Skip to content

/ aut

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Spark 3.0.0 + Hadoop 3.2 + Java 11 support. #385

Closed
wants to merge 50 commits into from
Closed

Spark 3.0.0 + Hadoop 3.2 + Java 11 support. #385

wants to merge 50 commits into from
+78 −263

Conversation

@ruebot
Copy link
Member

ruebot commented Nov 26, 2019

GitHub issue(s):

What does this Pull Request do?

This PR is #375 + Hadoop 3.2 support.

How should this be tested?

Same as #375 + It should be tested with spark-3.0.0-preview-bin-hadoop3.2.

Additional Notes:

Same as #375
ruebot and others added 30 commits Jul 25, 2019
@ruebot
Update Hadoop to 3.1.2
Verified
This commit was signed with a verified signature.
ruebot Nick Ruest
GPG key ID: 417FAF1A0E1080CD Learn about signing commits
3564749 
- Add copyMerge implementation in Scala (copyMerge is deprecated in
Hadoop 3)
- Update NERCombinedJson to use new copyMerge implementation
- 40 days and nights wandering the desert of pom.xml
@jrwiebe
Restore from master dependency on com.github.netarchivesuite.language…
e04aec0 
…-detector.
@ruebot
Initial Java 11 setup (#356); still need Spark compiled with Java 11 …
Verified
This commit was signed with a verified signature.
ruebot Nick Ruest
GPG key ID: 417FAF1A0E1080CD Learn about signing commits
04a80d6 
…to get tests passing https://stackoverflow.com/questions/53583199/pyspark-error-unsupported-class-file-major-version-55
@ruebot
Merge branch 'master' of github.com:archivesunleashed/aut into issue-356
Verified
This commit was signed with a verified signature.
ruebot Nick Ruest
GPG key ID: 417FAF1A0E1080CD Learn about signing commits
1c2ff7c
@ruebot
Update to Spark 3.0.0
Verified
This commit was signed with a verified signature.
ruebot Nick Ruest
GPG key ID: 417FAF1A0E1080CD Learn about signing commits
8a51a87 
- Some hacks to get a sucessful build
- Definitely need to loop back and clean-up a whole lot!
- Addresses #356
@ruebot
#356 scala updates
Verified
This commit was signed with a verified signature.
ruebot Nick Ruest
GPG key ID: 417FAF1A0E1080CD Learn about signing commits
06764f7
@ruebot
#356 verify-javadocs fixed -- make sure you have OpenJDK-11 fully ins…
Verified
This commit was signed with a verified signature.
ruebot Nick Ruest
GPG key ID: 417FAF1A0E1080CD Learn about signing commits
Loading status checks…
d301adc 
…talled 🤦, and a bunch more pom cleanup.
@ruebot
#356 update TravisCI; don't allow failures on Java 11.
Verified
This commit was signed with a verified signature.
ruebot Nick Ruest
GPG key ID: 417FAF1A0E1080CD Learn about signing commits
Loading status checks…
c86c398
@ruebot
#356 🤦 oops. uncommented that, and pushed.
Verified
This commit was signed with a verified signature.
ruebot Nick Ruest
GPG key ID: 417FAF1A0E1080CD Learn about signing commits
Loading status checks…
4a54d67
@ruebot
#356 Replace Cobertura with JaCoCo, and allow Java 8 to fail for now.
Verified
This commit was signed with a verified signature.
ruebot Nick Ruest
GPG key ID: 417FAF1A0E1080CD Learn about signing commits
Loading status checks…
41d5f45
@ruebot
#356 -- forgot to remove cobertura TravisCI build.
Verified
This commit was signed with a verified signature.
ruebot Nick Ruest
GPG key ID: 417FAF1A0E1080CD Learn about signing commits
Loading status checks…
b002a02
@ruebot
#356 remove test line.
Verified
This commit was signed with a verified signature.
ruebot Nick Ruest
GPG key ID: 417FAF1A0E1080CD Learn about signing commits
Loading status checks…
d1692b5
@ruebot
#356 wave goodbye to Java 8.
Verified
This commit was signed with a verified signature.
ruebot Nick Ruest
GPG key ID: 417FAF1A0E1080CD Learn about signing commits
Loading status checks…
a1b9607
@ruebot
#356 add Java 11 and Spark 3.0.0+ to README
Verified
This commit was signed with a verified signature.
ruebot Nick Ruest
GPG key ID: 417FAF1A0E1080CD Learn about signing commits
Loading status checks…
3b6744c
@ruebot
Merge branch 'master' into issue-356
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.
GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits
Loading status checks…
d59f9e6
@ruebot
Merge branch 'master' into issue-356
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.
GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits
Loading status checks…
eecd41a
@ruebot
Merge branch 'master' into issue-356
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.
GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits
Loading status checks…
fbc31d4
@ruebot
Merge branch 'master' into issue-356
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.
GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits
Loading status checks…
82d588a
@ruebot
Merge branch 'master' into issue-356
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.
GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits
Loading status checks…
f662dee
@ruebot
Merge branch 'master' into issue-356
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.
GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits
Loading status checks…
481dbf4
@ruebot
Merge branch 'issue-356' of github.com:archivesunleashed/aut into iss…
Verified
This commit was signed with a verified signature.
ruebot Nick Ruest
GPG key ID: 417FAF1A0E1080CD Learn about signing commits
Loading status checks…
65249ba 
…ue-329
@ruebot
Fix copyright header
Verified
This commit was signed with a verified signature.
ruebot Nick Ruest
GPG key ID: 417FAF1A0E1080CD Learn about signing commits
Loading status checks…
3ab17bf
@ruebot
Merge branch 'master' into issue-329
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.
GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits
Loading status checks…
a985973
@ruebot
Merge branch 'master' into issue-329
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.
GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits
Loading status checks…
6ec08a6
@ruebot
Merge branch 'master' into issue-329
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.
GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits
Loading status checks…
b82ee17
@ruebot
Merge branch 'master' into issue-329
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.
GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits
Loading status checks…
d3aa957
@ruebot
Merge branch 'master' into issue-329
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.
GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits
Loading status checks…
f89a59b
@ruebot
Spark 3.0.0-preview2
Verified
This commit was signed with a verified signature.
ruebot Nick Ruest
GPG key ID: 417FAF1A0E1080CD Learn about signing commits
Loading status checks…
85f2462
@ruebot
Merge branch 'master' into issue-329
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.
GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits
Loading status checks…
08bbed5
@ruebot
Merge branch 'master' into issue-329
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.
GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits
Loading status checks…
a84667e
ruebot added 7 commits Jan 7, 2020
@ruebot
Merge branch 'master' into issue-329
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.
GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits
Loading status checks…
00b817d
@ruebot
Merge branch 'master' into issue-329
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.
GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits
Loading status checks…
160f7b1
@ruebot
Merge branch 'master' into issue-329
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.
GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits
Loading status checks…
8f5bee1
@ruebot
Merge branch 'master' of github.com:archivesunleashed/aut into issue-329
Verified
This commit was signed with a verified signature.
ruebot Nick Ruest
GPG key ID: 417FAF1A0E1080CD Learn about signing commits
Loading status checks…
98b1b32
@ruebot
Merge branch 'master' into issue-329
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.
GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits
Loading status checks…
f072ca9
@ruebot
Merge branch 'master' into issue-329
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.
GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits
Loading status checks…
384bbc7
@ruebot
Merge branch 'master' into issue-329
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.
GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits
Loading status checks…
96ced04
@ruebot ruebot added this to In Progress in DataFrames and PySpark Feb 5, 2020
@ruebot ruebot added this to In Progress in 1.0.0 Release of AUT Feb 5, 2020
ruebot added 3 commits Feb 6, 2020
@ruebot
Merge branch 'master' into issue-329
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.
GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits
Loading status checks…
2bc906e
@ruebot
Merge branch 'master' into issue-329
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.
GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits
Loading status checks…
3c72660
@ruebot
Merge branch 'master' into issue-329
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.
GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits
Loading status checks…
f56b0ca
@ruebot
Copy link
Member Author

ruebot commented Apr 11, 2020

I believe we're going to continue to fail here until this is resolved. If that is resolved, it should also help out with our other hacks around making Tika work.
ruebot added 2 commits Apr 14, 2020
@ruebot
Merge branch 'master' of github.com:archivesunleashed/aut into issue-329
Verified
This commit was signed with a verified signature.
ruebot Nick Ruest
GPG key ID: 417FAF1A0E1080CD Learn about signing commits
Loading status checks…
eb21dd7
@ruebot
Merge branch 'master' into issue-329
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.
GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits
Loading status checks…
cc4274c
@ruebot
ruebot reviewed Apr 15, 2020
View changes
src/main/scala/io/archivesunleashed/util/copyMerge.scala Outdated
import java.io.IOException

/** Reimplementation of FileUtil.copyMerge . */
object CopyMerge {

This comment has been minimized.

Sign in to view
Copy link
@ruebot

ruebot Apr 15, 2020

Author Member

@lintool we're adding this, since copyMerge was deprecated and removed from Hadoop 3. Do we need to actually implement it again, or should we just use coalesce like we do here?

This comment has been minimized.

Sign in to view
Copy link
@lintool

lintool Apr 15, 2020

Member

I don't recall this being my code... but I think that's fine.
ruebot added 4 commits Apr 15, 2020
@ruebot
Merge branch 'master' into issue-329
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.
GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits
Loading status checks…
68856e3
@ruebot
Merge branch 'master' into issue-329
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.
GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits
Loading status checks…
3b5900d
@ruebot
Merge branch 'master' of github.com:archivesunleashed/aut into issue-329
Verified
This commit was signed with a verified signature.
ruebot Nick Ruest
GPG key ID: 417FAF1A0E1080CD Learn about signing commits
Loading status checks…
c873905
@ruebot
Merge branch 'master' into issue-329
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.
GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits
Loading status checks…
86dd21a
@ruebot ruebot removed this from In Progress in DataFrames and PySpark May 19, 2020
ruebot added 2 commits May 20, 2020
@ruebot
Merge branch 'master' into issue-329
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.
GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits
Loading status checks…
529b97b
@ruebot
Merge branch 'master' into issue-329
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.
GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits
Loading status checks…
a268059
@ruebot ruebot mentioned this pull request May 27, 2020
Closed
ruebot added 2 commits Jun 17, 2020
@ruebot
Merge branch 'master' of github.com:archivesunleashed/aut into issue-329
Verified
This commit was signed with a verified signature.
ruebot Nick Ruest
GPG key ID: 417FAF1A0E1080CD Learn about signing commits
Loading status checks…
fcabc4a
@ruebot
Don't need this anymore since NER is gone.
Verified
This commit was signed with a verified signature.
ruebot Nick Ruest
GPG key ID: 417FAF1A0E1080CD Learn about signing commits
Loading status checks…
0e23760
@ruebot ruebot removed this from In Progress in 1.0.0 Release of AUT Jun 18, 2020
@ruebot
Copy link
Member Author

ruebot commented Jun 29, 2020

Going to close this, and re-open it against main so we can remove master #490.
@ruebot ruebot closed this Jun 29, 2020
@ruebot ruebot mentioned this pull request Jun 29, 2020
Draft
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lintool lintool

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Linked issues

Successfully merging this pull request may close these issues.

None yet

3 participants
@ruebot @lintool @jrwiebe
You can’t perform that action at this time.