Skip to content
Please note that GitHub no longer supports your web browser.

We recommend upgrading to the latest Google Chrome or Firefox.

Learn more
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Spark 3.0.0 + Java 11 support. #375

Draft
wants to merge 13 commits into
base: master
from

Conversation

@ruebot
Copy link
Member

ruebot commented Nov 10, 2019

GitHub issue(s): #356

What does this Pull Request do?

Mostly pom.xml updates. Lots of cleanup, and updates. Big stuff, update to Apache Spark 3.0.0, and Java 11.

How should this be tested?

  • TravisCI
  • Exhaustive regression testing; basically hit everything in the documentation, and make sure it works.

Additional Notes:

  • BIG KICKER
    • I cannot get this to work with --packages. We hit an ugly dependency wall really quick. If we exclude, all the tests fail really bad. If I explicitly include it as a dependency, same thing. Tests fail really bad.
  • I'm going to leave this as a draft, and we shouldn't merge until there is an official Spark 3.0.0 release, and we make a decision on --packages.
  • We'll squash this all down, and make a nice commit message when the time comes.
ruebot added 5 commits Aug 31, 2019
- Some hacks to get a sucessful build
- Definitely need to loop back and clean-up a whole lot!
- Addresses #356
…talled :facepalm:, and a bunch more pom cleanup.
@ruebot ruebot requested review from lintool and ianmilligan1 Nov 10, 2019
ruebot added 4 commits Nov 10, 2019
@codecov

This comment has been minimized.

Copy link

codecov bot commented Nov 10, 2019

Codecov Report

Merging #375 into master will increase coverage by 2.56%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master     #375      +/-   ##
============================================
+ Coverage     76.36%   78.92%   +2.56%     
- Complexity        0       61      +61     
============================================
  Files            40       40              
  Lines          1413     1082     -331     
  Branches        268      114     -154     
============================================
- Hits           1079      854     -225     
+ Misses          217      184      -33     
+ Partials        117       44      -73
@lintool

This comment has been minimized.

Copy link
Member

lintool commented Nov 10, 2019

This is awesome! Did a quick check, everything looks sane to me.

ruebot added 3 commits Nov 10, 2019
@ianmilligan1

This comment has been minimized.

Copy link
Member

ianmilligan1 commented Nov 10, 2019

Thanks for this @ruebot! Have built it locally, but will take my time to exhaustively run through the docs before giving it the thumbs up.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.