Join GitHub today
GitHub is home to over 36 million developers working together to host and review code, manage projects, and build software together.
Sign upUpgrade to Hadoop 3.x #329
Comments
This comment has been minimized.
This comment has been minimized.
We'll have to sort out Preliminary StackOverFlow searching says we can implement our own version in Scala. Is that something we'd want to do, or should we look for a better solution to combine all the part files. ...and maybe there is a way to pull off a Anyway, I'll keep digging and check out the Hadoop 3.1.2 API docs. |
This comment has been minimized.
This comment has been minimized.
The Scala re-implementation looks good. I'd use it. |
This comment has been minimized.
This comment has been minimized.
Just dropping in to confirm that I have played around with the scala re-implementation for 3.1.1 and it works fine. |
ruebot
added
the
enhancement
label
Jul 25, 2019
This comment has been minimized.
This comment has been minimized.
Regarding the last paragraph of my initial comment:
Spark 2.x is not going to let us take advantage of Hadoop 3, since the official distribution of Spark is built with Hadoop 2. To confirm I tried removing the Hadoop exclusions from our fatjar (i.e., including Hadoop 3 in our fatjar), and then merging my S3-enabling changes with branch
The problem is This is the easiest way to get Spark + Hadoop 3:
Running spark-shell this way, the above test script works. This isn't that difficult, but it introduces complexity to the installation process, which is a turn off. Now we're asking people to download two packages, and it's easy enough to make a mistake setting the environment variable that will prevent Spark from starting. Maybe the Hadoop 3 upgrade isn't worth the hassle. (I say, after @ruebot does most of the work.) |
This comment has been minimized.
This comment has been minimized.
Funnily enough, I was going down the same path last night with the Hadoop-less Spark. It does complicate things significantly, and it might be worth talking this out on the next standup call we do (@ianmilligan1 @lintool sound good?) Worst case, we have this branch we can keep up-to-date from time-to-time, and maybe in the next year or so we'll see a Spark 3 with Hadoop 3 support. Based on following the Spark mailing list, that sounds reasonable since Spark 2.4.4 should be the last before 3.0.0.
|
This comment has been minimized.
This comment has been minimized.
Sounds good to chat about this, @ruebot – FWIW I don't think adding another download/export call is a game breaker if we think the benefits outweigh the disadvantages, as installing spark shell is already fairly complex. But I think it's worth further discussion for sure. |
jrwiebe commentedJul 24, 2019
AUT currently uses Hadoop 2.6.5. Though it is stable, at three years old it is beginning to show its age. I discovered this when testing S3 access (#319): hadoop-aws 2.6.5 is incapable of authenticating with temporary security credentials (probably an edge case) and with endpoints that require Signature Version 4 (many do). Upgrading to a current branch of Hadoop should be a matter of bringing other dependencies up to date, which might not be simple.
At present I would suggest going with version 3.1.2, the latest 3.1.x release. Its docs say: "This release is generally available (GA), meaning that it represents a point of API stability and quality that we consider production-ready." Or 3.0.0 -- all the Cloudera CDH 6 releases use that version, which is an indication its stability and wide use.
I'm not sure of the implications of using a distribution of Spark built with an older Hadoop to run our code that depends on Hadoop 3 (Spark 2.4.3 uses Hadoop 2.6.5). I wonder how the version conflicts would be resolved if we included the Hadoop 3 dependencies in our fatjar (we currently exclude them), and run it on Spark with Hadoop 2.6.5? I imagine it should work if we include Hadoop in the fatjar if we instruct people to use the version of Spark built without Hadoop. I think it's unreasonable to expect people to build Spark themselves, though.