Join GitHub today
GitHub is home to over 36 million developers working together to host and review code, manage projects, and build software together.
Sign upUpdate to Spark 2.4.3 and update Tika to 1.20. #321
+158
−91
Conversation
ruebot
requested review from
lintool and
ianmilligan1
Jul 4, 2019
ruebot
reviewed
Jul 4, 2019
| @@ -178,9 +178,13 @@ class CommandLineApp(conf: CmdAppConf) { | |||
|
|
|||
| def save(d: Dataset[Row]): Unit = { | |||
| if (!configuration.partition.isEmpty) { | |||
| d.coalesce(configuration.partition()).write.csv(saveTarget) | |||
| d.coalesce(configuration.partition()).write | |||
| .option("timestampFormat", "yyyy/MM/dd HH:mm:ss ZZ") | |||
This comment has been minimized.
This comment has been minimized.
ruebot
Jul 4, 2019
Author
Member
FYI: These were added for the newer version of Tika (underlying libraries). Timestamp format is required.
This comment has been minimized.
This comment has been minimized.
codecov-io
commented
Jul 4, 2019
•
Codecov Report
@@ Coverage Diff @@
## master #321 +/- ##
==========================================
+ Coverage 75.95% 76.08% +0.12%
==========================================
Files 41 41
Lines 1148 1154 +6
Branches 200 200
==========================================
+ Hits 872 878 +6
Misses 209 209
Partials 67 67
Continue to review full report at Codecov.
|
ianmilligan1
requested changes
Jul 4, 2019
|
Putting through its paces, things work well with example data (including DataFrames) except the language filter. The following script: import io.archivesunleashed._
import io.archivesunleashed.matchbox._
RecordLoader.loadArchives("example.arc.gz", sc)
.keepValidPages()
.keepDomains(Set("www.archive.org"))
.keepLanguages(Set("fr"))
.map(r => (r.getCrawlDate, r.getDomain, r.getUrl, RemoveHTML(r.getContentString)))
.saveAsTextFile("plain-text-fr/")Fails with et al. [full log here] Given the Tika issues around the language detector, probably somewhat to be expected? |
This comment has been minimized.
This comment has been minimized.
|
@ianmilligan1 That's what I was afraid of. |
This comment has been minimized.
This comment has been minimized.
|
Looks like I'm getting a slightly different error with: ERROR: I'll dig into it more. I think it might be a guava issue. |
This comment has been minimized.
This comment has been minimized.
|
Ah cool. We have 16 different occurrences of |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
ruebot commentedJul 4, 2019
•
edited
GitHub issue(s):
What does this Pull Request do?
Update to Spark 2.4.3 and update Tika to 1.20, and pulls in unfinished work by @jrwiebe and @borislin.
I had to tweak the language tests. I believe is because of the updated language detection in Tika, and that's why the test fixtures changed. Though, I'm not a 100% certain. So, definitely need a sanity check here.
How should this be tested?
TravisCI should take care of the first bit.
@ianmilligan1 would you mind putting this through a bunch of examples?
rm -rf ~/.m2/repository/* && mvn clean installrm -rf ~/.ivy2/* && spark-shell --packages "io.archivesunleashed:aut:0.17.1-SNAPSHOT"Additional Notes:
Once we get this in, and #318, might as well cut a 0.18.0 release. There's a fair bit done since the last release. It'd be nice to get #317 sorted, but that isn't a blocker.
@jrwiebe if you have some time, can you give this a sanity look too, since I'll built on a bit of what you were previously digging into.