Update Tika to 1.22; address security alerts. #337

ruebot · Aug 6, 2019

What does this Pull Request do?

Update Tika to 1.22
pom.xml surgery to get aut to build again with --packages
CVE-2019-10093, CVE-2019-10094, CVE-2019-10088

How should this be tested?

TravisCI
rm -rf ~/.m2/repository/* && mvn clean install && rm -rf ~/.ivy2/* && ~/bin/spark-2.4.3-bin-hadoop2.7/bin/spark-shell --packages io.archivesunleashed:aut:0.17.1-SNAPSHOT -i ~/318-test-lang.scala

318-test-lang.scala

import io.archivesunleashed._
import io.archivesunleashed.matchbox._

RecordLoader.loadArchives("/home/nruest/Projects/au/sample-data/geocites/1/*gz", sc)
.keepValidPages()
.keepDomains(Set("geocities.com"))
.keepLanguages(Set("en"))
.map(r => (r.getCrawlDate, r.getDomain, r.getUrl, RemoveHTML(r.getContentString)))
.saveAsTextFile("/tmp/plain-text-en/")

sys.exit

Additional Notes

@jrwiebe this might throw a wrench in our other work, but hopefully shouldn't.

ianmilligan1 · Aug 6, 2019

ianmilligan1 approved these changes Aug 6, 2019

View changes

Builds nicely locally. I tested language extraction on CPP data and it worked very well.

codecov-io · Aug 6, 2019

Codecov Report

Merging #337 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #337   +/-   ##
=======================================
  Coverage   75.97%   75.97%           
=======================================
  Files          39       39           
  Lines        1124     1124           
  Branches      197      197           
=======================================
  Hits          854      854           
  Misses        205      205           
  Partials       65       65

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 605afcc...1a3d2b9. Read the comment docs.

jrwiebe · Aug 6, 2019

@jrwiebe this might throw a wrench in our other work, but hopefully shouldn't.

If you're referring to our use of com.github.netarchivesuite.language-detector, we're good; it still works.

ruebot · Aug 6, 2019

Cool. I tested on my end too before I did the PR. Glad things are working on your end too!

ruebot requested a review from ianmilligan1 Aug 6, 2019

ianmilligan1 deleted the tika-CVE-2019-10093-CVE-2019-10094-CVE-2019-10088 branch Aug 6, 2019

archivesunleashed/aut

Update Tika to 1.22; address security alerts. #337

Update Tika to 1.22; address security alerts. #337

ruebot commented Aug 6, 2019

ruebot requested a review from ianmilligan1 Aug 6, 2019

ianmilligan1 approved these changes Aug 6, 2019

View changes

ianmilligan1 left a comment

This comment has been minimized.

codecov-io commented Aug 6, 2019

ianmilligan1 merged commit `2d14b92` into master Aug 6, 2019
3 checks passed

3 checks passed

ianmilligan1 deleted the tika-CVE-2019-10093-CVE-2019-10094-CVE-2019-10088 branch Aug 6, 2019

This comment has been minimized.

jrwiebe commented Aug 6, 2019

This comment has been minimized.

ruebot commented Aug 6, 2019

archivesunleashed/aut

Join GitHub today

Update Tika to 1.22; address security alerts. #337

Conversation

ruebot commented Aug 6, 2019

What does this Pull Request do?

How should this be tested?

Additional Notes

ruebot requested a review from ianmilligan1 Aug 6, 2019

ianmilligan1 approved these changes Aug 6, 2019 View changes

ianmilligan1 left a comment

This comment has been minimized.

codecov-io commented Aug 6, 2019

Codecov Report

Hide details View details ianmilligan1 merged commit 2d14b92 into master Aug 6, 2019 3 checks passed

3 checks passed

ianmilligan1 deleted the tika-CVE-2019-10093-CVE-2019-10094-CVE-2019-10088 branch Aug 6, 2019

This comment has been minimized.

jrwiebe commented Aug 6, 2019

This comment has been minimized.

ruebot commented Aug 6, 2019

ianmilligan1 approved these changes Aug 6, 2019

View changes

ianmilligan1 merged commit `2d14b92` into master Aug 6, 2019
3 checks passed