archivesunleashed/aut

Introduces the new extractPDFDetailsDF() method and brings in changes to make our use of Tika's MIME type detection more efficient, as well as POM updates to use a shaded version of tika-parsers in order to eliminate a dependency version conflict that has long been troublesome.

- Updates getImageBytes to getBinaryBytes
- Refactor SaveImage class to more general SaveBytes, and saveToDisk to saveImageToDisk
- Only instantiate Tika when the DetectMimeTypeTika singleton object is first referenced. See https://git.io/fj7g0.
- Use TikaInputStream to enabler container-aware detection. Until now we were only using the default Mime Magic detection. See https://tika.apache.org/1.22/detection.html#Container_Aware_Detection.
- Added generic saveToDisk method to save a bytes column of a DataFrame to files
- Updates tests
- Resolves #302
- Further addresses #308
- Includes work by @ruebot, see #340 for all commits before squash

- Remove all underscore imports, except shapeless
- Address all scalastyle warnings
- Update scalastyle config for magic numbers, and null (only used in
tests)

* Replace computeHash with ComputeMD5; resolves #333.

* I suppose these are redundant.

- Update Tika to 1.22
- pom.xml surgery to get aut to build again with --packages

- This commit will fall under @ruebot, but @jrwiebe did the heavy lifting here; see #336 for his commits before they were squashed down.
- Resolves #265
- Resolves #263
- Update Scaladocs

* Update POM to access data stored in Amazon S3, per #319
* In RecordLoader detect FileSystem based on path.
* Resolves #319

- Remove explicit Guava dependency (should have been remove in
0e701b2)
- Update Scala to 2.11.12; aligns with Spark 2.4.3
- Update Scala test
- Update Shapeless
- Update Scala lang parsers
- Fix a typo in a test

- Run black and isort on Python files.
- Move Spark config to example file.
- Update gitignore for 7a61f0e
additions.

- Resolves #229

* Update to Spark 2.4.3 and update Tika to 1.20.

- Resolves #295
- Resolves #308
- Resolves #286
- Pulls in unfinished work by @jrwiebe and @borislin.

* Add patched lang-detector

- Resolves #322
- Resolves #206
- Resolves #194

…Record and WARCRecord variables. Also removes unused headerResponseFormat variable. (#316)

- Exclude slf4j binding logback-classic
(mojohaus/cobertura-maven-plugin#6 (comment))

- CVE-2018-8024 https://nvd.nist.gov/vuln/detail/CVE-2018-8024
- CVE-2018-1334 https://nvd.nist.gov/vuln/detail/CVE-2018-1334
- CVE-2018-17190 https://nvd.nist.gov/vuln/detail/CVE-2018-17190
- CVE-2018-11770 https://nvd.nist.gov/vuln/detail/CVE-2018-11770

* Log opening and closing of archive files as per #156
* Remove redundant log message. Spark already logs the file that is to be read when an executor computes an RDD.

* Remove deprecated ExtractGraph.scala file.
* Remove deprecated ExtractGraphTest.scala file.

…& #164 (#292)

* Resolves #198
* Resolves #164
* Add getHttpStatus to ArchiveRecord class & trait
  - add .getHttpStatus to potential outputs
  - add tests for .getHttpStatus calls
  - improve ArchiveRecord testing overall.
* Add .getArchiveFile feature to ArchiveRecordImpl.
  - add getArchiveFile to trait
  - add getArchiveFile for ArchiveRecordImpl
  - add tests for getArchiveFile.
* Other code style fixes.
* Include updates to tests.

…zipWithUniqueIds() (#289)

* Resolves #243 
* Create GEXF with proper ids instead of hash to avoid collisions.
* Add WriteGEXF files.
* Add WriteGraph file and test.
* Add test for Graphml output.
* Add xml escaping for edges.
* Add test case for non-escaped edges.
* Add additional tests to cover for more potential cases of graphml and gexf files.
* Coverage for null cases in urls.

- CVE-2018-1338
- CVE-2018-11762
- CVE-2018-11761
- CVE-2016-6809
- CVE-2018-1339
- CVE-2018-11796
- CVE-2016-4434
- CVE-2018-1335

* Fix backslash in URL
* Add backslash test in ExtractDomainTest

- Follow on to
72cb5e2
- https://nvd.nist.gov/vuln/detail/CVE-2018-7489

- CVE-2017-752
- See also: https://nvd.nist.gov/vuln/detail/CVE-2017-7525