archivesunleashed/aut

…03-304-305

…mit 😱

false positives.

…03-304-305

- Add WordProcessor DF and binary extraction
- Add Spreadsheets DF and binary extraction
- Add Presentation Program DF and binary extraction
- Add tests for new DF and binary extractions
- Add test fixture for new DF and binary extractions
- Resolves #303
- Resolves #304
- Resolves #305
- Back out 39831c2 (We _might_ not have
to do this)

Ivy couldn't handle it, and specifying one for the custom tika-parsers artifact
was unnecessary.

…344)

- Move audio, pdf, and video DF extraction to tuple map
- Provide two MimeType columns; mime_type_web_server and mime_type_tika
- Update tests
- Resolves #342

- Add Audio & Video binary extraction.
- Add filename, and extenstion column to audio, pdf, and video DF
- Pass binary bytes instread of string to DetectMimeTypeTika in DF (s/getContentString/getBinaryBytes)
- Updates saveToDisk to use file extension from DF column
- Adds tests for Audio, PDF, and Video DF extraction
- Add test fixtures for Audio, PDF, and Video DF extraction
- Rename SaveBytesTest to SaveImageBytes test
- Eliminate bytes->string->bytes conversion that was causing data loss in DetectMimeTypeTika
- Update tika-parsers dep from JitPack
- Remove tweet cruft
- Resolves #306
- Resolves #307
- Includes work by @jrwiebe, see #341 for all commits before squash

Introduces the new extractPDFDetailsDF() method and brings in changes to make our use of Tika's MIME type detection more efficient, as well as POM updates to use a shaded version of tika-parsers in order to eliminate a dependency version conflict that has long been troublesome.

- Updates getImageBytes to getBinaryBytes
- Refactor SaveImage class to more general SaveBytes, and saveToDisk to saveImageToDisk
- Only instantiate Tika when the DetectMimeTypeTika singleton object is first referenced. See https://git.io/fj7g0.
- Use TikaInputStream to enabler container-aware detection. Until now we were only using the default Mime Magic detection. See https://tika.apache.org/1.22/detection.html#Container_Aware_Detection.
- Added generic saveToDisk method to save a bytes column of a DataFrame to files
- Updates tests
- Resolves #302
- Further addresses #308
- Includes work by @ruebot, see #340 for all commits before squash

- Remove all underscore imports, except shapeless
- Address all scalastyle warnings
- Update scalastyle config for magic numbers, and null (only used in
tests)

* Replace computeHash with ComputeMD5; resolves #333.

* I suppose these are redundant.

- Update Tika to 1.22
- pom.xml surgery to get aut to build again with --packages

- This commit will fall under @ruebot, but @jrwiebe did the heavy lifting here; see #336 for his commits before they were squashed down.
- Resolves #265
- Resolves #263
- Update Scaladocs

* Update POM to access data stored in Amazon S3, per #319
* In RecordLoader detect FileSystem based on path.
* Resolves #319

- Remove explicit Guava dependency (should have been remove in
0e701b2)
- Update Scala to 2.11.12; aligns with Spark 2.4.3
- Update Scala test
- Update Shapeless
- Update Scala lang parsers
- Fix a typo in a test

- Run black and isort on Python files.
- Move Spark config to example file.
- Update gitignore for 7a61f0e
additions.

- Resolves #229

* Update to Spark 2.4.3 and update Tika to 1.20.

- Resolves #295
- Resolves #308
- Resolves #286
- Pulls in unfinished work by @jrwiebe and @borislin.

* Add patched lang-detector

- Resolves #322
- Resolves #206
- Resolves #194

…Record and WARCRecord variables. Also removes unused headerResponseFormat variable. (#316)

- Exclude slf4j binding logback-classic
(mojohaus/cobertura-maven-plugin#6 (comment))