archivesunleashed/aut

* Add additional filters for fextFiles; resolves #362.

- Add filedesc, and dns filter (arc files)
- Add test case

- add discardMimeTypesDF
- add discardDateDF
- add discardUrlsDF
- add discardDomainsDF
- update tests
- addresses #223

- Add keepValidPagesDF
- Add HTTP status code column to all()
- Add test for keepValidPagesDF
- Addresses #223

- Add ExtractDateDF test
- Fix conditional logic of textFiles filter to resolve #390
- Add test for conditional logic fix for #390
- Remove cruft ExtractUrls, left over from Twitter analysis removal
(see:
https://github.com/lintool/warcbase/blob/cab311ed8b0bceb666865fa76dd3bc5a86402e13/warcbase-core/src/test/scala/org/warcbase/spark/matchbox/ExtractUrlsTest.scala)
- Tweak null/nothing on a few tests

- Add DetectLanguageDF
- Add ExtractBoilerpipeTextDF
- Add ExtractDateDF
- Update tests
- Rename existing ExtractDate, ExtractBoilerpipeText, DetectLanguage udfs by appending RDD
- Partially addresses #223

- Add tests for ExtractPopularImagesDF
- Rename ExtractPopularImages to ExtractPopularImagesRDD
- Addresses #223

- Add `all()` DataFrame method 
- Refactor fixity DataFrame UDFs
- Add ComputeImageSize UDF
- Add Python implementation of `all()`
- Addresses #223

- Part of work on #233

- Addresses #223

- Extend GetExtensionMime, ExtractImageLinks, ComputeMD5, and ComputeSHA1 to DataFrames
- Addresses #223

- Resolves #366

- Fully resolves #297 
- Overrides NER Classifier output to PERSON -> persons, LOCATION -> locations, ORGANIZATION -> organizations

- Add df ExtractLinks udf
- Add test

Tweaks the style of the license badge to look consistent with the other badges.

Resolves #368.

- Update Stanford core NLP
- Format NER output in json
- Add getPayloadDigest to ArchiveRecord
- Add test for getPayloadDigest
- Add payload digest to NER output
- Remove extractFromScrapeText
- Remove extractFromScrapeText test
- TODO: PERSON -> persons, LOCATION -> locations, ORGANIZATION -> organizations (involves writing a new class or overriding NER output 🤢

- Replace ExtractBaseDomain with ExtractDomain
- Closes #367
- Address bug in ArcTest; RemoveHTML -> RemoveHttpHeader
- Closes #369
- Wraps RemoveHttpHeader and RemoveHTML for use in data frames.
- Partially addresses #238
- Updates tests where necessary
- Punts on #368 UDF CaMeL cASe consistency issues

- Update tests where needed
- Add SHA1 method to ExtractImageDetails
- Add SHA1 to DataFrames binary extraction and analysis

- Add status code filter to keepValidPages
- Add MimeTypeTika to valid pages DF
- Update tests since we filter more and better now 😄
- Resolves #359

- Clean up doc comments
- Add test
- Resolves #352

- Add tests a few more filters in RecordLoader
- Add binary extration DataFrameLoader tests

* Add binary extration DataFrames to PySpark.
- Address #190
- Address #259
- Address #302
- Address #303
- Address #304
- Address #305
- Address #306
- Address #307
- Resolves #350 
- Update README

- Update LICENSE file to full Apache 2.0 license
- Reconfigure license-maven-plugin
- Update all license headers in java and scala files to include
copyright year, and project name
- Move LICENSE_HEADER.txt to config
- Update scalastyle config

This PR implements the strategy described in the discussion of the above issue to get an extension for a file described by a URL and a MIME type. It creates a GetExtensionMime object in the matchbox.

This PR also removes most of the filtering by URL from the image, audio, video, presentation, spreadsheet, and word processor document extraction methods, since these were returning false positives. (CSV and TSV files are a special case, since Tika detects them as "text/plain" based on content.)

Finally, I have inserted toLowerCase into the getUrl.endsWith() filter tests, which could possibly bring in some more CSV and TSV files

* Adds method for getting a file extension from a MIME type.
* Add getExtensions method to DetectMimeTypeTika.
* Matchbox object to get extension of URL
* Use GetExtensionMime for extraction methods; minor fixes.
* Remove tika-parsers classifier
* Remove most filtering by file extension from binary extraction methods; add CSV/TSV special cases.
* Fix GetExtensionMime case where URL has no extension but a MIME type is detected
* Insert `toLowerCase` into `getUrl.endsWith()` calls in io.archivesunleashed.packages; apply to `FilenameUtils.getExtension` in `GetExtensionMime`.
* Remove filtering on URL for audio, video, and images.
* Remove filtering on URL for images; add DF fields to image extraction
* Remove saveImageToDisk and its test
* Remove robots.txt check and extraneous imports
* Close files so we don't get too many files open again.
* Add GetExtensionMimeTest
* Resolve #343

- Add keep and discard by http status RecordLoader
- Add tests
- Clean up/add doc comments in RecordLoader
- Resolve #315

- Add Word Processor DF and binary extraction
- Add Spreadsheets DF and binary extraction
- Add Presentation Program DF and binary extraction
- Add Text files DF and binary extraction
- Add tests for new DF and binary extractions
- Add test fixtures for new DF and binary extractions
- Resolves #303
- Resolves #304
- Resolves #305
- Use aut-resources repo to distribute our shaded tika-parsers 1.22
- Close TikaInputStream
- Add RDD filters on MimeTypeTika values
- Add CodeCov configuration yaml
- Includes work by @jrwiebe, see #346 for all commits before squash

Ivy couldn't handle it, and specifying one for the custom tika-parsers artifact
was unnecessary.

…344)

- Move audio, pdf, and video DF extraction to tuple map
- Provide two MimeType columns; mime_type_web_server and mime_type_tika
- Update tests
- Resolves #342