SinghGursimran / aut

…chivesunleashed#422)

- Update ExtractImageLinksRDD to grab alt text
- Add alt_text column to imagegraph
- Update tests

…ivesunleashed#421)

* Rename imageLinks to imagegraph; resolves archivesunleashed#419

- Fully resolves this issue archivesunleashed/docker-aut#19
- archivesunleashed/docker-aut@37ce4e2
- archivesunleashed/docker-aut@082907a
- archivesunleashed/docker-aut@baee431

…sunleashed#416)

- Clean up test descriptions
- Rename typo filename

- Add test
- Addresses archivesunleashed#223

…d#414)

- Resolves archivesunleashed#413
- Update tests where necessary

…Addresses archivesunleashed#372.

- .all() column HttpStatus to http_status_code
- Adds archive_filename to .all()
- Significant README updates for setup
- See also: archivesunleashed/aut-docs#39

- Looks like repos are forcing https to be used now:
[WARNING] repository metadata for: 'artifact joda-time:joda-time' could not be retrieved from repository: maven due to an error: Failed to transfer file: http://repo.maven.apache.org/maven2/joda-time/joda-time/maven-metadata.xml. Return code is: 501 , ReasonPhrase:HTTPS Required.

- Clean-up variable names in RecordDFTest.scala
- Remove dos line endings on a number of files

- Addresses archivesunleashed#402

…d#401).

- Partially addresses  archivesunleashed#223 
- Add discardContentDF
- Add discardUrlPatternsDF
- Add discardLanguagesDF
- Add keepImagesDF
- Add keepContentDF
- Add keepUrlPatternsDF
- Add keepLanguagesDF
- Update tests

…shed#399)

- Add discardHttpStatusDF
- Add keepMimeTypesDF
- Add keepMimeTypesTikaDF
- Update tests

- Partially address archivesunleashed#223 
- Add keepHttpStatusDF
- Add keepDateDF
- Add keepUrlsDF
- Add keepDomainsDF
- Add tests

…shed#395)

…d#392. (archivesunleashed#394)

…archivesunleashed#393)

* Add additional filters for fextFiles; resolves archivesunleashed#362.

- Add filedesc, and dns filter (arc files)
- Add test case

- add discardMimeTypesDF
- add discardDateDF
- add discardUrlsDF
- add discardDomainsDF
- update tests
- addresses archivesunleashed#223

- Add keepValidPagesDF
- Add HTTP status code column to all()
- Add test for keepValidPagesDF
- Addresses archivesunleashed#223

- Add ExtractDateDF test
- Fix conditional logic of textFiles filter to resolve archivesunleashed#390
- Add test for conditional logic fix for archivesunleashed#390
- Remove cruft ExtractUrls, left over from Twitter analysis removal
(see:
https://github.com/lintool/warcbase/blob/cab311ed8b0bceb666865fa76dd3bc5a86402e13/warcbase-core/src/test/scala/org/warcbase/spark/matchbox/ExtractUrlsTest.scala)
- Tweak null/nothing on a few tests

- Add DetectLanguageDF
- Add ExtractBoilerpipeTextDF
- Add ExtractDateDF
- Update tests
- Rename existing ExtractDate, ExtractBoilerpipeText, DetectLanguage udfs by appending RDD
- Partially addresses archivesunleashed#223

…ed#386)

…hed#382).

- Add tests for ExtractPopularImagesDF
- Rename ExtractPopularImages to ExtractPopularImagesRDD
- Addresses archivesunleashed#223

- Add `all()` DataFrame method 
- Refactor fixity DataFrame UDFs
- Add ComputeImageSize UDF
- Add Python implementation of `all()`
- Addresses archivesunleashed#223

- Part of work on archivesunleashed#233

- Addresses archivesunleashed#223

- Extend GetExtensionMime, ExtractImageLinks, ComputeMD5, and ComputeSHA1 to DataFrames
- Addresses archivesunleashed#223

…vesunleashed#379)

- Resolves archivesunleashed#366

).

- Fully resolves archivesunleashed#297 
- Overrides NER Classifier output to PERSON -> persons, LOCATION -> locations, ORGANIZATION -> organizations

…eashed#377)

- Add df ExtractLinks udf
- Add test

Tweaks the style of the license badge to look consistent with the other badges.

Resolves archivesunleashed#368.