archivesunleashed/aut

* Fix backslash in URL
* Add backslash test in ExtractDomainTest

- Follow on to
72cb5e2
- https://nvd.nist.gov/vuln/detail/CVE-2018-7489

- CVE-2017-752
- See also: https://nvd.nist.gov/vuln/detail/CVE-2017-7525

…les. (#272)

* Fix exception when processing corrupted ARC files
* Filter out non-empty archive files in loadArchives()
* Fix archive files path pattern
* Resolves #246
* Resolves #271 
* Resolves #258

* ExtractBoilerpipeText now removes headers.

)

- Adds:
  - retweet_count
  - favorite_count
  - in_reply_to_status_id_str
  - in_reply_to_user_id_str
  - in_reply_to_screen_name
  - source
  - user.protected
  - user.profile_image_url
  - user.description
  - user.location
  - user.name
  - user.url
  - user.time_zone
- Updates some doc comments
- Updates tests

…). (#249)

* Fully resolves #196 
* Resolves #212

* Deal with wildcard import lint issues.
* Fix some magic numbers & duplicate string runs.
* Lint fixes, mostly explicit import warnings.
* All other scalastyle issues require refactoring.

…ssue 203 (#245)

* pom.xml change for GraphX
* Changes for GraphXSLS
* Changes for SLS graph
* Changes for GraphX
* Changes for converting WARC RDD to GraphX object
* Rename extractor to ExtractGraphX
* Various lint fixes (usually Magic Numbers)
* Remove illegal imports from scala style (we use wildcard imports a lot)
* Add WriteGraphXMLTest.

* Make the TravisCI build less verbose since we're hitting the 4MB log limit.
* Pin site.plugin and project-info-reports.plugin so mvn site builds.
  - See:
    - https://stackoverflow.com/questions/51091539/maven-site-plugins-3-3-java-lang-classnotfoundexception-org-apache-maven-doxia
    - https://travis-ci.org/archivesunleashed/aut/jobs/408259462#L3201-L3202

…esolve #235 (#236)

* initial implementation
* Data frame implementation of extractors.
* fix documentation.

* Save images from dataframe to disk
* Fix spacing
* Move save images to inline
* Refactor to chain and fix concurrency issue
* Add save image test
* Move saveToDisk to df

* Add Extract Image Details API
* Change check for jpeg and fix spacing
* Add tiff parser
* Use AutoDetectParser and read Numeric fields
* Use ComputeImageSize
* Hex encode hash and base64 encode image bytes
* Fix test
* Change df column names

…an be run from command line (#225)

* Resolves issue 195. Implement DomainFrequency, DomainGraph and PlainText extractor that can be run via command line in spark-submit, along with their tests

* Restructure CommandLineAppRunner to make it more robust. Add option to write GEXF output for DomainGraphExtractor (enable via --output-format GEXF). Add support for multiple input files. Other polish and cleanup.

* Extract Image Links DF API
* Add extract image links text
* Remove unnecessary comment from test
* Add doc comments
* Addresses #220

- Update tests to use workaround for SPARK-2243
- Comment out ExtractGraph test as per https://github.com/archivesunleashed/aut/pull/204/files#diff-4541b9834513985c360b64093fd45073
- Align Hadoop version with Apache Spark pom.xml https://github.com/apache/spark/blob/branch-2.3/pom.xml#L120

* Create issue templates

* DataFrameLoader - provides bridge to PySpark.
* Initial python classes for aut.
* Better packaging of Python modules.

* Initial stab at df.
* Initial stab of what link extraction would look like with DFs.
* Added test case.
* Docs.