archivesunleashed/aut

* ExtractBoilerpipeText now removes headers.

)

- Adds:
  - retweet_count
  - favorite_count
  - in_reply_to_status_id_str
  - in_reply_to_user_id_str
  - in_reply_to_screen_name
  - source
  - user.protected
  - user.profile_image_url
  - user.description
  - user.location
  - user.name
  - user.url
  - user.time_zone
- Updates some doc comments
- Updates tests

…). (#249)

* Fully resolves #196 
* Resolves #212

* Deal with wildcard import lint issues.
* Fix some magic numbers & duplicate string runs.
* Lint fixes, mostly explicit import warnings.
* All other scalastyle issues require refactoring.

…ssue 203 (#245)

* pom.xml change for GraphX
* Changes for GraphXSLS
* Changes for SLS graph
* Changes for GraphX
* Changes for converting WARC RDD to GraphX object
* Rename extractor to ExtractGraphX
* Various lint fixes (usually Magic Numbers)
* Remove illegal imports from scala style (we use wildcard imports a lot)
* Add WriteGraphXMLTest.

* Make the TravisCI build less verbose since we're hitting the 4MB log limit.
* Pin site.plugin and project-info-reports.plugin so mvn site builds.
  - See:
    - https://stackoverflow.com/questions/51091539/maven-site-plugins-3-3-java-lang-classnotfoundexception-org-apache-maven-doxia
    - https://travis-ci.org/archivesunleashed/aut/jobs/408259462#L3201-L3202

…esolve #235 (#236)

* initial implementation
* Data frame implementation of extractors.
* fix documentation.

* Save images from dataframe to disk
* Fix spacing
* Move save images to inline
* Refactor to chain and fix concurrency issue
* Add save image test
* Move saveToDisk to df

* Add Extract Image Details API
* Change check for jpeg and fix spacing
* Add tiff parser
* Use AutoDetectParser and read Numeric fields
* Use ComputeImageSize
* Hex encode hash and base64 encode image bytes
* Fix test
* Change df column names

…an be run from command line (#225)

* Resolves issue 195. Implement DomainFrequency, DomainGraph and PlainText extractor that can be run via command line in spark-submit, along with their tests

* Restructure CommandLineAppRunner to make it more robust. Add option to write GEXF output for DomainGraphExtractor (enable via --output-format GEXF). Add support for multiple input files. Other polish and cleanup.

* Extract Image Links DF API
* Add extract image links text
* Remove unnecessary comment from test
* Add doc comments
* Addresses #220

- Update tests to use workaround for SPARK-2243
- Comment out ExtractGraph test as per https://github.com/archivesunleashed/aut/pull/204/files#diff-4541b9834513985c360b64093fd45073
- Align Hadoop version with Apache Spark pom.xml https://github.com/apache/spark/blob/branch-2.3/pom.xml#L120

* Create issue templates

* DataFrameLoader - provides bridge to PySpark.
* Initial python classes for aut.
* Better packaging of Python modules.

* Initial stab at df.
* Initial stab of what link extraction would look like with DFs.
* Added test case.
* Docs.

, see also: json4s/json4s#316 (#205)

…n cha… (#200)

- update gitignore
- add site build to TravisCI config
- add scalastyle config
- improve scala docs on every scala file
- incorporate @greebie's work on scaladocs

* io.archivesunleashed.spark -> io.archivesunleashed package renaming.
* Resolves #188
* Resolves #178 
* Resolves #179 
* Addresses #180