archivesunleashed/aut

* Log opening and closing of archive files as per #156
* Remove redundant log message. Spark already logs the file that is to be read when an executor computes an RDD.

* Remove deprecated ExtractGraph.scala file.
* Remove deprecated ExtractGraphTest.scala file.

…& #164 (#292)

* Resolves #198
* Resolves #164
* Add getHttpStatus to ArchiveRecord class & trait
  - add .getHttpStatus to potential outputs
  - add tests for .getHttpStatus calls
  - improve ArchiveRecord testing overall.
* Add .getArchiveFile feature to ArchiveRecordImpl.
  - add getArchiveFile to trait
  - add getArchiveFile for ArchiveRecordImpl
  - add tests for getArchiveFile.
* Other code style fixes.
* Include updates to tests.

…zipWithUniqueIds() (#289)

* Resolves #243 
* Create GEXF with proper ids instead of hash to avoid collisions.
* Add WriteGEXF files.
* Add WriteGraph file and test.
* Add test for Graphml output.
* Add xml escaping for edges.
* Add test case for non-escaped edges.
* Add additional tests to cover for more potential cases of graphml and gexf files.
* Coverage for null cases in urls.

- CVE-2018-1338
- CVE-2018-11762
- CVE-2018-11761
- CVE-2016-6809
- CVE-2018-1339
- CVE-2018-11796
- CVE-2016-4434
- CVE-2018-1335

* Fix backslash in URL
* Add backslash test in ExtractDomainTest

- Follow on to
72cb5e2
- https://nvd.nist.gov/vuln/detail/CVE-2018-7489

- CVE-2017-752
- See also: https://nvd.nist.gov/vuln/detail/CVE-2017-7525

…les. (#272)

* Fix exception when processing corrupted ARC files
* Filter out non-empty archive files in loadArchives()
* Fix archive files path pattern
* Resolves #246
* Resolves #271 
* Resolves #258

* ExtractBoilerpipeText now removes headers.

)

- Adds:
  - retweet_count
  - favorite_count
  - in_reply_to_status_id_str
  - in_reply_to_user_id_str
  - in_reply_to_screen_name
  - source
  - user.protected
  - user.profile_image_url
  - user.description
  - user.location
  - user.name
  - user.url
  - user.time_zone
- Updates some doc comments
- Updates tests

…). (#249)

* Fully resolves #196 
* Resolves #212

* Deal with wildcard import lint issues.
* Fix some magic numbers & duplicate string runs.
* Lint fixes, mostly explicit import warnings.
* All other scalastyle issues require refactoring.

…ssue 203 (#245)

* pom.xml change for GraphX
* Changes for GraphXSLS
* Changes for SLS graph
* Changes for GraphX
* Changes for converting WARC RDD to GraphX object
* Rename extractor to ExtractGraphX
* Various lint fixes (usually Magic Numbers)
* Remove illegal imports from scala style (we use wildcard imports a lot)
* Add WriteGraphXMLTest.

* Make the TravisCI build less verbose since we're hitting the 4MB log limit.
* Pin site.plugin and project-info-reports.plugin so mvn site builds.
  - See:
    - https://stackoverflow.com/questions/51091539/maven-site-plugins-3-3-java-lang-classnotfoundexception-org-apache-maven-doxia
    - https://travis-ci.org/archivesunleashed/aut/jobs/408259462#L3201-L3202

…esolve #235 (#236)

* initial implementation
* Data frame implementation of extractors.
* fix documentation.

* Save images from dataframe to disk
* Fix spacing
* Move save images to inline
* Refactor to chain and fix concurrency issue
* Add save image test
* Move saveToDisk to df

* Add Extract Image Details API
* Change check for jpeg and fix spacing
* Add tiff parser
* Use AutoDetectParser and read Numeric fields
* Use ComputeImageSize
* Hex encode hash and base64 encode image bytes
* Fix test
* Change df column names