Skip to content
Please note that GitHub no longer supports your web browser.

We recommend upgrading to the latest Google Chrome or Firefox.

Learn more
Permalink
Tree: f9ce826989
Commits on Nov 14, 2019
  1. Finalize converting NER Classifier to WANE Format (#378).

    SinghGursimran authored and ruebot committed Nov 14, 2019
    - Fully resolves #297 
    - Overrides NER Classifier output to PERSON -> persons, LOCATION -> locations, ORGANIZATION -> organizations
Commits on Nov 12, 2019
  1. Add df ExtractLinks udf; resolves #238. (#377)

    SinghGursimran authored and ruebot committed Nov 12, 2019
    - Add df ExtractLinks udf
    - Add test
Commits on Nov 10, 2019
  1. Update README.md (#376)

    lintool authored and ruebot committed Nov 10, 2019
    Tweaks the style of the license badge to look consistent with the other badges.
Commits on Nov 7, 2019
  1. Change RemoveHttpHeader to RemoveHTTPHeader. (#374)

    SinghGursimran authored and ruebot committed Nov 7, 2019
    Resolves #368.
Commits on Nov 6, 2019
  1. Updates description. See archivesunleashed/aut-docs-new#18 (#373)

    ruebot authored and ianmilligan1 committed Nov 6, 2019
Commits on Nov 5, 2019
  1. Align NER output to WANE format; addresses #297 (#361)

    ruebot authored and ianmilligan1 committed Nov 5, 2019
    - Update Stanford core NLP
    - Format NER output in json
    - Add getPayloadDigest to ArchiveRecord
    - Add test for getPayloadDigest
    - Add payload digest to NER output
    - Remove extractFromScrapeText
    - Remove extractFromScrapeText test
    - TODO: PERSON -> persons, LOCATION -> locations, ORGANIZATION -> organizations (involves writing a new class or overriding NER output 🤢
  2. Various UDF implementation and cleanup for DF. (#370)

    lintool authored and ruebot committed Nov 5, 2019
    - Replace ExtractBaseDomain with ExtractDomain
    - Closes #367
    - Address bug in ArcTest; RemoveHTML -> RemoveHttpHeader
    - Closes #369
    - Wraps RemoveHttpHeader and RemoveHTML for use in data frames.
    - Partially addresses #238
    - Updates tests where necessary
    - Punts on #368 UDF CaMeL cASe consistency issues
Commits on Oct 14, 2019
  1. Update commons-compress to 1.19; CVE-2019-12402 (#365)

    ruebot authored and ianmilligan1 committed Oct 14, 2019
Commits on Oct 9, 2019
  1. Add ComputeSHA1 method; resolves #363. (#364)

    ruebot authored and ianmilligan1 committed Oct 9, 2019
    - Update tests where needed
    - Add SHA1 method to ExtractImageDetails
    - Add SHA1 to DataFrames binary extraction and analysis
Commits on Sep 11, 2019
  1. Update keepValidPages to include a filter on 200 OK. (#360)

    ruebot authored and ianmilligan1 committed Sep 11, 2019
    - Add status code filter to keepValidPages
    - Add MimeTypeTika to valid pages DF
    - Update tests since we filter more and better now 😄
    - Resolves #359
Commits on Sep 3, 2019
  1. Update to Spark 2.4.4 (#358)

    ruebot authored and ianmilligan1 committed Sep 3, 2019
Commits on Aug 27, 2019
Commits on Aug 23, 2019
  1. Add discardLanguage filter to RecordLoader. (#353)

    ruebot authored and ianmilligan1 committed Aug 23, 2019
    - Clean up doc comments
    - Add test
    - Resolves #352
Commits on Aug 22, 2019
  1. Improve test coverage. (#354)

    ruebot authored and ianmilligan1 committed Aug 22, 2019
    - Add tests a few more filters in RecordLoader
    - Add binary extration DataFrameLoader tests
Commits on Aug 21, 2019
  1. [maven-release-plugin] prepare for next development iteration

    ruebot committed Aug 21, 2019
  2. Add binary extraction DataFrames to PySpark. (#350)

    ruebot authored and ianmilligan1 committed Aug 21, 2019
    * Add binary extration DataFrames to PySpark.
    - Address #190
    - Address #259
    - Address #302
    - Address #303
    - Address #304
    - Address #305
    - Address #306
    - Address #307
    - Resolves #350 
    - Update README
  3. Update LICENSE and license headers. (#351)

    ruebot authored and ianmilligan1 committed Aug 21, 2019
    - Update LICENSE file to full Apache 2.0 license
    - Reconfigure license-maven-plugin
    - Update all license headers in java and scala files to include
    copyright year, and project name
    - Move LICENSE_HEADER.txt to config
    - Update scalastyle config
Commits on Aug 18, 2019
  1. Add method for determining binary file extension. (#349)

    jrwiebe authored and ruebot committed Aug 18, 2019
    This PR implements the strategy described in the discussion of the above issue to get an extension for a file described by a URL and a MIME type. It creates a GetExtensionMime object in the matchbox.
    
    This PR also removes most of the filtering by URL from the image, audio, video, presentation, spreadsheet, and word processor document extraction methods, since these were returning false positives. (CSV and TSV files are a special case, since Tika detects them as "text/plain" based on content.)
    
    Finally, I have inserted toLowerCase into the getUrl.endsWith() filter tests, which could possibly bring in some more CSV and TSV files
    
    * Adds method for getting a file extension from a MIME type.
    * Add getExtensions method to DetectMimeTypeTika.
    * Matchbox object to get extension of URL
    * Use GetExtensionMime for extraction methods; minor fixes.
    * Remove tika-parsers classifier
    * Remove most filtering by file extension from binary extraction methods; add CSV/TSV special cases.
    * Fix GetExtensionMime case where URL has no extension but a MIME type is detected
    * Insert `toLowerCase` into `getUrl.endsWith()` calls in io.archivesunleashed.packages; apply to `FilenameUtils.getExtension` in `GetExtensionMime`.
    * Remove filtering on URL for audio, video, and images.
    * Remove filtering on URL for images; add DF fields to image extraction
    * Remove saveImageToDisk and its test
    * Remove robots.txt check and extraneous imports
    * Close files so we don't get too many files open again.
    * Add GetExtensionMimeTest
    * Resolve #343
Commits on Aug 17, 2019
  1. Add keep and discard by http status. (#347)

    ruebot authored and ianmilligan1 committed Aug 17, 2019
    - Add keep and discard by http status RecordLoader
    - Add tests
    - Clean up/add doc comments in RecordLoader
    - Resolve #315
Commits on Aug 16, 2019
  1. Add office document binary extraction. (#346)

    ruebot authored and ianmilligan1 committed Aug 16, 2019
    - Add Word Processor DF and binary extraction
    - Add Spreadsheets DF and binary extraction
    - Add Presentation Program DF and binary extraction
    - Add Text files DF and binary extraction
    - Add tests for new DF and binary extractions
    - Add test fixtures for new DF and binary extractions
    - Resolves #303
    - Resolves #304
    - Resolves #305
    - Use aut-resources repo to distribute our shaded tika-parsers 1.22
    - Close TikaInputStream
    - Add RDD filters on MimeTypeTika values
    - Add CodeCov configuration yaml
    - Includes work by @jrwiebe, see #346 for all commits before squash
Commits on Aug 14, 2019
  1. Use version of tika-parsers without a classifier. (#345)

    jrwiebe authored and ruebot committed Aug 14, 2019
    Ivy couldn't handle it, and specifying one for the custom tika-parsers artifact
    was unnecessary.
  2. Use Tika's detected MIME type instead of ArchiveRecord getMimeType. (#…

    ruebot authored and ianmilligan1 committed Aug 14, 2019
    …344)
    
    - Move audio, pdf, and video DF extraction to tuple map
    - Provide two MimeType columns; mime_type_web_server and mime_type_tika
    - Update tests
    - Resolves #342
Commits on Aug 13, 2019
  1. Add audio & video binary extraction (#341)

    ruebot authored and ianmilligan1 committed Aug 13, 2019
    - Add Audio & Video binary extraction.
    - Add filename, and extenstion column to audio, pdf, and video DF
    - Pass binary bytes instread of string to DetectMimeTypeTika in DF (s/getContentString/getBinaryBytes)
    - Updates saveToDisk to use file extension from DF column
    - Adds tests for Audio, PDF, and Video DF extraction
    - Add test fixtures for Audio, PDF, and Video DF extraction
    - Rename SaveBytesTest to SaveImageBytes test
    - Eliminate bytes->string->bytes conversion that was causing data loss in DetectMimeTypeTika
    - Update tika-parsers dep from JitPack
    - Remove tweet cruft
    - Resolves #306
    - Resolves #307
    - Includes work by @jrwiebe, see #341 for all commits before squash
Commits on Aug 12, 2019
  1. Add PDF binary extraction. (#340)

    jrwiebe authored and ruebot committed Aug 12, 2019
    Introduces the new extractPDFDetailsDF() method and brings in changes to make our use of Tika's MIME type detection more efficient, as well as POM updates to use a shaded version of tika-parsers in order to eliminate a dependency version conflict that has long been troublesome.
    
    - Updates getImageBytes to getBinaryBytes
    - Refactor SaveImage class to more general SaveBytes, and saveToDisk to saveImageToDisk
    - Only instantiate Tika when the DetectMimeTypeTika singleton object is first referenced. See https://git.io/fj7g0.
    - Use TikaInputStream to enabler container-aware detection. Until now we were only using the default Mime Magic detection. See https://tika.apache.org/1.22/detection.html#Container_Aware_Detection.
    - Added generic saveToDisk method to save a bytes column of a DataFrame to files
    - Updates tests
    - Resolves #302
    - Further addresses #308
    - Includes work by @ruebot, see #340 for all commits before squash
Commits on Aug 8, 2019
  1. More scalastyle work; addresses #196. (#339)

    ruebot authored and ianmilligan1 committed Aug 8, 2019
    - Remove all underscore imports, except shapeless
    - Address all scalastyle warnings
    - Update scalastyle config for magic numbers, and null (only used in
    tests)
Commits on Aug 7, 2019
  1. Replace computeHash with ComputeMD5; resolves #333. (#338)

    ruebot authored and jrwiebe committed Aug 7, 2019
    * Replace computeHash with ComputeMD5; resolves #333.
    
    * I suppose these are redundant.
Commits on Aug 6, 2019
  1. Make ArchiveRecord.getContentBytes consistent,#334 (#335)

    ianmilligan1 authored and ruebot committed Aug 6, 2019
  2. Update Tika to 1.22; address security alerts. (#337)

    ruebot authored and ianmilligan1 committed Aug 6, 2019
    - Update Tika to 1.22
    - pom.xml surgery to get aut to build again with --packages
Commits on Jul 31, 2019
  1. Update test coverage for data frames (#336).

    ruebot authored and ianmilligan1 committed Jul 31, 2019
    - This commit will fall under @ruebot, but @jrwiebe did the heavy lifting here; see #336 for his commits before they were squashed down.
    - Resolves #265
    - Resolves #263
    - Update Scaladocs
Commits on Jul 25, 2019
  1. Enable S3 access (#332)

    jrwiebe authored and ruebot committed Jul 25, 2019
    * Update POM to access data stored in Amazon S3, per #319
    * In RecordLoader detect FileSystem based on path.
    * Resolves #319
Commits on Jul 23, 2019
  1. Updates to pom following 0e701b2 (#328)

    ruebot authored and ianmilligan1 committed Jul 23, 2019
    - Remove explicit Guava dependency (should have been remove in
    0e701b2)
    - Update Scala to 2.11.12; aligns with Spark 2.4.3
    - Update Scala test
    - Update Shapeless
    - Update Scala lang parsers
    - Fix a typo in a test
Commits on Jul 18, 2019
  1. Python formatting, and gitignore additions. (#326)

    ruebot authored and ianmilligan1 committed Jul 18, 2019
    - Run black and isort on Python files.
    - Move Spark config to example file.
    - Update gitignore for 7a61f0e
    additions.
  2. Move data frame fields names to snake_case. (#327)

    ruebot authored and ianmilligan1 committed Jul 18, 2019
    - Resolves #229
Commits on Jul 17, 2019
  1. Update to Spark 2.4.3 and update Tika to 1.20. (#321)

    ruebot authored and ianmilligan1 committed Jul 17, 2019
    * Update to Spark 2.4.3 and update Tika to 1.20.
    
    - Resolves #295
    - Resolves #308
    - Resolves #286
    - Pulls in unfinished work by @jrwiebe and @borislin.
    
    * Add patched lang-detector
Older
You can’t perform that action at this time.