Skip to content
Please note that GitHub no longer supports your web browser.

We recommend upgrading to the latest Google Chrome or Firefox.

Learn more

/aut

Permalink
Tree: 3ab17bf017
Commits on Nov 26, 2019

  1. Fix copyright header

    @ruebot
    ruebot committed Nov 26, 2019
    Loading status checks…
    Verified
    This commit was signed with a verified signature.
    ruebot Nick Ruest
    GPG key ID: 417FAF1A0E1080CD Learn about signing commits
    3ab17bf

  2. Merge branch 'issue-356' of github.com:archivesunleashed/aut into iss…

    @ruebot
    ruebot committed Nov 26, 2019
    Loading status checks…
     
    …ue-329
    Verified
    This commit was signed with a verified signature.
    ruebot Nick Ruest
    GPG key ID: 417FAF1A0E1080CD Learn about signing commits
    65249ba
Commits on Nov 21, 2019

  1. Merge branch 'master' into issue-356

    @ruebot
    ruebot committed Nov 21, 2019
    Loading status checks…
    Verified
    This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.
    GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits
    481dbf4

  2. Add "Extract popular images" DataFrame implementation (#382).

    @SinghGursimran @ruebot
    SinghGursimran authored and ruebot committed Nov 21, 2019
    Loading status checks…
     
    - Add tests for ExtractPopularImagesDF
- Rename ExtractPopularImages to ExtractPopularImagesRDD
- Addresses #223
    4042180

  3. Merge branch 'master' into issue-356

    @ruebot
    ruebot committed Nov 21, 2019
    Loading status checks…
    Verified
    This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.
    GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits
    f662dee

  4. Add all() method and refactor DF UDFs (#383).

    @SinghGursimran @ruebot
    SinghGursimran authored and ruebot committed Nov 21, 2019
    Loading status checks…
     
    - Add `all()` DataFrame method 
- Refactor fixity DataFrame UDFs
- Add ComputeImageSize UDF
- Add Python implementation of `all()`
- Addresses #223
    c4eaca9

  5. Rename pages() to webpages(). (#384)

    @ruebot @ianmilligan1
    ruebot authored and ianmilligan1 committed Nov 21, 2019
    Loading status checks…
     
    - Part of work on #233
    d8e8df3
Commits on Nov 19, 2019

  1. Merge branch 'master' into issue-356

    @ruebot
    ruebot committed Nov 19, 2019
    Loading status checks…
    Verified
    This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.
    GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits
    82d588a

  2. Append UDF with RDD or RF. (#381)

    @ruebot @ianmilligan1
    ruebot authored and ianmilligan1 committed Nov 19, 2019
    Loading status checks…
     
    - Addresses #223
    b98ba4b
Commits on Nov 18, 2019

  1. Merge branch 'master' into issue-356

    @ruebot
    ruebot committed Nov 18, 2019
    Loading status checks…
    Verified
    This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.
    GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits
    fbc31d4

  2. Extend more Matchbook utilities to DataFrames (#380).

    @SinghGursimran @ruebot
    SinghGursimran authored and ruebot committed Nov 18, 2019
    Loading status checks…
     
    - Extend GetExtensionMime, ExtractImageLinks, ComputeMD5, and ComputeSHA1 to DataFrames
- Addresses #223
    a081d7b
Commits on Nov 17, 2019

  1. Rename DF functions to be consistent with Python DF functions. (#379)

    @ruebot @ianmilligan1
    ruebot authored and ianmilligan1 committed Nov 17, 2019
    Loading status checks…
     
    - Resolves #366
    67ca17d
Commits on Nov 14, 2019

  1. Merge branch 'master' into issue-356

    @ruebot
    ruebot committed Nov 14, 2019
    Loading status checks…
    Verified
    This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.
    GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits
    eecd41a

  2. Finalize converting NER Classifier to WANE Format (#378).

    @SinghGursimran @ruebot
    SinghGursimran authored and ruebot committed Nov 14, 2019
    Loading status checks…
     
    - Fully resolves #297 
- Overrides NER Classifier output to PERSON -> persons, LOCATION -> locations, ORGANIZATION -> organizations
    f9ce826
Commits on Nov 12, 2019

  1. Add df ExtractLinks udf; resolves #238. (#377)

    @SinghGursimran @ruebot
    SinghGursimran authored and ruebot committed Nov 12, 2019
    Loading status checks…
     
    - Add df ExtractLinks udf
- Add test
    c353dae
Commits on Nov 10, 2019

  1. Merge branch 'master' into issue-356

    @ruebot
    ruebot committed Nov 10, 2019
    Loading status checks…
    Verified
    This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.
    GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits
    d59f9e6

  2. #356 add Java 11 and Spark 3.0.0+ to README

    @ruebot
    ruebot committed Nov 10, 2019
    Loading status checks…
    Verified
    This commit was signed with a verified signature.
    ruebot Nick Ruest
    GPG key ID: 417FAF1A0E1080CD Learn about signing commits
    3b6744c

  3. Update README.md (#376)

    @lintool @ruebot
    lintool authored and ruebot committed Nov 10, 2019
    Loading status checks…
     
    Tweaks the style of the license badge to look consistent with the other badges.
    107def2

  4. #356 wave goodbye to Java 8.

    @ruebot
    ruebot committed Nov 10, 2019
    Loading status checks…
    Verified
    This commit was signed with a verified signature.
    ruebot Nick Ruest
    GPG key ID: 417FAF1A0E1080CD Learn about signing commits
    a1b9607

  5. #356 remove test line.

    @ruebot
    ruebot committed Nov 10, 2019
    Loading status checks…
    Verified
    This commit was signed with a verified signature.
    ruebot Nick Ruest
    GPG key ID: 417FAF1A0E1080CD Learn about signing commits
    d1692b5

  6. #356 -- forgot to remove cobertura TravisCI build.

    @ruebot
    ruebot committed Nov 10, 2019
    Loading status checks…
    Verified
    This commit was signed with a verified signature.
    ruebot Nick Ruest
    GPG key ID: 417FAF1A0E1080CD Learn about signing commits
    b002a02

  7. #356 Replace Cobertura with JaCoCo, and allow Java 8 to fail for now.

    @ruebot
    ruebot committed Nov 10, 2019
    Loading status checks…
    Verified
    This commit was signed with a verified signature.
    ruebot Nick Ruest
    GPG key ID: 417FAF1A0E1080CD Learn about signing commits
    41d5f45

  8. #356 :facepalm: oops. uncommented that, and pushed.

    @ruebot
    ruebot committed Nov 10, 2019
    Loading status checks…
    Verified
    This commit was signed with a verified signature.
    ruebot Nick Ruest
    GPG key ID: 417FAF1A0E1080CD Learn about signing commits
    4a54d67

  9. #356 update TravisCI; don't allow failures on Java 11.

    @ruebot
    ruebot committed Nov 10, 2019
    Loading status checks…
    Verified
    This commit was signed with a verified signature.
    ruebot Nick Ruest
    GPG key ID: 417FAF1A0E1080CD Learn about signing commits
    c86c398

  10. #356 verify-javadocs fixed -- make sure you have OpenJDK-11 fully ins…

    @ruebot
    ruebot committed Nov 10, 2019
    Loading status checks…
     
    …talled :facepalm:, and a bunch more pom cleanup.
    Verified
    This commit was signed with a verified signature.
    ruebot Nick Ruest
    GPG key ID: 417FAF1A0E1080CD Learn about signing commits
    d301adc

  11. #356 scala updates

    @ruebot
    ruebot committed Nov 10, 2019
    Verified
    This commit was signed with a verified signature.
    ruebot Nick Ruest
    GPG key ID: 417FAF1A0E1080CD Learn about signing commits
    06764f7
Commits on Nov 9, 2019

  1. Update to Spark 3.0.0

    @ruebot
    ruebot committed Nov 9, 2019 
    - Some hacks to get a sucessful build
- Definitely need to loop back and clean-up a whole lot!
- Addresses #356
    Verified
    This commit was signed with a verified signature.
    ruebot Nick Ruest
    GPG key ID: 417FAF1A0E1080CD Learn about signing commits
    8a51a87
Commits on Nov 8, 2019

  1. Merge branch 'master' of github.com:archivesunleashed/aut into issue-356

    @ruebot
    ruebot committed Nov 8, 2019
    Verified
    This commit was signed with a verified signature.
    ruebot Nick Ruest
    GPG key ID: 417FAF1A0E1080CD Learn about signing commits
    1c2ff7c
Commits on Nov 7, 2019

  1. Change RemoveHttpHeader to RemoveHTTPHeader. (#374)

    @SinghGursimran @ruebot
    SinghGursimran authored and ruebot committed Nov 7, 2019
    Loading status checks…
     
    Resolves #368.
    25ca5a9
Commits on Nov 6, 2019

  1. Updates description. See archivesunleashed/aut-docs#18 (#373)

    @ruebot @ianmilligan1
    ruebot authored and ianmilligan1 committed Nov 6, 2019
    Loading status checks…
    08b9ae3
Commits on Nov 5, 2019

  1. Align NER output to WANE format; addresses #297 (#361)

    @ruebot @ianmilligan1
    ruebot authored and ianmilligan1 committed Nov 5, 2019 
    - Update Stanford core NLP
- Format NER output in json
- Add getPayloadDigest to ArchiveRecord
- Add test for getPayloadDigest
- Add payload digest to NER output
- Remove extractFromScrapeText
- Remove extractFromScrapeText test
- TODO: PERSON -> persons, LOCATION -> locations, ORGANIZATION -> organizations (involves writing a new class or overriding NER output 🤢
    379cc68

  2. Various UDF implementation and cleanup for DF. (#370)

    @lintool @ruebot
    lintool authored and ruebot committed Nov 5, 2019
    Loading status checks…
     
    - Replace ExtractBaseDomain with ExtractDomain
- Closes #367
- Address bug in ArcTest; RemoveHTML -> RemoveHttpHeader
- Closes #369
- Wraps RemoveHttpHeader and RemoveHTML for use in data frames.
- Partially addresses #238
- Updates tests where necessary
- Punts on #368 UDF CaMeL cASe consistency issues
    6686519
Commits on Oct 14, 2019

  1. Update commons-compress to 1.19; CVE-2019-12402 (#365)

    @ruebot @ianmilligan1
    ruebot authored and ianmilligan1 committed Oct 14, 2019
    Loading status checks…
    4e8b41d
Commits on Oct 9, 2019

  1. Add ComputeSHA1 method; resolves #363. (#364)

    @ruebot @ianmilligan1
    ruebot authored and ianmilligan1 committed Oct 9, 2019
    Loading status checks…
     
    - Update tests where needed
- Add SHA1 method to ExtractImageDetails
- Add SHA1 to DataFrames binary extraction and analysis
    03ac99c
Commits on Sep 11, 2019

  1. Update keepValidPages to include a filter on 200 OK. (#360)

    @ruebot @ianmilligan1
    ruebot authored and ianmilligan1 committed Sep 11, 2019
    Loading status checks…
     
    - Add status code filter to keepValidPages
- Add MimeTypeTika to valid pages DF
- Update tests since we filter more and better now 😄
- Resolves #359
    9b3e025
Older
You can’t perform that action at this time.