archivesunleashed/aut

Commits on Feb 20, 2020

Update Spark and Hadoop versions. (#426 )

ruebot committed Feb 20, 2020

Loading status checks…
```
- Update Spark to 2.4.5
- Update Hadoop to 2.7.4 (for RADOS/S3 support)
- Tweak README
```
Verified

This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.

GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits

9ce73a8

Commits on Feb 12, 2020

Add logic so UDFs that filter on url should also filter on src (#424 ).

SinghGursimran and ruebot committed Feb 12, 2020

Loading status checks…
```
- Resolves #418 
- Update tests

Co-authored-by: Nick Ruest <ruestn@gmail.com>
```
Verified

This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.

GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits

ebb5298

Commits on Feb 11, 2020

[skip travis] Add pre-print link to README. (#423 )

ruebot committed Feb 11, 2020
```
* [skip travis] Add pre-print link to README.
```
Verified

This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.

GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits

9474a09

Commits on Feb 10, 2020

Add img alt text to imagegraph(); resolves #420 . (#422 )

ruebot committed Feb 10, 2020

Loading status checks…
```
- Update ExtractImageLinksRDD to grab alt text
- Add alt_text column to imagegraph
- Update tests
```
Verified

This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.

GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits

8f1a9f1
Rename imageLinks to imagegraph; resolves #419 (#421 )

ruebot committed Feb 10, 2020

Loading status checks…
```
* Rename imageLinks to imagegraph; resolves #419
```
Verified

This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.

GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits

87c9734

Commits on Feb 6, 2020

Need --repositories flag with --packages. (#417 )

ruebot committed Feb 6, 2020

Loading status checks…
```
- Fully resolves this issue archivesunleashed/docker-aut#19
- archivesunleashed/docker-aut@37ce4e2
- archivesunleashed/docker-aut@082907a
- archivesunleashed/docker-aut@baee431
```
Verified

This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.

GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits

ae2f097
[maven-release-plugin] prepare for next development iteration

ruebot committed Feb 5, 2020

Loading status checks…

Verified

This commit was signed with a verified signature.

ruebot Nick Ruest

GPG key ID: 417FAF1A0E1080CD Learn about signing commits

e297369

Commits on Feb 5, 2020

[maven-release-plugin] prepare release aut-0.50.0

ruebot committed Feb 5, 2020

Verified

This commit was signed with a verified signature.

ruebot Nick Ruest

GPG key ID: 417FAF1A0E1080CD Learn about signing commits

4d1dcc9

Commits on Jan 21, 2020

Clean up test descriptions, addresses #372 . (#416 )

ruebot authored and ianmilligan1 committed Jan 21, 2020

Loading status checks…
```
- Clean up test descriptions
- Rename typo filename
```
ffef735
Add ExtractImageDetailsDF. (#415 )

SinghGursimran authored and ruebot committed Jan 21, 2020

Loading status checks…
```
- Add test
- Addresses #223
```
71b459c

Commits on Jan 18, 2020

Add crawl_date to binary DataFrames and imageLinks. (#414 )

ruebot authored and ianmilligan1 committed Jan 18, 2020

Loading status checks…
```
- Resolves #413
- Update tests where necessary
```
9e357cc

Commits on Jan 17, 2020

Various DataFrame implementation updates for documentation clean-up; …

ruebot authored and ianmilligan1 committed Jan 17, 2020

…Addresses #372.

- .all() column HttpStatus to http_status_code
- Adds archive_filename to .all()
- Significant README updates for setup
- See also: archivesunleashed/aut-docs#39

9277e68

Commits on Jan 16, 2020

Use https for maven repo. (#405 )

ruebot authored and ianmilligan1 committed Jan 16, 2020

- Looks like repos are forcing https to be used now:
[WARNING] repository metadata for: 'artifact joda-time:joda-time' could not be retrieved from repository: maven due to an error: Failed to transfer file: http://repo.maven.apache.org/maven2/joda-time/joda-time/maven-metadata.xml. Return code is: 501 , ReasonPhrase:HTTPS Required.

4c6875d

Commits on Jan 13, 2020

Test clean-up. (#404 )

ruebot authored and ianmilligan1 committed Jan 13, 2020

Loading status checks…
```
- Clean-up variable names in RecordDFTest.scala
- Remove dos line endings on a number of files
```
75b7502

Commits on Jan 12, 2020

Add language detection column to webpages. (#403 )

ruebot authored and ianmilligan1 committed Jan 12, 2020

Loading status checks…
```
- Addresses #402
```
bc0d663

Commits on Jan 10, 2020

Add more DataFrame Implementation Serializable APIs (#401 ).

SinghGursimran authored and ruebot committed Jan 10, 2020

- Partially addresses  #223 
- Add discardContentDF
- Add discardUrlPatternsDF
- Add discardLanguagesDF
- Add keepImagesDF
- Add keepContentDF
- Add keepUrlPatternsDF
- Add keepLanguagesDF
- Update tests

0ecc4f8

Commits on Jan 8, 2020

Filter blank src/dest out of webgraph. (#400 )

ruebot authored and ianmilligan1 committed Jan 8, 2020

Loading status checks…

3dc1545

Commits on Jan 7, 2020

Add more DF implementations for #223 . (#399 )

SinghGursimran authored and ruebot committed Jan 7, 2020

Loading status checks…
```
- Add discardHttpStatusDF
- Add keepMimeTypesDF
- Add keepMimeTypesTikaDF
- Update tests
```
be15375

Commits on Jan 5, 2020

Scala imports cleanup. (#398 )

ruebot authored and ianmilligan1 committed Jan 5, 2020

Loading status checks…

d5c7bf7

Commits on Dec 29, 2019

Add more serializable APIs for DataFrames (#396 )

SinghGursimran authored and ruebot committed Dec 29, 2019

Loading status checks…
```
- Partially address #223 
- Add keepHttpStatusDF
- Add keepDateDF
- Add keepUrlsDF
- Add keepDomainsDF
- Add tests
```
b915f82

Commits on Dec 19, 2019

Remove redundant test; addresses #64 . (#395 )

ruebot authored and ianmilligan1 committed Dec 19, 2019

Loading status checks…

2c96ff3

Commits on Dec 18, 2019

Add doc comments for webpages and webgraph; resolves #392 . (#394 )

ruebot authored and ianmilligan1 committed Dec 18, 2019

Loading status checks…

99e9d06
Add additional filters for fextFiles; resolves #362 . (#393 )

ruebot authored and ianmilligan1 committed Dec 18, 2019

Loading status checks…
```
* Add additional filters for fextFiles; resolves #362.

- Add filedesc, and dns filter (arc files)
- Add test case
```
8eb43ff

Commits on Dec 17, 2019

udf API implementations for DataFrame (#391 )

SinghGursimran authored and ruebot committed Dec 17, 2019

Loading status checks…
```
- add discardMimeTypesDF
- add discardDateDF
- add discardUrlsDF
- add discardDomainsDF
- update tests
- addresses #223
```
40a59de
Add Serializable APIs for DataFrames (#389 )

SinghGursimran authored and ruebot committed Dec 17, 2019

Loading status checks…
```
- Add keepValidPagesDF
- Add HTTP status code column to all()
- Add test for keepValidPagesDF
- Addresses #223
```
ca928d8

Add and update tests, resolve textFiles bug. (#388 )

ruebot authored and ianmilligan1 committed Dec 17, 2019

- Add ExtractDateDF test
- Fix conditional logic of textFiles filter to resolve #390
- Add test for conditional logic fix for #390
- Remove cruft ExtractUrls, left over from Twitter analysis removal
(see:
https://github.com/lintool/warcbase/blob/cab311ed8b0bceb666865fa76dd3bc5a86402e13/warcbase-core/src/test/scala/org/warcbase/spark/matchbox/ExtractUrlsTest.scala)
- Tweak null/nothing on a few tests

9e32284

Commits on Dec 5, 2019

Add new DataFrame matchbox udfs (#387 )

SinghGursimran authored and ruebot committed Dec 5, 2019

- Add DetectLanguageDF
- Add ExtractBoilerpipeTextDF
- Add ExtractDateDF
- Update tests
- Rename existing ExtractDate, ExtractBoilerpipeText, DetectLanguage udfs by appending RDD
- Partially addresses #223

079cd24

Commits on Nov 28, 2019

Clean-up underscore import, and scalastyle warnings. (#386 )

ruebot authored and ianmilligan1 committed Nov 28, 2019

Loading status checks…

560ed2b

Commits on Nov 21, 2019

Add "Extract popular images" DataFrame implementation (#382 ).

SinghGursimran authored and ruebot committed Nov 21, 2019

Loading status checks…
```
- Add tests for ExtractPopularImagesDF
- Rename ExtractPopularImages to ExtractPopularImagesRDD
- Addresses #223
```
4042180

Add all() method and refactor DF UDFs (#383 ).

SinghGursimran authored and ruebot committed Nov 21, 2019

- Add `all()` DataFrame method 
- Refactor fixity DataFrame UDFs
- Add ComputeImageSize UDF
- Add Python implementation of `all()`
- Addresses #223

c4eaca9

Rename pages() to webpages(). (#384 )

ruebot authored and ianmilligan1 committed Nov 21, 2019

Loading status checks…
```
- Part of work on #233
```
d8e8df3

Commits on Nov 19, 2019

Append UDF with RDD or RF. (#381 )

ruebot authored and ianmilligan1 committed Nov 19, 2019

Loading status checks…
```
- Addresses #223
```
b98ba4b

Commits on Nov 18, 2019

Extend more Matchbook utilities to DataFrames (#380 ).

SinghGursimran authored and ruebot committed Nov 18, 2019

Loading status checks…
```
- Extend GetExtensionMime, ExtractImageLinks, ComputeMD5, and ComputeSHA1 to DataFrames
- Addresses #223
```
a081d7b

Commits on Nov 17, 2019

Rename DF functions to be consistent with Python DF functions. (#379 )

ruebot authored and ianmilligan1 committed Nov 17, 2019

Loading status checks…
```
- Resolves #366
```
67ca17d

Commits on Nov 14, 2019

Finalize converting NER Classifier to WANE Format (#378 ).

SinghGursimran authored and ruebot committed Nov 14, 2019

Loading status checks…
```
- Fully resolves #297 
- Overrides NER Classifier output to PERSON -> persons, LOCATION -> locations, ORGANIZATION -> organizations
```
f9ce826

Older

archivesunleashed / aut