archivesunleashed/aut

Commits on May 19, 2020

Implement Scala Matchbox UDFs in Python. (#463 )

ruebot committed May 19, 2020

- Resolves #408
- Alphabetizes DataFrameloader functions
- Alphabetizes UDFs functions
- Move DataFrameLoader to df packages
- Move UDFs out of df into their own package
- Rename UDFs (no more DF tagged to the end).
- Update tests as necessary
- Partially addresses #410, #409
- Supersedes #412.

69007e2

Commits on May 10, 2020

Import clean-up for df package. (#462 )

ruebot committed May 10, 2020

Loading status checks…

Verified

This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.

GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits

1d01571

Commits on May 4, 2020

[maven-release-plugin] prepare for next development iteration

ruebot committed May 4, 2020

Loading status checks…

Verified

This commit was signed with a verified signature.

ruebot Nick Ruest

GPG key ID: 417FAF1A0E1080CD Learn about signing commits

4183a35
[maven-release-plugin] prepare release aut-0.70.0

ruebot committed May 4, 2020

Verified

This commit was signed with a verified signature.

ruebot Nick Ruest

GPG key ID: 417FAF1A0E1080CD Learn about signing commits

df43ac6
[skip travis] README updates (#460 )

ruebot committed May 4, 2020
```
- `$` should only be used if output is also shown (mdl)
- Add UserDoc badge, and yank buried documentation section
- Additional formatting and typo fixes
```
Verified

This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.

GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits

3ec5bba
Set spark-submit app name to be "aut - extractorName". (#459 )

ruebot committed May 4, 2020

Loading status checks…
```
- Resolves #458
```
Verified

This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.

GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits

ce6dc79

Commits on Apr 27, 2020

Add RemovePrefixWWWDF to DomainFrequencyExtractor. (#457 )

ruebot committed Apr 27, 2020

Loading status checks…
```
- Resolves #456
- Update test
```
Verified

This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.

GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits

0f3d732

Commits on Apr 23, 2020

[skip travis] Updating Java install instructions, resolves #445 (#455 )

ianmilligan1 committed Apr 23, 2020

Verified

This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.

GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits

9b1aae9

Commits on Apr 22, 2020

Add option to save to Parquet for app. (#454 )

ruebot committed Apr 22, 2020

Loading status checks…
```
- Resolves #448
- Update test
- Add CSV headers to coalesce CSV output
- Update README
```
Verified

This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.

GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits

aece1e3
Update PlainTextExtractor to output a single column; text. (#453 )

ruebot committed Apr 22, 2020

Loading status checks…
```
- Resolves #452
- PlainTextExtractor runs ExtractBoilerplate on `content`
- Update test
```
Verified

This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.

GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits

e91d01f

Commits on Apr 21, 2020

Add a number of additional app extractors. (#451 )

ruebot committed Apr 21, 2020

- Resolves #447
- Add AudioInformationExtractor, ImageInformationExtractor,
PDFInformationExtractor, PresentationProgramInformationExtractor,
SpreadsheetInformationExtractor, TextFilesInformationExtractor,
VideoInformationExtractor, WebGraphExtractor,
WordProcessorInformationExtractor
- Add tests for the new extractors
- Update CommandLineApp to use new extractors
- Add domain, and language column WebPagesExtractor
- Change "TEXT" to "csv"
- Lower case "GEXF" and "GRAPHML"

f1eb43b

Commits on Apr 20, 2020

Remove RDD option in app; DataFrame only now. (#450 )

ruebot committed Apr 20, 2020

Loading status checks…
```
- Resolves #449
- Updates and renames tests were applicable
- Update README to reflect updates
```
Verified

This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.

GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits

17ac324

Commits on Apr 15, 2020

[skip-travis] Add spark-submit option to README; resolves #444 . (#446 )

ruebot committed Apr 15, 2020

Loading status checks…

Verified

This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.

GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits

d5a0433
[maven-release-plugin] prepare for next development iteration

ruebot committed Apr 15, 2020

Loading status checks…

Verified

This commit was signed with a verified signature.

ruebot Nick Ruest

GPG key ID: 417FAF1A0E1080CD Learn about signing commits

35291bf
[maven-release-plugin] prepare release aut-0.60.0

ruebot committed Apr 15, 2020

Verified

This commit was signed with a verified signature.

ruebot Nick Ruest

GPG key ID: 417FAF1A0E1080CD Learn about signing commits

d0f9761

Commits on Apr 14, 2020

Remove WriteGraph; resolves #439 . (#441 )

ruebot committed Apr 14, 2020

Loading status checks…
```
* Cleanup WriteGraphML doc comments.
```
Verified

This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.

GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits

c1f9b31

Commits on Apr 13, 2020

Remove GraphX support; resolves #442 . (#443 )

ruebot committed Apr 13, 2020

Loading status checks…
```
- Remove graphx dependencies from pom
- Remove ExtractGraphX and related tests
- Remove WriteGraphXML and related tests
```
Verified

This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.

GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits

f357e0c

Commits on Apr 11, 2020

Add graphml output to CommandLineApp and DomainGraphExtractor. (#438 )

ruebot committed Apr 11, 2020

Loading status checks…
```
* Resolves #435
* Adds GRAPHML option to CommandLineApp
* Adds DataFrame method to DomainGraphExtractor
* Updates CommandLineApp, and WriteGraphML tests
```
Verified

This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.

GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits

943cf92

Commits on Apr 8, 2020

Align RDD and DF output for DomainGraphExtractor. (#437 )

ruebot committed Apr 8, 2020

Loading status checks…
```
- Resolves #436
- Remove WWW prefix for RDD was double escaping
- Update DF so it matches RDD output (it wasn't even close before
🤦)
- Update tests so they're basically testing the same thing
```
Verified

This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.

GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits

96899f4

Commits on Apr 7, 2020

Update log4j configuration to resolve #433 . (#434 )

ruebot committed Apr 7, 2020

Verified

This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.

GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits

b957e5d
Add imagegraph, and webgraph to command line app. (#432 )

ruebot committed Apr 7, 2020

Loading status checks…
```
- Resolves #431
- Adds webpages, and imagegraph to command line app
- Adds tests for new functionality
- Clean-up doc comments
- Convert files with dos line endings to unix line endings
- Update CommandLineApp tests
```
Verified

This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.

GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits

771ea82

Commits on Mar 23, 2020

Tweak hasDate to handle Seq. (#430 )

ruebot committed Mar 23, 2020

Loading status checks…
```
Tweak hasDate to handle Seq.
- Addresses #425
- Add test for hasDate
```
Verified

This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.

GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits

92b5f2d

Commits on Mar 18, 2020

Restyle keep/discard filter UDFs in the context of DataFrames (#429 )

ruebot committed Mar 18, 2020

Loading status checks…
```
Co-authored-by: g285sing <g285sing@student.cs.uwaterloo.ca> (@SinghGursimran)

- Resolves #425
- Replace all keep/discard DF udfs with `hasXYZ()`
- Update tests
```
Verified

This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.

GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits

e1d908b

Commits on Feb 20, 2020

Update Spark and Hadoop versions. (#426 )

ruebot committed Feb 20, 2020

Loading status checks…
```
- Update Spark to 2.4.5
- Update Hadoop to 2.7.4 (for RADOS/S3 support)
- Tweak README
```
Verified

This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.

GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits

9ce73a8

Commits on Feb 12, 2020

Add logic so UDFs that filter on url should also filter on src (#424 ).

SinghGursimran and ruebot committed Feb 12, 2020

Loading status checks…
```
- Resolves #418 
- Update tests

Co-authored-by: Nick Ruest <ruestn@gmail.com>
```
Verified

This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.

GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits

ebb5298

Commits on Feb 11, 2020

[skip travis] Add pre-print link to README. (#423 )

ruebot committed Feb 11, 2020
```
* [skip travis] Add pre-print link to README.
```
Verified

This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.

GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits

9474a09

Commits on Feb 10, 2020

Add img alt text to imagegraph(); resolves #420 . (#422 )

ruebot committed Feb 10, 2020

Loading status checks…
```
- Update ExtractImageLinksRDD to grab alt text
- Add alt_text column to imagegraph
- Update tests
```
Verified

This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.

GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits

8f1a9f1
Rename imageLinks to imagegraph; resolves #419 (#421 )

ruebot committed Feb 10, 2020

Loading status checks…
```
* Rename imageLinks to imagegraph; resolves #419
```
Verified

This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.

GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits

87c9734

Commits on Feb 6, 2020

Need --repositories flag with --packages. (#417 )

ruebot committed Feb 6, 2020

Loading status checks…
```
- Fully resolves this issue archivesunleashed/docker-aut#19
- archivesunleashed/docker-aut@37ce4e2
- archivesunleashed/docker-aut@082907a
- archivesunleashed/docker-aut@baee431
```
Verified

This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.

GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits

ae2f097
[maven-release-plugin] prepare for next development iteration

ruebot committed Feb 5, 2020

Loading status checks…

Verified

This commit was signed with a verified signature.

ruebot Nick Ruest

GPG key ID: 417FAF1A0E1080CD Learn about signing commits

e297369

Commits on Feb 5, 2020

[maven-release-plugin] prepare release aut-0.50.0

ruebot committed Feb 5, 2020

Verified

This commit was signed with a verified signature.

ruebot Nick Ruest

GPG key ID: 417FAF1A0E1080CD Learn about signing commits

4d1dcc9

Commits on Jan 21, 2020

Clean up test descriptions, addresses #372 . (#416 )

ruebot authored and ianmilligan1 committed Jan 21, 2020

Loading status checks…
```
- Clean up test descriptions
- Rename typo filename
```
ffef735
Add ExtractImageDetailsDF. (#415 )

SinghGursimran authored and ruebot committed Jan 21, 2020

Loading status checks…
```
- Add test
- Addresses #223
```
71b459c

Commits on Jan 18, 2020

Add crawl_date to binary DataFrames and imageLinks. (#414 )

ruebot authored and ianmilligan1 committed Jan 18, 2020

Loading status checks…
```
- Resolves #413
- Update tests where necessary
```
9e357cc

Commits on Jan 17, 2020

Various DataFrame implementation updates for documentation clean-up; …

ruebot authored and ianmilligan1 committed Jan 17, 2020

…Addresses #372.

- .all() column HttpStatus to http_status_code
- Adds archive_filename to .all()
- Significant README updates for setup
- See also: archivesunleashed/aut-docs#39

9277e68

Older

archivesunleashed / aut