archivesunleashed/aut

Commits on Feb 10, 2020

Python work check-in

ruebot committed Feb 10, 2020

Verified

This commit was signed with a verified signature.

ruebot Nick Ruest

GPG key ID: 417FAF1A0E1080CD Learn about signing commits

f237c35

Commits on Jan 23, 2020

Start adding filters; keep_valid_pages.

ruebot committed Jan 23, 2020

Loading status checks…
```
- TODO, make it object oriented
```
Verified

This commit was signed with a verified signature.

ruebot Nick Ruest

GPG key ID: 417FAF1A0E1080CD Learn about signing commits

5255c10
clean-up

ruebot committed Jan 23, 2020

Verified

This commit was signed with a verified signature.

ruebot Nick Ruest

GPG key ID: 417FAF1A0E1080CD Learn about signing commits

5803b4d

Commits on Jan 22, 2020

Merge branch 'master' into issue-409

ruebot committed Jan 22, 2020

Loading status checks…

Verified

This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.

GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits

3b16fd4

Commits on Jan 21, 2020

Clean up test descriptions, addresses #372 . (#416 )

ruebot authored and ianmilligan1 committed Jan 21, 2020

Loading status checks…
```
- Clean up test descriptions
- Rename typo filename
```
ffef735
Merge branch 'master' into issue-409

ruebot committed Jan 21, 2020

Loading status checks…

Verified

This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.

GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits

e8bf94a
Add ExtractImageDetailsDF. (#415 )

SinghGursimran authored and ruebot committed Jan 21, 2020

Loading status checks…
```
- Add test
- Addresses #223
```
71b459c
- Remove order udfs alphabetically

ruebot committed Jan 21, 2020

Loading status checks…
```
- Get detect_language setup
- ComputeSHA1 and MD5 need some work?
```
Verified

This commit was signed with a verified signature.

ruebot Nick Ruest

GPG key ID: 417FAF1A0E1080CD Learn about signing commits

825a933
Merge branch 'issue-409' of github.com:archivesunleashed/aut into iss…

ruebot committed Jan 21, 2020
```
…ue-409
```
Verified

This commit was signed with a verified signature.

ruebot Nick Ruest

GPG key ID: 417FAF1A0E1080CD Learn about signing commits

7060f5f
Setup external lib packaging for Python!!

ruebot committed Jan 21, 2020
```
Add remove_html udf
Rename remove_http_header to remove_http_headers
```
Verified

This commit was signed with a verified signature.

ruebot Nick Ruest

GPG key ID: 417FAF1A0E1080CD Learn about signing commits

0a094cc

Commits on Jan 19, 2020

Matchbox-pyhton1

g285sing committed Jan 19, 2020

Loading status checks…

2
5830ab9
computeMD5

g285sing committed Jan 19, 2020

Loading status checks…

e310611

Commits on Jan 18, 2020

Merge branch 'master' into issue-409

ruebot committed Jan 18, 2020

Loading status checks…

Verified

This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.

GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits

d5a4f06
Add crawl_date to binary DataFrames and imageLinks. (#414 )

ruebot authored and ianmilligan1 committed Jan 18, 2020

Loading status checks…
```
- Resolves #413
- Update tests where necessary
```
9e357cc
rename links to webgraph

ruebot committed Jan 18, 2020

Loading status checks…

Verified

This commit was signed with a verified signature.

ruebot Nick Ruest

GPG key ID: 417FAF1A0E1080CD Learn about signing commits

af2bc5a
Add some PySpark udfs

ruebot committed Jan 18, 2020

Loading status checks…
```
- Add remove_http_header, remove_prefix_www
- Rename extract_domain_func to extract_domain
- Formatting updates
- Addresses #409
```
Verified

This commit was signed with a verified signature.

ruebot Nick Ruest

GPG key ID: 417FAF1A0E1080CD Learn about signing commits

0e8489b

Commits on Jan 17, 2020

Various DataFrame implementation updates for documentation clean-up; …

ruebot authored and ianmilligan1 committed Jan 17, 2020

…Addresses #372.

- .all() column HttpStatus to http_status_code
- Adds archive_filename to .all()
- Significant README updates for setup
- See also: archivesunleashed/aut-docs#39

9277e68

Commits on Jan 16, 2020

Use https for maven repo. (#405 )

ruebot authored and ianmilligan1 committed Jan 16, 2020

- Looks like repos are forcing https to be used now:
[WARNING] repository metadata for: 'artifact joda-time:joda-time' could not be retrieved from repository: maven due to an error: Failed to transfer file: http://repo.maven.apache.org/maven2/joda-time/joda-time/maven-metadata.xml. Return code is: 501 , ReasonPhrase:HTTPS Required.

4c6875d

Commits on Jan 13, 2020

Test clean-up. (#404 )

ruebot authored and ianmilligan1 committed Jan 13, 2020

Loading status checks…
```
- Clean-up variable names in RecordDFTest.scala
- Remove dos line endings on a number of files
```
75b7502

Commits on Jan 12, 2020

Add language detection column to webpages. (#403 )

ruebot authored and ianmilligan1 committed Jan 12, 2020

Loading status checks…
```
- Addresses #402
```
bc0d663

Commits on Jan 10, 2020

Add more DataFrame Implementation Serializable APIs (#401 ).

SinghGursimran authored and ruebot committed Jan 10, 2020

- Partially addresses  #223 
- Add discardContentDF
- Add discardUrlPatternsDF
- Add discardLanguagesDF
- Add keepImagesDF
- Add keepContentDF
- Add keepUrlPatternsDF
- Add keepLanguagesDF
- Update tests

0ecc4f8

Commits on Jan 8, 2020

Filter blank src/dest out of webgraph. (#400 )

ruebot authored and ianmilligan1 committed Jan 8, 2020

Loading status checks…

3dc1545

Commits on Jan 7, 2020

Add more DF implementations for #223 . (#399 )

SinghGursimran authored and ruebot committed Jan 7, 2020

Loading status checks…
```
- Add discardHttpStatusDF
- Add keepMimeTypesDF
- Add keepMimeTypesTikaDF
- Update tests
```
be15375

Commits on Jan 5, 2020

Scala imports cleanup. (#398 )

ruebot authored and ianmilligan1 committed Jan 5, 2020

Loading status checks…

d5c7bf7

Commits on Dec 29, 2019

Add more serializable APIs for DataFrames (#396 )

SinghGursimran authored and ruebot committed Dec 29, 2019

Loading status checks…
```
- Partially address #223 
- Add keepHttpStatusDF
- Add keepDateDF
- Add keepUrlsDF
- Add keepDomainsDF
- Add tests
```
b915f82

Commits on Dec 19, 2019

Remove redundant test; addresses #64 . (#395 )

ruebot authored and ianmilligan1 committed Dec 19, 2019

Loading status checks…

2c96ff3

Commits on Dec 18, 2019

Add doc comments for webpages and webgraph; resolves #392 . (#394 )

ruebot authored and ianmilligan1 committed Dec 18, 2019

Loading status checks…

99e9d06
Add additional filters for fextFiles; resolves #362 . (#393 )

ruebot authored and ianmilligan1 committed Dec 18, 2019

Loading status checks…
```
* Add additional filters for fextFiles; resolves #362.

- Add filedesc, and dns filter (arc files)
- Add test case
```
8eb43ff

Commits on Dec 17, 2019

udf API implementations for DataFrame (#391 )

SinghGursimran authored and ruebot committed Dec 17, 2019

Loading status checks…
```
- add discardMimeTypesDF
- add discardDateDF
- add discardUrlsDF
- add discardDomainsDF
- update tests
- addresses #223
```
40a59de
Add Serializable APIs for DataFrames (#389 )

SinghGursimran authored and ruebot committed Dec 17, 2019

Loading status checks…
```
- Add keepValidPagesDF
- Add HTTP status code column to all()
- Add test for keepValidPagesDF
- Addresses #223
```
ca928d8

Add and update tests, resolve textFiles bug. (#388 )

ruebot authored and ianmilligan1 committed Dec 17, 2019

- Add ExtractDateDF test
- Fix conditional logic of textFiles filter to resolve #390
- Add test for conditional logic fix for #390
- Remove cruft ExtractUrls, left over from Twitter analysis removal
(see:
https://github.com/lintool/warcbase/blob/cab311ed8b0bceb666865fa76dd3bc5a86402e13/warcbase-core/src/test/scala/org/warcbase/spark/matchbox/ExtractUrlsTest.scala)
- Tweak null/nothing on a few tests

9e32284

Commits on Dec 5, 2019

Add new DataFrame matchbox udfs (#387 )

SinghGursimran authored and ruebot committed Dec 5, 2019

- Add DetectLanguageDF
- Add ExtractBoilerpipeTextDF
- Add ExtractDateDF
- Update tests
- Rename existing ExtractDate, ExtractBoilerpipeText, DetectLanguage udfs by appending RDD
- Partially addresses #223

079cd24

Commits on Nov 28, 2019

Clean-up underscore import, and scalastyle warnings. (#386 )

ruebot authored and ianmilligan1 committed Nov 28, 2019

Loading status checks…

560ed2b

Commits on Nov 21, 2019

Add "Extract popular images" DataFrame implementation (#382 ).

SinghGursimran authored and ruebot committed Nov 21, 2019

Loading status checks…
```
- Add tests for ExtractPopularImagesDF
- Rename ExtractPopularImages to ExtractPopularImagesRDD
- Addresses #223
```
4042180

Add all() method and refactor DF UDFs (#383 ).

SinghGursimran authored and ruebot committed Nov 21, 2019

- Add `all()` DataFrame method 
- Refactor fixity DataFrame UDFs
- Add ComputeImageSize UDF
- Add Python implementation of `all()`
- Addresses #223

c4eaca9

Older

archivesunleashed / aut