archivesunleashed/aut

Commits on Jul 18, 2019

Move data frame fields names to snake_case. (#327 )

ruebot authored and ianmilligan1 committed Jul 18, 2019

Loading status checks…
```
- Resolves #229
```
f35d54e

Commits on Jul 17, 2019

Update to Spark 2.4.3 and update Tika to 1.20. (#321 )

ruebot authored and ianmilligan1 committed Jul 17, 2019

* Update to Spark 2.4.3 and update Tika to 1.20.

- Resolves #295
- Resolves #308
- Resolves #286
- Pulls in unfinished work by @jrwiebe and @borislin.

* Add patched lang-detector

0e701b2

Commits on Jul 15, 2019

Remove Tweet utils. (#323 )

ruebot authored and ianmilligan1 committed Jul 15, 2019

Loading status checks…
```
- Resolves #322
- Resolves #206
- Resolves #194
```
20ffeeb

Commits on Jul 8, 2019

Test Java 8 & 11, and remove OracleJDK; resolves #324 . (#325 )

ruebot authored and ianmilligan1 committed Jul 8, 2019

Loading status checks…

4ce59c8

Commits on Jul 5, 2019

Add image analysis and extraction w/TensorFlow (#318 )

h324yang authored and ruebot committed Jul 5, 2019

Loading status checks…

7a61f0e

Commits on Apr 22, 2019

Makes ArchiveRecordImpl serializable by removing non-serializable ARC…

jrwiebe authored and ruebot committed Apr 22, 2019

Loading status checks…
```
…Record and WARCRecord variables. Also removes unused headerResponseFormat variable. (#316)
```
5cb05f7

Commits on Mar 23, 2019

Resolve cobertura-maven-plugin class issue; resolves #313 . (#314 )

ruebot authored and jrwiebe committed Mar 23, 2019

Loading status checks…
```
- Exclude slf4j binding logback-classic
(mojohaus/cobertura-maven-plugin#6 (comment))
```
8504190

Commits on Mar 18, 2019

Commits on Jan 31, 2019

Log closing of ARC and WARC files, resolves #156 (#301 )

jrwiebe authored and ruebot committed Jan 31, 2019

* Log opening and closing of archive files as per #156
* Remove redundant log message. Spark already logs the file that is to be read when an executor computes an RDD.

fc0178d

Commits on Jan 24, 2019

Delete saved image file; resolves #299 (#300 )

jrwiebe authored and ruebot committed Jan 24, 2019

Loading status checks…

1e69040

Commits on Nov 28, 2018

Remove Deprecated ExtractGraph app; resolves #291 . (#293 )

greebie authored and ruebot committed Nov 28, 2018

Loading status checks…
```
* Remove deprecated ExtractGraph.scala file.
* Remove deprecated ExtractGraphTest.scala file.
```
ab32df4

Add .getHttpStatus and .getArchiveFile to ArchiveRecordImpl class #198 …

greebie authored and ruebot committed Nov 28, 2018

…& #164 (#292)

* Resolves #198
* Resolves #164
* Add getHttpStatus to ArchiveRecord class & trait
  - add .getHttpStatus to potential outputs
  - add tests for .getHttpStatus calls
  - improve ArchiveRecord testing overall.
* Add .getArchiveFile feature to ArchiveRecordImpl.
  - add getArchiveFile to trait
  - add getArchiveFile for ArchiveRecordImpl
  - add tests for getArchiveFile.
* Other code style fixes.
* Include updates to tests.

7731b6d

Commits on Nov 22, 2018

Update license headers for #208 . (#290 )

ruebot authored and ianmilligan1 committed Nov 22, 2018

Loading status checks…

80b9e2b

Change Id generation for graphs from using hashes for urls to using .…

greebie authored and ruebot committed Nov 22, 2018

…zipWithUniqueIds() (#289)

* Resolves #243 
* Create GEXF with proper ids instead of hash to avoid collisions.
* Add WriteGEXF files.
* Add WriteGraph file and test.
* Add test for Graphml output.
* Add xml escaping for edges.
* Add test case for non-escaped edges.
* Add additional tests to cover for more potential cases of graphml and gexf files.
* Coverage for null cases in urls.

d3aebf4

Commits on Oct 19, 2018

CVE-2018-11771 update (#288 )

ruebot authored and ianmilligan1 committed Oct 19, 2018

Loading status checks…

e6080a7

Commits on Oct 18, 2018

CVE-2017-17485 update; follow-on to #281 . (#287 )

ruebot authored and ianmilligan1 committed Oct 18, 2018

Loading status checks…

3776489

Commits on Oct 17, 2018

Update Apache Tika - security vulnerabilities; resolves #131 . (#285 )

ruebot authored and ianmilligan1 committed Oct 17, 2018

Loading status checks…
```
- CVE-2018-1338
- CVE-2018-11762
- CVE-2018-11761
- CVE-2016-6809
- CVE-2018-1339
- CVE-2018-11796
- CVE-2016-4434
- CVE-2018-1335
```
b9260be
Only trigger TravisCI on master. (#283 )

ruebot authored and ianmilligan1 committed Oct 17, 2018

Loading status checks…

0d4f9bb
[skip travis] Update README (#284 )

ruebot authored and ianmilligan1 committed Oct 17, 2018

f8ee21e
Fix bug and unit test for ExtractDomain; resolves #277 (#278 )

borislin authored and ruebot committed Oct 17, 2018

Loading status checks…

2c66b09
Replace backslash with forward slash in URL; resolves #269 (#276 )

borislin authored and ruebot committed Oct 17, 2018

Loading status checks…
```
* Fix backslash in URL
* Add backslash test in ExtractDomainTest
```
7c3a80d
Missed something for #208 . (#282 )

ruebot authored and ianmilligan1 committed Oct 17, 2018

Loading status checks…

6b4ef94

Commits on Oct 16, 2018

CVE-2018-7489 fix. (#281 )

ruebot authored and ianmilligan1 committed Oct 16, 2018

Loading status checks…
```
- Follow on to
72cb5e2
- https://nvd.nist.gov/vuln/detail/CVE-2018-7489
```
4fe05a5
Update jackson-databind version; resolves #279 . (#280 )

ruebot authored and ianmilligan1 committed Oct 16, 2018

Loading status checks…
```
- CVE-2017-752
- See also: https://nvd.nist.gov/vuln/detail/CVE-2017-7525
```
72cb5e2

Commits on Oct 9, 2018

Clean-up pom.xml to remove plugin warnings; resolves #273 . (#274 )

ruebot authored and ianmilligan1 committed Oct 9, 2018

Loading status checks…

f19dc9a

Commits on Oct 4, 2018

[maven-release-plugin] prepare for next development iteration

ruebot committed Oct 4, 2018

Loading status checks…

Verified

This commit was signed with a verified signature.

ruebot Nick Ruest

GPG key ID: 417FAF1A0E1080CD Learn about signing commits

9796f50
[maven-release-plugin] prepare release aut-0.17.0

ruebot committed Oct 4, 2018

Loading status checks…

Verified

This commit was signed with a verified signature.

ruebot Nick Ruest

GPG key ID: 417FAF1A0E1080CD Learn about signing commits

694382c

Fix exception error when processing corrupted ARC files, and empty fi…

borislin authored and ruebot committed Oct 4, 2018

…les. (#272)

* Fix exception when processing corrupted ARC files
* Filter out non-empty archive files in loadArchives()
* Fix archive files path pattern
* Resolves #246
* Resolves #271 
* Resolves #258

b8e57ec

Commits on Sep 10, 2018

Update Bug report template. (#268 )

ruebot authored and ianmilligan1 committed Sep 10, 2018

Loading status checks…

c95a51d

Commits on Aug 11, 2018

ExtractBoilerpipeText to remove headers as well. #253 (#256 )

greebie authored and ruebot committed Aug 11, 2018

Loading status checks…
```
* ExtractBoilerpipeText now removes headers.
```
84a4c09

Add additional tweet fields to TweetUtils; partially address #194 . (#254

ruebot authored and ianmilligan1 committed Aug 11, 2018

)

- Adds:
  - retweet_count
  - favorite_count
  - in_reply_to_status_id_str
  - in_reply_to_user_id_str
  - in_reply_to_screen_name
  - source
  - user.protected
  - user.profile_image_url
  - user.description
  - user.location
  - user.name
  - user.url
  - user.time_zone
- Updates some doc comments
- Updates tests

e4cf9a7

Commits on Aug 10, 2018

Add support for full_text in tweets; resolve #192 . (#252 )

ruebot authored and ianmilligan1 committed Aug 10, 2018

Loading status checks…

62628b4
Get rid of 'filesystem-root relative reference' warning. (#251 )

ruebot authored and ianmilligan1 committed Aug 10, 2018

Loading status checks…

a5fa151

Commits on Aug 9, 2018

Remove stray characters from example commands. (#250 )

ruebot authored and ianmilligan1 committed Aug 9, 2018

Loading status checks…

9efde66
Deal with final scalastyle assessments, and Convert nulls to Option(T…

greebie authored and ruebot committed Aug 9, 2018

Loading status checks…
```
…). (#249)

* Fully resolves #196 
* Resolves #212
```
004ce1f

Older