Test and documentation inventory #372

ruebot · 2019-11-06T00:51:13Z

This ticket is meant to evaluate the current documentation against tests. Make sure the tests and examples are consistent. If there is a difference, not them below in a comment. Then create a pull request for each one, or all of them.

Current documentation examples are here, and pull requests should be made against that repo as necessary, and should reference this issue.

Where tests are missing, check and see if an issue exists. If one does not exist, please create one, or let us know in a comment.

ruebot · 2019-11-08T22:37:49Z

@SinghGursimran here's one for you.

ruebot · 2020-01-15T14:33:57Z

Documentation inventory and review. Lots to discuss here.

Filters

Python implementation of UDF filters.

Binary Analysis

Scala implementation; should we change the TODO to will not be implemented?
- Same here https://github.com/archivesunleashed/aut-docs/blob/master/current/image-analysis.md

Collection Analysis

https://github.com/archivesunleashed/aut-docs/blob/master/current/collection-analysis.md#extract-different-subdomains
- Scala DF
- Python DF
https://github.com/archivesunleashed/aut-docs/blob/master/current/collection-analysis.md#extract-http-status-codes
- Scala DF
- Python DF
https://github.com/archivesunleashed/aut-docs/blob/master/current/collection-analysis.md#extract-the-location-of-the-resource-in-arcs-and-warcs
- Scala DF
- Python DF

DF Results

https://github.com/archivesunleashed/aut-docs/blob/master/current/df-results.md#python
- Crib from TWUT

Index

Delete https://github.com/archivesunleashed/aut-docs/blob/master/current/index.md

Link Analysis

https://github.com/archivesunleashed/aut-docs/blob/master/current/link-analysis.md#extract-simple-site-link-structure
- Python
https://github.com/archivesunleashed/aut-docs/blob/master/current/link-analysis.md#extract-raw-url-link-structure
- Scala DF
- Python DF
https://github.com/archivesunleashed/aut-docs/blob/master/current/link-analysis.md#organize-links-by-url-pattern
- Scala DF
- Python DF
https://github.com/archivesunleashed/aut-docs/blob/master/current/link-analysis.md#organize-links-by-crawl-date
- Scala DF
https://github.com/archivesunleashed/aut-docs/blob/master/current/link-analysis.md#export-as-tsv
- Do we care about TSV? Can we just delete the section, or mention it in RDD results?
https://github.com/archivesunleashed/aut-docs/blob/master/current/link-analysis.md#filter-by-url
- Scala DF
- Python DF
https://github.com/archivesunleashed/aut-docs/blob/master/current/link-analysis.md#export-to-gephi
- Scala DF
- Python DF
- In progress by @SinghGursimran
https://github.com/archivesunleashed/aut-docs/blob/master/current/link-analysis.md#finding-hyperlinks-within-collection-on-pages-with-certain-keyword
- Scala RDD (won't implement?)
- Python DF

Setting up the Archives Unleashed Toolkit

Fix ToC
Review closer and align with README

Standard Derivatives

https://github.com/archivesunleashed/aut-docs/blob/master/current/standard-derivatives.md#create-the-archives-unleashed-cloud-scholarly-derivatives
- Scala DF (#397 and archivesunleashed/aut-docs#34)
- Python DF
https://github.com/archivesunleashed/aut-docs/blob/master/current/standard-derivatives.md#extract-binary-info
- Scala RDD (Won't implement language?)
https://github.com/archivesunleashed/aut-docs/blob/master/current/standard-derivatives.md#extract-binaries-to-disk
- Scala RDD (Won't implement language?)
- Python DF

Text Analysis

https://github.com/archivesunleashed/aut-docs/blob/master/current/text-analysis.md#extract-all-plain-text
- Scala DF
- Python DF
https://github.com/archivesunleashed/aut-docs/blob/master/current/text-analysis.md#extract-plain-text-without-http-headers
- Scala DF
- Python DF
https://github.com/archivesunleashed/aut-docs/blob/master/current/text-analysis.md#extract-plain-text-by-domain
- Scala DF
- Python DF
https://github.com/archivesunleashed/aut-docs/blob/master/current/text-analysis.md#extract-plain-text-by-url-pattern
- Scala DF
- Python DF
https://github.com/archivesunleashed/aut-docs/blob/master/current/text-analysis.md#extract-plain-text-minus-boilerplate
- Scala DF
- Python DF
https://github.com/archivesunleashed/aut-docs/blob/master/current/text-analysis.md#extract-plain-text-filtered-by-date
- Scala DF
- Python DF
https://github.com/archivesunleashed/aut-docs/blob/master/current/text-analysis.md#extract-plain-text-filtered-by-language
- Scala DF
- Python DF
https://github.com/archivesunleashed/aut-docs/blob/master/current/text-analysis.md#extract-plain-text-filtered-by-keyword
- Scala DF
- Python DF
https://github.com/archivesunleashed/aut-docs/blob/master/current/text-analysis.md#extract-raw-html
- Scala DF
- Python DF
https://github.com/archivesunleashed/aut-docs/blob/master/current/text-analysis.md#extract-named-entities
- I think we need to do a DF udf for NER #223.
Archives Unleashed Toolkit Walkthrough
- Lots of duplication with https://github.com/archivesunleashed/aut-docs/blob/master/current/setting-up-aut.md
  - Clean up for homework, datathon instructions?


        Address #372 - DRAFT


        Address archivesunleashed/aut#372 - DRAFT


        Various DataFrame implementation updates for documentation clean-up; …

…Addresses #372. - .all() column HttpStatus to http_status_code - Adds archive_filename to .all() - Significant README updates for setup - See also: archivesunleashed/aut-docs#39


        Update documentation for archivesunleashed/aut#372 (#39)

* Add DF results Python section * Add won't implement language to binary analysis. * Add won't implement language to standard derivatives. * Remove index, fix ToC in setting up. * Update README, add scala df to link analys, add TSV rdd results * text-analysis scala df * Add to be implemented * Resolves #22


        Clean up test descriptions, addresses #372.

- Rename typo filename - Remove potentially unnecessary tests


        Clean up test descriptions, addresses #372. (#416)

- Clean up test descriptions - Rename typo filename

ruebot · 2020-01-22T00:22:17Z

I think we're good here now.

ruebot added tests documentation labels Nov 6, 2019

ruebot mentioned this issue Nov 8, 2019

Migration of all RDD functionality over to DataFrames #223

Open

ruebot added this to To Do in 1.0.0 Release of AUT Nov 14, 2019

ruebot added a commit that referenced this issue Jan 16, 2020

Address #372 - DRAFT

Verified

This commit was signed with a verified signature.

ruebot Nick Ruest

GPG key ID: 417FAF1A0E1080CD Learn about signing commits

Loading status checks…

debf5d5

ruebot added a commit to archivesunleashed/aut-docs that referenced this issue Jan 16, 2020

Address archivesunleashed/aut#372 - DRAFT

Verified

This commit was signed with a verified signature.

ruebot Nick Ruest

GPG key ID: 417FAF1A0E1080CD Learn about signing commits

88789a9

This was referenced Jan 16, 2020

Various DataFrame implementation updates for documentation clean-up; Addresses #372. #406

Merged

Address https://github.com/archivesunleashed/aut/issues/372 - DRAFT archivesunleashed/aut-docs#39

Merged

ruebot mentioned this issue Jan 21, 2020

Clean up test descriptions, addresses #372. #416

Merged

ianmilligan1 added a commit that referenced this issue Jan 21, 2020

Clean up test descriptions, addresses #372. (#416)

Loading status checks…

ffef735

- Clean up test descriptions - Rename typo filename

ruebot closed this Jan 22, 2020

1.0.0 Release of AUT automation moved this from To Do to Done Jan 22, 2020

ruebot self-assigned this Jan 22, 2020

Please note that GitHub no longer supports your web browser.

archivesunleashed / aut

Test and documentation inventory #372

Test and documentation inventory #372

ruebot commented Nov 6, 2019

This comment has been minimized.

ruebot commented Nov 8, 2019

This comment has been minimized.

ruebot commented Jan 15, 2020

This comment has been minimized.

ruebot commented Jan 22, 2020

Please note that GitHub no longer supports your web browser.

archivesunleashed / aut

Join GitHub today

Test and documentation inventory #372

Test and documentation inventory #372

Comments

ruebot commented Nov 6, 2019

This comment has been minimized.

ruebot commented Nov 8, 2019

This comment has been minimized.

ruebot commented Jan 15, 2020

This comment has been minimized.

ruebot commented Jan 22, 2020