@SinghGursimran here's one for you.

Documentation inventory and review. Lots to discuss here.

Filters

• Python implementation of UDF filters.

Binary Analysis

• Scala implementation; should we change the TODO to will not be implemented?
  • Same here https://github.com/archivesunleashed/aut-docs/blob/master/current/image-analysis.md

Collection Analysis

• https://github.com/archivesunleashed/aut-docs/blob/master/current/collection-analysis.md#extract-different-subdomains
  • Scala DF
  • Python DF
• https://github.com/archivesunleashed/aut-docs/blob/master/current/collection-analysis.md#extract-http-status-codes
  • Scala DF
  • Python DF
• https://github.com/archivesunleashed/aut-docs/blob/master/current/collection-analysis.md#extract-the-location-of-the-resource-in-arcs-and-warcs
  • Scala DF
  • Python DF

DF Results

• https://github.com/archivesunleashed/aut-docs/blob/master/current/df-results.md#python
  • Crib from TWUT

Index

• Delete https://github.com/archivesunleashed/aut-docs/blob/master/current/index.md

Link Analysis

• https://github.com/archivesunleashed/aut-docs/blob/master/current/link-analysis.md#extract-simple-site-link-structure
  • Python
• https://github.com/archivesunleashed/aut-docs/blob/master/current/link-analysis.md#extract-raw-url-link-structure
  • Scala DF
  • Python DF
• https://github.com/archivesunleashed/aut-docs/blob/master/current/link-analysis.md#organize-links-by-url-pattern
  • Scala DF
  • Python DF
• https://github.com/archivesunleashed/aut-docs/blob/master/current/link-analysis.md#organize-links-by-crawl-date
  • Scala DF
• https://github.com/archivesunleashed/aut-docs/blob/master/current/link-analysis.md#export-as-tsv
  • Do we care about TSV? Can we just delete the section, or mention it in RDD results?
• https://github.com/archivesunleashed/aut-docs/blob/master/current/link-analysis.md#filter-by-url
  • Scala DF
  • Python DF
• https://github.com/archivesunleashed/aut-docs/blob/master/current/link-analysis.md#export-to-gephi
  • Scala DF
  • Python DF
  • In progress by @SinghGursimran
• https://github.com/archivesunleashed/aut-docs/blob/master/current/link-analysis.md#finding-hyperlinks-within-collection-on-pages-with-certain-keyword
  • Scala RDD (won't implement?)
  • Python DF

Setting up the Archives Unleashed Toolkit

• Fix ToC
• Review closer and align with README

Standard Derivatives

• https://github.com/archivesunleashed/aut-docs/blob/master/current/standard-derivatives.md#create-the-archives-unleashed-cloud-scholarly-derivatives
  • Scala DF (#397 and archivesunleashed/aut-docs#34)
  • Python DF
• https://github.com/archivesunleashed/aut-docs/blob/master/current/standard-derivatives.md#extract-binary-info
  • Scala RDD (Won't implement language?)
• https://github.com/archivesunleashed/aut-docs/blob/master/current/standard-derivatives.md#extract-binaries-to-disk
  • Scala RDD (Won't implement language?)
  • Python DF

Text Analysis

• https://github.com/archivesunleashed/aut-docs/blob/master/current/text-analysis.md#extract-all-plain-text

  • Scala DF
  • Python DF

• https://github.com/archivesunleashed/aut-docs/blob/master/current/text-analysis.md#extract-plain-text-without-http-headers

  • Scala DF
  • Python DF

• https://github.com/archivesunleashed/aut-docs/blob/master/current/text-analysis.md#extract-plain-text-by-domain

  • Scala DF
  • Python DF

• https://github.com/archivesunleashed/aut-docs/blob/master/current/text-analysis.md#extract-plain-text-by-url-pattern

  • Scala DF
  • Python DF

• https://github.com/archivesunleashed/aut-docs/blob/master/current/text-analysis.md#extract-plain-text-minus-boilerplate

  • Scala DF
  • Python DF

• https://github.com/archivesunleashed/aut-docs/blob/master/current/text-analysis.md#extract-plain-text-filtered-by-date

  • Scala DF
  • Python DF

• https://github.com/archivesunleashed/aut-docs/blob/master/current/text-analysis.md#extract-plain-text-filtered-by-language

  • Scala DF
  • Python DF

• https://github.com/archivesunleashed/aut-docs/blob/master/current/text-analysis.md#extract-plain-text-filtered-by-keyword

  • Scala DF
  • Python DF

• https://github.com/archivesunleashed/aut-docs/blob/master/current/text-analysis.md#extract-raw-html

  • Scala DF
  • Python DF

• https://github.com/archivesunleashed/aut-docs/blob/master/current/text-analysis.md#extract-named-entities

  • I think we need to do a DF udf for NER #223.

• Archives Unleashed Toolkit Walkthrough

  • Lots of duplication with https://github.com/archivesunleashed/aut-docs/blob/master/current/setting-up-aut.md
    • Clean up for homework, datathon instructions?