The Archives Unleashed Toolkit: Latest Documentation

The Archives Unleashed Toolkit is an open-source platform for analyzing web archives built on Apache Spark, which provides powerful tools for analytics and data processing.

This documentation is based on a cookbook approach, providing a series of "recipes" for addressing a number of common analytics tasks to provide inspiration for your own analysis. We generally provide examples for resilient distributed datasets (RDD) in Scala, and DataFrames in both Scala and Python. We leave it up to you to choose Scala or Python flavours of Spark.

If you want to learn more about Apache Spark, we highly recommend Spark: The Definitive Guide

Table of Contents

Our documentation is divided into several main sections, which cover the Archives Unleashed Toolkit workflow from analyzing collections to understanding and working with the results.

Getting Started

• Setting Things Up
• Using the Archives Unleashed Toolkit at Scale
• Archives Unleashed Toolkit Walkthrough
• DataFrame Schemas

Generating Results

• Collection Analysis: How do I...
  • Extract All URLs
  • Extract Top-Level Domains
  • Extract Different Subdomains
  • Extract HTTP Status Codes
  • Extract the Location of the Resource in ARCs and WARCs
• Text Analysis: How do I...
  • Extract All Plain Text
  • Extract Plain Text Without HTTP Headers
  • Extract Plain Text By Domain
  • Extract Plain Text by URL Pattern
  • Extract Plain Text Minus Boilerplate
  • Extract Plain Text Filtered by Date
  • Extract Plain Text Filtered by Language
  • Extract Plain Text Filtered by Keyword
  • Extract Raw HTML
  • Extract Named Entities
• Link Analysis: How do I...
  • Extract Simple Site Link Structure
  • Extract Raw URL Link Structure
  • Organize Links by URL Pattern
  • Organize Links by Crawl Date
  • Export as TSV
  • Filter by URL
  • Export to Gephi
• Image Analysis: How do I...
  • Extract Image Information
  • Extract Most Frequent Image URLs
  • Extract Most Frequent Images MD5 Hash
• Binary Analysis: How do I...
  • Extract Audio Information
  • Extract Image Information
  • Extract PDF Information
  • Extract Presentation Program File Information
  • Extract Spreadsheet Information
  • Extract Text File Information
  • Extract Video Information
  • Extract Word Processor File Information

Filtering Results

• Filters: A variety of ways to filter results.

Standard Derivatives

How do I...

• Create the Archives Unleashed Cloud Scholarly Derivatives
• Extract Binary Info
• Extract Binaries to Disk

What to do with Results

• What to do with DataFrame Results
• What to do with RDD Results

Further Reading

The toolkit grew out of a previous project called Warcbase. The following article provides a nice overview, much of which is still relevant:

Jimmy Lin, Ian Milligan, Jeremy Wiebe, and Alice Zhou. Warcbase: Scalable Analytics Infrastructure for Exploring Web Archives. ACM Journal on Computing and Cultural Heritage, 10(4), Article 22, 2017.

Acknowledgments

This work is primarily supported by the Andrew W. Mellon Foundation. Other financial and in-kind support comes from the Social Sciences and Humanities Research Council, Compute Canada, the Ontario Ministry of Research, Innovation, and Science, York University Libraries, Start Smart Labs, the Faculty of Arts and David R. Cheriton School of Computer Science at the University of Waterloo.

Any opinions, findings, and conclusions or recommendations expressed are those of the researchers and do not necessarily reflect the views of the sponsors.