Skip to content
Please note that GitHub no longer supports your web browser.

We recommend upgrading to the latest Google Chrome or Firefox.

Learn more
The Archives Unleashed Toolkit is an open-source toolkit for analyzing web archives.
Scala Java Python
Branch: master
Clone or download
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.github Update Bug report template. (#268) Sep 10, 2018
config Update LICENSE and license headers. (#351) Aug 21, 2019
src Change RemoveHttpHeader to RemoveHTTPHeader. (#374) Nov 7, 2019
.codecov.yml Add office document binary extraction. (#346) Aug 16, 2019
.gitignore Python formatting, and gitignore additions. (#326) Jul 18, 2019
.travis.yml Test Java 8 & 11, and remove OracleJDK; resolves #324. (#325) Jul 8, 2019
CODE_OF_CONDUCT.md
CONTRIBUTING.md warcbase core moves to aut Jul 5, 2017
LICENSE Update LICENSE and license headers. (#351) Aug 21, 2019
README.md Updates description. See archivesunleashed/aut-docs-new#18 (#373) Nov 6, 2019
pom.xml Align NER output to WANE format; addresses #297 (#361) Nov 5, 2019

README.md

The Archives Unleashed Toolkit

Build Status codecov Maven Central Javadoc Scaladoc LICENSE Contribution Guidelines

The Archives Unleashed Toolkit is an open-source platform for analyzing web archives built on Apache Spark, which provides powerful tools for analytics and data processing. This toolkit is part of the Archives Unleashed Project.

The toolkit grew out of a previous project called Warcbase. The following article provides a nice overview, much of which is still relevant:

Getting Started

Easy

If you have Apache Spark ready to go, it's as easy as:

$ spark-shell --packages "io.archivesunleashed:aut:0.18.0"

A little less easy

You can download the latest release here and include it like so:

$ spark-shell --jars /path/to/aut-0.18.0-fatjar.jar"

Even less easy

Build it yourself as per the instructions below:

Clone the repo:

$ git clone http://github.com/archivesunleashed/aut.git

You can then build The Archives Unleashed Toolkit.

$ mvn clean install

For the impatient, to skip tests:

$ mvn clean install -DskipTests

I want to use Docker!

Ok! Take a quick spin with aut with Docker.

Documentation! Or, how do I use this?

Once built or downloaded, you can follow the basic set of recipes and tutorials here.

License

Licensed under the Apache License, Version 2.0.

Acknowledgments

This work is primarily supported by the Andrew W. Mellon Foundation. Other financial and in-kind support comes from the Social Sciences and Humanities Research Council, Compute Canada, the Ontario Ministry of Research, Innovation, and Science, York University Libraries, Start Smart Labs, and the Faculty of Arts and David R. Cheriton School of Computer Science at the University of Waterloo.

Any opinions, findings, and conclusions or recommendations expressed are those of the researchers and do not necessarily reflect the views of the sponsors.

You can’t perform that action at this time.