The Archives Unleashed Toolkit
The Archives Unleashed Toolkit is an open-source platform for analyzing web archives built on Apache Spark, which provides powerful tools for analytics and data processing. The Toolkit is part of the Archives Unleashed Project.
Learn more about the Toolkit and how to use it by visiting our comprehensive documentation.
The following two articles provide an overview of the project:
- Nick Ruest, Jimmy Lin, Ian Milligan, and Samantha Fritz. The Archives Unleashed Project: Technology, Process, and Community to Improve Scholarly Access to Web Archives. Proceedings of the 2020 IEEE/ACM Joint Conference on Digital Libraries (JCDL 2020), Wuhan, China.
- Jimmy Lin, Ian Milligan, Jeremy Wiebe, and Alice Zhou. Warcbase: Scalable Analytics Infrastructure for Exploring Web Archives. ACM Journal on Computing and Cultural Heritage, 10(4), Article 22, 2017.
Dependencies
- Java 11
- Python 3.7.3+ (PySpark)
- Apache Spark 3.0.0+
More information on setting up dependencies can be found here.
Building
Clone the repo:
git clone http://github.com/archivesunleashed/aut.gitYou can then build The Archives Unleashed Toolkit.
mvn clean installUsage
The Toolkit can be used to submit a variety of extraction jobs with spark-submit, as well used as a library via spark-submit, pyspark, or in your own application. More information on using the Toolkit can be found here.
License
Licensed under the Apache License, Version 2.0.
Acknowledgments
This work is primarily supported by the Andrew W. Mellon Foundation. Other financial and in-kind support comes from the Social Sciences and Humanities Research Council, Compute Canada, the Ontario Ministry of Research, Innovation, and Science, York University Libraries, Start Smart Labs, and the Faculty of Arts and David R. Cheriton School of Computer Science at the University of Waterloo.
Any opinions, findings, and conclusions or recommendations expressed are those of the researchers and do not necessarily reflect the views of the sponsors.
