Skip to content
Branch: master
Go to file
Clone

Latest commit

ruebot committed 00e8166 Jun 25, 2020
Add Python formatter GitHub Action. (#489)
- Setup Python formatter that runs isort and black.
- Rename scala formatter, so all three are consistent now.
- Resolves #488
- Normalize all the formatters
- Sets all the formatters to run, but not apply changes. It would be helpful to do this, but we can't because we can't do
that, because we can't push to a person's PR.
- Apply isort/black
- Make sure isort and black are consistent in how they are opinionated
about imports
- Use scalafmt 2.3.2 (same as GH action)

Files

Permalink
Failed to load latest commit information.

README.md

The Archives Unleashed Toolkit

Build Status codecov Maven Central Javadoc Scaladoc UserDocs LICENSE Contribution Guidelines

The Archives Unleashed Toolkit is an open-source platform for analyzing web archives built on Apache Spark, which provides powerful tools for analytics and data processing. The Toolkit is part of the Archives Unleashed Project.

Learn more about the Toolkit and how to use it by visiting our comprehensive documentation.

The following two articles provide an overview of the project:

Dependencies

  • Java 11
  • Python 3.7.3+ (PySpark)
  • Apache Spark 3.0.0+

More information on setting up dependencies can be found here.

Building

Clone the repo:

git clone http://github.com/archivesunleashed/aut.git

You can then build The Archives Unleashed Toolkit.

mvn clean install

Usage

The Toolkit can be used to submit a variety of extraction jobs with spark-submit, as well used as a library via spark-submit, pyspark, or in your own application. More information on using the Toolkit can be found here.

License

Licensed under the Apache License, Version 2.0.

Acknowledgments

This work is primarily supported by the Andrew W. Mellon Foundation. Other financial and in-kind support comes from the Social Sciences and Humanities Research Council, Compute Canada, the Ontario Ministry of Research, Innovation, and Science, York University Libraries, Start Smart Labs, and the Faculty of Arts and David R. Cheriton School of Computer Science at the University of Waterloo.

Any opinions, findings, and conclusions or recommendations expressed are those of the researchers and do not necessarily reflect the views of the sponsors.

You can’t perform that action at this time.