Skip to content
The Archives Unleashed Toolkit is an open-source toolkit for analyzing web archives.
Scala Java Python
Branch: master
Clone or download
jrwiebe and ruebot Add PDF binary extraction. (#340)
Introduces the new extractPDFDetailsDF() method and brings in changes to make our use of Tika's MIME type detection more efficient, as well as POM updates to use a shaded version of tika-parsers in order to eliminate a dependency version conflict that has long been troublesome.

- Updates getImageBytes to getBinaryBytes
- Refactor SaveImage class to more general SaveBytes, and saveToDisk to saveImageToDisk
- Only instantiate Tika when the DetectMimeTypeTika singleton object is first referenced. See https://git.io/fj7g0.
- Use TikaInputStream to enabler container-aware detection. Until now we were only using the default Mime Magic detection. See https://tika.apache.org/1.22/detection.html#Container_Aware_Detection.
- Added generic saveToDisk method to save a bytes column of a DataFrame to files
- Updates tests
- Resolves #302
- Further addresses #308
- Includes work by @ruebot, see #340 for all commits before squash
Latest commit 73981a7 Aug 11, 2019

README.md

The Archives Unleashed Toolkit

Build Status codecov Maven Central Javadoc Scaladoc LICENSE Contribution Guidelines

The Archives Unleashed Toolkit is an open-source toolkit for analyzing web archives built around Apache Spark. This toolkit is part of the Archives Unleashed Project.

The toolkit grew out of a previous project called Warcbase. The following article provides a nice overview, much of which is still relevant:

Getting Started

Easy

If you have Apache Spark ready to go, it's as easy as:

$ spark-shell --packages "io.archivesunleashed:aut:0.17.0"

A little less easy

You can download the latest release here and include it like so:

$ spark-shell --jars /path/to/aut-0.17.0-fatjar.jar"

Even less easy

Build it yourself as per the instructions below:

Clone the repo:

$ git clone http://github.com/archivesunleashed/aut.git

You can then build The Archives Unleashed Toolkit.

$ mvn clean install

For the impatient, to skip tests:

$ mvn clean install -DskipTests

I want to use Docker!

Ok! Take a quick spin with aut with Docker.

Documentation! Or, how do I use this?

Once built or downloaded, you can follow the basic set of recipes and tutorials here.

License

Licensed under the Apache License, Version 2.0.

Acknowledgments

This work is primarily supported by the Andrew W. Mellon Foundation. Additional funding for the Toolkit has come from the U.S. National Science Foundation, Columbia University Library's Mellon-funded Web Archiving Incentive Award, the Natural Sciences and Engineering Research Council of Canada, the Social Sciences and Humanities Research Council of Canada, and the Ontario Ministry of Research and Innovation's Early Researcher Award program. Any opinions, findings, and conclusions or recommendations expressed are those of the researchers and do not necessarily reflect the views of the sponsors.

You can’t perform that action at this time.