Skip to content

/aut

The Archives Unleashed Toolkit is an open-source toolkit for analyzing web archives.
spark hadoop webarchives analysis apache-spark java scala digital-humanities pyspark dataframe big-data-analytics
Scala Java Python
  1. Scala 81.7%
  2. Java 14.6%
  3. Python 3.7%
Tree: 448601eaf9
Find File
Clone or download

Clone with HTTPS

Use Git or checkout with SVN using the web URL.

Open in Desktop Download ZIP

Launching GitHub Desktop...

If nothing happens, download GitHub Desktop and try again.

Launching GitHub Desktop...

If nothing happens, download GitHub Desktop and try again.

Launching Xcode...

If nothing happens, download Xcode and try again.

Launching Visual Studio...

If nothing happens, download the GitHub extension for Visual Studio and try again.

@jrwiebe @ruebot
jrwiebe and ruebot Add method for determining binary file extension. (#349) 
This PR implements the strategy described in the discussion of the above issue to get an extension for a file described by a URL and a MIME type. It creates a GetExtensionMime object in the matchbox.

This PR also removes most of the filtering by URL from the image, audio, video, presentation, spreadsheet, and word processor document extraction methods, since these were returning false positives. (CSV and TSV files are a special case, since Tika detects them as "text/plain" based on content.)

Finally, I have inserted toLowerCase into the getUrl.endsWith() filter tests, which could possibly bring in some more CSV and TSV files

* Adds method for getting a file extension from a MIME type.
* Add getExtensions method to DetectMimeTypeTika.
* Matchbox object to get extension of URL
* Use GetExtensionMime for extraction methods; minor fixes.
* Remove tika-parsers classifier
* Remove most filtering by file extension from binary extraction methods; add CSV/TSV special cases.
* Fix GetExtensionMime case where URL has no extension but a MIME type is detected
* Insert `toLowerCase` into `getUrl.endsWith()` calls in io.archivesunleashed.packages; apply to `FilenameUtils.getExtension` in `GetExtensionMime`.
* Remove filtering on URL for audio, video, and images.
* Remove filtering on URL for images; add DF fields to image extraction
* Remove saveImageToDisk and its test
* Remove robots.txt check and extraneous imports
* Close files so we don't get too many files open again.
* Add GetExtensionMimeTest
* Resolve #343
Latest commit 448601e Aug 17, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.github Update Bug report template. (#268) Sep 10, 2018
config/checkstyle More scalastyle work; addresses #196. (#339) Aug 8, 2019
src Add method for determining binary file extension. (#349) Aug 18, 2019
.codecov.yml Add office document binary extraction. (#346) Aug 16, 2019
.gitignore Python formatting, and gitignore additions. (#326) Jul 18, 2019
.travis.yml Test Java 8 & 11, and remove OracleJDK; resolves #324. (#325) Jul 8, 2019
CODE_OF_CONDUCT.md Created Code of Conduct file (#110) Nov 3, 2017
CONTRIBUTING.md warcbase core moves to aut Jul 5, 2017
LICENSE warcbase core moves to aut Jul 5, 2017
LICENSE_HEADER.txt Update license headers for #208. (#290) Nov 22, 2018
README.md [skip travis] Update README (#284) Oct 17, 2018
pom.xml Add office document binary extraction. (#346) Aug 16, 2019

README.md

The Archives Unleashed Toolkit

Build Status codecov Maven Central Javadoc Scaladoc LICENSE Contribution Guidelines

The Archives Unleashed Toolkit is an open-source toolkit for analyzing web archives built around Apache Spark. This toolkit is part of the Archives Unleashed Project.

The toolkit grew out of a previous project called Warcbase. The following article provides a nice overview, much of which is still relevant:

Getting Started

Easy

If you have Apache Spark ready to go, it's as easy as:

$ spark-shell --packages "io.archivesunleashed:aut:0.17.0"

A little less easy

You can download the latest release here and include it like so:

$ spark-shell --jars /path/to/aut-0.17.0-fatjar.jar"

Even less easy

Build it yourself as per the instructions below:

Clone the repo:

$ git clone http://github.com/archivesunleashed/aut.git

You can then build The Archives Unleashed Toolkit.

$ mvn clean install

For the impatient, to skip tests:

$ mvn clean install -DskipTests

I want to use Docker!

Ok! Take a quick spin with aut with Docker.

Documentation! Or, how do I use this?

Once built or downloaded, you can follow the basic set of recipes and tutorials here.

License

Licensed under the Apache License, Version 2.0.

Acknowledgments

This work is primarily supported by the Andrew W. Mellon Foundation. Additional funding for the Toolkit has come from the U.S. National Science Foundation, Columbia University Library's Mellon-funded Web Archiving Incentive Award, the Natural Sciences and Engineering Research Council of Canada, the Social Sciences and Humanities Research Council of Canada, and the Ontario Ministry of Research and Innovation's Early Researcher Award program. Any opinions, findings, and conclusions or recommendations expressed are those of the researchers and do not necessarily reflect the views of the sponsors.

You can’t perform that action at this time.