As @ruebot mentioned on Slack, running AUT with --packages produced error messages, e.g.:

19/08/14 19:35:28 ERROR SparkContext: Failed to add file:/root/.ivy2/jars/com.github.archivesunleashed.tika_tika-parsers-1.22.jar to Spark environment
java.io.FileNotFoundException: Jar /root/.ivy2/jars/com.github.archivesunleashed.tika_tika-parsers-1.22.jar not found
    at org.apache.spark.SparkContext.addJarFile$1(SparkContext.scala:1838)
...

This is because ivy isn't good at finding dependencies specified with a classifier.

Since the classifier wasn't doing any useful work I removed it from our fork of tika and pushed a new release. This PR updates our POM accordingly.

How should this be tested?

Something like this:

jrwiebe@tuna:~/aut$ rm -rf ~/.m2/repository/* && mvn clean install && rm -rf ~/.ivy2/* && time ~/spark-2.4.3-bin-hadoop2.7/bin/spark-shell --packages io.archivesunleashed:aut:0.17.1-SNAPSHOT

There should be no errors.

I tested by running this code:

import io.archivesunleashed._
import io.archivesunleashed.df._

val warc_path = "/home/jrwiebe/warcs/*.gz"

val df_pdf = RecordLoader.loadArchives(warc_path, sc).extractPDFDetailsDF();
val res_pdf = df_pdf.select($"bytes", $"extension").saveToDisk("bytes", "/home/jrwiebe/test/pdf", "extension")