There is a conflict between Tika's dependency on commons-compress and the version that is included in the Spark distribution, which under normal parameters causes calls to DetectMimeTypeTika from spark-shell to fail with a java.lang.NoSuchMethodError. For example, running the following code:

import io.archivesunleashed._
import io.archivesunleashed.matchbox._

val r = RecordLoader.loadArchives("/path/to/example.warc.gz", sc)

r.keepValidPages().map(r => DetectMimeTypeTika(r.getContentString)).take(5)

results in:

2019-02-18 16:39:07 ERROR Executor:91 - Exception in task 0.0 in stage 0.0 (TID 0)
java.lang.NoSuchMethodError: org.apache.commons.compress.archivers.ArchiveStreamFactory.detect(Ljava/io/InputStream;)Ljava/lang/String;
	at org.apache.tika.parser.pkg.ZipContainerDetector.detectArchiveFormat(ZipContainerDetector.java:160)
...

This is because current Spark distributions include old versions of commons-compress, and our code depends on version >= 1.14, which introduced the ArchiveStreamFactory.detect() method. Spark shell's classpath, which includes its jars/ directory, takes precedence over our dependencies.

I initially resolved this by adding an exclusion of poi-ooxml to the tika-parsers dependency in our POM, since poi-ooxml is the module that requires the newer commons-compress. This won't do, however, since we will want that module for detection of Microsoft Office formats.

A better solution is to prepend the correct commons-compress JAR to the spark-shell classpath with the --driver-class-path argument (i.e., the spark.driver.extraClassPath property). E.g,

$ spark-shell --jars target/aut-0.17.1-SNAPSHOT-fatjar.jar --driver-memory 4G --driver-class-path /home/jrwiebe/.m2/repository/org/apache/commons/commons-compress/1.18/commons-compress-1.18.jar

Obviously this is cumbersome, but until Spark is distributed with Hadoop 3, which shades Hadoop's dependencies, I don't see a better way. I've done quite a bit of research on this topic, and I don't see any solutions.

My question: Is there a better way? Perhaps some Maven magic, @lintool? Or is this just something we need to document?