I wasn't sure whether to add this as a comment in #306 (or one of the other extraction method threads), or to make it an issue.

I observed that while we're using DetectMimeTypeTika in methods like extractPDFDetailsDF, extractVideoDetailsDF, etc., we're using ArchiveRecord.getMimeType for reporting the MIME type. I think Tika's MIME type will tend to be more accurate than what is set by the web server. Do we want to go for accurate meta-description of the contents of our archive records, or do we want fidelity to what was collected from the web?

If we were wanted to use what we're getting from Tika, a method like extractAudioDetailsDF() could look something like this:

def extractAudioDetailsDF(rdd: RDD[ArchiveRecord]): DataFrame = {
    val records = rdd
    .map(r => 
        (r, DetectMimeTypeTika(r.getContentString)) // <------- (record, mime_type) tuple
    )
    .filter(r => r._2.startsWith("audio/")) // for example
    .map(r => {
      val bytes = r._1.getBinaryBytes
      val hash = new String(Hex.encodeHex(MessageDigest.getInstance("MD5").digest(bytes)))
      val encodedBytes = Base64.getEncoder.encodeToString(bytes)
      (r._1.getUrl, r._2, hash, encodedBytes)
    })
    .map(t => Row(t._1, t._2, t._3, t._4))

    val schema = new StructType()
    .add(StructField("url", StringType, true))
    .add(StructField("mime_type", StringType, true))
    .add(StructField("md5", StringType, true))
    .add(StructField("encodedBytes", StringType, true))

    val sqlContext = SparkSession.builder();
    sqlContext.getOrCreate().createDataFrame(records, schema)
}

I'm not sure about the memory implications of the r => (r, DetectMimeTypeTika(r.getContentString)) mapping.