I noticed when I was filtering for URLs ending in ".doc" that I was getting a lot of non-doc formats (HTML and text formats). I think it's less likely there will be such incorrect file extensions for the other binary formats we're targeting, but if we want a generic algorithm for determining the extension, I'd cover the .doc case by switching steps 1 and 2. I'd also use the plural method getExtensions and if the extension returned by FilenameUtils is contained in this list, I'd select that one.

We're already getting the MimeType with Tika, which is the only expensive operation in this process.

My method:

  def getExt(mimeType: String, url: String): String = {
    val tikaExtensions = DetectMimeTypeTika.getExtensions(mimetype)
    var ext = "unknown"
    // Tika method
    if (tikaExtensions.size == 1) {
      ext = tikaExtensions(0).substring(1)
    } else {
      // FilenameUtils method
      val fnuExt = FilenameUtils.getExtension(url)
      if (fnuExt != null && !fnuExt.isEmpty) {
        // Reconcile Tika list and FilenameUtils extension
        if (tikaExtensions.size > 1) {
          if (tikaExtensions.contains(fnuExt)) {
            ext = fnuExt
          } else {
            ext = tikaExtensions(0).substring(1)
          }
        } else { // tikaExtensions.size == 0 && fnuExt exists
          ext = fnuExt
        }
      } // else => unknown
    }
    ext
  }

I have an idea since I noticed these methods as I'm hacking on #346

https://github.com/archivesunleashed/aut/blob/master/src/main/scala/io/archivesunleashed/package.scala#L307-L313

https://github.com/archivesunleashed/aut/blob/master/src/main/scala/io/archivesunleashed/package.scala#L374-L380

Use what you have above, and combine it there. Maybe do something consistent with what we have with getMimeType where it uses web server Mime Type, and have keepTikaMimeTypes and discardTikaMimeTypes. It could clean-up a whole lot of what we have put in there the last couple of days.

Use what you have above, and combine it there.

Not sure what you mean by this line, unless you're just saying to put the method I wrote above in that section of package.scala. (Which I was about to do.)

@jrwiebe yep! that plus potentially mimicking those two exiting functions as well. That make sense?

Actually, now that I'm trying it I realize we don't want a getExtension method that applies to RDDs. I'm putting it in the matchbox.

Did it.

@ruebot Want to integrate this into PR #346? Or I could make a separate one after that's merged.

Let's do a separate one after #346. I want to make sure it comes in under your name 😃