Skip to content

/aut

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use Tika's detected MIME type instead of ArchiveRecord getMimeType? #342

Closed
jrwiebe opened this issue Aug 13, 2019 · 9 comments

Comments

@jrwiebe
Copy link
Contributor

commented Aug 13, 2019

I wasn't sure whether to add this as a comment in #306 (or one of the other extraction method threads), or to make it an issue.

I observed that while we're using DetectMimeTypeTika in methods like extractPDFDetailsDF, extractVideoDetailsDF, etc., we're using ArchiveRecord.getMimeType for reporting the MIME type. I think Tika's MIME type will tend to be more accurate than what is set by the web server. Do we want to go for accurate meta-description of the contents of our archive records, or do we want fidelity to what was collected from the web?

If we were wanted to use what we're getting from Tika, a method like extractAudioDetailsDF() could look something like this:

def extractAudioDetailsDF(rdd: RDD[ArchiveRecord]): DataFrame = {
    val records = rdd
    .map(r => 
        (r, DetectMimeTypeTika(r.getContentString)) // <------- (record, mime_type) tuple
    )
    .filter(r => r._2.startsWith("audio/")) // for example
    .map(r => {
      val bytes = r._1.getBinaryBytes
      val hash = new String(Hex.encodeHex(MessageDigest.getInstance("MD5").digest(bytes)))
      val encodedBytes = Base64.getEncoder.encodeToString(bytes)
      (r._1.getUrl, r._2, hash, encodedBytes)
    })
    .map(t => Row(t._1, t._2, t._3, t._4))

    val schema = new StructType()
    .add(StructField("url", StringType, true))
    .add(StructField("mime_type", StringType, true))
    .add(StructField("md5", StringType, true))
    .add(StructField("encodedBytes", StringType, true))

    val sqlContext = SparkSession.builder();
    sqlContext.getOrCreate().createDataFrame(records, schema)
}

I'm not sure about the memory implications of the r => (r, DetectMimeTypeTika(r.getContentString)) mapping.

@ruebot ruebot added DataFrames enhancement Scala labels Aug 13, 2019

@ruebot

This comment has been minimized.

Sign in to view
Copy link
Member

commented Aug 13, 2019

I agree with moving to Tika for all the MimeType detection for identification, but I also think we should tweak the DF columns to be mime_type_tika and mime_type_web_server since there might be a use case for comparing the two. Fun research study about how awful web servers are at identifying MimeTypes?
@ruebot

This comment has been minimized.

Sign in to view
Copy link
Member

commented Aug 13, 2019

Interesting, if I switch all the ExtractMediaDetails from r.getMimeType to DetectMimeTypeTika(r.getContentString), for the columns, the tests fail because the identification seems to get worse:

Results :

Tests in error: 
  Image DF extraction(io.archivesunleashed.ExtractImageDetailsTest): "[image/gif]" did not equal "[application/octet-stream]"
  Audio DF extraction(io.archivesunleashed.ExtractAudioDetailsTest): "a[udio/mpeg]" did not equal "a[pplication/octet-stream]"
  Video DF extraction(io.archivesunleashed.ExtractVideoDetailsTest): "[video/mp4]" did not equal "[application/octet-stream]"

@ruebot ruebot referenced this issue Aug 13, 2019

Merged

Add Audio & Video binary extraction #341

ruebot added a commit that referenced this issue Aug 13, 2019

@ruebot
hacking on #342
Verified
This commit was signed with a verified signature.
ruebot Nick Ruest
GPG key ID: 417FAF1A0E1080CD Learn about signing commits
54c1643
@ruebot

This comment has been minimized.

Sign in to view
Copy link
Member

commented Aug 13, 2019

@jrwiebe this what you're thinking? 54c1643
@jrwiebe

This comment has been minimized.

Sign in to view
Copy link
Contributor Author

commented Aug 13, 2019

extractPDFDetailsDF is what I was thinking, but I see the the audio and video methods don't use the same approach (i.e., the map(r => (r, DetectMimeTypeTika(r.getBinaryBytes)))). Is that intentional?
@ruebot

This comment has been minimized.

Sign in to view
Copy link
Member

commented Aug 14, 2019

Yeah intentional. I just picked one out of the three to implement before I got a 👍 or 👎 from you 😄
@jrwiebe

This comment has been minimized.

Sign in to view
Copy link
Contributor Author

commented Aug 14, 2019

👍
@jrwiebe

This comment has been minimized.

Sign in to view
Copy link
Contributor Author

commented Aug 14, 2019

I wish I knew more about how Spark runs this code. I wrote it this way to avoid calling Tika twice, but it's very possible the return value is cached and read from cache later on.
@ruebot

This comment has been minimized.

Sign in to view
Copy link
Member

commented Aug 14, 2019

Cool, I'll update the code later on this morning or afternoon, and compare the output to the last job I ran on #341 testing.
@ruebot

This comment has been minimized.

Sign in to view
Copy link
Member

commented Aug 14, 2019

Same numbers. I'm going to do a time test now on HEAD on master, and on what I'll push up here in a second.

4809 audio files
644 PDF files
232 video files
5685 total

ruebot added a commit that referenced this issue Aug 14, 2019

@ruebot
Use Tika's detected MIME type instead of ArchiveRecord getMimeType. 
- Move audio, pdf, and video DF extraction to tuple map
- Provide to MimeType columns; mime_type_web_server and mime_type_tika
- Update tests
- Resolves #342
Verified
This commit was signed with a verified signature.
ruebot Nick Ruest
GPG key ID: 417FAF1A0E1080CD Learn about signing commits
Loading status checks…
7ff5db2

@ruebot ruebot referenced this issue Aug 14, 2019

Merged

Use Tika's detected MIME type instead of ArchiveRecord getMimeType. #344

ruebot added a commit that referenced this issue Aug 14, 2019

@ruebot
Use Tika's detected MIME type instead of ArchiveRecord getMimeType. 
- Move audio, pdf, and video DF extraction to tuple map
- Provide two MimeType columns; mime_type_web_server and mime_type_tika
- Update tests
- Resolves #342
Verified
This commit was signed with a verified signature.
ruebot Nick Ruest
GPG key ID: 417FAF1A0E1080CD Learn about signing commits
Loading status checks…
47612ce

@ianmilligan1 ianmilligan1 closed this in #344 Aug 14, 2019

ianmilligan1 added a commit that referenced this issue Aug 14, 2019

@ruebot @ianmilligan1
Use Tika's detected MIME type instead of ArchiveRecord getMimeType. (#… 
…344)

- Move audio, pdf, and video DF extraction to tuple map
- Provide two MimeType columns; mime_type_web_server and mime_type_tika
- Update tests
- Resolves #342
Loading status checks…
01d12b4
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
DataFrames Scala enhancement
Projects
None yet
Milestone
No milestone
2 participants
@jrwiebe @ruebot
You can’t perform that action at this time.