Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ComputeSHA1 method; resolves #363. #364

Open
wants to merge 1 commit into
base: master
from

Conversation

@ruebot
Copy link
Member

commented Oct 8, 2019

GitHub issue(s): #363

What does this Pull Request do?

Add ComputeSHA1 method.

How should this be tested?

  • Unit tests, and TravisCI should take care of it.

Additional Notes:

Do we want to add this method to all our DataFrame methods? Basically add a sha1 column, along with the md5 column? If so, something like this here:

    /* Extract PDF bytes and PDF metadata. */
    def extractPDFDetailsDF(): DataFrame = {
      val records = rdd
        .map(r =>
            (r, (DetectMimeTypeTika(r.getBinaryBytes)))
            )
        .filter(r => r._2 == "application/pdf")
        .map(r => {
          val bytes = r._1.getBinaryBytes
          val md5Hash = new String(Hex.encodeHex(MessageDigest.getInstance("MD5").digest(bytes)))
          val sha1Hash = new String(Hex.encodeHex(MessageDigest.getInstance("MD5").digest(bytes)))
          val encodedBytes = Base64.getEncoder.encodeToString(bytes)
          val url = new URL(r._1.getUrl)
          val filename = FilenameUtils.getName(url.getPath())
          val extension = GetExtensionMime(url.getPath(), r._2)
          (r._1.getUrl, filename, extension, r._1.getMimeType,
            DetectMimeTypeTika(r._1.getBinaryBytes), md5Hash, sha1Hash, encodedBytes)
        })
        .map(t => Row(t._1, t._2, t._3, t._4, t._5, t._6, t._7, t._8))

      val schema = new StructType()
        .add(StructField("url", StringType, true))
        .add(StructField("filename", StringType, true))
        .add(StructField("extension", StringType, true))
        .add(StructField("mime_type_web_server", StringType, true))
        .add(StructField("mime_type_tika", StringType, true))
        .add(StructField("md5", StringType, true))
        .add(StructField("sha1", StringType, true))
        .add(StructField("bytes", StringType, true))

      val sqlContext = SparkSession.builder();
      sqlContext.getOrCreate().createDataFrame(records, schema)
    }
@ruebot ruebot requested review from lintool and ianmilligan1 Oct 8, 2019
@codecov

This comment has been minimized.

Copy link

commented Oct 8, 2019

Codecov Report

Merging #364 into master will increase coverage by 0.3%.
The diff coverage is 100%.

@@            Coverage Diff            @@
##           master     #364     +/-   ##
=========================================
+ Coverage   75.93%   76.24%   +0.3%     
=========================================
  Files          39       40      +1     
  Lines        1392     1410     +18     
  Branches      267      267             
=========================================
+ Hits         1057     1075     +18     
  Misses        218      218             
  Partials      117      117
Copy link
Member

left a comment

Builds nicely locally.

Re:

Do we want to add this method to all our DataFrame methods? Basically add a sha1 column, along with the md5 column? If so, something like this here:

That makes sense to me?

- Update tests where needed
- Add SHA1 method to ExtractImageDetails
- Add SHA1 to DataFrames binary extraction and analysis
@ruebot ruebot force-pushed the issue-363 branch from ffda0bb to 21f9434 Oct 8, 2019
@ruebot

This comment has been minimized.

Copy link
Member Author

commented Oct 8, 2019

I'll update the bleeding edge documentation once we're good to go here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.