Skip to content
Please note that GitHub no longer supports your web browser.

We recommend upgrading to the latest Google Chrome or Firefox.

Learn more
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ComputeSHA1 method; resolves #363. #364

Merged
merged 1 commit into from Oct 9, 2019

Conversation

@ruebot
Copy link
Member

commented Oct 8, 2019

GitHub issue(s): #363

What does this Pull Request do?

Add ComputeSHA1 method.

How should this be tested?

  • Unit tests, and TravisCI should take care of it.

Additional Notes:

Do we want to add this method to all our DataFrame methods? Basically add a sha1 column, along with the md5 column? If so, something like this here:

    /* Extract PDF bytes and PDF metadata. */
    def extractPDFDetailsDF(): DataFrame = {
      val records = rdd
        .map(r =>
            (r, (DetectMimeTypeTika(r.getBinaryBytes)))
            )
        .filter(r => r._2 == "application/pdf")
        .map(r => {
          val bytes = r._1.getBinaryBytes
          val md5Hash = new String(Hex.encodeHex(MessageDigest.getInstance("MD5").digest(bytes)))
          val sha1Hash = new String(Hex.encodeHex(MessageDigest.getInstance("MD5").digest(bytes)))
          val encodedBytes = Base64.getEncoder.encodeToString(bytes)
          val url = new URL(r._1.getUrl)
          val filename = FilenameUtils.getName(url.getPath())
          val extension = GetExtensionMime(url.getPath(), r._2)
          (r._1.getUrl, filename, extension, r._1.getMimeType,
            DetectMimeTypeTika(r._1.getBinaryBytes), md5Hash, sha1Hash, encodedBytes)
        })
        .map(t => Row(t._1, t._2, t._3, t._4, t._5, t._6, t._7, t._8))

      val schema = new StructType()
        .add(StructField("url", StringType, true))
        .add(StructField("filename", StringType, true))
        .add(StructField("extension", StringType, true))
        .add(StructField("mime_type_web_server", StringType, true))
        .add(StructField("mime_type_tika", StringType, true))
        .add(StructField("md5", StringType, true))
        .add(StructField("sha1", StringType, true))
        .add(StructField("bytes", StringType, true))

      val sqlContext = SparkSession.builder();
      sqlContext.getOrCreate().createDataFrame(records, schema)
    }
@ruebot ruebot requested review from lintool and ianmilligan1 Oct 8, 2019
@codecov

This comment has been minimized.

Copy link

commented Oct 8, 2019

Codecov Report

Merging #364 into master will increase coverage by 0.3%.
The diff coverage is 100%.

@@            Coverage Diff            @@
##           master     #364     +/-   ##
=========================================
+ Coverage   75.93%   76.24%   +0.3%     
=========================================
  Files          39       40      +1     
  Lines        1392     1410     +18     
  Branches      267      267             
=========================================
+ Hits         1057     1075     +18     
  Misses        218      218             
  Partials      117      117
Copy link
Member

left a comment

Builds nicely locally.

Re:

Do we want to add this method to all our DataFrame methods? Basically add a sha1 column, along with the md5 column? If so, something like this here:

That makes sense to me?

- Update tests where needed
- Add SHA1 method to ExtractImageDetails
- Add SHA1 to DataFrames binary extraction and analysis
@ruebot ruebot force-pushed the issue-363 branch from ffda0bb to 21f9434 Oct 8, 2019
@ruebot

This comment has been minimized.

Copy link
Member Author

commented Oct 8, 2019

I'll update the bleeding edge documentation once we're good to go here.

Copy link
Member

left a comment

New commit looks great! Runs nicely with this script:

import io.archivesunleashed._
import io.archivesunleashed.df._

val df = RecordLoader.loadArchives("example.arc.gz", sc).extractImageDetailsDF();
df.select($"url", $"filename", $"extension", $"mime_type_web_server", $"mime_type_tika", $"width", $"height", $"md5", $"sha1", $"bytes").orderBy(desc("md5")).show()

Screen Shot 2019-10-09 at 12 44 49 PM

@ianmilligan1 ianmilligan1 merged commit 03ac99c into master Oct 9, 2019
3 checks passed
3 checks passed
codecov/patch 100% of diff hit (target 75.93%)
Details
codecov/project 76.24% (+0.3%) compared to 9b3e025
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@ianmilligan1 ianmilligan1 deleted the issue-363 branch Oct 9, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.