More Data Frame Implementations + Code Refactoring #383

SinghGursimran · 2019-11-20T20:22:31Z

Added more Data Frame Implementations along with some Code Refactoring

For Testing:

import io.archivesunleashed._
import io.archivesunleashed.df._

RecordLoader.loadArchives("./ARCHIVEIT-227-QUARTERLY-XUGECV-20091218231727-00039-crawling06.us.archive.org-8091.warc.gz",sc)
			.keepImages()
			.all()
			.select(ComputeImageSizeDF($"bytes").as("size"), ComputeMD5DF($"bytes").as("md5"), ComputeSHA1DF($"bytes").as("sha1"))
			.show(20,false)


        Issue-368


        Issue238


        Issue238


        Merge branch 'master' of https://github.com/SinghGursimran/aut


        test


        Merge branch 'master' of https://github.com/archivesunleashed/aut


        Merge branch 'master' of https://github.com/archivesunleashed/aut


        Merge branch 'master' of https://github.com/archivesunleashed/aut


        code_Refactoring

codecov · 2019-11-20T20:41:04Z

Codecov Report

Merging #383 into master will increase coverage by 0.24%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master     #383      +/-   ##
==========================================
+ Coverage   76.23%   76.47%   +0.24%     
==========================================
  Files          40       40              
  Lines        1422     1437      +15     
  Branches      268      268              
==========================================
+ Hits         1084     1099      +15     
  Misses        221      221              
  Partials      117      117

ruebot · 2019-11-20T20:52:16Z

Interesting! I like it. Can you add a test?


        tests

ruebot

Couple small changes. Overall great!

Let's see what @lintool things about the name, and we might as well get this added to the Python side of things as well, so it works with PySpark: https://github.com/archivesunleashed/aut/blob/e2ec5a17502709de08c191fbf4783a3f6f0e8199/src/main/python/aut/common.py

I think you should see the implementation pattern there. But, if not, get a hold of me in Slack, and I'm happy to help out.

ruebot · 2019-11-20T22:44:34Z

src/main/scala/io/archivesunleashed/DataFrameLoader.scala

@@ -27,6 +27,12 @@ class DataFrameLoader(sc: SparkContext) {
      .pages()
  }

+  def pagesWithBytes(path: String): DataFrame = {


Let's get a doc comment for this method. You can just crib from the methods around it for the pattern.

ruebot · 2019-11-20T22:44:34Z

src/main/scala/io/archivesunleashed/package.scala

@@ -115,6 +115,24 @@ package object archivesunleashed {
      sqlContext.getOrCreate().createDataFrame(records, schema)
    }

+    /*Creates a column for Bytes as well in Dataframe. 
+      Call KeepImages OR KeepValidPages on RDD depending upon the requirement before calling this method */
+    def pagesWithBytes(): DataFrame = {


@lintool you like naming things, you good with this one, or you got something better? 😃

@SinghGursimran as per slack conversation, let's call this one all.

I can take care of renaming pages to html tomorrow or later tonight under the cover of a separate PR.

ruebot · 2019-11-20T22:44:34Z

src/main/scala/io/archivesunleashed/package.scala

@@ -30,7 +30,7 @@ import io.archivesunleashed.matchbox.ExtractDate.DateComponent.DateComponent
 import java.net.URI
 import java.net.URL
 import org.apache.spark.sql.{DataFrame, Row, SparkSession}
-import org.apache.spark.sql.types.{IntegerType, StringType, StructField, StructType}
+import org.apache.spark.sql.types.{IntegerType, StringType, BinaryType, StructField, StructType}


Let's get this in alphabetical order so Scalastyle is happy. Pedantic, sorry.

lintool · 2019-11-20T23:12:26Z

With DF, we shouldn't need pagesWithBytes, we should be okay with the widest possible table (as many fields as we need) in a single method. Spark is supposed to do the optimization behind the scenes, i.e., if you don't select it, Spark should be smart to project away it early on.

Worth testing just to make sure.

SinghGursimran · 2019-11-20T23:21:51Z

@lintool Initially, I added this only for images. I had to keep images for analysis. Instead of adding it specific to images, I created a general method considering "bytes" might be required in the future for non-image analysis as well.
Previous "pages" method filters out only valid pages without keeping images.
Plus, that "pages" method is being used in many use cases where we actually require only valid pages. So, I thought of not changing it.
If you want, I would change this "pagesWithBytes" to be specific for images.

ruebot

Updates from Slack convo

ruebot · 2019-11-21T00:35:43Z

src/main/scala/io/archivesunleashed/package.scala

@@ -115,6 +115,24 @@ package object archivesunleashed {
      sqlContext.getOrCreate().createDataFrame(records, schema)
    }

+    /*Creates a column for Bytes as well in Dataframe. 
+      Call KeepImages OR KeepValidPages on RDD depending upon the requirement before calling this method */
+    def pagesWithBytes(): DataFrame = {


@SinghGursimran as per slack conversation, let's call this one all.

I can take care of renaming pages to html tomorrow or later tonight under the cover of a separate PR.


        updates


        Merge branch 'master' into codeRefactoring


        python-all


        pyhton-changes

g285sing added 9 commits Nov 7, 2019

Issue-368

Loading status checks…

68922a2

Issue238

0db0093

Issue238

5817bf5

Merge branch 'master' of https://github.com/SinghGursimran/aut

Loading status checks…

d65cb41

test

Loading status checks…

4e5a066

Merge branch 'master' of https://github.com/archivesunleashed/aut

c60104f

Merge branch 'master' of https://github.com/archivesunleashed/aut

f18006e

Merge branch 'master' of https://github.com/archivesunleashed/aut

9141e2f

code_Refactoring

Loading status checks…

2aed5a0

tests

Loading status checks…

e2ec5a1

ruebot requested changes Nov 20, 2019

View changes

ruebot requested changes Nov 21, 2019

View changes

updates

Loading status checks…

3593561

ruebot mentioned this pull request Nov 21, 2019

Document new `.all()` method. #26

Open

ruebot approved these changes Nov 21, 2019

View changes

ruebot and others added 3 commits Nov 21, 2019

Merge branch 'master' into codeRefactoring

Verified

This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.

GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits

Loading status checks…

9761e71

python-all

7c1485f

pyhton-changes

Loading status checks…

0897f7c

ruebot merged commit c4eaca9 into archivesunleashed:master Nov 21, 2019
1 check was pending

1 check was pending

continuous-integration/travis-ci/pr The Travis CI build is in progress
Details

Please note that GitHub no longer supports your web browser.

archivesunleashed/aut

More Data Frame Implementations + Code Refactoring #383

More Data Frame Implementations + Code Refactoring #383

SinghGursimran commented Nov 20, 2019 •

edited

This comment has been minimized.

codecov bot commented Nov 20, 2019 •

edited

This comment has been minimized.

ruebot commented Nov 20, 2019

ruebot left a comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

lintool commented Nov 20, 2019

This comment has been minimized.

SinghGursimran commented Nov 20, 2019 •

edited

ruebot left a comment

This comment has been minimized.

Please note that GitHub no longer supports your web browser.

archivesunleashed/aut

Join GitHub today

More Data Frame Implementations + Code Refactoring #383

Conversation

SinghGursimran commented Nov 20, 2019 • edited

This comment has been minimized.

codecov bot commented Nov 20, 2019 • edited

Codecov Report

This comment has been minimized.

ruebot commented Nov 20, 2019

ruebot left a comment

This comment has been minimized.

ruebot Nov 20, 2019

This comment has been minimized.

ruebot Nov 20, 2019

This comment has been minimized.

ruebot Nov 21, 2019

This comment has been minimized.

ruebot Nov 20, 2019

This comment has been minimized.

lintool commented Nov 20, 2019

This comment has been minimized.

SinghGursimran commented Nov 20, 2019 • edited

ruebot left a comment

This comment has been minimized.

ruebot Nov 21, 2019

SinghGursimran commented Nov 20, 2019 •

edited

codecov bot commented Nov 20, 2019 •

edited

SinghGursimran commented Nov 20, 2019 •

edited