Skip to content

/aut

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add binary extraction DataFrames to PySpark. #350

Open
wants to merge 2 commits into
base: master
from
+77 −4

Conversation

@ruebot
Copy link
Member

commented Aug 20, 2019

GitHub issue(s):

What does this Pull Request do?

Add binary extraction DataFrames to PySpark.

How should this be tested?

  • TravisCI
  • Fire up a Jupyter Notebook:
$ PYSPARK_DRIVER_PYTHON=jupyter PYSPARK_DRIVER_PYTHON_OPTS=notebook ~/bin/spark-2.4.3-bin-hadoop2.7/bin/pyspark --jars /home/nruest/Projects/au/aut/target/aut-0.17.1-SNAPSHOT-fatjar.jar --driver-class-path /home/nruest/Projects/au/aut/target/aut-0.17.1-SNAPSHOT-fatjar.jar --py-files /home/nruest/Projects/au/aut/target/aut.zip

Then do this for each of the additions, and make sure it works:

from aut import *

archive = WebArchive(sc, sqlContext, "/home/nruest/Projects/au/aut/src/test/resources/warc/")

df = archive.spreadsheets()
df.printSchema()

Screenshot from 2019-08-20 13-25-51

Additional Notes:

Let's see what happens with the test coverage. I assume I'm going to have to add some things to src/test/scala/io/archivesunleashed/df/DataFrameLoaderTest.scala
@ruebot
Add binary extration DataFrames to PySpark. 
- Address #190
- Address #259
- Address #302
- Address #303
- Address #304
- Address #305
- Address #306
- Address #307
Verified
This commit was signed with a verified signature.
ruebot Nick Ruest
GPG key ID: 417FAF1A0E1080CD Learn about signing commits
Loading status checks…
1176fd5

@ruebot ruebot requested a review from ianmilligan1 Aug 20, 2019

@codecov

This comment has been minimized.

Sign in to view
Copy link

commented Aug 20, 2019

Codecov Report

Merging #350 into master will decrease coverage by 0.76%.
The diff coverage is 0%.

@@            Coverage Diff             @@
##           master     #350      +/-   ##
==========================================
- Coverage   75.52%   74.76%   -0.77%     
==========================================
  Files          39       39              
  Lines        1373     1387      +14     
  Branches      265      265              
==========================================
  Hits         1037     1037              
- Misses        220      234      +14     
  Partials      116      116
@ruebot

This comment has been minimized.

Sign in to view
Copy link
Member Author

commented Aug 20, 2019

Here's an even better test: https://github.com/archivesunleashed/aut/wiki/Using-AUT-with-PySpark

(Just swap out 0.18.0 with the path to aut-0.17.1-SNAPSHOT-fatjar.jar and aut.zip)
@ruebot
README updates
Verified
This commit was signed with a verified signature.
ruebot Nick Ruest
GPG key ID: 417FAF1A0E1080CD Learn about signing commits
Loading status checks…
88ab028
@ruebot ruebot referenced this pull request Aug 21, 2019
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ianmilligan1 ianmilligan1

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
1 participant
@ruebot
You can’t perform that action at this time.