Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add option to save to Parquet for app. #454

Merged
merged 6 commits into from Apr 22, 2020
Merged

Add option to save to Parquet for app. #454

merged 6 commits into from Apr 22, 2020

Conversation

@ruebot
Copy link
Member

ruebot commented Apr 22, 2020

#453 needs to be merged first

GitHub issue(s): #448

What does this Pull Request do?

Add option to save to Parquet for app.

  • Resolves #448
  • Update test
  • Add CSV headers to coalesced CSV output

How should this be tested?

  • TravisCI

Something like:

bin/spark-submit --class io.archivesunleashed.app.CommandLineAppRunner /home/nruest/Projects/au/aut/target/aut-0.60.1-SNAPSHOT-fatjar.jar --extractor AudioInformationExtractor --input /home/nruest/Projects/au/sample-data/geocities/* --output /home/nruest/Projects/au/sample-data/452-test/audioParquet --output-format parquet
bin/spark-submit --class io.archivesunleashed.app.CommandLineAppRunner /home/nruest/Projects/au/aut/target/aut-0.60.1-SNAPSHOT-fatjar.jar --extractor AudioInformationExtractor --input /home/nruest/Projects/au/sample-data/geocities/* --output /home/nruest/Projects/au/sample-data/452-test/audioParquetSingle --output-format parquet --partition 1
bin/spark-submit --class io.archivesunleashed.app.CommandLineAppRunner /home/nruest/Projects/au/aut/target/aut-0.60.1-SNAPSHOT-fatjar.jar --extractor AudioInformationExtractor --input /home/nruest/Projects/au/sample-data/geocities/* --output /home/nruest/Projects/au/sample-data/452-test/audioCsv  --partition 1

Additional Notes:

We should probably loop back around and clean all this up at some point. It's a little messy.

- Resolves #452
- PlainTextExtractor runs RemoveHTML, and ExtractBoilerplate on
`content`
- Update test
@ruebot ruebot requested review from lintool and ianmilligan1 Apr 22, 2020
- Resolves #448
- Update test
- Add CSV headers to coalesced CSV output
@ruebot ruebot force-pushed the issue-448 branch from 60fd670 to 04bd881 Apr 22, 2020
@ruebot

This comment has been minimized.

Copy link
Member Author

ruebot commented Apr 22, 2020

I'll get a documentation PR ready to go once archivesunleashed/aut-docs#58 is merged.

@codecov

This comment has been minimized.

Copy link

codecov bot commented Apr 22, 2020

Codecov Report

Merging #454 into master will decrease coverage by 0.11%.
The diff coverage is 72.00%.

@@            Coverage Diff             @@
##           master     #454      +/-   ##
==========================================
- Coverage   76.70%   76.59%   -0.12%     
==========================================
  Files          49       49              
  Lines        1421     1457      +36     
  Branches      264      279      +15     
==========================================
+ Hits         1090     1116      +26     
+ Misses        215      211       -4     
- Partials      116      130      +14     
ruebot added 3 commits Apr 22, 2020
@ruebot

This comment has been minimized.

Copy link
Member Author

ruebot commented Apr 22, 2020

Forgot to mention that I tested loading up both version of Parquet output into a Juypter notebook, and everything went smoothly.

@ianmilligan1 ianmilligan1 merged commit aece1e3 into master Apr 22, 2020
1 of 3 checks passed
1 of 3 checks passed
codecov/patch 72.00% of diff hit (target 76.70%)
Details
codecov/project 76.59% (+-0.12%) compared to e91d01f
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@ianmilligan1 ianmilligan1 deleted the issue-448 branch Apr 22, 2020
ruebot added a commit to archivesunleashed/aut-docs that referenced this pull request Apr 22, 2020
ruebot added a commit to archivesunleashed/aut-docs that referenced this pull request Apr 22, 2020
#59)

* Documentation update for archivesunleashed/aut#454
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

2 participants
You can’t perform that action at this time.