Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.
Sign upAdd imagegraph, and webgraph to command line app. #432
+422
−207
Conversation
This comment has been minimized.
This comment has been minimized.
codecov
bot
commented
Apr 6, 2020
•
Codecov Report
@@ Coverage Diff @@
## master #432 +/- ##
==========================================
+ Coverage 77.70% 77.99% +0.28%
==========================================
Files 41 43 +2
Lines 1534 1554 +20
Branches 282 286 +4
==========================================
+ Hits 1192 1212 +20
Misses 217 217
Partials 125 125 |
This comment has been minimized.
This comment has been minimized.
|
#433 has been created ( |
|
Tested locally on sample data and works like a charm (I borrowed the log4.properties file from the other PR). I suspect it generates GEXF as I think the work on this pre-dated our shift from GEXF to graphml as the main graph output. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
ruebot commentedApr 6, 2020
•
edited
GitHub issue(s): #431
What does this Pull Request do?
Add
imagegraph, andwebgraphto command line app.- Resolves #431
- Adds webpages, and imagegraph to command line app
- Adds tests for new functionality
- Clean-up doc comments
- Convert files with dos line endings to unix line endings
How should this be tested?
ImageGraphExtractorexample:bin/spark-submit --master local\[8\] --files /home/nruest/Projects/au/sample-data/log4j.properties --conf spark.driver.extraJavaOptions='-Dlog4j.configuration=file:/home/nruest/Projects/au/sample-data/log4j.properties' --class io.archivesunleashed.app.CommandLineAppRunner /home/nruest/Projects/au/aut/target/aut-0.50.1-SNAPSHOT-fatjar.jar --extractor ImageGraphExtractor --input /home/nruest/Projects/au/sample-data/geocities/* --output /home/nruest/Projects/au/sample-data/app-output/ImageGraphExtractor --dfWebPagesExtractorexample:bin/spark-submit --master local\[8\] --files /home/nruest/Projects/au/sample-data/log4j.properties --conf spark.driver.extraJavaOptions='-Dlog4j.configuration=file:/home/nruest/Projects/au/sample-data/log4j.properties' --class io.archivesunleashed.app.CommandLineAppRunner /home/nruest/Projects/au/aut/target/aut-0.50.1-SNAPSHOT-fatjar.jar --extractor WebPagesExtractor --input /home/nruest/Projects/au/sample-data/geocities/* --output /home/nruest/Projects/au/sample-data/app-output/WebPagesExtractor --dfAdditional Notes:
log4jconfig file required. I'll create a ticket for that, and work on a solution separately.DomainGraphExtractorwrites asGEXF. Anybody remember why this using GEXF? Shouldn’t it be graphml? https://github.com/archivesunleashed/aut/blob/master/src/main/scala/io/archivesunleashed/app/CommandLineApp.scala#L115-L123webgraphanddomainsbecause there is a bit of duplication withDomainFrequencyExtractorandDomainGraphExtractor, and there could also be some confusion here how the extractors are labeled.