Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ExtractPopularImages, WriteGEXF, and WriteGraphML to Python. #466

Merged
merged 3 commits into from May 27, 2020

Conversation

@ruebot
Copy link
Member

ruebot commented May 25, 2020

GitHub issue(s): #409

What does this Pull Request do?

Add ExtractPopularImages, WriteGEXF, and WriteGraphML to Python.

  • Partially addresses #409
  • Clean up formatting in app.py
  • Cleanup doc comments on the Scala side
  • TODO: Extract Entities

How should this be tested?

[nruest@bomba:/tmp]$ wc -l scala-example.gexf ~/Projects/au/sample-data/notebook-testing/test.gexf
  29186 scala-example.gexf
  29186 /home/nruest/Projects/au/sample-data/notebook-testing/test.gexf
  58372 total
[nruest@bomba:/tmp]$ wc -l scala-example.graphml ~/Projects/au/sample-data/notebook-testing/test.graphml
  33911 scala-example.graphml
  33911 /home/nruest/Projects/au/sample-data/notebook-testing/test.graphml
  67822 total
ruebot added 2 commits May 25, 2020
- Partially addresses #409
- Clean up formatting in app.py
- Cleanup doc comments on the Scala side
- TODO: Extract Entities
src/main/python/aut/app.py Outdated Show resolved Hide resolved
@ruebot ruebot requested review from ianmilligan1 and lintool May 25, 2020
@codecov
Copy link

codecov bot commented May 25, 2020

Codecov Report

Merging #466 into master will not change coverage.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##           master     #466   +/-   ##
=======================================
  Coverage   75.86%   75.86%           
=======================================
  Files          49       49           
  Lines        1442     1442           
  Branches      279      279           
=======================================
  Hits         1094     1094           
  Misses        218      218           
  Partials      130      130           
ruebot added a commit to archivesunleashed/aut-docs that referenced this pull request May 26, 2020
@ruebot
Copy link
Member Author

ruebot commented May 26, 2020

Here's the documentation update PR for when the time comes.

@ruebot ruebot marked this pull request as ready for review May 27, 2020
@ruebot
Copy link
Member Author

ruebot commented May 27, 2020

Last item here is Extract Entities. We only have an RDD implementation of that. I'd argue that NER is something down the chain folks can do with the derivatives, and we don't need to implement it in the Scala DataFrame or Python side.

...which means, we could go as far as completely removing all the NER functionality. Though, not 100% wedded to that idea.

@ruebot
Copy link
Member Author

ruebot commented May 27, 2020

Once we merge, I'm move those two notebooks over to the notebooks repo.

@ruebot ruebot added this to In review in DataFrames and PySpark May 27, 2020
@ruebot ruebot added this to In Progress in 1.0.0 Release of AUT May 27, 2020
Copy link
Member

ianmilligan1 left a comment

This all works perfectly - tested all three and the results are nice (loaded up in Gephi well too).

ruebot added a commit to archivesunleashed/notebooks that referenced this pull request May 27, 2020
@ruebot ruebot mentioned this pull request May 27, 2020
@ianmilligan1 ianmilligan1 merged commit 4322eb9 into master May 27, 2020
3 checks passed
3 checks passed
codecov/patch 100.00% of diff hit (target 75.86%)
Details
codecov/project 75.86% (+0.00%) compared to cfcbc48
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
1.0.0 Release of AUT automation moved this from In Progress to Done May 27, 2020
@ianmilligan1 ianmilligan1 deleted the issue-409 branch May 27, 2020
ianmilligan1 pushed a commit to archivesunleashed/aut-docs that referenced this pull request May 27, 2020
@ruebot ruebot moved this from In review to Done in DataFrames and PySpark May 27, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Linked issues

Successfully merging this pull request may close these issues.

None yet

2 participants
You can’t perform that action at this time.