Skip to content

/ aut

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement Python versions of Serializable APIs #410

Closed
ruebot opened this issue Jan 17, 2020 · 4 comments
Closed

Implement Python versions of Serializable APIs #410

ruebot opened this issue Jan 17, 2020 · 4 comments
Assignees
@ruebot
Labels
DataFrames PySpark Python documentation feature
Projects
DataFrames and PySpark

Comments

@ruebot
Copy link
Member

@ruebot ruebot commented Jan 17, 2020 •
edited by SinghGursimran

RDD Scala DF Python DF
keepValidPages keepValidPagesDF keepValidPagesDF
keepMimeTypes keepMimeTypesDF keepMimeTypesDF
keepMimeTypesTika keepMimeTypesTikaDF
keepHttpStatus keepHttpStatusDF
keepDate keepDateDF discardDateDF
keepUrls keepUrlsDF keepUrlsDF
keepUrlPatterns keepUrlPatternsDF keepUrlPatternsDF
keepImages keepImagesDF
keepDomains keepDomainsDF
keepLanguages keepLanguagesDF
keepContent keepContentDF keepContentDF
discardMimeTypes discardMimeTypesDF DiscardMimeTypesDF
discardMimeTypesTika discardMimeTypesTIkaDF
discardDate discardDateDF
discardUrls discardUrlsDF discardUrlsDF
discardHttpStatus discardHttpStatusDF
discardUrlPatterns discardUrlPatternsDF discardUrlPatternsDF
discardDomains discardDomainsDF
discardContent discardContentDF discardContentDF
discardLanguages discardLanguagesDF

Stealing @SinghGursimran's very helpful tables here 😃
@ruebot ruebot mentioned this issue Jan 17, 2020
Closed
@SinghGursimran
Copy link
Collaborator

@SinghGursimran SinghGursimran commented Feb 1, 2020 •
edited

https://gist.github.com/SinghGursimran/2e97126a966b4a5c7f4704e61f0eec82

I have added code for serializable APIs here. Still, few are left, will add that soon. With spark 2.4.4, it works in the terminal but not in the jupyter notebook (some issues with connection timeout for spark 2.4.4). With spark 2.3, it will work in both.

When the above issue is resolved, I will add the code to the project.
@ruebot ruebot added this to ToDo in DataFrames and PySpark Feb 5, 2020
ruebot added a commit that referenced this issue May 19, 2020
@ruebot
Implement Scala Matchbox UDFs in Python.
Verified
This commit was signed with a verified signature.
ruebot Nick Ruest
GPG key ID: 417FAF1A0E1080CD Learn about signing commits
Loading status checks…
9dc3ad7 
- Resolves #408
- Alphabetizes DataFrameloader functions
- Alphabetizes UDFs functions
- Move DataFrameLoader to df packages
- Move UDFs out of df into their own package
- Rename UDFs (no more DF tagged to the end).
- Update tests as necessary
- Partially addresses #410, #409
- Supersedes #412.
@ruebot ruebot mentioned this issue May 19, 2020
Merged
@ruebot
Copy link
Member Author

@ruebot ruebot commented May 19, 2020

I'm proposing that these should not be implemented as originally intended in #463, and should just implemented via the normal -- .filter(col("somecolumn").isin(somearray)) -- method.
@ianmilligan1
Copy link
Member

@ianmilligan1 ianmilligan1 commented May 19, 2020

As noted in #463, your above has a 👍 from me @ruebot.
@ruebot
Copy link
Member Author

@ruebot ruebot commented May 19, 2020

Cool. We'll mark this closed once we get all the documentation updated.
ianmilligan1 pushed a commit that referenced this issue May 19, 2020
@ruebot
Implement Scala Matchbox UDFs in Python. (#463)
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.
GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits
Loading status checks…
69007e2 
- Resolves #408
- Alphabetizes DataFrameloader functions
- Alphabetizes UDFs functions
- Move DataFrameLoader to df packages
- Move UDFs out of df into their own package
- Rename UDFs (no more DF tagged to the end).
- Update tests as necessary
- Partially addresses #410, #409
- Supersedes #412.
@ruebot ruebot moved this from ToDo to In Progress in DataFrames and PySpark May 19, 2020
@ruebot ruebot self-assigned this May 19, 2020
ruebot added a commit to archivesunleashed/aut-docs that referenced this issue May 20, 2020
@ruebot
Add documentation for Python filter examples.
Verified
This commit was signed with a verified signature.
ruebot Nick Ruest
GPG key ID: 417FAF1A0E1080CD Learn about signing commits
Loading status checks…
d9e307e 
- Resolve all TODO for Python that required a `.filter()`
- Linked some TODO open issues
- Resolves archivesunleashed/aut#410
ruebot added a commit to archivesunleashed/notebooks that referenced this issue May 20, 2020
@ruebot
Add PySpark documentation examples for archivesunleashed/aut#410
Verified
This commit was signed with a verified signature.
ruebot Nick Ruest
GPG key ID: 417FAF1A0E1080CD Learn about signing commits
5944105
@ruebot ruebot mentioned this issue May 20, 2020
Merged
DataFrames and PySpark automation moved this from In Progress to In review May 20, 2020
ianmilligan1 pushed a commit to archivesunleashed/aut-docs that referenced this issue May 20, 2020
@ruebot
Add documentation for Python filter examples. (#63)
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.
GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits
Loading status checks…
b6bff40 
- Resolve all TODO for Python that required a `.filter()`
- Linked some TODO open issues
- Resolves archivesunleashed/aut#410
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ruebot ruebot

Labels
DataFrames PySpark Python documentation feature
Projects
DataFrames and PySpark
  
In review
Milestone
No milestone
Linked pull requests

Successfully merging a pull request may close this issue.

Add documentation for Python filter examples.
3 participants
@ruebot @ianmilligan1 @SinghGursimran
You can’t perform that action at this time.