Skip to content

/ aut

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Discussion: Restyle UDFs in the context of DataFrames #425

Open
lintool opened this issue Feb 11, 2020 · 3 comments
Open

Discussion: Restyle UDFs in the context of DataFrames #425

lintool opened this issue Feb 11, 2020 · 3 comments
Labels
DataFrames Scala enhancement rdd
Projects
DataFrames and PySpark

Comments

@lintool
Copy link
Member

@lintool lintool commented Feb 11, 2020

Currently, we're doing something like this in DFs:

RecordLoader.loadArchives("./src/test/resources/arc/example.arc.gz",sc)
			.webgraph()
			.select($"src")
			.keepUrlPatternsDF(Set(".*index.*".r))
			.show(10,false)

This is a straightforward translation of what we've been doing in RDDs, so that's fine. However, in DF, something like this would be more fluent:

			.filter($"src".isInUrlPatterns(Set(".*index.*".r)))

This would require reimplementation of our all filters... let's discuss.
@ruebot

This comment has been minimized.

Sign in to view
Copy link
Member

@ruebot ruebot commented Feb 11, 2020

Pulling this in from Slack:

Looking at all the RDD filters, they're all basically the same implementation; there's a field, do this custom filter on it. So, a DF and RDD re-implementation could be very similar. Basically what you proposed, the filter UDF taking in two parameters. So, we could do something like this for both RDD and DF:

.filter($"col".isInUrlPatterns(Set(".*index.*".r)))

...and, if we play our cards right, we could just have one implementation for both 🤷‍♂
@lintool

This comment has been minimized.

Sign in to view
Copy link
Member Author

@lintool lintool commented Feb 11, 2020

we could just have one implementation for both

That would be great in the short term, but not necessary for the long term, IMO. Eventually, the DF functionality would be a superset of the RDD functionality, since we have no intention of backporting new DF features to RDD.
@ruebot ruebot mentioned this issue Feb 11, 2020
Open
@greebie

This comment has been minimized.

Sign in to view
Copy link
Contributor

@greebie greebie commented Feb 11, 2020

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
DataFrames Scala enhancement rdd
Projects
DataFrames and PySpark
  
ToDo
Milestone
No milestone
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants
@lintool @ruebot @greebie
You can’t perform that action at this time.