update for 'src' column #424

SinghGursimran · 2020-02-11T04:37:03Z

update for 'src' column

For Testing:

import io.archivesunleashed._
import io.archivesunleashed.df._

RecordLoader.loadArchives("./src/test/resources/arc/example.arc.gz",sc)
			.webgraph()
			.select($"src")
			.keepUrlPatternsDF(Set(".*index.*".r))
			.show(10,false)

RecordLoader.loadArchives("./src/test/resources/arc/example.arc.gz",sc)
			.webgraph()
			.select($"src")
			.discardUrlPatternsDF(Set(".*images.*".r))
			.show(10,false)

RecordLoader.loadArchives("./src/test/resources/arc/example.arc.gz",sc)
			.webgraph()
			.select($"src")
			.keepUrlsDF(Set("http://www.archive.org/","http://www.archive.org/index.php"))
			.show(10,false)

RecordLoader.loadArchives("./src/test/resources/arc/example.arc.gz",sc)
			.webgraph()
			.select($"src")
			.discardUrlsDF(Set("http://www.archive.org/","http://www.archive.org/index.php"))
			.show(10,false)

RecordLoader.loadArchives("./src/test/resources/arc/example.arc.gz",sc)
			.imagegraph()
			.select($"src")
			.keepDomainsDF(Set("www.archive.org"))
			.show(10,false)

RecordLoader.loadArchives("./src/test/resources/arc/example.arc.gz",sc)
			.webgraph()
			.select($"src")
			.discardDomainsDF(Set("www.archive.org"))
			.show(10,false)


        src_update

codecov · 2020-02-11T04:53:58Z

Codecov Report

Merging #424 into master will not change coverage.
The diff coverage is 100%.

@@           Coverage Diff           @@
##           master     #424   +/-   ##
=======================================
  Coverage   78.15%   78.15%           
=======================================
  Files          41       41           
  Lines        1584     1584           
  Branches      299      299           
=======================================
  Hits         1238     1238           
  Misses        218      218           
  Partials      128      128

ruebot · 2020-02-11T12:30:47Z

@SinghGursimran nice! I did a hasColumn function for a similar solution in twut. Can we get a test update too?

@lintool @ianmilligan1 do either of you see a use case for filtering on dest or image_url? Or is src, and url good enough here? If we add dest or image_url, we'd probably need to change the implementation to pass the column name as well.

ruebot · 2020-02-11T14:09:53Z

@SinghGursimran based on the new related issue (#425) let's just worry about getting the tests updated here, and don't worry about dest and image_url for now since the implementation of #425 would resolve that.

I'm running the right now on the entire GeoCities dataset for other project, and everything appears to be running smoothly 🙌


        tests


        Merge branch 'master' into src_update

src_update

Loading status checks…

bcc2895

ruebot added this to In review in DataFrames and PySpark Feb 11, 2020

g285sing and others added 2 commits Feb 12, 2020

tests

Loading status checks…

c7687e8

Merge branch 'master' into src_update

Verified

This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.

GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits

Loading status checks…

67629d6

ruebot approved these changes Feb 12, 2020

View changes

archivesunleashed / aut

update for 'src' column #424

update for 'src' column #424

SinghGursimran commented Feb 11, 2020

This comment has been minimized.

codecov bot commented Feb 11, 2020 •

edited

This comment has been minimized.

ruebot commented Feb 11, 2020

This comment has been minimized.

ruebot commented Feb 11, 2020

archivesunleashed / aut

Join GitHub today

update for 'src' column #424

update for 'src' column #424

Conversation

SinghGursimran commented Feb 11, 2020

This comment has been minimized.

codecov bot commented Feb 11, 2020 • edited

Codecov Report

This comment has been minimized.

ruebot commented Feb 11, 2020

This comment has been minimized.

ruebot commented Feb 11, 2020

codecov bot commented Feb 11, 2020 •

edited