Skip to content
Please note that GitHub no longer supports your web browser.

We recommend upgrading to the latest Google Chrome or Firefox.

Learn more

/aut

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

More Serializable APIs for DataFrames #396

Merged
merged 17 commits into from Dec 29, 2019
Merged

More Serializable APIs for DataFrames #396

merged 17 commits into from Dec 29, 2019
+75 −2

Conversation

@SinghGursimran
Copy link
Contributor

SinghGursimran commented Dec 29, 2019

More Serializable APIs for DataFrames

#223

For Testing:

keepHttpStatusDF

import io.archivesunleashed._
import io.archivesunleashed.df._

RecordLoader.loadArchives("./src/test/resources/arc/example.arc.gz",sc)
			.all()
			.select($"url",$"crawl_date",$"HttpStatus")
			.keepHttpStatusDF(Set("200"))
			.show(10,false)

keepDateDF

import io.archivesunleashed._
import io.archivesunleashed.df._

RecordLoader.loadArchives("./src/test/resources/arc/example.arc.gz",sc)
			.webpages()
			.select($"url",$"crawl_date")
			.keepDateDF(List("04"),"MM")
			.show(10,false)

keepUrlsDF

import io.archivesunleashed._
import io.archivesunleashed.df._

RecordLoader.loadArchives("./src/test/resources/arc/example.arc.gz",sc)
			.all()
			.select($"url",$"crawl_date",$"HttpStatus")
			.keepUrlsDF(Set("http://www.archive.org/","http://www.archive.org/robots.txt"))
			.show(10,false)

keepDomainsDF

import io.archivesunleashed._
import io.archivesunleashed.df._

RecordLoader.loadArchives("./src/test/resources/arc/example.arc.gz",sc)
			.all()
			.select($"url",$"crawl_date",$"HttpStatus")
			.keepDomainsDF(Set("www.archive.org"))
			.show(10,false)
g285sing added 17 commits Nov 7, 2019
g285sing
Issue-368
Loading status checks…
68922a2
g285sing
Issue238
0db0093
g285sing
Issue238
5817bf5
g285sing
Merge branch 'master' of https://github.com/SinghGursimran/aut
Loading status checks…
d65cb41
g285sing
test
Loading status checks…
4e5a066
g285sing
Merge branch 'master' of https://github.com/archivesunleashed/aut
c60104f
g285sing
Merge branch 'master' of https://github.com/archivesunleashed/aut
f18006e
g285sing
Merge branch 'master' of https://github.com/archivesunleashed/aut
9141e2f
g285sing
Merge remote-tracking branch 'upstream/master'
5e1cf23
g285sing
serializable-api
Loading status checks…
8d204d0
g285sing
APIs-DF
1c73e09
g285sing
Merge branch 'master' of https://github.com/SinghGursimran/aut
b3bcbe0
g285sing
Merge branch 'master' of https://github.com/archivesunleashed/aut
5bde3e7
g285sing
df-apis
Loading status checks…
08c90ad
g285sing
review_changes
de889c1
g285sing
Merge branch 'master' of https://github.com/archivesunleashed/aut
937925a
g285sing
new_ser_apis
Loading status checks…
e6834e5
@codecov

This comment has been minimized.

Sign in to view
Copy link

codecov bot commented Dec 29, 2019

Codecov Report

Merging #396 into master will increase coverage by 0.07%.
The diff coverage is 88.88%.

@@            Coverage Diff             @@
##           master     #396      +/-   ##
==========================================
+ Coverage   77.11%   77.19%   +0.07%     
==========================================
  Files          40       40              
  Lines        1486     1495       +9     
  Branches      280      280              
==========================================
+ Hits         1146     1154       +8     
- Misses        217      218       +1     
  Partials      123      123
@ruebot
ruebot approved these changes Dec 29, 2019
View changes
@ruebot ruebot merged commit b915f82 into archivesunleashed:master Dec 29, 2019
3 checks passed
3 checks passed
codecov/patch 88.88% of diff hit (target 77.11%)
Details
codecov/project 77.19% (+0.07%) compared to 2c96ff3
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ruebot ruebot

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
2 participants
@SinghGursimran @ruebot
You can’t perform that action at this time.