Skip to content
Please note that GitHub no longer supports your web browser.

We recommend upgrading to the latest Google Chrome or Firefox.

Learn more

/aut

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dataframe matchbox Implementations #387

Merged
merged 12 commits into from Dec 5, 2019
+72 −41

Conversation

@SinghGursimran
Copy link
Contributor

SinghGursimran commented Dec 4, 2019

Dataframe Implementations for ExtractDate, DetectLanguage and ExtarctBoilerpipeText

For Testing:

ExtractDate:

import io.archivesunleashed._
import io.archivesunleashed.df._
import org.apache.spark.sql.functions._


val df = RecordLoader.loadArchives("./src/test/resources/arc/example.arc.gz",sc)
			.webpages()
			.select(ExtractDateDF($"crawl_date",lit("YYYY")))
			.show(3,false)

DetectLanguage:

import io.archivesunleashed._
import io.archivesunleashed.df._

RecordLoader.loadArchives("./src/test/resources/arc/example.arc.gz",sc)
			.webpages()
			.select(DetectLanguageDF($"content"))
			.show(3,false)

ExtarctBoilerpipeText:

import io.archivesunleashed._
import io.archivesunleashed.df._

RecordLoader.loadArchives("./src/test/resources/arc/example.arc.gz",sc)
			.webpages()
			.select(ExtractBoilerpipeTextDF($"content"))
			.show(3,false)
g285sing added 10 commits Nov 7, 2019
g285sing
Issue-368
Loading status checks…
68922a2
g285sing
Issue238
0db0093
g285sing
Issue238
5817bf5
g285sing
Merge branch 'master' of https://github.com/SinghGursimran/aut
Loading status checks…
d65cb41
g285sing
test
Loading status checks…
4e5a066
g285sing
Merge branch 'master' of https://github.com/archivesunleashed/aut
c60104f
g285sing
Merge branch 'master' of https://github.com/archivesunleashed/aut
f18006e
g285sing
Merge branch 'master' of https://github.com/archivesunleashed/aut
9141e2f
g285sing
Merge branch 'master' of https://github.com/archivesunleashed/aut int…
9c172a9 
…o df-impl
g285sing
Matchbox-Implementations
Loading status checks…
8bd5634
@ruebot ruebot self-requested a review Dec 4, 2019
@codecov

This comment has been minimized.

Sign in to view
Copy link

codecov bot commented Dec 4, 2019

Codecov Report

Merging #387 into master will decrease coverage by 0.73%.
The diff coverage is 48.27%.

@@            Coverage Diff             @@
##           master     #387      +/-   ##
==========================================
- Coverage    76.7%   75.97%   -0.74%     
==========================================
  Files          41       41              
  Lines        1451     1469      +18     
  Branches      268      274       +6     
==========================================
+ Hits         1113     1116       +3     
- Misses        221      236      +15     
  Partials      117      117
@ruebot
ruebot approved these changes Dec 5, 2019
View changes
Copy link
Member

ruebot left a comment

Couple formatting and comment prose tweaks.

Tested, and works as expected. I'll get a PR in for the docs.
src/main/scala/io/archivesunleashed/df/package.scala Outdated
@@ -26,6 +26,7 @@ import java.util.Base64
/**
* UDFs for data frames.
*/

This comment has been minimized.

Sign in to view
Copy link
@ruebot

ruebot Dec 5, 2019

Member

Let's remove this blank line.
src/main/scala/io/archivesunleashed/matchbox/ExtractDateRDD.scala Outdated
@@ -49,4 +49,29 @@ object ExtractDate {
""
}
}

/** Extracts the wanted date component from a date (for DataFrames).

This comment has been minimized.

Sign in to view
Copy link
@ruebot

ruebot Dec 5, 2019

Member

Let's reword this to:

Extracts a provided date component from a date (for DataFrames).
g285sing added 2 commits Dec 5, 2019
g285sing
review_changes
Loading status checks…
a10a41d
g285sing
changes
Loading status checks…
21e29f3
@ruebot
ruebot approved these changes Dec 5, 2019
View changes
ruebot added a commit to archivesunleashed/aut-docs that referenced this pull request Dec 5, 2019
@ruebot
Updates for archivesunleashed/aut#387
Verified
This commit was signed with a verified signature.
ruebot Nick Ruest
GPG key ID: 417FAF1A0E1080CD Learn about signing commits
3e873d8
@ruebot ruebot mentioned this pull request Dec 5, 2019
Open
@ruebot

This comment has been minimized.

Sign in to view
Copy link
Member

ruebot commented Dec 5, 2019

#387
@ruebot ruebot merged commit 079cd24 into archivesunleashed:master Dec 5, 2019
1 of 3 checks passed
1 of 3 checks passed
codecov/patch 48.27% of diff hit (target 76.7%)
Details
codecov/project 75.97% (-0.74%) compared to 560ed2b
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ruebot ruebot

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
2 participants
@SinghGursimran @ruebot
You can’t perform that action at this time.