Skip to content
Please note that GitHub no longer supports your web browser.

We recommend upgrading to the latest Google Chrome or Firefox.

Learn more

/ aut

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add language detection column to webpages. #403

Merged
merged 1 commit into from Jan 12, 2020
Merged

Add language detection column to webpages. #403

merged 1 commit into from Jan 12, 2020
+6 −3

Conversation

@ruebot
Copy link
Member

ruebot commented Jan 12, 2020

GitHub issue(s): #402

What does this Pull Request do?

Add language detection column to webpages.

How should this be tested?

Something like:

import io.archivesunleashed.df._
import io.archivesunleashed.df._

RecordLoader.loadArchives("/home/nruest/Projects/au/sample-data/geocites/1", sc).webpages().show()

Should yield:

+----------+--------------------+--------------------+--------------------+--------+--------------------+
|crawl_date|                 url|mime_type_web_server|      mime_type_tika|language|             content|
+----------+--------------------+--------------------+--------------------+--------+--------------------+
|  20091027|http://geocities....|           text/html|           text/html|      en|HTTP/1.1 200 OK
...|
|  20091027|http://geocities....|           text/html|           text/html|      en|HTTP/1.1 200 OK
...|
|  20091027|http://www.geocit...|           text/html|           text/html|      en|HTTP/1.1 200 OK
...|
|  20091027|http://www.geocit...|           text/html|           text/html|      en|HTTP/1.1 200 OK
...|
|  20091027|http://geocities....|           text/html|           text/html|      en|HTTP/1.1 200 OK
...|
|  20091027|http://geocities....|           text/html|           text/html|      en|HTTP/1.1 200 OK
...|
|  20091027|http://www.talent...|           text/html|application/xhtml...|      en|HTTP/1.1 200 OK
...|
|  20091027|http://geocities....|           text/html|           text/html|      ms|HTTP/1.1 200 OK
...|
|  20091027|http://geocities....|           text/html|           text/html|      en|HTTP/1.1 200 OK
...|
|  20091027|http://www.geocit...|           text/html|           text/html|      en|HTTP/1.1 200 OK
...|
|  20091027|http://www.geocit...|           text/html|           text/html|      en|HTTP/1.1 200 OK
...|
|  20091027|http://geocities....|           text/html|           text/html|      en|HTTP/1.1 200 OK
...|
|  20091027|http://geocities....|           text/html|           text/html|      en|HTTP/1.1 200 OK
...|
|  20091027|http://www.geocit...|           text/html|           text/html|      en|HTTP/1.1 200 OK
...|
|  20091027|http://geocities....|           text/html|application/xhtml...|      en|HTTP/1.1 200 OK
...|
|  20091027|http://geocities....|           text/html|           text/html|      en|HTTP/1.1 200 OK
...|
|  20091027|http://geocities....|           text/html|           text/html|      en|HTTP/1.1 200 OK
...|
|  20091027|http://geocities....|           text/html|           text/html|      en|HTTP/1.1 200 OK
...|
|  20091027|http://www.infoca...|           text/html|           text/html|      en|HTTP/1.1 200 OK
...|
|  20091027|http://geocities....|           text/html|           text/html|      en|HTTP/1.1 200 OK
...|
+----------+--------------------+--------------------+--------------------+--------+--------------------+

Additional Notes:

If it comes up, we don't add Remote HTML and Headers on content because we leave that all in there for ExtractHyperlinks, so we can do the domains derivative.
@ruebot
Add language detection column to webpages.
Verified
This commit was signed with a verified signature.
ruebot Nick Ruest
GPG key ID: 417FAF1A0E1080CD Learn about signing commits
Loading status checks…
b1df9a9 
- Addresses #402
@ruebot ruebot requested review from lintool and ianmilligan1 Jan 12, 2020
@codecov

This comment has been minimized.

Sign in to view
Copy link

codecov bot commented Jan 12, 2020

Codecov Report

Merging #403 into master will increase coverage by 0.04%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master     #403      +/-   ##
==========================================
+ Coverage   77.48%   77.53%   +0.04%     
==========================================
  Files          40       40              
  Lines        1537     1540       +3     
  Branches      292      292              
==========================================
+ Hits         1191     1194       +3     
  Misses        218      218              
  Partials      128      128
@ianmilligan1

This comment has been minimized.

Sign in to view
Copy link
Member

ianmilligan1 commented Jan 12, 2020

Good idea - one of those important filters we have. I’ll try to review later today or first thing tomorrow.
@lintool
lintool approved these changes Jan 12, 2020
View changes
@ianmilligan1
ianmilligan1 approved these changes Jan 12, 2020
View changes
Copy link
Member

ianmilligan1 left a comment

Works nicely. For doc purposes, the example script you had was missing an import, but this checked out:

import io.archivesunleashed._
import io.archivesunleashed.df._

RecordLoader.loadArchives("example.arc.gz", sc).webpages().show()
@ianmilligan1 ianmilligan1 merged commit bc0d663 into master Jan 12, 2020
3 checks passed
3 checks passed
codecov/patch 100% of diff hit (target 77.48%)
Details
codecov/project 77.53% (+0.04%) compared to 0ecc4f8
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@ianmilligan1 ianmilligan1 deleted the issue-402 branch Jan 12, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lintool lintool

@ianmilligan1 ianmilligan1

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
3 participants
@ruebot @ianmilligan1 @lintool
You can’t perform that action at this time.