Skip to content

/ hypercane

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Make domain category lists for hc score dsa1-score configurable. #10

Open
shawnmjones opened this issue Jun 16, 2020 · 1 comment
Open

Make domain category lists for hc score dsa1-score configurable. #10

shawnmjones opened this issue Jun 16, 2020 · 1 comment
Assignees
@shawnmjones
Labels
enhancement

Comments

@shawnmjones
Copy link
Member

@shawnmjones shawnmjones commented Jun 16, 2020

As part of the DSA1 scoring equation, the original resource domain of the memento is given a different score based on its category according to Padia's 2012 work. Padia outlined the following categories:

  • news source
  • video sharing sites
  • social media domains
  • video sharing sites

Right now these domain lists are hard-coded and likely to change over time. Create a parameter that allows the user to supply them.
@shawnmjones shawnmjones self-assigned this Jun 16, 2020
@shawnmjones shawnmjones mentioned this issue Jun 25, 2020
Closed
@shawnmjones
Copy link
Member Author

@shawnmjones shawnmjones commented Jun 25, 2020

Scoring mementos by Padia's work depends upon a list of domains for the categories of news, image sharing sites, video sharing sites, blog sites, and social media sites. These lists will likely change over time and should be configurable by the end user.

The scoring is handled on lines 431 - 451 of hypercane/score/dsa1_ranking. We will have to make it generic.

hypercane/hypercane/score/dsa1_ranking.py

Lines 431 to 451 in 2e071d7

if domain in blog_sources:
return 0.4
elif domain in wikipedia_imagesharing_sources:
return 0.6
elif domain.upper() in pew_news_sources:
return 0.7
elif domain in w3newspapers_sources:
return 0.7
elif 'news' in domain:
return 0.7
elif domain in wikipedia_video_sources:
return 0.7
elif domain in adobe_socialmedia_sources:
return 0.5

The lists themselves are on lines 15 - 386 of that same file.

hypercane/hypercane/score/dsa1_ranking.py

Lines 15 to 386 in 2e071d7

blog_sources = [
'blogger.com',
'blogspot.com',
'wordpress.com',
'typepad.com'
]
# image sharing websites as per https://en.wikipedia.org/wiki/List_of_image-sharing_websites
wikipedia_imagesharing_sources = [
'500px.com',
'album2.com',
'bilddagboken.se',
'myphotodiary.com',
'kuvapaivakirja.fi',
'bildedagboka.no',
'billeddagbog.dk',
'deviantart.com',
'dronestagr.am',
'flickr.com',
'fotki.com',
'fotolog.com',
'fotolog.net',
'geograph.org.uk',
'photos.google.com',
'instagram.com',
'imgur.com',
'ipernity.com',
'jalbum.net',
'photobucket.com',
'pinterest.com',
'pixabay.com',
'securetribeapp.com',
'shutterflyinc.com',
'smugmug.com',
'snapfish.com',
'unsplash.com'
]
# video domains as per https://en.wikipedia.org/wiki/List_of_video_hosting_services#Specifically_dedicated_video_hosting_websites
wikipedia_video_sources = [
'acfun.cn',
'afreecatv.com',
'aparat.com',
'bigo.tv',
'bilibili.com',
'bitchute.com',
'dailymotion.com',
'godtube.com',
'iqiyi.com',
'liveleak.com',
'metacafe.com',
'mixer.com',
'nicovideo.jp',
'periscope.tv',
'rutube.ru',
'schooltube.com',
'smashcast.tv',
'trilulilu.ro',
'tudou.com',
'tune.pk',
'twitch.tv',
'vbox7.com',
'veoh.com',
'vimeo.com',
'youku.com',
'younow.com',
'youtube.com'
]
# social media domains as per https://helpx.adobe.com/analytics/kb/list-social-networks.html
adobe_socialmedia_sources = [
'12seconds.tv',
'4travel.jp',
'advogato.org',
'ameba.jp',
'anobii.com',
'answers.yahoo.com',
'asmallworld.net',
'avforums.com',
'backtype.com',
'badoo.com',
'bebo.com',
'bigadda.com',
'bigtent.com',
'biip.no',
'blackplanet.com',
'blog.seesaa.jp',
'blogspot.com',
'blogster.com',
'blomotion.jp',
'bolt.com',
'brightkite.com',
'buzznet.com',
'cafemom.com',
'care2.com',
'classmates.com',
'cloob.com',
'collegeblender.com',
'cyworld.co.kr',
'cyworld.com.cn',
'dailymotion.com',
'delicious.com',
'deviantart.com',
'digg.com',
'diigo.com',
'disqus.com',
'draugiem.lv',
'facebook.com',
'faceparty.com',
'fc2.com',
'flickr.com',
'flixster.com',
'fotolog.com',
'foursquare.com',
'friendfeed.com',
'friendsreunited.co.uk',
'friendsreunited.com',
'friendster.com',
'fubar.com',
'gaiaonline.com',
'geni.com',
'goodreads.com',
'grono.net',
'habbo.com',
'hatena.ne.jp',
'hi5.com',
'hotnews.infoseek.co.jp',
'hyves.nl',
'ibibo.com',
'identi.ca',
'imeem.com',
'instagram.com',
'intensedebate.com',
'irc-galleria.net',
'iwiw.hu',
'jaiku.com',
'jp.myspace.com',
'kaixin001.com',
'kaixin002.com',
'kakaku.com',
'kanshin.com',
'kozocom.com',
'last.fm',
'linkedin.com',
'livejournal.com',
'lnkd.in',
'matome.naver.jp',
'me2day.net',
'meetup.com',
'mister-wong.com',
'mixi.jp',
'mixx.com',
'mouthshut.com',
'mp.weixin.qq.com',
'multiply.com',
'mumsnet.com',
'myheritage.com',
'mylife.com',
'myspace.com',
'myyearbook.com',
'nasza-klasa.pl',
'netlog.com',
'nettby.no',
'netvibes.com',
'nextdoor.com',
'nicovideo.jp',
'ning.com',
'odnoklassniki.ru',
'ok.ru',
'orkut.com',
'pakila.jp',
'photobucket.com',
'pinterest.at',
'pinterest.be',
'pinterest.ca',
'pinterest.ch',
'pinterest.cl',
'pinterest.co',
'pinterest.co.kr',
'pinterest.co.uk',
'pinterest.com',
'pinterest.de',
'pinterest.dk',
'pinterest.es',
'pinterest.fr',
'pinterest.hu',
'pinterest.ie',
'pinterest.in',
'pinterest.jp',
'pinterest.nz',
'pinterest.ph',
'pinterest.pt',
'pinterest.se',
'plaxo.com',
'plurk.com',
'plus.google.com',
'plus.url.google.com',
'po.st',
'reddit.com',
'renren.com',
'skyrock.com',
'slideshare.net',
'smcb.jp',
'smugmug.com',
'sonico.com',
'studivz.net',
'stumbleupon.com',
't.163.com',
't.co',
't.hexun.com',
't.ifeng.com',
't.people.com.cn',
't.qq.com',
't.sina.com.cn',
't.sohu.com',
'tabelog.com',
'tagged.com',
'taringa.net',
'thefancy.com',
'toutiao.com',
'tripit.com',
'trombi.com',
'trytrend.jp',
'tuenti.com',
'tumblr.com',
'twine.com',
'twitter.com',
'uhuru.jp',
'viadeo.com',
'vimeo.com',
'vk.com',
'wayn.com',
'weibo.com',
'weourfamily.com',
'wer-kennt-wen.de',
'wordpress.com',
'xanga.com',
'xing.com',
'yammer.com',
'yaplog.jp',
'yelp.co.uk',
'yelp.com',
'youku.com',
'youtube.com',
'yozm.daum.net',
'yuku.com',
'zhihu.com',
'zooomr.com'
]
# news domains from https://pewresearch-org-preprod.go-vip.co/journalism/2019/07/23/state-of-the-news-media-methodology/#digital-native-news-outlet-audit
pew_news_sources = [
'12UP.COM',
'247SPORTS.COM',
'90MIN.COM',
'APLUS.COM',
'BGR.COM',
'BLEACHERREPORT.COM',
'BREITBART.COM',
'BUSINESSINSIDER.COM',
'BUSTLE.COM',
'BUZZFEED.COM',
'BUZZFEEDNEWS.COM',
'CHEATSHEET.COM',
'CINEMABLEND.COM',
'CNET.COM',
'COMICBOOK.COM',
'DAILYDOT.COM',
'DEADSPIN.COM',
'DIGITALTRENDS.COM',
'EATER.COM',
'ELITEDAILY.COM',
'ENGADGET.COM',
'FIVETHIRTYEIGHT.COM',
'GAMESPOT.COM',
'GIZMODO.COM',
'HELLOGIGGLES.COM',
'HOLLYWOODLIFE.COM',
'HUFFINGTONPOST.COM',
'IBTIMES.COM',
'IFLSCIENCE.COM',
'IGN.COM',
'IJR.COM',
'IJREVIEW.COM',
'INVESTOPEDIA.COM',
'JEZEBEL.COM',
'MARKETWATCH.COM',
'MASHABLE.COM',
'MAXPREPS.COM',
'MIC.COM',
'OPPOSINGVIEWS.COM',
'POLITICO.COM',
'POLYGON.COM',
'QZ.COM',
'RARE.US',
'RAWSTORY.COM',
'REFINERY29.COM',
'SALON.COM',
'SBNATION.COM',
'SLATE.COM',
'TECHRADAR.COM',
'THEBLAZE.COM',
'THEDAILYBEAST.COM',
'THEROOT.COM',
'THEVERGE.COM',
'THISISINSIDER.COM',
'THRILLIST.COM',
'TMZ.COM',
'TOPIX.COM',
'TOPIX.NET',
'UPROXX.COM',
'UPWORTHY.COM',
'VOX.COM'
]
# sources from https://www.w3newspapers.com/newssites/
w3newspapers_sources = [
'aljazeera.com',
'nytimes.com',
'wsj.com',
'huffpost.com',
'washingtonpost.com',
'latimes.com',
'reuters.com',
'abcnews.go.com',
'usatoday.com',
'bloomberg.com',
'nbcnews.com',
'dailymail.co.uk',
'theguardian.com',
'thesun.co.uk',
'mirror.co.uk',
'telegraph.co.uk',
'bbc.com',
'thestar.com',
'theglobeandmail.com',
'news.com.au',
'forbes.com',
'cnbc.com',
'chinadaily.com.cn',
'chron.com',
'nypost.com',
'usnews.com',
'dw.com',
'indiatimes.com',
'thehindu.com',
'indianexpress.com',
'hindustantimes.com',
'cbsnews.com',
'time.com',
'sfgate.com',
'thehill.com',
'thedailybeast.com',
'newsweek.com',
'theatlantic.com',
'nzherald.co.nz',
'herald.co.zw',
'vanguardngr.com',
'dailysun.co.za',
'thejakartapost.com',
'thestar.com.my',
'straitstimes.com',
'bangkokpost.com',
'japantimes.co.jp',
'thedailystar.net',
'dawn.com',
'alarabiya.net',
'hollywoodreporter.com',
'scmp.com',
'aljazeera.com',
'voanews.com'
]

@shawnmjones shawnmjones added enhancement and removed duplicate labels Jun 25, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@shawnmjones shawnmjones

Labels
enhancement
Projects
None yet
Milestone
No milestone
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
1 participant
@shawnmjones
You can’t perform that action at this time.