The Off-Topic Memento Toolkit

1. The Off-Topic Memento Toolkit Shawn M. Jones Michele C. Weigle Michael L. Nelson Old Dominion University Web Science and Digital Libraries Research Group @WebSciDL sjone@cs.odu.edu @shawnmjones mweigle@cs.odu.edu @weiglemc mln@cs.odu.edu @phonedude_mln Thanks to:

2. @shawnmjones @WebSciDL Many Curators Use Archive-It To Create Web Archive Collections 2 Archive-It makes it easy for curators to build collections and supply metadata for a collection.

3. @shawnmjones @WebSciDL When Building A Web Archive Collection…  Curators select web resources as seeds  Each version of a seed becomes a memento 3

4. @shawnmjones @WebSciDL When Building A Web Archive Collection…  Curators select web resources as seeds  Each version of a seed becomes a memento  They create a web archive collection with a purpose in mind 4

5. @shawnmjones @WebSciDL When Researchers Prepare to Analyze a Web Archive Collection… 5 Some collections have thousands of seeds. Remember: Each seed has one or more mementos. The sheer number of mementos to process means that researchers will need to quickly identify mementos with low information value. Off-topic mementos have low information value. We want to identify, not delete, these for further decision-making. We identify them to not consider them for selection as exemplars for storytelling. 81,014 seeds 486,227 seed mementos

6. @shawnmjones @WebSciDL How Can Mementos Go Off-Topic? 6

7. @shawnmjones @WebSciDL Mementos in a Collection Can Go Off-Topic: For Technical Reasons 7 http://wayback.archive-it.org/1068/20130306212205/http://bo.amnesty.org/ http://wayback.archive-it.org/1068/20120303011104/http://bo.amnesty.org/

8. @shawnmjones @WebSciDL Mementos in a Collection Can Go Off-Topic: Page Gone 8 http://wayback.archive-it.org/1068/20101221161732/http://www.acdauk.org.uk/ http://wayback.archive-it.org/1068/20110902210644/http://www.acdauk.org.uk/

9. @shawnmjones @WebSciDL Mementos in a Collection Can Go Off-Topic: Content Drift – A Change in Languages 9 http://wayback.archive-it.org/1068/20130306231537/http://ecwronline.org/ http://wayback.archive-it.org/1068/20110129043404/http://ecwronline.org/

10. @shawnmjones @WebSciDL Mementos in a Collection Can Go Off-Topic: Server Maintenance 10 http://wayback.archive-it.org/1068/20111202210620/http://amnestyghana.org/ http://wayback.archive-it.org/1068/20120302232416/http://amnestyghana.org/

11. @shawnmjones @WebSciDL Mementos in a Collection Can Go Off-Topic: Account Suspension 11 http://wayback.archive-it.org/1068/20110317151735/http://amnestymauritius.org/french/news.php http://wayback.archive-it.org/1068/20111202210625/http://amnestymauritius.org/cgi-sys/suspendedpage.cgi

12. @shawnmjones @WebSciDL Mementos in a Collection Can Go Off-Topic: Site Redesign 12 http://wayback.archive-it.org/1068/20120302224302/http://ombuds.am/main/ http://wayback.archive-it.org/1068/20100510173253/http://ombuds.am/main

13. @shawnmjones @WebSciDL Mementos in a Collection Can Go Off-Topic: Change Site Ownership 13 http://wayback.archive-it.org/1068/20090210190543/http://www.afapredesa.org/index.php http://wayback.archive-it.org/1068/20120302210439/http://www.afapredesa.org/

14. @shawnmjones @WebSciDL Mementos in a Collection Can Go Off-Topic: The Site Was Hacked 14 http://wayback.archive-it.org/2950/20120327032244/http://occupyevansville.org/ http://wayback.archive-it.org/2950/20120410032628/http://occupyevansville.org/

15. @shawnmjones @WebSciDL Mementos in a Collection Can Go Off-Topic: The Site Moves On From The Topic 15 http://wayback.archive-it.org/2358/20120803140009/http://www.bbc.co.uk/news/world/middle_east/ http://wayback.archive-it.org/2358/20110202225040/http://www.bbc.co.uk/news/world/middle_east/

16. @shawnmjones @WebSciDL Presenting the Off-Topic Memento Toolkit (OTMT) a tool for identifying these off-topic mementos 16

17. @shawnmjones @WebSciDL The Off-Topic Memento Toolkit (OTMT)  Currently in alpha status, the OTMT  Accepts a collection of mementos  Executes similarity measures on those mementos  Rates them as on or off-topic  Identifies, does not delete, off- topic mementos 17 https://github.com/oduwsdl/off-topic-memento-toolkit

18. @shawnmjones @WebSciDL Background and Related Work 18

19. @shawnmjones @WebSciDL Related Work – Similarity Measures for Documents 19 Manku (2007) Sorensen (1948) Dice (1945) Jaccard (1912) Simhash Charikar (2002) Sørensen-Dice Coefficient Jaccard Index Hajishirzi (2010) Cosine Similarity of TF-IDF Vectors Cosine Similarity of Latent Semantic Indexing Vectors Deerweister (1990) Adar (2009) Sivakumar (2015) Řehůřek (2011) Content Drift in Web Archives Jones (2016) Zittrain (2014)

20. @shawnmjones @WebSciDL Related Work – Similarity Measures for Documents 20 Manku (2007) Sorensen (1948) Dice (1945) Jaccard (1912) Simhash Charikar (2002) Sørensen-Dice Coefficient Jaccard Index Hajishirzi (2010) Cosine Similarity of TF-IDF Vectors Cosine Similarity of Latent Semantic Indexing Vectors Deerweister (1990) OTMT supports these similarity measures Adar (2009) Sivakumar (2015) Řehůřek (2011) Content Drift in Web Archives Jones (2016) Zittrain (2014)

21. @shawnmjones @WebSciDL Related Work – Similarity Measures for Documents 21 Manku (2007) Sorensen (1948) Dice (1945) Jaccard (1912) Simhash Charikar (2002) Sørensen-Dice Coefficient Jaccard Index Hajishirzi (2010) Cosine Similarity of TF-IDF Vectors Cosine Similarity of Latent Semantic Indexing Vectors Deerweister (1990) OTMT supports these similarity measures Adar (2009) Sivakumar (2015) Řehůřek (2011) Like these studies, we also use these similarity measures on mementos Content Drift in Web Archives Jones (2016) Zittrain (2014)

22. @shawnmjones @WebSciDL Related Work – Other Methods of Off-Topic Detection 22 AlNoamany (2016) Latent Dirichlet Allocation Blei (2003) Browser Thumbnails of Mementos AlSum (2012) Off-Topic Analysis

23. @shawnmjones @WebSciDL Related Work – Other Methods of Off-Topic Detection 23 AlNoamany (2016) Latent Dirichlet Allocation Blei (2003) Browser Thumbnails of Mementos AlSum (2012) Off-Topic Analysis Topic modeling should help us find off-topic documents, but which cluster is off- topic?

24. @shawnmjones @WebSciDL Related Work – Other Methods of Off-Topic Detection 24 AlNoamany (2016) Latent Dirichlet Allocation Blei (2003) Browser Thumbnails of Mementos AlSum (2012) Off-Topic Analysis Topic modeling should help us find off-topic documents, but which cluster is off- topic? It is costly to manually review browser thumbnails to find off-topic mementos

25. @shawnmjones @WebSciDL Related Work – Other Methods of Off-Topic Detection 25 AlNoamany (2016) Latent Dirichlet Allocation Blei (2003) Browser Thumbnails of Mementos AlSum (2012) Off-Topic Analysis We build on AlNoamany’s work to bring you the Off-Topic Memento Toolkit Topic modeling should help us find off-topic documents, but which cluster is off- topic? It is costly to manually review browser thumbnails to find off-topic mementos

26. @shawnmjones @WebSciDL Memento Protocol Terminology <http://a.example.org>;rel="original", <http://arxiv.example.net/timemap/http://a.example.org>; rel="self"; type="application/link-format" ; from="Tue, 20 Jun 2000 18:02:59 GMT" ; until="Wed, 21 Jun 2000 04:41:56 GMT", <http://arxiv.example.net/timegate/http://a.example.org>; rel="timegate", <http://arxiv.example.net/web/20000620180259/http://a.example.org>; rel="first memento"; datetime="Tue, 20 Jun 2000 18:02:59 GMT", <http://arxiv.example.net/web/20091027204954/http://a.example.org>; rel="last memento"; datetime="Tue, 27 Oct 2009 20:49:54 GMT", <http://arxiv.example.net/web/20000621011731/http://a.example.org>; rel="memento"; datetime="Wed, 21 Jun 2000 01:17:31 GMT", <http://arxiv.example.net/web/20000621044156/http://a.example.org>; rel="memento"; datetime="Wed, 21 Jun 2000 04:41:56 GMT" … 26 Each seed, or original resource, has a corresponding TimeMap listing all of that seed’s mementos and capture times, their memento- datetimes. URI-T: a URI for a TimeMap URI-M: a URI for a memento

27. @shawnmjones @WebSciDL Web Archives Augment Their Mementos 27 Banners Rewritten Links http://ws-dl.blogspot.com/2016/04/2016-04-27-mementos-in-raw.html http://ws-dl.blogspot.com/2016/08/2016-08-15-mementos-in-raw-take-two.html

28. @shawnmjones @WebSciDL The OTMT Uses Raw Mementos 28 Raw mementos are free of these augmentations. Archive-It and the Internet Archive provide access to raw mementos at special URIs. The OTMT finds these raw mementos and uses them in its similarity comparisons. http://ws-dl.blogspot.com/2016/04/2016-04-27-mementos-in-raw.html http://ws-dl.blogspot.com/2016/08/2016-08-15-mementos-in-raw-take-two.html

29. @shawnmjones @WebSciDL The OTMT Performs Preprocessing 29 <p class=“homepage-description”>The Women’s Initiatives for Gender Justice works globally to ensure justice for women and an independent and effective International Criminal Court.</p> ['The', 'Women', '’', 's', 'Initiatives', 'for', 'Gender', 'Justice', 'works', 'globally', 'to', 'ensure', 'justice', 'for', 'women', 'and', 'an', 'independent', 'and', 'effective', 'International', 'Criminal', 'Court', '.'] Tokenization Remove stop words ['Women', '’', 'Initiatives', 'Gender', 'Justice', 'works', 'globally', 'ensure', 'justice', 'women', 'independent', 'effective', 'International', 'Criminal', 'Court'] Stemming ['women', '’', 'initi', 'gender', 'justic', 'work', 'global', 'ensur', 'justic', 'women', 'independ', 'effect', 'intern', 'crimin', 'court'] Boilerplate removal The Women’s Initiatives for Gender Justice works globally to ensure justice for women and an independent and effective International Criminal Court.

30. @shawnmjones @WebSciDL We Evaluated the OTMT with a Gold Standard Dataset  In “Detecting off-topic pages within TimeMaps in Web archives”, AlNoamany performed a study to detect off-topic Mementos  The mementos were manually marked as on or off-topic  We reuse this dataset in our evaluation 30 https://github.com/oduwsdl/offtopic-goldstandard-data

31. @shawnmjones @WebSciDL TimeMap Measures Supported by OTMT 31

32. @shawnmjones @WebSciDL General algorithm  For each TimeMap in a collection 1. Get the first memento 2. Preprocess it 3. For each memento in the TimeMap 1. Get the memento 2. Preprocess it 3. Compute the similarity to the first memento using a given measure 4. Save the score 5. A threshold value determines if a memento is on or off-topic 32 First memento Considered memento

33. @shawnmjones @WebSciDL Structural Measures – Byte Count and Word Count 33 On-topic: 9599 bytes 183 words (after preprocessing) Off-topic: 401 bytes 22 words (after preprocessing) Off-topic mementos tend to have less bytes/words Scores range from 0 to -1

34. @shawnmjones @WebSciDL Set Operation Measures 34 Jaccard Distance Sørensen-Dice Distance Size of Intersection over size of union Twice the size of intersection over size of both sets Scores range from 0 to 1 ['women', '’', 'initi', 'gender', 'justic', 'current', 'work', 'uganda', 'democrat', 'republ', 'congo', 'libya'] ['women', '’', 'initi', 'gender', 'justic', 'work', 'uganda', 'democrat', 'republ', 'congo', 'sudan', 'central', 'african', 'republ', 'kenya', 'libya', 'kyrgyzstan'] Highlighted words are the intersection Words from Doc #1: Words from Doc #2:

35. @shawnmjones @WebSciDL Simhash of Term Frequencies 35 ('women', 4), ('justic', 4), ('’', 3), ('gender', 3), ('initi', 2), ('intern', 2), ('icc', 2), ('work', 2), ('republ', 2), ('human', 1), … 13221438115839111206 13797903006343525414 ('women', 4), ('justic', 4), ('’', 3), ('gender', 3), ('initi', 2), ('intern', 2), ('icc', 2), ('work', 2), ('human', 1), ('right', 1), … 6 bits Scores range from 0 to 64 bits Simhash Distance: Simhash of Terms and Frequencies from Document #1: Simhash of Terms and Frequencies from Document #2:

36. @shawnmjones @WebSciDL Simhash of raw content 36 The Women’s Initiatives for Gender Justice is an international women’s human rights organisation that advocates for gender justice through the International Criminal Court (ICC) and through domestic mechanisms, including peace negotiations and justice processes.We work with women and communities most affected by the armed conflict with a focus on countries with situations under investigation by the ICC. The Women’s Initiatives for Gender Justice currently works in Uganda, the Democratic Republic of the Congo and Libya. The Women’s Initiatives for Gender Justice is an international women’s human rights organisation that advocates for gender justice through the International Criminal Court (ICC) and through domestic mechanisms, including peace negotiations and justice processes. We work with women most affected by the conflict situations under investigation by the ICC. The Women’s Initiatives for Gender Justice works in Uganda, the Democratic Republic of the Congo, Sudan, the Central African Republic, Kenya, Libya and Kyrgyzstan. 12358429319379250844 12359555184926328508 6 bits Scores range from 0 to 64 bits Simhash of Document #1: Simhash of Document #2: Simhash Distance:

37. @shawnmjones @WebSciDL Cosine Similiarities 37 Take the cosine of the document vectors. Cosine of TF-IDF Vectors are formed from each document and their term frequencies. Cosine of Latent Semantic Indexing (LSI) Each vector is informed by LSI. Scores range from 1 to 0.

38. @shawnmjones @WebSciDL Using the OTMT 38

39. @shawnmjones @WebSciDL OTMT Installation Options 1. Pip from Pypi (preferred): pip install otmt 2. Experimental Docker Image: docker pull shawnmjones/otmt 3. Source Code: git clone https://github.com/oduwsdl/off-topic-memento- toolkit.git 39

40. @shawnmjones @WebSciDL OTMT Usage 40 # detect_off_topic -i archiveit=7877 -tm jaccard=0.80,bytecount=-0.50 -o outputfile.json Input Types for -i: • timemap – followed by 1 or more TimeMap URIs, separated by commas • warc – followed by 1 or more WARC files, separated by commas • archiveit – followed by an Archive-It collection ID TimeMap measures for -tm: • bytecount • wordcount • jaccard • sorensen • simhash-tf • simhash-raw • cosine • gensim_lsi Input OutputMeasure Output types for -ot: • json • csv

41. @shawnmjones @WebSciDL OTMT Output - JSON 41 "http://wayback.archive-it.org/1068/timemap/link/http://www.badil.org/": { "http://wayback.archive-it.org/1068/20130307084848/http://www.badil.org/": { "timemap measures": { "cosine": { "stemmed": true, "tokenized": true, "removed boilerplate": true, "comparison score": 0.10969941307631487, "topic status": "off-topic” }, "bytecount": { "stemmed": false, "tokenized": false, "removed boilerplate": false, "comparison score": 0.15971409055425445, "topic status": "on-topic" } }, "overall topic status": "off-topic" }, ... Measure Information Preprocessing status Measure Score On or off topic status by measure On or off topic status overall URI-T of TimeMap URI-M of Memento

42. @shawnmjones @WebSciDL OTMT Output - JSON 42 "http://wayback.archive-it.org/1068/timemap/link/http://www.badil.org/": { "http://wayback.archive-it.org/1068/20130307084848/http://www.badil.org/": { "timemap measures": { "cosine": { "stemmed": true, "tokenized": true, "removed boilerplate": true, "comparison score": 0.10969941307631487, "topic status": "off-topic” }, "bytecount": { "stemmed": false, "tokenized": false, "removed boilerplate": false, "comparison score": 0.15971409055425445, "topic status": "on-topic" } }, "overall topic status": "off-topic" }, ... URI-T of TimeMap URI-M of Memento Measure Information Preprocessing status Measure Score On or off topic status by measure On or off topic status overall If one measure scores as off-topic, the memento is considered off-topic

43. @shawnmjones @WebSciDL Supported Similarity Measures Measure Fully Equivalent Score Fully Dissimilar Score Preprocessing Performed OTMT -tm keyword Byte Count 0.0 -1.0 No bytecount Word Count 0.0 -1.0 Yes wordcount Jaccard Distance 0.0 1.0 Yes jaccard Sørensen-Dice 0.0 1.0 Yes sorensen Simhash of Term Frequencies 0 64 Yes simhash-tf Simhash or raw memento 0 64 No simhash-raw Cosine Similarity of TF-IDF Vectors 1.0 0 Yes cosine Cosine Similarity of LSI Vectors 1.0 0 Yes gensim_lsi 43

44. @shawnmjones @WebSciDL Establishing Reasonable Defaults 44

45. @shawnmjones @WebSciDL Experiment setup  For each measure: 1. Start the threshold at the score of complete dissimilarity 2. Test with the URI-Ms from the gold standard data set as if that threshold indicated off-topic 3. Compute F1 using real off-topic status of the memento from the gold standard data 4. Increment the threshold 5. Repeat 2 – 4 until the threshold matches complete equivalence score 45  Example using Byte Count: 1. Start threshold at -1 2. Test with the URI-Ms from the gold standard data set as if -1 indicated off- topic 3. Compute F1 using real off-topic status of the memento from the gold standard data 4. Increment the threshold to -0.99 5. Test with the URI-Ms from the gold standard data set as if -0.99 indicated off-topic 6. Compute F1 with real status 7. Increment to -0.98 8. Repeat until the threshold is 0

46. @shawnmjones @WebSciDL Our results do not match AlNoamany’s, but the world is not the same as it was in 2015… AlNoamany’s Study Our Study Year Conducted 2015 2017 Boilerplate Removal Boilerpipe (Java) Justext Tokenization and Stemming Scikit-learn NLTK 46 Other changes: • Download errors • Gold Standard Dataset updates

47. @shawnmjones @WebSciDL Simhash of Term Frequencies 47 Our Results: AlNoamany’s Results Not tested

48. @shawnmjones @WebSciDL Simhash of raw memento 48 Our Results: AlNoamany’s Results Not tested

49. @shawnmjones @WebSciDL Sørensen-Dice Distance Results 49 Our Results: AlNoamany’s Results Not tested

50. @shawnmjones @WebSciDL Jaccard Distance Results 50 Our Results: AlNoamany’s Results Best F1 Score: 0.538 Threshold: 0.95

51. @shawnmjones @WebSciDL Cosine Similarity of LSI Vectors 51 AlNoamany’s Results Not tested Our Results: Note: LSI scores are non-deterministic

52. @shawnmjones @WebSciDL Byte Count Results 52 AlNoamany’s Results Best F1 Score: 0.584 Threshold: -0.65 Our Results:

53. @shawnmjones @WebSciDL Cosine Similarity of TF-IDF Vectors 53 Our Results: AlNoamany’s Results Best F1 Score: 0.881 Threshold: 0.15 Best score in AlNoamany’s Results

54. @shawnmjones @WebSciDL Word Count Results 54 Best Score in Our Results: AlNoamany’s Results Best F1 Score: 0.806 Threshold: -0.85

55. @shawnmjones @WebSciDL Results Summarized – Best F1 Score is Word Count 55 AlNoamany's Results Results of this study Similarity Measure Best F1 Score Corresponding Accuracy Corresponding Threshold Best F1 Score Corresponding Accuracy Corresponding Threshold Word Count 0.806 0.982 -0.85 0.788 0.971 -0.7 Cosine Similarity of TF-IDF Vectors 0.881 0.983 0.15 0.766 0.965 0.12 Byte Count 0.584 0.962 -0.65 0.756 0.965 -0.39 Cosine Similarity of LSI Vectors Not tested 0.711 0.965 0.12 with 10 topics Jaccard Distance 0.538 0.962 0.95 0.651 0.953 0.94 Sørensen-Dice Distance Not tested 0.649 0.953 0.88 Simhash on raw memento content Not tested 0.578 0.934 25 Simhash on TF Not tested 0.523 0.942 28 Our word count measure came out ahead of AlNoamany’s. AlNoamany’s Cosine Similarity measure came out ahead of ours.

56. @shawnmjones @WebSciDL What about using measures together? 56 AlNoamany found that using cosine similarity of TF-IDF vectors and word count together produced even better results. Our best F1 score for word count alone was 0.788. Word count combined with LSI turned out to be slightly better with the same Accuracy. The success of word count appears to exert influence on the threshold of its partner measure, making its threshold more strict.

57. @shawnmjones @WebSciDL The Future of OTMT 57

58. @shawnmjones @WebSciDL Improving the OTMT  Bug fixes  Make LSI scores reproducible  New Measures  TimeMap Measures – compare first memento with considered memento:  Spamsum of the raw content – used by Andy Jackson at the UKWA  Cosine of LDA Vectors via Gensim  Collection Measures 1. Develop a collection-wide picture 2. Compare each memento against that picture  Control over preprocessing:  Options to change use a different boilerplate method  Options to turn off stemming or stop word removal 58

59. @shawnmjones @WebSciDL Conclusion 59

60. @shawnmjones @WebSciDL Motivation - Mementos Can Go Off-Topic 60 Hacked Moved on from topic Collections have a theme Seeds are selected to support that theme Mementos are versions of seeds Some of these versions are off-topic Identifying these off-topic mementos is key to some research activities, like summarization Web Page Gone Account Suspension

61. @shawnmjones @WebSciDL OTMT supports different similarity measures with thresholds established based on experimentation  Byte count  Word count  Jaccard distance  Sørensen-Dice distance  Simhash of term frequencies  Simhash of raw memento content  Cosine similarity of TF-IDF vectors  Cosine similarity of LSI vectors 61

62. @shawnmjones @WebSciDL Please try out the Off-Topic Memento Toolkit! 62 Thanks to: 1. Pip (preferred): pip install otmt 2. Experimental Docker Image: docker pull shawnmjones/otmt 3. Source Code: git clone https://github.com/oduwsdl/off-topic- memento-toolkit.git https://github.com/oduwsdl/off-topic-memento-toolkit https://github.com/oduwsdl/offtopic-goldstandard-data

The Off-Topic Memento Toolkit

Shawn Jones

The Off-Topic Memento Toolkit