Hmm. Let's think a bit more on this. Agreed that CPP is a bit too big. The current example data we're using isn't ideal. I think we'd like a small-ish collection with:

• multiple years;
• multiple domains;
• and in a dream world, multiple languages.

I think Victoria might have some ideal candidate collections. I can try to find a cycle to dig through some of the Archive-It pages, but @SamFritz if you have a moment do you want to take a look around the UVic archive-it pages and see if there are any ones that fit that criteria?

Actually, do any of these collections have manageable derivative sizes? (I don't have a UVic collection synced in the Cloud right)

• Trans Web
• BC Local Governments
• BC Teachers' Labour Dispute
• Anarchist Archives

If any of those stand out, I can write to UVic to see if they are interested in being used as "sample data."

The Trans Web:

• Gephi: 4.46MB
• Raw Network: 1.67MB
• Domains: 10.4KB
• Full Text: 1.81GB
• Text by Domains: needs to be run

British Columbia Local Governments:

• Gephi: 9.11MB
• Raw Network: 4.38MB
• Domains: 71.3KB
• Full Text: 30.7GB
• Text by Domains: needs to be run

B.C. Teachers' Labour Dispute (2014):

• Gephi: 2MB
• Raw Network: 751KB
• Domains: 19.9KB
• Full Text: 367MB
• Text by Domains: needs to be run

Trans Web:

• Gephi: 5.16MB
• Raw Network: 1.81MB
• Domains: 9.91KB
• Full Text: 1.49GB
• Text by Domains: 79.2MB

OK great, thanks @ruebot. I like BC Teachers Labour Dispute: neat topic, has mostly content from 2014 but also from 2015, fair number of domains, and different domains that take very divergent perspectives on the issue. Plus it's about the size that we could bundle with the image, knock on wood.

@greebie @ruebot @SamFritz provide any thoughts you might have on using this as a sample datasets.. if I get thumbs up, I'd like to reach out to UVic.

Once we're in agreement, I'll create a branch for it.

I have the UVIc account logged into my cloud account. I think the Teachers labor dispute has legs. I like the Transweb one, but I don't think it has much in terms of years available yet.

Should we have a section in the README like we do in docker-auk once we figure out which collection to use?

agreed, I think the BC Teachers Labour Dispute collection would work well, as a second runner I probably select the Trans Web collection (text wise it's a bit larger).

Perfect, thanks all. I'll send them an e-mail to see if there's interest.

The next spark job in the queue is for the BC Teachers collections. Should be done later tonight, or early tomorrow. I'll create a branch, and we'll see if it works. I think we'll be fine with the GitHub size limits.

I have the data ready. We need to work through #21 and #22 before I can move forward with this. Both are fairly straightforward, so hopefully we can get to this one by the end of the day worst case.

Back to the drawing board. We need a collection where all the derivatives are under a 100MB.

$ git push origin issue-13
Counting objects: 10, done.
Delta compression using up to 12 threads.
Compressing objects: 100% (10/10), done.
Writing objects: 100% (10/10), 72.35 MiB | 1000.00 KiB/s, done.
Total 10 (delta 2), reused 0 (delta 0)
remote: Resolving deltas: 100% (2/2), completed with 2 local objects.
remote: error: GH001: Large files detected. You may want to try Git Large File Storage - https://git-lfs.github.com.
remote: error: Trace: 11c76664f8efdae8bf95f093f60b634e
remote: error: See http://git.io/iEPt8g for more information.
remote: error: File data/4867-fulltext.txt is 360.63 MB; this exceeds GitHub's file size limit of 100.00 MB
To github.com:archivesunleashed/auk-notebooks.git
 ! [remote rejected] issue-13 -> issue-13 (pre-receive hook declined)
error: failed to push some refs to 'git@github.com:archivesunleashed/auk-notebooks.git'

Snowden Archive

First Nations and Indigenous Community websites

Nova Scotia FOIPOP portal breach

Legalization of cannabis in Nova Scotia

NGOs

Maybe we could just truncate the text? The script will only read the first 2500 lines anyway.

Yeah, I think truncating the text would work here. Trim the text to 35MB or so and just make clear that it’s a sample in the README?

Cool. 43k lines of text from the file is: 99M. That should do it. I'll test in a moment.

Ooops. Sorry - I had a comment and then closed the issue instead of deleting it.