Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Example collection #13

Closed
ruebot opened this Issue Mar 3, 2019 · 17 comments

Comments

Projects
None yet
4 participants
@ruebot
Copy link
Member

ruebot commented Mar 3, 2019

Do we want to use this one? If so, we should probably cite it in the notebook. We normally do Canadian Political Parties and Interest Groups, but those are some big derivatives.

screenshot from 2019-03-02 20-13-05

ruebot added a commit that referenced this issue Mar 3, 2019

Use NLTK stopwords, cite example dataset.
- Resolve #14
- Resolve #13
- Update notebooks to use NLTK stopwords
- Add NLTK stopwords

@ruebot ruebot self-assigned this Mar 3, 2019

@ianmilligan1

This comment has been minimized.

Copy link
Member

ianmilligan1 commented Mar 3, 2019

Hmm. Let's think a bit more on this. Agreed that CPP is a bit too big. The current example data we're using isn't ideal. I think we'd like a small-ish collection with:

  • multiple years;
  • multiple domains;
  • and in a dream world, multiple languages.

I think Victoria might have some ideal candidate collections. I can try to find a cycle to dig through some of the Archive-It pages, but @SamFritz if you have a moment do you want to take a look around the UVic archive-it pages and see if there are any ones that fit that criteria?

@ianmilligan1

This comment has been minimized.

Copy link
Member

ianmilligan1 commented Mar 3, 2019

Actually, do any of these collections have manageable derivative sizes? (I don't have a UVic collection synced in the Cloud right)

If any of those stand out, I can write to UVic to see if they are interested in being used as "sample data."

@ruebot

This comment has been minimized.

Copy link
Member Author

ruebot commented Mar 3, 2019

The Trans Web:

  • Gephi: 4.46MB
  • Raw Network: 1.67MB
  • Domains: 10.4KB
  • Full Text: 1.81GB
  • Text by Domains: needs to be run

British Columbia Local Governments:

  • Gephi: 9.11MB
  • Raw Network: 4.38MB
  • Domains: 71.3KB
  • Full Text: 30.7GB
  • Text by Domains: needs to be run

B.C. Teachers' Labour Dispute (2014):

  • Gephi: 2MB
  • Raw Network: 751KB
  • Domains: 19.9KB
  • Full Text: 367MB
  • Text by Domains: needs to be run

Trans Web:

  • Gephi: 5.16MB
  • Raw Network: 1.81MB
  • Domains: 9.91KB
  • Full Text: 1.49GB
  • Text by Domains: 79.2MB
@ianmilligan1

This comment has been minimized.

Copy link
Member

ianmilligan1 commented Mar 3, 2019

OK great, thanks @ruebot. I like BC Teachers Labour Dispute: neat topic, has mostly content from 2014 but also from 2015, fair number of domains, and different domains that take very divergent perspectives on the issue. Plus it's about the size that we could bundle with the image, knock on wood.

@greebie @ruebot @SamFritz provide any thoughts you might have on using this as a sample datasets.. if I get thumbs up, I'd like to reach out to UVic.

@ruebot

This comment has been minimized.

Copy link
Member Author

ruebot commented Mar 3, 2019

Once we're in agreement, I'll create a branch for it.

@greebie

This comment has been minimized.

Copy link
Collaborator

greebie commented Mar 3, 2019

I have the UVIc account logged into my cloud account. I think the Teachers labor dispute has legs. I like the Transweb one, but I don't think it has much in terms of years available yet.

@ruebot

This comment has been minimized.

Copy link
Member Author

ruebot commented Mar 4, 2019

Should we have a section in the README like we do in docker-auk once we figure out which collection to use?

ianmilligan1 added a commit that referenced this issue Mar 4, 2019

Use NLTK stopwords, update README (#15)
    - Resolve #14
    - Partially address #13
    - Resolve #17 
    - Update notebooks to use NLTK stopwords
    - Add NLTK stopwords
@SamFritz

This comment has been minimized.

Copy link
Member

SamFritz commented Mar 4, 2019

agreed, I think the BC Teachers Labour Dispute collection would work well, as a second runner I probably select the Trans Web collection (text wise it's a bit larger).

@ianmilligan1

This comment has been minimized.

Copy link
Member

ianmilligan1 commented Mar 4, 2019

Perfect, thanks all. I'll send them an e-mail to see if there's interest.

@ruebot

This comment has been minimized.

Copy link
Member Author

ruebot commented Mar 4, 2019

The next spark job in the queue is for the BC Teachers collections. Should be done later tonight, or early tomorrow. I'll create a branch, and we'll see if it works. I think we'll be fine with the GitHub size limits.

@ruebot

This comment has been minimized.

Copy link
Member Author

ruebot commented Mar 5, 2019

I have the data ready. We need to work through #21 and #22 before I can move forward with this. Both are fairly straightforward, so hopefully we can get to this one by the end of the day worst case.

@ruebot

This comment has been minimized.

Copy link
Member Author

ruebot commented Mar 5, 2019

Back to the drawing board. We need a collection where all the derivatives are under a 100MB.

$ git push origin issue-13
Counting objects: 10, done.
Delta compression using up to 12 threads.
Compressing objects: 100% (10/10), done.
Writing objects: 100% (10/10), 72.35 MiB | 1000.00 KiB/s, done.
Total 10 (delta 2), reused 0 (delta 0)
remote: Resolving deltas: 100% (2/2), completed with 2 local objects.
remote: error: GH001: Large files detected. You may want to try Git Large File Storage - https://git-lfs.github.com.
remote: error: Trace: 11c76664f8efdae8bf95f093f60b634e
remote: error: See http://git.io/iEPt8g for more information.
remote: error: File data/4867-fulltext.txt is 360.63 MB; this exceeds GitHub's file size limit of 100.00 MB
To github.com:archivesunleashed/auk-notebooks.git
 ! [remote rejected] issue-13 -> issue-13 (pre-receive hook declined)
error: failed to push some refs to 'git@github.com:archivesunleashed/auk-notebooks.git'
@greebie

This comment has been minimized.

Copy link
Collaborator

greebie commented Mar 5, 2019

Maybe we could just truncate the text? The script will only read the first 2500 lines anyway.

@ianmilligan1

This comment has been minimized.

Copy link
Member

ianmilligan1 commented Mar 5, 2019

Yeah, I think truncating the text would work here. Trim the text to 35MB or so and just make clear that it’s a sample in the README?

@ruebot

This comment has been minimized.

Copy link
Member Author

ruebot commented Mar 5, 2019

Cool. 43k lines of text from the file is: 99M. That should do it. I'll test in a moment.

@greebie greebie closed this Mar 5, 2019

@greebie greebie reopened this Mar 5, 2019

@greebie

This comment has been minimized.

Copy link
Collaborator

greebie commented Mar 5, 2019

Ooops. Sorry - I had a comment and then closed the issue instead of deleting it.

ruebot added a commit that referenced this issue Mar 5, 2019

Add B.C. Teachers' Labour Dispute (2014); resolves #13.
- Remove existing dataset
- Add 4867 data
- Update example notebook
- Update README

ruebot added a commit that referenced this issue Mar 5, 2019

Add B.C. Teachers' Labour Dispute (2014); resolves #13.
- Remove existing dataset
- Add 4867 data
- Update example notebook
- Update README
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.