Is your feature request related to a problem? Please describe.

Right now, users can receive the full-text of a web archival collection. Some of these are very big files, and more importantly, they often require a level of filtering to get to the useful content: filtering by date, for example, or filtering by domain.

At our datathons, we've had several teams who have compared the text of domains in a web archive collection to see what they are saying say about a supreme court nominee; labour disruptions; pipelines, etc.

Right now we have documentation on how to filter these files, but this requires that a scholar know how to use the command line and grep.

Describe the solution you'd like

Ideally, it would be nice to bake some of these grep commands into the Archives Unleashed Cloud. See the screenshot below:

The steps in this process would be:

1. Next to each of the domains, a "download" button is populated;
2. When that button is pressed, a background job starts that executes syntax similar to grep ',www.khyber.ca,' 9481-fulltext.txt > 9481-www-khyber-ca-text.txt. In general, it would be grep ,DOMAIN, COLLECTIONNUMBER-fulltext.txt > COLLECTIONNUMBER-domain-text.txt
3. An e-mail is sent to the user notifying them that the file is ready for download. Perhaps the icon would appear to download next to the "download" button or something.

The actual look and feel of the buttons will probably be different.

Describe alternatives you've considered

There are two main alternatives.

Option A: Using Spark

I'm suggesting a bash command here for two reasons:

1. The Spark queue can get backed up, and until full DataFrame implementation we don't want to have to pass over every single WARC just to get text from one domain;
2. Most importantly,, we delete the WARCs and we don't want to hold on to them longer than possible just in case somebody might run one of these jobs down the road. This runs off the full-text derivative which makes more sense to me from a resource perspective.

Option B: Pointing Users to Documentation

This is adding functionality that we tell people how to do here. I think that makes our service a bit more difficult. This way would let them just click a button, download the text file, and paste it into something like Voyant right away.