Permalink
Commits on Jun 29, 2020
-
-
ruebot committed
Jun 29, 2020 - Use Spark conf to configure Spark jobs instead of application.yml because delayed jobs needs to be restarted to in order to tweak Spark job settings - Add Spark conf example - Don't email user on failed jobs anymore. Just email Nick. - Move from master branch to main branch - Add Page count for seed jobs for troubleshooting wasapi issues
Commits on Jun 18, 2020
-
Co-authored-by: depfu[bot] <23717796+depfu[bot]@users.noreply.github.com>
-
Co-authored-by: depfu[bot] <23717796+depfu[bot]@users.noreply.github.com>
Commits on Jun 16, 2020
-
Co-authored-by: depfu[bot] <23717796+depfu[bot]@users.noreply.github.com>
Commits on Jun 11, 2020
-
Co-authored-by: depfu[bot] <23717796+depfu[bot]@users.noreply.github.com>
Commits on Jun 10, 2020
-
Co-authored-by: depfu[bot] <23717796+depfu[bot]@users.noreply.github.com>
Commits on Jun 5, 2020
-
Co-authored-by: depfu[bot] <23717796+depfu[bot]@users.noreply.github.com>
-
Co-authored-by: depfu[bot] <23717796+depfu[bot]@users.noreply.github.com>
Commits on Jun 2, 2020
-
Co-authored-by: depfu[bot] <23717796+depfu[bot]@users.noreply.github.com>
Commits on May 28, 2020
-
Co-authored-by: depfu[bot] <23717796+depfu[bot]@users.noreply.github.com>
Commits on May 27, 2020
-
Co-authored-by: depfu[bot] <23717796+depfu[bot]@users.noreply.github.com>
Commits on May 21, 2020
-
Co-authored-by: depfu[bot] <23717796+depfu[bot]@users.noreply.github.com>
Commits on May 19, 2020
-
Co-authored-by: depfu[bot] <23717796+depfu[bot]@users.noreply.github.com>
Commits on May 12, 2020
-
Co-authored-by: depfu[bot] <23717796+depfu[bot]@users.noreply.github.com>
Commits on May 9, 2020
-
Co-authored-by: depfu[bot] <23717796+depfu[bot]@users.noreply.github.com>
-
Co-authored-by: depfu[bot] <23717796+depfu[bot]@users.noreply.github.com>
Commits on Apr 30, 2020
-
-
-
- Add sub-job to textfilter job to create crawl frequency count data - Add helper method to prepare data for visualization - Update controller and view to show visualization - Add route for crawl date visualization data - Update Graphpass job to run two parallelized sub-jobs - Graphpass + combining part files - Removing output directories
Commits on Apr 28, 2020
-
-
- Update Spark job to run on directory, since wildcard will break things on directories with MANY files.
-
* Update Spark job to run auk jobs via spark-submit in parallel. - Update Rubocop config - Remove AUK Notebooks link - Update application config example - Tweak analyzed date helper to use filtered text to get date (last item that is ran in the pipeline) - Change name of "Full Text" derivative to "Web Page Text" since full text is misleading - Multiply data analyzed total by 3 since that's what we're doing in reality
Commits on Apr 24, 2020
-
Co-authored-by: depfu[bot] <23717796+depfu[bot]@users.noreply.github.com>
Commits on Apr 18, 2020
-
Co-authored-by: depfu[bot] <23717796+depfu[bot]@users.noreply.github.com>
Commits on Apr 17, 2020
Commits on Apr 16, 2020
-
-
-
/app/assets/images/Tutorial_domain_derivative_file.png -- 168.85kb -> 149.15kb (11.66%) Signed-off-by: ImgBotApp <ImgBotHelp@gmail.com> Co-authored-by: ImgBotApp <ImgBotHelp@gmail.com>
-
* Minor tweaks to domain page (follow up on #387) * Suggest using "Import"; text -> CSV * Updating image
-
- Resolves #386 - Move the faux txt derivatives to what they actually are; csv. - Update Spark job to use DataFrames - Update auk documentation and lessons with correct file extension (s/txt/csv) - Data migration needs to be completed on prod - rename full-text and full-domains - s/.txt/.csv/g - on all -fullurls.txt - remove the first and last character on each line. ( ) - TravisCI should only test Ruby 2.6.5 - Update tests to reflect changes - Rename text fixtures
Commits on Apr 14, 2020
Commits on Apr 6, 2020
-
Co-authored-by: depfu[bot] <23717796+depfu[bot]@users.noreply.github.com>