Perfect your code
With built-in code review tools, GitHub makes it easy to raise the quality bar before you ship. Join the 36 million developers who've merged over 200 million pull requests.
Sign up for free See pricing for teams and enterprisesPython formatting, and gitignore additions. #326
Conversation
ruebot
requested a review
from ianmilligan1
Jul 8, 2019
This comment has been minimized.
This comment has been minimized.
|
@ianmilligan1 I have all these steps save locally, so we can use them for documentation when the time comes |
This comment has been minimized.
This comment has been minimized.
codecov-io
commented
Jul 17, 2019
•
Codecov Report
@@ Coverage Diff @@
## master #326 +/- ##
=======================================
Coverage 74.97% 74.97%
=======================================
Files 39 39
Lines 1123 1123
Branches 197 197
=======================================
Hits 842 842
Misses 215 215
Partials 66 66Continue to review full report at Codecov.
|
ruebot
added some commits
Jul 17, 2019
ianmilligan1
approved these changes
Jul 18, 2019
|
Woohoo. Very great stuff. Lots of politician faces in this sample web archive of Canadian political parties:
Apologies for the delay on this – a few dozen Slack messages and wrangling and this was successfully built in a conda virtual environment (this guide was useful for future reference. For documentation purposes, on MacOS, the default URL for Spark master was formatted as |
This comment has been minimized.
This comment has been minimized.
|
Oh, that's good to know about the mac side of things. |
ruebot commentedJul 8, 2019
What does this Pull Request do?
Follow-on to 7a61f0e
blackandisorton Python files.How should this be tested?
I tested locally, and it was good to go. @ianmilligan1 if you want to test on your end, grab a small WARC (990/8471 is perfect!), then:
conda install pysparkconda install tensorflowconda install pyarrowexport PYSPARK_PYTHON=/home/ruestn/anaconda3/bin/pythonexport PYSPARK_DRIVER_PYTHON=/home/ruestn/anaconda3/bin/pythonBuild the branch locally
Pull down the models:
cd /tmp && wget http://download.tensorflow.org/models/object_detection/ssd_mobilenet_v1_fpn_shared_box_predictor_640x640_coco14_sync_2018_07_03.tar.gztar -xzvf ssd_mobilenet_v1_fpn_shared_box_predictor_640x640_coco14_sync_2018_07_03.tar.gzmkdir -p /PATH/TO/aut/src/main/python/tf/model/graph/ssd_mobilenet_v1_fpn_640x640/cp /tmp/ssd_mobilenet_v1_fpn_shared_box_predictor_640x640_coco14_sync_2018_07_03/frozen_inference_graph.pb /PATH/TO/aut/src/main/python/tf/model/graph/ssd_mobilenet_v1_fpn_640x640/mkdir -p /PATH/TO/aut/src/main/python/tf/model/category/cd /PATH/TO/aut/src/main/python/tf/model/category/wget https://raw.githubusercontent.com/tensorflow/models/master/research/object_detection/data/mscoco_label_map.pbtxtcp /PATH/TO/aut/src/main/python/tf/util/spark.conf.example /PATH/TO/aut/src/main/python/tf/util/spark.conf/PATH/TO/SPARK/sbin/start-master.sh/PATH/TO/SPARK/sbin/start-slave.sh 127.0.1.1:7077python /PATH/TO/aut/src/main/python/tf/detect.py --web_archive "/home/nruest/tmp/auk/990/8471/warcs/*" --aut_jar /home/nruest/Projects/au/aut/target/aut-0.17.1-SNAPSHOT-fatjar.jar --spark /home/nruest/bin/spark-2.4.1-bin-hadoop2.7/bin --master spark://127.0.1.1:7077 --img_model ssd --filter_size 50 50 --output_path /home/nruest/Projects/au/sample-data/aut-image-tf-testing-03python /PATH/TO/src/main/python/tf/extract_images.py --res_dir /home/nruest/Projects/au/sample-data/aut-image-tf-testing-03 --output_dir /home/nruest/Projects/au/sample-data/aut-image-tf-testing-image-output-03 --threshold 0.85