Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.
Sign upDocument the command line app. #51
Conversation
- Resolves #14 - Documents archivesunleashed/aut#431
This comment has been minimized.
This comment has been minimized.
|
Link to review new documentation: https://github.com/archivesunleashed/aut-docs/blob/issue-14/current/aut-spark-submit-app.md |
|
@ruebot, this is looking really good! I like the flow you've given to all the documentation. I have a few surface level suggestions. My only major comment is related to how to start off with the scripts - hopefully I'm not overthinking things. Look forward to working with this more! :) |
| @@ -0,0 +1,128 @@ | |||
| # Using the Toolkit with spark-submit | |||
|
|
|||
This comment has been minimized.
This comment has been minimized.
SamFritz
Apr 7, 2020
Member
Do these configuration options need to be used with a specific launch of the toolkit (e.g. package, uberjar, etc.)? At first glance, I guess I'm a little unsure of where to start or in terms of workflow, when this script would be introduced (e.g. use within or outside of sparkshell?)
This comment has been minimized.
This comment has been minimized.
ruebot
Apr 7, 2020
Author
Member
Adding a link to this https://spark.apache.org/docs/latest/submitting-applications.html
|
|
||
| The extration jobs have a basic outline of: | ||
|
|
||
| `spark-submit --class io.archivesunleashed.app.CommandLinAppRunner PATH_TO_AUT_JAR --extractor EXTRACTOR --input INPUT DIRECTORY --output OUTPUT DIRECTORY` |
This comment has been minimized.
This comment has been minimized.
SamFritz
Apr 7, 2020
Member
is it possible to put in an example of how this script would look if one of us were to run it, below the basic outline? I find that when I have an example it's a bit easier to see what needs to be changed in line.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
SamFritz
Apr 7, 2020
Member
Oh yeah, I realize the examples further down are more detailed - I just was going down a different line of thinking, so disregard my original comment
ruebot commentedApr 7, 2020
@lintool @ianmilligan1 @SamFritz here is a first crack at documenting the command line app. Please let me know how this approach works.