Skip to content

/browsertrix

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

API Refactor / Browser Profiles Support #15

Merged
merged 7 commits into from Apr 23, 2019
+899 −654

Conversation

Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
1 participant
@ikreymer
@ikreymer
Copy link
Member

commented Apr 23, 2019

  • Combine start and create options into create options, start just starts crawl!
  • Move crawling CLI to browsertrix crawl
  • Add support for browser profiles via CLI with browsertrix profile #14
  • Add support for cache options (always, never, default)
  • Add live mode/no-proxy, enable auto fetch worker for pywb proxy

ikreymer added some commits Apr 22, 2019

@ikreymer
api refactor: 
- '/crawls' create api accepts all options as POST body params, 'start' indicates if also starting
- '/crawl/<id>/start' api simply starts crawl
- 'cache' enum added to (always, default, never) to indicate aggressive cacheing (every request), default, or no cacheing
- browsertrix_cli: move crawl commands to 'crawls' subcommand and crawls.py to allow other subcommands
- tests: update tests for new api, fix name in test compose
Loading status checks…
6595ca7
@ikreymer
support for browser profiles as per #14 
update configs:
- enable auto fetch for pywb
- enable default fixed pool, no-proxy/live mode for creating profile browser
- profile: list, create, remove
- crawl create: can specify --profile flag, also --coll and --mode overrides
- consistently use 'behavior_time' param for behavior time
Loading status checks…
068f11e
@ikreymer
fix test docker compose 
add install-browsers.sh, use for travis install
Loading status checks…
f6df341
@ikreymer
make PROXY_HOST optional default to pywb for tests
Loading status checks…
d994c3a
@ikreymer
travis: set install-browsers to headless
Loading status checks…
95ff4fd
@ikreymer
lint pass on cli
Loading status checks…
4c4a3b6
@ikreymer
api refactor: update /crawl/<id>/urls to return 'scopes' and 'queue' … 
…with dict instead of dict as string

tests: update tests for api update, remove unused decoding functions
Loading status checks…
cc3c78f

@ikreymer ikreymer merged commit 83ebf10 into master Apr 23, 2019
2 checks passed

2 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details

@ikreymer ikreymer deleted the api-refactor branch Apr 24, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.