ruebot/heritrix3 forked from internetarchive/heritrix3

- Add badges to README
- Update Markdown
- Clean-up formatting
- Add LICENSE.txt in root
- Partially addresses internetarchive#233

Add CHANGELOG; address internetarchive#233.

fix some trough dedup bugs

…ler-bean

Remove suffix from warcWriter since it is no longer used.

set of frontier management changes to support CrawlHQ module

* master:
  replace System.err.println with logger.info
  Revert "Upgrade httpclient to 4.5.7 and handle cookies more compliantly"
  Removing outdated test.
  Disable questionalbe test.
  Avoid deprecated flag.
  Supply an iterator, for internetarchive#245
  Updated POM to use latest version.
  Update README.md
  Handle missing closing paren in srcset descriptor
  Teach jericho extractor srcset
  Don't run srcset test against jericho, it doesn't handle it
  Handle commas more compliantly when parsing srcset
  Ensure we start parsing full lines, for internetarchive#239.

These changes come from a private fork of H3, originally made by Kenji
Nagahashi, to create org.archive.crawler.frontier.PullingBdbFrontier,
which we intend to merge into 'contrib' of the official version in the
near future.

especially this:

-    writeUrlCache.remove("segmentId");
+    writeUrlCache.remove(segmentId);

and some improvements and tweaks

…6-upgrade-httpclient

Revert "Upgrade httpclient to 4.5.7 and handle cookies more compliantly"

Upgrade httpclient to 4.5.7 and handle cookies more compliantly

Trough dedup

Handle commas more compliantly when parsing srcset

... configure using rethinkdb url and segment id, instead of write url,
which means it can work if the segment gets reassigned and so forth
***backward incompatible change***

Update README.md

Fix typo in link

Commas are allowed if they're in the middle of the URL. Consequently:

    srcset="a,b,,c,"   => ["a,b,,c"]
    srcset="a, b,, c," => ["a", "b", "c"]

They occur particularly commonly in data: URLs before the base64 value.

Commas are also allowed in descriptors if they are enclosed by parens:

    srcset="a (b,c),d" => ["a", "d"]

Spec: https://html.spec.whatwg.org/multipage/images.html#parsing-a-srcset-attribute