Skip to content
Branch: master
Find file History
jrwiebe and ruebot Add method for determining binary file extension. (#349)
This PR implements the strategy described in the discussion of the above issue to get an extension for a file described by a URL and a MIME type. It creates a GetExtensionMime object in the matchbox.

This PR also removes most of the filtering by URL from the image, audio, video, presentation, spreadsheet, and word processor document extraction methods, since these were returning false positives. (CSV and TSV files are a special case, since Tika detects them as "text/plain" based on content.)

Finally, I have inserted toLowerCase into the getUrl.endsWith() filter tests, which could possibly bring in some more CSV and TSV files

* Adds method for getting a file extension from a MIME type.
* Add getExtensions method to DetectMimeTypeTika.
* Matchbox object to get extension of URL
* Use GetExtensionMime for extraction methods; minor fixes.
* Remove tika-parsers classifier
* Remove most filtering by file extension from binary extraction methods; add CSV/TSV special cases.
* Fix GetExtensionMime case where URL has no extension but a MIME type is detected
* Insert `toLowerCase` into `getUrl.endsWith()` calls in io.archivesunleashed.packages; apply to `FilenameUtils.getExtension` in `GetExtensionMime`.
* Remove filtering on URL for audio, video, and images.
* Remove filtering on URL for images; add DF fields to image extraction
* Remove saveImageToDisk and its test
* Remove robots.txt check and extraneous imports
* Close files so we don't get too many files open again.
* Add GetExtensionMimeTest
* Resolve #343
Latest commit 448601e Aug 17, 2019
Permalink
Type Name Latest commit message Commit time
..
Failed to load latest commit information.
main Add method for determining binary file extension. (#349) Aug 18, 2019
test Add method for determining binary file extension. (#349) Aug 18, 2019
You can’t perform that action at this time.