Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.
Sign upEncoding management #428
Encoding management #428
Comments
This comment has been minimized.
This comment has been minimized.
@alxdrdelaporte thanks for creating the issue! There was an issue template, and we use that to collect some very useful information to troubleshoot the issue. Can you please provide some more information: Environment information
I have some sample WARC/ARCs from BAnQ that should help me troubleshoot, but if you could provide one that you're working with, that would be very helpful. |
This comment has been minimized.
This comment has been minimized.
@alxdrdelaporte this make sense? That's Spark 2.4.5 with the 0.50.0 release. What needs to be sorted out:
All that said, this is all web archives, and crazy stuff from the web end up in them. Text encoding is definitely going to be a giant reemerging thorn. |
This comment has been minimized.
This comment has been minimized.
Thanks for your answer @ruebot (and sorry for not following the issue template, I didn't see there was one) As my workplace is in lockdown until further notice all projects are on hold and I am not able to provide more information for now. I will as soon as I can (the only thing I will not be able to provide at all is a sample WARC file, because duplicating or processing data outside of the premises is strictly forbidden). |
alxdrdelaporte commentedMar 12, 2020
I am currently working on a project involving content extraction from a certain number of WARC archives.
I use Archives Unleashed Toolkit to extract plain HTML content (script below) and it works very well.
However I encounter a problem regarding the output file produced by this script: the webpages (mostly in French) extracted from the WARC archive do not all have UTF-8 as their original encoding, so some characters are not rendered properly in the output (and replaced by �).
For example :
nouvelle page consacr�e au 114�me bataillon de marche de chasseurs alpins
As my research is mainly based on lexicon analysis, proper encoding and character rendering are an important matter.
I would be great if there was a way to deal with non-UTF encodings (such as windows-1252 or latin1) while extracting with AUT.