convert_to_utf8 is not converting files with certain characters

[NikKirstein]

I ran the conversion script on this data, and it recognizes that it is not in utf-8 and even produced an output file. However, the output file is still not in utf-8 and the script will not convert them. We think it is due to them containing this character.

This picture shows what the data looks like in it's original encoding: ISO-8859-2

This picture shows what happens when the encoding is changed to utf-8 manually in notepad++

[Gee-CityU]

Hi @NikKirstein
Thanks for checking the encoding. So...
Is ISO-8859-2 type of "Mac OS" encoding because the file is written on (maybe) Mac WPS?
How can we convert it to utf-8 by modifying the script?
Thanks you.

[markfullmer]

Hi @NikKirstein , a couple thoughts:

1. This may not be an encoding issue, but rather a normalization issue. If you run the file(s) through the (recently updated) text_normalization.py script first, the problematic characters may be converted to a more standard character. After that, you may or may not have to run it through the convert_to_utf8 script.
2. Failing that, could you send along one of the problematic files? I'll poke at it a bit.