UTF-8 file names decoding/encoding issue

[edouardhue]

With @symac's patched CommonsDownloader version of October 1st (see mail discussions), resulting file names on the filesystem are not properly encoded : two bytes UTF-8 chars (like é), that are properly encoded in the file list (Abbaye Saint-Pierre de Marcilhac-sur-Célé - Eglise.JPG,99999, get translated to two one byte chars, like in Abbaye_Saint-Pierre_de_Marcilhac-sur-Célé_-_Eglise. On the console, output is misencoded too, but not in the same way : Downloading Abbaye_Saint-Pierre_de_Marcilhac-sur-C├®l├®_-_Eglise.JPG.

Running with Python 2.7.8 in PowerShell under Windows 8.1 Pro N.

[JeanFred]

Quick answer on one point:

With @symac's patched CommonsDownloader version of October 1st (see mail discussions)

@symac edits were merged in #5 and #6

[JeanFred]

file names on the filesystem are not properly encoded : two bytes UTF-8 chars (like é), that are properly encoded in the file list (Abbaye Saint-Pierre de Marcilhac-sur-Célé - Eglise.JPG,99999, get translated to two one byte chars, like in Abbaye_Saint-Pierre_de_Marcilhac-sur-Célé_-_Eglise

Can’t reproduce with Python 2.7.3 under Ubuntu 12.04 Needs further investigation for Windows

[JeanFred]

On the console, output is misencoded too, but not in the same way : Downloading Abbaye_Saint-Pierre_de_Marcilhac-sur-C├®l├®_-_Eglise.JPG

Can’t reproduce either. Would not that rather be an issue with your Terminal which does not properly support Unicode @edouardhue ?

[edouardhue]

Garbage on the console is probably a terminal issue: it is the same with [http://github.com/edouardhue/commons-downloader]. But I don't have any issue with file names in the filesystem.