Unicode filenames on Windows with Python & subprocess.Popen()

Question

Why does the following occur:

>>> u'\u0308'.encode('mbcs')   #UMLAUT
'\xa8'
>>> u'\u041A'.encode('mbcs')   #CYRILLIC CAPITAL LETTER KA
'?'
>>>

I have a Python application accepting filenames from the operating system. It works for some international users, but not others.

For example, this unicode filename: u'\u041a\u0433\u044b\u044b\u0448\u0444\u0442'

will not encode with Windows 'mbcs' encoding (the one used by the filesystem, returned by sys.getfilesystemencoding()). I get '???????', indicating the encoder fails on those characters. But this makes no sense, since the filename came from the user to begin with.

Update: Here's the background to my reasons behind this... I have a file on my system with the name in Cyrillic. I want to call subprocess.Popen() with that file as an argument. Popen won't handle unicode. Normally I can get away with encoding the argument with the codec given by sys.getfilesystemencoding(). In this case it won't work

Answer 1

Docs for sys.getfilesystemencoding() say that for Windows NT and later, file names are natively Unicode. If you have a valid unicode file name, why would you bother encoding it using mbcs?

Docs for codecs module say that mbcs encodes using "ANSI code page" (which will differ depending on user's locale) so if the locale doesn't use Cyrillic characters, splat.

Edit: So your process is calling subprocess.Popen(). If your invoked process is under your control, the two processes ahould be able to agree to use UTF-8 as the Unicode Transport Format. Otherwise, you may need to ask on the pywin32 mailing list. In any case, edit your question to state the degree of control you have over the invoked process.

Answer 2

If you need to pass the name of an existing file, then you might have a better chance of success by passing the 8.3 version of the Unicode filename.

You need to have the pywin32 package installed, then you can do:

>>> import win32api
>>> win32api.GetShortPathName(u"C:\\Program Files")
'C:\\PROGRA~1'

I believe these short filenames use only ASCII characters, and therefore you should be able to use them as arguments to a command line.

Should you need to specify also filenames to be created, you can create them with zero size in advance from Python using Unicode filenames, and pass the short name of the file as an argument.

UPDATE: user bogdan says correctly that 8.3 filename generation can be disabled (I had it disabled, too, when I had Windows XP on my laptop), so you can't rely on them. So, as another more far-fetched approach when working on NTFS volumes, one can hard link the Unicode filenames to plain ASCII ones; pass the ASCII filenames to an external command and delete them afterwards.