Converting from ascii to utf-8 with Python

Question

Hello.

I have xmpp bot written in python. One of it's plugins is able to execute OS commands and send output to the user. As far as I know output should be unicode-like to send it over xmpp protocol. So I tried to handle it this way:

output = os.popen(cmd).read() 
if not isinstance(output, unicode):
   output = unicode(output,'utf-8','ignore')
bot.send(xmpp.Message(mess.getFrom(),output))

But when Russian symbols appear in output they aren't converted well.

sys.getdefaultencoding() 

says that default command prompt encoding is 'ascii', but when I try to do

output.decode('ascii') 

in python console I get

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0x92 in position 1: 
ordinal not in range(128)

OS: Win XP, Python 2.5.4 PS: Sorry for my English :(

Answer 1

Ascii has no defined character values above 127 0x7F. Perhaps you mean the Cyrillic code page? It's 866

See http://en.wikipedia.org/wiki/Code_page

edit: since this answer was marked correct presumably 886 worked, but as other answers have pointed it, 886 is not the only Russian language code page. If you use a code page different from the one that was used when the Russian symbols were encoded, you will get the wrong result.

Answer 2

sys.getdefaultencoding() returns python's default encoding - which is ASCII unless you have changed it. ASCII doesn't support Russian characters.

You need to work out what encoding the actual text is, either manually, or using the locale module.

Typically something like:

import locale
encoding = locale.getpreferredencoding(do_setlocale=True)¶

Answer 3

In Python 'cp855', 'cp866', 'cp1251', 'iso8859_5', 'koi8_r' are differing Russian code pages. You'll need to use the right one to decode the output of popen. In the Windows console, the 'chcp' command lists the code page used by console commands. That won't necessarily be the same code page as Windows applications. On US Windows, 'cp437' is used for the console and 'cp1252' is used for applications like Notepad.

Answer 4

You say """sys.getdefaultencoding() says that default command prompt encoding is 'ascii'"""

sys.getdefaultencoding says NOTHING about the "command prompt" encoding.

On Windows, sys.stdout.encoding should do the job. On my machine, it contains cp850 when Python is run in a Command Prompt window, and cp1252 in IDLE. Yours should contain cp866 and cp1251 respectively.

Update You say that you still need cp866 in IDLE. Note this:

IDLE 2.6.4      
>>> import os
>>> os.popen('chcp').read()
'Active code page: 850\n'
>>>

So when your app starts up, check if you are on Windows and if so, parse the result of os.popen('chcp').read(). The text before the : is probably locale-dependent. codepage = result.split()[-1] may be good enough "parsing". On Unix, which doesn't have a Windows/MS-DOS split personality, sys.stdout.encoding should be OK.