views:

121

answers:

2

I have a string that looks and behaves as follows (Python code provided). WTF?! What encoding is it in?

s = u'\x00Q\x00u\x00i\x00c\x00k'
>>> print s
Quick
>>>
>>> s == 'Quick'
False
>>>
>>> import re
>>> re.search('Quick', s)
>>>
>>> import chardet
>>> chardet.detect(s)
/usr/lib/pymodules/python2.6/chardet/universaldetector.py:69: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
  if aBuf[:3] == '\xEF\xBB\xBF':
/usr/lib/pymodules/python2.6/chardet/universaldetector.py:72: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
  elif aBuf[:4] == '\xFF\xFE\x00\x00':
/usr/lib/pymodules/python2.6/chardet/universaldetector.py:75: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
  elif aBuf[:4] == '\x00\x00\xFE\xFF':
/usr/lib/pymodules/python2.6/chardet/universaldetector.py:78: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
  elif aBuf[:4] == '\xFE\xFF\x00\x00':
/usr/lib/pymodules/python2.6/chardet/universaldetector.py:81: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
  elif aBuf[:4] == '\x00\x00\xFF\xFE':
/usr/lib/pymodules/python2.6/chardet/universaldetector.py:84: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
  elif aBuf[:2] == '\xFF\xFE':
/usr/lib/pymodules/python2.6/chardet/universaldetector.py:87: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
  elif aBuf[:2] == '\xFE\xFF':
{'confidence': 1.0, 'encoding': 'ascii'}
>>>
>>> chardet.detect(s)
{'confidence': 1.0, 'encoding': 'ascii'}
>>> 
+8  A: 

UTF-16 big endian

Delan Azabani
YES! Awesome, thanks a lot. Now I do *s.decode('utf-16-be')* to decode it and it's all fine.
ionut bizau
No problem; good luck!
Delan Azabani
+1  A: 

You have UTF-16BE without a BOM. As documented, chardet doesn't grok UTF-nnxE without a BOM.

>>> s = '\x00Q\x00u\x00i\x00c\x00k' #### Note: dropping the spurious `u` prefix
>>> s.decode('utf_16be')
u'Quick'
>>>

chardet is also not smart enough to raise a DontBeSilly exception if you feed it unicode :-)

John Machin