Convert Unicode to String in Python (containing extra symbols)

Question

How to you convert a Unicode string (containing extra characters like £ $, etc) into a python string?

Answer 1

title = u"Klüft skräms inför på fédéral électoral große"
import unicodedata
unicodedata.normalize('NFKD', title).encode('ascii','ignore')
'Kluft skrams infor pa federal electoral groe'

Answer 2

Here is an example:

>>> u = u'€€€'
>>> s = u.encode('utf8')
>>> s
'\xe2\x82\xac\xe2\x82\xac\xe2\x82\xac'

Answer 3

Well, if you're willing/ready to switch to Python 3 (which you may not be due to the backwards incompatibility with some Python 2 code), you don't have to do any converting; all text in Python 3 is represented with Unicode strings, which also means that there's no more usage of the u'<text>' syntax. You also have what are, in effect, strings of bytes, which are used to represent data (which may be an encoded string).

http://docs.python.org/3.1/whatsnew/3.0.html#text-vs-data-instead-of-unicode-vs-8-bit

(Of course, if you're currently using Python 3, then the problem is likely something to do with how you're attempting to save the text to a file.)

Answer 4

We need to know what Python version you are using, and what it is that you are calling a Unicode string.

Do the following on a short unicode_string that includes the currency symbols that are causing the bother:

Python 2.x : print type(unicode_string), repr(unicode_string)

Python 3.x : print type(unicode_string), ascii(unicode_string)

Then edit your question and copy/paste the results of the above print statement. DON'T retype the results.

Also look up near the top of your HTML and see if you can find something like this:

<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />

Tell us what yours says after charset=.

Then you stand a chance of getting meaningful answers.

Answer 5

If you have a unicode string, and you want to write this to a file, or other serialised form, you must first encode it into a particular representation that can be stored. There are several common unicode encodings, such as utf-16 (uses 2 bytes for most unicode characters) or utf-8 (1-4 bytes / codepoint depending on the character) etc. To convert that string into a particular encoding, you can use:

>>> s= u'£10"
>>> s.encode('utf8')
'\xc2\x9c10'
>>> s.encode('utf16')
'\xff\xfe\x9c\x001\x000\x00'

This raw string of bytes can be written to a file. However note that when reading it back, you must know what encoding it is in and decode it using that same encoding.

When writing to files, you can get rid of this manual encode / decode process by using the codecs module. So, to open a file that encodes all unicode strings into utf8, use:

import codecs
f = codecs.open('path/to/file.txt','w','utf8')
f.write(my_unicode_string)  # Stored on disk as UTF8

Do note that anything else that is using these files must understand what encoding the file is in if they want to read them. If you are the only one doing the reading/writing this is no problem, otherwise make sure that you write in a form understandable by whatever else uses the files.

In python 3, this form of file access is the default, and the builtin open function will take an encoding parameter and always translate to/from unicode strings (the default string object in python3) for files opened in text mode.

Answer 6

You can use encode to ASCII if you don't need to translate the non ASCII chars:

>>> a=u"aaaàçççñññ"
>>> type(a)
<type 'unicode'>
>>> a.encode('ascii','ignore')
'aaa'
>>> a.encode('ascii','replace')
'aaa???????'
>>>