views:

4467

answers:

9

In a text file, there is a string "I don't like this".

However, when I read it into a string, it becomes "I don\xe2\x80\x98t like this". I understand that \u2018 is the unicode representation of "'". I use

f1 = open (file1, "r")
text = f1.read()

command to do the reading.

Now, is it possible to read the string in such a way that when it is read into the string, it is "I don't like this", instead of "I don\xe2\x80\x98t like this like this"?

Second edit: I have seen some people use mapping to solve this problem, but really, is there no built-in conversion that does this kind of ANSI to unicode ( and vice versa) conversion?

+2  A: 

Actually, U+2018 is the Unicode representation of the special character ‘ . If you want, you can convert instances of that character to U+0027 with this code:

text = text.replace (u"\u2018", "'")

In addition, what are you using to write the file? f1.read() should return a string that looks like this:

'I don\xe2\x80\x98t like this'

If it's returning this string, the file is being written incorrectly:

'I don\u2018t like this'
John Millikin
Sorry! As you said, it is returning 'I don\xe2\x80\x98t like this'
Ngu Soon Hui
The 'I don\xe2\x80\x98t like this' that you're seeing is what Python would call a str. It appears to be the utf-8 encoding of u'I don\u2018t like this', which is a unicode instance in Python. Try calling .decode('utf-8') on the former or .encode('utf-8') on the latter.
Logan
U+0039 is the digit 9.
hop
@hop: oops, forgot ord() returns decimal instead of hex. Thank you for the catch.
John Millikin
A: 

This is Pythons way do show you unicode encoded strings. But i think you should be able to print the string on the screen or write it into a new file without any problems.

>>> test = u"I don\u2018t like this"
>>> test
u'I don\u2018t like this'
>>> print test
I don‘t like this
xardias
+14  A: 

Ref: http://www.amk.ca/python/howto/unicode

Reading Unicode from a file is therefore simple:

import codecs
f = codecs.open('unicode.rst', encoding='utf-8')
for line in f:
    print repr(line)

It's also possible to open files in update mode, allowing both reading and writing:

f = codecs.open('test', encoding='utf-8', mode='w+')
f.write(u'\u4500 blah blah blah\n')
f.seek(0)
print repr(f.readline()[:1])
f.close()

EDIT: I'm assuming that your intended goal is just to be able to read the file properly into a string in Python. If you're trying to convert to an ASCII string from Unicode, then there's really no direct way to do so, since the Unicode characters won't necessarily exist in ASCII.

If you're trying to convert to an ASCII string, try one of the following:

  1. Replace the specific unicode chars with ASCII equivalents, if you are only looking to handle a few special cases such as this particular example

  2. Use the unicodedata module's normalize() and the string.encode() method to convert as best you can to the next closest ASCII equivalent (Ref http://techxplorer.com/2006/07/18/converting-unicode-to-ascii-using-python/):

    >>> teststr

    u'I don\xe2\x80\x98t like this'

    >>> unicodedata.normalize('NFKD', teststr).encode('ascii', 'ignore')

    'I donat like this'

Jay
+4  A: 

But it really is "I don\u2018t like this" and not "I don't like this". The character u'\u2018' is a completely different character than "'" (and, visually, should correspond more to '`').

If you're trying to convert encoded unicode into plain ASCII, you could perhaps keep a mapping of unicode punctuation that you would like to translate into ASCII.

punctuation = {
  u'\u2018': "'",
  u'\u2019': "'",
}
for src, dest in punctuation.iteritems():
  text = text.replace(src, dest)

There are an awful lot of punctuation characters in unicode, however, but I suppose you can count on only a few of them actually being used by whatever application is creating the documents you're reading.

Logan
actually, if you make the dict map Unicode ordinals to Unicode ordinals ({0x2018: 0x27, 0x2019: 0x27}) you can just pass the entire dict to text.translate() to do all the replacing in one go.
Thomas Wouters
A: 

Some comments: I have seen some people use mapping to solve this problem, but really, is there no built-in conversion that does this kind of ANSI to unicode ( and vice versa) conversion? Thanks!

Ngu Soon Hui
There's not, because there are hundreds of thousands of Unicode code points. How would you decide which should be mapped to what ASCII characters?
John Millikin
john, your comment is wrong, at least in the general sense. the iconv lib can be used to transliterate unicode characters to ascii (even locale dependent.$ python -c 'print u"\u2018".encode("utf-8")' | iconv -t 'ascii//translit' | xxd0000000: 270a
hop
+3  A: 

There are a few points to consider.

A \u2018 character may appear only as a fragment of representation of a unicode string in Python, e.g. if you write:

>>> text = u'‘'
>>> print repr(text)
u'\u2018'

Now if you simply want to print the unicode string prettily, just use unicode's encode method:

>>> text = u'I don\u2018t like this'
>>> print text.encode('utf-8')
I don‘t like this

To make sure that every line from any file would be read as unicode, you'd better use the codecs.open function instead of just open, which allows you to specify file's encoding:

>>> import codecs
>>> f1 = codecs.open(file1, "r", "utf-8")
>>> text = f1.read()
>>> print type(text)
<type 'unicode'>
>>> print text.encode('utf-8')
I don‘t like this
DzinX
+1  A: 

There is a possibility that somehow you have a non-unicode string with unicode escape characters, e.g.:

>>> print repr(text)
'I don\\u2018t like this'

This actually happened to me once before. You can use a unicode_escape codec to decode the string to unicode and then encode it to any format you want:

>>> uni = text.decode('unicode_escape')
>>> print type(uni)
<type 'unicode'>
>>> print uni.encode('utf-8')
I don‘t like this
DzinX
+1  A: 

Leaving aside the fact that your text file is broken (U+2018 is a left quotation mark, not an apostrophe): iconv can be used to transliterate unicode characters to ascii.

You'll have to google for "iconvcodec", since the module seems not to be supported anymore and I can't find a canonical home page for it.

>>> import iconvcodec
>>> from locale import setlocale, LC_ALL
>>> setlocale(LC_ALL, '')
>>> u'\u2018'.encode('ascii//translit')
"'"

Alternatively you can use the iconv command line utility to clean up your file:

$ xxd foo
0000000: e280 980a                                ....
$ iconv -t 'ascii//translit' foo | xxd
0000000: 270a                                     '.
hop
A: 

You should take a look at the unicodedata module:

http://log.vaem.net/2008/06/python-translit-remove-accent.html

iconvcodec is dead and during my searches on google, I found that unicodedata was the answer.

Grégoire Cachet