UnicodeDecodeError when using socket.gethostname() result

Question

Some of my users report that the following code may raise a UnicodeDecodeError when the hostname contains non-ascii characters (however I haven't been able to replicate this on my Windows Vista machine):

    self.path = path
    self.lock_file = os.path.abspath(path) + ".lock"
    self.hostname = socket.gethostname()
    self.pid = os.getpid()
    dirname = os.path.dirname(self.lock_file)
    self.unique_name = os.path.join(dirname, "%s.%s" % (self.hostname, self.pid))

The last part of the traceback is:

    File "taskcoachlib\thirdparty\lockfile\lockfile.pyo", line 537, in FileLock
    File "taskcoachlib\thirdparty\lockfile\lockfile.pyo", line 296, in __init__
    File "taskcoachlib\thirdparty\lockfile\lockfile.pyo", line 175, in __init__
    File "ntpath.pyo", line 102, in join
    UnicodeDecodeError: 'ascii' codec can't decode byte 0xcf in position 7: ordinal not in range(128)

Any ideas on why and how to prevent it?

(The exception occurs with Python 2.5 on Windows XP)

Answer 1

Yes, if either the hostname or the dirname is a unicode string, it is likely to give you that error. The best solution is typically to make sure both are unicode, and not just one of them.

Answer 2

You want a unique string based on the hostname, but it's got Unicode characters in it. There are a variety of ways to reduce a Unicode string to an ascii string, depending on how you want to deal with non-ascii characters. Here's one:

self.hostname = socket.gethostname().encode('ascii', 'replace').replace('?', '_')

This will replace all non-ascii characters with a question mark, then change those to underscore (since file systems don't like questions marks in file names).

Answer 3

I don't think that there is a problem with the actual code that you've posted, even if socket.gethostname() returns a unicode object. There will be a problem when you attempt to use name such that it is converted to a string first:

import os
hostname = u'\u1306blah'
pid = os.getpid()
name = os.path.join(os.path.dirname('/tmp/blah.lock'), "%s.%s" % (hostname, pid))

>>> type(name)
<type 'unicode'>

>>> name
u'/tmp/\u1306blah.28292'

>>> print name
/tmp/ጆblah.29032

>>> str(name)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character u'\u1306' in position 5: ordinal not in range(128)

You can see that str(name) raises the exception that you're seeing, but everything looks OK up until that point. What are you doing with name once you've constructed it?

Answer 4

I don't think gethostname() is necessarily giving you a unicode object. It could be the directory name of lockfile. Regardless, one of them is a standard string with a non-ASCII (higher than 127) char in it and the other is a unicode string.

The problem is that the join function in the ntpath module (the module Python uses for os.path on Windows) attempts join the arguments given. This causes Python to try to convert the normal string parts to unicode. In your case the non-unicode string appears to have a non-ASCII character. This can't be reliably converted to unicode, so Python raises the exception.

A simpler way to trigger the problem:

>> from ntpath import join
>> join(u'abc', '\xff')
---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)

/home/msmits/<ipython console> in <module>()

/usr/lib64/python2.6/ntpath.pyc in join(a, *p)
    106                     path += b
    107                 else:
--> 108                     path += "\\" + b
    109             else:
    110                 # path is not empty and does not end with a backslash,

The traceback shows the problem line in ntpath.py.

You could work around this by using converting the args to join() to standard strings first as other answers suggest. Alternatively you could convert everything to unicode first. If a specific encoding is given to decode() high bytes can be converted to unicode.

For example:

>> '\xff'.decode('latin-1')
u'\xff'