Python: How do I force iso-8859-1 file output?

Question

How do I force Latin-1 (which I guess means iso-8859-1?) file output in Python?

Here's my code at the moment. It works, but trying to import the resulting output file into a Latin-1 MySQL table produces weird encoding errors.

outputFile = file( "textbase.tab", "w" )
for k, v in textData.iteritems():
    complete_line = k + '~~~~~' + v + '~~~~~' + " ENDOFTHELINE"
    outputFile.write(complete_line)
    outputFile.write( "\n" )
outputFile.close()

The resulting output file seems to be saved in "Western (Mac OS Roman)", but if I then save it in Latin-1, I still get strange encoding problems. How can I make sure that the strings used, and the file itself, are all encoded in Latin-1 as soon as they are generated?

The original strings (in the textData dictionary) have been parsed in from an RTF file - I don't know if that makes a difference.

I'm a bit new to Python and to encoding generally, so apologies if this is a dumb question. I have tried looking at the docs but haven't got very far.

I'm using Python 2.6.1.

Answer 1

I think it's just:

outputFile = file( "textbase.tab", "wb" )
for k, v in textData.iteritems():
    complete_line = k + '~~~~~' + v + '~~~~~' + " ENDOFTHELINE"
    outputFile.write((complete_line + "\n").encode("iso-8859-1"))
    outputFile.close()

As you alluded to, you need to make sure you are decoding the RTF file correctly too. For this to work, k and v should be unicode objects.

Answer 2

The main problem here is that you don't know what encoding your data is in. If we assume you are correct in that your file ends up being in Mac OS Roman, then you need to decode the data to unicode first, and then encode it as iso-8859-1.

inputFile = open("input.rtf", "rb") # The b flag is just a marker in Python 2.
data = inputFile.read().decode('mac_roman')
textData = yourparsefunctionhere(data)

outputFile = open( "textbase.tab", "wb" ) # don't use file()
for k, v in textData.iteritems():
    complete_line = k + '~~~~~' + v + '~~~~~' + " ENDOFTHELINE"
    outputFile.write((complete_line + "\n").encode("iso-8859-1"))
    outputFile.close()

But I wouldn't be surprised, since it's RTF, if it's Windows encoded, so you might want to try that too. I don't know how RTF specifies the encoding.

Answer 3

Simply use the codecs module for writing the file:

import codecs
outputFile = codecs.open("textbase.tab", "w", "ISO-8859-1")

Of course, the strings you write have to be Unicode strings (type unicode), they won't be converted if they are plain str objects (which are basically just arrays of bytes). I guess you are reading the RTF file with the normal Python file object as well, so you might have to convert that to using codecs.open as well.