Pickled file won't load on Mac/Linux

Question

I have an application that imports data from a pickled file. It works just fine in Windows but Mac and Linux behaviour is odd.

In OS X, the pickled file (file extension ".char") is unavailable as a selection unless I set the file type to *.*. Then, if I select a file that has the .char extension, it won't load, giving the error

unpickle_file = cPickle.load(char_file)

ValueError: could not convert string to float

However, if I create a file that doesn't have the .char extension, that file will load up just fine.

In Linux, when I use the "file open" dialog, my pickled files aren't visible, whether or not they have a file extension. However, I can see them under Nautilus or Dolphin. They simply don't exist to my application though.

---

Edit Here's the save code:

def createSaveFile(self):
        """Create the data files to be saved and save them.

        Creates a tuple comprised of a dictionary of general character information
        and the character's skills dictionary."""
        if self.file_name:
            self.save_data = ({'Name':self.charAttribs.name,

              <snip> 

                self.charAttribs.char_skills_dict)
            self.file = open(self.file_name, 'w')
            cPickle.dump(self.save_data, self.file)
        self.file.close()

Here's the open code:

 def getCharFile(self, event): # wxGlade: CharSheet.<event_handler>
        """Retrieve pickled character file from disk."""
        wildcard = "Character files (*.char) | *.char | All files (*.*) | *.*"        
        openDialog = wx.FileDialog(None, "Choose a character file", os.getcwd(),
        "", wildcard, wx.OPEN | wx.CHANGE_DIR)
        if openDialog.ShowModal() == wx.ID_OK:
            self.path = openDialog.GetPath()
        try:
            char_file =  open(self.path, "r")
            unpickle_file = cPickle.load(char_file)
            char_data, char_skills = unpickle_file
            self.displayCharacter(char_data, char_skills)
        except IOError:
            self.importError = wx.MessageDialog(self, 
            "The character file is not available!",
            "Character Import Error", wx.OK | wx.ICON_ERROR)
            self.importError.ShowModal()
            self.importError.Destroy()
            openDialog.Destroy()

Answer 1

Probably you didn't open the file in binary mode when writing and/or reading the pickled data. In this case newline format conversion will occur, which can break the binary data.

To open a file in binary mode you have to provide "b" as part of the mode string:

char_file = open('pickle.char', 'rb')

Answer 2

As mentioned by Adam, the problem is likely to be the newline format of the pickle file.

Unfortunately, the real problem is actually caused on save rather than load. This may be recoverable if you're using text mode pickles, rather than binary. Try opening the file in universal newline mode, which will cause python to guess what the right line-endings are ie:

char_file=open('filename.char','rU')

However, if you're using a binary format (cPickle.dump(file, 1)) you may have an unrecoverably corrupted pickle (even when loading in Windows) - if you're lucky and no \r\n characters show up then it may work, but as soon as this occurs you could end up with corrupted data, as there's no way to distinguish between a "real" \r\n code and one windows has inserted on seeing just \n.

The best way to handle things to be loaded in multiple platforms is to always save in binary mode. On your windows machine, when saving the pickle use:

char_file = open('filename.char','wb')
cPickle.dumps(data, char_file)

Answer 3

self.file = open(self.file_name, 'w')

Should be:

self.file = open(self.file_name, 'wb')

In your createSaveFile function, to save the file in binary mode (rather than text mode). You should also make sure you open the file in binary mode as well (rb).

If you don't use binary mode then Windows will convert all new-lines to \r\n and will effectively corrupt the file (at least as far as other OS's are concerned).