Pickle vs output to a file in python

Question

I have a program that outputs some lists that I want to store to work with later. For example, suppose it outputs a list of student names and another list of their midterm scores. I can store this output in the following two ways:

Standard File Output way:

newFile = open('trialWrite1.py','w')
newFile.write(str(firstNames))
newFile.write(str(midterm1Scores))
newFile.close()

The pickle way:

newFile = open('trialWrite2.txt','w')
cPickle.dump(firstNames, newFile)
cPickle.dump(midterm1Scores, newFile)
newFile.close()

Which technique is better or preferred? Is there an advantage of using one over the other?

Thanks

Answer 1

pickle is more generic -- it allows you to dump many different kinds of objects to a file for later use. The downside is that the interim storage is not very human-readable, and not in a standard format.

Writing strings to a file, on the other hand, is a much better interface to other activities or code. But it comes at the cost of having to parse the text back into your Python object again.

Both are fine for this simple (list?) data; I would use write( firstNames ) simply because there's no need to use pickle. In general, how to persist your data to the filesystem depends on the data!

---

For instance, pickle will happily pickle functions, which you can't do by simply writing the string representations.

>>> data = range
<class 'range'>
>>> pickle.dump( data, foo )
# stuff
>>> pickle.load( open( ..., "rb" ) )
<class 'range'.

Answer 2

I think the csv module might be a good fit here, since CSV is a standard format that can be both read and written by Python (and many other languages), and it's also human-readable. Usage could be as simple as

with open('trialWrite1.py','wb') as fileobj:
    newFile = csv.writer(fileobj)
    newFile.writerow(firstNames)
    newFile.writerow(midterm1Scores)

However, it'd probably make more sense to write one student per row, including their name and score. That can be done like this:

from itertools import izip
with open('trialWrite1.py','wb') as fileobj:
    newFile = csv.writer(fileobj)
    for row in izip(firstNames, midterm1Scores):
        newFile.writerow(row)

Answer 3

For a completely different approach, consider that Python ships with SQLite. You could store your data in a SQL database without adding any third-party dependencies.