Simple data storing in Python

Question

I'm looking for a simple solution using Python to store data as a flat file, such that each line is a string representation of an array that can be easily parsed.

I'm sure python has library for doing such a task easily but so far all the approaches I have found seemed like it would have been sloppy to get it to work and I'm sure there is a better approach. So far I've tried:

• the array.toFile() method but couldn't figure out how to get it to work with nested arrays of strings, it seemed geared towards integer data.
• Lists and sets do not have a toFile method built in, so I would have had to parse and encode it manually.
• CSV seemed like a good approach but this would also require manually parsing it, and did not allow me to simply append new lines at the end - so any new calls the the CSVWriter would overwrite the file existing data.

I'm really trying to avoid using databases (maybe SQLite but it seems a bit overkill) because I'm trying to develop this to have no software prerequisites besides Python.

Answer 1

http://docs.python.org/library/pickle.html

Answer 2

Must the file be human readable? If not, shelve is really easy to use.

Answer 3

I don't have enough reputation points to up vote Jeremy, but pickle is what you're looking for.

Answer 4

In addition to pickle (mentioned above), there's json (built in to 2.6, available via simplejson before that), and marshal. Also, there's a reader in the same csv module the writer is in.

UPDATE: As S. Lott pointed out in a comment, there's also YAML, available via PyYAML, among others.

Answer 5

I'm looking for a simple solution using Python to store data as a flat file, such that each line is a string representation of an array that can be easily parsed.

Is the data only ever going to be parsed by Python programs? If not, then I'd avoid pickle et al (shelve and marshal) since they're very Python specific. JSON and YAML have the important advantage that parsers are easily available for most any language.

Answer 6

This solution at SourceForge uses only standard Python modules:

y_serial.py module :: warehouse Python objects with SQLite

"Serialization + persistance :: in a few lines of code, compress and annotate Python objects into SQLite; then later retrieve them chronologically by keywords without any SQL. Most useful "standard" module for a database to store schema-less data."

http://yserial.sourceforge.net

SQLite is not "overkill" at all -- you will be amazed how simple it is; plus it solves more general data persistance issues.