Basic data storage with Python

Question

I need to store basic data of customer's and cars that they bought and payment schedule of these cars. These data come from GUI, written in Python. I don't have enough experience to use a database system like sql, so I want to store my data in a file as plain text. And it doesn't have to be online.

To be able to search and filter them, first I convert my data (lists of lists) to the string then when I need the data re-convert to the regular Python list syntax. I know it is a very brute-force way, but is it safe to do like that or can you advice me to another way?

Answer 1

You can use this lib to write an object into a file http://docs.python.org/library/pickle.html

Answer 2

It is never safe to save your database in a text format (or using pickle or whatever). There is a risk that problems while saving the data may cause corruption. Not to mention risks with your data being stolen.

As your dataset grows there may be a performance hit.

have a look at sqlite (or sqlite3) which is small and easier to manage than mysql. Unless you have a very small dataset that will fit in a text file.

P/S: btw, using berkeley db in python is simple, and you don't have to learn all the DB things, just import bsddb

Answer 3

The answer to use pickle is good, but I personally prefer shelve. It allows you to keep variables in the same state they were in between launches and I find it easier to use than pickle directly. http://docs.python.org/library/shelve.html

Answer 4

Writing data in a file isn't a safe way for datastorage. Better use a simple database libary like sqlalchemy. It is a ORM for easy database usage...

Answer 5

You can also keep simple data in plain text file. Then you have not much support, however, to check consistency of data, double values etc.

Here is my simple 'card file' type data in text file code snippet using namedtuple so that you can access values not only by index in line but by they header name:

# text based data input with data accessible
# with named fields or indexing
from __future__ import print_function ## Python 3 style printing
from collections import namedtuple
import string

filein = open("sample.dat")

datadict = {}

headerline = filein.readline().lower() ## lowercase field names Python style
## first non-letter and non-number is taken to be the separator
separator = headerline.strip(string.lowercase + string.digits)[0]
print("Separator is '%s'" % separator)

headerline = [field.strip() for field in headerline.split(separator)]
Dataline = namedtuple('Dataline',headerline)
print ('Fields are:',Dataline._fields,'\n')

for data in filein:
    data = [f.strip() for f in data.split(separator)]
    d = Dataline(*data)
    datadict[d.id] = d ## do hash of id values for fast lookup (key field)

## examples based on sample.dat file example
key = '123'
print('Email of record with key %s by field name is: %s' %
      (key, datadict[key].email))

## by number
print('Address of record with key %s by field number is: %s' %
      (key ,datadict[key][3]))

## print the dictionary in separate lines for clarity
for key,value in  datadict.items():
    print('%s: %s' % (key, value))

input('Ready') ## let the output be seen when run directly

""" Output:
Separator is ';'
Fields are: ('id', 'name', 'email', 'homeaddress') 

Email of record with key 123 by field name is: [email protected]
Address of record with key 123 by field number is: 456 happy st.
345: Dataline(id='345', name='tony', email='[email protected]', homeaddress='Espoo Finland')
123: Dataline(id='123', name='gishi', email='[email protected]', homeaddress='456 happy st.')
Ready
"""

Answer 6

I agree with the others that serious and important data would be more secure in some type of light database but can also feel sympathy for the wish to keep things simple and transparent.

So, instead of inventing your own text-based data-format I would suggest you use YAML

The format is human-readable for example:

List of things:
    - Alice
    - Bob
    - Evan

You load the file like this:

>>> import yaml
>>> file = open('test.yaml', 'r')
>>> list = yaml.load(file)

And list will look like this:

{'List of things': ['Alice', 'Bob', 'Evan']}

Of course you can do the reverse too and save data into YAML, the docs will help you with that.

At least another alternative to consider :)