How to save big "database-like" class in python

Question

Hi there,

I'm doing a project with reasonalby big DataBase. It's not a probper DB file, but a class with format as follows: DataBase.Nodes.Data=[[] for i in range(1,1000)] f.e. this DataBase is all together something like few thousands rows. Fisrt question - is the way I'm doing efficient, or is it better to use SQL, or any other "proper" DB, which I've never used actually. And the main question - I'd like to save my DataBase class with all record, and then re-open it with Python in another session. Is that possible, what tool should I use? cPickle - it seems to be only for strings, any other?

In matlab there's very useful functionality named save workspace - it saves all Your variables to a file that You can open at another session - this would be vary useful in python!

Answer 1

Pickle (cPickle) can handle any (picklable) Python object. So as long, as you're not trying to pickle thread or filehandle or something like that, you're ok.

Answer 2

Pickle should be able to serialise the data for you so that you can save it to file.

Alternatively if you don't need the features of a full featured RDBMS you could use a lightweight solution like SQLLite or a document store like MongoDB

Answer 3

The Pickle.dump function is very much like matlab's save feature. You just give it an object you wish to serialize and a file-like object to write it to. See the documentation for info and examples on how to use it.

The cPickle module is just like Pickle, but it is implemented in C so it can be much faster. You should probably use cPickle.