ansaurus

Question

Single-file storage for a Python application

Answer 1

+4 A:

shelve gives a mapping interface that allows you to store any pickleable type.

Ignacio Vazquez-Abrams 2010-09-01 11:08:57

Touché; shelve is probably ideal.

Autopulated 2010-09-01 11:10:54

IIUC, shelve uses pickle for serialization. How "stable" is the pickle format across OS / architectures? Will I be able to open a shelf stored on 32-bit Windows with 64-bit Windows? Cross-platform operation (Windows/OS X/Linux) is not required, but would be nice to have.

Remy Blank 2010-09-01 11:16:10

If you want to mitigate cross-platform concerns then you could replace the appropriate pickling methods with jsonpickle instead. http://jsonpickle.github.com/

Ignacio Vazquez-Abrams 2010-09-01 11:18:21

jsonpickle looks interesting, except for this part: "Loading a JSON string from an untrusted source represents a potential security vulnerability", and I remember reading the same about pickle. I'd prefer not having to worry about opening a project file from a (malicious) customer and have it wipe my hard disk :)

Remy Blank 2010-09-01 11:22:56

That's nothing pickle itself doesn't suffer from.

Ignacio Vazquez-Abrams 2010-09-01 11:40:03

@Remy Blank: The "security" hand-wringing is only if you're confronted with evil sociopaths who (a) know how to hack the pickle format and (b) feel it necessary to hack the pickle format instead of hacking your Python source. If they want to hack, they have your source. Don't worry needlessly. Use shelve with confidence.

S.Lott 2010-09-01 12:04:17

You can look at http://pyyaml.org/wiki/PyYAMLDocumentation YAML

Ib33X 2010-09-01 12:06:13

@S.Lott: Being able to hack the code is not the issue here, as I will install my own application instance from a trusted source (myself). The use case I want to avoid is getting a malicious project file from an "evil" customer (created with the application installed on his computer), and upon opening it on my computer, having all my files deleted. No need to hack the source (which they won't have anyway, BTW), if unpickling can trigger e.g. os.unlink(). We have had enough of that with Word files...

Remy Blank 2010-09-01 12:30:53

@Remy Blank: Please **update** the question to include this use case. Without the details of this additional use case, no one has any idea what your **real** requirements are.

S.Lott 2010-09-01 14:20:31

@S.Lott: I must be old-fashioned, but when I think of an "application", I see a program installed on a user's computer, not a web application. Word, not Google docs. But you are right, this was ambiguous.

Remy Blank 2010-09-02 20:58:15

@S.Lott: But I fail to see how this affects the security issue: having a malicious data file trigger an evil payload is unacceptable in both cases, "traditional" application or web application.

Remy Blank 2010-09-02 21:07:29

@Remy Blank: In a web application, no one hacks the database. You filter all inputs before putting them in the database. Since no one is allowed to tweak the database file, there's no security issue with a web application. A "desktop" application where the user can only corrupt their own data is none of your concern. The only thing that matters is **you accept data from malicious sociopaths** as part of your business model. Since you have that requirement -- which is very odd in my opinion -- you need to make that absolutely clear.

S.Lott 2010-09-03 15:00:21

@S.Lott: So you never open a Microsoft Word or PDF file created by someone else?

Remy Blank 2010-09-06 07:54:19

@Remy Blank: "Microsoft Word or PDF file created by someone else"? What? Are you talking about viruses? I use Mac OS X and Linux, so I'm aware that Windows has a problem with viruses, but I don't know precisely what you're talking about.

S.Lott 2010-09-07 13:17:59

@S.Lott: Call them viruses if you like (and no, they don't seem to be limited to Windows), but the fact is that opening a Microsoft Word or PDF file (a file that is supposed to contain only data) can trigger a destructive action, and you need "extra protective measures" like macro virus protection to be able to open them safely. This will never happen with e.g. plain text files. I would like my data files to be as safe as plain text files, that is, not have to worry about opening any data file from any source. This is a pretty basic requirement, and pickling doesn't give me that.

Remy Blank 2010-09-07 22:37:10

Answer 2

A:

Pickle it!

Make up your data structures however you like in memory, then just zap them to disk with a pickle when you want to save.

Autopulated 2010-09-01 11:09:08

I'd rather not pickle 10MB of data for every single change to the in-memory data.

Remy Blank 2010-09-01 11:10:53

See shelve, @Ignacio, above ;) -- though, that said, do you really want to be trying to keep disk data current with in-memory data? Sounds like a job for a proper transactional database... much easier to just write once when your application closes?

Autopulated 2010-09-01 11:11:36

@Autopulated: Yes, a transactional database is probably ideal, that's why I started looking at SQLite. Keeping disk data current with in-memory data is a design choice: I don't want the user to have to think about saving. He should be able to close the application at any time (or have it crash, for that matter) without loosing any data and without delay.

Remy Blank 2010-09-01 11:20:00

Answer 3

+4 A:

I would encourage you to look at SQLAlchemy like you discussed in your question. You can map tables in SQLite to Python objects. Using the SQLAlchemy session ( http://www.sqlalchemy.org/docs/session.html#what-does-the-session-do ) you can run queries, add objects to the tables and issue a session.commit command straight away in order to auto save data to SQLite.

New data element:

ed_user = User('ed', 'Ed Jones', 'edspassword') #user is the class you mapped the table to
session.add(ed_user)
session.commit() # basically auto saving here :)

That's what I'd use. I'm using SQLAlchemy for a project now and like what I see. For more see here: http://www.sqlalchemy.org/docs/ormtutorial.html#adding-new-objects

Edward Williams 2010-09-01 13:13:02

Answer 4

A:

Why not giving a try to couchdb ? As your data are heterogenous, it's the best approch. Easy to use and 10MB is nothing to it : http://pypi.python.org/pypi/CouchDB

boblefrag 2010-09-01 15:17:15

ansaurus

tags:

views:

answers:

Single-file storage for a Python application

related questions