dictionary interface for large data sets

views:

answers:

+2 Q:

dictionary interface for large data sets

I have a set of key/values (all text) that is too large to load in memory at once. I would like to interact with this data via a Python dictionary-like interface.

Does such a module already exist?

Reading key values should be efficient and values compressed on disk to save space.

Edit:

Ideally cross platform, but only using Linux for now
Needs to be thread safe

Assuming you are on a *nix platform, you are just BEGGING for tokyo-cabinet... It has a native set of ruby bindings... You can more information at 1978th.net...

LorenVS 2010-03-31 06:53:59

need a Python solution to integrate with existing code

Plumo 2010-03-31 13:19:07

Wow, I'm miss reading posts left right and center today... My bad

LorenVS 2010-03-31 14:00:58

+2 A:

Take a look at the shelve module. It provides dictionaries with various backing database systems include BerkleyDB and dbm.

Robert Christie 2010-03-31 07:09:55

unfortunately "does not support concurrent read/write access"

Plumo 2010-03-31 13:21:49

+2 A:

ZODB

gnibbler 2010-03-31 07:35:36

looks promising though is overkill for my particular use case. Is there a way to seamlessly compress the data on disk to save space?

Plumo 2010-03-31 13:36:42

+1 A:

Try bsddb(Python bindings for Berkley db)

Anurag Uniyal 2010-03-31 07:59:53

ended up creating my own solution based on this existing module.

Plumo 2010-04-29 00:37:48

ansaurus

tags:

views:

answers:

dictionary interface for large data sets

related questions