Pure Python persistent key and value based container (a hash like interface) with large file system support?

Question

I am looking for a (possibly) pure Python library for persistent hash table (btree or b+tree which would provide following features

1. Large file support (possibly in terabytes)
2. Fast enough and low memory footprint (looking for a descent balance between speed and memory)
3. Low cost of management
4. Reliability i.e. doesn't corrupt file once the content is written through the file system
5. Lastly a pure Python implementation. I am OK if it has C library but I am looking for a cross platform solution

I have looked into solutions like redis, shelve, tokyo cabinet. Tokyo cabinet is impressive and has a Python binding in the making at http://code.google.com/p/python-tokyocabinet/, but its Windows port is a work in progress.

Thanks for some good suggestions. I am currently exploring SQLite3 with Python. I got suggestions to use database engine but am more inclined towards a lean and mean persistent b+tree implementations

Answer 1

ZODB
http://pypi.python.org/pypi/ZODB3

Like Lennart says, use the latest version of course

Answer 2

ZODB is indeed a powerful tool, but maybe it's overkill.

You can hack your own solution in few Python lines : simply code a dictionary like object as a data base adapter. Try using this snippets, replacing the SQLite call to MySql and you should be done.

Answer 3

Use a relational database.

• Really fast when retrieving data based on a key, if you put an index in the key.
• Good scaling
• Don't get easily corrupted
• Tools already available for:
  • Backups
  • Replication
  • Clustering
• Cross-platform
• Works over the network
• Allow really fast JOINs, grouping, agreggation, and other complex queries, in case you need them

You can easily create a class that works like a dict or hash table, but uses the database as storage. You can make it cache as much as you want on memory.