Embedded non-relational (nosql) data store

Question

I'm thinking about using/implementing some kind of an embedded key-value (or document) store for my Windows desktop application. I want to be able to store various types of data (GPS tracks would be one example) and of course be able to query this data. The amount of data would be such that it couldn't all be loaded into memory at the same time.

I'm thinking about using sqlite as a storage engine for a key-value store, something like y-serial, but written in .NET. I've also read about FriendFeed's usage of MySQL to store schema-less data, which is a good pointer on how to use RDBMS for non-relational data. sqlite seems to be a good option because of its simplicity, portability and library size.

My question is whether there are any other options for an embedded non-relational store? It doesn't need to be distributable and it doesn't have to support transactions, but it does have to be accessible from .NET and it should have a small download size.

UPDATE: I've found an article titled SQLite as a Key-Value Database which compares sqlite with Berkeley DB, which is an embedded key-value store library.

Answer 1

Have you considered simply serializing an ADO DataSet for your data store?

Answer 2

Personally I would go for SQLite with NHibernate (and Fluent NHibernate). NHibernate can generate the database schema automatically for your classes, so you just need to specify what classes you want to persist, and that's quite easy with Fluent NHibernate. Furthermore, you can search for specific objects and you don't need to load all data to memory.

Best Regards
Oliver Hanappi

Answer 3

You can use document store MongoDB, it has a .Net driver and it doesn't have a schema. However it is not embedded, MongoDB runs as a separate process. See http://github.com/samus/mongodb-csharp

Answer 4

Windows has a built-in embedded non-relational store. It is called ESENT and is used by several Windows applications, including the Active Directory and Windows Desktop Search.

http://blogs.msdn.com/windowssdk/archive/2008/10/23/esent-extensible-storage-engine-api-in-the-windows-sdk.aspx

If you want .NET access you can use the ManagedEsent layer on CodePlex.

http://managedesent.codeplex.com/

That project has a PersistentDictionary class that implements a key-value store that implements the IDictionary interface, but is backed by a database.

Answer 5

Applying the KISS principle to your problem I would recommend you use files.

As in filename is the key. File contents is the value. Windows folder is the index.

Simple, quick, efficient, flexible, and foolproof (providing the fools have low intelligence).

Answer 6

Thanks for your kind mention of y_serial... more precisely, it is a Python module:

warehouse Python objects with SQLite

"Serialization + persistance :: in a few lines of code, compress and annotate Python objects into SQLite; then later retrieve them chronologically by keywords without any SQL. Most useful "standard" module for a database to store schema-less data."

http://yserial.sourceforge.net

In my experience, SQLite is a faster and more reliable choice than most databases (including PostgresQL and Berkeley DB) for the majority of projects -- and of course, it does not need a server daemon.

yserial is very easy to implement (and far faster than the "filename is the key / file contents is the value" approach ;-)

Answer 7

Could you create a simple sqlite database with two columns:

==documents==
id|data

and data would be json data.

You could also create a key table which would be:

==keys==
keyname|keyvalue|id

that would be indexed on keyname and keyvalue for quick lookups.

A single db file could be a collection, and you could create multiple db files for multiple collections.

You could use folders as "dbs" to match mongodb's hierarchy of db->collection->document

Answer 8

Take a look at RavenDB. It looks as though it can be embedded and is schemaless and works with .NET

From the website:

• Scalable infrastructure: Raven builds on top of existing, proven and scalable infrastructure
• Simple Windows configuration: Raven is simple to setup and run on windows as either a service or IIS7 website
• Transactional: Raven support System.Transaction with ACID transactions. If you put data in it, that data is going to stay there
• Map/Reduce: Easily define map/reduce indexes with Linq queries
• .NET Client API: Raven comes with a fully functional .NET client API which implements Unit of Work and much more
• RESTful: Raven is built around a RESTful API