Lightweight Store Mechanisms

Question

I'm about to write a small utility to organze and tag my mp3s.

What is the best way to store small amounts of data. More importantly, are there databases which exist where I don't need to install a client/server environment, I just include the library and I'm good?

I could use XML, but I'm afraid that the file size would become large and hard to handle, not to mention keeping the memory footprint small.

Thanks

EDIT: I haven't decided on the language, I wanted to make my decision independent of platform. If I had to choose, most likely .NET, second Java, third C++.

My apologies, this is for a Windows App.

Answer 1

Which language/platform are you talking about?

In the Java world I prefer using embedded databases such as HSQLDB, H2 or JavaDB (f.k.a. Derby).

They don't need installing and still provide the simple access you're used to from a "real" DBMS.

In the C/Python/Unixy world SQLite is a hot contender in that area.

Answer 2

SQLITE if you want the pain of a relational DB without a server install or hassle.

I would use one of the many text-serialization formats. I personally think that YAML 1.1 is the most powerful (built-in support for referential object graphs) and easiest to read/modify by a human (parsing is a bear, use a library such as PyYAML or JYaml or some .NET libaray).

Otherwise XML or JSON are adequate file formats.

Whichever format you use, just compress the file if you're concerned about disk usage. If you're worried about in-memory usage, then I don't see how your serialization format matters...

Answer 3

you could even use xml, json, an .ini file... a text file even

Answer 4

Another option is the various forms of the Berkeley database (eg, db3, db4, SleepyCat.)

Answer 5

Have a look at Prevayler - it's a serialization persistence framework (use xstream etc if you want to human-read your data), which is really fast, does not require annotations and "just works". Some basic info:

• It does impose a more rigorous transaction pattern, as it does not give you automatic rollback:
  • Ensure transaction will succeed (with current state of system) - e.g. does it make sense now?
  • [transaction is added to queue], and stored (for power reset etc)
  • transaction is executed and applied to the object structure.
• Writes of 1000's of transactions/sec
• Reads of 100,000's transactions/sec

I haven't used it much, but it's sooo much nicer to use for small projects (persisting any serializable object is so nice)

Oh - as for every one saying "what platform you running on?", Prevayler (java) has/had ports to quite a few platforms, but I can't find a decent list :(. I remember that there were around 5-7, but can only remember .NET.

Answer 6

If you're planning on storing everything in memory while your program does work on it, then serializing to a file using a basic load() and save() function that you write would be fine, and less pain than a full on DB.

In Java that can be done using standard Serialization (or can serialize to and from XML to make it somewhat human readable and editable).

It shouldn't affect your memory footprint at all as it is merely saving and restoring your objects. You just won't get transactions and random access and queries and all that good stuff.

Answer 7

I would advise a SQL like database (such as SQLLite). Today your requirements might make a full SQL database seem silly. But you never know how much this "little project" will grow over the years. When it does grow to the point where you have to have a SQL engine, you will be glad you didn't just serialize some Java objects or store stuff in JSON format.

Answer 8

On Windows you can use the built-in esent database engine. There is an API you can use from C++

http://blogs.msdn.com/windowssdk/archive/2008/10/23/esent-extensible-storage-engine-api-in-the-windows-sdk.aspx

There is also a managed interop layer that you can use from C# code:

http://www.codeplex.com/ManagedEsent