I would like to store data persistently for my application, but I don't really need a full blown relational database. I really could get by with a basic "cache"-like persistent storage where the structure is just a (key, value) pair.
In lieu of a database what are my best, scalable options?
If you want something really scalable, I wouldn't opt for a flat or XML file. As your data grows, it could kill your performance.
If you will have a lot of data at some stage, I would still opt for a database - I would take a look at something like SQLIte with a very simple schema to suit your needs.
There's always SQLite, a database that's stored in a file. SQLite already has built-in concurrency, so you don't have to worry about things like file locking, and it's really fast for reads.
If, however, you are doing lots of database changes, it's best to do them all at once inside a transaction. This will only write the changes to the file once, as opposed to every time an change query is issued. This dramatically increases the speed of doing multiple changes.
When a change query is issued, whether it's inside a tranasction or not, the whole database is locked until that query finishes. This means that extremely large transactions could adversely affect the performance of other processes because they must wait for the transaction to finish before they can access the database. In practice, I haven't found this to be that noticeable, but it's always good practice to try to minimize the number of database modifying queries you issue.
If you need scalability, then a RDBMS is your best bet. At the most basic level, you can serialize data structures into files - however, then you'd need to account for file locking issues which would limit concurrency.
SQLite is an SQL file-based database engine, which can run without a persistent database daemon (in PHP for example, it runs as an extension), however it too has concurrency issues (read the answers to this question which could help you decide if SQLite is right for you).
Unless you have a really good reason not to use a real DRBMS, I'd suggest you stick with MySQL or other "full-blown" engines.
If you are writing Java, then there are Java database implementations (Jared mentions hsqldb, there are others) that you can directly include.
SQLite is fine for static inclusion, however you can also include MySQL within your application if you're using a compatible language such as C.
I think you would appreciate having SQL available as well. XML files just don't cut it any longer, maybe a couple of years ago when writing PDA software, but even the iPhone and Android includes SQLite now.
if you want a 'persistent cache', and already use memcached, check memcachedb. it's a persistent hashtable using the memcached protocol, no need for a new client (but a new daemon)
For Key=Value pairs, you can use INI file format with simple load and save procedures to load it and save it to in-memory hash table.
That can later scale up to anythig, just by changing load and save procedures to work with db.
You could try CounchDB, it is a very flexible document-oriented database that doesn't force you to define a schema up-front. It was written in Erlang and thanks to this it is considered very scalable solution. It can be easily queried through a REST interface.
I ask a similar question recently. Here are some choices:
