Fastest and stable non-sql database?

Question

What is the fastest and most stable non-sql database to store big data and process thousands requests during the day (it's for traffic exchange service)? I've found Kdb+ and Berkeley DB. Are they good? Are there other options?

More details...

Each day server processes > 100K visits. For each visit I need to read corresponding stats from DB, write log to DB and update stats in DB, aka 3 operations with DB per visit. Traffic is continuously increasing. Thus DB engine should be fast. From one side DB will be managed by demon written on C, Erlang or any other low-level language. From another side DB will be managed by PHP scripts.

Answer 1

The file system itself is faster and more stable than almost anything else. It stores big data seamlessly and efficiently. The API is very simple.

You can store and retrieve from the file system very, very efficiently.

Since your question is a little thin on "requirements" it's hard to say much more.

Answer 2

What's wrong with SqlLite? Since you did explicitly state non-sql, Berkeley DB are based on key/value pairs which might not suffice for your needs if you wish to expand the datasets, even more so, how would you make that dataset relate to one another using key/value pairs....

On the other hand, Kdb+, looking at the FAQ on their website is a relational database that can handle SQL via their programming language Q...be aware, if the need to migrate appears, there could be potential hitches, such as incompatible dialects or a query that uses vendor specifics, hence the potential to get locked into that database and not being able to migrate at all...something to bear in mind for later on...

You need to be careful what you decide here and look at it from a long-term perspective, future upgrades, migration to another database, how easy would it be to up-scale, etc

Answer 3

One obvious entry in this category is Intersystems Caché. (Well, obvious to me...) Be aware, though, it's not cheap. (But I don't think Kdb+ is either.)

Answer 4

What about Redis?

http://code.google.com/p/redis/

Haven't try it yet did read about it and it seem to be a fast and stable enough for data storage. It also provides you with a decent anti-single-point-failure solution, as far as I understand.

Answer 5

Berkely DB is tried and tested and hardened and is at the heart of many mega-high transaction volume systems. One example is wireless carrier infrastructure that use huge LDAP stores (OpenWave, for example) to process more than 2 BILLION transactions per day. These systems also commonly have something like Oracle in the mix too for point in time recovery, but they use Berkeley DB as replicated caches.

Also, BDB is not limited to key value pairs in the simple sense of scalar values. You can store anything you want in the value, including arbitrary structures/records.