Writing your own storage-system: where to start?

Question

Hi, reading about NoSQL (http://nosql.eventbrite.com/), a movement aimed at encouraging the dropping of traditional relational databases in favor of custom, application-suited storage systems.

Intrigued by the idea of trying to write a small personal storage system (for the .net framework) as a learning pet project, what are you suggestions or useful links? Where to start? How to balance what's on the hard drive and what's in memory?

I think this could be an interesting opportunity to learn the insides of database inner work, but I really lack the most basic theory of it. Thanks.

Answer 1

Hi!

The SO question "What would it take to write my own database system?" has some usefull answers to your question!

Answer 2

Before you get going I would recommend looking into SQL Servers ability to store XML files as BLOB objects inside the relational database. Perhaps your storage system doesn't need to be "from scratch". It could be a hybrid on top of SQLs XML storage capability.

Answer 3

Well it all depends on the app you are building.

For example, if your app just needs to persist a few hundred objects and cut through them in a few ways and doesn't care if stuff gets corrupt once in a while. You could potentially just use LINQ to query a List and persist the List to disk once in a while.

If you need anything that has the magic ACID properties, well its going to take tons of work.

If you need something that supports Transactions, its going to take tons of work.

If you need something that understands the ANSI-SQL, you are going to have to write a parser, which is lots of work.

Before embarking on writing any kind of database I think you should understand a lot of database theory, get a book, read it.

Answer 4

The NoSQL movement is aimed at huge scale systems, at sizes where the relational model truly breaks. Before you start writing your own storage I highly recommend understanding the relational model, as is one of the best documented and well understood domains in CS. Start with the Gray's and Reuter's Transaction Processing, this book explains everything there is to know about implementing a classic RDBMS. Next on your list should be the Readings in Database Systems, this is a collection of the most relevant scientific papers and articles.

Answer 5

Take a look at the work done by the Prevayler guys. They make the observation that if you can fit the data in ram, most usage scenario's show much better performance, and a lot less writing of code, without a RDBMS. On the other hand the Google, Amazon guys show that for large amounts of data you do not want to use a RDBMS. As we're moving to 64-bit OS-es and pcs with lots of ram, RDBMS's are between a rock and a hard place.