Developing my own RDBMS

Question

I've always been fascinated by how these huge RDBMS servers can work for months, shuffling data all around without losing it, perform all kinds of queries and generally being very complex beasts.

I understand how much effort it takes to create even something as "simple" as SQLite, but I cannot resist the desire to try and develop my own RDBMS engine. Therefore, my question is: what books/online resources/etc. are there to this regard? I'd prefer more practice-oriented material rather than purely theoretical, but these will do as well.

I'd also prefer not to read SQLite/MySQL/PostgreSQL sources as this all are well beyond "simple" and are cluttered (being written in a fairly low-level language) with various technical and implementation details.

Answer 1

This question has been asked before in some permutation. I've seen this book recommended, Database Systems: The Complete Book. I've not read it, but I did order it. It covers basic theory and has a section on implementation.

I'm in the same boat you are. I would love to know how to implement a database, but the landscape for even regular database material on the web is a bit thin, let alone implementing one.

Other answers include reading the documentation and source code to SQLite, as well as the source code to other open source DBs (PostgreSQL, MySQL).

To add, maybe just good old fashioned feature copying. From my short stint in databases, we hear so much about how they work at a superficial level. So maybe you can take those concepts pretty literally and come up with implementations yourself. Like how indexes are just tables or how transcations work or how things are logged first then enacted upon, etc. My guess is that finding tutorial-centric material for database implementation will be really, really tough. But I'm right there with you! Stack project!!!

Answer 2

Ok, got one more-or-less practical. Database Management Systems by Raghu Ramakrishnan & Johannes Gehrke.

Answer 3

I second Database Management Systems and also recommend Readings in Database Systems.

I have read both. Database Management Systems is quite excellent. Readings in Database Systems is also filled with valuable information but is not as accessible.

Answer 4

I highly recommend you this book:

Database in Depth: The Relational Model for Practitioners by C.J. Date

Answer 5

I have an odd recommendation for you.

There are plenty of database engines that take in SQL.

Maybe take a different path and build a RDBMS so that what is passed in is directly the execution plan. It would have interesting applications.

Answer 6

For the theoretical part, I think that Edgar Frank "Ted" Codd (the father of Relational Databases) work is a must read (see Dr Codd at the C2 wiki too). The paper that started it all is "A Relational Model of Data for Large Shared Data Banks". Also check the work of Christopher J. Date and Hugh Darwen that have among others subsequently maintained and developed Codd's relational model. More precisely, look at The Third Manifesto and "An Introduction to Database Systems".

For the practical part, I'd warmly recommend to check Rel. Quoting the Rel page on Wikipedia:

Rel is an open source true relational database management system that implements a significant portion of Chris Date and Hugh Darwen's Tutorial D query language.

Primarily intended for teaching purposes, Rel is written in the Java programming language.

Rel might be much better for educational purpose than SQLite/MySQL/PostgreSQL in my opinion.

Answer 7

Developing a RDBMS can be a huge task but if you're doing this for learning purposes you don't have to go all the way. Simplify things. To begin with I wouldn't even created it ACID compliant.

If I were to create a RDBMS I would:

1) Create a simple storage engine. Simple tables and the most common datatypes. Skip implementing most indexes but the primary key index Skip transactions (to begin with) Skip optimizing database r/w

2) Query operators When you have libraries for storage I would implement the simple operators (projection, selection, cross join etc.)

3) Query parser When The simple operators had been implemented I would create a query parser for SQL. Not a fancy optimized one, but one that could parse simple SQL statements into query trees.

I wouldn't implement undo/redo logging, transaction support, odbc or any other "real" rdbms functionality in the first prototype/proof of concept solution.

Answer 8

The Developer FAQ on the postgresql website has a few interesting books and links suggestions (below on the page) http://wiki.postgresql.org/wiki/Developer%5FFAQ#What%5Fbooks%5Fare%5Fgood%5Ffor%5Fdevelopers.3F

Answer 9

Fundamentals of Database Systems by Elmasri & Navathe is a good one.

Answer 10

Apart from the books, i would strongly suggest you to take a look at the open source C# port of sqlite. Incase If you are a c# developer

Answer 11

If you are looking for source code, then Relational Database Management by Papazoglou and Valder contains a complete RDBMS, from B*-trees to language parser, implemented in (old-style) C. There's also a lot of explanatory text.

Answer 12

The H2 database is an open source full-featured RDBMS written in Java. The source code might provide valuable insights regarding the inner workings of a database.