In-memory database available across AppDomain boundary.

Question

Dear ladies and sirs.

I wonder if there is an in-memory database implementation, where the same in-memory database instance can be used across multiple AppDomains.

The motivation. Like many folks out there, we have integration tests for our client-server application. There are several test modes, from the most heavy one - the real life scenario, where the client, server and database are all located on different machines to the most lightweight one, where both the client and server are simply two AppDomains in the same process and the database is an sqlite file on the same machine.

Naturally, the lightweight scenario is the fastest and developers routinely use it, while the heavy one is run on our CI server on each build.

My goal is to further speed up the lightweight scenario by using an in-memory database.

The problem. OK, so sqlite has an in-memory database option, but such a database:

• is disposed of after a connection to it is closed.
• cannot have two open connections, new connections open another database instance.

Unfortunately, I need two - one for the test staging code, which runs in the client side AppDomain and the other - for the server side DAL.

I realize the rationale behind the way sqlite in-memory database works. I am also aware of the difficulties of having in-memory database shared between two AppDomains. How can one share a memory buffer between two AppDomains without reverting to memory mapped files (and I do not want to deal with any File System API)?

The only efficient solution that I see is having two AppDomains share an unmanaged memory buffer, where the handle to the buffer would be passed from one AppDomain to another. But are there any in-memory database implementations that support this setup?

(An inefficient solution would be to pass the whole database from one AppDomain to another and then back again).

I may be completely wrong with my analysis and there may be simple solutions which I miss. Anyway, I would like to know if anyone has encountered the same problem and how have they solved it.

P.S.

I would really like to avoid any kind of File API, like memory mapped files.

Answer 1

I don't know of any such database.

But it's fun to think about what form such a database could have. If we consider an app domain a light-weight process, then we can treat cross-domain talking (serialization) as message passing. Then we can define a simple message protocol that allows clients to send queries, and in response get results.

1. Your database should run in its own app domain
2. Each client should run in its own app domain
3. Standard serialization between app domains should be used to return query results. The easiest way to do that is to encapsulate queries in their own classes that execute when constructed. That way, you can use AppDomain.CreateInstance.
4. Your database should be able to handle requests from any thread.

I don't think this is a great idea though. There are too many details in how app domains work and interoperate to get this 100% right.

Answer 2

I have three solutions that come to mind, but all of them require some configuration. They all require you to create a third AppDomain where the in-memory database resides as a s singleton, and to code a class with a well-defined interface to this database.

The first solution is to just call code across AppDomains. The objects you pass must be "Remotable" which can be a big deal to set up. (Derive from MarshalByRefObject, etc) Not only that, any objects you pass to or from the AppDomains must be serializable. It's do-able, but sometimes it's not pretty.

A second solution is to use remoting. It's old technology now, but it still works. It basically formalizes the above call across AppDomains. I won't say any more about it here, I'm almost embarrassed by bringing it up.

A third solution is to use WCF and the namedPipeBinding which is fast. Both ends of a WCF call can be in the same applicaton and in different app domains. A side effect of coding it this way is that you'd be able to switch out the in-memory database for the fully featured one later, and change the binding to work across an intranet, or the internet.

The third solution is the one I'm currently using for a project similar to what you describe. Our goal was to isolate the database so we could try out nHibernate without "contaminating" the rest of the layers.