In memory database with socket capability

Question

Python --> SQLite --> ASP.NET C#

I am looking for an in memory database application that does not have to write the data it receives to disc. Basically, I'll be having a Python server which receives gaming UDP data and translates the data and stores it in the memory database engine.

I want to stay away from writing to disc as it takes too long. The data is not important, if something goes wrong, it simply flushes and fills up with the next wave of data sent by players.

Next, another ASP.NET server must be able to connect to this in memory database via TCP/IP at regular intervals, say once every second, or 10 seconds. It has to pull this data, and this will in turn update on a website that displays "live" game data.

I'm looking at SQlite, and wondering, is this the right tool for the job, anyone have any suggestions?

Thanks!!!

Answer 1

The application of SQlite depends on your data complexity.

If you need to perform complex queries on relational data, then it might be a viable option. If your data is flat (i.e. not relational) and processed as a whole, then some python-internal data structures might be applicable.

Answer 2

Totally not my field, but I think Redis is along these lines.

Answer 3

Perhaps AppFabric would work for you?

http://msdn.microsoft.com/en-us/windowsserver/ee695849.aspx

Answer 4

SQLite doesn't allow remote "connections" as far as I know, it only supports being invoked as an in-process library. However, you could try to use MySQL which, while heavier, supports remote connections and does have in-memory tables. See http://dev.mysql.com/doc/refman/5.5/en/memory-storage-engine.html

Answer 5

This sounds like a premature optimization (apologizes if you've already done the profiling). What I would suggest is go ahead and write the system in the simplest, cleanest way, but put a bit of abstraction around the database bits so they can easily by swapped out. Then profile it and find your bottleneck.

If it turns out it is the database, optimize the database in the usual way (indexes, query optimizations, etc...). If its still too slow, most databases support an in-memory table format. Or you can mount a RAM disk and mount individual tables or the whole database on it.