Real time data storage and access with .net

Question

Hello all,

Does anyone have any experience with receiving and updating a large volume of data, storing it, sorting it, and visualizing it very quickly?

Preferably, I'm looking for a .NET solution, but that may not be practical.

Now for the details...

I will receive roughly 1000 updates per second, some updates, some new rows of data records. But, it can also be very burst driven, with sometimes 5000 updates and new rows.

By the end of the day, I could have 4 to 5 million rows of data.

I have to both store them and also show the user updates in the UI. The UI allows the user to apply a number of filters to the data to just show what they want. I need to update all the records plus show the user these updates.

I have an visual update rate of 1 fps.

Anyone have any guidance or direction on this problem? I can't imagine I'm the first one to have to deal with something like this...

At first though, some sort of in memory database I would think, but will it be fast enough for querying for updates near the end of the day once I get a large enough data set? Or is that all dependent on smart indexing and queries?

Thanks in advance.

Answer 1

May be Oracle is more appropriate RDBMS solution fo you. The problem with your question is that at this "critical" levels there are too much variables and condition you need to deal with. Not only software, but hardware that you can have (It costs :)), connection speed, your expected common user system setup and more and more and more... Good Luck.

Answer 2

It's a very interesting and also challenging problem.

I would approach a pipeline design with processors implementing sorting, filtering, aggregation etc. The pipeline needs an async (threadsafe) input buffer that is processed in a timely manner (according to your 1fps req. under a second). If you can't do it, you need to queue the data somewhere, on disk or in memory depending on the nature of your problem.

Consequently, the UI needs to be implemented in a pull style rather than push, you only want to update it every second.

For datastore you have several options. Using a database is not a bad idea, since you need the data persisted (and I guess also queryable) anyway. If you are using an ORM, you may find NHibernate in combination with its superior second level cache a decent choice.

Many of the considerations might also be similar to those Ayende made when designing NHProf, a realtime profiler for NHibernate. He has written a series of posts about them on his blog.