simplest key value store for xml data with best performance for .NET

Question

I need a way to store large-numbers of variously sized XML data in a simple key/value store. So far I have tried two simple approaches:

1. RAM: Store everything in a ConcurrentDictionary instance
   • Problem: limited resources at project launch, a single server with 8GBs of RAM!
   • Note: Windows AppFabric could be a solution here in the future if we get more capable servers.
2. HDD: Store every value in its own file on the hard disk, with keys mapping to paths
   • Problem: latency
   • Question: how much of a difference could using an SSD make in the most optimistic scenario here?

Is there a ready-to-use database solution that could provide the middle-way, providing very fast read-access times while utilizing the HDD?

Write-access is only necessary for maintenance so any solution that specializes in least latency for read-only scenarios would work best. The data server should be communicable using WCF.

Answer 1

You might want to look at memcached if you are in .Net. It allows you start with in memory cache and move to redundant storage if required. It all depends on your data requirements and how often you are reading and writing the store.

As for a database, MSSQL provides XML type storage, so you could definately create a simple table with XML inside it, but it's not cheap and you did mention you wanted a NoSQL implementation. Note that if you need to search or sort your XML you may find having XPath and indexing available to be a bonus...