Efficient storage in .Net App

Question

I'm looking for the fastest, least memory consuming, stand alone storage method available for large amounts of data for my C# app.

My initial thoughts:

• Sql: no. not stand alone
• XML in flat file: no. takes too long to parse large amounts of data
• Other Options?

Basically what i'm looking for, is a way that i can load with my applications load, keep all the data in my app, and when the data in my app changes just update the storage location.

Answer 1

Binary serialiser? See http://msdn.microsoft.com/en-us/library/72hyey7b(VS.71).aspx for details.

This is a bit like an xml serialiser, though it's binary data. It's typically a lot smaller.

However, you have other things to consider, possibly:#

• Should the data be human-readable?
• Should other systems/programming languages be easily able to interpret it?
• Are there any concurrency issues with multiple users?
• Do you need to index or search the data somehow, or just load the whole lot back into memory again?

Answer 2

How large is large? Before dismissing XML, Double-check your assumption about the performance of XML parsing.
The reason XML is nice is because you can patch the data with a text editor. Can't do that easily with a binary store.

Also, SQL Server has an embedded version. It's called "Compact Edition", and it is embebddable into your C# app. Free license, too. Read more.

Answer 3

Sql: no. not stand alone

It depends on what you mean by "stand alone"... some DBMS are stand alone and don't need a server :

• SQL Server Compact Edition
• SQLite
• Other file-based DBMS...

Answer 4

You could use a in process (serverless) database like Sqlite, of Sql Server Compact

Answer 5

I'm not exactly sure what you're looking for but if i need some simple flat file storage in C# i usually just use linq2csv. its fast, efficient and type safe, with the added benefit that you can just edit the files in excel (or whatever) which makes it easy. Though its obviously not suited for hierarchical or highly relational data, but it is very easy for simple tabular data and much more compact than XML.

It provides very easy Write and read methods which would make it easy to sync changes to filesystem.

Answer 6

I think that maybe a virtual file system will do the work. For example, please see Solid File System . It lets you create a file system in a file, with certain DB-like search capabilities. And it seems to fit your needs nicely.