Storing large amounts of data in files. What is the most performant option?

Question

Currently doing XML serialization however, it is very slow. Looking for a way to save/load information from file very quickly not really interested in how it looks on disc (if anything I want it to be obscured as I don't want manual editing).

Thinking of binary format however I am not sure if it would be able to serialize properties which may be of a custom type etc.

Any idea's?

Answer 1

Binary serialization certainly works with properties of Custom Types and typically produces smaller files than XML serialization. It's certainly an approach you should consider if file size is an important factor for your situation.

Answer 2

What exactly is the data?

With xml, the obvious answer would be to use smoething like GZipStream to compress it - making it smaller and obscure. You could use BinaryFormatter but it is brittle and IMO unsuitable for long-term storage. I would say "protocol buffers", (maybe protobuf-net), but it depends what the "custom data" is. But if you are using XmlSerializer at the moment protobuf-net may work virtually without changes (maybe add a few attributes) - and it is (in every case I've seen to date) both smaller and faster than BinaryFormatter.

Here's the steep learning curve (see also: Getting Started):

[ProtoContract]
public class Person {
    [ProtoMember(1)]
    public int Id {get;set;}

    [ProtoMember(2)]
    public string Name {get;set;}

    //...
}

To be fair, it can get a little trickier if you are using inheritance - not much though. In many cases you can actually use your existing attributes - it'll work with xml / wcf attributes if an explicit element order is included.

Answer 3

You can try using Sqlite. It is very fast, and will give you complete database implementation with SQL queries on a file.

If you are thinking of trying binary formats, I suggest you try this first.

And can be used with ORM, and can be compressed and encrypted.

Answer 4

I agree with Am about using an embedded database like SQLite. It comes with significant benefits. The ability to layer an ORM on top of it is probably the most significant.

XML Serialization is handy, particularly when you need to be able to edit the XML by hand or process it with other XML tools like XSLT etc, but it also has some unavoidable performance problems. One important technique when using XML Serialization in .Net is to cache the XML Serializers. Or to have them created by sgen on build.

The reason to cache the XML Serializer is related to the fact that the .Net runtime will automatically generate, compile and load an assembly containing a serializer if it can't find one in an already loaded assembly. This process can be really slow. Also constructing a new XMLSerializer instance can be quite slow. Hence why you should cache it. Be careful when caching the serializer though as different XMLSerializer constructors can produce different serializer implementations which behave differently. Particular with respect to namespaces, etc.

Then of course there is the usual performance implications of parsing a lot of text. Unfortunately that isn't easy to avoid with XML.

One of the reasons SQLite is a better choice than XML is related to the fact that it is, at its core, a fixed length record storage system. Any binary file with fixed length records is going to be fast to read, index and scan. Fixed block size file formats are almost always screamingly fast to read and write. I would recommend implementing one at some point for your own education.

If you still want a text based format (for ease of interoperability) and don't need the benefits of an ORM then consider using the FileHelpers library.