Using Mapped Memory Files in C# to store reference types

Question

I need to store a dictionary to a file as fast as possible. Both key and value are objects and not guaranteed to be marked as Serializable. Also I prefer a method faster than serializing thousands of objects. So I looked into Mapped Memory Files support in .NET 4. However, it seems MemoryMappedViewAccessor only allows storage of structs and not reference types.

Is there a way of storing the memory used by a reference type of a file and reconstructing the object from that blob of memory (without binary serialization)?

Answer 1

This is the type of scenario that binary serialization was designed for. Is there some specific reason why you don't want to use that? Have you verified that it's 'too slow'? Sure, you can code your own custom serializer and probably make it more efficient for your specific scenario, but then you'll have to maintain it going forward. Is it worth the effort?

Answer 2

I believe that storing a blob of memory is simply unworkable because that memory, if it has reference types, will have pointers to other blocks of memory that will likely not apply next time the file is accessed. That's why binary serialization exists: to maintain these kinds of references. If you really want tight control, though, I would use System.IO.BinaryWriter and BinaryReader to have full control over exactly what is written to the file in what sequence, while minimizing overhead.

Answer 3

Memory mapped files are fundamentally incompatible with the garbage collector. Which is why it took so long for such a principal operating system feature to get supported by .NET. Reference types need to be serialized to the MMF view, MemoryMappedViewStream, no way around that. A similar restriction exists in unmanaged code, objects with pointers need to be flattened so the pointed-to objects are visible in the view as well.

Whether you serialize them to a MMF or to a file won't make any difference, the file system cache is implemented with MMFs as well. File writes are very fast, as long as the written data fits in available mappable memory. If that's an issue then look at a 64-bit operating system to solve that problem.