Streaming, custom serialization of huge objects in .NET

Question

In .NET, I would like to call a REST-style web service which expects a POST body which is a binary serialization of a class on the following form:

public class Test
{
    public string Name;
    public byte[] Data;
}

The bitstream should be exactly the same as when using BinaryFormatter.Serialize(). The problem is that the Data member could be really large and I (as the caller) am in turn acquiring it from a file stream. I do not want to first create a class instance in memory by reading all the data from the file, and then serialize it again just to call the web service. Instead, I envision using something similar to a BinaryWriter but which supports something like this pseudocode:

var w = new MagicBinaryWriter(myOutputStreamToPost);
w.BeginObject(typeof(Test));
w.WriteString("some string");
w.WriteByteArray(myFileStream);
w.EndObject();

It is worth noting that this is code that will run on a server, with potentially many concurrent users of this functionality, which is why I want to avoid storing all the data in memory.

I have been searching for low-level "building block" interfaces so to speak, preferably the ones that BinaryFormatter.Serialize() itself uses, but with no luck. Do they exist? Or at least very precise specs on how the serialization format of .NET looks, so I can roll my own?

Answer 1

Why don't you just decompile .net assemblies and take a look under the hood of how BinaryFormatter.Serialize() works? You could use redgate or some other decompiler.

Answer 2

You cannot realistically implement a replacement for BinaryFormatter. The exact format of the binary data it generates isn't documented. And the structure is complicated, it adds metadata to the actual object data that allows the reader to reconstruct the type of the object. It also isn't an exact one-to-one match with the collection layout, it flattens the object graph to some extent to avoid generating too much metadata. The amount of code in BinaryFormatter is quite stupendous.

Tackle this at the source and think of another representation of the data that doesn't consume so much memory.