Fast and compact object serialization in .NET

Question

Hi folks!

I want to use object serialization to communicate over the network between a mono server and Silverlight clients. It is pretty important that serialization is space efficient and pretty fast, as the server is going to host multiple real time games.

Any suggestions on what technique I should use? The BinaryFormatter adds a lot of overhead to serialized classes (Version, culture, class name, property names, etc) that is not required within this application.

What can I do to make this more space efficient? Thanks in advance!

Dennis

Answer 1

You could pass the data through a DeflateStream or GZipStream to compress it prior to transmission. These classes live in the System.IO.Compression namespace.

Answer 2

I think SOAP xml formatter is more efficient for your purpose.

Answer 3

You can use Protocol Buffers. I'm changing all my serialization code from BinaryFormatter with compression to Protocol Buffers and obtaining very good results. It's more efficient in both time and space.

There are two .NET implementations by Jon Skeet and Marc Gravell.

Answer 4

You could try using JSON. It's not as bandwidth efficient as protocol buffers but it would be a lot easier to monitor messages with tools like wireshark, which helps a lot when debugging problems. .net 3.5 comes with a json serializer.

Answer 5

As the author, I would invite you to try protobuf-net; it ships with binaries for both mono 2.0 and silverlight 2.0, and is fast and efficient. If you have any problems whatsoever, just drop me an e-mail (see my stackoverflow profile); support is free.

Jon's version (see the earlier accepted answer) is also very good, but IMO the protobuf-net version is more idiomatic for C# - Jon's would be ideal if you were talking C# to java, so you could have a similar API at both ends.

Answer 6

I had a very similar problem - saving to a file. But the following can also be used over a network as it was actually designed for remoting.

The solution is to use Simon Hewitt's library - see Optimizing Serialization in .NET - part 2.

Part 1 of the article states (the bold is my emphasis): "... If you've ever used .NET remoting for large amounts of data, you will have found that there are problems with scalability. For small amounts of data, it works well enough, but larger amounts take a lot of CPU and memory, generate massive amounts of data for transmission, and can fail with Out Of Memory exceptions. There is also a big problem with the time taken to actually perform the serialization - large amounts of data can make it unfeasible for use in apps ...."

I got a similar result for my particular application, 40 times faster saving and 20 times faster loading (from minutes to seconds). The size of the serialised data was also much reduced. I don't remember exactly, but it was at least 2-3 times.

It is quite easy to get started. However there is one gotcha: only use .NET serialisation for the very highest level datastructure (to get serialisation/deserialisation started) and then call the serialisation/deserialisation functions directly for the fields in the highest level datastructure. Otherwise there will not be any speed-up... For instance if a particular data structure (say Generic.List) is not supported by the library then .NET serialisation will used instead and this is a no-no. Instead serialise the list in client code (or similar). For an example see near "'This is our own encoding." in the same function as listed below.

For reference: code from my application - see near "Note: this is the only place where we use the built-in .NET ...".

Answer 7

I have some benchmarks for the leading .NET serializers available based on the Northwind dataset.

@marcgravell binary protobuf-net is the fastest implementations benchmarked that is about 7x faster than Microsoft fastest serializer available (the XML DataContractSerializer) in the BCL.

I also maintain some open-source high-performance .NET text serializers as well:

• JSV TypeSerializer a compact, clean, JSON+CSV-like format that's 3.1x quicker than the DataContractSerializer
• as well as a JsonSerializer that's 2.6x quicker.