Performance Tests of Serializations used by WCF Bindings

Answer 1

At the simplest level; simply serialize a shed-load of data, time it, and measure the bandwidth. And shed-load should include both large and small (but lots-of) payloads.

You should also consider with/without MTOM. And although I am perhaps biased I suggest you should include alternative WCF serializers such as protobuf-net (let me know if you need help hooking that in). From lots of work in the area it usually defeats all the ones you've mentioned by a decent margin on every measure.

Much of what is involved here can be investigated at the serializer level without even touching WCF, however that bypases base-64/MTOM so isn't a 100 percent image.

We can't, however, define your measures for you; only you can decide what is key. I do have a number of measure though - it is usually simply:

• serialize once to MemorySteam (and deserialize); this gets you the size and primes the JIT
• now keep that memory-stream (as a handy buffer) and (inside stopwatch) serialize many thousands if times. Divide. Rewind each time so you are overwriting (not extending).
• repeat but deserializibg many thousands of times. Divide.

Answer 2

Use an object of constant size; getting "size" information from a Type is messy and won't gain you much in terms of figuring out which is "best". Any object decorated as a DataContract can be serialized to binary (DataContract inherits Serializable), basic XML (any object with a default constructor can be serialized into XML) or DataContract XML (this requires the most markup to start with, but it's pretty simple).

For a running test, create a method that will take an Object and a Serializer. It should create a MemoryStream and start a StopWatch, then serialize the object into the MemoryStream (make sure to Flush()). Then it stops the stopwatch and returns you the results as a TimeSpan, and the Stream's length. Then reset and start the Stopwatch and deserialize the Stream, and remember that time. You can set the return results up as a simple struct.

Run this with the same object for each serializer you want to test. Output each of the results to the console or debug output, and may the best serializer win.

Generally speaking, I think you'll find:

• BinarySerializer will be fastest and smallest, as it has the least overhead of bytes to write while serializing. However, .NET's binary serializations are platform-specific; if you want to talk to anything but another .NET assembly that knows about your exact type, forget it.

• XMLSerializer, SoapSerializer and DataContractSerializer all output various forms of XML. DataContract is actually the simplest format (the XML is extremely basic because the handshake and other protocol/communication information is seperate) and will probably be rather fast. SOAP has a lot of bloat in the serialized file because of transport and metadata information, but is easy to generate as it's a pretty strict format. Basic XML serialization, because it's so flexible, has a lot of overhead, but can generate a very simple or very complex schema.

Answer 3

OK; I'll bite... here's some raw serializer metrics (emph: you may need to consider base-64/MTOM to get overall bandwidth requirements, plus whatever fixed overheads (both space and CPU) that WCF adds), however; results first:

BinaryFormatter
Length: 1314
Serialize: 6746
Deserialize: 6268

XmlSerializer
Length: 1049
Serialize: 3282
Deserialize: 5132

DataContractSerializer
Length: 911
Serialize: 1411
Deserialize: 4380

NetDataContractSerializer
Length: 1139
Serialize: 2014
Deserialize: 5645

JavaScriptSerializer
Length: 528
Serialize: 12050
Deserialize: 30558

(protobuf-net v2)
Length: 112
Serialize: 217
Deserialize: 250

(so I conclude protobuf-net v2 the winner...)

with test rig (compiled with optimizations, run at command line):

(and note I had to invent the Player class and some sample data):

using System;
using System.Diagnostics;
using System.IO;
using System.Runtime.Serialization;
using System.Runtime.Serialization.Formatters.Binary;
using System.Text;
using System.Web.Script.Serialization;
using System.Xml.Serialization;
using ProtoBuf.Meta;


static class Program
{
    static void Main()
    {
        var orig = new Game {
             Finished = true, GameGUID = Guid.NewGuid(), GameID = 12345, GameSetup = false, MaximumCardsInDeck = 20,
             Player = new Player { Name = "Fred"}, Player1 = new Player { Name = "Barney"}, Player1Connected = true,
             Player1EnvironmentSetup = true, Player1ID = 12345, Player1Won = 3, Player2Connected = true, Player2EnvironmentSetup = true,
             Player2ID = 23456, Player2Won = 0, Round = 4, RoundsToWin = 5, Started = true, StateXML = "not really xml",
             TimeEnded = null, TimeLimitPerTurn = 500, TimeStamp = new byte[] {1,2,3,4,5,6}, TimeStarted = DateTime.Today};
        const int LOOP = 50000;

        GC.Collect(GC.MaxGeneration, GCCollectionMode.Forced);
        GC.WaitForPendingFinalizers();
        using (var ms = new MemoryStream())
        {
            var ser = new BinaryFormatter();
            Console.WriteLine();
            Console.WriteLine(ser.GetType().Name);
            ser.Serialize(ms, orig);
            Console.WriteLine("Length: " + ms.Length);
            ms.Position = 0;
            ser.Deserialize(ms);

            var watch = Stopwatch.StartNew();
            for (int i = 0; i < LOOP; i++)
            {
                ms.Position = 0;
                ms.SetLength(0);
                ser.Serialize(ms, orig);
            }
            watch.Stop();
            Console.WriteLine("Serialize: " + watch.ElapsedMilliseconds);
            watch = Stopwatch.StartNew();
            for (int i = 0; i < LOOP; i++)
            {
                ms.Position = 0;
                ser.Deserialize(ms);
            }
            watch.Stop();
            Console.WriteLine("Deserialize: " + watch.ElapsedMilliseconds);
        }

        GC.Collect(GC.MaxGeneration, GCCollectionMode.Forced);
        GC.WaitForPendingFinalizers();
        using (var ms = new MemoryStream())
        {
            var ser = new XmlSerializer(typeof(Game));
            Console.WriteLine();
            Console.WriteLine(ser.GetType().Name);
            ser.Serialize(ms, orig);
            Console.WriteLine("Length: " + ms.Length);
            ms.Position = 0;
            ser.Deserialize(ms);

            var watch = Stopwatch.StartNew();
            for (int i = 0; i < LOOP; i++)
            {
                ms.Position = 0;
                ms.SetLength(0);
                ser.Serialize(ms, orig);
            }
            watch.Stop();
            Console.WriteLine("Serialize: " + watch.ElapsedMilliseconds);
            watch = Stopwatch.StartNew();
            for (int i = 0; i < LOOP; i++)
            {
                ms.Position = 0;
                ser.Deserialize(ms);
            }
            watch.Stop();
            Console.WriteLine("Deserialize: " + watch.ElapsedMilliseconds);
        }

        GC.Collect(GC.MaxGeneration, GCCollectionMode.Forced);
        GC.WaitForPendingFinalizers();
        using (var ms = new MemoryStream())
        {
            var ser = new DataContractSerializer(typeof(Game));
            Console.WriteLine();
            Console.WriteLine(ser.GetType().Name);
            ser.WriteObject(ms, orig);
            Console.WriteLine("Length: " + ms.Length);
            ms.Position = 0;
            ser.ReadObject(ms);

            var watch = Stopwatch.StartNew();
            for (int i = 0; i < LOOP; i++)
            {
                ms.Position = 0;
                ms.SetLength(0);
                ser.WriteObject(ms, orig);
            }
            watch.Stop();
            Console.WriteLine("Serialize: " + watch.ElapsedMilliseconds);
            watch = Stopwatch.StartNew();
            for (int i = 0; i < LOOP; i++)
            {
                ms.Position = 0;
                ser.ReadObject(ms);
            }
            watch.Stop();
            Console.WriteLine("Deserialize: " + watch.ElapsedMilliseconds);
        }

        GC.Collect(GC.MaxGeneration, GCCollectionMode.Forced);
        GC.WaitForPendingFinalizers();
        using (var ms = new MemoryStream())
        {
            var ser = new NetDataContractSerializer();
            Console.WriteLine();
            Console.WriteLine(ser.GetType().Name);
            ser.Serialize(ms, orig);
            Console.WriteLine("Length: " + ms.Length);
            ms.Position = 0;
            ser.Deserialize(ms);

            var watch = Stopwatch.StartNew();
            for (int i = 0; i < LOOP; i++)
            {
                ms.Position = 0;
                ms.SetLength(0);
                ser.Serialize(ms, orig);
            }
            watch.Stop();
            Console.WriteLine("Serialize: " + watch.ElapsedMilliseconds);
            watch = Stopwatch.StartNew();
            for (int i = 0; i < LOOP; i++)
            {
                ms.Position = 0;
                ser.Deserialize(ms);
            }
            watch.Stop();
            Console.WriteLine("Deserialize: " + watch.ElapsedMilliseconds);
        }

        GC.Collect(GC.MaxGeneration, GCCollectionMode.Forced);
        GC.WaitForPendingFinalizers();
        {
            var sb = new StringBuilder();
            var ser = new JavaScriptSerializer();
            Console.WriteLine();
            Console.WriteLine(ser.GetType().Name);
            ser.Serialize(orig, sb);
            Console.WriteLine("Length: " + sb.Length);
            ser.Deserialize(sb.ToString(), typeof(Game));

            var watch = Stopwatch.StartNew();
            for (int i = 0; i < LOOP; i++)
            {
                sb.Length = 0;
                ser.Serialize(orig, sb);
            }
            watch.Stop();
            string s = sb.ToString();
            Console.WriteLine("Serialize: " + watch.ElapsedMilliseconds);
            watch = Stopwatch.StartNew();
            for (int i = 0; i < LOOP; i++)
            {
                ser.Deserialize(s, typeof(Game));
            }
            watch.Stop();
            Console.WriteLine("Deserialize: " + watch.ElapsedMilliseconds);
        }

        GC.Collect(GC.MaxGeneration, GCCollectionMode.Forced);
        GC.WaitForPendingFinalizers();
        using (var ms = new MemoryStream())
        {
            var ser = CreateProto();
            Console.WriteLine();
            Console.WriteLine("(protobuf-net v2)");
            ser.Serialize(ms, orig);
            Console.WriteLine("Length: " + ms.Length);
            ms.Position = 0;
            ser.Deserialize(ms, null, typeof(Game));

            var watch = Stopwatch.StartNew();
            for (int i = 0; i < LOOP; i++)
            {
                ms.Position = 0;
                ms.SetLength(0);
                ser.Serialize(ms, orig);
            }
            watch.Stop();
            Console.WriteLine("Serialize: " + watch.ElapsedMilliseconds);
            watch = Stopwatch.StartNew();
            for (int i = 0; i < LOOP; i++)
            {
                ms.Position = 0;
                ser.Deserialize(ms, null, typeof(Game));
            }
            watch.Stop();
            Console.WriteLine("Deserialize: " + watch.ElapsedMilliseconds);
        }

        Console.WriteLine();
        Console.WriteLine("All done; any key to exit");
        Console.ReadKey();
    }
    static TypeModel CreateProto()
    {
        var meta = TypeModel.Create();
        meta.Add(typeof(Game), false).Add(Array.ConvertAll(typeof(Game).GetProperties(),prop=>prop.Name));
        meta.Add(typeof(Player), false).Add(Array.ConvertAll(typeof(Player).GetProperties(),prop=>prop.Name));
        return meta.Compile();
    }
}

[Serializable, DataContract]
public partial class Game
{
    [DataMember]
    public bool Finished { get; set; }
    [DataMember]
    public Guid GameGUID { get; set; }
    [DataMember]
    public long GameID { get; set; }
    [DataMember]
    public bool GameSetup { get; set; }
    [DataMember]
    public Nullable<int> MaximumCardsInDeck { get; set; }
    [DataMember]
    public Player Player { get; set; }
    [DataMember]
    public Player Player1 { get; set; }
    [DataMember]
    public bool Player1Connected { get; set; }
    [DataMember]
    public bool Player1EnvironmentSetup { get; set; }
    [DataMember]
    public long Player1ID { get; set; }
    [DataMember]
    public int Player1Won { get; set; }
    [DataMember]
    public bool Player2Connected { get; set; }
    [DataMember]
    public bool Player2EnvironmentSetup { get; set; }
    [DataMember]
    public long Player2ID { get; set; }
    [DataMember]
    public int Player2Won { get; set; }
    [DataMember]
    public int Round { get; set; }
    [DataMember]
    public Nullable<int> RoundsToWin { get; set; }
    [DataMember]
    public bool Started { get; set; }
    [DataMember]
    public string StateXML { get; set; }
    [DataMember]
    public Nullable<DateTime> TimeEnded { get; set; }
    [DataMember]
    public Nullable<int> TimeLimitPerTurn { get; set; }
    [DataMember]
    public byte[] TimeStamp { get; set; }
    [DataMember]
    public Nullable<DateTime> TimeStarted { get; set; }
}
[Serializable, DataContract]
public class Player
{
    [DataMember]
    public string Name { get; set; }
}

Answer 4

I also have benchmarks for different serializers in .NET that show @Marc Gravell's binary protobuf-net serializer as the clear winner. Although I maintain the fastest text serializers .NET which come closest to matching it and are also much faster than all the serializers that come in BCL in .NET.

These benchmarks are based on the Nortwind sample Database from Microsoft and show how much slower each serializer is compared to Protobuf-net's.

ProtoBuf.net(v1)                      1x
ServiceStack TypeSerializer           2.23x
ServiceStack JsonSerializer           2.58x
Microsoft DataContractSerializer      6.93x
NewtonSoft.Json                       7.83x
Microsoft BinaryFormatter             9.21x
Microsoft JsonDataContractSerializer  9.31x

The full benchmarks are available here

So if you prefer/need to use a fast text-serializer here are links to Service Stack's open source text serializers:

• JSON Serializer
• JSV Serializer

By the way the Microsoft's JavaScriptSerializer showed the worst performance and at times were 40x-100x slower than protobuf-nets. Took it out because they were slowing down my benchmarks :)

Answer 5

I've modified @Marc's benchmark source code and added results for ServiceStack's JSV and JSON Serializers Here are the results on my 3yo iMac:

BinaryFormatter
Length: 1313
Serialize: 3959
Deserialize: 3395

XmlSerializer
Length: 1049
Serialize: 1710
Deserialize: 2716

DataContractSerializer
Length: 911
Serialize: 712
Deserialize: 2117

NetDataContractSerializer
Length: 1138
Serialize: 1093
Deserialize: 4825

TypeSerializer
Length: 431
Serialize: 496
Deserialize: 887

JsonSerializer
Length: 507
Serialize: 558
Deserialize: 1213

Here is the source code that I added to @Marc's benchmark above.

GC.Collect(GC.MaxGeneration, GCCollectionMode.Forced);
GC.WaitForPendingFinalizers();
var sbJsv = new StringBuilder(4096);
using (var sw = new StringWriter(sbJsv))
{
    Console.WriteLine();
    Console.WriteLine(typeof(TypeSerializer).Name);
    TypeSerializer.SerializeToWriter(orig, sw);
    var jsv = sbJsv.ToString();
    Console.WriteLine("Length: " + sbJsv.Length);
    TypeSerializer.DeserializeFromString<Game>(jsv);

    var watch = Stopwatch.StartNew();
    for (int i = 0; i < LOOP; i++)
    {
        sbJsv.Length = 0;
        TypeSerializer.SerializeToWriter(orig, sw);
    }
    watch.Stop();
    Console.WriteLine("Serialize: " + watch.ElapsedMilliseconds);

    watch = Stopwatch.StartNew();
    for (int i = 0; i < LOOP; i++)
    {
        TypeSerializer.DeserializeFromString<Game>(jsv);
    }
    watch.Stop();
    Console.WriteLine("Deserialize: " + watch.ElapsedMilliseconds);
}

GC.Collect(GC.MaxGeneration, GCCollectionMode.Forced);
GC.WaitForPendingFinalizers();
var sbJson = new StringBuilder(4096);
using (var sw = new StringWriter(sbJson))
{
    Console.WriteLine();
    Console.WriteLine(typeof(JsonSerializer).Name);
    JsonSerializer.SerializeToWriter(orig, sw);
    var json = sbJson.ToString();
    Console.WriteLine("Length: " + sbJson.Length);
    JsonSerializer.DeserializeFromString<Game>(json);

    var watch = Stopwatch.StartNew();
    for (int i = 0; i < LOOP; i++)
    {
        sbJson.Length = 0;
        JsonSerializer.SerializeToWriter(orig, sw);
    }
    watch.Stop();
    Console.WriteLine("Serialize: " + watch.ElapsedMilliseconds);

    watch = Stopwatch.StartNew();
    for (int i = 0; i < LOOP; i++)
    {
        JsonSerializer.DeserializeFromString<Game>(json);
    }
    watch.Stop();
    Console.WriteLine("Deserialize: " + watch.ElapsedMilliseconds);
}

Note: I couldn't get a hold of @Marc's protobuf-net v2 r352 dlls he used for this so I've had to comment protobuf-net benchmarks out.