Serialization in .NET considering streamsize limitation.

Question

I am serializing an object, which has several sub-objects(which are also serializable). Can i check the size of the stream in between the serialization process? ( I just want to limit the size of the stream to which i am serializing )

Answer 1

Not using any built-in classes, but you can easily make a stream wrapper-class that has such a limit.

You would simply use it like this:

Stream limitedStream = new LimitedStream(realStream, 10 * 1024 * 1024); // 10MB max

and then serialize to that.

Answer 2

Depending on the serializer, it may be possible to drop items onto the end of the stream independently (and have it still possible to read the stream back again). Not with "pure" xml (due to the root element), but I know of at least one serializer that supports sequential object read/write (no prizes for guessing which).

If size is absolutely key, then perhaps go via a MemoryStream - something like (pseudo code)

using(Stream dest = ... )
using(MemoryStream buffer = new MemoryStream())
{
    foreach(SomeType obj in items) {
        buffer.SetLength(0); // reset to empty, but retaining buffer to reduce allocs
        someSerializer.Serialize(buffer, obj);

        if(enoughSpace) {
            // add our item
            dest.Write(buffer.GetBuffer(), 0, buffer.Length);
            // TODO: decrement space     
        } else {
            break; // or new file, whatever
        }
    }
}

Job done. This avoids writing partial objects by completing each individual object to the memory-stream (reusing the underlying memory whenever possible), and thus you only copy the data to the actual stream when you know it will fit.

Here's a complete example using protobuf-net:

using System;
using System.Collections.Generic;
using System.IO;
using System.Xml.Serialization;
using ProtoBuf;
[XmlType]      // not actually serialized as xml; simply it needs
class MyData   // a way of identifying the members, and it supports
{              // the Order property via XmlType/XmlElement
    [XmlElement(Order = 1)] public int Id { get; set; }
    [XmlElement(Order = 2)] public string Name { get; set; }
}
static class Program
{
    static IEnumerable<MyData> GetItems()
    {
        Random rand = new Random();
        int count = 0;
        while (true) // an infinite sequence of data; mwahahaahah
        {
            yield return new MyData
            {
                Id = rand.Next(0, 5000),
                Name = "Item " + count++
            };
        }
    }
    static void Main()
    {
        int space = 2048, count = 0;
        long checksum = 0;
        using(Stream dest = File.Create("out.bin"))
        using(MemoryStream buffer = new MemoryStream())
        {
            foreach (MyData obj in GetItems())
            {
                buffer.SetLength(0); // reset to empty, but retaining buffer to reduce allocs
                Serializer.SerializeWithLengthPrefix(buffer, obj, PrefixStyle.Base128, 1);
                int len = (int)buffer.Length;
                if(buffer.Length <= space) {
                    // add our item
                    dest.Write(buffer.GetBuffer(), 0, len);
                    space -= len;
                    checksum += obj.Id;
                    count++;
                } else {
                    break; // or new file, whatever
                }
            }
        }
        Console.WriteLine("Wrote " + count + " objects; chk = " + checksum);

        using (Stream source = File.OpenRead("out.bin"))
        {
            count = 0;
            checksum = 0;
            foreach (MyData item in
                Serializer.DeserializeItems<MyData>(source, PrefixStyle.Base128, 1))
            {
                count++;
                checksum += item.Id;
            }
        }

        Console.WriteLine("Read " + count + " objects; chk = " + checksum);
    }
}

You may be able to extend this approach to other serializers (such as BinaryFormatter or DataContractSerializer), but you'll presumably have to write your own length-prefix (at write), consume it (at read) and limit the data. Not rocket science, but not entirely trivial.