Possible ways to persist a string array to a stream without using serialization?

Question

What are possible ways to save string arrays to a stream without using serialization?

I'm particularly interested in strings since their lengths may vary. I also should be able to restore the array from stream. And, more importantly, I would like to be able to read only slices of an array without reading full array into memory, because potentially my arrays can be huge.

P.S. I know that there exist databases, that I shouldn't reinvent the wheel, etc, but I have my reasons to opt for hand made solution.

Thank you.

Answer 1

Well, saving data to a stream is serialization; the real trick is: what kind. For example, I assume you're talking about things like XmlSerializer or BinaryFormatter that require you to deserialize the whole thing, but that isn't always necessary.

By writing each string with a length-prefix, you should be able to seek past items you don't want pretty easily. The other option is to write (separately) an index of offsets, but that is sometimes overkill.

As a basic example, s here is "jkl", without it reading the entire stream or deserializing the unwanted strings; note that it could be optimized by (for example) using a variable-length encoding for the int (length), which would also fix the current assumption that endianness is the same between reader and writer:

static void Main()
{
    byte[] raw;
    using (MemoryStream ms = new MemoryStream())
    {
        // serialize all
        List<string> data = new List<string> {
           "abc", "def", "ghi", "jkl", "mno", "pqr" };
        foreach (string s in data)
        {
            byte[] buffer = Encoding.UTF8.GetBytes(s);
            byte[] lenBuffer = BitConverter.GetBytes(buffer.Length);
            ms.Write(lenBuffer, 0, lenBuffer.Length);
            ms.Write(buffer, 0, buffer.Length);
        }
        raw = ms.ToArray();
    }
    using (MemoryStream ms = new MemoryStream(raw))
    {
        int offset = 3, len;
        byte[] buffer = new byte[128];
        while (offset-- > 0)
        {
            Read(ms, ref buffer, 4);
            len = BitConverter.ToInt32(buffer, 0);
            ms.Seek(len, SeekOrigin.Current); // assume seekable, but
                                              // easy to read past if not
        }
        Read(ms, ref buffer, 4);
        len = BitConverter.ToInt32(buffer, 0);
        Read(ms, ref buffer, len);
        string s = Encoding.UTF8.GetString(buffer, 0, len);
    }
}
static void Read(Stream stream, ref byte[] buffer, int count)
{
    if (buffer.Length < count) buffer = new byte[count];
    int offset = 0;
    while (count > 0)
    {
        int bytes = stream.Read(buffer, offset, count);
        if (bytes <= 0) throw new EndOfStreamException();
        offset += bytes;
        count -= bytes;
    }
}