Easiest way to combine a dataset and other data in a single file?

Question

I have a dataset in C# which I can serialise using dataset.WriteXml(filename); but I want the file to contain other data as well (essentially some meta data about the dataset).

I could add another table to the dataset which contained the metadata, but I'd prefer to keep it separate, if at all possible.

Essentially I think I want to create a 'combination of files' file, that looks something like this:

size_of_file1
file1
size_of_file2
file2
... etc

Then, I'd like to load the file into memory, and split the file into separate streams, so that I can feed the dataset into dataset.ReadXml(stream); and the metadata into something else.

Sound possible? Can anyone tell me how I can do it?

Thanks

Tim

---

PS. I would like the resultant file to remain human-readable.

Answer 1

This is possible.

You should make it a binary (non-human-readable) file, and use the BinaryWriter and BinaryReader classes to read and write the lengths.

You can read a file from it like this:

using (FileStream combinedFile = File.Open(...))
using (var binReader = new BinaryReader(combinedFile)) {
    while(!combinedFile.EOF) {
        long segmentLength = binReader.ReadInt64();

        var bytes = new byte[segmentLength];
        long totalRead = 0;
        while(bytesRead < segmentLength) {
            int read = combinedFile.Read(bytes, totalRead, Math.Min(4096, segmentLength - totalRead));
            if (read == 0)
                throw new InvalidDataException();
        }

        yield return new MemoryStream(bytes);
    }
}

EDIT To make a human-readable file, write the segment length as a fixed-length string (padded to some reasonable number of digits, such as 16), optionally followed by a newline. The maximum segment size is the length of the string.

You can read a file from it like this:

const int LengthPadding = 16

using (FileStream combinedFile = File.Open(...))
using (var binReader = new BinaryReader(combinedFile)) {
    while(!combinedFile.EOF) {
        char[] segmentLengthChars = binReader.ReadChars(16);

        long segmentLength = long.Parse(new string(segmentLengthChars));

        binReader.ReadChars(2);  //Skip the newline

        var bytes = new byte[segmentLength];
        long totalRead = 0;
        while(bytesRead < segmentLength) {
            int read = combinedFile.Read(bytes, totalRead, Math.Min(4096, segmentLength - totalRead));
            if (read == 0)
                throw new InvalidDataException();
        }

        yield return new MemoryStream(bytes);
    }
}

To write the segment length, call

binWriter.WriteChars(length.ToString().PadLeft(16, '0').ToCharArray());
binWriter.WriteChars(new char[] { '\r', '\n' });

You should explicitly pass an encoding and CultureInfo.InvariantCulture where applicable.

Answer 2

You could extend the DataSet object add your metadata as properties and serialize that...

Answer 3

If it needs to be human readable use an XML file. Store the meta data in a element and the dataset in another element.

This thread will show you how

Answer 4

If you want to simply write the XML out one node after the other like this:

DataSet ds;
// populate ds with some data
string serialized;
using (System.IO.StringWriter sw = new System.IO.StringWriter())
{
   string metaData = "<MetaData version=\"1.0\" date=\"" + System.Xml.XmlConvert.ToString(DateTime.Now) + "\">" +
      "<Detail>Some more details</Detail></MetaData>";
   sw.Write(metaData);
   ds.WriteXml(sw, System.Data.XmlWriteMode.WriteSchema);
   sw.Close();
   serialized = sw.ToString();
}

Then you can read it as a series of nodes within an XML document fragment like this, taking advantage of the dataset's ability to use an XMLReader:

using (System.IO.StringReader sr = new System.IO.StringReader(serialized))
{
   System.Xml.XmlReaderSettings xs = new System.Xml.XmlReaderSettings();
   xs.ConformanceLevel = System.Xml.ConformanceLevel.Fragment;
   System.Xml.XmlReader xr = System.Xml.XmlTextReader.Create(sr, xs);
   xr.Read();
   string metaData = xr.ReadOuterXml();
   Console.WriteLine(metaData);
   ds = new System.Data.DataSet();
   ds.ReadXml(xr);
   ds.WriteXml(Console.Out);
}