C#/.NET - Custom Binary File Formats - Where to Start?

Question

I need to be able to store some data in a custom binary file format. I've never designed my own file format before. It needs to be a friendly format for traveling between the C#, Java and Ruby/Perl/Python worlds.

To start with the file will consist of records. A GUID field and a JSON/YAML/XML packet field. I'm not sure what to use as delimiters. A comma, tab or newline kind of thing seems too fragile. What does Excel do? or the pre-XML OpenOffice formats? Should you use ASCII chars 0 or 1. Not sure where to begin. Any articles or books on the topic?

This file format may expand later to include a "header section".

Note: To start with I'll be working in .NET, but I'd like the format to be easily portable.

UPDATE:
The processing of the "packets" can be slow, but navigation within the file format cannot. So I think XML is off the table.

Answer 1

ASCII chars 0 or 1 each take up several bits (just like any other character), so if you're storing it like that your "binary" file will be several times larger than it should be. At text file of zeros and ones is not exactly a binary file :)

You can use the BinaryWriter to write raw data directly to a file stream. The only part you need to figure out is translating your in-memory format (usually some kind of object graph) into a byte sequence that the BinaryWriter can consume.

However, if your primary interest is portability, I recommend against a binary format at all. XML is precisely designed to solve the portability and interoperability problem. It's verbose and weighty as a file format, but that's the trade-off you make to get those problems solved for you. If a human-readable format is off the table, Marc's answer is the way to go. No need to reinvent the portability wheel!

Answer 2

How about looking at using "protocol buffers"? Designed as an efficient, portable, version-tolerant general purpose binary format, it gives you C++, Java and Python in the google library, and C#, Perl, Ruby and others in the community ports?

Note that Guid doesn't have a specific data type, but you can shim it as a message with (essentially) a byte[].

Normally for .NET work, I'd recommend protobuf-net (but as the author, I'm somewhat biased) - however, if you intend to use other languages later you might do better (long term) using Jon's dotnet-protobufs; that'll give you a familiar API accross the platforms (where-as protobuf-net uses .NET idioms).

Answer 3

It depends on what type of data you will be writing in to the binary file and what is the purpose of the binary file. Are they class object or just record data? If it is record data i would recommend to put it in xml format. That way you can include an schema validation to validate that the file conforms with you standards. There are tools in both java and .NET to import and export data from / to xml format.

Answer 4

Suppose your format is:

    struct Format
    {
        struct Header // 1
        {
            byte a;
            bool b1, b2, b3, b4, b5, b6, b7, b8;
            string name;
        }
        struct Container // 1...*
        {
            MyTypeEnum Type;
            byte[] data;
        }
    }

    enum MyTypeEnum
    {
        Sound,
        Video,
        Image
    }

Then I'd have a sequential file with:

---

byte // a

byte // b

int // name size

char[] // name (which has the size specified above, remember a char is 16 bits in .NET)

int // MyTypeEnum type

int // data size

byte[] // data (which has the size specified above)

---

Then you can repeat the last three lines as many as you want.

To read you use the BinaryReader which has support for reading bytes, integers and series of bytes. There is also a BinaryWriter.

Further, remember that Microsoft .NET (thus on a Windows/Intel machine) is little-endian. So is the BinaryReader and BinaryWriter.