Binary files and cross platform compatability

Question

I have written a C++ library that saves my data (a collection of custom structs etc) into a binary file. I currently use (i.e. CREATE and CONSUME) the files locally, on my Windows (XP) machine. For simplicity, lets think of the library in two parts: a WRITER (Creates the files) and a READER or CONSUMER (simply reads data from the files).

Recently though, I would like to also CONSUME (i.e. read) the data files I have CREATED on my XP machine, on my Linux machine. I must point out at this stage that both machines are PCs (so have the same ENDIANess etc).

I can build a reader (and compile for Linux [Ubuntu 9.10 to be precise]), since I am the library creator. My question, before I embark down this road (of building the reader etc) is:

Assuming I have succesfully built the reader for Linux,

Can I simply copy accross, files that were CREATED on the windows (XP) machine to the Linux (Ubuntu 9.10) machine and use the Linux reader to successfully read the copied over file?

Answer 1

For the files to be binary compatible:

• endianness must match (as it does for you)
• bitfield packing order must be the same
• sizes and signedness of types must be the same
• the compiler must make the same decisions about padding and alignment

It's certainly possible for all of these conditions to be fulfilled, or for you to not happen to be hitting any cases for which they are not. At the very least, though, I'd add some sanity checks and/or sentinel members to detect problems.

Answer 2

Binary files should be compatible across machines with the same endianess.

The issue you may have in your code is the size of ints, you can't necessarily assume that the compiler on different OS's has the same size int. So either copy blocks of bytes and cast them, or use int16, int32 etc.

Answer 3

If:

• the machines have the same endianess (as you stated they have) and
• you do open the streams in binary mode, as text mode might do funny things e.g. with line-ends and
• you have programmed cleanly so you don't stumble over implementation-defined stuff like alignments, data type sizes, and struct packing,

then yes, your files should be portable.

The third bullet point is what makes a file format a "portable" one. Depending on what kind of data you have in your structs, it can be very easy or a bit tricky. Bitfields, or data being reinterpreted from a different type are especially tricky.

Answer 4

You might consider taking a look at the Boost Serialization Library. A lot of thought has been put into it, and it will handle many of the potential cross-platform incompatibilities for you. Of course, it's possible that it's overkill for your particular use case, especially if you've already got your writers & readers implemented.