What's the best way to read/write array contents from/to binary files in C#?

Question

I would like to read and write the contents of large, raw volume files (e.g. MRI scans). These files are just a sequence of e.g. 32 x 32 x 32 floats so they map well to 1D arrays. I would like to be able to read the contents of the binary volume files into 1D arrays of e.g. float or ushort (depending on the data type of the binary files) and similarly export the arrays back out to the raw volume files.

What's the best way to do this with C#? Read/Write them 1 element at a time with BinaryReader/BinaryWriter? Read them piece-wise into byte arrays with FileStream.Read and then do a System.Buffer.BlockCopy between arrays? Write my own Reader/Writer?

EDIT: It seems it's not possible to work with > 2GB arrays, but the question still stands for smaller arrays (around 256 MB or so)

Answer 1

You're not going to get arrays with more than 2GB data. From what I remember, there's a CLR limit of 1GB per object. It's possible that's been lifted for .NET 4 on 64-bit, but I haven't heard about it.

EDIT: According to this article the limit is 2GB, not 1GB - but you still won't manage more than 2GB.

Do you really have to have all the data in memory at one time? Can you work on chunks of it at a time?

EDIT: Okay, so it's now just about reading from a file into a float array? It's probably simplest to read chunks (either using BinaryReader.Read(byte[], int, int) or BinaryReader.ReadBytes(int)) and then use Buffer.BlockCopy to efficiently convert from bytes to floats etc. Note that this will be endian-sensitive, however. If you want to convert more robustly (so that you can change endianness later, or run on a big-endian platform) you'd probably want to call ReadFloat() repeatedly.

How convinced are you that you actually have a performance issue in this area of the code? It's worth doing the simplest thing that will work and then profiling it, to start with...