Is there a tried and true way of accessing specific areas of a very large (>1GB) binary file very fast in .Net

Question

Using a very large raw binary data file, I will be creating a large binary file (greater than 1 GB, usually greater than 4 GB) that contains sensor data from an arbituary number of sensors (1 to 64). The new processed file will be used by user in a GUI where they will be jumping around the file looking at different time periods or other index that may be defined. Its important that access is as smooth as possible, and I also would like to only load as much as needed (what the user sees plus maybe a bit of a buffer)

I would like to construct the processed data file such that it has an index at the beginning that is then used to jump to what ever point the person want to go.

Should I be writing this from scratch? or is there some helper libraries that may come in use?

The reason i wouldn't just tack some indexing onto the original raw data file is that it has been created in a way that is definitely not optimal for reading individual sensors (running compression, data sored in DCT blocks, etc...).

Answer 1

You can use a Memory Mapped File. In .NET 4.0 there are now wrappers: http://msdn.microsoft.com/en-us/library/system.io.memorymappedfiles.memorymappedfile.aspx

If you need to target an earlier version you will have to p/invoke the API methods yourself: http://msdn.microsoft.com/en-us/library/aa366537.aspx