Read data directly from file to RAM in C++

Question

Hi all,

Is there a way to directly read a binary file into RAM?

What I mean is, is there a way to tell the compiler, here's the file, here's the block of RAM, please put the file contents in the RAM, off you go, quickly as you can please.

Currently I'm stepping through the file with ifstream loading it into RAM (an array) 64bit block by 64 bit block. But I'm thinking that this must slow it down as it's like (here comes an analogy) using a thimble to bail the water out of a cup (the file) into a jug (the RAM) rather than just picking up the cup and tipping the entire contents into the jug in one go.

As a relative newcomer to this I may have completely the wrong idea about this - any guidence would be a great help.

I'm just after the quickest way to get a big file into RAM.

Thanks

Answer 1

mmap should work for you. Most implementations do not actually read the file until you reference the actual memory pages, but in any case you can access the file contents as if it was in RAM all the time.

Answer 2

You don't say which platform you are on, but this sounds like a job for memory-mapped files.

Answer 3

What prevents you from reading the file in one pass? Is it too big to fit in memory?

You can also use mapped file, UNIX : mmap, Windows : CreateFileMapping

Answer 4

I think what you need is memory mapped file, however the API depends on which OS you are on, in UNIX I believe it's mmap().

Answer 5

Your program is given virtual memory - some of it is mapped onto RAM, some onto swap file and you can't do much about it. However you can reduce the overhead of using the filesystem interface by just copying bigger blocks - several megabytes for example.

Answer 6

I'm not sure if this is what you're asking, but to speed up reading from a file you could buffer more bytes. the second argument of the read() method of fstream can be set to specify the input size. giving him, for example, a 32k (32*(2^10)) value could speed up the reading process. I suggest you do some benchmarking with different buffer size.

If you mean "avoiding to put the file on swap and put it ALWAYS in RAM" I think you can, since this is something the OS takes care.

Answer 7

Look at the ContentStreaming example in the DirectX SDK.

It uses the file mapping approach. Obviously, its uses the windows API, but you can find the corresponding API for other platforms as well.

The interesting thing is that it shows you the approach in context. (Reading in/paging in parts of a huge model (in this case terrain) on demand

Answer 8

Boost provides cross platform support for memory mapped files, among other things...

Boost Memory Mapped files