How big can a memory-mapped file be?

Question

What limits the size of a memory-mapped file? I know it can't be bigger than the largest continuous chunk of unallocated address space, and that there should be enough free disk space. But are there other limits?

Answer 1

There should be no other limits. Aren't those enough? ;-)

Answer 2

Under Windows: "The size of a file view is limited to the largest available contiguous block of unreserved virtual memory. This is at most 2 GB minus the virtual memory already reserved by the process. "

From MDSN.

I'm not sure about LINUX/OSX/Whatever Else, but it's probably also related to address space.

Answer 3

You're being too conservative: A memory-mapped file can be larger than the address space. The view of the memory-mapped file is limited by OS memory constraints, but that's only the part of the file you're looking at at one time. (And I guess technically you could map multiple views of discontinuous parts of the file at once, so aside from overhead and page length constraints, it's only the total # of bytes you're looking at that poses a limit. You could look at bytes [0 to 1024] and bytes [2⁴⁰ to 2⁴⁰ + 1024] with two separate views.)

In MS Windows, look at the MapViewOfFile function. It effectively takes a 64-bit file offset and a 32-bit length.

Answer 4

With FUSE on linux you could also make an in-memory filesystem that extends to disk on demand. I'm not sure that qualifies as memory mapped, and the distinction gets kind of blurred.

Answer 5

Wikipedia entry on the subject: http://en.wikipedia.org/wiki/Memory-mapped_file

Answer 6

This has been my experience when using memory-mapped files under Win32:

If your map the entire file into one segment, it normally taps out at around 750 MB, because it can't find a bigger contiguous block of memory. If you split it up into smaller segments, say 100MB each, you can get around 1500MB-1800MB depending on what else is running.

If you use the /3g switch you can get more than 2GB up to about 2700MB but OS performance is penalized.

I'm not sure about 64-bit, I've never tried it but I presume the max file size is then limited only by the amount of physical memory you have.