Memory usage when files are loaded via mmap()

views:

679

answers:

+1 Q:

Memory usage when files are loaded via mmap()

Hi,

Wanted to know what if there is a relationship between the maximum amount of memory that can be used to map a file through mmap() and the size of the RAM in a linux box. I tried to memory map some files , and I found that I am not able to map any more files when the "Mapped" usage comes close to the "MemTotal" ( viewed via cat /proc/meminfo).

From /proc/meminfo on the 64 bit linux box :

MemTotal: 32909628 kB

MemFree: 221744 kB

Buffers: 1800 kB

Mapped: 31642928 kB

CommitLimit: 38012616 kB

Committed_AS: 42641120 kB

VmallocTotal: 536870911 kB

VmallocUsed: 299920 kB

VmallocChunk: 536568999 kB

So, am I right in assuming that the "Mapped" usage would never exceed the "MemTotal". The confusion I have is I was of the opinion that the memory is always virtual when we mmap a file.

Thanks!

+1 A:

No, unless you have memlock()ed yourself into memory, you are most likely hitting an OS mapped address space limit. The fact that it's neatly coming out to 32GB, same as your RAM, is likely a coincidence.

Edit: Actually, if you're using MAP_PRIVATE then you may indeed be hitting memory limits (MAP_PRIVATE is accounted for as a private allocation, and so there needs to be physical RAM or swap to cover it unless aggressive overcommit is enabled). Try using MAP_SHARED if you're not already.

bdonlan 2009-08-06 14:13:41

Added some more info in my question. By OS mapped address space limit, do you mean the virtual memory limit. But I seem to have a lot of virtual memory at that time.

sim 2009-08-06 14:26:41

I mean the limit of how much address space a single process can have mapped at a time.

bdonlan 2009-08-06 14:35:49

I would think that would be a lot in a 64 bit linux box. My program loads only a 1.5 GB file to memory.

sim 2009-08-06 14:43:34

Wait, is your program running in 32-bit? It's only mapping 1.5G here?

bdonlan 2009-08-06 15:06:16

I ran a bunch of programs , each loading up a different 1.5 GB file. Yes, my program is running in 32-bit. But the heap and non heap size of each of these is not more than 2mb.

sim 2009-08-06 15:23:02

see my edit - just realized that there's a difference with MAP_PRIVATE

bdonlan 2009-08-06 15:34:18

I used READ_WRITE. So, I guess the mapped usage would be limited to the amount of physical memory that the box has irrespective of the mapped mode.

sim 2009-08-08 12:34:10

What's impostant is whether MAP_SHARED or MAP_PRIVATE is in the flags bitmask. MAP_SHARED with READ_WRITE will save the changes back to the disk file, and so there _shouldn't_ be a memory limit. MAP_PRIVATE will allocate memory for the changed pages, however.

bdonlan 2009-08-08 18:26:10

ansaurus

tags:

views:

answers:

Memory usage when files are loaded via mmap()

related questions