is there any way to reserve memory space to be used later by default Windows Memory Manager so that my application won't run out of memory if my program don't use space more than I have reserved at start of my program?
+1
A:
There is no point in doing this kind of thing when you have virtual memory.
Paul R
2010-03-22 14:28:08
I just want to do some optimization, virtual memory are slow since it involve disk read/write
uray
2010-03-22 14:30:55
Virtual memory is a abstraction of the memory to provide applications with what looks like contiguous segment of memory. I don't see what this has got to do with the question. I would assume you mean paging.
Yacoby
2010-03-22 14:33:38
@uray: you need to understand how virtual memory works
Paul R
2010-03-22 14:33:38
I would argue that there is a point. Virtual memory is still a limited resource.
John Burton
2010-03-22 14:34:08
@Yacoby: the point is that pre-allocating memory will have no useful end result in a demand-paged VM OS.
Paul R
2010-03-22 14:35:04
@JB: every process has the same VM address space available - pre-allocating chunks of it doesn't buy you anything.
Paul R
2010-03-22 14:35:50
@uray: Paul's point is that the virtual memory system won't tell you when memory is full and it is time to start using the "reserve", instead it will let you keep the "reserve" and start paging. Which puts in the "using slow disk-based memory" scenario that you are trying to avoid.
dmckee
2010-03-22 14:36:19
Huh? My server here has 1GB of physical memory and 1GB of swap space. If another programs decides to allocate 1.8GB of that then my program wouldn't be able to allocate the 1.5GB it needs.... I can see why it would be handy to make sure it would be able to get the memory it's going to need before starting a job. Virtual memory doesn't in any way alter that.
John Burton
2010-03-22 14:38:06
@JB: size of swap space has little do to with it - most VM systems are demand-paged - pretty much all you allocate when you allocate memory are page-table entries.
Paul R
2010-03-22 14:42:18
BTW--This is a marked contrast to less sophisticated memory systems such as the one used by Mac System 6, where grabbing a reserve chunk of memory early so that you could release it the first time an allocation came back as "Memory unavailable" *would* let you continue (though it was safer to fail gracefully).
dmckee
2010-03-22 14:46:10
@dmckee: exactly (I remember those days !) - pre-allocation can be a useful strategy in non-VM environments, but it's counter-productive if you have VM.
Paul R
2010-03-22 14:48:22
@Paul: is there any good article about memory management on VM O/S ?
uray
2010-03-22 14:51:29
@Paul R: perhaps then you could additionally answer the follow-up question, "is it possible to reserve *and commit* memory"? Sure, virtual memory leaves the question based on an incorrect premise, that "reserving" memory for a program is possible. But JB describes a valid problem, which may well be the problem of either this questioner, or people who in future find this question by searching. If someone doesn't understand that virtual memory renders "reserving" space for a process meaningless, they also aren't going to know that the magic word to find a decent answer is "commit".
Steve Jessop
2010-03-22 14:54:19
@Steve: the terminology is probably going to be OS-specific. Some operating systems support the concept of "wiring" physical memory. I've never heard the term "commit" in this context but no doubt it's used *somewhere*.
Paul R
2010-03-22 14:58:45
@uray - there's a good article here: http://lwn.net/Articles/253361/ and the Wikipedia entry is not bad: http://en.wikipedia.org/wiki/Virtual_memory
Paul R
2010-03-22 15:00:20
... even if the answer to that problem is "Sorry, you can't. You just have to allocate it, force it to commit by reading to it, and then your program have its own heap management to allocate from that chunk". That's better than "there is no point doing it, you don't have a problem", when in point of fact you do have a problem ;-)
Steve Jessop
2010-03-22 15:00:45
"no doubt it's used somewhere". Windows calls it committing, for example the MEM_COMMIT argument to VirtualAlloc. That's to back the virtual address range with *something*, not necessarily physical RAM. Could just be swap/pagefile. But I see that you've since teased out of the questioner that the intention is some kind of vague performance improvement, so in this instance I think you're quite right and there's no point. If JB wants his suggestion considered it's a different question :-)
Steve Jessop
2010-03-25 17:48:30
A:
Not implicitly. There is no way to tell the C memory manager (the one which gives you memory when you call malloc or new) to reserve space.
What you can do is use your own specialized allocator or buffer pool and manage the allocations and memory reservation manually.
Edit -
On windows you can use VirtualAllocEx and VirtualFreeEx to ask windows to reserve memory pages for you. But there is no way to tell malloc and new to use this memory. If you choose to go this path you'll need to implement your own memory manager.
shoosh
2010-03-22 14:29:09
I'am not asking strictly to C memory manager, is there any Win32/64 to do so?
uray
2010-03-22 14:31:32
A:
This isn't even remotely portable but in many systems memory you allocate is not returned to the operating system when you free it, it's just put back on the free list for that program.
So if you allocate a block big enough when your program starts up and then free it immediately then that memory will be available from then onwards for the memory allocator to hand out within that program.
You'd certainly have to make sure that your platform worked in that way though. At best it's unportable though, but at worst it shouldn't hurt anything.
John Burton
2010-03-22 14:33:18
This is more likely to have the *opposite* effect in a VM OS. You'll just be wasting page table entries and TLB entries and you will start to page sooner. The whole pre-allocation idea is seriously *old skool* - from the days when PC operating systems didn't have VM, just a finite pool of physical memory, shared between all processes.
Paul R
2010-03-22 14:44:51
Huh? VM isn't magical, there is still a finite amount of memory resources of any kind available. If you preallocate some of the resources to a specific process then you know that later on you'll have the memory you need in that process. Yes of course it will likely cause extra paging but that's the cost of reserving some of your memory on any system. I fail to see how this wouldn't work unless the standard library releases allocated memory back to the OS
John Burton
2010-03-22 15:22:23
A:
At any given moment, your application doesn't necessarily have any physical memory.
If your app is waiting on an IO operation for a while and the user starts editing a video, your app may disappear from physical memory altogether. It only exists in the page file. When the IO operation completes and one of your threads needs to start running again, you get some physical memory back.
All you really get to allocate is parts of your process's address space. Parts of it may be mapped to files (such as your EXE or any native DLLs that didn't need to be rebased). Other parts may be "memory allocation", but really that just means they are mapped to places in the page file.
None of it is "really" memory in any reliable sense.
The thing you can certainly run out of is room in your address space. If you allocate 100MB chunks a few times, plus a few smaller ones, it won't be long before you've fragmented the address space (which by default only gives you enough room for about 20 such chunks, and any given chunk has to fit into one contiguous region).
Daniel Earwicker
2010-03-22 14:43:22
A:
Though it's usually a really bad idea (as others have already pointed out), yes Windows does allow it. Look up VirtualLock if you really insist. At least 99% of the time, this is the wrong thing to do, but it's always possible (though extremely unlikely) that what you're doing falls into the fraction of a percent of things that justify it.
Jerry Coffin
2010-03-22 15:07:35
How does this answer the question? VirtualLock() simply keeps pages of memory from being swapped out to disk. It doesn't reserve any memory. Regardless of whether pages are locked, the amount of virtual memory available to the process remains unchanged. VirtualLock() is primarily useful for security and real-time related use cases to keep potentially sensitive data from being written to disk and to potentially improve determinism at run-time, respectively.
Void
2010-03-22 18:13:55