Information about PTE's (Page Table Entries) in Windows

Question

In order to find more easily buffer overflows I am changing our custom memory allocator so that it allocates a full 4KB page instead of only the wanted number of bytes. Then I change the page protection and size so that if the caller writes before or after its allocated piece of memory, the application immediately crashes.

Problem is that although I have enough memory, the application never starts up completely because it runs out of memory. This has two causes:

• since every allocation needs 4 KB, we probably reach the 2 GB limit very soon. This problem could be solved if I would make a 64-bit executable (didn't try it yet).
• even when I only need a few hundreds of megabytes, the allocations fail at a certain moment.

The second problem is the biggest one, and I think it's related to the maximum number of PTE's (page table entries, which store information on how Virtual Memory is mapped to physical memory, and whether pages should be read-only or not) you can have in a process.

My questions (or a cry-for-tips):

• Where can I find information about the maximum number of PTE's in a process?
• Is this different (higher) for 64-bit systems/applications or not?
• Can the number of PTE's be configured in the application or in Windows?

Thanks,

Patrick

PS. note for those who will try to argument that you shouldn't write your own memory manager:

• My application is rather specific so I really want full control over memory management (can't give any more details)
• Last week we had a memory overwrite which we couldn't find using the standard C++ allocator and the debugging functionality of the C/C++ run time (it only said "block corrupt" minutes after the actual corruption")
• We also tried standard Windows utilities (like GFLAGS, ...) but they slowed down the application by a factor of 100, and couldn't find the exact position of the overwrite either
• We also tried the "Full Page Heap" functionality of Application Verifier, but then the application doesn't start up either (probably also running out of PTE's)

Answer 1

There is what i thought was a great series of blog posts by Mark Russinovich on technet called "Pushing the limits of Windows..."

http://blogs.technet.com/markrussinovich/archive/2008/07/21/3092070.aspx

It has a few articles on virtual memory, paged nonpaged memory, physical memory and others.

He mentions little utilities he uses to take measurements about a systems resources.

Hopefully you will find your answers there.

Answer 2

In order to find more easily buffer overflows I am changing our custom memory allocator so that it allocates a full 4KB page instead of only the wanted number of bytes.

This has already been done. Application Verifier with PageHeap.

Info on PTEs and the Memory architecture can be found in Windows Internals, 5th Ed. and the Intel Manuals.

Is this different (higher) for 64-bit systems/applications or not?

Of course. 64bit Windows has a much larger address space, so clearly more PTEs are needed to map it.

Where can I find information about the maximum number of PTE's in a process?

This is not so important as the maximum amount of user address space available in a process. (The number of PTEs is this number divided by the page size.)

This is 2GB on 32 bit Windows and much bigger on x64 Windows. (The actual number varies, but it's "big enough").

Problem is that although I have enough memory, the application never starts up completely because it runs out of memory.

Are you a) leaking memory? b) using horribly inefficient algorithms?

Answer 3

Take a look at the implementation of OpenBSD malloc. Much of the same ideas (and more) implemented by very skilled folk.

Answer 4

A shotgun approach is to allocate those isolated 4KB entries at random. This means that you will need to rerun the same tests, with the same input repeatedly. Sometimes it will catch the error, if you're lucky.

A slightly smarter approach is to use another algorithm than just random - e.g. make it dependent on the call stack whether an allocation is isolated. Do you trust std::string users, for instance, and suspect raw malloc use?