Memory overwrite problem.

Question

Hello,

I have one C code app. which i was building using MS-VS2005. I had one output data buffer which was being allocated dynamically using malloc.

For some test cases, the memory size which was being malloc'd was falling short than the the actual output size in bytes which was generated. That larger sized output was written into the smaller sized buffer causing buffer overflow. As a result of which the test-run was crashing with MSVS-2005 showing up a window "Heap corruption ...."

I knew it had to do with some dynamic memory allocation, but i took long time to actually find the root cause, as i did not doubt the memory allocation because i was allocating large enough size necessary for the output. But one particular test case was generating more output than what i had calculated, hence the resulting crash.

My question is:

1.) What tools i can use to detect such dynamic memory buffer over-flow conditions. Can they also help detect any buffer overflow conditions(irrespective of whether the buffer/array is on heap, stack, global memory area)?

2.) Will memory leak tools(like say Purify) or code analysis tools like lint, klocworks would have helped in particular case? I believe they have to be run time analysis tools.

Thank you.

-AD.

Answer 1

PC-Lint can catch some forms of malloc/new size problems, but I'm not sure if it would have found yours.

VS2005 has good buffer overflow checking for stack objects in debug mode (runs at end of function). And it does periodic checking of the heap.

As for it helping to track down where the problems occurred, this is where I tend to start using macro's to dump all allocations to match against the corrupted memory later (when it's detected).

Painful process, so I'm keen to learn better ways also.

Answer 2

Something you can try is allocate enough pages + 1 using VirtualAlloc, use VirtualProtect with PAGE_READONLY | PAGE_GUARD flags on the last page, then align the suspected allocation so the end of the object is near the beginning of the protected page. If all goes well you should get an access violation when the guard page is accessed. It helps if you know approximately which allocation is overwritten. Otherwise it requires overriding all allocations which may require a lot of extra memory (at least 2 pages per allocation). A variation on this technique that I'm hereby christening as "statistical page guard" is to only randomly allocate memory for a relatively small percentage of allocations in that manner to avoid large bloat for small objects. Over a large number of execution runs you should be able to hit the error. The random number generator would have to be seeded off something like time in this case. Similarly you can allocate the guard page in front of the object if you suspect an overwrite at a lower address (can't do both at the same time but possible to randomly mix up as well).

An update: it turns out the gflags.exe (used to be pageheap.exe) microsoft utility already supports "statistical page guard" so i reinvented the wheel :) All you need to do is run gflags.exe /p /enable [/random 0-100] YourApplication.exe and run your app. If you are using a custom heap or custom guards on your heap allocations then you can simply switch to using HeapAlloc at least for catching bugs and then switch back. Gflags.exe is a part of Support Tools package and can be downloaded from the microsoft download center, just do a search there.

Answer 3

One solution, which I first encountered in the book Writing Solid Code, is to "wrap" the malloc() API with diagnostic code.

First, the diagnostic malloc() arranges to allocate additional bytes for a trailing sentinel. For example, an additional four bytes following the allocated memory are reserved and contain the characters 'FINE'.

Later, when the pointer from malloc() is passed to free(), a corresponding diagnostic version of free() is called. Before calling the standard implementation of free() and relinquishing the memory, the trailing sentinel is verified; it should be unmodified. If the sentinel is modified, then the block pointer has been misused at some point subsequent to being returned from the diagnostic malloc().

There are advantages of using a memory protection guard page rather than a sentinel pattern for detecting buffer overflows. In particular, with a pattern-based method, the illegal memory access is detected only after the fact. Only illegal writes are detected by the sentinel pattern method. The memory protection method catches both illegal reads and writes, and they are detected immediately as they occur.

Diagnostic wrapper functions for malloc() can also address other misuses of malloc(), such as multiple calls to free() for same memory block. Also, realloc() can be modified to always move blocks when executed in a debugging environment, to test the callers of realloc().

In particular, the diagnostic wrappers may record all of the blocks allocated and freed, and report on memory leaks when the program exits. Memory leaks are blocks which are allocated by not freed during the program execution.

When wrapping the malloc() API, one must wrap all of the related functions, including calloc(), realloc(), and strdup().

The typical way of wrapping these functions is via preprocessor macros:

#define malloc(s)   diagnostic_malloc(s, __FILE__, __LINE__)
/* etc.... */

If the need arises to code a call to the standard implementation (for example, the allocated block will be passed to a third-party, binary-only library which expects to free the block using the standard free() implementation, the original function names can be accessed rather than the preprocessor macro by using (malloc)(s) -- that is, place parentheses around the function name.