views:

149

answers:

2

I'm trying to debug a memory leak in a printer driver. I'm pretty sure it's a resource leak, not just a plain memory leak because analyzing heap with !heap -s in windbg doesn't show any increase. How do I monitor other kinds of objects with windbg? Number of GDI objects and open handles is not growing either, so what could it be?

The symptom of memory leak is that private bytes are growing up to 180Mb and then printing starts experiencing random problems.

+1  A: 

It could be direct calls to VirtualAlloc. Try to isolate the memory type using "!address -summary". Better yet, find a copy of vadump.exe from an old resource kit. It gives a more readable breakdown.

You might get some clues by diff'ing the output from two runs of windbg's !vadump command, and then dumping some of the newly allocated RAM. If you have symbol files and you dump RAM using the dps command, windbg will display symbol matches for each DWORD. In other words, if a value has a name in the symbol file, you'll see it. A nice example of that is when dumping C++ objects with vtables. The vtable has a symbol, so you'll see what type it is.

Last but not least, you could breakpoint VirtualAlloc and dump the stack for each call. Even absent a rigorous comparison between allocs and frees, you might notice an interesting call stack or allocation size. The breakpoint syntax to dump the stack and continue is bp kernel32!virtualalloc "kb;g"

Hope that helps, Gary Kratkin

Gary Kratkin
Thanks for the help. vadump.exe produces *a lot* of differences, kind of too much to look at. !address -summary shows that a region1886000 ( 25112) : 01.20% 50.05% : RegionUsageIsVADis growing. I guess this does point to VirtualAlloc. However the breakpoint in virtualalloc is not being hit at all. Does VirtualAlloc have any close relatives?
MK
Actually after diffing !address output I can see new 4Kb region being allocated in the RegionUsageIsVAD area. But how do I figure out what allocated it?
MK
A: 

Sorry, I should have specified a breakpoint on VirtualAllocEx in addition to VirtualAlloc. AFAIK, most process-initiated VAD allocations should hit the bp, except those that are implemented in the kernel, such as file mappings (CreateFileMapping/MapViewOfFile) and maybe LoadLibrary. -Gary

Gary Kratkin
I did put it in VirtualAllocEx and didn't get any hits.MapViewOfFile shows a lot of hits but none appear to contain my driver dll in the stack trace. For some reason I can't put a breakpoint on CreateFileMapping, windbg can't find that symbol.
MK
Oh, and the memory that is supposedly leaked is all 0s so that doesn't help :(
MK
It's CreateFileMappingA if the module is compiled ANSI, or CreateFileMappingW if Unicode. At this point I'd probably switch to kernel debugging and set a process-specific breakpoint on ZwAllocateVirtualMemory.
Gary Kratkin