Tracking actively used memory in Linux programs

Question

I'd like to track how much memory various programs touch while in specific states. For instance, say I have a graphical program. It might use substantially less memory when it's minimized, since it won't redraw the window, which requires reading images and fonts and executing a large number of library functions. These objects are still accessible in memory, but they aren't actually being used.

Tools like top are of limited use since they just tell how much memory is mapped into a program's address space, and how much of it is resident in physical RAM. Likewise, leak detectors will only tell when memory is inaccessible, not if if it just isn't being used.

Is there an existing tool that can track actively used / unused memory in this way? If possible, I'd like to track usage not only on the heap but also in memory storing program/library code.

---

EDIT: I'd like to clarify: I want to know much much memory a program actually reads, writes, or executes after a certain point, i.e., once it reaches a certain state. While the number of pages in the address space and the number of resident pages are important measurements, this is not what I'm looking for.

I'm pursuing three approaches right now:

1. I've written a library that clears protection bits of all memory regions (except the stack and its own code) read from /proc/self/maps using `mprotect`. It has a segfault handler that restores protection bits and increments a counter. I load it with `LD_PRELOAD`, and it starts tracking memory access on receipt of a signal. This has produced some genuine faults with seemingly bogus addresses (they aren't stored in any registers or nearby memory at the time of the fault).
2. I've written a `purge` program which allocates and reads from memory using `mmap` until `mmap` returns an error. This hopefully forces out all of the pages from the target process, which is suspended while `purge` is running. I then count the number of page-ins when the target process is resumed, using `pidstat`. This seems to work but it is a very blunt instrument. It doesn't give any information about what pages were touched.
3. I've been told that valgrind allows you to write plug-ins that cause certain actions to be executed on certain events, e.g., memory accesses. This looks promising so far.

Answer 1

The valgrind tool cachegrind does a good job of tracking memory usage.

There are also a couple of tools which show the output of cachegrind graphically.

Edit to respond to updated info:
Before the emergence of valgrind, I used a project called mpatrol.
It uses mprotect() to make the pages read-only and tracks access to
pages. It also allows specifying when to start and stop reporting,
like after the nth malloc, among others specifications.

It may do many, or all of the features you are looking for.

One caveat, it is much slower than valgrind.

Answer 2

This can be derived from data in /proc/pid/smaps. This breaks out an RSS value for each mapped area (including stack, text/data, and anonymously mapped regions), so you can see exactly what is resident for each loaded object as well as stack and heap.

Some links that might be helpful:

• http://bmaurer.blogspot.com/2006/03/memory-usage-with-smaps.html
• http://www.pixelbeat.org/scripts/ps_mem.py