How to measure lock contention?

Question

I'm reading http://lse.sourceforge.net/locking/dcache/dcache%5Flock.html, in which spinlock time for each functions is measured:

SPINLOCKS         HOLD            WAIT
  UTIL  CON    MEAN(  MAX )   MEAN(  MAX )(% CPU)     TOTAL NOWAIT SPIN RJECT  NAME
  5.3% 16.5%  0.6us(2787us)  5.0us(3094us)(0.89%)  15069563 83.5% 16.5%    0%  dcache_lock
 0.01% 10.9%  0.2us( 7.5us)  5.3us( 116us)(0.00%)    119448 89.1% 10.9%    0%    d_alloc+0x128
 0.04% 14.2%  0.3us(  42us)  6.3us( 925us)(0.02%)    233290 85.8% 14.2%    0%    d_delete+0x10
 0.00%  3.5%  0.2us( 3.1us)  5.6us(  41us)(0.00%)      5050 96.5%  3.5%    0%    d_delete+0x94

I'd like to know where these statistics are from. I tried oprofile, but it seems oprofile cannot measure lock holding and waiting time for a specific lock. And valgrind's drd slows down applications too much, which will make the result less accurate and also consume too much time. mutrace seems good, but as the name points out, I'm afraid it can only trace mutex exclusions.

So is there any other tool, or how to use the tools I mentioned above, to get lock contention statistics?

Thanks for your reply.

Answer 1

One way to tell is just get it running, pause it, and take a random stackshot of all the threads. Then do it again, several times. Then the fraction of stack samples that terminate in locking code is the percent of time you are after, roughly. It will also tell you which locations the locking is performed in. If you're after accuracy, take more samples. This works in any language or operating system.

Answer 2

Finally I find the performance measuring tool used in the article, which needs to patch kernel .

The introduction page can be found at http://oss.sgi.com/projects/lockmeter/, and the latest kernel patch corresponds to kernel version 2.6.16, which you can download here.