Determining which locks are most contended?

Question

Our application has ~10 threads doing separate tasks (no thread pools). We are not experiencing deadlock, but are always trying to lower latency to respond to a request so we are interested in determining which locks are the most contended. jconsole shows how often threads are blocked, and it isn't very often, but we still want to know which locks are the most contended.

We're running using the Sun JVM, so JLA from IBM is not useful, and we aren't running on Solaris so we can't use dTrace.

EDIT: I want to do this observation in production, where a profiler would slow the app unacceptably. This is a trading system, if we are slow, we lose money, so we don't run profilers in production. It is also quite hard to simulate the many exchanges we talk to in a performance test.

Answer 1

For similar problem in database, we log a line just before requesting and immediately after acquiring a lock. We also log one after releasing. We then post-process this data to generate the kind of stats you are looking for.

EDIT: On top of a developed system, AspectJ might be a good option to generate the logs.

Answer 2

Ted, I sympathize with your situation, but when performance is that critical, I'd recommend that you bite the bullet and simulate.

It shouldn't be as hard as you fear: instead of trying to generate message flow from your exchanges, why not record the incoming flow and replay it back on the simulation?

Without something like this, you're always going to be running into the Heisenberg problem: affecting the system you're measuring.