I am trying to find the bottleneck of a multithreaded service. I used ab with -c 100 and observed TPS around 350. The question is the peak CPU usage is 70%, memory is 10%. So it looks like it's not cpu or memory bound. It's not doing disk or network I/O. How do I determine what the bottleneck is?
A:
Do you have threads in deadlock waiting for each other to finish? Do you have threads waiting on database locks? Waiting threads can often be a bottleneck in that the system appears to slow down or wait, but you won't always see resources consumed.
Chris Aldrich
2010-06-28 19:37:25
@Chis Aldrich - how can cseric find those threads?
MaasSql
2010-06-28 19:38:46
@MaasSql - I know it is ridiculous expensive but the VS2010 profiler has some awesome features that can help you find dead-locks.
ChaosPandion
2010-06-28 19:46:07
As ChaosPandion noted you need to use a profiler (or potentially a code debugger that may show the threading). But the profiler is the best way to go. Without knowing what system is being used or what language it is harder to recommend a profiler tool. At my company, we tend to stick to Java and IBM stack products, so we stick to IBM profilers. Hope this was helpful.
Chris Aldrich
2010-06-28 19:58:56