high sys cpu and high ithr in pmap on solaris

Question

I have a mysql database running on solaris, from mpstat, I could see a thread has very high ithr and high sys cpu. How can I find where the ithr comes from and why the sys cpu is relatively high?

CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl

0 0 0 0 440 146 269 1 17 52 0 833 0 1 0 99 .......................................................................

7 0 0 0 19798 19738 914 22 4 242 0 3569 5 14 0 81 ........................................................................

15 0 0 0 1005 0 2035 2 0 52 0 8912 12 3 0 85

CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl

0 0 0 0 632 143 678 1 26 46 0 2243 4 2 0 94 ........................................................................

7 0 0 0 21068 20978 1409 27 9 276 0 5591 9 15 0 76

.................................................................

15 0 0 0 531 0 1071 2 2 45 0 4707 7 2 0 91

---

Answer 1

The Sun Studio tools include a performance analyzer that is designed to take advantage of all the latest features in Solaris. You could try using it to collect data on the mysql server. We're getting close to the next official release, but you can use a recent "express" release to check out the latest features. It's not open source, but it's free to use. http://developers.sun.com/sunstudio/downloads/express/index.jsp

Answer 2

That's probably a NIC's interrupt overhead. mpstat will show high values in ithr for the CPU's handling any NICs or HBAs, and a correspondingly high sys time. If you have Solaris 10, intrstat will show exactly how much time is spent handling hardware interrupts per CPU.