Hello,
Sometimes my application consumes 100 cpu. When I investigate crush dumps in windbg I always see huge "request queue" in the log and I am very suspicious of this huge amount of requests. Definitely I want to know where they come from, but at the first place I don't quite understand what it means. Does it mean that there are lots of waiting timer callbacks or something else? If it's timer callbacks how to find their sources?
0:000> !threadpool CPU utilization 93% Worker Thread: Total: 2 Running: 0 Idle: 2 MaxLimit: 200 MinLimit: 2 Work Request in Queue: 11943 AsyncTimerCallbackCompletion TimerInfo@0697a8e8 AsyncTimerCallbackCompletion TimerInfo@12b1f320 AsyncTimerCallbackCompletion TimerInfo@0697aa08 AsyncTimerCallbackCompletion TimerInfo@0697aff0 ... AsyncTimerCallbackCompletion TimerInfo@131bc718 -------------------------------------- Number of Timers: 30 -------------------------------------- Completion Port Thread:Total: 1 Free: 1 MaxFree: 4 CurrentLimit: 1 MaxLimit: 200 MinLimit: 2
Thanks!
MORE INFO:
The application doesn't use timers directly. That's why it is quite unexpected output for me. But the application heavily uses WCF and SQL (DbConnections). I checked all alive timers in the process and I see that most of them related to DbConnectionPool, some of them related to System.Web.* classes, but I cannot find relationship b/w these timers and those "AsyncTimerCallbackCompletion TimerInfo@12b1f320"
!dumpheap -type TimerCallback
Address MT Size
012629dc 66ef7104 32
01262a24 66ef7270 20
013db814 66ef7270 20
013e1c30 66ef3f94 32
...
104ec1c0 66ef7270 20
total 74 objects
Statistics:
MT Count TotalSize Class Name
66ef3f94 4 128 System.Threading.WaitOrTimerCallback
66ef7270 35 700 System.Threading._TimerCallback
66ef7104 35 1120 System.Threading.TimerCallback
0:004> !do 104d5a20
Name: System.Threading.TimerCallback
MethodTable: 66ef7104
EEClass: 66ca2ab8
Size: 32(0x20) bytes
(C:\Windows\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
Fields:
MT Field Offset Type VT Attr Value Name
66f10770 40000ff 4 System.Object 0 instance 104d56d8 _target
66f0ffc8 4000100 8 ...ection.MethodBase 0 instance 00000000 _methodBase
...
0:004> !do 104d56d8
Name: System.Data.ProviderBase.DbConnectionPool
MethodTable: 640af150
EEClass: 63fccb00
Size: 100(0x64) bytes
(C:\Windows\assembly\GAC_32\System.Data\2.0.0.0__b77a5c561934e089\System.Data.dll)
Fields:
MT Field Offset Type VT Attr Value Name
66f12da0 400152d 44 System.Int32 1 instance 170000 _cleanupWait
640b10b4 400152e 4 ...ctionPoolIdentity 0 instance 01264c00 _identity
640aee84 400152f 8 ...ConnectionFactory 0 instance 012601d4 _connectionFactory
...
Any thoughts or suggestions?
MORE INFO (2):
0:036> !threads
ThreadCount: 36
UnstartedThread: 0
BackgroundThread: 6
PendingThread: 0
DeadThread: 6
Hosted Runtime: no
PreEmptive GC Alloc Lock
ID OSID ThreadOBJ State GC Context Domain Count APT Exception
0 1 13e0 0026cf50 a020 Enabled 00000000:00000000 00263658 0 MTA
2 2 121c 0027a9d0 b220 Enabled 00000000:00000000 00263658 0 MTA (Finalizer)
3 3 1544 0030bc48 200b020 Enabled 1fce3a1c:1fce571c 00263658 0 MTA
4 5 1578 0032af10 180b220 Enabled 00000000:00000000 00263658 0 MTA (Threadpool Worker)
5 6 1288 003217f8 200b020 Enabled 00000000:00000000 00263658 0 MTA
6 7 128c 047250a0 200b020 Enabled 00000000:00000000 00263658 0 MTA
7 8 138 047285e8 200b020 Enabled 00000000:00000000 00263658 0 MTA
8 9 1268 04749b10 200b020 Enabled 00000000:00000000 00263658 0 MTA
9 a 13c0 047459f8 200b020 Enabled 00000000:00000000 00263658 0 MTA
10 b 1110 047460a0 200b020 Enabled 00000000:00000000 00263658 0 MTA
XXXX c 0 0474cb20 9820 Enabled 00000000:00000000 00263658 0 Ukn
XXXX d 0 0474d1c8 9820 Enabled 00000000:00000000 00263658 0 Ukn
11 e ec0 0474d870 200b020 Enabled 00000000:00000000 00263658 0 MTA
12 f 104c 0473a780 200b020 Enabled 00000000:00000000 00263658 0 MTA
XXXX 10 0 0473ae28 9820 Enabled 00000000:00000000 00263658 0 MTA
13 11 1168 0474ab38 200b020 Enabled 00000000:00000000 00263658 0 MTA
14 12 774 0474af20 200b020 Enabled 00000000:00000000 00263658 0 MTA
15 13 10dc 0474b308 200b020 Enabled 00000000:00000000 00263658 0 MTA
XXXX 14 0 0474b6f0 9820 Enabled 00000000:00000000 00263658 0 MTA
16 15 14b8 0474bad8 200b020 Enabled 00000000:00000000 00263658 0 MTA
18 17 12a0 047abbb8 200b020 Enabled 00000000:00000000 00263658 0 MTA
19 20 1600 047ae2c8 200b020 Enabled 1fef569c:1fef5c94 00263658 0 MTA
20 21 c2c 047ae6b0 80a220 Enabled 00000000:00000000 00263658 0 MTA (Threadpool Completion Port)
24 22 a94 069726c8 1220 Enabled 00000000:00000000 00263658 0 Ukn
26 30 43c 0ae64e78 180b220 Enabled 00000000:00000000 00263658 0 MTA (Threadpool Worker)
29 39 1608 06a651a8 200b020 Enabled 1fefdd5c:1feffc94 12eb6258 0 MTA
30 2f 3d0 06971b10 200b020 Enabled 1ff00d90:1ff01c94 1317a028 0 MTA
31 26 13fc 06971340 200b020 Enabled 1fefb504:1fefbc94 1317a028 0 MTA
32 46 1e0 1bc5c3e0 200b020 Enabled 1fe3d8f8:1fe3e038 1317a028 0 MTA
33 47 160c 1bc5cf98 200b020 Enabled 1fefd4f8:1fefdc94 1317a028 0 MTA
34 1e ffc 12bbbdc8 200b020 Enabled 1fef94f8:1fef9c94 131ba6c8 0 MTA
35 1c ac 12bbd150 200b020 Enabled 1ff02d90:1ff03c94 131ba6c8 0 MTA
36 28 1660 12e7c6a8 200b020 Enabled 1fef74f8:1fef7c94 131ba6c8 0 MTA
XXXX 37 0 047ac770 8801820 Enabled 00000000:00000000 00263658 0 Ukn (Threadpool Completion Port)
XXXX 2e 0 047ac388 8801820 Enabled 00000000:00000000 00263658 0 Ukn (Threadpool Completion Port)
38 36 15f4 047abfa0 880b220 Enabled 1fe3afa0:1fe3c038 00263658 0 MTA (Threadpool Completion Port)