ansaurus

Question

Answer 1

A:

If you can attach a debugger then the misbehaving threads will usually be the thread that comes up when you 'Break All'.

Otherwise I would probably take a bunch of those thread location snapshots and look to see if there are any threads which are consistently not in a wait (i.e. WaitForMultipleObjectEx). That should give you an idea of which threads are misbehaving and what code they're generally running.

And make sure you don't have any code like:

while(1)
  ;

:)

Aaron 2009-11-06 19:56:21

I know the 2 threads causing the issue. When I do !runaway they are at the top of the list!runaway 41:3420 0 days 5:31:57.781 48:345c 0 days 1:23:23.421But I'm at a lost for figuring out why those 2 threads are sticking around and what exactly they are doing and what line of code created them....If there's anything else you know that I can do to figure out what created these threads that'd be awesome :-)

Shane 2009-11-06 20:08:11

Answer 2

+1 A:

You can always stop the process with the debugger and check the stack trace a couple of times. If a thread is often not idling and in the same spot, you'll know a bit more about where it spends all time.

In the stuff you pasted I only see the stack trace for one thread, can you get the stack trace for all threads? (Sorry if this is way off, I'm used to doing this in unix mostly)

Mattias Nilsson 2009-11-06 20:03:53

yea I've done that... All the other threads are running and exiting and doing what they do just fine.... It's just these 2 threads that are sticking around and using up 100% of the cpu. The is currently load balanced and I have all traffic directed to a different site so the offending site has zero load right now and only these 2 threads are active and using up resources

Shane 2009-11-06 20:11:00

Answer 3

A:

Use ProcDump to get a memory dump when the CPU is spiking high. Then check the call stacks of all the threads. Also run perfmon and keep checking for thread that is using up most of the CPU. Hope this helps

Naveen 2009-11-06 21:48:24

Also check for % Time spent on GC in perfmon

Naveen 2009-11-06 21:49:39

yea I have a memory dump. The cpu is spiking high constantly... It's just sitting at 99 percent... I used perfmon to check the running threads and there are just 2 threads that are running busily (clrstacks of those posted above). Each thread has the exact same memory dumb and clrstack dump (posted above) and each thread is taking up 50 percent of the resource respectively...But I can't figure out what to do next to get any hints of what these two threads are doing or where they came from

Shane 2009-11-06 22:55:50

% Time spent on GC isn't that high it stays around 0.2 which I don't think is that high...

Shane 2009-11-06 23:40:54

I see call stack of only thread id 48. What about the other one?

Naveen 2009-11-07 00:30:31

Yea they both look identical so I figured I'd just post one of them

Shane 2009-11-07 01:44:31

it looks exactly the same as any of the other threads waiting around except for the parts about 1789e944 7a9390a2 System.Net.ConnectionPool.CleanupCallback()1789e954 7a938fc3 System.Net.ConnectionPool.CleanupCallbackWrapper(Timer, Int32, System.Object)1789e984 7aa97f5f System.Net.TimerThread+TimerNode.Fire()1789e9cc 7a584c84 System.Net.TimerThread+TimerQueue.Fire(Int32 ByRef)1789ea0c 7a55db8b System.Net.TimerThread.ThreadProc()So that's where I'm at

Shane 2009-11-07 02:00:43

Answer 4

+2 A:

Alright so I found the issue I did a !clrstack -p and than a !do on the system.net portions and it revealed the offending thread was a System.net.Servicepoint pointed at our smtp server..

Googled around and found that this is the issue http://connect.microsoft.com/VisualStudio/feedback/ViewFeedback.aspx?FeedbackID=337557 also described here http://www.vbforums.com/showthread.php?t=584384 It's an issue with the service point not correctly sending the quit command and disconnecting.. Which they are going to resolve in .net 4.0

Now I just have to code in the workarounds to ensure the service point closes up and that should work out

Thanks for everyones help

Shane 2009-11-07 08:57:51

+1 ... thanks for following up!

Adam Liss 2009-11-07 12:45:59

no problemand if anyone is curious what the magic fix was... I found 2 ways1) Just use IIS Pickup2) client.ServicePoint.MaxIdleTime = 1; client.ServicePoint.ConnectionLimit = 1;

Shane 2009-11-07 18:25:26

ansaurus

tags:

views:

answers:

Debugging high cpu usage

related questions