high cpu in IIS

Question

Hi all.

I'm developing a POS application that has a local database on each POS computer, and communicates with the server using WCF hosted in IIS. The application has been deployed in several customers for over a year now.

About a week ago, we've started getting reports from one of our customers that the server that the IIS is hosted on is very slow. When I've checked the issue, I saw the application pool with my process rocket to almost 100% cpu on an 8 cpu server.

I've checked the SQL Activity Monitor and network volume, and they showed no significant overload beyond what we usually see.

When checking the threads in Process Explorer, I saw lots of threads repeatedly calling CreateApplicationContext. I've tried installing .Net 2.0 SP1, according to some posts I found on the net, but it didn't solve the problem and replaced the function calls with CLRCreateManagedInstance.

I'm about to capture a dump using adplus and windbg of the IIS processes and try to figure out what's wrong.

Has anyone encountered something like this or has an idea which directory I should check ?

p.s. The same version of the application is deployed in another customer, and there it works just fine. I also tried rolling back versions (even very old versions) and it still behaves exactly the same.

Edit: well, problem solved, turns out I've had an SQL query in there that didn't limit the result set, and when the customer went over a certain number of rows, it started bogging down the server. Took me two days to find it, because of all the surrounding noise in the logs, but I waited for the night and took a dump then, which immediately showed me the query.

Answer 1

Going purely on gut and doing a complete guess, it sounds like there could be sort of exception happening, the exception is getting caught at the global exception handler in the global.asax, and the exception handler is also causing throwing an exception and dumping the process. Could also be due to a virus scanning utility locking up some files. I could be WAY off, though.

Answer 2

I've been stuck in the same hell...

http://stackoverflow.com/questions/392036/isolating-a-rampant-process-in-iis#392099

Answer 3

Don't eliminate the possibilty of a hardware issue. I had a server running slow and found it to be a motherboard issue. It was replaced under warranty.

Answer 4

Usually this has nothing to do with hardware and everything to do with how IIS is configured coupled with some slightly long running queries (100+ milliseconds).

Under your application pool configuration set your web garden setting to something like 20 or more.

The web garden setting is pretty much the number of threads available to process requests for your application. If it's set to 1 then a single query could block handling other requests until it completes.

I have an app that's handling close to 3.5 million requests a day. When the web garden was set to 1, the web server CPU stayed at 100% and had a LOT of requests dropped. When I upped it to 50, the web server CPU dropped to just under 2% and no requests were dropped.