ASP.NET Web Services troubleshooting?

Question

Working with one of our partners, we have developed now two separate sets of web services for their use. The first one was a simple "post to an https URL" style web service, which we facilitated by building a web page in ASP.NET that inspected the arguments in the URL, and then acted accordingly. This "web service" (if you can call it that) has been very stable.

At some point, the partner asked us to begin using SOAP based web services. At their request, we built them a new set of web services largely based on the previous objects, reimplemented as an actual "Web Service". This web service has not been very stable: around once a week, Nagios will alert us that our web service is not responding - and a quick iisreset does the trick.

Analyzing the log output and working in a debugger has not led us to anything concrete. The volume on this new web service is actually much lower than the HTTP web service. I think this could be a code problem or a platform problem, or of course something in between.

We've tried, with little improvement:

• To duplicate the behavior in the lab
• Debugging in the Visual Studio debugger
• Tinkering with IIS options to give it its own application pool

My question, what are the next steps for troubleshooting?

Environment: Windows Server 2003 Standard Edition R2 Service Pack 2 32 bit, Visual Studio 2005, MS SQL 2005, .NET Framework 2.0.50727

Answer 1

You may get some answers by profiling your webservices and understanding how they are using their resources. perfmon and procmon are both very useful tools in this regard.

EDIT: Since you say errors happen after about a week, the only thing I can think of is resource usage. Ensure your DB connections are being cleaned up, and any opened files (system call to the exe) are being closed.

Also, if your webservices can tolerate it, IIS has a setting that triggers a periodic recycle of an App Pool to handle cases where performance degrades over time. Its dirty, but it may work well for your case.

Answer 2

Since there isn't much to go on - here's another odd issue we came up against regarding our web services.

When the web service stops responding how is memory utilization? We have experienced issues with memory and memory fragmentation relating to busy web services on a system (there was also other things running causing additional fragmentation). When we re-factored the web services to load from smaller dll's and depend on other libraries (instead of one large library) we were able to resolve the memory fragmentation.

To identify what was occurring we would take a dump from the offending iis worker process where the app pool resided and then reviewed that using WinDbg. http://www.microsoft.com/whdc/devtools/debugging/default.mspx

Additionally we used DebugDiag to take the postmortem dumps. http://www.iis.net/downloads/default.aspx?tabid=34&g=6&i=1286

Hope this provides another direction to look at.