an ASP.NET performance bottleneck mystery

Question

A classic ASP.NET app - AppSrv + MS SQL DB. Both servers are heavy-lifters 8 cores, 20 GB of RAM. When load testing, the throughput goes somewhere to 400 VirtualUsers (according to LoadRunner) with CPU being approximately 30% utilized an DB server primarily idling - response times go dramatically up, to the point of unresponsive.

The usual suspects, such as Max Pool being exhausted and conn limit on ASP.NET set are not at fault: Max Pool is set to 200 and about 80 conns are used; conn limit is set to 0.

I ran with ANTS profiler the code and it showed that Thread blocking did not contribute significantly.

Ideas very very welcome!

Answer 1

We had bottleneck issues similar to yours when using WWF.

Replace workflows with code.

Its not fix, but if you find workflows are the problem then you can move on to fixing the issue.

Answer 2

Does the host utilize App Pool?

Did you try increase the number to 5 to 10 in

An Application Pool -> Performance -> 
Web Garden -> Max Number of worker processes

Answer 3

When no cpu's are hosed and pages tend to become unresponsive, my first hunch is database locking. One thread can hold up the rest when not releasing a lock on a table, which results in non responsive pages while idling your db server.

Can you use a sql monitoring tool to verify proper usege on the database machine and see if all users get their data properly?

One other thing; what info can you give on memory use?

Answer 4

Add logging to your app with some appropriate unique request ID so you can effectively trace how long each operation within a page load takes. For instance, logging before and after every database call will show whether the database calls are taking a long time or not.

As others have suggested, adding logging/profiling on the database side will show whether that's deadlocked (or similar).

Answer 5

Also check for network saturation. Make sure you aren't maxing out the network connection between your load test machine and the web server. Also, if you are returning alot of heavy text/binary data between your web and database server monitor that network connection.

Answer 6

We recently tuned our web application using the following IIS performance settings guide which proved very succesfull.

There were two server specific settings we changed;

Working Set Memory Usage - Servers running Windows Server™ 2003 are configured by default to give preference to the file system cache over the working set when allocating memory. Microsoft does this because Windows benefits from having a large file system cache. Being that IIS rides on top of the Windows operating system, it also benefits from having a large file system cache. If your server is a dedicated IIS Server, though, you might see better performance if you shift priority to the working set instead. The reason behind this is if preference is given to the file system cache, the pageable code is often written to virtual memory. The next time this information is needed, something else must be paged to virtual memory and the previously paged information must be read into physical memory before it can be used. This results in very slow processing.

Network Throughput - By default, servers running Windows Server 2003 are configured to give preference to the file system cache over the working sets of processes when allocating memory (through the server property Maximize data throughput for file sharing). Although IIS 6.0–based servers benefit from a large file system cache, giving the file system cache preference often causes the IIS 6.0 pageable code to be written to disk, which results in lengthy processing delays. To avoid these processing delays, set server properties to maximize data throughput for network applications.

The following services are not required on a dedicated Web server:

• Alerter
• ClipBook
• Computer Browser
• DHCP Client
• DHCP Server
• Fax Service
• File Replication
• Infrared Monitor
• Internet Connection Sharing
• Messenger
• NetMeeting Remote Desktop Sharing
• Network DDE
• Network DDE DSDM
• NWLink NetBIOS
• NWLink IPX/SPX
• Print Spooler
• TCP/IP NetBIOS Helper Service
• Telephony
• Telnet
• Uninterruptible Power Supply

Answer 7

It may be a long shot, but you can try running your code through the Patterns and Practices Checker to see if it finds any low hanging fruit.

Answer 8

Problem is probably in your machine.config file.

You should check following configuration parameters:

• maxconnection
• maxIoThreads
• maxWorkerThreads
• minFreeThreads
• minLocalRequestFreeThreads

For short description check: IIS 6.0 Tuning for Performance

There are some differences between ASP.NET 1.1 and ASP.NET 2.0

ASP.NET 1.1

Guidelines about some practical values you can find in:

• Microsoft Kb article 821268
• MSDN Article "Chapter 6 — Improving ASP.NET Performance - Threading Explained"

Default values are way to low for any practical use and should be corrected according to suggestions in linked articles.

ASP.NET 2.0

Parameters are moved to processModel section and parameters are auto configured. Besides auto configuration, you can manually set parameter values, so you should check if:

• processModel is enabled
• autoConfig is set to true
  or
• parameters are set to correct values

For detailed description check: ASP.Net 2.0 processModel