Tuning a WCF File Server

Question

I'm tuning a server and need some guidance. This server provides the following features:

1. an ASPX page with a DB call that is downloaded approximately every 3 minutes by several thousand machines
2. a simple ASP.NET admin web site used by a tiny number of people but that must be highly available
3. a WCF service that provides file synchronization to another several thousand clients using BasicHttpBinding and Streamed messages.

When an update is available to the file-sync clients (#3), they swarm the server clamoring for their new data. Without aggressive throttling they eat up all available ASP.NET worker threads and cause #1 and #2 to hang. This, as the kids say, sux0rs.

I've reproduced this problem on a 4-proc server. With a test client creating 100 simultaneous downloads, I see that 48 downloads are running simultaneously; in perfmon I see the remainder queueing up in ASP.NET->Requests In Application Queue. This jibes with the "12 concurrent threads per CPU" default setting described in this article. CPU and memory look fine; it's really just shuttling bytes with no processing.

As I see it, there are a couple of things I could do, possibly in concert.

1. Up the maxIOThreads and maxWorkerThreads in machine.config
2. Set up a web garden
3. Redo the client so downloads are handled directly instead of via WCF (eg link directly to the file instead of streaming it in a message).

My questions:

• Aside from trial and error, what are good guidelines for increasing the number of threads given that each is going to be in use for a pretty long time with low CPU/memory usage? If I go this route, how do I ensure the task switching for the increased number of threads doesn't overwhelm the system and cause more problems?
• Would a web garden help in this situation?
• Will I see much better performance if I get the download out of WCF and let IIS handle it entirely?

Best, roufamatic

Answer 1

Done right, WCF is the best tool for the job. If you're sending a static file, I'd suggest looking into an async service impl (or an async HttpHandler if you want to do it in ASP.NET without WCF) that's built on a FileStream with the Asynchronous flag set, and do the client writes async as well. This will allow you to make much more efficient use of the worker threads and offload the work to the OS. Downloading 1000 copies of the same file concurrently shouldn't cause your server to bat an eye if the impl is done right.