IIS performance problem trying to implement an XMPP-like protocol

Question

Hi, we're have a client that needs to get interactive messages from a server, from clients that are distributed around the world behind all kinds of firewalls with all kinds of ports closed. The only thing we can rely on is HTTP port 80 (and HTTPS 443).

The design is basically modeled after XMPP (the Jabber protocol), using our client and IIS. The client issues GET requests to a .NET Handler; the handler holds the request open for a while looking for messages. If any messages arrive, they are immediately sent to the client; if not, after a timeout the connection is closed with a "no-data" response. The client immediately reopens the communication.

Well, theoretically.

What's actually happening is first, IIS can't handle more than about 100 simultaneous requests - others are all queued, and there can be a several minute lag between "connected" and IIS recognizing that the client called in. Second, about half the time the client times out without any response from the server (the client timeout is five minutes longer than the server's).

POST always works. Other data served on the same web server works. Web services on the same server work. This is an out-of-the-box installation on Windows 2K3 Server.

Is there a configuration option we're missing, or is there something else I should look at to address this?

Thanks.

Answer 1

If IIS doesn't fit your requirements, you should choose another web server such as Apache (with Mod_mono) or LightTPD.

BTW, you can tunnel XMPP through HTTP using XMPP Over BOSH. No need to invent a custom protocol.

Answer 2

I think you're hitting ASP.NET thread pool limits, rather than IIS ones. Look into creating a asynchronous HTTP handler (IHttpAsyncHandler) as when they block/wait they aren't tying up the thread pool (they use completion ports instead).

Update: Came across this recently that seems to concur with my thinking: CodeProject: Scalable COMET Combined with ASP.NET

Answer 3

XMPP was never designed for high performance applications. The messages must traverse through the entire stack to the application layer, and there is a lot of XML parsing. Have you considered using some other standard besides XMPP?

Answer 4

Out of the box, windows needs some tweeking. I had to implement a comet server in asp.net and ran into some silly defaults. After reading these links (sorry, had to break the links, remove the x from the hxttp)

• hxttp://stackoverflow.com/questions/612242/iis-7-0-503-errors-with-generic-handler-ashx-implementing-ihttpasynchandler
• hxttp://blogs.technet.com/b/winserverperformance/archive/2008/07/25/tuning-windows-server-2008-for-php.aspx
• hxttp://smallvoid.com/article/winnt-tcpip-max-limit.html
• hxttp://publib.boulder.ibm.com/infocenter/wasinfo/v6r1/index.jsp?topic=/com.ibm.websphere.express.doc/info/exp/ae/rprf_plugin.html
• hxttp://support.microsoft.com/kb/820129
• hxttp://msdn.microsoft.com/en-us/library/ee37705(BTS.10).aspx
• hxttp://blogs.msdn.com/b/tmarq/archive/2007/07/21/asp-net-thread-usage-on-iis-7-0-and-6-0.aspx

I came up with the following changes that were made to our windows 2k8 server.

• reg add HKLM\System\CurrentControlSet\Services\HTTP\Parameters /v MaxConnections /t REG_DWORD /d 1000000 /f
• reg add HKLM\System\CurrentControlSet\Services\TcpIp\Parameters /v TcpTimedWaitDelay /t REG_DWORD /d 30 /f
• reg add HKLM\SOFTWARE\Microsoft\ASP.NET\2.0.50727.0 /v MaxConcurrentThreadsPerCPU /t REG_DWORD /d 0 /f
• reg add HKLM\SOFTWARE\Microsoft\ASP.NET\2.0.50727.0 /v MaxConcurrentRequestsPerCPU /t REG_DWORD /d 30000 /f
• appcmd.exe set apppool "[app pool name]" /queueLength:65535
• appcmd.exe set config /section:serverRuntime /appConcurrentRequestLimit:100000
• reg add HKLM\System\CurrentControlSet\Services\TcpIp\Parameters /v MaxUserPort /t REG_DWORD /d 65534 /f
• reg add HKLM\System\CurrentControlSet\Services\TcpIp\Parameters /v MaxFreeTcbs /t REG_DWORD /d 2000 /f
• reg add HKLM\System\CurrentControlSet\Services\TcpIp\Parameters /v MaxHashTableSize /t REG_DWORD /d 2048 /f reg add HKLM\System\CurrentControlSet\Services\InetInfo\Parameters /v MaxPoolThreads /t REG_DWORD /d 80 /f
• appcmd set config /section:processModel /requestQueueLimit:100000 /commit:MACHINE

I don't know if all the changes were required or optimal, but with some quik tesing against a test server, we achived over 30k executing connections and 5k requests per second. Couldn't go further because i ran out of client machines to run the tests from.