WCF Cold Startup

Question

I use WCF in a fairly demanding environment. One behavior that I have observed is something that I have taken to calling the cold startup. When I first startup a client that is calling a service there seem to be a lot of failures in the first calls. For instance I can watch the first say ten calls go through and then the next 200 calls fail all once. I am talking to the service asynchronously. The service then run and responds fine. I can see that it is an endpoint (potentially) issue not an operation issue since multiple different operations will all fail. It feels like there is a lock and the endpoint stalls and reset itself and is then fine although I have no evidence to back this up.

There are no errors in the server side trace. My client side logs show a lot of the following exception:

System.ServiceModel.CommunicationException: The server did not provide a meaningful reply; this might be caused by a contract mismatch, a premature session shutdown or an internal server error.

I have considered implementing a smoothing algorithm to even out the service calls since there tend to be a lot of them on startup. Has anyone else seen similar behavior? Thanks.

Steve

EDIT: The service is hosted in a Windows Service.

EDIT: Thanks for the comments guys.

I have from the outset set the numbers on the various queues to be pretty high. I have some hard won knowledge about how to scale WCF services. I allow 2048 concurrent connections. I have set the listenBackLog etc...

I implemented a smoothing for the first 1000 calls and that seemed to get it. It may be JIT related but I have no proof about that. For the time being I am going to leave it and see if it reoccurs.

Answer 1

Out of interest, how are you hosting the WCF server? IIS has convenient pooling (via a NLB such as F5), but has the app-pool recycle issue, and the lag caused by IIS spawning (on first demand) the app-pool/domain/etc in the first place. Self hosted (windows service etc) tends to be more linear and predictable for performance, especially if (during service-start) you forcibly load everything, perhaps by making a request to yourself (if you see what I mean).

If you don't need to cluster the service, consider self-hosting it. See if that helps.

Answer 2

Have you tried increasing the number of queued connections for the endpoint? It could be that 10 are queued and while the .NET CLR is JITing your code, the rest of the incomming connections are refused until your code starts running.

Answer 3

This is an older topic, but I find myself in a similar situation. The OP stated

I implemented a smoothing for the first 1000 calls and that seemed to get it.

What exactly does he mean by "smoothing" I did a google search and it doesnt seem to be a keyword...