WCF communication between 2 servers crashes after IIS7 process recycle

Question

Hi guys,

I am kind of stumped with this one, and was hoping I could find some answers here.

Basically, I have an ASP.NET application that is running across 2 servers. Server A has all of the business logic/data access exposed as web services, and Server B has the website which talks to those services (via WCF, with net.tcp binding).

The problem occurs a few seconds after a recycle of my app pool is initiated by IIS on Server A. The recycle happens after the allotted time (using the default of 29 hours set in IIS).

In the server log (of Server A):

A worker process with process id of '####' serving application pool 'AppPoolName' has requested a recycle because the worker process reached its allowed processing time limit.

I believe that this is normal behavior. The problem is that a few seconds later, I get this exception on Server B:

This channel can no longer be used to send messages as the output session was auto-closed due to a server-initiated shutdown. Either disable auto-close by setting the DispatchRuntime.AutomaticInputSessionShutdown to false, or consider modifying the shutdown protocol with the remote server.

This doesn't happen on every recycle; I assume that it happens when someone is hitting the site with a request WHILE the recycle happens.

Furthermore, my application is down until I intervene; this exception continues to occur every time a subsequent request is made to the page. I intervene by editting the web.config (by adding a space or something benign to the end of file) and saving it- I assume that that causes my application to recompile and brings the services back up. I also have experimented with running a batch file that does this for me every time the exception happens ;)

Now, I could barely find any information on this exception, and I've been looking for a while. Most of the information I did find pertains to WCF settings that I am not using.

I already read up on "DispatchRuntime.AutomaticInputSessionShutdown" and I don't think it pertains to this situation. This particular property refers to the service shutting down automatically in response to behavior on the client side, which is not what is happening here. Here, the service is shutdown because of IIS.

I did read this which went through some sort of work around to bring the service back up automatically, but I am really looking to understand what is going on here, not to hack around it!

I have started playing around with the settings in IIS7, specifically turning on/off Overlapped Recycling and increasing the process startup/shutdown times. I am wondering whether it is safe to turn off recycling completely (I believe if I put 0 for the recycling time interval?) But again, I want to know what's going on!

Anyway, if you need more information, let me know. Thanks in advance!

Answer 1

This is probably related to how you open and close WCF connections.

If you open a proxy when your app starts and then continue to use this, a break in the connection, which is caused by a restart on the server side. Results in a error on the client side, since the server that the proxy was talking to is no longer there.

When you restart the client side (changing the web.config) new proxies are created against a server that is running.

The way to fix this is to make sure that you close a WCF connection after you use it.

http://www.codeguru.com/csharp/.net/net_wcf/article.php/c15941/

Answer 2

You should also make sure that you're using the correct SessionMode for your Web Service. I remember having similar trouble with some of my Services until I sorted out the correct mode. This is especially true when you're mixing this with any other authentication mode that is not "None".

This link might have some pointer.

http://msdn.microsoft.com/en-us/library/ms731193.aspx

Answer 3

My suggestion is to simply stop using IIS to host your services. Unless there is something you really need from IIS, I would recommend just writing a standard Windows Service to host your WCF endpoints.

If you can't do that, then by all means turn off recycling. AppPool recycling is mainly there because web developers write crappy code. I know that sounds rather blunt, but if you have enough sense to write code that doesn't leak then there is no reason to have IIS constantly restart your program.