Hopefully someone can help us as we're reaching as far as investigation can go!
We've got a simple asynchronous socket server written in C# that accepts connections from an ASP.NET web application, is sent a message, performs some processing (usually against a DB but other systems too) and then sends a response back to the client. The client is in charge of closing the connection.
We've been having issues where if the system is under heavy load over a long period of time (days usually), CLOSE_WAIT sockets build up on the server box (netstat -a) to an extent that the process will not accept any further connections. At that point we have to bounce the process and off it runs again.
We've tried running some load tests of our ASP.NET application to attempt to replicate the problem (because inferring some issue from the code wasn't possible). We think we've managed this and ended up with a WireShark packet trace of the issue manifesting itself as a SocketException in the socket server's logs:
System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host at System.Net.Sockets.Socket.BeginSend(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags, AsyncCallback callback, Object state)
I've tried to reproduce the issue from the packet trace as a single threaded process directly talking to the socket server (using the same code the ASP.NET app does) and am unable.
Has anybody got any suggestions of next things to try, check for or obvious things we may be doing wrong?