Socket server crashing with unhandled exception

Question

We have a c# (3.5 framework) socket server which is a console app, after about 3000 connections (or less, it's quite random), we get an unhandled exception which crashes the app completely.

We're really struggling to find out what's happening and where, the only info we get is below, can anyone shed any light? It should be noted that EVERYTHING is wrapped up in try catch{}

Description: Stopped working

Problem signature: Problem Event Name: CLR20r3 Problem Signature 01: qrushrserver.exe Problem Signature 02: 1.0.0.0 Problem Signature 03: 4bf56a0c Problem Signature 04: System Problem Signature 05: 2.0.0.0 Problem Signature 06: 49cc5ec9 Problem Signature 07: 2c0b Problem Signature 08: 40 Problem Signature 09: System.Net.Sockets.Socket OS Version: 6.0.6002.2.2.0.1296.17 Locale ID: 2057

Faulting application app_name.exe, version 1.0.0.0, time stamp 0x4bf56a0c, faulting module mscorwks.dll, version 2.0.50727.4200, time stamp 0x4a9ee32d, exception code 0xc0000005, fault offset 0x00000000001c89ca, process id 0x%9, application start time 0x%10.

.NET Runtime version 2.0.50727.4200 - Fatal Execution Engine Error (000007FEF8E4664E) (80131506)

Answer 1

Your program is dying on an Access Violation. While there's no conclusive evidence, this most likely happened when the garbage collector or finalizer thread was running. Which would make the exception uncatchable.

This is almost always caused by heap corruption. It is very unlikely that it is the .NET Socket code that caused the corruption, that code has been put to the test billions of times. Although you are giving it a good work-out. The much more likely cause is some kind of unmanaged code that's used by your program. Some kind of COM server, perhaps you P/Invoke something. It could also be some sort of add-in to the machine, like a virus scanner.

Finding the true cause is going to be difficult. Start with the environmental stuff, boot Windows in safe mode with network support. Run SysInternals' AutoRuns utility to disable stuff that gets started automatically. Good luck with it, you'll need it.

Answer 2

I guess you're using the Async methods rather than Begin/End.

Are you calling Dispose() on your SocketAsyncEventArgs when you're done with them? I find that if you don't and keep allocating new ones then you chew through memory for some reason, they don't seem to be collected... I found this a real problem with high number of read/write operations on a high number of connections.