.Net Windows Service Throws EventType clr20r3 system.data.sqlclient.sql error

Question

I have a .Net/c# 2.0 windows service. The entry point is wrapped in a try catch block that notifies me of problems and allows the service to continue operating normally yet when I look at the server's application event log I see a number of "EventType clr20r3" errors that are causing the service to die unexpectedly. The catch block has a "catch (Exception ex)" and does not rethrow the exception.

Each sql commands is of the type "CommandType.StoredProcedure" and are executed with SqlDataReader's. These sproc calls function correctly 99% of time and have all been thoroughly unit tested, profiled, and QA'd. I additionally wrapped these calls in try catch blocks just to be sure and am still experiencing these unhandled exceptions.

This only in our production environment and cannot be duplicated in our dev or staging environments (even under heavy load).

Why would my error handling not catch this particular error? Is there anyway to capture more detail as to the root cause of the problem?

Here is an example of the event log:

EventType clr20r3, P1 RDC.OrderProcessorService, 
P2 1.0.0.0, 
P3 4ae6a0d0, 
P4 system.data, 
P5 2.0.0.0, P6 4889deaf, 
P7 2490, 
P8 2c, 
P9 system.data.sqlclient.sql, 
P10 NIL.

Additionally

The Order Processor service terminated unexpectedly.  
It has done this 1 time(s).  
The following corrective action will be taken in 60000 milliseconds: 
Restart the service.

Answer 1

This error is most likely happening on a thread other than your entry point.

You need to somehow define exception handling for your background threads as well.

If you are using WCF to wrap your service, you have the ability to define how faults and other exceptions are handled. Is this how you're hosting your service?

EDIT: If you aren't spawning anything asynchronously, and it's not a WCF/etc app, try handling AppDomain.UnhandledException and logging the detailed error there.

Answer 2

Answer from comments. Glad to have helped. :)

Have you tried adding an AppDomain.UnhandledException event handler? msdn.microsoft.com/en-us/library/… Pretty much the problem is you have an unhandled SQL exception somewhere in your code.

Not necessarily, but without seeing your code or the libraries you are using, there is no way to know if the exception is occuring in a different thread, in a 3rd party library, etc... By wiring an AppDomain.UnhandledException handler, you should at least by able to log the exception details and figure out where your problem point is. Just cast the EventArg to ((Exception)e.ExceptionObject).Message and you should at least have a better idea where your unhandled exception is.