Explain this obscure ASP.NET bug

Question

I was just pushing a website to production, which worked fine in a QA environment. The patch involved DLL dependency updates, and some database scripts.

I will cut to the chase and explain the error that took me 4 hours to figure out. One of my database scripts created a stored procedure, which was missing [dbo] before its name. So instead of becoming [dbo].[myProcedure] it became [shawn].[myProcedure]. When the script was executed from code via the System.Data.SqlClient library, it killed the entire ASP.NET worker process. This caused the application to restart and redirect me back to the login screen. The procedure was only called conditionally, and quite infrequently, so it was difficult to reproduce.

Because it took out the entire ASP.NET worker process, it was extremely difficult to debug. I had no way of catching the error. Eventually I noticed this in the Windows Event Log:

Exception: System.Runtime.Serialization.SerializationException

Message: Unable to find assembly 'Shawn.Core, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null'.

(Note: Root namespace renamed to protect the innocent.)

And then, 2 seconds later:

EventType clr20r3, P1 w3wp.exe, P2 6.0.3790.3959, P3 45d6968e, P4 mscorlib, P5 2.0.0.0, P6 4889dc80, P7 4687, P8 4b, P9 shawn.data.dataaccess, P10 NIL.

At this point I figured it was a mismatched assembly version, GAC conflict? or something similar. I checked every reference throughout my project, and tried tons of different things. I eventually looked for all references to the namespace Shawn.Data.DataAccess and logged to a file every few lines to find where the application was failing. This led me to the correct stored procedure.

TLDR: Why would the lack of [dbo] on a single script cause the entire ASP.NET worker process to crash?

Answer 1

It crashed because your data access functionality could not find the stored procedure, and when you called it, well, since the stored procedure couldn't be found, the run time generated an unhandled exception.

As for it killing the entire ASP.NET worker process, that's because you had an unhandled exception. The process could not recover from the exception because it was not handled (no try/catch around your read, etc.), therefore the only thing it could do is restart the process.

As for the why the [dbo] versus [shawn] causes a crash... well, that has to do with ownership of objects in SQL Server. Users are able to access any object owned by DBO without specifying an owner, but if the object is owned by an account that is NOT DBO, then you must provide a fully qualified name. (i.e. [shawn].myStoredProcedure). Read more here: http://www.sqlteam.com/article/understanding-the-difference-between-owners-and-schemas-in-sql-server.

Answer 2

You must be doing something really unusual to bring down the worker process. It doesn't have anything to do with the database error - it's got to be related to where that error happens in your code, likely on a background thread, but even then ASP.NET generally protects itself from being brought down by uncaught exceptions.

It's best practice to always surround all code in any methods that are executed on threads with a try-catch.