Site update, testing was fine, after deployment, again fine, once user load increases, FAIL?

Question

We are using ASP.NET MVC with LINQ to SQL. We added some features and tested them all to perfection on our QA box. We are using Windows Server 2003 and SQL Server 2005. So when we pushed out changes to the Live web server we also used Red Gate SQL Compare to push new database changes to the LIVE database. We tested again between the few of us, no problems. Time for bed.

The morning comes and users are starting to hit the app, and BOOM. We have no idea why this would happen as we have not been doing any new types of code things that we were not doing before. However we did notice that during the SQL Compare sync the names of all the foreign keys were different between the two databases, not the IDs in the tables, FK_AssetAsset_A0EB67 to FK_AssetAsset_B67EF8 (for example, don't remember the exact number of trailing mixed characters during the SQL Compare), we are not sure why but that is another variable in this problem.

Strangely once this was all pushed out we could then replicate the errors on QA, but not before everything was pushed to LIVE.

QA and LIVE databases are on the same SQL Server, but the apps are on different instances of Windows Server 2003.

Errors generated:

Index was outside the bounds of the array.

Invalid attempt to call FieldCount when reader is closed.

Server failed to resume the transaction.

There is already an open DataReader associated with this Command which must be closed first.

A transport-level error has occurred when sending the request to the server.

A transport-level error has occurred when receiving results from the server.

Invalid attempt to call Read when reader is closed.

Invalid attempt to call MetaData when reader is closed.

Count must be positive and count must refer to a location within the string/array/collection. Parameter name: count

ExecuteReader requires an open and available Connection. The connection's current state is connecting.

Any one have any idea what the heck could have happened?

---

EDIT: Since we were able to replicate the errors all of a sudden on QA, it might not be a user load issue... Needless to say we all feel really screwed here.

Answer 1

Concurrency always brings bugs out of the woodwork. I'd recommend you check for objects that could be shared among requests (such as static members and singletons) and refactor your code so that as little as possible is shared.

As far as specifics go, for the error "There is already an open DataReader associated with this Command which must be closed first," you may want to try adding MultipleActiveResultSets=True to your connection strings.

Answer 2

It sounds like you're crossing the streams a bit and trying to share DataContexts across requests. My suggestion would be to wire in a dependancy injection framework that creates a new instance of the dependancy for each request.

I use Castle's IoC and wire it into the controller factory so that when it sees a dependancy on a repository it creates a new instance of that repository for each request. If you go this route let me know and I can shoot you a few more resources.