views:

102

answers:

1

Hi there, got a weird question for you based on some weird happenings on our site. We have four .DLLs in our ASP.Net solution (ie. managed code in the /bin folder). This morning we updated all four of them, the site was tested and worked fine.

This afternoon I uploaded a new version of one of the DLLs, and it took the whole site down, much to our dismay. I quickly swapped out the new one for the old one, taking the site back into the state before I'd uploaded the DLL, but the site was still down. We were forced to restart the entire server, but still no joy: As far as we were concerned, the site had been returned to exactly the same state it had been in before I'd uploaded anything, and it still wasn't working.

So, after a bit of investigating we discovered that it was a bug in one of the DLLs that had been uploaded that morning (i.e not related to the DLL I'd uploaded in the afternoon).

What we seem to have discovered is that, for some odd reason, the AppDomain didn't recycle properly when we uploaded our DLLs in the morning. Some of the changes made to the site went live, but at least one of the DLLs remained cached, instead of being updated.

When I uploaded a DLL later that afternoon, it caused an AppDomain recycle which then included the bugged DLL that was uploaded this morning.

Has anyone else had problem with the AppDomain not recycling when it should? Or possibly have another explanation for what it could be? AFAIK the AppDomain should be recycled every time there's a change made to any file in the bin/ folder... If so, how did it accidentally cache an older version of a file and not recycle properly?

Here's Scott Forsyth talking about AppDomain recycling: http://weblogs.asp.net/owscott/archive/2006/02/21/ASP.NET-v2.0-2D00-AppDomain-recycles_2C00_-more-common-than-before.aspx

+1  A: 

Sometimes the site won't recycle if it is under load as IIS thinks it is better to keep running than take everyone down for the amount of time the site requires to restart.

At most places I've been the best practice is to always issue an IISRESET after updating the site files. This ensures everything gets recycled after a deployment.

Chris Lively
Awesome response, thanks a lot! Can I ask: How do we "issue" and IISRESET? Thanks again.
Django Reinhardt
From the web server, go to the command line (if 2008 server, it has to be run as administrator), and type "IISRESET" That will cause IIS to dump all current transactions and recycle everything.
Chris Lively
Googled it: Just restarting IIS through IIS Manager does the same thing, I think. Thought it might, but I wanted to make sure. Thanks.
Django Reinhardt
Another thing, if you have a web farm, do it one server at a time. Basically, take one box out of the load balance rotation, wait until it completes the active requests, then update the code and reset it. Finally, put it back in the rotation and start the process over with the next box. This ensures no downtime. Of course, if the update requires changing a database schema, then you might just wait until off hours and do the whole thing at once.
Chris Lively
Looks like we wrote at the same time. Is there any different between manually typing IISRESET and clicking "Reset" in IIS manager, do you know? (I ask because we had a weird problem last Friday that was only cured after we reset the entire machine -- clicking "Reset" didn't seem to do anything).
Django Reinhardt
I have no idea. Every time I've seen IIS be reset it's always from the command line (even MS docs say do it that way). You might ask over at serverfault.com
Chris Lively
Cool, thanks again.
Django Reinhardt