Is it acceptable to have unhandled exceptions in ASP.NET web applications?

views:

156

answers:

+5 Q:

Is it acceptable to have unhandled exceptions in ASP.NET web applications?

I'm working with a third party vendor at the moment who has supplied an ASP.Net web application. The web app generates around 200 unhandled exceptions per day, which end up as emails in my in-box. Upon investigation it turns out that most of these errors are triggered by the GoogleBot web crawler indexing the site and triggering access to another third party web service, which is rate-limiting the requests. When a request limit is exceeded, the third party web service refuses the request, this results in an unhandled exception in the web server and an HTTP/500 status code. The exception looks like this:

Exception: Exception of type 'System.Web.HttpUnhandledException' was thrown., Stack Trace:    at System.Web.UI.Page.HandleError(Exception e)
   at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)
   at System.Web.UI.Page.ProcessRequest(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)
   at System.Web.UI.Page.ProcessRequest()
   at System.Web.UI.Page.ProcessRequest(HttpContext context)
   at ASP.views_products_detail_aspx.ProcessRequest(HttpContext context)
   at System.Web.Mvc.ViewPage.RenderView(ViewContext viewContext)
   at System.Web.Mvc.ViewResultBase.ExecuteResult(ControllerContext context)
   at System.Web.Mvc.ControllerActionInvoker.c__DisplayClass11.b__e()
   at System.Web.Mvc.ControllerActionInvoker.InvokeActionResultFilter(IResultFilter filter, ResultExecutingContext preContext, Func`1 continuation)
   at System.Web.Mvc.ControllerActionInvoker.InvokeActionResultWithFilters(ControllerContext controllerContext, IList`1 filters, ActionResult actionResult)
   at System.Web.Mvc.ControllerActionInvoker.InvokeAction(ControllerContext controllerContext, String actionName)

The web app developer seems unwilling to handle these errors for reasons I don't really understand. Their approach is to throttle the GoogleBot until the errors stop happening (Google indexes quite aggressivley, generating around 5,000 hits per day). While I accept that throttling the GoogleBot would work, it seems like a cop-out to me. I've always considered unhandled exceptions to be bugs. Shouldn't the web app handle these errors? It is ever acceptable to allow an HTTP/500 to happen? What do the web developers out there think?

+2 A:

No, it's not acceptable. The web app should, at the very least, catch exceptions in global.asax.cs and do something reasonable with them.

Steven Sudit 2010-07-20 19:38:16

I suspect that is how I'm getting emails, it seems like the app is catching the exception and logging it. That's a sensible approach, but the constant noise from the server renders the notifications useless. The noise would, of course, abate if I were to throttle the GoogleBot.

Tim Long 2010-07-20 19:44:42

No, that's just bad design in terms of error reporting. It should log the errors somewhere convenient, and then notify you that errors have been logged, allowing you to click on a link to view them. It should not email you with each new error, though. Rather, it should only notify you of errors that occur after you view the last batch.

Steven Sudit 2010-07-20 19:56:04

"constant noise from the server renders the notifications useless" Be careful. If you don't watch the exceptions you'll miss the ones that matters. Glancing at 200 emails is not a big deal.

jfar 2010-07-20 20:42:26

@jfar: I believe the method I suggested would inform you of all 200 exceptions, but do so with only 1 email.

Steven Sudit 2010-07-20 21:18:11

-1 The web app does catch the exceptions and do something reasonable with them. Unless the specification says different - does it?

Philip Smith 2010-07-20 23:22:41

@Philip: I'm not sure that spamming Tim qualifies as reasonable. There are better ways to notify the administrator, such as the one mentioned above.

Steven Sudit 2010-07-21 13:26:37

I know why Philip downvoted me, but that doesn't explain the other two downvotes. It would be helpful if these downvoters offered their basis, even if it's just to ditto Philip.

Steven Sudit 2010-07-21 18:50:44

@Philip: As this is an expected condition, I would expect the error to be caught at the point of failure and not allowed to bubble up to a global exception handler. To my mind, unhandled exceptions are things you couldn't plan for - like running out of memory or disk full.

Tim Long 2010-07-22 03:02:43

The web app is handling the errors by e-mailing you.

What I am wondering is, what if a real human site visitor experience this error, how would you want it handled? I imagine you would want to get e-mailed if a real human visitor experienced this error. Maybe some of these errors are from real human visitors because Google used up their quota?

It seems you need a more graceful way overall of making sure the site scales without the third-party service. It's not clear to me which vendor (the ASP.NET developer or the third party service) should work on it but that's more of a project management issue.

Marc 2010-07-20 20:03:29

@Marc, if I understand correctly, the problem is that a call to his site causes it to call a service that imposes a rate limit. The GoogleBot is triggering this limit by calling very aggressively. The solution would be to use caching to avoid these calls.

Steven Sudit 2010-07-20 20:12:36

@Steve - exactly! the fix may be to add caching, not to improve the error handling. I am always a fan of good error handling but if your error handling is overloaded probably better to fix the error.

Marc 2010-07-22 16:45:49

@Mark: I'll gladly agree that adding caching addresses the root cause better and is more important. I still feel that spamming the admin when an expected (if unwanted) condition occurs is itself a problem.

Steven Sudit 2010-07-22 17:24:29

+2 A:

The web developer is unwilling to handle the exception in a web service?

That's rather frightening, as there are many, many reasons why a web service might cause an exception. Instead of asking to fix the GoogleBot issue, try asking the developer to allow the application to degrade gracefully if the web service is unavailable.

Damien Dennehy 2010-07-20 22:05:30

The web service is not the web developers. The web developer is simply reporting an error reported to them. That is perfectly reasonable and in no way frightening.

Philip Smith 2010-07-20 23:18:43

+1 A:

IMO this is actually rather acceptable. The problem is you have reached your service saturation point as it stands. There are 2 ways to handle this: either spend the money and time required to increase the saturation point and be allocated more service or spend the time and money to make yourself not be dependent on said service.

Edit: There is a 3rd option which would be to follow foot steps of the NY Times treating Google as a thief of your services and just ban them. Of course this is really just like putting your head in the sand but it's an option.

Chris Marisic 2010-07-20 22:14:39

I totally agree. The developer has done nothing wrong here.

Philip Smith 2010-07-20 23:23:59

+1 There is hardly a way for a a web application to deliver the right logging strategy out of the box. It has to be adjusted to the traffic, the availability of 3rd party services and the time administrators are willing to spent reading the errormails and logs.This is something that you will get when you have a service contract with the company that developed the application. You might get it for free if you have a long standing relation with them.

Malcolm Frexner 2010-07-21 11:04:55

The exceptions caused by GoogleBot turn out not to be exceptional, so sending out hundreds of notifications is not a reasonable response. The right answer is to use caching or throttling to cure the problem. But in the interim, the GoogleBot exceptions should be quietly logged without spam.

Steven Sudit 2010-07-21 18:53:28

@Steven that's a rather whining statement, sure it would be reasonable to make the notifications less frequent perhaps a daily batch but seems to be the most insignificant part of the OP's question.

Chris Marisic 2010-07-21 20:08:01

Not less frequent: gone. Any "exception" that you fully expect is not exceptional anymore. And, actually, that was the *whole point* of the question. He asked whether it's ok to have unhandled exceptions, and I'm directly answering it with a "no" and an explanation. You can disagree with my view and pointlessly accuse me of whining, but when you say that I didn't answer the question, then that is a factual error.

Steven Sudit 2010-07-21 20:13:21

+2 A:

There's really several questions here: should the web site display exceptions (No), should the web site display something more friendly to users (Yes), should the web site return a 500 error to Googlebot when it can't proceed (Probably), should you ask Googlebot to slow down (Yes), should you deliver 500 exception emails a day with no throttling or summarization (Probably not).

More detail:-

Using google.com/webmasters you can request that Google indexes your site less aggressively.

You should never show an Exception to users, you should always catch it and display a friendly error page BUT you need to be careful to preserve the HTTP code when you display that page (e.g. 404 or 500) since if you return a page with code = 200 then that error page will find its way into search engine indexes.

Any error handler should rate limit how often it sends email messages when errors occur.

A well written error handler should also allow the suppression of errors you know will happen - e.g. certain search engines insist on requesting pages that do not exist.

If Google can get you into a rate limit situation then high traffic from users may be able to do the same so overall it sounds like you need some kind of caching solution here too.

Hightechrider 2010-07-20 22:32:04

Good advice, and comprehensive.

Steven Sudit 2010-07-21 13:28:04

A situation outside the control of the vendor has occurred. The vendor has logged the error and sent a notification to you. The vendor cannot control the limiting on the web service or the number of visitors to your site. You are in control of both.

Unless the specification stated that the emails should be throttled in some way the vendors position is sound. You want work done you pay for it.

So you have three choices

Throttle google in some way;
Extend your web service allowance; or
Pay for the website to be altered.

Philip Smith 2010-07-20 23:17:01

Whether there should be a charge for fixing the web site is a different matter from whether unhandled exceptions are acceptable.

Steven Sudit 2010-07-21 18:55:25

The matter of payment is not in question here, this isn't about whether or not they get paid, it is purely a technical matter.

Tim Long 2010-07-22 02:50:00

@Tim @Steven - There is no "unhandled exception" in the question. The exception is "handled" by sending the website operator an email. The "real" question is was the behaviour specified and does the current behaviour conform to that.

Philip Smith 2010-07-22 07:47:18

@Philip: Fine, replace "unhandled" with "poorly handled" and retry.

Steven Sudit 2010-07-22 13:49:16

@Steven - Excellent now we understand the question. Which was is it acceptable? The answer to which must and can only be "what was specified". And the OP has singularly failed to respond to that question, so I suspect that what he has is a result of him removing the "well handling" from the project to get a discount. Which takes us right back to my original point with its three options.

Philip Smith 2010-07-22 14:15:11

Clearly, you still don't understand the question. The exception should have been handled close to where it occurred, so that the code can check if it's just GoogleBot up to its usual antics and therefore suppress the exception. Letting it bubble up, whether to a spam module or the default handler for otherwise unhandled exceptions, is an error.

Steven Sudit 2010-07-22 14:26:38

@Steven - That is not the question at all.

Philip Smith 2010-07-22 14:58:41

I find myself disagreeing with you. Coincidentally, it does not seem as if the original poster, Tim Long, agrees with you, either. Perhaps this means something.

Steven Sudit 2010-07-22 15:35:42

ansaurus

tags:

views:

answers:

Is it acceptable to have unhandled exceptions in ASP.NET web applications?

related questions