Error Tracking in live/production web applications

Question

I currently work at a company that has a lot of custom applications that are made internally. There are not currently standards for a lot of things. I would like to implement a way to record/track errors that happen in this programs (most are asp.net).

I am currently thinking of handling this in the Global.asax in the Application Error method. First trying to save the information to an Error Log/Tracking database, and if that fails, try sending an e-mail.

What type of information is the most useful to get from the error message and other application variables (page, username etc).

I am currently thinking of using two tables, one to get the general error and application information, and a second that holds the exception information. This will be a one to many relationship to hand the inner Exceptions that can come from one Application level exception.

I am sure that I am missing a lot of details and would like to hear your strategy for handling this issue.

Answer 1

It actually sounds like you've thought out the solution pretty well.

I work in web development shop, so in addition to logging errors, I also make sure that any system critical errors such as failing db query's and the like are also emailed so that we can jump on them and fix them right away.

An alternative to consider would be to email a report each day that provides information on each error thrown in each application and any relevant information that you would want in the email.

We log all of our errors into a single table for that application (of course that's easier to do when your working on applications where the databases are all contained in-house) including the timestamp of the error and the full text of the exception message that has also been emailed.

The exception message contains a function trace so you know what the original calling function was as well as file and line numbers for all functions in the stack. This makes back-tracking any given bug super simple.

Answer 2

Jeff Atwood wrote a great Exception Handler that you should look at on CodeProject

I would attempt to collect as much information as possible. Stack Information Session Information Location of Error

I would also setup a webservice that the applications call to save off the exception information into your system.

Answer 3

When logging to the database fails, you could write to the server's Event Log, or to a Log.txt file. You choice here partly depends upon whether you have access to the web servers containing those.

Sending error notification emails is a good idea if you need a speedy response. But there's no list of the errors to scan through them.

I do this by creating a special custom exception class with properties including user information (ID, name if available), all of the session variables, all of the form variables (so you can see how the form was populated and what the user entered), and some indication beyond the stack trace of where the error occurred (which page, which class, which method). Also include the name of the stored procedure called and the parameters send, or the SQL itself. Also all of the Exception properties (stack trace, etc). And DisplayMessage and InternalMessage fields. The DisplayMessage is displayed to the user. The InternalMessage is recorded in the log, and displayed on the custom error page when in debug mode.

I do the logging in Page_Error on the base page, and as a fail-safe in Application_Error.

Sometimes I add a key-value pair in web.config to contain my email address (or a distribution list). If there is a vaue in that key, a notification email gets sent. If there's no value, no eomail is sent. Then, during testing or early production, I can get immediate personal notification of the problem. After the initial period passes, I remove the email address in web.config.

Answer 4

This is what I do, and I still got a bug because of an email problem.

I never keep anything on the DB, cause if the DB has an error I will never have that error on the DB cause, logically, the insert will fail!

So I email to a special email address like [email protected]

Subject: [Application name] [Time Stamp: ddmmyyyy hhmmss] Message: Application, Error Message, Stack Trace Information plus session variables like username and server variables like Referrer.

the best friend of an ASP.NET developer is the stack trace information, it is here that you will know what went wrong, what was the call and where it was calling.

the only problem that you have in this system, is that you will not get anything if the email has a problem (some exception when sending the email), and for that I started add to a monthly XML file [errorLog_ mmm_yyyy.xml] as well and made a simple "drag-and-drop" page with a gridview that loaded the XML for the month and year that I wanted to check the errors.

try 
{
   // production code
}
catch(Exception ex)
{
   Utilities.Mail.SendError(ex);
}

or the best way: Add it to the Application_Error in global.asax:

<%@ Application Language="C#" %>
<%@ Import Namespace="System.Diagnostics" %>
<script language="C#" runat="server">
void Application_Error(object sender, EventArgs e)
{
   //get reference to the source of the exception chain
   Exception ex = Server.GetLastError().GetBaseException();

   //log the details of the exception and page state to the
   //Windows Event Log
   EventLog.WriteEntry("myWebApplication name",
     "MESSAGE: " + ex.Message + 
     "\nSOURCE: " + ex.Source +
     "\nFORM: " + Request.Form.ToString() + 
     "\nQUERYSTRING: " + Request.QueryString.ToString() +
     "\nTARGETSITE: " + ex.TargetSite +
     "\nSTACKTRACE: " + ex.StackTrace, 
     EventLogEntryType.Error);

   Utilities.Mail.SendError(ex);
}
</script>

with the code above you add the error to the event log, I append the error to the XML file in the SendError(Exception) function.

Answer 5

logging is good, but application monitoring is better.

caveat: i am the author of CALM

Answer 6

I think ELMAH may be what you're looking for.

Answer 7

Be very careful when logging potentially confidential information which has arrived over https. The user will expect it to be encrypted end-to-end (which may be a legal requirement). Ensure you don't send it via plain email.

Normally I'd say log all of the request including get, post, cookies, form state (in ASPNET), user agent, other headers, date/time, server machine etc

BUT under some cases that would result in some information being logged which you're not supposed to permanently record (such as credit card numbers being passed through to a payment provider). Sending it by email is even worse.

It's worth doing a check if HTTPS is on, and if so, reduce the amount of information you log to avoid this problem. I just sent through a string to say whether the field was empty or non-empty.

Answer 8

If you use a logging library (like Log4Net) you can set up various logging appenders to log to email, DB, file, event log, etc while having a single log call in your code. This post covers everything you need to get started.

There's also ASP.NET health monitoring.

EDIT: another poster mentioned ELMAH, which is great for recording ASP.NET errors, and can be dynamically 'injected' into running apps.

Answer 9

We were frustrated with the website error reporting applications that were out there so we wrote our own at Clearwind Consulting. It's free and we use it internally for our projects. It gives complete error information such as full error codes, browser type, traceback, a snippet of the code that caused the error, error frequency graphed over 30 days and more. At Clearwind, we do web application development. We needed a tool that would give us real data we could efficiently use about our websites.

You can choose how to get notification of when your website gives an error. RSS, email, or whatever method works best for you can all be used to get error notifications. And yes, you can send the errors to your iPhone while you're having a drink. It is fully written in Django so you can use JSON, RoR, XML, CSV, etc. if you desire to tag or format your data. Or just come to the website.

If you like, look at www.areciboapp.com.

It's free, easily added to (or removed from) any website and gives real information to enable you to fix the errors, rather than just a 404. No hard sell here. Just an invite to come take a look and see if it might help you and your website. We think it might.

Answer 10

It's a pretty good idea to log to a custom event log on the server as well as any other logging & notification. If the error was caused by an infrastructure issue, the same problem might also keep the error handler from sending emails or saving to a database.