What do we log and why do we log it?

Question

This has been bugging me for quite some time.

Reading various questions on SO, blogs and listening to colleagues, I keep hearing how important "logging" is. How various logging frameworks stack up against each other, and how there are so many to pick from it's (apparently) ridiculous.

Now, I know what logging is. What I don't know is what is supposed to be logged and why. Sure, I can guess. Exceptions? Sounds like something one might want to log... but which exceptions? And is it only exceptions? And what do I do with the logged information? If it's an in-house app, then that could probably be put to good use, but if it's a commercial desktop application, how is the log of... whatever... helping anyone? I doubt regular users would be peeking inside. Is it then something you ask the users to provide on request?

I'm deeply frustrated by my own ignorance in this.

It's also surprising how little information there is about this. The info on the websites of the various logging frameworks is all written for an audience that already knows what it wants to log, and knows why it needs to do so. Same things goes for the various discussions on SO about logging, like for instance this highly voted up question on Logging best practices. For a question with so many votes, it's almost comical how there's next to nothing in there that would answer my what and why questions.

So being finally fed up, I'm asking here: what do people log, and why do they log it?

Answer 1

Logging provides a way, and one of the few ways, to supply useful information about the runtime of an application in production.

Debugging will tell you about the runtime of your application during development, but often, problems arise after deployment. Logging is a useful means by which you can, at runtime, give yourself some information about the behavior of your application in the deployed state, while only minimally affecting the runtime of the application.

This is useful, as you mentioned, for exceptions - by logging exceptions, you provide a way to "see" how your application is failing in places where you can't attach a debugger. The logs can be provided to you (by request), and help you diagnose this problem.

In addition, logging is often useful for other purposes. For example, my company's licensing software provides detailed logging, both for diagnostic purposes (for us to help troubleshoot), but also in order for our clients to be able to properly track and bill their time (many of our clients bill through usage time of our software to their clients, and by looking at the "logs", they can track usage by user, etc).

Pretty much any time you want to get an idea of the runtime behavior of your application in deployment, logging can be a useful tool.

Answer 2

Logging is very helpful for a headless service such as a network server; otherwise you have no real idea what the program is doing with all the messages it is supposed to process. If a message is never received, how will you figure out why without logging?

In my current project, we have a network server (and other processes) that all log data to the Unix syslog. So for example, instead of calling printf to print to console, they all call a different function to write a line to syslog. This is very useful since you can just do a tail -f /var/log/messages to see what is happening at any given time, and all of the logs are in the same place so you can cross-reference what different processes are doing.

Another nice thing about syslog is that it automatically cleans up the log files after a configurable interval of time. This is important since otherwise the logs will fill up the whole disk.

Obviously what exactly you log is a subjective and difficult-to-answer question. Probably the best answer is "log what you think will be useful to you later". When in doubt it is probably a good idea to log a bit more than necessary, but you have to be careful not to be too verbose or your log file will just be full of spam.

Answer 3

IMHO it's highly personal taste, what you want to log. Where it's not, it's defined by strict needs.

Take a web server, for example. Per default, almost all web servers log out of the box every access. The concrete fields may be different, but the mere fact is, that for the admin it is important to know, what is going on. Therefore, the logs have some common fields like the request's IP address to identify spoofing and DoS attacks.

Then, there are fields like the User Agent string or the returned status code, or, say, the referrer. These are (besides their possible use in the above case) there for statistics: The webmaster might want to know which browsers requested which page, where they came from and so on. Here the logging is done just in the case someone could be interested. And that is where the personal taste (read, interests) comes in. You can mostly configure, what is logged, and you will do so based on what you think might be important somewhen in the future.

Answer 4

Generally almost anything and everything can have a rationale for getting logged, though I think there are 3 general areas for why things get logged:

1. Errors and bugs to be found - If your database server goes down, there should be a mechanism to record this and send out an alert. Thus, this is something to be recorded so that if someone asks, "Why couldn't X do Y at time Z?" the answer can easily be given. This is partially a CYA type of situation in addition to being somewhat simple here.

2. Performance optimization - This is where other things can be logged like just information notifications so that someone else can take all the data and try to identify bottlenecks so that someone can know whether to upgrade the network gear, memory of a server, processor or something else.

3. Analytics - This is where things like Webtrends may be used for business users wanting to know how many users did we have at one time or some other simple request that if there are logs, these can be pulled and used for gathering statistics.

Answer 5

I think it's important to log events from exceptions to informational messages. The key is to be able to modify the logging level that is actually captured and recorded by a listener at runtime, this way you can gather more information only when needed.

Answer 6

I think that you should log anything that is likely to be useful in diagnosing systems for which manual issue reproduction would be difficult. That is not a concrete answer, however, the answer for every application will ultimately be quite a bit different anyway.

For example, exceptions are obviously useful, but also it can be useful to log key pieces of state information in order to insure that the resulting exception can be logically traced back to its causes. Sometimes reproducing problems from a production environment can be difficult, so these logs can be life-savers.

For commercial/desktop applications, things are more difficult. The user's are not likely to look in these logs, and many of them will just get confused if they try to look at them. The best thing there is to log similarly to your server based apps, but just rely on them to send them to you when there are issues. A simple "send logs to server" button in the application can be extremely useful here.

Answer 7

To me, logging is way of finding out what's really going on in your application. This can be as simple as just logging exceptions that you didn't realize were being thrown.

I think usage logging is something that is often times missing from applications. You can be spending months developing a feature, but there's no way of really know if the feature is even being used unless you log its usage. This can give you great insight not only into fixing bugs, but into where you can be most effective in expanding your application.

Answer 8

I think that the most important things to log, besides exceptions, are the steps to reproduce a specific state. In my company there are many different applications which manipulate a lot of data. Most bugs in the applications are related to wrong manipulation. When a customer complains about the app, we can try to reproduce the situation following the steps the customer followed before -- but better than that, if we have access to enough (and clear) logging, we can deduce exactly what happened.

A little example would look like:

10:44:13 [INFO] getFromDB()
10:44:14 [INFO] applyTaxes(23, 13, 44)
10:44:14 [INFO] removeDiscount(12)
10:44:15 [INFO] saveToDB()

Looking at the log I know what methods, with what parameters, and in which order, were called towards the data. This is the kind of logging that I find useful.

Answer 9

I can tell you what we log, and why we log it. Maybe that will help shed light on what decisions could be made in a concrete sense.

The application is a medical imaging application, so the care that people receive is directly related to how well the application works. If the application crashes when patient data entry is happening, that's bad, but it would be much worse to require that more images be acquired (too much extra exposure to the patient, too much time lost, etc).

So, with that in mind, we set things up using an MVC pattern. Each object communicates to other objects using a Message object, and the message object is identified by a string that defines the message as well as a payload object. That payload also has a .ToString method.

The default, for us, is any inter-object communication is logged. The exception happens when the communication is extremely frequent and not helpful. For instance, when an image is being acquired, it's being constantly rewritten to both screen and to disk, and adding logging to that bogs down the system too much to be useful.

Why do we log this much information? The application is quite complicated, with at least three or four threads running simultaneously. By having a record of everything that happens, we can determine where things are going wrong.

We can also determine where things are going right. We can use our log files to see operations that are being used all the time; by figuring out, for instance, that users are always searching their local patient databases (something we don't expect them to do, which would entail a discussion that would be quite lengthy in and of itself), then we can decide to spend more programming time on that section. If we see that certain operations, which we'd expect to happen quickly are happening with large lags in between them, we can determine whether we're doing too much for the machine to make it non-responsive, or if the user is actually not progressing through the steps as we would expect them to. This kind of information (analytics, I suppose the term is), makes us more responsive to the way that users actually use the program.

Answer 10

I use logging to make my debugging life easier.

While coding I regularly need to know what the state of a variable is quickly and without any other noise so I can locate the info I want.

For example, in a web app, I have many ways to 'display' some debug information but so far just logging to a file which I'm tailing in another terminal window is enough for my needs.

Of course, over time I have refined my logging functions, they stay out of the way but provide a powerful way of probing the system when I need to.

Answer 11

I worked on an app that logged all SQL select statements used to generate export pdf files. We tracked user, date+time the query and the parameters bound to the query. Logs would be uploaded to a second site each night at close of business. As well as a copy of each export file created. Not that we wanted to but the lawyers insisted on it.

Logging doesn't have to be about errors and debugging. There are cases where logs are in themselves pretty useful. Or in that case, required.

As another example, logging which function is run can be useful to gather data on how your app is used. Are reports run mostly on Friday? Are new jobs created mostly on Monday? Are there apps where 95% of the use are during last three days of the month? Are users insisting that searches are slow when less than 5% take longer than 2 seconds?

Answer 12

And don't forget database logging. We log every data change made to our data in audit tables. We log every step of every data import or export so that when they fail we can find out where they failed and we can tell the client when a particular file they sent us was put in the system (which also helps us track down the problems it created if they sent us bad data). We also log all our application errors to database tables so they can be easily analyzed for patterns and problems. And of course the database itself has it's own logs that log when jobs fail and when the server goes down, etc.

The need for logging in my opinion is:

1. to give the information you need to identify and fix problems on production that might be hard to reproduce just from a bug report.

2. to give you data to analyze to see if you can identify patterns to fix larger more systemic problems. It also may allow you to see if people are doing things they shouldn't be doing. Or to see as someone else pointed out, if there are parts of the application that are not being used.

3. To protect your data from bad changes (it is much easier to restore from a data change log also called an audit table) when someone accidentally (or maliciously) deletes the whole user table or when a client sends (an example not picked at random) the lastname in the first name column and vice versa and you need to get rid of 10,000 new bad records. It is generally faster to restore from logging tables than to restore an entire datbase to fix issues in one small part of the database.

4. to provide a record to use as evidence in a legal action when someone does maliciously change the data.

I think logging is very necessary when the product is at a client location and not your location because it may the only way that you can see what happened when they hit a problem. Users are notoriously bad at writing bug reports and remote users are even less likely to do so (we can train our own people usually but this may be the only time you hear from a particularl client.) Also problems at a client often happen for a long time before someone actaully tells the vendor about them. And what if the problem happened in running the quarterly reports but they did a workaround to get the job done. Are you going to wait a whole quarter until they run those reports again to see the symptoms? Is anybody goign to remember to get you to monitor while they did it? Clients also are often running on a set of equipment that may not match what you tested on and problems can come from that. Again logging is your best chace of being able to identify issues related to differnt environments becasuse they may hit errors you never got with your environment.

You may also want to design some logging that can be turned on and off easily if the logging itself is affecting performance. For instance creating a profile trace of the execution of database queries is a useful logging technique, but it usually affects performance so much it is only run on prod very rarely when a problem can't be isolated any other way.

Answer 13

I usually compare the logging idea with the "paper trail" idea in accounting. Accountants usually requires a lot of paper stuff when a certain process is performed. This is for there presentation later on auditing. This is much like the logging idea. We logged several important cases to be able to determine the process that took place on a certain process. This serves a our paper trail for the application if something might have went wrong. We can easily determine the errors by searching for [CRITICAL] or [ERROR] log tags. Something useful for investigation, you will have a quick idea on your issues at hand.

Answer 14

I don't get to write a ton of code, usually just for internal things, but my motto is somewhat similar to Bob the Builder:

CAN WE LOG IT? YES WE CAN!

My logs tend to tell the entire story of what the program is doing, to the point that other people don't like it. But, by God, it makes it easy for me to debug what went wrong.

Answer 15

I think what decides what's important to log is your application. It's difficult to state rules about what to log in general. We could list some things like errors, exceptions, but these might be also irrelevant for some applications.

So, try to figure out what would be useful to know if you need information later. Ask yourself: if something happens, do I need to know what was going on in this part of the application? As you can realize, there is no definitive answer.

Just some tips about implementing logging:

• try to define a good structure for your log information, so you can do automatic log analysis and filtering
• define several logging levels, so you can increase the amount of information only when you want.
• implement size limits, if you think you can have issues about that. For example, you log size could be at most 10 Mb per day.
• take care about the information you are logging for security reasons.

Answer 16

In my company, PA-DSS compliance rules mandate that all user actions must be accounted for. I must be able to prove that user X went to webpage C to perform action A at this time. I also must not present any detailed error information to the user. Both reasons dictate why the application logs.

What the app logs is very verbose. It logs every db transaction performed as a result of a webpage submit.

I find logging requirements highly dependent on business rules and demands.

Answer 17

For me the logging epiphany happens at the point where you get a bug report - "Brian was using the application and something funny happened and his screen went black.".

Brian's the CEO, this is all the information you're going to get from him and you absolutely have to fix this before he gets back in tomorrow. This is where you learn to love your log files (because you do have some don't you...).

Log all your errors - log anything that might cause you a problem down the line, or anything that might be useful when tracking down an error. Never, ever swallow an error without logging it.