Logging vs. Debugging

Question

Background: I've inherited a web application that is intended to create on-the-fly connections between local and remote equipment. There are a tremendous number of moving parts recently: the app itself has changed significantly; the development toolchain was just updated; and both the local and remote equipment have been "modified" to support those changes.

The bright side is that it has a reasonable logging system that will write debug messages to a file, and it will log to both the file and a real-time user screen. I have an opportunity to re-work the entire log/debug mechanism.

Examples:

• All messages are time-stamped and prefixed with a severity level.
• Logs are for the customer. They record the system's responses to his/her requests.
• Any log that identifies a problem also suggests a solution.
• Debugs are for developers and Tech Support. They reveal the system internals.
• Debugs indicate the function and/or line that generated them.
• The customer can adjust the debug level on the fly to set the verbosity.

Question: What best practices have you used as a developer, or seen as a consumer, that generate useful logs and debugs?

---

Edit: Many helpful suggestions so far, thanks! To clarify: I'm more interested in what to log: content, format, etc.--and the reasons for doing so--than specific tools.

What was it about the best logs you've seen that made them most helpful?

Thanks for your help!

Answer 1

The absolutley most valueable thing done with any logging framework is a "1-click" tool that gathers all logs and mail them to me even when the application is deployed on a machine belonging to a customer.

And make good choices at what to log so you can roughly follow the main paths in your application.

As frameworks I've used the standards (log4net, log4java, log4c++)

do NOT implement your own logging framework, when there already is a good one out-of-the-box. Most people who do just reinvent the wheel.

Answer 2

Don't confuse Logging, Tracing and Error Reporting, some people I know do and it creates one hell of a log file to grep through in order to get the information I want.

If I want to have everything churned out, I seperate into the following:

• Tracing -> Dumps every action and step, timestamped, with input and output data of that stage (ugliest and largest file)
• Logging -> Log the business process steps only, client does enquiry so log the enquiry criteria and output data nothing more.
• Error Reporting / Debugging -> Exceptions logged detailing where it occurred, timestamped, input/output data if possible, user information etc

That way if any errors occurred and the Error/Debug log doesn't contain enough information for my liking I can always do a grep -A 50 -B 50 'timestamp' tracing_file to get more detail.

EDIT: As has also been said, sticking to standard packages like the built in logging module for python as an example is always good. Rolling your own is not a great idea unless the language does not have one in it's standard library. I do like wrapping the logging in a small function generally taking the message and value for determining which logs it goes to, ie. 1 - tracing, 2 - logging, 4 - debugging so sending a value of 7 drops to all 3 etc.

Answer 3

Use an existing logging format, such as that used by Apache, and you can then piggyback on the many tools available for analysing the format.

Answer 4

I would just setup your logging system to have multiple logging levels, on the services I write I have a logging/audit for almost every action and it's assigned a audit level 1-5 the higher the number the more audit events you get.

1. The very basic logging: starting, stopping, and restarting
2. Basic logging: Processing x number of files etc
3. Standard logging: Beginning to Processing, Finished processing, etc
4. Advanced logging: Beginning and ending of every stage in Processing
5. Everything : every action taken

you set the audit level in a config file so it can be changed on the fly.

Answer 5

Some general rules-of-thumb I have found to be useful in server-side applications:

• requestID - assign a request ID to each incoming (HTTP) request and then log that on every log line, so you can easily grep those logs later by that ID and find all relevant lines. If you think it is very tedious to add that ID to every log statement, then at least java logging frameworks have made it transparent with the use of Mapped Diagnostic Context (MDC).
• objectID - if your application/service deals with manipulating some business objects that have primary key, then it is useful to attach also that primary key to diagnostic context. Later, if someone comes with question "when was this object manipulated?" you can easily grep by the objectID and see all log records related to that object. In this context it is (sometimes) useful to actually use Nested Diagnostic Context instead of MDC.
• when to log? - at least you should log whenever you cross an important service/component boundary. That way you can later reconstruct the call-flow and drill down to the particular codebase that seems to cause the error.

As I'm a Java developer, I will also give my experience with Java APIs and frameworks.

API

I'd recommend to use Simple Logging Facade for Java (SLF4J) - in my experience, it is the best facade to logging:

• full-featured: it has not followed the least-common denominator approach (like commons-logging); instead, it is using degrade gracefully approach.
• has adapters for practically all popular Java logging frameworks (e.g. log4j)
• has solutions available on how to redirect all legacy logging APIs (log4j, commons-logging) to SLF4J

Implementation The best implementation to use with SLF4J is logback - written by the same guy who also created SLF4J API.