What Is a Good Layout for a Standard Error Log File?

Question

I am trying to design the error and warning log file for my desktop program.

As my program reads the user's input file, it may find syntax errors or invalid data of some sort. Once everything is read and the program is processing the data, more problems may be found.

I want to write messages about these into a simple text file. I may also want to include informational text to indicate progress, time, memory use, etc. I'll want to include line numbers and maybe even the actual input lines that are causing the errors.

This will be a file that the user will want to browse through, so obviously it has to be well laid out and easy to use.

Do you know of any style guides for this, or have you seen an error log file that has made you think to yourself: "Now that's a well designed log file!"

---

Followup:

The first three answers are actually more applicable for a server or event log.

I'm really looking for a format for a log file for my desktop program to detail any problems it finds with the input file and the success (or failure) of its processing thereof.

I'm sure there are some desktop applications that you've used that produce such log files. Have you seen any good ones?

Answer 1

I write logs to csv files and find it very convenient (you can use them as data base or open them with a spreadsheet software for sophisticated analysis). The sample log file is:

Date;Time;Severity;TID;Module;This;Source;Message
2010/01/21;08:47:05:205;DEBUG;4936;MAIN;0x00000000;DllMain@36;DLL_PROCESS_ATTACH
2010/01/21;08:47:05:205;DEBUG;4936;MAIN;0x00000000;DllMain@40;DLL_PROCESS_DETACH

I'm not sure whether it suits your needs but I think you've got the main idea. Good luck!

Answer 2

There are few log files I've been truly impressed by - actually, I struggle to think of any (and am failing to find any). In my experience, the problems that arise come about in part because the logs are formatted for an insider (programmer) to understand, rather than either for another program or an outsider (non-programmer) to understand.

Crucial information is often omitted; sometimes even date and time information. I'd recommend making those easily parsable; I'd use an ISO 8601 notation such as '2010-01-22T10:23:21-08:00' (including the time zone, note). I'd include process (and thread) ID; I'd consider including program name, arguments (eg file name); I'd include the user ID in some form too. Some of this might only be needed once per run, but other bits might be needed for each message. It depends in part on whether the log file is unique per run or shared across runs, and whether a single file can be used by more than one user/process.

You then need to decide how you are going to identify the message content - and the end of the message content. Especially for machine parsing (but also for human parsing), it is helpful if the content is unambiguously formatted and the end can be detected. You may need to escape content (this is the step that is most frequently undone, so if the error message is about a mal-formed error message, it is really hard to tell what's data from the file and what's new error message about the data in the file). In a multi-threaded program - or where the log file might be shared between processes - you have to consider how the writes will be buffered as well, especially if you have to deal with long lines. You want each message to be individually separate in the log file. This is not necessarily trivial--you might even need to deal with a locking protocol to ensure controlled access.

Consider whether you need to provide a pretty-printer for the log.

Consider whether an XML-based format (with begin and end tags) would help. I am not a big fan of XML, but it does have some advantages for machine-processing.

Answer 3

syslog and log4j are widely used formats with which sysadmins may be familiar, and there are tools for working with them. You should at least look at them before doing your own format.

For Windows it's pretty much always the Application Event Log, which I don't really like as much as a text file, but it's the Right Way. I've seen a few apps that write text files in their program directory or something, but it's always ad hoc, never standard. McAfee VirusScan, for example, logs files in C:\Documents and Settings\All Users\Application Data\McAfee\DesktopProtection. My machine has lots of install logs in C:\Documents and Settings\username\Local Settings\temp*.log and C:\WINDOWS\temp*.log. But again, they're all different and ad hoc and not noticeably well-designed