Best XML format for log events in terms of tool support for data mining and visualization?

Question

We want to be able to create log files from our Java application which is suited for later processing by tools to help investigate bugs and gather performance statistics.

Currently we use the traditional "log stuff which may or may not be flattened into text form and appended to a log file", but this works the best for small amounts of information read by a human.

After careful consideration the best bet has been to store the log events as XML snippets in text files (which is then treated like any other log file), and then download them to the machine with the appropriate tool for post processing.

I'd like to use as widely supported an XML format as possible, and right now I am in the "research-then-make-decision" phase. I'd appreciate any help both in terms of XML format and tools and I'd be happy to write glue code to get what I need.

What I've found so far:

log4j XML format: Supported by chainsaw and Vigilog. Lilith XML format: Supported by Lilith

Uninvestigated tools:

Microsoft Log Parser: Apparently supports XML. OS X log viewer:

plus there is a lot of tools on http://www.loganalysis.org/sections/parsing/generic-log-parsers/

Any suggestions?

Answer 1

Unfortunately, I can't give you the answer you are looking for, but I would like to warn you of something to consider when logging to XML. For example:

<log>
 <msg level="info">I'm a log message</msg>
 <msg level="info">I'm another message</msg>
 <!-- maybe you won't even get here -->
 <msg level="fatal">My server just ate a flaming death

In the above snippet of a potential XML log you can see the biggest drawback of logging to XML. When a catastrophic failure happens, your log format becomes broken because it requires closing tags. However, if you are using a program that parses your primary log output, this shouldn't be too much of a problem.

Answer 2

One of the nice things in log4j is that it offers nice possibilities for customizing the log formats and where those are written to.

So instead of choosing some log file format, I'd choose some logging library that allows to change the format and allows also getting the log directly to some program.

Answer 3

It appears that the Lilith log viewer contains an XML-format which is well suited for dealing with the extra facilities available in logback and not only the log4j things.

It is - for now - the best bet so far :)

---

I adapted the log4j xmllayout class to logback, which works with chainsaw.

---

As I have not been able to find a suitable log viewer capable of visualizing event information (instead of just presenting all events in a table) I have for now decided to create a very terse xml layout containing machine parsable information based on the above which can then be postprocessed by the Microsoft LogParser to any format I need.

Answer 4

I'd advise you consider logback-access for events.

Other than that, anything using JMX, as it was made to match the feature set of SNMP.

Answer 5

If you are defining your own XML log file writing, you do not need to worry about having a closing and opening tag in order to produce valid XML. Elijah's answer is right in that you do have the issue if you want to create an XML document, but that is not necessary straight off. The W3 standard also defines XML Entities (see section 4.3 of the W3's XML 1.0 spec, second edition, which unfortunately I cannot link to for you because I do not have enough points), which would be more suitable for log-style continual appending to a file without rewriting parts of it. You can then create a referencing XML wrapper document if you need to work with an actual XML document rather than an XML entity (see http://www.perlmonks.org/?node_id=217788#217797 for an example)