What is a good way to format logs?

Question

I'm designing an application which includes the need to log all incoming messages I receive from a Telnet connection. The text is largely plain though can include ANSI tags that provide text colour and formatting (16 colours, bold, underline, etc).

I'm would like to format my logs to store the text with formatting, date/time and potentially other meta data later. My first thoughts was all XML but this could impact my ability to write a fast search tool later. My current idea is Date/Time + text in one file with meta-data stored in another XML file, referenced by line number.

Is this a good solution? Also, where and how should I store the formatting commands? The original ANSI tags would disrupt the plain but having them in two different files might be awkward.

Additional: Thanks to some answers so far, though I should mention that most of the time the messages will be person to person communications rather than system messages. A more primitive IRC of sorts. Its up to my user to decide later (by adding meta data) which messages were important. This is the raw on the record log that filtered or edited logs might derive from.

Answer 1

If you are catpuring logging information for future searching and anaylsis perhaps a database would be a better answer.

As for your solution. Flat files do not scale well at all where as a database scale much better. I wouldn't split the files either, that just compounds the scalability issue. If you have to use a flat file I would probably try keeping the meta data in a csv (less over head) and the data in a series of files indexed by the csv file. That way all the data doesn't impact your index file. Just my thoughts.

Answer 2

G'day,

Definitely do the logging in flat file and add munge scripts to turn it into XMl later.

First suggestion would be to make sure that all date/time strings are in ISO 8601 format, namely YYYY-MM-DD hh:mm:ss.

Second is to make your categories, e.g. exception, fatal, error, warning, info, etc. really stand out in your logs.

Then aybe look at some of the vim syntax files and create a new syntax for your log format so that important log entries really stand out.

It's not really that hard to take one of the standard syntax files and modify it to handle your log strings.

HTH.

cheers,

Rob

Answer 3

My first suggestion would be to use a drop-in logging tool like log4net, which will make formatting much more automatic.

If you are going to go the route of two files (and I agree with Craig that a database is probably a better choice,) you can probably save yourself a lot of heartache by having one file that is as sparse as you can make it for later fast searches and one that holds all the information in one place (metadata and data) rather than creating a metadata-only format.

Answer 4

I'm going to "split the fence" and say use the database for all of your analysis/archiving log entries (such as your Telnet communications). This will grant you the benefits of full text searching, columns, and easy ways to search out the data.

Use a flat file (or XML format since the file shouldn't be too big) for any of your debug/critical error type logs.

If you have a broken database connection, or something has gone wacky with your table structure, logging to the DB will be meaningless.

Come to think of it, if you are looking for a slightly more "lightweight" solution, you could use SQLite to log all your telnet traffic so that you can leverage the advantage of the DB structure, but also have the availability of the file.

With another nod to log4net, you could easily accomplish this with the ADO appender they have.

Answer 5

I'm not sure exactly what you are trying to accomplish. Telnet is usually thought of as a character-at-a-time protocol, so when you say "incoming messages" do you mean each character is a message? Or the entire user's session is a message?

I'll make some assumtions. You have users logging in via telnet and you want to capture everything they do while they are logged in. Later, you want to be able to associate the stuff they did with that user and the time and date they did it. You'll need to be able to search later to find out "who did 'rm *' as root?"

I would store each user's session as a separate file, with a naming convention that includes the user's login and a timestamp.

e.g. 2008_09_08_14_52_07_nidonocu

Within the the file, I would capture each byte received, assuming they will mostly be plain text characters.

e.g.

ls
cd www
ls
vi index.html
/copyright 2007
llllllllllllr8:wq
exit

Write the 8-bit ANSI characters to the file as well. You should be able to use a text editor and grep to do basic audits and searches. You could use a binary file viewer or get more sophisticated later if you need to actually read the 8-bit data.

Backups, archiving, purging, etc. can all be done using regular file system tools and scripting.

My apologies if my assumptions are wrong.

--
Bruce