A simple log file format

Question

Hi,

I'm not sure if it was asked, but I couldn't find anything like this.

My program uses a simple .txt file for log purposes, It just creates/opens a file and appends lines.

After some time, I started to log quite a lot of activities, so the file became too large and hardly readable. I know, that it's not write way to do this, but I simply need to have a readable file.

So I thought maybe there's a simple file format for log files and a soft to view it or if you'd have any other suggestions on this question?

Thanks for help in advance.

UPDATE:

It's access 97 application. I'm logging some activities like Form Loading, SELECT/INSERT/UPDATE to MS SQL Server ... The log file isn't really big, I just write the duration of operations, so I need a simple way to do this.

The Log file is created on a user's machine. It's used for monitoring purposes logging some activities' durations.

Is there a way of viewing that kind of simple Log file highlighted with an existing tool?

Simply, I'd like to:

1) Write smth like "'CurrentTime' 'ActivityName' 'Duration in milliseconds' " (maybe some additional information like query string) into a file.

2) Open it with a tool and view it highlighted or somehow more readable.

ANSWER: I've found a nice tool to do all I've wanted. Check my answer.

LogExpert

Answer 1

The 3 W's :

When, what and where.

For viewing something like multitail ("tail on steroids") http://www.vanheusden.com/multitail/

or for pure ms windows try mtail http://ophilipp.free.fr/op_tail.htm

And to keep your files readable, you might want to start new files when if the filesize of the current log file is over certain limit. Example:

• activity0.log (1 mb)
• activity1.log (1 mb)
• activity2.log (1 mb)
• activity3.log (1 mb)
• activity4.log (205 bytes)

Answer 2

First, check you're only logging things that are useful.

If it's all useful, make sure it is easily parsable by tools such as grep, that way you can find the info you want. Make sure you have the type of log entry, the date/time all conforming to a layout.

Build yourself a few scripts to extract the information for you. Alternatively, use separate log files for different types of entries.

Answer 3

Basically you better just split logs according to severity. You'll rarely need to read all logs for the whole system. For example apache allows to configure error log and access log, pretty obvious what info exactly they have.

If you're under linux system grep is your best tool to search through logs for specific entries.

Answer 4

Look at popular logfiles like /var/log/syslog on Unix to get ideas:

MMM DD HH:MM:SS hostname process[pid]: message

Example out of my syslog:

May 11 12:58:39 raphaelm anacron[1086]: Normal exit (1 job run)

But to give you the perfect answer we'd need more information about what you are logging, how much and how you want to read the logs.

Answer 5

If only the size of the log file is the problem, I recommend using logrotate or something similar. logrotate watches log files and, depending on how you configured it, after a given time or when the log file exceeds a given size, it moves the log file to an archive directory and optionally compresses it. Then the original log file is truncated. For example, you could configure it to archive the log file every 24 hours or whenever the files size exceeds 500kb.

Answer 6

If this is a program, you might investigate apache logging libraries (http://logging.apache.org/) Out of the box, they'll give you a decent logging format out of the box. They're also customizable, so you can simplify your parsing job.

If this is a script, see some of the other answers.

Answer 7

A fairly standard way to deal with logging from an application into a plain text file is to:

• split the logs into different program functional areas.
• rotate the logs on a daily/weekly basis (i.e. split the log on a size or date basis)
  • so the current log is "mylog.log" or whatever, and yesterday's was "mylog.log.1" or "mylog.ddmmyyyy.log"

This keeps the size of the active log manageable. And then you can just have expiry rules so that old logs get thrown away on a regular basis.

In addition it would be a good idea to have different log levels for your application (info/warning/error/fatal) so that you're not logging more than is necessary.

Answer 8

LogExpert

I've found it here. Filter is better, than in mtail. There's an option of highlighting just adding a string and the app is nice and readable. You can customize columns as you like.

http://stackoverflow.com/questions/113121/best-tail-log-file-visualization-freeware-tool