Log analyzer for non-web-applications

Question

Hi all,

I have been searching for some kind of tool to analyse log files that are in custom format (my own format for my own program for example). There are many tools that are designed for web-traffic log analysing, but I haven't found suitable tools for custom log files. I'd be happy to find one where I can collect for example all the timestamps and draw conclusions or graphs based on that. For example show TPS (transactions per second).

I'd preferably use free programs. Basically it can be any kind of data analyser as long as it's capable of parsing log files. I would write a program of my own if I only needed to analyse only one kind of files, but I have need to analyse wide variety of files.

I did try to google for suitable tool for a long time but every analyser seems to be designed for web traffic.

Answer 1

This question seems a little bit broad to me, but your one single example (a graph using timestamps to compute/show transactions per seconds) screams RRD to me.

While this can help you analyze and graph your data, it's pretty lowlevel and might need some amount of work on your site. If you have no problems with web applications you might want to look at Cacti which is based on RRD and gives you a graphical way to analyze your data and to configure the datasources. You still probably need to create scripts to extract the data of your choice from logfiles/whatever, but you get the "GUI" for free.

Bottom line: Write some (g)awk/perl scripts and use tools like those mentioned above and you should be good to go with little effort.

Answer 2

With Log Parser you get info by simple sql query.

At cmd line for text file use like that:

LogParser -i:textline "select * from 'c:\path\*.txt' where text like '%TEXT TO FIND%'"

or try with GUI.

Answer 3

at a previous job, I maintain some perl scripts to summarize this type of data and put it in a postgres database....then used www.databeacon.com for analysis...but after you get the information in a database can generate any charts you want.....

Answer 4

You can try importing your file into a spreadsheet or database app like Excel or Access.