Parsing a growing logfile

Question

I want to do a logfile parser using ruby, this parser should parse the log file while it grows. It should parse line by line until the end and then wait (somehow?) for more lines to come, so my question is how to best handle it growing?

edit: Would prefer a portable way of doing this, even though my logfile is on Windows (for the moment).

Answer 1

For Windows, you can use Directory Change Notifications. You tell Windows (with FindFirstChangeNotification) to monitor the directory c:/foo/logs, then Windows updates your handle when something happens in that directory. At that point, you check to see if the change involves a file you care about.

Ruby has bindings for the Win32 API, and there is an example of getting these notifications.

Answer 2

There is a good script posted at http://www.biterscripting.com/SS_WebLogParser.html . It is a sample script written for web server logs, but can be used as a starting point for writing your own log parser for logs of any kind. To use it in a continuous way, while the log file keeps growing, here is a script.

# Script LogParser.txt
# Go in a continuous loop, sleeping 1 hr each time.
while (true)
do
    # The number of lines in the log file the last time we checked is in following
    # variable. Initially, it will be 0.
    var int lines_old

    # Read the log file into a str variable.
    var str log ; cat "file.log" > $log

    # Get the number of lines found this time.
    var str lines_new ; set $lines_new = { len -e $log } 

    # Strip off the first $lines lines.
    lex -e (makestr(int($lines))+"]") $log > null

    # The new lines are now available in $log. Process them with something similar to
    # SS_WebLogParser script.

    # Update $lines_old, then, sleep.
    set $lines_old = $lines_new
    sleep 3600                                  # 3600 seconds = 1 hour
done

To try,

1. Save this script into, say, C:\LogParser.txt (since you are on windows).
2. Download biterscripting. Google it up.

3. Call our script by entering the following command.

   script "\LogParser.txt"

If you need to use any of their sample scripts, install them with the following command.

script "http://www.biterscripting.com/Download/SS_AllSamples.txt"

Patrick