How can I view log files in Linux and apply custom filers while viewing?

Question

I need to read through some gigantic log files on a Linux system. There's a lot of clutter in the logs. At the moment I'm doing something like this:

cat logfile.txt | grep -v "IgnoreThis\|IgnoreThat" | less

But it's cumbersome -- every time I want to add another filter, I need to quit less and edit the command line. Some of the filters are relatively complicated and may be multi-line.

I'd like some way to apply filters as I am reading through the log, and a way to save these filters somewhere.

Is there a tool that can do this for me? I can't install new software so hopefully it's something that would already be installed -- e.g., less, vi, something in a Python or Perl lib, etc.

Changing the code that generates the log to generate less is not an option.

Answer 1

see the man page of less. there are some options you can use to search for words for example. It has line editing mode as well.

Answer 2

Try the multitail tool - as well as letting you view multile logs at once, I'm pretty sure it lets you apply regex filters interactively.

Answer 3

Based on ghostdog74's answer and the less manpage, I came up with this:

~/.bashrc:

export LESSOPEN='|~/less-filter.sh %s'
export LESS=-R  # to allow ANSI colors

---

~/less-filter.sh:

#!/bin/sh
case "$1" in
*logfile*.log*) ~/less-filter.sed < $1
  ;;
esac

---

~/less-filter.sed:

/deleteLinesLikeThis/d  # to filter out lines
s/this/that/  # to change text on lines (useful to colorize using ANSI escapes)

---

Then:

• less logfileFooBar.log.1 -- applies the filter applies automatically.
• cat logfileFooBar.log.1 | less -- to see the log without filtering

This is adequate for now but I would still like to be able to edit the filters on the fly.