Apply formatting to unix shell

Question

I've been looking at some server logs using tail -f recently, and have thought that it'd be much easier to see some things if I could format the output. Really all I'm looking for is a way to perhaps colour certain words (determined by a regex), and perhaps remove certain words (again, determined by a regex).

I know there's programs which visualize server logs in real time and whatnot, but I'm more interested in this.

Answer 1

I think what you're looking for is some kind of sed script that will surround the words you choose with ANSI Color escape sequences. (hmm... lemme look).

EDIT OK, got it:

Here's an example for outputting "Hello" in dark red:

echo -e "\033[31mHello\033[0m"

What's happening? First of all, I'm using echo -e so that echo doesn't convert the slashes into on-screen slashes, but rather reads the escape sequence of \033 as a single escaped character. This is actually just 33 in octal, which is 27 (the ordinal for the ESC key).

So what is really being sent to the screen is something like:

<ESC>[32mHello<ESC>[0m

Which the ANSI display interprets as "first do the command 32m, output "Hello", and then do the command 0m.

In this case, the command 32m means "set the forground color to 2", and since color #2 is dark red, the "pen" used by the terminal will now be dark red. Which means that from this point onwards, all the text that will be displayed on screen will be dark red.

When we're done outputting the text that supposed to be red, we wish to reset the colors, so we call the command 0m which resets the colors to normal.

For a list of all the escape codes, look up in [http://en.wikipedia.org/wiki/ANSI_escape_code Wikipedia] or just google for it.

So all your sed script has to do is replace the words you choose with the words surrounded by the colors. For example, to replace the word "Feb" in /var/log/messages, do the following:

tail /var/log/messages | sed -e "s/Feb/\\o033[31m&\\o033[0m/"

(You might have to do this as root to actually read /var/log/messages)

All that sed did was search for the word "Feb" and surround it with the same escape sequence as we used above.

You could expand it to color multiple words:

tail /var/log/messages | sed -e "s/\(Feb\|Mar\|Apr\)/\\o033[31m&\\o033[0m/g"

Which would color "Feb", "Mar", "Apr" - each in dark red.

I hope this gives you some idea of how to do what you need!

Answer 2

Pipe the output of tail -f into sed, and add in some ANSI escape codes. For example, the following will colorize all numbers in red (color 31) and all quoted strings in bright yellow (color 93):

RED=`echo -en '\e[31m'`
YELLOW=`echo -en '\e[93m'`
RESET=`echo -en '\e[00m'`
tail -f file | sed -E "s/([0-9]+)/$RED\1$RESET/g;s/(\"[^\"]*\")/$YELLOW\1$RESET/g"

Answer 3

Thanks to both scraimer and Adam, I came up with pretty much what I was after, and I thought I'd share it here for anyone else:

RED=`echo -en '\e[31m'`
YELLOW=`echo -en '\e[93m'`
RESET=`echo -en '\e[00m'`

# line breaks here are just for formatting
tail -f ~/access-logs/access.log | sed -e "s@\([0-9\.]\+\) - - 
    \[[0-9]\+/[a-zA-Z]\+/[0-9]\+:\([0-9]\+:[0-9]\+:[0-9]\+\) +[0-9]\+] \"
    \(.\+\) HTTP/1\.[01]\" \([0-9]\+\) \([-0-9]\+\) \"\([^\"]\+\)\".*
    @\n\2 $YELLOW\4$RESET (\1)\n$RED\3$RESET\nBytes: \5\nFrom: \6@g"

# here's the full line if you wanted to copy it
# tail -f ~/access-logs/access.log | sed -e "s@\([0-9\.]\+\) - - \[[0-9]\+/[a-zA-Z]\+/[0-9]\+:\([0-9]\+:[0-9]\+:[0-9]\+\) +[0-9]\+] \"\(.\+\) HTTP/1\.[01]\" \([0-9]\+\) \([-0-9]\+\) \"\([^\"]\+\)\".*@\n\2 $YELLOW\4$RESET (\1)\n$RED\3$RESET\nBytes: \5\nFrom: \6@g"

Which turns this:

182.108.1.20 - - [04/Feb/2009:23:24:41 +1000] "GET /images/exam_room.jpg HTTP/1.0" 200 8559 "http://www.myserver.com/courses/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"

Into this (but with some colour):

23:24:41 200 (182.108.1.20)
GET /images/exam_room.jpg
Bytes: 8559
From: http://www.myserver.com/courses/