Apache log analyzer which outputs user flows

Question

Hi Folks

I'm looking for a peice of software which can help me debug issues on a website by using apache logs. Here's the scenario.

I have a client on a website who has just performed something which is unexpected, I have his IP address because it got logged with a purchased transaction.

Is there a program out there that'll let me view step by step which a selected user did on the site, using the apache logs.

e.g. IP address a.b.c.d set...

IP address a.b.c.d
Time Date / Path
18:02     / index.htm
18:03     / shop.htm
18:04     / product1.htm

That way I can see exactly how a user arrived where they did.

Thanks for any help you can give.

Answer 1

That's one line of bash. I hope you've got your apache log type set to "Combined", and a bash shell at your disposal. Then the command

grep $IP $LOGFILE | awk '{ print $4$5, $7, "("$11")" }'

will yield something like this (Referers are in parentheses):

[07/Mar/2010:14:11:45+0100] /Doku/strawberrylimes.html ("-")
[07/Mar/2010:14:11:45+0100] /doku.css ("http://da.andaka.org/Doku/strawberrylimes.html")
[07/Mar/2010:14:11:45+0100] /images/zutaten.jpg ("http://da.andaka.org/Doku/strawberrylimes.html")
[07/Mar/2010:14:11:56+0100] /images/prost.jpg ("http://da.andaka.org/Doku/strawberrylimes.html")
[07/Mar/2010:14:11:56+0100] /images/vollermixer.jpg ("http://da.andaka.org/Doku/strawberrylimes.html")
[07/Mar/2010:14:11:57+0100] /favicon.ico ("-")

(Just remember to replace $IP and $LOGFILE with your values ... the results are from my webserver and were produced by one page visit.)

If you're only interested in, say, requests for .html files, extend the command line with another grep:

grep $IP $LOGFILE | awk '{ print $4$5, $7, "("$11")" }' | grep -E '.+ .+\.html.* .+'

The last grep filters out any lines that do not have a ".html" string in the second column of the output. But it also filters out all requests on directories!