YQL Web Server Log File regex example.

Question

I am looking for example code using YQL to handle a web server log file.

YQL table for Apache access logs asks "What about a table for Apache access logs?" to which spullara replies:

If we add something like this it will likely be with a regex based line reader that you could then apply to apache logs.

later Paul Tarjan says:

There is now a regex table

http://developer.yahoo.com/yql/console/?q=select%20%20from%20regex%20where%20expression%20%3D%20%22%28.%29%22%20and%20text%3D%22test%22&env=http%3A%2F%2Fdatatables.org%2Falltables.env

So, it looks the regex feature has been added to YQL.

I would like to see a complete YQL example using this regex feature using this sample log file. Here is a line from the sample log file:

69.12.127.10 - - 02/Sep/2009:01:53:52 MDT "GET /holeman/cgi-bin/get_page.cgi?../cxx/sw_stack" HTTP/1.1 200 999 Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.2) Gecko/20090810 Ubuntu/9.10 (karmic) Firefox/3.5.2

Display just the IP addresses and pages referenced which for the above sample line would be:

69.12.127.10 /holeman/cgi-bin/get_page.cgi?../cxx/sw_stack

In the Paul Tarjan example the values for expression and text are hard coded. How would one reference the text in the file http://sky.prohosting.com/holeman/yql/serverlog/20090902.txt? Also, for expression how can one use a reference rather than a hard coded value?

Answer 1

Here's a very primitive regex to do what you want.

$.getJSON('http://query.yahooapis.com/v1/public/yql?q=select%20match1%2Cmatch7%20from%20regex%20where%20expression%20%3D%20%22%5E(%5C%5CS%2B)%20(%5C%5CS%2B)%20(%5C%5CS%2B)%20(%5C%5CS%2B)%20(%5C%5CS%2B)%20%5C%5C%5C%22(%5C%5CS%2B)%20(%5C%5CS%2B)%5C%5C%5C%22%22%20and%20text%3D%2269.12.127.10%20-%20-%2002%2FSep%2F2009%3A01%3A53%3A52%20MDT%20%5C%22GET%20%2Fholeman%2Fcgi-bin%2Fget_page.cgi%5C%5C%3F..%2Fcxx%2Fsw_stack%5C%22%20HTTP%2F1.1%20200%20999%20Mozilla%2F5.0%20(X11%3B%20U%3B%20Linux%20x86_64%3B%20en-US%3B%20rv%3A1.9.1.2)%20Gecko%2F20090810%20Ubuntu%2F9.10%20(karmic)%20Firefox%2F3.5.2%0A%22&format=json&env=http%3A%2F%2Fdatatables.org%2Falltables.env&callback=?', function(data) {
  console.log( data.query.results.matches.match1 );
  console.log( data.query.results.matches.match7 );
  });