What's the Right Tool for automating refresh+scrub of a page behind a login?

Question

A friend just asked me if I could help him in automating something that is taking up a lot of his time. He grades essays online so basically there is a website that he logs into and sits on a page refreshing it until some data appears (new essays to be graded). Instead of doing this he would like to simply be notified when there are new items that require his attention.

There is no API to work off of and a login is required to access website. What in your opinion is the right tool for this job?

I'll post my idea as an answer but I'm curious as to what everyone suggests.

Edit: He is running windows (Windows Vista). Browser doesn't really matter as long as the site runs in it.

Answer 1

My idea is to write a script for Firefox's Greasemonkey plug-in.

Basically he would log into the page and turn on the script which would constantly be refreshing and scrubbing the site for new items. If some are found it pops up a message and plays a noise (or something like that).

I've never worked with Greasemonkey before but it seems like something like this should be pretty simple.

Answer 2

You can write a small Ruby script using Waitr (see) which will actually use a browser and allow you to scrape data, or scrubyt, (examples).

Here is what scrubyt looks like. I'd recommend you do something like generate an email or IM message but can do whatever you like. Schedule it to run in CRON somewhere - beauty of this approach is it doesn't matter if his computer is on, browser is open, etc.

#simple ebay example

require 'rubygems'
require 'scrubyt'

ebay_data = Scrubyt::Extractor.define do

     fetch 'http://www.ebay.com/'
     fill_textfield 'satitle', 'ipod'
     submit

     record "//table[@class='nol']" do
       name "//td[@class='details']/div/a"
     end
end

puts ebay_data.to_xml