How to build a web crawler to find a specific advert, which is in an iframe loaded by Javascript

Question

I'm trying to find all instances of an advert on a website. The advert is in an iframe which is loaded by javascript (it doesn't appear at all if javascript is turned off). Detecting the advert itself is extremely simple, both the name of the flash file and the target of the href always contain a certain string.

What would be the best "starting point" for achieving this? At the moment I'm considering an Adobe AIR app, which could crawl the site and examine the DOM to find the ad, and would run javascript and load the content of the iframe. The other option I can think of is using Firefox as the platform (using maybe GreaseMonkey or Selenium? I don't really know how to leverage Firefox like this).

Does anyone know of anything suitable to build this, or have any suggestions on using Firefox to do it?

---

Extra details:

Being CPU intensive isn't really an issue, nor is anything depending on a browser being open. This doesn't need to run on a headless server, it will be running on a powerful desktop box. OS is also not an issue. It would be advantageous if the crawler loaded each page multiple times, as the advert is in rotation. While the crawler does need to execute the javascript and load the content of the iframe, it does not need to be able to display flash files.

Answer 1

If the ad is only displayed when javascript is enabled, you are going to have a problem, as no crawler is going to be able to read the web page in that matter.

Is there something in the javascript code itself that could be a tipoff to where the add is displayed? If so, maybe you can check that.

I've tried similar stuff before, and I used BeautifulSoup in python, and it worked really well.

Answer 2

I think You don't want a crawler. You are going to run it on a single page and not want it to look around the internet through links, right?

If so - You want to find something on the page with javascript on. You then just have to use javascript.

You'll need:

1. the site :)
2. correct rights to access its content - use greasemonkey for FF or user scripts in Opera
3. a code similar to this jQuery sampe:

finding stuff in iframes:

$('iframe').each(function(){
     $(this).contents().find('object').each(function(){
      if($(this).attr('name').match(/regex/)){
        $(this).remove(); //or do whatever You want
       }
      });
    });

caution: accessing iframe contents may differ in browsers and is influenced by time when You run the script

Answer 3

An alternative to using a "browser as a crawler" is using HTMLUnit as the page says, it's:

HtmlUnit is a "GUI-Less browser for Java programs". It models HTML documents and provides an API that allows you to invoke pages, fill out forms, click links, etc... just like you do in your "normal" browser.

It has fairly good JavaScript support (which is constantly improving) and is able to work even with quite complex AJAX libraries, simulating either Firefox or Internet Explorer depending on the configuration you want to use.