Screen scrape a web page that uses javaScript and frames

Question

Hi, I want to scrape data from www.marktplaats.nl . I want to analyze the scraped description, price, date and views in Excel/Access.

I tried to scrape data with Ruby (nokogiri, scrapi) but nothing worked. (on other sites it worked well) The main problem is that for example selectorgadget and the add-on firebug (Firefox) don’t find any css I can use to scrape the page. On other sites I can extract the css with selectorgadget or firebug and use it with nokogiri or scrapi. Due to lack of experience it is difficult to identify the problem and therefore searching for a solution isn’t easy.

Can you tell me where to start solving this problem and where I maybe can find more info about a similar scraping process?

Thanks in advance!

Answer 1

Iframes aren't a problem - just access the embedded iframe URL directly. You will find that it redirects in the browser unless you disable JavaScript.

Description and date can be extracted straight from HTML source. However prices are images which will make scraping them more cumbersome.

Answer 2

You can try IRobotSoft web scraper. It has good frame support and is free.

Answer 3

I used excel web query and it works perfect. You can find a lot about scraping with excel on youtube if you search for mrexcel. Thanks, Mello