Dynamic screen scraping with PHP and getting past javascript

Question

I have a website that provides a price comparison for students textbooks. I wrote a ruby script to go class by class and grab all the textbook information and store it in a database that the website can query for book information. The problem is that the bookstore keeps changing the books needed for each class so I need to figure out a way to get rid of the database and screen scrape on the fly with a script on a web page.

The schools website uses ASP to dynamically load the books needed after the page has loaded and I don't know a lick of ASP to figure out how to reverse engineer it. I have poured through the source code and used firebug and have no clue where the data is being loaded from in order to go directly there. The URL my ruby script uses is http://bookstore.umbc.edu/SelectCourses.aspx?src=2&type=2&stoid=9&trm=Fall%2009&cid=4156 where cid is the unique number for the class. When loaded with curl I get the following - bookscrooge.com/curltest.php (which is what you would expect).

Anyone have any incite or looking to make a little quick cash / partner in a startup? Mike

Answer 1

If you want to know where ASP is loading the book data from, the simplest solution would be to ask the bookstore themselves. It's possible they might not want to tell you (or that they don't know), but it would certainly be a lot easier than trying to reverse engineer a language you don't know. Alternatively, try and find someone who does know ASP and who is willing to help you. I'm sure Stack Overflow must be swarming with such people.

Good luck!