Screen Scraping with Ruby on Rails

Question

Hi everyone,

Am a newbie in RoR. I want to develop a rails app, which could search flight info from my rails app, fetch the flight informations from various airlines sites(for example; airasia and malaysia airlines). display those results back in my rails app.
My search module should include the following textfields:

From: Kuala Lumpur To: Dubai Departure Date:2/11/2009 Return date: 19/11/2009 Departure Time: 12:00 Number of tickets: 2

user gets to enter the information above on my site and
a script should search flights according to that information on various airlines sites and
display the results with the tag of which airlines is providing this flight.

Can either use "Scrapi" or "Scrubyt"

can anyone offer an assistance?

Thanks, Hassan

Answer 1

can anyone offer an assistance?

Don't. Find another get rich quick scheme. Or at least don't rely on screen scraping.

There's so much potential for it to go wrong. Find another way to get the information to aggregate. Use API's if they exist and store the schedules in the databases (they rarely change on a week to week basis). Storing things locally makes things faster for your customers, it also allows you to provide more services.

Here are just a few of the potential problems with screen scraping:

• Minor updates to the sites you're scraping can break your tool.
• Scraping attempts from your server's IP can be denied.
• Depending on the information, you could possibly even get sued.