tags:

views:

147

answers:

1
Q: 

Price Comparison Script for Products

How would you build a price comparison script? I know Amazon offers a public API, but I saw these two sites goodreads, bookdope which compare book prices, retrieve prices from Walmart and others websites that do not offer APIs. How do you get prices from sites that do not have an API?

I'm using C# and ASP.NET MVC.

+1  A: 

If the site does not offer an API then you need to do what is called screen scraping. You get the HTML of the page and you have your program extract the information out of it.

This works, but beware of the problems.

Companies ofen don't like you doing this. If they wanted you to get the information they'd have provided an API. They may even detect a high number of hits from your specific location and block you.

Above all else, the page is designed for a human to read in a browser. They can change it at any time and your application may fail as a result.

Colin Mackay  2009-07-16 18:45:54
This is the express train to insanity John.We tried this 10 years ago and have since decided that this is a very bad idea.Every subtle change to the sites you are [sraping] will potentially bring your site to a screaming and spectacular halt.I'd give the sites a call and see if there is a way to get the data. You never know, they may be willing to play ball.
griegs  2009-07-17 05:13:56
Indeed - That's why I said there were problems with screen scraping.
Colin Mackay  2009-07-17 06:33:51

related questions

Displaying Flash content in a C# WinForms application
How to get the value of built, encoded ViewState?
Unhandled Exception Handler in .NET 1.1
How do I connect to a database and loop over a recordset in C#?
How do I most elegantly express left join with aggregate SQL as LINQ query
Get a new object instance from a Type in C#
.NET Testing Framework Advice
Automatically update version number
What is the difference between an int and an Integer in Java/C#?
How to write to Web.Config in Medium Trust ?
WinForms ComboBox data binding gotcha
How do you sort a C# dictionary by value?
Adding Scripting functionality to .NET applications
Floating Point Number parsing: Is there a Catch All algorithm?
How do I print an HTML document from a web service?
Decoding T-SQL CAST in C#/VB.net
Anatomy of a "Memory Leak"
How do I get a distinct, ordered list of names from a DataTable using Linq
Reliable Timer in a Console Application
How do I fill a DataSet or a DataTable from a LINQ query resultset ?
What's the difference between Math.Floor() and Math.Truncate() in .NET?
How do I calculate relative time?
How do I calculate someone's age in C#?
Are there any conversion tools for porting Visual J# code to C#?
When setting a form's opacity should I use a decimal or double?