Writing a C# program that scans ecommerce website and extracts products pictures + prices + description from them

Question

Hi,

I'm developing an ecommerce search engine that allows you to search for products in a lot of ecommerce websites.

How do I approach the matter?

I need an application that will be able to scan websites, parse their HTML and determine which of the images in the website are product images, which are product descriptions, which are product prices.

Would be happy to hear any idea, example.

Thanks in advance.

edit: My question is not how to get the HTML from the websites(which is called screen scraping) but more on how to parse that information and understand which of the html contains the actual data i am looking for, and which is not.

Answer 1

You may find this thread helpful in your quest. I had outlined the basic steps there. Here's the link to all questions tagged as "Screen-scraping" on SO. Also, lots of material on the web - Google.

Answer 2

Most of the sites you'd be scraping (more correctly web-scraping) have partner APIs for "reseller" type deals. For you to circumvent that with screen scraping will quickly find your IP blocked by their traffic servers, and potentially put you in a legal situation.

This is ethically dubious at best.