In asp.net how to screen scrape multiple records when paging is implemented for the results?

Question

I know how to screen scrap a page and read the data. But,I need help on how to get all results when paged.

Will HTML Agility Pack help in this issue or any other tools available for this or any other way?

Answer 1

First of all I would try to avoid screen-scraping unless there is really no other way to gain access to the data. In an environment with limited traffic (e.g. Intranet) screen-scraping could be a useful technique but in a live web environment most site owners are opposing it.

However, I think the first question is how the paging was implemented in the first place? Successful screen scraping is pretty much down to having an understanding about how the target site works.