scraping dynamic content

Question

Hi,

I am working on a web scraping project. do any body have idea of scraping dynamic content.

Dynamic content on base of query string is similar to static content but dynamic content based on some event of a control within same page is the point where i am stuck. because in this case page url remain same.

I am using C#.

Thanks in advance

Answer 1

Your question is rather general.

I'm not sure what you mean by event of a control, but as long as a browser generates http request you can catch it using tools like Firebug for Firefox or tools built in Google Chrome and see what is actually being sent to the server. So called AJAX requests are nothing else than standard http requests, it's just that web page is not reloaded as a whole.

Based on that information and page source it is possible to figure out how to create range of reguests that would simulate user interaction with dynamic elements on the page.