Is it possible crawl ASP.NET pages?

Question

Is there a way to crawl some ASP.NET pages that uses doPostBack as events calling?

Example:

Page1.aspx:

Contains 1 LinkButton that redirects to Page2.aspx

Code-behind for LinkButton Click event:

 Response.Redirect("Page2.aspx")

In client side this code is generated on click event:

doPostBack(...

Is it possible crawl pages using only HttpWebRequest?

I know that use Response.Redirect is not a good idea in this case, but I don't have choice.

Answer 1

Yes, it's possible if the code follows a well predictable pattern. You would have to gather the form data from the page and simulate what the doPostBack function does (putting some values in some hidden fields), and send a POST request to the server. What you get back would be a redirection page, so you would have to parse that to get the url of the target page.

If you mean if search engines like Google will crawl the pages, then that is very unlikely. They might attempt to follow some common patterns of posting and script linking, but generally you need to use proper links between the pages to be sure that they are crawlable.