ansaurus

Question

Answer 1

+2 A:

You can try with Selenium. Take a look at: http://seleniumhq.org/. Also, you can write web crawler in many languages of your choice. If you prefer .NET, take a look at HttpWebRequestClass.

vucetica 2010-07-10 11:20:40

Answer 2

+1 A:

It's not clear from your question, but I assume you're trying to make your C# app automate a Web browser that's running as a separate application (as opposed to embedding a WebBrowser control in your app and automating that).

This sounds like the sort of thing WatiN is meant to automate. It's intended for automated tests (which may or may not be what you're doing) but gives you an easy way to drive an external instance of either IE or FireFox, navigate to pages, find text fields and type into them, click buttons, etc.

A sample code snippet from their site:

using (var browser = new IE("http://www.google.com"))
{
    browser.TextField(Find.ByName("q")).TypeText("WatiN");
    browser.Button(Find.ByName("btnG")).Click();

    Assert.IsTrue(browser.ContainsText("WatiN"));
}

Joe White 2010-07-10 11:34:32

In short i want dynamic content of the page which is coming against event of a control, based on some value. which in my case is onchange event of dropdrown based on selected value of the dropdown

Sakhawat Ali 2010-07-10 11:36:04

I am writing a web scrapper and for that i want dynamic content of that page. am i clear now?

Sakhawat Ali 2010-07-10 11:40:13

@Sakhawat, Yes to some extent you are clear now. :-)

GS_Guy 2010-07-10 11:42:12

Initially, from OP's description, I thought the site used AJAX, so something like WatiN would be useful (since it loads the site in a browser and actually runs all the JavaScript). However, I looked at the site OP linked to, and it actually does a postback -- and that can be easily simulated with HTTP calls with something like the HTML Agility Pack, which should be faster than out-of-process automation. So while I think WatiN would work, it may not be the best solution for this particular problem.

Joe White 2010-07-10 12:00:33

@Jeo thanks for all your helpful replays, i think i am very close to the solution. if you look into the source on this page you will find this:onchange="javascript:setTimeout('__doPostBack(\'dnn$ctr410$listing$priceDateList\',\'\')', 0)"now if i use HTTPWebRequest object to get content, what should i give in postback data?

Sakhawat Ali 2010-07-10 12:58:30

That depends on what the page's __doPostBack method does. The easiest thing is probably to use a tool like Firebug or Fiddler to see what actual data is being posted. I can't help you there, never having used either for anything like this -- so if you have trouble figuring it out, it's probably worth posting a new SO question.

Joe White 2010-07-10 15:58:53

Answer 3

A:

I think you are trying to do page scraping. If yes then HTML Agility Pack is best choice for you to use in your .NET application and scrap the page programmatically.

GS_Guy 2010-07-10 11:40:21

ansaurus

tags:

views:

answers:

navigating site programmatically

related questions