site scraping: drop down controls and search button click

Question

Imagine a web site, where there are 2 drop down controls like min age and max age. After I choose min age and max age, I would like to hit the search button. How do I get the search results into an output stream and then I could navigate the DOM HTML. I already know how to naviate through dom html via htmlagilitypack. I just do not know how to invoke the drop down make selections and invoke search button. I want to do all this from console application. Is it even possible? I am using C# visual studio 2010.

Thanks..

Answer 1

If the html looks like this:

<form action="someurl">
    <select name="SomeOption">
    <option value="val1">val1</option>
    <option value="val2">val2</option>
    </select>
</form>

And you want to submit the form with "val1", then you need to execute an HTTP POST operation to "someurl" including the argument SomeOption=val1. That is essentially what the browser is doing. Of course the form you are trying to simulate will be a bit more complicated.

Is that what you were asking?

BTW, if you want to see exactly what is going on when the browser submits the form, check out Fiddler, an excellent tool made by a colleague at Microsoft.