How do I create a web crawler in ASP.NET?

Question

I am wondering if there is a way to make a web bot/crawler for a website in ASP.NET.

I have to grab information from one of our payment providers, but they do not have an API so the only current way to grab the information automatically would be to log in to their page and then fill out a form and retrieve the information.

Is there any way to do this?

Answer 1

• http://www.codeproject.com/KB/IP/Crawler.aspx

• http://arachnode.net/

Answer 2

you can do this with the webclient() in .net but I wouldn't do it from a web page. Make it a service you can call and get data from.