WebBrowsing in C# - Libraries, Tools etc. - Anything like Mechanize in Perl?

views:

478

answers:

+8 Q:

WebBrowsing in C# - Libraries, Tools etc. - Anything like Mechanize in Perl?

Looking for something similar to Mechanize for .NET C#.

If you don't know what Mechanize is.. http://search.cpan.org/dist/WWW-Mechanize/

I will maintain a list of suggestions here. Anything for browsing/posting/screen scraping (Other than WebRequest and WebBrowser Control).

Parsing

HTMLAgilityPack - http://www.codeplex.com/htmlagilitypack

Web App Testing

WatiN - Web Application Testing Framework (.NET) - http://watin.sourceforge.net/
Selenium - http://seleniumhq.org/
Design Canvas - But it costs money =(

Other

Tools for finding page strucutre - Firebug for Firefox or Internet Explorer Developer Toolbar for IE

Note: WatiN is pretty close to what I am looking for, except it opens up a browser, which is annoying and awesome at the same time. Depends on what you are doing. Converted some stuff to run in Watin...fun to watch.

+2 A:

You need to use the HTML Agility Pack, which can parse tag soup from real websites into a DOM structure.

SLaks 2010-01-27 20:05:52

That's good for parsing the pages once you get them, but what about everything else? Like, logging into sites, managing cookies, handling redirection etc?

Blankasaurus 2010-01-27 20:14:52

Use the `WebClient` class, or HAP's `HtmlWeb` class, or the `HttpWebRequest` class with a `CookieContainer`.

SLaks 2010-01-27 20:15:11

I know you can do it that way in .NET, looking for something a little higher level. Like Mechanize. Its OK if there isn't one, I just was curious if there was a library that did what I have done using WebClient etc.

Blankasaurus 2010-01-27 20:20:37

A friend of mine wrote a program that does what my C# program does using Mechanize, and it is 13 lines. Mine is WAY more. 13 LINES! =P

Blankasaurus 2010-01-27 20:22:10

+1 A:

You can use the WebBrowser control, which can be automated to an extent.

John Saunders 2010-01-27 20:29:08

@Downvoter: why bother to downvote if you can't be bothered to say why you downvoted? Do you think I _care_ about 2 points?

John Saunders 2010-01-29 21:02:20

You want HttpWebRequest for automating web requests and HtmlAgilityPack for processing the resulting HTML.

qntmfred 2010-01-27 20:33:42

+2 A:

I've been using WatiN to great effect. It's an easy way to 1) automate user input w/ IE and 2) navigate the DOM.

ZaijiaN 2010-01-27 21:23:01

+1 Best answer so far. Thanks!

Blankasaurus 2010-01-28 03:06:13

WatiN requires actually launching browser windows. Mechanize is in-memory only

qntmfred 2010-01-28 04:28:47

It is a bit slow, but it is fun to watch the browser be automated. Even more fun is to call .Highlight on whichever part of the DOM you're processing, and you can watch the processing happen.

ZaijiaN 2010-01-28 15:02:04

Another benefit of WatiN - since it interacts with IE, it can process the *live* DOM - that is, there's no problem if the page been built by javascript. I don't believe the HTML agility pack can do that.

ZaijiaN 2010-01-30 04:48:44

@qntmfred: You can hide those windows: "IE.Settings.MakeNewIeInstanceVisible = false". There is another property which handles dialogs: "IE.Settings.AutoStartDialogWatcher".

tsocks 2010-03-30 08:05:34

+1 A:

You can also use Selenium. It's for unit testing web sites. It has a java application that drives the browser and a C# interface that you can write your code in. It also has the downside of showing the browser, but it's pretty full featured in terms of control, waiting on responses and getting the results.

Pete McKinney 2010-01-29 14:35:56

+1 A:

Design Canvas is the best tool out there for this type of thing. Works with IE, Firefox, Safari, or an in-memory browser. It allows you to record and then playback any kind of web interaction.

jaws 2010-02-23 05:09:39

ansaurus

tags:

views:

answers:

WebBrowsing in C# - Libraries, Tools etc. - Anything like Mechanize in Perl?

related questions