How do I automate navigation to a website that requires authentication?

Question

Here's what I'm trying to achieve. I would like to write a script that will navigate to a website that requires me to be authenticated as myself, say Facebook, Live Spaces, Twitter or any other, and then have that script search for certain information on one of the pages of the website.

I've done something similar in the past with the Windows.Forms WebBrowser control, which is a full blown implementation of IE that can be controlled through code and will store whatever cookies you get once you're authenticated, but it was very unfriendly to modify and I was hoping to use a scripting language instead, maybe Powershell or something of that sort.

Are there maybe some good tutorials about this out there on the web?

Thanks!

Answer 1

There are many alternatives. In the past I have automated my browser around authenticated sites with Chickenfoot, or from the console with Mechanize.

Then crawl the site and examine each pages HTML for your data.

Answer 2

If Python is your first choice, Mechanize + Twill + Beautiful Soup.
Have a look to SO screen-scraping tag too.

Answer 3

You can use IRobotSoft web scraper. It automates an IE WebBrowser control for web data mining. See their forum here http://www.irobotsoft.org/bb/ for typical questions.