Is there an equivalent of Don Libes's *expect* tool for scripting interaction with web pages?

Question

In the bad old days of interactive console applications, Don Libes created a tool called Expect, which enabled you to write Tcl scripts that interacted with these applications, much as a user would. Expect had two tremendous benefits:

• It was possible to script interactions that otherwise would have had to be repeated by hand, tediously. A classic example was dialup Internet access hell (from the days before PPP).

• It was possible to write scripts to test one's own interactive applications, programmatically, as part of a regression suite.

Today most interactive applications are on the web, not on the console. Hence my question: is there any tool that provides the ability to interact with web pages and web forms programmatically, much as Expect provides the ability to interact with console applications programmatically?

(The closest thing I am aware of is Chickenfoot.)

Answer 1

You might be looking for Selenium

Answer 2

I've used Selenium RC in conjunction with Python to drive web page interactions programmatically. This has allowed me to write pretty extensive user tests in which forms and inputs are driven and their results are measured.

Check out the Selenium IDE on Firefox (as mentioned above). It allows you to record tests in the browser and play them back, either using the IDE itself, or the Remote Control app.

Answer 3

Perl Mechanize works pretty well for this exact issue.

HTTPS and some authentication issues are tricky at times. I will be posting couple questions about those in the future.

Answer 4

In addition to Selenium, if you're doing the Ruby/Rails thing, there's Webrat.

Answer 5

I did a ton of Expect work in a former life and always thought Don Libes' Expect book was one of the best-written and most enlightening technical books I'd ever seen.

Hands down I would say that Perl's WWW::Mechanize library is what you want. I note above that you were having trouble finding documentation. There is good documentation for it! Look up the module's distribution on search.cpan.org and see what all is packaged with it. There's a FAQ, Cookbook with examples, etc. Plus I've always been able to get help on the web. If you can't get it here, try at use.perl.org or perlmonks.org. WWW::Mechanize's author, Andy Lester, is present on Stack Overflow. (He's also an all around friendly and helpful guy.)

I believe WWW::Mechanize also has a program that is analogous to Expect's autoexpect program: you set up a proxy process running this program as a server, point your browser to it as a proxy, perform the actions you want to automate, and then the proxy program gives you a WWW::Mechanize program for you to use as a base for your project. (If it works like autoexpect, you will certainly want to make modifications from there.)

As mentioned above, WWW::Mechanize is a browser (to be more exact, it is a web client or http client) that happens to be programmable. The last time I looked, there was even work in progress to make it support JavaScript.