What is the best way to do web scripting/web macros?

Question

I'm trying to streamline some of our tasks at my place of work, and it seems that quite a lot of our developers' time is spent doing semi-mechanical tasks on the web (specifically, editing online stores that use web-based interfaces). As such, I've been looking into some solutions that will allow these tasks to be done by scripts since I figure that could save us quite a bit of time per task. So before I really started digging into any of these, I was just wondering if the Stack Overflow community had any recommendations about which web scripting/macro solution would be the best.

Here are the requirements:

• Must be able to interact with web forms (not just downloading a page and scraping the file - the script must edit controls within a web form and then submit that form)
• The forms we have to edit are secure forms, so the scripting solution must be able to handle that (ie it's of no use for us to have an incredibly powerful scripting solution if a human being will have to sit there and watch it and manually re-login every few minutes)
• It would be really, really, really preferable if it could read local files and do some basic string replacement/manipulation on them (e.g. it would be nice to have a list of variables or some HTML code in a text file and then have the script replace the token "STORENAME" with that particular store's name before it inserts the code into the form)

Here are the scripting solutions that are on my radar so far (I haven't really looked into any of these, although I have played around with Chickenfoot):

• Chickenfoot
• iMacros for Firefox
• libwww-perl
• libwww for unix and C (while searching for libwww for perl I came across this, which I did not know existed until now)
• a more "general" macro solution like AutoHotKey

Has anyone here on Stack Overflow tried any of these solutions? If so, what did you like or dislike about them? Can anyone recommend one that is not on the list? (This is by no means an exclusive or exhaustive list). I would really love to automate a lot of our mechanical processes, and I hope the Stack Overflow community can help us out so we can hopefully avoid that much of the mind-numbing part of the work :). Thanks in advance!

edit: Re: platform - We have primarily WindowsXP terminals at work, but 1) we do have a few Mac test PCs, so OS X is a viable option, and 2) if it would mean automating a lot of these tasks, I'll build a Linux box if that is necessary. So platform is pretty much a non-issue.

Answer 1

I really like the watir framework (running on ruby) -- programmable, easy, uses IE to automate the Web (and so you get true browser behavior). Of course, it's Windows only -- you didn't specify whether that's what you use.

We've used watir to automate our tests of a webapp, and we're very satisfied.

Answer 2

@Asaf Bartov, actually, Watir is not Windows only. There are also FireWatir that works on Windows, Linux and Mac, and SafariWatir that works on Mac.

http://wtr.rubyforge.org/ http://wiki.openqa.org/display/WTR/FireWatir http://safariwatir.rubyforge.org/

Answer 3

A few suggestions:

• Write a Firefox extension
• Write a Greasemonkey script
• Write a Ubiquity script

Edit: I'm assuming these would act on web pages containing the data you want to submit to the forms.

Answer 4

If you're using libwww-perl, then don't stop there. Use WWW::Mechanize, which is a wrapper around LWP that does the browser interaction you want. The downside: It doesn't handle JavaScript (yet).

WWW::Mechanize home page

Answer 5

I'm actually a big fan of Chickenfoot, which I have started using a lot recently to provide scripting capabilities for Zotero, another Firefox extension.

Answer 6

Another option is use AutoIT + Firefox + mozrepl + FF.au3. This is powerful as AutoIT operates outside the browser (it's a fantastic automation tool in itself) so you can work through submitting multiple browser forms without losing context. Then the FF.au3 script lets you simply control Firefox via AutoIT3.

Answer 7

Hi, we use the iMacros tool you mentioned with good success:

Firefox + iMacros

Note that it also comes as free IE addon! The great thing about this is that the exactly same macros work in both browsers. IMHO very useful for quick and easy web testing and general web automation.

Tim

Answer 8

Use:

Firefox + iMacros to fetch:
Go to the page you like (e.g yahoo.com), bring iMacro in focus, hit record and search for some text. When first page shows, Save the page in a file.

Now you have a page you wanted.
You are done fetching at this point.
Save the macro and give it a name. Next time you can just run the macro you just saved.

You then clean up the html code you got by using this:

# this is php, do the proper thing in the language of your choice
# one or more spaces is equal to one space, remove tab, return  ...etc.
return ereg_replace(' +', ' ', trim(preg_replace("/(\r\n|\r|\t|\n|\'|\")/s", '', $string)

Now you have a clean html page with one line, a long one.
Now you can cut and slice as you wish by using a regEx.

Things to know:
You can start the macro in firefox via a cronjob (commandline) if you need full automation.

Answer 9

my impression of IMacros is you need to pay to unlock a lot of the goodies, so I went with Chickenfoot. Unfortunately the Chickenfoot mailing list is very quiet lately, so am not confident about its future...

Answer 10

I'm using iMacro/.net for a data extraction project. I like the ease of scripting for iMacros, which is important since the application may have up to 60+ websites to extract data from. For this project the overhead cost of running iMacros is worth the saved development time. It's not the ideal solution, but it's one way to interface with another website's data, assuming they don't provide a proper interface. The neat part is that I've created a small server cluster of iMacros scrapers to help distribute the load. Right now users can access the application servers through a website for adhoc requests, but eventually i'd like it to evolve into something more streamed line.

Answer 11

I'm a big fan of Selenium

http://www.seleniumhq.com

designed to handle automated browser testing though with scripting support (multiple languages) you can design macro templates that will fill in forms and such based on variables

also: anyone know a good macro that works with Flash?? Doesn't seem javascript is able to cross that line