Automatic testing of web pages (and generating from use cases by DSL)

Question

Hi,

My goal is:

• Our customers could generate new web-tests.
• Our continuous integration server makes a test-environment deployment; it should execute the tests against it
• The test could also be run against some other environment.

(Final acceptance tests should be made by the customer, to test fonts etc, but this would be a great pre-acceptance check for our test-environment. Customers could focus on other things than now.)

Usually some property (like text field id) has changed or something and the tests will break in a few weeks. It seems that recorded tests broke often, so it's better to easily record a new one than trying to maintenance and modify an old test.

Now, I found a whole new approach. Maybe recording is not the right way. How about, if our customers could make use cases in a human readable own language which the machine would understand and compile to web-recording (with Domain Specific Language, DSL). This is not sci-fi, it has been already made, so read on. :-)

I have tried to use these automatic web testing frameworks:

• Visual Studio web test (Customers can't execute)
• Selenium (Works only with Firefox, our customers have IE)
• WatiN (.NET version of Watir, recorder seems to be a bit buggy)
• HP Quick Test Pro (Not easy enough to make new tests)

None of these have provided actually what I need... but Selenium is the closest one.

Our customers speak Finnish, so in the beginning of a software project, in specification phase, user writes a use case like:

• Avaa "OmaLomake"
• Syötä "Tuomas" kohtaan "nimi"
• Paina "Seuraava"

Translation:

• Open "MyForm"
• Insert "Tuomas" into field "name"
• Press "Next"

Now... This is a human-readable use case, but also it can be compiled to automatic web acceptance test. Open, Insert, into field and Press are keywords, others are values.

What kind of DSL tool would be good for this?

Microsoft is making a new DSL-making-tool in their Oslo-project called MGrammar. It means that you can make a custom language to make it easy for non-technical people to work with machines. (The same basic idea that was (and failed) with Cobol and Visual Basic.)

I found that someone has already made this kind of DSL with MGrammar, but it is for Watin, not Selenium: http://www.codinginstinct.com/2008/11/creating-watin-dsl-using-mgrammar.html

So the continuous integration server process will be:

1. Fetch a new version from source control (as usual).
2. Build, run unit tests and analyze the code (as usual).
3. Make an installation package and tag version in version control (as usual).
4. Compile use cases to web tests
5. Run web tests
6. Accept/Reject the software :-)

Running a web-test in the continuous integration server usually means a lot of configuration work. So, before I try this, I'm curious, what do you think? Have you used same kind of setup, and what are your experiences? (What exact environment?)

How about DSL, will it have enough power for use cases or will it be another endless development task? Will the customers ever generate the tests?

Answer 1

First of all, Selenium does work with IE and other browsers as well as Firefox; cross browser support is one of its strengths. Here's the list of supported browsers.

However, if you want a human language-based DSL for writing your tests, take a look at Cucumber - the syntax is almost exactly like your example above. Cucumber already has Finnish language support - see the examples at this link.

Answer 2

Fitnesse and Selenium Integration tools such as Selenesse(http://github.com/marisaseal/selenesse) or Fitnium(http://www.magneticreason.com/tools/fitnium/fitnium.html) can also serve your purpose. However, you need to find answer for who will put the element locators in the test cases written by customers. If customers put the locators using the recorders, it may not be possible to maintain. If customers write the steps and a automation tester/developer can put those locators using regex, custom location strategy, this approach may work.

Answer 3

The TestPlan software uses a specialize language for writing tests. It is highly domain specific and works very well in web environments. It supports the Selenium backend so you gain that compatibility, plus it can run without a browser, for even faster tests. I have used it on some fairly large web projects in the type of setup you are looking for.

Your example script might look like this:

GotoURL /SomePage
Click MyForm
SubmitForm with
   %Params:name% Tuomos
   %Submit% value:Next
end

That's it. It nicely describes what the user wants to do and is a functioning test. You can combine scripts into units and have custom function as well. So if you really wanted you could write the Finish equivalents to the names.