Automated testing of GUI

Question

This question isn't about unit-testing. And it is for a desktop product.

This is about testing of the gui and testing that the right stuff is input in the right text box at the right time.

A company I used to work at used WinRunner (different department so I don't know much more that that), but that has now been shutdown by HP but they don't seem that bothered whether you stay with HP or go elsewhere. You can't read about the product until you've signed up which is annoying.

The tool has to work with MFC (non-negotiable) and the ideal tool will also...

• be automated.
• be scriptable.
• work with different screen resolutions automatically.
• be able to 'spy' on individual static text boxes, etc.
• intuitive enough so non-programmers can create the scripts.
• have reporting tools, including email of individual users.

What do other SO users do for automated GUI testing?

Answer 1

See User Interface Testing.

Answer 2

Desktop or Web apps alike have the same testing pattern here (I've worked in both).

Place as little logic as possible in the UI and test everything below it. So you say, but what if I want to test that such and such happens when a button is clicked? The method that is called when that button is clicked should call out to another class that actually does the thinking.

You might say, but I'm using some static classes/methods that can only exist in my UI, wrap those with an adapter and utilize that interface in order to make your code testable.

The parts of your GUI tests that you want to automate should be automated below the UI. There are parts of your GUI that you can't automate. Checking to make sure things "look right", or testing that you can see certain elements, etc. All of that is what your humans should be doing. Ensuring that events are properly firing, and that values are being appropriately returned from your business objects, that's all unit tests.

I can see from your question that you've already pretty much decided you need an automated GUI tester, but that's not the right tool for this job. If you decide to use that you're trying to find the best way of doing the wrong thing.

If you think that this isn't about unit testing because you're testing GUI interactions, then I can guarantee you're not unit testing close enough to your UI. If you were, you'd feel like most of what you'd be testing was redundant.

If you disagree with me, please post some reasons and we'll hash this out.

Answer 3

We use the SAFS framework for Rational Robot (RRAFS). There are also SAFS implementations for WinRunner (WRAFS) and it looks like they have a new "Image-Based Testing" implementation, which I'm not familiar with.

This framework does a nice job of seperating the UI implementation from the test scripts. I've tested four releases of a web application developed by two different teams (one team using classic ASP, one using ASP.NET) and I only had to change the application map of my UI objects, the tests themselves didn't need to change.

That said, the language of the framework is cumbersome and takes getting used to. It's not very robust, in terms of language constructs, but with some effort you can do anything you need to. It's sort of like "programming" in Windows Batch language, but for tests ;)

To address your individual requirements above:

1) The tool has to work with MFC (non-negotiable). The SAFS framework uses a 3rd party "record-playback" tool to drive the tests, like Rational Robot or Mercury WinRunner. If that tool can interact with MFC apps, then the framework can. I don't know how the "Image-Based Testing" implementation drives the tests, but I'd guess it can also work with MFC.

2) be automated. The SAFS framework integrates with the STAF framework, which will allow you to automate the execution of your tests. I have a proof-of-concept test that uses STAF to start a VM image from a pool of images, install the application under test, run the RRAFS test, and put the results on a web server for others to get at.

3) be scriptable. Yes, but as mentioned, it's not the most robust programming language. I wrote an Excel add-in that our testers use to write their tests that simplifies things a little bit.

4) work with different screen resolutions automatically. Yes, since it's looking "under the covers" at the UI objects and not the screen. Except for maybe the "Image-based Testing" option...

5) be able to 'spy' on individual static text boxes, etc. Yes, you can wait for a UI object to appear, disapper, to have a value, for a value to be changed, etc.

6) intuitive enough so non-programmers can create the scripts. With some training. We've had limited success. Some QA folks can write the tests, some struggle.

7) have reporting tools, including email of individual users. Yes, using the STAF framework you can post results to a web server, send out emails, etc.

Answer 4

There are a lot more (open source) alternatives if you are testing a web product. For a desktop product, some popular general purpose desktop GUI automation tools below (in no particular order). I've worked with all of these personally, and they all get the job done. If you choose to go with a vendor tool, get POCs for the ones you are considering, and make the decision based on what tool is the best fit for the company in general. One tool may be a better fit for a particular application, but there may be other projects/applications to consider.

• QuickTest Pro (HP's successor to WinRunner)
• Rational Functional Tester (IBM's successor to Robot)
• TestPartner

Answer 5

Lots of good answers here, but I wanted to address this goal statement, specifically:

• intuitive enough so non-programmers can create the scripts

I can understand why you'd want this, but it's a lot harder than you might think. While you can find any number of tools out there that'll claim to make writing scripts easy, in practice, you're going to need at least some people on your automation team who understand programming. Writing scripts that are reasonably robust is going to involve one or more of looping, if/then/else, and subroutine calls. Not the kind of thing that non-programmers are going to find intuitive.

Be especially wary of the idea that you can "record" a person using a tool, then play it back for testing. That sort of "automation" is often so brittle that you'll end up modifying or re-recording the script for nearly every change in the software.

Answer 6

Hello Grahm -

Coming from a strong Mercury/HP background, I would highly recommend using QuickTest Professional for your GUI testing. It has a lot of the same functionality as WinRunner, but without a lot of code. Simple GUI checks can be done through the QTP interface with minimal, if any, custom VB code. Checks for text next to boxes can be done with simple compares using the datasheet in QTP.

If you are used to WinRunner, and know VBScript (not so much TSL), then I would definantly look at QTP.

As far as your other requirements, QTP also has the Spy feature, like WinRunner, that will list all properties and actions you can perform on objects. And as for simplicity of use, at my old job, we would have business or system testers create simple smoke scripts, then I would take them and code them for more in-depth testing (multiple data values, error checking, etc). And as for reporting, QTP will do simple reporting of pass/fail/warning on tags that you put in, along with custom data you can input. So you could use a case statement to populate your output values based on your results. It won't do e-mail naitevly, but if you integrate with TestDirector/QualityCenter, you can setup through there, along with automating the kick-off of your scripts, and parameterizing data right from there (which is nice to send back to testers to have populate data without being involved in the script itseld).

Pat