Testing a Gui-heavy WPF application.

Question

We (my colleagues) have a messy 12 y.o. mature app that is GUI-based, and the current plan is to add new dialogs & other GUI in WPF, as well as replace some of the older dialogs in WPF as well. At the same time we wish to be able to test that Monster - GUI automation in a maintainable way. Some challenges:

1. The application is massive.
2. It constantly gains new features.
3. It is being changed around (bug fixes, patches).
4. It has a back end, and a layer in-between. The state of it can get out of whack if you beat it to death.

What we want is:

• Some tool that can automate testing of WPF.
• auto-discovery of what the inputs and the outputs of the dialog are. An old test should still work if you add a label that does nothing. It should fail, however, if you remove a necessary text field. It would be very nice if the test suite was easy to maintain, if it ran and did not break most of the time.
• Every new dialog should be created with testability in mind.

At this point I do not know exactly what I want, so I am marking this as a community wiki. If having to test a huge GUI-based app rings the bell (even if not in WPF), then please share your good, bad and ugly experiences here.

Answer 1

One of the main strengths of WPF is actually the ability to NOT need UI specific testing, in my opinion. Using a M-V-VM approach would allow you to take the logic out of the UI/messy-GUI-area. Having a unit testable ViewModel (especially if you are writing new dialogs!) lets you write unit tests that emulate the clicking of your GUI.

I'd really reconsider what you want to accomplish by moving to WPF and what you want to achieve with some type of automated testing of a WPF GUI. Once this has been established, look into transitioning from WinForms to WPF if it still fits your needs.

Answer 2

As Jimmy Lyke says, most of your testing should be focused on the ViewModel. This consists of writing unit tests for the ViewModel - basically sending commands and setting and getting properties.

Once that is done, 95% of your testing is out of the way. If you want to take it a step further and test the view beyond the manual "walkthroughs" testing you would do anyway, there are a number of simple "sanity checks" you can easily automate to make sure you didn't accidentally delete an important TextBox or render a visual indicator invisible.

Each of the following techniques can be automated using some simple automation code that uses a "shotgun" approach by blindly running the visual tree and assuming nothing about the actual UI structure.

• To verify that all the ViewModel data is bound, find all Visuals and Freezables (using the visual tree) and check each bound property for its BindingExpression's binding path.

• To verify that all the ViewModel data is displayed somehow, vary the ViewModel data using a script and after each change uses RenderTargetBitmap to capture the UI and compare it with before the data change to make sure the UI has changed.

• To verify that property values are being updated correctly, find all Visuals and Freezables, and scans and records all bound properties on them, then make the ViewModel change, rescan, and searche for the expected change to any property of the given type. (To double-check you can then use the bitmap comparison technique on the particular Visual affected.)

• To verify that all commands are accessible, set a known ViewModel state then fire every command bound to a button that is visible to see if any of them trigger the ICommand or otherwise update ViewModel state.

• To verify that a ViewModel property is actually editable by the user, change the content or selection of every visible TextBox, ComboBox, ListBox to see if any of them affect the property.

• To get an opportunity to check any change that affects the UI, keep a database containing bitmap snapshots of each views in various ViewModel states in a set of different window sizes. When a new version of the app is built, run the same snapshot system and compare with the previous bitmaps. If anything at all has changed, generate a manual task for QA personnel to visually compare the old and new bitmaps to see if anything important has changed.

Other test automation is possible on the view, but the above will give you a start.

Again I must point out that it is best to focus on thoroughly testing the ViewModel. Bugs in the view itself are quite rare, typically detected quickly, and usually trivial to fix. But once ViewModel testing is thorough, it makes sense to do some automation of the view testing.

Answer 3

You have a very large application. I'm guessing it has lots of logic wrapped up with the presentation layer and you'll never be given the time to refactor the beast to separate the view from the rest of the logic.

You don't have a lot of choices here, but:

1. Refactor the code as you go along. This may be little method extractions so you can unit test or shifting to a proper model.
2. Use one or more of the variety of Windows GUI testing tools. Note, if you are planning a lot of layout and/or control changes, delay this as long as possible. The tools in this article will use absolute positioning of actions, control linked (sometimes by internal ids) or a mixture of both. Since they usually have to be trained without using code (aimed at QC testers, not programmers), your tests will stop running after the change.
3. Invest in human testers. While not a good choice, it does improve ending quality and starts to get management thinking more about the costs of refactoring the application.

Answer 4

OK, your app sounds large! I can share my experiences around an application we engineered recently; it was a GUI talking web services to a server that in turn contacted multiple databases and other web services. The client base was around 15,000 users… Either way - this is a lot of work no matter how you approach it; the upside is it will help you not chew your nails off each time you make a release!

MVVM

In general I would also recommend the MVVM pattern and do as much testing as possible without the GUI. GUI testing is just plain hard! I like Josh Smith’s article on MSDN: "WPF Apps With The Model-View-ViewModel Design Pattern" (http://msdn.microsoft.com/en-us/magazine/dd419663.aspx)

Test Scripting

The trick with this app was that we had a lot to test, the guts of it were constantly moving and there were (strangely enough) not enough people to get the testing job done for each iteration.

My solution was to come up with a custom testing tool that leveraged existing libraries. We had a simple script engine that read a file and executed commands. In effect we developed a DSL (http://en.wikipedia.org/wiki/Domain-specific_language) for testing our specific application. The DSL included some simple commands to signal what "window" it was testing, any specific "setup" signals and then a series of commands followed by assertions. It looked something like this:

Test NewXyzItem
Setup Clear_items

Press Some_button
Enter_text_into Name Bobby Joe
(...)
Press Save

Assert_db_has_itemX SomeKey

The format of each line is

"command" "argument" [data]

The scripts go into groups of directories and the "test runner" loads them up, parses them and executes them. Creating logs and reports as you go is useful, I got added in hook for making screen-shots etc which came in handy. If you are interested in implementing something likke this and would like a hand let me know.

The handy thing here was that we could make blanket changes to the test strategy.

Writing the scripts becomes pretty simple which is important because you end up with many, many scripts. The controls are discovered by name so you follow a convention (e.g. “Name” may be "NameTextBox" in code, or “Save” could be "SaveButton").

You can actually harness NUnit etc to be your test runner too.

NOTE - Just run the tests interactively, getting GUI test to work with CI is difficult and problematic...

Data and Testing

One major thing here is that the data management was a huge part of the test problem and cannot be overlooked. Our “fresh deployment” was also very long but some parts were external and we had no control over the freshness of the data. The way we handled the cleaning was to provide hooks through the scripting that allowed us to easily remove objects before tests. Not optimal but was rarely an issue.

Libraries

The library that you may find most useful in "White" (http://white.codeplex.com/) It can test windows apps in general – i.e both WPF and WinForms. Essentially you end up coding things like this:

Button button = window.Get<Button>("SaveButton");
button.Click();

If your app makes async calls you will need to come up with a strategy for the test runner to know when the async call is finished, perhaps though the status bar or something. It depend how you hook in…

Again, a lot of work but it’s worth it.

PK :-)

Answer 5

To test WPF apps there are a few things we have had success with:

1. PowerShell
2. TestPlant

and possibly would be the new VSTS 2010 features, though we haven't tried them