Is testing from inside an application via embedded scripting a good idea?

Question

I'm working on a large GUI Program and even after years of development I still have not a single test case. I removed a lot of the needs by using Eiffel together with disciplined coding and Design By Contract.

But sometimes I feel that having unit tests might help me. But whenever I try to write some down I soon run into the problem with GUI testing (IMHO still an open challenge at the research side) and trying to isolate the code from the environment seems to be even more difficult.

Think of my work as writing very complex plugins for something like Eclipse.

So my new idea is to add a Lua scripting interface to the application and run the tests inside the program instead of separate unit tests. Or should I really try and spend a lot of refactoring hours (and mockup object writing) to make the application unit testing capable?

Answer 1

Scripting is a pretty bad choice for testing, as:

1. it won't catch actual GUI or GUI-to-model-code integration bugs
2. it will have many, many script-to-code integration bugs
3. it is a very slow and inefficient way of testing for model-code bugs
4. It takes more work to set up and maintain than any conceivable unit test framework

If you happen to have scripting in your product for some other reason, then testing it is obviously necessary. But it is pretty much never going to be a win to add it purely for testing.

Unit tests (or unit-style integration tests) are probably the most obvious alternative. Others are:

• GUI automation, by a product or tool, based on playback and record.
• GUI scripting, which is kind of like application scripting, but its the GUI model not the application that does the work to support scripts, and hopefully your toolkit of choice has already done this.
• GUI-based test coding, which is like GUI scripting, but in your main development language: you write code which fills in text fields, presses buttons, etc.

Which you pick should be based mostly on the kind and quantity of bugs you can live with your customers hitting.

Answer 2

It can be a good idea, as part of a balanced approach to testing. The disadvantages that soru notes are valid, but these may or may not be a big deal, depending on the project.

• Execution Speed: GUI automation using an external tool is typically very slow. An embedded scripting engine, while not as fast as compile-time unit tests, will be much faster than GUI tests.
• Not reinventing the wheel: When creating GUI test frameworks, you often have to replicate application logic in the test code. With a scripting engine, you can simply call the code directly.
• Headless Execution: With GUI test automation, the agent you execute the test on will require a GUI desktop, which has certain limitations when attempting to farm out test jobs to other machines (need to be logged on interactively, screen not locked, etc...) When coding to the API with a scripting engine, this is not an issue.

That having been said, one additional drawback to automated testing via an embedded scripting engine, is that it can be cumbersome to author the tests. The test author needs to know the application model, and the application model needs to support the scripting. Classes and methods need to be exposed to the scripting engine, and this support may be spotty, depending on the project, or part of the project you are trying to test. In my experience, this leads to digging through the source code to figure out what classes / methods need to be invoked in order to write the test (although, as a tester, digging through the source code ain't such a bad idea).

Answer 3

In my opinion, any approach to testing that will:

• Give you greater confidence in your code, and
• Improve the quality of the product

... is worthwhile doing.

Unit testing is only a means to an end.

If your intuition tells you writing tests in embedded Lua will give you the most bang for the buck, I say go for it.