automated system testing approaches for combination of PC + embedded SW

Question

The system consists of a hardware device and a PC. The PC runs a GUI that controls the hardware device. To test the device various test hardware is used that can be controlled by RS232 or GPIB. To automate this testing the test software would have to control the test hardware and interact with the PC GUI.

What is the best way you know of to automate the testing of this system?

There are lots of ways to test a system like this. The tests can be controlled by COTS test software or by a custom test software built in house. Possibly a combination of COTS packages would be the best approach - one that specializes in GUI testing and one for the device. I've seen Labview applied to testing devices - not sure if it can do GUIs too. Maybe leaving off the GUI layer of the Windows app and exposing an automation layer is the best approach. What do you think? What have you tried?

Voting up the better techniques should be very interesting.

"the best way" would have to take into account things like: most effective, maintainability, implementation time, cost, etc.

Answer 1

Create custom test software package with a xUnit framework.

Answer 2

In situations like this with multiple seperate components it's almost certainly best to test each component in isolation first, and then test the integrated whole.

So create frameworks and tests for testing:
The Windows GUI
The automation layer.
The hardware device.

While automation can be achieved for most components with an unlimited budget, and an unlimited amount of time, it's often more worthwhile to test the key components, and use people to test the rest.

Users are obviously capable of breaking anything, so involve them once you've ironed out the majority of the issues. This will also mean you get valuable feedback.

Finally... only use this approach if having bugs in the hardware/software is acceptable.
I wouldn't recommend this approach if you're writing aeroplane control software for example...