Unit Testing Embedded Software

Question

What best practices have you used in unit testing embedded software that are peculiar to embedded systems?

Answer 1

Embedded software may have come a long way in the last 10 years but we generally did the following:

• for algorithms that didn't depend on the target hardware, we simply had unit tests that were built and tested on a non-embedded platform.
• for stuff that did require the hardware, unit tests were conditionally compiled into the code to use whatever hardware was available. In our case, it was a serial port on the target pushing the results to another, more capable, machine where the tests were checked for correctness.
• Depending on the hardware, you could sometimes dummy up a "virtual" device on a non-embedded platform. This usually consisted of having another thread of execution (or signal function) changing memory used by the program. Useful for memory mapped I/O but not IRQs and such.
• typically, you could only unit test a small subset of the complete code at a time (due to memory constraints).
• for testing of time-sensitive things, we didn't. Plain and simple. The hardware we used (8051 and 68302) was not always functional if it ran too slow. That sort of debugging had to be done initially with a CRO (oscilloscope) and (when we had some more money) an ICE (in-circuit emulator).

Hopefully the situation has improved since I last did it. I wouldn't wish that pain on my worst enemy.

Answer 2

Lots of embedded processors are available on eval boards, so although you may not have your real i/o devices, often you can execute a good deal of your algorithms and logic on one of these kinds of things, often w/hardware debugging available via jtag. And 'unit' tests usually are more about your logic than your i/o anyway. Problem is usually getting your test artifacts back out of one of these environments.

Answer 3

Embedded systems is a wide topic but in general, let's think of it as a specific-purpose product that combines both hardware and software. My embedded background is from mobile phones which is just a small subset of all embedded systems. I'll try to keep the following points a bit on the abstract side:

• Abstract out hardware dependencies whenever possible. This way you can run your unit tests on mocked "hardware" and also test various rare/exceptional cases that would be harder to test on target. To prevent abstraction costs, you can use e.g. conditional compilation.

• Have as little as possible depend on the hardware.

• Unit tests running on an emulator or cross-compiler environment still does not guarantee the code works on target hardware. You must test on target as well. Test on target as early as possible.

Answer 4

Split the code between device-dependent & device-independent. The independent code can be unit-tested without too much pain. The dependent code will simply need to be hand-tested until you have a smooth communications interface.

If you're writing the communications interface, I'm sorry.

Answer 5

There can be a lot to be gained by unit testing in a PC environment (compiling your code with a PC C compiler and running your code in a PC unit testing framework), with several provisos:

1. This doesn't apply to testing your low-level code, including start-up code, RAM tests, hardware drivers. You'll have to use more direct unit testing of those.
2. Your embedded system's compiler has to be trustworthy, so you're not hunting for bugs created by the compiler.
3. Your code has to be layered architecture, with hardware abstraction. You may need to write hardware driver simulators for your PC unit testing framework.
4. You should always use the stdint.h types such as uint16_t rather than plain unsigned int etc.

We've followed these rules, and found that after unit testing the application-layer code in a PC unit test framework, we can have a good amount of confidence that it works well.

Advantages of unit testing on the PC platform:

1. You don't face the problem of running out of ROM space on your embedded platform due to adding a unit testing framework.
2. The compile-link-run cycle is typically faster and simpler on the PC platform (and avoids the 'write/download' step which can potentially be several minutes).
3. You have more options for visualising progress (some embedded applications have limited I/O peripherals), storing input/output data for analysis, running more time-consuming tests.
4. You can use readily available PC-based unit test frameworks that aren't available/suitable for an embedded platform.

Answer 6

Voice of inexperience here, but this is something I've been thinking about as well lately. It seems to me that the best approach would be either

A) Write as much of your hardware-independent application code as you can in a PC environment, before you write it on the target, and write your unit tests at the same time (doing it this on the PC first should help force you to separate the hardware-independent stuff). This way you can use your choice of unit testers, then test the hardware-dependent stuff the old fashioned way - with RS-232 and/or oscilloscopes and I/O pins signalling time-dependent data, depending on how fast it has to run.

B) Write it all on the target hardware, but have a make target to conditionally compile a unit test build that will run unit tests and output the results (or data that can be analyzed for results) via RS-232 or some other means. If you don't have a lot of memory, this can be tricky.

Edit 7/3/2009 I just had another thought about how to unit test hardware dependent stuff. If your hardware events are happening too fast to record with RS-232, but you don't want to manually sift through tons of oscilloscope data checking to see if your I/O pin flags rise and fall as expected, you can use a PC card with integrated DIO (such as National Instruments' line of Data Acquisition cards) to automatically evaluate the timing of those signals. You would then just need to write the software on your PC to control the data acquisition card to synchronize with the currently running unit test.

Answer 7

There's lots of good answers here, some things that haven't been mentioned is to have diagnostic code running in order to:

Log HAL events (interrupts, bus messages, etc)

Have code to keep track of your resources, (all active semaphores, thread activity)

Have a capture ram mechanism to copy the heap and memory content to persistent storage (hard disk or equivalent) to detect and debug deadlocks, livelocks, memory leaks, buffer overflows, etc.

Answer 8

You might want to check out Test Driven Development for Embedded C by James W. Grenning. The book is scheduled to be published in August 2010, but the beta book is available now on The Pragmatic Bookshelf.