Testing Code for Embedded Application

Question

Background:

I am developing a largish project using at Atmel AVR atmega2560. This project contains a lot of hardware based functions (7 SPI devices, 2 I2C, 2 RS485 MODBUS ports, lots of Analogue and Digital I/O). I have developed "drivers" for all of these devices which provide the main application loop with an interface to access the required data.

Question:

The project I am developing will eventually have to meet SIL standards.

I would like to be able to test the code and provide a good level of code coverage. However I am unable to find any information to get me started on how such a testing framework should be set up.

The idea is that I can have a suite of automated tests which will allow future bug fixes and feature additions to be tested to see if they break the code. The thing is I don't understand how the code can be tested on chip.

Do I require hardware to monitor the I/O on the device and emulate externally connected devices? Any pointers that could be provided would be highly appreciated.

--Steve

Answer 1

Never run unit tests on or against the real hardware. Always mock your I/O interfaces. Otherwise, you can't simulate error conditions and, more importantly, you can't rely on the test to succeed.

So what you need is to split your app into various pieces that you can test independently. Simulator (or mock) all hardware that you need for those tests and run them on your development PC.

That should cover most of your code and leaves you with the drivers. Try to make as much of the driver code as possible work without the hardware. For the rest, you'll have to figure a way to make the code run on the hardware. This usually means you must create a test bed with external devices which respond to signals, etc. Since this is brittle (as in "your tests can't make this work automatically"), you must run these tests manually after preparing the hardware.

Answer 2

Vectorcast is a commercial tool to run unit tests on the hardware with code coverage.

Answer 3

This is a very good question - a common concern for embedded developers. Unfortunately, most embedded developers aren't as concerned as you are and only test the code on real hardware. But as another answer pointed out, this can basically test just the nominal functionality of the code and not the corner/error cases.

There is no single and simple solution to this problem. Some guidelines and techniques exist, however, to do a relatively good job.

First, separate your code into layers. One layer should be "hardware agnostic" - i.e. function calls. Do not ask the user to write into HW registers directly. The other (lower) layer deals with the HW. This layer can be "mocked" in order to test the higher level. The lower level can not be really tested without the HW, but it's not going to change often and needs deep HW integration, so it's not an issue.

A "test harness" will be all your high-level HW agnostic code with a "fake" lower level specifically for testing. This can simulate the HW devices for correct and incorrect functionality and thus allow you to run automated tests on the PC.

Answer 4

Do you have a JTAG connector? You may be able to use JTAG to simulate error conditions on the chip.

Answer 5

I like to separate the tasks. For instance, when I made a circular buffer for my Atmel AVR I wrote it all in Code::Blocks and compiled it with the regular GCC compiler instead of the AVR GCC compiler, then I create a unit test for it. I used a special header file to provide the proper data types that I wanted to work with (uint8_t for example). I found errors with the unit tests, fixed them, then took the fixed code over to AVR Studio and integrated it. After that I used wrote support functions and ISRs to fit the buffer into useful code (ie, pop one byte off the buffer, push it into the UART data output register, append a string constant to the buffer for a printf function, etc). Then I used the AVR simulator to make sure that my ISRs and functions were being called and that the right data showed up in registers. After that I programmed it onto the chip and it worked perfectly.

I greatly prefer the debugging capabilities of Code::Blocks compared to AVR Studio so I use the above approach whenever I can. When I can't I'm usually dealing with hardware only. For instance I have a timer that automatically produces a square wave. The best I could do was see that the pin bit was being twiddled in the simulator. After that I just had to hook a scope up and make sure.

I like to use a multi-level approach when debugging problems. For instance with the clock the first layer is 'Put a probe on the clock pin and see if there's a signal there'. If not, probe out the pin on the uC and look for the signal. Then, I coded a debug interface in one of my UARTs where I can look at specific register values and make sure they are what they're supposed to be. So if it doesn't work the next step is 'call up the register value and ensure it's correct.'

Try to think ahead four steps or so whenever you're planning your debugging. There should be +5V here, but what if there isn't? Write into the debug interface a way to toggle the pin and see if that changes it. What if that doesn't work? Do something else, etc etc etc. You get to a point where you run into 'I HAVE NO IDEA WHY THIS DANG THING DOESN'T WORK!!!!' but hopefully you'll figure out the reason beforehand.