Creating Non-Destructive Software

Question

From here I was shocked to learn the massive destructive capabilties of untested software (Read Ariane5 integer overflow and Therac -25).
They say that not all software logical conditions can be tested in a lab environment, I don't know how true it is?
What can I do to thoroughly test my software in the lab before it leaves for the field?
What kind of tools I could use and how would they help be to write Non-destructive software to the best of its abilities?

Answer 1

though you can never cover all of the possible faults and errors...

but if you use unit testing (in java JUnit) well enough, the chance of getting clean software that works as desired grows a lot.

do useful tests in your unit testing and try to think what else could happen to your code, to the execution path the machine may take (even if not expected) and fix the code to let the machine prevent such outcomes.

for sure, in java for example, there might be errors you cannot fix yourself without modifying the JVM. these are cases you cannot cover easily.

you're always limited to the framework you're using, right?

regards

Answer 2

Testing can only prove the presence of bugs, not their absence.

If you're writing software for mission-critical applications (i.e. where lives are at stake), there really is no alternative to using formal methods to hammer down the behaviour of your software exactly, and extensive peer review procedures to verify the specifications and proofs are correct.

There exist theorem provers and automatic program derivation tools you can use, but on a fundamental level, this kind of software development is very labour-intensive work.

Answer 3

If you really want to know if your software is safe then use a tool such as what LDRA have to offer. www.ldra.com

Answer 4

Testing the code alone wont do the trick. You need to test the specifications as well. You can easily write perfectly tested code, which follows an erroneous written specification.

Using live-data from the beginning (in simulation env.) helps -at least- to soften the heavy wake-up call when actually going "live" with your system.

Must-use tools might also be: Code Coverage Analysis Tools (like EclEmma)

The aspect of Pair Programming cannot be emphasized enough. Its so hard to get used to it, but helps to eliminate so many little mistakes right where they occurr.

Answer 5

This is an older stackoverflow posting with quite a lot of fan-out about techniques for safety-critical software. It has links to documentation about various formal methods for correctness proving, tools for static code analysis and other literature on the subject.

Answer 6

I would consider using a strongly typed language like ML or Ada if you can afford a language switch of course. A functional language would be a better fit too for everything related to testing. Thirdly, support of units of measure is something I guess could remove a lot of errors too. F# does support it directly, see here: http://blogs.msdn.com/andrewkennedy/archive/2008/08/29/units-of-measure-in-f-part-one-introducing-units.aspx

Answer 7

Agreing with other answers.

Also, you can use Jester http://jester.sourceforge.net/ to improve your testing.

Answer 8

To create good software you should design tests just like you design your software itself: with what the spec says. Don’t only create test at the end to prove your program is working, create them at the beginning / in the middle.

Tests should be written with destruction in mind. So don’t switch back and forth. Having a full day only writing destructive tests to find those bugs can be good. (Always depends on how easy it is to switch context for you; constructive to destructive and the other way around.) Having another coder do the tests can also be a vast improvements, as he may think about things you wouldn’t.

Tests should be Unit tests. Automated, fast running tests testing single components and functions. Use mocking to separate your components for testing. Of course a good OO approach is needed for you to be able to really test. Doing these tests also forces you do make a good software design; which is a good thing as well.

Then use a code coverage tool to check that you did test all that needs to be tested. (Often GUI packages do not need to be tested, but all the logic extensively ofc.) Be sure to check the coverage tool for it’s quality though. Is it C0, C1, C2?

In the end, test your application against your specification. Don’t just run it and check if it works. The spec is the important thing. Make a checklist.

As for tools, for Java: JUnit for unit testing EasyMock for mocking in your unit tests Cobertura is a free coverage tool you may want to check.

And you’ll definitely need an UML tool to design your software. Eclipse Modeling Framework, ArgoUML, or the commercial excellent Visual Paradigm.