Best way for testing compiled code to return expected output/errors

Question

How do you test if compiled code returns the expected output or fails as expected?

I have worked out a working example below, but it is not easily extendable. Every additional test would require additional nesting parentheses. Of course I could split this into other files, but do you have any suggestions on how to improve this?. Also I'm planning to use this from make test stanza in a makefile, so I do not expect other people to install something that isn't installed by default, just for testing it. And stdout should also remain interleaved with stderr.

simplified example:

./testFoo || echo execution failed

./testBar && echo expected failure

(./testBaz && (./testBaz 2>&1 | cmp -s - foo.tst && ( ./testFoo && echo and so on 
     || echo testFoo's execution failed )|| echo testBaz's does not match   )
  || echo testBaz's execution failed

my current tester looks like this (for one test):

\#!/bin/bash
compiler1 $1 && (compiler2 -E --make $(echo $1 | sed 's/^\(.\)\(.*\)\..*$/\l\1\2/') && (./$(echo $1 | sed 's/^\(.\)\(.*\)\..*$/\l\1\2/') || echo execution failed)  || less $(echo $1 | sed 's/^\(.\)\(.*\)\..*$/\l\1\2/').err) || echo compile failed

Answer 1

I suggest to start looking for patterns here. For example, you could use the file name as the pattern and then create some additional files that encode the expected result.

You can then use a simple script to run the command and verify the result (instead of repeating the test code again and again).

For example, a file testFoo.exec with the content 0 means that it must succeed (or at least return with 0) while testBar.exec would contain 1.

textBaz.out would then contain the expected output. You don't need to call testBaz several times; you can redirect the output in the first call and then look at $? to see if the call succeeded or not. If it did, then you can directly verify the output (without starting the command again).

Answer 2

My own simple minded test harness works like this:

• every test is represented by a bash script with an extension .test - these all live in the same directory

• when I create a test, I run the test script and examine the output carefully, if it looks good it goes into a directory called good_results, in a file with the same name as the test that generated it

• the main testing script finds all the .test scripts and executes each of them in turn, producing a temporary output file. This is diff'd with the matching file in the good_results directory and any differences reported

Itv took me about half an hour to write this and get it working, but it has proved invaluable!