What do you need from a test harness?

Question

I'm one of the people involved in the Test Anything Protocol (TAP) IETF group (if interested, feel free to join the mailing list). Many programming languages are starting to adopt TAP as their primary testing protocol and they want more from it than what we currently offer. As a result, we'd like to get feedback from people who have a background in xUnit, TestNG or any other testing framework/methodology.

Basically, aside from a simple pass/fail, what information do you need from a test harness? Just to give you some examples:

• Filename and line number (if applicable)
• Start and end time
• Diagnostic output such as the difference between what you got and what you expected.

And so on ...

Answer 1

It must be very, very easy to write a test, and equally easy to run them. That, to me, is the single most important feature of a testing harness. If someone has to fire up a GUI or jump through a bunch of hoops to write a test, they won't use it.

Answer 2

To what you said I'd add:

• Method/function/class name
• Coverage counting tool, with exceptions (Do not count these methods)
• Result of N last runs available
• Mandate that ways to easily parse test results must exist

Answer 3

Most definitely all things from your list for each individual item:

• Filename
• Line number
• Namespace/class/function name
• Test coverage
• Start time and end time
• And/or total time (this would be more useful for me than the top two items)
• Diagnostic output such as the difference between what you got and what you expected.

From the top of my head not much else but for the group of tests I would like to know

• group name
• total execution time

Answer 4

Any sort of diagnostic output - especially on failure is critical. If a test fails, you don't want to always have to rerun the test under a debugger to see what happened - there should be some cludes in the output.

I also like to see a before and after snapshot of critical system variables like memory or hard disk space available as those can provide great clues as well.

Finally, if you're using random seeds for any of the tests, write the seed out to the logfile so that the test can be reproduced if necessary.

Answer 5

An arbitrary set of tags - so I can mark a test as, for example "integration, UI, admin".

(you knew I was going to ask for this didn't you :-)

Answer 6

I'd like the ability to concatenate and nest TAP streams.

Answer 7

I use TAP as output protocol for a set of simple C++ test methods, and have seen the following shortcomings:

• test steps cannot be put into groups (there's only the grouping into several test scripts; but for running all tests in our software, I need at least one more level of grouping, so that a single test step would be identified by like "DB connection" -> "Reconnection Test" -> "test step #3")
• seeing differences between expected and actual output is useful; I either print the diff to stderr (as comment) or actually launch a graphical diff tool
• the protocol and tools must be really language-independent. For example, so far I only know of the Perl "prove" tool for running tests, which is limited to running Perl scripts

In the end, the test output must be suitable as basis for easily generating an HTML report file which lists succeeded tests very concisely, gives detailed output for failed tests, and makes it possible to quickly jump into the IDE to the failing test line.

Answer 8

A unique id (uuid, md5sum) to be able to identify an individual test -- say, for use when inserting test results in a database, or identifying them in a bug tracker to make it possible for QA to rerun an individual test.

This would also make it possible to trace an individual test's behavior from build-to-build through the entire lifecycle of multiple revisions of a product. This could eventually allow larger-scale correlations between 'historic' events (new hire, product release, hardware upgrades) and the profile(s) of tests that fail as a result of such events.

I'm also thinking that TAP should be emitted through a dedicated side-channel rather than mixed in with stdout. I'm not sure this is under the scope of the protocol definition.

Answer 9

• optional ascii coloured output, green for good, yellow for pending, red for errors

• the idea of things being pending

• a summary at the end of the test report of commands that will run the individual tests where

• List item

  • something went wrong

  • something in the test was pending