Reporting unit test results.

Question

I am writing a paper for a scientific conference. Just to be thorough, in that paper I said I made unit tests for the components in the system among all the other tests (system testing, usability etc) I made.

In the results section, for the unit tests I simply said that all the tests passed. I submitted the draft to my adviser and now he's telling me that I need to give a more detailed summary of the unit tests I made.

I have never heard of writing a result summary for unit testing. Is there such a thing?

Answer 1

Here are some additional metrics you could consider reporting:

• The number of unit tests
• Code coverage

In addition to that, you may choose to describe which unit testing framework you used, and whether you used a particular methodology (Test-Driven Development or Test-After).

Answer 2

A detailed summary of the unit tests could include a measure of code coverage perhaps, or a chart showing where you have achieved code coverage. But even this is not particularly valuable. This is part of the problem with unit tests - it's easy to add a unit test to add additional code coverage without really adding to the confidence you have in your code.

If you have some specific tests that, for example, test that known input/output combinations worked, or that test the handling of certain edge or exception cases, then you can document those. If you used test-driven development, where the test comes from the problem and its analysis rather than from the code in a "retrofitted" manner, then say that as well.

By all means put some simple metrics in there, but you always have to take these with a pinch of salt...

Answer 3

On most runners, the minimal output will also show the number of tests run (how can you tell it found any tests?).

Other features one might see are:

• how many tests were skipped (on frameworks that support it)
• code coverage
• execution time (how long did the tests take)
• the actual names of the unit tests or their descriptions.

Answer 4

If the unit tests are unit tests a la jUnit, then could tweak your framework to display the name of the tests (names of TestCase methods) and the assertion messages, if any to generate a raw description of the unit tests. I had to do that once to satisfy the request of a QA guy.

Perhaps he believed the unit tests were executed manually, and just asked details about them, does the paper make it clear the unit tests are automatic ?