What are the best practices regarding unit testing a library/framework ?

Question

Context

I'm struggling to write a set of unit-tests for a library/framework I'm designing. For context, please, think of my library as an object layer over a hierarchical set of related objects.

Question

Basically, I'm trying to adhere to the principles and best practices regarding unit testing as seen in some posts here, but they seem conflicting with regards to specifically unit testing a library or framework.

For instance, a basic test concerns itself with "creating an artifact". And another one with "removing an artifact". But, since a unit test is supposed to be standalone and restore the state of the world after its completion, both those tests seem somewhat related anyway : when testing the artifact creation, we need to cleanup the state at the end of the test by actually removing it. This means that artifact removal is itself implicitly tested. And the same reasoning applies to testing artifact removal : in order to setup the world so that artifact removal is testable, we need to create a new artifact first.

The situation is compounded when we need to unit test the creation and removal of related sub-artifacts, for which we will need to setup the world accordingly.

What I'm leaning towards is perform a set of related unit tests in sequence, so that each unit test is discrete (i.e. only tests one thing and one thing only), but depends on previous tests in the sequence to progressively setup the world. Then, my sequence could look like this:

[create artifact]->[create sub artifact]->[remove sub artifact]->[remove artifact]

With this principle, the entire library/framework is unit-tested, and the state of the world is restored at the end of the entire run of the test suite. However, this means that any failure in the middle of the test suite "breaks the world".

What are some best practices and guidelines that may be useful to reconcile those conflicting needs?

References

What makes a good unit test?

Is it bad form to have unit test calling other unit tests?

Answer 1

Don't know what language you are programming with. In Java, you can create TestSuites with the JUnit API. You add single sub-tests to the TestSuite and run the TestSuite as a whole.

Answer 2

By "restore the state of the world" you mean clean up a database perhaps, or something similar?

In which case you probably already have some unit tests which mock out your persistence layer. This would let you run unit tests in isolation, which would not depend on preserving state between tests.

So that leaves you with your other tests, which sound a bit more like "black box" tests, or integration tests, or whatever you want to call them. They rely on external state which has to be set up, monitored, torn down etc.

You should definitely expect them to be more brittle than unit tests.

My personal opinion is that once you get to these tests... it really depends. What you're suggesting doesn't sound unreasonable.

I tend to build a strong suite of isolated unit tests, and rely on the tests you've described for final "glue testing". So they're not too detailed and don't try and exercise every facet of a library.

Answer 3

Good unit tests are independent. The order in which the tests are run (or even if other tests are run at all) should not matter.

Set up your fixture for each test, then tear it down (restore the state) afterwards. Most of the time you won't need a teardown method (implementation depends on the test framework) if you keep your variables local to the test method.

If your library keeps track of state, you may have other issues (such as thread safety).

Answer 4

Here's some best practices that will help:

• Make tests independent, so they can be run alone, and in any sequence.
• Start every test in a known, clean state.
• Build up the environment using fixtures. Again look at db:unit. I like to have a good "canonical" state of the database, and restore it before each test (or roll back).

I disagree with one assumption you have: that a test must RESTORE its state before it finishes. You can also say it's the responsibility of the test runner to do this; tools like db:unit do this. They allow you to write your tests more freely.

Your sequential test makes sense, but it becomes difficult in practice. (I've tried) As you mentioned, you don't get good reports of the failures. In addition, as the program grows beyond 4 tests, you'll need to figure out where in the sequence to insert a test so that the data is just right for it. It will quickly become too hard to reason about. (Think 100s or 1000s of tests). That's why test fixturs (or factories) make sense.

Kent Beck's Test Driven Development: By Example discusses this topic thoroughly.

Answer 5

Going by the strict definition of unit testing used by Glen Beck and co, it could well actually be a bad idea to create unit tests in your situation.

What you definitely and unarguably need for a library to be used by anyone outside shouting range of your workstation is a full set of user-level API system tests. Ones that exercise the complete system in the same way client code would, not piece-by-piece in isolation.

Once you have those, then depending on:

• execution time (influenced by your language and nature of what you are doing).
• internal hidden complexity
• size of the API
• number of simultaneous developers on the library
• build time

it could well be that you can set up a complete set of system tests that covers everything that needs covering and executes in seconds. You could supplement this by a small set of focused unit tests on the lowest-level and most complex 10% of the code, but full coverage of everything at unit level could well be wasted effort.

Note: in the case of a system that is a library, a unit test tool like Junit is perfectly well suited to creating such complete-system tests. So a lot of people would end up calling them unit tests. Which is fine, as long as you don't fall for the argument that giving them that name changes anything about the way they should work.

The qualities that make good unit tests not only do not apply, they may well be actively harmful.

Answer 6

Read the book "XUnit Test Patterns" but Gerard Meszaros (http://xunitpatterns.com/). It covers the subject well.