Semi-automated testing of external libraries and error-prone interactions

Question

Recently I have been trying to use unit tests in my code, and I like the idea in principle. However, the parts of my code that I am most eager to test are those error-prone areas which unit tests alone don't handle very well; for example:

• Network code
• Filesystem interactions
• Database interactions
• Communication with hardware (e.g. specialized devices that talk over RS-232)
• Calls to quirky third-party libraries

I understand that mock objects are typically used in these situations, but I'm looking for a way to feel confident that the mock objects are correctly simulating the situations I want to test.

For example, suppose I want to write a mock that simulates what happens when the database server is restarted. To do this, I would want to first verify that the database library I'm using will actually throw a particular exception if the DB server is restarted. Right now, I write code like:

def checkDatabaseDropout():
    connectToDatabase()
    raw_input("Shut down the database and press Enter")
    try:
        testQuery()
        assert False, "Database should have thrown an exception"
    except DatabaseError, ex:
        pass

Running this requires a fair amount of manual intervention, but it at least it gives me a verifiable set of assumptions I can work with in my code, and it lets me check those assumptions when I upgrade the library, switch to a different underlying database, etc.

My question is: are there better ways of handling this? Are there frameworks that support this kind of semi-automated testing? Or do people generally use other techniques at this end of the testing spectrum?

Answer 1

I try to not foresee these kinds of things.

Even though I'm doing close to 100% TDD, at the end of the day, I'm still building a complete system, so I also test that the entire application runs as expected. Such System Tests can capture and reproduce the kind of scenarios you talk about.

Once I know how to reproduce a given scenario, I can always write a unit test that reproduces it.

So in other words, I currently tend to work with two configurations:

• Fully automated unit tests
• Manual system tests.

These can interact and feed each other, iteratively making each easier and better to work with.