With Python unittest, how do I create and use a "callable object that returns a test suite"?

Question

I'm learning Python and have been trying to understand more about the details of Python's unittest module. The documentation includes the following:

For the ease of running tests, as we will see later, it is a good idea to provide in each test module a callable object that returns a pre-built test suite:

def suite():
    suite = unittest.TestSuite()
    suite.addTest(WidgetTestCase('testDefaultSize'))
    suite.addTest(WidgetTestCase('testResize'))
    return suite

As far as I can tell, the purpose of doing this is not explained. In addition, I was unable to figure out how one would use such a method. I tried several things without success (aside from learning about the error messages I got):

import unittest

def average(values):
    return sum(values) / len(values)

class MyTestCase(unittest.TestCase):
    def testFoo(self):
        self.assertEqual(average([10,100]),55)

    def testBar(self):
        self.assertEqual(average([11]),11)

    def testBaz(self):
        self.assertEqual(average([20,20]),20)

    def suite():
        suite = unittest.TestSuite()
        suite.addTest(MyTestCase('testFoo'))
        suite.addTest(MyTestCase('testBar'))
        suite.addTest(MyTestCase('testBaz'))
        return suite

if __name__ == '__main__':
    # s = MyTestCase.suite()
    # TypeError: unbound method suite() must be called 
    # with MyTestCase instance as first argument

    # s = MyTestCase.suite(MyTestCase())
    # ValueError: no such test method in <class '__main__.MyTestCase'>: runTest

    # s = MyTestCase.suite(MyTestCase('testFoo'))
    # TypeError: suite() takes no arguments (1 given)

The following "worked" but seems awkward and it required that I change the method signature of suite() to 'def suite(self):'.

s = MyTestCase('testFoo').suite()
unittest.TextTestRunner().run(s)

Answer 1

The documentation is saying suite() should be a function in the module, rather than a method in your class. It appears to be merely a convenience function so you can later aggregate test suites for many modules:

alltests = unittest.TestSuite([
  my_module_1.suite(),
  my_module_2.suite(),
])

Your problems with calling your function are all related to it being a method in the class, yet not written as a method. The self parameter is the "current object", and is required for instance methods. (E.g. a.b(1, 2) is conceptually the same as b(a, 1, 2).) If the method operates on the class instead of instances, read about classmethod. If it is just grouped with the class for convenience but doesn't operate on the class nor instances, then read about staticmethod. Neither of these will particularly help you use unittest, but they might help explain why you saw what you did.

Answer 2

The very first error message you got is meaningful, and explains a lot.

print MyTestCase.suite # <unbound method MyTestCase.suite>

Unbound. It means that you cannot call it unless you bind it to an instance. It's actually the same for MyTestCase.run:

print MyTestCase.run # <unbound method MyTestCase.run>

Maybe for now you don't understand why you can't call suite, but please leave it aside for now. Would you try to call run on the class, like above? Something like:

MyTestCase.run() # ?

Probably not, right? It does not make sense to write this, and it will not work, because run is an instance method, and needs a self instance to work on. Well it appears that Python "understands" suite in the same way it understands run, as an unbound method.

Let's see why:

If you try to put the suite method out of the class scope, and define it as a global function, it just works:

import unittest

def average(values):
    return sum(values) / len(values)

class MyTestCase(unittest.TestCase):
    def testFoo(self):
        self.assertEqual(average([10,100]),55)

    def testBar(self):
        self.assertEqual(average([11]),11)

    def testBaz(self):
        self.assertEqual(average([20,20]),20)

def suite():
    suite = unittest.TestSuite()
    suite.addTest(MyTestCase('testFoo'))
    suite.addTest(MyTestCase('testBar'))
    suite.addTest(MyTestCase('testBaz'))
    return suite

print suite() # <unittest.TestSuite tests=[<__main__.MyTestCase testMethod=testFoo>, <__main__.MyTestCase testMethod=testBar>, <__main__.MyTestCase testMethod=testBaz>]>

But you don't want it out of the class scope, because you want to call MyTestCase.suite()

You probably thought that since suite was sort of "static", or instance-independent, it did not make sense to put a self argument, did you? It's right.

But if you define a method inside a Python class, Python will expect that method to have a self argument as a first argument. Just omitting the self argument does not make your method static automatically. When you want to define a "static" method, you have to use the staticmethod decorator:

@staticmethod
def suite():
    suite = unittest.TestSuite()
    suite.addTest(MyTestCase('testFoo'))
    suite.addTest(MyTestCase('testBar'))
    suite.addTest(MyTestCase('testBaz'))
    return suite

This way Python does not consider MyTestCase as an instance method but as a function:

print MyTestCase.suite # <function suite at 0x...>

And of course now you can call MyTestCase.suite(), and that will work as expected.

if __name__ == '__main__':
    s = MyTestCase.suite()
    unittest.TextTestRunner().run(s) # Ran 3 tests in 0.000s, OK

Answer 3

One thing that's important to note, nose is a tool that really simplifies running tests. It allows you to specify exactly which tests to run from the command-line as well as walking through each directory and running each test.

Answer 4

As already been mentioned, documentation refers to suite() as a method in the module, and unittest (strangely) doesn't seem to have any special recognition for this method, so you do have to call it explicitly. However, if you use a tool called "testoob", it automatically calls the suite() method (if you specify the defaultTest="suite" argument to its main()) and adds several other features on top of the base unittest package. It also provides options for generating XML files that includes collected stdout and stderr from these tests (which is big plus for automated tests) and can also generate HTML reports (though you would need to install additional packages). I couldn't find a way to automatically discover all the tests which nose claims to support, so nose is probably a better option.