Writing a re-usable unittest.TestCase method

Question

I'm writing tests using the unittest package, and I want to avoid repeated code. I am going to carry out a number of tests which all require a very similar method, but with only one value different each time. A simplistic and useless example would be:

class ExampleTestCase(unittest.TestCase):

    def test_1(self):
        self.assertEqual(self.somevalue, 1)

    def test_2(self):
        self.assertEqual(self.somevalue, 2)

    def test_3(self):
        self.assertEqual(self.somevalue, 3)

    def test_4(self):
        self.assertEqual(self.somevalue, 4)

Is there a way to write the above example without repeating all the code each time, but instead writing a generic method, e.g.

    def test_n(self, n):
        self.assertEqual(self.somevalue, n)

and telling unittest to try this test with different inputs?

Answer 1

Perhaps something like:

def test_many(self):
    for n in range(0,1000):
        self.assertEqual(self.somevalue, n)

Answer 2

I guess what you want is "parameterized tests".

I don't think unittest module supports this (unfortunately), but if I were adding this feature it would look something like this:

# Will run the test for all combinations of parameters
@RunTestWith(x=[0, 1, 2, 3], y=[-1, 0, 1])
def testMultiplication(self, x, y):
  self.assertEqual(multiplication.multiply(x, y), x*y)

With the existing unittest module, a simple decorator like this won't be able to "replicate" the test multiple times, but I think this is doable using a combination of a decorator and a metaclass (metaclass should observe all 'test*' methods and replicate (under different auto-generated names) those that have a decorator applied).

Answer 3

If you really want to have multiple unitttest then you need multiple methods. The only way to get that is through some sort of code generation. You can do that through a metaclasses, or by tweaking the class after the definition, including (if you are using Python 2.6) through a class decorator.

Here's a solution which looks for the special 'multitest' and 'multitest_values' members and uses those to build the test methods on the fly. Not elegant, but it does roughly what you want:

import unittest
import inspect

class SomeValue(object):
    def __eq__(self, other):
        return other in [1, 3, 4]

class ExampleTestCase(unittest.TestCase):
    somevalue = SomeValue()

    multitest_values = [1, 2, 3, 4]
    def multitest(self, n):
        self.assertEqual(self.somevalue, n)

    multitest_gt_values = "ABCDEF"
    def multitest_gt(self, c):
        self.assertTrue(c > "B", c)


def add_test_cases(cls):
    values = {}
    functions = {}
    # Find all the 'multitest*' functions and
    # matching list of test values.
    for key, value in inspect.getmembers(cls):
        if key.startswith("multitest"):
            if key.endswith("_values"):
                values[key[:-7]] = value
            else:
                functions[key] = value

    # Put them together to make a list of new test functions.
    # One test function for each value
    for key in functions:
        if key in values:
            function = functions[key]
            for i, value in enumerate(values[key]):
                def test_function(self, function=function, value=value):
                    function(self, value)
                name ="test%s_%d" % (key[9:], i+1)
                test_function.__name__ = name
                setattr(cls, name, test_function)

add_test_cases(ExampleTestCase)

if __name__ == "__main__":
    unittest.main()

This is the output from when I run it

% python stackoverflow.py
.F..FF....
======================================================================
FAIL: test_2 (__main__.ExampleTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "stackoverflow.py", line 34, in test_function
    function(self, value)
  File "stackoverflow.py", line 13, in multitest
    self.assertEqual(self.somevalue, n)
AssertionError: <__main__.SomeValue object at 0xd9870> != 2

======================================================================
FAIL: test_gt_1 (__main__.ExampleTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "stackoverflow.py", line 34, in test_function
    function(self, value)
  File "stackoverflow.py", line 17, in multitest_gt
    self.assertTrue(c > "B", c)
AssertionError: A

======================================================================
FAIL: test_gt_2 (__main__.ExampleTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "stackoverflow.py", line 34, in test_function
    function(self, value)
  File "stackoverflow.py", line 17, in multitest_gt
    self.assertTrue(c > "B", c)
AssertionError: B

----------------------------------------------------------------------
Ran 10 tests in 0.001s

FAILED (failures=3)

You can immediately see some of the problems which occur with code generation. Where does "test_gt_1" come from? I could change the name to the longer "test_multitest_gt_1" but then which test is 1? Better here would be to start from _0 instead of _1, and perhaps in your case you know the values can be used as a Python function name.

I do not like this approach. I've worked on code bases which autogenerated test methods (in one case using a metaclass) and found it was much harder to understand than it was useful. When a test failed it was hard to figure out the source of the failure case, and it was hard to stick in debugging code to probe the reason for the failure.

(Debugging failures in the example I wrote here isn't as hard as that specific metaclass approach I had to work with.)

Answer 4

Write a single test method that performs all of your tests and captures all of the results, write your own diagnostic messages to stderr, and fail the test if any of its subtests fail:

def test_with_multiple_parameters(self):
    failed = False
    for k in sorted(self.test_parameters.keys()):
        if not self.my_test(self.test_parameters[k]):
           print >> sys.stderr, "Test {0} failed.".format(k)
           failed = True
    self.assertFalse(failed)

Note that of course my_test()'s name can't begin with test.

Answer 5

A more data-oriented approach might be clearer than the one used in Andrew Dalke's answer:

"""Parametrized unit test.

Builds a single TestCase class which tests if its
  `somevalue` method is equal to the numbers 1 through 4.

This is accomplished by
  creating a list (`cases`)
  of dictionaries which contain test specifications
  and then feeding the list to a function which creates a test case class.

When run, the output shows that three of the four cases fail,
  as expected:

>>> import sys
>>> from unittest import TextTestRunner
>>> run_tests(TextTestRunner(stream=sys.stdout, verbosity=9))
... # doctest: +ELLIPSIS
Test if self.somevalue equals 4 ... FAIL
Test if self.somevalue equals 1 ... FAIL
Test if self.somevalue equals 3 ... FAIL
Test if self.somevalue equals 2 ... ok
<BLANKLINE>
======================================================================
FAIL: Test if self.somevalue equals 4
----------------------------------------------------------------------
Traceback (most recent call last):
  ...
AssertionError: 2 != 4
<BLANKLINE>
======================================================================
FAIL: Test if self.somevalue equals 1
----------------------------------------------------------------------
Traceback (most recent call last):
  ...
AssertionError: 2 != 1
<BLANKLINE>
======================================================================
FAIL: Test if self.somevalue equals 3
----------------------------------------------------------------------
Traceback (most recent call last):
  ...
AssertionError: 2 != 3
<BLANKLINE>
----------------------------------------------------------------------
Ran 4 tests in ...s
<BLANKLINE>
FAILED (failures=3)
"""

from unittest import TestCase, TestSuite, defaultTestLoader

cases = [{'name': "somevalue_equals_one",
          'doc': "Test if self.somevalue equals 1",
          'value': 1},
         {'name': "somevalue_equals_two",
          'doc': "Test if self.somevalue equals 2",
          'value': 2},
         {'name': "somevalue_equals_three",
          'doc': "Test if self.somevalue equals 3",
          'value': 3},
         {'name': "somevalue_equals_four",
          'doc': "Test if self.somevalue equals 4",
          'value': 4}]

class BaseTestCase(TestCase):
    def setUp(self):
        self.somevalue = 2

def test_n(self, n):
    self.assertEqual(self.somevalue, n)

def make_parametrized_testcase(class_name, base_classes, test_method, cases):
    def make_parametrized_test_method(name, value, doc=None):
        def method(self):
            return test_method(self, value)
        method.__name__ = "test_" + name
        method.__doc__ = doc
        return (method.__name__, method)

    test_methods = (make_parametrized_test_method(**case) for case in cases)
    class_dict = dict(test_methods)
    return type(class_name, base_classes, class_dict)


TestCase = make_parametrized_testcase('TestOneThroughFour',
                                      (BaseTestCase,),
                                      test_n,
                                      cases)

def make_test_suite():
    load = defaultTestLoader.loadTestsFromTestCase
    return TestSuite(load(TestCase))

def run_tests(runner):
    runner.run(make_test_suite())

if __name__ == '__main__':
    from unittest import TextTestRunner
    run_tests(TextTestRunner(verbosity=9))

I'm not sure what voodoo is involved in determining the order in which the tests are run, but the doctest passes consistently for me, at least.

For more complex situations it's possible to replace the values element of the cases dictionaries with a tuple containing a list of arguments and a dict of keyword arguments. Though at that point you're basically coding lisp in python.