How do I get nose to discover dynamically-generated testcases?

Question

This is a follow-up to a previous question of mine.

In the previous question, methods were explored to implement what was essentially the same test over an entire family of functions, ensuring testing did not stop at the first function that failed.

My preferred solution used a metaclass to dynamically insert the tests into a unittest.TestCase. Unfortunately, nose does not pick this up because nose statically scans for test cases.

How do I get nose to discover and run such a TestCase? Please refer here for an example of the TestCase in question.

Answer 1

You could try to generate the testcase classes with type()

class UnderTest_MixIn(object):

    def f1(self, i):
        return i + 1

    def f2(self, i):
        return i + 2

SomeDynamicTestcase = type(
    "SomeDynamicTestcase", 
    (UnderTest_MixIn, unittest.TestCase), 
    {"even_more_dynamic":"attributes .."}
)

# or even:

name = 'SomeDynamicTestcase'
globals()[name] = type(
    name, 
    (UnderTest_MixIn, unittest.TestCase), 
    {"even_more_dynamic":"attributes .."}
)

This should be created when nose tries to import your test_module so it should work.

The advantage of this approach is that you can create many combinations of tests dynamically.

Answer 2

Nose has a "test generator" feature for stuff like this. You write a generator function that yields each "test case" function you want it to run, along with its args. Following your previous example, this could check each of the functions in a separate test:

import unittest
import numpy

from somewhere import the_functions

def test_matrix_functions():
    for function in the_functions:
        yield check_matrix_function, function

def check_matrix_function(function)
    matrix1 = numpy.ones((5,10))
    matrix2 = numpy.identity(5)
    output = function(matrix1, matrix2)
    assert matrix1.shape == output.shape, \
           "%s produces output of the wrong shape" % str(function)