How to accumulate state across tests in py.test

Question

I currently have a project and tests similar to these.

class mylib:
    @classmethod
    def get_a(cls):
        return 'a'

    @classmethod
    def convert_a_to_b(cls, a):
        return 'b'

    @classmethod
    def works_with(cls, a, b):
        return True

class TestMyStuff(object):
    def test_first(self):
        self.a = mylib.get_a()

    def test_conversion(self):
        self.b = mylib.convert_a_to_b(self.a)

    def test_a_works_with_b(self):
        assert mylib.works_with(self.a, self.b)

With py.test 0.9.2, these tests (or similar ones) pass. With later versions of py.test, test_conversion and test_a_works_with_b fail with 'TestMyStuff has no attribute a'.

I am guessing this is because with later builds of py.test, a separate instance of TestMyStuff is created for each method that is tested.

What is the proper way to write these tests such that results can be given for each of the steps in the sequence, but the state from a previous (successful) test can (must) be used to perform subsequent tests?

Answer 1

Good unit test practice is to avoid state accumulated across tests. Most unit test frameworks go to great lengths to prevent you from accumulating state. The reason is that you want each test to stand on its own. This lets you run arbitrary subsets of your tests, and ensures that your system is in a clean state for each test.

Answer 2

I partly agree with Ned in that it's good to avoid somewhat random sharing of test state. But i also think it is sometimes useful to accumulate state incrementally during tests.

With py.test you can actually do that by making it explicit that you want to share test state. Your example rewritten to work:

class State:
    """ holding (incremental) test state """

def pytest_funcarg__state(request):
    return request.cached_setup(
        setup=lambda: State(),
        scope="module"
    )

class mylib:
    @classmethod
    def get_a(cls):
        return 'a'

    @classmethod
    def convert_a_to_b(cls, a):
        return 'b'

    @classmethod
    def works_with(cls, a, b):
        return True

class TestMyStuff(object):
    def test_first(self, state):
        state.a = mylib.get_a()

    def test_conversion(self, state):
        state.b = mylib.convert_a_to_b(state.a)

    def test_a_works_with_b(self, state):
        mylib.works_with(state.a, state.b)

You can run this with recent py.test versions. Each functions receives a "state" object and the "funcarg" factory creates it initially and caches it over the module scope. Together with the py.test guarantee that tests are run in file order the test functions can be rather they will work incrementally on the test "state".

However, It is a bit fragile because if you select just the running of "test_conversion" via e.g. "py.test -k test_conversion" then your test will fail because the first test hasn't run. I think that some way to do incremental tests would be nice so maybe we can eventually find a totally robust solution.

HTH, holger

Answer 3

As I spent more time with this problem, I realized there was an implicit aspect to my question that I neglected to specify. In most scenarios, I found that I wanted to accumulate state within a single class, but discard it when the test class had completed.

What I ended up using for some of my classes, where the class itself represented a process that accumulated state, I stored the accumulated state in the class object itself.

class mylib:
    @classmethod
    def get_a(cls):
        return 'a'

    @classmethod
    def convert_a_to_b(cls, a):
        return 'b'

    @classmethod
    def works_with(cls, a, b):
        return True

class TestMyStuff(object):
    def test_first(self):
        self.__class__.a = mylib.get_a()

    def test_conversion(self):
        self.__class__.b = mylib.convert_a_to_b(self.a)

    def test_a_works_with_b(self):
        mylib.works_with(self.a, self.b)

The advantage to this approach is it keeps the state encapsulated within the test class (there are no auxiliary functions that have to be present for the test to run), and it would be suitably awkward for a different class to expect the TestMyStuff state to be present when the different class runs.

I think each of these approaches discussed thusfar have their merits, and intend to use each approach where it fits best.