Reusing Generators in Different Unit Tests

Question

I'm running into an issue while unit-testing a Python project that I'm working on which uses generators. Simplified, the project/unit-test looks like this:

I have a setUp() function which creates a Person instance. Person is a class that has a generator, next_task(), which yields the next task that a Person has.

I now have two unit-tests that test different things about the way the generator works, using a for loop. The first test works exactly as I'd expect, and the second one never even enters the loop. In both unit tests, the first line of code is:

for rank, task in enumerate(self.person.next_task()):

My guess is that this isn't working because the same generator function is being used in two separate unit tests. But that doesn't seem like the way that generators or unit-tests are supposed to work. Shouldn't I be able to iterate twice across the list of tasks? Also, shouldn't each unit-test be working with an essentially different instance of the Person, since the Person instance is created in setUp()?

Answer 1

The yielded results of a generator are consumed by the first for loop that uses it. Afterwards, the generator function returns and is finished - and thus empty. As the second unit test uses the very same generator object, it doesn't enter the loop. You have to create a new generator for the second unit test, or use itertools.tee to make N separate iterators out of one generator.

Generators do not work the way you think. On each call to generatorObject.next(), the next yielded result is returned, but the result does not get stored anywhere. That's why generators are often used for lazy operations. If you want to reuse the results, you can use itertools.tee as I said, or convert the generator to a tuple/list of results.

A hint on using itertools.tee from the documentation:

This itertool may require significant auxiliary storage (depending on how much temporary data needs to be stored). In general, if one iterator uses most or all of the data before another iterator starts, it is faster to use list() instead of tee().

Answer 2

If you are really creating a new Person object in setUp then it should work as you expect. There are several reasons why it may not be working:

1) you are initialising the Person's tasks from another iterator, and that is exhausted by the second time you create Person.

2) You are creating a new Person object each time but the task generator is a class variable instead of an instance variable, so is shared between the class instances.

3) You think you are creating a new Person object but in reality you are not for some reason. Perhaps it is implemented as a singleton.

4) the unittest setUp method is broken.

Of these I think (4) is least likely, but we would need to see more of you code before we can track down the real problem.