Python Generator - what not to use it for

Question

Just looking at Python generators, real impressed with them, but are there any things not to use them for? I was thinking of past C coding where reading from a file, or user actions would be areas. For example, could the generator be used to prompt the user for input (base data entry?) and the calling function process that input? are there any performance or cleanup issues to be concerned with?

Answer 1

Generators don't persist well.

Generally, you get an error trying to persist a generator object.

>>> def generatorForEvenKeys( aDictionary ):
    for k in aDictionary:
        if k % 2 == 0: yield aDictionary[k]

>>> x = generatorForEvenKeys( someDictionary )
>>> pickle.dump(x,file('temp.dat','wb'))

Gets you the following error:

TypeError: can't pickle generator objects

Answer 2

One problem with generators is that they get "consumed." This means that if you need to iterate over the sequence again, you need to create the generator again.

If lazy evaluation is an issue, then you probably don't want a generator expression. For example, if you want to perform all your calculations up front (e.g. so that you can release a resource), then a list comprehension or for loop is probably best.

If you use psyco, you'll get a significant speed increase for list expressions and for loops, but not for generators.

Also rather obviously, if you need to get the length of your sequence up front, then you don't want a generator.

Answer 3

You use a generator when you want to have something be iterateable, without holding the entire list in memory (this is why xrange supports much longer sequences than range in Python 2.x and lower)

When you need to load the whole "list of stuff to yield" into memory, there's not much point in using a generator - you may as well just return a list.

For a (slightly contrived) example:

def my_pointless_generator(x):
    thedata = range(x) # or thedata = list(range(x)) in Python 3.x
    for x in thedata:
        yield x

..can be rewritten just as efficiently as..

def my_pointless_generator(x):
    return range(x)