python: attributes on a generator object

Question

Is it possible to create an attribute on a generator object?

Here's a very simple example:

def filter(x):
    for line in myContent:
        if line == x:
            yield x

Now say I have a lot of these filter generator objects floating around... maybe some of them are anonymous... I want to go back later and interrogate them for what they are filtering for. Is there a way I can a) interrogate the generator object for the value of x or b) set an attribute with the value of x that I can later interrogate?

Thanks

Answer 1

Yes.

class Filter( object ):
    def __init__( self, content ):
        self.content = content
    def __call__( self, someParam ):
        self.someParam = someParam
        for line in self.content:
            if line == someParam:
                yield line

Answer 2

Unfortunately, generator objects (the results returned from calling a generator function) do not support adding arbitrary attributes. You can work around it to some extent by using an external dict indexed by the generator objects, since such objects are usable as keys into a dict. So where you'd like to do, say:

a = filter(23)
b = filter(45)
...
a.foo = 67
...
x = random.choice([a,b])
if hasattr(x, 'foo'): munge(x.foo)

you may instead do:

foos = dict()
a = filter(23)
b = filter(45)
...
foos[a] = 67
...
x = random.choice([a,b])
if x in foos: munge(foos[x])

For anything fancier, use a class instead of a generator (one or more of the class's methods can be generators, after all).

Answer 3

No. You can't set arbitrary attributes on generators.

As S. Lott points out, you can have a object that looks like a generator, and acts like a generator. And if it looks like a duck, and acts like a duck, you've got yourself the very definition of duck typing, right there.

It won't support generator attributes like gi_frame without the appropriate proxy methods, however.

Answer 4

Thinking about the problem, there is a way of having generators carry around a set of attributes. It's a little crazy--I'd strongly recommend Alex Martelli's suggestion instead of this--but it might be useful in some situations.

my_content = ['cat', 'dog days', 'catfish', 'dog', 'catalog']

def filter(x):
    _query = 'I\'m looking for %r' % x

    def _filter():
        query = yield None
        for line in my_content:
            while query:
                query = yield _query

            if line.startswith(x):
                query = yield line

        while query:
            query = yield _query

    _f = _filter()
    _f.next()
    return _f

for d in filter('dog'):
    print 'Found %s' % d

cats = filter('cat')
for c in cats:
    looking = cats.send(True)
    print 'Found %s (filter %r)' % (c, looking)

If you want to ask the generator what it's filtering on, just call send with a value that evaluates to true. Of course, this code is probably too clever by half. Use with caution.

Answer 5

If you want to interrogate them for debugging purposes, then the following function will help:

import inspect

def inspect_generator(g):
    sourcecode = open(g.gi_code.co_filename).readlines()
    gline = g.gi_code.co_firstlineno
    generator_code = inspect.getblock(sourcecode[gline-1:])

    output = "Generator %r from %r\n" % (g.gi_code.co_name, g.gi_code.co_filename)
    output += "".join("%4s: %s" % (idx+gline, line) for idx, line in enumerate(generator_code))

    output += "Local variables:\n"
    output += "".join("%s = %r\n" % (key,value) for key,value in g.gi_frame.f_locals.items())

    return output

print inspect_generator(filter(6))
"""Output:
Generator 'filter' from 'generator_introspection.py'
   1: def filter(x):
   2:     for line in myContent:
   3:         if line == x:
   4:             yield x
Local variables:
x = 6
"""

If you want to interrogate them to implement functionality then classes implementing the iterator protocol are probably a better idea.

Answer 6

I just wrote a decorator to do this here: http://code.activestate.com/recipes/577057-generator-attributes/