Can I be warned when I used a generator function by accident

Question

I was working with generator functions and private functions of a class. I am wondering

1. Why when yielding (which in my one case was by accident) in __someFunc that this function just appears not to be called from within __someGenerator. Also what is the terminology I want to use when referring to these aspects of the language?
2. Can the python interpreter warn of such instances?

Below is an example snippet of my scenario.

class someClass():
    def __init__(self):
        pass

    #Copy and paste mistake where yield ended up in a regular function
    def __someFunc(self):
     print "hello"
     #yield True #if yielding in this function it isn't called

    def __someGenerator (self):
     for i in range(0, 10):
      self.__someFunc()
      yield True
     yield False

    def someMethod(self):
     func = self.__someGenerator()
     while func.next():
      print "next"

sc = someClass()
sc.someMethod()

I got burned on this and spent some time trying to figure out why a function just wasn't getting called. I finally discovered I was yielding in function I didn't want to in.

Answer 1

A "generator" isn't so much a language feature, as a name for functions that "yield." Yielding is pretty much always legal. There's not really any way for Python to know that you didn't "mean" to yield from some function.

This PEP http://www.python.org/dev/peps/pep-0255/ talks about generators, and may help you understand the background better.

I sympathize with your experience, but compilers can't figure out what you "meant for them to do", only what you actually told them to do.

Answer 2

Because the return keyword is applicable in both generator functions and regular functions, there's nothing you could possibly check (as @Christopher mentions). The return keyword in a generator indicates that a StopIteration exception should be raised.

If you try to return with a value from within a generator (which doesn't make sense, since return just means "stop iteration"), the compiler will complain at compile-time -- this may catch some copy-and-paste mistakes:

>>> def foo():
...     yield 12
...     return 15
... 
  File "<stdin>", line 3
SyntaxError: 'return' with argument inside generator

I personally just advise against copy and paste programming. :-)

From the PEP:

Note that return means "I'm done, and have nothing interesting to return", for both generator functions and non-generator functions.

Answer 3

I'll try to answer the first of your questions.

A regular function, when called like this:

val = func()

executes its inside statements until it ends or a return statement is reached. Then the return value of the function is assigned to val.

If a compiler recognizes the function to actually be a generator and not a regular function (it does that by looking for yield statements inside the function -- if there's at least one, it's a generator), the scenario when calling it the same way as above has different consequences. Upon calling func(), no code inside the function is executed, and a special <generator> value is assigned to val. Then, the first time you call val.next(), the actual statements of func are being executed until a yield or return is encountered, upon which the execution of the function stops, value yielded is returned and generator waits for another call to val.next().

That's why, in your example, function __someFunc didn't print "hello" -- its statements were not executed, because you haven't called self.__someFunc().next(), but only self.__someFunc().

Unfortunately, I'm pretty sure there's no built-in warning mechanism for programming errors like yours.

Answer 4

Python doesn't know whether you want to create a generator object for later iteration or call a function. But python isn't your only tool for seeing what's going on with your code. If you're using an editor or IDE that allows customized syntax highlighting, you can tell it to give the yield keyword a different color, or even a bright background, which will help you find your errors more quickly, at least. In vim, for example, you might do:

:syntax keyword Yield yield
:highlight yield ctermbg=yellow guibg=yellow ctermfg=blue guifg=blue

Those are horrendous colors, by the way. I recommend picking something better. Another option, if your editor or IDE won't cooperate, is to set up a custom rule in a code checker like pylint. An example from pylint's source tarball:

from pylint.interfaces import IRawChecker
from pylint.checkers import BaseChecker

class MyRawChecker(BaseChecker):
    """check for line continuations with '\' instead of using triple
    quoted string or parenthesis
    """

    __implements__ = IRawChecker

    name = 'custom_raw'
    msgs = {'W9901': ('use \\ for line continuation',
                     ('Used when a \\ is used for a line continuation instead'
                      ' of using triple quoted string or parenthesis.')),
           }
    options = ()

    def process_module(self, stream):
        """process a module

        the module's content is accessible via the stream object
        """
        for (lineno, line) in enumerate(stream):
            if line.rstrip().endswith('\\'):
                self.add_message('W9901', line=lineno)


def register(linter):
    """required method to auto register this checker"""
    linter.register_checker(MyRawChecker(linter))

The pylint manual is available here: http://www.logilab.org/card/pylint_manual And vim's syntax documentation is here: http://www.vim.org/htmldoc/syntax.html

Answer 5

We do this.

Generators have names with "generate" or "gen" in their name. It will have a yield statement in the body. Pretty easy to check visually, since no method is much over 20 lines of code.

Other methods don't have "gen" in their name.

Also, we do not every use __ (double underscore) names under any circumstances. 32,000 lines of code. Non __ names.

The "generator vs. non-generator" method function is entirely a design question. What did the programmer "intend" to happen. The compiler can't easily validate your intent, it can only validate what you actually typed.