Different results from yield vs return

Question

I don't really understand how yield statement works in this situation. The problem says that given an expression without parentheses, write a function to generate all possible fully parenthesized (FP) expressions. Say, the input is '1+2+3+4' which should be generated to 5 FP expressions:

1. (1+(2+(3+4)))
2. (1+((2+3)+4))
3. ((1+2)+(3+4))
4. ((1+(2+3))+4)
5. (((1+2)+3)+4)

My code is as follows.

OPS = ('+', '-', '*', '/')
def f(expr):
    """
    Generates FP exprs
    Recursive formula: f(expr1[op]expr2) = (f(expr1) [op] f(expr2))
    """
    if expr.isdigit(): yield expr
#       return [expr]

#   ret = []
    first = ''
    i = 0
    while i < len(expr):
        if expr[i] not in OPS:
            first += expr[i]
            i += 1
        else:
            op = expr[i]
            i += 1
            second = expr[i:]
            firstG, secondG = f(first), f(second)
            for e in ('(' + e1 + op + e2 + ')' for e1 in firstG for e2 in secondG):
                yield e
#               ret.append(e)
            first += op
#    return ret

If I use return statement (the commented out lines), then the code works as expected. However, when I change to yield statement as the code shows, I only get the first 4 results. If the number of operands of the input expression is increased, then of course more results will be lost. For example, for the input '1+2+3+4+5', I only get 8 instead of 14.

I finally figure out the way to make the code work by commenting out the line firstG, secondG = f(first), f(second) and replace the line

for e in ('(' + e1 + op + e2 + ')' for e1 in firstG for e2 in secondG):

by

for e in ('(' + e1 + op + e2 + ')' for e1 in f(first) for e2 in f(second)):

That means some 'information' of the generator is lost because of the line firstG, secondG = f(first), f(second) but I can't figure out the real reason. Could you guys give me some ideas?

Answer 1

The problem is you're iterating over generators instead of lists in the yield version, specifically secondG which is exhausted after one loop. Change the line to this and it works:

firstG, secondG = f(first), list(f(second))

Or, you can change your loop:

for e in ("(%s%s%s)" % (e1, op, e2) for e1 in f(first) for e2 in f(second)):
#                               new generator object every loop  ^^^^^^^^^

The non-yield version works because you return lists, which can be iterated over again, unlike generators. Also note you only iterate over firstG once, so it's not affected.

Remember that this:

r = [v for a in A for b in B]

Is equivalent to:

r = []
for a in A:
  for b in B:
    r.append(v)

Which more clearly shows the repeated loop over B.

Another example:

def y():
  yield 1
  yield 2
  yield 3
def r():
  return [1, 2, 3]

vy = y()
for v in vy:
  print v
for v in vy:
  print v

print "---"

vr = r()
for v in vr:
  print v
for v in vr:
  print v