Python generators in various languages

Question

How do you emulate Python style generators in your favorite language? I found this one in Scheme. It must be interesting to see other implementations, especially in those languages that don't have first-class continuations.

Answer 1

Here is an example in C++ that simulates generators using fibers:

Yield Return Iterator for Native C++ Using Fibers

The "yield return" iterator is a language feature that was created for one reason: simplicity. It is generally much easier to iterate across whole collectionl, storing all context needed in local variables, rather than crafting a complicated, custom iterator object that stores its state across subsequent retrieval operations.

There are also the primitive C routines setjmp, longjmp to achieve similar results.
(Lua coroutines are implemented with the above method)

Answer 2

Monads can be used to represent generators (even if the semantics are little different).

So any language allowing us to define monadic operations within a special syntax can be used here.

• VB.NET/C# (Linq - but C# already got yield return)
• Scala (For-comprehensions)
• Haskell (do-notation)
• F#/OCaml (Computation expressions/Perform)

Ruby can emulate generators through its builtin continuation capabilities.

Answer 3

Common Lisp, although does not have native continuations, allows to create delimited continuations using CPS transformers like cl-cont. So generators in Common Lisp can be written in pretty much the same way as Scheme generators.

By the way, continuation-based generators have one trait that Python and C# generators lack: the yield may be called in the dynamic extent of the generator function call. Python and C# generators allow yield to be placed only inside the body of a generator.

Answer 4

In JavaScript 1.7+ I usually only have to add a few parentheses and brackets. Everything else is about the same. JavaScript 1.7 introduced pythonic generators and iterators among other things.

Generator expressions:

# Python
(x + 1 for x in y if x > 100)

// JavaScript 1.8+
(x + 1 for (x in y) if (x > 100))

Generators

# Python
def simpleRange(n):
    for i in xrange(n):
        yield i

for n in simpleRange(5):
     print(n)


// JavaScript 1.7+
function simpleRange(n) {
    for (let i = 0; i < n; i++)
        yield i;
}

for (n in simpleRange(5))
    print(n);

List/Array comprehensions

# Python
[x + 1 for x in y if x > 100]

// JavaScript 1.7+
[x + 1 for (x in y) if (x > 100)]

Answer 5

Ruby:

Generator function:

def simple_range(n)
    Enumerator.new do |y|    
        (0..n).each { |v| y.yield(v) }
    end
end

Answer 6

I wouldn't use yield at all in Lisp/Scheme.

'yield' requires some kind of co-routine or continuation facility in the language. Many uses of yield can be implemented in a simpler functional way.

YIELD is basically related to the famous COME-FROM operator. ;-) Here, a call in some place can lead to different places in some other routine, depending on its execution context. So a routine suddenly has multiple entry points whose order is determined at runtime. For simple uses this may be fine, but I would argue that for more complex code reasoning about the code would get harder.

Take the Scheme example linked in the question:

(define/y (step) 
  (yield 1)
  (yield 2)
  (yield 3)
  'finished)

(list (step) (step) (step))

Calling (step) multiple times returns different values then.

I would just create a closure:

(define step
  (let ((state '(1 2 3 finished)))
    (lambda ()
      (pop state))))

This breaks above function with a yield into two different things: a variable that carries the state and a simple function that changes the state. The state is no longer implicitly coded into the execution sequence.

 (list (step) (step) (step))))

One can imagine similar solutions for other uses of yield.

Compare that with the generators from the Common Lisp SERIES library:

(let ((x (generator (scan '(1 2 3 finished)))))
  (list (next-in x)
        (next-in x)
        (next-in x)))

If we look at this Python example from another answer

def simpleRange(n):
    for i in xrange(n):
        yield i

for n in simpleRange(5):
     print(n)

we can see that it duplicates the control structure. Both the calling place and the generator use a FOR iteration control structure. Using closures, we can get rid of the use of control structures inside the generator, by only providing the state transition code.