How do Enumerators work in Ruby 1.9.1?

Question

This question is not about how to use Enumerators in Ruby 1.9.1 but rather im curious how they work. Here is some code:

class Bunk
    def initialize
        @h = [*1..100]
    end

    def each
        if !block_given?
            enum_for(:each)
        else
            0.upto(@h.length) { |i|
                yield @h[i]
            }
        end
    end
end

In the above code I can go: e = Bunk.new.each. And then e.next, e.next to get each successive element...but how exactly is it suspending execution and then resuming at the right spot ? I am aware that if the yield in the 0.upto is replaced with Fiber.yield then it's easy to understand...but that is not the case here, it is a plain old yield! so how does it work? I have looked at enumerator.c but it's neigh on incomprehensible for me. Maybe someone could provide an implementation in ruby (using fibers, not 1.8.6 style continuation-based enumerators) that makes it all clear?

thanks :D

Answer 1

Actually in your e = Bunk.new.each the else clause is not executed initially. Instead the 'if !block_given' clause executes and returns an enumerator object. The enumerator object does keep a fiber object internally. (At least that is what it looks like in enumerator.c)

When you call e.each it is calling a method on an enumerator which uses a fiber internally to keep track of its execution context. This method calls the Bunk.each method using the fibers execution context. The Bunk.each call here does execut the else clause and yields up the value.

I do not know how yield is implemented or how a fiber tracks the execution context. I haven't looked at that code. Almost all of the enumerator and fiber magic is implemented in C.

Are you really asking how fibers and yield are implemented? What level of detail are you looking for?

If I am off base please correct me.

Answer 2

Here's a plain ruby enumerator that uses Fibers and should pretty much behave like the original:

class MyEnumerator
  include Enumerable

  def initialize(obj, iterator_method)
    @f = Fiber.new do
      obj.send(iterator_method) do |*args|
        Fiber.yield(*args)
      end
      raise StopIteration
    end
  end

  def next
    @f.resume
  end

  def each
    loop do
      yield self.next
    end
  rescue StopIteration
    self
  end
end

And before someone complains about exceptions as flow control: The real Enumerator raises StopIteration at the end, too, so I just emulated the original behaviour.

Usage:

>> enum = MyEnumerator.new([1,2,3,4], :each_with_index)
=> #<MyEnumerator:0x9d184f0 @f=#<Fiber:0x9d184dc>
>> enum.next
=> [1, 0]
>> enum.next
=> [2, 1]
>> enum.to_a
=> [[3, 2], [4, 3]]

Answer 3

As the other posters noted, I believe it creates its own private "fiber" [in 1.9]. In 1.8.7 (or 1.8.6 if you use the backports gem) somehow or other it does the same thing (perhaps because all threads in 1.8 are the equivalent of fibers, it just uses them?)

Thus in 1.9 and 1.8.x, if you chain several of them together a.each_line.map.each_with_index { }

It actually flows through that whole chain with each line, kind of like a pipe on the command line

http://pragdave.blogs.pragprog.com/pragdave/2007/12/pipelines-using.html

HTH.