Encapsulating a (pure Ruby) Ragel parser for infinite streams?

Question

I want to parse a continuous stream of bytes (from a socket) with a state machine using Ragel

However, all the Examples I have found are either parsing a complete file in one pass (like the Gherkin lexer or are using Ragels C Target (like the mongrel HTTP1.1 Parser)

I'm looking for some advice or examples on how to instantiate a Ragel State machine and then add bytes to it, keeping the existing state intact.

The final interface I am looking for is something like:

parser = MyStreamParser.new(Grammar)
parser.on_token { |t| puts t.inspect }

# I can't parse lines seperately because tokens can span multiple lines.
$stdin.each_line do |line|
  parser.add(line)
end

Any advice on how to do that in Ragel is greatly appreciated. I'd rather use that than code another state machine by hand.

Maybe Ragel is not the right tool? If not: What should I use instead?

Answer 1

It may not be exactly what you are looking for, but Dhaka is another decent parser generator to take a look at. I'm not sure that will help, but it has served me well in the past.

Answer 2

At first glance, Ragel doesn't look very Ruby-like. Have you taken a look at Statemachine? It looks like you can feed the state machine events (characters, in your problem) one at a time.