Parsing a structured text file in Ruby

Question

How can i easily parse a document which has this structure

description
some line of text
another line of text
more lines of text

quality
3 47 88 4 4 4  4

text: type 1
stats some funny stats

description
some line of text2
another line of text2
more lines of text2

quality
1 2  4 6 7

text: type 1
stats some funny stats

.
.
.

Ideally i would want an array of hash structures where each hash represents a 'section' of the document and probably should look like this:

{:description => "some line of text another line of text more lines of text", :quality => "3 47 88 4 4 4 4", :text =>type 1, :stats => "some funny stats"}

Answer 1

You should look for the indicator lines (description, quality, text and stats) in a loop and fill the hash while processing the document line by line.

Another option would be to use regular expressions and parse the document at once, but you don't really need regular expressions here, and if you're not familiar with them, I'd have to recommend against regexes.

UPDATE:

sections = []

File.open("deneme") do |f|
  current = {:description => "", :text => "", :quality => "", :stats => ""}
  inDescription = false
  inQuality = false

  f.each_line do |line|
    if inDescription
      if line.strip == ""
        inDescription = false
      else
        current[:description] += line
      end
    elsif inQuality
      current[:quality] = line.strip
      inQuality = false
    elsif line.strip == "description"
      inDescription = true
    elsif line.strip == "quality"
      inQuality = true
    elsif line.match(/^text: /)
      current[:text] = line[6..-1].strip
    elsif line.match(/^stats /)
      current[:stats] = line[6..-1].strip
      sections.push(current)
      current = {:description => "", :text => "", :quality => "", :stats => ""}
    end
  end
end

Answer 2

Your input looks pretty close to YAML, so I would convert the input to valid YAML (using method like Can's) then use standard ruby library to load it. Then when your user runs into something they didn't think of in their brilliant markup, tell them to just use YAML :)

Answer 3

The regex version:

ary = str.scan(/description\n(.*?)\n\nquality\n(.*?)\n\ntext:([^\n]+)\nstats([^\n]+)/m).inject([]) do |n, (desc, qual, text, stats)|
  n << { :description => desc.gsub("\n", ' '), :quality => qual, :text => text, :stats => stats }
end

Answer 4

One parsing trick is to read data in paragraph mode -- a chunk at a time. If your sub-sections are consistently delimited by 2 newlines (or if you can use a pre-process to impose such consistency), paragraph-reading might be useful.

Aside from the special handling needed for the 'text' sub-section, the example below is fairly general, requiring only that you declare the name of the last sub-section.

# Paragraph mode.
$/ = "\n\n"

last_subsection = 'text'
data = []

until DATA.eof
    data.push({})
    while true
        line = DATA.readline

        # Determine which sub-section we are in.
        ss = nil
        line.sub!( %r"\A(\w+):?\s*" ) { ss = $1; '' }

        # Special handling: split the 'text' item into 2 subsections.
        line, data[-1]['stats'] = line.split(/\nstats +/, 2) if ss == 'text'

        data[-1][ss] = line
        break if ss == last_subsection
    end

    # Cleanup newlines however you like.
    data[-1].each_key { |k| data[-1][k].gsub!(/\n/, ' ') }
end

# Check
data.each { |d| puts; d.each { |k,v| puts k + ' => ' + v } }

__END__
# Data not shown here...