Very strange problem with Gettext and Haml on Rails / rake updatepo broke

Question

Hi,

I'm on Rails 2.3.3 and using Haml 2.0.9 for my templates and Gettext-Rails 2.0.4 for the translation. Haml works like a charm and gettext is also working like it should.

But I cant get Gettext to parse Haml Files when using "rake updatepo". I created a custom parser like this:

# lib/haml_parser.rb
require 'gettext_rails/tools'
require 'haml'
# Haml gettext parser
module HamlParser
  module_function

  def target?(file)
    File.extname(file) == ".haml"
  end

  def parse(file, ary = [])
    haml = Haml::Engine.new(IO.readlines(file).join)
    code = haml.precompiled.split(/$/)
    GetText::RubyParser.parse_lines(file, code, ary)
  end
end

GetText::RGetText.add_parser(HamlParser)

My Rakefile looks like this:

# Rakefile
require(File.join(File.dirname(__FILE__), 'config', 'boot'))

require 'rake'
require 'rake/testtask'
require 'rake/rdoctask'

require 'tasks/rails'

desc "Create mo-files for L10n"
task :makemo do
  require 'gettext_rails/tools'
  GetText.create_mofiles(true)  #(true, "po", "locale")
end

desc "Update pot/po files to match new version."
task :updatepo do
  require 'gettext_rails/tools'
  require 'haml_parser'
  MY_APP_TEXT_DOMAIN = "APP" 
  MY_APP_VERSION     = "APP 1.1.0"
  GetText.update_pofiles(MY_APP_TEXT_DOMAIN, Dir.glob("{app,lib}/**/*.{rb,rhtml,html.erb,haml,html.haml,rjs}"),
                         MY_APP_VERSION)
end

This follows the known approach for parsing Haml files ( http://www.paulgillard.me.uk/2008/3/8/rails-haml-and-gettext ).

The problem: No MessageIds are recognized from my Haml files. I checked with "puts" in the Haml-Parser if it tried the right files, could parse them and so on. Everything seemed to be fine, it just recognize anything and always returned only the already found msgids and for the Haml file an empty Array.

The strange thing: When I enter this in my console, everything works:

$$ script/console
Loading development environment (Rails 2.3.3)
>> require 'gettext_rails/tools'
=> []
>> require 'haml'
=> []
>> file = "app/views/sessions/new.haml"
=> "app/views/sessions/new.haml"
>> haml = Haml::Engine.new(IO.readlines(file).join)
=> #<Haml::Engine:0x4254104 @tab_change=0, @block_opened=false, @inden [...]
>> code = haml.precompiled.split(/$/)
=> [" content_for :head do;", "\nhaml_temp =  stylesheet_link [...]
>> GetText::RubyParser.parse_lines(file, code, [])
=> [["Login", "app/views/sessions/new.haml:4"], [...]

As you can see everything works here (I left out the long returns). I'm kind of freaking out why this isn't the case in my Rake Task.

Anyone has an idea? You would really make me a happy Nerd!

Thanks!

Answer 1

ruby_gettext_extractor might help fix the gettext string extraction. Especially when you're using HAML 2.2 which changed it's internal format and completely broke the gettext extraction process.

Answer 2

Try modifying your haml_parser.rb to see where the parser gets stuck.

  def parse(file, ary = [])
    haml = Haml::Engine.new(IO.readlines(file).join)
    code = haml.precompiled.split(/$/)
    puts "Parsing Haml File: #{file}"
    RubyParser.parse_lines(file, code, ary)
  end

I'm having problems of my own in upgrading my setup.

Rails 2.3.4
gettext 2.0.4
haml 2.2.8

From what I have gathered it is largely due to the change of in HAML 2.2 Engine that made it's output unparseable by the gettext's default RubyParser.

I have tried the gems retoo-ruby_gettext_extractor (0.2.1) that depends on ruby_parser without success.

This ruby_parser has a RubyParser class of it's own and conflicts with Gettext's RubyParser module. So here's a classical case of why we should namespace our code. -_-

After hacking it to work by changing the names of the RubyParser in the gems, I can make it work to parse en template files, using retoo's gettext_extractor/rubyparser gems. However, the parser doesn't work on my non-latin template files. My HamlParser class as below. Anyone can have a go and fix this? A million thanks!

# haml_parser.rb
require 'rubygems'
require 'haml'
require 'gettext_rails/tools'
require 'ruby_gettext_extractor'

module HamlParser
  module_function

  def target?(file)
    File.extname(file) == '.haml'
  end

  def parse(file, ary = [])
    bypass = ! File.basename(file, 
'.haml').match(/(vi|zh|zh_HK|id|th)$/).nil?
    puts "HamlParser:#{file}:bypass:#{bypass}"
    return ary if bypass

    haml = Haml::Engine.new(IO.readlines(file).join)
    result = nil
    begin
      #result = GetText::RubyParser.parse_lines(file, 
haml.precompiled.split(/$/), ary)
      result = RubyGettextExtractor.parse_string(haml.precompiled, file, 
ary)
    rescue Exception => e
      puts "Error:#{file}"
      raise e
    end
    result
  end
end

GetText::RGetText.add_parser(HamlParser)

Answer 3

Just had the same problem and after a bit of debugging I found the cause: most of the output in Haml is generated by instructions inside of #{} in "" strings and that construct seems to fool the RubyParser class. If you had something like "#{_('some text')}" in a "normal" Ruby file, this would not be recognized, too.

My solution was to shuffle the Haml-precompiled code a little bit and get the _() calls outside of "#{}". That what the regexp does in line 4.

  def parse(file, ary = [])
    haml = Haml::Engine.new(IO.readlines(file).join)
    code = haml.precompiled
    code = code.gsub(/(.*#\{(_hamlout.adjust_tabs\(\d+\);\s*)?haml_temp)\s*=\s*(_\(['"].+['"]\))/) { |m| "haml_temp = #{$3}; #{$1}" }
    code = code.split(/$/)
    GetText::RubyParser.parse_lines(file, code, ary)
  end

This fixes about 80% of cases in my project, which was good enough for me. The remaining 20% sometimes can be fixed by changing:

%tag= _('text')

into:

%tag
  = _('text')

And if that does not help, you can use following trick:

- txt = _('text')
= txt

Didn't try the ruby_gettext_extractor gem yet, though.