How do I replace accented Latin characters in Ruby?

Question

I have an ActiveRecord model, Foo, which has a name field. I'd like users to be able to search by name, but I'd like the search to ignore case and any accents. Thus, I'm also storing a canonical_name field against which to search:

class Foo
  validates_presence_of :name

  before_validate :set_canonical_name

  private

  def set_canonical_name
    self.canonical_name ||= canonicalize(self.name) if self.name
  end

  def canonicalize(x)
    x.downcase.  # something here
  end
end

I need to fill in the "something here" to replace the accented characters. Is there anything better than

x.downcase.gsub(/[àáâãäå]/,'a').gsub(/æ/,'ae').gsub(/ç/, 'c').gsub(/[èéêë]/,'e')....

And, for that matter, since I'm not on Ruby 1.9, I can't put those Unicode literals in my code. The actual regular expressions will look much uglier.

Answer 1

You probably want Unicode decomposition ("NFD"). After decomposing the string, just filter out anything not in [A-Za-z]. æ will decompose to "ae", ã to "a~" (approximately - the diacritical will become a separate character) so the filtering leaves a reasonable approximation.

Answer 2

iconv:

http://groups.google.com/group/ruby-talk-google/browse_frm/thread/8064dcac15d688ce?

=============

a perl module which i can't understand:

http://www.ahinea.com/en/tech/accented-translate.html

============

brute force (there's a lot of htose critters!:

http://projects.jkraemer.net/acts_as_ferret/wiki#UTF-8support

http://snippets.dzone.com/posts/show/2384

Answer 3

Convert the text to normalization form D, remove all codepoints with unicode category non spacing mark (Mn), and convert it back to normalization form C. This will strip all diacritics, and your problem is reduced to a case insensitive search.

See http://blogs.msdn.com/michkap/archive/2005/02/19/376617.aspx and http://blogs.msdn.com/michkap/archive/2007/05/14/2629747.aspx for details.

Answer 4

I think that you maybe don't really what to go down that path. If you are developing for a market that has these kind of letters your users probably will think you are a sort of ...pip. Because 'å' isn't even close to 'a' in any meaning to a user. Take a different road and read up about searching in a non-ascii way. This is just one of those cases someone invented unicode and collation.

Answer 5

Rails has already a builtin for normalizing, you just have to use this to normalize your string to form KD and remove the other chars like this:

>> "àáâãäå".chars.normalize(:kd).gsub(/[^\x00-\x7F]/n,'').downcase.to_s
=> "aaaaaa"

Answer 6

The key is to use two columns: canonical_text and original_text. Use original_text for display and canonical_text for searches. That way, if a user searches for "Visual Cafe," she sees the "Visual Café" result. If she really wants a different item called "Visual Cafe," it can be saved separately.

To get the characters in a Ruby 1.8 source file, do something like this:

register_replacement([0x008A].pack('U'), 'S')

Answer 7

lol.. i just tryed this.. and it is working.. iam still not pretty sure why.. but when i use this 4 lines of code:

• str = str.gsub(/[^a-zA-Z0-9 ]/,"")
• str = str.gsub(/[ ]+/," ")
• str = str.gsub(/ /,"-")
• str = str.downcase

it automaticly removes any accent from filenames.. which i was trying to remove(accent from filenames and renaming them than) hope it helped :)

Answer 8

For anyone reading this wanting to strip all non-ascii characters this might be useful, I used the first example successfully.