How to ignore accents in highlight function.

Question

I have a micro-mini-search engine that highlights the search terms in my rails app. The search ignores accents and the highlight is case insensitive. Almost perfect. But, for instance if I have a record with the text "pão de queijo" and searches for "pao de queijo" the record is returned but the iext is not highlighted. Similarly, if I search for "pÃo de queijo" the record is returned but not highlighted properly.

My code is as simple as:

<%= highlight(result_pessoa.observacoes, search_string, '<span style="background-color: yellow;">\1</span>') %>

Answer 1

it sounds like you are using two different methods for deciding whether a match has occured or not: one for your search, and one for the hilight. Use the same method as your search for your highlighting and it should pick it up, no?

Answer 2

Maybe you're searching UTF-8 strings directly against a MySQL database?

A properly configured MySQL server (and likely any other mainstream database server) will perform correctly with case-insensitive and accent-insensitive comparisons.

That's not the case of Ruby, though. As of version 1.8 Ruby does not support Unicode strings. So you are getting correct results from your database server but the Rails' highlight function, which uses gsub, fails to find your search string. You need to reimplement highlight using an Unicode-aware string library, like ICU4R.

Answer 3

that broke your links

Answer 4

Hi!

I've just submitted a patch to Rails thats solves this.

http://rails.lighthouseapp.com/projects/8994-ruby-on-rails/tickets/3593-patch-support-for-highlighting-with-ignoring-special-chars

  # Highlights one or more +phrases+ everywhere in +text+ by inserting it into
  # a <tt>:highlighter</tt> string. The highlighter can be specialized by passing <tt>:highlighter</tt>
  # as a single-quoted string with \1 where the phrase is to be inserted (defaults to
  # '<strong class="highlight">\1</strong>')
  #
  # ==== Examples
  #   highlight('You searched for: rails', 'rails')
  #   # => You searched for: <strong class="highlight">rails</strong>
  #
  #   highlight('You searched for: ruby, rails, dhh', 'actionpack')
  #   # => You searched for: ruby, rails, dhh
  #
  #   highlight('You searched for: rails', ['for', 'rails'], :highlighter => '<em>\1</em>')
  #   # => You searched <em>for</em>: <em>rails</em>
  #
  #   highlight('You searched for: rails', 'rails', :highlighter => '<a href="search?q=\1">\1</a>')
  #   # => You searched for: <a href="search?q=rails">rails</a>
  #
  #   highlight('Šumné dievčatá', ['šumňe', 'dievca'], :ignore_special_chars => true)
  #   # => <strong class="highlight">Šumné</strong> <strong class="highlight">dievča</strong>tá  
  #
  # You can still use <tt>highlight</tt> with the old API that accepts the
  # +highlighter+ as its optional third parameter:
  #   highlight('You searched for: rails', 'rails', '<a href="search?q=\1">\1</a>')     # => You searched for: <a href="search?q=rails">rails</a>
  def highlight(text, phrases, *args)
    options = args.extract_options!
    unless args.empty?
      options[:highlighter] = args[0] || '<strong class="highlight">\1</strong>'
    end
    options.reverse_merge!(:highlighter => '<strong class="highlight">\1</strong>')

    if text.blank? || phrases.blank?
      text
    else
      haystack = text.clone
      match = Array(phrases).map { |p| Regexp.escape(p) }.join('|')
      if options[:ignore_special_chars]
        haystack = haystack.mb_chars.normalize(:kd)
        match = match.mb_chars.normalize(:kd).gsub(/[^\x00-\x7F]+/n, '').gsub(/\w/, '\0[^\x00-\x7F]*')
      end
      highlighted = haystack.gsub(/(#{match})(?!(?:[^<]*?)(?:["'])[^<>]*>)/i, options[:highlighter])
      highlighted = highlighted.mb_chars.normalize(:kc) if options[:ignore_special_chars]
      highlighted
    end
  end

Answer 5

Here's my article that explains elegant solution where you don't need Rails or ActiveSupport.