Encoding problems with hpricot

I am getting the following encoding error when trying to scrap web pages with hpricot in ruby 1.9:

Encoding::CompatibilityError: incompatible character encodings: ASCII-8BIT and UTF-8

I can reproduce the error by doing the following:

ska:~ sam$ rvm 1.9.2@hpricot
ska:~ sam$ ruby -v
ruby 1.9.2dev (2010-05-31 revision 28117) [x86_64-darwin10.4.0]
ska:~ sam$ gem list

*** LOCAL GEMS ***

hpricot (0.8.2)
rake (0.8.7)
rdoc (2.5.8)
ska:~ sam$ irb
ruby-1.9.2-preview3 > require 'rubygems'
 => false 
ruby-1.9.2-preview3 > require 'hpricot'
 => true 
ruby-1.9.2-preview3 > require 'open-uri'
 => true 

ruby-1.9.2-preview3 > page = Hpricot(open('http://www.imdb.com/title/tt0435761/'))
 => #<Hpricot::Doc "\n" {doctype "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 4.01 Transitional//EN\" \"http://www.w3.org/TR/html4/loose.dtd\"&gt;"} "\n" {elem <html xmlns:og="http://opengraphprotocol.org/schema/" xmlns:fb="http://www.facebook.com/2008/fbml"&gt; "\n" {elem <head> "\n" __TRUNCATED__


ruby-1.9.2-preview3 > page.search("//div[@class = 'info-content").collect { |f| f.inner_text }.join(', ')

Encoding::CompatibilityError: incompatible character encodings: ASCII-8BIT and UTF-8
        from /Users/sam/.rvm/gems/ruby-1.9.2-preview3@hpricot/gems/hpricot-0.8.2/lib/hpricot/traverse.rb:160:in `join'
        from /Users/sam/.rvm/gems/ruby-1.9.2-preview3@hpricot/gems/hpricot-0.8.2/lib/hpricot/traverse.rb:160:in `inner_text'
        from /Users/sam/.rvm/gems/ruby-1.9.2-preview3@hpricot/gems/hpricot-0.8.2/lib/hpricot/traverse.rb:160:in `block in inner_text'
        from /Users/sam/.rvm/gems/ruby-1.9.2-preview3@hpricot/gems/hpricot-0.8.2/lib/hpricot/traverse.rb:160:in `map'
        from /Users/sam/.rvm/gems/ruby-1.9.2-preview3@hpricot/gems/hpricot-0.8.2/lib/hpricot/traverse.rb:160:in `inner_text'
        from /Users/sam/.rvm/gems/ruby-1.9.2-preview3@hpricot/gems/hpricot-0.8.2/lib/hpricot/traverse.rb:160:in `block in inner_text'
        from /Users/sam/.rvm/gems/ruby-1.9.2-preview3@hpricot/gems/hpricot-0.8.2/lib/hpricot/traverse.rb:160:in `map'
        from /Users/sam/.rvm/gems/ruby-1.9.2-preview3@hpricot/gems/hpricot-0.8.2/lib/hpricot/traverse.rb:160:in `inner_text'
        from /Users/sam/.rvm/gems/ruby-1.9.2-preview3@hpricot/gems/hpricot-0.8.2/lib/hpricot/traverse.rb:160:in `block in inner_text'
        from /Users/sam/.rvm/gems/ruby-1.9.2-preview3@hpricot/gems/hpricot-0.8.2/lib/hpricot/traverse.rb:160:in `map'
        from /Users/sam/.rvm/gems/ruby-1.9.2-preview3@hpricot/gems/hpricot-0.8.2/lib/hpricot/traverse.rb:160:in `inner_text'
        from /Users/sam/.rvm/gems/ruby-1.9.2-preview3@hpricot/gems/hpricot-0.8.2/lib/hpricot/traverse.rb:160:in `block in inner_text'
        from /Users/sam/.rvm/gems/ruby-1.9.2-preview3@hpricot/gems/hpricot-0.8.2/lib/hpricot/traverse.rb:160:in `map'
        from /Users/sam/.rvm/gems/ruby-1.9.2-preview3@hpricot/gems/hpricot-0.8.2/lib/hpricot/traverse.rb:160:in `inner_text'
        from /Users/sam/.rvm/gems/ruby-1.9.2-preview3@hpricot/gems/hpricot-0.8.2/lib/hpricot/traverse.rb:160:in `block in inner_text'
        from /Users/sam/.rvm/gems/ruby-1.9.2-preview3@hpricot/gems/hpricot-0.8.2/lib/hpricot/traverse.rb:160:in `map'
        from /Users/sam/.rvm/gems/ruby-1.9.2-preview3@hpricot/gems/hpricot-0.8.2/lib/hpricot/traverse.rb:160:in `inner_text'
        from /Users/sam/.rvm/gems/ruby-1.9.2-preview3@hpricot/gems/hpricot-0.8.2/lib/hpricot/traverse.rb:160:in `block in inner_text'
        from /Users/sam/.rvm/gems/ruby-1.9.2-preview3@hpricot/gems/hpricot-0.8.2/lib/hpricot/traverse.rb:160:in `map'
        from /Users/sam/.rvm/gems/ruby-1.9.2-preview3@hpricot/gems/hpricot-0.8.2/lib/hpricot/traverse.rb:160:in `inner_text'
        from /Users/sam/.rvm/gems/ruby-1.9.2-preview3@hpricot/gems/hpricot-0.8.2/lib/hpricot/traverse.rb:160:in `block in inner_text'
        from /Users/sam/.rvm/gems/ruby-1.9.2-preview3@hpricot/gems/hpricot-0.8.2/lib/hpricot/traverse.rb:160:in `map'
        from /Users/sam/.rvm/gems/ruby-1.9.2-preview3@hpricot/gems/hpricot-0.8.2/lib/hpricot/traverse.rb:160:in `inner_text'
        from (irb):5:in `block in irb_binding'
        from (irb):5:in `collect'
        from (irb):5
        from /Users/sam/.rvm/rubies/ruby-1.9.2-preview3/bin/irb:17:in `<main>'ruby-1.9.2-preview3 >

Your right that is an error, however still get an encoding error. Maybe i should try 1.9.1

Sam 2010-07-06 09:40:42

1.9.1 has changes to handle encodings. I haven't seen cases where 1.9.1 handled text better than 1.8.7, but that's probably because I haven't had to do any conversions lately. Anymore I think 1.9.1 is plenty stable and enough modules are working with it, so I use it as my default.

Greg 2010-07-06 16:26:42

ansaurus

tags:

views:

answers:

Encoding problems with hpricot

related questions