Ruby HTML scraper written in Hpricot having trouble with escaped HTML

Question

I am trying to scrape this page: http://www.udel.edu/dining/menus/russell.html. I have written a scraper in Ruby using the Hpricot library.

problem: HTML page is escaped and I need to display it unescaped

example: "M&M" should be "M&M"  
example: "Entrée" should be "Vegetarian Entrée"  

I have tried using the CGI library in Ruby (not too successful) and the HTMLEntities gem that I found through this Stack Overflow post.

HTMLEntities works during testing:

require 'rubygems' 
require 'htmlentities'
require 'cgi'

h = HTMLEntities.new
puts "h.decode('Entrée') = #{h.decode("Entrée")}"

blank = " "
puts "h.decode blank = #{h.decode blank}"
puts "CGI.unescapeHTML blank = |#{CGI.unescapeHTML blank}|"

puts "h.decode '<th width=86 height=59 scope=row>Vegetarian Entr&eacute;e</th> ' = |#{h.decode '<th width=86 height=59 scope=row>Vegetarian Entr&eacute;e</th> '}|"  

correctly yields

h.decode('Entrée') = Entrée
h.decode blank =  
CGI.unescapeHTML blank = | |
h.decode '<th width=86 height=59 scope=row>Vegetarian Entr&eacute;e</th> ' = |<th width=86 height=59 scope=row>Vegetarian Entrée</th> |

However, when I go to use it on a file with open-uri it does not work properly:

require 'rubygems'
require 'hpricot'
require 'open-uri'
require 'htmlentities'
require 'cgi'
f = open("http://www.udel.edu/dining/menus/russell.html")
htmlentity = HTMLEntities.new
while line = f.gets
  puts htmlentity.decode line
end

Incorrectly yields things like:

<th width="60" height="59" scope="row">Vegetarian Entrée</th>

and

<th scope="row"> </th>  // note: was originally '&nbsp;' to indicate a blank

but correctly handles M&M by yielding:

<td valign="middle" class="menulineA">M&M Brownies</td>

Am I treating the escaped HTML incorrectly? I don't understand why it works in some cases and not in others.

I am running ruby 1.8.7 (2009-06-12 patchlevel 174) [i486-linux]

Any help/suggestion is appreciated. Thanks.

Answer 1

HTMLEntities seems to work but you have an encoding problem. The terminal you're printing on is probably set up for a latin charset and barfs on the utf-8 characters output by your script.

In what environment are you running ruby ?

The reason '&' displays correctly is that it's an ascii character and thus will display the same in most encodings.The problem is that it's not supposed to happen alone in an xml document and could pose problems later when you feed your decoded file to hpricot. I believe the proper way would be to parse with hpricot and then pass what you're extracting from the document to HTMLEntity.