Timeout Error with Hpricot in Rails Controller

Question

Hey--I'm writing a basic Rails app that uses the digg API. I'm trying to parse the xml data that digg's api provides with hpricot, but when testing the page, the browser hangs until I eventually catch the Timeout::Error exception.

Here's the code for the controller:

require 'rubygems'
require 'hpricot'
require 'open-uri'

appkey = 'http://mportiz08.homeip.net/twigg'
query = CGI::escape(params[:id].gsub('_', ' ').gsub('#', ''))

@request = 'http://services.digg.com/search/stories?query=' + query + '&appkey=' + appkey
@response = Hpricot( open(@request) )

And here's the stack trace:

/usr/lib/ruby/1.8/timeout.rb:60:in `rbuf_fill'
/usr/lib/ruby/1.8/net/protocol.rb:134:in `rbuf_fill'
/usr/lib/ruby/1.8/net/protocol.rb:116:in `readuntil'
/usr/lib/ruby/1.8/net/protocol.rb:126:in `readline'
/usr/lib/ruby/1.8/net/http.rb:2020:in `read_status_line'
/usr/lib/ruby/1.8/net/http.rb:2009:in `read_new'
/usr/lib/ruby/1.8/net/http.rb:1050:in `request'
/usr/lib/ruby/1.8/open-uri.rb:248:in `open_http'
/usr/lib/ruby/1.8/net/http.rb:543:in `start'
/usr/lib/ruby/1.8/open-uri.rb:242:in `open_http'
/usr/lib/ruby/1.8/open-uri.rb:616:in `buffer_open'
/usr/lib/ruby/1.8/open-uri.rb:164:in `open_loop'
/usr/lib/ruby/1.8/open-uri.rb:162:in `catch'
/usr/lib/ruby/1.8/open-uri.rb:162:in `open_loop'
/usr/lib/ruby/1.8/open-uri.rb:132:in `open_uri'
/usr/lib/ruby/1.8/open-uri.rb:518:in `open'
/usr/lib/ruby/1.8/open-uri.rb:30:in `open'
/home/marcus/dev/ruby/twigg/app/controllers/stories_controller.rb:15:in `view'
/var/lib/gems/1.8/gems/actionpack-2.3.3/lib/action_controller/base.rb:1327:in `send'
/var/lib/gems/1.8/gems/actionpack-2.3.3/lib/action_controller/base.rb:1327:in `perform_action_without_filters'
/var/lib/gems/1.8/gems/actionpack-2.3.3/lib/action_controller/filters.rb:617:in `call_filters'
/var/lib/gems/1.8/gems/actionpack-2.3.3/lib/action_controller/filters.rb:610:in `perform_action_without_benchmark'
/var/lib/gems/1.8/gems/actionpack-2.3.3/lib/action_controller/benchmarking.rb:68:in `perform_action_without_rescue'
/var/lib/gems/1.8/gems/actionpack-2.3.3/lib/action_controller/benchmarking.rb:68:in `perform_action_without_rescue'
/var/lib/gems/1.8/gems/actionpack-2.3.3/lib/action_controller/rescue.rb:160:in `perform_action_without_flash'
/var/lib/gems/1.8/gems/actionpack-2.3.3/lib/action_controller/flash.rb:146:in `perform_action'
/var/lib/gems/1.8/gems/actionpack-2.3.3/lib/action_controller/base.rb:527:in `send'
/var/lib/gems/1.8/gems/actionpack-2.3.3/lib/action_controller/base.rb:527:in `process_without_filters'
/var/lib/gems/1.8/gems/actionpack-2.3.3/lib/action_controller/filters.rb:606:in `process'
/var/lib/gems/1.8/gems/actionpack-2.3.3/lib/action_controller/base.rb:391:in `process'
/var/lib/gems/1.8/gems/actionpack-2.3.3/lib/action_controller/base.rb:386:in `call'
/var/lib/gems/1.8/gems/actionpack-2.3.3/lib/action_controller/routing/route_set.rb:434:in `call'

I'm just learning Rails, and I can't figure out what's going on. Any ideas as to why this is happening?

Update

I tried the same exact process with a locally saved xml file, and it worked perfectly--the problem might have something to do with open-uri and the remote xml

Answer 1

Mayb diggs only accept webbrowser?

I think HPricot sends another user-header? Would be good to find out what headers become sent from hpricot?

Answer 2

Lichtamberg--you were right. The problem was that digg required a user agent header to be sent with the request: see here

I just modified the open method call like this:

@response = Hpricot( open(@request, 'User-Agent' => 'twigg') )