Recieving a 404 HTTPError on a working page in Ruby Script

Question

This is my first time asking a question, please be gentle!

I have a Rails application that handles content for a whole bunch of domains (over 100 so far). Each domain either points to where my app is hosted (Heroku, if you're interested), or the original place it was hosted. Every time a domain is ready, it needs to point to the heroku servers, so that my app can serve content for it.

To check to see if a domain has successfully been changed over from its original location to my application, I'm writing a script that looks for a special hidden tag I included in them. If it finds the tag, then the domain is pointing to my app. If not, it hasn't been changed, which I record.

The problem is that, at least for one domain so far, I'm getting a 404 OpenURI::HTTPError exception for my script. Which is strange, because I can visit the site just fine and I can even get it via curl. Does anyone know why a working site would get an error like this? Here's the important snippet:

require 'rubygems'
require 'open-uri'
require 'hpricot'
...
url = "http://www.#{domainname}.com"
doc = Hpricot(open(url)) #<---- Problem right here.
...

Thanks for all of your help!

Answer 1

Welcome to SO!

Here would be my debugging method:

1. See if you can replicate in irb with open-uri alone, no Hpricot:

$ irb -rubygems -ropen-uri

>> open('http://www.somedomain.com')

1. Look in your Heroku log to see if it even touches the server.
2. Look in your original server's log for the same.
3. Throw open something like Wireshark to see the HTTP transaction, and see if a 404 is indeed coming back.

Start with that, and come back with your results.