How do I debug a Net::HTTPInternalServerError error when using Mechanize?

Question

c:/ruby/lib/ruby/gems/1.8/gems/mechanize-1.0.0/lib/mechanize.rb:259:in `get': 500 => Net::HTTPInternalServerError (Mechanize::ResponseCodeError)

I get the above error when I try to navigate to the following webpage

http://fakewebsite.com//admin/edit_building.cfm?page=buildings&updateMode=yes&id=1251

I can navigate just fine when copy and paste the link into a browser.

Note: This website does require a login, which I handle with code like the following

$agent = Mechanize.new
$agent.get('http://fakewebsite//admin/login.cfm?res=-5')
form = $agent.page.forms.first
form.EMail = "admin"
form.Password = "password"
form.submit

This login works just fine, since I access other webpages protected by the login.

I'm not sure where to go from here, any suggestions?

---

I solved the problem with some help from below. Turns out after I successfully login into the site, it takes me to a home page. This home page has several iframes in it pointing to other webpages. Mechanize doesn't automatically retrieve these pages. When I manually retrieved them I discovered that these pages were setting cookies that were necessary to access other pages on the site. I had mistakenly assumed that all cookies would get set after the login process was completed but before I arrived at the home page.

Answer 1

I'd start by enabling logging:

agent.log = Logger.new(STDOUT)

This is going to give you all headers, incoming and outgoing, which you can use to understand the problem. Additionally, I'd use an http sniffer like Scoop that would show me what's going on when I manually enter this url in the browser. Then you'll be able to compare what your browser is doing with what mechanize is doing and find the problem.

Also, try setting the user-agent because some websites do terminate a connection if mechanize honestly introduces itself.

agent.user_agent_alias = 'Mac Safari'