Using open-uri how can I get the contents of a redirecting page?

Question

I ultimately want to get data from this page:

http://www.canadapost.ca/cpotools/apps/track/personal/findByTrackNumber?trackingNumber=0656887000494793

But that page forwards to:

http://www.canadapost.ca/cpotools/apps/track/personal/findByTrackNumber?execution=eXs1

So when I use open (open-uri) to try and fetch the data, it throws a RuntimeError error saying HTTP redirection loop:

So I'm not really sure how to get that data after it redirects and throws that error.

Answer 1

The site seems to be doing some of the redirection logic with sessions. If you don't send back the session cookies they are sending on the first request you will end up in a redirect loop. IMHO it's a crappy implementation on their part.

However, I tried to pass the cookies back to them, but I didn't get it to work, so I can't be completely sure that that is all that's going on here.

Answer 2

You need a tool like Mechanize. From it's description:

The Mechanize library is used for automating interaction with websites. Mechanize automatically stores and sends cookies, follows redirects, can follow links, and submit forms. Form fields can be populated and submitted. Mechanize also keeps track of the sites that you have visited as a history.

which is exactly what you need. So,

sudo gem install mechanize

then

require 'mechanize'
agent = WWW::Mechanize.new
page = agent.get "http://www.canadapost.ca/cpotools/apps/track/personal/findByTrackNumber trackingNumber=0656887000494793"
puts page.body

and you're ready to rock 'n' roll.