urllib2: submitting a form and then redirecting

Question

My goal is to come up with a portable urllib2 solution that would POST a form and then redirect the user to what comes out. The POSTing part is simple:

request = urllib2.Request('https://some.site/page', data=urllib.urlencode({'key':'value'}))
response = urllib2.urlopen(request)

Providing data sets request type to POST. Now, what I suspect all the data I should care about comes from response.info() & response.geturl(). I should do a self.redirect(response.geturl()) inside a get(self) method of webapp.RequestHandler.

But what should I do with headers? Anything else I've overlooked? Code snippets are highly appreciated. :)

TIA.

EDIT: Here's a naive solution I came up with. Redirects but the remote server shows an error indicating that there's no match to the previously POSTed form:

info = response.info()
for key in info:
    self.response.headers[key] = info[key]
self.response.headers['Location'] = response.geturl()
self.response.set_status(302)
self.response.clear()

Answer 1

I suspect this will almost always fail. When you POST a form, the URL you end up at is just the URL you posted to. Sending someone else to this URL, or even visiting it again with the same browser that just POSTed, is going to do a GET and the page won't have the form data that was POSTed. The only way this would work is if the site redirected after the POST to a URL containing some sort of session info.

Answer 2

The standard way to follow redirects with urllib2 is to use the HTTPRedirectHandler. (Not sure what you mean by 'what comes out' but I'm assuming a standard http redirect here, javascript redirect is a different beast)

# Created handler
redirectionHandler = urllib2.HTTPRedirectHandler() 

# 2 apply the handler to an opener
opener = urllib2.build_opener(redirectionHandler)

# 3. Install the openers
urllib2.install_opener(opener)


request = urllib2.Request('https://some.site/page', data=urllib.urlencode({'key':'value'}))
response = urllib2.urlopen(request)

See urllib2.HTTPRedirectHandler for details on the handler.

Answer 3

You will find using mechanize much easier than using urllib2 directly

http://wwwsearch.sourceforge.net/mechanize/