Help converting code using httlib2 to use urllib2

Question

What am I trying to do?

Visit a site, retrieve cookie, visit the next page by sending in the cookie info. It all works but httplib2 is giving me one too many problems with socks proxy on one site.

http = httplib2.Http()
main_url = 'http://mywebsite.com/get.aspx?id='+ id +'&rows=25'
response, content = http.request(main_url, 'GET', headers=headers)
main_cookie = response['set-cookie']
referer = 'http://google.com'
headers = {'Content-type': 'application/x-www-form-urlencoded', 'Cookie': main_cookie, 'User-Agent' : USER_AGENT, 'Referer' : referer}

How to do the same exact thing using urllib2 (cookie retrieving, passing to the next page on the same site)?

Thank you.

Answer 1

Use cookielib, it'll handle all the cookie related work as automatically as a web browser would.

Example:

import urllib2
import cookielib

cookie_jar = cookielib.LWPCookieJar()
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cookie_jar))

#Get the first page with the cookie, installing it in the cookie jar automatically
opener.open("http://yoursite.com/set-cookie")

#Get the second page, passing on the cookie in the cookiejar.
opener.open("http://yoursite.com/other")

#Alternatively, you can also install this opener as the default urllib2 opener:
urllib2.install_opener(opener)
#Now all urllib2 requests will use cookies:
urllib2.urlopen("http://yoursite.com/other")