httplib2 giving internal server error 500 with proxy

Question

Following is the code and error it throws. It works fine without the proxy http = httplib2.Http() .

When I try the same http proxy in Firefox, it works fine.

Any pointers are highly appreciated!

Usage :

http = httplib2.Http(proxy_info = httplib2.ProxyInfo(socks.PROXY_TYPE_HTTP, '68.48.25.158', 25681))

main_url = 'http://www.mywebsite.com'
response, content = http.request(main_url, 'GET')

Error :

  File "testproxy.py", line 17, in <module>
    response, content = http.request(main_url, 'GET')
  File "/home/kk/bin/pythonlib/httplib2/__init__.py", line 1129, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
  File "/home/kk/bin/pythonlib/httplib2/__init__.py", line 901, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "/home/kk/bin/pythonlib/httplib2/__init__.py", line 862, in _conn_request
    conn.request(method, request_uri, body, headers)
  File "/usr/lib/python2.5/httplib.py", line 866, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python2.5/httplib.py", line 889, in _send_request
    self.endheaders()
  File "/usr/lib/python2.5/httplib.py", line 860, in endheaders
    self._send_output()
  File "/usr/lib/python2.5/httplib.py", line 732, in _send_output
    self.send(msg)
  File "/usr/lib/python2.5/httplib.py", line 699, in send
    self.connect()
  File "/home/kk/bin/pythonlib/httplib2/__init__.py", line 740, in connect
    self.sock.connect(sa)
  File "/home/kk/bin/pythonlib/socks.py", line 383, in connect
    self.__negotiatehttp(destpair[0],destpair[1])
  File "/home/kk/bin/pythonlib/socks.py", line 349, in __negotiatehttp
    raise HTTPError((statuscode,statusline[2]))
socks.HTTPError: (500, 'Internal Server Error')

Answer 1

Is the SOCKS client library installed and available to your code? The proxy support only works if the SOCKS library is installed.

Answer 2

Make sure the proxy isn't transparent. I don't know too much about this, but evidently a transparent proxy enables the server to see you're using a proxy, and perhaps even access your IP. Some websites will definitely shut down any requests that appear to originate from a proxy (for fear of bots). That may mean either throwing a fake internal server error or actually encountering an error. For me, using an anonymous proxy has always solved that problem. Since you said it works without the proxy, I would start there.