python - urrlib2 request https site - getting 400 error

Question

Hi...

using the following snip of code to access a url with a post.

i can get it using wget and the following: wget --post-data 'p_calling_proc=bwckschd.p_disp_dyn_sched&p_term=201010' https://spectrumssb2.memphis.edu/pls/PROD/bwckgens.p%5Fproc%5Fterm%5Fdate

for some reason, i'm having an issue with my python text, in that i get a errorcode of 400. (and of course the browser works as expected)

any thoughts/comments/etc...

the python test that i have:

//==========================================

import urllib 
import urllib2
import sys, string
import time
import mechanize

Request = urllib2.Request
urlopen = urllib2.urlopen

headers ={'User-Agent': 'Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)'}
query = "p_calling_proc%3Dbwckschd.p_disp_dyn_sched%26p_term%3D201010"
url1="https://spectrumssb2.memphis.edu/pls/PROD/bwckgens.p_proc_term_date"

req = Request(url1, query, headers)

test1=0
test=0
while test==0:
  print "aaaaattttt \n"
  try: 
    res = urlopen(req)
    #req = Request(url1, query, headers)
    print "aaaappppp \n"
    #urllib2.URLError, (e)
    #print e
  except urllib2.HTTPError, e:
    print "ffff1111 "+str(e.code)+"\n"
    if e.code:
      test1=1
      print "error ..sleep \n"
      time.sleep(1)
    else:
      test1=0
  except urllib2.URLError, e:
    print e.reason
    #print "ffff3333 "+e.code+"\n"
    if e.reason:
      test1=1
      print "error ..sleep \n"
      time.sleep(1)
    else:
      test1=0
  #print "ddd "+e.code +"\n"
  #print e
  if test1==0:
    test=1

print "test1 = "+str(test1)+"\n"
#res = urlopen(req)
print "gggg 000000000000\n"
s = res.read()

.

---

any thoughts/comments would be appreciated..

thanks

Answer 1

I think your query string is not quite right. Try using the urllib.urlencode() method to generate the query, a la

urllib.urlencode([ ('param1', value1), ('param2',value2) ])

Answer 2

Try not encoding the query string. The &'s and ='s in the POST data don't need to be urlencoded. If the web app on the remote end does not expect the %xx encoding in the query string, it won't be able to parse it.

Here's curl's HTTP request headers:

POST / HTTP/1.1
User-Agent: curl/7.19.4 (universal-apple-darwin10.0) libcurl/7.19.4 OpenSSL/0.9.8k zlib/1.2.3
Host: 127.0.0.1 
Accept: */*
Content-Length: 188
Expect: 100-continue

bwckschd.p_disp_dyn_sched&p_term=201010

And here's the HTTP request headers from your python:

POST / HTTP/1.1
Accept-Encoding: identity
Content-Length: 60
Host: 127.0.0.1
Content-Type: application/x-www-form-urlencoded
Connection: close
User-Agent: Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)

p_calling_proc%3Dbwckschd.p_disp_dyn_sched%26p_term%3D201010