Help with cURL in Python

Question

Hi,

I have to POST a request to a server. In the API documentation of the website there is this example that uses cURL in PHP:

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'http://api.website.com');
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, "request=$wrapper");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
;
$data = curl_exec($ch);
curl_close($ch);

But my app is done using Python, so I tried to write something like that but this code doesn't work:

req = urllib2.Request(url, formatted)  
response = urllib2.urlopen(req)  
html = response.read()  
print html+"\n\n"  

Can you help me write a working conversion of a PHP cURL program to Python?

Thank you!!

Answer 1

curl is for Python too: http://pycurl.sourceforge.net/

The example could be translated into Python and pycurl like this:

import pycurl
c = pycurl.Curl()
c.setopt(pycurl.URL, "http://api.website.com")
c.setopt(pycurl.POST, 1)
c.setopt(pycurl.POSTFIELDS, "request=%s" % wrapper)
import StringIO
b = StringIO.StringIO()
c.setopt(pycurl.WRITEFUNCTION, b.write)
c.perform()
c.close()
data = b.getvalue()

Your Python code using urllib2 looks OK, it should be working. Probably there is an error in something other you did not mention in question; could you be please more specific?

Answer 2

Taking your code, this should actually work.

 import urllib 
 import urllib2

 url = 'http://api.website.com/' 
 values = {'some_key':'some_value'} 
 data = urllib.urlencode(values) 
 req = urllib2.Request(url, data) 
 response = urllib2.urlopen(req) 
 page = response.read()
 print page + '\n\n'

What is the error you are getting?

Answer 3

Consider using a packet sniffer to figure out if cURL is sending User-Agent information. If it is, and the service is expecting that information, then use the add_header() method on your Request (from urllib2 documentation, bottom of page):

import urllib2
req = urllib2.Request('http://api.website.com/')
# Your parameter encoding here
req.add_header('User-agent', 'Mozilla/5.0')
r = urllib2.urlopen(req)
# Process the response

Answer 4

It's quite embarrassing but... the only problem with my code that uses urllib and urllib2 is... that this code do a GET and not a POST !!!

Here my scan using Wireshark:

1- using urllib and urllib2

Hypertext Transfer Protocol
    GET / HTTP/1.1\r\n
        [Expert Info (Chat/Sequence): GET / HTTP/1.1\r\n]
            [Message: GET / HTTP/1.1\r\n]
            [Severity level: Chat]
            [Group: Sequence]
        Request Method: GET
        Request URI: /
        Request Version: HTTP/1.1
    Accept-Encoding: identity\r\n
    Host: api.apptrackr.org\r\n
    Connection: close\r\n
    User-Agent: Python-urllib/2.6\r\n
    \r\n

2- using PyCurl

Hypertext Transfer Protocol
    POST / HTTP/1.1\r\n
        [Expert Info (Chat/Sequence): POST / HTTP/1.1\r\n]
            [Message: POST / HTTP/1.1\r\n]
            [Severity level: Chat]
            [Group: Sequence]
        Request Method: POST
        Request URI: /
        Request Version: HTTP/1.1
    User-Agent: PycURL/7.19.5\r\n
    Host: api.website.com\r\n
    Accept: */*\r\n
    Content-Length: 365\r\n
        [Content length: 365]
    Content-Type: application/x-www-form-urlencoded\r\n
    \r\n
Line-based text data: application/x-www-form-urlencoded
    [truncated]         request=%7B%22enc_key%22%3A%22o37vOsNetKgprRE0VsBYefYViP4%2ByB3pjxfkfCYtpgiQ%2ByxONgkhhsxtqAwaXwCrrgx%2BPDuDtMRZNI1ez//4Zw%3D%3D%22%2C%22format%22%3A%22RSA_RC4_Sealed%22%2C%22profile%22%3A%22Ldn%22%2C%22request%22%3A%22bQ%2BHm/

so the code works, but it's not right for me because i need a POST, but i will prefer to use NOT PyCurl. Any idea?

Thank you very much!!