pycurl fails but curl (from bash) works in ubuntu

Question

I'm using curl and pycurl to connect to a secure 3rd party api and when I use pycurl I'm getting authentication errors back from the server, but when I use curl on the command line and do the same thing it works. I set both to verbose mode and am seeing some differences in the request, but I can't seem to figure out what the error is.

They seem to be using different encryption methods, perhaps that is the problem? If anyone has ideas on different options to try with pycurl or suggestions for recompiling pycurl to work like curl, that would be awesome. Thanks.

Here are my pycurl settings, fyi:

    buffer = cStringIO.StringIO()

    curl = pycurl.Curl()
    curl.setopt(pycurl.VERBOSE,1) 
    curl.setopt(pycurl.POST, 1)
    curl.setopt(pycurl.POSTFIELDS, post_data)
    curl.setopt(pycurl.TIMEOUT_MS, self.HTTP_TIMEOUT)
    curl.setopt(pycurl.URL, url)
    curl.setopt(pycurl.FOLLOWLOCATION, self.HTTP_FOLLOW_REDIRECTS)
    curl.setopt(pycurl.MAXREDIRS, self.HTTP_MAX_REDIRECTS)
    curl.setopt(pycurl.WRITEFUNCTION, buffer.write)
    curl.setopt(pycurl.NOSIGNAL, 1)
    curl.setopt(pycurl.SSLCERT, self.path_to_ssl_cert)

    curl.setopt(pycurl.SSL_VERIFYPEER, 0)

    # 1/0

    try:
        curl.perform()

...

Oh, last thing: the same python script I'm using works on my Mac laptop but doesn't work on the ubuntu server I'm trying to setup.

python test.py 
18:09:13,299 root INFO fetching: https://secure.....
* About to connect() to secure.... 1129 (#0)
*   Trying 216....... * connected
* Connected to secure.... port 1129 (#0)
* found 102 certificates in /etc/ssl/certs/ca-certificates.crt
*        server certificate verification OK
*        common name: secure.... (matched)
*        server certificate expiration date OK
*        server certificate activation date OK
*        certificate public key: RSA
*        certificate version: #3
*        subject: .......
*        start date: Sat, 14 Feb 2009 22:45:27 GMT
*        expire date: Mon, 15 Feb 2010 22:45:27 GMT
*        issuer: ...
*        compression: NULL
*        cipher: AES 128 CBC
*        MAC: SHA
User-Agent: PycURL/7.16.4
Host: secure....
Accept: */*
Content-Length: 387
Content-Type: application/x-www-form-urlencoded

< HTTP/1.1 200 OK
< Content-Length: 291
< 
* Connection #0 to host secure.... left intact
* Closing connection #0


 curl -v -d '...' --cert cert.pem  https://secure....
* About to connect() to secure.... port 1129 (#0)
*   Trying 216....... connected
* Connected to secure.... port 1129 (#0)
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* SSLv2, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Request CERT (13):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS handshake, CERT verify (15):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using AES256-SHA
* Server certificate:
*        subject: .......
*        start date: 2009-02-14 22:45:27 GMT
*        expire date: 2010-02-15 22:45:27 GMT
*        common name: secure.... (matched)
*        issuer: ... Certificate Authority
* SSL certificate verify ok.
> User-Agent: curl/7.16.4 (i486-pc-linux-gnu) libcurl/7.16.4 OpenSSL/0.9.8e zlib/1.2.3.3 libidn/1.0
> Host: secure....:1129
> Accept: */*
> Content-Length: 387
> Content-Type: application/x-www-form-urlencoded
> 
< HTTP/1.1 200 OK
< Content-Length: 342

Answer 1

I have a little trouble understanding the code/output fragments you have posted. Is the actual error message included?

Problems with SSL/TLS are often because of the X.509 certificate infrastructure. There are "Certificate Authorities" (CA) like Verisign, RapidSSL etc. which digitally "sign" the certificates of servers. To check these signatures, you need the so called "root certificate" of the CA who signed the certificate of the server ("issuer") you are connecting to.

Usually operating systems come with a fair amount of certificates pre-installed. And often Browsers, the OS and certain libraries all have their own list of certificates. On a Mac you can see them, if you start the program "Keychain Access" and open the "System Roots" keychain.

So I suggest you check if the cert is missing from Ubuntu and if so to add it there. (Maybe that is all saved in /etc/ssl/certs/)

Answer 2

Ubuntu pycurl uses GnuTLS while ubuntu curl command line uses OpenSSL. There are differences e.g. in supported certificate formats.

I for one cannot comprehend this decision taken by ubuntu devs/packagers. I stumbled on this once and could not work around it, luckily there are other distributions than ubuntu :-)

You could always try to complain to "humanity towards others."