TLS connection with timeouts (and a few other difficulties)

Question

I have a HTTP client in Python which needs to use TLS. I need not only to make encrypted connections but also to retrieve info from the remote machine, such as the certificate issuer. I need to make connection to many HTTP servers, often badly behaved, so I absolutely need to have a timeout. With non-TLS connections, mysocket.settimeout(5) does what I want.

Among the many TLS Python modules:

python-gnutls does not allow to use settimeout() on sockets because it uses non-blocking sockets:

gnutls.errors.OperationWouldBlock: Function was interrupted.

python-openssl has a similar issue:

OpenSSL.SSL.WantReadError

The SSL module of the standard library does not work with Python 2.5.

Other libraries like TLSlite apparently does not give access to the metadata of the certificate.

The program is threaded so I cannot use signals. I need detailed control on the HTTP dialog so I cannot use a standard library like urllib2.

Background: this is the survey project DNSwitness. Relevant SO threads: Timeout on a Python function call and How to limit execution time of a function call in Python.

Answer 1

Although I've never used it for exactly this purpose, Twisted should do what you want. The only downside is that it's a rather large library, and you will also need to install PyOpenSSL (Twisted depends on it). If you've never used it before, Twisted's callback-based architecture can take some getting used to (you really want to read the tutorials before starting).

But aside from that, it's designed around the idea of managing a lot of connections, it of course lets you specify timeouts, reconnects, etc., and you can retrieve certificate info (see here).

Answer 2

I assume the problems you're having is the following, you're opening a connection using PyOpenSSL and you always get a WantReadError exception. And you can't distinguish between this error and a timeout. Consider the following example:

#!/usr/bin/python

import OpenSSL
import socket
import struct

context = OpenSSL.SSL.Context(OpenSSL.SSL.TLSv1_METHOD)
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.settimeout(5)
connection = OpenSSL.SSL.Connection(context,s)
connection.connect(("www.gmail.com",443))

# Put the socket in blocking mode
connection.setblocking(1)

# Set the timeout using the setsockopt
tv = struct.pack('ii', int(6), int(0))
connection.setsockopt(socket.SOL_SOCKET, socket.SO_RCVTIMEO, tv)

print "Connected to " , connection.getpeername()
print "Sate " , connection.state_string()

while True:
    try:
        connection.do_handshake()
        break
    except OpenSSL.SSL.WantReadError:
        print "Exception"
        pass
print "Sate " , connection.state_string()


print connection.send("koekoek\r\n")


while True:
    try:
        recvstr = connection.recv(1024)
        break
    except OpenSSL.SSL.WantReadError:
        print "Exception"
        pass

print recvstr

This will open an SSL connection to gmail, send an invalid string, read the response and print it. Note that: * the connection is explicitely set to blocking-mode * the recv timeout is explicitely set to in this case 6 seconds.

Now what will the behavior be, when the timeout occurs, the WantReadError exception will be thornw, in this case after waiting for 6 seconds. (You can remove the while True to avoid the retry, but in this case i added them for testing). The timeout set on the socket only appears to be effective in the connect() call.

An alternative would be when keeping the sockets in non-blocking mode which probably applies for the GNUTLS case as well is to perform the timekeeping yourself, you get the time when you launch the call, and in the while True, try: except WantReadError you perform the check every time yourself to see if you haven't been waiting for too long.

Answer 3

I would also recommend Twisted, and using M2Crypto for the TLS parts.