Python: Asynchronous http requests sent in order with automatic handling of cookies?

Question

Hello,

I am coding a python (2.6) interface to a web service. I need to communicate via http so that :

• Cookies are handled automatically,
• The requests are asynchronous,
• The order in which the requests are sent is respected (the order in which the responses to these requests are received does not matter).

I have tried what could be easily derived from the build-in libraries, facing different problems :

• Using httplib and urllib2, the requests are synchronous unless I use thread, in which case the order is not guaranteed to be respected,
• Using asyncore, there was no library to automatically deal with cookies send by the web service.

After some googling, it seems that there are many examples of python scripts or libraries that match 2 out of the 3 criteria, but not the 3 of them. I am thinking of reading through the cookielib sources and adapting what I need of it to asyncore (or only to my application in a ad hoc manner), but it seems strange that nothing like this exists yet, as I guess I am not the only one interested. If anyone knows of pointers about this problem, it would be greatly appreciated.

Thank you.

Edit to clarify :

What I am doing is a local proxy that interfaces my IRC client with a webchat. It creates a socket that listens to IRC connections, then upon receiving one, it logs in the webchat via http. I don't have access to the behaviour of the webchat, and it uses cookies for session IDs. When client sends several IRC requests to my python proxy, I have to forward them to the webchat's server via http and with cookies. I also want to do this asynchronously (I don't want to wait for the http response before I send the next request), and currently what happens is that the order in which the http requests are sent is not the order in which the IRC commands were received.

I hope this clarifies the question, and I will of course detail more if it doesn't.

Answer 1

Using httplib and urllib2, the requests are synchronous unless I use thread, in which case the order is not guaranteed to be respected

How would you know that the order has been respected unless you get your response back from the first connection before you send the response to the second connection? After all, you don't care what order the responses come in, so it's very possible that the responses come back in the order you expect but that your requests were processed in the wrong order!

The only way you can guarantee the ordering is by waiting for confirmation that the first request has successfully arrived (eg. you start receiving the response for it) before beginning the second request. You can do this by not launching the second thread until you reach the response handling part of the first thread.