how to do asynchronous http requests with epoll and python 3.1

Question

there is an interesting page http://scotdoyle.com/python-epoll-howto.html about how to do asnchronous / non-blocking / AIO http serving in python 3.

there is the tornado web server which does include a non-blocking http client. i have managed to port parts of the server to python 3.1, but the implementation of the client requires pyCurl and seems to have problems (with one participant stating how ‘Libcurl is such a pain in the neck’, and looking at the incredibly ugly pyCurl page i doubt pyCurl will arrive in py3+ any time soon).

now that epoll is available in the standard library, it should be possible to do asynchronous http requests out of the box with python. i really do not want to use asyncore or whatnot; epoll has a reputation for being the ideal tool for the task, and it is part of the python distribution, so using anything but epoll for non-blocking http is highly counterintuitive (prove me wrong if you feel like it).

oh, and i feel threading is horrible. no threading. i use stackless.

people further interested in the topic of asynchronous http should not miss out on this talk by peter portante at PyCon2010; also of interest is the keynote, where speaker antonio rodriguez at one point emphasizes the importance of having up-to-date web technology libraries right in the standard library.

edit there may be many ways to achieve high throughputs with little resource consumption. however, it is not only me who thinks that abandoning threads (and other, partially outdated, methods); the people over at the google skipfish project seem to think similar: they say skipfish uses a

Multiplexing single-thread, fully asynchronous network I/O and data processing model that eliminates memory management, scheduling, and IPC inefficiencies present in some multi-threaded clients.

Answer 1

In this poll, months ago, Giampaolo Rodolá said

I have a patch for adding epoll() support which is almost ready (asyncore supports poll(), not epoll()), it's just a matter of writing tests.

I'm not sure where to find that patch, but it seems a simple enough task that it should be readily portable into asyncore (the standard python library approach to async servers and clients -- sure, way behind Twisted or Tornado, but that's what the stdlib has;-).

Answer 2

Python 3 lacks libraries. You seem interested in libraries to help you do stuff, so why exactly are you using python 3 to solve your problem?

Answer 3

FWIW, I have a library at: http://github.com/mnot/nbhttp/

The client side is in production as part of redbot.org.

I haven't tried it in Python 3, but it may help as a basis for your work.

Cheers,