Hi there,
I'm serving a python script through WSGI. The script accesses a web resource through urllib, computes the resource and then returns a value.
Problem is that urllib doesn't seem to handle many concurrent requests to a precise URL.
As soon as the requests go up to 30 concurrent request, the requests slow to a crawl! :(
Help would be much appreciated! :D
Yeah, urllib doesn't do much concurrency. Every time you urlopen, it has to set up the connection, send the HTTP request, and get the status code and headers from the response (and possibly handle a redirect from there). So although you get to read the body of the response at your own pace, the majority of the waiting time for the request will have already happened.
If you need more concurrency, you'll probably have to pick up some kind of asynchronous network IO tool (eg. Eventlet seems to have a suitable example on its front page), or just launch each urlopen in its own thread.