Cancel a group of HTTP requests in twisted

Question

I'm making several HTTP requests with twisted.web.client.getPage, and would like to be able to cancel some of them at the user's request. Ideally I would like to do something like:

# Pseudocode, getPage doesn't work like this:
getPage(url1, "group1")
getPage(url2, "group1")
getPage(url3, "group1")
...

# Later on
reactor.cancel_all("group1")

Maybe I could add all the Deferreds to a DeferredList, but I have a lot of small requests, so most of the requests would be finished at a given time anyway (plus, I don't know if you can add Deferreds to a existing DeferredList)... Is there a more idiomatic solution?

Answer 1

I am not providing the solution but pointing out to a following relevant discussion on twisted mailing list.

• http://www.mail-archive.com/[email protected]/msg01193.html

Answer 2

You are describing two separate issues. First, can an HTTP request made with getPage be cancelled at all? No, it can't. Second, can operations be grouped together so that they can all be cancelled simultaneously. Sure, that doesn't involve anything very special:

def cancel(group):
    for job in group:
        job.cancel()

group = []
group.append(job1)
group.append(job2)
...
cancel(group)

Nothing particular about Twisted here - this is just creating a collection and then operating on it. You don't need the reactor to help or anything. What you do need is a way to cancel an individual operation. The latest release of Twisted adds Deferred.cancel (so, contrary to the older post linked to in pyfunc's answer, Deferreds do have a notion of being cancelled now). However, for this to actually do anything, each API which creates Deferreds - for example, getPage - has to be updated to perform the relevant cancellation operation. As of Twisted 10.1, getPage has not been updated.

So you can either implement cancellation for getPage (and contribute it to Twisted, please!) or you can forget about actually cancelling the HTTP request and instead just ignore the result when it arrives.