python twisted threading

Question

Hi can you please tell me how use different functions in different thread using thread pool in twisted...say

I have a list of ids x=[1,2,3,4] where 1,2,...etc are ids(I got from data base and each one contains python script in some where disk).

what I want to do is

scanning of x traverse on list and run every script in different thread until they completed

Answer 1

Threads in Twisted are primarily used via twisted.internet.threads.deferToThread. Alternatively, there's a new interface which is slightly more flexible, twisted.internet.threads.deferToThreadPool. Either way, the answer is roughly the same, though. Iterate over your data and use one of these functions to dispatch it to a thread. You get back a Deferred from either which will tell you what the result is, when it is available.

from twisted.internet.threads import deferToThread
from twisted.internet.defer import gatherResults
from twisted.internet import reactor

def double(n):
    return n * 2

data = [1, 2, 3, 4]

results = []
for datum in data:
    results.append(deferToThread(double, datum))

d = gatherResults(results)
def displayResults(results):
    print 'Doubled data:', results
d.addCallback(displayResults)
d.addCallback(lambda ignored: reactor.stop())

reactor.run()

You can read more about threading in Twisted in the threading howto.

Answer 2

Thanx Calderone, your code helped me a lot.

I have few doubts like I can resize threadpool size by this way.

from twisted.internet import reactor
reactor.suggestThreadPoolSize(30)

say all 30 available threads are busy & there is still some ids in list(dict or tuple) 1-In this situation all ids will be traversed? I mean as soon as thread is free next tool(id) will be assigned to freed thread? 2-there is also some cases one tools must be executed before second tool & one tool output will be used by another tool,how will it be managed in twisted thread. 3