Difference between LoopingCall and callInThread in Python's Twisted.

Question

I'm trying to figure out the differences between a task.LoopingCall and a reactor.callInThread in Twisted.

All my self.sendLine's in the LoopingCall are performed immediately. The ones in the callInThread are not. They're only sent after the one in the LoopingCall has finished. Even though I'm sending the right delimiter.

Why is that? What's the difference? Aren't they both threads?

This is the server:

from twisted.internet import reactor, protocol, task
from twisted.protocols import basic
from twisted.python import log
import sys
import time
import threading
import Queue

class ServerProtocol(basic.LineOnlyReceiver):
    delimiter = '\0'
    clientReady = 1

    def __init__(self):
        print 'New client has logged on. Waiting for initialization'

    def lineReceived(self, line):
        if line.startswith('I'):
            print 'Data started with I: '+line
            user = dict(uid=line[1:6], x=line[6:9], y=line[9:12])
            self.factory.users[user['uid']] = user
            log.msg(repr(self.factory.users))
            self.startUpdateClient(user)
            reactor.callInThread(self.transferToClient)
            self.sendLine(user['uid'] + ' - Beginning - Initialized')
            print user['uid'] + ' - Beginning - Initialized'
        elif line.startswith('P'):
            print 'Ping!'
        elif line[0:3] == 'ACK':
            print 'Received ACK'
            self.clientReady = 1
        #else:
            #self.transport.loseConnection()

    def _updateClient(self, user):
        if self._running == 0:
            self._looper.stop()
            return
        self._running -= 1
        self._test += 1
        print user['uid'] + ' Sending test data' + str(self._test)
        self.sendLine(user['uid'] + ' Test Queue Data #%d' % (self._test,) + '\0')

    def startUpdateClient(self, user):
        self._running, self._test = 25, 0
        self._looper = task.LoopingCall(self._updateClient, user)
        self._looper.start(1, now=False)
        print user['uid'] + ' - Startupdateclient'

    def transferToClient(self):
        test = 20
        while test > 0:
            if self.clientReady == 1:
                test = test-1
                print 'Reactor test ' + str(test) + ' - ' + str(time.time())
                self.clientReady = 0
                self.sendLine('This is reactortest ' + str(test) + ' - ' + str(time.time()) +' \0')

class Server(protocol.ServerFactory):
    protocol = ServerProtocol
    def __init__(self):
        self.users = {}

if __name__ == '__main__':
    log.startLogging(sys.stderr)
    reactor.listenTCP(2000, Server())
    reactor.run()

This is the client:

#!/usr/bin/env python

import socket
import time

host = 'localhost'
port = 2000
size = 1024
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((host,port))
s.send('I12345070060\0')
running = 1

while running:
    s.send('ACK\0')
    data = s.recv(size)
    if data:
        print 'Received:', data 
    else:
        print 'Closing'
        s.close()
        running=0

Answer 1

Have you looked at the docs for LoopingCall? No thread involved -- it runs (every second, the way you're calling its start method) on the main thread, i.e., typically, the thread of the reactor. callInThread is the only one of the two that causes the function to be run on a separate thread.

Answer 2

Why is that? What's the difference? Aren't they both threads?

No. LoopingCall uses callLater; it runs the calls in the reactor.

All my self.sendLine's in the LoopingCall are performed immediately.

Yep, as they should be.

The ones in the callInThread are not.

It's not so much that they're not performed, it's that because you called a reactor API from a thread, which you are never ever ever allowed to do, you have put your program into a state where everything is completely broken, forever. Every future API call may produce bizarre, broken results, or no results, or random, inexplicable crashes.

You know, the normal way multithreaded programs work ;-).

To repeat: every API in twisted, with the sole exception of callFromThread (and by extension things which call callFromThread, like blockingCallFromThread), is not thread safe. Unfortunately, putting in warnings for every single API would be both a code maintenance nightmare, so several users have discovered this constraint in the same way that you have, by calling an API and noticing something weird.

If you have some code which runs in a thread that needs to call a reactor API, use callFromThread or blockingCallFromThread and it will dispatch the call to the reactor thread, where everything should work smoothly. However, for stuff like timed calls, there's really no need to use threads at all, and they would needlessly complicate your program.