Python TCP Server, writing to clients?

Question

Hi

I have a tcp server which uses the select call to multiplex reading from clients. I have a client class (MClient) which manages the decoding of incoming data packets

while(1)

    rlist, wlist, xlist = select( input_sockets, output_sockets, [] , 1)

    for insock in rlist: #any clients????

        if insock is server_socket:
            new_socket, addr = server_socket.accept()
            input_sockets.append(new_socket)    
            clients[insock.fileno()] = MClient(new_socket, addr, client_id) #dict of clients        
        else:
            data = insock.recv(512)
            if data:
                clients[insock.fileno()].ProcessPacket(data)
            else:
                input_sockets.remove(insock)
                del clients[insock.fileno()]

    #handle writing to sockets
    for outsock in wlist: 
        ....not done yet



   #do some other stuff not associated with the socket

I am confused as to how to handle sending data back to the client, i.e how to write to the list 'output_sockets'. I am thinking of setting a flag in my MClient object which indicates that I have data to send back to the client. I would then in my server loop check each of the clients to see if this flag was set, and then poulate the output_list with the corresponding socket. When the socket is available for write, I would then call the appropriate clients write function.

This scheme does not seem very elegant, I would like to handle the writing in the main server loop. How would I accomplish this?

Thanks

Answer 1

The implementation you describe in your first paragraph sounds like the conventional way of implementing a server using select.

If you want to be able to write to the client in the "main server loop", by which I assume you mean having code that looks something like:

request = socket.recv()
response = process_request(request)
socket.send(response)

then you'll need to have a separate thread per client.

Answer 2

Here's something I wrote a while back to learn about processing multiple connections with a single thread. It is by no means perfect but illustrates what you want to do. The client object manages the read and write streams of the connection, and makes sure the server has the client socket in the right select() lists. This implements a simple protocol where messages are terminated by newlines. The pumpXXXX() functions just block read/write the streams and manage the read/write buffers. Complete messages are processed only when newlines are found in the buffers.

import socket
import select

class Client(object):

    '''This object is created for each client connection.  It tracks
    what has been read, what has been written, and processes complete
    messages terminated by newlines.  It responds by returning the
    original message wrapped in square brackets and terminated by a
    newline. '''

    def __init__(self,who,sock,server):

        '''who - client address
        sock - client socket
        server - server object for this client
        '''

        self.who = who
        self.readbuf = ''
        self.writbuf = ''
        self.server = server
        self.sock = sock

    def close(self):

        '''Removes client from server's reader/writer queues and
        closes the connection.'''

        self.sock.close()
        if self.sock in self.server.readers:
            self.server.readers.remove(self.sock)
        if self.sock in self.server.writers:
            self.server.writers.remove(self.sock)
        self.server.data.pop(self.sock)

    def pumprecv(self):

        '''Server calls pumprecv() when something is readable from the
        client socket.  The data is appended to the client's read
        buffer.mro Complete messages (if any) are then removed from
        the buffer and processed.'''

        try:
            tmp = self.sock.recv(1000)
        except socket.error,e:
            print 'recv',e
            self.close()
        else:                
            if tmp:
                self.readbuf += tmp

                # Complete messages are processed
                while '\n' in self.readbuf:
                    msg,self.readbuf = self.readbuf.split('\n',1)
                    print self.who,msg
                    self.writbuf += '[' + msg + ']\n'
                    # New data to send.  Make sure client is in the
                    # server's writer queue.
                    if self.sock not in self.server.writers:
                        self.server.writers.append(self.sock)
            else:
                self.close()

    def pumpsend(self):
        try:
            # send some data.  tmp is #chars sent (may not be all in writbuf).
            tmp = self.sock.send(self.writbuf)
        except socket.error,e:
            print 'send:',e
            self.close()
        else:
            # Removed sent characters from writbuf.
            self.writbuf = self.writbuf[tmp:]
            # If writbuf is empty, remove socket from server's write queue.
            if not self.writbuf:
                self.server.writers.remove(self.sock)

class Server(object):
    def __init__(self,ip='127.0.0.1',port=9999):
        self.ssock = socket.socket()
        self.ssock.bind((ip,port))
        self.ssock.listen(5)
        self.readers = [self.ssock]
        self.data = {}
        self.writers = []
        self.quit = False

    def pumpaccept(self):

        '''Called when server socket is readable to accept a
        connection and create a Client object.'''

        csock,who = self.ssock.accept()
        print 'Connected %s:%d' % who
        self.readers.append(csock)
        self.data[csock] = Client(who,csock,self)

    def serve(self):
        while not self.quit or self.writers:
            readable,writable,other = select.select(self.readers,self.writers,[],1.0)
            # Operate on copies of the queues since the pumpXXX() commands can modify the lists.
            if self.ssock in readable[:]:
                self.pumpaccept()
                readable.remove(self.ssock)
            for reader in readable[:]:
                self.data[reader].pumprecv()
            for writer in writable[:]:
                self.data[writer].pumpsend()

            if not readable and not writable and not other:
                print '.',

if __name__ == '__main__':
    srv = Server()
    srv.serve()

I tested this by starting the server in one console, and running the following code in other consoles to test multiple connections. Make multiple connections, alternate sends from different windows, and send partial messages to see how the server responds.

>>> from socket import *
>>> s=socket()
>>> s.connect(('localhost',9999))
>>> s.send('one\ntwo\nthree')
13
>>> s.send('\nfour\n')
6
>>> s.recv(1024)
'[one]\n[two\three]\n[four]\n'
>>> s.close()

Output should look something like:

. . . . . . . . . . . . . . . . . . . Connected 127.0.0.1:1514
. . . . . . . . . ('127.0.0.1', 1514) one
. . . . . . . ('127.0.0.1', 1514) two
. . . ('127.0.0.1', 1514) three
('127.0.0.1', 1514) four
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .