Design question on Python network programming

Question

I'm currently writing a project in Python which has a client and a server part. I have troubles with the network communication, so I need to explain some things...

The client mainly does operations the server tells him to and sends the results of the operations back to the server. I need a way to communicate bidirectional on a TCP socket.

Current Situation

I currently use a LineReceiver of the Twisted framework on the server side, and a plain Python socket (and ssl) on client side (because I was unable to correctly implement a Twisted PushProducer). There is a Queue on the client side which gets filled with data which should be sent to the server; a subprocess continuously pulls data from the queue and sends it to the server (see code below).

This scenario works well, if only the client pushes its results to the manager. There is no possibility the server can send data to the client. More accurate, there is no way for the client to receive data the server has sent.

The Problem

I need a way to send commands from the server to the client.

I thought about listening for incoming data in the client loop I use to send data from the queue:

def run(self):
    while True:
        data = self.queue.get()
        logger.debug("Sending: %s", repr(data))
        data = cPickle.dumps(data)
        self.socket.write(data + "\r\n")
        # Here would be a good place to listen on the socket

But there are several problems with this solution:

• the SSLSocket.read() method is a blocking one
• if there is no data in the queue, the client will never receive any data

Yes, I could use Queue.get_nowait() instead of Queue.get(), but all in all it's not a good solution, I think.

The Question

Is there a good way to achieve this requirements with Twisted? I really do not have that much skills on Twisted to find my way round in there. I don't even know if using the LineReceiver is a good idea for this kind of problem, because it cannot send any data, if it does not receive data from the client. There is only a lineReceived event.

Is Twisted (or more general any event driven framework) able to solve this problem? I don't even have real event on the communication side. If the server decides to send data, it should be able to send it; there should not be a need to wait for any event on the communication side, as possible.

Answer 1

"I need a way to send commands from the server to the client."

Don't do this. It inverts the usual meaning of "client" and "server". Clients take the active role and send stuff or request stuff from the server.

Is Twisted (or more general any event driven framework) able to solve this problem?

It shouldn't. You're inverting the role of client and server.

If the server decides to send data, it should be able to send it;

False, actually.

The server is constrained to wait for clients to request data. That's generally the accepted meaning of "client" and "server".

---

"One to send commands to the client and one to transmit the results to the server. Does this solution sound more like a standard client-server communication for you?"

No.

If a client sent messages to a server and received responses from the server, it would meet more usual definitions.

Sometimes, this sort of thing is described as having "Agents" which are -- each -- a kind of server and a "Controller" which is a single client of all these servers.

The controller dispatches work to the agents. The agents are servers -- they listen on a port, accept work from the controller, and do work. Each Agent must do two concurrent things (usually via the select API):

• Monitor a well-known socket on which it will receive work from the one-and-only client.

• Do the work (in the background).

This is what Client-Server usually means.

If each Agent is a Server, you'll find lots of libraries will support this. This is the way everyone does it.

Answer 2

"I don't even know if using the LineReceiver is a good idea for this kind of problem, because it cannot send any data, if it does not receive data from the client. There is only a lineReceived event."

You can send data using protocol.transport.write from anywhere, not just in lineReceived.