How to correctly relay TCP traffic between sockets?

Question

I'm trying to write some Python code that will establish an invisible relay between two TCP sockets. My current technique is to set up two threads, each one reading and subsequently writing 1kb of data at a time in a particular direction (i.e. 1 thread for A to B, 1 thread for B to A).

This works for some applications and protocols, but it isn't foolproof - sometimes particular applications will behave differently when running through this Python-based relay. Some even crash.

I think that this is because when I finish performing a read on socket A, the program running there considers its data to have already arrived at B, when in fact I - the devious man in the middle - have yet to send it to B. In a situation where B isn't ready to receive the data (whereby send() blocks for a while), we are now in a state where A believes it has successfully sent data to B, yet I am still holding the data, waiting for the send() call to execute. I think this is the cause of the difference in behaviour that I've found in some applications, while using my current relaying code. Have I missed something, or does that sound correct?

If so, my real question is: is there a way around this problem? Is it possible to only read from socket A when we know that B is ready to receive data? Or is there another technique that I can use to establish a truly 'invisible' two-way relay between [already open & established] TCP sockets?

Answer 1

I don't think that's likely to be your problem.

In general, the sending application can't tell when the receiving application actually calls recv() to read the data: the sender's send() may have completed, but the TCP implementations in the source & destination OS will be doing buffering, flow control, retransmission, etc.

Even without your relay in the middle, the only way for A to "consider its data to have already arrived at B" is to receive a response from B saying "yep, I got it".

Answer 2

Is it possible to only read from socket A when we know that B is ready to receive data?

Sure: use select.select on both sockets A and B (if it returns saying only one of them is ready, use it on the other one), and only read from A and write to B when you know they're both ready. E.g.:

import select

def fromAtoB(A, B):
    r, w = select.select([A], [B], [])
    if not r: select.select([A], [], [])
    elif not w: select.select([], [B], [])
    B.sendall(A.recv(4096))

Answer 3

If you're using blocking operations (not async), you'll probably stumble on many problems. My advice is to use asynchronous IO, and for that I suggest using pyevent (python bindings for libevent).

IMO, your concern makes sense if there's a very high throughput, which may cause a general slowdown. In this extraordinary situation, your application could miss packets due to full cache, and retransmission (protocol dependent) is very likely to occur. Another possibility for the same scenario, is that endpoint applications with strict timeout handling might behaviour differently.

Additionally, a common problem I see in this kind of application is a missing flush upon disconnect. For example, if endpoint A sends data and disconnects right after that, you must send this data to endpoint B prior to closing the connection with B.

Answer 4

Perhaps the application you're proxying is poorly written.

For instance, if I call recv(fd, buf, 4096, 0); I'm not promised 4096 bytes. The system makes a best-effort to provide it.

If 1k isn't a multiple of your application's recv or send sizes, and the application is broken, then grouping the data sent into 1k blocks will break the app.