I need a TCP option (ioctl) to send data immediately.

Question

I've got an unusual situation: I'm using a Linux system in an embedded situation (Intel box, currently using a 2.6.20 kernel.) which has to communicate with an embedded system that has a partially broken TCP implementation. As near as I can tell right now they expect each message from us to come in a separate Ethernet frame! They seem to have problems when messages are split across Ethernet frames.

We are on the local network with the device, and there are no routers between us (just a switch).

We are, of course, trying to force them to fix their system, but that may not end up being feasible.

I've already set TCP_NODELAY on my sockets (I connect to them), but that only helps if I don't try to send more than one message at a time. If I have several outgoing messages in a row, those messages tend to end up in one or two Ethernet frames, which causes trouble on the other system.

I can generally avoid the problem by using a timer to avoid sending messages too close together, but that obviously limits our throughput. Further, if I turn the time down too low, I risk network congestion holding up packet transmits and ending up allowing more than one of my messages into the same packet.

Is there any way that I can tell whether the driver has data queued or not? Is there some way I can force the driver to send independent write calls in independent transport layer packets? I've had a look through the socket(7) and tcp(7) man pages and I didn't find anything. It may just be that I don't know what I'm looking for.

Obviously, UDP would be one way out, but again, I don't think we can make the other end change anything much at this point.

Any help greatly appreciated.

Answer 1

IIUC, setting the TCP_NODELAY option should flush all packets (i.e. tcp.c implements setting of NODELAY with a call to tcp_push_pending_frames). So if you set the socket option after each send call, you should get what you want.

Answer 2

Maybe, set TCP_NODELAY and set your MTU low enough so that there would be at most 1 message per frame? Oh, and add "dont-fragment" flag on outgoing packets

Answer 3

See if this question: will help you. http://stackoverflow.com/questions/105681/tcp-send-queue-depth

Answer 4

Have you tried opening a new socket for each message and closing it immediately? The overhead may be nauseating,but this should delimit your messages.

Answer 5

In the worst case scenario you could go one level lower (raw sockets), where you have better control over the packets sent, but then you'd have to deal with all the nitty-gritty of TCP.

Answer 6

Maybe you could try putting the tcp stack into low-latency mode:

echo 1 > /proc/sys/net/ipv4/tcp_low_latency

That should favor emitting packets as quickly as possible over combining data. Read the man on tcp(7) for more information.

Answer 7

You cannot work around a problem unless you're sure what the problem is.

If they've done the newbie mistake of assuming that recv() receives exactly one message then I don't see a way to solve it completely. Sending only one message per Ethernet frame is one thing, but if multiple Ethernet frames arrive before the receiver calls recv() it will still get multiple messages in one call.

Network congestion makes it practically impossible to prevent this (while maintaining decent throughput) even if they can tell you how often they call recv().