Using TCP for real-time commands: Nagle arithmetic causes huge delays, what should I do?

Question

I'm writing a socket server and flash game client. The game requires real-time commands such as movement and turning. It is important for these commands to be sent by the server to the client as soon as possible because the other clients will otherwise desynchronise a lot with the moving/turning client.

This is an example of the problem caused by the Nagle arithmetic:

Note: see the command table below if you wish to understand what these commands mean.

First one is the ship I moved (moved forward + right, forward was received but right not)

The client sending commands:

84796: Sending data: 2#4
84796: Sending data: 2#2
84904: Sending data: 2#3
84904: Sending data: 2#0
86187: Sending data: 2#4
86188: Sending data: 2#2
86374: Sending data: 2#3
86404: Sending data: 2#0

The client receiving commands:

79244: Raw receive: 3#3#4$
79244: New command: 3#3#4
79398: Raw receive: 3#3#2$3#3#3$3#3#0$
79399: New command: 3#3#2
79399: New command: 3#3#3
79399: New command: 3#3#0
80635: Raw receive: 3#3#4$
80635: New command: 3#3#4
80908: Raw receive: 3#3#2$3#3#3$3#3#0$
80908: New command: 3#3#2
80908: New command: 3#3#3
80908: New command: 3#3#0

"moment" is a strange term that doesn't mean what I am trying to say, but here it seems the amount of time in milliseconds after the previous command

1. move forward send by client A (moment: 0), received by client B (moment: 0)

2. turn right send by client A (moment: 0), received by client B (moment: 155)

3. stop moving send by client A (moment: 108), received by client B (moment: 0)

4. stop turning send by client A (moment: 0), received by client B (moment: 0)

5. move forward send by client A (moment: 1283), received by client B (moment: 1236)

6. turn right send by client A (moment: 1), received by client B (moment: 273)

7. stop movement send by client A (moment: 186), received by client B (moment: 0)

8. stop turning send by client A (moment: 30), received by client B (moment: 0)

This is the command table corresponding with the commands:

Client-> Server

2# (movement info)
0) now not turning
1) now turning left
2) now turning right
3) now not moving
4) now moving forward

Server-> Client

3# (movement info)
[shipId]#
0) now not turning
1) now turning left
2) now turning right
3) now not moving
4) now moving forward

So, what you can see is that the commands are totally desynched because of "Nagle". This causes the stop movement command to be received by other clients at the same time as the start movement command, causing that player to not move at all.

This is why I need these commands to be send in real-time, as fast as possible by the TCP server. An easy fix would be to simply disable Nagle. However, I have googled (note that his suggestion about tcp message partial is implemented in my system but has nothing to do with timing) a bit and noticed that people absolutely not recommend disabling Nagle.

Is it true that I should not disable the Nagle arithmetic for this cause and should instead look for an other solution? Why (not)?

Thanks in advance. - Tom

Answer 1

You want to use TCP_NODELAY to turn off the nagle algorithm for that socket. There are settings in modern OSes to disable it on a system-wide basis, and that IS frowned upon, but there's not reason you shouldn't do it with a socket that you KNOW you need low latency on.

setsockopt(TCP_NODELAY)

Answer 2

Note that your switch, router, ISP's swithcgear.... etc may all also nagle their black little guts out. The dirty little beasts can read packets, so they take liberties and rewrite them sometimes..

Take a look at RTP ( which runs over UDP) for a more network -survivable approach.

Answer 3

If it needs to be as fast as possible, you should use UDP.

However, note that TCP provides guarantee of delivery and order whereas UDP provides none.

That point would seem to be a critically important point in this application so you would have to layer your own acknowledgement/retry command ordering mechanism on top.

If you want to stick with TCP, TCP__NODELAY will help. As your game advances you may end up requiring a stream of data that TCP will be well suited to without TCP_NODELAY.

RFC1006 formalizes a mechanism for sending packets over a TCP stream.

Answer 4

If your goal is to have the ships end up doing the same thing on everyone's screen, you're going to have to send position updates. You can't assume that just because client A did:

• Turn Left
• Wait 100ms
• Stop turning

that Client B will see that exact same 100ms delay. The Nagle buffering just makes this more extreme - but there will always be jitter and varying delays.

You really need to send something like:

• (I am at position X1, Y1 with heading Z1) Turn Left
• (I am at position X2, Y2 with heading Z2) Stop Turning

So that the clients are continually resynchronising.

Answer 5

Note that Windows implements delayed ACK as well as Nagle. This can give you an extra 200ms of delay on individual TCP messages. As far as I know, you can't change this on Windows except via a registry key (and a hotpatch to enable the registry key in some cases).