Aggregated throughput of multiple connections (Linux)

Question

Why do multiple connections from the same host achieve better throughput than multiple connections from different hosts?

I suspect is something kernel-related (Linux) but a confirmation would help a lot.

More details I have 1 receiver process, let's call it R. It accepts incoming connections and receives data using select().

I have 3 sender processes S1, S2, S3. They connect to R and send data at a fixed rate, i.e. 200Mbit per second each.

If S1, S2 and S3 are on the same machine, I get better results than having each one of them on a different machine. (R is in both cases on some other machine)

Example: R on host0, S1, S2, S3 on host2, R receives at 600Mbit/s

R on host0, S1 on host1, S2 on host2, S3 on host3, R receives at 480Mbit/s

This looks counter-intuitive for me, I expected the opposite since in the second case senders don't have to share the network card and the processor (Not that expect processor or network card to be bottlenecks...)

[The hosts above are nodes in a linux cluster with a dedicated full-duplex Gigabit switch. They are running 2.6.24-24-generic (latest Ubuntu i guess)]

Answer 1

This is probably because when the senders are all on one machine, the outgoing packets are all nicely queued and sent one-at-a-time (just by virtue of the fact that they're all going through one NIC).

Whereas in the multiple-senders case, two machines will often send packets at the same time, and it is then up to the network switch to queue them. This will manifest itself to the sending TCP as a jittery latency - sometimes a packet will be switched straight through to the receiver, other times it will have to wait for one or two packets from the other senders inside the switch queues.

I expect that the latency jitter on its own would be enough to knock that much off your bandwidth - remember that to sustain 200mbps with standard TCP windows, you need a minimum round-trip-time of 2.6ms, which is pretty tight.