Linux Socket Buffer Imbalance

Question

I have a simple scenario, where two servers are connected through a gigabit link. I run iperf on both sides to measure the throughput.

What surprises me, whenever I run the traffic bidirectionally, it always favor one side only (eg. ~900Mbps vs. ~100Mbps). If I run the traffic unidirectional, each side got ~900Mbps.

If I connect one of the server (the lower memory) to another server, the bidirectional traffic was balanced out. So definitely not the iperf problem.

Other facts:

• One server has pretty big memory (~12GB), while the other only has ~4GB.
• Both server have the same TCP memory configurations, in this case, the same TCP w/r mem, core w/r mem, TX queue length.
• Both are using the same ethernet card (E1000 driver).
• Same Linux version, RedHat 2.6.9. The big server runs 64-bit version, due to the 12GB memory.
• Both have no other traffic, beside small SSH, and occasional ping every second.
• Both have "tcp_moderate_rcvbuf" on.

Questions:

1. Why the imbalanced?
2. Which area should I look to see if the socket buffer is heavily utilized on one side, and how?
3. Beside iperf, what are other good software (not hardware/tester) to measure the performance?
4. What's the best way to get an insight on how Linux allocating the buffer, from the Ethernet ring buffer, TCP buffer, socket buffer, and other buffers?
5. What could be the other entity that might impact the throughput that I haven't covered above?
6. Is there any documentation that explains how Linux distributes the memory allocation between user, kernel, device drivers, and network stack?

Any advise is deeply appreciated.

Answer 1

Suggestions:

• Take a look at the actual settings on your Ethernet interfaces. "ethtool" is one way to get a thorough look. "ifconfig" tells you something, though less. (Both probably in /usr/sbin/.) Finding kernel messages with "dmesg" might tell you something. Looking at link error rates might reveal something too.
• Querying your switch for its idea of port state might also reveal what's really going on. (Not relevant if you're just using a CAT5 cable between interfaces, without a switch.)
• Since one pair of machines works as you expect, while another pair of machines doesn't, I'm thinking about some anomaly with duplex autonegotiation. Half-duplex is unusual for GigE but perhaps your switch or NIC is causing it. Discovering a half-duplex setting anywhere, or especially a disagreement between a host and its switch about port state, could be possible cause.

Answer 2

still sounds like a duplex mismatch to me.