Having trouble achieving 1Gbit UDP throughput

Question

For UDP packets with a payload less then 1470, is it possible to achieve 1Gbit throughput? Due to the small packet size, there should be some bottlenecks in achieving such throughput (I/O, OS, network, etc.). I imagine drivers and hardware might have to be tuned to small packet/high throughput. Has anybody attempted successfully achieved 1Gbit throughput with small UDP packets?

Answer 1

There's a good tutorial on tweaking your network settings (in Linux) to achieve true gigabit speed here: http://datatag.web.cern.ch/datatag/howto/tcp.html

Answer 2

What type of network connection are you using? If you're using a 1000BaseTx/Fx link, don't expect more than 80% throughput with maximum sized packets. As your packet size decreases, the overhead for spacing, synchronization, Ethernet headers, IP headers and UDP headers increases in relation to the payload and therefore degrades your maximum throughput even more.

Answer 3

Check the documentation for the switch you're using. Switches are constrained in the number of packets per second (pps) they can deliver and often can't sustain 1GBps if you're sending packets with significantly smaller than the maximum payload size.

Another thing to check is whether your network card is doing interrupt coalescing, and what is the maximum number of send/receive descriptors it can support. At that level of throughput the interrupt service time and context switching time can become a big overhead on the host system even with a modern CPU and memory system.

Also if you're using gigabit over copper, the smallest ethernet frame the card will emit is 512 bytes, so smaller messages will be padded to that size. This is because of requirements for carrier sense/collision detection.

Answer 4

I've previously done some experimenting with throughput on gigabit links on relatively standard pc hardware, albeit doing just transmits (via tcpreplay), rather than udp.

The biggest bottleneck that I found was in just getting packets to the NIC itself. This can be significantly improved by using a high speed bus to interface to your NIC (eg. a 4x pci-express NIC). But even with this there was a very definate packet/second limit. Obviously increasing the packet size would allow you to utilize more of your bandwidth while reducing processor load.

Along the same lines as the comment by Steve Moyer, there is a theoretical limit for the utilization of any network. In my experiments (which were being done on a completely quiet network) I was seeing a maximum of approximately (and only off the top of my memory) 900Mb/s. This was with cpu loads of 30 to 40%.

It's more likely that the limitation is going to be imposed by your system hardware (ie. PC) than your network infrastructure - any network switch worth its salt should be capable of sustaining full speed network access with small packets - certainly at much higher rates than most PCs can cope with.

Answer 5

I've found hardware has a significantly lower packet-per-second limit than the networks theoretical capacity. For a Broadcomm BCM5704S I hit this at 69,000 pps compared to 1,488,100pps of gigabit.

Some more numbers I reported here, http://code.google.com/p/openpgm/