Multicast Send Performance

Question

We recently completed an analysis of multicast sending performance. Happily, Java and C performed almost identically as we tested different traffic sending rates on Windows and Solaris.

However, we noticed that the time to send a multicast message increases as the time between sends increases. The more frequently we call send, the less time it takes to complete the send call.

The application lets us control the amount of time we wait between calling send, below you see the time increasing as the delay between packets goes up. When sending 1000 packets/second (1 ms wait time), it only takes 13 microseconds to call send. At 1 packet/second (1000 ms wait time), that time increases to 20 microseconds.

Wait time (ms)                      us to send
0                                   8.67   
1                                   12.97  
10                                  13.06  
100                                 18.03  
1000                                20.82
10000                               57.20

We see this phenomenon from both Java and C and on both Windows and Solaris. We’re testing on a Dell 1950 server, with an Intel Pro 1000 dual port network card. Micro-benchmarking is hard, especially in Java, but we don’t think this is related to JITing or GC.

The Java code and the command line I'm using for the tests are at: http://www.moneyandsoftware.com/2009/09/18/multicast-send-performance/

Answer 1

Some theories:

My first thought was that I would consider caching to be a factor here - if the tasks and values are still on the stack or in recent short-term memory, you may find it is able to send them faster. As time increases, the chance it's still available decreases, so it would on average take longer.

However, I would expect there to be an upper limit if this were the case...some point at which it's always in cache.

An alternative reasoning is that there's a memory leak or some performance degradation over time in your app/test/platform. This would also (if it exists) mean that the longer you wait, the longer it has time to degrade the performance, and thus the longer it would take to send.

ALSO - if you're taking longer between packets to send them - you may exceed the address learning timeouts - both IP tables and MAC tables. If these tables/caches have expired, they'd need to relearn them before forwarding the packet on.

Good luck!

Answer 2

The code to perform those tasks is cached closer to the CPU (maybe even still in the registers) when a call just occurred.

Answer 3

It might be an artifact of interrupt coalescing with the NIC on that particular host, check this article by 29 West on the topic, they show how latency can increase to 125μs on an e1000 NIC,

http://www.29west.com/docs/THPM/latency-interrupt-coalescing.html

Answer 4

How are you waiting between sends? Have you tried busy waiting so you don't give up the CPU?