How to debug packet loss ?

Question

I wrote a C++ application (running on Linux) that serves an RTP stream of about 400 kbps. To most destinations this works fine, but some destinations expericence packet loss. The problematic destinations seem to have a slower connection in common, but it should be plenty fast enough for the stream I'm sending.

Since these destinations are able to receive similar RTP streams for other applications without packet loss, my application might be at fault.

I already verified a few things: - in a tcpdump, I see all RTP packets going out on the sending machine - there is a UDP send buffer in place (I tried sizes between 64KB and 300KB) - the RTP packets mostly stay below 1400 bytes to avoid fragmentation

What can a sending application do to minimize the possibility of packet loss and what would be the best way to debug such a situation ?

Answer 1

This may not be the answer you want, but if I had packet loss problems I'd try to switch my application to use TCP, and have most worries of packet loss taken off my mind.

Answer 2

You should try reducing the rate you send packets. A slow connection can mean all sorts of things, and trying to send it packets (small or large) at a high rate won't help.

Answer 3

netstat has several usefull option to debug the situation.

First one is netstat -su (dump UDP statistics):

dima@linux-z8mw:/media> netstat -su                                                      
IcmpMsg:                                                                                 
    InType3: 679
    InType4: 20
    InType11: 548
    OutType3: 100
Udp:
    12945 packets received
    88 packets to unknown port received.
    0 packet receive errors
    13139 packets sent
    RcvbufErrors: 0
    SndbufErrors: 0
UdpLite:
    InDatagrams: 0
    NoPorts: 0
    InErrors: 0
    OutDatagrams: 0
    RcvbufErrors: 0
    SndbufErrors: 0
IpExt:
    InNoRoutes: 0
    InTruncatedPkts: 0
    InMcastPkts: 3877
    OutMcastPkts: 3881
    InBcastPkts: 0
    OutBcastPkts: 0
    InOctets: 7172779304
    OutOctets: 785498393
    InMcastOctets: 525749
    OutMcastOctets: 525909
    InBcastOctets: 0
    OutBcastOctets: 0

Notice "RcvbufErrors" and "SndbufErrors"

Additional option is to monitor receive and send UDP buffers of the process:

dima@linux-z8mw:/media> netstat -ua
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
udp        0      0 *:bootpc                *:*
udp        0      0 *:40134                 *:*
udp        0      0 *:737                   *:*
udp        0      0 *:mdns                  *:*

Here you need to look at Recv-Q and Send-Q column of the connection you're interested. If the values high and don't drop to zero, than the process can not handle the load.

You can use these commands on sending and on receiving machine.

Also you can use mtr, which combines traceroute and ping - it pings each hop in route. This may detect a slow hop in your route. Run it on oth machines to check connectivity to the second one.

Answer 4

Don't send out packets in big bursty chunks.

The packet loss is usually caused by slow routers with limited packet buffer sizes. The slow router might be able to handle 1 Mbps just fine if it has time to send out say, 10 packets before receiving another 10, but if the 100 Mbps sender side sends it a big chunk of 50 packets it has no choice but to drop 40 of them.

Try spreading out the sending so that you write only what is necessary to write in each time period. If you have to write one packet every fifth of a second, do it that way instead of writing 5 packets per second.

Answer 5

RTP typically uses UDP, which is inherently lossy. Packets could be lost anywhere between sender and receiver, so local debug will show you nothing useful.

Obvious things to do:

• a: Reduce the overall data rate
• b: Reduce the 'peak' data rate, by sending small packets more often rather than one huge chunk every few seconds. ie, REDUCE your UDP send buffer - maybe even to just 1400 bytes.
• c: See if you can switch to a TCP variant of RTP.

If all else fails, WireShark is your friend. It will give you a true picture of how much data - and when is being sent by your app.