The same driver for multiple network cards -- performance bottleneck?

Question

I'm using driver e1000e for multiple Intel network cards (Intel EXPI9402PT, based on 82571EB chip). The problem is that when I'm trying to utilize maximum speed (1GB) on more than one interface, speed on each interface starts to drop down.

I have my own driver in kernel space designed to just sent given packets. It just allocs packets by:

skb = dev_alloc_skb(packet->len);

and them sends them by:

result = dev->hard_start_xmit(skb,dev);

Each interface has its own instance of the driver.

For one interface I get: 120435948 bytes/sec.

For two interfaces I get: 61080233 bytes/sec and 60515294 bytes/sec.

For three interfaces I get: 28564020 bytes/sec, 27111184 bytes/sec, 27118907 bytes/sec.

What can be the cause? Is the hard_start_xmit function reentrant?

Answer 1

This is most likely due to a lack of bandwidth over your motherboard.

If you're trying to pump 3 Gb/s of information through a bus slower than 3 Gb/s, you'll have problems. What sort of bus are these cards on?

There may be a fix, but I think this is a physical limitation of the board, not necessarily your driver.

Answer 2

When I add the numbers together for 2 interfaces, the net result is slightly bigger than the output for a single interface. To me, this means the system is being slightly more efficient when using both interfaces. One possible reason might be better CPU utilization or possibly bus utilization. But note that the result is only slightly better and probably indicates that the resource causing the bottle neck is limited to 121MB/s. Once the load (3 active interfaces) exceeds this limit, performance drops dramatically to 82MB/s.

It is hard to pin down the exact cause without some additional measurements, but my guesses would be

1. CPU limited : Adding multiple CPU to the system would rule this out as a problem.
2. Memory limited : Remember that even if the device is in a x4 or x8 slot, the connection to main memory (i.e. where the SKB live) may not be able to sustain that load.
3. Interrupt limited : The packets per second might be high enough that switching in and out of interrupt contexts is hurting performance. This is less likely as most drivers are good about interrupt coalescing, but if possible, switch the driver to a polled mode to rule this out.