Speaker
Description
To optimize the PCI data path on the Azure MANA NIC, we implemented support for Completion Queue Entry, or CQE, coalescing. In the regular receive path, each packet can generate a separate CQE that must be processed by the driver, increasing interrupt pressure and CPU & PCI overhead. CQE coalescing reduces this cost by merging multiple CQEs into one aggregated completion that represents several received network packets. This allows the driver to process more packets per interrupt and reduces the amount of per-packet overhead required in the fast path. The improvement is especially important on ARM64 platforms, where interrupt handling cost and cache efficiency can strongly affect networking throughput. In our measurements, CQE coalescing provided a throughput enhancement ranging from 33% to 62% while also increasing the number of packets handled per interrupt.
We also added two new parameters to both the ethtool kernel API and the ethtool command line: ETHTOOL_A_COALESCE_RX_CQE_FRAMES and ETHTOOL_A_COALESCE_RX_CQE_NSECS as suggested by netdev maintainers. These parameters expose CQE coalescing through the standard Linux networking configuration model instead of relying on a driver-specific private flag. The frame-based setting controls how many packet completions may be combined before a coalesced CQE is generated, while the time-based setting controls how long the device may wait before reporting the aggregated completion. Together, they allow administrators to tune the balance between throughput, CPU utilization, interrupt rate, and receive latency for different workloads. Other NIC drivers that support similar CQE coalescing behavior can adopt the same interface, improving consistency across Linux networking devices.