Speaker
Description
Systems that treat devices as untrusted protect themselves using confidential computing memory protection. Linux uses swiotlb to handle the necessary buffer bouncing transparently to drivers. For high-performance NICs operating at scale, however, the overhead of allocating and managing bounce buffers can substantially exceed the cost of the memory copies themselves.
Receive queues create long-lived pressure on the swiotlb pool. Pages are DMA-mapped when they enter a page_pool and normally remain mapped while they are recycled, often until they leave the pool or the queue is destroyed. Across many receive queues, these mappings can retain a substantial portion of the available bounce-buffer memory, even after the pool has been increased to the largest practical size for the system. As occupancy increases, allocations search additional swiotlb areas and contend on their per-area locks. This primarily affects transmission, where an SKB’s linear area and fragments can each require a separate bounce-buffer allocation, copy, synchronization, and reclamation sequence.
To evaluate an alternative to per-mapping swiotlb bouncing, we implemented a prototype mlx5e datapath based on preallocated DMA-coherent staging memory. With 16 transmit-heavy TCP streams, one per queue, the swiotlb datapath reduced throughput by 70% relative to a baseline without buffer bouncing. The explicit staging datapath reduced this loss to 10–20%.
In this talk, we will compare the generic swiotlb approach with the driver-managed bounce buffers, present detailed measurements, and analyze the bottlenecks we identified. We will conclude by discussing what an upstream solution should look like: driver-specific staging buffers, a generic networking abstraction or improvements to swiotlb scalability.