Speaker
Description
Coco VMs rely on bounce buffering to use the guest's vCPUs to encrypt and decrypt data for DMA. This memory copy is performed via the SWIOTLB bounce buffer.
Persistent disk read operations handle completions within their storage interface’s interrupt handler. When SWIOTLB is disabled, the interrupt handler is invoked only after the DMA data transfer is completed. This makes the handler only responsible for quick clean up operations. With SWIOTLB enabled, the read data is currently entirely copied by a vCPU during this interrupt handler which dramatically increases handling time.
The memory copy is expensive and should not be done within an interrupt handler since it prolongs the time interrupting other tasks. This problem is exacerbated by each storage interface’s interrupt handler being pinned to run only a specific vCPU. Heavy disk read workloads can cause these pinned vCPUs to spend the entire duration of the workload in the storage interface’s interrupt handler thus starving other tasks. Google has seen cases of these workloads causing softlockups on the pinned vCPUs.
I have a prototype which defers the SWIOTLB’s memory copy out of the interrupt handler and into a workqueue. This eliminates the softlockup problem. It also allows for the memory copy to be scheduled on other vCPUs; not just the ones pinned to the interrupt handlers. For VMs with a higher vCPU count, this approach increases bandwidth and IOPS. Although, deferring the work has the drawback of slightly decreasing bandwidth for VMs with a low number of vCPUs and slightly increasing latency regardless of vCPU count.
I would like to gather feedback for my approach and discuss questions I have regarding its enablement. Is this slight latency increase tolerable? Should this feature be enabled by default for VMs with forced SWIOTLB or should the user of the guest kernel be responsible for enabling it? If we decide to enable it by default, should we only enable it for VMs with a higher number of vCPUs? If we decide the user of the guest kernel is responsible for setting it, how should they be able to set it?