Speaker
Description
RDMA delivers high-throughput, low-latency networking by bypassing the kernel and letting applications communicate directly with the hardware. CRIU, by contrast, works by freezing running processes and serializing their state so they can be restored later. Bringing the two together is difficult precisely because of what makes RDMA fast: RDMA bypasses the kernel abstractions CRIU would normally use to checkpoint. CRIU already migrates live TCP connections but relies on filesystem attributes and a hook in the socket interface. RDMA is conceptually similar but the interfaces required to save/restore look very different. Our work aims to close this gap with an approach we demonstrate on NVIDIA ConnectX and BlueField devices using existing SRIOV VF migration support, as well as on RXE/Soft-RoCE. In both cases the RDMA connection survives checkpoint/restore intact: when peers are checkpointed together, the connection is never torn down and neither side sees QP errors or forced reconnects.
This matters most for machine learning, where RDMA carries communication for large distributed training and inference jobs that are expensive to start and stop. The ability to checkpoint and restore these jobs enables defragmenting a cluster to improve utilization, recovering seamlessly from hardware failures, time-sharing expensive resources between seasonal workloads (for example, inference by day and training by night), and migrating jobs to cheaper resources as availability changes. It also standardizes the save/restore workflow across frameworks, simplifying resource management for infra owners. Crucially, these jobs typically run on bare metal, so virtual machine live migration—the main existing alternative—will not be adopted by many would-be users.
To get there, the talk will first propose the concrete kernel interfaces required to support checkpoint/restore for RDMA, and explain how our implementation in CRIU uses them to checkpoint and restore a connection. Then we will then turn to mlx5, which has supported live migrating RDMA connections inside of QEMU VMs for some time. By reusing the same device capabilities and firmware APIs that already power SRIOV VM live migration, we show how realistic machine learning workloads can be checkpointed and restored on Linux using networking hardware available today.