Speaker
Description
After moving OpenVZ containers to cgroup-v2 we are struggling a bit to reach
feature parity with what we had before. One such feature is running nested
Docker containers inside an OpenVZ (system) container — part of making our
containers behave as close to a regular server as possible.
In cgroup-v2 the device controller was reformed drastically: device
availability can only be controlled by special BPF_PROG_TYPE_CGROUP_DEVICE
programs attached to cgroups. Docker naturally relies on this, and systemd also
employs it for its own services — so a migrated container without these
programs comes back with its device policy silently dropped. The problem is
that the BPF interface is asymmetric: the kernel accepts a program, but gives
no way to get it back in a reloadable form. The verified/JITed instructions it
can report are not portable — the verifier rewrites context accesses and helper
calls into offsets and addresses specific to the running kernel, so they can
neither pass verification again nor work on another kernel. Mainstream solved
the same problem for seccomp a decade ago — commit f8e529ed941ba ("seccomp,
ptrace: add support for dumping seccomp filters") added an API to retrieve
loaded filters specifically for C/R — but for general BPF programs no such interface
exists to this day.
We took the seccomp approach for cgroup device programs in the Virtuozzo
kernel: keep a copy of the original, pre-verification instructions at load time
and report it through BPF_OBJ_GET_INFO_BY_FD in a new
bpf_prog_info::orig_prog_insns field. The encoding (struct bpf_insn) and the
CGROUP_DEVICE context are stable UAPI, so on restore the program is simply
reloaded and the destination kernel re-verifies and re-JITs it. On top of this
API, CRIU now dumps programs and their per-cgroup attachments (via
BPF_PROG_QUERY), the bpf-prog anon-inode fds held by processes, and on restore
re-attaches everything once the cgroup tree is recreated, preserving the
original sharing topology.
In this talk we will go over the kernel and CRIU sides of the design and
discuss whether such an interface could be accepted in mainstream, along with
the open problems on the way to generic BPF checkpoint/restore: original
instructions for arbitrary program types, maps and their contents, links, and
pinned objects.