Speaker
Description
Production incidents often require answering concrete questions about current kernel state: which tasks are blocked, what is on each runqueue, where D-state tasks are stuck, or what reclaim and I/O state looks like while the host is still alive. Existing tools such as drgn and crash are powerful, but live inspection can require substantial kernel memory traversal from userspace and may be costly on an already slow host.
Ithildin’s Snapshot mode explores a different design point: pre-approved, bounded, live kernel-state capture using BPF-assisted in-kernel traversal and compact userspace output. The goal is not to replace debuggers or tracing, but to provide low-perturbation answers to common operational questions while the system is still running.
This discussion will cover early measurements comparing snapshot-style capture against drgn for task lists, D-state stacks, and runqueue state under stressed workloads. More importantly, it will ask what BPF and kernel infrastructure would make this pattern easier, safer, and faster: suitable BPF program types, iterator support, helper/kfunc availability, verifier constraints, stack capture, output buffering, adding new iterators, and defining precise safety bounds.
Discussion questions
- Is BPF iterator-based live state capture the right model for this
class of production diagnostics? - Should it use a new/dedicated BPF program type?
- What helper, kfunc, or iterator gaps make common snapshots awkward
today? - How easy should it be to add a new snapshot over a kernel structure
such as tasks, runqueues, block I/O, reclaim state, or cgroups? - What consistency guarantees are realistic for live snapshots without
stopping the world?
Expected outcome
Get feedback on whether this model is useful, what BPF/kernel interfaces would improve it and which iterators are worth standardizing