Speaker
Description
CoreWeave is a bare-metal cloud provider. A typical machine in our fleet has at least 2TB of RAM, and customers tend to run close to that limit. That has two immediate consequences: the crash-kernel memory reservation matters a lot (every GB we reserve is a GB the customer doesn't get), and when a machine panics the resulting crash dump is huge. Because it's so big, the downtime during a crash, and how soon we can hand the machine back to the customer, ends up gated almost entirely on makedumpfile. At the fleet level it gets even more challenging as bugs triggered by a bad kernel upgrade or some fancy customer workload lead to getting tons of crash dumps at once and all of them are due to the same bug. Finally there’s us, a lean team of 2 kernel engineers at CoreWeave that have to look at all the incoming crash dumps. We lean on AI a lot, at least for triaging failures and knocking out the easy RCAs. That field has progressed and gotten more accurate over time but we still get the occasional confident hallucination.
Our ideas and projects so far:
-
(Project in-progress at the time of submission) Fingerprint the failure before we even try to capture a dump. Compute a cheap signature of the crash from within the crash kernel, compare it against what we've already seen, and if it's a known one, skip collection altogether. (This probably needs a network connection out of the crash kernel to check against a central store, and we're not sure yet how painful that is to pull off)
-
(Idea) Decouple capturing the crashed memory from turning it into a proper dump. Instead of running makedumpfile to completion before we reboot, grab the crashed memory (maybe with some very light filtering), reboot early to give the machine back, and do the heavy transform/filter later. (There may be Kexec HandOver-related discussion here).
-
(Idea) An alternative on the above that we're weighing is to skip the general-purpose dump entirely and use drgn+sdb to take ultra-filtered dumps on the spot, driven by a preconfigured list of data structures we care about.
-
(Project in-progress at the time of submission) Use out-of-band kernel introspection from the DPU to grab ultra-filtered dumps the moment we know a panic happened, without even waiting for the crash kernel to come up (we got drgn+sdb on the DPU reading host memory over RDMA).
-
(Implemented) For the AI triaging/RCA side, hand the agent a Docker container preloaded with the crash dump plus matching debug info, the kernel source for that exact version, and some hints on where to look for existing issues (bug trackers, the mailing list, etc.). We also drop in a small agents.md whose main job is to keep the agent from spinning in endless loops.
Topics we'd like to discuss:
-
How do folks capture dumps on multi-TB hosts at fleet scale? Full vs. aggressively filtered, local NVMe vs. a network target, and what capture times are people living with? And how do you settle on the crash-kernel memory reservation?
-
Is anyone else deduplicating crash dumps? If so, what crash-signature/dedup schemes are you using, and is there any appetite for converging on a shared, drgn-based triage library instead of everyone rolling their own?
-
How are people using AI tools for crash dump analysis, and which setups have worked best? How do you keep the confident hallucinations in check?
-
Is anyone else doing out-of-band introspection of production hosts (DPU, BMC, PCIe P2P, RDMA)? How do you handle it, and do you use drgn?
-
How do you keep historical crash dump data around? Jira and/or database plus Grafana? Something else?