Speakers
Description
Motivation:
Kexec Hand-Over (KHO) helps minimize downtime during kernel updates by preserving state across the kexec reboot. However, there are some scenarios which need a deeper form of reboot to be effective -- for example, performing a firmware update or a device-tree refresh, or even to avoid known hardware/platform bugs that make the kexec path itself unreliable. In such cases, the kernel's "warm reboot" facility could be leveraged, which performs a full reboot (Firmware + Bootloader + OS/kernel init), but with the guarantee that this entire flow leaves DRAM contents intact (distinct from a "cold reboot", where DRAM is also cleared). We would like to extend KHO (and its memory state preservation feature via LUO & memfd, in particular) to also work across warm reboots, so as to minimize downtime during update scenarios that necessitate a warm reboot.
Key Challenges:
When trying to extend KHO/LUO/memfd to warm reboots, we ran into 2 key technical challenges, described below. We were able to solve the first one (supported by a working prototype), but we need the community's insights & feedback to help solve the second challenge (our proposed solution approaches are outlined below).
Challenge 1: Discovering preserved KHO state in the next kernel (without passing an FDT)
During a kexec, the handover from the first kernel to the second one is explicit -- by passing an FDT with info embedded as to where to find the preserved KHO state in memory. However, in the case of a warm reboot, there is no opportunity for the first kernel to do a similar hand-off, and yet, the second kernel should be able to discover where the KHO state was preserved, in order to restore it.
We solved this problem by using a classic Computer Science technique -- i.e., by adding a level of indirection :-). While the KHO state can be allocated & preserved anywhere in memory, the KHO metadata info is kept in a small header and it is passed to the next kernel. On UEFI platforms an EFI variable can be used for this purpose, whereas on DT based platforms, the header can be kept in a pre-determined location, and a DT node added to inform the kernel about it. This retains the KHO allocation flexibility. We also added checksums to verify the integrity of the preserved metadata to make this mechanism robust against stale/circumstantial data left in that fixed location.
Challenge 2: Preventing firmware from clobbering preserved KHO state across a warm reboot
The warm reboot flow includes all the usual init sequence across components like firmware and bootloader, leading up to launching the Linux kernel. The memory used by these components before Linux is launched can be termed as "FW-Transient", as they are only needed during that transition period. One common optimization technique used on memory-constrained systems is to allow firmware to temporarily re-use the same memory region that Linux has access to, but only when the kernel isn't running. Thus, FW-Transient memory during a warm reboot transition intentionally overlaps with Linux's usable memory, to avoid wasted permanent memory carveouts for firmware's temporary use. However, this introduces a problem for KHO, where the firmware can trample on memory regions that the kernel expects to pass unmodified to the next kernel across a warm reboot.
Delving deeper into the platform/firmware constraints around the use of FW-Transient memory, we found that while the memory maps vary widely between platforms, they do remain fixed (pre-determined at design time) for a given platform. This implies that while we can't move FW-Transient memory to cooperate with KHO, KHO can instead discover the boundaries of this region to avoid saving state-to-be-preserved in that region. But the exact mechanism for how to demarcate and export that FW-Transient region info from the firmware to the kernel across a variety of platforms (UEFI vs DT) remains an open question.
Using this insight, we have 2 solution approaches that we would like to propose to solve this challenge. The first one is to influence the sizing and placement of the kho-scratch region such that it fully overlaps with FW-Transient memory. If feasible, this would be an elegant solution because KHO won't have to do anything else to prevent firmware from clobbering its preserved state, as the kho-scratch region is meant for non-preserved allocations -- by definition. However, if there are any memory-management constraints that prevent kho-scratch placement in this way, we could also explore migrating preserved KHO state out of the FW-Transient region just before the first kernel's shutdown. This latter approach has its own potential pitfalls, such as difficulties in maintaining KHO state integrity post page migration and the cost of migrating pages in a performance sensitive path (i.e., the need for fast reboot to minimize downtime).
We would like to present these challenges and solution approaches and brainstorm with the Linux kernel, Firmware, and Bootloader communities at LPC on the best approach to solve this problem to enable KHO across warm reboots.