Speaker
Description
In 2024, hotplug on our NVMe storage fleet — NTB behind Microchip/Switchtec switches — was
not stable on mainline, so we ran Keith Busch's out-of-tree bus-locking series on a private
branch. It held. Then a new hardware generation arrived, we needed a kernel we could
validate against, and that meant mainline: 6.8, now 6.17. Going upstream meant giving up the
structural work that had been keeping us alive, and we started hitting problems again. This
is the story of that round trip, and where I think it should go next.
The good news is real, so I lead with it: since Keith Busch's ALPSS 2024 talk "The Problem
with PCI Subsystems," the subsystem has closed most of what bit us. The child use-after-free
at pci_dev_wait is fixed (11a1f4bc4736, v6.11). pciehp suppresses reset- and DPC-induced
link events (v6.16). DPC recovery holds a port reference (a1ed752bc7cb, v7.1).
reset_subordinate is hotplug-safe (8238cb69c01f, v7.1). DPC is decoupled from per-device
AER binding for v7.3 (97ca178c899d), which resolves — properly — a layering hack I had
been carrying in production. The point fixes landed.
The structural work did not come with us, and its absence is where the new problems live.
Hierarchy mutation still sits behind one global pci_rescan_remove_lock; Keith's per-bus
locking and subordinate-bus refcounting are still unmerged, two years on. Where exclusion
became coordination, the budgets do not match: pci_dpc_recovered() waits 4 s for a
recovery bounded at roughly 62 s. And a failure nobody was watching turned up — after a
switch-subtree reset, 3 of 8 endpoints cannot place a 1 GiB BAR that boot places without
complaint.
I am not here to complain; my crash was fixed and I will say so. I bring crash dumps, source
analysis, a reproducible switch topology, and fleet-scale test capacity — the validation
bandwidth the subsystem said it was short on — and three questions about the way forward. I
want to leave the room agreeing on which one is worth fixing next, and who validates it.