# sched_ext Microconference - LPC 2026 Report The sched_ext microconference took place on October 7th, 2026 and featured six talks, preceded by an opening discussion on the future direction of sched_ext. Session page: [https://lpc.events/event/20/sessions/270/](https://lpc.events/event/20/sessions/270/) Three main topics dominated the discussions: - **Priority inversion and how to handle it in sched_ext**: lock holder and lock waiter preemption, time slice extension, and making proxy execution work together with BPF schedulers. - **Paravirtualized scheduling**: letting the host and guest schedulers cooperate across the VM boundary, with sched_ext as the policy engine on the host. - **Device and topology-aware scheduling**: using hardware feedback (Intel HFI), device/accelerator locality and cache topology to drive task placement. A recurring theme across all of them was the need for the kernel to expose the right information (lock ownership, guest priorities, hardware hints, task-to-device associations) so that BPF schedulers can make better decisions, while keeping the policy itself in BPF. ## Intro: where sched_ext is going [Video](https://www.youtube.com/watch?v=4vYPuuy4T0M) Instead of the usual status update, the intro briefly outlined the general direction of the project: - **Move more into BPF.** With BPF arenas, queuing and idle CPU management can be implemented entirely in BPF, and several schedulers are already converging on this model. Once they all move, a significant amount of in-kernel sched_ext code can be removed. - **Composability.** Long discussed, now a good time to tackle it: reusable BPF/user-space building blocks (load balancer from one scheduler, idle CPU selection from another, etc.) that can be combined into new schedulers. - **Rust everywhere.** Most schedulers today are Rust in user space plus C for the BPF part. The trend, aligned with the wider BPF community, is to generate the BPF code from Rust as well, so a single language and shared abstractions can be used to access scheduling data structures from both sides. Prototypes exist but this is not there yet. ## Priority inversion ### Making proxy execution compatible with sched_ext *Andrea Righi (NVIDIA), John Stultz (Google)* - [Video](https://www.youtube.com/watch?v=utS74Dvhp_M) John recapped proxy execution as "scheduler-led priority inheritance": instead of propagating priorities (which is easy for RT but very hard for DEADLINE and fair), mutex-blocked tasks stay on the runqueue, and when the scheduler picks one, the blocked-on chain is walked to find the owner, which runs using the donor's scheduling context. If the owner is on another CPU, the donor is "migrated" to the owner's runqueue for selection and accounting purposes only, it never actually runs there. Until now `CONFIG_SCHED_PROXY_EXEC` and `CONFIG_SCHED_CLASS_EXT` were mutually exclusive, since BPF schedulers could not be made aware of the split donor/owner context. **Both can now be enabled together; the change is queued for 7.4.** This is particularly relevant for distributions, which can now ship both and let users experiment with either independently. Design decisions: - Proxy support is **opt-in per scheduler via a flag**. Schedulers that set it receive mutex waiters in `ops.enqueue()` (marked as blocked enqueues) and can decide what to do with them, e.g. prioritize them. Schedulers that do not set it see waiters simply block as before. - Proxy execution remains driven by the scheduler: nothing happens unless the BPF scheduler actually dispatches the blocked task. Dispatching it is the "admission" that triggers the core proxy machinery. The BPF scheduler stays in control. - sched_ext reuses the core proxy machinery, so improvements there (e.g. sleeping owner handling, future futex support) are automatically inherited. - It works across scheduling classes: e.g., an RT donor blocked on a sched_ext owner works transparently, and the BPF scheduler sees nothing because the owner runs within the RT task's scheduling context. At the time of the talk, the implementation generated several unnecessary `ops.enqueue()` calls during proxy migrations (switching to idle, removing from the original CPU, adding to the owner CPU), and the BPF scheduler could see a blocked task enqueued on the owner's CPU, outside the waiter's affinity mask (harmless, since the waiter never runs there, but confusing for schedulers making strong per-CPU assumptions). The plan presented was to collapse these into a single enqueue so proxy bookkeeping is fully transparent to BPF; this improvement is now also queued for v7.4. Results were shown for scx_cosmos with an artificially created mutex priority inversion scenario, where enabling proxy execution resolved the inversion and lowered latency. Simply enabling it does not bring improvements across the board, though: the benefits show up mainly where tasks are deliberately biased (e.g., restricting background work in Android), or in cases like Steam updating Vulkan shaders while gaming, where a heavily contended mutex otherwise makes the whole system laggy. Discussion points: - **User-space locks**: futexes are not covered since `FUTEX_WAIT` does not identify the owner. A new futex op that carries ownership is being explored; passing user-space owner hints (e.g., via BPF maps or futex flags) is inherently racy. A "directed yield" (timed proxy to a specific task) was mentioned as future work. - **rwsems / semaphores**: writer boosting is supported via a patch; boosting multiple readers is harder and has overhead. Plain semaphores have no owner and cannot be proxied. - **Observability**: there are no upstream tracepoints for proxy execution yet, which makes traces confusing (the owner runs under the donor's context). Something similar to the existing PI tracepoints was requested. - **Accounting**: vruntime is charged to the donor, but CPU time is credited to the task actually running (so CPU timers fire correctly), while throttling must apply to the donor. ### Taming latency spikes caused by lock holder and lock waiter preemption *Changwoo Min (Igalia)* - [Video](https://www.youtube.com/watch?v=aj72z7FLxLk) Changwoo distinguished two related problems: - **Lock holder preemption**: the lock owner is preempted while in its critical section, stretching it. - **Lock waiter preemption**: the owner releases the lock and wakes the waiter, but the scheduler does not consider the waiter urgent, so the lock is logically handed over but the critical section is still effectively stretched. The motivating case is the Steam Frame VR headset, which runs Linux with scx_lavd. Its tracking cameras produce 30 frames/s, so each frame must be processed within 33ms, using only ~7-8ms of CPU time. But AAA games often push CPU utilization to 80-95%, and the frame processing pipeline is split into small stages coordinated through pthread mutexes, condition variables (futexes) and SysV semaphores. A few occurrences of lock holder/waiter preemption within a single frame are enough to miss the deadline, producing wrong head-position estimates and visible glitches. The problem is generic: average latency and throughput are barely affected, but tail latency (P99) is destroyed. Making the pipeline RT or SCHED_DEADLINE was tried and did not work well: the work is not latency-critical in itself, RT preempted fair tasks that were more latency-sensitive, and too many RT tasks ended up competing with each other. Three existing approaches were reviewed: 1. **Extend** ("I hold a lock, give me a bit more time"): the rseq-based time slice extension now in mainline (works with both EEVDF and sched_ext, but targets very short user-space critical sections), and a more generous LAVD variant that traces futex wait/wake slow paths and grants one extra tick to a task that likely owns a futex. 2. **Donate** ("I'm blocked on your lock, run on my slice"): proxy execution. Powerful, but reactive. 3. **Run the waiter immediately** when the lock is handed over: this was the missing piece. LAVD now hooks futex and SysV semaphore wakeups and boosts the latency criticality of the woken waiter, giving it an earlier virtual deadline. With LHP/LWP mitigations, LAVD significantly reduced scheduling delays above 2ms across four games, and no missed camera frames were observed during one hour of gameplay in the tested titles. The open question is how to make this less ad hoc. The scheduler needs to know how many locks a task holds, whether it is waiting on a lock, and whether an extension was granted, for both kernel and user-space locks. Options discussed: instrumenting all kernel locks, the recently posted lock holder tracking series (opt-in tracking of hand-picked kernel locks, motivated by memcg lock holder preemption), the existing `contention_begin/end` tracepoints (imprecise semantics), and extending the rseq-based shared memory approach to more user-space lock types. From the discussion: - Abuse of the extension is bounded by fairness: a task that constantly extends its slice is deprioritized. - Why not just rely on proxy execution? Proxy is reactive (owner scheduled out, donor scheduled in, owner scheduled in again), with extra overhead. The consensus was that the two approaches are complementary: slice extension is like a spinlock (cheap, good for short critical sections), proxy execution is like a mutex (heavier, but handles long ones). ## Paravirtualized scheduling ### Bridging the VM Boundary: Scheduling Passthrough via pvsched and sched_ext *Josh Don, Vineeth Remanan Pillai (Google)* - [Video](https://www.youtube.com/watch?v=CrGicO8tFhs) From the host's point of view, a VM is just a set of opaque vCPU threads: the host scheduler has no idea whether a vCPU is running an RT audio thread, an IRQ handler or a SCHED_IDLE batch job. This leads to priority inversion and poor arbitration of CPU time, especially in overcommitted environments. The work started in ChromeOS (Android apps run in a VM, and latency-sensitive Android threads boosted inside the guest were still preempted on the host, causing UI jitter) and is now also used for large, densely-packed VMs in Google's fleet. Unlike approaches that only push host information into the guest, pvsched is **bidirectional**: - Guest to host: which threads are currently on each vCPU, their priority, cgroup information, "don't preempt me" signals, etc. - Host to guest: what the host can actually provide, e.g. "right now I can only run vCPUs 0-3". Architecture (v3, not yet posted): earlier versions put the scheduling logic (v1) and then the negotiation (v2) in KVM, which KVM maintainers rejected. In v3 negotiation happens in the VMM through a virtual **pvsched PCI device**: the host advertises supported policies, the guest picks one and allocates per-vCPU or global shared memory (guest-initiated, so no on-the-fly memslot creation is needed), and a pvsched kernel module sets up the shared region and exposes hooks (VM entry/exit, sched_switch, IRQ/NMI entry, preemption disable). Policies can be kernel modules or BPF struct_ops programs. Abuse prevention is policy-specific. The default policy throttles a vCPU boosted for more than a tunable time (500ms by default) and then applies a cooldown. CAP_SYS_NICE is required on the VMM side, so creating a VM does not grant RT privileges. An audience member from the KVM side agreed that keeping all of this policy out of KVM was the right call, and that host actions must never be more privileged than what a normal task could do. Where sched_ext comes in on the host: - Bounded boosting of vCPUs that request it, beyond what their accumulated vruntime/lag would allow under EEVDF. - **Gang scheduling** of a VM's vCPUs across CPUs, which helps a lot with spinlocks and IPIs inside the guest and does not map well onto EEVDF. - Global boosting independent of the cgroup hierarchy, via custom auxiliary queues. And optionally in the guest: **vCPU funneling**, concentrating work on the vCPUs the host can actually run concurrently (reducing VM exits and vCPU-to-vCPU switches under host contention), and ideally letting the host make most decisions, down to which guest thread to run, to reduce the double-scheduling penalty. The same shared-memory channel could also be used for other BPF-based host/guest policies, e.g. BPF-driven OOM victim selection across memcgs inside VMs. Discussion: - **BPF arenas as an alternative** to a dedicated device: a host scheduler could allocate an arena, hand out per-guest chunks and get zero-copy communication with no kernel changes, needing only a way to pin arena memory. Someone from the audience pointed out that allocating memory on the host and mapping it into the guest on the fly is slow (memslot creation, guest notification), which is why guest-initiated allocation was chosen. However, this cost can be avoided: the host BPF scheduler can create the arena once when it is started and hand out per-guest chunks from the preallocated arena, mapping each chunk when the guest is set up rather than on the fly, which keeps the fast path cheap. It was also noted that the device model keeps the handshake arch-agnostic and evolvable without kernel changes. - There is related work from others in this area; the agreement was that the various efforts should converge on a single kernel module, possibly alongside a BPF-arena-based path. - Standardization: post the kernel module first, then the QEMU device, then other VMMs (crosvm, Cloud Hypervisor). Virtio was suggested as a possible transport. - Relation to the preferred-CPUs / steal time work (Shrikanth Hegde): related, but reactive, while pvsched allows an explicit policy. ## Device and topology-aware scheduling ### Automatic placement of accelerator workloads *Balbir Singh, Lee Trager (NVIDIA)* - [Video](https://www.youtube.com/watch?v=aW_MdW_mf6g) On multi-socket, multi-GPU systems, performance is often limited by CPU/GPU locality. Customers are typically told to use `numactl` pinning and to disable NUMA balancing, which is static and requires the admin to know the topology and device usage in advance. scx_cosmos already places tasks near their GPU, but relies on NVIDIA's proprietary NVML library, so it is vendor-specific and has no visibility into host memory placement. The proposal is a generic feedback loop for any accelerator or PCI device (GPU, NIC, NVMe): 1. The framework (transparently to the application) tells the scheduler which device it is about to use. Merely opening a device is not a good signal: many applications open all devices to query them and then use only one. 2. The sched_ext scheduler migrates the task to the device's node. 3. Telemetry (PMU-based remote access counters, bandwidth) is fed back, and either the application calls `move_pages()` or NUMA balancing moves the memory. A proof of concept (LD_PRELOAD hooking the accelerator launch call, six pinned PMU events) showed that running on the wrong node costs about 67% in bandwidth, and that correct placement eliminated remote access events. In internal tests, removing `numactl` from some customer workloads halved performance, which is what this approach aims to make unnecessary. Two kernel pieces are proposed: - **BPF-driven NUMA balancing**: NUMA balancing is currently disabled whenever a sched_ext scheduler is loaded. A series enabling it was recently posted, along with a kfunc letting a BPF scheduler request a preferred memory node and influence page migration (the scheduler requests, it does not migrate pages itself). - **Process-to-device association**: a generic way to know which devices a process uses. Open questions and discussion: - Exposing PMU telemetry to non-root applications raises side-channel concerns; IPC is fairly safe, cache/TLB misses are a gray area. An allowlist of audited events was suggested. - Is a BPF map the right interface, or is a common user-space library needed so all frameworks can provide hints? - Migrating network-heavy tasks across nodes may interact badly with per-CPU queues and TCP flows. - Gregory Price (mm NUMA placement maintainer) found the BPF extension compelling but warned that NUMA balancing itself needs fixes first (async migration, charging the work to the right task), to avoid "piling hacks on hacks". Since enabling it under sched_ext reuses the existing mechanism without changing it, that part can proceed in parallel, with `move_pages()` driven by telemetry as an interim solution. ### HFI plug-in for sched_ext *Srinivas Pandruvada (Intel)* - [Video](https://www.youtube.com/watch?v=kWyQb-rRmzM) The goal was to match Windows gaming performance on a hybrid Intel handheld running Linux with scx_lavd. Intel's Hardware Feedback Interface (HFI) is not new (productized on servers years ago), but it has never been used on clients. The firmware/P-unit has a global view of the power budget across CPU, graphics and other subsystems, and emits sparse hints about per-CPU performance and efficiency capability, including "do not schedule here". Hints can be ignored, but the hardware then reacts more aggressively and eventually throttles. Results with Shadow of the Tomb Raider at 10W TDP: following HFI hints visibly increases schedulable CPU time (the heatmap of "excluded" CPUs grows over time if hints are ignored) and gives about **+3 FPS with LAVD**, much more than the LAVD vs EEVDF difference. This work is also a good example of how sched_ext speeds up prototyping support for new hardware and topology features. By the speaker's own account, they were not a sched_ext expert and spent less than a month on it, and the changes fit naturally into LAVD's existing design. By contrast, HFI is still not consumed by the in-kernel scheduler: earlier hardware generations were too noisy, previous related patches were not well received, and as noted in the discussion, without an existing framework such changes "can be debated for years". With sched_ext the idea could be validated on real hardware with real games first, building the case for broader adoption. The implementation is small and opt-in: - HFI updates arrive via netlink and are consumed by a Rust user-space thread, which updates a BPF map (single writer, generation counter, no locks). - **CPU exclusion**: excluded CPUs are cleared from LAVD's active/overflow masks and checked in about ten other places (`select_cpu` paths, shared DSQ consumption, kicking already-running tasks). Excluded is not offline: kthreads and IRQs can still run there. If every CPU is excluded, nothing is excluded. - **Capacity capping**: HFI capacity is folded into LAVD's effective capacity, so more CPUs are activated for the same demand. - Reuses the existing 10ms core compaction timer, no new BPF timers. Discussion: - Why is HFI not used by the in-kernel scheduler? Older generations were too noisy, and the firmware was tuned mostly on Windows workloads; the last two generations treat Linux on equal footing, so it is worth revisiting. - Qualcomm's WALT has a similar CPU isolation feature (thermal and low-utilization cases), implemented downstream via Android vendor hooks. - Shrikanth Hegde's preferred-CPU framework could be a natural in-kernel hook for HFI exclusion. The thermal pressure signal filter currently only allows slowing down the reaction time, and patches to make it configurable in both directions may be revisited given how fast firmware can now report changes. - The HFI code is structured as a library and was proposed for inclusion in the scx library so other schedulers can reuse it. ### Stickiness: keeping tasks cache-warm in the work-conserving LAVD scheduler *Gavin Guo (Igalia)* - [Video](https://www.youtube.com/watch?v=syrnzMx9Iq4) LAVD is work-conserving: each domain has a DSQ, any idle CPU in the domain consumes from its head, and `select_cpu` favors idle cores over cache-warm ones. This hurts cache-sensitive workloads (e.g. edge routing with in-memory routing tables, memcached). Focusing on intra-domain (L2) stickiness, three approaches were compared: - **Warmness** (David, Meta): models cache warmth as a temperature that saturates after ~500us of running and decays afterwards. The remaining warmth is a budget compared against the estimated wait on the previous CPU. It gave noticeable P99 improvements for short waits (~10us), with less benefit for longer ones. - **CPU reservation** (Chen Yu's 2023 idea): if a task's average sleep time is below a threshold, its CPU is reserved and skipped by other wakeups. - **Look-ahead dispatch** (Gavin): an idle CPU scans a small window of the domain DSQ for a task that is warm on it instead of always taking the head. This trades deadline order for locality. With memcached it showed no noticeable latency improvement under light and moderate load, and added overhead under heavy and overcommitted load. Discussion: staying on a busy CPU is a short-term gain, while an idle CPU may offer long-term uninterrupted execution, so the right answer is very workload- and topology-dependent (shared vs non-shared L2/L3, snoop efficiency). Suggestions included using PMU counters (as scx_cosmos already does) or Intel RDT/CMT for cache occupancy (limited by the number of monitoring IDs), tracking only latency-critical tasks, and broader workloads such as stress-ng and video encoders. ## Takeaways and next steps - **Priority inversion** is now a first-class concern for sched_ext. Proxy execution and sched_ext can be enabled together (queued for v7.4), with opt-in BPF support and a single transparent `ops.enqueue()` for blocked tasks; follow-ups include proxy tracepoints and futex ownership. Time slice extension, waiter boosting and proxy execution are complementary, and the scheduler needs generic lock ownership/waiting information for both kernel and user-space locks. - **Paravirtualized scheduling** is converging: pvsched v3 is to be posted upstream, and the different efforts (pvsched device, BPF arena-based shared memory, related proposals) should consolidate into a common framework, with sched_ext as the natural host-side policy engine (bounded boosting, gang scheduling). - **Device and topology-aware scheduling** is gaining traction: accelerator-aware placement needs a generic task-to-device association and BPF-controlled NUMA balancing, which in turn depends on cleanups of the in-kernel NUMA balancing. HFI hints give measurable gains on hybrid Intel parts and may land as a reusable scx library component, a good example of how quickly sched_ext lets new hardware and topology features be prototyped and validated. Cache-aware placement in work-conserving schedulers remains an open problem: the trade-off between cache warmth and immediate execution is highly workload- and topology-dependent, and hardware counters (PMU, RDT) may help quantify it. - More broadly, the project direction is to move more logic into BPF, relying on BPF arenas, build composable scheduler components, and write the BPF side in Rust too.