The sched_ext Microconference was very active this year at Linux Plumbers in Prague. It featured
six talks, preceded by an opening discussion on the future direction of sched_ext.
(Written by Andrea Righi)
Three main topics dominated the discussions
- Priority inversion and how to handle it in sched_ext: lock holder and
lock waiter preemption, time slice extension, and making proxy execution
work together with BPF schedulers. - Paravirtualized scheduling: letting the host and guest schedulers
cooperate across the VM boundary, with sched_ext as the policy engine on
the host. - Device and topology-aware scheduling: using hardware feedback (Intel
HFI), device/accelerator locality and cache topology to drive task
placement.
Intro: where sched_ext is going
The opening discussion outlined three directions for sched_ext development:
- Move more into BPF. With BPF arenas, queuing and idle CPU management can
be implemented entirely in BPF, and several schedulers are already
converging on this model. Once they all move, a significant amount of
in-kernel sched_ext code can be removed. - Composability. Long discussed, now a good time to tackle it: reusable
BPF/user-space building blocks (load balancer from one scheduler, idle CPU
selection from another, etc.) that can be combined into new schedulers. - Generate BPF code from Rust. Most schedulers today are Rust in
user space plus C for the BPF part. The trend, aligned with the wider
BPF community, is to generate the BPF code from Rust as well, so a
single language and shared abstractions can be used to access
scheduling data structures from both sides. Prototypes exist but this
is not there yet.
Priority inversion
Making proxy execution compatible with sched_ext
(Andrea Righi and John Stultz)
Proxy execution lets a mutex-blocked task lend its scheduling context to the
lock owner. The work presented makes this compatible with sched_ext, with
support enabled explicitly by each BPF scheduler. Dispatching a blocked task
triggers the core proxy machinery, keeping the BPF scheduler in control.
The compatibility changes and follow-up improvements are queued for Linux v7.4.
Distributions will be able to enable both `CONFIG_SCHED_PROXY_EXEC` and
`CONFIG_SCHED_CLASS_EXT` in the same kernel, removing their previous mutual
exclusion. The follow-up changes will also make proxy migrations transparent to
BPF, without exposing intermediate bookkeeping through extra enqueue callbacks.
Discussion points:
- User-space locks: futexes are not covered since `FUTEX_WAIT` does not
identify the owner. A new futex op that carries ownership is being explored;
passing user-space owner hints (e.g., via BPF maps or futex flags) is
inherently racy. A “directed yield” (timed proxy to a specific task) was
mentioned as future work. - rwsems / semaphores: writer boosting is supported via a patch; boosting
multiple readers is harder and has overhead. Plain semaphores have no owner
and cannot be proxied. - Observability: there are no upstream tracepoints for proxy execution
yet, which makes traces confusing (the owner runs under the donor’s
context). Something similar to the existing PI tracepoints was requested.
Taming latency spikes caused by lock holder and lock waiter preemption
(Changwoo Min)
On the Steam Frame VR headset, camera processing must finish within 33ms but
needs only about 7–8ms of CPU time. Under heavy gaming load, preempting a lock
holder or delaying a woken lock waiter can stretch the pipeline enough to miss
that deadline.
LAVD addresses both cases with time slice extension and boosting of lock
waiters. The mitigations reduced scheduling delays above 2ms across four games,
with no missed camera frames observed during an hour of gameplay in the tested
titles.
A common interface for lock ownership and waiting information, covering kernel
and user-space locks, would help replace scheduler-specific heuristics.
Discussion points:
- Abuse of the extension is bounded by fairness: a task that constantly
extends its slice is deprioritized. - Why not just rely on proxy execution? Proxy is reactive (owner scheduled
out, donor scheduled in, owner scheduled in again), with extra overhead.
The consensus was that the two approaches are complementary: slice
extension is like a spinlock (cheap, good for short critical sections),
proxy execution is like a mutex (heavier, but handles long ones).
Paravirtualized scheduling
Bridging the VM Boundary: Scheduling Passthrough via pvsched and sched_ext
(Josh Don and Vineeth Remanan Pillai)
A host scheduler normally sees opaque vCPU threads, without knowing whether
they are running urgent guest work or background jobs. pvsched provides a
bidirectional channel: guests communicate priorities and preemption hints,
while hosts report which vCPUs they can run concurrently.
The proposed v3 design moves negotiation into the VMM through a virtual PCI
device, keeping policy outside KVM. On the host, sched_ext can implement bounded
vCPU boosting and gang scheduling. Guests can concentrate work on the vCPUs the
host can actually run, reducing the cost of double scheduling.
The next revision is still to be posted upstream.
Discussion points:
- BPF arenas as an alternative to a dedicated device: a host scheduler
could allocate an arena, hand out per-guest chunks and get zero-copy
communication with no kernel changes, needing only a way to pin arena
memory. Someone from the audience pointed out that allocating memory on
the host and mapping it into the guest on the fly is slow (memslot
creation, guest notification), which is why guest-initiated allocation was
chosen. However, this cost can be avoided: the host BPF scheduler can
create the arena once when it is started and hand out per-guest chunks
from the preallocated arena, mapping each chunk when the guest is set up
rather than on the fly, which keeps the fast path cheap. It was also
noted that the device model keeps the handshake arch-agnostic and
evolvable without kernel changes. - There is related work from others in this area; the agreement was that
the various efforts should converge on a single kernel module, possibly
alongside a BPF-arena-based path. - Standardization: post the kernel module first, then the QEMU device, then
other VMMs. - The preferred-CPU framework (steal governor) is related, but reactive,
while pvsched allows an explicit policy.
Device and topology-aware scheduling
Automatic placement of accelerator workloads
(Balbir Singh and Lee Trager)
CPU and memory placement can have a large effect on accelerator performance,
but manual pinning requires applications or administrators to know the topology
and device usage in advance. The proposed alternative combines hints about
which device a task is using, sched_ext task placement, and memory-access
telemetry to guide page placement. PMU counters were also suggested as a source
of remote-access and bandwidth telemetry to inform those decisions.
Two interfaces are needed: a generic process-to-device association and BPF
control over NUMA placement preferences. A series enabling NUMA balancing under
sched_ext had recently been posted, together with a kfunc for requesting a
preferred memory node.
Discussion points:
- Exposing PMU telemetry to non-root applications raises side-channel
concerns; IPC is fairly safe, cache/TLB misses are a gray area. An
allowlist of audited events was suggested. - Is a BPF map the right interface, or is a common user-space library needed
so all frameworks can provide hints? - Migrating network-heavy tasks across nodes may interact badly with per-CPU
queues and TCP flows. - NUMA balancing needs improvements, including asynchronous migration and
charging migration work to the right task. Enabling it under sched_ext
reuses the existing mechanism, so that work can proceed in parallel with
those fixes. Telemetry-driven `move_pages()` can serve as an interim solution.
HFI plug-in for sched_ext
(Srinivas Pandruvada)
Intel’s Hardware Feedback Interface supplies per-CPU performance and efficiency
hints that reflect the platform’s changing power budget. Integrating these
hints into LAVD improved Shadow of the Tomb Raider performance by about three
frames per second on a hybrid Intel handheld at a 10W power limit.
The prototype took less than a month. A Rust user-space thread receives HFI
updates over netlink and passes them to BPF through a map; LAVD then adjusts CPU
selection and effective capacity.
Discussion points:
- Why is HFI not used by the in-kernel scheduler? Older generations were too
noisy, and the firmware was tuned mostly on Windows workloads; the last two
generations treat Linux on equal footing, so it is worth revisiting. - Qualcomm’s WALT has a similar CPU isolation feature (thermal and
low-utilization cases), implemented downstream via Android vendor hooks. - The preferred-CPU framework (steal governor) could be a natural in-kernel
hook for HFI exclusion. The thermal pressure signal filter currently only allows
slowing down the reaction time, and patches to make it configurable in both
directions may be revisited given how fast firmware can now report changes. - The HFI code is structured as a library and was proposed for inclusion in
the scx library so other schedulers can reuse it.
Stickiness: keeping tasks cache-warm in the work-conserving LAVD scheduler
(Gavin Guo)
Keeping a task on a cache-warm CPU can improve performance, but waiting for that
CPU competes with the benefit of running immediately elsewhere. The session
compared cache-warmth modelling, CPU reservation, and look-ahead dispatch from a
shared queue.
Results depended on the workload and approach. Warmth-based placement improved
P99 latency for short waits, while look-ahead dispatch showed no noticeable
memcached latency improvement under light or moderate load and added overhead
under heavier load.
Discussion points:
- Staying on a busy CPU is a short-term gain, while an idle CPU may offer
long-term uninterrupted execution, so the right answer is very workload-
and topology-dependent (shared vs non-shared L2/L3, snoop efficiency). - Suggestions included using PMU counters (as scx_cosmos already does) or Intel
RDT/CMT for cache occupancy (limited by the number of monitoring IDs),
tracking only latency-critical tasks, and broader workloads such as
stress-ng and video encoders.
