{"id":1386,"date":"2026-10-28T14:57:14","date_gmt":"2026-10-28T14:57:14","guid":{"rendered":"https:\/\/lpc.events\/blog\/current\/?p=1386"},"modified":"2026-10-28T14:57:14","modified_gmt":"2026-10-28T14:57:14","slug":"the-sched_exit-microconference","status":"publish","type":"post","link":"https:\/\/lpc.events\/blog\/current\/index.php\/2026\/10\/28\/the-sched_exit-microconference\/","title":{"rendered":"The sched_exit Microconference"},"content":{"rendered":"<p>The <a href=\"https:\/\/lpc.events\/event\/20\/sessions\/270\/\">sched_ext Microconference<\/a> was very active this year at Linux Plumbers in Prague. It featured<br \/>\nsix talks, preceded by an opening discussion on the future direction of sched_ext.<br \/>\n(Written by Andrea Righi)<\/p>\n<p><strong>Three main topics dominated the discussions<\/strong><\/p>\n<ul>\n<li><strong>Priority inversion and how to handle it in sched_ext<\/strong>: lock holder and<br \/>\n  lock waiter preemption, time slice extension, and making proxy execution<br \/>\n  work together with BPF schedulers.<\/li>\n<li><strong>Paravirtualized scheduling<\/strong>: letting the host and guest schedulers<br \/>\n  cooperate across the VM boundary, with sched_ext as the policy engine on<br \/>\n  the host.<\/li>\n<li><strong>Device and topology-aware scheduling<\/strong>: using hardware feedback (Intel<br \/>\n  HFI), device\/accelerator locality and cache topology to drive task<br \/>\n  placement.<\/li>\n<\/ul>\n<p><strong>Intro: where sched_ext is going<\/strong><\/p>\n<p><a href=\"https:\/\/www.youtube.com\/watch?v=4vYPuuy4T0M\">The opening discussion<\/a> outlined three directions for sched_ext development:<\/p>\n<ul>\n<li><strong>Move more into BPF.<\/strong> With BPF arenas, queuing and idle CPU management can<br \/>\n  be implemented entirely in BPF, and several schedulers are already<br \/>\n  converging on this model. Once they all move, a significant amount of<br \/>\n  in-kernel sched_ext code can be removed.<\/li>\n<li><strong>Composability.<\/strong> Long discussed, now a good time to tackle it: reusable<br \/>\n  BPF\/user-space building blocks (load balancer from one scheduler, idle CPU<br \/>\n  selection from another, etc.) that can be combined into new schedulers.<\/li>\n<li><strong>Generate BPF code from Rust.<\/strong> Most schedulers today are Rust in<br \/>\n  user space plus C for the BPF part. The trend, aligned with the wider<br \/>\n  BPF community, is to generate the BPF code from Rust as well, so a<br \/>\n  single language and shared abstractions can be used to access<br \/>\n  scheduling data structures from both sides. Prototypes exist but this<br \/>\n  is not there yet.<\/li>\n<\/ul>\n<p><strong>Priority inversion<\/strong><\/p>\n<p><a href=\"https:\/\/www.youtube.com\/watch?v=utS74Dvhp_M\">Making proxy execution compatible with sched_ext<\/a><br \/>\n(Andrea Righi and John Stultz)<\/p>\n<p>Proxy execution lets a mutex-blocked task lend its scheduling context to the<br \/>\nlock owner. The work presented makes this compatible with sched_ext, with<br \/>\nsupport enabled explicitly by each BPF scheduler. Dispatching a blocked task<br \/>\ntriggers the core proxy machinery, keeping the BPF scheduler in control.<\/p>\n<p>The compatibility changes and follow-up improvements are queued for Linux v7.4.<br \/>\nDistributions will be able to enable both `CONFIG_SCHED_PROXY_EXEC` and<br \/>\n`CONFIG_SCHED_CLASS_EXT` in the same kernel, removing their previous mutual<br \/>\nexclusion. The follow-up changes will also make proxy migrations transparent to<br \/>\nBPF, without exposing intermediate bookkeeping through extra enqueue callbacks.<\/p>\n<p><strong>Discussion points:<\/strong><\/p>\n<ul>\n<li><strong>User-space locks<\/strong>: futexes are not covered since `FUTEX_WAIT` does not<br \/>\n  identify the owner. A new futex op that carries ownership is being explored;<br \/>\n  passing user-space owner hints (e.g., via BPF maps or futex flags) is<br \/>\n  inherently racy. A &#8220;directed yield&#8221; (timed proxy to a specific task) was<br \/>\n  mentioned as future work.<\/li>\n<li><strong>rwsems \/ semaphores<\/strong>: writer boosting is supported via a patch; boosting<br \/>\n  multiple readers is harder and has overhead. Plain semaphores have no owner<br \/>\n  and cannot be proxied.<\/li>\n<li><strong>Observability<\/strong>: there are no upstream tracepoints for proxy execution<br \/>\n  yet, which makes traces confusing (the owner runs under the donor&#8217;s<br \/>\n  context). Something similar to the existing PI tracepoints was requested.<\/li>\n<\/ul>\n<p><a href=\"https:\/\/www.youtube.com\/watch?v=aj72z7FLxLk\">Taming latency spikes caused by lock holder and lock waiter preemption<\/a><br \/>\n(Changwoo Min)<\/p>\n<p>On the Steam Frame VR headset, camera processing must finish within 33ms but<br \/>\nneeds only about 7\u20138ms of CPU time. Under heavy gaming load, preempting a lock<br \/>\nholder or delaying a woken lock waiter can stretch the pipeline enough to miss<br \/>\nthat deadline.<\/p>\n<p>LAVD addresses both cases with time slice extension and boosting of lock<br \/>\nwaiters. The mitigations reduced scheduling delays above 2ms across four games,<br \/>\nwith no missed camera frames observed during an hour of gameplay in the tested<br \/>\ntitles.<\/p>\n<p>A common interface for lock ownership and waiting information, covering kernel<br \/>\nand user-space locks, would help replace scheduler-specific heuristics.<\/p>\n<p><strong>Discussion points:<\/strong><\/p>\n<ul>\n<li>Abuse of the extension is bounded by fairness: a task that constantly<br \/>\n  extends its slice is deprioritized.<\/li>\n<li>Why not just rely on proxy execution? Proxy is reactive (owner scheduled<br \/>\n  out, donor scheduled in, owner scheduled in again), with extra overhead.<br \/>\n  The consensus was that the two approaches are complementary: slice<br \/>\n  extension is like a spinlock (cheap, good for short critical sections),<br \/>\n  proxy execution is like a mutex (heavier, but handles long ones).<\/li>\n<\/ul>\n<p><strong>Paravirtualized scheduling<\/strong><\/p>\n<p><a href=\"https:\/\/www.youtube.com\/watch?v=CrGicO8tFhs\">Bridging the VM Boundary: Scheduling Passthrough via pvsched and sched_ext<\/a><br \/>\n(Josh Don and Vineeth Remanan Pillai)<\/p>\n<p>A host scheduler normally sees opaque vCPU threads, without knowing whether<br \/>\nthey are running urgent guest work or background jobs. pvsched provides a<br \/>\nbidirectional channel: guests communicate priorities and preemption hints,<br \/>\nwhile hosts report which vCPUs they can run concurrently.<\/p>\n<p>The proposed v3 design moves negotiation into the VMM through a virtual PCI<br \/>\ndevice, keeping policy outside KVM. On the host, sched_ext can implement bounded<br \/>\nvCPU boosting and gang scheduling. Guests can concentrate work on the vCPUs the<br \/>\nhost can actually run, reducing the cost of double scheduling.<\/p>\n<p>The next revision is still to be posted upstream.<\/p>\n<p><strong>Discussion points:<\/strong><\/p>\n<ul>\n<li><strong>BPF arenas as an alternative<\/strong> to a dedicated device: a host scheduler<br \/>\n  could allocate an arena, hand out per-guest chunks and get zero-copy<br \/>\n  communication with no kernel changes, needing only a way to pin arena<br \/>\n  memory. Someone from the audience pointed out that allocating memory on<br \/>\n  the host and mapping it into the guest on the fly is slow (memslot<br \/>\n  creation, guest notification), which is why guest-initiated allocation was<br \/>\n  chosen. However, this cost can be avoided: the host BPF scheduler can<br \/>\n  create the arena once when it is started and hand out per-guest chunks<br \/>\n  from the preallocated arena, mapping each chunk when the guest is set up<br \/>\n  rather than on the fly, which keeps the fast path cheap. It was also<br \/>\n  noted that the device model keeps the handshake arch-agnostic and<br \/>\n  evolvable without kernel changes.<\/li>\n<li>There is related work from others in this area; the agreement was that<br \/>\n  the various efforts should converge on a single kernel module, possibly<br \/>\n  alongside a BPF-arena-based path.<\/li>\n<li>Standardization: post the kernel module first, then the QEMU device, then<br \/>\n  other VMMs.<\/li>\n<li>The preferred-CPU framework (steal governor) is related, but reactive,<br \/>\n  while pvsched allows an explicit policy.<\/li>\n<\/ul>\n<p><strong>Device and topology-aware scheduling<\/strong><\/p>\n<p><a href=\"https:\/\/www.youtube.com\/watch?v=aW_MdW_mf6g\">Automatic placement of accelerator workloads<\/a><br \/>\n(Balbir Singh and Lee Trager)<\/p>\n<p>CPU and memory placement can have a large effect on accelerator performance,<br \/>\nbut manual pinning requires applications or administrators to know the topology<br \/>\nand device usage in advance. The proposed alternative combines hints about<br \/>\nwhich device a task is using, sched_ext task placement, and memory-access<br \/>\ntelemetry to guide page placement. PMU counters were also suggested as a source<br \/>\nof remote-access and bandwidth telemetry to inform those decisions.<\/p>\n<p>Two interfaces are needed: a generic process-to-device association and BPF<br \/>\ncontrol over NUMA placement preferences. A series enabling NUMA balancing under<br \/>\nsched_ext had recently been posted, together with a kfunc for requesting a<br \/>\npreferred memory node.<\/p>\n<p><strong>Discussion points:<\/strong><\/p>\n<ul>\n<li>Exposing PMU telemetry to non-root applications raises side-channel<br \/>\n  concerns; IPC is fairly safe, cache\/TLB misses are a gray area. An<br \/>\n  allowlist of audited events was suggested.<\/li>\n<li>Is a BPF map the right interface, or is a common user-space library needed<br \/>\n  so all frameworks can provide hints?<\/li>\n<li>Migrating network-heavy tasks across nodes may interact badly with per-CPU<br \/>\n  queues and TCP flows.<\/li>\n<li>NUMA balancing needs improvements, including asynchronous migration and<br \/>\n  charging migration work to the right task. Enabling it under sched_ext<br \/>\n  reuses the existing mechanism, so that work can proceed in parallel with<br \/>\n  those fixes. Telemetry-driven `move_pages()` can serve as an interim solution.<\/li>\n<\/ul>\n<p><a href=\"https:\/\/www.youtube.com\/watch?v=kWyQb-rRmzM\">HFI plug-in for sched_ext<\/a><br \/>\n(Srinivas Pandruvada)<\/p>\n<p>Intel&#8217;s Hardware Feedback Interface supplies per-CPU performance and efficiency<br \/>\nhints that reflect the platform&#8217;s changing power budget. Integrating these<br \/>\nhints into LAVD improved Shadow of the Tomb Raider performance by about three<br \/>\nframes per second on a hybrid Intel handheld at a 10W power limit.<\/p>\n<p>The prototype took less than a month. A Rust user-space thread receives HFI<br \/>\nupdates over netlink and passes them to BPF through a map; LAVD then adjusts CPU<br \/>\nselection and effective capacity.<\/p>\n<p><strong>Discussion points:<\/strong><\/p>\n<ul>\n<li>Why is HFI not used by the in-kernel scheduler? Older generations were too<br \/>\n  noisy, and the firmware was tuned mostly on Windows workloads; the last two<br \/>\n  generations treat Linux on equal footing, so it is worth revisiting.<\/li>\n<li>Qualcomm&#8217;s WALT has a similar CPU isolation feature (thermal and<br \/>\n  low-utilization cases), implemented downstream via Android vendor hooks.<\/li>\n<li>The preferred-CPU framework (steal governor) could be a natural in-kernel<br \/>\n  hook for HFI exclusion. The thermal pressure signal filter currently only allows<br \/>\n  slowing down the reaction time, and patches to make it configurable in both<br \/>\n  directions may be revisited given how fast firmware can now report changes.<\/li>\n<li>The HFI code is structured as a library and was proposed for inclusion in<br \/>\n  the scx library so other schedulers can reuse it.<\/li>\n<\/ul>\n<p><a href=\"https:\/\/www.youtube.com\/watch?v=syrnzMx9Iq4\">Stickiness: keeping tasks cache-warm in the work-conserving LAVD scheduler<\/a><br \/>\n(Gavin Guo)<\/p>\n<p>Keeping a task on a cache-warm CPU can improve performance, but waiting for that<br \/>\nCPU competes with the benefit of running immediately elsewhere. The session<br \/>\ncompared cache-warmth modelling, CPU reservation, and look-ahead dispatch from a<br \/>\nshared queue.<\/p>\n<p>Results depended on the workload and approach. Warmth-based placement improved<br \/>\nP99 latency for short waits, while look-ahead dispatch showed no noticeable<br \/>\nmemcached latency improvement under light or moderate load and added overhead<br \/>\nunder heavier load.<\/p>\n<p><strong>Discussion points:<\/strong><\/p>\n<ul>\n<li>Staying on a busy CPU is a short-term gain, while an idle CPU may offer<br \/>\n  long-term uninterrupted execution, so the right answer is very workload-<br \/>\n  and topology-dependent (shared vs non-shared L2\/L3, snoop efficiency).<\/li>\n<li>Suggestions included using PMU counters (as scx_cosmos already does) or Intel<br \/>\n  RDT\/CMT for cache occupancy (limited by the number of monitoring IDs),<br \/>\n  tracking only latency-critical tasks, and broader workloads such as<br \/>\n  stress-ng and video encoders.<\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>The sched_ext Microconference was very active this year at Linux Plumbers in Prague. It featured six talks, preceded by an opening discussion on the future direction of sched_ext. (Written by Andrea Righi) Three main topics dominated the discussions Priority inversion and how to handle it in sched_ext: lock holder and lock waiter preemption, time slice [&hellip;]<\/p>\n","protected":false},"author":9,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"inline_featured_image":false,"footnotes":""},"categories":[15],"tags":[],"class_list":["post-1386","post","type-post","status-publish","format-standard","hentry","category-microconference"],"_links":{"self":[{"href":"https:\/\/lpc.events\/blog\/current\/index.php\/wp-json\/wp\/v2\/posts\/1386","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/lpc.events\/blog\/current\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/lpc.events\/blog\/current\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/lpc.events\/blog\/current\/index.php\/wp-json\/wp\/v2\/users\/9"}],"replies":[{"embeddable":true,"href":"https:\/\/lpc.events\/blog\/current\/index.php\/wp-json\/wp\/v2\/comments?post=1386"}],"version-history":[{"count":5,"href":"https:\/\/lpc.events\/blog\/current\/index.php\/wp-json\/wp\/v2\/posts\/1386\/revisions"}],"predecessor-version":[{"id":1391,"href":"https:\/\/lpc.events\/blog\/current\/index.php\/wp-json\/wp\/v2\/posts\/1386\/revisions\/1391"}],"wp:attachment":[{"href":"https:\/\/lpc.events\/blog\/current\/index.php\/wp-json\/wp\/v2\/media?parent=1386"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/lpc.events\/blog\/current\/index.php\/wp-json\/wp\/v2\/categories?post=1386"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/lpc.events\/blog\/current\/index.php\/wp-json\/wp\/v2\/tags?post=1386"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}