Description
The Linux System Monitoring and Observability MC brings together developers, maintainers, system engineers, and researchers to tackle unsolved problems in understanding, monitoring, and maintaining the health of Linux systems at scale.
Engineers managing millions of Linux servers face monitoring and observability challenges that no single team can solve alone. This track provides a forum to surface those challenges, share partial approaches, and leave with concrete next steps.
The goal is to have these engineers together to discuss the direction and strategy for better monitoring of Linux systems.
Track Objectives
- Surface the most pressing unsolved problems in Linux monitoring and observability
- Identify gaps in existing kernel interfaces, tooling, and infrastructure
- Build consensus on priorities and approaches for the upstream community
- This track invites participants to bring their hardest open questions, pain points, and gaps in current tooling, so the community can collaboratively work toward solutions.
Target Audience
- Hyperscaler Engineers: System reliability engineers who encounter monitoring gaps at scale that others may share
- Kernel Developers: Contributors working on tracing, performance counters, and diagnostic interfaces who want to understand real-world pain points
- Monitoring Tool Developers: Creators of observability platforms who have hit kernel or infrastructure limitations
- System Administrators: Operations teams who can articulate what breaks, what's missing, and what's too hard
- Performance Engineers: Specialists who can identify where current observability falls short for optimization work
Problems involving any of the following (but not limited to) are in scope:
- eBPF/BPF: Tracing and monitoring programs: limitations, missing features, safety constraints
- ftrace/perf: Kernel tracing infrastructure gaps
- Runtime Sanitizers: KFENCE, KASAN: coverage gaps, performance trade-offs, production usability
- Hardware Interfaces: EDAC, MCE, ACPI error reporting, missing integrations, inadequate interfaces
- bpftrace, systemd, netconsole: usability and scalability issues
- kdump/crash/drgn: crash analysis workflow pain points
- perf, memory profilers, below, strobelight, OpenTelemetry: analysis gaps and scaling challenges
Things that made progress given and were discussed in the Micro conference:
-
Lack of NMI on some architecture and how to collect information about it. (Breno)
https://lore.kernel.org/all/rs4igmsjrm6r2aio4nbe5jos3vcqk2u4bjhltjwtj2pn3cquip@kv3grgec7qrb/
https://lkml.org/lkml/2026/3/30/1280 -
Improvements in the LAVD monitoring system (Gavin Guo)
-
Page owner tracking (Mauricio)
https://lore.kernel.org/all/20251205231721.104505-1-mfo@igalia.com/ -
Kmemleak detection in the fleet (Breno)
https://lore.kernel.org/all/20260323-kmemleak_report-v1-1-ba2cdd9c11b9@debian.org/ -
Memory failure and clean crashes
https://lore.kernel.org/all/20260413-ecc_panic-v3-0-1dcbb2f12bc4@debian.org/ -
Track kernels doing kexec
https://lore.kernel.org/all/20260309-kho-v8-0-c3abcf4ac750@debian.org/ -
TCP Reset Observibility (Jason)
https://mailarchive.ietf.org/arch/msg/tcpm/d27ntz9UM4tb4-cxJNxfYw8zCSE/ -
Always-on 7x24 network latency monitor (Jason)
-
Who is planning to submit topics being discussed in the MC to SOSP 2026
-
Relay monitoring (Jason)
-
Diagnostic check and api in relay and future upstreaming discussion
-
Improving memcg statistics collection (JP)
https://lore.kernel.org/all/20260401203752.643259-1-jp.kobryn@linux.dev/ -
General improvements for DAMON (SJ)
-
Brendan Gregg (OpenAI)05/10/2026, 10:00
A new era is upon performance engineering where we ask AI to do much of our analysis and thinking: we run "performance prompts" (and skills and agents) instead of "performance tools." These can analyze all sources, including flame graphs, bcc and bpftrace tools (eBPF), Ftrace, PMCs, and MSRs; they can also propose new metrics and tools. How well does it currently work, and what does this mean...
Go to contribution page -
John Harrison (Igalia)05/10/2026, 10:20
Debugging end user kernel issues is a problem
Reporting bugs is hard - important information is scattered across many log files, crash files, sysfs entries, etc. Most end users don't know where or how to file a bug. They might not even know that a crash has happened (just a glitch on the screen or a log entry somewhere).
When a bug is reported, triage can be difficult. Likely only...
Go to contribution page -
Yuan Tan05/10/2026, 10:35
Linux kernel bugs are reported through fuzzers, static analyzers, mailing lists, and increasingly AI-assisted tools. These reports vary widely in quality: some are duplicates or false positives, while others describe exploitable vulnerabilities or issues with little practical impact. Maintainers still need to determine what is real, what matters, and what deserves attention first.
We are...
Go to contribution page -
Sam Crossley (Meta)05/10/2026, 10:50
This talk explores Meta's approaches to identifying production issues rooted in the Linux kernel. Our investigation process, spanning detection, correlation, and deep triage, relies on a unified framework of telemetry and monitoring tools. A common challenge arising in hyperscale kernel releases is that of robustly comparing kernel performance across lifecycle phases, despite vast variations...
Go to contribution page -
David Dai, Shakeel Butt05/10/2026, 11:05
We've observed issues in the fleet where kernel lock holders are unable to obtain CPU time, causing waiters to block and stall overall system wide progress which can be broken into 3 main phenomena:
CPU Bandwidth Limiter: Kernel lock holders being throttled(This should be solved in sched/fair on newer kernels, but presents a problem for BPF schedulers that support cpu.max)
Go to contribution page
CPUSET... -
Imran Khan05/10/2026, 11:20
Production incidents often require answering concrete questions about current kernel state: which tasks are blocked, what is on each runqueue, where D-state tasks are stuck, or what reclaim and I/O state looks like while the host is still alive. Existing tools such as drgn and crash are powerful, but live inspection can require substantial kernel memory traversal from userspace and may be...
Go to contribution page -
Jakub Sitnicki (Cloudflare)05/10/2026, 12:00
Operating a network at scale, we routinely need to answer a deceptively simple question: what path did this packet take through the kernel, and where did it get dropped? On our edge, a single packet can cross several network namespaces, get encapsulated in GRE or IPIP, be encrypted with IPsec, and pass through multiple nftables chains before it leaves the box. Tagging the payload the way we...
Go to contribution page -
Serapheim Dimitropoulos (CoreWeave)05/10/2026, 12:20
CoreWeave is a bare-metal cloud provider. A typical machine in our fleet has at least 2TB of RAM, and customers tend to run close to that limit. That has two immediate consequences: the crash-kernel memory reservation matters a lot (every GB we reserve is a GB the customer doesn't get), and when a machine panics the resulting crash dump is huge. Because it's so big, the downtime during a...
Go to contribution page -
Omar Sandoval05/10/2026, 12:40
Breakpoint support for drgn is currently underway (https://github.com/osandov/drgn/issues/626). Support is planned for:
- KGDB
- Linux kernel QEMU guests
- Live kernels in production (a sequel to my 2023 LPC session).
- Userspace processes (ptrace)
Each of these targets has its challenges and deficiencies. I will discuss these problems as well as next steps.
Go to contribution page