Description
The Linux kernel has grown in complexity over the years. Complete understanding of how it works via code inspection has become virtually impossible. Today, tracing is used to follow the kernel as it performs its complex tasks. Tracing is used today for much more than simply debugging. Its framework has become the way for other parts of the Linux kernel to enhance and even make possible new features. Live kernel patching is based on the infrastructure of function tracing, as well as BPF. It is now even possible to model the behavior and correctness of the system via runtime verification which attaches to trace points. There is still much more that is happening in this space, and this microconference will be the forum to explore current and new ideas.
This year, focus will also be on perf events:
Perf events are a mechanism for presenting performance counters and software events that occur running Linux to users. There are kernel and userland components to perf events, with the kernel presenting or extending APIs and the perf tool presenting this to users
Results and accomplishments from the last time (2023):
- Masami's work on accessing function entry data from function return probes (kprobe and fprobe) was merged for v6.9.
- eventfs is now dynamically created and fully working following robust discussions with Linus.
- Work on sframes was paused due to other priorities but is still a topic of interest.
- Discussions on integrating User events with libside are ongoing.
- User events added multi-format events.
Topics for this year:
- Feedback about the tracing/perf subsystems overall (e.g. how can people help the maintainers).
- Reboot persistent in-memory tracing buffers, this would make ftrace a very powerful debugging and performance analysis tool for kexec and could also be used for post crash debugging.
- Dynamic change of ftrace events to improve symbolic printing.
- Userspace instrumentation (libside), including discussion of its impacts on the User events ABI.
- Collect state dump events from kernel drivers (e.g. dump wifi interfaces configuration at a given point in time through trace buffers).
- Current work implementing performance monitoring in the kernel,
- User land profiling and analysis tools using the perf event API,
- Improving the kernel perf event and PMU APIs,
- Interaction between perf events and subsystems like cgroups, kvm, drm, bpf, etc.,
- Improving the perf tool and its interfaces in particular w.r.t. to scalability of the tool,
- Implementation of new perf features and tools using eBPF, like the ones in tools/perf/util/bpf_skel/.
- Further use of type information to augment the perf tools,
- Novel uses of perf events for debugging and correctness,
- New challenges in performance monitoring for the Linux kernel,
- Regression testing/CI integration for the perf kernel infrastructure and tools,
- Improving documentation,
- Security aspects of using tracing/perf tools,
Key attendees:
- Steven Rostedt
- Masami Hiramatsu
- Mathieu Desnoyers
- Alexei Starovoitov
- Peter Zijlstra
- Mark Rutland
- Beau Belgrave
- Florent Revest
- Jiri Olsa
- Tom Zanussi
- Alexander Graf
- Johannes Berg
- Arnaldo Carvalho de Melo
- Ian Rogers
- Namhyung Kim
- Stephane Eranian
-
Ian Rogers (Google)20/09/2024, 10:00
15 years ago the perf events subsystem evolved from prior subsystems like OProfile, adding features like kernel profiling, tracing and system wide profiling. The subsystem continues to evolve, often driven by hardware, tracing and BPF developments. However, the purpose of this talk isn’t to look back at the perf event subsystem, the perf tool and ongoing improvements, it is to take advantage...
Go to contribution page -
Namhyung Kim (Google)20/09/2024, 10:23
Recently perf tools added data type profiling which can find type information in the sample data using debug information (DWARF). It added the basic usage in the perf report and annotate with new sort keys. But it can be extended by more targeted commands like perf mem and perf c2c. We can discuss what it will be looked like and if there are other cases where the type info can be...
Go to contribution page -
Mr Masami Hiramatsu (Google)20/09/2024, 10:45
Linux kernel supports many debugging feature. Tracing events by dynamic probes are one of them. This explains what kind of probes we already support and discuss what will be next. This will includes;
Go to contribution page
- Kprobes
- Fprobe/tprobe
- Eprobe
- Uprobe
- Perf probe and BTF -
Alessandro Carminati20/09/2024, 11:08
I developed
Go to contribution pagekas-aliasto address the issue of duplicate symbols in the kernel. This solution effectively handles duplicate symbols originating from the main kernel image binary and also provides a method for managing symbols in the modules.
However, the current implementation has a challenge that remains unresolved: it modifies an input file during the make process to insert aliases into... -
Alexander Aring (Red Hat)20/09/2024, 12:00
This talk is about a crazy idea that the Linux tracing ring buffers directly operate on your NIC tx/rx ring buffers. You may ask why? Doing that the whole Linux tracing subsystem gets Linux “networkified” as being part of the Linux networking ecosystem and you can do the same things like you do with all your other networking stuff. For example: classification, filtering, etc. Those operations...
Go to contribution page -
Mathieu Desnoyers (EfficiOS Inc.)20/09/2024, 12:23
The goal of the SIDE specification is to enable instrumentation of various runtimes and languages. The libside library is a reference implementation of the SIDE specification for C/C++ user-space.
The SIDE specification covers:
- an event and field description ABI,
- a type system,
- a captured arguments layout ABI,
- and how to sample internal application state.
It specifies how...
Go to contribution page -
Mathieu Desnoyers (EfficiOS Inc.), Michael Jeanson (EfficiOS)20/09/2024, 12:45
There are scenarios where tracer inputs come from user-space and are not paged in. Tracepoints invoked at system call entry immediately after an exec(2) system call are very likely to require page faults to access arguments located in the ELF data section.
This issue is not limited to system call instrumentation, it also affects instrumentation ABIs such as User Events.
Discuss our...
Go to contribution page -
Steven Rostedt20/09/2024, 13:07
The runtime verification utility has been in the kernel for a few years now. It is a way to actively verify that the system is acting as it expects to be by the use of creating a formal model, compiling it, and adding it to the kernel. Then it attaches to tracepoints and when one of those tracepoints is triggered, it moves the model state to the next node....
Go to contribution page -
Steven Rostedt
The Linux kernel has grown in complexity over the years. Complete understanding of how it works via code inspection has become virtually impossible. Today, tracing is used to follow the kernel as it performs its complex tasks. Tracing is used today for much more than simply debugging. Its framework has become the way for other parts of the Linux kernel to enhance and even make possible new...
Go to contribution page -
Steven Rostedt
The Linux kernel has grown in complexity over the years. Complete understanding of how it works via code inspection has become virtually impossible. Today, tracing is used to follow the kernel as it performs its complex tasks. Tracing is used today for much more than simply debugging. Its framework has become the way for other parts of the Linux kernel to enhance and even make possible new...
Go to contribution page -
The Linux kernel has grown in complexity over the years. Complete understanding of how it works via code inspection has become virtually impossible. Today, tracing is used to follow the kernel as it performs its complex tasks. Tracing is used today for much more than simply debugging. Its framework has become the way for other parts of the Linux kernel to enhance and even make possible new...
Go to contribution page -
The Linux kernel has grown in complexity over the years. Complete understanding of how it works via code inspection has become virtually impossible. Today, tracing is used to follow the kernel as it performs its complex tasks. Tracing is used today for much more than simply debugging. Its framework has become the way for other parts of the Linux kernel to enhance and even make possible new...
Go to contribution page
