The Kernel Testing & Dependability Micro-Conference was back again at Plumbers this year, in Prague and with a wide variety of topics. In addition to the well-established frameworks such as KUnit and kselftest, some new frontiers are being explored with the spotlight on LLM-assisted tools. Developers are also getting special treats with new kci-dev features and expert insights into local build testing at scale.

Primary author: Guillaume Tucker

kci-dev

What Changed, What Works, and What Kernel Developers Still Need – by Arisu Tachibana (virtual)

The kci-dev command line tool enables developers to interact with KernelCI. Here’s a summary of the latest changes:

• Core implementation moved to the kcidev Python module as an API library
• Standalone patch file testing enabled on top of a Git checkout
• Submission of results to KCIDB enabled with artifacts uploaded to storage
• Experimental MCP added on top of the API

The tool also benefits from these new plugins:

• kci-patchwork to apply patch series from Patchwork and submit test job
• kci_release_review to compare revisions and create reports

Plugins can help extend functionality so they should be a good way of driving future kci-dev development. The MCP may also enable new use cases. Please join the KernelCI mailing list and Discord channel as per the contact page if you wish to take part and discuss things further.

Build Identity

Kernel Builds Deserve an Identity: Portable Kernel Artifacts and Traceable Test Results – by Cong Wang

The raw metadata provided by uname, Git, IDs from various CI systems etc. does not provide any standard or consistent way to identify a build. As such, bug reports can’t directly refer to a single reproducible binary.

Cong Wang’s proposal is the Kernel Bundle Image (KBI) project. It draws a parallel with how OCI solved this for container images. The idea is essentially to create OCI images for kernel builds. However, it still has a low level of adoption and isn’t used by current automated testing systems.

How may KBI gain more traction?

Where should the line be drawn about what to include in the kernel build ID? There hasn’t been a clear consensus on this yet, whether it should include device trees, firmware blobs and other artifacts.

Yocto / OE also has a way to reproduce entire OS builds, including the kernel. This may be used as a guideline or potentially combined with the KBI approach.

It’s also worth noting that TuxMake is able to reproduce binary-identical builds for a given set of inputs (container image and build parameters), which does not constitute a build ID per se but could still be potentially very helpful for this purpose.

A related initiative is KernelSbom since a bill of materials could define the input to a KBI image: source with metadata, main build artifacts, additional contents (firmware, BPF).

KUnit Configuration

Rethinking KUnit Configuration – by David Gow

The session started with a poll: roughly half of the room use KUnit regularly and 3 people answered yes to the last question about running KUnit with multiple configurations. This tends to show that the tool is well established for standard usage but few developers use it in an expert way.

The main issue covered by this session is about limitations with the .kunitconfig file and qemu_config Python module, especially the lack of output format options and the fact that each config covers only a single KUnit run.

Proposal: Determine a single standard configuration mechanism to go beyond limitations while being easy to share between developers. Passing metadata through to the results would also be especially useful for CI systems which can deal with more details and trace the results back to their origin.

Should this be solved with a new config system or by extending an existing one?

• Option 1: extend .kunitconfig (currently a plain Kconfig fragment)
• Option 2: native Python module since kunit.py is already in Python
• Any other options?

David’s preferred option is to extend .kunitconfig.

For multiple runs bundled together, it’s important to be able to parse the results as if they were individual ones e.g. objects embedded within a bigger blob. One potential solution is to produce results in separate files (KTAP).

Multiple runs are useful for CI integration with a single entry point. It also removes the overhead of setting up many short-lived individual jobs. This doesn’t however appear to be too useful for manual runs. It can help a bit with git bisect run although multiple KUnit commands may be bundled within a single script.

It’s also worth noting that having isolated kernel runs would prevent a single crash from breaking the whole series.

The overall consensus is that multiple runs in a single command or config aren’t probably that useful except in particular cases e.g. CI hooks.

Build Testing

Optimizing local kernel build testing – by Arnd Bergmann

As a sequel from Arnd’s talk at Kernel Recipes, this session is to discuss using a manual, local setup to achieve wide build testing coverage. The focus is on linux-next, with -Werror enabled to try and catch all new build issues.

The best approach when something breaks is to send a fix if possible as it is a lot more effective than just reporting bugs. By doing this for many years, Arnd has fixed thousands of bugs upstream (3000+) and touched every subsystem in the kernel.

The most common problems are Kconfig dependency bugs since its logic is complicated and deceptive, especially with tristate options. The new kconfirm tool aims at solving this issue via static analysis.

Arnd keeps a small config fragment to maximise the effectiveness of build testing:

# maximize search space, disable options not worth testing
CONFIG_COMPILE_TEST=y
# avoid long build times
# PROFILE_ALL_BRANCHES is not set
# reduce compile-time dependencies
CONFIG_STANDALONE=y
CONFIG_PREVENT_FIRMWARE_BUILD=y

Should orchestration of parallel builds be integrated upstream?

This sounds useful although the discussion didn’t expand on how to approach this topic in practice. The general makefile and arm64-specific testrandconfig64 files attached have been kindly provided by Arnd as a reference based on his local setup. Please feel free to reach out to him via email should you wish to discuss this further.

Should local build regression reports be sent to KCIDB? With a proposed fix in a patch?

Several people would welcome this and a follow-up discussion is intended to take place with the KernelCI team.

kcov and kselftest

Kselftest coverage with kcov – by Shuah Khan

Thousands of tests are available in-tree via kselftest and KUnit, but what is their coverage? Shuah has made an experimental tool to run kselftest with kcov and start looking into the issue. Coverage data is not ready to be published yet so the discussion was more about the potential use cases. You can already expect to see a more in-depth follow-up analysis on this topic next year and other general kselftest sessions at every future LPC.

How is code coverage useful?

When working with unknown customer workloads, comparing their coverage with well-known tests gives an idea of how much the tests are covering these workloads.

One issue with enabling kcov at build time is that it’s not suitable for production kernels. However, ftrace is usually enabled so it could be used in combination with kcov with different granularity (function-level only coverage with ftrace).

Another dimension to the problem is about evaluating other tools e.g. LLVM/Clang coverage. See also the LLM fuzzing session on a related topic.

KFuzzTest

KFuzzTest: One year on – by Ethan Graham

The aim of KFuzzTest is to combine the benefits of fuzzing and unit testing. A new, experimental API is now available in the latest version of the patch series to make it easier to write tests and facilitate adoption.

The next core feature which is currently under development is stateful fuzzing. Initial trials found a known bug which had already been fixed but this showed the tool’s potential.

Can fuzzing target specific code?

This is not trivial. In some cases, hardware mocks may be used to enable coverage on specific drivers. Generally speaking, KFuzzTest’s aim is to narrow down fuzzing to isolated functions like a unit test would typically do. The effectiveness of this method is still being debated, perhaps the sweet spot lies in between this and syzkaller’s broader approach.

How about stateful fuzzing?

Stateful fuzzing in userspace exists and enables setting extra boundaries. Kernel fuzzing is typically run from outside the kernel, kind of guessing how to invoke the syscalls. Having the kernel asking a userspace fuzzer has led to successful experiments in the past. So overall, stateful fuzzing is feasible.

How about KUnit overlap?

David Gow is in favour of adding functionality that lets userspace pass data into KUnit tests. This could also facilitate making reproducers. However, the stateful approach is not a good fit for KUnit with long strings of operations as unit tests are meant to be run in isolation.

LLM Fuzzing

LLM-Assisted Fuzzing with Syzkaller – by Chuyang Wang & Aleksandr Nogikh

The issue when fuzzing with high-level descriptions is that they don’t go deep enough to reach good code coverage. Hand-written seed programs partially solve this but the approach doesn’t scale.

The proposed solution here is to rely on LLM-generated ones instead. While attempting to reach areas never covered by syzbot, an overall 15% increased coverage was reached over a few months using this approach. This uncovered a new issue with the LLM trying to evade syzkaller’s sandboxing environment. The next step is going to integrate with syzbot.

Is AI-driven cheaper?

Probably but not measured in terms of actual costs. One seed program could then expand coverage.

How about fuzzing drivers?

This is tricky because testing drivers often requires access to real hardware. USB is an exception as physical devices are not strictly necessary. This has enabled the team to significantly increase fuzzing coverage and report many bugs over a short period of time.

Still useful to upstream hand-written seed programs?

Yes, these are complementary approaches.

KernelCI in the Cloud

Testing the kernel where the fleet runs: KernelCI in the cloud – by Norbert Manthey & Stanislav Uschakow

Going beyond the historical use cases of KernelCI which originated on embedded platforms, this session is about testing kernels deployed in the Cloud. The system covers x86_64 and Graviton (arm64) VMs in an AWS lab. Additionally, nested KVM guests on particular VM types are also supported. It provides the typical scalability and variety of platforms one would typically expect to find in this environment. Developers may also run the tests themselves using their own AWS account.

Current upstream kernel coverage includes mainline, linux-next and LTS. Runtime coverage is limited to baseline boot testing and basic checks. Performance testing is next on the roadmap.

How is A/B testing done in the AWS Cloud?

The approach is to run both A and B kernels with the same VM definition to reduce dependencies on the environment (hardware of physical host, virtualization stack, firmware, …). Keeping the kernel as the single moving part enforces it to be the only root cause for any difference found in test results. The code for this A/B testing is made available as a GitHub repository under the KernelCI project umbrella.

How does A/B testing on AWS work with live migrations?

Tests are not relying on a file system for now to keep resource dependencies low. Hypervisor updates can cause one VM to stop but tests are run multiple times and in a distributed way so the impact is mitigated.

Are you running KUnit?

KernelCI is running KUnit in other environments, not in AWS. It’s unclear whether doing this would increase existing coverage.

Which kernel trees have the priority?

Testing linux-next and mainline kernels should be the priority. Covering LTS is also useful to verify that fixes get backported successfully as they would typically be applied to the AWS forks too although those are outside the scope of KernelCI.

Are any more details of the test runs available?

In order to give maintainers access to more details about test runs, the AWS lab setup needs to expand its integration with KernelCI in general. Additional features have been requested during LPC such as being able to automatically bisect issues and sending regressions as direct replies to patches via emails threads. Those reports should also be sent to the kernel regressions list.

Testing BoF

Note – The speaker has apologised for not being able to host the last session of the schedule due to personal reasons. It was replaced with an impromptu Testing BoF instead.

One particular point was discussed:

How to address kselftest failing to build and run?

• Tests may fail to build due to missing dependencies. The kselftest_deps.sh tool can be used to discover those. Integrating it into the kselftest Makefile is being considered in order to report them directly to the user.
• Test build failures don’t cause the overall kselftest build to fail. This is good, although reporting this in a better way might be helpful.
• Build and run regressions need better reporting, they could be made more visible using a tag in email subjects.

Wrapping Up

One lesson learnt is that AI-assisted development tends to be more about data analysis while the initial scope of the Testing MC is rather to focus on low-level tools. These may be seen as optimised, primitive functions producing raw data that LLMs can then leverage and potentially use to orchestrate test runs. Having a session about LLM-assisted fuzzing was a good fit though as it showcased how AI can make use of existing testing components in novel ways.

Last but not least, many thanks to all the speakers for reviewing this blog post, the conference committee and all the attendees for making Plumbers such a great experience. It was a pleasure to be a co-organiser of the Testing MC again this year.

See you at LPC 2027 🐧