Linux Plumbers Conference 2026

Following the bread crumbs: Tracing packets through the kernel with per-skb BPF metadata

Not scheduled

20m

Linux System Monitoring and Observability MC Linux System Monitoring and Observability MC

Speaker

Jakub Sitnicki (Cloudflare)

Description

Operating a network at scale, we routinely need to answer a deceptively simple question: what path did this packet take through the kernel, and where did it get dropped? On our edge, a single packet can cross several network namespaces, get encapsulated in GRE or IPIP, be encrypted with IPsec, and pass through multiple nftables chains before it leaves the box. Tagging the payload the way we tag L7 requests is not an option here: it's opaque, encrypted, and rewritten along the way.

We'll walk through our attempt to build an mtr-like tool that, instead of a list of hosts, shows the kernel waypoints a packet hits — interfaces, netns boundaries, tunnel encap/decap, IPsec encrypt/decrypt, nftables verdicts, and drops — each with a timestamp.

We started with Retis, Red Hat's eBPF-based packet tracer, which already does most of the heavy lifting. We'll cover what worked out of the box and where we hit its limits:

• It correlates events by hashing the skb data area (skb->head); we found
  tracking by skb address to be both cheaper (~15% fewer trigger events on a busy box) and a better fit.
• The "tcpdump model" mismatch: Retis filters at every probe, capturing many
  flows. For a traceroute-like UX we instead want to lock on to a packet once,
  then have downstream probes fire purely on identity.

This is where the observability problem becomes a kernel interface problem. To follow a packet across transformations, and to let BPF programs annotate a packet at one hook and read it back at another, we need per-packet state that lives and dies with the sk_buff. The BPF-map "side-stash" works today but is costly: to avoid leaking entries it has to hook the consume_skb release path, which fires for every one of the hundreds of thousands of packets passing through the system each second, even when we only care about a handful of them. So we started an upstream effort to add a better building block: a BPF metadata buffer embedded in an skb extension chunk.

We'll update the audience on where that work has moved since we last presented it at Netdev 0x1A in July, and on the challenges we faced wiring Retis up to use it.

We'd like to hear whether others have felt the same gap in production, and if not, how they observe a packet's journey through their systems today.