Speaker
Description
Identity-based micro-segmentation depends on building a complete graph of workload-to-workload communication. For TCP, this is tractable: connections have clear lifecycle events observable via sock_ops and inet_sock_set_state tracepoints, and conntrack entries persist long enough for user space collection. UDP has none of this, and in production Kubernetes environments, UDP is everywhere: DNS queries,NTP,syslog,SNMP custom service discovery protocols, and increasingly, QUIC.
Generally in production micro-segmentation deployment, 15-20% of observed flows are UDP-based, and we consistently undercount them by 30-40% compared to packet-capture ground truth. This means if you have some ML engine in place, then this ML-based policy engine generates policies with blind spots: it doesn't suggest rules for UDP flows it never observed, leaving them to hit default-deny and break applications after policy enforcement.
What I'll cover:
- Why UDP flow observability is structurally harder- TCP connections:
SYN creates conntrack entry -> observable via ctnetlink ->
inet_sock_set_state tracepoint marks lifecycle. UDP: first packet
creates a conntrack entry with a short timeout (30s default, 120s
for "assured"), and single-datagram exchanges (DNS request->response)
may age out before user space collects them. I'll present data from
production system showing the correlation between conntrack
timeout settings and UDP flow miss rates. - Three approaches and their costs- (1) Tracing every
udp_sendmsg/udp_recvmsg via kprobe/fentry: complete visibility but
3-5% throughput overhead on UDP-heavy workloads.
(2) Increasing conntrack UDP timeouts: reduces miss rate but bloats
conntrack table size (we measured 4x table growth) and increases
memory pressure. (3) BPF socket filter at cgroup level: catches
socket-level events but misses raw UDP (common in legacy workloads
that use raw sockets) - Proposal: lightweight UDP flow event via sock_ops- TCP already has
BPF_SOCK_OPS_ACTIVE_ESTABLISHED_CB and
BPF_SOCK_OPS_PASSIVE_ESTABLISHED_CB. I propose adding analogous
callbacks for UDP- BPF_SOCK_OPS_UDP_FIRST_SEND and
BPF_SOCK_OPS_UDP_FIRST_RECV- that fire once per unique (source,
dest, sport, dport) tuple within a configurable window. This gives
flow-level visibility without per-packet tracing overhead. I'll
present the proposed semantics, the kernel-side implementation
sketch, and the estimated overhead based on prototype measurements. - Conntrack event reliability at scale — Even for flows that conntrack
does capture, we lose 5-8% of ctnetlink events under high connection
rates (>50K new flows/sec) due to netlink socket buffer overflow.