Speakers
Description
We've observed issues in the fleet where kernel lock holders are unable to obtain CPU time, causing waiters to block and stall overall system wide progress which can be broken into 3 main phenomena:
CPU Bandwidth Limiter: Kernel lock holders being throttled(This should be solved in sched/fair on newer kernels, but presents a problem for BPF schedulers that support cpu.max)
CPUSET Constraints: Kernel lock holders constrained by affinity and unable get sufficient time on CPUs
General Priority Inversion: Higher priority tasks blocked from lower priority lock holders
We’d like to discuss/describe how these problems manifest in the fleet and how they’re observed. For example, using tracing tools such as "wprof" to operate in continuous monitor modes that trigger to capture stalls in the system that would otherwise be difficult to reproduce.