Speaker
Description
Building a driver for a new NIC normally means writing a kernel module. Whether you upstream it or maintain it out of tree, it requires a lot of effort. Isn't there a lighter-weight approach?
This talk shows one way to write the driver logic in userspace, without writing a kernel module, while the device still shows up as a Linux network device like "eth0". This is possible with existing in-kernel mechanisms today, but it is not yet a complete solution. Let's discuss what is required for a userspace driver mechanism.
Moving driver logic to userspace reduces the pain of maintaining out-of-tree kernel modules: you stop keeping up with kernel version changes and a critical bug in the driver does not crash the kernel. Other subsystems have similar concepts: FUSE for filesystems, ublk for block devices. The networking subsystem also has building blocks for processing networking in userspace, and they are divided into two groups.
One group keeps the network stack in kernel and hands packets off to userspace below the driver. TUNTAP is a classic example. It's a virtual device used for tunneling and virtual machines, and exchanges packets with userspace rather than with a wire. VDUSE is another example: it lets you implement a vDPA device in userspace, intended as a userspace backend for virtual networks of virtual machines and containers; combined with virtio-vdpa, it provides a function similar to TUNTAP.
The other group does the opposite: VFIO and UIO provide only a thin device-handling layer in the kernel and move the implementation above the driver, including the network stack, out to userspace.
By combining the two groups, and placing the actual driver logic in a userspace daemon between them, we can realize a NIC driver in userspace. One of its use cases is an FPGA-based NIC, which acts as a fully programmable NIC. Because its CPU-facing interface -- descriptor formats, register layouts, and so on -- is entirely up to the FPGA programmer, the driver variations are virtually unlimited.
I have been experimenting with building userspace NIC drivers with TUNTAP/VDUSE and VFIO. For testing, I used an e1000e device emulated in a VM as a common simple device rather than special hardware. Unsurprisingly, there is a performance penalty due to packet copies, high CPU load of the userspace daemon, and so on. Both TUNTAP and VDUSE are also limited by their specific interfaces, such as statistics and netdev features. I'll introduce the various difficulties I faced and discuss how userspace network drivers should be built. Are TUNTAP/VDUSE and VFIO the right shape, or does Linux networking want its own dedicated mechanism? Is it possible to implement a driver's data path in BPF?