Speaker
Description
While cameras have been ubiquitous in Linux systems for more than a decade, vendors have historically been very reluctant to disclose any information about Image Signal Processors (ISP), leading to the proliferation of out-of-tree kernel drivers and closed-source userspace stacks. The situation started to change with the launch of the libcamera project at the end of 2018, and progress has accelerated over the past couple of years with more and more vendors jumping on board. Even Qualcomm recently posted an initial ISP driver in a timid but very real first step, a move that was unthinkable just a couple of years ago.
There is plenty of work left to do, as the increased interest from vendors lays bare the lack of investment of the previous decade that leaves many technical issues unsolved. This microconference will bring representatives of kernel subsystems (mainly V4L2, but also DRM), userspace frameworks (libcamera, GStreamer, PipeWire, ...), image sensor vendors and ISP vendors in the same room to discuss and solve open issues.
Example Topics
The following example topics have already been proposed by potential participants.
Low (sub-frame) video pipeline latency
In a typical video pipeline, a sink (e.g. encoder or compositor) waits until a camera finishes writing a frame to memory before stating to consume it. This latency can be reduced by starting the consumer before the frame is fully available. Lack of standardized hardware synchronization primitives for sub-frame latency makes this particularly tricky. Kernel APIs need to be designed, in collaboration with userspace components such as libcamera, GStreamer, PipeWire and Wayland.
Real‑time for camera
Cameras and ISPs have intrinsic real-time constraints, as control algorithms running on the CPU need to produce parameters and schedule jobs with the hardware in time to process frames produced by image sensors. This requires precise timestamping, job scheduling policies, bounded processing latency, and standardized latency/jitter measurement, which are all new areas for the V4L2 API.
Enumeration for ISP parameters and statistics
The last few years saw multiple new ISP drivers being merged in the Linux kernel, in cooperation with SoC vendors. As cameras and ISPs remain an area where vendors are particularly cautious, it is crucial to give vendors a way to upstream drivers with limited scope that can then be extended incrementally. The V4L2 ISP extensible parameters API has been developed for this purpose, and a V4L2 ISP extensible statistics API has also been proposed. Both are however missing a scheme to discover the features supported by a particular driver version.
V4L2 ISP helpers
Now that the kernel includes multiple ISP drivers, we start to see common patterns with duplicated code across drivers. To scale to more ISPs without suffering from cargo-cult mistakes and inconsistent behaviours between drivers, analyzing common patterns and factoring them to helpers is needed. This requires cooperation between developers and maintainers of ISP drivers.
Scaling the media controller graph
The Media Controller API is a key component of kernel drivers involved in camera pipelines. Modern SoCs increasingly include hardware pipelines made of IP cores from different vendors, assembled in different ways. The media controller subsystem does not support this well as it assumes that all pipelines are built around one principal component. Discussions on how to address this issue have just started, and should progress enough by the time of LPC to benefit from a face-to-face session.
Results from previous editions
A Complex Cameras MC took place at LPC 2024. The discussions have mostly focussed on strategies to encourage vendors to participate in mainline development and open their camera stacks, at least partly. This was a piece of a larger outreach effort that produced results: drivers for ISPs from Amlogic, Arm, DreamChip, NXP, Qualcomm, Raspberry Pi, Rockchip and VeriSilicon are now actively developed, and either available in the mainline kernel or under review on public mailing lists.
The camera & ISP microconference stems from this outreach effort but takes a new form. With a critical mass of vendors participating in upstream development, we can now switch the focus from diplomacy and politics to solving technical problems.