Speaker
Description
Debugging end user kernel issues is a problem
Reporting bugs is hard - important information is scattered across many log files, crash files, sysfs entries, etc. Most end users don't know where or how to file a bug. They might not even know that a crash has happened (just a glitch on the screen or a log entry somewhere).
When a bug is reported, triage can be difficult. Likely only one report but is it really a one-off issue, never to be seen again? Was all the important info included? Were the logs collected but two hours after the bug occurred and have wrapped? ...
Bugs can cascade - a non-fatal issue can sometimes lead to a fatal issue later in time. A timeline of reports from a single system/boot is needed to identify cause/effect relationships. Bugs can also cross subsystem boundaries and a bug logged against driver X might really have been caused by module Y but team Y might never get to know about it because team X didn't pass it on properly (or at all).
Solution: Remove the user
Automatically generate a dump file - a daemon can ensure it captures all important information (e.g. devcoredump and other sysfs entries plus dmesg, syslog, etc. plus timeline info such as boot time, hashed processor id, etc.) and immediately at the time of the bug occurrance. Daemon then sends the bug report a server.
First level server can be local, but ultimately bugs are sent to a global server. Server does first step triage - build timeline of bug reports with the same session id, check for known instability issues earlier in the timeline, match new bug to existing entries, etc. It would have plugins to help analyse driver specific data blocks (e.g. devcoredump files) for customised triage. Sends notification to driver/module owner when a new bug is identified and provides database for investigating bugs - developers can see how many times a given bug has really occurred, what platforms/hardware/kernels are affected, etc.