Description
Databases utilize and depend on a variety of kernel interfaces and are critically dependent on their specification, conformance to specification, and performance. Failure in any of these results in data loss, loss in revenue, or degraded experience or if discovered early, software debt. Specific interfaces can also remove small or large parts of user space code creating greater efficiencies.
This microconference will get a group of database developers together to talk about how their databases work, along with kernel developers currently developing a particular database-focused technology to talk about its interfaces and intended use.
Database developers are expected to cover:
The architecture of their database;
The kernel interfaces utilized, particularly those critical to performance and integrity
What is a general performance profile of their database with respect to kernel interfaces;
What kernel difficulties they have experienced;
What kernel interfaces are particularly useful;
What kernel interfaces would have been nice to use, but were discounted for a particular reason;
Particular pieces of their codebase that have convoluted implementations due to missing syscalls; and
The direction of database development and what interfaces to newer hardware, like NVDIMM, atomic write storage, would be desirable.
The aim for kernel developers attending is to:
Gain a relationship with database developers;
Understand where in development kernel code they will need additional input by database developers;
Gain an understanding on how to run database performance tests (or at least who to ask);
Gain appreciation for previous work that has been useful; and
Gain an understanding of what would be useful aspects to improve.
The aim for database developers attending is to:
Gain an understanding of who is implementing the functionality they need;
Gain an understanding of kernel development;
Learn about kernel features that exist, and how they can be incorporated into their implementation; and
Learn how to run a test on a new kernel feature.
If you are interested in participating in this microconference and have topics to propose, please use the CfP process. More topics will be added based on CfP for this microconference.
MC lead
Daniel Black daniel@linux.ibm.com
Quick introduction of people. Frame discussion. Will be quick I promise.
many devs are excited about the progress reported on this new stuff, but is it followed / considered by kernel devs.? what kind of gain to expect? any potential issues or feedback to share?
for example, for a write-ahead logging, one needs to guarantee that writes to log are completed before the corresponding data pages are written. fsync() on the log file does this, but it is an overkill for this.
Postgres (and many other databases) have, until fairly recently, assumed that IO errors would a) be reliably signalled by fsync/fdatasync/... b) repeating an fsync after a failure would either result in another failure, or the IO operations would succeed.
That turned out not to be true: See also https://lwn.net/Articles/752063/
While a few improvements have been made, both in postgres and...
At MongoDB, we implemented an eBPF tool to collect and display a complete time-series view of information about all threads whether they are on- or off-CPU. This allows us to inspect where the database server spends its time, both in userspace and in kernel. Its minimal overhead allows to deploy it in production.
This can be an effective method to collect diagnostic information in the field...
there is "backlog" option used in MySQL for both IP and UNIX sockets, but seems like it has a significant overhead on heavy connect/disconnect activity workloads (e.g. like most of Web apps which are doing "connect; SQL query; disconnect") -- any explanation/ reason for this? can it be improved?
users are very worry about any kind of overhead due kernel patches applied to solve Intel CPU issues (Spectre/Meltdown/etc.) -- what others are observing? what kind of workloads / test cases do you use for evaluation?
