Speaker
Description
Cross-node Memory as a Linux Storage Tier：Exploring POSIX-based Shared Memory
Background & Motivation
Memory-semantic interconnects such as CXL 3.0 and Huawei United Bus make remote memory directly addressable. This raises a question for Linux: should cross-node memory become another storage tier that can be exposed through existing POSIX filesystem interfaces?
Our Exploration: XMFS
To explore this question, we built XMFS, an in-kernel prototype that exports cross-node shared memory through a standard POSIX filesystem interface. Initial experiments with metadata-intensive workloads and container image sharing suggest the approach is practical.
Engineering Challenges Encountered
During development we repeatedly encountered four areas where existing kernel abstractions are either missing or insufficient:
• Metadata synchronization: Current VFS and filesystem metadata management assume a single kernel instance coordinating namespace and inode updates. What generic kernel mechanisms are needed to efficiently synchronize metadata across multiple kernels without introducing centralized bottlenecks?
• Software cache coherence: The Linux page cache and MM subsystem rely on hardware cache coherence within a machine. When memory is shared across nodes without hardware coherence, should software cache-coherence policies live inside individual filesystems, the MM subsystem, or new generic kernel infrastructure?
• Integrating hybrid transports: Linux currently exposes memory-semantic interconnects and RDMA through different subsystems and programming models. How should these be integrated so that filesystems can transparently use both without exposing transport-specific behavior to applications?
• Capacity management & Tiering: Linux already provides memory tiering, page migration, and NUMA-aware memory management. Should cross-node shared memory participate in these mechanisms, and what interfaces are needed between filesystems and MM to support it?
Topics for Discussion
Rather than presenting XMFS as a finished filesystem, we hope to discuss whether Linux should treat cross-node memory as a first-class storage tier, which abstractions belong in VFS/MM rather than individual filesystems, and what a practical upstream path could look like. We believe these questions will become increasingly relevant as memory-semantic interconnects become more widely available.