Speakers
Description
As the memory hierarchy deepens at both ends, with HBM adding a faster tier on top and CXL adding cheaper, slower capacity below, keeping hot data in the fast tier and shedding cold data downward makes page migration central to NUMA, tiered-memory and coherent CPU-GPU systems (where device memory is exposed as NUMA nodes).
Profiling move_pages(2) shows the folio copy dominates (~97% of migration time for a 2MB THP), making it the primary scaling bottleneck. Today this path is largely sequential: folios are copied one at a time by a single CPU, leaving DMA engines, idle cores and memory bandwidth underused.
To tap that idle hardware, this work separates the folio content copy from the rest of migration behind a pluggable migrator. The batch path:
- unmaps a batch of folios and flushes the TLB once,
- asks a migrator to copy the eligible folios,
- marks copied folios so the move phase skips the per-folio copy,
- completes the move through the existing flow [1].
The goal is not just the faster copy, but simplification and making different optimizations plug in as composable pieces rather than as special cases. So, the work proceeds in two directions: accelerating the dominant copy phase of migration and restructuring the migration core so that this and other independent optimizations can be added without further complicating it.
Accelerating the copy
Three complementary, measured directions:
1. dcbm - a DMA-based migrator using dmaengine devices, for bulk copy across multiple channels.
2. mtcopy - multi-threaded CPU copy across idle cores, a software fallback where no DMA/offload engine is available.
3. folio_copy() bulk copy for large folios (one bulk copy over the contiguous range); helps even without offload, independent of the batch framework [2].
I measured this with move_pages() on 1GB anonymous memory, dual-socket EPYC Zen 3, node 0 -> 1. Batch copy offload speeds up migration by several times for large folios: up to ~6x for a 2MB THP via DMA (dcbm, 16 channels) and ~3.8x via multi-threaded CPU copy (mtcopy, 8 threads).
Separately, a related demotion effort uses non-temporal stores to reduce cache pollution and CXL-side read traffic [3].
Beyond the copy: the rmap phase
Once the copy is offloaded, per-base-page rmap work dominates for PTE-mapped large folios (mTHP). The unmap and restore rmap walks each run once per subpage. Batching those walks is the next lever, and its payoff grows with folio size: in this setup a 1MB folio reaches ~5.6x over vanilla with DMA offload, compared to ~2x with offload alone [7].
Reworking the core so these compose
Several ongoing efforts optimize different migration phases, and each one adds flags or special cases to the current monolithic path [1][3][4]. They run into the same limitation: ->migrate_folio() only gets enum migrate_mode, which describes the blocking semantics but not the migration intent (for example, promotion versus demotion). So, new behaviour ends up either extending migrate_mode or going through a side channel. For example, non-temporal demotion added MIGRATE_ASYNC_NON_TEMPORAL_STORES, while the DMA offload encoded FOLIO_CONTENT_COPIED in migrate_info.
I am prototyping two changes:
-
Pass a small context struct into ->migrate_folio() carrying mode, reason, and future attributes. This separates blocking behavior from migration intent and makes "how to copy" (cached, non-temporal, offload, and so on) be its own field.
-
Express migration as explicit phases:
lock -> unmap -> [copy] -> move -> remap -> release
A shared engine categorizes the folios into separate lists: LRU folios, hugetlb folios and movable_ops pages each provide their migrate implementation. This gets rid of the folio_test_hugetlb() and page_has_movable_ops() checks scattered through the path today and makes the flow easier to follow.
Discussion
I'd like to use the session to validate the direction of the migration core work and how ongoing efforts should compose.
Key questions:
- Interface evolution: should ->migrate_folio() take a context struct? It touches every callback. Is that acceptable, and is it the right shape?
- Core structure: is an explicit phased migration engine with per-class callbacks a reasonable model for composability and maintainability? How far should movable_ops be decoupled from migrate_pages()?
- Copy helpers: The folio_copy() bulk path can regress without FSRM. Is the fix to improve memcpy() for !FSRM, or to add a dedicated copy_pages() helper?
- Batching trade-offs: What batch-sizes are acceptable when balancing throughput against first-folio latency? Should this be a tunable depending on hardware?
Roadmap:
Scatter-gather copy (DMA_MEMCPY_SG); additional migrators (SDXI [6]); topology-aware thread placement for multi-threaded copy; hot page promotion (pghot [5]) reusing the offload path; migration-side rmap batching; continued simplification of the migration core.
References:
[1] Migrator/offload RFC: https://lore.kernel.org/all/20260630-shivank-batch-migrate-offload-v6-0-da95d7e8b8a2@amd.com
[2] folio_copy optimization: https://lore.kernel.org/all/20260427142036.111940-4-shivankg@amd.com
[3] Non-temporal demotion RFC: https://lore.kernel.org/all/20260526-rfc-nt-demote-v1-0-eb9c9422daef@zptcorp.com
[4] migrate_mode/migrate_folio() ABI extensibility (Memory Hotness and Promotion call):
https://lore.kernel.org/linux-mm/37010218-ecad-4f75-961d-686d51b6677d@amd.com
[5] pghot: https://lore.kernel.org/all/20260504060924.344313-2-bharata@amd.com
[6] SDXI: https://lore.kernel.org/all/20260605-sdxi-base-v3-0-4d38ca2bdffe@amd.com
[7] rmap walk batch: https://lore.kernel.org/linux-mm/20260712-migrate-rmap-batch-v1-0-872a734431d1@amd.com/