Design: memory-fault tolerance and multi-region support for CXL memory

[congwang-mk]

A daxfs user asked whether daxfs can tolerate memory faults on CXL memory, and whether it should support multiple memory ranges with dynamic add and remove. This issue carries the design spec for review. Nothing is implemented yet; see section 13 for the proposed phases.

---

daxfs memory-fault tolerance and multi-region design

Status: draft for review
Date: 2026-09-04
Format version: 8 to 9

1. Problem

Users want daxfs to survive CXL memory failures. Today a poisoned line consumed by a kernel-side copy is a machine check panic, a poisoned line under an mmap is a SIGBUS with no record, and a CXL device or link loss takes the whole filesystem with it. The image is one contiguous range, so there is no way to keep the index on trusted memory while the bulk data sits on CXL, and no way to add capacity after creation.

2. Principle

daxfs cannot make memory reliable. Its job is to bound the blast radius: a failure costs exactly the pages that lived on the failed memory, never the index that finds them, and never the kernel.

The format is already region-relative. Base data offsets are relative to the base region, bucket values and pool offsets are relative to the overlay region, and pcache slot offsets are relative to the pcache region. Only the superblock's region offsets are absolute. So the region is the unit of placement, failure, and hotplug. No new address space is introduced.

The behaviour mirrors regular filesystems:

Event	Regular filesystem	daxfs
Bad data sector	EIO on that page; drive remaps on write	EIO on that page; poison sentinel in the index; next full write allocates elsewhere
Bad metadata sector	XFS shutdown, ext4 remount-ro	shutdown flag in superblock, all peers go read-only
Missing device	btrfs degraded, XFS realtime device absent	region marked dead, references into it return EIO
Add device	btrfs device add	append a live directory entry

3. Goals and non-goals

Goals:

1. A poisoned data page never panics the kernel. Reads return EIO, mmap touches deliver SIGBUS, and the page is not touched again.
2. Metadata poison shuts the filesystem down read-only rather than corrupting or crashing.
3. The image may span several memory regions with distinct roles, so metadata can live on local DRAM while data lives on CXL.
4. Data regions can be added to a mounted filesystem, and any peer kernel discovers them from shared memory without new mount options.
5. A data region can be declared dead at runtime, by the kernel's memory failure path or by an administrator, and the filesystem keeps running.

Non-goals:

• Redundancy. No mirroring or erasure coding. Losing the only copy of a page is an EIO, as in any filesystem without RAID.
• Online removal with migration. Peer kernels hold raw pointers and PFN mappings that daxfs cannot revoke. Removal is fail-stop (mark dead) or unmount. Single-mounter drain is a possible later addition.
• Growing the bucket array or the pcache slot array. Both stay fixed at creation time.
• Multi-host CXL 3.0 coherence. Unchanged from docs/COHERENCE.md; the validation gate there still applies.

4. Terminology

• Region: one contiguous memory range with a role. Mapped separately by the kernel.
• Home region: region 0. Holds the superblock, the region directory, the base inode table and directory entries, the overlay header, buckets, and metadata pool, and the pcache header and slot metadata. Everything that points somewhere lives here.
• Data region: any region of kind DATA_POOL, BASE_DATA, or PCACHE_DATA. Holds only leaves that nothing else links through.
• Live / dead: directory state. Dead regions are never read.

5. On-DAX format changes

5.1 Superblock

DAXFS_VERSION becomes 9. Fields are added inside the current reserved[3984] area, so the struct stays 4 KiB.

struct daxfs_super {
	/* existing fields unchanged through pcache_hash_shift */

	__le32 flags;			/* DAXFS_SB_SHUTDOWN */
	__le32 region_count;		/* Entries used in regions[] */
	__le32 base_data_region;	/* Region holding base file data */
	__le32 pcache_data_region;	/* Region holding pcache slot data */
	struct daxfs_region regions[DAXFS_MAX_REGIONS];	/* 32 x 64 B */

	__u8   reserved[...];		/* Pad to 4KB */
};

#define DAXFS_SB_SHUTDOWN	(1 << 0)
#define DAXFS_MAX_REGIONS	32

flags is CAS-updated through the coherence helpers. region_count only grows. base_data_region and pcache_data_region default to 0, which reproduces today's single-region layout.

data_offset in the superblock and data_offset in each base inode become relative to the start of region base_data_region. When that is region 0 the value is base_offset + old_value, which mkdaxfs computes. slot_data_offset in the pcache header becomes relative to region pcache_data_region.

5.2 Region directory entry

struct daxfs_region {
	__u8   uuid[16];	/* CXL region UUID, or random for phys */
	__le64 phys;		/* HPA where the creator mapped it */
	__le64 size;
	__le32 kind;
	__le32 state;
	__le64 gen;		/* Incremented on every state change */
	__u8   reserved[16];
};

#define DAXFS_REGION_HOME	0
#define DAXFS_REGION_DATA_POOL	1
#define DAXFS_REGION_BASE_DATA	2
#define DAXFS_REGION_PCACHE_DATA	3

#define DAXFS_RSTATE_EMPTY	0
#define DAXFS_RSTATE_ADDING	1
#define DAXFS_RSTATE_LIVE	2
#define DAXFS_RSTATE_DEAD	3

Entry 0 is always the home region and always LIVE. State transitions are EMPTY to ADDING to LIVE to DEAD, each a single CAS on state followed by a publish fence. No transition goes backwards. phys is valid for every kernel on the same machine, which is the multikernel case. Cross-host mounts resolve uuid instead; see section 9.

5.3 Data pool region

A DATA_POOL region is self-contained: a header followed by page-aligned raw pages, exactly as the overlay pool stores data pages today.

#define DAXFS_DPOOL_MAGIC	0x64706f6c	/* "dpol" */
#define DAXFS_DPOOL_VERSION	1

struct daxfs_dpool_header {
	__le32 magic;
	__le32 version;
	__le64 pool_offset;	/* From region start, page aligned */
	__le64 pool_size;
	__le64 pool_alloc;	/* Atomic bump allocator */
	__le64 free_data;	/* Tagged free list head, as in overlay */
	__u8   reserved[4096 - 40];
};

The bump allocator and the tagged free list are the existing overlay code with the header pointer swapped. The free list head lives inside the region it serves, so a dead region's allocator state is never read.

5.4 Bucket value encoding

Today a bucket value is a pool offset relative to the overlay pool. Overlay pool sizes are bounded well below 2^48 and the free list already masks offsets to 48 bits. The high byte becomes a region index for DATA keys only:

#define DAXFS_OVL_VAL_OFF_MASK	0x0000FFFFFFFFFFFFULL
#define DAXFS_OVL_VAL_REGION(v)	((v) >> 56)
#define DAXFS_OVL_VAL_OFF(v)	((v) & DAXFS_OVL_VAL_OFF_MASK)
#define DAXFS_OVL_VAL_MAKE(r, off)	(((__u64)(r) << 56) | (off))
#define DAXFS_OVL_VAL_POISON	((__u64)-1)

Region 0 means the home overlay pool, which is where data pages go when no DATA_POOL region exists. That keeps empty-mode and split-mode images built by the old layout logic valid under the new encoding. INODE, DIRLIST, and DIRENT values always refer to the home pool and never carry a region index; the key already tells the reader which kind of value it is looking at.

DAXFS_OVL_VAL_POISON is a sentinel meaning the page exists but is unreadable. A lookup that finds it returns ERR_PTR(-EIO), not NULL, so the read path does not fall through to the base image.

5.5 pcache slot state

PCACHE_STATE_POISONED takes the unused value 3 in the two state bits. A poisoned slot is never pinned, filled, or evicted. The slot is lost capacity, like a remapped sector.

5.6 Base image poison

Base data is immutable and has no bucket. A poisoned base page is recorded by inserting an overlay bucket with key DAXFS_OVL_KEY_DATA(ino, pgoff) and value DAXFS_OVL_VAL_POISON. The overlay already shadows base pages, so no new mechanism is needed. A read-only image without an overlay cannot record poison and simply returns EIO on every access to that page.

6. Runtime structures

struct daxfs_region_map {
	void *mem;
	phys_addr_t phys;
	size_t size;
	struct dax_device *dax_dev;	/* NULL for raw phys and dmabuf */
	u32 index;
};

struct daxfs_info {
	...
	struct daxfs_region_map *regions[DAXFS_MAX_REGIONS];	/* NULL = not mapped */
	struct mutex region_lock;	/* Serialises lazy mapping and add/dead */
	...
};

struct daxfs_dpool {
	struct daxfs_dpool_header *header;
	void *pool;
	u64 pool_size;
	u32 region;
};

struct daxfs_overlay {
	...
	struct daxfs_dpool *dpools[DAXFS_MAX_REGIONS];	/* Indexed by region */
	u32 alloc_hint;			/* Last region allocated from */
};

info->mem and info->size remain as aliases for region 0 so the existing superblock, base inode, overlay, and pcache metadata code is untouched.

6.1 Pointer resolution

daxfs_mem_ptr(info, offset) keeps its meaning for region 0. A new daxfs_region_ptr(info, region, offset, len) returns a pointer, or ERR_PTR(-EIO) when the directory state is not LIVE, or ERR_PTR(-ENXIO) when the region is not yet mapped and lazy mapping fails. It reads regions[region].state from the home region on every call. That is one load from a hot cacheline and is the whole cost of fault checking on the data path.

daxfs_in_region(info, ptr, len) becomes a scan over the mapped region array (at most 32 entries) and returns the region index, so the copy helpers can check liveness for pointers that came from the per-inode xarray cache without going through a bucket lookup.

daxfs_mem_phys and daxfs_mem_offset take the region into account through the same scan.

6.2 Lazy mapping

A peer kernel mounts with only the home region. When a lookup or an allocation meets a region index it has not mapped, it takes region_lock, re-reads the directory entry, and maps phys/size with memremap if the state is LIVE. Allocation and the fault path are both process context, so sleeping is fine. A region that is ADDING is treated as not yet present: allocation skips it and lookups cannot meet it because nothing has been published into it.

7. Allocation policy

• INODE, DIRENT, and DIRLIST entries always come from the home pool.
• DATA pages come from a live DATA_POOL region if any exists, otherwise from the home pool. The allocator starts at alloc_hint, and on -ENOSPC moves to the next live DATA_POOL region and updates the hint. Batch allocation never spans regions, which preserves the contiguity the read path relies on.
• Freed data pages return to the free list of the region they came from. Freeing into a dead region is a no-op.
• The home pool is never used for data once a DATA_POOL region is live. This is what keeps the index off the risky memory. A mount option data_in_home can relax it for images that want a single range.

8. Failure handling

8.1 Synchronous poison on copy

Every copy out of DAX memory switches to the machine-check-safe variant. _copy_mc_to_iter is a drop-in for _copy_to_iter and keeps the hardened-usercopy bypass; on kernels without CONFIG_ARCH_HAS_COPY_MC it is defined to _copy_to_iter, so the change is free where recovery is impossible anyway. Kernel-to-kernel copies (COW from base into an overlay page, the anonymous-page fallback in the fault handler, symlink targets) use copy_mc_to_kernel.

A short copy means poison. The caller:

1. Returns -EIO (or VM_FAULT_SIGBUS from a fault) for that page. Bytes already copied are returned as a short read first, as today.
2. Records the page so it is not consumed again: - overlay data page: CAS the bucket value from its current offset to DAXFS_OVL_VAL_POISON. The page is not returned to any free list. - base data page: insert a poison bucket as in 5.6. - pcache data page: CAS the slot to POISONED with refcount 0. - metadata (an entry in the home pool, a bucket, a base inode, a dirent array, a pcache slot record): set DAXFS_SB_SHUTDOWN.
3. Drops the pointer from the per-inode xarray cache and calls unmap_mapping_range on the inode for that page, so a MAP_SHARED PFN mapping established earlier is torn down. A later touch faults, the fault handler sees the sentinel, and returns SIGBUS.

Metadata loads that are plain dereferences (bucket state words, entry headers, free list links) are not wrapped. They are small, they live on the home region, and a poison there is a shutdown condition. If the home region is on memory the user does not trust, the honest answer is that a metadata poison may still panic on such a load. This is documented.

8.2 Asynchronous notification

When a region is backed by a device-dax node, the kernel can report a poisoned range without anyone consuming it. daxfs registers as the dax holder for that device with notify_failure(dax_dev, off, len, flags). The handler:

1. Maps (dax_dev, off) to (region, offset).
2. If the region is the home region or len covers the whole region, marks the region DEAD (home: sets shutdown as well).
3. Otherwise finds every page overlapping the range. Overlay data pages are found by scanning the bucket array for DATA keys whose value resolves into the range. Base data pages by scanning the base inode table. pcache pages by scanning slot records. All three arrays are flat and bounded, and the scan runs only on a failure.
4. Applies step 2 and 3 of 8.1 to each page found.

Today fs_dax_get_by_bdev is the only holder registration entry point and it requires a block device, and device-dax does not forward memory_failure to a holder the way pmem does. Both need small upstream additions; until they land, the DAXFS_IOC_REGION_FAIL ioctl (8.4) covers the same path from userspace, driven by the CXL event trace.

Raw physical regions without struct pages cannot receive notify_failure. On those, only 8.1 applies, plus the ioctl.

What each backend can detect, and by which path:

Backend	User touch	Kernel copy	Async report	Device gone
device-dax	SIGBUS	EIO via copy_mc	holder callback (phase D)	udev rule + ioctl
raw phys	SIGBUS	EIO via copy_mc	ioctl only	ioctl only
dma-buf	SIGBUS	EIO via copy_mc	ioctl only	not applicable

A userspace touch is detected by the CPU and turned into SIGBUS by the arch handler on every backend, but without a holder callback daxfs is not told, so no sentinel is recorded and the next reader consumes the line again. Only the holder path or the ioctl closes that gap; the kernel copy path closes it for reads and writes on every backend.

CXL media events reach the kernel as mailbox event log entries and are emitted as trace events carrying the host physical address and region UUID. In the kernels examined for this spec they are not fed into memory_failure, so on their own they reach daxfs only through a userspace consumer such as rasdaemon invoking region fail. If the target kernel routes CXL events into memory_failure, the device-dax holder path in phase D picks them up with no further work.

8.3 Region death

A DATA_POOL, BASE_DATA, or PCACHE_DATA region marked DEAD causes:

• daxfs_region_ptr to return -EIO for every reference into it. No memory in the region is touched, including its allocator header.
• the allocator to skip it.
• the notifying kernel to walk the arrays as in 8.2 step 3 and unmap every mapping into the region. Other kernels do the same lazily: a read through a cached pointer fails the liveness check in the copy helper, which drops the cached pointer and unmaps.
• a dead PCACHE_DATA region to make every pcache lookup a miss. On the host that degrades to reading the backing file directly for each request; spawn kernels, which cannot read the backing file, get EIO. Re-creating the pcache in a live region is a later addition.
• a dead BASE_DATA region to make base file reads EIO unless the page has an overlay copy.

The home region DEAD or DAXFS_SB_SHUTDOWN set causes every mutating operation to return -EIO and every lookup that needs the home region to fail. Reads that need nothing from the home region still succeed (a cached overlay pointer in a live data region, for example), which is the same behaviour XFS gives after a shutdown for pages already in the page cache. remount cannot clear it; unmount and repair.

8.4 Administrative interface

#define DAXFS_IOC_REGION_ADD	_IOW('D', 2, struct daxfs_region_arg)
#define DAXFS_IOC_REGION_FAIL	_IOW('D', 3, struct daxfs_region_arg)
#define DAXFS_IOC_REGION_INFO	_IOR('D', 4, struct daxfs_region_info)

struct daxfs_region_arg {
	__u8  uuid[16];
	__u64 phys;
	__u64 size;
	__u32 kind;
	__s32 fd;		/* device-dax fd, or -1 for phys */
	__u64 off, len;		/* FAIL only: sub-range, 0/0 = whole region */
};

REGION_ADD: map the range, write a daxfs_dpool_header, CAS the next EMPTY directory entry to ADDING, publish the header, then CAS to LIVE. Only DATA_POOL can be added after creation. REGION_FAIL with a sub-range runs 8.2 step 3; with the whole region it marks it DEAD.

daxfs-inspect grows region list, region add, and region fail, resolving a /dev/daxX.Y argument through sysfs the way mkdaxfs already does in dax_device_info.

9. Mount and discovery

Existing options are unchanged. The home region is what phys=/size= or dmabuf= names. New repeatable option:

region=/dev/dax1.0
region=0x200000000:0x40000000

Each entry is matched to a directory entry by uuid when the argument is a dax device (from the CXL region's sysfs uuid), else by phys. A region= option is only needed when the kernel wants a device-dax backing for failure notification, or when phys in the directory is not valid on this host. Without it, every LIVE entry is mapped lazily by phys as in 6.2. Spawn kernels in a multikernel deployment therefore need nothing new.

For cross-host CXL, phys differs per host, so the host must supply region= for every data region and the home region must itself be on the fabric. That is the only configuration where metadata cannot be kept on local DRAM, and the shutdown path is what protects it.

10. mkdaxfs

• Always writes a version 9 superblock with region_count = 1, entry 0 describing the home region, and base_data_region = pcache_data_region = 0. This is today's layout under the new encoding.
• New --data-region PATH|phys:size (repeatable) creates DATA_POOL regions at creation time. Split and empty modes then size the home overlay pool for metadata only (inode, dirent, dirlist entries), and the -O pool size applies to the data regions.
• New --base-data-region PATH|phys:size places base file data in a separate BASE_DATA region. Directory dirent arrays and symlink targets stay in the home region because they are metadata.
• New --pcache-data-region PATH|phys:size likewise for pcache slot data.
• -V/validate checks that every non-empty directory entry has a sane size and kind and that base_data_region and pcache_data_region refer to LIVE entries of the right kind.

11. CXL mapping

• One daxfs region is one CXL region exposed as device-dax. Interleaved CXL regions widen the blast radius to every device in the interleave; single-device regions isolate failures. The choice belongs to the administrator and daxfs is indifferent.
• The CXL region UUID is the directory entry's uuid.
• Consumed poison lands in 8.1. Media error events are trace events (see 8.2) and reach daxfs through rasdaemon and region fail unless the kernel routes them into memory_failure. Poison is cacheline granular; daxfs marks the enclosing page.
• Link loss or surprise removal is a PCIe AER event, not a memory error. The CXL driver tears the region down and the dax devices under it disappear, and no filesystem holding a mapping of the range is told. A udev rule on the dax device removal event runs region fail to mark the region DEAD. Until that runs, loads from the range are platform-defined (poison or all-ones on most parts) and are caught by 8.1.
• Dynamic capacity add surfaces new dax extents; region add consumes them. A release request is refused while the filesystem is mounted and the region is LIVE. Single-mounter drain is future work.
• Metadata on DRAM and data on CXL is also the right latency placement: index lookups are latency bound, bulk copies are bandwidth bound.

11a. Multikernel considerations

• A machine check is delivered to the CPU that consumed the poison. Peer kernels learn about a bad page or a dead region only through the sentinel and directory state in shared memory. This is by design: it costs one load on a hot cacheline and needs no IPI.
• On Intel parts without local machine check (LMCE) enabled, a machine check is broadcast to every logical CPU, including CPUs a spawn kernel owns. Each kernel's handler then waits for every CPU it knows about to rendezvous, and with CPUs split across kernels that wait times out and panics. Deployments must confirm LMCE is enabled, or the multikernel machine check handler must account for CPUs it does not own. This is a platform requirement, not something daxfs can work around.
• Spawn kernels run no CXL or dax driver. They never receive a holder callback or a device removal event, and only the host does. The host is therefore the kernel that marks regions dead and records async poison; spawns only record poison they consume themselves.

12. Compatibility

Version 8 images are refused, as with every previous bump. There is no in-place upgrade; the tool can rebuild an image from its source directory. The bucket value encoding is the only change that touches existing on-DAX words, and region index 0 makes an unchanged value mean what it meant before, so the kernel change to overlay_lookup is a mask and a region check.

13. Implementation phases

Each phase is independently shippable.

Phase A: poison containment, no format change. Switch every DAX copy to the copy_mc variants. Add the poison sentinel (a reserved value, so no version bump), the POISONED pcache state, and the shutdown flag (a reserved superblock word). Add DAXFS_IOC_REGION_FAIL operating on region 0 sub-ranges. Add a fault-injection knob for testing. Touches dax_mem.c, file.c, overlay.c, pcache.c, dir.c, super.c.

Phase B: region directory and data pools. Version bump, directory in the superblock, daxfs_dpool, bucket value region index, per-region mapping and liveness checks, lazy mapping, allocation policy, REGION_ADD, REGION_INFO, mkdaxfs --data-region, daxfs-inspect region. Touches the format header, dax_mem.c, overlay.c, super.c, validate.c, both tools.

Phase C: base data and pcache data regions. base_data_region, pcache_data_region, the mkdaxfs options, and the relative-offset changes in file.c and pcache.c.

Phase D: dax holder integration. Device-dax backing for regions, holder registration, notify_failure. Depends on the two upstream additions named in 8.2 and is the phase most likely to slip; the ioctl covers the gap.

14. Testing

• Fault injection. A debugfs file daxfs/<mount>/inject_poison taking region:offset:len makes the copy helpers return short for that range and makes daxfs_region_ptr behave as if poisoned. This is how Phase A and B are tested without CXL hardware.
• Phase A. test_overlay.sh gains cases: read of a poisoned overlay page returns EIO and later reads do not touch the page (verified by clearing the injection and observing EIO persists); poisoned base page with and without an overlay; poisoned pcache slot; metadata poison flips shutdown and writes return EIO; test_mmap gains a SIGBUS case for a MAP_SHARED page poisoned after mapping.
• Phase B. Create an image with two data regions, write until the first is full and confirm the allocator moved; region add on a mounted filesystem and write into the new region from a second kernel that mounted before the add; region fail on a data region and confirm EIO on its files, success on others, and continued writes into live regions; df accounting across regions.
• Real hardware. On a CXL device with poison injection support (inject_poison under the device's debugfs) and on x86 with EINJ, repeat the Phase A cases against real machine checks. This is the only test that proves the copy_mc recovery path on the platform.

15. Open questions

1. Should data_in_home be the default for images with no DATA_POOL region, or should mkdaxfs refuse to create a writable image without one when --data-region is available? Default proposed: allow, so existing workflows keep working.
2. Is 32 regions enough? It fits the superblock with room to spare; 64 would need a separate directory block. Proposed: 32.
3. Should notify_failure on a home-region sub-range that hits only data pages in the home pool (the data_in_home case) be treated as page poison rather than shutdown? Proposed: yes, scan first, shut down only if the range touches something that is not a data page.