Skip to content

fix: discover DEV_DAX devices whose sysfs path has no region component - #81

Open
seohui-XCENA wants to merge 1 commit into
mainfrom
fix/hmem-dax-discovery
Open

fix: discover DEV_DAX devices whose sysfs path has no region component#81
seohui-XCENA wants to merge 1 commit into
mainfrom
fix/hmem-dax-discovery

Conversation

@seohui-XCENA

@seohui-XCENA seohui-XCENA commented Aug 28, 2026

Copy link
Copy Markdown
Collaborator

🤔 Background & Motivation (Why)

The resource manager finds no pool on a machine whose DAX device sits under a platform hmem device, even though the device is present and healthy (daxctl reports mode: devdax, /dev/dax0.0 exists, the UUID header is written):

$ maru-resource-manager --state-dir $(mktemp -d)
no CXL/DAX devices found — starting with empty pool
ready — listening on 0.0.0.0:9850
pools empty, rescanning for CXL/DAX devices...      (every 10s, forever)

With no pool there is nothing to allocate from, so maru-server and MaruHandler cannot do anything either — Maru is unusable on such a machine, single node included.

scanDevices() derived the dax_region index by searching the sysfs symlink target for region<N>. That works for pmem (.../ndbus0/region0/dax_region0/dax0.0) and for a CXL region, but an hmem-backed device has no region component in its path:

$ readlink /sys/bus/dax/devices/dax0.0
../../../devices/platform/hmem.0/dax0.0

so the lookup failed and the device was skipped with no log line — which is why the only visible symptom was "no devices found".

🏗️ Design Changes

  • The dax_region index now comes from the device name, with the old link-target parse kept as a fallback. This removes a path dependency rather than adding one: the old code needed the sysfs hierarchy to spell region<N>, and that hierarchy differs per provider and per platform. The name does not — alloc_dax_region and devm_create_dev_dax are built into the kernel core rather than into any module, and dax_pmem, dax_cxl and dax_hmem all call into them, so dax<dax_region id>.<dev_dax id> is produced by one shared code path for every provider. Keeping the old parse as a fallback also means the set of discovered devices is a superset of what it was before, so a working setup cannot regress.
  • How the device path itself is found is untouched — it never went through the symlink.
  • Devices skipped during discovery are now logged with the reason.
  • New internal function parseRegionIndexFromDaxName(), declared in pool_manager.h so tests can link it.
  • No change to pool id numbering, the on-disk metadata format, the WAL format, or the wire protocol.

📝 Implementation Details

Only the region-id derivation changes. How the device path is found is untouched — it never went through the symlink:

Before After
device path /dev/ + entry name from readdir("/sys/bus/dax/devices") unchanged
region id first region<N> found anywhere in the sysfs link target first number in the device name; link target only as fallback
Before:  getRegionIndexForDax("dax0.0")
           readlink("/sys/bus/dax/devices/dax0.0") -> ".../platform/hmem.0/dax0.0"
             target.find("region") -> npos  ->  skipped, silently

After:   getRegionIndexForDax("dax0.0")
           parseRegionIndexFromDaxName("dax0.0") -> 0      <- readlink not reached
           (link-target parse runs only if the name does not parse)

Pool ids are unchanged

Worth a reviewer's eye, since poolId is a persistence key — it names pool_<id>.meta (metadata.cpp:33), is rejected on mismatch when loading (metadata.cpp:54), and is what WAL replay resolves a pool by (wal.cpp:125-127). A renumbering would orphan existing state.

It does not happen here. Wherever discovery already worked, both sources yield the same number: dax_region->id equals the nd_region id for pmem and the CXL region id for CXL, and that is precisely the region<N> in the path. Where discovery failed there was no pool and no metadata to be compatible with.

✅ Tests

  • Unit tests
  • Integration tests
  • Manual tests
  • No tests needed (reason: )

Unit — 3 new gtest cases in tests/test_dax_region_index.cpp; ctest 19/19, clean under -Wall -Wextra -Wpedantic.

Case Covers
ParsesRegionIndexFromDeviceName dax0.0, dax0.1, dax3.7, dax12.0
RejectsMalformedNames dax, dax0, dax0., dax.0, daxfoo, dax_region, pmem0, ""
ResolvesHmemDeviceWhoseLinkTargetHasNoRegion asserts the hmem link target has no region component, and that the name still resolves — the regression

Manual, on an hmem-backed devdax device: the rescan loop is gone and the pool registers at startup. STATS_REQ reports the pool, and ALLOC_REQ → mmap → write → FREE_REQ completes. The full stack (resource manager + maru-server + two MaruHandler clients storing and retrieving over several rounds) runs green, with the two clients receiving disjoint ranges.

Not covered: no pmem or CXL-region machine was available, so the claim that those topologies are unaffected rests on the two arguments above rather than on a run. On such a machine it can be checked in a minute — for every device the first number of the name should equal dax_region/id, and pool_<id>.meta should keep its existing filename.

🔗 Related Issues (optional)

📦 Release Note (for auto-generation / write in English)

NEW

CHANGED

  • Resource Manager: skipped DAX devices are now logged with the reason instead of being passed over silently.

FIXED

  • Resource Manager: DEV_DAX devices whose sysfs path carries no region<N> component (e.g. a dax_region on a platform hmem device) were skipped during discovery, leaving the daemon with no pools and no way to allocate.

IMPORTANT NOTES

DEV_DAX discovery derived the dax_region index by searching the sysfs
symlink target for "region<N>". A device whose dax_region hangs off a
platform hmem device resolves to ".../devices/platform/hmem.0/dax0.0",
which has no region component, so scanDevices() skipped it and the daemon
reported "no CXL/DAX devices found" and rescanned forever with no pool to
allocate from.

Derive the index from the device name instead: the kernel names every
dev_dax "dax<dax_region id>.<dev_dax id>" whatever the provider is, so it
does not depend on where the dax_region sits in the hierarchy. The link
parse stays as a fallback, and pool ids are unchanged where discovery
already worked -- the "region<N>" in a pmem or CXL path is the same region
whose id names the device -- which matters because poolId keys
pool_<id>.meta and the WAL allocation records. The FS_DAX branch of
scanDevices() has had the same name-based fallback all along.

Also log skipped devices; the silent skip made an undiscoverable device
indistinguishable from an absent one.
@github-actions

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant