fix(daemon): keep held runtime files fresh and exit when one is lost - #3
halindrome wants to merge 8 commits into
Conversation
macOS com.apple.tmp_cleaner deletes /tmp files whose atime, mtime and ctime are all older than three days. The daemon's long-held lock files are empty and never written, so under a daemon that lives past three days the cleaner unlinks them while they are held. The next participant then creates and locks a fresh inode: forked index workers see the cohort daemon marker as free, read the live daemon as uncoordinated and exit 1 on every reindex, and a lost IPC lifetime lock or cohort lifetime file stops protecting the generation at all. A lost socket identity marker additionally makes listener close leave the socket pair behind. The host lifetime loop now runs a heartbeat every 60 s over every file the generation holds (cohort daemon claim and lease, participant guard, IPC lifetime reservation, identity marker). Each file is refreshed with futimens through its held descriptor, only after proving the path still names that descriptor, so the cleaner never selects it. If any file is already lost, the daemon logs daemon.lifetime_end reason=coordination_file_lost and stops cleanly; the next client starts a coordinated generation. Re-claiming in place would race that client. Windows needs no refresh (no age-based cleaner; held lock handles deny deletion), so its entry points only validate the handles. Closes DeusData#2178 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Shane McCarron <shane.mccarron@corvexconnect.com>
- Separate loss from transient failure. Touch helpers now report OK / lost / transient (version_cohort: OK/UNSAFE/IO; ipc: 1/0/-1). Only a proven unlink or replacement stops the daemon. A refresh failure on a still-valid file (EROFS, EPERM, EMFILE on the marker reopen) logs daemon.coordination_touch_failed once, retries every tick, and logs daemon.coordination_touch_recovered when it clears. - Cohort and claim locks now prove the path still names the held inode (private_file_revalidate against the stored directory + name), not just st_nlink, so rename-away, hard-link+unlink and a replaced directory are caught like unlink. - The runtime service's self-owned participant guard is refreshed by the listener heartbeat too. - Tests: rename-away detection; cohort lease/claim touch (lease with only lifetime held is OK, each unlinked file reported independently); IPC listener touch now checks timestamps are actually refreshed and covers the participant guard and identity-marker replacement. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Shane McCarron <shane.mccarron@corvexconnect.com>
QA Round 1Contract: DeusData#2178. Acceptance criteria are root-cause fixes 1 and 2: keep held files fresh, and detect loss then exit cleanly. The sentinel wording and the promotion of the worker failure reason into the daemon log (issue items 3a and 3b) are out of scope. Review panel: 3 Claude lenses (contract-security, regression-edges, schema-tests) plus a synthesizer. None failed. Contract Verification
After the round 1 fixes: AC3 is addressed by finding 1 and AC7 by findings 4–6. AC6 is still partial, by design (see finding 7). Finding 1: [claude] Cohort and claim lock loss is detected only through st_nlink on the held inode, with no check that the path still names that inode
Finding 2: [claude] Transient or IO errors during the heartbeat are reported as a lost coordination file and stop the daemon
Finding 3: [claude] No automated test covers the host wiring: host_coordination_touch, stop on loss, the log event, permanent generations, and no false positive at the t=0 check
Finding 4: [claude] The cohort and participant-guard touch helpers are untested, including the NULL-lock semantics that keep a running daemon alive
Finding 5: [claude] The IPC listener touch test checks only return values and never that timestamps were refreshed
Finding 6: [claude] Replacement (unlink then recreate at the same name) is not tested, so neither the AC2 hazard nor the dev/ino check is protected against regression
Finding 7: [claude] The Windows touch paths have no test, including the platform-neutral half that checks an intact handle is not reported lost
Finding 8: [claude] The socket and its anchor are deliberately not refreshed, and loss of the socket is not detected
Finding 9: [claude] After a long sleep there is a blind window: the monotonic clock does not advance during macOS sleep, so after more than 3 days asleep tmp_cleaner can run before the next touch
Finding 10: [claude] The runtime-service start path's owned_participant_guard is not refreshed by the listener heartbeat
Summary
Fixed: 1, 2, 4, 5, 6, 10. Accepted with rationale: 3, 7. No change needed: 8, 9. Schema ChangeNone. No SQL, migrations or DDL; all three lenses agree. Other notes
NEW SAST findings (GitHub code scanning, open alerts on
|
- Classify touch failures explicitly. A file is lost only on proof that the path no longer names it: ENOENT, st_nlink 0, another inode, a symlink or other non-regular entry, or a replaced lock directory. Any other syscall failure (EIO, ESTALE, ENOMEM, EACCES on reopen) is transient, so a flaky runtime volume no longer stops the daemon. The POSIX private-lock touch and the ipc lock/marker touches share this rule; the Windows touch reports a failed handle query as transient. - The identity marker is classified by path (fstatat, no follow) before it is reopened, so replacement by a symlink is now loss, not a transient reopen failure. - Tests: symlink replacement of the identity marker (lost); an unreadable marker is transient (-1) and recovers (1) once readable; the cohort touch test ages both held files and asserts they were refreshed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Shane McCarron <shane.mccarron@corvexconnect.com>
QA Round 2Contract: DeusData#2178 (AC1–AC7, issue items 3a and 3b out of scope). Review panel: 3 Claude lenses plus a synthesizer, none failed. CMM graph tools were available this round. The branch was re-synced with upstream Contract Verification
After the round 2 fixes, AC3's partial verdict (the symlink case) is addressed by finding 2, and the false-loss routes behind AC5 and AC6 are addressed by finding 1. AC6 stays partial only because the Windows touch tests are SKIP_PLATFORM, which was accepted in round 1. Finding 1: [claude] POSIX lock touch reports any validation error as 'lost', which stops the daemon instead of treating the error as transient. The Windows touch maps a GetFileInformationByHandle failure to loss.
Finding 2: [claude] Identity-marker touch treats replacement by a symlink (or by an unreadable same-name file) as transient instead of lost
Finding 3: [claude] The production host participant-guard touch, cbm_daemon_runtime_service_touch_listener, and the round-1 external_guard wiring are never called by a test
Finding 4: [claude] The transient (-1 / IO) result added in round 1 has no test
Finding 5: [claude] The version_cohort touch test checks only status codes, not that timestamps were refreshed
Summary
Fixed: 1, 2, 4, 5. Accepted with rationale: 3. Schema ChangeNone. All three lenses agree. Pre-existing issues discovered
Verification
NEW SAST findings (GitHub code scanning, open alerts on
|
QA Round 3 — cleanContract: DeusData#2178 (AC1–AC7). Panel: 3 Claude lenses plus a synthesizer, none failed. Branch re-synced with upstream All three lenses declare diminishing returns. No lens found a functional defect. One lens returned zero findings. The 6 merged findings are all minor: five test-coverage or test-robustness notes, and one code-hardening asymmetry that no lens could turn into a reachable failure. Contract Verification
AC1–AC5 and AC7 pass. AC6 remains partial for the reason accepted in round 1: the Windows touch entry points are compile- and CI-verified but their behaviour is not exercised by a test (the POSIX-only tests are Finding 1: [claude] Loss-detection-to-stop wiring in host_wait_for_lifetime has no automated test; two new public touch entry points have zero test references
Finding 2: [claude] ipc.c posix_path_names_inode is never exercised for held lock files - only plain unlink (st_nlink==0) is tested for lifetime.lock and the participant guard
Finding 3: [claude] private_held_identity directory-replacement and undetermined (-1) branches have no test
Finding 4: [claude] ipc.c touch path omits the runtime-directory identity check that every other ipc.c lock path (and private_file_lock.c) performs
Finding 5: [claude] Identity-marker LOSS_REPLACE_FILE case can false-pass on filesystems that reuse inode numbers
Finding 6: [claude] Windows DeusData#2178 heartbeat is unverified beyond compilation - all three new tests SKIP_PLATFORM on _WIN32
Summary
Nothing fixed this round: every finding is advisory, and no acceptance criterion fails. Round 1 raised 10 findings (6 fixed), round 2 raised 5 (4 fixed), round 3 raised 0 functional defects. Schema ChangeNone. Verification
NEW SAST findings (GitHub code scanning, open alerts on
|
|
QA complete (3 rounds, round 3 clean). Upstream draft PR opened: DeusData#2289, with these three round reports carried over. This fork PR stays open as the QA trail and is not for merging. |
Add cbm_daemon_host_wait_for_lifetime_for_test, which runs the production lifetime loop over caller-held coordination handles, and a runtime test that ages the claim marker, unlinks the cohort lifetime lock, and asserts the tick refreshes the marker, logs daemon.lifetime_end reason=coordination_file_lost file=cohort_lease, and stops the runtime. The seam runs an ephemeral generation, so a missed loss ends at the initial client window instead of hanging. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbYKGdU7pqG6AEuMQuJxiF Signed-off-by: Shane McCarron <shane.mccarron@corvexconnect.com>
Restore main's formatting on pre-existing lines that a whole-file clang-format pass had reflowed, so the diff carries only the DeusData#2178 change. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbYKGdU7pqG6AEuMQuJxiF Signed-off-by: Shane McCarron <shane.mccarron@corvexconnect.com>
Reflow the host-loop seam call to clang-format and note that the fixture's convenience start joins its own participant guard, unlike the host's start_reserved. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NbYKGdU7pqG6AEuMQuJxiF Signed-off-by: Shane McCarron <shane.mccarron@corvexconnect.com>
Fork-internal QA PR. QA rounds for the upstream fix run here, so each fix commit starts fork CI rather than upstream CI. When a round comes back clean, an upstream draft PR against
DeusData/codebase-memory-mcp:mainwill close DeusData#2178, with these QA reports carried over.Problem
macOS
com.apple.tmp_cleaner(launchd, daily at midnight, or on wake if the machine was asleep) deletes/tmpfiles whose atime, mtime and ctime are all older than 3 days. The daemon's long-held lock files are empty and never written, so under a daemon that lives longer than 3 days the cleaner unlinks them while they are held. The next participant then creates and locks a fresh inode:OK, so it reads the live daemon asUNCOORDINATEDand exits 1 on every reindex (the "pre-cohort/unknown" wedge)..sock.identitylost:posix_listener_artifacts_remove_if_matchesrefuses to unlink anything, so even a cleandaemon.stopleaves a markerless.sock/.ancpair behind.Field evidence (2026-09-21): the daemon started Sep 16, and the daemon/admission cohort locks and the IPC lifetime lock all have a birth time of Sep 21 09:24:05, after the machine woke.
Fix
The host lifetime loop now runs a heartbeat every 60 s over every file the generation holds:
cbm_version_cohort_daemon_claim_touchcbm_version_cohort_lease_touchcbm_daemon_ipc_participant_guard_touchcbm_daemon_ipc_listener_touch(throughcbm_daemon_runtime_service_touch_listener)futimensthrough the held descriptor, and only after proving the path still names that descriptor (st_nlink == 1/ fd-vs-path inode). Refreshing by path would keep a replacement inode fresh and hide the loss.daemon.lifetime_end reason=coordination_file_lost file=<artifact>and stops cleanly, permanent daemons included. The next client then starts a coordinated generation. Re-claiming in place would race that client.Tests
private_file_lock_touch_refreshes_held_file_and_detects_unlink: ages a held file, touches it (timestamps refreshed), unlinks it (touch →UNSAFE), and confirms that a peer EX then succeeds, which is the hazard itself.daemon_ipc_listener_touch_detects_lost_runtime_files: a live listener passes touch; after unlinking the lifetime lock, or in a second case the identity marker, touch reports the loss.CBM_RUNTIME_DIRwith the built binary:coordination_file_lost file=cohort_daemon_claimand a clean stop after 59 s; the socket pair was removed..sock.identity→file=listenerand a clean stop; the nextdaemon startsucceeded (the fix: recover markerless stale rendezvous sockets during activation DeusData/codebase-memory-mcp#1894 reclaim cleared the pair).scripts/test.sh: 8109 passed, 1 failed, 10 skipped. The failure isdaemon_runtimetest_daemon_runtime.c:3938(ASSERT(marker_published)), which also fails intermittently on unmodified upstream/maindef38f3e(4 of 6 runs; this branch 6 of 7), and all three round-1 reviewers confirmed the diff cannot reach it.Refs DeusData#2178 (the upstream PR will carry
Closes #2178).🤖 Generated with Claude Code