fix(hook-augment): cache the image fingerprint per process and announce a missed deadline on stderr

[bmcnaboe]

The build-identity fingerprint is a SHA-256 over the entire ~295 MB executable (~1.1 s per hash on an M5 Pro, ~2.3 s on the machine where this was first measured). A daemon pays it twice at start (the supervisor's startup capture and the runtime service's active-image check), and a peer whose image is not inode-identical to the daemon's own is re-hashed in full on every rendezvous.

• Cache the fingerprint per process at runtime_process_image_reference_acquire, keyed by (device, inode, size, mtime, ctime): the tuple the acquire path already verifies as stable across the hash, so a rebuilt or replaced binary rolls the key. Every hit is still bracketed by the same before/after stat and process-maps checks, so a cached digest stays bound to the verified image.
• A missed hook deadline is announced on stderr as well as the timeouts log, so a fired deadline is never a silent 0-byte exit 0. The breadcrumb is formatted before the log path is resolved, so stderr gets it even when the log cannot be opened. CBM_HOOK_DEADLINE_MS is unchanged.
• daemon <start|stop|status> is listed in --help.

SHA-256 itself is unchanged: one portable scalar implementation on every platform (the earlier CommonCrypto path was dropped at review).

Measured (M5 Pro, 296 MB image, sandboxed profile, src/ of this repo indexed)

	main (339b3f4)	this branch
cli index_repository (spawns the internal daemon)	6471 ms	5198 ms
daemon start → ready	6497 ms	5329 ms
hook-augment against the warm daemon, 5 runs	1140–1151 ms, context emitted	1129–1160 ms, context emitted

The cache removes the daemon's second start-up hash. A cold hook-augment hashes exactly once on both binaries, so it is unchanged by the cache; whether it fits the 2 s budget is a function of the per-machine scalar hash cost. That is why this no longer claims to close #1335 on its own.

Tests

• daemon_runtime_fingerprint_cache_hit_miss_key_roll: miss → hit → key roll on mtime and on inode, via a test seam that stubs the image hash and counts calls.
• cli_hook_augment_deadline_breadcrumb_issue858 extended: the breadcrumb is asserted on stderr as well as in the log.
• scripts/test.sh --suites "cli daemon_runtime" (ASan+UBSan) and scripts/smoke-local.sh pass locally on arm64.

The sanitized-build timeout backstops that used to ride along here live in #1742.

Refs #1335, #2058

[github-actions]

Thanks for opening this — it has been seen, and it is queued.

This note is automated, but it is not a brush-off: it exists so you know where your PR stands instead of having to guess from silence.

Current review status: working through a backlog. 0.9.1-rc.1 is out, so the release freeze that held reviews is over — but it left a large queue of open pull requests behind it, and we are reading through them oldest-first. The background is in discussion #1144.

What that means for this PR, concretely:

• It will not be closed for inactivity. No stale bot touches pull requests here.
• It may still sit a while before a human reads it. That is on us, not on you.
• Older PRs are read first, so a recent one is not being skipped — it is behind a queue.

Things that will genuinely speed it up whenever review does happen:

• Keep it rebased on main — the tree is moving quickly right now, and a conflicting branch cannot be reviewed as the diff you intended.
• Get CI green, or say which failures you believe are pre-existing.
• Keep the change to one claim. Bundled features and refactors get split before they get merged, which costs you a round trip.
• Every commit needs a sign-off (git commit -s) — CI enforces DCO.

If this fixes a bug, a reproduction we can run is worth more than a description of the symptom.

Thanks for contributing, and sorry in advance for the wait.

[DeusData]

Thank you for measuring the fingerprint cost, covering cache rollover, and making missed hook deadlines observable. Runtime identity caching and platform SHA paths touch startup and trust boundaries, so we need more time to review them carefully. The contribution queue is quite full, but we will return with grounded feedback as soon as possible.

[DeusData]

Reviewed. The diagnosis is excellent and it closes the other half of a symptom this repo has chased before. One CI failure I am not willing to call environmental, and I would rather say why than wave it through.

The substance

This is the second cause of the same silent failure. The comment above ha_deadline_ms records #858: the original 300 ms budget self-terminated on cold starts so augmentation "never appeared in real sessions (0/24 observed)". #1880 found that a typo in CBM_HOOK_DEADLINE_MS silently clamps to 50 ms. You have found that the hook spends ~2.3 s SHA-256'ing its own ~295 MB executable before doing anything at all. Three different routes to a hook that emits nothing and exits 0.

Keying the cache on (device, inode, size, mtime, ctime) is the right identity. A rebuilt binary rolls every one of those, so the cache cannot serve a stale fingerprint for a different image — which is the failure mode that would make this worse than no cache.

Asserting the CommonCrypto path bit-identical to the scalar one is what makes the hardware path safe to take. A faster hash that disagrees on one input would be a very unpleasant bug to find later, and cli_sha256_platform_path_matches_scalar is the test that stops it.

But the most valuable line in this PR is the third bullet. Recording a missed deadline on both the timeouts log and stderr, "never a silent 0-byte exit 0", is what turns this class of problem from invisible into reportable. Even with the cache, some future machine will be slow enough; the difference is whether anyone can tell.

The failure I am not attributing

pr-smoke (macos-14) fails at scripts/smoke-test.sh:549:

FAIL: config get auto_watch printed '' (expected the stored value or the default 'true')

I checked, and I cannot honestly call this unrelated:

• auto_watch is a real key (CBM_CONFIG_AUTO_WATCH, cli.h:425), so the test is not stale.
• You add zero lines touching config handling, which argues it is not yours.
• But the assertion runs config get ... 2>/dev/null || true, so an empty result means the binary failed or printed nothing — not that the key is missing. And this PR changes what happens at process start, in runtime_process_image_reference_acquire, for every command including config get, against a fresh CBM_CACHE_DIR.

That is a plausible mechanism, and "adds no config code" does not rule it out.

The run is also from 20 August, and main has moved a great deal since — it was briefly broken by a duplicate-symbol merge yesterday, repaired by #1993, and #1703 landed after that. So the result may simply be stale.

Please rebase and let it re-run. If it clears, it was the base. If it persists, the place to look is fingerprint acquisition on a cold cache directory — a startup path failing before config get can print is the shape that would produce exactly this output.

This is also DIRTY, so a rebase is needed regardless.

Three well-argued fixes to the hook-augment path from you now. Thank you — this one has the most user-visible payoff of the three.

[DeusData]

The pr-smoke (macos-14) red is real, it is reproducible, and it is caused by your speed-up — but not by a mistake in it. I chased it to the bottom locally on an arm64 Mac, so you don't have to guess from a swallowed stderr.

What the smoke sees

Phase 3z3 runs CBM_CACHE_DIR=<fresh dir> codebase-memory-mcp config get auto_watch immediately after Phase 3z2, which drives the MCP stdio server through 24 pipelined calls and lets it exit. On your build that config get prints nothing and exits 1 — twice in a row across your last two runs, while the six other PRs re-validated against the same main pass it. Ubuntu and Windows pass on your branch too.

What stderr actually says

codebase-memory-mcp: CBM could not start because the active account daemon uses a different
cache directory (active cache 91a55c…; requested cache 92e0e2…). Close all CBM sessions and
commands, then retry.

Mechanism (measured)

The MCP stdio session auto-starts --cbm-daemon-internal, and that daemon outlives its last client by a short window — this is pre-existing on main:

binary	daemon gone after client exit
main	0.56 s / 0.58 s / 0.57 s
this PR	0.31 s / 0.31 s / 0.30 s

The next one-shot CLI (config get is LOCAL_CLI) computes its own build fingerprint in main_build_identity() before it looks for a daemon. On main that is the scalar SHA-256 of the whole executable — ~1 s here, ~3–4 s on the macos-14 runner — which is longer than the daemon's linger, so the CLI never meets it. Your CommonCrypto path does the same hash in ~0.1 s, so the CLI arrives while the old daemon is still visible, the version-cohort acquisition sees a live daemon bound to a different cache directory, and refuses hard. Control experiment on your binary, same repro:

• no pause between 3z2 and 3z3 → printed='' rc=1 (2/2)
• sleep 1 between them → printed='true' rc=0

Linux/Windows keep the scalar hash, so the mask stays in place there — which is exactly why only the macOS leg goes red.

Why this is yours to close, and how

The smoke assertion is a real production race that your speed-up exposes deterministically: any command under a different CBM_CACHE_DIR issued within ~0.5 s of an MCP session ending is refused. Removing that startup cost is the whole point of the PR, so it needs to land with the race closed rather than re-masked. Two production-side shapes that fit:

1. Deterministic hand-off on the daemon side — the stdio client does not return until the internal daemon has released its cohort/endpoint registration (or the daemon's exit is made synchronous with its last client). This is the wait-for-the-asserted-state shape and is what I'd lean to.
2. Tolerant admission on the CLI side — cohort acquisition treats a daemon whose last client is gone (draining) as absent, or retries a cache-directory conflict against a draining daemon within its existing MAIN_STARTUP_TIMEOUT_MS deadline instead of failing on first sight.

What is not on the table is a sleep in the smoke, or budget-tuning the linger — the race would still be there for real users on fast machines.

Local repro that reproduces it 2/2 (CBM_RUNTIME_DIR just isolates you from your own installed daemon):

export CBM_RUNTIME_DIR=$(mktemp -d)
python3 - "$BIN" <<'PY'   # the smoke's own Phase 3z2 snippet
import json, subprocess, sys
BIN=sys.argv[1]; rpc=lambda i,m,p: json.dumps({"jsonrpc":"2.0","id":i,"method":m,"params":p})
lines=[rpc(1,"initialize",{"protocolVersion":"2025-06-18","capabilities":{},"clientInfo":{"name":"smoke-pipe","version":"0"}}),
       json.dumps({"jsonrpc":"2.0","method":"notifications/initialized"})]
lines+=[rpc(100+i,"tools/call",{"name":"list_projects","arguments":{}}) for i in range(24)]
subprocess.run([BIN],input="\n".join(lines)+"\n",capture_output=True,text=True,timeout=300)
PY
CBM_CACHE_DIR=$(mktemp -d) "$BIN" config get auto_watch; echo "rc=$?"

Everything else on your branch is green apart from the standing test-windows-guards red, which is ours. The substance of the PR still reads well to me — and finding this race is a real contribution in its own right; a slow hash was hiding a genuine lifecycle bug.

[DeusData]

Closing the loop on the macOS pr-smoke red: it is not your change, but your change is what exposed it — and that turned out to be useful.

The failing config get under a fresh CBM_CACHE_DIR arrives a few hundred milliseconds after the internal daemon's last stdio client exits. In that window the draining daemon still holds the cohort lifetime lock, so the CLI reads a holder record with a different cache root and cbm_version_cohort_acquire refused immediately instead of honouring its deadline. Main's slow scalar self-hash happened to push the CLI past that window; your CommonCrypto hash removes the accidental margin and the race shows up on every run. Filed as #2046 with the full trace.

The fix lives in #2047 (the cohort acquire now retries a mismatched holder until the caller's deadline, the same handoff wait host.c already does for the claim marker). With that applied on top of your head, the previously failing sequence passes 3/3 locally. So please don't work around it in this branch — once #2047 lands I will update this PR against main and re-run the matrix, and this branch stays exactly the hash change it is.

Thanks for the patience on this one, and for a change that flushed out a real lifecycle bug on its way in.

[DeusData]

Thank you for this — the fingerprint cache is exactly the right idea for the hook-budget problem (#1335/#2058), and the design is genuinely careful: keying on (dev, inode, size, mtime, ctime) is a nice touch, since ctime moves on any in-place write and can't be rolled back with utimes, so a same-size/reset-mtime tamper still misses; and every cache hit is still bracketed by the acquire path's stat + process-maps checks, so a hit stays bound to the verified image. That's the security-relevant part and it holds up.

One change we'd like before this lands: please drop the Apple CommonCrypto SHA-256 path (src/foundation/sha256.c/.h) and keep just the cache. The cache alone solves the deadline — once the fingerprint is memoized, the ~2.3 s hash runs at most once per process, so CommonCrypto only speeds a cache miss. For a lean C codebase, a permanent macOS-only, platform-divergent crypto path is a maintenance cost we'd rather not take on for that marginal miss-path win. Keeping a single scalar SHA-256 everywhere keeps the behaviour identical across platforms.

Two smaller notes: (1) CI is red only on the pr-smoke (macos-14) lane (the rest of the matrix — full unit suites on macOS/Linux/Windows, tsan/msan, CodeQL, lint, the security gates — is green), so it looks isolated to the macOS smoke/packaging step; could you take a look? Happy to help triage if it's not obvious. (2) The sanitized-build test-timeout backstops in tests/test_daemon_runtime.c are fine in principle (upper-bound liveness, not the assertion), but they're a bit orthogonal to the caching change — your call whether to keep them here or split them out.

No rush — flip it out of draft whenever you're ready and we'll take another pass. Really appreciate the careful work on this.

[bmcnaboe]

Thanks — all three addressed, plus one measurement you should see before the next pass.

CommonCrypto is gone. src/foundation/sha256.{c,h} are byte-identical to main again; what remains is the per-process fingerprint cache, the stderr breadcrumb, and the --help line. Rebased onto main (339b3f4) as one signed-off commit, so the conflict is cleared.

pr-smoke (macos-14) was the #2046 race you traced on Sept 3: the CommonCrypto hash let config get reach the draining internal daemon inside its linger window. #2047 (merged Sept 4) retries a mismatched cohort holder until the caller's deadline, and this rebase picks that up; with the scalar hash back, the branch's one-shot CLI start-up timing is main's again anyway. scripts/smoke-local.sh passes locally on arm64 (3z2 → 3z3 included).

Backstops are out of this PR; they already live in #1742, unchanged.

The measurement. With the hardware path gone I timed both binaries in a sandboxed profile (M5 Pro, 296 MB image, src/ of this repo indexed, warm permanent daemon):

	main (339b3f4)	this branch
cli index_repository (spawns the internal daemon)	6471 ms	5198 ms
daemon start → ready	6497 ms	5329 ms
hook-augment, 5 runs	1140–1151 ms	1129–1160 ms

The cache saves the daemon's second start-up hash (the supervisor capture plus the runtime service's active-image check) and the full re-hash of any peer that is not inode-identical. But a cold hook-augment hashes exactly once on main and on this branch — the peer check on the same inode is metadata-only, so there is no second hash for the cache to absorb — and its latency is unchanged: one scalar hash per invocation (1.15 s here, ~2.3 s on the machine that first hit the deadline). So "the cache alone solves the deadline" doesn't hold for the cold hook; whether it fits the 2 s budget is now purely the per-machine hash cost. I've retitled the PR and dropped Fixes #1335 to match what it delivers; the stderr breadcrumb is what makes the miss visible where it still happens.

If you want the cold hook's hash gone without a platform crypto path, the shape I'd propose as a follow-up is persisting the same-keyed digest in the owner-only account cache root (already the storage authority for admission), so a cold process skips the hash when the tuple matches and a rebuilt binary still rolls it. That is a trust-boundary call, so I'd rather have your steer before writing it.

Out of draft and ready for another pass.

[bmcnaboe]

CI note: the one red lane, test / test-windows-guards, is section_cold_storm failing at endpoint creation before any code this PR touches (same failure on main on Sept 12, tracked in #2057); a rerun should clear it if you can trigger one.