feat(watcher): add paused external-wait supervision

[kunchenguid]

Intent

Add a paused/awaiting-external crew state so an in-flight crewmate intentionally idling on a KNOWN external wait (an upstream tool release, a vendor rate-limit reset, a scheduled window) does not trip the possible-wedge stale escalation like a genuine wedge. This was a deferred item from the 2026-07-07 herdr incident (docs/herdr-backend.md), reproduced live on 2026-07-09/10 in this home when a crew held parked awaiting an upstream release and its idle pane tripped repeated 'stale ... possible wedge' escalations all day.

Design decisions and tradeoffs a diff-only reviewer would not know:

• New verb 'paused: ' that a crew (or firstmate steering it) appends. It is DELIBERATELY NOT added to the captain-relevant verb set (FM_CLASSIFY_CAPTAIN_RE_DEFAULT): a pause is a 'stop wedge-nagging this idle pane' signal, not work to keep surfacing. The initial pause still surfaces once via the existing not-provably-working signal path in always-on mode, and is self-handled (routine) in away/afk mode - the intended present-vs-away split.
• One-owner vocabulary in bin/fm-classify-lib.sh: added FM_CLASSIFY_PAUSED_VERB_DEFAULT, FM_PAUSE_RESURFACE_SECS_DEFAULT (3600s, chosen far longer than the 240s wedge threshold but finite so a forgotten pause cannot rot invisibly), status_is_paused, and crew_absorb_class. I intentionally REFACTORED crew_is_provably_working to delegate to the new crew_absorb_class (which returns working|paused|none from ONE fm-crew-state.sh read) so both the watcher and daemon share the classification and there is no duplicated logic; crew_is_paused is a sibling delegating wrapper. This refactor is behavior-preserving for crew_is_provably_working (covered by the existing classifier test).
• Watcher (bin/fm-watch.sh): a declared pause is absorbed like a working crew but on the long PAUSE_RESURFACE_SECS re-surface cadence instead of the wedge timer. The re-surface age is anchored on the pause's own STATUS-FILE mtime (not a per-hash marker) deliberately, so a churny idle pane - a ticking clock or token counter - cannot keep resetting the cadence the way the hash-tied wedge timer would; a .paused-resurfaced- throttle marker makes it fire once per window. This asymmetry with the wedge timer (which does reset on churn) is intentional: a wedge is about a FROZEN pane, a pause is about a DECLARED wait regardless of pane animation. The afk one-shot branch is untouched (daemon owns triage there).
• Daemon (bin/fm-supervise-daemon.sh): classify_stale returns a distinct 'pause' action; handle_wake records a .subsuper-paused- marker and drops any wedge marker (and vice versa) so the two are mutually exclusive; housekeeping adds a pause re-surface recheck on the long cadence that re-surfaces a recheck or clears the marker on resume/unpause. The daemon uses the cheap status_is_paused verb read plus a busy recheck (mirroring its existing wedge path), not an fm-crew-state call, to stay cheap.
• fm-crew-state.sh reports 'state: paused' distinctly (map_log_state) so a supervisor sees a declared pause and its reason rather than a wedge-suspect idle.
• Safety preserved: a crew that goes idle WITHOUT declaring a pause classifies exactly as before (crew_absorb_class returns none -> surfaces); covered by the existing not-working and wedge regression tests, which still pass.
• FM_PAUSE_RESURFACE_SECS_DEFAULT carries a '# shellcheck disable=SC2034' because it is read by the two consumer scripts, not the library itself - matching the repo's existing convention for externally-consumed constants.
• Docs updated consistently: AGENTS.md sections 8 and 11, the three brief-scaffold status state lists in bin/fm-brief.sh (distinguishing paused: from blocked:), and the docs/herdr-backend.md deferral item marked RESOLVED. Tests colocated in tests/fm-watch-triage.test.sh, tests/fm-daemon.test.sh, and tests/fm-crew-state.test.sh. shellcheck bin/.sh bin/backends/.sh tests/*.sh is clean.

This is a firstmate-repo change; per its no-mistakes contract .no-mistakes/ stays gitignored and CI rejects tracked .no-mistakes paths.

What Changed

• Add a configurable paused: crew status for declared external waits, including distinct paused reporting in crew-state output and generated briefs.
• Update the watcher and AFK supervision daemon to suppress wedge escalation during declared waits, recheck them on a bounded long cadence, and restore normal tracking when work resumes or terminates.
• Document paused-wait behavior and configuration, with regression coverage for watcher, daemon, crew-state, and brief transitions.

Risk Assessment

⚠️ Medium: The implementation is a broad, stateful coordination change across normal and AFK supervisors, but the final marker transitions and added regressions cover the material risks identified in prior rounds.

Testing

The supplied baseline full suite had already passed; focused daemon and crew-state suites passed, and controlled production watcher/daemon/tmux scenarios demonstrated that declared pauses suppress wedge escalation, re-surface as awaiting-external checks, and retain actionable AFK daemon handoff. Reviewer-visible transcripts are recorded above.

Evidence: AFK paused-state end-to-end transcript

AFK daemon self-handled stale: crew:fm-external-wait as paused, injected bounded awaiting-external rechecks, and emitted no wedge marker or possible-wedge escalation.

Paused external-wait AFK end-to-end transcript
Input status: paused: awaiting the upstream release
Observed daemon handoff and triage:

[2026-07-10T06:21:12-0700] wake: stale: crew:fm-external-wait
[2026-07-10T06:21:12-0700] self-handle (paused): stale: crew:fm-external-wait -> paused (awaiting external), rechecked on a long cadence: paused: awaiting the upstream release

Supervisor-visible recheck injected after the bounded 3s cadence:
tmux send-keys -t supervisor -l �Supervisor escalate (       1 event(s)): paused 3s (awaiting external, recheck whether the wait still holds): crew:fm-external-wait (pre-read; re-arm not needed — watcher daemon-managed)
tmux send-keys -t supervisor Enter
tmux send-keys -t supervisor -l �Supervisor escalate (       1 event(s)): paused 3s (awaiting external, recheck whether the wait still holds): crew:fm-external-wait (pre-read; re-arm not needed — watcher daemon-managed)
tmux send-keys -t supervisor Enter
tmux send-keys -t supervisor -l �Supervisor escalate (       1 event(s)): paused 3s (awaiting external, recheck whether the wait still holds): crew:fm-external-wait (pre-read; re-arm not needed — watcher daemon-managed)
tmux send-keys -t supervisor Enter

Persisted tracking after recheck:
pause marker: present (window reset for future rechecks)
wedge marker: absent
wedge wording: absent

Result: PASS - daemon received a plain stale window, classified paused, and surfaced an awaiting-external recheck without a wedge escalation.

Evidence: Watcher pause re-surface transcript

A fresh declared pause was absorbed, then re-surfaced after the shortened pause cadence as awaiting external, never as a wedge.

Paused external-wait normal watcher end-to-end transcript
Input status: paused: holding for the upstream tool release
Fresh declared pause: watcher remained running=1; queued wake=0; wedge timer present=0
Bounded re-surface after status age exceeded the 3s test cadence:
stale: crew:fm-normal-external-wait (paused 10s, awaiting external - declared pause, rechecked on a long cadence not a wedge; confirm the wait still holds)
wedge wording: absent
Result: PASS - a fresh pause is absorbed without wedge timing and later re-surfaces as an awaiting-external recheck.

Evidence: Crew-state paused output transcript

Crew-state output reports the declared paused state and reason distinctly.

Paused crew-state end-to-end transcript
Input status: paused: awaiting the vendor rate-limit reset
Supervisor read: state: paused · source: status-log · awaiting the vendor rate-limit reset
Result: PASS - the production current-state reader exposes the declared external wait as paused.

- Outcome: 🔧 1 issue found → auto-fixed ✅ across 2 runs (1h0m39s)

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

🔧 **Review** - 2 issues found → auto-fixed (7) ✅

• 🚨 bin/fm-watch.sh:601 - Pause mode is decided only for a new pane hash. If firstmate appends paused: to an already-stale, unchanged pane, sf == h bypasses crew_absorb_class and retains the wedge timer, producing a possible wedge; AFK similarly retains its existing stale marker because no new stale wake is emitted. Reclassify or reset stale state when status enters or leaves pause.
• ⚠️ bin/fm-crew-state.sh:129 - The configurable pause verb is not honored here. status_is_paused accepts FM_CLASSIFY_PAUSED_VERB, but this fallback maps only literal paused, so a custom verb is paused in the daemon but becomes state: unknown for the always-on watcher. Use the shared configured verb or helper.

🔧 Fix: Captain: fix paused-state transitions
3 errors still open:

• 🚨 bin/fm-watch.sh:657 - A paused pane whose output changes every poll never reaches handle_paused_stale: this branch clears the pause flag and resets the stale hash. A ticking clock or token counter, explicitly meant not to reset pause cadence, can therefore prevent any bounded re-surface.
• 🚨 bin/fm-supervise-daemon.sh:906 - AFK mode records a pause marker only for a stale: wake. A new paused: append arrives as a routine signal:, and if the watcher already has the same stale hash from before AFK it emits no later stale wake. The daemon then never performs the promised long-cadence recheck.
• 🚨 bin/fm-watch.sh:629 - Once a pause flag exists, this path trusts the append-only paused: tail without consulting crew_absorb_class. If the crew starts a validation run without another status append and later becomes idle, the active run is hidden on the hour-long pause cadence instead of returning to wedge tracking, despite fm-crew-state.sh treating run state as authoritative.

🔧 Fix: Captain: fix paused-state supervision transitions
3 errors still open:

• 🚨 bin/fm-supervise-daemon.sh:479 - When a paused task appends a non-paused status such as working:, this removes .subsuper-paused-* without restoring normal stale tracking. The watcher has already suppressed the unchanged idle hash, so AFK housekeeping has no marker to recheck and a resumed-but-idle task can evade wedge escalation indefinitely.
• 🚨 bin/fm-supervise-daemon.sh:726 - Entering AFK after normal-mode pause absorption does not seed .subsuper-paused-*. The retained watcher stale hash prevents a new stale wake, while housekeeping only scans daemon pause markers, so the promised long-cadence recheck never begins.
• 🚨 bin/fm-brief.sh:154 - Secondmates are instructed to use paused:, but normal-mode watcher processing skips kind=secondmate before pause handling. Their initial signal surfaces once, then no periodic pause recheck is possible outside AFK.

🔧 Fix: Captain: fix paused supervision handoffs
3 errors still open:

• 🚨 bin/fm-supervise-daemon.sh:474 - A terminal transition such as paused: -> done: or blocked: replaces the pause marker with a normal stale marker. The signal escalates, but this marker remains and housekeeping later emits a false possible wedge for the intentionally idle terminal pane. The watcher-marker migration has the same outcome.
• 🚨 bin/fm-watch.sh:554 - A secondmate that leaves paused: is skipped before its existing watcher pause markers are cleared. If AFK begins later, migration converts that stale marker into normal stale tracking and the healthy idle secondmate is escalated as a possible wedge.
• 🚨 bin/fm-supervise-daemon.sh:485 - AFK handoff only migrates an existing watcher .paused-* flag. If a pause signal was surfaced in normal mode before the next stale poll creates that flag, and the pane hash was already suppressed, AFK receives neither a new stale wake nor a daemon pause marker. The declared wait then never gets its bounded recheck.

🔧 Fix: Reconcile paused supervision markers
1 error still open:

• 🚨 bin/fm-classify-lib.sh:73 - status_is_captain_relevant applies its unanchored patterns to the entire status line before pause handling. A valid reason such as paused: waiting for upstream checks green becomes captain-relevant: AFK signal handling escalates it, while normal stale handling takes the terminal path before pause absorption. Give a leading pause verb precedence over captain-relevance so a pause's behavior cannot depend on its reason text.

🔧 Fix: Captain: prioritize paused states over captain relevance
1 error still open:

• 🚨 bin/fm-watch.sh:675 - When the status remains paused: but the authoritative reader reports working, this repeat-stale path clears pause tracking and recreates .stale-since-* on every poll. A frozen unchanged active run therefore never reaches FM_STALE_ESCALATE_SECS; preserve the wedge timer across these periodic working rechecks.

🔧 Fix: Captain: preserve paused-working wedge timer
1 warning still open:

• ⚠️ bin/fm-brief.sh:155 - The classifier supports FM_CLASSIFY_PAUSED_VERB, but all generated briefs still instruct crews to append literal paused:. With a configured verb such as awaiting, crews emit an unrecognized state and their declared wait can resume normal stale/wedge handling. Render the configured verb in every brief scaffold.

🔧 Fix: Captain: honor configured pause verb in briefs
✅ Re-checked - no issues remain.

🔧 **Test** - 1 issue found → auto-fixed ✅

• 🚨 bin/fm-watch.sh:708 - In AFK mode, the changed paused-pane path runs before the watcher’s one-shot daemon handoff, suppressing that handoff. Its later re-surface sends stale: <window> (paused ...); the daemon treats the decorated text as the window identity, cannot find status, and drops it as transient stale. This leaves paused AFK triage incomplete.
• command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; for t in tests/*.test.sh; do echo "== $t =="; bash "$t" || rc=1; done; exit "$rc"
• Provided successful baseline: command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; for t in tests/*.test.sh; do echo "== $t =="; bash "$t" || rc=1; done; exit "$rc"
• bin/fm-session-start.sh
• bash tests/fm-watch-triage.test.sh
• bash tests/fm-daemon.test.sh
• bash tests/fm-crew-state.test.sh
• bash tests/fm-brief.test.sh
• Real-tmux normal paused watcher scenario with short FM_PAUSE_RESURFACE_SECS, plus bin/fm-crew-state.sh and bin/fm-wake-drain.sh evidence capture
• Real-tmux AFK watcher scenario verifying the expected daemon handoff is absent
• Real-tmux bin/fm-supervise-daemon.sh scenario verifying the decorated paused re-surface is rejected as unresolved transient stale

🔧 Fix: Captain: fix AFK paused watcher handoff
✅ Re-checked - no issues remain.

• command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; for t in tests/*.test.sh; do echo "== $t =="; bash "$t" || rc=1; done; exit "$rc"
• Baseline full suite supplied by the task: command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; for t in tests/*.test.sh; do echo "== $t =="; bash "$t" || rc=1; done; exit "$rc" (previously successful)
• bash tests/fm-daemon.test.sh
• bash tests/fm-crew-state.test.sh
• bash tests/fm-watch-triage.test.sh - all scheduled assertions, including paused AFK handoff, appeared in the captured log without not ok; the runner did not return its final wrapper exit line
• Production bin/fm-supervise-daemon.sh with a production watcher child and controlled idle tmux pane in AFK mode
• Production bin/fm-watch.sh with controlled pause absorption and bounded re-surfacing
• bin/fm-crew-state.sh state-wait with controlled paused status

✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.