Harden agent transport recovery and diagnostics

[kjgbot]

Root cause

Direct agent CLIs were always spawned with a piped stdin. The worker opened the
sidechannel around startup and closed the unused pipe after a delay, but Codex
had already detected non-TTY stdin and entered its additional-input lifecycle.
When that process exited, the SDK retained only a generic nonzero code and
stderr. The kernel then conflated semantic iteration with infrastructure
recovery: crashes consumed no semantic iteration and therefore had no separate
bound, while f.agent() exposed no recovery controls.

Change

• Open the sidechannel before spawn and decide the child stdin contract during a
  bounded pre-spawn enrollment window. An unattended CLI starts with stdin
  ignored/EOF; only an already-connected drive peer gets a pipe.
• Journal bounded, redacted direct-transport evidence: phase, cause, exit code,
  signal, safe OS error code, retryability, and stderr tail.
• Classify only a closed set of transport failures as crashed: signal close,
  close without status, selected transient spawn errors, and the exact historical
  Codex stdin-lifecycle signature. Ordinary nonzero exits stay terminal
  worker_error; timeout stays timeout.
• Add maxIterations, transportRetries, and recoveryMode to f.agent(),
  authored lowering, YAML/kernel compilation, reverse compilation, diagnostics,
  docs, and examples.
• Split semantic and transport retry budgets in the kernel. The transport budget
  is deterministic and bounded across both reported crashes and abandoned
  leases; replacement attempts preserve the idempotency key and pinned revisions.
• Add the direct Codex lifecycle fixture plus SDK, authored-flow, compiler,
  diagnostic, kernel, pin/idempotency, exhaustion, and crash/resume tests.

Compatibility and safety

• maxIterations remains 1 by default.
• transportRetries defaults to one additional classified infrastructure
  attempt, preserving the existing one-crash resume contract while removing the
  former unbounded loop. Explicit 0 disables transport recovery.
• The SDK omits an undeclared transport budget, preserving legacy canonical
  authored specs and hashes; an explicitly authored zero round-trips.
• reset, inspect, and manual retain the RFC Appendix A recovery meanings.
• The authored root now has one semantic iteration plus seven transport retries,
  so a returned body worker_error cannot replay semantic side effects while a
  genuinely lost process remains resumable.

Verification evidence

Rust workspace

Command:

export PATH="$HOME/.rustup/toolchains/stable-aarch64-apple-darwin/bin:$PATH" CARGO_INCREMENTAL=0 CARGO_PROFILE_TEST_DEBUG=0
../ops/cargo.sh test --workspace --quiet

Captured result lines, in test-binary execution order (exit 0):

test result: ok. 51 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
test result: ok. 10 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
test result: ok. 40 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
test result: ok. 6 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
test result: ok. 67 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
test result: ok. 10 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
test result: ok. 32 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

The toolchain also emitted non-fatal rust-objcopy warnings because its
libLLVM.dylib is absent; the command exited 0 and every test binary above
completed.

SDK supported runtime suite

Command:

export RELAYFLOWD_BIN="$HOME/.relayflows-toolchain/target/1166253295/debug/relayflowd"
npx vitest run --exclude tests/authored-node-runtime.test.ts

Captured output:

Test Files  145 passed | 1 skipped (146)
Tests  2357 passed | 3 skipped (2360)
Duration  100.20s

SDK typecheck and build

Command:

npm run typecheck && npm run typecheck:tests && npm run build

Captured output (exit 0):

> @relayflows/sdk@2.0.22 typecheck
> tsc --noEmit && tsc -p tsconfig.type-tests.json

> @relayflows/sdk@2.0.22 typecheck:tests
> tsc -p tsconfig.tests.json

> @relayflows/sdk@2.0.22 build
> tsc && node scripts/make-cli-executable.mjs

Surface and packed-consumer gates

Commands:

cd packages/surface && npm test
bash scripts/surface-package-gate.sh

Captured output:

Test Files  9 passed (9)
Tests  46 passed (46)
PACKED_RUNTIME_OK name=packed-runtime-consumer completionReason=success
PACKED_RUNTIME_REFUSAL_OK invalidHeaders=9 forgedHandle=refused
PACKED_TYPESCRIPT_OK
Test Files  1 passed (1)
Tests  26 passed (26)

packages/surface build, main typecheck, and regression typecheck also exited 0.

Review follow-up at exact head 139690f

The review delta closes the manual-recovery gap: worker-reported crashed or lease_expired completions now park under recoveryMode: manual before consulting the retry budget, including a zero budget. The wait.human append uses the journaled start pin, and resume repairs a process death between the park placeholder and wait append idempotently. The general timer test again requires a retryable crash, max_transport_retries is always journaled, and older journals still read through the serde default. Internal helpers remain crate-private.

Committed literal evidence is under kernel/evidence/501/, including mutation-transcript.txt (red mutations, checksummed restore, green rerun), green-kernel.txt, green-sdk-authored-node-runtime.txt, and codex-live-probe.txt.

Exact-head local commands and captured result excerpts:

$ ../ops/cargo.sh test --workspace --quiet
...
test result: ok. 53 passed; 0 failed
test result: ok. 41 passed; 0 failed
test result: ok. 72 passed; 0 failed
exit_code=0

$ ../ops/cargo.sh test -p relayflowd-core --lib -- machine::recovery_tests
test result: ok. 3 passed; 0 failed
$ ../ops/cargo.sh test -p relayflowd --test manual_recovery
test result: ok. 2 passed; 0 failed
$ ../ops/cargo.sh test -p relayflowd --test crash_resume manual_recovery
test result: ok. 1 passed; 0 failed; 40 filtered out

$ npm ci --ignore-scripts
$ npm install --no-save --package-lock=false --ignore-scripts ../../dist/publish/relayflows-surface-2.0.22.tgz
$ npm run typecheck && npm run typecheck:tests && npm run build
exit_code=0

$ RELAYFLOWD_BIN=... npx vitest run tests/pty-sidechannel.test.ts tests/worker-cli.test.ts tests/authored-flow.test.ts tests/step-failure-diagnostic.test.ts
Test Files  4 passed (4)
Tests  82 passed (82)

$ RELAYFLOWD_BIN=... npx vitest run tests/authored-root.test.ts tests/authored-agent-permissions.test.ts tests/authored-run-failure-evidence.test.ts tests/deterministic-llm.test.ts tests/spec-parity.test.ts tests/verb-field-lint.test.ts tests/agent-transcript-live.test.ts
Test Files  7 passed (7)
Tests  185 passed (185)

$ cd packages/schema && npm test -- --run tests/parity.test.ts
80 pass
0 fail

Repository-wide gate blockers (reported, not hidden)

• ../ops/cargo.sh fmt --all -- --check exits 1 on existing formatting drift
  across files including relayflowd/src/engine/remote.rs, relayflowd/src/lib.rs,
  crash fixtures, schema, memory, and state. First captured hunk:

Diff in /Users/khaliqgant/Projects/AgentWorkforce/flows/kernel/relayflowd/src/engine/remote.rs:434:
-                answer.insert(
-                    "attribution".to_owned(),
-                    Value::from("client_asserted"),
-                );
+                answer.insert("attribution".to_owned(), Value::from("client_asserted"));

• ../ops/cargo.sh clippy --workspace -- -D warnings exits 101 on three existing
  collapsible_if findings:

relayflowd-core/src/schema.rs:125
relayflowd-core/src/spec.rs:77
relayflowd-core/src/memoization.rs:102

• The excluded standalone authored-node-runtime suite cannot launch under the
  installed Node 26.7.0:

/opt/homebrew/Cellar/node/26.7.0/bin/node: bad option: --experimental-transform-types

• packages/surface npm run typecheck:examples has existing URL-global errors:

../../workflows/stuck-run-triage.flow.ts(77,12): error TS2304: Cannot find name 'URL'.
../../workflows/stuck-run-triage.flow.ts(79,15): error TS2552: Cannot find name 'URL'. Did you mean 'url'?

No merge is requested; this PR is ready for independent review at commit
139690f754805ee961bb7f12e2c5d0f4f8b72f46.

---

Note

High Risk
Changes kernel agent completion, recovery mode, and journal semantics—incorrect logic could redispatch when a human park was intended or leave runs stuck without a repairable wait.

Overview
Closes a manual recovery gap: when an agent step uses recoveryMode: manual, worker-reported crashed / lease_expired completions now park with wait.human (using journaled start pins for diff_ref) instead of taking the transport retry path—matching abandonment handling. manual_park_wait is shared so the two death paths cannot drift.

Transport vs semantic retry is enforced in completion_actions: only classified transport failures can consume max_transport_retries; ordinary worker_error stays terminal. max_transport_retries is always written on step.attempt.started (including 0).

Resume repairs a torn manual park (park completion journaled but wait.human missing) by appending the wait once from existing journal facts.

Docs/README add f.agent maxIterations / transportRetries / recoveryMode, PTY pre-spawn drive enrollment, and richer failure diagnostics. Regression tests cover manual park, torn-park repair, and bounded transport retry; kernel/evidence/501/ captures mutation and green-run logs.

^{Reviewed by Cursor Bugbot for commit b266979. Bugbot is set up for automated code reviews on this repo. Configure here.}

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: c7518982-93ee-44ce-bb17-6e83efc46669

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[kjgbot]

Surface build/typecheck evidence for the success claim in the PR body.

Command:

npm run build && npm run typecheck && npm run typecheck:regressions

Captured output (exit 0):

> @relayflows/surface@2.0.22 build
> tsc

> @relayflows/surface@2.0.22 typecheck
> tsc --noEmit

> @relayflows/surface@2.0.22 typecheck:regressions
> tsc -p ../../regressions/tsconfig.json && tsc -p tsconfig.test.json && node scripts/check-generated-helpers.mjs

HELPERS_GENERATED_OK airtable.ts, asana.ts, azure-blob.ts, box.ts, calendly.ts, clickup.ts, clients.ts, cloudflare.ts, confluence.ts, daytona.ts, docker-hub.ts, dropbox.ts, fathom.ts, gcp.ts, gcs.ts, github.ts, gitlab.ts, gmail.ts, google-calendar.ts, google-drive.ts, granola.ts, hubspot.ts, index.ts, intercom.ts, jira.ts, linear.ts, mailgun.ts, mixpanel.ts, neon.ts, notion.ts, onedrive.ts, pipedrive.ts, postgres.ts, posthog.ts, providers.ts, ramp.ts, recall.ts, reddit.ts, redis.ts, s3.ts, salesforce.ts, segment.ts, sendgrid.ts, sharepoint.ts, shopify.ts, shortcut.ts, slack.ts, stripe.ts, teams.ts, telegram.ts, webhook-server.ts, x.ts, zendesk.ts

[kjgbot]

Follow-up after the schema validation check found an omitted generated artifact.

Fix: committed regenerated packages/schema/flows.schema.json, constrained both authoring transportRetries and kernel max_transport_retries to integer/minimum 0, and added negative, fractional, and explicit-zero parity cases. New exact head: f3bd47f.

Command:

cd packages/schema
npm test

Captured output:

(pass) structural parity: negative transport retry
(pass) structural parity: fractional transport retry
(pass) structural parity: zero transport retry

80 pass
0 fail
3940 expect() calls
Ran 80 tests across 2 files. [2.19s]

Deterministic regeneration evidence from the same suite:

(pass) regeneration is byte-stable and committed schema has not drifted [283.30ms]

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit f3bd47f. Configure here.}

[kjgbot]

maintainability lens — FAIL

PR #501 — Maintainability review

Blockers

• kernel/relayflowd-core/src/machine/tests.rs:363-368 (all_backing_off_steps_return_timers). The test's precondition changed from failure_reason: Some(CompletionReason::WorkerError) / detail = "stub rejection" to None + no detail. The point of that test is to prove that every failing lane in a parallel spec produces a wake timer; under the new rules WorkerError is terminal, so instead of choosing a transport failure that still retries, the test now feeds a successful completion. Its name and comment still describe "backing off," but a regression in transport-failure backoff scheduling will no longer trip it. Either restore a retryable failure case (Crashed / LeaseExpired) or rename+re-scope this test — silent coverage loss on a load-bearing scheduling path.

Concerns

• kernel/DESIGN.md:61 vs. kernel/relayflowd-core/src/entry.rs:207-215. DESIGN.md says the journaled max_transport_retries field is "omitted at that default" (default=1). The AttemptStartedPayload uses skip_serializing_if = "is_zero_u32" — it omits zero, not the default. A reader auditing journal payloads will find the field always present (1) except when explicitly zero. Meanwhile spec.rs:678-686 skips on the actual default (is_default_max_transport_retries). Two different skip rules for the same conceptual field, plus prose that contradicts one of them, is exactly the kind of trap this diff is meant to prevent.

• packages/sdk/src/worker-cli.ts:461-467 (codex_stdin_lifecycle). Retryability turns on an exact stderr string match ("Reading additional input from stdin...") with no reference to a Codex version, source link, or fallback signal. Any upstream Codex change silently reclassifies this as nonzero_exit → terminal, undoing the whole point of the new transport class. Please pin the Codex version this shape was observed against and journal a warning if this branch ever doesn't fire.

• Retry classification is split across two layers. worker-cli.ts sets transport.retryable, cli-transport-evidence.ts:agentCompletionReason uses it to pick 'crashed', and machine.rs:completion_actions re-derives retryability from CompletionReason::{Crashed, LeaseExpired}. Six-months-later reader must trace three files to answer "why did this attempt retry?" — worth a paragraph in DESIGN.md tying the SDK classifier, the completion-reason alphabet, and the kernel predicate together.

• packages/sdk/src/worker.ts:156-163 duplicates redacted stderr. transport.stderr_tail is already bounded/redacted in transportEvidence, but the wrapper's stderr_tail is re-redacted+re-bounded with a 'worker stderr: ' prefix. Two truncated copies of the same bytes travel into trajectory_tail; a future stderr policy fix will need to be applied twice.

Notes

• packages/sdk/src/authored-root.ts:291 now grants transportRetries: 7, but no test in authored-root.test.ts exercises the actual transport-retry path — only that a body failure terminalizes at attempt 1. The 7-budget is currently dead-lettered from a test perspective.
• README.md:15-18 restates every default explicitly, which teaches readers the wrong reflex for a system built around "omit at default."
• compile.ts:validateAuthoringRetryDefaults now returns number | undefined — a validator returning an extracted value is an odd shape. Consider parseAuthoringRetry or splitting.

REVIEW_FAILED

[kjgbot]

history lens — FAIL

Blocker

• packages/sdk/src/worker.ts:129 now reports classified transport losses as crashed, but kernel/relayflowd-core/src/machine.rs:367-412 retries every budget-eligible crashed completion without checking RecoveryMode::Manual. The manual-mode park logic exists only for abandoned leases in kernel/relayflowd-core/src/machine/recovery.rs:81-145. Consequently, a signal close, statusless close, transient spawn failure, or Codex stdin-lifecycle failure reported through step.complete is redispatched instead of parked. This newly contradicts RFC-0001 Appendix A rule 4 (docs/RFC-0001-everything-is-a-relayflow.md:240-243) and the diff’s own statement that “manual parks rather than redispatching” (docs/SURFACE.md:344-352). Add the manual disposition/wait.human handling to the reported-crash path and pin it with a test.

This also echoes DRIVE-LOG’s recorded #252 failure mode: retry behavior was described more strongly than the actual control flow supported. Here the mismatch is directly observable from the two separate completion paths, so it is not an aspirational deferral.

Concern

• kernel/DESIGN.md:61 says max_transport_retries is omitted at its default of one, while kernel/relayflowd-core/src/entry.rs:207-212 omits zero instead. That makes the journal documentation inaccurate and loses explicit-zero diagnostic provenance. It is not independently a history-lens blocker, but should be reconciled.

Notes

The bounded transport budget, narrow failure classification, preserved idempotency key/pins, and terminal handling of ordinary nonzero exits align with recorded history and RFC semantics. The two commit subjects make no false test, evidence, or scope claims.

REVIEW_FAILED

[kjgbot]

structure lens — MISSING

[kjgbot]

🎯 review-swarm: FAILED (M:fail H:fail S:missing)

Lens transcripts posted as sibling comments above.

[kjgbot]

Final-head handoff for 139690f754805ee961bb7f12e2c5d0f4f8b72f46:

• The prior maintainability blocker is fixed: all_backing_off_steps_return_timers again uses CompletionReason::Crashed.
• The prior history/Bugbot blocker is fixed: reported crashed/lease_expired completions under manual recovery park before retry-budget evaluation, using the journaled start pin.
• A newly found crash-prefix defect is also fixed: resume idempotently repairs a death between the park placeholder and wait.human append.
• Review cleanup split the recovery tests into a focused module, kept the sentinel crate-private, reconciled DESIGN/journal behavior, and replaced narrative mutation claims with a literal red/checksummed-restore/green transcript.

Exact-head local output is recorded in the PR body and committed under kernel/evidence/501/. GitHub checks at this head:

guard                 pass   6s
validate              pass   13s
packed-consumer       pass   50s
linux-x64-artifact    pass   7m31s
Cursor Bugbot         pass   5m11s

The independent final-head reviewer reported no blocker across maintainability/history/structure and re-executed the extracted recovery tests; its node-local full report is being relayed for portable attachment. The repository Review swarm workflow is currently disabled, so the old M:fail H:fail S:missing comment is stale at f3bd47fe; I have asked the coordinator to rerun it or post the equivalent exact-head preswarm.

No merge performed; human review/merge remains required.

[khaliqgant]

Merge conflict resolution note (b266979): reverted the branch's transportRetries/recoveryMode additions to examples/software-factory/software-factory.flow.ts.

Main's hosted-capability isolation (#552) pins that file by sha256 — Hosted capability isolation accepts only the reviewed Software Factory base source fires when the bytes differ. The reviewed hash 49c99322… matches main's version exactly; the branch's edits invalidated it and broke babysitter-native-extension.test.ts in CI.

The transport-retry feature itself is unchanged — only the example usage was reverted. If the factory flow should adopt retries, that requires a re-reviewed base-source pin; I left it out rather than updating a security pin to accommodate this branch.

Verified: npx vitest run tests/babysitter-native-extension.test.ts tests/canonical-software-factory.test.ts — 44 passed at b266979.