diff --git a/docs/SURFACE.md b/docs/SURFACE.md index 29c6887f..9585f606 100644 --- a/docs/SURFACE.md +++ b/docs/SURFACE.md @@ -1736,6 +1736,32 @@ typed `run_not_found` refusal. A dropped connection, request failure, or journal may already have changed and the CLI cannot honestly claim the resume was refused before a write. +A timed-out read is handled separately: flow execution clients (`flows run` +and `flows resume`, authored and declarative) retry only `run.get`, +`journal.read`, `stream.read`, and `subscription.inspect` within a 300-second +total budget. Reads use a separate session, one in flight per body client, +with increasing attempt bounds and jitter. Interactive clients keep the +single-shot 30-second default; writes are never retried by this policy. +Unretried timeouts retain `journal client: timed out after ms`. +Exhausted reads instead name the verb, attempts, elapsed time, total read +budget, and possible CPU load. + +After a read budget expires, the CLI probes a fresh connection. A responding +daemon reports `daemon_unresponsive`; a failed probe reports +`daemon_unreachable` (also used for initial attach failures). A failed probe +cannot prove the daemon is dead: its diagnostic names both unreachability and +CPU load. Both reports exit 1, carry the known run/root id, retain +`status: running`, and include `flows resume`. This parks the CLI execution +without manufacturing a terminal journal fact; it does not claim a journaled +human park or verify journal integrity. The root worker session closes so the +kernel can recover its attempt, preserving completed work for resume. + +Worker completion waits use `run.watch` pushes with a snapshot every two +seconds for the live lease deadline. A scoped watch session closes after each +wait because the protocol has no unwatch verb. A heartbeat timeout relinquishes +worker ownership as lease loss, leaving recovery to the kernel instead of +recording `worker_error` against the body. + A step that ran and failed is **not** one of those. It reports `step_failed` with `status: failed`, for every step type and for authored TypeScript flows as well as declarative ones. `protocol_error` is reserved for an outcome the diff --git a/evidence/run-read-timeout/baseline-setup.md b/evidence/run-read-timeout/baseline-setup.md new file mode 100644 index 00000000..aa9e8e1a --- /dev/null +++ b/evidence/run-read-timeout/baseline-setup.md @@ -0,0 +1,17 @@ +The handshake comparison used an unmodified detached checkout of +`3c58ee16d10a9e2400db980f5bbafac84e437f20`. The two probe sources are copied +verbatim from the `/tmp` scripts named in their captured output. They use this +session's absolute checkout/build paths; adjust those paths when reproducing +on another machine. + +Setup used (from the implementation checkout): + +```sh +git worktree add --detach /tmp/relayflow-read-timeout-baseline 3c58ee16d10a9e2400db980f5bbafac84e437f20 +ln -s /home/daytona/.relayflow-v2-supervisor/durable/repository/packages/sdk/node_modules /tmp/relayflow-read-timeout-baseline/packages/sdk/node_modules +npm run build --prefix /tmp/relayflow-read-timeout-baseline/packages/sdk +``` + +The baseline probe loads SDK code from that checkout while keeping the daemon +binary and stub fixture path identical to the current-code probe. It compares +that one fixture handshake failure, not the full suite. diff --git a/evidence/run-read-timeout/final-focused.log b/evidence/run-read-timeout/final-focused.log new file mode 100644 index 00000000..ce9b8742 --- /dev/null +++ b/evidence/run-read-timeout/final-focused.log @@ -0,0 +1,59 @@ +$ cd packages/sdk && RELAYFLOWD_BIN=/home/daytona/.relayflows-toolchain/target/2962130851/debug/relayflowd npx vitest run tests/journal-client-read-timeout.test.ts tests/journal-client.test.ts tests/journal-client-completion.test.ts tests/journal-client-subscriptions.test.ts tests/running-step-watch.test.ts tests/heartbeat-timeout.test.ts tests/run-daemon-unresponsive.test.ts tests/authored-root.test.ts tests/classify-outcome.test.ts tests/cli.test.ts tests/direct-run-worker-lease.test.ts tests/resume-worker-lease.test.ts tests/worker-lease.test.ts tests/worker-lease-lost.test.ts tests/worker-lease-sweep.test.ts tests/run-read-load-live.test.ts tests/worker-lease-lost-live.test.ts tests/flow-executor-chain.test.ts tests/agent-transcript-live.test.ts tests/human-live.test.ts --maxWorkers=1 --minWorkers=1 + + RUN v2.1.9 /home/daytona/.relayflow-v2-supervisor/durable/repository/packages/sdk + + ✓ tests/cli.test.ts (71 tests) 6026ms + ✓ flows check CLI > binds a checked relative wrapper to the flow directory for worker execution 566ms + ✓ flows check CLI > resolves a bare PATH-resolved claude with no declared model, in an isolated PATH 488ms + ✓ flows run/resume CLI over the journal protocol > follows a dispatched worker step instead of reporting a protocol error 2053ms + ✓ flows run/resume CLI over the journal protocol > follows a worker wait past a locally expired lease until the daemon settles it 2045ms +(node:44939) [FLOWS_ROOT_LEASE_LOST] Warning: authored root run_id=root-run attempt=1: lease_conflict: attempt has no active worker lease. Waiting for the kernel to retry it. +(Use `node --trace-warnings ...` to show where the warning was created) + ✓ tests/authored-root.test.ts (26 tests) 398ms + ✓ tests/journal-client.test.ts (17 tests) 89ms + ✓ tests/flow-executor-chain.test.ts (14 tests) 10315ms + ✓ flow executor LLM and output-binding chain > runs f.llm -> f.agent -> f.run with schema-verified journal output and the exact allowed model 1022ms + ✓ flow executor LLM and output-binding chain > runs a dollar-budgeted authored Claude agent with the same default used by preflight 700ms + ✓ flow executor LLM and output-binding chain > runs the exact authored flagship f.llm -> f.agent -> f.run path through the durable CLI root 1595ms + ✓ flow executor LLM and output-binding chain > resumes an interrupted durable authored root without replaying completed flagship effects 3382ms + ✓ flow executor LLM and output-binding chain > passes a declarative verified value through an agent into a deterministic artifact 779ms + ✓ flow executor LLM and output-binding chain > flows run consumes YAML bindings and resume reuses the original journal output 1068ms + ✓ tests/agent-transcript-live.test.ts (4 tests) 3394ms + ✓ the transcript digest through the built CLI, a real daemon and the local agent > preserves structured agent failure details and its completed root index 875ms + ✓ the transcript digest through the built CLI, a real daemon and the local agent > preserves structured llm failure details and its completed root index 818ms + ✓ the transcript digest through the built CLI, a real daemon and the local agent > journals the digest in trajectory_tail on a successful agent step and writes the file it points at 851ms + ✓ the transcript digest through the built CLI, a real daemon and the local agent > on a failed agent step, names the failure and the transcript in the terminal diagnostic, redacted 848ms + ✓ tests/classify-outcome.test.ts (11 tests) 7422ms + ✓ classifyOutcome > gives up and reports when a running run never becomes classifiable 2009ms + ✓ the remedy on a worker park > follows a step through a retry backoff longer than the unclassified bound 3005ms + ✓ the remedy on a worker park > follows a run.start outcome that is already running on a retried attempt 2001ms + ✓ tests/human-live.test.ts (3 tests) 7854ms + ✓ f.human against a real daemon > parks with the question, refuses wrong answers, records one, and resumes to success 4785ms + ✓ f.human against a real daemon > a "no" is a value the body branches on: declined, exit 0, no effect 1916ms + ✓ f.human against a real daemon > refuses to answer a run the daemon does not know 1152ms + ✓ tests/worker-lease.test.ts (7 tests) 20ms + ✓ tests/worker-lease-lost.test.ts (17 tests) 27ms + ✓ tests/journal-client-read-timeout.test.ts (13 tests) 1104ms + ✓ a recovered read timeout does not become an authored callback failure 368ms + ✓ tests/worker-lease-lost-live.test.ts (3 tests) 943ms + ✓ reports journal success after completion rejects with lease_conflict 327ms + ✓ reports journal success when a renewal rejects after completion landed 327ms + ✓ tests/run-read-load-live.test.ts (2 tests) 2821ms + ✓ completes a CPU-saturating deterministic flow with reads in flight and preserves its journal 2095ms + ✓ drains read and watch promises before an authored flow completes 725ms + ✓ tests/worker-lease-sweep.test.ts (4 tests) 8ms + ✓ tests/journal-client-completion.test.ts (6 tests) 102ms + ✓ tests/resume-worker-lease.test.ts (3 tests) 6ms + ✓ tests/direct-run-worker-lease.test.ts (3 tests) 10ms + ✓ tests/running-step-watch.test.ts (2 tests) 2122ms + ✓ uses pushes for completion with lease-cadence reads and releases its watcher 2118ms + ✓ tests/journal-client-subscriptions.test.ts (1 test) 8ms + ✓ tests/run-daemon-unresponsive.test.ts (2 tests) 4ms + ✓ tests/heartbeat-timeout.test.ts (1 test) 16ms + + Test Files 20 passed (20) + Tests 210 passed (210) + Start at 10:41:25 + Duration 54.29s (transform 1.45s, setup 97ms, collect 8.09s, tests 42.69s, environment 3ms, prepare 1.05s) + +exit=0 diff --git a/evidence/run-read-timeout/full-suite-final.log b/evidence/run-read-timeout/full-suite-final.log new file mode 100644 index 00000000..0f4c7146 --- /dev/null +++ b/evidence/run-read-timeout/full-suite-final.log @@ -0,0 +1,1970 @@ +$ cd packages/sdk && RELAYFLOWD_BIN=/home/daytona/.relayflows-toolchain/target/2962130851/debug/relayflowd npm test + +> @relayflows/sdk@2.0.42 test +> sh scripts/test.sh + + +> @relayflows/sdk@2.0.42 test:prep +> ( cd ../../kernel && sh ../ops/cargo.sh build ) && ( [ ! -d ../../testdata/preflight ] || find ../../testdata/preflight -name '*-cli' -type f -exec chmod +x {} + ) + + Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.07s + +> @relayflows/sdk@2.0.42 typecheck +> tsc --noEmit && tsc -p tsconfig.type-tests.json + + +> @relayflows/sdk@2.0.42 build +> tsc && node scripts/make-cli-executable.mjs + + +> @relayflows/sdk@2.0.42 typecheck:tests +> tsc -p tsconfig.tests.json + + + RUN v2.1.9 /home/daytona/.relayflow-v2-supervisor/durable/repository/packages/sdk + +stdout | tests/live-kernel.test.ts +LIVE_KERNEL relayflowd=/home/daytona/.relayflows-toolchain/target/2962130851/debug/relayflowd +LIVE_KERNEL flows=/home/daytona/.relayflow-v2-supervisor/durable/repository/packages/sdk/dist/cli.js + + ✓ tests/preflight.test.ts (70 tests) 115ms + ✓ tests/cli.test.ts (71 tests) 5824ms + ✓ flows check CLI > binds a checked relative wrapper to the flow directory for worker execution 460ms + ✓ flows check CLI > resolves a bare PATH-resolved claude with no declared model, in an isolated PATH 413ms + ✓ flows run/resume CLI over the journal protocol > follows a dispatched worker step instead of reporting a protocol error 2038ms + ✓ flows run/resume CLI over the journal protocol > follows a worker wait past a locally expired lease until the daemon settles it 2045ms + ✓ tests/cloud-read.test.ts (46 tests) 55ms +stdout | tests/live-kernel.test.ts > surface resume after a real daemon kill > resumes a three-step run with each successful completion exactly once +LIVE_KERNEL kill -9 pid=44150 run=01M45TEQ6WQ827KKF8VV8WPXD1 while step=two state=Running + + ❯ tests/live-kernel.test.ts (32 tests | 8 failed) 57015ms + ✓ built flows CLI against live relayflowd > twenty-six-step reuses 25 durable completions after editing the failed final step 2821ms + ✓ built flows CLI against live relayflowd > runs rung (a), parks rung (b), and keeps JSON report-shaped 2765ms + ✓ built flows CLI against live relayflowd > allows a deterministic run to exceed the bounded request timeout 32494ms + ✓ built flows CLI against live relayflowd > follows a live worker dispatch through flows run 668ms + ✓ built flows CLI against live relayflowd > runs an agent CLI end to end through the SDK worker 466ms + ✓ built flows CLI against live relayflowd > f.agent lowers to a real agent step and dispatches through a live worker 578ms + ✓ built flows CLI against live relayflowd > can always get a parked run to a late-attaching worker 5588ms + ✓ built flows CLI against live relayflowd > reports a real manual-recovery NeedsHuman state as parked 485ms + × built flows CLI against live relayflowd > runs hn-monitor analyze-story end-to-end via a stub agent CLI (gate 2 clause 2 demo) 543ms + → expected { …(12) } to match object { output: { …(3) }, …(1) } +(22 matching properties omitted from actual) + × built flows CLI against live relayflowd > hn-monitor analyze-story FAILS verification when the CLI omits required schema fields 543ms + → expected { …(12) } to match object { …(3) } +(21 matching properties omitted from actual) + × built flows CLI against live relayflowd > agent step preserves the CliResult wrapper as output when the CLI emits non-JSON text 506ms + → expected null not to be null + × built flows CLI against live relayflowd > AgentWorker exposes wake_context to the CLI via RELAYFLOW_WAKE_CONTEXT env var (real analyzer prerequisite) 635ms + → Cannot read properties of null (reading 'story_title') + × built flows CLI against live relayflowd > AgentWorker leaves RELAYFLOW_WAKE_CONTEXT UNSET when the run has no wake_context (undefined-vs-null pin) 587ms + → Cannot read properties of null (reading 'env_present') + ✓ built flows CLI against live relayflowd > AgentWorker passes a declared model to an identified wrapper as RELAYFLOW_MODEL 589ms + ✓ built flows CLI against live relayflowd > AgentWorker refuses a nonconforming journal-submitted wrapper before exposing RELAYFLOW_MODEL 522ms + ✓ built flows CLI against live relayflowd > AgentWorker executes the raw claude adapter with its real model flag 462ms + ✓ built flows CLI against live relayflowd > AgentWorker executes the raw codex adapter with its real model flag 635ms + × built flows CLI against live relayflowd > AgentWorker leaves RELAYFLOW_MODEL UNSET when the step declares no model 1454ms + → Cannot read properties of null (reading 'story_title') + × built flows CLI against live relayflowd > hn-monitor analyze-story reaches done through the real Claude analyzer CLI 29ms + → LIVE_ANALYZER_UNAVAILABLE: "/home/daytona/.relayflow-v2-supervisor/durable/repository/testdata/preflight/analyze-story-claude-cli" does not identify as relayflows-agent-cli-v1 — failing because gate-2 acceptance requires the real analyzer to execute. Set RELAYFLOWS_ALLOW_ANALYZER_SKIP=1 only if this run is not gate evidence. + ✓ built flows CLI against live relayflowd > preflights before journaling and names an unreachable socket 904ms + ✓ built flows CLI against live relayflowd > starts exactly one daemon when two runs race for one empty data dir 569ms + ✓ subprocess_gate output capture against live relayflowd > journals the gate command output and reports both tails 1007ms + ✓ surface resume after a real daemon kill > resumes a three-step run with each successful completion exactly once 882ms + × a relayflow can be scheduled: tick source against live relayflowd > a tick spawns a real run whose step reports the SCHEDULED instant 495ms + → expected null to deeply equal { schedule_id: 'heartbeat-1m', …(3) } + ✓ tests/cloud-live.test.ts (55 tests) 2058ms + ✓ argv and wiring > routes both live invocations through runCli, and refuses a missing run id 2004ms + ❯ tests/hosted-extension-isolation.test.ts (22 tests | 14 failed) 350ms + × hosted extension capability isolation > executes the exact capability-only handler for a queued receipt 9ms + → bubblewrap is unavailable + × hosted extension capability isolation > executes the exact capability-only handler for a duplicate receipt 2ms + → bubblewrap is unavailable + × hosted extension capability isolation > launches through the captured process primitive after builtin export synchronization 5ms + → promise rejected "Error: bubblewrap is unavailable { code: '…' }" instead of resolving + × hosted extension capability isolation > ignores inherited launcher overrides and decodes manifests with the captured Buffer intrinsic 5ms + → promise rejected "Error: bubblewrap is unavailable { code: '…' }" instead of resolving + × hosted extension capability isolation > streams verified bytes when the live store is replaced and no writable staging path exists 248ms + → promise rejected "Error: Hosted extension sandbox exited wi… { code: '…' }" instead of resolving + × hosted extension capability isolation > mounts pinned private Surface bytes when the live package changes before launch 4ms + → promise rejected "Error: bubblewrap is unavailable { code: '…' }" instead of resolving + × hosted extension capability isolation > refuses oversized Surface files through the bounded descriptor reader 6ms + → expected Error: bubblewrap is unavailable { code: '…' } to match object { code: 'plugin_unsupported', …(1) } + × hosted extension capability isolation > shields verified Surface files before async settlement 4ms + → promise rejected "Error: bubblewrap is unavailable { code: '…' }" instead of resolving + × hosted extension capability isolation > preserves a typed host refusal while disclosing only a fixed marker to the child 2ms + → expected Error: bubblewrap is unavailable { code: '…' } to be Error: private Cloud policy detail { code: '…' } // Object.is equality + × hosted extension capability isolation > denies ambient credentials, host files, writes, network, subprocesses, and undeclared context verbs 7ms + → bubblewrap is unavailable + × hosted extension capability isolation > enforces OS address-space and data bounds on native Buffer allocation 2ms + → expected Error: bubblewrap is unavailable { code: '…' } to match object { code: 'plugin_unsupported', …(1) } + × hosted extension capability isolation > blocks extra handler fields and authority-bearing receipt fields at the parent port 2ms + → expected Error: bubblewrap is unavailable { code: '…' } to match object { code: 'plugin_event_unroutable' } + × hosted extension capability isolation > constructs adapter authority with the captured freeze intrinsic 1ms + → bubblewrap is unavailable + × hosted extension capability isolation > writes the Surface manifest and protocol without inherited toJSON behavior 2ms + → promise rejected "Error: bubblewrap is unavailable { code: '…' }" instead of resolving + ✓ tests/cloud-deploy.test.ts (117 tests) 2599ms + ✓ tests/cloud-sync.test.ts (40 tests) 736ms + ✓ tests/observer-link.test.ts (44 tests) 177ms + ✓ tests/authored-flow.test.ts (38 tests) 806ms + ✓ tests/plugin-extension.test.ts (94 tests) 464ms + ✓ tests/cloud-transcript-codex.test.ts (44 tests) 21ms + ✓ tests/worker-cli.test.ts (25 tests) 27087ms + ✓ registered CLI model defaults > passes the same priced Claude default to the real provider invocation 482ms + ✓ registered CLI model defaults > uses the explicitly supplied agent environment for the provider subprocess 427ms + ✓ registered CLI model defaults > dispatches canonical generic bytes with the preflight-proved adapter identity 493ms + ✓ direct transport lifecycle evidence > classifies only the exact Codex stdin lifecycle signature as retryable 410ms + ✓ direct transport lifecycle evidence > records a signal close separately from an ordinary nonzero exit 849ms + ✓ direct transport lifecycle evidence > records a spawn error code without treating a missing executable as transient 450ms + ✓ direct transport lifecycle evidence > journals classified lifecycle evidence and reports crashed instead of generic worker_error 610ms + ✓ step discovery environment > names the run, step, attempt and an absolute data dir for a direct agent spawn 582ms + ✓ step discovery environment > exports none of the four without a data dir, even when the worker inherited them 478ms + ✓ wrapper discovery environment > sets the four names from the dispatch and still refuses ambient values and other secrets 466ms + ✓ wrapper discovery environment > exports none of the four to a wrapper without a data dir, even when the worker inherited them 486ms + ✓ custom wrapper execution identity > passes an explicit safe environment at identification and execution 449ms + ✓ custom wrapper execution identity > refuses a wrapper symlink retarget before delivering private values 453ms + ✓ custom wrapper execution identity > bounds wrapper execution after acknowledgement 498ms + ✓ custom wrapper execution identity > bounds captured wrapper output 470ms + ✓ custom wrapper execution identity > refuses a duplicate execute protocol frame 432ms + ✓ custom wrapper execution bounds are reader-owned > resolves when a conforming wrapper leaks a stdio pipe to a background helper 697ms + ✓ custom wrapper execution bounds are reader-owned > resolves when the leaked helper inherits stderr only 673ms + ✓ custom wrapper execution bounds are reader-owned > resolves when a wrapper leaks a stdio pipe and exits before identifying 3447ms + ✓ custom wrapper execution bounds are reader-owned > journals a completionReason at the default bound when a wrapper leaks a stdio pipe 11463ms + ✓ custom wrapper execution bounds are reader-owned > accepts an execute token and an over-8KiB payload flushed in one write 434ms + ✓ custom wrapper execution bounds are reader-owned > accepts the same over-8KiB payload whether or not it coalesces with the execute token 1430ms + ✓ custom wrapper execution bounds are reader-owned > still bounds an un-terminated handshake buffer and names the bound 441ms + ✓ delivers the journaled memory pack to the real wrapper and excludes its charge from completion usage 462ms + ✓ tests/flow-extension-compose.test.ts (33 tests) 6663ms + ✓ composing flow extensions onto a base flow > composes two extensions in declaration order, and the order is the lockfile order 556ms + ✓ composing flow extensions onto a base flow > flows check reports the composition and keeps the composed triggers deliverable 1122ms + ✓ composing flow extensions onto a base flow > uses extension permissions for hosted deploy preflight and the deploy body 324ms +(node:46445) ExperimentalWarning: SQLite is an experimental feature and might change at any time +(Use `node --trace-warnings ...` to show where the warning was created) + ✓ tests/cli-status.test.ts (27 tests) 1171ms + ✓ flows status > resolves the run with no arguments from inside a worker-spawned agent 940ms + ✓ tests/run-state.test.ts (28 tests) 34ms + ✓ tests/cloud-run.test.ts (59 tests) 597ms + ✓ tests/relay-cli-surface.test.ts (87 tests) 42ms + ✓ tests/agent-transcript.test.ts (29 tests) 332ms +(node:46773) [FLOWS_ROOT_LEASE_LOST] Warning: authored root run_id=root-run attempt=1: lease_conflict: attempt has no active worker lease. Waiting for the kernel to retry it. +(Use `node --trace-warnings ...` to show where the warning was created) + ✓ tests/authored-root.test.ts (26 tests) 400ms + ✓ tests/babysitter-native-extension.test.ts (41 tests | 1 skipped) 1473ms + ✓ tests/shipped-source-model-provenance.test.ts (1 test) 109230ms + ✓ shipped-source model provenance > does not treat mutable aliases or incomplete named agents as pinned 109229ms + ✓ tests/authored-completion-detail.test.ts (59 tests) 223ms + ✓ tests/shipped-source-worker-call-forms.test.ts (1 test) 17255ms + ✓ shipped-source worker call forms > fails closed for computed keys, assertions, binds, call, and apply 17254ms + ✓ tests/step-failure-diagnostic.test.ts (26 tests) 72ms + ✓ tests/stop-process-group.test.ts (11 tests) 14220ms + ✓ every stop reaches the process group, not just the direct child > exits the run after an execution-timeout stop 970ms + ✓ every stop reaches the process group, not just the direct child > exits the run after a protocol terminate stop 486ms + ✓ every stop reaches the process group, not just the direct child > kills a SIGTERM-deaf grandchild after a protocol terminate stop 1508ms + ✓ every stop reaches the process group, not just the direct child > kills a SIGTERM-deaf grandchild after an execution-timeout stop 1913ms + ✓ every stop reaches the process group, not just the direct child > holds the loop open long enough for the escalation to run 1102ms + ✓ every stop reaches the process group, not just the direct child > bounds forced-stop confirmation when a group remains unprovable 1006ms + ✓ a wrapper that exits with no execution deadline still drains > reports the wrapper result and reaps a grandchild holding its pipes 707ms + ✓ a wrapper that exits with no execution deadline still drains > reaps a SIGTERM-deaf grandchild holding its pipes 1780ms + ✓ a wrapper that exits with no execution deadline still drains > settles on its own deadline when an escaped holder withholds close 4427ms + ✓ tests/mcp.test.ts (34 tests) 13995ms + ✓ MCP preflight and transports > flows check refuses an undeclared server with exit 2 and no daemon 865ms + ✓ MCP preflight and transports > flows check reports a refusing server and leaves no PID 944ms + ✓ MCP preflight and transports > kills a SIGTERM-resistant silent child after a parent-owned handshake deadline 1313ms + ✓ MCP preflight and transports > reaps a SIGTERM-resistant descendant with inherit stdio before cleanup finishes 1061ms + ✓ MCP preflight and transports > reaps a SIGTERM-resistant descendant with ignore stdio before cleanup finishes 2021ms + ✓ MCP preflight and transports > rejects close when forced group death remains unprovable 2065ms + ✓ MCP preflight and transports > cancels force escalation when close follows an exited child 1179ms + ✓ MCP preflight and transports > waits for child close when the transport cannot own a process group 1055ms + ✓ MCP preflight and transports > cancels force escalation for an already-closed child without a process group 1177ms + ✓ MCP preflight and transports > reports malformed connection configuration as config_invalid 803ms + ✓ authored MCP effects against the real kernel > reports a dropped tool connection as a failed CLI run 647ms + ✓ tests/step-attempt-history.test.ts (23 tests) 27ms + ✓ tests/daemon-lifecycle.test.ts (42 tests) 41ms + ✓ tests/close-pr-flow.test.ts (35 tests) 21131ms + ✓ close-pr journaled repair loop > reads an existing Bugbot finding, repairs in the same worktree, pushes and re-verifies before merging 2505ms + ✓ close-pr journaled repair loop > opens a PR and feeds failed CI logs into the repair agent 2128ms + ✓ close-pr journaled repair loop > pins the current generated model for codex when no override is supplied 4998ms + ✓ close-pr journaled repair loop > pins the current generated model for claude when no override is supplied 3900ms + ✓ close-pr journaled repair loop > parks after exactly three nonconverging repairs, with accumulated blockers 2385ms + ✓ close-pr journaled repair loop > can converge on the third repair 2486ms + ✓ close-pr journaled repair loop > executes the deterministic commit and force-push steps against a local Git remote, including a no-op repair 2527ms + ❯ tests/authored-node-runtime.test.ts (16 tests | 16 skipped) 12ms + ✓ tests/artifact-gates.test.ts (27 tests) 242ms + ❯ tests/hosted-extension-protocol.test.ts (27 tests | 8 failed) 40086ms + × hosted extension hostile protocol > uses captured JSON intrinsics for the complete parent boundary 6ms + → bubblewrap is unavailable + × hosted extension hostile protocol > rejects an import-time different PR frame with zero adapter calls 5ms + → expected Error: bubblewrap is unavailable { code: '…' } to match object { code: 'plugin_event_unroutable' } + × hosted extension hostile protocol > rejects an import-time different delivery frame with zero adapter calls 2ms + → expected Error: bubblewrap is unavailable { code: '…' } to match object { code: 'plugin_event_unroutable' } + × hosted extension hostile protocol > rejects an import-time different event frame with zero adapter calls 2ms + → expected Error: bubblewrap is unavailable { code: '…' } to match object { code: 'plugin_event_unroutable' } + × hosted extension hostile protocol > rejects two forged calls after the authoritative first outcome settles 10004ms + → hostile child did not invoke the adapter + × hosted extension hostile protocol > waits for a pending adapter to reject after a forged child error 10004ms + → hostile child did not invoke the adapter + × hosted extension hostile protocol > waits for a pending adapter to resolve after a forged child error 10005ms + → hostile child did not invoke the adapter + × hosted extension hostile protocol > returns a typed adapter rejection even when the hostile child hangs 10005ms + → hostile child did not invoke the adapter + ✓ tests/hosted-base-snapshot.test.ts (18 tests) 1873ms + ✓ hosted base private snapshot > stops streaming project entries at the shared count limit 1797ms + ✓ tests/cloud-connect.test.ts (24 tests) 3023ms + ✓ hosted verbs connect before they submit > flows run --cloud submits once the prompt connected the integration 2120ms + ✓ tests/journal-client.test.ts (17 tests) 81ms + ✓ tests/authored-node-result.test.ts (45 tests) 20ms + ✓ tests/verb-field-lint.test.ts (99 tests) 389ms + ✓ tests/bundle.test.ts (26 tests) 9078ms + ✓ immutable bundles > returns exit 2 naming a byte-flipped payload and refuses to reuse corruption 406ms + ✓ immutable bundles > verifies with --verify in any position and answers --json with one object 799ms + ✓ immutable bundles > refuses --out with --verify rather than ignoring the destination 401ms + ✓ immutable bundles > builds and verifies the canonical YAML fixture through the compiled CLI 1253ms + ✓ immutable bundles > emits the ephemeral warning on CLI stderr and uses the default output directory 793ms + ✓ immutable bundles > refuses build-provable CLI resolution errors without environment probes 431ms + ✓ immutable bundles > builds a standalone TS fixture twice with identical executable hashes 2353ms + ✓ immutable bundles > refuses to label installed dependency drift with lockfile pins 384ms + ✓ immutable bundles > refuses invalid CLI arguments %j 396ms + ✓ immutable bundles > refuses invalid CLI arguments "--out" 396ms + ✓ immutable bundles > refuses invalid CLI arguments "--verify" 397ms + ✓ immutable bundles > refuses invalid CLI arguments "--verify" 394ms + ✓ immutable bundles > refuses invalid CLI arguments "--out" 529ms + ✓ tests/validate.test.ts (68 tests) 34ms + ✓ tests/flow-executor-chain.test.ts (14 tests) 9665ms + ✓ flow executor LLM and output-binding chain > runs f.llm -> f.agent -> f.run with schema-verified journal output and the exact allowed model 842ms + ✓ flow executor LLM and output-binding chain > runs a dollar-budgeted authored Claude agent with the same default used by preflight 547ms + ✓ flow executor LLM and output-binding chain > runs the exact authored flagship f.llm -> f.agent -> f.run path through the durable CLI root 1465ms + ✓ flow executor LLM and output-binding chain > resumes an interrupted durable authored root without replaying completed flagship effects 3225ms + ✓ flow executor LLM and output-binding chain > passes a declarative verified value through an agent into a deterministic artifact 761ms + ✓ flow executor LLM and output-binding chain > flows run consumes YAML bindings and resume reuses the original journal output 1096ms + ✓ tests/agent-relay-transport.test.ts (17 tests) 2272ms + ✓ Relay completion at the journal boundary > does not complete at readiness and journals exact output, receipt, and priced accounting 1006ms + ✓ Relay completion at the journal boundary > aborts polling on rejected renewal and never writes a stale completion 1000ms + ✓ tests/shipped-source-flow-provenance-repairs.test.ts (1 test) 30332ms + ✓ shipped-source flow provenance repairs > audits repaired writer, alias, binding, and cyclic provenance forms 30331ms + ✓ tests/tick-source.test.ts (33 tests) 21ms + ✓ tests/authored-step-graph.test.ts (25 tests) 837ms + ✓ the authored step DAG > does not walk a long-running step's own polling chain to find its dependents' edges 337ms + ✓ tests/cloud-mirror.test.ts (14 tests) 33ms + ✓ tests/pr-review-post.test.ts (21 tests) 1982ms + ✓ tests/authored-flow-lifecycle-executor.test.ts (27 tests) 587ms + ✓ tests/step-failure-excerpt.test.ts (42 tests) 195ms + ✓ tests/authored-step-terminal-completion.test.ts (13 tests) 4878ms + ✓ readCompletedStepOutput on an adopted gated child whose gate has not started > refuses a producer success when the wait ends with the run still not terminal 2006ms + ✓ readCompletedStepOutput given a suspended known status (#441) > refuses a producer success while the child is still not terminal 2006ms +(node:52566) ExperimentalWarning: SQLite is an experimental feature and might change at any time +(Use `node --trace-warnings ...` to show where the warning was created) + ✓ tests/cloud-mirror-step.test.ts (17 tests) 12ms + ✓ tests/cli-replay.test.ts (38 tests) 1314ms + ✓ flows replay > --json is byte-identical across two CLI invocations (diff) 941ms + ✓ tests/authored-flow-slack.test.ts (7 tests) 1753ms + ✓ authored Slack helper effects > replays after SIGKILL before confirm with the same token and one successful completion 560ms + ✓ authored Slack helper effects > replays after SIGKILL before complete with the same token and one successful completion 528ms + ✓ tests/wrapper-execution-duration.test.ts (7 tests) 10822ms + ✓ keeps the handshake deadline independent of the removed execution deadline 10054ms + ✓ still lets a lease abort stop an unlimited wrapper before it produces output 511ms + ✓ tests/tick-runner.test.ts (22 tests) 2354ms + ✓ CLI argument parsing refuses coercion rather than accepting it > refuses --interval-ms fractional as an invocation error 393ms + ✓ CLI argument parsing refuses coercion rather than accepting it > refuses --interval-ms exponent notation as an invocation error 385ms + ✓ CLI argument parsing refuses coercion rather than accepting it > refuses --interval-ms hex as an invocation error 385ms + ✓ CLI argument parsing refuses coercion rather than accepting it > refuses --interval-ms trailing text as an invocation error 390ms + ✓ CLI argument parsing refuses coercion rather than accepting it > refuses --interval-ms empty as an invocation error 391ms + ✓ CLI argument parsing refuses coercion rather than accepting it > accepts an exact integer and proceeds past parsing 385ms + ✓ tests/authored-step-index.test.ts (17 tests) 16ms + ✓ tests/named-gate-diagnostics.test.ts (20 tests) 640ms + ✓ tests/authored-agent-artifacts.test.ts (5 tests) 325ms + ✓ tests/direct-input.test.ts (6 tests) 6022ms + ✓ direct .flow.ts input through the built CLI and live runtime > returns exit 3 for an authored human handoff and persists its outcome 592ms + ✓ direct .flow.ts input through the built CLI and live runtime > returns exit 1 for an authored step_failed verdict and persists its outcome 632ms + ✓ direct .flow.ts input through the built CLI and live runtime > executes inline and file JSON input through relayflowd 2421ms + ✓ direct .flow.ts input through the built CLI and live runtime > refuses missing and malformed input before contacting relayflowd 1517ms + ✓ direct .flow.ts input through the built CLI and live runtime > does not run the authored body before daemon availability 478ms + ✓ direct .flow.ts input through the built CLI and live runtime > refuses oversized file input before contacting relayflowd 381ms + ✓ tests/worker-transcript.test.ts (9 tests) 866ms + ✓ tests/gate-contract.test.ts (20 tests) 151ms + ❯ tests/canonical-software-factory.test.ts (25 tests | 25 failed) 152ms + × canonical software-factory metadata contract > opens the actual catalog flow with the ticket title and exactly one GitHub closing line 11ms + → expected an @relayflows/surface flow handle + × canonical software-factory metadata contract > writes a Linear closing reference that the GitHub integration links back 7ms + → expected an @relayflows/surface flow handle + × canonical software-factory metadata contract > accepts a Linear team key that carries digits 6ms + → expected an @relayflows/surface flow handle + × canonical software-factory metadata contract > stops before push when a Linear ticket has no linkable identifier () 6ms + → expected an @relayflows/surface flow handle + × canonical software-factory metadata contract > stops before push when a Linear ticket has no linkable identifier (not-an-issue) 7ms + → expected an @relayflows/surface flow handle + × canonical software-factory metadata contract > fails closed before push when the Linear closing reference is duplicated in the final body 7ms + → expected an @relayflows/surface flow handle + × canonical software-factory metadata contract > fails closed before push when the body carries a foreign closing reference 6ms + → expected an @relayflows/surface flow handle + × canonical software-factory metadata contract > does not mistake ordinary closing-verb prose for a reference 6ms + → expected an @relayflows/surface flow handle + × canonical software-factory metadata contract > normalizes whitespace and caps the title at 240 Unicode code points 7ms + → expected an @relayflows/surface flow handle + × canonical software-factory metadata contract > fails closed before push when GitHub identity is missing or the final body duplicates its closing line 0ms + → expected an @relayflows/surface flow handle + × canonical software-factory review scope > fails closed for explicit defects 6ms + → expected an @relayflows/surface flow handle + × canonical software-factory review scope > fails closed for no verdict 5ms + → expected an @relayflows/surface flow handle + × canonical software-factory review scope > fails closed for contradictory blocked and unverified 6ms + → expected an @relayflows/surface flow handle + × canonical software-factory review scope > fails closed for contradictory passed and unverified 5ms + → expected an @relayflows/surface flow handle + × canonical software-factory review scope > fails closed for contradictory passed and blocked 7ms + → expected an @relayflows/surface flow handle + × canonical software-factory review scope > fails closed for empty unverified 7ms + → expected an @relayflows/surface flow handle + × canonical software-factory review scope > distinguishes a missing verification prerequisite from a defect 6ms + → expected an @relayflows/surface flow handle + × canonical software-factory review scope > preserves the passed body without a scope marker or draft 6ms + → expected an @relayflows/surface flow handle + × canonical software-factory review scope > rejects a forged scope for local before publication 6ms + → expected an @relayflows/surface flow handle + × canonical software-factory review scope > rejects a forged scope for github before publication 5ms + → expected an @relayflows/surface flow handle + × canonical software-factory review scope > scopes the post-review hook refusal 6ms + → expected an @relayflows/surface flow handle + × canonical software-factory review scope > scopes the merge-gate hook refusal 6ms + → expected an @relayflows/surface flow handle + × canonical software-factory review scope > rejects malformed head "" before publication 6ms + → expected an @relayflows/surface flow handle + × canonical software-factory review scope > rejects malformed head "not-a-sha" before publication 7ms + → expected an @relayflows/surface flow handle + × canonical software-factory review scope > rejects malformed head "a'; touch injected; #" before publication 6ms + → expected an @relayflows/surface flow handle + ✓ tests/authored-human.test.ts (13 tests) 91ms + ✓ tests/wrapper-exit-drain.test.ts (10 tests) 3512ms + ✓ reports a signalled wrapper death while a pipe is held, with its output intact 379ms + ✓ lets a lease abort outrank a successful exit still being drained 537ms + ✓ does not let a later drain overwrite abort while group confirmation is pending 631ms + ✓ tests/cloud-schedule.test.ts (17 tests) 5275ms + ✓ schedule lowering > marks a non-grid cron as Cloud-only rather than approximating it, with a silence budget from its own cadence 2913ms + ✓ flows check prints declared schedules > shows the lowering for a fixed interval and the Cloud-only note for a real cron 1956ms + ✓ tests/authored-run-failure-evidence.test.ts (9 tests) 848ms + ✓ the child index after the process that wrote it is gone > still names every child, with its own run id, after a daemon restart 484ms + ✓ tests/worker-cli-result-exit.test.ts (7 tests) 34844ms + ✓ a Claude agent step completes on its result, not only on process exit > settles a hung, successful run within the grace and stops its whole tree 32619ms + ✓ a Claude agent step completes on its result, not only on process exit > maps an error result on a hung run to a failed exit 31619ms + ✓ a Claude agent step completes on its result, not only on process exit > leaves a hang before any result to the existing stops 32014ms + ✓ an agent tree does not outlive the process that spawned it > kills the agent group when the run process is terminated by SIGTERM 795ms + ✓ an agent tree does not outlive the process that spawned it > removes a wrapper alias on host exit after group death stays unprovable 1341ms + ✓ tests/cli-hn-monitor.test.ts (16 tests) 95ms + ✓ tests/spec-parity.test.ts (46 tests) 505ms + ✓ tests/authored-activity.test.ts (16 tests) 108ms + ✓ tests/run-projection.test.ts (13 tests) 12ms + ✓ tests/agent-artifacts-live.test.ts (5 tests) 4163ms + ✓ agent artifacts and gates through the built CLI, a real daemon and the local agent > journals the files the agent wrote, including under a dot-directory, and every artifact gate passes on that journal 1030ms + ✓ agent artifacts and gates through the built CLI, a real daemon and the local agent > fails the run when the artifact_exists gate names a file the agent did not write 803ms + ✓ agent artifacts and gates through the built CLI, a real daemon and the local agent > fails the run with the author reason when a predicate gate returns false, journaling the verdict 877ms + ✓ review follow-ups > applies a predicate gate on a helper step too, and journals its verdict 790ms + ✓ review follow-ups > records predicate verdicts on the root run so a resume reuses them instead of re-running the closure 662ms + ✓ tests/daemon-lifecycle-live.test.ts (9 tests) 6889ms + ✓ flows run against a data dir with no daemon (§6 test 7) > cold start spawns exactly one daemon, the run succeeds, and the daemon outlives the CLI 492ms + ✓ flows run against a data dir with no daemon (§6 test 7) > polls, bounded, for a daemon that holds the lock before it binds 1369ms + ✓ flows run against a data dir with no daemon (§6 test 7) > attaches to a serving daemon that has not published a connection file 505ms + ✓ flows run against a data dir with no daemon (§6 test 7) > a second run attaches to the daemon the first one started, spawning nothing 964ms + ✓ flows run against a data dir with no daemon (§6 test 7) > detects a stale connection file left by a hard kill and starts a fresh daemon 1010ms + ✓ concurrent invocations against one empty data dir (§6 test 15) > ends with exactly one daemon owning the socket, and both runs succeed 1203ms + ✓ refusals from a spawn that cannot produce a daemon > names relayflowd_not_found rather than falling through to PATH 413ms + ✓ refusals from a spawn that cannot produce a daemon > names daemon_start_failed and quotes the daemon log when startup dies 470ms + ✓ refusals from a spawn that cannot produce a daemon > refuses a daemon speaking another protocol version instead of binding over it 464ms + ✓ tests/agent-transcript-live.test.ts (4 tests) 3517ms + ✓ the transcript digest through the built CLI, a real daemon and the local agent > preserves structured agent failure details and its completed root index 852ms + ✓ the transcript digest through the built CLI, a real daemon and the local agent > preserves structured llm failure details and its completed root index 806ms + ✓ the transcript digest through the built CLI, a real daemon and the local agent > journals the digest in trajectory_tail on a successful agent step and writes the file it points at 885ms + ✓ the transcript digest through the built CLI, a real daemon and the local agent > on a failed agent step, names the failure and the transcript in the terminal diagnostic, redacted 973ms + ✓ tests/resume-local-agent.test.ts (8 tests) 13ms +(node:55014) Warning: Transcript tail for run-9/analyze attempt 1 (stdout) could not be written; the step continues without it: EACCES: permission denied, mkdir '/tmp/transcript-tail-1nBIzB/runs/run-9/steps' +(Use `node --trace-warnings ...` to show where the warning was created) + ✓ tests/transcript-tail.test.ts (11 tests) 1192ms + ✓ direct agent spawn > tees stdout and stderr into tail files that name the dispatch 527ms + ✓ direct agent spawn > completes the step when the tail directory cannot be created 384ms + ✓ tests/advisory-outcome.test.ts (23 tests) 72ms + ✓ tests/shipped-source-cubic-regressions.test.ts (5 tests) 11768ms + ✓ shipped-source adversarial provenance regressions > retains callable writes through member mutation, spreads, this, holes, and alternate receivers 6294ms + ✓ shipped-source adversarial provenance regressions > does not manufacture intrinsic writes or comma-left callables 2442ms + ✓ shipped-source adversarial provenance regressions > treats constructor and implicit-this receiver escapes as possible worker writes 666ms + ✓ shipped-source adversarial provenance regressions > maps pure helper parameters and refuses mutations or conditional assignments as proofs 1755ms + ✓ shipped-source adversarial provenance regressions > does not apply later alias assignments retroactively to copied values 610ms + ✓ tests/agent-cwd-live.test.ts (4 tests) 2928ms + ✓ agents drive two checkouts under one run root > runs each CLI in its declared directory, and the run is not refused as an unknown field 868ms + ✓ agents drive two checkouts under one run root > gates on an artifact path relative to the agent directory 779ms + ✓ a declaration this contract cannot honour is refused at its own edge > refuses a path that climbs out of the run root without starting a run at all 600ms + ✓ a declaration this contract cannot honour is refused at its own edge > fails the step with the reason when the directory is not there, rather than running somewhere else 680ms + ✓ tests/authored-helpers.test.ts (6 tests) 3363ms + ✓ runs every available provider through the real kernel and resumes completed effects without a second write 1593ms + ✓ replays after SIGKILL before confirm with the same token and one successful completion 712ms + ✓ replays after SIGKILL before complete with the same token and one successful completion 741ms + ✓ tests/cloud-mirror-session.test.ts (12 tests) 14ms + ✓ tests/agent-cwd.test.ts (53 tests) 50ms + ✓ tests/pty-sidechannel.test.ts (14 tests) 8465ms + ✓ view attach preserves worker completion and marks only drive 970ms + ✓ drive attach preserves worker completion and marks only drive 460ms + ✓ passthrough attach preserves worker completion and marks only drive 907ms + ✓ none attach preserves worker completion and marks only drive 947ms + ✓ none subscriber lets an unattended CLI read EOF 506ms + ✓ view subscriber lets an unattended CLI read EOF 497ms + ✓ passthrough subscriber lets an unattended CLI read EOF 494ms + ✓ incomplete subscriber lets an unattended CLI read EOF 513ms + ✓ unattended Codex receives closed stdin before startup instead of entering its additional-input lifecycle 591ms + ✓ ends driven stdin when the last drive peer disconnects after spawn 730ms + ✓ rejects drive after EOF without marking human intervention 951ms + ✓ delivers all drive bytes in order across child stdin backpressure 866ms + ✓ tests/authored-flow-operation.test.ts (24 tests) 507ms + ✓ tests/budget-attribution.test.ts (10 tests) 9ms + ✓ tests/helper-reference.test.ts (30 tests) 22ms + ✓ tests/cloud-mirror-live.test.ts (3 tests) 2782ms + ✓ a local run on the Cloud dashboard > registers, publishes its steps and its log, and reports terminal last 547ms + ✓ a local run on the Cloud dashboard > sends nothing at all unless it is asked to 1358ms + ✓ a local run on the Cloud dashboard > publishes a composed authored flow as one connected dashboard DAG 835ms + ✓ tests/flow-requirements.test.ts (14 tests) 720ms + ✓ flows check prints REQUIRES > names the helper, the harness and the mcp server of an authored flow 508ms + ✓ tests/backlog-picker.test.ts (14 tests) 41ms + ✓ tests/plugin-store-bounds.test.ts (11 tests) 69ms + ✓ tests/classify-outcome.test.ts (11 tests) 7417ms + ✓ classifyOutcome > gives up and reports when a running run never becomes classifiable 2007ms + ✓ the remedy on a worker park > follows a step through a retry backoff longer than the unclassified bound 3005ms + ✓ the remedy on a worker park > follows a run.start outcome that is already running on a retried attempt 2000ms + ✓ tests/authored-completion-recovery.test.ts (5 tests) 375ms + ✓ tests/backlog-picker-flow.test.ts (6 tests) 262ms + ✓ tests/hosted-extension-protocol-intrinsics.test.ts (6 tests) 13ms + ✓ tests/preflight-permissions-unenforced.test.ts (17 tests) 517ms + ✓ an authored TypeScript body is out of reach, and spec.ts says so > checks clean on a .flow.ts whose body declares permissions 473ms + ✓ tests/budget-unmetered-live.test.ts (4 tests) 1245ms + ✓ unmetered budget spend through the live kernel > runs an unpriced step under a dollar budget without tripping it, journaling unknown dollars 444ms + ❯ tests/stuck-run-triage.test.ts (22 tests | 22 failed) 74ms + × stuck-run-triage input validation > refuses an 8-character run-id prefix: Cloud has no prefix lookup 4ms + → expected [Function] to throw error matching /not full Cloud run ids: c649fe14/ but got 'expected an @relayflows/surface flow …' + × stuck-run-triage input validation > refuses the whole batch when any id is invalid, rather than dropping it 1ms + → expected [Function] to throw error matching /not full Cloud run ids: nope!/ but got 'expected an @relayflows/surface flow …' + × stuck-run-triage input validation > refuses an empty batch 0ms + → expected [Function] to throw error matching /needs runIds/ but got 'expected an @relayflows/surface flow …' + × stuck-run-triage input validation > refuses a batch too large for the edge step lease 0ms + → expected [Function] to throw error matching /exceeds the 8 that fit/ but got 'expected an @relayflows/surface flow …' + × stuck-run-triage input validation > accepts eight ids — the incident batch is inside the bound 3ms + → promise rejected "TypeError: expected an @relayflows/surfac…" instead of resolving + × stuck-run-triage apiUrl > refuses to send the Cloud bearer token to an unapproved origin 0ms + → expected [Function] to throw error matching /refusing to send the Cloud bearer to…/\ but got 'expected an @relayflows/surface flow …' + × stuck-run-triage apiUrl > refuses a non-URL apiUrl 1ms + → expected [Function] to throw error matching /is not a URL/ but got 'expected an @relayflows/surface flow …' + × stuck-run-triage apiUrl > allows an approved origin and uses it in the curl 0ms + → expected an @relayflows/surface flow handle + × stuck-run-triage apiUrl > defaults to production Cloud 0ms + → expected an @relayflows/surface flow handle + × stuck-run-triage apiUrl > never publishes a run record the fetch did not produce 0ms + → expected an @relayflows/surface flow handle + × stuck-run-triage edge collection > names the Worker on every wrangler invocation 0ms + → expected an @relayflows/surface flow handle + × stuck-run-triage edge collection > accepts caller-supplied Workers and rejects option-shaped ones 0ms + → expected an @relayflows/surface flow handle + × stuck-run-triage edge collection > falls back when GNU timeout is absent, as it is on macOS 0ms + → expected an @relayflows/surface flow handle + × stuck-run-triage edge collection > runs the tails concurrently so wall time does not scale with the batch 0ms + → expected an @relayflows/surface flow handle + × stuck-run-triage edge collection > records wrangler's own exit status rather than head's 0ms + → expected an @relayflows/surface flow handle + × stuck-run-triage shell text > parses under both sh and bash 0ms + → expected an @relayflows/surface flow handle + × stuck-run-triage shell text > collects tails with no GNU timeout on PATH, as on a stock macOS 58ms + → expected an @relayflows/surface flow handle + × stuck-run-triage agents > declares read-only permissions on every agent 0ms + → expected an @relayflows/surface flow handle + × stuck-run-triage agents > tells the forensics agents their evidence is untrusted 0ms + → expected an @relayflows/surface flow handle + × stuck-run-triage fan-out > refuses a duplicate run id: two tails would share one evidence file 1ms + → expected [Function] to throw error matching /duplicate runIds: c649fe14-0c2e-4e51-…/ but got 'expected an @relayflows/surface flow …' + × stuck-run-triage fan-out > refuses a duplicate Worker name for the same reason 0ms + → expected [Function] to throw error matching /duplicate workers: w-one/ but got 'expected an @relayflows/surface flow …' + × stuck-run-triage fan-out > bounds ids x workers, not just ids 0ms + → expected [Function] to throw error matching /24 concurrent tails, over the 16/ but got 'expected an @relayflows/surface flow …' + ✓ tests/hosted-extension-routing.test.ts (7 tests) 7ms + ✓ tests/webhook.test.ts (9 tests) 784ms + ✓ webhook ingress > checks TS declarations against flows.json without invoking handlers 675ms + ✓ tests/redact.test.ts (54 tests) 10ms + ✓ tests/cli-probe.test.ts (17 tests) 1830ms + ✓ tests/authored-advisory-run-live.test.ts (16 tests) 883ms + ✓ tests/webhook-live.test.ts (6 tests) 9746ms + ✓ executes and deduplicates 'app_mention' only for its provider and matching payload 1471ms + ✓ executes and deduplicates 'reaction_added' only for its provider and matching payload 1447ms + ✓ executes and deduplicates 'pull_request' only for its provider and matching payload 1448ms + ✓ flows serve-webhook writes JSON before the daemon starts, then journals and archives exactly once 1444ms + ✓ replays a dropped file after SIGKILL before spawn 478ms + ✓ resumes the same journal after SIGKILL after spawn and before acknowledgement 3457ms + ✓ tests/authored-parallel-agents.test.ts (8 tests) 8916ms + ✓ authored steps under local workers with capacity > runs more concurrent f.llm calls than the worker holds side by side, never more than its capacity 1086ms + ✓ authored steps under local workers with capacity > completes more concurrent f.agent calls than the worker holds: the overflow waits for a slot instead of parking 1473ms + ✓ authored steps under local workers with capacity > runs agents in distinct working directories side by side (the kernel carries cwd) 596ms + ✓ authored steps under local workers with capacity > serializes agents whose cwd is a symlink alias of the same directory 1014ms + ✓ authored steps under local workers with capacity > serializes agents whose cwd is a directory nested inside the other 1036ms + ✓ authored steps under local workers with capacity > never starts queued agents once the body has failed 1568ms + ✓ authored steps under local workers with capacity > never starts a queued agent when the agent holding the only slot fails 1570ms + ✓ authored steps under local workers with capacity > parks the overflow when the body is not told the capacity (the defect this closes) 573ms + ✓ tests/local-agent-live.test.ts (7 tests) 42312ms + ✓ built CLI local agent against a real daemon > dispatches through the wrapper and keeps --json stdout report-shaped 766ms + ✓ built CLI local agent against a real daemon > runs beyond the initial 30-second lease without a second invocation 35817ms + ✓ built CLI local agent against a real daemon > renders actual agent completion in text output 888ms + ✓ built CLI local agent against a real daemon > returns a failed run when the agent process fails 813ms + ✓ built CLI local agent against a real daemon > prints a remedy that runs the parked authored flow to completion 1608ms + ✓ built CLI local agent against a real daemon > prints a remedy that runs, for a run started from an input file too long to be one 1818ms + ✓ built CLI local agent against a real daemon > refuses a workspace it cannot pin before invoking the agent 600ms + ✓ tests/cli-watch.test.ts (10 tests) 16514ms + ✓ flows check --watch > rechecks syntax errors, clears once, and returns the last refusal on Ctrl-C 1335ms + ✓ flows check --watch > streams JSON lines without ANSI, recovers after atomic saves, and exits zero after repair 1867ms + ✓ flows check --watch > coalesces 20 concurrent saves into at most two rechecks 1870ms + ✓ flows check --watch > watches transitive relative use imports, cycles, and nearest config changes 2461ms + ✓ flows check --watch > refreshes the import graph and notices missing imports being created 2520ms + ✓ flows check --watch > reloads authored TypeScript instead of reusing the first imported definition 1804ms + ✓ flows check --watch > detects a nearer config appearing and falls back after it is deleted 1973ms + ✓ flows check --watch > keeps watching after the target is deleted and recreated 1905ms + ✓ flows check --watch > queues changes during a slow check without overlapping checks 775ms + ✓ tests/check-worker-surface.test.ts (11 tests) 80ms + ✓ tests/human-live.test.ts (3 tests) 6825ms + ✓ f.human against a real daemon > parks with the question, refuses wrong answers, records one, and resumes to success 3963ms + ✓ f.human against a real daemon > a "no" is a value the body branches on: declined, exit 0, no effect 1759ms + ✓ f.human against a real daemon > refuses to answer a run the daemon does not know 1102ms +(node:58939) ExperimentalWarning: SQLite is an experimental feature and might change at any time +(Use `node --trace-warnings ...` to show where the warning was created) + ✓ tests/authored-status-detail.test.ts (20 tests) 124ms + ✓ tests/authored-step-failed.test.ts (10 tests) 42ms + ✓ tests/budget-preflight.test.ts (25 tests) 18ms + ✓ tests/authored-retried-child-resume-live.test.ts (2 tests) 1732ms + ✓ an authored step whose child run retried an attempt > resumes after a kill mid-step and reads the terminal completion, not the crashed one 1619ms + ✓ tests/provider-trigger-contract.test.ts (7 tests) 872ms + ✓ provider trigger contract > accepts one subscription from each of five providers and refuses a bogus event on any of them 510ms + ✓ provider trigger contract > fails `flows check` before deployment and passes once the event is real 357ms + ✓ tests/shipped-source-flow-header-provenance.test.ts (1 test) 2910ms + ✓ shipped-source flow header provenance > fails closed for unsafe authored and declarative headers 2910ms + ✓ tests/work-package-consumer.test.ts (13 tests) 111ms + ✓ tests/helpers-fanout.test.ts (96 tests) 159ms +(node:59280) ExperimentalWarning: SQLite is an experimental feature and might change at any time +(Use `node --trace-warnings ...` to show where the warning was created) + ✓ tests/authored-step-graph-live.test.ts (1 test) 944ms + ✓ the authored step DAG through the live kernel > carries labels and predecessors on every index record and journal step, ids unchanged 943ms + ✓ tests/named-gate-journal.test.ts (5 tests) 1760ms + ✓ a lowered subprocess_gate journals the gate command's streams > persists what the gate printed before a timeout killed it 828ms + ✓ a gate on an agent step > journals the command's streams and surfaces them in the authored failure 647ms + ✓ tests/generate-triggers.test.ts (7 tests) 1403ms + ✓ discovers new adapters, preserves exact event names, and prefers adapter-local mappings 404ms + ✓ tests/local-agent-remedy.test.ts (16 tests) 13ms + ✓ tests/yaml-local-agent-live.test.ts (9 tests) 6529ms + ✓ YAML --local-agent through the built CLI and real daemon > runs with the checked step CLI and model and journals done 601ms + ✓ YAML --local-agent through the built CLI and real daemon > runs with the checked named CLI and model and journals done 622ms + ✓ YAML --local-agent through the built CLI and real daemon > runs with the checked flow CLI and model and journals done 590ms + ✓ YAML --local-agent through the built CLI and real daemon > runs with the checked project CLI and model and journals done 582ms + ✓ YAML --local-agent through the built CLI and real daemon > still parks without --local-agent 539ms + ✓ YAML --local-agent through the built CLI and real daemon > resumes a parked spec run with --local-agent instead of parking identically again 1004ms + ✓ YAML --local-agent through the built CLI and real daemon > parks a second time with a different message when the attached worker is not eligible 1020ms + ✓ YAML --local-agent through the built CLI and real daemon > reports the agent process failure 852ms + ✓ YAML --local-agent through the built CLI and real daemon > preserves declared workspace surfaces that the local worker cannot pin 717ms + ✓ tests/yaml-helpers.test.ts (34 tests) 77ms + ✓ tests/direct-run-failure.test.ts (12 tests) 41ms + ✓ tests/authored-agent-permissions.test.ts (27 tests) 902ms + ✓ tests/authored-detail-live.test.ts (3 tests) 3737ms + ✓ carries done("step_failed", { detail }) into the report, the journal and flows status 2146ms + ✓ reports a detail a caller sliced through an emoji, instead of hanging on it 603ms + ✓ leaves a one-argument done("step_failed") reporting exactly as it always did 987ms + ✓ tests/live-event-activities.test.ts (2 tests) 202ms + ✓ tests/software-garden-babysitter-composition.test.ts (6 tests | 2 skipped) 43ms + ✓ tests/worker-lease.test.ts (7 tests) 18ms + ✓ tests/worker-lease-lost.test.ts (17 tests) 25ms + ✓ tests/webhook-hardening.test.ts (11 tests) 58ms + ✓ tests/authored-probe-cache.test.ts (5 tests) 21050ms + ✓ authored run CLI probe cache > nine calls have no expired attempts at capacity 1 6998ms + ✓ authored run CLI probe cache > nine calls have no expired attempts at capacity 4 5396ms + ✓ authored run CLI probe cache > does not serialize nine starts behind nine probes, and probes again on a new run 7027ms + ✓ authored run CLI probe cache > caches probe failures while refusing all nine calls 397ms + ✓ authored run CLI probe cache > shares probe results across agent calls too 1232ms + ✓ tests/yaml-declared-streams-live.test.ts (8 tests) 5063ms + ✓ YAML declared streams through the built CLI and real daemon > runs with the checked step CLI and model and journals done 609ms + ✓ YAML declared streams through the built CLI and real daemon > runs with the checked named CLI and model and journals done 599ms + ✓ YAML declared streams through the built CLI and real daemon > runs with the checked flow CLI and model and journals done 606ms + ✓ YAML declared streams through the built CLI and real daemon > runs with the checked project CLI and model and journals done 587ms + ✓ YAML declared streams through the built CLI and real daemon > still parks without --local-agent 539ms + ✓ YAML declared streams through the built CLI and real daemon > reports the agent process failure 613ms + ✓ YAML declared streams through the built CLI and real daemon > resumes the journaled stream declarations after the input YAML changes 1001ms + ✓ YAML declared streams through the built CLI and real daemon > preserves declared workspace surfaces that the local worker cannot pin 508ms + ✓ tests/human-to.test.ts (8 tests) 13ms +(node:61719) ExperimentalWarning: SQLite is an experimental feature and might change at any time +(Use `node --trace-warnings ...` to show where the warning was created) + ✓ tests/authored-dispatch-live.test.ts (2 tests) 2417ms + ✓ durable child flow dispatch > runs a declared child in the parent DAG and replays it without repeating effects 1621ms + ✓ durable child flow dispatch > lowers a named gate attached to the dispatch receipt 794ms + ✓ tests/plugin-loader.test.ts (9 tests) 195ms + ✓ tests/worker-lease-drift-live.test.ts (3 tests) 36205ms + ✓ worker lease under clock skew between worker and daemon > renews before the daemon deadline when the worker clock runs 75s behind the daemon 36062ms + ✓ tests/deploy.test.ts (11 tests) 5316ms + ✓ flows deploy file buckets > publishes the full signed layout byte-for-byte and redeploys as a noop 824ms + ✓ flows deploy file buckets > answers --json with one object per outcome 830ms + ✓ flows deploy file buckets > reports a refusal as JSON under --json 398ms + ✓ flows deploy file buckets > refuses a missing local bundle before creating the bucket 394ms + ✓ flows deploy file buckets > refuses an unreachable bucket before copying 403ms + ✓ flows deploy file buckets > refuses an unwritable bucket 410ms + ✓ flows deploy file buckets > refuses local tampering of spec.canonical.json 405ms + ✓ flows deploy file buckets > refuses local tampering of identity.json 419ms + ✓ flows deploy file buckets > refuses asset bundles instead of using daemon-relative files 392ms + ✓ flows deploy file buckets > never labels a corrupt existing deployment as a noop 813ms + ✓ tests/bin.test.ts (12 tests) 5976ms + ✓ built flows binary > prints the installed SDK version for --version 383ms + ✓ built flows binary > prints the installed SDK version for -V 389ms + ✓ built flows binary > keeps version flags strict when extra arguments are supplied 377ms + ✓ built flows binary > prints the installed SDK version from the standalone binary for --version 973ms + ✓ built flows binary > prints the installed SDK version from the standalone binary for -V 987ms + ✓ built flows binary > refuses through a symlink to the built artifact 383ms + ✓ built flows binary > refuses through a symlinked directory component 400ms + ✓ built flows binary > classifies a signal-terminated auth probe as probe_failed 859ms + ✓ built flows binary > classifies an unavailable PATH resolver as probe_failed 420ms + ✓ built flows binary > does not describe a present non-executable CLI as missing 403ms + ✓ built flows binary > runs one auth probe for three steps sharing a flow CLI 399ms + ✓ tests/babysitter-catalog-export.test.ts (14 tests) 759ms + ✓ Babysitter catalog artifact export > CLI refuses an existing output and leaves no file on validation failure 606ms + ✓ tests/journal-client-read-timeout.test.ts (13 tests) 1065ms + ✓ a recovered read timeout does not become an authored callback failure 337ms + ✓ tests/communication.test.ts (10 tests) 13ms + ✓ tests/communication-review.test.ts (6 tests) 324ms + ✓ tests/transport-evidence-bounds.test.ts (7 tests) 11ms + ✓ tests/typed-output.test.ts (14 tests) 209ms + ✓ tests/shipped-source-models.test.ts (2 tests) 39331ms + ✓ first-party shipped source model pins > gives every TypeScript agent and LLM an explicit supported pair or a pinned named-agent declaration 39293ms + ✓ tests/agent-timeout-live.test.ts (3 tests) 4254ms + ✓ journals timeout, stops the process, runs a predicate gate and publishes under a budget header 1156ms + ✓ replays timeout after killing the root and daemon, without executing the agent again 1973ms + ✓ lets an author reject timeout through a predicate gate 1123ms + ✓ tests/effect-channel.test.ts (5 tests) 397ms + ✓ tests/worker-slots.test.ts (8 tests) 7ms + ✓ tests/step-lease.test.ts (36 tests) 66445ms + ✓ f.run leases against the live kernel > enforces 10000 ms for 'sleep 5; printf ok' 5065ms + ✓ f.run leases against the live kernel > enforces 40000 ms for 'sleep 31; printf ok' 31101ms + ✓ f.run leases against the live kernel > enforces 30000 ms for 'sleep 31; printf ok' 30086ms + ✓ tests/json-schema-bound.test.ts (71 tests) 2320ms + ✓ JSON Schema termination bound > walks a deep schema with an explicit stack rather than recursion 1896ms + ✓ tests/cloud-mirror-transport.test.ts (5 tests) 9ms + ✓ tests/mcp-lifecycle.test.ts (4 tests) 19ms + ✓ tests/communication-worker.test.ts (17 tests) 1708ms + ✓ tests/model-selection.test.ts (10 tests) 17ms + ✓ tests/f-memory.test.ts (7 tests) 781ms + ✓ tests/relayflowd-path.test.ts (10 tests) 5ms + ✓ tests/agent-artifacts.test.ts (9 tests) 20ms + ✓ tests/authored-plugin-effect.test.ts (6 tests) 65ms + ✓ tests/agent-timeout-worker.test.ts (9 tests) 2190ms + ✓ stops claude at its declared limit, preserving work and stopping descendants 523ms + ✓ stops codex at its declared limit, preserving work and stopping descendants 517ms + ✓ stops wrapper at its declared limit, preserving work and stopping descendants 536ms + ✓ tests/hosted-hardening.test.ts (11 tests) 15ms + ✓ tests/local-dev-ux.test.ts (8 tests) 14ms + ✓ tests/authored-preflight.test.ts (5 tests) 214ms + ✓ tests/deterministic-llm.test.ts (5 tests) 53ms + ✓ tests/relay-cli-surface-live.test.ts (3 tests) 416ms + ✓ tests/authored-declined.test.ts (13 tests) 51ms + ✓ tests/resume-failure.test.ts (2 tests) 5ms + ✓ tests/dependency-validation.test.ts (6 tests) 759ms + ✓ dependency validation > accepts a valid 10,000-step reverse chain through every direct public boundary 417ms + ✓ tests/worker-lease-lost-live.test.ts (3 tests) 878ms + ✓ reports journal success after completion rejects with lease_conflict 315ms + ✓ tests/authored-hooks.test.ts (5 tests) 6ms + ✓ tests/input-binding.test.ts (12 tests) 201ms + ✓ tests/yaml-helper-effect.test.ts (4 tests) 83ms + ✓ tests/cloud-mirror-ledger.test.ts (7 tests) 1121ms + ✓ the mirror ledger > keeps the newest entries when the count cap bites 1109ms + ✓ tests/scope-preflight.test.ts (6 tests) 8ms + ✓ tests/run-from-digest.test.ts (6 tests) 4580ms + ✓ flows run digest input > submits the sealed canonical spec through the normal journal path without checkout 431ms + ✓ flows run digest input > uses a verified cache hit even after the bucket is removed 453ms + ✓ flows run digest input > resolves deploy.bucket from flows.json and honors explicit override 1216ms + ✓ flows run digest input > refuses an unconfigured bucket 809ms + ✓ flows run digest input > refuses tampered spec.canonical.json before creating run data 810ms + ✓ flows run digest input > refuses tampered identity.json before creating run data 859ms + ✓ tests/build-gate.test.ts (3 tests) 1201ms + ✓ flows build gates on flows check green (#318) > refuses a flow with an unresolvable named-agent CLI and leaves no artifacts 388ms + ✓ flows build gates on flows check green (#318) > --json emits one CheckReport object on stdout on refusal, exits 2, no artifacts 394ms + ✓ flows build gates on flows check green (#318) > builds the bundle on success (regression: gate must not block valid flows) 418ms + ✓ tests/scope-compiler.test.ts (25 tests) 11ms + ✓ tests/worker-cli-abort.test.ts (3 tests) 3748ms + ✓ stops claude and its process group when lease ownership is lost 1365ms + ✓ stops wrapper.mjs and its process group when lease ownership is lost 1350ms + ✓ fails closed when abort cannot prove the process group is gone 1033ms + ✓ tests/hn-poller.test.ts (6 tests) 7ms + ✓ tests/plugin-add.test.ts (7 tests) 1144ms + ✓ typechecks the augmented verb and rejects unknown namespaces 855ms + ✓ tests/authored-step-failed-exit.test.ts (3 tests) 9ms + ✓ tests/run-read-load-live.test.ts (2 tests) 3060ms + ✓ completes a CPU-saturating deterministic flow with reads in flight and preserves its journal 2431ms + ✓ drains read and watch promises before an authored flow completes 628ms + ✓ tests/retried-step-failure.test.ts (1 test) 904ms + ✓ a retried step failing differently through the live kernel > reports both attempts and says the evidence differs 903ms + ✓ tests/dir-watcher-poller.test.ts (6 tests) 7ms + ✓ tests/subscription-report.test.ts (8 tests) 5ms + ✓ tests/worker-lease-sweep.test.ts (4 tests) 9ms + ✓ tests/model-pricing.test.ts (10 tests) 5ms + ✓ tests/authored-use-loader.test.ts (6 tests) 693ms + ✓ tests/yaml-helper-live.test.ts (1 test) 957ms + ✓ runs compiled YAML helpers through the built CLI and kernel effect journal 957ms + ✓ tests/provider-trigger-executor.test.ts (4 tests) 228ms + ✓ tests/journal-client-completion.test.ts (6 tests) 104ms + ✓ tests/transcript-tail-close.test.ts (2 tests) 1303ms + ✓ a stalled transcript-tail close > does not hold the spawn open past its bounded window 647ms + ✓ a stalled tail close beside a transcript that finished > still journals the transcript pointer 654ms + ✓ tests/wrapper-artifacts-cwd.test.ts (2 tests) 123ms + ✓ tests/hello-deterministic.test.ts (5 tests) 17ms + ✓ tests/transcript-exclusion-timeout.test.ts (1 test) 287ms + ✓ tests/cli-adapter.test.ts (4 tests) 5ms + ✓ tests/communication-mixed-resume.test.ts (1 test) 169ms + ✓ tests/work-package-validator.test.ts (7 tests) 5ms + ✓ tests/activity-preflight.test.ts (10 tests) 21ms + ✓ tests/authored-declined-live.test.ts (1 test) 1606ms + ✓ runs an input guard and resumes its completed declined root without repeated effects 1605ms + ✓ tests/cli-answer.test.ts (15 tests) 10ms + ✓ tests/bundle-preflight.test.ts (4 tests) 880ms + ✓ bundle execution preflight > ignores surrounding cache configuration on a verified cache hit 448ms + ✓ bundle execution preflight > uses the built alias for a nameless flow even in a digest-only cache directory 416ms + ✓ tests/agent-timeout-outcome.test.ts (6 tests) 18ms + ✓ tests/agent-relay-hardening.test.ts (12 tests) 16ms + ✓ tests/resume-worker-lease.test.ts (3 tests) 6ms + ✓ tests/communication-preflight.test.ts (13 tests) 31ms + ✓ tests/direct-run-worker-lease.test.ts (3 tests) 9ms + ↓ tests/real-cli-adapters.test.ts (3 tests | 3 skipped) + ✓ tests/memoization.test.ts (57 tests) 54ms + ✓ tests/shipped-source-call-array-forwarding.test.ts (1 test) 2781ms + ✓ shipped-source call-array forwarding > maps rest indexes and nested same-helper actuals for workers and flows 2780ms + ✓ tests/local-agent-environment.test.ts (8 tests) 6ms + ✓ tests/fs-descriptor.test.ts (1 test) 4ms + ✓ tests/running-step-watch.test.ts (2 tests) 2120ms + ✓ uses pushes for completion with lease-cadence reads and releases its watcher 2117ms + ✓ tests/runtime-dependencies.test.ts (1 test) 210ms + ✓ tests/parse-json-output.test.ts (7 tests) 4ms + ✓ tests/reported-cost.test.ts (4 tests) 3ms + ✓ tests/journal-client-subscriptions.test.ts (1 test) 8ms + ✓ tests/authored-model-probe-kinds.test.ts (2 tests) 199ms + ✓ tests/communication-environment-preflight.test.ts (6 tests) 5ms + ✓ tests/budget-authored-live.test.ts (2 tests) 181ms + ✓ tests/slack-writeback.test.ts (1 test) 257ms + ✓ tests/authored-surface-authority.test.ts (2 tests) 19ms + ✓ tests/adapters/claude.test.ts (7 tests) 4ms + ✓ tests/worker-cli-cwd.test.ts (2 tests) 360ms + ✓ runAgentCli — cwd propagation (flows#357) > omits cwd when not provided (inherits parent cwd) 354ms + ✓ tests/adapters/codex.test.ts (7 tests) 5ms + ✓ tests/shipped-source-parameter-provenance.test.ts (1 test) 1630ms + ✓ shipped-source parameter provenance > keeps caller-supplied identifier and destructured keys unknown 1629ms + ✓ tests/slack-block-kit.test.ts (5 tests) 14ms + ✓ tests/agent-timeout.test.ts (23 tests) 35ms + ✓ tests/communication-history.test.ts (1 test) 3ms + ✓ tests/adapters/registry.test.ts (4 tests) 4ms + ✓ tests/authored-declined-report.test.ts (6 tests) 7ms + ✓ tests/promise-ancestry.test.ts (2 tests) 260ms + ✓ tests/agent-cwd-validation.test.ts (2 tests) 397ms + ✓ declarative agent cwd > is refused by `flows check` on a YAML flow before anything runs 394ms + ✓ tests/communication-refusal.test.ts (1 test) 14ms + ✓ tests/bundle-transport.test.ts (20 tests) 2491ms + ✓ digest references > accepts and deploys the build output for hello 463ms + ✓ digest references > accepts and deploys the build output for Hello 413ms + ✓ digest references > accepts and deploys the build output for hello.world 408ms + ✓ digest references > accepts and deploys the build output for hello_world 397ms + ✓ digest references > accepts and deploys the build output for 123 408ms + ✓ digest references > accepts and deploys the build output for A_b.c-1 399ms + ✓ tests/cli-cloud-mirror-flag.test.ts (4 tests) 3ms + ✓ tests/catalog-plugins.test.ts (2 tests) 4ms + ✓ tests/check-command-cwd.test.ts (1 test) 23ms + ✓ tests/communication-lazy.test.ts (1 test) 5ms + ✓ tests/cli-progress-wait.test.ts (2 tests) 4ms + ✓ tests/run-digest-live.test.ts (1 test) 924ms + ✓ executes a deployed digest on the real kernel after deleting the authoring tree 923ms + ✓ tests/run-daemon-unresponsive.test.ts (2 tests) 4ms + ✓ tests/placement.test.ts (54 tests) 18ms + ✓ tests/preflight-run-cache.test.ts (1 test) 5ms + ✓ tests/heartbeat-timeout.test.ts (1 test) 15ms + ✓ tests/canonical-tree.test.ts (1 test) 2ms + ✓ tests/local-agent-stream-selection.test.ts (1 test) 5ms + ✓ tests/run-digest.test.ts (4 tests) 1552ms + ✓ digest run configuration refusals > reports config_invalid before fetching or starting a run for {invalid json 397ms + ✓ digest run configuration refusals > reports config_invalid before fetching or starting a run for {"deploy":{}} 383ms + ✓ digest run configuration refusals > reports config_invalid before fetching or starting a run for {"deploy":{"bucket":123}} 380ms + ✓ digest run configuration refusals > reports config_invalid before fetching or starting a run for {"deploy":{"bucket":""}} 391ms + ✓ tests/communication-tools.test.ts (1 test) 64ms + ✓ tests/authored-admission.test.ts (2 tests) 3ms + ✓ tests/memory.test.ts (18 tests) 8ms + ✓ tests/worker-platform.test.ts (1 test) 3ms +$ /usr/local/share/nvm/versions/node/v25.6.0/bin/node scripts/pack-release.mjs surface /tmp/flows-cli-package-h2FUqF/tarballs +PACK_OK @relayflows/surface@2.0.42: package/dist/index.js, package/dist/index.d.ts, package/dist/runtime.js, package/dist/runtime.d.ts +$ /usr/local/share/nvm/versions/node/v25.6.0/bin/node scripts/pack-release.mjs sdk /tmp/flows-cli-package-h2FUqF/tarballs +PACK_OK @relayflows/sdk@2.0.42: package/dist/index.js, package/dist/index.d.ts, package/dist/cli.js +$ /usr/local/share/nvm/versions/node/v25.6.0/bin/node scripts/pack-release.mjs relayflows /tmp/flows-cli-package-h2FUqF/tarballs +PACK_OK relayflows@2.0.42: package/bin/flows.js +$ npm install --ignore-scripts --omit=optional --no-audit --no-fund /tmp/flows-cli-package-h2FUqF/tarballs/relayflows-2.0.42.tgz /tmp/flows-cli-package-h2FUqF/tarballs/relayflows-sdk-2.0.42.tgz /tmp/flows-cli-package-h2FUqF/tarballs/relayflows-surface-2.0.42.tgz + +added 150 packages in 5s +$ /tmp/flows-cli-package-h2FUqF/local/node_modules/.bin/flows check examples/dependency-upgrade-bot.flow.ts +REQUIRES claude (agent "upgrader") +CHECK PASSED examples/dependency-upgrade-bot.flow.ts +$ /usr/local/share/nvm/versions/node/v25.6.0/bin/node /tmp/flows-cli-package-h2FUqF/local/node_modules/relayflows/bin/flows.js check examples/dependency-upgrade-bot.flow.ts +REQUIRES claude (agent "upgrader") +CHECK PASSED examples/dependency-upgrade-bot.flow.ts +$ npm install -g --prefix /tmp/flows-cli-package-h2FUqF/prefix --ignore-scripts --omit=optional --no-audit --no-fund /tmp/flows-cli-package-h2FUqF/tarballs/relayflows-2.0.42.tgz /tmp/flows-cli-package-h2FUqF/tarballs/relayflows-sdk-2.0.42.tgz /tmp/flows-cli-package-h2FUqF/tarballs/relayflows-surface-2.0.42.tgz + +added 294 packages in 5s +$ npm install --ignore-scripts --omit=optional --no-audit --no-fund /tmp/flows-cli-package-h2FUqF/tarballs/relayflows-surface-2.0.42.tgz + +added 140 packages in 2s +$ /tmp/flows-cli-package-h2FUqF/prefix/bin/flows check examples/dependency-upgrade-bot.flow.ts +REQUIRES claude (agent "upgrader") +CHECK PASSED examples/dependency-upgrade-bot.flow.ts +$ /usr/local/share/nvm/versions/node/v25.6.0/bin/node /tmp/flows-cli-package-h2FUqF/prefix/lib/node_modules/relayflows/bin/flows.js check examples/dependency-upgrade-bot.flow.ts +REQUIRES claude (agent "upgrader") +CHECK PASSED examples/dependency-upgrade-bot.flow.ts +CLI_PACKAGE_OK: local and global installs + ✓ tests/cli-package-gate.test.ts (1 test) 19640ms + ✓ checks authored flows with locally and globally installed release tarballs 19639ms + ✓ tests/event-await-cli.test.ts (1 test) 13231ms + ✓ parks, restarts, and replays two event wakes through the actual CLI 13228ms + ✓ tests/authored-parallel-llm.test.ts (6 tests) 115215ms + ✓ parallel llm capacity 1 > deduplicates concurrent preflight probes 3257ms + ✓ parallel llm capacity 1 > completes nine calls without expired child leases during slow preflight 8921ms + ✓ parallel llm capacity 1 > keeps the durable root lease alive across two cold models 47937ms + ✓ parallel llm capacity 4 > deduplicates concurrent preflight probes 1356ms + ✓ parallel llm capacity 4 > completes nine calls without expired child leases during slow preflight 7251ms + ✓ parallel llm capacity 4 > keeps the durable root lease alive across two cold models 46493ms + ✓ tests/shipped-source-worker-invocations.test.ts (1 test) 756344ms + ✓ shipped-source worker invocation resolution > fails closed across direct, extracted, bound, mutable, and escaped callables 756343ms + +⎯⎯⎯⎯⎯⎯ Failed Suites 1 ⎯⎯⎯⎯⎯⎯⎯ + + FAIL tests/authored-node-runtime.test.ts [ tests/authored-node-runtime.test.ts ] +AssertionError: expected '1.3.6' to be '1.4.0' // Object.is equality + +Expected: "1.4.0" +Received: "1.3.6" + + ❯ tests/authored-node-runtime.test.ts:18:77 + 16| + 17| beforeAll(() => { + 18| expect(spawnSync(bun, ['--version'], { encoding: 'utf8' }).stdout.tr… + | ^ + 19| expect(existsSync(daemon), 'build the current kernel or set RELAYFLO… + 20| stage = mkdtempSync(join(tmpdir(), 'authored-standalone-build-')); + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[1/78]⎯ + +⎯⎯⎯⎯⎯⎯ Failed Tests 77 ⎯⎯⎯⎯⎯⎯⎯ + + FAIL tests/canonical-software-factory.test.ts > canonical software-factory metadata contract > opens the actual catalog flow with the ticket title and exactly one GitHub closing line +TypeError: expected an @relayflows/surface flow handle + ❯ Module.getFlowDefinition node_modules/@relayflows/surface/src/flow.ts:169:11 + ❯ runCanonical tests/canonical-software-factory.test.ts:73:22 + 71| }; + 72| + 73| const definition = getFlowDefinition(softwareFactory); + | ^ + 74| return definition.body(context as never, { issue, approver: 'khaliq'… + 75| root, + ❯ tests/canonical-software-factory.test.ts:89:26 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[2/78]⎯ + + FAIL tests/canonical-software-factory.test.ts > canonical software-factory metadata contract > writes a Linear closing reference that the GitHub integration links back +TypeError: expected an @relayflows/surface flow handle + ❯ Module.getFlowDefinition node_modules/@relayflows/surface/src/flow.ts:169:11 + ❯ runCanonical tests/canonical-software-factory.test.ts:73:22 + 71| }; + 72| + 73| const definition = getFlowDefinition(softwareFactory); + | ^ + 74| return definition.body(context as never, { issue, approver: 'khaliq'… + 75| root, + ❯ tests/canonical-software-factory.test.ts:106:26 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[3/78]⎯ + + FAIL tests/canonical-software-factory.test.ts > canonical software-factory metadata contract > accepts a Linear team key that carries digits +TypeError: expected an @relayflows/surface flow handle + ❯ Module.getFlowDefinition node_modules/@relayflows/surface/src/flow.ts:169:11 + ❯ runCanonical tests/canonical-software-factory.test.ts:73:22 + 71| }; + 72| + 73| const definition = getFlowDefinition(softwareFactory); + | ^ + 74| return definition.body(context as never, { issue, approver: 'khaliq'… + 75| root, + ❯ tests/canonical-software-factory.test.ts:115:26 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[4/78]⎯ + + FAIL tests/canonical-software-factory.test.ts > canonical software-factory metadata contract > stops before push when a Linear ticket has no linkable identifier () + FAIL tests/canonical-software-factory.test.ts > canonical software-factory metadata contract > stops before push when a Linear ticket has no linkable identifier (not-an-issue) +TypeError: expected an @relayflows/surface flow handle + ❯ Module.getFlowDefinition node_modules/@relayflows/surface/src/flow.ts:169:11 + ❯ runCanonical tests/canonical-software-factory.test.ts:73:22 + 71| }; + 72| + 73| const definition = getFlowDefinition(softwareFactory); + | ^ + 74| return definition.body(context as never, { issue, approver: 'khaliq'… + 75| root, + ❯ tests/canonical-software-factory.test.ts:126:28 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[5/78]⎯ + + FAIL tests/canonical-software-factory.test.ts > canonical software-factory metadata contract > fails closed before push when the Linear closing reference is duplicated in the final body +TypeError: expected an @relayflows/surface flow handle + ❯ Module.getFlowDefinition node_modules/@relayflows/surface/src/flow.ts:169:11 + ❯ runCanonical tests/canonical-software-factory.test.ts:73:22 + 71| }; + 72| + 73| const definition = getFlowDefinition(softwareFactory); + | ^ + 74| return definition.body(context as never, { issue, approver: 'khaliq'… + 75| root, + ❯ tests/canonical-software-factory.test.ts:138:26 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[6/78]⎯ + + FAIL tests/canonical-software-factory.test.ts > canonical software-factory metadata contract > fails closed before push when the body carries a foreign closing reference +TypeError: expected an @relayflows/surface flow handle + ❯ Module.getFlowDefinition node_modules/@relayflows/surface/src/flow.ts:169:11 + ❯ runCanonical tests/canonical-software-factory.test.ts:73:22 + 71| }; + 72| + 73| const definition = getFlowDefinition(softwareFactory); + | ^ + 74| return definition.body(context as never, { issue, approver: 'khaliq'… + 75| root, + ❯ tests/canonical-software-factory.test.ts:150:26 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[7/78]⎯ + + FAIL tests/canonical-software-factory.test.ts > canonical software-factory metadata contract > does not mistake ordinary closing-verb prose for a reference +TypeError: expected an @relayflows/surface flow handle + ❯ Module.getFlowDefinition node_modules/@relayflows/surface/src/flow.ts:169:11 + ❯ runCanonical tests/canonical-software-factory.test.ts:73:22 + 71| }; + 72| + 73| const definition = getFlowDefinition(softwareFactory); + | ^ + 74| return definition.body(context as never, { issue, approver: 'khaliq'… + 75| root, + ❯ tests/canonical-software-factory.test.ts:162:26 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[8/78]⎯ + + FAIL tests/canonical-software-factory.test.ts > canonical software-factory metadata contract > normalizes whitespace and caps the title at 240 Unicode code points +TypeError: expected an @relayflows/surface flow handle + ❯ Module.getFlowDefinition node_modules/@relayflows/surface/src/flow.ts:169:11 + ❯ runCanonical tests/canonical-software-factory.test.ts:73:22 + 71| }; + 72| + 73| const definition = getFlowDefinition(softwareFactory); + | ^ + 74| return definition.body(context as never, { issue, approver: 'khaliq'… + 75| root, + ❯ tests/canonical-software-factory.test.ts:171:26 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[9/78]⎯ + + FAIL tests/canonical-software-factory.test.ts > canonical software-factory metadata contract > fails closed before push when GitHub identity is missing or the final body duplicates its closing line +TypeError: expected an @relayflows/surface flow handle + ❯ Module.getFlowDefinition node_modules/@relayflows/surface/src/flow.ts:169:11 + ❯ tests/canonical-software-factory.test.ts:180:24 + 178| + 179| it('fails closed before push when GitHub identity is missing or the … + 180| const definition = getFlowDefinition(softwareFactory); + | ^ + 181| const commands: string[] = []; + 182| let completionReason = ''; + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[10/78]⎯ + + FAIL tests/canonical-software-factory.test.ts > canonical software-factory review scope > fails closed for explicit defects + FAIL tests/canonical-software-factory.test.ts > canonical software-factory review scope > fails closed for no verdict + FAIL tests/canonical-software-factory.test.ts > canonical software-factory review scope > fails closed for contradictory blocked and unverified + FAIL tests/canonical-software-factory.test.ts > canonical software-factory review scope > fails closed for contradictory passed and unverified + FAIL tests/canonical-software-factory.test.ts > canonical software-factory review scope > fails closed for contradictory passed and blocked + FAIL tests/canonical-software-factory.test.ts > canonical software-factory review scope > fails closed for empty unverified +TypeError: expected an @relayflows/surface flow handle + ❯ Module.getFlowDefinition node_modules/@relayflows/surface/src/flow.ts:169:11 + ❯ runCanonical tests/canonical-software-factory.test.ts:73:22 + 71| }; + 72| + 73| const definition = getFlowDefinition(softwareFactory); + | ^ + 74| return definition.body(context as never, { issue, approver: 'khaliq'… + 75| root, + ❯ tests/canonical-software-factory.test.ts:214:26 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[11/78]⎯ + + FAIL tests/canonical-software-factory.test.ts > canonical software-factory review scope > distinguishes a missing verification prerequisite from a defect +TypeError: expected an @relayflows/surface flow handle + ❯ Module.getFlowDefinition node_modules/@relayflows/surface/src/flow.ts:169:11 + ❯ runCanonical tests/canonical-software-factory.test.ts:73:22 + 71| }; + 72| + 73| const definition = getFlowDefinition(softwareFactory); + | ^ + 74| return definition.body(context as never, { issue, approver: 'khaliq'… + 75| root, + ❯ tests/canonical-software-factory.test.ts:224:26 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[12/78]⎯ + + FAIL tests/canonical-software-factory.test.ts > canonical software-factory review scope > preserves the passed body without a scope marker or draft +TypeError: expected an @relayflows/surface flow handle + ❯ Module.getFlowDefinition node_modules/@relayflows/surface/src/flow.ts:169:11 + ❯ runCanonical tests/canonical-software-factory.test.ts:73:22 + 71| }; + 72| + 73| const definition = getFlowDefinition(softwareFactory); + | ^ + 74| return definition.body(context as never, { issue, approver: 'khaliq'… + 75| root, + ❯ tests/canonical-software-factory.test.ts:235:26 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[13/78]⎯ + + FAIL tests/canonical-software-factory.test.ts > canonical software-factory review scope > rejects a forged scope for local before publication + FAIL tests/canonical-software-factory.test.ts > canonical software-factory review scope > rejects a forged scope for github before publication +TypeError: expected an @relayflows/surface flow handle + ❯ Module.getFlowDefinition node_modules/@relayflows/surface/src/flow.ts:169:11 + ❯ runCanonical tests/canonical-software-factory.test.ts:73:22 + 71| }; + 72| + 73| const definition = getFlowDefinition(softwareFactory); + | ^ + 74| return definition.body(context as never, { issue, approver: 'khaliq'… + 75| root, + ❯ tests/canonical-software-factory.test.ts:243:26 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[14/78]⎯ + + FAIL tests/canonical-software-factory.test.ts > canonical software-factory review scope > scopes the post-review hook refusal + FAIL tests/canonical-software-factory.test.ts > canonical software-factory review scope > scopes the merge-gate hook refusal +TypeError: expected an @relayflows/surface flow handle + ❯ Module.getFlowDefinition node_modules/@relayflows/surface/src/flow.ts:169:11 + ❯ runCanonical tests/canonical-software-factory.test.ts:73:22 + 71| }; + 72| + 73| const definition = getFlowDefinition(softwareFactory); + | ^ + 74| return definition.body(context as never, { issue, approver: 'khaliq'… + 75| root, + ❯ tests/canonical-software-factory.test.ts:250:26 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[15/78]⎯ + + FAIL tests/canonical-software-factory.test.ts > canonical software-factory review scope > rejects malformed head "" before publication + FAIL tests/canonical-software-factory.test.ts > canonical software-factory review scope > rejects malformed head "not-a-sha" before publication + FAIL tests/canonical-software-factory.test.ts > canonical software-factory review scope > rejects malformed head "a'; touch injected; #" before publication +TypeError: expected an @relayflows/surface flow handle + ❯ Module.getFlowDefinition node_modules/@relayflows/surface/src/flow.ts:169:11 + ❯ runCanonical tests/canonical-software-factory.test.ts:73:22 + 71| }; + 72| + 73| const definition = getFlowDefinition(softwareFactory); + | ^ + 74| return definition.body(context as never, { issue, approver: 'khaliq'… + 75| root, + ❯ tests/canonical-software-factory.test.ts:259:26 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[16/78]⎯ + + FAIL tests/hosted-extension-isolation.test.ts > hosted extension capability isolation > executes the exact capability-only handler for a queued receipt + FAIL tests/hosted-extension-isolation.test.ts > hosted extension capability isolation > executes the exact capability-only handler for a duplicate receipt +Error: bubblewrap is unavailable + ❯ unsupported src/hosted-extension-sandbox.ts:503:9 + 501| + 502| function unsupported(message: string): never { + 503| throw new PluginError('plugin_unsupported', message); + | ^ + 504| } + 505| + ❯ executable src/hosted-extension-sandbox.ts:484:18 + ❯ Module.runHostedExtensionSandbox src/hosted-extension-sandbox.ts:176:17 + ❯ Module.runVerifiedNativeExtensionSandbox src/hosted-extension-isolation.ts:209:16 + ❯ tests/hosted-extension-isolation.test.ts:221:26 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[17/78]⎯ + + FAIL tests/hosted-extension-isolation.test.ts > hosted extension capability isolation > launches through the captured process primitive after builtin export synchronization +AssertionError: promise rejected "Error: bubblewrap is unavailable { code: '…' }" instead of resolving + ❯ tests/hosted-extension-isolation.test.ts:283:11 + 281| input: descriptor(), + 282| babysitterTurn: { queue: async () => ({ receiptId: 'receipt-… + 283| })).resolves.toEqual({ completionReason: 'success', capability… + | ^ + 284| } finally { + 285| process.execPath = originalExecPath; + +Caused by: Error: bubblewrap is unavailable + ❯ unsupported src/hosted-extension-sandbox.ts:503:9 + ❯ executable src/hosted-extension-sandbox.ts:484:18 + ❯ Module.runHostedExtensionSandbox src/hosted-extension-sandbox.ts:176:17 + ❯ Module.runVerifiedNativeExtensionSandbox src/hosted-extension-isolation.ts:209:16 + ❯ tests/hosted-extension-isolation.test.ts:277:22 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯ +Serialized Error: { code: 'plugin_unsupported' } +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[18/78]⎯ + + FAIL tests/hosted-extension-isolation.test.ts > hosted extension capability isolation > ignores inherited launcher overrides and decodes manifests with the captured Buffer intrinsic +AssertionError: promise rejected "Error: bubblewrap is unavailable { code: '…' }" instead of resolving + ❯ tests/hosted-extension-isolation.test.ts:373:11 + 371| input: descriptor('delivery-options'), + 372| babysitterTurn: { queue: async () => ({ receiptId: 'receipt-… + 373| })).resolves.toEqual({ completionReason: 'success', capability… + | ^ + 374| } finally { + 375| Buffer.prototype.toString = bufferToString; + +Caused by: Error: bubblewrap is unavailable + ❯ unsupported src/hosted-extension-sandbox.ts:503:9 + ❯ executable src/hosted-extension-sandbox.ts:484:18 + ❯ Module.runHostedExtensionSandbox src/hosted-extension-sandbox.ts:176:17 + ❯ Module.runVerifiedNativeExtensionSandbox src/hosted-extension-isolation.ts:209:16 + ❯ tests/hosted-extension-isolation.test.ts:367:22 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯ +Serialized Error: { code: 'plugin_unsupported' } +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[19/78]⎯ + + FAIL tests/hosted-extension-isolation.test.ts > hosted extension capability isolation > streams verified bytes when the live store is replaced and no writable staging path exists +AssertionError: promise rejected "Error: Hosted extension sandbox exited wi… { code: '…' }" instead of resolving + ❯ tests/hosted-extension-isolation.test.ts:431:7 + 429| return { receiptId: 'receipt-1', status: 'queued' }; + 430| } }, + 431| })).resolves.toEqual({ completionReason: 'success', capabilityCall… + | ^ + 432| expect(calls).toBe(1); + 433| expect(readFileSync(join(installed.directory, 'babysitter.flow.ts'… + +Caused by: Error: Hosted extension sandbox exited without a valid completion (exit 1): /tmp/hosted-bwrap-wrapper-Bvf1hY/bwrap-wrapper:20 +if (child.error) throw child.error; + ^ + +Error: spawnSync /usr/bin/bwrap ENOENT + at Object.spawnSync (node:internal/child_process:1103:20) + at spawnSync (node:child_process:911:24) + at Object. (/tmp/hosted-bwrap-wrapper-Bvf1hY/bwrap-wrapper:19:15) + at Module._compile (node:internal/modules/cjs/loader:1809:14) + at Object..js (node:internal/modules/cjs/loader:1940:10) + at Module.load (node:internal/modules/cjs/loader:1530:32) + at Module._load (node:internal/modules/cjs/loader:1332:12) + at wrapModuleLoad (node:internal/modules/cjs/loader:255:19) + at Module.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:154:5) + at node:internal/main/run_main_module:33:47 { + errno: -2, + code: 'ENOENT', + syscall: 'spawnSync /usr/bin/bwrap', + path: '/usr/bin/bwrap', + spawnargs: [ + '--unshare-all', + '--die-with-parent', + '--new-session', + '--clearenv', + '--cap-drop', + 'ALL', + '--dir', + '/usr', + '--ro-bind', + '/usr/lib', + '/usr/lib', + '--ro-bind', + '/usr/lib64', + '/usr/lib64', + '--ro-bind', + '/usr/lib', + '/lib', + '--ro-bind', + '/usr/lib64', + '/lib64', + '--proc', + '/proc', + '--dev', + '/dev', + '--tmpfs', + '/tmp', + '--dir', + '/runtime', + '--dir', + '/extension', + '--dir', + '/extension/node_modules', + '--dir', + '/extension/node_modules/@relayflows', + '--dir', + '/extension/node_modules/@relayflows/surface', + '--dir', + '/extension/node_modules/@relayflows/surface/dist', + '--dir', + '/extension/node_modules/@relayflows/surface/dist/helpers', + '--dir', + '/extension/node_modules/@relayflows/surface/dist/triggers', + '--dir', + '/extension/src', + '--perms', + '0500', + '--ro-bind-data', + '4', + '/runtime/node', + '--perms', + '0400', + '--ro-bind-data', + '5', + '/runtime/runner.mjs', + '--perms', + '0400', + '--ro-bind-data', + '6', + '/extension/node_modules/@relayflows/surface/package.json', + '--perms', + '0400', + '--ro-bind-data', + '7', + '/extension/node_modules/@relayflows/surface/index.js', + '--perms', + '0400', + '--ro-bind-data', + '8', + '/extension/node_modules/@relayflows/surface/runtime.js', + '--perms', + '0400', + '--ro-bind-data', + '9', + '/extension/node_modules/@relayflows/surface/dist/flow.js', + '--perms', + '0400', + '--ro-bind-data', + '10', + '/extension/node_modules/@relayflows/surface/dist/helpers/providers.js', + '--perms', + '0400', + '--ro-bind-data', + '11', + '/extension/node_modules/@relayflows/surface/dist/provider-trigger.js', + '--perms', + '0400', + '--ro-bind-data', + '12', + '/extension/node_modules/@relayflows/surface/dist/schedule.js', + '--perms', + '0400', + '--ro-bind-data', + '13', + '/extension/node_modules/@relayflows/surface/dist/triggers.js', + '--perms', + '0400', + '--ro-bind-data', + '14', + '/extension/node_modules/@relayflows/surface/dist/triggers/github.js', + '--perms', + ... 27 more items + ] +} + +Node.js v25.6.0 + + ❯ Object. ../../../../../../../tmp/hosted-bwrap-wrapper-Bvf1hY/bwrap-wrapper:19:15 + ❯ refuse src/hosted-extension-protocol.ts:135:21 + ❯ ChildProcess. src/hosted-extension-protocol.ts:234:21 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯ +Serialized Error: { code: 'plugin_unsupported' } +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[20/78]⎯ + + FAIL tests/hosted-extension-isolation.test.ts > hosted extension capability isolation > mounts pinned private Surface bytes when the live package changes before launch +AssertionError: promise rejected "Error: bubblewrap is unavailable { code: '…' }" instead of resolving + ❯ tests/hosted-extension-isolation.test.ts:456:7 + 454| return { receiptId: 'receipt-1', status: 'queued' }; + 455| } }, + 456| })).resolves.toEqual({ completionReason: 'success', capabilityCall… + | ^ + 457| expect(calls).toBe(1); + 458| expect(readFileSync(join(surfaceRoot, 'dist/flow.js'), 'utf8')).to… + +Caused by: Error: bubblewrap is unavailable + ❯ unsupported src/hosted-extension-sandbox.ts:503:9 + ❯ executable src/hosted-extension-sandbox.ts:484:18 + ❯ Module.runHostedExtensionSandbox src/hosted-extension-sandbox.ts:176:17 + ❯ Module.runVerifiedNativeExtensionSandbox src/hosted-extension-isolation.ts:209:16 + ❯ tests/hosted-extension-isolation.test.ts:443:18 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯ +Serialized Error: { code: 'plugin_unsupported' } +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[21/78]⎯ + + FAIL tests/hosted-extension-isolation.test.ts > hosted extension capability isolation > refuses oversized Surface files through the bounded descriptor reader +AssertionError: expected Error: bubblewrap is unavailable { code: '…' } to match object { code: 'plugin_unsupported', …(1) } + +- Expected ++ Received + +- Object { ++ PluginError { + "code": "plugin_unsupported", +- "message": StringContaining "cannot read pinned Surface runtime flow.js", + } + + ❯ tests/hosted-extension-isolation.test.ts:489:5 + 487| }); + 488| let calls = 0; + 489| await expect(runVerifiedNativeExtensionSandbox({ + | ^ + 490| artifact: await artifact(), + 491| manifest: validateFlowExtensionManifest(manifest()), + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[22/78]⎯ + + FAIL tests/hosted-extension-isolation.test.ts > hosted extension capability isolation > shields verified Surface files before async settlement +AssertionError: promise rejected "Error: bubblewrap is unavailable { code: '…' }" instead of resolving + ❯ tests/hosted-extension-isolation.test.ts:532:9 + 530| surfaceRoot, + 531| babysitterTurn: { queue: async () => ({ receiptId: 'receipt-1'… + 532| })).resolves.toEqual({ completionReason: 'success', capabilityCa… + | ^ + 533| } finally { + 534| if (previous === undefined) delete (Array.prototype as { then?: … + +Caused by: Error: bubblewrap is unavailable + ❯ unsupported src/hosted-extension-sandbox.ts:503:9 + ❯ executable src/hosted-extension-sandbox.ts:484:18 + ❯ Module.runHostedExtensionSandbox src/hosted-extension-sandbox.ts:176:17 + ❯ Module.runVerifiedNativeExtensionSandbox src/hosted-extension-isolation.ts:209:16 + ❯ tests/hosted-extension-isolation.test.ts:525:20 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯ +Serialized Error: { code: 'plugin_unsupported' } +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[23/78]⎯ + + FAIL tests/hosted-extension-isolation.test.ts > hosted extension capability isolation > preserves a typed host refusal while disclosing only a fixed marker to the child +AssertionError: expected Error: bubblewrap is unavailable { code: '…' } to be Error: private Cloud policy detail { code: '…' } // Object.is equality + +- Expected ++ Received + +- [Error: private Cloud policy detail] ++ [Error: bubblewrap is unavailable] + + ❯ tests/hosted-extension-isolation.test.ts:560:5 + 558| provider: 'github', eventType: 'pull_request.labeled', deliveryI… + 559| }); + 560| await expect(runVerifiedNativeExtensionSandbox({ + | ^ + 561| artifact: await artifact(source), manifest: validateFlowExtensio… + 562| babysitterTurn: { queue: async () => { throw refusal; } }, + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[24/78]⎯ + + FAIL tests/hosted-extension-isolation.test.ts > hosted extension capability isolation > denies ambient credentials, host files, writes, network, subprocesses, and undeclared context verbs +Error: bubblewrap is unavailable + ❯ unsupported src/hosted-extension-sandbox.ts:503:9 + 501| + 502| function unsupported(message: string): never { + 503| throw new PluginError('plugin_unsupported', message); + | ^ + 504| } + 505| + ❯ executable src/hosted-extension-sandbox.ts:484:18 + ❯ Module.runHostedExtensionSandbox src/hosted-extension-sandbox.ts:176:17 + ❯ Module.runVerifiedNativeExtensionSandbox src/hosted-extension-isolation.ts:209:16 + ❯ tests/hosted-extension-isolation.test.ts:624:13 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[25/78]⎯ + + FAIL tests/hosted-extension-isolation.test.ts > hosted extension capability isolation > enforces OS address-space and data bounds on native Buffer allocation +AssertionError: expected Error: bubblewrap is unavailable { code: '…' } to match object { code: 'plugin_unsupported', …(1) } + +- Expected ++ Received + +- Object { ++ PluginError { + "code": "plugin_unsupported", +- "message": StringMatching /(?:Failed to allocate memory|Array buffer allocation failed)/u, + } + + ❯ tests/hosted-extension-isolation.test.ts:646:5 + 644| }); + 645| let calls = 0; + 646| await expect(runVerifiedNativeExtensionSandbox({ + | ^ + 647| artifact: installed, + 648| manifest: validateFlowExtensionManifest(manifest()), + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[26/78]⎯ + + FAIL tests/hosted-extension-isolation.test.ts > hosted extension capability isolation > blocks extra handler fields and authority-bearing receipt fields at the parent port +AssertionError: expected Error: bubblewrap is unavailable { code: '…' } to match object { code: 'plugin_event_unroutable' } + +- Expected ++ Received + +- Object { +- "code": "plugin_event_unroutable", ++ PluginError { ++ "code": "plugin_unsupported", + } + + ❯ tests/hosted-extension-isolation.test.ts:675:5 + 673| }); + 674| let calls = 0; + 675| await expect(runVerifiedNativeExtensionSandbox({ + | ^ + 676| artifact: await artifact(source), manifest: validateFlowExtensio… + 677| babysitterTurn: { queue: async () => { calls += 1; return { rece… + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[27/78]⎯ + + FAIL tests/hosted-extension-isolation.test.ts > hosted extension capability isolation > constructs adapter authority with the captured freeze intrinsic +Error: bubblewrap is unavailable + ❯ unsupported src/hosted-extension-sandbox.ts:503:9 + 501| + 502| function unsupported(message: string): never { + 503| throw new PluginError('plugin_unsupported', message); + | ^ + 504| } + 505| + ❯ executable src/hosted-extension-sandbox.ts:484:18 + ❯ Module.runHostedExtensionSandbox src/hosted-extension-sandbox.ts:176:17 + ❯ Module.runVerifiedNativeExtensionSandbox src/hosted-extension-isolation.ts:209:16 + ❯ tests/hosted-extension-isolation.test.ts:745:22 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[28/78]⎯ + + FAIL tests/hosted-extension-isolation.test.ts > hosted extension capability isolation > writes the Surface manifest and protocol without inherited toJSON behavior +AssertionError: promise rejected "Error: bubblewrap is unavailable { code: '…' }" instead of resolving + ❯ tests/hosted-extension-isolation.test.ts:788:11 + 786| babysitterTurn: { queue: async () => ({ receiptId: 'receipt-… + 787| timeoutMs: 3_000, + 788| })).resolves.toEqual({ completionReason: 'success', capability… + | ^ + 789| } finally { + 790| if (previous === undefined) delete (Object.prototype as { toJS… + +Caused by: Error: bubblewrap is unavailable + ❯ unsupported src/hosted-extension-sandbox.ts:503:9 + ❯ executable src/hosted-extension-sandbox.ts:484:18 + ❯ Module.runHostedExtensionSandbox src/hosted-extension-sandbox.ts:176:17 + ❯ Module.runVerifiedNativeExtensionSandbox src/hosted-extension-isolation.ts:209:16 + ❯ tests/hosted-extension-isolation.test.ts:781:22 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯ +Serialized Error: { code: 'plugin_unsupported' } +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[29/78]⎯ + + FAIL tests/hosted-extension-protocol.test.ts > hosted extension hostile protocol > uses captured JSON intrinsics for the complete parent boundary +Error: bubblewrap is unavailable + ❯ unsupported src/hosted-extension-sandbox.ts:503:9 + 501| + 502| function unsupported(message: string): never { + 503| throw new PluginError('plugin_unsupported', message); + | ^ + 504| } + 505| + ❯ executable src/hosted-extension-sandbox.ts:484:18 + ❯ Module.runHostedExtensionSandbox src/hosted-extension-sandbox.ts:176:17 + ❯ Module.runVerifiedNativeExtensionSandbox src/hosted-extension-isolation.ts:209:16 + ❯ tests/hosted-extension-protocol.test.ts:326:24 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[30/78]⎯ + + FAIL tests/hosted-extension-protocol.test.ts > hosted extension hostile protocol > rejects an import-time different PR frame with zero adapter calls + FAIL tests/hosted-extension-protocol.test.ts > hosted extension hostile protocol > rejects an import-time different delivery frame with zero adapter calls + FAIL tests/hosted-extension-protocol.test.ts > hosted extension hostile protocol > rejects an import-time different event frame with zero adapter calls +AssertionError: expected Error: bubblewrap is unavailable { code: '…' } to match object { code: 'plugin_event_unroutable' } + +- Expected ++ Received + +- Object { +- "code": "plugin_event_unroutable", ++ PluginError { ++ "code": "plugin_unsupported", + } + + ❯ tests/hosted-extension-protocol.test.ts:447:5 + 445| ])('rejects an import-time %s frame with zero adapter calls', async … + 446| let calls = 0; + 447| await expect(runVerifiedNativeExtensionSandbox({ + | ^ + 448| artifact: await artifact(hostileImport([frame, { type: 'error', … + 449| manifest: validateFlowExtensionManifest(manifest()), dispatch: d… + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[31/78]⎯ + + FAIL tests/hosted-extension-protocol.test.ts > hosted extension hostile protocol > rejects two forged calls after the authoritative first outcome settles + FAIL tests/hosted-extension-protocol.test.ts > hosted extension hostile protocol > waits for a pending adapter to reject after a forged child error + FAIL tests/hosted-extension-protocol.test.ts > hosted extension hostile protocol > waits for a pending adapter to resolve after a forged child error + FAIL tests/hosted-extension-protocol.test.ts > hosted extension hostile protocol > returns a typed adapter rejection even when the hostile child hangs +Error: hostile child did not invoke the adapter + ❯ Timeout._onTimeout tests/hosted-extension-protocol.test.ts:135:45 + 133| async function waitForInvocation(invoked: Promise): Promise((resolve, reject) => { + 135| const timeout = setTimeout(() => reject(new Error('hostile child d… + | ^ + 136| void invoked.then(() => { clearTimeout(timeout); resolve(); }, rej… + 137| }); + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[32/78]⎯ + + FAIL tests/live-kernel.test.ts > built flows CLI against live relayflowd > runs hn-monitor analyze-story end-to-end via a stub agent CLI (gate 2 clause 2 demo) +AssertionError: expected { …(12) } to match object { output: { …(3) }, …(1) } +(22 matching properties omitted from actual) + +- Expected ++ Received + + Object { +- "output": Object { +- "reasoning": "stub agent runtime — deterministic output for gate-2 clause-2 demo", +- "relevance_score": 5, +- "story_title": "stub", +- }, ++ "output": null, + "verification": Object { +- "gate": "json_schema", +- "verdict": "pass", ++ "gate": "execution", ++ "verdict": "fail", + }, + } + + ❯ tests/live-kernel.test.ts:657:36 + 655| && (entry as { step_id?: string }).step_id === 'analyze-story', + 656| ) as { payload: { output: unknown; verification: unknown } } | und… + 657| expect(stepCompleted?.payload).toMatchObject({ + | ^ + 658| output: { + 659| story_title: 'stub', + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[33/78]⎯ + + FAIL tests/live-kernel.test.ts > built flows CLI against live relayflowd > hn-monitor analyze-story FAILS verification when the CLI omits required schema fields +AssertionError: expected { …(12) } to match object { …(3) } +(21 matching properties omitted from actual) + +- Expected ++ Received + + Object { +- "completionReason": "retries_exhausted", ++ "completionReason": "worker_error", + "output": null, + "verification": Object { +- "gate": "json_schema", ++ "gate": "execution", + "verdict": "fail", + }, + } + + ❯ tests/live-kernel.test.ts:752:36 + 750| // its verification record names the json_schema rejection. The re… + 751| // parsed value is nulled before the completion is persisted. + 752| expect(stepCompleted?.payload).toMatchObject({ + | ^ + 753| completionReason: 'retries_exhausted', + 754| output: null, + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[34/78]⎯ + + FAIL tests/live-kernel.test.ts > built flows CLI against live relayflowd > agent step preserves the CliResult wrapper as output when the CLI emits non-JSON text +AssertionError: expected null not to be null + ❯ tests/live-kernel.test.ts:823:24 + 821| // here (parseJsonOutput returned null on non-JSON stdout) and + 822| // these assertions would all fail. + 823| expect(output).not.toBeNull(); + | ^ + 824| expect(output.exit_code).toBe(0); + 825| expect(output.stdout_tail).toContain('looked at the story'); + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[35/78]⎯ + + FAIL tests/live-kernel.test.ts > built flows CLI against live relayflowd > AgentWorker exposes wake_context to the CLI via RELAYFLOW_WAKE_CONTEXT env var (real analyzer prerequisite) +TypeError: Cannot read properties of null (reading 'story_title') + ❯ tests/live-kernel.test.ts:891:42 + 889| ) as { payload: { output: { story_title: string; reasoning: string… + 890| expect(stepCompleted).toBeDefined(); + 891| expect(stepCompleted!.payload.output.story_title).toBe(`echoed:${s… + | ^ + 892| expect(stepCompleted!.payload.output.reasoning).toContain(String(s… + 893| + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[36/78]⎯ + + FAIL tests/live-kernel.test.ts > built flows CLI against live relayflowd > AgentWorker leaves RELAYFLOW_WAKE_CONTEXT UNSET when the run has no wake_context (undefined-vs-null pin) +TypeError: Cannot read properties of null (reading 'env_present') + ❯ tests/live-kernel.test.ts:958:38 + 956| ) as { payload: { output: { env_present: boolean } } } | undefined; + 957| expect(completed).toBeDefined(); + 958| expect(completed!.payload.output.env_present).toBe(false); + | ^ + 959| + 960| delete process.env.RELAYFLOW_WAKE_CONTEXT; + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[37/78]⎯ + + FAIL tests/live-kernel.test.ts > built flows CLI against live relayflowd > AgentWorker leaves RELAYFLOW_MODEL UNSET when the step declares no model +TypeError: Cannot read properties of null (reading 'story_title') + ❯ tests/live-kernel.test.ts:1194:38 + 1192| expect(completed).toBeDefined(); + 1193| // UNSET, not EMPTY and not the leaked parent value. + 1194| expect(completed!.payload.output.story_title).toBe('model:UNSET'); + | ^ + 1195| + 1196| delete process.env.RELAYFLOW_MODEL; + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[38/78]⎯ + + FAIL tests/live-kernel.test.ts > built flows CLI against live relayflowd > hn-monitor analyze-story reaches done through the real Claude analyzer CLI +Error: LIVE_ANALYZER_UNAVAILABLE: "/home/daytona/.relayflow-v2-supervisor/durable/repository/testdata/preflight/analyze-story-claude-cli" does not identify as relayflows-agent-cli-v1 — failing because gate-2 acceptance requires the real analyzer to execute. Set RELAYFLOWS_ALLOW_ANALYZER_SKIP=1 only if this run is not gate evidence. + ❯ tests/live-kernel.test.ts:1223:15 + 1221| const notice = `LIVE_ANALYZER_UNAVAILABLE: ${readiness.detail}`; + 1222| if (process.env['RELAYFLOWS_ALLOW_ANALYZER_SKIP'] !== '1') { + 1223| throw new Error( + | ^ + 1224| `${notice} — failing because gate-2 acceptance requires the … + 1225| + 'Set RELAYFLOWS_ALLOW_ANALYZER_SKIP=1 only if this run is … + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[39/78]⎯ + + FAIL tests/live-kernel.test.ts > a relayflow can be scheduled: tick source against live relayflowd > a tick spawns a real run whose step reports the SCHEDULED instant +AssertionError: expected null to deeply equal { schedule_id: 'heartbeat-1m', …(3) } + +- Expected: +Object { + "lag_ms": 43000, + "schedule_id": "heartbeat-1m", + "scheduled_for_ms": 1764000000000, + "slot": 29400000, +} + ++ Received: +null + + ❯ tests/live-kernel.test.ts:1739:39 + 1737| // The bound: the run reports the grid instant and its own lag, so… + 1738| // backfilled run can tell it is running for a slot from the past. + 1739| expect(completed!.payload.output).toEqual({ + | ^ + 1740| schedule_id: 'heartbeat-1m', + 1741| slot: 29_400_000, + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[40/78]⎯ + + FAIL tests/stuck-run-triage.test.ts > stuck-run-triage input validation > refuses an 8-character run-id prefix: Cloud has no prefix lookup +AssertionError: expected [Function] to throw error matching /not full Cloud run ids: c649fe14/ but got 'expected an @relayflows/surface flow …' + +- Expected: +/not full Cloud run ids: c649fe14/ + ++ Received: +"expected an @relayflows/surface flow handle" + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[41/78]⎯ + + FAIL tests/stuck-run-triage.test.ts > stuck-run-triage input validation > refuses the whole batch when any id is invalid, rather than dropping it +AssertionError: expected [Function] to throw error matching /not full Cloud run ids: nope!/ but got 'expected an @relayflows/surface flow …' + +- Expected: +/not full Cloud run ids: nope!/ + ++ Received: +"expected an @relayflows/surface flow handle" + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[42/78]⎯ + + FAIL tests/stuck-run-triage.test.ts > stuck-run-triage input validation > refuses an empty batch +AssertionError: expected [Function] to throw error matching /needs runIds/ but got 'expected an @relayflows/surface flow …' + +- Expected: +/needs runIds/ + ++ Received: +"expected an @relayflows/surface flow handle" + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[43/78]⎯ + + FAIL tests/stuck-run-triage.test.ts > stuck-run-triage input validation > refuses a batch too large for the edge step lease +AssertionError: expected [Function] to throw error matching /exceeds the 8 that fit/ but got 'expected an @relayflows/surface flow …' + +- Expected: +/exceeds the 8 that fit/ + ++ Received: +"expected an @relayflows/surface flow handle" + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[44/78]⎯ + + FAIL tests/stuck-run-triage.test.ts > stuck-run-triage input validation > accepts eight ids — the incident batch is inside the bound +AssertionError: promise rejected "TypeError: expected an @relayflows/surfac…" instead of resolving + ❯ tests/stuck-run-triage.test.ts:62:40 + 60| it('accepts eight ids — the incident batch is inside the bound', asy… + 61| const ids = Array.from({ length: 8 }, (_, i) => `${ID_A.slice(0, -… + 62| await expect(drive({ runIds: ids })).resolves.toBeDefined(); + | ^ + 63| }); + 64| }); + +Caused by: TypeError: expected an @relayflows/surface flow handle + ❯ Module.getFlowDefinition node_modules/@relayflows/surface/src/flow.ts:169:11 + ❯ drive tests/stuck-run-triage.test.ts:34:9 + ❯ tests/stuck-run-triage.test.ts:62:18 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[45/78]⎯ + + FAIL tests/stuck-run-triage.test.ts > stuck-run-triage apiUrl > refuses to send the Cloud bearer token to an unapproved origin +AssertionError: expected [Function] to throw error matching /refusing to send the Cloud bearer to…/\ but got 'expected an @relayflows/surface flow …' + +- Expected: +/refusing to send the Cloud bearer token to https:\/\/evil\.example/ + ++ Received: +"expected an @relayflows/surface flow handle" + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[46/78]⎯ + + FAIL tests/stuck-run-triage.test.ts > stuck-run-triage apiUrl > refuses a non-URL apiUrl +AssertionError: expected [Function] to throw error matching /is not a URL/ but got 'expected an @relayflows/surface flow …' + +- Expected: +/is not a URL/ + ++ Received: +"expected an @relayflows/surface flow handle" + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[47/78]⎯ + + FAIL tests/stuck-run-triage.test.ts > stuck-run-triage apiUrl > allows an approved origin and uses it in the curl +TypeError: expected an @relayflows/surface flow handle + ❯ Module.getFlowDefinition node_modules/@relayflows/surface/src/flow.ts:169:11 + ❯ drive tests/stuck-run-triage.test.ts:34:9 + 32| done: () => {}, + 33| }; + 34| await getFlowDefinition(triage).body(f as never… + | ^ + 35| return rec; + 36| } + ❯ tests/stuck-run-triage.test.ts:77:23 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[48/78]⎯ + + FAIL tests/stuck-run-triage.test.ts > stuck-run-triage apiUrl > defaults to production Cloud +TypeError: expected an @relayflows/surface flow handle + ❯ Module.getFlowDefinition node_modules/@relayflows/surface/src/flow.ts:169:11 + ❯ drive tests/stuck-run-triage.test.ts:34:9 + 32| done: () => {}, + 33| }; + 34| await getFlowDefinition(triage).body(f as never… + | ^ + 35| return rec; + 36| } + ❯ tests/stuck-run-triage.test.ts:82:23 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[49/78]⎯ + + FAIL tests/stuck-run-triage.test.ts > stuck-run-triage apiUrl > never publishes a run record the fetch did not produce +TypeError: expected an @relayflows/surface flow handle + ❯ Module.getFlowDefinition node_modules/@relayflows/surface/src/flow.ts:169:11 + ❯ drive tests/stuck-run-triage.test.ts:34:9 + 32| done: () => {}, + 33| }; + 34| await getFlowDefinition(triage).body(f as never… + | ^ + 35| return rec; + 36| } + ❯ tests/stuck-run-triage.test.ts:89:34 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[50/78]⎯ + + FAIL tests/stuck-run-triage.test.ts > stuck-run-triage edge collection > names the Worker on every wrangler invocation +TypeError: expected an @relayflows/surface flow handle + ❯ Module.getFlowDefinition node_modules/@relayflows/surface/src/flow.ts:169:11 + ❯ drive tests/stuck-run-triage.test.ts:34:9 + 32| done: () => {}, + 33| }; + 34| await getFlowDefinition(triage).body(f as never… + | ^ + 35| return rec; + 36| } + ❯ tests/stuck-run-triage.test.ts:98:33 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[51/78]⎯ + + FAIL tests/stuck-run-triage.test.ts > stuck-run-triage edge collection > accepts caller-supplied Workers and rejects option-shaped ones +TypeError: expected an @relayflows/surface flow handle + ❯ Module.getFlowDefinition node_modules/@relayflows/surface/src/flow.ts:169:11 + ❯ drive tests/stuck-run-triage.test.ts:34:9 + 32| done: () => {}, + 33| }; + 34| await getFlowDefinition(triage).body(f as never… + | ^ + 35| return rec; + 36| } + ❯ tests/stuck-run-triage.test.ts:107:33 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[52/78]⎯ + + FAIL tests/stuck-run-triage.test.ts > stuck-run-triage edge collection > falls back when GNU timeout is absent, as it is on macOS +TypeError: expected an @relayflows/surface flow handle + ❯ Module.getFlowDefinition node_modules/@relayflows/surface/src/flow.ts:169:11 + ❯ drive tests/stuck-run-triage.test.ts:34:9 + 32| done: () => {}, + 33| }; + 34| await getFlowDefinition(triage).body(f as never… + | ^ + 35| return rec; + 36| } + ❯ tests/stuck-run-triage.test.ts:115:33 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[53/78]⎯ + + FAIL tests/stuck-run-triage.test.ts > stuck-run-triage edge collection > runs the tails concurrently so wall time does not scale with the batch +TypeError: expected an @relayflows/surface flow handle + ❯ Module.getFlowDefinition node_modules/@relayflows/surface/src/flow.ts:169:11 + ❯ drive tests/stuck-run-triage.test.ts:34:9 + 32| done: () => {}, + 33| }; + 34| await getFlowDefinition(triage).body(f as never… + | ^ + 35| return rec; + 36| } + ❯ tests/stuck-run-triage.test.ts:123:33 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[54/78]⎯ + + FAIL tests/stuck-run-triage.test.ts > stuck-run-triage edge collection > records wrangler's own exit status rather than head's +TypeError: expected an @relayflows/surface flow handle + ❯ Module.getFlowDefinition node_modules/@relayflows/surface/src/flow.ts:169:11 + ❯ drive tests/stuck-run-triage.test.ts:34:9 + 32| done: () => {}, + 33| }; + 34| await getFlowDefinition(triage).body(f as never… + | ^ + 35| return rec; + 36| } + ❯ tests/stuck-run-triage.test.ts:129:33 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[55/78]⎯ + + FAIL tests/stuck-run-triage.test.ts > stuck-run-triage shell text > parses under both sh and bash +TypeError: expected an @relayflows/surface flow handle + ❯ Module.getFlowDefinition node_modules/@relayflows/surface/src/flow.ts:169:11 + ❯ drive tests/stuck-run-triage.test.ts:34:9 + 32| done: () => {}, + 33| }; + 34| await getFlowDefinition(triage).body(f as never… + | ^ + 35| return rec; + 36| } + ❯ tests/stuck-run-triage.test.ts:137:23 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[56/78]⎯ + + FAIL tests/stuck-run-triage.test.ts > stuck-run-triage shell text > collects tails with no GNU timeout on PATH, as on a stock macOS +TypeError: expected an @relayflows/surface flow handle + ❯ Module.getFlowDefinition node_modules/@relayflows/surface/src/flow.ts:169:11 + ❯ drive tests/stuck-run-triage.test.ts:34:9 + 32| done: () => {}, + 33| }; + 34| await getFlowDefinition(triage).body(f as never… + | ^ + 35| return rec; + 36| } + ❯ tests/stuck-run-triage.test.ts:157:33 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[57/78]⎯ + + FAIL tests/stuck-run-triage.test.ts > stuck-run-triage agents > declares read-only permissions on every agent +TypeError: expected an @relayflows/surface flow handle + ❯ Module.getFlowDefinition node_modules/@relayflows/surface/src/flow.ts:169:11 + ❯ drive tests/stuck-run-triage.test.ts:34:9 + 32| done: () => {}, + 33| }; + 34| await getFlowDefinition(triage).body(f as never… + | ^ + 35| return rec; + 36| } + ❯ tests/stuck-run-triage.test.ts:176:23 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[58/78]⎯ + + FAIL tests/stuck-run-triage.test.ts > stuck-run-triage agents > tells the forensics agents their evidence is untrusted +TypeError: expected an @relayflows/surface flow handle + ❯ Module.getFlowDefinition node_modules/@relayflows/surface/src/flow.ts:169:11 + ❯ drive tests/stuck-run-triage.test.ts:34:9 + 32| done: () => {}, + 33| }; + 34| await getFlowDefinition(triage).body(f as never… + | ^ + 35| return rec; + 36| } + ❯ tests/stuck-run-triage.test.ts:182:23 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[59/78]⎯ + + FAIL tests/stuck-run-triage.test.ts > stuck-run-triage fan-out > refuses a duplicate run id: two tails would share one evidence file +AssertionError: expected [Function] to throw error matching /duplicate runIds: c649fe14-0c2e-4e51-…/ but got 'expected an @relayflows/surface flow …' + +- Expected: +/duplicate runIds: c649fe14-0c2e-4e51-9a6a-4f0d1b0f77aa/ + ++ Received: +"expected an @relayflows/surface flow handle" + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[60/78]⎯ + + FAIL tests/stuck-run-triage.test.ts > stuck-run-triage fan-out > refuses a duplicate Worker name for the same reason +AssertionError: expected [Function] to throw error matching /duplicate workers: w-one/ but got 'expected an @relayflows/surface flow …' + +- Expected: +/duplicate workers: w-one/ + ++ Received: +"expected an @relayflows/surface flow handle" + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[61/78]⎯ + + FAIL tests/stuck-run-triage.test.ts > stuck-run-triage fan-out > bounds ids x workers, not just ids +AssertionError: expected [Function] to throw error matching /24 concurrent tails, over the 16/ but got 'expected an @relayflows/surface flow …' + +- Expected: +/24 concurrent tails, over the 16/ + ++ Received: +"expected an @relayflows/surface flow handle" + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[62/78]⎯ + +⎯⎯⎯⎯⎯⎯ Unhandled Errors ⎯⎯⎯⎯⎯⎯ + +Vitest caught 1 unhandled error during the test run. +This might cause false positive tests. Resolve unhandled errors to make sure your tests are not affected. + +⎯⎯⎯⎯ Unhandled Rejection ⎯⎯⎯⎯⎯ +Error: bubblewrap is unavailable + ❯ unsupported src/hosted-extension-sandbox.ts:503:9 + 501| + 502| function unsupported(message: string): never { + 503| throw new PluginError('plugin_unsupported', message); + | ^ + 504| } + 505| + ❯ executable src/hosted-extension-sandbox.ts:484:18 + ❯ Module.runHostedExtensionSandbox src/hosted-extension-sandbox.ts:176:17 + ❯ Module.runVerifiedNativeExtensionSandbox src/hosted-extension-isolation.ts:209:16 + ❯ tests/hosted-extension-protocol.test.ts:471:17 + ❯ node_modules/@vitest/runner/dist/index.js:533:5 + ❯ runTest node_modules/@vitest/runner/dist/index.js:1056:11 + ❯ runSuite node_modules/@vitest/runner/dist/index.js:1205:15 + ❯ runSuite node_modules/@vitest/runner/dist/index.js:1205:15 + ❯ runFiles node_modules/@vitest/runner/dist/index.js:1262:5 + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯ +Serialized Error: { code: 'plugin_unsupported' } +This error originated in "tests/hosted-extension-protocol.test.ts" test file. It doesn't mean the error was thrown inside the file itself, but while it was running. +The latest test that might've caused the error is "rejects two forged calls after the authoritative first outcome settles". It might mean one of the following: +- The error was thrown, while Vitest was running this test. +- If the error occurred after the test had been completed, this was the last documented test before it was thrown. +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯ + + Test Files 6 failed | 248 passed | 1 skipped (255) + Tests 77 failed | 3774 passed | 22 skipped (3873) + Errors 1 error + Start at 10:40:22 + Duration 757.43s (transform 4.18s, setup 1.09s, collect 64.50s, tests 1765.01s, environment 32ms, prepare 10.44s) + +exit=1 diff --git a/evidence/run-read-timeout/handshake-probe-baseline.mjs b/evidence/run-read-timeout/handshake-probe-baseline.mjs new file mode 100644 index 00000000..eb71885c --- /dev/null +++ b/evidence/run-read-timeout/handshake-probe-baseline.mjs @@ -0,0 +1,25 @@ +import {mkdtempSync,readFileSync,rmSync} from 'node:fs'; +import {join} from 'node:path'; +import {tmpdir} from 'node:os'; +import {spawn} from 'node:child_process'; +const root='/home/daytona/.relayflow-v2-supervisor/durable/repository'; +const {JournalClient}=await import('/tmp/relayflow-read-timeout-baseline'+'/packages/sdk/dist/journal-client.js'); +const {AgentWorker}=await import('/tmp/relayflow-read-timeout-baseline'+'/packages/sdk/dist/worker.js'); +const {socketPathFor}=await import('/tmp/relayflow-read-timeout-baseline'+'/packages/sdk/dist/daemon-connection.js'); +const data=mkdtempSync(join(tmpdir(),'rf-read-probe-')); +const daemon=spawn('/home/daytona/.relayflows-toolchain/target/2962130851/debug/relayflowd',['--data-dir',data,'serve'],{stdio:'ignore'}); +let client,worker; +try { + for(let i=0;i<100;i++) { + client=new JournalClient(socketPathFor(data)); + try {await client.connect();await client.hello('probe');break;} catch {client.close();await new Promise(r=>setTimeout(r,20));} + } + worker=new AgentWorker(client,{workerId:'probe',pins:{workspace:[{surface:'repo',revision_id:'rev-a'}],streams:[]}}); + await worker.attach(); + const spec=JSON.parse(readFileSync(root+'/testdata/hn-monitor.spec.canonical.json','utf8')); + for(const step of spec.steps) if(step.id==='analyze-story')step.cli=root+'/testdata/preflight/analyze-story-stub-cli'; + const outcome=await client.eventSubmit(spec,{type:'hn.story_posted',payload:{id:42000042,type:'story'}}); + for(let i=0;i<100;i++){if((await client.runGet(outcome.run.run_id)).steps['analyze-story'].state==='done')break;await new Promise(r=>setTimeout(r,50));} + const {entries}=await client.journalRead(outcome.run.run_id,1,1000); + console.log(JSON.stringify(entries.filter(e=>e.entry_type==='step.completed'),null,2)); +}finally{await worker?.close();client?.close();daemon.kill('SIGTERM');await new Promise(r=>daemon.once('exit',r));rmSync(data,{recursive:true,force:true});} diff --git a/evidence/run-read-timeout/handshake-probe.mjs b/evidence/run-read-timeout/handshake-probe.mjs new file mode 100644 index 00000000..60241ad8 --- /dev/null +++ b/evidence/run-read-timeout/handshake-probe.mjs @@ -0,0 +1,25 @@ +import {mkdtempSync,readFileSync,rmSync} from 'node:fs'; +import {join} from 'node:path'; +import {tmpdir} from 'node:os'; +import {spawn} from 'node:child_process'; +const root='/home/daytona/.relayflow-v2-supervisor/durable/repository'; +const {JournalClient}=await import(root+'/packages/sdk/dist/journal-client.js'); +const {AgentWorker}=await import(root+'/packages/sdk/dist/worker.js'); +const {socketPathFor}=await import(root+'/packages/sdk/dist/daemon-connection.js'); +const data=mkdtempSync(join(tmpdir(),'rf-read-probe-')); +const daemon=spawn('/home/daytona/.relayflows-toolchain/target/2962130851/debug/relayflowd',['--data-dir',data,'serve'],{stdio:'ignore'}); +let client,worker; +try { + for(let i=0;i<100;i++) { + client=new JournalClient(socketPathFor(data)); + try {await client.connect();await client.hello('probe');break;} catch {client.close();await new Promise(r=>setTimeout(r,20));} + } + worker=new AgentWorker(client,{workerId:'probe',pins:{workspace:[{surface:'repo',revision_id:'rev-a'}],streams:[]}}); + await worker.attach(); + const spec=JSON.parse(readFileSync(root+'/testdata/hn-monitor.spec.canonical.json','utf8')); + for(const step of spec.steps) if(step.id==='analyze-story')step.cli=root+'/testdata/preflight/analyze-story-stub-cli'; + const outcome=await client.eventSubmit(spec,{type:'hn.story_posted',payload:{id:42000042,type:'story'}}); + for(let i=0;i<100;i++){if((await client.runGet(outcome.run.run_id)).steps['analyze-story'].state==='done')break;await new Promise(r=>setTimeout(r,50));} + const {entries}=await client.journalRead(outcome.run.run_id,1,1000); + console.log(JSON.stringify(entries.filter(e=>e.entry_type==='step.completed'),null,2)); +}finally{await worker?.close();client?.close();daemon.kill('SIGTERM');await new Promise(r=>daemon.once('exit',r));rmSync(data,{recursive:true,force:true});} diff --git a/evidence/run-read-timeout/live-failure-probe-baseline.log b/evidence/run-read-timeout/live-failure-probe-baseline.log new file mode 100644 index 00000000..947cc038 --- /dev/null +++ b/evidence/run-read-timeout/live-failure-probe-baseline.log @@ -0,0 +1,45 @@ +$ node /tmp/relayflow-read-probe-baseline.mjs +[ + { + "at_ms": 1791197583957, + "attempt": 1, + "entry_type": "step.completed", + "payload": { + "budget": { + "dollars": "0", + "dollars_unmetered": true, + "tokens_in": 0, + "tokens_out": 0 + }, + "completed_by": "probe", + "completionReason": "worker_error", + "disposition": "step_done", + "effects": [], + "end_pins": { + "streams": [], + "workspace": [] + }, + "input_hash": "44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a", + "next_attempt_at_ms": null, + "output": null, + "spend": { + "dollars": 0, + "dollars_unmetered": true, + "tokens_input": 0, + "tokens_output": 0, + "wallclock_ms": 888 + }, + "step_spec_hash": "12d9633edd0ffd882a0b1586fc347ce36c84d32656fe82f7e0b8d9d97cecc087", + "verification": { + "detail": "{\"artifacts\":[],\"exit_code\":null,\"stderr_tail\":\"CLI \\\"/home/daytona/.relayflow-v2-supervisor/durable/repository/testdata/preflight/analyze-story-stub-cli\\\" exited before completing the relayflows-agent-cli-v1 same-process handshake.\",\"stdout_tail\":\"\"}", + "gate": "execution", + "verdict": "fail" + } + }, + "run_id": "01M45V45PHWH45AD8128KRDAR7", + "segment_id": 1, + "seq": 7, + "step_id": "analyze-story" + } +] +exit=0 diff --git a/evidence/run-read-timeout/live-failure-probe.log b/evidence/run-read-timeout/live-failure-probe.log new file mode 100644 index 00000000..7d969c68 --- /dev/null +++ b/evidence/run-read-timeout/live-failure-probe.log @@ -0,0 +1,45 @@ +$ node /tmp/relayflow-read-probe.mjs +[ + { + "at_ms": 1791197554045, + "attempt": 1, + "entry_type": "step.completed", + "payload": { + "budget": { + "dollars": "0", + "dollars_unmetered": true, + "tokens_in": 0, + "tokens_out": 0 + }, + "completed_by": "probe", + "completionReason": "worker_error", + "disposition": "step_done", + "effects": [], + "end_pins": { + "streams": [], + "workspace": [] + }, + "input_hash": "44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a", + "next_attempt_at_ms": null, + "output": null, + "spend": { + "dollars": 0, + "dollars_unmetered": true, + "tokens_input": 0, + "tokens_output": 0, + "wallclock_ms": 723 + }, + "step_spec_hash": "12d9633edd0ffd882a0b1586fc347ce36c84d32656fe82f7e0b8d9d97cecc087", + "verification": { + "detail": "{\"artifacts\":[],\"exit_code\":null,\"stderr_tail\":\"CLI \\\"/home/daytona/.relayflow-v2-supervisor/durable/repository/testdata/preflight/analyze-story-stub-cli\\\" exited before completing the relayflows-agent-cli-v1 same-process handshake.\",\"stdout_tail\":\"\"}", + "gate": "execution", + "verdict": "fail" + } + }, + "run_id": "01M45V38MZYZ0F87JWZVAJBZ6F", + "segment_id": 1, + "seq": 7, + "step_id": "analyze-story" + } +] +exit=0 diff --git a/evidence/run-read-timeout/mutation-heartbeat.log b/evidence/run-read-timeout/mutation-heartbeat.log new file mode 100644 index 00000000..e0770eaa --- /dev/null +++ b/evidence/run-read-timeout/mutation-heartbeat.log @@ -0,0 +1,53 @@ +Source: packages/sdk/src/worker-lease.ts +Original SHA256: 44c13f9f61ac13cb8c0f872f4c22b1c212e020d05880ed080ea7205ba6007213 +Replaced: + if (error instanceof JournalRequestTimeoutError && error.verb === 'step.heartbeat') { + throw new WorkerLeaseLostError('renewal_expired', error.message, { cause: error }); + } + +With: + + +REVERTED +$ cd packages/sdk && npx vitest run tests/heartbeat-timeout.test.ts -t 'a heartbeat timeout' --maxWorkers=1 --minWorkers=1 + + RUN v2.1.9 /home/daytona/.relayflow-v2-supervisor/durable/repository/packages/sdk + + ❯ tests/heartbeat-timeout.test.ts (1 test | 1 failed) 20ms + × a heartbeat timeout is lease loss rather than a worker body failure 20ms + → expected JournalRequestTimeoutError: journal clien… { …(5) } to be an instance of WorkerLeaseLostError + +⎯⎯⎯⎯⎯⎯⎯ Failed Tests 1 ⎯⎯⎯⎯⎯⎯⎯ + + FAIL tests/heartbeat-timeout.test.ts > a heartbeat timeout is lease loss rather than a worker body failure +AssertionError: expected JournalRequestTimeoutError: journal clien… { …(5) } to be an instance of WorkerLeaseLostError + ❯ tests/heartbeat-timeout.test.ts:16:5 + 14| try { + 15| await client.connect(); + 16| await expect(withWorkerLease(client, { + | ^ + 17| run_id: 'run', step_id: 'step', attempt: 1, step_type: 'agent', … + 18| lease_id: 'lease', lease_deadline_ms: Date.now() + 30_000, lease… + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[1/1]⎯ + + Test Files 1 failed (1) + Tests 1 failed (1) + Start at 10:38:23 + Duration 509ms (transform 171ms, setup 14ms, collect 315ms, tests 20ms, environment 0ms, prepare 45ms) + +exit=1 + +RESTORED SHA256: 44c13f9f61ac13cb8c0f872f4c22b1c212e020d05880ed080ea7205ba6007213 +$ cd packages/sdk && npx vitest run tests/heartbeat-timeout.test.ts -t 'a heartbeat timeout' --maxWorkers=1 --minWorkers=1 + + RUN v2.1.9 /home/daytona/.relayflow-v2-supervisor/durable/repository/packages/sdk + + ✓ tests/heartbeat-timeout.test.ts (1 test) 15ms + + Test Files 1 passed (1) + Tests 1 passed (1) + Start at 10:38:24 + Duration 491ms (transform 169ms, setup 15ms, collect 300ms, tests 15ms, environment 0ms, prepare 43ms) + +exit=0 diff --git a/evidence/run-read-timeout/mutation-reader.log b/evidence/run-read-timeout/mutation-reader.log new file mode 100644 index 00000000..546fe785 --- /dev/null +++ b/evidence/run-read-timeout/mutation-reader.log @@ -0,0 +1,57 @@ +Source: packages/sdk/src/journal-client.ts +Original SHA256: 9a390f44f9a1375f0e13be91786f24e848bba7fa07b93b28b119c7ca1d24027c +Replaced: +(reader ?? this).requestOnce(verb, params, remaining) +With: +this.requestOnce(verb, params, remaining) + +REVERTED +$ cd packages/sdk && npx vitest run tests/journal-client-read-timeout.test.ts -t 'serves a bounded read' --maxWorkers=1 --minWorkers=1 + + RUN v2.1.9 /home/daytona/.relayflow-v2-supervisor/durable/repository/packages/sdk + + ❯ tests/journal-client-read-timeout.test.ts (13 tests | 1 failed | 12 skipped) 111ms + × serves a bounded read while an unbounded command is in flight 110ms + → expected true to be false // Object.is equality + +⎯⎯⎯⎯⎯⎯⎯ Failed Tests 1 ⎯⎯⎯⎯⎯⎯⎯ + + FAIL tests/journal-client-read-timeout.test.ts > serves a bounded read while an unbounded command is in flight +AssertionError: expected true to be false // Object.is equality + +- Expected ++ Received + +- false ++ true + + ❯ tests/journal-client-read-timeout.test.ts:36:23 + 34| await inFlight; + 35| expect(await client.runGet('run')).toMatchObject({ status: 'running'… + 36| expect(commandDone).toBe(false); + | ^ + 37| await command; + 38| }); + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[1/1]⎯ + + Test Files 1 failed (1) + Tests 1 failed | 12 skipped (13) + Start at 10:38:14 + Duration 588ms (transform 169ms, setup 15ms, collect 306ms, tests 111ms, environment 0ms, prepare 44ms) + +exit=1 + +RESTORED SHA256: 9a390f44f9a1375f0e13be91786f24e848bba7fa07b93b28b119c7ca1d24027c +$ cd packages/sdk && npx vitest run tests/journal-client-read-timeout.test.ts -t 'serves a bounded read' --maxWorkers=1 --minWorkers=1 + + RUN v2.1.9 /home/daytona/.relayflow-v2-supervisor/durable/repository/packages/sdk + + ✓ tests/journal-client-read-timeout.test.ts (13 tests | 12 skipped) 108ms + + Test Files 1 passed (1) + Tests 1 passed | 12 skipped (13) + Start at 10:38:15 + Duration 586ms (transform 171ms, setup 15ms, collect 307ms, tests 108ms, environment 0ms, prepare 42ms) + +exit=0 diff --git a/evidence/run-read-timeout/mutation-root-parking.log b/evidence/run-read-timeout/mutation-root-parking.log new file mode 100644 index 00000000..a8d7556f --- /dev/null +++ b/evidence/run-read-timeout/mutation-root-parking.log @@ -0,0 +1,70 @@ +Source: packages/sdk/src/authored-root.ts +Original SHA256: ef090c86e849b5b4d77700672b60f9a42979e6649e6207a800a1e7561ca6be1e +Replaced: + if ((error instanceof JournalRequestTimeoutError && (READ_ONLY_VERBS.has(error.verb) || error.verb === 'run.watch')) + || (error instanceof AuthoredFlowExecutionError && error.code === 'daemon_unresponsive')) { + const parked = new AuthoredFlowExecutionError('daemon_unresponsive', + `${error.message}. The run remains resumable. Continue with: ${resumeCommand(dispatch.run_id, options.dataDir, options.localAgentStream !== undefined)}.`); + parked.rootRunId = dispatch.run_id; + throw parked; + } + +With: + + +REVERTED +$ cd packages/sdk && npx vitest run tests/authored-root.test.ts -t 'leaves the root resumable after a read timeout' --maxWorkers=1 --minWorkers=1 + + RUN v2.1.9 /home/daytona/.relayflow-v2-supervisor/durable/repository/packages/sdk + + ❯ tests/authored-root.test.ts (25 tests | 1 failed | 24 skipped) 22ms + × durable authored root > leaves the root resumable after a read timeout without waiting for redispatch 21ms + → expected JournalRequestTimeoutError: journal clien… { …(5) } to match object { code: 'daemon_unresponsive', …(2) } +(6 matching properties omitted from actual) + +⎯⎯⎯⎯⎯⎯⎯ Failed Tests 1 ⎯⎯⎯⎯⎯⎯⎯ + + FAIL tests/authored-root.test.ts > durable authored root > leaves the root resumable after a read timeout without waiting for redispatch +AssertionError: expected JournalRequestTimeoutError: journal clien… { …(5) } to match object { code: 'daemon_unresponsive', …(2) } +(6 matching properties omitted from actual) + +- Expected ++ Received + +- Object { +- "code": "daemon_unresponsive", +- "message": StringContaining "flows resume", +- "rootRunId": "root-run", +- } ++ [JournalRequestTimeoutError: journal client: run.get timed out after 10ms] + + ❯ tests/authored-root.test.ts:411:5 + 409| const loaded = await fixture(false, 0, async () => { throw new Jou… + 410| const journal = new RootJournal(); + 411| await expect(executeDurableAuthoredFlow(loaded, journal as unknown… + | ^ + 412| { dataDir: '/unused', admissionKey: 'read-timeout' })).rejects.t… + 413| code: 'daemon_unresponsive', rootRunId: 'root-run', message: e… + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[1/1]⎯ + + Test Files 1 failed (1) + Tests 1 failed | 24 skipped (25) + Start at 10:38:25 + Duration 1.36s (transform 704ms, setup 18ms, collect 1.16s, tests 22ms, environment 0ms, prepare 46ms) + +exit=1 + +RESTORED SHA256: ef090c86e849b5b4d77700672b60f9a42979e6649e6207a800a1e7561ca6be1e +$ cd packages/sdk && npx vitest run tests/authored-root.test.ts -t 'leaves the root resumable after a read timeout' --maxWorkers=1 --minWorkers=1 + + RUN v2.1.9 /home/daytona/.relayflow-v2-supervisor/durable/repository/packages/sdk + + ✓ tests/authored-root.test.ts (25 tests | 24 skipped) 18ms + + Test Files 1 passed (1) + Tests 1 passed | 24 skipped (25) + Start at 10:38:27 + Duration 1.32s (transform 674ms, setup 15ms, collect 1.13s, tests 18ms, environment 0ms, prepare 45ms) + +exit=0 diff --git a/evidence/run-read-timeout/mutation-watch-cadence.log b/evidence/run-read-timeout/mutation-watch-cadence.log new file mode 100644 index 00000000..c10f7873 --- /dev/null +++ b/evidence/run-read-timeout/mutation-watch-cadence.log @@ -0,0 +1,51 @@ +Source: packages/sdk/src/cli/running-step.ts +Original SHA256: 0f8b6b9e160ebb047b5a83ad48ad41e2c201d5ba905c27c3a872acd3b9b288c2 +Replaced: +const LEASE_POLL_MS = 2_000; +With: +const LEASE_POLL_MS = 50; + +REVERTED +$ cd packages/sdk && npx vitest run tests/running-step-watch.test.ts -t 'uses pushes for completion' --maxWorkers=1 --minWorkers=1 + + RUN v2.1.9 /home/daytona/.relayflow-v2-supervisor/durable/repository/packages/sdk + + ❯ tests/running-step-watch.test.ts (2 tests | 1 failed | 1 skipped) 2110ms + × uses pushes for completion with lease-cadence reads and releases its watcher 2109ms + → expected 41 to be less than or equal to 1 + +⎯⎯⎯⎯⎯⎯⎯ Failed Tests 1 ⎯⎯⎯⎯⎯⎯⎯ + + FAIL tests/running-step-watch.test.ts > uses pushes for completion with lease-cadence reads and releases its watcher +AssertionError: expected 41 to be less than or equal to 1 + ❯ tests/running-step-watch.test.ts:37:19 + 35| await waitForRunningStep(client, 'run', { id: 'step', type: 'agent… + 36| expect(performance.now() - start).toBeGreaterThan(2000); + 37| expect(reads).toBeLessThanOrEqual(1); + | ^ + 38| await sleep(10); + 39| expect(watchClosed).toBe(true); + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[1/1]⎯ + + Test Files 1 failed (1) + Tests 1 failed | 1 skipped (2) + Start at 10:38:16 + Duration 2.60s (transform 179ms, setup 16ms, collect 313ms, tests 2.11s, environment 0ms, prepare 43ms) + +exit=1 + +RESTORED SHA256: 0f8b6b9e160ebb047b5a83ad48ad41e2c201d5ba905c27c3a872acd3b9b288c2 +$ cd packages/sdk && npx vitest run tests/running-step-watch.test.ts -t 'uses pushes for completion' --maxWorkers=1 --minWorkers=1 + + RUN v2.1.9 /home/daytona/.relayflow-v2-supervisor/durable/repository/packages/sdk + + ✓ tests/running-step-watch.test.ts (2 tests | 1 skipped) 2120ms + ✓ uses pushes for completion with lease-cadence reads and releases its watcher 2119ms + + Test Files 1 passed (1) + Tests 1 passed | 1 skipped (2) + Start at 10:38:20 + Duration 2.61s (transform 186ms, setup 15ms, collect 322ms, tests 2.12s, environment 0ms, prepare 42ms) + +exit=0 diff --git a/evidence/run-read-timeout/mutations.py b/evidence/run-read-timeout/mutations.py new file mode 100644 index 00000000..5bcea4a9 --- /dev/null +++ b/evidence/run-read-timeout/mutations.py @@ -0,0 +1,57 @@ +"""Reproduce the four reviewed-plan mutations; always restore source bytes.""" +from pathlib import Path +import hashlib +import shlex +import subprocess + +ROOT = Path(__file__).resolve().parents[2] +SDK = ROOT / 'packages/sdk' +CASES = [ + ('reader', 'packages/sdk/src/journal-client.ts', + '(reader ?? this).requestOnce(verb, params, remaining)', + 'this.requestOnce(verb, params, remaining)', + 'tests/journal-client-read-timeout.test.ts', 'serves a bounded read'), + ('watch-cadence', 'packages/sdk/src/cli/running-step.ts', + 'const LEASE_POLL_MS = 2_000;', 'const LEASE_POLL_MS = 50;', + 'tests/running-step-watch.test.ts', 'uses pushes for completion'), + ('heartbeat', 'packages/sdk/src/worker-lease.ts', + """ if (error instanceof JournalRequestTimeoutError && error.verb === 'step.heartbeat') { + throw new WorkerLeaseLostError('renewal_expired', error.message, { cause: error }); + } +""", '', 'tests/heartbeat-timeout.test.ts', 'a heartbeat timeout'), + ('root-parking', 'packages/sdk/src/authored-root.ts', + """ if ((error instanceof JournalRequestTimeoutError && (READ_ONLY_VERBS.has(error.verb) || error.verb === 'run.watch')) + || (error instanceof AuthoredFlowExecutionError && error.code === 'daemon_unresponsive')) { + const parked = new AuthoredFlowExecutionError('daemon_unresponsive', + `${error.message}. The run remains resumable. Continue with: ${resumeCommand(dispatch.run_id, options.dataDir, options.localAgentStream !== undefined)}.`); + parked.rootRunId = dispatch.run_id; + throw parked; + } +""", '', 'tests/authored-root.test.ts', 'leaves the root resumable after a read timeout'), +] + +for name, source, old, new, test, pattern in CASES: + path = ROOT / source + original = path.read_bytes() + assert original.count(old.encode()) == 1, (name, 'mutation must match once') + digest = hashlib.sha256(original).hexdigest() + command = ['npx', 'vitest', 'run', test, '-t', pattern, '--maxWorkers=1', '--minWorkers=1'] + log = ROOT / 'evidence/run-read-timeout' / f'mutation-{name}.log' + with log.open('w') as output: + output.write(f'Source: {source}\nOriginal SHA256: {digest}\nReplaced:\n{old}\nWith:\n{new}\n') + try: + path.write_bytes(original.replace(old.encode(), new.encode())) + output.write('\nREVERTED\n$ cd packages/sdk && ' + shlex.join(command) + '\n') + output.flush() + failed = subprocess.run(command, cwd=SDK, stdout=output, stderr=subprocess.STDOUT) + output.write(f'exit={failed.returncode}\n') + finally: + path.write_bytes(original) + assert path.read_bytes() == original + output.write(f'\nRESTORED SHA256: {hashlib.sha256(path.read_bytes()).hexdigest()}\n') + output.write('$ cd packages/sdk && ' + shlex.join(command) + '\n') + output.flush() + passed = subprocess.run(command, cwd=SDK, stdout=output, stderr=subprocess.STDOUT) + output.write(f'exit={passed.returncode}\n') + print(f'{name}: reverted exit={failed.returncode}; restored exit={passed.returncode}; {log.relative_to(ROOT)}', flush=True) + assert failed.returncode == 1 and passed.returncode == 0, name diff --git a/evidence/run-read-timeout/resume-live.log b/evidence/run-read-timeout/resume-live.log new file mode 100644 index 00000000..1fe792d9 --- /dev/null +++ b/evidence/run-read-timeout/resume-live.log @@ -0,0 +1,15 @@ +$ cd packages/sdk && npx vitest run tests/read-timeout-resume-live.test.ts +$ cd packages/sdk && npx vitest run tests/read-timeout-resume-live.test.ts + + RUN v2.1.9 /home/daytona/.relayflow-v2-supervisor/durable/repository/packages/sdk + + ✓ tests/read-timeout-resume-live.test.ts (1 test) 1512ms + ✓ parks an unreadable authored root and resumes without repeating its journaled effect 1511ms + + Test Files 1 passed (1) + Tests 1 passed (1) + Start at 10:49:12 + Duration 3.20s (transform 891ms, setup 54ms, collect 1.39s, tests 1.51s, environment 0ms, prepare 43ms) + +exit=0 +exit=0 diff --git a/evidence/run-read-timeout/typecheck-final.json b/evidence/run-read-timeout/typecheck-final.json new file mode 100644 index 00000000..2c0063d8 --- /dev/null +++ b/evidence/run-read-timeout/typecheck-final.json @@ -0,0 +1,10 @@ +{ + "command": "cd packages/sdk && npx tsc --noEmit && npx tsc -p tsconfig.tests.json", + "result": { + "chunk_id": "10aa44", + "wall_time_seconds": 0.000001521, + "exit_code": 0, + "original_token_count": 0, + "output": "" + } +} diff --git a/evidence/run-read-timeout/verification.md b/evidence/run-read-timeout/verification.md new file mode 100644 index 00000000..6d6f5b02 --- /dev/null +++ b/evidence/run-read-timeout/verification.md @@ -0,0 +1,336 @@ +Captured verification. Commands and output below are verbatim; the full-suite log is linked from summary.md. + +## final-focused.log + +```text +$ cd packages/sdk && RELAYFLOWD_BIN=/home/daytona/.relayflows-toolchain/target/2962130851/debug/relayflowd npx vitest run tests/journal-client-read-timeout.test.ts tests/journal-client.test.ts tests/journal-client-completion.test.ts tests/journal-client-subscriptions.test.ts tests/running-step-watch.test.ts tests/heartbeat-timeout.test.ts tests/run-daemon-unresponsive.test.ts tests/authored-root.test.ts tests/classify-outcome.test.ts tests/cli.test.ts tests/direct-run-worker-lease.test.ts tests/resume-worker-lease.test.ts tests/worker-lease.test.ts tests/worker-lease-lost.test.ts tests/worker-lease-sweep.test.ts tests/run-read-load-live.test.ts tests/worker-lease-lost-live.test.ts tests/flow-executor-chain.test.ts tests/agent-transcript-live.test.ts tests/human-live.test.ts --maxWorkers=1 --minWorkers=1 + + RUN v2.1.9 /home/daytona/.relayflow-v2-supervisor/durable/repository/packages/sdk + + ✓ tests/cli.test.ts (71 tests) 6026ms + ✓ flows check CLI > binds a checked relative wrapper to the flow directory for worker execution 566ms + ✓ flows check CLI > resolves a bare PATH-resolved claude with no declared model, in an isolated PATH 488ms + ✓ flows run/resume CLI over the journal protocol > follows a dispatched worker step instead of reporting a protocol error 2053ms + ✓ flows run/resume CLI over the journal protocol > follows a worker wait past a locally expired lease until the daemon settles it 2045ms +(node:44939) [FLOWS_ROOT_LEASE_LOST] Warning: authored root run_id=root-run attempt=1: lease_conflict: attempt has no active worker lease. Waiting for the kernel to retry it. +(Use `node --trace-warnings ...` to show where the warning was created) + ✓ tests/authored-root.test.ts (26 tests) 398ms + ✓ tests/journal-client.test.ts (17 tests) 89ms + ✓ tests/flow-executor-chain.test.ts (14 tests) 10315ms + ✓ flow executor LLM and output-binding chain > runs f.llm -> f.agent -> f.run with schema-verified journal output and the exact allowed model 1022ms + ✓ flow executor LLM and output-binding chain > runs a dollar-budgeted authored Claude agent with the same default used by preflight 700ms + ✓ flow executor LLM and output-binding chain > runs the exact authored flagship f.llm -> f.agent -> f.run path through the durable CLI root 1595ms + ✓ flow executor LLM and output-binding chain > resumes an interrupted durable authored root without replaying completed flagship effects 3382ms + ✓ flow executor LLM and output-binding chain > passes a declarative verified value through an agent into a deterministic artifact 779ms + ✓ flow executor LLM and output-binding chain > flows run consumes YAML bindings and resume reuses the original journal output 1068ms + ✓ tests/agent-transcript-live.test.ts (4 tests) 3394ms + ✓ the transcript digest through the built CLI, a real daemon and the local agent > preserves structured agent failure details and its completed root index 875ms + ✓ the transcript digest through the built CLI, a real daemon and the local agent > preserves structured llm failure details and its completed root index 818ms + ✓ the transcript digest through the built CLI, a real daemon and the local agent > journals the digest in trajectory_tail on a successful agent step and writes the file it points at 851ms + ✓ the transcript digest through the built CLI, a real daemon and the local agent > on a failed agent step, names the failure and the transcript in the terminal diagnostic, redacted 848ms + ✓ tests/classify-outcome.test.ts (11 tests) 7422ms + ✓ classifyOutcome > gives up and reports when a running run never becomes classifiable 2009ms + ✓ the remedy on a worker park > follows a step through a retry backoff longer than the unclassified bound 3005ms + ✓ the remedy on a worker park > follows a run.start outcome that is already running on a retried attempt 2001ms + ✓ tests/human-live.test.ts (3 tests) 7854ms + ✓ f.human against a real daemon > parks with the question, refuses wrong answers, records one, and resumes to success 4785ms + ✓ f.human against a real daemon > a "no" is a value the body branches on: declined, exit 0, no effect 1916ms + ✓ f.human against a real daemon > refuses to answer a run the daemon does not know 1152ms + ✓ tests/worker-lease.test.ts (7 tests) 20ms + ✓ tests/worker-lease-lost.test.ts (17 tests) 27ms + ✓ tests/journal-client-read-timeout.test.ts (13 tests) 1104ms + ✓ a recovered read timeout does not become an authored callback failure 368ms + ✓ tests/worker-lease-lost-live.test.ts (3 tests) 943ms + ✓ reports journal success after completion rejects with lease_conflict 327ms + ✓ reports journal success when a renewal rejects after completion landed 327ms + ✓ tests/run-read-load-live.test.ts (2 tests) 2821ms + ✓ completes a CPU-saturating deterministic flow with reads in flight and preserves its journal 2095ms + ✓ drains read and watch promises before an authored flow completes 725ms + ✓ tests/worker-lease-sweep.test.ts (4 tests) 8ms + ✓ tests/journal-client-completion.test.ts (6 tests) 102ms + ✓ tests/resume-worker-lease.test.ts (3 tests) 6ms + ✓ tests/direct-run-worker-lease.test.ts (3 tests) 10ms + ✓ tests/running-step-watch.test.ts (2 tests) 2122ms + ✓ uses pushes for completion with lease-cadence reads and releases its watcher 2118ms + ✓ tests/journal-client-subscriptions.test.ts (1 test) 8ms + ✓ tests/run-daemon-unresponsive.test.ts (2 tests) 4ms + ✓ tests/heartbeat-timeout.test.ts (1 test) 16ms + + Test Files 20 passed (20) + Tests 210 passed (210) + Start at 10:41:25 + Duration 54.29s (transform 1.45s, setup 97ms, collect 8.09s, tests 42.69s, environment 3ms, prepare 1.05s) + +exit=0 +``` + +## resume-live.log + +```text +$ cd packages/sdk && npx vitest run tests/read-timeout-resume-live.test.ts +$ cd packages/sdk && npx vitest run tests/read-timeout-resume-live.test.ts + + RUN v2.1.9 /home/daytona/.relayflow-v2-supervisor/durable/repository/packages/sdk + + ✓ tests/read-timeout-resume-live.test.ts (1 test) 1512ms + ✓ parks an unreadable authored root and resumes without repeating its journaled effect 1511ms + + Test Files 1 passed (1) + Tests 1 passed (1) + Start at 10:49:12 + Duration 3.20s (transform 891ms, setup 54ms, collect 1.39s, tests 1.51s, environment 0ms, prepare 43ms) + +exit=0 +exit=0 +``` + +## mutation-reader.log + +```text +Source: packages/sdk/src/journal-client.ts +Original SHA256: 9a390f44f9a1375f0e13be91786f24e848bba7fa07b93b28b119c7ca1d24027c +Replaced: +(reader ?? this).requestOnce(verb, params, remaining) +With: +this.requestOnce(verb, params, remaining) + +REVERTED +$ cd packages/sdk && npx vitest run tests/journal-client-read-timeout.test.ts -t 'serves a bounded read' --maxWorkers=1 --minWorkers=1 + + RUN v2.1.9 /home/daytona/.relayflow-v2-supervisor/durable/repository/packages/sdk + + ❯ tests/journal-client-read-timeout.test.ts (13 tests | 1 failed | 12 skipped) 111ms + × serves a bounded read while an unbounded command is in flight 110ms + → expected true to be false // Object.is equality + +⎯⎯⎯⎯⎯⎯⎯ Failed Tests 1 ⎯⎯⎯⎯⎯⎯⎯ + + FAIL tests/journal-client-read-timeout.test.ts > serves a bounded read while an unbounded command is in flight +AssertionError: expected true to be false // Object.is equality + +- Expected ++ Received + +- false ++ true + + ❯ tests/journal-client-read-timeout.test.ts:36:23 + 34| await inFlight; + 35| expect(await client.runGet('run')).toMatchObject({ status: 'running'… + 36| expect(commandDone).toBe(false); + | ^ + 37| await command; + 38| }); + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[1/1]⎯ + + Test Files 1 failed (1) + Tests 1 failed | 12 skipped (13) + Start at 10:38:14 + Duration 588ms (transform 169ms, setup 15ms, collect 306ms, tests 111ms, environment 0ms, prepare 44ms) + +exit=1 + +RESTORED SHA256: 9a390f44f9a1375f0e13be91786f24e848bba7fa07b93b28b119c7ca1d24027c +$ cd packages/sdk && npx vitest run tests/journal-client-read-timeout.test.ts -t 'serves a bounded read' --maxWorkers=1 --minWorkers=1 + + RUN v2.1.9 /home/daytona/.relayflow-v2-supervisor/durable/repository/packages/sdk + + ✓ tests/journal-client-read-timeout.test.ts (13 tests | 12 skipped) 108ms + + Test Files 1 passed (1) + Tests 1 passed | 12 skipped (13) + Start at 10:38:15 + Duration 586ms (transform 171ms, setup 15ms, collect 307ms, tests 108ms, environment 0ms, prepare 42ms) + +exit=0 +``` + +## mutation-watch-cadence.log + +```text +Source: packages/sdk/src/cli/running-step.ts +Original SHA256: 0f8b6b9e160ebb047b5a83ad48ad41e2c201d5ba905c27c3a872acd3b9b288c2 +Replaced: +const LEASE_POLL_MS = 2_000; +With: +const LEASE_POLL_MS = 50; + +REVERTED +$ cd packages/sdk && npx vitest run tests/running-step-watch.test.ts -t 'uses pushes for completion' --maxWorkers=1 --minWorkers=1 + + RUN v2.1.9 /home/daytona/.relayflow-v2-supervisor/durable/repository/packages/sdk + + ❯ tests/running-step-watch.test.ts (2 tests | 1 failed | 1 skipped) 2110ms + × uses pushes for completion with lease-cadence reads and releases its watcher 2109ms + → expected 41 to be less than or equal to 1 + +⎯⎯⎯⎯⎯⎯⎯ Failed Tests 1 ⎯⎯⎯⎯⎯⎯⎯ + + FAIL tests/running-step-watch.test.ts > uses pushes for completion with lease-cadence reads and releases its watcher +AssertionError: expected 41 to be less than or equal to 1 + ❯ tests/running-step-watch.test.ts:37:19 + 35| await waitForRunningStep(client, 'run', { id: 'step', type: 'agent… + 36| expect(performance.now() - start).toBeGreaterThan(2000); + 37| expect(reads).toBeLessThanOrEqual(1); + | ^ + 38| await sleep(10); + 39| expect(watchClosed).toBe(true); + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[1/1]⎯ + + Test Files 1 failed (1) + Tests 1 failed | 1 skipped (2) + Start at 10:38:16 + Duration 2.60s (transform 179ms, setup 16ms, collect 313ms, tests 2.11s, environment 0ms, prepare 43ms) + +exit=1 + +RESTORED SHA256: 0f8b6b9e160ebb047b5a83ad48ad41e2c201d5ba905c27c3a872acd3b9b288c2 +$ cd packages/sdk && npx vitest run tests/running-step-watch.test.ts -t 'uses pushes for completion' --maxWorkers=1 --minWorkers=1 + + RUN v2.1.9 /home/daytona/.relayflow-v2-supervisor/durable/repository/packages/sdk + + ✓ tests/running-step-watch.test.ts (2 tests | 1 skipped) 2120ms + ✓ uses pushes for completion with lease-cadence reads and releases its watcher 2119ms + + Test Files 1 passed (1) + Tests 1 passed | 1 skipped (2) + Start at 10:38:20 + Duration 2.61s (transform 186ms, setup 15ms, collect 322ms, tests 2.12s, environment 0ms, prepare 42ms) + +exit=0 +``` + +## mutation-heartbeat.log + +```text +Source: packages/sdk/src/worker-lease.ts +Original SHA256: 44c13f9f61ac13cb8c0f872f4c22b1c212e020d05880ed080ea7205ba6007213 +Replaced: + if (error instanceof JournalRequestTimeoutError && error.verb === 'step.heartbeat') { + throw new WorkerLeaseLostError('renewal_expired', error.message, { cause: error }); + } + +With: + + +REVERTED +$ cd packages/sdk && npx vitest run tests/heartbeat-timeout.test.ts -t 'a heartbeat timeout' --maxWorkers=1 --minWorkers=1 + + RUN v2.1.9 /home/daytona/.relayflow-v2-supervisor/durable/repository/packages/sdk + + ❯ tests/heartbeat-timeout.test.ts (1 test | 1 failed) 20ms + × a heartbeat timeout is lease loss rather than a worker body failure 20ms + → expected JournalRequestTimeoutError: journal clien… { …(5) } to be an instance of WorkerLeaseLostError + +⎯⎯⎯⎯⎯⎯⎯ Failed Tests 1 ⎯⎯⎯⎯⎯⎯⎯ + + FAIL tests/heartbeat-timeout.test.ts > a heartbeat timeout is lease loss rather than a worker body failure +AssertionError: expected JournalRequestTimeoutError: journal clien… { …(5) } to be an instance of WorkerLeaseLostError + ❯ tests/heartbeat-timeout.test.ts:16:5 + 14| try { + 15| await client.connect(); + 16| await expect(withWorkerLease(client, { + | ^ + 17| run_id: 'run', step_id: 'step', attempt: 1, step_type: 'agent', … + 18| lease_id: 'lease', lease_deadline_ms: Date.now() + 30_000, lease… + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[1/1]⎯ + + Test Files 1 failed (1) + Tests 1 failed (1) + Start at 10:38:23 + Duration 509ms (transform 171ms, setup 14ms, collect 315ms, tests 20ms, environment 0ms, prepare 45ms) + +exit=1 + +RESTORED SHA256: 44c13f9f61ac13cb8c0f872f4c22b1c212e020d05880ed080ea7205ba6007213 +$ cd packages/sdk && npx vitest run tests/heartbeat-timeout.test.ts -t 'a heartbeat timeout' --maxWorkers=1 --minWorkers=1 + + RUN v2.1.9 /home/daytona/.relayflow-v2-supervisor/durable/repository/packages/sdk + + ✓ tests/heartbeat-timeout.test.ts (1 test) 15ms + + Test Files 1 passed (1) + Tests 1 passed (1) + Start at 10:38:24 + Duration 491ms (transform 169ms, setup 15ms, collect 300ms, tests 15ms, environment 0ms, prepare 43ms) + +exit=0 +``` + +## mutation-root-parking.log + +```text +Source: packages/sdk/src/authored-root.ts +Original SHA256: ef090c86e849b5b4d77700672b60f9a42979e6649e6207a800a1e7561ca6be1e +Replaced: + if ((error instanceof JournalRequestTimeoutError && (READ_ONLY_VERBS.has(error.verb) || error.verb === 'run.watch')) + || (error instanceof AuthoredFlowExecutionError && error.code === 'daemon_unresponsive')) { + const parked = new AuthoredFlowExecutionError('daemon_unresponsive', + `${error.message}. The run remains resumable. Continue with: ${resumeCommand(dispatch.run_id, options.dataDir, options.localAgentStream !== undefined)}.`); + parked.rootRunId = dispatch.run_id; + throw parked; + } + +With: + + +REVERTED +$ cd packages/sdk && npx vitest run tests/authored-root.test.ts -t 'leaves the root resumable after a read timeout' --maxWorkers=1 --minWorkers=1 + + RUN v2.1.9 /home/daytona/.relayflow-v2-supervisor/durable/repository/packages/sdk + + ❯ tests/authored-root.test.ts (25 tests | 1 failed | 24 skipped) 22ms + × durable authored root > leaves the root resumable after a read timeout without waiting for redispatch 21ms + → expected JournalRequestTimeoutError: journal clien… { …(5) } to match object { code: 'daemon_unresponsive', …(2) } +(6 matching properties omitted from actual) + +⎯⎯⎯⎯⎯⎯⎯ Failed Tests 1 ⎯⎯⎯⎯⎯⎯⎯ + + FAIL tests/authored-root.test.ts > durable authored root > leaves the root resumable after a read timeout without waiting for redispatch +AssertionError: expected JournalRequestTimeoutError: journal clien… { …(5) } to match object { code: 'daemon_unresponsive', …(2) } +(6 matching properties omitted from actual) + +- Expected ++ Received + +- Object { +- "code": "daemon_unresponsive", +- "message": StringContaining "flows resume", +- "rootRunId": "root-run", +- } ++ [JournalRequestTimeoutError: journal client: run.get timed out after 10ms] + + ❯ tests/authored-root.test.ts:411:5 + 409| const loaded = await fixture(false, 0, async () => { throw new Jou… + 410| const journal = new RootJournal(); + 411| await expect(executeDurableAuthoredFlow(loaded, journal as unknown… + | ^ + 412| { dataDir: '/unused', admissionKey: 'read-timeout' })).rejects.t… + 413| code: 'daemon_unresponsive', rootRunId: 'root-run', message: e… + +⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[1/1]⎯ + + Test Files 1 failed (1) + Tests 1 failed | 24 skipped (25) + Start at 10:38:25 + Duration 1.36s (transform 704ms, setup 18ms, collect 1.16s, tests 22ms, environment 0ms, prepare 46ms) + +exit=1 + +RESTORED SHA256: ef090c86e849b5b4d77700672b60f9a42979e6649e6207a800a1e7561ca6be1e +$ cd packages/sdk && npx vitest run tests/authored-root.test.ts -t 'leaves the root resumable after a read timeout' --maxWorkers=1 --minWorkers=1 + + RUN v2.1.9 /home/daytona/.relayflow-v2-supervisor/durable/repository/packages/sdk + + ✓ tests/authored-root.test.ts (25 tests | 24 skipped) 18ms + + Test Files 1 passed (1) + Tests 1 passed | 24 skipped (25) + Start at 10:38:27 + Duration 1.32s (transform 674ms, setup 15ms, collect 1.13s, tests 18ms, environment 0ms, prepare 45ms) + +exit=0 +``` diff --git a/kernel/DAEMON-LIFECYCLE.md b/kernel/DAEMON-LIFECYCLE.md index 3d739fab..257db138 100644 --- a/kernel/DAEMON-LIFECYCLE.md +++ b/kernel/DAEMON-LIFECYCLE.md @@ -559,3 +559,15 @@ SDK (`npm test` in `packages/sdk`): Test 15 is the one that matters most and is the hardest to fake: it must spawn real processes against a real temp data dir, because the property under test is enforced by `flock(2)`, not by any code we could stub. + +## Reads during long-running commands + +The daemon handles frames sequentially per connection; `run.start`, +`run.resume`, and `step.complete` can drive deterministic commands before +replying. The SDK's flow execution clients therefore send read-only requests +on an unconditional lazy reader session, serialized and retried within a +bounded budget. Worker registrations, leases, writes, and watches retain their +own session ordering. Watch pushes come from the hub and can arrive while a +command is in flight. A read timeout is not a terminal run fact: the CLI probes +a fresh connection and reports a resumable interruption (see `docs/SURFACE.md` +§5). No daemon protocol or runtime release is required for this policy. diff --git a/packages/sdk/src/authored-flow-error.ts b/packages/sdk/src/authored-flow-error.ts index d43aeeb9..e2ec5828 100644 --- a/packages/sdk/src/authored-flow-error.ts +++ b/packages/sdk/src/authored-flow-error.ts @@ -35,6 +35,7 @@ export type AuthoredFlowExecutionErrorCode = * re-dispatch the root to it. The run is not failed: completed steps are * journaled and `flows resume ` continues it. */ + | 'daemon_unresponsive' | 'root_lease_lost' | 'unsupported_completion' | 'unsupported_gate' diff --git a/packages/sdk/src/authored-node-entry.ts b/packages/sdk/src/authored-node-entry.ts index ac6f6fda..b4f58523 100644 --- a/packages/sdk/src/authored-node-entry.ts +++ b/packages/sdk/src/authored-node-entry.ts @@ -1,3 +1,4 @@ +import { FLOW_READ_BUDGET_MS, JournalRequestTimeoutError, READ_ONLY_VERBS } from './journal-read-policy.js'; import { Worker } from 'node:worker_threads'; import { createHash, createHmac } from 'node:crypto'; import { readFileSync, writeSync } from 'node:fs'; @@ -74,7 +75,7 @@ try { const loaded = await loadPinnedAuthoredSource(request.metadata, true); if (request.localAgentStream !== request.metadata.localAgentStream) throw new Error('authored root local agent surface mismatch'); if (request.workerCapacity !== undefined && !isAgentCapacity(request.workerCapacity)) throw new Error('invalid authored worker capacity'); - client = new JournalClient(request.socketPath); + client = new JournalClient(request.socketPath, { readBudgetMs: FLOW_READ_BUDGET_MS }); await client.connect(); await client.hello('flows-authored-node'); const result = await executeAuthoredFlow(loaded.handle, client, request.metadata.inputPresent ? request.metadata.input : undefined, { @@ -99,6 +100,10 @@ try { // `parkCause` travels for the same reason: it is the difference between // "attach a worker" and "a human has to recover this", and this process is // the only one that saw the child's classification. + // Preserve a read interruption across the Node/Bun error frame so the + // lease-owning parent can leave the root resumable. + ...(error instanceof JournalRequestTimeoutError && (READ_ONLY_VERBS.has(error.verb) || error.verb === 'run.watch') + ? { code: 'daemon_unresponsive' } : {}), ...(error instanceof AuthoredFlowExecutionError ? { code: error.code, completionReason: error.completionReason, runId: error.runId, details: error.details, diff --git a/packages/sdk/src/authored-root.ts b/packages/sdk/src/authored-root.ts index 1929fd38..97470332 100644 --- a/packages/sdk/src/authored-root.ts +++ b/packages/sdk/src/authored-root.ts @@ -1,3 +1,4 @@ +import { JournalRequestTimeoutError, READ_ONLY_VERBS } from './journal-read-policy.js'; import { loadPinnedAuthoredSource } from './authored-source-authority.js'; import { assertAuthoredRuntimeAvailable, runAuthoredInNode } from './authored-node-runner.js'; import { createHash, randomUUID } from 'node:crypto'; @@ -368,6 +369,13 @@ async function driveRoot( // A lost lease is owned by the kernel's retry path, not by this attempt's // terminalization or suspension handling. if (isLeaseLost(error)) throw error; + if ((error instanceof JournalRequestTimeoutError && (READ_ONLY_VERBS.has(error.verb) || error.verb === 'run.watch')) + || (error instanceof AuthoredFlowExecutionError && error.code === 'daemon_unresponsive')) { + const parked = new AuthoredFlowExecutionError('daemon_unresponsive', + `${error.message}. The run remains resumable. Continue with: ${resumeCommand(dispatch.run_id, options.dataDir, options.localAgentStream !== undefined)}.`); + parked.rootRunId = dispatch.run_id; + throw parked; + } if (error instanceof AuthoredFlowExecutionError && error.code === 'subscription_suspended' && error.suspension !== undefined) { diff --git a/packages/sdk/src/cli/direct-run.ts b/packages/sdk/src/cli/direct-run.ts index 90457370..7109ff59 100644 --- a/packages/sdk/src/cli/direct-run.ts +++ b/packages/sdk/src/cli/direct-run.ts @@ -1,3 +1,5 @@ +import { daemonUnresponsiveReport, isReadInterruption } from './journal-timeout.js'; +import { FLOW_READ_BUDGET_MS } from '../journal-read-policy.js'; import { onWorkerFailure } from '../worker-lease.js'; import { McpStepError } from '../authored-mcp.js'; import { randomUUID } from 'node:crypto'; @@ -67,7 +69,7 @@ export async function runDirectFlow( } const socketPath = socketFor(dataDir); const base: RunReport = { ...emptyReport('run'), path }; - const client = new JournalClient(socketPath); + const client = new JournalClient(socketPath, { readBudgetMs: FLOW_READ_BUDGET_MS }); const connected = await connect(client, 'run', dataDir, base, options); if (connected !== undefined) return connected; @@ -212,6 +214,7 @@ export async function runDirectFlow( if (error instanceof AuthoredFlowExecutionError && (error.code === 'step_failed' || error.code === 'gate_failed')) { return authoredStepFailure('run', base, socketPath, error); } + if (isReadInterruption(error)) return daemonUnresponsiveReport('run', base, socketPath, error, base.rootRunId, options, dataDir); if (error instanceof AuthoredFlowExecutionError && error.code === 'root_lease_lost') { return rootLeaseLostReport('run', base, socketPath, error); } diff --git a/packages/sdk/src/cli/journal-timeout.ts b/packages/sdk/src/cli/journal-timeout.ts new file mode 100644 index 00000000..8edce8af --- /dev/null +++ b/packages/sdk/src/cli/journal-timeout.ts @@ -0,0 +1,30 @@ +import { probeSocket } from '../daemon-connection.js'; +import { AuthoredFlowExecutionError } from '../authored-flow-error.js'; +import { resumeCommand } from '../authored-human.js'; +import { JournalRequestTimeoutError, READ_ONLY_VERBS } from '../journal-read-policy.js'; +import type { CheckReport } from './check.js'; +import type { RunExecution, RunReport, RunLifecycleOptions, RunCommand } from './run.js'; + +export function isReadInterruption(error: unknown): boolean { + return (error instanceof JournalRequestTimeoutError + && (READ_ONLY_VERBS.has(error.verb) || error.verb === 'run.watch')) + || (error instanceof AuthoredFlowExecutionError && error.code === 'daemon_unresponsive'); +} + +export async function daemonUnresponsiveReport(command: RunCommand, + base: CheckReport | RunReport, socketPath: string, error: unknown, fallbackRunId?: string, + options: RunLifecycleOptions = {}, dataDir?: string): Promise { + const runId = error instanceof AuthoredFlowExecutionError ? error.rootRunId ?? fallbackRunId : fallbackRunId; + const reachable = (await probeSocket(socketPath)).reachable; + const kind = reachable ? 'daemon_unresponsive' : 'daemon_unreachable'; + const detail = reachable ? 'relayflowd answers a fresh connection but the run read timed out; CPU load may be delaying it.' + : 'relayflowd did not answer a fresh connection; it may be unreachable or delayed by CPU load.'; + return { exitCode: 1, report: { + ...base, command, ok: false, socketPath, status: 'running', + ...(runId === undefined ? {} : { runId, rootRunId: runId }), + diagnostics: [...base.diagnostics, { severity: 'failure', kind, + message: `${detail} ${error instanceof Error ? error.message : String(error)}` + + (runId === undefined ? '' : ` The run remains resumable. Continue with: ${resumeCommand(runId, dataDir ?? options.dataDir, options.localAgent === true)}.`), + }], + } }; +} diff --git a/packages/sdk/src/cli/run.ts b/packages/sdk/src/cli/run.ts index 5ec49286..abc8d54f 100644 --- a/packages/sdk/src/cli/run.ts +++ b/packages/sdk/src/cli/run.ts @@ -1,3 +1,7 @@ +import { waitForRunningStep, type RunningStep } from './running-step.js'; +export { LEASE_SWEEP_GRACE_MS } from './running-step.js'; +import { daemonUnresponsiveReport, isReadInterruption } from './journal-timeout.js'; +import { FLOW_READ_BUDGET_MS } from '../journal-read-policy.js'; import { onWorkerFailure } from '../worker-lease.js'; import { communicationInstruction } from '../communication/spec.js'; import { checkCommunicationEnvironment, CommunicationEnvironmentError } from '../communication/preflight.js'; @@ -203,7 +207,7 @@ async function executeCheckedFlow( catch (error) { return { exitCode: 2, report: { ...base, diagnostics: [...base.diagnostics, { severity: 'refusal', kind: 'probe_failed', message: error instanceof Error ? error.message : 'Communication environment could not be checked.' }] } }; } } - const client = new JournalClient(socketPath); + const client = new JournalClient(socketPath, { readBudgetMs: FLOW_READ_BUDGET_MS }); const connected = await connect(client, 'run', dataDir, base, options); if (connected !== undefined) return connected; @@ -225,6 +229,7 @@ async function executeCheckedFlow( } if (options.onJournalEntry !== undefined) client.on('entry', options.onJournalEntry); const outcome = await startWatched(client, spec, options); + base.runId = outcome.run_id; // `run.start { watch: true }` is an event stream, not the source of the // root identity. A short deterministic run can finish before its first // watched entry is delivered (and an older daemon may refuse `watch`), so @@ -239,6 +244,7 @@ async function executeCheckedFlow( } return execution; } catch (error) { + if (isReadInterruption(error)) return daemonUnresponsiveReport('run', base, socketPath, error, base.runId, options, dataDir); if (error instanceof CommunicationEnvironmentError) return { exitCode: 2, report: { ...base, diagnostics: [...base.diagnostics, { severity: 'refusal', kind: 'probe_failed', message: error.message }] } }; if (error instanceof JournalProtocolError && ( @@ -289,7 +295,7 @@ export async function resumeFlow( const agentEnvironment = localAgentEnvironment(); const socketPath = socketFor(dataDir); const base = emptyReport('resume'); - const client = new JournalClient(socketPath); + const client = new JournalClient(socketPath, { readBudgetMs: FLOW_READ_BUDGET_MS }); const connected = await connect(client, 'resume', dataDir, base, options); if (connected !== undefined) return connected; @@ -408,6 +414,7 @@ export async function resumeFlow( if (error instanceof AuthoredFlowExecutionError && (error.code === 'step_failed' || error.code === 'gate_failed')) { return authoredStepFailure('resume', base, socketPath, error, runId); } + if (isReadInterruption(error)) return daemonUnresponsiveReport('resume', base, socketPath, error, runId, options, dataDir); if (error instanceof AuthoredFlowExecutionError && error.code === 'root_lease_lost') { return rootLeaseLostReport('resume', base, socketPath, error, runId); } @@ -822,6 +829,19 @@ export async function connect( /// needs a stubbed client rather than a real run -- the integration test that /// found it reproduced the bug roughly 1 time in 12. export async function classifyOutcome( + client: JournalClient, command: RunCommand, outcome: RunOutcome, + base: CheckReport | RunReport, socketPath: string, options: RunLifecycleOptions, +): Promise { + try { return await classifyOutcomeInner(client, command, outcome, base, socketPath, options); } + catch (error) { + // Authored child calls have no CLI socket path: let their durable root + // handle the interruption, rather than turn it into a child step failure. + if (socketPath !== '' && isReadInterruption(error)) return daemonUnresponsiveReport(command, base, socketPath, error, outcome.run_id, options); + throw error; + } +} + +async function classifyOutcomeInner( client: JournalClient, command: RunCommand, outcome: RunOutcome, @@ -1021,10 +1041,6 @@ interface OutOfBandInspection { backoffStep?: ParkedStep; } -interface RunningStep extends ParkedStep { - leaseDeadlineMs: number; -} - // Bound on re-polling a run that reports `running` with no identifiable step. // 40 x 50ms = 2s, far longer than the sub-second window observed in #179, and // short enough that a genuinely stuck run still reports rather than hangs. @@ -1071,63 +1087,7 @@ async function inspectOutOfBandStep( }; } -// Match kernel/relayflowd/src/server/client.rs: allow the lease sweep to dispatch a retry. -export const LEASE_SWEEP_GRACE_MS = 5_000; -// How often to re-ask the daemon about a step whose lease looks expired here. const EXPIRED_LEASE_POLL_MS = 250; -// A deadline unchanged for a whole lease (relayflowd LEASE_RENEWAL_MS, 30s) -// plus the sweep grace was renewed by nobody: the attempt is dead and the -// daemon never swept it. Fail rather than hang. -const STALE_LEASE_MS = 30_000 + LEASE_SWEEP_GRACE_MS; - -async function waitForRunningStep( - client: JournalClient, - runId: string, - runningStep: RunningStep, - options: RunLifecycleOptions, -): Promise { - let leaseDeadlineMs = runningStep.leaseDeadlineMs; - if (!Number.isFinite(leaseDeadlineMs)) { - throw new Error(`running step "${runningStep.id}" omitted lease_deadline_ms`); - } - options.onWait?.({ - runId, - stepId: runningStep.id, - stepType: runningStep.type, - leaseDeadlineMs, - }); - // When the lease deadline last changed, on this process's monotonic clock. - // A live worker renews every ~10s, moving the deadline; a dead lease the - // daemon has not swept keeps it. That -- not a comparison of the daemon's - // deadline with our wall clock -- is what separates the two under skew. - let deadlineSeenAt = performance.now(); - while (true) { - throwIfCanceled(options.signal, runningStep.id); - const remainingMs = leaseDeadlineMs + LEASE_SWEEP_GRACE_MS - Date.now(); - if (performance.now() - deadlineSeenAt > STALE_LEASE_MS) { - throw new Error( - `worker lease for step "${runningStep.id}" expired at ${leaseDeadlineMs} without completion`, - ); - } - // Past the deadline by THIS clock alone is not an expiry: the daemon may - // still hold the lease (skew) or be sweeping it into a retry. Keep - // following the step, at a slower poll, until the daemon says otherwise. - await delay(remainingMs <= 0 ? EXPIRED_LEASE_POLL_MS : Math.min(50, remainingMs), options.signal); - const snapshot = await client.runGet(runId); - const step = snapshot.steps[runningStep.id]; - if (step?.state !== 'running') return; - if (step.lease_deadline_ms !== undefined && step.lease_deadline_ms !== leaseDeadlineMs) { - leaseDeadlineMs = step.lease_deadline_ms; - deadlineSeenAt = performance.now(); - options.onWait?.({ - runId, - stepId: runningStep.id, - stepType: runningStep.type, - leaseDeadlineMs, - }); - } - } -} export function protocolFailure( command: RunCommand, diff --git a/packages/sdk/src/cli/running-step.ts b/packages/sdk/src/cli/running-step.ts new file mode 100644 index 00000000..7f666d91 --- /dev/null +++ b/packages/sdk/src/cli/running-step.ts @@ -0,0 +1,90 @@ +import { JournalProtocolError, type JournalClient } from '../journal-client.js'; +import { AuthoredFlowExecutionError } from '../authored-flow-error.js'; +import type { JournalEvent } from '../journal-reader.js'; +import type { RunLifecycleOptions, ParkedStep } from './run.js'; + +export const LEASE_SWEEP_GRACE_MS = 5_000; +const LEASE_POLL_MS = 2_000; +const STALE_LEASE_MS = 30_000 + LEASE_SWEEP_GRACE_MS; +export interface RunningStep extends ParkedStep { leaseDeadlineMs: number } + +/** Watch completion; snapshots only refresh the non-journaled live lease deadline. */ +export async function waitForRunningStep(client: JournalClient, runId: string, + runningStep: RunningStep, options: RunLifecycleOptions): Promise { + let leaseDeadlineMs = runningStep.leaseDeadlineMs; + if (!Number.isFinite(leaseDeadlineMs)) { + throw new Error(`running step "${runningStep.id}" omitted lease_deadline_ms`); + } + const notify = () => options.onWait?.({ runId, stepId: runningStep.id, + stepType: runningStep.type, leaseDeadlineMs }); + notify(); + let deadlineSeenAt = performance.now(); + // The protocol has no unwatch. Closing this scoped, non-worker session removes + // the server watcher, even on cancellation or a read/registration failure. + const watch = client.createPeer(); + let settled = false; + let ready = false; + let finish: () => void = () => {}; + let cancel: () => void = () => {}; + const completed = new Promise((resolve, reject) => { + finish = resolve; + cancel = () => reject(new Error(`worker wait for step "${runningStep.id}" was canceled`)); + }); + // The subscription handshake itself may be pending when cancellation arrives. + void completed.catch(() => {}); + const onEntry = (entry: JournalEvent) => { + if (entry.run_id !== runId) return; + if (entry.entry_type === 'run.completed') settled = true; + else if (entry.step_id === runningStep.id) { + if (entry.entry_type === 'step.attempt.started') settled = false; + if (entry.entry_type === 'step.completed' || entry.entry_type === 'wait.human') settled = true; + } + // Fold the entire replay before deciding: an older attempt may have failed. + if (ready && settled) finish(); + }; + watch.on('entry', onEntry); + options.signal?.addEventListener('abort', cancel, { once: true }); + if (options.signal?.aborted) cancel(); + try { + await Promise.race([watch.connect().then(() => watch.hello('flows-step-watch')).catch(error => { + if (error instanceof JournalProtocolError) throw error; + throw new AuthoredFlowExecutionError('daemon_unresponsive', + `could not establish the completion watch: ${error instanceof Error ? error.message : String(error)}`); + }).then(() => watch.runWatch(runId)), completed]); + ready = true; + if (settled) return; + while (true) { + let timer: ReturnType | undefined; + let releaseTimer: () => void = () => {}; + try { + const elapsed = await Promise.race([ + new Promise(resolve => { releaseTimer = () => resolve(false); timer = setTimeout(() => resolve(true), LEASE_POLL_MS); }), + completed.then(() => false), + ]); + if (!elapsed) return; + } finally { clearTimeout(timer); releaseTimer(); } + // Drain the read before leaving the authored promise scope. A raced but + // unresolved read would look like unawaited derived work to that scope. + const snapshot = await client.runGet(runId).catch(error => { + if (settled) return undefined; + throw error; + }); + if (settled || snapshot === undefined || snapshot.steps[runningStep.id]?.state !== 'running') return; + const deadline = snapshot.steps[runningStep.id]?.lease_deadline_ms; + if (deadline !== undefined && deadline !== leaseDeadlineMs) { + leaseDeadlineMs = deadline; + deadlineSeenAt = performance.now(); + notify(); + } else if (performance.now() - deadlineSeenAt > STALE_LEASE_MS) { + // A stalled sweep/renewal is not evidence that the body itself failed. + throw new AuthoredFlowExecutionError('daemon_unresponsive', + `worker lease for step "${runningStep.id}" expired at ${leaseDeadlineMs} without completion; relayflowd's lease sweep may be delayed by CPU load`); + } + } + } finally { + finish(); + options.signal?.removeEventListener('abort', cancel); + watch.off('entry', onEntry); + watch.close(); + } +} diff --git a/packages/sdk/src/failure-kinds.ts b/packages/sdk/src/failure-kinds.ts index 7a05c98a..09cc78fd 100644 --- a/packages/sdk/src/failure-kinds.ts +++ b/packages/sdk/src/failure-kinds.ts @@ -149,6 +149,7 @@ export const RUN_FAILURE_KINDS = [ * reach this process. The run is NOT failed (`status: running`): its * completed steps are journaled and `flows resume ` continues it. */ + 'daemon_unresponsive', 'root_lease_lost', ] as const; diff --git a/packages/sdk/src/index.ts b/packages/sdk/src/index.ts index 50909fa8..d69f03a0 100644 --- a/packages/sdk/src/index.ts +++ b/packages/sdk/src/index.ts @@ -217,7 +217,7 @@ export type { } from './protocol.js'; export { JOURNAL_WRITE_FAILED, PROTOCOL_VERSION } from './protocol.js'; -export { JournalClient, type JournalClientOptions } from './journal-client.js'; +export { JournalClient, JournalRequestTimeoutError, type JournalClientOptions } from './journal-client.js'; export { AgentWorker, type AgentWorkerOptions } from './worker.js'; export { diff --git a/packages/sdk/src/journal-client.ts b/packages/sdk/src/journal-client.ts index 9fe998ee..112a9bde 100644 --- a/packages/sdk/src/journal-client.ts +++ b/packages/sdk/src/journal-client.ts @@ -9,6 +9,8 @@ // stream plumbing. The client's framing and failure behavior is covered by a // loopback double in tests. +import { JournalReadPolicy, JournalRequestTimeoutError, READ_ONLY_VERBS } from './journal-read-policy.js'; +export { JournalRequestTimeoutError } from './journal-read-policy.js'; import { EventEmitter } from 'node:events'; export { walkJournal, JournalReadError, type JournalEvent, type JournalReadFailure } from './journal-reader.js'; import { randomUUID } from 'node:crypto'; @@ -27,6 +29,8 @@ import { import type { KernelRunSpec, StepType } from './spec.js'; export interface JournalClientOptions { + /** Opt-in total budget for read-only requests; interactive clients stay single-shot. */ + readBudgetMs?: number; /** Override the timeout for bounded protocol requests (ms). Default 30000. */ requestTimeoutMs?: number; /** @@ -61,9 +65,14 @@ export class JournalProtocolError extends Error { } export class JournalClient extends EventEmitter { + private readonly reads = new JournalReadPolicy(); + private reader: JournalClient | undefined; + private readerReady: Promise | undefined; + private readonly readBudgetMs?: number; /** Additive request capabilities the daemon advertised at `hello`; none until then. */ private features: ReadonlySet = new Set(); private socket: Socket | null = null; + private connectingSocket: Socket | undefined; /** Why the connection ended, so a later "not connected" names its cause rather than hiding it. */ private disconnectCause: Error | undefined; private buffer = ''; @@ -76,6 +85,7 @@ export class JournalClient extends EventEmitter { options: JournalClientOptions = {}, ) { super(); + this.readBudgetMs = options.readBudgetMs; this.requestTimeoutMs = options.requestTimeoutMs ?? 30_000; this.connectTimeoutMs = options.connectTimeoutMs ?? 2_000; } @@ -92,6 +102,7 @@ export class JournalClient extends EventEmitter { return new Promise((resolve, reject) => { if (this.socket) return resolve(); const socket = createConnection({ path: this.socketPath }); + this.connectingSocket = socket; const timer = setTimeout(() => { socket.removeAllListeners(); socket.destroy(); @@ -112,10 +123,13 @@ export class JournalClient extends EventEmitter { socket.on('data', (chunk) => this.onData(chunk)); socket.on('close', () => { const closed = new Error('journal client: connection closed'); + this.reads.close(); + this.reader?.close(closed); this.disconnectCause ??= closed; this.failAll(closed); }); this.disconnectCause = undefined; + this.connectingSocket = undefined; this.socket = socket; resolve(); }); @@ -131,6 +145,9 @@ export class JournalClient extends EventEmitter { const closed = cause === undefined ? new Error('journal client: closed by caller') : new Error(`journal client: closed after ${cause instanceof Error ? cause.message : String(cause)}`, { cause }); + this.reads.close(); + this.reader?.close(cause); + this.connectingSocket?.destroy(closed); this.disconnectCause ??= closed; this.failAll(closed); this.socket?.destroy(); @@ -181,7 +198,31 @@ export class JournalClient extends EventEmitter { this.pending.clear(); } - private request( + private async readSession(): Promise { + if (this.readerReady === undefined) { + const reader = this.createPeer(); + this.reader = reader; + this.readerReady = reader.connect().then(() => reader.hello('flows-reader')).then(() => reader) + .catch(() => { reader.close(); return undefined; }); + } + return this.readerReady; + } + + private request(verb: V, params: VerbContract[V]['params'], + timeoutMs: number | null = this.requestTimeoutMs): Promise { + if (this.socket && !this.socket.destroyed && this.readBudgetMs !== undefined && timeoutMs !== null && READ_ONLY_VERBS.has(verb)) { + return this.reads.read(verb, timeoutMs, this.readBudgetMs, async bound => { + const started = performance.now(); + const reader = await this.readSession(); + const remaining = bound - (performance.now() - started); + if (remaining <= 0) throw new JournalRequestTimeoutError(verb, bound); + return (reader ?? this).requestOnce(verb, params, remaining); + }); + } + return this.requestOnce(verb, params, timeoutMs); + } + + private requestOnce( verb: V, params: VerbContract[V]['params'], timeoutMs: number | null = this.requestTimeoutMs, @@ -196,10 +237,11 @@ export class JournalClient extends EventEmitter { return; } const id = randomUUID(); + const started = performance.now(); const frame: Request = { id, verb: verb as string, params }; const timer = timeoutMs === null ? undefined : setTimeout(() => { this.pending.delete(id); - reject(new Error(`journal client: ${verb} timed out after ${timeoutMs}ms`)); + reject(new JournalRequestTimeoutError(verb, timeoutMs, 1, performance.now() - started)); }, timeoutMs); this.pending.set(id, { resolve: resolve as (v: unknown) => void, reject, timer }); this.socket.write(JSON.stringify(frame) + '\n', (err) => { diff --git a/packages/sdk/src/journal-read-policy.ts b/packages/sdk/src/journal-read-policy.ts new file mode 100644 index 00000000..76f666a7 --- /dev/null +++ b/packages/sdk/src/journal-read-policy.ts @@ -0,0 +1,79 @@ +import { setMaxListeners } from 'node:events'; +import { setTimeout as sleep } from 'node:timers/promises'; + +export const READ_ONLY_VERBS = new Set(['run.get', 'journal.read', 'stream.read', 'subscription.inspect']); +export const FLOW_READ_BUDGET_MS = 300_000; + +export class JournalRequestTimeoutError extends Error { + constructor( + readonly verb: string, + readonly timeoutMs: number, + readonly attempts = 1, + readonly elapsedMs = timeoutMs, + readonly readBudgetMs?: number, + ) { + super(readBudgetMs === undefined + ? `journal client: ${verb} timed out after ${timeoutMs}ms` + : `journal client: ${verb} timed out after ${attempts} attempts in ${Math.round(elapsedMs)}ms (read budget ${readBudgetMs}ms); relayflowd may be delayed by CPU load`); + this.name = 'JournalRequestTimeoutError'; + } +} + +/** Serial admission bounds read amplification, including retries, per body client. */ +export class JournalReadPolicy { + private tail: Promise = Promise.resolve(); + private readonly closed = new AbortController(); + + constructor() { + // Concurrent children share this cancellation signal; each waiter removes its listener. + setMaxListeners(0, this.closed.signal); + } + + close(): void { this.closed.abort(new Error('journal client: closed by caller')); } + + async read(verb: string, timeoutMs: number, budgetMs: number, + request: (timeoutMs: number) => Promise): Promise { + const started = performance.now(); + const deadline = started + budgetMs; + let attempts = 0; + const exhausted = () => new JournalRequestTimeoutError(verb, timeoutMs, attempts, + performance.now() - started, budgetMs); + const signal = this.closed.signal; + let timer: ReturnType | undefined; + let onClose: () => void = () => {}; + let deliver: (value: T) => void = () => {}; + let fail: (error: unknown) => void = () => {}; + const response = new Promise((resolve, reject) => { + deliver = resolve; + fail = reject; + timer = setTimeout(() => reject(exhausted()), budgetMs); + onClose = () => reject(signal.reason); + signal.addEventListener('abort', onClose, { once: true }); + if (signal.aborted) onClose(); + }); + const job = this.tail.then(async () => { + while (true) { + signal.throwIfAborted(); + const remaining = deadline - performance.now(); + if (remaining <= 0) throw exhausted(); + attempts += 1; + try { + return await request(Math.min(remaining, timeoutMs * 2 ** Math.max(0, attempts - 2))); + } catch (error) { + if (!(error instanceof JournalRequestTimeoutError)) throw error; + const left = deadline - performance.now(); + if (left <= 0) throw exhausted(); + // Jitter plus escalating bounds keep a slow daemon from collecting a retry storm. + await sleep(Math.min(left, Math.min(1000, timeoutMs / 4) * (0.5 + Math.random())), undefined, { signal }); + } + } + }); + this.tail = job.catch(() => {}); + void job.then(deliver, fail); + try { return await response; } + finally { + clearTimeout(timer); + signal.removeEventListener('abort', onClose); + } + } +} diff --git a/packages/sdk/src/worker-lease.ts b/packages/sdk/src/worker-lease.ts index 8396f165..9475b542 100644 --- a/packages/sdk/src/worker-lease.ts +++ b/packages/sdk/src/worker-lease.ts @@ -1,4 +1,4 @@ -import { JournalProtocolError, type JournalClient } from './journal-client.js'; +import { JournalProtocolError, JournalRequestTimeoutError, type JournalClient } from './journal-client.js'; import type { StepDispatchEvent } from './protocol.js'; export class WorkerLeaseLostError extends Error { @@ -72,7 +72,12 @@ export async function withWorkerLease( const sentAt = performance.now(); const result = await untilAborted(client.stepHeartbeat( dispatch.run_id, dispatch.step_id, dispatch.attempt, dispatch.lease_id, - ), controller.signal); + ), controller.signal).catch(error => { + if (error instanceof JournalRequestTimeoutError && error.verb === 'step.heartbeat') { + throw new WorkerLeaseLostError('renewal_expired', error.message, { cause: error }); + } + throw error; + }); controller.signal.throwIfAborted(); // A response handled after local expiry cannot revive ownership, even // if its future deadline was issued before this event loop stalled. diff --git a/packages/sdk/tests/authored-root.test.ts b/packages/sdk/tests/authored-root.test.ts index 589c64fb..2aa0eed0 100644 --- a/packages/sdk/tests/authored-root.test.ts +++ b/packages/sdk/tests/authored-root.test.ts @@ -12,9 +12,9 @@ import { resumeDurableAuthoredFlow, type AuthoredRootMetadata, } from '../src/authored-root.js'; -import { JournalProtocolError, type JournalClient } from '../src/journal-client.js'; +import { JournalProtocolError, JournalRequestTimeoutError, type JournalClient } from '../src/journal-client.js'; import type { RunOutcome, StepDispatchEvent } from '../src/protocol.js'; -import { AuthoredHumanParked } from '../src/authored-flow-error.js'; +import { AuthoredFlowExecutionError, AuthoredHumanParked } from '../src/authored-flow-error.js'; vi.mock('../src/authored-flow-loader.js', async importOriginal => ({ ...await importOriginal(), @@ -383,6 +383,42 @@ describe('durable authored root', () => { expect(journal.peer.completions).toEqual([{ attempt: 2, reason: 'success' }]); }); + it('does not terminalize a root whose heartbeat times out', async () => { + const loaded = await fixture(); + const journal = new RootJournal(); + const controller = new AbortController(); + journal.peer.stepHeartbeat = async () => { + controller.abort(); + throw new JournalRequestTimeoutError('step.heartbeat', 10); + }; + await expect(executeDurableAuthoredFlow(loaded, journal as unknown as JournalClient, undefined, + { dataDir: '/unused', admissionKey: 'heartbeat-timeout', lifecycle: { signal: controller.signal } })) + .rejects.toThrow(); + expect(journal.peer.completions).toEqual([]); + }); + + it('still terminalizes a timed-out mutation as a body failure', async () => { + const loaded = await fixture(false, 0, async () => { throw new JournalRequestTimeoutError('stream.append', 10); }); + const journal = new RootJournal(); + await expect(executeDurableAuthoredFlow(loaded, journal as unknown as JournalClient, undefined, + { dataDir: '/unused', admissionKey: 'write-timeout' })).rejects.toMatchObject({ verb: 'stream.append' }); + expect(journal.peer.completions).toEqual([{ attempt: 1, reason: 'worker_error' }]); + }); + + it.each([false, true])('leaves the root resumable after a read timeout without waiting for redispatch (Node frame=%s)', async nodeFrame => { + const timeout = new JournalRequestTimeoutError('run.get', 10); + const loaded = await fixture(false, 0, async () => { + throw nodeFrame ? new AuthoredFlowExecutionError('daemon_unresponsive', timeout.message) : timeout; + }); + const journal = new RootJournal(); + await expect(executeDurableAuthoredFlow(loaded, journal as unknown as JournalClient, undefined, + { dataDir: '/unused', admissionKey: 'read-timeout' })).rejects.toMatchObject({ + code: 'daemon_unresponsive', rootRunId: 'root-run', message: expect.stringContaining('flows resume'), + }); + expect(journal.peer.completions).toEqual([]); + expect(journal.resumeCalls).toBe(1); + }); + it('terminalizes a returned body failure without replaying semantic side effects', async () => { const loaded = await fixture(true); const journal = new RootJournal(); diff --git a/packages/sdk/tests/classify-outcome.test.ts b/packages/sdk/tests/classify-outcome.test.ts index 47cb6556..d46da85a 100644 --- a/packages/sdk/tests/classify-outcome.test.ts +++ b/packages/sdk/tests/classify-outcome.test.ts @@ -1,3 +1,4 @@ +import { EventEmitter } from 'node:events'; import { describe, expect, it } from 'vitest'; import { classifyOutcome } from '../src/cli/run.js'; @@ -163,6 +164,9 @@ describe('the remedy on a worker park', () => { const until = Date.now() + 3_000; let resumes = 0; const client = { + createPeer: () => Object.assign(new EventEmitter(), { + connect: async () => {}, hello: async () => {}, runWatch: async () => {}, close: () => {}, + }), runGet: async () => (Date.now() < until ? backoff : completed), runResume: async (): Promise => { resumes += 1; @@ -245,6 +249,9 @@ describe('the remedy on a worker park', () => { budget: { tokens_in: 0, tokens_out: 0, dollars: '0' }, }; const client = { + createPeer: () => Object.assign(new EventEmitter(), { + connect: async () => {}, hello: async () => {}, runWatch: async () => {}, close: () => {}, + }), runGet: async () => (Date.now() < until ? running : { ...running, status: 'completed', steps: { answer: { type: 'agent', state: 'done' } } }), runResume: async (): Promise => diff --git a/packages/sdk/tests/cli.test.ts b/packages/sdk/tests/cli.test.ts index c8b6138a..e9d9c86b 100644 --- a/packages/sdk/tests/cli.test.ts +++ b/packages/sdk/tests/cli.test.ts @@ -1010,6 +1010,7 @@ describe('flows run/resume CLI over the journal protocol', () => { let snapshots = 0; await startCliLoopback(dataDir, { hello: sendOk, + 'run.watch': ctx => sendResult(ctx, {}), 'run.start': (ctx) => sendResult(ctx, { run_id: 'run-worker', status: 'parked', @@ -1059,6 +1060,7 @@ describe('flows run/resume CLI over the journal protocol', () => { let snapshots = 0; await startCliLoopback(dataDir, { hello: sendOk, + 'run.watch': ctx => sendResult(ctx, {}), 'run.start': (ctx) => sendResult(ctx, { run_id: 'run-stale-worker', status: 'parked', @@ -1066,7 +1068,7 @@ describe('flows run/resume CLI over the journal protocol', () => { completed_steps: 1, }), 'run.get': (ctx) => { - const running = snapshots++ < 4; + const running = snapshots++ < 1; sendResult(ctx, { run_id: 'run-stale-worker', status: running ? 'running' : 'completed', @@ -1095,13 +1097,14 @@ describe('flows run/resume CLI over the journal protocol', () => { expect(output.stderr.join('\n')).toContain('WAITING [worker_lease]'); expect(output.stderr.join('\n')).toContain(`until ${leaseDeadlineMs}`); expect(output.stderr.join('\n')).not.toContain('worker lease for step "answer" expired'); - expect(snapshots).toBeGreaterThan(4); + expect(snapshots).toBeGreaterThan(1); }); it('allows a caller to cancel a worker-lease wait', async () => { const dataDir = temporaryProject('flows-run-cancel-'); await startCliLoopback(dataDir, { hello: sendOk, + 'run.watch': ctx => sendResult(ctx, {}), 'run.start': (ctx) => sendResult(ctx, { run_id: 'run-cancel', status: 'parked', diff --git a/packages/sdk/tests/direct-run-worker-lease.test.ts b/packages/sdk/tests/direct-run-worker-lease.test.ts index 2b1bbe76..f0d2e9ee 100644 --- a/packages/sdk/tests/direct-run-worker-lease.test.ts +++ b/packages/sdk/tests/direct-run-worker-lease.test.ts @@ -1,3 +1,4 @@ +import { AuthoredFlowExecutionError } from '../src/authored-flow-error.js'; import { afterEach, expect, it, vi } from 'vitest'; import { runDirectFlow } from '../src/cli/direct-run.js'; import { JournalClient, JournalProtocolError } from '../src/journal-client.js'; @@ -32,3 +33,17 @@ it.each([false, true])('direct run handles LLM errors with leaseLost=%s', async expect(result.exitCode).toBe(1); expect(JSON.stringify(result.report)).toContain(leaseLost ? 'connection closed' : 'cli exploded'); }); + +it('reports a read interruption without losing the resumable root', async () => { + vi.spyOn(JournalClient.prototype, 'connect').mockResolvedValue(); + vi.spyOn(JournalClient.prototype, 'hello').mockResolvedValue({} as never); + vi.spyOn(LlmWorker.prototype, 'attach').mockResolvedValue(); + const error = new AuthoredFlowExecutionError('daemon_unresponsive', 'read timed out under CPU load'); + error.rootRunId = 'saved-root'; + vi.mocked(executeDurableAuthoredFlow).mockRejectedValueOnce(error); + const result = await runDirectFlow('flow.ts', '{}', '/unused'); + expect(result).toMatchObject({ exitCode: 1, report: { + command: 'run', runId: 'saved-root', rootRunId: 'saved-root', status: 'running', + diagnostics: [{ kind: 'daemon_unresponsive', message: expect.stringContaining('flows resume --data-dir /unused saved-root') }], + } }); +}); diff --git a/packages/sdk/tests/heartbeat-timeout.test.ts b/packages/sdk/tests/heartbeat-timeout.test.ts new file mode 100644 index 00000000..e4db308b --- /dev/null +++ b/packages/sdk/tests/heartbeat-timeout.test.ts @@ -0,0 +1,24 @@ +import { once } from 'node:events'; +import { expect, it } from 'vitest'; +import { JournalClient } from '../src/journal-client.js'; +import { withWorkerLease, WorkerLeaseLostError } from '../src/worker-lease.js'; +import { sendOk, sockPath, startLoopback } from './journal-client-loopback.js'; + +it('a heartbeat timeout is lease loss rather than a worker body failure', async () => { + const path = sockPath(); + let heartbeats = 0; + const server = startLoopback(path, { hello: sendOk, 'step.heartbeat': () => { heartbeats++; } }); + await once(server, 'listening'); + const client = new JournalClient(path, { requestTimeoutMs: 10 }); + let executed = false; + try { + await client.connect(); + await expect(withWorkerLease(client, { + run_id: 'run', step_id: 'step', attempt: 1, step_type: 'agent', spec: {}, + lease_id: 'lease', lease_deadline_ms: Date.now() + 30_000, lease_ttl_ms: 30_000, + idempotency_key: 'key', pins: { workspace: [], streams: [] }, + }, async () => { executed = true; })).rejects.toBeInstanceOf(WorkerLeaseLostError); + expect(executed).toBe(false); + expect(heartbeats).toBe(1); + } finally { client.close(); await new Promise(resolve => server.close(() => resolve())); } +}); diff --git a/packages/sdk/tests/journal-client-loopback.ts b/packages/sdk/tests/journal-client-loopback.ts index caa434a7..ef35a709 100644 --- a/packages/sdk/tests/journal-client-loopback.ts +++ b/packages/sdk/tests/journal-client-loopback.ts @@ -24,33 +24,34 @@ export interface FrameCtx { } export interface LoopbackHandlers { - 'subscription.inspect'?: (ctx: FrameCtx, params: Record) => void; - 'subscription.deliver'?: (ctx: FrameCtx, params: Record) => void; - 'subscription.fence_overflow'?: (ctx: FrameCtx, params: Record) => void; - hello?: (ctx: FrameCtx) => void; - 'run.start'?: (ctx: FrameCtx, params: Record) => void; - 'run.resume'?: (ctx: FrameCtx, params: Record) => void; - 'run.cancel'?: (ctx: FrameCtx, params: Record) => void; - 'run.get'?: (ctx: FrameCtx, params: Record) => void; - 'run.watch'?: (ctx: FrameCtx, params: Record) => void; - 'worker.attach'?: (ctx: FrameCtx, params: Record) => void; - 'step.heartbeat'?: (ctx: FrameCtx, params: Record) => void; - 'effect.record'?: (ctx: FrameCtx, params: Record) => void; - 'effect.confirm'?: (ctx: FrameCtx, params: Record) => void; - 'step.complete'?: (ctx: FrameCtx, params: Record) => void; - 'step.wait'?: (ctx: FrameCtx, params: Record) => void; - 'event.emit'?: (ctx: FrameCtx, params: Record) => void; - 'subscription.open'?: (ctx: FrameCtx, params: Record) => void; - 'subscription.next'?: (ctx: FrameCtx, params: Record) => void; - 'subscription.close'?: (ctx: FrameCtx, params: Record) => void; - 'journal.read'?: (ctx: FrameCtx, params: Record) => void; - 'stream.append'?: (ctx: FrameCtx, params: Record) => void; - 'stream.read'?: (ctx: FrameCtx, params: Record) => void; + 'subscription.inspect'?: (ctx: FrameCtx, params: Record) => void | Promise; + 'subscription.deliver'?: (ctx: FrameCtx, params: Record) => void | Promise; + 'subscription.fence_overflow'?: (ctx: FrameCtx, params: Record) => void | Promise; + hello?: (ctx: FrameCtx) => void | Promise; + 'run.start'?: (ctx: FrameCtx, params: Record) => void | Promise; + 'run.resume'?: (ctx: FrameCtx, params: Record) => void | Promise; + 'run.cancel'?: (ctx: FrameCtx, params: Record) => void | Promise; + 'run.get'?: (ctx: FrameCtx, params: Record) => void | Promise; + 'run.watch'?: (ctx: FrameCtx, params: Record) => void | Promise; + 'worker.attach'?: (ctx: FrameCtx, params: Record) => void | Promise; + 'step.heartbeat'?: (ctx: FrameCtx, params: Record) => void | Promise; + 'effect.record'?: (ctx: FrameCtx, params: Record) => void | Promise; + 'effect.confirm'?: (ctx: FrameCtx, params: Record) => void | Promise; + 'step.complete'?: (ctx: FrameCtx, params: Record) => void | Promise; + 'step.wait'?: (ctx: FrameCtx, params: Record) => void | Promise; + 'event.emit'?: (ctx: FrameCtx, params: Record) => void | Promise; + 'subscription.open'?: (ctx: FrameCtx, params: Record) => void | Promise; + 'subscription.next'?: (ctx: FrameCtx, params: Record) => void | Promise; + 'subscription.close'?: (ctx: FrameCtx, params: Record) => void | Promise; + 'journal.read'?: (ctx: FrameCtx, params: Record) => void | Promise; + 'stream.append'?: (ctx: FrameCtx, params: Record) => void | Promise; + 'stream.read'?: (ctx: FrameCtx, params: Record) => void | Promise; } -export function startLoopback(path: string, handlers: LoopbackHandlers): Server { +export function startLoopback(path: string, handlers: LoopbackHandlers, options: { serialize?: boolean } = {}): Server { const server = createServer((socket) => { let buffer = ''; + let queue = Promise.resolve(); const send = (obj: unknown): void => { socket.write(JSON.stringify(obj) + '\n'); }; @@ -62,11 +63,14 @@ export function startLoopback(path: string, handlers: LoopbackHandlers): Server buffer = buffer.slice(nl + 1); if (line.length === 0) continue; const req = JSON.parse(line) as { id: string; verb: keyof LoopbackHandlers; params: Record }; - const handler = handlers[req.verb] as ((ctx: FrameCtx, params: Record) => void) | undefined; + const handler = handlers[req.verb] as ((ctx: FrameCtx, params: Record) => void | Promise) | undefined; if (handler === undefined) { send({ id: req.id, ok: false, error: { code: 'unknown_verb', message: req.verb } }); } else { - handler({ id: req.id, socket, send }, req.params); + const handle = () => handler({ id: req.id, socket, send }, req.params); + // server.rs handles one frame at a time per connection. + if (options.serialize) queue = queue.then(handle); + else void handle(); } } }); diff --git a/packages/sdk/tests/journal-client-read-timeout.test.ts b/packages/sdk/tests/journal-client-read-timeout.test.ts new file mode 100644 index 00000000..2efeac22 --- /dev/null +++ b/packages/sdk/tests/journal-client-read-timeout.test.ts @@ -0,0 +1,141 @@ +import { once } from 'node:events'; +import { setTimeout as sleep } from 'node:timers/promises'; +import type { Server } from 'node:net'; +import { afterEach, expect, it } from 'vitest'; +import { JournalClient, JournalRequestTimeoutError } from '../src/journal-client.js'; +import { HELLO_SPEC, sendOk, sendResult, sockPath, startLoopback, type LoopbackHandlers } from './journal-client-loopback.js'; + +const clients: JournalClient[] = []; +const servers: Server[] = []; +afterEach(async () => { + for (const client of clients.splice(0)) client.close(); + await Promise.all(servers.splice(0).map(server => new Promise(resolve => server.close(() => resolve())))); +}); +async function setup(handlers: LoopbackHandlers, budget: number | undefined = 1000) { + const path = sockPath(); + const server = startLoopback(path, { hello: sendOk, ...handlers }, { serialize: true }); + servers.push(server); + await once(server, 'listening'); + const client = new JournalClient(path, { requestTimeoutMs: 50, readBudgetMs: budget }); + clients.push(client); + await client.connect(); + return client; +} + +it('serves a bounded read while an unbounded command is in flight', async () => { + let started!: () => void; + const inFlight = new Promise(resolve => { started = resolve; }); + const client = await setup({ + 'run.start': async ctx => { started(); await sleep(100); sendResult(ctx, { status: 'completed' }); }, + 'run.get': ctx => sendResult(ctx, { status: 'running' }), + }); + let commandDone = false; + const command = client.runStart(HELLO_SPEC).then(result => { commandDone = true; return result; }); + await inFlight; + expect(await client.runGet('run')).toMatchObject({ status: 'running' }); + expect(commandDone).toBe(false); + await command; +}); + +it('retries a read with a fresh id and ignores the late first reply', async () => { + const ids: string[] = []; + let first: Parameters[0]; + const client = await setup({ 'run.get': ctx => { + ids.push(ctx.id); + if (ids.length === 1) first = ctx; + else { sendResult(first, { status: 'stale' }); sendResult(ctx, { status: 'completed' }); } + } }); + expect(await client.runGet('run')).toMatchObject({ status: 'completed' }); + expect(new Set(ids).size).toBe(2); +}); + +it('never retries mutating verbs', async () => { + let writes = 0; + const client = await setup({ 'stream.append': () => { writes += 1; } }); + await expect(client.streamAppend('run', 'stream', {})).rejects.toMatchObject({ + verb: 'stream.append', attempts: 1, message: 'journal client: stream.append timed out after 50ms', + }); + expect(writes).toBe(1); +}); + +it('exhausts a total read budget with a typed diagnostic', async () => { + const client = await setup({ 'run.get': () => {} }, 250); + const error = await client.runGet('run').catch(error => error); + expect(error).toBeInstanceOf(JournalRequestTimeoutError); + expect(error.attempts).toBeGreaterThan(1); + expect(error.message).toContain('read budget 250ms'); + expect(error.message).toContain('CPU load'); +}); + +it('keeps interactive reads single-shot by default', async () => { + let reads = 0; + // Explicitly omit the opt-in. + const client = await setup({ 'run.get': () => { reads += 1; } }, 0); + client.close(); + const interactive = new JournalClient(client.socketPath, { requestTimeoutMs: 10 }); + clients.push(interactive); + await interactive.connect(); + await expect(interactive.runGet('run')).rejects.toMatchObject({ attempts: 1, readBudgetMs: undefined }); + expect(reads).toBe(1); +}); + +it('serializes concurrent reads and stops queued reads on close', async () => { + let reads = 0; + let received!: () => void; + const entered = new Promise(resolve => { received = resolve; }); + const client = await setup({ 'run.get': () => { reads += 1; received(); } }); + const results = Promise.allSettled(Array.from({ length: 8 }, () => client.runGet('run'))); + await entered; + client.close(); + expect((await results).every(result => result.status === 'rejected')).toBe(true); + expect(reads).toBe(1); +}); + +it('does not retry protocol rejections', async () => { + let reads = 0; + const client = await setup({ 'run.get': ctx => { + reads += 1; + ctx.send({ id: ctx.id, ok: false, error: { code: 'run_not_found', message: 'absent' } }); + } }); + await expect(client.runGet('absent')).rejects.toMatchObject({ code: 'run_not_found' }); + expect(reads).toBe(1); +}); + +it('a recovered read timeout does not become an authored callback failure', async () => { + const { flow } = await import('@relayflows/surface'); + const { executeAuthoredFlow } = await import('../src/authored-flow-executor.js'); + let reads = 0; + const client = await setup({ + 'run.start': (ctx, params) => sendResult(ctx, { run_id: (params['spec'] as { steps: Array<{ id: string }> }).steps[0]!.id, status: 'completed', completion_reason: 'success', completed_steps: 1 }), + 'journal.read': (ctx, params) => { + if (++reads === 1) return; + sendResult(ctx, { entries: [{ seq: 1, run_id: params['run_id'], step_id: params['run_id'], entry_type: 'step.completed', + payload: { completionReason: 'success', output: { stdout_tail: 'ok', exit_code: 0 } } }] }); + }, + }); + const result = await executeAuthoredFlow(flow('retried-read', async f => { + await f.run('echo ok'); + f.done('success'); + }), client, undefined); + expect(result.completionReason).toBe('success'); + expect(reads).toBe(3); +}); + +it('falls back to the primary connection if reader setup is refused', async () => { + const client = await setup({ + hello: ctx => ctx.send({ id: ctx.id, ok: false, error: { code: 'busy', message: 'reader unavailable' } }), + 'run.get': ctx => sendResult(ctx, { status: 'completed' }), + }); + expect(await client.runGet('run')).toMatchObject({ status: 'completed' }); +}); + +it.each(['run.get', 'journal.read', 'stream.read', 'subscription.inspect'] as const)('retries only the allowlisted read %s', async verb => { + let attempts = 0; + const client = await setup({ [verb]: ctx => { if (++attempts === 2) sendResult(ctx, {}); } }); + const result = verb === 'run.get' ? client.runGet('run') + : verb === 'journal.read' ? client.journalRead('run', 1) + : verb === 'stream.read' ? client.streamRead('run', 'stream', 0) + : client.subscriptionInspect({ run_id: 'run', subscription_id: 'subscription' }); + await result; + expect(attempts).toBe(2); +}); diff --git a/packages/sdk/tests/read-timeout-resume-live.test.ts b/packages/sdk/tests/read-timeout-resume-live.test.ts new file mode 100644 index 00000000..67730987 --- /dev/null +++ b/packages/sdk/tests/read-timeout-resume-live.test.ts @@ -0,0 +1,60 @@ +import { readFileSync, writeFileSync } from 'node:fs'; +import { join } from 'node:path'; +import { expect, it } from 'vitest'; +import { JournalClient } from '../src/journal-client.js'; +import { loadAuthoredFlow } from '../src/authored-flow-loader.js'; +import { executeDurableAuthoredFlow, resumeDurableAuthoredFlow } from '../src/authored-root.js'; +import { LlmWorker } from '../src/llm-worker.js'; +import { chainFixture, shellWord } from './flow-chain-fixture.js'; + +it('parks an unreadable authored root and resumes without repeating its journaled effect', async () => { + const fixture = chainFixture(); + const journal = await fixture.connect(); + const { pid } = JSON.parse(readFileSync(join(fixture.data, 'connection.json'), 'utf8')); + const worker = new LlmWorker(journal, 'park-read-worker'); + const client = new JournalClient(journal.socketPath, { requestTimeoutMs: 100, readBudgetMs: 500 }); + const effects = join(fixture.root, 'effects'); + writeFileSync(fixture.flowPath, `import {flow} from '@relayflows/surface'; +export default flow('park-read', async f => { + await f.run(${JSON.stringify(`echo saved >> ${shellWord(effects)}`)}); + await f.llm('hello', {model:'test-model', output:{type:'object'}}); + await f.run(${JSON.stringify(`echo after >> ${shellWord(effects)}`)}); + f.done('success'); +}); +`); + const get = client.runGet.bind(client); + let interrupted = false; + // Suspend service exactly when the body first reads its running child. The + // request and retry timers are real; no synthetic rejection is injected. + client.runGet = runId => { + if (!interrupted) { interrupted = true; process.kill(pid, 'SIGSTOP'); } + return get(runId); + }; + let rootRunId = ''; + try { + await client.connect(); + await worker.attach(); + const loaded = await loadAuthoredFlow(fixture.flowPath); + try { + await expect(executeDurableAuthoredFlow(loaded, client, undefined, { + dataDir: fixture.data, admissionKey: 'park-read', onAdmitted: id => { rootRunId = id; }, + })).rejects.toMatchObject({ code: 'daemon_unresponsive', rootRunId: expect.any(String) }); + } finally { process.kill(pid, 'SIGCONT'); client.runGet = get; } + expect(interrupted).toBe(true); + expect(readFileSync(effects, 'utf8')).toBe('saved\n'); + const entries = (await journal.journalRead(rootRunId, 1, 1000)).entries as Array<{ + entry_type: string; step_id: string; payload: { completionReason?: string }; + }>; + expect(entries.some(entry => entry.step_id === 'authored-root' + && entry.entry_type === 'step.completed' && entry.payload.completionReason === 'worker_error')).toBe(false); + expect(await resumeDurableAuthoredFlow(rootRunId, client, { dataDir: fixture.data })) + .toMatchObject({ rootRunId, completionReason: 'success' }); + expect(readFileSync(effects, 'utf8')).toBe('saved\nafter\n'); + expect(await journal.runGet(rootRunId)).toMatchObject({ status: 'completed' }); + } finally { + process.kill(pid, 'SIGCONT'); + client.close(); + await worker.close(); + await fixture.close(); + } +}, 30_000); diff --git a/packages/sdk/tests/resume-worker-lease.test.ts b/packages/sdk/tests/resume-worker-lease.test.ts index 91a5b304..102f2e25 100644 --- a/packages/sdk/tests/resume-worker-lease.test.ts +++ b/packages/sdk/tests/resume-worker-lease.test.ts @@ -1,3 +1,4 @@ +import { AuthoredFlowExecutionError } from '../src/authored-flow-error.js'; import { afterEach, expect, it, vi } from 'vitest'; import { JournalClient, JournalProtocolError } from '../src/journal-client.js'; import { LlmWorker } from '../src/llm-worker.js'; @@ -34,3 +35,17 @@ it.each([false, true])('resume handles LLM errors with leaseLost=%s', async leas expect(JSON.stringify(result.report)).toContain(leaseLost ? 'connection closed' : 'cli exploded'); expect(warning).toHaveBeenCalledTimes(leaseLost ? 1 : 0); }); + +it('reports a read interruption without losing the resumable root', async () => { + vi.spyOn(JournalClient.prototype, 'connect').mockResolvedValue(); + vi.spyOn(JournalClient.prototype, 'hello').mockResolvedValue({} as never); + vi.spyOn(LlmWorker.prototype, 'attach').mockResolvedValue(); + const error = new AuthoredFlowExecutionError('daemon_unresponsive', 'read timed out under CPU load'); + error.rootRunId = 'saved-root'; + vi.mocked(resumeDurableAuthoredFlow).mockRejectedValueOnce(error); + const result = await resumeFlow('saved-root', '/unused', { localAgent: true }); + expect(result).toMatchObject({ exitCode: 1, report: { + command: 'resume', runId: 'saved-root', rootRunId: 'saved-root', status: 'running', + diagnostics: [{ kind: 'daemon_unresponsive', message: expect.stringContaining('flows resume --data-dir /unused --local-agent saved-root') }], + } }); +}); diff --git a/packages/sdk/tests/run-daemon-unresponsive.test.ts b/packages/sdk/tests/run-daemon-unresponsive.test.ts new file mode 100644 index 00000000..641f1f22 --- /dev/null +++ b/packages/sdk/tests/run-daemon-unresponsive.test.ts @@ -0,0 +1,21 @@ +import { afterEach, expect, it, vi } from 'vitest'; +import { classifyOutcome } from '../src/cli/run.js'; +import { JournalRequestTimeoutError, type JournalClient } from '../src/journal-client.js'; +import { probeSocket } from '../src/daemon-connection.js'; +vi.mock('../src/daemon-connection.js', async original => ({ + ...await original(), probeSocket: vi.fn(), +})); +afterEach(() => vi.clearAllMocks()); +it.each([true, false])('reports an unreadable declarative run as resumable (reachable=%s)', async reachable => { + vi.mocked(probeSocket).mockResolvedValue({ reachable }); + const client = { runGet: async () => { throw new JournalRequestTimeoutError('run.get', 10, 3, 100, 100); } }; + const result = await classifyOutcome(client as unknown as JournalClient, 'run', { + run_id: 'saved-run', status: 'parked', completion_reason: null, completed_steps: 30, + }, { command: 'run', ok: true, diagnostics: [] } as never, '/socket', {}); + expect(result).toMatchObject({ exitCode: 1, report: { + runId: 'saved-run', rootRunId: 'saved-run', status: 'running', + diagnostics: [{ kind: reachable ? 'daemon_unresponsive' : 'daemon_unreachable', + message: expect.stringContaining('flows resume saved-run') }], + } }); + expect(probeSocket).toHaveBeenCalledWith('/socket'); +}); diff --git a/packages/sdk/tests/run-read-load-live.test.ts b/packages/sdk/tests/run-read-load-live.test.ts new file mode 100644 index 00000000..5b7e6de0 --- /dev/null +++ b/packages/sdk/tests/run-read-load-live.test.ts @@ -0,0 +1,68 @@ +import { execFileSync } from 'node:child_process'; +import { availableParallelism } from 'node:os'; +import { readFileSync } from 'node:fs'; +import { join } from 'node:path'; +import { expect, it } from 'vitest'; +import { toKernelSpec } from '../src/compile.js'; +import { JournalClient } from '../src/journal-client.js'; +import { chainFixture, shellWord } from './flow-chain-fixture.js'; + +it('completes a CPU-saturating deterministic flow with reads in flight and preserves its journal', async () => { + const fixture = chainFixture(); + const primary = await fixture.connect(); + const client = new JournalClient(primary.socketPath, { requestTimeoutMs: 100, readBudgetMs: 15_000 }); + let processors = availableParallelism(); + if (process.platform === 'linux') { + // Saturate the daemon's entire CPU allocation without starving unrelated + // test fixtures whose synthetic worker leases last only 30 milliseconds. + const cpu = readFileSync('/proc/self/status', 'utf8').match(/^Cpus_allowed_list:\s*(\d+)/m)![1]!; + const { pid } = JSON.parse(readFileSync(join(fixture.data, 'connection.json'), 'utf8')); + execFileSync('taskset', ['-apc', cpu, String(pid)]); + processors = 1; + } + const stamp = join(fixture.root, 'effects'); + let started!: (runId: string) => void; + const admitted = new Promise(resolve => { started = resolve; }); + client.on('entry', entry => { if (entry.entry_type === 'run.spawned') started(entry.run_id); }); + // One runnable CPU burner per available processor, launched by the flow itself. + // The shell trap owns all burners even if the deterministic step is terminated. + const command = `pids=''; trap 'kill $pids 2>/dev/null; wait' EXIT; ` + + `i=0; while [ "$i" -lt ${processors} ]; do yes >/dev/null & pids="$pids $!"; i=$((i+1)); done; sleep 2`; + try { + await client.connect(); + await client.hello('load-regression'); + const execution = client.runStart(toKernelSpec({ version: '0.1.0', name: 'read-under-load', steps: [ + { id: 'saved', type: 'deterministic', command: `echo saved >> ${shellWord(stamp)}` }, + { id: 'cpu', type: 'deterministic', command, dependsOn: ['saved'] }, + { id: 'after', type: 'deterministic', command: 'echo survived', dependsOn: ['cpu'] }, + ] }), undefined, undefined, true); + const runId = await admitted; + const reads = Promise.all(Array.from({ length: 8 }, () => client.runGet(runId))); + const [outcome, snapshots] = await Promise.all([execution, reads]); + expect(outcome).toMatchObject({ status: 'completed', completion_reason: 'success', completed_steps: 3 }); + expect(snapshots.every(snapshot => snapshot.run_id === runId)).toBe(true); + const entries = (await client.journalRead(runId, 1, 1000)).entries as Array<{ entry_type: string }>; + expect(entries.filter(entry => entry.entry_type === 'step.completed')).toHaveLength(3); + expect(await client.runResume(runId)).toMatchObject({ status: 'completed', completion_reason: 'success' }); + expect(readFileSync(stamp, 'utf8')).toBe('saved\n'); + } finally { client.close(); await fixture.close(); } +}, 30_000); + +it('drains read and watch promises before an authored flow completes', async () => { + const { flow } = await import('@relayflows/surface'); + const { executeAuthoredFlow } = await import('../src/authored-flow-executor.js'); + const { LlmWorker } = await import('../src/llm-worker.js'); + const fixture = chainFixture(); + const workerClient = await fixture.connect(); + const worker = new LlmWorker(workerClient, 'read-policy-worker'); + const client = new JournalClient(workerClient.socketPath, { readBudgetMs: 5_000 }); + try { + await client.connect(); + await worker.attach(); + const result = await executeAuthoredFlow(flow('read-policy-scope', async f => { + await f.llm('hello', { model: 'test-model', output: { type: 'object' } }); + f.done('success'); + }), client, undefined, { flowPath: fixture.flowPath }); + expect(result.completionReason).toBe('success'); + } finally { await worker.close(); client.close(); await fixture.close(); } +}, 30_000); diff --git a/packages/sdk/tests/running-step-watch.test.ts b/packages/sdk/tests/running-step-watch.test.ts new file mode 100644 index 00000000..b55bf635 --- /dev/null +++ b/packages/sdk/tests/running-step-watch.test.ts @@ -0,0 +1,59 @@ +import { once } from 'node:events'; +import { setTimeout as sleep } from 'node:timers/promises'; +import { expect, it } from 'vitest'; +import { JournalClient } from '../src/journal-client.js'; +import { waitForRunningStep } from '../src/cli/running-step.js'; +import { sendOk, sendResult, sockPath, startLoopback } from './journal-client-loopback.js'; + +it('uses pushes for completion with lease-cadence reads and releases its watcher', async () => { + let reads = 0; + let watchClosed = false; + const path = sockPath(); + const server = startLoopback(path, { + hello: sendOk, + 'run.watch': (ctx, params) => { + const entry = (run: string, type: string) => ctx.send({ event: 'entry', data: { + run_id: run, step_id: 'step', entry_type: type, + } }); + // An older failed attempt in replay must not finish the current one. + entry('run', 'step.completed'); + entry('run', 'step.attempt.started'); + sendResult(ctx, {}); + entry('other-run', 'step.completed'); + setTimeout(() => entry(String(params['run_id']), 'step.completed'), 2100); + ctx.socket.once('close', () => { watchClosed = true; }); + }, + 'run.get': ctx => { reads++; sendResult(ctx, { steps: { step: { + state: 'running', lease_deadline_ms: Date.now() + 30_000, + } } }); }, + }); + await once(server, 'listening'); + const client = new JournalClient(path); + try { + await client.connect(); + const start = performance.now(); + await waitForRunningStep(client, 'run', { id: 'step', type: 'agent', leaseDeadlineMs: Date.now() + 30_000 }, {}); + expect(performance.now() - start).toBeGreaterThan(2000); + expect(reads).toBeLessThanOrEqual(1); + await sleep(10); + expect(watchClosed).toBe(true); + } finally { client.close(); await new Promise(resolve => server.close(() => resolve())); } +}); + +it('cancels a pending watch registration and removes its connection', async () => { + const path = sockPath(); + let registered!: () => void; + const registration = new Promise(resolve => { registered = resolve; }); + const server = startLoopback(path, { hello: sendOk, 'run.watch': () => registered() }); + await once(server, 'listening'); + const client = new JournalClient(path); + const controller = new AbortController(); + try { + await client.connect(); + const waiting = waitForRunningStep(client, 'run', { id: 'step', type: 'agent', leaseDeadlineMs: Date.now() + 30_000 }, { signal: controller.signal }); + const rejected = expect(waiting).rejects.toThrow('was canceled'); + await registration; + controller.abort(); + await rejected; + } finally { client.close(); await new Promise(resolve => server.close(() => resolve())); } +}); diff --git a/packages/sdk/tests/worker-lease-sweep.test.ts b/packages/sdk/tests/worker-lease-sweep.test.ts index 1cc66b93..c3348a30 100644 --- a/packages/sdk/tests/worker-lease-sweep.test.ts +++ b/packages/sdk/tests/worker-lease-sweep.test.ts @@ -1,3 +1,4 @@ +import { EventEmitter } from 'node:events'; import { afterEach, expect, it, vi } from 'vitest'; import { classifyOutcome, emptyReport } from '../src/cli/run.js'; import type { JournalClient } from '../src/journal-client.js'; @@ -14,10 +15,13 @@ function fixture() { }); const parked = { run_id: 'run', status: 'parked' as const, completion_reason: null, completed_steps: 0 }; const client = { + createPeer: () => Object.assign(new EventEmitter(), { + connect: async () => {}, hello: async () => {}, runWatch: async () => {}, close: () => {}, + }), runGet: vi.fn().mockResolvedValue(snapshot('running')), runResume: vi.fn().mockResolvedValue(parked), }; - const run = () => classifyOutcome(client as unknown as JournalClient, 'run', parked, emptyReport('run'), '/unused', {}); + const run = () => classifyOutcome(client as unknown as JournalClient, 'run', parked, emptyReport('run'), '', {}); return { client, snapshot, run }; } @@ -31,14 +35,14 @@ it('waits through an expired snapshot until the kernel retries and completes', a .mockResolvedValue({ run_id: 'run', status: 'completed', completion_reason: 'success', completed_steps: 1 }); const execution = run(); const assertion = expect(execution).resolves.toMatchObject({ exitCode: 0, report: { completionReason: 'success' } }); - await Promise.all([assertion, vi.advanceTimersByTimeAsync(1_000)]); + await Promise.all([assertion, vi.advanceTimersByTimeAsync(4_000)]); expect(client.runResume).toHaveBeenCalledTimes(2); }); it('still fails when the kernel never resolves an expired lease after sweep grace', async () => { const { run } = fixture(); const assertion = expect(run()).rejects.toThrow('without completion'); - await vi.advanceTimersByTimeAsync(35_500); + await vi.advanceTimersByTimeAsync(36_500); await assertion; }); @@ -46,7 +50,7 @@ it('bounds an unchanged lease when the CLI clock is behind the daemon', async () const { client, snapshot, run } = fixture(); client.runGet.mockResolvedValue(snapshot('running', Date.now() + 75_000)); const assertion = expect(run()).rejects.toThrow('without completion'); - await vi.advanceTimersByTimeAsync(35_500); + await vi.advanceTimersByTimeAsync(36_500); await assertion; }); @@ -69,5 +73,5 @@ it('follows a live step whose renewed deadline looks expired on a skewed CLI clo client.runGet.mockResolvedValue(snapshot('completed')); client.runResume.mockResolvedValue({ run_id: 'run', status: 'completed', completion_reason: 'success', completed_steps: 1 }); const assertion = expect(execution).resolves.toMatchObject({ exitCode: 0, report: { completionReason: 'success' } }); - await Promise.all([assertion, vi.advanceTimersByTimeAsync(1_000)]); + await Promise.all([assertion, vi.advanceTimersByTimeAsync(4_000)]); });