Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
72 changes: 69 additions & 3 deletions docs/SURFACE.md
Original file line number Diff line number Diff line change
Expand Up @@ -246,7 +246,8 @@ No process runs between events: the handler wakes, executes to its next await, p
one, while a worker that dies stops renewing and the lease expires. The run's
wallclock budget is separate and stops *new* work, draining whatever is
already running rather than cancelling a step mid-flight. A wrapper step and
a native step therefore get the same duration.
a native step therefore get the same duration by default. Agent steps may
declare the per-step `timeout` described below, enforced on both paths.

`flows check` resolves the binary (a path is relative to the declaring flow
or project config; a bare name resolves via `PATH`) and caches each resolved
Expand Down Expand Up @@ -347,7 +348,8 @@ authoring-time narrowing, not a kernel guarantee.

### Per-agent permissions in TypeScript

Supported `f.agent` calls accept an optional `permissions` declaration:
Supported `f.agent` calls accept an optional `timeout` (see Agent step timeouts)
and an optional `permissions` declaration:

```ts
const draft = await f.agent("writer", {
Expand Down Expand Up @@ -641,6 +643,68 @@ the kernel kills the command's process group and journals `completionReason: tim
`f.run` refuses with code `lease_exceeded`. The override applies only to that
invocation; calls without options retain the default.

### Agent step timeouts

`f.agent(name, { task, timeout?: string | number })` accepts the same duration
syntax as `f.run`: numeric milliseconds or strings with `ms`, `s`, or `m`
(including `'1.5s'`). Omitting `timeout` keeps the existing unlimited CLI
execution duration. The authoring/`flows check` ceiling is **60 minutes**
(3600000 ms), inclusive: measured repair work reached 44m09s, so 60m allows
about 35% headroom and leaves room in a 2h flow for publishing. This is a
compiler/SDK validation limit; the kernel validates positive milliseconds
fitting in `i64` and carries the declaration to the worker.

```ts
const repair = await f.agent('repair', { task: 'Repair failing checks.', timeout: '45m' });
if (repair.completionReason === 'timeout') {
await f.run('git push'); // publish what the agent already committed
}
f.done('success');
```

**A declared timeout resolves; it does not throw.** `AgentResult` includes
`completionReason: 'success' | 'timeout'`. Handle the timeout by branching on
that field, not with `catch`. The worker stops the CLI process group at the
execution deadline, using the existing graceful stop and forced-kill escalation,
then journals `step.completed` with `completionReason: 'timeout'`. Settlement
includes process-stop confirmation, so it can occur shortly after the deadline.
Failure to confirm the stop remains a failure, not a recoverable timeout.
The agent child run remains failed (`step_failed`) in status/dashboard views;
the authored flow may continue and succeed. Other failures still throw.
A predicate `.gate(result => ...)` still runs on this result; returning false
lets an author fail the flow on timeout. Named data gates require successful
producer output; a timeout that prevents such a gate from running still fails
the operation, rather than bypassing the declared check.

On timeout, `summary` contains journaled timeout evidence and `artifacts` is
`[]`. Native CLI partial output is retained in transport failure evidence;
native artifact paths remain under `trajectory_tail.transcript.artifacts.paths`
in the journal (a bounded list). Wrappers discard partial stdout on timeout
and provide the deadline message; they have no transcript artifact list. A
wrapper result envelope emitted before the deadline still supplies its usage.
No workspace reset occurs: committed work and uncommitted edits remain, and
the following step sees that potentially dirty tree even with
`recoveryMode: 'reset'`. Unlike crash/lease recovery in RFC Appendix A, timeout
settles the step with no successor attempt. Resume replays the recorded timeout
and does not execute that agent again. Incurred spend remains charged:
reported usage is priced as usual, and usage never reported before the deadline
is journaled as dollar-unmetered rather than as a measured $0.

`timeout` with `maxIterations > 1` is refused, so semantic iterations cannot
multiply the limit. Transport recovery before a timeout can start a new CLI
execution with its own timer; this is an execution deadline, not a total flow
budget. `transport: 'relay'` with a timeout is refused because the local worker
cannot stop the remote process. Authored option errors use the existing
`agent_cli_unresolved` refusal, before child admission.

Declarative YAML/JSON uses `timeoutMs: 2700000` on agent steps: **integer
milliseconds only**, no duration strings. It lowers to journal spec
`timeout_ms`. This enforced duration is separate from
`requirements.expectedDurationMs`, which is only a placement estimate.
Use matching SDK, daemon, and worker versions: older daemons refuse the new
field, while older workers do not enforce it. The native and wrapper paths
both honor the declaration; no default wrapper duration cap is introduced.

### Repair before failure: `onNonZero: 'record'`

A deterministic step is gated on its exit code by default: a nonzero exit fails
Expand Down Expand Up @@ -781,7 +845,9 @@ vocabulary:
recorded before author code can reach the operation, so a `catch` cannot hide
it. `done("step_failed")` does not change that: it declares a verdict about
checks the body ran and read for itself, and is not a way to continue past a
step that failed.
step that failed. A declared agent timeout is the explicit exception:
it journals a failed step but resolves with `completionReason: 'timeout'`,
allowing the flow to continue (see Agent step timeouts).
3. **Finish your derived work before `done()`.** If a handler chained onto a step
is still in flight when the body returns, the run is refused with
`unsettled_derived_work` rather than recorded as a success nobody can prove.
Expand Down
65 changes: 65 additions & 0 deletions evidence/agent-timeout/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
# Agent timeout verification

Commands ran from the repository root unless a `cd` is shown. The linked files
contain captured stdout/stderr, including failures; they are not paraphrases.
Dependencies were installed in surface and SDK, the SDK's surface dependency was
linked to this checkout's built surface, and `ops/cargo.sh` installed its private
toolchain. No workflow files or gate workflows were changed.

| Literal command | Captured output |
| --- | --- |
| `npm run build --prefix packages/sdk` | [build-sdk.txt](build-sdk.txt) |
| `sh ops/cargo.sh build --manifest-path kernel/Cargo.toml` | [build-kernel.txt](build-kernel.txt) |
| `npm run typecheck --prefix packages/sdk` | [typecheck.txt](typecheck.txt) |
| `npm run typecheck:tests --prefix packages/sdk` | [test-typecheck.txt](test-typecheck.txt) |
| `sh ops/cargo.sh test --manifest-path kernel/Cargo.toml -p relayflowd-core` | [kernel.txt](kernel.txt) |
| `npm run generate --prefix packages/schema` | [schema-generate.txt](schema-generate.txt) |
| `npm test --prefix packages/schema` | [schema-tests.txt](schema-tests.txt) |

Final SDK regression command, with [captured output](final-regressions.txt):

```sh
cd packages/sdk && npx vitest run tests/agent-timeout.test.ts tests/agent-timeout-outcome.test.ts tests/agent-timeout-worker.test.ts tests/agent-timeout-live.test.ts tests/step-lease.test.ts tests/spec-parity.test.ts tests/verb-field-lint.test.ts tests/wrapper-execution-duration.test.ts tests/stop-process-group.test.ts tests/worker-cli.test.ts tests/authored-node-result.test.ts tests/authored-retried-child-resume-live.test.ts tests/authored-agent-artifacts.test.ts
```

Earlier broader command, with [captured failures](regression-tests.txt):

```sh
cd packages/sdk && npx vitest run tests/agent-timeout.test.ts tests/agent-timeout-worker.test.ts tests/agent-timeout-live.test.ts tests/step-lease.test.ts tests/spec-parity.test.ts tests/verb-field-lint.test.ts tests/wrapper-execution-duration.test.ts tests/stop-process-group.test.ts tests/worker-cli.test.ts tests/authored-node-result.test.ts tests/live-kernel.test.ts tests/authored-retried-child-resume-live.test.ts tests/authored-agent-artifacts.test.ts
```

That run found a stale per-verb descriptor expectation (updated to admit agent
`timeoutMs`) and nine live-kernel failures in this checkout. The checkout inherits
`type: commonjs` from `/home/daytona/package.json`; its extensionless ESM fixtures
did not emit their wrapper handshake. A daemon-discovery test also needed an
explicit binary path.

The live-kernel suite was rerun in `/tmp/agent-timeout-baseline`, an isolated
worktree **overlaid with the changed source, new files, and current SDK dist**.
Despite the directory name, this is the implemented code, not a baseline test.
Its SDK dependencies link to the installed dependencies in the working checkout.
No fixtures were rewritten to change their behavior. The analyzer opt-out flag
allows the existing suite to report unavailable live Claude access if needed;
the captured output records whether the analyzer actually executed.

```sh
cd /tmp/agent-timeout-baseline/packages/sdk && RELAYFLOWD_BIN=/home/daytona/.relayflows-toolchain/target/2962130851/debug/relayflowd RELAYFLOWS_ALLOW_ANALYZER_SKIP=1 ./node_modules/.bin/vitest run tests/live-kernel.test.ts
```

[Captured isolated live-kernel output](isolated-live-kernel.txt).

The timeout-specific runtime and kill/resume tests ran in the original checkout
as part of the final SDK regression command. They do not depend on the isolated
checkout or on live provider access.

The remaining real Claude analyzer failure was also run against the original
branch head (`c88c3d0`) in `/tmp/agent-timeout-original`, with its own original
surface and SDK rebuilt. It used the same daemon executable. The original
source produced the same execution-gate failure; this is not a green live
analyzer acceptance claim.

```sh
cd /tmp/agent-timeout-original/packages/sdk && RELAYFLOWD_BIN=/home/daytona/.relayflows-toolchain/target/2962130851/debug/relayflowd ./node_modules/.bin/vitest run tests/live-kernel.test.ts -t 'hn-monitor analyze-story reaches done through the real Claude analyzer CLI'
```

[Captured original-head analyzer output](original-analyzer.txt).
4 changes: 4 additions & 0 deletions evidence/agent-timeout/build-kernel.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
Compiling relayflowd-core v0.1.0 (/home/daytona/.relayflow-v2-supervisor/durable/repository/kernel/relayflowd-core)
Compiling relayflowd-journal v0.1.0 (/home/daytona/.relayflow-v2-supervisor/durable/repository/kernel/relayflowd-journal)
Compiling relayflowd v0.1.0 (/home/daytona/.relayflow-v2-supervisor/durable/repository/kernel/relayflowd)
Finished `dev` profile [unoptimized + debuginfo] target(s) in 3.00s
4 changes: 4 additions & 0 deletions evidence/agent-timeout/build-sdk.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@

> @relayflows/sdk@2.0.39 build
> tsc && node scripts/make-cli-executable.mjs

67 changes: 67 additions & 0 deletions evidence/agent-timeout/final-regressions.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@

RUN v2.1.9 /home/daytona/.relayflow-v2-supervisor/durable/repository/packages/sdk

✓ tests/authored-node-result.test.ts (45 tests) 35ms
✓ tests/verb-field-lint.test.ts (99 tests) 384ms
✓ tests/wrapper-execution-duration.test.ts (7 tests) 10858ms
✓ keeps the handshake deadline independent of the removed execution deadline 10062ms
✓ still lets a lease abort stop an unlimited wrapper before it produces output 511ms
✓ tests/stop-process-group.test.ts (11 tests) 14607ms
✓ every stop reaches the process group, not just the direct child > exits the run after an execution-timeout stop 1003ms
✓ every stop reaches the process group, not just the direct child > exits the run after a protocol terminate stop 653ms
✓ every stop reaches the process group, not just the direct child > kills a SIGTERM-deaf grandchild after a protocol terminate stop 1584ms
✓ every stop reaches the process group, not just the direct child > kills a SIGTERM-deaf grandchild after an execution-timeout stop 1944ms
✓ every stop reaches the process group, not just the direct child > holds the loop open long enough for the escalation to run 1104ms
✓ every stop reaches the process group, not just the direct child > bounds forced-stop confirmation when a group remains unprovable 1009ms
✓ a wrapper that exits with no execution deadline still drains > reports the wrapper result and reaps a grandchild holding its pipes 797ms
✓ a wrapper that exits with no execution deadline still drains > reaps a SIGTERM-deaf grandchild holding its pipes 1886ms
✓ a wrapper that exits with no execution deadline still drains > settles on its own deadline when an escaped holder withholds close 4289ms
✓ tests/authored-agent-artifacts.test.ts (5 tests) 356ms
✓ tests/spec-parity.test.ts (46 tests) 538ms
✓ tests/authored-retried-child-resume-live.test.ts (2 tests) 1499ms
✓ an authored step whose child run retried an attempt > resumes after a kill mid-step and reads the terminal completion, not the crashed one 1361ms
✓ tests/agent-timeout-live.test.ts (3 tests) 4486ms
✓ journals timeout, stops the process, runs a predicate gate and publishes under a budget header 1231ms
✓ replays timeout after killing the root and daemon, without executing the agent again 2078ms
✓ lets an author reject timeout through a predicate gate 1176ms
✓ tests/agent-timeout-worker.test.ts (8 tests) 1981ms
✓ stops claude at its declared limit, preserving work and stopping descendants 523ms
✓ stops codex at its declared limit, preserving work and stopping descendants 519ms
✓ stops wrapper at its declared limit, preserving work and stopping descendants 540ms
✓ tests/agent-timeout-outcome.test.ts (6 tests) 12ms
✓ tests/agent-timeout.test.ts (23 tests) 38ms
✓ tests/worker-cli.test.ts (25 tests) 30887ms
✓ registered CLI model defaults > passes the same priced Claude default to the real provider invocation 532ms
✓ registered CLI model defaults > uses the explicitly supplied agent environment for the provider subprocess 581ms
✓ registered CLI model defaults > dispatches canonical generic bytes with the preflight-proved adapter identity 571ms
✓ direct transport lifecycle evidence > classifies only the exact Codex stdin lifecycle signature as retryable 583ms
✓ direct transport lifecycle evidence > records a signal close separately from an ordinary nonzero exit 1000ms
✓ direct transport lifecycle evidence > records a spawn error code without treating a missing executable as transient 405ms
✓ direct transport lifecycle evidence > journals classified lifecycle evidence and reports crashed instead of generic worker_error 554ms
✓ step discovery environment > names the run, step, attempt and an absolute data dir for a direct agent spawn 624ms
✓ step discovery environment > exports none of the four without a data dir, even when the worker inherited them 499ms
✓ wrapper discovery environment > sets the four names from the dispatch and still refuses ambient values and other secrets 530ms
✓ wrapper discovery environment > exports none of the four to a wrapper without a data dir, even when the worker inherited them 521ms
✓ custom wrapper execution identity > passes an explicit safe environment at identification and execution 577ms
✓ custom wrapper execution identity > refuses a wrapper symlink retarget before delivering private values 505ms
✓ custom wrapper execution identity > bounds wrapper execution after acknowledgement 557ms
✓ custom wrapper execution identity > bounds captured wrapper output 553ms
✓ custom wrapper execution identity > refuses a duplicate execute protocol frame 552ms
✓ custom wrapper execution bounds are reader-owned > resolves when a conforming wrapper leaks a stdio pipe to a background helper 1867ms
✓ custom wrapper execution bounds are reader-owned > resolves when the leaked helper inherits stderr only 1849ms
✓ custom wrapper execution bounds are reader-owned > resolves when a wrapper leaks a stdio pipe and exits before identifying 3464ms
✓ custom wrapper execution bounds are reader-owned > journals a completionReason at the default bound when a wrapper leaks a stdio pipe 11584ms
✓ custom wrapper execution bounds are reader-owned > accepts an execute token and an over-8KiB payload flushed in one write 506ms
✓ custom wrapper execution bounds are reader-owned > accepts the same over-8KiB payload whether or not it coalesces with the execute token 1499ms
✓ custom wrapper execution bounds are reader-owned > still bounds an un-terminated handshake buffer and names the bound 507ms
✓ delivers the journaled memory pack to the real wrapper and excludes its charge from completion usage 466ms
✓ tests/step-lease.test.ts (36 tests) 66522ms
✓ f.run leases against the live kernel > enforces 10000 ms for 'sleep 5; printf ok' 5084ms
✓ f.run leases against the live kernel > enforces 40000 ms for 'sleep 31; printf ok' 31086ms
✓ f.run leases against the live kernel > enforces 30000 ms for 'sleep 31; printf ok' 30128ms

Test Files 13 passed (13)
Tests 316 passed (316)
Start at 20:46:54
Duration 83.60s (transform 1.76s, setup 138ms, collect 6.15s, tests 132.20s, environment 2ms, prepare 666ms)

Loading
Loading