Skip to content

relayflowd run.get times out under CPU load and kills the run: a flow's own test suite can destroy 30 steps of journaled work #613

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
agent-relay-code wants to merge 1 commit into
base: main
Choose a base branch
from
+3,657 −103
Draft

relayflowd run.get times out under CPU load and kills the run: a flow's own test suite can destroy 30 steps of journaled work #613

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 26 additions & 0 deletions docs/SURFACE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1736,6 +1736,32 @@ typed `run_not_found` refusal. A dropped connection, request failure, or
journal may already have changed and the CLI cannot honestly claim the resume
was refused before a write.

A timed-out read is handled separately: flow execution clients (`flows run`
and `flows resume`, authored and declarative) retry only `run.get`,
`journal.read`, `stream.read`, and `subscription.inspect` within a 300-second
total budget. Reads use a separate session, one in flight per body client,
with increasing attempt bounds and jitter. Interactive clients keep the
single-shot 30-second default; writes are never retried by this policy.
Unretried timeouts retain `journal client: <verb> timed out after <ms>ms`.
Exhausted reads instead name the verb, attempts, elapsed time, total read
budget, and possible CPU load.

After a read budget expires, the CLI probes a fresh connection. A responding
daemon reports `daemon_unresponsive`; a failed probe reports
`daemon_unreachable` (also used for initial attach failures). A failed probe
cannot prove the daemon is dead: its diagnostic names both unreachability and
CPU load. Both reports exit 1, carry the known run/root id, retain
`status: running`, and include `flows resume`. This parks the CLI execution
without manufacturing a terminal journal fact; it does not claim a journaled
human park or verify journal integrity. The root worker session closes so the
kernel can recover its attempt, preserving completed work for resume.

Worker completion waits use `run.watch` pushes with a snapshot every two
seconds for the live lease deadline. A scoped watch session closes after each
wait because the protocol has no unwatch verb. A heartbeat timeout relinquishes
worker ownership as lease loss, leaving recovery to the kernel instead of
recording `worker_error` against the body.

A step that ran and failed is **not** one of those. It reports `step_failed`
with `status: failed`, for every step type and for authored TypeScript flows
as well as declarative ones. `protocol_error` is reserved for an outcome the
Expand Down
17 changes: 17 additions & 0 deletions evidence/run-read-timeout/baseline-setup.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
The handshake comparison used an unmodified detached checkout of
`3c58ee16d10a9e2400db980f5bbafac84e437f20`. The two probe sources are copied
verbatim from the `/tmp` scripts named in their captured output. They use this
session's absolute checkout/build paths; adjust those paths when reproducing
on another machine.

Setup used (from the implementation checkout):

```sh
git worktree add --detach /tmp/relayflow-read-timeout-baseline 3c58ee16d10a9e2400db980f5bbafac84e437f20
ln -s /home/daytona/.relayflow-v2-supervisor/durable/repository/packages/sdk/node_modules /tmp/relayflow-read-timeout-baseline/packages/sdk/node_modules
npm run build --prefix /tmp/relayflow-read-timeout-baseline/packages/sdk
```

The baseline probe loads SDK code from that checkout while keeping the daemon
binary and stub fixture path identical to the current-code probe. It compares
that one fixture handshake failure, not the full suite.
59 changes: 59 additions & 0 deletions evidence/run-read-timeout/final-focused.log
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
$ cd packages/sdk && RELAYFLOWD_BIN=/home/daytona/.relayflows-toolchain/target/2962130851/debug/relayflowd npx vitest run tests/journal-client-read-timeout.test.ts tests/journal-client.test.ts tests/journal-client-completion.test.ts tests/journal-client-subscriptions.test.ts tests/running-step-watch.test.ts tests/heartbeat-timeout.test.ts tests/run-daemon-unresponsive.test.ts tests/authored-root.test.ts tests/classify-outcome.test.ts tests/cli.test.ts tests/direct-run-worker-lease.test.ts tests/resume-worker-lease.test.ts tests/worker-lease.test.ts tests/worker-lease-lost.test.ts tests/worker-lease-sweep.test.ts tests/run-read-load-live.test.ts tests/worker-lease-lost-live.test.ts tests/flow-executor-chain.test.ts tests/agent-transcript-live.test.ts tests/human-live.test.ts --maxWorkers=1 --minWorkers=1

RUN v2.1.9 /home/daytona/.relayflow-v2-supervisor/durable/repository/packages/sdk

✓ tests/cli.test.ts (71 tests) 6026ms
✓ flows check CLI > binds a checked relative wrapper to the flow directory for worker execution 566ms
✓ flows check CLI > resolves a bare PATH-resolved claude with no declared model, in an isolated PATH 488ms
✓ flows run/resume CLI over the journal protocol > follows a dispatched worker step instead of reporting a protocol error 2053ms
✓ flows run/resume CLI over the journal protocol > follows a worker wait past a locally expired lease until the daemon settles it 2045ms
(node:44939) [FLOWS_ROOT_LEASE_LOST] Warning: authored root run_id=root-run attempt=1: lease_conflict: attempt has no active worker lease. Waiting for the kernel to retry it.
(Use `node --trace-warnings ...` to show where the warning was created)
✓ tests/authored-root.test.ts (26 tests) 398ms
✓ tests/journal-client.test.ts (17 tests) 89ms
✓ tests/flow-executor-chain.test.ts (14 tests) 10315ms
✓ flow executor LLM and output-binding chain > runs f.llm -> f.agent -> f.run with schema-verified journal output and the exact allowed model 1022ms
✓ flow executor LLM and output-binding chain > runs a dollar-budgeted authored Claude agent with the same default used by preflight 700ms
✓ flow executor LLM and output-binding chain > runs the exact authored flagship f.llm -> f.agent -> f.run path through the durable CLI root 1595ms
✓ flow executor LLM and output-binding chain > resumes an interrupted durable authored root without replaying completed flagship effects 3382ms
✓ flow executor LLM and output-binding chain > passes a declarative verified value through an agent into a deterministic artifact 779ms
✓ flow executor LLM and output-binding chain > flows run consumes YAML bindings and resume reuses the original journal output 1068ms
✓ tests/agent-transcript-live.test.ts (4 tests) 3394ms
✓ the transcript digest through the built CLI, a real daemon and the local agent > preserves structured agent failure details and its completed root index 875ms
✓ the transcript digest through the built CLI, a real daemon and the local agent > preserves structured llm failure details and its completed root index 818ms
✓ the transcript digest through the built CLI, a real daemon and the local agent > journals the digest in trajectory_tail on a successful agent step and writes the file it points at 851ms
✓ the transcript digest through the built CLI, a real daemon and the local agent > on a failed agent step, names the failure and the transcript in the terminal diagnostic, redacted 848ms
✓ tests/classify-outcome.test.ts (11 tests) 7422ms
✓ classifyOutcome > gives up and reports when a running run never becomes classifiable 2009ms
✓ the remedy on a worker park > follows a step through a retry backoff longer than the unclassified bound 3005ms
✓ the remedy on a worker park > follows a run.start outcome that is already running on a retried attempt 2001ms
✓ tests/human-live.test.ts (3 tests) 7854ms
✓ f.human against a real daemon > parks with the question, refuses wrong answers, records one, and resumes to success 4785ms
✓ f.human against a real daemon > a "no" is a value the body branches on: declined, exit 0, no effect 1916ms
✓ f.human against a real daemon > refuses to answer a run the daemon does not know 1152ms
✓ tests/worker-lease.test.ts (7 tests) 20ms
✓ tests/worker-lease-lost.test.ts (17 tests) 27ms
✓ tests/journal-client-read-timeout.test.ts (13 tests) 1104ms
✓ a recovered read timeout does not become an authored callback failure 368ms
✓ tests/worker-lease-lost-live.test.ts (3 tests) 943ms
✓ reports journal success after completion rejects with lease_conflict 327ms
✓ reports journal success when a renewal rejects after completion landed 327ms
✓ tests/run-read-load-live.test.ts (2 tests) 2821ms
✓ completes a CPU-saturating deterministic flow with reads in flight and preserves its journal 2095ms
✓ drains read and watch promises before an authored flow completes 725ms
✓ tests/worker-lease-sweep.test.ts (4 tests) 8ms
✓ tests/journal-client-completion.test.ts (6 tests) 102ms
✓ tests/resume-worker-lease.test.ts (3 tests) 6ms
✓ tests/direct-run-worker-lease.test.ts (3 tests) 10ms
✓ tests/running-step-watch.test.ts (2 tests) 2122ms
✓ uses pushes for completion with lease-cadence reads and releases its watcher 2118ms
✓ tests/journal-client-subscriptions.test.ts (1 test) 8ms
✓ tests/run-daemon-unresponsive.test.ts (2 tests) 4ms
✓ tests/heartbeat-timeout.test.ts (1 test) 16ms

Test Files 20 passed (20)
Tests 210 passed (210)
Start at 10:41:25
Duration 54.29s (transform 1.45s, setup 97ms, collect 8.09s, tests 42.69s, environment 3ms, prepare 1.05s)

exit=0
Loading
Loading