[Kyber5323]

Implements specs ADR decisions/cloud/workflow/20261021-node-retry-and-resume-contract.md on top of main 5ba0d25. The ADR is approved and stays approved in this change. A separate specs landing run opens the approved → implemented PR once this PR's review passes; that specs PR merges after this PR.

What changes

• Retry policy at publish: agentConfig.retry is validated with the four named gate codes (agent_retry_not_an_object, agent_retry_enabled_invalid, agent_retry_max_retries_invalid, agent_retry_initial_delay_invalid). Absent or null selects the default enabled=true, maxRetries=2, initialDelaySeconds=10; frozen snapshots are never re-gated.
• Waiting state: an agent_failed session within budget commits retrying with retryAt from the database clock. The failed attempt's evidence and startedAt are kept, the run stays running, and its serial session slot stays held.
• Wake and budget: the scheduler starts attempt n+1 only when retryAt is due, through the ordinary session path, under the run's exclusive transaction and version CAS. The wait is initialDelaySeconds × 2^(n-1), capped at 600s, and the budget is maxRetries + 1 attempts.
• Per-attempt identity: migration 0069_workflow_node_attempt.sql adds workflow_node_attempt to execution_work, node_executions and thread_commands and re-keys the session and end-request partial unique indexes on it. Main took 0063–0068 while this branch was in review (the last, 0068_scheduled_manifest_completeness.sql, from cloud#154), so this migration follows them; the migration README count and the test name/comments follow the new number. The attempt travels from the work item to the execution and command; the Controller wire shape is unchanged.
• Cancel and exhaustion: cancellation wins over a waiting retry and starts no later attempt; budget exhaustion, a disabled policy and every non-agent_failed reason fail the node and release the Workspace.
• Resume contract: POST /api/v1/tenants/:tid/workflows/:wfid/runs/:rid/resume uses the cancel authorization, accepts only optional mode (keep, node_files, checkpoint), and answers every valid request with 501 workflow_run_resume_unavailable. It writes no run state and no idempotency record, so a replay is the same refusal; there is no UI entry.
• UI and docs: the run view shows retrying, the attempt number and the next retry instant; both workflow guides document the behavior and link the ADR. The frontend keeps retrying and carries no awaiting_input, which cloud#131 removed from the public status surface.

Tests

• task check — formatting, lint, proto drift, full PostgreSQL-backed suite
• task test:race — race-enabled full suite
• task frontend:check — generated-client drift, lint, types, tests, module docs, build

Rebased onto main 5ba0d25. node_execution.go keeps both main's dual-delivery concurrency fence and this branch's attempt column, and the scheduled-artifact enqueue passes the new attempt argument as IssueRun work. Regenerating api/openapi.json and the frontend client produces no diff.

Refs ora-space/circus#469

🤖 Generated with Claude Code

[Kyber5323]

Reviewed at 5607ad0 (base ed3badd). The retry core follows D1–D6 and I found no behavioral defect in the implementation:

• D1: the publish gate and the total runtime parser agree on the shape and bounds; absent/null/invalid reads as the default policy; the wait is initialDelaySeconds × 2^(n-1) capped at 600s; the budget is maxRetries + 1.
• D2: the decision reads result.reason before the status mapping, only agent_failed retries, and the agent_plugin_unavailable detail is lifted only at terminal settlement.
• D3/D5: cancellation runs first, then the retrying branch (which always returns), then running-session settlement; retryAt comes from clock_timestamp(); cancellation clears retryAt and starts no attempt even when the due instant has passed.
• D4: migration 0062 is forward-only, adds NOT NULL DEFAULT 1 columns and re-keys the two partial unique indexes; the attempt is copied from the claimed work item, never from the wire; the claim join and delivery ownership check carry it; every evidence/end/cancel lookup is attempt-scoped; the Controller projection still excludes the internal column.
• D7 and the contract: route shape, authorization-before-availability, 400 before the 501, no idempotency record, the mode enum and the 501 description all match; the generated OpenAPI/frontend client and the zh/en docs are consistent.

Two evidence gaps to close before this can carry the "Only The Current Attempt Can Settle Its Node" core case:

1. workflow_retry_db_test.go does not exercise cross-attempt command scoping. Every command path in the tests is attempt 1. The core case requires a direct PostgreSQL test where:

   • attempt 1 has an undelivered end_session and attempt 2's session is live: claimThreadCommands must not return attempt 1's command for attempt 2, and recordThreadCommandDelivered with attempt 1's command and attempt 2's execution must return dispatch_conflict;
   • a repeated/late attempt 1 terminal receipt is posted after attempt 2 starts, and the pass must leave node_states at running/attempt 2. The current replay assertion runs while the node is still retrying, so it does not cover the attempt-2-in-flight projection.

   The code for both is present and looks right; the tests are what is missing. This also lets the landing run record those obligations as Covered instead of Missing (specs test-cases/cloud/workflow/retry-and-resume.md).

2. internal/core/migrations/README.md:9 and README.en.md:9 say 63 SQL files; the tree at 5607ad0 has 64. (The base was already one behind: 63 files, README 62.) Please set both to 64.

Non-blocking for the owner: I agree with the literal D2 reading of the #147 interaction recorded in the PR body — the decision keys on the session's result.reason, so a session-reported agent_failed with detail=agent_plugin_unavailable stays retryable, and narrowing the closed set is an ADR change, not an implementation detail. That question can be settled in the owner's review or a separate decision issue.

[zhangshilong123]

Reviewed at 9f72d15 (base d085bb9). The retry implementation matches D1–D6 and the resume registration matches D7, and the two evidence gaps from the earlier review are closed at this head: TestWorkflowRetryKeepsAttemptOneRequestAndResultOutOfAttemptTwo covers cross-attempt claim/delivery plus a late attempt-1 receipt while attempt 2 is in flight, and both migration READMEs now say 64. CI is green and the file list is all deliverables.

Two inaccuracies in the new nodeStates wording (internal/contract/openapi.go:137, regenerated into api/openapi.json and frontend/src/api/generated.schemas.ts):

1. "startedAt is kept from the first attempt" — a node that retries twice has a retrying state whose startedAt is attempt 2's start, not the first attempt's. startWorkflowAgentSession writes a fresh StartedAt for every attempt and commitWorkflowAgentTerminal keeps the failed attempt's value, so the description should say "the failed attempt's start".
2. "finishedAt is absent" — workflowNodeState.object() always emits finishedAt; a retrying state carries "". Please say it is empty. (Omitting the key instead would change the wire shape for every status, so the wording is the smaller fix.)

Please fix both in openapi.go, regenerate the contract and the frontend client, and re-request review.

Non-blocking: resumeWorkflowRunAs (integration/workflow_run_resume_test.go:42) takes a uid parameter it never uses — the request identity comes from user — so it can be dropped.

[Kyber5323]

@zhangshilong123 both wording issues are fixed on the new head 6cb8f7e (base d085bb9):

• internal/contract/openapi.go now says a retrying state's startedAt is the failed attempt's start and that finishedAt is empty.
• api/openapi.json and frontend/src/api/generated.schemas.ts were regenerated with task frontend:generate; no hand edits.
• Non-blocking: resumeWorkflowRunAs no longer takes the unused uid parameter.

Local gates on 6cb8f7e: task check, task test:race and task frontend:check all pass. GitHub CI is running on the new head. Please re-review.

[zhangshilong123]

Reviewed at 6cb8f7e (base d085bb9). The two requested fixes are correct: the nodeStates description now names the failed attempt's start and an empty finishedAt, api/openapi.json and frontend/src/api/generated.schemas.ts carry exactly the regenerated wording, and the unused uid parameter is gone from resumeWorkflowRunAs.

Two blocking items before this head can be reviewed for merge:

1. The branch is CONFLICTING with main (mergeStateStatus: DIRTY). The conflicts are 7 files under frontend/src/features/workflows/, caused by #131 (f65172b) removing awaiting_input while this branch still carries it from its d085bb9 base. Please rebase onto current upstream/main (49a427e or newer) and resolve by dropping awaiting_input and keeping retrying; then re-check that no awaiting_input reference is left in the touched frontend files (run-node-state.ts, run-status-style.ts, run-status-mark.test.tsx, run-status-style.test.ts, translations.ts, types.ts).

2. No CI has run on 6cb8f7e: the commit has 0 check-runs and there is no workflow run for it, so the task check / task test:race / task frontend:check result claimed in ora-space/circus#469 is not evidenced on GitHub. After the rebase, push and let the Backend and Frontend workflows finish green; if they do not start, say so on the run issue rather than waiting.

Also, before merge the landing run has to open the specs PR that turns decisions/cloud/workflow/20261021-node-retry-and-resume-contract.md from approved into implemented. Per the protocol that PR is opened once this review passes and merged after this one, so the description's "follows after this PR merges" has the order reversed; please correct it when you update the PR.

[zhangshilong123]

Review 结论：需要修改（fa60a7e，base 6e302ef）

上一轮的两项修正、以及这次 rebase 对 cloud#131 冲突的处理都已核对：

• nodeStates 描述（失败尝试的开始时间、finishedAt 为空）与 resumeWorkflowRunAs 的未使用参数清理都正确。
• frontend/src 与 api/openapi.json 已无 awaiting_input，retrying 保留；internal/contract/openapi.go 只保留“存储保留值、没有写入者”的注释。生成的契约与前端 client 无手改。
• fa60a7e 上 7 个检查全部通过（check、frontend、integration-0–integration-3、packages）。
• PR 描述里 ADR 保持 approved、implemented 的 specs PR 在 review 通过后开、产品 PR 合并后合并，顺序正确。

但当前 head 还不能合并，原因都是 main 在本分支上次 rebase 之后又前进了：

1. 与当前 main 5ba0d25 冲突（mergeStateStatus: DIRTY）。冲突文件：internal/core/migrations/README.md、internal/core/migrations/README.en.md、internal/core/node_execution.go。node_execution.go 里 main（#154/#158）把分发并发栅栏改成允许 deliver_revision 与 deliver_scheduled_artifacts 并存，本分支在同一个 INSERT 上加了 workflow_node_attempt；rebase 时两者都要保留：main 的新栅栏加回去，INSERT 带上 attempt 列并从领取的工单复制。请 rebase 到当前 upstream/main。

2. 迁移编号与 main 同号：main 现在已有 0063_scheduled_run_results.sql 到 0068_scheduled_manifest_completeness.sql，本分支的 0063_workflow_node_attempt.sql 需要顺延为 0069_workflow_node_attempt.sql（memory/rules/migration-numbering.md），并同步：

   • 迁移 README 两种语言的行与计数。注意 main 的计数行目前写 62，但索引里已有 70 个文件（这是 main 已有的过时值）；加上本迁移后正确是 71。
   • TestMigration0063WorkflowNodeAttemptAppliesFreshAndUpgrades 测试名、migrationsBefore(t, "0063_workflow_node_attempt.sql")、before 0063 等引用。
   • internal/core/control_projection_db_test.go 与 internal/core/workflow_run_session.go 里提到 0063 的注释。
   • PR 描述里的 base（6e302ef → 最新 main）和迁移编号。

rebase 后重新 push，等 CI 绿，再重新打 status:in-review，我复审。

[Kyber5323]

Addressed both blockers in 4a1efb2, rebased onto main 5ba0d25:

• internal/core/node_execution.go keeps main's dual-delivery concurrency fence and this branch's workflow_node_attempt column. The rebase also surfaced the added attempt parameter in scheduled_artifact_delivery.go's enqueueExecutionWork call, which now passes an empty node and attempt 0 as IssueRun work.
• The migration is now 0069_workflow_node_attempt.sql; both migration READMEs list it at 0069 with the count at 71 (main's 70 files plus this one), the upgrade wording starts from 0068, and the test name, migrationsBefore(...) call, failure message, comments and test schema prefix follow. The PR description is updated to match.

Local task check, task test:race and task frontend:check pass, and regenerating api/openapi.json and the frontend client produces no diff. CI is green on 4a1efb2 (check, packages, frontend, integration-0–integration-3).

@zhangshilong123 please take another look when you have a moment.

[zhangshilong123]

已审 head 4a1efb2（base 5ba0d25）。上一轮提出的两项阻塞都已解决，本轮未发现新的阻塞项，Approve。

已核对通过

• rebase 冲突：mergeStateStatus: CLEAN。node_execution.go 同时保留 main 的双交付并发栅栏和本分支的 workflow_node_attempt 列；scheduled_artifact_delivery.go 按非图节点给 enqueueExecutionWork 传空节点和 attempt 0，与其它 IssueRun 调用一致；attempt 只从领取的工作项复制到执行行，Controller 线上形状不变。
• 迁移编号：0069_workflow_node_attempt.sql 接在 main 最新的 0068 之后（main 70 个 SQL + 本迁移 = 71，README 两份的计数行、索引行、说明段、测试名 TestMigration0069WorkflowNodeAttemptAppliesFreshAndUpgrades、升级起点 0068 与 schema 前缀都已同步），符合 memory/rules/migration-numbering.md。
• 实现与 ADR D1–D7：发布门禁四个命名错误码与完整对象校验；agent_failed 按会话原因闭集判定后才映射状态；retrying 的 attempt/retryAt 用数据库时钟计算、保留失败尝试的 startedAt/error/output.reason 并清空 finishedAt，运行保持 running 且不释放 Workspace；到期只按已提交的 retryAt 唤醒且并发只放出一个 attempt；证据、end_session、取消路径全部按 attempt 限定，跨 attempt 投递被拒绝；取消优先于等待；预算耗尽按最后一次证据失败并释放；resume 按最终请求形状校验后一律 501、不写状态与幂等记录，没有 UI 入口。
• 测试：等待→到期→第 2 次尝试结算、并发 pass 只放出一个 attempt、跨 attempt 命令/结果隔离、预算耗尽、不可重试原因、取消优先、迁移升级，均有真实 PostgreSQL 证据。
• CI：4a1efb2 上 check、packages、frontend、integration-0–integration-3 七个检查全部通过。
• 范围与描述：改动文件只有代码、测试、迁移、生成的契约与前端客户端、正式文档；PR 描述写明 ADR 保持 approved 及落地运行顺序。

后续

review 已通过，接下来由落地运行在 specs 开 PR 把 decisions/cloud/workflow/20261021-node-retry-and-resume-contract.md 改为 implemented，产品 PR 合并后合并该 specs PR。