feat(workflows): retry agent nodes and register the resume contract as unavailable

api/openapi.json

@@ -5913,7 +5913,7 @@
           },
           "nodeStates": {
             "additionalProperties": true,
-            "description": "Per-node execution state keyed by node id: status, displayName, startedAt, finishedAt, output, and error — the reason, present only when the node failed.",
+            "description": "Per-node execution state keyed by node id: status, displayName, startedAt, finishedAt, output, and error — the reason, present only when the node failed or is retrying. An agent node can be `retrying`, which means its last attempt failed with a retryable reason and the run is waiting for the next one: `retryAt` is the database-clock instant that attempt becomes due, `error` and `output.reason` name the failed attempt's reason, `startedAt` is the failed attempt's start, `finishedAt` is empty, and the run itself stays `running`. `attempt` is the 1-based attempt the state describes, present whenever the node has started at least once, so a terminal node states which attempt ended it; a node's retry budget is `maxRetries + 1` attempts.",
             "type": "object"
           },
           "pluginVersion": {
@@ -31197,6 +31197,175 @@
         ]
       }
     },
+    "/api/v1/tenants/{tid}/workflows/{wfid}/runs/{rid}/resume": {
+      "post": {
+        "description": "Public requests require a gateway service credential plus a caller-bound user credential. Active tenant membership is checked before lookup. Shared projects expose safe runtime summaries; runtime content and use require the verified creator or current tenant administrator. Conflicting mutations additionally require an effective server-confirmed control session. Workflows are tenant-owned graph documents. `graph` is the authored document (nodes, edges, viewport, editor annotations, global variables), stored and returned whole — the editor is its only reader, so no field inside it is validated or indexed here. A live workflow name is unique per tenant; archiving one frees its name. Resumes a terminal failed workflow run inside the same run identity. The route is registered against its final request contract, but no resume mode can run in this phase: every valid request answers 501 workflow_run_resume_unavailable, changes no run, node or Workspace state, and records no idempotency entry, so repeating the request returns the same refusal instead of a replay. mode selects how much Workspace state rolls back — keep (the default when omitted), node_files or checkpoint — and any other value is 400 invalid_field_value; the body carries only mode, and any other field is 400 unknown_field. Authorization is exactly the cancellation request's: the run's initiator and the tenant's administrators on a run carrying frozen executor bindings, and an active tenant member who is neither answers 403 workflow_run_resume_forbidden. A missing or mismatched tenant, workflow or run, and a soft-deleted run are all 404 not_found. Mutation version conflicts return 409; a missing required version returns 428. Unknown fields are rejected. Lists use ascending UUID pagination.",
+        "operationId": "post_api_v1_tenants_tid_workflows_wfid_runs_rid_resume",
+        "parameters": [
+          {
+            "in": "path",
+            "name": "tid",
+            "required": true,
+            "schema": {
+              "format": "uuid",
+              "type": "string"
+            }
+          },
+          {
+            "in": "path",
+            "name": "wfid",
+            "required": true,
+            "schema": {
+              "format": "uuid",
+              "type": "string"
+            }
+          },
+          {
+            "in": "path",
+            "name": "rid",
+            "required": true,
+            "schema": {
+              "format": "uuid",
+              "type": "string"
+            }
+          },
+          {
+            "description": "Scoped to tenant and user. Same key and canonical method/path/body returns the original response before version validation; changed request is 409.",
+            "in": "header",
+            "name": "Idempotency-Key",
+            "required": true,
+            "schema": {
+              "maxLength": 200,
+              "minLength": 1,
+              "type": "string"
+            }
+          }
+        ],
+        "requestBody": {
+          "content": {
+            "application/json": {
+              "schema": {
+                "additionalProperties": false,
+                "properties": {
+                  "mode": {
+                    "enum": [
+                      "keep",
+                      "node_files",
+                      "checkpoint"
+                    ],
+                    "type": "string"
+                  }
+                },
+                "type": "object"
+              }
+            }
+          },
+          "required": true
+        },
+        "responses": {
+          "400": {
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/Error"
+                }
+              }
+            },
+            "description": "Invalid JSON/field/input, missing idempotency key, invalid pagination or evidence"
+          },
+          "401": {
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/Error"
+                }
+              }
+            },
+            "description": "Invalid, forged, expired, wrong-audience, untrusted, or caller-mismatched credential"
+          },
+          "403": {
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/Error"
+                }
+              }
+            },
+            "description": "Disabled user, inactive/missing membership, wrong service role, or admin required"
+          },
+          "404": {
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/Error"
+                }
+              }
+            },
+            "description": "Resource absent or outside authorized tenant/owner scope"
+          },
+          "409": {
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/Error"
+                }
+              }
+            },
+            "description": "Version/idempotency conflict, resource_in_use, closed admission, stale epoch/Node/sandbox, incomplete effect, invalid transition, unconfirmed termination/idle, last_admin, space_last_owner, space_slug_conflict, or default_space_protected"
+          },
+          "428": {
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/Error"
+                }
+              }
+            },
+            "description": "Version precondition required"
+          },
+          "500": {
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/Error"
+                }
+              }
+            },
+            "description": "Internal error; no SQL or secret details are exposed"
+          },
+          "501": {
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/Error"
+                }
+              }
+            },
+            "description": "The request is valid but the operation is not available in this deployment (e.g. workflow_run_resume_unavailable)"
+          },
+          "503": {
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/Error"
+                }
+              }
+            },
+            "description": "External capability not wired in this deployment (a port is Unavailable), e.g. form_descriptor_unavailable or assist_unavailable"
+          }
+        },
+        "security": [
+          {
+            "serviceCredential": [],
+            "userCredential": []
+          }
+        ],
+        "summary": "POST /api/v1/tenants/:tid/workflows/:wfid/runs/:rid/resume",
+        "tags": [
+          "workflows"
+        ]
+      }
+    },
     "/api/v1/tenants/{tid}/workflows/{wfid}/runs/{rid}/statistics": {
       "get": {
         "description": "Read-only active-member model statistics for a WorkflowRun. Covers all recorded model-gateway HTTP attempts owned by this WorkflowRun, independently of request-page pagination. The run must belong to the named workflow and be live; any active tenant member who can read the run can read its statistics. This is not a historical request backfill or billing ledger. Each metric exposes its own known/total coverage; absent values remain null, including a run with no observations. Input/cache semantics follow the source protocol; reasoning is an output subset. No query parameters accepted; private, no-store.",

docs/workflows/workflows.en.md

@@ -113,6 +113,15 @@ node states, session or Workspace; the scheduler settles it on its next pass (se
 bindings frozen on the run: on a bound run only the initiator and the tenant's administrators may cancel, and any
 other member answers `403 workflow_run_cancel_forbidden`; an unbound run keeps the Issue Thread's member rule.
 
+**The resume endpoint is registered but unavailable in this phase.** `POST .../workflows/:wfid/runs/:rid/resume`
+validates the request against the final contract: `mode` is optional and is one of `keep` (the default), `node_files`
+and `checkpoint`; an unknown field is `400 unknown_field` and an invalid value is `400 invalid_field_value`.
+Authorization is cancellation's: on a bound run only the initiator and the tenant's administrators may call it, and
+any other member answers `403 workflow_run_resume_forbidden`. Every valid request answers
+`501 workflow_run_resume_unavailable` and writes nothing — no run state and no idempotency record — so a repeat is
+the same refusal rather than a replay of a recorded success. A failed run's Workspace is already declared deleted,
+leaving `keep` nothing to resume onto, so the frontend offers no resume entry point.
+
 ## Scheduling
 
 WorkflowRuns are carried forward by a scheduling loop in the Cloud server (`AdvanceWorkflowRunsOnce`, every 10
@@ -160,6 +169,18 @@ seconds):
   of its Workspace. In the first phase this session has no Git Server
   access (reads and writes get `403`), and its Workspace is not listed in an IssueRun session's Workspace directory;
   see [Git Server](../repositories/git-server.en.md).
+- **Agent-node retry is decided by the `agentConfig.retry` frozen on the node.** A session retries only when it ends
+  as `agent_failed`, the policy is enabled and the budget is not spent: the budget is `maxRetries + 1` attempts
+  (`maxRetries` ranges 0–5 and defaults to 2), and the next attempt starts after `initialDelaySeconds × 2^(n-1)`
+  (`initialDelaySeconds` ranges 0–300 and defaults to 10, with a 600-second cap per wait). While it waits, the node
+  commits `retrying`: the run stays `running`, the node keeps the failed attempt's `startedAt`, `error` and
+  `output.reason`, records `retryAt` from the database clock and clears `finishedAt`; it keeps the run's single
+  session slot until `retryAt` arrives. The new attempt has its own execution identity (`workflow_node_attempt`), so
+  the previous attempt's result and end request are never reused. A spent budget, a disabled policy or a reason
+  outside `agent_failed` fails the node with the last reason, and the run reaches a terminal state and releases its
+  Workspace. At publish, `agentConfig.retry` must be a complete object; absent or `null` means the default policy,
+  and unknown keys are ignored. A frozen historical snapshot is never re-gated, and a policy that cannot be parsed
+  runs as the default.
 - **A workflow session is single-turn, and Cloud decides when it ends.** Once the initial turn's `turnEnded`
   receipt arrives (with a `stop_reason` other than `cancelled`), the scheduler ends the session through the existing
   `EndSession` command relay; the session result the Node reports remains the node's only terminal evidence. If the
@@ -231,6 +252,7 @@ rewrites imported documents or published versions.
 - [Workflows are executed by a Cloud-side scheduler: WorkflowRun is the execution aggregate](https://github.com/ora-space/specs/blob/main/decisions/cloud/workflow/0-cloud-side-scheduler-owns-workflow-execution.md) (approved)
 - [Workflow agent nodes reference executors by executor slot; graphs no longer store Agent CLIs or model IDs](https://github.com/ora-space/specs/blob/main/decisions/cloud/workflow/20261015-executor-slots-replace-agent-model-ids.md) (approved)
 - [A workflow session ends after its initial turn by the Cloud scheduler, with a bounded wait on the end request](https://github.com/ora-space/specs/blob/main/decisions/cloud/workflow/20261019-workflow-session-end-after-initial-turn.md)
+- [Workflow agent nodes retry under a retry policy; the resume endpoint is registered against its final contract but unavailable in the first phase](https://github.com/ora-space/specs/blob/main/decisions/cloud/workflow/20261021-node-retry-and-resume-contract.md) (approved)
 - [Plugin asset import produces published snapshots: the asset lifecycle never rewrites snapshots, and runs freeze only the snapshotId](https://github.com/ora-space/specs/blob/main/decisions/cloud/workflow/20261016-asset-import-and-published-snapshots.md)
 - [Workflow plugin assets are fetched by Cloud and imported document by document as editable copies](https://github.com/ora-space/specs/blob/main/decisions/cloud/plugin-marketplace/20261016-workflow-plugin-assets.md)
 - [Personal slot mappings resolve a workflow's executors; each run pins one model binding per slot](https://github.com/ora-space/specs/blob/main/decisions/cloud/model-access/20261015-executor-slot-mappings-and-per-slot-run-bindings.md) (approved)

docs/workflows/workflows.md

@@ -81,6 +81,12 @@ Project 缺失时统一返回 `409 workflow_run_project_required`，不可用、
 `pending` 和 `running` 的运行接受请求；已经进入终态的运行返回 `409 run_not_cancellable`。授权按运行上冻结的模型绑定判断：带绑定的运行只有发起人和
 租户管理员可以取消，其他成员返回 `403 workflow_run_cancel_forbidden`；没有绑定的运行沿用 Issue 线程的成员规则。
 
+**恢复接口已注册，但本阶段不可用。** `POST .../workflows/:wfid/runs/:rid/resume` 按最终契约校验请求体：`mode` 可选，取值
+`keep`（默认）、`node_files`、`checkpoint`，未知字段返回 `400 unknown_field`，非法取值返回 `400 invalid_field_value`；授权与取消相同，
+带冻结绑定的运行只有发起人和租户管理员可以调用，其他成员返回 `403 workflow_run_resume_forbidden`。任何合法请求都返回
+`501 workflow_run_resume_unavailable`，不改动运行状态，也不写幂等记录——重复请求得到同样的拒绝，而不是重放一条记录下来的成功。运行的
+Workspace 在失败时已经声明删除，`keep` 没有可恢复的状态，所以前端不提供恢复入口。
+
 ## 调度
 
 WorkflowRun 由 Cloud server 里的调度循环推进（`AdvanceWorkflowRunsOnce`，每 10 秒一轮）：
@@ -107,6 +113,13 @@ WorkflowRun 由 Cloud server 里的调度循环推进（`AdvanceWorkflowRunsOnce
   已提交的结果按它自己的原因结算，Node 不可达仍报 `node_unavailable`，两者都不成立时才轮到窗口；窗口把运行带入终态之后才到达的结果
   保留在执行行上，不改写已经终态的运行。运行进入终态后，Cloud 声明删除这次运行的 Workspace。第一阶段这个会话没有 Git Server 访问（读写都返回 `403`），它的 Workspace 也不进 IssueRun 会话的 Workspace
   目录，见 [Git Server](../repositories/git-server.md)。
+- **Agent 节点重试由冻结在节点上的 `agentConfig.retry` 决定。** 只有会话以 `agent_failed` 结束、策略启用且预算未用完时才重试：预算是
+  `maxRetries + 1` 次尝试（`maxRetries` 取值 0–5，默认 2），下一次尝试在 `initialDelaySeconds × 2^(n-1)` 之后开始（`initialDelaySeconds`
+  取值 0–300，默认 10，单次等待上限 600 秒）。等待期间节点提交 `retrying`：运行仍是 `running`，节点保留失败尝试的 `startedAt`、
+  `error` 和 `output.reason`，写入数据库时钟计算的 `retryAt` 并清空 `finishedAt`；它继续占用运行唯一的会话位置，直到 `retryAt` 到达才启动
+  下一次尝试。新尝试使用自己的执行身份（`workflow_node_attempt`），上一次尝试的结果和结束请求不会被它复用。预算用完、策略关闭或原因不在
+  `agent_failed` 内时，节点以最后一次的原因失败，运行进入终态并释放 Workspace。发布时 `agentConfig.retry` 必须是完整对象，缺失或为 `null`
+  表示默认策略，未知键忽略；已冻结的历史快照不重新校验，无法解析的策略按默认值执行。
 - **工作流会话是单回合的，结束由 Cloud 决定。** 初始回合的 `turnEnded` 回执（`stop_reason` 不是 `cancelled`）到达后，调度器通过既有的 `EndSession`
   命令让 Node 结束会话；Node 上报的会话执行结果仍是节点终态的唯一证据。结束请求放出后超过确认窗口仍没有会话终态时，节点以
   `session_end_deadline_exceeded` 失败，运行进入终态并释放 Workspace——不会再有永久 `running` 的运行。
@@ -151,6 +164,7 @@ WorkflowRun 的实际状态更新成功后，在同一事务里将状态、时
 - [工作流由 Cloud 侧调度器执行：WorkflowRun 是执行聚合](https://github.com/ora-space/specs/blob/main/decisions/cloud/workflow/0-cloud-side-scheduler-owns-workflow-execution.md)（approved）
 - [工作流 agent 节点按执行器槽位引用执行器，图里不再存 agent CLI 与模型 ID](https://github.com/ora-space/specs/blob/main/decisions/cloud/workflow/20261015-executor-slots-replace-agent-model-ids.md)（approved）
 - [工作流会话在初始回合结束后由 Cloud 调度器结束，结束请求有界等待](https://github.com/ora-space/specs/blob/main/decisions/cloud/workflow/20261019-workflow-session-end-after-initial-turn.md)
+- [工作流 agent 节点按重试策略重试，恢复接口按最终契约注册但在首期不可用](https://github.com/ora-space/specs/blob/main/decisions/cloud/workflow/20261021-node-retry-and-resume-contract.md)（approved）
 - [插件资产导入产出已发布快照：资产生命周期不改写快照，运行只冻结 snapshotId](https://github.com/ora-space/specs/blob/main/decisions/cloud/workflow/20261016-asset-import-and-published-snapshots.md)
 - [Workflow 插件资产由 Cloud 自行取包、逐文档导入为可编辑副本](https://github.com/ora-space/specs/blob/main/decisions/cloud/plugin-marketplace/20261016-workflow-plugin-assets.md)
 - [个人槽位映射为工作流解析执行器，每次运行按槽位锁定多组模型绑定](https://github.com/ora-space/specs/blob/main/decisions/cloud/model-access/20261015-executor-slot-mappings-and-per-slot-run-bindings.md)（approved）