feat(automation): stamp status: 'stranded' on the resume catch arm and pin the re-armed run's exactly-once — the #13937 services half (shape 4)

.changeset/stranded-run-status-stamp.md

@@ -0,0 +1,63 @@
+---
+"@objectstack/service-automation": minor
+---
+
+feat(automation): a resume that consumed the pause and then failed downstream answers `status: 'stranded'` (#13937)
+
+The services half of the #13937 shape-4 ruling (maintainer 2026-09-01):
+`resumeInternal`'s consumption order is kept — the suspension is consumed
+before downstream nodes run, which is what buys exactly-once across a crash —
+and the state that order leaves behind when a downstream node throws now
+carries the platform-level name #14384 put on the contract.
+
+`AutomationEngine.resume()` (and every engine continuation that reaches the
+same catch arm) returns `{ success: false, status: 'stranded', … }` where it
+returned no `status` at all. Stamped on that one exit only: the pause a
+durable decision was waiting on is gone, the run is recorded `failed`, and it
+can be re-armed only by the explicit operator verb
+`restoreConsumedSuspension` (#13909 slice 2, already published) — never by
+`resume` (which answers `RUN_NOT_FOUND`) and never automatically. Distinct
+from `'failed'` on purpose: that one says the run ran and was rejected; this
+one says a recorded continuation stopped mid-flight and an operator has
+something to repair. The result's verdict and the restore verb are held to
+agree by test: a stranded result is exactly a restorable run.
+
+Not changed: the run's RECORDED status (the run log, `getRun`, `listRuns`, the
+durable `sys_automation_run` history row) stays `failed` — that vocabulary is
+`ExecutionStatus` in `@objectstack/spec`, which the ruling did not widen; the
+durable discriminator for the condition remains the snapshot the terminal row
+carries. No resume semantics move for any pausing node type; shapes 2 and 3
+of the decision stay excluded.
+
+Also in this change, under the same ruling's exactly-once guarantee, two
+repairs to how `restoreConsumedSuspension` finds a stranded run's snapshot:
+
+- The durable run-history row of a stranded run now records the PAUSE node in
+  `node_id`. It recorded the node that threw — the run's last step — and the
+  object store read that column back as the snapshot's node, so a restore
+  from the row (after a restart, or on another replica) re-armed the run at
+  the failed node and the next resume skipped it while reporting the run
+  completed. The throwing node stays in the row's step log and `error`.
+  Visible on the Runs surface: `sys_automation_run`'s row title and highlight
+  set are built from `node_id` (`titleFormat '{flow_name} · {node_id}'`), so a
+  stranded run's row now names the PAUSED node — the one an operator can
+  re-arm — where it named the node that threw; ordinary completed / failed
+  rows are unchanged. The `node_id` and `variables_json` field descriptions
+  carry this carve-out, the way `node_type`'s already did.
+- The verb reads the durable row and its own per-process journal as two
+  witnesses of one strand instead of trusting either alone. The hot copy is
+  preferred when both describe the same pause (it is the verbatim object the
+  failure was journalled from). A row that carries no snapshot is read as
+  "the run moved on" only when this process's own history write landed —
+  the replica that stranded a run used to keep a hot copy that could re-arm
+  the run after another replica had restored, resumed and finished it, and
+  the next resume re-ran every node after the pause. A snapshot the object
+  store could not persist (over its 256 KiB row budget) is now recorded in
+  the row as dropped, with the pause it belonged to, so the replica holding
+  the hot copy still restores and any other replica is refused with a reason
+  that names the budget and the remedy.
+
+In-memory and store-less deployments observe no behaviour difference. On the
+object store, same-replica restores re-arm the pause node on every path, and
+restores from the row alone do too; restores across replicas of a run that
+finished elsewhere are refused.

packages/services/service-automation/src/consumed-suspension-restore.test.ts

@@ -14,10 +14,13 @@
  * the REST run surface has no cancel or retry route, and none of the engine's
  * public methods moved such a run out.
  *
- * ⛔ **This file changes nothing about that ordering.** Which ordering is right
- * is #13937, unruled and in the maintainer's hands. What is pinned here is the
- * EXIT: `restoreConsumedSuspension` puts the consumed suspension back so the
- * run is resumable again, under the ordering exactly as it is.
+ * ⛔ **This file changes nothing about that ordering.** #13937 ruled it (shape
+ * 4, maintainer 2026-09-01): the order stays, for exactly-once across a crash,
+ * and this verb is the operator exit. What is pinned here is that EXIT:
+ * `restoreConsumedSuspension` puts the consumed suspension back so the run is
+ * resumable again, under the ordering exactly as it is. The NAME the ruling
+ * gave the state (`AutomationResult.status: 'stranded'`) is pinned beside it
+ * in `stranded-run-status.test.ts`.
  *
  * ## What is pinned, and why each one is here
  *
@@ -554,17 +557,19 @@ describe('#13909 — across a restart: the deployment shape this exists for', ()
     });
 });
 
-describe('#13909 — what this slice deliberately does NOT do', () => {
-    it('leaves AutomationResult.status and the run\'s recorded status alone', async () => {
+describe('#13909 — what this verb deliberately does NOT do', () => {
+    it('names the condition on the RESULT only — the run\'s recorded status stays failed', async () => {
         const { engine } = newEngine(new InMemorySuspendedRunStore());
         const started = await engine.execute('strand_flow', ctx);
         const runId = started.runId as string;
         const failed = await engine.resume(runId);
 
-        // ⛔ No new platform status is minted for the condition — naming it is
-        // an explicit same-batch sub-item of #13937, because what it should be
-        // called depends on which resume-ordering shape is ruled.
-        expect(failed.status).toBeUndefined();
+        // The #13937 shape-4 ruling put the name on `AutomationResult.status`
+        // (#14384) and the catch arm now stamps it (the producer half, pinned
+        // in full in `stranded-run-status.test.ts`). It did NOT widen
+        // `ExecutionStatus`: the run's recorded lifecycle stays `failed`, in
+        // the log and in the durable history row.
+        expect(failed.status).toBe('stranded');
         expect((await engine.getRun(runId))?.status).toBe('failed');
     });
 
@@ -574,10 +579,11 @@ describe('#13909 — what this slice deliberately does NOT do', () => {
         const runId = started.runId as string;
 
         // The consumption still precedes the traversal: measured from inside
-        // the downstream node, the suspension is already gone. If a future
-        // change made the pause survive a downstream throw (#13937 shape 2),
-        // THIS is the assertion that should be reconsidered — deliberately, not
-        // by accident.
+        // the downstream node, the suspension is already gone. #13937 ruled
+        // this order KEPT (shape 4); shape 2 — the pause surviving a downstream
+        // throw — reopens only together with a durable claim/lease that keeps
+        // exactly-once, and THIS is the assertion such a change must flip
+        // deliberately, in the same batch, not by accident.
         let suspendedDuringTraversal: boolean | undefined;
         registerDownstream(engine, {
             throws: true,