fix(plugin-approvals, service-automation): report a repairable strand, a dropped snapshot and a cascade-failed run apart — through a read-only engine member, not the wire (#15358 B′)

.changeset/stranded-run-state-discriminator.md

@@ -0,0 +1,17 @@
+---
+"@objectstack/service-automation": minor
+"@objectstack/plugin-approvals": minor
+---
+
+The stranded-request inspection tells a repairable strand from a cascade-failed run — through a dedicated read-only engine member, not through the wire (#15358, ruling B′).
+
+`ApprovalService.inspectStrandedRequests` keyed on `run.status === 'failed'`, which over-reports in one direction: a **cascade-failed** run — an ancestor `failAncestors` failed while it was parked at its `subflow` node, whose pause `failSuspendedRun` consumed and journalled nothing — has the same terminal `failed` row as the #13909 strand, so both came back `runState: 'failed'`, and `restoreConsumedSuspension` re-arms one and refuses the other (`NO_CONSUMED_SUSPENSION`). The engine's discriminator (the consumed-suspension snapshot on the durable `RunRecord`) is deliberately NOT on the `ExecutionLogEntry` that `getRun` answers, because `GET /automation/:name/runs/:runId` serves that object verbatim — so the plugin could not read it, and reading its absence as "not a strand" would have called the repairable row dead.
+
+**`@objectstack/service-automation` — additive, `minor`.** `AutomationEngine.inspectConsumedSuspension(runId)` answers whether `restoreConsumedSuspension` would have a consumed suspension to put back, from the SAME two witnesses that verb reads (this process's hot journal and the durable row, reconciled by the same `rowSupersedesJournal` / `persisted` / drop-notice rules — the read is now one private method both call), and re-arms nothing. Four answers, none folded: `repairable: true` (with the pause it would re-arm and which witness answered); `SNAPSHOT_DROPPED` (the strand happened, the store could not persist the snapshot, and this process holds no hot copy — repairable only by the replica that stranded it, while it lives); `NO_CONSUMED_SUSPENSION` (cascade-failed or never paused); `RUN_SUSPENDED` (already resumable). It REJECTS when a store cannot be read — an outage is unknown, not "nothing to restore". The result type is exported as `ConsumedSuspensionInspection`. `restoreConsumedSuspension` behaves exactly as before; nothing on `ExecutionLogEntry`, the run-detail route, or `@objectstack/spec` changes.
+
+**`@objectstack/plugin-approvals` — additive on two published types, `minor`.**
+
+- `ApprovalResumeSurface` gains the optional `inspectConsumedSuspension?(runId)`, declared the way `listSuspendedRunsDurable` is: a method `AutomationEngine` already implements, widening no engine surface.
+- `StrandedRunState` splits `'failed'` three ways and keeps `'missing'` untouched: `'repairable'` (the #13909 strand — restore, then `continueRestoredRun`), `'snapshot_dropped'` (its own class: as `repairable` it over-reports, as `unrepairable` it is a false negative), and `'unrepairable'` (the cascade-failed / never-paused run, #15222's shape — nothing re-arms it). **`'failed'` stays a member, on purpose**: it is what a `failed` row is reported as when the attached surface has no `inspectConsumedSuspension` (an engine build older than this plugin, or a host double). Absence of the discriminator is fail-closed for a report — the row is reported, undifferentiated, never labelled `unrepairable` and never dropped. A thrown read counts `undetermined`, as the other two oracles' do.
+
+A consumer switching exhaustively over `StrandedRunState` gains three arms; nothing it matched before stops arriving. The inspection's summary log adds `runRepairable` / `runSnapshotDropped` / `runUnrepairable` beside the existing counts.

packages/plugins/plugin-approvals/src/approval-service.ts

@@ -182,8 +182,47 @@ export interface ApprovalResumeSurface {
    * pause it cannot see is a pause this verb will not act on.
    */
   listSuspendedRunsDurable?(): Promise<Array<{ runId: string; flowName: string; nodeId: string; correlation?: string }>>;
+  /**
+   * [#15358] Whether the consumed suspension behind a `failed` run SURVIVES —
+   * i.e. whether the engine's `restoreConsumedSuspension` would re-arm it.
+   * Read by {@link ApprovalService.inspectStrandedRequests}, for `failed` rows
+   * only, to tell the #13909 strand (repairable) from a cascade-failed
+   * ancestor — the `failAncestors` shape the engine itself calls NOT stranded,
+   * because `failSuspendedRun` consumed the ancestor's pause and journalled
+   * nothing, so no verb re-arms it (#15222's shape). {@link getRun} answers
+   * `status: 'failed'` for both: the discriminator is deliberately NOT on
+   * `ExecutionLogEntry`, which `GET /automation/:name/runs/:runId` serves
+   * verbatim. Ruled B′ on #15358 (2026-09-07): it is published as a dedicated
+   * read-only engine member, never on the wire.
+   *
+   * ⚠️ Declares a method `AutomationEngine` ALREADY implements publicly
+   * (`inspectConsumedSuspension`); it widens no engine surface. It answers
+   * from the same two witnesses the restore verb reads — this process's hot
+   * journal and the durable row — so what it calls repairable IS what that
+   * verb restores. The three negatives are distinct on purpose (see
+   * {@link StrandedRunState}); the middle one, `'SNAPSHOT_DROPPED'`, is a
+   * strand the store could not persist, repairable only by the process that
+   * stranded it while that process lives.
+   *
+   * Optional, and its absence is FAIL-CLOSED for a report: a `failed` row the
+   * engine cannot be asked about is reported as today's undifferentiated
+   * `'failed'` — ⛔ never as `'unrepairable'` (that calls the repairable row
+   * dead: the #15555 false-negative harm, one surface over) and ⛔ never
+   * skipped (that hides the row). Absence of the discriminator is not
+   * evidence of anything. Rejects when a store cannot be read; the inspection
+   * counts such a row `undetermined`, exactly as it does a thrown
+   * {@link hasSuspendedRun}.
+   */
+  inspectConsumedSuspension?(runId: string): Promise<
+    | { repairable: true }
+    | { repairable: false; reason: 'RUN_SUSPENDED' | 'SNAPSHOT_DROPPED' | 'NO_CONSUMED_SUSPENSION' }
+  >;
 }
 
+/** What {@link ApprovalResumeSurface.inspectConsumedSuspension} answers. */
+type ConsumedSuspensionVerdict =
+  Awaited<ReturnType<NonNullable<ApprovalResumeSurface['inspectConsumedSuspension']>>>;
+
 /**
  * Optional messaging surface (ADR-0012 `messaging` service). When attached,
  * thread interactions (reassign / remind / request-info / comment) notify the
@@ -349,7 +388,12 @@ function classifyStrandedRunState(run: { status?: string } | null | undefined):
   // No history row at all — the #4469 shape this inspection was built for.
   if (!run) return 'missing';
   switch (run.status) {
-    // The resume consumed the pause and a downstream node threw. Reported.
+    // A terminal `failed` row. Reported — and, when the engine can be asked,
+    // refined by the third oracle ({@link refineFailedRunState}) into which of
+    // the three `failed` shapes it is. This arm alone cannot tell them apart:
+    // the row reads identically for a resume that consumed the pause and
+    // threw downstream (repairable) and for an ancestor `failAncestors`
+    // cascade-failed (not). `'failed'` here means "reported, undifferentiated".
     case 'failed':
       return 'failed';
 
@@ -377,26 +421,94 @@ function classifyStrandedRunState(run: { status?: string } | null | undefined):
 }
 
 /**
- * WHY a terminal request's run is unrecoverable — the two shapes the inspection
- * reports, which have different causes and different remedies (#13909).
+ * [#15358] The third oracle's verdict over a `failed` row — which of the three
+ * differentiated shapes it is, read from the engine's own consumed-suspension
+ * witnesses — or `undefined` for the one answer that means the row is not
+ * stranded after all.
+ *
+ * ⛔ Absence of the discriminator is NOT an input here, on purpose: this runs
+ * only when the engine answered. A surface without the member never reaches
+ * it and the row stays `'failed'` (see {@link StrandedRunState}) — reading
+ * "the engine could not be asked" as "not a strand" would call the repairable
+ * row dead, which is #15555's false negative one surface over.
+ */
+function refineFailedRunState(verdict: ConsumedSuspensionVerdict): StrandedRunState | undefined {
+  if (verdict.repairable) return 'repairable';
+  switch (verdict.reason) {
+    // The strand happened; the store could not keep the snapshot, and the
+    // engine asked holds no hot copy. Its own class — see the type below.
+    case 'SNAPSHOT_DROPPED':
+      return 'snapshot_dropped';
+    // Neither witness holds anything: cascade-failed, or never paused.
+    case 'NO_CONSUMED_SUSPENSION':
+      return 'unrepairable';
+    // Re-armed between the two reads (an operator's restore landed while this
+    // scan was running): the run is alive and resumable, which is what the
+    // first oracle would have said a moment later. Not stranded.
+    case 'RUN_SUSPENDED':
+      return undefined;
+    // An answer this build does not know (an engine ahead of this plugin).
+    // Fail-closed exactly as an absent member: reported, undifferentiated —
+    // never condemned on a word this code cannot read.
+    default:
+      return 'failed';
+  }
+}
+
+/**
+ * WHY a terminal request's run is unrecoverable — the shapes the inspection
+ * reports, which have different causes and different remedies (#13909;
+ * split three ways by the #15358 B′ ruling, 2026-09-07).
  *
  *  - `missing` — `getRun` finds no history row at all (#4469's original shape):
  *    the run was lost before it could record anything, typically a pause that
  *    never reached a durable store and did not survive a restart.
- *  - `failed` — the run DID record a terminal `failed` row. The engine consumes
- *    a suspension *before* running the downstream nodes
- *    (`AutomationEngine.resumeInternal`: `forgetSuspendedRun(run, 'resumed')`
- *    precedes `traverseNext`), so a downstream node that merely THREW threw with
- *    the pause already gone — the catch arm recorded `failed` and there is no
- *    suspension left to resume. The decision is durable, the flow stopped
- *    mid-continuation, and no verb moves the run out of that state.
+ *
+ * The other four all describe a run that DID record a terminal `failed` row.
+ * The engine consumes a suspension *before* running the downstream nodes
+ * (`AutomationEngine.resumeInternal`: `forgetSuspendedRun(run, 'resumed')`
+ * precedes `traverseNext`), so a downstream node that merely THREW threw with
+ * the pause already gone — the catch arm recorded `failed` and there is no
+ * suspension left to resume. The decision is durable and the flow stopped
+ * mid-continuation. What differs is whether anything can put the pause back,
+ * and `status` alone cannot say — the three differentiated members come from
+ * the engine's own consumed-suspension witnesses
+ * ({@link ApprovalResumeSurface.inspectConsumedSuspension}):
+ *
+ *  - `repairable` — the engine holds the consumed suspension (in its journal
+ *    or on the durable row): the #13909 strand. `restoreConsumedSuspension`
+ *    re-arms it and {@link ApprovalService.continueRestoredRun} re-issues the
+ *    decision. The remedy is a repair.
+ *  - `snapshot_dropped` — the run DID strand, but the store could not persist
+ *    the snapshot (over its row budget) and the engine asked holds no hot
+ *    copy. Repairable only by the process that stranded it, while that process
+ *    lives; the restore verb refuses it elsewhere naming the budget. ⛔ Not
+ *    folded into either neighbour: as `repairable` it over-reports, as
+ *    `unrepairable` it is #15555's false negative. Reported as what it is.
+ *  - `unrepairable` — the engine holds NO consumed suspension: the run was
+ *    cascade-failed (`failAncestors` → `failSuspendedRun`, which consumes the
+ *    ancestor's pause and journals nothing — #15222's shape, the one the
+ *    engine's own words call not a strand), never paused at all, or did
+ *    strand and its snapshot is no longer held (the journal evicted a copy
+ *    whose write never landed; the run was restored and then finished; a
+ *    store class without `loadTerminal`, after a restart) — the engine's
+ *    `NO_CONSUMED_SUSPENSION` covers all three and does not say which. The
+ *    label is faithful to the verb: nothing re-arms it; the remedy is a new
+ *    run, not a restore.
+ *  - `failed` — the engine COULD NOT BE ASKED which of the three it is: the
+ *    attached surface has no `inspectConsumedSuspension` (an engine build
+ *    older than this plugin, or a test double). Today's undifferentiated
+ *    label, kept on purpose as the fail-closed fallback (#15358 ruling, item
+ *    1): absence of the discriminator is not evidence, so the row is reported
+ *    and its repairability left unstated — ⛔ never `unrepairable`, ⛔ never
+ *    dropped from the report.
  *
  * ⚠️ This names the shapes for the REPORT only. It is not a run state: the
  * engine's own vocabulary is still `'completed' | 'paused' | 'failed'`
  * (`AutomationResult.status`) and nothing persists or queries "stranded".
  * Giving the condition a platform-level name is #13909's own deliverable.
  */
-export type StrandedRunState = 'missing' | 'failed';
+export type StrandedRunState = 'missing' | 'failed' | 'repairable' | 'snapshot_dropped' | 'unrepairable';
 
 /**
  * The continuation an approvals door already issued once, kept so it can be
@@ -464,14 +576,18 @@ export interface StrandedApprovalRequest {
   /**
    * The `flow_run_id` that resolves to no live suspension and no recoverable
    * run — see `runState`: no history row at all (`missing`), or a terminal
-   * `failed` row (`failed`).
+   * `failed` row (`repairable` / `snapshot_dropped` / `unrepairable`, or
+   * `failed` when the engine could not say which).
    */
   runId: string;
   /**
    * Which unrecoverable shape this is — see {@link StrandedRunState}. Carried
-   * because the two need different remedies: a `missing` run has no history to
-   * read, while a `failed` one has a step log and an error message naming the
-   * node that threw.
+   * because the shapes need different remedies: a `missing` run has no history
+   * to read; a `repairable` one is put back by `restoreConsumedSuspension` and
+   * continued by {@link ApprovalService.continueRestoredRun}; an
+   * `unrepairable` one (#15222's cascade-failed ancestor) is refused by that
+   * verb and needs a new run. An operator reading `'failed'` has to ask the
+   * engine directly.
    */
   runState: StrandedRunState;
   flowName?: string;
@@ -4263,8 +4379,22 @@ export class ApprovalService implements IApprovalService {
    * ⚠️ The widening does NOT reverse the conservatism: `completed`, `cancelled`
    * and `paused` are each still skipped, for reasons named one at a time in
    * `classifyStrandedRunState`, and an unrecognised status is skipped too.
-   * What the widening buys is that a `failed` run is now reported with
-   * `runState: 'failed'` instead of counted as healthy.
+   * What the widening buys is that a `failed` run is now reported instead of
+   * counted as healthy.
+   *
+   * **A THIRD oracle tells the `failed` rows apart (#15358).** `status ===
+   * 'failed'` over-reports in one specific direction: a cascade-failed run
+   * — an ancestor `failAncestors` failed while it was parked at its `subflow`
+   * node, whose pause `failSuspendedRun` consumed and journalled nothing — has
+   * the same terminal row as the #13909 strand, and `restoreConsumedSuspension`
+   * refuses it. The engine's discriminator (the consumed-suspension snapshot)
+   * is deliberately NOT on the object `getRun` answers, so it is asked through
+   * a dedicated read-only member, `inspectConsumedSuspension`, and only for
+   * `failed` rows: the answer splits `'failed'` into `'repairable'`,
+   * `'snapshot_dropped'` and `'unrepairable'` (see {@link StrandedRunState}).
+   * A surface without that member leaves the row `'failed'` — reported,
+   * undifferentiated — because absence of the discriminator is not evidence
+   * of anything; a thrown read counts `undetermined`, like the other two.
    *
    * ⚠️ **What this can and cannot size.** It makes the condition *visible* in a
    * deployment; it is not itself a census, and it says nothing about this
@@ -4339,12 +4469,36 @@ export class ApprovalService implements IApprovalService {
       // #13909 — the widened verdict. `undefined` means "not a shape this
       // reports": healthy, deliberate, or unresolvable. See
       // `classifyStrandedRunState` for which, and why each one.
-      const runState = classifyStrandedRunState(terminal);
+      let runState = classifyStrandedRunState(terminal);
       if (!runState) continue;
 
+      // #15358 — the third oracle, for `failed` rows only: does the engine hold
+      // the consumed suspension a restore would put back? Asked through the
+      // dedicated read-only member, never inferred from the row. Absence of
+      // the member is FAIL-CLOSED for a report — the row stays `'failed'`,
+      // reported and undifferentiated; a thrown read is `undetermined`, as for
+      // the other two oracles. See `refineFailedRunState` and
+      // `StrandedRunState` for the three answers and why none is folded.
+      if (runState === 'failed' && typeof this.automation.inspectConsumedSuspension === 'function') {
+        let verdict: ConsumedSuspensionVerdict;
+        try {
+          verdict = await this.automation.inspectConsumedSuspension(runId);
+        } catch (err: any) {
+          undetermined++;
+          this.logger?.warn?.('[approvals] stranded-request scan could not read the consumed-suspension state', {
+            request: raw?.id, run: runId, error: err?.message ?? String(err),
+          });
+          continue;
+        }
+        const refined = refineFailedRunState(verdict);
+        if (!refined) continue;   // re-armed between the two reads — alive after all
+        runState = refined;
+      }
+
       // Neither suspended nor recoverable: the run this decision was supposed to
       // advance is gone (`missing`) or terminally failed mid-continuation with
-      // its pause already consumed (`failed`).
+      // its pause already consumed (`repairable` / `snapshot_dropped` /
+      // `unrepairable` — or `failed`, when the engine could not say which).
       const config = parseJson<ApprovalNodeConfig>(
         raw.node_config_json, { approvers: [], behavior: 'first_response' } as any,
       );
@@ -4383,6 +4537,9 @@ export class ApprovalService implements IApprovalService {
         scanned: rows.length, stranded: stranded.length, undetermined,
         runMissing: stranded.filter(s => s.runState === 'missing').length,
         runFailed: stranded.filter(s => s.runState === 'failed').length,
+        runRepairable: stranded.filter(s => s.runState === 'repairable').length,
+        runSnapshotDropped: stranded.filter(s => s.runState === 'snapshot_dropped').length,
+        runUnrepairable: stranded.filter(s => s.runState === 'unrepairable').length,
         requests: stranded.map(s => `${s.requestId}@${s.nodeId ?? '?'} → run ${s.runId} (${s.runState})`),
       });
     }