Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
71 changes: 71 additions & 0 deletions .changeset/execution-step-metrics-failure-slot.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
---
'@objectstack/spec': minor
---

feat(spec): `ExecutionStepMetrics` gains an optional `failures` slot, and `FlowRunSummary.failed` is declared as the fold INCLUDING what a delegating node rolled up from its child (maintainer ruling 2026-09-06 on #15617, spec half)

Additive. Nothing an author writes is renamed, retired or narrowed; no accept
set shrinks. One optional key is declared on a runtime-produced schema and the
prose of a published contract is reconciled with itself.

**What was wrong.** `FlowRunSummary` said two things about `failed`. Its
header paragraph declared that a `subflow` node rolls its child run's totals
up into the parent — "this summary answers *what did this run cause*" — while
the field itself declared `failed = Σ nodes[].failures`, a fold over the
parent's own node executions. For a parent that delegates its rows to a
`subflow` (or a `map` item) those give different answers, and the engine could
only satisfy the second one: `ExecutionStepMetrics` carried `selected` /
`acted` / `unmeasuredEffect` and no failure slot, so a child's contained
failures had no path into the parent's fold. Measured on the real engine by
the services seat (#15617): parent `loop { subflow(child) }` → parent
`failed=0` while the five child summaries carried `failed=[0,0,0,0,1]` —
`acted` rolled up, `failed` did not.

**What this declares.**

- `ExecutionStepMetrics.failures` (optional, integer ≥ 0): node executions
that failed inside a child run this execution delegated to and went on from
— a `subflow` child or a `map` item whose run COMPLETED while containing
failures, i.e. the child's `summary.failed`, rolled up. It folds into the
delegating node's `nodes[].failures` and so into the run-level `failed` —
the same fold shape `acted` has, but not the same rule at the failed-child
boundary (next bullet). Absent means the step delegated nothing, or its
child tracked no count, or the producer did not track it (every step the
engine emits between this release and the engine half) — never zero.
- It is NOT the step's own outcome. A step that failed is `status: 'failure'`
and counts once through `nodes[].failures`, as before; a child that FAILED
— whether or not it also contained failures before it failed — is
precisely that step failure: its own `failed`, contained and fatal alike,
stays on the child's run row and nothing rides up, so one failure is never
counted twice. This is where the rule parts from `acted`, which does carry
a failed child's writes up to the parent. The control the card measured (a
failing child → parent `failed=1`) keeps counting exactly as today.
- `FlowRunSummary.failed` is declared, at the field, as the fold of
`nodes[].failures` INCLUDING what a delegating node rolled up; the
`FlowRunNodeSummary.failures` describe names the roll-up path, and its
`status` describe states that a delegating node whose child contained
failures reads `success` beside `failures > 0` — status is judged on the
node's own executions.

**What this does not do yet.** This is the contract half of a two-lane
landing (contract first). No producer populates `failures` in this release:
`subflow-node.ts` and the `map` node roll the child's contained failures into
the slot in the services half, #16314, and only then does a parent's
`failed` start counting them. Until that lands, every `ExecutionStepMetrics`
the engine emits is byte-identical to today's, `failed` is numerically what it
was, and the flow-run reference page keeps the narrowed wording PR #15609
shipped ("node executions **of this run**") on purpose — it is widened when
both halves are in.

**Consumers.** A reader of `ExecutionStepMetrics` sees one more optional
number and nothing else changes shape; a consumer that already sums
`nodes[].failures` to cross-check `failed` keeps agreeing with it, because the
fold is unchanged — the roll-up enters the per-node array, not beside it. Two
consequences of that placement are part of the contract from this release,
even though no producer populates the slot yet: on a delegating node
`nodes[].failures` may exceed `runs` (`runs: 5, failures: 15` is a legal
shape — five subflow executions whose children each contained three), and it
is no longer only that node's own failed executions, so a reader that derived
"this node's executions that failed" or a failure RATE from `failures / runs`
must read a delegating node's number as "failures this node caused, its
child's contained ones included".
18 changes: 10 additions & 8 deletions content/docs/references/automation/execution.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -144,7 +144,7 @@ const result = CheckpointSchema.parse(data);
| **iteration** | `integer` | optional | Zero-based iteration of the enclosing `loop`, carried through any nesting — a step inside a `parallel` branch that is itself inside a loop body carries the loop's iteration here and its branch index on `branch`. A step inside a `try` / `catch` region that is itself inside a loop body carries the enclosing loop's iteration — a try/catch region has no index of its own — while `regionKind` stays `try` / `catch`. |
| **branch** | `integer` | optional | Zero-based index of the enclosing `parallel` branch. Present only on a step inside a parallel branch; absent everywhere else. When the parallel node is itself inside a loop body, the loop iteration is reported through `iteration`, never here. |
| **regionKind** | `string` | optional | Region kind the step ran in: loop-body \| parallel-branch \| try \| catch. Stays `try` / `catch` for a step inside a try/catch region nested in a loop body; the loop is reported through `iteration`. For `parallel-branch` the branch index is reported through `branch`, and the enclosing loop iteration — when the parallel node sits inside a loop body — through `iteration`. |
| **metrics** | `{ selected?: integer; acted?: integer; unmeasuredEffect?: boolean }` | optional | Records this step selected / acted on, as reported by the node executor |
| **metrics** | `{ selected?: integer; acted?: integer; unmeasuredEffect?: boolean; failures?: integer }` | optional | Records this step selected / acted on — and, for a step that delegated to a child run (`subflow`, a `map` item), the failures that child contained — as reported by the node executor |
| **skippedBy** | `{ nodeId: string; edgeId?: string; label?: string }` | optional | The gate that closed, when `status` is `skipped` |

### Nested Shape: `ExecutionLog.summary`
Expand All @@ -155,7 +155,7 @@ const result = CheckpointSchema.parse(data);
| **acted** | `integer` | ✅ | Total records written / effects dispatched by the run |
| **skipped** | `integer` | ✅ | Total node executions a closed gate prevented |
| **unmeasured** | `integer` | optional | Total executions that may have caused an effect the platform cannot count. Absent = not tracked (an older run), which is not the same as zero. |
| **failed** | `integer` | optional | Total node executions that failed — a fold of `nodes[].failures`. On a run that completed every one of them was contained (caught by a `try_catch` or routed down a `fault` edge) and the run went on. Absent = not tracked (an older run), which is not the same as zero. |
| **failed** | `integer` | optional | Total node executions that failed — a fold of `nodes[].failures`, INCLUDING what a delegating node (`subflow` / `map`) rolled up from a child run that COMPLETED while containing failures: this total answers "what did this run cause", subflows included, so a parent whose child lost rows does not read `failed: 0`. A child that FAILED — whether or not it also contained failures before it failed — counts once, as the delegating step's own failure, and its own `failed` stays on its row (unlike `acted`, which carries a failed child's writes). On a run that completed every one of them was contained (caught by a `try_catch` or routed down a `fault` edge) and the run went on. Absent = not tracked (an older run), which is not the same as zero. |
| **nodes** | `{ nodeId: string; nodeType: string; nodeLabel?: string; status: Enum<'success' \| 'failure' \| 'skipped'>; … }[]` | ✅ | Per-node breakdown, in first-execution order |
| **gates** | `{ nodeId: string; targetNodeId: string; edgeId?: string; label?: string; … }[]` | ✅ | Gates that closed during the run, most-skipped first |
| **detailOmitted** | `boolean` | optional | Set when persistence dropped `nodes`/`gates` to keep the stored row bounded — the totals are still exact. Declared so empty arrays are never mistaken for "nothing ran". |
Expand Down Expand Up @@ -201,7 +201,7 @@ const result = CheckpointSchema.parse(data);
| **iteration** | `integer` | optional | Zero-based iteration of the enclosing `loop`, carried through any nesting — a step inside a `parallel` branch that is itself inside a loop body carries the loop's iteration here and its branch index on `branch`. A step inside a `try` / `catch` region that is itself inside a loop body carries the enclosing loop's iteration — a try/catch region has no index of its own — while `regionKind` stays `try` / `catch`. |
| **branch** | `integer` | optional | Zero-based index of the enclosing `parallel` branch. Present only on a step inside a parallel branch; absent everywhere else. When the parallel node is itself inside a loop body, the loop iteration is reported through `iteration`, never here. |
| **regionKind** | `string` | optional | Region kind the step ran in: loop-body \| parallel-branch \| try \| catch. Stays `try` / `catch` for a step inside a try/catch region nested in a loop body; the loop is reported through `iteration`. For `parallel-branch` the branch index is reported through `branch`, and the enclosing loop iteration — when the parallel node sits inside a loop body — through `iteration`. |
| **metrics** | `{ selected?: integer; acted?: integer; unmeasuredEffect?: boolean }` | optional | Records this step selected / acted on, as reported by the node executor |
| **metrics** | `{ selected?: integer; acted?: integer; unmeasuredEffect?: boolean; failures?: integer }` | optional | Records this step selected / acted on — and, for a step that delegated to a child run (`subflow`, a `map` item), the failures that child contained — as reported by the node executor |
| **skippedBy** | `{ nodeId: string; edgeId?: string; label?: string }` | optional | The gate that closed, when `status` is `skipped` |

### Nested Shape: `ExecutionStepLog.error`
Expand All @@ -219,6 +219,7 @@ const result = CheckpointSchema.parse(data);
| **selected** | `integer` | optional | Records this node READ or matched (a `get_record` query, a lookup) |
| **acted** | `integer` | optional | Records this node WROTE (created / updated / deleted) or effects it dispatched (notifications delivered) |
| **unmeasuredEffect** | `boolean` | optional | This execution may have caused an effect the platform cannot count (an external write through a connector). NOT interchangeable with `acted: 0` — it says the count is unknown, not that it is zero. |
| **failures** | `integer` | optional | Node executions that failed inside a child run this execution delegated to and went on from — a `subflow` child or a `map` item whose run COMPLETED while containing failures: its `summary.failed`, rolled up so the parent answers "what did this run cause". Folds into this node's `failures` and so into the run-level `failed`. NOT this execution's own outcome: a step that failed is `status: 'failure'` and counts once through `nodes[].failures`, and a child that FAILED — whether or not it also contained failures before it failed — is exactly that step failure: its own `failed`, contained and fatal alike, stays on the child's run row and nothing rides up here (unlike `acted`, which does carry a failed child's writes). Absent = delegated nothing, or the child tracked no count, or the producer did not track it; never zero. |

### Nested Shape: `ExecutionStepLog.skippedBy`

Expand All @@ -240,6 +241,7 @@ const result = CheckpointSchema.parse(data);
| **selected** | `integer` | optional | Records this node READ or matched (a `get_record` query, a lookup) |
| **acted** | `integer` | optional | Records this node WROTE (created / updated / deleted) or effects it dispatched (notifications delivered) |
| **unmeasuredEffect** | `boolean` | optional | This execution may have caused an effect the platform cannot count (an external write through a connector). NOT interchangeable with `acted: 0` — it says the count is unknown, not that it is zero. |
| **failures** | `integer` | optional | Node executions that failed inside a child run this execution delegated to and went on from — a `subflow` child or a `map` item whose run COMPLETED while containing failures: its `summary.failed`, rolled up so the parent answers "what did this run cause". Folds into this node's `failures` and so into the run-level `failed`. NOT this execution's own outcome: a step that failed is `status: 'failure'` and counts once through `nodes[].failures`, and a child that FAILED — whether or not it also contained failures before it failed — is exactly that step failure: its own `failed`, contained and fatal alike, stays on the child's run row and nothing rides up here (unlike `acted`, which does carry a failed child's writes). Absent = delegated nothing, or the child tracked no count, or the producer did not track it; never zero. |


---
Expand Down Expand Up @@ -281,9 +283,9 @@ const result = CheckpointSchema.parse(data);
| **nodeId** | `string` | ✅ | Node ID |
| **nodeType** | `string` | ✅ | Node action type (e.g., "get_record", "decision") |
| **nodeLabel** | `string` | optional | Human-readable node label |
| **status** | `Enum<'success' \| 'failure' \| 'skipped'>` | ✅ | Terminal status of the node across the run — `failure` if any execution failed, else `success` if any succeeded, else `skipped` |
| **status** | `Enum<'success' \| 'failure' \| 'skipped'>` | ✅ | Terminal status of the node across the run — `failure` if any execution failed, else `success` if any succeeded, else `skipped`. Judged on this node's OWN executions: a delegating node (`subflow` / `map`) whose child completed while containing failures reads `success` here with `failures > 0` |
| **runs** | `integer` | ✅ | Times the node executed (loop iterations and parallel branches each count) |
| **failures** | `integer` | ✅ | Executions that failed — a failure a `try_catch` caught or a `fault` edge routed counts here too; the run-level `failed` is the sum of this across `nodes` |
| **failures** | `integer` | ✅ | Executions that failed — a failure a `try_catch` caught or a `fault` edge routed counts here too — plus what a delegating execution rolled up from a child run that COMPLETED (`metrics.failures`: the contained failures of a `subflow` child or a `map` item). On a delegating node this may therefore exceed `runs` and is no longer only this node's own failed executions; a child that FAILED adds only the step's own failure (unlike `acted`, which carries a failed child's writes too); the run-level `failed` is the sum of this across `nodes` |
| **skipped** | `integer` | ✅ | Times a closed gate kept this node from running at all |
| **selected** | `integer` | optional | Records read across every execution — omitted for a node that reads none |
| **acted** | `integer` | optional | Records written / effects dispatched across every execution — omitted for a node that writes none |
Expand All @@ -302,7 +304,7 @@ const result = CheckpointSchema.parse(data);
| **acted** | `integer` | ✅ | Total records written / effects dispatched by the run |
| **skipped** | `integer` | ✅ | Total node executions a closed gate prevented |
| **unmeasured** | `integer` | optional | Total executions that may have caused an effect the platform cannot count. Absent = not tracked (an older run), which is not the same as zero. |
| **failed** | `integer` | optional | Total node executions that failed — a fold of `nodes[].failures`. On a run that completed every one of them was contained (caught by a `try_catch` or routed down a `fault` edge) and the run went on. Absent = not tracked (an older run), which is not the same as zero. |
| **failed** | `integer` | optional | Total node executions that failed — a fold of `nodes[].failures`, INCLUDING what a delegating node (`subflow` / `map`) rolled up from a child run that COMPLETED while containing failures: this total answers "what did this run cause", subflows included, so a parent whose child lost rows does not read `failed: 0`. A child that FAILED — whether or not it also contained failures before it failed — counts once, as the delegating step's own failure, and its own `failed` stays on its row (unlike `acted`, which carries a failed child's writes). On a run that completed every one of them was contained (caught by a `try_catch` or routed down a `fault` edge) and the run went on. Absent = not tracked (an older run), which is not the same as zero. |
| **nodes** | `{ nodeId: string; nodeType: string; nodeLabel?: string; status: Enum<'success' \| 'failure' \| 'skipped'>; … }[]` | ✅ | Per-node breakdown, in first-execution order |
| **gates** | `{ nodeId: string; targetNodeId: string; edgeId?: string; label?: string; … }[]` | ✅ | Gates that closed during the run, most-skipped first |
| **detailOmitted** | `boolean` | optional | Set when persistence dropped `nodes`/`gates` to keep the stored row bounded — the totals are still exact. Declared so empty arrays are never mistaken for "nothing ran". |
Expand All @@ -314,9 +316,9 @@ const result = CheckpointSchema.parse(data);
| **nodeId** | `string` | ✅ | Node ID |
| **nodeType** | `string` | ✅ | Node action type (e.g., "get_record", "decision") |
| **nodeLabel** | `string` | optional | Human-readable node label |
| **status** | `Enum<'success' \| 'failure' \| 'skipped'>` | ✅ | Terminal status of the node across the run — `failure` if any execution failed, else `success` if any succeeded, else `skipped` |
| **status** | `Enum<'success' \| 'failure' \| 'skipped'>` | ✅ | Terminal status of the node across the run — `failure` if any execution failed, else `success` if any succeeded, else `skipped`. Judged on this node's OWN executions: a delegating node (`subflow` / `map`) whose child completed while containing failures reads `success` here with `failures > 0` |
| **runs** | `integer` | ✅ | Times the node executed (loop iterations and parallel branches each count) |
| **failures** | `integer` | ✅ | Executions that failed — a failure a `try_catch` caught or a `fault` edge routed counts here too; the run-level `failed` is the sum of this across `nodes` |
| **failures** | `integer` | ✅ | Executions that failed — a failure a `try_catch` caught or a `fault` edge routed counts here too — plus what a delegating execution rolled up from a child run that COMPLETED (`metrics.failures`: the contained failures of a `subflow` child or a `map` item). On a delegating node this may therefore exceed `runs` and is no longer only this node's own failed executions; a child that FAILED adds only the step's own failure (unlike `acted`, which carries a failed child's writes too); the run-level `failed` is the sum of this across `nodes` |
| **skipped** | `integer` | ✅ | Times a closed gate kept this node from running at all |
| **selected** | `integer` | optional | Records read across every execution — omitted for a node that reads none |
| **acted** | `integer` | optional | Records written / effects dispatched across every execution — omitted for a node that writes none |
Expand Down
1 change: 1 addition & 0 deletions packages/spec/authorable-surface/automation.json
Original file line number Diff line number Diff line change
Expand Up @@ -141,6 +141,7 @@
"automation/ExecutionStepLog:startedAt",
"automation/ExecutionStepLog:status",
"automation/ExecutionStepMetrics:acted",
"automation/ExecutionStepMetrics:failures",
"automation/ExecutionStepMetrics:selected",
"automation/ExecutionStepMetrics:unmeasuredEffect",
"automation/ExecutionStepSkipReason:edgeId",
Expand Down
Loading
Loading