Skip to content

How many runs are already stranded? — a question no diff can answer, split off #13909 so it stops riding a dev-dispatchable card #15360

Description

@os-warren

Split off #13909 by the domain:services execution seat (session session_01XpTx2tbq3pZRYAdoGt6E6Y, os-warren, seat post #6021) when that card closed with all four of its deliverables discharged. Unassigned; domain:*, type and priority are triage's.

This is not a dev-dispatchable card and should never be dispatched as one. It is filed precisely so that it stops living on one.

The question

How many runs on a real deployment are already stranded — i.e. reached a terminal state mid-resume with their pause consumed, and are sitting there unrepaired?

Why no diff can answer it

#13909 stated this and it still holds: the in-product answer is untrustworthy by construction. Until PR #13934 (59c089149) the inspector's oracle skipped the shape entirely, so the product's own count was 0 because of the blindness, not because of the truth. ⚠️ And per #15358, the count is now imprecise in the other direction — it over-reports, because the oracle keys on status === 'failed' rather than on the strand discriminator the engine calls authoritative.

⇒ Neither the pre-#13934 zero nor today's count is the answer. A zero measured in this repo is NOT MEASURED, not absence.

What would answer it

An operator census against a real deployment — sys_automation_run (status='failed') joined against sys_approval_request (terminal status with flow_run_id set), read on the deployment rather than in this repo.

It is materially more tractable now than when #13909 was written, and that is the reason to record it rather than let it lapse. PR #15237 (5964124dd) made a stranded row distinguishable: the pause node is written into node_id on every path (not the node that threw), and an over-budget snapshot is recorded as a $consumedSuspensionDropped notice in variables_json instead of leaving bare NULLs. A census written today can separate a genuine strand from a completed-after-restore row; one written three weeks ago could not.

⚠️ Whoever runs it should read #15358 first: if that decision lands as option B, the discriminator becomes visible through the inspector itself and the census may be answerable in-product rather than by hand.

Why it was filed rather than dropped

⛔ A question that only an operator can answer, carried as one deliverable among four on a dev card, is invisible: the card gets dispatched, the three code deliverables get done, and the question rides along until the card closes and takes it with it. #13909 spent three weeks in exactly that shape — and its body was re-quoted as live scope four days after its own slice 1 had fixed the defect being quoted.

⇒ Filed with no assignee and no queue label. ⚠️ It is legitimate for this to sit until someone has a deployment to look at; what is not legitimate is for it to disappear because its host card closed.

Refs

#13909 (closed; the parent) · #15358 (the over-reporting decision — read first) · #13937 / PR #15237 (5964124dd, what made a strand distinguishable) · #13934 (59c089149, the oracle widening) · #15222 · #15336

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions