fix(service-automation): a restored suspension carries the pause, not the failed attempt - #16396
Conversation
…d attempt's `resumeInternal` rebuilt the flow scope as `new Map(Object.entries( run.variables))` — keys copied, value objects SHARED — and `journalConsumedSuspension` then shallow-copied `run` and journalled it as the pause "VERBATIM". An executor that keeps state in the scope and updates it in place (`map`'s `<nodeId>.$mapState`) therefore wrote through into the snapshot `restoreConsumedSuspension` hands an operator, so the exit re-armed a pause carrying state that belonged to the failed attempt. Measured: the durable row held `started: 1` at the pause and the restore put back `started: 99`. The copy is taken before the failed attempt runs — the same line that already captures `stepCountAtPause`, for the same reason. No later placement works: the node mutates and THEN throws, so a copy taken at journal or restore time copies the corruption. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XpTx2tbq3pZRYAdoGt6E6Y
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XpTx2tbq3pZRYAdoGt6E6Y
📓 Docs Drift CheckThis PR changes 1 package(s): ⛔ 2 release-owned page(s) name something this change touched. These are read-only:
What this run could not see
Coarse fallback — 5 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): Which tree this was computed onThis run read A worktree cut from an older # while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 97670b10f2693ea2b8738d9019ba5c44a4cd2b25 && git checkout 97670b10f2693ea2b8738d9019ba5c44a4cd2b25
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 4998efa71773154561c471075f4ef12566ecc455 d3ab0677673554881ad1a7742f021de4ec30a8ae && git checkout -B drift-repro 4998efa71773154561c471075f4ef12566ecc455 && git merge --no-ff d3ab0677673554881ad1a7742f021de4ec30a8ae
node scripts/docs-audit/affected-docs.mjs --json 4998efa71773154561c471075f4ef12566ecc455
|
Fixes #15660
The card was filed as a reading, not a measurement, and its first job was to decide whether it reproduces at all. It reproduces — but not by the mechanism the card named. Both halves are below.
It reproduces
Driven end to end against the real engine, exactly the path the card named as the only exposure (the #13937 consumed-suspension restore):
started: 1restoreConsumedSuspensionput backstarted: 99started: 99An operator repairing a stranded run got a snapshot stamped as "the pause" that carried state written by the attempt that failed.
The mechanism is NOT the one the card named
The card attributed the aliasing to "the in-memory suspended-run store keeps the object by identity rather than serialising it". It does not.
InMemorySuspendedRunStoreJSON round-trips on bothsaveandload— pinned in this PR, because the next reader should not re-derive a fix from a premise that is false.The aliasing is minted on the resume, not at the suspend.
resumeInternalrebuilds the flow scope asnew Map(Object.entries(run.variables)): that copies the keys and shares every value object with the parked snapshot.journalConsumedSuspensionthen shallow-copiesrunand journals it as the pause "verbatim" — a guarantee it delivered for top-level keys only. So an executor that keeps state in the scope and updates it in place writes straight through into the snapshot the operator exit hands back.mapis the concrete instance (nodeId.$mapState) and, by a census re-derived here rather than recalled, still the only executor in-repo that does this.Both proposed fix placements are refuted by that same measurement
The card offered three candidates and asked that the choice be made after the repro, with cost measured. Two of them are refuted:
The copy has to be taken before the failed attempt runs. That is the line that already captures
stepCountAtPause, for the same reason, so the fix sits beside the code that had the same problem and solved it.The cost the card flagged NOT MEASURED
MAX_MAP_ITEMS)Measured against what the same path already spends: one suspend+resume round trip against a configured store already performs four full clones of this same payload (
savex1,loadx2,recordTerminalx1 — counted, not read). This adds a fifth, on a path whose production cost is a durable round trip. The existing objection to a JSON clone was that it would run inside a catch arm already handling a failure, where a circular value would throw the operator's repair away; on the happy path it is just a value we decline to copy, and the fallback is the previous behaviour with a warning.Verification
git rev-parse --short HEAD= d3ab067 — every result below is from that tree.origin/main(mutation confirmed on disk: blob hash equals theorigin/mainblob, marker count 3 to 0), the new pin is 2 failed | 27 passed; both control arms and all 24 tests of the existingconsumed-suspension-restore.test.tsstay green. With the fix: 29 passed. Restore proved by an emptygit diff HEADand disk hash equal to the HEAD blob.--listFiles(1 hit each) rather than assumed.node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack --commands— and all 56 run. Two returned exit 3PREREQUISITE NOT MET(check:dual-build-cjs-loads,check:type-check-debt); both were re-run afterturbo run buildand returned real greens.Scope
No contract surface: no
packages/spec/**, no exported type or signature changed. The resume ordering is untouched, and a running flow sees exactly the scope it saw before — only what gets journalled changes.Not folded in, per the card's boundaries: #15616 / PR #15648 (the
mapstate lifetime) and #15646 (therunRegionsuspend-arm residue) remain open and are not addressed here.🤖 Generated with Claude Code
https://claude.ai/code/session_01XpTx2tbq3pZRYAdoGt6E6Y
Generated by Claude Code