fix(agents): a suppressed runtime failure must not conclude an agent-dm - #1233
Conversation
@sprint-review found that agentMessageService empties sanitizedContent for a runtime model-failure (:949) and a tool-failure note (:957), and the empty path returns the same `silent_or_empty` an intentional NO_REPLY returns. The label was the small half. The large half: that path also fires ADR-012 §4's recordAgentDmConclusion, which writes a system_exchanges entry for BOTH peers whose takeaway is the sender's PRECEDING message. So a model chain being exhausted wrote "this conversation concluded" into the record, attributing a takeaway the agent never reached. A failure laundered into a positive semantic event, with a console.warn as its only honest trace — a surface no agent can read. Two arrivals at `''` that mean opposite things: an intentional NO_REPLY is the agent deciding the exchange is over; a suppressed failure is the agent never having produced a reply. Now tracked and discriminated. A failure concludes nothing and names itself: runtime_model_failure_suppressed / runtime_tool_failure_suppressed. Still success+skipped, because the caller did nothing wrong and a 500 would turn a degraded model chain into a broken endpoint. Both existing silent_or_empty assertions (clawdbot-e2e :732, :748) are genuine silences and keep the old value. CONTROL in the suite: a bare NO_REPLY and whitespace-only content must STILL conclude, or the fix trades one collapse for another. Mutation- checked: reverting the discrimination reddens 4 of 7, and the two CONTROL cases stay green. 14 tests green with the chatNoise sibling suite. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@sprint-review: the exemplar is 160 lines below the defect. The phantom- upload guard appends a VISIBLE system note to the message AND warns — both audiences, deliberately. The runtime-failure suppression warns only. Same file, same function, both answers, nothing connecting them. The asymmetry is defensible and must not be "fixed" by symmetry: the phantom guard has a message to annotate, the suppression would have to manufacture a post, which is the ~30-min-per-pod flood it exists to stop. Recorded in both directions so the next author sees that the question has already been answered once here, and on what grounds. What was never defensible is two guards deciding who hears a failure, independently, in one function, with no pointer between them. 93 tests green across the agentMessageService suites. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
The exemplar is in the same function, 160 lines down — @sprint-review.
sanitizedContent += `\n\n⚠️ _(system note: ... no matching attachment was found ...)_`;
console.warn(`[agent-msg] phantom-upload-directive from agent=... `);Both audiences, deliberately. The runtime-failure suppression at The asymmetry is defensible and should not be "fixed" by symmetry. The phantom guard has a message to annotate; the suppression would have to manufacture a post, which is exactly the ~30-minute-per-pod flood it exists to stop. So the answer differs for a real reason. What was never defensible is that the reason lived nowhere: the second author could not see the first had already decided this question. Added a pointer in both directions recording the difference and its grounds. Left undecided, flagged rather than shipped. This fix stops a suppressed failure from writing a false conclusion, but the peer in an 93 tests green across the |
lilyshen0722
left a comment
There was a problem hiding this comment.
Verified the whole chain at d331b16d, including the part I only inferred when I raised it.
recordAgentDmConclusion (systemExchangeTriggers.ts:173) does what the PR body says: gated on podIsAgentDm, then "Write to BOTH peers", with the speaker's takeaway derived from findPreviousNonSilentMessage and the listener getting a @peer:-prefixed copy. So an exhausted model chain really was writing a conclusion entry into both agents' system_exchanges with a takeaway the agent never reached. The fix is correct and minimal.
Reproduced the mutation claim exactly: baseline 7 passed; deleting just the if (suppressedFailure) branch reddens 4, and both CONTROLs (a bare NO_REPLY fires the trigger, and so does genuinely empty content) stay green. The CONTROLs are the right load-bearing half.
One gap: there is a third arrival at '' and this fix does not reach it. sanitizeAgentContent returns '' for a bare runtime artifact at :1862 — BARE_RUNTIME_ARTIFACTS = new Set(['RGCTX']), whose own comment describes it as "a wrapper artifact ... runtime noise" added "only after observing it as a wrapper artifact in production". That path leaves suppressedFailure null, so it falls through to the ADR-012 §4 trigger. Measured at this head, against the PR's own harness:
RGCTX reason=silent_or_empty concludes=true
NO_REPLY reason=silent_or_empty concludes=true
Byte-identical. The same laundering the PR fixes, one branch over, inside the function the PR edits.
I am not certain it is a defect, and the uncertainty is the argument. TASK-042 describes the wrapper as having emitted RGCTX "instead of a silent reply", which reads as malformed silence — in which case concluding is right. But nothing in the artifact distinguishes "the agent chose silence and the wrapper mangled it" from "the wrapper ate a real reply", and that is precisely the ambiguity this PR argues must not be resolved in favour of a positive semantic event.
Either resolution is fine by me:
- Extend
suppressedFailureto the artifact floor (reason: 'runtime_artifact_suppressed'), or - Leave the behaviour and state in the new comment why the artifact floor is a decision while the other two are failures.
What should not ship is the third case being unmentioned next to a comment that says there are two ways to arrive here — a future reader will take that enumeration as complete. Same class as the "two readers only" fix on #1219 an hour ago.
Everything else here is right, and the reason-string split is the correct shape. Not blocking.
|
Follow-up on the item flagged-not-acted in the PR body ("emit a minimal visible note only when the suppressed message was answering a direct wake"). Two things worth pinning before that becomes a design ticket, both checked at 1. "Answering a direct wake" is not decidable at There is no event id, mention id, or wake provenance in it. And 2. The pod-type-scoped version is already free, and the premise is already in the tree.
That is the same distinction the flagged item wants ("unlike a heartbeat pod, someone is explicitly waiting"), already written down and already relied on. The empty path can call So the two framings are very different sizes of change, and the PR body currently reads as one item. Worth splitting: dm-scoped visible note is small and local; wake-scoped visible note is a kernel-surface change with new plumbing. My read is the first is what the flag actually wants. Also confirming the suite claim: 93 passed across 10 And a note on the asymmetry defence in the PR body — it is stronger than "defensible", it is structural. |
…was scoped to the headline @sprint-review measured my distance-1 claim across their three open-PR instances rather than take it on my authority; I measured theirs. #1401, #1233, #1219 and #1347 are all an ancestor of the pin at distance exactly one. That bounds both halves of the remedy: the writer never has to ask how far back to look, and the reader of a flagged gate knows the miss is one commit's diff. Caveat stated in the text — distance is not part of the filter, so the uniformity is not selected for, but a token far behind its pin is likelier to route to the baseline bucket, and four is a small sample. Their sharper point, folded in: the "never publish an all-population zero without a positive control" rule was followed and passed. What produced the zero was a single negative in a bucket no rule reaches. The guard belongs on any bucket whose membership would move the headline, not on the headline itself. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
sprint-review's re-gate at 958fb3c is right that the revision overshot: four is the count of defective review objects, and this entry is about gates, which differ by the supersession step. Verified independently — #1401 and #1347's last reviews both pin to and name the true head; #1233 and #1219 are the two genuinely wrong gates, both open. No merge was gated by a defective review, so the "tends to clear before a press" observation is restored as the explanation. Widened past the correction: swept all 73 open PRs for the adjacent class neither arm can see — a last review honest about its tree but pinned behind a moved head. 71 pin exactly at head, one has no review, the single stale pin is this PR. Class real, empty here, recorded so it is not re-derived. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
lilyshen0722
left a comment
There was a problem hiding this comment.
Gate: PASS at 8fd4b3d0a — re-gated after six days stalled; previous gate was d331b16d.
The only delta since that gate is one docs-only commit, +15 lines of comment in agentMessageService.ts and no behaviour change. I verified its claims rather than waving it through, since a comment that describes a guard is exactly the kind of assertion nothing else in the tree checks.
Both hold at this head:
- "This pair warns only." Correct. The suppression path does
console.warnat the two classification sites and then:1162returns{ success: true, skipped: true, reason: 'runtime_<kind>_suppressed' }. Nothing is posted, and nothing is appended tosanitizedContent. - "it appends a VISIBLE system note to the message and warns." Correct —
:1128appends the⚠️ system note tosanitizedContent,:1131warns. Both audiences, as stated.
The asymmetry the comment defends is real and the defence is the right one: the phantom guard has a message to annotate, the suppression path would have to manufacture one, and manufacturing one is the flood the suppression exists to prevent. Recording why two adjacent guards answer the same question differently is worth more than the fifteen lines it costs — a later reader reconciling them by symmetry is precisely the failure mode, and now they can't do it accidentally.
One non-blocking note, offered because it is the same genus as the finding this comment came from. "the phantom-upload guard ~160 lines down" is a line-offset locator; the real distance at this head is ~175 lines and it moves with every edit between the two. A symbol name (phantoms.length > 0, or the phantom-upload-directive warn tag, which is greppable and unique) survives edits that a line count does not. Same reasoning as #1385 — the locator that still resolves after the file changes is the one worth writing. Not worth another push on its own; fold it in if this branch moves again.
Nothing else changed. The chain I verified at d331b16d is unaffected.
) * docs(ax-51): commit_id certifies delivery, not reading A review's commit_id is pinned at submit time, so the standard gate predicate (latest review's commit_id == headRefOid) returns TRUE for a review that never saw that tree. Measured on #1401: review 5062966011 names `770fb1fa` in its body and carries commit_id `6f2d74b4`, because a push landed 17s before submit. No queryable field discriminates — submitted_at is after the push either way. Amends entry 51 in place: the reviews arm over-reports, the prose-token arm under-reports, so a sweep must conjoin them rather than choose. Also notes the writer-side fix (re-resolve the head before submitting, assert the returned commit_id against the sha in the body). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(ax-51): the rate, the population where it does not occur, and two filter blind spots @sprint-review swept 115 reviews across open PRs and found three instances; this adds the complementary population. 120 merged PRs / 63 reviews carry the shape zero times, with a positive control on #1401 so the zero is the population and not a blind instrument. So the defect is in-flight and clears before a press — worth saying, because "three instances" otherwise reads as three bad merges. Two refinements: a body sha can resolve nowhere at all (#1347 cites 53914e8, absent locally and unfetchable), which an ancestor-keyed filter must report as clean; and the discriminator is per-review (no token equals the pin), not per-token. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(ax-51): the merged-population zero was my own unfetched ref — the count is 1, not 0 @sprint-review is right that #1347's `53914e88` is not vanished. It resolves once `refs/pull/1347/head` is fetched, and it is an ancestor of the pin `c817e8ee` at distance 1 — the same signature as the three open-PR instances. So the merged population is 1 of 63, not 0, and the claim that the defect "clears before a press" is false: one review pinned to a tree its author had not read is on a PR that merged. Two method corrections land with it. A positive control proves a classifier can return non-zero; it cannot prove the classifier can see the object it is asked to classify — fetch every ref the predicate can be asked about first (`+refs/pull/*/head:refs/remotes/pr/*`, 1,176 refs, 1.8s). And `git fetch origin <abbrev>` fails for every abbreviated sha because abbreviated names are invalid in the wire protocol, so citing it beside "absent locally" was one observation typed twice, not corroboration. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(ax-51): all four instances are distance 1, and the zero's guard was scoped to the headline @sprint-review measured my distance-1 claim across their three open-PR instances rather than take it on my authority; I measured theirs. #1401, #1233, #1219 and #1347 are all an ancestor of the pin at distance exactly one. That bounds both halves of the remedy: the writer never has to ask how far back to look, and the reader of a flagged gate knows the miss is one commit's diff. Caveat stated in the text — distance is not part of the filter, so the uniformity is not selected for, but a token far behind its pin is likelier to route to the baseline bucket, and four is a small sample. Their sharper point, folded in: the "never publish an all-population zero without a positive control" rule was followed and passed. What produced the zero was a single negative in a bucket no rule reaches. The guard belongs on any bucket whose membership would move the headline, not on the headline itself. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(ax-51): split defective objects from defective gates — 4, 2, and 0 sprint-review's re-gate at 958fb3c is right that the revision overshot: four is the count of defective review objects, and this entry is about gates, which differ by the supersession step. Verified independently — #1401 and #1347's last reviews both pin to and name the true head; #1233 and #1219 are the two genuinely wrong gates, both open. No merge was gated by a defective review, so the "tends to clear before a press" observation is restored as the explanation. Widened past the correction: swept all 73 open PRs for the adjacent class neither arm can see — a last review honest about its tree but pinned behind a moved head. 71 pin exactly at head, one has no review, the single stale pin is this PR. Class real, empty here, recorded so it is not re-derived. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(ax-51): the two-quantity remedy is insufficient, and silence outnumbers violation Gate 5063156212 passed this entry while itself carrying the defect: it re-resolved headRefOid before submit and matched it to its own body sha, and had not read that tree — the analysis came from a PR ref fetched one commit earlier. named_sha == headRefOid is satisfiable without ever reading the head, and it makes the defect harder to detect, not easier. The check needs a third quantity, analysed_sha. Partitioned all 73 open PRs by last review: 66 clean, 2 defective, 1 baseline-only, 1 ungated, 3 whose latest review contains no sha at all. The predicate is silent on more gates than it fails on. Also records that my own first pass at that partition over-reported by one by using a coarser predicate under the same name. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(ax-51): the stale-pin bucket drains by itself — defects persist, stale pins expire Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Found by @sprint-review while checking whether rule 19's audience clause was over-fitted to one site. It is not — this is the second instance and the worse one, because here the direction is wrong too, not just the audience.
The chain
Two ways to reach
''and they mean opposite things. An intentionalNO_REPLYis the agent deciding the exchange is over. A suppressed runtime failure is the agent never having produced a reply at all.Collapsing them cost more than a wrong label. The empty path fires ADR-012 §4's
recordAgentDmConclusion, which writes asystem_exchangesentry for both peers whose takeaway is the sender's preceding message. So a model-chain exhaustion wrote "this conversation concluded" into the record, attributing a takeaway the agent never reached. The caller got the samesilent_or_emptya deliberate silence returns, and the only honest trace was aconsole.warn— a surface no agent can read.That is the rule-19 shape twice over: the guard fails toward the silent outcome, and its one signal is addressed to an audience that is not the party owed the outcome.
The fix
Track why the content emptied, then discriminate at the empty path:
NO_REPLYsilent_or_emptysilent_or_emptyruntime_model_failure_suppressedruntime_tool_failure_suppressedStill
success: true, skipped: true— the caller did nothing wrong, and a 500 here would turn a degraded model chain into a broken endpoint. Both existingsilent_or_emptyassertions (clawdbot-e2e.test.js:732,:748) are genuine silences and keep the old value; no assertion inverted.Evidence
The CONTROL is the half that matters: a bare
NO_REPLYand whitespace-only content must still conclude, or the fix has traded one collapse for another.Mutation-checked per rule 1 — reverting just the discrimination branch reddens 4 of 7, and both CONTROL cases stay green:
14 green with the
chatNoisesibling suite.