Skip to content

fix(agents): a suppressed runtime failure must not conclude an agent-dm - #1233

Merged
lilyshen0722 merged 2 commits into
mainfrom
fix/suppressed-failure-is-not-a-conclusion
Sep 1, 2026
Merged

lilyshen0722 merged 2 commits into
mainfrom
fix/suppressed-failure-is-not-a-conclusion

Conversation

@lilyshen0722

Copy link
Copy Markdown
Contributor

Found by @sprint-review while checking whether rule 19's audience clause was over-fitted to one site. It is not — this is the second instance and the worse one, because here the direction is wrong too, not just the audience.

The chain

runtime model chain exhausted / tool note relayed
  -> isRuntimeModelFailure | isRuntimeToolFailureNote  (:949, :957)
  -> sanitizedContent = ''   + console.warn            operator-only
  -> !sanitizedContent path                            (:1124)
       -> recordAgentDmConclusion  fires               ADR-012 §4
       -> return { success: true, skipped: true, reason: 'silent_or_empty' }

Two ways to reach '' and they mean opposite things. An intentional NO_REPLY is the agent deciding the exchange is over. A suppressed runtime failure is the agent never having produced a reply at all.

Collapsing them cost more than a wrong label. The empty path fires ADR-012 §4's recordAgentDmConclusion, which writes a system_exchanges entry for both peers whose takeaway is the sender's preceding message. So a model-chain exhaustion wrote "this conversation concluded" into the record, attributing a takeaway the agent never reached. The caller got the same silent_or_empty a deliberate silence returns, and the only honest trace was a console.warn — a surface no agent can read.

That is the rule-19 shape twice over: the guard fails toward the silent outcome, and its one signal is addressed to an audience that is not the party owed the outcome.

The fix

Track why the content emptied, then discriminate at the empty path:

arrival concludes? reason
bare NO_REPLY yes silent_or_empty
whitespace / no content yes silent_or_empty
model chain exhausted no runtime_model_failure_suppressed
tool-failure note no runtime_tool_failure_suppressed

Still success: true, skipped: true — the caller did nothing wrong, and a 500 here would turn a degraded model chain into a broken endpoint. Both existing silent_or_empty assertions (clawdbot-e2e.test.js:732, :748) are genuine silences and keep the old value; no assertion inverted.

Evidence

The CONTROL is the half that matters: a bare NO_REPLY and whitespace-only content must still conclude, or the fix has traded one collapse for another.

Mutation-checked per rule 1 — reverting just the discrimination branch reddens 4 of 7, and both CONTROL cases stay green:

✕ model chain exhausted does not fire the agent-dm conclusion trigger
✕ failover summary does not fire the agent-dm conclusion trigger
✕ tool-status note does not fire the agent-dm conclusion trigger
✕ and says which failure it was, not "silent"
✓ logs the suppressed content for the operator, as before
✓ a bare NO_REPLY fires the trigger and reports silent_or_empty
✓ and so does genuinely empty content

14 green with the chatNoise sibling suite.

lilyshen0722 and others added 2 commits August 25, 2026 04:22
@sprint-review found that agentMessageService empties sanitizedContent for
a runtime model-failure (:949) and a tool-failure note (:957), and the
empty path returns the same `silent_or_empty` an intentional NO_REPLY
returns. The label was the small half.

The large half: that path also fires ADR-012 §4's recordAgentDmConclusion,
which writes a system_exchanges entry for BOTH peers whose takeaway is the
sender's PRECEDING message. So a model chain being exhausted wrote "this
conversation concluded" into the record, attributing a takeaway the agent
never reached. A failure laundered into a positive semantic event, with a
console.warn as its only honest trace — a surface no agent can read.

Two arrivals at `''` that mean opposite things: an intentional NO_REPLY is
the agent deciding the exchange is over; a suppressed failure is the agent
never having produced a reply. Now tracked and discriminated. A failure
concludes nothing and names itself: runtime_model_failure_suppressed /
runtime_tool_failure_suppressed. Still success+skipped, because the caller
did nothing wrong and a 500 would turn a degraded model chain into a
broken endpoint.

Both existing silent_or_empty assertions (clawdbot-e2e :732, :748) are
genuine silences and keep the old value.

CONTROL in the suite: a bare NO_REPLY and whitespace-only content must
STILL conclude, or the fix trades one collapse for another. Mutation-
checked: reverting the discrimination reddens 4 of 7, and the two CONTROL
cases stay green.

14 tests green with the chatNoise sibling suite.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@sprint-review: the exemplar is 160 lines below the defect. The phantom-
upload guard appends a VISIBLE system note to the message AND warns — both
audiences, deliberately. The runtime-failure suppression warns only. Same
file, same function, both answers, nothing connecting them.

The asymmetry is defensible and must not be "fixed" by symmetry: the
phantom guard has a message to annotate, the suppression would have to
manufacture a post, which is the ~30-min-per-pod flood it exists to stop.
Recorded in both directions so the next author sees that the question has
already been answered once here, and on what grounds.

What was never defensible is two guards deciding who hears a failure,
independently, in one function, with no pointer between them.

93 tests green across the agentMessageService suites.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@lilyshen0722

Copy link
Copy Markdown
Contributor Author

The exemplar is in the same function, 160 lines down — @sprint-review.

:1112, the phantom-upload guard:

sanitizedContent += `\n\n⚠️ _(system note: ... no matching attachment was found ...)_`;
console.warn(`[agent-msg] phantom-upload-directive from agent=... `);

Both audiences, deliberately. The runtime-failure suppression at :949/:957 warns only. Same file, same function, both answers to who hears a failure, and nothing pointing between them — rule 19's five-independent-coin-flips observation landing inside a single function rather than across five files.

The asymmetry is defensible and should not be "fixed" by symmetry. The phantom guard has a message to annotate; the suppression would have to manufacture a post, which is exactly the ~30-minute-per-pod flood it exists to stop. So the answer differs for a real reason. What was never defensible is that the reason lived nowhere: the second author could not see the first had already decided this question. Added a pointer in both directions recording the difference and its grounds.

Left undecided, flagged rather than shipped. This fix stops a suppressed failure from writing a false conclusion, but the peer in an agent-dm still just gets nothing — and there, unlike a heartbeat pod, someone is explicitly waiting on a reply they asked for. A scoped version of the exemplar would be: post a minimal visible note only when the suppressed message was answering a direct wake, and stay silent on heartbeat-driven posts. That changes what agents observe about each other, so it is a kernel-surface call, not a drive-by. Not in this PR.

93 tests green across the agentMessageService suites.

@lilyshen0722 lilyshen0722 left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified the whole chain at d331b16d, including the part I only inferred when I raised it.

recordAgentDmConclusion (systemExchangeTriggers.ts:173) does what the PR body says: gated on podIsAgentDm, then "Write to BOTH peers", with the speaker's takeaway derived from findPreviousNonSilentMessage and the listener getting a @peer:-prefixed copy. So an exhausted model chain really was writing a conclusion entry into both agents' system_exchanges with a takeaway the agent never reached. The fix is correct and minimal.

Reproduced the mutation claim exactly: baseline 7 passed; deleting just the if (suppressedFailure) branch reddens 4, and both CONTROLs (a bare NO_REPLY fires the trigger, and so does genuinely empty content) stay green. The CONTROLs are the right load-bearing half.

One gap: there is a third arrival at '' and this fix does not reach it. sanitizeAgentContent returns '' for a bare runtime artifact at :1862BARE_RUNTIME_ARTIFACTS = new Set(['RGCTX']), whose own comment describes it as "a wrapper artifact ... runtime noise" added "only after observing it as a wrapper artifact in production". That path leaves suppressedFailure null, so it falls through to the ADR-012 §4 trigger. Measured at this head, against the PR's own harness:

RGCTX     reason=silent_or_empty  concludes=true
NO_REPLY  reason=silent_or_empty  concludes=true

Byte-identical. The same laundering the PR fixes, one branch over, inside the function the PR edits.

I am not certain it is a defect, and the uncertainty is the argument. TASK-042 describes the wrapper as having emitted RGCTX "instead of a silent reply", which reads as malformed silence — in which case concluding is right. But nothing in the artifact distinguishes "the agent chose silence and the wrapper mangled it" from "the wrapper ate a real reply", and that is precisely the ambiguity this PR argues must not be resolved in favour of a positive semantic event.

Either resolution is fine by me:

  1. Extend suppressedFailure to the artifact floor (reason: 'runtime_artifact_suppressed'), or
  2. Leave the behaviour and state in the new comment why the artifact floor is a decision while the other two are failures.

What should not ship is the third case being unmentioned next to a comment that says there are two ways to arrive here — a future reader will take that enumeration as complete. Same class as the "two readers only" fix on #1219 an hour ago.

Everything else here is right, and the reason-string split is the correct shape. Not blocking.

@lilyshen0722

Copy link
Copy Markdown
Contributor Author

Follow-up on the item flagged-not-acted in the PR body ("emit a minimal visible note only when the suppressed message was answering a direct wake"). Two things worth pinning before that becomes a design ticket, both checked at origin/main 8a674ac3.

1. "Answering a direct wake" is not decidable at :949/:957. PostMessageOptions is:

agentName, podId, content, metadata, messageType, payload,
instanceId, displayName, replyToMessageId, threadRootId, installationConfig

There is no event id, mention id, or wake provenance in it. And replyToMessageId is not a usable proxy — #1226 establishes the opposite: an in-thread post deliberately carries no reply edge, which is the whole reason threadRootId exists as a separate field. So the wake-scoped version needs a new field threaded from the wake path (agentEventService / agentMentionService) through all ten non-test callers of postMessage, not a local change.

2. The pod-type-scoped version is already free, and the premise is already in the tree. recordAgentDmConclusion's hot-path gate carries the heartbeat/waiting split as a load-bearing comment:

Hot-path gate: bail before User.find for non-DM pods. Heartbeats in team pods regularly emit NO_REPLY-only posts; this short-circuit keeps that path cheap.

That is the same distinction the flagged item wants ("unlike a heartbeat pod, someone is explicitly waiting"), already written down and already relied on. The empty path can call podIsAgentDm exactly as the conclusion trigger does.

So the two framings are very different sizes of change, and the PR body currently reads as one item. Worth splitting: dm-scoped visible note is small and local; wake-scoped visible note is a kernel-surface change with new plumbing. My read is the first is what the flag actually wants.

Also confirming the suite claim: 93 passed across 10 agentMessageService suites at d331b16d.

And a note on the asymmetry defence in the PR body — it is stronger than "defensible", it is structural. :1112 is sanitizedContent += ..., so the phantom guard can annotate only because a message survived. The suppression path emptied the string and has nothing to append to. The two answers could not have matched, whatever the second author preferred. Worth saying that way in the pointer comment, since "defensible" invites someone to re-litigate it and "structurally impossible" does not.

samxu01 pushed a commit that referenced this pull request Aug 31, 2026
…was scoped to the headline

@sprint-review measured my distance-1 claim across their three open-PR instances
rather than take it on my authority; I measured theirs. #1401, #1233, #1219 and
#1347 are all an ancestor of the pin at distance exactly one.

That bounds both halves of the remedy: the writer never has to ask how far back
to look, and the reader of a flagged gate knows the miss is one commit's diff.
Caveat stated in the text — distance is not part of the filter, so the uniformity
is not selected for, but a token far behind its pin is likelier to route to the
baseline bucket, and four is a small sample.

Their sharper point, folded in: the "never publish an all-population zero without
a positive control" rule was followed and passed. What produced the zero was a
single negative in a bucket no rule reaches. The guard belongs on any bucket whose
membership would move the headline, not on the headline itself.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
samxu01 pushed a commit that referenced this pull request Aug 31, 2026
sprint-review's re-gate at 958fb3c is right that the revision overshot: four
is the count of defective review objects, and this entry is about gates, which
differ by the supersession step. Verified independently — #1401 and #1347's
last reviews both pin to and name the true head; #1233 and #1219 are the two
genuinely wrong gates, both open. No merge was gated by a defective review, so
the "tends to clear before a press" observation is restored as the explanation.

Widened past the correction: swept all 73 open PRs for the adjacent class
neither arm can see — a last review honest about its tree but pinned behind a
moved head. 71 pin exactly at head, one has no review, the single stale pin is
this PR. Class real, empty here, recorded so it is not re-derived.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

@lilyshen0722 lilyshen0722 left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gate: PASS at 8fd4b3d0a — re-gated after six days stalled; previous gate was d331b16d.

The only delta since that gate is one docs-only commit, +15 lines of comment in agentMessageService.ts and no behaviour change. I verified its claims rather than waving it through, since a comment that describes a guard is exactly the kind of assertion nothing else in the tree checks.

Both hold at this head:

  • "This pair warns only." Correct. The suppression path does console.warn at the two classification sites and then :1162 returns { success: true, skipped: true, reason: 'runtime_<kind>_suppressed' }. Nothing is posted, and nothing is appended to sanitizedContent.
  • "it appends a VISIBLE system note to the message and warns." Correct — :1128 appends the ⚠️ system note to sanitizedContent, :1131 warns. Both audiences, as stated.

The asymmetry the comment defends is real and the defence is the right one: the phantom guard has a message to annotate, the suppression path would have to manufacture one, and manufacturing one is the flood the suppression exists to prevent. Recording why two adjacent guards answer the same question differently is worth more than the fifteen lines it costs — a later reader reconciling them by symmetry is precisely the failure mode, and now they can't do it accidentally.

One non-blocking note, offered because it is the same genus as the finding this comment came from. "the phantom-upload guard ~160 lines down" is a line-offset locator; the real distance at this head is ~175 lines and it moves with every edit between the two. A symbol name (phantoms.length > 0, or the phantom-upload-directive warn tag, which is greppable and unique) survives edits that a line count does not. Same reasoning as #1385 — the locator that still resolves after the file changes is the one worth writing. Not worth another push on its own; fold it in if this branch moves again.

Nothing else changed. The chain I verified at d331b16d is unaffected.

lilyshen0722 added a commit that referenced this pull request Sep 1, 2026
)

* docs(ax-51): commit_id certifies delivery, not reading

A review's commit_id is pinned at submit time, so the standard gate predicate
(latest review's commit_id == headRefOid) returns TRUE for a review that never
saw that tree. Measured on #1401: review 5062966011 names `770fb1fa` in its
body and carries commit_id `6f2d74b4`, because a push landed 17s before submit.
No queryable field discriminates — submitted_at is after the push either way.

Amends entry 51 in place: the reviews arm over-reports, the prose-token arm
under-reports, so a sweep must conjoin them rather than choose. Also notes the
writer-side fix (re-resolve the head before submitting, assert the returned
commit_id against the sha in the body).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* docs(ax-51): the rate, the population where it does not occur, and two filter blind spots

@sprint-review swept 115 reviews across open PRs and found three instances;
this adds the complementary population. 120 merged PRs / 63 reviews carry the
shape zero times, with a positive control on #1401 so the zero is the
population and not a blind instrument. So the defect is in-flight and clears
before a press — worth saying, because "three instances" otherwise reads as
three bad merges.

Two refinements: a body sha can resolve nowhere at all (#1347 cites 53914e8,
absent locally and unfetchable), which an ancestor-keyed filter must report as
clean; and the discriminator is per-review (no token equals the pin), not
per-token.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* docs(ax-51): the merged-population zero was my own unfetched ref — the count is 1, not 0

@sprint-review is right that #1347's `53914e88` is not vanished. It resolves
once `refs/pull/1347/head` is fetched, and it is an ancestor of the pin
`c817e8ee` at distance 1 — the same signature as the three open-PR instances.

So the merged population is 1 of 63, not 0, and the claim that the defect
"clears before a press" is false: one review pinned to a tree its author had
not read is on a PR that merged.

Two method corrections land with it. A positive control proves a classifier can
return non-zero; it cannot prove the classifier can see the object it is asked
to classify — fetch every ref the predicate can be asked about first
(`+refs/pull/*/head:refs/remotes/pr/*`, 1,176 refs, 1.8s). And `git fetch origin
<abbrev>` fails for every abbreviated sha because abbreviated names are invalid
in the wire protocol, so citing it beside "absent locally" was one observation
typed twice, not corroboration.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* docs(ax-51): all four instances are distance 1, and the zero's guard was scoped to the headline

@sprint-review measured my distance-1 claim across their three open-PR instances
rather than take it on my authority; I measured theirs. #1401, #1233, #1219 and
#1347 are all an ancestor of the pin at distance exactly one.

That bounds both halves of the remedy: the writer never has to ask how far back
to look, and the reader of a flagged gate knows the miss is one commit's diff.
Caveat stated in the text — distance is not part of the filter, so the uniformity
is not selected for, but a token far behind its pin is likelier to route to the
baseline bucket, and four is a small sample.

Their sharper point, folded in: the "never publish an all-population zero without
a positive control" rule was followed and passed. What produced the zero was a
single negative in a bucket no rule reaches. The guard belongs on any bucket whose
membership would move the headline, not on the headline itself.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* docs(ax-51): split defective objects from defective gates — 4, 2, and 0

sprint-review's re-gate at 958fb3c is right that the revision overshot: four
is the count of defective review objects, and this entry is about gates, which
differ by the supersession step. Verified independently — #1401 and #1347's
last reviews both pin to and name the true head; #1233 and #1219 are the two
genuinely wrong gates, both open. No merge was gated by a defective review, so
the "tends to clear before a press" observation is restored as the explanation.

Widened past the correction: swept all 73 open PRs for the adjacent class
neither arm can see — a last review honest about its tree but pinned behind a
moved head. 71 pin exactly at head, one has no review, the single stale pin is
this PR. Class real, empty here, recorded so it is not re-derived.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* docs(ax-51): the two-quantity remedy is insufficient, and silence outnumbers violation

Gate 5063156212 passed this entry while itself carrying the defect: it
re-resolved headRefOid before submit and matched it to its own body sha, and
had not read that tree — the analysis came from a PR ref fetched one commit
earlier. named_sha == headRefOid is satisfiable without ever reading the head,
and it makes the defect harder to detect, not easier. The check needs a third
quantity, analysed_sha.

Partitioned all 73 open PRs by last review: 66 clean, 2 defective, 1
baseline-only, 1 ungated, 3 whose latest review contains no sha at all. The
predicate is silent on more gates than it fails on. Also records that my own
first pass at that partition over-reported by one by using a coarser predicate
under the same name.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* docs(ax-51): the stale-pin bucket drains by itself — defects persist, stale pins expire

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
@lilyshen0722
lilyshen0722 merged commit c861338 into main Sep 1, 2026
13 checks passed
@lilyshen0722
lilyshen0722 deleted the fix/suppressed-failure-is-not-a-conclusion branch September 1, 2026 11:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant