Skip to content

feat(agent): context trace, partial-success terminal, and scored self-critique - #140

Merged
hung12ct merged 10 commits into
mainfrom
feat/loop-observability-quality-gates
Aug 2, 2026
Merged

feat(agent): context trace, partial-success terminal, and scored self-critique#140
hung12ct merged 10 commits into
mainfrom
feat/loop-observability-quality-gates

Conversation

@hung12ct

@hung12ct hung12ct commented Aug 2, 2026

Copy link
Copy Markdown
Owner

Summary

Three additive capabilities on the agent loop, each closing a gap that a host cannot fill from outside the package.

ContextTraceEvent — context pruning is no longer invisible. The loop prunes before every LLM call, and the most common path (no MaxTokenBudget set) reported nothing at all. When an adopter asked "why did the agent forget the schema from turn 2?", no artifact existed to answer from. The event names each rewritten message with a closed-enum reason (soft-trim / outlier-discarded / args-truncated), a policy (default / budget-warn / budget-emergency), and before/after token estimates. ContextRef.Index lines up with Sessions.History, and CorrelationID matches the one on ToolCallEvent, so a trace joins back to both the stored transcript and the call that produced it.

Emitted only when a prune actually changed something — a turn whose context fits emits nothing, and the token sweeps only run when there is something to report.

DegradedEvent — a terminal for work that half-landed. The vocabulary was binary: Done, or an error. There was no way to say "the artifact was written, the derived bookkeeping failed." Reporting Done hides a real inconsistency; reporting an error invites a retry that duplicates the expensive write. A tool raises one by returning a normal (non-error) tools.Result with Degraded: &tools.Degradation{Reason, Artifacts, Unreliable}.

The event precedes the terminal rather than replacing it, so DoneEvent still fires and consumers that ignore it are unaffected. The loop also appends a partial-success note to the text the model reads, telling it not to redo the half that landed — without that, the host knows and the model doesn't.

Scorer — self-critique keeps the best round, not the last. Reflect accepted any textually-different revision, so round 3 won even when round 1 was better, and round 1 was unrecoverable. The only guard was string equality, which a cosmetic rewrite defeats. With a Scorer set, the original answer is round 0 and each revision must strictly beat the incumbent; a rejected revision is rolled back so the next round critiques the incumbent rather than the discarded draft. Ties keep the incumbent, and a revision the scorer cannot rank is discarded — an unranked candidate cannot be shown to be an improvement.

Scorer is the shared interface planned for best-of-K, landed here with its first (sequential) consumer.

Changed (breaking)

  • EventVisitor gains VisitContextTrace and VisitDegraded. External implementers must add both. This is the documented purpose of the interface — a new payload type forces every visitor to handle it.
  • ReflectedEvent.Score is *float64, not float64. Zero is a legitimate score (a 0–100 rubric can return 0; a negated-latency scorer treats 0 as optimal), and float64 + omitempty erased it from the wire and made it indistinguishable from "unscored" in Go.
  • tools.Result gains Degraded *Degradation and a String() method. The method is load-bearing, not cosmetic: without it the new pointer field makes go vet reject %s/%q on a Result in downstream modules, breaking builds for callers who never touch the feature. Verified against a separate consumer module. Note that %v on a Result now prints its text rather than the struct.

Defaults and cost

All three are zero-cost when unused. Scorer is nil by default, preserving the previous last-wins reflect behavior exactly. Degraded is nil on every existing tool. The context trace is silent unless a prune fires. Measured overhead on a Run with none of them active: 47 → 50 allocs/op (the per-Run degradation accumulator, which cannot be gated since whether a tool will degrade is unknowable up front).

Review

Two code-reviewer passes. The first covered the three feature commits and found four real defects, each verified independently before fixing:

  • a degraded result was entering the tool cache, so a later session replayed a partial-success note for work that never ran while the host saw a clean turn;
  • a speculative execution that was discarded lost its degradation entirely — the double-write the feature exists to prevent;
  • ReflectedEvent.Score erasing a legitimate zero;
  • a doc claim that DegradedEvent always precedes the terminal, which is false on the deferred-sweep path (it trails ErrorEvent / LimitExhaustedEvent there).

The second pass covered those fixes and found a regression introduced by one of them: OnToolResult can recover an error into a success, so filing a degradation at execution time dropped the report for a recovered speculative call. Consumed results now file exactly once after the hook chain; orphaned speculations file at discard time, where the hook never runs. The two paths are mutually exclusive and cannot double-count.

Tests

25 new tests. Each load-bearing assertion was mutation-checked — the fix reverted in a scratch copy to confirm the test actually fails. That caught one hollow test: the reflect rollback assertion originally passed with the rollback deleted, because it only checked the final answer. It now asserts which answer each critique round was handed.

gofmt -l . empty · make vet clean · make lint 0 issues · make build · make test · make test-race all green.

Follow-ups (not in this PR)

  • Reflect / Scorer are not configurable from YAML — Go-side fields only.
  • No built-in tool raises a Degradation and the demo does not render either new event, so both are discoverable only via godoc.
  • BestOfK, the parallel consumer of Scorer, is still open.
  • CHANGELOG.md entry lands with the version tag.

hung12ct added 10 commits August 2, 2026 10:39
Pruning ran before every LLM call and reported nothing on the default
unbudgeted path. ContextTraceEvent names each rewritten message, its
reason, and the token delta — emitted only when something changed.
A tool whose artifact landed but whose derived state failed had to lie
with Done or discard good work with Error. tools.Degradation raises it;
DegradedEvent precedes the terminal, and the model is told not to retry
the half that succeeded.
…e last

Reflect accepted any textually different revision, so a critique pass
could silently make the answer worse and the earlier text was gone.
With a Scorer set, a revision must beat the incumbent to be adopted;
nil Scorer keeps today's behavior.
A cache hit replayed the partial-success note to a later turn's model
while short-circuiting before recordDegradation, so the host saw a clean
turn for work that never ran.
A retry reset or stream error drops the spec entry before it is awaited,
losing the report for a tool that really executed and half-succeeded —
the double-write the feature exists to prevent.
float64 with omitempty erased a legitimate 0 from the wire and left it
indistinguishable from an unscored round in Go — 0 is valid on a 0-100
rubric and optimal for a negated-latency scorer.
DegradedEvent trails the terminal frame on the cap/error sweep path, not
precedes it. Duplicate ContextRef.Index is unreachable under current
thresholds, and per-ref token estimates exclude tool-call arguments.
The *Degradation field made go vet reject %s/%q on a Result, failing
builds for callers that never use the degradation feature. Also check
the pointer before the ctx walk in recordDegradation.
OnToolResult can recover an error into a success, so filing at execution
time dropped the report for a recovered speculative call. Consumed
results now file once after the hook; orphaned speculations file at
discard, where the hook never runs.
enforceTokenBudget concatenates the truncation and prune passes, so
Index ascends within a group but not across the slice.
@hung12ct
hung12ct merged commit 930e1b2 into main Aug 2, 2026
2 checks passed
@hung12ct
hung12ct deleted the feat/loop-observability-quality-gates branch August 2, 2026 05:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant