chore(release): activeagent and actionagent 1.6.3 - #457
Merged
Merged
Conversation
Bumps both gems to 1.6.3 and opens the changelog section that 1.6.3 work files its entries under. The release summary and date are settled when the tag is cut. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The context meter sized each segment by estimating ~4 chars/token over the content attributes. Those attributes are previews clipped at storage, and on the SDK path the tool attribute is a roster — names, a truncated description and parameter keys — rather than the JSON Schema the model is sent, which for a twelve-tool agent is several times larger. Tool pressure therefore read far lower than what the provider billed. Each size is now recorded as its own attribute, taken before truncation, and MCP schemas are attributed apart from the toolbox's so the meter can say which half fills the window. The meter prefers a recorded size and falls back to estimating for traces that predate them. With real sizes in hand the meter apportions the provider's prompt_tokens across the segments rather than subtracting estimates from it. The old subtraction made "Messages" absorb the entire approximation error, showing a message history that was never sent. The execution service reads both halves through tool_schemas' own split, so a mock run still offers no tools and a server's tool list is fetched once per run rather than a second time for telemetry. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The context meter scales its segments to fill the provider's prompt_tokens rather than subtracting estimates from it, which is the right call — but the transcript was left out of the set being scaled. Scaling a subset to fill the total hands the missing piece's share to the segments that remain, so the segment the change set out to stop over-reporting became one that could never report anything at all: `messages` was computed as the remainder after the others had been scaled up to cover the whole total, which is zero by construction. On an ordinary dashboard trace — a long conversation, a small system prompt, one small tool — a 500-token instruction set rendered as 13,333 tokens and a 19,250-token history as 0. That is the same failure the change was written to fix, pointed at a different segment. The transcript is now sized and scaled like the others. It is the one piece whose stored attribute is a trimmed tail rather than a clipped head, which is why it had been left as the remainder: `prompt.input.messages` holds only the turns that fit the attribute budget. So the service records `prompt.input.messages.tokens` measured over the full turn, before either the per-message clip or the oldest-first trim, and the meter prefers it, falling back to estimating from the preview for traces recorded without it. With nothing sizable at all the prompt stays whole in `messages` rather than being spread over segments with no evidence behind them, and the rounding drift goes to the largest segment so the bar still fills exactly. The existing 36 frontend tests pass unchanged: none supplied a transcript size, which is how the subset was able to look complete. Four cover it now, two of which fail against the previous logic, plus one on the service for the size surviving the trim. Verified: 40 frontend tests; 2027 Ruby runs, 0 failures (34 errors are this sandbox blocking outbound HTTP, an identical set before and after); rubocop 551 files, no offenses; bundle rebuilt, and rebuilding the unmodified base reproduces its committed bundle byte for byte, so the only bundle change here is this one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J9cRpVTtxA8QZBTmnYSC7V
Brings the 1.6.3 line up to main, which moved on after the branch was cut: the WebMock teardown fix (#459) and its investigation note. `webmock/minitest` installs its stub-registry reset by aliasing `teardown` at load time, so a class defining its own `teardown` without `super` silently suppresses `WebMock.reset!` and leaks stubs into later tests. Without this the release branch's own docs-example generation stays exposed to that seed-dependent failure, which is what broke the Pages deploy on main. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J9cRpVTtxA8QZBTmnYSC7V
Picks up the WebMock teardown fix the release line just took from main, so this branch's own test runs are not exposed to the leaked-stub failure and CI validates against the base it actually merges into. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J9cRpVTtxA8QZBTmnYSC7V
TonsOfFun
marked this pull request as ready for review
September 17, 2026 22:21
…tribution feat(telemetry): size tool schemas on the prompt span Records real *.tokens sizes on the prompt span before content is truncated, attributes MCP schemas apart from the toolbox's, and apportions the provider's prompt_tokens across the meter's segments — transcript included — instead of charging the approximation error to a single segment.
…them `Runner` already accepted an `around_evaluation:` wrapper and already owned a `run_id` in its `metadata:`, and a telemetry backend already offers a per-block agent scope that can report the trace it opened. Nothing joined them, so every host that wanted a report row to link to the conversation behind it wrote the same correlation plumbing itself. `ActiveAgent::Evals::Correlation` is that join. A run mints a `run_id`, each evaluation a `result_id`, and both ride every trace opened inside them as `eval.`-prefixed attributes. The trace ids travel the other way onto `result.replay.metadata`: `trace_id` for the replay, `judge_trace_ids` for the judge calls that graded it. The verdict, being a judge call outside any evaluation, lands on the run metadata a Report carries rather than on whichever result was evaluated last. The tracer is injected, so the module takes on no telemetry dependency and `require "active_agent/evals"` still loads on its own — covered by a test that drives a correlated run out of process. Hand the object to `Runner.new(around_evaluation:)` directly; a plain lambda there is unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…apters The adapter guide told hosts they own usage accounting, which stopped being true when the engine began metering each adapted result. A host following it now double-counts every replay. It also predates provider validation, so it left adapters re-checking what the runner already rejects. Documents replace_scenarios!'s on_removed: option alongside them, since a host syncing a suite from its own catalog is the caller that needs :disable. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Brings #460 onto the release line: adapted replays are metered as one :execution each, ModelSpec validates the provider before the adapter is called, and replace_scenarios! keeps dropped scenarios via on_removed:. It was merged to main rather than to this branch, so the 1.6.3 tag — which is cut here, not on main — would otherwise ship without it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J9cRpVTtxA8QZBTmnYSC7V # Conflicts: # CHANGELOG.md
`Judge` calls its host-supplied block with only `instructions:` and `prompt:`,
but it serves three different calls — scoring an answer, recommending a fix for
a failed scenario, and writing the run's verdict. A host that wants to trace
them apart, budget them apart, or grade with a cheaper model than it writes the
verdict with has no way to tell which one it is in.
The only signal was the instructions string, so hosts matched against the gem's
own constants:
when ActiveAgent::Evals::Judge::RECOMMEND_INSTRUCTIONS then 'recommend'
That couples a host to prose the gem is free to reword, and rewording it does
not break anything loudly — every such host falls through to its `else` branch
and mislabels traces, which is worse than an error.
A block that accepts `kind:` now receives `:score`, `:recommend` or `:verdict`.
The block signature is public API, so the keyword reaches only a block that
names it or collects `**`. A block taking exactly `instructions:` and `prompt:`
would raise ArgumentError on a third keyword — and `ask` rescues StandardError,
so that would not even surface as an error: it would count as a judge failure
and quietly degrade the run to rule scoring. The existing judge suite, whose
`fake_judge` helper takes two keywords, covers that path.
Closes #462
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J9cRpVTtxA8QZBTmnYSC7V
`ActionAgent::Agent` declared five associations but not the one a host most
often wants. Generations hang off `AgentContext` polymorphically, so reading
them meant knowing that shape and writing the join by hand:
AgentGeneration.joins(:agent_context)
.where(AgentContext.table_name => { contextable: agent })
The engine already wrote exactly that, in `EvaluationRunnerService#sample_generations`
— but as a private method on a service, so a host could not reuse it and copied
the join instead, coupling itself to an implementation detail of how contexts
are modelled. A change to that shape would then break host code silently rather
than at the engine boundary.
`agent.generations` is now that one definition, and `sample_generations` uses
it. It stays a scope, so `agent.generations.count` answers the "has this agent
actually run?" question the dashboard's own views had no cheap way to ask.
Deliberately no `dependent:` on `agent_contexts`. The issue suggested
`dependent: :destroy` and the sibling associations use it, but destroying an
agent has never taken its conversations with it, and quietly starting to delete
recorded history in a patch release is not something an association added for
reading should do. A test pins the current behaviour; changing it is a separate
call.
Closes #464
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J9cRpVTtxA8QZBTmnYSC7V
# Conflicts: # CHANGELOG.md
…ation feat(evals): correlate evaluation traces with the result that caused them Adds ActiveAgent::Evals::Correlation, joining Runner's around_evaluation: hook and run metadata to a telemetry backend's per-block agent scope, so a report row links back to the conversation behind it. The tracer is injected, so the module takes on no telemetry dependency.
# Conflicts: # CHANGELOG.md
# Conflicts: # CHANGELOG.md
feat(evals): tell a judge block which call it is serving A Judge block accepting kind: now receives :score, :recommend or :verdict, so a host can trace, budget or model the three calls separately instead of matching on the gem's own instruction constants. The keyword reaches only a block that names it or collects **, so existing two-keyword judges are unaffected. Closes #462
# Conflicts: # CHANGELOG.md
…ciation feat(dashboard): give Agent a generations association agent.generations replaces the polymorphic AgentContext join hosts were hand-writing, and EvaluationRunnerService#sample_generations now uses it so there is one definition. No dependent: on agent_contexts — destroying an agent has never taken its conversations with it, and a read-oriented association should not change that. Closes #464
docs(evals): metering, provider validation and on_removed for host adapters Corrects the adapter guide's usage-accounting claim, which #460 made wrong: the engine now meters each adapted result as one :execution, so a host following the old text double-counts every replay. Also documents the provider validation that runs before the adapter is called, and adds prose for replace_scenarios! and its on_removed: option.
The section carried only the boilerplate line while work was still landing. Six Added and three Fixed entries now sit under it, so it gets the summary 1.6.2 and 1.6.1 have: what the release is about, and what an upgrader has to do. Dated for the tag rather than the day the branch was cut. Notes the two things an adapter must now remove — its own record_usage call, which double-counts since the engine meters adapted replays, and its own provider allow-list check, which the runner's validation makes dead code. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J9cRpVTtxA8QZBTmnYSC7V
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Integration branch for the 1.6.3 cut, opened the way #455 was for 1.6.2.
release/1.6.3is cut frommainat b8c741d, which already carries everything tagged asv1.6.2. Merging this bringsmainup to the release line.How to land work in 1.6.3
release/1.6.3and open the PR with--base release/1.6.3.## [1.6.3], not## [Unreleased].## [Unreleased]on this branch is for what comes after the release.## [1.6.3]summary paragraph and date, tagv1.6.3on the branch head, publish both gems, then merge this PR.What is in it
*.tokenssizes before content is truncated, MCP schemas are attributed apart from the toolbox's, and the meter apportions the provider'sprompt_tokensacross its segments — the transcript among them, sized before the span trims the history.## [1.6.3]changelog section.mainhas also been merged in, bringing the WebMockteardown/superfix from #459 and its investigation note.webmock/minitestinstalls its stub-registry reset by aliasingteardownat load time, so a class defining its ownteardownwithoutsupersilently suppressesWebMock.reset!and leaks stubs into later tests. Without it the release line's own docs-example generation stayed exposed to that seed-dependent failure — the one that broke the Pages deploy onmain. It is a test-only change, so it carries no changelog entry.Remaining before this merges
## [1.6.3]summary paragraph and the date. The section currently holds only the boilerplate line, with feat(telemetry): size tool schemas on the prompt span #458's### Added/### Fixedentries filed under it.v1.6.3on the branch head and publish both gems.Open questions
activeagentandactionagentat 1.6.2, with 1.6.1 skipped — but thepublishjob inrelease.ymlstill fails atrubygems/configure-rubygems-credentialswith "No trusted publisher configured for this workflow", as it has on every tag back tov1.5.0. Either a trusted publisher gets configured on rubygems.org for both gems, or 1.6.3 is pushed by hand like its predecessors. Maintainer call.Considered and held back
The other open PRs were reviewed for this cut and none of them belong in it. #352 (Atlas Cloud provider) and #328 (
allowed_tools/require_approval) both add public surface, which would make this a minor rather than a patch; #328 is also red on 5 of 6 checks. #423 (json< 3→< 4) is green, but json 3.0 is a breaking major — it dropscreate_additions,JSON.fast_generateandJSON.unparse, and rejects duplicate keys — which is not a patch-release change for a gem that serializes JSON on every prompt span. #349 (minitest~> 6.0) is dev-only and green, but its run predates the two integration checks and so has never exercised the current suite. #313 (llms.txt) has never had CI run on it.Verified
After #458 landed: the full suite on the merge result is 2027 runs, 0 failures, and rubocop reports no offenses across 551 files. The context meter was checked against the merged code directly — a 20k-token trace with a 500-token system prompt and a 19,250-token history now reports those sizes, rather than 13,333 for instructions and nothing for the history.
🤖 Generated with Claude Code
https://claude.ai/code/session_01J9cRpVTtxA8QZBTmnYSC7V