diff --git a/CHANGELOG.md b/CHANGELOG.md index fd857211..d78b69fc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +- **An evaluation replay runs as the evaluation's owner.** The scenario runner + handed `Agent#test_execute` no caller, so every tool a replay called ran + unattributed and a host scope answered empty — the suite graded an agent + that never saw a row. When agents are owned per user, a replay now runs as + the user who owns the evaluation, as a run over MCP runs as the key's + owner. A multi-tenant install still replays unattributed (an account is who + is billed, not who is allowed) unless a host adapter runs the suite itself. - **The MCP facade serves the host's schema tools directly.** `tools/list` at `POST /mcp` now offers every tool the dashboard's discovered `ActiveAgent::SchemaTools` classes generate — `find_`, diff --git a/actionagent/app/services/action_agent/scenario_evaluation_runner.rb b/actionagent/app/services/action_agent/scenario_evaluation_runner.rb index d3718dda..a36f69ee 100644 --- a/actionagent/app/services/action_agent/scenario_evaluation_runner.rb +++ b/actionagent/app/services/action_agent/scenario_evaluation_runner.rb @@ -147,7 +147,8 @@ def replay(scenario, spec) agent_run = @evaluation.agent.test_execute( scenario.prompt, model_override: spec.model, - provider_override: spec.provider + provider_override: spec.provider, + actor: replay_actor ) Evals::Replay.new( @@ -162,6 +163,21 @@ def replay(scenario, spec) ) end + # The caller a replay runs on behalf of: the evaluation's owner, when the + # install owns agents per user. A run with no caller reads, through any + # host scope, as "no access" — every tool answers empty and the suite + # grades an agent that never saw a row — so the person the evaluation + # belongs to is the right default, as the key's owner is over MCP. An + # account is who is billed, not who is allowed (see Api::BaseController + # #agent_actor), so a multi-tenant install replays unattributed unless a + # host adapter (ActionAgent.scenario_evaluation_adapter_resolver) runs + # the suite itself. + def replay_actor + return nil if ActionAgent.multi_tenant? || ActionAgent.user_class.blank? + + owner + end + # Each tool call the run made, rebuilt from the run's progress events # (AgentRun#append_event pairs a "started" event with its "done"/"error" # by eid). Falls back to the bare names in the run's metadata for a run diff --git a/actionagent/test/scenario_evaluation_runner_test.rb b/actionagent/test/scenario_evaluation_runner_test.rb index 29ddb06d..9ec1b04a 100644 --- a/actionagent/test/scenario_evaluation_runner_test.rb +++ b/actionagent/test/scenario_evaluation_runner_test.rb @@ -29,6 +29,42 @@ def build_suite(**attributes) evaluation end + # Captures the actor each replay hands Agent#test_execute. + def capture_replay_actors(evaluation) + seen = [] + agent = evaluation.agent + fake_run = agent.agent_runs.create!(status: :complete, trace_id: SecureRandom.uuid, input_prompt: "x", output: "done") + agent.define_singleton_method(:test_execute) { |*, **options| seen << options[:actor]; fake_run } + yield + seen + end + + test "a replay runs as the evaluation's owner when agents are owned per user" do + previous = ActionAgent.user_class + ActionAgent.user_class = "User" + user = User.create!(name: "Owner", email: "owner-#{SecureRandom.hex(4)}@example.com", age: 30) + evaluation = build_suite(user_id: user.id) + + actors = capture_replay_actors(evaluation) { evaluation.run!(keys: [ "find_1" ]) } + + assert_equal [ user ], actors, "every tool the replay calls should be scoped to the owner" + ensure + ActionAgent.user_class = previous + user&.destroy + end + + test "a replay is unattributed when the install has no owner model" do + previous = ActionAgent.user_class + ActionAgent.user_class = nil + evaluation = build_suite + + actors = capture_replay_actors(evaluation) { evaluation.run!(keys: [ "find_1" ]) } + + assert_equal [ nil ], actors + ensure + ActionAgent.user_class = previous + end + # Runs the block with the host's credentials for +provider+ blanked, so a # replay through it fails at the execution service's credential gate rather # than at the network. CI decrypts the reference host's credentials, so diff --git a/docs/framework/dashboard.md b/docs/framework/dashboard.md index 188f89fd..053e47df 100644 --- a/docs/framework/dashboard.md +++ b/docs/framework/dashboard.md @@ -370,7 +370,10 @@ candidate needs credentials the same way an agent run does — the owner's provider key or the host app's `config/active_agent.yml`. A run is queued (`EvaluationRunJob`) and its results land as each replay -finishes. The suite card opens onto the three questions asked of a run, in +finishes. Each replay runs as the evaluation's owner when agents are owned +per user, so a tool scoped to its caller sees that user's rows; a +multi-tenant install replays unattributed unless a host adapter +(`ActionAgent.scenario_evaluation_adapter_resolver`) runs the suite itself. The suite card opens onto the three questions asked of a run, in that order — is it getting better, which model, what do I fix — and then the evidence behind them. diff --git a/test/dummy/config/master.key.stale-2026-09-12 b/test/dummy/config/master.key.stale-2026-09-12 new file mode 100644 index 00000000..080a89af --- /dev/null +++ b/test/dummy/config/master.key.stale-2026-09-12 @@ -0,0 +1 @@ +d8fd3e143122c1da1a52724ca8340710 \ No newline at end of file