feat(evals): a scenario replay runs as the evaluation's owner

CHANGELOG.md

@@ -9,6 +9,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+- **An evaluation replay runs as the evaluation's owner.** The scenario runner
+  handed `Agent#test_execute` no caller, so every tool a replay called ran
+  unattributed and a host scope answered empty — the suite graded an agent
+  that never saw a row. When agents are owned per user, a replay now runs as
+  the user who owns the evaluation, as a run over MCP runs as the key's
+  owner. A multi-tenant install still replays unattributed (an account is who
+  is billed, not who is allowed) unless a host adapter runs the suite itself.
 - **The MCP facade serves the host's schema tools directly.** `tools/list`
   at `POST <mount>/mcp` now offers every tool the dashboard's discovered
   `ActiveAgent::SchemaTools` classes generate — `find_<records>`,

actionagent/app/services/action_agent/scenario_evaluation_runner.rb

@@ -147,7 +147,8 @@ def replay(scenario, spec)
       agent_run = @evaluation.agent.test_execute(
         scenario.prompt,
         model_override: spec.model,
-        provider_override: spec.provider
+        provider_override: spec.provider,
+        actor: replay_actor
       )
 
       Evals::Replay.new(
@@ -162,6 +163,21 @@ def replay(scenario, spec)
       )
     end
 
+    # The caller a replay runs on behalf of: the evaluation's owner, when the
+    # install owns agents per user. A run with no caller reads, through any
+    # host scope, as "no access" — every tool answers empty and the suite
+    # grades an agent that never saw a row — so the person the evaluation
+    # belongs to is the right default, as the key's owner is over MCP. An
+    # account is who is billed, not who is allowed (see Api::BaseController
+    # #agent_actor), so a multi-tenant install replays unattributed unless a
+    # host adapter (ActionAgent.scenario_evaluation_adapter_resolver) runs
+    # the suite itself.
+    def replay_actor
+      return nil if ActionAgent.multi_tenant? || ActionAgent.user_class.blank?
+
+      owner
+    end
+
     # Each tool call the run made, rebuilt from the run's progress events
     # (AgentRun#append_event pairs a "started" event with its "done"/"error"
     # by eid). Falls back to the bare names in the run's metadata for a run

actionagent/test/scenario_evaluation_runner_test.rb

@@ -29,6 +29,42 @@ def build_suite(**attributes)
     evaluation
   end
 
+  # Captures the actor each replay hands Agent#test_execute.
+  def capture_replay_actors(evaluation)
+    seen = []
+    agent = evaluation.agent
+    fake_run = agent.agent_runs.create!(status: :complete, trace_id: SecureRandom.uuid, input_prompt: "x", output: "done")
+    agent.define_singleton_method(:test_execute) { |*, **options| seen << options[:actor]; fake_run }
+    yield
+    seen
+  end
+
+  test "a replay runs as the evaluation's owner when agents are owned per user" do
+    previous = ActionAgent.user_class
+    ActionAgent.user_class = "User"
+    user = User.create!(name: "Owner", email: "owner-#{SecureRandom.hex(4)}@example.com", age: 30)
+    evaluation = build_suite(user_id: user.id)
+
+    actors = capture_replay_actors(evaluation) { evaluation.run!(keys: [ "find_1" ]) }
+
+    assert_equal [ user ], actors, "every tool the replay calls should be scoped to the owner"
+  ensure
+    ActionAgent.user_class = previous
+    user&.destroy
+  end
+
+  test "a replay is unattributed when the install has no owner model" do
+    previous = ActionAgent.user_class
+    ActionAgent.user_class = nil
+    evaluation = build_suite
+
+    actors = capture_replay_actors(evaluation) { evaluation.run!(keys: [ "find_1" ]) }
+
+    assert_equal [ nil ], actors
+  ensure
+    ActionAgent.user_class = previous
+  end
+
   # Runs the block with the host's credentials for +provider+ blanked, so a
   # replay through it fails at the execution service's credential gate rather
   # than at the network. CI decrypts the reference host's credentials, so

docs/framework/dashboard.md

@@ -370,7 +370,10 @@ candidate needs credentials the same way an agent run does — the owner's
 provider key or the host app's `config/active_agent.yml`.
 
 A run is queued (`EvaluationRunJob`) and its results land as each replay
-finishes. The suite card opens onto the three questions asked of a run, in
+finishes. Each replay runs as the evaluation's owner when agents are owned
+per user, so a tool scoped to its caller sees that user's rows; a
+multi-tenant install replays unattributed unless a host adapter
+(`ActionAgent.scenario_evaluation_adapter_resolver`) runs the suite itself. The suite card opens onto the three questions asked of a run, in
 that order — is it getting better, which model, what do I fix — and then
 the evidence behind them.
 

test/dummy/config/master.key.stale-2026-09-12

@@ -0,0 +1 @@
+d8fd3e143122c1da1a52724ca8340710