feat: let the engine run host agent classes, and teach SchemaTools about Rails enums

CHANGELOG.md

@@ -7,6 +7,77 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Added
+
+- **The Evaluations page is rebuilt around runs** (`actionagent`). Evaluations
+  are the top level; every run is kept and listed with its movement against
+  the run before it (`+3 passed vs #2`, `partial run`, `#1 failed`), and a
+  sampling evaluation's run opens to a page of its own at
+  `<mount>/evaluations/:id/runs/:run_id` — a scorecard per model cohort, the
+  judge's verdict, the criteria × models matrix and what the run asks to fix.
+  A scenario suite's runs are the same full-width list; a row selects the run
+  the suite's model scorecards, fix items and scenario matrix show.
+- **What a run cost is two figures, not one.** The agent's spend — what the
+  replayed or sampled interactions cost to serve, with a `per_interaction`
+  rate, the operating cost a per-conversation budget is set against — is
+  reported apart from the judge's, the judge model's own calls, which run
+  agent-to-agent and offline. Every judge call is metered under what it was
+  for (`scores["_judge_usage"]`: calls, tokens, estimated cost and how many
+  calls scored, recommended, ruled or authored KPIs), and `EvaluationRun#usage`
+  carries both sides. The page shows them on every run row, on the run, on a
+  page tile and in the footer, so the cost of operating an agent is never
+  inflated by the cost of checking it.
+- A generation-sampling run records `scores["_cohorts"]`: per model, how many
+  generations were sampled, how many cleared every criterion, their latency
+  and tokens, and what those interactions cost to serve.
+- `GET /api/evaluations` carries `run_count` and a `previous_run` summary per
+  evaluation, and every serialized run its `number` in the evaluation's
+  history, oldest first.
+- The dashboard's object lists hold their metric columns in place: a trace,
+  interaction or evaluation run with nothing in a column prints a dash there
+  rather than sliding its neighbours over (`MetaStrip`).
+- `ActiveAgent::Base.rendered_instructions` renders an agent's instructions
+  outside a generation, for a dashboard mirroring the class and for tests
+  asserting what a model is told. Both otherwise reached a private renderer
+  through `send`.
+- `ActionAgent::AgentSync` mirrors host agent classes into dashboard `Agent`
+  records, setting the `agent_class_name` that `AgentRelease` already expects a
+  host to have written. The code owns what an agent is (name, description,
+  instructions, tools — rewritten every sync); the operator owns how it runs
+  (provider, model, status — set on create and preserved), so a model chosen in
+  the dashboard survives the next deploy.
+- `ActionAgent.run_host_agent_classes` (default `false`) runs an agent that
+  mirrors a host class as that class, rather than as one rebuilt from the
+  record's `tools` and `instructions` columns. Dashboard-authored agents, which
+  name no class, keep using the dynamic runtime either way; a class name that no
+  longer resolves falls back to it rather than failing the run.
+
+### Changed
+
+- The engine's judge blocks take `ActiveAgent::Evals::Judge`'s `kind:`, so a
+  scenario run's score, recommendation and verdict calls are metered apart.
+
+## [1.6.4] - 2026-09-22
+
+Releases `activeagent` and `actionagent` 1.6.4 from one tag. A patch on 1.6.3
+carrying one fix to `SchemaTools`, for a filter that answered confidently and
+wrongly instead of failing.
+
+### Fixed
+
+- A Rails enum is offered to the model as its names (`{type: "string", enum:
+  [...]}`) instead of the integer backing it. `SchemaGenerator` reads enums from
+  inclusion validators and never consulted `defined_enums`, so a `status` column
+  reached the model as a bare integer with no labels.
+- A filter value outside an enum — alone or inside an IN list — is rejected,
+  naming the valid values, instead of matching no rows. `status: "pending"`
+  returned `{count: 0}`, which an agent reports as a fact, indistinguishable
+  from "none match". Same reasoning as the unknown-operator rejection in
+  `range_predicates!`.
+- An enum is no longer offered the range form. Its integer backing is a
+  declaration-order artefact, so `status: {gt: 1}` was a meaningless filter that
+  still returned a confident count.
+
 ## [1.6.3] - 2026-09-18
 
 Releases `activeagent` and `actionagent` 1.6.3 from one tag.

actionagent/app/controllers/action_agent/api/evaluations_controller.rb

@@ -46,14 +46,20 @@ def index
         render json: { evaluations: evaluations.map { |evaluation| serialize(evaluation) } }
       end
 
+      # Runs listed per evaluation on GET /api/evaluations/:id. The rest of
+      # the history stays reachable by run id; `run_count` says how long it is.
+      RUN_HISTORY_LIMIT = 20
+
       # GET /api/evaluations/:id
       def show
         evaluation = evaluations_scope.find(params[:id])
+        run_count = evaluation.evaluation_runs.count
+        runs = evaluation.evaluation_runs.recent.limit(RUN_HISTORY_LIMIT).to_a
 
         render json: {
           evaluation: serialize(evaluation).merge(
             scenarios: evaluation.scenarios.ordered.map(&:as_json_summary),
-            runs: evaluation.evaluation_runs.recent.limit(20).map { |run| serialize_run(run) }
+            runs: runs.each_with_index.map { |run, index| serialize_run(run, number: run_count - index) }
           )
         }
       end
@@ -98,8 +104,12 @@ def create
       def run
         evaluation = current_evaluation
         run = start_run(evaluation, selection_params)
+        evaluation.reload
 
-        render json: { evaluation: serialize(evaluation.reload), run: serialize_run(run) }
+        render json: {
+          evaluation: serialize(evaluation),
+          run: serialize_run(run, number: evaluation.evaluation_runs.count)
+        }
       end
 
       # GET /api/evaluations/:id/runs/:run_id
@@ -116,7 +126,8 @@ def show_run
 
         render json: {
           evaluation: serialize(evaluation),
-          run: serialize_run(run).merge(results: results.map(&:as_json_summary), fix_items: safe_fix_items(run))
+          run: serialize_run(run, number: run_number(evaluation, run))
+            .merge(results: results.map(&:as_json_summary), fix_items: safe_fix_items(run))
         }
       end
 
@@ -328,7 +339,9 @@ def compare_models_param
       end
 
       def serialize(evaluation)
-        latest = evaluation.latest_run
+        # size reads the preloaded association on index and COUNTs elsewhere.
+        run_count = evaluation.evaluation_runs.size
+        latest, previous = recent_runs(evaluation, 2)
 
         {
           id: evaluation.id,
@@ -344,22 +357,50 @@ def serialize(evaluation)
           scenario_count: evaluation.scenarios.size,
           scenario_groups: evaluation.scenario_suite? ? evaluation.scenario_groups : [],
           created_at: evaluation.created_at.iso8601,
-          latest_run: latest ? serialize_run(latest) : nil
+          run_count: run_count,
+          latest_run: latest ? serialize_run(latest, number: run_count) : nil,
+          # Just enough of the run before it for the list to show movement
+          # ("+3 passed vs #2") without a request per evaluation.
+          previous_run: previous ? serialize_run_summary(previous, number: run_count - 1) : nil
         }
       end
 
-      def serialize_run(run)
-        {
-          id: run.id,
-          status: run.status,
+      # Newest first. Sorts the preloaded association when index loaded it
+      # rather than issuing one ORDER BY query per evaluation.
+      def recent_runs(evaluation, limit)
+        runs = evaluation.evaluation_runs
+        if runs.loaded?
+          runs.sort_by { |run| [ run.created_at, run.id ] }.reverse.first(limit)
+        else
+          runs.recent.limit(limit).to_a
+        end
+      end
+
+      # A run's position in its evaluation's history, oldest = 1.
+      def run_number(evaluation, run)
+        evaluation.evaluation_runs.where("created_at < ? OR (created_at = ? AND id <= ?)", run.created_at, run.created_at, run.id).count
+      end
+
+      # `number` is the run's position in its evaluation's history, oldest =
+      # 1, so the dashboard can say "Run #3" and "vs #2".
+      def serialize_run(run, number: nil)
+        serialize_run_summary(run, number: number).merge(
           scores: run.scores,
           selection: run.selection,
           models: run.models,
+          usage: run.usage,
+          error_message: run.error_message
+        )
+      end
+
+      def serialize_run_summary(run, number: nil)
+        {
+          id: run.id,
+          number: number,
+          status: run.status,
           average_score: safe_average_score(run),
           samples_evaluated: run.samples_evaluated,
           samples_passed: run.samples_passed,
-          usage: run.usage,
-          error_message: run.error_message,
           completed_at: run.completed_at&.iso8601,
           created_at: run.created_at.iso8601
         }

actionagent/app/models/action_agent/evaluation_run.rb

@@ -79,26 +79,74 @@ def average_score
       (values.sum.to_f / values.size).round(3)
     end
 
-    # Aggregate usage over the run's scenario results, for display after a
-    # run: estimated cost, token totals, summed model time, and the run's
-    # wall-clock runtime. Returns nil for a generation-sampling run, which
-    # replays nothing itself.
+    # The judge's own spend on this run — calls, tokens, estimated cost and
+    # how many calls served each purpose — as the runner recorded it; nil
+    # for a run that never asked a judge.
+    def judge_usage
+      value = scores&.dig("_judge_usage")
+      value.is_a?(Hash) ? value : nil
+    end
+
+    # Per-model summaries of a generation-sampling run's cohorts, keyed by
+    # model; empty for a scenario run or a run recorded before they were.
+    def cohorts
+      value = scores&.dig("_cohorts")
+      value.is_a?(Hash) ? value : {}
+    end
+
+    # What the run spent, for display after it: the agent's side and the
+    # judge's, kept apart because they answer different questions.
+    #
+    # The agent's side is the operating figure — what the interactions cost
+    # to serve. For a scenario run that is the replays' estimated cost,
+    # tokens and summed model time (`replays` of them); for a
+    # generation-sampling run it is the sampled generations' (`samples`),
+    # which were served before the run and cost it nothing. `per_interaction`
+    # is that cost spread over the interactions, the number a per-conversation
+    # budget is set against.
+    #
+    # `judge` is the evaluation's own overhead: the judge model's calls
+    # (scoring, recommending, the verdict, authoring KPIs), which run
+    # agent-to-agent and offline. It is present only when a judge was asked.
+    #
+    # Returns nil for a run that recorded nothing on either side.
     def usage
       totals = scenario_results.pick(
         Arel.sql("COUNT(*)"), Arel.sql("SUM(cost)"), Arel.sql("SUM(input_tokens)"),
         Arel.sql("SUM(output_tokens)"), Arel.sql("SUM(duration_ms)")
       )
       replays = totals&.first.to_i
-      return nil if replays.zero?
+      judge = judge_usage
+      runtime_ms = completed_at.present? ? ((completed_at - created_at) * 1000).round : nil
 
-      {
-        replays: replays,
-        cost: totals[1]&.to_f,
-        input_tokens: totals[2].to_i,
-        output_tokens: totals[3].to_i,
-        model_time_ms: totals[4].to_i,
-        runtime_ms: completed_at.present? ? ((completed_at - created_at) * 1000).round : nil
-      }
+      if replays.positive?
+        cost = totals[1]&.to_f
+        {
+          replays: replays,
+          cost: cost,
+          per_interaction: cost && (cost / replays).round(6),
+          input_tokens: totals[2].to_i,
+          output_tokens: totals[3].to_i,
+          model_time_ms: totals[4].to_i,
+          runtime_ms: runtime_ms,
+          judge: judge
+        }.compact
+      elsif cohorts.any?
+        samples = cohorts.values.sum { |cohort| cohort["samples"].to_i }
+        costs = cohorts.values.filter_map { |cohort| cohort["cost"] }
+        cost = costs.any? ? costs.sum.to_f.round(6) : nil
+        {
+          samples: samples,
+          cost: cost,
+          per_interaction: cost && samples.positive? ? (cost / samples).round(6) : nil,
+          input_tokens: cohorts.values.sum { |cohort| cohort["input_tokens"].to_i },
+          output_tokens: cohorts.values.sum { |cohort| cohort["output_tokens"].to_i },
+          runtime_ms: runtime_ms,
+          judge: judge
+        }.compact
+      elsif judge
+        { runtime_ms: runtime_ms, judge: judge }.compact
+      end
     end
 
     # Route templates for the report's fix item actions, relative to the

actionagent/app/services/action_agent/agent_execution_service.rb

@@ -500,6 +500,21 @@ def generate!
       tool_definitions = tool_schemas
       service = self
 
+      # A dashboard-authored agent has no Ruby class — it is rows: a tool
+      # selection, instructions typed in the builder. That is the common case
+      # and the runtime below builds a class for it.
+      #
+      # An agent mirrored from host code is the other case: the class exists,
+      # already declares its own tools (SchemaTools rosters, delegations) and
+      # renders its own instructions, and none of that is reachable through
+      # `tools` + `instructions` columns. Running the real class keeps the
+      # dashboard evaluating what production runs, instead of a rebuilt
+      # lookalike. Both runtimes stay; which one applies is decided by whether
+      # the class resolves.
+      if (host_class = resolved_host_class)
+        return run_host_class(host_class, actor: actor, action: action, run_trace_id: run_trace_id)
+      end
+
       agent_class = Class.new(ActiveAgent::Base) do
         # SolidAgent persists contexts under self.class.name; anonymous
         # classes would fail its agent_name presence validation.
@@ -677,6 +692,40 @@ def record_tool_spans(root_span, response)
       end
     end
 
+    # The host class this agent mirrors, when it names one that resolves to a
+    # runnable ActiveAgent::Base subclass. Anything else — no class name, a
+    # class that no longer exists, a name that resolves to something else — is
+    # nil, and the dynamic runtime handles the record as before.
+    #
+    # @return [Class, nil]
+    def resolved_host_class
+      return nil unless ActionAgent.run_host_agent_classes
+
+      name = @agent_record.agent_class_name.presence
+      return nil if name.blank?
+
+      klass = name.safe_constantize
+      klass if klass.is_a?(Class) && klass < ActiveAgent::Base
+    end
+
+    # Runs the host's own class. Its tools, delegations and instructions come
+    # from the code, so the engine supplies only what is the run's business:
+    # the caller, and the trace to correlate against.
+    def run_host_class(klass, actor:, action:, run_trace_id:)
+      generation = klass.as(actor).public_send(action, **host_action_arguments(klass, action))
+      generation.prompt_options[:trace_id] = run_trace_id if generation.respond_to?(:prompt_options)
+      generation.generate_now
+    end
+
+    # A code agent's action takes named arguments (`ask(question:)`), so the
+    # run's message is passed under the action's own keyword rather than as a
+    # bare message the signature would reject.
+    def host_action_arguments(klass, action)
+      contract = klass.try(:delegation_contracts)&.dig(action.to_sym)
+      keyword = contract&.try(:parameters)&.keys&.first
+      keyword ? { keyword.to_sym => user_text } : {}
+    end
+
     def provider_available?(name)
       # The gem's mock provider is a test double: accepted only in the test
       # environment so app runs can never store fabricated output.