Test job pooling configurations for scenario evaluations (sequential today)

[TonsOfFun]

Scenario evaluations run strictly sequentially. A 33-question catalog takes ~2.5–3.5 minutes of wall clock that is almost entirely spent waiting on one provider call at a time, and a 100-question catalog is ~10 minutes. The work is IO-bound and embarrassingly parallel, so this is throughput left on the floor.

Worth testing pooling configurations against real suites rather than picking a number — the same exercise we ran for the RubyKaigi ragents demo.

Where the sequencing is

ActiveAgent::Evals::Runner#call:

results = @scenarios.flat_map do |scenario|
  @models.map do |spec|
    evaluate_with_context(scenario, spec).tap { |result| @on_result&.call(result) }
  end
end

EvaluationRunJob enqueues one job for the whole suite, not one per scenario, so the queue's own concurrency never comes into play. Measured on a live host: 4.6s/scenario on a light suite, 6.5s/scenario on a tool-heavy one (13 scenarios in 84.9s).

What to test

The interesting question is not "add threads" but which shape wins, and where the ceiling is:

1. In-runner thread pool — a bounded pool inside Runner#call, width configurable. Simplest, keeps one job per run, keeps the Report assembly where it is.
2. Job-per-scenario fan-out — EvaluationRunJob enqueues one job per (scenario × model) and a completion check assembles the Report. Uses the host's existing queue concurrency, survives a worker restart, but needs a barrier and makes partial-run reporting harder.
3. Job-per-model, sequential within — a middle ground for multi-model comparisons, where each cohort is independent.

Vary pool width (1, 2, 4, 8, 16) against a real catalog and record wall clock, provider error rate, and cost. The useful output is a recommended default plus the shape of the curve where it stops helping.

Constraints any design has to respect

• on_result is not thread-safe. ScenarioEvaluationRunner closes over a bare Array:

  persisted = []
  on_result = lambda do |result|
    raise ArgumentError, "..." unless expected.include?(pair) && !persisted.include?(pair)
    persist(run, records.fetch(result.scenario.key), result)
    persisted << pair
  end

  Concurrent results race both the duplicate check and the append.

• Host adapters may hold thread-local state. The scenario_evaluation_adapter_resolver path (fix(evals): preserve grades, import suites, and publish reports #414) lets a host supply its own runner; one such adapter wraps replays in PublishingDomain.with_publishing_domain, which stores the domain thread-locally. A worker thread would not inherit it, so any pooling has to either propagate that context explicitly or leave the parallelism decision to the adapter.

• Provider rate limits become the real ceiling. 33-wide against one provider key will rate-limit before it saturates anything else. A pool width that is fast on a mock provider may be slower end-to-end on a live one once retries start.

• ActiveRecord connection pool. Each worker persisting a result needs a connection; pool width above ActiveRecord::Base.connection_pool.size trades provider waiting for connection waiting.

Why it matters beyond speed

A catalog that takes minutes is a catalog people run rarely. The evaluation loop is most useful when it is cheap enough to run on every agent change, which is a throughput problem before it is a features problem.

Related

• Execute a host's MCP tools from the dashboard, so hosts do not need their own evaluation runner #419 / feat(dashboard): call a host's MCP tools instead of only describing them #420 — host MCP tool execution; a tool-heavy suite is exactly the case where per-scenario latency is dominated by IO.
• fix(evals): preserve grades, import suites, and publish reports #414 — the adapter hook a host runner plugs into, and the second constraint above.

[TonsOfFun]

Picking option 2 (job-per-scenario fan-out) as the direction, rather than an in-runner thread pool.

The reason is that concurrency should be the host's decision, not the engine's. An engine-owned thread pool hard-codes a width into a library that gets mounted into apps whose queue backends, worker counts and provider quotas we do not know. Fanning out over Active Job instead means an evaluation runs at whatever concurrency the mounting app already configured for its :agents queue — a Solid Queue app with 8 workers gets 8-wide for free, a single-worker dev box stays sequential, and neither needs a setting in the engine.

That also inherits the queue's existing operational properties rather than reimplementing them: retries and backoff, worker-restart survival (a lost job is redriven; a lost thread loses the run), per-queue concurrency limits, and whatever observability the host already has on its queues. AgentExecutionJob on queue_as :agents is the natural unit — the fan-out enqueues one per (scenario × model).

The work is then:

• A barrier. EvaluationRunJob stops being the thing that runs the suite and becomes the thing that enqueues the children; the run completes when the last child persists its result. A counter on the run (expected vs. persisted) is enough — no need for the parent to block a worker while it waits.
• Make on_result safe under concurrency. Today it closes over a bare Array (persisted << pair) and does its duplicate check against that array; with independent jobs the natural fix is to move both to the database — a uniqueness constraint on (run, scenario, model) makes the duplicate check atomic and idempotent, which a redriven job needs anyway.
• Report assembly moves to the barrier, after the last result lands, instead of being the return value of Runner#call.
• Propagate host context explicitly. The scenario_evaluation_adapter_resolver path stores things thread-locally (Sparkle's adapter sets PublishingDomain.with_publishing_domain); a separate job does not inherit the enqueuing thread's locals, so whatever the adapter needs has to be serialized into the job arguments and re-established in perform.
• Keep a sequential path. perform_now in a loop should remain available for tests and for hosts that want it, so the change is a scheduling strategy rather than a hard dependency on a queue being drained.

Rate limits stay the real ceiling, but they become the host's ceiling to manage with the tools it already has (queue concurrency limits, a dedicated queue) rather than a number the engine guesses.

(Generated using Claude Code)

[TonsOfFun]

Scheduling note: this is 1.6, not 1.5.

1.5 is a minimal release cut to get a host app off path:/git: Gemfile pins and back onto released gems, so it carries only what that needs. The fan-out changes the evaluation runner's completion contract, which deserves its own review cycle rather than riding along with a packaging release.

The three constraints to preserve, verified against main today:

• on_result is not thread-safe (ScenarioEvaluationRunner, the allowed/persisted Sets): the guard reads persisted and then appends to it with no lock, so concurrent results race both the duplicate check and the append.
• The run enforces an exact completion contract: it raises unless reported and persisted each equal expected. A fan-out has to satisfy that from N independent jobs, which is what makes the barrier and idempotent persistence load-bearing rather than nice-to-have — a redriven job must not trip the duplicate raise.
• Host adapters hold thread-local state: the scenario_evaluation_adapter_resolver path lets a host supply its own runner, and one such adapter wraps replays in PublishingDomain.with_publishing_domain. A separate job does not inherit the enqueuing thread's locals, so that context has to be serialized into the job arguments and re-established in perform.

(Generated using Claude Code)