Repository navigation
Test model and effort selection with <TestAgent> #984
Description
Story
As a workflow author, I want Markdown tests to exercise explicit Session model
and effort selection through <TestAgent>, so I can test the workflow's real
Agent/Session/Prompt path without calling a model provider or changing the
workflow's configuration for the test.
A scenario declares fictional model IDs and their effort choices, then tests a
workflow containing an ordinary configured Session:
<Agent name="reviewer">
<Session name="review" model="fixture-model" effort="high">
<Prompt>Review the supplied change.</Prompt>
</Session>
</Agent>The scenario observes the selected model and effort when its prompt starts.
The exact scenario configuration syntax belongs to the design of this story;
the example above uses the existing Session API.
Current gap
Session model and effort selection is delivered by #828 / #829. TestAgent's ACP
worker supplies no model or effort choices and no configuration-selection
behavior. A workflow specifying a model therefore refuses with Model choices are unavailable for codex. before the mocked prompt can run.
This blocked the local phase/ticket workflow tests while reviewing #896 / #983.
A test-only role boundary can verify packet routing, but it does not test the
actual configured Session path. Removing the production model prop would leave
that behavior untested.
Acceptance
- A TestAgent scenario can declare model choices, the selected model, and effort
choices for each model. IDs are supplied by the test rather than tied to a
real provider's current catalog. - A configured Session applies and verifies the specified model, refreshes that
model's effort choices, applies the specified effort, and only then runs the
scenario prompt. The scenario can assert the configuration actually consumed. - Selecting another model uses its declared effort choices. An unknown model
or unsupported effort refuses before the scenario prompt, using the existing
Session diagnostics. - Scenarios can represent unavailable configuration choices. A Session omitting
model and effort retains the existing unconfigured behavior. - Configuration state belongs to the scenario's existing session and isolation
lifetime. Reusing a named session preserves its identity; independent test
scenarios do not share configuration state. - The test completes through ordinary TestAgent ACP routing without a real
provider, provider credentials, a production configuration override or a
substituted Phase.Role implementation.
Evidence
Add focused TestAgent component/ACP-worker coverage and a Markdown integration
test containing a configured Session. Include a model change with different
effort choices and a negative control proving invalid selection starts no
prompt. Run the affected files with deno task test <test-file> while
implementing; required CI owns exhaustive runtime coverage.
The observed implementation boundaries are
packages/test-agent/src/worker/acp-server.ts and the existing Session
configuration contract in packages/acp/src/session-configuration.ts.
This story adds test-provider capability. It does not change Session ownership,
configuration semantics, live provider/model selection, or workflow policy.
Activity