Skip to content

Test model and effort selection with <TestAgent> #984

Closed

Description

@taras

Story

As a workflow author, I want Markdown tests to exercise explicit Session model
and effort selection through <TestAgent>, so I can test the workflow's real
Agent/Session/Prompt path without calling a model provider or changing the
workflow's configuration for the test.

A scenario declares fictional model IDs and their effort choices, then tests a
workflow containing an ordinary configured Session:

<Agent name="reviewer">
  <Session name="review" model="fixture-model" effort="high">
    <Prompt>Review the supplied change.</Prompt>
  </Session>
</Agent>

The scenario observes the selected model and effort when its prompt starts.
The exact scenario configuration syntax belongs to the design of this story;
the example above uses the existing Session API.

Current gap

Session model and effort selection is delivered by #828 / #829. TestAgent's ACP
worker supplies no model or effort choices and no configuration-selection
behavior. A workflow specifying a model therefore refuses with Model choices are unavailable for codex. before the mocked prompt can run.

This blocked the local phase/ticket workflow tests while reviewing #896 / #983.
A test-only role boundary can verify packet routing, but it does not test the
actual configured Session path. Removing the production model prop would leave
that behavior untested.

Acceptance

  • A TestAgent scenario can declare model choices, the selected model, and effort
    choices for each model. IDs are supplied by the test rather than tied to a
    real provider's current catalog.
  • A configured Session applies and verifies the specified model, refreshes that
    model's effort choices, applies the specified effort, and only then runs the
    scenario prompt. The scenario can assert the configuration actually consumed.
  • Selecting another model uses its declared effort choices. An unknown model
    or unsupported effort refuses before the scenario prompt, using the existing
    Session diagnostics.
  • Scenarios can represent unavailable configuration choices. A Session omitting
    model and effort retains the existing unconfigured behavior.
  • Configuration state belongs to the scenario's existing session and isolation
    lifetime. Reusing a named session preserves its identity; independent test
    scenarios do not share configuration state.
  • The test completes through ordinary TestAgent ACP routing without a real
    provider, provider credentials, a production configuration override or a
    substituted Phase.Role implementation.

Evidence

Add focused TestAgent component/ACP-worker coverage and a Markdown integration
test containing a configured Session. Include a model change with different
effort choices and a negative control proving invalid selection starts no
prompt. Run the affected files with deno task test <test-file> while
implementing; required CI owns exhaustive runtime coverage.

The observed implementation boundaries are
packages/test-agent/src/worker/acp-server.ts and the existing Session
configuration contract in packages/acp/src/session-configuration.ts.

This story adds test-provider capability. It does not change Session ownership,
configuration semantics, live provider/model selection, or workflow policy.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions