Skip to content

Support model and effort configuration in TestAgent scenarios - #988

Merged
taras merged 5 commits into
mainfrom
agent/testagent-model-options-984-delivery
Oct 26, 2026
Merged

taras merged 5 commits into
mainfrom
agent/testagent-model-options-984-delivery

Conversation

@taras

@taras taras commented Oct 23, 2026

Copy link
Copy Markdown
Owner

Configured Sessions in Markdown tests can now select scenario-defined model and effort choices through the ordinary TestAgent ACP worker. Invalid or unavailable choices refuse before the scenario prompt, and each scenario retains its own configuration state.

This also includes the retained REPL source-admission fixes needed to run the workflow against an observed checkout while keeping native role sessions in their original directory.

Focused TestAgent, shared execution-harness, and REPL ownership tests passed. Both original independent reviewers passed signed head e4cedff5f8a0f7f96464a8fbb455b092822788e8; its merge with current main preserves the reviewed source tree.

Runner weight measurement is in progress: https://github.com/taras/executable.md/actions/runs/40032899108. The resulting artifact commit, refreshed final-head reviews, and required CI remain pending. This PR stays draft until those delivery gates pass.

Closes #984.

@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

PR #988: Support model and effort configuration in TestAgent scenarios

31 files, +1312 / -27

Scope

🔴 PR has 1339 lines changed. Split into focused PRs.

🟡 1339 lines changed. PRs under 400 receive more thorough review.

🟡 31 files changed. Are all changes related?

🟡 Changes span 6 directories.

Structural

🟡 Type declarations with no consumers: GateKind.
Symbol Declared at Refs in diff Why flagged
GateKind packages/test-agent/tests/worker-lifecycle.test.ts:168 1 referenced ≤1× within the added diff (pre-existing usages not counted)

Oxlint structural signals:

  • no-unnecessary-type-assertion ×5: packages/testing/src/execution-harness.ts, packages/test-agent/src/protocol.ts
  • no-unused-vars ×3: packages/test-agent/src/components.ts
  • no-empty-function ×2: packages/test-agent/src/components.ts, packages/test-agent/src/worker/run.ts

Slop

✅ Slop indicators look low.

Static Analysis

Oxlint: 30 diagnostics across 9 files (10 rules)
Density: 0.023 violations/added-line

unbound-method (6): packages/testing/src/execution-harness.ts, packages/test-agent/src/worker/bridge.ts, packages/test-agent/src/controller.ts
no-shadow (5): packages/test-agent/src/configuration.ts, packages/test-agent/src/controller.ts, packages/testing/src/execution-harness.ts
no-unnecessary-type-assertion (5): packages/testing/src/execution-harness.ts, packages/test-agent/src/protocol.ts
no-unused-vars (3): packages/test-agent/src/components.ts
no-floating-promises (3): packages/test-agent/src/components.ts, packages/test-agent/src/worker/run.ts
no-underscore-dangle (2): packages/test-agent/src/worker/acp-server.ts
no-empty-function (2): packages/test-agent/src/components.ts, packages/test-agent/src/worker/run.ts
consistent-return (2): packages/testing/src/child-configuration.ts, packages/testing/src/execution-harness.ts
consistent-function-scoping (1): packages/test-agent/src/worker/bridge.ts
no-unsafe-type-assertion (1): packages/testing/src/execution-harness.ts

Correctness

No extraneous code patterns detected.

Comment thread architecture.md Outdated
linked registration, common directory, object format and captured repository
identity before live targeting work, including before Dir can ensure a missing
root. Observation registrations remain separate from mutation/publication and
retained Files placements.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need this?

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the retained-REPL source prerequisite bundled with #984, rather than part of model/effort selection. The REPL was opened in one checkout while implementation source lived in another existing linked worktree; Dir alone did not admit that checkout to retained Git reads. --observe-worktree permits those four reads without granting Git mutations/publication. Existing histories now contain its version-1 placement records, so the final runtime must retain their reader. The paragraph should lead with this concrete use case.

* a path a later `<Dir>` sibling renders at.
*/
placements(): readonly ReplManagedPlacement[];
guard?(target: string): Operation<void>;

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is this for?

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This check runs before Dir can create its target directory. Without it, a deleted worktree could be recreated as an empty directory before Git rejects it. Completed replay bypasses the check; new work validates that the configured linked checkout still exists and matches its recorded identity. guard hides that purpose. I'll request validateObservationTarget as a required recorder method, keeping the deliberate no-op when no observation feature is installed. That correction is not implemented yet.

}
}

export function* useObservationLease(commonDirectory: string, checkout: string): Operation<void> {

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lease is not a permitted terminology

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed: this is an advisory lock, so the new helper should be named useObservationLock. It currently refuses a second participating XMD observation entry until the first ends; it does not exclude ordinary Git writers. Read-only Git safety alone does not justify that exclusivity—it is the separately accepted D3 ownership policy. I'll request the naming correction without changing its key or cleanup. Removing reader exclusivity would be a separate behavioral proposal.

/** The native repository work this entry asks for, credential-free. */
readonly operations: DenoRepositoryOperations;
prepare?(path: string): Operation<void>;
ownObservation?(target: ReplGitCheckout): Operation<void>;

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is this?

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the entry owner's callback for a new read of a restored observation checkout. It rechecks the current grant and linked-worktree identity and acquires the entry's observation lock. Replaying a saved page does neither. The only production constructor always supplies this callback, so the optional hook is unnecessary internal wiring. I'll make it a required dependency in the correction request while retaining the live validation.

Comment thread packages/cli/src/repl/repositories/deno/observations.ts Outdated
Comment thread packages/cli/src/repl/repositories/deno/observations.ts Outdated
@taras taras mentioned this pull request Oct 24, 2026
3 of 4 tasks
@taras
taras force-pushed the agent/testagent-model-options-984-delivery branch from d9c4380 to e016900 Compare October 26, 2026 17:03
@taras
taras marked this pull request as ready for review October 26, 2026 17:09
@taras
taras enabled auto-merge October 26, 2026 17:09
@taras
taras merged commit dfe4c73 into main Oct 26, 2026
44 checks passed
@taras
taras deleted the agent/testagent-model-options-984-delivery branch October 26, 2026 17:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Test model and effort selection with <TestAgent>

1 participant