Opening this for discussion before sending a PR, because the placement question is worth settling first and the answer changes the shape of the change.
What this is about
We have a set of multi-turn, tool-using benchmark lanes (tau2, tau3) plus a routing-accuracy scorer, built to answer a question the existing benchmark/ harness does not: when the router sends a turn to the weak tier, was that the right call?
benchmark/ today measures whether Terminal-Bench tasks pass through a routed server. These lanes measure routing decisions against per-task ground truth derived from floor and anchor arms. Complementary, but different enough that they may not want to be neighbours.
Question 1 — where in the tree?
Candidates we considered:
benchmark/benches/<lane>/ — what we prototyped. Reuses the existing benchmark/ conventions, and benchmark/*.md is already whitelisted in the CI paths filter. Downside: benchmark/benches/ reads oddly, and it puts two different evaluation philosophies under one root.benchmark/<lane>/ — flat, as siblings of closed_book_proxy/ and server-configs/. Simpler, but the lane runners are a different kind of artifact from the Harbor tooling already there.examples/experimental/<name>/ — the litellm precedent, with its own pyproject.toml/uv.lock. Best if the harness should carry dependencies that must stay out of the root lockfile.- A separate repo — these drive third-party harnesses that must be checked out separately anyway.
Note .gitignore already reserves benchmark/perf/results/ for a perf sub-harness that does not exist yet, which suggests some intent about how benchmark/ subdivides — worth knowing what that was.
Question 2 — what belongs in-tree?
Our current answer, for discussion:
In: lane runners, route profiles (native TOML), the routing-log-to-predictions joiner, the scorer, per-domain task-id lists.
Out: all run artifacts (logs, routing logs, transcripts), and all ground truth. Every GT path in the scorer is a required argument with no default — some source datasets are gated or do-not-redistribute, and GT derived from one pair of tiers says nothing about another deployment's, so shipping it would be actively misleading.
Uncertain: the profiles. Eight operating points across two traffic domains, each with an inline classifier rubric. They are genuinely useful as worked examples of mode = "custom" routing, but they encode one deployment's judgement about what "aggressive" means. Do those belong in the repo, in docs, or nowhere?
Question 3 — dependencies
The scorer's optional dashboard needs matplotlib. We deliberately did not add it to pyproject.toml (the slim-install smoke test, and lockfile churn) and instead documented uv run --with matplotlib. Is that the preferred pattern for benchmark-only tooling, or should such deps go in the dev group?
Status
Implemented and working against the native server: the lanes drive switchyard-server --config <profile> --routing-log-file <path>, and the joiner attributes turns to tasks via the task field the routing log already records from x-switchyard-intake-task (falling back to session_id). All profiles are validated with --dry-run in CI-able form.
Happy to send it wherever it should go, or to drop parts that are not wanted in-tree.
Opening this for discussion before sending a PR, because the placement question is worth settling first and the answer changes the shape of the change.
What this is about
We have a set of multi-turn, tool-using benchmark lanes (tau2, tau3) plus a routing-accuracy scorer, built to answer a question the existing
benchmark/harness does not: when the router sends a turn to the weak tier, was that the right call?benchmark/today measures whether Terminal-Bench tasks pass through a routed server. These lanes measure routing decisions against per-task ground truth derived from floor and anchor arms. Complementary, but different enough that they may not want to be neighbours.Question 1 — where in the tree?
Candidates we considered:
benchmark/benches/<lane>/— what we prototyped. Reuses the existingbenchmark/conventions, andbenchmark/*.mdis already whitelisted in the CI paths filter. Downside:benchmark/benches/reads oddly, and it puts two different evaluation philosophies under one root.benchmark/<lane>/— flat, as siblings ofclosed_book_proxy/andserver-configs/. Simpler, but the lane runners are a different kind of artifact from the Harbor tooling already there.examples/experimental/<name>/— thelitellmprecedent, with its ownpyproject.toml/uv.lock. Best if the harness should carry dependencies that must stay out of the root lockfile.Note
.gitignorealready reservesbenchmark/perf/results/for a perf sub-harness that does not exist yet, which suggests some intent about howbenchmark/subdivides — worth knowing what that was.Question 2 — what belongs in-tree?
Our current answer, for discussion:
In: lane runners, route profiles (native TOML), the routing-log-to-predictions joiner, the scorer, per-domain task-id lists.
Out: all run artifacts (logs, routing logs, transcripts), and all ground truth. Every GT path in the scorer is a required argument with no default — some source datasets are gated or do-not-redistribute, and GT derived from one pair of tiers says nothing about another deployment's, so shipping it would be actively misleading.
Uncertain: the profiles. Eight operating points across two traffic domains, each with an inline classifier rubric. They are genuinely useful as worked examples of
mode = "custom"routing, but they encode one deployment's judgement about what "aggressive" means. Do those belong in the repo, in docs, or nowhere?Question 3 — dependencies
The scorer's optional dashboard needs
matplotlib. We deliberately did not add it topyproject.toml(the slim-install smoke test, and lockfile churn) and instead documenteduv run --with matplotlib. Is that the preferred pattern for benchmark-only tooling, or should such deps go in thedevgroup?Status
Implemented and working against the native server: the lanes drive
switchyard-server --config <profile> --routing-log-file <path>, and the joiner attributes turns to tasks via thetaskfield the routing log already records fromx-switchyard-intake-task(falling back tosession_id). All profiles are validated with--dry-runin CI-able form.Happy to send it wherever it should go, or to drop parts that are not wanted in-tree.