feat(server): opt-in cost savings endpoint and live dashboard - #378
feat(server): opt-in cost savings endpoint and live dashboard#378michaelneale wants to merge 4 commits into
Conversation
Adds per-target pricing (USD per 1M tokens) to the server TOML and, when any target is priced, registers GET /v1/savings (JSON) and GET /dashboard (self-contained live HTML page). Savings compare actual routed spend against a baseline model - what the same traffic would have cost if every request had been served by the most capable target. Classifier calls are counted as routing overhead against the savings. Pricing semantics (base input / cache read / cache write / output buckets) match switchyard.cli.launchers.cost_estimator. Fully opt-in: no pricing in config means no new routes and no behaviour change. Signed-off-by: Michael Neale <michael.neale@gmail.com>
…by model id Signed-off-by: Michael Neale <michael.neale@gmail.com>
Signed-off-by: Michael Neale <michael.neale@gmail.com>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (4)
🚧 Files skipped from review as they are similar to previous changes (2)
WalkthroughThe change adds optional model pricing and savings accounting. Configured servers expose ChangesSavings reporting
Estimated code review effort: 3 (Moderate) | ~30 minutes Mergeability Score: ⚪ Minimal · up to The change adds opt-in cost-savings reporting without altering routing behavior when pricing is not configured, and no actionable merge-blocking risk remains beyond normal checks. Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (4)
crates/switchyard-server/src/savings_dashboard.html (2)
130-144: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueUse
fmtfor the call count, asrenderModelsdoes.Line 141 interpolates
m.callsdirectly.renderModelspasses the same field throughfmt(m.calls)on line 153. Align the two so the distribution row groups thousands the same way as the table.This also removes the only raw interpolation of a server-supplied value in the template. The static analysis
inner-outer-htmlwarnings on lines 133-142 and 159-163 are handled:escalready wraps every model name, and the remaining values are numeric. Themanual-sanitizationhint recommends DOMPurify, which does not apply here. The page is embedded withinclude_str!and is intentionally dependency-free.♻️ Proposed consistency change
- <div class="pct">${m.calls} calls · ${pct.toFixed(1)}%</div> + <div class="pct">${fmt(m.calls)} calls · ${pct.toFixed(1)}%</div>🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/switchyard-server/src/savings_dashboard.html` around lines 130 - 144, Update renderDist to format m.calls with the existing fmt helper in the distribution row, matching renderModels while leaving the percentage calculation unchanged.Source: Linters/SAST tools
195-199: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueReport a failed reset to the user.
The reset handler ignores the response status. If
POST /v1/stats/resetreturns an error, the page callsrefresh()and shows the unchanged counters with the status still set to "live". The operator gets no signal that the reset failed.♻️ Proposed change
$("reset").addEventListener("click", async () => { - await fetch("/v1/stats/reset", {method:"POST"}); - refresh(); + try { + const r = await fetch("/v1/stats/reset", {method:"POST"}); + if(!r.ok) throw new Error(await r.text()); + } catch(e) { + $("status").textContent = "reset failed"; + return; + } + refresh(); });🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/switchyard-server/src/savings_dashboard.html` around lines 195 - 199, Update the reset click handler near the $("reset") listener to inspect the POST /v1/stats/reset response before calling refresh(). On a non-success response, report the reset failure through the page’s existing user-visible status/error mechanism and avoid presenting the unchanged counters as successfully reset; retain the refresh flow only for successful resets.crates/switchyard-server/src/savings.rs (2)
91-102: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueConsider rounding the per-model costs too.
computeappliesround6to the snapshot totals but stores the rawf64forModelSavings::costandModelSavings::baseline_cost. The JSON response therefore mixes rounded totals with full-precision per-model values, and the per-model column can show long floats to any consumer that does not format them. The bundled dashboard formats withtoFixed, so nothing is visibly wrong today.Apply
round6to both per-model fields for a consistent response contract.♻️ Proposed consistency change
- cost, - baseline_cost: would_be, + cost: round6(cost), + baseline_cost: round6(would_be),Also applies to: 199-205
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/switchyard-server/src/savings.rs` around lines 91 - 102, Apply the existing round6 helper to both cost and baseline_cost when constructing each ModelSavings entry in compute, while leaving the raw calculations and priced flag unchanged.
45-48: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueRestrict
is_emptyunless it is part of the external API. No in-repository code callsSavingsConfig::is_empty; usepub(crate)or remove it if external callers do not need it.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/switchyard-server/src/savings.rs` around lines 45 - 48, Restrict the visibility of SavingsConfig::is_empty from public external API access to pub(crate), or remove the method if it is not required by external callers; preserve its existing pricing.is_empty behavior if retained.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/switchyard-server/src/config.rs`:
- Around line 129-159: Validate every pricing rate in apply_savings before
converting entries with into_model_price. Reject negative or non-finite f64
values, including NaN and infinities, by returning a ServerError during
configuration loading; only construct SavingsConfig after all rates pass
validation, preserving the existing baseline_model checks.
In `@crates/switchyard-server/src/savings_dashboard.html`:
- Around line 167-202: Update refresh and the polling setup to prevent
concurrent requests: track an in-flight state, return immediately when refresh
is already running or document.hidden is true, and clear the state in all
completion paths. Replace the unconditional setInterval polling with
visibility-aware scheduling that resumes on visibility changes and preserves
manual refresh behavior; do not apply the React-specific setstate-same-var hint.
In `@crates/switchyard-server/src/savings.rs`:
- Around line 105-119: Update the classifier-cost loop in the savings
calculation to append each classifier model lacking a price to the existing
unpriced_models collection, matching the routed-model loop’s behavior. Keep its
cost at zero while ensuring the model is surfaced for reporting and dashboard
warnings.
---
Nitpick comments:
In `@crates/switchyard-server/src/savings_dashboard.html`:
- Around line 130-144: Update renderDist to format m.calls with the existing fmt
helper in the distribution row, matching renderModels while leaving the
percentage calculation unchanged.
- Around line 195-199: Update the reset click handler near the $("reset")
listener to inspect the POST /v1/stats/reset response before calling refresh().
On a non-success response, report the reset failure through the page’s existing
user-visible status/error mechanism and avoid presenting the unchanged counters
as successfully reset; retain the refresh flow only for successful resets.
In `@crates/switchyard-server/src/savings.rs`:
- Around line 91-102: Apply the existing round6 helper to both cost and
baseline_cost when constructing each ModelSavings entry in compute, while
leaving the raw calculations and priced flag unchanged.
- Around line 45-48: Restrict the visibility of SavingsConfig::is_empty from
public external API access to pub(crate), or remove the method if it is not
required by external callers; preserve its existing pricing.is_empty behavior if
retained.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 65f3e994-00e7-43b1-9749-e66f9d453e24
📒 Files selected for processing (7)
CHANGELOG.mdcrates/switchyard-server/src/config.rscrates/switchyard-server/src/lib.rscrates/switchyard-server/src/savings.rscrates/switchyard-server/src/savings_dashboard.htmldocs/operations/cost_savings.mdmkdocs.yml
Signed-off-by: Michael Neale <michael.neale@gmail.com>
|
@grahamking do we need dashboard stuff as part of switchyard ?? |
|
@michaelneale Thanks! That looks really neat. Give us a few days to play with it. In the meantime would you be able to address the Code Rabbit review comments? |
|
thanks @grahamking on it |
|
@grahamking should be addressed now |
|
🤖 (posted by micn's AI agent) All CodeRabbit findings are addressed in da9703b:
fmt, clippy, and all tests green. Happy to iterate while you try it out — and if the consensus lands on JSON-endpoint-only, the @coderabbitai review |
|
✅ Action performedReview finished.
|
Prose and accuracy reviewI reviewed the public documentation and dashboard copy against the implementation, the repository's existing cost estimator, and current provider pricing. I found three actionable accuracy issues; I did not repeat the CodeRabbit findings already addressed in
|
4 participants
What
Adds opt-in cost savings reporting to
switchyard-server: an optional[pricing]table in the deployment TOML enablesGET /v1/savings(JSON) andGET /dashboard(self-contained live HTML page) that compare actual routed spend against a baseline model — "what would this traffic have cost if every request went to the most capable target".Why
The routing algorithms (llm_classifier, stage_router, escalation) exist largely to spend the capable model only on turns that need it. Today the only way to see what that actually saved is post-hoc analysis of
/v1/statsdumps (as the Pythoncost_estimatorin the launchers does). This makes the payoff visible live while a coding agent runs through the proxy.How
Purely additive. No
[pricing]table → the endpoints are not registered and behavior is unchanged. Pricing never influences routing decisions.Config:
[pricing]is keyed by model id (target.id), matching how stats are keyed, so one entry covers all targets sharing a model.Pricing semantics (base input / cache read / cache write / output buckets) match
switchyard.cli.launchers.cost_estimator.Classifier/judge traffic is priced separately as
classifier_costand deducted from savings, so routing overhead is charged honestly against the result.Models serving traffic without a pricing entry are costed at zero and surfaced in
unpriced_models(and as a dashboard warning) so under-counting is visible.The dashboard is a single embedded HTML file with no external dependencies, polling
/v1/savingsevery 2s. Counters reset with the existingPOST /v1/stats/reset.Endpoint registration is gated the same way as the existing session-stats route.
Sample
/v1/savingsoutputFrom a real session (goose coding agent through an llm_classifier route, cheap judge, Sonnet weak / Opus strong, plus some Opus-pinned A/B traffic):
{ "total_requests": 11, "actual_cost": 0.2864, "baseline_cost": 0.4722, "classifier_cost": 0.0026, "saved": 0.1858, "saved_pct": 39.35, "baseline_model": "claude-opus-5", "models": { "claude-opus-5": { "calls": 4, "cost": 0.1581, "baseline_cost": 0.1581 }, "claude-sonnet-5": { "calls": 7, "cost": 0.1256, "baseline_cost": 0.3141 } }, "unpriced_models": [] }Testing
[savings]without pricing rejected, unpriced baseline rejected)cargo test -p switchyard-server— 62 tests green; fmt and clippy cleanNotes for reviewers
Summary by CodeRabbit
/v1/savingsmetrics and a live/dashboardwith spending, savings, routing, per-model costs, warnings, refresh, and reset controls.