Model/effort default recommendations from compare data (design-gated)

[iamtoruk]

Model/effort default recommendations from compare data (design-gated)

Part of the acting-layer epic. Blocked by the journal and --apply issues. Additionally gated: post a short design comment on this issue (recommendation thresholds, exact settings keys verified against current Claude Code docs, rollout plan) and get it approved before writing code.

Idea

compare already computes per-model one-shot rate, cost per edit turn, and per-category (classifier) splits from real local history. When one model demonstrably matches another's edit reliability at a fraction of the cost for a given project's workload, CodeBurn can recommend, and optionally apply, a per-project default model change. This is the no-proxy version of "routing": it changes a default in config, never a live request, and the user sees exactly what changed.

What to build (after design approval)

Recommendation engine (src/act/model-defaults.ts)

Using src/compare-stats.ts aggregates per project over the last 30 days:

• Consider a pair (current dominant model A, candidate B) only when both have at least 30 edit turns in the project (raise if the design review says so).
• Recommend B when: B one-shot rate >= A one-shot rate minus 3 percentage points, and B cost per edit turn <= 60% of A's, computed on this project's own turns.
• Never recommend across capability tiers for Debugging-heavy projects: if the project's classifier split is more than 40% Debugging, require B one-shot rate >= A (no tolerance).
• Output: recommendation objects with the full evidence (turn counts, rates, costs) for display.

Surface

• codeburn compare (non-interactive summary path) and codeburn optimize gain a low-key recommendation block listing qualifying projects with evidence, and the apply hint.
• codeburn act apply-model <project>: writes the model default into <project>/.claude/settings.json (verify the exact key against current docs at implementation time; do not guess) via the action journal (kind: model-default). Undo restores.
• Always print the evidence and the escape hatch (/model in-session overrides per session) when applying.

Explicit non-goals

• No per-turn routing, no proxy, no request rewriting. If the recommendation is wrong for a particular task, the user overrides in-session; nothing intercepts anything.
• No cross-provider recommendations in v1 (Claude Code projects recommend Claude models only; Codex analogue is a follow-up).
• No automatic apply. This kind is never included in optimize --apply --yes; it requires the explicit per-project command.

Tests

• fixture stats where B qualifies (rates/costs asserted against the thresholds), where B fails the one-shot tolerance, where turn counts are insufficient, and where the Debugging-share rule blocks an otherwise-qualifying pair
• apply writes only the expected key into project settings, preserves the rest, journals, undo restores byte-identical
• recommendation block absent when nothing qualifies

Acceptance

• On real data, recommendations only appear with the full evidence line, and applying one is a single reviewable config edit with undo.
• Design comment approved on this issue before any implementation PR.

Part of epic #602.

[iamtoruk]

Design for review, per the gate on this issue. Implementation starts only after this comment is approved.

Settings keys (verified against code.claude.com/docs/en/settings, 2026-07-03)

• model: model ID string, supported in project .claude/settings.json ("Override the default model to use for Claude Code"). --model and ANTHROPIC_MODEL override it per session, which is the user's escape hatch.
• effortLevel: "low" | "medium" | "high" | "xhigh", same scopes.

v1 writes only model. An effortLevel recommendation is deferred: compare has no per-effort stats today, and we do not recommend what we cannot evidence.

Recommendation thresholds

For a pair (dominant model A, candidate B) per project over the trailing 30 days, recommend B only when ALL hold:

1. Volume: at least 30 edit turns for A and 30 for B in this project. No cross-project borrowing.
2. Reliability: B one-shot rate >= A one-shot rate minus 3 percentage points.
3. Economics: B cost per edit turn <= 60% of A's, computed on this project's own turns.
4. Debugging guard: if the project's turn mix is more than 40% Debugging, the tolerance in rule 2 drops to zero (B must meet or beat A).
5. Recency: both models observed in the project within the last 14 days.
6. Same provider family only (Claude to Claude). Codex analogue is a follow-up issue.

Presentation

The recommendation block (in compare's summary and optimize) always shows the full evidence: turn counts, both one-shot rates, both cost-per-edit figures, and the exact file and key it would write. No evidence line, no recommendation.

Rollout and safety

• Apply only via an explicit codeburn act apply-model <project>. Never part of optimize --apply --yes; the picker lists it as manual with a pointer to the explicit command.
• The write goes through the action journal (kind model-default) with the stale-plan expectedHash check; codeburn act undo restores the settings file byte-identical.
• act report shows the applied row as correlation only in v1: one-shot rate before vs after, never a token or dollar claim.
• Quality tripwire: if the post-apply one-shot rate over at least 20 edit turns drops more than 5 points below the pre-apply baseline, the report row is flagged "quality regression, consider undo". No auto-revert in v1.
• The /model per-session override is printed with every apply so nobody feels locked in.

Approve or adjust the thresholds here and implementation follows the same implement, adversarial-review, fix flow the rest of the epic used.

[ozymandiashh]

Design looks solid to me. I'll pick up the v1 implementation in a focused PR.

Scope I'm locking for this PR:

• write only the model key in project .claude/settings.json
• keep effortLevel deferred because compare has no per-effort evidence today
• use the six thresholds from the design comment:
  • at least 30 edit turns for both dominant model A and candidate B in the same project
  • B one-shot rate >= A one-shot rate minus 3 percentage points
  • B cost per edit turn <= 60% of A's
  • debugging-heavy projects require B >= A with no tolerance
  • both models observed in the last 14 days
  • same provider family only
• apply only through explicit codeburn act apply-model <project>
• never include this in optimize --apply --yes
• journal the edit as model-default, with undo restoring the settings file byte-identical
• show the evidence line and the /model escape hatch on apply
• report post-apply quality as correlation only in v1, including the 5pp regression tripwire

Planned code paths:

• src/act/model-defaults.ts
• src/act/cli.ts for act apply-model
• recommendation blocks in compare and optimize
• tests for threshold qualification, insufficient volume, debugging guard, recency, and apply/undo round trip.

I will keep this strictly config-level. No proxy, no per-turn routing, no request rewriting.