feat(trg): let a case pin how many times it is drawn

[yordis]

• A pass-wide --attempts forces one price on every case: a cheap, stable case pays for the flaky one's repeats, and an expensive case makes a deep pass unaffordable. Whether a case needs a second sample is a property of the case, so the case is where it belongs.
• The operator still wins when they name a count, so a smoke pass or a deep one over a whole suite stays one flag away. That is why the flag is left unset rather than defaulted: a count defaulted at the CLI cannot be told from one an operator typed, and every case that pinned its own would be silently overridden.
• The summary is read as the denominator of the pass. Multiplying one count by the case count would claim runs the report does not contain the moment a case pins its own.
• mirror_path tells a reader where a run's outputs live, so it has to be cut from the count that run's own cell took. Taken from the pass-wide maximum, a case drawn once lands under attempt-1 because some other case was drawn five times, and the unsuffixed path the reader was promised holds nothing.

[cursor]

PR Summary

Medium Risk
Changes eval run scheduling, report totals, and on-disk mirror layout semantics; behavior is heavily tested but operators and suite authors may see different run counts and paths than before when mixing per-case attempts.

Overview
Eval cases can now declare an optional attempts count in evals.json (schema AttemptCount, minimum 1) so flaky or expensive cases can differ from the pass default without forcing one draw count on the whole suite.

Precedence is implemented in effective_attempts: an explicit --attempts on eval run overrides every case; otherwise each case uses its own pin or falls back to the recommended 3 draws. The CLI flag is Option<AttemptCount> with no clap default so “operator did not pass the flag” stays distinct from “operator chose N,” which is required for overrides to work.

Reporting and docs layout follow per (case × scenario) cell draw counts: build_runs / build_summaries sum actual draws per case instead of multiplying a single pass-wide count, and DrawsPerCell drives alias indexes and mirror paths so a case drawn once keeps an unsuffixed scenario path even when another case in the same pass has multiple attempts.

^{Reviewed by Cursor Bugbot for commit c108d7d. Bugbot is set up for automated code reviews on this repo. Configure here.}

[coderabbitai]

Warning

Review limit reached

Next included review available in 9 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Advanced

Run ID: 4a91749a-06e3-45e3-a2fc-8952282ecd6b

📥 Commits

Reviewing files that changed from the base of the PR and between edb031d and c108d7d.

📒 Files selected for processing (6)

• crates/trg/schemas/evals.json.schema.json
• crates/trg/src/agentskills/evals.rs
• crates/trg/src/agentskills/layout.rs
• crates/trg/src/agentskills/report.rs
• crates/trg/src/agentskills/sampling.rs
• crates/trg/src/commands/ai/skills/eval/run.rs

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}