feat(trg): let an eval fixture opt into read-only enforcement

[yordis]

• A case that reads a file had no way to say the file must survive the run unchanged. An agent that edited a fixture in place still passed, as long as the case's own assertions never happened to look, so the suite was quietly measuring a different input than the one it declared.
• Permissions are not the guarantee and are not treated as one here. An agent can delete a read-only file and write a fresh one in its place without ever touching a mode bit, so the staged copy is hashed against its source after the run and a mismatch fails the case regardless of what the case's own graders found. Clearing write bits is a courtesy that catches the accidental case early; the hash is what actually holds.
• The violation is reported as an ordinary failing assertion rather than through a new channel of its own. That is what makes it reach grading.json, the exit code, and the existing ci gates without any of them being taught about this feature, and it means the fixture path an operator needs is in the evidence string rather than in a count.
• The wire form stays additive on purpose: a bare string still parses, still means writable, and still serializes back as a bare string, so no existing suite changes meaning and no suite gets rewritten into the object form just by round-tripping. Only a fixture that opts into read_only pays for the longer form.
• Both forms deserialize through the same path validation that was already there, so the new syntax cannot become a way around the existing escape and absolute-path rejection.

[cursor]

PR Summary

Medium Risk
Touches eval staging, runner outcomes, report schema, and grading pass/fail logic; behavior is additive for existing suites but changes what counts as a failed case when read-only fixtures are used.

Overview
Eval cases can mark staged fixtures as read-only so read-heavy cases are not scored against agent-mutated inputs.

Manifest wire format: files entries stay backward compatible—a bare string still means writable and serializes back as a string. Opt-in read-only uses { "path": "...", "mode": "read_only" } (or writable on the object form). Schemas and reference docs describe the new shape and read_only_fixture_violations on runs.

Runner behavior: After staging, read-only fixtures get write bits cleared on Unix (directories stay writable so workspace reset still works). When any runner finishes, staged copies are content-hashed against the skill source (same machinery as skill_integrity); mismatches populate read_only_fixture_violations on the outcome and run record, with run warnings naming each path.

Grading: Violations become mechanical failing assertions (fixture path in evidence), so a case cannot pass on other graders alone. Existing CI/verify paths pick this up without new gates.

^{Reviewed by Cursor Bugbot for commit d07a50b. Bugbot is set up for automated code reviews on this repo. Configure here.}

[coderabbitai]

Warning

Review limit reached

Next included review available in 31 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: 47666584-dcb3-4e34-b94c-bf034a76f96e

📥 Commits

Reviewing files that changed from the base of the PR and between dd3d8d3 and d07a50b.

📒 Files selected for processing (11)

• crates/trg/docs/reference/ai-skills-eval.md
• crates/trg/schemas/evals.json.schema.json
• crates/trg/schemas/report.json.schema.json
• crates/trg/src/agentskills/cache.rs
• crates/trg/src/agentskills/ci.rs
• crates/trg/src/agentskills/evals.rs
• crates/trg/src/agentskills/grading.rs
• crates/trg/src/agentskills/report.rs
• crates/trg/src/agentskills/runner/mod.rs
• crates/trg/src/agentskills/schemas.rs
• crates/trg/src/commands/ai/skills/eval/run.rs

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit d07a50b. Configure here.}

[cursor]

Cache hits drop fixture violations

High Severity

apply_cache_hit copies skill_integrity but not read_only_fixture_violations. A completed run that mutated a read-only fixture is still cached, and a later cache or --reuse-completed hit leaves the field empty. Grading then turns that field into the failing assertion, so the reused case can pass even though the original run failed the fixture check.

 

^{Reviewed by Cursor Bugbot for commit d07a50b. Configure here.}