feat(diff-scope): measure and capture diffs with Git instead of copying a baseline - #281
Merged
Conversation
Three places spawned git with the operator's configuration held off, each with its own copy of the setup: the source resolver, the task-repository lifecycle, and — next — diff-scope measurement. Move `IsolatedGit` to `core`, where `run_git` already lives, and give `run` an `env` parameter so the baseline commit's committer identity rides on the shared helper instead of a parallel one. `BASELINE_REF` moves with it. It was private to the runner, but it is the contract between whoever writes the ref and whoever measures against it, so it needs one spelling. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Diff-scope snapshotted a task's start state by copying every file in the environment into `diff-scope-baseline/files`, then walked both trees to produce four counters. Against a real codebase that doubles disk per environment and adds a full tree walk per task — and it never produced the diff itself, which is the evidence a judge needs to answer whether the code got better. Every environment is now a Git repository marked with `eval-magic/baseline` at the state the agent started from, so Git can supply both. Measurement seeds a scratch index from that ref, brings it up to the working tree with one `git add`, and diffs the two trees: creations, modifications, and deletions fall out of one pass, and untracked creations are not missed. The scratch index lives outside the repository, so an eval that ran git itself keeps its own index and HEAD. Each run now also gets `diff.patch` beside its metrics, capped and marked when a diff runs past the cap, and a changed-file list in `diff-scope.json`. What counts is what Git counts, which changes two documented behaviors. The codebase's own `.gitignore` now applies, so a run that compiles no longer reports its build output as thousands of touched files — the same rule the baseline commit was already built under. And nested repository metadata is no longer measurable at all, because Git indexes no path with a `.git` component. Renames are switched off deliberately: a rename is two touched files, which is what the metric has always meant. The four existing integration tests pass with their metric expectations unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #255. Part of #244, depends on #252 (landed).
Why
capture_iteration_baselinessnapshotted a task's start state by copying every file in theenvironment into
diff-scope-baseline/files, then walked both trees at ingest time to produce fourcounters. Against a real codebase that doubles disk per environment and adds a full tree walk per
task — and it never produced the diff itself, which is exactly the evidence #244's judge needs to
answer "is the code better?"
#252 left every environment a Git repository marked with
refs/eval-magic/baseline, so Git cansupply both the metrics and the patch for free.
What changed
Measurement seeds a scratch index from the baseline ref, brings it up to the working tree with one
git add, and diffs the two trees. The scratch index lives outside the repository, so an eval thatran
gititself keeps its own index andHEAD.Each run now gets:
diff-scope.json— the four counters as before, plus a changed-file list withadded/modified/deletedstatuses and a patch recorddiff.patch— the diff itself,-U3, capped at 1 MiB with an explicit truncation marker. Alwayspresent; empty for a run that changed nothing.
Deleted: the
diff-scope-baseline/copy tree and its capture pass, the hand-rolled Myers counter,core::fs::copy_entry(its only caller), and thesimilarandwalkdirdependencies. Net −31lines despite the added feature.
IsolatedGitmoved fromsrc/source/git.rstosrc/core/git.rsand absorbedorchestrate/git.rs's third hand-rolled copy of the same git-config isolation (first commit,reviewable on its own). That isolation is load-bearing rather than tidiness: an operator's
core.excludesFileorcore.autocrlfwould otherwise change the numbers a published benchmarkreports.
Behavior changes worth reviewing
Two documented behaviors change, because what counts is now what Git counts:
.gitignoreapplies. A run that compiles no longer reports its buildoutput as thousands of touched files. This is the same rule the baseline commit was already built
under. Existing
diff_scopethresholds tuned against ignored output will measure smaller..gitcomponent.
docs/progressive-enhancements.mdlost its "nested repository metadata and all othernew files count" claim, and
schema/evals.schema.json'smax_files_toucheddescription wasrewritten (it still described Myers byte-lines).
Three edge cases also diverge from the old counter: a mode-only change now counts as one touched
file with zero lines (the copy walk missed it entirely); a binary file counts as one touched
file with zero lines (the copy walk ran a byte-line diff over it); a changed symlink counts one
added and one removed line. None affect the ported test expectations.
Renames are switched off deliberately (
--no-renames): a rename is two touched files, one createdand one deleted, which is what the metric has always meant.
Before / after
Same agent edit — one file modified, one added — against a real repo whose
.gitignorecoverstarget/, with 5 KB of build output written:Before:
files_touched: 3(build output counted), no diff artifact.After:
{ "files_touched": 2, "lines_added": 4, "lines_removed": 1, "hunks": 2, "files": [ { "path": "src/greeting.rs", "status": "added", "lines_added": 3, "lines_removed": 0 }, { "path": "src/main.rs", "status": "modified", "lines_added": 1, "lines_removed": 1 } ], "patch": { "path": "diff.patch", "bytes": 379, "truncated": false } }with
diff.patchholding the readable, applyable diff beside it.Verification
The four pre-existing
tests/run/diff_scope.rstests pass with their metric expectationsunmodified — the parity #255 asks for. Two pins were mutation-checked: reverting
--no-renamesorforcing the measurement
addkills exactly its own test.Also verified end to end against a real Git codebase: gitignored build output excluded, no-change
arm gets zero metrics and an empty patch,
find -name diff-scope-baselinereturns nothing, and theenvironment's own index and
HEADare untouched by measurement.Mode B parity has its own case (
revision_mode_measures_and_captures_the_diff_for_both_arms).Provenance needed no new work — #252 and #253 already land source and resolved SHA on all four
surfaces — so it is covered by assertion rather than by new code.
🤖 Generated with Claude Code