Skip to content

feat(diff-scope): measure and capture diffs with Git instead of copying a baseline - #281

Merged
slowdini merged 2 commits into
devfrom
feat/git-diff-capture
Aug 20, 2026
Merged

feat(diff-scope): measure and capture diffs with Git instead of copying a baseline#281
slowdini merged 2 commits into
devfrom
feat/git-diff-capture

Conversation

@slowdini

Copy link
Copy Markdown
Owner

Closes #255. Part of #244, depends on #252 (landed).

Why

capture_iteration_baselines snapshotted a task's start state by copying every file in the
environment into diff-scope-baseline/files, then walked both trees at ingest time to produce four
counters. Against a real codebase that doubles disk per environment and adds a full tree walk per
task — and it never produced the diff itself, which is exactly the evidence #244's judge needs to
answer "is the code better?"

#252 left every environment a Git repository marked with refs/eval-magic/baseline, so Git can
supply both the metrics and the patch for free.

What changed

Measurement seeds a scratch index from the baseline ref, brings it up to the working tree with one
git add, and diffs the two trees. The scratch index lives outside the repository, so an eval that
ran git itself keeps its own index and HEAD.

Each run now gets:

  • diff-scope.json — the four counters as before, plus a changed-file list with added /
    modified / deleted statuses and a patch record
  • diff.patch — the diff itself, -U3, capped at 1 MiB with an explicit truncation marker. Always
    present; empty for a run that changed nothing.

Deleted: the diff-scope-baseline/ copy tree and its capture pass, the hand-rolled Myers counter,
core::fs::copy_entry (its only caller), and the similar and walkdir dependencies. Net −31
lines despite the added feature.

IsolatedGit moved from src/source/git.rs to src/core/git.rs and absorbed
orchestrate/git.rs's third hand-rolled copy of the same git-config isolation (first commit,
reviewable on its own). That isolation is load-bearing rather than tidiness: an operator's
core.excludesFile or core.autocrlf would otherwise change the numbers a published benchmark
reports.

Behavior changes worth reviewing

Two documented behaviors change, because what counts is now what Git counts:

  • The codebase's own .gitignore applies. A run that compiles no longer reports its build
    output as thousands of touched files. This is the same rule the baseline commit was already built
    under. Existing diff_scope thresholds tuned against ignored output will measure smaller.
  • Nested repository metadata is no longer measurable — Git indexes no path with a .git
    component. docs/progressive-enhancements.md lost its "nested repository metadata and all other
    new files count" claim, and schema/evals.schema.json's max_files_touched description was
    rewritten (it still described Myers byte-lines).

Three edge cases also diverge from the old counter: a mode-only change now counts as one touched
file with zero lines (the copy walk missed it entirely); a binary file counts as one touched
file with zero lines (the copy walk ran a byte-line diff over it); a changed symlink counts one
added and one removed line. None affect the ported test expectations.

Renames are switched off deliberately (--no-renames): a rename is two touched files, one created
and one deleted, which is what the metric has always meant.

Before / after

Same agent edit — one file modified, one added — against a real repo whose .gitignore covers
target/, with 5 KB of build output written:

Before: files_touched: 3 (build output counted), no diff artifact.

After:

{
  "files_touched": 2,
  "lines_added": 4,
  "lines_removed": 1,
  "hunks": 2,
  "files": [
    { "path": "src/greeting.rs", "status": "added",    "lines_added": 3, "lines_removed": 0 },
    { "path": "src/main.rs",     "status": "modified", "lines_added": 1, "lines_removed": 1 }
  ],
  "patch": { "path": "diff.patch", "bytes": 379, "truncated": false }
}

with diff.patch holding the readable, applyable diff beside it.

Verification

cargo fmt --all -- --check
cargo clippy --all-targets --all-features -- -D warnings
EVAL_MAGIC_REQUIRE_POSIX_TOOLS=1 cargo test --all-targets    # 1214 passed, 0 failed, 0 skipped
cargo build --release

The four pre-existing tests/run/diff_scope.rs tests pass with their metric expectations
unmodified
— the parity #255 asks for. Two pins were mutation-checked: reverting --no-renames or
forcing the measurement add kills exactly its own test.

Also verified end to end against a real Git codebase: gitignored build output excluded, no-change
arm gets zero metrics and an empty patch, find -name diff-scope-baseline returns nothing, and the
environment's own index and HEAD are untouched by measurement.

Mode B parity has its own case (revision_mode_measures_and_captures_the_diff_for_both_arms).
Provenance needed no new work — #252 and #253 already land source and resolved SHA on all four
surfaces — so it is covered by assertion rather than by new code.

🤖 Generated with Claude Code

slowdini and others added 2 commits August 20, 2026 03:41
Three places spawned git with the operator's configuration held off, each
with its own copy of the setup: the source resolver, the task-repository
lifecycle, and — next — diff-scope measurement. Move `IsolatedGit` to
`core`, where `run_git` already lives, and give `run` an `env` parameter so
the baseline commit's committer identity rides on the shared helper instead
of a parallel one.

`BASELINE_REF` moves with it. It was private to the runner, but it is the
contract between whoever writes the ref and whoever measures against it, so
it needs one spelling.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Diff-scope snapshotted a task's start state by copying every file in the
environment into `diff-scope-baseline/files`, then walked both trees to
produce four counters. Against a real codebase that doubles disk per
environment and adds a full tree walk per task — and it never produced the
diff itself, which is the evidence a judge needs to answer whether the code
got better.

Every environment is now a Git repository marked with `eval-magic/baseline`
at the state the agent started from, so Git can supply both. Measurement
seeds a scratch index from that ref, brings it up to the working tree with
one `git add`, and diffs the two trees: creations, modifications, and
deletions fall out of one pass, and untracked creations are not missed. The
scratch index lives outside the repository, so an eval that ran git itself
keeps its own index and HEAD.

Each run now also gets `diff.patch` beside its metrics, capped and marked
when a diff runs past the cap, and a changed-file list in `diff-scope.json`.

What counts is what Git counts, which changes two documented behaviors. The
codebase's own `.gitignore` now applies, so a run that compiles no longer
reports its build output as thousands of touched files — the same rule the
baseline commit was already built under. And nested repository metadata is
no longer measurable at all, because Git indexes no path with a `.git`
component. Renames are switched off deliberately: a rename is two touched
files, which is what the metric has always meant.

The four existing integration tests pass with their metric expectations
unchanged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@slowdini
slowdini merged commit ab2c66b into dev Aug 20, 2026
8 checks passed
@slowdini
slowdini deleted the feat/git-diff-capture branch August 20, 2026 16:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Measure and capture diffs with Git instead of copying a baseline

1 participant