Feat/skill source - #279
Merged
Merged
Conversation
The module's own docs say it knows nothing about what it is sourcing, but every message it emits said "codebase". A skill resolved through it would have reported `codebase path '...' is not a directory`, naming the one thing the operator cannot act on. The caller now supplies the noun, and the resolution carries it so materialization reads it back. Also records `dirty` alongside the existing uncommitted-changes warning. A warning is advice the operator may miss; the flag is evidence, and a subject copied as it sits on disk cannot be cited without it. The probe is scoped with `-- .` so a skill that is one directory among many in a repository is not called dirty the moment some other skill is edited. Part of #253. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`CodebaseRecord` is about to carry the skill under test as well as the codebase a task environment is built from. Leaving it named for one of its two subjects would misdescribe every use of the other. Pure rename: `CodebaseUse` flattens the record, so no JSON key moves and every artifact serializes byte-identically. Part of #253. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Artifacts defaulted to `<cwd>/.eval-magic`, so running an eval from a skills repository dropped iteration trees, envs, and benchmarks inside the very repository under measurement. The eval home now derives from the skill directory instead of the cwd: `EVAL_MAGIC_WORKSPACE_DIR`, else `$XDG_DATA_HOME/eval-magic`, else `~/.local/share/eval-magic` — mirroring the `EVAL_MAGIC_CONFIG_DIR` ladder already used for descriptor layers. `--workspace-dir` still wins over both. The derived default is namespaced by `<skill-dir-name>-<digest>`. Without it a single global root would interleave the iterations of two skills that share a name and come from different repositories, where `--iteration N` could reach the wrong one. The digest is a hand-rolled FNV-1a rather than `DefaultHasher`, which has no cross-release stability guarantee: this names a directory operators re-type and generated commands embed, so a toolchain upgrade must not silently relocate it. An operator upgrading mid-campaign gets a notice naming the old directory and the `--workspace-dir` value that keeps it reachable — suppressed when the resolved root already is that directory, and when the only thing there is the `harnesses/` descriptor layer, which is an unrelated use of the same name and does not move. Part of #253. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The codebase was a sourced, copied, SHA-recorded input while the skill was read in place from wherever the operator's cwd happened to be — two mental models in one command, and a report could pin the codebase commit while the skill side was "whatever was on disk at the time". The skill now resolves through the same resolver and is copied into `iteration-N/.skills/`, a sibling of `.codebase/`. Every condition stages from that copy, and the resolved source plus its revision reach `conditions.json`, each `dispatch.json` task, each `run.json`, `benchmark.json`, and `BASELINE.md`. The copy is the working tree as it sits, not a checkout: Mode B's new arm *is* the uncommitted edit under test, and Mode A's ordinary loop is edit-then-run, so a committed-state copy would measure the wrong bytes. `dirty` records when that happened and the warning says the run measured the uncommitted work — the opposite of the codebase warning, where a clean checkout leaves it behind. The resolver now reports only the fact; each caller phrases the consequence it owns. The sibling roster is recorded at resolution and staging copies exactly those names, so what the artifacts claim and what the environments hold cannot drift apart. `promote-baseline` follows the recorded pointer rather than the operator's current selection, so it still writes `<skill>/evals/baseline/` — and refuses loudly when that skill has moved, rather than writing one skill's baseline into another. Also drops the dead `stage_root` override in `command_run`: it pointed at an `env/` directory this layout stopped producing, and nothing inside `run` read it back. Part of #253. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`detect_live_source_reads` scanned agent shell commands for a bare relative path to the live skill, computed from the operator's cwd. That only ever made sense while the eval home sat inside the skill's own tree: with it outside, a bare relative token in an agent command resolves against the environment and cannot name the live skill. The absolute-path branch stays. The live source still exists on disk and an agent can still name it outright, so that remains a real finding. `repo_root` stays with it — read-tool arguments may be relative and are resolved against it. The three "staged copy under .claude/skills is not flagged" tests go with the branch: they existed to pin the config-dir lookbehind inside `references_bare_rel`, and would have passed vacuously once it was gone, which is worse than no coverage. Also drops teardown's cwd sweep of staged skills. Staging is env-scoped — `run` places nothing at the invocation cwd — so the sweep was residue from when it did. The lifecycle test's `.claude` assertion moves to just after `run`, where it can still fail if cwd staging ever comes back; after teardown it could only pass. The cwd guard disarm stays, since `teardown-guard` is a documented cwd-only command. Part of #253. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`grade` read eval definitions and held-out command-check setup files from the live skill tree. An edit between `run` and `grade` therefore changed what a finished run was measured against, with nothing recording that it had — the same provenance hole this ticket closes, one phase later. Both now come from `iteration-N/.skills/<skill>`, falling back to the live tree for iterations prepared before skills were sourced. Live-source detection keeps the live path, which is the one thing it is looking for. Part of #253. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`--mode revision` stages a snapshot in one arm and the live skill in the other, so it is the mode where a half-applied change would hide. Pins that both arms name something the runner placed inside the eval home, and that revision runs record the skill source like new-skill runs do. Part of #253. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The isolation guide explained what a dispatch can *load* but not what the runner *places*, which is now the larger half of the story: everything the agent can see is a copy, and the eval home lives outside the skill's own repository. Adds a section covering the copy, what `skill_source` records, why `dirty` matters before publishing, and why an absolute-path read of the live directory is still a finding. The codebase guide's verification snippet names the skill alongside the codebase, since the two are recorded the same way; `--skill-dir` help says the roster is captured at resolution; and `--help` says where artifacts land. Part of #253. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Review follow-ups to the copied-input change. `promote-baseline` writes into the skill the run recorded, but read "Promoted from commit" from the operator's current selection, so the table could label a baseline with a commit from a different repository. `git_cwd` existed to allow exactly that difference and no caller ever wanted it; the commit now comes from the tree the baseline lands in. `--no-stage` populates no skills directory, so recording a roster of siblings "staged alongside" the skill described an environment that never existed. Extracts the `context` and `promote` test modules into sibling files, the convention `adapters/guard` already uses: both had grown past the point where the module fits the file it exercises. Part of #253. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Grading reads `run.json` and nothing else, so the carrier from `dispatch.json` into the record is the link that ties a graded result to a skill revision. It had no test of its own; the codebase equivalent beside it did. Part of #253. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #253