From 23a9864fa93f40a1bb097e0651de9daf6c775c54 Mon Sep 17 00:00:00 2001 From: Justin McLean Date: Mon, 27 Jul 2026 12:23:46 +1000 Subject: [PATCH 1/2] docs(spec-loop): add spec-gap staleness verification spec Generated-by: Claude (claude-opus-5) --- tools/spec-loop/specs/spec-gap-staleness.md | 218 ++++++++++++++++++++ uv.lock | 16 +- 2 files changed, 226 insertions(+), 8 deletions(-) create mode 100644 tools/spec-loop/specs/spec-gap-staleness.md diff --git a/tools/spec-loop/specs/spec-gap-staleness.md b/tools/spec-loop/specs/spec-gap-staleness.md new file mode 100644 index 00000000..fa0ce738 --- /dev/null +++ b/tools/spec-loop/specs/spec-gap-staleness.md @@ -0,0 +1,218 @@ + + +--- +title: Spec-gap staleness verification +status: proposed +kind: feature +mode: infra +source: > + Observed drift: nine "Known gaps" bullets across eight specs described + capabilities that had already shipped, and the plan beat queued work + from them. Implemented by tools/spec-loop/PROMPT_plan.md, + tools/spec-loop/PROMPT_update.md, and new checks in + tools/spec-validator. +acceptance: + - The plan beat cannot queue a work item from a Known-gap bullet + without citing evidence that the gap is still open. + - Admissible evidence is a call site, current file content, or a + counted value; the presence of a name is not admissible. + - A Known-gap bullet may carry a machine-evaluable close condition, + and the validator fails when that condition is already met. + - The validator fails when two specs claim different counts for the + same skill family. + - Every check is deterministic given a working tree and requires no + model call. +--- + +# Spec-gap staleness verification + +## What it does + +Stops a `## Known gaps` bullet outliving the gap it describes. + +A spec's Known-gaps section is not documentation. The spec-loop plan +beat reads it to choose the next work item, so a bullet describing +already-shipped code sends the loop at finished work, and a bullet that +was accurate a month ago is indistinguishable from one accurate today. +Nothing in the toolchain can currently tell the two apart: +`spec-validator` checks frontmatter keys, section presence, SPDX +headers, and the existence of paths named in Validation blocks, but has +no view on whether a gap statement is still true. + +This spec adds three mechanisms. An evidence rule on the plan and +update beats refuses to queue work from an unverified gap. An optional +close-condition predicate on a gap bullet makes staleness decidable for +the bullets that have a crisp end state. A cross-spec count check +catches the narrow case of two specs disagreeing about a countable +fact. + +## Where it lives + +- `tools/spec-loop/PROMPT_plan.md`: the evidence rule. A work item + derived from a Known-gap bullet must carry a verification line. +- `tools/spec-loop/PROMPT_update.md`: the same rule applied when the + update beat rewrites a Known-gaps section, so a bullet is retained + only if it is re-verified. +- `tools/spec-validator/`: `validate_gap_predicates` (HARD on a + satisfied predicate, SOFT on an unannotated bullet) and + `validate_cross_spec_counts` (HARD). +- `tools/spec-inventory/`: reused, not changed. `section_bullets` + already extracts `Known gaps` bullets per spec. +- `tools/spec-loop/.last-sync`: unchanged. It scopes the update beat's + diff and is not a staleness signal. + +## Behaviour & contract + +### The evidence rule (plan and update beats) + +Before a Known-gap bullet becomes a work item, the beat records one of +exactly three kinds of evidence that the gap is still open: + +| Evidence | Shape | Example | +|---|---|---| +| Call site | `file:line` where behaviour is or is not wired | `no call to validate_x in run_validation` | +| File content | current text or absence at a named path | `audit-record-schema.md does not exist` | +| Counted value | a number derived from the tree | `6 skills carry family:repo-health` | + +The presence of a name is explicitly **not** admissible. A function +defined but never called, a hook named in a config comment but not +wired, and a grep hit inside a docstring each satisfy a name match +while the described gap remains open or closed independently of it. +Where the distinction is load-bearing, the evidence names the call +site, not the definition. + +A bullet the beat cannot verify is neither queued nor deleted. It is +carried forward marked unverified, so the next beat sees that the check +was attempted and failed rather than silently re-deriving work from it. + +### Close-condition predicates + +A Known-gap bullet may carry a trailing HTML comment stating the +condition under which the gap is closed: + +```markdown +- **Branch-name confidentiality validation is missing.** Security-fix + workflows already require neutral branch names, but no deterministic + check scans skill/docs examples for CVE IDs or embargoed terms. + +``` + +An HTML comment is invisible in rendered markdown and rides alongside +its bullet the way SPDX markers already ride alongside file content. + +The vocabulary is deliberately three predicates, which is what the nine +observed stale bullets require between them: + +| Predicate | Closed when | Evaluation | +|---|---|---| +| `symbol-called:::` | `` is both defined and called within `` | parse call sites, not definitions | +| `path-exists:` | `` exists in the tree | filesystem check | +| `skill-count:=` | exactly `` skills carry `family:` | frontmatter count | + +`symbol-called` deliberately requires a call site rather than a +definition, because a defined-but-unwired function is the exact shape +that makes a name match untrustworthy. + +Severity is split so that adoption is incremental: + +- A predicate that **evaluates to closed** is a HARD violation. The + bullet is provably describing shipped work. +- A predicate that is **malformed or unresolvable** is a HARD + violation. A broken check is worse than no check because it reads as + a passing one. +- A bullet with **no predicate** is a SOFT advisory naming the bullet + and suggesting either a predicate or the explicit narrative marker + below. This is today's behaviour plus a nudge, so nothing breaks on + the day this lands. + +Not every gap has an end state a machine can see. A bullet whose gap is +inherently narrative declares that rather than being left silent: + +```markdown + +``` + +This is the correct annotation for the eleven "experimental, no adopter +pilot has run" bullets in the current tree. They are status notes whose +truth depends on facts outside the repository, and they should never be +machine-checked. The vocabulary's inability to express them is a +feature: the bullets it cannot describe are the bullets it should not +judge. + +### `validate_cross_spec_counts` (HARD) + +Extract claims of the form " skills" scoped to a named family +from any spec section, resolve the live count of skills whose +frontmatter carries that `family:`, and compare. Two specs asserting +different counts for one family is a violation regardless of which is +right, because at most one can be. Narrow and exact: no prose +inference, no threshold, no false positives. + +## Out of scope + +- Deciding whether an arbitrary prose bullet is true. That needs a + model call, and a model call in a deterministic validator would break + the reproducibility the other checks rely on. +- Editing spec text. Every check reports; a human or the update beat + rewrites. +- **Git-recency staleness detection, refuted by measurement.** An + earlier draft proposed a SOFT advisory comparing each spec's last + commit against commits touching the paths in its `Where it lives` + section. Measured against the eight specs known to carry stale + bullets, the signal does not separate them: five scored at or below + the median and two scored zero, while the highest-scoring spec + (`good-first-issue-sweep.md`, 65 commits ahead) had entirely accurate + gaps. Re-measuring per bullet with `git blame` failed the same way, + with eleven specs tied at the history ceiling across both + populations. The two populations move in opposite directions from + what the metric assumes: a narrative gap stays true while its area + churns, and a capability gap goes stale the moment an unrelated PR + ships the capability without touching the spec. Recorded here so the + approach is not re-proposed. +- Removing or reworking `tools/spec-loop/.last-sync`. + +## Acceptance criteria + +1. `PROMPT_plan.md` states the three admissible evidence kinds and that + a name match is not one of them. +2. `PROMPT_update.md` requires a retained Known-gap bullet to be + re-verified against the same rule. +3. `validate_gap_predicates` parses all three predicates plus the + narrative marker, and is registered in the validator orchestrator. +4. A satisfied predicate and a malformed predicate each produce a HARD + violation; an unannotated bullet produces a SOFT advisory only. +5. `validate_cross_spec_counts` flags a family whose claimed count in + one spec differs from its claimed count in another. +6. Every check has unit tests covering the firing case, the clean case, + and the skip case. +7. Running the validator on the live tree reproduces the known + five-versus-six repo-health family discrepancy. + +## Validation + +```bash +uv run --directory tools/spec-validator --group dev pytest +uv run --directory tools/spec-validator --group dev spec-validate +test -f tools/spec-loop/PROMPT_plan.md +test -f tools/spec-loop/PROMPT_update.md +``` + +## Known gaps + +- **Not yet built.** Both prompt edits and both validator checks are + proposed here; nothing has shipped. + +- **The existing bullets need a one-off backfill.** Nine bullets across + eight specs describe shipped capabilities, and eleven more are + narrative status notes needing the `none (narrative)` marker. + Annotating them is manual and independent of building the checks. +- **A predicate can drift from its bullet.** Nothing binds the comment + to the prose above it, so an edit that rewrites a bullet without + updating its predicate leaves a check that passes while describing + something else. Mitigated by the HARD malformed-predicate rule but + not eliminated. +- **The vocabulary will need extending.** Three predicates cover the + nine observed cases. A fourth shape will appear; each addition is a + parser branch plus tests, and the bar for adding one should be a real + bullet that needs it rather than an anticipated one. diff --git a/uv.lock b/uv.lock index b70568a4..f0fe7246 100644 --- a/uv.lock +++ b/uv.lock @@ -369,7 +369,7 @@ dev = [ [package.metadata.requires-dev] dev = [ - { name = "mypy", specifier = ">=2.1.0" }, + { name = "mypy", specifier = ">=2.3.0" }, { name = "pytest", specifier = ">=9.1.1" }, { name = "ruff", specifier = ">=0.15.21" }, ] @@ -495,7 +495,7 @@ dev = [ [package.metadata.requires-dev] dev = [ - { name = "mypy", specifier = ">=2.1.0" }, + { name = "mypy", specifier = ">=2.3.0" }, { name = "pytest", specifier = ">=9.1.1" }, { name = "ruff", specifier = ">=0.15.21" }, ] @@ -945,7 +945,7 @@ provides-extras = ["mcp"] [package.metadata.requires-dev] dev = [ { name = "mcp", specifier = ">=1.28.1" }, - { name = "mypy", specifier = ">=2.1.0" }, + { name = "mypy", specifier = ">=2.3.0" }, { name = "pytest", specifier = ">=9.1.1" }, { name = "ruff", specifier = ">=0.15.21" }, ] @@ -1359,7 +1359,7 @@ dev = [ [package.metadata.requires-dev] dev = [ - { name = "mypy", specifier = ">=2.1.0" }, + { name = "mypy", specifier = ">=2.3.0" }, { name = "pytest", specifier = ">=9.1.1" }, { name = "ruff", specifier = ">=0.15.21" }, ] @@ -1570,7 +1570,7 @@ dev = [ [package.metadata.requires-dev] dev = [ - { name = "mypy", specifier = ">=2.1.0" }, + { name = "mypy", specifier = ">=2.3.0" }, { name = "pytest", specifier = ">=9.1.1" }, { name = "ruff", specifier = ">=0.15.21" }, ] @@ -1612,7 +1612,7 @@ dev = [ [package.metadata.requires-dev] dev = [ - { name = "mypy", specifier = ">=2.1.0" }, + { name = "mypy", specifier = ">=2.3.0" }, { name = "pytest", specifier = ">=9.1.1" }, { name = "ruff", specifier = ">=0.15.21" }, ] @@ -1633,7 +1633,7 @@ dev = [ [package.metadata.requires-dev] dev = [ - { name = "mypy", specifier = ">=2.1.0" }, + { name = "mypy", specifier = ">=2.3.0" }, { name = "pytest", specifier = ">=9.1.1" }, { name = "ruff", specifier = ">=0.15.21" }, ] @@ -1843,7 +1843,7 @@ dev = [ [package.metadata.requires-dev] dev = [ - { name = "mypy", specifier = ">=2.1.0" }, + { name = "mypy", specifier = ">=2.3.0" }, { name = "pytest", specifier = ">=9.1.1" }, { name = "ruff", specifier = ">=0.15.21" }, ] From fdacd3e186bc7be1a1155c2466fc93bb66a90e24 Mon Sep 17 00:00:00 2001 From: Jarek Potiuk Date: Sat, 1 Aug 2026 23:28:14 +0200 Subject: [PATCH 2/2] fixup: correct the repo-health count claim and record landing order MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two review fixes on the spec text. The uv.lock change in the parent commit is left alone: it corrects genuine stale-lock drift (the lock at the base commit had 26 mypy specifiers at >=2.1.0 against 18 in the pyprojects; regenerating brings both to 18/9). - Correct acceptance criterion 7. The discrepancy is three-way, not five-versus-six: triage-mode.md claims five skills, repo-health-family.md claims six, and seven carry family: repo-health in frontmatter. The seventh, audit-finding-fix, is in neither prose list — plausibly deliberate, since it fixes findings rather than producing them. The check must report all three numbers and name the disputed skill rather than presume frontmatter is authoritative. - Note that validate_cross_spec_counts compares against the live count independently, so the case where every spec agrees and all are stale is caught too. Agreement reads as confirmation, which makes it the more dangerous shape. - Record the landing order in Known gaps. The check is HARD and criterion 7 requires it to fire on a violation present in the tree today, so landing it before the counts are reconciled turns main red. Only the SOFT advisory is safe to land first. Generated-by: Claude Code (Opus 5) --- tools/spec-loop/specs/spec-gap-staleness.md | 29 +++++++++++++++++++-- 1 file changed, 27 insertions(+), 2 deletions(-) diff --git a/tools/spec-loop/specs/spec-gap-staleness.md b/tools/spec-loop/specs/spec-gap-staleness.md index fa0ce738..01a936ca 100644 --- a/tools/spec-loop/specs/spec-gap-staleness.md +++ b/tools/spec-loop/specs/spec-gap-staleness.md @@ -146,9 +146,20 @@ Extract claims of the form " skills" scoped to a named family from any spec section, resolve the live count of skills whose frontmatter carries that `family:`, and compare. Two specs asserting different counts for one family is a violation regardless of which is -right, because at most one can be. Narrow and exact: no prose +right, because at most one can be. The live count is compared +independently of that, so the case where every spec agrees and all of +them are stale is caught too — which is the more dangerous shape, since +agreement reads as confirmation. Narrow and exact: no prose inference, no threshold, no false positives. +One caveat the implementation has to face. A count disagreement can mean +the prose is stale, or it can mean membership itself is contested: a +skill may carry `family:` in frontmatter while the prose deliberately +omits it because it is a different kind of thing. The check reports the +discrepancy and names the skills on each side; it does not assume the +frontmatter is right. Deciding membership is a human call, and a HARD +check that presumes an answer will be argued with rather than fixed. + ## Out of scope - Deciding whether an arbitrary prose bullet is true. That needs a @@ -187,7 +198,14 @@ inference, no threshold, no false positives. 6. Every check has unit tests covering the firing case, the clean case, and the skip case. 7. Running the validator on the live tree reproduces the known - five-versus-six repo-health family discrepancy. + repo-health family discrepancy, which is three-way rather than + two-way: `triage-mode.md` claims five skills, + `repo-health-family.md` claims six, and seven carry + `family: repo-health` in frontmatter. The seventh, + `audit-finding-fix`, appears in neither prose list — plausibly + because it fixes findings rather than producing them. The check + must report all three numbers and name the disputed skill, not + pick a winner. ## Validation @@ -207,6 +225,13 @@ test -f tools/spec-loop/PROMPT_update.md eight specs describe shipped capabilities, and eleven more are narrative status notes needing the `none (narrative)` marker. Annotating them is manual and independent of building the checks. +- **Landing order matters, and only the SOFT rule is safe to land + first.** The unannotated-bullet advisory is SOFT, so it changes + nothing on the day it ships. `validate_cross_spec_counts` is HARD and + acceptance criterion 7 requires it to fire on a violation that exists + in the tree today, so landing it before the repo-health counts are + reconciled turns main red for everyone. Reconcile first, then land + the HARD check. - **A predicate can drift from its bullet.** Nothing binds the comment to the prose above it, so an edit that rewrites a bullet without updating its predicate leaves a check that passes while describing