From f537ce201beeac3c78b943e874ebf51e62048910 Mon Sep 17 00:00:00 2001 From: Oriol Torrent Florensa Date: Sun, 6 Sep 2026 12:07:18 +0200 Subject: [PATCH] docs(unic-archon-dlc): record the sixth check as shipped, and what building it taught MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit WI 43028 merged as PR 5854 (`f77ee51` on DXP-DesignSystem's `develop`), so F1's "a sixth check" row stops describing work and starts describing an artefact. F4's "five checks" becomes the F1 table, which is the count. The new F5 holds what the three `unic-dlc-pr-review` iterations produced beyond the check itself. Every row is one shape — a green that means nothing — found in four places in one day, three of them while building the instrument run 3 is read through: - a check that returned `[]` on the barrel shape `develop` gets the day the first component lands, past 12 tests including four failure fixtures; - so a positive control per check, with the technique per-check and 43020 as the counter-example that four of the six controls are still owed; - with no CI, the PR description is the only record that a check ran; - `/pr-review` iteration 3 produced eight findings, wrote none and exited 0 (#465); - four of the Consumer's eight declared commands cannot fail; - `estimations` moved `off` to `both` between the runs, retiring run 2's cost and halt figures. The last two rows are what the seal must mark not-comparable rather than score. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01THGdEfJM9zVBvMmTZPwTQk --- .../research/2026-08-31-dlc-run-2-findings.md | 135 ++++++++++++++++-- 1 file changed, 121 insertions(+), 14 deletions(-) diff --git a/docs/research/2026-08-31-dlc-run-2-findings.md b/docs/research/2026-08-31-dlc-run-2-findings.md index 158fc7f4..0240a585 100644 --- a/docs/research/2026-08-31-dlc-run-2-findings.md +++ b/docs/research/2026-08-31-dlc-run-2-findings.md @@ -232,16 +232,16 @@ different question: _what would have caught this without anyone reading anything ## F1 — Mechanisms (M) -| From | The rule that already existed | What would catch it | -| --------------------------------------------------------- | ----------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| **11** `dxp:font-body` absent | ADR-0005, explicitly, written from run 1's fix | An assertion that every component root's class list carries it. **A document has now failed at this twice.** | -| **1**, **15** `cn` extends one scale of four | none — but `utils.ts`'s own comment diagnoses the general defect and fixes one case | A check that **every** custom `---*` in `styles.css` appears in `utils.ts`. Kills both findings with one test. | -| **3** `argTypes` incomplete | `CLAUDE.md`, **and** an acceptance criterion on all seven slices | A check that every prop in the component's type has an `argTypes` entry. **This failed three times after being made a criterion — a criterion is not a mechanism.** | -| **12** icon-only badge unnamed | the component's own warning | An a11y rule that flags an unnamed interactive element. **axe does not flag an unnamed `span`, so "a11y passes" held vacuously.** | -| **12**, second half (added 2026-09-02, PR 5843) | none | The badge root at `5e683a7` is a plain `span`, not interactive, so an interactive-naming rule (43024) does not catch it. Finding 12 is a **text-alternative** defect: an element whose only content is an `aria-hidden` glyph, conveying a status and exposing nothing. Needs its own rule and work item; candidate shape from `DS-43020`: no accessible name, no text content, at least one element child, every descendant `aria-hidden`. **Not filed yet.** | -| **a sixth check** (added 2026-09-02, PR 5843 iteration 2) | none | **A component with no `test`-tagged story is invisible to 43022, 43023 and 43024**: the hooks fire on every rendered story, and a build that ships a component and no story satisfies all five mechanisms while enforcing none. Closing it needs a check that walks `packages/ui-react/src/` for a component directory no story renders. Its shape is a decision, not an implementer's guess: whether an icon counts, whether a `DEMOS` composition counts, whether a component may ship story-less behind a tag. **Load-bearing for run 3; filed 2026-09-02 as [WI 43028](https://dev.azure.com/FZAG/dxp/_workitems/edit/43028)** under 42989, predecessor 43020, with the three shape decisions derived from the bar and stated in the body. PR 5843 thread 65652 holds the evidence. | -| **4** one story without a snapshot | `CLAUDE.md` § The bar, item 3 | `count(test-tagged stories) == count(snapshots)`. Round 2 did this arithmetic by hand; nothing runs it. | -| **25** badge and icon position | **the contract records the arrangement with y-coordinates** | An assertion on document order and geometry — **blocked by F3, because the bar may forbid the assertion that would prove it** | +| From | The rule that already existed | What would catch it | +| ---------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **11** `dxp:font-body` absent | ADR-0005, explicitly, written from run 1's fix | An assertion that every component root's class list carries it. **A document has now failed at this twice.** | +| **1**, **15** `cn` extends one scale of four | none — but `utils.ts`'s own comment diagnoses the general defect and fixes one case | A check that **every** custom `---*` in `styles.css` appears in `utils.ts`. Kills both findings with one test. | +| **3** `argTypes` incomplete | `CLAUDE.md`, **and** an acceptance criterion on all seven slices | A check that every prop in the component's type has an `argTypes` entry. **This failed three times after being made a criterion — a criterion is not a mechanism.** | +| **12** icon-only badge unnamed | the component's own warning | An a11y rule that flags an unnamed interactive element. **axe does not flag an unnamed `span`, so "a11y passes" held vacuously.** | +| **12**, second half (added 2026-09-02, PR 5843) | none | The badge root at `5e683a7` is a plain `span`, not interactive, so an interactive-naming rule (43024) does not catch it. Finding 12 is a **text-alternative** defect: an element whose only content is an `aria-hidden` glyph, conveying a status and exposing nothing. Needs its own rule and work item; candidate shape from `DS-43020`: no accessible name, no text content, at least one element child, every descendant `aria-hidden`. **Not filed yet.** | +| **a sixth check** (added 2026-09-02, PR 5843 iteration 2 · **SHIPPED 2026-09-06**) | none | **A component with no `test`-tagged story is invisible to 43022, 43023 and 43024**: the hooks fire on every rendered story, and a build that ships a component and no story satisfies the other mechanisms while enforcing none. Closing it needed a check that walks `packages/ui-react/src/` for a component directory no story renders. Its shape was a decision, not an implementer's guess: whether an icon counts, whether a `DEMOS` composition counts, whether a component may ship story-less behind a tag — all three ruled in the body. **Filed 2026-09-02 as [WI 43028](https://dev.azure.com/FZAG/dxp/_workitems/edit/43028)** under 42989, predecessor 43020. **Merged 2026-09-06 as PR 5854 (`f77ee51` on the Consumer's `develop`), 43028 Resolved**: `checks/component-stories.ts` plus 17 tests, `story-tags.ts` gains `excludingTags`, and `countStories` becomes the one reader of the tag arithmetic so 43022 and 43028 cannot drift apart. Three `unic-dlc-pr-review` iterations, twenty-three findings. PR 5843 thread 65652 holds the original evidence; PR 5854 thread 65722 holds the review. **What it produced beyond the check is in F5.** | +| **4** one story without a snapshot | `CLAUDE.md` § The bar, item 3 | `count(test-tagged stories) == count(snapshots)`. Round 2 did this arithmetic by hand; nothing runs it. | +| **25** badge and icon position | **the contract records the arrangement with y-coordinates** | An assertion on document order and geometry — **blocked by F3, because the bar may forbid the assertion that would prove it** | ## F2 — Reads (R) @@ -266,10 +266,11 @@ different question: _what would have caught this without anyone reading anything **Widened 2026-09-02 15:58 (maintainer).** Run 3 also carries [#441](https://github.com/unic/unic-agents-plugins/issues/441) (after [#452](https://github.com/unic/unic-agents-plugins/issues/452)), so one run assesses two legs: on the specs leg, whether -the grilling fires and the halts halt; on the build leg, whether the five checks (ADO 43020–43024) reach the implementer. -This is not a confound, because the five checks test properties a PRD cannot mask (`dxp:font-body` on the root, an +the grilling fires and the halts halt; on the build leg, whether the checks in the F1 table above reach the implementer. +**Six of them ship as of 2026-09-06** — ADO 43020–43024 and 43028 — and the table is the count, not this sentence. +This is not a confound, because the checks test properties a PRD cannot mask (`dxp:font-body` on the root, an `argTypes` entry per prop, a snapshot per test story, every `@theme` scale in `cn()`, a name on every interactive -element), so each leg's result is observable on its own. What the run can no longer say is whether the build got +element, a rendered story per component directory), so each leg's result is observable on its own. What the run can no longer say is whether the build got better _overall_ because of one change or the other, and that was never the hypothesis. The sealed predictions must cover both legs. @@ -322,7 +323,7 @@ run-2-versus-run-3 diff is new infrastructure rather than a changed chain. **Acc Sharpened 2026-09-02, while amending 43020–43024: `apps/storybook-react`'s `test` script pins `--project=storybook` and `packages/ui-react` has no `test` script, so the new node-environment project is invisible to root `pnpm test` until one of those changes. That script change is part of the same new -infrastructure and belongs in the confound, not in run 3's report as a surprise. And the five checks' "Done +infrastructure and belongs in the confound, not in run 3's report as a surprise. And the checks' "Done when" lines were all satisfiable on the empty `develop` tree — the all-negative shape #381 already measured — so each now proves its negative path on a fixture (WI revisions of 2026-09-02). @@ -335,6 +336,112 @@ the fixture proofs of 43023 and 43024. Run 3's diff against run 2 carries both. --- +## F5 — What building the mechanisms taught, which the mechanisms did not + +Added 2026-09-06, from WI 43028 (the sixth check, PR 5854) and the three `unic-dlc-pr-review` iterations against it. +**Every row here is about a green that means nothing.** That is one shape, found in four places in a single day, and +three of the four were found while building the instrument run 3 will be read through. + +### 1. A mechanism can read green by construction + +43028's check shipped with 12 tests, four of them fixtures written to fail. Iteration 1 found a hole none reached: + +```ts +import { BadgeMain } from './badge-main/BadgeMain.js' +export { BadgeMain } // no module specifier +``` + +That clause hit the skip written for `export {}`, so the check returned `[]` directories, `[]` gaps, green run. **The +barrel shape that triggers it is the one `develop` gets the day the first component lands**, if whoever adds it +writes a local re-export. The check would have been silently inert exactly when it first mattered. Fixed by turning +the skip on the name count rather than on the specifier. + +**Why this outranks the bug.** The checks are run 3's treatment. If a check can pass vacuously, then in the run's +output _"the mechanism fired and found nothing"_ and _"the mechanism was inert"_ are the same observation — and +telling those two apart is the whole of run 3's question. + +### 2. So every check gets a positive control, and the technique is per-check + +**The obligation is universal; the technique is not.** Prove each check can fail against the run's own tree before +the result is read. A check with no working control scores **inert**, never pass — a finding about the mechanism, +which is a different finding from "the implementer produced no defect". + +| Check | Reads | Its control | +| ----------- | ------------------------------------------------------------- | -------------------------------------------------------------------- | +| 43028 | the barrel `packages/ui-react/src/index.ts` | append a failing export, watch it fail naming the directory, revert | +| 43020 | stylesheets, via `uiReactStyleSheets()` — **throws on empty** | **not** a barrel append: needs a `@theme` scale `cn()` does not know | +| 43021–43024 | stories | unwritten. Four controls owed before run 3's result is read | + +Fixtures do not substitute. Fixtures are what missed row 1. + +**The correction that produced this table is itself the finding.** The first version of the rule generalised 43028's +technique to all six checks, and 43020 breaks it. The same session over-claimed the same way inside +`checks/README.md`'s new paragraph — the paragraph it added to stop over-claims — writing "every real-tree call here +passes on an empty set", false of 43020 for the same reason. Two people, one hour, each inside prose whose subject +was that move: + +> **A quantifier over a set of mechanisms is an unverified claim per member until each member is named.** "Every +> real-tree call here" and "each of the six" are cheaper to write than the enumeration and read as if the +> enumeration had been done. + +### 3. With no CI, the PR description is the only record that a check ran + +Three of iteration 2's four findings were in the **PR description**, not the code. One was Critical: it claimed 12 +tests while the file ran 16, because a later commit falsified the author's own evidence row and two re-reads missed +it. + +`DXP-DesignSystem` has no CI, so nothing re-runs what a description asserts. **A description patched rather than +rewritten is an unversioned test report.** The treatment is cheap and belongs in every Consumer opener: **quote the +runner's output rather than asserting it — a quoted count goes stale visibly, an asserted one does not.** The author +then repeated the shape in its own handover message, after the Box had caught it twice, and sharpened the entry: +**a number written from memory is not evidence, whether or not it was ever true.** + +### 4. The reviewer itself failed the same bar, on its first observation + +`unic-dlc-pr-review` iteration 3 (run `a7e55042`, 2026-09-05) reviewed the diff, produced **eight findings across +both axes**, wrote none to disk, and exited 0 reporting success. The `review` node's `node_output` holds all 14038 +characters of the review; its last line is **"Standards axis still running."** — it returned believing one of its +two axes had not finished. `synthesize` then read the empty directory and emitted three zeros, `reconcile` emitted +`verdicts: 0, unmatched_priors: 18`, and `post` refused because publishing would have meant inventing 18 verdicts. + +**A Box that reviewed and lost its review is indistinguishable, at the exit code, from a Box that found nothing.** +`post`'s refusal reads as "nothing to say", not as "something broke". The eight findings were recovered by hand out +of `~/.archon/archon.db`; six were real and were fixed, including one falsehood the check itself would have printed. +Filed as [#465](https://github.com/unic/unic-agents-plugins/issues/465), p1. **Not a blocker for run 3 — a caution +about reading it**, because `/pr-review` is one of the mechanisms run 3 is read through. + +### 5. Four of the Consumer's eight declared commands cannot fail + +Measured 2026-09-05 against `sdlc_needs` in `.archon/unic-dlc.config.yaml`, each documented in a comment beside its +own key: + +| Key | Why a pass proves nothing | +| ---------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `lint` | `eslint-plugin-only-warn` downgrades every rule, and `apps/storybook-react` runs bare `eslint .` with no `--max-warnings 0` — and that app holds every story, check and test | +| `format` | `prettier --write` rewrites the tree and exits 0 either way | +| `coverage` | `Unknown% (0/0)`; vitest does not collect an untested file outside the project root, and the stories import `@repo/ui-react`, which resolves to `dist/index.js`. Structural, not a state waiting on the first component | +| `test` | `passWithNoTests: true`. Inert today at 51 tests; a tag-filter change that empties the run turns the merge floor into a pass | + +`test` and `check-types` are the two that mean something today. **Any run-3 prediction about `lint`, `format` or +`coverage` is unfalsifiable and must not be scored.** A related non-defect, recorded so nobody reads it as +evidence: `sdlc_needs.install` is a two-command chain and the Box reports one `ok`, so a fetched chromium and a +warm-cache no-op are the same line. + +### 6. `estimations` changed the baseline between run 2 and run 3 + +`estimations` was `'off'` from 2026-08-17 through run 2 and is `'both'` since `1995549` (2026-09-05 16:17), kept for +run 3 by the maintainer's ruling. It turns on the provisional wave in `/specs` and the definitive wave in +`/tickets`. **Not a confound for the treatment** — planning legs, separately observable from whether a check fires — +but it retires two of the five numbers in § D: + +- **Cost**: build 6, review round 1 3, round 2 2. Two extra planning waves add cost run 2 never paid. +- **Unattended: no — four halts in `/specs` alone.** An estimation wave can halt, so a run-3 halt count is counting + a different command. + +**Dedup, reconcile and praise still compare.** Together with the three commands in row 5, these two are what the +seal marks not-comparable rather than scores; do not restate them as a total anywhere, because the list is the +count. + # E. Proposed tickets ## `unic-agents-plugins` (GitHub) — FILED 2026-09-01