Skip to content

feat: close the audit's reference and content blind spots, ship lint.yml in the kit #31

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mcarter-astronautdev merged 3 commits into from
Aug 2, 2026
Merged

feat: close the audit's reference and content blind spots, ship lint.yml in the kit #31

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
3832f20
feat: close the audit's reference and content blind spots, ship lint.…
mcarter-astronautdev Aug 2, 2026
d0f0c6e
docs: record actionlint as a required check on main, and the enforce_…
mcarter-astronautdev Aug 2, 2026
474f2f2
fix: address the review — audit guards, SC2015, and four wrong doc cl…
mcarter-astronautdev Aug 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
64 changes: 48 additions & 16 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,8 +42,9 @@ one wave.

**Decided 2026-08-02: shipped as `v1.11.0` rather than folded into the v1.10.0 wave.** Folding it in
would have put `claude.yml` content on 18 branches that exists in no tag, and `tools/fleet-pin-audit.sh`
compares only stub pin lines against the latest tag — never `templates/` content — so it would have
reported the fleet uniform and green over the gap. Executed as: merge → tag `v1.11.0` at that merge
compared only stub pin lines against the latest tag — never `templates/` content — so it would have
reported the fleet uniform and green over the gap. (That blind spot is now closed: the audit also
checks `templates/` against the latest tag and every waved file against `templates/`.) Executed as: merge → tag `v1.11.0` at that merge
commit → repin all six stubs to `v1.11.0` → **one** wave. That is what `v1.7.0` and `v1.8.0` each did.

**Note on the pin sequence:** `v1.9.0` (`a54c91e`, the store-secret rename) never got its kit repin
Expand Down Expand Up @@ -109,8 +110,9 @@ Waved to all 21 repin targets on 2026-08-02; fleet uniform, 108 pins, zero stale
green on the new names, old-name secrets deleted.
- **Reusables unchanged.** Note this release **never got its kit repin commit** — `templates/` sat at
`v1.8.0`'s SHA while the deployed fleet was waved to `a54c91e`, leaving the fleet a release *ahead*
of the kit templates until `v1.11.0` closed it. That gap is invisible to
`tools/fleet-pin-audit.sh`; see [`docs/fleet-operations.md`](docs/fleet-operations.md).
of the kit templates until `v1.11.0` closed it. That gap was invisible to
`tools/fleet-pin-audit.sh` at the time; its reference check now catches exactly this shape — see
[`docs/fleet-operations.md`](docs/fleet-operations.md).

### `v1.8.0` (2026-07-31, kit-only)

Expand Down Expand Up @@ -168,8 +170,11 @@ Waved to all 21 repin targets on 2026-08-02; fleet uniform, 108 pins, zero stale
prefixes, NFC-normalize, **collapse whitespace**, and diff against the blockquote portion of
`claude.yml`'s `--append-system-prompt`. The whitespace collapse is mandatory — the kit flattens
canonical's paragraph break to a single space (forced by the no-newline constraint), so a strict
byte compare reports a false failure. Nothing else re-checks this: `fleet-pin-audit.sh` greps only
stub pin lines, and `DRIVER_AGENTS_REF` is a raw SHA in an env var that no bot can bump.
byte compare reports a false failure. **Nothing else re-checks this.** `fleet-pin-audit.sh` proves
the fleet matches `templates/github/claude.yml` — it cannot prove that file's blockquote still
matches canonical at the new pin, and `DRIVER_AGENTS_REF` is a raw SHA in an env var that no bot
can bump. The audit catches a *fleet* that fell behind `templates/`; only this step catches
`templates/` falling behind driver-agents.
2. Repin every caller stub in `templates/github/` to that tag's SHA and commit. Until this
lands, the kit's stubs still point at the PREVIOUS tag's reusables.
- **If the release ADDS a reusable**, its stub lands *in this step*, not in the PR that added the
Expand All @@ -178,7 +183,8 @@ Waved to all 21 repin targets on 2026-08-02; fleet uniform, 108 pins, zero stale
`bonsai-status-sync.yml`'s stub landed at `v1.11.0`. `lint.yml` fails the build on any stub left
carrying a placeholder pin, so this step cannot be silently skipped.
3. Only then re-copy `templates/github/` into consumer repos (`tools/fleet-pin-audit.sh --stale`
to confirm the fleet converged afterwards).
to confirm the fleet converged afterwards — it now checks waved file **content** against
`templates/`, not just the pin line, and exits non-zero on any drift, so a wave can gate on it).
- **When a full workflow becomes a stub** (as `bonsai-status-sync.yml` did — this applies to the
v1.11.0 wave specifically), the wave diff
contains a `templates/github/` path AND a `.github/workflows/` path with the SAME basename. The
Expand Down Expand Up @@ -220,26 +226,48 @@ produces a silent `startup_failure` — no check run, no notification).

**The onboarding kit lives here: `templates/github/`** (moved from `driver-bonsai-mcp` 2026-07-15). It
carries the six caller stubs above plus `claude.yml` (the implementer, still a full per-repo workflow),
`shopify-tool-smoke.yml` (store repos only) and `pull_request_template.md`.
`shopify-tool-smoke.yml` (store repos only), `lint.yml` (actionlint over the installing repo's own
workflows) and `pull_request_template.md`.

**Not every repo takes the whole kit.** A repo that is not on the Bonsai → PR pipeline can install
`pr-first-review.yml` + `lint.yml` alone and skip the rest as inert weight. That subset is proposed for
[`driver-agents`](https://github.com/DriverDigital/driver-agents/pull/6) and
[`driver-agents-app`](https://github.com/DriverDigital/driver-agents-app/pull/2) — **both PRs are open,
not merged**, and they should land only *after* this repo ships `templates/github/lint.yml`, since until
then the file they install has no upstream source to be re-copied from. The trade-off is written up in
`templates/github/README.md` under *Partial install*.

**`bonsai-status-sync.yml` finished converting at `v1.11.0`.** The reusable landed 2026-08-02 and its stub
landed in this tag's repin commit, so the kit now installs a 66-line stub instead of the old 190-line copy —
see *Release + repin order* above and [`docs/reusable-conversion-scope.md`](docs/reusable-conversion-scope.md).
The two-step was deliberate and matches how `dependabot-keep-current` was added: a new reusable's stub cannot
be pinned until the tag containing that reusable exists, so the reusable lands first and the stub follows in
the repin commit. `lint.yml` fails the build on any stub still carrying a placeholder pin. **The fleet has not
been waved yet** — consumer repos still run the 190-line copy until the v1.11.0 wave.
the repin commit. `lint.yml` fails the build on any stub still carrying a placeholder pin. **The v1.11.0 wave
has landed** — `tools/fleet-pin-audit.sh` reads clean across all 21 repo@branch pairs (108 pin rows at
`90f0d066`, 127 files byte-identical to `templates/` after store-handle normalization, verified 2026-08-02),
so every consumer repo now runs the 66-line stub.

**`claude.yml` stays a per-repo copy** — that half of the conversion is tabled pending the OIDC spike (whether
Claude App token minting survives inside a cross-repo reusable), so it remains the kit's main drift surface
and the reason re-copies still need care.

Two files in `.github/workflows/` are **this repo's own CI**, not products — they are `workflow_call`-free
and never ship to the fleet: `lint.yml` (actionlint + shellcheck over the reusables *and* the kit, so a
broken workflow can't reach consumer repos — it reports red on the PR, but **pin `actionlint` as a
required check** if you want it to actually block a merge) and `dependabot-auto-merge.yml` (auto-merges
this repo's own `github-owned` Dependabot bumps; the `claude-code-action` group is deliberately excluded,
so those land by hand).
broken workflow can't reach consumer repos) and `dependabot-auto-merge.yml` (auto-merges this repo's own
`github-owned` Dependabot bumps; the `claude-code-action` group is deliberately excluded, so those land by
hand).

**`actionlint` is a required status check on `main`** (set 2026-08-02) — before that, `lint.yml` could
report red without being able to block. Note the name collision: this repo's own `lint.yml` and the kit's
`templates/github/lint.yml` are **different files**. The kit one runs actionlint over the installing repo's
`.github/workflows/` and nothing else; this one additionally lints `templates/github/`, gates on placeholder
pins, and asserts `claude.yml`'s system prompt still tokenizes. Both use the job id `actionlint`, so the
required-check context string is the same either way. `enforce_admins` stays **`false`** here, deliberately
— which means an admin can still merge past a red `actionlint`. Requiring the check makes it binding for
everyone else and puts a red X in front of an admin who previously had nothing to override; that was worth
having on its own. Flipping the flag would break this repo's own release habit — six commits on `main`,
`v1.9.0`'s included, were pushed directly with no PR. Detail in
[`docs/fleet-operations.md`](docs/fleet-operations.md#branch-protection).

## The three identities

Expand Down Expand Up @@ -299,8 +327,12 @@ task. The `config/reviewers.json` copy in **this** repo is reference only — no

## First-run / required-check

The `dependabot-validate` job **always runs and branches internally** (non-Dependabot PRs no-op green) — a
*skipped* required check counts as not-passed and would block every human PR, so it must never be `if:`-skipped.
The `dependabot-validate` job **always runs and branches internally** (non-Dependabot PRs no-op green), and
must never be `if:`-skipped. The mechanism is worth stating precisely, because the intuitive version is
wrong: GitHub *does* accept a check run whose conclusion is `skipped`. The problem is that `if:`-skipping
the **caller** job means the reusable never starts, so the nested `validate / validate` context is never
created at all — and a required context with **no check run** for the head commit blocks forever. Reason
about whether a check run exists for the head SHA, not about the word "skipped".
After the first run on a test PR: pin the **exact required-check context GitHub reports** — for a
reusable-workflow job it is `<caller-job-id> / <reusable-job-id>`, expected **`validate / validate`** (the
workflow display name is NOT part of the context; copy the literal string from the first run's checks list).
Expand Down
109 changes: 88 additions & 21 deletions docs/fleet-operations.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,22 +95,45 @@ Dependabot drift — but survey, don't assume.

---

## What the pin audit cannot see

`tools/fleet-pin-audit.sh` greps only
`DriverDigital/workflows/.github/workflows/<name>@<sha>` and compares the SHA to the latest tag.
Three consequences:

- **A file with no `uses:` line is invisible.** An unconverted 190-line copy has none, so the audit
cannot tell a repo that was skipped by a stub conversion from one that never carried the file.
- **Content is never compared.** `DRIVER_AGENTS_REF` is a raw SHA in an `env:` block, and the
system-prompt text is just text. A fleet running kit content from no tag reports clean.
- **The reference itself can drift.** The audit compares against the latest *tag*, never against
`templates/`. When those disagree the audit reports uniform while real drift sits in the source of
truth — which is exactly how the v1.9.0 gap went unnoticed for a day.

This is why the release order requires the tag to contain what gets waved, and why a
`DRIVER_AGENTS_REF` bump must re-run the canonical parity check by hand.
## What the pin audit checks — and the one thing it still cannot see

`tools/fleet-pin-audit.sh` used to grep only
`DriverDigital/workflows/.github/workflows/<name>@<sha>` and compare that SHA to the latest tag,
which left three holes. Two of them let the v1.9.0 gap read green for a day. All three are now
checked; the script runs them in this order and exits non-zero if any fires:

1. **Reference** — every `uses:` pin in `templates/github/*.yml` equals the latest tag's SHA.
Checks 2 and 3 measure the fleet *against* `templates/`, so a stale reference makes both of them
lie. This is the v1.9.0 failure exactly: the wave repinned the fleet to `a54c91e` while the kit's
own stubs still said `80c35fe`, and an audit that only ever compared deployed pins to the latest
tag called the fleet uniform throughout. When this fires, nothing below it means anything —
fix step 2 of the README's release order first.
2. **Pins** — each deployed caller stub's `uses:` SHA vs that tag. The original check, unchanged.
3. **Content** — the whole waved file vs its `templates/github/` source, line for line. This is
what closes the other two holes: a file with **no `uses:` line at all** (an unconverted 190-line
copy of what is now a 66-line stub) is no longer invisible, and `DRIVER_AGENTS_REF` — a raw SHA
in an `env:` block that no bot can bump — is now compared like any other line.

Two things worth knowing about check 3:

- **Exactly two things are normalized away.** First, `SHOPIFY_STORE_NAME` — the one difference a
correctly-waved repo is *supposed* to have. Second, trailing blank lines and the final newline:
the three stub-rails-only pairs (`Team-Laird@develop`, `The-Gathery@develop`,
`driver-bonsai-mcp@main`) were waved without a final newline and are otherwise identical, and nine
permanently-red rows for a byte nobody can act on is how a detector stops being read. Internal
blank lines *are* compared. Everything else that differs is reported, third-party action pins
included: a repo whose Dependabot bumped `actions/checkout` past the kit's pin is drift worth
seeing, and it means the kit is behind, not that the repo is wrong.
- **`DriverDigital/workflows` itself is skipped.** Its `.github/workflows/` holds the *reusables*,
which share basenames with the stubs that call them — `pr-first-review.yml` is a ~200-line
reusable there and a 25-line stub in the kit — so comparing it against `templates/` would report
seven phantom drifts — the six stubs plus `lint.yml`, whose kit copy is a trimmed version of
this repo's own CI file of the same name.

**Still unchecked: the tripwire parity between `templates/` and canonical.** The audit proves the
fleet matches `templates/github/claude.yml`; it cannot prove that file's `--append-system-prompt`
blockquote still matches driver-agents `docs/agent-instructions-shopify.md` at the pinned
`DRIVER_AGENTS_REF`. That comparison is by hand, at release time — step 1 of the release order.

---

Expand Down Expand Up @@ -153,9 +176,53 @@ the installed stub — the wave covers it anyway.

`enforce_admins` is `false` fleet-wide, which is what makes direct-push waves work. Two live kit
branches have **no protection at all** — `studio-sulzer@main` and `Team-Laird@develop` (404 on the
protection endpoint). Every other kit branch is protected. The kit's onboarding steps assume a
human-approver rule exists, so on those two a bot signal alone could satisfy a merge.
protection endpoint). Every other kit branch has a protection object — but **having one is not the
same as requiring a human**, and the gap is wider than those two. Surveyed across all 21 pairs
2026-08-02:

| Pairs | `required_approving_review_count` | |
|---|---|---|
| 11 | `1` | Avara, Driver-Digital-Website, Kissy-Kissy, LaPointe, LittleMe, The-Gathery, client-workspaces, driver-bonsai-mcp, foundrae-blackridge, plugins, vite-plugin-shopify-clean |
| **8** | **`0`** | **every Palmers branch** — `main`, `-au`, `-ca`, `-in`, `-ma`, `-me`, `-sa`, `-uk` |
| **2** | **no protection at all** | **`studio-sulzer@main`, `Team-Laird@develop`** |

The kit's onboarding steps assume a human-approver rule exists. On **10** of the 21 pairs it does
not, so a bot signal alone could satisfy a merge — not the 2 this section used to name.

On this repo, `main` requires **`actionlint`** (set 2026-08-02; before that `required_status_checks`
had `strict: true` but empty `contexts`, so `lint.yml` could report red without being able to block).
The context is the **job id** at `.github/workflows/lint.yml:28` — the workflow-level `name:` is not
part of it. Applied through the narrow sub-resource, never a whole-object `PUT`:

```bash
gh api -X PATCH repos/DriverDigital/workflows/branches/main/protection/required_status_checks \
-f 'contexts[]=actionlint'
```

`PUT /branches/{branch}/protection` **replaces** the entire protection object, so any field left out
of the body is silently deleted — the 1-approval review rule included. The `PATCH` above touches
`required_status_checks` and nothing else; diffing the full object before and after confirmed only
`contexts`/`checks` moved. GitHub bound the context to the Actions app (`app_id: 15368`) on its own,
which is the stricter outcome: only a check run from Actions can satisfy it.

**`enforce_admins` stays `false` here — deliberately, and know what that buys.** With it `false`, an
admin can merge past *everything*: a red `actionlint`, no approval, `strict` or not. So requiring the
check does not make it unbypassable for Maria — it makes it unbypassable for everyone else, and it
puts a red X in front of an admin who would otherwise have had nothing to override. That is the
actual value, and it was worth having either way.

Flipping it to `true` was considered and rejected, and the reason is stronger than uniformity: **this
repo is itself pushed to directly.** No repin *wave* targets it, but six commits on `main` have no
associated PR — including `a54c91e`, the v1.9.0 release commit, pushed 2026-08-01:

```
a54c91e feat(kit): rename store app secrets to DRIVER_ENGINEERING_APP_* (v1.9.0)
5bee8c4 Onboarding kit moves here from driver-bonsai-mcp (repo-split Stage 0)
34826c9 fix: dependabot-validate falls back to npm install when no lockfile — v1.5.4
c960ec1 fix: allowed_bots claude[bot] so round-1 actually reviews — v1.5.2
69bd828 fix: drop the gh-based author re-check — v1.5.1
499a51e feat: ticketed-review reusable (Phase 3 ticketed-rail auto-review loop) — v1.5.0
```

On this repo, `main` has `required_status_checks` with `strict: true` but empty `contexts` — so
`lint.yml` reports red without being able to block. The context string to add is **`actionlint`**
(the job id at `.github/workflows/lint.yml:28`; the workflow-level `name:` is not part of it).
`enforce_admins: true` would have blocked every one of those. Releases here have repeatedly gone out
as direct pushes, so the flag stays `false` until that stops being true.
Loading