From 8e33c73a3d953009841575869f04536ebaad5a29 Mon Sep 17 00:00:00 2001 From: Stefan Ayala Date: Fri, 10 Jul 2026 17:44:12 -0700 Subject: [PATCH] docs(models): GPT-5.6 Sol supersedes GPT-5.5 as cross-model reviewer MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit OpenAI shipped a new flagship family (gpt-5.6-sol / gpt-5.6-terra / gpt-5.6-luna) superseding GPT-5.5. Updates the reviewer-model guidance inherited by every consumer repo: AI_SETUP_LANES.md, README.md, CLAUDE_CODE_SDLC_WIZARD.md, both SKILL.md copies, AGENTS.md, and this repo's own CLAUDE.md now name GPT-5.6 Sol (fallback: Terra) at xhigh, with historical citations to GPT-5.5/5.4 left untouched. Adds a documented (non-default) escalation to max/Pro mode for unusually risky PRs in AI_SETUP_LANES.md's Final Review Policy, and 8 new per-location regression tests in test-doc-consistency.sh — mutation-verified against Sol/Terra/Luna swaps and a version-bump scenario, not just aggregate "mentions 5.6 somewhere" checks. Reviewed by Fable and Codex (GPT-5.6 Sol, xhigh) across 3 rounds to CERTIFIED. ROADMAP #439 --- AGENTS.md | 2 +- AI_SETUP_LANES.md | 26 +++--- CLAUDE.md | 2 +- CLAUDE_CODE_SDLC_WIZARD.md | 10 +-- README.md | 10 +-- ROADMAP.md | 1 + cowork/skills/sdlc/SKILL.md | 2 +- skills/sdlc/SKILL.md | 2 +- tests/test-doc-consistency.sh | 158 ++++++++++++++++++++++++++++++++++ 9 files changed, 187 insertions(+), 26 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 7c11b628..38ac4241 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -13,7 +13,7 @@ Meta-repository — SDLC Wizard documentation, automation, and a zero-dep Node.j ## AI Setup Lanes -This repo recommends four setup lanes — Setup A (Sonnet 5 + Fable advisor, GPT-5.5 xhigh reviewer, recommended default), Setup B (Opus 4.6 Stability: Fable advisor, Opus 4.6 max driver, GPT-5.5 xhigh reviewer — legacy flagship), Setup C (OpusPlan Hybrid: Opus 4.8 plan mode + Sonnet 5 execute, GPT-5.5 xhigh reviewer), and Setup D (Claude Lite: Sonnet driver, no reviewer). See [`AI_SETUP_LANES.md`](AI_SETUP_LANES.md) for the full pick list. +This repo recommends four setup lanes — Setup A (Sonnet 5 + Fable advisor, GPT-5.6 Sol xhigh reviewer, recommended default), Setup B (Opus 4.6 Stability: Fable advisor, Opus 4.6 max driver, GPT-5.6 Sol xhigh reviewer — legacy flagship), Setup C (OpusPlan Hybrid: Opus 4.8 plan mode + Sonnet 5 execute, GPT-5.6 Sol xhigh reviewer), and Setup D (Claude Lite: Sonnet driver, no reviewer). See [`AI_SETUP_LANES.md`](AI_SETUP_LANES.md) for the full pick list. The lanes are guidance, not a hard rule — maintainer override is always allowed. diff --git a/AI_SETUP_LANES.md b/AI_SETUP_LANES.md index e371e208..03ca2508 100644 --- a/AI_SETUP_LANES.md +++ b/AI_SETUP_LANES.md @@ -10,10 +10,10 @@ This is **guidance, not a hard rule**. Maintainer override is always allowed. |------|-------|--------| | **Advisor** | Fable 5 (via `advisorModel: "fable"`) | `high` (server-side) | | **Driver** | Sonnet 5 (`claude-sonnet-5`) | `high` default, `/effort xhigh` for hard tasks | -| **Reviewer** | Codex (GPT-5.5) xhigh | — | +| **Reviewer** | Codex (GPT-5.6 Sol) xhigh | — | | **Escalation** | Opus 4.8 xhigh or Fable 5 review | When stuck or high-stakes | -The new standard. Sonnet 5 beats Opus 4.6 on every coding benchmark (SWE-bench Verified 85.2% vs 80.8%, Terminal-Bench 80.4% vs 65.4%) while using ~5x less Max quota per turn. Fable 5 advises at key decision points via native `advisorModel` (v2.1.170+). GPT-5.5 xhigh reviews cross-family. Escalate to Opus 4.8 xhigh for the hardest debugging or architecture decisions — don't run Opus as the daily driver (burns Max limits 2-3x faster). +The new standard. Sonnet 5 beats Opus 4.6 on every coding benchmark (SWE-bench Verified 85.2% vs 80.8%, Terminal-Bench 80.4% vs 65.4%) while using ~5x less Max quota per turn. Fable 5 advises at key decision points via native `advisorModel` (v2.1.170+). GPT-5.6 Sol xhigh reviews cross-family. Escalate to Opus 4.8 xhigh for the hardest debugging or architecture decisions — don't run Opus as the daily driver (burns Max limits 2-3x faster). **Effort escalation ladder:** Start at `high` (Sonnet 5's default and sweet spot). Raise to `xhigh` when doing hard debugging, multi-file migrations, or long agent runs. `max` is rarely worth it — doubles cost for marginal gains per CodeRabbit testing. @@ -25,7 +25,7 @@ The new standard. Sonnet 5 beats Opus 4.6 on every coding benchmark (SWE-bench V |------|-------|--------| | **Advisor** | Fable 5 (via `advisorModel: "fable"`) | `high` (server-side) | | **Driver** | Opus 4.6 (`claude-opus-4-6`) | `max` | -| **Reviewer** | Codex (GPT-5.5) xhigh | — | +| **Reviewer** | Codex (GPT-5.6 Sol) xhigh | — | The consistency-first lane. Opus 4.6 is the only model where `max` effort works without overthinking — months of field data, Active through at least Feb 2027, proven predictable. Lower benchmark scores than Sonnet 5 but higher consistency. Choose this when you've tuned prompts to Opus 4.6 behavior and reliability matters more than peak capability. @@ -38,9 +38,9 @@ The consistency-first lane. Opus 4.6 is the only model where `max` effort works | **Planner** | Opus 4.8 (via Plan Mode — Shift+Tab) | `xhigh` | | **Advisor** | Fable 5 or Opus 4.8 (via `advisorModel`) | — | | **Driver** | Sonnet 5 (auto execute mode) | `high` | -| **Reviewer** | Codex (GPT-5.5) xhigh | — | +| **Reviewer** | Codex (GPT-5.6 Sol) xhigh | — | -Cost-efficient hybrid using CC's native `opusplan` alias. Opus 4.8 reasons during Plan Mode, Sonnet 5 executes. Max-bundled — no API credit drain. Pin `model: "opusplan"` + `advisorModel: "fable"` in project settings. Sonnet 5 now uses 1M context natively (no `[1m]` suffix needed). GPT-5.5 xhigh is the cross-model reviewer. +Cost-efficient hybrid using CC's native `opusplan` alias. Opus 4.8 reasons during Plan Mode, Sonnet 5 executes. Max-bundled — no API credit drain. Pin `model: "opusplan"` + `advisorModel: "fable"` in project settings. Sonnet 5 now uses 1M context natively (no `[1m]` suffix needed). GPT-5.6 Sol xhigh is the cross-model reviewer. **Note:** Opus 4.6 cannot advise Sonnet 5 (rejected in the advisor pairing table). Use Fable 5 or Opus 4.8 as advisor for this lane. @@ -116,11 +116,13 @@ Setup D is for work where SDLC discipline overhead exceeds the value: ## Final Review Policy -**Setups A, B, and C end at GPT-5.5 xhigh as the cross-model reviewer.** Claude can't grade its own homework — the reviewer always belongs to a different lab with different blind spots. See [CLAUDE_CODE_SDLC_WIZARD.md → "Cross-Model Review (Codex)"](CLAUDE_CODE_SDLC_WIZARD.md) for the handoff protocol. +**Setups A, B, and C end at GPT-5.6 Sol xhigh as the cross-model reviewer.** Claude can't grade its own homework — the reviewer always belongs to a different lab with different blind spots. See [CLAUDE_CODE_SDLC_WIZARD.md → "Cross-Model Review (Codex)"](CLAUDE_CODE_SDLC_WIZARD.md) for the handoff protocol. **Setup D has no reviewer** — the blast radius doesn't justify it. If you're unsure whether a task is truly Lite, it probably isn't. Escalate. -If GPT-5.5 isn't available on your OpenAI account, Codex auto-falls back to GPT-5.4 — still keep `model_reasoning_effort="xhigh"`. Lower reasoning misses subtle bugs that the reviewer is the last gate to catch. +If GPT-5.6 Sol isn't available on your OpenAI account, Codex auto-falls back to Terra — still keep `model_reasoning_effort="xhigh"`. Lower reasoning misses subtle bugs that the reviewer is the last gate to catch. + +**Escalation for unusually risky PRs:** `xhigh` is the evidence-based default — OpenAI's own migration guidance is to preserve the prior effort baseline, and no published data shows `max` or Pro mode catching meaningfully more real bugs than `xhigh` on ordinary PR review. For a PR you'd genuinely lose sleep over (security-sensitive, high blast radius, touches the installer or a consumer-facing template), escalate the reviewer to `max` or Pro mode — a once-per-PR gate is exactly the kind of low-frequency, high-stakes call site where the extra cost is easiest to justify. Don't make it the default. ## Version Requirement @@ -159,11 +161,11 @@ Whichever path you use, the cross-model PR review gate still applies. - Drop to Setup A (Sonnet 5) for the remainder of the day — same discipline, far less quota burn - Or drop to Setup D for grunt work that doesn't need deep reasoning -- Or use Sonnet directly for the final mechanical edits, then run the GPT-5.5 reviewer over the whole diff at the end +- Or use Sonnet directly for the final mechanical edits, then run the GPT-5.6 Sol reviewer over the whole diff at the end **Setup D uses Sonnet** — same model as Setup A's driver, Max-bundled. One less model to manage if you're already on Setup A. -The reviewer (GPT-5.5 xhigh) is billed against your OpenAI account, separately. Watch both bills. +The reviewer (GPT-5.6 Sol xhigh) is billed against your OpenAI account, separately. Watch both bills. ## Autocompact Thresholds @@ -201,11 +203,11 @@ Credit allocations: Pro $20/mo, Max 5x $100/mo, Max 20x $200/mo. **No rollover.* ### What this means for the lanes -- **Setup A — Sonnet 5 + Fable advisor:** Sonnet 5's native 1M context — interactive session, Max-bundled, no `[1m]` suffix needed. Fable 5 advisor via `advisorModel: "fable"` — also Max-bundled. GPT-5.5 xhigh reviewer on ChatGPT subscription. Roughly 5x less Max quota consumed per turn than Setup B. -- **Setup B — Opus 4.6 Stability:** Opus 4.6 max driver on Max, 1M context included at standard rates (see above). Fable 5 advisor, Max-bundled. GPT-5.5 xhigh reviewer, separate. +- **Setup A — Sonnet 5 + Fable advisor:** Sonnet 5's native 1M context — interactive session, Max-bundled, no `[1m]` suffix needed. Fable 5 advisor via `advisorModel: "fable"` — also Max-bundled. GPT-5.6 Sol xhigh reviewer on ChatGPT subscription. Roughly 5x less Max quota consumed per turn than Setup B. +- **Setup B — Opus 4.6 Stability:** Opus 4.6 max driver on Max, 1M context included at standard rates (see above). Fable 5 advisor, Max-bundled. GPT-5.6 Sol xhigh reviewer, separate. - **Setup C — OpusPlan Hybrid:** **fully Max-bundled.** `opusplan` uses Opus (plan mode) + Sonnet (execute mode), both at their native context windows — no credit drain. - **⚠️ Avoid `sonnet[1m]` as a manual pin outside Setup A/C:** if your provider or gateway doesn't resolve Sonnet 5 to its native 1M automatically, forcing a `[1m]`-suffixed pin on an older Sonnet can draw from your usage credits pool ($3/$15 per Mtok) instead of your Max subscription. The `/model` picker shows this explicitly — watch for "Draws from usage credits." -- **Reviewer (GPT-5.5 xhigh) in all three triads:** billed against your OpenAI account, completely separate from Anthropic. +- **Reviewer (GPT-5.6 Sol xhigh) in all three triads:** billed against your OpenAI account, completely separate from Anthropic. - **CI loops that use `claude -p` post-June-15:** these now bill against the separate Anthropic credit pool, not your Max subscription. The wizard's CI shepherd loops (E2E scoring, weekly-update jobs) are local-only on the maintainer's machine and stay on Max; consumer-repo CI integrations may need to budget the new credit pool. ### Bottom line diff --git a/CLAUDE.md b/CLAUDE.md index 773b43d7..65a4d9e7 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -72,7 +72,7 @@ Key concepts: - PRs require 1 approving review and passing CI (`validate`) - E2E signal is advisory-only now, via `tests/e2e/local-shepherd.sh` run locally on maintainer's Max subscription (ROADMAP #212 Option 1) - Admin enforcement is on — no bypassing, even for repo owners -- **Fable is the primary reviewer/advisor** (via `advisor()` or Fable subagent fallback when advisor is down). Codex (GPT-5.5) xhigh is the **cross-model safety check** — default to running before committing and pushing. Skip only with logged justification (e.g., single-line typo fix). Incident 2026-06-09: 4 PRs shipped without cross-model check, all had issues +- **Fable is the primary reviewer/advisor** (via `advisor()` or Fable subagent fallback when advisor is down). Codex (GPT-5.6 Sol) xhigh is the **cross-model safety check** — default to running before committing and pushing. Skip only with logged justification (e.g., single-line typo fix). Incident 2026-06-09: 4 PRs shipped without cross-model check, all had issues ## Special Notes diff --git a/CLAUDE_CODE_SDLC_WIZARD.md b/CLAUDE_CODE_SDLC_WIZARD.md index 3bb72c2b..8a79923a 100644 --- a/CLAUDE_CODE_SDLC_WIZARD.md +++ b/CLAUDE_CODE_SDLC_WIZARD.md @@ -257,7 +257,7 @@ Claude Code's **effort level** controls how much thinking the model does before | Opus 4.8 (escalation only) | `xhigh` | `max` triggers excessive reasoning on 4.7/4.8 — documented 40-60x cache-token jump vs `high` (see "Opus 4.6 Stability" tier below) | | Fable 5 (advisor) | `high` (its own designed default) | Adaptive thinking always on; `xhigh`/`max` mainly move the thinking-token budget with small marginal gain for review work | | Opus 4.6 (Stability lane) | `max` | The one model where `max` doesn't overthink — no `xhigh` support at all (only low/medium/high/max) | -| OpenAI/Codex (cross-model reviewer) | `xhigh` (their highest) | Always — lower reasoning misses subtle bugs the reviewer exists to catch | +| OpenAI/Codex (cross-model reviewer) | `xhigh` default, escalate to `max`/Pro for unusually risky PRs | Lower reasoning misses subtle bugs the reviewer exists to catch; see `AI_SETUP_LANES.md`'s Final Review Policy for when to escalate | **Strict effort behavior (Opus 4.7+, carried forward in 4.8):** - **`xhigh` was introduced in 4.7** — sits between `high` and `max`, designed for coding and agentic work (30+ minute tasks with token budgets in the millions) @@ -1087,7 +1087,7 @@ This is **Setup C (OpusPlan Hybrid/Saver)** in `AI_SETUP_LANES.md`. CC's native |-------|--------------------|--------------------|----------------------| | Planner | Opus 4.8 `xhigh` (Plan Mode) | Sonnet 5 `high`→`xhigh` | Opus 4.6 `max` | | Driver | Sonnet 5 `high` (execute mode) | Sonnet 5 `high`→`xhigh` | Opus 4.6 `max` | -| Reviewer | GPT-5.5 xhigh | GPT-5.5 xhigh | GPT-5.5 xhigh | +| Reviewer | GPT-5.6 Sol xhigh | GPT-5.6 Sol xhigh | GPT-5.6 Sol xhigh | **How to opt in:** ```json @@ -1110,7 +1110,7 @@ Set effort per-session with `/effort` (planner `xhigh`, driver `high`) rather th **Prove-It Gate (#233 acceptance criterion):** mixed-mode ships only if pair-tested on 3+ simple repos shows Sonnet-coder + Opus-reviewer produces ≥ same SDLC scores as full-Opus baseline. The first version of the heuristic ships v1.38.0; pair-test results land in CHANGELOG before recommending mixed-mode as the default for any tier. **Tradeoffs (be honest):** -- The Sonnet driver will drop some fine-grained self-review moves compared to an Opus-coder run — it's fast, less deliberate. The Opus planner and GPT-5.5 reviewer catch them, but expect more "fix in round 2" cycles. +- The Sonnet driver will drop some fine-grained self-review moves compared to an Opus-coder run — it's fast, less deliberate. The Opus planner and GPT-5.6 Sol reviewer catch them, but expect more "fix in round 2" cycles. - Mixed-mode disables auto-mode (same as any pinned model). The pin is per-session — to switch back, remove the `model` line. ### Latest tier — Opus 4.8 (escalation model, #395) @@ -3852,12 +3852,12 @@ Use an independent AI model from a different company as a code reviewer. The aut **Why this works:** Two AI systems from different companies (e.g., Claude writes, GPT reviews) provide adversarial diversity. They have fundamentally different training, different failure modes, and different strengths. What one misses, the other catches. -**Use the best model at the deepest reasoning.** This is your quality gate — don't economize on it. Always use the latest, most capable model available (**GPT-5.5 if you have access**, otherwise GPT-5.4) at maximum reasoning effort (`xhigh` — this is non-negotiable, lower settings miss subtle errors). Cheaper/faster models miss things. The whole point is catching what the authoring model couldn't. +**Use the best model at the deepest reasoning.** This is your quality gate — don't economize on it. Always use the latest, most capable model available (**GPT-5.6 Sol if you have access**, otherwise Terra) at `xhigh` reasoning effort (non-negotiable — lower settings miss subtle errors; preserve this baseline rather than downgrading it on a model bump). Cheaper/faster models miss things. The whole point is catching what the authoring model couldn't. For unusually risky PRs, escalating to `max` or Pro mode is defensible — see `AI_SETUP_LANES.md`'s Final Review Policy — but `xhigh` is the default; no published data shows the higher tiers catching meaningfully more real bugs on ordinary PR review. **Prerequisites:** - Codex CLI installed: `npm i -g @openai/codex` - OpenAI API key configured: `export OPENAI_API_KEY=...` -- Codex CLI picks up your OpenAI account's best available model automatically. If you have GPT-5.5 access, `codex exec` uses it; otherwise it falls back to GPT-5.4. No config change needed on your side. +- Codex CLI picks up your OpenAI account's best available model automatically. If you have GPT-5.6 Sol access, `codex exec` uses it; otherwise it falls back to Terra. No config change needed on your side. - This is a local workflow tool — not required for CI/CD **The Protocol:** diff --git a/README.md b/README.md index 58a429b1..c1b1f04d 100644 --- a/README.md +++ b/README.md @@ -123,7 +123,7 @@ export OPENAI_API_KEY=sk-... codex --version # confirm ready ``` -That's it. Codex picks up your OpenAI account's best available model automatically — **if you have GPT-5.5, it uses GPT-5.5; otherwise GPT-5.4**. No model config needed. +That's it. Codex picks up your OpenAI account's best available model automatically — **if you have GPT-5.6 Sol, it uses Sol; otherwise it falls back to Terra**. No model config needed. **How to use it:** after Claude's self-review passes, write a one-file mission brief and run: @@ -175,7 +175,7 @@ Or pin in `.claude/settings.json`: { "model": "sonnet", "advisorModel": "fable", "effortLevel": "xhigh" } ``` -**Effort is model-aware, not blanket `max`.** Sonnet 5: `high` default, `/effort xhigh` for hard tasks. Opus 4.8: `xhigh` (its own `max` overthinks). Opus 4.6: `max` (its one `xhigh`-less sweet spot). Set per-session with `/effort`, not a shell-rc or settings env var — persisting effort that way silently overrides a later `/effort` change after you switch models (see `SDLC.md`'s Lessons Learned for a real incident this caused). OpenAI/Codex reviewer: `xhigh` (its highest). +**Effort is model-aware, not blanket `max`.** Sonnet 5: `high` default, `/effort xhigh` for hard tasks. Opus 4.8: `xhigh` (its own `max` overthinks). Opus 4.6: `max` (its one `xhigh`-less sweet spot). Set per-session with `/effort`, not a shell-rc or settings env var — persisting effort that way silently overrides a later `/effort` change after you switch models (see `SDLC.md`'s Lessons Learned for a real incident this caused). OpenAI/Codex reviewer: `xhigh` default — escalate to `max`/Pro mode only for unusually risky PRs (see `AI_SETUP_LANES.md`'s Final Review Policy). ### Four Setup Lanes @@ -183,9 +183,9 @@ The wizard defines four AI coding setups in [`AI_SETUP_LANES.md`](AI_SETUP_LANES | Lane | Advisor | Driver | Reviewer | Escalation | |------|---------|--------|----------|------------| -| **A — Recommended** | Fable 5 (advisorModel) | Sonnet 5, `high`→`xhigh` | GPT-5.5 xhigh | Opus 4.8 xhigh or Fable review | -| **B — Stability** | Fable 5 (advisorModel) | Opus 4.6 max | GPT-5.5 xhigh | None | -| **C — Saver** | Fable 5 or Opus 4.8 (advisorModel) | Opus 4.8 plans, Sonnet 5 executes | GPT-5.5 xhigh | None | +| **A — Recommended** | Fable 5 (advisorModel) | Sonnet 5, `high`→`xhigh` | GPT-5.6 Sol xhigh | Opus 4.8 xhigh or Fable review | +| **B — Stability** | Fable 5 (advisorModel) | Opus 4.6 max | GPT-5.6 Sol xhigh | None | +| **C — Saver** | Fable 5 or Opus 4.8 (advisorModel) | Opus 4.8 plans, Sonnet 5 executes | GPT-5.6 Sol xhigh | None | | **D — Lite** | None | Sonnet 5, `medium` | None | None | Setup D's whole point: **the discipline of knowing when NOT to use discipline.** When blast radius is low and you just need fast cheap hands, skip the SDLC overhead. diff --git a/ROADMAP.md b/ROADMAP.md index 7e840ab9..483709a9 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -99,5 +99,6 @@ Living tracker of projects shipped using this wizard. **Rule:** only list projec | 436 | Fable self-enforcement audit — codex-gate-check.sh + tdd-pretool-check.sh never actually blocked | **Maintainer pain event 2026-07-04:** "I hate having to remind 'don't forget to cross-model review with codex'." Root-cause audit found two enforcement hooks that printed the right warning text but always `exit 0` — including on their "REQUIRED"/blocking branches. Claude Code only denies a `PreToolUse` call on `exit 2` + stderr; `exit 0` always allows regardless of stdout content. Both hooks had shipped this way since introduction, silently decorative the whole time. **Scope shipped:** (a) `codex-gate-check.sh` — the two "CROSS-MODEL REVIEW REQUIRED" branches now `exit 2`; (b) `tdd-pretool-check.sh` — new edit-ordering gate blocks (`exit 2`) writes to `src/**` unless a test file was touched earlier in the session (proxy for TDD RED — a bash hook can't run the suite to confirm a real failing test, this is the best mechanically-available approximation); (c) new `Stop` hook (`codex-review-stop-check.sh`) closes a third gap — a session can end with `git commit` never invoked, so the Bash-triggered gate never fires; non-blocking (Stop can't deny), warns once per session on significant uncommitted changes with no REVIEWED/CERTIFIED handoff; (d) fixed 3 pre-existing distribution-parity bugs surfaced by a new stronger test (`test_hooks_json_script_parity`, compares actual registered scripts per hook event, not just event-name presence) — `codex-gate-check.sh` was never wired into the plugin's `hooks/hooks.json` (plugin users had zero cross-model gate), `goal-confidence-check.sh` was missing from the project's own `.claude/settings.json`, `token-spike-check.sh` was missing from the CLI template and from `cli/init.js`'s FILES array entirely (never actually copied to consumer repos by `npx ... init`). **Entry gate:** maintainer pain event — the exact mechanism this whole wizard exists to provide (mechanical enforcement instead of verbal reminders) was non-functional. **Verified:** 22 test files run clean post-fix (185 hook tests run twice for isolation, plus doc-consistency/cowork-drift/cli/plugin/audit-session-load/tdd-fires-once and 15 more referencing the changed files). | | 437 | Codex-gate handoff staleness — `.reviews/handoff.json` status isn't checked for freshness or branch match | **Found while shipping #436, 2026-07-04:** `codex-gate-check.sh` only checks `.reviews/handoff.json`'s `status` field (`CERTIFIED`/`REVIEWED` → allow commit). It does not check `review_id`, `branch`, `requested_at` staleness, or that `files_changed` overlaps the actually-staged diff. Live evidence: a week-old `REVIEWED` handoff (`review_id: "drift-baseline-fix-001"`, dated 2026-06-28, for an already-merged and unrelated PR #431) was still sitting in `.reviews/` and would have silently satisfied the gate for #436's entirely different, never-reviewed changes had it not been manually overwritten before committing. **Independently re-confirmed 2026-07-05** via the v1.84.0 post-ship retrospective (Fable + Codex dual audit): 2 real commits landed after round-11 CERTIFIED and both sailed through the gate on the same stale certification — proving the gap live a second time, on this exact hook. **Design settled (Fable-reviewed 2026-07-05), simpler than the branch/files_changed idea above:** certification writes `commit_sha` (HEAD at cert time) into `handoff.json`; the gate compares current `git rev-parse HEAD` to that SHA — mismatch (or missing `commit_sha`, e.g. old-format handoff files) = stale, exit 2. This directly answers the open design question below: it allows exactly one commit after certification (HEAD still equals the recorded SHA at that commit's `PreToolUse` check) and blocks the next one until re-cert, rather than needing a branch/superset-diff heuristic. `CODEX_GATE_SKIP=1` remains the logged-justification escape hatch for legitimate small follow-ups. **Original open design question (superseded by the above):** how to avoid false-positiving on legitimate small follow-up commits after a review already covered the substance (e.g. a one-line typo fix Codex itself suggested) — a naive "any new diff invalidates the handoff" rule would defeat the recheck/dialogue loop the protocol already relies on. **Entry gate:** maintainer pain event — demonstrated live twice now, not theoretical. Not fixed in #436 (separate, non-trivial concern; scope discipline). **Status: ✅ DONE 2026-07-05** (v1.86.0 — Pass 2 of the v1.84.0 retrospective fix plan, standalone TDD PR, 2 new tests + 2 existing tests updated to real git-repo fixtures, 191/191 hook tests green). | | 438 | Cross-repo Fable audit sweep — track and follow up on the 5 GH issues filed across the ecosystem | **Maintainer call-out 2026-07-04:** "use fable to review/upgrade these repos while it's free... reduce bloat and fix and review all that stuff," extended across every repo that matters, not just this one. Fable-led review/upgrade/bloat-reduction issues were filed on 5 repos as a time-boxed sweep before the July 1-7 free-inclusion window closes: [claude-gdlc-wizard#14](https://github.com/BaseInfinity/claude-gdlc-wizard/issues/14), [claude-rdlc-wizard#9](https://github.com/BaseInfinity/claude-rdlc-wizard/issues/9), [audit-agent#1](https://github.com/BaseInfinity/audit-agent/issues/1) (ADLC), [anticheat#1](https://github.com/BaseInfinity/anticheat/issues/1), [pdlc#6](https://github.com/BaseInfinity/pdlc/issues/6). This repo's own equivalent is tracked separately as #236 (bloat-hunt, re-triggered 2026-07-04 with the same Fable-window urgency) — #438 is specifically the cross-repo tracking/follow-up, since those 5 issues live in repos this ROADMAP doesn't otherwise cover. **Scope:** (a) follow up on each filed issue once picked up — verify Fable was actually used (not silently misrouted to another model, per the `CLAUDE_CODE_SUBAGENT_MODEL` bug found the same night, see #436 commit history) via the subagent self-identifying its model in its report; (b) after the July 7 window, note in each issue whether it was completed in time or needs re-scoping for metered Fable; (c) if a genuine pattern/finding recurs across 3+ of these repos (e.g. the same stale-doc-vs-lessons-learned contradiction found in this repo's own SDLC.md), consider whether it's worth a wizard-level fix (a hook/test/template change here) rather than 5 separate one-off fixes; (d) **maintainer add 2026-07-04:** extend the audit to measure token efficiency and ROI per repo — for each swept repo, capture roughly how many tokens the Fable pass spent (`/usage` or session cost) against what it actually fixed/found (bugs caught, bloat removed, doc drift corrected), so the sweep reports back "was this worth it" per repo, not just "did it run." Design not yet started — needs a lightweight before/after measure (e.g. char-count or token-bloat-audit delta plus a qualitative finding count) that doesn't require its own new tooling per repo. **Entry gate:** maintainer pain event — issues filed but nothing tracks whether the cross-repo sweep actually happened or paid off. **Recurring pattern confirmed 2026-07-05** (this is the #438(c) trigger): the v1.84.0 post-ship retrospective independently re-found the exact stale-doc-vs-lessons-learned pattern #438(c) called out by name (see #437's staleness gap, and the CI-Feedback-Loop/Convergence gaps fixed in `CLAUDE_CODE_SDLC_WIZARD.md` the same day) — a wizard-level fix (this repo's own doc + hook) is underway rather than 5 one-off fixes. **Maintainer intent 2026-07-05:** once this repo's fixes land, maintainer will run the Fable-led upgrade on each sibling repo (gdlc, rdlc, etc.) pointed at this repo's now-updated guidelines, so the siblings pick up similar patterns/discipline rather than each independently reinventing them — the 5 filed issues are the vehicle for that, not just a bloat-reduction pass. | +| 439 | GPT-5.6 family (Sol/Terra/Luna) supersedes GPT-5.5 — reviewer-model doc sweep | **Found 2026-07-10:** OpenAI shipped a new flagship family — `gpt-5.6-sol` (flagship, "strongest for complex coding, computer use, research, cybersecurity"), `gpt-5.6-terra` (balanced, "competitive with GPT-5.5 at lower cost"), `gpt-5.6-luna` (high-volume/cheap) — verified against primary OpenAI docs (`developers.openai.com/api/docs/guides/latest-model`, `learn.chatgpt.com/docs/models`, `developers.openai.com/api/docs/pricing`), not inferred. This repo names GPT-5.5 as the cross-model reviewer in 6 live-guidance files inherited by every consumer repo — all updated to GPT-5.6 Sol: `AI_SETUP_LANES.md`, `README.md`, `CLAUDE_CODE_SDLC_WIZARD.md`, `skills/sdlc/SKILL.md` + `cowork/skills/sdlc/SKILL.md` (both copies, moved together per the doc-duplication-drift lesson from v1.85.0), `AGENTS.md`, and this repo's own `CLAUDE.md`. **Explicitly untouched (historical record, not live guidance):** `CHANGELOG.md`, `ROADMAP_ARCHIVE.md`, `ROADMAP.md`'s #302 citation, README.md's Andon Labs vending-bench citation (L151, names the model an actual past eval ran on), and this file's own L113 E2E-audit citation — rewriting any of these would falsify what model was actually used at the time; regression tests explicitly guard both untouched citations. **Design decision — auto-detect preserved, no script/hook changes:** a Fable-subagent scoping pass recommended pinning an explicit `-c model=gpt-5.6-sol` in `scripts/codex-review-with-progress.sh` and `CI_CD.md`'s invocation. Direct file reads found this was already a deliberate, documented design (self-updating without maintenance at each model bump — proven live: the CLI's own default already resolved to Sol with zero config change) independently stated in 3 places; pinning would trade that property away for marginal audit-trail precision. Kept auto-detect; this is a pure prose fix, not a behavior change. **Reasoning-tier question resolved:** user asked whether an "ultra" tier exists above `max` for the final review gate. Sourced research (run against Sol itself once the local Codex CLI was upgraded past its `gpt-5.6-sol`-incompatible v0.142.3) confirmed no — the effort axis is still `none/low/medium/high/xhigh/max`; Codex's "Ultra" is a separate multi-agent orchestration mode, not deeper single-model reasoning. No OpenAI-published data compares `xhigh` vs `max` for PR-style review specifically. Kept `xhigh` as the default (preserves the prior baseline per OpenAI's own migration guidance), documented Pro/`max` as an escalation option for unusually risky PRs in `AI_SETUP_LANES.md`'s Final Review Policy, mirroring the existing Sonnet-5→Opus-4.8 escalation pattern. **Incident during verification:** the pre-edit full-suite baseline run corrupted `README.md` and `tests/e2e/score-history.jsonl` — `tests/test-persist-score-history.sh`'s git-fixture setup silently fell back to running real git operations (bogus `"initial"`/`"seed history"` commits, a `feature/x` branch) against the actual repo when its tmpdir creation failed under a sandboxed shell, rather than hard-failing. Root-caused via `git reflog`, both files restored from the known-good commit, garbage branches removed (explicit user confirmation obtained before any force-delete), `main`/`origin/main` were never touched. Flagged as a separate follow-up bug in `test-persist-score-history.sh` (needs to hard-fail on tmpdir-creation failure instead of silently using cwd) — not fixed here, out of scope; the same test independently reproduced a second symptom of this fragility during this task's own final regression sweep (its fixture push got blocked by the afterhours hook, unrelated to model-name content — confirmed no repo damage this time). **Second gap caught before review:** re-grepping all 7 target files from scratch (rather than trusting prior session state) found `CLAUDE.md:75` still said "GPT-5.5" — dropped during the corruption/recovery above and never redone. Caught RED via its own regression test, fixed, GREEN. **Fable review (fresh instance):** clean — independently re-grepped every spelling of "5.5"/"5.4" repo-wide (including files outside the original scope list, e.g. `RESEARCH_58_CLAW_OMO_OMX.md`) and confirmed every remaining hit is a genuine historical citation. Found and fixed one real nit: a stale comment claiming the test helper used a bash nameref (it doesn't — bash 3.x/macOS has none; it echoes for command-substitution capture). **Codex xhigh (Sol) cross-model review — 3 rounds to CERTIFIED, not rubber-stamped:** round 1 found 2 P1s — (a) the new escalation-to-`max`/Pro paragraph in `AI_SETUP_LANES.md` made 2 pre-existing lines (`README.md:178`, `CLAUDE_CODE_SDLC_WIZARD.md:260`) stale, since both still called `xhigh` "the highest" tier; fixed both to describe `xhigh` as the default with escalation, outside the original 6-file scope but directly caused by this diff; (b) the 8 new regression tests only checked for the `5.6` substring, so they passed under a demonstrated Sol→Terra (primary reviewer) or Terra→Luna (fallback target) swap — fixed by requiring `Sol`/`Terra` by name, mutation-verified against the reviewer's own 3 example mutations. Round 2 found a further gap in that same fix: requiring `Sol` alone (without also requiring `5.6`) still passed a demonstrated GPT-5.6 Sol → GPT-5.7 Sol mutation — fixed by refactoring the test helper to require a comma-separated list of ALL of `5.6` + `Sol`/`Terra` together per line, re-verified against all 4 mutations (the 3 from round 1 plus the new one). Round 3: CERTIFIED, no further findings. **Entry gate:** this repo is the canonical source other wizard siblings inherit "GPT-5.5 xhigh reviewer" guidance from — stale model guidance here propagates everywhere. **Verified:** 8 new per-location regression tests in `tests/test-doc-consistency.sh` (RED confirmed against pre-edit content, GREEN after; mutation-tested against 4 distinct swap scenarios, not just 2), full test suite before/after comparison, Fable review, 3-round Codex xhigh (Sol) cross-model review to CERTIFIED. | _67 fully-resolved rows archived → [ROADMAP_ARCHIVE.md](ROADMAP_ARCHIVE.md), including #231 (shepherd migration, the single largest row in the pre-cut file)._ diff --git a/cowork/skills/sdlc/SKILL.md b/cowork/skills/sdlc/SKILL.md index a8071a72..af93cb87 100644 --- a/cowork/skills/sdlc/SKILL.md +++ b/cowork/skills/sdlc/SKILL.md @@ -137,7 +137,7 @@ The loop goes back to PLANNING, not TDD RED. Run `/code-review`; issues at confi ## Cross-Model Review (REQUIRED for High-Stakes) -**When to run:** high-stakes changes (auth, payments, data), releases/publishes, complex refactors. **When to skip (log justification):** trivial, hotfixes, risk < review cost. **Prerequisites:** Codex CLI + OpenAI API key. **Reviewer:** `gpt-5.5` xhigh — adversarial diversity. +**When to run:** high-stakes changes (auth, payments, data), releases/publishes, complex refactors. **When to skip (log justification):** trivial, hotfixes, risk < review cost. **Prerequisites:** Codex CLI + OpenAI API key. **Reviewer:** `gpt-5.6-sol` xhigh — adversarial diversity. PROTOCOL is universal across domains; only `review_instructions` and `verification_checklist` change. diff --git a/skills/sdlc/SKILL.md b/skills/sdlc/SKILL.md index a8071a72..af93cb87 100644 --- a/skills/sdlc/SKILL.md +++ b/skills/sdlc/SKILL.md @@ -137,7 +137,7 @@ The loop goes back to PLANNING, not TDD RED. Run `/code-review`; issues at confi ## Cross-Model Review (REQUIRED for High-Stakes) -**When to run:** high-stakes changes (auth, payments, data), releases/publishes, complex refactors. **When to skip (log justification):** trivial, hotfixes, risk < review cost. **Prerequisites:** Codex CLI + OpenAI API key. **Reviewer:** `gpt-5.5` xhigh — adversarial diversity. +**When to run:** high-stakes changes (auth, payments, data), releases/publishes, complex refactors. **When to skip (log justification):** trivial, hotfixes, risk < review cost. **Prerequisites:** Codex CLI + OpenAI API key. **Reviewer:** `gpt-5.6-sol` xhigh — adversarial diversity. PROTOCOL is universal across domains; only `review_instructions` and `verification_checklist` change. diff --git a/tests/test-doc-consistency.sh b/tests/test-doc-consistency.sh index 868279b8..2667fa34 100755 --- a/tests/test-doc-consistency.sh +++ b/tests/test-doc-consistency.sh @@ -1211,6 +1211,164 @@ test_roadmap_no_duplicate_row_numbers_across_files test_roadmap_key_open_items_present test_roadmap_archive_has_key_archived_items +# ──────────────────────────────────────────── +# ROADMAP #439: GPT-5.6 (Sol) supersedes GPT-5.5 as cross-model reviewer +# Per-location checks, not whole-file counts (per #437's CHANGELOG lesson: +# aggregate checks have slack that lets one unfixed line hide behind an +# unrelated match elsewhere in the same file). Each line must both GAIN +# "5.6" and LOSE "5.5" (and "5.4" on fallback-chain lines) — a half-applied +# edit (mentions 5.6 nearby but leaves the old 5.5/5.4 name in place) fails. +# ──────────────────────────────────────────── + +# Checks one line of a file both contains a required substring and does NOT +# contain any of the forbidden substrings. Echoes a descriptive failure (or +# nothing on success) for the caller to collect via command substitution, +# rather than calling fail() directly, so one test function can report every +# stale line in one message. (bash 3.x on macOS has no namerefs.) +_check_line_has_and_lacks() { + local file="$1" line_num="$2" must_have_csv="$3" + shift 3 + local content + content="$(sed -n "${line_num}p" "$file")" + if [ -z "$content" ]; then + echo "${file}:${line_num}(line-missing)" + return + fi + local required + for required in ${must_have_csv//,/ }; do + if ! printf '%s' "$content" | grep -qi "$required"; then + echo "${file}:${line_num}(missing '$required')" + return + fi + done + for forbidden in "$@"; do + if printf '%s' "$content" | grep -qi "$forbidden"; then + echo "${file}:${line_num}(stale '$forbidden')" + return + fi + done +} + +test_ai_setup_lanes_reviewer_is_gpt56() { + local F="$REPO_ROOT/AI_SETUP_LANES.md" + if [ ! -f "$F" ]; then fail "AI_SETUP_LANES.md not found"; return; fi + local bad="" + # "5\.6" AND "Sol" (not just one or the other) so a Sol->Terra swap, or a + # future GPT-5.7 Sol rename, both fail. + for n in 13 16 28 41 43 119 164 168 206 207 210; do + bad="$bad$(_check_line_has_and_lacks "$F" "$n" "5\.6,Sol" "5\.5")" + done + # L123 is the fallback-chain line: must name "5\.6" AND BOTH Sol (primary) + # and Terra (fallback target) so a Terra->Luna swap also fails. + bad="$bad$(_check_line_has_and_lacks "$F" 123 "5\.6,Sol,Terra" "5\.5" "5\.4")" + if [ -z "$bad" ]; then + pass "AI_SETUP_LANES.md: all reviewer-model lines reference GPT-5.6 Sol/Terra, none reference stale GPT-5.5/5.4" + else + fail "AI_SETUP_LANES.md stale reviewer-model reference(s):$bad" + fi +} + +test_readme_reviewer_is_gpt56() { + local F="$REPO_ROOT/README.md" + if [ ! -f "$F" ]; then fail "README.md not found"; return; fi + local bad="" + # L126 is the fallback-chain line: must name "5\.6" AND BOTH Sol and Terra. + bad="$bad$(_check_line_has_and_lacks "$F" 126 "5\.6,Sol,Terra" "5\.5" "5\.4")" + for n in 186 187 188; do + bad="$bad$(_check_line_has_and_lacks "$F" "$n" "5\.6,Sol" "5\.5")" + done + if [ -z "$bad" ]; then + pass "README.md: all reviewer-model lines reference GPT-5.6 Sol/Terra, none reference stale GPT-5.5/5.4" + else + fail "README.md stale reviewer-model reference(s):$bad" + fi +} + +# L151 is a historical eval citation (Andon Labs Vending-Bench actually used +# GPT-5.5 at the time) — must NOT be rewritten, or the citation becomes +# factually wrong about what model that benchmark run used. +test_readme_vending_bench_citation_untouched() { + local F="$REPO_ROOT/README.md" + if [ ! -f "$F" ]; then fail "README.md not found"; return; fi + if sed -n '151p' "$F" | grep -q "GPT-5\.5"; then + pass "README.md L151 vending-bench citation still names GPT-5.5 (historical, untouched)" + else + fail "README.md L151 vending-bench citation no longer names GPT-5.5 — historical citation was rewritten" + fi +} + +test_wizard_doc_reviewer_is_gpt56() { + local F="$REPO_ROOT/CLAUDE_CODE_SDLC_WIZARD.md" + if [ ! -f "$F" ]; then fail "CLAUDE_CODE_SDLC_WIZARD.md not found"; return; fi + local bad="" + bad="$bad$(_check_line_has_and_lacks "$F" 1090 "5\.6,Sol" "5\.5")" + bad="$bad$(_check_line_has_and_lacks "$F" 1113 "5\.6,Sol" "5\.5")" + # L3855 and L3860 are fallback-chain lines: must name "5\.6" AND BOTH Sol + # and Terra. + bad="$bad$(_check_line_has_and_lacks "$F" 3855 "5\.6,Sol,Terra" "5\.5" "5\.4")" + bad="$bad$(_check_line_has_and_lacks "$F" 3860 "5\.6,Sol,Terra" "5\.5" "5\.4")" + if [ -z "$bad" ]; then + pass "CLAUDE_CODE_SDLC_WIZARD.md: all reviewer-model lines reference GPT-5.6 Sol/Terra, none reference stale GPT-5.5/5.4" + else + fail "CLAUDE_CODE_SDLC_WIZARD.md stale reviewer-model reference(s):$bad" + fi +} + +# L113 is a historical audit citation (the E2E benchmark critique actually +# ran on GPT-5.4 at the time) — must NOT be rewritten. +test_wizard_doc_e2e_audit_citation_untouched() { + local F="$REPO_ROOT/CLAUDE_CODE_SDLC_WIZARD.md" + if [ ! -f "$F" ]; then fail "CLAUDE_CODE_SDLC_WIZARD.md not found"; return; fi + if sed -n '113p' "$F" | grep -q "GPT-5\.4"; then + pass "CLAUDE_CODE_SDLC_WIZARD.md L113 E2E-audit citation still names GPT-5.4 (historical, untouched)" + else + fail "CLAUDE_CODE_SDLC_WIZARD.md L113 E2E-audit citation no longer names GPT-5.4 — historical citation was rewritten" + fi +} + +# Both copies (canonical + cowork) must move together — this is the exact +# doc-duplication-drift risk documented in project memory (v1.85.0: a +# protocol documented twice, fixed in only one copy). +test_skill_files_reviewer_is_gpt56() { + local bad="" + bad="$bad$(_check_line_has_and_lacks "$REPO_ROOT/skills/sdlc/SKILL.md" 140 "5\.6,sol" "5\.5")" + bad="$bad$(_check_line_has_and_lacks "$REPO_ROOT/cowork/skills/sdlc/SKILL.md" 140 "5\.6,sol" "5\.5")" + if [ -z "$bad" ]; then + pass "skills/sdlc/SKILL.md and cowork/skills/sdlc/SKILL.md both reference GPT-5.6 Sol reviewer" + else + fail "skill file(s) stale reviewer-model reference(s):$bad" + fi +} + +test_agents_md_reviewer_is_gpt56() { + local bad="" + bad="$bad$(_check_line_has_and_lacks "$REPO_ROOT/AGENTS.md" 16 "5\.6,Sol" "5\.5")" + if [ -z "$bad" ]; then + pass "AGENTS.md: lane summary references GPT-5.6 Sol reviewer" + else + fail "AGENTS.md stale reviewer-model reference(s):$bad" + fi +} + +test_claude_md_reviewer_is_gpt56() { + local bad="" + bad="$bad$(_check_line_has_and_lacks "$REPO_ROOT/CLAUDE.md" 75 "5\.6,Sol" "5\.5")" + if [ -z "$bad" ]; then + pass "CLAUDE.md: cross-model safety check references GPT-5.6 Sol reviewer" + else + fail "CLAUDE.md stale reviewer-model reference(s):$bad" + fi +} + +test_ai_setup_lanes_reviewer_is_gpt56 +test_readme_reviewer_is_gpt56 +test_readme_vending_bench_citation_untouched +test_wizard_doc_reviewer_is_gpt56 +test_wizard_doc_e2e_audit_citation_untouched +test_skill_files_reviewer_is_gpt56 +test_agents_md_reviewer_is_gpt56 +test_claude_md_reviewer_is_gpt56 + # ──────────────────────────────────────────── # Summary # ────────────────────────────────────────────