From 87ed40977a4021bcd4f95e93b2f45881630c1a51 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Mon, 3 Aug 2026 22:51:46 +0900 Subject: [PATCH 001/163] docs(integrations): test the native-restore thesis against the code, not the prior plan ocx restore already restores native Codex WITHOUT stopping the proxy (src/cli/help.ts:18), and POST /api/stop restores before it drains (management-api.ts:181). So the Codex half of this unit needs no durable operation-state engine; both directions already exist as CLI verbs. Records the three real asymmetries (enable is syncModelsToCodex not bare inject; a post-injection root model selection is destroyed; resume history is reversible but not byte-identical), and the defect that matters for the GUI: restoreNativeCodex() collapses a structured history failure into a message string while keeping success:true, so a card trusting that boolean would report a clean disable while routed threads stay hidden. Names the hazard that replaces the rollback engine: no persisted per-client desired state, so startup/ensure/sync/api-sync can silently re-inject. --- .../001_native_restore_thesis.md | 144 ++++++++++++++++++ 1 file changed, 144 insertions(+) create mode 100644 devlog/_plan/260803_codex_desktop_toggle/001_native_restore_thesis.md diff --git a/devlog/_plan/260803_codex_desktop_toggle/001_native_restore_thesis.md b/devlog/_plan/260803_codex_desktop_toggle/001_native_restore_thesis.md new file mode 100644 index 000000000..50d0a87f2 --- /dev/null +++ b/devlog/_plan/260803_codex_desktop_toggle/001_native_restore_thesis.md @@ -0,0 +1,144 @@ +# The native-restore thesis, tested against the code + +Research doc. No diffs here — the implementation designs live in the decade +docs. This one records what the code actually does, because the previous +unit's plan was built on a claim the code does not support. + +## The claim under test + +`000_plan.md` asserts that Codex and Claude Desktop both need a durable +operation-state engine — a versioned discriminated journal entry, prepare/commit +with restart reconciliation, and a field-scoped config writer — before either +can get a toggle. The owner's counter-thesis: the proxy keeps RUNNING while a +client is switched back to its native path, so each client only needs to be +returned to a path that already exists, not replayed from a recorded snapshot. + +For Codex the counter-thesis is correct, and the evidence is not subtle. + +## Codex: the toggle pair already ships as a CLI + +`ocx restore` is documented in `src/cli/help.ts:18-20` as: + +> Restore native Codex config without stopping the proxy; `restore back` +> re-points codex at the running proxy. + +That is the toggle, both directions, with the proxy up. `src/cli/index.ts:745` +calls `restoreNativeCodex()` with no lifecycle operation anywhere near it, and +`src/cli/index.ts:757` implements the enable direction as `syncModelsToCodex(live.port)` +against a proxy it first proves is live via `findLiveProxy()`. + +Stronger still: `POST /api/stop` (`src/server/management-api.ts:181`) calls +`restoreNativeCodex()` FIRST and only then schedules the drain and exit. The +restore therefore already executes while the listener is serving. Whether the +proxy later stops is irrelevant to the restore itself. + +The service-stop path does verify the listener is gone before restoring +(`src/service.ts:2571`), but that check enforces the requested outcome "service +stopped" so success is not claimed while a supervisor respawns the process. It +is not a precondition inside `restoreNativeCodex()`. + +**Conclusion:** no durable operation-state engine is required to make Codex +switchable. The disable path exists, the enable path exists, and both are +proxy-agnostic. + +## What restore actually touches, and where it is not symmetric + +Four state groups, when no external `model_provider` owns the config: + +| Group | Restore behavior | Evidence | +|---|---|---| +| `config.toml` + `opencodex.config.toml` | Byte-exact from the journal when the injected hash still matches; otherwise strip owned fragments | `src/codex/journal.ts:109`, `src/codex/inject.ts:770` | +| Injection journal | Deleted on a complete restore, retained on a partial one | `src/codex/journal.ts:133` | +| Model catalog | Pristine backup + post-sync native additions, or drop slash-qualified routed rows keeping native ones | `src/codex/catalog/sync.ts:572-590` | +| Resume history | May update `state_5.sqlite`, patch/append rollout JSONL, consume the backup manifest | `src/codex/history-provider.ts:413,656,691` | + +When an external provider such as `custom` owns `model_provider`, restore removes +only the stale journal and deliberately leaves everything else alone +(`src/codex/inject.ts:765`). That is a pre-existing courtesy to a user who moved +off us by hand, and the toggle must preserve it rather than "fixing" it. + +Three asymmetries matter for the toggle's honesty: + +1. **Enable is `syncModelsToCodex()`, not `injectCodexConfig()`.** Injection + selects and writes a catalog path but does not build the routed rows; + `src/codex/sync.ts:83-110` refreshes the catalog and then injects, and that + is what `ocx restore back` uses. A toggle wired to bare injection would + re-point Codex at a catalog that no longer lists the routed models. +2. **A post-injection root model selection is destroyed, not restored.** If the + user edited config after we injected, the hash no longer matches, the + fallback strip runs, and a root `model = "provider/slug"` line is removed + (`src/codex/inject.ts:315,700`). Re-enabling has no record of that selection. + The dialog copy must not promise to put it back. +3. **Resume history is reversible but not byte-identical.** Restore patches line + one when safe and appends a `session_meta` + (`src/codex/history-provider.ts:81,444`); re-enabling appends another + provider change rather than deleting that history. + +## The history lock, and why the current return shape is not enough + +The write path mirrors Codex's five-second SQLite busy timeout and retries twice +with a 500 ms delay (`src/codex/history-provider.ts:25,526`). A Codex app or IDE +holding the WAL writer lock is exactly what makes it fail. + +At the low-level boundary the failure is structured: + +```ts +return withHistoryRetry(...) ?? { rows: 0, files: 0, failed: true }; +``` + +Recoverable busy/lock/permission failures return `failed: true`; corruption and +programming errors throw (`src/codex/history-provider.ts:511,536,577`). + +But `restoreNativeCodex()` discards that structure. `history.failed` only edits +the message string, and `success` stays `cfg.success` +(`src/codex/inject.ts:787,794`). A config restore that succeeded while the +history stayed locked returns roughly: + +``` +{ success: true, message: "... history could NOT be restored ..." } +``` + +So a GUI that trusts `success` reports a clean disable while routed threads stay +tagged `opencodex` and remain invisible in the native app. And `failed: true` +itself conflates lock contention with `EPERM`/`EACCES`. + +**Design consequence for `020`:** the toggle route must not consume +`restoreNativeCodex()`'s boolean. It needs the structured per-artifact outcome — +config, catalog, history — with the history failure carrying a classified reason, +so the card can render "disabled, but your routed threads are still hidden +because the Codex app is holding the history database" instead of a green check +or a raw 500. Parsing the message string is not acceptable. + +## The real hazard is not the socket + +The sibling explorer's central finding, and the one that reshapes this unit: +**there is no persisted per-client desired state.** After a disable, the next +proxy startup, `ocx ensure`, `ocx sync`, a provider/model mutation, or +`POST /api/sync` can silently re-inject Codex (`src/cli/index.ts:318,365`, +`src/server/management/config-routes.ts:261`). + +That is the actual engineering work this unit needs — and it is a much smaller +thing than the operation-state engine `000_plan.md` specified. A durable +desired-state flag per client, defaulting ON so no existing setup dies on +upgrade, consulted by every automatic apply path. It is a switch's memory, not a +rollback engine. + +`030` and the durable-state design pick this up; `010` is retired as specified +because the crash-recovery machinery it described is not what the evidence asks +for. + +## Observable consequences of a Codex disable + +What the consequence dialog must actually name: + +- plain `codex` returns to its native provider path (`src/codex/inject.ts:688`) +- routed `provider/model` catalog rows disappear; native rows survive + (`src/codex/catalog/sync.ts:578,590`) +- the generated `opencodex` profile is removed or replaced, and managed native + subagent defaults are restored or stripped (`src/codex/inject.ts:703`) +- previously routed threads are retagged to their native provider — or stay + hidden if the history DB was locked (`src/codex/history-provider.ts:677`) +- the proxy keeps serving Claude, Grok, the exported file clients, and direct API + callers, because restore contains no lifecycle operation at all + +That last line is the whole point of the feature. From 535663c30711d63e1a0474bb458d202bed42e83f Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Mon, 3 Aug 2026 22:54:46 +0900 Subject: [PATCH 002/163] =?UTF-8?q?docs(integrations):=20Desktop=20disable?= =?UTF-8?q?=20is=20buildable=20=E2=80=94=20standard=20mode=20is=20a=20docu?= =?UTF-8?q?mented=20contract?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The prior unit concluded Desktop removal was impossible. It conflated two requirements: restoring the exact prior selection (still impossible, we never recorded appliedId) and returning the user to standard Claude (a documented behavior we can aim at). Anthropic's configuration reference states third-party mode activates only when inferenceProvider and its credentials are valid; otherwise Desktop launches in standard mode. So we point appliedId at a present, readable, credential-free config with no inferenceProvider instead of guessing what an absent or dangling appliedId does — all four unproven behaviors are designed around rather than relied on. Also kills the 'just pick Default' shortcut with local evidence: this machine's _meta.json has a Default entry whose .json does not exist. And settles the tested hypothesis: 'ocx claude desktop default' sets a model-family default inside our own profile and never touches appliedId, so it is not the restore verb its name suggests. --- .../002_desktop_standard_mode.md | 113 ++++++++++++++++++ 1 file changed, 113 insertions(+) create mode 100644 devlog/_plan/260803_codex_desktop_toggle/002_desktop_standard_mode.md diff --git a/devlog/_plan/260803_codex_desktop_toggle/002_desktop_standard_mode.md b/devlog/_plan/260803_codex_desktop_toggle/002_desktop_standard_mode.md new file mode 100644 index 000000000..4b0f8e7d3 --- /dev/null +++ b/devlog/_plan/260803_codex_desktop_toggle/002_desktop_standard_mode.md @@ -0,0 +1,113 @@ +# Claude Desktop: standard mode is documented, so disable is buildable + +Research doc. This one overturns the load-bearing conclusion of +`../260803_integrations_toggle_all/001_removal_path_inventory.md` §Claude Desktop. + +## What the previous conclusion got right, and where it overreached + +Right, and re-confirmed here: + +- `apply` sets `_meta.json`'s `appliedId` to our id **unconditionally** and never + records the previous value (`src/claude/desktop-3p.ts:345,358`). Exact + "put back whatever was selected before" is impossible from our bookkeeping. +- `.json.bak` holds the previous profile, and that profile carries + `inferenceGatewayApiKey` — so the backup can contain a credential and nothing + cleans it up (`src/claude/desktop-3p.ts:371`). +- `/status.applied` is derived from the saved fingerprint, not from actual + selection (`src/server/management/agent-settings-routes.ts:797`), so a disable + that forgets to clear the markers keeps reporting `applied: true`. +- `desktopAutoApply` is enabled by ABSENCE (`src/types.ts:456`); its guard + suppresses only an explicit `false`, and the subagent-model update route can + re-create a removed profile (`agent-settings-routes.ts:130,518`). + +The overreach was treating "cannot restore the exact prior selection" as +"cannot safely disable". **Those are different requirements**, and only the +first one is blocked. + +## The official semantics that make disable safe + +Primary Anthropic documentation, opened and read (not inferred from our devlog): + +| Source | lastmod | +|---|---| +| [Configuration reference](https://claude.com/docs/third-party/claude-desktop/configuration) | 2026-07-24 | +| [In-app configuration](https://claude.com/docs/third-party/claude-desktop/in-app-configuration) | 2026-07-17 | +| [Claude API provider](https://claude.com/docs/third-party/claude-desktop/claude-api) | 2026-07-17 | +| [Gateway provider](https://claude.com/docs/third-party/claude-desktop/gateway) | 2026-07-29 | + +The load-bearing sentence: third-party mode activates **only** when +`inferenceProvider` and that provider's required credentials are valid; +otherwise Desktop launches in **standard mode**. Configuration is read once at +launch. + +That is a documented contract we can aim at deliberately. We do not need to +guess what Desktop does with a missing or dangling `appliedId` — we can point it +at a configuration that is *present, readable, and deliberately without an +`inferenceProvider`*, which the docs say yields standard mode. + +## What stayed UNPROVEN, and why it no longer blocks us + +| Question | Status | +|---|---| +| Is there a documented "return to standard" UI button? | UNPROVEN | +| What happens when `appliedId` is absent? | UNPROVEN | +| What happens when `appliedId` dangles? | UNPROVEN | +| Is a `Default` entry guaranteed? | UNPROVEN | + +The design simply avoids all four. It never deletes `appliedId`, never leaves it +dangling, and never selects an entry merely because it is named `Default`. + +This machine proves why that last guard matters: the real `_meta.json` has a +`Default` entry **whose `.json` does not exist**. A disable that "just picks +Default" would have pointed Desktop at a missing file. The one observation the +earlier RCA generalized from was the exception, not the rule. + +The installed Desktop bundle (v1.18286.0) does appear to seed a `Default`/`{}` +profile when metadata does not yet exist, and its current reader does fall back +when the selected file is unreadable — but that is a bundle observation, not a +contract, and it does not upgrade any row above. + +## `default` is not the restore verb + +The hypothesis worth testing was that `ocx claude desktop default` already +performs an official restore. It does not. + +`default ` sets the default model **inside one opencodex +model family** (opus/fable/sonnet/haiku) via `setDesktopFamilyDefault`, saves +`claudeCode.desktopProfile`, and returns without touching Desktop's config +library or `appliedId` at all (`src/cli/claude-desktop.ts:148`, +`src/claude/desktop-profile.ts:219`). `move` likewise only edits our own routing +profile (`src/cli/claude-desktop.ts:138`). + +Neither is a disable mechanism. The name collides with the concept; the behavior +does not. + +## The disable sequence + +Ordered so Desktop is never pointed at something missing: + +1. Persist `desktopAutoApply: false` **first**, so no concurrent auto-apply + re-creates what we are about to remove. +2. Write a new opencodex-owned, credential-free configuration — no + `inferenceProvider` — under a fresh UUID, and set `appliedId` to it. +3. Only then remove the old `opencodex` entry, its `.json`, and its `.bak` + (the credential-bearing file). +4. Clear `appliedFingerprint` and `appliedAt` so `/status` stops claiming + `applied: true`. +5. Preserve `desktopProfile.assignments`/`defaults` so re-enabling does not throw + away the user's model organization. This machine currently has 33 assignments. + +Two explicit non-choices: do not route through `inferenceProvider: "anthropic"`, +because that is direct Claude API billing rather than the user's normal +subscription mode; and do not reuse the `default` subcommand. + +Because configuration is read once at launch, the disable does not take effect +until Desktop restarts. The consequence dialog must say that plainly rather than +implying an instant switch. + +## Residual, deliberately deferred + +Recording the previous `appliedId` at apply time is still worth doing, but it now +buys only the stronger feature: "restore exactly whichever *other* third-party +provider was active before opencodex." Returning the user to standard Claude does +not need it. That keeps it out of this unit's critical path. From 13f48dbfc6c17a39d03a53eab52bca9200fb384e Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Mon, 3 Aug 2026 22:58:45 +0900 Subject: [PATCH 003/163] docs(integrations): name the two defects the toggle work actually has to fix MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 003 — a switch has no memory. Only Claude Code has a durable desired state; everything else reads observed disk artifacts, and the six-client ownership record is DELETED by disable, so it structurally cannot carry an OFF. The shipped Grok toggle already fails this way: OFF persists nothing and every ocx start / ensure / api-grok-apply rewrites the fence. Proposes a default-ON OcxConfig.clientIntegrations map, and names the paths that must NOT be gated — crash-journal repair, ownership checks, owned teardown, and shared transports, since disabling an integration means stop writing that client's config, never stop serving. 004 — one rejected modality value poisons a whole client config. gjc rejects audio on zenmux/meta-muse-spark-1.1 and falls back to its built-in list. Pi carries the identical bug (upstream schema is text|image and it returns an EMPTY config on failure), unobserved only because its file is empty here. Same class as the Codex 'video' incident that showed zero apps. Fix belongs at the client-dialect boundary, not in ExportModel or normalizeExportModels. --- .../003_durable_desired_state.md | 133 ++++++++++++++++++ .../004_export_modality_poisoning.md | 122 ++++++++++++++++ 2 files changed, 255 insertions(+) create mode 100644 devlog/_plan/260803_codex_desktop_toggle/003_durable_desired_state.md create mode 100644 devlog/_plan/260803_codex_desktop_toggle/004_export_modality_poisoning.md diff --git a/devlog/_plan/260803_codex_desktop_toggle/003_durable_desired_state.md b/devlog/_plan/260803_codex_desktop_toggle/003_durable_desired_state.md new file mode 100644 index 000000000..34f8811ae --- /dev/null +++ b/devlog/_plan/260803_codex_desktop_toggle/003_durable_desired_state.md @@ -0,0 +1,133 @@ +# Desired state vs observed state — the thing this unit actually needs + +Research doc. It replaces the durable-operation-state design `000_plan.md` WP1 +asked for. That design solved crash recovery; the evidence says the real defect +is that a switch has no memory. + +## The requirement, stated exactly + +"Turning a switch off must leave the proxy running and serving every other +client. I might want everything except Codex on." + +That has two halves, and only the first was ever examined: + +1. **The mutation must not stop the proxy.** Already true for Codex + (`001_native_restore_thesis.md`) and for the two shipped toggles. +2. **The OFF must survive.** Not true today for anything except Claude Code. + +## The distinction everything turns on + +| | Desired state | Observed state | +|---|---|---| +| Means | what the user asked for | what is currently on disk | +| Survives a restart | yes | only accidentally | +| Can gate an automatic path | yes | no — it IS the thing the path rewrites | + +Almost every artifact we have is observed, not desired: + +| Artifact | Which | Client | +|---|---|---| +| `config.claudeCode.enabled` | **desired** | Claude Code | +| `claudeCode.desktopAutoApply` | desired *policy*, but only "auto-rewrite a saved profile" — not "Desktop is enabled" | Claude Desktop | +| `desktopProfile.appliedFingerprint` / `appliedAt` | observed | Claude Desktop | +| Six-client ownership records | observed provenance — and **disable deletes the record** (`src/integrations/writer.ts:373-384`), so it structurally cannot carry an OFF | the six file clients | +| Grok fence presence | observed disk artifact (`src/grok/inspect.ts:19-44`) | Grok | +| Codex journal | observed recovery artifact (`src/codex/journal.ts:10-18`) | Codex | +| `codexAutoStart` | desired *lifecycle* policy; false makes `ocx ensure` skip the proxy entirely (`src/cli/index.ts:358-364`) | not a client flag at all | + +The ownership-record row is the sharpest proof. A record that is deleted by the +very operation whose intent we want to remember can never be that memory. + +## The shipped Grok toggle is already broken this way + +Not a hypothetical. `PUT` OFF strips the fence and returns, persisting nothing +(`src/server/management/native-integration-routes.ts:228-256`). Then: + +- every `ocx start` rewrites it (`src/cli/index.ts:334-341`) +- both branches of `ocx ensure` rewrite it (`src/cli/index.ts:372-379`, `:398-404`) +- service start, login start, dashboard restart and tray restart all funnel into + startup (`src/service.ts:318-340`, `src/server/management/system-restart.ts:90-170`) +- `POST /api/grok/apply` regenerates it directly (`agent-settings-routes.ts:639-652`) + +`syncGrokConfig` has no enabled check at all (`src/grok/sync.ts:29-65`). So a +user who turns Grok off gets it back on the next restart, silently. That is a +regression against the switch we shipped hours ago, and it lands in this unit +because the fix is the same schema. + +Claude Code is the counter-example that proves the shape works: `enabled` is +persisted and every automatic consumer honors it — system env +(`src/server/system-env.ts:251-256`), the launcher (`src/cli/claude.ts:236-242`), +agent sync (`src/claude/agents-inject.ts:247-254`), inbound +(`src/server/claude-messages.ts:65-69`), model discovery +(`src/server/index.ts:493-502`). + +## Codex's automatic re-apply paths + +All ungated today: + +| Path | Trigger | +|---|---| +| `ocx start` | every proxy start (`src/cli/index.ts:318-341`) | +| `ocx ensure` (both branches) | tray, restart, many commands (`src/cli/index.ts:365-411`) | +| `POST /api/sync` | dashboard sync (`config-routes.ts:261-268`) | +| `ocx sync`, `ocx restore back` | explicit, but should still respect OFF | +| `ocx models custom add/remove` | any custom-model edit with a live proxy (`src/cli/models.ts:102-206`) | +| provider/model/combo mutations | via `refreshCodexCatalogBestEffort` (`management-api.ts:105-112`) — catalog only, not injection | + +The last row matters for scoping: provider and model mutations rewrite catalog +artifacts but do NOT call `injectCodexConfig`, so they need their own gate rather +than riding on the sync gate. + +`ocx opencode` deserves its own line: it injects `provider.opencodex` inline via +`OPENCODE_CONFIG_CONTENT`, and that inline layer outranks disk config +(`src/cli/opencode.ts:461-477,531-572`). The six file clients look safe from +auto-reapply only because no automatic writer calls them — an accident, not a +guarantee — and this path already bypasses it. + +## The schema + +In `OcxConfig`, not the integrations store — the store holds observed state by +construction: + +```ts +type ClientIntegrationId = + | "codex" | "claude-code" | "claude-desktop" | "grok" + | "opencode" | "pi" | "hermes" | "openclaw" | "kimi" | "gajae"; + +clientIntegrations?: Partial>; +``` + +Effective state is `config.clientIntegrations?.[client] !== false`. A missing +map, a missing key, and an explicit `true` all mean ON, so **no existing setup +changes behavior on upgrade**. That defaulting is not a convenience; it is the +only acceptable migration for a feature that can silently unplug someone's +working client. + +Compatibility rules: + +- `claude-code` absent → fall back to `config.claudeCode?.enabled !== false`, and + mirror both during the transition, so an existing Claude OFF never migrates + itself back to ON. +- Never infer Desktop OFF from `desktopAutoApply: false`. Different intent. +- **Desired intent persists independently of mutation success.** If a disable + hits an ownership refusal or drift, keep desired `false` and report + "desired OFF, observed conflict". Otherwise the next automatic path quietly + undoes what the user asked for. + +## What must NOT be gated + +A gate in the wrong place turns a safety mechanism off: + +- Codex crash-journal reconciliation (`src/codex/journal.ts:148-162`) — it repairs + our own stale state. +- Ownership, drift and compare-before-write checks (`src/integrations/writer.ts:171-223`). +- Owned teardown on stop/uninstall (`src/service.ts:2587-2594`). Stopping the + service must never rewrite desired ON into OFF. +- Grok's non-loopback credential-safety cleanup (`src/grok/inject.ts:359-380`). +- **Shared transports.** Codex OFF must not disable `/v1/responses`, which other + clients use; Claude Code and Desktop flags must not shut down their shared + `/v1/messages`. Disabling an integration means "stop writing into that client's + config", never "stop serving". + +That last rule is the user's requirement restated as an invariant, and it is the +one an implementer is most likely to violate while feeling productive. diff --git a/devlog/_plan/260803_codex_desktop_toggle/004_export_modality_poisoning.md b/devlog/_plan/260803_codex_desktop_toggle/004_export_modality_poisoning.md new file mode 100644 index 000000000..5c23869fa --- /dev/null +++ b/devlog/_plan/260803_codex_desktop_toggle/004_export_modality_poisoning.md @@ -0,0 +1,122 @@ +# One rejected modality value poisons a whole client config + +Research doc. A live bug, its blast radius across the other five exporters, and +where the fix belongs. + +## The observed failure + +Gajae Code refuses to load its entire config: + +``` +Failed to load config file models, Schema error: +/providers/opencodex/models/30/input/2: Invalid option: expected one of "text"|"image" +``` + +Model index 30 (0-based) is `zenmux/meta-muse-spark-1.1`, whose `input` we wrote +as `[text, image, audio]`. Index 2 of that array is `audio`. + +The user-visible consequence is out of all proportion to the cause: gjc falls +back to its built-in Anthropic list, so every routed model disappears at once. +One value in one model takes down the whole file. + +## Why we emit a value the client rejects + +Our internal modality vocabulary is `text | image | audio` +(`src/server/management/model-routes.ts:13`, `src/cli/models.ts`), and the two +affected exporters copy it through verbatim: + +- `buildGajaeClientConfig` — `src/clients/config-export.ts:765` +- `buildPiClientConfig` — `src/clients/config-export.ts:659` + +Both clients accept only `text | image`. Gajae's installed schema pins it at +`@gajae-code/coding-agent/src/config/models-config-schema.ts:119`, and Pi's +upstream source does the same in `packages/coding-agent/src/core/model-config.ts:156-169`, +with whole-file rejection at `:267-274` — Pi returns an EMPTY model config on a +schema failure rather than dropping the offending entry. + +**So Pi carries the identical bug.** It has not been observed only because this +machine's `~/.pi/agent/models.json` is currently empty. This is a latent live +defect, not a hypothetical. + +A stale comment at `config-export.ts:649` still calls Pi's schema UNVERIFIED. +It is verified now, and it says `text|image`. + +## This exact bug already happened once + +`tests/catalog-input-modality-enum.test.ts:5-12` records the precedent in its own +words: zenmux advertised `video`, we wrote it through verbatim, and the Codex app +reported `unknown variant 'video'` **while showing zero apps** — because the +catalog is referenced from config, so the rejection cascaded into plugins, apps +and MCP servers. + +`ensureStrictCatalogFields` (`src/codex/catalog/parsing.ts`) was the fix for the +Codex path: filter to the accepted enum, and fall back to `["text"]` rather than +an empty list, because a modality-less entry would leave the client unable to +tell the model takes prompts at all. + +The lesson did not generalize to the client exporters. Same class, same shape, +different destination. + +## The other four exporters + +| Client | Emits modalities? | Residual risk | +|---|---|---| +| OpenCode | no | ids/names/context unvalidated against client semantics | +| Hermes | no | selector strings unvalidated | +| OpenClaw | no | ids/names/context unvalidated | +| Kimi | no | alias/model characters unchecked; non-finite contexts already omitted | + +No modality exposure outside Pi and Gajae. The residual rows are marked INFERRED +by the audit — only Gajae's and Pi's schemas were directly verified — so they are +recorded as follow-up, not folded into this unit. + +Two things that are NOT at risk, checked rather than assumed: syntax injection +via quotes/newlines is prevented by the serializers +(`src/integrations/serialize.ts:52,167,224`), and numeric handling already omits +zero/negative/NaN/Infinity and floors fractions (`config-export.ts:420,431`). +There is no upper sanity cap on context, which is a real but separate gap. + +## Where the fix goes + +At the **client-dialect boundary**, as one helper beside `authoritativeContextWindow`: + +```ts +inputModalitiesForClient(client, inputModalities) +``` + +Accepted vocabulary `text|image` for both current callers; preserve order, +dedupe, and fall back to `["text"]` when filtering empties the list — matching +the Codex precedent exactly. + +Called from the two emission sites only: `config-export.ts:659` (Pi) and `:765` +(Gajae). + +Two rejected alternatives, each for a concrete reason: + +- **Not in `ExportModel` construction.** The management and CLI boundaries carry + catalog modalities verbatim on purpose (`src/server/management/model-rows.ts:91`, + `src/cli/export-command.ts:82`). Stripping `audio` globally would destroy valid + internal metadata before the destination is known. +- **Not inside `normalizeExportModels`.** Its documented and tested contract is + first-wins dedupe plus deterministic sorting (`config-export.ts:508`, + `tests/client-config-export.test.ts:261`). Overloading it hides the behavior + from the place a reader would look for it. + +## What the existing tests do and do not pin + +Pinned: Pi's `["text"]` default, `[text,image]` preservation, numeric +omission/clamping, and a byte-exact golden +(`tests/client-config-export.test.ts:184,191,201,343`). Gajae's tests pin allowed +**field names**, not value enums +(`tests/client-config-export-new-clients.test.ts:251`). + +Nothing exercises `audio` or any rejected modality, which is precisely why 91 +green tests coexist with a config the client refuses to load. The regression test +must assert the emitted VALUE vocabulary for both clients, and the fix must not +disturb the byte-exact golden beyond the intended modality change. + +## Verification bar + +A unit test alone does not close this. The criterion is the actually emitted +file: re-apply the gajae integration and confirm gjc loads it with no schema +error. The bug was found in a real file, and it gets closed in one. From f375325a27cb923b364353e908f241b70018465b Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Mon, 3 Aug 2026 23:02:57 +0900 Subject: [PATCH 004/163] docs(integrations): re-plan the unit around desired state, not a rollback engine The operation-state engine (010, never written) is dropped: research 001-003 shows Codex restore already works with the proxy up, Desktop disable has a documented standard-mode target, and the actual missing piece is a switch that remembers being off. New phase map, dependency-ordered: modality filter (independent, fixes a live failure), desired-state schema (the foundation both toggles consume and the fix for the shipped Grok regression), API keys row, then Codex and Desktop as parallel siblings. 010 is the first phase written to diff level: one client-dialect modality helper called from the Pi and Gajae builders only, with the whole-catalog assertion that would actually have caught the bug the per-entry tests missed. --- .../260803_codex_desktop_toggle/000_plan.md | 157 ++++++++++-------- .../010_modality_boundary.md | 142 ++++++++++++++++ 2 files changed, 232 insertions(+), 67 deletions(-) create mode 100644 devlog/_plan/260803_codex_desktop_toggle/010_modality_boundary.md diff --git a/devlog/_plan/260803_codex_desktop_toggle/000_plan.md b/devlog/_plan/260803_codex_desktop_toggle/000_plan.md index 33b319e0e..aca5666a2 100644 --- a/devlog/_plan/260803_codex_desktop_toggle/000_plan.md +++ b/devlog/_plan/260803_codex_desktop_toggle/000_plan.md @@ -1,81 +1,104 @@ -# Codex and Claude Desktop toggles +# Client integration switches: Codex, Claude Desktop, and the memory they need -Split out of `260803_integrations_toggle_all` after its fourth audit -(`../260803_integrations_toggle_all/007_audit_synthesis_r4.md`). That unit keeps -Claude Code and Grok, whose toggles need no durable operation state. These two -need one, and it is the shared dependency four audit rounds kept circling. +Split out of `260803_integrations_toggle_all` after its fourth audit. That unit +shipped Claude Code and Grok. This one was scoped around a durable +operation-state engine — and a research cycle against the real code says that +engine solves the wrong problem. -## Why these two are together and separate +> **Re-planned 260803 after four research passes.** The rollback engine +> (`010_operation_state.md`, never written) is **dropped**. Evidence: +> `001`-`004`. The replacement is smaller, and it also fixes a defect in the +> toggle we shipped hours ago. -| | Codex | Claude Desktop | -|---|---|---| -| Artifacts | `config.toml`, `opencodex.config.toml`, model catalog, resume history (SQLite) | `.json`, `.bak`, `_meta.json`, four fields in our config | -| Owner | us, mostly | **Claude Desktop** — we edit another app's registry | -| Prior state recoverable? | from its own journal, when hashes still match | only from `_meta.json` bytes; `appliedId` was never recorded | -| Removal code exists? | yes, `restoreNativeCodex` | **no** | +## The requirement in the owner's words + +> "스위치를 꺼도 프록시는 살아있어야 돼. 코덱스 말고 다른 것만 켜고 싶을 수도 있잖아." + +Turning a client off must leave the proxy running and serving every other +client. That is two obligations: the mutation must not stop the proxy, and the +OFF must survive a restart. -Neither can be undone by re-running its enable path, which is what makes Claude -Code and Grok cheap. Codex's enable is not the inverse of its disable — the -journal fallback means a disable may strip fragments rather than restore bytes, -and a later enable writes today's catalog, not yesterday's arrangement. Desktop -cannot re-derive which profile the user had selected at all. +## What the research changed -So both need an operation record that outlives the request, and that record is -the first phase. +| Prior claim | What the code says | +|---|---| +| Codex needs a durable operation-state engine | `ocx restore` already restores native Codex **without stopping the proxy** (`src/cli/help.ts:18`), and `ocx restore back` is the enable direction. Both exist. (`001`) | +| Desktop removal is impossible to do safely | Anthropic documents that a selected config without a valid `inferenceProvider` launches **standard mode**. We aim at that instead of guessing. (`002`) | +| The missing piece is crash recovery | The missing piece is **desired state**. Only Claude Code has one; Grok's shipped toggle is silently re-enabled by the next `ocx start`. (`003`) | + +The fourth research doc (`004`) is an unrelated live defect found while looking: +one out-of-enum modality value makes a client reject its entire config. It joins +this unit as an independent work-phase (LOOP-UNIT-CHAIN-01). ## Read first -From the parent unit, all still authoritative: +In this unit, all four written this cycle: + +- `001_native_restore_thesis.md` — Codex restore, its asymmetries, the history lock +- `002_desktop_standard_mode.md` — the official standard-mode contract, and why `default` is not the restore verb +- `003_durable_desired_state.md` — desired vs observed, the Grok regression, the schema +- `004_export_modality_poisoning.md` — the gjc/Pi enum defect -- `001_removal_path_inventory.md` — what each client's disable actually costs -- `003`, `004`, `005`, `007` — the four audit syntheses; `007` is why this unit - exists -- `002_consequence_dialog_ux.md` — dialog direction; the Codex and Desktop copy - lives there +From the parent unit, still authoritative: `001_removal_path_inventory.md` +(what each disable costs) and `002_consequence_dialog_ux.md` (dialog direction). +Its `007_audit_synthesis_r4.md` is the reason this unit exists, but its central +conclusion is now **superseded** by `001`-`003` here. ## Phases +Dependency-ordered (PHASE-SPLIT-01): the schema is the foundation every switch +consumes, so it goes first even though it ships no visible switch. + | Phase | Doc | Deliverable | |---|---|---| -| WP1 | `010_operation_state.md` | The durable operation record: a versioned discriminated journal entry, prepare/commit, restart reconciliation, a field-scoped config writer | -| WP2 | `020_codex_toggle.md` | Codex disable/enable on top of it | -| WP3 | `030_desktop_toggle.md` | Desktop removal + rollback, `appliedProfileId` schema work | - -WP2 and WP3 are **parallel siblings**, not a sequence: Desktop does not depend on -Codex (audit r4 #11). Both depend on WP1 and nothing else. - -## What WP1 must deliver - -Named by audit round 4, findings #1, #2, #5, #3: - -1. **A versioned discriminated journal entry.** Today's `JournalEntry` - (`src/integrations/journal.ts:34-51`) is file-shaped: one `configPath`, one - `SnapshotRef`, one `resultFingerprint`. A routing description and three - library members do not fit it. Needs `file-v1 | native-state-v1 | - desktop-hybrid-v1` with validated serialized state, per-member fingerprints, - and retention behavior. -2. **Prepare/commit with restart reconciliation.** A crash between the mutation - and the append leaves no undo state at all. Idempotent re-apply does not fix - this — it helps once a state exists, and none was recorded. The prepared - record must be durable before the mutation and resolved after. -3. **A field-scoped config writer.** `saveConfigPreservingClaudeCode` persists - the whole live object and its own docstring says a `providers` hand edit is - clobbered (`src/config.ts:2132-2135`); worse, when disk and caller both - changed `claudeCode`, the caller's stale subtree wins, so one toggle field can - clobber a concurrent Desktop-profile edit. Desktop's four fields need a write - that reloads from disk and touches only named paths. - -## Carried-forward findings - -Everything in `005` §Carried forward plus round 4's, in particular: Codex's -pre-state must capture the effective injection mode and history policy, not just -a routing kind (r4 #4); `injectCodexConfig` takes a concrete `catalogPath`, so -"selector" needs a resolver (r4 #4); Desktop's ordering is dangling-pointer-safe -but not transactionally crash-safe and must say so (r4 #6); auto-apply must be -suppressed while an operation is prepared (r4 #6). - -## Status - -Not started. `020` and `030` carry their pre-split content and are **stale** -against the operation-state design WP1 has not written yet; their next P -re-verifies them against it before any build. +| WP2 | `010_modality_boundary.md` | The client-dialect modality filter. Independent of everything else; fixes a live user-visible failure | +| WP3 | `020_desired_state.md` | `OcxConfig.clientIntegrations`, default-ON, consulted by every automatic apply path — including the Grok regression | +| WP4 | `030_api_keys_row.md` | API keys out of the card grid into their own row | +| WP5 | `040_codex_toggle.md` | The Codex switch on top of WP3, with a structured restore result | +| WP6 | `050_desktop_toggle.md` | The Desktop switch via documented standard mode | + +WP1 was this cycle: the research above plus this roadmap. + +WP2 and WP4 are independent of the rest and of each other. WP5 and WP6 are +parallel siblings that both depend on WP3. + +## Scope boundary + +IN: `src/clients/config-export.ts`, `src/types.ts`, `src/config.ts`, +`src/codex/sync.ts`, `src/grok/sync.ts`, `src/cli/index.ts`, +`src/cli/opencode.ts`, `src/claude/desktop-3p.ts`, +`src/server/management/native-integration-routes.ts`, +`src/server/management/agent-settings-routes.ts`, `src/server/management-api.ts`, +`gui/src/pages/integrations/*`, `gui/src/styles-integrations.css`, +`gui/src/i18n/*`, `tests/`, `gui/tests/`. + +OUT: releases, publishing, deploys, tags; starring the repository; rewriting the +six-client file machinery in `src/integrations/`; `docs-site` restructuring; +recording the previous `appliedId` (deferred, `002` §Residual). + +## Criteria + +- C1 — gjc loads our emitted config with no schema error, proven from the real + file; Pi's identical exposure is closed in the same change. +- C2 — a disabled client stays disabled across a proxy restart, an `ocx ensure`, + and a `POST /api/sync`. +- C3 — an upgrading user with no `clientIntegrations` key sees no behavior change. +- C4 — disabling any client never stops the proxy and never disables a shared + transport used by another client. +- C5 — Codex toggles both directions from the overview with the proxy running. +- C6 — a Codex disable blocked by the held history DB is an explained refusal + naming the cause, never a raw 500 and never a false green. +- C7 — Desktop's disable points `appliedId` at a present, readable, + credential-free config and removes the credential-bearing `.bak`. +- C8 — API keys render as a row above the grid, observed rendered. +- C9 — typecheck, full test, gui lint, gui test, privacy scan all green. + +## Risk register + +| Risk | Mitigation | +|---|---| +| A gate silently unplugs a working client on upgrade | Absent key means ON, everywhere, with a test for the absent-config case (`003`) | +| Gating a safety path | Explicit do-not-gate list: journal repair, ownership checks, owned teardown, shared transports (`003`) | +| Desktop pointed at a missing file | Never delete `appliedId`, never leave it dangling, never pick an entry by the name `Default` — this machine's `Default` is already dangling (`002`) | +| The modality fix erases valid internal metadata | Filter at the client-dialect boundary only; management and CLI keep carrying `audio` verbatim (`004`) | +| A green suite hides the real failure | 91 tests pass today beside a config gjc refuses to load. Every criterion names a live artifact, not a unit test | diff --git a/devlog/_plan/260803_codex_desktop_toggle/010_modality_boundary.md b/devlog/_plan/260803_codex_desktop_toggle/010_modality_boundary.md new file mode 100644 index 000000000..b0c5c3e36 --- /dev/null +++ b/devlog/_plan/260803_codex_desktop_toggle/010_modality_boundary.md @@ -0,0 +1,142 @@ +# WP2 — the client-dialect modality filter + +Research: `004_export_modality_poisoning.md`. Read it first; this doc is the diff. + +Independent of every other phase. It fixes a failure the user is hitting right +now, so it goes first despite not being a switch. + +## IN / OUT + +IN: `src/clients/config-export.ts` (MODIFY), +`tests/client-export-modality-enum.test.ts` (NEW). + +OUT: `src/server/management/model-rows.ts`, `src/cli/export-command.ts`, +`src/codex/catalog/*`, `normalizeExportModels`. All four deliberately keep +carrying `audio` — the internal vocabulary is correct, only two destinations are +narrower. + +## The helper + +MODIFY `src/clients/config-export.ts`, immediately after `outputBudgetFor` +(currently line 432) so the two value-normalizing helpers sit together: + +```ts +/** + * Modalities a given client's schema will actually accept. + * + * Our internal vocabulary is `text | image | audio` (model-routes.ts + * ALLOWED_INPUT_MODALITIES). Pi and Gajae both accept only `text | image`, and + * both reject the WHOLE config file over one out-of-enum value — Gajae reports + * `/providers/opencodex/models/N/input/2: Invalid option` and falls back to its + * built-in list, Pi returns an empty model config. So a single `audio` model + * takes every routed model down with it. + * + * This is the same defect the Codex catalog had with `video`, where the app + * showed zero apps (tests/catalog-input-modality-enum.test.ts). The fix is the + * same shape: filter to what the destination accepts, and fall back to `text` + * rather than an empty list — a modality-less entry would leave the client + * unable to tell the model takes prompts at all. + * + * Deliberately NOT applied in ExportModel construction: the management and CLI + * boundaries carry catalog modalities verbatim on purpose, and stripping `audio` + * globally would destroy valid metadata before the destination is known. + */ +const CLIENT_INPUT_MODALITIES: Record<"pi" | "gajae", ReadonlySet> = { + pi: new Set(["text", "image"]), + gajae: new Set(["text", "image"]), +}; + +function inputModalitiesForClient( + client: "pi" | "gajae", + modalities: readonly string[] | undefined, +): string[] { + const accepted = CLIENT_INPUT_MODALITIES[client]; + const kept: string[] = []; + for (const value of modalities ?? []) { + if (accepted.has(value) && !kept.includes(value)) kept.push(value); + } + return kept.length > 0 ? kept : ["text"]; +} +``` + +Order-preserving and deduping, so `[text, image, audio]` becomes `[text, image]` +and the existing byte-exact golden is unaffected for models that never carried +`audio`. + +## Call site 1 — Pi + +`buildPiClientConfig`, currently line 659: + +```diff + const entry: PiModelEntry = { + id: model.namespaced, + name: exportModelLabel(model), +- // Text is the one modality every routed model supports; anything richer must come +- // from the catalog rather than an assumption. +- input: model.inputModalities && model.inputModalities.length > 0 ? [...model.inputModalities] : ["text"], ++ // Text is the one modality every routed model supports; anything richer must come ++ // from the catalog rather than an assumption — and must still be inside the ++ // enum Pi accepts, because Pi returns an EMPTY model config on a schema ++ // failure rather than dropping the offending entry. ++ input: inputModalitiesForClient("pi", model.inputModalities), + }; +``` + +Also MODIFY the stale docstring above `buildPiClientConfig` (line 649), which +still says Pi's schema is UNVERIFIED. It is verified now — upstream +`packages/coding-agent/src/core/model-config.ts:156-169` pins `text|image`, and +`:267-274` is the whole-file rejection. Replace the "UNVERIFIED" sentence with +that citation. + +## Call site 2 — Gajae + +`buildGajaeClientConfig`, currently line 765: + +```diff + const entry: GajaeModelEntry = { + id: model.namespaced, + name: exportModelLabel(model), +- input: model.inputModalities && model.inputModalities.length > 0 +- ? [...model.inputModalities] +- : ["text"], ++ input: inputModalitiesForClient("gajae", model.inputModalities), + }; +``` + +## Test — `tests/client-export-modality-enum.test.ts` (NEW) + +Named to sit beside `catalog-input-modality-enum.test.ts`, whose incident this +repeats. Cases: + +1. `audio` is dropped from a Gajae entry — the exact live failure, using + `zenmux/meta-muse-spark-1.1` with `[text, image, audio]`, asserting + `[text, image]`. +2. The same for Pi, so the latent half is pinned too. +3. A model whose only modality is rejected falls back to `["text"]`, never `[]`. +4. `[text, image]` survives untouched in both. +5. Order and dedupe: `[image, text, image]` yields `[image, text]`. +6. A whole-catalog assertion: no emitted Pi or Gajae `input` value is outside + `text|image`, given a catalog containing `audio`. This is the one that would + have caught the bug, since the per-entry tests all passed while the file was + broken. + +## Verification + +A unit test does not close this — 91 tests were green beside a config gjc +refuses to load. + +1. `bun run typecheck`, `bun run test` +2. Re-apply the gajae integration through the running proxy +3. `grep -c audio ~/.gjc/agent/models.yml` → 0 inside the opencodex block +4. Launch gjc and confirm the model list loads with no schema error + +Step 4 is the criterion. Steps 1-3 are necessary and insufficient. + +## Accept criteria + +- C1 — gjc loads the emitted config with no schema error, observed in the real + file, and Pi's identical exposure is closed in the same change. +- No emitted Pi/Gajae `input` value outside the client's enum, asserted over a + whole catalog rather than one entry. +- The byte-exact export goldens still pass, changing only where `audio` was + previously emitted. From 98a6fb9224acbc7eea7d7df88e121f9e6857b518 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Mon, 3 Aug 2026 23:26:16 +0900 Subject: [PATCH 005/163] docs(integrations): write every implementation phase to diff level, retire the superseded two MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 020 desired-state, 030 api-keys row, 040 codex toggle, 050 desktop toggle — each a copy-paste-executable design against the current tree, with real diffs, test plans naming specific files and cases, and live-proof verification rather than a green suite. The pre-split 020_codex_toggle and 030_desktop_toggle move to _retired/ with a note on which premise failed: Codex did not need a captured pre-state because ocx restore already exists, and Desktop's removal was judged impossible only by conflating exact-prior-selection restore with return-to-standard-mode. --- .../020_desired_state.md | 779 ++++++++++++++++++ .../030_api_keys_row.md | 332 ++++++++ .../040_codex_toggle.md | 775 +++++++++++++++++ .../050_desktop_toggle.md | 627 ++++++++++++++ .../_retired/000_why_retired.md | 23 + .../{ => _retired}/020_codex_toggle.md | 0 .../{ => _retired}/030_desktop_toggle.md | 0 7 files changed, 2536 insertions(+) create mode 100644 devlog/_plan/260803_codex_desktop_toggle/020_desired_state.md create mode 100644 devlog/_plan/260803_codex_desktop_toggle/030_api_keys_row.md create mode 100644 devlog/_plan/260803_codex_desktop_toggle/040_codex_toggle.md create mode 100644 devlog/_plan/260803_codex_desktop_toggle/050_desktop_toggle.md create mode 100644 devlog/_plan/260803_codex_desktop_toggle/_retired/000_why_retired.md rename devlog/_plan/260803_codex_desktop_toggle/{ => _retired}/020_codex_toggle.md (100%) rename devlog/_plan/260803_codex_desktop_toggle/{ => _retired}/030_desktop_toggle.md (100%) diff --git a/devlog/_plan/260803_codex_desktop_toggle/020_desired_state.md b/devlog/_plan/260803_codex_desktop_toggle/020_desired_state.md new file mode 100644 index 000000000..73f38fec7 --- /dev/null +++ b/devlog/_plan/260803_codex_desktop_toggle/020_desired_state.md @@ -0,0 +1,779 @@ +# WP3 — durable per-client desired state + +Research: `003_durable_desired_state.md`. Read it first; this doc is the diff. + +The shipped Grok switch removes its fence but records no intent. The next +`ocx start` calls `syncGrokConfig` and writes the fence back +(`src/cli/index.ts:334-341`, `src/grok/sync.ts:29-65`). WP3 gives that OFF a +durable owner before WP5 and WP6 add two more switches with the same failure +mode. + +## IN / OUT + +| Path | Change | Why it is in WP3 | +|---|---|---| +| `src/types.ts` | MODIFY | Defines the persisted client-id vocabulary and `OcxConfig.clientIntegrations`. | +| `src/config.ts` | MODIFY | Parses the map without turning one malformed value into an all-clients reset; owns effective-state reads and transition writes. | +| `src/codex/sync.ts` | MODIFY | Stops every Codex catalog/injection sync while Codex is desired OFF. | +| `src/server/management-api.ts` | MODIFY | Gates the direct catalog refresher that provider/model routes call, and moves Claude agent sync to the compatibility helper. | +| `src/grok/sync.ts` | MODIFY | Closes every start/ensure/restart path at the shared sync owner. | +| `src/server/management/native-integration-routes.ts` | MODIFY | Persists Grok intent before touching its file and mirrors Claude Code's old/new keys. | +| `src/server/management/agent-settings-routes.ts` | MODIFY | Requires both Desktop desired ON and `desktopAutoApply`, and mirrors Claude Code's settings route. | +| `src/cli/opencode.ts` | MODIFY | Stops the inline provider layer from bypassing an OpenCode OFF. | +| `src/cli/claude.ts` | MODIFY | Reads Claude Code desired state through the compatibility helper. | +| `src/claude/agents-inject.ts` | MODIFY | Reads Claude Code desired state through the compatibility helper. | +| `src/server/system-env.ts` | MODIFY | Reads Claude Code desired state through the compatibility helper. | +| `src/server/claude-messages.ts` | MODIFY | Removes the existing Claude-Code-only gate from the shared Messages transport. | +| `src/server/index.ts` | MODIFY | Removes the existing Claude-Code-only gate from shared Anthropic model discovery. | +| `tests/client-integration-desired-state.test.ts` | NEW | Pins schema defaulting, malformed-key salvage, compatibility, and mirroring. | +| `tests/client-integration-auto-gates.test.ts` | NEW | Drives every automatic gate and proves its writer is not called. | +| `tests/client-integration-transport-isolation.test.ts` | NEW | Proves one disabled client cannot shut down another client's transport. | +| `tests/native-grok-toggle.test.ts` | MODIFY | Pins persist-before-mutate and desired/observed conflict reporting. | +| `tests/native-claude-code-toggle.test.ts` | MODIFY | Pins both-key mirroring, including the old-value idempotent case. | +| `tests/claude-management-api.test.ts` | MODIFY | Pins mirroring through the older `/api/claude-code` route. | +| `tests/claude-messages-endpoint.test.ts` | MODIFY | Replaces the shipped transport-403 assertion with the shared-transport invariant. | + +OUT: + +| Path / surface | Reason | +|---|---| +| `gui/` | WP3 has no new switch. WP5 and WP6 consume this contract. | +| `src/integrations/writer.ts` and the six-client ownership store | Observed provenance is deleted on disable (`writer.ts:373-384`); it cannot own durable OFF intent. | +| `src/codex/journal.ts` | Crash reconciliation repairs our stale write and must run regardless of desired state. | +| `src/service.ts` stop/uninstall teardown | Teardown removes dead proxy pointers; it must neither consult nor rewrite desired state. | +| `src/grok/inject.ts` non-loopback cleanup | Credential-safety cleanup remains unconditional. | +| `/v1/responses`, `/v1/messages`, `/v1/messages/count_tokens` | They are shared transports, not client installation state. No desired-state check belongs in them. | +| Codex/Desktop mutation implementations | WP5 and WP6 own those operations. WP3 supplies only the state contract and automatic-path gates. | +| releases, publishing, deploys, tags, repository starring | No delivery or user-identity action belongs in a foundation phase. | + +## The schema and its one reader + +MODIFY `src/types.ts` immediately before `OcxConfig` (currently line 533), then +put the field beside `claudeCode` (currently line 544): + +```diff + export interface OcxApiKeyEntry { + id: string; + name: string; + key: string; + createdAt: string; + } + ++export type ClientIntegrationId = ++ | "codex" ++ | "claude-code" ++ | "claude-desktop" ++ | "grok" ++ | "opencode" ++ | "pi" ++ | "hermes" ++ | "openclaw" ++ | "kimi" ++ | "gajae"; ++ + export interface OcxConfig { + port: number; +``` + +```diff + /** One-time migration marker for Antigravity's static catalog default. */ + googleAntigravityStaticCatalogVersion?: 1; + /** Claude Code inbound + launcher settings. */ + claudeCode?: OcxClaudeCodeConfig; ++ /** ++ * The user's durable ON/OFF intent for each client integration, separate from ++ * whatever config happens to be present on disk right now. ++ * ++ * Missing entries deliberately mean ON. Existing installations pre-date this ++ * map, and treating absence as OFF would silently unplug working clients on the ++ * first upgraded start — the same restart path that currently resurrects a Grok ++ * fence after its shipped switch removed it. ++ */ ++ clientIntegrations?: Partial>; + /** + * Up to 5 routed model ids ("/") to feature FIRST in the injected Codex catalog. +``` + +MODIFY `src/config.ts`. The parser salvages each known key independently. A +single hand-edited `"codex": "false"` becomes absent/ON, but it cannot discard a +valid `"grok": false` next to it; unknown future keys pass through so an older +binary does not erase a newer client's intent on save. + +```diff + import { + isWirePinnedModel, + MODEL_ADAPTER_OVERRIDE_ALLOWED, + OPENAI_PROVIDER_TIER_VERSION, + pinnedWireAdapter, + REASONING_SUMMARY_DELIVERY_VALUES, ++ type ClientIntegrationId, + type OcxClaudeCodeConfig, + type OcxConfig, +``` + +```diff + const apiKeyEntrySchema = z.object({ + key: z.string().refine(isUsableApiKeySecret), + // Degrades to "" here; every schema consumer then runs `normalizeApiKeyIds`, + // which fills it deterministically so the id is stable across loads. + id: z.string().catch(""), + name: z.string().catch(""), + createdAt: z.string().catch(""), + }).passthrough(); + ++const clientIntegrationsSchema = z.object({ ++ codex: z.boolean().optional().catch(undefined), ++ "claude-code": z.boolean().optional().catch(undefined), ++ "claude-desktop": z.boolean().optional().catch(undefined), ++ grok: z.boolean().optional().catch(undefined), ++ opencode: z.boolean().optional().catch(undefined), ++ pi: z.boolean().optional().catch(undefined), ++ hermes: z.boolean().optional().catch(undefined), ++ openclaw: z.boolean().optional().catch(undefined), ++ kimi: z.boolean().optional().catch(undefined), ++ gajae: z.boolean().optional().catch(undefined), ++}).passthrough(); ++ + const configSchema = z.object({ + port: z.number().int().min(0).max(65535).default(10100), +``` + +```diff + googleAntigravityStaticCatalogVersion: z.literal(1).optional().catch(undefined), ++ // Per-key catches preserve every valid OFF beside one malformed hand edit. A ++ // malformed whole map degrades to absent, which is the upgrade-safe ON default. ++ clientIntegrations: clientIntegrationsSchema.optional().catch(undefined), + providerContextCaps: z.record(z.string(), z.number().int().positive()).optional(), +``` + +Add the only effective-state reader beside the existing config feature gates +(`websocketsEnabled`, currently line 1909). No caller may open-code +`?.[client] ?? true`: Claude Code's old field is the transition exception. + +```diff + export function websocketsEnabled(config: Pick): boolean { + return config.websockets === true; + } + ++/** ++ * Resolve durable client intent without mistaking an absent upgrade-era key for ++ * an opt-out. Claude Code alone predates the shared map, so its old explicit OFF ++ * remains authoritative until a route has mirrored both representations. ++ */ ++export function clientIntegrationEnabled( ++ config: Pick, ++ client: ClientIntegrationId, ++): boolean { ++ const desired = config.clientIntegrations?.[client]; ++ if (desired !== undefined) return desired !== false; ++ if (client === "claude-code") return config.claudeCode?.enabled !== false; ++ return true; ++} ++ ++/** Write the transition representation in one place so Claude OFF cannot split-brain. */ ++export function setClientIntegrationEnabled( ++ config: OcxConfig, ++ client: ClientIntegrationId, ++ enabled: boolean, ++): void { ++ config.clientIntegrations = { ...config.clientIntegrations, [client]: enabled }; ++ if (client === "claude-code") { ++ config.claudeCode = { ...(config.claudeCode ?? {}), enabled }; ++ } ++} ++ + // --------------------------------------------------------------------------- + // Hand-edit protection for the `claudeCode` subtree (devlog 260726_claude_auth_auto/040 H1). +``` + +Truth table: + +| New key | Legacy `claudeCode.enabled` | Effective state | +|---|---:|---:| +| absent | absent / `true` | ON | +| absent | `false` | OFF | +| `true` | any | ON | +| `false` | any | OFF | + +The new key wins once present. Both Claude mutation routes write both, so the +legacy fallback can never migrate an existing Claude OFF back to ON. + +## Gate 1 — Codex's shared sync owner + +MODIFY `src/codex/sync.ts`. The return is a successful no-op because desired OFF +is policy, not a failed catalog refresh. The gate precedes the external-provider +branch too; that branch still calls `injectCodexConfig` (`sync.ts:56-70`). + +```diff +-import { applyProxyEnv, loadConfig } from "../config"; ++import { applyProxyEnv, clientIntegrationEnabled, loadConfig } from "../config"; +``` + +```diff + export async function syncModelsToCodex( + port?: number, + config: OcxConfig = loadConfig(), + log: Pick | null = console, + deps: CodexSyncDeps = defaultDeps, + ): Promise { ++ if (!clientIntegrationEnabled(config, "codex")) { ++ return { ++ ok: true, ++ added: 0, ++ catalogPath: null, ++ catalogExists: false, ++ catalogWritten: false, ++ cacheSynced: false, ++ message: "Codex integration sync skipped: desired state is OFF.", ++ }; ++ } + const p = port ?? config.port ?? 10100; +``` + +This one gate covers `ocx start`, both `ocx ensure` branches, `POST /api/sync`, +`ocx sync`, `ocx restore back`, custom-model edits, and the direct CLI provider +sync caller (`src/cli/index.ts:318-341,365-411,756-829`, +`src/cli/models.ts:102-206`, `src/cli/provider.ts:235`). + +## Gate 2 — provider/model catalog refreshes that bypass sync + +MODIFY `src/server/management-api.ts`. `refreshCodexCatalogBestEffort` directly +calls `refreshCodexModelCatalog` today (`management-api.ts:105-112`), so putting +the check only in `syncModelsToCodex` leaves every provider/model/combo mutation +able to rewrite Codex artifacts. + +```diff + import { + DEFAULT_SUBAGENT_MODELS, ++ clientIntegrationEnabled, + codexAutoStartEnabled, +``` + +```diff + async function refreshCodexCatalogBestEffort(): Promise { ++ if (!clientIntegrationEnabled(config, "codex")) return; + if (deps.refreshCodexCatalog) return deps.refreshCodexCatalog(); + try { + const { refreshCodexModelCatalog } = await import("../codex/refresh"); + await refreshCodexModelCatalog(config); +``` + +The gate comes before the injected dependency. Otherwise tests can pass while a +production caller bypasses policy through a configured seam. + +## Gate 3 — every Grok startup/ensure caller + +MODIFY `src/grok/sync.ts`, before catalog fetch and before the writer. Do not add +`"disabled"` to `GrokInjectResult.skippedReason`: those values are writer policy +outcomes from `injectGrokConfig`; desired OFF never reaches that writer. + +```diff + import { visibleNativeSlugs, filterCatalogVisibleModels, nativeOpenAiContextWindow, type CatalogModel } from "../codex/catalog"; ++import { clientIntegrationEnabled } from "../config"; + import type { OcxConfig } from "../types"; +``` + +```diff + export async function syncGrokConfig( + port: number, + config: OcxConfig, + opts: { hostname?: string; grokHome?: string } = {}, + deps: GrokSyncDeps = { fetchAllModels: defaultFetchAllModels, injectGrokConfig }, + ): Promise { ++ if (!clientIntegrationEnabled(config, "grok")) { ++ return { ++ ok: true, ++ changed: false, ++ message: "Grok config sync skipped: desired state is OFF.", ++ }; ++ } + let models: GrokInjectModel[]; +``` + +This closes all three real callers: start and both ensure branches +(`src/cli/index.ts:334-341,372-379,398-404`) plus `/api/grok/apply`, whose flight +loads fresh persisted config before calling this helper +(`src/server/management/agent-settings-routes.ts:94-107,639-652`). + +## Gate 4 — Desktop auto-apply is two policies, not one + +MODIFY `src/server/management/agent-settings-routes.ts`. Desktop desired state +and `desktopAutoApply` answer different questions: “may opencodex manage Desktop?” +and “may provider changes rewrite the saved managed profile?” Both must allow the +write. + +```diff + import { + DEFAULT_SUBAGENT_MODELS, ++ clientIntegrationEnabled, + codexAutoStartEnabled, +``` + +```diff + /** Best-effort Desktop 3P config auto-reconcile when providers change. */ + async function autoApplyDesktopBestEffort(): Promise { + try { ++ if (!clientIntegrationEnabled(config, "claude-desktop")) return; + if (config.claudeCode?.desktopAutoApply === false) return; + if (!config.claudeCode?.desktopProfile) return; +``` + +An absent desired key and absent `desktopAutoApply` both preserve the current +auto-apply behavior. `desktopAutoApply: false` must never be migrated into +Desktop desired OFF (`003_durable_desired_state.md:106-115`). + +## Gate 5 — `ocx opencode` cannot inject around disk state + +MODIFY `src/cli/opencode.ts`. The command builds `OPENCODE_CONFIG_CONTENT`, whose +provider block outranks global, project, and custom disk config +(`opencode.ts:461-477`). INFERRED decision: an explicit invocation while desired +OFF refuses with exit 1, matching `ocx claude`; launching an unwired OpenCode from +a command whose contract says “wired to the local proxy” would be a false green. + +```diff +-import { loadConfig } from "../config"; ++import { clientIntegrationEnabled, loadConfig } from "../config"; +``` + +```diff + export async function cmdOpencode(args: string[]): Promise { + const config = loadConfig(); ++ if (!clientIntegrationEnabled(config, "opencode")) { ++ console.error("OpenCode integration is disabled (config.clientIntegrations.opencode=false — turn it ON before using `ocx opencode`)."); ++ return 1; ++ } + const live = await ensureProxyForOpencode(config); +``` + +The gate precedes `ensureProxyForOpencode`; a disabled client command must not +start the proxy merely to refuse later. + +## Claude Code transition consumers + +The new map is authoritative when present; mirroring is compatibility, not a +license for old consumers to open-code the legacy field forever. Replace the +three Claude-Code-specific automatic gates and the management agent-sync gate. + +MODIFY `src/cli/claude.ts`: + +```diff +-import { loadConfig } from "../config"; ++import { clientIntegrationEnabled, loadConfig } from "../config"; +``` + +```diff + export async function cmdClaude(args: string[]): Promise { + const config = loadConfig(); +- if (config.claudeCode?.enabled === false) { +- console.error("Claude inbound is disabled (config.claudeCode.enabled=false — flip the Claude ON toggle in the GUI or edit config)."); ++ if (!clientIntegrationEnabled(config, "claude-code")) { ++ console.error("Claude Code integration is disabled — turn it ON before using `ocx claude`."); + return 1; + } +``` + +MODIFY `src/claude/agents-inject.ts`: + +```diff +-import { DEFAULT_SUBAGENT_MODELS, hasOwnProvider } from "../config"; ++import { clientIntegrationEnabled, DEFAULT_SUBAGENT_MODELS, hasOwnProvider } from "../config"; +``` + +```diff + export function injectClaudeAgentDefs(config: OcxConfig, windows: Record, configDir?: string): string[] | null { +- if (config.claudeCode?.enabled === false || config.claudeCode?.injectAgents === false) { ++ if (!clientIntegrationEnabled(config, "claude-code") || config.claudeCode?.injectAgents === false) { +``` + +MODIFY `src/server/system-env.ts`: + +```diff +-import { getConfigDir } from "../config"; ++import { clientIntegrationEnabled, getConfigDir } from "../config"; +``` + +```diff + export async function injectSystemEnv(port: number, config: OcxConfig): Promise { + if (process.platform !== "darwin") return { injected: false, reason: "not macOS" }; +- if (config.claudeCode?.enabled === false) return { injected: false, reason: "claude disabled" }; ++ if (!clientIntegrationEnabled(config, "claude-code")) return { injected: false, reason: "claude disabled" }; +``` + +MODIFY the already-open `src/server/management-api.ts` import above, then: + +```diff + async function syncClaudeAgentDefsBestEffort(): Promise { + try { + const { injectClaudeAgentDefs } = await import("../claude/agents-inject"); +- if (config.claudeCode?.enabled === false || config.claudeCode?.injectAgents === false) { ++ if (!clientIntegrationEnabled(config, "claude-code") || config.claudeCode?.injectAgents === false) { +``` + +## Persist desired intent before the Grok mutation + +MODIFY `src/server/management/native-integration-routes.ts`. The config write is +the intent commit; fence inspection/removal/injection is observation and may +refuse. Never roll the committed flag back because the file conflicted. + +```diff +-import { readRuntimePort, saveConfigPreservingClaudeCode } from "../../config"; ++import { clientIntegrationEnabled, readRuntimePort, saveConfigPreservingClaudeCode, setClientIntegrationEnabled } from "../../config"; +``` + +The response must keep the two states separate. `state` remains observed disk +state for compatibility; `desiredEnabled` is the persisted intent. + +```diff + export interface NativeStatus { + clientId: NativeIntegrationClientId; + state: "absent" | "current" | "unsafe"; ++ desiredEnabled: boolean; + installed: boolean; +``` + +```diff + export interface NativeToggleEnvelope { + ok: true; + clientId: NativeIntegrationClientId; + changed: boolean; + state: NativeStatus["state"]; ++ desiredEnabled: boolean; + message: string; +``` + +```diff + export interface NativeRefusalEnvelope { + error: string; + code: "native_integration_refused" | "native_integration_failed"; + clientId: NativeIntegrationClientId; + reason: NativeRefusalReason; + message: string; ++ desiredEnabled?: boolean; ++ observedState?: NativeStatus["state"]; + } +``` + +Change `claudeCodeEnabled` into a compatibility alias and report desired state +from both GET rows: + +```diff + /** Absent means ON: the six read sites all treat only an explicit `false` as off. */ + export function claudeCodeEnabled(config: ManagementContext["config"]): boolean { +- return config.claudeCode?.enabled !== false; ++ return clientIntegrationEnabled(config, "claude-code"); + } +``` + +```diff + return { + clientId: "claude", + state: claudeCodeEnabled(config) ? "current" : "absent", ++ desiredEnabled: claudeCodeEnabled(config), +``` + +```diff +-function grokStatus(): NativeStatus { ++function grokStatus(config: ManagementContext["config"]): NativeStatus { + const seen = inspectGrokConfig(); +``` + +```diff + return { + clientId: "grok", + state, ++ desiredEnabled: clientIntegrationEnabled(config, "grok"), + installed: seen.kind !== "not_installed", +``` + +```diff + if (url.pathname === "/api/native-integrations" && req.method === "GET") { + const { getConfigPath } = await import("../../config"); + return jsonResponse({ +- clients: [claudeStatus(config, getConfigPath()), grokStatus()], ++ clients: [claudeStatus(config, getConfigPath()), grokStatus(config)], +``` + +In `handleGrokToggle`, persist immediately after body validation and before the +first inspector. If config persistence fails, do not touch the Grok file; that is +the only failure allowed to prevent the desired-state commit. + +```diff + } + const enabled = body.enabled; ++ if (config.clientIntegrations?.grok !== enabled) { ++ const previousClientIntegrations = config.clientIntegrations; ++ setClientIntegrationEnabled(config, "grok", enabled); ++ const persist = deps.saveConfigPreservingClaudeCode ?? saveConfigPreservingClaudeCode; ++ try { ++ persist(config); ++ } catch (error) { ++ if (previousClientIntegrations === undefined) delete config.clientIntegrations; ++ else config.clientIntegrations = previousClientIntegrations; ++ if (isConfigLockError(error)) { ++ return isLockContention(error) ++ ? refusal(409, "grok", "config_busy", ++ "Another process is saving the configuration right now. Desired state was not changed; try again in a moment.") ++ : refusal(500, "grok", "write_failed", ++ `Desired state could not be saved: ${error instanceof Error ? error.message : String(error)}`); ++ } ++ throw error; ++ } ++ } + + /* + * The inspector runs BEFORE either delegate, in BOTH directions (012 §In +``` + +Every success envelope in this function adds `desiredEnabled: enabled`. Every +post-persist refusal adds `desiredEnabled: enabled` and the last observed state. +The ownership refusal is the regression case: + +```diff + const owned = assertNativeTeardownOwned(); +- if (!owned.ok) return refusal(409, "grok", "home_mismatch", owned.message); ++ if (!owned.ok) { ++ return jsonResponse({ ++ error: "native integration change refused", ++ code: "native_integration_refused", ++ clientId: "grok", ++ reason: "home_mismatch", ++ desiredEnabled: false, ++ observedState: "current", ++ message: `${owned.message} Desired OFF was saved; the observed Grok block is still present.`, ++ } satisfies NativeRefusalEnvelope, 409); ++ } +``` + +Apply the same shape to orphaned-marker, late writer refusal, and catalog failure: +desired remains what was saved; `observedState` comes from the inspector rather +than from the requested direction. The route may say “desired OFF, observed +conflict”; it must never answer “still ON” as though the request disappeared. + +## Mirror Claude Code during transition + +The native route currently skips persistence when legacy effective state already +matches (`native-integration-routes.ts:393-400`). That is no longer enough: an +old `{ claudeCode: { enabled: false } }` must acquire the new false key even +though its effective state is already OFF. + +```diff + const enabled = body.enabled; +- if (claudeCodeEnabled(config) === enabled) { ++ const alreadyMirrored = config.clientIntegrations?.["claude-code"] === enabled ++ && config.claudeCode?.enabled === enabled; ++ if (alreadyMirrored) { + return jsonResponse({ + ok: true, clientId: "claude", changed: false, + state: enabled ? "current" : "absent", ++ desiredEnabled: enabled, +``` + +```diff +- const next = { ...(config.claudeCode ?? {}), enabled }; ++ setClientIntegrationEnabled(config, "claude-code", enabled); ++ const next = config.claudeCode!; +``` + +All native Claude success envelopes add `desiredEnabled: enabled`. + +The older `/api/claude-code` route in +`src/server/management/agent-settings-routes.ts` already persists the legacy +field (`agent-settings-routes.ts:941,1060-1070`); mirror the map before that same +save rather than introducing a second write: + +```diff + } + config.claudeCode = next; + // Stamp the migration sentinel on EVERY persist of this block. The migration reads +@@ + // would be converted into a sticky manual subscription by the next startServer, and + // auto would survive exactly one proxy lifetime with no way back. + if (!next.authModeMigratedAt) next.authModeMigratedAt = new Date().toISOString(); ++ if (body.enabled !== undefined) { ++ setClientIntegrationEnabled(config, "claude-code", next.enabled !== false); ++ } + const { saveConfigPreservingClaudeCode: save } = await import("../../config"); +``` + +Import `setClientIntegrationEnabled` from `../../config` in the existing import +block. The setter runs after the migration sentinel because it reassigns +`config.claudeCode`; this order guarantees the mirrored object is the stamped +object that the existing save persists. + +## What must NOT be gated + +| Surface | Required invariant | +|---|---| +| `src/codex/journal.ts:148-162` | `reconcileJournal` always repairs a dead process's stale Codex state. Desired OFF is not permission to leave a half-applied journal. | +| `src/integrations/writer.ts:171-223` | Path resolution, ownership, parse, drift, and compare-before-write checks always run when a mutation is requested. | +| `src/service.ts:2587-2594` | Stop restores native Codex and strips Grok's dead proxy pointer. It does not write `clientIntegrations`; stopping is not opting out. | +| `src/grok/inject.ts:359-380` | A non-loopback bind always strips the unsafe loopback fence, even when desired Grok state is ON. | +| `/v1/responses` | Codex desired OFF stops Codex config/catalog writes, not the Responses transport used by OpenCode, Pi, Hermes, OpenClaw, Kimi, and Gajae. | +| `/v1/messages` and `/v1/messages/count_tokens` | Claude Code/Desktop desired state stops client-specific wiring, not the Anthropic transport shared by both clients and external callers. | + +The last invariant exposes an already-shipped contradiction. Today +`claudeCode.enabled=false` returns 403 from both Messages handlers +(`src/server/claude-messages.ts:65-69,536-548,868-872`) and empties shared +Anthropic model discovery (`src/server/index.ts:493-502`). Remove those gates; +do not replace them with `clientIntegrationEnabled`. + +MODIFY `src/server/claude-messages.ts`: + +```diff +-function claudeInboundDisabled(config: OcxConfig): Response | null { +- if (config.claudeCode?.enabled === false) { +- return anthropicErrorResponse(403, "Claude inbound is disabled (GUI: Claude ON toggle / config.claudeCode.enabled)", "permission_error"); +- } +- return null; +-} +- + async function readAnthropicBody(req: Request, budget: TranslatorBudget): Promise { +``` + +```diff + ): Promise { + logCtx.surface = "claude"; +- const disabled = claudeInboundDisabled(config); +- if (disabled) { +- if (logIds) addFinalRequestLog(logIds.requestId, logIds.start, logCtx, 403, { closeReason: "non_stream" }); +- return disabled; +- } + + let anthropicBody: unknown; +``` + +```diff + /** Documented approximation: serialize system+messages+tools, run the char estimator. */ + export async function handleClaudeCountTokens(req: Request, config: OcxConfig): Promise { +- const disabled = claudeInboundDisabled(config); +- if (disabled) return disabled; +- + let body: unknown; +``` + +MODIFY `src/server/index.ts`: + +```diff + const wantsAnthropicList = req.headers.get("anthropic-version") !== null + || url.searchParams.get("flavor") === "anthropic"; + if (wantsAnthropicList && !url.searchParams.has("client_version")) { +- if (config.claudeCode?.enabled === false) return jsonResponse({ data: [] }, 200, req, config); + // Build Desktop 3P registry so inbound alias resolution works for subsequent requests. +``` + +## Test plan + +### `tests/client-integration-desired-state.test.ts` (NEW) + +| Case | Activation and assertion | +|---|---| +| Absent-config upgrade | Load a config with no `clientIntegrations`; every id is effective ON. This is C3's upgrade case, not merely a helper call with a fabricated object. | +| Missing key / explicit true / explicit false | For every id: absent and true are ON; only false is OFF. | +| Claude legacy fallback | New key absent + `claudeCode.enabled=false` is OFF; absent/true is ON. | +| New Claude key wins | New true overrides legacy false, and new false overrides legacy true. | +| Per-key malformed salvage | Persist `{ codex: "false", grok: false }`; load yields Codex ON and Grok OFF, without falling back to a default config or losing providers. | +| Future-key preservation | An unknown boolean key survives load/save so an older binary does not erase a newer client's intent. | +| Setter mirroring | `setClientIntegrationEnabled(..., "claude-code", value)` writes both keys and preserves every unrelated Claude field. Other ids touch only the map. | + +### `tests/client-integration-auto-gates.test.ts` (NEW) + +| Gate | Activation and observable proof | +|---|---| +| Codex sync OFF | Inject spies for catalog refresh and `injectCodexConfig`; both remain at zero, including the external-provider branch. Result is the explicit successful skip. | +| Codex absent/ON | The same spies fire, pinning upgrade behavior. | +| Direct management refresh OFF | Trigger a provider and a custom-model route with `refreshCodexCatalog` injected; count stays zero. This proves the bypass gate, not `syncModelsToCodex`. | +| Grok sync OFF | Inject fetch and writer spies; neither fires. Repeat with no map and prove both fire. | +| Desktop two-key gate | Exercise provider mutation with a saved Desktop profile for all four combinations of desired ON/OFF and `desktopAutoApply` true/false; write occurs only when both policies allow it. | +| OpenCode OFF | Invoke the command through injectable launch seams; proxy ensure, catalog fetch, env build, and spawn remain uncalled, exit is 1. | +| Claude compatibility consumers | New-map false with legacy field absent blocks launcher/system-env/agent writes; absent new key + legacy false does the same. | + +### Route regressions + +MODIFY `tests/native-grok-toggle.test.ts`: + +1. Disable persists `clientIntegrations.grok=false` before `stripGrokConfig`. +2. Ownership refusal leaves that persisted false and returns + `desiredEnabled:false`, `observedState:"current"`. +3. Orphaned fence, late writer refusal, and catalog failure keep the requested + intent and report observed state; none rolls the flag back. +4. A config-lock failure calls neither strip nor inject and says desired state was + not saved. +5. The next `syncGrokConfig` with the saved config calls neither catalog nor + writer — the exact `ocx start` regression. + +MODIFY `tests/native-claude-code-toggle.test.ts` and +`tests/claude-management-api.test.ts`: both routes mirror old/new values; an old +legacy OFF plus absent new key is not treated as a no-op; unrelated Claude fields +and the auth-mode migration sentinel survive. + +### `tests/client-integration-transport-isolation.test.ts` (NEW) + +Start the real Bun proxy with `clientIntegrations.codex=false` and +`clientIntegrations["claude-code"]=false`. Assert `/healthz` stays healthy; +an invalid `/v1/responses` request reaches its normal validation response rather +than an integration-disabled response; invalid `/v1/messages` and +`/v1/messages/count_tokens` requests return their normal 400 contract, never the +old 403. Fetch Anthropic model discovery and assert it is not emptied by Claude +Code OFF. This is the case proving a disabled client does not break another +client's transport. + +MODIFY `tests/claude-messages-endpoint.test.ts:786-803` to remove the old test +that requires 403. Keeping it would encode the C4 violation as a regression. + +## Verification + +Static and suite gates: + +```bash +bun test tests/client-integration-desired-state.test.ts +bun test tests/client-integration-auto-gates.test.ts +bun test tests/native-grok-toggle.test.ts tests/native-claude-code-toggle.test.ts tests/claude-management-api.test.ts +bun test tests/client-integration-transport-isolation.test.ts tests/claude-messages-endpoint.test.ts +bun run typecheck +bun run test +bun run privacy:scan +``` + +Live proof uses the already-running proxy at `localhost:10100`; a green suite is +not restart persistence: + +1. Record `curl -fsS http://localhost:10100/healthz` and its `pid`. +2. Through the authenticated dashboard/API, turn Grok OFF. Confirm + `GET /api/native-integrations` reports `desiredEnabled:false` and observed + `absent`, or the explicit observed conflict if ownership/drift refused removal. +3. Run `ocx ensure`, then re-read both the status and `~/.grok/config.toml`. + Desired remains false and no managed fence reappears. Repeat after a real + proxy restart; `/healthz` returns with a new PID and Grok remains OFF. +4. Turn Codex OFF in the WP5 surface, run `POST /api/sync` and one provider edit, + then prove neither Codex config nor catalog artifact changed. `/healthz` remains + healthy. +5. With Claude Code desired OFF, send an invalid body to both shared paths: + + ```bash + curl -sS -o /tmp/ocx-messages-proof.json -w '%{http_code}\n' \ + -H 'content-type: application/json' -d '{}' \ + http://localhost:10100/v1/messages + curl -sS -o /tmp/ocx-count-proof.json -w '%{http_code}\n' \ + -H 'content-type: application/json' -d '{}' \ + http://localhost:10100/v1/messages/count_tokens + ``` + + Both reach normal request validation (400), not integration policy (403). + Read the bodies back; a status code without the response body is not proof of + which branch ran. +6. Re-read `/healthz`; the proxy stayed serving throughout. Compare the PID with + step 1 for the no-stop toggle operations and with the post-restart PID for the + restart persistence case. + +## Accept criteria + +| Roadmap criterion | WP3 closure | +|---|---| +| C2 — disabled survives restart, ensure, and `/api/sync` | Grok OFF is persisted before mutation; the shared Grok sync and Codex sync/direct-refresh owners are gated. Tests activate each path, and live proof checks the real fence after ensure and restart. | +| C3 — absent config changes nothing on upgrade | The absent-map load test proves every integration effective ON; missing keys and explicit true remain ON. Claude's absent-key fallback preserves a legacy explicit OFF. | +| C4 — disable never stops proxy or another transport | No lifecycle path is touched. Journal, teardown, credential cleanup, `/v1/responses`, and `/v1/messages` remain unconditional; the existing Claude transport/model-discovery gates are removed and the real proxy isolation test proves reachability. | + +WP3 is complete only when desired and observed state can disagree honestly. A +successful file removal with no persisted flag is still the shipped Grok bug; a +persisted OFF reported as observed OFF when the file is still present is a new +lie, not a fix. diff --git a/devlog/_plan/260803_codex_desktop_toggle/030_api_keys_row.md b/devlog/_plan/260803_codex_desktop_toggle/030_api_keys_row.md new file mode 100644 index 000000000..11c85c442 --- /dev/null +++ b/devlog/_plan/260803_codex_desktop_toggle/030_api_keys_row.md @@ -0,0 +1,332 @@ +# WP4 — API keys out of the client-card grid + +> "이 api 키는 박스로 두지말고 위쪽에 한 라인으로 빼자" +> +> "don't leave this API key as a box/card - pull it out to the top as a single line/row." + +Independent of WP2/WP3. This is a layout and summary-semantics change; the +`/api/keys` read already returns the only fact this surface needs. + +## IN / OUT + +IN: `gui/src/pages/integrations/overview-clients.ts` (MODIFY), +`gui/src/pages/integrations/IntegrationsOverview.tsx` (MODIFY), +`gui/src/styles-integrations.css` (MODIFY), +`gui/src/i18n/en.ts` (MODIFY), `gui/src/i18n/ko.ts` (MODIFY), +`gui/src/i18n/ja.ts` (MODIFY), `gui/src/i18n/zh.ts` (MODIFY), +`gui/src/i18n/de.ts` (MODIFY), `gui/src/i18n/ru.ts` (MODIFY), +`gui/tests/integrations-overview-rows.test.ts` (MODIFY), +`gui/tests/overview-state-merge.test.ts` (MODIFY), and +`gui/tests/integrations-surfaces.test.tsx` (MODIFY). No implementation file is NEW. + +OUT: `gui/src/pages/integrations/integration-api.ts` — `loadApiKeyCount` +already reduces `/api/keys` to `number | null` (`:283-287`); API and polling +semantics do not change. `gui/src/pages/integrations/IntegrationStateBadge.tsx` +— reuse its `unknown/current/absent` vocabulary. `src/`, `tests/`, and +`docs-site/` — this neither changes the management contract nor setup behavior. + +## Why keys is not a card + +Every grid card represents a client that can be detected, applied, stale, or +unsafe against client-owned configuration. Most carry a switch and a config +path; the exceptions still describe routing or a profile that can drift. + +API keys have neither property. They cannot be installed as a client, toggled, +or drift from a config file. Their entire overview state is the count returned +by `loadApiKeyCount`: zero keys, N issued keys, or an unsettled read, plus a way +to open the keys tab. Painting that credential inventory as one peer in the +client grid is the asymmetry that caused both the orphaned card and the false +summary totals. The type boundary should now say the same thing as the layout. + +## Row model — `overview-clients.ts` + +`OverviewClientId` stops admitting `keys`. `buildOverviewRows` still owns the +normalization of all overview sources, but its result separates the credential +row from client rows instead of returning one mixed array (`:24-59`, `:153-176`, +`:352-397`). + +```diff + export type OverviewClientId = + | "codex" +- | "keys" + | "claude" + | "claudeDesktop" + | "grok" + | FileIntegrationClientId; + ++export interface ApiKeysOverviewRow { ++ hash: "integrations/keys"; ++ labelKey: TKey; ++ state: "unknown" | "absent" | "current"; ++ detailKey: TKey | null; ++ detailVars: Record | null; ++} ++ ++export interface OverviewRows { ++ keysRow: ApiKeysOverviewRow; ++ rows: OverviewRow[]; ++} + + /** API keys are issued or not; there is no config file to drift. */ +-function keysRow(count: number | null): OverviewRow { +- const base = { +- id: "keys" as const, +- hash: "integrations/keys", +- labelKey: "integrations.tab.keys" as TKey, +- toggle: null, +- toggleBlocked: null, +- togglePath: null, +- status: null, +- detail: null, +- }; ++function keysRow(count: number | null): ApiKeysOverviewRow { ++ const base = { ++ hash: "integrations/keys" as const, ++ labelKey: "integrations.tab.keys" as TKey, ++ }; + if (count === null) { +- return { ...base, state: "unknown", installed: false, applied: false, detailKey: null, detailVars: null }; ++ return { ...base, state: "unknown", detailKey: null, detailVars: null }; + } + return { + ...base, + state: count > 0 ? "current" : "absent", +- installed: true, +- applied: count > 0, + detailKey: count > 0 ? "integrations.detail.keyCount" : "integrations.detail.keyNone", + detailVars: count > 0 ? { count: String(count) } : null, + }; + } + +-export function buildOverviewRows(sources: OverviewSources): OverviewRow[] { ++export function buildOverviewRows(sources: OverviewSources): OverviewRows { + const nativeClaude = sources.native?.find(status => status.clientId === "claude"); + const nativeGrok = sources.native?.find(status => status.clientId === "grok"); + const statusByClient = new Map(sources.clients.map(status => [status.clientId, status])); + const rows: OverviewRow[] = [ + codexRow(sources.codex), +- keysRow(sources.keyCount), + claudeRow(sources.claude, nativeClaude, sources.nativeSettled), + claudeDesktopRow(sources.claudeDesktop), + grokRow(sources.grok, nativeGrok, sources.nativeSettled), + ]; + // Existing file-client loop stays byte-for-byte unchanged. +- return rows; ++ return { keysRow: keysRow(sources.keyCount), rows }; + } +``` + +Also change the file docstring's “Five more sources join the grid” to “Four +more client sources join the grid; API keys return separately,” change +`OverviewRow`'s “other five” detail comment to “other four,” and replace the +catalog-order comment at `:352-356` with the actual client order: Codex, Claude, +Claude Desktop, Grok, then the file clients. Comments must not keep claiming +that keys joins the grid after the type no longer permits it. + +`ApiKeysOverviewRow` deliberately has no `installed`, `applied`, `toggle`, or +`status`. Re-adding one of those fields would recreate the semantic leak this +phase removes. `gui/tests/overview-state-merge.test.ts` changes only its helper +to search `buildOverviewRows(...).rows`; native-state behavior is otherwise +untouched. + +## Render — `IntegrationsOverview.tsx` + +The row goes immediately **below the summary strip** and above onboarding, +notices, and the grid (`:433-498`). The summary is the page-level aggregate; +the keys row is one credential surface with one action. Putting keys above the +summary would promote one surface over the aggregate, while merging it into the +strip would make “Manage keys” look like a bulk-status control beside “Disable +all.” Below preserves aggregate → individual → client catalog hierarchy. + +```diff + import { + buildOverviewRows, + countOverviewRows, ++ type ApiKeysOverviewRow, + type OverviewRow, + } from "./overview-clients"; + ++/** ++ * Credentials are one explicit action, not a clickable client card. ++ * ++ * Do not stretch the title over this row. The card overlay exists because a ++ * card also contains a switch; this row has no nested-control problem to ++ * solve. A plain Manage keys button is the one tab stop, so its visible label, ++ * focus ring, Enter, and Space behavior all come from a native button. ++ */ ++function ApiKeysRow({ row }: { row: ApiKeysOverviewRow }) { ++ const t = useT(); ++ const detail = row.detailKey ? t(row.detailKey, row.detailVars ?? undefined) : null; ++ return ( ++
++
++

{t(row.labelKey)}

++ {detail &&

{detail}

} ++
++ ++ ++
++ ); ++} + +- const rows = buildOverviewRows({ ++ const { keysRow, rows } = buildOverviewRows({ + clients, + clientsSettled, + codex: codexResource.state.data ?? null, + keyCount: keysResource.state.data ?? null, + claude: claudeResource.state.data ?? null, + claudeDesktop: claudeDesktopResource.state.data ?? null, + grok: grokResource.state.data ?? null, + native, + nativeSettled, + }); + const counts = countOverviewRows(rows); + + + ++ ++ +

{t("integrations.onboarding")}

+ + {/* +- The grid used to disappear entirely when no FILE client was installed, +- which now means hiding Codex, API keys, Claude and Grok because the ++ The grid used to disappear entirely when no FILE client was installed, ++ which means hiding Codex, Claude and Grok because the + user has not installed OpenCode. The "nothing detected" panel is about + the file clients specifically, so it sits BELOW the grid and says so + instead of replacing everything. + */} +``` + +The row itself has no click handler and no `tabIndex`. Its heading, count, and +badge remain readable in document order; only the native action button enters +the tab sequence, once, at the same visual position. This does not copy the +card's `::after` overlay, does not add a second `tabIndex={-1}` action, and +cannot swallow a future control through stacking order. + +## CSS — `styles-integrations.css` + +Insert after the summary rules (`:11-14`), before `.integration-cards`. This is +a full-width divider row, not `.integration-card` with `grid-column: 1 / -1`: +it has no raised background, surrounding border, radius, or card hover state. + +```diff + .integration-summary-label { font-size: var(--text-caption); color: var(--muted); } + .integration-summary .btn { margin-left: auto; } + ++/* API keys are credential inventory, not a client card. A bottom rule keeps ++ the full-width row in the page flow without making it an orphaned wide card. */ ++.integration-api-keys-row { display: flex; align-items: center; flex-wrap: wrap; gap: 8px 12px; min-height: 44px; padding: 0 0 14px; border-bottom: 1px solid var(--border); margin-bottom: 14px; } ++.integration-api-keys-copy { display: flex; align-items: baseline; flex: 1 1 220px; min-width: 0; flex-wrap: wrap; gap: 4px 10px; } ++.integration-api-keys-copy h4, ++.integration-api-keys-copy .integration-meta { margin: 0; } ++.integration-api-keys-row .btn { margin-inline-start: auto; flex: 0 0 auto; } +``` + +`flex-wrap` is intentional: at 320-390 px, long German/Russian action copy may +move beside or below the badge instead of clipping. It remains one credential +row surface. At normal widths, `flex: 1 1 220px` keeps title/count together on +the left and badge/action on the right. The existing global `:focus-visible` +rule owns the button ring; the rendered keyboard check below proves it. + +## Counts + +Keys leave all four totals. `countOverviewRows(rows)` remains unchanged because +its input is now client rows only. + +- `detected`: a keys API endpoint exists on every dashboard; that is not a + detected client. Counting it makes the total increase even with zero keys. +- `applied`: issuing a credential is not applying opencodex to a client. Two + keys are inventory, not two applied integrations — and the old code counted + either number as exactly one anyway. +- `stale`: keys have no config file and cannot enter this state. +- `unknown`: a failed `/api/keys` read remains visible as the row's Checking + badge, but must not inflate the number of client integrations whose state is + unknown. + +This changes current totals by at most one. That is not a regression hidden by +the layout move; it corrects what the labels already claim to measure. The +“Disable all” button remains file-client-only (`IntegrationsOverview.tsx:293-350`) +and is unaffected. + +## i18n + +The existing title/count keys stay unchanged. Add one action key to every +locale; hardcoding “Manage keys” in JSX is forbidden by `gui/AGENTS.md`. + +| Key | File | Value | +|---|---|---| +| `integrations.action.manageKeys` | `gui/src/i18n/en.ts` | `Manage keys` | +| `integrations.action.manageKeys` | `gui/src/i18n/ko.ts` | `키 관리` | +| `integrations.action.manageKeys` | `gui/src/i18n/ja.ts` | `キーを管理` | +| `integrations.action.manageKeys` | `gui/src/i18n/zh.ts` | `管理密钥` | +| `integrations.action.manageKeys` | `gui/src/i18n/de.ts` | `Schlüssel verwalten` | +| `integrations.action.manageKeys` | `gui/src/i18n/ru.ts` | `Управлять ключами` | + +Reused in all six locales: `integrations.tab.keys`, +`integrations.detail.keyCount`, `integrations.detail.keyNone`, and the three +state labels selected by `IntegrationStateBadge`. + +## Test plan + +MODIFY `gui/tests/integrations-overview-rows.test.ts`: + +1. Destructure `{ keysRow, rows }`; assert null count gives a keys `unknown` + row while client `unknown` count is 4, not 5. +2. Assert zero gives `absent` + `keyNone`, and two gives `current` + + `{ count: "2" }`. With Codex, Claude, Desktop, Grok, and one file client + applied, changing key count through null/zero/two leaves `applied === 5`. +3. Update settled lengths from 5 to 4 and unsettled lengths from 11 to 10; + assert no member of `rows` has id `keys`. +4. Keep every existing Codex/Desktop/file-client mapping case against `rows`. + +MODIFY `gui/tests/overview-state-merge.test.ts`: its `row()` helper reads the +`.rows` member. No assertion changes. + +MODIFY `gui/tests/integrations-surfaces.test.tsx` with a mounted overview case: + +1. `[data-client="keys"]` exists, but + `.integration-cards [data-client="keys"]` is null. +2. DOM order is `.integration-summary` → keys row → `.integration-cards`. +3. The row renders issued/none/unknown copy from `/api/keys`; an API-key failure + does not change the client summary's unknown total. +4. Exactly one button exists inside the row, has `tabIndex === 0`, shows the + localized action, and clicking it navigates to `#integrations/keys`. + +## Verification + +1. `cd gui && bun test tests/integrations-overview-rows.test.ts tests/overview-state-merge.test.ts tests/integrations-surfaces.test.tsx` +2. `cd gui && bun test tests && bun run lint && bun run lint:i18n && bun run build` +3. `bun run typecheck`, `bun run test`, `bun run privacy:scan` +4. Start the real dashboard and open `http://localhost:10100/#integrations` at + 1280 px and 390 px. Inspect the screenshot and live DOM: one keys row below + the summary, no keys descendant inside `.integration-cards`, no wide-card + chrome, and the grid begins with Codex. At 390 px, switch through all six + locales and confirm the action text wraps when needed rather than clipping. +5. Keyboard-tab to “Manage keys”; observe a visible focus ring, activate with + Enter and Space, and confirm `#integrations/keys` opens. Repeat with zero + keys and a failed keys read to observe absent and unknown states. + +Step 4 is **C-RENDER-GROUNDING-01**. The change is not verified by typecheck, +DOM assertions, or a produced-but-unread screenshot: it must be OBSERVED +rendered at `localhost:10100`, and any layout defect must be fixed and observed +again. + +## Accept criteria + +- C8 — API keys render as one full-width row above `.integration-cards`, below + the aggregate summary, observed rendered at `localhost:10100`. +- The keys row is absent from the card grid and from all client summary totals; + its own issued/none/unknown state remains visible. +- One native button is the complete keyboard path to the keys tab; no stretched + card overlay or duplicate tab stop is introduced. +- C9's GUI tests, lint, i18n lint, build, repository typecheck/test, and privacy + scan pass before WP4 is called done. diff --git a/devlog/_plan/260803_codex_desktop_toggle/040_codex_toggle.md b/devlog/_plan/260803_codex_desktop_toggle/040_codex_toggle.md new file mode 100644 index 000000000..06f14751f --- /dev/null +++ b/devlog/_plan/260803_codex_desktop_toggle/040_codex_toggle.md @@ -0,0 +1,775 @@ +# WP5 — the Codex CLI toggle, with artifact-level restore truth + +Research: `001_native_restore_thesis.md`. Read it first; this doc is the diff. + +The failure this phase closes is concrete: `restoreNativeCodex()` can leave routed +threads hidden when the history database is locked, yet return `success: true` +because that boolean is copied from config restore alone +(`src/codex/inject.ts:783-794`). Disable and enable already exist as +`restoreNativeCodex()` and `syncModelsToCodex(port)` while the proxy keeps serving +(`src/cli/index.ts:745-768`); WP3 already adds the durable, default-ON +`clientIntegrations.codex` intent (`003_durable_desired_state.md:87-115`). This +phase adds the missing artifact-level result, classifies the held-history failure, +registers Codex in the existing native-integration route family, and gives the +overview card an honest switch. It adds no operation journal or lifecycle engine. + +## IN / OUT + +IN: + +- `src/codex/history-provider.ts` (MODIFY) — retain the exhausted retry's + classified reason instead of reducing it to `null`. +- `src/codex/inject.ts` (MODIFY) — return config/catalog/history results and make + aggregate success mean all required artifacts succeeded. +- `src/server/management/context.ts` (MODIFY) — add Codex mutation seams so route + tests cannot touch the developer's real Codex home. +- `src/server/management/native-integration-routes.ts` (MODIFY) — add Codex to + GET and `PUT /api/native-integrations/codex`, persisting WP3 intent before the + client mutation. +- `gui/src/pages/integrations/overview-clients.ts` (MODIFY), + `gui/src/pages/integrations/IntegrationsOverview.tsx` (MODIFY), + `gui/src/pages/integrations/native-api.ts` (MODIFY), and + `gui/src/pages/integrations/refusal-copy.ts` (MODIFY) — wire the card, dialog, + structured native API vocabulary, and localized refusal copy. +- `gui/src/i18n/en.ts`, `de.ts`, `ja.ts`, `ko.ts`, `ru.ts`, `zh.ts` (MODIFY) — + source copy plus all five translations required by `gui/AGENTS.md:13-19`. +- `tests/native-codex-toggle.test.ts` (NEW), + `tests/codex-history-provider.test.ts` (MODIFY), + `tests/codex-journal.test.ts` (MODIFY), + `gui/tests/integrations-overview-rows.test.ts` (MODIFY), + `gui/tests/overview-state-merge.test.ts` (MODIFY), and + `gui/tests/consequence-dialog.test.tsx` (MODIFY). + +OUT: + +- `src/codex/sync.ts` — enable delegates to the existing full catalog refresh + + injection path at lines 83-110; changing it is not needed. +- `src/cli/index.ts`, `src/server/management-api.ts`, and `src/service.ts` — their + existing `restoreNativeCodex().success` checks become more truthful through the + widened return type; no lifecycle caller needs a new state machine. +- `/v1/responses` and every data-plane router — a client flag gates automatic + Codex config writes, never the shared transport (`003_durable_desired_state.md:117-130`). +- `src/integrations/writer.ts`, operation records, snapshots, undo routes, and the + superseded `src/integrations/native/codex.ts` idea — turning the switch back on + is `syncModelsToCodex`, not replay. +- `gui/dist`, docs publishing, releases, deployment, and any live proxy mutation. + +## The structured result + +MODIFY `src/codex/history-provider.ts` at the current +`CodexHistorySyncResult` (`:162-168`): + +```ts +export type CodexHistoryFailureReason = "busy" | "permission"; + +export interface CodexHistorySyncResult { + rows: number; + files: number; + ejectedRows?: number; + /** The mutation was skipped after every retry; zero rows is not a successful no-op. */ + failed?: true; + /** + * Why the retry budget was exhausted. `failed` alone caused the Codex-toggle + * incident: SQLITE_BUSY and EACCES both became the same boolean, then + * restoreNativeCodex converted that boolean to prose while keeping + * `success: true`. Callers need this discriminator to recommend a retry only + * for contention and to stop treating an ACL failure as a lock that will pass. + */ + failureReason?: CodexHistoryFailureReason; +} +``` + +MODIFY `src/codex/inject.ts` above `restoreNativeCodex()` (currently line 764): + +```ts +export type CodexRestoreArtifactState = "ok" | "skipped" | "failed"; + +export interface CodexRestoreConfigResult { + state: CodexRestoreArtifactState; + changed: boolean; + action: "journal-restored" | "owned-fields-stripped" | "external-provider-preserved" | "failed"; + message: string; +} + +export interface CodexRestoreCatalogResult { + state: CodexRestoreArtifactState; + changed: boolean; + removed: number; + kept: number; + path: string | null; + message: string; +} + +export interface CodexRestoreHistoryResult { + state: CodexRestoreArtifactState; + changed: boolean; + reason?: CodexHistoryFailureReason; + rows: number; + files: number; + ejectedRows: number; + message: string; +} + +export interface CodexNativeRestoreResult { + /** + * True only when every artifact required for a native Codex view succeeded. + * The former boolean described config only, so the held-history incident + * returned true while routed threads remained tagged opencodex and invisible. + * Consumers must inspect `artifacts` for the failed boundary; they must never + * recover structure by parsing `message`. + */ + success: boolean; + message: string; + externalProvider?: string; + artifacts: { + config: CodexRestoreConfigResult; + catalog: CodexRestoreCatalogResult; + history: CodexRestoreHistoryResult; + }; +} +``` + +`restoreNativeCodex(): CodexNativeRestoreResult` keeps the existing operation +order but catches and records each artifact boundary separately. Config uses +`restoreJournalState()` and then the existing `removeCodexConfig()` fallback +(`src/codex/inject.ts:770-774`); catalog delegates once to +`restoreCodexCatalog()` (`src/codex/catalog/sync.ts:572-597`); history delegates +once to `syncCodexHistoryProvider("openai", ...)` (`src/codex/inject.ts:775-783`). +Aggregate `success` is `config.state !== "failed" && catalog.state !== "failed" +&& history.state !== "failed"`. Existing callers can keep reading `.success` and +`.message`, but a history failure now makes `.success === false`. + +The external-provider courtesy is a successful skip, not a fake restore. When +`currentExternalCodexModelProvider()` returns (currently lines 765-768), remove +only the stale journal and return all three artifacts as `state: "skipped"`, +config action `external-provider-preserved`, and `externalProvider`. No catalog +or history function runs. That preserves the existing behavior for `custom` +while giving the card a stable fact to show. + +## The route + +Method and path: `PUT /api/native-integrations/codex` with +`Content-Type: application/json`. + +Request: + +```ts +{ enabled: boolean } +``` + +Success (`200`), using the existing envelope and adding optional Codex detail: + +```ts +{ + ok: true; + clientId: "codex"; + changed: boolean; + state: "absent" | "current" | "unsafe"; + message: string; + reason?: "external_provider_preserved" | "catalog_warning"; + artifacts?: CodexNativeRestoreResult["artifacts"]; +} +``` + +Disable persists `clientIntegrations.codex = false` first, then checks teardown +ownership and calls `restoreNativeCodex()`. That order is intentional: WP3 says +desired OFF survives an ownership refusal or drift so a later automatic apply +cannot reverse the user's request (`003_durable_desired_state.md:112-115`). Use a +cloned config for persistence, then update the request's in-memory config only +after persistence succeeds; a failed config lock must not create an in-memory-only +OFF. Enable likewise persists `true`, resolves the running listener from +`readRuntimePort(process.pid)` with request/config fallback, and calls +`syncModelsToCodex(port, config, null)`. Bare `injectCodexConfig()` is forbidden: +it does not rebuild routed catalog rows (`src/codex/sync.ts:83-110`). + +GET `/api/native-integrations` adds a Codex row. Its `state` is observed routing +from `getCodexRoutingKind()` (`src/codex/inject.ts:255-273`), not merely desired +intent: `opencodex-local` is `current`, `native` is `absent`, and +`custom-local|custom-remote|unknown` is `unsafe` unless an external +`model_provider` explains it, in which case it is `absent` with +`reason: "external_provider_preserved"` and a message naming that provider. +`disableBlocked` carries `home_mismatch` only while teardown would touch our +artifacts. Consume WP3's `clientIntegrationEnabled()` and +`setClientIntegrationEnabled()` owners (`020_desired_state.md:149-183`); no WP5 +caller open-codes the map's defaulting rule. + +Every refusal uses the existing +`refusal(status, clientId, reason, message)` function unchanged +(`src/server/management/native-integration-routes.ts:76-87`): + +| HTTP | reason | Trigger | Observable state | +|---|---|---|---| +| 409 | `config_busy` | WP3 desired-state persistence loses a real `SQLITE_BUSY` lock race | No durable intent or Codex artifact changed; retry is correct | +| 409 | `home_mismatch` | disable sees an installed service owned by another Codex/OpenCodex home | Desired OFF is durable; Codex artifacts are untouched | +| 409 | `history_busy` | config and catalog restored, history retries exhaust on busy/locked contention | Desired OFF is durable; native routing is active, but routed threads remain hidden until retry | +| 500 | `history_permission` | config and catalog restored, history fails with `EPERM`/`EACCES` | Desired OFF is durable; user must fix permissions, not wait | +| 500 | `write_failed` | desired-state persistence cannot open its lock, config/catalog restore fails, or enable sync returns `ok: false` | Message names the failed boundary; no retry promise unless the cause is known | + +Malformed JSON and non-boolean `enabled` retain the route family's existing +plain `400` responses (`native-integration-routes.ts:206-215,381-391`); these are +request errors, not native refusals. An external provider is not a refusal: the +desired flag changes and the response is `200`, reason +`external_provider_preserved`, while the config/catalog/history stay untouched. + +Refusal/failure response (the existing envelope, unchanged): + +```ts +{ + error: "native integration change refused" | "native integration change failed"; + code: "native_integration_refused" | "native_integration_failed"; + clientId: "codex"; + reason: "config_busy" | "home_mismatch" | "history_busy" + | "history_permission" | "write_failed"; + message: string; +} +``` + +Invalid bodies remain `{ error: "invalid JSON body" }` or +`{ error: "enabled must be a boolean" }` with HTTP 400. + +MODIFY `src/server/management/native-integration-routes.ts`: + +```diff ++import { ++ currentExternalCodexModelProvider, ++ getCodexConfigPath, ++ getCodexRoutingKind, ++ restoreNativeCodex, ++ type CodexNativeRestoreResult, ++} from "../../codex/inject"; ++import { syncModelsToCodex } from "../../codex/sync"; +-import { readRuntimePort, saveConfigPreservingClaudeCode } from "../../config"; ++import { ++ clientIntegrationEnabled, readRuntimePort, saveConfigPreservingClaudeCode, ++ setClientIntegrationEnabled, ++} from "../../config"; +@@ +-export type NativeIntegrationClientId = "claude" | "grok"; ++export type NativeIntegrationClientId = "codex" | "claude" | "grok"; +@@ + | "config_busy" ++ | "history_busy" ++ | "history_permission" + | "write_failed"; +@@ + export interface NativeStatus { +@@ + disableBlocked: { reason: NativeRefusalReason; message: string } | null; ++ reason?: "external_provider_preserved"; ++ externalProvider?: string; + } +@@ +- reason?: string; ++ reason?: "non_loopback_removed" | "non_loopback_superseded" ++ | "external_provider_preserved" | "catalog_warning"; ++ externalProvider?: string; ++ artifacts?: CodexNativeRestoreResult["artifacts"]; +@@ ++function codexStatus(ctx: ManagementContext): NativeStatus { ++ const { deps } = ctx; ++ const externalProvider = (deps.currentExternalCodexModelProvider ++ ?? currentExternalCodexModelProvider)(); ++ const routing = (deps.getCodexRoutingKind ?? getCodexRoutingKind)(); ++ const owned = routing === "opencodex-local" ? assertNativeTeardownOwned() : null; ++ return { ++ clientId: "codex", ++ state: externalProvider ? "absent" ++ : routing === "opencodex-local" ? "current" ++ : routing === "native" ? "absent" : "unsafe", ++ installed: true, ++ configPath: getCodexConfigPath(), ++ disableBlocked: owned && !owned.ok ++ ? { reason: "home_mismatch", message: owned.message } : null, ++ ...(externalProvider ? { ++ reason: "external_provider_preserved" as const, externalProvider, ++ } : {}), ++ }; ++} +@@ +- clients: [claudeStatus(config, getConfigPath()), grokStatus()], ++ clients: [codexStatus(ctx), claudeStatus(config, getConfigPath()), grokStatus()], +@@ ++ if (url.pathname === "/api/native-integrations/codex" && req.method === "PUT") { ++ let body: { enabled?: unknown }; ++ try { ++ body = await readManagementJsonBody(req); ++ } catch (error) { ++ rethrowManagementBodyTooLarge(error); ++ return jsonResponse({ error: "invalid JSON body" }, 400); ++ } ++ if (typeof body.enabled !== "boolean") { ++ return jsonResponse({ error: "enabled must be a boolean" }, 400); ++ } ++ const enabled = body.enabled; ++ ++ const desiredChanged = clientIntegrationEnabled(config, "codex") !== enabled; ++ if (desiredChanged) { ++ const next = { ...config, clientIntegrations: { ...config.clientIntegrations } }; ++ setClientIntegrationEnabled(next, "codex", enabled); ++ const persist = deps.saveConfigPreservingClaudeCode ?? saveConfigPreservingClaudeCode; ++ try { ++ persist(next); ++ } catch (error) { ++ if (!isConfigLockError(error)) throw error; ++ return isLockContention(error) ++ ? refusal(409, "codex", "config_busy", ++ "Another process is saving the configuration right now. Try again in a moment.") ++ : refusal(500, "codex", "write_failed", ++ `The configuration lock could not be acquired: ${error instanceof Error ? error.message : String(error)}`); ++ } ++ // Change the request-scoped object only after durable persistence succeeds. ++ setClientIntegrationEnabled(config, "codex", enabled); ++ } ++ ++ if (!enabled) { ++ const owned = assertNativeTeardownOwned(); ++ if (!owned.ok) return refusal(409, "codex", "home_mismatch", owned.message); ++ const restore = (deps.restoreNativeCodex ?? restoreNativeCodex)(); ++ if (!restore.success) { ++ const history = restore.artifacts.history; ++ const otherArtifactsOk = restore.artifacts.config.state !== "failed" ++ && restore.artifacts.catalog.state !== "failed"; ++ if (otherArtifactsOk && history.state === "failed" && history.reason === "busy") { ++ return refusal(409, "codex", "history_busy", history.message); ++ } ++ if (otherArtifactsOk && history.state === "failed" && history.reason === "permission") { ++ return refusal(500, "codex", "history_permission", history.message); ++ } ++ return refusal(500, "codex", "write_failed", restore.message); ++ } ++ return jsonResponse({ ++ ok: true, clientId: "codex", ++ changed: desiredChanged || Object.values(restore.artifacts).some(a => a.changed), ++ state: "absent", ++ message: restore.message, artifacts: restore.artifacts, ++ ...(restore.externalProvider ? { ++ reason: "external_provider_preserved" as const, ++ externalProvider: restore.externalProvider, ++ } : {}), ++ } satisfies NativeToggleEnvelope); ++ } ++ ++ const externalProvider = (deps.currentExternalCodexModelProvider ++ ?? currentExternalCodexModelProvider)(); ++ const runtime = (deps.readRuntimePort ?? readRuntimePort)(process.pid); ++ const port = runtime?.port ?? (Number(url.port) || config.port); ++ const synced = await (deps.syncModelsToCodex ?? syncModelsToCodex)(port, config, null); ++ if (!synced.ok) return refusal(500, "codex", "write_failed", synced.message); ++ return jsonResponse({ ++ ok: true, clientId: "codex", changed: desiredChanged || !externalProvider, ++ state: externalProvider ? "absent" : "current", message: synced.message, ++ ...(externalProvider ? { ++ reason: "external_provider_preserved" as const, externalProvider, ++ } : synced.warning ? { reason: "catalog_warning" as const } : {}), ++ } satisfies NativeToggleEnvelope); ++ } +``` + +MODIFY `src/server/management/context.ts` with typed optional seams for +`restoreNativeCodex`, `syncModelsToCodex`, `getCodexRoutingKind`, and +`currentExternalCodexModelProvider`. The production defaults are the real +functions; `tests/native-codex-toggle.test.ts` supplies deterministic results. +This follows the existing reason for `saveConfigPreservingClaudeCode` and Grok's +writer/catalog seams (`context.ts:12-37`). + +## History-lock classification + +The current low-level code cannot distinguish the two outcomes after retry. It +recognizes `SQLITE_BUSY`, `SQLITE_LOCKED`, `EBUSY`, `EPERM`, and `EACCES` in one +predicate (`src/codex/history-provider.ts:511-523`), then `withHistoryRetry()` +discards the final error and returns `null` (`:536-548`). The caller therefore has +no code left to inspect at line 577. Saying the GUI can classify this today would +be false. + +The minimal change is one classifier and one internal discriminated retry helper: + +```ts +export function classifyRecoverableHistoryError( + error: unknown, +): CodexHistoryFailureReason | null { + const code = typeof error === "object" && error && "code" in error + ? String((error as { code?: unknown }).code) : ""; + const message = error instanceof Error ? error.message.toLowerCase() : String(error).toLowerCase(); + if (["SQLITE_BUSY", "SQLITE_LOCKED", "EBUSY"].includes(code) + || message.includes("database is locked") + || message.includes("database is busy") + || message.includes("resource busy")) return "busy"; + if (["EPERM", "EACCES"].includes(code) + || message.includes("operation not permitted") + || message.includes("permission denied")) return "permission"; + return null; +} +``` + +`isRecoverableHistoryError(error)` becomes +`classifyRecoverableHistoryError(error) !== null`, preserving its public boolean +contract and existing tests. New internal `withHistoryRetryResult()` returns +`{ ok: true, value } | { ok: false, reason }`; exported `withHistoryRetry()` wraps +it and still returns `T | null`, preserving callers/tests at +`tests/codex-history-provider.test.ts:309-361`. `syncCodexHistoryProvider()` uses +the detailed helper and emits `{ rows: 0, files: 0, failed: true, +failureReason: retry.reason }`. Neither `restoreNativeCodex`, the route, nor the +GUI parses error prose. + +## GUI + +`codexRow` currently hard-codes `toggle: null` and ignores the native family +(`gui/src/pages/integrations/overview-clients.ts:118-150`). Make its status merge +match Claude/Grok: find `nativeCodex`, wait for `nativeSettled`, set +`toggle: "codex"`, `toggleBlocked`, and `togglePath` from that row, and keep the +badge based on observed `native.state`. When `native.reason` is +`external_provider_preserved`, use the localized detail key with the structured +provider name so the card says another provider owns routing instead of saying +opencodex is applied. + +MODIFY `gui/src/pages/integrations/overview-clients.ts`: + +```diff +-function codexRow(payload: CodexRoutingPayload | null): OverviewRow { ++function codexRow( ++ payload: CodexRoutingPayload | null, ++ native: NativeStatus | undefined, ++ nativeSettled: boolean, ++): OverviewRow { + const base = { +@@ +- toggle: null, +- toggleBlocked: null, +- togglePath: null, ++ toggle: "codex" as const, ++ toggleBlocked: native?.disableBlocked ?? null, ++ togglePath: native?.configPath ?? null, +@@ ++ if (!nativeSettled) return { ...base, state: "unknown", installed: false, applied: false, detailKey: null }; ++ if (!native) return { ...base, toggle: null, state: "unknown", installed: false, applied: false, detailKey: null }; ++ if (native.reason === "external_provider_preserved") { ++ return { ...base, state: native.state, installed: true, applied: false, ++ detail: null, detailKey: "integrations.native.msg.codexExternalProvider", ++ detailVars: { provider: native.externalProvider ?? "" } }; ++ } +@@ + export function buildOverviewRows(sources: OverviewSources): OverviewRow[] { ++ const nativeCodex = sources.native?.find(status => status.clientId === "codex"); +@@ +- codexRow(sources.codex), ++ codexRow(sources.codex, nativeCodex, sources.nativeSettled), +``` + +`IntegrationsOverview.tsx` adds `CODEX_DISABLE_COPY`, admits `codex` anywhere the +native toggle union is narrowed, refreshes `codexResource` after the mutation, +and chooses copy by `pendingToggle.id` instead of always rendering Grok's copy. + +```diff ++const CODEX_DISABLE_COPY: ConsequenceCopy = { ++ titleKey: "integrations.dialog.codex.title", ++ changesKey: "integrations.dialog.codex.changes", ++ breakageKey: "integrations.dialog.codex.breakage", ++ undoKey: "integrations.dialog.codex.undo", ++ sideEffectKey: "integrations.dialog.codex.sideEffect", ++ confirmKey: "integrations.dialog.codex.confirm", ++}; +@@ +- } else if (row.toggle === "claude" || row.toggle === "grok") { ++ } else if (row.toggle === "codex" || row.toggle === "claude" || row.toggle === "grok") { +@@ +- if (row.status || next || row.id === "claude" || row.toggle === null) { ++ if (row.status || next || row.id === "claude" || row.toggle === null) { + void toggleCard(row, next); + return; + } +- // Grok disable is the only native action that edits another program's file. ++ // Codex and Grok disable both alter another client's on-disk state and earn ++ // the consequence gate; Claude changes only our own flag and stays immediate. +@@ +- copy={{ ...GROK_DISABLE_COPY, vars: { path: pendingToggle.togglePath ?? "" } }} ++ copy={{ ++ ...(pendingToggle.id === "codex" ? CODEX_DISABLE_COPY : GROK_DISABLE_COPY), ++ vars: { path: pendingToggle.togglePath ?? "" }, ++ }} +``` + +The same `IntegrationsOverview.tsx` diff handles the Codex-only success caveat and +refreshes both observed sources: + +```diff ++ } else if (result.reason === "external_provider_preserved") { ++ setCardResult(row.id, { tone: "ok", text: t( ++ "integrations.native.msg.codexExternalProvider", ++ { provider: result.externalProvider ?? "" }, ++ ) }); ++ } +@@ + const refreshNativeDetails = () => { + nativeResource.refresh(); ++ codexResource.refresh(); + claudeResource.refresh(); +``` + +MODIFY `gui/src/pages/integrations/native-api.ts`; both runtime allowlists must +widen with the TypeScript unions, or a valid server refusal will be downgraded to +an opaque `NativeApiError` (`native-api.ts:51-75`): + +```diff +-export type NativeIntegrationClientId = "claude" | "grok"; ++export type NativeIntegrationClientId = "codex" | "claude" | "grok"; +@@ + | "config_busy" ++ | "history_busy" ++ | "history_permission" + | "write_failed"; +@@ + export interface NativeStatus { +@@ + disableBlocked: { reason: NativeRefusalReason; message: string } | null; ++ reason?: "external_provider_preserved"; ++ externalProvider?: string; +@@ + export interface NativeToggleEnvelope { +@@ + reason?: string; ++ externalProvider?: string; ++ artifacts?: CodexNativeRestoreArtifacts; +@@ +-const NATIVE_CLIENTS = new Set(["claude", "grok"]); ++const NATIVE_CLIENTS = new Set(["codex", "claude", "grok"]); +@@ + "config_busy", ++ "history_busy", ++ "history_permission", + "write_failed", +``` + +Define the GUI's structural `CodexNativeRestoreArtifacts` beside the envelope; +do not import a runtime type across the `src/`/`gui/` package boundary. The shape +matches the three server artifacts exactly and keeps `reason` typed as +`"busy" | "permission"` on history. + +```ts +export interface CodexNativeRestoreArtifacts { + config: { + state: "ok" | "skipped" | "failed"; + changed: boolean; + action: "journal-restored" | "owned-fields-stripped" + | "external-provider-preserved" | "failed"; + message: string; + }; + catalog: { + state: "ok" | "skipped" | "failed"; + changed: boolean; + removed: number; + kept: number; + path: string | null; + message: string; + }; + history: { + state: "ok" | "skipped" | "failed"; + changed: boolean; + reason?: "busy" | "permission"; + rows: number; + files: number; + ejectedRows: number; + message: string; + }; +} +``` + +MODIFY `gui/src/pages/integrations/refusal-copy.ts` at the existing native reason +switch (`:56-71`): + +```diff + if (refusal.reason === "not_installed") return t("integrations.native.error.notInstalled"); + if (refusal.reason === "config_busy") return t("integrations.native.error.configBusy"); ++ if (refusal.reason === "history_busy") return t("integrations.native.error.historyBusy"); ++ if (refusal.reason === "history_permission") return t("integrations.native.error.historyPermission"); + return refusal.message || t("integrations.error.generic"); +``` + +The English source text is exact: + +- `integrations.dialog.codex.title` — `Disable the Codex integration?` +- `integrations.dialog.codex.changes` — `opencodex will remove its routing from {path}, remove its generated profile, restore the native model catalog, and retag resumable threads for native Codex.` +- `integrations.dialog.codex.breakage` — `Plain codex will connect directly to OpenAI, and models routed from other providers will disappear from Codex. The proxy and /v1/responses stay running for other clients.` +- `integrations.dialog.codex.undo` — `Turning this back on rebuilds the routed catalog from the models available then and injects Codex again. Resume history is made usable in the matching direction, but its files are not restored byte for byte.` +- `integrations.dialog.codex.sideEffect` — `If you selected a routed root model after opencodex injected the config, disabling removes that model selection and turning the integration back on cannot reconstruct it; select the model again. If an external model_provider owns Codex, opencodex removes only its stale journal and leaves the config, catalog, and history unchanged.` +- `integrations.dialog.codex.confirm` — `Disable` + +The model-selection sentence is deliberately stronger than the superseded copy. +The fallback strip removes any root slash-qualified `model = "provider/slug"` +after post-injection drift (`src/codex/inject.ts:315-327,688-705`), and no restore +record exists from which enable could rebuild it. Resume history is reversible in +provider meaning but appends/patches metadata rather than restoring bytes +(`src/codex/history-provider.ts:52-90,480-508`). + +Refusal/success copy in `gui/src/pages/integrations/refusal-copy.ts` and the native +result branch: + +- `history_busy` — `Codex routing is disabled, but routed threads are still hidden because Codex or an IDE is holding the history database. Close Codex and the IDE, then turn the integration off again.` +- `history_permission` — `Codex routing is disabled, but opencodex does not have permission to retag the history database. Fix the Codex history file permissions, then turn the integration off again.` +- `external_provider_preserved` — `Codex is using the external model provider {provider}. opencodex left its config, catalog, and history unchanged.` + +`history_busy` and `history_permission` replace server prose by reason, just as +`orphaned_marker` and `config_busy` do today +(`gui/src/pages/integrations/refusal-copy.ts:56-71`). `write_failed` continues to +show the server's boundary-specific message. A refusal is rendered in the card's +notice area after the dialog closes; it never opens a second modal, matching the +established direction (`../260803_integrations_toggle_all/002_consequence_dialog_ux.md:124-149`). + +## i18n + +Add these exact keys to all six locale files: + +```text +integrations.dialog.codex.title +integrations.dialog.codex.changes +integrations.dialog.codex.breakage +integrations.dialog.codex.undo +integrations.dialog.codex.sideEffect +integrations.dialog.codex.confirm +integrations.native.error.historyBusy +integrations.native.error.historyPermission +integrations.native.msg.codexExternalProvider +``` + +`gui/src/i18n/en.ts` is the English source and `TKey` authority. Add matching +translations to `de.ts`, `ja.ts`, `ko.ts`, `ru.ts`, and `zh.ts`; do not hardcode +the dialog or refusal text in JSX (`gui/AGENTS.md:13-30`). `{path}` appears in +`changes`; `{provider}` appears in `codexExternalProvider`. + +## Test plan + +`tests/codex-history-provider.test.ts`: + +1. `SQLITE_BUSY`, `SQLITE_LOCKED`, `EBUSY`, and the existing lock/busy message + fallbacks classify as `busy`. +2. `EPERM`, `EACCES`, `operation not permitted`, and `permission denied` classify + as `permission`. +3. Corruption/programming errors classify `null` and still throw. +4. Exhausted detailed retry preserves the last reason; exported + `withHistoryRetry()` still returns `null` for compatibility. +5. `syncCodexHistoryProvider()` against a held real `BEGIN IMMEDIATE` transaction + returns `failed: true, failureReason: "busy"`; an ACL/code fixture returns + `permission`. The real lock case is the activation proof, not only a mocked object. + +`tests/codex-journal.test.ts`: + +1. A complete restore reports all three artifact objects and `success: true`. +2. A config failure, catalog failure, and history failure each name only that + boundary and make aggregate success false. +3. External `model_provider = "custom"` removes only the journal, invokes neither + catalog nor history mutation, and returns three structured skips. +4. A drifted post-injection root `model = "provider/slug"` is removed; reinjection + does not recreate it. This pins the dialog's destructive sentence. + +`tests/native-codex-toggle.test.ts`: + +1. GET includes `clientId: "codex"` and reports observed native/current/unsafe + routing without reading desired intent as disk truth. +2. Missing WP3 flag defaults ON; explicit false survives a fresh config load. +3. Disable persists false, passes ownership, calls structured restore once, and + never calls a stop/drain function. +4. Enable persists true and calls `syncModelsToCodex` with the running listener's + port; a test fails if the route calls bare injection. +5. Held history returns HTTP 409, reason `history_busy`, code + `native_integration_refused`; config/catalog are already native and desired OFF + remains persisted. The response is never 200/green and never raw 500. +6. Permission failure returns HTTP 500, reason `history_permission`, code + `native_integration_failed`, with no retry advice. +7. Home mismatch returns 409 after desired OFF is persisted and before any Codex + artifact mutation. +8. External `custom` returns 200 `external_provider_preserved`; all three artifact + states are skipped and the status row carries the courtesy message. +9. Invalid JSON/non-boolean bodies return 400; config-lock contention is 409 while + an unopenable lock is 500, matching `tests/native-claude-code-toggle.test.ts:120-154`. +10. Start a real test proxy with another routed client, disable Codex through the + management route, then POST that client's deliberately local fixture request + to `/v1/responses` and assert its expected response. Also assert `/healthz` + identifies the same PID before and after. This proves the shared endpoint and + process stayed alive; checking only the PUT response would not prove C4. + +GUI tests: + +1. `gui/tests/integrations-overview-rows.test.ts` — settled native Codex gains a + toggle/path/blocker; missing or unsettled native evidence remains unknown with + no active switch; external provider renders the courtesy detail and no applied + claim. +2. `gui/tests/overview-state-merge.test.ts` — widen client/reason validators and + prove localized `history_busy` and `history_permission`; keep raw + `write_failed` detail. +3. `gui/tests/consequence-dialog.test.tsx` — render the Codex copy in slot order, + assert the root-model loss, non-byte-identical history, external-provider + courtesy, and `/v1/responses` survival sentences, plus focus return and the + pending double-submit guard already tested for Grok. + +## Verification + +Static and automated gates: + +```bash +bun run typecheck +bun test tests/codex-history-provider.test.ts tests/codex-journal.test.ts tests/native-codex-toggle.test.ts +bun run test +cd gui && bun test tests && bun run lint && bun run lint:i18n && bun run build +cd .. && bun run privacy:scan +``` + +Live HTTP proof uses the already-running proxy at `localhost:10100`; do not call +`ocx stop` or `ocx restore`. Run only in the implementation C phase, after taking +a copy of the user's current Codex config for inspection and with an admin token +supplied by the maintainer: + +```bash +export OCX_LIVE_BASE=http://localhost:10100 +export OCX_ADMIN_TOKEN="${OPENCODEX_ADMIN_AUTH_TOKEN:?set the live management token without printing it}" + +curl -fsS "$OCX_LIVE_BASE/healthz" > .tmp/wp5-health-before.json +curl -fsS -H "x-opencodex-api-key: $OCX_ADMIN_TOKEN" \ + "$OCX_LIVE_BASE/api/native-integrations" > .tmp/wp5-native-before.json + +curl -fsS -X PUT -H "x-opencodex-api-key: $OCX_ADMIN_TOKEN" \ + -H 'content-type: application/json' -d '{"enabled":false}' \ + "$OCX_LIVE_BASE/api/native-integrations/codex" > .tmp/wp5-disable.json +curl -fsS "$OCX_LIVE_BASE/healthz" > .tmp/wp5-health-disabled.json +curl -sS -o .tmp/wp5-other-client.json -w '%{http_code}\n' \ + -H "x-opencodex-api-key: $OCX_ADMIN_TOKEN" -H 'content-type: application/json' \ + -d '{}' "$OCX_LIVE_BASE/v1/responses" + +curl -fsS -X PUT -H "x-opencodex-api-key: $OCX_ADMIN_TOKEN" \ + -H 'content-type: application/json' -d '{"enabled":true}' \ + "$OCX_LIVE_BASE/api/native-integrations/codex" > .tmp/wp5-enable.json +curl -fsS -H "x-opencodex-api-key: $OCX_ADMIN_TOKEN" \ + "$OCX_LIVE_BASE/api/native-integrations" > .tmp/wp5-native-after.json +curl -fsS "$OCX_LIVE_BASE/healthz" > .tmp/wp5-health-after.json +``` + +Read every JSON artifact. Disable must show Codex `absent` (or the classified +history refusal), enable must show `current` unless the explicit external-provider +courtesy applies, and all three health files must identify the same running proxy. +The `/v1/responses` invalid-body probe must reach the data-plane handler (an +expected validation/auth response, not connection refusal or 404); the automated +fixture test above is the stronger proof that another client completes a routed +request without spending live provider credits. Finally inspect the emitted Codex +catalog after enable and prove at least one current `provider/model` row exists; +an injected config beside a native-only catalog does not satisfy C5. + +For the held-history activation, open Codex so its writer lock is genuinely held, +disable once, and require HTTP 409 `history_busy` plus the localized card notice. +Close Codex/IDE, disable again, and require success with history `state: "ok"`. +Do not simulate this live proof by editing the response or matching its message. + +## Accept criteria + +- **C5 — Codex toggles both directions from the overview with the proxy running.** + Disable calls structured `restoreNativeCodex`; enable calls + `syncModelsToCodex(runtimePort)`. Live GET and on-disk catalog/config evidence + agree after both directions, and no stop/drain path runs. +- **C6 — a held history DB is explained, never false green.** A real held lock + yields `409 native_integration_refused / history_busy`; config and catalog are + reported separately, desired OFF persists, the card says why routed threads + remain hidden, and no layer parses `message`. +- **C4 — other clients keep serving.** The proxy PID/health identity is unchanged + across disable and enable, `/v1/responses` remains registered, and another + client's fixture request completes while Codex is OFF. +- External `model_provider` ownership remains untouched and visible on the card; + resume history is described as semantically reversible, not byte-identical; + and the dialog states that a post-injection routed root model selection is + destroyed and cannot be reconstructed. diff --git a/devlog/_plan/260803_codex_desktop_toggle/050_desktop_toggle.md b/devlog/_plan/260803_codex_desktop_toggle/050_desktop_toggle.md new file mode 100644 index 000000000..1ba67ae1f --- /dev/null +++ b/devlog/_plan/260803_codex_desktop_toggle/050_desktop_toggle.md @@ -0,0 +1,627 @@ +# WP6 — Claude Desktop toggle: pivot to standard mode, then remove credentials + +Research: `002_desktop_standard_mode.md`. Read it first; this doc is the diff. +The official contract is Anthropic's +[Claude Desktop configuration reference](https://claude.com/docs/third-party/claude-desktop/configuration) +(last modified 2026-07-24), as captured with the other primary citations in `002`. + +The concrete failure mode is a disable that deletes `.json` while +`_meta.json.appliedId` still names that id: Desktop opens the selected file by id, +so the next launch points at something missing. The superseded `030` correctly +noticed that pointer hazard, but concluded that exact restoration of the previous +selection was required and therefore removal needed a durable operation-state +engine. That joined two different requirements. Exact restoration is still +impossible because apply overwrites `appliedId` without recording its previous +value (`src/claude/desktop-3p.ts:345-358`); returning to standard Claude is +documented and achievable. This phase writes and selects a present, readable, +credential-free `{}` profile first, then removes the old opencodex profile and +its credential-bearing backup. It does not add an operation-state engine. + +## IN / OUT + +IN: + +- `src/claude/desktop-3p.ts` — MODIFY: add the standard-mode remover and make + apply prefer the selected opencodex row when an interrupted cleanup left two. +- `src/cli/claude-desktop.ts` — MODIFY: explicit CLI apply is the enable + direction and persists WP3 desired ON plus `desktopAutoApply: true` first. +- `src/server/management/agent-settings-routes.ts` — MODIFY: gate auto-apply on + WP3 desired state, re-check after its await, expose desired state in `/status`, + and make explicit `/apply` an enable action. +- `src/server/management/native-integration-routes.ts` — MODIFY: add + `claude-desktop` status and `PUT` toggle using the existing typed + success/refusal/single-flight pattern (`:31-86`, `:164-174`, `:371-445`). +- `gui/src/pages/integrations/native-api.ts` — MODIFY: carry the new native id, + refusal reasons, and residual paths. +- `gui/src/pages/integrations/integration-api.ts` — MODIFY: parse Desktop desired + state from the existing rich status route. +- `gui/src/pages/integrations/overview-clients.ts` — MODIFY: give + `claudeDesktopRow` a toggle and keep desired switch state separate from observed + `applied` state. +- `gui/src/pages/integrations/IntegrationsOverview.tsx` — MODIFY: route the toggle, + select Desktop dialog copy, and render localized partial/refusal outcomes. +- `gui/src/pages/integrations/refusal-copy.ts` — MODIFY: translate Desktop's + metadata refusal and incomplete credential cleanup. +- `gui/src/pages/ClaudeDesktop.tsx` — MODIFY: show desired OFF honestly and label + Save + Apply as an enable action while OFF. +- `gui/src/i18n/{en,ko,ja,zh,de,ru}.ts` — MODIFY: exact keys below. +- `tests/desktop-3p-removal.test.ts` — NEW: filesystem and crash-boundary cases. +- `tests/native-claude-desktop-toggle.test.ts` — NEW: route, ordering, persistence, + refusal, and auto-apply cases. +- `tests/claude-messages-endpoint.test.ts` — MODIFY: shared transport remains live. +- `gui/tests/integrations-overview-rows.test.ts` — MODIFY: desired/observed mapping. +- `gui/tests/integrations-surfaces.test.tsx` — MODIFY: switch, dialog, and outcome. +- `gui/tests/claude-desktop-locale.test.ts` — MODIFY: six-locale parity. + +OUT: + +- `src/types.ts` and `src/config.ts` — WP3 already owns + `clientIntegrations["claude-desktop"]`, default-ON parsing, + `clientIntegrationEnabled`, and `setClientIntegrationEnabled`. WP6 consumes + those helpers and does not open-code the map (`020_desired_state.md:149-183`). +- `src/claude/desktop-3p-paths.ts` — path resolution is already one tested owner; + the remover consumes `resolveDesktop3pConfigLibraryPath()` unchanged (`:67-78`). +- `src/claude/desktop-profile.ts` — assignments/defaults are preserved as-is; no + new profile field is needed. +- `/v1/messages` and `src/server/claude-messages.ts` — shared transport is not a + Desktop lifecycle switch. Claude Code must continue using it. +- `inferenceProvider: "anthropic"` — this means direct Claude API billing, not + normal subscription mode, so it is not a disable fallback. +- A native Desktop "return to standard" button — UNPROVEN and not called. +- Recording the previous `appliedId` — explicitly deferred. It would enable exact + restoration of another prior third-party selection, not standard-mode disable. +- The user's live Claude Desktop config library — no implementation or C-gate + command mutates it without a separate, explicit approval. + +## What we depend on and what we refuse to depend on + +We depend on one official contract: third-party mode activates only when +`inferenceProvider` and that provider's required credentials are valid; otherwise +Desktop launches in standard mode. Desktop reads the configuration once at launch +([configuration reference](https://claude.com/docs/third-party/claude-desktop/configuration), +`002:27-46`). Therefore `{}` is deliberate: valid JSON, no +`inferenceProvider`, no credential fields. + +We refuse to depend on all four UNPROVEN behaviors: + +1. absent `appliedId` being safe; +2. dangling `appliedId` being tolerated; +3. a `Default` entry being guaranteed; +4. a native "return to standard" UI action existing. + +The algorithm never removes `appliedId`, never points it at an unreadable or +missing file, never chooses by `name === "Default"`, and never automates Desktop's +UI. The local evidence makes the third refusal load-bearing: this machine's real +`_meta.json` has a `Default` row whose `.json` does not exist (`002:60-63`). + +## Core diff — select a safe target before cleanup + +MODIFY `src/claude/desktop-3p.ts`. Add `unlinkSync` to the existing fs import, +export the result vocabulary beside `Desktop3pConfigLibraryOptions`, and place the +remover after `writeDesktop3pConfig` and before `atomicReplaceDesktopConfig`: + +```diff +-import { copyFileSync, existsSync, mkdirSync, readFileSync } from "node:fs"; ++import { copyFileSync, existsSync, mkdirSync, readFileSync, unlinkSync } from "node:fs"; +@@ ++export type Desktop3pRemoveReason = ++ | "unsafe_metadata" ++ | "write_failed" ++ | "cleanup_incomplete"; ++ ++export interface Desktop3pRemoveResult { ++ ok: boolean; ++ changed: boolean; ++ libraryPath: string; ++ standardProfilePath?: string; ++ reason?: Desktop3pRemoveReason; ++ message?: string; ++ residualPaths?: string[]; ++} ++ ++export interface Desktop3pRemoveDeps { ++ randomId?: typeof randomUUID; ++ writeFile?: typeof atomicWriteFile; ++ unlinkFile?: typeof unlinkSync; ++} +``` + +The function signature is fixed here, and it lives in +`src/claude/desktop-3p.ts` because `parseMetadata`, metadata ownership, atomic +writes, and profile-path construction already live there: + +```ts +export function removeDesktop3pConfig( + options: Desktop3pConfigLibraryOptions = {}, + deps: Desktop3pRemoveDeps = {}, +): Desktop3pRemoveResult +``` + +`options` gives tests the same pure path seam used by +`resolveDesktop3pConfigLibraryPath`; `deps` gives failure tests deterministic ids, +atomic-write failure, and unlink failure without patching globals. Production +passes neither. The function does not accept or return credential values and +never logs profile contents. + +The implementation is this ordered state machine, expressed against the current +writer: + +```diff + export function writeDesktop3pConfig(/* existing args */) { +@@ +- const existing = metadata.entries.find(entry => entry?.name === "opencodex" && typeof entry.id === "string"); ++ // If disable was interrupted after selecting its replacement, reuse the ++ // selected opencodex row, not an older non-selected cleanup row. ++ const existing = metadata.entries.find(entry => ++ entry?.name === "opencodex" && entry.id === metadata.appliedId ++ ) ?? metadata.entries.find(entry => ++ entry?.name === "opencodex" && typeof entry.id === "string" ++ ); +@@ + } ++ ++export function removeDesktop3pConfig( ++ options: Desktop3pConfigLibraryOptions = {}, ++ deps: Desktop3pRemoveDeps = {}, ++): Desktop3pRemoveResult { ++ const libraryPath = resolveDesktop3pConfigLibraryPath(options); ++ const metadataPath = join(libraryPath, "_meta.json"); ++ const randomId = deps.randomId ?? randomUUID; ++ const writeFile = deps.writeFile ?? atomicWriteFile; ++ const unlinkFile = deps.unlinkFile ?? unlinkSync; ++ // 1. Parse and validate before the first Desktop-library write. A non-string ++ // id, path separator, duplicate non-selected opencodex row, or malformed ++ // entries array is unsafe_metadata: desired OFF is already persisted by the ++ // caller, but this function touches no Desktop bytes. ++ ++ // 2a. If appliedId already selects an opencodex row whose file parses exactly ++ // as a credential-free object with no inferenceProvider, this is a retry. ++ // Reuse it; do not allocate another replacement. ++ // 2b. Otherwise allocate randomId(), atomically write "{}\n" to the fresh ++ // .json, and verify it can be read and parsed before publishing the id. ++ ++ // 3. Atomically write metadata with BOTH rows still present and appliedId set ++ // to the new standard row. From here on Desktop's selected id always resolves. ++ ++ // 4. Remove the old .json.bak FIRST, then old .json through ++ // unlinkFile. Keep the old metadata row until both deletions succeed: it is ++ // the retry locator after a crash. Missing files count as already cleaned; ++ // no file content is printed. ++ ++ // 5. Atomically remove the old metadata row LAST. Preserve every unrelated ++ // entry and every unknown top-level/entry field byte-semantically through ++ // object spreads, as writeDesktop3pConfig does today. ++ ++ // Any failure after step 3 returns cleanup_incomplete plus residualPaths. ++ // appliedId remains on the readable standard profile; the caller does not ++ // clear appliedFingerprint/appliedAt until a retry completes steps 4-5. ++} +``` + +Path validation is deletion policy, not format cleanup. The old id must be one +path component: reject `/`, `\\`, `..`, NUL, or a resolved path outside +`libraryPath`. Multiple non-selected `name === "opencodex"` rows are ambiguous and +REFUSE `unsafe_metadata`; the function does not guess which user-visible row to +delete. A missing `_meta.json` is not unsafe: create the standard file and a new +metadata document with one selected opencodex row. A dangling `Default` row is +preserved untouched. + +The standard file is exactly `{}` plus a newline. Do not send +`inferenceProvider: "anthropic"`; do not copy any old field into the replacement; +do not call `atomicReplaceDesktopConfig` for the fresh target, because that would +create another `.bak` the disable then has to explain. + +## Persist intent before touching Desktop + +WP3 provides desired state. Both CLI apply and the existing POST apply become +explicit enable actions: + +```diff + // src/cli/claude-desktop.ts:45-49 ++import { loadConfig, saveConfigPreservingClaudeCode, setClientIntegrationEnabled } from "../config"; +@@ + const config = loadConfig(); ++ setClientIntegrationEnabled(config, "claude-desktop", true); + const state = await buildClaudeDesktopState(config, profile); +- config.claudeCode = { ...(config.claudeCode ?? {}), desktopProfile: state.profile }; ++ config.claudeCode = { ++ ...(config.claudeCode ?? {}), ++ desktopAutoApply: true, ++ desktopProfile: state.profile, ++ }; + saveConfigPreservingClaudeCode(config); +``` + +```diff + // src/server/management/agent-settings-routes.ts:735-738 + const state = await buildClaudeDesktopState(config, profileOverride); +- config.claudeCode = { ...(config.claudeCode ?? {}), desktopProfile: state.profile }; ++ setClientIntegrationEnabled(config, "claude-desktop", true); ++ config.claudeCode = { ++ ...(config.claudeCode ?? {}), ++ desktopAutoApply: true, ++ desktopProfile: state.profile, ++ }; + saveConfigPreservingClaudeCode(config); +``` + +The disable endpoint follows `native-integration-routes.ts` rather than inventing +a second response grammar. Extend `NativeIntegrationClientId` with +`"claude-desktop"`, extend refusal reasons with `unsafe_metadata` and +`cleanup_incomplete`, add a module-level Desktop single-flight, include its status +in GET, and add `PUT /api/native-integrations/claude-desktop`: + +```diff +-export type NativeIntegrationClientId = "claude" | "grok"; ++export type NativeIntegrationClientId = "claude" | "claude-desktop" | "grok"; +@@ + export type NativeRefusalReason = + | "not_installed" + | "orphaned_marker" ++ | "unsafe_metadata" ++ | "cleanup_incomplete" + | "home_mismatch" +@@ + export interface NativeRefusalEnvelope { +@@ ++ residualPaths?: string[]; + } +``` + +WP3 has already added `desiredEnabled` to `NativeStatus`, +`NativeToggleEnvelope`, and post-persist refusal envelopes +(`020_desired_state.md:423-453`). Desktop uses those fields; it does not add a +second desired-state property. + +Disable body, in this exact order: + +```ts +// PUT { enabled: false } +// 1. Persist BOTH suppressors before the Desktop mutation. +setClientIntegrationEnabled(config, "claude-desktop", false); +config.claudeCode = { + ...(config.claudeCode ?? {}), + desktopAutoApply: false, +}; +persist(config); + +// 2-3. Fresh readable standard profile -> appliedId pivot -> old .bak/.json/row. +const removed = removeDesktop3pConfig(); +if (!removed.ok) { + // unsafe_metadata: 409 refused; no Desktop bytes changed. + // write_failed/cleanup_incomplete: 500 failed. For cleanup_incomplete include + // residualPaths; desired OFF and auto-apply suppression remain persisted. + return desktopRefusal(removed); +} + +// 4. Only complete cleanup clears observed apply markers. Preserve the profile's +// assignments/defaults (33 assignments on this machine) and every other field. +const profile = config.claudeCode?.desktopProfile; +if (profile) { + const { appliedFingerprint: _fingerprint, appliedAt: _appliedAt, ...preserved } = profile; + config.claudeCode = { ...config.claudeCode, desktopProfile: preserved }; + persist(config); +} + +return jsonResponse({ + ok: true, + clientId: "claude-desktop", + changed: removed.changed, + state: "absent", + reason: "desktop_standard_mode", + message: "Claude Desktop is configured for standard mode; restart required", +}); +``` + +The endpoint does not stop, restart, or reconfigure the proxy. Enable persists +desired `true` and `desktopAutoApply: true` first, then runs the same state build +and `writeDesktop3pConfig` path as POST apply, and finally records fingerprint/time. +If generation fails, desired ON remains visible while observed state remains off; +the response is a failure, not false green. No enable or disable branch touches +`config.claudeCode.enabled`, so Claude Code's use of `/v1/messages` is unchanged. + +## Crash-safety: exact residual state at every boundary + +There is no transaction across opencodex `config.json`, Desktop `_meta.json`, and +three profile paths. The ordering preserves a valid selected pointer; it does not +make the whole disable transactional. + +| Process dies after | State on disk | Classification / retry | +|---|---|---| +| desired OFF + `desktopAutoApply:false`, before standard file | Desktop still selects the old gateway profile; automatic re-apply is suppressed | Pointer-safe only; disable is not effective. Retry starts the mutation. | +| standard `{}` file, before first metadata write | Old profile still selected; the fresh credential-free file is orphaned and its generated id was not durably recorded | Pointer-safe only; retry creates another fresh target. The harmless orphan may remain because identifying it after process death would require the operation record this design deliberately does not add. No claim of transactional cleanup or semantic disable. | +| metadata points to standard, before `.bak` deletion | Next Desktop launch is standard mode; old profile and backup still contain credentials | Pointer-safe and semantically disabled on next launch, but security cleanup is incomplete. Old metadata row locates both files for retry. | +| `.bak` deleted, before old `.json` deletion | Selected standard file exists; one old credential-bearing profile remains | Pointer-safe, not security-complete. Retry deletes the old profile. | +| old `.json` deleted, before old row removal | Selected standard file exists; stale non-selected row may name a missing file | Applied-pointer-safe, not registry-clean. Retry removes that exact old row. | +| old row removed, before markers clear | Desktop library and credential cleanup are complete; `/status` still derives `applied:true` from the stale fingerprint (`agent-settings-routes.ts:797-804`) | Runtime files are safe; bookkeeping is false. Retry clears markers without changing assignments/defaults. | +| markers clear | Desired OFF, auto-apply OFF, selected standard profile readable, old profile and `.bak` absent | Disable is complete on disk. A Desktop process already running can still be using launch-time state until restart. | + +The first metadata write contains both rows and points at the new one. Cleanup +deletes the `.bak` first because it is an otherwise unmanaged credential copy, +then the old config, then its metadata row. Removing the row first would lose the +only crash-retry locator; deleting either file before the pointer pivot would +recreate the original dangling-selection bug. + +## The `.bak` is a security obligation + +`atomicReplaceDesktopConfig` copies the prior profile to `.json.bak` +(`src/claude/desktop-3p.ts:371-380`), and the profile contains +`inferenceGatewayApiKey`. Nothing removes it today (`002:13-15`). A successful +disable MUST end with both old `.json` and `.json.bak` absent. A response +cannot say success when either remains: return `cleanup_incomplete`, include only +residual file paths, keep desired OFF, and keep the old metadata row as the retry +locator. Never include file contents, parsed credential fields, or credential +values in logs, errors, tests, screenshots, or the API envelope. + +## Auto-apply suppression + +The located automatic caller is `PUT /api/subagent-models`: after saving and two +other refreshes, it awaits `autoApplyDesktopBestEffort()` +(`agent-settings-routes.ts:518-528`). Its current guard checks only +`desktopAutoApply === false` before `fetchAllModels` (`:130-151`). Persisting OFF +first prevents later calls; a second check after the await closes the already +started in-process race: + +```diff + async function autoApplyDesktopBestEffort(): Promise { + try { ++ if (!clientIntegrationEnabled(config, "claude-desktop")) return; + if (config.claudeCode?.desktopAutoApply === false) return; + if (!config.claudeCode?.desktopProfile) return; +@@ + const allModels = await fetchAllModels(config); + const routed = /* existing mapping */; ++ // The toggle can persist OFF while fetchAllModels was awaiting. Re-check ++ // immediately before the synchronous writer. ++ if (!clientIntegrationEnabled(config, "claude-desktop")) return; ++ if (config.claudeCode?.desktopAutoApply === false) return; + const result = writeDesktop3pConfig(/* existing args */); +``` + +If auto-apply has already entered the synchronous writer, JavaScript completes +that write before the toggle handler runs; the later disable then pivots away and +cleans it. If it is awaiting model discovery, the second guard stops it. This is +in-process ordering, not a cross-process file lock; a second opencodex process is +INFERRED possible and is reported by post-write status, not claimed excluded. + +Extend `/api/claude-desktop/status` without changing the existing observed fields: + +```diff + return jsonResponse({ ++ enabled: clientIntegrationEnabled(config, "claude-desktop"), + applied: savedFingerprint !== null, +@@ + }); +``` + +Desired state drives the switch. `applied`, `stale`, and `activeProfile` remain +observed evidence and drive badge/count detail. A failed disable may therefore +show switch OFF with an amber observed-state notice; that is the truthful +"desired OFF, observed conflict" state required by WP3. + +## GUI — one switch, one consequence dialog + +`claudeDesktopRow` currently hard-codes `toggle: null` +(`gui/src/pages/integrations/overview-clients.ts:240-277`). Give it the native id +and a separate optional `toggleOn` so the summary count does not become desired +state by accident: + +```diff + export interface OverviewRow { +@@ + applied: boolean; ++ /** Desired switch position; absent means use observed `applied`. */ ++ toggleOn?: boolean; +@@ + function claudeDesktopRow( + payload: ClaudeDesktopPayload | null, ++ native: NativeStatus | undefined, ++ nativeSettled: boolean, + ): OverviewRow { +@@ +- toggle: null, +- toggleBlocked: null, +- togglePath: null, ++ toggle: "claude-desktop", ++ toggleBlocked: native?.disableBlocked ?? null, ++ togglePath: native?.configPath ?? null, ++ toggleOn: native?.desiredEnabled ?? (payload?.enabled !== false), +``` + +`OverviewCard` renders `on={row.toggleOn ?? row.applied}`. The Desktop row is +unknown and non-actionable until both its rich status and native status settle. +Desired OFF + stale marker is amber, not green; desired ON + no applied marker is +absent with the switch ON and `integrations.detail.desktopDesiredOnNotApplied`. + +Add `DESKTOP_DISABLE_COPY` beside `GROK_DISABLE_COPY` and branch on +`pendingToggle.id`. Exact English source copy: + +> **Disable Claude Desktop integration?** +> +> `{path}` will be updated to select a new credential-free opencodex profile with +> no inference provider. The previous opencodex profile and its backup will be +> removed. +> +> Claude Desktop will stop using models routed through opencodex and return to +> standard Claude. +> +> Turning it back on regenerates the opencodex profile from your saved model +> assignments. It cannot restore whichever profile was selected before +> opencodex was first applied. +> +> **Claude Desktop reads this configuration only at launch. Fully quit and reopen +> Claude Desktop for this change to take effect.** + +Confirm label: **Disable**. The restart sentence is `sideEffectKey`, not a toast +added after confirmation: the user sees the delayed effect before choosing. +There is no claim that the current Desktop process switched instantly. + +The refusal/partial copy is equally exact: + +- `unsafe_metadata` — "Claude Desktop's metadata could not be read safely, so + its library was not changed. The requested Off state was saved and automatic + apply remains disabled. Repair `{path}/_meta.json`, then try again." +- `config_busy` — reuse the existing native lock copy: nothing was persisted and + retry is appropriate. +- `cleanup_incomplete` — "Claude Desktop is pointed at standard mode, but old + opencodex credential files remain at: `{paths}`. Remove them manually before + treating cleanup as complete." This is a failed partial outcome, not a refusal + pretending nothing changed. +- `write_failed` before the pointer pivot — use the server message and say no + Desktop library change completed; desired OFF remains saved if step 1 passed. + +On the Desktop page, add `enabled` to `DesktopStatus`. When false, the status bar +says: "Claude Desktop integration is off. Desktop reads configuration only at +launch; if it was open during the change, fully quit and reopen it." Save remains +available because assignments/defaults are intentionally preserved; Save + Apply +reads **Enable and apply** and goes through the explicit enable path. + +## i18n + +Add every key to exactly these six locale files: + +``` +gui/src/i18n/en.ts +gui/src/i18n/ko.ts +gui/src/i18n/ja.ts +gui/src/i18n/zh.ts +gui/src/i18n/de.ts +gui/src/i18n/ru.ts +``` + +Exact keys (English is the source of truth / `TKey`): + +```text +integrations.dialog.desktop.title +integrations.dialog.desktop.changes +integrations.dialog.desktop.breakage +integrations.dialog.desktop.undo +integrations.dialog.desktop.restart +integrations.dialog.desktop.confirm +integrations.detail.desktopDesiredOff +integrations.detail.desktopDesiredOnNotApplied +integrations.native.error.desktopUnsafeMetadata +integrations.native.error.desktopCleanupIncomplete +integrations.native.msg.desktopDisabled +integrations.native.msg.desktopEnabled +claudeDesktop.status.disabled +claudeDesktop.enableApply +``` + +`changes` interpolates `{path}`; `desktopUnsafeMetadata` interpolates `{path}`; +`desktopCleanupIncomplete` interpolates `{paths}`. Do not put a credential value +or profile JSON into any interpolation. + +## Test plan + +`tests/desktop-3p-removal.test.ts` uses +`OPENCODEX_CLAUDE_DESKTOP_CONFIG_DIR`/explicit options and `mkdtempSync`; it never +resolves the user's live library. + +1. Normal disable writes a fresh UUID `{}` profile, verifies it is readable, + points `appliedId` to it, removes only the old opencodex row, and preserves + unrelated rows/top-level fields. +2. **Dangling Default fixture:** `_meta.json` contains `Default`, but its + `.json` does not exist. Disable neither selects nor changes that row; the + selected fresh standard file exists. This reproduces the real-machine fact. +3. The replacement JSON has no `inferenceProvider`, + `inferenceGatewayApiKey`, or other credential field. Do not assert by printing + values; assert key absence. +4. The old `.json.bak` exists before removal and is absent after. This is a + mandatory security assertion, not incidental cleanup. The old `.json` is + also absent. +5. Crash fixtures resume from every table row above: selected standard + both old + files; backup gone; both files gone + old row present; markers handled at the + route layer. Every retry ends with one selected opencodex row and no old files. +6. Metadata malformed, path-escaping id, and two non-selected opencodex cleanup + rows each REFUSE `unsafe_metadata` without a Desktop-library write. +7. Injected delete failure returns `cleanup_incomplete`, keeps the old row as + locator, reports residual paths only, and leaves selected standard readable. +8. Idempotent retry allocates no second standard profile. +9. Re-enable prefers the selected standard row, overwrites it through the normal + gateway writer, and does not revive an old interrupted-cleanup id. + +`tests/native-claude-desktop-toggle.test.ts` follows the injected-persist seam in +`tests/native-claude-code-toggle.test.ts:18-43`: + +1. Absent WP3 key reads desired ON; upgrade behavior is unchanged. +2. Disable persists desired false and `desktopAutoApply:false` before the remover + seam is called; a spy records call order. +3. Successful cleanup clears only `appliedFingerprint`/`appliedAt` and preserves + all assignments/defaults (include 33 assignments to pin the observed scale). +4. `unsafe_metadata` leaves desired OFF persisted and markers intact, returning + 409 with the typed refusal. +5. Cleanup partial returns 500, residual paths, selected-standard state, desired + OFF, and no false success. +6. Config `SQLITE_BUSY` refuses before any Desktop mutation; broken lock is 500. +7. Two concurrent PUTs produce one `config_busy` and no overlapping remover. +8. Auto-apply that is paused in `fetchAllModels`, then disabled, hits the second + guard and never calls `writeDesktop3pConfig`. This activates the race fix at + `agent-settings-routes.ts:137-139`, not merely its first guard. +9. Enable and explicit POST apply persist desired true, keep assignments/defaults, + regenerate the gateway profile, and record markers only after write success. + +MODIFY `tests/claude-messages-endpoint.test.ts`: start from Claude Code enabled, +perform the Desktop disable PUT against a temp config library, then send a valid +Claude Code `/v1/messages` request through the same test server. Assert it reaches +the existing transport/adapter path rather than 403/404. Also assert the proxy +health endpoint still responds. This is the C4 proof that Desktop OFF does not +shut down the shared transport. + +GUI cases: + +- `integrations-overview-rows.test.ts`: switch uses desired state while badge and + applied count use observed state; OFF + stale marker is not green. +- `integrations-surfaces.test.tsx`: Desktop card has a keyboard-operable switch; + disable opens the Desktop—not Grok—dialog; all five paragraphs render in order; + confirm calls `/api/native-integrations/claude-desktop`; restart-required text + is visible before confirm; focus returns to the switch. +- `claude-desktop-locale.test.ts`: all 14 keys exist and are non-empty in all six + locales. + +## Verification + +Implementation C-gate commands: + +```bash +bun run typecheck +bun test --isolate tests/desktop-3p-removal.test.ts tests/native-claude-desktop-toggle.test.ts tests/claude-messages-endpoint.test.ts tests/claude-management-api.test.ts +bun run test +bun run privacy:scan +cd gui && bun test tests +cd gui && bun run lint +cd gui && bun run lint:i18n +cd gui && bun run build +``` + +Render grounding: open the Integrations overview in the real dashboard, activate +the Desktop OFF switch with keyboard, screenshot the open dialog at desktop and +constrained width, read the screenshot back, and verify the restart sentence is +visible before confirmation. In browser QA, point the server at a temporary +`OPENCODEX_CLAUDE_DESKTOP_CONFIG_DIR`; do not confirm against the live daemon. + +Real-machine proof is deliberately read-only. It may parse only +`configLibrary/_meta.json` to report row names/ids and `existsSync(.json)` +booleans, confirming the known dangling `Default` shape and that path resolution +targets the installed Desktop library. It must not open or print any profile or +`.bak` contents. All destructive activation proof runs against `mkdtempSync` on +the same machine. Do **not** call the live PUT endpoint, remover, `unlink`, or +apply route against the user's real Desktop library; that would change the active +selection and delete credential-bearing files without separate approval. + +## Accept criteria + +- **C7** (`000_plan.md:91-92`) — after a successful disable, `_meta.json.appliedId` + names a present, readable `{}` profile with no `inferenceProvider` or credential + fields; the previous opencodex `.json` and `.json.bak` are absent. The + dangling-Default fixture proves no `Default` assumption entered the path. +- **C4** (`000_plan.md:86-87`) — disable does not stop/restart the proxy and does + not change `claudeCode.enabled`; a Claude Code request still traverses + `/v1/messages` after Desktop is disabled, and proxy health remains live. +- Desired OFF and `desktopAutoApply:false` are durable before the Desktop write; + a failed/partial mutation reports desired OFF versus observed residue rather + than silently re-enabling. +- Assignments/defaults survive disable and enable byte-semantically as parsed + data; only `appliedFingerprint`/`appliedAt` are cleared after complete cleanup. +- The dialog states before confirmation that a full Desktop quit/reopen is + required. No UI or API claims the running Desktop process changed instantly. diff --git a/devlog/_plan/260803_codex_desktop_toggle/_retired/000_why_retired.md b/devlog/_plan/260803_codex_desktop_toggle/_retired/000_why_retired.md new file mode 100644 index 000000000..0dabd786c --- /dev/null +++ b/devlog/_plan/260803_codex_desktop_toggle/_retired/000_why_retired.md @@ -0,0 +1,23 @@ +# Why these two are retired + +`020_codex_toggle.md` and `030_desktop_toggle.md` were written against the +durable operation-state design this unit was originally scoped around: a +versioned discriminated journal entry, prepare/commit with restart +reconciliation, and a field-scoped config writer. + +That design was dropped after the research cycle recorded in `../001`-`../004`. +Two of its premises did not survive contact with the code: + +- **`020` assumed Codex needed a captured pre-state.** `ocx restore` already + restores native Codex without stopping the proxy (`src/cli/help.ts:18`), and + `ocx restore back` is the enable direction. The toggle pair exists; what was + missing is a switch that remembers being off. +- **`030` assumed Desktop removal was unsafe at any price.** It conflated + restoring the exact prior selection (still impossible) with returning the user + to standard Claude, which Anthropic documents as the behavior of a selected + config with no valid `inferenceProvider`. + +They are kept rather than deleted because their factual inventories are still +accurate and their reasoning explains why the replacement designs refuse certain +shortcuts. The live phases are `../040_codex_toggle.md` and +`../050_desktop_toggle.md`. diff --git a/devlog/_plan/260803_codex_desktop_toggle/020_codex_toggle.md b/devlog/_plan/260803_codex_desktop_toggle/_retired/020_codex_toggle.md similarity index 100% rename from devlog/_plan/260803_codex_desktop_toggle/020_codex_toggle.md rename to devlog/_plan/260803_codex_desktop_toggle/_retired/020_codex_toggle.md diff --git a/devlog/_plan/260803_codex_desktop_toggle/030_desktop_toggle.md b/devlog/_plan/260803_codex_desktop_toggle/_retired/030_desktop_toggle.md similarity index 100% rename from devlog/_plan/260803_codex_desktop_toggle/030_desktop_toggle.md rename to devlog/_plan/260803_codex_desktop_toggle/_retired/030_desktop_toggle.md From c8c213bf45fcd4da6b3da1480901e951762d937c Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Mon, 3 Aug 2026 23:46:36 +0900 Subject: [PATCH 006/163] =?UTF-8?q?docs(integrations):=20close=20audit=20r?= =?UTF-8?q?ound=201=20=E2=80=94=20accept=20all=20eleven,=20including=20the?= =?UTF-8?q?=20one=20I=20flagged=20myself?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit FAIL, 7 High. Five of them are one defect: I designed a flag and called it a state machine. Dropping the operation-state engine was right for its rollback half and wrong for its coordination half — single-flight, ordering, restart reconciliation — which I removed without noticing it was load-bearing for a different reason. Sharpest form: persist OFF, crash before the remover runs, and restart only skips future writes, so desired OFF and observed ON never reconcile. My claim in 003 that the boolean survives a restart was true of the boolean and false of the system. Also accepts #1, the decision I had flagged as most likely to matter: the invariant 'disabling must not stop serving' is correct for installation state and wrong for ingress admission. claudeCode.enabled is the documented kill switch for /v1/messages; removing it would silently reopen an ingress for a user whose switch still reads OFF. Keep both gates, drive them through the new helper. And #5, my dispatch error: WP5 and WP6 were written in parallel against the same route and do not compose — WP6 re-types the union without the Codex entry WP5 adds. WP3 takes the shared contract; they become sequential. Replacement is smaller than the old engine: mutatePersistedConfig (already in the tree at config.ts:1854, which I failed to look for), per-client single-flight with a re-read before the write, and startup convergence. --- .../005_audit_synthesis.md | 129 ++++++++++++++++++ 1 file changed, 129 insertions(+) create mode 100644 devlog/_plan/260803_codex_desktop_toggle/005_audit_synthesis.md diff --git a/devlog/_plan/260803_codex_desktop_toggle/005_audit_synthesis.md b/devlog/_plan/260803_codex_desktop_toggle/005_audit_synthesis.md new file mode 100644 index 000000000..cbb0dbd44 --- /dev/null +++ b/devlog/_plan/260803_codex_desktop_toggle/005_audit_synthesis.md @@ -0,0 +1,129 @@ +# Audit round 1 — synthesis + +Verdict: **FAIL**, 7 High plus 3 Medium and 1 Low. Independent reviewer, auditing +the roadmap against the real tree rather than against its own claims. + +What survived matters as much as what failed, so it goes first. + +## Confirmed by the reviewer + +- The **overturned conclusion is justified.** The full operation-state engine is + not inherently required for "OFF means converge to native/standard mode". The + predecessor unit's central finding really was too strong. +- The **official Anthropic citation is genuine.** The reviewer opened + the configuration reference itself and confirmed both the standard-mode + sentence and read-once-at-launch. +- Pi's `text|image` restriction is real, re-confirmed from current upstream docs. +- Absent `clientIntegrations` keys really do default ON. +- The whole-catalog modality assertion really would have caught the gjc bug. +- Numbered lexicographic split and diff-level coverage are satisfied. + +So the direction holds and WP2 is clean. Everything below is about the machinery +the roadmap put around it. + +## The one defect seen five ways + +Findings #2, #3, #4, #6 and #7 are one thing: **I designed a flag and called it a +state machine.** + +A durable boolean answers "what does the user want". It does not answer: + +- who writes it safely when two processes write at once (#2) +- who writes it at all for six of the ten clients (#3) +- what the CLI prints when it skips because of it (#4) +- what stops an in-flight writer that read it before the flip (#6) +- what converges observed state when we crash between persist and mutate (#7) + +I dropped the operation-state engine because its *rollback* half was +disproportionate. That was right. But the engine also carried a *coordination* +half — single-flight, ordering, restart reconciliation — and I dropped that too, +without noticing it was load-bearing for a different reason. + +#7 is the sharpest version. I wrote in `003` that a boolean "survives a restart", +which is true of the boolean and false of the system: persist OFF, crash before +the remover runs, and restart merely *skips future writes*. Desired OFF, observed +ON, forever, with nothing that ever reconciles them. The reviewer is right that +this is not established by anything I wrote. + +**Disposition: accept all five.** The replacement is not the old engine. It is +three specific mechanisms, and they belong in WP3 where the flag lives: + +1. **Field-scoped persistence.** `mutatePersistedConfig` already exists + (`src/config.ts:1854`) — I specified `saveConfigPreservingClaudeCode` without + checking whether the better primitive was already in the tree. It was. This + also removes the "mutate live object before persisting" bug the reviewer found + in the shipped route. +2. **Per-client single-flight around every irreversible write**, with desired + state re-read immediately before the write, not only at entry. +3. **Startup reconciliation**: desired OFF is a *converge* instruction, so + startup re-runs the idempotent remover rather than only skipping. + +That is meaningfully smaller than a versioned discriminated journal with +prepare/commit, and it is what the evidence actually demands. + +## The finding I most need to accept, and why + +**#1 — removing the Claude transport gates.** I had flagged this myself as the +decision most likely to matter, and the reviewer's judgment is that it is wrong. + +My reasoning was the invariant "disabling an integration means stop writing that +client's config, never stop serving." That invariant is correct **for +installation state** and I over-applied it to **ingress admission**. +`claudeCode.enabled` is documented and implemented as the kill switch for +`/v1/messages` (`src/server/claude-messages.ts:65-69`) and empties Anthropic +discovery (`src/server/index.ts:496`). An upgrading user whose switch reads OFF +would silently get an ingress that starts accepting traffic again. That is a +shipped-behavior regression dressed as a principle. + +**Accept.** Keep both gates, drive them through +`clientIntegrationEnabled(config, "claude-code")`, and add the compatibility test +proving a legacy `enabled: false` config still gets 403 and an empty model list +after migration. The invariant survives in its correct form: Codex OFF must not +close `/v1/responses`, which no client-specific gate ever guarded. + +## #5 — the phases do not compose + +WP5 and WP6 were dispatched in parallel and each wrote a diff against the tree as +it is today, not against the tree as the other leaves it. WP6 even re-types the +client union as `"claude" | "claude-desktop" | "grok"` — dropping the Codex entry +WP5 adds. + +That is my dispatch error, not the writers'. Two authors editing the same route +and the same union need one contract they both consume. + +**Accept.** WP3 owns the complete shared contract: the four-client union, the +status/success response schema including `desiredEnabled`, the refusal envelope, +and the status helpers. WP5 and WP6 are rewritten against it, and they are +**sequential, not parallel**, wherever their diffs touch the same file. + +## #8 and #9 — Desktop status honesty + +Both accepted, and #9 is the more embarrassing: a status read that hardcodes +`installed: true` and creates the config library when metadata is missing would +manufacture Claude Desktop directories on a machine that never had Desktop. A +read must not write. Observed state gets derived from `_meta.json`'s selected id +and the selected profile's actual contents, not from our own fingerprint. + +## #10, #11 + +#10 accepted: the test plans cover the happy paths and none of the five failure +modes above. Each new mechanism lands with the test that would catch its absence. + +#11 accepted with a correction to my own habit: several line citations drifted by +a few lines, and I asserted a `lastmod` date the reviewer could not see on the +page. The semantic claim was verified; the metadata was not. Cite what the page +says, not what the sitemap claimed about it. + +## Nothing rebutted + +Eleven findings, eleven accepted. That is not deference — I checked #1's gates, +#2's `mutatePersistedConfig`, and #3's toggle route in the tree before accepting. + +## Next + +A is a loop, and a FAIL round never exits it (AUDIT-LOOP-01). The amendments +above go into `020`, `040` and `050`, WP3 absorbs the shared contract and the +three coordination mechanisms, then the SAME reviewer re-audits. + +WP2 (`010_modality_boundary.md`) is untouched by every finding and stays as +written. From 16792a68f5c9ea08e3dd8330618edfec5280f5cb Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Tue, 4 Aug 2026 00:10:40 +0900 Subject: [PATCH 007/163] docs(integrations): fold every accepted blocker back into the three failing phases WP3 keeps the Claude ingress and discovery gates instead of deleting them, now driven by clientIntegrationEnabled; swaps whole-object saving for field-scoped mutatePersistedConfig; gives all six file clients a writer so the opencode guard is actually reachable; adds per-client single-flight with a persisted re-read immediately before every irreversible write; makes desired OFF a converge instruction that startup re-runs; and takes ownership of the four-client contract WP5 and WP6 both consume. WP5 pulls the CLI back into scope, because WP3's gate changes what ocx restore and restore back do: restore persists OFF, restore back persists ON, and a skipped sync is a discriminated result rather than a bare ok:true that would print success while doing nothing. WP6 stops its status read from creating a Desktop library on a machine that never had Desktop, derives observed state from the selected profile's real contents rather than our own fingerprint, and lands after WP5. --- .../020_desired_state.md | 1069 +++++++++-------- .../040_codex_toggle.md | 682 ++++++++--- .../050_desktop_toggle.md | 835 ++++++++++--- 3 files changed, 1731 insertions(+), 855 deletions(-) diff --git a/devlog/_plan/260803_codex_desktop_toggle/020_desired_state.md b/devlog/_plan/260803_codex_desktop_toggle/020_desired_state.md index 73f38fec7..a568b0f98 100644 --- a/devlog/_plan/260803_codex_desktop_toggle/020_desired_state.md +++ b/devlog/_plan/260803_codex_desktop_toggle/020_desired_state.md @@ -1,55 +1,69 @@ -# WP3 — durable per-client desired state +# WP3 — durable desired state, admission, and convergence -Research: `003_durable_desired_state.md`. Read it first; this doc is the diff. +Research: `003_durable_desired_state.md`. Audit disposition: +`005_audit_synthesis.md`. Read both first; this doc is the amended diff. The shipped Grok switch removes its fence but records no intent. The next `ocx start` calls `syncGrokConfig` and writes the fence back -(`src/cli/index.ts:334-341`, `src/grok/sync.ts:29-65`). WP3 gives that OFF a -durable owner before WP5 and WP6 add two more switches with the same failure -mode. +(`src/cli/index.ts:334-350`, `src/grok/sync.ts:29-65`). The failed first draft +added a boolean but omitted the writers, ordering, and restart work that make the +boolean govern the system. It also proposed removing Claude's shipped ingress +kill switch. That decision is reversed here, plainly: the audit was right. An +upgrading user with legacy `claudeCode.enabled=false` must not silently regain +Claude ingress. + +WP3 therefore owns more than a flag. It owns the shared desired-state contract, +field-scoped persistence, the per-client operation boundary, the six file-client +writer, startup reconciliation, and the response grammar WP5 and WP6 consume. ## IN / OUT | Path | Change | Why it is in WP3 | |---|---|---| -| `src/types.ts` | MODIFY | Defines the persisted client-id vocabulary and `OcxConfig.clientIntegrations`. | -| `src/config.ts` | MODIFY | Parses the map without turning one malformed value into an all-clients reset; owns effective-state reads and transition writes. | -| `src/codex/sync.ts` | MODIFY | Stops every Codex catalog/injection sync while Codex is desired OFF. | -| `src/server/management-api.ts` | MODIFY | Gates the direct catalog refresher that provider/model routes call, and moves Claude agent sync to the compatibility helper. | -| `src/grok/sync.ts` | MODIFY | Closes every start/ensure/restart path at the shared sync owner. | -| `src/server/management/native-integration-routes.ts` | MODIFY | Persists Grok intent before touching its file and mirrors Claude Code's old/new keys. | -| `src/server/management/agent-settings-routes.ts` | MODIFY | Requires both Desktop desired ON and `desktopAutoApply`, and mirrors Claude Code's settings route. | -| `src/cli/opencode.ts` | MODIFY | Stops the inline provider layer from bypassing an OpenCode OFF. | +| `src/types.ts` | MODIFY | Defines the complete ten-client desired-state vocabulary and `OcxConfig.clientIntegrations`. | +| `src/config.ts` | MODIFY | Parses the map, resolves legacy Claude intent, and mutates one selected key through the existing `mutatePersistedConfig` primitive. | +| `src/integrations/desired-state.ts` | NEW | Owns per-client single-flight, last-moment persisted-state checks, and reconciliation result types. | +| `src/integrations/reconcile.ts` | NEW | Converges desired OFF to observed absent for the six file clients and the native handlers registered by WP3/WP5/WP6. | +| `src/integrations/state.ts` | MODIFY | Adds required `desiredEnabled` to the six-client status helper. | +| `src/integrations/writer.ts` | MODIFY | Re-reads persisted intent immediately before apply/disable/restore commits. | +| `src/server/management/integration-routes.ts` | MODIFY | Persists the six-client desired state before applying/removing files; GET/status also reconciles stale OFF. | +| `src/codex/sync.ts` | MODIFY | Stops Codex catalog/injection while OFF, joins the per-client flight, and re-checks before each artifact write. | +| `src/grok/sync.ts` | MODIFY | Stops Grok fetch/write while OFF, joins the same Grok flight as every other caller, and re-checks before injection. | +| `src/server/management-api.ts` | MODIFY | Gates the direct Codex catalog refresher and moves Claude agent sync to the compatibility helper. | +| `src/server/management/native-integration-routes.ts` | MODIFY | Owns the complete four-client native contract, field-scoped Claude/Grok persistence, status helpers, and native reconciliation entry points. | +| `src/server/management/agent-settings-routes.ts` | MODIFY | Routes Grok/Desktop background writes through the shared flight and mutation contract. | +| `src/cli/index.ts` | MODIFY | Runs OFF reconciliation on start and both ensure branches before any automatic apply. | +| `src/cli/opencode.ts` | MODIFY | Refuses the inline provider writer after a real OpenCode OFF and re-checks before spawn. | | `src/cli/claude.ts` | MODIFY | Reads Claude Code desired state through the compatibility helper. | | `src/claude/agents-inject.ts` | MODIFY | Reads Claude Code desired state through the compatibility helper. | | `src/server/system-env.ts` | MODIFY | Reads Claude Code desired state through the compatibility helper. | -| `src/server/claude-messages.ts` | MODIFY | Removes the existing Claude-Code-only gate from the shared Messages transport. | -| `src/server/index.ts` | MODIFY | Removes the existing Claude-Code-only gate from shared Anthropic model discovery. | -| `tests/client-integration-desired-state.test.ts` | NEW | Pins schema defaulting, malformed-key salvage, compatibility, and mirroring. | -| `tests/client-integration-auto-gates.test.ts` | NEW | Drives every automatic gate and proves its writer is not called. | -| `tests/client-integration-transport-isolation.test.ts` | NEW | Proves one disabled client cannot shut down another client's transport. | -| `tests/native-grok-toggle.test.ts` | MODIFY | Pins persist-before-mutate and desired/observed conflict reporting. | -| `tests/native-claude-code-toggle.test.ts` | MODIFY | Pins both-key mirroring, including the old-value idempotent case. | -| `tests/claude-management-api.test.ts` | MODIFY | Pins mirroring through the older `/api/claude-code` route. | -| `tests/claude-messages-endpoint.test.ts` | MODIFY | Replaces the shipped transport-403 assertion with the shared-transport invariant. | +| `src/server/claude-messages.ts` | MODIFY | Keeps both shipped Claude ingress gates and changes only their reader to `clientIntegrationEnabled`. | +| `src/server/index.ts` | MODIFY | Keeps Anthropic discovery gated and changes only its reader to `clientIntegrationEnabled`. | +| `tests/client-integration-desired-state.test.ts` | NEW | Pins migration, per-field mutation, contention/retry, and preservation. | +| `tests/client-integration-auto-gates.test.ts` | NEW | Pins automatic gates, shared flights, last-moment re-checks, and OpenCode activation. | +| `tests/client-integration-reconciliation.test.ts` | NEW | Pins persist/mutate crash points and startup/ensure/status convergence. | +| `tests/management-integration-routes.test.ts` | MODIFY | Pins the six-client persist-before-mutate route and desired/observed responses. | +| `tests/native-grok-toggle.test.ts` | MODIFY | Pins field-scoped persistence, conflict reporting, and retry after lock refusal. | +| `tests/native-claude-code-toggle.test.ts` | MODIFY | Replaces the pinned live-object mutation bug with no-mutation-before-commit coverage. | +| `tests/claude-management-api.test.ts` | MODIFY | Pins compatibility mirroring through the older Claude route. | +| `tests/claude-messages-endpoint.test.ts` | MODIFY | Keeps the legacy OFF => 403 contract through the new reader. | OUT: | Path / surface | Reason | |---|---| -| `gui/` | WP3 has no new switch. WP5 and WP6 consume this contract. | -| `src/integrations/writer.ts` and the six-client ownership store | Observed provenance is deleted on disable (`writer.ts:373-384`); it cannot own durable OFF intent. | -| `src/codex/journal.ts` | Crash reconciliation repairs our stale write and must run regardless of desired state. | -| `src/service.ts` stop/uninstall teardown | Teardown removes dead proxy pointers; it must neither consult nor rewrite desired state. | -| `src/grok/inject.ts` non-loopback cleanup | Credential-safety cleanup remains unconditional. | -| `/v1/responses`, `/v1/messages`, `/v1/messages/count_tokens` | They are shared transports, not client installation state. No desired-state check belongs in them. | -| Codex/Desktop mutation implementations | WP5 and WP6 own those operations. WP3 supplies only the state contract and automatic-path gates. | -| releases, publishing, deploys, tags, repository starring | No delivery or user-identity action belongs in a foundation phase. | +| `gui/` | WP3 adds the contract, not a new card. WP5 and WP6 consume it. | +| `src/codex/journal.ts` | Crash reconciliation repairs a half-applied Codex write regardless of desired state (`src/codex/journal.ts:148-162`). | +| `src/service.ts` stop/uninstall teardown | Teardown removes dead proxy pointers; it neither consults nor rewrites desired state (`src/service.ts:2587-2594`). | +| `src/grok/inject.ts` non-loopback cleanup | Credential-safety cleanup remains unconditional (`src/grok/inject.ts:359-380`). | +| `/v1/responses` | No client-specific gate guards it today. Codex OFF must not close the transport used by other clients. | +| Codex/Desktop remover implementations | WP5 and WP6 implement those two removers against this contract. Their shared-file work is sequential, not parallel. | +| releases, publishing, deploys, tags, repository starring | No delivery or user-identity action belongs in this phase. | -## The schema and its one reader +## The schema and effective-state reader -MODIFY `src/types.ts` immediately before `OcxConfig` (currently line 533), then -put the field beside `claudeCode` (currently line 544): +MODIFY `src/types.ts` immediately before `OcxConfig` (`src/types.ts:521-533`), +then put the field beside `claudeCode` (`src/types.ts:541-545`): ```diff export interface OcxApiKeyEntry { @@ -72,32 +86,19 @@ put the field beside `claudeCode` (currently line 544): + | "gajae"; + export interface OcxConfig { - port: number; ``` ```diff - /** One-time migration marker for Antigravity's static catalog default. */ - googleAntigravityStaticCatalogVersion?: 1; /** Claude Code inbound + launcher settings. */ claudeCode?: OcxClaudeCodeConfig; -+ /** -+ * The user's durable ON/OFF intent for each client integration, separate from -+ * whatever config happens to be present on disk right now. -+ * -+ * Missing entries deliberately mean ON. Existing installations pre-date this -+ * map, and treating absence as OFF would silently unplug working clients on the -+ * first upgraded start — the same restart path that currently resurrects a Grok -+ * fence after its shipped switch removed it. -+ */ ++ /** Durable user intent. Missing map/key means ON for upgrade compatibility. */ + clientIntegrations?: Partial>; - /** - * Up to 5 routed model ids ("/") to feature FIRST in the injected Codex catalog. ``` MODIFY `src/config.ts`. The parser salvages each known key independently. A -single hand-edited `"codex": "false"` becomes absent/ON, but it cannot discard a -valid `"grok": false` next to it; unknown future keys pass through so an older -binary does not erase a newer client's intent on save. +hand-edited `"codex": "false"` becomes absent/ON without discarding a valid +`"grok": false` beside it. The object stays `.passthrough()` so an older binary +does not erase a newer client's key on its next field-scoped mutation. ```diff import { @@ -114,8 +115,6 @@ binary does not erase a newer client's intent on save. ```diff const apiKeyEntrySchema = z.object({ key: z.string().refine(isUsableApiKeySecret), - // Degrades to "" here; every schema consumer then runs `normalizeApiKeyIds`, - // which fills it deterministically so the id is stable across loads. id: z.string().catch(""), name: z.string().catch(""), createdAt: z.string().catch(""), @@ -133,33 +132,23 @@ binary does not erase a newer client's intent on save. + kimi: z.boolean().optional().catch(undefined), + gajae: z.boolean().optional().catch(undefined), +}).passthrough(); -+ - const configSchema = z.object({ - port: z.number().int().min(0).max(65535).default(10100), ``` ```diff googleAntigravityStaticCatalogVersion: z.literal(1).optional().catch(undefined), -+ // Per-key catches preserve every valid OFF beside one malformed hand edit. A -+ // malformed whole map degrades to absent, which is the upgrade-safe ON default. + clientIntegrations: clientIntegrationsSchema.optional().catch(undefined), providerContextCaps: z.record(z.string(), z.number().int().positive()).optional(), ``` -Add the only effective-state reader beside the existing config feature gates -(`websocketsEnabled`, currently line 1909). No caller may open-code -`?.[client] ?? true`: Claude Code's old field is the transition exception. +Add the only effective-state reader beside `websocketsEnabled` +(`src/config.ts:1909-1911`). No caller may open-code `?.[client] ?? true` because +Claude Code's old field is the migration exception. ```diff export function websocketsEnabled(config: Pick): boolean { return config.websockets === true; } -+/** -+ * Resolve durable client intent without mistaking an absent upgrade-era key for -+ * an opt-out. Claude Code alone predates the shared map, so its old explicit OFF -+ * remains authoritative until a route has mirrored both representations. -+ */ +export function clientIntegrationEnabled( + config: Pick, + client: ClientIntegrationId, @@ -169,21 +158,6 @@ Add the only effective-state reader beside the existing config feature gates + if (client === "claude-code") return config.claudeCode?.enabled !== false; + return true; +} -+ -+/** Write the transition representation in one place so Claude OFF cannot split-brain. */ -+export function setClientIntegrationEnabled( -+ config: OcxConfig, -+ client: ClientIntegrationId, -+ enabled: boolean, -+): void { -+ config.clientIntegrations = { ...config.clientIntegrations, [client]: enabled }; -+ if (client === "claude-code") { -+ config.claudeCode = { ...(config.claudeCode ?? {}), enabled }; -+ } -+} -+ - // --------------------------------------------------------------------------- - // Hand-edit protection for the `claudeCode` subtree (devlog 260726_claude_auth_auto/040 H1). ``` Truth table: @@ -195,243 +169,152 @@ Truth table: | `true` | any | ON | | `false` | any | OFF | -The new key wins once present. Both Claude mutation routes write both, so the -legacy fallback can never migrate an existing Claude OFF back to ON. - -## Gate 1 — Codex's shared sync owner - -MODIFY `src/codex/sync.ts`. The return is a successful no-op because desired OFF -is policy, not a failed catalog refresh. The gate precedes the external-provider -branch too; that branch still calls `injectCodexConfig` (`sync.ts:56-70`). - -```diff --import { applyProxyEnv, loadConfig } from "../config"; -+import { applyProxyEnv, clientIntegrationEnabled, loadConfig } from "../config"; -``` - -```diff - export async function syncModelsToCodex( - port?: number, - config: OcxConfig = loadConfig(), - log: Pick | null = console, - deps: CodexSyncDeps = defaultDeps, - ): Promise { -+ if (!clientIntegrationEnabled(config, "codex")) { -+ return { -+ ok: true, -+ added: 0, -+ catalogPath: null, -+ catalogExists: false, -+ catalogWritten: false, -+ cacheSynced: false, -+ message: "Codex integration sync skipped: desired state is OFF.", -+ }; -+ } - const p = port ?? config.port ?? 10100; -``` - -This one gate covers `ocx start`, both `ocx ensure` branches, `POST /api/sync`, -`ocx sync`, `ocx restore back`, custom-model edits, and the direct CLI provider -sync caller (`src/cli/index.ts:318-341,365-411,756-829`, -`src/cli/models.ts:102-206`, `src/cli/provider.ts:235`). +## A2 — field-scoped persistence uses the primitive that already exists -## Gate 2 — provider/model catalog refreshes that bypass sync +The failed draft invented `saveConfigPreservingClaudeCode` as the desired-state +writer and mutated the request's live `config` before saving it. The shipped +Claude route still demonstrates the bug: it assigns `config.claudeCode = next` +at `src/server/management/native-integration-routes.ts:402-415`, then persistence +can refuse at `:420-431`. The regression test explains why a retry needs a fresh +object (`tests/native-claude-code-toggle.test.ts:213-218`). -MODIFY `src/server/management-api.ts`. `refreshCodexCatalogBestEffort` directly -calls `refreshCodexModelCatalog` today (`management-api.ts:105-112`), so putting -the check only in `syncModelsToCodex` leaves every provider/model/combo mutation -able to rewrite Codex artifacts. +Do not add that writer. `mutatePersistedConfig` already has the required real +contract (`src/config.ts:1825-1832,1854-1906`): -```diff - import { - DEFAULT_SUBAGENT_MODELS, -+ clientIntegrationEnabled, - codexAutoStartEnabled, +```ts +export function mutatePersistedConfig( + mutate: (config: OcxConfig) => { changed: boolean; value: T }, +): + | { status: "committed" | "unchanged"; value: T } + | { status: "unavailable"; reason: "missing" | "invalid" | "conflict" }; ``` -```diff - async function refreshCodexCatalogBestEffort(): Promise { -+ if (!clientIntegrationEnabled(config, "codex")) return; - if (deps.refreshCodexCatalog) return deps.refreshCodexCatalog(); - try { - const { refreshCodexModelCatalog } = await import("../codex/refresh"); - await refreshCodexModelCatalog(config); -``` - -The gate comes before the injected dependency. Otherwise tests can pass while a -production caller bypasses policy through a configured seam. - -## Gate 3 — every Grok startup/ensure caller - -MODIFY `src/grok/sync.ts`, before catalog fetch and before the writer. Do not add -`"disabled"` to `GrokInjectResult.skippedReason`: those values are writer policy -outcomes from `injectGrokConfig`; desired OFF never reaches that writer. - -```diff - import { visibleNativeSlugs, filterCatalogVisibleModels, nativeOpenAiContextWindow, type CatalogModel } from "../codex/catalog"; -+import { clientIntegrationEnabled } from "../config"; - import type { OcxConfig } from "../types"; -``` - -```diff - export async function syncGrokConfig( - port: number, - config: OcxConfig, - opts: { hostname?: string; grokHome?: string } = {}, - deps: GrokSyncDeps = { fetchAllModels: defaultFetchAllModels, injectGrokConfig }, - ): Promise { -+ if (!clientIntegrationEnabled(config, "grok")) { -+ return { -+ ok: true, -+ changed: false, -+ message: "Grok config sync skipped: desired state is OFF.", -+ }; -+ } - let models: GrokInjectModel[]; -``` - -This closes all three real callers: start and both ensure branches -(`src/cli/index.ts:334-341,372-379,398-404`) plus `/api/grok/apply`, whose flight -loads fresh persisted config before calling this helper -(`src/server/management/agent-settings-routes.ts:94-107,639-652`). - -## Gate 4 — Desktop auto-apply is two policies, not one - -MODIFY `src/server/management/agent-settings-routes.ts`. Desktop desired state -and `desktopAutoApply` answer different questions: “may opencodex manage Desktop?” -and “may provider changes rewrite the saved managed profile?” Both must allow the -write. - -```diff - import { - DEFAULT_SUBAGENT_MODELS, -+ clientIntegrationEnabled, - codexAutoStartEnabled, -``` +It reads the current disk bytes, clones before invoking the callback, re-runs the +callback against the latest snapshot, and commits the confirmed clone under the +shared SQLite mutation lock. Build the one-key mutation on top of that signature: ```diff - /** Best-effort Desktop 3P config auto-reconcile when providers change. */ - async function autoApplyDesktopBestEffort(): Promise { - try { -+ if (!clientIntegrationEnabled(config, "claude-desktop")) return; - if (config.claudeCode?.desktopAutoApply === false) return; - if (!config.claudeCode?.desktopProfile) return; ++export interface ClientIntegrationMutationValue { ++ config: OcxConfig; ++ desiredEnabled: boolean; ++} ++ ++export function mutateClientIntegrationEnabled( ++ client: ClientIntegrationId, ++ enabled: boolean, ++): PersistedConfigMutationOutcome { ++ return mutatePersistedConfig(config => { ++ const mapAlreadyMatches = config.clientIntegrations?.[client] === enabled; ++ const legacyAlreadyMatches = client !== "claude-code" ++ || config.claudeCode?.enabled === enabled; ++ if (mapAlreadyMatches && legacyAlreadyMatches) { ++ return { changed: false, value: { config, desiredEnabled: enabled } }; ++ } ++ config.clientIntegrations = { ...config.clientIntegrations, [client]: enabled }; ++ if (client === "claude-code") { ++ config.claudeCode = { ...(config.claudeCode ?? {}), enabled }; ++ } ++ return { changed: true, value: { config, desiredEnabled: enabled } }; ++ }); ++} ``` -An absent desired key and absent `desktopAutoApply` both preserve the current -auto-apply behavior. `desktopAutoApply: false` must never be migrated into -Desktop desired OFF (`003_durable_desired_state.md:106-115`). - -## Gate 5 — `ocx opencode` cannot inject around disk state +Only the callback-local clone is mutated. A route uses `outcome.value.config` for +the following file operation; it does not patch `ctx.config` before or after the +commit. A later status read loads persisted state. This prevents one long-lived +request object from overwriting a newer disk edit and makes lock refusal retryable +with the same object. -MODIFY `src/cli/opencode.ts`. The command builds `OPENCODE_CONFIG_CONTENT`, whose -provider block outranks global, project, and custom disk config -(`opencode.ts:461-477`). INFERRED decision: an explicit invocation while desired -OFF refuses with exit 1, matching `ocx claude`; launching an unwired OpenCode from -a command whose contract says “wired to the local proxy” would be a false green. - -```diff --import { loadConfig } from "../config"; -+import { clientIntegrationEnabled, loadConfig } from "../config"; -``` - -```diff - export async function cmdOpencode(args: string[]): Promise { - const config = loadConfig(); -+ if (!clientIntegrationEnabled(config, "opencode")) { -+ console.error("OpenCode integration is disabled (config.clientIntegrations.opencode=false — turn it ON before using `ocx opencode`)."); -+ return 1; -+ } - const live = await ensureProxyForOpencode(config); -``` +The helper touches only `clientIntegrations[client]`, plus +`claudeCode.enabled` for the one compatibility client. It preserves all other +client keys, providers, API settings, and unrelated `claudeCode` fields. -The gate precedes `ensureProxyForOpencode`; a disabled client command must not -start the proxy merely to refuse later. +Required persistence tests: -## Claude Code transition consumers +1. Two writers toggling different clients from the same stale starting object + both survive in the final file. +2. Simultaneous toggles of different clients preserve both keys; simultaneous + opposing toggles of one client serialize to one whole outcome, never a split + legacy/new representation. +3. A held mutation lock refuses without changing the live object; retry with the + same object succeeds after release. +4. Claude mirroring preserves `authMode`, `injectAgents`, `desktopProfile`, + `desktopAutoApply`, and unknown hand-edited Claude fields. -The new map is authoritative when present; mirroring is compatibility, not a -license for old consumers to open-code the legacy field forever. Replace the -three Claude-Code-specific automatic gates and the management agent-sync gate. +## A1 reversal — keep client-specific Claude ingress admission -MODIFY `src/cli/claude.ts`: +The earlier invariant was over-broad. Client installation state must not stop the +proxy or a different client, but ingress admission is itself a client contract. +`claudeCode.enabled` is the documented, shipped kill switch used before body work +for `/v1/messages` and `/v1/messages/count_tokens` +(`src/server/claude-messages.ts:65-69,543-548,868-872`) and before Anthropic model +discovery (`src/server/index.ts:493-502`). Removing those checks would turn a +legacy OFF into ON during upgrade. -```diff --import { loadConfig } from "../config"; -+import { clientIntegrationEnabled, loadConfig } from "../config"; -``` +KEEP every gate and change only the reader: ```diff - export async function cmdClaude(args: string[]): Promise { - const config = loadConfig(); ++import { clientIntegrationEnabled } from "../config"; +@@ + function claudeInboundDisabled(config: OcxConfig): Response | null { - if (config.claudeCode?.enabled === false) { -- console.error("Claude inbound is disabled (config.claudeCode.enabled=false — flip the Claude ON toggle in the GUI or edit config)."); + if (!clientIntegrationEnabled(config, "claude-code")) { -+ console.error("Claude Code integration is disabled — turn it ON before using `ocx claude`."); - return 1; + return anthropicErrorResponse(403, "Claude inbound is disabled (GUI: Claude ON toggle / config.claudeCode.enabled)", "permission_error"); } -``` - -MODIFY `src/claude/agents-inject.ts`: - -```diff --import { DEFAULT_SUBAGENT_MODELS, hasOwnProvider } from "../config"; -+import { clientIntegrationEnabled, DEFAULT_SUBAGENT_MODELS, hasOwnProvider } from "../config"; -``` - -```diff - export function injectClaudeAgentDefs(config: OcxConfig, windows: Record, configDir?: string): string[] | null { -- if (config.claudeCode?.enabled === false || config.claudeCode?.injectAgents === false) { -+ if (!clientIntegrationEnabled(config, "claude-code") || config.claudeCode?.injectAgents === false) { -``` - -MODIFY `src/server/system-env.ts`: - -```diff --import { getConfigDir } from "../config"; -+import { clientIntegrationEnabled, getConfigDir } from "../config"; + return null; + } ``` ```diff - export async function injectSystemEnv(port: number, config: OcxConfig): Promise { - if (process.platform !== "darwin") return { injected: false, reason: "not macOS" }; -- if (config.claudeCode?.enabled === false) return { injected: false, reason: "claude disabled" }; -+ if (!clientIntegrationEnabled(config, "claude-code")) return { injected: false, reason: "claude disabled" }; + const wantsAnthropicList = req.headers.get("anthropic-version") !== null + || url.searchParams.get("flavor") === "anthropic"; + if (wantsAnthropicList && !url.searchParams.has("client_version")) { +- if (config.claudeCode?.enabled === false) return jsonResponse({ data: [] }, 200, req, config); ++ if (!clientIntegrationEnabled(config, "claude-code")) { ++ return jsonResponse({ data: [] }, 200, req, config); ++ } ``` -MODIFY the already-open `src/server/management-api.ts` import above, then: +The corrected invariant is precise: -```diff - async function syncClaudeAgentDefsBestEffort(): Promise { - try { - const { injectClaudeAgentDefs } = await import("../claude/agents-inject"); -- if (config.claudeCode?.enabled === false || config.claudeCode?.injectAgents === false) { -+ if (!clientIntegrationEnabled(config, "claude-code") || config.claudeCode?.injectAgents === false) { -``` +| Surface | Desired OFF behavior | +|---|---| +| Codex `/v1/responses` | Remains admitted. No Codex client gate guarded this transport. | +| Claude Code `/v1/messages` and `/count_tokens` | Returns the shipped 403 because this is Claude Code ingress admission. | +| Anthropic-flavored model discovery | Returns an empty list while Claude Code is OFF, preserving the shipped kill switch. | +| Proxy lifecycle | Remains running. No toggle calls stop/restart/uninstall. | +| A different client's writer/transport | Remains available unless that different client's own desired key is OFF. | -## Persist desired intent before the Grok mutation +Compatibility activation: load a legacy file containing only +`claudeCode.enabled=false`, run the migration/load path with no new key, then hit +both Messages handlers and Anthropic discovery. Both handlers still return 403 +and discovery still returns `{ data: [] }`. This test must fail if either old +gate is removed. -MODIFY `src/server/management/native-integration-routes.ts`. The config write is -the intent commit; fence inspection/removal/injection is observation and may -refuse. Never roll the committed flag back because the file conflicted. +## A6 — the complete shared native contract, consumed rather than redefined -```diff --import { readRuntimePort, saveConfigPreservingClaudeCode } from "../../config"; -+import { clientIntegrationEnabled, readRuntimePort, saveConfigPreservingClaudeCode, setClientIntegrationEnabled } from "../../config"; -``` +The first roadmap dispatched WP5 and WP6 in parallel even though both edit +`native-integration-routes.ts` and its client union. Their proposed unions do not +compose: one adds Codex and the other adds Desktop. WP3 defines the final contract +once. WP5 runs first where shared files overlap; WP6 rebases on WP5 and runs +second. They may proceed independently only on disjoint files. -The response must keep the two states separate. `state` remains observed disk -state for compatibility; `desiredEnabled` is the persisted intent. +MODIFY `src/server/management/native-integration-routes.ts:31-74`: ```diff +-export type NativeIntegrationClientId = "claude" | "grok"; ++export type NativeIntegrationClientId = ++ | "codex" ++ | "claude" ++ | "claude-desktop" ++ | "grok"; +@@ export interface NativeStatus { clientId: NativeIntegrationClientId; state: "absent" | "current" | "unsafe"; + desiredEnabled: boolean; installed: boolean; -``` - -```diff +@@ export interface NativeToggleEnvelope { ok: true; clientId: NativeIntegrationClientId; @@ -439,227 +322,375 @@ state for compatibility; `desiredEnabled` is the persisted intent. state: NativeStatus["state"]; + desiredEnabled: boolean; message: string; -``` - -```diff +@@ export interface NativeRefusalEnvelope { error: string; code: "native_integration_refused" | "native_integration_failed"; clientId: NativeIntegrationClientId; reason: NativeRefusalReason; message: string; -+ desiredEnabled?: boolean; ++ desiredEnabled: boolean; + observedState?: NativeStatus["state"]; ++ residualPaths?: string[]; } ``` -Change `claudeCodeEnabled` into a compatibility alias and report desired state -from both GET rows: +The status helpers own the native-id mapping and make omission a type error: ```diff - /** Absent means ON: the six read sites all treat only an explicit `false` as off. */ - export function claudeCodeEnabled(config: ManagementContext["config"]): boolean { -- return config.claudeCode?.enabled !== false; -+ return clientIntegrationEnabled(config, "claude-code"); - } ++function desiredClientId(clientId: NativeIntegrationClientId): ClientIntegrationId { ++ return clientId === "claude" ? "claude-code" : clientId; ++} ++ ++function desiredEnabledForNative( ++ config: Pick, ++ clientId: NativeIntegrationClientId, ++): boolean { ++ return clientIntegrationEnabled(config, desiredClientId(clientId)); ++} ++ ++function withDesiredState( ++ config: Pick, ++ observed: Omit, ++): NativeStatus { ++ return { ++ ...observed, ++ desiredEnabled: desiredEnabledForNative(config, observed.clientId), ++ }; ++} ``` -```diff - return { - clientId: "claude", - state: claudeCodeEnabled(config) ? "current" : "absent", -+ desiredEnabled: claudeCodeEnabled(config), -``` +`claudeStatus`, `grokStatus`, and later `codexStatus`/`desktopStatus` return through +`withDesiredState`. Every success literal supplies `desiredEnabled`; every refusal +uses a single serializer that supplies the persisted intent and, after persistence, +the last observed state. WP5 and WP6 delete their local union/schema diffs and use +these helpers. + +The six file-client schema follows the same two-state rule. MODIFY +`src/integrations/state.ts:30-42` and the route envelopes at +`src/server/management/integration-routes.ts:44-55`: ```diff --function grokStatus(): NativeStatus { -+function grokStatus(config: ManagementContext["config"]): NativeStatus { - const seen = inspectGrokConfig(); + export interface IntegrationStatus { + clientId: IntegrationClientId; + state: IntegrationState; ++ desiredEnabled: boolean; + installed: boolean; ``` ```diff - return { - clientId: "grok", - state, -+ desiredEnabled: clientIntegrationEnabled(config, "grok"), - installed: seen.kind !== "not_installed", +-export type IntegrationToggleEnvelope = +- | ({ clientId: IntegrationClientId } & ApplyResult) +- | ({ clientId: IntegrationClientId } & DisableResult); ++export type IntegrationToggleEnvelope = ( ++ | ({ clientId: IntegrationClientId } & ApplyResult) ++ | ({ clientId: IntegrationClientId } & DisableResult) ++) & { desiredEnabled: boolean }; ``` +`readIntegrationState` is the status helper every surface already uses +(`src/integrations/state.ts:225-289`); add +`desiredEnabled: clientIntegrationEnabled(input.config, input.clientId)` to all +three return sites, including unsafe path-resolution and unreadable-file returns. + +## A3 — six file clients persist intent before touching their files + +The failed draft put `ocx opencode` behind a desired-state guard but gave +OpenCode, Pi, Hermes, OpenClaw, Kimi, and Gajae no writer for that state. The +real switch is `PUT /api/client-integrations/:clientId`, which currently goes +straight from body validation to `applyIntegration`/`disableIntegration` +(`src/server/management/integration-routes.ts:507-527`). Route it through the same +field-scoped mutation first: + ```diff - if (url.pathname === "/api/native-integrations" && req.method === "GET") { - const { getConfigPath } = await import("../../config"); - return jsonResponse({ -- clients: [claudeStatus(config, getConfigPath()), grokStatus()], -+ clients: [claudeStatus(config, getConfigPath()), grokStatus(config)], + const parsed = await readJsonBody(ctx); + if (parsed instanceof Response) return parsed; +@@ + try { ++ const persisted = mutateClientIntegrationEnabled(requestedClient, parsed.enabled); ++ if (persisted.status === "unavailable") { ++ return desiredStatePersistenceFailure(requestedClient, persisted.reason, ctx); ++ } ++ const operationConfig = persisted.value.config; +- const input = await buildIntegrationWriteInput(requestedClient, ctx, integrationStore()); ++ const input = await buildIntegrationWriteInput( ++ requestedClient, ++ ctx, ++ integrationStore(), ++ operationConfig, ++ ); + const result = await runClientIntegrationFlight( + requestedClient, + parsed.enabled ? "apply" : "disable", +- input.io?.now ?? Date.now, + () => Promise.resolve(parsed.enabled + ? applyIntegration(input) + : disableIntegration(input)), + ); +- if (!result.ok) return writerFailureResponse(requestedClient, result, ctx); +- return jsonResponse(result satisfies IntegrationToggleEnvelope, 200, req, ctx.config); ++ if (!result.ok) { ++ return writerFailureResponse(requestedClient, result, ctx, { ++ desiredEnabled: parsed.enabled, ++ observedState: readIntegrationState(input).state, ++ }); ++ } ++ return jsonResponse({ ++ ...result, ++ desiredEnabled: parsed.enabled, ++ } satisfies IntegrationToggleEnvelope, 200, req, operationConfig); ``` -In `handleGrokToggle`, persist immediately after body validation and before the -first inspector. If config persistence fails, do not touch the Grok file; that is -the only failure allowed to prevent the desired-state commit. +The helper takes the committed snapshot explicitly so model export and the writer +cannot fall back to the stale request object: ```diff - } - const enabled = body.enabled; -+ if (config.clientIntegrations?.grok !== enabled) { -+ const previousClientIntegrations = config.clientIntegrations; -+ setClientIntegrationEnabled(config, "grok", enabled); -+ const persist = deps.saveConfigPreservingClaudeCode ?? saveConfigPreservingClaudeCode; -+ try { -+ persist(config); -+ } catch (error) { -+ if (previousClientIntegrations === undefined) delete config.clientIntegrations; -+ else config.clientIntegrations = previousClientIntegrations; -+ if (isConfigLockError(error)) { -+ return isLockContention(error) -+ ? refusal(409, "grok", "config_busy", -+ "Another process is saving the configuration right now. Desired state was not changed; try again in a moment.") -+ : refusal(500, "grok", "write_failed", -+ `Desired state could not be saved: ${error instanceof Error ? error.message : String(error)}`); -+ } -+ throw error; -+ } -+ } + async function buildIntegrationWriteInput( + clientId: IntegrationClientId, + ctx: ManagementContext, + store: IntegrationStateStore, ++ config: OcxConfig = ctx.config, + ): Promise { + return { + clientId, +- models: await loadExportModels(ctx.config), +- config: ctx.config, +- port: Number(ctx.url.port) || ctx.config.port, ++ models: await loadExportModels(config), ++ config, ++ port: Number(ctx.url.port) || config.port, +``` + +Ordering and failure behavior are fixed: + +1. Invalid body: no intent and no file change. +2. Missing/invalid/conflicted config or lock refusal: no intent and no file change; + return retryable `config_busy` only for real contention. +3. Intent committed, file mutation succeeds: return desired and observed state. +4. Intent committed, file mutation refuses/fails: never roll intent back. Return + `desiredEnabled` plus freshly inspected `observedState` and the writer's recovery + fields. Startup/ensure/status retries convergence. + +Activation test: create a real temporary OpenCode config, disable OpenCode through +the real management route, then invoke `cmdOpencode`. The command must refuse before +proxy ensure/spawn, and the OpenCode file bytes must remain unchanged. This proves +the CLI guard is reachable from the state the real switch writes. + +## A4 — per-client single-flight and the last-moment write check + +Entry checks alone are racy. Codex awaits catalog work at +`src/codex/sync.ts:83-108` and then injects at `:110`; Grok awaits model discovery +at `src/grok/sync.ts:35-57` and then injects at `:61-65`. Either can start ON, +pause, persist OFF in another request, and write after OFF. + +NEW `src/integrations/desired-state.ts` owns one operation boundary for all ten +ids. It replaces route-local Grok/apply and six-client flight maps. The boundary +has two layers: + +- an in-process promise map joins an identical operation and refuses a competing + direction; +- an OS-backed SQLite transaction in a per-client coordinator file prevents a + CLI/startup process and the server's GUI/background process from writing the + same client concurrently. Separate files preserve concurrency between different + clients. Process exit releases the transaction; there is no stale lease row. + +Every GUI route, CLI writer, startup sync, ensure sync, Desktop auto-apply, Grok +apply, Codex refresh, and WP5/WP6 native mutation reaches +`runClientIntegrationFlight(clientId, operationKey, operation)`. It enters exactly +once at the lowest shared owner of the irreversible write: a route that delegates +to `syncGrokConfig` or `syncModelsToCodex` does not acquire an outer flight and +deadlock the same client. Direct strip/restore routes acquire it themselves. No +surface keeps its own map. + +The flight is necessary but not sufficient. Every irreversible write calls this +immediately before commit: + +```ts +export function requirePersistedClientIntent( + client: ClientIntegrationId, + expectedEnabled: boolean, +): { ok: true; config: OcxConfig } | { + ok: false; + reason: "desired_state_changed" | "desired_state_unavailable"; +}; +``` + +The helper reads a fresh valid disk snapshot. Missing or invalid state fails +closed; it never falls back to a stale request object. Apply/inject/spawn requires +ON. Disable/removal requires OFF. The check is placed after async catalog/model +work and after compare-before-write, directly before each of these boundaries: + +| Writer | Last-moment check | +|---|---| +| Codex catalog refresh | before catalog atomic replace | +| `injectCodexConfig` | after model resolution, before config/journal mutation | +| `injectGrokConfig` | after catalog resolution, before fenced-file write | +| six-client `commit` | after the existing byte recheck (`src/integrations/writer.ts:292-317,367-384`), before snapshot/file/record commit | +| Desktop profile/meta writers | before each selected-profile or metadata write | +| `cmdOpencode` | once at entry and again immediately before spawning with `OPENCODE_CONFIG_CONTENT` | - /* - * The inspector runs BEFORE either delegate, in BOTH directions (012 §In -``` +If the expected intent changed, the writer returns a typed skip/refusal and writes +nothing. A caller may retry under the new direction; it may not continue with the +old snapshot. + +Deterministic race test: pause Codex and Grok model resolution on a controlled +promise, persist that client OFF through the real mutation helper, release the +promise, and assert catalog/inject spies remain zero. Repeat one file client with +its commit hook. The observable proof is unchanged target bytes, not only a skip +message. + +## Automatic gates use the same owner -Every success envelope in this function adds `desiredEnabled: enabled`. Every -post-persist refusal adds `desiredEnabled: enabled` and the last observed state. -The ownership refusal is the regression case: +The entry gates remain useful because they avoid needless catalog work, but each +one enters the shared flight and still performs the last-moment check above. + +### Codex + +MODIFY `src/codex/sync.ts:49-55`: ```diff - const owned = assertNativeTeardownOwned(); -- if (!owned.ok) return refusal(409, "grok", "home_mismatch", owned.message); -+ if (!owned.ok) { -+ return jsonResponse({ -+ error: "native integration change refused", -+ code: "native_integration_refused", -+ clientId: "grok", -+ reason: "home_mismatch", -+ desiredEnabled: false, -+ observedState: "current", -+ message: `${owned.message} Desired OFF was saved; the observed Grok block is still present.`, -+ } satisfies NativeRefusalEnvelope, 409); -+ } +-import { applyProxyEnv, loadConfig } from "../config"; ++import { applyProxyEnv, clientIntegrationEnabled, loadConfig } from "../config"; +@@ + ): Promise { ++ if (!clientIntegrationEnabled(loadConfig(), "codex")) { ++ return codexDesiredStateSkip(); ++ } ++ return runClientIntegrationFlight("codex", "sync", async () => { + const p = port ?? config.port ?? 10100; ``` -Apply the same shape to orphaned-marker, late writer refusal, and catalog failure: -desired remains what was saved; `observedState` comes from the inspector rather -than from the requested direction. The route may say “desired OFF, observed -conflict”; it must never answer “still ON” as though the request disappeared. +Close the flight after the existing result return. `refreshCodexCatalogBestEffort` +at `src/server/management-api.ts:105-112` uses the same flight/reader rather than +a separate boolean check, so provider/model routes cannot bypass ordering. -## Mirror Claude Code during transition +### Grok -The native route currently skips persistence when legacy effective state already -matches (`native-integration-routes.ts:393-400`). That is no longer enough: an -old `{ claudeCode: { enabled: false } }` must acquire the new false key even -though its effective state is already OFF. +MODIFY `src/grok/sync.ts:29-35`: ```diff - const enabled = body.enabled; -- if (claudeCodeEnabled(config) === enabled) { -+ const alreadyMirrored = config.clientIntegrations?.["claude-code"] === enabled -+ && config.claudeCode?.enabled === enabled; -+ if (alreadyMirrored) { - return jsonResponse({ - ok: true, clientId: "claude", changed: false, - state: enabled ? "current" : "absent", -+ desiredEnabled: enabled, ++import { clientIntegrationEnabled, loadConfig } from "../config"; ++import { runClientIntegrationFlight } from "../integrations/desired-state"; +@@ + ): Promise { ++ if (!clientIntegrationEnabled(loadConfig(), "grok")) { ++ return { ok: true, changed: false, message: "Grok config sync skipped: desired state is OFF." }; ++ } ++ return runClientIntegrationFlight("grok", "sync", async () => { + let models: GrokInjectModel[]; ``` +Close the flight after injection. `/api/grok/apply` at +`src/server/management/agent-settings-routes.ts:639-657` deletes its local +`grokApplyFlight`; the shared owner covers start, both ensure branches, GUI apply, +toggle, and background work. + +### Desktop and Claude consumers + +Desktop auto-apply at `src/server/management/agent-settings-routes.ts:130-150` +requires both policies and enters the Desktop flight: + ```diff -- const next = { ...(config.claudeCode ?? {}), enabled }; -+ setClientIntegrationEnabled(config, "claude-code", enabled); -+ const next = config.claudeCode!; + async function autoApplyDesktopBestEffort(): Promise { + try { ++ if (!clientIntegrationEnabled(loadConfig(), "claude-desktop")) return; + if (config.claudeCode?.desktopAutoApply === false) return; ``` -All native Claude success envelopes add `desiredEnabled: enabled`. +`desktopAutoApply:false` is not migrated into Desktop OFF. Claude launcher, +agent injection, and system-env replace direct legacy reads with +`clientIntegrationEnabled(config, "claude-code")` as in the first draft. Claude +ingress and discovery retain their gates as specified in A1. + +### OpenCode -The older `/api/claude-code` route in -`src/server/management/agent-settings-routes.ts` already persists the legacy -field (`agent-settings-routes.ts:941,1060-1070`); mirror the map before that same -save rather than introducing a second write: +MODIFY `src/cli/opencode.ts:531-533`: ```diff - } - config.claudeCode = next; - // Stamp the migration sentinel on EVERY persist of this block. The migration reads -@@ - // would be converted into a sticky manual subscription by the next startServer, and - // auto would survive exactly one proxy lifetime with no way back. - if (!next.authModeMigratedAt) next.authModeMigratedAt = new Date().toISOString(); -+ if (body.enabled !== undefined) { -+ setClientIntegrationEnabled(config, "claude-code", next.enabled !== false); -+ } - const { saveConfigPreservingClaudeCode: save } = await import("../../config"); + export async function cmdOpencode(args: string[]): Promise { + const config = loadConfig(); ++ if (!clientIntegrationEnabled(config, "opencode")) { ++ console.error("OpenCode integration is disabled — turn it ON before using `ocx opencode`."); ++ return 1; ++ } + const live = await ensureProxyForOpencode(config); ``` -Import `setClientIntegrationEnabled` from `../../config` in the existing import -block. The setter runs after the migration sentinel because it reassigns -`config.claudeCode`; this order guarantees the mirrored object is the stamped -object that the existing save persists. +The command enters the OpenCode flight and repeats +`requirePersistedClientIntent("opencode", true)` immediately before spawn. OFF +must not start the proxy merely to refuse later. -## What must NOT be gated +## A5 — startup reconciliation: OFF means converge, not skip -| Surface | Required invariant | -|---|---| -| `src/codex/journal.ts:148-162` | `reconcileJournal` always repairs a dead process's stale Codex state. Desired OFF is not permission to leave a half-applied journal. | -| `src/integrations/writer.ts:171-223` | Path resolution, ownership, parse, drift, and compare-before-write checks always run when a mutation is requested. | -| `src/service.ts:2587-2594` | Stop restores native Codex and strips Grok's dead proxy pointer. It does not write `clientIntegrations`; stopping is not opting out. | -| `src/grok/inject.ts:359-380` | A non-loopback bind always strips the unsafe loopback fence, even when desired Grok state is ON. | -| `/v1/responses` | Codex desired OFF stops Codex config/catalog writes, not the Responses transport used by OpenCode, Pi, Hermes, OpenClaw, Kimi, and Gajae. | -| `/v1/messages` and `/v1/messages/count_tokens` | Claude Code/Desktop desired state stops client-specific wiring, not the Anthropic transport shared by both clients and external callers. | +Persist OFF, crash before the remover, restart: the first draft would skip future +apply and leave desired OFF / observed ON forever. Desired OFF is therefore a +converge instruction. -The last invariant exposes an already-shipped contradiction. Today -`claudeCode.enabled=false` returns 403 from both Messages handlers -(`src/server/claude-messages.ts:65-69,536-548,868-872`) and empties shared -Anthropic model discovery (`src/server/index.ts:493-502`). Remove those gates; -do not replace them with `clientIntegrationEnabled`. +NEW `src/integrations/reconcile.ts` exposes: -MODIFY `src/server/claude-messages.ts`: +```ts +export interface ClientReconcileResult { + clientId: ClientIntegrationId; + desiredEnabled: boolean; + observedState: "absent" | "current" | "stale" | "conflict" | "unsafe"; + resolved: boolean; + message: string; +} -```diff --function claudeInboundDisabled(config: OcxConfig): Response | null { -- if (config.claudeCode?.enabled === false) { -- return anthropicErrorResponse(403, "Claude inbound is disabled (GUI: Claude ON toggle / config.claudeCode.enabled)", "permission_error"); -- } -- return null; --} -- - async function readAnthropicBody(req: Request, budget: TranslatorBudget): Promise { +export async function reconcileDisabledClientIntegrations( + trigger: "startup" | "ensure" | "status", + options?: { only?: readonly ClientIntegrationId[] }, +): Promise; ``` +For every client whose fresh persisted intent is OFF: + +1. Inspect observed state. +2. If absent, report resolved without writing. +3. If applied/current/stale, enter that client's shared flight, re-read OFF, and + run the existing idempotent remover. +4. Re-inspect. Report `resolved:true` only when observed state is absent. +5. Preserve OFF and return an unresolved conflict for ownership, drift, unsafe + metadata, history lock, or write failure. Never report desired OFF as observed + OFF merely because the remover was attempted. + +The six file clients use `disableIntegration`; Grok uses `stripGrokConfig`; Claude +Code has no external artifact and resolves from the persisted admission flag. WP5 +registers Codex's `restoreNativeCodex` remover, then WP6 registers Desktop's +standard-mode remover. Registration is exhaustive over `ClientIntegrationId`, so +the final WP6 build cannot compile with either new native client omitted. + +Invoke reconciliation at these real boundaries: + ```diff - ): Promise { - logCtx.surface = "claude"; -- const disabled = claudeInboundDisabled(config); -- if (disabled) { -- if (logIds) addFinalRequestLog(logIds.requestId, logIds.start, logCtx, 403, { closeReason: "non_stream" }); -- return disabled; -- } - - let anthropicBody: unknown; + // src/cli/index.ts:169-177 + async function handleStart(options: { block?: boolean } = {}) { +@@ + const requestedPort = parsePortOption(); ++ await reconcileDisabledClientIntegrations("startup"); + if (!currentExternalCodexModelProvider()) reconcileJournal(); ``` ```diff - /** Documented approximation: serialize system+messages+tools, run the char estimator. */ - export async function handleClaudeCountTokens(req: Request, config: OcxConfig): Promise { -- const disabled = claudeInboundDisabled(config); -- if (disabled) return disabled; -- - let body: unknown; + // src/cli/index.ts:358-365 + async function handleEnsure() { + if (!currentExternalCodexModelProvider()) reconcileJournal(); ++ await reconcileDisabledClientIntegrations("ensure"); + const config = loadConfig(); ``` -MODIFY `src/server/index.ts`: +Both collection and per-client GET routes call status reconciliation for the ids +being read before `readIntegrationState`/native status helpers run. Status returns +the unresolved result in `disableBlocked` or response diagnostics; it does not +hide a conflict and does not flip desired state back ON. -```diff - const wantsAnthropicList = req.headers.get("anthropic-version") !== null - || url.searchParams.get("flavor") === "anthropic"; - if (wantsAnthropicList && !url.searchParams.has("client_version")) { -- if (config.claudeCode?.enabled === false) return jsonResponse({ data: [] }, 200, req, config); - // Build Desktop 3P registry so inbound alias resolution works for subsequent requests. -``` +Crash-point tests use a hook immediately after `mutateClientIntegrationEnabled` +returns and before the remover begins. Terminate the simulated request there, +then invoke each of startup, ensure, and status reconciliation. For OpenCode and +Grok, assert the previously applied bytes are removed. For a drift/ownership +fixture, assert bytes remain, desired stays false, and the unresolved conflict is +reported. WP5 and WP6 add the same crash point for Codex and Desktop when their +removers land. ## Test plan @@ -667,113 +698,89 @@ MODIFY `src/server/index.ts`: | Case | Activation and assertion | |---|---| -| Absent-config upgrade | Load a config with no `clientIntegrations`; every id is effective ON. This is C3's upgrade case, not merely a helper call with a fabricated object. | -| Missing key / explicit true / explicit false | For every id: absent and true are ON; only false is OFF. | -| Claude legacy fallback | New key absent + `claudeCode.enabled=false` is OFF; absent/true is ON. | -| New Claude key wins | New true overrides legacy false, and new false overrides legacy true. | -| Per-key malformed salvage | Persist `{ codex: "false", grok: false }`; load yields Codex ON and Grok OFF, without falling back to a default config or losing providers. | -| Future-key preservation | An unknown boolean key survives load/save so an older binary does not erase a newer client's intent. | -| Setter mirroring | `setClientIntegrationEnabled(..., "claude-code", value)` writes both keys and preserves every unrelated Claude field. Other ids touch only the map. | +| Absent-config upgrade | Load a file with no map; all ten ids are ON. | +| Claude legacy fallback | New key absent + legacy false is OFF; new key wins once present. | +| Per-key malformed salvage | `{ codex: "false", grok: false }` yields Codex ON and Grok OFF without losing providers. | +| Two stale writers | Different client toggles both survive because each callback rebases on latest disk. | +| Simultaneous toggles | Different client keys both commit; neither whole-object snapshot wins. | +| Lock refusal and retry | Refusal changes neither disk nor live object; retry with that same object succeeds. | +| Claude field preservation | Mirroring changes only the new key and legacy `enabled`; every unrelated Claude field survives. | ### `tests/client-integration-auto-gates.test.ts` (NEW) -| Gate | Activation and observable proof | +| Case | Activation and assertion | |---|---| -| Codex sync OFF | Inject spies for catalog refresh and `injectCodexConfig`; both remain at zero, including the external-provider branch. Result is the explicit successful skip. | -| Codex absent/ON | The same spies fire, pinning upgrade behavior. | -| Direct management refresh OFF | Trigger a provider and a custom-model route with `refreshCodexCatalog` injected; count stays zero. This proves the bypass gate, not `syncModelsToCodex`. | -| Grok sync OFF | Inject fetch and writer spies; neither fires. Repeat with no map and prove both fire. | -| Desktop two-key gate | Exercise provider mutation with a saved Desktop profile for all four combinations of desired ON/OFF and `desktopAutoApply` true/false; write occurs only when both policies allow it. | -| OpenCode OFF | Invoke the command through injectable launch seams; proxy ensure, catalog fetch, env build, and spawn remain uncalled, exit is 1. | -| Claude compatibility consumers | New-map false with legacy field absent blocks launcher/system-env/agent writes; absent new key + legacy false does the same. | - -### Route regressions - -MODIFY `tests/native-grok-toggle.test.ts`: - -1. Disable persists `clientIntegrations.grok=false` before `stripGrokConfig`. -2. Ownership refusal leaves that persisted false and returns - `desiredEnabled:false`, `observedState:"current"`. -3. Orphaned fence, late writer refusal, and catalog failure keep the requested - intent and report observed state; none rolls the flag back. -4. A config-lock failure calls neither strip nor inject and says desired state was - not saved. -5. The next `syncGrokConfig` with the saved config calls neither catalog nor - writer — the exact `ocx start` regression. - -MODIFY `tests/native-claude-code-toggle.test.ts` and -`tests/claude-management-api.test.ts`: both routes mirror old/new values; an old -legacy OFF plus absent new key is not treated as a no-op; unrelated Claude fields -and the auth-mode migration sentinel survive. - -### `tests/client-integration-transport-isolation.test.ts` (NEW) - -Start the real Bun proxy with `clientIntegrations.codex=false` and -`clientIntegrations["claude-code"]=false`. Assert `/healthz` stays healthy; -an invalid `/v1/responses` request reaches its normal validation response rather -than an integration-disabled response; invalid `/v1/messages` and -`/v1/messages/count_tokens` requests return their normal 400 contract, never the -old 403. Fetch Anthropic model discovery and assert it is not emptied by Claude -Code OFF. This is the case proving a disabled client does not break another -client's transport. - -MODIFY `tests/claude-messages-endpoint.test.ts:786-803` to remove the old test -that requires 403. Keeping it would encode the C4 violation as a regression. +| Codex/Grok OFF at entry | Catalog and writer spies stay zero. | +| Codex/Grok OFF during fetch | Pause resolution, persist OFF, release; target bytes and writer counts remain unchanged. | +| Six-client last-moment check | Flip direction at the commit hook; no file/snapshot/record is written. | +| Desktop two-policy gate | Write occurs only when desired ON and `desktopAutoApply` permits it. | +| Real OpenCode activation | Disable through real PUT, invoke `cmdOpencode`; ensure/spawn stay zero and config bytes are unchanged. | +| Shared-flight coverage | GUI, CLI, startup, ensure, and background callers for one client cannot overlap; a different client can proceed. | + +### Route and compatibility regressions + +- `tests/management-integration-routes.test.ts`: all six PUTs persist intent before + file work; post-persist refusal returns required desired plus observed state. +- `tests/native-grok-toggle.test.ts`: same ordering, lock refusal/retry, and no + rollback of intent after ownership/catalog/write failure. +- `tests/native-claude-code-toggle.test.ts`: legacy OFF mirrors even when effective + state already matches; a failed persist leaves the supplied config object + untouched and the same object can retry. +- `tests/claude-management-api.test.ts`: the older route uses the same field-scoped + mutation and preserves migration sentinels/other Claude fields. +- `tests/claude-messages-endpoint.test.ts`: legacy `enabled:false` still returns + 403 from Messages and count-tokens; Anthropic discovery remains empty. + +### `tests/client-integration-reconciliation.test.ts` (NEW) + +For each current remover, persist OFF and stop at the post-persist/pre-mutate hook. +Run startup, ensure, and status reconciliation independently and prove observed +state becomes absent. Fault fixtures prove drift/ownership/unsafe removals remain +unresolved and visible without changing desired OFF. WP5/WP6 append Codex/Desktop +cases sequentially when those removers exist. ## Verification -Static and suite gates: - ```bash bun test tests/client-integration-desired-state.test.ts bun test tests/client-integration-auto-gates.test.ts -bun test tests/native-grok-toggle.test.ts tests/native-claude-code-toggle.test.ts tests/claude-management-api.test.ts -bun test tests/client-integration-transport-isolation.test.ts tests/claude-messages-endpoint.test.ts +bun test tests/client-integration-reconciliation.test.ts +bun test tests/management-integration-routes.test.ts tests/native-grok-toggle.test.ts +bun test tests/native-claude-code-toggle.test.ts tests/claude-management-api.test.ts +bun test tests/claude-messages-endpoint.test.ts bun run typecheck bun run test bun run privacy:scan ``` -Live proof uses the already-running proxy at `localhost:10100`; a green suite is -not restart persistence: - -1. Record `curl -fsS http://localhost:10100/healthz` and its `pid`. -2. Through the authenticated dashboard/API, turn Grok OFF. Confirm - `GET /api/native-integrations` reports `desiredEnabled:false` and observed - `absent`, or the explicit observed conflict if ownership/drift refused removal. -3. Run `ocx ensure`, then re-read both the status and `~/.grok/config.toml`. - Desired remains false and no managed fence reappears. Repeat after a real - proxy restart; `/healthz` returns with a new PID and Grok remains OFF. -4. Turn Codex OFF in the WP5 surface, run `POST /api/sync` and one provider edit, - then prove neither Codex config nor catalog artifact changed. `/healthz` remains - healthy. -5. With Claude Code desired OFF, send an invalid body to both shared paths: - - ```bash - curl -sS -o /tmp/ocx-messages-proof.json -w '%{http_code}\n' \ - -H 'content-type: application/json' -d '{}' \ - http://localhost:10100/v1/messages - curl -sS -o /tmp/ocx-count-proof.json -w '%{http_code}\n' \ - -H 'content-type: application/json' -d '{}' \ - http://localhost:10100/v1/messages/count_tokens - ``` - - Both reach normal request validation (400), not integration policy (403). - Read the bodies back; a status code without the response body is not proof of - which branch ran. -6. Re-read `/healthz`; the proxy stayed serving throughout. Compare the PID with - step 1 for the no-stop toggle operations and with the post-restart PID for the - restart persistence case. +Live activation proof: + +1. Record `/healthz` and PID. +2. Disable Grok and OpenCode through their real management routes. Confirm each + status has `desiredEnabled:false` and honest observed state. +3. Run `ocx ensure`, restart the proxy, and read the target files. Neither managed + contribution reappears; `/healthz` returns with the proxy still serving. +4. Disable OpenCode, run `ocx opencode`, and observe refusal before ensure/spawn. +5. With a legacy-only `claudeCode.enabled=false`, call Messages, count-tokens, and + Anthropic discovery. Observe 403, 403, and an empty model list. Then call an + invalid `/v1/responses` request and observe its normal validation response, + never a client-disabled response. +6. Inject a post-persist crash for one file client, restart, and observe the + remover converge it to absent. Repeat with drift and observe the unresolved + conflict while desired remains OFF. ## Accept criteria | Roadmap criterion | WP3 closure | |---|---| -| C2 — disabled survives restart, ensure, and `/api/sync` | Grok OFF is persisted before mutation; the shared Grok sync and Codex sync/direct-refresh owners are gated. Tests activate each path, and live proof checks the real fence after ensure and restart. | -| C3 — absent config changes nothing on upgrade | The absent-map load test proves every integration effective ON; missing keys and explicit true remain ON. Claude's absent-key fallback preserves a legacy explicit OFF. | -| C4 — disable never stops proxy or another transport | No lifecycle path is touched. Journal, teardown, credential cleanup, `/v1/responses`, and `/v1/messages` remain unconditional; the existing Claude transport/model-discovery gates are removed and the real proxy isolation test proves reachability. | - -WP3 is complete only when desired and observed state can disagree honestly. A -successful file removal with no persisted flag is still the shipped Grok bug; a -persisted OFF reported as observed OFF when the file is still present is a new -lie, not a fix. +| C2 — disabled survives restart, ensure, and `/api/sync` | Every real switch writes intent; every automatic writer reads it; startup/ensure/status converge residual applied state. | +| C3 — absent config changes nothing on upgrade | Missing map/key is ON for all ten clients, except legacy Claude explicit OFF remains OFF. | +| C4 — disable never stops proxy or another client | No lifecycle operation is added; `/v1/responses` stays ungated; Claude's own ingress admission remains gated; every other client is governed only by its own key. | +| Coordination | `mutatePersistedConfig` prevents stale whole-object saves, the per-client flight orders every surface, and each writer re-reads intent immediately before commit. | +| Shared contract | Native union is `codex | claude | claude-desktop | grok`; status/success always include `desiredEnabled`; WP5 then WP6 consume it sequentially. | + +WP3 is complete only when desired and observed state can disagree honestly and +the system keeps trying to reconcile that disagreement. A removed file with no +persisted intent is still the shipped Grok bug. A persisted OFF reported as +observed OFF while bytes remain is a new lie. A legacy Claude OFF that accepts +traffic again is a compatibility regression. diff --git a/devlog/_plan/260803_codex_desktop_toggle/040_codex_toggle.md b/devlog/_plan/260803_codex_desktop_toggle/040_codex_toggle.md index 06f14751f..e6ce8bd71 100644 --- a/devlog/_plan/260803_codex_desktop_toggle/040_codex_toggle.md +++ b/devlog/_plan/260803_codex_desktop_toggle/040_codex_toggle.md @@ -1,17 +1,49 @@ -# WP5 — the Codex CLI toggle, with artifact-level restore truth +# WP5 — the Codex GUI and CLI toggle, with artifact-level restore truth Research: `001_native_restore_thesis.md`. Read it first; this doc is the diff. The failure this phase closes is concrete: `restoreNativeCodex()` can leave routed threads hidden when the history database is locked, yet return `success: true` because that boolean is copied from config restore alone -(`src/codex/inject.ts:783-794`). Disable and enable already exist as -`restoreNativeCodex()` and `syncModelsToCodex(port)` while the proxy keeps serving -(`src/cli/index.ts:745-768`); WP3 already adds the durable, default-ON -`clientIntegrations.codex` intent (`003_durable_desired_state.md:87-115`). This -phase adds the missing artifact-level result, classifies the held-history failure, -registers Codex in the existing native-integration route family, and gives the -overview card an honest switch. It adds no operation journal or lifecycle engine. +(`src/codex/inject.ts:764-794`). There is a second false green in the existing +CLI: `ocx restore back` calls `syncModelsToCodex()` at +`src/cli/index.ts:756`, treats any `ok` result as applied, and prints “now routes +through opencodex” at `src/cli/index.ts:763`; the native restore call is at +`src/cli/index.ts:768`, not the stale `src/cli/index.ts:745` citation used by the +research note. If WP3 turns desired OFF into a bare +successful skip, that command claims a write which did not happen. In the other +direction, `ocx restore` restores native artifacts but records no OFF, so startup's +sync at `src/cli/index.ts:319` can put the routing back. + +**The earlier CLI-out-of-scope position is reversed.** WP3's gate changes the +meaning of existing CLI commands, so WP5 owns their user-visible result and +exit-code behavior. The decision is: **`ocx restore` / `ocx eject` persist desired +OFF; `ocx restore back` / `ocx eject back` persist desired ON.** These are explicit +user integration actions, unlike stop, shutdown, or uninstall cleanup, which keep +their existing intent-neutral behavior. This makes CLI and GUI two front ends to +the same desired-state transition and makes either result survive restart. + +This phase adds the missing artifact-level result, classifies the held-history +failure, consumes WP3's shared native-integration contract and per-client +coordinator, registers Codex in that route family, and gives the overview card an +honest switch. It adds no operation journal or lifecycle engine. + +## Dependency and landing order + +WP3 lands first and owns the complete shared contract: the +`"codex" | "claude" | "claude-desktop" | "grok"` native-client union, status and +success envelopes with **required** `desiredEnabled`, the refusal envelope and +helper, desired/observed status helpers, field-scoped intent mutation, and the +per-client single-flight. WP5 imports and extends that contract; it does not +redeclare any of those owners (`020_desired_state.md:172-230,294-367,479-534,623-661`). +WP5 then lands **before WP6**. This is sequential, +not parallel, because WP5 and WP6 both touch +`src/server/management/native-integration-routes.ts`, +`gui/src/pages/integrations/native-api.ts`, +`gui/src/pages/integrations/overview-clients.ts`, +`gui/src/pages/integrations/IntegrationsOverview.tsx`, refusal copy, locale files, +and shared GUI tests (`050_desktop_toggle.md:50-71,104-119,376-382,722-781`). WP6 rebases +its Desktop additions onto WP5's Codex additions. ## IN / OUT @@ -21,6 +53,16 @@ IN: classified reason instead of reducing it to `null`. - `src/codex/inject.ts` (MODIFY) — return config/catalog/history results and make aggregate success mean all required artifacts succeeded. +- `src/codex/sync.ts` (MODIFY) — consume WP3's coordinator and return a + discriminated disabled skip; re-read desired state at the catalog/cache and + config/history write boundaries. +- `src/cli/index.ts` (MODIFY) — persist explicit restore intent, distinguish + applied/disabled/failed sync outcomes, and print no applied claim on a skip. +- `src/cli/models.ts` and `src/cli/provider.ts` (MODIFY) — report an intentional + disabled skip without turning the provider/custom-model mutation into a false + failure or a silent sync success. +- `src/server/management/config-routes.ts` (MODIFY) — preserve the discriminated + skip in `POST /api/sync` instead of flattening it through `ok`. - `src/server/management/context.ts` (MODIFY) — add Codex mutation seams so route tests cannot touch the developer's real Codex home. - `src/server/management/native-integration-routes.ts` (MODIFY) — add Codex to @@ -42,11 +84,14 @@ IN: OUT: -- `src/codex/sync.ts` — enable delegates to the existing full catalog refresh + - injection path at lines 83-110; changing it is not needed. -- `src/cli/index.ts`, `src/server/management-api.ts`, and `src/service.ts` — their - existing `restoreNativeCodex().success` checks become more truthful through the - widened return type; no lifecycle caller needs a new state machine. +- `src/server/management-api.ts`'s route dispatcher and `src/service.ts` — no new + lifecycle state machine. The coordinator and gates they call are WP3-owned. +- Stop, signal shutdown, service teardown, and uninstall desired-state writes — + their calls at `src/cli/index.ts:257,528,591` restore dead pointers as cleanup + but do not opt the user out. Only the explicit restore/eject command at + `src/cli/index.ts:745-790` changes desired Codex intent. The cleanup writers + still use WP3's Codex flight; they use its unconditional `teardown` policy, not + the OFF-only toggle policy. - `/v1/responses` and every data-plane router — a client flag gates automatic Codex config writes, never the shared transport (`003_durable_desired_state.md:117-130`). - `src/integrations/writer.ts`, operation records, snapshots, undo routes, and the @@ -54,10 +99,146 @@ OUT: is `syncModelsToCodex`, not replay. - `gui/dist`, docs publishing, releases, deployment, and any live proxy mutation. +## A disabled sync is not an applied sync + +WP3 owns the gate and coordinator; WP5 consumes its result in every existing CLI +caller. Against the current `CodexSyncResult` at `src/codex/sync.ts:9-22`, WP3's +contract is discriminated instead of returning only `ok: true` with zero counts: + +```diff +export interface CodexSyncResult { ++ /** `skipped` is policy truth, never evidence that Codex was written. */ ++ status: "applied" | "skipped"; + ok: boolean; ++ skippedReason?: "desired_disabled" | "desired_state_changed" ++ | "desired_state_unavailable"; +@@ + } +``` + +Every current applied return at `src/codex/sync.ts:61-70,114-129` adds +`status: "applied"`. Desired OFF returns this exact shape before catalog work: + +```ts +{ + status: "skipped", + skippedReason: "desired_disabled", + ok: true, + added: 0, + catalogPath: null, + catalogExists: false, + catalogWritten: false, + cacheSynced: false, + message: "Codex integration is OFF; no Codex config, catalog, cache, or history was changed.", +} +``` + +`ok: true` means the policy was honored; `status` says whether the requested write +was applied. No caller may infer application from `ok` alone. The API preserves +both fields. `POST /api/sync` at `src/server/management/config-routes.ts:261-268` +returns HTTP 200 for a disabled policy skip and HTTP 500 only for an attempted +apply with `ok: false`; its JSON still contains `status` and `skippedReason`. + +## CLI is an explicit desired-state surface + +The current restore block is `src/cli/index.ts:745-790`. Amend that real block, +using WP3's field-scoped desired-state mutation and Codex single-flight owner: +`mutateClientIntegrationEnabled()` is WP3's field-scoped owner. ON delegates to +`syncModelsToCodex()`, which acquires the shared flight itself; OFF wraps the +direct restore in that flight. + +```diff + case "restore": + case "eject": { + if (args[1] === "back") { +@@ + if (!live) { + console.error("No running proxy found. Run 'ocx start' — it injects opencodex automatically."); + process.exit(1); + } ++ // Explicit enable: commit desired ON before entering sync's Codex flight. ++ const desired = mutateClientIntegrationEnabled("codex", true); ++ if (desired.status === "unavailable") { ++ console.error(`Codex desired state was not saved (${desired.reason}).`); ++ process.exitCode = desired.reason === "conflict" ? 2 : 1; ++ break; ++ } +- const synced = await syncModelsToCodex(live.port); ++ const synced = await syncModelsToCodex(live.port, desired.value.config); +- if (!synced.ok) { +- process.exitCode = 1; ++ if (synced.status === "skipped") { ++ // OFF won a later serialized transition. Never print the applied claim. ++ process.exitCode = 2; ++ console.error("Codex integration is OFF; restore back did not change Codex. Retry after the competing integration change finishes."); ++ break; ++ } ++ if (!synced.ok) { ++ process.exitCode = 1; + console.error("Plain `codex` was not switched back to opencodex. Fix the reported Codex config issue and retry."); + break; + } +@@ + break; + } ++ // Explicit disable: unlike stop/uninstall cleanup, restore/eject records OFF. ++ const desired = mutateClientIntegrationEnabled("codex", false); ++ if (desired.status === "unavailable") { ++ console.error(`Codex desired state was not saved (${desired.reason}).`); ++ process.exitCode = desired.reason === "conflict" ? 2 : 1; ++ break; ++ } + let r: { success: boolean; message: string }; +@@ +- try { +- r = restoreNativeCodex(); +- } catch (err) { +- r = { success: false, message: err instanceof Error ? err.message : String(err) }; +- } ++ try { ++ r = await runClientIntegrationFlight( ++ "codex", ++ "disable", ++ () => Promise.resolve(restoreNativeCodex({ ++ beforeWrite: () => requirePersistedClientIntent("codex", false), ++ })), ++ ); ++ } catch (err) { ++ r = { success: false, message: err instanceof Error ? err.message : String(err) }; ++ } +@@ +- if (r.success) { +- console.log("Plain `codex` now runs natively (no proxy). Switch back with: ocx restore back"); ++ if (r.success) { ++ console.log("Codex integration is OFF and plain `codex` now runs natively. Switch back with: ocx restore back"); +``` + +Exit `2` means “the requested integration transition was not applied because a +retryable policy/config flight won”; exit `1` means a non-retryable mutation or +artifact failure; exit `0` means the explicit transition reached its stated +observed result. This distinction matters to scripts without relabeling a +deliberate `ocx sync` policy skip as corruption. + +Every direct `syncModelsToCodex()` caller branches on `status` before `ok`: + +| Caller | Disabled output | Exit / parent outcome | +|---|---|---| +| startup (`src/cli/index.ts:319`) | one indented line: `Codex integration OFF; startup left Codex native.` | proxy start remains 0 | +| ensure, live and newly started (`src/cli/index.ts:367-369,409-411`) | same explicit skip, then the existing proxy-running line | ensure remains 0 | +| `restore back` (`src/cli/index.ts:751-764`) | never prints “now routes”; prints the competing-OFF error above | 2 for skipped, 1 for attempted failure | +| `sync` (`src/cli/index.ts:827-842`) | `Codex integration is OFF; sync skipped and no Codex files changed.` | 0: requested policy is already satisfied | +| custom-model refresh (`src/cli/models.ts:102-107`) | `Custom model saved; Codex integration is OFF, so its catalog was not changed.` | parent mutation remains 0 | +| provider `--sync` (`src/cli/provider.ts:232-239`) | `Provider saved; Codex integration is OFF, so Codex sync was skipped.` | provider mutation remains 0 | +| `POST /api/sync` / `ocx system sync` (`src/server/management/config-routes.ts:261-268`, `src/cli/system-command.ts:104-107`) | structured `status:"skipped", skippedReason:"desired_disabled"`; human formatter prints the message | HTTP/CLI 0 | + +Background provider/model refreshes do not print, but they must return/record the +same skipped reason for tests and diagnostics. A silent background no-op is +acceptable; a foreground command claiming an apply is not. + ## The structured result MODIFY `src/codex/history-provider.ts` at the current -`CodexHistorySyncResult` (`:162-168`): +`CodexHistorySyncResult` (`src/codex/history-provider.ts:162-168`): ```ts export type CodexHistoryFailureReason = "busy" | "permission"; @@ -127,9 +308,17 @@ export interface CodexNativeRestoreResult { history: CodexRestoreHistoryResult; }; } + +export interface CodexNativeRestoreOptions { + /** Called separately immediately before config/journal, catalog, and history writes. */ + beforeWrite?: () => + | { ok: true; config: OcxConfig } + | { ok: false; reason: "desired_state_changed" | "desired_state_unavailable" }; +} ``` -`restoreNativeCodex(): CodexNativeRestoreResult` keeps the existing operation +`restoreNativeCodex(options: CodexNativeRestoreOptions = {}): +CodexNativeRestoreResult` keeps the existing operation order but catches and records each artifact boundary separately. Config uses `restoreJournalState()` and then the existing `removeCodexConfig()` fallback (`src/codex/inject.ts:770-774`); catalog delegates once to @@ -157,7 +346,8 @@ Request: { enabled: boolean } ``` -Success (`200`), using the existing envelope and adding optional Codex detail: +Success (`200`), consuming WP3's envelope and adding optional Codex detail. +`desiredEnabled` is required; omitting it was the WP5/WP6 composition bug: ```ts { @@ -165,22 +355,29 @@ Success (`200`), using the existing envelope and adding optional Codex detail: clientId: "codex"; changed: boolean; state: "absent" | "current" | "unsafe"; + desiredEnabled: boolean; message: string; reason?: "external_provider_preserved" | "catalog_warning"; + externalProvider?: string; artifacts?: CodexNativeRestoreResult["artifacts"]; } ``` -Disable persists `clientIntegrations.codex = false` first, then checks teardown -ownership and calls `restoreNativeCodex()`. That order is intentional: WP3 says +Disable enters WP3's Codex flight, persists `clientIntegrations.codex = false` +through its field-scoped `mutatePersistedConfig()` owner, then checks teardown +ownership and calls structured `restoreNativeCodex()`. That order is intentional: desired OFF survives an ownership refusal or drift so a later automatic apply -cannot reverse the user's request (`003_durable_desired_state.md:112-115`). Use a -cloned config for persistence, then update the request's in-memory config only -after persistence succeeds; a failed config lock must not create an in-memory-only -OFF. Enable likewise persists `true`, resolves the running listener from +cannot reverse the request. `mutatePersistedConfig()` already clones, rebases, +freshness-checks, and commits under the shared config lock +(`src/config.ts:1846-1906`); WP5 must not clone a long-lived request config and +whole-file-save it. + +Enable uses the same flight, persists true, resolves the running listener from `readRuntimePort(process.pid)` with request/config fallback, and calls -`syncModelsToCodex(port, config, null)`. Bare `injectCodexConfig()` is forbidden: -it does not rebuild routed catalog rows (`src/codex/sync.ts:83-110`). +`syncModelsToCodex(port, freshlyLoadedConfig, null)`. Bare +`injectCodexConfig()` is forbidden: the full sync refreshes the catalog before +injecting (`src/codex/sync.ts:83-110`). Every success response reports the +freshly re-read desired value, not the request body copied back as fact. GET `/api/native-integrations` adds a Codex row. Its `state` is observed routing from `getCodexRoutingKind()` (`src/codex/inject.ts:255-273`), not merely desired @@ -189,29 +386,31 @@ intent: `opencodex-local` is `current`, `native` is `absent`, and `model_provider` explains it, in which case it is `absent` with `reason: "external_provider_preserved"` and a message naming that provider. `disableBlocked` carries `home_mismatch` only while teardown would touch our -artifacts. Consume WP3's `clientIntegrationEnabled()` and -`setClientIntegrationEnabled()` owners (`020_desired_state.md:149-183`); no WP5 -caller open-codes the map's defaulting rule. +artifacts. Consume WP3's status helper and effective-state reader; no WP5 caller +open-codes the map's defaulting rule. The current route has no desired field +(`src/server/management/native-integration-routes.ts:41-74`); WP3 adds it before +this diff lands. -Every refusal uses the existing -`refusal(status, clientId, reason, message)` function unchanged -(`src/server/management/native-integration-routes.ts:76-87`): +Every refusal consumes WP3's widened refusal helper, including desired and last +observed state after persistence. WP5 does not call the old four-argument helper +at `src/server/management/native-integration-routes.ts:76-87`: | HTTP | reason | Trigger | Observable state | |---|---|---|---| -| 409 | `config_busy` | WP3 desired-state persistence loses a real `SQLITE_BUSY` lock race | No durable intent or Codex artifact changed; retry is correct | +| 409 | `config_busy` | WP3 desired-state persistence or Codex single-flight loses a real contention race | No unreported Codex write occurs; retry is correct | | 409 | `home_mismatch` | disable sees an installed service owned by another Codex/OpenCodex home | Desired OFF is durable; Codex artifacts are untouched | | 409 | `history_busy` | config and catalog restored, history retries exhaust on busy/locked contention | Desired OFF is durable; native routing is active, but routed threads remain hidden until retry | | 500 | `history_permission` | config and catalog restored, history fails with `EPERM`/`EACCES` | Desired OFF is durable; user must fix permissions, not wait | | 500 | `write_failed` | desired-state persistence cannot open its lock, config/catalog restore fails, or enable sync returns `ok: false` | Message names the failed boundary; no retry promise unless the cause is known | Malformed JSON and non-boolean `enabled` retain the route family's existing -plain `400` responses (`native-integration-routes.ts:206-215,381-391`); these are +plain `400` responses +(`src/server/management/native-integration-routes.ts:206-215,381-391`); these are request errors, not native refusals. An external provider is not a refusal: the desired flag changes and the response is `200`, reason `external_provider_preserved`, while the config/catalog/history stay untouched. -Refusal/failure response (the existing envelope, unchanged): +Refusal/failure response (WP3's shared envelope, not a WP5 redefinition): ```ts { @@ -221,13 +420,21 @@ Refusal/failure response (the existing envelope, unchanged): reason: "config_busy" | "home_mismatch" | "history_busy" | "history_permission" | "write_failed"; message: string; + desiredEnabled: boolean; + observedState?: "absent" | "current" | "unsafe"; } ``` Invalid bodies remain `{ error: "invalid JSON body" }` or `{ error: "enabled must be a boolean" }` with HTTP 400. -MODIFY `src/server/management/native-integration-routes.ts`: +MODIFY `src/server/management/native-integration-routes.ts`. The contract types, +union, refusal helper, and flight primitive in this hunk are imports/owners landed +by WP3; WP5 adds only Codex detail and behavior. The real current GET anchor is +`clients: [claudeStatus(config, getConfigPath()), grokStatus()]` at +`src/server/management/native-integration-routes.ts:374-378`; WP3 first changes +Grok to its desired-aware helper, so the WP5 hunk below deliberately applies to +that post-WP3 line rather than pretending the phases are parallel: ```diff +import { @@ -238,40 +445,20 @@ MODIFY `src/server/management/native-integration-routes.ts`: + type CodexNativeRestoreResult, +} from "../../codex/inject"; +import { syncModelsToCodex } from "../../codex/sync"; --import { readRuntimePort, saveConfigPreservingClaudeCode } from "../../config"; +import { -+ clientIntegrationEnabled, readRuntimePort, saveConfigPreservingClaudeCode, -+ setClientIntegrationEnabled, ++ loadConfig, mutateClientIntegrationEnabled, readRuntimePort, +} from "../../config"; -@@ --export type NativeIntegrationClientId = "claude" | "grok"; -+export type NativeIntegrationClientId = "codex" | "claude" | "grok"; -@@ - | "config_busy" -+ | "history_busy" -+ | "history_permission" - | "write_failed"; -@@ - export interface NativeStatus { -@@ - disableBlocked: { reason: NativeRefusalReason; message: string } | null; -+ reason?: "external_provider_preserved"; -+ externalProvider?: string; - } -@@ -- reason?: string; -+ reason?: "non_loopback_removed" | "non_loopback_superseded" -+ | "external_provider_preserved" | "catalog_warning"; -+ externalProvider?: string; -+ artifacts?: CodexNativeRestoreResult["artifacts"]; -@@ ++import { ++ requirePersistedClientIntent, ++ runClientIntegrationFlight, ++} from "../../integrations/desired-state"; +function codexStatus(ctx: ManagementContext): NativeStatus { + const { deps } = ctx; + const externalProvider = (deps.currentExternalCodexModelProvider + ?? currentExternalCodexModelProvider)(); + const routing = (deps.getCodexRoutingKind ?? getCodexRoutingKind)(); + const owned = routing === "opencodex-local" ? assertNativeTeardownOwned() : null; -+ return { ++ return withDesiredState(loadConfig(), { + clientId: "codex", + state: externalProvider ? "absent" + : routing === "opencodex-local" ? "current" @@ -283,83 +470,99 @@ MODIFY `src/server/management/native-integration-routes.ts`: + ...(externalProvider ? { + reason: "external_provider_preserved" as const, externalProvider, + } : {}), -+ }; ++ }); +} @@ -- clients: [claudeStatus(config, getConfigPath()), grokStatus()], -+ clients: [codexStatus(ctx), claudeStatus(config, getConfigPath()), grokStatus()], +- clients: [claudeStatus(config, getConfigPath()), grokStatus(config)], ++ clients: [codexStatus(ctx), claudeStatus(config, getConfigPath()), grokStatus(config)], @@ + if (url.pathname === "/api/native-integrations/codex" && req.method === "PUT") { -+ let body: { enabled?: unknown }; -+ try { -+ body = await readManagementJsonBody(req); -+ } catch (error) { -+ rethrowManagementBodyTooLarge(error); -+ return jsonResponse({ error: "invalid JSON body" }, 400); -+ } -+ if (typeof body.enabled !== "boolean") { -+ return jsonResponse({ error: "enabled must be a boolean" }, 400); -+ } -+ const enabled = body.enabled; -+ -+ const desiredChanged = clientIntegrationEnabled(config, "codex") !== enabled; -+ if (desiredChanged) { -+ const next = { ...config, clientIntegrations: { ...config.clientIntegrations } }; -+ setClientIntegrationEnabled(next, "codex", enabled); -+ const persist = deps.saveConfigPreservingClaudeCode ?? saveConfigPreservingClaudeCode; ++ let body: { enabled?: unknown }; + try { -+ persist(next); ++ body = await readManagementJsonBody(req); + } catch (error) { -+ if (!isConfigLockError(error)) throw error; -+ return isLockContention(error) -+ ? refusal(409, "codex", "config_busy", -+ "Another process is saving the configuration right now. Try again in a moment.") -+ : refusal(500, "codex", "write_failed", -+ `The configuration lock could not be acquired: ${error instanceof Error ? error.message : String(error)}`); ++ rethrowManagementBodyTooLarge(error); ++ return jsonResponse({ error: "invalid JSON body" }, 400); + } -+ // Change the request-scoped object only after durable persistence succeeds. -+ setClientIntegrationEnabled(config, "codex", enabled); -+ } ++ if (typeof body.enabled !== "boolean") { ++ return jsonResponse({ error: "enabled must be a boolean" }, 400); ++ } ++ const enabled = body.enabled; ++ const desired = mutateClientIntegrationEnabled("codex", enabled); ++ if (desired.status === "unavailable") { ++ return desiredStatePersistenceFailure("codex", desired.reason, ctx); ++ } ++ const operationConfig = desired.value.config; + -+ if (!enabled) { -+ const owned = assertNativeTeardownOwned(); -+ if (!owned.ok) return refusal(409, "codex", "home_mismatch", owned.message); -+ const restore = (deps.restoreNativeCodex ?? restoreNativeCodex)(); ++ if (!enabled) { ++ const owned = assertNativeTeardownOwned(); ++ if (!owned.ok) return refusal({ ++ status: 409, clientId: "codex", reason: "home_mismatch", ++ desiredEnabled: false, observedState: codexStatus(ctx).state, ++ message: owned.message, ++ }); ++ const restore = await runClientIntegrationFlight( ++ "codex", ++ "disable", ++ () => Promise.resolve((deps.restoreNativeCodex ?? restoreNativeCodex)({ ++ beforeWrite: () => requirePersistedClientIntent("codex", false), ++ })), ++ ); + if (!restore.success) { + const history = restore.artifacts.history; + const otherArtifactsOk = restore.artifacts.config.state !== "failed" + && restore.artifacts.catalog.state !== "failed"; + if (otherArtifactsOk && history.state === "failed" && history.reason === "busy") { -+ return refusal(409, "codex", "history_busy", history.message); ++ return refusal({ status: 409, clientId: "codex", reason: "history_busy", ++ desiredEnabled: false, observedState: "absent", message: history.message }); + } + if (otherArtifactsOk && history.state === "failed" && history.reason === "permission") { -+ return refusal(500, "codex", "history_permission", history.message); ++ return refusal({ status: 500, clientId: "codex", reason: "history_permission", ++ desiredEnabled: false, observedState: "absent", message: history.message }); + } -+ return refusal(500, "codex", "write_failed", restore.message); ++ return refusal({ status: 500, clientId: "codex", reason: "write_failed", ++ desiredEnabled: false, observedState: codexStatus(ctx).state, ++ message: restore.message }); + } + return jsonResponse({ + ok: true, clientId: "codex", -+ changed: desiredChanged || Object.values(restore.artifacts).some(a => a.changed), ++ changed: desired.status === "committed" ++ || Object.values(restore.artifacts).some(a => a.changed), + state: "absent", ++ desiredEnabled: false, + message: restore.message, artifacts: restore.artifacts, + ...(restore.externalProvider ? { + reason: "external_provider_preserved" as const, + externalProvider: restore.externalProvider, + } : {}), + } satisfies NativeToggleEnvelope); -+ } ++ } + + const externalProvider = (deps.currentExternalCodexModelProvider + ?? currentExternalCodexModelProvider)(); + const runtime = (deps.readRuntimePort ?? readRuntimePort)(process.pid); -+ const port = runtime?.port ?? (Number(url.port) || config.port); -+ const synced = await (deps.syncModelsToCodex ?? syncModelsToCodex)(port, config, null); -+ if (!synced.ok) return refusal(500, "codex", "write_failed", synced.message); ++ const port = runtime?.port ?? (Number(url.port) || operationConfig.port); ++ // syncModelsToCodex is the lowest shared ON owner and enters the Codex ++ // flight itself; acquiring an outer route flight would self-deadlock. ++ const synced = await (deps.syncModelsToCodex ?? syncModelsToCodex)( ++ port, operationConfig, null, ++ ); ++ if (synced.status === "skipped") return refusal({ ++ status: 409, clientId: "codex", reason: "config_busy", ++ desiredEnabled: false, observedState: codexStatus(ctx).state, ++ message: "Desired OFF superseded this enable before its write; retry the explicit enable.", ++ }); ++ if (!synced.ok) return refusal({ status: 500, clientId: "codex", ++ reason: "write_failed", desiredEnabled: true, ++ observedState: codexStatus(ctx).state, message: synced.message }); + return jsonResponse({ -+ ok: true, clientId: "codex", changed: desiredChanged || !externalProvider, ++ ok: true, clientId: "codex", ++ changed: desired.status === "committed" || !externalProvider, + state: externalProvider ? "absent" : "current", message: synced.message, ++ desiredEnabled: true, + ...(externalProvider ? { -+ reason: "external_provider_preserved" as const, externalProvider, ++ reason: "external_provider_preserved" as const, ++ externalProvider, + } : synced.warning ? { reason: "catalog_warning" as const } : {}), + } satisfies NativeToggleEnvelope); + } @@ -370,14 +573,94 @@ MODIFY `src/server/management/context.ts` with typed optional seams for `currentExternalCodexModelProvider`. The production defaults are the real functions; `tests/native-codex-toggle.test.ts` supplies deterministic results. This follows the existing reason for `saveConfigPreservingClaudeCode` and Grok's -writer/catalog seams (`context.ts:12-37`). +writer/catalog seams (`src/server/management/context.ts:12-37`). + +## Coordination consumed from WP3 + +An entry guard is insufficient. Today catalog discovery awaits inside +`syncModelsToCodex()` before the irreversible injection at +`src/codex/sync.ts:83-110`; OFF can be persisted during that await. WP5 therefore +uses WP3's one Codex flight for **every** producer, not a route-local promise: + +- GUI `PUT /api/native-integrations/codex`; +- CLI `restore`, `restore back`, `sync`, startup, and both ensure paths; +- stop/shutdown/uninstall restore under the intent-neutral `teardown` operation; +- `POST /api/sync` and provider/model/combo refreshes; +- custom-model/provider CLI sync; and +- startup/background catalog refresh and injection. + +The flight serializes irreversible client operations. Desired intent may still +change while catalog work awaits — that is why the last-moment read is separate +and mandatory. It is the same client key across GUI, CLI, startup, and background +entry points; adding another `let codexToggleFlight` in the route would leave the +accepted race intact. A second write operation joins an identical reconciliation +or receives the shared `config_busy`/retry outcome for an opposing transition; it +never overlaps another Codex write. + +Serialization does not replace fresh policy reads. Desired state is loaded from +persisted config — never only from the request-scoped `config` object — +immediately before each irreversible boundary: + +| Direction | Irreversible boundary | Required fresh state | +|---|---|---| +| ON | catalog/cache write inside `refreshCodexModelCatalog()` | ON | +| ON | config/profile/journal write inside `injectCodexConfig()` | ON | +| ON | history provider retag inside injection | ON | +| OFF | journal restore or owned-field strip in `restoreNativeCodex()` | OFF | +| OFF | catalog restore in `restoreCodexCatalog()` | OFF | +| OFF | history retag/manifest consumption in `syncCodexHistoryProvider("openai")` | OFF | + +The lower write owners accept the WP3 flight's `beforeWrite()` assertion, because +a check immediately before calling an async catalog helper is still too early. +If the assertion observes the opposite desired state, that boundary does not +write and returns the discriminated superseded/config-busy outcome. Already +completed earlier artifacts are reported as changed; the response never rolls +intent back and never claims the opposite observed state. + +Lifecycle teardown is the named exception to the *gate*, not to the fresh read or +flight. It re-reads desired intent at each boundary for diagnostics but restores +native artifacts regardless, because a stopped proxy must not leave a dead +pointer. It never writes the desired map. A later start sees the preserved ON and +may re-enable; that is teardown recovery, not an explicit `ocx restore` opt-out. + +The external `model_provider` courtesy remains ahead of Codex-owned writes: +desired state changes, the stale journal is removed where current behavior does +so (`src/codex/inject.ts:765-768`), and config/catalog/history are structured +skips. WP5 does not seize an externally owned provider in either direction. + +## Desired OFF converges after interruption + +Desired OFF is not merely permission to skip future ON writes. Startup enters the +same Codex flight and re-runs the idempotent remover whenever persisted intent is +OFF. That reconciliation runs before startup would otherwise sync at +`src/cli/index.ts:319`; it does not stop the proxy or disable `/v1/responses`. +WP5 registers Codex's observed-state probe (`getCodexRoutingKind`) and remover +(`restoreNativeCodex`) in WP3's exhaustive reconciliation registry; it does not +add another startup hook. WP3 already invokes the registry for startup, ensure, +and status (`020_desired_state.md:629-685`). + +The retry walk is explicit: + +| Persisted/observed state after a crash | Startup re-run | +|---|---| +| OFF saved; config, catalog, and history still routed | restore/strip config, restore catalog, retag history | +| config native; catalog still has routed rows; history still tagged opencodex | config reports `skipped`/unchanged, catalog removes routed rows, history retags | +| config and catalog native; history still tagged opencodex or backup manifest remains | first two artifacts report unchanged, history alone retries and consumes the manifest on success | +| all three native; stale journal remains | journal cleanup completes; all other artifacts are unchanged | +| external `model_provider` owns routing | remove only the stale opencodex journal and return three courtesy skips | + +If history is busy, startup reports the classified partial outcome and leaves OFF +durable; the next startup or explicit OFF retries. Permission failure is also +re-run but remains a 500/non-retry-advice outcome until permissions change. No +state is reconstructed by parsing `restoreNativeCodex().message`. ## History-lock classification The current low-level code cannot distinguish the two outcomes after retry. It recognizes `SQLITE_BUSY`, `SQLITE_LOCKED`, `EBUSY`, `EPERM`, and `EACCES` in one predicate (`src/codex/history-provider.ts:511-523`), then `withHistoryRetry()` -discards the final error and returns `null` (`:536-548`). The caller therefore has +discards the final error and returns `null` +(`src/codex/history-provider.ts:536-548`). The caller therefore has no code left to inspect at line 577. Saying the GUI can classify this today would be false. @@ -416,8 +699,10 @@ GUI parses error prose. `codexRow` currently hard-codes `toggle: null` and ignores the native family (`gui/src/pages/integrations/overview-clients.ts:118-150`). Make its status merge match Claude/Grok: find `nativeCodex`, wait for `nativeSettled`, set -`toggle: "codex"`, `toggleBlocked`, and `togglePath` from that row, and keep the -badge based on observed `native.state`. When `native.reason` is +`toggle: "codex"`, `toggleBlocked`, `togglePath`, and `toggleOn` from that row, +and keep the badge/applied count based on observed `native.state`. The switch is +driven by required `native.desiredEnabled`; this is how a partial disable shows +OFF while the observed badge remains current/amber. When `native.reason` is `external_provider_preserved`, use the localized detail key with the structured provider name so the card says another provider owns routing instead of saying opencodex is applied. @@ -425,6 +710,14 @@ opencodex is applied. MODIFY `gui/src/pages/integrations/overview-clients.ts`: ```diff + export interface OverviewRow { +@@ + applied: boolean; ++ /** Desired switch position; absent means use observed `applied`. */ ++ toggleOn?: boolean; +@@ + } + -function codexRow(payload: CodexRoutingPayload | null): OverviewRow { +function codexRow( + payload: CodexRoutingPayload | null, @@ -439,6 +732,7 @@ MODIFY `gui/src/pages/integrations/overview-clients.ts`: + toggle: "codex" as const, + toggleBlocked: native?.disableBlocked ?? null, + togglePath: native?.configPath ?? null, ++ toggleOn: native?.desiredEnabled, @@ + if (!nativeSettled) return { ...base, state: "unknown", installed: false, applied: false, detailKey: null }; + if (!native) return { ...base, toggle: null, state: "unknown", installed: false, applied: false, detailKey: null }; @@ -455,6 +749,26 @@ MODIFY `gui/src/pages/integrations/overview-clients.ts`: + codexRow(sources.codex, nativeCodex, sources.nativeSettled), ``` +MODIFY the current switch/request sites at +`gui/src/pages/integrations/IntegrationsOverview.tsx:113-128,499-507`: + +```diff +- on={row.applied} ++ on={row.toggleOn ?? row.applied} +@@ +- label={row.applied ++ label={(row.toggleOn ?? row.applied) +@@ +- onToggle={row.toggle ? () => requestToggle(row, !row.applied) : null} ++ onToggle={row.toggle ++ ? () => requestToggle(row, !(row.toggleOn ?? row.applied)) ++ : null} +``` + +WP6 consumes this `toggleOn` owner for Desktop. When rebased after WP5, WP6 drops +its duplicate interface-field hunk at `050_desktop_toggle.md:727-731` and keeps +only the Desktop-row use at `050_desktop_toggle.md:733-745`. + `IntegrationsOverview.tsx` adds `CODEX_DISABLE_COPY`, admits `codex` anywhere the native toggle union is narrowed, refreshes `codexResource` after the mutation, and chooses copy by `pendingToggle.id` instead of always rendering Grok's copy. @@ -505,38 +819,23 @@ refreshes both observed sources: claudeResource.refresh(); ``` -MODIFY `gui/src/pages/integrations/native-api.ts`; both runtime allowlists must -widen with the TypeScript unions, or a valid server refusal will be downgraded to -an opaque `NativeApiError` (`native-api.ts:51-75`): +`gui/src/pages/integrations/native-api.ts` consumes WP3's complete mirror. WP3 +replaces the current two-client union and runtime allowlist at lines `3` and +`51-62` with all four native clients, requires `desiredEnabled` on status and +success, and owns every shared refusal reason. WP5 does **not** repeat those +diffs. It only adds Codex-specific optional detail to the already-WP3-owned +status/success interfaces: ```diff --export type NativeIntegrationClientId = "claude" | "grok"; -+export type NativeIntegrationClientId = "codex" | "claude" | "grok"; -@@ - | "config_busy" -+ | "history_busy" -+ | "history_permission" - | "write_failed"; -@@ export interface NativeStatus { @@ - disableBlocked: { reason: NativeRefusalReason; message: string } | null; + reason?: "external_provider_preserved"; + externalProvider?: string; @@ export interface NativeToggleEnvelope { @@ - reason?: string; + externalProvider?: string; + artifacts?: CodexNativeRestoreArtifacts; -@@ --const NATIVE_CLIENTS = new Set(["claude", "grok"]); -+const NATIVE_CLIENTS = new Set(["codex", "claude", "grok"]); -@@ - "config_busy", -+ "history_busy", -+ "history_permission", - "write_failed", ``` Define the GUI's structural `CodexNativeRestoreArtifacts` beside the envelope; @@ -574,7 +873,7 @@ export interface CodexNativeRestoreArtifacts { ``` MODIFY `gui/src/pages/integrations/refusal-copy.ts` at the existing native reason -switch (`:56-71`): +switch (`gui/src/pages/integrations/refusal-copy.ts:56-71`): ```diff if (refusal.reason === "not_installed") return t("integrations.native.error.notInstalled"); @@ -659,6 +958,20 @@ the dialog or refusal text in JSX (`gui/AGENTS.md:13-30`). `{path}` appears in catalog nor history mutation, and returns three structured skips. 4. A drifted post-injection root `model = "provider/slug"` is removed; reinjection does not recreate it. This pins the dialog's destructive sentence. +5. Desired-OFF reconciliation resumes from each partial boundary: config native + with routed catalog/history; config+catalog native with routed history; and all + native with only stale journal cleanup. Each re-run changes only remaining + artifacts and ends converged. +6. Before-write desired assertions flip ON between each pair of artifact writes; + the next boundary does not write and the structured result reports the exact + completed/aborted artifacts. + +`tests/codex-sync-api.test.ts` (the current structured-result owner at +`tests/codex-sync-api.test.ts:47-84`): desired OFF returns +`status:"skipped"/skippedReason:"desired_disabled"`; catalog fetch, catalog/cache +write, injection, history, and project-warning probes remain uncalled. A desired +flip during paused catalog discovery is observed at the lower before-write guard, +so no late catalog or config write lands after OFF. `tests/native-codex-toggle.test.ts`: @@ -685,13 +998,41 @@ the dialog or refusal text in JSX (`gui/AGENTS.md:13-30`). `{path}` appears in to `/v1/responses` and assert its expected response. Also assert `/healthz` identifies the same PID before and after. This proves the shared endpoint and process stayed alive; checking only the PUT response would not prove C4. +11. GET and every 200 response require `desiredEnabled`; every post-persist + refusal includes desired plus observed state. A schema/parser test fails if + Codex, Claude, Desktop, or Grok disappears from WP3's four-client union. +12. Pause GUI enable in catalog fetch, issue CLI restore/OFF, then release the + fetch. The shared Codex flight and lower before-write assertion permit no late + catalog/config/history write. Repeat with startup and background refresh as + the paused producer. Route-local-only exclusion fails this test. + +`tests/cli-restore-back.test.ts` changes from source-string assertions +(`tests/cli-restore-back.test.ts:11-35`) to isolated process-level cases. Every child receives temporary +`OPENCODEX_HOME` and `CODEX_HOME`, `CI=1`, and a reserved non-10100 port; teardown +terminates only the recorded child PID and removes only those temporary roots. + +1. Start desired OFF with native artifacts, run `ocx restore back`, and assert + exit 0, desired ON on a fresh config load, routed config plus at least one + routed catalog row, and no disabled-skip text. Restart that isolated proxy and + assert desired remains ON and observed routing remains current. +2. Start desired ON/current, run `ocx restore`, and assert exit 0, desired OFF on + a fresh config load, native config/catalog/history, and the explicit OFF + sentence. Restart and assert startup reconciliation leaves desired OFF and + observed native. +3. Force the distinct disabled result after `restore back`'s initial liveness + check through the coordinator seam. Assert it never prints “now routes”, exits + 2, and reports the competing OFF. An attempted artifact failure exits 1. +4. Run `ocx sync` while OFF: exit 0 with explicit skipped copy and no Codex file + timestamp change. Exercise startup, ensure, custom-model, provider `--sync`, + and `ocx system sync` to pin every caller outcome in the table above. GUI tests: 1. `gui/tests/integrations-overview-rows.test.ts` — settled native Codex gains a - toggle/path/blocker; missing or unsettled native evidence remains unknown with - no active switch; external provider renders the courtesy detail and no applied - claim. + toggle/path/blocker; required desired state drives `toggleOn` while observed + state drives badge/count; missing or unsettled native evidence remains unknown + with no active switch; external provider renders the courtesy detail and no + applied claim. 2. `gui/tests/overview-state-merge.test.ts` — widen client/reason validators and prove localized `history_busy` and `history_permission`; keep raw `write_failed` detail. @@ -706,62 +1047,40 @@ Static and automated gates: ```bash bun run typecheck -bun test tests/codex-history-provider.test.ts tests/codex-journal.test.ts tests/native-codex-toggle.test.ts +bun test tests/codex-history-provider.test.ts tests/codex-journal.test.ts tests/codex-sync-api.test.ts tests/native-codex-toggle.test.ts tests/cli-restore-back.test.ts bun run test cd gui && bun test tests && bun run lint && bun run lint:i18n && bun run build cd .. && bun run privacy:scan ``` -Live HTTP proof uses the already-running proxy at `localhost:10100`; do not call -`ocx stop` or `ocx restore`. Run only in the implementation C phase, after taking -a copy of the user's current Codex config for inspection and with an admin token -supplied by the maintainer: - -```bash -export OCX_LIVE_BASE=http://localhost:10100 -export OCX_ADMIN_TOKEN="${OPENCODEX_ADMIN_AUTH_TOKEN:?set the live management token without printing it}" - -curl -fsS "$OCX_LIVE_BASE/healthz" > .tmp/wp5-health-before.json -curl -fsS -H "x-opencodex-api-key: $OCX_ADMIN_TOKEN" \ - "$OCX_LIVE_BASE/api/native-integrations" > .tmp/wp5-native-before.json - -curl -fsS -X PUT -H "x-opencodex-api-key: $OCX_ADMIN_TOKEN" \ - -H 'content-type: application/json' -d '{"enabled":false}' \ - "$OCX_LIVE_BASE/api/native-integrations/codex" > .tmp/wp5-disable.json -curl -fsS "$OCX_LIVE_BASE/healthz" > .tmp/wp5-health-disabled.json -curl -sS -o .tmp/wp5-other-client.json -w '%{http_code}\n' \ - -H "x-opencodex-api-key: $OCX_ADMIN_TOKEN" -H 'content-type: application/json' \ - -d '{}' "$OCX_LIVE_BASE/v1/responses" - -curl -fsS -X PUT -H "x-opencodex-api-key: $OCX_ADMIN_TOKEN" \ - -H 'content-type: application/json' -d '{"enabled":true}' \ - "$OCX_LIVE_BASE/api/native-integrations/codex" > .tmp/wp5-enable.json -curl -fsS -H "x-opencodex-api-key: $OCX_ADMIN_TOKEN" \ - "$OCX_LIVE_BASE/api/native-integrations" > .tmp/wp5-native-after.json -curl -fsS "$OCX_LIVE_BASE/healthz" > .tmp/wp5-health-after.json -``` - -Read every JSON artifact. Disable must show Codex `absent` (or the classified -history refusal), enable must show `current` unless the explicit external-provider -courtesy applies, and all three health files must identify the same running proxy. -The `/v1/responses` invalid-body probe must reach the data-plane handler (an -expected validation/auth response, not connection refusal or 404); the automated -fixture test above is the stronger proof that another client completes a routed -request without spending live provider credits. Finally inspect the emitted Codex -catalog after enable and prove at least one current `provider/model` row exists; -an injected config beside a native-only catalog does not satisfy C5. - -For the held-history activation, open Codex so its writer lock is genuinely held, -disable once, and require HTTP 409 `history_busy` plus the localized card notice. -Close Codex/IDE, disable again, and require success with history `state: "ok"`. -Do not simulate this live proof by editing the response or matching its message. +No C-gate command targets the user's proxy on port 10100 or the user's real +Codex home. The process-level CLI/restart tests above are the activation proof: +they launch a recorded child on a reserved non-10100 port with temporary +`OPENCODEX_HOME`/`CODEX_HOME`, drive both restore directions, terminate that child, +restart it, and inspect the emitted config/catalog/history. The held-history case +uses a real `BEGIN IMMEDIATE` lock against the temporary `state_5.sqlite`, then +retries after releasing it. This proves 409 busy versus 500 permission without +opening the installed Codex app or mutating live state. + +Render grounding uses an isolated test server and temporary Codex home. Open the +Integrations overview, keyboard-activate Codex OFF, capture desktop and constrained +width screenshots, and read them back. The dialog must show root-model destruction, +non-byte-identical history, external-provider courtesy, and `/v1/responses` +survival before confirmation. After a forced history-busy response, the switch +must show desired OFF while the observed card remains partial; after retry it must +show observed native. No browser run confirms against port 10100. ## Accept criteria - **C5 — Codex toggles both directions from the overview with the proxy running.** Disable calls structured `restoreNativeCodex`; enable calls - `syncModelsToCodex(runtimePort)`. Live GET and on-disk catalog/config evidence + `syncModelsToCodex(runtimePort)`. Isolated-process GET and on-disk catalog/config evidence agree after both directions, and no stop/drain path runs. +- **CLI has the same durable meaning.** `ocx restore` persists OFF and + `ocx restore back` persists ON. The process-level test restarts after each and + proves the chosen state remains observed. A disabled sync is explicitly + `status:"skipped"`; no caller prints an applied claim, and exit 0/1/2 follows + the caller table rather than `ok` alone. - **C6 — a held history DB is explained, never false green.** A real held lock yields `409 native_integration_refused / history_busy`; config and catalog are reported separately, desired OFF persists, the card says why routed threads @@ -769,6 +1088,13 @@ Do not simulate this live proof by editing the response or matching its message. - **C4 — other clients keep serving.** The proxy PID/health identity is unchanged across disable and enable, `/v1/responses` remains registered, and another client's fixture request completes while Codex is OFF. +- **OFF converges and ON cannot write late.** Startup re-runs the remover from + every config/catalog/history partial state. The shared Codex flight covers GUI, + CLI, startup, and background producers, and persisted desired state is re-read + immediately before every irreversible write. +- **WP3's contract remains singular.** Server and GUI status/success shapes always + include required `desiredEnabled`; WP5 declares no competing native-client + union or refusal helper. WP5 lands before WP6 in every overlapping file. - External `model_provider` ownership remains untouched and visible on the card; resume history is described as semantically reversible, not byte-identical; and the dialog states that a post-injection routed root model selection is diff --git a/devlog/_plan/260803_codex_desktop_toggle/050_desktop_toggle.md b/devlog/_plan/260803_codex_desktop_toggle/050_desktop_toggle.md index 1ba67ae1f..7c593b0b1 100644 --- a/devlog/_plan/260803_codex_desktop_toggle/050_desktop_toggle.md +++ b/devlog/_plan/260803_codex_desktop_toggle/050_desktop_toggle.md @@ -2,10 +2,11 @@ Research: `002_desktop_standard_mode.md`. Read it first; this doc is the diff. The official contract is Anthropic's -[Claude Desktop configuration reference](https://claude.com/docs/third-party/claude-desktop/configuration) -(last modified 2026-07-24), as captured with the other primary citations in `002`. +[Claude Desktop configuration reference](https://claude.com/docs/third-party/claude-desktop/configuration). +The page itself supports the standard-mode and read-once-at-launch semantics used +below; this document no longer asserts a `lastmod` date the page does not show. -The concrete failure mode is a disable that deletes `.json` while +The first concrete failure mode is a disable that deletes `.json` while `_meta.json.appliedId` still names that id: Desktop opens the selected file by id, so the next launch points at something missing. The superseded `030` correctly noticed that pointer hazard, but concluded that exact restoration of the previous @@ -17,22 +18,45 @@ documented and achievable. This phase writes and selects a present, readable, credential-free `{}` profile first, then removes the old opencodex profile and its credential-bearing backup. It does not add an operation-state engine. +The audit exposed a second, more basic failure in the first revision of this +phase: **its status read and OFF path could install a filesystem footprint.** +`writeDesktop3pConfig()` calls `mkdirSync(libraryPath, { recursive: true })` +before reading metadata (`src/claude/desktop-3p.ts:343-345`), while the first +revision hard-coded `installed: true` and treated missing metadata as permission +to create a standard profile. On a machine that never created a Claude Desktop +config library, merely reading status or requesting OFF would manufacture one. +That position is reversed here plainly: reads never write; an absent library is +`not_installed`; OFF with no owned Desktop state is a successful idempotent no-op; +only an explicit setup/apply/enable action may create the library. + +The audit also rejected our bookkeeping as observed truth. A saved +`appliedFingerprint` says only what opencodex last intended to write. The user can +select another Desktop profile, edit/delete the selected profile, or return to +standard mode while that marker remains. Every status below therefore starts at +`_meta.json.appliedId`, proves the selected file exists and parses as an object, +classifies its `inferenceProvider` and credential-field shape, and only then uses +the fingerprint to distinguish current from drifted. Desired state remains a +separate required field. + ## IN / OUT IN: - `src/claude/desktop-3p.ts` — MODIFY: add the standard-mode remover and make - apply prefer the selected opencodex row when an interrupted cleanup left two. + apply prefer the selected opencodex row when an interrupted cleanup left two; + add the read-only installation/library and selected-profile inspector. - `src/cli/claude-desktop.ts` — MODIFY: explicit CLI apply is the enable direction and persists WP3 desired ON plus `desktopAutoApply: true` first. - `src/server/management/agent-settings-routes.ts` — MODIFY: gate auto-apply on - WP3 desired state, re-check after its await, expose desired state in `/status`, - and make explicit `/apply` an enable action. + WP3 desired state, join WP3's per-client flight, re-read persisted intent after + its await, expose desired and observed state in `/status`, and make explicit + `/apply` an enable action. - `src/server/management/native-integration-routes.ts` — MODIFY: add `claude-desktop` status and `PUT` toggle using the existing typed success/refusal/single-flight pattern (`:31-86`, `:164-174`, `:371-445`). -- `gui/src/pages/integrations/native-api.ts` — MODIFY: carry the new native id, - refusal reasons, and residual paths. +- `gui/src/pages/integrations/native-api.ts` — MODIFY: consume WP3's complete + four-client native contract and WP5's Codex additions, then extend only the + Desktop reason allowlist/residual detail. WP6 adds no competing union or envelope. - `gui/src/pages/integrations/integration-api.ts` — MODIFY: parse Desktop desired state from the existing rich status route. - `gui/src/pages/integrations/overview-clients.ts` — MODIFY: give @@ -57,8 +81,12 @@ OUT: - `src/types.ts` and `src/config.ts` — WP3 already owns `clientIntegrations["claude-desktop"]`, default-ON parsing, - `clientIntegrationEnabled`, and `setClientIntegrationEnabled`. WP6 consumes - those helpers and does not open-code the map (`020_desired_state.md:149-183`). + `clientIntegrationEnabled`, `mutateClientIntegrationEnabled`, the complete native + status/success/refusal contract, its status helpers, per-client single-flight, + startup reconciliation, and field-scoped persistence. WP6 consumes those owners + and does not open-code the map, redefine an envelope, or replace the whole + `claudeCode` subtree (`020_desired_state.md`; `src/config.ts:1846-1884`). + The existing `desktopAutoApply` field is at `src/types.ts:458-459`, not line 456. - `src/claude/desktop-3p-paths.ts` — path resolution is already one tested owner; the remover consumes `resolveDesktop3pConfigLibraryPath()` unchanged (`:67-78`). - `src/claude/desktop-profile.ts` — assignments/defaults are preserved as-is; no @@ -73,6 +101,25 @@ OUT: - The user's live Claude Desktop config library — no implementation or C-gate command mutates it without a separate, explicit approval. +## Composition order — this phase is after WP5 + +The first revision called WP5 and WP6 parallel siblings. That is wrong wherever +they touch the same route and GUI contract. WP5 adds `codex` to +`NativeIntegrationClientId`, the server GET list, `native-api.ts`'s runtime +allowlists, the overview merge, refusal reasons, and Codex restore detail +(`040_codex_toggle.md:230-366,416-545`). WP6 is **sequential after WP5** for those +files and its diffs apply to WP5's output, not today's three-client tree. + +WP3 remains the shared-contract owner. By the time WP6 starts, the shared client +union is already the complete `"codex" | "claude" | "claude-desktop" | "grok"` +contract; `NativeStatus` and every successful toggle response already require +`desiredEnabled`; the refusal envelope and status constructors are already +defined; and both GUI runtime allowlists already admit all four clients and shared +reasons. WP6 imports/uses those definitions. It does **not** repeat the old diff +from `"claude" | "grok"`, because that would delete WP5's `codex` entry. +Feature-specific Desktop reason literals extend WP3's existing reason union; the +envelope itself and its required fields are not re-declared. + ## What we depend on and what we refuse to depend on We depend on one official contract: third-party mode activates only when @@ -94,20 +141,43 @@ missing file, never chooses by `name === "Default"`, and never automates Desktop UI. The local evidence makes the third refusal load-bearing: this machine's real `_meta.json` has a `Default` row whose `.json` does not exist (`002:60-63`). -## Core diff — select a safe target before cleanup +## Core diff — probe first, then select a safe target before cleanup MODIFY `src/claude/desktop-3p.ts`. Add `unlinkSync` to the existing fs import, -export the result vocabulary beside `Desktop3pConfigLibraryOptions`, and place the -remover after `writeDesktop3pConfig` and before `atomicReplaceDesktopConfig`: +export the read-only observation and removal vocabulary beside +`Desktop3pConfigLibraryOptions`, and place the inspector/remover after +`writeDesktop3pConfig` and before `atomicReplaceDesktopConfig`: ```diff -import { copyFileSync, existsSync, mkdirSync, readFileSync } from "node:fs"; -+import { copyFileSync, existsSync, mkdirSync, readFileSync, unlinkSync } from "node:fs"; ++import { copyFileSync, existsSync, mkdirSync, readFileSync, statSync, unlinkSync } from "node:fs"; @@ ++export type Desktop3pObservedKind = ++ | "not_installed" ++ | "no_owned_state" ++ | "standard" ++ | "gateway" ++ | "foreign" ++ | "unsafe"; ++ ++export interface Desktop3pLibraryObservation { ++ kind: Desktop3pObservedKind; ++ libraryPath: string; ++ metadataPath: string; ++ selectedId: string | null; ++ selectedProfilePath: string | null; ++ selectedOwned: boolean; ++ fingerprint: string | null; ++ reason?: "metadata_unreadable" | "selected_missing" | "profile_unreadable" ++ | "provider_credentials_invalid" | "ambiguous_owned_rows"; ++} ++ +export type Desktop3pRemoveReason = + | "unsafe_metadata" + | "write_failed" -+ | "cleanup_incomplete"; ++ | "cleanup_incomplete" ++ | "desired_state_changed" ++ | "desired_state_unavailable"; + +export interface Desktop3pRemoveResult { + ok: boolean; @@ -119,13 +189,59 @@ remover after `writeDesktop3pConfig` and before `atomicReplaceDesktopConfig`: + residualPaths?: string[]; +} + ++export type Desktop3pIntentGuardResult = ++ | { ok: true } ++ | { ok: false; reason: "desired_state_changed" | "desired_state_unavailable" }; ++ +export interface Desktop3pRemoveDeps { + randomId?: typeof randomUUID; + writeFile?: typeof atomicWriteFile; + unlinkFile?: typeof unlinkSync; ++ beforeWrite?: () => Desktop3pIntentGuardResult; +} ``` +`inspectDesktop3pConfigLibrary(options)` is the one read-only probe used by both +status routes and the remover. It resolves the path but never calls `mkdirSync`, +`atomicWriteFile`, `copyFileSync`, or `unlinkSync`: + +```diff ++export function inspectDesktop3pConfigLibrary( ++ options: Desktop3pConfigLibraryOptions = {}, ++): Desktop3pLibraryObservation { ++ const libraryPath = resolveDesktop3pConfigLibraryPath(options); ++ const metadataPath = join(libraryPath, "_meta.json"); ++ // Operational installation means the config library already exists as a ++ // directory. This does not claim to detect every installed-but-never-launched ++ // Desktop bundle; it is the filesystem boundary opencodex can safely prove. ++ if (!existsSync(libraryPath) || !statSync(libraryPath).isDirectory()) { ++ return { kind: "not_installed", libraryPath, metadataPath, ++ selectedId: null, selectedProfilePath: null, selectedOwned: false, ++ fingerprint: null }; ++ } ++ if (!existsSync(metadataPath)) { ++ return { kind: "no_owned_state", libraryPath, metadataPath, ++ selectedId: null, selectedProfilePath: null, selectedOwned: false, ++ fingerprint: null }; ++ } ++ ++ // Parse metadata, resolve ONLY appliedId, require its entry and file to exist, ++ // parse that file as a non-array object, and inspect field NAMES/TYPES only. ++ // Never return profile JSON or any credential value. ++ // - a selected non-opencodex row is foreign; ++ // - selected opencodex + no inferenceProvider and no credential fields is standard; ++ // - selected opencodex + gateway provider + required gateway URL/key fields is gateway; ++ // - dangling/unreadable/invalid provider-credential combinations are unsafe. ++ // Compute the fingerprint only after the selected file passes those checks. ++} +``` + +`not_installed` is deliberately an operational/library result, not proof that no +Claude Desktop application bundle exists anywhere. An installed app that has never +created its 3P library is indistinguishable without platform-specific bundle +guessing; the safe answer for status and OFF is still “no manageable Desktop state” +and no write. Tests inject `options` and never inspect the user's real library. + The function signature is fixed here, and it lives in `src/claude/desktop-3p.ts` because `parseMetadata`, metadata ownership, atomic writes, and profile-path construction already live there: @@ -169,6 +285,12 @@ writer: + const randomId = deps.randomId ?? randomUUID; + const writeFile = deps.writeFile ?? atomicWriteFile; + const unlinkFile = deps.unlinkFile ?? unlinkSync; ++ const beforeWrite = deps.beforeWrite; ++ const observed = inspectDesktop3pConfigLibrary(options); ++ // 0. An absent library, missing metadata, or metadata with no owned row is an ++ // idempotent success/no-op. OFF must not call mkdirSync or create a standard ++ // profile when no opencodex Desktop state exists. ++ + // 1. Parse and validate before the first Desktop-library write. A non-string + // id, path separator, duplicate non-selected opencodex row, or malformed + // entries array is unsafe_metadata: desired OFF is already persisted by the @@ -182,6 +304,9 @@ writer: + + // 3. Atomically write metadata with BOTH rows still present and appliedId set + // to the new standard row. From here on Desktop's selected id always resolves. ++ // Call beforeWrite() immediately before the fresh profile write, metadata ++ // pivot, each unlink, and final metadata write. A changed/unavailable desired ++ // state returns a typed refusal and performs no later mutation. + + // 4. Remove the old .json.bak FIRST, then old .json through + // unlinkFile. Keep the old metadata row until both deletions succeed: it is @@ -198,13 +323,43 @@ writer: +} ``` +The existing writer also receives an optional WP3 intent guard; a route-level +check before model discovery is not close enough to either atomic commit: + +```diff + export function writeDesktop3pConfig( + /* existing parameters */, ++ deps: { beforeWrite?: () => Desktop3pIntentGuardResult } = {}, + ): { written: boolean; path: string; reason?: string; fingerprint?: string } { +@@ ++ const beforeProfile = deps.beforeWrite?.(); ++ if (beforeProfile && !beforeProfile.ok) { ++ return desiredStateWriteRefusal(configPath, beforeProfile.reason); ++ } + const { backupPath } = atomicReplaceDesktopConfig(configPath, configJson); + try { ++ const beforeMetadata = deps.beforeWrite?.(); ++ if (beforeMetadata && !beforeMetadata.ok) { ++ return desiredStateWriteRefusal(configPath, beforeMetadata.reason); ++ } + atomicWriteFile(metadataPath, /* ... */); +``` + +Explicit apply, native enable, and auto-apply pass +`() => requirePersistedClientIntent("claude-desktop", true)`. The remover passes +the same helper with `false` before every profile write, metadata write, and +unlink. `desired_state_changed` and `desired_state_unavailable` map through WP3's +shared refusal serializer; they never become a generic green no-op. + Path validation is deletion policy, not format cleanup. The old id must be one path component: reject `/`, `\\`, `..`, NUL, or a resolved path outside `libraryPath`. Multiple non-selected `name === "opencodex"` rows are ambiguous and REFUSE `unsafe_metadata`; the function does not guess which user-visible row to -delete. A missing `_meta.json` is not unsafe: create the standard file and a new -metadata document with one selected opencodex row. A dangling `Default` row is -preserved untouched. +delete. **Reversal from the first revision:** a missing library, missing +`_meta.json`, or metadata with no opencodex-owned row does not create a standard +file or metadata. Removal returns `{ ok: true, changed: false }`; a dangling +unrelated `Default` row is preserved untouched. Only `writeDesktop3pConfig` reached +from an explicit setup/apply/enable action retains permission to `mkdirSync`. The standard file is exactly `{}` plus a newline. Do not send `inferenceProvider: "anthropic"`; do not copy any old field into the replacement; @@ -213,103 +368,214 @@ create another `.bak` the disable then has to explain. ## Persist intent before touching Desktop -WP3 provides desired state. Both CLI apply and the existing POST apply become -explicit enable actions: +WP3 provides desired state, the per-client coordinator, startup reconciliation, +and the shared response constructors. Both CLI apply and the existing POST apply +become explicit enable actions. **The earlier `saveConfigPreservingClaudeCode` +examples are withdrawn.** They mutate a request snapshot and save a whole object, +which can overwrite a concurrent sibling-field update. Every persistence below +uses `mutatePersistedConfig()` (`src/config.ts:1846-1884`); its callback changes +only `clientIntegrations["claude-desktop"]`, +`claudeCode.desktopAutoApply`, and the named `desktopProfile` fields. ```diff - // src/cli/claude-desktop.ts:45-49 -+import { loadConfig, saveConfigPreservingClaudeCode, setClientIntegrationEnabled } from "../config"; + // src/cli/claude-desktop.ts:45-83 ++import { loadConfig, mutatePersistedConfig } from "../config"; ++import { requirePersistedClientIntent, runClientIntegrationFlight } from "../integrations/desired-state"; @@ const config = loadConfig(); -+ setClientIntegrationEnabled(config, "claude-desktop", true); const state = await buildClaudeDesktopState(config, profile); - config.claudeCode = { ...(config.claudeCode ?? {}), desktopProfile: state.profile }; -+ config.claudeCode = { -+ ...(config.claudeCode ?? {}), -+ desktopAutoApply: true, -+ desktopProfile: state.profile, -+ }; - saveConfigPreservingClaudeCode(config); +- saveConfigPreservingClaudeCode(config); ++ // A live daemon owns persistence + writing inside POST /apply's Desktop flight. ++ // Do not save locally first: that would be outside the daemon's OFF ordering. + const live = await (deps.findLiveProxyImpl ?? findLiveProxy)(); + if (live) { /* existing POST delegation, carrying state.profile */ } ++ return runClientIntegrationFlight("claude-desktop", "explicit-apply", async () => { ++ const persisted = mutatePersistedConfig(next => { ++ next.clientIntegrations = { ...next.clientIntegrations, "claude-desktop": true }; ++ next.claudeCode ??= {}; ++ next.claudeCode.desktopAutoApply = true; ++ next.claudeCode.desktopProfile = state.profile; ++ return { changed: true, value: undefined }; ++ }); ++ if (persisted.status === "unavailable") { ++ return { ok: false, path: "", reason: "config unavailable" }; ++ } ++ const permit = requirePersistedClientIntent("claude-desktop", true); ++ if (!permit.ok) return { ok: false, path: "", reason: permit.reason }; ++ return writeDesktop3pConfig(/* existing values from permit.config */, { ++ beforeWrite: () => requirePersistedClientIntent("claude-desktop", true), ++ }); ++ }); ``` ```diff // src/server/management/agent-settings-routes.ts:735-738 - const state = await buildClaudeDesktopState(config, profileOverride); +- const state = await buildClaudeDesktopState(config, profileOverride); - config.claudeCode = { ...(config.claudeCode ?? {}), desktopProfile: state.profile }; -+ setClientIntegrationEnabled(config, "claude-desktop", true); -+ config.claudeCode = { -+ ...(config.claudeCode ?? {}), -+ desktopAutoApply: true, -+ desktopProfile: state.profile, -+ }; - saveConfigPreservingClaudeCode(config); +- saveConfigPreservingClaudeCode(config); ++ return runClientIntegrationFlight("claude-desktop", "explicit-apply", async () => { ++ const persisted = mutatePersistedConfig(next => { ++ next.clientIntegrations = { ...next.clientIntegrations, "claude-desktop": true }; ++ next.claudeCode ??= {}; ++ next.claudeCode.desktopAutoApply = true; ++ if (profileOverride) next.claudeCode.desktopProfile = profileOverride; ++ return { changed: true, value: next }; ++ }); ++ if (persisted.status === "unavailable") { ++ const desiredEnabled = clientIntegrationEnabled(loadConfig(), "claude-desktop"); ++ return jsonResponse({ ++ error: `Desktop desired state was not saved: ${persisted.reason}`, ++ desiredEnabled, ++ }, persisted.reason === "conflict" ? 409 : 500); ++ } ++ const state = await buildClaudeDesktopState(persisted.value, profileOverride); ++ // Continue to the guarded writer below; close the flight after marker commit. +@@ +- if (!result.written) return jsonResponse({ error: result.reason, saved: true, path: result.path }, 500); ++ if (!result.written) return jsonResponse({ ++ error: result.reason ?? "Claude Desktop apply failed", ++ saved: true, path: result.path, desiredEnabled: true, ++ }, 500); +@@ +- return jsonResponse({ ok: true, saved: true, applied: true, path: result.path, fingerprint: result.fingerprint }); ++ return jsonResponse({ ok: true, saved: true, applied: true, ++ desiredEnabled: true, path: result.path, fingerprint: result.fingerprint }); ++ }); ``` The disable endpoint follows `native-integration-routes.ts` rather than inventing -a second response grammar. Extend `NativeIntegrationClientId` with -`"claude-desktop"`, extend refusal reasons with `unsafe_metadata` and -`cleanup_incomplete`, add a module-level Desktop single-flight, include its status -in GET, and add `PUT /api/native-integrations/claude-desktop`: +a second response grammar. There is intentionally **no client-union or envelope +diff here**. WP3's four-client contract already includes `claude-desktop`, the +`residualPaths` extension, and required `desiredEnabled` on status, success, and +refusal/failure responses. WP6 appends `unsafe_metadata` and +`cleanup_incomplete` only to `NativeRefusalReason`, calls WP3's +`withDesiredState` and widened `refusal({ ... })` helpers, and adds the Desktop PUT +after WP5's Codex route. ```diff --export type NativeIntegrationClientId = "claude" | "grok"; -+export type NativeIntegrationClientId = "claude" | "claude-desktop" | "grok"; -@@ export type NativeRefusalReason = - | "not_installed" - | "orphaned_marker" + /* WP3 shared reasons + WP5 Codex reasons */ + | "unsafe_metadata" -+ | "cleanup_incomplete" - | "home_mismatch" -@@ - export interface NativeRefusalEnvelope { ++ | "cleanup_incomplete"; @@ -+ residualPaths?: string[]; + export interface NativeStatus { + /* WP3 required fields + WP5 Codex detail */ +- reason?: "external_provider_preserved"; ++ reason?: "external_provider_preserved" | "not_installed"; } ``` -WP3 has already added `desiredEnabled` to `NativeStatus`, -`NativeToggleEnvelope`, and post-persist refusal envelopes -(`020_desired_state.md:423-453`). Desktop uses those fields; it does not add a -second desired-state property. +Add the GET row with desired and observed fields kept separate. It reports the +library path, not a guessed `Default` profile path: + +```diff ++function desktopStatus(): NativeStatus { ++ const persisted = loadConfig(); ++ const observed = inspectDesktop3pConfigLibrary(); // read-only; never mkdir/write ++ const saved = persisted.claudeCode?.desktopProfile?.appliedFingerprint ?? null; ++ const fingerprintMatches = observed.kind === "gateway" ++ && saved !== null && observed.fingerprint === saved; ++ return withDesiredState(persisted, { ++ clientId: "claude-desktop", ++ state: fingerprintMatches ? "current" ++ : observed.kind === "unsafe" || observed.kind === "gateway" ? "unsafe" ++ : "absent", ++ installed: observed.kind !== "not_installed", ++ configPath: observed.libraryPath, ++ disableBlocked: null, ++ ...(observed.kind === "not_installed" ? { reason: "not_installed" as const } : {}), ++ }); ++} +@@ +- clients: [codexStatus(ctx), claudeStatus(config, getConfigPath()), grokStatus(config)], ++ clients: [ ++ codexStatus(ctx), ++ claudeStatus(config, getConfigPath()), ++ desktopStatus(), ++ grokStatus(config), ++ ], +``` + +`disableBlocked` is null intentionally. The read-only inspector can classify +malformed state as `unsafe`, but the PUT remains available so desired OFF can be +recorded. An absent library returns `installed:false`, `state:"absent"`, reason +`not_installed`, and the actual required `desiredEnabled` without creating a +directory. A malformed file is returned by PUT as desired OFF + observed unsafe, +not hidden as an unavailable action. Disable body, in this exact order: ```ts // PUT { enabled: false } -// 1. Persist BOTH suppressors before the Desktop mutation. -setClientIntegrationEnabled(config, "claude-desktop", false); -config.claudeCode = { - ...(config.claudeCode ?? {}), - desktopAutoApply: false, -}; -persist(config); - -// 2-3. Fresh readable standard profile -> appliedId pivot -> old .bak/.json/row. -const removed = removeDesktop3pConfig(); -if (!removed.ok) { - // unsafe_metadata: 409 refused; no Desktop bytes changed. - // write_failed/cleanup_incomplete: 500 failed. For cleanup_incomplete include - // residualPaths; desired OFF and auto-apply suppression remain persisted. - return desktopRefusal(removed); -} - -// 4. Only complete cleanup clears observed apply markers. Preserve the profile's -// assignments/defaults (33 assignments on this machine) and every other field. -const profile = config.claudeCode?.desktopProfile; -if (profile) { - const { appliedFingerprint: _fingerprint, appliedAt: _appliedAt, ...preserved } = profile; - config.claudeCode = { ...config.claudeCode, desktopProfile: preserved }; - persist(config); -} - -return jsonResponse({ - ok: true, - clientId: "claude-desktop", - changed: removed.changed, - state: "absent", - reason: "desktop_standard_mode", - message: "Claude Desktop is configured for standard mode; restart required", +return runClientIntegrationFlight("claude-desktop", "disable", async () => { + // 1. Persist BOTH suppressors from the newest on-disk snapshot. The callback + // patches named fields only; it never assigns a replacement claudeCode subtree. + const persisted = mutatePersistedConfig(next => { + next.clientIntegrations = { ...next.clientIntegrations, "claude-desktop": false }; + next.claudeCode ??= {}; + next.claudeCode.desktopAutoApply = false; + return { changed: true, value: undefined }; + }); + if (persisted.status === "unavailable") { + return refusalFromDesiredFailure("claude-desktop", persisted); + } + + // 2. Probe only after intent commits. With no library/metadata/owned row this is + // a successful no-op and MUST NOT create a directory, profile, or _meta.json. + const before = inspectDesktop3pConfigLibrary(); + const permitRemoval = requirePersistedClientIntent("claude-desktop", false); + if (!permitRemoval.ok) { + const desiredEnabled = clientIntegrationEnabled(loadConfig(), "claude-desktop"); + return refusal({ + status: 409, clientId: "claude-desktop", reason: "config_busy", + desiredEnabled, observedState: desktopStatus().state, + message: `Desktop removal was skipped: ${permitRemoval.reason}.`, + }); + } + const removed = removeDesktop3pConfig({}, { + beforeWrite: () => requirePersistedClientIntent("claude-desktop", false), + }); + if (!removed.ok) { + // unsafe_metadata: 409 refused; no Desktop bytes changed. + // write_failed/cleanup_incomplete: 500 failed. Every post-commit envelope has + // desiredEnabled:false; cleanup_incomplete includes residualPaths only. + const desiredStateReason = removed.reason === "desired_state_changed" + || removed.reason === "desired_state_unavailable"; + return refusal({ + status: removed.reason === "unsafe_metadata" || desiredStateReason ? 409 : 500, + clientId: "claude-desktop", + reason: desiredStateReason ? "config_busy" : removed.reason ?? "write_failed", + desiredEnabled: false, + observedState: desktopStatus().state, + message: removed.message ?? "Claude Desktop cleanup did not complete.", + residualPaths: removed.residualPaths, + }); + } + + // 3. Only complete cleanup clears marker fields, again from a fresh snapshot. + // Assignments/defaults and every sibling Claude field remain untouched. + mutatePersistedConfig(next => { + const profile = next.claudeCode?.desktopProfile; + if (!profile) return { changed: false, value: undefined }; + delete profile.appliedFingerprint; + delete profile.appliedAt; + return { changed: true, value: undefined }; + }); + + const after = inspectDesktop3pConfigLibrary(); + return jsonResponse({ + ok: true, + clientId: "claude-desktop", + changed: removed.changed, + state: "absent", + desiredEnabled: false, + reason: before.kind === "not_installed" || before.kind === "no_owned_state" + ? "not_installed" : "desktop_standard_mode", + message: after.kind === "not_installed" || after.kind === "no_owned_state" + ? "No Claude Desktop-managed state exists; nothing was created or removed." + : "Claude Desktop is configured for standard mode; restart required", + } satisfies NativeToggleEnvelope); }); ``` @@ -320,21 +586,146 @@ If generation fails, desired ON remains visible while observed state remains off the response is a failure, not false green. No enable or disable branch touches `config.claudeCode.enabled`, so Claude Code's use of `/v1/messages` is unchanged. +All four writers enter the **same WP3 per-client flight**: automatic apply, +`POST /api/claude-desktop/apply`, native PUT enable, and native PUT disable. The +CLI delegates to POST when a daemon is live; its no-daemon fallback uses the same +coordinator. WP3 joins an identical operation and refuses a competing direction; +it does not coalesce ON and OFF. Whichever direction owns the flight reaches a +whole outcome while the competitor receives a typed contention/refusal and may +retry. Immediately before every irreversible +`writeDesktop3pConfig` or `removeDesktop3pConfig` call, the operation reloads +persisted desired state. Auto-apply skips unless it is still ON; explicit enable +is allowed only after its own ON commit; removal runs only while OFF. A stale +request-scoped `config` object is never the authority for that final check. + +WP3's in-process promise map is backed by its per-client SQLite coordinator, so a +cooperating CLI/startup process and the server cannot write Desktop concurrently +(`020_desired_state.md:486-515`). This still cannot exclude the user, Claude +Desktop, or a non-cooperating process editing the library; the post-write +inspector and drift response expose those changes rather than calling them current. + +The enable branch is concrete, not an internal HTTP call back into the same +server. Dynamic imports preserve `management-api.ts`'s current cycle boundary +(`native-integration-routes.ts:176-183`): + +```diff ++ return runClientIntegrationFlight("claude-desktop", "enable", async () => { ++ const persisted = mutatePersistedConfig(next => { ++ next.clientIntegrations = { ...next.clientIntegrations, "claude-desktop": true }; ++ next.claudeCode ??= {}; ++ next.claudeCode.desktopAutoApply = true; ++ return { changed: true, value: undefined }; ++ }); ++ if (persisted.status === "unavailable") { ++ return refusalFromDesiredFailure("claude-desktop", persisted); ++ } ++ ++ const { buildClaudeDesktopState } = await import("../management-api"); ++ const { desktopVisibleNativeSlugs } = await import("../../codex/catalog"); ++ const { writeDesktop3pConfig } = await import("../../claude/desktop-3p"); ++ const fresh = loadConfig(); ++ const state = await buildClaudeDesktopState(fresh); ++ const routed = state.models ++ .filter(model => model.available && !model.route.startsWith("native/")) ++ .map(model => { ++ const slash = model.route.indexOf("/"); ++ return { ++ provider: model.route.slice(0, slash), ++ id: model.route.slice(slash + 1), ++ contextWindow: model.contextWindow, ++ }; ++ }); ++ // Re-read immediately before the irreversible writer, inside the same flight. ++ // A queued OFF cannot cross this point unnoticed. ++ const permitWrite = requirePersistedClientIntent("claude-desktop", true); ++ if (!permitWrite.ok) { ++ const desiredEnabled = clientIntegrationEnabled(loadConfig(), "claude-desktop"); ++ return refusal({ status: 409, clientId: "claude-desktop", reason: "config_busy", ++ desiredEnabled, observedState: desktopStatus().state, ++ message: `Desktop enable was skipped: ${permitWrite.reason}.` }); ++ } ++ const beforeWrite = permitWrite.config; ++ const written = writeDesktop3pConfig( ++ Number(ctx.url.port) || beforeWrite.port, ++ [...desktopVisibleNativeSlugs(beforeWrite)], ++ routed, ++ beforeWrite.apiKeys?.[0]?.key, ++ "static", ++ state.profile, ++ { beforeWrite: () => requirePersistedClientIntent("claude-desktop", true) }, ++ ); ++ if (!written.written) return refusal({ ++ status: 500, clientId: "claude-desktop", reason: "write_failed", ++ desiredEnabled: true, observedState: desktopStatus().state, ++ message: written.reason ?? "Claude Desktop apply failed.", ++ }); ++ mutatePersistedConfig(next => { ++ next.claudeCode ??= {}; ++ next.claudeCode.desktopProfile = { ++ ...state.profile, ++ appliedFingerprint: written.fingerprint, ++ appliedAt: new Date().toISOString(), ++ }; ++ return { changed: true, value: undefined }; ++ }); ++ return jsonResponse({ ++ ok: true, clientId: "claude-desktop", changed: true, ++ state: "current", desiredEnabled: true, ++ message: "Claude Desktop gateway profile applied; restart required.", ++ } satisfies NativeToggleEnvelope); ++ }); +``` + +The implementation should extract the repeated state-to-routed-model mapping +from the existing POST apply into one local helper in +`agent-settings-routes.ts` only if that avoids byte-for-byte duplication without +creating a cross-module cycle. It must not call the management endpoint over +loopback or invent another transport. + +`POST /api/claude-desktop/apply` returns `desiredEnabled: true` on success and on +every failure after the ON commit. A failure before persistence reports the +current persisted desired state through WP3's refusal helper. The native enable +and disable successes, native GET row, rich status GET, startup reconciliation +diagnostic, and all post-commit refusals follow the same rule: no response shape +omits `desiredEnabled`, and no caller reconstructs it from `applied`. + ## Crash-safety: exact residual state at every boundary There is no transaction across opencodex `config.json`, Desktop `_meta.json`, and three profile paths. The ordering preserves a valid selected pointer; it does not -make the whole disable transactional. +make the whole disable transactional. WP3 now treats desired OFF as a **converge +instruction**, not merely a gate: startup reconciliation enters the same +`claude-desktop` flight and re-runs this remover while OFF. That reverses the first +revision's false claim that persisting a boolean alone survived a crash. -| Process dies after | State on disk | Classification / retry | +The remover must be idempotent from every state it can inherit, including states +created by older builds or a hand edit: + +| State at startup/retry | Read-only classification | What the idempotent re-run does | |---|---|---| -| desired OFF + `desktopAutoApply:false`, before standard file | Desktop still selects the old gateway profile; automatic re-apply is suppressed | Pointer-safe only; disable is not effective. Retry starts the mutation. | -| standard `{}` file, before first metadata write | Old profile still selected; the fresh credential-free file is orphaned and its generated id was not durably recorded | Pointer-safe only; retry creates another fresh target. The harmless orphan may remain because identifying it after process death would require the operation record this design deliberately does not add. No claim of transactional cleanup or semantic disable. | -| metadata points to standard, before `.bak` deletion | Next Desktop launch is standard mode; old profile and backup still contain credentials | Pointer-safe and semantically disabled on next launch, but security cleanup is incomplete. Old metadata row locates both files for retry. | -| `.bak` deleted, before old `.json` deletion | Selected standard file exists; one old credential-bearing profile remains | Pointer-safe, not security-complete. Retry deletes the old profile. | -| old `.json` deleted, before old row removal | Selected standard file exists; stale non-selected row may name a missing file | Applied-pointer-safe, not registry-clean. Retry removes that exact old row. | -| old row removed, before markers clear | Desktop library and credential cleanup are complete; `/status` still derives `applied:true` from the stale fingerprint (`agent-settings-routes.ts:797-804`) | Runtime files are safe; bookkeeping is false. Retry clears markers without changing assignments/defaults. | -| markers clear | Desired OFF, auto-apply OFF, selected standard profile readable, old profile and `.bak` absent | Disable is complete on disk. A Desktop process already running can still be using launch-time state until restart. | +| Library directory absent | `not_installed` | Returns unchanged success. It does not call `mkdirSync`, create `_meta.json`, or create a profile. Startup reconciliation is complete for this client. | +| Library exists, metadata absent, or metadata has no opencodex-owned row | `no_owned_state` or `foreign` | Returns unchanged success and preserves all files. OFF is already converged because there is no owned gateway state to remove. | +| Desired OFF persisted; old opencodex gateway is still selected | `gateway` (desired/observed drift) | Creates and verifies one fresh `{}` target, pivots `appliedId`, then performs cleanup. This closes “crash after persist, before mutate.” | +| Fresh `{}` file exists but metadata still selects the old gateway | `gateway`; orphan id is not discoverable | Creates another target and proceeds. The first credential-free orphan may remain; without a journal it cannot be identified safely. This leak is acknowledged, not called full cleanup. | +| Metadata selects standard; old row + `.bak` + `.json` remain | `standard`, with retry locator | Reuses the selected standard row, deletes `.bak`, then `.json`, then the old row. It allocates no second selected profile. | +| Metadata selects standard; `.bak` is already absent | `standard`, with retry locator | Missing backup is already-clean; deletes old `.json`, then the old row. | +| Metadata selects standard; both old files are absent but old row remains | `standard`, with retry locator | Removes exactly the stale old row; unrelated rows survive. | +| Old row is gone; selected standard is valid; apply markers remain | `standard`; markers are not observed truth | Performs no Desktop write, then field-scoped persistence deletes only `appliedFingerprint`/`appliedAt`. Status already reports standard/absent before marker cleanup. | +| Selected standard and markers are clean | `standard` | Returns unchanged success. A Desktop process already running may still use launch-time state until restart. | +| Selected owned id is missing/unreadable, provider/credentials conflict, metadata is malformed, or cleanup rows are ambiguous | `unsafe` | Refuses without a Desktop write on every retry, leaves desired OFF durable, and emits only typed reason/path evidence. Startup records the non-secret reconciliation failure for user action; it never guesses or deletes. | + +Startup reconciliation does not run apply while desired ON; existing startup/auto +paths remain separately gated. It invokes only the OFF converger, and a failure +does not flip desired state back to ON. Explicit OFF, startup OFF reconciliation, +explicit enable, POST apply, and auto-apply all share the same flight, so the +startup remover cannot overlap an explicit writer in the same process. + +Desktop is the exception to WP3's generic `trigger: "status"` convergence hook. +For `claude-desktop`, status-triggered reconciliation performs inspection only and +returns the unresolved/resolved diagnostic without invoking the remover. Startup +and ensure re-run the remover; an explicit OFF request runs it; GET never does. +Otherwise a status read with desired OFF could create the standard pivot or delete +credential files, violating the audit's non-negotiable “read never writes” rule. The first metadata write contains both rows and points at the new one. Cleanup deletes the `.bak` first because it is an otherwise unmanaged credential copy, @@ -365,39 +756,96 @@ started in-process race: ```diff async function autoApplyDesktopBestEffort(): Promise { try { -+ if (!clientIntegrationEnabled(config, "claude-desktop")) return; - if (config.claudeCode?.desktopAutoApply === false) return; - if (!config.claudeCode?.desktopProfile) return; ++ await runClientIntegrationFlight("claude-desktop", "auto-apply", async () => { ++ const initial = loadConfig(); ++ const library = inspectDesktop3pConfigLibrary(); ++ if (!clientIntegrationEnabled(initial, "claude-desktop")) return; ++ if (initial.claudeCode?.desktopAutoApply === false) return; ++ if (!initial.claudeCode?.desktopProfile) return; ++ // Auto-apply is reconciliation, not setup. Only explicit apply/enable may ++ // create a missing library or first owned profile. ++ if (library.kind === "not_installed" || library.kind === "no_owned_state" ++ || library.kind === "foreign") return; @@ - const allModels = await fetchAllModels(config); + const allModels = await fetchAllModels(initial); const routed = /* existing mapping */; -+ // The toggle can persist OFF while fetchAllModels was awaiting. Re-check -+ // immediately before the synchronous writer. -+ if (!clientIntegrationEnabled(config, "claude-desktop")) return; -+ if (config.claudeCode?.desktopAutoApply === false) return; - const result = writeDesktop3pConfig(/* existing args */); ++ // The toggle can persist OFF while fetchAllModels was awaiting. Re-read ++ // persisted config immediately before the irreversible writer. ++ const permitWrite = requirePersistedClientIntent("claude-desktop", true); ++ if (!permitWrite.ok) return; ++ const beforeWrite = permitWrite.config; ++ if (beforeWrite.claudeCode?.desktopAutoApply === false) return; ++ if (inspectDesktop3pConfigLibrary().kind === "not_installed") return; ++ const result = writeDesktop3pConfig(/* values from beforeWrite */, { ++ beforeWrite: () => requirePersistedClientIntent("claude-desktop", true), ++ }); ++ // Persist markers with mutatePersistedConfig, changing marker fields only. ++ }); ``` If auto-apply has already entered the synchronous writer, JavaScript completes -that write before the toggle handler runs; the later disable then pivots away and -cleans it. If it is awaiting model discovery, the second guard stops it. This is -in-process ordering, not a cross-process file lock; a second opencodex process is -INFERRED possible and is reported by post-write status, not claimed excluded. +that guarded write before the competing OFF flight can acquire the coordinator; +the later disable then pivots away and cleans it. If auto-apply is awaiting model +discovery and desired state changes through a non-overlapping committed path, the +last-moment guard stops it. The WP3 coordinator covers cooperating processes; +post-write status still detects Desktop/user/non-cooperating edits. -Extend `/api/claude-desktop/status` without changing the existing observed fields: +Replace `/api/claude-desktop/status`'s bookkeeping inference with the shared +read-only inspector. The old route makes `applied` equal +`savedFingerprint !== null` (`agent-settings-routes.ts:797-804`); that is the +stale-by-construction behavior being removed: ```diff - return jsonResponse({ -+ enabled: clientIntegrationEnabled(config, "claude-desktop"), - applied: savedFingerprint !== null, +- const libraryPath = resolveDesktop3pConfigLibraryPath(); +- /* first name === "opencodex" lookup + direct profile hash */ +- const stale = savedFingerprint !== null && onDiskFingerprint !== null +- && savedFingerprint !== onDiskFingerprint; ++ const persisted = loadConfig(); ++ const observed = inspectDesktop3pConfigLibrary(); ++ const desiredEnabled = clientIntegrationEnabled(persisted, "claude-desktop"); ++ const savedFingerprint = persisted.claudeCode?.desktopProfile?.appliedFingerprint ?? null; ++ const applied = observed.kind === "gateway"; ++ const fingerprintMatches = applied && savedFingerprint !== null ++ && observed.fingerprint === savedFingerprint; ++ const stale = applied && !fingerprintMatches; ++ const drift = desiredEnabled ++ ? !fingerprintMatches ++ : applied || observed.kind === "unsafe"; @@ + return jsonResponse({ ++ desiredEnabled, ++ installed: observed.kind !== "not_installed", ++ observedKind: observed.kind, ++ applied, + appliedAt, + savedFingerprint, ++ onDiskFingerprint: observed.fingerprint, ++ configPath: observed.selectedProfilePath, ++ activeProfile: observed.selectedOwned, + stale, ++ drift, ++ driftReason: !drift ? null ++ : observed.kind === "unsafe" ? observed.reason ++ : desiredEnabled ? "desired_on_not_current" : "desired_off_gateway_selected", + health, }); ``` -Desired state drives the switch. `applied`, `stale`, and `activeProfile` remain -observed evidence and drive badge/count detail. A failed disable may therefore -show switch OFF with an amber observed-state notice; that is the truthful -"desired OFF, observed conflict" state required by WP3. +There is no fallback to the first `name === "opencodex"` row. During interrupted +cleanup both rows have that name, and only `_meta.json.appliedId` identifies what +Desktop will read. If the selected row/file is missing, unreadable, non-object, or +has an invalid provider/credential combination, status is `unsafe`; it does not +hash an old non-selected profile and does not expose parsed contents. A selected +foreign row or valid standard profile is observed `applied:false` even if our +saved fingerprint remains. A selected valid gateway profile is `applied:true`; +only a matching saved fingerprint is current. That ordering makes fingerprint a +last corroborating check, never the source of observed state. + +`desiredEnabled` drives the switch. `observedKind`, `applied`, `stale`, +`activeProfile`, and `drift` drive badge/count/detail. A user-selected foreign +profile, hand-edited gateway, deleted selected file, standard-mode pivot, and +desired-OFF/gateway-selected conflict therefore remain distinguishable instead of +collapsing into “applied because our marker exists.” ## GUI — one switch, one consequence dialog @@ -425,22 +873,67 @@ state by accident: + toggle: "claude-desktop", + toggleBlocked: native?.disableBlocked ?? null, + togglePath: native?.configPath ?? null, -+ toggleOn: native?.desiredEnabled ?? (payload?.enabled !== false), ++ toggleOn: native?.desiredEnabled, ``` `OverviewCard` renders `on={row.toggleOn ?? row.applied}`. The Desktop row is unknown and non-actionable until both its rich status and native status settle. -Desired OFF + stale marker is amber, not green; desired ON + no applied marker is -absent with the switch ON and `integrations.detail.desktopDesiredOnNotApplied`. +Once settled, a missing/non-boolean `desiredEnabled` is a contract parse failure, +not permission to infer ON from `applied` or an old marker. The fallback in +`OverviewCard` remains for client rows that do not have a desired-state contract; +the Desktop row always supplies the required field. +Desired OFF + an actually selected gateway (or unsafe selected state) is amber, +not green; a stale marker beside a selected standard/foreign profile is not +treated as applied. Desired ON + no current selected gateway is absent/drifted +with the switch ON and `integrations.detail.desktopDesiredOnNotApplied`. + +MODIFY `gui/src/pages/integrations/integration-api.ts` to consume the required +rich-status shape, without recreating the native union from WP3: + +```diff + export interface ClaudeDesktopPayload { ++ desiredEnabled: boolean; ++ installed: boolean; ++ observedKind: "not_installed" | "no_owned_state" | "standard" ++ | "gateway" | "foreign" | "unsafe"; + applied: boolean; + stale: boolean; ++ drift: boolean; ++ driftReason: string | null; +@@ ++ if (typeof value.desiredEnabled !== "boolean" ++ || typeof value.installed !== "boolean" ++ || typeof value.observedKind !== "string" ++ || typeof value.drift !== "boolean") return null; +``` + +`native-api.ts` is not modified to add `claude-desktop` or `desiredEnabled`: WP3 +already shipped those, and WP5 has already added Codex detail. On that post-WP5 +module, WP6 extends only the Desktop reasons; `residualPaths` is parsed through +WP3's existing envelope field: + +```diff + export type NativeRefusalReason = + /* WP3 shared + WP5 Codex reasons */ ++ | "unsafe_metadata" ++ | "cleanup_incomplete"; +@@ + const NATIVE_REASONS = new Set([ + /* existing post-WP5 values */ ++ "unsafe_metadata", ++ "cleanup_incomplete", + ]); +``` Add `DESKTOP_DISABLE_COPY` beside `GROK_DISABLE_COPY` and branch on `pendingToggle.id`. Exact English source copy: > **Disable Claude Desktop integration?** > -> `{path}` will be updated to select a new credential-free opencodex profile with -> no inference provider. The previous opencodex profile and its backup will be -> removed. +> If `{path}` contains an opencodex-managed gateway profile, it will be updated to +> select a new credential-free opencodex profile with no inference provider, and +> the previous profile and backup will be removed. If no Claude Desktop library or +> managed profile exists, nothing will be created or removed. > > Claude Desktop will stop using models routed through opencodex and return to > standard Claude. @@ -470,7 +963,8 @@ The refusal/partial copy is equally exact: - `write_failed` before the pointer pivot — use the server message and say no Desktop library change completed; desired OFF remains saved if step 1 passed. -On the Desktop page, add `enabled` to `DesktopStatus`. When false, the status bar +On the Desktop page, consume required `desiredEnabled` from `DesktopStatus`; do +not add a second `enabled` alias. When false, the status bar says: "Claude Desktop integration is off. Desktop reads configuration only at launch; if it was open during the change, fully quit and reopen it." Save remains available because assignments/defaults are intentionally preserved; Save + Apply @@ -500,6 +994,10 @@ integrations.dialog.desktop.restart integrations.dialog.desktop.confirm integrations.detail.desktopDesiredOff integrations.detail.desktopDesiredOnNotApplied +integrations.detail.desktopSelectedElsewhere +integrations.detail.desktopProfileDrift +integrations.detail.desktopObservedUnsafe +integrations.detail.desktopNotInstalled integrations.native.error.desktopUnsafeMetadata integrations.native.error.desktopCleanupIncomplete integrations.native.msg.desktopDisabled @@ -518,34 +1016,50 @@ or profile JSON into any interpolation. `OPENCODEX_CLAUDE_DESKTOP_CONFIG_DIR`/explicit options and `mkdtempSync`; it never resolves the user's live library. -1. Normal disable writes a fresh UUID `{}` profile, verifies it is readable, +1. Point the path override at a child directory that does not exist. Inspector, + native GET, rich status GET, and OFF each return `not_installed`/unchanged with + required `desiredEnabled`; the child path remains absent after every call. + Repeat with an existing empty library and no `_meta.json`: OFF is a no-op and + creates neither metadata nor a profile. Explicit apply is the first operation + allowed to create them. Finally persist desired OFF beside an existing selected + gateway and call both GETs: they report drift but every Desktop-library byte and + mtime remains unchanged; only startup/ensure/explicit OFF may converge it. +2. Normal disable of an existing selected gateway writes a fresh UUID `{}` profile, verifies it is readable, points `appliedId` to it, removes only the old opencodex row, and preserves unrelated rows/top-level fields. -2. **Dangling Default fixture:** `_meta.json` contains `Default`, but its +3. **Dangling Default fixture:** `_meta.json` contains `Default`, but its `.json` does not exist. Disable neither selects nor changes that row; the selected fresh standard file exists. This reproduces the real-machine fact. -3. The replacement JSON has no `inferenceProvider`, +4. The replacement JSON has no `inferenceProvider`, `inferenceGatewayApiKey`, or other credential field. Do not assert by printing values; assert key absence. -4. The old `.json.bak` exists before removal and is absent after. This is a +5. The old `.json.bak` exists before removal and is absent after. This is a mandatory security assertion, not incidental cleanup. The old `.json` is also absent. -5. Crash fixtures resume from every table row above: selected standard + both old +6. Crash fixtures resume from every table row above: selected standard + both old files; backup gone; both files gone + old row present; markers handled at the - route layer. Every retry ends with one selected opencodex row and no old files. -6. Metadata malformed, path-escaping id, and two non-selected opencodex cleanup + route layer. Every retry removes locatable old credential files and leaves one + selected standard row. The pre-metadata orphan case explicitly permits the one + unlocatable credential-free orphan documented above. +7. Metadata malformed, path-escaping id, and two non-selected opencodex cleanup rows each REFUSE `unsafe_metadata` without a Desktop-library write. -7. Injected delete failure returns `cleanup_incomplete`, keeps the old row as +8. Injected delete failure returns `cleanup_incomplete`, keeps the old row as locator, reports residual paths only, and leaves selected standard readable. -8. Idempotent retry allocates no second standard profile. -9. Re-enable prefers the selected standard row, overwrites it through the normal +9. Idempotent retry from a selected standard profile allocates no second profile. +10. Re-enable prefers the selected standard row, overwrites it through the normal gateway writer, and does not revive an old interrupted-cleanup id. +11. Inspector fixtures cover selected foreign, selected standard, valid gateway, + edited gateway fingerprint drift, selected file deleted, invalid provider / + credential shape, and malformed metadata. It returns only classification, + paths, and hashes—never profile contents or credential values. `tests/native-claude-desktop-toggle.test.ts` follows the injected-persist seam in `tests/native-claude-code-toggle.test.ts:18-43`: -1. Absent WP3 key reads desired ON; upgrade behavior is unchanged. -2. Disable persists desired false and `desktopAutoApply:false` before the remover +1. Absent WP3 key reads desired ON; every status/success fixture includes the + required `desiredEnabled` field, including not-installed and no-op OFF. +2. Disable field-scoped-patches desired false and `desktopAutoApply:false` through + `mutatePersistedConfig` before the remover seam is called; a spy records call order. 3. Successful cleanup clears only `appliedFingerprint`/`appliedAt` and preserves all assignments/defaults (include 33 assignments to pin the observed scale). @@ -554,12 +1068,26 @@ resolves the user's live library. 5. Cleanup partial returns 500, residual paths, selected-standard state, desired OFF, and no false success. 6. Config `SQLITE_BUSY` refuses before any Desktop mutation; broken lock is 500. -7. Two concurrent PUTs produce one `config_busy` and no overlapping remover. -8. Auto-apply that is paused in `fetchAllModels`, then disabled, hits the second +7. A competing persisted update to another `claudeCode` field between read and + commit survives disable, enable, POST apply, and marker cleanup. This proves + field-scoped rebasing rather than whole-subtree replacement. +8. Explicit POST apply paused before its writer, then a competing OFF arrives on + the same per-client flight. The competitor is typed-refused; no ON/OFF overlap + occurs and maximum concurrent Desktop mutators is one. Repeat with native + enable versus OFF and from a second process against WP3's SQLite coordinator. +9. Auto-apply that is paused in `fetchAllModels`, then disabled, hits the second guard and never calls `writeDesktop3pConfig`. This activates the race fix at `agent-settings-routes.ts:137-139`, not merely its first guard. -9. Enable and explicit POST apply persist desired true, keep assignments/defaults, +10. Enable and explicit POST apply persist desired true, keep assignments/defaults, regenerate the gateway profile, and record markers only after write success. +11. Persist OFF, stop before invoking the remover, then run WP3 startup + reconciliation. It enters the same flight, re-runs removal, and converges the + selected gateway to standard mode. Repeat every partial state in the table; + no retry creates a library when none exists. +12. Rich status derives observation from selected id + selected file + parsed + provider/credential shape before fingerprint. Changing selection, deleting or + editing the selected file, and selecting standard mode all change observed + status while the saved marker stays constant; desired state does not change. MODIFY `tests/claude-messages-endpoint.test.ts`: start from Claude Code enabled, perform the Desktop disable PUT against a temp config library, then send a valid @@ -571,12 +1099,15 @@ shut down the shared transport. GUI cases: - `integrations-overview-rows.test.ts`: switch uses desired state while badge and - applied count use observed state; OFF + stale marker is not green. + applied count use observed state; OFF + gateway selected, ON + foreign selected, + edited/deleted selected profile, and standard mode each show explicit drift. + `not_installed` is not rendered as applied and no missing `desiredEnabled` is + silently inferred from bookkeeping. - `integrations-surfaces.test.tsx`: Desktop card has a keyboard-operable switch; disable opens the Desktop—not Grok—dialog; all five paragraphs render in order; confirm calls `/api/native-integrations/claude-desktop`; restart-required text is visible before confirm; focus returns to the switch. -- `claude-desktop-locale.test.ts`: all 14 keys exist and are non-empty in all six +- `claude-desktop-locale.test.ts`: all 18 keys exist and are non-empty in all six locales. ## Verification @@ -613,14 +1144,26 @@ selection and delete credential-bearing files without separate approval. - **C7** (`000_plan.md:91-92`) — after a successful disable, `_meta.json.appliedId` names a present, readable `{}` profile with no `inferenceProvider` or credential - fields; the previous opencodex `.json` and `.json.bak` are absent. The - dangling-Default fixture proves no `Default` assumption entered the path. + fields and the previous opencodex `.json`/`.bak` are absent **when an owned + gateway existed**. If no library/metadata/owned row existed, successful OFF is + unchanged and creates nothing. The dangling-Default fixture proves no `Default` + assumption entered either path. - **C4** (`000_plan.md:86-87`) — disable does not stop/restart the proxy and does not change `claudeCode.enabled`; a Claude Code request still traverses `/v1/messages` after Desktop is disabled, and proxy health remains live. - Desired OFF and `desktopAutoApply:false` are durable before the Desktop write; - a failed/partial mutation reports desired OFF versus observed residue rather - than silently re-enabling. + startup reconciliation re-runs the idempotent remover after a crash, and every + partial-state fixture states exactly what the retry removes, preserves, or + refuses. A failed/partial mutation reports desired OFF versus observed residue + rather than silently re-enabling. +- Native/rich status is read-only and derives observation from selected id, + selected-file existence, parsed provider/credential shape, then fingerprint. + Desired state is required separately on every status/success/post-commit + refusal; changing Desktop state behind opencodex surfaces drift immediately. +- Auto-apply, POST apply, native enable, native disable, and startup OFF + reconciliation share WP3's `claude-desktop` flight and re-read persisted desired + state immediately before each irreversible Desktop write. Persistence uses + `mutatePersistedConfig` field patches, so unrelated `claudeCode` fields survive. - Assignments/defaults survive disable and enable byte-semantically as parsed data; only `appliedFingerprint`/`appliedAt` are cleared after complete cleanup. - The dialog states before confirmation that a full Desktop quit/reopen is From abb402e53ca5068b33315da2dd60cbd73e09eb28 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Tue, 4 Aug 2026 00:22:53 +0900 Subject: [PATCH 008/163] =?UTF-8?q?docs(integrations):=20round=202=20fails?= =?UTF-8?q?=20too=20=E2=80=94=20the=20phase=20map=20is=20the=20defect,=20n?= =?UTF-8?q?ot=20the=20fixes?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Five closed, six still open, six NEW including three High. A converging audit closes more than it opens; this one did the opposite, and one new finding is that my fix for 'startup must converge' can tear down another installed service's Codex/Grok state from a different OPENCODEX_HOME, because the reconciliation registry calls the removers without assertNativeTeardownOwned. A repair that is worse than the defect it repairs is the signal to stop patching. Root cause: I coupled ten clients into one schema. A ten-key clientIntegrations map forced every phase to touch every client's write path, so each repair round widened the blast radius. The evidence is that WP2 passed both rounds untouched — it is the only phase that changes one thing at one boundary. Replan: ship the modality fix and the API-keys row alone, re-scope desired state to Codex only, leave Claude Code's ingress gates entirely alone, and move Desktop behind Codex to be audited on its own. Same four deliverables, sliced along ownership boundaries instead of along a schema. Also fixes the stale citations round 1 flagged and round 2 found unfixed: restore is cli/index.ts:770 not :745, the sync is :756 not :757, the Desktop field is types.ts:458-459 not :456, and the doc lastmod dates are removed because the pages do not show them — the semantic claims were verified twice. --- .../001_native_restore_thesis.md | 6 +- .../002_desktop_standard_mode.md | 18 ++-- .../006_audit_synthesis_r2.md | 90 +++++++++++++++++++ 3 files changed, 103 insertions(+), 11 deletions(-) create mode 100644 devlog/_plan/260803_codex_desktop_toggle/006_audit_synthesis_r2.md diff --git a/devlog/_plan/260803_codex_desktop_toggle/001_native_restore_thesis.md b/devlog/_plan/260803_codex_desktop_toggle/001_native_restore_thesis.md index 50d0a87f2..edba568ba 100644 --- a/devlog/_plan/260803_codex_desktop_toggle/001_native_restore_thesis.md +++ b/devlog/_plan/260803_codex_desktop_toggle/001_native_restore_thesis.md @@ -22,10 +22,10 @@ For Codex the counter-thesis is correct, and the evidence is not subtle. > Restore native Codex config without stopping the proxy; `restore back` > re-points codex at the running proxy. -That is the toggle, both directions, with the proxy up. `src/cli/index.ts:745` +That is the toggle, both directions, with the proxy up. `src/cli/index.ts:770` calls `restoreNativeCodex()` with no lifecycle operation anywhere near it, and -`src/cli/index.ts:757` implements the enable direction as `syncModelsToCodex(live.port)` -against a proxy it first proves is live via `findLiveProxy()`. +`src/cli/index.ts:756` implements the enable direction as `syncModelsToCodex(live.port)` +against a proxy it first proves is live via `findLiveProxy()` (`:751`). Stronger still: `POST /api/stop` (`src/server/management-api.ts:181`) calls `restoreNativeCodex()` FIRST and only then schedules the drain and exit. The diff --git a/devlog/_plan/260803_codex_desktop_toggle/002_desktop_standard_mode.md b/devlog/_plan/260803_codex_desktop_toggle/002_desktop_standard_mode.md index 4b0f8e7d3..77976443d 100644 --- a/devlog/_plan/260803_codex_desktop_toggle/002_desktop_standard_mode.md +++ b/devlog/_plan/260803_codex_desktop_toggle/002_desktop_standard_mode.md @@ -16,7 +16,7 @@ Right, and re-confirmed here: - `/status.applied` is derived from the saved fingerprint, not from actual selection (`src/server/management/agent-settings-routes.ts:797`), so a disable that forgets to clear the markers keeps reporting `applied: true`. -- `desktopAutoApply` is enabled by ABSENCE (`src/types.ts:456`); its guard +- `desktopAutoApply` is enabled by ABSENCE (`src/types.ts:458-459`); its guard suppresses only an explicit `false`, and the subagent-model update route can re-create a removed profile (`agent-settings-routes.ts:130,518`). @@ -26,14 +26,16 @@ first one is blocked. ## The official semantics that make disable safe -Primary Anthropic documentation, opened and read (not inferred from our devlog): +Primary Anthropic documentation, opened and read (not inferred from our devlog). +Dates are deliberately omitted: the sitemap `lastmod` values originally recorded +here are not visible on the pages themselves, and an independent reviewer could +not confirm them. The semantic claims below WERE independently re-verified by +that reviewer, twice. -| Source | lastmod | -|---|---| -| [Configuration reference](https://claude.com/docs/third-party/claude-desktop/configuration) | 2026-07-24 | -| [In-app configuration](https://claude.com/docs/third-party/claude-desktop/in-app-configuration) | 2026-07-17 | -| [Claude API provider](https://claude.com/docs/third-party/claude-desktop/claude-api) | 2026-07-17 | -| [Gateway provider](https://claude.com/docs/third-party/claude-desktop/gateway) | 2026-07-29 | +- [Configuration reference](https://claude.com/docs/third-party/claude-desktop/configuration) +- [In-app configuration](https://claude.com/docs/third-party/claude-desktop/in-app-configuration) +- [Claude API provider](https://claude.com/docs/third-party/claude-desktop/claude-api) +- [Gateway provider](https://claude.com/docs/third-party/claude-desktop/gateway) The load-bearing sentence: third-party mode activates **only** when `inferenceProvider` and that provider's required credentials are valid; diff --git a/devlog/_plan/260803_codex_desktop_toggle/006_audit_synthesis_r2.md b/devlog/_plan/260803_codex_desktop_toggle/006_audit_synthesis_r2.md new file mode 100644 index 000000000..c7376f209 --- /dev/null +++ b/devlog/_plan/260803_codex_desktop_toggle/006_audit_synthesis_r2.md @@ -0,0 +1,90 @@ +# Audit round 2 — synthesis, and the replan it forces + +Verdict: **FAIL**. Five of eleven closed, six still open, and **six NEW findings** +including three High. + +## The number that decides this round + +| Round | Closed | Still open | New | +|---|---|---|---| +| 1 | — | — | 11 | +| 2 | 5 | 6 | 6 | + +A converging audit closes more than it opens. This one did not. And the new +findings are not nitpicks — they are the same *class* as the old ones, found one +layer deeper: + +- round 1 said "persistence is unsafe" → round 2 says the fix drops the + `authModeMigratedAt` sentinel, so a client toggle silently pins the user's + Claude auth mode (`src/claude/auth-mode-migration.ts:16-31`) +- round 1 said "in-flight writers can re-enable" → round 2 says teardown callers + in `src/service.ts:2587-2592` and `management-api.ts:181-186` still bypass the + flight entirely +- round 1 said "startup must converge" → round 2 says the convergence I added can + **tear down another installed service's Codex/Grok state** from a different + `OPENCODEX_HOME`, because the registry calls the removers directly without + `assertNativeTeardownOwned` + +That last one is the tell. My fix for a finding *created a worse defect than the +finding*. And #5 (GUI union ownership) is still open after I explicitly assigned +the shared contract to WP3 — because I put the server contract there and left the +GUI parser unowned. + +## Root cause (LOOP-REPAIR-01 → root-cause mode) + +Two failed repair rounds on the same failure means stop patching and diagnose. + +The diagnosis: **I coupled ten clients into one schema change.** `clientIntegrations` +as a ten-key map forced every phase to touch every client's write path, so each +round of fixes widened the blast radius instead of narrowing it. Codex, Grok, +Desktop, Claude Code and six file clients each have different ownership rules, +different teardown callers, and different migration histories. One map made them +one problem. + +The evidence that this is the cause rather than bad luck: **WP2 passed both +rounds untouched.** It is the only phase that changes one thing at one boundary. + +## The replan + +Return to P (LOOP-REPAIR-01 escalation) with a decoupled map. Not a smaller +objective — the same four deliverables, sliced so each is independently +auditable. + +1. **WP2 modality fix ships first, alone.** Clean through two adversarial rounds. + It fixes a live user-facing failure and depends on nothing here. +2. **WP4 API-keys row ships second, alone.** Never audited as blocking; pure GUI; + no coupling to the schema. +3. **Desired state is re-scoped to ONE client: Codex.** A `codex` flag, its + gates, its single-flight, its ownership preflight, its CLI semantics. Not a + ten-key map. Grok's regression gets its own later phase reusing whatever + shape survives audit. +4. **Claude Code's gates are left exactly as they are.** Round 1 #1 proved I had + no business touching that ingress; the amended plan keeps them, and now the + honest move is to not route them through a new helper at all in this unit. +5. **Desktop moves behind Codex** and is re-audited on its own once the + one-client shape is proven. The goal explicitly permits an evidenced deferral, + and round 2 #3 (no coherent rule when a foreign profile is selected) plus #6 + (marker cleanup failure ignored) say it is not ready. + +## What carries forward regardless + +Verified across both rounds and not in dispute: the native-restore thesis +(`001`), the official standard-mode contract (`002`, re-opened by the reviewer in +round 2), the Grok regression (`003`), and the modality defect (`004`). The +research holds. It was the *phase map* that was wrong, which is exactly what +PHASE-SPLIT-01 exists to catch and what I got wrong by slicing along a schema +instead of along ownership boundaries. + +## Carried-forward findings for the re-scoped phases + +Every open and new finding stays on the ledger, attached to whichever +single-client phase inherits it: + +| Finding | Inherits | +|---|---| +| r1 #5 GUI contract, r2 #1 auth sentinel | the Codex phase | +| r2 #2 ownership preflight in reconciliation | the Codex phase — **blocking** | +| r1 #6/#7 flight + convergence, incl. teardown callers | the Codex phase | +| r2 #4 existing-disabled migration, r2 #5 mutating GET | the file-client phase | +| r1 #8, r2 #3, r2 #6 | the Desktop phase | +| r1 #11 stale citations in `001`/`002` | fix now, in this commit | From 9483d428d3533d2b4975bf2de30ebc4548996718 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Tue, 4 Aug 2026 00:24:15 +0900 Subject: [PATCH 009/163] docs(integrations): re-slice the roadmap along ownership boundaries MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit WP2 modality fix and WP3 api-keys row ship first and alone, because each changes one thing at one boundary — the only property that survived both audit rounds. Desired state re-scopes to Codex only; Grok gets its own later phase reusing whatever shape WP4 proves; Desktop moves behind Codex. Three exclusions, each an accepted finding rather than a convenience: Claude Code's ingress gates are not touched at all, the six file clients get no flag in this unit, and Desktop is deferred pending its own audit. Adds C7 for the foreign-home teardown hazard round 2 found, and a risk-register row for the failure mode round 2 actually demonstrated — a repair worse than its defect. --- .../260803_codex_desktop_toggle/000_plan.md | 64 +++++++++++++------ 1 file changed, 43 insertions(+), 21 deletions(-) diff --git a/devlog/_plan/260803_codex_desktop_toggle/000_plan.md b/devlog/_plan/260803_codex_desktop_toggle/000_plan.md index aca5666a2..400b5d8bd 100644 --- a/devlog/_plan/260803_codex_desktop_toggle/000_plan.md +++ b/devlog/_plan/260803_codex_desktop_toggle/000_plan.md @@ -46,21 +46,40 @@ conclusion is now **superseded** by `001`-`003` here. ## Phases -Dependency-ordered (PHASE-SPLIT-01): the schema is the foundation every switch -consumes, so it goes first even though it ships no visible switch. - -| Phase | Doc | Deliverable | -|---|---|---| -| WP2 | `010_modality_boundary.md` | The client-dialect modality filter. Independent of everything else; fixes a live user-visible failure | -| WP3 | `020_desired_state.md` | `OcxConfig.clientIntegrations`, default-ON, consulted by every automatic apply path — including the Grok regression | -| WP4 | `030_api_keys_row.md` | API keys out of the card grid into their own row | -| WP5 | `040_codex_toggle.md` | The Codex switch on top of WP3, with a structured restore result | -| WP6 | `050_desktop_toggle.md` | The Desktop switch via documented standard mode | - -WP1 was this cycle: the research above plus this roadmap. - -WP2 and WP4 are independent of the rest and of each other. WP5 and WP6 are -parallel siblings that both depend on WP3. +**Re-sliced after audit round 2** (`006_audit_synthesis_r2.md`). The first map +sliced along a schema — one ten-key `clientIntegrations` map — which forced every +phase to touch every client's write path. Two audit rounds widened rather than +narrowed. This map slices along **ownership boundaries**: one client family per +phase, each independently auditable. + +| Phase | Doc | Deliverable | Audit state | +|---|---|---|---| +| WP2 | `010_modality_boundary.md` | The client-dialect modality filter | **clean through two rounds** | +| WP3 | `030_api_keys_row.md` | API keys out of the card grid into their own row | never blocking | +| WP4 | `020_desired_state.md` → re-scoped | Desired state for **Codex only**, plus its gates, single-flight, ownership preflight and CLI semantics | re-scope pending | +| WP5 | `040_codex_toggle.md` | The Codex switch on WP4's flag | rewrite pending | +| WP6 | *(new doc)* | Grok's desired state, reusing the shape WP4 proved | not started | +| WP7 | `050_desktop_toggle.md` | The Desktop switch via documented standard mode | deferred behind WP5 | + +WP1 was the research cycle plus this roadmap, twice audited. + +WP2 and WP3 depend on nothing here and on each other not at all — they ship +first, alone, because each changes one thing at one boundary. That property is +the only thing that survived both audit rounds intact. + +Three deliberate exclusions, each an accepted audit finding rather than a +convenience: + +- **Claude Code's ingress gates are not touched at all.** Round 1 #1 established + that `claudeCode.enabled` is the documented kill switch for `/v1/messages`; + the honest conclusion is not to route it through a new helper in this unit. +- **The six file clients get no desired-state flag here.** Round 2 #4 (existing + explicit OFF choices are not migrated) and #5 (a mutating GET) both belong to a + phase that does not exist yet, and inventing it under audit pressure is what + produced round 2's new findings. +- **Desktop is behind Codex, not beside it.** Round 2 #3 found no coherent rule + when a foreign profile is selected. The goal explicitly permits an evidenced + deferral; this is one. ## Scope boundary @@ -80,18 +99,19 @@ recording the previous `appliedId` (deferred, `002` §Residual). - C1 — gjc loads our emitted config with no schema error, proven from the real file; Pi's identical exposure is closed in the same change. -- C2 — a disabled client stays disabled across a proxy restart, an `ocx ensure`, - and a `POST /api/sync`. +- C2 — Codex stays disabled across a proxy restart, an `ocx ensure`, and a + `POST /api/sync`. - C3 — an upgrading user with no `clientIntegrations` key sees no behavior change. -- C4 — disabling any client never stops the proxy and never disables a shared - transport used by another client. +- C4 — disabling Codex never stops the proxy and never closes `/v1/responses`. - C5 — Codex toggles both directions from the overview with the proxy running. - C6 — a Codex disable blocked by the held history DB is an explained refusal naming the cause, never a raw 500 and never a false green. -- C7 — Desktop's disable points `appliedId` at a present, readable, - credential-free config and removes the credential-bearing `.bak`. +- C7 — startup convergence never touches state owned by a service running from a + different `OPENCODEX_HOME` (round 2 #2, blocking). - C8 — API keys render as a row above the grid, observed rendered. - C9 — typecheck, full test, gui lint, gui test, privacy scan all green. +- C10 — Desktop ships with a proven restore path OR is deferred with recorded + evidence. Both outcomes close this criterion. ## Risk register @@ -102,3 +122,5 @@ recording the previous `appliedId` (deferred, `002` §Residual). | Desktop pointed at a missing file | Never delete `appliedId`, never leave it dangling, never pick an entry by the name `Default` — this machine's `Default` is already dangling (`002`) | | The modality fix erases valid internal metadata | Filter at the client-dialect boundary only; management and CLI keep carrying `audio` verbatim (`004`) | | A green suite hides the real failure | 91 tests pass today beside a config gjc refuses to load. Every criterion names a live artifact, not a unit test | +| A repair introduces a worse defect than the one it fixes | Round 2 produced exactly this. One client family per phase, re-audited before the next starts — never a cross-client rewrite under audit pressure | +| Convergence tears down a foreign home's state | Every native remover runs `assertNativeTeardownOwned` before removal, and reconciliation runs only after service-ownership resolution (round 2 #2) | From 06179a8e5c499f1f219623e3a1e1ae43b8a2081b Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Tue, 4 Aug 2026 00:39:30 +0900 Subject: [PATCH 010/163] =?UTF-8?q?docs(integrations):=20fold=20round=203?= =?UTF-8?q?=20into=20WP2=20=E2=80=94=20unknown=20and=20incompatible=20are?= =?UTF-8?q?=20different=20inputs?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The audit caught a real design error, not a nit. My helper mapped an audio-only model to [text], which advertises a capability the model does not have — and that input is reachable three ways: ocx models add --modalities audio, POST /api/custom-models, and provider discovery. So the helper now returns null for incompatible and the builders drop the row, while a model with NO declared modalities still gets [text] because unknown is not incompatible. Omitting a model costs a row in a picker; fabricating text costs a model that fails at call time with no explanation. Also replaces the grep-based verification, which would false-positive on 'audio' appearing in a model id or another provider's preserved block, with a parsed check of providers.opencodex.models[*].input. Completes the carried-forward ledger the audit found incomplete: every finding from both rounds now has a status and an inheriting phase, with the deferred file-client work named FOLLOWUP-FILECLIENT-01 rather than left pointing at a phase that does not exist. Renames the two pending docs so decade order matches phase order. --- .../260803_codex_desktop_toggle/000_plan.md | 8 +- .../006_audit_synthesis_r2.md | 41 +++-- .../010_modality_boundary.md | 145 +++++++++++++----- ...30_api_keys_row.md => 020_api_keys_row.md} | 0 ..._desired_state.md => 030_desired_state.md} | 0 5 files changed, 139 insertions(+), 55 deletions(-) rename devlog/_plan/260803_codex_desktop_toggle/{030_api_keys_row.md => 020_api_keys_row.md} (100%) rename devlog/_plan/260803_codex_desktop_toggle/{020_desired_state.md => 030_desired_state.md} (100%) diff --git a/devlog/_plan/260803_codex_desktop_toggle/000_plan.md b/devlog/_plan/260803_codex_desktop_toggle/000_plan.md index 400b5d8bd..8a2814875 100644 --- a/devlog/_plan/260803_codex_desktop_toggle/000_plan.md +++ b/devlog/_plan/260803_codex_desktop_toggle/000_plan.md @@ -55,12 +55,16 @@ phase, each independently auditable. | Phase | Doc | Deliverable | Audit state | |---|---|---|---| | WP2 | `010_modality_boundary.md` | The client-dialect modality filter | **clean through two rounds** | -| WP3 | `030_api_keys_row.md` | API keys out of the card grid into their own row | never blocking | -| WP4 | `020_desired_state.md` → re-scoped | Desired state for **Codex only**, plus its gates, single-flight, ownership preflight and CLI semantics | re-scope pending | +| WP3 | `020_api_keys_row.md` | API keys out of the card grid into their own row | never blocking | +| WP4 | `030_desired_state.md` → re-scoped | Desired state for **Codex only**, plus its gates, single-flight, ownership preflight and CLI semantics | re-scope pending | | WP5 | `040_codex_toggle.md` | The Codex switch on WP4's flag | rewrite pending | | WP6 | *(new doc)* | Grok's desired state, reusing the shape WP4 proved | not started | | WP7 | `050_desktop_toggle.md` | The Desktop switch via documented standard mode | deferred behind WP5 | +The two pending docs were renamed so the decade order matches the phase order: +reading the unit lexicographically now gives the build order, which is the whole +point of the numbering convention. + WP1 was the research cycle plus this roadmap, twice audited. WP2 and WP3 depend on nothing here and on each other not at all — they ship diff --git a/devlog/_plan/260803_codex_desktop_toggle/006_audit_synthesis_r2.md b/devlog/_plan/260803_codex_desktop_toggle/006_audit_synthesis_r2.md index c7376f209..f918b7a54 100644 --- a/devlog/_plan/260803_codex_desktop_toggle/006_audit_synthesis_r2.md +++ b/devlog/_plan/260803_codex_desktop_toggle/006_audit_synthesis_r2.md @@ -77,14 +77,33 @@ instead of along ownership boundaries. ## Carried-forward findings for the re-scoped phases -Every open and new finding stays on the ledger, attached to whichever -single-client phase inherits it: - -| Finding | Inherits | -|---|---| -| r1 #5 GUI contract, r2 #1 auth sentinel | the Codex phase | -| r2 #2 ownership preflight in reconciliation | the Codex phase — **blocking** | -| r1 #6/#7 flight + convergence, incl. teardown callers | the Codex phase | -| r2 #4 existing-disabled migration, r2 #5 mutating GET | the file-client phase | -| r1 #8, r2 #3, r2 #6 | the Desktop phase | -| r1 #11 stale citations in `001`/`002` | fix now, in this commit | +Round 3 found my first version of this table incomplete — findings were missing, +two were assigned to one phase when three inherit them, and two pointed at a +"file-client phase" the roadmap does not contain. A ledger that quietly drops an +obligation is worse than no ledger, because the next author reads absence as +closure. Complete version, every finding from both rounds: + +| Finding | Status | Inherits | +|---|---|---| +| r1 #1 Claude ingress gates | **scope-eliminated** — this unit no longer touches them | — | +| r1 #2 unsafe persistence | open | WP4 (Codex), then reused by WP6/WP7 | +| r1 #3 file clients have no writer | **deferred** — no file-client flag in this unit; tracked as `FOLLOWUP-FILECLIENT-01` | a future unit | +| r1 #4 false-green CLI | open | WP4 | +| r1 #5 GUI contract ownership | open | WP4 defines it; **WP5 and WP7 both extend it** | +| r1 #6 in-flight writers | open | WP4 for Codex, WP6 for Grok, WP7 for Desktop — each must prove its OWN callers | +| r1 #7 restart reconciliation | open | same three, per client | +| r1 #8 Desktop status from stale bookkeeping | open | WP7 | +| r1 #9 Desktop status can create files | closed in r2 | WP7 keeps the test | +| r1 #10 test adequacy | open | **every** phase — each lands the test that would catch its own absence | +| r1 #11 stale citations | fixed | `001`, `002`, and `010` in this cycle | +| r2 #1 auth-mode sentinel dropped | open | WP4 | +| r2 #2 reconciliation can tear down a foreign home | open — **blocking** | WP4, and every native remover after it | +| r2 #3 Desktop foreign-selected profile has no rule | open | WP7 — the reason Desktop is deferred | +| r2 #4 existing-disabled clients not migrated | **deferred** with r1 #3 | `FOLLOWUP-FILECLIENT-01` | +| r2 #5 mutating GET | open | WP4 — status GETs stay inspection-only | +| r2 #6 Desktop marker cleanup failure ignored | open | WP7 | + +`FOLLOWUP-FILECLIENT-01` is a named placeholder, not a phase: the six file +clients get no desired-state flag in this unit, so r1 #3 and r2 #4 are recorded +as owed work rather than silently dropped. Inventing that phase under audit +pressure is what produced round 2's new findings. diff --git a/devlog/_plan/260803_codex_desktop_toggle/010_modality_boundary.md b/devlog/_plan/260803_codex_desktop_toggle/010_modality_boundary.md index b0c5c3e36..eaf6d2367 100644 --- a/devlog/_plan/260803_codex_desktop_toggle/010_modality_boundary.md +++ b/devlog/_plan/260803_codex_desktop_toggle/010_modality_boundary.md @@ -33,75 +33,124 @@ MODIFY `src/clients/config-export.ts`, immediately after `outputBudgetFor` * * This is the same defect the Codex catalog had with `video`, where the app * showed zero apps (tests/catalog-input-modality-enum.test.ts). The fix is the - * same shape: filter to what the destination accepts, and fall back to `text` - * rather than an empty list — a modality-less entry would leave the client - * unable to tell the model takes prompts at all. + * same shape — filter to what the destination accepts — with one deliberate + * difference, below. * - * Deliberately NOT applied in ExportModel construction: the management and CLI - * boundaries carry catalog modalities verbatim on purpose, and stripping `audio` - * globally would destroy valid metadata before the destination is known. - */ + * UNKNOWN and INCOMPATIBLE are not the same input, and the Codex fix could + * conflate them safely only because its enum is wider. A model with no declared + * modalities is unknown, and `text` is the honest floor: every routed model + * takes prompts. A model that declares `["audio"]` and nothing else is + * INCOMPATIBLE with a text|image client, and rewriting it to `["text"]` would + * advertise a capability the model does not have. That input is reachable: + * `ocx models add --modalities audio` accepts it (src/cli/models.ts:139-146), + * `/api/custom-models` accepts it (model-routes.ts:13), and provider discovery + * can return an audio-only list (src/codex/catalog/provider-fetch.ts:341). + * + * So unknown falls back to text, and incompatible omits the model. Omitting one + * model costs the user that row in a picker; fabricating `text` costs them a + * model that fails at call time with no explanation. + * + * Deliberately NOT applied in ExportModel construction: the management and CLI + * boundaries carry catalog modalities verbatim on purpose, and stripping `audio` + * globally would destroy valid metadata before the destination is known. + */ const CLIENT_INPUT_MODALITIES: Record<"pi" | "gajae", ReadonlySet> = { pi: new Set(["text", "image"]), gajae: new Set(["text", "image"]), }; +/** + * `null` means "this model cannot be represented for this client" — the caller + * drops the row. Deliberately not an empty array, which a caller could spread + * into a config without noticing. + */ function inputModalitiesForClient( client: "pi" | "gajae", modalities: readonly string[] | undefined, -): string[] { +): string[] | null { + const declared = modalities ?? []; + // Nothing declared is unknown, not incompatible. + if (declared.length === 0) return ["text"]; const accepted = CLIENT_INPUT_MODALITIES[client]; const kept: string[] = []; - for (const value of modalities ?? []) { + for (const value of declared) { if (accepted.has(value) && !kept.includes(value)) kept.push(value); } - return kept.length > 0 ? kept : ["text"]; + return kept.length > 0 ? kept : null; } ``` -Order-preserving and deduping, so `[text, image, audio]` becomes `[text, image]` -and the existing byte-exact golden is unaffected for models that never carried -`audio`. +Order-preserving and deduping, so `[text, image, audio]` becomes `[text, image]`. +The existing byte-exact golden is unaffected: its fixture declares no modalities, +which still emits `["text"]` through the unknown branch. ## Call site 1 — Pi -`buildPiClientConfig`, currently line 659: +`buildPiClientConfig`, currently line 653. The `map` becomes a `for` because the +helper now filters as well as transforms, and a `null` sentinel inside a `map` +would need a second pass: ```diff - const entry: PiModelEntry = { - id: model.namespaced, - name: exportModelLabel(model), +- const models: PiModelEntry[] = normalizeExportModels(ctx.models).map(model => { +- const entry: PiModelEntry = { +- id: model.namespaced, +- name: exportModelLabel(model), - // Text is the one modality every routed model supports; anything richer must come - // from the catalog rather than an assumption. - input: model.inputModalities && model.inputModalities.length > 0 ? [...model.inputModalities] : ["text"], -+ // Text is the one modality every routed model supports; anything richer must come -+ // from the catalog rather than an assumption — and must still be inside the -+ // enum Pi accepts, because Pi returns an EMPTY model config on a schema -+ // failure rather than dropping the offending entry. -+ input: inputModalitiesForClient("pi", model.inputModalities), - }; +- }; ++ const models: PiModelEntry[] = []; ++ for (const model of normalizeExportModels(ctx.models)) { ++ // Pi returns an EMPTY model config on a schema failure rather than dropping ++ // the offending entry, so one out-of-enum value costs every routed model. ++ const input = inputModalitiesForClient("pi", model.inputModalities); ++ // An audio-only model has no honest representation here; claiming `text` ++ // would fail at call time instead, so the row is dropped. ++ if (input === null) continue; ++ const entry: PiModelEntry = { id: model.namespaced, name: exportModelLabel(model), input }; + const context = authoritativeContextWindow(model.contextWindow); + if (context !== undefined) { + entry.contextWindow = context; + entry.maxTokens = outputBudgetFor(context); + } +- return entry; +- }); ++ models.push(entry); ++ } ``` Also MODIFY the stale docstring above `buildPiClientConfig` (line 649), which -still says Pi's schema is UNVERIFIED. It is verified now — upstream -`packages/coding-agent/src/core/model-config.ts:156-169` pins `text|image`, and -`:267-274` is the whole-file rejection. Replace the "UNVERIFIED" sentence with -that citation. +still says Pi's schema is UNVERIFIED. It is verified now. Cite the *stable* doc +rather than a line range that has already drifted between audit rounds +(`packages/coding-agent/docs/models.md` states the accepted values), and name the +behavior rather than the line: Pi returns an empty model config when validation +fails. A line-pinned citation into a moving upstream file is a comment that rots. ## Call site 2 — Gajae -`buildGajaeClientConfig`, currently line 765: +`buildGajaeClientConfig`, currently line 761, takes the identical shape: ```diff - const entry: GajaeModelEntry = { - id: model.namespaced, - name: exportModelLabel(model), +- const models: GajaeModelEntry[] = normalizeExportModels(ctx.models).map(model => { +- const entry: GajaeModelEntry = { +- id: model.namespaced, +- name: exportModelLabel(model), - input: model.inputModalities && model.inputModalities.length > 0 - ? [...model.inputModalities] - : ["text"], -+ input: inputModalitiesForClient("gajae", model.inputModalities), - }; -``` +- }; ++ const models: GajaeModelEntry[] = []; ++ for (const model of normalizeExportModels(ctx.models)) { ++ const input = inputModalitiesForClient("gajae", model.inputModalities); ++ if (input === null) continue; ++ const entry: GajaeModelEntry = { id: model.namespaced, name: exportModelLabel(model), input }; + ``` + +with the same `return entry;` → `models.push(entry);` change at the loop tail. + +Gajae's enum is at `models-config-schema.ts:141` in the installed +`@gajae-code/coding-agent` — line 119, which `004` cited, only opens the model +schema object. Cite the installed version alongside the line. ## Test — `tests/client-export-modality-enum.test.ts` (NEW) @@ -112,13 +161,19 @@ repeats. Cases: `zenmux/meta-muse-spark-1.1` with `[text, image, audio]`, asserting `[text, image]`. 2. The same for Pi, so the latent half is pinned too. -3. A model whose only modality is rejected falls back to `["text"]`, never `[]`. -4. `[text, image]` survives untouched in both. -5. Order and dedupe: `[image, text, image]` yields `[image, text]`. -6. A whole-catalog assertion: no emitted Pi or Gajae `input` value is outside - `text|image`, given a catalog containing `audio`. This is the one that would - have caught the bug, since the per-entry tests all passed while the file was - broken. +3. **An audio-only model is OMITTED from both exports, not rewritten to + `["text"]`.** Assert its id is absent from `models` entirely. The fixture + builds it the way a user reaches it — `ocx models add --modalities audio` + accepts exactly this (`src/cli/models.ts:139-146`). +4. A model with NO declared modalities still emits `["text"]`. Unknown is not + incompatible, and this branch is what keeps the byte-exact golden stable. +5. `[text, image]` survives untouched in both. +6. Order and dedupe: `[image, text, image]` yields `[image, text]`. +7. A whole-catalog assertion over a catalog carrying BOTH a mixed + `[text, image, audio]` model and an audio-ONLY model: every emitted Pi and + Gajae `input` value is inside `text|image`, and no emitted model claims a + modality its catalog entry did not have. This is the case that would have + caught the live bug — every per-entry test passed while the file was broken. ## Verification @@ -127,7 +182,11 @@ refuses to load. 1. `bun run typecheck`, `bun run test` 2. Re-apply the gajae integration through the running proxy -3. `grep -c audio ~/.gjc/agent/models.yml` → 0 inside the opencodex block +3. **Parse** the emitted YAML and assert every + `providers.opencodex.models[*].input` value is in `text|image`. A `grep` for + `audio` is the wrong check: the string also appears in model ids, display + names, and other providers' preserved blocks, so it can fail while our output + is correct. Same structural check against Pi's JSON. 4. Launch gjc and confirm the model list loads with no schema error Step 4 is the criterion. Steps 1-3 are necessary and insufficient. @@ -138,5 +197,7 @@ Step 4 is the criterion. Steps 1-3 are necessary and insufficient. file, and Pi's identical exposure is closed in the same change. - No emitted Pi/Gajae `input` value outside the client's enum, asserted over a whole catalog rather than one entry. +- No model is advertised to a client with a modality it does not have: an + incompatible model is omitted, never rewritten to `text`. - The byte-exact export goldens still pass, changing only where `audio` was previously emitted. diff --git a/devlog/_plan/260803_codex_desktop_toggle/030_api_keys_row.md b/devlog/_plan/260803_codex_desktop_toggle/020_api_keys_row.md similarity index 100% rename from devlog/_plan/260803_codex_desktop_toggle/030_api_keys_row.md rename to devlog/_plan/260803_codex_desktop_toggle/020_api_keys_row.md diff --git a/devlog/_plan/260803_codex_desktop_toggle/020_desired_state.md b/devlog/_plan/260803_codex_desktop_toggle/030_desired_state.md similarity index 100% rename from devlog/_plan/260803_codex_desktop_toggle/020_desired_state.md rename to devlog/_plan/260803_codex_desktop_toggle/030_desired_state.md From 1a76376ba828908272a20eba23d1dabe2ac60c3f Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Tue, 4 Aug 2026 00:41:51 +0900 Subject: [PATCH 011/163] fix(clients): stop one rejected modality from poisoning a whole client config MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit gjc refused its entire config with /providers/opencodex/models/30/input/2: Invalid option: expected one of text|image, and fell back to its built-in list — every routed model gone. Index 30 is zenmux/meta-muse-spark-1.1, which advertises audio. Our internal vocabulary is text|image|audio and both Pi and Gajae accept only text|image, so the exporters were copying a value that costs the whole file. Pi carried the identical bug, unobserved only because its config was empty here; it returns an EMPTY model config on a schema failure. Same class as the Codex 'video' incident that showed zero apps, fixed at the client-dialect boundary rather than in ExportModel or normalizeExportModels, so the management and CLI surfaces keep carrying audio verbatim. Unknown and incompatible are treated differently: nothing declared falls back to text, since every routed model takes prompts, but a model declaring only audio is dropped rather than rewritten to text — that input is reachable via ocx models add --modalities audio, /api/custom-models, and provider discovery, and claiming text would fail at call time with no explanation. Test proven by reverting the fix: 5 of 7 cases fail without it, including the whole-catalog assertion the existing per-entry tests lacked while the real file was broken. Existing 91 export tests unchanged. --- src/clients/config-export.ts | 89 ++++++++++++--- tests/client-export-modality-enum.test.ts | 132 ++++++++++++++++++++++ 2 files changed, 207 insertions(+), 14 deletions(-) create mode 100644 tests/client-export-modality-enum.test.ts diff --git a/src/clients/config-export.ts b/src/clients/config-export.ts index dabc64aad..0a4a84ed0 100644 --- a/src/clients/config-export.ts +++ b/src/clients/config-export.ts @@ -433,6 +433,55 @@ function outputBudgetFor(context: number): number { return Math.min(SCHEMA_REQUIRED_OUTPUT_BUDGET, context); } +/** + * Modalities a given client's schema will actually accept. + * + * Our internal vocabulary is `text | image | audio` (ALLOWED_INPUT_MODALITIES in + * src/server/management/model-routes.ts). Pi and Gajae accept only + * `text | image`, and both reject the WHOLE config file over one out-of-enum + * value — Gajae reports `/providers/opencodex/models/N/input/2: Invalid option` + * and falls back to its built-in list, Pi returns an empty model config. So a + * single `audio` model takes every routed model down with it. That is not + * hypothetical: zenmux/meta-muse-spark-1.1 advertises audio and did exactly + * this. It is also the same defect the Codex catalog had with `video`, where + * the app showed zero apps (tests/catalog-input-modality-enum.test.ts). + * + * UNKNOWN and INCOMPATIBLE are different inputs, and the Codex fix could + * conflate them safely only because its enum is wider. A model with nothing + * declared is unknown, and `text` is the honest floor — every routed model takes + * prompts. A model declaring `["audio"]` and nothing else is incompatible with a + * text|image client, and rewriting it to `["text"]` would advertise a capability + * it does not have. That input is reachable three ways: `ocx models add + * --modalities audio`, `/api/custom-models`, and provider discovery. + * + * So unknown falls back to text and incompatible returns null, which drops the + * row. Omitting a model costs the user a line in a picker; fabricating `text` + * costs them a model that fails at call time with no explanation. + * + * Deliberately NOT applied in `ExportModel` construction: the management and CLI + * boundaries carry catalog modalities verbatim on purpose, and stripping `audio` + * globally would destroy valid metadata before the destination is known. + */ +const CLIENT_INPUT_MODALITIES: Record<"pi" | "gajae", ReadonlySet> = { + pi: new Set(["text", "image"]), + gajae: new Set(["text", "image"]), +}; + +/** `null` means the model cannot be represented for this client — drop the row. */ +function inputModalitiesForClient( + client: "pi" | "gajae", + modalities: readonly string[] | undefined, +): string[] | null { + const declared = modalities ?? []; + if (declared.length === 0) return ["text"]; + const accepted = CLIENT_INPUT_MODALITIES[client]; + const kept: string[] = []; + for (const value of declared) { + if (accepted.has(value) && !kept.includes(value)) kept.push(value); + } + return kept.length > 0 ? kept : null; +} + /** * Label shared by every client: `" ()"`. The * provider suffix is what makes two same-named models from different upstreams @@ -646,25 +695,34 @@ export interface GajaeGeneratedConfig { * `reasoning` is a boolean in Pi while our catalog carries an effort list — mapping one * to the other would be a guess. * - * Pi's schema is UNVERIFIED against a real installation (001 §2); this contract is ours, - * not a claim about Pi's acceptance. + * Pi's input enum IS verified: its documented model configuration accepts only + * `text` and `image`, and a validation failure yields an EMPTY model config + * rather than dropping the offending entry — one bad value costs every routed + * model. The rest of this contract (omitting `cost` and `reasoning`) is still + * ours rather than a claim about Pi's acceptance. */ function buildPiClientConfig(ctx: ExportContext): PiGeneratedConfig { - const models: PiModelEntry[] = normalizeExportModels(ctx.models).map(model => { + const models: PiModelEntry[] = []; + for (const model of normalizeExportModels(ctx.models)) { + // Text is the one modality every routed model supports; anything richer must come + // from the catalog rather than an assumption — and must still be inside the enum + // Pi accepts, because one rejected value empties the whole config. + const input = inputModalitiesForClient("pi", model.inputModalities); + // An audio-only model has no honest representation here; claiming `text` + // would fail at call time instead, so the row is dropped. + if (input === null) continue; const entry: PiModelEntry = { id: model.namespaced, name: exportModelLabel(model), - // Text is the one modality every routed model supports; anything richer must come - // from the catalog rather than an assumption. - input: model.inputModalities && model.inputModalities.length > 0 ? [...model.inputModalities] : ["text"], + input, }; const context = authoritativeContextWindow(model.contextWindow); if (context !== undefined) { entry.contextWindow = context; entry.maxTokens = outputBudgetFor(context); } - return entry; - }); + models.push(entry); + } return { providers: { [OPENCODE_PROVIDER_ID]: { @@ -758,21 +816,24 @@ function buildKimiClientConfig(ctx: ExportContext): KimiGeneratedConfig { } function buildGajaeClientConfig(ctx: ExportContext): GajaeGeneratedConfig { - const models: GajaeModelEntry[] = normalizeExportModels(ctx.models).map(model => { + const models: GajaeModelEntry[] = []; + for (const model of normalizeExportModels(ctx.models)) { + // Gajae's enum is text|image and it rejects the whole file over one bad + // value, naming the offending index in the error. + const input = inputModalitiesForClient("gajae", model.inputModalities); + if (input === null) continue; const entry: GajaeModelEntry = { id: model.namespaced, name: exportModelLabel(model), - input: model.inputModalities && model.inputModalities.length > 0 - ? [...model.inputModalities] - : ["text"], + input, }; const context = authoritativeContextWindow(model.contextWindow); if (context !== undefined) { entry.contextWindow = context; entry.maxTokens = outputBudgetFor(context); } - return entry; - }); + models.push(entry); + } return { providers: { [OPENCODE_PROVIDER_ID]: { diff --git a/tests/client-export-modality-enum.test.ts b/tests/client-export-modality-enum.test.ts new file mode 100644 index 000000000..bcefd5039 --- /dev/null +++ b/tests/client-export-modality-enum.test.ts @@ -0,0 +1,132 @@ +import { describe, expect, test } from "bun:test"; +import { + buildClientConfig, + OPENCODE_PROVIDER_ID, + type ExportContext, + type ExportModel, + type GajaeGeneratedConfig, + type PiGeneratedConfig, +} from "../src/clients/config-export"; +import type { OcxConfig } from "../src/types"; + +/** + * Sibling of catalog-input-modality-enum.test.ts, whose incident this repeats + * at a different boundary. + * + * Our internal modality vocabulary is text|image|audio. Pi and Gajae accept only + * text|image, and BOTH reject the whole config over one out-of-enum value — + * Gajae falls back to its built-in list, Pi returns an empty model config. This + * actually happened: zenmux advertises audio on meta-muse-spark-1.1, we wrote it + * through verbatim, and gjc reported + * `/providers/opencodex/models/30/input/2: Invalid option: expected one of + * "text"|"image"` while showing none of the routed models. + * + * The per-entry tests in client-config-export.test.ts all passed while that file + * was broken, which is why the whole-catalog assertion at the bottom exists. + */ + +const CONFIG: OcxConfig = { + port: 10100, + hostname: "127.0.0.1", + defaultProvider: "mock", + providers: { mock: { adapter: "openai-chat", baseUrl: "http://127.0.0.1/v1" } }, +} as unknown as OcxConfig; + +function ctx(models: ExportModel[]): ExportContext { + return { baseUrl: "http://127.0.0.1:10100/v1", models, config: CONFIG }; +} + +function piModels(models: ExportModel[]) { + return (buildClientConfig("pi", ctx(models)) as PiGeneratedConfig) + .providers[OPENCODE_PROVIDER_ID].models; +} + +function gajaeModels(models: ExportModel[]) { + return (buildClientConfig("gajae", ctx(models)) as GajaeGeneratedConfig) + .providers[OPENCODE_PROVIDER_ID].models; +} + +/** The live failure, by its real id and real modality list. */ +const MIXED: ExportModel = { + namespaced: "zenmux/meta-muse-spark-1.1", + provider: "zenmux", + id: "meta-muse-spark-1.1", + contextWindow: 1_048_576, + inputModalities: ["text", "image", "audio"], +}; + +/** + * Reachable three ways: `ocx models add --modalities audio` (src/cli/models.ts), + * POST /api/custom-models (model-routes.ts ALLOWED_INPUT_MODALITIES), and + * provider discovery returning an audio-only list. + */ +const AUDIO_ONLY: ExportModel = { + namespaced: "p/audio-only", + provider: "p", + id: "audio-only", + inputModalities: ["audio"], +}; + +describe("exported modalities stay inside the enum each client accepts", () => { + test("audio is dropped from a mixed Gajae entry rather than written through", () => { + expect(gajaeModels([MIXED])[0]?.input).toEqual(["text", "image"]); + }); + + test("the same holds for Pi, whose exposure was latent only because its file was empty", () => { + expect(piModels([MIXED])[0]?.input).toEqual(["text", "image"]); + }); + + test("a model with NO acceptable modality is omitted, never rewritten to text", () => { + // Claiming text would advertise a capability the model does not have, and it + // would fail at call time with no explanation. Losing the row is the lesser + // cost — and unlike a fabricated modality, it is visible. + expect(gajaeModels([AUDIO_ONLY])).toEqual([]); + expect(piModels([AUDIO_ONLY])).toEqual([]); + }); + + test("an undeclared modality list still yields text — unknown is not incompatible", () => { + // Every routed model takes prompts, so text is the honest floor here. This + // branch is also what keeps the byte-exact goldens stable. + const bare: ExportModel = { namespaced: "p/bare", provider: "p", id: "bare" }; + expect(gajaeModels([bare])[0]?.input).toEqual(["text"]); + expect(piModels([bare])[0]?.input).toEqual(["text"]); + const empty: ExportModel = { ...bare, namespaced: "p/empty", id: "empty", inputModalities: [] }; + expect(gajaeModels([empty])[0]?.input).toEqual(["text"]); + }); + + test("an already-acceptable list survives untouched", () => { + const vision: ExportModel = { + namespaced: "p/vision", provider: "p", id: "vision", inputModalities: ["text", "image"], + }; + expect(gajaeModels([vision])[0]?.input).toEqual(["text", "image"]); + expect(piModels([vision])[0]?.input).toEqual(["text", "image"]); + }); + + test("order is preserved and duplicates collapse", () => { + const dup: ExportModel = { + namespaced: "p/dup", provider: "p", id: "dup", inputModalities: ["image", "text", "image"], + }; + expect(gajaeModels([dup])[0]?.input).toEqual(["image", "text"]); + }); + + test("no emitted entry in a whole catalog carries a value its client rejects", () => { + // The assertion that would have caught the live bug: every per-entry test + // above can pass while one model in a real catalog still poisons the file. + const catalog: ExportModel[] = [ + MIXED, + AUDIO_ONLY, + { namespaced: "p/bare", provider: "p", id: "bare" }, + { namespaced: "p/vision", provider: "p", id: "vision", inputModalities: ["text", "image"] }, + ]; + for (const models of [gajaeModels(catalog), piModels(catalog)]) { + expect(models.length).toBe(3); + for (const entry of models) { + for (const value of entry.input) { + expect(["text", "image"]).toContain(value); + } + } + // And the incompatible one is gone rather than silently retyped. + expect(models.map(m => m.id)).not.toContain("p/audio-only"); + } + }); +}); From 6c9ef8e4d4ad0e803f3137fe66a416b0a3711910 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Tue, 4 Aug 2026 00:49:09 +0900 Subject: [PATCH 012/163] =?UTF-8?q?docs(integrations):=20WP2=20evidence=20?= =?UTF-8?q?=E2=80=94=20A/B=20against=20gajae's=20own=20schema,=20not=20a?= =?UTF-8?q?=20unit=20test?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The client ships its schema as source, so it can be imported and run against our real emitted bytes. Feeding it an audio-bearing entry reproduces the user's exact error text from models-config-schema.ts:141, which makes it a real oracle rather than an approximation. A/B on the emitted file: HEAD~1 fails at models[31].input[2], HEAD passes, and the model count is unchanged at 34 — meta-muse-spark-1.1 is still exported, now as [text, image]. The fix removes a rejected value, not a model. Also records a method failure worth keeping: the first A/B reported BEFORE -> PASS because the git stash used to revert the file conflicted and never removed the fix, so the 'before' run measured the fixed code. Re-done by extracting HEAD~1 directly. A verification that silently measures the wrong build and reports green is the exact class of failure this unit exists to stop. --- .../011_wp2_evidence.md | 75 +++++++++++++++++++ 1 file changed, 75 insertions(+) create mode 100644 devlog/_plan/260803_codex_desktop_toggle/011_wp2_evidence.md diff --git a/devlog/_plan/260803_codex_desktop_toggle/011_wp2_evidence.md b/devlog/_plan/260803_codex_desktop_toggle/011_wp2_evidence.md new file mode 100644 index 000000000..4c1b6fd50 --- /dev/null +++ b/devlog/_plan/260803_codex_desktop_toggle/011_wp2_evidence.md @@ -0,0 +1,75 @@ +# WP2 — verification evidence + +The criterion in `010` was never "a unit test passes". 91 export tests were +green while the real file was broken. So the close-out is an A/B against the +client's own schema, on the real emitted bytes. + +## The decisive check + +`@gajae-code/coding-agent@0.7.11` ships its schema as source, so it can be +imported and run against our output directly rather than approximated. + +``` +ModelsConfigSchema.safeParse({... input: ["text","image","audio"] ...}) +→ FAIL {"path":["providers","opencodex","models",0,"input",2], + "message":"Invalid option: expected one of \"text\"|\"image\""} +``` + +That is the user's reported error, reproduced from the schema itself +(`models-config-schema.ts:141`, `input: z.array(z.enum(["text","image"]))`). +The bug and the oracle now agree, which is what makes the A/B meaningful. + +## A/B on the real emitted file + +Both runs: `bun run src/cli/index.ts export --client gajae --out `, output +parsed as YAML and handed to the real schema. + +| | Models | audio-bearing | Schema | +|---|---|---|---| +| `HEAD~1` (pre-fix) | 34 | `zenmux/meta-muse-spark-1.1` | **FAIL** at `models[31].input[2]` | +| `HEAD` (fixed) | 34 | none | **PASS** | + +Two details worth keeping: + +- **The model count is unchanged.** `meta-muse-spark-1.1` is still exported, now + as `[text, image]`. The fix removes a rejected value, not a model — the + omission branch only fires for a model with nothing acceptable left, and this + catalog has none. +- The pre-fix index is 31 here versus 30 in the user's report. The catalog moved + between then and now; the offending entry is the same one. + +## Structural check, not grep + +Audit round 3 rejected `grep -c audio` as the verification: the string also +occurs in model ids, display names and other providers' preserved blocks, so it +can fail while our output is correct. The check walks +`providers.opencodex.models[*].input` and asserts every value is in +`text|image`. Result: `out-of-enum input values: NONE` across 34 models. + +## Gates + +| Gate | Result | +|---|---| +| `bun x tsc --noEmit` | clean | +| `bun run test` | **7632 pass, 0 fail**, 7 skip, 507 files | +| `bun run privacy:scan` | passed | +| `tests/client-export-modality-enum.test.ts` | 7 pass | +| existing export/catalog suites | 91 pass | + +## The test earns its place + +Reverting `config-export.ts` alone makes **5 of 7** new cases fail, including the +whole-catalog assertion. A test that passes with and without the fix would have +been decoration. + +## Note on method + +A first attempt at this A/B reported "BEFORE FIX → PASS", which was wrong: the +`git stash push` used to revert the file hit a conflict and the fix was never +actually removed, so the "before" run measured the fixed code. Re-done by +extracting `HEAD~1:src/clients/config-export.ts` directly. The worktree was +restored afterwards and matches HEAD; the six pre-existing stashes are intact. + +Recording this because the failure mode — a verification that silently measures +the wrong build and reports green — is exactly the class of thing this unit +exists to stop. From a0f05af5c7a318db95457570980bc59d40d4442b Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Tue, 4 Aug 2026 01:04:49 +0900 Subject: [PATCH 013/163] =?UTF-8?q?docs(integrations):=20fold=20the=20WP3?= =?UTF-8?q?=20audit=20=E2=80=94=20a=20credential=20row=20must=20not=20say?= =?UTF-8?q?=20'Not=20applied'?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The blocking finding is a contradiction I wrote myself: the doc argues at length that issuing a credential is not applying an integration, removes applied from the model, and drops keys from the Applied total — then renders IntegrationStateBadge, whose current/absent labels are exactly 'Applied' and 'Not applied' in all six locales. The live zero-key card says 미적용 today. The row now carries no badge; the detail line is the state, and an unsettled read gets its own key rather than borrowing the badge's 'Unknown'. Also: the summary labels move with their scope, because Detected silently goes 5 to 4 on this machine the moment the phase ships and bare 'Detected' never said it counted clients. The pinned test value is updated, not deleted. And two test-quality fixes: failExtraSources fails five sources at once so it cannot show an API-key failure alone leaves client totals alone — the fixture gets an independent keys control and a settled native response — and the keyboard assertion queries tabbable descendants with Enter/Space activation instead of counting buttons, which would pass through the exact regression it is meant to catch. --- .../020_api_keys_row.md | 92 +++++++++++++++++-- 1 file changed, 83 insertions(+), 9 deletions(-) diff --git a/devlog/_plan/260803_codex_desktop_toggle/020_api_keys_row.md b/devlog/_plan/260803_codex_desktop_toggle/020_api_keys_row.md index 11c85c442..58f792f33 100644 --- a/devlog/_plan/260803_codex_desktop_toggle/020_api_keys_row.md +++ b/devlog/_plan/260803_codex_desktop_toggle/020_api_keys_row.md @@ -163,7 +163,14 @@ all.” Below preserves aggregate → individual → client catalog hierarchy. +

{t(row.labelKey)}

+ {detail &&

{detail}

} + -+ ++ {/* ++ NOT IntegrationStateBadge. It renders `current` as "Applied" and ++ `absent` as "Not applied" in all six locales (LABEL_KEYS in ++ IntegrationStateBadge.tsx:12), which is exactly the claim this phase ++ argues is false — the live zero-key card says `미적용` today. The ++ detail line above already carries the honest state (checking, none ++ issued, N issued), so a badge here could only re-say it wrongly. ++ */} + + {/* + Below the aggregate, above the client catalog. The summary is the + page-level total; this is one credential surface with one action. + Putting it above the summary would promote one surface over the + aggregate, and merging it into the strip would make "Manage keys" read + as a bulk control beside "Disable all". + */} + +

{t("integrations.onboarding")}

{statesResource.state.kind === "failed-cold" && ( {t("integrations.error.load")} @@ -568,3 +595,31 @@ export default function IntegrationsOverview({ ); } +/** + * Credentials are one explicit action, not a clickable client card. + * + * No `IntegrationStateBadge`: it renders `current` as "Applied" and `absent` as + * "Not applied" in all six locales, which is the one thing a credential row + * must not claim — issuing a key does not apply an integration. The detail line + * IS the state, and `data-key-state` is what keeps the four states testable and + * stylable without borrowing client vocabulary. + * + * The card overlay is also deliberately absent. It exists because a card holds + * a switch as well as a title; this row has no nested-control problem to solve, + * so one plain button is the whole keyboard path. + */ +function ApiKeysRow({ row }: { row: ApiKeysOverviewRow }) { + const t = useT(); + const detail = row.detailKey ? t(row.detailKey, row.detailVars ?? undefined) : null; + return ( +
+
+

{t(row.labelKey)}

+ {detail &&

{detail}

} +
+ +
+ ); +} diff --git a/gui/src/pages/integrations/integration-api.ts b/gui/src/pages/integrations/integration-api.ts index 3fdbf592b..927f1417a 100644 --- a/gui/src/pages/integrations/integration-api.ts +++ b/gui/src/pages/integrations/integration-api.ts @@ -280,9 +280,26 @@ export async function loadCodexRoutingStatus(apiBase: string, signal?: AbortSign }; } -export async function loadApiKeyCount(apiBase: string, signal?: AbortSignal): Promise { - const body = await readOptional<{ keys?: unknown }>(fetch(`${apiBase}/api/keys`, { signal })); - if (!body || !Array.isArray(body.keys)) return null; +/** + * Throws on a failed or malformed read rather than returning null. + * + * `readOptional` is right for surfaces that treat "no answer" and "empty" the + * same. This one cannot: the overview says "Checking…" while a read is in + * flight and "Key status unavailable" once it has settled badly, and a + * successfully-returned null collapses both into `ready-empty` with no polling + * to ever correct it — so the row would claim the user has no keys because a + * request failed. Throwing is what produces `failed-cold` / `failed-with-stale`, + * which is the signal the row reads. Aborts never reach a state: an aborted + * generation is discarded before either data or failure is published. + */ +export async function loadApiKeyCount(apiBase: string, signal?: AbortSignal): Promise { + const response = await fetch(`${apiBase}/api/keys`, { signal }); + // These two strings are diagnostics for the failure path, never rendered: + // the row shows the localized `integrations.detail.keyUnavailable` instead. + // eslint-disable-next-line local-i18n/no-hardcoded-ui-strings -- rejection reason, not UI text + if (!response.ok) throw new Error(`/api/keys responded ${response.status}`); + const body = await readJsonIfOk<{ keys?: unknown }>(response); + if (!body || !Array.isArray(body.keys)) throw new Error("/api/keys returned an unexpected body"); return body.keys.length; } diff --git a/gui/src/pages/integrations/overview-clients.ts b/gui/src/pages/integrations/overview-clients.ts index e4eb0a004..f0d190e6e 100644 --- a/gui/src/pages/integrations/overview-clients.ts +++ b/gui/src/pages/integrations/overview-clients.ts @@ -23,12 +23,39 @@ import type { NativeStatus } from "./native-api"; export type OverviewClientId = | "codex" - | "keys" | "claude" | "claudeDesktop" | "grok" | FileIntegrationClientId; +/** How far the `/api/keys` read has got, since the count alone cannot say. */ +export type ApiKeyReadPhase = "checking" | "unavailable" | "settled"; + +/** + * The credential row, deliberately NOT an `OverviewRow`. + * + * API keys cannot be installed, toggled, or drift from a config file, so they + * have no `installed`, `applied`, `toggle`, or `status`. Re-adding one would + * recreate the semantic leak this shape removes. + * + * `state` uses credential vocabulary rather than the client + * `unknown|absent|current` triple, because those words carry "applied" — + * `IntegrationStateBadge` renders them as "Applied" and "Not applied" in all + * six locales, which is exactly the claim a credential row must never make. + */ +export interface ApiKeysOverviewRow { + hash: "integrations/keys"; + labelKey: TKey; + state: "checking" | "unavailable" | "none-issued" | "issued"; + detailKey: TKey | null; + detailVars: Record | null; +} + +export interface OverviewRows { + keysRow: ApiKeysOverviewRow; + rows: OverviewRow[]; +} + export interface OverviewRow { id: OverviewClientId; /** Tab this card opens. Claude Desktop opens Claude's nested route. */ @@ -85,6 +112,11 @@ export interface OverviewSources { clientsSettled: boolean; codex: CodexRoutingPayload | null; keyCount: number | null; + /** + * Read phase for `keyCount`. Separate because a settled zero and a failed + * read are both null-adjacent facts that must not render the same way. + */ + keyPhase: ApiKeyReadPhase; claude: ClaudeCodePayload | null; claudeDesktop: ClaudeDesktopPayload | null; grok: GrokPayload | null; @@ -151,25 +183,26 @@ function codexRow(payload: CodexRoutingPayload | null): OverviewRow { } /** API keys are issued or not; there is no config file to drift. */ -function keysRow(count: number | null): OverviewRow { +function keysRow(phase: ApiKeyReadPhase, count: number | null): ApiKeysOverviewRow { const base = { - id: "keys" as const, - hash: "integrations/keys", + hash: "integrations/keys" as const, labelKey: "integrations.tab.keys" as TKey, - toggle: null, - toggleBlocked: null, - togglePath: null, - status: null, - detail: null, }; - if (count === null) { - return { ...base, state: "unknown", installed: false, applied: false, detailKey: null, detailVars: null }; + // Every branch names a detail key. The detail line is the ONLY state + // expression — there is no badge — so a null one would render a row with no + // state at all. + if (phase === "checking") { + return { ...base, state: "checking", detailKey: "integrations.detail.keyChecking", detailVars: null }; + } + // `count === null` is defensive: a failed read is already `unavailable`, and + // claiming "no keys issued" because a request failed is a statement about the + // account of the user that we cannot support. + if (phase === "unavailable" || count === null) { + return { ...base, state: "unavailable", detailKey: "integrations.detail.keyUnavailable", detailVars: null }; } return { ...base, - state: count > 0 ? "current" : "absent", - installed: true, - applied: count > 0, + state: count > 0 ? "issued" : "none-issued", detailKey: count > 0 ? "integrations.detail.keyCount" : "integrations.detail.keyNone", detailVars: count > 0 ? { count: String(count) } : null, }; @@ -354,14 +387,13 @@ function fileRow(status: IntegrationStatus): OverviewRow { * clients, then the file clients in their existing order. It matches the tab * strip above the grid, so the eye moves the same way in both. */ -export function buildOverviewRows(sources: OverviewSources): OverviewRow[] { +export function buildOverviewRows(sources: OverviewSources): OverviewRows { const nativeClaude = sources.native?.find(status => status.clientId === "claude"); const nativeGrok = sources.native?.find(status => status.clientId === "grok"); // One lookup table, not a find per client (react-doctor js-index-maps). const statusByClient = new Map(sources.clients.map(status => [status.clientId, status])); const rows: OverviewRow[] = [ codexRow(sources.codex), - keysRow(sources.keyCount), claudeRow(sources.claude, nativeClaude, sources.nativeSettled), claudeDesktopRow(sources.claudeDesktop), grokRow(sources.grok, nativeGrok, sources.nativeSettled), @@ -393,7 +425,7 @@ export function buildOverviewRows(sources: OverviewSources): OverviewRow[] { }); } } - return rows; + return { keysRow: keysRow(sources.keyPhase, sources.keyCount), rows }; } export interface OverviewCounts { diff --git a/gui/src/styles-integrations.css b/gui/src/styles-integrations.css index 6ef9bf1b9..50ecd3e2d 100644 --- a/gui/src/styles-integrations.css +++ b/gui/src/styles-integrations.css @@ -18,6 +18,15 @@ down while a one-line one did not. Reserving the detail line's height keeps the switches on one baseline across the row. */ .integration-cards { display: grid; grid-template-columns: repeat(auto-fill, minmax(260px, 1fr)); gap: 12px; list-style: none; padding: 0; margin: 14px 0; } +/* + One full-width row, not a wide card: no grid cell, no hover border, no + stretched title. `flex-wrap` is what keeps long German/Russian action copy + moving below the label at 320-390px instead of clipping. +*/ +.integration-api-keys-row { display: flex; flex-wrap: wrap; align-items: center; gap: 10px; padding: 12px 14px; margin: 14px 0 0; border: 1px solid var(--border); border-radius: var(--radius); background: var(--raised); } +.integration-api-keys-copy { display: flex; flex-direction: column; gap: 2px; flex: 1 1 220px; min-width: 0; } +.integration-api-keys-copy h4 { margin: 0; } +.integration-api-keys-row .integration-meta { margin: 0; min-height: 0; } .integration-card { position: relative; display: flex; flex-direction: column; gap: 8px; padding: 14px; border: 1px solid var(--border); border-radius: var(--radius); background: var(--raised); } .integration-card-head { display: flex; align-items: center; justify-content: space-between; gap: 8px; } .integration-card-head h4 { margin: 0; } diff --git a/gui/tests/api-key-count-loader.test.ts b/gui/tests/api-key-count-loader.test.ts new file mode 100644 index 000000000..9c5507e7f --- /dev/null +++ b/gui/tests/api-key-count-loader.test.ts @@ -0,0 +1,57 @@ +import { afterEach, describe, expect, test } from "bun:test"; +import { loadApiKeyCount } from "../src/pages/integrations/integration-api"; + +/** + * `loadApiKeyCount` used to swallow every failure into `null` via + * `readOptional`. The overview then saw a SUCCESSFUL empty result, with no + * polling to correct it, and rendered "No keys issued" — a claim about the + * account of the user that a failed request cannot support. + * + * It now throws, which is what produces `failed-cold` / `failed-with-stale` and + * lets the row say "Key status unavailable" instead. These cases pin that + * contract, because the mounted overview test only exercises one of the five + * ways it can now fail. + */ + +const realFetch = globalThis.fetch; +afterEach(() => { globalThis.fetch = realFetch; }); + +function respondWith(body: string, init: ResponseInit = {}) { + globalThis.fetch = (async () => new Response(body, { + status: 200, + headers: { "content-type": "application/json" }, + ...init, + })) as typeof fetch; +} + +describe("loadApiKeyCount distinguishes empty from failed", () => { + test("an empty key list resolves 0 — zero is data, not a failure", async () => { + respondWith(JSON.stringify({ keys: [] })); + expect(await loadApiKeyCount("http://x")).toBe(0); + }); + + test("a populated list resolves its length", async () => { + respondWith(JSON.stringify({ keys: [{ key: "a" }, { key: "b" }] })); + expect(await loadApiKeyCount("http://x")).toBe(2); + }); + + test("a non-ok status rejects instead of reading as no keys", async () => { + respondWith("{}", { status: 500 }); + expect(loadApiKeyCount("http://x")).rejects.toThrow(/500/); + }); + + test("a malformed body rejects", async () => { + respondWith("not json at all"); + expect(loadApiKeyCount("http://x")).rejects.toThrow(/unexpected body/); + }); + + test("a non-array `keys` rejects — the case readOptional turned into 'no keys issued'", async () => { + respondWith(JSON.stringify({ keys: "nope" })); + expect(loadApiKeyCount("http://x")).rejects.toThrow(/unexpected body/); + }); + + test("a network rejection propagates rather than being swallowed", async () => { + globalThis.fetch = (async () => { throw new Error("offline"); }) as typeof fetch; + expect(loadApiKeyCount("http://x")).rejects.toThrow(/offline/); + }); +}); diff --git a/gui/tests/integrations-overview-rows.test.ts b/gui/tests/integrations-overview-rows.test.ts index 9f361373a..4cd926dc9 100644 --- a/gui/tests/integrations-overview-rows.test.ts +++ b/gui/tests/integrations-overview-rows.test.ts @@ -31,6 +31,7 @@ function sources(overrides: Partial = {}): OverviewSources { clientsSettled: true, codex: null, keyCount: null, + keyPhase: "settled", claude: null, claudeDesktop: null, grok: null, @@ -38,21 +39,22 @@ function sources(overrides: Partial = {}): OverviewSources { }; } -function rowById(rows: ReturnType, id: string) { - const found = rows.find(row => row.id === id); +function rowById(built: ReturnType, id: string) { + const found = built.rows.find(row => row.id === id); if (!found) throw new Error(`no row for ${id}`); return found; } test("a null source is unknown, never absent, and is counted in neither total", () => { - const rows = buildOverviewRows(sources()); - for (const id of ["codex", "keys", "claude", "claudeDesktop", "grok"]) { - expect(rowById(rows, id).state).toBe("unknown"); + const built = buildOverviewRows(sources()); + for (const id of ["codex", "claude", "claudeDesktop", "grok"]) { + expect(rowById(built, id).state).toBe("unknown"); } - const counts = countOverviewRows(rows); + const counts = countOverviewRows(built.rows); expect(counts.detected).toBe(0); expect(counts.applied).toBe(0); - expect(counts.unknown).toBe(5); + // Four, not five: keys is a credential surface and never a client row. + expect(counts.unknown).toBe(4); }); test("Codex reads routingInjected, not status", () => { @@ -119,7 +121,7 @@ test("file clients keep their existing badge and applied semantics", () => { expect(rowById(rows, "kimi").installed).toBe(false); expect(rowById(rows, "gajae").state).toBe("unsafe"); - const counts = countOverviewRows(rows); + const counts = countOverviewRows(rows.rows); expect(counts.detected).toBe(5); expect(counts.applied).toBe(2); expect(counts.stale).toBe(1); @@ -134,27 +136,28 @@ test("every client counts toward the summary, not just the file six", () => { claudeDesktop: { applied: true, stale: true, activeProfile: true }, grok: { present: true, models: [{}, {}] }, })); - const counts = countOverviewRows(rows); - // codex + keys + claude + desktop + grok + opencode - expect(counts.applied).toBe(6); + const counts = countOverviewRows(rows.rows); + // codex + claude + desktop + grok + opencode. Keys are deliberately absent: + // an issued credential is not an applied client. + expect(counts.applied).toBe(5); expect(counts.stale).toBe(1); expect(counts.unknown).toBe(0); }); test("an unsettled file list renders unknown rows instead of dropping them", () => { - const rows = buildOverviewRows(sources({ clients: [], clientsSettled: false })); - expect(rows).toHaveLength(11); - expect(rowById(rows, "kimi").state).toBe("unknown"); + const built = buildOverviewRows(sources({ clients: [], clientsSettled: false })); + expect(built.rows).toHaveLength(10); + expect(rowById(built, "kimi").state).toBe("unknown"); // Once settled, a client the server omitted is genuinely gone. const settled = buildOverviewRows(sources({ clients: [], clientsSettled: true })); - expect(settled).toHaveLength(5); + expect(settled.rows).toHaveLength(4); + expect(settled.rows.some(row => row.hash === "integrations/keys")).toBe(false); }); test("each row points at its own tab", () => { const rows = buildOverviewRows(sources({ clientsSettled: false })); expect(rowById(rows, "codex").hash).toBe("integrations/codex"); - expect(rowById(rows, "keys").hash).toBe("integrations/keys"); expect(rowById(rows, "claude").hash).toBe("integrations/claude"); expect(rowById(rows, "claudeDesktop").hash).toBe("integrations/claude/desktop"); expect(rowById(rows, "grok").hash).toBe("integrations/grok"); diff --git a/gui/tests/integrations-surfaces.test.tsx b/gui/tests/integrations-surfaces.test.tsx index 6d203f5e9..3f84f323c 100644 --- a/gui/tests/integrations-surfaces.test.tsx +++ b/gui/tests/integrations-surfaces.test.tsx @@ -566,7 +566,11 @@ test("every reachable client gets a card, not just the file six", async () => { const clientIds = Array.from(container.querySelectorAll(".integration-card")) .map(card => (card as unknown as HTMLElement).getAttribute("data-client")); expect(clientIds).toContain("codex"); - expect(clientIds).toContain("keys"); + // Keys deliberately absent: a credential is not a client card. It renders as + // its own row above the grid instead. + expect(clientIds).not.toContain("keys"); + expect(container.querySelector(".integration-cards [data-client='keys']")).toBeNull(); + expect(container.querySelector(".integration-api-keys-row")).not.toBeNull(); expect(clientIds).toContain("claude"); expect(clientIds).toContain("claudeDesktop"); expect(clientIds).toContain("grok"); @@ -594,12 +598,16 @@ test("a source that cannot be read is unknown, never 'not applied'", async () => failExtraSources = true; await mountOverview(); - for (const id of ["codex", "keys", "claude", "claudeDesktop", "grok"]) { + for (const id of ["codex", "claude", "claudeDesktop", "grok"]) { const badge = container.querySelector( `.integration-card[data-client='${id}'] .badge`, ) as unknown as HTMLElement | null; expect(badge?.getAttribute("data-integration-state")).toBe("unknown"); } + // The keys row says the same thing in credential words: a failed read is + // "unavailable", never "no keys issued". + const keysRow = container.querySelector(".integration-api-keys-row") as unknown as HTMLElement | null; + expect(keysRow?.getAttribute("data-key-state")).toBe("unavailable"); // The file client still reports its real state. const hermes = container.querySelector( ".integration-card[data-client='hermes'] .badge", diff --git a/gui/tests/overview-state-merge.test.ts b/gui/tests/overview-state-merge.test.ts index 5eb15da9e..e1697fa42 100644 --- a/gui/tests/overview-state-merge.test.ts +++ b/gui/tests/overview-state-merge.test.ts @@ -20,6 +20,7 @@ function sources(overrides: Partial = {}): OverviewSources { clientsSettled: true, codex: null, keyCount: null, + keyPhase: "settled", claude: { enabled: false, authMode: "subscription" }, claudeDesktop: null, grok: { present: false, models: [] }, @@ -30,7 +31,7 @@ function sources(overrides: Partial = {}): OverviewSources { } function row(overrides: Partial, id: "claude" | "grok") { - const found = buildOverviewRows(sources(overrides)).find(candidate => candidate.id === id); + const found = buildOverviewRows(sources(overrides)).rows.find(candidate => candidate.id === id); if (!found) throw new Error(`missing ${id} row`); return found; } From feaac9c8d324332a10fb63e3a232c5ac80f674b2 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Tue, 4 Aug 2026 01:35:48 +0900 Subject: [PATCH 019/163] =?UTF-8?q?docs(integrations):=20WP3=20evidence=20?= =?UTF-8?q?=E2=80=94=20rendered,=20observed,=20and=20isolated=20from=20the?= =?UTF-8?q?=20live=20proxy?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The user's 10100 proxy runs the installed build without this change, so the row was observed from a second proxy started off the dev tree on 10399 under a mktemp OPENCODEX_HOME, then stopped by PID. 10100 answered 200 throughout. A dev-tree start against the real home would have rewritten the user's client configs mid-session. At 1440px: one full-width row between the summary and the grid, no badge, and the grid starting at Codex CLI with no keys card. At 390px it keeps its shape without clipping. agbrowse resolves the control as a real button in the accessibility tree — the Enter/Space assertion happy-dom could not make. Also records that lint rejected the two throw strings as untranslated UI, and why exactly one scoped disable is correct: they are rejection reasons the user never sees, and a blanket disable would hide the next real one. --- .../021_wp3_evidence.md | 69 +++++++++++++++++++ 1 file changed, 69 insertions(+) create mode 100644 devlog/_plan/260803_codex_desktop_toggle/021_wp3_evidence.md diff --git a/devlog/_plan/260803_codex_desktop_toggle/021_wp3_evidence.md b/devlog/_plan/260803_codex_desktop_toggle/021_wp3_evidence.md new file mode 100644 index 000000000..1d35fb9ee --- /dev/null +++ b/devlog/_plan/260803_codex_desktop_toggle/021_wp3_evidence.md @@ -0,0 +1,69 @@ +# WP3 — verification evidence + +C-RENDER-GROUNDING-01 applies: a layout change is not verified by a green suite +or a produced-but-unread screenshot. The row was rendered, observed, and +inspected. + +## How it was observed without touching the live proxy + +The user's proxy on 10100 runs the installed v2.10.0 build, which does not carry +this change. So a second proxy was started from the dev tree on port 10399 under +an `OPENCODEX_HOME` from `mktemp -d`, observed, and then stopped by PID. The +10100 proxy answered 200 before and after. + +That isolation is not politeness. A dev-tree start against the real home would +have rewritten the user's client configs mid-session. + +## What the render showed + +At 1440 px: + +- API keys render as **one full-width row** between the summary strip and the + card grid — the requested `위쪽에 한 라인`. +- The row carries the title, `No keys issued`, and a `Manage keys` button. **No + badge**, so nothing on it says "Not applied" about a credential. +- The card grid begins with **Codex CLI**. There is no keys card anywhere in it. +- The summary reads `Clients detected 9` / `Configured clients 3`, so the labels + now state the scope their numbers actually measure. + +At 390 px the row keeps its shape: title and detail stack on the left, the +action stays on one line, nothing clips. + +`agbrowse snapshot --interactive` resolves the control as +`e75 button "Manage keys"` — a real button in the accessibility tree, which is +what earns platform Enter/Space behavior. That is the assertion happy-dom cannot +make, which is why the audit moved it here. + +![keys row at 1440px](/Users/jun/.browser-agent/screenshots/screenshot_1785774782191.png) + +## Gates + +| Gate | Result | +|---|---| +| root `bun x tsc --noEmit` | clean | +| `gui` typecheck | clean | +| `gui bun test tests` | **564 pass, 0 fail**, 107 files | +| `bun run lint:gui` | clean, 0 errors 0 warnings | +| `bun run lint:i18n` | clean | +| `gui bun run build` | succeeded | +| `tests/api-key-count-loader.test.ts` | 6 pass | + +The lint run earned its place: it rejected the two `throw new Error` strings as +untranslated UI text. They are rejection reasons that never reach a user — the +row renders `integrations.detail.keyUnavailable` instead — so one carries a +scoped disable with that reason, and the second disable was removed once eslint +reported it as unused. A blanket disable would have hidden the next real one. + +## Tests that had to change, and why that is the point + +Four existing tests failed after the change, each asserting the contract this +phase removes: `keys` as a client row, `unknown` as its state, 5 settled rows, +11 unsettled, and an applied total of 6. They were updated rather than deleted, +and two now assert the inverse — that no `[data-client="keys"]` exists inside +`.integration-cards`, and that a failed key read renders `data-key-state` +`unavailable` rather than "no keys issued". + +The loader test is the one that would have caught the original defect: a +`{ keys: "nope" }` body used to become "No keys issued" through `readOptional`, +which is a claim about the account of the user that a malformed response cannot +support. From 75d33a138cac3764e758c306fd0a13b99341a0e8 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Tue, 4 Aug 2026 01:59:54 +0900 Subject: [PATCH 020/163] docs(integrations): re-scope WP4 to Codex alone MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The ten-key version failed two audit rounds, and 006 diagnosed the coupling itself as the defect: ten clients with different ownership rules, teardown callers and migration histories in one schema change, so every repair widened the blast radius. The two phases that shipped clean each changed one thing at one boundary. So WP4 is now Codex only: one clientIntegrations.codex key in a shape WP6 and WP7 can extend, field-scoped persistence through mutatePersistedConfig, gates on Codex's own automatic re-apply paths, a write flight keyed by CODEX_HOME with a fresh intent read before each irreversible write, and startup convergence that re-runs the remover behind assertNativeTeardownOwned so an ocx start from another OPENCODEX_HOME cannot strip a foreign service's state. Explicitly removed rather than quietly dropped: Claude Code's ingress gates (round 1 #1 — this unit does not touch them at all), the six file clients (deferred as FOLLOWUP-FILECLIENT-01), Grok and Desktop, and the four-client native contract that belongs to WP5. The auth-mode sentinel finding is answered rather than handled: a Codex-only flag never creates or modifies the claudeCode block, so the migration path is unreachable, and a regression test pins that. --- .../030_desired_state.md | 1040 ++++++----------- 1 file changed, 377 insertions(+), 663 deletions(-) diff --git a/devlog/_plan/260803_codex_desktop_toggle/030_desired_state.md b/devlog/_plan/260803_codex_desktop_toggle/030_desired_state.md index a568b0f98..066585dca 100644 --- a/devlog/_plan/260803_codex_desktop_toggle/030_desired_state.md +++ b/devlog/_plan/260803_codex_desktop_toggle/030_desired_state.md @@ -1,69 +1,93 @@ -# WP3 — durable desired state, admission, and convergence - -Research: `003_durable_desired_state.md`. Audit disposition: -`005_audit_synthesis.md`. Read both first; this doc is the amended diff. - -The shipped Grok switch removes its fence but records no intent. The next -`ocx start` calls `syncGrokConfig` and writes the fence back -(`src/cli/index.ts:334-350`, `src/grok/sync.ts:29-65`). The failed first draft -added a boolean but omitted the writers, ordering, and restart work that make the -boolean govern the system. It also proposed removing Claude's shipped ingress -kill switch. That decision is reversed here, plainly: the audit was right. An -upgrading user with legacy `claudeCode.enabled=false` must not silently regain -Claude ingress. - -WP3 therefore owns more than a flag. It owns the shared desired-state contract, -field-scoped persistence, the per-client operation boundary, the six file-client -writer, startup reconciliation, and the response grammar WP5 and WP6 consume. +# WP4 — Codex desired state and ownership-safe convergence + +Research: `003_durable_desired_state.md`. Re-scope authority: +`006_audit_synthesis_r2.md`. This document replaces the failed ten-client +version of WP4. + +The failure to prevent is concrete: a user persists Codex OFF, the process dies +before native restore finishes, and the next `ocx start` writes OpenCodex routing +back because `syncModelsToCodex` is unconditional (`src/cli/index.ts:318-320`, +`src/codex/sync.ts:49-110`). The first repair made that worse: startup +reconciliation called the native remover without checking service ownership, so +a start from a different `OPENCODEX_HOME` could strip Codex state used by the +installed service (`006_audit_synthesis_r2.md`, round 2 #2). + +That incident decides the scope. The two phases that shipped cleanly each changed +one thing at one boundary (`010_modality_boundary.md`, `020_api_keys_row.md`). WP4 +therefore changes desired state for **Codex only**. It does not establish a +cross-client contract. + +## What exists, and what WP4 adds + +Already present: + +- `mutatePersistedConfig` clones and rebases a callback under the config mutation + lock, then returns `committed | unchanged | unavailable`; callers do not need a + second persistence mechanism (`src/config.ts:1825-1906`). +- `syncModelsToCodex` owns the normal catalog-plus-injection path + (`src/codex/sync.ts:49-129`), while provider/model/combo routes bypass it through + `refreshCodexCatalogBestEffort` (`src/server/management-api.ts:105-112`). +- `restoreNativeCodex` is the idempotent Codex remover + (`src/codex/inject.ts:759-795`), and `assertNativeTeardownOwned` is the shipped + foreign-home preflight (`src/integrations/native/ownership-preflight.ts:19-35`). +- crash-journal reconciliation already repairs an abandoned injection + (`src/codex/journal.ts:148-162`). + +WP4 adds one persisted Codex flag, one Codex write coordinator, last-moment +persisted-state checks for automatic apply writes, and OFF reconciliation at start +and ensure. WP5 adds the management route and GUI switch that call the writer; +WP4 does not define their response schema. ## IN / OUT -| Path | Change | Why it is in WP3 | +| Path | Change | Why it is in WP4 | |---|---|---| -| `src/types.ts` | MODIFY | Defines the complete ten-client desired-state vocabulary and `OcxConfig.clientIntegrations`. | -| `src/config.ts` | MODIFY | Parses the map, resolves legacy Claude intent, and mutates one selected key through the existing `mutatePersistedConfig` primitive. | -| `src/integrations/desired-state.ts` | NEW | Owns per-client single-flight, last-moment persisted-state checks, and reconciliation result types. | -| `src/integrations/reconcile.ts` | NEW | Converges desired OFF to observed absent for the six file clients and the native handlers registered by WP3/WP5/WP6. | -| `src/integrations/state.ts` | MODIFY | Adds required `desiredEnabled` to the six-client status helper. | -| `src/integrations/writer.ts` | MODIFY | Re-reads persisted intent immediately before apply/disable/restore commits. | -| `src/server/management/integration-routes.ts` | MODIFY | Persists the six-client desired state before applying/removing files; GET/status also reconciles stale OFF. | -| `src/codex/sync.ts` | MODIFY | Stops Codex catalog/injection while OFF, joins the per-client flight, and re-checks before each artifact write. | -| `src/grok/sync.ts` | MODIFY | Stops Grok fetch/write while OFF, joins the same Grok flight as every other caller, and re-checks before injection. | -| `src/server/management-api.ts` | MODIFY | Gates the direct Codex catalog refresher and moves Claude agent sync to the compatibility helper. | -| `src/server/management/native-integration-routes.ts` | MODIFY | Owns the complete four-client native contract, field-scoped Claude/Grok persistence, status helpers, and native reconciliation entry points. | -| `src/server/management/agent-settings-routes.ts` | MODIFY | Routes Grok/Desktop background writes through the shared flight and mutation contract. | -| `src/cli/index.ts` | MODIFY | Runs OFF reconciliation on start and both ensure branches before any automatic apply. | -| `src/cli/opencode.ts` | MODIFY | Refuses the inline provider writer after a real OpenCode OFF and re-checks before spawn. | -| `src/cli/claude.ts` | MODIFY | Reads Claude Code desired state through the compatibility helper. | -| `src/claude/agents-inject.ts` | MODIFY | Reads Claude Code desired state through the compatibility helper. | -| `src/server/system-env.ts` | MODIFY | Reads Claude Code desired state through the compatibility helper. | -| `src/server/claude-messages.ts` | MODIFY | Keeps both shipped Claude ingress gates and changes only their reader to `clientIntegrationEnabled`. | -| `src/server/index.ts` | MODIFY | Keeps Anthropic discovery gated and changes only its reader to `clientIntegrationEnabled`. | -| `tests/client-integration-desired-state.test.ts` | NEW | Pins migration, per-field mutation, contention/retry, and preservation. | -| `tests/client-integration-auto-gates.test.ts` | NEW | Pins automatic gates, shared flights, last-moment re-checks, and OpenCode activation. | -| `tests/client-integration-reconciliation.test.ts` | NEW | Pins persist/mutate crash points and startup/ensure/status convergence. | -| `tests/management-integration-routes.test.ts` | MODIFY | Pins the six-client persist-before-mutate route and desired/observed responses. | -| `tests/native-grok-toggle.test.ts` | MODIFY | Pins field-scoped persistence, conflict reporting, and retry after lock refusal. | -| `tests/native-claude-code-toggle.test.ts` | MODIFY | Replaces the pinned live-object mutation bug with no-mutation-before-commit coverage. | -| `tests/claude-management-api.test.ts` | MODIFY | Pins compatibility mirroring through the older Claude route. | -| `tests/claude-messages-endpoint.test.ts` | MODIFY | Keeps the legacy OFF => 403 contract through the new reader. | - -OUT: - -| Path / surface | Reason | +| `src/types.ts` | MODIFY | Adds a one-key `OcxClientIntegrationsConfig` and its optional `OcxConfig.clientIntegrations` home. | +| `src/config.ts` | MODIFY | Parses the Codex key, resolves absent as ON, re-reads persisted intent, and mutates only that field through the real `mutatePersistedConfig` signature. | +| `src/codex/desired-state.ts` | NEW | Owns the Codex-only process/OS write flight, last-moment authority checks, owned restore wrapper, and OFF reconciliation. | +| `src/codex/sync.ts` | MODIFY | Admits automatic Codex sync only while desired ON and passes a fresh-write authority through catalog and injection. | +| `src/codex/refresh.ts` | MODIFY | Carries the authority to the direct catalog/cache path used outside `syncModelsToCodex`. | +| `src/codex/catalog/sync.ts` | MODIFY | Re-reads desired ON after model gathering and immediately before catalog/cache replacement. | +| `src/codex/catalog/bundled.ts` | MODIFY | Prevents fallback catalog materialization before a fresh desired-ON check. | +| `src/codex/catalog/parsing.ts` | MODIFY | Calls the authority separately before each pristine-backup copy/write. | +| `src/codex/inject.ts` | MODIFY | Re-checks desired ON at the injection commit boundaries and makes the unchecked remover internal to the owned wrapper. | +| `src/server/management-api.ts` | MODIFY | Gives provider/model/combo refreshes their own Codex gate and routes `/api/stop` through the owned remover. | +| `src/server/management/config-routes.ts` | MODIFY | Makes `POST /api/sync` report an intentional desired-OFF skip instead of false success. | +| `src/cli/index.ts` | MODIFY | Reconciles OFF after journal repair, explains start/ensure skips, and routes every CLI/shutdown remover through the owned flight. | +| `src/cli/init.ts` | MODIFY | Uses the write flight without turning explicit bootstrap into an automatic desired-state gate. | +| `src/service.ts` | MODIFY | Routes service stop/uninstall removers through the same owned flight without changing desired state. | +| `tests/codex-desired-state.test.ts` | NEW | Pins schema defaulting, field-scoped persistence, auth-sentinel isolation, and unavailable/conflict behavior. | +| `tests/codex-desired-state-race.test.ts` | NEW | Pins in-flight OFF, crash-point convergence, single-flight, and foreign-home refusal. | +| `tests/codex-sync-api.test.ts` | MODIFY | Pins sync and `POST /api/sync` OFF semantics. | +| `tests/codex-inject-integration.test.ts` | MODIFY | Pins guarded commit boundaries and the owned native remover. | +| `tests/codex-journal.test.ts` | MODIFY | Proves journal repair still runs while desired Codex state is OFF. | +| `tests/service.test.ts`, `tests/uninstall.test.ts` | MODIFY | Proves owned teardown remains unconditional with respect to desired ON/OFF. | +| `tests/server-auth.test.ts` | MODIFY | Proves Codex OFF does not gate the shared `/v1/responses` transport. | + +OUT, deliberately: + +| Path / surface | Disposition | |---|---| -| `gui/` | WP3 adds the contract, not a new card. WP5 and WP6 consume it. | -| `src/codex/journal.ts` | Crash reconciliation repairs a half-applied Codex write regardless of desired state (`src/codex/journal.ts:148-162`). | -| `src/service.ts` stop/uninstall teardown | Teardown removes dead proxy pointers; it neither consults nor rewrites desired state (`src/service.ts:2587-2594`). | -| `src/grok/inject.ts` non-loopback cleanup | Credential-safety cleanup remains unconditional (`src/grok/inject.ts:359-380`). | -| `/v1/responses` | No client-specific gate guards it today. Codex OFF must not close the transport used by other clients. | -| Codex/Desktop remover implementations | WP5 and WP6 implement those two removers against this contract. Their shared-file work is sequential, not parallel. | -| releases, publishing, deploys, tags, repository starring | No delivery or user-identity action belongs in this phase. | - -## The schema and effective-state reader - -MODIFY `src/types.ts` immediately before `OcxConfig` (`src/types.ts:521-533`), -then put the field beside `claudeCode` (`src/types.ts:541-545`): +| `src/server/claude-messages.ts`, `src/server/index.ts`, `src/cli/claude.ts`, `src/claude/agents-inject.ts`, `src/server/system-env.ts` | **Dropped from WP4.** Round 1 #1 established `claudeCode.enabled` as the shipped Claude Code kill switch. WP4 neither changes it nor routes it through a helper. | +| `src/integrations/state.ts`, `src/integrations/writer.ts`, `src/server/management/integration-routes.ts`, `src/cli/opencode.ts` | **Moved out.** The six file clients are `FOLLOWUP-FILECLIENT-01`; this removes the old gates, writer changes, mutating GET, and migration claims rejected by round 2 #4/#5. | +| `src/grok/**`, Grok routes | **Moved to WP6.** Grok will add its own key and prove its own callers after the Codex shape passes. | +| `src/claude/desktop-3p.ts`, Desktop routes | **Moved to WP7.** Desktop keeps its separate ownership/profile questions. | +| `src/server/management/native-integration-routes.ts`, `gui/` | **Moved to WP5.** WP5 owns the Codex route, GUI parser, and UI contract. WP4 does not define a `codex | claude | claude-desktop | grok` union or `desiredEnabled` response schema. | +| desired-state admission in `ocx init` | **Not added. INFERRED:** `ocx init` is a user-commanded setup operation, not one of the automatic re-apply paths named by this phase. Its direct injection still uses the Codex write flight so it cannot overlap another irreversible Codex write. | +| `/v1/responses` | Never gated. It is a shared transport used by clients other than native Codex. | +| releases, publishing, deploys, tags, repository starring | No delivery or identity action belongs in this phase. | + +## The flag: one key in an extension-safe object + +Use a map-shaped object with **one key today**, not a top-level `codexEnabled` +field and not the prior ten-key union. A top-level field would force WP6 and WP7 +to invent unrelated names and helpers; a ten-key type would recreate the coupling +that failed two audits. A one-key object preserves the upgrade-safe extension +point while making WP4 incapable of claiming ownership over another client. + +MODIFY `src/types.ts` immediately before `OcxConfig` (current +`src/types.ts:521-533`) and place the field beside `claudeCode` +(`src/types.ts:533-545`): ```diff export interface OcxApiKeyEntry { @@ -73,65 +97,35 @@ then put the field beside `claudeCode` (`src/types.ts:541-545`): createdAt: string; } -+export type ClientIntegrationId = -+ | "codex" -+ | "claude-code" -+ | "claude-desktop" -+ | "grok" -+ | "opencode" -+ | "pi" -+ | "hermes" -+ | "openclaw" -+ | "kimi" -+ | "gajae"; -+ ++export interface OcxClientIntegrationsConfig { ++ /** Durable desired state for native Codex. Missing means ON. */ ++ codex?: boolean; ++} + export interface OcxConfig { ``` ```diff /** Claude Code inbound + launcher settings. */ claudeCode?: OcxClaudeCodeConfig; -+ /** Durable user intent. Missing map/key means ON for upgrade compatibility. */ -+ clientIntegrations?: Partial>; ++ /** Per-client durable intent. WP4 owns only `codex`; later phases extend one key at a time. */ ++ clientIntegrations?: OcxClientIntegrationsConfig; ``` -MODIFY `src/config.ts`. The parser salvages each known key independently. A -hand-edited `"codex": "false"` becomes absent/ON without discarding a valid -`"grok": false` beside it. The object stays `.passthrough()` so an older binary -does not erase a newer client's key on its next field-scoped mutation. - -```diff - import { - isWirePinnedModel, - MODEL_ADAPTER_OVERRIDE_ALLOWED, - OPENAI_PROVIDER_TIER_VERSION, - pinnedWireAdapter, - REASONING_SUMMARY_DELIVERY_VALUES, -+ type ClientIntegrationId, - type OcxClaudeCodeConfig, - type OcxConfig, -``` +MODIFY `src/config.ts` after `apiKeyEntrySchema` +(`src/config.ts:909-918`). The nested schema stays `.passthrough()` so a binary +from WP4 does not erase a later WP6/WP7 key during a field-scoped mutation. ```diff - const apiKeyEntrySchema = z.object({ - key: z.string().refine(isUsableApiKeySecret), - id: z.string().catch(""), name: z.string().catch(""), createdAt: z.string().catch(""), }).passthrough(); +const clientIntegrationsSchema = z.object({ + codex: z.boolean().optional().catch(undefined), -+ "claude-code": z.boolean().optional().catch(undefined), -+ "claude-desktop": z.boolean().optional().catch(undefined), -+ grok: z.boolean().optional().catch(undefined), -+ opencode: z.boolean().optional().catch(undefined), -+ pi: z.boolean().optional().catch(undefined), -+ hermes: z.boolean().optional().catch(undefined), -+ openclaw: z.boolean().optional().catch(undefined), -+ kimi: z.boolean().optional().catch(undefined), -+ gajae: z.boolean().optional().catch(undefined), +}).passthrough(); + + const configSchema = z.object({ ``` ```diff @@ -140,647 +134,367 @@ does not erase a newer client's key on its next field-scoped mutation. providerContextCaps: z.record(z.string(), z.number().int().positive()).optional(), ``` -Add the only effective-state reader beside `websocketsEnabled` -(`src/config.ts:1909-1911`). No caller may open-code `?.[client] ?? true` because -Claude Code's old field is the migration exception. +Effective state is `config.clientIntegrations?.codex !== false`. Missing object, +missing key, and explicit `true` all mean ON. A malformed hand edit such as +`{ "codex": "false", "future-client": false }` degrades only `codex` to absent/ON +and preserves the unknown future key; it does not invalidate providers or create +another client's block. -```diff - export function websocketsEnabled(config: Pick): boolean { - return config.websockets === true; - } +## Field-scoped persistence and the auth sentinel -+export function clientIntegrationEnabled( -+ config: Pick, -+ client: ClientIntegrationId, -+): boolean { -+ const desired = config.clientIntegrations?.[client]; -+ if (desired !== undefined) return desired !== false; -+ if (client === "claude-code") return config.claudeCode?.enabled !== false; -+ return true; -+} -``` - -Truth table: - -| New key | Legacy `claudeCode.enabled` | Effective state | -|---|---:|---:| -| absent | absent / `true` | ON | -| absent | `false` | OFF | -| `true` | any | ON | -| `false` | any | OFF | - -## A2 — field-scoped persistence uses the primitive that already exists - -The failed draft invented `saveConfigPreservingClaudeCode` as the desired-state -writer and mutated the request's live `config` before saving it. The shipped -Claude route still demonstrates the bug: it assigns `config.claudeCode = next` -at `src/server/management/native-integration-routes.ts:402-415`, then persistence -can refuse at `:420-431`. The regression test explains why a retry needs a fresh -object (`tests/native-claude-code-toggle.test.ts:213-218`). - -Do not add that writer. `mutatePersistedConfig` already has the required real -contract (`src/config.ts:1825-1832,1854-1906`): +The real primitive is synchronous and callback-based +(`src/config.ts:1854-1856`): ```ts export function mutatePersistedConfig( - mutate: (config: OcxConfig) => { changed: boolean; value: T }, -): - | { status: "committed" | "unchanged"; value: T } - | { status: "unavailable"; reason: "missing" | "invalid" | "conflict" }; + mutate: (config: OcxConfig) => PersistedConfigMutation, +): PersistedConfigMutationOutcome; ``` -It reads the current disk bytes, clones before invoking the callback, re-runs the -callback against the latest snapshot, and commits the confirmed clone under the -shared SQLite mutation lock. Build the one-key mutation on top of that signature: +Build the Codex writer directly on it after `websocketsEnabled` +(`src/config.ts:1909-1911`): ```diff -+export interface ClientIntegrationMutationValue { + export function websocketsEnabled(config: Pick): boolean { + return config.websockets === true; + } + ++export function codexDesiredEnabled( ++ config: Pick, ++): boolean { ++ return config.clientIntegrations?.codex !== false; ++} ++ ++export interface CodexDesiredMutationValue { + config: OcxConfig; + desiredEnabled: boolean; +} + -+export function mutateClientIntegrationEnabled( -+ client: ClientIntegrationId, ++export function mutateCodexDesiredEnabled( + enabled: boolean, -+): PersistedConfigMutationOutcome { ++): PersistedConfigMutationOutcome { + return mutatePersistedConfig(config => { -+ const mapAlreadyMatches = config.clientIntegrations?.[client] === enabled; -+ const legacyAlreadyMatches = client !== "claude-code" -+ || config.claudeCode?.enabled === enabled; -+ if (mapAlreadyMatches && legacyAlreadyMatches) { ++ if (config.clientIntegrations?.codex === enabled) { + return { changed: false, value: { config, desiredEnabled: enabled } }; + } -+ config.clientIntegrations = { ...config.clientIntegrations, [client]: enabled }; -+ if (client === "claude-code") { -+ config.claudeCode = { ...(config.claudeCode ?? {}), enabled }; -+ } ++ config.clientIntegrations = { ...config.clientIntegrations, codex: enabled }; + return { changed: true, value: { config, desiredEnabled: enabled } }; + }); +} ``` -Only the callback-local clone is mutated. A route uses `outcome.value.config` for -the following file operation; it does not patch `ctx.config` before or after the -commit. A later status read loads persisted state. This prevents one long-lived -request object from overwriting a newer disk edit and makes lock refusal retryable -with the same object. - -The helper touches only `clientIntegrations[client]`, plus -`claudeCode.enabled` for the one compatibility client. It preserves all other -client keys, providers, API settings, and unrelated `claudeCode` fields. +Only the callback-local clone is mutated. A future WP5 route must use +`outcome.value.config` after `committed | unchanged`; it must never patch the +long-lived management `config` before persistence succeeds. `missing`, `invalid`, +and `conflict` leave both disk and the supplied live object unchanged. -Required persistence tests: +Round 2 #1 is **unreachable for this Codex-only shape**. The mutation above writes +only the sibling `clientIntegrations` object. It never reads, spreads, creates, or +assigns `config.claudeCode`. `runClaudeAuthModeMigration` returns immediately when +that block is absent (`src/claude/auth-mode-migration.ts:16-20`) and is invoked on +startup only afterward (`src/server/index.ts:290-294`). Therefore a Codex flag +write cannot create the pre-upgrade sentinel condition. The test still runs the +real migration after OFF and ON mutations and asserts: no `claudeCode` block, +`runClaudeAuthModeMigration(...) === false`, and no persisted `authMode` or +`authModeMigratedAt`. -1. Two writers toggling different clients from the same stale starting object - both survive in the final file. -2. Simultaneous toggles of different clients preserve both keys; simultaneous - opposing toggles of one client serialize to one whole outcome, never a split - legacy/new representation. -3. A held mutation lock refuses without changing the live object; retry with the - same object succeeds after release. -4. Claude mirroring preserves `authMode`, `injectAgents`, `desktopProfile`, - `desktopAutoApply`, and unknown hand-edited Claude fields. +## One Codex write flight, with a last-moment authority read -## A1 reversal — keep client-specific Claude ingress admission +Entry checks do not close the race. `syncModelsToCodex` can pause in provider +model gathering (`src/codex/sync.ts:83-108`), while another process persists OFF, +then continue into injection at `src/codex/sync.ts:110`. -The earlier invariant was over-broad. Client installation state must not stop the -proxy or a different client, but ingress admission is itself a client contract. -`claudeCode.enabled` is the documented, shipped kill switch used before body work -for `/v1/messages` and `/v1/messages/count_tokens` -(`src/server/claude-messages.ts:65-69,543-548,868-872`) and before Anthropic model -discovery (`src/server/index.ts:493-502`). Removing those checks would turn a -legacy OFF into ON during upgrade. - -KEEP every gate and change only the reader: - -```diff -+import { clientIntegrationEnabled } from "../config"; -@@ - function claudeInboundDisabled(config: OcxConfig): Response | null { -- if (config.claudeCode?.enabled === false) { -+ if (!clientIntegrationEnabled(config, "claude-code")) { - return anthropicErrorResponse(403, "Claude inbound is disabled (GUI: Claude ON toggle / config.claudeCode.enabled)", "permission_error"); - } - return null; - } -``` - -```diff - const wantsAnthropicList = req.headers.get("anthropic-version") !== null - || url.searchParams.get("flavor") === "anthropic"; - if (wantsAnthropicList && !url.searchParams.has("client_version")) { -- if (config.claudeCode?.enabled === false) return jsonResponse({ data: [] }, 200, req, config); -+ if (!clientIntegrationEnabled(config, "claude-code")) { -+ return jsonResponse({ data: [] }, 200, req, config); -+ } -``` - -The corrected invariant is precise: - -| Surface | Desired OFF behavior | -|---|---| -| Codex `/v1/responses` | Remains admitted. No Codex client gate guarded this transport. | -| Claude Code `/v1/messages` and `/count_tokens` | Returns the shipped 403 because this is Claude Code ingress admission. | -| Anthropic-flavored model discovery | Returns an empty list while Claude Code is OFF, preserving the shipped kill switch. | -| Proxy lifecycle | Remains running. No toggle calls stop/restart/uninstall. | -| A different client's writer/transport | Remains available unless that different client's own desired key is OFF. | - -Compatibility activation: load a legacy file containing only -`claudeCode.enabled=false`, run the migration/load path with no new key, then hit -both Messages handlers and Anthropic discovery. Both handlers still return 403 -and discovery still returns `{ data: [] }`. This test must fail if either old -gate is removed. - -## A6 — the complete shared native contract, consumed rather than redefined - -The first roadmap dispatched WP5 and WP6 in parallel even though both edit -`native-integration-routes.ts` and its client union. Their proposed unions do not -compose: one adds Codex and the other adds Desktop. WP3 defines the final contract -once. WP5 runs first where shared files overlap; WP6 rebases on WP5 and runs -second. They may proceed independently only on disjoint files. - -MODIFY `src/server/management/native-integration-routes.ts:31-74`: - -```diff --export type NativeIntegrationClientId = "claude" | "grok"; -+export type NativeIntegrationClientId = -+ | "codex" -+ | "claude" -+ | "claude-desktop" -+ | "grok"; -@@ - export interface NativeStatus { - clientId: NativeIntegrationClientId; - state: "absent" | "current" | "unsafe"; -+ desiredEnabled: boolean; - installed: boolean; -@@ - export interface NativeToggleEnvelope { - ok: true; - clientId: NativeIntegrationClientId; - changed: boolean; - state: NativeStatus["state"]; -+ desiredEnabled: boolean; - message: string; -@@ - export interface NativeRefusalEnvelope { - error: string; - code: "native_integration_refused" | "native_integration_failed"; - clientId: NativeIntegrationClientId; - reason: NativeRefusalReason; - message: string; -+ desiredEnabled: boolean; -+ observedState?: NativeStatus["state"]; -+ residualPaths?: string[]; - } -``` - -The status helpers own the native-id mapping and make omission a type error: - -```diff -+function desiredClientId(clientId: NativeIntegrationClientId): ClientIntegrationId { -+ return clientId === "claude" ? "claude-code" : clientId; -+} -+ -+function desiredEnabledForNative( -+ config: Pick, -+ clientId: NativeIntegrationClientId, -+): boolean { -+ return clientIntegrationEnabled(config, desiredClientId(clientId)); -+} -+ -+function withDesiredState( -+ config: Pick, -+ observed: Omit, -+): NativeStatus { -+ return { -+ ...observed, -+ desiredEnabled: desiredEnabledForNative(config, observed.clientId), -+ }; -+} -``` - -`claudeStatus`, `grokStatus`, and later `codexStatus`/`desktopStatus` return through -`withDesiredState`. Every success literal supplies `desiredEnabled`; every refusal -uses a single serializer that supplies the persisted intent and, after persistence, -the last observed state. WP5 and WP6 delete their local union/schema diffs and use -these helpers. - -The six file-client schema follows the same two-state rule. MODIFY -`src/integrations/state.ts:30-42` and the route envelopes at -`src/server/management/integration-routes.ts:44-55`: - -```diff - export interface IntegrationStatus { - clientId: IntegrationClientId; - state: IntegrationState; -+ desiredEnabled: boolean; - installed: boolean; -``` - -```diff --export type IntegrationToggleEnvelope = -- | ({ clientId: IntegrationClientId } & ApplyResult) -- | ({ clientId: IntegrationClientId } & DisableResult); -+export type IntegrationToggleEnvelope = ( -+ | ({ clientId: IntegrationClientId } & ApplyResult) -+ | ({ clientId: IntegrationClientId } & DisableResult) -+) & { desiredEnabled: boolean }; -``` - -`readIntegrationState` is the status helper every surface already uses -(`src/integrations/state.ts:225-289`); add -`desiredEnabled: clientIntegrationEnabled(input.config, input.clientId)` to all -three return sites, including unsafe path-resolution and unreadable-file returns. - -## A3 — six file clients persist intent before touching their files - -The failed draft put `ocx opencode` behind a desired-state guard but gave -OpenCode, Pi, Hermes, OpenClaw, Kimi, and Gajae no writer for that state. The -real switch is `PUT /api/client-integrations/:clientId`, which currently goes -straight from body validation to `applyIntegration`/`disableIntegration` -(`src/server/management/integration-routes.ts:507-527`). Route it through the same -field-scoped mutation first: - -```diff - const parsed = await readJsonBody(ctx); - if (parsed instanceof Response) return parsed; -@@ - try { -+ const persisted = mutateClientIntegrationEnabled(requestedClient, parsed.enabled); -+ if (persisted.status === "unavailable") { -+ return desiredStatePersistenceFailure(requestedClient, persisted.reason, ctx); -+ } -+ const operationConfig = persisted.value.config; -- const input = await buildIntegrationWriteInput(requestedClient, ctx, integrationStore()); -+ const input = await buildIntegrationWriteInput( -+ requestedClient, -+ ctx, -+ integrationStore(), -+ operationConfig, -+ ); - const result = await runClientIntegrationFlight( - requestedClient, - parsed.enabled ? "apply" : "disable", -- input.io?.now ?? Date.now, - () => Promise.resolve(parsed.enabled - ? applyIntegration(input) - : disableIntegration(input)), - ); -- if (!result.ok) return writerFailureResponse(requestedClient, result, ctx); -- return jsonResponse(result satisfies IntegrationToggleEnvelope, 200, req, ctx.config); -+ if (!result.ok) { -+ return writerFailureResponse(requestedClient, result, ctx, { -+ desiredEnabled: parsed.enabled, -+ observedState: readIntegrationState(input).state, -+ }); -+ } -+ return jsonResponse({ -+ ...result, -+ desiredEnabled: parsed.enabled, -+ } satisfies IntegrationToggleEnvelope, 200, req, operationConfig); -``` - -The helper takes the committed snapshot explicitly so model export and the writer -cannot fall back to the stale request object: - -```diff - async function buildIntegrationWriteInput( - clientId: IntegrationClientId, - ctx: ManagementContext, - store: IntegrationStateStore, -+ config: OcxConfig = ctx.config, - ): Promise { - return { - clientId, -- models: await loadExportModels(ctx.config), -- config: ctx.config, -- port: Number(ctx.url.port) || ctx.config.port, -+ models: await loadExportModels(config), -+ config, -+ port: Number(ctx.url.port) || config.port, -``` - -Ordering and failure behavior are fixed: - -1. Invalid body: no intent and no file change. -2. Missing/invalid/conflicted config or lock refusal: no intent and no file change; - return retryable `config_busy` only for real contention. -3. Intent committed, file mutation succeeds: return desired and observed state. -4. Intent committed, file mutation refuses/fails: never roll intent back. Return - `desiredEnabled` plus freshly inspected `observedState` and the writer's recovery - fields. Startup/ensure/status retries convergence. - -Activation test: create a real temporary OpenCode config, disable OpenCode through -the real management route, then invoke `cmdOpencode`. The command must refuse before -proxy ensure/spawn, and the OpenCode file bytes must remain unchanged. This proves -the CLI guard is reachable from the state the real switch writes. - -## A4 — per-client single-flight and the last-moment write check - -Entry checks alone are racy. Codex awaits catalog work at -`src/codex/sync.ts:83-108` and then injects at `:110`; Grok awaits model discovery -at `src/grok/sync.ts:35-57` and then injects at `:61-65`. Either can start ON, -pause, persist OFF in another request, and write after OFF. - -NEW `src/integrations/desired-state.ts` owns one operation boundary for all ten -ids. It replaces route-local Grok/apply and six-client flight maps. The boundary -has two layers: - -- an in-process promise map joins an identical operation and refuses a competing - direction; -- an OS-backed SQLite transaction in a per-client coordinator file prevents a - CLI/startup process and the server's GUI/background process from writing the - same client concurrently. Separate files preserve concurrency between different - clients. Process exit releases the transaction; there is no stale lease row. - -Every GUI route, CLI writer, startup sync, ensure sync, Desktop auto-apply, Grok -apply, Codex refresh, and WP5/WP6 native mutation reaches -`runClientIntegrationFlight(clientId, operationKey, operation)`. It enters exactly -once at the lowest shared owner of the irreversible write: a route that delegates -to `syncGrokConfig` or `syncModelsToCodex` does not acquire an outer flight and -deadlock the same client. Direct strip/restore routes acquire it themselves. No -surface keeps its own map. - -The flight is necessary but not sufficient. Every irreversible write calls this -immediately before commit: +NEW `src/codex/desired-state.ts` owns one Codex coordinator: ```ts -export function requirePersistedClientIntent( - client: ClientIntegrationId, - expectedEnabled: boolean, -): { ok: true; config: OcxConfig } | { - ok: false; - reason: "desired_state_changed" | "desired_state_unavailable"; -}; -``` +export type CodexWriteDirection = "apply" | "remove"; -The helper reads a fresh valid disk snapshot. Missing or invalid state fails -closed; it never falls back to a stale request object. Apply/inject/spawn requires -ON. Disable/removal requires OFF. The check is placed after async catalog/model -work and after compare-before-write, directly before each of these boundaries: +export type PersistedCodexAuthority = + | { ok: true; config: OcxConfig } + | { ok: false; reason: "desired_state_changed" | "desired_state_unavailable" }; -| Writer | Last-moment check | -|---|---| -| Codex catalog refresh | before catalog atomic replace | -| `injectCodexConfig` | after model resolution, before config/journal mutation | -| `injectGrokConfig` | after catalog resolution, before fenced-file write | -| six-client `commit` | after the existing byte recheck (`src/integrations/writer.ts:292-317,367-384`), before snapshot/file/record commit | -| Desktop profile/meta writers | before each selected-profile or metadata write | -| `cmdOpencode` | once at entry and again immediately before spawning with `OPENCODE_CONFIG_CONTENT` | +export interface CodexReconcileResult { + trigger: "startup" | "ensure"; + desiredEnabled: boolean; + observedState: "absent" | "applied" | "conflict" | "unavailable"; + resolved: boolean; + reason?: "home_mismatch" | "history_locked" | "write_failed" | "codex_write_busy"; + message: string; +} -If the expected intent changed, the writer returns a typed skip/refusal and writes -nothing. A caller may retry under the new direction; it may not continue with the -old snapshot. +export function requirePersistedCodexIntent( + expectedEnabled: boolean, +): PersistedCodexAuthority; -Deterministic race test: pause Codex and Grok model resolution on a controlled -promise, persist that client OFF through the real mutation helper, release the -promise, and assert catalog/inject spies remain zero. Repeat one file client with -its commit hook. The observable proof is unchanged target bytes, not only a skip -message. +export async function runCodexWriteFlight( + direction: CodexWriteDirection, + operation: () => Promise, +): Promise; -## Automatic gates use the same owner +export function runCodexWriteFlightSync( + direction: CodexWriteDirection, + operation: () => T, +): T; -The entry gates remain useful because they avoid needless catalog work, but each -one enters the shared flight and still performs the last-moment check above. +export function restoreNativeCodexOwned(): { success: boolean; message: string }; -### Codex +export async function reconcileCodexDesiredState( + trigger: "startup" | "ensure", +): Promise; +``` -MODIFY `src/codex/sync.ts:49-55`: +**INFERRED design choice:** the coordinator has one in-process tail and one +OS-backed SQLite transaction at +`getCodexHome()/opencodex-write.sqlite` (`src/codex/paths.ts:32-35`). The lock is +keyed by the native target, not `OPENCODEX_HOME`: two OpenCodex homes can point at +the same `CODEX_HOME`, and they must not acquire different locks for the same +files. The config mutation lock is deliberately not reused: holding it across +model fetch would prevent OFF from being persisted, which is the race this phase +must handle. A second process waits with a bounded timeout; timeout returns +`codex_write_busy` and writes nothing. The sync form exists for +shutdown/`process.on("exit")`, where a Promise cannot be awaited. + +`requirePersistedCodexIntent` uses `readConfigDiagnostics` +(`src/config.ts:1691-1708`), not a request's captured config. A missing or invalid +file is unavailable at a write boundary and fails closed; it is not reinterpreted +as an upgrade-time ON after an operation has already begun. + +Every automatic apply path enters `runCodexWriteFlight("apply", ...)`, and the +authority is re-read after its last await and immediately before each commit +boundary. `InjectCodexOptions` and the catalog helpers receive a +`beforeWrite(boundary)` callback; they call it again for every separate file or DB +write rather than treating several writes as one group: + +| Boundary | Current write | WP4 check | +|---|---|---| +| bundled fallback | `materializeBundledCodexCatalog` at `src/codex/catalog/bundled.ts:213-219` | pass the authority into `loadCatalogForSync`; re-read before fallback materialization | +| pristine backups | `copyFileSync` / `atomicWriteFile` at `src/codex/catalog/parsing.ts:428-444` | re-read separately inside `writePristineCatalogBackup` before each backup copy/write | +| catalog | `atomicWriteFile(catalogPath, ...)` at `src/codex/catalog/sync.ts:568` | `requirePersistedCodexIntent(true)` after `gatherRoutedModels` and directly before replace | +| models cache | `atomicWriteFile(activeCodexModelsCachePath(), ...)` at `src/codex/catalog/sync.ts:600-613` | re-read before cache replacement | +| injection journal | `writeJournal(...)` at `src/codex/inject.ts:521-527` | re-read before recording an apply transaction | +| config | first atomic write at `src/codex/inject.ts:593-596` | re-read immediately before `CODEX_CONFIG_PATH` replacement | +| profile | second atomic write at `src/codex/inject.ts:595-597` | re-read again immediately before `CODEX_PROFILE_PATH` replacement | +| journal injected marker | `markJournalInjectedState(...)` at `src/codex/inject.ts:597` | re-read again before advancing journal state | +| history mutation | `syncCodexHistoryProvider` / `migrateHistoryToOpenai` at `src/codex/inject.ts:598-603` | re-read before the DB mutation | +| native remove | `restoreNativeCodex` body at `src/codex/inject.ts:764-795` | ownership preflight first; startup reconciliation also re-reads OFF immediately before remove | + +`src/codex/inject.ts` renames the raw remover to +`restoreNativeCodexUnchecked`; only `src/codex/desired-state.ts` may import it. +All production callers import `restoreNativeCodexOwned` instead. A source-shape +test rejects any other import of the unchecked symbol. This makes the round 2 #2 +preflight an owned boundary rather than a convention each caller can forget. + +Stop, uninstall, and explicit restore use the owned remover but **do not require +desired OFF** and never rewrite the flag. They are safety teardown, not user-intent +mutation (`src/service.ts:2587-2594`). Startup/ensure reconciliation uses the same +remover with the additional fresh-OFF check. + +`src/service.ts` must not statically import the new wrapper. The wrapper imports +`assertNativeTeardownOwned`, whose current implementation imports `service.ts` +(`src/integrations/native/ownership-preflight.ts:14-17`); a static reverse import +would create `service -> desired-state -> ownership-preflight -> service`. Remove +the current static raw-remover import at `src/service.ts:15` and dynamically import +`restoreNativeCodexOwned` inside the already-async stop/uninstall branches before +calling it. **INFERRED:** this is the smallest way to keep the shipped preflight as +the authority without broadening WP4 into a service-ownership module extraction. + +## Automatic Codex gates + +### Normal sync path + +MODIFY `src/codex/sync.ts:49-55` so the entry gate avoids unnecessary fetches and +the write flight covers catalog plus injection as one Codex operation: ```diff --import { applyProxyEnv, loadConfig } from "../config"; -+import { applyProxyEnv, clientIntegrationEnabled, loadConfig } from "../config"; -@@ + export async function syncModelsToCodex( + port?: number, + config: OcxConfig = loadConfig(), + log: Pick | null = console, + deps: CodexSyncDeps = defaultDeps, ): Promise { -+ if (!clientIntegrationEnabled(loadConfig(), "codex")) { -+ return codexDesiredStateSkip(); -+ } -+ return runClientIntegrationFlight("codex", "sync", async () => { ++ if (!codexDesiredEnabled(config)) return codexDesiredOffSyncResult(log); ++ return runCodexWriteFlight("apply", async () => { const p = port ?? config.port ?? 10100; ``` -Close the flight after the existing result return. `refreshCodexCatalogBestEffort` -at `src/server/management-api.ts:105-112` uses the same flight/reader rather than -a separate boolean check, so provider/model routes cannot bypass ordering. +Close the flight after the existing return at `src/codex/sync.ts:114-129`. +`CodexSyncResult` adds optional `skippedReason: "desired-off" | +"desired-state-unavailable" | "codex-write-busy"`. Desired OFF is an intentional +no-write result, not a claim that catalog/injection completed. -### Grok +`ocx start` and both `ocx ensure` branches remain callers at +`src/cli/index.ts:318-320,358-411`; they inspect `skippedReason` and print +`Codex auto-apply skipped: desired state is OFF.` once. They do not stop the +proxy, alter its port, or skip another client's setup. -MODIFY `src/grok/sync.ts:29-35`: +`POST /api/sync` at `src/server/management/config-routes.ts:261-268` returns a +409 `codex_desired_off` envelope when the sync result says desired OFF. It must +not return the current 200-shaped success for an operation that intentionally +wrote nothing. **INFERRED:** 409 distinguishes a valid request blocked by current +desired state from a server fault; 200 would preserve the false-green finding and +500 would misclassify an intentional policy decision. Other sync failures keep +their existing 500 behavior. -```diff -+import { clientIntegrationEnabled, loadConfig } from "../config"; -+import { runClientIntegrationFlight } from "../integrations/desired-state"; -@@ - ): Promise { -+ if (!clientIntegrationEnabled(loadConfig(), "grok")) { -+ return { ok: true, changed: false, message: "Grok config sync skipped: desired state is OFF." }; -+ } -+ return runClientIntegrationFlight("grok", "sync", async () => { - let models: GrokInjectModel[]; -``` +### Provider/model/combo refresh bypass -Close the flight after injection. `/api/grok/apply` at -`src/server/management/agent-settings-routes.ts:639-657` deletes its local -`grokApplyFlight`; the shared owner covers start, both ensure branches, GUI apply, -toggle, and background work. - -### Desktop and Claude consumers - -Desktop auto-apply at `src/server/management/agent-settings-routes.ts:130-150` -requires both policies and enters the Desktop flight: +The management helper currently calls `refreshCodexModelCatalog(config)` directly +(`src/server/management-api.ts:105-112`), so a gate only in +`syncModelsToCodex` is insufficient. Replace it with a separately gated entry: ```diff - async function autoApplyDesktopBestEffort(): Promise { + async function refreshCodexCatalogBestEffort(): Promise { +- if (deps.refreshCodexCatalog) return deps.refreshCodexCatalog(); try { -+ if (!clientIntegrationEnabled(loadConfig(), "claude-desktop")) return; - if (config.claudeCode?.desktopAutoApply === false) return; -``` - -`desktopAutoApply:false` is not migrated into Desktop OFF. Claude launcher, -agent injection, and system-env replace direct legacy reads with -`clientIntegrationEnabled(config, "claude-code")` as in the first draft. Claude -ingress and discovery retain their gates as specified in A1. - -### OpenCode - -MODIFY `src/cli/opencode.ts:531-533`: - -```diff - export async function cmdOpencode(args: string[]): Promise { - const config = loadConfig(); -+ if (!clientIntegrationEnabled(config, "opencode")) { -+ console.error("OpenCode integration is disabled — turn it ON before using `ocx opencode`."); -+ return 1; -+ } - const live = await ensureProxyForOpencode(config); -``` - -The command enters the OpenCode flight and repeats -`requirePersistedClientIntent("opencode", true)` immediately before spawn. OFF -must not start the proxy merely to refuse later. - -## A5 — startup reconciliation: OFF means converge, not skip - -Persist OFF, crash before the remover, restart: the first draft would skip future -apply and leave desired OFF / observed ON forever. Desired OFF is therefore a -converge instruction. - -NEW `src/integrations/reconcile.ts` exposes: - -```ts -export interface ClientReconcileResult { - clientId: ClientIntegrationId; - desiredEnabled: boolean; - observedState: "absent" | "current" | "stale" | "conflict" | "unsafe"; - resolved: boolean; - message: string; -} - -export async function reconcileDisabledClientIntegrations( - trigger: "startup" | "ensure" | "status", - options?: { only?: readonly ClientIntegrationId[] }, -): Promise; +- const { refreshCodexModelCatalog } = await import("../codex/refresh"); +- await refreshCodexModelCatalog(config); ++ const { refreshCodexCatalogIfDesired } = await import("../codex/sync"); ++ await refreshCodexCatalogIfDesired(async freshConfig => { ++ if (deps.refreshCodexCatalog) return deps.refreshCodexCatalog(); ++ const { refreshCodexModelCatalog } = await import("../codex/refresh"); ++ await refreshCodexModelCatalog(freshConfig); ++ }); + } catch { + /* catalog absent */ + } + } ``` -For every client whose fresh persisted intent is OFF: - -1. Inspect observed state. -2. If absent, report resolved without writing. -3. If applied/current/stale, enter that client's shared flight, re-read OFF, and - run the existing idempotent remover. -4. Re-inspect. Report `resolved:true` only when observed state is absent. -5. Preserve OFF and return an unresolved conflict for ownership, drift, unsafe - metadata, history lock, or write failure. Never report desired OFF as observed - OFF merely because the remover was attempted. +`refreshCodexCatalogIfDesired` loads persisted state, enters the same Codex write +flight, and passes the same last-moment authority into `refreshCodexModelCatalog`. +The injected test dependency is inside that gate, not an early-return bypass. +Provider/model/combo routes therefore cannot bypass OFF, but no Claude, Grok, +Desktop, or file-client reader changes. -The six file clients use `disableIntegration`; Grok uses `stripGrokConfig`; Claude -Code has no external artifact and resolves from the persisted admission flag. WP5 -registers Codex's `restoreNativeCodex` remover, then WP6 registers Desktop's -standard-mode remover. Registration is exhaustive over `ClientIntegrationId`, so -the final WP6 build cannot compile with either new native client omitted. +## Startup reconciliation: OFF means remove again -Invoke reconciliation at these real boundaries: +The order in `handleStart` matters. Journal repair remains unconditional and runs +first; desired OFF convergence runs second; automatic sync runs later and observes +OFF: ```diff - // src/cli/index.ts:169-177 async function handleStart(options: { block?: boolean } = {}) { @@ const requestedPort = parsePortOption(); -+ await reconcileDisabledClientIntegrations("startup"); - if (!currentExternalCodexModelProvider()) reconcileJournal(); -``` - -```diff - // src/cli/index.ts:358-365 - async function handleEnsure() { if (!currentExternalCodexModelProvider()) reconcileJournal(); -+ await reconcileDisabledClientIntegrations("ensure"); - const config = loadConfig(); ++ await reconcileCodexDesiredState("startup"); + const existingPid = readPid(); ``` -Both collection and per-client GET routes call status reconciliation for the ids -being read before `readIntegrationState`/native status helpers run. Status returns -the unresolved result in `disableBlocked` or response diagnostics; it does not -hide a conflict and does not flip desired state back ON. - -Crash-point tests use a hook immediately after `mutateClientIntegrationEnabled` -returns and before the remover begins. Terminate the simulated request there, -then invoke each of startup, ensure, and status reconciliation. For OpenCode and -Grok, assert the previously applied bytes are removed. For a drift/ownership -fixture, assert bytes remain, desired stays false, and the unresolved conflict is -reported. WP5 and WP6 add the same crash point for Codex and Desktop when their -removers land. +Apply the same reconciliation after journal repair in `handleEnsure` +(`src/cli/index.ts:358-365`). Reconciliation does exactly this: + +1. Fresh-read desired state. ON or unavailable performs no removal. +2. For OFF, enter the Codex remove flight and fresh-read OFF again. +3. Immediately before removal, call `assertNativeTeardownOwned` inside the flight. +4. If ownership is foreign, return unresolved `home_mismatch`; preserve OFF and + every Codex byte. Otherwise call `restoreNativeCodexUnchecked`. +5. Inspect the native artifacts again. Report resolved only when OpenCodex routing, + profile, and proxy-routed catalog residue are absent. A history lock remains an + explained unresolved result; desired OFF is not rolled back. + +The crash point is after `mutateCodexDesiredEnabled(false)` commits and before the +remover starts. Restarting from that fixture must execute steps 1-5 again. A GET +route is not used as a repair trigger; round 2 #5's mutating-GET design is dropped. + +## Do not gate these paths + +- `reconcileJournal` remains unconditional (`src/codex/journal.ts:148-162`). It + repairs an abandoned transaction before desired-state convergence decides the + final direction. +- ownership and drift inspection always run. Desired OFF never bypasses + `assertNativeTeardownOwned`. +- stop, uninstall, shutdown, and explicit native restore always remove state they + own, regardless of desired ON, and never persist OFF + (`src/service.ts:2587-2594`). +- `/v1/responses` remains admitted. Codex OFF means “stop automatically writing + native Codex configuration,” not “stop serving Responses.” +- no Claude Code, Grok, Desktop, or file-client path reads the Codex key. ## Test plan -### `tests/client-integration-desired-state.test.ts` (NEW) +### `tests/codex-desired-state.test.ts` (NEW) | Case | Activation and assertion | |---|---| -| Absent-config upgrade | Load a file with no map; all ten ids are ON. | -| Claude legacy fallback | New key absent + legacy false is OFF; new key wins once present. | -| Per-key malformed salvage | `{ codex: "false", grok: false }` yields Codex ON and Grok OFF without losing providers. | -| Two stale writers | Different client toggles both survive because each callback rebases on latest disk. | -| Simultaneous toggles | Different client keys both commit; neither whole-object snapshot wins. | -| Lock refusal and retry | Refusal changes neither disk nor live object; retry with that same object succeeds. | -| Claude field preservation | Mirroring changes only the new key and legacy `enabled`; every unrelated Claude field survives. | +| Upgrade default | Load config with no `clientIntegrations`; Codex is ON and no bytes are rewritten. | +| One-key parser | `codex:false` loads OFF; malformed `codex:"false"` degrades to ON while a future unknown key survives a field mutation. | +| Field-scoped commit | Mutate OFF from a stale live object; unrelated providers, API keys, and unknown fields survive. The live object is unchanged. | +| Lock/conflict refusal | Hold the real config mutation lock; mutation changes neither disk nor live object. Retry succeeds after release. | +| Auth sentinel unreachable | Start with no `claudeCode`, mutate Codex OFF then ON, reload, run `runClaudeAuthModeMigration`; it returns false and never creates `authMode` or `authModeMigratedAt`. | -### `tests/client-integration-auto-gates.test.ts` (NEW) +### `tests/codex-desired-state-race.test.ts` (NEW) | Case | Activation and assertion | |---|---| -| Codex/Grok OFF at entry | Catalog and writer spies stay zero. | -| Codex/Grok OFF during fetch | Pause resolution, persist OFF, release; target bytes and writer counts remain unchanged. | -| Six-client last-moment check | Flip direction at the commit hook; no file/snapshot/record is written. | -| Desktop two-policy gate | Write occurs only when desired ON and `desktopAutoApply` permits it. | -| Real OpenCode activation | Disable through real PUT, invoke `cmdOpencode`; ensure/spawn stay zero and config bytes are unchanged. | -| Shared-flight coverage | GUI, CLI, startup, ensure, and background callers for one client cannot overlap; a different client can proceed. | - -### Route and compatibility regressions - -- `tests/management-integration-routes.test.ts`: all six PUTs persist intent before - file work; post-persist refusal returns required desired plus observed state. -- `tests/native-grok-toggle.test.ts`: same ordering, lock refusal/retry, and no - rollback of intent after ownership/catalog/write failure. -- `tests/native-claude-code-toggle.test.ts`: legacy OFF mirrors even when effective - state already matches; a failed persist leaves the supplied config object - untouched and the same object can retry. -- `tests/claude-management-api.test.ts`: the older route uses the same field-scoped - mutation and preserves migration sentinels/other Claude fields. -- `tests/claude-messages-endpoint.test.ts`: legacy `enabled:false` still returns - 403 from Messages and count-tokens; Anthropic discovery remains empty. - -### `tests/client-integration-reconciliation.test.ts` (NEW) - -For each current remover, persist OFF and stop at the post-persist/pre-mutate hook. -Run startup, ensure, and status reconciliation independently and prove observed -state becomes absent. Fault fixtures prove drift/ownership/unsafe removals remain -unresolved and visible without changing desired OFF. WP5/WP6 append Codex/Desktop -cases sequentially when those removers exist. +| OFF during model fetch | Pause `gatherRoutedModels`, persist OFF through the real writer, release; catalog, cache, journal, config, profile, and history writer counts remain zero. | +| Direct refresh bypass | Invoke a real provider/model route while OFF; `refreshCodexCatalogBestEffort` performs no catalog/cache write. | +| Single-flight | Hold one apply at the fetch seam, start a second-process refresh and a remove; no two Codex commit sections overlap, and the final operation re-reads the newest intent. | +| Crash after persist | Commit OFF, abort before remove, run startup and ensure independently; each converges a seeded applied Codex fixture to native state. | +| Foreign home | Seed install state for home A, run OFF startup reconciliation from home B; `assertNativeTeardownOwned` returns `home_mismatch`, all Codex bytes remain exact, and desired OFF remains persisted. | +| Stop does not change intent | With desired ON, run owned stop/uninstall teardown; artifacts are removed and the flag remains ON. | + +### Existing regressions + +- `tests/codex-sync-api.test.ts`: OFF at entry avoids fetch/inject; OFF during + fetch returns the typed skip; `POST /api/sync` is 409 `codex_desired_off`, not + 200 and not 500. +- `tests/codex-inject-integration.test.ts`: each injected write authority seam is + reachable independently (journal, config, profile, journal mark, history); only + `src/codex/desired-state.ts` imports the unchecked remover. +- `tests/codex-journal.test.ts`: seed a dead-PID journal while desired OFF; journal + reconciliation runs, then OFF convergence removes residual routing. +- `tests/service.test.ts` and `tests/uninstall.test.ts`: every production native + remover passes through `assertNativeTeardownOwned`; foreign-home teardown writes + nothing; owned teardown never changes desired state. +- `tests/server-auth.test.ts`: add a live-server case with + `clientIntegrations.codex=false`; `POST /v1/responses` reaches the same normal + validation/routing response as ON, never a client-disabled response. ## Verification +All tests use temporary `OPENCODEX_HOME`, `CODEX_HOME`, config, catalog, profile, +journal, history, and service-install-state fixtures. Do not point any command at +the user's live proxy on port 10100. + ```bash -bun test tests/client-integration-desired-state.test.ts -bun test tests/client-integration-auto-gates.test.ts -bun test tests/client-integration-reconciliation.test.ts -bun test tests/management-integration-routes.test.ts tests/native-grok-toggle.test.ts -bun test tests/native-claude-code-toggle.test.ts tests/claude-management-api.test.ts -bun test tests/claude-messages-endpoint.test.ts +bun test tests/codex-desired-state.test.ts +bun test tests/codex-desired-state-race.test.ts +bun test tests/codex-sync-api.test.ts tests/codex-inject-integration.test.ts +bun test tests/codex-journal.test.ts tests/service.test.ts tests/uninstall.test.ts tests/server-auth.test.ts bun run typecheck bun run test bun run privacy:scan ``` -Live activation proof: - -1. Record `/healthz` and PID. -2. Disable Grok and OpenCode through their real management routes. Confirm each - status has `desiredEnabled:false` and honest observed state. -3. Run `ocx ensure`, restart the proxy, and read the target files. Neither managed - contribution reappears; `/healthz` returns with the proxy still serving. -4. Disable OpenCode, run `ocx opencode`, and observe refusal before ensure/spawn. -5. With a legacy-only `claudeCode.enabled=false`, call Messages, count-tokens, and - Anthropic discovery. Observe 403, 403, and an empty model list. Then call an - invalid `/v1/responses` request and observe its normal validation response, - never a client-disabled response. -6. Inject a post-persist crash for one file client, restart, and observe the - remover converge it to absent. Repeat with drift and observe the unresolved - conflict while desired remains OFF. +Live proof is the subprocess case in `tests/codex-desired-state-race.test.ts`, not +the installed `ocx`: it launches the repository CLI with isolated homes and an +ephemeral non-10100 port and records PID and `/healthz`. First it commits OFF and +converges while the process stays alive. In a separate run it kills at the +post-persist/pre-remove seam and relaunches from the same isolated home. It proves: + +1. `/healthz` returns from the same PID after the live OFF mutation; disabling + native Codex did not stop or replace the proxy. +2. native Codex routing/profile/catalog residue converges to absent after restart. +3. an invalid `/v1/responses` request reaches its normal validation response, not + a desired-state gate. +4. repeating from a foreign `OPENCODEX_HOME` leaves every Codex artifact byte-exact + and reports `home_mismatch`. + +The test must print the isolated roots, chosen port, before/after hashes, and +reconciliation result so the C-phase evidence proves the live path rather than +only a mocked helper. It must tear down only its recorded subprocess and temp +directory. ## Accept criteria -| Roadmap criterion | WP3 closure | +| Roadmap criterion | WP4 closure | |---|---| -| C2 — disabled survives restart, ensure, and `/api/sync` | Every real switch writes intent; every automatic writer reads it; startup/ensure/status converge residual applied state. | -| C3 — absent config changes nothing on upgrade | Missing map/key is ON for all ten clients, except legacy Claude explicit OFF remains OFF. | -| C4 — disable never stops proxy or another client | No lifecycle operation is added; `/v1/responses` stays ungated; Claude's own ingress admission remains gated; every other client is governed only by its own key. | -| Coordination | `mutatePersistedConfig` prevents stale whole-object saves, the per-client flight orders every surface, and each writer re-reads intent immediately before commit. | -| Shared contract | Native union is `codex | claude | claude-desktop | grok`; status/success always include `desiredEnabled`; WP5 then WP6 consume it sequentially. | - -WP3 is complete only when desired and observed state can disagree honestly and -the system keeps trying to reconcile that disagreement. A removed file with no -persisted intent is still the shipped Grok bug. A persisted OFF reported as -observed OFF while bytes remain is a new lie. A legacy Claude OFF that accepts -traffic again is a compatibility regression. +| C2 — Codex stays disabled across restart, ensure, and `/api/sync` | OFF is persisted through `mutatePersistedConfig`; startup/ensure converge residual apply state; sync and direct catalog refresh re-read OFF before writing; `/api/sync` reports the skip honestly. | +| C3 — absent config changes nothing on upgrade | Missing object/key remains ON. The one-key parser and no-write upgrade test prove existing installs keep current behavior. | +| C4 — disabling Codex never stops proxy or closes `/v1/responses` | No lifecycle or transport gate is added. The isolated live proof keeps `/healthz` and the Responses route reachable while native Codex state is removed. | +| C7 — foreign-home startup touches nothing | Every production remover runs `assertNativeTeardownOwned` first; the foreign-home crash-recovery fixture proves byte-exact refusal with desired OFF preserved. | + +WP4 is complete only when an operation that began ON can resume after OFF and +still perform zero Codex writes, and when a restart can finish an interrupted OFF +without touching a foreign service's state. A boolean without those two proofs is +the failed first draft in a smaller file. From b8350d92ac8264384f02788f7347e95f927517c9 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Tue, 4 Aug 2026 02:15:38 +0900 Subject: [PATCH 021/163] =?UTF-8?q?docs(integrations):=20close=20the=20WP4?= =?UTF-8?q?=20audit=20=E2=80=94=20the=20re-scope=20held,=20Codex=20is=20ju?= =?UTF-8?q?st=20concurrent?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four High, all accepted. But the shape of this round is different from the two that failed before it: the reviewer independently confirmed the doc is Codex-only, the flag is extension-safe, and the auth-sentinel claim holds even under a second key. Three rounds of coupling findings vanished in one pass, so the 006 diagnosis was right. What remains is all Codex. #1 and #2 are one defect: I designed a check where I needed a lock. Re-reading intent before a write narrows a window rather than closing it, and my own test would have passed while the bug was live because it flips OFF before the check instead of between check and write. The ownership preflight fails open by design, which is defensible for a route a human is watching and wrong for unattended startup convergence — and my ordering put both journal repair and my own lock file ahead of the check that was supposed to protect them. #3 is an artifact I did not know existed: restoreNativeCodex never touches models_cache.json, so a converged OFF can still advertise routed models. Same shape as the WP2 bug — invisible to a test asserting only the artifacts it already knew about. #4 is the false-green class again, one phase after I wrote a section about it: I gated syncModelsToCodex under ocx restore back without deciding what that documented reverse switch does when Codex is durably OFF. --- .../007_audit_synthesis_wp4.md | 103 ++++++++++++++++++ 1 file changed, 103 insertions(+) create mode 100644 devlog/_plan/260803_codex_desktop_toggle/007_audit_synthesis_wp4.md diff --git a/devlog/_plan/260803_codex_desktop_toggle/007_audit_synthesis_wp4.md b/devlog/_plan/260803_codex_desktop_toggle/007_audit_synthesis_wp4.md new file mode 100644 index 000000000..105bf1207 --- /dev/null +++ b/devlog/_plan/260803_codex_desktop_toggle/007_audit_synthesis_wp4.md @@ -0,0 +1,103 @@ +# WP4 audit — synthesis + +Verdict: **FAIL**, 4 High blocking plus 3 Medium. Fresh reviewer, auditing the +re-scoped Codex-only doc. + +## What the re-scope bought + +The reviewer confirmed, by independent grep and trace, the things the previous +two rounds kept failing on: + +- **It really is Codex-only.** No surviving non-Codex gate, writer, or type + change; the Claude/Grok/Desktop/file-client mentions are exclusions or the + auth-sentinel proof. +- **The flag shape is extension-safe** and absent-key-means-ON holds against the + real load/save path. +- **The auth-sentinel claim holds**, and holds even when a second key is added + under `clientIntegrations`. + +So the diagnosis in `006` was right: slicing by ownership fixed the divergence. +Three rounds of findings about coupling are gone in one pass. + +What replaced them is narrower and, unlike the previous rounds, all of it is +about **Codex itself**. + +## The one defect behind #1 and #2 + +I designed a check where I needed a lock. + +`#1`: my "re-read intent immediately before the write" closes a window, it does +not eliminate one. An apply reads ON, another process commits OFF through the +*separate* config lock, and the apply proceeds into `atomicWriteFile`. Every +check/write pair has that gap, including the several writes hidden inside one +history callback. My own test would have passed while the bug was live, because +it flips OFF *before* the check rather than between check and write. + +`#2`: the ownership preflight fails open by design +(`ownership-preflight.ts:21`), and `readServiceInstallState` returns `null` for +corrupt, unreadable, and missing-mirror states alike (`service.ts:165`). That is +defensible for an interactive route where a human reads a refusal. It is wrong +for **unattended startup convergence**, where fail-open means "remove a foreign +home's Codex state and tell nobody". Worse, my ordering was wrong twice over: +startup runs `reconcileJournal` before my preflight, and my own flight database +would be created under `$CODEX_HOME` before the preflight runs — so the +"byte-exact refusal" claim is already false by the time the check happens. + +**Accept both.** The replacement: + +- one per-`CODEX_HOME` linearization lock covering **both** desired-state commits + and native commit sections, with model gathering left outside it +- a tri-state ownership answer — `owned | foreign | unknown` — where automatic + convergence fails **closed** on `foreign` and `unknown` +- ownership resolved before journal repair and before any lock artifact exists, + with the lock stored outside `CODEX_HOME`, keyed by canonical-path hash + +## #3 — the artifact I did not know about + +`restoreNativeCodex` restores config, profile, catalog and history. It never +touches `models_cache.json`, and apply writes routed data there +(`catalog/sync.ts:600`). So a converged OFF can report success while native +Codex still advertises routed models from the integration we just disabled. + +This is the same shape as the WP2 bug: a state that only shows up in the real +artifact, invisible to a test that asserts the artifacts it already knew about. +Cache restoration joins the remover and the observed-state inspection. + +## #4 — the command I gated without deciding what it means + +`ocx restore back` is the documented reverse switch (`cli/help.ts:18`). I gated +`syncModelsToCodex` beneath it and never said what the command does when Codex is +durably OFF — so it either cannot perform its documented job, or prints "now +routes through opencodex" after writing nothing. That is the false-green class +again, one phase after I wrote a doc section about it. + +Resolution, adopting the reviewer's split: + +| Path | Behavior while OFF | +|---|---| +| `POST /api/sync` | 409; an automatic-ish surface does not override intent | +| `ocx sync` | refuse, naming the switch | +| `ocx restore back` | **explicit enable**: persist ON atomically, then apply | +| `ocx init` | may establish ON — it writes a fresh config and asks separately | + +And a skipped sync returns `ok: false` with a `skippedReason`, not a bare `ok` +that every caller reads as success. + +## Medium findings + +- **#5** `mutatePersistedConfig` **throws** `ConfigMutationLockError` on lock + contention; `unavailable` means rebase exhaustion. I documented three outcomes + and missed the thrown branch, so a route following my instructions leaks a 500. +- **#6** two processes sharing `OPENCODEX_HOME` with different `CODEX_HOME`s can + both write the legacy `catalog-backup.json`. The hashed backup usually masks + it, which is exactly why it needs a test rather than a comment. +- **#7** SOT-SYNC: `docs-site` lifecycle pages still promise unconditional + sync/restore-back. A user-visible command semantics change has to land there. + +All accepted. + +## Next + +A is a loop; a FAIL never exits it. Amendments go into `030`, then the SAME +reviewer re-audits. Nothing about the phase map changes — this round produced no +evidence of divergence, only of Codex being genuinely concurrent. From 5eaa71f73062a3525929a3d316302d10797cecf7 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Tue, 4 Aug 2026 02:28:54 +0900 Subject: [PATCH 022/163] docs(integrations): give WP4 a lock where it had a check MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit D1: re-reading intent before a write narrows the race rather than closing it, and the test I wrote for it flipped OFF before the check instead of between check and write, so it would have passed while the bug was live. Replaced with one per-CODEX_HOME linearization lock covering both intent commits and native write sections, model gathering left outside it, and deterministic seams after authority approval so the race is testable at the point it actually exists. D2: the ownership preflight fails open, and readServiceInstallState collapses corrupt, unreadable and missing-mirror into the same null. Acceptable where a human reads the refusal; wrong for unattended startup convergence. Now a tri-state owned|foreign|unknown that fails closed on the latter two, resolved before journal repair and before any lock artifact exists, with the lock moved outside CODEX_HOME. D3: restoreNativeCodex never touched models_cache.json, so a converged OFF could still advertise routed models. Cache restoration joins the remover and the observed-state read, with crash fixtures at the cache boundary. D4: ocx restore back is an explicit enable verb — it persists ON and applies, while api/sync 409s and ocx sync refuses by name. A skipped sync returns ok:false with a reason instead of an ok every caller reads as success. D5-D7: the thrown lock-contention branch, the shared legacy catalog backup across two CODEX_HOMEs, and the docs-site lifecycle pages this phase changes. --- .../030_desired_state.md | 498 ++++++++++++------ 1 file changed, 339 insertions(+), 159 deletions(-) diff --git a/devlog/_plan/260803_codex_desktop_toggle/030_desired_state.md b/devlog/_plan/260803_codex_desktop_toggle/030_desired_state.md index 066585dca..977157f8c 100644 --- a/devlog/_plan/260803_codex_desktop_toggle/030_desired_state.md +++ b/devlog/_plan/260803_codex_desktop_toggle/030_desired_state.md @@ -12,6 +12,15 @@ reconciliation called the native remover without checking service ownership, so a start from a different `OPENCODEX_HOME` could strip Codex state used by the installed service (`006_audit_synthesis_r2.md`, round 2 #2). +Audit round 4 found the same incident still possible in the proposed fix +(`007_audit_synthesis_wp4.md`, #1-#3). A fresh authority read was only a check, +not a lock: OFF could commit through the separate config lock after an apply read +ON and before `atomicWriteFile`. Startup also repaired the journal before proving +ownership, and the proposed lock created a file inside the foreign `CODEX_HOME` +before that proof. Finally, restore omitted `models_cache.json`, so a reported OFF +could still advertise routed slugs. This revision replaces those mechanisms +plainly; it does not describe the prior re-read design as sufficient. + That incident decides the scope. The two phases that shipped cleanly each changed one thing at one boundary (`010_modality_boundary.md`, `020_api_keys_row.md`). WP4 therefore changes desired state for **Codex only**. It does not establish a @@ -33,36 +42,46 @@ Already present: - crash-journal reconciliation already repairs an abandoned injection (`src/codex/journal.ts:148-162`). -WP4 adds one persisted Codex flag, one Codex write coordinator, last-moment -persisted-state checks for automatic apply writes, and OFF reconciliation at start -and ensure. WP5 adds the management route and GUI switch that call the writer; -WP4 does not define their response schema. +WP4 adds one persisted Codex flag and one per-`CODEX_HOME` linearization lock that +covers both desired-state commits and bounded native commit sections. Provider +model gathering stays outside the lock. Ownership is tri-state and is resolved +before journal repair or lock creation, then rechecked inside the lock. OFF +reconciliation restores config, profile, catalog, cache, and history at start and +ensure. WP5 adds the management route and GUI switch that call the writer; WP4 +defines the writer failures WP5 must map but not WP5's full response schema. ## IN / OUT | Path | Change | Why it is in WP4 | |---|---|---| | `src/types.ts` | MODIFY | Adds a one-key `OcxClientIntegrationsConfig` and its optional `OcxConfig.clientIntegrations` home. | -| `src/config.ts` | MODIFY | Parses the Codex key, resolves absent as ON, re-reads persisted intent, and mutates only that field through the real `mutatePersistedConfig` signature. | -| `src/codex/desired-state.ts` | NEW | Owns the Codex-only process/OS write flight, last-moment authority checks, owned restore wrapper, and OFF reconciliation. | -| `src/codex/sync.ts` | MODIFY | Admits automatic Codex sync only while desired ON and passes a fresh-write authority through catalog and injection. | -| `src/codex/refresh.ts` | MODIFY | Carries the authority to the direct catalog/cache path used outside `syncModelsToCodex`. | -| `src/codex/catalog/sync.ts` | MODIFY | Re-reads desired ON after model gathering and immediately before catalog/cache replacement. | -| `src/codex/catalog/bundled.ts` | MODIFY | Prevents fallback catalog materialization before a fresh desired-ON check. | -| `src/codex/catalog/parsing.ts` | MODIFY | Calls the authority separately before each pristine-backup copy/write. | -| `src/codex/inject.ts` | MODIFY | Re-checks desired ON at the injection commit boundaries and makes the unchecked remover internal to the owned wrapper. | +| `src/config.ts` | MODIFY | Parses the Codex key, resolves absent as ON, and mutates only that field through `mutatePersistedConfig`; documents its thrown lock branch. | +| `src/codex/desired-state.ts` | NEW | Owns the external per-home linearization lock, test seams, tri-state ownership gate, owned restore wrapper, observed-state inspection, and OFF reconciliation. | +| `src/codex/sync.ts` | MODIFY | Separates model gathering from the bounded apply commit and returns `ok:false` plus `skippedReason` for every no-write result. | +| `src/codex/refresh.ts` | MODIFY | Splits gathered catalog data from the bounded catalog/cache commit used outside `syncModelsToCodex`. | +| `src/codex/catalog/sync.ts` | MODIFY | Commits catalog/cache only while holding the shared linearization lock; restores or invalidates `models_cache.json` during native removal. | +| `src/codex/catalog/bundled.ts` | MODIFY | Moves fallback materialization into the bounded native commit section. | +| `src/codex/catalog/parsing.ts` | MODIFY | Stops replacing the shared legacy backup after a target-hashed backup exists. | +| `src/codex/inject.ts` | MODIFY | Moves every config/profile/journal/history write into one bounded commit callback and makes the unchecked remover internal to the owned wrapper. | | `src/server/management-api.ts` | MODIFY | Gives provider/model/combo refreshes their own Codex gate and routes `/api/stop` through the owned remover. | | `src/server/management/config-routes.ts` | MODIFY | Makes `POST /api/sync` report an intentional desired-OFF skip instead of false success. | -| `src/cli/index.ts` | MODIFY | Reconciles OFF after journal repair, explains start/ensure skips, and routes every CLI/shutdown remover through the owned flight. | -| `src/cli/init.ts` | MODIFY | Uses the write flight without turning explicit bootstrap into an automatic desired-state gate. | -| `src/service.ts` | MODIFY | Routes service stop/uninstall removers through the same owned flight without changing desired state. | +| `src/cli/index.ts` | MODIFY | Resolves ownership before journal repair, reconciles OFF, refuses explicit sync while OFF, makes `restore back` an explicit enable, and routes every remover through the owned lock. | +| `src/cli/init.ts` | MODIFY | May establish ON after its separate injection prompt, then uses the same linearized apply operation. | +| `src/cli/provider.ts`, `src/cli/models.ts` | MODIFY | Inspect typed sync skips so provider/model mutations do not claim Codex refresh success while OFF. | +| `src/service.ts` | MODIFY | Exposes tri-state ownership diagnostics and routes service stop/uninstall removers through the same per-home lock without changing desired state. | | `tests/codex-desired-state.test.ts` | NEW | Pins schema defaulting, field-scoped persistence, auth-sentinel isolation, and unavailable/conflict behavior. | -| `tests/codex-desired-state-race.test.ts` | NEW | Pins in-flight OFF, crash-point convergence, single-flight, and foreign-home refusal. | +| `tests/codex-desired-state-race.test.ts` | NEW | Pins post-approval OFF, crash-point convergence, linearization, and foreign/unknown refusal. | | `tests/codex-sync-api.test.ts` | MODIFY | Pins sync and `POST /api/sync` OFF semantics. | -| `tests/codex-inject-integration.test.ts` | MODIFY | Pins guarded commit boundaries and the owned native remover. | +| `tests/codex-inject-integration.test.ts` | MODIFY | Pins locked commit boundaries and the owned native remover. | | `tests/codex-journal.test.ts` | MODIFY | Proves journal repair still runs while desired Codex state is OFF. | +| `tests/codex-catalog-restore.test.ts`, `tests/codex-models-cache-invalidate.test.ts` | MODIFY | Pin cache cleanup and two-home backup isolation. | +| `tests/cli-restore-back.test.ts`, `tests/cli-provider.test.ts`, `tests/startup-prompt.test.ts` | MODIFY | Pin every CLI caller's OFF or explicit-enable meaning. | +| `tests/cli-models-desired-state.test.ts` | NEW | Pins custom-model save plus honest Codex-refresh skip while OFF. | +| `tests/cli-init-desired-state.test.ts` | NEW | Pins fresh init plus affirmative/negative injection prompt semantics. | | `tests/service.test.ts`, `tests/uninstall.test.ts` | MODIFY | Proves owned teardown remains unconditional with respect to desired ON/OFF. | | `tests/server-auth.test.ts` | MODIFY | Proves Codex OFF does not gate the shared `/v1/responses` transport. | +| `docs-site/src/content/docs/reference/cli/lifecycle.md` | MODIFY | Documents start/ensure/sync OFF behavior and `restore back` as explicit enable. | +| `docs-site/src/content/docs/reference/configuration.md` | MODIFY | Documents `clientIntegrations.codex`, absent-means-ON, and desired versus observed state. | OUT, deliberately: @@ -73,10 +92,15 @@ OUT, deliberately: | `src/grok/**`, Grok routes | **Moved to WP6.** Grok will add its own key and prove its own callers after the Codex shape passes. | | `src/claude/desktop-3p.ts`, Desktop routes | **Moved to WP7.** Desktop keeps its separate ownership/profile questions. | | `src/server/management/native-integration-routes.ts`, `gui/` | **Moved to WP5.** WP5 owns the Codex route, GUI parser, and UI contract. WP4 does not define a `codex | claude | claude-desktop | grok` union or `desiredEnabled` response schema. | -| desired-state admission in `ocx init` | **Not added. INFERRED:** `ocx init` is a user-commanded setup operation, not one of the automatic re-apply paths named by this phase. Its direct injection still uses the Codex write flight so it cannot overlap another irreversible Codex write. | +| desired-OFF refusal in `ocx init` | **Not added. INFERRED:** init writes a fresh config and asks separately before injection, so accepting that prompt may establish Codex ON. It still participates in the same linearization lock. | | `/v1/responses` | Never gated. It is a shared transport used by clients other than native Codex. | | releases, publishing, deploys, tags, repository starring | No delivery or identity action belongs in this phase. | +English lifecycle and configuration pages are canonical (`docs-site/AGENTS.md:5-10`). +The corresponding `ko`, `ja`, `zh-cn`, and `ru` pages must be updated in the same +implementation phase or left without a contradictory promise; unconditional +start/ensure/sync/restore-back wording may not survive in a translated locale. + ## The flag: one key in an extension-safe object Use a map-shaped object with **one key today**, not a top-level `codexEnabled` @@ -170,7 +194,7 @@ Build the Codex writer directly on it after `websocketsEnabled` + desiredEnabled: boolean; +} + -+export function mutateCodexDesiredEnabled( ++export function mutateCodexDesiredEnabledUnlocked( + enabled: boolean, +): PersistedConfigMutationOutcome { + return mutatePersistedConfig(config => { @@ -183,11 +207,34 @@ Build the Codex writer directly on it after `websocketsEnabled` +} ``` +The `Unlocked` suffix is deliberate: only `src/codex/desired-state.ts` may import +this primitive, and a source-shape assertion enforces that ownership. All +production callers use the locked setter below. Exporting a friendly-looking raw +setter would let the next caller recreate the separate-lock race this audit found. + Only the callback-local clone is mutated. A future WP5 route must use `outcome.value.config` after `committed | unchanged`; it must never patch the long-lived management `config` before persistence succeeds. `missing`, `invalid`, and `conflict` leave both disk and the supplied live object unchanged. +The prior draft listed only the return union. That was insufficient because +`withConfigMutationLockSync` throws `ConfigMutationLockError` when SQLite lock +acquisition fails (`src/config.ts:1768-1793`); `unavailable` is the later +missing/invalid/rebase-exhaustion result (`src/config.ts:1830-1834,1859-1906`), +not contention. The Codex setter and every route/CLI caller handle four paths: + +| Mutation result | Contract | +|---|---| +| `committed` | Copy `outcome.value.config` into the live object and continue. | +| `unchanged` | Copy the freshly loaded `outcome.value.config` into the live object and continue without a second native write. | +| `unavailable` | Non-retryable desired-state refusal (`missing`, `invalid`, or `conflict`); disk and live object stay unchanged. | +| thrown `ConfigMutationLockError` with `cause.code === "SQLITE_BUSY"` | Retryable busy: HTTP 409 `config_busy` (503 is also acceptable at a service boundary); CLI exits nonzero and says retry. | +| thrown `ConfigMutationLockError` with any other cause | Non-retryable `write_failed`; HTTP 500 and CLI exits nonzero. | + +This follows the existing distinction at +`src/server/management/native-integration-routes.ts:144-161`, where mapping the +whole exception class to retryable would lie about an unopenable database. + Round 2 #1 is **unreachable for this Codex-only shape**. The mutation above writes only the sibling `clientIntegrations` object. It never reads, spreads, creates, or assigns `config.claudeCode`. `runClaudeAuthModeMigration` returns immediately when @@ -198,129 +245,150 @@ real migration after OFF and ON mutations and asserts: no `claudeCode` block, `runClaudeAuthModeMigration(...) === false`, and no persisted `authMode` or `authModeMigratedAt`. -## One Codex write flight, with a last-moment authority read +## One per-home linearization lock — a check is not a lock -Entry checks do not close the race. `syncModelsToCodex` can pause in provider -model gathering (`src/codex/sync.ts:83-108`), while another process persists OFF, -then continue into injection at `src/codex/sync.ts:110`. +The previous design's bare re-read was insufficient. `syncModelsToCodex` can pause +in provider model gathering (`src/codex/sync.ts:83-108`), and even a re-read after +that pause leaves a check/write gap: another process can commit OFF through the +separate config mutation transaction before the apply reaches +`atomicWriteFile`. The several config/profile/journal/history writes at +`src/codex/inject.ts:524-603` have the same defect. -NEW `src/codex/desired-state.ts` owns one Codex coordinator: +NEW `src/codex/desired-state.ts` owns one linearization boundary: ```ts -export type CodexWriteDirection = "apply" | "remove"; +export type NativeCodexOwnership = + | { state: "owned" } + | { state: "foreign"; message: string } + | { state: "unknown"; message: string }; + +export type CodexNativeWriteBoundary = + | "journal" | "config" | "profile" | "journal-injected" | "history" + | "catalog-backup" | "catalog" | "models-cache" | "remove"; -export type PersistedCodexAuthority = - | { ok: true; config: OcxConfig } - | { ok: false; reason: "desired_state_changed" | "desired_state_unavailable" }; +export type CodexDesiredMutationResult = + | { ok: true; status: "committed" | "unchanged"; config: OcxConfig } + | { ok: false; reason: "missing" | "invalid" | "conflict" | "config_busy" | "write_failed"; + retryable: boolean; message: string }; export interface CodexReconcileResult { trigger: "startup" | "ensure"; desiredEnabled: boolean; observedState: "absent" | "applied" | "conflict" | "unavailable"; resolved: boolean; - reason?: "home_mismatch" | "history_locked" | "write_failed" | "codex_write_busy"; + reason?: "home_mismatch" | "ownership_unknown" | "history_locked" + | "write_failed" | "codex_write_busy"; message: string; } -export function requirePersistedCodexIntent( - expectedEnabled: boolean, -): PersistedCodexAuthority; - -export async function runCodexWriteFlight( - direction: CodexWriteDirection, - operation: () => Promise, -): Promise; - -export function runCodexWriteFlightSync( - direction: CodexWriteDirection, - operation: () => T, -): T; - +export function inspectNativeCodexOwnership(): NativeCodexOwnership; +export function withCodexHomeLinearizationLockSync(operation: () => T): T; +export function setCodexDesiredEnabled(enabled: boolean): CodexDesiredMutationResult; +export function setCodexBeforeNativeWriteForTests( + hook: ((boundary: CodexNativeWriteBoundary) => void) | null, +): void; export function restoreNativeCodexOwned(): { success: boolean; message: string }; - -export async function reconcileCodexDesiredState( +export function reconcileCodexDesiredState( trigger: "startup" | "ensure", -): Promise; +): CodexReconcileResult; ``` -**INFERRED design choice:** the coordinator has one in-process tail and one -OS-backed SQLite transaction at -`getCodexHome()/opencodex-write.sqlite` (`src/codex/paths.ts:32-35`). The lock is -keyed by the native target, not `OPENCODEX_HOME`: two OpenCodex homes can point at -the same `CODEX_HOME`, and they must not acquire different locks for the same -files. The config mutation lock is deliberately not reused: holding it across -model fetch would prevent OFF from being persisted, which is the race this phase -must handle. A second process waits with a bounded timeout; timeout returns -`codex_write_busy` and writes nothing. The sync form exists for -shutdown/`process.on("exit")`, where a Promise cannot be awaited. - -`requirePersistedCodexIntent` uses `readConfigDiagnostics` -(`src/config.ts:1691-1708`), not a request's captured config. A missing or invalid -file is unavailable at a write boundary and fails closed; it is not reinterpreted -as an upgrade-time ON after an operation has already begun. - -Every automatic apply path enters `runCodexWriteFlight("apply", ...)`, and the -authority is re-read after its last await and immediately before each commit -boundary. `InjectCodexOptions` and the catalog helpers receive a -`beforeWrite(boundary)` callback; they call it again for every separate file or DB -write rather than treating several writes as one group: - -| Boundary | Current write | WP4 check | +**INFERRED design choice:** canonicalize the effective `CODEX_HOME`, hash that +canonical path with SHA-256, and store the SQLite lock at +`join(tmpdir(), "opencodex-native-locks", + ".sqlite")`, mode `0600` in a +mode-`0700` directory. It is outside `CODEX_HOME` and independent of +`OPENCODEX_HOME`, so two OpenCodex homes targeting one native home serialize on +one lock without writing a lock artifact into the target. Process exit releases +the SQLite transaction. The callback is synchronous and bounded: there is no +provider fetch, model discovery, sleep, or other `await` while it is held. + +The ordering invariant is: + +1. Inspect ownership **before** journal repair and before resolving/creating the + lock path. `foreign` and `unknown` fail closed for startup/ensure and create no + lock file. No installed service is `owned`; an installed service with no valid + mirror is `unknown`. +2. Gather provider models and calculate candidate bytes outside the lock. +3. Acquire the per-home lock, inspect ownership again inside it, then fresh-read + desired intent. A changed/invalid intent aborts with no native write. +4. Perform the bounded native commit while still holding the lock. Every desired + ON/OFF setter acquires this **same lock before committing intent**, so OFF + cannot linearize between authority approval and a native write. +5. Release the lock before logging, app-server handling, or any network work. + +`inspectNativeCodexOwnership` replaces the automatic route's use of +`assertNativeTeardownOwned`. The existing helper fails open on unrelated errors +(`src/integrations/native/ownership-preflight.ts:21-35`), while +`readServiceInstallState` returns one `null` for corrupt, unreadable, and absent +mirrors (`src/service.ts:165-175`). MODIFY `src/service.ts` to expose a diagnostic +read that distinguishes: no installed service (`owned`), valid same-home state +(`owned`), valid different-home state (`foreign`), and an installed service whose +state is corrupt, unreadable, or missing (`unknown`). Interactive teardown may +retain its current human-facing policy; unattended convergence may not. + +The authority check is once per locked commit, not a series of unlocked checks. +The test seam fires **after** ownership and desired-state approval and immediately +before every irreversible write below, while the same lock is still held: + +| Boundary | Current write | WP4 locked commit | |---|---|---| -| bundled fallback | `materializeBundledCodexCatalog` at `src/codex/catalog/bundled.ts:213-219` | pass the authority into `loadCatalogForSync`; re-read before fallback materialization | -| pristine backups | `copyFileSync` / `atomicWriteFile` at `src/codex/catalog/parsing.ts:428-444` | re-read separately inside `writePristineCatalogBackup` before each backup copy/write | -| catalog | `atomicWriteFile(catalogPath, ...)` at `src/codex/catalog/sync.ts:568` | `requirePersistedCodexIntent(true)` after `gatherRoutedModels` and directly before replace | -| models cache | `atomicWriteFile(activeCodexModelsCachePath(), ...)` at `src/codex/catalog/sync.ts:600-613` | re-read before cache replacement | -| injection journal | `writeJournal(...)` at `src/codex/inject.ts:521-527` | re-read before recording an apply transaction | -| config | first atomic write at `src/codex/inject.ts:593-596` | re-read immediately before `CODEX_CONFIG_PATH` replacement | -| profile | second atomic write at `src/codex/inject.ts:595-597` | re-read again immediately before `CODEX_PROFILE_PATH` replacement | -| journal injected marker | `markJournalInjectedState(...)` at `src/codex/inject.ts:597` | re-read again before advancing journal state | -| history mutation | `syncCodexHistoryProvider` / `migrateHistoryToOpenai` at `src/codex/inject.ts:598-603` | re-read before the DB mutation | -| native remove | `restoreNativeCodex` body at `src/codex/inject.ts:764-795` | ownership preflight first; startup reconciliation also re-reads OFF immediately before remove | +| bundled fallback | `materializeBundledCodexCatalog` at `src/codex/catalog/bundled.ts:213-219` | candidate resolution outside; materialization inside lock | +| pristine backup | `copyFileSync` / `atomicWriteFile` at `src/codex/catalog/parsing.ts:428-444` | each target-hashed backup write inside lock | +| catalog | `atomicWriteFile(catalogPath, ...)` at `src/codex/catalog/sync.ts:568` | gathered candidate committed inside lock | +| models cache | `atomicWriteFile(activeCodexModelsCachePath(), ...)` at `src/codex/catalog/sync.ts:600-613` | replacement/restoration inside lock | +| injection journal | `writeJournal(...)` at `src/codex/inject.ts:521-527` | inside lock | +| config/profile/marker | writes at `src/codex/inject.ts:593-597` | all inside the same lock | +| history mutation | callback at `src/codex/inject.ts:598-603` | the complete callback, including its hidden writes, inside lock | +| native remove | `restoreNativeCodex` body at `src/codex/inject.ts:764-795` | ownership and OFF rechecked, then complete remove inside lock | + +At the deterministic seam, start an OFF setter in a second process and prove its +intent commit cannot complete until the held apply write exits. Then release the +seam, let apply linearize, let OFF linearize next and remove it, and assert the +final disk state is OFF. The existing “OFF during model fetch” case remains and +must still produce zero writes because OFF commits before lock admission. These +are different races; the old fetch-only test would pass while the check/write bug +was live. `src/codex/inject.ts` renames the raw remover to `restoreNativeCodexUnchecked`; only `src/codex/desired-state.ts` may import it. -All production callers import `restoreNativeCodexOwned` instead. A source-shape -test rejects any other import of the unchecked symbol. This makes the round 2 #2 -preflight an owned boundary rather than a convention each caller can forget. - -Stop, uninstall, and explicit restore use the owned remover but **do not require -desired OFF** and never rewrite the flag. They are safety teardown, not user-intent -mutation (`src/service.ts:2587-2594`). Startup/ensure reconciliation uses the same -remover with the additional fresh-OFF check. - -`src/service.ts` must not statically import the new wrapper. The wrapper imports -`assertNativeTeardownOwned`, whose current implementation imports `service.ts` -(`src/integrations/native/ownership-preflight.ts:14-17`); a static reverse import -would create `service -> desired-state -> ownership-preflight -> service`. Remove -the current static raw-remover import at `src/service.ts:15` and dynamically import -`restoreNativeCodexOwned` inside the already-async stop/uninstall branches before -calling it. **INFERRED:** this is the smallest way to keep the shipped preflight as -the authority without broadening WP4 into a service-ownership module extraction. +Stop, uninstall, shutdown, and explicit `ocx restore` use the owned lock but do +not require desired OFF and never rewrite intent. `src/service.ts` dynamically +imports the wrapper in its async stop/uninstall branches to avoid the existing +`service -> desired-state -> ownership-preflight -> service` cycle +(`src/integrations/native/ownership-preflight.ts:14-17`). ## Automatic Codex gates -### Normal sync path +### Normal sync path and explicit CLI meanings -MODIFY `src/codex/sync.ts:49-55` so the entry gate avoids unnecessary fetches and -the write flight covers catalog plus injection as one Codex operation: +A skipped operation is not successful. MODIFY `src/codex/sync.ts:9-22,49-55` so +the entry gate avoids unnecessary fetches and every no-write result is typed: ```diff + export interface CodexSyncResult { + ok: boolean; +@@ ++ skippedReason?: "desired-off" | "desired-state-unavailable" | "codex-write-busy"; + } +@@ export async function syncModelsToCodex( port?: number, config: OcxConfig = loadConfig(), log: Pick | null = console, deps: CodexSyncDeps = defaultDeps, ): Promise { -+ if (!codexDesiredEnabled(config)) return codexDesiredOffSyncResult(log); -+ return runCodexWriteFlight("apply", async () => { ++ if (!codexDesiredEnabled(config)) { ++ return { ok: false, added: 0, catalogPath: null, catalogExists: false, ++ catalogWritten: false, cacheSynced: false, ++ skippedReason: "desired-off", message: "Codex integration is OFF." }; ++ } const p = port ?? config.port ?? 10100; ``` -Close the flight after the existing return at `src/codex/sync.ts:114-129`. -`CodexSyncResult` adds optional `skippedReason: "desired-off" | -"desired-state-unavailable" | "codex-write-busy"`. Desired OFF is an intentional -no-write result, not a claim that catalog/injection completed. +The function gathers the candidate model/catalog state outside the lock, then +calls the locked commit described above. Desired OFF, unavailable authority, and +lock timeout all return `ok:false` plus `skippedReason`; no caller may infer success +from a bare resolved Promise. `ocx start` and both `ocx ensure` branches remain callers at `src/cli/index.ts:318-320,358-411`; they inspect `skippedReason` and print @@ -328,12 +396,29 @@ no-write result, not a claim that catalog/injection completed. proxy, alter its port, or skip another client's setup. `POST /api/sync` at `src/server/management/config-routes.ts:261-268` returns a -409 `codex_desired_off` envelope when the sync result says desired OFF. It must -not return the current 200-shaped success for an operation that intentionally -wrote nothing. **INFERRED:** 409 distinguishes a valid request blocked by current -desired state from a server fault; 200 would preserve the false-green finding and -500 would misclassify an intentional policy decision. Other sync failures keep -their existing 500 behavior. +409 `codex_desired_off` envelope with `ok:false` and the `skippedReason` when OFF. +It must not return the current success-shaped body for an operation that wrote +nothing. **INFERRED:** 409 distinguishes a valid request blocked by current intent +from a server fault; 200 preserves the false-green incident and 500 misclassifies +policy. Busy is retryable 409/503; non-contention write failure remains 500. + +The command split is explicit, because the old design gated the shared backend +without defining its callers: + +| Caller | Behavior while durably OFF | +|---|---| +| `ocx start`, both `ocx ensure` branches | Start/keep the proxy, print one auto-apply skip, write no Codex artifact. | +| `POST /api/sync` | 409 `codex_desired_off`, `ok:false`, `skippedReason:"desired-off"`. | +| `ocx sync` | Exit nonzero with `Codex integration is OFF. Enable it with 'ocx restore back' or the Codex integration switch.` | +| `ocx provider ... --sync`, custom-model add/remove | Preserve their primary config mutation, report Codex refresh skipped with the same actionable switch text, and perform no Codex write (`src/cli/provider.ts:235`, `src/cli/models.ts:105`). | +| `ocx restore back` / `ocx eject back` | **Explicit enable verb:** require a live proxy, gather models, acquire the shared lock, persist ON, then apply before releasing it. | +| `ocx init` | After writing a fresh config, an affirmative injection answer may establish ON and apply under the shared lock (`src/cli/init.ts:176-198`). | + +“Atomically persists ON then applies” means one linearized operation, not rollback +of config intent if a later native file fails: ON commits first under the lock; +apply failure returns nonzero and the next convergence may retry. It never prints +the current success line at `src/cli/index.ts:762-763` unless both the ON commit +and bounded apply succeed. ### Provider/model/combo refresh bypass @@ -359,50 +444,125 @@ The management helper currently calls `refreshCodexModelCatalog(config)` directl } ``` -`refreshCodexCatalogIfDesired` loads persisted state, enters the same Codex write -flight, and passes the same last-moment authority into `refreshCodexModelCatalog`. +`refreshCodexCatalogIfDesired` loads persisted state, gathers outside the shared +lock, and commits catalog/cache inside it after the locked ownership/intent check. The injected test dependency is inside that gate, not an early-return bypass. Provider/model/combo routes therefore cannot bypass OFF, but no Claude, Grok, Desktop, or file-client reader changes. -## Startup reconciliation: OFF means remove again +### Target-keyed backups, not one shared legacy writer + +Two processes can share `OPENCODEX_HOME` while targeting different +`CODEX_HOME`s. The target-hashed path is safe +(`src/codex/catalog/parsing.ts:40-44`); the legacy +`OPENCODEX_HOME/catalog-backup.json` is not (`:36-38,440-445`). The hash usually +masks the collision, which is why this needs an activated two-home test. + +Stop writing the legacy path. Keep it as a read-only upgrade fallback at +`readCatalogBackup` (`src/codex/catalog/parsing.ts:419-422`), but every new +snapshot is target-hashed and an existing valid snapshot is never replaced: + +```diff + export function ensureCatalogBackup(catalogPath: string, catalog: RawCatalog): void { + const dir = getConfigDir(); + if (!existsSync(dir)) mkdirSync(dir, { recursive: true }); + writePristineCatalogBackup(catalogBackupPathFor(catalogPath), catalogPath, catalog); +- if (isDefaultCatalogPath(catalogPath)) writePristineCatalogBackup(legacyCatalogBackupPath(), catalogPath, catalog); + } +``` + +The two-home fixture uses one `OPENCODEX_HOME`, distinct canonical +`CODEX_HOME`s, and simultaneous first syncs. It asserts two different hashed +snapshots contain their own native slugs, a pre-seeded valid legacy file remains +byte-exact (and an absent one is not created), and each restore reads only its +target snapshot. + +## Startup reconciliation: ownership before any target mutation -The order in `handleStart` matters. Journal repair remains unconditional and runs -first; desired OFF convergence runs second; automatic sync runs later and observes -OFF: +The prior ordering was wrong. `handleStart` currently calls `reconcileJournal` +before server setup (`src/cli/index.ts:169-176`), and journal repair can replace +config/profile bytes (`src/codex/journal.ts:121-134`). Automatic convergence must +resolve ownership before that repair and before any lock artifact exists: ```diff async function handleStart(options: { block?: boolean } = {}) { @@ const requestedPort = parsePortOption(); - if (!currentExternalCodexModelProvider()) reconcileJournal(); -+ await reconcileCodexDesiredState("startup"); +- if (!currentExternalCodexModelProvider()) reconcileJournal(); ++ const ownership = inspectNativeCodexOwnership(); ++ if (ownership.state === "owned") { ++ reconcileCodexDesiredState("startup"); // lock -> recheck -> journal -> OFF remove ++ } else { ++ reportCodexConvergenceRefusal(ownership); // no repair and no lock path resolution ++ } const existingPid = readPid(); ``` -Apply the same reconciliation after journal repair in `handleEnsure` +Apply the identical preflight to `handleEnsure` before its current journal call (`src/cli/index.ts:358-365`). Reconciliation does exactly this: -1. Fresh-read desired state. ON or unavailable performs no removal. -2. For OFF, enter the Codex remove flight and fresh-read OFF again. -3. Immediately before removal, call `assertNativeTeardownOwned` inside the flight. -4. If ownership is foreign, return unresolved `home_mismatch`; preserve OFF and - every Codex byte. Otherwise call `restoreNativeCodexUnchecked`. -5. Inspect the native artifacts again. Report resolved only when OpenCodex routing, - profile, and proxy-routed catalog residue are absent. A history lock remains an - explained unresolved result; desired OFF is not rolled back. +1. Outside the lock, classify ownership. `foreign` and `unknown` return unresolved + and create no lock file. The proxy lifecycle may continue, but automatic Codex + journal/apply/remove writes are suppressed. +2. For `owned`, acquire the external per-home lock and classify ownership again. + A changed answer aborts before journal repair. +3. Under that lock, run `reconcileJournal` unconditionally for the owned home, then + fresh-read desired state. +4. ON ends reconciliation; later automatic sync gathers outside and re-enters the + same lock. OFF calls `restoreNativeCodexUnchecked` before releasing the lock. +5. Inspect config, profile, catalog, **models cache**, journal, and routed history. + Report resolved only when no OpenCodex routing or routed slug remains. A history + lock or unreadable cache is explained unresolved; desired OFF stays persisted. + +`foreign` means a valid install record names another canonical home. `unknown` +covers corrupt JSON, unreadable state, and an installed service with a missing +mirror. The tests drive all four cases plus a valid same-home record. The +valid-foreign and all unknown cases hash every native artifact before/after and +assert the external lock path does not exist; that is the byte-exact refusal claim +the prior startup ordering could not support. + +### The remover owns `models_cache.json` too + +`restoreNativeCodex` currently restores journal/config, catalog, and history at +`src/codex/inject.ts:764-795`, but apply writes routed models into +`models_cache.json` at `src/codex/catalog/sync.ts:600-613`. Reporting OFF while +that cache still advertises routed slugs repeats the WP2 incident: the test sees +only artifacts it already knew to assert. + +Add `restoreCodexModelsCache` beside `restoreCodexCatalog`. After catalog restore, +rewrite the cache from the restored catalog with an expired wrapper; if the +catalog is unavailable, parse the existing cache and remove only routed slugs. +Missing cache is success. Unreadable or unwritable cache is failure, not the +current swallowed `false` from `invalidateCodexModelsCache` (`:601-616`): + +```diff + const cat = restoreCodexCatalog(); ++ const cache = restoreCodexModelsCache(); +@@ +- return { success: cfg.success, message: `${msg}${historyMsg}` }; ++ return { ++ success: cfg.success && cache.success, ++ message: `${msg}${cache.message}${historyMsg}`, ++ }; +``` + +Observed-state inspection parses both `readCodexCatalogPath()` and +`activeCodexModelsCachePath()` (`src/codex/catalog/parsing.ts:68-75`) and treats +any routed slug in either as `applied`/`conflict`, never `absent`. Crash fixtures +stop immediately after cache replacement and after each later write. Every rerun +asserts no routed slug remains in **either** catalog or cache. -The crash point is after `mutateCodexDesiredEnabled(false)` commits and before the +The crash point is after `setCodexDesiredEnabled(false)` commits and before the remover starts. Restarting from that fixture must execute steps 1-5 again. A GET route is not used as a repair trigger; round 2 #5's mutating-GET design is dropped. ## Do not gate these paths -- `reconcileJournal` remains unconditional (`src/codex/journal.ts:148-162`). It - repairs an abandoned transaction before desired-state convergence decides the - final direction. -- ownership and drift inspection always run. Desired OFF never bypasses - `assertNativeTeardownOwned`. +- `reconcileJournal` remains unconditional **after ownership is proved**, and runs + under the shared lock before desired-state convergence chooses the final + direction (`src/codex/journal.ts:148-162`). +- ownership and drift inspection always run. Automatic convergence fails closed + on both `foreign` and `unknown`. - stop, uninstall, shutdown, and explicit native restore always remove state they own, regardless of desired ON, and never persist OFF (`src/service.ts:2587-2594`). @@ -418,8 +578,8 @@ route is not used as a repair trigger; round 2 #5's mutating-GET design is dropp |---|---| | Upgrade default | Load config with no `clientIntegrations`; Codex is ON and no bytes are rewritten. | | One-key parser | `codex:false` loads OFF; malformed `codex:"false"` degrades to ON while a future unknown key survives a field mutation. | -| Field-scoped commit | Mutate OFF from a stale live object; unrelated providers, API keys, and unknown fields survive. The live object is unchanged. | -| Lock/conflict refusal | Hold the real config mutation lock; mutation changes neither disk nor live object. Retry succeeds after release. | +| Field-scoped commit | Mutate OFF from a stale live object; unrelated providers, API keys, and unknown fields survive. The supplied live object changes only from `committed`/`unchanged` output. | +| Return-versus-throw matrix | Drive `committed`, `unchanged`, and `unavailable`; then hold a real config transaction for `SQLITE_BUSY` and inject a non-contention acquisition failure. Busy is retryable 409/503, broken lock is non-retryable write failure, and neither failure changes disk/live state. | | Auth sentinel unreachable | Start with no `claudeCode`, mutate Codex OFF then ON, reload, run `runClaudeAuthModeMigration`; it returns false and never creates `authMode` or `authModeMigratedAt`. | ### `tests/codex-desired-state-race.test.ts` (NEW) @@ -427,25 +587,40 @@ route is not used as a repair trigger; round 2 #5's mutating-GET design is dropp | Case | Activation and assertion | |---|---| | OFF during model fetch | Pause `gatherRoutedModels`, persist OFF through the real writer, release; catalog, cache, journal, config, profile, and history writer counts remain zero. | +| OFF at post-approval seam | Pause after locked ownership/ON approval and before each named write, start an OFF setter in another process, and prove OFF cannot commit until apply releases the shared lock. Then OFF commits/removes and final observed state is absent. | | Direct refresh bypass | Invoke a real provider/model route while OFF; `refreshCodexCatalogBestEffort` performs no catalog/cache write. | -| Single-flight | Hold one apply at the fetch seam, start a second-process refresh and a remove; no two Codex commit sections overlap, and the final operation re-reads the newest intent. | -| Crash after persist | Commit OFF, abort before remove, run startup and ensure independently; each converges a seeded applied Codex fixture to native state. | -| Foreign home | Seed install state for home A, run OFF startup reconciliation from home B; `assertNativeTeardownOwned` returns `home_mismatch`, all Codex bytes remain exact, and desired OFF remains persisted. | +| Hidden history writes | Fire the post-approval seam before every write inside the history callback; no callback write overlaps an OFF intent commit. | +| Single linearization lock | Hold one commit, start second-process refresh, ON setter, OFF setter, and remove; no native commit or desired-state commit overlaps, and final state follows lock acquisition order. | +| Crash after persist/cache | Crash after OFF intent and immediately after cache replacement, then run startup and ensure independently. Every rerun removes routed slugs from both catalog and cache. | +| Ownership matrix | Valid same-home is `owned`; valid foreign is `foreign`; corrupt, unreadable, and installed-service-with-missing-mirror are `unknown`. Foreign/unknown preserve all bytes and create no external lock file. | | Stop does not change intent | With desired ON, run owned stop/uninstall teardown; artifacts are removed and the flag remains ON. | ### Existing regressions -- `tests/codex-sync-api.test.ts`: OFF at entry avoids fetch/inject; OFF during - fetch returns the typed skip; `POST /api/sync` is 409 `codex_desired_off`, not - 200 and not 500. -- `tests/codex-inject-integration.test.ts`: each injected write authority seam is - reachable independently (journal, config, profile, journal mark, history); only +- `tests/codex-sync-api.test.ts`: every no-write result has `ok:false` and + `skippedReason`; `POST /api/sync` is 409 `codex_desired_off`, not 200/500. +- `tests/cli-restore-back.test.ts`: `ocx sync` refuses OFF with the switch command; + `restore/eject back` persists ON and applies under one lock, reports success only + after apply, and leaves ON persisted if apply later fails. +- `tests/startup-prompt.test.ts`: start and both ensure branches continue the proxy + while OFF and make no native write. `tests/cli-provider.test.ts` plus the model + case in `tests/cli-models-desired-state.test.ts` prove provider/model callers + report a skipped refresh rather than silently claiming Codex was updated. + `tests/cli-init-desired-state.test.ts` proves an affirmative injection prompt + may establish ON and a negative answer performs no Codex write. +- `tests/codex-inject-integration.test.ts`: every post-approval write seam is + reachable (journal, config, profile, journal mark, each history write); only `src/codex/desired-state.ts` imports the unchecked remover. -- `tests/codex-journal.test.ts`: seed a dead-PID journal while desired OFF; journal - reconciliation runs, then OFF convergence removes residual routing. +- `tests/codex-catalog-restore.test.ts`: a shared `OPENCODEX_HOME` with two + `CODEX_HOME`s creates two independent hashed snapshots and never replaces the + legacy path. `tests/codex-models-cache-invalidate.test.ts` pins restore from + catalog, fallback filtering, missing cache, and unreadable/unwritable refusal. +- `tests/codex-journal.test.ts`: seed a dead-PID journal while desired OFF; owned + repair runs under the lock before OFF convergence. Foreign/unknown runs do not + repair the journal or create the lock. - `tests/service.test.ts` and `tests/uninstall.test.ts`: every production native - remover passes through `assertNativeTeardownOwned`; foreign-home teardown writes - nothing; owned teardown never changes desired state. + remover passes through the owned lock; foreign/unknown automatic teardown writes + nothing; explicit owned teardown never changes desired state. - `tests/server-auth.test.ts`: add a live-server case with `clientIntegrations.codex=false`; `POST /v1/responses` reaches the same normal validation/routing response as ON, never a client-disabled response. @@ -459,11 +634,13 @@ the user's live proxy on port 10100. ```bash bun test tests/codex-desired-state.test.ts bun test tests/codex-desired-state-race.test.ts -bun test tests/codex-sync-api.test.ts tests/codex-inject-integration.test.ts +bun test tests/codex-sync-api.test.ts tests/codex-inject-integration.test.ts tests/cli-restore-back.test.ts tests/cli-provider.test.ts tests/cli-models-desired-state.test.ts tests/cli-init-desired-state.test.ts tests/startup-prompt.test.ts +bun test tests/codex-catalog-restore.test.ts tests/codex-models-cache-invalidate.test.ts bun test tests/codex-journal.test.ts tests/service.test.ts tests/uninstall.test.ts tests/server-auth.test.ts bun run typecheck bun run test bun run privacy:scan +cd docs-site && bun install --frozen-lockfile && bun run build ``` Live proof is the subprocess case in `tests/codex-desired-state-race.test.ts`, not @@ -474,11 +651,12 @@ post-persist/pre-remove seam and relaunches from the same isolated home. It prov 1. `/healthz` returns from the same PID after the live OFF mutation; disabling native Codex did not stop or replace the proxy. -2. native Codex routing/profile/catalog residue converges to absent after restart. +2. native Codex routing/profile/catalog/cache residue converges to absent after restart; + neither catalog nor cache contains a routed slug after any crash-point rerun. 3. an invalid `/v1/responses` request reaches its normal validation response, not a desired-state gate. -4. repeating from a foreign `OPENCODEX_HOME` leaves every Codex artifact byte-exact - and reports `home_mismatch`. +4. repeating valid-foreign, corrupt, unreadable, and missing-mirror ownership cases + leaves every Codex artifact byte-exact and creates no per-home lock file. The test must print the isolated roots, chosen port, before/after hashes, and reconciliation result so the C-phase evidence proves the live path rather than @@ -489,12 +667,14 @@ directory. | Roadmap criterion | WP4 closure | |---|---| -| C2 — Codex stays disabled across restart, ensure, and `/api/sync` | OFF is persisted through `mutatePersistedConfig`; startup/ensure converge residual apply state; sync and direct catalog refresh re-read OFF before writing; `/api/sync` reports the skip honestly. | +| C2 — Codex stays disabled across restart, ensure, and `/api/sync` | OFF and every native commit share one per-home lock; startup/ensure converge catalog and cache residue; every skipped caller returns/reports failure honestly. | | C3 — absent config changes nothing on upgrade | Missing object/key remains ON. The one-key parser and no-write upgrade test prove existing installs keep current behavior. | | C4 — disabling Codex never stops proxy or closes `/v1/responses` | No lifecycle or transport gate is added. The isolated live proof keeps `/healthz` and the Responses route reachable while native Codex state is removed. | -| C7 — foreign-home startup touches nothing | Every production remover runs `assertNativeTeardownOwned` first; the foreign-home crash-recovery fixture proves byte-exact refusal with desired OFF preserved. | - -WP4 is complete only when an operation that began ON can resume after OFF and -still perform zero Codex writes, and when a restart can finish an interrupted OFF -without touching a foreign service's state. A boolean without those two proofs is -the failed first draft in a smaller file. +| C7 — foreign-home startup touches nothing | Tri-state ownership is resolved before journal/lock creation and rechecked under the external lock; foreign and unknown fixtures prove byte-exact refusal and no new lock artifact. | +| Public contract stays synchronized | English lifecycle/configuration docs name OFF behavior and explicit-enable semantics; translated lifecycle/config pages contain no contradictory unconditional promise. | + +WP4 is complete only when no native write can occur after OFF has linearized, an +OFF started at the post-approval seam is forced to wait and then wins by removing +the prior apply, cache and catalog are both clean after every crash rerun, and +foreign/unknown startup creates no artifact at all. A boolean plus an unlocked +re-read is the failed draft, not this design. From c5907146c7695847eb9bcb51750cc628662e817c Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Tue, 4 Aug 2026 02:45:36 +0900 Subject: [PATCH 023/163] =?UTF-8?q?docs(integrations):=20WP4=20round=202?= =?UTF-8?q?=20=E2=80=94=20the=20lock=20is=20not=20implementable=20over=20t?= =?UTF-8?q?his=20code=20yet?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three closed, four open, five new blockers. Unlike the earlier divergence this is not the phase map: the reviewer re-confirmed the Codex-only re-scope holds. It is narrower and more honest — Codex's write path was never designed to be interrupted. The concrete proofs. The management refresh dependency is an opaque () => Promise that gathers and writes in one call, so 'gather outside the lock, commit inside' cannot be done to it. The history write is two 5s SQLite busy waits plus a synchronous 500ms sleep plus row-dependent rollout writes, so my 'bounded synchronous commit' can hold the lock ~10.5s on the server event loop — a dashboard OFF would stop serving every other client, which is the exact thing this feature exists to prevent. I also deleted a guard I did not recognize: startup suppresses journal repair under an external model_provider, and I replaced that authority with service-home ownership. Satisfying this design needs a gather/commit seam in the management contract, history moved off the event loop or given a fail-fast mode, an async lock protocol with deadlines and fairness, and a hardened per-user lock namespace. That is a concurrency substrate, not a boolean with a check. Recording the Codex switch as blocked on that prerequisite rather than patching a third round. The two shipped deliverables stand on their own evidence; WP6 and WP7 depend on the same substrate, which is why the chain stops here instead of continuing to the next work-phase. --- .../008_audit_synthesis_wp4_r2.md | 106 ++++++++++++++++++ 1 file changed, 106 insertions(+) create mode 100644 devlog/_plan/260803_codex_desktop_toggle/008_audit_synthesis_wp4_r2.md diff --git a/devlog/_plan/260803_codex_desktop_toggle/008_audit_synthesis_wp4_r2.md b/devlog/_plan/260803_codex_desktop_toggle/008_audit_synthesis_wp4_r2.md new file mode 100644 index 000000000..498fe2378 --- /dev/null +++ b/devlog/_plan/260803_codex_desktop_toggle/008_audit_synthesis_wp4_r2.md @@ -0,0 +1,106 @@ +# WP4 audit round 2 — synthesis, and a scope decision + +Verdict: **FAIL**. Three closed (#4, #6, #7), four still open, **five new High**. + +## The count that matters + +| Round | Closed | Still open | New | +|---|---|---|---| +| 1 | — | — | 7 (4 blocking) | +| 2 | 3 | 4 | 6 (5 blocking) | + +Same signature as the earlier divergence: fixes opening more than they close. +But the diagnosis is different this time, and it is not the phase map. + +## What round 2 actually discovered + +My lock is not implementable over the code it has to wrap. + +- **The refresh dependency has no seam.** `refreshCodexCatalog?: () => Promise` + (`src/server/management/context.ts:11`) gathers AND writes inside one opaque + async call. My design says "gather outside the lock, commit inside it". You + cannot do that to a callback that does both and returns nothing. Calling it + outside leaves writes unprotected; calling it inside breaks my own + no-await-under-lock rule (new #1). +- **The history write is not bounded.** Two SQLite busy waits at 5 s plus a + synchronous `Bun.sleepSync(500)` (`history-provider.ts:31,537`), then + row-dependent rollout writes. My "bounded synchronous commit" can hold the + lock for ~10.5 s **on the server's event loop**, so a dashboard-initiated OFF + stops serving every other client — the precise thing this feature exists to + prevent (new #3). +- **I deleted a guard I did not recognize.** Startup suppresses journal repair + when `currentExternalCodexModelProvider()` is set. I replaced that with + service-home ownership and called it "unconditional". Those are different + authorities, and the result can write our journal baseline over a config + another provider owns (new #2). +- **Persisted ON can read as OFF.** `/api/sync` gates on the server's + long-lived `config` object before reaching my fresh read, so a CLI enable in + another process leaves the running server refusing (new #4). +- **`unchanged` is not `converged`.** I let a no-op config commit skip native + convergence, which false-greens OFF-with-residue and ON-with-missing-artifacts + (new #5). + +Every one is accepted. None is a nitpick; #3 alone would ship a switch that +freezes the proxy. + +## The decision this forces + +Rounds 1 and 2 of WP4 are not the coupling failure `006` diagnosed — the +reviewer re-confirmed the re-scope holds. They are something narrower and more +honest: **Codex's write path was never designed to be interrupted**, and making +it safely interruptible is its own unit of work. + +The evidence, plainly: to satisfy this design I would have to introduce a +gather/commit seam into the management refresh contract, move history mutation +off the server event loop or give it a fail-fast mode, define an async lock +acquisition protocol with deadlines and fairness, and specify a per-user lock +namespace with symlink hardening. That is a concurrency substrate. It is not +"add a boolean and check it". + +I am not going to keep patching a design whose foundation the audit keeps +finding underneath it. Two rounds, eleven accepted findings, five of them +discovered only after my own fixes created them. + +## What that means for the goal + +The objective has four deliverables. Two shipped and are proven: + +| | Status | +|---|---| +| gjc/Pi modality poisoning | **DONE** — proven against the client's own schema | +| API keys row | **DONE** — rendered, observed, isolated | +| Codex toggle | **BLOCKED on a prerequisite this unit uncovered** | +| Desktop toggle | deferred behind Codex, by design | + +The Codex switch is not blocked by anything external, and it is not +unachievable. It is blocked by a real finding: the safe-interruption substrate +has to exist first, and its scope is a unit, not a work-phase inside this one. + +Recording it as `NEEDS_HUMAN` rather than continuing is the honest call under +LOOP-CONTINUE-01. The rule forbids shrinking the objective to escape a loop; it +does not require pretending a newly-discovered prerequisite is small. The +owner should decide whether to fund the substrate now or ship the two proven +deliverables and revisit. + +## Carried forward, in full + +Every open and new finding, for whoever picks this up: + +| Finding | Status | Belongs to | +|---|---|---| +| r1 #1 lock linearization | open — lock is right, wrapping is not implementable | substrate unit | +| r1 #2 ownership tri-state | open — external-provider guard regression (r2 #2) | substrate unit | +| r1 #3 `models_cache.json` | open — invalidation conflated with restoration (r2 #6) | substrate unit | +| r1 #4 caller semantics | **closed** | — | +| r1 #5 lock-error branch | open — `conflict` wrongly called non-retryable | substrate unit | +| r1 #6 legacy catalog backup | **closed** | — | +| r1 #7 docs-site sync | **closed** | — | +| r2 #1 refresh contract has no seam | open — **blocking**, needs a contract change | substrate unit | +| r2 #3 history is unbounded on the event loop | open — **blocking** | substrate unit | +| r2 #4 stale in-memory config admission | open — **blocking** | substrate unit | +| r2 #5 `unchanged` skips convergence | open — **blocking** | substrate unit | +| r2 #7 lock namespace hardening | open | substrate unit | + +WP6 (Grok) and WP7 (Desktop) both depend on the same substrate, so neither is +startable ahead of it. That dependency is why the loop stops here rather than +chaining to the next work-phase. From 99ccfeaee41453d548e46bddbedbbb7f37537590 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Tue, 4 Aug 2026 09:41:22 +0900 Subject: [PATCH 024/163] docs(substrate): research the four parts of a safe-interruption write path MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 001 catalog seam: draws the exact gather/commit line through refresh.ts and traces all 16 management callers. Hardest problem named — stopping a locked commit from applying a candidate gathered against an obsolete config. 002 history: maps every blocking operation and separates server-process from CLI-process callers, since a CLI blocking itself is acceptable and the server blocking is not. Recommends worker isolation plus fail-fast convergence, and names the constraint that history traversal has no finite work bound. 003 lock: async SQLite lock with finite deadlines and acquired|busy|refused, namespaced per-user under the real home rather than a shared tmpdir, keyed by a realpathed home. Hardest question is identifying a missing home across case-sensitive and case-insensitive filesystems. 004 ownership: one admission order for start, ensure and routes, with foreign and unknown failing closed before any artifact exists. Creation is proven by a ledger recording baseline absence plus a post-image hash, not by a filename or marker — which is what makes absence restorable. Hardest case is baseline absence followed by native edits: preservation wins and restoration reports a conflict rather than deleting user data. --- .../001_catalog_seam.md | 429 ++++++++++++++++ .../002_history_off_the_loop.md | 486 ++++++++++++++++++ .../003_lock_protocol.md | 453 ++++++++++++++++ .../004_ownership_and_convergence.md | 391 ++++++++++++++ 4 files changed, 1759 insertions(+) create mode 100644 devlog/_plan/260804_codex_write_substrate/001_catalog_seam.md create mode 100644 devlog/_plan/260804_codex_write_substrate/002_history_off_the_loop.md create mode 100644 devlog/_plan/260804_codex_write_substrate/003_lock_protocol.md create mode 100644 devlog/_plan/260804_codex_write_substrate/004_ownership_and_convergence.md diff --git a/devlog/_plan/260804_codex_write_substrate/001_catalog_seam.md b/devlog/_plan/260804_codex_write_substrate/001_catalog_seam.md new file mode 100644 index 000000000..eb3ffa2fa --- /dev/null +++ b/devlog/_plan/260804_codex_write_substrate/001_catalog_seam.md @@ -0,0 +1,429 @@ +# Part 1 research — Codex catalog gather/commit seam + +The failure is a check/write gap, not a missing `try/catch`. The failed OFF design +needed provider discovery outside the per-`CODEX_HOME` lock and catalog/cache +mutation inside it, but the only management dependency returns `Promise` +and the production function performs both halves before it resolves +(`src/server/management/context.ts:9-18`, +`devlog/_plan/260803_codex_desktop_toggle/008_audit_synthesis_wp4_r2.md:17-24`). +Calling that operation outside the lock leaves native writes after OFF can +linearize; calling it inside the lock admits provider I/O and a 10-second bundled +catalog subprocess under a rule that forbids slow work there +(`devlog/_plan/260803_codex_desktop_toggle/030_desired_state.md:250-310`, +`src/codex/catalog/bundled.ts:127-143`). + +This document specifies only the catalog substrate needed to remove that +contradiction. It does not specify the desired-state flag, lock implementation, +history writer, ownership preflight, management toggle, or GUI; those were +separate responsibilities in the failed design +(`devlog/_plan/260803_codex_desktop_toggle/030_desired_state.md:53-84`). + +## 1. Current call graph + +There are two production entry paths to `refreshCodexModelCatalog`: the +management helper dynamically imports it, while `syncModelsToCodex` imports it +directly (`src/server/management-api.ts:105-113`, `src/codex/sync.ts:1-4,24-34,83-108`). +The management helper is then awaited at 16 mutation sites; no production caller +uses `void`, an unhandled promise, or a detached task +(`src/server/management/provider-routes.ts:147,338,487,512,527,546`, +`src/server/management/model-routes.ts:214,313,352,390,404,440`, +`src/server/management/combo-routes.ts:198,216`, +`src/server/management/agent-settings-routes.ts:280,525`). + +The existing dependency and production paths are not equivalent. An injected +`deps.refreshCodexCatalog` returns before the helper's `try`, so its rejection +reaches the management error mapper; the production dynamic import and refresh +are inside a blanket catch and disappear +(`src/server/management-api.ts:105-112,150-163`). The management API already has +a `CatalogGatherBusyError` to 503/`Retry-After` mapping, but production refresh +contention cannot reach it through this helper +(`src/server/management-api.ts:75,150-163`, +`src/codex/catalog/provider-fetch.ts:76-78,670-693`). + +### 1.1 `refreshCodexCatalogBestEffort` callers + +Every row below has the same current result handling: persist the primary +mutation, await a `Promise`, discard any refresh result, and return the +ordinary success body shown at the cited route tail. + +| # | Trigger | Current action after persistence | Typed-outcome change | +|---|---|---|---| +| P1 | `POST /api/providers` add or overwrite | Clears that provider's model cache, awaits refresh, then returns `{ success: true, name }` (`src/server/management/provider-routes.ts:96-99,132-148`). | Keep the provider mutation successful, but attach the catalog outcome instead of implying that Codex also changed. | +| P2 | Ordinary `PATCH /api/providers?name=...` field edit or enabled toggle | The standalone default/mode branches return earlier; the ordinary branch saves, optionally clears discovery cache, awaits refresh, then returns provider state (`src/server/management/provider-routes.ts:193-208,210-305,330-344`). | Return provider success plus committed/skipped/failed catalog status. | +| P3 | `DELETE /api/providers?name=...` | Reassigns the default if needed, saves, clears the deleted provider cache, awaits refresh, then returns success (`src/server/management/provider-routes.ts:449-488`). | Preserve deletion success and report whether stale Codex rows remain. | +| P4 | `PUT /api/provider-context-caps` with `value` | Saves the new global cap, clears affected provider caches, awaits refresh, then returns the cap view (`src/server/management/provider-routes.ts:495-513`). | Preserve cap success and expose catalog disposition. | +| P5 | `PUT /api/provider-context-caps` with `setAll` | Saves all cap toggles, clears affected caches, awaits refresh, then returns the cap view (`src/server/management/provider-routes.ts:516-528`). | Preserve cap success and expose catalog disposition. | +| P6 | `PUT /api/provider-context-caps` for one provider | Saves the provider cap, clears its cache, awaits refresh, then returns the cap view (`src/server/management/provider-routes.ts:531-547`). | Preserve cap success and expose catalog disposition. | +| M1 | `PUT /api/disabled-models` | Saves the blocklist, awaits refresh, then returns `{ ok: true, disabled }` (`src/server/management/model-routes.ts:206-215`). | Preserve blocklist success and state explicitly when Codex did not consume it. | +| M2 | `PUT /api/model-visibility` | Saves the combined allowlist/blocklist edit, awaits refresh, then returns visibility success (`src/server/management/model-routes.ts:218-255,277-314`). | Preserve visibility success and expose catalog disposition. | +| M3 | `POST /api/custom-models` | Saves the new custom row, awaits refresh, then returns the row with 201 (`src/server/management/model-routes.ts:321-353`). | Preserve 201 and report whether the row reached Codex. | +| M4 | `PUT /api/custom-models/:id` | Saves the edited row, awaits refresh, then returns it (`src/server/management/model-routes.ts:356-391`). | Preserve edit success and report whether Codex consumed it. | +| M5 | `DELETE /api/custom-models/:id` | Saves removal, awaits refresh, then returns `{ ok: true }` (`src/server/management/model-routes.ts:394-405`). | Preserve removal success and report whether a stale native row may remain. | +| M6 | `PUT /api/selected-models` | Saves/clears the provider allowlist, awaits refresh, then returns selection success (`src/server/management/model-routes.ts:426-441`). | Preserve selection success and expose catalog disposition. | +| C1 | `PUT /api/combos` create, update, or rename | Saves combo and migrated references, clears combo runtime state, awaits refresh, optionally syncs Claude agents, then returns success (`src/server/management/combo-routes.ts:83-115,150-200`). | Codex failure must not suppress the already-saved combo or the independent Claude sync; include a catalog status. | +| C2 | `DELETE /api/combos?id=...` | Saves removal, clears combo runtime state, awaits refresh, then returns success (`src/server/management/combo-routes.ts:203-217`). | Preserve deletion success and report whether the retired combo row remains in Codex. | +| A1 | `PUT /api/v2` | Applies Codex feature/config writers, awaits refresh, then returns the effective settings and warnings (`src/server/management/agent-settings-routes.ts:154-178,240-294`). | Add a typed catalog warning/status; do not turn an already-landed feature write into 5xx. | +| A2 | `PUT /api/subagent-models` | Saves chosen models, awaits Codex refresh, then independently syncs Claude agents and Desktop before returning success (`src/server/management/agent-settings-routes.ts:495-528`). | Continue the Claude/Desktop work after any Codex skip/failure and attach the Codex disposition. | + +`config-routes.ts`, `logs-usage-routes.ts`, and `oauth-account-routes.ts` destructure +the shared helper but do not invoke it; the complete production invocation set is +the 16 sites above +(`src/server/management/config-routes.ts:77`, +`src/server/management/logs-usage-routes.ts:124`, +`src/server/management/oauth-account-routes.ts:115`). + +### 1.2 Direct `refreshCodexModelCatalog` caller + +`syncModelsToCodex` is the only non-test direct caller. It skips catalog refresh +for an externally owned model provider, otherwise catches every refresh throw, +logs a warning, and still calls `injectCodexConfig`; its final `ok` is the +injection result rather than catalog success +(`src/codex/sync.ts:49-71,73-129`). This fallback is pinned: a thrown refresh must +still call injection with an undefined catalog path and return `ok: true` plus a +warning when injection succeeds (`tests/codex-sync-api.test.ts:148-166`). + +Its second-order production callers are all awaited: custom-model save and +provider `--sync` catch and warn, start suppresses failure, both ensure paths +catch and warn, `restore back` and explicit `sync` inspect `ok`, and `POST +/api/sync` maps `ok` to HTTP 200/500 +(`src/cli/models.ts:102-107`, `src/cli/provider.ts:232-237`, +`src/cli/index.ts:318-320,358-411,745-763,827-840`, +`src/server/management/config-routes.ts:261-268`). A typed desired-OFF or native +lock-busy result must stop before `injectCodexConfig`; network/auth catalog +degradation must retain the existing injection fallback +(`devlog/_plan/260803_codex_desktop_toggle/030_desired_state.md:372-423`). + +## 2. Where gathering ends and committing begins + +`refreshCodexModelCatalog` currently looks small because `syncCatalogModels` +contains the mixed operation. It awaits that function, checks whether the +returned path exists, then calls a cache invalidator that reads the just-written +catalog and writes `models_cache.json` +(`src/codex/refresh.ts:40-52`, `src/codex/catalog/sync.ts:601-616`). + +### 2.1 Gathering: permitted to be async or slow, forbidden to write + +The gather phase must include all of the following work: + +1. Resolve the effective catalog path and load an in-memory source catalog, + existing target catalog, native template, and baseline + (`src/codex/catalog/sync.ts:513-525,545-565`). +2. Run bundled catalog discovery when needed. `loadBundledCodexCatalog` can resolve + the Codex runtime and execute `codex debug models --bundled` with a 10-second + timeout, so it cannot run under the write lock + (`src/codex/catalog/bundled.ts:127-165,188-210`). +3. Await `gatherRoutedModels`. It admits one gather flight, resolves provider auth, + and fans out provider model discovery with `Promise.all` + (`src/codex/catalog/provider-fetch.ts:410-490,670-717`). +4. Filter visible models, build routed entries, merge native/routed state, clamp + fields, and serialize the final catalog in memory + (`src/codex/catalog/sync.ts:533-568`). +5. Build the expired cache wrapper in memory instead of rereading the catalog + after commit; the wrapper shape is currently assembled immediately before its + write (`src/codex/catalog/sync.ts:600-613`). +6. Decide whether pristine backup payloads are needed and capture their bytes in + the candidate. Current backup creation performs `existsSync`, catalog reads, + `copyFileSync`, `mkdirSync`, and `atomicWriteFile`; those writes cannot remain + in gather (`src/codex/catalog/parsing.ts:428-444`). + +`loadCatalogForSync` is itself mixed. Its bundled/source reads belong to gather, +but its final fallback calls `materializeBundledCodexCatalog`, whose +`mkdirSync` plus `atomicWriteFile` are writes +(`src/codex/catalog/bundled.ts:213-234`). The split therefore cannot merely move +the `await` at `syncCatalogModels:526`; it must replace the materializing fallback +with a pure in-memory source and defer materialization bytes to commit +(`src/codex/catalog/sync.ts:507-531`). + +### 2.2 Committing: synchronous, bounded, and byte-oriented + +The commit phase consumes already-built payloads and performs no provider fetch, +auth refresh, subprocess, catalog merge, JSON parsing, or `await`. Its complete +write set is: + +- create a parent/config directory only when one of the prepared writes needs it + (`src/codex/catalog/bundled.ts:213-218`, + `src/codex/catalog/parsing.ts:440-444`); +- write zero, one, or two prepared pristine backup payloads, replacing the current + `copyFileSync`/`atomicWriteFile` choice with candidate bytes + (`src/codex/catalog/parsing.ts:428-444`); +- write the prepared catalog bytes with `atomicWriteFile` + (`src/codex/catalog/sync.ts:565-569`); +- write the prepared expired cache wrapper with `atomicWriteFile` + (`src/codex/catalog/sync.ts:601-613`). + +`atomicWriteFile` is synchronous: it writes a temporary file, hardens it, renames +it, and scrubs/removes a failed temporary file before rethrowing +(`src/config.ts:178-230`). The number of candidate writes is fixed by the payload +set rather than provider/model count, which is the bounded property required by +the failed lock design +(`devlog/_plan/260803_codex_desktop_toggle/030_desired_state.md:250-310`). + +`syncCodexModelsCacheFromCatalog` is not in the production refresh path and must +not become the commit primitive: it rereads the catalog and writes raw catalog +bytes to the cache path, while the active invalidator requires an expired wrapper +(`src/codex/refresh.ts:29-38`, `src/codex/catalog/sync.ts:600-613`). + +## 3. Proposed contract + +**INFERRED contract:** expose exactly these two operations from the catalog owner, +because the audit requires the lock owner to schedule them separately +(`devlog/_plan/260803_codex_desktop_toggle/008_audit_synthesis_wp4_r2.md:46-58`). + +```ts +export interface CodexCatalogCandidate { /* opaque, readonly, branded */ } + +export async function gatherCodexCatalogCandidate( + config: OcxConfig, +): Promise; + +export function commitCodexCatalogCandidate( + candidate: CodexCatalogCandidate, +): CodexCatalogCommitResult; +``` + +The candidate must carry: + +- final catalog bytes, expired cache-wrapper bytes, target catalog/cache paths, + and optional pristine-backup path/byte pairs; these are the complete current + write products (`src/codex/catalog/sync.ts:568,601-613`, + `src/codex/catalog/parsing.ts:428-444`); +- `added`, `comboOmissions`, source/catalog existence, and the result metadata now + returned after mutation (`src/codex/refresh.ts:8-15,44-52`); +- a digest of the catalog-affecting config snapshot, including credential changes + without retaining or exposing credential values; provider auth and config both + influence gathered rows (`src/codex/catalog/provider-fetch.ts:410-490,696-717`); +- canonical target identity, resolved catalog path, and a digest-or-absence marker + for the base catalog from which the merge was assembled; the merge deliberately + preserves existing routed/user-native rows from that file + (`src/codex/catalog/sync.ts:513-525,430-468`); +- typed provider-discovery notices and the fallback actually used. Current network, + HTTP, malformed-response, and destination-policy failures degrade to stale or + configured models rather than throwing (`src/codex/catalog/provider-fetch.ts:494-570,616-632`); +- an internal one-shot identity so the same candidate cannot be committed twice. + **INFERRED:** two commits of one snapshot can overwrite a later refresh even + when the lock serializes both (`src/codex/catalog/sync.ts:568`). + +The candidate must not carry a mutable `OcxConfig` reference, an open file handle, +or a callback. **INFERRED:** those would allow mutation after gather or smuggle +slow work back into commit, recreating the opaque all-in-one dependency rejected +by the audit +(`devlog/_plan/260803_codex_desktop_toggle/008_audit_synthesis_wp4_r2.md:19-24`). + +The payload fields are mostly JSON-compatible, but the candidate is deliberately +not a persistence or IPC format. **INFERRED:** a private brand/one-shot token and +freshness evidence are process-local; serializing and later replaying it would +turn a short gather/commit handoff into an unbounded stale-write mechanism. + +The management test seam must split too. Keeping +`refreshCodexCatalog?: () => Promise` as an early-return override would let +the injected path continue bypassing lock admission, exactly as it does now +(`src/server/management/context.ts:9-18`, `src/server/management-api.ts:105-112`). +**INFERRED:** inject a paired gather/commit dependency (or one object containing +both) so tests can substitute candidate production and byte commit independently; +the management orchestrator, desired-state check, and lock remain outside that +pair. + +## 4. Freshness contract + +A candidate can become stale between gather and commit. The lock serializes +commits; it does not freeze the config, target path, or catalog while gathering +happens outside it +(`devlog/_plan/260803_codex_desktop_toggle/030_desired_state.md:250-310`). + +**INFERRED invalidators:** commit must refuse with `stale_candidate` when any of +these differ under the lock from the candidate's evidence: + +1. effective desired state is OFF or unavailable; +2. catalog-affecting config digest changed; +3. canonical `CODEX_HOME`, resolved catalog path, or cache path changed; +4. base catalog digest/existence changed, including a prior refresh committing + while this candidate waited; +5. ownership is no longer the same owned target; or +6. the candidate was already consumed. + +The need for checks 1 and 5 comes from the failed OFF ordering, while checks 2-4 +close the stale-config and stale-merge window created by gathering outside the +lock +(`devlog/_plan/260803_codex_desktop_toggle/030_desired_state.md:250-310`, +`src/codex/catalog/sync.ts:513-525,430-468`). A newly appearing valid pristine +backup is not permission to overwrite it: current behavior is create-once, so +commit skips that prepared backup write if another commit created it first +(`src/codex/catalog/parsing.ts:428-444`). + +Provider inventory changing upstream after gather does not by itself invalidate +the candidate. **INFERRED:** it is a point-in-time discovery result, just as the +current fetch/cache path is; a later refresh supersedes it +(`src/codex/catalog/provider-fetch.ts:481-510,613-632`). Wall-clock age alone is +also not a correctness test; revision evidence is. + +## 5. Typed refresh outcome + +`refreshCodexCatalogBestEffort` should remain non-throwing by design, but +"best effort" must mean a returned disposition, not `Promise` plus a blanket +catch (`src/server/management-api.ts:105-112`). **INFERRED contract:** the +orchestrator returns this closed shape: + +```ts +type CodexCatalogRefreshOutcome = + | { status: "committed"; result: CodexCatalogRefreshResult; notices: CatalogGatherNotice[] } + | { status: "skipped"; reason: "catalog_unavailable" | "desired_off" | + "gather_busy" | "lock_busy" | "stale_candidate"; retryable: boolean } + | { status: "failed"; reason: "provider_network" | "provider_auth" | "disk"; + phase: "gather" | "commit"; retryable: boolean; writes?: CatalogWriteReceipt }; +``` + +The real dispositions and caller meanings are: + +| Condition | Evidence | Outcome and required handling | +|---|---|---| +| Per-provider network/HTTP/policy failure with stale or configured fallback | Discovery records failure and returns fallback rows (`src/codex/catalog/provider-fetch.ts:494-570,616-632`). | Commit can succeed with a typed notice; management mutations stay successful and `syncModelsToCodex` keeps its warning/fallback behavior. | +| Network failure that prevents any candidate | `Promise.all` rejects if a provider-stage operation escapes its local fallback (`src/codex/catalog/provider-fetch.ts:696-717`). | `failed/provider_network`, no commit; management returns primary success plus warning, sync may continue injection fallback. | +| Missing OAuth token | Discovery returns configured rows (`src/codex/catalog/provider-fetch.ts:475-479`). | A committed candidate with `provider_auth` notice, not a route failure. | +| Auth/token resolution throws before fetch fallback | Token resolution occurs before the fetch `try` (`src/codex/catalog/provider-fetch.ts:428,512-516`). | `failed/provider_auth`, no commit; management returns primary success plus warning, sync may continue injection fallback. | +| Gather admission occupied | `tryAcquire` throws `CatalogGatherBusyError` (`src/codex/catalog/provider-fetch.ts:670-684`). | `skipped/gather_busy`, retryable; best-effort mutation routes do not become 503 after their primary write, but report retryable skip. | +| Desired Codex state is OFF | The failed design requires a fresh desired-state check under the shared lock before any native write (`devlog/_plan/260803_codex_desktop_toggle/030_desired_state.md:250-310,372-423`). | `skipped/desired_off`, not retryable until intent changes; no catalog, cache, or injection write. | +| Native write lock is occupied | The failed design classifies lock timeout as a no-write result (`devlog/_plan/260803_codex_desktop_toggle/030_desired_state.md:372-423`). | `skipped/lock_busy`, retryable; no commit and no injection. | +| Candidate revision changed | Gathering reads and merges current on-disk rows before its write (`src/codex/catalog/sync.ts:513-525,430-468`). | `skipped/stale_candidate`, retryable after regather; never commit against changed config/catalog. | +| No usable catalog source | Current sync returns `catalogWritten:false`, and refresh reports absent without touching cache (`src/codex/catalog/sync.ts:513-515`, `src/codex/refresh.ts:44-52`). | `skipped/catalog_unavailable`; retain current native catalog/injection fallback. | +| Backup/catalog/cache filesystem failure | Backup errors are currently swallowed, catalog write throws, and cache write is caught as `false` (`src/codex/catalog/sync.ts:527-531,568`, `src/codex/catalog/sync.ts:601-616`). | `failed/disk` with per-write receipt. Do not claim rollback across files; report partial catalog/cache state and let convergence retry. | + +`CatalogWriteReceipt` must identify which fixed writes landed before failure because +catalog and cache are separate atomic replacements, not one transaction +(`src/codex/catalog/sync.ts:568,601-613`). **INFERRED:** a disk failure after the +catalog rename but before cache rename is a partial commit, so a single boolean +cannot tell an explicit sync whether a long-lived Codex app-server may be stale; +the CLI already reacts to either `catalogWritten` or `cacheSynced` +(`src/cli/index.ts:827-840`). + +### 5.1 Caller policy + +All 16 management callers are best-effort with respect to Codex catalog refresh: +their primary config mutation is persisted before refresh, and their current +success response follows the awaited call +(`src/server/management/provider-routes.ts:132-148,330-344,479-488`, +`src/server/management/model-routes.ts:311-314,350-353,387-405,437-441`, +`src/server/management/combo-routes.ts:190-217`, +`src/server/management/agent-settings-routes.ts:240-294,518-528`). +**INFERRED:** they must keep their existing 2xx/201 primary outcome for every +catalog disposition and add a small `catalogRefresh` field containing +`status/reason/retryable`; rolling back or returning 5xx would falsely say the +provider/model/combo/setting mutation did not land. + +`syncModelsToCodex` has two policies. Provider network/auth, unavailable catalog, +and ordinary disk failure preserve the tested injection fallback and return a +warning/receipt; desired OFF, native lock busy, or stale locked approval return +`ok:false` before injection because those mean native writes are not authorized +(`src/codex/sync.ts:83-129`, `tests/codex-sync-api.test.ts:148-166`, +`devlog/_plan/260803_codex_desktop_toggle/030_desired_state.md:372-423`). Explicit +`POST /api/sync` can then map desired OFF to 409, retryable busy to 409/503, and +non-retryable disk failure to 500 instead of mapping every `ok:false` to 500 +(`src/server/management/config-routes.ts:261-268`, +`devlog/_plan/260803_codex_desktop_toggle/030_desired_state.md:396-407`). + +## 6. Existing tests and contract breakage + +The split is a contract change, not an internal rename: + +- `tests/codex-refresh.test.ts` pins one async all-in-one + `syncCatalogModels` dependency followed by `existsSync` and cache invalidation, + the exact result booleans, real catalog rewrite, and cache success/failure + (`tests/codex-refresh.test.ts:60-216`). It must split gather assertions from + fixed-write commit assertions and add stale-candidate/partial-write receipts. +- `tests/codex-sync-api.test.ts` stubs `refreshCodexModelCatalog` with the current + result object and pins catch-and-continue injection fallback + (`tests/codex-sync-api.test.ts:47-166,227-255`). It must stub typed outcomes and + separately pin desired-OFF/lock-busy no-injection behavior. +- `tests/codex-models-cache-invalidate.test.ts` calls the real refresh through + `syncModelsToCodex` and pins `catalogWritten || cacheSynced` as the app-server + restart gate (`tests/codex-models-cache-invalidate.test.ts:114-177`). Its + assertions must consume the write receipt without losing that gate. +- `tests/injection-model-api.test.ts` directly verifies that the current refresh + forwards the mutable config object to `syncCatalogModels` + (`tests/injection-model-api.test.ts:373-401`). It must instead verify candidate + fingerprint/input snapshot behavior. +- `tests/model-visibility-management-api.test.ts` counts exactly one injected + void refresh per successful mutation and expects 200 responses + (`tests/model-visibility-management-api.test.ts:49-89,92-106`). It must use the + paired seam and assert the response-side catalog disposition. +- `tests/management-provider-validation.test.ts` pins no refresh for dedicated + provider mode changes and one refresh for ordinary provider edits + (`tests/management-provider-validation.test.ts:1751-1823,1843-1865`). Those call + counts remain, but the injected contract and response assertions change. +- `tests/combo-management-api.test.ts` has a `Promise` helper and a real + callback that invokes `syncCatalogModels` to prove DELETE retires the final + combo row (`tests/combo-management-api.test.ts:117-142,682-725`). It must drive + gather then commit through the new seam. +- `tests/combos.test.ts` and `tests/codex-v2-gate.test.ts` encode the same + `() => Promise` dependency in helpers or route fixtures + (`tests/combos.test.ts:119-143`, `tests/codex-v2-gate.test.ts:637-745`). They + need paired no-write stubs and typed success outcomes. +- Stub-only management fixtures in integration, client-config, response-shadow, + and combo-failover tests inject the old void callback to prevent real-home + writes (`tests/management-integration-routes.test.ts:125`, + `tests/management-client-config-route.test.ts:84-96,246`, + `tests/responses-shadow-intercept.test.ts:200`, + `tests/server-combo-failover-e2e.test.ts:346`). They break at the dependency + type even where their exercised route never refreshes. +- `tests/catalog-input-modality-enum.test.ts` constructs a direct + `ManagementContext` with a void best-effort helper and therefore breaks when + that context returns a typed outcome + (`tests/catalog-input-modality-enum.test.ts:90`). + +No caller depends on fire-and-forget timing. All production sites await the +operation, and route tests count refreshes synchronously after the response +(`src/server/management/provider-routes.ts:147,338,487,512,527,546`, +`src/server/management/model-routes.ts:214,313,352,390,404,440`, +`tests/model-visibility-management-api.test.ts:78-106`). What callers do depend +on is failure isolation: management mutations still succeed after their primary +write, and `syncModelsToCodex` still injects when ordinary catalog refresh fails +(`src/server/management-api.ts:105-112`, `tests/codex-sync-api.test.ts:148-166`). + +## 7. Risks and the hardest problem + +The main risk is an optimistic-concurrency window: gather reads config and the +existing catalog, waits on provider discovery, and assembles a merge while +another process can change either input +(`src/codex/catalog/sync.ts:513-526,430-468`). A lock acquired only for commit +orders writers but does not prove that the candidate was built from the state now +being overwritten +(`devlog/_plan/260803_codex_desktop_toggle/030_desired_state.md:250-310`). + +**INFERRED:** the single hardest problem is defining the revision key that makes +that proof exact without rerunning slow work under the lock. A config-only digest +is insufficient because a newer refresh or Codex itself can replace the base +catalog; a catalog-only digest is insufficient because provider enablement, +allowlists, custom models, combos, modality metadata, and credentials all affect +assembly/discovery +(`src/codex/catalog/sync.ts:513-565`, +`src/codex/catalog/provider-fetch.ts:410-490,696-717`). The candidate therefore +needs both a catalog-affecting config digest and target/base-catalog revision, and +commit must reject rather than “best effort” overwrite when either changed. + +Other risks follow from that decision: + +- Conservative full-config fingerprinting can cause harmless retries after an + unrelated config edit, but a hand-maintained subset can miss a future + catalog-affecting field; **INFERRED:** conservative invalidation is safer for + Part 1 because catalog inputs already span providers, visibility, combos, and + subagent settings (`src/codex/catalog/sync.ts:533-565`). +- Backup, catalog, and cache writes are separately atomic, so a disk error can + leave a catalog/cache mismatch; the receipt and later convergence must make + that partial state visible (`src/codex/catalog/sync.ts:527-531,568,601-616`). +- A second candidate gathered from the same initial revision can wait behind the + first and then overwrite it; base-catalog revision validation makes the second + return `stale_candidate` and regather instead + (`src/codex/catalog/sync.ts:513-525,568`). +- Current provider discovery frequently degrades instead of throwing, so a typed + outcome added only around exceptions would still lose network/auth evidence; + notices must be gathered from the provider discovery result/status path + (`src/codex/catalog/provider-fetch.ts:494-570,616-632`). +- The bundled fallback currently writes while loading; leaving that one call + untouched would preserve the audit failure even if the final catalog/cache + writes move (`src/codex/catalog/bundled.ts:213-234`). + +The seam is sufficient only when gather can be paused indefinitely with zero +native write, OFF or a config/catalog revision change can win during that pause, +and commit either performs the fixed prepared write set or returns a typed no-write +or partial-write receipt +(`devlog/_plan/260803_codex_desktop_toggle/030_desired_state.md:585-597`, +`devlog/_plan/260803_codex_desktop_toggle/008_audit_synthesis_wp4_r2.md:19-24`). diff --git a/devlog/_plan/260804_codex_write_substrate/002_history_off_the_loop.md b/devlog/_plan/260804_codex_write_substrate/002_history_off_the_loop.md new file mode 100644 index 000000000..ff2551e9a --- /dev/null +++ b/devlog/_plan/260804_codex_write_substrate/002_history_off_the_loop.md @@ -0,0 +1,486 @@ +# Part 2 — history mutation must leave the server event loop + +Research doc. No implementation diffs here. This records the blocking boundary, +the process split, and the acceptance test the later phase design must satisfy. + +## The failure this substrate must prevent + +A dashboard OFF cannot call the current history restore in the server process. +`restoreNativeCodex()` calls `syncCodexHistoryProvider("openai")` synchronously, +and `POST /api/stop` calls `restoreNativeCodex()` before it schedules drain and +exit (`src/codex/inject.ts:764-794`, `src/server/management-api.ts:167-194`). +The incident audit rejected that shape because two 5-second SQLite waits, one +500 ms synchronous sleep, and row-dependent file work can freeze the listener +that is meant to keep serving every other client +(`devlog/_plan/260803_codex_desktop_toggle/008_audit_synthesis_wp4_r2.md:25-30`). + +The invariant for this part is therefore narrower than “history eventually +converges”: while a Codex history operation is waiting on SQLite or walking +rollouts, `/healthz` and existing data-plane traffic must continue to run on the +server event loop (`src/server/index.ts:536-550`). + +## What the current history path does synchronously + +The normal provider-sync path is synchronous from entry to return. The exported +function runs a synchronous no-op probe when requested, otherwise wraps the +entire unsafe mutation in a synchronous retry loop +(`src/codex/history-provider.ts:551-579`). + +### SQLite wait and retry budget + +- Every normal writable database connection sets `PRAGMA busy_timeout` from + `historyDbBusyTimeoutMs`, whose production default is 5,000 ms + (`src/codex/history-provider.ts:25-49`). +- `withHistoryRetry` makes two attempts by default. After the first recoverable + failure it calls `Bun.sleepSync(500)` before trying the whole mutation again + (`src/codex/history-provider.ts:526-548`). +- The lock-contention portion can therefore consume roughly `5,000 + 500 + + 5,000 = 10,500 ms`. This is not a whole-operation ceiling: it excludes every + row scan, rollout read/write, fsync, manifest operation, and successful SQLite + statement before or after the contended statement + (`src/codex/history-provider.ts:526-548`, + `src/codex/history-provider.ts:581-699`). +- Recoverable currently includes SQLite busy/locked, filesystem busy, and + permission failures. All of those collapse to `failed: true`; hard failures + throw (`src/codex/history-provider.ts:511-524`, + `src/codex/history-provider.ts:577-578`). + +### Rows, rollouts, and database writes + +The forward direction materializes every resumable `openai` row and every +interactive `opencodex`/`exec` row with synchronous `.all()` calls. It then +iterates both arrays once for rollout mutation and again inside the SQLite +transaction before executing two set-based updates +(`src/codex/history-provider.ts:585-650`). + +The restore direction reads every backup entry into an array, mutates each +entry's rollout, updates each entry inside a transaction, consumes the manifest, +then queries and iterates every remaining interactive `opencodex` row in the +ejection pass (`src/codex/history-provider.ts:656-695`, +`src/codex/history-provider.ts:475-508`). When the manifest is empty, restore +goes directly to that unbounded ejection query (`src/codex/history-provider.ts:656-665`). + +For each rollout selected for a provider/source change, the path can do all of +the following synchronously: + +1. Read the entire JSONL file into memory, split it into lines, and scan backward + for the latest `session_meta` (`src/codex/history-provider.ts:258-274`, + `src/codex/history-provider.ts:419-431`). +2. Reopen the file read/write and grow a 64 KiB probe until the first newline, + with a 16 MiB first-line stop; a safe shrink writes line one in place and + calls `fsyncSync` (`src/codex/history-provider.ts:102-157`). +3. Reopen with `O_APPEND`, write the trailing metadata line in a loop, call + `fsyncSync` again, and close the descriptor (`src/codex/history-provider.ts:67-79`, + `src/codex/history-provider.ts:444-457`). + +The line-one probe is bounded at 16 MiB, but the preceding full-rollout read is +not. The selected database row count, backup-entry count, number of rollout +files, total rollout bytes, and filesystem flush latency have no cap in this +module (`src/codex/history-provider.ts:102-124`, +`src/codex/history-provider.ts:263-274`, +`src/codex/history-provider.ts:585-650`, +`src/codex/history-provider.ts:656-695`). + +**INFERRED cost model:** for forward sync, the unbounded term is driven by +`N = openaiRows + execRows` and the sum of those rollouts' bytes. For restore it +is driven by `E = backup entries`, plus `R = remaining opencodex rows` found by +the ejection pass, and the sum of the corresponding rollout bytes. A changed +rollout can add two synchronous fsyncs, so file count matters independently of +byte count (`src/codex/history-provider.ts:67-79`, +`src/codex/history-provider.ts:102-157`, +`src/codex/history-provider.ts:585-650`, +`src/codex/history-provider.ts:656-695`). There is no finite worst-case duration +to quote from the code; `~10.5 s + row/file work` is the honest bound statement. + +### Backup manifest handling + +The manifest path is derived from the normalized `state_5.sqlite` path and lives +under the OpenCodex config directory. Reading it uses synchronous existence, +whole-file read, and JSON parse; writing either synchronously unlinks an empty +manifest or creates its parent and atomically rewrites the whole JSON object +(`src/codex/history-provider.ts:16-22`, +`src/codex/history-provider.ts:204-227`, `src/config.ts:190-230`). Invalid JSON, +an invalid version/shape, or a manifest naming a different state DB is treated as +an empty manifest (`src/codex/history-provider.ts:204-217`). + +Forward sync inserts every selected row into that in-memory object before one +whole-manifest write (`src/codex/history-provider.ts:229-238`, +`src/codex/history-provider.ts:606-608`). Restore materializes all manifest +values and removes the manifest only after the transaction succeeds +(`src/codex/history-provider.ts:656-691`). Manifest size is therefore another +row-count-dependent synchronous term, not a constant-size marker. + +### Other synchronous work in the file + +The quarantine reconstruction helpers are not on the normal provider-sync call +graph, but they are also synchronous: plain rollouts use `readFileSync`; `.zst` +rollouts use `readFileSync` plus `zstdDecompressSync`, capped at 64 MiB decoded, +then scan every JSONL line (`src/codex/history-provider.ts:14-14`, +`src/codex/history-provider.ts:340-389`). They must remain outside the server +event loop if a later caller exposes them through management. + +The read-only pending probe still synchronously reads/parses the whole manifest, +opens SQLite read-only, sets a 100 ms busy timeout, and performs `count(*)` +(`src/codex/history-provider.ts:743-775`). It is short with respect to SQLite +contention, but its manifest read remains proportional to manifest size. + +## Which process blocks today + +The distinction is not “CLI command versus dashboard button”; it is the process +in which the final call executes. + +| Entry | Process that executes history work | Consequence | +|---|---|---| +| `ocx start` | **Server process.** The listener is bound first, then the same process awaits `syncModelsToCodex`, whose injection calls history sync/migration. | The already-live listener can stop progressing during startup history work. (`src/cli/index.ts:195-204`, `src/cli/index.ts:318-322`, `src/codex/sync.ts:49-58`, `src/codex/sync.ts:110-110`, `src/codex/inject.ts:598-603`) | +| History migration guardian | **Server process.** `handleStart` creates it after sync; each timer tick calls the synchronous pending probe and then a one-attempt migration. | It removes `sleepSync`, but one 5-second SQLite wait and all row/file work still run on the event loop. (`src/cli/index.ts:318-322`, `src/codex/history-migration-guardian.ts:43-70`, `src/codex/history-provider.ts:713-731`) | +| `POST /api/sync` | **Server process.** The management route awaits `syncModelsToCodex`, which reaches injection and history. | A dashboard/provider refresh can block all clients. (`src/server/management/config-routes.ts:261-268`, `src/codex/sync.ts:83-110`, `src/codex/inject.ts:598-603`) | +| `POST /api/stop` | **Server process.** It calls `restoreNativeCodex` synchronously before scheduling drain. | This is the direct incident path: the server is still serving when history restore runs. (`src/server/management-api.ts:167-194`, `src/codex/inject.ts:764-794`) | +| Signal/exit cleanup | **Server process.** `syncCleanup` may restore after the async drain, and is also registered on process exit. | It can delay shutdown; after drain it is not the normal availability hazard, but it uses the same blocking primitive. (`src/cli/index.ts:242-265`, `src/cli/index.ts:284-310`) | +| `ocx stop` | **Both.** The CLI first asks the live proxy to `POST /api/stop`; after that returns/exits, `handleStop` calls restore again in the CLI process. | The first restore can freeze the server; only the second restore is acceptably self-blocking. (`src/lib/process-control.ts:55-94`, `src/cli/index.ts:479-528`) | +| `ocx restore` / `eject` | **CLI process.** It calls restore directly. | Blocking only that command is acceptable, provided its result remains truthful. (`src/cli/index.ts:745-775`) | +| `ocx sync`, `restore back`, or `ensure` against an already-live proxy | **CLI process.** These commands call `syncModelsToCodex` in their own process. | Their own terminal can wait without starving the proxy. (`src/cli/index.ts:365-380`, `src/cli/index.ts:747-760`, `src/cli/index.ts:827-840`) | +| `ocx ensure` when it starts a proxy | **Both.** The child runs the startup sync after binding; the parent later performs another sync. | The child's startup sync is a server-loop hazard; the parent's sync is not. (`src/cli/index.ts:384-412`, `src/cli/index.ts:195-204`, `src/cli/index.ts:318-322`) | +| `ocx service stop` / uninstall | **Server, then CLI, when a tracked proxy is live.** Service cleanup reaches `stopProxy`, which uses `/api/stop`; the service command later restores in its own process. | As with `ocx stop`, the server-side first pass is the hazard. (`src/service.ts:2172-2197`, `src/lib/process-control.ts:64-94`, `src/service.ts:2564-2595`, `src/service.ts:2610-2632`) | +| `ocx recover-history --legacy-openai` | **CLI process.** It calls the legacy restore directly. | Its synchronous wait is local to the command. (`src/cli/index.ts:711-724`) | + +There is no current Codex toggle management route. The native-integration module +explicitly excludes Codex because its state spans multiple artifacts and a live +database (`src/server/management/native-integration-routes.ts:1-15`). Any new +dashboard OFF route would become a server-process caller unless the substrate +changes this boundary. + +## What `skipWhenProvablyNoop` actually buys + +`restoreNativeCodex` enables `skipWhenProvablyNoop` only for Design B/loopback, +derived as `!shouldInjectApiAuthHeader(loadConfig())`; unreadable config and +legacy/non-loopback mode retain the unconditional write-open behavior +(`src/codex/inject.ts:775-783`). + +When enabled for the `openai` direction and the state DB exists, the optimization +runs `countPendingOpencodexHistory`. It skips only when the read succeeds, zero +interactive rows remain tagged `opencodex`, and the backup manifest has zero +entries (`src/codex/history-provider.ts:551-575`). A failed/unknown probe, any +pending row, any backup entry, a non-`openai` direction, or a caller that omitted +the option falls through to the full write attempt +(`src/codex/history-provider.ts:551-579`). + +The migration helper has the same proof-of-no-work gate unconditionally, while +its guardian reduces attempts to one (`src/codex/history-provider.ts:713-731`, +`src/codex/history-migration-guardian.ts:43-45`). Existing tests pin both the +byte-identical steady-state skip and the required fall-through when work remains +(`tests/codex-history-provider.test.ts:398-422`, +`tests/codex-history-provider.test.ts:443-456`). + +It helps the server only in the already-converged steady state. It does not help +the OFF case this unit must make safe: routed threads or backup entries are the +reason history work is necessary, and either fact deliberately disables the +skip. It also cannot bound the file work after a successful probe identifies +pending rows (`src/codex/history-provider.ts:560-579`, +`src/codex/history-provider.ts:656-695`). + +## Options + +### A. Fail-fast automatic convergence only + +Automatic/server callers can use one attempt and a very short writable SQLite +busy timeout, then return a classified deferred result instead of sleeping or +waiting five seconds. The retry helper already proves that `attempts: 1` performs +no sleep, and the read probe already establishes a 100 ms precedent +(`src/codex/history-provider.ts:536-548`, +`src/codex/history-provider.ts:743-775`, +`tests/codex-history-provider.test.ts:358-369`). + +User observation: OFF can return quickly under active SQLite contention with +“Codex is off; history is pending because the database is busy.” The proxy keeps +serving because the lock wait is short, but a small event-loop pause remains. + +Correctness: no history mutation is declared complete on contention; desired OFF +and config/catalog removal can remain committed while history is explicitly +pending. The next automatic retry or a manual CLI restore can finish it. + +Limit: fail-fast bounds only SQLite waiting. If the lock is free, the same server +call can still iterate an unbounded row set, read unbounded rollout bytes, and +fsync per changed file (`src/codex/history-provider.ts:585-650`, +`src/codex/history-provider.ts:656-695`). **INFERRED:** option A alone cannot prove +the `/healthz` invariant for a large but uncontended history. + +GUI requirement: render OFF and history convergence as separate facts. A green +OFF badge cannot imply that routed threads are visible; the previous restore +result already allows `success: true` while only the message says history failed +(`src/codex/inject.ts:787-794`, +`devlog/_plan/260803_codex_desktop_toggle/001_native_restore_thesis.md:92-110`). + +### B. Move the entire history operation off the event loop + +The repository already uses Bun `Worker` for synchronous filesystem/SQLite +storage work specifically to keep it off the proxy event loop +(`src/storage/policy-worker.ts:1-5`, `src/storage/policy-job.ts:295-344`). A +separate liveness test blocks that Worker for 1.2 seconds while sampling +`/healthz` and an active stream (`tests/storage-restore-job-responsive.test.ts:152-216`). + +User observation: the OFF request can remain in “converging history” while the +proxy continues serving. With an async job response, the route can return before +an unbounded history walk completes; the GUI polls the durable state. + +Correctness: the worker runs the existing ordered mutation and reports its +structured result. Server shutdown must join or terminate it without allowing a +later worker to overlap the same history files. + +Bun-specific constraint from this repository: `Worker.terminate()` does not wait +for thread reclamation. Windows and macOS need explicit close tracking, +serialized spawns, and post-close settle time; the existing lifecycle records +1,500 ms on Windows and 250 ms on macOS +(`src/storage/worker-lifecycle.ts:1-17`, +`src/storage/worker-lifecycle.ts:43-60`, +`src/storage/worker-lifecycle.ts:150-209`). Reusing the idea without its lifecycle +discipline would trade an event-loop freeze for teardown races. + +A second Bun/test constraint is path binding. `history-provider.ts` derives +`STATE_DB_PATH` from a module-level `CODEX_HOME` constant +(`src/codex/history-provider.ts:16-22`, `src/codex/paths.ts:6-29`). Existing +integration tests use a subprocess with `CODEX_HOME` and `OPENCODEX_HOME` set +before import for exactly this reason +(`tests/codex-inject-integration.test.ts:13-42`). A history Worker must receive +explicit resolved state/backup paths, or set its environment before dynamically +importing the module; it must not assume a parent test's late environment mutation +is visible. The storage Worker already passes explicit home/env data because +Workers may not see parent mutations on every platform +(`src/storage/policy-worker.ts:7-18`, `src/storage/policy-worker.ts:30-44`). + +A subprocess gives stronger crash and module-environment isolation, and this repo +already launches Bun/TypeScript child work with `process.execPath` +(`tests/codex-inject-integration.test.ts:15-42`, +`src/update/job.ts:390-409`). Its costs are process startup, a separate IPC/result +contract, and cross-platform child ownership. Detached Windows children can also +inherit a listener handle, which the update path avoids with a platform-specific +launcher (`src/update/job.ts:380-430`). **INFERRED:** a non-detached, owned child +can avoid that particular leak, but it still needs explicit shutdown and output +bounds. + +GUI requirement: show `converging` while the worker/job is active, then either +`converged` or an actionable unresolved state. Do not hold the switch in an +indeterminate visual state with no durable status behind it. + +### C. Worker plus fail-fast automatic mode — recommended + +Use an owned Worker for every server-process history mutation, and give automatic +attempts a short SQLite timeout with no synchronous retry sleep. Keep the current +full retry budget for explicit CLI-only recovery, where blocking the invoking +terminal does not deny proxy service (`src/codex/history-provider.ts:526-548`, +`src/cli/index.ts:711-724`, `src/cli/index.ts:745-775`). + +The Worker is an execution boundary, not mutation authority. Part 4's read-only +ownership/provenance admission must pass before dispatch; `SQLITE_BUSY`, +permissions, or an unreadable manifest remain unresolved capability facts and do +not grant permission to mutate (`devlog/_plan/260804_codex_write_substrate/004_ownership_and_convergence.md:49-71`, +`devlog/_plan/260804_codex_write_substrate/004_ownership_and_convergence.md:203-214`). + +This combination closes both dimensions of the incident: + +- The Worker moves successful large row/file walks and fsyncs off the listener's + event loop (`src/storage/policy-worker.ts:1-5`, + `src/storage/policy-job.ts:295-344`). +- Fail-fast mode prevents one automatic job from occupying the history mutation + slot for 10.5 seconds when Codex owns the database, and leaves retries to the + durable convergence scheduler (`src/codex/history-provider.ts:526-548`, + `src/codex/history-migration-guardian.ts:54-92`). + +Prefer the in-repo Worker pattern over a new subprocess protocol unless Worker +teardown testing finds a history-specific Bun defect. The project already owns +admission, spawn serialization, close tracking, timeout, and drain concepts for +Workers (`src/storage/worker-lifecycle.ts:30-44`, +`src/storage/worker-lifecycle.ts:92-143`, +`src/storage/worker-lifecycle.ts:150-209`). **INFERRED:** history needs its own +single-flight/admission key rather than sharing storage cleanup's global worker +slot, because the resources and user-visible job states are different; both must +still be drained during server shutdown. + +## “Unresolved history” is a durable fact + +The backup manifest is recovery material, not a sufficient status record. It can +be empty while no-backup `opencodex` rows still need the ejection path, and a +failed read probe returns zero-looking counts with `failed: true` +(`src/codex/history-provider.ts:656-665`, +`src/codex/history-provider.ts:734-775`). The GUI must never derive “done” from +manifest absence or numeric zero while `failed` is set. + +Record convergence under the OpenCodex config root, not in `CODEX_HOME`. The +recommended concrete location is `getConfigDir()/integrations/codex.json`, with +desired integration state and history convergence in the same atomic record; do +not create a second history-only status file. The existing integration state +convention places owned durable records below `getConfigDir()/integrations` and +writes them atomically +(`src/integrations/ownership.ts:60-66`, +`src/integrations/ownership.ts:74-106`). For this unit, the concrete durable fact +should live in the Codex desired-state record from Part 1, with a history section +keyed by the normalized state-DB identity already used for backup naming +(`src/codex/history-provider.ts:16-22`). + +The record needs these semantics, not merely these labels: + +| Field | Meaning | +|---|---| +| desired integration state | `off` is committed before the history attempt. It is the authority that makes a restart retry removal instead of re-injecting. | +| history state | `pending`, `running`, `blocked`, `converged`, or `unknown`; `converged` is legal only after a clean post-probe reports zero pending rows and zero backup entries. | +| reason | At minimum distinguish SQLite contention from permission failure, unreadable/schema-unknown state, worker failure/timeout, and cancellation during shutdown. Current `failed: true` conflates these classes (`src/codex/history-provider.ts:511-524`, `src/codex/history-provider.ts:734-775`). | +| evidence | Last-attempt time, attempt count, nullable pending-row/backup-entry counts, and the state-DB identity. Counts are null/unknown when the probe failed. | +| retry | Next eligible retry time and whether automatic retry remains armed. | + +**INFERRED state rule:** write `pending` in the same durable desired-state +operation before dispatching the Worker; change it to `converged` only after the +Worker succeeds and a clean post-probe proves both counts are zero. A crash +between those writes therefore leaves a retryable false negative (“pending even +if work landed”), never a false green. The existing guardian already uses a +post-probe before treating zero-row success as done +(`src/codex/history-migration-guardian.ts:68-83`). + +Retry ownership should be explicit: + +- The running server schedules bounded automatic attempts while desired state is + OFF; its current guardian already has unref'd timed ticks and a finite budget, + but today it stops after about an hour and only logs the failure + (`src/codex/history-migration-guardian.ts:34-40`, + `src/codex/history-migration-guardian.ts:54-92`). +- Every proxy startup re-arms a durable pending/unknown record; startup must + dispatch the Worker rather than run the mutation inline. The current startup + starts its guardian only after the synchronous initial sync, which is the + boundary this unit changes (`src/cli/index.ts:318-322`). +- An explicit CLI restore/recover command can request the full retry budget in + its own process and must update the same durable result afterward + (`src/cli/index.ts:711-724`, `src/cli/index.ts:745-775`). + +The user learns the truth through all control surfaces. The Codex integration +GET response should expose desired state plus history state/reason/counts; the +GUI should render “Off — history pending” and say routed threads can remain hidden +until retry succeeds. `ocx doctor` already reports clean, pending counts, and +locked/unreadable unknown state from the read-only probe, so it should add the +durable reason/next retry rather than invent a second definition +(`src/cli/doctor.ts:891-902`). + +## Acceptance tests + +### The gate: real SQLite contention while `/healthz` stays responsive + +Add a server-boundary test beside the existing Worker responsiveness test. It +must use isolated `CODEX_HOME` and `OPENCODEX_HOME`; the repository already has a +helper that creates and restores an isolated Codex home +(`tests/helpers/isolated-codex-home.ts:1-23`). + +Concrete method: + +1. In the isolated Codex home, create a production-shaped `state_5.sqlite`, one + interactive `opencodex` row, and its rollout. The current fixture schema and + row shape are pinned in `tests/codex-history-provider.test.ts:27-89`. +2. Spawn an owned Bun child process that opens that exact SQLite file, executes + `BEGIN IMMEDIATE`, writes a ready marker, and holds the transaction until a + release marker appears. Existing multiprocess lock tests use the same owned + child + marker handshake and enforce cleanup in `finally` + (`tests/oauth-refresh-lock-multiprocess.test.ts:58-101`, + `tests/config-mutation-lock.test.ts:48-92`). This is real cross-process SQLite + writer contention, not a mocked `SQLITE_BUSY`. +3. After the ready marker, start `startServer(0)` and issue the future Codex OFF + management request. Give the history Worker a test-only busy timeout/hold long + enough to sample deterministically (for example 1,200 ms), while production + automatic mode remains short. The server test seam already forwards management + dependencies after authentication (`src/server/index.ts:351-367`, + `src/server/index.ts:541-550`). +4. Do not await OFF first. While its promise is pending, issue at least six + `/healthz` requests at 40 ms intervals, require every response to be 200, and + assert each post-warmup latency is below one third of the contention window. + This is the existing measured liveness pattern, not a sleep-and-assume check + (`tests/storage-restore-job-responsive.test.ts:175-210`). +5. Also keep one streaming data-plane response active during the same window and + prove all chunks arrive. `/healthz` alone proves listener scheduling; the + stream proves an already-admitted client continues to receive service + (`tests/storage-restore-job-responsive.test.ts:182-210`). +6. Release the child transaction in `finally`, await its zero exit, await/drain + the history Worker, and shut the server down. Worker tests must join threads + before deleting homes because Bun termination is not a join + (`src/storage/worker-lifecycle.ts:1-17`, + `tests/storage-restore-job-responsive.test.ts:53-80`). +7. Assert the OFF response/status says history is unresolved with reason + `sqlite_busy`, that the durable record survives a fresh server instance, and + that a later retry after lock release converges and clears the warning. A clean + post-probe must report both counts as zero + (`src/codex/history-provider.ts:734-775`, + `src/codex/history-migration-guardian.ts:68-83`). + +The liveness pass condition is quantitative: the OFF operation is demonstrably +still contending while every health sample stays well below the contention +window and the stream completes. A unit test that merely injects a busy error, or +a test that checks `/healthz` only before and after OFF, does not exercise the +incident. + +### Other tests required by the recommendation + +- `tests/codex-history-worker.test.ts` — Worker result parity: forward, manifest restore, no-manifest ejection, + line-one patch, trailing append, and manifest consumption match the existing + synchronous results (`tests/codex-history-provider.test.ts:92-290`). +- `tests/codex-history-convergence.test.ts` — automatic fail-fast: one real busy attempt performs no `sleepSync`, persists + unresolved state, and schedules a later attempt; hard errors retain their + classified reason (`tests/codex-history-provider.test.ts:293-369`, + `tests/history-migration-guardian.test.ts:40-79`). +- `tests/codex-history-worker.test.ts` — no-op: clean Design B state does not spawn a Worker or write the DB/rollout; + pending rows and manifest entries do spawn one + (`tests/codex-history-provider.test.ts:398-456`, + `tests/history-migration-guardian.test.ts:24-38`). +- `tests/codex-history-convergence.test.ts` — crash/cancel: killing or timing out the Worker leaves durable `pending` or + `unknown`, never `converged`, and the next server start retries. Teardown joins + before a second worker starts (`src/storage/worker-lifecycle.ts:123-143`, + `src/storage/worker-lifecycle.ts:150-209`). +- `tests/codex-history-worker-responsive.test.ts` — the real-contention `/healthz` + and active-stream measurement specified above + (`tests/storage-restore-job-responsive.test.ts:152-216`). +- `tests/codex-history-process-routing.test.ts` — process routing: direct CLI restore uses the full synchronous budget; startup, + `/api/sync`, `/api/stop`, guardian, and the future toggle never call the + synchronous mutation on the server event loop. The current call sites to pin + are `src/cli/index.ts:318-322`, `src/server/management/config-routes.ts:261-268`, + `src/server/management-api.ts:167-194`, and + `src/codex/history-migration-guardian.ts:59-83`. +- `tests/codex-integration-history-state.test.ts` plus the owning GUI test — API/GUI truth: OFF with pending history is not a success-only envelope; state, + reason, counts/unknown, retry, and the hidden-thread warning survive reload. + The current string-only failure attached to `success: cfg.success` is the + regression target (`src/codex/inject.ts:787-794`). + +## Existing tests that pin current behavior + +`tests/codex-history-provider.test.ts` is the primary contract. It pins forward +retagging, append-only metadata, line-one in-place restoration, oversized first +line handling, backup restore, foreign-DB manifest refusal, exec-source repair, +no-backup ejection, and explicit legacy recovery +(`tests/codex-history-provider.test.ts:92-290`). It also pins recoverable error +classification, two-attempt retry, no retry for hard errors, custom attempt +budgets, the no-sleep one-attempt mode, pending counts, idempotent migration, +missing-DB manifest retention, and `skipWhenProvablyNoop` +(`tests/codex-history-provider.test.ts:293-456`). + +`tests/history-migration-guardian.test.ts` pins the current scheduler semantics: +no work in steady state, retries through contention, finite give-up, cancellation, +retry after an unknown probe, and refusal to call zero-row success complete while +backup entries remain (`tests/history-migration-guardian.test.ts:24-136`). + +`tests/codex-inject-integration.test.ts` pins isolated-process path binding and +the external-provider guard that prevents history/config mutation when another +provider owns Codex (`tests/codex-inject-integration.test.ts:13-42`, +`tests/codex-inject-integration.test.ts:247-355`). + +The service/CLI tests currently pin call ordering only: service stop restores +after tracked proxy stop, service uninstall restores after uninstall, and the CLI +exposes restore/legacy recovery (`tests/service.test.ts:646-661`, +`tests/uninstall.test.ts:17-27`). No existing Codex-history test measures server +liveness under real SQLite contention; the closest proven pattern is the storage +Worker liveness test (`tests/storage-restore-job-responsive.test.ts:152-216`). + +## Recommendation + +Choose **C: an owned Worker for every server-process history operation, plus a +fail-fast automatic SQLite budget and durable unresolved-history state**. Keep +the full synchronous retry path only for explicit CLI-process recovery. + +The hardest constraint is not the 10.5-second lock budget. It is that a successful +history mutation has no finite work bound: row count, manifest entries, rollout +count, total JSONL bytes, and fsync latency all scale with user history +(`src/codex/history-provider.ts:585-650`, +`src/codex/history-provider.ts:656-695`). Only moving the whole mutation off the +server event loop can satisfy the availability invariant in both contended and +uncontended large-history cases. diff --git a/devlog/_plan/260804_codex_write_substrate/003_lock_protocol.md b/devlog/_plan/260804_codex_write_substrate/003_lock_protocol.md new file mode 100644 index 000000000..ba09d5991 --- /dev/null +++ b/devlog/_plan/260804_codex_write_substrate/003_lock_protocol.md @@ -0,0 +1,453 @@ +# Codex write substrate, part 3 — lock protocol + +The failure to prevent is not merely two writers entering together. The failed +OFF design could block Bun's event loop while waiting, could derive two lock keys +for one symlinked default home, and could let a hostile entry in a shared temp +directory decide where SQLite opened. Its own race test also required an OFF +setter to wait for an apply and then remove it, but the proposed synchronous API +never said whether contention waited, timed out, or failed immediately +(`devlog/_plan/260803_codex_desktop_toggle/030_desired_state.md:284-303,344-350`). +Audit therefore rejected the design as a missing concurrency substrate, not as a +missing boolean (`devlog/_plan/260803_codex_desktop_toggle/008_audit_synthesis_wp4_r2.md:46-58`). + +This document specifies that substrate boundary. It is research and protocol, +not an implementation diff. + +## Decision + +Use an **async, cross-process, SQLite-backed per-canonical-`CODEX_HOME` lock**. +Acquisition has a required finite deadline and returns one of `acquired`, `busy`, +or `refused`; expected contention is never an exception. SQLite's OS lock is the +holder authority. There is no PID/mtime lease takeover and no stale-lock unlink. +Acquiring it proves exclusion only, never ownership: service-home, external-provider, +journal, provenance, and desired-state admission must pass read-only before the +caller asks the lock module to create or open anything, then be rechecked under +the lock (`devlog/_plan/260804_codex_write_substrate/004_ownership_and_convergence.md:106-139`). + +The protocol is deliberately **barging-allowed**, not FIFO. A contender sleeps +asynchronously with bounded jitter and retries `BEGIN IMMEDIATE` until its +deadline. This gives no request-order or starvation guarantee when three or more +processes contend. It does give the property WP4 actually needs: with one current +holder and one waiting OFF setter, the setter either acquires after release and +then removes, or returns typed `busy` at its deadline. A test may assert that +two-party sequence; it may not assert global arrival order. + +The database lives in a private, per-OS-user namespace derived from the canonical +login home, not `tmpdir()`, not `CODEX_HOME`, and not `OPENCODEX_HOME`: + +```text +/.opencodex/native-write-locks/v1/ + .sqlite +``` + +`` is the full lowercase hexadecimal SHA-256 of +`"opencodex-codex-write-lock-v1\0" + canonicalCodexHome`. The domain prefix makes +the key unusable as an accidental alias for another hash namespace. Keeping all +64 hex characters avoids introducing a truncation collision into a correctness +key. + +**INFERRED:** a fixed home-relative namespace is preferable to a platform/fallback +ladder here. Two processes owned by one user must derive the same lock even when +one has `XDG_RUNTIME_DIR` and another does not, or when one has a custom +`OPENCODEX_HOME`. The repository has no general runtime-directory resolver: its +only `XDG_RUNTIME_DIR` logic discovers the systemd user bus and falls back to +`/run/user/` (`src/service.ts:1977-2007`); `LOCALAPPDATA` is used to discover +third-party Windows data, not OpenCodex coordination (`src/codex/plugins-doctor.ts:111`, +`src/claude/desktop-3p-paths.ts:60`). OpenCodex's own default per-user state root +is already `~/.opencodex` (`src/config.ts:527-535`). + +## Existing coordinator: config mutation + +`withConfigMutationLockSync` is the first prior art to preserve. + +| Question | Current answer | +|---|---| +| Where | `getConfigDir()/config-mutation.sqlite`; sidecars are registered as owned config paths (`src/config.ts:1718-1762`). | +| Preparation | It asserts the test-home boundary before creating anything, creates/chmods the config directory to `0700`, and attempts Windows ACL hardening (`src/config.ts:1731-1756`). | +| Acquisition | It opens Bun SQLite, chmods the database to `0600`, sets `busy_timeout=0`, then executes `BEGIN IMMEDIATE` (`src/config.ts:1784-1791`). | +| Busy behavior | Fail immediately. `SQLITE_BUSY` becomes `ConfigMutationLockError("Config mutation already in progress")`; every other open/acquisition failure becomes the same typed exception with a different message and original `cause` (`src/config.ts:1792-1800`). | +| Critical section | The callback is synchronous. The transaction commits after it returns; callback or commit failure rolls back and rethrows (`src/config.ts:1803-1814`). | +| Release | Closing the SQLite handle releases the OS lock in `finally` (`src/config.ts:1815-1818`). | +| Stale holder | There is no lease or stale-file deletion. Process exit releases the SQLite lock (`src/config.ts:1767-1773`); the abrupt-exit regression proves a later writer acquires without recovery (`tests/config-mutation-lock.test.ts:105-129`). | +| Reentrancy | Reentrant only on the current synchronous call stack through `configMutationLockDepth`; returning a Promise is forbidden (`src/config.ts:1765-1783`). | + +This lock should remain fail-fast and synchronous. It protects short config and +credential commits, and its comment explicitly avoids freezing the Bun event loop +(`src/config.ts:1767-1773`). Changing it into the native-write lock would expand +its critical section across async native work and recreate the audit failure. + +The new coordinator should look like it in three ways: SQLite is the process-crash +authority, `busy_timeout=0` prevents SQLite from synchronously parking Bun, and +the database is persistent while the transaction is ephemeral. It deliberately +differs in three ways: acquisition retries asynchronously to a deadline, expected +outcomes are returned rather than thrown, and async reentrancy is refused rather +than inferred from one process-global depth counter. + +## The repository already has three concurrency families + +Adding another ad hoc `*.lock` algorithm would increase disagreement about stale +ownership, release, and contention. The current tree has these families. + +### 1. OS-backed SQLite transactions + +- Config mutation is synchronous and fail-fast as described above. +- Native-profile switching is already async: it repeatedly opens a stable lock + file, tries `BEGIN IMMEDIATE` with `busy_timeout=0`, sleeps 50 ms on busy, and + returns retryable `NATIVE_PROFILE_BUSY` after a 5 s deadline + (`src/codex/native-profile-manager.ts:77,313-370`). It keeps the transaction + across an awaited operation and releases by rollback/close + (`src/codex/native-profile-manager.ts:373-387`). +- Native-main shared/exclusive claims use the same stable-file helper and SQLite. + The exclusive claim retries asynchronously only until its caller-supplied + deadline (`src/codex/native-main-claim.ts:107-160`). The long-lived owner also + uses `busy_timeout=0; BEGIN IMMEDIATE`, publishes `contended`, and schedules an + async retry (`src/codex/native-main-owner.ts:160-194`). + +This is the family the new lock belongs to. In particular, the stable-file helper +already rejects non-regular/symlink entries, opens with `O_NOFOLLOW` on POSIX, +captures `(dev, ino)`, and detects path substitution after open +(`src/codex/native-main-lock-file.ts:58-125`). Its Windows hardening is already a +required ACL operation (`src/codex/native-main-lock-file.ts:127-131`). The new +coordinator should reuse or generalize that owner; it should not copy those checks +into a fourth file. + +### 2. Exclusive-create files with stale recovery + +- Prompt-layer writes use `flag: "wx"`, PID plus acquisition age, atomic rename + quarantine for stale takeover, and token-checked release + (`src/codex/prompt-lock.ts:28-44,75-121,124-142`). +- OAuth refresh uses an async deadline and jitter, but its lock is an exclusive + file and stale recovery compares snapshots before unlink + (`src/oauth/store.ts:171-193`). +- Codex credential refresh also polls an exclusive file asynchronously, but it + declares staleness from age and unlinks the path (`src/codex/account-store.ts:299-373`). +- Shim autorestore uses an exclusive directory containing a token-named owner + record; it combines PID, creation time, mtime, inode identity, and a second + snapshot before stale deletion (`src/codex/shim.ts:722-801,804-840`). + +These locks exist because their files are themselves the ownership record. They +are not the model for native writes. A paused but live process, PID reuse, clock +jump, or path replacement turns stale-file deletion into correctness policy. +SQLite already provides crash release without any of those guesses. + +### 3. Process-local flights + +- `runIntegrationMutationFlight` joins an identical per-client request for 120 s, + rejects a different request as busy for up to 10 minutes, and forgets the entry + afterward (`src/server/management/integration-routes.ts:35-37,83-95,146-183`). +- Storage cleanup returns `already_running` from one process-local Promise slot + (`src/storage/policy-job.ts:415-438`). +- Credential, quota, lifecycle, image-description, and catalog-prime paths use + Promise/Map single-flights for duplicate suppression; for example Codex token + refresh joins a grant-keyed Promise and separately takes its cross-process file + lock (`src/codex/account-store.ts:290-302,400-433`). + +Flights collapse work in one process. They do not serialize a CLI, service, and +second OpenCodex process targeting the same native home. A process-local flight +may sit in front of the new lock as an optimization, but it cannot be the lock. + +## Acquisition contract + +The public acquisition result is a closed union: + +```ts +type CodexWriteLockAcquireResult = + | { status: "acquired"; handle: CodexWriteLockHandle; waitedMs: number } + | { status: "busy"; reason: "deadline" | "cancelled"; retryable: true; waitedMs: number } + | { status: "refused"; reason: + | "codex_home_missing" + | "codex_home_unsafe" + | "authority_not_proven" + | "namespace_unsafe" + | "lock_path_unsafe" + | "unsupported_filesystem" + | "reentrant" + | "lock_unavailable"; + retryable: false; message: string }; +``` + +This is a shape specification, not an implementation diff. Programmer errors +such as a negative timeout may throw before I/O; filesystem, ACL, SQLite-open, +contention, cancellation, and path-validation outcomes do not. + +Acquisition takes a required `timeoutMs`, an optional `AbortSignal`, and injectable +clock/sleep seams. **INFERRED:** production callers cap `timeoutMs` at 30 s; the +Codex OFF setter uses 15 s, while startup/background convergence uses 5 s. There +is no unbounded default. A zero timeout is a valid fail-fast probe. + +The algorithm is: + +1. Resolve and validate canonical `CODEX_HOME`. Missing or unsafe returns + `refused` before namespace creation. +2. Require the caller's pre-lock authority receipt. A missing/stale receipt is + `refused`; the lock itself does not infer ownership from path access. +3. Resolve and validate the per-user namespace and expected lock path without + following links. +4. Open a stable regular lock file, enforce private ownership/mode/ACL, and retain + its identity handle. +5. Open SQLite with `busy_timeout=0`, force rollback-journal mode, and attempt + `BEGIN IMMEDIATE`. +6. On `SQLITE_BUSY`, close the candidate handles, check cancellation/deadline, + then `await` a jittered 25-75 ms delay and retry from stable-path validation. + Another contender may barge during that delay. +7. On success, re-assert path identity and return `acquired`. The handle owns the + SQLite transaction until explicit release. +8. Release rolls back, closes SQLite, then closes the stable side descriptor. + The persistent database is not unlinked. + +The critical operation may `await`; acquisition waiting and the operation itself +therefore do not stop the server event loop. That is the key difference from the +failed synchronous proposal, whose wrapped history path could synchronously hold +for about 10.5 s (`devlog/_plan/260803_codex_desktop_toggle/008_audit_synthesis_wp4_r2.md:25-30`). + +### Fairness + +Fairness is **weak, deadline-bounded, barging allowed**: + +- no FIFO queue or ticket is claimed; +- jitter reduces lock-step polling but does not establish ordering; +- every waiter terminates by acquisition, cancellation, refusal, or deadline; +- a newly arriving process may acquire before an older sleeper; +- no caller may use acquisition order as business ordering. + +The linearization order is the order of successful SQLite transactions, not +request arrival. The old test phrase “final state follows lock acquisition order” +remains valid (`devlog/_plan/260803_codex_desktop_toggle/030_desired_state.md:589-594`); +“final state follows request arrival order” would be a new and false promise. + +### Stale holder + +PID and mtime are diagnostics only. They are not takeover authority. + +- If a holder exits or crashes, the OS closes its SQLite handle and releases the + transaction. The next retry can acquire, as the config-lock crash test already + demonstrates (`tests/config-mutation-lock.test.ts:105-129`). +- If a holder is alive but hung, it is not stale. Contenders return `busy` at + their deadlines. Stealing from a live process would permit two native writers. +- The database file may outlive every holder. Its age says nothing about lock + ownership and is never grounds for deletion. + +This means there is no PID-reuse problem, no heartbeat lease to tune, and no +unlink race. Operational recovery from a genuinely hung live holder is to stop +that process, not to let another process guess. + +### Reentrancy + +The lock is non-reentrant for the same logical async operation. A nested attempt +for the same canonical home returns `refused/reentrant` immediately. It must not +reuse the config lock's process-global depth counter: while an async holder is +suspended, an unrelated request in the same process could otherwise observe a +positive depth and enter the critical section without owning SQLite. + +Unrelated tasks in the same process are ordinary contenders and may wait. The +implementation therefore needs logical-owner context, not a single global +boolean. A helper that already receives an acquired handle may call internal +`...Unlocked` operations explicitly; it must not acquire again. + +## Namespace and path hardening + +### Why the temp directory is rejected + +`tmpdir()/opencodex-native-locks` is a global name on systems where the temp root +is shared. The first user can create that directory with inaccessible ownership, +and an existing symlink can redirect later lock opens. A requested mode on +`mkdir` or SQLite create does not repair or authenticate an existing entry. The +failed design specified only modes, not identity checks +(`devlog/_plan/260803_codex_desktop_toggle/030_desired_state.md:296-303`). + +The repository uses `tmpdir()` for isolated tests and disposable probes, such as +the Codex runtime probe (`src/codex/runtime.ts:251-253`), not for a production +cross-process ownership namespace. The lock therefore stays under the canonical +OS user's home. + +### Validation rules + +Every path component introduced by OpenCodex is validated before SQLite sees it. + +On POSIX: + +- canonical user home must be an existing directory resolved with + `realpathSync.native`; +- `.opencodex`, `native-write-locks`, and `v1` must each be a real directory, not + a symlink, owned by `process.getuid()`, with effective mode `0700`; +- the database, when present, must be a regular non-symlink file owned by the + same uid with effective mode `0600`; +- open uses `O_NOFOLLOW`, then `fstat`; `lstat` after open must match the retained + `(dev, ino)` before acquisition and before release; +- any existing SQLite sidecar at the expected `-journal`, `-wal`, or `-shm` name + must also be a same-owner regular non-symlink file with mode `0600`. The lock + database uses rollback-journal mode, so WAL/SHM are unexpected residue and cause + refusal rather than silent reuse. + +On Windows: + +- the same `lstat`/realpath identity rules reject symlinks, junctions, and + reparse-point substitution; +- directory and file ACL hardening is required, not best effort, using the same + Windows secret-path owners already used by stable native lock files + (`src/codex/native-main-lock-file.ts:127-131`); +- post-open stable identity is rechecked even though POSIX `O_NOFOLLOW` is not + available there. + +Creation is allowed only after the parent has passed validation. Each new +directory is created `0700`, then re-read and validated; the database is created +`0600`, then re-read and validated. An existing owned entry with a broader mode, +an entry owned by another uid, a symlink/reparse point, unexpected sidecar, or a +path whose identity changes is **not** chmodded, unlinked, renamed, or recreated +by acquisition. It returns typed `refused`. Automatic repair would mutate an +entry whose ownership is precisely what could not be established. + +No acquisition path escalates privileges. The response names the unsafe path in +redacted form and tells the user to inspect/remove it themselves. + +## Canonical `CODEX_HOME` identity + +The current split is a real lock-splitting bug. Explicit `CODEX_HOME` is +realpathed after an existence/directory check (`src/codex/paths.ts:6-21`), while +the default returns `~/.codex` (or a WSL-detected home) without realpathing +(`src/codex/home.ts:135-146`, `src/codex/paths.ts:22-24`). If `~/.codex` is a +symlink, default and explicit spellings can name the same directory but hash +different strings. + +The lock protocol has one resolver for both forms: + +1. Select the raw effective home with existing precedence: nonblank + `CODEX_HOME`, otherwise `defaultCodexHome()`. +2. Expand leading `~`, make the path absolute, require an existing directory, + then call `realpathSync.native` regardless of whether the source was explicit + or default. +3. On Windows, normalize separators and case-fold the real path before hashing; + the repository already treats Windows path identity case-insensitively in + diagnostics (`src/codex/home.ts:164-183`). On macOS and Linux preserve the + real path returned by the filesystem. For an existing case-insensitive macOS + directory, realpath supplies the stored directory-entry spelling; a + case-sensitive volume keeps distinct paths distinct. +4. Hash the normalized canonical string with the domain-separated full SHA-256 + described above. + +Consequences: + +- default `~/.codex`, explicit `~/.codex`, an absolute spelling, and any symlink + to the same existing directory take the same lock; +- Windows case/separator variants take the same lock; +- two different existing canonical directories take different locks; +- a missing home returns `refused/codex_home_missing` and creates no namespace or + lock artifact. + +Refusing a missing home is deliberate. Canonicalizing the deepest existing parent +and appending missing segments cannot satisfy both requirements on all filesystems: +lowercasing the absent suffix aliases two distinct future homes on case-sensitive +APFS, while preserving it splits one future home on case-insensitive APFS. There +is no inode or canonical directory entry to resolve yet. Native writes already +require a real target; creation/installation of `CODEX_HOME` is a separate +operation and must complete before this lock can protect it. + +**INFERRED:** bind mounts or other namespace aliases that `realpath` does not +collapse are outside the supported equivalence set. Supporting them would require +a portable directory file-identity key and a decision about remote/network +filesystems. The protocol returns `refused/unsupported_filesystem` where SQLite +locking semantics or stable file identity cannot be established; it does not +pretend path hashing solved that case. + +## `mutatePersistedConfig`: real outcomes and retryability + +The previous design's statement that all `unavailable` outcomes were +non-retryable was wrong specifically for `conflict`. `conflict` means three fresh +snapshot checks observed continuing byte movement, not malformed authority +(`src/config.ts:1841,1870-1913`). Retrying the whole mutation after backoff is the +intended recovery once the competing writer settles. + +The complete taxonomy is: + +| Actual outcome | Where | Retry policy | +|---|---|---| +| returned `committed` | Fresh bytes stayed stable through revalidation and `persistConfigUnlocked` completed (`src/config.ts:1885-1910`). | Success; do not retry. | +| returned `unchanged` | The first callback or confirmation callback says no change (`src/config.ts:1877-1879,1894-1898`). | Success; do not retry. Native convergence is a separate decision and must not be skipped merely because config was unchanged. | +| returned `unavailable/missing` | The file is absent before lock or at an under-lock read (`src/config.ts:1864-1869,1871-1875,1885-1888,1900-1903`). | Not an immediate retry. Retry only after the config is restored/created by an authorized path. | +| returned `unavailable/invalid` | The file exists but is unreadable/invalid at the same read points (`src/config.ts:1698-1711,1849-1850,1864-1903`). | Not an immediate retry. Requires repair of the file or permissions. | +| returned `unavailable/conflict` | All three rebase attempts observe competing byte changes (`src/config.ts:1841,1885-1912`). | **Retryable** from the beginning with bounded backoff and a fresh native-lock deadline. No partial config commit occurred. | +| thrown `ConfigMutationLockError`, cause `SQLITE_BUSY` | Config transaction acquisition lost (`src/config.ts:1784-1800`). | Retryable contention. While the OFF setter already owns the outer native lock, it may retry only within the remaining outer deadline; it must not release and silently reorder. | +| thrown `ConfigMutationLockError`, non-busy cause | Coordinator database could not be opened/acquired (`src/config.ts:1792-1800`). | Refused/write failure, not blind retry. Permissions, path, disk, or SQLite setup may be broken. | +| thrown callback/hook error | Either mutation callback or the test seam throws inside the transaction (`src/config.ts:1877-1883,1894-1898`). | Domain-specific failure. The generic primitive cannot label it retryable. | +| thrown persistence/commit/close-path error | Atomic config persistence or SQLite commit fails; the catch rolls back when possible and rethrows (`src/config.ts:1803-1818,1909-1910`). | Treat as non-retryable/commit-ambiguous until disk is reread. Never automatically issue a second OFF/ON write from the exception alone. | + +`PersistedConfigMutationOutcome` describes only the returned branches +(`src/config.ts:1832-1839`). Callers that claim an exhaustive status matrix while +ignoring exceptions are not exhaustive. + +## Lock ordering and deadlock analysis + +The OFF setter needs one atomic ordering point across desired intent and native +state. It therefore takes locks in this order: + +```text +Codex native-write lock (async, per canonical CODEX_HOME) + -> config mutation lock (sync, per OPENCODEX_HOME) + -> release config lock before any further await + -> native remove/converge +-> release native-write lock +``` + +No code may acquire the native-write lock from inside +`withConfigMutationLockSync`, `mutatePersistedConfig`'s callback, or any helper +called by those callbacks. The config callback stays synchronous; waiting for the +config lock uses its existing fail-fast exception and a bounded outer async retry. + +The current tree has no inverse nesting: + +- every direct `withConfigMutationLockSync` call is in `src/config.ts` or the + credential-store wrapper (`src/config.ts:1826-1829,1861-1913,2144-2176`; + `src/codex/account-store.ts:278-285`); +- the only external `mutatePersistedConfig` caller mutates plan strings in its + callback and performs no native operation (`src/codex/auth-api.ts:660-701`); +- config persistence writes only config/account files; it does not import or call + native-profile, native-main, prompt, shim, integration, or proposed Codex-write + acquisition; +- the existing native-profile lock is acquired inside `NativeProfileManager` + operations and does not appear under a config-lock callback + (`src/codex/native-profile-manager.ts:313-388,553,610`). + +Therefore adding only the edge `native-write -> config` cannot close a cycle in +the current graph. The proof must remain mechanical: a source-shape regression +should enumerate all config-lock callbacks and reject imports/calls into the +native-write owner, while native-write tests exercise config busy, conflict, and +success under one held outer transaction. + +Two qualifications matter: + +1. `withConfigMutationLockSync` has synchronous reentrancy + (`src/config.ts:1765-1783`). That does not authorize async native-lock + reentrancy and does not change the global order. +2. Codex credential refresh currently has its own file lock and can later take the + config lock (`src/codex/account-store.ts:433-469`). Native write critical + sections must not perform credential refresh or provider network work. That is + already required by the gather/commit split; violating it would add a longer + temporal dependency even if it did not immediately form an inverse native-lock + edge. + +## Protocol assertions for the later implementation plan + +The implementation phase should be rejected unless tests activate all of these +paths: + +1. A held lock keeps Bun responsive; an OFF waiter acquires after release and its + mutation wins in the two-party race. +2. Deadline expiry returns typed `busy` without a native or config write. +3. A crashed child releases SQLite ownership without PID/mtime cleanup. +4. A live hung holder is never stolen after any age. +5. Default, explicit, absolute, and symlink spellings of one existing home map to + one full hash; distinct homes map to different hashes. +6. Missing home refuses before namespace creation. +7. Wrong owner/mode, symlink/reparse namespace component, substituted database, + and unexpected sidecar each return `refused` and are not repaired. +8. A nested same-task acquisition refuses immediately; a separate same-process + task waits normally. +9. Barging is permitted by contract: tests assert exclusion and deadlines, not + FIFO arrival order. +10. Outer native-lock plus config `SQLITE_BUSY` and config `conflict` each retry + only within the remaining outer deadline; non-busy config failure does not. + +The hardest unresolved-looking question was missing-home identity on +case-insensitive filesystems. It is resolved here by refusing to invent a lock +key before a canonical directory exists. Any later requirement to coordinate +creation of `CODEX_HOME` is a different lock domain and must not weaken this one. diff --git a/devlog/_plan/260804_codex_write_substrate/004_ownership_and_convergence.md b/devlog/_plan/260804_codex_write_substrate/004_ownership_and_convergence.md new file mode 100644 index 000000000..d69e7d6f7 --- /dev/null +++ b/devlog/_plan/260804_codex_write_substrate/004_ownership_and_convergence.md @@ -0,0 +1,391 @@ +# Part 4 — ownership authority and the meaning of convergence + +Research document. No implementation diff. The failure under examination is the +one the second WP4 audit exposed: an unattended start can decide that an unreadable +service record means "no owner", repair a dead-PID journal, and overwrite files that +an external provider or another OpenCodex home owns. The current code makes each +individual choice look reasonable, but it asks four different ownership questions +as though they were one (`007_audit_synthesis_wp4.md:25-64`, +`008_audit_synthesis_wp4_r2.md:31-41`). + +The substrate must answer those questions separately. A persisted switch says what +the user wants. It does not prove which installation may write, whether another +provider owns `config.toml`, whether a journal writer is still alive, or whether the +bytes on disk are still the bytes OpenCodex wrote. + +## The concrete failure + +The dangerous fixture is not an exotic race: + +1. an old, markerless version-1 journal exists with a dead PID; +2. `config.toml` now selects an external `model_provider`; +3. the journal has no `injectedConfigHash`, so `restoreJournalState` treats the + current config as unchanged (`src/codex/journal.ts:109-123`); +4. startup currently avoids that write only because it checks + `currentExternalCodexModelProvider()` before `reconcileJournal` + (`src/cli/index.ts:169-177`, `src/cli/index.ts:358-365`); +5. replacing that check with service-home ownership deletes a different authority + and lets the journal baseline overwrite the externally managed config. + +The external-provider check must therefore precede journal parsing, PID recovery, +desired-state convergence, and lock creation. In this fixture the entire Codex +artifact set, including the old journal, must remain byte-exact. The current +`restoreNativeCodex` behavior of deleting the journal when an external provider is +active is not acceptable for unattended convergence (`src/codex/inject.ts:764-770`). + +## Four kinds of decision, not one + +The word “guard” hid the design error. The substrate has four decision classes: + +- **authority** — who is allowed to mutate this Codex home; +- **intent** — whether the persisted target is ON or OFF; +- **serialization** — whether this operation is the one allowed to commit now; +- **capability** — whether the selected files and database can be read and written. + +Only authority can answer “may this process touch these bytes?”. Intent answers +“which direction should it converge?”. A lock answers “when?”. A filesystem or +SQLite error answers “can it finish?”. None may be used as a fallback for another. + +## Distinct authorities that can refuse a mutation + +| Authority | Question it answers | Current owner and current failure | Interactive route | Unattended convergence | +|---|---|---|---|---| +| Service-home ownership | Does an installed OpenCodex service claim this canonical `CODEX_HOME` and `OPENCODEX_HOME` pair? | `assertServiceEnvironmentMatchesInstall` compares the recorded homes (`src/service.ts:216-234`). `readServiceInstallState` skips unreadable/corrupt mirrors and returns the same `null` used for absence (`src/service.ts:165-175`), while `assertNativeTeardownOwned` converts every non-mismatch error to success (`src/integrations/native/ownership-preflight.ts:21-35`). | Fail closed on `foreign`; fail closed on `unknown` for Codex file mutation and return an actionable refusal. An explicit future override would be a separate user-consent surface, not implicit fail-open. | Fail closed on both `foreign` and `unknown`; continue starting the proxy, but perform zero Codex writes. | +| External `model_provider` | Has the user delegated this Codex config to a provider other than native `openai` or `opencodex`? | `externalCodexModelProvider` resolves the effective project provider and treats any other value as external (`src/codex/inject.ts:28-36`). Start and ensure currently consult it before journal recovery, but the failed design removed it in favor of service ownership. | Fail closed for apply, restore, repair, journal deletion, catalog/cache cleanup, and history mutation. Report the external provider. | Same, with byte-exact preservation. External ownership is not “already converged”; it is a blocked/deferred state. | +| Journal validity and writer liveness | Is there a valid OpenCodex transaction to recover, and can its writer still be active? | The journal records only PID and timestamp (`src/codex/journal.ts:10-18`). `reconcileJournal` treats a live PID or `EPERM` as active and any other probe failure as dead (`src/codex/journal.ts:148-159`). `readJournal` deletes malformed or unknown-version bytes while merely trying to inspect them (`src/codex/journal.ts:97-105`). | A valid live/permission-unknown writer blocks. A valid dead writer permits recovery only after the higher authorities pass. Invalid/unknown-version journal bytes are `unknown`, preserved, and block automatic writes. | Identical fail-closed policy. PID reuse may delay recovery, which is safer than overwriting a live transaction. A future journal should add an instance token/process-start identity; PID alone is not proof of identity. | +| Artifact provenance and drift | Are the current bytes still OpenCodex’s post-image, or did another actor edit them after apply? | Journal hashes protect only config/profile, and missing hashes are interpreted as unchanged (`src/codex/journal.ts:114-129`). Catalog restore infers ownership from slash-qualified slugs and a backup (`src/codex/catalog/sync.ts:572-597`). Cache invalidation has no pre-image at all (`src/codex/catalog/sync.ts:600-616`). Managed-default marker ambiguity is already a local refusal (`src/codex/inject.ts:506-517`, `src/codex/inject.ts:683-709`). | Exact rollback is allowed only from a recorded baseline and matching post-image. A structurally owned fragment may be removed while preserving unrelated edits. Ambiguous markers, unknown provenance, or conflicting hashes fail closed and name the artifact. | Same. Unattended code may never turn “I cannot prove ownership” into deletion. | + +Two additional gates can refuse work, but they are not ownership authorities: + +- the persisted desired flag is the **intent authority**. It must be freshly read; + it never grants permission to cross one of the four authorities above; +- the per-home linearization transaction is **serialization**. Contention or an + unopenable lock refuses the attempt, but owning the lock does not make foreign + files ours. The current config mutation database is created when its path is + resolved/opened (`src/config.ts:1757-1762`, `src/config.ts:1775-1800`), which is + why an ownership decision must happen before a Codex lock path is opened. + +History `SQLITE_BUSY`, `SQLITE_LOCKED`, `EPERM`, and `EACCES` are capability +failures, not evidence that another OpenCodex home owns the state +(`src/codex/history-provider.ts:511-548`). They make convergence unresolved; they +do not authorize a partial success. + +## The service-home tri-state API + +**INFERRED specification:** the ownership API must return evidence, not a boolean: + +```ts +type NativeCodexOwnership = + | { state: "owned"; codexHome: string; opencodexHome: string; evidence: "no-service" | "matching-install" } + | { state: "foreign"; codexHome: string; opencodexHome: string; recordedCodexHome: string; recordedOpenCodexHome: string; message: string } + | { state: "unknown"; codexHome?: string; opencodexHome?: string; reason: "service-state-missing" | "service-state-corrupt" | "service-state-unreadable" | "service-state-conflict" | "path-unresolvable"; message: string }; +``` + +This is a research contract, not a proposed source diff. Its truth table is: + +| Read-only evidence | Result | +|---|---| +| No service registration and no service-state mirror | `owned/no-service` — no installed service claims the target. | +| Service is installed and every readable mirror agrees with the canonical current homes | `owned/matching-install`. | +| A valid mirror names a different canonical Codex or OpenCodex home | `foreign`, even if a second path is missing. A valid foreign claim is not erased by absence elsewhere. | +| Service is installed but no valid mirror exists | `unknown/service-state-missing`. | +| Any required mirror is unreadable/corrupt, or two valid mirrors disagree | `unknown`; do not skip the bad mirror and accept the convenient one. | +| Current or recorded paths cannot be canonicalized safely | `unknown/path-unresolvable`. | + +The truth table above is also **INFERRED** from the information the current service +reader collapses; no shipped function currently correlates manager registration, +all mirrors, and canonical paths. The probe must be read-only. It may inspect +service-manager registration and every +known state mirror, but it may not create a directory, harden an ACL, open SQLite, +delete a malformed mirror, or “repair” state. Current `loadConfig()` is unsuitable +for the admission read because it hardens paths and can write an invalid-config +backup (`src/config.ts:1503-1510`, `src/config.ts:1544-1549`). The existing +`readConfigDiagnostics()` path reads and validates without that backup side effect +(`src/config.ts:1679-1715`). + +## One admission order for start, ensure, and routes + +The invariant is stronger than “check before commit”: + +> Before all relevant authorities answer, do not create or modify a lock file, +> SQLite database, journal, catalog backup, catalog, cache, config, profile, +> history row, or rollout line. + +The common admission sequence is: + +1. Resolve the effective paths without creating them. Canonicalize the existing + `CODEX_HOME`, `OPENCODEX_HOME`, active config, active catalog, cache, journal, + history DB, and provenance paths. A path-resolution failure is `unknown`. +2. Inspect service-home ownership. `foreign` or `unknown` stops Codex mutation. +3. Read `config.toml` without mutation and resolve external `model_provider`. + External ownership stops every Codex mutation, including journal cleanup. +4. Inspect the journal without deleting or rewriting it. Validate schema, then + classify the recorded writer as `alive | dead | unknown`. `alive` and `unknown` + stop recovery and new apply. Invalid bytes are preserved and stop mutation. +5. Inspect provenance for every artifact that either direction could touch. + Unknown or conflicting provenance stops that artifact and therefore prevents a + full-convergence claim. +6. Read the persisted desired flag from disk with the pure diagnostic reader. Do + not consult a server-captured `config` object. +7. For ON only, gather provider models and calculate candidate catalog/config bytes + outside the lock. Gathering may await network I/O, but it writes nothing. +8. Only now open the per-canonical-`CODEX_HOME` lock outside `CODEX_HOME`. Part 3 + places the full-hash SQLite database in the private real-user-home namespace, + explicitly rejecting both `tmpdir()` and either configurable home + (`003_lock_protocol.md:35-47`, `003_lock_protocol.md:246-260`). Even that external + lock artifact is forbidden before steps 1-6 answer. +9. Inside the lock, re-run steps 1-6 from disk. This is the authority and intent + linearization point. A changed answer aborts with no native write. +10. Recover a valid dead-writer journal first, then commit the desired ON or OFF + artifact transition. Recovery never runs past an external-provider veto. +11. Read observed state while still serialized. Release the lock before logging, + app-server handling, network work, or retries. + +`startServer()` currently invalidates `models_cache.json` unconditionally before +listen (`src/server/index.ts:362-403`). That is a Codex write and therefore belongs +behind this admission sequence; moving only `reconcileJournal` is insufficient. + +### Startup + +`handleStart` currently reconciles the journal before it checks the existing proxy +and later calls `syncModelsToCodex` after listen (`src/cli/index.ts:169-177`, +`src/cli/index.ts:312-321`). Startup should run the read-only authority sequence +before either journal repair or `startServer` cache invalidation. A refusal does +not prevent the proxy from listening for other clients. It suppresses all Codex +recovery/apply/remove work and records a specific unresolved reason. + +After the proxy is live, desired ON may gather and enter the locked commit. Desired +OFF enters the same lock and restores. A crash after intent persistence but before +restore is ordinary: the next startup sees OFF, observes residue, and retries. + +### Ensure + +Ensure uses the same order before its current journal call +(`src/cli/index.ts:358-369`). The already-live and spawn-a-child branches must both +admit Codex separately; successful proxy health does not imply Codex convergence. +The parent’s post-spawn sync at `src/cli/index.ts:398-412` must receive the same +fresh authority and desired-state answer as the child. + +### Route entry + +Management authentication and request-body validation may happen first because +they do not touch Codex artifacts. Immediately after that, every Codex-mutating +route performs the common read-only admission. A status GET remains inspection-only +and never repairs. + +For `POST /api/sync`, the fresh desired-state read happens before gather, then all +authorities and intent are re-read under the lock before commit. The route currently +passes the server’s startup-captured `config` into `syncModelsToCodex` +(`src/server/management/config-routes.ts:261-268`); that object cannot be an +admission source. + +For an explicit toggle, `unchanged` means only that the desired flag already had +the requested value. The route still inspects and converges native artifacts under +the same lock. OFF intent is committed before removal so a crash is recoverable. +ON intent may remain persisted if apply later fails; observed state then reports +partial and startup/ensure retries it. Explicit native restore does not silently +change desired state. + +## Artifact inventory and absence restoration + +Filename is not provenance. A path called `opencodex-catalog.json` may predate the +current operation; a file at a custom `model_catalog_json` path may be user-owned; +and a missing file may be created later by either OpenCodex or Codex. The substrate +needs a durable per-operation artifact ledger that records, before the first write. +This ledger is an **INFERRED requirement**; the current journal and backup files do +not carry enough pre-image/post-image evidence for the full inventory. It records: + +- canonical path and artifact kind; +- baseline state: `absent` or `present` with exact bytes/hash and relevant metadata; +- the OpenCodex post-image hash after each successful write; +- structural ownership facts where byte identity is expected to drift, such as + exact routed slugs and history row originals; +- transaction identity and completion state. + +The ledger belongs in the Codex integration’s owned record under +`OPENCODEX_HOME`, not in `CODEX_HOME`; Part 2 selects that same record for desired +state plus durable history convergence (`002_history_off_the_loop.md:298-335`). It +is created only after pre-lock authority passes and while the native-write lock is +held. + +An artifact is “created by us” only when the ledger says its baseline was absent +and records the successful OpenCodex post-image. A familiar filename, slash in a +slug, marker comment, mtime, or presence in `OPENCODEX_HOME` is supporting evidence, +never sufficient by itself. + +| Artifact | Current behavior | Restore when baseline was present | Restore when baseline was absent | +|---|---|---|---| +| `config.toml` root routing | Injection journals only when the file exists (`src/codex/journal.ts:60-81`) and later writes routing/profile/default fragments. Fallback restore strips structurally owned fragments (`src/codex/inject.ts:688-740`). | If current bytes match the recorded post-image, restore exact baseline bytes. If unrelated edits occurred, remove only unambiguous owned fragments and preserve edits; report historical restore as partial. Ambiguous markers or unknown provenance block. | If current bytes exactly match our post-image, delete the file. If another actor added content, preserve that content while stripping only owned fragments; absence is no longer safely reproducible, so report a preserved-drift conflict rather than deleting the user’s edits. | +| Embedded `[profiles.opencodex]` profile in `config.toml` | Removed as part of the config transform (`src/codex/inject.ts:696-705`). | Covered by the config baseline/post-image; never treat it as independent permission to overwrite the rest of the file. | Remove the owned section only. Delete the whole config only when the baseline was absent and the whole current file still equals our post-image. | +| `opencodex.config.toml` generated profile file | Journal stores either exact profile bytes or `null`; restore writes bytes or unlinks (`src/codex/journal.ts:71-90`, `src/codex/journal.ts:125-131`). `removeCodexConfig` otherwise unlinks by filename (`src/codex/inject.ts:723-742`). | Restore exact baseline only if current hash equals our post-image. Drift is a conflict unless an owned subsection format exists. | Unlink only when the ledger records baseline absent and the current hash equals our post-image. A same-named untracked file is not ours. | +| Active catalog (`opencodex-catalog.json` or custom path) | The active path comes from root `model_catalog_json`, else the default (`src/codex/catalog/parsing.ts:167-176`). Sync may materialize an absent catalog (`src/codex/catalog/bundled.ts:213-234`) and later overwrites it (`src/codex/catalog/sync.ts:507-569`). Restore uses backup plus native additions or removes routed rows (`src/codex/catalog/sync.ts:572-597`). | Exact post-image permits exact baseline restore. On drift, restore the baseline’s native fields and preserve verified post-apply native additions while removing the exact routed entries recorded by the ledger. Unknown JSON/provenance blocks. | Delete only if the current file is still the recorded OpenCodex post-image. If Codex or the user added native rows, remove recorded routed rows and preserve native rows, but report that baseline absence could not be restored without data loss. | +| Hashed and legacy catalog backups | Backups are created once and best-effort; both hashed and legacy paths may be written (`src/codex/catalog/parsing.ts:419-445`). Restore reads them but does not consume them. | Preserve exact pre-existing bytes. A collision between an existing backup and the operation’s expected baseline is `unknown`, never replacement authority. | Backups created by this transaction are internal rollback artifacts: delete them only after every dependent catalog/cache restore is complete. A ledger entry, not the backup filename, proves they were created by us. | +| `models_cache.json` | Apply rewrites it with an expired wrapper containing the current catalog (`src/codex/catalog/sync.ts:600-613`); `restoreNativeCodex` never calls a cache restore (`src/codex/inject.ts:770-794`). Errors are swallowed into `false`. | Restore exact baseline when current hash equals our post-image. If Codex refreshed it meanwhile, preserve native cache data and remove only ledger-recorded routed rows when parseable; unreadable or ambiguous cache is unresolved. | This is the audit’s invalidation/restoration distinction: if apply created it and it still matches our post-image, delete it. Rewriting an expired native wrapper is not restoration of absence. If a native process changed it, preserve the changed file, remove only proven routed residue, and report preserved drift. | +| Injection journal | Current version stores config/profile pre-images, optional post hashes, PID, and timestamp (`src/codex/journal.ts:8-18`). Complete restore deletes it (`src/codex/journal.ts:133-140`); malformed reads also delete it (`src/codex/journal.ts:97-105`). | A pre-existing valid journal represents an earlier transaction and must be recovered or explicitly superseded under its own rules; it is not overwritten merely because a new apply starts. Pre-existing invalid bytes are preserved and block. | A journal created for this operation is deleted only after every artifact it protects reaches its terminal restore state. Partial restore retains it. | +| Resume history database (`state_5.sqlite`) | Apply changes selected `threads` rows after recording row originals (`src/codex/history-provider.ts:581-650`). Restore writes original fields, then ejects leftover `opencodex` rows (`src/codex/history-provider.ts:656-699`). | Restore each recorded row’s exact provider/source/user-event fields. Do not claim completion while any selected row remains `opencodex` or its backup entry remains. SQLite/WAL bytes are not expected to be byte-exact. | Apply already checks DB existence and returns without opening when absent (`src/codex/history-provider.ts:581-583`); it must not create the DB. Absence therefore remains absence. | +| History backup manifest | Missing manifest is treated as an empty map; apply records each original only once, and an empty manifest is unlinked (`src/codex/history-provider.ts:204-237`). | Preserve unrelated/pre-existing entries and consume only transaction-owned entries after their DB and rollout observations agree. Corrupt or wrong-DB manifests are `unknown`, not silently empty. | Create only after at least one row is selected. Delete when all owned entries are restored; a zero-entry file must not remain. | +| Rollout JSONL files | OpenCodex never creates them on this path. It patches line one when length permits and appends a last-writer-wins `session_meta` (`src/codex/history-provider.ts:52-100`, `src/codex/history-provider.ts:412-457`). Current restore catches per-file failure and can still consume the manifest (`src/codex/history-provider.ts:667-695`). | Restore semantically, not byte-exactly: latest metadata and the first-line provider reader must both resolve to the native target. Keep the backup entry until that observation passes. Concurrent Codex turns remain untouched. | Apply must not create a missing rollout (`src/codex/history-provider.ts:419-423`). If a recorded rollout disappears, do not recreate it from partial metadata; resolve only when the corresponding DB row is also gone, otherwise report unresolved. | + +The config/catalog/cache “baseline absent plus later native edit” case has no +lossless automatic answer. Deleting restores absence and destroys new data; +preserving new data means historical byte restoration is incomplete. The substrate +must prefer preservation and expose the conflict. That is the hardest ownership +problem in this part. + +## Observed state is not desired state + +The persisted flag is one boolean with an absent key in a valid file meaning ON. +An unreadable or invalid config is desired-state `unknown`, not default ON; treating +parse failure as intent would recreate the fail-open ownership bug. The flag answers +only the target direction. Observed state is a read-only projection over the actual +Codex artifacts and authorities. + +The projection must read: + +1. service-home ownership and external provider; +2. `config.toml` root `model_provider`, owned `openai_base_url`, active + `model_catalog_json`, embedded profile, root routed model, and managed defaults; +3. `opencodex.config.toml` existence, hash, and provenance; +4. the active catalog’s parse state and all ledger-recorded routed slugs; +5. `models_cache.json` in both wrapper and raw-catalog shapes, including routed + slugs and provenance; +6. journal absence/validity, writer liveness, transaction identity, and whether + its post hashes match current config/profile; +7. history DB rows still tagged `opencodex`, remaining backup entries, and each + touched rollout’s latest and first-line provider observations; +8. catalog backup/provenance residue whose baseline was absent. + +**INFERRED projection:** the aggregate is not a boolean: + +| Observed state | Meaning | +|---|---| +| `applied` | Config routing is active and every required profile/catalog/cache/history artifact agrees with the ON transaction. History expectation is mode-dependent: loopback Design B remains `openai`, while legacy non-loopback routing may use `opencodex` (`src/codex/inject.ts:57-63`, `src/codex/inject.ts:775-783`). | +| `absent` | No OpenCodex routing/profile/routed catalog or cache row/pending journal/history residue remains, and transaction-created rollback artifacts whose baseline was absent are consumed. | +| `partial` | Some ON or OFF artifacts match and others do not; this includes crash residue and missing ON artifacts after `ocx restore` or stop. | +| `external` | Another `model_provider` owns Codex. No mutation is allowed, even if OpenCodex residue is visible. | +| `blocked` | Service ownership is foreign/unknown, a journal writer is live/unknown, provenance is ambiguous, or an artifact cannot be inspected. | + +“Converged” is the relation between fresh desired and observed state: + +- desired ON + observed `applied` = converged; +- desired OFF + observed `absent` = converged; +- every other pair is not converged, with `external`/`blocked` carrying the + authority that prevents repair. + +This definition is why `unchanged` must still converge. A no-op desired-state +commit says only that the boolean already matched. OFF may still have routed cache +rows after a crash; ON may still have missing config/catalog/profile after restore +or stop. The route must inspect, execute the appropriate transition, and re-inspect +before reporting convergence (`008_audit_synthesis_wp4_r2.md:36-41`). + +Operational convergence and historical restoration should be reported separately. +An OFF transition that removes every owned routed fragment while preserving a +post-apply native edit can be operationally absent but historically +`preserved-drift`; it must not claim byte-exact restoration. + +## Fresh desired-state admission for a long-lived server + +Recommend **re-read at every Codex mutation admission**, not a watcher and not a +process-local version counter. + +The route should use the pure persisted-config diagnostic reader, require a valid +file-backed result, and extract only the Codex desired flag. It repeats the read +inside the linearization lock. The long-lived `config` object may still serve +unrelated request routing, but it is not authority for Codex mutation. + +Cost per Codex-mutating request: one file open/read, JSON parse, and schema +validation before gather, plus the same bounded read under the commit lock. This is +O(config-file bytes), normally two local reads on an infrequent management/startup +path. It adds no resident watcher, debounce/rearm behavior, cross-platform rename +edge cases, or sidecar version file. A watcher can be an optimization later, but its +cache must never replace the admission read. A version counter cannot detect a CLI +or manual edit unless every writer participates, which is the exact stale-object +assumption that failed here (`src/server/management/config-routes.ts:261-268`, +`src/server/index.ts:362-364`). + +## Tests that prove the authority and convergence contract + +All cases use temporary `CODEX_HOME` and `OPENCODEX_HOME`; none starts, stops, +syncs, restores, or ensures the live proxy on port 10100. + +### Ownership and ordering + +- `tests/codex-ownership-authority.test.ts` — `service ownership distinguishes + owned, foreign, and unknown without side effects`: cover no service, matching + mirrors, foreign canonical home, installed-with-missing-mirror, corrupt, + unreadable, conflicting mirrors, and unresolvable paths. Hash every fixture and + assert no lock/database/journal path was created. +- `tests/codex-ownership-authority.test.ts` — **`dead-PID markerless journal plus + external provider preserves every byte`**: seed a version-1 journal with no + injected hashes and a dead PID; set an external `model_provider`; include config, + generated profile, catalog, both backup forms, cache, history manifest, DB, and + rollout sentinels. Run startup admission and ensure admission independently. + Assert byte-exact equality, unchanged mtimes where supported, journal still + present, zero lock artifacts, and an `external` refusal. This strengthens the + current dead-PID test, which presently expects markerless overwrite and journal + deletion (`tests/codex-journal.test.ts:53-75`). +- `tests/codex-ownership-authority.test.ts` — `invalid journal inspection is + read-only`: corrupt and unknown-version journals remain byte-exact and block; + this intentionally reverses the current deletion expectation + (`tests/codex-journal.test.ts:77-89`). +- `tests/codex-ownership-authority.test.ts` — `authority precedes lock creation`: + instrument every lock/path factory and artifact writer; foreign, unknown, + external, live-journal, and unknown-journal cases must hit none. + +### Artifact absence and drift + +- `tests/codex-artifact-provenance.test.ts` — `baseline absence is restored for + config, profile, catalog, backups, cache, journal, and history manifest`: apply + into an empty isolated home, verify the ledger records `absent`, restore without + drift, and assert every transaction-created artifact is absent again. +- `tests/codex-models-cache-restore.test.ts` — **`cache absent before apply is + absent after OFF`**: begin with no `models_cache.json`, apply routed catalog data, + assert apply creates the cache, then OFF must unlink it rather than rewrite an + expired native wrapper. A second OFF is a no-write success. This is distinct from + the existing invalidation test that proves creation (`tests/codex-models-cache-invalidate.test.ts:41-55`). +- `tests/codex-artifact-provenance.test.ts` — `absent baseline plus native drift is + preserved and reported`: after apply creates config/catalog/cache, add native + content through an independent fixture writer. OFF removes only ledger-owned + routing, preserves native additions, and returns operational `absent` plus + historical `preserved-drift`, never byte-exact success. +- `tests/codex-artifact-provenance.test.ts` — `pre-existing same-named files are not + owned by filename`: omit the ledger or give it a conflicting transaction/hash; + restore refuses rather than unlinking. +- `tests/codex-artifact-provenance.test.ts` — `pre-existing catalog backups survive`: + exact backup bytes remain after apply/restore, while transaction-created hashed + and legacy backups are consumed only after catalog and cache restore. + +### Observed state and unchanged convergence + +- `tests/codex-observed-state.test.ts` — table-drive `applied`, `absent`, every + one-artifact `partial`, `external`, unreadable/invalid `blocked`, stale first-line + rollout metadata, and a non-empty history manifest with no matching DB row. +- `tests/codex-convergence.test.ts` — `desired OFF unchanged still removes crash + residue`: persist OFF first, seed each routed artifact one at a time, invoke the + setter again, and prove every case converges rather than returning after + `unchanged`. +- `tests/codex-convergence.test.ts` — `desired ON unchanged rebuilds removed + artifacts`: persist ON, remove config/profile/catalog/cache independently, and + prove the no-op flag commit still applies and re-inspects. +- `tests/codex-sync-api.test.ts` — `server observes CLI desired-state change at + admission`: construct the server with a stale ON object, persist OFF from a + subprocess, call `POST /api/sync`, and assert refusal plus zero gather/write; + then persist ON from the subprocess and assert the same running server admits. +- `tests/codex-convergence-order.test.ts` — startup, both ensure branches, explicit + route, sync route, restore, stop, and uninstall all emit an ordered trace proving + `path -> service ownership -> external provider -> journal -> provenance -> fresh + intent -> gather -> lock -> recheck -> commit -> observe`. Foreign/external traces + end before `lock`. + +The final oracle is artifact state, not a green return envelope: cache absence, +journal preservation, first-line rollout provider, and no-created-lock assertions +are each read back after the operation. + +## Decision + +Service-home ownership, external provider ownership, journal writer ownership, and +artifact provenance are independent vetoes. The common admission path consults +them in that order before intent, gather, and lock creation, then rechecks them +inside the lock. Desired state is freshly read for every mutation, and convergence +is proved from config, profile, catalog, cache, journal, history, and rollouts — not +from whether the desired-state write happened to change one JSON field. From 7e67a8d06311de2471b0a25e41cf85f97007cc69 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Tue, 4 Aug 2026 09:43:42 +0900 Subject: [PATCH 025/163] =?UTF-8?q?docs(substrate):=20lock=20the=20roadmap?= =?UTF-8?q?=20=E2=80=94=20four=20parts,=20dependency-ordered?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit WP9 and WP10 are genuinely independent and both land before WP11, because a lock around an unsplittable gather-and-write, or around a ten-second blocking history call, is the failure the previous unit already proved. WP11 then has something bounded to wrap, and WP12 is last because its admission order must run before the lock module creates anything, so it needs the real construction sequence to point at. Thirteen criteria, each naming a live artifact rather than a unit test: C3 in particular MEASURES /healthz under real contention instead of asserting it. States plainly what this unit does not claim: the write path does not become transactional. A crash mid-commit still leaves partial state; what changes is that it is detectable and the next convergence re-runs against it. --- .../260804_codex_write_substrate/000_plan.md | 118 ++++++++++++++++++ 1 file changed, 118 insertions(+) create mode 100644 devlog/_plan/260804_codex_write_substrate/000_plan.md diff --git a/devlog/_plan/260804_codex_write_substrate/000_plan.md b/devlog/_plan/260804_codex_write_substrate/000_plan.md new file mode 100644 index 000000000..161136ff3 --- /dev/null +++ b/devlog/_plan/260804_codex_write_substrate/000_plan.md @@ -0,0 +1,118 @@ +# A Codex write path that can be safely interrupted + +The prerequisite three integration switches are blocked on. + +## Why this unit exists + +`../260803_codex_desktop_toggle/` shipped two deliverables and proved the switch +itself is cheap: `ocx restore` already returns Codex to its native path without +stopping the proxy. Then two audit rounds on the durable-OFF flag failed, and the +diagnosis in that unit's `008_audit_synthesis_wp4_r2.md` was not the flag: + +> Codex's write path was never designed to be interrupted. + +Every attempt to add "check the flag before writing" hit the same wall. A check +is not a lock; the catalog refresh cannot be split around one; the history write +would freeze the proxy for every other client if a lock were held across it; and +the ownership guard that was supposed to protect a foreign home fails open, after +the artifacts it guards have already been created. + +So this unit builds the substrate. **It ships no switch.** The switches +(`WP4`/`WP5` Codex, `WP6` Grok, `WP7` Desktop in the prior unit) become small +once it exists. + +## The four parts, and why they are ordered this way + +Dependency order (PHASE-SPLIT-01), not effort. Each phase closes with something +independently verifiable. + +| Phase | Doc | Delivers | Depends on | +|---|---|---|---| +| WP9 | `010_catalog_seam.md` | `gatherCodexCatalogCandidate` / `commitCodexCatalogCandidate` + a typed outcome | — | +| WP10 | `020_history_isolation.md` | history off the server event loop, fail-fast under convergence | — | +| WP11 | `030_lock_protocol.md` | the async per-home lock with a hardened namespace | WP9, WP10 | +| WP12 | `040_ownership_convergence.md` | tri-state authority, admission order, absence restoration | WP11 | + +WP9 and WP10 are genuinely independent: one makes catalog work *splittable*, the +other makes history work *non-blocking*. Neither needs a lock to be useful, and +both must exist before a lock is worth taking — a lock around an unsplittable +gather-and-write, or around a ten-second blocking history call, is the failure +the last unit already proved. + +WP11 then has something bounded to wrap. WP12 sits last because the admission +order it defines must run *before* the lock module creates anything, so it needs +the lock's real construction sequence to point at. + +## Research, all written this cycle + +- `001_catalog_seam.md` — the gather/commit line drawn through `refresh.ts`, all + 16 management callers traced, and the stale-candidate problem named +- `002_history_off_the_loop.md` — every blocking operation mapped, server-process + vs CLI-process callers separated, worker isolation + fail-fast recommended +- `003_lock_protocol.md` — async SQLite lock, `acquired | busy | refused`, + barging-allowed, per-user namespace under the real home, realpathed key +- `004_ownership_and_convergence.md` — four independent vetoes in one admission + order, provenance by baseline-absence ledger rather than filename, and why + `unchanged` must still converge + +Carried forward from the prior unit: all eleven open findings listed in +`../260803_codex_desktop_toggle/008_audit_synthesis_wp4_r2.md`. Each is assigned +to a phase in the table above and re-stated in that phase's decade doc. + +## Scope boundary + +IN: `src/codex/**`, `src/server/management/context.ts` and its callers, +`src/config.ts`, `src/service.ts`, `src/integrations/native/**`, `tests/`, +`docs-site` lifecycle and configuration pages, this unit. + +OUT: the GUI switches (they follow once this lands), `gui/**`, releases, +publishing, deploys, tags, npm, starring the repository. The six file clients +remain `FOLLOWUP-FILECLIENT-01` from the prior unit. + +## Criteria + +- C1 — catalog work gathers outside a lock and commits inside it, and a failure + is a typed outcome rather than a swallowed exception. +- C2 — a stale candidate cannot be committed: a config or base-catalog revision + change between gather and commit is detected and refused. +- C3 — a dashboard-initiated OFF never blocks another client. Measured, not + argued: `/healthz` stays responsive while real SQLite history contention is + active. +- C4 — history that cannot be resolved is recorded as unresolved and retried, + never silently reported as success. +- C5 — lock acquisition is async with a finite deadline and returns + `acquired | busy | refused`; contention is never an exception. +- C6 — two spellings of the same home take the SAME lock; two different homes + never do. Symlinked, default, explicit and case-differing spellings all tested. +- C7 — the lock namespace is per-user, outside `CODEX_HOME`, and rejects a + symlinked or wrong-owner path rather than trusting it. +- C8 — automatic convergence refuses on `foreign` AND `unknown` ownership, and + creates no artifact — no lock file, no database, no journal write — before the + answer is known. +- C9 — the external-`model_provider` guard survives as an authority distinct from + service-home ownership. +- C10 — an artifact that did not exist before apply is *removed* on convergence, + not merely filtered; and a baseline-absent artifact the user has since edited is + preserved with a reported conflict rather than deleted. +- C11 — `unchanged` desired state still converges observed state. +- C12 — a desired-state change made by another process is honored by the running + server without a restart. +- C13 — typecheck, full test, gui lint, privacy scan green; no regression in the + 8000-test suite. + +## Risk register + +| Risk | Mitigation | +|---|---| +| A lock held across unbounded work freezes the proxy | WP10 lands before WP11; the locked section is synchronous and bounded by construction, and C3 measures it rather than asserting it | +| The substrate becomes a fifth concurrency pattern | `003` catalogues the existing `withConfigMutationLockSync` and `runIntegrationMutationFlight` and states where the new lock deliberately differs | +| Deadlock against the config mutation lock | One stated ordering, plus a proof that no inverse nesting exists today | +| Convergence deletes something the user owns | Provenance is a recorded baseline-absence plus post-image hash, never a filename or marker; on conflict, preservation wins and the operation reports rather than deletes | +| Another round of divergence | One phase, one boundary, one audit. WP2 and WP3 of the prior unit passed clean on exactly that property; WP4 failed twice without it | + +## What this unit does not claim + +It does not make Codex's write path transactional. A crash mid-commit still +leaves partial state; what changes is that the partial state is *detectable* and +the next convergence *re-runs* against it. `004` §Artifact inventory is explicit +about which artifacts can be restored to absence and which can only be reported. From 6ab4d5f82b9acb6c2a0a6418268298ac45d69116 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Tue, 4 Aug 2026 10:01:35 +0900 Subject: [PATCH 026/163] docs(substrate): write all four implementation phases to diff level MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 010 catalog seam: staleness guard is a SHA-256 over canonical catalog-affecting config plus the exact base-catalog bytes-or-absence and target paths, so a commit refuses a candidate gathered against a config that has since moved. Candidates are consumed once and branded module-private. 020 history: the whole SQLite/rollout mutation moves into an owned Bun Worker with serializable messages and explicit death handling. Automatic server work gets 100ms/one attempt/no sleep while explicit CLI work keeps the current 5000ms/two attempts — the asymmetry is the point, since a CLI blocking itself is acceptable and the server blocking is not. C3 measures six /healthz calls and a live stream during deterministic cross-process BEGIN IMMEDIATE contention rather than asserting responsiveness. 030 lock: explicit and default homes resolve identically and must exist, both realpathed, case-folded only on Windows. Missing homes are refused before hashing, admission or namespace creation, which is what dissolves the case-sensitivity ambiguity 003 flagged as its hardest question. 040 ownership: provenance is a baseline record plus a verified post-image hash, so a baseline-absent artifact is deleted only when the hash still matches and a user edit is preserved and reported as a conflict instead. The server rereads persisted intent before gather and again under the lock. --- .../010_catalog_seam.md | 836 +++++++++++++++++ .../020_history_isolation.md | 871 ++++++++++++++++++ .../030_lock_protocol.md | 800 ++++++++++++++++ .../040_ownership_convergence.md | 848 +++++++++++++++++ 4 files changed, 3355 insertions(+) create mode 100644 devlog/_plan/260804_codex_write_substrate/010_catalog_seam.md create mode 100644 devlog/_plan/260804_codex_write_substrate/020_history_isolation.md create mode 100644 devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md create mode 100644 devlog/_plan/260804_codex_write_substrate/040_ownership_convergence.md diff --git a/devlog/_plan/260804_codex_write_substrate/010_catalog_seam.md b/devlog/_plan/260804_codex_write_substrate/010_catalog_seam.md new file mode 100644 index 000000000..0824b8533 --- /dev/null +++ b/devlog/_plan/260804_codex_write_substrate/010_catalog_seam.md @@ -0,0 +1,836 @@ +# WP9 — split Codex catalog gather from commit + +Research: `001_catalog_seam.md`. Read it first; this doc is the diff. + +The incident is r2 #1: the OFF design needed provider discovery outside a +per-`CODEX_HOME` lock and native writes inside it, but management exposes one +`Promise` callback that gathers and writes before resolving +(`../260803_codex_desktop_toggle/008_audit_synthesis_wp4_r2.md:17-24`, +`src/server/management/context.ts:9-18`). Today `refreshCodexModelCatalog` awaits +the mixed `syncCatalogModels`, checks the file that function may just have +written, and then performs a second cache write (`src/codex/refresh.ts:40-52`, +`src/codex/catalog/sync.ts:507-569,600-616`). This phase adds an opaque, +point-in-time candidate, a pure gather, a fixed synchronous commit, a revision +check, and typed dispositions. It does not add the native write lock; WP11 owns +that. Until WP11, the existing orchestrator calls gather and then commit directly, +so the split is independently useful and the revision guard is exercised, without +pretending a lock already exists. + +## IN / OUT + +IN — production: + +- `src/codex/refresh.ts` (MODIFY) — canonical candidate, revisions, gather/commit + exports, outcome union, and the compatibility orchestrator. +- `src/codex/catalog.ts` (MODIFY) — re-export the prepared catalog contract from + the existing facade; do not create a second catalog entry point. +- `src/codex/catalog/sync.ts` (MODIFY) — turn assembly into a write-free prepared + payload and expose the fixed synchronous writer. +- `src/codex/catalog/bundled.ts` (MODIFY) — replace the materializing sync fallback + with in-memory catalog bytes for the candidate. +- `src/codex/catalog/parsing.ts` (MODIFY) — prepare create-once pristine-backup + bytes without writing; retain restore's existing writer path. +- `src/codex/catalog/provider-fetch.ts` (MODIFY) — return typed degradation notices + and distinguish token resolution from provider-network failure. +- `src/codex/sync.ts` (MODIFY) — consume typed refresh outcomes and preserve the + existing ordinary-failure injection fallback. +- `src/server/management/context.ts` (MODIFY) — replace the void callback with the + paired candidate seam and typed best-effort result. +- `src/server/management-api.ts` (MODIFY) — orchestrate the injected or production + pair, never swallow the disposition. +- `src/server/management/provider-routes.ts` (MODIFY) — six persisted mutations + attach `catalogRefresh`. +- `src/server/management/model-routes.ts` (MODIFY) — six persisted mutations attach + `catalogRefresh`. +- `src/server/management/combo-routes.ts` (MODIFY) — two persisted mutations attach + `catalogRefresh` without suppressing Claude follow-up work. +- `src/server/management/agent-settings-routes.ts` (MODIFY) — two persisted + mutations attach `catalogRefresh` without suppressing Claude/Desktop follow-up + work. +- `src/server/management/config-routes.ts` (MODIFY) — explicit `/api/sync` maps + typed authorization/contention/disk results instead of flattening every failure + to 500. +- `structure/03_catalog-and-subagents.md` (MODIFY) — source-of-truth statement for + the candidate/revision contract. +- `structure/05_gui-and-management-api.md` (MODIFY) — source-of-truth statement for + best-effort mutation responses versus explicit sync. +- `docs-site/src/content/docs/reference/management-api.md` (MODIFY), plus the + matching `ja`, `ko`, `ru`, and `zh-cn` files (MODIFY) — document the additive + `catalogRefresh` field and explicit-sync status mapping. + +IN — tests: + +- `tests/codex-refresh.test.ts` (MODIFY) — pure gather, bounded commit, one-shot, + stale config/base revisions, and partial write receipts. +- `tests/codex-sync-api.test.ts` (MODIFY) — typed fallback and no-injection cases. +- `tests/codex-models-cache-invalidate.test.ts` (MODIFY) — receipt-driven app-server + invalidation behavior. +- `tests/injection-model-api.test.ts` (MODIFY) — immutable config snapshot rather + than forwarding a mutable config reference. +- `tests/model-visibility-management-api.test.ts` (MODIFY), + `tests/management-provider-validation.test.ts` (MODIFY), + `tests/combo-management-api.test.ts` (MODIFY), `tests/combos.test.ts` (MODIFY), + and `tests/codex-v2-gate.test.ts` (MODIFY) — paired seam and response disposition. +- `tests/management-integration-routes.test.ts` (MODIFY), + `tests/management-client-config-route.test.ts` (MODIFY), + `tests/responses-shadow-intercept.test.ts` (MODIFY), + `tests/server-combo-failover-e2e.test.ts` (MODIFY), and + `tests/catalog-input-modality-enum.test.ts` (MODIFY) — fixture type migration; + no new behavior in routes that do not refresh. + +OUT: + +- `src/integrations/native/**`, `src/service.ts`, and a native write lock — WP11 + creates the lock and wraps `commitCodexCatalogCandidate`; WP9 must compile and + behave correctly without it. +- Desired-state OFF reads and ownership admission — WP12 owns those authorities. + The outcome union reserves `desired_off` and the orchestrator handles it, but + WP9 does not invent a flag or emit that result. +- `gui/**` — responses are additive and the current dashboard does not need a new + visual state in this substrate phase. +- `src/codex/history-provider.ts` — WP10 isolates history separately. +- `syncCodexModelsCacheFromCatalog` — retain the explicit raw-copy utility at + `src/codex/refresh.ts:29-32`; it is not the active expired-wrapper commit. +- Transactional rollback — catalog and cache are separate atomic replacements; + a receipt reports partial progress instead of claiming all-or-nothing behavior + (`src/config.ts:178-230`, `src/codex/catalog/sync.ts:568,601-613`). + +## The candidate and the two operations + +MODIFY `src/codex/refresh.ts`. The public type is structurally opaque because its +brand symbol is module-private; the payload lives in a `WeakMap`, so it cannot be +JSON-serialized, reconstructed by a caller, or inspected for credentials. The +state holds strings, paths, revisions, notices, and result metadata only — no +mutable `OcxConfig`, file handle, callback, or promise. + +```ts +const codexCatalogCandidateBrand: unique symbol = Symbol("CodexCatalogCandidate"); + +export interface CodexCatalogCandidate { + readonly [codexCatalogCandidateBrand]: true; +} + +interface CodexCatalogCandidateState { + readonly prepared: PreparedCodexCatalogCommit; + readonly revision: CodexCatalogRevision; + readonly result: Omit; + readonly notices: readonly CatalogGatherNotice[]; + consumed: boolean; +} + +const candidateStates = new WeakMap(); + +/** + * Discover providers and load every source needed to assemble the exact catalog, + * backup, and expired-cache bytes that a later commit may write. + * + * WHY: provider auth, network I/O, bundled `codex debug models --bundled`, JSON + * parsing, merging, and serialization are unbounded relative to a native-write + * critical section. This operation therefore performs no mkdir/copy/write/rename + * and returns an opaque point-in-time candidate instead of exposing writable + * payloads to callers. + */ +export async function gatherCodexCatalogCandidate( + config: OcxConfig, +): Promise; + +/** + * Revalidate and consume one gathered candidate, then perform only its prepared + * create-once backup writes, catalog replacement, and expired-cache replacement. + * + * WHY: r2 #1 can be fixed only if the eventual lock owner can call a synchronous, + * fixed-write function. Rechecking config and base-catalog revisions here prevents + * a candidate assembled from obsolete state from overwriting a newer catalog. + * No provider call, auth resolution, subprocess, parse, merge, serialization, or + * await is permitted below this boundary. + */ +export function commitCodexCatalogCandidate( + candidate: CodexCatalogCandidate, +): CodexCatalogCommitResult; +``` + +`gatherCodexCatalogCandidate` calls a write-free `prepareCatalogSync(config)` in +`src/codex/catalog/sync.ts`. That helper returns final catalog bytes, expired-cache +wrapper bytes, optional pristine-backup path/byte pairs, exact target paths, +`added`, `comboOmissions`, and notices. +The gather then freezes a tiny branded handle and stores the internal state in the +`WeakMap`. `commitCodexCatalogCandidate` rejects a missing/consumed handle as +`stale_candidate`, marks a valid handle consumed before the first write, compares +revisions, and invokes `writePreparedCatalogCommit` only on an exact match. + +Mark-before-write is deliberate. Retrying a partially written candidate would +replay old bytes after a later convergence. A disk failure returns its receipt; +the caller regathers instead of recommitting the consumed handle. + +## C2 — the revision guard + +The guard uses content revisions, not mtimes. An mtime can change without content, +can be restored, and has platform-dependent resolution; the merge at +`src/codex/catalog/sync.ts:520-565` depends on exact catalog bytes. **INFERRED:** +SHA-256 of canonical config input and exact base-catalog bytes is the smallest +evidence that detects every input change relevant to this candidate while avoiding +raw credential retention. + +```ts +type ContentRevision = + | { readonly state: "absent" } + | { readonly state: "present"; readonly sha256: string }; + +interface CodexCatalogRevision { + readonly configSha256: string; + readonly baseCatalog: ContentRevision; + readonly codexHome: string; + readonly catalogPath: string; + readonly cachePath: string; +} +``` + +Gather captures exactly these values: + +1. `configSha256`: SHA-256 of stable-key JSON containing the complete `providers` + object (including auth mode and credential/env references), `disabledModels`, + `customModels`, `combos`, `subagentModels`, `multiAgentMode`, + `providerContextCaps`, `contextCapValue`, `modelCacheTtlMs`, `websockets`, and the + fresh `isMultiAgentV2Enabled()` value. Those are the inputs consumed by provider + gathering and final assembly (`src/codex/catalog/provider-fetch.ts:98-142,670-719,785-814`, + `src/codex/catalog/sync.ts:533-565`). The digest includes credential values so a + key rotation invalidates stale discovery, but the candidate never retains or + returns those values. +2. `baseCatalog`: `absent`, or SHA-256 of the exact bytes read from `catalogPath` + before parsing. This is the merge source whose routed and user-native rows are + preserved (`src/codex/catalog/sync.ts:517-523,430-468`). +3. `codexHome`: `realpathSync.native(resolveCodexHomeDir())`, plus the resolved + `catalogPath` and `activeCodexModelsCachePath()`. This prevents an environment or + config-path change from redirecting prepared bytes to another home + (`structure/02_config-and-codex-home.md:3-21`, + `src/codex/catalog/parsing.ts:73,167`). + +At commit, synchronously and before any mkdir/write, recompute: + +- the config digest from `readConfigDiagnostics().config` plus a fresh feature-flag + read; +- the canonical home and both target paths; and +- the exact current catalog content digest/absence marker. + +Every field must equal the candidate revision. Any mismatch returns +`{ status: "skipped", reason: "stale_candidate", retryable: true }`, consumes the +candidate, and produces an all-false write receipt. No backup directory, backup, +catalog, or cache is created. A newly appearing backup does not make the candidate +stale: backups are create-once; the commit rechecks each optional backup path and +skips that one write if another actor already created it +(`src/codex/catalog/parsing.ts:428-444`). + +Provider inventory changing upstream after gather is not a revision mismatch. The +candidate represents one completed discovery. A subsequent refresh may supersede +it; wall-clock age is not used (`src/codex/catalog/provider-fetch.ts:481-510,670-717`). + +Without WP11 there is no cross-process critical section around compare-and-write. +WP9 still makes the operation correct for every revision change completed before +commit begins and for all same-process interleavings because commit contains no +`await`. **INFERRED:** an external process can still replace the catalog after the +comparison and before rename; WP11 closes that remaining TOCTOU by placing this +unchanged synchronous function under the shared per-home lock. WP9 must not claim +cross-process linearizability before that phase. + +## Typed outcomes + +MODIFY `src/codex/refresh.ts` with one closed public outcome and a public, +credential-free management projection: + +```ts +export type CatalogGatherNotice = { + kind: "provider_degraded"; + reason: "provider_network" | "provider_auth"; + fallback: "stale" | "configured"; +}; + +export interface CatalogWriteReceipt { + catalogBackup: boolean; + legacyBackup: boolean; + catalog: boolean; + cache: boolean; +} + +export type CodexCatalogCommitResult = + | { status: "committed"; result: CodexCatalogRefreshResult; writes: CatalogWriteReceipt } + | { status: "skipped"; reason: "stale_candidate"; retryable: true; writes: CatalogWriteReceipt } + | { status: "failed"; reason: "disk"; phase: "commit"; retryable: true; writes: CatalogWriteReceipt }; + +export type CodexCatalogSkipReason = + | "catalog_unavailable" + | "desired_off" + | "gather_busy" + | "lock_busy" + | "stale_candidate"; + +export type CodexCatalogRefreshOutcome = + | { status: "committed"; result: CodexCatalogRefreshResult; notices: readonly CatalogGatherNotice[]; writes: CatalogWriteReceipt } + | { status: "skipped"; reason: CodexCatalogSkipReason; retryable: boolean; writes: CatalogWriteReceipt } + | { status: "failed"; reason: "provider_network" | "provider_auth" | "disk"; phase: "gather" | "commit"; retryable: boolean; writes: CatalogWriteReceipt }; + +export type CatalogRefreshDisposition = + | { status: "committed"; degraded: boolean } + | { status: "skipped"; reason: CodexCatalogSkipReason; retryable: boolean } + | { status: "failed"; reason: "provider_network" | "provider_auth" | "disk"; retryable: boolean; partialWrite: boolean }; +``` + +Do not expose provider names, URLs, token text, catalog paths, or digests through +`CatalogRefreshDisposition`. `committed` plus `degraded:true` is how a stale/static +provider fallback remains visible without turning the primary route into a failure. +`partialWrite` is `true` when any receipt bit is true. + +`refreshCodexModelCatalog(config, deps)` becomes the direct gather-then-commit +orchestrator for WP9. It maps `CatalogGatherBusyError` to `skipped/gather_busy`, +typed token-resolution failure to `failed/provider_auth`, escaped provider fetch +failure to `failed/provider_network`, no source to `skipped/catalog_unavailable`, +and commit errors to their returned result. It does not emit `desired_off` or +`lock_busy`; WP11/WP12 add those admission results without changing callers. + +### Every management caller + +All 16 rows are best-effort by design. For every variant they retain their current +2xx/201 primary status, attach `catalogRefresh`, and never roll back the config +mutation that already landed. `committed` reports `degraded:false|true`; OFF, +gather/lock contention, and stale candidates report `skipped` with reason and +retryability; auth/network/disk report `failed`. The route-specific continuation is +the only difference. + +| Exact outcome | `catalogRefresh` projection on every row below | +|---|---| +| committed, no notices | `{ status: "committed", degraded: false }` | +| committed with provider network/auth fallback | `{ status: "committed", degraded: true }` | +| `desired_off` | `{ status: "skipped", reason: "desired_off", retryable: false }` | +| `gather_busy` | `{ status: "skipped", reason: "gather_busy", retryable: true }` | +| `lock_busy` | `{ status: "skipped", reason: "lock_busy", retryable: true }` | +| `stale_candidate` | `{ status: "skipped", reason: "stale_candidate", retryable: true }` | +| `catalog_unavailable` | `{ status: "skipped", reason: "catalog_unavailable", retryable: false }` | +| `provider_auth` | `{ status: "failed", reason: "provider_auth", retryable: false, partialWrite: false }` | +| `provider_network` | `{ status: "failed", reason: "provider_network", retryable: true, partialWrite: false }` | +| `disk` | `{ status: "failed", reason: "disk", retryable: true, partialWrite: }` | + +| # | Caller and current line | Best-effort | Committed / degraded | OFF / gather busy / lock busy / stale | Auth / network / disk | +|---|---|---|---|---|---| +| P1 | provider add/overwrite, `src/server/management/provider-routes.ts:147-148` | YES | Return current 200 plus disposition. | Same 200; provider remains saved. | Same 200; no rollback. | +| P2 | ordinary provider edit/toggle, `src/server/management/provider-routes.ts:338-344` | YES | Return current 200 plus disposition. | Same 200; edited provider remains saved. | Same 200; no rollback. | +| P3 | provider delete, `src/server/management/provider-routes.ts:479-488` | YES | Return current 200 plus disposition. | Same 200; warn stale native rows through disposition. | Same 200; no rollback. | +| P4 | global context-cap value, `src/server/management/provider-routes.ts:503-513` | YES | `respond` includes disposition. | Same 200 and cap body. | Same 200 and cap body. | +| P5 | all context-cap toggles, `src/server/management/provider-routes.ts:516-528` | YES | `respond` includes disposition. | Same 200 and cap body. | Same 200 and cap body. | +| P6 | one provider context cap, `src/server/management/provider-routes.ts:531-547` | YES | `respond` includes disposition. | Same 200 and cap body. | Same 200 and cap body. | +| M1 | disabled models, `src/server/management/model-routes.ts:208-215` | YES | Return current 200 plus disposition. | Same 200; blocklist remains saved. | Same 200; no rollback. | +| M2 | model visibility, `src/server/management/model-routes.ts:221-314` | YES | Return current 200 plus disposition. | Same 200; visibility intent remains saved. | Same 200; no rollback. | +| M3 | custom model create, `src/server/management/model-routes.ts:321-353` | YES | Preserve 201; append disposition. | Preserve 201. | Preserve 201. | +| M4 | custom model edit, `src/server/management/model-routes.ts:356-391` | YES | Return current 200 plus disposition. | Same 200. | Same 200. | +| M5 | custom model delete, `src/server/management/model-routes.ts:394-405` | YES | Return current 200 plus disposition. | Same 200; possible stale row is explicit. | Same 200. | +| M6 | selected models, `src/server/management/model-routes.ts:426-441` | YES | Return current 200 plus disposition. | Same 200; allowlist remains saved. | Same 200. | +| C1 | combo create/update/rename, `src/server/management/combo-routes.ts:190-200` | YES | Return current 200 plus disposition. | Same 200; still run Claude sync when `shouldSyncClaudeAgentDefs`. | Same 200; still run Claude sync. | +| C2 | combo delete, `src/server/management/combo-routes.ts:203-217` | YES | Return current 200 plus disposition. | Same 200. | Same 200. | +| A1 | v2/settings write, `src/server/management/agent-settings-routes.ts:224-294` | YES | Return current 200 plus disposition and existing warnings. | Same 200; feature/config writes remain authoritative. | Same 200; no rollback. | +| A2 | subagent model write, `src/server/management/agent-settings-routes.ts:518-528` | YES | Return current 200 plus disposition. | Same 200; still run Claude and Desktop follow-ups. | Same 200; still run both follow-ups. | + +Explicit sync is not in that table. `syncModelsToCodex` keeps injection fallback for +`catalog_unavailable`, provider degradation, provider auth/network failure, and disk +failure, matching the current catch-and-continue contract at +`src/codex/sync.ts:83-110` and `tests/codex-sync-api.test.ts:148-166`. It returns +before `injectCodexConfig` for `desired_off`, `lock_busy`, or `stale_candidate`: +those are authorization/serialization refusals, not missing catalog data. Gather +busy is retryable and also returns before injection so an explicit sync cannot +claim fresh native state while another revision is being assembled. `/api/sync` +maps `desired_off` and `stale_candidate` to 409, `gather_busy`/`lock_busy` to 503 +with `Retry-After: 1`, and non-fallback disk failure to 500 +(`src/server/management/config-routes.ts:261-268`). + +## Diff + +### Catalog preparation and fixed writes + +MODIFY `src/codex/catalog/bundled.ts` at current lines 225-234. The fallback remains +in memory; it does not materialize a source while loading: + +```diff + export function loadCatalogForSync(path: string): RawCatalog | null { +@@ + return readCatalog(catalogBackupPathFor(path)) + ?? (isDefaultCatalogPath(path) ? readCatalog(legacyCatalogBackupPath()) : null) + ?? readCatalog(activeCodexModelsCachePath()) +- ?? materializeBundledCodexCatalog(path) + ?? catalog; + } +``` + +Retain `materializeBundledCodexCatalog` for its existing public callers +(`src/codex/catalog.ts:6`); only catalog gather stops calling it. + +MODIFY `src/codex/catalog/parsing.ts` around current lines 428-444. Extract a pure +backup planner beside the existing restore-facing writer: + +```diff ++export interface PreparedCatalogBackup { ++ path: string; ++ bytes: string; ++ kind: "catalog" | "legacy"; ++} ++ ++export function prepareCatalogBackups( ++ catalogPath: string, ++ catalog: RawCatalog, ++ onDiskBytes: string | null, ++): PreparedCatalogBackup[] { ++ const source = onDiskBytes === null ? null : parseCatalogJson(onDiskBytes); ++ const pristineBytes = source && !catalogHasRoutedEntries(source) ++ ? onDiskBytes ++ : !catalogHasRoutedEntries(catalog) ++ ? JSON.stringify(catalog, null, 2) + "\n" ++ : null; ++ if (pristineBytes === null) return []; ++ return [ ++ { path: catalogBackupPathFor(catalogPath), bytes: pristineBytes, kind: "catalog" }, ++ ...(isDefaultCatalogPath(catalogPath) ++ ? [{ path: legacyCatalogBackupPath(), bytes: pristineBytes, kind: "legacy" as const }] ++ : []), ++ ]; ++} ++ + export function writePristineCatalogBackup(backupPath: string, catalogPath: string, catalog: RawCatalog): void { +``` + +MODIFY `src/codex/catalog/sync.ts` at current lines 507-569 and 600-616. The full +assembly remains where it is, but its output is bytes, not mutations: + +```diff +-export async function syncCatalogModels(config: OcxConfig): Promise<{ ++export interface PreparedCodexCatalogCommit { + added: number; + path: string; +- catalogWritten: boolean; + comboOmissions: ComboCatalogOmission[]; +-}> { ++ catalogBytes: string | null; ++ cachePath: string; ++ cacheBytes: string | null; ++ backups: PreparedCatalogBackup[]; ++ baseCatalogBytes: string | null; ++ notices: CatalogGatherNotice[]; ++} ++ ++export async function prepareCatalogSync(config: OcxConfig): Promise { + const catalogPath = readCodexCatalogPath(); +- const catalog = loadCatalogForSync(catalogPath); +- if (!catalog) return { added: 0, path: catalogPath, catalogWritten: false, comboOmissions: [] }; ++ const catalog = loadCatalogForSync(catalogPath); ++ if (!catalog) return emptyPreparedCatalogCommit(catalogPath); ++ const baseCatalogBytes = readFileOrNull(catalogPath); + + // The bundled catalog is a reliable native template on the default path, but it is not the +@@ +- const onDiskCatalog = readCatalog(catalogPath); ++ const onDiskCatalog = baseCatalogBytes === null ? null : parseCatalogJson(baseCatalogBytes); +@@ +- const goModels = await gatherRoutedModels(config, { comboOmissions }); +- try { +- // Once-only: preserve the PRISTINE pre-opencodex catalog as the native-priority baseline +- // (later syncs would otherwise overwrite it with featured-modified priorities). +- ensureCatalogBackup(catalogPath, catalog); +- } catch { /* backup best-effort */ } ++ const notices: CatalogGatherNotice[] = []; ++ const goModels = await gatherRoutedModels(config, { comboOmissions, notices }); ++ const backups = prepareCatalogBackups(catalogPath, catalog, baseCatalogBytes); +@@ +- atomicWriteFile(catalogPath, JSON.stringify(catalog, null, 2) + "\n"); +- return { added: goEntries.length, path: catalogPath, catalogWritten: true, comboOmissions }; ++ const catalogBytes = JSON.stringify(catalog, null, 2) + "\n"; ++ const cacheBytes = JSON.stringify({ ++ fetched_at: "2000-01-01T00:00:00Z", ++ client_version: "0.0.0", ++ models: catalog.models ?? catalog, ++ }, null, 2) + "\n"; ++ return { ++ added: goEntries.length, ++ path: catalogPath, ++ comboOmissions, ++ catalogBytes, ++ cachePath: activeCodexModelsCachePath(), ++ cacheBytes, ++ backups, ++ baseCatalogBytes, ++ notices, ++ }; + } ++ ++export function writePreparedCatalogCommit( ++ prepared: PreparedCodexCatalogCommit, ++): { result: CodexCatalogRefreshResult; writes: CatalogWriteReceipt } { ++ // No await, parsing, serialization, provider call, or subprocess below this line. ++ // Set each receipt bit only after its atomic replacement returns. ++ // Backup writes remain create-once: existsSync(path) means skip, never overwrite. ++ // On failure throw CatalogCommitDiskError carrying the receipt completed so far. ++} +``` + +`writePreparedCatalogCommit` performs at most four atomic replacements in this +fixed order: keyed backup, optional legacy backup, catalog, cache. The final catalog +replacement is also what materializes an absent default catalog; there is no fifth +"source" write. It creates only the parent directories needed by a prepared write. +The count never scales with providers or models. `invalidateCodexModelsCache` remains +for startup/manual cache invalidation at `src/codex/catalog/sync.ts:600-616`; it is +not called by the candidate commit. + +MODIFY `src/codex/catalog/provider-fetch.ts` at current lines 410-428, 494-510, and +670-693. `resolveModelsAuthToken` rejection becomes `CatalogProviderAuthError`, and +each fallback pushes one sanitized notice with `reason` and `fallback`; no provider +name or exception text enters the public notice. The flight result carries notices +so same-key joiners receive the exact degradation set from the flight they joined, +just as `comboOmissions` is flight-local at lines 696-700. + +MODIFY the facade at current `src/codex/catalog.ts:6,11-12`: + +```diff +-export { isSpawnableCodexCandidate, codexExecInvocation, loadBundledCodexCatalog, materializeBundledCodexCatalog, loadCatalogTemplate } from "./catalog/bundled"; ++export { isSpawnableCodexCandidate, codexExecInvocation, loadBundledCodexCatalog, materializeBundledCodexCatalog, loadCatalogForSync, loadCatalogTemplate } from "./catalog/bundled"; +@@ +-export { MAX_SPAWN_AGENT_MODEL_OVERRIDES, effectiveSubagentRoster, buildCatalogEntries, resetCatalogRuntimeStateForTests, orderForSubagents, mergeCatalogEntriesForSync, syncCatalogModels, restoreCodexCatalog, invalidateCodexModelsCache } from "./catalog/sync"; ++export { MAX_SPAWN_AGENT_MODEL_OVERRIDES, effectiveSubagentRoster, buildCatalogEntries, resetCatalogRuntimeStateForTests, orderForSubagents, mergeCatalogEntriesForSync, prepareCatalogSync, writePreparedCatalogCommit, restoreCodexCatalog, invalidateCodexModelsCache } from "./catalog/sync"; ++export type { PreparedCodexCatalogCommit } from "./catalog/sync"; +``` + +`syncCatalogModels` is removed only after `rg -n "syncCatalogModels" src tests` +shows every production and test import migrated. This is an intentional internal +contract replacement, not a silent facade break. + +### `src/codex/refresh.ts` + +Replace current lines 1-27 and 34-53 with the types and operations above. The +orchestrator diff is: + +```diff +-export async function refreshCodexModelCatalog( +- config: OcxConfig, +- deps: RefreshDeps = defaultDeps, +-): Promise { +- const result = await deps.syncCatalogModels(config); +- const catalogExists = deps.existsSync(result.path); +- const catalogWritten = result.catalogWritten === true; +- const comboOmissions = result.comboOmissions ?? []; +- if (!catalogExists) { +- return { ...result, catalogExists, catalogWritten: false, cacheSynced: false, comboOmissions }; +- } +- const cacheSynced = deps.invalidateCodexModelsCache(); +- return { ...result, catalogExists, catalogWritten, cacheSynced, comboOmissions }; ++export async function refreshCodexModelCatalog( ++ config: OcxConfig, ++ deps: CatalogCandidateDeps = defaultCandidateDeps, ++): Promise { ++ try { ++ const candidate = await deps.gatherCodexCatalogCandidate(config); ++ return toRefreshOutcome(deps.commitCodexCatalogCandidate(candidate)); ++ } catch (error) { ++ return catalogGatherFailureOutcome(error); ++ } + } +``` + +`CatalogCandidateDeps` owns only deterministic readers/writers needed to test the +candidate. Production defaults use `prepareCatalogSync`, +`writePreparedCatalogCommit`, `readConfigDiagnostics`, feature-state/path readers, +and SHA-256. Tests inject all filesystem/config observations; no test touches the +user's real home. + +### Management contract and orchestrator + +MODIFY `src/server/management/context.ts` at current lines 9-18 and 54-70: + +```diff ++import type { ++ CatalogRefreshDisposition, ++ CodexCatalogCandidate, ++ CodexCatalogCommitResult, ++} from "../../codex/refresh"; +@@ +- refreshCodexCatalog?: () => Promise; ++ codexCatalog?: { ++ gather: (config: OcxConfig) => Promise; ++ commit: (candidate: CodexCatalogCandidate) => CodexCatalogCommitResult; ++ }; +@@ +- refreshCodexCatalogBestEffort: () => Promise; ++ refreshCodexCatalogBestEffort: () => Promise; +``` + +The pair is one optional object so a test cannot inject gather without commit or +commit without gather. It does not own desired state or locking; WP11 wraps the +same `commit` call at the orchestrator. + +MODIFY `src/server/management-api.ts` at current lines 105-113 and remove the now +dead `CatalogGatherBusyError` route-level mapper at lines 159-163. Best-effort means +non-throwing typed disposition for both production and injected paths: + +```diff +- async function refreshCodexCatalogBestEffort(): Promise { +- if (deps.refreshCodexCatalog) return deps.refreshCodexCatalog(); +- try { +- const { refreshCodexModelCatalog } = await import("../codex/refresh"); +- await refreshCodexModelCatalog(config); +- } catch { +- /* catalog absent */ +- } ++ async function refreshCodexCatalogBestEffort(): Promise { ++ const refresh = await import("../codex/refresh"); ++ const outcome = deps.codexCatalog ++ ? await refresh.refreshCodexModelCatalog(config, { ++ gatherCodexCatalogCandidate: deps.codexCatalog.gather, ++ commitCodexCatalogCandidate: deps.codexCatalog.commit, ++ }) ++ : await refresh.refreshCodexModelCatalog(config); ++ return refresh.catalogRefreshDisposition(outcome); + } +``` + +### Caller sites + +Every caller captures the result immediately where it currently awaits. The +response-spread pattern is identical; the examples below cover all response shapes. + +MODIFY `src/server/management/provider-routes.ts` current lines 147-148, 338-344, +487-488, and 500-547: + +```diff +- await refreshCodexCatalogBestEffort(); +- return jsonResponse({ success: true, name }); ++ const catalogRefresh = await refreshCodexCatalogBestEffort(); ++ return jsonResponse({ success: true, name, catalogRefresh }); +@@ +- await refreshCodexCatalogBestEffort(); ++ const catalogRefresh = await refreshCodexCatalogBestEffort(); + return jsonResponse({ + success: true, + name, + disabled: config.providers[name]!.disabled === true, + hasApiKey: !!config.providers[name]!.apiKey, ++ catalogRefresh, + }); +@@ +- await refreshCodexCatalogBestEffort(); +- return jsonResponse({ success: true, ...(fallbackDefault ? { defaultProvider: fallbackDefault } : {}) }); ++ const catalogRefresh = await refreshCodexCatalogBestEffort(); ++ return jsonResponse({ success: true, ...(fallbackDefault ? { defaultProvider: fallbackDefault } : {}), catalogRefresh }); +@@ +- const respond = () => jsonResponse({ ok: true, cap: DEFAULT_PROVIDER_CONTEXT_CAP, value: globalContextCapValue(config), caps: providerContextCaps(config) }); ++ const respond = async () => { ++ const catalogRefresh = await refreshCodexCatalogBestEffort(); ++ return jsonResponse({ ++ ok: true, cap: DEFAULT_PROVIDER_CONTEXT_CAP, value: globalContextCapValue(config), ++ caps: providerContextCaps(config), catalogRefresh, ++ }); ++ }; +@@ +- await refreshCodexCatalogBestEffort(); +- return respond(); ++ return respond(); +``` + +Apply the last two-line replacement to all three cap branches at current lines +512-513, 527-528, and 546-547. + +MODIFY `src/server/management/model-routes.ts` current lines 214-215, 313-314, +352-353, 390-391, 404-405, and 440-441: + +```diff +- await refreshCodexCatalogBestEffort(); +- return jsonResponse({ ok: true, disabled }); ++ const catalogRefresh = await refreshCodexCatalogBestEffort(); ++ return jsonResponse({ ok: true, disabled, catalogRefresh }); +@@ +- await refreshCodexCatalogBestEffort(); +- return jsonResponse({ ok: true, scope, provider, enabled: body.enabled, disabled }); ++ const catalogRefresh = await refreshCodexCatalogBestEffort(); ++ return jsonResponse({ ok: true, scope, provider, enabled: body.enabled, disabled, catalogRefresh }); +@@ +- await refreshCodexCatalogBestEffort(); +- return jsonResponse(entry, 201); ++ const catalogRefresh = await refreshCodexCatalogBestEffort(); ++ return jsonResponse({ ...entry, catalogRefresh }, 201); +@@ +- await refreshCodexCatalogBestEffort(); +- return jsonResponse(cm); ++ const catalogRefresh = await refreshCodexCatalogBestEffort(); ++ return jsonResponse({ ...cm, catalogRefresh }); +@@ +- await refreshCodexCatalogBestEffort(); +- return jsonResponse({ ok: true }); ++ const catalogRefresh = await refreshCodexCatalogBestEffort(); ++ return jsonResponse({ ok: true, catalogRefresh }); +@@ +- await refreshCodexCatalogBestEffort(); +- return jsonResponse({ ok: true, provider, selected: models }); ++ const catalogRefresh = await refreshCodexCatalogBestEffort(); ++ return jsonResponse({ ok: true, provider, selected: models, catalogRefresh }); +``` + +MODIFY `src/server/management/combo-routes.ts` current lines 198-200 and 216-217. +Capture before independent follow-up work, but do not return early: + +```diff +- await refreshCodexCatalogBestEffort(); ++ const catalogRefresh = await refreshCodexCatalogBestEffort(); + if (shouldSyncClaudeAgentDefs) await syncClaudeAgentDefsBestEffort(); +- return jsonResponse({ success: true, id, model: newPublicModel, combo: normalized }); ++ return jsonResponse({ success: true, id, model: newPublicModel, combo: normalized, catalogRefresh }); +@@ +- await refreshCodexCatalogBestEffort(); +- return jsonResponse({ success: true, id }); ++ const catalogRefresh = await refreshCodexCatalogBestEffort(); ++ return jsonResponse({ success: true, id, catalogRefresh }); +``` + +MODIFY `src/server/management/agent-settings-routes.ts` current lines 280-294 and +525-528: + +```diff +- await refreshCodexCatalogBestEffort(); ++ const catalogRefresh = await refreshCodexCatalogBestEffort(); +@@ + agentsMaxDepthAppliesWhenV2Disabled: !enabled, + warnings, ++ catalogRefresh, + }); +@@ +- await refreshCodexCatalogBestEffort(); ++ const catalogRefresh = await refreshCodexCatalogBestEffort(); + await syncClaudeAgentDefsBestEffort(); + await autoApplyDesktopBestEffort(); +- return jsonResponse({ ok: true, applied: chosen }); ++ return jsonResponse({ ok: true, applied: chosen, catalogRefresh }); +``` + +### Explicit sync + +MODIFY `src/codex/sync.ts` current lines 9-22 and 83-110. Add +`catalogRefresh?: CodexCatalogRefreshOutcome` to `CodexSyncResult` (the existing +external-provider branch at lines 56-71 performs no catalog attempt); replace the +throw/catch with a switch. `committed` fills the existing booleans from its result. +`committed` with notices sets a warning but continues injection. Ordinary gather +failure, unavailable catalog, and disk failure also continue with +`catalogPathForInjection = undefined`, preserving the pinned fallback. OFF/busy/stale +return `ok:false` before line 110 with the typed outcome attached. + +MODIFY `src/server/management/config-routes.ts` current lines 261-268: + +```diff + const result = await syncModelsToCodex(undefined, config, null); ++ const status = explicitSyncHttpStatus(result.catalogRefresh, result.ok); ++ const response = jsonResponse({ + ...attachStaleAppServerHint(result), + ...(result.ok ? {} : { error: result.message }), +- }, result.ok ? 200 : 500); ++ }, status, req, config); ++ if (status === 503) response.headers.set("Retry-After", "1"); ++ return response; +``` + +`jsonResponse` currently accepts exactly data, status, request, and config +(`src/server/auth-cors.ts:184-188`), so the header is set on the returned response; +do not invent a fifth argument or silently omit `Retry-After`. + +## Tests + +### `tests/codex-refresh.test.ts` + +Replace the all-in-one dependency tests at current lines 60-216 with split cases: + +1. Gather runs bundled source loading, provider discovery, assembly, serialization, + backup preparation, and cache-wrapper preparation; injected write spies remain + zero before commit. +2. Commit performs only the fixed write list in order. Inject every async/provider/ + parser dependency with a function that throws if called during commit. +3. `catalog_unavailable` returns skipped with no write. +4. Provider HTTP/network fallback produces committed plus a + `provider_degraded/provider_network` notice. +5. Missing OAuth token fallback produces committed plus a + `provider_degraded/provider_auth` notice; token-resolution throw produces + `failed/provider_auth` with no commit. +6. `CatalogGatherBusyError` becomes retryable `skipped/gather_busy`. +7. Catalog write succeeds and cache write fails: `failed/disk`, receipt has + `catalog:true`, `cache:false`, and the candidate is consumed. +8. A second commit of the same candidate returns `stale_candidate` and writes zero. +9. Create-once backup appears after gather: commit skips that backup, writes catalog + and cache, and does not overwrite backup bytes. + +The C2 activation cases are mandatory: + +10. Gather candidate A; change one catalog-affecting persisted config field before + commit; assert `stale_candidate`, all-false receipt, and byte-identical catalog, + cache, and backup directory state. +11. Gather candidate A; replace the base catalog bytes with candidate B's committed + catalog; commit A; assert `stale_candidate` and B's bytes survive. +12. Gather from absent catalog; create a catalog before commit; assert absence versus + presence is a revision mismatch. +13. Change a non-catalog config key such as `shutdownTimeoutMs`; assert the canonical + catalog-config digest is unchanged and commit succeeds. This prevents whole-file + hashing from turning unrelated settings into false contention. + +### Caller and sync tests + +- `tests/model-visibility-management-api.test.ts`: inject gather success and commit + `stale_candidate`; assert HTTP 200, persisted disabled state, and + `catalogRefresh.status === "skipped"`. This is the proof that a best-effort caller + did not become loud. +- `tests/management-provider-validation.test.ts`: retain zero refresh for standalone + default/mode branches and exactly one paired gather/commit for ordinary edits; + assert the response disposition. +- `tests/combo-management-api.test.ts`: DELETE removes the final combo row through + real gather/commit; a disk failure still returns 200 with failed disposition. +- `tests/codex-v2-gate.test.ts`: scalar/feature writes remain applied when commit is + busy or stale; route remains 200. +- `tests/codex-sync-api.test.ts`: provider network/auth and ordinary disk failures + still invoke injection; desired OFF, gather busy, lock busy, and stale candidate + do not. Assert the exact `CodexSyncResult.catalogRefresh` in every branch. +- `tests/codex-models-cache-invalidate.test.ts`: app-server restart hint remains + keyed to `receipt.catalog || receipt.cache`, including partial commit. +- Fixture-only files listed in IN compile with the paired seam and never touch the + real home. + +## Verification + +Static gates: + +1. `bun test tests/codex-refresh.test.ts tests/codex-sync-api.test.ts tests/codex-models-cache-invalidate.test.ts` +2. `bun test tests/model-visibility-management-api.test.ts tests/management-provider-validation.test.ts tests/combo-management-api.test.ts tests/combos.test.ts tests/codex-v2-gate.test.ts` +3. `bun run typecheck` +4. `bun run test` +5. `bun run privacy:scan` +6. `bun --cwd docs-site run build` + +The live proxy on 10100 is not used, restarted, synced, restored, ensured, or +stopped. Runtime proof uses isolated temporary homes and a separate process/port: + +1. Create temporary `OPENCODEX_HOME` and `CODEX_HOME`, seed a known base catalog, + and run a test harness that calls `gatherCodexCatalogCandidate` only. Before/after + recursive file manifests must be byte-identical. This proves gather has no native + writes, not merely that mocks saw none. +2. In the same isolated harness, mutate persisted catalog-affecting config after + gather and call commit. Observe `stale_candidate`, all-false receipt, and + byte-identical catalog/cache/backups. Then regather and commit; parse the catalog + and expired cache wrapper and assert the wrapper models equal candidate catalog + models. +3. Start an isolated proxy on a non-10100 ephemeral port with the same temporary + homes. Send one management visibility mutation while the injected commit returns + `stale_candidate`; observe HTTP 200, persisted mutation, and skipped disposition. + Send explicit `/api/sync` with injected `lock_busy`; observe 503 plus + `Retry-After: 1` and no injection write. +4. Record the before/after manifest and JSON responses in the WP9 completion section + before moving this unit to `_fin/`. A green suite without the fired stale branch + does not satisfy C2. + +## Accept criteria + +- C1 (`000_plan.md:74-75`) — `gatherCodexCatalogCandidate` performs discovery, + loading, assembly, serialization, cache-wrapper construction, and backup + preparation with a byte-identical isolated-home manifest; commit is synchronous + and restricted to the fixed prepared write set. Every failure/skip is represented + by `CodexCatalogRefreshOutcome`, and all 16 management callers report a public + disposition while preserving their current primary success semantics. +- C2 (`000_plan.md:76-77`) — config digest mismatch, base-catalog digest/absence + mismatch, target-home/path mismatch, and candidate reuse all return + `stale_candidate` before any write. Tests activate both config and base-catalog + changes and prove the newer bytes survive. WP11 later places this same synchronous + compare-and-commit operation under the shared lock to close the remaining + cross-process check/write window; WP9 neither assumes nor fabricates that lock. diff --git a/devlog/_plan/260804_codex_write_substrate/020_history_isolation.md b/devlog/_plan/260804_codex_write_substrate/020_history_isolation.md new file mode 100644 index 000000000..dec9ef236 --- /dev/null +++ b/devlog/_plan/260804_codex_write_substrate/020_history_isolation.md @@ -0,0 +1,871 @@ +# WP10 — history isolation: one client turning off cannot freeze every client + +Research: `002_history_off_the_loop.md`. Read it first; this doc is the diff. + +Today, a server-side native restore enters `syncCodexHistoryProvider("openai")` +on the listener thread before `/api/stop` schedules drain, so a Codex SQLite +writer lock can hold the proxy for roughly 10.5 seconds and a successful +row/rollout traversal has no finite work bound at all +(`src/server/management-api.ts:167-194`, `src/codex/inject.ts:759-794`, +`src/codex/history-provider.ts:526-699`). That is the incident: turning one +client off can stop every other client, which is the exact opposite of the +integration switch's purpose +(`../260803_codex_desktop_toggle/008_audit_synthesis_wp4_r2.md:25-30`). This +phase adds an owned Bun Worker boundary for every history operation executed by +the server process, a fail-fast automatic SQLite mode, and a durable unresolved +history fact. Explicit CLI-process recovery keeps the existing synchronous +budget because blocking its own terminal does not starve the proxy. This phase +does not add the WP11 native-write lock and does not depend on it. + +## IN / OUT + +IN: + +- `src/codex/history-provider.ts` (MODIFY) — make busy timeout and retry mode + explicit per invocation, and preserve a classified recoverable failure. +- `src/codex/history-convergence.ts` (NEW) — own + `getConfigDir()/integrations/codex.json`, its schema, fail-closed updates, and + user-facing history status. +- `src/codex/history-worker.ts` (NEW) — Worker entry point; set the captured + environment before dynamically importing the history provider. +- `src/codex/history-job.ts` (NEW) — per-state-DB single flight, Worker IPC, + watchdog, close tracking, and durable state transitions. +- `src/codex/inject.ts` (MODIFY) — await the injected history executor and split + synchronous CLI restore from asynchronous server restore. +- `src/codex/sync.ts` (MODIFY) — carry an explicit `inline | worker` execution + choice to injection; default remains CLI-safe `inline`. +- `src/codex/history-migration-guardian.ts` (MODIFY) — schedule the Worker job; + never call the synchronous probe or mutation on a daemon tick. +- `src/cli/index.ts` (MODIFY) — startup selects `worker`; explicit `sync`, + `restore`, `eject`, `ensure`, and service-command processes retain `inline`. +- `src/server/management/context.ts` (MODIFY) — add the existing-style sync seam + needed to drive a real management request deterministically in the liveness + test (`src/server/management/context.ts:9-50`). +- `src/server/management/config-routes.ts` (MODIFY) — `/api/sync` selects Worker + execution and `GET /api/codex/history` exposes the durable fact. +- `src/server/management-api.ts` (MODIFY) — `/api/stop` awaits the server-safe + restore and returns pending/blocked history honestly before scheduling drain. +- `src/server/lifecycle.ts` (MODIFY) — cancel, join, and persist cancellation for + a history Worker before listener teardown, beside the storage Worker joins + (`src/server/lifecycle.ts:407-445`). +- `src/cli/doctor.ts` (MODIFY) — combine the live read-only probe with the durable + reason, attempt time, and next retry (`src/cli/doctor.ts:891-902`). +- `tests/codex-history-provider.test.ts` (MODIFY), + `tests/history-migration-guardian.test.ts` (MODIFY), + `tests/codex-sync-api.test.ts` (MODIFY), and + `tests/shutdown-drain.test.ts` (MODIFY) — pin changed contracts in their + existing owners. +- `tests/codex-history-worker.test.ts` (NEW), + `tests/codex-history-convergence.test.ts` (NEW), + `tests/codex-history-worker-responsive.test.ts` (NEW), and + `tests/codex-history-process-routing.test.ts` (NEW) — isolate Worker parity, + durable retry truth, measured server liveness, and process routing. + +OUT: + +- `gui/**` — this substrate exposes a truthful management status; the switch UI + belongs to the later Codex-toggle unit (`000_plan.md:20-22`, + `000_plan.md:62-70`). +- `docs-site/**` — no user-facing switch or configuration key ships in WP10. + The later toggle phase documents the final control surface. +- `src/codex/history-provider.ts` traversal/chunking — batching does not create a + finite bound for row count, rollout bytes, file count, or fsync latency; the + Worker boundary is the availability fix + (`src/codex/history-provider.ts:581-699`, + `002_history_off_the_loop.md:474-486`). +- `src/storage/worker-lifecycle.ts` — history has a different resource key and + job state. Sharing the storage reservation would let a cleanup spawn terminate + or serialize behind unrelated history work (`src/storage/worker-lifecycle.ts:40-50`, + `src/storage/worker-lifecycle.ts:123-143`). WP10 copies no storage mutation + authority; it reuses its close/join discipline in a dedicated controller. +- `src/codex/lock*`, lock files, lock directories, and WP11 protocol — no native + write lock exists yet. The only lock used here is the already-shipped, + zero-wait `withConfigMutationLockSync` around the small + `integrations/codex.json` read-modify-write, not around Codex files, SQLite, or + Worker execution (`src/config.ts:1767-1808`). If that state write cannot acquire + immediately, the history mutation is not dispatched. +- History mutation authority — the Worker executes only after its caller's + current authority checks. WP12 will strengthen that admission; moving code to + another thread is not permission to write (`002_history_off_the_loop.md:272-276`). +- Subprocess isolation — Bun 1.3.14 is pinned in CI + (`.github/workflows/ci.yml:220-222`) and the repository already runs synchronous + SQLite/filesystem work in TypeScript Workers (`src/storage/restore-job.ts:156-234`). + A subprocess is fallback work only if Worker teardown proves a history-specific + Bun defect. + +## Worker boundary + +### Why this is a Worker, and what Bun actually guarantees + +Bun Workers run TypeScript/ES modules without a compile step, communicate through +structured-clone `postMessage`, report module-resolution failures through `error`, +and emit `close` when marked terminated. Bun's own documentation also warns that +Worker termination remains experimental and that the thread can take time to +fully exit ([Bun Workers](https://bun.sh/docs/runtime/workers)). The repository has +already converted that warning into a stronger local rule: attach `close` at spawn, +do not treat `terminate()` as a join, and wait an OS-settle window on Windows and +macOS (`src/storage/worker-lifecycle.ts:1-17`, +`src/storage/worker-lifecycle.ts:150-209`). WP10 follows that local rule. + +The Worker runs the whole history unit, not merely the contended statement: + +1. optional read-only no-op probe; +2. SQLite open, queries, transactions, and close; +3. backup-manifest read/write; +4. every rollout read, line-one patch, append, and fsync; +5. the final read-only pending probe for an `openai` restore. + +Moving only `Database` calls is insufficient because full JSONL reads, per-file +patches/appends, and fsync are also synchronous and unbounded +(`src/codex/history-provider.ts:67-79`, +`src/codex/history-provider.ts:102-157`, +`src/codex/history-provider.ts:258-274`, +`src/codex/history-provider.ts:581-699`). + +### Serializable request and response + +`src/codex/history-worker.ts` accepts one plain-data message: + +```ts +export interface HistoryWorkerRequest { + type: "run"; + requestId: string; + targetProvider: "openai" | "opencodex"; + stateDbPath: string; + backupPath: string; + busyTimeoutMs: number; + attempts: number; + delayMs: number; + skipWhenProvablyNoop: boolean; + env: { CODEX_HOME?: string; OPENCODEX_HOME?: string }; +} + +export type HistoryWorkerResponse = + | { + type: "done"; + requestId: string; + result: CodexHistorySyncResult; + postProbe: PendingHistoryCount | null; + } + | { + type: "error"; + requestId: string; + reason: "permission_denied" | "state_unreadable" | "worker_error"; + }; +``` + +Every crossing value is a string, finite number, null, or plain object containing +those values. No `Database`, `Error`, callback, config object, file handle, or +class instance crosses structured clone. The parent resolves `stateDbPath` and +`backupPath` before spawn. The Worker applies the captured homes and only then +dynamically imports `history-provider.ts`; this avoids the current module-level +`CODEX_HOME` binding selecting a parent test's stale home +(`src/codex/history-provider.ts:16-22`, `src/codex/paths.ts:6-29`). The repository's +storage Worker records the same environment caveat +(`src/storage/restore-worker.ts:16-40`). + +The parent accepts a message only when `requestId` matches and the payload passes a +shape guard. `done` is not automatically `converged`: for target `openai`, +`postProbe` must be non-failed with both `pendingRows === 0` and +`backupEntries === 0` (`src/codex/history-provider.ts:734-775`). A provider result +with `failed: true`, a malformed message, or a failed/nonnull post-probe is durable +unresolved state. + +### Failure, timeout, and death + +**INFERRED design decision:** `src/codex/history-job.ts` owns one active +operation per normalized state-DB id. +Same-target callers join the same Promise; an opposite-target caller gets +`history_operation_busy` and does not overwrite the active attempt. This is an +in-process single flight, not the WP11 cross-process native-write lock. + +The parent resolves outcomes in this order: + +- valid `done` message → classify from mutation result and post-probe; +- valid `error` message → `blocked` with the Worker-provided safe reason; +- `worker.onerror` → `unknown / worker_error`; +- `close` before a valid terminal message → `unknown / worker_died`; +- 10-minute watchdog → terminate and join, then `pending / worker_timeout`; +- shutdown cancellation → terminate and join, then + `pending / shutdown_cancelled`. + +A `done` result with `failureReason: "sqlite_busy"` becomes retryable +`pending / sqlite_busy`; permission becomes `blocked / permission_denied`; a +failed or structurally unknown post-probe becomes `unknown / state_unreadable`. +Only the clean zero/zero post-probe reaches `converged` for target `openai`. + +**INFERRED containment decision:** ten minutes matches the existing storage restore watchdog +(`src/storage/restore-job.ts:40-46`, `src/storage/restore-job.ts:190-215`). It is a +containment deadline, not a claim that history finishes in ten minutes. Because +the work has no finite bound, timeout can interrupt a legitimate large history; +the pre-dispatch durable `pending` fact therefore remains authoritative, the next +startup retries, and the explicit CLI command remains the unbounded operator path. +The Worker closes itself in `finally`; the parent still calls its join helper on +every terminal path, because Bun `close` does not prove immediate OS thread reclaim +(`src/storage/restore-worker.ts:43-55`, +`src/storage/worker-lifecycle.ts:176-199`). + +## Fail-fast automatic mode + +The current writable connection reads one mutable global +`historyDbBusyTimeoutMs = 5000`, and `withHistoryRetry` defaults to two attempts +with `Bun.sleepSync(500)` between them +(`src/codex/history-provider.ts:25-49`, +`src/codex/history-provider.ts:526-548`). WP10 makes the policy explicit: + +| Caller | Execution | SQLite busy timeout | Attempts / delay | Reason | +|---|---|---:|---:|---| +| Server startup, `/api/sync`, `/api/stop`, guardian, future toggle | Worker | **100 ms** | **1 / 0 ms** | Automatic convergence must release the history slot quickly when Codex owns SQLite. The read-only probe already uses 100 ms (`src/codex/history-provider.ts:749-774`). | +| Explicit CLI `restore`, `eject`, `recover-history`, `sync`, `restore back`, `ensure` parent | CLI process, inline | **5,000 ms** | **2 / 500 ms** | The invoking terminal may wait for a transient Codex lock; this preserves today's operator behavior (`src/codex/history-provider.ts:25-49`, `src/codex/history-provider.ts:526-548`). | + +Automatic mode does not call `sleepSync`; its scheduler delay is the retry. The +100 ms budget bounds only lock waiting. It does not and cannot bound a successful +row/file walk; that is why fail-fast without Worker isolation failed the research +gate (`002_history_off_the_loop.md:183-205`). + +## Unresolved history is a durable fact + +### Location and exact shape + +**INFERRED schema decision:** the record is +`join(getConfigDir(), "integrations", "codex.json")`, beneath +`OPENCODEX_HOME`, never `CODEX_HOME`. This reuses the repository's owned +integration directory and atomic-write convention +(`src/integrations/ownership.ts:60-71`, +`src/integrations/ownership.ts:94-106`). It is the one future Codex integration +record, not a second history-only file. WP10 writes `version` and `history`; WP12 +may add desired state and the artifact ledger without moving history. + +```json +{ + "version": 1, + "history": { + "<16-hex normalized state DB id>": { + "stateDbPath": "/absolute/CODEX_HOME/state_5.sqlite", + "backupPath": "/absolute/OPENCODEX_HOME/codex-history-backup-.json", + "targetProvider": "openai", + "state": "pending", + "reason": "sqlite_busy", + "attemptId": "uuid", + "attemptCount": 3, + "lastAttemptAt": "2026-08-04T00:00:00.000Z", + "pendingRows": null, + "backupEntries": 4, + "automaticRetry": true, + "nextRetryAt": "2026-08-04T00:01:00.000Z" + } + } +} +``` + +`state` is `pending | running | blocked | converged | unknown`. `reason` is null +only for `converged`; otherwise it is one of `sqlite_busy`, `permission_denied`, +`state_unreadable`, `state_write_busy`, `history_operation_busy`, `worker_error`, +`worker_died`, `worker_timeout`, or `shutdown_cancelled`. Counts are nullable: +failed probes mean unknown, never numeric zero. The key uses the same normalized +state-DB hash already used for backup naming +(`src/codex/history-provider.ts:16-22`). + +Before spawn, the parent writes `pending` with a new `attemptId`. After creating an +idle Worker but before posting `run`, it writes `running`. Both updates use the +already-shipped zero-wait config mutation transaction only around read/merge/atomic +write (`src/config.ts:1767-1808`). If either write is busy or fails, the Worker is +not messaged and no history mutation starts. A terminal update applies only when +the stored `attemptId` still matches; an older in-process completion cannot turn a +newer attempt green. If the final state write fails after mutation, the record +stays `running` or `pending`, which is a retryable false negative rather than a +false success. + +`converged` for target `openai` is legal only after the clean post-probe proves +zero pending rows and zero backup entries. Manifest absence alone is insufficient: +the no-backup ejection path can still have work, and a failed probe currently +returns zero-looking counts with `failed: true` +(`src/codex/history-provider.ts:656-665`, +`src/codex/history-provider.ts:749-775`). + +### Retry ownership and user visibility + +- A running server retries `pending`, `unknown`, and retryable `blocked` entries + every 60 seconds, at most 60 ticks per process lifetime. The durable record keeps + `automaticRetry: true`; after the in-process budget, `nextRetryAt: null` means + “next proxy startup,” not “abandoned.” This preserves the current finite guardian + cadence while replacing its event-loop mutation + (`src/codex/history-migration-guardian.ts:34-40`, + `src/codex/history-migration-guardian.ts:54-92`). +- Every proxy startup treats persisted `pending`, `running`, `blocked`, or + `unknown` as retryable. A stale `running` state is not proof a Worker survived + its process. +- Explicit CLI recovery runs inline with the full budget, then writes the same + record. It never reports success while the durable entry remains unresolved. + +The user sees the fact in three places. `GET /api/codex/history` returns the entry; +`/api/sync` and `/api/stop` include the same status in their response; and +`ocx doctor` prints, for example: + +```text +-- Codex resume history unresolved: sqlite_busy + 4 routed thread(s) may remain hidden in native Codex + automatic retry: 2026-08-04T00:01:00.000Z; run `ocx restore` after closing Codex to retry now +``` + +The wording says “may remain hidden” when counts are null. OFF/config restoration +and history convergence are separate facts; no `success: true` envelope may erase +the warning. That fixes the current shape where `restoreNativeCodex` returns +`success: cfg.success` while history failure exists only in message text +(`src/codex/inject.ts:783-794`). + +## Diff + +Line anchors below are against current HEAD `7e67a8d06311de2471b0a25e41cf85f97007cc69`. + +### `src/codex/history-provider.ts` + +Make writable busy timeout invocation-local while preserving the test override as +the explicit-mode default: + +```diff + let historyDbBusyTimeoutMs = 5000; ++export const AUTOMATIC_HISTORY_DB_BUSY_TIMEOUT_MS = 100; +@@ +-function openStateDb(stateDbPath: string): Database { ++function openStateDb(stateDbPath: string, busyTimeoutMs = historyDbBusyTimeoutMs): Database { + const db = new Database(stateDbPath); + try { +- db.exec(`PRAGMA busy_timeout = ${historyDbBusyTimeoutMs}`); ++ db.exec(`PRAGMA busy_timeout = ${Math.max(0, Math.floor(busyTimeoutMs))}`); +``` + +Extend the existing result and options; do not replace `failed`, because current +callers and tests use it (`src/codex/history-provider.ts:160-168`, +`src/codex/history-provider.ts:565-579`): + +```diff + export interface CodexHistorySyncResult { + rows: number; + files: number; + ejectedRows?: number; + failed?: true; ++ failureReason?: "sqlite_busy" | "permission_denied" | "state_unreadable"; + } ++ ++export interface HistoryExecutionOptions { ++ skipWhenProvablyNoop?: boolean; ++ busyTimeoutMs?: number; ++ attempts?: number; ++ delayMs?: number; ++ sleepFn?: (ms: number) => void; ++} +@@ + export function syncCodexHistoryProvider( + provider: CodexHistoryProvider, + stateDbPath = STATE_DB_PATH, + backupPath = HISTORY_BACKUP_PATH, +- opts: { skipWhenProvablyNoop?: boolean } = {}, ++ opts: HistoryExecutionOptions = {}, + ): CodexHistorySyncResult { +@@ +- return withHistoryRetry(() => syncCodexHistoryProviderUnsafe(provider, stateDbPath, backupPath)) +- ?? { rows: 0, files: 0, failed: true }; ++ let failureReason: CodexHistorySyncResult["failureReason"]; ++ const result = withHistoryRetry( ++ () => syncCodexHistoryProviderUnsafe(provider, stateDbPath, backupPath, opts.busyTimeoutMs), ++ { ++ attempts: opts.attempts, ++ delayMs: opts.delayMs, ++ sleepFn: opts.sleepFn, ++ onRecoverableError: error => { failureReason = classifyHistoryFailure(error); }, ++ }, ++ ); ++ return result ?? { rows: 0, files: 0, failed: true, failureReason: failureReason ?? "state_unreadable" }; + } +@@ +-function syncCodexHistoryProviderUnsafe(provider: CodexHistoryProvider, stateDbPath: string, backupPath: string): CodexHistorySyncResult { ++function syncCodexHistoryProviderUnsafe(provider: CodexHistoryProvider, stateDbPath: string, backupPath: string, busyTimeoutMs?: number): CodexHistorySyncResult { +@@ +- const db = openStateDb(stateDbPath); ++ const db = openStateDb(stateDbPath, busyTimeoutMs); +``` + +Apply the same `busyTimeoutMs` parameter to the restore-side `openStateDb` at +`src/codex/history-provider.ts:660` and to `migrateHistoryToOpenai` at +`src/codex/history-provider.ts:719-731`. Add +`onRecoverableError?: (error: unknown) => void` to `withHistoryRetry`'s `io` +parameter and call it immediately after the recoverability check at +`src/codex/history-provider.ts:544`, before the attempts check. +`classifyHistoryFailure` maps SQLite busy/locked to +`sqlite_busy`, `EPERM`/`EACCES`/permission text to `permission_denied`, and the +remaining recoverable class to `state_unreadable`; hard errors still throw. Existing +default behavior remains 5,000 ms, two attempts, and 500 ms delay +(`src/codex/history-provider.ts:511-548`). + +### `src/codex/history-worker.ts` (NEW) + +Implement the message contract above. The critical order is: + +```ts +self.onmessage = async (event: MessageEvent) => { + if (!isHistoryWorkerRequest(event.data)) return; + const request = event.data; + try { + if (request.env.CODEX_HOME) process.env.CODEX_HOME = request.env.CODEX_HOME; + if (request.env.OPENCODEX_HOME) process.env.OPENCODEX_HOME = request.env.OPENCODEX_HOME; + const { countPendingOpencodexHistory, syncCodexHistoryProvider } = await import("./history-provider"); + const result = syncCodexHistoryProvider(request.targetProvider, request.stateDbPath, request.backupPath, { + busyTimeoutMs: request.busyTimeoutMs, + attempts: request.attempts, + delayMs: request.delayMs, + skipWhenProvablyNoop: request.skipWhenProvablyNoop, + }); + const postProbe = request.targetProvider === "openai" + ? countPendingOpencodexHistory(request.stateDbPath, request.backupPath) + : null; + self.postMessage({ type: "done", requestId: request.requestId, result, postProbe }); + } catch (error) { + self.postMessage({ + type: "error", + requestId: request.requestId, + reason: classifyWorkerThrownError(error), + }); + } finally { + try { (self as unknown as { close?: () => void }).close?.(); } catch {} + } +}; +``` + +`classifyWorkerThrownError` emits only the reason enum, not raw error strings or +paths. The request guard rejects non-finite/negative numeric policy fields and +non-absolute paths. + +### `src/codex/history-convergence.ts` and `src/codex/history-job.ts` (NEW) + +Export the normalized identity/path resolver from `history-provider.ts` so the +job, state record, and backup file cannot implement three subtly different hashes: + +```diff +-function historyBackupPathFor(stateDbPath: string): string { ++export function historyStateDbId(stateDbPath: string): string { + const normalized = process.platform === "win32" ? resolve(stateDbPath).toLowerCase() : resolve(stateDbPath); +- const id = createHash("sha256").update(normalized).digest("hex").slice(0, 16); +- return join(getConfigDir(), `codex-history-backup-${id}.json`); ++ return createHash("sha256").update(normalized).digest("hex").slice(0, 16); + } ++function historyBackupPathFor(stateDbPath: string): string { ++ return join(getConfigDir(), `codex-history-backup-${historyStateDbId(stateDbPath)}.json`); ++} ++export function resolveCodexHistoryPaths(stateDbPath = STATE_DB_PATH): { stateDbPath: string; backupPath: string } { ++ return { stateDbPath: resolve(stateDbPath), backupPath: historyBackupPathFor(stateDbPath) }; ++} +``` + +Also export `CodexHistoryProvider`; the Worker and job import the owner type instead +of restating a parallel union. + +`history-convergence.ts` owns the schema and only these operations: + +```ts +historyStateDbId(stateDbPath: string): string; +readCodexHistoryConvergence(stateDbPath?: string): HistoryConvergenceEntry | null; +beginCodexHistoryAttempt(input: AttemptInput): HistoryConvergenceEntry; +markCodexHistoryAttemptRunning(attemptId: string): HistoryConvergenceEntry; +finishCodexHistoryAttempt(attemptId: string, outcome: HistoryJobOutcome): HistoryConvergenceEntry; +``` + +All three writes execute a synchronous, no-await callback inside +`withConfigMutationLockSync`; malformed/unknown-version `codex.json` is +`state_unreadable` and is preserved, not replaced. `begin` fails before Worker +dispatch if the record cannot be durably written. `finish` compares `attemptId` +inside the transaction and leaves a newer entry untouched. + +`history-job.ts` exports: + +```ts +export type HistoryExecution = "automatic" | "explicit"; +export function runCodexHistoryJob(input: { + targetProvider: "openai" | "opencodex"; + execution: HistoryExecution; + stateDbPath?: string; + backupPath?: string; + skipWhenProvablyNoop?: boolean; +}): Promise; +export function runCodexHistoryInline(input: { + targetProvider: "openai" | "opencodex"; + stateDbPath?: string; + backupPath?: string; + skipWhenProvablyNoop?: boolean; +}): HistoryJobOutcome; +export function runLegacyCodexHistoryRecoveryInline(input?: { + stateDbPath?: string; +}): HistoryJobOutcome; +export function abortCodexHistoryJobAsync(): Promise; +export function setCodexHistoryJobTestHooks(hooks: { + automaticBusyTimeoutMs?: number; + workerTimeoutMs?: number; +} | null): void; +``` + +`automatic` always spawns `new Worker(new URL("./history-worker.ts", +import.meta.url).href)`, sends 100/1/0, and uses the durable transitions above. +`runCodexHistoryInline` invokes `syncCodexHistoryProvider` in the caller process +with defaults, then writes the same terminal state; the async job delegates to it +for `explicit`. The test hooks change timing only; there is no +`runInProcess` hook in the liveness test because that would make C3 vacuous. + +### `src/codex/inject.ts` + +Extend `InjectCodexOptions` at `src/codex/inject.ts:66-73` and replace the direct +call at `src/codex/inject.ts:601-603`: + +```diff + export interface InjectCodexOptions { + catalogPath?: string | null; ++ historyExecution?: "automatic" | "explicit"; + } +@@ +- const history = config?.syncResumeHistory !== false +- ? (legacyMode ? syncCodexHistoryProvider("opencodex") : migrateHistoryToOpenai()) ++ const history = config?.syncResumeHistory !== false ++ ? await runCodexHistoryJob({ ++ targetProvider: legacyMode ? "opencodex" : "openai", ++ execution: options.historyExecution ?? "explicit", ++ }) + : { rows: 0, files: 0 }; +``` + +Factor `src/codex/inject.ts:765-783` into `prepareNativeCodexRestore()` +(external-provider guard, journal/config/catalog work, and +`skipWhenProvablyNoop`) and `src/codex/inject.ts:784-794` into +`finishNativeCodexRestore(prepared, history)`. The public synchronous CLI contract +remains, while the server gets an async sibling: + +```diff + export function restoreNativeCodex(): { success: boolean; message: string } { +- const activeProvider = currentExternalCodexModelProvider(); +- // ... current config/catalog setup ... +- const history = syncCodexHistoryProvider("openai", undefined, undefined, { skipWhenProvablyNoop }); +- // ... current message formatting ... ++ const prepared = prepareNativeCodexRestore(); ++ if (prepared.done) return prepared.result; ++ const history = runCodexHistoryInline({ targetProvider: "openai", skipWhenProvablyNoop: prepared.skipWhenProvablyNoop }); ++ return finishNativeCodexRestore(prepared, history); + } ++ ++/** Exit-hook fallback: restore bounded config/catalog state and leave history unresolved. */ ++export function restoreNativeCodexWithoutHistory(): CodexRestoreResult { ++ const prepared = prepareNativeCodexRestore(); ++ if (prepared.done) return prepared.result; ++ return finishNativeCodexRestore(prepared, preserveConvergedOrLeaveCodexHistoryPending("shutdown_cancelled")); ++} ++ ++export async function restoreNativeCodexInServer(): Promise { ++ const prepared = prepareNativeCodexRestore(); ++ if (prepared.done) return prepared.result; ++ const history = await runCodexHistoryJob({ ++ targetProvider: "openai", ++ execution: "automatic", ++ skipWhenProvablyNoop: prepared.skipWhenProvablyNoop, ++ }); ++ return finishNativeCodexRestore(prepared, history); ++} +``` + +`CodexRestoreResult` adds `history: HistoryConvergenceEntry | null`. Its `success` +continues to describe config/catalog restoration for compatibility, but every +caller must render `history.state !== "converged"` separately; the formatter keeps +the hidden-thread warning from `src/codex/inject.ts:787-793`. + +### Process-aware callers + +`syncModelsToCodex` carries an explicit fifth options object rather than inferring +from port or whether a proxy happens to be live; those are not process identity: + +```diff + export async function syncModelsToCodex( + port?: number, + config: OcxConfig = loadConfig(), + log: Pick | null = console, + deps: CodexSyncDeps = defaultDeps, ++ options: { historyExecution?: "automatic" | "explicit" } = {}, + ): Promise { +@@ +- const result = await deps.injectCodexConfig(p, config, {}); ++ const result = await deps.injectCodexConfig(p, config, { ++ ...(options.historyExecution ? { historyExecution: options.historyExecution } : {}), ++ }); +@@ +- const result = await deps.injectCodexConfig(p, config, { catalogPath: catalogPathForInjection }); ++ const result = await deps.injectCodexConfig(p, config, { ++ catalogPath: catalogPathForInjection, ++ ...(options.historyExecution ? { historyExecution: options.historyExecution } : {}), ++ }); +``` + +Server callers opt in; CLI callers retain the default: + +```diff + // src/cli/index.ts:318-322 — this is the server process after listen +- await syncModelsToCodex(port).catch(() => {}); ++ await syncModelsToCodex(port, config, console, undefined, { historyExecution: "automatic" }).catch(() => {}); + + // src/server/management/config-routes.ts:261-268 +- const result = await syncModelsToCodex(undefined, config, null); ++ const sync = ctx.deps.syncModelsToCodex ?? syncModelsToCodex; ++ const result = await sync(undefined, config, null, undefined, { historyExecution: "automatic" }); + + // src/server/management-api.ts:167-194 +- const { restoreNativeCodex } = await import("../codex/inject"); ++ const { restoreNativeCodexInServer } = await import("../codex/inject"); +@@ +- const restore = restoreNativeCodex(); ++ const restore = await restoreNativeCodexInServer(); +@@ +- return jsonResponse(restore.success +- ? { success: true, message: `Proxy stopping, native Codex restored.${grokNote}` } ++ return jsonResponse(restore.success ++ ? { success: true, history: restore.history, message: `Proxy stopping, native Codex restored.${historyNote}${grokNote}` } + : { success: false, message: `Proxy stopping, but native Codex restore failed: ${restore.message}. Run \`ocx restore\`.${grokNote}` }); +``` + +`historyNote` explicitly says routed threads remain hidden when state is not +`converged`. The 200 ms drain timer is scheduled only after the awaited automatic +attempt returns; under lock contention that is one 100 ms Worker attempt. Under a +large uncontended traversal the request may remain pending, but `/healthz` and +data-plane traffic continue; if the request watchdog/shutdown cancels it, durable +state remains pending. + +The guardian's current synchronous `countFn` and `migrateFn` dependencies at +`src/codex/history-migration-guardian.ts:24-31` become async +`readStateFn`/`runJobFn`. Its tick awaits the single-flight job and schedules from +the durable terminal state; it never calls `countPendingOpencodexHistory` or +`migrateHistoryToOpenai` on the server thread (`src/codex/history-migration-guardian.ts:59-83`). + +`drainAndShutdown` adds `abortCodexHistoryJobAsync()` to the `Promise.allSettled` +join group at `src/server/lifecycle.ts:415-418` and logs it under +`[codex-history]`. The abort +function writes `shutdown_cancelled` before resolving. The synchronous +`process.on("exit")` fallback in `src/cli/index.ts:305-310` must never start a Worker +or run history inline; graceful signal paths await the server-safe cleanup before +calling `process.exit`, while the exit fallback can only leave/rewrite unresolved +state. + +The signal/exit caller split is explicit; the synchronous exit hook restores only +bounded config/catalog state, while the graceful async path runs history in a +Worker after drain: + +```diff + // src/cli/index.ts:242-266 +- const restored = restoreNativeCodex(); ++ const restored = restoreNativeCodexWithoutHistory(); +@@ + // src/cli/index.ts:295-301 + try { + await drainAndShutdown(server, config.shutdownTimeoutMs ?? 5000); + } finally { ++ if (!isRecyclingForExit() && !process.env.OCX_SERVICE && !currentExternalCodexModelProvider()) { ++ const historyRestore = await restoreNativeCodexInServer(); ++ if (!historyRestore.success) cleanupSucceeded = false; ++ } + const restored = syncCleanup(); + process.exit(restored ? 0 : 1); + } +``` + +`/api/stop` already awaits `restoreNativeCodexInServer` before its drain timer; +the later exit hook sees idempotently restored config/catalog and does no history +work. A crash that reaches only the synchronous exit hook leaves history pending +for startup instead of freezing exit or pretending convergence. + +The fallback helper preserves an already durable `converged` entry. It writes +`shutdown_cancelled` only when history is absent, running, or already unresolved; +an idempotent exit hook must not turn the `/api/stop` Worker's proven zero/zero +result back into a false negative. + +All commands that execute after the proxy is stopped or in a separate CLI process +remain unchanged at their call sites: `handleStop`'s second restore +(`src/cli/index.ts:527-534`), explicit restore/eject +(`src/cli/index.ts:745-776`), service stop/uninstall +(`src/service.ts:2564-2595`, `src/service.ts:2610-2632`). Their synchronous +self-block is intentional. Legacy recovery keeps its narrower operation but routes +through the state-writing inline wrapper: + +```diff + // src/cli/index.ts:711-724 +- const r = restoreLegacyOpenaiHistory(); ++ const r = runLegacyCodexHistoryRecoveryInline(); +``` + +That wrapper calls the existing `restoreLegacyOpenaiHistory`, performs the same +post-probe, and updates `integrations/codex.json`; it does not broaden legacy +recovery into manifest restore. + +### Durable read surface + +Add `syncModelsToCodex?: typeof syncModelsToCodex` to `ManagementApiDeps`, use it +for `/api/sync`, and add this authenticated route beside it: + +```diff + if (url.pathname === "/api/sync" && req.method === "POST") { + // worker-aware sync above + } ++ ++ if (url.pathname === "/api/codex/history" && req.method === "GET") { ++ const { readCurrentCodexHistoryConvergence } = await import("../../codex/history-convergence"); ++ return jsonResponse({ history: readCurrentCodexHistoryConvergence() }); ++ } +``` + +`ocx doctor` keeps its live read-only probe, because the durable fact can be stale, +but it treats a failed probe as unknown and prints durable reason/retry metadata +instead of the current generic locked-or-unreadable line +(`src/cli/doctor.ts:891-902`). + +## Test plan + +### C3 — real SQLite contention with measured `/healthz` + +`tests/codex-history-worker-responsive.test.ts` is a server-boundary test, not a +mocked busy-error unit test: + +1. Install isolated `CODEX_HOME` and `OPENCODEX_HOME` with + `installIsolatedCodexHome`; create a production-shaped `threads` table, one + interactive `opencodex` row, and a matching rollout using the fixture at + `tests/codex-history-provider.test.ts:27-89`. +2. Spawn an owned Bun child with `Bun.spawn([process.execPath, "-e", source])`. + The child opens that exact `state_5.sqlite`, executes + `PRAGMA busy_timeout=0; BEGIN IMMEDIATE; UPDATE threads SET has_user_event = + has_user_event`, writes `holder-ready`, and loops with `Bun.sleepSync(10)` until + `holder-release` exists. The ready/release handshake and `finally` cleanup match + `tests/config-mutation-lock.test.ts:48-92`. This is a separate-process SQLite + writer lock, not a stubbed `SQLITE_BUSY`. +3. Set only `setCodexHistoryJobTestHooks({ automaticBusyTimeoutMs: 1_200 })` so + contention lasts long enough to sample. Do not set an in-process execution hook. +4. Start `startServer(0)` with `managementApi.syncModelsToCodex` injected so the + real `/api/sync` request calls real `injectCodexConfig` and the real Worker while + catalog fetch is a deterministic local stub. Start the management POST but do + not await it. +5. In parallel, issue a real `/v1/responses` request to a local test upstream that + emits eight SSE chunks 50 ms apart; assert all eight arrive. This proves an + already-admitted data-plane client still progresses. +6. While the management request is still pending and the child still owns the + transaction, issue six `/healthz` requests 40 ms apart. Require every status to + be 200. Discard the first warmup latency and require every remaining sample to + be below `Math.floor(1_200 / 3) = 400 ms`. Also assert the management operation + duration is at least 1,100 ms, proving the health samples overlapped contention. + This copies the measured pattern at + `tests/storage-restore-job-responsive.test.ts:175-210`; checking health only + before and after the operation is not acceptance evidence. +7. Assert the management result and `GET /api/codex/history` both report unresolved + `sqlite_busy`, with null unknown counts where the probe failed and a next retry. +8. In `finally`, write `holder-release`, await child exit code 0, reset/join the + history Worker, drain the server, restore both homes, and delete fixtures. No + test may delete a Worker-owned home before the Worker join; the repository has + already observed Bun isolate failures from that ordering + (`tests/storage-restore-job-responsive.test.ts:53-80`, + `src/storage/worker-lifecycle.ts:1-17`). +9. Start a fresh server after lock release, trigger the persisted retry, and assert + both the API and live post-probe become `converged` with zero/zero counts. This + closes persistence and retry, not only responsiveness. + +### Focused cases + +- `tests/codex-history-provider.test.ts` — automatic options set one attempt and a + 100 ms writable timeout; no sleep callback fires; explicit defaults still make + two attempts with one 500 ms sleep; every recoverable class gets the right reason; + hard errors still throw (`tests/codex-history-provider.test.ts:293-369`). +- `tests/codex-history-worker.test.ts` — parity for forward retag, manifest restore, + no-manifest ejection, line-one patch, trailing append, manifest consumption, and + no-op. Assert only plain-data messages cross. Existing behavioral oracle: + `tests/codex-history-provider.test.ts:92-290`. +- `tests/codex-history-worker.test.ts` — malformed message ignored; dynamic import + sees captured homes; valid `error`, `onerror`, early `close`, watchdog timeout, + and shutdown cancellation each join and classify once. +- `tests/codex-history-convergence.test.ts` — pending is durable before dispatch; + state-write busy means no Worker spawn; final-write failure leaves pending/running; + stale `attemptId` cannot overwrite a newer attempt; corrupt/unknown-version + `codex.json` is preserved and blocks; failed probe counts are null; only clean + zero/zero post-probe permits `converged`. +- `tests/history-migration-guardian.test.ts` — no synchronous provider probe on a + tick, single-flight retries, finite 60-tick budget, startup re-arm of stale + running/pending/blocked/unknown, and no retry for converged. Preserve the scheduler + expectations currently covered at `tests/history-migration-guardian.test.ts:24-136`. +- `tests/codex-history-process-routing.test.ts` — startup, `/api/sync`, `/api/stop`, + guardian, and graceful server cleanup select automatic Worker execution; explicit + CLI restore/eject/recover/sync/ensure and service cleanup select inline full-budget + execution. Assert by injected executors, not source-string matching. +- `tests/codex-sync-api.test.ts` — execution option reaches both external-provider + and normal injection paths at `src/codex/sync.ts:49-70` and + `src/codex/sync.ts:83-124`; result carries history status. +- `tests/shutdown-drain.test.ts` — drain awaits history termination, records + `shutdown_cancelled`, and stops the listener even if join rejects. Existing + storage joins remain unchanged (`src/server/lifecycle.ts:407-445`). +- `tests/codex-history-convergence.test.ts` — doctor and management status say + routed threads remain hidden for pending/blocked/unknown, survive module reload, + and never collapse failed-probe zeroes into success. + +## Verification + +Static and focused gates: + +```bash +bun run typecheck +bun test tests/codex-history-provider.test.ts +bun test tests/codex-history-worker.test.ts +bun test tests/codex-history-convergence.test.ts +bun test tests/history-migration-guardian.test.ts +bun test tests/codex-sync-api.test.ts +bun test tests/codex-history-process-routing.test.ts +bun test tests/shutdown-drain.test.ts +bun test tests/codex-history-worker-responsive.test.ts +bun run privacy:scan +bun run test +``` + +Runtime measurement is the output of the responsiveness test, which must print or +attach this evidence on failure and success: + +```text +lock_ready_at= +history_request_started_at= +health_ms=[...five post-warmup samples...] +max_health_ms= threshold_ms=400 +stream_chunks=8 +history_elapsed_ms== 1100> +history_state=pending reason=sqlite_busy +child_exit=0 history_workers_live=0 +``` + +The acceptance command is the real test invocation, not a prose assertion: + +```bash +bun test tests/codex-history-worker-responsive.test.ts --timeout 30000 +``` + +Do not run `ocx start`, `ocx stop`, `ocx sync`, `ocx restore`, or `ocx ensure` as +verification against the live proxy on port 10100. The isolated test server binds +port 0 and the lock child touches only its temporary Codex home. + +## Accept criteria + +- **C3 — measured availability:** during a real cross-process + `BEGIN IMMEDIATE` lock on the exact history database, the automatic history + operation remains pending for at least 1,100 ms, all six `/healthz` requests are + 200, every post-warmup sample is below 400 ms, and an eight-chunk data-plane + stream completes. The operation executes in a Worker; no synchronous fallback + is enabled in this test. +- **C3 — unbounded work boundary:** SQLite queries, all row/manifest traversal, + rollout reads/writes, and fsync stay inside the Worker. No claim that the + operation is “fast” substitutes for this boundary. +- **C4 — no silent success:** before any automatic mutation, a durable + `pending`/`running` record exists. Busy, permissions, unreadable state, timeout, + Worker death, and shutdown cancellation remain non-converged with classified + reasons and nullable unknown counts. +- **C4 — proof before green:** `converged` for native restore requires a clean + post-probe with `pendingRows=0` and `backupEntries=0`; manifest absence, + `failed: true`, or a successful zero-row mutation is insufficient. +- **C4 — retry and visibility:** the running server retries on its bounded cadence, + every startup re-arms durable unresolved work, explicit CLI recovery updates the + same record, and management responses plus `ocx doctor` state that routed threads + may remain hidden until convergence. +- CLI-process commands retain today's 5,000 ms / two-attempt / 500 ms budget; + server-process callers use Worker + 100 ms / one attempt / no sleep. +- WP10 creates no native-write lock and no GUI switch. It is independently useful + and independently testable before WP11. diff --git a/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md b/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md new file mode 100644 index 000000000..3226b3966 --- /dev/null +++ b/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md @@ -0,0 +1,800 @@ +# WP11 — one bounded writer per canonical `CODEX_HOME` + +Research: `003_lock_protocol.md`. Read it first; this document is the implementation +diff for its Decision. + +The failure is lock splitting plus event-loop denial, not a missing mutex. Today the +default home can remain an unresolved `~/.codex` spelling while an explicit home is +realpathed (`src/codex/paths.ts:6-24`), and the nearest reusable SQLite lock opens a +stable file but repairs its mode after open (`src/codex/native-main-lock-file.ts:74-131`). +The rejected Codex-OFF design could therefore let two spellings of one home take two +locks, or wait synchronously around history work that can exceed 10.5 seconds +(`devlog/_plan/260803_codex_desktop_toggle/008_audit_synthesis_wp4_r2.md:17-30`). +WP9 and WP10 land first: WP9 supplies an already-gathered, fixed-size synchronous +catalog commit (`001_catalog_seam.md:137-159`), and WP10 moves unbounded history work +to an owned Worker (`002_history_off_the_loop.md:264-296,474-486`). This phase adds +only the async cross-process acquisition substrate around those bounded commit +sections. It does not add desired state or decide ownership. + +All current-code citations and diff context below were rechecked on 2026-08-04 at +`7e67a8d06311de2471b0a25e41cf85f97007cc69`. + +## IN / OUT + +IN: + +- `src/codex/codex-write-lock.ts` (**NEW**) — canonical-home resolution, namespace + validation, finite async retry, admission callbacks, and the synchronous locked + callback. +- `src/codex/native-main-lock-file.ts` (**MODIFY**) — preserve its existing stable + descriptor owner and add only a caller-supplied ACL time cap. WP11 reuses + `openStableLockFile`/`assertStableLockFile`; it does not copy their descriptor + lifetime or `(dev, ino)` substitution checks. +- `src/lib/windows-secret-acl.ts` (**MODIFY**) — allow a stricter caller deadline to + cap the existing required `icacls` sequence. Existing callers retain the current + 5-second default. +- `tests/codex-write-lock.test.ts` (**NEW**) — result taxonomy, canonicalization, + namespace refusal, sync-callback, reentrancy, deadline, and real-process coverage. +- `tests/helpers/codex-write-lock-child.ts` (**NEW**) — an owned Bun process that + acquires the real SQLite transaction and holds one finite synchronous section. +- `tests/windows-secret-acl.test.ts` (**MODIFY**) — prove the optional caller cap is + forwarded without weakening the required ACL failure behavior. + +OUT: + +- `src/config.ts` — `withConfigMutationLockSync` stays synchronous and fail-fast; + changing it would recreate the listener freeze its docstring prevents + (`src/config.ts:1767-1818`). +- WP9 catalog gather/commit implementation, WP10 history Worker implementation, + and all of WP12 ownership, desired-state, provenance, convergence, API, CLI, GUI, + and docs wiring. +- `src/codex/paths.ts` global behavior. WP11 canonicalizes for the lock without + changing every existing `CODEX_HOME` consumer at module import. +- PID files, heartbeat rows, leases, stale-file deletion, lock-database unlink, + FIFO tickets, and process-local queueing. +- `gui/**`, service lifecycle, proxy start/stop/sync/restore/ensure, release, deploy, + and the live proxy on port 10100. + +No-code/configuration reuse is insufficient: process-local flights do not coordinate +two processes, and the existing native-main databases live inside `CODEX_HOME` and +have different lifetime semantics. Reusing the stable-file owner is sufficient for +the dangerous descriptor/open race, so this phase extends that owner instead of +adding another raw `openSync` implementation. + +## API and ownership boundary + +### Public contract + +Add the following real TypeScript contract at the top of +`src/codex/codex-write-lock.ts`: + +```ts +export const CODEX_WRITE_LOCK_MAX_TIMEOUT_MS = 30_000; + +export type CodexWriteLockRefusalReason = + | "codex_home_missing" + | "codex_home_unsafe" + | "authority_not_proven" + | "namespace_unsafe" + | "lock_path_unsafe" + | "unsupported_filesystem" + | "reentrant" + | "lock_unavailable"; + +export type CodexWriteLockResult = + | { + status: "acquired"; + value: T; + waitedMs: number; + lockId: string; + } + | { + status: "busy"; + reason: "deadline" | "cancelled"; + retryable: true; + waitedMs: number; + } + | { + status: "refused"; + reason: CodexWriteLockRefusalReason; + retryable: false; + message: string; + }; + +export type CodexWriteLockAdmissionPhase = "before_namespace" | "under_lock"; + +export type CodexWriteLockAdmissionResult = + | { status: "admitted" } + | { status: "refused"; message: string }; + +export interface CodexWriteLockContext { + readonly canonicalCodexHome: string; + readonly lockId: string; +} + +export interface CodexWriteLockOptions { + /** + * Optional explicit target. When absent, a nonblank process CODEX_HOME wins; + * otherwise defaultCodexHome() supplies ~/.codex or the existing WSL default. + * Explicit and default targets pass through the same existing-directory + * realpath algorithm before identity or namespace work. + */ + codexHome?: string; + + /** + * Required total acquisition budget in milliseconds, including namespace ACL + * validation and every BEGIN IMMEDIATE attempt. Must be finite, integral, and + * within 0..CODEX_WRITE_LOCK_MAX_TIMEOUT_MS. Zero performs one fail-fast attempt. + */ + timeoutMs: number; + + /** Cancellation is a typed busy outcome; it is never thrown as contention. */ + signal?: AbortSignal; + + /** + * Read-only WP12 admission. It runs once after canonical-home resolution but + * BEFORE this module creates or opens a namespace entry, then again while the + * SQLite transaction is held. The lock proves exclusion only; an admission + * callback must independently prove service-home, external-provider, journal, + * provenance, and desired-state authority. + */ + admit( + phase: CodexWriteLockAdmissionPhase, + context: CodexWriteLockContext, + ): CodexWriteLockAdmissionResult; +} + +type Synchronous = T extends PromiseLike ? never : T; + +/** + * Acquire the per-canonical-CODEX_HOME cross-process write lock, execute one + * synchronous bounded commit, and release the SQLite transaction before this + * Promise resolves. + * + * Waiting is asynchronous and barging-allowed: SQLITE_BUSY closes candidate + * handles, sleeps with bounded jitter, and retries only until timeoutMs. The + * locked callback itself MUST NOT return a Promise or perform provider I/O, + * subprocess work, history walking, retry sleeps, or any other awaitable work. + * WP9 prepares catalog bytes before this call; WP10 owns history in a Worker. + * Keeping only their fixed commit/admission work here prevents a lock holder from + * becoming the event-loop outage that blocked the earlier OFF design. + * + * Contention, cancellation, filesystem validation, ACL failure, and SQLite-open + * failure return busy/refused. Invalid timing arguments and exceptions thrown by + * the caller's admission or locked callback remain programmer/domain exceptions. + */ +export async function withCodexWriteLock( + options: CodexWriteLockOptions, + locked: (context: CodexWriteLockContext) => Synchronous, +): Promise>; +``` + +There is deliberately no public handle and no `release()` method. A handle would +let a caller retain the transaction across an `await`, making “bounded” a comment +rather than an API boundary. The conditional return type rejects an ordinary +`async` callback at typecheck; the implementation also checks for a thenable after +invocation, rolls back immediately, and throws `TypeError` as a programmer error. +It never awaits a callback result. + +This narrows the research-level allowance at `003_lock_protocol.md:198-201` without +changing its Decision: acquisition remains async; the held operation is now +synchronous because WP9/WP10 remove the two reasons it previously needed to await. +The roadmap already records this stronger construction at `000_plan.md:105-109`. + +### Admission order + +The implementation order is fixed: + +```text +resolve existing canonical CODEX_HOME read-only +derive lockId and detect same-task reentrancy read-only +options.admit("before_namespace", context) read-only + refused -> return authority_not_proven; create NOTHING +resolve/validate real login home read-only +validate/create each private namespace component +validate/open stable database and BEGIN IMMEDIATE +options.admit("under_lock", context) read-only + refused -> rollback/close; do not run locked callback +locked(context) synchronous, bounded +assert stable path, ROLLBACK, close SQLite, close side fd +``` + +The callback names “admission”, not “ownership receipt”, because WP11 must not +manufacture a token that WP12 could accidentally treat as authority. The first call +closes the creation-before-knowledge bug; the second closes the check/lock race. +An `acquired` result means both admissions passed and the callback ran under the OS +transaction. It does not mean OpenCodex owns every artifact the callback might name. + +## Canonicalization — C6 + +### Exact algorithm + +`canonicalCodexHome(options)` implements these steps, in this order: + +1. Select raw input as `options.codexHome` when it is nonblank; otherwise use a + nonblank `process.env.CODEX_HOME`; otherwise call `defaultCodexHome()`. This + retains today's default/WSL precedence (`src/codex/home.ts:121-146`). A supplied + blank `options.codexHome` is a programmer error, not a request for default. +2. Expand only a leading `~` through the existing `expandUserPath`, then `resolve` + to an absolute path. Do not lowercase, Unicode-normalize, or append unresolved + suffixes. +3. `statSync` the target. `ENOENT`/`ENOTDIR` returns + `refused/codex_home_missing`; another read error or a non-directory returns + `refused/codex_home_unsafe`. No namespace function has run yet. +4. Call `realpathSync.native` for **both** default and explicit input. This collapses + `~`, dot segments, trailing separators, and every symlink in the existing path. +5. On Windows only, feed `win32.normalize(realPath).toLowerCase()` to the hash. The + existing diagnostics already compare Windows paths case-insensitively + (`src/codex/home.ts:164-183`). On macOS and Linux, hash the exact string returned + by `realpathSync.native`. +6. Refuse the already-recognized unsupported target classes: Windows UNC homes and + WSL `/mnt/` homes, using `nativeMainOwnerFilesystemSupported` + (`src/codex/native-main-owner.ts:75-91`). This phase does not claim portable + lock identity across network hosts or filesystem namespaces. +7. Hash exactly + `"opencodex-codex-write-lock-v1\0" + normalizedCanonicalHome` as UTF-8 with + SHA-256, lowercase hex, all 64 characters. + +For an existing case-insensitive macOS directory, `realpathSync.native` returns the +filesystem's stored directory-entry spelling, so `/Users/A/.CODEX` and +`/Users/A/.codex` converge. On a case-sensitive APFS volume, those can be two real +directories and must remain two identities. Windows can case-fold safely because a +single Windows namespace does not distinguish those spellings. Linux remains +case-sensitive. + +Two consequences are acceptance requirements, not examples: + +- default `~/.codex`, explicit `~/.codex`, its absolute spelling, and any symlink + to that same existing directory contend on one SQLite file; +- two different existing directories produce different 64-character IDs and can + acquire concurrently. + +### The missing-home question + +Missing homes are refused before hashing and before resolving the login-home lock +namespace. There is no portable alternative. If WP11 canonicalized the deepest +existing parent and preserved the absent suffix, `Foo` and `foo` would split one +future home on case-insensitive APFS. If it lowercased the suffix, they would alias +two future homes on case-sensitive APFS. Until the directory exists there is no +inode, filesystem-returned spelling, or case-behavior answer. Creation/installation +of `CODEX_HOME` is therefore another operation and another lock domain. + +The implementation hunk in the new module is: + +```diff +diff --git a/src/codex/codex-write-lock.ts b/src/codex/codex-write-lock.ts +new file mode 100644 +--- /dev/null ++++ b/src/codex/codex-write-lock.ts +@@ ++import { createHash } from "node:crypto"; ++import { lstatSync, mkdirSync, realpathSync, statSync } from "node:fs"; ++import { homedir } from "node:os"; ++import { join, resolve, win32 } from "node:path"; ++import { AsyncLocalStorage } from "node:async_hooks"; ++import { Database } from "bun:sqlite"; ++ ++import { expandUserPath } from "../config"; ++import { hardenSecretDirAsync } from "../lib/windows-secret-acl"; ++import { defaultCodexHome } from "./home"; ++import { ++ assertStableLockFile, ++ hardenStableLockFile, ++ openStableLockFile, ++ StableLockPathUnsafeError, ++ type StableLockFile, ++} from "./native-main-lock-file"; ++import { nativeMainOwnerFilesystemSupported } from "./native-main-owner"; ++ ++const LOCK_DOMAIN = "opencodex-codex-write-lock-v1\0"; ++const LOCK_NAMESPACE_PARTS = [".opencodex", "native-write-locks", "v1"] as const; ++const heldHomes = new AsyncLocalStorage>(); ++ ++function normalizeCanonicalHome(path: string, platform = process.platform): string { ++ return platform === "win32" ? win32.normalize(path).toLowerCase() : path; ++} ++ ++function lockIdFor(canonicalCodexHome: string): string { ++ return createHash("sha256") ++ .update(LOCK_DOMAIN) ++ .update(canonicalCodexHome) ++ .digest("hex"); ++} ++ ++function rawCodexHome(explicit: string | undefined): string { ++ if (explicit !== undefined) { ++ if (!explicit.trim()) throw new TypeError("codexHome must not be blank"); ++ return explicit; ++ } ++ return process.env.CODEX_HOME?.trim() || defaultCodexHome(); ++} ++ ++function canonicalCodexHome(explicit: string | undefined): ++ | { status: "ok"; path: string } ++ | Extract, { status: "refused" }> { ++ const absolute = resolve(expandUserPath(rawCodexHome(explicit))); ++ try { ++ if (!statSync(absolute).isDirectory()) { ++ return refused("codex_home_unsafe", "CODEX_HOME is not an existing directory."); ++ } ++ const real = realpathSync.native(absolute); ++ if (!nativeMainOwnerFilesystemSupported(real)) { ++ return refused("unsupported_filesystem", "CODEX_HOME uses an unsupported filesystem identity."); ++ } ++ return { status: "ok", path: normalizeCanonicalHome(real) }; ++ } catch (error) { ++ const code = errorCode(error); ++ return code === "ENOENT" || code === "ENOTDIR" ++ ? refused("codex_home_missing", "CODEX_HOME must exist before native writes can be locked.") ++ : refused("codex_home_unsafe", "CODEX_HOME could not be resolved safely."); ++ } ++} +``` + +`refused` and `errorCode` are private constructors in the same file; messages never +include the raw path or username. + +## Namespace and hardening — C7 + +### Exact path + +The namespace is independent of both `CODEX_HOME` and `OPENCODEX_HOME`: + +```text +realpathSync.native(homedir()) + /.opencodex + /native-write-locks + /v1 + /.sqlite +``` + +The login home itself may resolve through a symlink because it is immediately +realpathed. The three OpenCodex-owned descendants may not be symlinks, junctions, +or other path substitutions. A custom `OPENCODEX_HOME` never changes this path. + +### Component validation + +`ensurePrivateLockNamespace(deadline)` walks one component at a time; it never uses +recursive `mkdir`: + +1. Resolve and `stat` the login home, then `realpathSync.native` it. +2. For each descendant, `lstat` first. `ENOENT` permits one `mkdirSync(path, + { mode: 0o700 })`; `EEXIST` restarts validation. Any existing non-directory, + symlink, junction/reparse redirect, or realpath mismatch returns + `refused/namespace_unsafe`. +3. On POSIX, require `process.getuid()` and exact `(mode & 0o7777) === 0o700` plus + `stats.uid === process.getuid()`. Existing broader/narrower modes and another + uid are refused; WP11 never chmods, renames, unlinks, or recreates them. +4. On Windows, compare case-folded `resolve(path)` and `realpathSync.native(path)` + to reject junction/reparse redirection, then run the existing async directory + ACL owner with `required: true` and the remaining outer deadline. A failed or + timed-out required ACL operation returns `refused/namespace_unsafe`; it never + proceeds to SQLite. The helper grants only the current user before removing + inheritance and broad SIDs (`src/lib/windows-secret-acl.ts:217-328,404-494`). +5. Re-`lstat` and re-run identity/mode checks after creation/hardening before + descending to the next component. + +For the database and SQLite sidecars: + +- Before open, any existing `.sqlite` or `.sqlite-journal` must be a regular + non-symlink entry; on POSIX it must have the same uid and exact `0600` mode. +- Existing `-wal` or `-shm` is refused as `lock_path_unsafe`. WP11 forces rollback + journal mode, so those names are unexpected state, not files to clean up. +- `openStableLockFile` performs `O_NOFOLLOW` on POSIX, then `fstat`; its retained + side descriptor and reference count prevent a sibling close from releasing this + process's SQLite lock (`src/codex/native-main-lock-file.ts:35-55,74-125`). +- After open, validate the descriptor's regular-file/uid/mode metadata, compare + path `(dev, ino)` through `assertStableLockFile`, run required Windows file ACL + hardening within the remaining deadline, and assert identity again before SQLite. +- SQLite executes `PRAGMA busy_timeout = 0`, `PRAGMA locking_mode = NORMAL`, verifies + `PRAGMA journal_mode = DELETE`, then tries `BEGIN IMMEDIATE`. SQLite's OS lock is + the only holder authority. +- Assert stable identity immediately after `BEGIN IMMEDIATE`, immediately before + the synchronous callback, and once more before rollback/close. Close SQLite + before closing the retained side descriptor. + +Any validation failure before the locked callback maps to `refused/namespace_unsafe` +or `refused/lock_path_unsafe`; ACL/SQLite setup failures map to the narrower safe +reason when known, otherwise `refused/lock_unavailable`. The implementation does not +repair or delete the suspect entry. Diagnostics identify only the component role +(`v1 namespace`, `lock database`, `journal sidecar`), never its full home path. + +The database persists after release. There is no `unlinkSync` in the module. A +crashed process loses its transaction when the OS closes SQLite; an old database +mtime or dead PID grants no takeover rights. A live hung process remains the holder, +and contenders return `busy/deadline`. + +### Existing helper changes + +The stable-file owner currently gives required Windows hardening its own fixed +deadline (`src/codex/native-main-lock-file.ts:127-131`). Add an optional caller cap +without changing existing call sites: + +```diff +diff --git a/src/codex/native-main-lock-file.ts b/src/codex/native-main-lock-file.ts +--- a/src/codex/native-main-lock-file.ts ++++ b/src/codex/native-main-lock-file.ts +@@ -127,6 +127,10 @@ +-export async function hardenStableLockFile(path: string): Promise { ++export async function hardenStableLockFile(path: string, timeoutMs?: number): Promise { + try { chmodSync(path, 0o600); } catch { /* Windows ACL below is authoritative there. */ } + if (process.platform === "win32") { +- await hardenSecretPathAsync(path, { required: true, timeoutMemoKey: path }); ++ await hardenSecretPathAsync(path, { ++ required: true, ++ timeoutMemoKey: path, ++ timeoutMs, ++ }); + } + } +``` + +WP11 does **not** call `hardenStableLockFile` on POSIX: its unconditional `chmodSync` +is compatible with existing native-main users but forbidden for this strict namespace. +WP11 validates exact POSIX metadata instead. On Windows the existing ACL operation is +the authoritative platform control. + +Cap the ACL helper's existing configured budget, leaving all current callers +unchanged: + +```diff +diff --git a/src/lib/windows-secret-acl.ts b/src/lib/windows-secret-acl.ts +--- a/src/lib/windows-secret-acl.ts ++++ b/src/lib/windows-secret-acl.ts +@@ -53,2 +53,4 @@ + timeoutMemoKey?: string; ++ /** Optional stricter caller budget; never enlarges OPENCODEX_ACL_TIMEOUT_MS. */ ++ timeoutMs?: number; + } +@@ -68,7 +70,10 @@ +-function resolveHardenDeadlineMs(): number { ++function resolveHardenDeadlineMs(opts: HardenOptions): number { + const raw = env["OPENCODEX_ACL_TIMEOUT_MS"]?.trim(); +- if (!raw) return HARDEN_DEADLINE_DEFAULT_MS; + const parsed = Number(raw); +- if (!Number.isSafeInteger(parsed)) return HARDEN_DEADLINE_DEFAULT_MS; +- return Math.min(HARDEN_DEADLINE_MAX_MS, Math.max(HARDEN_DEADLINE_MIN_MS, parsed)); ++ const configured = raw && Number.isSafeInteger(parsed) ++ ? Math.min(HARDEN_DEADLINE_MAX_MS, Math.max(HARDEN_DEADLINE_MIN_MS, parsed)) ++ : HARDEN_DEADLINE_DEFAULT_MS; ++ if (opts.timeoutMs === undefined) return configured; ++ if (!Number.isFinite(opts.timeoutMs) || opts.timeoutMs <= 0) return 1; ++ return Math.max(1, Math.min(configured, Math.floor(opts.timeoutMs))); + } +@@ -426 +431 @@ function hardenEntry( +- const deadline = nowFn() + resolveHardenDeadlineMs(); ++ const deadline = nowFn() + resolveHardenDeadlineMs(opts); +@@ -470 +475 @@ async function hardenEntryAsync( +- const deadline = nowFn() + resolveHardenDeadlineMs(); ++ const deadline = nowFn() + resolveHardenDeadlineMs(opts); +``` + +The `tests/windows-secret-acl.test.ts` addition injects the existing async runner, +calls `hardenSecretDirAsync(path, { required: true, timeoutMs: 37 })`, and asserts +every runner invocation receives `<= 37`; a failed runner still rejects. This is a +deadline plumbing test, not an ACL mock standing in for the Windows CI job. + +## Acquisition loop, release, and reentrancy — C5 + +The core new-file hunk is: + +```diff +diff --git a/src/codex/codex-write-lock.ts b/src/codex/codex-write-lock.ts +new file mode 100644 +--- /dev/null ++++ b/src/codex/codex-write-lock.ts +@@ ++const RETRY_MIN_MS = 25; ++const RETRY_MAX_MS = 75; ++ ++function isBusy(error: unknown): boolean { ++ const code = errorCode(error); ++ const message = error instanceof Error ? error.message : String(error); ++ return code === "SQLITE_BUSY" ++ || code === "SQLITE_LOCKED" ++ || /database (?:is|table is) locked/i.test(message); ++} ++ ++function jitter(random = Math.random): number { ++ return RETRY_MIN_MS + Math.floor(random() * (RETRY_MAX_MS - RETRY_MIN_MS + 1)); ++} ++ ++async function abortableSleep(ms: number, signal?: AbortSignal): Promise { ++ if (signal?.aborted) return false; ++ return new Promise(resolveSleep => { ++ let settled = false; ++ const finish = (completed: boolean): void => { ++ if (settled) return; ++ settled = true; ++ clearTimeout(timer); ++ signal?.removeEventListener("abort", onAbort); ++ resolveSleep(completed); ++ }; ++ const onAbort = (): void => finish(false); ++ const timer = setTimeout(() => finish(true), ms); ++ signal?.addEventListener("abort", onAbort, { once: true }); ++ }); ++} ++ ++function release(database: Database | undefined, file: StableLockFile | undefined): void { ++ try { database?.exec("ROLLBACK"); } catch { /* close remains the OS release */ } ++ try { database?.close(); } catch { /* transaction is already ending */ } ++ try { file?.close(); } catch { /* SQLite closed before the side descriptor */ } ++} ++ ++export async function withCodexWriteLock( ++ options: CodexWriteLockOptions, ++ locked: (context: CodexWriteLockContext) => Synchronous, ++): Promise> { ++ assertTimeout(options.timeoutMs); ++ const startedAt = performance.now(); ++ const deadline = startedAt + options.timeoutMs; ++ const canonical = canonicalCodexHome(options.codexHome); ++ if (canonical.status !== "ok") return canonical; ++ const lockId = lockIdFor(canonical.path); ++ const context = { canonicalCodexHome: canonical.path, lockId } as const; ++ if (heldHomes.getStore()?.has(canonical.path)) { ++ return refused("reentrant", "A nested Codex write attempted to acquire the same home."); ++ } ++ if (options.signal?.aborted) return busy("cancelled", startedAt); ++ const preflight = options.admit("before_namespace", context); ++ if (preflight.status === "refused") { ++ return refused("authority_not_proven", preflight.message); ++ } ++ ++ return heldHomes.run(new Set([...(heldHomes.getStore() ?? []), canonical.path]), async () => { ++ const target = await ensurePrivateLockTarget(lockId, deadline); ++ if (target.status !== "ok") return target; ++ let attempted = false; ++ for (;;) { ++ let file: StableLockFile | undefined; ++ let database: Database | undefined; ++ let callerCodeStarted = false; ++ try { ++ attempted = true; ++ ({ file, database } = await openCandidate(target, deadline)); ++ database.exec("BEGIN IMMEDIATE"); ++ assertStableLockFile(target.databasePath, file); ++ callerCodeStarted = true; ++ const underLock = options.admit("under_lock", context); ++ if (underLock.status === "refused") { ++ release(database, file); ++ return refused("authority_not_proven", underLock.message); ++ } ++ assertStableLockFile(target.databasePath, file); ++ const value = locked(context); ++ if (value && typeof value === "object" && "then" in value) { ++ throw new TypeError("Codex write locked callback must be synchronous"); ++ } ++ callerCodeStarted = false; ++ assertStableLockFile(target.databasePath, file); ++ release(database, file); ++ return { status: "acquired", value, waitedMs: elapsed(startedAt), lockId }; ++ } catch (error) { ++ release(database, file); ++ if (callerCodeStarted) throw error; ++ if (!isBusy(error)) return mapAcquireRefusal(error); ++ if (options.signal?.aborted) return busy("cancelled", startedAt); ++ const remaining = deadline - performance.now(); ++ if (attempted && remaining <= 0) return busy("deadline", startedAt); ++ const slept = await abortableSleep(Math.min(jitter(), remaining), options.signal); ++ if (!slept) return busy("cancelled", startedAt); ++ } ++ } ++ }); ++} +``` + +`ensurePrivateLockTarget` and `openCandidate` implement the namespace rules above. +`openCandidate` always closes both handles on failure and maps +`StableLockPathUnsafeError` to `lock_path_unsafe`. It validates sidecars afresh on +every retry because another process may replace a path while this contender sleeps. + +The timeout is monotonic (`performance.now`), required, and total. A zero timeout +still gets exactly one `BEGIN IMMEDIATE`; if busy, it returns immediately. No SQLite +busy timeout, ACL subprocess, or retry sleep may exceed the remaining outer budget. +Jitter is uniformly bounded to integer 25–75 ms and clipped to the deadline. +Contenders may barge after any sleep; no test or caller may infer FIFO order. + +Only `SQLITE_BUSY`/`SQLITE_LOCKED` enters the retry loop. Filesystem, ACL, malformed +database, unexpected journal mode, identity, and permission failures are refusals, +not contention. There is no catch that turns callback exceptions into `busy` or +`refused`; after release they propagate unchanged. + +## Deadlock order and current inverse-nesting proof + +The only legal nested order is: + +```text +Codex write lock (async acquisition; synchronous held callback) + -> withConfigMutationLockSync / mutatePersistedConfig + -> return before the Codex callback returns + -> fixed native commit +-> release Codex write lock +``` + +Never call `withCodexWriteLock` from inside `withConfigMutationLockSync`, from a +`mutatePersistedConfig` mutation callback, or from a helper reached by either +callback. Outer config contention remains `ConfigMutationLockError`; WP12 may retry +that synchronous acquisition only while the outer Codex deadline remains. It must +not release and silently reorder the requested state change. + +Fresh search on the current tree found no inverse edge: + +```text +$ rg -n 'withConfigMutationLockSync\(|mutatePersistedConfig\(' src --glob '*.ts' +src/config.ts:1829: withConfigMutationLockSync(() => persistConfigUnlocked(config)); +src/config.ts:1870: return withConfigMutationLockSync(() => { +src/config.ts:2145: withConfigMutationLockSync(() => { +src/codex/account-store.ts:281: return withConfigMutationLockSync(fn); +src/codex/auth-api.ts:670: outcome = mutatePersistedConfig(persistedConfig => { +``` + +The three config-owned sections perform config snapshots/persistence only +(`src/config.ts:1821-1829,1861-1913,2144-2176`). The account wrapper is a direct +typed-error translation (`src/codex/account-store.ts:278-285`). The sole external +`mutatePersistedConfig` callback updates plan strings and performs no Codex native +operation (`src/codex/auth-api.ts:660-701`). None imports the new module today. +Therefore adding the future WP12 edge `codex-write -> config` cannot close a cycle +in the current graph. + +Add a source-shape case to `tests/codex-write-lock.test.ts` that reruns this inventory +over `src/config.ts`, `src/codex/account-store.ts`, and `src/codex/auth-api.ts`, and +fails if `codex-write-lock` or `withCodexWriteLock` appears inside an existing +config-lock callback. This test protects inverse nesting; it does not reject a WP12 +orchestrator that correctly acquires Codex first and calls config second. + +## Test plan + +### `tests/helpers/codex-write-lock-child.ts` (NEW) + +The helper accepts `CODEX_HOME`, `HOLD_MS`, and marker paths through its environment. +It calls the production `withCodexWriteLock` with a 5-second deadline and an +always-admitted test callback, writes `READY_PATH`, then executes one finite +`Bun.sleepSync(HOLD_MS)` inside the synchronous locked callback. It prints the typed +result as one JSON line and exits nonzero unless status is `acquired`. It never +opens SQLite directly; contention must exercise the production namespace and API. + +### `tests/codex-write-lock.test.ts` (NEW) + +Every test creates both a fake login home and existing Codex homes below one test +root, sets `HOME`/`USERPROFILE`, and restores them in `afterEach`. It never resolves +the real user's `.codex` or `.opencodex`. + +1. **Real two-process exclusion and barging contract.** Spawn the child on home A, + wait for its ready marker, then call the production API in the parent. A 100 ms + deadline returns `{ status:"busy", reason:"deadline" }` and the parent callback + does not run. A second parent waiter with 2 s acquires after the child's bounded + release. A timer increments while waiting, proving retry sleep is async. Assert + exclusion and eventual two-party acquisition only, never arrival order. +2. **Crash release, no stale recovery.** Child acquires and exits from inside its + callback. After its zero exit, parent acquires the persistent database without + unlink, PID, mtime, quarantine, or recovery marker. This mirrors the existing + OS-release proof (`tests/config-mutation-lock.test.ts:105-129`). +3. **Live holder is never stolen.** Hold longer than two successive parent deadlines; + both return busy, the database inode is unchanged, and no path is removed. Age is + not takeover authority. +4. **Deadline and cancellation.** Zero gets one fail-fast attempt; finite expiry is + typed busy; an already-aborted signal returns `busy/cancelled`; a signal fired + during jitter cancels the timer and returns the same. None throws contention. +5. **Callback boundary.** A synchronous value appears in `acquired.value`; a thrown + domain error propagates after release; an `async` callback is a compile-time + `@ts-expect-error`; a cast thenable activates the runtime `TypeError` and releases + the lock for a later call. +6. **Admission order.** `before_namespace` refusal leaves + `.opencodex/native-write-locks` absent. Under-lock refusal may leave the persistent + database but never calls the locked callback. Record phase order exactly as + `before_namespace, under_lock, locked`. +7. **Same-task reentrancy.** Calling the API again for the same canonical home from + the callback returns `refused/reentrant` without waiting. A separately started + same-process task is an ordinary contender and acquires after release. +8. **Default/explicit/absolute/tilde.** With default login `.codex` existing, delete + `CODEX_HOME` and have the child hold the default spelling. Parent attempts using + explicit `~/.codex` and the absolute path both return busy on the same `lockId`. +9. **Symlinked home.** Child holds a real directory; parent targets a symlink to it. + The parent is busy and the one expected full-hash database exists. Reverse the + spellings so the default itself is the symlink; the result is identical. +10. **Case behavior.** Create `CaseHome`, then probe `casehome`. If the platform + resolves both to the same existing directory, assert contention and one ID. If + the alternate spelling is missing, first assert `codex_home_missing` creates no + namespace; then create the second directory and assert both acquire concurrently + with distinct IDs. On Windows, slash and drive-letter case variants also share + one ID. +11. **Distinct homes.** Hold home A in the child and acquire home B immediately in + the parent. Assert two different 64-hex IDs and database paths. +12. **Missing home.** Test explicit and default missing paths. Both return + `codex_home_missing`, `admit` is not called, and the fake login home still has no + `.opencodex` descendant. This activates the case-sensitive/case-insensitive + resolution rather than testing only a pure hash helper. +13. **Namespace symlinks.** Independently replace `.opencodex`, + `native-write-locks`, and `v1` with a real symlink/junction. Each returns + `namespace_unsafe`, preserves the entry/target byte-for-byte, and creates no DB. +14. **Database/sidecar substitution.** A symlink database, symlink `-journal`, or + existing `-wal`/`-shm` returns `lock_path_unsafe` and is not removed. A test hook + swaps the DB after stable open and proves `(dev, ino)` revalidation refuses. +15. **POSIX owner/mode.** Existing namespace modes `0755`/`0700` and DB modes + `0644`/`0600` cover refusal/success. Inject a mismatched effective uid for the + deterministic wrong-owner branch; when CI runs as uid 0, additionally `chown` + a fixture and prove the real metadata branch. No test expects chmod repair. +16. **Windows ACL and junctions.** On `windows-latest`, create real directory + junctions for each namespace component and require refusal. Inject the existing + `icacls` runner for required failure/timeout mapping, while the normal success + case runs the real required ACL path. UNC and WSL DrvFS identities return + `unsupported_filesystem` through the existing predicate. +17. **Malformed database and rollback journal.** Preserve malformed bytes and return + `lock_unavailable`; accept a same-owner/mode regular rollback journal, refuse + wrong metadata, and never silently switch to WAL. +18. **Deadlock source shape.** Re-run the inventory described above and pin + `Codex-write -> config`, with no inverse callback acquisition. + +The wrong-owner uid injection is only for a branch a non-root CI process cannot +materialize. Symlink, mode, substitution, SQLite contention, process crash, and +deadline tests all use real filesystem/process behavior. + +## Verification + +No verification command starts, stops, syncs, restores, or ensures the proxy. Port +10100 remains untouched. + +Run in this order after WP9 and WP10 are present and the diff is implemented: + +```bash +bun test tests/codex-write-lock.test.ts --test-name-pattern "real two-process exclusion" +bun test tests/codex-write-lock.test.ts tests/windows-secret-acl.test.ts tests/native-main-claim.test.ts tests/native-main-owner-lifetime.test.ts tests/config-mutation-lock.test.ts +bun run typecheck +bun run test +bun run privacy:scan +``` + +The first command is the required real two-process contention run: the child holds +the production SQLite transaction, the parent expires once as typed `busy`, remains +event-loop responsive, then acquires after release. A mocked `SQLITE_BUSY`, two +connections in one process, or a pure lock-ID test does not satisfy it. + +Run the focused test and typecheck on macOS, Linux, and Windows. Windows must execute +the real junction and ACL-success cases; POSIX must execute exact uid/mode checks. +The full suite is required because `native-main-lock-file.ts` and +`windows-secret-acl.ts` are shared owners even though their existing defaults are +preserved. + +## Deliberate residuals + +- `realpath` does not collapse bind-mount or filesystem-namespace aliases. Portable + directory file identity and cross-host/network-filesystem coordination remain + unsupported, as `003_lock_protocol.md:345-350` already marks **INFERRED**. WP11 + refuses the target classes the repository can identify; it does not claim every + alias can be detected portably. +- The concrete 15-second OFF and 5-second startup/background budgets in + `003_lock_protocol.md:173-176` remain caller-policy in WP12. WP11 enforces only the + required finite `0..30_000` ms API bound. +- Creation of a missing `CODEX_HOME` remains a separate lock domain. No future WP12 + convenience path may weaken missing-home refusal in this module. + +## Accept criteria + +- **C5 — finite async acquisition and typed contention.** Every call supplies an + integral `0..30_000` ms total deadline. Real cross-process `BEGIN IMMEDIATE` + contention yields `busy/deadline`, cancellation yields `busy/cancelled`, ordinary + acquisition yields `acquired`, and unsafe setup yields `refused`; contention is + never an exception. Retry sleeps are async 25–75 ms bounded jitter, FIFO is not + claimed, the locked callback is synchronous/bounded, and no PID/mtime takeover or + stale-file unlink exists. +- **C6 — one identity per real home.** Both explicit and default homes require an + existing directory and pass through `realpathSync.native`; Windows additionally + normalizes/case-folds. Default, explicit, absolute, tilde, symlink, separator, and + case-equivalent spellings contend on one full SHA-256 lock, while two different + existing canonical directories acquire independently. A missing home refuses + before admission, hashing side effects, or namespace creation. +- **C7 — private hardened namespace.** The database path is exactly + `/.opencodex/native-write-locks/v1/.sqlite`, never + `tmpdir`, `CODEX_HOME`, or `OPENCODEX_HOME`. POSIX requires real same-uid `0700` + directories and a same-uid `0600` regular DB/rollback journal; Windows requires + non-junction identity plus successful required per-user ACL hardening within the + outer deadline. Symlink, wrong-owner/mode, substitution, WAL/SHM residue, ACL + failure, and unsupported filesystem identity refuse without chmod repair, rename, + unlink, or recreation. + +WP11 is complete only when the focused real-process activation, cross-platform +hardening cases, typecheck, full tests, and privacy scan all pass. It still does not +authorize a native write: WP12 must supply the two read-only admissions and handle +config-lock retry/commit outcomes under this exclusion boundary. diff --git a/devlog/_plan/260804_codex_write_substrate/040_ownership_convergence.md b/devlog/_plan/260804_codex_write_substrate/040_ownership_convergence.md new file mode 100644 index 000000000..a6c5dc022 --- /dev/null +++ b/devlog/_plan/260804_codex_write_substrate/040_ownership_convergence.md @@ -0,0 +1,848 @@ +# WP12 — ownership authority and convergence + +Research: `004_ownership_and_convergence.md`. Read it first; this document is the +implementation diff for its Decision. Source citations and diff context below were +re-verified at `7e67a8d06311de2471b0a25e41cf85f97007cc69` on 2026-08-04. + +The failure to prevent is data loss, not an untidy status result. Today a dead-PID +version-1 journal with no injected hashes is treated as permission to replay its +baseline (`src/codex/journal.ts:109-134`), while a corrupt service-state mirror is +collapsed into the same `null` as no service (`src/service.ts:165-175`) and the +native teardown preflight converts that uncertainty to success +(`src/integrations/native/ownership-preflight.ts:21-35`). If startup replaces the +separate external-provider check with that service check, it can overwrite a +`config.toml` now owned by another provider. WP9-WP11 add a gather/commit seam, +off-event-loop history work, and a bounded per-home lock. WP12 adds the authority +admission that must run before that lock can create anything, the provenance needed +to prove what OpenCodex created, and an observed-state projection that distinguishes +desired intent from actual convergence. + +## IN / OUT + +IN: + +| Path | Change | Why | +|---|---|---| +| `src/types.ts` | MODIFY | Adds only `clientIntegrations.codex?: boolean`; absent means desired ON. | +| `src/config.ts` | MODIFY | Parses the one-key extension-safe object and exports a pure, file-backed Codex intent reader. | +| `src/service.ts` | MODIFY | Stops skipping bad service-state mirrors and exposes read-only registration/mirror evidence. | +| `src/integrations/native/ownership-preflight.ts` | MODIFY | Replaces the fail-open boolean preflight with the tri-state mutation authority. | +| `src/codex/integration-record.ts` | NEW | Owns `getConfigDir()/integrations/codex.json`, including WP10 history state and the exact provenance ledger below. | +| `src/codex/ownership-convergence.ts` | NEW | Owns read-only admission, gather/lock/recheck orchestration, observed state, and convergence results. | +| `src/codex/journal.ts` | MODIFY | Makes journal inspection read-only and typed; recovery becomes an under-lock operation over a previously inspected dead writer. | +| `src/codex/inject.ts` | MODIFY | Preserves external-provider bytes, removes filename-based deletion authority, and exposes only receipt-gated apply/restore commits. | +| `src/codex/sync.ts` | MODIFY | Delegates apply to the common convergence owner instead of gathering/writing from a startup-captured config object. | +| `src/codex/catalog/sync.ts` | MODIFY | Records catalog/cache post-images and restores baseline absence; cache invalidation is no longer an unowned write. | +| `src/server/index.ts` | MODIFY | Removes the unconditional startup cache write at current line 403. | +| `src/cli/index.ts` | MODIFY | Routes start and both ensure branches through the one admission order; proxy startup survives an ownership refusal. | +| `src/server/management/config-routes.ts` | MODIFY | Makes `/api/sync` reread persisted intent instead of passing the server-captured `config`. | +| `tests/codex-ownership-authority.test.ts` | NEW | Pins owned/foreign/unknown and the no-artifact-before-answer invariant. | +| `tests/codex-artifact-provenance.test.ts` | NEW | Pins baseline absence, matching-post-image deletion, and preserved-drift conflict behavior. | +| `tests/codex-observed-state.test.ts` | NEW | Pins the complete observed projection and desired/observed convergence relation. | +| `tests/codex-convergence-order.test.ts` | NEW | Pins the trace order for startup, ensure, sync, apply, restore, stop, and uninstall entry points. | +| `tests/codex-journal.test.ts` | MODIFY | Reverses corrupt/unknown journal deletion and markerless automatic replay expectations. | +| `tests/codex-models-cache-restore.test.ts` | NEW | Proves an apply-created cache returns to absence and native drift is preserved. | +| `tests/codex-sync-api.test.ts` | MODIFY | Proves one running server observes CLI intent changes made by another process. | +| `tests/service.test.ts`, `tests/uninstall.test.ts` | MODIFY | Pin mirror conflict/unreadable evidence and fail-closed teardown. | +| `docs-site/src/content/docs/reference/cli/lifecycle.md` | MODIFY | Documents blocked/external/partial convergence without claiming that proxy startup failed. | +| `docs-site/src/content/docs/reference/configuration.md` | MODIFY | Documents `clientIntegrations.codex`, absent-means-ON, and desired versus observed state. | + +The predecessor names `src/codex/write-lock.ts` (WP11) and +`src/codex/history-convergence.ts` (WP10) are consumed but not redesigned here. +WP12 may modify their exported record composition/types only where the exact +`integrations/codex.json` schema below requires it; it must not weaken WP10's +off-event-loop boundary or WP11's acquisition protocol. + +OUT: `gui/**`, Grok, Claude Code, Claude Desktop, the six file integrations, +provider transport, releases, publishing, deployment, tags, npm, and the live proxy +on port 10100. WP12 supplies state/result types for the later Codex toggle, but it +does not add that route or render a switch. It does not promise byte-exact rollback +after a user or Codex has edited a baseline-absent artifact; that case can only be +preserved and reported. + +## The tri-state authority + +The public API belongs at the existing native preflight boundary. It returns the +canonical homes and evidence because a boolean cannot distinguish “another home +owns this” from “the ownership record could not be read”. + +```ts +/** + * Whether this process may mutate native Codex artifacts for one canonical home. + * + * `owned` is positive evidence: either no service registration and no mirror + * exist, or every readable/required mirror agrees with the installed service and + * the current canonical homes. `foreign` is a valid claim by another home. + * `unknown` means the evidence needed to choose is missing or cannot be trusted. + * Callers must permit native writes only for `owned`. + */ +export type NativeCodexOwnership = + | { + state: "owned"; + evidence: "no-service" | "matching-install"; + codexHome: string; + opencodexHome: string; + } + | { + state: "foreign"; + codexHome: string; + opencodexHome: string; + recordedCodexHome: string; + recordedOpenCodexHome: string; + message: string; + } + | { + state: "unknown"; + reason: + | "service-state-missing" + | "service-state-corrupt" + | "service-state-unreadable" + | "service-state-conflict" + | "service-registration-unknown" + | "path-unresolvable"; + codexHome?: string; + opencodexHome?: string; + message: string; + }; + +/** + * Read service registration and every known install-state mirror without repair, + * directory creation, SQLite open, chmod, unlink, rename, or config loading. + */ +export function inspectNativeCodexOwnership(): NativeCodexOwnership; +``` + +`assertNativeTeardownOwned` currently fails open for every error that is not the +specific mismatch class (`src/integrations/native/ownership-preflight.ts:25-35`). +That behavior was written for an interactive teardown route where a human sees the +result and can immediately repair a stale service record. Automatic convergence is +unattended: it runs during startup, ensure, server requests, crash recovery, and +later retries. In that setting an unreadable authority record cannot be converted +to deletion permission. A false refusal leaves residue that can be inspected; a +false success can destroy a newer config, catalog, or cache. Therefore both +`foreign` and `unknown` refuse, while the proxy itself may continue serving. + +### Actual diff — `src/integrations/native/ownership-preflight.ts:14-35` + +```diff + import { +- assertServiceEnvironmentMatchesInstall, +- isServiceOwnershipError, ++ inspectServiceInstallOwnership, + } from "../../service"; + +-export type NativeTeardownOwnership = { ok: true } | { ok: false; message: string }; ++export type NativeTeardownOwnership = ++ | { ok: true; ownership: Extract } ++ | { ok: false; ownership: Exclude; message: string }; + ++/** ++ * Read-only native mutation authority. Only `owned` authorizes a Codex write; ++ * foreign and unknown evidence are equally non-authorizing. ++ */ ++export function inspectNativeCodexOwnership(): NativeCodexOwnership { ++ return inspectServiceInstallOwnership(); ++} ++ + export function assertNativeTeardownOwned(): NativeTeardownOwnership { +- try { +- assertServiceEnvironmentMatchesInstall(); +- return { ok: true }; +- } catch (error) { +- if (isServiceOwnershipError(error)) { +- // The message names both the recorded and the current home — that is the +- // refusal text, verbatim, because the user has to act on it. +- return { ok: false, message: error.message }; +- } +- // Unrelated failure (corrupt state file, IO): mirror +- // `serviceEnvironmentOwnedHere` and fail open rather than wedging the route +- // behind a check whose own input is broken. +- return { ok: true }; +- } ++ const ownership = inspectNativeCodexOwnership(); ++ return ownership.state === "owned" ++ ? { ok: true, ownership } ++ : { ok: false, ownership, message: ownership.message }; + } +``` + +The `NativeCodexOwnership` declaration is inserted above +`NativeTeardownOwnership`; it is shown in full in the API block above and is not +duplicated in the diff. + +### Actual diff — `src/service.ts:165-175` + +The low-level reader must retain all mirror outcomes rather than returning the +first convenient valid row. `serviceRegistration` is derived from the existing +platform registration probes used by `diagnoseService` at +`src/service.ts:2370-2416`, but returns `unknown` when the platform probe itself +cannot establish presence. It is read-only; it does not call install, repair, +start, stop, or uninstall. + +```diff +-function readServiceInstallState(): ServiceInstallState | null { +- for (const path of serviceStatePaths()) { +- try { +- const parsed = parseServiceInstallState(JSON.parse(readFileSync(path, "utf8"))); +- if (parsed) return parsed; +- } catch { +- /* try the next known state path */ +- } +- } +- return null; +-} ++export type ServiceInstallStateRead = ++ | { status: "absent"; path: string } ++ | { status: "valid"; path: string; state: ServiceInstallState } ++ | { status: "corrupt"; path: string; message: string } ++ | { status: "unreadable"; path: string; message: string }; ++ ++/** Read every known mirror without creating, deleting, or repairing any path. */ ++export function readServiceInstallStates(): readonly ServiceInstallStateRead[] { ++ return serviceStatePaths().map(path => { ++ try { ++ const parsed = parseServiceInstallState(JSON.parse(readFileSync(path, "utf8"))); ++ return parsed ++ ? { status: "valid" as const, path, state: parsed } ++ : { status: "corrupt" as const, path, message: "invalid service-state schema" }; ++ } catch (error) { ++ const code = (error as NodeJS.ErrnoException).code; ++ if (code === "ENOENT") return { status: "absent" as const, path }; ++ if (error instanceof SyntaxError) { ++ return { status: "corrupt" as const, path, message: error.message }; ++ } ++ return { status: "unreadable" as const, path, message: error instanceof Error ? error.message : String(error) }; ++ } ++ }); ++} +``` + +Immediately after this reader, add `inspectServiceInstallOwnership()`. Its truth +table is exact: + +| Registration evidence | Mirror evidence | Result | +|---|---|---| +| absent | all absent | `owned/no-service` | +| installed | all required mirrors valid, canonical pairs equal each other and current pair | `owned/matching-install` | +| any | any valid mirror names another canonical pair | `foreign` | +| installed | all absent | `unknown/service-state-missing` | +| any | corrupt | `unknown/service-state-corrupt` | +| any | unreadable | `unknown/service-state-unreadable` | +| any | two valid canonical pairs disagree | `unknown/service-state-conflict` | +| unknown | no valid decisive foreign claim | `unknown/service-registration-unknown` | +| any | current or recorded path cannot be canonicalized | `unknown/path-unresolvable` | + +A valid foreign claim wins over an absent sibling mirror, but never over a corrupt, +unreadable, or conflicting mirror: those are `unknown`, because the complete evidence +set is not trustworthy. Existing `readServiceBackend`, diagnostics, and interactive +service commands may keep a compatibility helper that selects one valid state; native +mutation admission must use only the all-mirror reader. + +## One admission order (C8) + +Every start, ensure branch, sync, apply, restore, stop, uninstall, and retry uses one +sequence. No caller may select a subset or reorder it: + +1. **Canonical paths.** Resolve existing canonical `CODEX_HOME`, `OPENCODEX_HOME`, + effective `config.toml`, generated profile, active catalog, cache, journal, + history DB, rollouts, integration record, and WP11 lock path without creating + any component. Failure is `unknown/path-unresolvable`. +2. **Service ownership.** Call `inspectNativeCodexOwnership`. `foreign` or + `unknown` returns `blocked` and stops this sequence. +3. **External provider.** Read the effective project `model_provider` from + `config.toml` without mutation. An external provider returns `external` and + stops every journal, config, profile, catalog, cache, history, rollout, backup, + provenance, and lock write. +4. **Journal/liveness.** Inspect without cleanup. Invalid/unknown-version bytes, + a live writer, or liveness `unknown` block. A valid dead writer is recoverable + only after provenance also authorizes it. +5. **Provenance.** Read and validate `integrations/codex.json` without creating it. + A missing record is legal only when no OpenCodex residue requiring ownership + proof exists. Corrupt, wrong-version, conflicting transaction, missing post-image, + or artifact/hash disagreement blocks the corresponding transition. +6. **Fresh intent.** Call `readPersistedCodexIntent`; only diagnostics with + `source === "file"` are authoritative. Missing/unreadable/invalid config is + `unknown`, never default ON. +7. **Gather.** For desired ON, run WP9 provider/catalog gathering outside the lock. + It may await network I/O and must not write. Desired OFF has no gather step. +8. **Lock/recheck.** Only now call WP11 acquisition. Its construction order remains + canonical-home validation -> authority receipt -> private namespace validation -> + stable lock file -> SQLite open -> `BEGIN IMMEDIATE` + (`003_lock_protocol.md:178-196`). Once acquired, repeat steps 1-6 from disk and + compare the new authority/intent digest with the pre-lock receipt. Any change + aborts before a native write. +9. **Commit/observe.** Recover an authorized dead journal first, then apply or + restore using the locked candidate/ledger. Read observed state while still + serialized. Release before logs, HTTP response shaping, network retries, or + app-server handling. + +Testable invariant: + +> Until steps 1-6 have returned authoritative answers, the filesystem snapshot must +> show no new lock file, SQLite database or sidecar, directory, journal, integration +> record, catalog backup, catalog, cache, config, profile, history manifest, history +> row, or rollout line. A `foreign`, `unknown`, `external`, live-writer, or unknown- +> journal trace ends before the first `lock:*` event. + +WP11's lock database is outside both configurable homes, but it is still an artifact +and is forbidden before the answer is known. Passing a path that happens to be +writable is not an authority receipt. + +## External `model_provider` remains a distinct authority (C9) + +Service-home ownership answers: “does another OpenCodex service installation claim +this canonical `CODEX_HOME`/`OPENCODEX_HOME` pair?” It says nothing about who owns +the contents of `config.toml`. The external-provider guard answers: “has the user +delegated effective Codex routing to a provider other than native `openai` or +`opencodex`?” A matching OpenCodex service can coexist with a newly selected +external provider; service ownership may be `owned` while config mutation authority +is absent. + +The previous design deleted this guard by substituting the service-home check. That +was wrong (`008_audit_synthesis_wp4_r2.md:31-35`). The external check stays after +service ownership and before journal inspection. It vetoes apply, restore, repair, +journal deletion, catalog/cache cleanup, history changes, and rollout changes. The +result is `external`, not “already converged”. + +### Actual diff — `src/codex/inject.ts:481-503,764-770` + +```diff + const activeProvider = externalCodexModelProvider(rawContent); + if (activeProvider) { +- // A launcher may have journaled before the provider manager took ownership. Never let shutdown +- // replay that stale snapshot over externally managed config. +- removeJournal(); + const nativeSubagentDefaultsWarning = configuredManagedSubagentDefaults(config) + ? `Native Codex sub-agent defaults were not injected: external model_provider ${tomlString(activeProvider)} owns config.toml.` + : undefined; +``` + +```diff + export function restoreNativeCodex(): { success: boolean; message: string } { + const activeProvider = currentExternalCodexModelProvider(); + if (activeProvider) { +- removeJournal(); +- return { success: true, message: `External Codex provider ${tomlString(activeProvider)} preserved; no native restore was needed.` }; ++ return { ++ success: false, ++ message: `Native Codex restore blocked: external model_provider ${tomlString(activeProvider)} owns config.toml; no Codex artifact was changed.`, ++ }; + } +``` + +Then make the writing body `restoreNativeCodexUnlocked(receipt)` internal to +`ownership-convergence.ts`; public callers receive the typed common convergence +result. `removeCodexConfig` may perform structural removal only with ledger entries +for the exact fragments and current transaction. Its current filename-only profile +unlink at `src/codex/inject.ts:723-742` is removed. + +## Provenance ledger and absence restoration (C10) + +### Location and exact record + +The one owned operational record is +`getConfigDir()/integrations/codex.json`. It is outside `CODEX_HOME`; it composes +WP10 history convergence with WP12 provenance. Desired intent remains the fresh +file-backed `clientIntegrations.codex` value in the main config so there is one +intent authority. `lastAdmittedDesired` below is evidence, not a writable intent. + +```ts +export interface CodexIntegrationRecordV1 { + version: 1; + /** Diagnostic snapshot only; never used instead of readPersistedCodexIntent(). */ + lastAdmittedDesired?: "on" | "off"; + history: Record; + provenance: { + activeTransactionId: string | null; + transactions: Record; + }; +} + +export interface CodexArtifactTransaction { + id: string; + desired: "on" | "off"; + state: "prepared" | "committing" | "applied" | "restoring" | "restored" | "conflict"; + startedAt: string; + completedAt?: string; + artifacts: Record; +} + +export type CodexArtifactKind = + | "config" + | "profile" + | "catalog" + | "catalog-backup" + | "cache" + | "journal" + | "history-manifest" + | "history-row" + | "rollout"; + +export interface CodexArtifactLedgerRow { + kind: CodexArtifactKind; + canonicalPath: string; + baseline: + | { state: "absent" } + | { state: "present"; sha256: string; bytesBase64?: string; mode?: number }; + /** Written only after the candidate write succeeds and its bytes are read back. */ + postImage: { sha256: string; recordedAt: string } | null; + ownedStructure?: { + routedSlugs?: string[]; + configFragments?: string[]; + historyRows?: Array<{ + threadId: string; + modelProvider: string | null; + source: string | null; + rolloutPath: string | null; + }>; + rolloutProviders?: Array<{ + path: string; + firstLine: string | null; + latest: string | null; + }>; + }; + restore: + | { state: "pending" } + | { state: "restored-exact"; recordedAt: string } + | { state: "restored-structural"; recordedAt: string } + | { state: "preserved-drift"; recordedAt: string; currentSha256: string; message: string } + | { state: "blocked"; recordedAt: string; message: string }; +} +``` + +`bytesBase64` is required for byte-restorable present baselines (config, profile, +catalog, cache, and pre-existing backups). It is omitted for history DB/rollout +rows, which restore semantically from `ownedStructure`; copying SQLite or JSONL +bytes would overwrite concurrent native work. The record reader validates version, +transaction ids, canonical unique paths, SHA-256 width, base64/hash agreement, and +the single active transaction. A malformed record is `blocked/provenance-unknown`. + +### When rows are written + +1. After pre-lock admission passes and WP11 is acquired, re-read all baselines. +2. Before the first native artifact write, atomically persist one `prepared` + transaction containing a row for every artifact the commit can touch. This is + where baseline `absent` is recorded. +3. Set the transaction to `committing`; perform one candidate write. +4. After that write returns, read the resulting bytes, compute full SHA-256, and + atomically persist `postImage`. Only then may the row prove “created by us”. +5. Repeat steps 3-4 per artifact. Set `applied` only after observed state verifies + every required ON artifact. Partial/crashed work retains the active transaction. + +A filename, marker, slash-qualified slug, mtime, backup name, or file location is +never creation proof. Creation requires both `baseline.state === "absent"` and a +non-null successful `postImage.sha256`. A crash after a native write but before the +post-image ledger update leaves `postImage:null`; that is intentionally unknown and +cannot authorize automatic deletion. + +### Restoration rules + +- Baseline present + current hash equals post-image: restore exact baseline bytes, + then verify the baseline hash. +- Baseline absent + current hash equals post-image: unlink, then verify absence. +- Baseline absent + current hash differs: preservation wins. If the format is + parseable and `ownedStructure` identifies exact OpenCodex fragments/rows, remove + only those fragments and preserve native additions. Report operational + `absent` with historical `preserved-drift`; never report byte-exact restoration. +- Baseline absent + drift is unparseable/ambiguous: make no write and report a + conflict. Deleting would destroy user data; rewriting would invent a baseline. +- Missing ledger, null post-image, wrong transaction, or hash mismatch without an + exact structural owner: preserve and block. + +The hardest case is deliberate: config, catalog, or cache was absent; OpenCodex +created it; Codex or the user later added native data. OFF must not delete that +file. It removes only proven routed residue when possible and reports +`preserved-drift`; otherwise it preserves the entire file and reports `blocked`. +Historical absence cannot be restored without data loss, so the implementation +must say so. + +### Actual diff — `src/codex/journal.ts:97-107,148-162` + +```diff +-function readJournal(): Journal | null { +- if (!existsSync(JOURNAL_PATH)) return null; ++export type JournalInspection = ++ | { state: "absent" } ++ | { state: "invalid"; reason: "corrupt" | "unknown-version"; message: string } ++ | { state: "valid"; journal: Journal; writer: "alive" | "dead" | "unknown"; postImageKnown: boolean }; ++ ++/** Inspect journal bytes and writer liveness without deleting or rewriting them. */ ++export function inspectJournal(): JournalInspection { ++ if (!existsSync(JOURNAL_PATH)) return { state: "absent" }; + try { +- const journal = JSON.parse(readFileSync(JOURNAL_PATH, "utf-8")) as Journal; +- if (journal.version !== 1) throw new Error("unknown version"); +- return journal; +- } catch { +- removeJournal(); +- return null; ++ const value = JSON.parse(readFileSync(JOURNAL_PATH, "utf-8")) as Partial; ++ if (value.version !== 1) return { state: "invalid", reason: "unknown-version", message: "unsupported journal version" }; ++ const journal = value as Journal; ++ let writer: "alive" | "dead" | "unknown"; ++ try { process.kill(journal.pid, 0); writer = "alive"; } ++ catch (error) { ++ const code = (error as NodeJS.ErrnoException).code; ++ writer = code === "ESRCH" ? "dead" : "unknown"; ++ } ++ return { ++ state: "valid", ++ journal, ++ writer, ++ postImageKnown: typeof journal.injectedConfigHash === "string" && journal.injectedProfileHash !== undefined, ++ }; ++ } catch (error) { ++ return { state: "invalid", reason: "corrupt", message: error instanceof Error ? error.message : String(error) }; + } + } +``` + +```diff +-export function reconcileJournal(): boolean { +- const journal = readJournal(); +- if (!journal) return false; +- try { +- process.kill(journal.pid, 0); +- return false; +- } catch (e: unknown) { +- if ((e as NodeJS.ErrnoException).code === "EPERM") { +- return false; +- } +- } +- const restored = restoreJournalState(); ++export function reconcileJournalUnlocked( ++ inspection: Extract, ++): RestoreJournalResult { ++ if (inspection.writer !== "dead" || !inspection.postImageKnown) { ++ return { configRestored: false, profileRestored: false, configChanged: false, profileChanged: false, complete: false }; ++ } ++ const restored = restoreJournalState(inspection.journal); +- if (!restored.configRestored && !restored.profileRestored) return false; +- console.error(`⚠️ Previous session (PID ${journal.pid}) did not shut down cleanly. Codex state restored from journal.`); +- return true; ++ return restored; + } +``` + +The final implementation returns `RestoreJournalResult` consistently; no log is +emitted under the lock. `restoreJournalState` accepts the inspected journal and no +longer calls a reader that could change the authority answer. Markerless version-1 journals are +valid but `postImageKnown:false`; provenance cannot prove current bytes, so +automatic recovery blocks instead of assuming unchanged. + +## Observed state and `unchanged` convergence (C11) + +`inspectCodexObservedState` is read-only and returns: + +```ts +export type CodexObservedState = + | { state: "applied"; historical: "exact"; artifacts: CodexArtifactObservation[] } + | { state: "absent"; historical: "exact" | "preserved-drift"; artifacts: CodexArtifactObservation[] } + | { state: "partial"; historical: "exact" | "preserved-drift" | "unknown"; artifacts: CodexArtifactObservation[] } + | { state: "external"; provider: string; artifacts: CodexArtifactObservation[] } + | { state: "blocked"; reasons: string[]; artifacts: CodexArtifactObservation[] }; + +export interface CodexConvergenceResult { + desired: "on" | "off" | "unknown"; + observed: CodexObservedState; + converged: boolean; + changed: boolean; + refusal?: "foreign" | "unknown" | "external" | "journal-active" | "provenance" | "lock-busy"; + message: string; +} + +export interface CodexSyncConvergenceResult extends CodexConvergenceResult { + ok: boolean; + retryable: boolean; + added: number; + catalogPath: string | null; + catalogExists: boolean; + catalogWritten: boolean; + cacheSynced: boolean; +} +``` + +The observer reads all of these before answering “is Codex currently applied?”: + +1. service ownership and external provider; +2. `config.toml` root `model_provider`, owned `openai_base_url`, active + `model_catalog_json`, embedded `[profiles.opencodex]`, routed root model, and + managed defaults; +3. generated profile existence, bytes/hash, and provenance; +4. active catalog parse state, provenance, and every transaction-recorded routed slug; +5. `models_cache.json` in wrapper or raw-catalog shape, provenance, and routed slugs; +6. journal validity, writer liveness, transaction identity, and post-image matches; +7. history DB rows tagged `opencodex`, backup-manifest entries, and each touched + rollout's first-line and latest provider observations; +8. catalog backup and transaction residue, especially artifacts whose baseline was absent. + +Desired ON converges only with observed `applied`; desired OFF converges only with +observed `absent`. `external`, `blocked`, and `partial` never converge. Operational +absence with `historical:"preserved-drift"` is converged for routing but is not an +exact historical restore, and the response must expose both facts. + +`mutatePersistedConfig` already distinguishes `unchanged` from `committed` +(`src/config.ts:1837-1839,1877-1913`). `unchanged` says only that the boolean already +matched. It never skips observation or work: desired OFF may have a routed cache row +after a crash, and desired ON may be missing a profile/catalog after explicit restore. +Both paths run admission, converge, and re-observe. + +## Fresh admission in a long-lived server (C12) + +Add this pure reader beside `readConfigDiagnostics` at `src/config.ts:1714-1715`: + +```ts +/** Read Codex intent from persisted, schema-valid config; never use fallback defaults as authority. */ +export function readPersistedCodexIntent(): + | { state: "known"; desired: "on" | "off" } + | { state: "unknown"; reason: "missing" | "invalid" } { + const diagnostics = readConfigDiagnostics(); + if (diagnostics.source !== "file") { + return { state: "unknown", reason: diagnostics.source === "default" ? "missing" : "invalid" }; + } + return { state: "known", desired: diagnostics.config.clientIntegrations?.codex === false ? "off" : "on" }; +} +``` + +`src/types.ts:533-545` gains the one-key `OcxClientIntegrationsConfig`, and +`src/config.ts:916-940` gains a `.passthrough()` nested schema. Unknown future +integration keys survive a field-scoped mutation. This substrate defines the +reader and writer, but not a GUI/toggle route. + +Every Codex-mutating request performs one config file open/read, JSON parse, and +schema validation before gather, then repeats that bounded read under the WP11 +lock. Cost is two O(config-file-bytes) local reads per infrequent mutation request, +zero resident watchers, and zero cross-process cache protocol. A watcher may later +reduce diagnostics latency, but it may never replace the two admission reads. + +### Actual diff — `src/server/management/config-routes.ts:261-268` + +```diff + if (url.pathname === "/api/sync" && req.method === "POST") { +- const { syncModelsToCodex } = await import("../../codex/sync"); ++ const { convergeCodexToPersistedIntent } = await import("../../codex/ownership-convergence"); + const { attachStaleAppServerHint } = await import("../../codex/app-server-processes"); +- const result = await syncModelsToCodex(undefined, config, null); ++ // The server-captured `config` at handleConfigRoutes line 77 is request-routing ++ // state, not mutation authority. This call rereads disk before gather and lock. ++ const result = await convergeCodexToPersistedIntent({ source: "api-sync", log: null }); + return jsonResponse({ + ...attachStaleAppServerHint(result), + ...(result.ok ? {} : { error: result.message }), +- }, result.ok ? 200 : 500); ++ }, result.ok ? 200 : result.retryable ? 409 : 503); + } +``` + +The API result adapter retains the existing sync fields (`added`, `catalogPath`, +`catalogExists`, `catalogWritten`, `cacheSynced`) from WP9 and adds desired, +observed, converged, refusal, and retryable. It does not read `config` to gate Codex. + +### Actual diff — `src/cli/index.ts:169-177,318-321,358-369,398-412` + +```diff +-import { reconcileJournal } from "../codex/journal"; ++import { convergeCodexToPersistedIntent } from "../codex/ownership-convergence"; +``` + +```diff + async function handleStart(options: { block?: boolean } = {}) { +@@ + const requestedPort = parsePortOption(); +- if (!currentExternalCodexModelProvider()) reconcileJournal(); + const existingPid = readPid(); +``` + +```diff + await maybeShowStarPrompt(); // once-only Yes/No GitHub-star prompt on first interactive start +- await syncModelsToCodex(port).catch(() => {}); ++ const codex = await convergeCodexToPersistedIntent({ source: "startup", port, log: console }); ++ if (!codex.converged) console.error(`⚠️ ${codex.message}`); + if (!currentExternalCodexModelProvider() && !shouldInjectApiAuthHeader(config) && config.syncResumeHistory !== false) { +``` + +```diff + async function handleEnsure() { +- if (!currentExternalCodexModelProvider()) reconcileJournal(); + const config = loadConfig(); +@@ + if (live) { +- await syncModelsToCodex(live.port).catch(e => { +- console.error(`⚠️ Model sync skipped: ${e instanceof Error ? e.message : String(e)}`); +- }); ++ const codex = await convergeCodexToPersistedIntent({ source: "ensure-live", port: live.port, log: console }); ++ if (!codex.converged) console.error(`⚠️ ${codex.message}`); +``` + +```diff +- await syncModelsToCodex(port).catch(e => { +- console.error(`⚠️ Model sync skipped: ${e instanceof Error ? e.message : String(e)}`); +- }); ++ const codex = await convergeCodexToPersistedIntent({ source: "ensure-parent", port, log: console }); ++ if (!codex.converged) console.error(`⚠️ ${codex.message}`); +``` + +The spawned child runs the startup path and the parent runs `ensure-parent`; both +admit independently from disk. The operation is idempotent under WP11, so the second +observer either proves convergence or performs residue repair. `src/server/index.ts` +removes import/current line 403 `invalidateCodexModelsCache()`; cache mutation occurs +only in the admitted commit. Thus the proxy can bind and serve other clients even +when Codex admission returns foreign, unknown, or external. + +## Common orchestrator + +`src/codex/ownership-convergence.ts` exports exactly these entry points: + +```ts +export interface CodexAdmissionReceipt { + canonicalCodexHome: string; + canonicalOpenCodexHome: string; + authorityDigest: string; + desired: "on" | "off"; + journalTransactionId: string | null; + provenanceRevision: string; +} + +/** Steps 1-6 only. This function performs no write and opens no coordinator. */ +export function inspectCodexMutationAdmission(): + | { status: "admitted"; receipt: CodexAdmissionReceipt } + | { status: "blocked" | "external"; result: CodexConvergenceResult }; + +/** Run the fixed admission -> gather -> lock/recheck -> commit/observe sequence. */ +export async function convergeCodexToPersistedIntent(options: { + source: "startup" | "ensure-live" | "ensure-parent" | "api-sync" | "explicit" | "teardown"; + port?: number; + log?: Pick | null; +}): Promise; + +/** Read the complete artifact projection without repair. */ +export function inspectCodexObservedState(): CodexObservedState; +``` + +The under-lock recheck computes a new receipt and requires equality of canonical +homes, authority digest, desired intent, journal transaction, and provenance +revision. WP9 candidate revisions are checked separately before catalog commit. +No caller can supply `desired` or ownership as an option; test seams replace the +readers, not the verdict. + +## Test plan + +All tests use fresh temporary `CODEX_HOME`, `OPENCODEX_HOME`, real-user-home lock +namespace overrides supplied by WP11's test seam, and port `0`. None invokes +`ocx start`, `ocx stop`, `ocx sync`, `ocx restore`, or `ocx ensure`, and none reaches +the live listener on 10100. + +### Authority and ordering + +1. `tests/codex-ownership-authority.test.ts` — no service/no mirrors is owned; + matching mirrors are owned; foreign canonical pair is foreign; installed plus + missing mirror, corrupt mirror, unreadable mirror, conflicting valid mirrors, + registration unknown, and unresolvable paths are unknown. +2. **Dead-PID markerless journal plus external provider, byte-exact preservation.** + Seed version-1 journal without injected hashes and dead PID, external + `model_provider`, config, profile, catalog, both backup forms, cache, history + manifest, SQLite DB, and rollout sentinels. Run startup admission and ensure + admission independently. Hash all bytes before/after; assert identical bytes, + journal present, no mtime change where supported, result `external`, and no lock, + SQLite, provenance, journal, or native artifact created. +3. Foreign-home run asserting **NO artifact was created**. Snapshot both homes and + the WP11 namespace; call startup, ensure, API sync, and teardown entries. Assert + the trace ends at `service:foreign`, directory trees and hashes are identical, + and lock DB/sidecars, integration record, journal, catalog backup, and cache are absent. +4. Invalid and unknown-version journals remain byte-exact and return blocked. + `EPERM`/liveness-unknown is not treated as dead. +5. Ordered trace table for every entry point: + `paths -> service -> external -> journal -> provenance -> intent -> gather -> + lock -> paths -> service -> external -> journal -> provenance -> intent -> + recover -> commit -> observe`. Desired OFF omits only `gather`; every refusal + ends before `lock`. + +### Provenance and restoration + +1. Apply into absent config/profile/catalog/backups/cache/journal/history-manifest; + assert each row first records `baseline:absent`, then a read-back post-image hash. +2. Restore with unchanged post-images; assert every transaction-created artifact + returns to absence and the transaction reaches `restored`. +3. **Cache absence restoration.** Begin without `models_cache.json`, apply routed + data, prove ledger absence + successful cache post-image, then desired OFF must + unlink it. A second OFF is a no-write success. This is different from the current + creation-only assertion at `tests/codex-models-cache-invalidate.test.ts:41-55`. +4. Baseline absent followed by native edits for config, catalog, and cache. Add + native content after apply. OFF preserves native additions, removes only exact + ledger-owned routing, and reports operational absent plus historical + `preserved-drift`. Unparseable drift is fully preserved and returns blocked. +5. Crash after artifact write but before post-image ledger write; restart sees + `postImage:null`, preserves the artifact, and reports provenance conflict. +6. Pre-existing same-named profile/catalog/cache/backup with no matching ledger is + never unlinked. Present-baseline exact restore requires current hash == post-image. +7. History/rollout restoration stays semantic: originals remain in the manifest + until DB rows and both first-line/latest rollout observations agree. + +### Observed state and fresh intent + +1. Table-drive applied, absent, each one-artifact partial, external, blocked, and + preserved-drift historical status. Include stale first-line rollout metadata and + a non-empty manifest with no matching DB row. +2. Persist desired OFF first, seed one residue artifact at a time, perform an + `unchanged` OFF write, and prove every case still converges. +3. Persist desired ON first, remove config/profile/catalog/cache one at a time, + perform an `unchanged` ON write, and prove reconstruction plus re-observation. +4. **Running server honors another process.** Construct the server with stale ON + in memory, persist OFF from a subprocess, call `/api/sync`, and assert no gather + or native write. Persist ON from the subprocess, call the same running server, + and assert gather/apply occurs without restart. Repeat with invalid persisted + config and assert unknown/no write. + +## Verification + +Fresh implementation gates: + +```bash +bun run typecheck +bun test tests/codex-ownership-authority.test.ts +bun test tests/codex-artifact-provenance.test.ts tests/codex-models-cache-restore.test.ts +bun test tests/codex-observed-state.test.ts tests/codex-convergence-order.test.ts +bun test tests/codex-journal.test.ts tests/codex-sync-api.test.ts tests/service.test.ts tests/uninstall.test.ts +bun run test +bun run lint:gui +bun run privacy:scan +``` + +Live proof is the real in-process server/subprocess case in +`tests/codex-sync-api.test.ts`, bound to port `0` with isolated homes. Its evidence +must show one server PID, two separate config-writer PIDs, OFF causing zero gather +and zero native writes, then ON causing the ordered gather/lock/commit path without +server restart. Artifact proof is the post-test tree/hash receipt from the external, +foreign, cache-absence, and preserved-drift fixtures. A green response envelope or +green suite without those read-backs is insufficient. Do not use the live proxy on +10100 for WP12 verification. + +## Accept criteria + +- **C8 — authority before artifacts.** Foreign and unknown service ownership fail + closed. Tests prove no lock file/database/sidecar, directory, journal, provenance + record, or native artifact is created before paths, service ownership, external + provider, journal/liveness, provenance, and fresh intent all answer. +- **C9 — separate external authority.** A matching service does not override an + external effective `model_provider`; apply, restore, repair, journal cleanup, + catalog/cache/history/rollout writes all remain byte-exactly suppressed. +- **C10 — creation and preservation.** “Created by us” requires ledger baseline + absence plus successful read-back post-image hash. Matching post-images restore + absence. Later native edits are preserved and reported as conflict or + `preserved-drift`; no byte-exact claim is made where none is possible. +- **C11 — observed convergence.** Config, profile, catalog, cache, journal, history, + rollouts, backups, and provenance are inspected. `unchanged` intent still runs + convergence and post-observation; only ON/applied and OFF/absent are converged. +- **C12 — fresh server admission.** Every Codex mutation rereads file-backed intent + before gather and under lock. The subprocess test proves a CLI OFF and later ON + are honored by the same running server at a cost of two O(config-file-bytes) + local reads per mutation request. + +## Explicitly open after `004` + +The Decision leaves two future hardening items, neither of which may be silently +claimed here: journal liveness still identifies a writer by PID only, so PID reuse +can conservatively delay recovery until a later journal version records a process- +start/instance token; and terminal provenance transaction retention/compaction needs +a bounded policy after enough production evidence exists. Neither gap permits fail- +open mutation. Unknown liveness or provenance remains blocked, and preservation wins. From 2e8b27ddc384fe5469947a51268370046d869497 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Tue, 4 Aug 2026 10:06:37 +0900 Subject: [PATCH 027/163] docs(substrate): a field report that proves two of this unit's criteria MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The owner found opencodex missing from gjc's /provider list, and Pi refusing to start at all: models.json was three bytes, {}, which violates its own schema because providers is a required key. The journal shows both were disabled during this project's own investigative sessions yesterday and never re-applied. Two defects, both this unit's business. First, disable can leave a file that is invalid rather than merely empty — removing the last member of a required container is not the same as removing a member, and the writer does not distinguish them. Second, nothing reconciles an integration that is off but should be on, which is the desired-state gap seen from the other side: a disable performed for one purpose stayed in effect for a day, silently, across restarts. Amends 040: the artifact inventory needs a baseline class for containers the client requires to be non-empty, so a remover can tell restore-to-absent from restore-to-valid-minimum and never writes a bare {} unless that is what preceded us. Both files were re-exported and verified — gjc passes its real schema with the user's own profiles block preserved, and both now carry the WP2 modality fix. --- .../005_disable_leaves_a_broken_file.md | 98 +++++++++++++++++++ 1 file changed, 98 insertions(+) create mode 100644 devlog/_plan/260804_codex_write_substrate/005_disable_leaves_a_broken_file.md diff --git a/devlog/_plan/260804_codex_write_substrate/005_disable_leaves_a_broken_file.md b/devlog/_plan/260804_codex_write_substrate/005_disable_leaves_a_broken_file.md new file mode 100644 index 000000000..c01254a81 --- /dev/null +++ b/devlog/_plan/260804_codex_write_substrate/005_disable_leaves_a_broken_file.md @@ -0,0 +1,98 @@ +# A disable left two clients broken, and one of them invalid + +Field report, 260804. Not a hypothetical: the owner hit it while using the +product, and it lands in this unit because it is the same class of defect the +substrate exists to prevent. + +## What the owner saw + +`gjc` ran `/provider` and opencodex was not in the list. Pi was worse — it +refused to start at all: + +``` +Error: models.json error: Invalid models.json schema: +- providers: must have required properties providers + File: /Users/jun/.pi/agent/models.json +``` + +## What was actually on disk + +`~/.pi/agent/models.json` was **three bytes**: `{}`. + +`~/.gjc/agent/models.yml` was seven lines — the user's own `profiles:` block and +nothing else. No `providers:` key at all. + +The integration journal explains both: + +``` +disable pi 2026-08-03T13:50:15Z /Users/jun/.pi/agent/models.json +disable gajae 2026-08-04T01:02:56Z /Users/jun/.gjc/agent/models.yml +``` + +Those disables came from investigative work in this project's own sessions. The +integration was never re-applied, and nothing ever told the user their clients +were now unconfigured. + +## The two distinct defects + +**1. Disable can leave a file that is invalid, not merely empty.** + +Pi's schema requires a `providers` key. Removing our block removed the last +provider, so the writer left `{}` — syntactically fine, semantically illegal. +The client does not fall back; it refuses to load. Removing the only occupant of +a required container is not the same as removing an occupant, and the writer does +not distinguish them. + +gjc degrades more gracefully — it drops back to its built-in list — but the +outcome is the same for the user: the routed models are gone with no explanation. + +**2. Nothing reconciles an integration that is off but should be on.** + +This is exactly the gap `../260803_codex_desktop_toggle/003_durable_desired_state.md` +named for Codex and Grok, seen from the other side. There is no desired state, so +there is nothing to notice that observed state has diverged from it. A disable +performed for one purpose stays in effect indefinitely, silently, across restarts. + +## Why it belongs to this unit + +`000_plan.md` C10 already says an artifact that did not exist before apply must be +*removed* on convergence, and C11 says `unchanged` desired state must still +converge observed state. This incident is the concrete proof that both criteria +are load-bearing rather than theoretical: + +- C10's inverse case is here — Pi's file DID exist before apply and had to keep + existing in a **valid** shape after removal. The provenance ledger in + `040_ownership_convergence.md` records baseline state precisely so a remover can + tell "restore to absent" from "restore to a valid minimal document". +- C11 is the reason a user should never be left in this state for a day: with + desired state recorded, startup convergence sees ON-with-missing-artifacts and + re-applies. + +**Amendment to `040`:** the artifact inventory must treat "the client's schema +requires this container to be non-empty" as a distinct baseline class. Removing +the last member is either a restore-to-baseline (if we created the file) or a +restore-to-valid-minimum (if it pre-existed) — never a bare `{}` unless `{}` is +what was there before us. + +## Immediate remediation performed + +Both files were re-exported from the dev tree and verified: + +| | Result | +|---|---| +| `~/.pi/agent/models.json` | `providers.opencodex`, 34 models, no out-of-enum modality | +| `~/.gjc/agent/models.yml` | gajae's real `ModelsConfigSchema` **PASS**, 34 models, user `profiles:` preserved | + +The gjc merge appended our block to the existing document rather than replacing +it, so the user's `codex` profile survived. Prior copies are at +`/tmp/pi_models_before.json` and `/tmp/gjc_models_before.yml`. + +Both files now also carry the WP2 modality fix, so the `audio` value that broke +gjc's whole config is gone from the emitted output. + +## Follow-up owed + +`FOLLOWUP-FILECLIENT-01` (the six file clients' desired state, deferred from the +prior unit) now has a user-visible incident attached to it. The empty-container +defect is narrower and should be fixed in the writer regardless of when that +follow-up runs. From 3130356b338fd344319ff0c35c5058657be4d8a4 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Tue, 4 Aug 2026 10:20:43 +0900 Subject: [PATCH 028/163] =?UTF-8?q?docs(substrate):=20close=20audit=20roun?= =?UTF-8?q?d=201=20=E2=80=94=20a=20contract=20phase,=20because=20four=20pa?= =?UTF-8?q?rallel=20authors=20collided?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit FAIL, 7 High. Four of them are one defect: I dispatched four phase docs in parallel and each wrote a correct design for its own boundary, then they met. Two phases both declared ownership of integrations/codex.json with different required version-1 shapes; three each defined the /api/sync contract; the lock forbids awaitable work while history lives in a Worker, so the lock never serializes history at all; and 16 management callers were rewired to a direct commit helper that no phase's admission covers. That is the prior unit's defect at four times the scale, and I had written the fix into that unit's own synthesis: when phases share a contract, one phase owns it and the others consume it. So WP8b lands first and owns the record schema, the sync response, the convergence entry point, the generation counters and the module names. Three findings are real design errors rather than composition. mutatePersistedConfig documents that a writer ignoring the coordinator can change bytes after the final check, so recheck-under-lock does not linearize — it needs a generation verified before and after commit. My staleness guard hashes contents, which an A-to-B-to-A cycle passes, while C2 promised revision detection. And the lock namespace derives from homedir(), which reads HOME/USERPROFILE, so a service and a CLI for the same user can take different locks and defeat exclusion entirely; my tests set both consistently and would never have caught it. Also corrects a false claim the audit caught: this unit DOES ship a config switch. clientIntegrations.codex plus convergence that obeys it is a switch even without a setter or GUI, so the plan now says exactly what ships. Three technical bets survived independent verification: Bun Workers carry the proposed messages, a crashed lock holder does not wedge the OS lock, and refusing a missing CODEX_HOME does not break first run. --- .../260804_codex_write_substrate/000_plan.md | 62 ++++++--- .../006_audit_synthesis.md | 125 ++++++++++++++++++ 2 files changed, 170 insertions(+), 17 deletions(-) create mode 100644 devlog/_plan/260804_codex_write_substrate/006_audit_synthesis.md diff --git a/devlog/_plan/260804_codex_write_substrate/000_plan.md b/devlog/_plan/260804_codex_write_substrate/000_plan.md index 161136ff3..d877ce8cb 100644 --- a/devlog/_plan/260804_codex_write_substrate/000_plan.md +++ b/devlog/_plan/260804_codex_write_substrate/000_plan.md @@ -17,31 +17,44 @@ would freeze the proxy for every other client if a lock were held across it; and the ownership guard that was supposed to protect a foreign home fails open, after the artifacts it guards have already been created. -So this unit builds the substrate. **It ships no switch.** The switches -(`WP4`/`WP5` Codex, `WP6` Grok, `WP7` Desktop in the prior unit) become small -once it exists. +So this unit builds the substrate. -## The four parts, and why they are ordered this way +**What it ships, stated honestly** (audit #12 caught the earlier claim that this +unit "ships no switch", which was false): it ships the persisted config field +`clientIntegrations.codex` and the convergence semantics that obey it. It does +NOT ship the management setter or any GUI control. A user could set the field by +hand; nothing in the product offers it yet. The switch surfaces +(`WP5` Codex UI, `WP6` Grok, `WP7` Desktop in the prior unit) become small once +this exists. -Dependency order (PHASE-SPLIT-01), not effort. Each phase closes with something -independently verifiable. +## The phases + +**Re-planned after audit round 1** (`006_audit_synthesis.md`). The first map had +four parallel authors and no owner for the surfaces they share, so they collided +on four of them: the `integrations/codex.json` record, the `/api/sync` contract, +the convergence entry point, and the module name. That is the same defect the +prior unit had at smaller scale, and I reproduced it. + +So a contract phase lands first and owns every shared surface. The rest consume +it rather than inventing their share. | Phase | Doc | Delivers | Depends on | |---|---|---|---| -| WP9 | `010_catalog_seam.md` | `gatherCodexCatalogCandidate` / `commitCodexCatalogCandidate` + a typed outcome | — | -| WP10 | `020_history_isolation.md` | history off the server event loop, fail-fast under convergence | — | -| WP11 | `030_lock_protocol.md` | the async per-home lock with a hardened namespace | WP9, WP10 | +| WP8b | `005_contract.md` *(to write)* | The shared surfaces: record schema + owner, `/api/sync` response contract, the single convergence entry point, generation counters, module names, and the config-snapshot admission result | — | +| WP9 | `010_catalog_seam.md` | gather/commit split + typed outcome, consuming the contract | WP8b | +| WP10 | `020_history_isolation.md` | history off the event loop, and the cross-process history protocol | WP8b | +| WP11 | `030_lock_protocol.md` | the async per-home lock, per-USER namespace | WP8b, WP9, WP10 | | WP12 | `040_ownership_convergence.md` | tri-state authority, admission order, absence restoration | WP11 | +| WP13 | `050_composed_acceptance.md` *(to write)* | one acceptance suite against real production entry points | all | -WP9 and WP10 are genuinely independent: one makes catalog work *splittable*, the -other makes history work *non-blocking*. Neither needs a lock to be useful, and -both must exist before a lock is worth taking — a lock around an unsplittable -gather-and-write, or around a ten-second blocking history call, is the failure -the last unit already proved. +WP9 and WP10 remain independent of each other and both precede WP11: a lock +around an unsplittable gather-and-write, or around a ten-second blocking history +call, is the failure the last unit already proved. WP12 stays last of the four +because its admission must run before the lock module creates anything. -WP11 then has something bounded to wrap. WP12 sits last because the admission -order it defines must run *before* the lock module creates anything, so it needs -the lock's real construction sequence to point at. +WP13 exists because audit #11 showed the per-phase criteria are provable inside +their own phase and break at the seams — the "8000 green tests beside a broken +real file" class this plan already warns about. ## Research, all written this cycle @@ -99,6 +112,19 @@ remain `FOLLOWUP-FILECLIENT-01` from the prior unit. server without a restart. - C13 — typecheck, full test, gui lint, privacy scan green; no regression in the 8000-test suite. +- C14 — every one of the 16 management catalog callers goes through the single + convergence entry point; none can commit catalog bytes bypassing ownership, + provenance, intent or the lock (audit #2). +- C15 — history is serialized ACROSS processes, including the manifest and + rollout writes that happen outside its SQLite transaction + (`history-provider.ts:606,626`). Proven by two processes converging in + opposite directions, not by a same-process flight test (audit #1). +- C16 — one owner and one schema for `integrations/codex.json`; a record written + by any phase is readable by every other (audit #3). +- C17 — a config or catalog A→B→A cycle between gather and commit is DETECTED. + Content equality is not revision equality (audit #6). +- C18 — two processes for the same OS user with different `HOME`/`USERPROFILE` + take the SAME lock (audit #7). ## Risk register @@ -109,6 +135,8 @@ remain `FOLLOWUP-FILECLIENT-01` from the prior unit. | Deadlock against the config mutation lock | One stated ordering, plus a proof that no inverse nesting exists today | | Convergence deletes something the user owns | Provenance is a recorded baseline-absence plus post-image hash, never a filename or marker; on conflict, preservation wins and the operation reports rather than deletes | | Another round of divergence | One phase, one boundary, one audit. WP2 and WP3 of the prior unit passed clean on exactly that property; WP4 failed twice without it | +| Parallel authors collide on a shared surface | WP8b owns every shared surface and lands first; the rest consume it. Round 1 proved that dispatching four parallel authors without a contract owner reproduces the prior unit's defect at four times the scale | +| A criterion provable only inside its own phase | WP13 re-proves the composed system through real production entry points | ## What this unit does not claim diff --git a/devlog/_plan/260804_codex_write_substrate/006_audit_synthesis.md b/devlog/_plan/260804_codex_write_substrate/006_audit_synthesis.md new file mode 100644 index 000000000..ab5341f01 --- /dev/null +++ b/devlog/_plan/260804_codex_write_substrate/006_audit_synthesis.md @@ -0,0 +1,125 @@ +# Substrate audit round 1 — synthesis + +Verdict: **FAIL**, 7 High blocking, 5 Medium, 1 Low. Fresh reviewer, auditing the +whole unit as an implementer would apply it. + +## What the reviewer proved rather than accepted + +This round did unusually heavy independent verification, and three results +matter regardless of the findings: + +- **Bun Workers do what `020` assumes.** The reviewer ran a probe: TypeScript + ESM workers load, structured clone carries the proposed message shapes, and + the repo already uses Workers in storage restore/policy paths. Every field + crossing the boundary is plain data. +- **A crashed lock holder does NOT wedge the lock.** They ran the existing + abrupt-holder regression on Bun 1.3.14: the OS releases the SQLite transaction + on process death, so `030`'s no-stale-takeover decision is safe. +- **Refusing a missing `CODEX_HOME` does not break first run.** The native + injection path already rejects a missing `config.toml`, so no legitimate caller + depends on materializing a home. + +So the three riskiest *technical* bets in the unit survived. What failed is +composition. + +## The defect behind #1 through #4 + +I dispatched four phase docs in parallel and each one wrote a correct design for +its own boundary. Then they met. + +| Finding | The collision | +|---|---| +| #1 | `030`'s locked callback forbids awaitable work; `020` puts history in a Worker with only an in-process flight. So the lock never serializes history at all — and the real history path writes the manifest and rollout files OUTSIDE its SQLite transaction (`history-provider.ts:606,626`), so two processes going opposite directions can still corrupt each other | +| #2 | `010` rewires 16 management callers to a direct gather/commit helper; `040` replaces startup, ensure and `/api/sync` but never touches those 16. A provider edit therefore still commits catalog bytes with no ownership, provenance, intent or lock check | +| #3 | `020` and `040` both declare themselves owner of `integrations/codex.json`, each with its own required `version: 1` shape. A record written by one is malformed to the other | +| #4 | `010`, `020` and `040` each define the `/api/sync` contract, and `040`'s version drops `Retry-After`, bypasses `020`'s seam, and returns neither `catalogRefresh` nor `history` | + +This is the same failure the previous unit had — two docs redefining one union — +and I reproduced it at four times the scale. **The lesson I did not apply: when +phases share a contract, ONE phase must own it and the others consume it.** I +wrote that sentence into the previous unit's synthesis and then dispatched four +parallel authors with no shared-contract owner. + +## The three findings that are real design errors, not composition + +**#5 — "recheck under the lock" still does not linearize.** `mutatePersistedConfig` +says so itself (`config.ts:1855-1857`): a writer that ignores the coordinator can +change bytes after the final check, because the filesystem has no portable +conditional rename. My ordering releases the config lock before native +convergence, so A can recheck ON, B can persist OFF, and A still writes. Accept: +this needs a monotonic generation verified before AND after the native commit, +with a post-commit mismatch left unresolved and re-converged. + +**#6 — my staleness guard is a state comparison, not a revision guard.** Hashing +current config and base-catalog bytes passes an A→B→A cycle, and a parent symlink +can retarget while the textual path is unchanged. `010`'s own criterion (C2) says +"revision", and I implemented "contents". Accept: a monotonic catalog generation +plus stable target identity, not path strings. + +**#7 — the lock namespace is not per-user.** `homedir()` reads `HOME` / +`USERPROFILE` first, so a service and a CLI for the same user can land on +different lock databases and defeat exclusion entirely. My tests set both +consistently and would never have caught it. Accept: derive from OS-provided +effective-user identity. + +## Medium and Low + +All accepted. Two are worth naming: + +- **#12 — the unit ships a switch after claiming it does not.** `040` adds + `clientIntegrations.codex`, treats false as OFF, and makes convergence obey it. + That is a config-file switch even without a GUI. `000_plan.md` says "It ships no + switch", which is now false. Either say so plainly or move the boolean out. +- **#11 — the 13 criteria are not sufficient for the COMPOSED system.** Several + are provable only inside their own phase and break at the seams; C3 tests an + `/api/sync` seam that `040` removes. This is the "8000 green tests beside a + broken real file" class the plan itself warns about, so it needs a final + composed acceptance suite against real production entry points. + +## Disposition + +Thirteen findings, thirteen accepted, nothing rebutted. I verified #1's +out-of-transaction manifest write and #5's config caveat in the source myself +before accepting. + +## The correction + +Not another parallel round. The phase map gains a **contract phase** that lands +FIRST and owns every shared surface: + +- the `integrations/codex.json` record: one owner, one schema, provenance as an + extension-safe optional field +- the `/api/sync` response contract: one adapter carrying catalog disposition, + history state and desired/observed projection +- the module name (`codex-write-lock.ts`, per #13) +- the convergence entry point that all 16 management callers and the three + lifecycle paths funnel through + +Then WP9-WP12 CONSUME it instead of each inventing their share. Sequential, one +audit each, which is the property that made the two shipped phases of the +previous unit clean. + +`005_disable_leaves_a_broken_file.md` — the live incident found mid-audit — feeds +the same contract phase: the artifact inventory needs a baseline class for +containers a client requires to be non-empty. + +## Carried forward + +| Finding | Inherits | +|---|---| +| #1 history linearization | contract phase (protocol), WP10 (implementation) | +| #2 the 16 bypassing callers | contract phase (entry point), WP9 | +| #3 record ownership | contract phase — **it is the contract** | +| #4 `/api/sync` contract | contract phase | +| #5 config generation | contract phase (generation), WP12 | +| #6 catalog generation + target identity | contract phase (generation), WP9 | +| #7 per-user namespace | WP11 | +| #8 config snapshot reuse | contract phase (admission returns a snapshot) | +| #9 history retry dormancy | WP10 | +| #10 provenance recovery + ABA | WP12 | +| #11 composed acceptance suite | a final verification phase | +| #12 scope honesty | `000_plan.md`, now | +| #13 module name | contract phase | + +Prior-unit findings r1 #1 and r2 #4 remain open and are re-homed above rather +than being counted as housed. From 04f5ac7e95ad4b9a0a596655462c3eacaaf272e3 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Tue, 4 Aug 2026 10:22:09 +0900 Subject: [PATCH 029/163] docs(substrate): one owner for every surface four authors collided on MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The record gets a single owner and a schema where every section is OPTIONAL at v1, so a record written before a section existed is valid rather than malformed — that is what lets WP10 land before WP12 with no migration, and it is exactly what round 1 got wrong with two incompatible version-1 shapes. convergeCodex becomes the only way Codex-owned bytes are written, with a discriminated outcome instead of the bare catch that swallows everything today. Best-effort callers keep their 2xx; what changes is that the outcome is visible. An import-guard test keeps that true as callers are added. Two generation counters replace the content hash, because mutatePersistedConfig documents that a writer ignoring the coordinator can change bytes after the final check, and because a content hash passes an A-to-B-to-A cycle. Both are read before AND after the native commit: a post-commit mismatch is not converged, it is unresolved and re-converged. That is weaker than a transaction and says so. History gets its own cross-process lock held inside the Worker across the whole unit, because the real path writes the manifest and rollouts outside its SQLite transaction, so two processes corrupt each other through files SQLite never guarded. Ordering is native then history, never the inverse. And the baseline classes come straight from the live incident: a removal that empties a container the client requires is a third class, not the same as removing a member. --- .../005_contract.md | 247 ++++++++++++++++++ 1 file changed, 247 insertions(+) create mode 100644 devlog/_plan/260804_codex_write_substrate/005_contract.md diff --git a/devlog/_plan/260804_codex_write_substrate/005_contract.md b/devlog/_plan/260804_codex_write_substrate/005_contract.md new file mode 100644 index 000000000..d5d762223 --- /dev/null +++ b/devlog/_plan/260804_codex_write_substrate/005_contract.md @@ -0,0 +1,247 @@ +# WP8b — the shared surfaces, owned once + +Audit round 1 (`006_audit_synthesis.md`) failed four ways on one cause: four +phase docs, written in parallel, each invented its share of a surface they all +touch. Two claimed ownership of `integrations/codex.json` with incompatible +`version: 1` shapes. Three defined `/api/sync`. The lock forbade awaitable work +while history moved into a Worker, so the lock serialized nothing. And 16 +management callers were rewired to a helper no phase's admission covered. + +This phase lands first and owns all of it. WP9-WP12 consume; they do not extend. + +## IN / OUT + +IN: `src/codex/integration-record.ts` (NEW — the sole owner), +`src/codex/convergence.ts` (NEW — the single entry point), +`src/codex/generation.ts` (NEW), `src/server/management/context.ts` (MODIFY), +`tests/codex-integration-record.test.ts` (NEW), +`tests/codex-convergence-contract.test.ts` (NEW). + +OUT: every behavior. This phase defines shapes, names and one funnel; it moves no +catalog bytes, touches no history, takes no lock. That is deliberate — a contract +phase that also implements is a contract phase nobody can audit separately. + +## 1. The record: one owner, one schema + +`020` and `040` both wrote `integrations/codex.json` with a required `version: 1` +containing different fields, so a record from either is malformed to the other +(audit #3). + +```ts +/** + * The single durable record for the Codex integration. + * + * ONE owner. WP10 (history state) and WP12 (provenance) both write here, and + * both go through `updateIntegrationRecord` — never their own read/merge/write. + * Round 1 had two owners and two schemas for this exact file. + * + * Every section is OPTIONAL at v1. A record written before a section existed is + * VALID, not malformed: absence means "that subsystem has not spoken yet". This + * is what lets WP10 land before WP12 without a migration. + */ +export interface CodexIntegrationRecord { + version: 1; + history?: CodexHistoryState; + provenance?: CodexProvenanceLedger; + /** Bumped by every native commit. See §3. */ + generation?: number; +} +``` + +`updateIntegrationRecord(mutate)` does one read-modify-write under the same +coordinator the config uses, and **preserves unknown top-level keys verbatim** so +a newer version's record survives an older binary. + +Unreadable or unparseable is not "empty": it fails closed and the caller reports +rather than silently starting a fresh record. Losing provenance silently is how +`005_disable_leaves_a_broken_file.md` became possible. + +## 2. One convergence entry point + +Audit #2: `010` rewires 16 management callers to a direct gather/commit helper, +and `040` never touches them, so a provider edit commits catalog bytes with no +ownership, provenance, intent or lock check. Today that helper is +`refreshCodexCatalogBestEffort` (`src/server/management-api.ts:105-112`) and its +entire error handling is `catch { /* catalog absent */ }`. + +```ts +/** + * The ONLY way Codex-owned bytes are written. Startup, ensure, /api/sync, the + * CLI verbs and all 16 management mutation callbacks funnel here. + * + * The funnel is the point: admission, generation checks and the lock live in one + * place, so a new caller cannot forget them. Round 1's 16 callers each held + * their own path to a commit. + */ +export async function convergeCodex(request: ConvergeRequest): Promise; + +export interface ConvergeRequest { + /** What the caller wants. `observe` writes nothing and is the status read. */ + intent: "apply" | "remove" | "observe"; + /** Why, for the record and for log attribution. */ + reason: "startup" | "ensure" | "api-sync" | "cli" | "management-mutation"; + /** Automatic callers fail fast and defer; explicit ones may wait. See §5. */ + mode: "automatic" | "explicit"; + deadlineMs: number; +} +``` + +`ConvergeOutcome` is a discriminated union, never a thrown exception for an +expected condition: + +```ts +export type ConvergeOutcome = + | { kind: "converged"; changed: boolean; observed: CodexObservedState; generation: number } + | { kind: "skipped"; reason: "desired-off" | "already-converged"; observed: CodexObservedState } + | { kind: "refused"; authority: "service-home" | "external-provider" | "journal" | "provenance"; message: string } + | { kind: "busy"; surface: "lock" | "history" | "config"; retryAfterMs: number } + | { kind: "deferred"; unresolved: readonly ("history")[]; observed: CodexObservedState } + | { kind: "failed"; surface: string; message: string }; +``` + +**Best-effort callers stay best-effort.** The 16 management callbacks keep their +2xx and report the outcome in a `catalogRefresh` field; they do not start +failing loudly because a catalog refresh deferred. What changes is that the +outcome is *visible* instead of swallowed by a bare `catch`. + +## 3. Generations, because content equality is not revision equality + +Audit #5 and #6. `mutatePersistedConfig` documents its own limit +(`src/config.ts:1855-1857`): + +> A writer that ignores the coordinator can still change bytes after the final +> check because the filesystem has no portable conditional rename. + +And a content hash passes an A→B→A cycle, which may still have moved the cache, +the backup or the provenance ledger. + +So two monotonic counters, both in the record from §1: + +- **`generation`** — bumped by every cooperating native commit. +- the config's own revision, read as part of the admission snapshot in §4. + +The rule, and it is the whole point of the counters: + +> Read both immediately before the native commit, and **again immediately +> after**. A post-commit mismatch means somebody wrote underneath us: the +> outcome is NOT `converged`, the record is left `unresolved`, and convergence +> re-runs. We never claim a commit we cannot prove was the last one. + +That is weaker than a transaction and it is honest about it: we detect +interference rather than prevent it. + +**Target identity, not path strings.** A candidate records the canonical parent +directory and the file identity (dev+inode where available) of each target, not +the textual path — a parent symlink can retarget while the path string is +unchanged, and `atomicWriteFile` only resolves the effective target at commit +time (`src/config.ts:190-199`). + +## 4. Admission returns a snapshot, not a boolean + +Audit #8: `040`'s intent reader returns ON/OFF while `010`'s gather needs a full +`OcxConfig`, so either gather uses the stale server object or the claimed +"two reads" is wrong. + +```ts +export interface AdmissionSnapshot { + config: Readonly; + configDigest: string; + intent: "on" | "off"; + generation: number; + ownership: "owned" | "foreign" | "unknown"; +} +``` + +One read produces all of it. Gather consumes `config` — **that exact object**, +never a re-read and never the server's long-lived one. The under-lock recheck +compares `configDigest` and `generation` rather than reading the config again. + +Read count per mutation: **one** before gather, **two** cheap counter reads +around the commit. `010`'s independent `readConfigDiagnostics()` call is removed. + +## 5. `/api/sync`, defined once + +Audit #4: three phases defined this route and the last one dropped `Retry-After` +and both payload fields. + +| `ConvergeOutcome` | Status | Body | +|---|---|---| +| `converged` | 200 | `{ ok: true, changed, observed, catalogRefresh, history }` | +| `skipped` (`desired-off`) | 409 | `{ ok: false, reason: "desired-off", observed }` | +| `skipped` (`already-converged`) | 200 | `{ ok: true, changed: false, observed }` | +| `refused` | 409 | `{ ok: false, authority, message, observed }` | +| `busy` | 503 + `Retry-After` | `{ ok: false, surface, retryAfterMs }` | +| `deferred` | 200 | `{ ok: true, changed, unresolved, observed }` | +| `failed` | 500 | `{ error: message, surface }` | + +`busy` is 503 with `Retry-After` because it is transient and the client should +retry; `refused` and `desired-off` are 409 because retrying changes nothing until +a human acts. `deferred` is 200 because the requested work DID happen — history +is outstanding and named, not failed. + +## 6. History is serialized across processes, not just in one + +Audit #1, the finding with no home. `030`'s locked callback is synchronous and +forbids awaitable work; `020` puts history in a Worker with an in-process flight. +So the lock never covers history — and the real path writes the backup manifest +and rollout files OUTSIDE its SQLite transaction +(`src/codex/history-provider.ts:606,626`), so two processes converging in +opposite directions corrupt each other through files SQLite never guarded. + +The contract: **history has its own cross-process lock, acquired INSIDE the +Worker, held across the whole history unit** — manifest read, rollout writes and +the DB transaction together. It is a sibling of the §3 native lock, not nested +inside it, precisely because the native lock's section must stay synchronous. + +Ordering, to make the absence of deadlock checkable: **native lock → history +lock, never the inverse.** The native section releases before the Worker is +asked for history. + +## 7. Names + +Audit #13. Fixed here so no phase invents a variant: + +| Thing | Module | +|---|---| +| the native write lock | `src/codex/codex-write-lock.ts` | +| the record | `src/codex/integration-record.ts` | +| the entry point | `src/codex/convergence.ts` | +| generations | `src/codex/generation.ts` | +| history worker | `src/codex/history-worker.ts` | + +## 8. Baseline classes, from a live incident + +`005_disable_leaves_a_broken_file.md`: a disable left `~/.pi/agent/models.json` +as `{}`, which violates Pi's schema because `providers` is required. The client +refused to start. + +So a provenance baseline is one of three, and a remover must know which: + +| Class | On removal | +|---|---| +| `absent` | delete, if the post-image hash still matches | +| `present` | restore the recorded baseline bytes | +| `present-required-nonempty` | restore to a VALID minimum for that client — never a bare `{}` unless `{}` is what preceded us | + +## Test plan + +`tests/codex-integration-record.test.ts`: a v1 record with only `history` is +valid to a provenance reader and vice versa (audit #3); unknown top-level keys +survive a write; unparseable fails closed rather than resetting. + +`tests/codex-convergence-contract.test.ts`: every `ConvergeOutcome` variant maps +to the §5 row, `busy` carries `Retry-After`, and a best-effort management caller +still returns 2xx while reporting a non-converged disposition. + +A grep-level guard test: no module outside `convergence.ts` imports the catalog +commit or the remover directly. That is how C14 stays true as callers are added. + +## Accept criteria + +- C14 — all 16 management callers funnel through `convergeCodex`, enforced by the + import guard test. +- C16 — one owner, one schema; a record from any phase reads in every other. +- C17 — an A→B→A cycle between gather and commit is detected by generation, and + a parent-symlink retarget is detected by target identity. +- Contributes to C15 (the history protocol is specified here, implemented in + WP10) and to C2/C12 (generations and the admission snapshot). From d7859bdb134f4cafd788d850e550fa87101400cf Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Tue, 4 Aug 2026 10:41:27 +0900 Subject: [PATCH 030/163] =?UTF-8?q?docs(substrate):=20round=202=20?= =?UTF-8?q?=E2=80=94=20I=20declared=20ownership=20without=20transferring?= =?UTF-8?q?=20it?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit One closed, eleven open, five new. The cause is nameable: 005 says it owns the record, the route and the entry point, and 020/030/040 still contain their own versions because I never rewrote them. The reviewer's list of contradicting sections runs thirty-odd entries. A contract nobody collected is a fifth opinion, which is round 1's defect one level up. The most valuable finding came from the reviewer EXECUTING a claim rather than reading it. I had accepted their round-1 fix of using os.userInfo().homedir instead of homedir() for the lock namespace; they ran it on our pinned Bun 1.3.14 and both returned the fake HOME. I reproduced it. But the same probe shows the way out — uid and username are real — so the namespace keys on effective-user identity, uid on POSIX and account SID on Windows, never a home path. Three new High findings accepted. The request let the CALLER choose apply vs remove, so /api/sync while OFF could skip instead of removing residue; it now carries converge|observe and the direction comes from admitted intent. WP8b as written cannot land first because it declares a runtime entry point while being OUT of every behavior. And the generation counter would treat its own successful commit as interference, since a bump-on-every-commit counter always mismatches after a write — it needs an expected N-to-N+1 transition with a transaction id. Also reverses my own scope creep: present-required-nonempty came from the live Pi incident and belongs in FOLLOWUP-FILECLIENT-01, not in a Codex unit. Housing a finding in the wrong unit is not housing it. The correction is to collapse the four phase docs into the contract rather than run a third round of parallel edits, because two rounds have now shown four docs cannot be kept consistent by review alone. --- .../007_audit_synthesis_r2.md | 130 ++++++++++++++++++ 1 file changed, 130 insertions(+) create mode 100644 devlog/_plan/260804_codex_write_substrate/007_audit_synthesis_r2.md diff --git a/devlog/_plan/260804_codex_write_substrate/007_audit_synthesis_r2.md b/devlog/_plan/260804_codex_write_substrate/007_audit_synthesis_r2.md new file mode 100644 index 000000000..7894d25f0 --- /dev/null +++ b/devlog/_plan/260804_codex_write_substrate/007_audit_synthesis_r2.md @@ -0,0 +1,130 @@ +# Substrate audit round 2 — synthesis + +Verdict: **FAIL**. One closed (#12), eleven still open, five new (three High). + +## The honest read of this round + +I wrote a contract phase and it did not do the job, for a reason worth naming: +**I declared ownership without transferring it.** `005` says it owns the record, +the route and the entry point — and `020`, `030`, `040` still contain their own +versions, because I never rewrote them. The reviewer's "PHASE DOCS THAT MUST +CHANGE" section is 30-odd concrete sections long. Declaring a contract and +leaving three docs that contradict it is not a contract; it is a fifth opinion. + +That is the same shape as round 1, one level up. Round 1: four authors, no owner. +Round 2: an owner who did not collect. + +## The finding that changes a design decision + +**#7 — the per-user namespace.** I accepted the reviewer's round-1 fix +(`os.userInfo().homedir` instead of `homedir()`) and specified it. The reviewer +then **ran it on our pinned Bun 1.3.14** and both returned the fake `HOME`. I +reproduced it: + +``` +HOME=/tmp/fakehome bun -e '...' +homedir: /tmp/fakehome +userInfo: /tmp/fakehome +uid: 501 username: jun +``` + +So the fix I wrote does not work in the runtime we ship on. But the probe also +shows the way out: **`uid` and `username` are real**. The namespace must be keyed +on effective-user identity — uid on POSIX, account SID on Windows — not on any +home path, since every home path in this runtime is environment-controlled. + +This is the single most valuable thing either round produced, and it only +surfaced because the reviewer executed the claim instead of reading it. + +## The three new High findings, all accepted + +**N1 — the caller picks the direction.** `ConvergeRequest.intent` accepts +`apply | remove`, which lets `/api/sync` skip while OFF instead of removing +residue, violating C11 — and it contradicts `040`, which says callers cannot +supply desired state. Fix: the request carries `converge | observe` only, and +the direction is derived from admitted persisted intent. The caller says *when*, +never *which way*. + +**N2 — WP8b cannot land first as written.** It is "OUT: every behavior" yet +declares a runtime entry point and references types later phases define. A +throwing placeholder is not a safe first commit, and a compatibility shim would +bypass the very authority it establishes. Fix: WP8b becomes a complete +types/validators/adapter phase that rewires nothing, OR it lands the whole safe +funnel. Either way **every phase must typecheck and preserve behavior at its own +commit** — that is what "one phase, one boundary" has to mean operationally. + +**N3 — the generation protocol treats its own commit as interference.** A +counter bumped by every native commit, compared before and after, always +mismatches after a successful write. I specified a mechanism whose success +condition is indistinguishable from its failure condition. Fix: an expected +transition — `N → N+1 by us`, identified by a transaction id — with explicit +bump ownership and crash ordering. + +## #5, restated because I got the direction wrong + +C2 says a stale candidate cannot be committed. I specified detect-after-commit +and promised re-convergence, which permits exactly the write C2 forbids. The +reviewer points at `030`'s own text: the native lock may hold the config +mutation lock through the synchronous re-read and commit. So **prevention is +available for cooperating writers**, and post-commit detection is only for +writers that ignore the coordinator. Accept, with the retry bounded by +`deadlineMs` and a typed unresolved reason when it expires. + +## #4 — scope creep I introduced + +`present-required-nonempty` came from the live Pi incident and names no baseline +bytes, no client schema and no validator, while the six file clients are +explicitly out of scope. The reviewer is right: for Codex, `present` plus exact +baseline bytes already expresses restoration. The class goes back to +`FOLLOWUP-FILECLIENT-01` where the incident belongs. + +I added it because the incident was fresh and I wanted it housed. Housing a +finding in the wrong unit is not housing it. + +## Disposition + +Sixteen open items, all accepted, nothing rebutted. I verified the Bun probe +myself before accepting #7. + +## The correction, and why it is not another rewrite round + +Two audit rounds have now failed on the same axis: documents that disagree with +each other. The fix is not a third round of parallel edits — it is to **collapse +the four phase docs into the contract**, because the audit has demonstrated that +four docs cannot be kept consistent by review alone. + +Concretely: + +1. `005_contract.md` absorbs every shared surface **completely** — full section + types, the exhaustive outcome union, the single adapter, generations with + expected transitions, the uid/SID namespace, and the corrected + `converge | observe` request. +2. `010`, `020`, `030`, `040` are rewritten as **consumers**: each keeps only its + own mechanism and imports everything shared. The reviewer's section list is + the checklist. +3. Each phase must typecheck and preserve behavior at its own commit (N2). +4. `050_composed_acceptance.md` is written before implementation, not after. + +That is a real amount of work, and it is smaller than shipping a substrate whose +four documents contradict each other in thirty places. + +## Carried forward + +| Finding | Disposition | +|---|---| +| #1 history overtaking + CLI inline | contract: one history lock, expected-transition rejection, all callers | +| #2 funnel not provable by grep | contract: writers move to an internal module, reachability-enforced | +| #3 three schemas | contract: complete section types; `020`/`040` import them | +| #4 route mapped three times | contract: one exhaustive adapter; remove from all three | +| #5 detection vs prevention | contract: hold config lock through commit for cooperating writers | +| #6 one counter, two jobs | contract: separate config and native generations, expected transitions | +| #7 namespace | **uid/SID**, proven necessary by the Bun probe | +| #8 digest with nothing to compare | contract: authoritative re-read inside the commit; withdraw the one-read claim | +| #9 60-tick dormancy | `020`: capped backoff that never becomes permanent | +| #10 provenance recovery + ABA | `040`: operator adoption path; narrow C10 to current-byte drift | +| #11 WP13 placeholder | write `050` before implementation | +| #12 scope honesty | **closed** | +| #13 module names | `040` still names two wrong modules | +| N1 caller-chosen direction | contract: `converge \| observe` | +| N2 WP8b cannot land first | contract: types/validators/adapter only, or the whole funnel | +| N3 self-interference | contract: expected transition with a transaction id | From 2d5e080dea3e7000bf2111b381c7c1a3c4f5fb11 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Tue, 4 Aug 2026 10:43:45 +0900 Subject: [PATCH 031/163] docs(substrate): make the contract complete, so it can actually be collected MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Round 2 said I declared ownership without transferring it. This version defines every shared surface fully rather than gesturing at it. The namespace change is the one proven necessary by running code: Bun 1.3.14 returns an environment-controlled home from BOTH os.homedir() and os.userInfo().homedir, so the fix I accepted in round 1 does not work where we ship. The same probe shows uid and username are real, so the lock keys on effective-user identity — uid on POSIX, SID on Windows — and never on a home path. The test that matters is two children with different HOME values taking the same lock, which my first version could not have failed. Generations become an expected transition with a transaction id, because a counter bumped by every commit and compared before/after always mismatches after a successful write — I had specified a mechanism whose success condition was indistinguishable from failure. Prevention now covers cooperating writers by holding the config lock through the commit, since C2 says a stale candidate cannot be committed and detect-after-commit permits exactly that write. History carries the commit expectation so an overtaken job is rejected before mutation rather than racing at scheduling, and every caller including explicit CLI takes the lock — one a caller can skip is not a lock. The funnel is proven by module-graph reachability instead of grep, since the tree already has wrappers, re-exports and dynamic imports that grep misses. And present-required-nonempty is removed: it came from the Pi incident and belongs to FOLLOWUP-FILECLIENT-01, not a Codex unit. --- .../005_contract.md | 351 ++++++++++++++---- 1 file changed, 280 insertions(+), 71 deletions(-) diff --git a/devlog/_plan/260804_codex_write_substrate/005_contract.md b/devlog/_plan/260804_codex_write_substrate/005_contract.md index d5d762223..1de08e910 100644 --- a/devlog/_plan/260804_codex_write_substrate/005_contract.md +++ b/devlog/_plan/260804_codex_write_substrate/005_contract.md @@ -2,24 +2,44 @@ Audit round 1 (`006_audit_synthesis.md`) failed four ways on one cause: four phase docs, written in parallel, each invented its share of a surface they all -touch. Two claimed ownership of `integrations/codex.json` with incompatible -`version: 1` shapes. Three defined `/api/sync`. The lock forbade awaitable work -while history moved into a Worker, so the lock serialized nothing. And 16 -management callers were rewired to a helper no phase's admission covered. +touch. Round 2 (`007_audit_synthesis_r2.md`) failed because the first version of +this document **declared** ownership without **transferring** it — `020`, `030` +and `040` still carried their own record schema, their own route mapping and +their own module names, in roughly thirty places. -This phase lands first and owns all of it. WP9-WP12 consume; they do not extend. +So this document is now the complete definition of every shared surface, and the +four phase docs are rewritten as consumers against the reviewer's section list. +A contract nobody collected is a fifth opinion. ## IN / OUT -IN: `src/codex/integration-record.ts` (NEW — the sole owner), +IN: `src/codex/integration-record.ts` (NEW — sole owner of the record), `src/codex/convergence.ts` (NEW — the single entry point), -`src/codex/generation.ts` (NEW), `src/server/management/context.ts` (MODIFY), +`src/codex/convergence-types.ts` (NEW — every shared type), +`src/codex/generation.ts` (NEW), `src/codex/user-identity.ts` (NEW — §7), +`src/server/management/sync-response.ts` (NEW — the one adapter), `tests/codex-integration-record.test.ts` (NEW), -`tests/codex-convergence-contract.test.ts` (NEW). +`tests/codex-convergence-contract.test.ts` (NEW), +`tests/codex-user-identity.test.ts` (NEW). -OUT: every behavior. This phase defines shapes, names and one funnel; it moves no -catalog bytes, touches no history, takes no lock. That is deliberate — a contract -phase that also implements is a contract phase nobody can audit separately. +OUT: catalog mechanics (WP9), history mechanics (WP10), lock mechanics (WP11), +ownership mechanics (WP12). This phase owns *shapes and the funnel*, not the +work inside them. + +### What "lands first" has to mean (round 2 N2) + +The reviewer showed the previous version could not land: it was "OUT: every +behavior" while declaring a runtime `convergeCodex`, and a throwing placeholder +is not a safe commit. + +So WP8b lands **types, validators, the record owner, the identity resolver and +the response adapter — and rewires nothing.** `convergeCodex` is declared here +as a type only; WP9 supplies its first real implementation and rewires the +catalog callers at that commit. + +**Invariant for every phase in this unit:** each phase typechecks and preserves +behavior at its own commit. No phase may leave a placeholder that a later phase +is required to replace before the tree is correct. ## 1. The record: one owner, one schema @@ -48,9 +68,45 @@ export interface CodexIntegrationRecord { } ``` +### The section types, defined HERE (round 2 #3) + +The first version referenced `CodexHistoryState` and `CodexProvenanceLedger` +without defining them, so `020` and `040` kept their own. Both live in +`convergence-types.ts` and both phases import them: + +```ts +export interface CodexHistoryState { + status: "converged" | "pending" | "running" | "blocked" | "unknown"; + /** Why it is not converged, when it is not. */ + reason?: "db-busy" | "permission" | "worker-died" | "overtaken"; + attempts: number; + /** null means "no timer armed"; see 020 — it must never mean "never again". */ + nextRetryAt: string | null; + /** The transition this state belongs to, so an overtaken job is detectable. */ + txId: string | null; + /** Unknown keys from a newer writer, preserved verbatim. */ + readonly [extra: string]: unknown; +} + +export interface CodexProvenanceEntry { + artifact: CodexArtifactId; + baseline: { kind: "absent" } | { kind: "present"; sha256: string }; + /** Hash of what WE wrote. null when the write did not complete. */ + postImage: string | null; + txId: string; + at: string; +} + +export interface CodexProvenanceLedger { + entries: readonly CodexProvenanceEntry[]; + readonly [extra: string]: unknown; +} +``` + `updateIntegrationRecord(mutate)` does one read-modify-write under the same -coordinator the config uses, and **preserves unknown top-level keys verbatim** so -a newer version's record survives an older binary. +coordinator the config uses, and **preserves unknown keys verbatim at every +level** — top-level and inside each section — so a newer version's record +survives an older binary. Unreadable or unparseable is not "empty": it fails closed and the caller reports rather than silently starting a fresh record. Losing provenance silently is how @@ -76,8 +132,17 @@ entire error handling is `catch { /* catalog absent */ }`. export async function convergeCodex(request: ConvergeRequest): Promise; export interface ConvergeRequest { - /** What the caller wants. `observe` writes nothing and is the status read. */ - intent: "apply" | "remove" | "observe"; + /** + * The caller says WHEN, never WHICH WAY. + * + * Round 2 N1: an `apply | remove` request let `/api/sync` skip while desired + * state was OFF instead of removing residue, which violates C11 and + * contradicts the rule that callers cannot supply desired state. The + * direction is derived from admitted persisted intent, full stop. + * + * `observe` writes nothing and is the status read. + */ + action: "converge" | "observe"; /** Why, for the record and for log attribution. */ reason: "startup" | "ensure" | "api-sync" | "cli" | "management-mutation"; /** Automatic callers fail fast and defer; explicit ones may wait. See §5. */ @@ -91,12 +156,25 @@ expected condition: ```ts export type ConvergeOutcome = - | { kind: "converged"; changed: boolean; observed: CodexObservedState; generation: number } - | { kind: "skipped"; reason: "desired-off" | "already-converged"; observed: CodexObservedState } - | { kind: "refused"; authority: "service-home" | "external-provider" | "journal" | "provenance"; message: string } + | { kind: "converged"; direction: "applied" | "removed"; changed: boolean; + observed: CodexObservedState; generation: number; + catalogRefresh: CatalogDisposition; history: CodexHistoryState } + | { kind: "skipped"; reason: "already-converged"; + observed: CodexObservedState; catalogRefresh: CatalogDisposition; history: CodexHistoryState } + | { kind: "refused"; authority: "service-home" | "external-provider" | "journal" | "provenance"; + message: string; observed: CodexObservedState } | { kind: "busy"; surface: "lock" | "history" | "config"; retryAfterMs: number } - | { kind: "deferred"; unresolved: readonly ("history")[]; observed: CodexObservedState } + | { kind: "deferred"; direction: "applied" | "removed"; changed: boolean; + unresolved: readonly UnresolvedSurface[]; + observed: CodexObservedState; catalogRefresh: CatalogDisposition; history: CodexHistoryState } | { kind: "failed"; surface: string; message: string }; + +/** + * Note what is NOT here: `desired-off`. Desired OFF is not a skip — it is a + * `converged` with `direction: "removed"`. That is round 2 N1: the old shape let + * a sync while OFF return "skipped" and leave routed residue on disk. + */ +export type UnresolvedSurface = "history"; ``` **Best-effort callers stay best-effort.** The 16 management callbacks keep their @@ -104,37 +182,88 @@ export type ConvergeOutcome = failing loudly because a catalog refresh deferred. What changes is that the outcome is *visible* instead of swallowed by a bare `catch`. -## 3. Generations, because content equality is not revision equality +## 3. Generations: an expected transition, not a bare counter + +Round 1 #5/#6 and round 2 N3. Three separate defects lived here. -Audit #5 and #6. `mutatePersistedConfig` documents its own limit -(`src/config.ts:1855-1857`): +`mutatePersistedConfig` documents its own limit (`src/config.ts:1855-1857`): > A writer that ignores the coordinator can still change bytes after the final > check because the filesystem has no portable conditional rename. -And a content hash passes an A→B→A cycle, which may still have moved the cache, -the backup or the provenance ledger. +A content hash passes an A→B→A cycle. And my first counter was **bumped by every +native commit and compared before/after** — so a successful write always +mismatched. I specified a mechanism whose success condition was +indistinguishable from its failure condition. -So two monotonic counters, both in the record from §1: +### Two counters, not one + +```ts +/** Bumped by every cooperating CONFIG write. Owned by src/config.ts. */ +export interface ConfigGeneration { readonly value: number; } -- **`generation`** — bumped by every cooperating native commit. -- the config's own revision, read as part of the admission snapshot in §4. +/** Bumped by every cooperating NATIVE commit. Owned by convergence.ts. */ +export interface NativeGeneration { readonly value: number; } +``` -The rule, and it is the whole point of the counters: +Round 2 #6: the previous version said "two counters, both in the record" and +then defined one. They are distinct because they answer different questions — +did the user's configuration move, versus did somebody else write Codex's files. -> Read both immediately before the native commit, and **again immediately -> after**. A post-commit mismatch means somebody wrote underneath us: the -> outcome is NOT `converged`, the record is left `unresolved`, and convergence -> re-runs. We never claim a commit we cannot prove was the last one. +### The expected transition -That is weaker than a transaction and it is honest about it: we detect -interference rather than prevent it. +```ts +export interface CommitExpectation { + /** Read at admission. */ + readonly nativeBefore: number; + /** What OUR commit will produce. Always nativeBefore + 1. */ + readonly nativeAfter: number; + /** Identifies the commit that performed the bump. */ + readonly txId: string; +} +``` -**Target identity, not path strings.** A candidate records the canonical parent -directory and the file identity (dev+inode where available) of each target, not -the textual path — a parent symlink can retarget while the path string is -unchanged, and `atomicWriteFile` only resolves the effective target at commit -time (`src/config.ts:190-199`). +The rule, stated so a test can check it: + +> After the commit, the record must show **exactly** `nativeAfter` AND `txId` +> equal to ours. `nativeAfter` with a different `txId` is another writer that +> raced us to the same number. Anything else is interference: the outcome is +> `deferred` with the surface named, never `converged`. + +The bump is written **inside** the same synchronous section as the commit, by +the committer, so there is no window where the files moved and the counter did +not. On crash between file write and counter bump, the next convergence sees a +stale counter and re-converges — which is safe because convergence is idempotent +by construction. + +### Prevention for cooperating writers (round 2 #5) + +C2 says a stale candidate **cannot be committed**. Detect-after-commit permits +exactly the write C2 forbids, and `030` already allows the fix: the native lock +may hold the config mutation lock across the synchronous re-read and commit. + +So: + +| Writer | Mechanism | +|---|---| +| cooperating (ours) | **prevented** — config lock held through re-read and commit | +| non-cooperating (hand edit, foreign tool) | **detected** after the fact, reported `deferred` | + +Re-gather is bounded by `deadlineMs`. On expiry the outcome is `deferred` with a +typed reason and another convergence is scheduled — the retry loop terminates on +a deadline, not on hope (round 1 #5's missing termination rule). + +### Target identity, honestly bounded + +A candidate records the canonical parent directory and the file identity +(dev+inode where available) of each target, not the textual path — a parent +symlink can retarget while the path string is unchanged, and `atomicWriteFile` +resolves the effective target only at commit (`src/config.ts:190-199`). + +**What this does not do** (round 2 #6): it cannot detect a parent-symlink A→B→A +that happens entirely between two checks. C17 is therefore scoped to *cooperating +transitions and single-direction drift*, not to arbitrary filesystem ABA. Claiming +otherwise would be a promise the filesystem does not offer. ## 4. Admission returns a snapshot, not a boolean @@ -167,37 +296,107 @@ and both payload fields. | `ConvergeOutcome` | Status | Body | |---|---|---| | `converged` | 200 | `{ ok: true, changed, observed, catalogRefresh, history }` | -| `skipped` (`desired-off`) | 409 | `{ ok: false, reason: "desired-off", observed }` | -| `skipped` (`already-converged`) | 200 | `{ ok: true, changed: false, observed }` | +| `skipped` (`already-converged`) | 200 | `{ ok: true, changed: false, observed, catalogRefresh, history }` | | `refused` | 409 | `{ ok: false, authority, message, observed }` | | `busy` | 503 + `Retry-After` | `{ ok: false, surface, retryAfterMs }` | | `deferred` | 200 | `{ ok: true, changed, unresolved, observed }` | | `failed` | 500 | `{ error: message, surface }` | `busy` is 503 with `Retry-After` because it is transient and the client should -retry; `refused` and `desired-off` are 409 because retrying changes nothing until -a human acts. `deferred` is 200 because the requested work DID happen — history -is outstanding and named, not failed. +retry; `refused` is 409 because retrying changes nothing until a human acts. +`deferred` is 200 because the requested work DID happen — history is outstanding +and named, not failed. + +There is no `desired-off` row, per §2: a converge while OFF removes and returns +`converged { direction: "removed" }`. + +**One adapter, one place.** `src/server/management/sync-response.ts` exports a +single exhaustive `toSyncResponse(outcome): Response`. `010`, `020` and `040` +each mapped this route themselves (round 1 #4, still open in round 2); none of +them may now. The exhaustiveness is enforced by a `never` check on the union, so +adding an outcome variant without a row fails typecheck. + +## 6. History: one lock, and no overtaking + +Round 1 #1 had no home; round 2 showed my first answer had two holes. + +The real apply path writes manifest → rollouts → DB +(`history-provider.ts:606,611,626`); restore writes rollouts → DB → manifest +deletion → a second ejection (`:657,667,677,691`). SQLite guards only one of +those steps, so two processes corrupt each other through the files it never sees. + +**One cross-process history lock**, acquired inside the Worker, held across the +entire unit — manifest, rollouts and the DB transaction together, including the +final post-probe. It is a sibling of the native lock, not nested, because the +native section must stay synchronous. + +Two things round 2 caught: + +**Explicit CLI history still ran inline** (`020:216-219,868-870`), outside any +lock. Every history caller takes this lock — server, CLI, startup, retry. A lock +one caller can skip is not a lock. + +**Sibling locks permit overtaking.** A releases the native lock after committing +ON; B commits native OFF; B's history removal can then run before A's history +apply, leaving native OFF with history ON. So the history job carries the +`CommitExpectation` from §3: + +> A history job whose `nativeBefore` no longer matches the record has been +> overtaken. It is **rejected before any mutation**, and the winning transition +> converges history itself. Overtaking is detected at the point of work, not +> raced at the point of scheduling. + +That also bounds the livelock the reviewer raised: a rejected job does not retry +into the same race, it defers to the newer transition. -## 6. History is serialized across processes, not just in one +Ordering, so absence of deadlock is checkable: **native lock → history lock, +never the inverse**, and they are never held simultaneously. -Audit #1, the finding with no home. `030`'s locked callback is synchronous and -forbids awaitable work; `020` puts history in a Worker with an in-process flight. -So the lock never covers history — and the real path writes the backup manifest -and rollout files OUTSIDE its SQLite transaction -(`src/codex/history-provider.ts:606,626`), so two processes converging in -opposite directions corrupt each other through files SQLite never guarded. +## 7. The lock namespace keys on effective user, not on any home path -The contract: **history has its own cross-process lock, acquired INSIDE the -Worker, held across the whole history unit** — manifest read, rollout writes and -the DB transaction together. It is a sibling of the §3 native lock, not nested -inside it, precisely because the native lock's section must stay synchronous. +Round 1 #7 said `homedir()` reads `HOME`/`USERPROFILE`, so a service and a CLI +for the same user can take different locks. I accepted the fix — use +`os.userInfo().homedir` — and specified it. -Ordering, to make the absence of deadlock checkable: **native lock → history -lock, never the inverse.** The native section releases before the Worker is -asked for history. +The reviewer then **ran it on our pinned Bun 1.3.14**, and I reproduced the run: -## 7. Names +``` +HOME=/tmp/fakehome bun -e '...' +homedir: /tmp/fakehome +userInfo: /tmp/fakehome +uid: 501 username: jun +``` + +Both home accessors return the fake environment path in this runtime. The +accepted fix does not work where we ship. + +But the same probe shows the way out: **`uid` and `username` are real.** So the +coordination namespace keys on effective-user IDENTITY, never on a home path: + +```ts +/** + * Effective-user identity for the lock namespace. + * + * NOT a home path. Bun 1.3.14 returns an environment-controlled home from both + * os.homedir() AND os.userInfo().homedir, so any home-derived namespace can be + * split by a service and a CLI that see different HOME values — which defeats + * exclusion entirely, silently. + */ +export type UserIdentity = + | { platform: "posix"; uid: number } + | { platform: "win32"; sid: string }; +``` + +The lock path is then +`/opencodex/native-write-locks/v1//.sqlite`, +with the per-user directory created mode `0700` and validated by `lstat` before +use — a symlink or a wrong owner is a refusal, never a trust. + +The test that matters, and the one my first version could not have failed: two +child processes with **different** `HOME`/`USERPROFILE` values must take the +**same** lock. + +## 8. Names Audit #13. Fixed here so no phase invents a variant: @@ -209,19 +408,21 @@ Audit #13. Fixed here so no phase invents a variant: | generations | `src/codex/generation.ts` | | history worker | `src/codex/history-worker.ts` | -## 8. Baseline classes, from a live incident +## 9. Baseline classes -`005_disable_leaves_a_broken_file.md`: a disable left `~/.pi/agent/models.json` -as `{}`, which violates Pi's schema because `providers` is required. The client -refused to start. +Two, not three. A provenance baseline is `absent` or `present`, and `present` +carries the exact baseline bytes — which already expresses restoration for every +Codex artifact. -So a provenance baseline is one of three, and a remover must know which: +The `present-required-nonempty` class I added in the first version is **removed** +(round 2 #4). It came from the live Pi incident in +`005_disable_leaves_a_broken_file.md`, where a disable left `models.json` as `{}` +and violated Pi's required-`providers` schema. That is real, and it belongs to +`FOLLOWUP-FILECLIENT-01` with the rest of the six file clients — which this unit +lists as out of scope. It named no baseline bytes, no client schema and no +validator, because a Codex unit has nowhere to get them. -| Class | On removal | -|---|---| -| `absent` | delete, if the post-image hash still matches | -| `present` | restore the recorded baseline bytes | -| `present-required-nonempty` | restore to a VALID minimum for that client — never a bare `{}` unless `{}` is what preceded us | +Housing a finding in the wrong unit is not housing it. ## Test plan @@ -233,8 +434,16 @@ survive a write; unparseable fails closed rather than resetting. to the §5 row, `busy` carries `Retry-After`, and a best-effort management caller still returns 2xx while reporting a non-converged disposition. -A grep-level guard test: no module outside `convergence.ts` imports the catalog -commit or the remover directly. That is how C14 stays true as callers are added. +**The funnel must be provable, not grepped** (round 2 #2). A grep guard misses a +wrapper in the same module, a re-export, an alias and a dynamic import — and the +tree has all four today: `refreshCodexModelCatalog` wraps the catalog writers +(`refresh.ts:40-52`), `restoreNativeCodex` wraps config/catalog/history removal +(`inject.ts:764-794`), and `catalog.ts:11` re-exports the direct writers. + +So the low-level writers move into `src/codex/internal/` whose **only** permitted +importer is `convergence.ts`, and the guard test walks the module dependency +GRAPH — static imports, dynamic imports, re-exports and aliases — asserting no +other path reaches them. Reachability, not spelling. ## Accept criteria From ee182744af6958478523fb97ece6af2efb63b082 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Tue, 4 Aug 2026 11:09:47 +0900 Subject: [PATCH 032/163] =?UTF-8?q?docs(substrate):=20collect=20the=20cont?= =?UTF-8?q?ract=20=E2=80=94=20four=20docs=20become=20consumers?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Round 2's verdict was that 005 declared ownership without transferring it, and the reviewer's list of contradicting sections ran thirty-odd entries. This is the transfer: 2931 lines deleted, 1531 added, because the shared surfaces stop being repeated four times. 010 gives up management outcomes, its own revision schema, the helper orchestrator and the route mapping; it keeps write-free gathering and the fixed synchronous commit, and it now supplies the first real convergeCodex while rewiring all 16 catalog callers. 020 gives up record ownership, the inline CLI bypass, process-only exclusion and the permanent 60-tick dormancy; it keeps Worker isolation and the sibling history lock, now with expectation-based overtaking prevention so an overtaken job is rejected before it mutates. 030 deletes every home-derived namespace design, since the pinned Bun probe showed both os.homedir() and os.userInfo().homedir follow an environment home; it keeps bounded lock mechanics on a uid/SID namespace and now states the native-then-config ordering with non-nested history sequencing. 040 deletes the competing record definition, the second entry point and the two wrong module names; it keeps authority evidence, provenance and restoration, now on the contract's two baseline classes with an explicit ledger-recovery path. --- .../010_catalog_seam.md | 1092 +++++---------- .../020_history_isolation.md | 1123 +++++----------- .../030_lock_protocol.md | 1061 +++++---------- .../040_ownership_convergence.md | 1186 ++++++----------- 4 files changed, 1531 insertions(+), 2931 deletions(-) diff --git a/devlog/_plan/260804_codex_write_substrate/010_catalog_seam.md b/devlog/_plan/260804_codex_write_substrate/010_catalog_seam.md index 0824b8533..fcde090bd 100644 --- a/devlog/_plan/260804_codex_write_substrate/010_catalog_seam.md +++ b/devlog/_plan/260804_codex_write_substrate/010_catalog_seam.md @@ -1,836 +1,466 @@ # WP9 — split Codex catalog gather from commit -Research: `001_catalog_seam.md`. Read it first; this doc is the diff. - -The incident is r2 #1: the OFF design needed provider discovery outside a -per-`CODEX_HOME` lock and native writes inside it, but management exposes one -`Promise` callback that gathers and writes before resolving -(`../260803_codex_desktop_toggle/008_audit_synthesis_wp4_r2.md:17-24`, -`src/server/management/context.ts:9-18`). Today `refreshCodexModelCatalog` awaits -the mixed `syncCatalogModels`, checks the file that function may just have -written, and then performs a second cache write (`src/codex/refresh.ts:40-52`, -`src/codex/catalog/sync.ts:507-569,600-616`). This phase adds an opaque, -point-in-time candidate, a pure gather, a fixed synchronous commit, a revision -check, and typed dispositions. It does not add the native write lock; WP11 owns -that. Until WP11, the existing orchestrator calls gather and then commit directly, -so the split is independently useful and the revision guard is exercised, without -pretending a lock already exists. +Research: `001_catalog_seam.md`. Shared contract: `005_contract.md`. Read both +before implementing this diff. + +The incident is still r2 #1: the active refresh combines provider discovery, +catalog assembly, native writes, and cache invalidation in one awaited function +(`src/codex/refresh.ts:40-52`, `src/codex/catalog/sync.ts:507-569,600-616`). The +16 management mutations then call a `Promise` helper whose only failure +policy is a swallowed exception (`src/server/management-api.ts:105-112`, +`src/server/management/context.ts:54-69`). That shape cannot place slow gathering +outside a lock and a fixed commit inside it. + +This phase fixes that catalog mechanism. It is the **first real implementation** +of the contract's `convergeCodex`. It does not define another +entry point, record, route mapping, admission shape, or shared result union. WP8b +landed those declarations without rewiring behavior (`005_contract.md` §What +"lands first"). WP9 consumes them and rewires the catalog callers in the same +commit, so this phase typechecks and preserves the callers' 2xx/201 behavior on +its own. Nothing here waits for WP10-WP12 to make the tree buildable. + +All current-code citations and diff context below were rechecked on 2026-08-04 at +`2d5e080dea3e7000bf2111b381c7c1a3c4f5fb11`. ## IN / OUT -IN — production: - -- `src/codex/refresh.ts` (MODIFY) — canonical candidate, revisions, gather/commit - exports, outcome union, and the compatibility orchestrator. -- `src/codex/catalog.ts` (MODIFY) — re-export the prepared catalog contract from - the existing facade; do not create a second catalog entry point. -- `src/codex/catalog/sync.ts` (MODIFY) — turn assembly into a write-free prepared - payload and expose the fixed synchronous writer. -- `src/codex/catalog/bundled.ts` (MODIFY) — replace the materializing sync fallback - with in-memory catalog bytes for the candidate. -- `src/codex/catalog/parsing.ts` (MODIFY) — prepare create-once pristine-backup - bytes without writing; retain restore's existing writer path. -- `src/codex/catalog/provider-fetch.ts` (MODIFY) — return typed degradation notices - and distinguish token resolution from provider-network failure. -- `src/codex/sync.ts` (MODIFY) — consume typed refresh outcomes and preserve the - existing ordinary-failure injection fallback. -- `src/server/management/context.ts` (MODIFY) — replace the void callback with the - paired candidate seam and typed best-effort result. -- `src/server/management-api.ts` (MODIFY) — orchestrate the injected or production - pair, never swallow the disposition. -- `src/server/management/provider-routes.ts` (MODIFY) — six persisted mutations - attach `catalogRefresh`. -- `src/server/management/model-routes.ts` (MODIFY) — six persisted mutations attach - `catalogRefresh`. -- `src/server/management/combo-routes.ts` (MODIFY) — two persisted mutations attach - `catalogRefresh` without suppressing Claude follow-up work. -- `src/server/management/agent-settings-routes.ts` (MODIFY) — two persisted - mutations attach `catalogRefresh` without suppressing Claude/Desktop follow-up - work. -- `src/server/management/config-routes.ts` (MODIFY) — explicit `/api/sync` maps - typed authorization/contention/disk results instead of flattening every failure - to 500. -- `structure/03_catalog-and-subagents.md` (MODIFY) — source-of-truth statement for - the candidate/revision contract. -- `structure/05_gui-and-management-api.md` (MODIFY) — source-of-truth statement for - best-effort mutation responses versus explicit sync. -- `docs-site/src/content/docs/reference/management-api.md` (MODIFY), plus the - matching `ja`, `ko`, `ru`, and `zh-cn` files (MODIFY) — document the additive - `catalogRefresh` field and explicit-sync status mapping. +IN — catalog mechanism: + +- `src/codex/refresh.ts` (MODIFY) — opaque candidate and catalog-private gather / + commit outcomes. +- `src/codex/catalog.ts` (MODIFY) — retain the existing facade while moving direct + writers behind the contract's internal boundary. +- `src/codex/catalog/sync.ts` (MODIFY) — write-free preparation and one fixed, + synchronous writer. +- `src/codex/catalog/bundled.ts` (MODIFY) — in-memory fallback during gather. +- `src/codex/catalog/parsing.ts` (MODIFY) — prepare create-once backup bytes without + writing them. +- `src/codex/catalog/provider-fetch.ts` (MODIFY) — sanitized, catalog-private + degradation notices. +- `src/codex/convergence.ts` (MODIFY) — implement the contract-declared + `convergeCodex` for the first time. This phase consumes `AdmissionSnapshot`, + `CommitExpectation`, `CatalogDisposition`, and `ConvergeOutcome` from + `convergence-types.ts`; it does not redefine them. +- `src/codex/internal/catalog-commit.ts` (NEW/MOVE) — the prepared catalog/cache/ + backup writer, reachable only from `convergence.ts`, as required by + `005_contract.md` §Test plan. +- `src/codex/sync.ts` (MODIFY) — delegate catalog work to `convergeCodex`; retain + the existing ordinary-failure injection fallback until later phases replace + more of the native path. + +IN — production callers: + +- `src/server/management/context.ts`, `src/server/management-api.ts` (MODIFY) — + inject/call `convergeCodex`, not a catalog-specific orchestrator. +- `src/server/management/provider-routes.ts` (MODIFY) — six mutations report the + contract's `CatalogDisposition` while retaining their primary status. +- `src/server/management/model-routes.ts` (MODIFY) — six mutations, same rule. +- `src/server/management/combo-routes.ts` (MODIFY) — two mutations, without + suppressing Claude follow-up work. +- `src/server/management/agent-settings-routes.ts` (MODIFY) — two mutations, + without suppressing Claude/Desktop follow-up work. +- `src/server/management/config-routes.ts` (MODIFY) — explicit sync calls + `convergeCodex` and hands the result to the contract adapter. +- `structure/03_catalog-and-subagents.md`, + `structure/05_gui-and-management-api.md` (MODIFY) — document the production + funnel and best-effort mutation semantics. +- `docs-site/src/content/docs/reference/management-api.md` and matching `ja`, + `ko`, `ru`, `zh-cn` pages (MODIFY) — additive `catalogRefresh`; route statuses + are referenced from the contract, not copied into these implementation notes. IN — tests: -- `tests/codex-refresh.test.ts` (MODIFY) — pure gather, bounded commit, one-shot, - stale config/base revisions, and partial write receipts. -- `tests/codex-sync-api.test.ts` (MODIFY) — typed fallback and no-injection cases. -- `tests/codex-models-cache-invalidate.test.ts` (MODIFY) — receipt-driven app-server - invalidation behavior. -- `tests/injection-model-api.test.ts` (MODIFY) — immutable config snapshot rather - than forwarding a mutable config reference. -- `tests/model-visibility-management-api.test.ts` (MODIFY), - `tests/management-provider-validation.test.ts` (MODIFY), - `tests/combo-management-api.test.ts` (MODIFY), `tests/combos.test.ts` (MODIFY), - and `tests/codex-v2-gate.test.ts` (MODIFY) — paired seam and response disposition. -- `tests/management-integration-routes.test.ts` (MODIFY), - `tests/management-client-config-route.test.ts` (MODIFY), - `tests/responses-shadow-intercept.test.ts` (MODIFY), - `tests/server-combo-failover-e2e.test.ts` (MODIFY), and - `tests/catalog-input-modality-enum.test.ts` (MODIFY) — fixture type migration; - no new behavior in routes that do not refresh. +- `tests/codex-refresh.test.ts`, `tests/codex-sync-api.test.ts`, + `tests/codex-models-cache-invalidate.test.ts`, + `tests/injection-model-api.test.ts` (MODIFY) — pure gather, fixed commit, real + generation invalidation, and compatibility behavior. +- `tests/model-visibility-management-api.test.ts`, + `tests/management-provider-validation.test.ts`, + `tests/combo-management-api.test.ts`, `tests/combos.test.ts`, + `tests/codex-v2-gate.test.ts`, + `tests/management-integration-routes.test.ts`, + `tests/management-client-config-route.test.ts`, + `tests/responses-shadow-intercept.test.ts`, + `tests/server-combo-failover-e2e.test.ts`, and + `tests/catalog-input-modality-enum.test.ts` (MODIFY) — migrate fixtures to + `convergeCodex`; routes that do not refresh retain zero calls. +- `tests/codex-convergence-contract.test.ts` (MODIFY) — add production module-graph + reachability checks and the 16-caller funnel proof. WP8b created this test file; + WP9 extends it rather than creating a second guard. OUT: -- `src/integrations/native/**`, `src/service.ts`, and a native write lock — WP11 - creates the lock and wraps `commitCodexCatalogCandidate`; WP9 must compile and - behave correctly without it. -- Desired-state OFF reads and ownership admission — WP12 owns those authorities. - The outcome union reserves `desired_off` and the orchestrator handles it, but - WP9 does not invent a flag or emit that result. -- `gui/**` — responses are additive and the current dashboard does not need a new - visual state in this substrate phase. -- `src/codex/history-provider.ts` — WP10 isolates history separately. -- `syncCodexModelsCacheFromCatalog` — retain the explicit raw-copy utility at - `src/codex/refresh.ts:29-32`; it is not the active expired-wrapper commit. -- Transactional rollback — catalog and cache are separate atomic replacements; - a receipt reports partial progress instead of claiming all-or-nothing behavior - (`src/config.ts:178-230`, `src/codex/catalog/sync.ts:568,601-613`). - -## The candidate and the two operations - -MODIFY `src/codex/refresh.ts`. The public type is structurally opaque because its -brand symbol is module-private; the payload lives in a `WeakMap`, so it cannot be -JSON-serialized, reconstructed by a caller, or inspected for credentials. The -state holds strings, paths, revisions, notices, and result metadata only — no -mutable `OcxConfig`, file handle, callback, or promise. +- The `integrations/codex.json` schema and updater, `AdmissionSnapshot`, + `CommitExpectation`, `ConvergeRequest`, `ConvergeOutcome`, `CatalogDisposition`, + and `toSyncResponse` — owned by `005_contract.md` §§1-5. This document deletes + its old versions instead of restating them. +- Management status/header ownership. `/api/sync` is mapped only by + `src/server/management/sync-response.ts` (`005_contract.md` §5). +- Desired-state, ownership, journal, and provenance policy — WP12 consumes the same + funnel and strengthens admission; WP9 does not reserve fake outcomes for it. +- The native write lock — WP11. WP9's commit is synchronous now so WP11 can wrap it + later without changing the catalog contract. +- History isolation/locking — WP10. +- `gui/**`, transactional rollback, release/deploy actions, and the live proxy on + port 10100. + +## The catalog-private candidate + +**INFERRED implementation choice:** the candidate is opaque and one-shot. Its payload is held in a module-private +`WeakMap`, so callers cannot inspect credentials, substitute bytes, serialize it, +or reconstruct a stale candidate. ```ts -const codexCatalogCandidateBrand: unique symbol = Symbol("CodexCatalogCandidate"); +const candidateBrand: unique symbol = Symbol("CodexCatalogCandidate"); export interface CodexCatalogCandidate { - readonly [codexCatalogCandidateBrand]: true; + readonly [candidateBrand]: true; } -interface CodexCatalogCandidateState { +interface CandidateState { readonly prepared: PreparedCodexCatalogCommit; - readonly revision: CodexCatalogRevision; - readonly result: Omit; + readonly admittedGeneration: number; + readonly targetIdentities: readonly CatalogTargetIdentity[]; readonly notices: readonly CatalogGatherNotice[]; consumed: boolean; } -const candidateStates = new WeakMap(); - -/** - * Discover providers and load every source needed to assemble the exact catalog, - * backup, and expired-cache bytes that a later commit may write. - * - * WHY: provider auth, network I/O, bundled `codex debug models --bundled`, JSON - * parsing, merging, and serialization are unbounded relative to a native-write - * critical section. This operation therefore performs no mkdir/copy/write/rename - * and returns an opaque point-in-time candidate instead of exposing writable - * payloads to callers. - */ +const states = new WeakMap(); + export async function gatherCodexCatalogCandidate( - config: OcxConfig, + admission: AdmissionSnapshot, ): Promise; -/** - * Revalidate and consume one gathered candidate, then perform only its prepared - * create-once backup writes, catalog replacement, and expired-cache replacement. - * - * WHY: r2 #1 can be fixed only if the eventual lock owner can call a synchronous, - * fixed-write function. Rechecking config and base-catalog revisions here prevents - * a candidate assembled from obsolete state from overwriting a newer catalog. - * No provider call, auth resolution, subprocess, parse, merge, serialization, or - * await is permitted below this boundary. - */ export function commitCodexCatalogCandidate( candidate: CodexCatalogCandidate, -): CodexCatalogCommitResult; + expectation: CommitExpectation, +): CatalogCommitOutcome; ``` -`gatherCodexCatalogCandidate` calls a write-free `prepareCatalogSync(config)` in -`src/codex/catalog/sync.ts`. That helper returns final catalog bytes, expired-cache -wrapper bytes, optional pristine-backup path/byte pairs, exact target paths, -`added`, `comboOmissions`, and notices. -The gather then freezes a tiny branded handle and stores the internal state in the -`WeakMap`. `commitCodexCatalogCandidate` rejects a missing/consumed handle as -`stale_candidate`, marks a valid handle consumed before the first write, compares -revisions, and invokes `writePreparedCatalogCommit` only on an exact match. - -Mark-before-write is deliberate. Retrying a partially written candidate would -replay old bytes after a later convergence. A disk failure returns its receipt; -the caller regathers instead of recommitting the consumed handle. - -## C2 — the revision guard - -The guard uses content revisions, not mtimes. An mtime can change without content, -can be restored, and has platform-dependent resolution; the merge at -`src/codex/catalog/sync.ts:520-565` depends on exact catalog bytes. **INFERRED:** -SHA-256 of canonical config input and exact base-catalog bytes is the smallest -evidence that detects every input change relevant to this candidate while avoiding -raw credential retention. +The exact `AdmissionSnapshot` returned by the contract admission is passed to +gather. `prepareCatalogSync` receives `admission.config` — **that object**, not the +server's captured config and not a separate `readConfigDiagnostics()` result. This +is the transfer required by `005_contract.md` §4. A gather that reopens config has +reintroduced the stale-object disagreement audit #8 identified. + +Gather performs provider auth/network work, source loading, parsing, merging, +serialization, cache-wrapper construction, and backup planning. It performs no +`mkdir`, copy, write, rename, journal mutation, or integration-record update. The +isolated-home before/after manifest is the acceptance evidence for that claim. + +Commit marks the candidate consumed before the first write, validates the shared +generation/identity evidence, and performs at most four atomic replacements in a +fixed order: keyed backup, optional legacy backup, catalog, cache. Retrying a +partially written candidate would replay old bytes after a later transition, so a +second call is a catalog-private `candidate-consumed` result and never writes. + +## C2 — generation and target identity, not content revision + +The old plan owned a `ContentRevision` and hashed config/catalog bytes. That design +is deleted. Content equality passes A→B→A, and a textual path does not reveal a +parent-symlink retarget. The shared mechanism is `005_contract.md` §3: + +- `AdmissionSnapshot.generation` identifies the cooperating config generation used + by gather; +- `CommitExpectation { nativeBefore, nativeAfter, txId }` identifies the one native + transition this commit is allowed to perform; +- each prepared target records canonical parent identity plus file identity where + available, not merely a path string; +- the config mutation coordinator stays held through the authoritative re-read and + synchronous commit for cooperating writers; +- after commit, native generation must be exactly `nativeAfter` with this `txId`. + +The catalog phase supplies the target observations and refuses its private commit +when the shared validator rejects them. It does not invent a third counter or a +catalog-specific revision schema. + +The bound is stated narrowly. Cooperating transitions are prevented from committing +a stale candidate. A single-direction target retarget or replacement is detected. +The mechanism does **not** claim to detect an arbitrary parent-symlink A→B→A that +occurs entirely between checks; `005_contract.md` §3 explicitly scopes C17 that +way. Provider inventory changing upstream after a completed gather is also not +filesystem interference; a later convergence may supersede that snapshot. + +WP9 is independently correct before WP11: its synchronous no-await commit prevents +same-process interleaving and rejects generation/identity evidence that changed +before commit. It does not claim cross-process exclusion until WP11 installs the +native lock. The catalog API does not change when that lock lands. + +## Catalog-internal outcomes only + +The prior document published `CodexCatalogRefreshOutcome`, +`CatalogRefreshDisposition`, and a skip-reason union that included future +`desired_off` and `lock_busy`. Those shared versions are deleted. The contract owns +the public result and management projection (`005_contract.md` §2). + +WP9 keeps only facts needed inside the catalog implementation: ```ts -type ContentRevision = - | { readonly state: "absent" } - | { readonly state: "present"; readonly sha256: string }; - -interface CodexCatalogRevision { - readonly configSha256: string; - readonly baseCatalog: ContentRevision; - readonly codexHome: string; - readonly catalogPath: string; - readonly cachePath: string; -} +type CatalogGatherOutcome = + | { kind: "prepared"; candidate: CodexCatalogCandidate } + | { kind: "unavailable" } + | { kind: "degraded"; candidate: CodexCatalogCandidate; notices: readonly CatalogGatherNotice[] } + | { kind: "failed"; surface: "provider-auth" | "provider-network"; retryable: boolean }; + +type CatalogCommitOutcome = + | { kind: "committed"; result: CodexCatalogRefreshResult; writes: CatalogWriteReceipt } + | { kind: "stale"; reason: "generation" | "target-identity" | "candidate-consumed" } + | { kind: "failed"; surface: "disk"; writes: CatalogWriteReceipt }; ``` -Gather captures exactly these values: - -1. `configSha256`: SHA-256 of stable-key JSON containing the complete `providers` - object (including auth mode and credential/env references), `disabledModels`, - `customModels`, `combos`, `subagentModels`, `multiAgentMode`, - `providerContextCaps`, `contextCapValue`, `modelCacheTtlMs`, `websockets`, and the - fresh `isMultiAgentV2Enabled()` value. Those are the inputs consumed by provider - gathering and final assembly (`src/codex/catalog/provider-fetch.ts:98-142,670-719,785-814`, - `src/codex/catalog/sync.ts:533-565`). The digest includes credential values so a - key rotation invalidates stale discovery, but the candidate never retains or - returns those values. -2. `baseCatalog`: `absent`, or SHA-256 of the exact bytes read from `catalogPath` - before parsing. This is the merge source whose routed and user-native rows are - preserved (`src/codex/catalog/sync.ts:517-523,430-468`). -3. `codexHome`: `realpathSync.native(resolveCodexHomeDir())`, plus the resolved - `catalogPath` and `activeCodexModelsCachePath()`. This prevents an environment or - config-path change from redirecting prepared bytes to another home - (`structure/02_config-and-codex-home.md:3-21`, - `src/codex/catalog/parsing.ts:73,167`). - -At commit, synchronously and before any mkdir/write, recompute: - -- the config digest from `readConfigDiagnostics().config` plus a fresh feature-flag - read; -- the canonical home and both target paths; and -- the exact current catalog content digest/absence marker. - -Every field must equal the candidate revision. Any mismatch returns -`{ status: "skipped", reason: "stale_candidate", retryable: true }`, consumes the -candidate, and produces an all-false write receipt. No backup directory, backup, -catalog, or cache is created. A newly appearing backup does not make the candidate -stale: backups are create-once; the commit rechecks each optional backup path and -skips that one write if another actor already created it -(`src/codex/catalog/parsing.ts:428-444`). - -Provider inventory changing upstream after gather is not a revision mismatch. The -candidate represents one completed discovery. A subsequent refresh may supersede -it; wall-clock age is not used (`src/codex/catalog/provider-fetch.ts:481-510,670-717`). - -Without WP11 there is no cross-process critical section around compare-and-write. -WP9 still makes the operation correct for every revision change completed before -commit begins and for all same-process interleavings because commit contains no -`await`. **INFERRED:** an external process can still replace the catalog after the -comparison and before rename; WP11 closes that remaining TOCTOU by placing this -unchanged synchronous function under the shared per-home lock. WP9 must not claim -cross-process linearizability before that phase. - -## Typed outcomes - -MODIFY `src/codex/refresh.ts` with one closed public outcome and a public, -credential-free management projection: +These types do not cross the `convergence.ts` boundary. `convergeCodex` projects +them into the contract's `ConvergeOutcome` and `CatalogDisposition`; no route +switches on catalog-private variants. Provider names, URLs, token text, paths, +digests, and raw exceptions never enter the public disposition. Partial disk writes +are derived from the receipt and cause a fresh convergence, never a replay of the +candidate. -```ts -export type CatalogGatherNotice = { - kind: "provider_degraded"; - reason: "provider_network" | "provider_auth"; - fallback: "stale" | "configured"; -}; - -export interface CatalogWriteReceipt { - catalogBackup: boolean; - legacyBackup: boolean; - catalog: boolean; - cache: boolean; -} +## Diff — preparation and fixed writes -export type CodexCatalogCommitResult = - | { status: "committed"; result: CodexCatalogRefreshResult; writes: CatalogWriteReceipt } - | { status: "skipped"; reason: "stale_candidate"; retryable: true; writes: CatalogWriteReceipt } - | { status: "failed"; reason: "disk"; phase: "commit"; retryable: true; writes: CatalogWriteReceipt }; - -export type CodexCatalogSkipReason = - | "catalog_unavailable" - | "desired_off" - | "gather_busy" - | "lock_busy" - | "stale_candidate"; - -export type CodexCatalogRefreshOutcome = - | { status: "committed"; result: CodexCatalogRefreshResult; notices: readonly CatalogGatherNotice[]; writes: CatalogWriteReceipt } - | { status: "skipped"; reason: CodexCatalogSkipReason; retryable: boolean; writes: CatalogWriteReceipt } - | { status: "failed"; reason: "provider_network" | "provider_auth" | "disk"; phase: "gather" | "commit"; retryable: boolean; writes: CatalogWriteReceipt }; - -export type CatalogRefreshDisposition = - | { status: "committed"; degraded: boolean } - | { status: "skipped"; reason: CodexCatalogSkipReason; retryable: boolean } - | { status: "failed"; reason: "provider_network" | "provider_auth" | "disk"; retryable: boolean; partialWrite: boolean }; -``` - -Do not expose provider names, URLs, token text, catalog paths, or digests through -`CatalogRefreshDisposition`. `committed` plus `degraded:true` is how a stale/static -provider fallback remains visible without turning the primary route into a failure. -`partialWrite` is `true` when any receipt bit is true. - -`refreshCodexModelCatalog(config, deps)` becomes the direct gather-then-commit -orchestrator for WP9. It maps `CatalogGatherBusyError` to `skipped/gather_busy`, -typed token-resolution failure to `failed/provider_auth`, escaped provider fetch -failure to `failed/provider_network`, no source to `skipped/catalog_unavailable`, -and commit errors to their returned result. It does not emit `desired_off` or -`lock_busy`; WP11/WP12 add those admission results without changing callers. - -### Every management caller - -All 16 rows are best-effort by design. For every variant they retain their current -2xx/201 primary status, attach `catalogRefresh`, and never roll back the config -mutation that already landed. `committed` reports `degraded:false|true`; OFF, -gather/lock contention, and stale candidates report `skipped` with reason and -retryability; auth/network/disk report `failed`. The route-specific continuation is -the only difference. - -| Exact outcome | `catalogRefresh` projection on every row below | -|---|---| -| committed, no notices | `{ status: "committed", degraded: false }` | -| committed with provider network/auth fallback | `{ status: "committed", degraded: true }` | -| `desired_off` | `{ status: "skipped", reason: "desired_off", retryable: false }` | -| `gather_busy` | `{ status: "skipped", reason: "gather_busy", retryable: true }` | -| `lock_busy` | `{ status: "skipped", reason: "lock_busy", retryable: true }` | -| `stale_candidate` | `{ status: "skipped", reason: "stale_candidate", retryable: true }` | -| `catalog_unavailable` | `{ status: "skipped", reason: "catalog_unavailable", retryable: false }` | -| `provider_auth` | `{ status: "failed", reason: "provider_auth", retryable: false, partialWrite: false }` | -| `provider_network` | `{ status: "failed", reason: "provider_network", retryable: true, partialWrite: false }` | -| `disk` | `{ status: "failed", reason: "disk", retryable: true, partialWrite: }` | - -| # | Caller and current line | Best-effort | Committed / degraded | OFF / gather busy / lock busy / stale | Auth / network / disk | -|---|---|---|---|---|---| -| P1 | provider add/overwrite, `src/server/management/provider-routes.ts:147-148` | YES | Return current 200 plus disposition. | Same 200; provider remains saved. | Same 200; no rollback. | -| P2 | ordinary provider edit/toggle, `src/server/management/provider-routes.ts:338-344` | YES | Return current 200 plus disposition. | Same 200; edited provider remains saved. | Same 200; no rollback. | -| P3 | provider delete, `src/server/management/provider-routes.ts:479-488` | YES | Return current 200 plus disposition. | Same 200; warn stale native rows through disposition. | Same 200; no rollback. | -| P4 | global context-cap value, `src/server/management/provider-routes.ts:503-513` | YES | `respond` includes disposition. | Same 200 and cap body. | Same 200 and cap body. | -| P5 | all context-cap toggles, `src/server/management/provider-routes.ts:516-528` | YES | `respond` includes disposition. | Same 200 and cap body. | Same 200 and cap body. | -| P6 | one provider context cap, `src/server/management/provider-routes.ts:531-547` | YES | `respond` includes disposition. | Same 200 and cap body. | Same 200 and cap body. | -| M1 | disabled models, `src/server/management/model-routes.ts:208-215` | YES | Return current 200 plus disposition. | Same 200; blocklist remains saved. | Same 200; no rollback. | -| M2 | model visibility, `src/server/management/model-routes.ts:221-314` | YES | Return current 200 plus disposition. | Same 200; visibility intent remains saved. | Same 200; no rollback. | -| M3 | custom model create, `src/server/management/model-routes.ts:321-353` | YES | Preserve 201; append disposition. | Preserve 201. | Preserve 201. | -| M4 | custom model edit, `src/server/management/model-routes.ts:356-391` | YES | Return current 200 plus disposition. | Same 200. | Same 200. | -| M5 | custom model delete, `src/server/management/model-routes.ts:394-405` | YES | Return current 200 plus disposition. | Same 200; possible stale row is explicit. | Same 200. | -| M6 | selected models, `src/server/management/model-routes.ts:426-441` | YES | Return current 200 plus disposition. | Same 200; allowlist remains saved. | Same 200. | -| C1 | combo create/update/rename, `src/server/management/combo-routes.ts:190-200` | YES | Return current 200 plus disposition. | Same 200; still run Claude sync when `shouldSyncClaudeAgentDefs`. | Same 200; still run Claude sync. | -| C2 | combo delete, `src/server/management/combo-routes.ts:203-217` | YES | Return current 200 plus disposition. | Same 200. | Same 200. | -| A1 | v2/settings write, `src/server/management/agent-settings-routes.ts:224-294` | YES | Return current 200 plus disposition and existing warnings. | Same 200; feature/config writes remain authoritative. | Same 200; no rollback. | -| A2 | subagent model write, `src/server/management/agent-settings-routes.ts:518-528` | YES | Return current 200 plus disposition. | Same 200; still run Claude and Desktop follow-ups. | Same 200; still run both follow-ups. | - -Explicit sync is not in that table. `syncModelsToCodex` keeps injection fallback for -`catalog_unavailable`, provider degradation, provider auth/network failure, and disk -failure, matching the current catch-and-continue contract at -`src/codex/sync.ts:83-110` and `tests/codex-sync-api.test.ts:148-166`. It returns -before `injectCodexConfig` for `desired_off`, `lock_busy`, or `stale_candidate`: -those are authorization/serialization refusals, not missing catalog data. Gather -busy is retryable and also returns before injection so an explicit sync cannot -claim fresh native state while another revision is being assembled. `/api/sync` -maps `desired_off` and `stale_candidate` to 409, `gather_busy`/`lock_busy` to 503 -with `Retry-After: 1`, and non-fallback disk failure to 500 -(`src/server/management/config-routes.ts:261-268`). - -## Diff - -### Catalog preparation and fixed writes - -MODIFY `src/codex/catalog/bundled.ts` at current lines 225-234. The fallback remains -in memory; it does not materialize a source while loading: +MODIFY `src/codex/catalog/bundled.ts` at current lines 225-234. Loading a fallback +during gather stays in memory: ```diff export function loadCatalogForSync(path: string): RawCatalog | null { -@@ - return readCatalog(catalogBackupPathFor(path)) + return readCatalog(path) + ?? readCatalog(catalogBackupPathFor(path)) ?? (isDefaultCatalogPath(path) ? readCatalog(legacyCatalogBackupPath()) : null) ?? readCatalog(activeCodexModelsCachePath()) - ?? materializeBundledCodexCatalog(path) - ?? catalog; + ?? loadBundledCodexCatalog(); } ``` -Retain `materializeBundledCodexCatalog` for its existing public callers -(`src/codex/catalog.ts:6`); only catalog gather stops calling it. +Retain `materializeBundledCodexCatalog` for existing explicit callers. Only gather +stops using a materializing fallback. -MODIFY `src/codex/catalog/parsing.ts` around current lines 428-444. Extract a pure -backup planner beside the existing restore-facing writer: +MODIFY `src/codex/catalog/sync.ts` at current lines 507-569 and 600-616. Assembly +returns bytes and observations; the writer accepts no config and performs no await: ```diff -+export interface PreparedCatalogBackup { -+ path: string; -+ bytes: string; -+ kind: "catalog" | "legacy"; -+} -+ -+export function prepareCatalogBackups( -+ catalogPath: string, -+ catalog: RawCatalog, -+ onDiskBytes: string | null, -+): PreparedCatalogBackup[] { -+ const source = onDiskBytes === null ? null : parseCatalogJson(onDiskBytes); -+ const pristineBytes = source && !catalogHasRoutedEntries(source) -+ ? onDiskBytes -+ : !catalogHasRoutedEntries(catalog) -+ ? JSON.stringify(catalog, null, 2) + "\n" -+ : null; -+ if (pristineBytes === null) return []; -+ return [ -+ { path: catalogBackupPathFor(catalogPath), bytes: pristineBytes, kind: "catalog" }, -+ ...(isDefaultCatalogPath(catalogPath) -+ ? [{ path: legacyCatalogBackupPath(), bytes: pristineBytes, kind: "legacy" as const }] -+ : []), -+ ]; -+} -+ - export function writePristineCatalogBackup(backupPath: string, catalogPath: string, catalog: RawCatalog): void { -``` - -MODIFY `src/codex/catalog/sync.ts` at current lines 507-569 and 600-616. The full -assembly remains where it is, but its output is bytes, not mutations: - -```diff --export async function syncCatalogModels(config: OcxConfig): Promise<{ -+export interface PreparedCodexCatalogCommit { - added: number; - path: string; -- catalogWritten: boolean; - comboOmissions: ComboCatalogOmission[]; --}> { -+ catalogBytes: string | null; -+ cachePath: string; -+ cacheBytes: string | null; -+ backups: PreparedCatalogBackup[]; -+ baseCatalogBytes: string | null; -+ notices: CatalogGatherNotice[]; -+} -+ -+export async function prepareCatalogSync(config: OcxConfig): Promise { +-export async function syncCatalogModels(config: OcxConfig): Promise { ++export async function prepareCatalogSync( ++ config: Readonly, ++): Promise { const catalogPath = readCodexCatalogPath(); -- const catalog = loadCatalogForSync(catalogPath); -- if (!catalog) return { added: 0, path: catalogPath, catalogWritten: false, comboOmissions: [] }; -+ const catalog = loadCatalogForSync(catalogPath); -+ if (!catalog) return emptyPreparedCatalogCommit(catalogPath); -+ const baseCatalogBytes = readFileOrNull(catalogPath); - - // The bundled catalog is a reliable native template on the default path, but it is not the -@@ -- const onDiskCatalog = readCatalog(catalogPath); -+ const onDiskCatalog = baseCatalogBytes === null ? null : parseCatalogJson(baseCatalogBytes); -@@ -- const goModels = await gatherRoutedModels(config, { comboOmissions }); -- try { -- // Once-only: preserve the PRISTINE pre-opencodex catalog as the native-priority baseline -- // (later syncs would otherwise overwrite it with featured-modified priorities). -- ensureCatalogBackup(catalogPath, catalog); -- } catch { /* backup best-effort */ } -+ const notices: CatalogGatherNotice[] = []; -+ const goModels = await gatherRoutedModels(config, { comboOmissions, notices }); -+ const backups = prepareCatalogBackups(catalogPath, catalog, baseCatalogBytes); -@@ + const baseCatalogBytes = readFileOrNull(catalogPath); + // Existing merge logic, provider gather, backup planning, serialization. - atomicWriteFile(catalogPath, JSON.stringify(catalog, null, 2) + "\n"); -- return { added: goEntries.length, path: catalogPath, catalogWritten: true, comboOmissions }; -+ const catalogBytes = JSON.stringify(catalog, null, 2) + "\n"; -+ const cacheBytes = JSON.stringify({ -+ fetched_at: "2000-01-01T00:00:00Z", -+ client_version: "0.0.0", -+ models: catalog.models ?? catalog, -+ }, null, 2) + "\n"; +- invalidateCodexModelsCache(); +- return result; + return { -+ added: goEntries.length, -+ path: catalogPath, -+ comboOmissions, + catalogBytes, -+ cachePath: activeCodexModelsCachePath(), + cacheBytes, + backups, -+ baseCatalogBytes, ++ targets: observeCatalogTargetIdentities(catalogPath, cachePath, backups), ++ result, + notices, + }; } + +export function writePreparedCatalogCommit( + prepared: PreparedCodexCatalogCommit, -+): { result: CodexCatalogRefreshResult; writes: CatalogWriteReceipt } { -+ // No await, parsing, serialization, provider call, or subprocess below this line. -+ // Set each receipt bit only after its atomic replacement returns. -+ // Backup writes remain create-once: existsSync(path) means skip, never overwrite. -+ // On failure throw CatalogCommitDiskError carrying the receipt completed so far. ++): CatalogWriteReceipt { ++ // Fixed order; set each receipt bit only after atomic replacement returns. +} ``` -`writePreparedCatalogCommit` performs at most four atomic replacements in this -fixed order: keyed backup, optional legacy backup, catalog, cache. The final catalog -replacement is also what materializes an absent default catalog; there is no fifth -"source" write. It creates only the parent directories needed by a prepared write. -The count never scales with providers or models. `invalidateCodexModelsCache` remains -for startup/manual cache invalidation at `src/codex/catalog/sync.ts:600-616`; it is -not called by the candidate commit. +Move `writePreparedCatalogCommit` and every lower-level direct catalog/cache writer +used by convergence into `src/codex/internal/catalog-commit.ts`. +`src/codex/catalog.ts:11` currently +re-exports `syncCatalogModels`; remove that direct writer export after all production +and test imports migrate. The dependency-graph test, not an `rg` spelling guard, +proves no alias, re-export, wrapper, or dynamic import reaches the writers outside +`convergence.ts` (`005_contract.md` §Test plan). -MODIFY `src/codex/catalog/provider-fetch.ts` at current lines 410-428, 494-510, and -670-693. `resolveModelsAuthToken` rejection becomes `CatalogProviderAuthError`, and -each fallback pushes one sanitized notice with `reason` and `fallback`; no provider -name or exception text enters the public notice. The flight result carries notices -so same-key joiners receive the exact degradation set from the flight they joined, -just as `comboOmissions` is flight-local at lines 696-700. +## The first production `convergeCodex` -MODIFY the facade at current `src/codex/catalog.ts:6,11-12`: +WP8b declared this function as a type only. WP9 now adds a non-placeholder +implementation in `src/codex/convergence.ts`: ```diff --export { isSpawnableCodexCandidate, codexExecInvocation, loadBundledCodexCatalog, materializeBundledCodexCatalog, loadCatalogTemplate } from "./catalog/bundled"; -+export { isSpawnableCodexCandidate, codexExecInvocation, loadBundledCodexCatalog, materializeBundledCodexCatalog, loadCatalogForSync, loadCatalogTemplate } from "./catalog/bundled"; -@@ --export { MAX_SPAWN_AGENT_MODEL_OVERRIDES, effectiveSubagentRoster, buildCatalogEntries, resetCatalogRuntimeStateForTests, orderForSubagents, mergeCatalogEntriesForSync, syncCatalogModels, restoreCodexCatalog, invalidateCodexModelsCache } from "./catalog/sync"; -+export { MAX_SPAWN_AGENT_MODEL_OVERRIDES, effectiveSubagentRoster, buildCatalogEntries, resetCatalogRuntimeStateForTests, orderForSubagents, mergeCatalogEntriesForSync, prepareCatalogSync, writePreparedCatalogCommit, restoreCodexCatalog, invalidateCodexModelsCache } from "./catalog/sync"; -+export type { PreparedCodexCatalogCommit } from "./catalog/sync"; ++export async function convergeCodex( ++ request: ConvergeRequest, ++): Promise { ++ const admission = inspectAdmissionSnapshot(); ++ if (request.action === "observe") return observeWithoutWrite(admission); ++ ++ const gathered = admission.intent === "on" ++ ? await gatherCodexCatalogCandidate(admission) ++ : null; ++ ++ return coordinateCurrentNativeBehavior({ ++ request, ++ admission, ++ gathered, ++ commitCatalog: commitCodexCatalogCandidate, ++ }); ++} ``` -`syncCatalogModels` is removed only after `rg -n "syncCatalogModels" src tests` -shows every production and test import migrated. This is an intentional internal -contract replacement, not a silent facade break. +`coordinateCurrentNativeBehavior` is real at this commit: it preserves the existing +apply/injection/history behavior and uses the new catalog seam. It is not a throw, +TODO, compatibility path around `convergeCodex`, or promise that WP10/WP12 must land +before WP9 works. Later phases replace mechanisms behind this same entry point. -### `src/codex/refresh.ts` +Desired direction comes only from `admission.intent`; callers pass +`action:"converge"`, never `apply` or `remove`. Desired OFF therefore performs the +current removal path and is not a catalog `skipped` outcome (`005_contract.md` §2). -Replace current lines 1-27 and 34-53 with the types and operations above. The -orchestrator diff is: +## Every management caller uses the funnel -```diff --export async function refreshCodexModelCatalog( -- config: OcxConfig, -- deps: RefreshDeps = defaultDeps, --): Promise { -- const result = await deps.syncCatalogModels(config); -- const catalogExists = deps.existsSync(result.path); -- const catalogWritten = result.catalogWritten === true; -- const comboOmissions = result.comboOmissions ?? []; -- if (!catalogExists) { -- return { ...result, catalogExists, catalogWritten: false, cacheSynced: false, comboOmissions }; -- } -- const cacheSynced = deps.invalidateCodexModelsCache(); -- return { ...result, catalogExists, catalogWritten, cacheSynced, comboOmissions }; -+export async function refreshCodexModelCatalog( -+ config: OcxConfig, -+ deps: CatalogCandidateDeps = defaultCandidateDeps, -+): Promise { -+ try { -+ const candidate = await deps.gatherCodexCatalogCandidate(config); -+ return toRefreshOutcome(deps.commitCodexCatalogCandidate(candidate)); -+ } catch (error) { -+ return catalogGatherFailureOutcome(error); -+ } - } -``` - -`CatalogCandidateDeps` owns only deterministic readers/writers needed to test the -candidate. Production defaults use `prepareCatalogSync`, -`writePreparedCatalogCommit`, `readConfigDiagnostics`, feature-state/path readers, -and SHA-256. Tests inject all filesystem/config observations; no test touches the -user's real home. - -### Management contract and orchestrator - -MODIFY `src/server/management/context.ts` at current lines 9-18 and 54-70: +Delete `refreshCodexCatalogBestEffort` from +`src/server/management-api.ts:105-112` and +`src/server/management/context.ts:54-69`. Replace it with one injected production +funnel: ```diff -+import type { -+ CatalogRefreshDisposition, -+ CodexCatalogCandidate, -+ CodexCatalogCommitResult, -+} from "../../codex/refresh"; -@@ -- refreshCodexCatalog?: () => Promise; -+ codexCatalog?: { -+ gather: (config: OcxConfig) => Promise; -+ commit: (candidate: CodexCatalogCandidate) => CodexCatalogCommitResult; -+ }; -@@ - refreshCodexCatalogBestEffort: () => Promise; -+ refreshCodexCatalogBestEffort: () => Promise; ++ convergeCodex: (request: ConvergeRequest) => Promise; ``` -The pair is one optional object so a test cannot inject gather without commit or -commit without gather. It does not own desired state or locking; WP11 wraps the -same `commit` call at the orchestrator. - -MODIFY `src/server/management-api.ts` at current lines 105-113 and remove the now -dead `CatalogGatherBusyError` route-level mapper at lines 159-163. Best-effort means -non-throwing typed disposition for both production and injected paths: - ```diff - async function refreshCodexCatalogBestEffort(): Promise { - if (deps.refreshCodexCatalog) return deps.refreshCodexCatalog(); - try { - const { refreshCodexModelCatalog } = await import("../codex/refresh"); - await refreshCodexModelCatalog(config); -- } catch { -- /* catalog absent */ -- } -+ async function refreshCodexCatalogBestEffort(): Promise { -+ const refresh = await import("../codex/refresh"); -+ const outcome = deps.codexCatalog -+ ? await refresh.refreshCodexModelCatalog(config, { -+ gatherCodexCatalogCandidate: deps.codexCatalog.gather, -+ commitCodexCatalogCandidate: deps.codexCatalog.commit, -+ }) -+ : await refresh.refreshCodexModelCatalog(config); -+ return refresh.catalogRefreshDisposition(outcome); - } +- } catch { /* catalog absent */ } +- } ++ const converge = deps.convergeCodex ++ ?? (await import("../codex/convergence")).convergeCodex; ``` -### Caller sites - -Every caller captures the result immediately where it currently awaits. The -response-spread pattern is identical; the examples below cover all response shapes. - -MODIFY `src/server/management/provider-routes.ts` current lines 147-148, 338-344, -487-488, and 500-547: +Each of the 16 current awaits — provider 6 +(`src/server/management/provider-routes.ts:147,338,487,512,527,546`), model 6 +(`src/server/management/model-routes.ts:214,313,352,390,404,440`), combo 2 +(`src/server/management/combo-routes.ts:198,216`), and agent settings 2 +(`src/server/management/agent-settings-routes.ts:280,525`) — becomes: ```diff -- await refreshCodexCatalogBestEffort(); -- return jsonResponse({ success: true, name }); -+ const catalogRefresh = await refreshCodexCatalogBestEffort(); -+ return jsonResponse({ success: true, name, catalogRefresh }); -@@ -- await refreshCodexCatalogBestEffort(); -+ const catalogRefresh = await refreshCodexCatalogBestEffort(); - return jsonResponse({ - success: true, - name, - disabled: config.providers[name]!.disabled === true, - hasApiKey: !!config.providers[name]!.apiKey, -+ catalogRefresh, - }); -@@ -- await refreshCodexCatalogBestEffort(); -- return jsonResponse({ success: true, ...(fallbackDefault ? { defaultProvider: fallbackDefault } : {}) }); -+ const catalogRefresh = await refreshCodexCatalogBestEffort(); -+ return jsonResponse({ success: true, ...(fallbackDefault ? { defaultProvider: fallbackDefault } : {}), catalogRefresh }); -@@ -- const respond = () => jsonResponse({ ok: true, cap: DEFAULT_PROVIDER_CONTEXT_CAP, value: globalContextCapValue(config), caps: providerContextCaps(config) }); -+ const respond = async () => { -+ const catalogRefresh = await refreshCodexCatalogBestEffort(); -+ return jsonResponse({ -+ ok: true, cap: DEFAULT_PROVIDER_CONTEXT_CAP, value: globalContextCapValue(config), -+ caps: providerContextCaps(config), catalogRefresh, -+ }); -+ }; -@@ -- await refreshCodexCatalogBestEffort(); -- return respond(); -+ return respond(); +-const catalogRefresh = await refreshCodexCatalogBestEffort(); ++const outcome = await convergeCodex({ ++ action: "converge", ++ reason: "management-mutation", ++ mode: "automatic", ++ deadlineMs: MANAGEMENT_CODEX_CONVERGENCE_DEADLINE_MS, ++}); ++const catalogRefresh = outcomeCatalogDisposition(outcome); ``` -Apply the last two-line replacement to all three cap branches at current lines -512-513, 527-528, and 546-547. - -MODIFY `src/server/management/model-routes.ts` current lines 214-215, 313-314, -352-353, 390-391, 404-405, and 440-441: +`outcomeCatalogDisposition` projects only into the contract-declared +`CatalogDisposition`; it does not define a second management union. Every route +keeps its current 200/201 and the persisted mutation, appends `catalogRefresh`, and +continues unrelated Claude/Desktop work. That is the best-effort behavior promised +by `005_contract.md` §2. -```diff -- await refreshCodexCatalogBestEffort(); -- return jsonResponse({ ok: true, disabled }); -+ const catalogRefresh = await refreshCodexCatalogBestEffort(); -+ return jsonResponse({ ok: true, disabled, catalogRefresh }); -@@ -- await refreshCodexCatalogBestEffort(); -- return jsonResponse({ ok: true, scope, provider, enabled: body.enabled, disabled }); -+ const catalogRefresh = await refreshCodexCatalogBestEffort(); -+ return jsonResponse({ ok: true, scope, provider, enabled: body.enabled, disabled, catalogRefresh }); -@@ -- await refreshCodexCatalogBestEffort(); -- return jsonResponse(entry, 201); -+ const catalogRefresh = await refreshCodexCatalogBestEffort(); -+ return jsonResponse({ ...entry, catalogRefresh }, 201); -@@ -- await refreshCodexCatalogBestEffort(); -- return jsonResponse(cm); -+ const catalogRefresh = await refreshCodexCatalogBestEffort(); -+ return jsonResponse({ ...cm, catalogRefresh }); -@@ -- await refreshCodexCatalogBestEffort(); -- return jsonResponse({ ok: true }); -+ const catalogRefresh = await refreshCodexCatalogBestEffort(); -+ return jsonResponse({ ok: true, catalogRefresh }); -@@ -- await refreshCodexCatalogBestEffort(); -- return jsonResponse({ ok: true, provider, selected: models }); -+ const catalogRefresh = await refreshCodexCatalogBestEffort(); -+ return jsonResponse({ ok: true, provider, selected: models, catalogRefresh }); -``` +## Explicit sync consumes the adapter -MODIFY `src/server/management/combo-routes.ts` current lines 198-200 and 216-217. -Capture before independent follow-up work, but do not return early: +The old status table and manual `Retry-After` logic are deleted. The contract owns +them in `005_contract.md` §5. `src/server/management/config-routes.ts:261-268` +only invokes the funnel and adapter: ```diff -- await refreshCodexCatalogBestEffort(); -+ const catalogRefresh = await refreshCodexCatalogBestEffort(); - if (shouldSyncClaudeAgentDefs) await syncClaudeAgentDefsBestEffort(); -- return jsonResponse({ success: true, id, model: newPublicModel, combo: normalized }); -+ return jsonResponse({ success: true, id, model: newPublicModel, combo: normalized, catalogRefresh }); -@@ -- await refreshCodexCatalogBestEffort(); -- return jsonResponse({ success: true, id }); -+ const catalogRefresh = await refreshCodexCatalogBestEffort(); -+ return jsonResponse({ success: true, id, catalogRefresh }); + if (url.pathname === "/api/sync" && req.method === "POST") { +- const result = await syncModelsToCodex(undefined, config, null); +- return jsonResponse(result, result.ok ? 200 : 500); ++ const outcome = await convergeCodex({ ++ action: "converge", ++ reason: "api-sync", ++ mode: "explicit", ++ deadlineMs: EXPLICIT_CODEX_CONVERGENCE_DEADLINE_MS, ++ }); ++ return toSyncResponse(outcome); + } ``` -MODIFY `src/server/management/agent-settings-routes.ts` current lines 280-294 and -525-528: +No phase-local route helper chooses status, body, or headers. A new outcome variant +must fail the contract adapter's exhaustive `never` check, not silently take a WP9 +default branch. -```diff -- await refreshCodexCatalogBestEffort(); -+ const catalogRefresh = await refreshCodexCatalogBestEffort(); -@@ - agentsMaxDepthAppliesWhenV2Disabled: !enabled, - warnings, -+ catalogRefresh, - }); -@@ -- await refreshCodexCatalogBestEffort(); -+ const catalogRefresh = await refreshCodexCatalogBestEffort(); - await syncClaudeAgentDefsBestEffort(); - await autoApplyDesktopBestEffort(); -- return jsonResponse({ ok: true, applied: chosen }); -+ return jsonResponse({ ok: true, applied: chosen, catalogRefresh }); -``` +## Tests -### Explicit sync +### Catalog mechanism + +`tests/codex-refresh.test.ts` replaces the all-in-one dependency tests with: + +1. gather uses `AdmissionSnapshot.config`, performs provider/parse/assembly work, + and leaves a real isolated-home recursive manifest byte-identical; +2. commit invokes only the fixed writer list; injected provider/parser/subprocess + functions throw if reached beneath the synchronous boundary; +3. disk failure returns the exact partial receipt and consumes the candidate; +4. a second commit writes nothing; +5. a create-once backup appearing after gather is preserved; +6. provider auth/network degradation stays sanitized and projects through + `ConvergeOutcome.catalogRefresh`. + +### Real generation invalidation — C2/C17 + +The old config/content-hash tests are removed. Activation uses the production +generation owners: + +1. admit/gather A; perform a cooperating persisted config transition A→B→A through + the real config mutation API; commit A and assert generation rejection before + every catalog/cache/backup write; +2. gather A; complete another cooperating native transition with its own `txId`; + assert A's `CommitExpectation` is rejected and the newer bytes survive; +3. retarget a canonical parent once between gather and commit; assert target- + identity rejection and zero writes; +4. document, but do not falsely test as guaranteed, a complete parent-symlink + A→B→A between checks — it is outside C17's contract bound; +5. change an unrelated config field through the real config API and assert the + generation still invalidates the candidate. Generation is transition identity, + not semantic-field equality. + +### Production funnel + +- Extend `tests/codex-convergence-contract.test.ts` to walk the TypeScript module + graph (static imports, dynamic imports, aliases, and re-exports) and prove every + direct writer in `src/codex/internal/catalog-commit.ts` is reachable only from + `convergence.ts`. +- Drive all 16 real management routes with an injected `convergeCodex`, assert one + call using persisted admission rather than the route's captured config, preserve + each primary 2xx/201, and observe the additive `catalogRefresh`. +- A refused/deferred catalog attempt must not suppress combo Claude work or agent + settings Claude/Desktop work. +- Drive `POST /api/sync` and assert exact response behavior through + `toSyncResponse`; do not duplicate the contract's status table in this suite. -MODIFY `src/codex/sync.ts` current lines 9-22 and 83-110. Add -`catalogRefresh?: CodexCatalogRefreshOutcome` to `CodexSyncResult` (the existing -external-provider branch at lines 56-71 performs no catalog attempt); replace the -throw/catch with a switch. `committed` fills the existing booleans from its result. -`committed` with notices sets a warning but continues injection. Ordinary gather -failure, unavailable catalog, and disk failure also continue with -`catalogPathForInjection = undefined`, preserving the pinned fallback. OFF/busy/stale -return `ok:false` before line 110 with the typed outcome attached. +## Verification -MODIFY `src/server/management/config-routes.ts` current lines 261-268: +Static/focused gates for the WP9 commit: -```diff - const result = await syncModelsToCodex(undefined, config, null); -+ const status = explicitSyncHttpStatus(result.catalogRefresh, result.ok); -+ const response = jsonResponse({ - ...attachStaleAppServerHint(result), - ...(result.ok ? {} : { error: result.message }), -- }, result.ok ? 200 : 500); -+ }, status, req, config); -+ if (status === 503) response.headers.set("Retry-After", "1"); -+ return response; +```bash +bun test tests/codex-refresh.test.ts tests/codex-convergence-contract.test.ts +bun test tests/codex-sync-api.test.ts tests/codex-models-cache-invalidate.test.ts +bun test tests/model-visibility-management-api.test.ts tests/management-provider-validation.test.ts tests/combo-management-api.test.ts tests/codex-v2-gate.test.ts +bun run typecheck +bun run test +bun run privacy:scan +bun --cwd docs-site run build ``` -`jsonResponse` currently accepts exactly data, status, request, and config -(`src/server/auth-cors.ts:184-188`), so the header is set on the returned response; -do not invent a fifth argument or silently omit `Retry-After`. - -## Tests - -### `tests/codex-refresh.test.ts` - -Replace the all-in-one dependency tests at current lines 60-216 with split cases: - -1. Gather runs bundled source loading, provider discovery, assembly, serialization, - backup preparation, and cache-wrapper preparation; injected write spies remain - zero before commit. -2. Commit performs only the fixed write list in order. Inject every async/provider/ - parser dependency with a function that throws if called during commit. -3. `catalog_unavailable` returns skipped with no write. -4. Provider HTTP/network fallback produces committed plus a - `provider_degraded/provider_network` notice. -5. Missing OAuth token fallback produces committed plus a - `provider_degraded/provider_auth` notice; token-resolution throw produces - `failed/provider_auth` with no commit. -6. `CatalogGatherBusyError` becomes retryable `skipped/gather_busy`. -7. Catalog write succeeds and cache write fails: `failed/disk`, receipt has - `catalog:true`, `cache:false`, and the candidate is consumed. -8. A second commit of the same candidate returns `stale_candidate` and writes zero. -9. Create-once backup appears after gather: commit skips that backup, writes catalog - and cache, and does not overwrite backup bytes. - -The C2 activation cases are mandatory: - -10. Gather candidate A; change one catalog-affecting persisted config field before - commit; assert `stale_candidate`, all-false receipt, and byte-identical catalog, - cache, and backup directory state. -11. Gather candidate A; replace the base catalog bytes with candidate B's committed - catalog; commit A; assert `stale_candidate` and B's bytes survive. -12. Gather from absent catalog; create a catalog before commit; assert absence versus - presence is a revision mismatch. -13. Change a non-catalog config key such as `shutdownTimeoutMs`; assert the canonical - catalog-config digest is unchanged and commit succeeds. This prevents whole-file - hashing from turning unrelated settings into false contention. - -### Caller and sync tests - -- `tests/model-visibility-management-api.test.ts`: inject gather success and commit - `stale_candidate`; assert HTTP 200, persisted disabled state, and - `catalogRefresh.status === "skipped"`. This is the proof that a best-effort caller - did not become loud. -- `tests/management-provider-validation.test.ts`: retain zero refresh for standalone - default/mode branches and exactly one paired gather/commit for ordinary edits; - assert the response disposition. -- `tests/combo-management-api.test.ts`: DELETE removes the final combo row through - real gather/commit; a disk failure still returns 200 with failed disposition. -- `tests/codex-v2-gate.test.ts`: scalar/feature writes remain applied when commit is - busy or stale; route remains 200. -- `tests/codex-sync-api.test.ts`: provider network/auth and ordinary disk failures - still invoke injection; desired OFF, gather busy, lock busy, and stale candidate - do not. Assert the exact `CodexSyncResult.catalogRefresh` in every branch. -- `tests/codex-models-cache-invalidate.test.ts`: app-server restart hint remains - keyed to `receipt.catalog || receipt.cache`, including partial commit. -- Fixture-only files listed in IN compile with the paired seam and never touch the - real home. - -## Verification +Runtime proof uses temporary `OPENCODEX_HOME`/`CODEX_HOME` and port `0`. It never +starts, stops, syncs, restores, or ensures the live proxy on port 10100. -Static gates: - -1. `bun test tests/codex-refresh.test.ts tests/codex-sync-api.test.ts tests/codex-models-cache-invalidate.test.ts` -2. `bun test tests/model-visibility-management-api.test.ts tests/management-provider-validation.test.ts tests/combo-management-api.test.ts tests/combos.test.ts tests/codex-v2-gate.test.ts` -3. `bun run typecheck` -4. `bun run test` -5. `bun run privacy:scan` -6. `bun --cwd docs-site run build` - -The live proxy on 10100 is not used, restarted, synced, restored, ensured, or -stopped. Runtime proof uses isolated temporary homes and a separate process/port: - -1. Create temporary `OPENCODEX_HOME` and `CODEX_HOME`, seed a known base catalog, - and run a test harness that calls `gatherCodexCatalogCandidate` only. Before/after - recursive file manifests must be byte-identical. This proves gather has no native - writes, not merely that mocks saw none. -2. In the same isolated harness, mutate persisted catalog-affecting config after - gather and call commit. Observe `stale_candidate`, all-false receipt, and - byte-identical catalog/cache/backups. Then regather and commit; parse the catalog - and expired cache wrapper and assert the wrapper models equal candidate catalog - models. -3. Start an isolated proxy on a non-10100 ephemeral port with the same temporary - homes. Send one management visibility mutation while the injected commit returns - `stale_candidate`; observe HTTP 200, persisted mutation, and skipped disposition. - Send explicit `/api/sync` with injected `lock_busy`; observe 503 plus - `Retry-After: 1` and no injection write. -4. Record the before/after manifest and JSON responses in the WP9 completion section - before moving this unit to `_fin/`. A green suite without the fired stale branch - does not satisfy C2. +1. Gather only; compare full before/after manifests. +2. Fire the real config generation A→B→A transition; observe zero native writes. +3. Regather and converge; parse catalog and cache bytes and verify the cache models + match the committed catalog. +4. Drive one best-effort management mutation through the real server boundary; + observe its primary 2xx and contract disposition. +5. Drive explicit sync through `convergeCodex` and `toSyncResponse`. ## Accept criteria -- C1 (`000_plan.md:74-75`) — `gatherCodexCatalogCandidate` performs discovery, - loading, assembly, serialization, cache-wrapper construction, and backup - preparation with a byte-identical isolated-home manifest; commit is synchronous - and restricted to the fixed prepared write set. Every failure/skip is represented - by `CodexCatalogRefreshOutcome`, and all 16 management callers report a public - disposition while preserving their current primary success semantics. -- C2 (`000_plan.md:76-77`) — config digest mismatch, base-catalog digest/absence - mismatch, target-home/path mismatch, and candidate reuse all return - `stale_candidate` before any write. Tests activate both config and base-catalog - changes and prove the newer bytes survive. WP11 later places this same synchronous - compare-and-commit operation under the shared lock to close the remaining - cross-process check/write window; WP9 neither assumes nor fabricates that lock. +- **C1** — gather is write-free and commit is synchronous/fixed. Catalog failures + remain catalog-private until projected through `ConvergeOutcome`; all 16 callers + preserve their primary success behavior and expose the contract disposition. +- **C2 / C17 (contract-scoped)** — real config/native generation changes and + single-direction target-identity drift reject before write. No content hash or + path string is presented as arbitrary filesystem ABA protection. +- **C14** — the production module graph proves all 16 management callers funnel + through `convergeCodex`, and no other importer reaches the direct catalog writers. +- **N2** — the WP9 commit contains the first working `convergeCodex`, rewires its + callers in that same commit, passes typecheck, and preserves current behavior. + It has no placeholder whose correctness depends on WP10-WP12. diff --git a/devlog/_plan/260804_codex_write_substrate/020_history_isolation.md b/devlog/_plan/260804_codex_write_substrate/020_history_isolation.md index dec9ef236..bb5c82451 100644 --- a/devlog/_plan/260804_codex_write_substrate/020_history_isolation.md +++ b/devlog/_plan/260804_codex_write_substrate/020_history_isolation.md @@ -1,138 +1,135 @@ # WP10 — history isolation: one client turning off cannot freeze every client -Research: `002_history_off_the_loop.md`. Read it first; this doc is the diff. - -Today, a server-side native restore enters `syncCodexHistoryProvider("openai")` -on the listener thread before `/api/stop` schedules drain, so a Codex SQLite -writer lock can hold the proxy for roughly 10.5 seconds and a successful -row/rollout traversal has no finite work bound at all -(`src/server/management-api.ts:167-194`, `src/codex/inject.ts:759-794`, -`src/codex/history-provider.ts:526-699`). That is the incident: turning one -client off can stop every other client, which is the exact opposite of the -integration switch's purpose -(`../260803_codex_desktop_toggle/008_audit_synthesis_wp4_r2.md:25-30`). This -phase adds an owned Bun Worker boundary for every history operation executed by -the server process, a fail-fast automatic SQLite mode, and a durable unresolved -history fact. Explicit CLI-process recovery keeps the existing synchronous -budget because blocking its own terminal does not starve the proxy. This phase -does not add the WP11 native-write lock and does not depend on it. +Research: `002_history_off_the_loop.md`. Shared contract: `005_contract.md`. + +Today, native apply/restore performs synchronous SQLite, manifest, rollout, and +fsync work on the caller thread (`src/codex/inject.ts:602,764-794`, +`src/codex/history-provider.ts:565-699`). The manifest and rollout writes are +outside the provider's SQLite transaction: apply writes manifest and rollouts +before the DB transaction (`src/codex/history-provider.ts:606-648`); restore writes +rollouts, then DB, then manifest, then performs a second ejection +(`src/codex/history-provider.ts:656-695`). A SQLite busy timeout therefore explains +only one stall and serializes only one part of the state transition. + +The previous WP10 moved server work to a Worker but left explicit CLI work inline, +claimed there was no cross-process history lock, and owned a second +`integrations/codex.json` schema. Round 2 showed all three are one failure: an +opposite-direction process can overtake the Worker through the unguarded files, and +the CLI can skip the only exclusion path. This rewrite consumes the contract's +sibling history-lock protocol and record section (`005_contract.md` §§1, 3, 6). + +WP10 is independently landable. WP8b already supplies the record updater, shared +types, generations, and user-identity resolver; WP9 supplies the working +`convergeCodex`. WP10 adds the real history lock and Worker implementation in the +same commit that routes every history caller through it. It does not wait for the +WP11 native lock or WP12 provenance implementation to typecheck or preserve +behavior. + +All current-code citations and diff context below were rechecked on 2026-08-04 at +`2d5e080dea3e7000bf2111b381c7c1a3c4f5fb11`. ## IN / OUT IN: -- `src/codex/history-provider.ts` (MODIFY) — make busy timeout and retry mode - explicit per invocation, and preserve a classified recoverable failure. -- `src/codex/history-convergence.ts` (NEW) — own - `getConfigDir()/integrations/codex.json`, its schema, fail-closed updates, and - user-facing history status. -- `src/codex/history-worker.ts` (NEW) — Worker entry point; set the captured - environment before dynamically importing the history provider. -- `src/codex/history-job.ts` (NEW) — per-state-DB single flight, Worker IPC, - watchdog, close tracking, and durable state transitions. -- `src/codex/inject.ts` (MODIFY) — await the injected history executor and split - synchronous CLI restore from asynchronous server restore. -- `src/codex/sync.ts` (MODIFY) — carry an explicit `inline | worker` execution - choice to injection; default remains CLI-safe `inline`. -- `src/codex/history-migration-guardian.ts` (MODIFY) — schedule the Worker job; - never call the synchronous probe or mutation on a daemon tick. -- `src/cli/index.ts` (MODIFY) — startup selects `worker`; explicit `sync`, - `restore`, `eject`, `ensure`, and service-command processes retain `inline`. -- `src/server/management/context.ts` (MODIFY) — add the existing-style sync seam - needed to drive a real management request deterministically in the liveness - test (`src/server/management/context.ts:9-50`). -- `src/server/management/config-routes.ts` (MODIFY) — `/api/sync` selects Worker - execution and `GET /api/codex/history` exposes the durable fact. -- `src/server/management-api.ts` (MODIFY) — `/api/stop` awaits the server-safe - restore and returns pending/blocked history honestly before scheduling drain. -- `src/server/lifecycle.ts` (MODIFY) — cancel, join, and persist cancellation for - a history Worker before listener teardown, beside the storage Worker joins - (`src/server/lifecycle.ts:407-445`). -- `src/cli/doctor.ts` (MODIFY) — combine the live read-only probe with the durable - reason, attempt time, and next retry (`src/cli/doctor.ts:891-902`). -- `tests/codex-history-provider.test.ts` (MODIFY), - `tests/history-migration-guardian.test.ts` (MODIFY), - `tests/codex-sync-api.test.ts` (MODIFY), and - `tests/shutdown-drain.test.ts` (MODIFY) — pin changed contracts in their - existing owners. -- `tests/codex-history-worker.test.ts` (NEW), - `tests/codex-history-convergence.test.ts` (NEW), - `tests/codex-history-worker-responsive.test.ts` (NEW), and - `tests/codex-history-process-routing.test.ts` (NEW) — isolate Worker parity, - durable retry truth, measured server liveness, and process routing. +- `src/codex/history-provider.ts` (MODIFY) — invocation-local retry policy, + classified internal failures, shared state-DB identity/path resolver, and a + post-probe callable while the history lock is still held. +- `src/codex/history-worker.ts` (NEW) — Worker entry point; applies captured homes, + acquires the sibling cross-process history lock, rejects overtaken work, performs + the entire history unit, probes, records, and releases. +- `src/codex/history-job.ts` (NEW) — request validation, Worker IPC/watchdog/join, + history-lock target construction, capped retry scheduling, and conversion of job + facts to the contract's `CodexHistoryState`. +- `src/codex/convergence.ts` (MODIFY) — add history execution behind the existing + `convergeCodex`; callers still use only the contract request/result. +- `src/codex/integration-record.ts` (MODIFY only through its public updater) — no + schema change. WP10 calls `updateIntegrationRecord` to write the optional + `history` section and native expected transition atomically. +- `src/codex/inject.ts`, `src/codex/sync.ts` (MODIFY) — remove direct history + execution paths and return their current non-history receipts to convergence. +- `src/codex/history-migration-guardian.ts` (MODIFY) — schedule convergence from + durable state; never probe or mutate history on the listener thread. +- `src/cli/index.ts`, `src/cli/models.ts`, `src/cli/provider.ts`, `src/cli/v2.ts`, + `src/service.ts` (MODIFY where they currently trigger Codex native/history work) + — startup, explicit CLI, stop/uninstall, retry, and ensure use `convergeCodex`. +- `src/server/management-api.ts`, `src/server/management/config-routes.ts`, + `src/server/lifecycle.ts` (MODIFY) — server work uses the same funnel and awaits + Worker termination during drain. +- `src/cli/doctor.ts` (MODIFY) — combine a live read-only probe with the contract + history section. +- `tests/codex-history-provider.test.ts`, + `tests/history-migration-guardian.test.ts`, + `tests/codex-sync-api.test.ts`, and `tests/shutdown-drain.test.ts` (MODIFY), plus + `tests/codex-history-worker.test.ts`, + `tests/codex-history-worker-responsive.test.ts`, + `tests/codex-history-process-routing.test.ts` (NEW). OUT: -- `gui/**` — this substrate exposes a truthful management status; the switch UI - belongs to the later Codex-toggle unit (`000_plan.md:20-22`, - `000_plan.md:62-70`). -- `docs-site/**` — no user-facing switch or configuration key ships in WP10. - The later toggle phase documents the final control surface. -- `src/codex/history-provider.ts` traversal/chunking — batching does not create a - finite bound for row count, rollout bytes, file count, or fsync latency; the - Worker boundary is the availability fix - (`src/codex/history-provider.ts:581-699`, - `002_history_off_the_loop.md:474-486`). -- `src/storage/worker-lifecycle.ts` — history has a different resource key and - job state. Sharing the storage reservation would let a cleanup spawn terminate - or serialize behind unrelated history work (`src/storage/worker-lifecycle.ts:40-50`, - `src/storage/worker-lifecycle.ts:123-143`). WP10 copies no storage mutation - authority; it reuses its close/join discipline in a dedicated controller. -- `src/codex/lock*`, lock files, lock directories, and WP11 protocol — no native - write lock exists yet. The only lock used here is the already-shipped, - zero-wait `withConfigMutationLockSync` around the small - `integrations/codex.json` read-modify-write, not around Codex files, SQLite, or - Worker execution (`src/config.ts:1767-1808`). If that state write cannot acquire - immediately, the history mutation is not dispatched. -- History mutation authority — the Worker executes only after its caller's - current authority checks. WP12 will strengthen that admission; moving code to - another thread is not permission to write (`002_history_off_the_loop.md:272-276`). -- Subprocess isolation — Bun 1.3.14 is pinned in CI - (`.github/workflows/ci.yml:220-222`) and the repository already runs synchronous - SQLite/filesystem work in TypeScript Workers (`src/storage/restore-job.ts:156-234`). - A subprocess is fallback work only if Worker teardown proves a history-specific - Bun defect. +- Any `integrations/codex.json` path, version, parser, merge algorithm, or schema. + `src/codex/integration-record.ts` and `CodexHistoryState` are owned by + `005_contract.md` §1. The former `history-convergence.ts` schema owner is deleted + from this plan. +- The claim that no cross-process history lock exists. WP10 owns its implementation + now because the history unit is not safe without it. +- The native lock and its namespace mechanics — WP11. The history lock is a sibling, + never a nested substitute (`005_contract.md` §6). +- `/api/sync` status, body, or header mapping — `toSyncResponse` owns that contract + (`005_contract.md` §5). +- Ownership/provenance/desired-state policy — WP12. A Worker receives an authority + snapshot identity; it does not invent authority. +- Traversal chunking, GUI, release/deploy operations, and the live proxy on 10100. ## Worker boundary -### Why this is a Worker, and what Bun actually guarantees +The Worker contains the whole mutable history unit: -Bun Workers run TypeScript/ES modules without a compile step, communicate through -structured-clone `postMessage`, report module-resolution failures through `error`, -and emit `close` when marked terminated. Bun's own documentation also warns that -Worker termination remains experimental and that the thread can take time to -fully exit ([Bun Workers](https://bun.sh/docs/runtime/workers)). The repository has -already converted that warning into a stronger local rule: attach `close` at spawn, -do not treat `terminate()` as a join, and wait an OS-settle window on Windows and -macOS (`src/storage/worker-lifecycle.ts:1-17`, -`src/storage/worker-lifecycle.ts:150-209`). WP10 follows that local rule. +1. acquire the sibling cross-process history lock; +2. validate `CommitExpectation` and authority snapshot identity; +3. optional no-op probe; +4. SQLite open, query, transaction, and close; +5. manifest read/write; +6. every rollout read, line-one patch, append, and fsync; +7. final post-probe; +8. update the contract record while still serialized; +9. release the history lock. -The Worker runs the whole history unit, not merely the contended statement: +Moving only `Database` calls is insufficient because the current manifest and +rollout mutations surround the DB transaction (`src/codex/history-provider.ts:606-648,656-695`). +Moving only automatic/server callers is insufficient because the explicit CLI path +currently reaches `restoreNativeCodex()` and `syncModelsToCodex()` directly +(`src/cli/index.ts:528,591,756,768,829`). A lock one caller can skip is not a lock. -1. optional read-only no-op probe; -2. SQLite open, queries, transactions, and close; -3. backup-manifest read/write; -4. every rollout read, line-one patch, append, and fsync; -5. the final read-only pending probe for an `openai` restore. +The server remains responsive because all synchronous/unbounded history work is in +the Worker. Explicit CLI also uses the Worker; its larger wait budget may block its +own terminal, but never the proxy listener and never bypasses cross-process +serialization. -Moving only `Database` calls is insufficient because full JSONL reads, per-file -patches/appends, and fsync are also synchronous and unbounded -(`src/codex/history-provider.ts:67-79`, -`src/codex/history-provider.ts:102-157`, -`src/codex/history-provider.ts:258-274`, -`src/codex/history-provider.ts:581-699`). +## Serializable request and response -### Serializable request and response - -`src/codex/history-worker.ts` accepts one plain-data message: +The request carries the identity of every authority the Worker must revalidate. It +does not carry a mutable config object or a caller-chosen desired direction. ```ts +import type { + CodexHistoryState, + CommitExpectation, +} from "./convergence-types"; + export interface HistoryWorkerRequest { type: "run"; requestId: string; targetProvider: "openai" | "opencodex"; stateDbPath: string; backupPath: string; + lockIdentity: { + userIdentity: UserIdentity; + stateDbId: string; + }; + expectation: CommitExpectation; + /** Digest/id of the AdmissionSnapshot that authorized this transition. */ + authoritySnapshotId: string; busyTimeoutMs: number; attempts: number; delayMs: number; @@ -144,728 +141,346 @@ export type HistoryWorkerResponse = | { type: "done"; requestId: string; - result: CodexHistorySyncResult; - postProbe: PendingHistoryCount | null; + state: CodexHistoryState; + postProbe: PendingHistoryCount; + expectation: CommitExpectation; + authoritySnapshotId: string; } | { type: "error"; requestId: string; - reason: "permission_denied" | "state_unreadable" | "worker_error"; + reason: "db-busy" | "permission" | "worker-died" | "overtaken"; }; ``` -Every crossing value is a string, finite number, null, or plain object containing -those values. No `Database`, `Error`, callback, config object, file handle, or -class instance crosses structured clone. The parent resolves `stateDbPath` and -`backupPath` before spawn. The Worker applies the captured homes and only then -dynamically imports `history-provider.ts`; this avoids the current module-level -`CODEX_HOME` binding selecting a parent test's stale home -(`src/codex/history-provider.ts:16-22`, `src/codex/paths.ts:6-29`). The repository's -storage Worker records the same environment caveat -(`src/storage/restore-worker.ts:16-40`). - -The parent accepts a message only when `requestId` matches and the payload passes a -shape guard. `done` is not automatically `converged`: for target `openai`, -`postProbe` must be non-failed with both `pendingRows === 0` and -`backupEntries === 0` (`src/codex/history-provider.ts:734-775`). A provider result -with `failed: true`, a malformed message, or a failed/nonnull post-probe is durable -unresolved state. - -### Failure, timeout, and death - -**INFERRED design decision:** `src/codex/history-job.ts` owns one active -operation per normalized state-DB id. -Same-target callers join the same Promise; an opposite-target caller gets -`history_operation_busy` and does not overwrite the active attempt. This is an -in-process single flight, not the WP11 cross-process native-write lock. - -The parent resolves outcomes in this order: - -- valid `done` message → classify from mutation result and post-probe; -- valid `error` message → `blocked` with the Worker-provided safe reason; -- `worker.onerror` → `unknown / worker_error`; -- `close` before a valid terminal message → `unknown / worker_died`; -- 10-minute watchdog → terminate and join, then `pending / worker_timeout`; -- shutdown cancellation → terminate and join, then - `pending / shutdown_cancelled`. - -A `done` result with `failureReason: "sqlite_busy"` becomes retryable -`pending / sqlite_busy`; permission becomes `blocked / permission_denied`; a -failed or structurally unknown post-probe becomes `unknown / state_unreadable`. -Only the clean zero/zero post-probe reaches `converged` for target `openai`. - -**INFERRED containment decision:** ten minutes matches the existing storage restore watchdog -(`src/storage/restore-job.ts:40-46`, `src/storage/restore-job.ts:190-215`). It is a -containment deadline, not a claim that history finishes in ten minutes. Because -the work has no finite bound, timeout can interrupt a legitimate large history; -the pre-dispatch durable `pending` fact therefore remains authoritative, the next -startup retries, and the explicit CLI command remains the unbounded operator path. -The Worker closes itself in `finally`; the parent still calls its join helper on -every terminal path, because Bun `close` does not prove immediate OS thread reclaim -(`src/storage/restore-worker.ts:43-55`, -`src/storage/worker-lifecycle.ts:176-199`). - -## Fail-fast automatic mode - -The current writable connection reads one mutable global -`historyDbBusyTimeoutMs = 5000`, and `withHistoryRetry` defaults to two attempts -with `Bun.sleepSync(500)` between them -(`src/codex/history-provider.ts:25-49`, -`src/codex/history-provider.ts:526-548`). WP10 makes the policy explicit: - -| Caller | Execution | SQLite busy timeout | Attempts / delay | Reason | -|---|---|---:|---:|---| -| Server startup, `/api/sync`, `/api/stop`, guardian, future toggle | Worker | **100 ms** | **1 / 0 ms** | Automatic convergence must release the history slot quickly when Codex owns SQLite. The read-only probe already uses 100 ms (`src/codex/history-provider.ts:749-774`). | -| Explicit CLI `restore`, `eject`, `recover-history`, `sync`, `restore back`, `ensure` parent | CLI process, inline | **5,000 ms** | **2 / 500 ms** | The invoking terminal may wait for a transient Codex lock; this preserves today's operator behavior (`src/codex/history-provider.ts:25-49`, `src/codex/history-provider.ts:526-548`). | - -Automatic mode does not call `sleepSync`; its scheduler delay is the retry. The -100 ms budget bounds only lock waiting. It does not and cannot bound a successful -row/file walk; that is why fail-fast without Worker isolation failed the research -gate (`002_history_off_the_loop.md:183-205`). - -## Unresolved history is a durable fact - -### Location and exact shape - -**INFERRED schema decision:** the record is -`join(getConfigDir(), "integrations", "codex.json")`, beneath -`OPENCODEX_HOME`, never `CODEX_HOME`. This reuses the repository's owned -integration directory and atomic-write convention -(`src/integrations/ownership.ts:60-71`, -`src/integrations/ownership.ts:94-106`). It is the one future Codex integration -record, not a second history-only file. WP10 writes `version` and `history`; WP12 -may add desired state and the artifact ledger without moving history. - -```json -{ - "version": 1, - "history": { - "<16-hex normalized state DB id>": { - "stateDbPath": "/absolute/CODEX_HOME/state_5.sqlite", - "backupPath": "/absolute/OPENCODEX_HOME/codex-history-backup-.json", - "targetProvider": "openai", - "state": "pending", - "reason": "sqlite_busy", - "attemptId": "uuid", - "attemptCount": 3, - "lastAttemptAt": "2026-08-04T00:00:00.000Z", - "pendingRows": null, - "backupEntries": 4, - "automaticRetry": true, - "nextRetryAt": "2026-08-04T00:01:00.000Z" - } - } -} -``` +Every crossing value is plain structured-clone data. The parent resolves absolute +paths and lock identity before spawn. The Worker applies captured homes before the +dynamic import because `history-provider.ts:16-22` currently binds path-derived +state at module load. The request guard rejects non-finite/negative numeric fields, +non-absolute paths, malformed identities, invalid expectations, and blank snapshot +ids. + +`requestId` rejects stray messages. `authoritySnapshotId` rejects a job admitted +for different service/external/journal/provenance/intent evidence. The +`CommitExpectation` rejects a transition overtaken after native commit. These are +not optional diagnostics; missing fields make the message invalid and no mutation +starts. + +## One sibling history lock + +`src/codex/history-job.ts` constructs the history lock from the contract-owned +effective-user identity plus normalized state-DB identity. It uses a private, +persistent SQLite transaction with finite async acquisition and no PID/mtime stale +takeover. The Worker acquires it **inside the Worker** and holds it over manifest, +rollouts, DB, final probe, and terminal record update. -`state` is `pending | running | blocked | converged | unknown`. `reason` is null -only for `converged`; otherwise it is one of `sqlite_busy`, `permission_denied`, -`state_unreadable`, `state_write_busy`, `history_operation_busy`, `worker_error`, -`worker_died`, `worker_timeout`, or `shutdown_cancelled`. Counts are nullable: -failed probes mean unknown, never numeric zero. The key uses the same normalized -state-DB hash already used for backup naming -(`src/codex/history-provider.ts:16-22`). - -Before spawn, the parent writes `pending` with a new `attemptId`. After creating an -idle Worker but before posting `run`, it writes `running`. Both updates use the -already-shipped zero-wait config mutation transaction only around read/merge/atomic -write (`src/config.ts:1767-1808`). If either write is busy or fails, the Worker is -not messaged and no history mutation starts. A terminal update applies only when -the stored `attemptId` still matches; an older in-process completion cannot turn a -newer attempt green. If the final state write fails after mutation, the record -stays `running` or `pending`, which is a retryable false negative rather than a -false success. - -`converged` for target `openai` is legal only after the clean post-probe proves -zero pending rows and zero backup entries. Manifest absence alone is insufficient: -the no-backup ejection path can still have work, and a failed probe currently -returns zero-looking counts with `failed: true` -(`src/codex/history-provider.ts:656-665`, -`src/codex/history-provider.ts:749-775`). - -### Retry ownership and user visibility - -- A running server retries `pending`, `unknown`, and retryable `blocked` entries - every 60 seconds, at most 60 ticks per process lifetime. The durable record keeps - `automaticRetry: true`; after the in-process budget, `nextRetryAt: null` means - “next proxy startup,” not “abandoned.” This preserves the current finite guardian - cadence while replacing its event-loop mutation - (`src/codex/history-migration-guardian.ts:34-40`, - `src/codex/history-migration-guardian.ts:54-92`). -- Every proxy startup treats persisted `pending`, `running`, `blocked`, or - `unknown` as retryable. A stale `running` state is not proof a Worker survived - its process. -- Explicit CLI recovery runs inline with the full budget, then writes the same - record. It never reports success while the durable entry remains unresolved. - -The user sees the fact in three places. `GET /api/codex/history` returns the entry; -`/api/sync` and `/api/stop` include the same status in their response; and -`ocx doctor` prints, for example: +The native and history locks are siblings: ```text --- Codex resume history unresolved: sqlite_busy - 4 routed thread(s) may remain hidden in native Codex - automatic retry: 2026-08-04T00:01:00.000Z; run `ocx restore` after closing Codex to retry now +native transition: acquire native -> synchronous native commit -> release native +history transition: acquire history -> validate expectation -> mutate/probe/record -> release history ``` -The wording says “may remain hidden” when counts are null. OFF/config restoration -and history convergence are separate facts; no `success: true` envelope may erase -the warning. That fixes the current shape where `restoreNativeCodex` returns -`success: cfg.success` while history failure exists only in message text -(`src/codex/inject.ts:783-794`). +They are never held simultaneously. The history Worker never acquires the native +lock, and the native synchronous callback never spawns/awaits the Worker. This is +the checkable deadlock rule from `005_contract.md` §6. -## Diff +### Overtaking prevention -Line anchors below are against current HEAD `7e67a8d06311de2471b0a25e41cf85f97007cc69`. +Sibling locks alone allow this sequence: A commits native ON, B commits native OFF, +B removes history, then A applies history. The request therefore carries A's +`CommitExpectation`. -### `src/codex/history-provider.ts` +Immediately after taking the history lock and before any probe or mutation, the +Worker reads the integration record. The job is legal only when the record still +names the transition expected by the request. If another native transition has +advanced the generation/transaction identity, the Worker returns +`CodexHistoryState { status:"pending", reason:"overtaken", ... }`, performs no +history write, and does **not** retry itself. The winning/newer transition owns the +next convergence. -Make writable busy timeout invocation-local while preserving the test override as -the explicit-mode default: +The final post-probe and record update happen before release. A clean mutation +followed by an unlocked probe is not evidence: another process could change rows or +the manifest in between. For target `openai`, `converged` requires a non-failed +probe with `pendingRows === 0` and `backupEntries === 0`; manifest absence or a +zero-row mutation alone is insufficient (`src/codex/history-provider.ts:749-775`). -```diff - let historyDbBusyTimeoutMs = 5000; -+export const AUTOMATIC_HISTORY_DB_BUSY_TIMEOUT_MS = 100; -@@ --function openStateDb(stateDbPath: string): Database { -+function openStateDb(stateDbPath: string, busyTimeoutMs = historyDbBusyTimeoutMs): Database { - const db = new Database(stateDbPath); - try { -- db.exec(`PRAGMA busy_timeout = ${historyDbBusyTimeoutMs}`); -+ db.exec(`PRAGMA busy_timeout = ${Math.max(0, Math.floor(busyTimeoutMs))}`); -``` +## Failure, timeout, and death -Extend the existing result and options; do not replace `failed`, because current -callers and tests use it (`src/codex/history-provider.ts:160-168`, -`src/codex/history-provider.ts:565-579`): - -```diff - export interface CodexHistorySyncResult { - rows: number; - files: number; - ejectedRows?: number; - failed?: true; -+ failureReason?: "sqlite_busy" | "permission_denied" | "state_unreadable"; - } -+ -+export interface HistoryExecutionOptions { -+ skipWhenProvablyNoop?: boolean; -+ busyTimeoutMs?: number; -+ attempts?: number; -+ delayMs?: number; -+ sleepFn?: (ms: number) => void; -+} -@@ - export function syncCodexHistoryProvider( - provider: CodexHistoryProvider, - stateDbPath = STATE_DB_PATH, - backupPath = HISTORY_BACKUP_PATH, -- opts: { skipWhenProvablyNoop?: boolean } = {}, -+ opts: HistoryExecutionOptions = {}, - ): CodexHistorySyncResult { -@@ -- return withHistoryRetry(() => syncCodexHistoryProviderUnsafe(provider, stateDbPath, backupPath)) -- ?? { rows: 0, files: 0, failed: true }; -+ let failureReason: CodexHistorySyncResult["failureReason"]; -+ const result = withHistoryRetry( -+ () => syncCodexHistoryProviderUnsafe(provider, stateDbPath, backupPath, opts.busyTimeoutMs), -+ { -+ attempts: opts.attempts, -+ delayMs: opts.delayMs, -+ sleepFn: opts.sleepFn, -+ onRecoverableError: error => { failureReason = classifyHistoryFailure(error); }, -+ }, -+ ); -+ return result ?? { rows: 0, files: 0, failed: true, failureReason: failureReason ?? "state_unreadable" }; - } -@@ --function syncCodexHistoryProviderUnsafe(provider: CodexHistoryProvider, stateDbPath: string, backupPath: string): CodexHistorySyncResult { -+function syncCodexHistoryProviderUnsafe(provider: CodexHistoryProvider, stateDbPath: string, backupPath: string, busyTimeoutMs?: number): CodexHistorySyncResult { -@@ -- const db = openStateDb(stateDbPath); -+ const db = openStateDb(stateDbPath, busyTimeoutMs); -``` +The parent owns one process-local Worker flight only to avoid duplicate threads; +cross-process exclusion comes from the sibling lock, not this map. Same-transition +callers may join. Opposite transitions do not overwrite each other: each reaches +the lock and the older one is rejected by its expectation. -Apply the same `busyTimeoutMs` parameter to the restore-side `openStateDb` at -`src/codex/history-provider.ts:660` and to `migrateHistoryToOpenai` at -`src/codex/history-provider.ts:719-731`. Add -`onRecoverableError?: (error: unknown) => void` to `withHistoryRetry`'s `io` -parameter and call it immediately after the recoverability check at -`src/codex/history-provider.ts:544`, before the attempts check. -`classifyHistoryFailure` maps SQLite busy/locked to -`sqlite_busy`, `EPERM`/`EACCES`/permission text to `permission_denied`, and the -remaining recoverable class to `state_unreadable`; hard errors still throw. Existing -default behavior remains 5,000 ms, two attempts, and 500 ms delay -(`src/codex/history-provider.ts:511-548`). +Outcome order: -### `src/codex/history-worker.ts` (NEW) +- valid `done` + clean under-lock post-probe -> contract `converged` state; +- SQLite/history-lock busy -> `pending/db-busy` with next retry; +- permission/refusal -> `blocked/permission`; +- expectation/snapshot mismatch -> `pending/overtaken`, no self-retry; +- `worker.onerror`, malformed terminal message, early close, or watchdog -> + `unknown/worker-died`; +- shutdown cancellation -> persist non-converged state, join, then drain. -Implement the message contract above. The critical order is: +The Worker closes in `finally`; the parent still waits for `close`/join using the +repository's existing discipline (`src/storage/worker-lifecycle.ts:150-209`). A +watchdog is containment, not convergence. It may interrupt legitimate large +history, so timeout can never be recorded as success. -```ts -self.onmessage = async (event: MessageEvent) => { - if (!isHistoryWorkerRequest(event.data)) return; - const request = event.data; - try { - if (request.env.CODEX_HOME) process.env.CODEX_HOME = request.env.CODEX_HOME; - if (request.env.OPENCODEX_HOME) process.env.OPENCODEX_HOME = request.env.OPENCODEX_HOME; - const { countPendingOpencodexHistory, syncCodexHistoryProvider } = await import("./history-provider"); - const result = syncCodexHistoryProvider(request.targetProvider, request.stateDbPath, request.backupPath, { - busyTimeoutMs: request.busyTimeoutMs, - attempts: request.attempts, - delayMs: request.delayMs, - skipWhenProvablyNoop: request.skipWhenProvablyNoop, - }); - const postProbe = request.targetProvider === "openai" - ? countPendingOpencodexHistory(request.stateDbPath, request.backupPath) - : null; - self.postMessage({ type: "done", requestId: request.requestId, result, postProbe }); - } catch (error) { - self.postMessage({ - type: "error", - requestId: request.requestId, - reason: classifyWorkerThrownError(error), - }); - } finally { - try { (self as unknown as { close?: () => void }).close?.(); } catch {} - } -}; -``` +## Fail-fast automatic mode and explicit mode -`classifyWorkerThrownError` emits only the reason enum, not raw error strings or -paths. The request guard rejects non-finite/negative numeric policy fields and -non-absolute paths. +The provider currently uses a mutable global 5,000 ms busy timeout and two retries +with a synchronous 500 ms sleep (`src/codex/history-provider.ts:25-49,526-548`). +Make the policy invocation-local: -### `src/codex/history-convergence.ts` and `src/codex/history-job.ts` (NEW) +| Caller mode | Worker lock / SQLite wait | Attempts / delay | Reason | +|---|---:|---:|---| +| automatic (startup, management, guardian, stop) | 100 ms | 1 / 0 ms | Defer quickly; listener availability is the requirement. | +| explicit CLI | 5,000 ms | 2 / 500 ms | Preserve today's operator wait budget, but inside the Worker and under the same lock. | -Export the normalized identity/path resolver from `history-provider.ts` so the -job, state record, and backup file cannot implement three subtly different hashes: +Automatic mode never calls `sleepSync` on the parent. Explicit delay may use +`sleepSync` inside the Worker because it cannot starve the proxy or bypass the +history lock. ```diff --function historyBackupPathFor(stateDbPath: string): string { -+export function historyStateDbId(stateDbPath: string): string { - const normalized = process.platform === "win32" ? resolve(stateDbPath).toLowerCase() : resolve(stateDbPath); -- const id = createHash("sha256").update(normalized).digest("hex").slice(0, 16); -- return join(getConfigDir(), `codex-history-backup-${id}.json`); -+ return createHash("sha256").update(normalized).digest("hex").slice(0, 16); + export interface HistoryExecutionOptions { + skipWhenProvablyNoop?: boolean; ++ busyTimeoutMs?: number; ++ attempts?: number; ++ delayMs?: number; ++ sleepFn?: (ms: number) => void; } -+function historyBackupPathFor(stateDbPath: string): string { -+ return join(getConfigDir(), `codex-history-backup-${historyStateDbId(stateDbPath)}.json`); -+} -+export function resolveCodexHistoryPaths(stateDbPath = STATE_DB_PATH): { stateDbPath: string; backupPath: string } { -+ return { stateDbPath: resolve(stateDbPath), backupPath: historyBackupPathFor(stateDbPath) }; -+} ``` -Also export `CodexHistoryProvider`; the Worker and job import the owner type instead -of restating a parallel union. +Apply `busyTimeoutMs` to both apply and restore database opens. Keep hard errors +throwing inside the Worker so its boundary can classify them once; do not turn +programming/data corruption into `db-busy`. -`history-convergence.ts` owns the schema and only these operations: +## Durable state consumes the contract record -```ts -historyStateDbId(stateDbPath: string): string; -readCodexHistoryConvergence(stateDbPath?: string): HistoryConvergenceEntry | null; -beginCodexHistoryAttempt(input: AttemptInput): HistoryConvergenceEntry; -markCodexHistoryAttemptRunning(attemptId: string): HistoryConvergenceEntry; -finishCodexHistoryAttempt(attemptId: string, outcome: HistoryJobOutcome): HistoryConvergenceEntry; -``` +Delete the former “Location and exact shape” JSON and the planned +`src/codex/history-convergence.ts`. The path, top-level version, extension policy, +and section schema belong to `005_contract.md` §1. -All three writes execute a synchronous, no-await callback inside -`withConfigMutationLockSync`; malformed/unknown-version `codex.json` is -`state_unreadable` and is preserved, not replaced. `begin` fails before Worker -dispatch if the record cannot be durably written. `finish` compares `attemptId` -inside the transaction and leaves a newer entry untouched. - -`history-job.ts` exports: +Both `history-worker.ts` and `history-job.ts` import: ```ts -export type HistoryExecution = "automatic" | "explicit"; -export function runCodexHistoryJob(input: { - targetProvider: "openai" | "opencodex"; - execution: HistoryExecution; - stateDbPath?: string; - backupPath?: string; - skipWhenProvablyNoop?: boolean; -}): Promise; -export function runCodexHistoryInline(input: { - targetProvider: "openai" | "opencodex"; - stateDbPath?: string; - backupPath?: string; - skipWhenProvablyNoop?: boolean; -}): HistoryJobOutcome; -export function runLegacyCodexHistoryRecoveryInline(input?: { - stateDbPath?: string; -}): HistoryJobOutcome; -export function abortCodexHistoryJobAsync(): Promise; -export function setCodexHistoryJobTestHooks(hooks: { - automaticBusyTimeoutMs?: number; - workerTimeoutMs?: number; -} | null): void; +import type { CodexHistoryState } from "./convergence-types"; +import { + readIntegrationRecord, + updateIntegrationRecord, +} from "./integration-record"; ``` -`automatic` always spawns `new Worker(new URL("./history-worker.ts", -import.meta.url).href)`, sends 100/1/0, and uses the durable transitions above. -`runCodexHistoryInline` invokes `syncCodexHistoryProvider` in the caller process -with defaults, then writes the same terminal state; the async job delegates to it -for `explicit`. The test hooks change timing only; there is no -`runInProcess` hook in the liveness test because that would make C3 vacuous. +They never parse or atomically replace `integrations/codex.json` themselves. A +state transition is one `updateIntegrationRecord(record => ({ ...record, history: +next }))`; unknown keys and the provenance section survive. Corrupt/unparseable +records fail closed. `txId` links the state to the native transition and +`nextRetryAt:null` means only “no timer armed now,” never “never again.” -### `src/codex/inject.ts` +The durable contract has no per-state-DB schema invented here. If multiple state +DBs need internal scheduling metadata, it remains an in-memory/job-private map; +the shared `CodexHistoryState` is the current convergence fact exposed to every +consumer. -Extend `InjectCodexOptions` at `src/codex/inject.ts:66-73` and replace the direct -call at `src/codex/inject.ts:601-603`: +## Retry ownership — no permanent dormancy -```diff - export interface InjectCodexOptions { - catalogPath?: string | null; -+ historyExecution?: "automatic" | "explicit"; - } -@@ -- const history = config?.syncResumeHistory !== false -- ? (legacyMode ? syncCodexHistoryProvider("opencodex") : migrateHistoryToOpenai()) -+ const history = config?.syncResumeHistory !== false -+ ? await runCodexHistoryJob({ -+ targetProvider: legacyMode ? "opencodex" : "openai", -+ execution: options.historyExecution ?? "explicit", -+ }) - : { rows: 0, files: 0 }; -``` +Delete “every 60 seconds, at most 60 ticks per process” and the interpretation of +`nextRetryAt:null` as next-startup-only. That creates permanent dormancy in a +long-lived process (carried finding #9). -Factor `src/codex/inject.ts:765-783` into `prepareNativeCodexRestore()` -(external-provider guard, journal/config/catalog work, and -`skipWhenProvablyNoop`) and `src/codex/inject.ts:784-794` into -`finishNativeCodexRestore(prepared, history)`. The public synchronous CLI contract -remains, while the server gets an async sibling: +**INFERRED scheduling choice:** the guardian uses capped exponential backoff with +deterministic testable jitter: -```diff - export function restoreNativeCodex(): { success: boolean; message: string } { -- const activeProvider = currentExternalCodexModelProvider(); -- // ... current config/catalog setup ... -- const history = syncCodexHistoryProvider("openai", undefined, undefined, { skipWhenProvablyNoop }); -- // ... current message formatting ... -+ const prepared = prepareNativeCodexRestore(); -+ if (prepared.done) return prepared.result; -+ const history = runCodexHistoryInline({ targetProvider: "openai", skipWhenProvablyNoop: prepared.skipWhenProvablyNoop }); -+ return finishNativeCodexRestore(prepared, history); - } -+ -+/** Exit-hook fallback: restore bounded config/catalog state and leave history unresolved. */ -+export function restoreNativeCodexWithoutHistory(): CodexRestoreResult { -+ const prepared = prepareNativeCodexRestore(); -+ if (prepared.done) return prepared.result; -+ return finishNativeCodexRestore(prepared, preserveConvergedOrLeaveCodexHistoryPending("shutdown_cancelled")); -+} -+ -+export async function restoreNativeCodexInServer(): Promise { -+ const prepared = prepareNativeCodexRestore(); -+ if (prepared.done) return prepared.result; -+ const history = await runCodexHistoryJob({ -+ targetProvider: "openai", -+ execution: "automatic", -+ skipWhenProvablyNoop: prepared.skipWhenProvablyNoop, -+ }); -+ return finishNativeCodexRestore(prepared, history); -+} +```text +delay(attempt) = min(MAX_HISTORY_RETRY_MS, + BASE_HISTORY_RETRY_MS * 2^min(attempt, BACKOFF_EXPONENT_CAP)) ``` -`CodexRestoreResult` adds `history: HistoryConvergenceEntry | null`. Its `success` -continues to describe config/catalog restoration for compatibility, but every -caller must render `history.state !== "converged"` separately; the formatter keeps -the hidden-thread warning from `src/codex/inject.ts:787-793`. +It schedules at most one timer and one Worker per current `txId`. It may back off +to the cap but never exhausts into a permanent state. Startup re-arms any unresolved +record whose timer was lost. A successful convergence clears the timer. An +`overtaken` job does not retry the losing transition; it schedules one observation +of the current generation so the winner owns work. -### Process-aware callers +This loop has a finite delay per attempt and no finite lifetime attempt count. +Shutdown cancels the current timer/Worker and leaves durable unresolved state for +the next process. -`syncModelsToCodex` carries an explicit fifth options object rather than inferring -from port or whether a proxy happens to be live; those are not process identity: +## Process-aware callers use `convergeCodex` -```diff - export async function syncModelsToCodex( - port?: number, - config: OcxConfig = loadConfig(), - log: Pick | null = console, - deps: CodexSyncDeps = defaultDeps, -+ options: { historyExecution?: "automatic" | "explicit" } = {}, - ): Promise { -@@ -- const result = await deps.injectCodexConfig(p, config, {}); -+ const result = await deps.injectCodexConfig(p, config, { -+ ...(options.historyExecution ? { historyExecution: options.historyExecution } : {}), -+ }); -@@ -- const result = await deps.injectCodexConfig(p, config, { catalogPath: catalogPathForInjection }); -+ const result = await deps.injectCodexConfig(p, config, { -+ catalogPath: catalogPathForInjection, -+ ...(options.historyExecution ? { historyExecution: options.historyExecution } : {}), -+ }); -``` - -Server callers opt in; CLI callers retain the default: +Delete `runCodexHistoryInline`, `HistoryExecution = "automatic" | "explicit"` as a +public alternate entry point, and every caller selection that bypasses convergence. +Mode is already in `ConvergeRequest`. ```diff - // src/cli/index.ts:318-322 — this is the server process after listen -- await syncModelsToCodex(port).catch(() => {}); -+ await syncModelsToCodex(port, config, console, undefined, { historyExecution: "automatic" }).catch(() => {}); - - // src/server/management/config-routes.ts:261-268 -- const result = await syncModelsToCodex(undefined, config, null); -+ const sync = ctx.deps.syncModelsToCodex ?? syncModelsToCodex; -+ const result = await sync(undefined, config, null, undefined, { historyExecution: "automatic" }); - - // src/server/management-api.ts:167-194 -- const { restoreNativeCodex } = await import("../codex/inject"); -+ const { restoreNativeCodexInServer } = await import("../codex/inject"); -@@ -- const restore = restoreNativeCodex(); -+ const restore = await restoreNativeCodexInServer(); -@@ -- return jsonResponse(restore.success -- ? { success: true, message: `Proxy stopping, native Codex restored.${grokNote}` } -+ return jsonResponse(restore.success -+ ? { success: true, history: restore.history, message: `Proxy stopping, native Codex restored.${historyNote}${grokNote}` } - : { success: false, message: `Proxy stopping, but native Codex restore failed: ${restore.message}. Run \`ocx restore\`.${grokNote}` }); +-const history = syncCodexHistoryProvider("openai", ...); ++const outcome = await convergeCodex({ ++ action: "converge", ++ reason: "cli", ++ mode: "explicit", ++ deadlineMs: EXPLICIT_CODEX_CONVERGENCE_DEADLINE_MS, ++}); ``` -`historyNote` explicitly says routed threads remain hidden when state is not -`converged`. The 200 ms drain timer is scheduled only after the awaited automatic -attempt returns; under lock contention that is one 100 ms Worker attempt. Under a -large uncontended traversal the request may remain pending, but `/healthz` and -data-plane traffic continue; if the request watchdog/shutdown cancels it, durable -state remains pending. - -The guardian's current synchronous `countFn` and `migrateFn` dependencies at -`src/codex/history-migration-guardian.ts:24-31` become async -`readStateFn`/`runJobFn`. Its tick awaits the single-flight job and schedules from -the durable terminal state; it never calls `countPendingOpencodexHistory` or -`migrateHistoryToOpenai` on the server thread (`src/codex/history-migration-guardian.ts:59-83`). - -`drainAndShutdown` adds `abortCodexHistoryJobAsync()` to the `Promise.allSettled` -join group at `src/server/lifecycle.ts:415-418` and logs it under -`[codex-history]`. The abort -function writes `shutdown_cancelled` before resolving. The synchronous -`process.on("exit")` fallback in `src/cli/index.ts:305-310` must never start a Worker -or run history inline; graceful signal paths await the server-safe cleanup before -calling `process.exit`, while the exit fallback can only leave/rewrite unresolved -state. - -The signal/exit caller split is explicit; the synchronous exit hook restores only -bounded config/catalog state, while the graceful async path runs history in a -Worker after drain: +Server startup/management/guardian uses `mode:"automatic"`; explicit CLI sync, +restore, eject, recover-history, ensure, and service cleanup uses +`mode:"explicit"`. Both modes reach the **same Worker and same history lock**. +`src/codex/inject.ts:602,783` loses direct provider calls; it exposes only bounded +native apply/restore receipts to convergence. -```diff - // src/cli/index.ts:242-266 -- const restored = restoreNativeCodex(); -+ const restored = restoreNativeCodexWithoutHistory(); -@@ - // src/cli/index.ts:295-301 - try { - await drainAndShutdown(server, config.shutdownTimeoutMs ?? 5000); - } finally { -+ if (!isRecyclingForExit() && !process.env.OCX_SERVICE && !currentExternalCodexModelProvider()) { -+ const historyRestore = await restoreNativeCodexInServer(); -+ if (!historyRestore.success) cleanupSucceeded = false; -+ } - const restored = syncCleanup(); - process.exit(restored ? 0 : 1); - } +`src/codex/convergence.ts` sequence at this phase is: + +```text +admit current snapshot -> gather if ON -> native commit -> release native section +-> dispatch history Worker(expectation, authoritySnapshotId) -> observe -> outcome ``` -`/api/stop` already awaits `restoreNativeCodexInServer` before its drain timer; -the later exit hook sees idempotently restored config/catalog and does no history -work. A crash that reaches only the synchronous exit hook leaves history pending -for startup instead of freezing exit or pretending convergence. +Automatic calls may return `deferred` with unresolved `history`; explicit calls +wait only through their request deadline. Neither reports `converged` while history +is outstanding. -The fallback helper preserves an already durable `converged` entry. It writes -`shutdown_cancelled` only when history is absent, running, or already unresolved; -an idempotent exit hook must not turn the `/api/stop` Worker's proven zero/zero -result back into a false negative. +The synchronous `process.on("exit")` hook cannot await a Worker. It performs no +history mutation and leaves/records unresolved state; graceful signal and command +paths call convergence before exit. This preserves process shutdown without +inventing an inline escape hatch. -All commands that execute after the proxy is stopped or in a separate CLI process -remain unchanged at their call sites: `handleStop`'s second restore -(`src/cli/index.ts:527-534`), explicit restore/eject -(`src/cli/index.ts:745-776`), service stop/uninstall -(`src/service.ts:2564-2595`, `src/service.ts:2610-2632`). Their synchronous -self-block is intentional. Legacy recovery keeps its narrower operation but routes -through the state-writing inline wrapper: +## Durable read surface -```diff - // src/cli/index.ts:711-724 -- const r = restoreLegacyOpenaiHistory(); -+ const r = runLegacyCodexHistoryRecoveryInline(); -``` +`GET /api/codex/history` may expose the contract record's `history` section through +an authenticated read-only route. It imports `readIntegrationRecord`; it does not +define a second state type. -That wrapper calls the existing `restoreLegacyOpenaiHistory`, performs the same -post-probe, and updates `integrations/codex.json`; it does not broaden legacy -recovery into manifest restore. +`POST /api/sync` is not redefined here. It already calls `convergeCodex` and +`toSyncResponse` after WP9 (`005_contract.md` §5). WP10 only ensures the resulting +`ConvergeOutcome` contains the contract `history` state. `ocx doctor` retains its +live read-only probe because durable state can be stale, but failed probes are +unknown rather than zero-looking success. -### Durable read surface +## Key diffs -Add `syncModelsToCodex?: typeof syncModelsToCodex` to `ManagementApiDeps`, use it -for `/api/sync`, and add this authenticated route beside it: +### Worker owns lock, mutation, post-probe, and record ```diff - if (url.pathname === "/api/sync" && req.method === "POST") { - // worker-aware sync above - } -+ -+ if (url.pathname === "/api/codex/history" && req.method === "GET") { -+ const { readCurrentCodexHistoryConvergence } = await import("../../codex/history-convergence"); -+ return jsonResponse({ history: readCurrentCodexHistoryConvergence() }); ++self.onmessage = async (event: MessageEvent) => { ++ const request = parseHistoryWorkerRequest(event.data); ++ applyCapturedHomes(request.env); ++ const lock = await acquireHistoryLock(request.lockIdentity, requestDeadline(request)); ++ if (lock.status !== "acquired") return postHistoryBusy(request, lock); ++ try { ++ const current = readIntegrationRecord(); ++ if (!expectationStillCurrent(current, request.expectation, request.authoritySnapshotId)) { ++ return postOvertaken(request); ++ } ++ const result = syncCodexHistoryProvider(request.targetProvider, request.stateDbPath, request.backupPath, policy(request)); ++ const postProbe = countPendingOpencodexHistory(request.stateDbPath, request.backupPath); ++ const state = classifyHistoryState(result, postProbe, request.expectation.txId); ++ updateIntegrationRecord(record => ({ ...record, history: state })); ++ self.postMessage({ type: "done", requestId: request.requestId, state, postProbe, expectation: request.expectation, authoritySnapshotId: request.authoritySnapshotId }); ++ } finally { ++ lock.release(); ++ closeWorker(); + } ++}; ``` -`ocx doctor` keeps its live read-only probe, because the durable fact can be stale, -but it treats a failed probe as unknown and prints durable reason/retry metadata -instead of the current generic locked-or-unreadable line -(`src/cli/doctor.ts:891-902`). +`release()` above is private to Worker implementation; unlike the native public +API, no caller can retain it across unrelated work. + +### Convergence dispatch, no inline branch + +```diff +-historyExecution === "explicit" +- ? runCodexHistoryInline(input) +- : runCodexHistoryJob(input) ++await runCodexHistoryJob({ ++ ...input, ++ mode: request.mode, ++ expectation, ++ authoritySnapshotId: admittedSnapshotId(admission), ++}) +``` ## Test plan -### C3 — real SQLite contention with measured `/healthz` - -`tests/codex-history-worker-responsive.test.ts` is a server-boundary test, not a -mocked busy-error unit test: - -1. Install isolated `CODEX_HOME` and `OPENCODEX_HOME` with - `installIsolatedCodexHome`; create a production-shaped `threads` table, one - interactive `opencodex` row, and a matching rollout using the fixture at - `tests/codex-history-provider.test.ts:27-89`. -2. Spawn an owned Bun child with `Bun.spawn([process.execPath, "-e", source])`. - The child opens that exact `state_5.sqlite`, executes - `PRAGMA busy_timeout=0; BEGIN IMMEDIATE; UPDATE threads SET has_user_event = - has_user_event`, writes `holder-ready`, and loops with `Bun.sleepSync(10)` until - `holder-release` exists. The ready/release handshake and `finally` cleanup match - `tests/config-mutation-lock.test.ts:48-92`. This is a separate-process SQLite - writer lock, not a stubbed `SQLITE_BUSY`. -3. Set only `setCodexHistoryJobTestHooks({ automaticBusyTimeoutMs: 1_200 })` so - contention lasts long enough to sample. Do not set an in-process execution hook. -4. Start `startServer(0)` with `managementApi.syncModelsToCodex` injected so the - real `/api/sync` request calls real `injectCodexConfig` and the real Worker while - catalog fetch is a deterministic local stub. Start the management POST but do - not await it. -5. In parallel, issue a real `/v1/responses` request to a local test upstream that - emits eight SSE chunks 50 ms apart; assert all eight arrive. This proves an - already-admitted data-plane client still progresses. -6. While the management request is still pending and the child still owns the - transaction, issue six `/healthz` requests 40 ms apart. Require every status to - be 200. Discard the first warmup latency and require every remaining sample to - be below `Math.floor(1_200 / 3) = 400 ms`. Also assert the management operation - duration is at least 1,100 ms, proving the health samples overlapped contention. - This copies the measured pattern at - `tests/storage-restore-job-responsive.test.ts:175-210`; checking health only - before and after the operation is not acceptance evidence. -7. Assert the management result and `GET /api/codex/history` both report unresolved - `sqlite_busy`, with null unknown counts where the probe failed and a next retry. -8. In `finally`, write `holder-release`, await child exit code 0, reset/join the - history Worker, drain the server, restore both homes, and delete fixtures. No - test may delete a Worker-owned home before the Worker join; the repository has - already observed Bun isolate failures from that ordering - (`tests/storage-restore-job-responsive.test.ts:53-80`, - `src/storage/worker-lifecycle.ts:1-17`). -9. Start a fresh server after lock release, trigger the persisted retry, and assert - both the API and live post-probe become `converged` with zero/zero counts. This - closes persistence and retry, not only responsiveness. - -### Focused cases - -- `tests/codex-history-provider.test.ts` — automatic options set one attempt and a - 100 ms writable timeout; no sleep callback fires; explicit defaults still make - two attempts with one 500 ms sleep; every recoverable class gets the right reason; - hard errors still throw (`tests/codex-history-provider.test.ts:293-369`). -- `tests/codex-history-worker.test.ts` — parity for forward retag, manifest restore, - no-manifest ejection, line-one patch, trailing append, manifest consumption, and - no-op. Assert only plain-data messages cross. Existing behavioral oracle: - `tests/codex-history-provider.test.ts:92-290`. -- `tests/codex-history-worker.test.ts` — malformed message ignored; dynamic import - sees captured homes; valid `error`, `onerror`, early `close`, watchdog timeout, - and shutdown cancellation each join and classify once. -- `tests/codex-history-convergence.test.ts` — pending is durable before dispatch; - state-write busy means no Worker spawn; final-write failure leaves pending/running; - stale `attemptId` cannot overwrite a newer attempt; corrupt/unknown-version - `codex.json` is preserved and blocks; failed probe counts are null; only clean - zero/zero post-probe permits `converged`. -- `tests/history-migration-guardian.test.ts` — no synchronous provider probe on a - tick, single-flight retries, finite 60-tick budget, startup re-arm of stale - running/pending/blocked/unknown, and no retry for converged. Preserve the scheduler - expectations currently covered at `tests/history-migration-guardian.test.ts:24-136`. -- `tests/codex-history-process-routing.test.ts` — startup, `/api/sync`, `/api/stop`, - guardian, and graceful server cleanup select automatic Worker execution; explicit - CLI restore/eject/recover/sync/ensure and service cleanup select inline full-budget - execution. Assert by injected executors, not source-string matching. -- `tests/codex-sync-api.test.ts` — execution option reaches both external-provider - and normal injection paths at `src/codex/sync.ts:49-70` and - `src/codex/sync.ts:83-124`; result carries history status. -- `tests/shutdown-drain.test.ts` — drain awaits history termination, records - `shutdown_cancelled`, and stops the listener even if join rejects. Existing - storage joins remain unchanged (`src/server/lifecycle.ts:407-445`). -- `tests/codex-history-convergence.test.ts` — doctor and management status say - routed threads remain hidden for pending/blocked/unknown, survive module reload, - and never collapse failed-probe zeroes into success. +### Opposite-direction cross-process serialization + +1. Seed production-shaped DB, manifest, and rollouts in isolated homes. +2. Process A converges ON and pauses after acquiring the real history lock. +3. Process B converges OFF. Assert B cannot mutate manifest, rollout, or DB while A + holds the lock. +4. Let B win the newer native `CommitExpectation`; release A. Assert A is rejected + as `overtaken` before its first history write and B alone produces final OFF + history. +5. Reverse direction/order and repeat. Final history must match the highest native + generation, not Worker scheduling order. + +This is real two-process SQLite/filesystem behavior. A same-process flight or two +connections without rollout sentinels does not satisfy C15. + +### CLI contention + +- Hold the production history lock in a child. Invoke an explicit CLI convergence + through its function-level command handler with `mode:"explicit"`; assert it + waits/returns the typed contract outcome and performs no inline provider call. +- In parallel trigger automatic server convergence; assert listener health/data + plane progress while both processes contend. +- Release, join both Workers, and prove one serialized winner. The test inspects the + integration record through its owner, not a WP10 parser. + +### Post-probe under lock + +- Inject a competing child that attempts to change a history row and manifest at + the probe seam. Assert it cannot proceed until after terminal state is recorded + and lock released. +- A failed probe, nonzero pending rows, or nonempty backup entries remains + non-converged. Only clean zero/zero becomes `converged`. + +### Retry and death + +- Advance a fake monotonic clock through exponential growth and the cap; prove a + later timer always exists for unresolved current work and no 60-tick terminal + state exists. +- Restart/module reload re-arms unresolved state. +- Worker error, malformed response, early close, watchdog, cancellation, and final + record-write failure remain non-converged and join exactly once. +- An overtaken transition does not retry itself. + +### Measured responsiveness — C3 + +Keep the real `BEGIN IMMEDIATE` holder and overlapping `/healthz` plus eight-chunk +SSE test from the prior plan, but route the request through production +`convergeCodex`. Bind port `0`; use temporary homes. Require the management/history +request to overlap contention, every health response to be 200, the stream to +complete, and the durable state to report `db-busy` before succeeding after release. ## Verification -Static and focused gates: - ```bash bun run typecheck -bun test tests/codex-history-provider.test.ts -bun test tests/codex-history-worker.test.ts -bun test tests/codex-history-convergence.test.ts -bun test tests/history-migration-guardian.test.ts -bun test tests/codex-sync-api.test.ts -bun test tests/codex-history-process-routing.test.ts -bun test tests/shutdown-drain.test.ts -bun test tests/codex-history-worker-responsive.test.ts +bun test tests/codex-history-provider.test.ts tests/codex-history-worker.test.ts +bun test tests/history-migration-guardian.test.ts tests/codex-history-process-routing.test.ts +bun test tests/codex-sync-api.test.ts tests/shutdown-drain.test.ts +bun test tests/codex-history-worker-responsive.test.ts --timeout 30000 bun run privacy:scan bun run test ``` -Runtime measurement is the output of the responsiveness test, which must print or -attach this evidence on failure and success: - -```text -lock_ready_at= -history_request_started_at= -health_ms=[...five post-warmup samples...] -max_health_ms= threshold_ms=400 -stream_chunks=8 -history_elapsed_ms== 1100> -history_state=pending reason=sqlite_busy -child_exit=0 history_workers_live=0 -``` - -The acceptance command is the real test invocation, not a prose assertion: - -```bash -bun test tests/codex-history-worker-responsive.test.ts --timeout 30000 -``` - -Do not run `ocx start`, `ocx stop`, `ocx sync`, `ocx restore`, or `ocx ensure` as -verification against the live proxy on port 10100. The isolated test server binds -port 0 and the lock child touches only its temporary Codex home. +The responsiveness test prints lock-ready time, overlapping health latencies, +stream chunk count, history elapsed time/state, child exit, and live Worker count. +No verification command invokes `ocx start`, `ocx stop`, `ocx sync`, `ocx restore`, +or `ocx ensure`; port 10100 remains untouched. ## Accept criteria -- **C3 — measured availability:** during a real cross-process - `BEGIN IMMEDIATE` lock on the exact history database, the automatic history - operation remains pending for at least 1,100 ms, all six `/healthz` requests are - 200, every post-warmup sample is below 400 ms, and an eight-chunk data-plane - stream completes. The operation executes in a Worker; no synchronous fallback - is enabled in this test. -- **C3 — unbounded work boundary:** SQLite queries, all row/manifest traversal, - rollout reads/writes, and fsync stay inside the Worker. No claim that the - operation is “fast” substitutes for this boundary. -- **C4 — no silent success:** before any automatic mutation, a durable - `pending`/`running` record exists. Busy, permissions, unreadable state, timeout, - Worker death, and shutdown cancellation remain non-converged with classified - reasons and nullable unknown counts. -- **C4 — proof before green:** `converged` for native restore requires a clean - post-probe with `pendingRows=0` and `backupEntries=0`; manifest absence, - `failed: true`, or a successful zero-row mutation is insufficient. -- **C4 — retry and visibility:** the running server retries on its bounded cadence, - every startup re-arms durable unresolved work, explicit CLI recovery updates the - same record, and management responses plus `ocx doctor` state that routed threads - may remain hidden until convergence. -- CLI-process commands retain today's 5,000 ms / two-attempt / 500 ms budget; - server-process callers use Worker + 100 ms / one attempt / no sleep. -- WP10 creates no native-write lock and no GUI switch. It is independently useful - and independently testable before WP11. +- **C3** — all synchronous/unbounded history work is in the Worker; real contention + overlaps responsive `/healthz` and data-plane traffic. +- **C4** — unresolved work is durably represented by the contract + `CodexHistoryState`, retried with capped non-permanent backoff, and never collapsed + into success. Clean post-probe occurs under the history lock. +- **C15** — opposite-direction processes serialize manifest, rollouts, DB, probe, + and record update; `CommitExpectation` prevents overtaking. +- Explicit CLI and automatic server/startup/retry callers all enter through + `convergeCodex` and the same sibling history lock. No inline escape hatch remains. +- **N2** — WP10 imports the WP8b record/types and extends WP9's working funnel. Its + commit typechecks and preserves behavior without any WP11/WP12 placeholder. diff --git a/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md b/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md index 3226b3966..875fb92fa 100644 --- a/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md +++ b/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md @@ -1,800 +1,459 @@ # WP11 — one bounded writer per canonical `CODEX_HOME` -Research: `003_lock_protocol.md`. Read it first; this document is the implementation -diff for its Decision. - -The failure is lock splitting plus event-loop denial, not a missing mutex. Today the -default home can remain an unresolved `~/.codex` spelling while an explicit home is -realpathed (`src/codex/paths.ts:6-24`), and the nearest reusable SQLite lock opens a -stable file but repairs its mode after open (`src/codex/native-main-lock-file.ts:74-131`). -The rejected Codex-OFF design could therefore let two spellings of one home take two -locks, or wait synchronously around history work that can exceed 10.5 seconds -(`devlog/_plan/260803_codex_desktop_toggle/008_audit_synthesis_wp4_r2.md:17-30`). -WP9 and WP10 land first: WP9 supplies an already-gathered, fixed-size synchronous -catalog commit (`001_catalog_seam.md:137-159`), and WP10 moves unbounded history work -to an owned Worker (`002_history_off_the_loop.md:264-296,474-486`). This phase adds -only the async cross-process acquisition substrate around those bounded commit -sections. It does not add desired state or decide ownership. +Research: `003_lock_protocol.md`. Shared contract: `005_contract.md`. + +The failure is lock splitting plus event-loop denial, not a missing mutex. Two +spellings of one existing Codex home can reach different textual paths +(`src/codex/paths.ts:6-24`), while a lock held across provider discovery or history +walking would recreate the 10.5-second listener stall that blocked the previous OFF +design. WP9 has already made catalog commit synchronous and fixed-size; WP10 has +already put history behind its own sibling Worker-held lock. WP11 supplies only the +native acquisition and coordinated synchronous commit section. + +The prior plan still invented admission callback/result types and based the +namespace on `homedir()`. Round 2 invalidated both choices. Admission is the +contract's exact `AdmissionSnapshot`; and the pinned Bun 1.3.14 probe showed both +`os.homedir()` and `os.userInfo().homedir` follow environment-controlled home. +Effective-user identity — uid on POSIX, SID on Windows — is the namespace authority +(`005_contract.md` §§4, 7). + +WP11 is independently landable. It consumes WP8b's identity/generation/types, +WP9's synchronous candidate commit, and WP10's separate history protocol. The WP11 +commit typechecks and preserves the working WP9/WP10 funnel. WP12 later supplies +stronger ownership/provenance decisions through the same `AdmissionSnapshot`; it is +not required to replace a placeholder before this phase works. All current-code citations and diff context below were rechecked on 2026-08-04 at -`7e67a8d06311de2471b0a25e41cf85f97007cc69`. +`2d5e080dea3e7000bf2111b381c7c1a3c4f5fb11`. ## IN / OUT IN: -- `src/codex/codex-write-lock.ts` (**NEW**) — canonical-home resolution, namespace - validation, finite async retry, admission callbacks, and the synchronous locked - callback. -- `src/codex/native-main-lock-file.ts` (**MODIFY**) — preserve its existing stable - descriptor owner and add only a caller-supplied ACL time cap. WP11 reuses - `openStableLockFile`/`assertStableLockFile`; it does not copy their descriptor - lifetime or `(dev, ino)` substitution checks. -- `src/lib/windows-secret-acl.ts` (**MODIFY**) — allow a stricter caller deadline to - cap the existing required `icacls` sequence. Existing callers retain the current - 5-second default. -- `tests/codex-write-lock.test.ts` (**NEW**) — result taxonomy, canonicalization, - namespace refusal, sync-callback, reentrancy, deadline, and real-process coverage. -- `tests/helpers/codex-write-lock-child.ts` (**NEW**) — an owned Bun process that - acquires the real SQLite transaction and holds one finite synchronous section. -- `tests/windows-secret-acl.test.ts` (**MODIFY**) — prove the optional caller cap is - forwarded without weakening the required ACL failure behavior. +- `src/codex/codex-write-lock.ts` (NEW) — exact contract module name; canonical + target identity, effective-user namespace, finite async acquisition, synchronous + coordinated commit, release, and typed lock mechanics. +- `src/codex/convergence.ts` (MODIFY) — place WP9's fixed catalog/native commit + under the new lock and pass the contract `AdmissionSnapshot`/`CommitExpectation`. +- `src/codex/generation.ts` (MODIFY through its public owner API) — allocate and + verify native expected transitions; no parallel counter. +- `src/codex/integration-record.ts` (MODIFY through `updateIntegrationRecord`) — + persist native generation/tx identity inside the synchronous coordinated section. +- `src/codex/native-main-lock-file.ts` (MODIFY) — reuse stable descriptor and + substitution checks; add only a caller-supplied ACL deadline cap. +- `src/lib/windows-secret-acl.ts` (MODIFY) — accept a stricter remaining deadline; + current callers retain the 5-second default. +- `tests/codex-write-lock.test.ts` (NEW), + `tests/helpers/codex-write-lock-child.ts` (NEW), and + `tests/windows-secret-acl.test.ts` (MODIFY). OUT: -- `src/config.ts` — `withConfigMutationLockSync` stays synchronous and fail-fast; - changing it would recreate the listener freeze its docstring prevents - (`src/config.ts:1767-1818`). -- WP9 catalog gather/commit implementation, WP10 history Worker implementation, - and all of WP12 ownership, desired-state, provenance, convergence, API, CLI, GUI, - and docs wiring. -- `src/codex/paths.ts` global behavior. WP11 canonicalizes for the lock without - changing every existing `CODEX_HOME` consumer at module import. -- PID files, heartbeat rows, leases, stale-file deletion, lock-database unlink, - FIFO tickets, and process-local queueing. -- `gui/**`, service lifecycle, proxy start/stop/sync/restore/ensure, release, deploy, - and the live proxy on port 10100. - -No-code/configuration reuse is insufficient: process-local flights do not coordinate -two processes, and the existing native-main databases live inside `CODEX_HOME` and -have different lifetime semantics. Reusing the stable-file owner is sufficient for -the dangerous descriptor/open race, so this phase extends that owner instead of -adding another raw `openSync` implementation. - -## API and ownership boundary - -### Public contract - -Add the following real TypeScript contract at the top of -`src/codex/codex-write-lock.ts`: +- New admission, authority, generation, record, observed-state, or convergence + result shapes. WP11 imports `AdmissionSnapshot`, `CommitExpectation`, and + `UserIdentity` from the contract modules (`005_contract.md` §§1-4, 7). The + native-lock result below remains owned by this lock module; it is a mechanism + result projected by `convergence.ts`, not a competing convergence union. +- History mutation/locking. WP10's history lock is a sibling and is never nested. +- Provider gathering or any awaited history work inside the native held section. +- Desired-state, service ownership, external-provider, journal, and provenance + policy — WP12. WP11 compares snapshots and enforces order; it does not decide + what `owned` means. +- `src/codex/paths.ts` global behavior, PID files, leases, stale-file deletion, + FIFO tickets, process-local queueing, GUI, releases/deploys, and port 10100. + +No-code/config reuse is insufficient: process-local flights do not coordinate two +processes. Reusing `native-main-lock-file.ts` is required for its stable descriptor +and `(dev, ino)` checks (`src/codex/native-main-lock-file.ts:35-55,74-131`); WP11 +does not add another raw-open owner. + +## Public contract consumes `AdmissionSnapshot` + +The shared type names and result taxonomy already exist after WP8b. WP11 implements +them in `src/codex/codex-write-lock.ts`; it does not publish the former +`CodexWriteLockAdmissionPhase` or `CodexWriteLockAdmissionResult` unions. ```ts -export const CODEX_WRITE_LOCK_MAX_TIMEOUT_MS = 30_000; +import type { + AdmissionSnapshot, + CommitExpectation, +} from "./convergence-types"; -export type CodexWriteLockRefusalReason = - | "codex_home_missing" - | "codex_home_unsafe" - | "authority_not_proven" - | "namespace_unsafe" - | "lock_path_unsafe" - | "unsupported_filesystem" - | "reentrant" - | "lock_unavailable"; +export const CODEX_WRITE_LOCK_MAX_TIMEOUT_MS = 30_000; export type CodexWriteLockResult = - | { - status: "acquired"; - value: T; - waitedMs: number; - lockId: string; - } - | { - status: "busy"; - reason: "deadline" | "cancelled"; - retryable: true; - waitedMs: number; - } + | { status: "acquired"; value: T; waitedMs: number; lockId: string } + | { status: "busy"; reason: "deadline" | "cancelled"; retryable: true; waitedMs: number } | { status: "refused"; - reason: CodexWriteLockRefusalReason; + reason: + | "codex_home_missing" + | "codex_home_unsafe" + | "authority_not_proven" + | "namespace_unsafe" + | "lock_path_unsafe" + | "unsupported_filesystem" + | "reentrant" + | "lock_unavailable"; retryable: false; message: string; }; -export type CodexWriteLockAdmissionPhase = "before_namespace" | "under_lock"; - -export type CodexWriteLockAdmissionResult = - | { status: "admitted" } - | { status: "refused"; message: string }; - -export interface CodexWriteLockContext { - readonly canonicalCodexHome: string; - readonly lockId: string; -} - export interface CodexWriteLockOptions { - /** - * Optional explicit target. When absent, a nonblank process CODEX_HOME wins; - * otherwise defaultCodexHome() supplies ~/.codex or the existing WSL default. - * Explicit and default targets pass through the same existing-directory - * realpath algorithm before identity or namespace work. - */ codexHome?: string; - - /** - * Required total acquisition budget in milliseconds, including namespace ACL - * validation and every BEGIN IMMEDIATE attempt. Must be finite, integral, and - * within 0..CODEX_WRITE_LOCK_MAX_TIMEOUT_MS. Zero performs one fail-fast attempt. - */ timeoutMs: number; - - /** Cancellation is a typed busy outcome; it is never thrown as contention. */ signal?: AbortSignal; + /** Read-only snapshot obtained before any namespace creation. */ + admitted: AdmissionSnapshot; + /** - * Read-only WP12 admission. It runs once after canonical-home resolution but - * BEFORE this module creates or opens a namespace entry, then again while the - * SQLite transaction is held. The lock proves exclusion only; an admission - * callback must independently prove service-home, external-provider, journal, - * provenance, and desired-state authority. + * Authoritative synchronous re-read while native + config coordination is held. + * It returns the exact shared shape; no lock-specific admission union exists. */ - admit( - phase: CodexWriteLockAdmissionPhase, - context: CodexWriteLockContext, - ): CodexWriteLockAdmissionResult; + readAdmissionUnderLock(): AdmissionSnapshot; +} + +export interface CodexWriteCommitContext { + readonly canonicalCodexHome: string; + readonly lockId: string; + readonly admission: AdmissionSnapshot; + readonly expectation: CommitExpectation; } type Synchronous = T extends PromiseLike ? never : T; -/** - * Acquire the per-canonical-CODEX_HOME cross-process write lock, execute one - * synchronous bounded commit, and release the SQLite transaction before this - * Promise resolves. - * - * Waiting is asynchronous and barging-allowed: SQLITE_BUSY closes candidate - * handles, sleeps with bounded jitter, and retries only until timeoutMs. The - * locked callback itself MUST NOT return a Promise or perform provider I/O, - * subprocess work, history walking, retry sleeps, or any other awaitable work. - * WP9 prepares catalog bytes before this call; WP10 owns history in a Worker. - * Keeping only their fixed commit/admission work here prevents a lock holder from - * becoming the event-loop outage that blocked the earlier OFF design. - * - * Contention, cancellation, filesystem validation, ACL failure, and SQLite-open - * failure return busy/refused. Invalid timing arguments and exceptions thrown by - * the caller's admission or locked callback remain programmer/domain exceptions. - */ export async function withCodexWriteLock( options: CodexWriteLockOptions, - locked: (context: CodexWriteLockContext) => Synchronous, + commit: (context: CodexWriteCommitContext) => Synchronous, ): Promise>; ``` -There is deliberately no public handle and no `release()` method. A handle would -let a caller retain the transaction across an `await`, making “bounded” a comment -rather than an API boundary. The conditional return type rejects an ordinary -`async` callback at typecheck; the implementation also checks for a thenable after -invocation, rolls back immediately, and throws `TypeError` as a programmer error. -It never awaits a callback result. - -This narrows the research-level allowance at `003_lock_protocol.md:198-201` without -changing its Decision: acquisition remains async; the held operation is now -synchronous because WP9/WP10 remove the two reasons it previously needed to await. -The roadmap already records this stronger construction at `000_plan.md:105-109`. +`CodexWriteLockResult` is the lock module's own bounded mechanism result. +`convergence.ts` exhaustively projects it into `ConvergeOutcome`; no route consumes +it directly. There is no public handle or release method. The conditional return rejects ordinary `async` callbacks at typecheck; +the implementation also detects a cast thenable, rolls back, and throws a +`TypeError`. Provider I/O, subprocesses, serialization, history walking, retry +sleeps, and any other awaitable work are forbidden beneath `commit`. -### Admission order +## Admission and synchronous config-record section -The implementation order is fixed: +The fixed order is: ```text -resolve existing canonical CODEX_HOME read-only -derive lockId and detect same-task reentrancy read-only -options.admit("before_namespace", context) read-only - refused -> return authority_not_proven; create NOTHING -resolve/validate real login home read-only -validate/create each private namespace component -validate/open stable database and BEGIN IMMEDIATE -options.admit("under_lock", context) read-only - refused -> rollback/close; do not run locked callback -locked(context) synchronous, bounded -assert stable path, ROLLBACK, close SQLite, close side fd +resolve canonical existing CODEX_HOME read-only +derive lock id + detect same-task reentrancy read-only +compare options.admitted to target home read-only + non-authorizing snapshot -> return refused; create NOTHING +resolve effective UserIdentity + OS runtime root read-only +validate/create private user namespace +validate/open stable DB; BEGIN IMMEDIATE native lock held +withConfigMutationLockSync config lock held + authoritative readAdmissionUnderLock() fresh snapshot + compare digest + config generation + intent + ownership + allocate CommitExpectation (N -> N+1, this txId) + commit(context) synchronous + updateIntegrationRecord(nativeAfter + txId + section edits) + verify exact expected transition +release config lock +assert stable lock path; ROLLBACK; close DB + side fd ``` -The callback names “admission”, not “ownership receipt”, because WP11 must not -manufacture a token that WP12 could accidentally treat as authority. The first call -closes the creation-before-knowledge bug; the second closes the check/lock race. -An `acquired` result means both admissions passed and the callback ran under the OS -transaction. It does not mean OpenCodex owns every artifact the callback might name. - -## Canonicalization — C6 - -### Exact algorithm - -`canonicalCodexHome(options)` implements these steps, in this order: - -1. Select raw input as `options.codexHome` when it is nonblank; otherwise use a - nonblank `process.env.CODEX_HOME`; otherwise call `defaultCodexHome()`. This - retains today's default/WSL precedence (`src/codex/home.ts:121-146`). A supplied - blank `options.codexHome` is a programmer error, not a request for default. -2. Expand only a leading `~` through the existing `expandUserPath`, then `resolve` - to an absolute path. Do not lowercase, Unicode-normalize, or append unresolved - suffixes. -3. `statSync` the target. `ENOENT`/`ENOTDIR` returns - `refused/codex_home_missing`; another read error or a non-directory returns - `refused/codex_home_unsafe`. No namespace function has run yet. -4. Call `realpathSync.native` for **both** default and explicit input. This collapses - `~`, dot segments, trailing separators, and every symlink in the existing path. -5. On Windows only, feed `win32.normalize(realPath).toLowerCase()` to the hash. The - existing diagnostics already compare Windows paths case-insensitively - (`src/codex/home.ts:164-183`). On macOS and Linux, hash the exact string returned - by `realpathSync.native`. -6. Refuse the already-recognized unsupported target classes: Windows UNC homes and - WSL `/mnt/` homes, using `nativeMainOwnerFilesystemSupported` - (`src/codex/native-main-owner.ts:75-91`). This phase does not claim portable - lock identity across network hosts or filesystem namespaces. -7. Hash exactly - `"opencodex-codex-write-lock-v1\0" + normalizedCanonicalHome` as UTF-8 with - SHA-256, lowercase hex, all 64 characters. - -For an existing case-insensitive macOS directory, `realpathSync.native` returns the -filesystem's stored directory-entry spelling, so `/Users/A/.CODEX` and -`/Users/A/.codex` converge. On a case-sensitive APFS volume, those can be two real -directories and must remain two identities. Windows can case-fold safely because a -single Windows namespace does not distinguish those spellings. Linux remains -case-sensitive. - -Two consequences are acceptance requirements, not examples: - -- default `~/.codex`, explicit `~/.codex`, its absolute spelling, and any symlink - to that same existing directory contend on one SQLite file; -- two different existing directories produce different 64-character IDs and can - acquire concurrently. - -### The missing-home question - -Missing homes are refused before hashing and before resolving the login-home lock -namespace. There is no portable alternative. If WP11 canonicalized the deepest -existing parent and preserved the absent suffix, `Foo` and `foo` would split one -future home on case-insensitive APFS. If it lowercased the suffix, they would alias -two future homes on case-sensitive APFS. Until the directory exists there is no -inode, filesystem-returned spelling, or case-behavior answer. Creation/installation -of `CODEX_HOME` is therefore another operation and another lock domain. - -The implementation hunk in the new module is: +This replaces the former two generic admission callbacks. The first +`AdmissionSnapshot` is enough to refuse before namespace creation. The second is +an authoritative re-read inside the coordinated commit; WP11 does not reduce it to +a boolean or manufacture an authority receipt. + +`withConfigMutationLockSync` is already synchronous, fail-fast, and reentrant only +for the current synchronous stack (`src/config.ts:1767-1818`). The native lock may +hold it because no await occurs. Config-generation reads/updates and +`updateIntegrationRecord` happen before that callback returns. The native +generation bump and `txId` are persisted in the same record update as the native +commit result, so another cooperating writer cannot observe moved native bytes with +an old generation. + +If the config coordinator is busy, the attempt releases the native lock and retries +only while the outer monotonic deadline remains; deadline expiry returns typed +`busy`. It never releases and commits against the old admission. A non-cooperating +filesystem writer remains detectable after commit, as scoped by `005_contract.md` +§3; WP11 does not promise a portable conditional rename that `src/config.ts:1853-1859` +explicitly says the filesystem lacks. + +## Canonical `CODEX_HOME` identity — C6 + +1. Select nonblank explicit `codexHome`, else nonblank `process.env.CODEX_HOME`, + else `defaultCodexHome()` (`src/codex/home.ts:121-146`). Blank explicit input is + a programmer error. +2. Expand only leading `~`, resolve absolute, and require an existing directory. + Missing/non-directory refuses before identity namespace work. +3. `realpathSync.native` every spelling, default and explicit. +4. Refuse known unsupported UNC/WSL DrvFS target classes through the existing + predicate (`src/codex/native-main-owner.ts:75-91`). +5. Windows normalizes/case-folds the canonical result; POSIX hashes the exact + realpath string. +6. Hash `"opencodex-codex-write-lock-v1\0" + normalizedCanonicalHome` with full + SHA-256 lowercase hex. + +Default, explicit, absolute, tilde, and symlink spellings of one existing directory +must contend on one lock. Two distinct existing directories must not. A missing +home is refused: preserving an unresolved suffix would either split one future home +on case-insensitive filesystems or alias two on case-sensitive ones. -```diff -diff --git a/src/codex/codex-write-lock.ts b/src/codex/codex-write-lock.ts -new file mode 100644 ---- /dev/null -+++ b/src/codex/codex-write-lock.ts -@@ -+import { createHash } from "node:crypto"; -+import { lstatSync, mkdirSync, realpathSync, statSync } from "node:fs"; -+import { homedir } from "node:os"; -+import { join, resolve, win32 } from "node:path"; -+import { AsyncLocalStorage } from "node:async_hooks"; -+import { Database } from "bun:sqlite"; -+ -+import { expandUserPath } from "../config"; -+import { hardenSecretDirAsync } from "../lib/windows-secret-acl"; -+import { defaultCodexHome } from "./home"; -+import { -+ assertStableLockFile, -+ hardenStableLockFile, -+ openStableLockFile, -+ StableLockPathUnsafeError, -+ type StableLockFile, -+} from "./native-main-lock-file"; -+import { nativeMainOwnerFilesystemSupported } from "./native-main-owner"; -+ -+const LOCK_DOMAIN = "opencodex-codex-write-lock-v1\0"; -+const LOCK_NAMESPACE_PARTS = [".opencodex", "native-write-locks", "v1"] as const; -+const heldHomes = new AsyncLocalStorage>(); -+ -+function normalizeCanonicalHome(path: string, platform = process.platform): string { -+ return platform === "win32" ? win32.normalize(path).toLowerCase() : path; -+} -+ -+function lockIdFor(canonicalCodexHome: string): string { -+ return createHash("sha256") -+ .update(LOCK_DOMAIN) -+ .update(canonicalCodexHome) -+ .digest("hex"); -+} -+ -+function rawCodexHome(explicit: string | undefined): string { -+ if (explicit !== undefined) { -+ if (!explicit.trim()) throw new TypeError("codexHome must not be blank"); -+ return explicit; -+ } -+ return process.env.CODEX_HOME?.trim() || defaultCodexHome(); -+} -+ -+function canonicalCodexHome(explicit: string | undefined): -+ | { status: "ok"; path: string } -+ | Extract, { status: "refused" }> { -+ const absolute = resolve(expandUserPath(rawCodexHome(explicit))); -+ try { -+ if (!statSync(absolute).isDirectory()) { -+ return refused("codex_home_unsafe", "CODEX_HOME is not an existing directory."); -+ } -+ const real = realpathSync.native(absolute); -+ if (!nativeMainOwnerFilesystemSupported(real)) { -+ return refused("unsupported_filesystem", "CODEX_HOME uses an unsupported filesystem identity."); -+ } -+ return { status: "ok", path: normalizeCanonicalHome(real) }; -+ } catch (error) { -+ const code = errorCode(error); -+ return code === "ENOENT" || code === "ENOTDIR" -+ ? refused("codex_home_missing", "CODEX_HOME must exist before native writes can be locked.") -+ : refused("codex_home_unsafe", "CODEX_HOME could not be resolved safely."); -+ } -+} -``` +## Namespace and hardening — C7 -`refused` and `errorCode` are private constructors in the same file; messages never -include the raw path or username. +### No home accessor participates -## Namespace and hardening — C7 +Delete `homedir()` from the import list and delete the prior +`realpathSync.native(homedir())/.opencodex/...` design. The pinned Bun probe in +`005_contract.md` §7 proves both home accessors can be changed by `HOME`; using +`os.userInfo().homedir` would preserve the defect. -### Exact path +Consume `UserIdentity` and the resolver from `src/codex/user-identity.ts`: + +```ts +import { + resolveEffectiveUserIdentity, + resolveOsRuntimeDirectory, +} from "./user-identity"; +``` -The namespace is independent of both `CODEX_HOME` and `OPENCODEX_HOME`: +The exact path is: ```text -realpathSync.native(homedir()) - /.opencodex - /native-write-locks - /v1 - /.sqlite +/opencodex/native-write-locks/v1//.sqlite ``` -The login home itself may resolve through a symlink because it is immediately -realpathed. The three OpenCodex-owned descendants may not be symlinks, junctions, -or other path substitutions. A custom `OPENCODEX_HOME` never changes this path. +`` is encoded from `{ platform:"posix", uid }` or +`{ platform:"win32", sid }`; it is never username, `HOME`, `USERPROFILE`, +`CODEX_HOME`, or `OPENCODEX_HOME`. This matches `005_contract.md` §7. WP8b's +identity resolver is the sole platform owner; WP11 does not add a second SID lookup. ### Component validation -`ensurePrivateLockNamespace(deadline)` walks one component at a time; it never uses -recursive `mkdir`: - -1. Resolve and `stat` the login home, then `realpathSync.native` it. -2. For each descendant, `lstat` first. `ENOENT` permits one `mkdirSync(path, - { mode: 0o700 })`; `EEXIST` restarts validation. Any existing non-directory, - symlink, junction/reparse redirect, or realpath mismatch returns - `refused/namespace_unsafe`. -3. On POSIX, require `process.getuid()` and exact `(mode & 0o7777) === 0o700` plus - `stats.uid === process.getuid()`. Existing broader/narrower modes and another - uid are refused; WP11 never chmods, renames, unlinks, or recreates them. -4. On Windows, compare case-folded `resolve(path)` and `realpathSync.native(path)` - to reject junction/reparse redirection, then run the existing async directory - ACL owner with `required: true` and the remaining outer deadline. A failed or - timed-out required ACL operation returns `refused/namespace_unsafe`; it never - proceeds to SQLite. The helper grants only the current user before removing - inheritance and broad SIDs (`src/lib/windows-secret-acl.ts:217-328,404-494`). -5. Re-`lstat` and re-run identity/mode checks after creation/hardening before - descending to the next component. - -For the database and SQLite sidecars: - -- Before open, any existing `.sqlite` or `.sqlite-journal` must be a regular - non-symlink entry; on POSIX it must have the same uid and exact `0600` mode. -- Existing `-wal` or `-shm` is refused as `lock_path_unsafe`. WP11 forces rollback - journal mode, so those names are unexpected state, not files to clean up. -- `openStableLockFile` performs `O_NOFOLLOW` on POSIX, then `fstat`; its retained - side descriptor and reference count prevent a sibling close from releasing this - process's SQLite lock (`src/codex/native-main-lock-file.ts:35-55,74-125`). -- After open, validate the descriptor's regular-file/uid/mode metadata, compare - path `(dev, ino)` through `assertStableLockFile`, run required Windows file ACL - hardening within the remaining deadline, and assert identity again before SQLite. -- SQLite executes `PRAGMA busy_timeout = 0`, `PRAGMA locking_mode = NORMAL`, verifies - `PRAGMA journal_mode = DELETE`, then tries `BEGIN IMMEDIATE`. SQLite's OS lock is - the only holder authority. -- Assert stable identity immediately after `BEGIN IMMEDIATE`, immediately before - the synchronous callback, and once more before rollback/close. Close SQLite - before closing the retained side descriptor. - -Any validation failure before the locked callback maps to `refused/namespace_unsafe` -or `refused/lock_path_unsafe`; ACL/SQLite setup failures map to the narrower safe -reason when known, otherwise `refused/lock_unavailable`. The implementation does not -repair or delete the suspect entry. Diagnostics identify only the component role -(`v1 namespace`, `lock database`, `journal sidecar`), never its full home path. - -The database persists after release. There is no `unlinkSync` in the module. A -crashed process loses its transaction when the OS closes SQLite; an old database -mtime or dead PID grants no takeover rights. A live hung process remains the holder, -and contenders return `busy/deadline`. - -### Existing helper changes - -The stable-file owner currently gives required Windows hardening its own fixed -deadline (`src/codex/native-main-lock-file.ts:127-131`). Add an optional caller cap -without changing existing call sites: +Walk components one at a time; never recursive-mkdir across an unvalidated parent. + +- Existing components are `lstat`ed and must be real directories, not symlinks, + junctions, or reparse redirects. `ENOENT` permits one `mkdirSync(..., 0700)`, + followed by the same validation. +- POSIX requires exact effective uid and mode `0700` for directories, `0600` for + the DB/rollback journal. Wrong owner/mode refuses; WP11 does not chmod a suspect + existing path. +- Windows validates non-junction identity and runs the existing required per-user + ACL owner within the remaining outer deadline + (`src/lib/windows-secret-acl.ts:217-328,404-494`). Failure/timeout refuses. +- Existing DB or `-journal` must be regular, same-user private entries. Existing + `-wal`/`-shm` refuses; WP11 forces rollback journal mode. +- `openStableLockFile` retains the side descriptor; validate descriptor metadata, + assert path identity before/after SQLite open, after `BEGIN IMMEDIATE`, before + commit, and before close. +- SQLite uses `busy_timeout=0`, `locking_mode=NORMAL`, and verified + `journal_mode=DELETE`. The OS transaction is holder authority. + +The DB persists after release. There is no unlink, stale takeover, heartbeat, PID, +or mtime authority. Process death releases the OS lock; a live hung holder remains +the holder and contenders reach their deadline. + +### Core new-module diff ```diff -diff --git a/src/codex/native-main-lock-file.ts b/src/codex/native-main-lock-file.ts ---- a/src/codex/native-main-lock-file.ts -+++ b/src/codex/native-main-lock-file.ts -@@ -127,6 +127,10 @@ --export async function hardenStableLockFile(path: string): Promise { -+export async function hardenStableLockFile(path: string, timeoutMs?: number): Promise { - try { chmodSync(path, 0o600); } catch { /* Windows ACL below is authoritative there. */ } - if (process.platform === "win32") { -- await hardenSecretPathAsync(path, { required: true, timeoutMemoKey: path }); -+ await hardenSecretPathAsync(path, { -+ required: true, -+ timeoutMemoKey: path, -+ timeoutMs, -+ }); - } - } ++import { createHash } from "node:crypto"; ++import { lstatSync, mkdirSync, realpathSync, statSync } from "node:fs"; ++import { join, resolve, win32 } from "node:path"; ++import { AsyncLocalStorage } from "node:async_hooks"; ++import { Database } from "bun:sqlite"; ++ ++import { withConfigMutationLockSync } from "../config"; ++import { updateIntegrationRecord } from "./integration-record"; ++import { resolveEffectiveUserIdentity, resolveOsRuntimeDirectory } from "./user-identity"; ++ ++function lockDatabasePath(canonicalHome: string): string { ++ const identity = resolveEffectiveUserIdentity(); ++ const identityPart = encodeUserIdentity(identity); ++ const homeId = sha256(LOCK_DOMAIN + canonicalHome); ++ return join(resolveOsRuntimeDirectory(identity), "opencodex", "native-write-locks", "v1", identityPart, `${homeId}.sqlite`); ++} ``` -WP11 does **not** call `hardenStableLockFile` on POSIX: its unconditional `chmodSync` -is compatible with existing native-main users but forbidden for this strict namespace. -WP11 validates exact POSIX metadata instead. On Windows the existing ACL operation is -the authoritative platform control. +No `node:os` home accessor is imported. -Cap the ACL helper's existing configured budget, leaving all current callers -unchanged: - -```diff -diff --git a/src/lib/windows-secret-acl.ts b/src/lib/windows-secret-acl.ts ---- a/src/lib/windows-secret-acl.ts -+++ b/src/lib/windows-secret-acl.ts -@@ -53,2 +53,4 @@ - timeoutMemoKey?: string; -+ /** Optional stricter caller budget; never enlarges OPENCODEX_ACL_TIMEOUT_MS. */ -+ timeoutMs?: number; - } -@@ -68,7 +70,10 @@ --function resolveHardenDeadlineMs(): number { -+function resolveHardenDeadlineMs(opts: HardenOptions): number { - const raw = env["OPENCODEX_ACL_TIMEOUT_MS"]?.trim(); -- if (!raw) return HARDEN_DEADLINE_DEFAULT_MS; - const parsed = Number(raw); -- if (!Number.isSafeInteger(parsed)) return HARDEN_DEADLINE_DEFAULT_MS; -- return Math.min(HARDEN_DEADLINE_MAX_MS, Math.max(HARDEN_DEADLINE_MIN_MS, parsed)); -+ const configured = raw && Number.isSafeInteger(parsed) -+ ? Math.min(HARDEN_DEADLINE_MAX_MS, Math.max(HARDEN_DEADLINE_MIN_MS, parsed)) -+ : HARDEN_DEADLINE_DEFAULT_MS; -+ if (opts.timeoutMs === undefined) return configured; -+ if (!Number.isFinite(opts.timeoutMs) || opts.timeoutMs <= 0) return 1; -+ return Math.max(1, Math.min(configured, Math.floor(opts.timeoutMs))); - } -@@ -426 +431 @@ function hardenEntry( -- const deadline = nowFn() + resolveHardenDeadlineMs(); -+ const deadline = nowFn() + resolveHardenDeadlineMs(opts); -@@ -470 +475 @@ async function hardenEntryAsync( -- const deadline = nowFn() + resolveHardenDeadlineMs(); -+ const deadline = nowFn() + resolveHardenDeadlineMs(opts); -``` +## Acquisition, release, and reentrancy — C5 -The `tests/windows-secret-acl.test.ts` addition injects the existing async runner, -calls `hardenSecretDirAsync(path, { required: true, timeoutMs: 37 })`, and asserts -every runner invocation receives `<= 37`; a failed runner still rejects. This is a -deadline plumbing test, not an ACL mock standing in for the Windows CI job. +The total timeout is required, finite, integral, and within `0..30_000` ms. +Acquisition uses monotonic `performance.now()`. Zero receives one fail-fast +`BEGIN IMMEDIATE`. Only SQLite busy/locked retries; filesystem, ACL, malformed DB, +identity, permission, and journal-mode failures are refusals. -## Acquisition loop, release, and reentrancy — C5 +Retry sleeps are async uniformly bounded 25-75 ms, clipped to remaining deadline, +and abortable. Barging is allowed; no caller/test infers FIFO. Candidate SQLite and +side descriptors close after every failed attempt. -The core new-file hunk is: +`AsyncLocalStorage>` rejects same-task same-home reentrancy. +A separate task is an ordinary contender. Caller exceptions propagate after +rollback/release; they are never converted to busy/refused. ```diff -diff --git a/src/codex/codex-write-lock.ts b/src/codex/codex-write-lock.ts -new file mode 100644 ---- /dev/null -+++ b/src/codex/codex-write-lock.ts -@@ -+const RETRY_MIN_MS = 25; -+const RETRY_MAX_MS = 75; -+ -+function isBusy(error: unknown): boolean { -+ const code = errorCode(error); -+ const message = error instanceof Error ? error.message : String(error); -+ return code === "SQLITE_BUSY" -+ || code === "SQLITE_LOCKED" -+ || /database (?:is|table is) locked/i.test(message); -+} -+ -+function jitter(random = Math.random): number { -+ return RETRY_MIN_MS + Math.floor(random() * (RETRY_MAX_MS - RETRY_MIN_MS + 1)); -+} -+ -+async function abortableSleep(ms: number, signal?: AbortSignal): Promise { -+ if (signal?.aborted) return false; -+ return new Promise(resolveSleep => { -+ let settled = false; -+ const finish = (completed: boolean): void => { -+ if (settled) return; -+ settled = true; -+ clearTimeout(timer); -+ signal?.removeEventListener("abort", onAbort); -+ resolveSleep(completed); -+ }; -+ const onAbort = (): void => finish(false); -+ const timer = setTimeout(() => finish(true), ms); -+ signal?.addEventListener("abort", onAbort, { once: true }); -+ }); -+} -+ -+function release(database: Database | undefined, file: StableLockFile | undefined): void { -+ try { database?.exec("ROLLBACK"); } catch { /* close remains the OS release */ } -+ try { database?.close(); } catch { /* transaction is already ending */ } -+ try { file?.close(); } catch { /* SQLite closed before the side descriptor */ } -+} -+ -+export async function withCodexWriteLock( -+ options: CodexWriteLockOptions, -+ locked: (context: CodexWriteLockContext) => Synchronous, -+): Promise> { -+ assertTimeout(options.timeoutMs); -+ const startedAt = performance.now(); -+ const deadline = startedAt + options.timeoutMs; -+ const canonical = canonicalCodexHome(options.codexHome); -+ if (canonical.status !== "ok") return canonical; -+ const lockId = lockIdFor(canonical.path); -+ const context = { canonicalCodexHome: canonical.path, lockId } as const; -+ if (heldHomes.getStore()?.has(canonical.path)) { -+ return refused("reentrant", "A nested Codex write attempted to acquire the same home."); -+ } -+ if (options.signal?.aborted) return busy("cancelled", startedAt); -+ const preflight = options.admit("before_namespace", context); -+ if (preflight.status === "refused") { -+ return refused("authority_not_proven", preflight.message); -+ } -+ -+ return heldHomes.run(new Set([...(heldHomes.getStore() ?? []), canonical.path]), async () => { -+ const target = await ensurePrivateLockTarget(lockId, deadline); -+ if (target.status !== "ok") return target; -+ let attempted = false; -+ for (;;) { -+ let file: StableLockFile | undefined; -+ let database: Database | undefined; -+ let callerCodeStarted = false; -+ try { -+ attempted = true; -+ ({ file, database } = await openCandidate(target, deadline)); -+ database.exec("BEGIN IMMEDIATE"); -+ assertStableLockFile(target.databasePath, file); -+ callerCodeStarted = true; -+ const underLock = options.admit("under_lock", context); -+ if (underLock.status === "refused") { -+ release(database, file); -+ return refused("authority_not_proven", underLock.message); -+ } -+ assertStableLockFile(target.databasePath, file); -+ const value = locked(context); -+ if (value && typeof value === "object" && "then" in value) { -+ throw new TypeError("Codex write locked callback must be synchronous"); -+ } -+ callerCodeStarted = false; -+ assertStableLockFile(target.databasePath, file); -+ release(database, file); -+ return { status: "acquired", value, waitedMs: elapsed(startedAt), lockId }; -+ } catch (error) { -+ release(database, file); -+ if (callerCodeStarted) throw error; -+ if (!isBusy(error)) return mapAcquireRefusal(error); -+ if (options.signal?.aborted) return busy("cancelled", startedAt); -+ const remaining = deadline - performance.now(); -+ if (attempted && remaining <= 0) return busy("deadline", startedAt); -+ const slept = await abortableSleep(Math.min(jitter(), remaining), options.signal); -+ if (!slept) return busy("cancelled", startedAt); -+ } -+ } -+ }); -+} ++const value = withConfigMutationLockSync(() => { ++ const current = options.readAdmissionUnderLock(); ++ assertAdmissionStillCurrent(options.admitted, current); ++ const expectation = beginExpectedNativeTransition(); ++ const result = commit({ canonicalCodexHome, lockId, admission: current, expectation }); ++ updateIntegrationRecord(record => commitExpectedTransition(record, expectation, result)); ++ assertExpectedTransition(readIntegrationRecord(), expectation); ++ return result; ++}); ``` -`ensurePrivateLockTarget` and `openCandidate` implement the namespace rules above. -`openCandidate` always closes both handles on failure and maps -`StableLockPathUnsafeError` to `lock_path_unsafe`. It validates sidecars afresh on -every retry because another process may replace a path while this contender sleeps. - -The timeout is monotonic (`performance.now`), required, and total. A zero timeout -still gets exactly one `BEGIN IMMEDIATE`; if busy, it returns immediately. No SQLite -busy timeout, ACL subprocess, or retry sleep may exceed the remaining outer budget. -Jitter is uniformly bounded to integer 25–75 ms and clipped to the deadline. -Contenders may barge after any sleep; no test or caller may infer FIFO order. - -Only `SQLITE_BUSY`/`SQLITE_LOCKED` enters the retry loop. Filesystem, ACL, malformed -database, unexpected journal mode, identity, and permission failures are refusals, -not contention. There is no catch that turns callback exceptions into `busy` or -`refused`; after release they propagate unchanged. +The commit callback performs no logging or response shaping. Those occur after both +locks release. -## Deadlock order and current inverse-nesting proof +## Deadlock order and sibling history sequence -The only legal nested order is: +Legal order: ```text -Codex write lock (async acquisition; synchronous held callback) - -> withConfigMutationLockSync / mutatePersistedConfig - -> return before the Codex callback returns - -> fixed native commit --> release Codex write lock +native lock + -> config mutation lock + -> authoritative AdmissionSnapshot re-read + -> config generation read/update when config changes + -> synchronous native commit + -> integration-record native generation + txId update + -> release config +-> release native + +history lock (later, in Worker) + -> reject stale CommitExpectation / authoritySnapshotId + -> manifest + rollouts + DB + post-probe + history record +-> release history ``` -Never call `withCodexWriteLock` from inside `withConfigMutationLockSync`, from a -`mutatePersistedConfig` mutation callback, or from a helper reached by either -callback. Outer config contention remains `ConfigMutationLockError`; WP12 may retry -that synchronous acquisition only while the outer Codex deadline remains. It must -not release and silently reorder the requested state change. +The native and history locks are **not nested**. Native releases before the Worker +acquires history; history never acquires native/config. A stale history job is +generation/transaction-rejected before mutation, so sibling sequencing cannot let +an old ON job overtake a newer OFF transition (`005_contract.md` §6). -Fresh search on the current tree found no inverse edge: +Never call `withCodexWriteLock` from inside `withConfigMutationLockSync` or a +`mutatePersistedConfig` callback. Current inverse-edge search found config-owned +callbacks at `src/config.ts:1829,1870,2145`, the account wrapper at +`src/codex/account-store.ts:281`, and auth mutation at +`src/codex/auth-api.ts:670`; none currently imports the new lock. Add a dependency- +graph test that protects this direction. Source substring matching inside one file +is not enough. -```text -$ rg -n 'withConfigMutationLockSync\(|mutatePersistedConfig\(' src --glob '*.ts' -src/config.ts:1829: withConfigMutationLockSync(() => persistConfigUnlocked(config)); -src/config.ts:1870: return withConfigMutationLockSync(() => { -src/config.ts:2145: withConfigMutationLockSync(() => { -src/codex/account-store.ts:281: return withConfigMutationLockSync(fn); -src/codex/auth-api.ts:670: outcome = mutatePersistedConfig(persistedConfig => { -``` +## Shared helper deadline changes -The three config-owned sections perform config snapshots/persistence only -(`src/config.ts:1821-1829,1861-1913,2144-2176`). The account wrapper is a direct -typed-error translation (`src/codex/account-store.ts:278-285`). The sole external -`mutatePersistedConfig` callback updates plan strings and performs no Codex native -operation (`src/codex/auth-api.ts:660-701`). None imports the new module today. -Therefore adding the future WP12 edge `codex-write -> config` cannot close a cycle -in the current graph. +`native-main-lock-file.ts` keeps ownership of stable descriptors. Add only an +optional stricter timeout for Windows hardening: -Add a source-shape case to `tests/codex-write-lock.test.ts` that reruns this inventory -over `src/config.ts`, `src/codex/account-store.ts`, and `src/codex/auth-api.ts`, and -fails if `codex-write-lock` or `withCodexWriteLock` appears inside an existing -config-lock callback. This test protects inverse nesting; it does not reject a WP12 -orchestrator that correctly acquires Codex first and calls config second. +```diff +-export async function hardenStableLockFile(path: string): Promise { ++export async function hardenStableLockFile(path: string, timeoutMs?: number): Promise { + try { chmodSync(path, 0o600); } catch {} + if (process.platform === "win32") { +- await hardenSecretPathAsync(path, { required: true, timeoutMemoKey: path }); ++ await hardenSecretPathAsync(path, { required: true, timeoutMemoKey: path, timeoutMs }); + } + } +``` + +`windows-secret-acl.ts` clamps the caller value to the existing configured budget; +it may shorten but never enlarge it. Existing callers that omit `timeoutMs` retain +current behavior. Required ACL failure still rejects. ## Test plan -### `tests/helpers/codex-write-lock-child.ts` (NEW) - -The helper accepts `CODEX_HOME`, `HOLD_MS`, and marker paths through its environment. -It calls the production `withCodexWriteLock` with a 5-second deadline and an -always-admitted test callback, writes `READY_PATH`, then executes one finite -`Bun.sleepSync(HOLD_MS)` inside the synchronous locked callback. It prints the typed -result as one JSON line and exits nonzero unless status is `acquired`. It never -opens SQLite directly; contention must exercise the production namespace and API. - -### `tests/codex-write-lock.test.ts` (NEW) - -Every test creates both a fake login home and existing Codex homes below one test -root, sets `HOME`/`USERPROFILE`, and restores them in `afterEach`. It never resolves -the real user's `.codex` or `.opencodex`. - -1. **Real two-process exclusion and barging contract.** Spawn the child on home A, - wait for its ready marker, then call the production API in the parent. A 100 ms - deadline returns `{ status:"busy", reason:"deadline" }` and the parent callback - does not run. A second parent waiter with 2 s acquires after the child's bounded - release. A timer increments while waiting, proving retry sleep is async. Assert - exclusion and eventual two-party acquisition only, never arrival order. -2. **Crash release, no stale recovery.** Child acquires and exits from inside its - callback. After its zero exit, parent acquires the persistent database without - unlink, PID, mtime, quarantine, or recovery marker. This mirrors the existing - OS-release proof (`tests/config-mutation-lock.test.ts:105-129`). -3. **Live holder is never stolen.** Hold longer than two successive parent deadlines; - both return busy, the database inode is unchanged, and no path is removed. Age is - not takeover authority. -4. **Deadline and cancellation.** Zero gets one fail-fast attempt; finite expiry is - typed busy; an already-aborted signal returns `busy/cancelled`; a signal fired - during jitter cancels the timer and returns the same. None throws contention. -5. **Callback boundary.** A synchronous value appears in `acquired.value`; a thrown - domain error propagates after release; an `async` callback is a compile-time - `@ts-expect-error`; a cast thenable activates the runtime `TypeError` and releases - the lock for a later call. -6. **Admission order.** `before_namespace` refusal leaves - `.opencodex/native-write-locks` absent. Under-lock refusal may leave the persistent - database but never calls the locked callback. Record phase order exactly as - `before_namespace, under_lock, locked`. -7. **Same-task reentrancy.** Calling the API again for the same canonical home from - the callback returns `refused/reentrant` without waiting. A separately started - same-process task is an ordinary contender and acquires after release. -8. **Default/explicit/absolute/tilde.** With default login `.codex` existing, delete - `CODEX_HOME` and have the child hold the default spelling. Parent attempts using - explicit `~/.codex` and the absolute path both return busy on the same `lockId`. -9. **Symlinked home.** Child holds a real directory; parent targets a symlink to it. - The parent is busy and the one expected full-hash database exists. Reverse the - spellings so the default itself is the symlink; the result is identical. -10. **Case behavior.** Create `CaseHome`, then probe `casehome`. If the platform - resolves both to the same existing directory, assert contention and one ID. If - the alternate spelling is missing, first assert `codex_home_missing` creates no - namespace; then create the second directory and assert both acquire concurrently - with distinct IDs. On Windows, slash and drive-letter case variants also share - one ID. -11. **Distinct homes.** Hold home A in the child and acquire home B immediately in - the parent. Assert two different 64-hex IDs and database paths. -12. **Missing home.** Test explicit and default missing paths. Both return - `codex_home_missing`, `admit` is not called, and the fake login home still has no - `.opencodex` descendant. This activates the case-sensitive/case-insensitive - resolution rather than testing only a pure hash helper. -13. **Namespace symlinks.** Independently replace `.opencodex`, - `native-write-locks`, and `v1` with a real symlink/junction. Each returns - `namespace_unsafe`, preserves the entry/target byte-for-byte, and creates no DB. -14. **Database/sidecar substitution.** A symlink database, symlink `-journal`, or - existing `-wal`/`-shm` returns `lock_path_unsafe` and is not removed. A test hook - swaps the DB after stable open and proves `(dev, ino)` revalidation refuses. -15. **POSIX owner/mode.** Existing namespace modes `0755`/`0700` and DB modes - `0644`/`0600` cover refusal/success. Inject a mismatched effective uid for the - deterministic wrong-owner branch; when CI runs as uid 0, additionally `chown` - a fixture and prove the real metadata branch. No test expects chmod repair. -16. **Windows ACL and junctions.** On `windows-latest`, create real directory - junctions for each namespace component and require refusal. Inject the existing - `icacls` runner for required failure/timeout mapping, while the normal success - case runs the real required ACL path. UNC and WSL DrvFS identities return - `unsupported_filesystem` through the existing predicate. -17. **Malformed database and rollback journal.** Preserve malformed bytes and return - `lock_unavailable`; accept a same-owner/mode regular rollback journal, refuse - wrong metadata, and never silently switch to WAL. -18. **Deadlock source shape.** Re-run the inventory described above and pin - `Codex-write -> config`, with no inverse callback acquisition. - -The wrong-owner uid injection is only for a branch a non-root CI process cannot -materialize. Symlink, mode, substitution, SQLite contention, process crash, and -deadline tests all use real filesystem/process behavior. +`tests/helpers/codex-write-lock-child.ts` imports and calls the production API. It +accepts explicit test paths/timing through its environment, prints one typed result, +and never opens SQLite directly. + +### Real-process identity and exclusion + +1. Child holds home A; parent deadline returns typed busy and its callback does not + run; a timer advances while waiting; parent later acquires after release. +2. Abrupt holder exit releases without unlink/stale recovery; live holder is never + stolen across repeated deadlines. +3. Default/explicit/absolute/tilde/symlink/case-equivalent spellings of one existing + home produce one ID; distinct homes acquire independently. +4. Missing homes refuse before namespace creation. + +### Effective-user namespace activation — carried #7/C18 + +Run real pinned-Bun child processes, not pure resolver mocks: + +1. Child A: `HOME=`, `USERPROFILE=`; hold the production lock. +2. Child B: `HOME=`, `USERPROFILE=`; same OS user and + `CODEX_HOME`; assert busy on the **same** DB path/lock id. +3. Repeat with `HOME=` and independently different + `USERPROFILE=`. +4. On Windows, vary `USERPROFILE` while retaining the real account SID. On POSIX, + vary both variables independently while retaining the real uid. +5. Assert exactly one namespace under the uid/SID component. A test that sets HOME + and USERPROFILE to the same fake value in both children is insufficient because + it cannot catch the original split. + +Use `process.execPath` and assert the pinned Bun version expected by CI before the +probe. Do not substitute Node or a same-process environment mutation. + +### Admission/config-record ordering + +- Non-authorizing pre-snapshot leaves the runtime namespace absent. +- Under-lock authoritative snapshot mismatch calls no commit and writes no record. +- A cooperating config transition while gather is outside the lock prevents stale + commit. +- A successful commit shows exact `nativeAfter` and this `txId`; another tx at the + same numeric generation is interference. +- Inject config-lock contention; assert bounded retry/typed busy and no stale + commit. +- Prove config/native generation and integration-record update occur inside the + synchronous section by blocking a contender at each seam. + +### Boundary/hardening + +- Compile-time async callback rejection plus runtime thenable rejection and release. +- Callback throw releases then propagates. +- Namespace symlink/junction, wrong owner/mode, DB/journal substitution, WAL/SHM, + malformed DB, ACL failure/timeout, unsupported filesystem all refuse without + repair/deletion. +- Windows CI executes real SID/junction/ACL success; POSIX executes real uid/mode. +- Dependency graph proves no inverse config->native acquisition and no history/native + nesting. ## Verification -No verification command starts, stops, syncs, restores, or ensures the proxy. Port -10100 remains untouched. - -Run in this order after WP9 and WP10 are present and the diff is implemented: - ```bash bun test tests/codex-write-lock.test.ts --test-name-pattern "real two-process exclusion" +bun test tests/codex-write-lock.test.ts --test-name-pattern "HOME and USERPROFILE independently" bun test tests/codex-write-lock.test.ts tests/windows-secret-acl.test.ts tests/native-main-claim.test.ts tests/native-main-owner-lifetime.test.ts tests/config-mutation-lock.test.ts bun run typecheck bun run test bun run privacy:scan ``` -The first command is the required real two-process contention run: the child holds -the production SQLite transaction, the parent expires once as typed `busy`, remains -event-loop responsive, then acquires after release. A mocked `SQLITE_BUSY`, two -connections in one process, or a pure lock-ID test does not satisfy it. - -Run the focused test and typecheck on macOS, Linux, and Windows. Windows must execute -the real junction and ACL-success cases; POSIX must execute exact uid/mode checks. -The full suite is required because `native-main-lock-file.ts` and -`windows-secret-acl.ts` are shared owners even though their existing defaults are -preserved. +Run focused tests/typecheck on macOS, Linux, and Windows. No command starts, stops, +syncs, restores, or ensures the proxy; port 10100 is untouched. ## Deliberate residuals -- `realpath` does not collapse bind-mount or filesystem-namespace aliases. Portable - directory file identity and cross-host/network-filesystem coordination remain - unsupported, as `003_lock_protocol.md:345-350` already marks **INFERRED**. WP11 - refuses the target classes the repository can identify; it does not claim every - alias can be detected portably. -- The concrete 15-second OFF and 5-second startup/background budgets in - `003_lock_protocol.md:173-176` remain caller-policy in WP12. WP11 enforces only the - required finite `0..30_000` ms API bound. -- Creation of a missing `CODEX_HOME` remains a separate lock domain. No future WP12 - convenience path may weaken missing-home refusal in this module. +- `realpath` does not collapse bind-mount/filesystem-namespace aliases. Unsupported + network target classes are refused; arbitrary cross-namespace identity is not + claimed. +- The caller chooses a finite automatic/explicit deadline within the API cap. WP11 + owns enforcement, not later policy values. +- Missing `CODEX_HOME` creation remains another operation/domain. +- Non-cooperating arbitrary filesystem ABA is outside the contract's proof bound. ## Accept criteria -- **C5 — finite async acquisition and typed contention.** Every call supplies an - integral `0..30_000` ms total deadline. Real cross-process `BEGIN IMMEDIATE` - contention yields `busy/deadline`, cancellation yields `busy/cancelled`, ordinary - acquisition yields `acquired`, and unsafe setup yields `refused`; contention is - never an exception. Retry sleeps are async 25–75 ms bounded jitter, FIFO is not - claimed, the locked callback is synchronous/bounded, and no PID/mtime takeover or - stale-file unlink exists. -- **C6 — one identity per real home.** Both explicit and default homes require an - existing directory and pass through `realpathSync.native`; Windows additionally - normalizes/case-folds. Default, explicit, absolute, tilde, symlink, separator, and - case-equivalent spellings contend on one full SHA-256 lock, while two different - existing canonical directories acquire independently. A missing home refuses - before admission, hashing side effects, or namespace creation. -- **C7 — private hardened namespace.** The database path is exactly - `/.opencodex/native-write-locks/v1/.sqlite`, never - `tmpdir`, `CODEX_HOME`, or `OPENCODEX_HOME`. POSIX requires real same-uid `0700` - directories and a same-uid `0600` regular DB/rollback journal; Windows requires - non-junction identity plus successful required per-user ACL hardening within the - outer deadline. Symlink, wrong-owner/mode, substitution, WAL/SHM residue, ACL - failure, and unsupported filesystem identity refuse without chmod repair, rename, - unlink, or recreation. - -WP11 is complete only when the focused real-process activation, cross-platform -hardening cases, typecheck, full tests, and privacy scan all pass. It still does not -authorize a native write: WP12 must supply the two read-only admissions and handle -config-lock retry/commit outcomes under this exclusion boundary. +- **C5** — finite async acquisition yields typed acquired/busy/refused behavior; + callback is synchronous/bounded; no stale takeover or FIFO claim exists. +- **C6** — all real spellings of one existing home share one lock; distinct homes + do not; missing homes refuse before artifacts. +- **C7/C18** — namespace keys on effective uid/SID beneath the OS runtime directory, + never any home accessor. Real pinned-Bun children with independently varied HOME + and USERPROFILE prove one lock for one user/home. +- Config generation, authoritative admission re-read, native commit, expected + native generation/txId, and integration-record updates share the synchronous + native->config section. +- Native and history locks are never nested; stale history jobs are rejected by + generation/transaction identity. +- **N2** — WP11 extends the already-working funnel and typechecks/preserves behavior + at its own commit; WP12 strengthens admission without supplying missing mechanics. diff --git a/devlog/_plan/260804_codex_write_substrate/040_ownership_convergence.md b/devlog/_plan/260804_codex_write_substrate/040_ownership_convergence.md index a6c5dc022..f6914cb24 100644 --- a/devlog/_plan/260804_codex_write_substrate/040_ownership_convergence.md +++ b/devlog/_plan/260804_codex_write_substrate/040_ownership_convergence.md @@ -1,21 +1,36 @@ # WP12 — ownership authority and convergence -Research: `004_ownership_and_convergence.md`. Read it first; this document is the -implementation diff for its Decision. Source citations and diff context below were -re-verified at `7e67a8d06311de2471b0a25e41cf85f97007cc69` on 2026-08-04. +Research: `004_ownership_and_convergence.md`. Shared contract: +`005_contract.md`. The failure to prevent is data loss, not an untidy status result. Today a dead-PID -version-1 journal with no injected hashes is treated as permission to replay its -baseline (`src/codex/journal.ts:109-134`), while a corrupt service-state mirror is -collapsed into the same `null` as no service (`src/service.ts:165-175`) and the -native teardown preflight converts that uncertainty to success -(`src/integrations/native/ownership-preflight.ts:21-35`). If startup replaces the -separate external-provider check with that service check, it can overwrite a -`config.toml` now owned by another provider. WP9-WP11 add a gather/commit seam, -off-event-loop history work, and a bounded per-home lock. WP12 adds the authority -admission that must run before that lock can create anything, the provenance needed -to prove what OpenCodex created, and an observed-state projection that distinguishes -desired intent from actual convergence. +version-1 journal without injected hashes can replay its baseline +(`src/codex/journal.ts:109-162`), corrupt service-state mirrors collapse toward the +same absence result used for no service (`src/service.ts:165-175`), and native +teardown fails open for errors outside its one mismatch class +(`src/integrations/native/ownership-preflight.ts:21-35`). A matching service-home +check also does not authorize overwriting a `config.toml` whose effective +`model_provider` is now external. + +WP9-WP11 already provide the working `convergeCodex` funnel, catalog split, +history protocol, generations, integration-record owner, and native lock. WP12 +completes the mechanisms behind that funnel: tri-state service authority, +file-backed intent, journal/provenance admission, restoration, and observed-state +inspection. It does **not** add another record module, another convergence module, +another route mapping, or another public result union. + +The prior plan named `write-lock.ts`, created `ownership-convergence.ts`, redefined +`integrations/codex.json`, and exported `convergeCodexToPersistedIntent`. Those are +deleted. Contract names are exact: `codex-write-lock.ts`, `integration-record.ts`, +and `convergence.ts` (`005_contract.md` §8). + +WP12 is independently landable: it modifies the existing working funnel and +contract implementations in one commit, rewires every remaining lifecycle caller +in that commit, and typechecks/preserves behavior without a future phase. WP13 may +re-prove composition; it is not required to make WP12 correct. + +All current-code citations and diff context below were rechecked on 2026-08-04 at +`2d5e080dea3e7000bf2111b381c7c1a3c4f5fb11`. ## IN / OUT @@ -23,779 +38,464 @@ IN: | Path | Change | Why | |---|---|---| -| `src/types.ts` | MODIFY | Adds only `clientIntegrations.codex?: boolean`; absent means desired ON. | -| `src/config.ts` | MODIFY | Parses the one-key extension-safe object and exports a pure, file-backed Codex intent reader. | -| `src/service.ts` | MODIFY | Stops skipping bad service-state mirrors and exposes read-only registration/mirror evidence. | -| `src/integrations/native/ownership-preflight.ts` | MODIFY | Replaces the fail-open boolean preflight with the tri-state mutation authority. | -| `src/codex/integration-record.ts` | NEW | Owns `getConfigDir()/integrations/codex.json`, including WP10 history state and the exact provenance ledger below. | -| `src/codex/ownership-convergence.ts` | NEW | Owns read-only admission, gather/lock/recheck orchestration, observed state, and convergence results. | -| `src/codex/journal.ts` | MODIFY | Makes journal inspection read-only and typed; recovery becomes an under-lock operation over a previously inspected dead writer. | -| `src/codex/inject.ts` | MODIFY | Preserves external-provider bytes, removes filename-based deletion authority, and exposes only receipt-gated apply/restore commits. | -| `src/codex/sync.ts` | MODIFY | Delegates apply to the common convergence owner instead of gathering/writing from a startup-captured config object. | -| `src/codex/catalog/sync.ts` | MODIFY | Records catalog/cache post-images and restores baseline absence; cache invalidation is no longer an unowned write. | -| `src/server/index.ts` | MODIFY | Removes the unconditional startup cache write at current line 403. | -| `src/cli/index.ts` | MODIFY | Routes start and both ensure branches through the one admission order; proxy startup survives an ownership refusal. | -| `src/server/management/config-routes.ts` | MODIFY | Makes `/api/sync` reread persisted intent instead of passing the server-captured `config`. | -| `tests/codex-ownership-authority.test.ts` | NEW | Pins owned/foreign/unknown and the no-artifact-before-answer invariant. | -| `tests/codex-artifact-provenance.test.ts` | NEW | Pins baseline absence, matching-post-image deletion, and preserved-drift conflict behavior. | -| `tests/codex-observed-state.test.ts` | NEW | Pins the complete observed projection and desired/observed convergence relation. | -| `tests/codex-convergence-order.test.ts` | NEW | Pins the trace order for startup, ensure, sync, apply, restore, stop, and uninstall entry points. | -| `tests/codex-journal.test.ts` | MODIFY | Reverses corrupt/unknown journal deletion and markerless automatic replay expectations. | -| `tests/codex-models-cache-restore.test.ts` | NEW | Proves an apply-created cache returns to absence and native drift is preserved. | -| `tests/codex-sync-api.test.ts` | MODIFY | Proves one running server observes CLI intent changes made by another process. | -| `tests/service.test.ts`, `tests/uninstall.test.ts` | MODIFY | Pin mirror conflict/unreadable evidence and fail-closed teardown. | -| `docs-site/src/content/docs/reference/cli/lifecycle.md` | MODIFY | Documents blocked/external/partial convergence without claiming that proxy startup failed. | -| `docs-site/src/content/docs/reference/configuration.md` | MODIFY | Documents `clientIntegrations.codex`, absent-means-ON, and desired versus observed state. | - -The predecessor names `src/codex/write-lock.ts` (WP11) and -`src/codex/history-convergence.ts` (WP10) are consumed but not redesigned here. -WP12 may modify their exported record composition/types only where the exact -`integrations/codex.json` schema below requires it; it must not weaken WP10's -off-event-loop boundary or WP11's acquisition protocol. - -OUT: `gui/**`, Grok, Claude Code, Claude Desktop, the six file integrations, -provider transport, releases, publishing, deployment, tags, npm, and the live proxy -on port 10100. WP12 supplies state/result types for the later Codex toggle, but it -does not add that route or render a switch. It does not promise byte-exact rollback -after a user or Codex has edited a baseline-absent artifact; that case can only be -preserved and reported. - -## The tri-state authority - -The public API belongs at the existing native preflight boundary. It returns the -canonical homes and evidence because a boolean cannot distinguish “another home -owns this” from “the ownership record could not be read”. - -```ts -/** - * Whether this process may mutate native Codex artifacts for one canonical home. - * - * `owned` is positive evidence: either no service registration and no mirror - * exist, or every readable/required mirror agrees with the installed service and - * the current canonical homes. `foreign` is a valid claim by another home. - * `unknown` means the evidence needed to choose is missing or cannot be trusted. - * Callers must permit native writes only for `owned`. - */ -export type NativeCodexOwnership = - | { - state: "owned"; - evidence: "no-service" | "matching-install"; - codexHome: string; - opencodexHome: string; - } - | { - state: "foreign"; - codexHome: string; - opencodexHome: string; - recordedCodexHome: string; - recordedOpenCodexHome: string; - message: string; - } - | { - state: "unknown"; - reason: - | "service-state-missing" - | "service-state-corrupt" - | "service-state-unreadable" - | "service-state-conflict" - | "service-registration-unknown" - | "path-unresolvable"; - codexHome?: string; - opencodexHome?: string; - message: string; - }; - -/** - * Read service registration and every known install-state mirror without repair, - * directory creation, SQLite open, chmod, unlink, rename, or config loading. - */ -export function inspectNativeCodexOwnership(): NativeCodexOwnership; -``` - -`assertNativeTeardownOwned` currently fails open for every error that is not the -specific mismatch class (`src/integrations/native/ownership-preflight.ts:25-35`). -That behavior was written for an interactive teardown route where a human sees the -result and can immediately repair a stale service record. Automatic convergence is -unattended: it runs during startup, ensure, server requests, crash recovery, and -later retries. In that setting an unreadable authority record cannot be converted -to deletion permission. A false refusal leaves residue that can be inspected; a -false success can destroy a newer config, catalog, or cache. Therefore both -`foreign` and `unknown` refuse, while the proxy itself may continue serving. - -### Actual diff — `src/integrations/native/ownership-preflight.ts:14-35` +| `src/types.ts` | MODIFY | Add `clientIntegrations.codex?: boolean`; absent means desired ON. | +| `src/config.ts` | MODIFY | Parse the extension-safe object; own config generation bumps and authoritative `AdmissionSnapshot` reads. | +| `src/service.ts` | MODIFY | Preserve all service registration/mirror evidence instead of skipping corrupt/unreadable rows. | +| `src/integrations/native/ownership-preflight.ts` | MODIFY | Tri-state read-only service-home authority; only owned permits native mutation. | +| `src/codex/convergence.ts` | MODIFY | Complete admission, provenance, restore, observation, and lifecycle routing behind the contract entry point. | +| `src/codex/convergence-types.ts` | IMPORT ONLY | Consume `AdmissionSnapshot`, `CodexObservedState`, `ConvergeOutcome`, `CodexProvenanceLedger`, and section types; no WP12 union. | +| `src/codex/integration-record.ts` | USE/MODIFY THROUGH OWNER API | Read/update provenance and native transition through the contract owner; no path/schema/parser here. | +| `src/codex/codex-write-lock.ts` | CONSUME | Correct WP11 module name; no lock redesign. | +| `src/codex/journal.ts` | MODIFY | Read-only typed inspection; authorized recovery only inside convergence. | +| `src/codex/inject.ts` | MODIFY | Receipt-gated internal apply/restore mechanics; remove filename-based deletion authority. | +| `src/codex/sync.ts` | MODIFY | Remove the remaining alternate native orchestration; delegate to `convergeCodex`. | +| `src/codex/catalog/sync.ts` | MODIFY | Report post-images and perform provenance-authorized restoration behind convergence. | +| `src/server/index.ts` | MODIFY | Remove unconditional cache invalidation at current line 403. | +| `src/cli/index.ts`, `src/service.ts` | MODIFY | Route startup, ensure, explicit restore/eject, stop, uninstall, and recovery through `convergeCodex`. | +| `src/server/management/config-routes.ts` | MODIFY | Call `convergeCodex` and the contract response adapter only. | +| `tests/codex-ownership-authority.test.ts`, `tests/codex-artifact-provenance.test.ts`, `tests/codex-observed-state.test.ts`, `tests/codex-convergence-order.test.ts`, `tests/codex-models-cache-restore.test.ts` | NEW | Authority, provenance, observation, ordering, and current-byte drift. | +| `tests/codex-journal.test.ts`, `tests/codex-sync-api.test.ts`, `tests/service.test.ts`, `tests/uninstall.test.ts`, `tests/codex-convergence-contract.test.ts` | MODIFY | Recovery, fresh intent, fail-closed service behavior, and production funnel. | +| `docs-site/src/content/docs/reference/cli/lifecycle.md`, `docs-site/src/content/docs/reference/configuration.md` | MODIFY | Refusal/recovery and persisted intent; link route behavior to the contract adapter. | + +OUT: + +- `src/codex/ownership-convergence.ts` — deleted from the plan. There is one entry + module, `src/codex/convergence.ts`. +- Ownership of `src/codex/integration-record.ts`, its path/schema/validators, or + `/api/sync` mapping — `005_contract.md` §§1, 5. +- A module named `src/codex/write-lock.ts`; the consumer import is + `src/codex/codex-write-lock.ts`. +- New request/result/observed-state/provenance section unions. All shared shapes + come from `convergence-types.ts`. +- GUI, Grok, Claude Code/Desktop, six file integrations, provider transport, + release/publish/deploy actions, and the live proxy on 10100. +- The Pi required-nonempty file-client incident. The third baseline class is + removed and remains `FOLLOWUP-FILECLIENT-01` (`005_contract.md` §9). + +## Tri-state service-home authority + +The service preflight owns evidence collection, not the shared convergence result. +Its implementation may use a private/local discriminated union so it can explain +why an `AdmissionSnapshot.ownership` is `owned | foreign | unknown`; it must not +export a second convergence outcome. + +Truth table: + +| Registration evidence | Mirror evidence | Admission ownership | +|---|---|---| +| absent | all absent | `owned` | +| installed | all required mirrors valid and canonical pairs match current | `owned` | +| any | decisive valid mirror names another pair | `foreign` | +| installed | all absent | `unknown` | +| any | corrupt, unreadable, conflicting mirrors | `unknown` | +| unknown | no decisive valid foreign claim | `unknown` | +| any | any required path cannot be canonicalized | `unknown` | + +Corrupt/unreadable/conflicting evidence wins over a convenient absent sibling. A +false refusal leaves inspectable residue; a false success can destroy newer user +state. Only `owned` reaches a native write. + +`src/service.ts:165-175` becomes an all-mirror read that distinguishes absent, +valid, corrupt, and unreadable. Registration probes remain read-only and return +unknown when the platform cannot establish presence. Existing diagnostics may keep +a compatibility “first valid state” view; mutation admission may not use it. ```diff - import { -- assertServiceEnvironmentMatchesInstall, -- isServiceOwnershipError, -+ inspectServiceInstallOwnership, - } from "../../service"; - --export type NativeTeardownOwnership = { ok: true } | { ok: false; message: string }; -+export type NativeTeardownOwnership = -+ | { ok: true; ownership: Extract } -+ | { ok: false; ownership: Exclude; message: string }; - -+/** -+ * Read-only native mutation authority. Only `owned` authorizes a Codex write; -+ * foreign and unknown evidence are equally non-authorizing. -+ */ -+export function inspectNativeCodexOwnership(): NativeCodexOwnership { -+ return inspectServiceInstallOwnership(); -+} -+ - export function assertNativeTeardownOwned(): NativeTeardownOwnership { -- try { -- assertServiceEnvironmentMatchesInstall(); -- return { ok: true }; -- } catch (error) { -- if (isServiceOwnershipError(error)) { -- // The message names both the recorded and the current home — that is the -- // refusal text, verbatim, because the user has to act on it. -- return { ok: false, message: error.message }; -- } -- // Unrelated failure (corrupt state file, IO): mirror -- // `serviceEnvironmentOwnedHere` and fail open rather than wedging the route -- // behind a check whose own input is broken. +-export function assertNativeTeardownOwned(): { ok: boolean; message?: string } { +- try { assertServiceEnvironmentMatchesInstall(); return { ok: true }; } +- catch (error) { +- if (isServiceOwnershipError(error)) return { ok: false, message: error.message }; - return { ok: true }; - } -+ const ownership = inspectNativeCodexOwnership(); -+ return ownership.state === "owned" -+ ? { ok: true, ownership } -+ : { ok: false, ownership, message: ownership.message }; - } +-} ++export function inspectNativeCodexOwnership(): NativeCodexOwnershipEvidence { ++ return inspectAllServiceRegistrationAndMirrors(); ++} ``` -The `NativeCodexOwnership` declaration is inserted above -`NativeTeardownOwnership`; it is shown in full in the API block above and is not -duplicated in the diff. +`NativeCodexOwnershipEvidence` is phase-internal evidence projected to +`AdmissionSnapshot.ownership`; it is not a public shared result family. -### Actual diff — `src/service.ts:165-175` +## One admission order — exact `AdmissionSnapshot` -The low-level reader must retain all mirror outcomes rather than returning the -first convenient valid row. `serviceRegistration` is derived from the existing -platform registration probes used by `diagnoseService` at -`src/service.ts:2370-2416`, but returns `unknown` when the platform probe itself -cannot establish presence. It is read-only; it does not call install, repair, -start, stop, or uninstall. +Every startup, ensure branch, management mutation, explicit sync/restore/eject, +stop, uninstall, retry, and observe uses this sequence. No caller selects a subset: -```diff --function readServiceInstallState(): ServiceInstallState | null { -- for (const path of serviceStatePaths()) { -- try { -- const parsed = parseServiceInstallState(JSON.parse(readFileSync(path, "utf8"))); -- if (parsed) return parsed; -- } catch { -- /* try the next known state path */ -- } -- } -- return null; --} -+export type ServiceInstallStateRead = -+ | { status: "absent"; path: string } -+ | { status: "valid"; path: string; state: ServiceInstallState } -+ | { status: "corrupt"; path: string; message: string } -+ | { status: "unreadable"; path: string; message: string }; -+ -+/** Read every known mirror without creating, deleting, or repairing any path. */ -+export function readServiceInstallStates(): readonly ServiceInstallStateRead[] { -+ return serviceStatePaths().map(path => { -+ try { -+ const parsed = parseServiceInstallState(JSON.parse(readFileSync(path, "utf8"))); -+ return parsed -+ ? { status: "valid" as const, path, state: parsed } -+ : { status: "corrupt" as const, path, message: "invalid service-state schema" }; -+ } catch (error) { -+ const code = (error as NodeJS.ErrnoException).code; -+ if (code === "ENOENT") return { status: "absent" as const, path }; -+ if (error instanceof SyntaxError) { -+ return { status: "corrupt" as const, path, message: error.message }; -+ } -+ return { status: "unreadable" as const, path, message: error instanceof Error ? error.message : String(error) }; -+ } -+ }); -+} -``` +1. Resolve existing canonical `CODEX_HOME`, `OPENCODEX_HOME`, config/profile, + catalog/cache, journal, history, rollouts, and integration-record targets without + creating anything. +2. Read all service registration/mirror evidence. Foreign/unknown refuses. +3. Read effective project `model_provider`. External refuses separately. +4. Inspect journal/liveness without cleanup. Invalid/unknown-version, live writer, + or unknown liveness refuses. +5. Read/validate the contract integration record without creating it. Missing is + legal only when no residue needs provenance proof; corrupt/lost/conflicting + provenance refuses. +6. Authoritatively read persisted config, config generation, intent, and ownership; + return one exact `AdmissionSnapshot`: -Immediately after this reader, add `inspectServiceInstallOwnership()`. Its truth -table is exact: +```ts +const admission: AdmissionSnapshot = { + config: diagnostics.config, + configDigest, + intent, + generation: configGeneration.value, + ownership, +}; +``` -| Registration evidence | Mirror evidence | Result | -|---|---|---| -| absent | all absent | `owned/no-service` | -| installed | all required mirrors valid, canonical pairs equal each other and current pair | `owned/matching-install` | -| any | any valid mirror names another canonical pair | `foreign` | -| installed | all absent | `unknown/service-state-missing` | -| any | corrupt | `unknown/service-state-corrupt` | -| any | unreadable | `unknown/service-state-unreadable` | -| any | two valid canonical pairs disagree | `unknown/service-state-conflict` | -| unknown | no valid decisive foreign claim | `unknown/service-registration-unknown` | -| any | current or recorded path cannot be canonicalized | `unknown/path-unresolvable` | - -A valid foreign claim wins over an absent sibling mirror, but never over a corrupt, -unreadable, or conflicting mirror: those are `unknown`, because the complete evidence -set is not trustworthy. Existing `readServiceBackend`, diagnostics, and interactive -service commands may keep a compatibility helper that selects one valid state; native -mutation admission must use only the all-mirror reader. - -## One admission order (C8) - -Every start, ensure branch, sync, apply, restore, stop, uninstall, and retry uses one -sequence. No caller may select a subset or reorder it: - -1. **Canonical paths.** Resolve existing canonical `CODEX_HOME`, `OPENCODEX_HOME`, - effective `config.toml`, generated profile, active catalog, cache, journal, - history DB, rollouts, integration record, and WP11 lock path without creating - any component. Failure is `unknown/path-unresolvable`. -2. **Service ownership.** Call `inspectNativeCodexOwnership`. `foreign` or - `unknown` returns `blocked` and stops this sequence. -3. **External provider.** Read the effective project `model_provider` from - `config.toml` without mutation. An external provider returns `external` and - stops every journal, config, profile, catalog, cache, history, rollout, backup, - provenance, and lock write. -4. **Journal/liveness.** Inspect without cleanup. Invalid/unknown-version bytes, - a live writer, or liveness `unknown` block. A valid dead writer is recoverable - only after provenance also authorizes it. -5. **Provenance.** Read and validate `integrations/codex.json` without creating it. - A missing record is legal only when no OpenCodex residue requiring ownership - proof exists. Corrupt, wrong-version, conflicting transaction, missing post-image, - or artifact/hash disagreement blocks the corresponding transition. -6. **Fresh intent.** Call `readPersistedCodexIntent`; only diagnostics with - `source === "file"` are authoritative. Missing/unreadable/invalid config is - `unknown`, never default ON. -7. **Gather.** For desired ON, run WP9 provider/catalog gathering outside the lock. - It may await network I/O and must not write. Desired OFF has no gather step. -8. **Lock/recheck.** Only now call WP11 acquisition. Its construction order remains - canonical-home validation -> authority receipt -> private namespace validation -> - stable lock file -> SQLite open -> `BEGIN IMMEDIATE` - (`003_lock_protocol.md:178-196`). Once acquired, repeat steps 1-6 from disk and - compare the new authority/intent digest with the pre-lock receipt. Any change - aborts before a native write. -9. **Commit/observe.** Recover an authorized dead journal first, then apply or - restore using the locked candidate/ledger. Read observed state while still - serialized. Release before logs, HTTP response shaping, network retries, or - app-server handling. +7. If intent is ON, WP9 gather receives `admission.config` — **that exact object**. + OFF does not gather. +8. Call WP11 with `admitted: admission`. Under native->config coordination, + authoritatively re-read steps 1-6 into a second `AdmissionSnapshot` and compare + digest, generation, intent, ownership, canonical targets, journal identity, and + provenance identity. +9. Recover an authorized dead journal, establish baselines, commit apply/remove, + write the expected native generation/`txId`, and inspect observed state inside + the coordinated section. Release before logging/HTTP shaping. +10. Run WP10 history afterward under its sibling lock with the same + `CommitExpectation` and authority snapshot identity; stale jobs are rejected. Testable invariant: -> Until steps 1-6 have returned authoritative answers, the filesystem snapshot must -> show no new lock file, SQLite database or sidecar, directory, journal, integration -> record, catalog backup, catalog, cache, config, profile, history manifest, history -> row, or rollout line. A `foreign`, `unknown`, `external`, live-writer, or unknown- -> journal trace ends before the first `lock:*` event. +> Before steps 1-6 return an authorizing `AdmissionSnapshot`, there is no new lock +> namespace/database/sidecar, integration record, journal, catalog backup, catalog, +> cache, config, profile, history manifest/row, or rollout line. -WP11's lock database is outside both configurable homes, but it is still an artifact -and is forbidden before the answer is known. Passing a path that happens to be -writable is not an authority receipt. +### Prevention and detection are different claims -## External `model_provider` remains a distinct authority (C9) +For cooperating writers, stale commit is **prevented**: the config coordinator is +held through authoritative re-read and synchronous native commit. This is available +because `withConfigMutationLockSync` is synchronous (`src/config.ts:1767-1818`) and +`mutatePersistedConfig` already reruns against fresh snapshots +(`src/config.ts:1853-1913`). -Service-home ownership answers: “does another OpenCodex service installation claim -this canonical `CODEX_HOME`/`OPENCODEX_HOME` pair?” It says nothing about who owns -the contents of `config.toml`. The external-provider guard answers: “has the user -delegated effective Codex routing to a provider other than native `openai` or -`opencodex`?” A matching OpenCodex service can coexist with a newly selected -external provider; service ownership may be `owned` while config mutation authority -is absent. +For non-cooperating writers, portable conditional rename is unavailable +(`src/config.ts:1853-1859`). Bounded post-commit generation/target/current-byte +checks **detect** interference and return `deferred`; they do not retroactively +claim prevention. Regather/retry ends at `deadlineMs`, after which unresolved work +is named. This distinction is the correction required by audit #5 and +`005_contract.md` §3. -The previous design deleted this guard by substituting the service-home check. That -was wrong (`008_audit_synthesis_wp4_r2.md:31-35`). The external check stays after -service ownership and before journal inspection. It vetoes apply, restore, repair, -journal deletion, catalog/cache cleanup, history changes, and rollout changes. The -result is `external`, not “already converged”. +## External `model_provider` is a separate veto — C9 -### Actual diff — `src/codex/inject.ts:481-503,764-770` +Service-home ownership answers who claims this OpenCodex installation. It does not +answer who owns effective `config.toml` routing. A matching service can coexist +with a newly selected external provider; that provider blocks apply, restore, +journal recovery/deletion, catalog/cache/history/rollout mutation, provenance +adoption, and lock creation. -```diff - const activeProvider = externalCodexModelProvider(rawContent); - if (activeProvider) { -- // A launcher may have journaled before the provider manager took ownership. Never let shutdown -- // replay that stale snapshot over externally managed config. -- removeJournal(); - const nativeSubagentDefaultsWarning = configuredManagedSubagentDefaults(config) - ? `Native Codex sub-agent defaults were not injected: external model_provider ${tomlString(activeProvider)} owns config.toml.` - : undefined; -``` +Remove journal deletion from the external branch: ```diff - export function restoreNativeCodex(): { success: boolean; message: string } { - const activeProvider = currentExternalCodexModelProvider(); - if (activeProvider) { -- removeJournal(); -- return { success: true, message: `External Codex provider ${tomlString(activeProvider)} preserved; no native restore was needed.` }; -+ return { -+ success: false, -+ message: `Native Codex restore blocked: external model_provider ${tomlString(activeProvider)} owns config.toml; no Codex artifact was changed.`, -+ }; - } + const activeProvider = externalCodexModelProvider(rawContent); + if (activeProvider) { +- removeJournal(); + return externalAuthorityRefusal(activeProvider); + } ``` -Then make the writing body `restoreNativeCodexUnlocked(receipt)` internal to -`ownership-convergence.ts`; public callers receive the typed common convergence -result. `removeCodexConfig` may perform structural removal only with ledger entries -for the exact fragments and current transaction. Its current filename-only profile -unlink at `src/codex/inject.ts:723-742` is removed. - -## Provenance ledger and absence restoration (C10) +External is projected through the contract's `refused` authority/result. It is not +“already converged.” -### Location and exact record +## Contract-owned provenance record -The one owned operational record is -`getConfigDir()/integrations/codex.json`. It is outside `CODEX_HOME`; it composes -WP10 history convergence with WP12 provenance. Desired intent remains the fresh -file-backed `clientIntegrations.codex` value in the main config so there is one -intent authority. `lastAdmittedDesired` below is evidence, not a writable intent. +Delete the former `CodexIntegrationRecordV1`, transaction, artifact, ledger-row, +and restore unions. Import the section types: ```ts -export interface CodexIntegrationRecordV1 { - version: 1; - /** Diagnostic snapshot only; never used instead of readPersistedCodexIntent(). */ - lastAdmittedDesired?: "on" | "off"; - history: Record; - provenance: { - activeTransactionId: string | null; - transactions: Record; - }; -} - -export interface CodexArtifactTransaction { - id: string; - desired: "on" | "off"; - state: "prepared" | "committing" | "applied" | "restoring" | "restored" | "conflict"; - startedAt: string; - completedAt?: string; - artifacts: Record; -} - -export type CodexArtifactKind = - | "config" - | "profile" - | "catalog" - | "catalog-backup" - | "cache" - | "journal" - | "history-manifest" - | "history-row" - | "rollout"; - -export interface CodexArtifactLedgerRow { - kind: CodexArtifactKind; - canonicalPath: string; - baseline: - | { state: "absent" } - | { state: "present"; sha256: string; bytesBase64?: string; mode?: number }; - /** Written only after the candidate write succeeds and its bytes are read back. */ - postImage: { sha256: string; recordedAt: string } | null; - ownedStructure?: { - routedSlugs?: string[]; - configFragments?: string[]; - historyRows?: Array<{ - threadId: string; - modelProvider: string | null; - source: string | null; - rolloutPath: string | null; - }>; - rolloutProviders?: Array<{ - path: string; - firstLine: string | null; - latest: string | null; - }>; - }; - restore: - | { state: "pending" } - | { state: "restored-exact"; recordedAt: string } - | { state: "restored-structural"; recordedAt: string } - | { state: "preserved-drift"; recordedAt: string; currentSha256: string; message: string } - | { state: "blocked"; recordedAt: string; message: string }; -} +import type { + CodexProvenanceEntry, + CodexProvenanceLedger, +} from "./convergence-types"; +import { + readIntegrationRecord, + updateIntegrationRecord, +} from "./integration-record"; ``` -`bytesBase64` is required for byte-restorable present baselines (config, profile, -catalog, cache, and pre-existing backups). It is omitted for history DB/rollout -rows, which restore semantically from `ownedStructure`; copying SQLite or JSONL -bytes would overwrite concurrent native work. The record reader validates version, -transaction ids, canonical unique paths, SHA-256 width, base64/hash agreement, and -the single active transaction. A malformed record is `blocked/provenance-unknown`. - -### When rows are written - -1. After pre-lock admission passes and WP11 is acquired, re-read all baselines. -2. Before the first native artifact write, atomically persist one `prepared` - transaction containing a row for every artifact the commit can touch. This is - where baseline `absent` is recorded. -3. Set the transaction to `committing`; perform one candidate write. -4. After that write returns, read the resulting bytes, compute full SHA-256, and - atomically persist `postImage`. Only then may the row prove “created by us”. -5. Repeat steps 3-4 per artifact. Set `applied` only after observed state verifies - every required ON artifact. Partial/crashed work retains the active transaction. - -A filename, marker, slash-qualified slug, mtime, backup name, or file location is -never creation proof. Creation requires both `baseline.state === "absent"` and a -non-null successful `postImage.sha256`. A crash after a native write but before the -post-image ledger update leaves `postImage:null`; that is intentionally unknown and -cannot authorize automatic deletion. - -### Restoration rules - -- Baseline present + current hash equals post-image: restore exact baseline bytes, - then verify the baseline hash. -- Baseline absent + current hash equals post-image: unlink, then verify absence. -- Baseline absent + current hash differs: preservation wins. If the format is - parseable and `ownedStructure` identifies exact OpenCodex fragments/rows, remove - only those fragments and preserve native additions. Report operational - `absent` with historical `preserved-drift`; never report byte-exact restoration. -- Baseline absent + drift is unparseable/ambiguous: make no write and report a - conflict. Deleting would destroy user data; rewriting would invent a baseline. -- Missing ledger, null post-image, wrong transaction, or hash mismatch without an - exact structural owner: preserve and block. - -The hardest case is deliberate: config, catalog, or cache was absent; OpenCodex -created it; Codex or the user later added native data. OFF must not delete that -file. It removes only proven routed residue when possible and reports -`preserved-drift`; otherwise it preserves the entire file and reports `blocked`. -Historical absence cannot be restored without data loss, so the implementation -must say so. - -### Actual diff — `src/codex/journal.ts:97-107,148-162` +WP12 writes only `record.provenance` through `updateIntegrationRecord`; history, +generation, unknown top-level keys, and unknown section keys survive. Unparseable +or wrong-version record fails closed. No WP12 code joins +`getConfigDir()/integrations/codex.json`, validates the top-level schema, or runs a +parallel read/merge/write (`005_contract.md` §1). -```diff --function readJournal(): Journal | null { -- if (!existsSync(JOURNAL_PATH)) return null; -+export type JournalInspection = -+ | { state: "absent" } -+ | { state: "invalid"; reason: "corrupt" | "unknown-version"; message: string } -+ | { state: "valid"; journal: Journal; writer: "alive" | "dead" | "unknown"; postImageKnown: boolean }; -+ -+/** Inspect journal bytes and writer liveness without deleting or rewriting them. */ -+export function inspectJournal(): JournalInspection { -+ if (!existsSync(JOURNAL_PATH)) return { state: "absent" }; - try { -- const journal = JSON.parse(readFileSync(JOURNAL_PATH, "utf-8")) as Journal; -- if (journal.version !== 1) throw new Error("unknown version"); -- return journal; -- } catch { -- removeJournal(); -- return null; -+ const value = JSON.parse(readFileSync(JOURNAL_PATH, "utf-8")) as Partial; -+ if (value.version !== 1) return { state: "invalid", reason: "unknown-version", message: "unsupported journal version" }; -+ const journal = value as Journal; -+ let writer: "alive" | "dead" | "unknown"; -+ try { process.kill(journal.pid, 0); writer = "alive"; } -+ catch (error) { -+ const code = (error as NodeJS.ErrnoException).code; -+ writer = code === "ESRCH" ? "dead" : "unknown"; -+ } -+ return { -+ state: "valid", -+ journal, -+ writer, -+ postImageKnown: typeof journal.injectedConfigHash === "string" && journal.injectedProfileHash !== undefined, -+ }; -+ } catch (error) { -+ return { state: "invalid", reason: "corrupt", message: error instanceof Error ? error.message : String(error) }; - } - } +## When provenance entries are written + +1. After pre-lock admission and authoritative under-lock re-read, read every + artifact baseline. +2. Before the first native write, persist contract `CodexProvenanceEntry` rows for + every artifact this transition may touch, with this `txId` and one of the two + contract baselines. +3. Commit one artifact. +4. Read current bytes after the successful write and persist its `postImage` hash. +5. Repeat; then write the expected native generation/`txId` and observe. + +A filename, marker, slug, mtime, backup name, or location is not creation proof. A +crash after native write but before `postImage` leaves unknown provenance and cannot +authorize automatic deletion. + +## Two baseline classes, no third + +Consume exactly `005_contract.md` §9: + +- `absent` — no baseline artifact existed; +- `present` — the contract representation carries the exact baseline needed for + restoration plus its hash. + +There is no `present-required-nonempty`. The Pi `models.json {}` incident belongs +to `FOLLOWUP-FILECLIENT-01`; a Codex artifact phase has no file-client schema or +validator with which to implement that class. + +Restoration: + +- present + current bytes match our post-image -> restore exact contract baseline, + then verify baseline hash; +- absent + current bytes match our post-image -> unlink, then verify absence; +- current bytes differ -> preserve; remove only exact provenance-owned structure + when the format and ledger make that operation unambiguous, then report + operational absence with historical drift; +- unparseable/ambiguous drift, missing/null post-image, wrong transaction, or lost/ + corrupt ledger -> write nothing and refuse on provenance. + +### C10 is current-byte drift, not historical no-edit proof + +A SHA-256 comparison proves only that the bytes observed **now** equal the recorded +post-image. It cannot prove the artifact was never edited and reverted between +observations. C10 is therefore narrowed to current-byte drift detection and safe +restoration from current evidence. No test or documentation may claim detection of +an edit-and-revert ABA that leaves identical bytes. + +## Lost/corrupt ledger operator recovery — carried #10 + +Automatic convergence always refuses lost/corrupt provenance and preserves native +bytes. “Start a fresh record” is not recovery; it silently turns unknown artifacts +into owned artifacts. + +**INFERRED operator-recovery UX:** provide one explicit operator-only adoption flow +in the existing CLI, separate from normal convergence: + +```text +ocx restore --adopt-current-codex-baseline ``` +The flag is rejected in service/agent-driven/automatic contexts and requires an +interactive confirmation naming the canonical Codex home and that current bytes +will become the baseline. It performs read-only service/external/journal checks +first, requires the proxy stopped and no live journal writer, asks the +`integration-record.ts` owner to atomically move an unreadable record to a +timestamped sibling quarantine (preserving its bytes), then uses +`updateIntegrationRecord` against the now-absent canonical path to create a +valid record whose `present` baselines are the exact current bytes. A lost record +has no quarantine source but follows the same exact-current-baseline validation. It +changes no Codex native artifact. + +If even one target is unreadable/ambiguous, adoption aborts before replacing the +record. A subsequent explicit `convergeCodex` performs the requested apply/remove +from the adopted baseline. The command prints the quarantine path and resulting +`txId`; automatic callers receive only the provenance refusal and operator +instruction. Tests never auto-confirm this action. + +This is a recovery path, not a second convergence entry point: adoption establishes +authority evidence; all native mutation still goes through `convergeCodex`. + +## Journal inspection and recovery + +`src/codex/journal.ts` gains a read-only inspection result local to the journal +module. Corrupt/unknown-version bytes are preserved. PID `EPERM`/unknown is not +dead. A markerless version-1 journal may be structurally valid but lacks post-image +proof and blocks automatic replay. + ```diff -export function reconcileJournal(): boolean { - const journal = readJournal(); -- if (!journal) return false; -- try { -- process.kill(journal.pid, 0); -- return false; -- } catch (e: unknown) { -- if ((e as NodeJS.ErrnoException).code === "EPERM") { -- return false; -- } -- } -- const restored = restoreJournalState(); -+export function reconcileJournalUnlocked( -+ inspection: Extract, +- // read may delete malformed journal; dead PID may replay automatically +-} ++export function inspectJournal(): JournalInspection { ++ // Read/validate/liveness only; no delete, rename, repair, or replay. ++} ++ ++function reconcileJournalUnlocked( ++ inspection: AuthorizedDeadJournal, +): RestoreJournalResult { -+ if (inspection.writer !== "dead" || !inspection.postImageKnown) { -+ return { configRestored: false, profileRestored: false, configChanged: false, profileChanged: false, complete: false }; -+ } -+ const restored = restoreJournalState(inspection.journal); -- if (!restored.configRestored && !restored.profileRestored) return false; -- console.error(`⚠️ Previous session (PID ${journal.pid}) did not shut down cleanly. Codex state restored from journal.`); -- return true; -+ return restored; - } ++ // Called only by convergence inside the coordinated commit. ++} ``` -The final implementation returns `RestoreJournalResult` consistently; no log is -emitted under the lock. `restoreJournalState` accepts the inspected journal and no -longer calls a reader that could change the authority answer. Markerless version-1 journals are -valid but `postImageKnown:false`; provenance cannot prove current bytes, so -automatic recovery blocks instead of assuming unchanged. +No log is emitted while locks are held. -## Observed state and `unchanged` convergence (C11) +## Observed state consumes contract types — C11 -`inspectCodexObservedState` is read-only and returns: +Delete `CodexObservedState`, `CodexConvergenceResult`, and +`CodexSyncConvergenceResult` from this document. `inspectCodexObservedState` +returns the contract's `CodexObservedState`; `convergeCodex` returns the contract's +`ConvergeOutcome`. -```ts -export type CodexObservedState = - | { state: "applied"; historical: "exact"; artifacts: CodexArtifactObservation[] } - | { state: "absent"; historical: "exact" | "preserved-drift"; artifacts: CodexArtifactObservation[] } - | { state: "partial"; historical: "exact" | "preserved-drift" | "unknown"; artifacts: CodexArtifactObservation[] } - | { state: "external"; provider: string; artifacts: CodexArtifactObservation[] } - | { state: "blocked"; reasons: string[]; artifacts: CodexArtifactObservation[] }; - -export interface CodexConvergenceResult { - desired: "on" | "off" | "unknown"; - observed: CodexObservedState; - converged: boolean; - changed: boolean; - refusal?: "foreign" | "unknown" | "external" | "journal-active" | "provenance" | "lock-busy"; - message: string; -} - -export interface CodexSyncConvergenceResult extends CodexConvergenceResult { - ok: boolean; - retryable: boolean; - added: number; - catalogPath: string | null; - catalogExists: boolean; - catalogWritten: boolean; - cacheSynced: boolean; -} -``` +The observer reads service/external authority, managed config fragments, profile, +catalog/cache and routed slugs, journal/liveness, provenance/generation/tx identity, +history DB/manifest/rollouts, backups, and partial transaction residue. It performs +no repair. -The observer reads all of these before answering “is Codex currently applied?”: - -1. service ownership and external provider; -2. `config.toml` root `model_provider`, owned `openai_base_url`, active - `model_catalog_json`, embedded `[profiles.opencodex]`, routed root model, and - managed defaults; -3. generated profile existence, bytes/hash, and provenance; -4. active catalog parse state, provenance, and every transaction-recorded routed slug; -5. `models_cache.json` in wrapper or raw-catalog shape, provenance, and routed slugs; -6. journal validity, writer liveness, transaction identity, and post-image matches; -7. history DB rows tagged `opencodex`, backup-manifest entries, and each touched - rollout's first-line and latest provider observations; -8. catalog backup and transaction residue, especially artifacts whose baseline was absent. - -Desired ON converges only with observed `applied`; desired OFF converges only with -observed `absent`. `external`, `blocked`, and `partial` never converge. Operational -absence with `historical:"preserved-drift"` is converged for routing but is not an -exact historical restore, and the response must expose both facts. +Desired ON converges only when the contract observer says applied. Desired OFF +converges only when residue is removed/restored. External/refused/partial remains +non-converged. Current-byte structural preservation can be operationally removed +while still reporting historical drift; it cannot be described as byte-exact +restoration. `mutatePersistedConfig` already distinguishes `unchanged` from `committed` -(`src/config.ts:1837-1839,1877-1913`). `unchanged` says only that the boolean already -matched. It never skips observation or work: desired OFF may have a routed cache row -after a crash, and desired ON may be missing a profile/catalog after explicit restore. -Both paths run admission, converge, and re-observe. - -## Fresh admission in a long-lived server (C12) +(`src/config.ts:1837-1839,1877-1913`). `unchanged` intent never skips observation or +work: OFF may retain crash residue; ON may be missing a profile/catalog after an +explicit restore. -Add this pure reader beside `readConfigDiagnostics` at `src/config.ts:1714-1715`: - -```ts -/** Read Codex intent from persisted, schema-valid config; never use fallback defaults as authority. */ -export function readPersistedCodexIntent(): - | { state: "known"; desired: "on" | "off" } - | { state: "unknown"; reason: "missing" | "invalid" } { - const diagnostics = readConfigDiagnostics(); - if (diagnostics.source !== "file") { - return { state: "unknown", reason: diagnostics.source === "default" ? "missing" : "invalid" }; - } - return { state: "known", desired: diagnostics.config.clientIntegrations?.codex === false ? "off" : "on" }; -} -``` +## Fresh admission in a long-lived server — C12 -`src/types.ts:533-545` gains the one-key `OcxClientIntegrationsConfig`, and -`src/config.ts:916-940` gains a `.passthrough()` nested schema. Unknown future -integration keys survive a field-scoped mutation. This substrate defines the -reader and writer, but not a GUI/toggle route. +The old “one config read” cost claim is withdrawn. WP9 gather uses the exact config +object from the first `AdmissionSnapshot`, but `005_contract.md` §§3-4 requires an +authoritative re-read inside the coordinated commit. These are compatible duties, +not one read: -Every Codex-mutating request performs one config file open/read, JSON parse, and -schema validation before gather, then repeats that bounded read under the WP11 -lock. Cost is two O(config-file-bytes) local reads per infrequent mutation request, -zero resident watchers, and zero cross-process cache protocol. A watcher may later -reduce diagnostics latency, but it may never replace the two admission reads. +1. full persisted read before gather -> `AdmissionSnapshot A`; gather uses + `A.config`; +2. full authoritative persisted re-read under native->config coordination -> + `AdmissionSnapshot B`; compare B to A before commit; +3. cheap expected-transition checks around/after commit. -### Actual diff — `src/server/management/config-routes.ts:261-268` +No resident watcher or server-captured config is authority. The config reader +returns unknown for missing/unreadable/invalid persisted config; default fallback +ON is not sufficient to mutate. ```diff - if (url.pathname === "/api/sync" && req.method === "POST") { -- const { syncModelsToCodex } = await import("../../codex/sync"); -+ const { convergeCodexToPersistedIntent } = await import("../../codex/ownership-convergence"); - const { attachStaleAppServerHint } = await import("../../codex/app-server-processes"); -- const result = await syncModelsToCodex(undefined, config, null); -+ // The server-captured `config` at handleConfigRoutes line 77 is request-routing -+ // state, not mutation authority. This call rereads disk before gather and lock. -+ const result = await convergeCodexToPersistedIntent({ source: "api-sync", log: null }); - return jsonResponse({ - ...attachStaleAppServerHint(result), - ...(result.ok ? {} : { error: result.message }), -- }, result.ok ? 200 : 500); -+ }, result.ok ? 200 : result.retryable ? 409 : 503); - } ++function readCodexAdmissionSnapshot(): ++ | AdmissionSnapshot ++ | Extract { ++ const diagnostics = readConfigDiagnostics(); ++ if (diagnostics.source !== "file") return contractAdmissionRefusal(diagnostics.source); ++ return admittedSnapshotFromPersistedConfig(diagnostics); ++} ``` -The API result adapter retains the existing sync fields (`added`, `catalogPath`, -`catalogExists`, `catalogWritten`, `cacheSynced`) from WP9 and adds desired, -observed, converged, refusal, and retryable. It does not read `config` to gate Codex. - -### Actual diff — `src/cli/index.ts:169-177,318-321,358-369,398-412` +The helper is module-private and returns only contract shapes; it does not publish +an `AdmissionSnapshotResult` union. -```diff --import { reconcileJournal } from "../codex/journal"; -+import { convergeCodexToPersistedIntent } from "../codex/ownership-convergence"; -``` +`src/types.ts` adds the one-key client-integrations object; the config schema is +passthrough so unknown future integration keys survive a scoped mutation. -```diff - async function handleStart(options: { block?: boolean } = {}) { -@@ - const requestedPort = parsePortOption(); -- if (!currentExternalCodexModelProvider()) reconcileJournal(); - const existingPid = readPid(); -``` +## `/api/sync` calls the contract adapter only -```diff - await maybeShowStarPrompt(); // once-only Yes/No GitHub-star prompt on first interactive start -- await syncModelsToCodex(port).catch(() => {}); -+ const codex = await convergeCodexToPersistedIntent({ source: "startup", port, log: console }); -+ if (!codex.converged) console.error(`⚠️ ${codex.message}`); - if (!currentExternalCodexModelProvider() && !shouldInjectApiAuthHeader(config) && config.syncResumeHistory !== false) { -``` +Delete WP12's status logic and custom result adapter. Current route +`src/server/management/config-routes.ts:261-268` becomes: ```diff - async function handleEnsure() { -- if (!currentExternalCodexModelProvider()) reconcileJournal(); - const config = loadConfig(); -@@ - if (live) { -- await syncModelsToCodex(live.port).catch(e => { -- console.error(`⚠️ Model sync skipped: ${e instanceof Error ? e.message : String(e)}`); -- }); -+ const codex = await convergeCodexToPersistedIntent({ source: "ensure-live", port: live.port, log: console }); -+ if (!codex.converged) console.error(`⚠️ ${codex.message}`); -``` - -```diff -- await syncModelsToCodex(port).catch(e => { -- console.error(`⚠️ Model sync skipped: ${e instanceof Error ? e.message : String(e)}`); -- }); -+ const codex = await convergeCodexToPersistedIntent({ source: "ensure-parent", port, log: console }); -+ if (!codex.converged) console.error(`⚠️ ${codex.message}`); + if (url.pathname === "/api/sync" && req.method === "POST") { +- const result = await syncModelsToCodex(undefined, config, null); +- return jsonResponse(result, result.ok ? 200 : 500); ++ const outcome = await convergeCodex({ ++ action: "converge", ++ reason: "api-sync", ++ mode: "explicit", ++ deadlineMs: EXPLICIT_CODEX_CONVERGENCE_DEADLINE_MS, ++ }); ++ return toSyncResponse(outcome); + } ``` -The spawned child runs the startup path and the parent runs `ensure-parent`; both -admit independently from disk. The operation is idempotent under WP11, so the second -observer either proves convergence or performs residue repair. `src/server/index.ts` -removes import/current line 403 `invalidateCodexModelsCache()`; cache mutation occurs -only in the admitted commit. Thus the proxy can bind and serve other clients even -when Codex admission returns foreign, unknown, or external. +Status, body, and `Retry-After` belong only to +`src/server/management/sync-response.ts` (`005_contract.md` §5). -## Common orchestrator +## One common entry point -`src/codex/ownership-convergence.ts` exports exactly these entry points: +`src/codex/convergence.ts` exports the contract's `convergeCodex` and no +`convergeCodexToPersistedIntent`, `inspectCodexMutationAdmission` public receipt, +or WP12-specific request/result type. ```ts -export interface CodexAdmissionReceipt { - canonicalCodexHome: string; - canonicalOpenCodexHome: string; - authorityDigest: string; - desired: "on" | "off"; - journalTransactionId: string | null; - provenanceRevision: string; -} - -/** Steps 1-6 only. This function performs no write and opens no coordinator. */ -export function inspectCodexMutationAdmission(): - | { status: "admitted"; receipt: CodexAdmissionReceipt } - | { status: "blocked" | "external"; result: CodexConvergenceResult }; - -/** Run the fixed admission -> gather -> lock/recheck -> commit/observe sequence. */ -export async function convergeCodexToPersistedIntent(options: { - source: "startup" | "ensure-live" | "ensure-parent" | "api-sync" | "explicit" | "teardown"; - port?: number; - log?: Pick | null; -}): Promise; - -/** Read the complete artifact projection without repair. */ -export function inspectCodexObservedState(): CodexObservedState; +export async function convergeCodex( + request: ConvergeRequest, +): Promise; ``` -The under-lock recheck computes a new receipt and requires equality of canonical -homes, authority digest, desired intent, journal transaction, and provenance -revision. WP9 candidate revisions are checked separately before catalog commit. -No caller can supply `desired` or ownership as an option; test seams replace the -readers, not the verdict. +Callers say when/reason/mode/deadline. They never supply desired state, ownership, +journal verdict, provenance verdict, or apply/remove direction. `action:"observe"` +is the one read-only public operation; internal admission/observer helpers stay +module-private unless another contract phase explicitly owns them. + +Rewire remaining startup/ensure/restore/eject/stop/uninstall paths in the same WP12 +commit. Current direct sites include startup/ensure sync +(`src/cli/index.ts:319,367,409`), explicit restore/sync +(`src/cli/index.ts:528,591,756,768,829`), service restore +(`src/service.ts:2587,2625`), and server stop restore +(`src/server/management-api.ts:168-181`). A module-graph test proves none reaches +native writers except through `convergence.ts`. + +Remove unconditional `invalidateCodexModelsCache()` from +`src/server/index.ts:403`; cache mutation occurs only in admitted convergence. ## Test plan -All tests use fresh temporary `CODEX_HOME`, `OPENCODEX_HOME`, real-user-home lock -namespace overrides supplied by WP11's test seam, and port `0`. None invokes -`ocx start`, `ocx stop`, `ocx sync`, `ocx restore`, or `ocx ensure`, and none reaches -the live listener on 10100. +All tests use temporary homes, real contract record owner, port `0`, and production +`convergeCodex`. None invokes live CLI lifecycle commands or port 10100. ### Authority and ordering -1. `tests/codex-ownership-authority.test.ts` — no service/no mirrors is owned; - matching mirrors are owned; foreign canonical pair is foreign; installed plus - missing mirror, corrupt mirror, unreadable mirror, conflicting valid mirrors, - registration unknown, and unresolvable paths are unknown. -2. **Dead-PID markerless journal plus external provider, byte-exact preservation.** - Seed version-1 journal without injected hashes and dead PID, external - `model_provider`, config, profile, catalog, both backup forms, cache, history - manifest, SQLite DB, and rollout sentinels. Run startup admission and ensure - admission independently. Hash all bytes before/after; assert identical bytes, - journal present, no mtime change where supported, result `external`, and no lock, - SQLite, provenance, journal, or native artifact created. -3. Foreign-home run asserting **NO artifact was created**. Snapshot both homes and - the WP11 namespace; call startup, ensure, API sync, and teardown entries. Assert - the trace ends at `service:foreign`, directory trees and hashes are identical, - and lock DB/sidecars, integration record, journal, catalog backup, and cache are absent. -4. Invalid and unknown-version journals remain byte-exact and return blocked. - `EPERM`/liveness-unknown is not treated as dead. -5. Ordered trace table for every entry point: - `paths -> service -> external -> journal -> provenance -> intent -> gather -> - lock -> paths -> service -> external -> journal -> provenance -> intent -> - recover -> commit -> observe`. Desired OFF omits only `gather`; every refusal - ends before `lock`. - -### Provenance and restoration - -1. Apply into absent config/profile/catalog/backups/cache/journal/history-manifest; - assert each row first records `baseline:absent`, then a read-back post-image hash. -2. Restore with unchanged post-images; assert every transaction-created artifact - returns to absence and the transaction reaches `restored`. -3. **Cache absence restoration.** Begin without `models_cache.json`, apply routed - data, prove ledger absence + successful cache post-image, then desired OFF must - unlink it. A second OFF is a no-write success. This is different from the current - creation-only assertion at `tests/codex-models-cache-invalidate.test.ts:41-55`. -4. Baseline absent followed by native edits for config, catalog, and cache. Add - native content after apply. OFF preserves native additions, removes only exact - ledger-owned routing, and reports operational absent plus historical - `preserved-drift`. Unparseable drift is fully preserved and returns blocked. -5. Crash after artifact write but before post-image ledger write; restart sees - `postImage:null`, preserves the artifact, and reports provenance conflict. -6. Pre-existing same-named profile/catalog/cache/backup with no matching ledger is - never unlinked. Present-baseline exact restore requires current hash == post-image. -7. History/rollout restoration stays semantic: originals remain in the manifest - until DB rows and both first-line/latest rollout observations agree. +1. Table-drive owned/foreign/unknown service evidence including corrupt, + unreadable, conflicting, missing, unknown registration, and unresolvable paths. +2. External provider + dead markerless journal: byte-exact before/after across + config/profile/catalog/backups/cache/history/rollouts; journal remains; no lock + or record creation. +3. Foreign/unknown startup, ensure, API sync, teardown, and management mutation end + before first lock event and preserve full manifests. +4. Invalid/unknown-version journal and unknown liveness preserve bytes and refuse. +5. Trace exact order through `convergeCodex`; OFF omits gather only. + +### Bounded interference + +1. Gather from snapshot A; cooperating config writer changes generation before + lock. Under-lock snapshot B rejects before commit — prevention. +2. Inject a non-cooperating byte change after the final coordinated read but before + post-commit check. Outcome is deferred/preserved and a bounded retry is scheduled + — detection, not prevention. +3. Exhaust `deadlineMs`; assert typed unresolved outcome and no unbounded loop. +4. Native expected generation with another `txId` at the same number is + interference. +5. Stale history `CommitExpectation` is rejected before mutation. + +### Provenance/restoration/recovery + +1. Apply from absent and present baselines through `convergeCodex`; verify each + contract entry precedes native write and post-image follows read-back. +2. Matching current post-image restores exact present baseline or absence. +3. Current-byte drift preserves native additions; exact structural removal occurs + only with unambiguous provenance. Unparseable drift blocks. +4. Edit then revert to the same bytes: test only that current equality permits the + contract action; explicitly do **not** assert no edit occurred. +5. Crash between native write and post-image record; restart preserves/refuses. +6. Lost record and corrupt record: every automatic/normal explicit convergence + refuses and preserves bytes. +7. Operator adoption with no confirmation does nothing; confirmed isolated CLI + flow quarantines the bad record, writes exact current present baselines through + the owner, changes no native bytes, then normal `convergeCodex` succeeds. +8. Adoption aborts atomically on one unreadable target, external provider, live + writer, running proxy, or noninteractive/agent-driven invocation. ### Observed state and fresh intent -1. Table-drive applied, absent, each one-artifact partial, external, blocked, and - preserved-drift historical status. Include stale first-line rollout metadata and - a non-empty manifest with no matching DB row. -2. Persist desired OFF first, seed one residue artifact at a time, perform an - `unchanged` OFF write, and prove every case still converges. -3. Persist desired ON first, remove config/profile/catalog/cache one at a time, - perform an `unchanged` ON write, and prove reconstruction plus re-observation. -4. **Running server honors another process.** Construct the server with stale ON - in memory, persist OFF from a subprocess, call `/api/sync`, and assert no gather - or native write. Persist ON from the subprocess, call the same running server, - and assert gather/apply occurs without restart. Repeat with invalid persisted - config and assert unknown/no write. +1. Drive applied, absent, each one-artifact partial, external/refused, current-byte + drift, stale rollout, and nonempty manifest states through `action:"observe"`. +2. Persist OFF with one residue at a time; unchanged config mutation still converges. +3. Persist ON with one required artifact missing; unchanged mutation reconstructs + and re-observes. +4. One running server starts with stale ON in memory; a subprocess persists OFF; + `/api/sync` removes through fresh admission. Subprocess persists ON; same server + gathers/applies without restart. Invalid config refuses/no write. +5. Route assertions use `toSyncResponse`; no duplicated status table. -## Verification +### Production funnel -Fresh implementation gates: +Walk static/dynamic imports, aliases, wrappers, and re-exports. Every lifecycle and +management native writer must be reachable only through `convergence.ts`. Grepping +for `convergeCodex` alone is insufficient (`005_contract.md` §Test plan). + +## Verification ```bash bun run typecheck @@ -803,46 +503,42 @@ bun test tests/codex-ownership-authority.test.ts bun test tests/codex-artifact-provenance.test.ts tests/codex-models-cache-restore.test.ts bun test tests/codex-observed-state.test.ts tests/codex-convergence-order.test.ts bun test tests/codex-journal.test.ts tests/codex-sync-api.test.ts tests/service.test.ts tests/uninstall.test.ts +bun test tests/codex-convergence-contract.test.ts bun run test bun run lint:gui bun run privacy:scan ``` -Live proof is the real in-process server/subprocess case in -`tests/codex-sync-api.test.ts`, bound to port `0` with isolated homes. Its evidence -must show one server PID, two separate config-writer PIDs, OFF causing zero gather -and zero native writes, then ON causing the ordered gather/lock/commit path without -server restart. Artifact proof is the post-test tree/hash receipt from the external, -foreign, cache-absence, and preserved-drift fixtures. A green response envelope or -green suite without those read-backs is insufficient. Do not use the live proxy on -10100 for WP12 verification. +Live proof is the in-process server/subprocess test bound to port `0` with isolated +homes. Evidence names one server PID, two config-writer PIDs, prevention/deferred +interference traces, recovery quarantine/record hashes, and observed ON/OFF results. +A green response envelope without artifact read-back is insufficient. Never use the +live proxy on 10100. ## Accept criteria -- **C8 — authority before artifacts.** Foreign and unknown service ownership fail - closed. Tests prove no lock file/database/sidecar, directory, journal, provenance - record, or native artifact is created before paths, service ownership, external - provider, journal/liveness, provenance, and fresh intent all answer. -- **C9 — separate external authority.** A matching service does not override an - external effective `model_provider`; apply, restore, repair, journal cleanup, - catalog/cache/history/rollout writes all remain byte-exactly suppressed. -- **C10 — creation and preservation.** “Created by us” requires ledger baseline - absence plus successful read-back post-image hash. Matching post-images restore - absence. Later native edits are preserved and reported as conflict or - `preserved-drift`; no byte-exact claim is made where none is possible. -- **C11 — observed convergence.** Config, profile, catalog, cache, journal, history, - rollouts, backups, and provenance are inspected. `unchanged` intent still runs - convergence and post-observation; only ON/applied and OFF/absent are converged. -- **C12 — fresh server admission.** Every Codex mutation rereads file-backed intent - before gather and under lock. The subprocess test proves a CLI OFF and later ON - are honored by the same running server at a cost of two O(config-file-bytes) - local reads per mutation request. - -## Explicitly open after `004` - -The Decision leaves two future hardening items, neither of which may be silently -claimed here: journal liveness still identifies a writer by PID only, so PID reuse -can conservatively delay recovery until a later journal version records a process- -start/instance token; and terminal provenance transaction retention/compaction needs -a bounded policy after enough production evidence exists. Neither gap permits fail- -open mutation. Unknown liveness or provenance remains blocked, and preservation wins. +- **C8** — exact `AdmissionSnapshot` authority precedes every artifact; foreign and + unknown fail closed. Gather uses its config; authoritative re-read occurs inside + coordinated commit. +- **C9** — external provider remains a separate veto and preserves all bytes. +- **C10 (narrowed)** — two contract baseline classes only. Matching current + post-images restore; current-byte drift preserves/reports. No hash claims to prove + absence of edit-and-revert. Lost/corrupt ledger has an explicit, confirmed, + non-mutating adoption path; automatic behavior refuses. +- **C11** — observed state is the contract `CodexObservedState`; unchanged intent + still converges and re-observes. +- **C12** — the same running server honors subprocess OFF then ON using a pre-gather + snapshot and authoritative under-lock re-read; the old one-read cost claim is + withdrawn. +- `/api/sync` calls only `convergeCodex` + `toSyncResponse`; no WP12 status/header + owner exists. +- There is one convergence entry point and one shared result family. +- **N2** — WP12 rewires all remaining callers and passes its own typecheck/tests in + the same commit. WP13 adds composed proof, not missing implementation. + +## Explicitly open after WP12 + +Journal liveness still identifies a writer by PID only, so PID reuse may delay +recovery until a later journal version records a process-instance token. Terminal +provenance retention/compaction still needs a bounded policy after production +evidence. Neither gap permits fail-open mutation; preservation wins. From 47e7cac27723fa09dd7bb1bacac402b1e579b358 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Tue, 4 Aug 2026 11:24:30 +0900 Subject: [PATCH 033/163] docs(substrate): round 3 closes five, and names the one omission behind four more MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit First round that closed more than it opened. The ownership transfer worked: the reviewer re-walked their own thirty-entry checklist and most entries are structurally transferred. What remains is mostly one omission. CodexArtifactId, CodexObservedState and CatalogDisposition appear in the contract's signatures and are defined nowhere in the unit — I verified that myself. That cascades into four findings: the schema is incomplete, the adapter's never-check cannot be written, the AdmissionSnapshot lacks fields WP12 compares, and WP8b cannot typecheck. The reviewer fed my bodyless convergeCodex declaration to the real TypeScript compiler and got TS2391. Two design errors survived. The transition check compares nativeBefore after a commit that was supposed to change it, the record has no txId field so same-number-different-writer is undetectable by construction, and checking once after acquiring the history lock only moves the race. And the module graph cannot see what I asked it to: inject.ts and journal.ts are mixed read/write modules, so module-level reachability cannot tell a reader import from reaching a writer. Also accepts that WP9's rewiring does not preserve behavior — routing 16 catalog-only callbacks through a coordinator that injects config, profile and history means an ordinary provider edit starts doing all of that before the safety mechanics exist. That violates the invariant I wrote one round earlier. 050_composed_acceptance.md lands with this: 36 production entry points, 14 route shapes, 16 catalog-write call sites, and only C13 not provable through a production entry point. --- .../008_audit_synthesis_r3.md | 125 ++++++ .../050_composed_acceptance.md | 410 ++++++++++++++++++ 2 files changed, 535 insertions(+) create mode 100644 devlog/_plan/260804_codex_write_substrate/008_audit_synthesis_r3.md create mode 100644 devlog/_plan/260804_codex_write_substrate/050_composed_acceptance.md diff --git a/devlog/_plan/260804_codex_write_substrate/008_audit_synthesis_r3.md b/devlog/_plan/260804_codex_write_substrate/008_audit_synthesis_r3.md new file mode 100644 index 000000000..a9dba5d8e --- /dev/null +++ b/devlog/_plan/260804_codex_write_substrate/008_audit_synthesis_r3.md @@ -0,0 +1,125 @@ +# Substrate audit round 3 — synthesis + +Verdict: **FAIL**. Five closed (#9, #12, #13, N1, N4), eleven open, four new. + +## The trend is real, and so is the remaining gap + +| Round | Closed | Open | New | +|---|---|---|---| +| 1 | — | 13 | 13 | +| 2 | 1 | 11 | 5 | +| 3 | **5** | 11 | 4 | + +Round 3 is the first round that closed more than it opened. The *ownership +transfer* worked: the reviewer re-walked their own thirty-entry checklist and +confirmed most entries are structurally transferred. `/api/sync` has one owner, +the module names are normalized, the caller no longer picks direction, the retry +dormancy is gone, and the scope creep is reversed. + +What remains is narrower and mostly one thing. + +## The one thing: I referenced types I never defined + +`CodexArtifactId`, `CodexObservedState` and `CatalogDisposition` appear in the +contract's signatures and **are defined nowhere in the unit**. I verified this +myself — a grep for their definitions across all nine docs returns nothing. + +That single omission cascades into four separate findings: + +- **#3** the schema is not complete +- **#4** the adapter's exhaustive `never` check cannot be written +- **N2** WP8b cannot typecheck — the reviewer fed my bodyless + `export async function convergeCodex(...)` to the actual TypeScript compiler + and got **TS2391, "Function implementation is missing"** +- **checklist 8** the `AdmissionSnapshot` lacks fields WP12 says it compares + +A contract that references undefined types is not a contract. I wrote "the +complete definition of every shared surface" at the top of a document that was +not complete. + +## The design errors that survived + +**#1 / N3 — the expected transition is still wrong on both sides.** + +I wrote "reject when `nativeBefore` no longer matches", but after the native +commit the record holds `nativeAfter` — so the check compares against the value +that is *supposed* to have changed. Worse, the reviewer showed that even +corrected, checking once after acquiring the history lock only *moves* the race: +a newer transition can commit while the old Worker is still traversing files. + +And the record has **no `txId` field at all**, so "same number, different txId" +is undetectable by construction. + +Accept. Either a transition gate shared by native commits and the whole history +unit, or — the honest alternative — narrow the claim from *prevention* to +*detect-and-repair*, and guarantee the latest transition is durably scheduled +even when its Worker never spawned. + +**#2 — the module graph cannot see what I asked it to see.** + +I said writers move to `src/codex/internal/` with `convergence.ts` as sole +importer. But WP10's history worker must reach history writers directly, and +`inject.ts` / `journal.ts` are *mixed* read/write modules — a module-level graph +cannot tell a safe reader import from reaching a writer when both live in one +file. Accept: symbol-aware reachability, a published writer inventory, and +per-domain permitted roots (`convergence.ts` for native/catalog, +`history-worker.ts` for history). + +**#7 — uid/SID is the right key on the wrong root.** + +The key is settled. But `` calls an undefined resolver, and if a +service and a CLI derive different roots from `TMPDIR` / `XDG_RUNTIME_DIR` / +`LOCALAPPDATA`, adding the same uid underneath does not stop the split. The +environment problem I fixed at the leaf is still present at the root. + +**N-new 2 — WP9's rewiring does not preserve behavior.** + +The 16 management callbacks refresh catalog and cache only. Routing them through +a coordinator that also injects config, profile, journal and history means an +ordinary provider edit starts doing all of that — *before* WP10-WP12 land the +safety mechanics. That violates the invariant I wrote one round earlier: every +phase preserves behavior at its own commit. Accept: WP9's funnel is +catalog-only for management callers. + +**N-new 1 — adoption can enshrine routed state as "native".** + +If the ledger is lost while config still carries opencodex routing, adopting +current bytes as the baseline makes OFF *preserve* the routing forever. Accept: +adoption requires a verified native-clean state, or is split into salvage +(remove proven residue) then adopt. + +## The honesty corrections + +**#6 / #10 — `000_plan.md` still overclaims.** C17 is narrowed in the contract's +prose to cooperating transitions, and C10 is narrowed in WP12 to current-byte +drift, but the plan still promises unrestricted ABA detection and historical +edit detection. A criterion narrowed in one document and left broad in the +top-level plan is not narrowed. + +**N-new 4 — "no window" is false.** I wrote that the file writes and the counter +bump have "no window" between them, then described a crash in that window three +lines later. Process exclusion does not make separate filesystem replacements +atomic. The claim becomes "no cooperating interleaving while the process is +alive", with per-artifact crash recovery stated. + +## Disposition + +Fifteen open items, all accepted. I verified the missing type definitions and +the TS2391 form myself. + +## Next + +No structural replan this round — the structure is now right and the reviewer +says so. The work is finishing what the contract started: + +1. Define the three missing types, add `currentTxId` to the record, and complete + `AdmissionSnapshot` with the fields WP12 compares. +2. Make `convergeCodex` a type alias in WP8b, not a bodyless declaration. +3. Fix the transition check side, and choose prevention-with-a-gate or + detect-and-repair explicitly. +4. Specify the runtime-root algorithm per platform. +5. Scope WP9's management funnel to catalog-only. +6. Narrow C10 and C17 in `000_plan.md` to match the documents. +7. Write `050_composed_acceptance.md` (already dispatched in parallel). + +Then re-audit with the same reviewer. diff --git a/devlog/_plan/260804_codex_write_substrate/050_composed_acceptance.md b/devlog/_plan/260804_codex_write_substrate/050_composed_acceptance.md new file mode 100644 index 000000000..f8d7dbdb6 --- /dev/null +++ b/devlog/_plan/260804_codex_write_substrate/050_composed_acceptance.md @@ -0,0 +1,410 @@ +# WP13 — composed acceptance at the production boundary + +The failure this phase prevents is not a bad helper. It is a system in which each +helper passes its own tests while one production path still writes around the +substrate. That is already the current shape: startup restores a journal before the +server exists (`src/cli/index.ts:169-176`), server construction rewrites +`models_cache.json` directly (`src/server/index.ts:362-403`), sixteen management +call sites invoke a swallowed best-effort catalog writer +(`src/server/management-api.ts:105-112`), explicit restore writes history inline +(`src/codex/inject.ts:764-794`), and the retry guardian gives up after sixty ticks +(`src/codex/history-migration-guardian.ts:34-35,87-90`). A phase-local green test +can miss every one of those seams. + +This document specifies the one suite that is allowed to claim C1-C18 for the +composed system. All citations were rechecked on 2026-08-04 at +`ee182744af6958478523fb97ece6af2efb63b082`. The substrate modules named by +`005_contract.md` do not exist at that revision; the current red signatures below +are therefore predictions grounded in the production call graph, not claims that +an unimplemented suite was run. + +## IN / OUT + +IN: a future `tests/codex-composed-acceptance.test.ts` plus narrowly named child +fixtures under `tests/helpers/`; the production CLI, server, management routes, +service dispatcher, convergence entry point, real filesystem, real Bun Workers, +and real SQLite files. + +OUT: mocks of `convergeCodex`, direct calls to phase-local gather/commit/history +helpers as acceptance proof, the live proxy on port 10100, the user's homes, GUI +controls, six file integrations, release/deploy/publish work, arbitrary filesystem +ABA, and historical edit-and-revert detection. + +## The proof rule + +Every acceptance case has two required observations: + +1. **RED on the pre-substrate revision.** Run the same case against the parent of + the substrate implementation and capture the named failure below. A compile + failure caused only by a missing future import is not enough when the current + production path can be exercised; the red artifact must show the current wrong + byte, wrong ordering, blocked listener, bypass, or terminal retry state. +2. **GREEN on the composed revision.** Run through the production entry point and + read back native bytes, the sole integration record, lock/history state, HTTP + response, and child exits as applicable. A spy count or green helper test alone + is not acceptance evidence. + +The suite records the parent SHA, composed SHA, case id, entry-point id, child PIDs, +temporary roots, transition ids/generations, and the red/green oracle. This is how +we know a test is not decoration that passes on both revisions. + +## Production entry-point census — 36 rows + +The count is by independently invokable command/route or independently scheduled +production path. Aliases that execute the same branch are one row. The management +surface has **14 route shapes and 16 current catalog-write call sites** because +`PUT /api/provider-context-caps` has three mutation branches. + +| ID | Production entry point | Current write/reconciliation edge | +|---|---|---| +| P01 | `ocx init` / `ocx setup`, answer Yes to injection | dispatches `runInit` (`src/cli/index.ts:727-732`), which saves config and calls `injectCodexConfig` (`src/cli/init.ts:176-198`) | +| P02 | `ocx start` | dispatches `handleStart` (`src/cli/index.ts:734-736`) and calls `syncModelsToCodex` after bind (`src/cli/index.ts:318-321`) | +| P03 | foreground start graceful shutdown / process-exit cleanup | the start-installed cleanup calls `restoreNativeCodex` (`src/cli/index.ts:240-266`) and is registered for signals and exit (`src/cli/index.ts:284-310`) | +| P04 | `ocx ensure` | live and newly spawned branches call `syncModelsToCodex` (`src/cli/index.ts:358-381,398-412`) | +| P05 | `ocx sync` | dispatch calls `syncModelsToCodex` (`src/cli/index.ts:827-842`) | +| P06 | `ocx sync-cache` | directly calls `invalidateCodexModelsCache` (`src/cli/index.ts:849-855`) | +| P07 | `ocx restore` / `ocx eject` | dispatch calls `restoreNativeCodex` (`src/cli/index.ts:745-790`) | +| P08 | `ocx restore back` | the reverse branch calls `syncModelsToCodex` (`src/cli/index.ts:747-764`) | +| P09 | `ocx stop` | `handleStop` restores native state (`src/cli/index.ts:456-551`), dispatched at `src/cli/index.ts:737-743` | +| P10 | `ocx uninstall` / `ocx remove` | restores native state before deleting owned OpenCodex state (`src/cli/index.ts:554-638`), dispatched at `src/cli/index.ts:795-798` | +| P11 | `ocx recover-history --legacy-openai` | directly calls `restoreLegacyOpenaiHistory` (`src/cli/index.ts:711-724,792-794`) | +| P12 | `ocx provider add ... --sync` | live-proxy branch calls `syncModelsToCodex` (`src/cli/provider.ts:130-146,216-239`) | +| P13 | `ocx models add` | dispatches the custom add and live sync (`src/cli/models.ts:110-166,315-319`) | +| P14 | `ocx models remove` | dispatches the custom remove and live sync (`src/cli/models.ts:183-206,321-323`) | +| P15 | `ocx v2 mode ...` | persists mode then calls `syncModelsToCodex` (`src/cli/v2.ts:143-168`) | +| P16 | `ocx v2 on|off` | changed transition calls `syncModelsToCodex` (`src/cli/v2.ts:172-198`) | +| P17 | startup reconciliation path | journal replay is before bind (`src/cli/index.ts:169-176`), server construction directly invalidates cache (`src/server/index.ts:362-403`), and start arms the history guardian (`src/cli/index.ts:318-322`) | +| P18 | `POST /api/stop` | stops service, directly restores Codex, then drains (`src/server/management-api.ts:167-194`) | +| P19 | `POST /api/sync` | calls `syncModelsToCodex` with the server-captured config (`src/server/management/config-routes.ts:261-268`) | +| P20 | `POST /api/providers` | provider create reaches catalog write (`src/server/management/provider-routes.ts:99-147`) | +| P21 | `PATCH /api/providers?name=...` | provider edit reaches catalog write (`src/server/management/provider-routes.ts:151-338`) | +| P22 | `DELETE /api/providers?name=...` | provider delete reaches catalog write (`src/server/management/provider-routes.ts:449-487`) | +| P23 | `PUT /api/provider-context-caps` | all/global/per-provider branches reach three write calls (`src/server/management/provider-routes.ts:495-546`) | +| P24 | `PUT /api/disabled-models` | persists then refreshes (`src/server/management/model-routes.ts:208-215`) | +| P25 | `PUT /api/model-visibility` | persists then refreshes (`src/server/management/model-routes.ts:221-314`) | +| P26 | `POST /api/custom-models` | creates then refreshes (`src/server/management/model-routes.ts:321-353`) | +| P27 | `PUT /api/custom-models/:id` | updates then refreshes (`src/server/management/model-routes.ts:356-391`) | +| P28 | `DELETE /api/custom-models/:id` | deletes then refreshes (`src/server/management/model-routes.ts:394-405`) | +| P29 | `PUT /api/selected-models` | persists allowlist then refreshes (`src/server/management/model-routes.ts:426-441`) | +| P30 | `PUT /api/combos` | saves combo then refreshes before Claude follow-up (`src/server/management/combo-routes.ts:83-200`) | +| P31 | `DELETE /api/combos?id=...` | deletes combo then refreshes (`src/server/management/combo-routes.ts:203-217`) | +| P32 | `PUT /api/v2` | saves agent settings then refreshes (`src/server/management/agent-settings-routes.ts:178-280`) | +| P33 | `PUT /api/subagent-models` | saves roster, refreshes, then runs Claude/Desktop follow-up (`src/server/management/agent-settings-routes.ts:518-528`) | +| P34 | `ocx service start` | service dispatcher starts the installed wrapper (`src/service.ts:2511-2563`), whose baked command is `ocx start --port ...` (`src/service.ts:340,1378`) | +| P35 | `ocx service stop` | verifies stop, then directly restores native Codex (`src/service.ts:2564-2595`) | +| P36 | `ocx service uninstall` / `remove` | removes service, then directly restores native Codex (`src/service.ts:2610-2635`) | + +`ocx restart` and tray restart compose P09/P04 or P03/P02 +(`src/cli/index.ts:939-949,963-967`); service install eventually launches P02; they +do not own another Codex writer. CLI runtime model/combo commands that call the +management API are covered by the receiving P20-P33 route. If implementation finds +another production edge, this count changes and C14 remains red until the row and +runtime matrix are amended. + +## Harness: real isolated processes, never the user's state + +The parent creates one root with `mkdtempSync(join(tmpdir(), +"ocx-composed-"))`, then creates explicit `codex/`, `ocx/`, `home-a/`, `home-b/`, +`userprofile-a/`, `userprofile-b/`, `runtime/`, and local-provider fixture +directories beneath it. No path is derived from the parent process's `HOME`, +`USERPROFILE`, `CODEX_HOME`, or `OPENCODEX_HOME`. + +Every OpenCodex child is spawned as: + +```ts +Bun.spawn([ + process.execPath, + resolve(repoRoot, "src/cli/index.ts"), + ...argv, +], { + cwd: fixtureRoot, + env: { + ...minimalAllowlistedEnvironment, + HOME: fakeHome, + USERPROFILE: fakeUserProfile, + CODEX_HOME: codexHome, + OPENCODEX_HOME: ocxHome, + XDG_RUNTIME_DIR: sharedRuntimeRoot, + OPENCODEX_API_AUTH_TOKEN: fixtureToken, + }, + stdin: "pipe", + stdout: "pipe", + stderr: "pipe", +}); +``` + +This executes the production CLI module in a second Bun process. It does not import +a command handler into the test process and does not install or call a global +`ocx`. Interactive P01 input is sent through the child's stdin. Servers bind port +`0`; the harness reads the isolated runtime-port record and verifies `/healthz` +reports that same PID/port. Provider discovery points only at a local fixture +server. No external API call is permitted. + +The `saveConfigPreservingClaudeCode` warning is binding: a route fixture without an +injected persistence seam can overwrite the developer's real home +(`src/server/management/context.ts:9-19`). Therefore the composed suite never +constructs management routes around an in-memory config. It starts the real server +with the temporary `OPENCODEX_HOME` and sends authenticated HTTP to it. + +Setup helpers may seed production-shaped files and hold a real SQLite transaction; +they may not replace convergence, admission, lock, history, or writer functions. +Synchronization is by child IPC/stdout sentinels, HTTP completion, SQLite lock +ownership, and file/record observations—never `sleep` as readiness. Each child has +a hard watchdog, all Workers are joined, all spawned PIDs are proven exited, and +only then is the known temporary root removed. A teardown failure fails the case. + +## Runnable composed scenarios + +### A — every entry reaches one funnel + +Parameterize P01-P36. Seed an authorizing isolated installation, invoke the real +entry, and read the integration record transition id plus a recursive before/after +manifest. Every native mutation must have exactly one admitted transaction; OFF +entries must produce a removal transaction, not a skip. P20-P33 retain their +existing primary 2xx/201 behavior and expose the contract disposition; P30/P33 +still complete their Claude/Desktop follow-up. + +**RED today:** `convergence.ts` and the integration record do not exist; P06, P11, +P17, P18, P24-P33, P35, and P36 visibly reach direct writers. The management rows +also pass through the bare catch at `src/server/management-api.ts:105-112`, so no +typed transaction can be observed. **GREEN:** all 36 rows yield either one recorded +transition or a typed no-write refusal/busy outcome, and the module graph has no +writer reachable outside `convergence.ts`. + +### B — two-process race after approval, before commit + +Process A invokes P19 with desired ON and gathers from a local provider whose HTTP +response is held after request receipt. The fixture emits `GATHER_ENTERED`. Process +B uses a production config mutation route to persist B and then A again, completing +both cooperating generations while A remains inside gather. Release the provider +response; A reaches the native/config section with its old admission. Assert A is +rejected before the first catalog/cache/backup/config/profile write. Regather then +succeeds. + +**RED today:** P19 passes the server-captured config to `syncModelsToCodex` +(`src/server/management/config-routes.ts:261-264`), gather and write are one awaited +function (`src/codex/catalog/sync.ts:507-568`), and there is no generation. A writes +its stale catalog. **GREEN:** the A candidate records zero writes and a fresh +production call commits. + +### C — `/healthz` during real SQLite contention + +Seed a production-shaped `state_5.sqlite`, manifest, and eight rollout files. A +holder child opens that exact DB with `bun:sqlite`, executes `BEGIN IMMEDIATE`, and +prints `DB_LOCK_HELD` only after the transaction succeeds. While the holder waits +for parent IPC, invoke P19 so history work overlaps the lock. Before +releasing it, require ten authenticated-independent `/healthz` responses and an +eight-chunk local SSE request to complete. Each health request has a 500 ms +watchdog; the whole overlap has a 2 s watchdog. Then release the transaction, +observe durable `pending/db-busy`, and wait for the production retry to converge. + +**RED today:** history uses a 5 s SQLite busy timeout and synchronous retry sleep on +the caller thread (`src/codex/history-provider.ts:526-548,565-578`), so the 500 ms +health watchdog fires. **GREEN:** history waits in the Worker; all health responses +and chunks complete while the DB remains locked, and the later retry clears the +durable unresolved state. + +### D — foreign/unknown authority creates nothing + +Create foreign service-home evidence, then repeat with corrupt and unreadable +mirror evidence. Snapshot the whole temp root and the expected OS-runtime namespace, +invoke P02, P04, P19, one of P20-P33, P07, P18, P35, and P36, and compare byte/path +manifests. Assert no lock directory, SQLite DB/journal, integration record, native +journal, backup, catalog/cache, config/profile, history manifest/row, or rollout +line was created or changed. + +**RED today:** ownership preflight applies only to teardown and fails open on +non-mismatch errors (`src/integrations/native/ownership-preflight.ts:21-35`); +management catalog writes never call it. At least the management/startup rows +create or rewrite native artifacts. **GREEN:** every row refuses before the first +artifact and reports `foreign` or `unknown`. + +### E — one effective user, different environment homes, same lock + +Two real children use the same existing `CODEX_HOME`, OS account, and runtime root. +Child A gets `HOME=home-a`, `USERPROFILE=userprofile-a` and holds the production +lock. Child B gets `HOME=home-b`, `USERPROFILE=userprofile-b` and invokes P19 with a +zero deadline. Assert B receives typed busy with the same lock id/path. Repeat by +varying HOME alone and USERPROFILE alone. Assert one uid/SID namespace and no lock +artifact below either fake home or `CODEX_HOME`. + +**RED today:** there is no native cross-process lock or uid/SID namespace, so both +callbacks can write. **GREEN:** B is busy until A releases, then acquires the one +uid/SID-scoped database. + +### F — canonical-home matrix and lock taxonomy + +Invoke P19 from children using default, explicit, absolute, tilde, symlink, and +platform case-equivalent spellings of one existing home; they must contend on one +lock. Two different homes acquire independently. Missing home, namespace symlink, +wrong-owner/mode, malformed DB, and finite-deadline contention return typed +`refused` or `busy`, never throw; a normal run proves `acquired` through a +`converged` response. + +**RED today:** no such exclusion exists, so same-home contenders both mutate and +unsafe namespace fixtures are not classified. **GREEN:** the exact +`acquired | busy | refused` taxonomy is observable through P19 and no refused case +runs its commit. + +### G — config A→B→A and target retarget + +Use Scenario B's barrier, but require B to persist A→B→A through production config +mutations. A's content digest ends equal to its admitted digest. Assert A still +fails on generation before mutation. In a second run, retarget a parent symlink once +between gather and commit and assert target-identity refusal with zero writes. + +**RED today:** equality is the only available observation and there is no monotonic +generation/target expectation; the stale A bytes commit. **GREEN:** generation +detects the cooperating ABA and target identity detects single-direction retarget. +An arbitrary parent-symlink A→B→A wholly between checks is deliberately not claimed +(`005_contract.md` §3). + +### H — history overtaking is rejected before mutation + +Hold the production history lock with a helper child after transition A has +committed native ON but before A's Worker mutates history. Process B invokes P07 or +P19 with desired OFF and commits the newer native expectation. Release the history +lock. Place sentinels in the manifest, every rollout, and DB before release; assert +A returns `pending/overtaken` without changing any sentinel, then B alone produces +OFF history. Reverse ON/OFF and repeat. + +**RED today:** manifest and rollouts are outside SQLite's transaction +(`src/codex/history-provider.ts:606-648,656-695`) and there is no history lock or +expected transition, so scheduling order can overwrite the newer direction. +**GREEN:** the losing expectation is rejected before its first probe/write and the +highest native generation owns final history. + +### I — retry beyond the old horizon, without restart + +Seed the sole record with unresolved current history at `attempts: 60` and +`nextRetryAt` due now, then start one P02 server and keep it alive. Hold the real DB +through the first retry, observe the attempt advance beyond 60 and another finite +timer remain armed, then release. Wait no longer than one exported production +backoff cap plus a 2 s watchdog and assert the same PID converges; no restart/module +reload is allowed. + +**RED today:** `DEFAULT_MAX_TICKS = 60` and the terminal branch stops scheduling +(`src/codex/history-migration-guardian.ts:34-35,87-92`). **GREEN:** attempt 61+ +retains a timer and the same long-lived process converges after contention clears. + +### J — unchanged OFF with residue and unchanged ON with absence + +For OFF, persist desired OFF, leave one OpenCodex-owned artifact at a time +(config route, profile, catalog row, cache, journal, history manifest/row, rollout), +then invoke P19 without changing intent. Every residue must be removed/restored and +observed OFF. For ON, persist desired ON, delete one required artifact at a time, +invoke P19 without changing intent, and require reconstruction plus observed ON. +For C12, keep one P02 server alive across both halves: a second real CLI process +persists OFF, P19 removes through fresh admission, that process persists ON, and +the same server PID applies through fresh admission. An invalid persisted config +must refuse without changing any native byte. + +**RED today:** no persisted `clientIntegrations.codex` intent exists; P19 always +calls the apply-oriented `syncModelsToCodex`, while restore is a separate command. +OFF residue is reapplied and ON absence is not judged against a shared observed +state. **GREEN:** callers say only `converge`; admitted intent selects direction and +unchanged intent still repairs observed state. + +### K — current-byte provenance, recovery, and one schema + +Sequence P02 apply, P07 remove, P08 apply, P18 remove, and P02 startup using the same +isolated homes. After each transition, read the record only through its production +owner and assert history, provenance, generation, unknown top-level keys, and +unknown section keys survive. For an absent baseline, matching post-image removal +must restore absence. For a present baseline, restore exact bytes. Change current +bytes after apply and require preservation/conflict. Corrupt or remove the record +and require automatic refusal before mutation. + +**RED today:** there is no integration record; restore filters catalog rows and +deletes profile by filename (`src/codex/inject.ts:723-741`, +`src/codex/catalog/sync.ts:572-597`) without baseline/post-image authority. +**GREEN:** every phase reads one extension-safe schema and current-byte drift is +preserved/reported. Edit-then-revert to identical bytes is outside C10. + +### L — external provider remains an independent veto + +Create an owned service home but set an external root `model_provider`, add a dead +journal and native residue, then invoke P02, P04, P19, P20, P07, P18, P35, and P36. +Assert byte-exact preservation and no lock/record creation; the response names +`external-provider`, not service ownership or already-converged. + +**RED today:** direct management refresh and startup cache invalidation bypass the +external guard, while restore deletes the journal even when external +(`src/codex/inject.ts:764-769`). **GREEN:** the external-provider veto is checked +after service authority and before every artifact for every row. + +## C1-C18 matrix + +| Criterion | Production proof | What fails on the current revision | +|---|---|---| +| C1 | P19 and P20-P33 through A/B: provider response barrier proves gather is write-free; fixed commit receipts and typed outcomes prove the bounded commit | gather writes backup/catalog in one awaited function (`src/codex/catalog/sync.ts:507-568`); management errors are swallowed (`src/server/management-api.ts:105-112`) | +| C2 | P19 through B/G: cooperating config change after admission rejects before every native write | no generation/under-lock authoritative admission exists, so stale captured config commits | +| C3 | P19 through C while `/healthz` and SSE complete under held real SQLite write transaction | synchronous 5 s busy wait/retry runs on the listener thread | +| C4 | P02/P19 through C/I: unresolved is durable, attempt 61+ is armed, same PID later converges | current guardian terminates at 60 and has no durable typed record | +| C5 | P19 through F: converged proves acquired, zero-deadline contention proves busy, unsafe namespace proves refused | no native lock API/taxonomy exists; both contenders write | +| C6 | P19 through F across equivalent and distinct real homes | textual-path callers have no common cross-process lock | +| C7 | P19 through D/E/F: no home-derived namespace; wrong-owner/symlink namespace refuses | no per-user namespace exists; current paths are home/environment-derived elsewhere | +| C8 | P02/P04/P07/P18/P19/P20/P35/P36 through D, with full manifest including runtime lock path | current ownership check is teardown-only and fails open; management/startup bypass it | +| C9 | P02/P04/P07/P18/P19/P20/P35/P36 through L | management/startup write around the guard and external restore deletes the journal | +| C10 | P02/P07/P08/P18 through K | filename/filter restore has no baseline/post-image record and cannot restore proven absence safely | +| C11 | P19 through J for OFF-with-residue and ON-with-absence | no persisted Codex intent/observer; `/api/sync` always follows the old apply seam | +| C12 | one P02 server plus a subprocess config writer, then P19 OFF and ON through J | P19 passes the long-lived captured `config` object (`src/server/management/config-routes.ts:261-264`) | +| C13 | **Not provable through a production entry point.** Run typecheck, full suite, GUI lint, privacy scan, docs build, then require this composed suite's case manifest and red/green evidence | those static/broad gates can be green today while A-L are red; C13 alone proves none of C1-C12 | +| C14 | A drives all P01-P36; P20-P33 cover 14 route shapes/16 calls; module-graph reachability and transition receipts must agree | 16 management call sites and multiple CLI/startup paths reach direct writers instead of one funnel | +| C15 | P02/P07/P19 through H in both directions with manifest/rollout/DB sentinels | only DB substeps are transactional; processes can overtake file writes | +| C16 | P02/P07/P08/P18/P17 sequence through K, preserving both optional sections and unknown keys | no shared record owner/schema exists | +| C17 | P19 through G with production config A→B→A and one-way parent retarget | no generation or stable target expectation exists; equal content passes | +| C18 | P19 through E with independently different HOME and USERPROFILE in real children | no uid/SID lock exists, so environment-home variation does not contend | + +C13 is the only criterion not provable through a production entry point. It is a +meta-gate and must remain labelled that way; substituting its commands for A-L is +the exact false proof WP13 exists to prevent. + +## Determinism and runtime budget + +- No random timing decides a race. Provider-response gates, `BEGIN IMMEDIATE` + ownership, child READY messages, generation/record observations, and explicit IPC + releases define every ordering. +- Random ids are printed and compared, never predicted. The fixture clock may seed + persisted due timestamps, but the production scheduler and real monotonic + deadlines execute; no fake timer or mocked Worker satisfies C3/C4. +- The focused composed suite has a **120 s hard budget** on macOS/Linux and **180 s** + on Windows. All cases except C/I should finish within 10 s each; C is capped at + 2 s of held contention, and I may wait one production backoff cap. Exceeding the + budget is failure, not a retry-green allowance. +- Case order is irrelevant. Every case gets a fresh root and port 0. The suite runs + serially initially because it intentionally contends on per-user runtime + namespaces; parallelization is allowed only after distinct runtime roots and PIDs + are proven in the case manifest. +- Windows CI must exercise real SID, junction/reparse, ACL, and USERPROFILE behavior; + POSIX CI must exercise real uid, symlink, and mode/owner behavior. Platform skips + are limited to the opposite platform's primitive, never the shared criterion. + +## What this suite deliberately does not prove + +- It does not prove crash-atomic native commits. The contract promises detectable + partial state and later convergence, not a filesystem transaction. +- It does not prove arbitrary filesystem A→B→A wholly between observations, + bind-mount alias collapse, or cross-namespace identity. C17 covers cooperating + generations and single-direction target drift. +- It does not prove that matching current bytes were never edited and reverted. + C10 is current-byte evidence only. +- It does not prove GUI controls, Grok, Claude Code/Desktop, or the six file-client + integrations; those are outside this unit. +- It does not prove provider correctness or internet availability. Provider gather + is a deterministic local fixture. +- It does not prove release packaging, installation, deployment, or the live proxy + on 10100. Published-launcher smoke belongs to release verification after this + source-level production-boundary suite passes. +- It does not make C13 evidence for C1-C12. Green typecheck/full-suite/privacy output + remains necessary and insufficient. + +## Acceptance of WP13 + +WP13 passes only when A-L are red on the pre-substrate revision for the named +observable reason, green on the composed revision, all 36 production rows are in +the case manifest, C1-C12 and C14-C18 have artifact-level production evidence, C13 +is separately green, every child/Worker is joined, and the only removed paths are +the suite's explicit temporary roots. A missing row, a same-process substitute for +E/H, a mocked convergence function, a test that passes on both revisions, or a +green broad suite beside any red composed case is a failure. From 0f99666c85970b438832f74d417ebc6b980cadfe Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Tue, 4 Aug 2026 11:37:44 +0900 Subject: [PATCH 034/163] docs(substrate): define what the contract referenced, and stop overclaiming MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The three types the contract used in its own signatures — CodexArtifactId, CodexObservedState, CatalogDisposition — are now defined, along with the history reason union and nullable probe counts WP10 needs. The author reports all ten TypeScript blocks compile with zero diagnostics, which is the check that would have caught the TS2391 the reviewer found by compiling my bodyless convergeCodex. That declaration is now a ConvergeCodex type alias, with WP9 supplying the first catalog-scoped implementation. The record gains {nativeGeneration, currentTxId} with pairwise compare-and-set, so 'same number, different writer' is representable at all — it was structurally undetectable before. On the overtaking race I chose detect-and-repair over a shared gate, and said so: the latest history work is persisted before its Worker spawns, and a stale Worker cannot overwrite that schedule. Claiming prevention would have meant claiming a guarantee the sibling-lock arrangement does not provide. The lock root no longer reads any mutable environment variable — hardened per-uid directories on POSIX, token SID plus Known Folder on Windows, with no fallback. Fixing uid at the leaf while the root still came from TMPDIR would have left the split it was meant to close. WP9's management funnel is scoped to catalog only, because routing 16 catalog-and-cache callbacks through a coordinator that also writes config, profile and history would change what a provider edit does before the safety mechanics exist. And the 'no window' claim is explicitly reversed: process exclusion does not make separate filesystem replacements plus a record write atomic. C10 and C17 in the plan are narrowed to match what the documents actually promise. --- .../260804_codex_write_substrate/000_plan.md | 18 +- .../005_contract.md | 468 +++++++++++++++--- .../010_catalog_seam.md | 183 ++++--- 3 files changed, 532 insertions(+), 137 deletions(-) diff --git a/devlog/_plan/260804_codex_write_substrate/000_plan.md b/devlog/_plan/260804_codex_write_substrate/000_plan.md index d877ce8cb..fe31ad2d5 100644 --- a/devlog/_plan/260804_codex_write_substrate/000_plan.md +++ b/devlog/_plan/260804_codex_write_substrate/000_plan.md @@ -40,12 +40,12 @@ it rather than inventing their share. | Phase | Doc | Delivers | Depends on | |---|---|---|---| -| WP8b | `005_contract.md` *(to write)* | The shared surfaces: record schema + owner, `/api/sync` response contract, the single convergence entry point, generation counters, module names, and the config-snapshot admission result | — | +| WP8b | `005_contract.md` | The shared surfaces: record schema + owner, `/api/sync` response contract, the single convergence entry point, generation counters, module names, and the config-snapshot admission result | — | | WP9 | `010_catalog_seam.md` | gather/commit split + typed outcome, consuming the contract | WP8b | | WP10 | `020_history_isolation.md` | history off the event loop, and the cross-process history protocol | WP8b | | WP11 | `030_lock_protocol.md` | the async per-home lock, per-USER namespace | WP8b, WP9, WP10 | | WP12 | `040_ownership_convergence.md` | tri-state authority, admission order, absence restoration | WP11 | -| WP13 | `050_composed_acceptance.md` *(to write)* | one acceptance suite against real production entry points | all | +| WP13 | `050_composed_acceptance.md` | one acceptance suite against real production entry points | all | WP9 and WP10 remain independent of each other and both precede WP11: a lock around an unsplittable gather-and-write, or around a ten-second blocking history @@ -104,9 +104,10 @@ remain `FOLLOWUP-FILECLIENT-01` from the prior unit. answer is known. - C9 — the external-`model_provider` guard survives as an authority distinct from service-home ownership. -- C10 — an artifact that did not exist before apply is *removed* on convergence, - not merely filtered; and a baseline-absent artifact the user has since edited is - preserved with a reported conflict rather than deleted. +- C10 — an artifact that did not exist before apply is *removed* only when its + current bytes match the recorded post-image; current-byte drift is preserved with + a reported conflict rather than deleted. A hash proves only current equality. It + cannot prove that no edit-and-revert occurred between observations. - C11 — `unchanged` desired state still converges observed state. - C12 — a desired-state change made by another process is honored by the running server without a restart. @@ -121,8 +122,11 @@ remain `FOLLOWUP-FILECLIENT-01` from the prior unit. opposite directions, not by a same-process flight test (audit #1). - C16 — one owner and one schema for `integrations/codex.json`; a record written by any phase is readable by every other (audit #3). -- C17 — a config or catalog A→B→A cycle between gather and commit is DETECTED. - Content equality is not revision equality (audit #6). +- C17 — a cooperating config/native A→B→A transition between gather and commit is + detected by generation identity, and single-direction target-identity drift is + detected. This does not promise detection of an arbitrary non-cooperating + filesystem A→B→A entirely between checks: a hash or equal current bytes cannot + prove that no edit-and-revert occurred (audit #6). - C18 — two processes for the same OS user with different `HOME`/`USERPROFILE` take the SAME lock (audit #7). diff --git a/devlog/_plan/260804_codex_write_substrate/005_contract.md b/devlog/_plan/260804_codex_write_substrate/005_contract.md index 1de08e910..d25847b70 100644 --- a/devlog/_plan/260804_codex_write_substrate/005_contract.md +++ b/devlog/_plan/260804_codex_write_substrate/005_contract.md @@ -63,8 +63,10 @@ export interface CodexIntegrationRecord { version: 1; history?: CodexHistoryState; provenance?: CodexProvenanceLedger; - /** Bumped by every native commit. See §3. */ - generation?: number; + /** Bumped by every cooperating native commit. See §3. */ + nativeGeneration?: number; + /** The transaction that owns `nativeGeneration`; null is legal only at zero. */ + currentTxId?: string | null; } ``` @@ -76,21 +78,61 @@ without defining them, so `020` and `040` kept their own. Both live in ```ts export interface CodexHistoryState { - status: "converged" | "pending" | "running" | "blocked" | "unknown"; - /** Why it is not converged, when it is not. */ - reason?: "db-busy" | "permission" | "worker-died" | "overtaken"; + status: "converged" | "pending" | "running" | "blocked" | "unknown" | "not-evaluated"; + /** + * Why it is not converged, when it is not. These are terminal observations + * for one attempt, not reasons to collapse the durable retry schedule. + */ + reason?: + | "db-busy" + | "permission" + | "unreadable" + | "schema" + | "timeout" + | "shutdown-cancelled" + | "worker-died" + | "overtaken" + | "record-write-failed"; attempts: number; /** null means "no timer armed"; see 020 — it must never mean "never again". */ nextRetryAt: string | null; /** The transition this state belongs to, so an overtaken job is detectable. */ txId: string | null; + /** null means the final probe could not produce a trustworthy row count. */ + pendingRows: number | null; + /** null means the final probe could not produce a trustworthy manifest count. */ + backupEntries: number | null; /** Unknown keys from a newer writer, preserved verbatim. */ readonly [extra: string]: unknown; } +/** + * Every mutable Codex artifact for which the provenance ledger can authorize a + * restore. Embedded config fragments share the `config` entry because they are + * committed and restored as one file. Dynamic history ids name the exact row or + * rollout whose semantic pre-image is retained. + */ +export type CodexArtifactId = + | { readonly kind: "config" } + | { readonly kind: "generated-profile" } + | { readonly kind: "active-catalog"; readonly canonicalPath: string } + | { readonly kind: "catalog-backup"; readonly form: "hashed" | "legacy"; + readonly canonicalPath: string } + | { readonly kind: "models-cache" } + | { readonly kind: "injection-journal" } + | { readonly kind: "history-row"; readonly stateDbId: string; readonly threadId: string } + | { readonly kind: "history-manifest"; readonly stateDbId: string; + readonly canonicalPath: string } + | { readonly kind: "history-manifest-entry"; readonly stateDbId: string; + readonly threadId: string } + | { readonly kind: "history-rollout"; readonly stateDbId: string; + readonly canonicalPath: string }; + export interface CodexProvenanceEntry { artifact: CodexArtifactId; - baseline: { kind: "absent" } | { kind: "present"; sha256: string }; + baseline: + | { kind: "absent" } + | { kind: "present"; sha256: string; bytesBase64: string }; /** Hash of what WE wrote. null when the write did not complete. */ postImage: string | null; txId: string; @@ -101,12 +143,137 @@ export interface CodexProvenanceLedger { entries: readonly CodexProvenanceEntry[]; readonly [extra: string]: unknown; } + +export type CodexArtifactObservation = + | "applied" + | "absent" + | "missing" + | "residue" + | "drifted" + | "unreadable" + | "invalid" + | "not-evaluated" + | "unknown"; + +/** + * Read-only proof of what Codex has now, not what persisted intent requests. + * `isApplied` is true only for aggregate `applied`; a partial surface can never + * be flattened into true. OFF is operationally converged only at `absent`. + */ +export interface CodexObservedState { + aggregate: "applied" | "absent" | "partial" | "external" | "blocked" | "not-evaluated"; + /** null only for a catalog-scoped request that deliberately did not observe. */ + isApplied: boolean | null; + desired: "on" | "off" | "unknown"; + /** null only when aggregate is `not-evaluated`. */ + converged: boolean | null; + authority: { + service: "owned" | "foreign" | "unknown"; + externalProvider: string | null; + }; + surfaces: { + config: CodexArtifactObservation; + profile: CodexArtifactObservation; + catalog: CodexArtifactObservation; + cache: CodexArtifactObservation; + journal: "absent" | "pending" | "live" | "invalid" | "unknown" | "not-evaluated"; + history: { + state: CodexHistoryState; + database: CodexArtifactObservation; + manifest: CodexArtifactObservation; + rollouts: CodexArtifactObservation; + }; + provenance: { + state: "verified" | "missing" | "conflict" | "unreadable" | "unknown" | "not-evaluated"; + nativeGeneration: number | null; + currentTxId: string | null; + }; + }; +} + +export type CatalogNotice = "provider-auth" | "provider-network" | "fallback"; + +/** Sanitized catalog fact safe to append to management mutation responses. */ +export type CatalogDisposition = + | { status: "committed"; changed: boolean; degraded: boolean; + notices: readonly CatalogNotice[] } + | { status: "skipped"; + reason: "not-requested" | "catalog-unavailable" | "busy" | "stale" | "refused"; + retryable: boolean } + | { status: "failed"; reason: "provider-auth" | "provider-network" | "disk"; + phase: "gather" | "commit"; retryable: boolean; partialWrite: boolean }; ``` -`updateIntegrationRecord(mutate)` does one read-modify-write under the same -coordinator the config uses, and **preserves unknown keys verbatim at every -level** — top-level and inside each section — so a newer version's record -survives an older binary. +`CatalogDisposition` contains no provider name, URL, token text, path, digest or +raw exception. `CodexObservedState.aggregate` follows the five-state projection +already derived from the real config/profile/catalog/cache/journal/history surfaces +(`devlog/_plan/260804_codex_write_substrate/004_ownership_and_convergence.md:235-273`); +its nested observations keep the +one-artifact partial cases testable instead of hiding them behind that aggregate. +For a clean history convergence, `reason` is absent and both probe counts are zero. +An unreadable DB/manifest uses `unreadable`; a readable but unsupported table or +manifest shape uses `schema`; a watchdog uses `timeout`; graceful drain uses +`shutdown-cancelled`; and failure of the terminal CAS uses +`record-write-failed` in the returned observation while leaving the previously +persisted `pending` schedule intact. Any failed/unavailable final probe stores null, +never a zero-looking count. +`not-evaluated` is an ephemeral projection used only by WP9's catalog-scoped +compatibility outcome; it is never persisted as durable history and never answers +`isApplied` or `converged` with a false-looking boolean. + +### Durable read/update and initialization + +```ts +export interface IntegrationRecordVersion { + readonly nativeGeneration: number; + readonly currentTxId: string | null; +} + +export type IntegrationRecordRead = + | { kind: "missing"; record: null; version: { nativeGeneration: 0; currentTxId: null } } + | { kind: "ready"; record: CodexIntegrationRecord; version: IntegrationRecordVersion } + | { kind: "legacy-ambiguous"; record: CodexIntegrationRecord } + | { kind: "invalid"; message: string }; + +export type ReadIntegrationRecord = () => IntegrationRecordRead; + +export type IntegrationRecordUpdate = + | { kind: "updated"; record: CodexIntegrationRecord; version: IntegrationRecordVersion } + | { kind: "conflict"; current: IntegrationRecordVersion } + | { kind: "invalid"; message: string }; + +/** + * Compare `expected` against both native fields, apply `mutate`, and atomically + * replace the record while the caller's coordinator is held. A mismatch writes + * nothing. The updater preserves unknown keys at every object level. + */ +export type UpdateIntegrationRecord = ( + expected: IntegrationRecordVersion, + mutate: (record: CodexIntegrationRecord) => CodexIntegrationRecord, +) => IntegrationRecordUpdate; +``` + +WP8b implements and exports `const readIntegrationRecord: ReadIntegrationRecord` +and `const updateIntegrationRecord: UpdateIntegrationRecord` from +`src/codex/integration-record.ts`; these are executable functions in that phase, +not ambient declarations. + +A missing file normalizes to `{ nativeGeneration: 0, currentTxId: null }`; the +first successful update creates `version:1` and persists both native fields even +when it writes only `history` or `provenance`. A v1 record with neither native field +has the same initial meaning, which keeps the history-only/provenance-only landing +order valid. A positive generation without a nonblank `currentTxId`, a txId without +its generation, `generation` from the abandoned draft schema, or `null` paired with +a nonzero generation is `legacy-ambiguous`: automatic mutation fails closed and an +explicit observation/recovery must establish a current pair. It is never silently +coerced to the initial state. + +`updateIntegrationRecord` does one read-modify-write under the caller's coordinator. +Native transition N uses expected `{N,currentTxId}` and writes `{N+1,newTxId}`; +history/provenance completion uses the exact current pair it started from. Thus a +late Worker receives `conflict` rather than overwriting a competing txId at the same +number. Unknown keys survive top-level and section updates so a newer writer's +record survives an older binary. Unreadable or unparseable is not "empty": it fails closed and the caller reports rather than silently starting a fresh record. Losing provenance silently is how @@ -129,7 +296,9 @@ entire error handling is `catch { /* catalog absent */ }`. * place, so a new caller cannot forget them. Round 1's 16 callers each held * their own path to a commit. */ -export async function convergeCodex(request: ConvergeRequest): Promise; +export type ConvergeCodex = ( + request: ConvergeRequest, +) => Promise; export interface ConvergeRequest { /** @@ -143,6 +312,11 @@ export interface ConvergeRequest { * `observe` writes nothing and is the status read. */ action: "converge" | "observe"; + /** + * WP9 management mutations use `catalog`; explicit/lifecycle convergence uses + * `full`. Scope limits work, but still never lets the caller choose direction. + */ + scope: "catalog" | "full"; /** Why, for the record and for log attribution. */ reason: "startup" | "ensure" | "api-sync" | "cli" | "management-mutation"; /** Automatic callers fail fast and defer; explicit ones may wait. See §5. */ @@ -156,8 +330,12 @@ expected condition: ```ts export type ConvergeOutcome = + | { kind: "catalog-only"; changed: boolean; + observed: CodexObservedState; catalogRefresh: CatalogDisposition; + history: CodexHistoryState } | { kind: "converged"; direction: "applied" | "removed"; changed: boolean; - observed: CodexObservedState; generation: number; + observed: CodexObservedState; nativeGeneration: number; + currentTxId: string; catalogRefresh: CatalogDisposition; history: CodexHistoryState } | { kind: "skipped"; reason: "already-converged"; observed: CodexObservedState; catalogRefresh: CatalogDisposition; history: CodexHistoryState } @@ -166,6 +344,7 @@ export type ConvergeOutcome = | { kind: "busy"; surface: "lock" | "history" | "config"; retryAfterMs: number } | { kind: "deferred"; direction: "applied" | "removed"; changed: boolean; unresolved: readonly UnresolvedSurface[]; + nativeGeneration: number; currentTxId: string; observed: CodexObservedState; catalogRefresh: CatalogDisposition; history: CodexHistoryState } | { kind: "failed"; surface: string; message: string }; @@ -174,9 +353,36 @@ export type ConvergeOutcome = * `converged` with `direction: "removed"`. That is round 2 N1: the old shape let * a sync while OFF return "skipped" and leave routed residue on disk. */ -export type UnresolvedSurface = "history"; +export type UnresolvedSurface = + | "config" + | "native" + | "catalog" + | "cache" + | "journal" + | "provenance" + | "history"; ``` +This is deliberately a type alias, not the bodyless function declaration that +produced TS2391 in audit round 3. WP8b exports the type and lands no runtime +placeholder. WP9 supplies the first `convergeCodex` implementation and assigns it +to `ConvergeCodex` in the same commit that rewires catalog callers. + +WP9's `scope:"catalog"` implementation is **catalog-only**: it gathers, +commits, and reports catalog/cache/backup disposition while preserving each route's +primary 2xx/201. It does not inject config/profile, recover journals, or dispatch +history before WP10-WP12 land those mechanisms. WP12 strengthens that same funnel +to full observed-state convergence; there is no second entry point. +Its `catalog-only` outcome sets non-catalog observations to `not-evaluated` and uses +an ephemeral history value with `status:"not-evaluated"`, zero attempts, null txId, +timer, and probe counts. It does not claim either full direction. + +An unresolved surface also names its scheduler. `config` schedules a fresh +pre-gather admission; `native`, `catalog`, `cache`, `journal`, and `provenance` +schedule a full convergence for the record's current transaction; `history` +schedules the history guardian for that transaction. A scheduling write itself is +part of the durable record update, not a best-effort callback after the response. + **Best-effort callers stay best-effort.** The 16 management callbacks keep their 2xx and report the outcome in a `catalogRefresh` field; they do not start failing loudly because a catalog refresh deferred. What changes is that the @@ -230,11 +436,21 @@ The rule, stated so a test can check it: > raced us to the same number. Anything else is interference: the outcome is > `deferred` with the surface named, never `converged`. -The bump is written **inside** the same synchronous section as the commit, by -the committer, so there is no window where the files moved and the counter did -not. On crash between file write and counter bump, the next convergence sees a -stale counter and re-converges — which is safe because convergence is idempotent -by construction. +The earlier “there is no window” claim was wrong. Process exclusion cannot make +separate file replacements and the integration-record replacement atomic. Holding +native + config coordination provides **no cooperating interleaving while the +process is alive**; a crash can still leave any prefix of the artifact sequence with +the old record pair. + +Recovery is therefore artifact-specific. Config, generated profile, catalog, +hashed/legacy backups, cache, and journal recover only from their ledger baseline +plus matching post-image; a missing/null post-image preserves and refuses. History +rows, manifest entries, and rollouts remain `pending` and are re-probed/repaired by +the history guardian. A missing record with native residue or an invalid/ambiguous +record refuses automatic deletion. On restart, observation compares every artifact +to the ledger/current pair, records the unresolved surfaces, and schedules a fresh +current transition; idempotence is required but is not described as filesystem +atomicity. ### Prevention for cooperating writers (round 2 #5) @@ -272,21 +488,63 @@ Audit #8: `040`'s intent reader returns ON/OFF while `010`'s gather needs a full "two reads" is wrong. ```ts +/** The minimal, working WP8b/WP9 snapshot; it authorizes catalog work only. */ +export interface CatalogAdmissionSnapshot { + config: Readonly; + generation: number; + targets: Readonly<{ + catalog: string; + cache: string; + catalogBackups: readonly string[]; + }>; +} + export interface AdmissionSnapshot { config: Readonly; configDigest: string; intent: "on" | "off"; generation: number; ownership: "owned" | "foreign" | "unknown"; + externalProvider: string | null; + canonicalTargets: Readonly<{ + codexHome: string; + opencodexHome: string; + config: string; + profile: string; + catalog: string; + cache: string; + journal: string; + integrationRecord: string; + catalogBackups: readonly string[]; + historyDb: string; + historyManifest: string; + historyRollouts: readonly string[]; + }>; + journalIdentity: string; + provenanceIdentity: string; + /** Digest of every authority field above; passed to the history Worker. */ + authoritySnapshotId: string; } ``` -One read produces all of it. Gather consumes `config` — **that exact object**, -never a re-read and never the server's long-lived one. The under-lock recheck -compares `configDigest` and `generation` rather than reading the config again. - -Read count per mutation: **one** before gather, **two** cheap counter reads -around the commit. `010`'s independent `readConfigDiagnostics()` call is removed. +The earlier one-read claim is withdrawn. There are three authoritative observation +points, each with a different job: + +1. **Pre-gather:** fully read persisted config and all authority/target fields into + snapshot A. Gather consumes `A.config` — that exact object, never the server's + long-lived one. +2. **Under-lock:** while native + config coordination is held, fully re-read snapshot + B and compare digest, config generation, intent, ownership, external provider, + canonical targets, journal identity, and provenance identity. A mismatch rejects + before the first native write. +3. **Post-commit:** re-read persisted config and observe every native/catalog/history + surface into `CodexObservedState`. The outcome is not `converged` unless this + observation agrees with admitted intent and the exact expected native pair. + +The config reader at all three points is the persisted diagnostic reader. A missing, +unreadable, or invalid file produces unknown/refusal; it never falls back to the +server's captured object. `010`'s independent gather-time +`readConfigDiagnostics()` remains removed because snapshot A already owns that read. ## 5. `/api/sync`, defined once @@ -295,17 +553,18 @@ and both payload fields. | `ConvergeOutcome` | Status | Body | |---|---|---| +| `catalog-only` | 200 | `{ ok: true, changed, observed, catalogRefresh, history }` | | `converged` | 200 | `{ ok: true, changed, observed, catalogRefresh, history }` | | `skipped` (`already-converged`) | 200 | `{ ok: true, changed: false, observed, catalogRefresh, history }` | | `refused` | 409 | `{ ok: false, authority, message, observed }` | | `busy` | 503 + `Retry-After` | `{ ok: false, surface, retryAfterMs }` | -| `deferred` | 200 | `{ ok: true, changed, unresolved, observed }` | +| `deferred` | 200 | `{ ok: true, changed, unresolved, observed, catalogRefresh, history }` | | `failed` | 500 | `{ error: message, surface }` | `busy` is 503 with `Retry-After` because it is transient and the client should retry; `refused` is 409 because retrying changes nothing until a human acts. -`deferred` is 200 because the requested work DID happen — history is outstanding -and named, not failed. +`deferred` is 200 because the admitted bounded work DID happen — one or more +durably scheduled surfaces are outstanding and named, not collapsed into success. There is no `desired-off` row, per §2: a converge while OFF removes and returns `converged { direction: "removed" }`. @@ -316,14 +575,15 @@ each mapped this route themselves (round 1 #4, still open in round 2); none of them may now. The exhaustiveness is enforced by a `never` check on the union, so adding an outcome variant without a row fails typecheck. -## 6. History: one lock, and no overtaking +## 6. History: one lock, with overtaking detected and repaired Round 1 #1 had no home; round 2 showed my first answer had two holes. The real apply path writes manifest → rollouts → DB -(`history-provider.ts:606,611,626`); restore writes rollouts → DB → manifest -deletion → a second ejection (`:657,667,677,691`). SQLite guards only one of -those steps, so two processes corrupt each other through the files it never sees. +(`src/codex/history-provider.ts:606,611,626`); restore writes rollouts → DB → +manifest deletion → a second ejection +(`src/codex/history-provider.ts:657,667,677,691`). SQLite guards only one of those +steps, so two processes corrupt each other through the files it never sees. **One cross-process history lock**, acquired inside the Worker, held across the entire unit — manifest, rollouts and the DB transaction together, including the @@ -332,27 +592,47 @@ native section must stay synchronous. Two things round 2 caught: -**Explicit CLI history still ran inline** (`020:216-219,868-870`), outside any -lock. Every history caller takes this lock — server, CLI, startup, retry. A lock -one caller can skip is not a lock. +**Explicit CLI history still runs inline** through direct sync/restore calls +(`src/cli/index.ts:528,591,756,768,829`), outside any future history lock. Every +history caller takes this lock — server, CLI, startup, retry. A lock one caller can +skip is not a lock. **Sibling locks permit overtaking.** A releases the native lock after committing -ON; B commits native OFF; B's history removal can then run before A's history -apply, leaving native OFF with history ON. So the history job carries the -`CommitExpectation` from §3: - -> A history job whose `nativeBefore` no longer matches the record has been -> overtaken. It is **rejected before any mutation**, and the winning transition -> converges history itself. Overtaking is detected at the point of work, not -> raced at the point of scheduling. - -That also bounds the livelock the reviewer raised: a rejected job does not retry -into the same race, it defers to the newer transition. +ON; B commits native OFF while A traverses history. Checking once after A acquires +the history lock only moves the race: B can still advance the native pair before A +finishes. The previous check against `nativeBefore` was also the wrong side. After +A's native commit, the record is expected to contain A's +`{ nativeAfter, txId }`, not `nativeBefore`. + +This contract chooses **detect-and-repair**, not a transition gate shared across +the complete history unit. The guarantee is eventual convergence to the latest +durable native transition: + +1. The native record CAS that writes `{nativeAfter, txId}` also writes + `history:{status:"pending", txId, nextRetryAt:, ...}` for that same + transition **before** any Worker spawn. If spawn never occurs or the Worker dies, + the guardian/startup reader still has durable work to schedule. +2. A Worker checks that the record contains its `{nativeAfter, txId}` immediately + after acquiring the history lock. A mismatch returns `pending/overtaken` without + mutation and schedules observation of the current pair. +3. Because a newer native transition can commit during traversal, the Worker uses + `updateIntegrationRecord({nativeGeneration:nativeAfter,currentTxId:txId}, ...)` + for its terminal state. A CAS conflict means its result is stale; it does not + overwrite the newer transition's pending schedule and returns `overtaken`. +4. If an old Worker mutated history before detecting that final conflict, the newest + transition remains durably pending and runs after the old Worker releases the + history lock. Therefore stale history may exist temporarily, but it cannot become + the terminal recorded state or cancel repair of the winner. + +This is narrower than prevention. No test or caller may claim an old Worker cannot +write after a newer native commit; the testable claim is that the latest pair stays +durably scheduled and eventually owns the clean under-lock post-probe, even across +spawn failure, Worker death, or process restart. Ordering, so absence of deadlock is checkable: **native lock → history lock, never the inverse**, and they are never held simultaneously. -## 7. The lock namespace keys on effective user, not on any home path +## 7. The lock namespace has one environment-independent root per effective user Round 1 #7 said `homedir()` reads `HOME`/`USERPROFILE`, so a service and a CLI for the same user can take different locks. I accepted the fix — use @@ -387,14 +667,63 @@ export type UserIdentity = | { platform: "win32"; sid: string }; ``` -The lock path is then -`/opencodex/native-write-locks/v1//.sqlite`, -with the per-user directory created mode `0700` and validated by `lstat` before -use — a symlink or a wrong owner is a refusal, never a trust. +The key alone was not enough. The earlier `` called an undefined +resolver and allowed service/CLI processes to choose different parents through +`TMPDIR`, `XDG_RUNTIME_DIR`, or `LOCALAPPDATA`. `resolveOsRuntimeDirectory` is now +the sole algorithm below and reads none of those variables. + +```ts +/** + * Resolve the effective account from operating-system identity APIs only. + * Failure is a typed namespace refusal; username/home/environment fallback is + * forbidden because it can split one account across two lock databases. + */ +export type ResolveEffectiveUserIdentity = () => UserIdentity; + +/** + * Return the canonical, private per-user runtime root. The result never depends + * on HOME, USERPROFILE, TMPDIR, XDG_RUNTIME_DIR, TEMP, TMP or LOCALAPPDATA. + */ +export type ResolveOsRuntimeDirectory = (identity: UserIdentity) => string; +``` + +WP8b implements and exports constants of both function types from +`src/codex/user-identity.ts`; it does not ship declarations without bodies. + +Exact platform algorithm: + +- **macOS and Linux:** obtain the effective uid from `getuid(2)` (Bun + `process.getuid()` is the public call). Require `/tmp` to resolve by + `realpathSync.native`, be a real directory owned by uid 0, and have the sticky + bit plus world-write/search semantics. The root is + `/opencodex-runtime-v1-`. Create that one component with + mode `0700`; on every use, `lstat`/descriptor checks require a non-symlink real + directory, exact effective uid, and exact `0700`. There is no `/run/user`, + `XDG_RUNTIME_DIR`, `TMPDIR`, home, or cwd fallback. Failure refuses. +- **Windows:** open the current process effective token with `TOKEN_QUERY`, call + `GetTokenInformation(TokenUser)`, and canonicalize it with + `ConvertSidToStringSidW`. Blank/malformed SID or any API failure refuses; account + name and `USERPROFILE` are not fallbacks. Resolve `FOLDERID_LocalAppData` with + `SHGetKnownFolderPath` for that same effective token, ignoring the `LOCALAPPDATA` + environment variable. The root is + `/OpenCodex/Runtime/v1/`. Resolve and inspect each + component without following a reparse redirect, then require/harden an ACL owned + by that SID that grants only that SID, `SYSTEM`, and `Administrators`. Known-folder, + SID, canonicalization, reparse, owner, or ACL failure refuses; there is no temp or + ProgramData fallback. + +The lock database path is +`/native-write-locks/.sqlite`. +POSIX directories are `0700` and files `0600`; Windows applies the required ACL to +the root, database, and rollback journal. Every existing component is checked before +use and again through stable descriptors around SQLite open/transaction boundaries. +A symlink, junction/reparse redirect, wrong owner, broad mode/ACL, or substituted +path is a refusal, never something the resolver repairs in place. The test that matters, and the one my first version could not have failed: two -child processes with **different** `HOME`/`USERPROFILE` values must take the -**same** lock. +child processes with different `HOME`, `USERPROFILE`, `TMPDIR`, `XDG_RUNTIME_DIR`, +`TEMP`, `TMP`, and `LOCALAPPDATA` values but the same effective uid/SID and canonical +`CODEX_HOME` must resolve the same root and take the same lock. ## 8. Names @@ -428,17 +757,38 @@ Housing a finding in the wrong unit is not housing it. `tests/codex-integration-record.test.ts`: a v1 record with only `history` is valid to a provenance reader and vice versa (audit #3); unknown top-level keys -survive a write; unparseable fails closed rather than resetting. +survive a write; unparseable fails closed rather than resetting. Missing and +field-free v1 records initialize to `{0,null}` and persist both native fields on +first update. Generation-only/tx-only legacy records refuse. Two updates expecting +the same pair race; exactly one updates and the loser returns `conflict` without +overwriting the winner's txId or pending history schedule. `tests/codex-convergence-contract.test.ts`: every `ConvergeOutcome` variant maps to the §5 row, `busy` carries `Retry-After`, and a best-effort management caller -still returns 2xx while reporting a non-converged disposition. +still returns 2xx while reporting a non-converged disposition. Compile the exported +type block with the repository TypeScript compiler so WP8b cannot regress to a +bodyless TS2391 declaration. Table-drive each artifact observation and require +`isApplied` only for the fully applied aggregate. + +`tests/codex-user-identity.test.ts`: real child processes vary every environment +home/runtime variable named in §7 and resolve one root/lock for one effective uid or +SID. POSIX activates wrong owner/mode/symlink and non-sticky `/tmp` refusal through a +resolver seam; Windows CI activates token/SID failure, known-folder failure, reparse, +owner, and broad-ACL refusal. No case falls back to an environment directory. + +WP10's Worker tests pause an old Worker during traversal, commit a newer transition, +then let the old mutation finish. Its terminal CAS must conflict, the newer pending +state must survive, and the guardian must repair it. Repeat with spawn suppressed, +Worker death, timeout, shutdown cancellation, unreadable/schema probes, and terminal +record-write failure; every failed probe count is null and the latest transition +remains durably schedulable. **The funnel must be provable, not grepped** (round 2 #2). A grep guard misses a wrapper in the same module, a re-export, an alias and a dynamic import — and the tree has all four today: `refreshCodexModelCatalog` wraps the catalog writers -(`refresh.ts:40-52`), `restoreNativeCodex` wraps config/catalog/history removal -(`inject.ts:764-794`), and `catalog.ts:11` re-exports the direct writers. +(`src/codex/refresh.ts:40-52`), `restoreNativeCodex` wraps config/catalog/history +removal (`src/codex/inject.ts:764-794`), and `src/codex/catalog.ts:11` re-exports +the direct writers. So the low-level writers move into `src/codex/internal/` whose **only** permitted importer is `convergence.ts`, and the guard test walks the module dependency @@ -452,5 +802,7 @@ other path reaches them. Reachability, not spelling. - C16 — one owner, one schema; a record from any phase reads in every other. - C17 — an A→B→A cycle between gather and commit is detected by generation, and a parent-symlink retarget is detected by target identity. -- Contributes to C15 (the history protocol is specified here, implemented in - WP10) and to C2/C12 (generations and the admission snapshot). +- Contributes to C15 with detect-and-repair: the latest native pair is durably + pending before spawn, a stale Worker cannot replace its record, and the guardian + eventually repairs history. WP10 implements that protocol. Also contributes to + C2/C12 (generations and the three-read admission/observation sequence). diff --git a/devlog/_plan/260804_codex_write_substrate/010_catalog_seam.md b/devlog/_plan/260804_codex_write_substrate/010_catalog_seam.md index fcde090bd..ec0ed9455 100644 --- a/devlog/_plan/260804_codex_write_substrate/010_catalog_seam.md +++ b/devlog/_plan/260804_codex_write_substrate/010_catalog_seam.md @@ -12,15 +12,23 @@ policy is a swallowed exception (`src/server/management-api.ts:105-112`, outside a lock and a fixed commit inside it. This phase fixes that catalog mechanism. It is the **first real implementation** -of the contract's `convergeCodex`. It does not define another -entry point, record, route mapping, admission shape, or shared result union. WP8b -landed those declarations without rewiring behavior (`005_contract.md` §What -"lands first"). WP9 consumes them and rewires the catalog callers in the same -commit, so this phase typechecks and preserves the callers' 2xx/201 behavior on -its own. Nothing here waits for WP10-WP12 to make the tree buildable. +of the contract's `convergeCodex`, but the earlier decision to send management +mutations through full apply/injection/history convergence is reversed. Those 16 +callers currently refresh catalog and cache only +(`src/server/management-api.ts:105-112`, `src/codex/refresh.ts:40-52`); changing a +provider must not start rewriting `config.toml`, profile, journal, or history in the +WP9 commit. WP9 therefore implements a catalog-scoped request and rewires only that +behavior. WP12 installs the authoritative full funnel and rewires `/api/sync` and +the remaining lifecycle callers after WP10-WP11 supply their safety mechanics. + +WP9 does not define another entry point, record, route mapping, admission shape, or +shared result union. WP8b lands the minimal concrete primitives listed below, not +declarations that require WP12 to become executable. WP9 consumes them and rewires +the catalog callers in the same commit, so this phase typechecks and preserves the +callers' 2xx/201 and native-write behavior on its own. All current-code citations and diff context below were rechecked on 2026-08-04 at -`2d5e080dea3e7000bf2111b381c7c1a3c4f5fb11`. +`47e7cac27723fa09dd7bb1bacac402b1e579b358`. ## IN / OUT @@ -38,20 +46,20 @@ IN — catalog mechanism: - `src/codex/catalog/provider-fetch.ts` (MODIFY) — sanitized, catalog-private degradation notices. - `src/codex/convergence.ts` (MODIFY) — implement the contract-declared - `convergeCodex` for the first time. This phase consumes `AdmissionSnapshot`, - `CommitExpectation`, `CatalogDisposition`, and `ConvergeOutcome` from - `convergence-types.ts`; it does not redefine them. + `convergeCodex` for the first time. The management path consumes the catalog-scoped + request/snapshot, generation tokens, `CatalogDisposition`, and `ConvergeOutcome` + from `convergence-types.ts`; it does not call WP12's full admission or observer. - `src/codex/internal/catalog-commit.ts` (NEW/MOVE) — the prepared catalog/cache/ backup writer, reachable only from `convergence.ts`, as required by `005_contract.md` §Test plan. -- `src/codex/sync.ts` (MODIFY) — delegate catalog work to `convergeCodex`; retain - the existing ordinary-failure injection fallback until later phases replace - more of the native path. +- `src/codex/sync.ts` (NO CHANGE) — explicit sync remains on its current full native + path in WP9. WP12 rewires it after the full admission/observation mechanics exist. IN — production callers: - `src/server/management/context.ts`, `src/server/management-api.ts` (MODIFY) — - inject/call `convergeCodex`, not a catalog-specific orchestrator. + inject/call `convergeCodex` with `scope: "catalog"`, not a catalog-specific entry + point and not the full convergence scope. - `src/server/management/provider-routes.ts` (MODIFY) — six mutations report the contract's `CatalogDisposition` while retaining their primary status. - `src/server/management/model-routes.ts` (MODIFY) — six mutations, same rule. @@ -59,8 +67,8 @@ IN — production callers: suppressing Claude follow-up work. - `src/server/management/agent-settings-routes.ts` (MODIFY) — two mutations, without suppressing Claude/Desktop follow-up work. -- `src/server/management/config-routes.ts` (MODIFY) — explicit sync calls - `convergeCodex` and hands the result to the contract adapter. +- `src/server/management/config-routes.ts` (NO CHANGE) — WP9 leaves explicit sync at + current lines 261-268; WP12 moves it to full convergence and the contract adapter. - `structure/03_catalog-and-subagents.md`, `structure/05_gui-and-management-api.md` (MODIFY) — document the production funnel and best-effort mutation semantics. @@ -95,15 +103,53 @@ OUT: and `toSyncResponse` — owned by `005_contract.md` §§1-5. This document deletes its old versions instead of restating them. - Management status/header ownership. `/api/sync` is mapped only by - `src/server/management/sync-response.ts` (`005_contract.md` §5). + `src/server/management/sync-response.ts` (`005_contract.md` §5), but WP12 is the + phase that first connects that adapter to the production route. - Desired-state, ownership, journal, and provenance policy — WP12 consumes the same funnel and strengthens admission; WP9 does not reserve fake outcomes for it. - The native write lock — WP11. WP9's commit is synchronous now so WP11 can wrap it later without changing the catalog contract. - History isolation/locking — WP10. +- Full admission, observed-state projection, apply/remove direction, and production + `/api/sync`/lifecycle rewiring — WP12. WP9 must not call their future helpers. - `gui/**`, transactional rollback, release/deploy actions, and the live proxy on port 10100. +## WP8b prerequisites that make WP9 self-contained + +The earlier draft assumed `inspectAdmissionSnapshot`, config/native generation +owners, and observed-state projection would already exist. They do not: WP12 owns +the full authority read and observer (`040_ownership_convergence.md:42,119-157,327-378`). +A WP9 diff that calls those helpers cannot land independently. + +WP8b must therefore add these minimal **working** primitives before WP9, with focused +typecheck/tests in the WP8b commit: + +1. `ConvergeRequest.scope` with at least `"catalog" | "full"`, plus a concrete + catalog request constructor. The production management callbacks use only + `scope: "catalog"`; `"full"` remains the compatibility/current-behavior branch + until WP12 replaces it with authoritative admission. +2. A concrete `CatalogAdmissionSnapshot` plus catalog-scoped snapshot reader that + accepts the same `OcxConfig` object the current callback already uses and captures + only the config generation and catalog target identities WP9 validates. It + performs no service ownership, external-provider, journal, provenance, desired- + state, history, or observed-state work and is not `inspectAdmissionSnapshot`. +3. Concrete config/native generation owners: every cooperating persisted-config + commit bumps the config generation through the existing config mutation owner, + and the integration-record owner reads/advances native generation plus `txId`. + WP9 may consume those tokens; it may not assume WP12 will add their storage or + bump sites later. +4. One contract projection for catalog-only completion. WP9 supplies the real + `CatalogDisposition`; history and observed sections are synthesized as + **no-change/not-evaluated**, never by invoking WP10 history or WP12 observation. + +These are substrate primitives, not a partial ownership implementation. WP12 still +owns the authoritative full `AdmissionSnapshot`, fresh under-coordination re-read, +authority/provenance checks, real observed-state projection, and full caller funnel. +Moving the concrete generation owners forward is an explicit ownership correction +to `040_ownership_convergence.md:42`: WP12 consumes those WP8b owners instead of +introducing them after WP9 has already depended on them. + ## The catalog-private candidate **INFERRED implementation choice:** the candidate is opaque and one-shot. Its payload is held in a module-private @@ -128,7 +174,7 @@ interface CandidateState { const states = new WeakMap(); export async function gatherCodexCatalogCandidate( - admission: AdmissionSnapshot, + admission: CatalogAdmissionSnapshot, ): Promise; export function commitCodexCatalogCandidate( @@ -137,11 +183,12 @@ export function commitCodexCatalogCandidate( ): CatalogCommitOutcome; ``` -The exact `AdmissionSnapshot` returned by the contract admission is passed to -gather. `prepareCatalogSync` receives `admission.config` — **that object**, not the -server's captured config and not a separate `readConfigDiagnostics()` result. This -is the transfer required by `005_contract.md` §4. A gather that reopens config has -reintroduced the stale-object disagreement audit #8 identified. +The catalog-scoped snapshot receives the same config object the current management +callback already uses. `prepareCatalogSync` receives `admission.config` — **that +object**, not a separate `readConfigDiagnostics()` result. The generation token +detects a cooperating persisted transition before commit. WP12 later replaces this +limited input with its authoritative full admission; WP9 does not import that future +helper. Gather performs provider auth/network work, source loading, parsing, merging, serialization, cache-wrapper construction, and backup planning. It performs no @@ -273,7 +320,7 @@ and test imports migrate. The dependency-graph test, not an `rg` spelling guard, proves no alias, re-export, wrapper, or dynamic import reaches the writers outside `convergence.ts` (`005_contract.md` §Test plan). -## The first production `convergeCodex` +## The first production `convergeCodex` is catalog-scoped for management WP8b declared this function as a type only. WP9 now adds a non-placeholder implementation in `src/codex/convergence.ts`: @@ -282,30 +329,28 @@ implementation in `src/codex/convergence.ts`: +export async function convergeCodex( + request: ConvergeRequest, +): Promise { -+ const admission = inspectAdmissionSnapshot(); -+ if (request.action === "observe") return observeWithoutWrite(admission); -+ -+ const gathered = admission.intent === "on" -+ ? await gatherCodexCatalogCandidate(admission) -+ : null; ++ if (request.scope === "catalog") { ++ const admission = captureCatalogAdmissionSnapshot(request); ++ const gathered = await gatherCodexCatalogCandidate(admission); ++ const catalog = commitCatalogAgainstCurrentGeneration(admission, gathered); ++ return projectCatalogOnlyOutcome(catalog, { ++ history: "no-change", ++ observed: "no-change-not-evaluated", ++ }); ++ } + -+ return coordinateCurrentNativeBehavior({ -+ request, -+ admission, -+ gathered, -+ commitCatalog: commitCodexCatalogCandidate, -+ }); ++ return coordinateLegacyFullBehavior(request); +} ``` -`coordinateCurrentNativeBehavior` is real at this commit: it preserves the existing -apply/injection/history behavior and uses the new catalog seam. It is not a throw, -TODO, compatibility path around `convergeCodex`, or promise that WP10/WP12 must land -before WP9 works. Later phases replace mechanisms behind this same entry point. - -Desired direction comes only from `admission.intent`; callers pass -`action:"converge"`, never `apply` or `remove`. Desired OFF therefore performs the -current removal path and is not a catalog `skipped` outcome (`005_contract.md` §2). +`projectCatalogOnlyOutcome` reports the actual catalog/cache/backup result and +synthesizes history and observed fields as no-change/not-evaluated. It never calls +config injection, profile, journal, history, restoration, or WP12 observation. +`coordinateLegacyFullBehavior` is only a typed adapter over the existing full path; +WP9 does not route management or `/api/sync` into it. WP12 replaces that branch with +the authoritative full funnel and rewires the production full callers. This is a +plain reversal of the earlier WP9 design, which had made provider/model edits perform +full native convergence before its safety phases existed. ## Every management caller uses the funnel @@ -341,6 +386,7 @@ Each of the 16 current awaits — provider 6 -const catalogRefresh = await refreshCodexCatalogBestEffort(); +const outcome = await convergeCodex({ + action: "converge", ++ scope: "catalog", + reason: "management-mutation", + mode: "automatic", + deadlineMs: MANAGEMENT_CODEX_CONVERGENCE_DEADLINE_MS, @@ -354,29 +400,14 @@ keeps its current 200/201 and the persisted mutation, appends `catalogRefresh`, continues unrelated Claude/Desktop work. That is the best-effort behavior promised by `005_contract.md` §2. -## Explicit sync consumes the adapter - -The old status table and manual `Retry-After` logic are deleted. The contract owns -them in `005_contract.md` §5. `src/server/management/config-routes.ts:261-268` -only invokes the funnel and adapter: - -```diff - if (url.pathname === "/api/sync" && req.method === "POST") { -- const result = await syncModelsToCodex(undefined, config, null); -- return jsonResponse(result, result.ok ? 200 : 500); -+ const outcome = await convergeCodex({ -+ action: "converge", -+ reason: "api-sync", -+ mode: "explicit", -+ deadlineMs: EXPLICIT_CODEX_CONVERGENCE_DEADLINE_MS, -+ }); -+ return toSyncResponse(outcome); - } -``` +## Explicit sync is deliberately not rewired in WP9 -No phase-local route helper chooses status, body, or headers. A new outcome variant -must fail the contract adapter's exhaustive `never` check, not silently take a WP9 -default branch. +`src/server/management/config-routes.ts:261-268` continues to call +`syncModelsToCodex(undefined, config, null)` in the WP9 commit. Moving that route to +`convergeCodex` here would require the full admission, observed-state projection, +history safety, and response semantics that WP10-WP12 have not landed. WP12 performs +the real diff to `scope: "full"` plus `toSyncResponse`; until then the current route +status/body behavior remains unchanged. ## Tests @@ -384,7 +415,7 @@ default branch. `tests/codex-refresh.test.ts` replaces the all-in-one dependency tests with: -1. gather uses `AdmissionSnapshot.config`, performs provider/parse/assembly work, +1. gather uses `CatalogAdmissionSnapshot.config`, performs provider/parse/assembly work, and leaves a real isolated-home recursive manifest byte-identical; 2. commit invokes only the fixed writer list; injected provider/parser/subprocess functions throw if reached beneath the synchronous boundary; @@ -419,12 +450,16 @@ generation owners: direct writer in `src/codex/internal/catalog-commit.ts` is reachable only from `convergence.ts`. - Drive all 16 real management routes with an injected `convergeCodex`, assert one - call using persisted admission rather than the route's captured config, preserve + `scope: "catalog"` call using the same config object as today's callback, preserve each primary 2xx/201, and observe the additive `catalogRefresh`. +- For every management route, inject spies that fail on config/profile/journal/history + writes and assert zero calls; assert history and observed result sections are the + contract's no-change/not-evaluated projection. - A refused/deferred catalog attempt must not suppress combo Claude work or agent settings Claude/Desktop work. -- Drive `POST /api/sync` and assert exact response behavior through - `toSyncResponse`; do not duplicate the contract's status table in this suite. +- Drive `POST /api/sync` and assert it still follows the pre-WP9 + `syncModelsToCodex` route behavior. The `toSyncResponse` production proof belongs + to WP12/WP13. ## Verification @@ -449,13 +484,15 @@ starts, stops, syncs, restores, or ensures the live proxy on port 10100. match the committed catalog. 4. Drive one best-effort management mutation through the real server boundary; observe its primary 2xx and contract disposition. -5. Drive explicit sync through `convergeCodex` and `toSyncResponse`. +5. Drive explicit sync and prove WP9 left its current route behavior unchanged. ## Accept criteria - **C1** — gather is write-free and commit is synchronous/fixed. Catalog failures remain catalog-private until projected through `ConvergeOutcome`; all 16 callers preserve their primary success behavior and expose the contract disposition. + Their request is `scope: "catalog"`; config/profile/journal/history write spies stay + at zero, and history/observed fields are no-change/not-evaluated. - **C2 / C17 (contract-scoped)** — real config/native generation changes and single-direction target-identity drift reject before write. No content hash or path string is presented as arbitrary filesystem ABA protection. @@ -463,4 +500,6 @@ starts, stops, syncs, restores, or ensures the live proxy on port 10100. through `convergeCodex`, and no other importer reaches the direct catalog writers. - **N2** — the WP9 commit contains the first working `convergeCodex`, rewires its callers in that same commit, passes typecheck, and preserves current behavior. - It has no placeholder whose correctness depends on WP10-WP12. + WP8b already supplies the concrete catalog snapshot/request/projection and both + generation owners; WP9 calls no WP12 admission or observer placeholder. WP12 is + explicitly the phase that replaces the legacy full branch and rewires full callers. From bd1065a85c4981a1662e3676243d7de3b4ed9c81 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Tue, 4 Aug 2026 11:59:15 +0900 Subject: [PATCH 035/163] =?UTF-8?q?docs(substrate):=20round=204=20?= =?UTF-8?q?=E2=80=94=20a=20Critical=20that=20would=20have=20torn=20down=20?= =?UTF-8?q?the=20real=20proxy?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit WP13 invokes ocx service start/stop/uninstall as production entry points, and service identifiers are global constants, not derived from any home: com.opencodex.proxy at service.ts:42, a fixed Task Scheduler task on Windows, a fixed systemd user unit on Linux. A mktemp OPENCODEX_HOME does not namespace a launchd label, so the suite I commissioned to prove safety would have stopped and uninstalled the owner's service. I checked: launchctl lists com.opencodex.proxy running as PID 72848 on this machine right now. The unit exists because turning one client off must not disturb anything else, and its acceptance suite would have taken down the proxy. The architectural finding: my CAS is not a CAS. updateIntegrationRecord does read/compare/replace under the caller's coordinator, but native and history hold different non-overlapping coordinators, so a Worker can read pair N, a native transition can write its schedule at N+1, and the Worker can replace the JSON with stale N. Detect-and-repair was chosen precisely for that sequence, so three findings rest on it. And I keyed the lock on CODEX_HOME while leaving the state it protects under each OPENCODEX_HOME, so two installs sharing one Codex home serialize their writes and then consult different counters. Nine findings closed, and the reviewer verified rather than accepted them — they hand-counted the 16 management catalog writes and read the service labels out of the source. I verified the compile claim myself: ten blocks give zero diagnostics once the real OcxConfig import is added, so round 3's TS2391 is genuinely gone. Four of the eight remaining blockers are one question I have now gotten wrong twice: where cross-process transition state lives. A JSON file under one home cannot be compare-and-swapped and cannot be shared by two homes. --- .../009_audit_synthesis_r4.md | 132 ++++++++++++++++++ 1 file changed, 132 insertions(+) create mode 100644 devlog/_plan/260804_codex_write_substrate/009_audit_synthesis_r4.md diff --git a/devlog/_plan/260804_codex_write_substrate/009_audit_synthesis_r4.md b/devlog/_plan/260804_codex_write_substrate/009_audit_synthesis_r4.md new file mode 100644 index 000000000..b261b2650 --- /dev/null +++ b/devlog/_plan/260804_codex_write_substrate/009_audit_synthesis_r4.md @@ -0,0 +1,132 @@ +# Substrate audit round 4 — synthesis + +Verdict: **FAIL**, 8 blocking. Nine closed, and one **Critical** that would have +damaged the owner's machine. + +## The Critical, first + +WP13 invokes `ocx service start/stop/uninstall` as production entry points +(`050:93`). Service identifiers are **global constants**, not derived from any +home: `com.opencodex.proxy` (`src/service.ts:42`), a fixed Task Scheduler task +on Windows (`:1868,1894`), a fixed systemd user unit on Linux (`:2045,2069`). + +A `mktemp` `OPENCODEX_HOME` does not namespace a launchd label. So the suite I +commissioned to prove safety would have **stopped and uninstalled the real +service**. + +I checked this machine: + +``` +launchctl list | grep opencodex +72848 0 com.opencodex.proxy +``` + +It is installed and running right now. This is not theoretical. + +The whole unit exists because turning one client off must not disturb anything +else, and its acceptance suite would have torn down the owner's proxy. Rows +P34-P36 leave the workstation suite; they run only on a disposable host proven +to have no installed service, and that requirement is stated in the doc rather +than assumed. + +Two smaller versions of the same mistake: lock artifacts land in the fixed +per-user runtime root, outside the `mkdtemp` root the harness deletes, so every +case leaks; and the wrong-owner fixture (`050:235`) cannot be built by an +unprivileged CI account at all, since you cannot create a file owned by another +uid. + +## The architectural finding: my CAS is not a CAS + +`updateIntegrationRecord` does read/compare/replace "under the caller's +coordinator" (`005:246`) — but native and history hold **different, +non-overlapping** coordinators (`005:632`). So: + +1. the history Worker reads pair `N` +2. the native transition writes its pending schedule at `N+1` +3. the Worker replaces the JSON with stale `N` + +A separate-file read-modify-write is not conditional just because each writer +holds *a* lock. And this is exactly the overtaking sequence detect-and-repair was +chosen to handle, so #1, #5 and N3 all rest on it. + +**Accept.** The pair and the schedule move into a conditional row update in the +coordinator that already exists for config mutation, or a single narrow +cross-process record lock is shared by both domains. A JSON file cannot carry +this invariant. + +## The finding that reframes the key + +New #2: the native lock is keyed on canonical `CODEX_HOME` alone (`005:715`), +while `integrations/codex.json` lives under each `OPENCODEX_HOME`. Two opencodex +installs pointing at one Codex home therefore **serialize their writes and then +consult different generation counters** — each stale Worker sees its own tx as +current. + +I keyed the lock correctly and left the state it protects keyed differently. The +generation and schedule belong in the `CODEX_HOME`-keyed coordinator, or a +competing opencodex owner is refused outright. + +## What actually closed + +Nine findings, and the reviewer verified rather than accepted: + +- **#4** the adapter's `never` check is now structurally implementable +- **#9, #10, #12, #13, N1, N4, N5** all closed +- **Round-3 New #2** WP9's management funnel is catalog-only and forbids + config/profile/journal/history writes +- **Round-3 New #4** the "no window" reversal held + +The reviewer also **counted the census by hand**: exactly 16 management catalog +writes — 6 provider, 6 model, 2 combo, 2 agent-settings — matching WP13's claim, +and confirmed the 14-route/16-site distinction is correct. + +## The compile claim, measured + +I verified it myself. The ten blocks as printed give two `TS2304` for +`OcxConfig`; adding the real import gives **zero diagnostics**. So the TS2391 +that failed round 3 is genuinely gone, and the residue is a missing import line +in the document, not a design defect. The reviewer independently reproduced +both, and additionally found `TS2345` on WP12's request object — it omits +`scope` — and that WP12 still prints the old bodyless form. + +That is the useful pattern of this whole audit: claims get compiled, censuses +get counted, and service labels get read out of the source. + +## Honest position after four rounds + +The reviewer's judgment: *"still not implementable end to end by an outsider. +The remaining work is architectural, not polish."* + +I agree, and the trend supports continuing rather than stopping: + +| Round | Closed | Open | New | +|---|---|---|---| +| 1 | — | 13 | 13 | +| 2 | 1 | 11 | 5 | +| 3 | 5 | 11 | 4 | +| 4 | **9** | 8 | 5 | + +But four of the eight remaining blockers are one question I have now gotten +wrong twice: **where does cross-process transition state live?** A JSON file +under one home cannot be compare-and-swapped, and cannot be shared by two homes. +Answer that once and #1, #5, #6, N3 and New #2 collapse together. + +## Next + +1. Move generation + schedule into the existing config-mutation SQLite + coordinator with a conditional row update, keyed so two opencodex homes + sharing one Codex home cannot diverge. This is the load-bearing change. +2. Give `src/config.ts` the durable counter API, and add it to WP8b's IN list — + WP9 already delegates the owner there while WP8b's scope excludes the file. +3. One exported resolver returns the FINAL lock path; consumers stop + re-appending identity and version segments. +4. Publish the writer inventory with permitted roots per domain, since + `history-worker.ts` must reach history writers and `convergence.ts` cannot be + the only root. +5. WP13: remove P34-P36 from the workstation suite, add the mid-traversal + repair scenario, and confine or account for lock artifacts. +6. Adoption requires a verified native-clean observation, or splits into salvage + then adopt. +7. Small: `scope` in WP12's request, the bodyless form in WP12, the contract's + final C17 text, unknown-key passthrough at nested levels, and the compile + prelude import. From a68b4d90779b30a2f40f5a541973973a6fe9e7cd Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Tue, 4 Aug 2026 12:18:29 +0900 Subject: [PATCH 036/163] docs(substrate): move transition state into a real coordinator, and stop the suite from killing the proxy MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The load-bearing change: {nativeGeneration, currentTxId} and the pending history schedule leave integrations/codex.json for a sibling SQLite coordinator keyed by effective user and canonical CODEX_HOME hash, with conditional row updates. A JSON read/compare/replace was never a CAS — native and history hold different coordinators, so a Worker could read pair N, a native transition could write its schedule at N+1, and the Worker could overwrite it with stale N. Keying on CODEX_HOME also fixes the second half: two opencodex homes sharing one Codex home now consult the same row instead of diverging. The JSON keeps only version, provenance and recursively preserved extension keys. WP13 drops P34-P36 and the three other global-service callers from the workstation suite entirely, behind an explicit disposable-account gate with the exact launchctl/schtasks/systemctl checks and required empty results. Scenario H now tests the race that matters: A holds BEGIN IMMEDIATE and mutates a manifest and a rollout, B commits, and A's terminal update conflicts so the live guardian repairs B. Lock-artifact cleanup is scoped to the exact per-case SQLite path and its journal siblings. Adoption now requires an all-surface structurally verified native-clean observation and refuses on residue or ambiguity, so recovery cannot enshrine opencodex routing as the native baseline. Each amended doc compiles clean on its own. Combining all nine still fails, and 010_catalog_seam.md is the source — it was not in this round's scope and still prints bodyless declarations and undefined catalog types. Recorded here rather than claimed fixed. --- .../005_contract.md | 433 +++++++++++++----- .../020_history_isolation.md | 275 ++++++++--- .../040_ownership_convergence.md | 294 +++++++++--- .../050_composed_acceptance.md | 221 +++++++-- 4 files changed, 935 insertions(+), 288 deletions(-) diff --git a/devlog/_plan/260804_codex_write_substrate/005_contract.md b/devlog/_plan/260804_codex_write_substrate/005_contract.md index d25847b70..569cb5475 100644 --- a/devlog/_plan/260804_codex_write_substrate/005_contract.md +++ b/devlog/_plan/260804_codex_write_substrate/005_contract.md @@ -13,18 +13,23 @@ A contract nobody collected is a fifth opinion. ## IN / OUT -IN: `src/codex/integration-record.ts` (NEW — sole owner of the record), +IN: `src/config.ts` (MODIFY — durable config-generation API), +`src/codex/integration-record.ts` (NEW — sole owner of the JSON record), +`src/codex/transition-state.ts` (NEW — sole owner of the CODEX_HOME-keyed +SQLite transition row), `src/codex/convergence.ts` (NEW — the single entry point), `src/codex/convergence-types.ts` (NEW — every shared type), `src/codex/generation.ts` (NEW), `src/codex/user-identity.ts` (NEW — §7), `src/server/management/sync-response.ts` (NEW — the one adapter), `tests/codex-integration-record.test.ts` (NEW), +`tests/codex-transition-state.test.ts` (NEW), `tests/codex-convergence-contract.test.ts` (NEW), `tests/codex-user-identity.test.ts` (NEW). -OUT: catalog mechanics (WP9), history mechanics (WP10), lock mechanics (WP11), -ownership mechanics (WP12). This phase owns *shapes and the funnel*, not the -work inside them. +OUT: catalog mechanics (WP9), history mechanics (WP10), native-lock acquisition +and retry mechanics (WP11), ownership mechanics (WP12). The final coordinator +path, transition table/CAS, config-generation API, shapes and funnel are IN; +the domain work performed while those coordinators are held is OUT. ### What "lands first" has to mean (round 2 N2) @@ -32,10 +37,10 @@ The reviewer showed the previous version could not land: it was "OUT: every behavior" while declaring a runtime `convergeCodex`, and a throwing placeholder is not a safe commit. -So WP8b lands **types, validators, the record owner, the identity resolver and -the response adapter — and rewires nothing.** `convergeCodex` is declared here -as a type only; WP9 supplies its first real implementation and rewires the -catalog callers at that commit. +So WP8b lands **types, validators, both durable-state owners, the config-generation +API, the final coordinator-path resolver and the response adapter — and rewires +nothing.** `convergeCodex` is declared here as a type only; WP9 supplies its first +real implementation and rewires the catalog callers at that commit. **Invariant for every phase in this unit:** each phase typechecks and preserves behavior at its own commit. No phase may leave a placeholder that a later phase @@ -47,26 +52,28 @@ is required to replace before the tree is correct. containing different fields, so a record from either is malformed to the other (audit #3). +**TypeScript compile prelude.** The TypeScript fences in this document are +concatenated contract fragments. Compile them in document order after prepending +`import type { OcxConfig } from "../types";`. `OcxConfig` is the real export used +by `src/config.ts:34`; omitting this prelude gives TS2304 even though the contract +itself is otherwise valid. + ```ts /** - * The single durable record for the Codex integration. + * The non-CAS JSON record for the Codex integration. * - * ONE owner. WP10 (history state) and WP12 (provenance) both write here, and - * both go through `updateIntegrationRecord` — never their own read/merge/write. - * Round 1 had two owners and two schemas for this exact file. + * ONE owner. WP12 writes provenance here through `updateIntegrationRecord` — + * never its own read/merge/write. Cross-process transition state is deliberately + * absent; it belongs to the CODEX_HOME-keyed SQLite row below. * - * Every section is OPTIONAL at v1. A record written before a section existed is - * VALID, not malformed: absence means "that subsystem has not spoken yet". This - * is what lets WP10 land before WP12 without a migration. + * Provenance is OPTIONAL at v1. A record written before WP12 is valid, and + * unknown extension sections from a newer writer remain valid and preserved. */ export interface CodexIntegrationRecord { version: 1; - history?: CodexHistoryState; provenance?: CodexProvenanceLedger; - /** Bumped by every cooperating native commit. See §3. */ - nativeGeneration?: number; - /** The transaction that owns `nativeGeneration`; null is legal only at zero. */ - currentTxId?: string | null; + /** Unknown keys from a newer writer survive every older-writer update. */ + readonly [extra: string]: unknown; } ``` @@ -137,6 +144,8 @@ export interface CodexProvenanceEntry { postImage: string | null; txId: string; at: string; + /** Entry-level extensions are preserved, not only ledger/top-level keys. */ + readonly [extra: string]: unknown; } export interface CodexProvenanceLedger { @@ -221,63 +230,202 @@ never a zero-looking count. compatibility outcome; it is never persisted as durable history and never answers `isApplied` or `converged` with a false-looking boolean. -### Durable read/update and initialization +### Durable state: the JSON CAS was wrong + +The previous contract called `updateIntegrationRecord` a CAS because it compared +two JSON fields and replaced the file while *the caller's* coordinator was held. +That was wrong. Native and history callers hold different coordinators, so an old +history Worker can read JSON at N, a native transition can replace it at N+1, and +the Worker can then replace the file with stale N. Serialization under two +non-overlapping locks is not compare-and-swap. + +The key was wrong as well. `integrations/codex.json` is under `OPENCODEX_HOME`, but +native exclusion is keyed by canonical `CODEX_HOME`. Two OpenCodex installations +sharing one Codex home therefore serialized and then consulted different counters. +The pair and all history scheduling/terminal state move to one SQLite row in the +final CODEX_HOME-keyed coordinator database. The JSON record keeps exactly +`version`, the provenance ledger, and unknown extension members; none is the +authority for transition admission, Worker overtaking, or retry scheduling. ```ts -export interface IntegrationRecordVersion { +export interface CodexTransitionVersion { readonly nativeGeneration: number; readonly currentTxId: string | null; } +export interface CodexTransitionState extends CodexTransitionVersion { + /** Durable schedule and latest terminal observation for this exact pair. */ + readonly history: CodexHistoryState; + readonly historySchedule: null | Readonly<{ + direction: "apply" | "remove"; + authoritySnapshotId: string; + }>; +} + export type IntegrationRecordRead = - | { kind: "missing"; record: null; version: { nativeGeneration: 0; currentTxId: null } } - | { kind: "ready"; record: CodexIntegrationRecord; version: IntegrationRecordVersion } - | { kind: "legacy-ambiguous"; record: CodexIntegrationRecord } + | { kind: "missing"; record: null } + | { kind: "ready"; record: CodexIntegrationRecord } | { kind: "invalid"; message: string }; export type ReadIntegrationRecord = () => IntegrationRecordRead; export type IntegrationRecordUpdate = - | { kind: "updated"; record: CodexIntegrationRecord; version: IntegrationRecordVersion } - | { kind: "conflict"; current: IntegrationRecordVersion } + | { kind: "updated"; record: CodexIntegrationRecord } | { kind: "invalid"; message: string }; /** - * Compare `expected` against both native fields, apply `mutate`, and atomically - * replace the record while the caller's coordinator is held. A mismatch writes - * nothing. The updater preserves unknown keys at every object level. + * Update only non-CAS JSON data. Callers may not add transition or schedule + * fields. The updater preserves unknown keys at every object level. */ export type UpdateIntegrationRecord = ( - expected: IntegrationRecordVersion, mutate: (record: CodexIntegrationRecord) => CodexIntegrationRecord, ) => IntegrationRecordUpdate; + +export type TransitionStateRead = + | { kind: "ready"; state: CodexTransitionState } + | { kind: "legacy-ambiguous"; message: string } + | { kind: "unavailable"; reason: "busy" | "unsafe-path" | "database" }; + +export type TransitionStateUpdate = + | { kind: "updated"; state: CodexTransitionState } + | { kind: "conflict"; current: CodexTransitionState } + | { kind: "unavailable"; reason: "busy" | "unsafe-path" | "database" }; + +export type ReadCodexTransitionState = () => TransitionStateRead; + +/** Publish N+1 and its pending schedule with one conditional SQLite UPDATE. */ +export type BeginCodexTransition = ( + expected: CodexTransitionVersion, + next: Readonly<{ + txId: string; + direction: "apply" | "remove"; + authoritySnapshotId: string; + nextRetryAt: string; + }>, +) => TransitionStateUpdate; + +/** Change only history columns when the exact native pair still owns the row. */ +export type UpdateCodexHistoryTransition = ( + expected: CodexTransitionVersion, + history: CodexHistoryState, +) => TransitionStateUpdate; ``` WP8b implements and exports `const readIntegrationRecord: ReadIntegrationRecord` and `const updateIntegrationRecord: UpdateIntegrationRecord` from -`src/codex/integration-record.ts`; these are executable functions in that phase, -not ambient declarations. - -A missing file normalizes to `{ nativeGeneration: 0, currentTxId: null }`; the -first successful update creates `version:1` and persists both native fields even -when it writes only `history` or `provenance`. A v1 record with neither native field -has the same initial meaning, which keeps the history-only/provenance-only landing -order valid. A positive generation without a nonblank `currentTxId`, a txId without -its generation, `generation` from the abandoned draft schema, or `null` paired with -a nonzero generation is `legacy-ambiguous`: automatic mutation fails closed and an -explicit observation/recovery must establish a current pair. It is never silently -coerced to the initial state. - -`updateIntegrationRecord` does one read-modify-write under the caller's coordinator. -Native transition N uses expected `{N,currentTxId}` and writes `{N+1,newTxId}`; -history/provenance completion uses the exact current pair it started from. Thus a -late Worker receives `conflict` rather than overwriting a competing txId at the same -number. Unknown keys survive top-level and section updates so a newer writer's -record survives an older binary. - -Unreadable or unparseable is not "empty": it fails closed and the caller reports -rather than silently starting a fresh record. Losing provenance silently is how -`005_disable_leaves_a_broken_file.md` became possible. +`src/codex/integration-record.ts`, plus +`readCodexTransitionState`, `beginCodexTransition`, and +`updateCodexHistoryTransition` from +`src/codex/transition-state.ts`; these are executable functions in that phase, not +ambient declarations. + +The coordinator is a **sibling**, not an extension of `config-mutation.sqlite`. +The existing database path is derived from `getConfigDir()` +(`src/config.ts:1731-1762`), whose resolver reads `OPENCODEX_HOME` +(`src/config.ts:530-534,1254-1256`); extending it would repeat the split-key +defect. The sibling uses the same Bun SQLite pattern — private +file, `busy_timeout=0`, `BEGIN IMMEDIATE`, process-exit lock release +(`src/config.ts:1767-1818`) — but its final database path is keyed by effective +user plus canonical `CODEX_HOME` (§7). WP11's native exclusion transaction and +both transition-state callers open this same database. + +The exact singleton row is: + +```sql +CREATE TABLE codex_transition_state ( + singleton INTEGER PRIMARY KEY CHECK (singleton = 1), + native_generation INTEGER NOT NULL CHECK (native_generation >= 0), + current_tx_id TEXT, + history_status TEXT NOT NULL, + history_reason TEXT, + history_attempts INTEGER NOT NULL CHECK (history_attempts >= 0), + history_next_retry_at TEXT, + history_tx_id TEXT, + history_direction TEXT CHECK (history_direction IN ('apply', 'remove')), + history_authority_snapshot_id TEXT, + history_pending_rows INTEGER, + history_backup_entries INTEGER, + updated_at TEXT NOT NULL, + CHECK (history_status IN + ('converged', 'pending', 'running', 'blocked', 'unknown')), + CHECK (history_reason IS NULL OR history_reason IN + ('db-busy', 'permission', 'unreadable', 'schema', 'timeout', + 'shutdown-cancelled', 'worker-died', 'overtaken', 'record-write-failed')), + CHECK (history_pending_rows IS NULL OR history_pending_rows >= 0), + CHECK (history_backup_entries IS NULL OR history_backup_entries >= 0), + CHECK ((native_generation = 0 AND current_tx_id IS NULL) + OR (native_generation > 0 AND length(trim(current_tx_id)) > 0)), + CHECK ((native_generation = 0 + AND history_tx_id IS NULL + AND history_direction IS NULL + AND history_authority_snapshot_id IS NULL) + OR (native_generation > 0 + AND history_tx_id = current_tx_id + AND length(trim(history_authority_snapshot_id)) > 0)), + CHECK (native_generation > 0 OR + (history_status = 'unknown' + AND history_reason IS NULL + AND history_attempts = 0 + AND history_next_retry_at IS NULL + AND history_pending_rows IS NULL + AND history_backup_entries IS NULL)) +); +``` + +The observation columns project to `CodexHistoryState`; direction and authority +snapshot are schedule metadata required to restart the exact Worker after process +death. `not-evaluated` remains ephemeral and is rejected by the table. A native +transition publishes its winner and schedule atomically with this null-safe conditional update +(SQLite `IS` is required for the initial null txId): + +```sql +UPDATE codex_transition_state + SET native_generation = ?, current_tx_id = ?, + history_status = 'pending', history_reason = NULL, + history_attempts = 0, history_next_retry_at = ?, history_tx_id = ?, + history_direction = ?, history_authority_snapshot_id = ?, + history_pending_rows = NULL, history_backup_entries = NULL, + updated_at = ? + WHERE singleton = 1 + AND native_generation = ? + AND current_tx_id IS ?; +``` + +The first two bound values are `{nativeAfter,newTxId}`; the last two are the +expected `{nativeBefore,currentTxId}`. Worker claim/retry/terminal updates use the +same `WHERE native_generation = ? AND current_tx_id IS ?` predicate and additionally +require `history_tx_id IS ?`; they change only `history_*`, never the native pair. +The row count, not a later JSON read, is the CAS result. + +A zero-row native result means another transition won despite this caller's +admission: do not write JSON or spawn its Worker; any native bytes already committed +are unresolved and the current row's winner owns repair. Re-admit if the deadline +permits, otherwise return `deferred`. A zero-row Worker result means `overtaken`: +do not write the JSON record, do not clear the +winner's timer, and schedule from the row returned by a fresh read. A zero-row +guardian update means its timer was stale and is replaced from the current row. +Database busy/unavailable is typed `busy`/`deferred`; no caller guesses success. + +Initialization first verifies the no-legacy/native-clean precondition while the +native lock excludes another initializer, then uses one `BEGIN IMMEDIATE` +transaction: create the table, then +`INSERT OR IGNORE` singleton 1 as `{0,null}` with an `unknown` history observation, +zero attempts and no txId/direction/authority/timer/counts, and sets +`PRAGMA user_version = 1`. That initialization is legal only when the +JSON has no legacy `nativeGeneration`, `currentTxId`, `generation`, or durable +`history` member and native observation finds no unresolved routed residue. When +the row is absent, any such legacy field or native residue is `legacy-ambiguous`; automatic +mutation refuses and explicit salvage/native-clean adoption must establish the row. +An `OPENCODEX_HOME`-local positive pair is never imported because a second home may +hold a different claimant. Once the row exists, legacy JSON fields have no authority +and are removed on the next successful non-CAS record update while all unrelated +unknown keys survive. + +A missing JSON file is valid and the first provenance update creates `{version:1}`. +Unreadable/unparseable JSON is not empty: provenance mutation fails closed. Unknown +members survive at the record, ledger, and individual `CodexProvenanceEntry` levels; +tests seed a nested future key in an entry and require deep-equal preservation +after an older-writer update. ## 2. One convergence entry point @@ -408,14 +556,39 @@ indistinguishable from its failure condition. /** Bumped by every cooperating CONFIG write. Owned by src/config.ts. */ export interface ConfigGeneration { readonly value: number; } -/** Bumped by every cooperating NATIVE commit. Owned by convergence.ts. */ +/** Bumped by every cooperating NATIVE commit. Owned by transition-state.ts. */ export interface NativeGeneration { readonly value: number; } + +export type ConfigGenerationRead = + | { kind: "ready"; generation: ConfigGeneration } + | { kind: "unavailable"; reason: "busy" | "database" }; + +export type ConfigGenerationBump = + | { kind: "updated"; generation: ConfigGeneration } + | { kind: "conflict"; current: ConfigGeneration } + | { kind: "unavailable"; reason: "busy" | "database" }; + +export type ReadConfigGeneration = () => ConfigGenerationRead; +export type BumpConfigGeneration = (expected: ConfigGeneration) => ConfigGenerationBump; ``` Round 2 #6: the previous version said "two counters, both in the record" and then defined one. They are distinct because they answer different questions — did the user's configuration move, versus did somebody else write Codex's files. +WP8b adds executable `readConfigGeneration` and `bumpConfigGeneration` exports to +`src/config.ts` with the callable types above. They use a singleton +`config_generation(singleton INTEGER PRIMARY KEY CHECK(singleton=1), value INTEGER +NOT NULL CHECK(value>=0))` row in the existing `config-mutation.sqlite`. Creation +and `INSERT OR IGNORE (1,0)` happen under that database's `BEGIN IMMEDIATE`. +`bumpConfigGeneration({value:N})` executes +`UPDATE config_generation SET value = value + 1 WHERE singleton = 1 AND value = N`; +one changed row returns N+1, zero rows returns `conflict` with a fresh current read, +and busy/open failure returns `unavailable`. Every cooperating persisted config +commit calls the bump before committing the SQLite transaction; unchanged mutations +do not bump. This closes the former scope hole: WP9 delegates this owner to WP8b, +and `src/config.ts` is now explicitly IN. + ### The expected transition ```ts @@ -431,16 +604,16 @@ export interface CommitExpectation { The rule, stated so a test can check it: -> After the commit, the record must show **exactly** `nativeAfter` AND `txId` +> After the commit, the coordinator row must show **exactly** `nativeAfter` AND `txId` > equal to ours. `nativeAfter` with a different `txId` is another writer that > raced us to the same number. Anything else is interference: the outcome is > `deferred` with the surface named, never `converged`. The earlier “there is no window” claim was wrong. Process exclusion cannot make -separate file replacements and the integration-record replacement atomic. Holding +separate file replacements and the coordinator-row update atomic. Holding native + config coordination provides **no cooperating interleaving while the process is alive**; a crash can still leave any prefix of the artifact sequence with -the old record pair. +the old coordinator pair. Recovery is therefore artifact-specific. Config, generated profile, catalog, hashed/legacy backups, cache, and journal recover only from their ledger baseline @@ -608,17 +781,17 @@ This contract chooses **detect-and-repair**, not a transition gate shared across the complete history unit. The guarantee is eventual convergence to the latest durable native transition: -1. The native record CAS that writes `{nativeAfter, txId}` also writes - `history:{status:"pending", txId, nextRetryAt:, ...}` for that same - transition **before** any Worker spawn. If spawn never occurs or the Worker dies, - the guardian/startup reader still has durable work to schedule. -2. A Worker checks that the record contains its `{nativeAfter, txId}` immediately - after acquiring the history lock. A mismatch returns `pending/overtaken` without - mutation and schedules observation of the current pair. +1. The native coordinator CAS writes `{nativeAfter, txId}` and the complete + `history_status='pending'` schedule in the **same SQLite row update** before any + Worker spawn. If spawn never occurs or the Worker dies, the guardian/startup + reader still has durable work to schedule. +2. A Worker checks that the coordinator row contains its `{nativeAfter, txId}` + immediately after acquiring the history lock. A mismatch returns + `pending/overtaken` without mutation and schedules observation of the current row. 3. Because a newer native transition can commit during traversal, the Worker uses - `updateIntegrationRecord({nativeGeneration:nativeAfter,currentTxId:txId}, ...)` - for its terminal state. A CAS conflict means its result is stale; it does not - overwrite the newer transition's pending schedule and returns `overtaken`. + the §1 conditional SQLite update for its terminal history state. A zero-row CAS + means its result is stale; it does not touch JSON, overwrite the newer pending + schedule, or clear the winner's timer, and returns `overtaken`. 4. If an old Worker mutated history before detecting that final conflict, the newest transition remains durably pending and runs after the old Worker releases the history lock. Therefore stale history may exist temporarily, but it cannot become @@ -629,8 +802,13 @@ write after a newer native commit; the testable claim is that the latest pair st durably scheduled and eventually owns the clean under-lock post-probe, even across spawn failure, Worker death, or process restart. -Ordering, so absence of deadlock is checkable: **native lock → history lock, -never the inverse**, and they are never held simultaneously. +Ordering, so absence of deadlock is checkable: the native callback performs its +transition-row UPDATE in the native coordinator transaction, then releases it before +history dispatch. A Worker holds the history lock while traversing and attempts only +a fail-fast short coordinator CAS at claim/terminal boundaries; it never invokes the +native callback or waits on config coordination. `SQLITE_BUSY` leaves the current +pending row intact and retries after the history lock is released. Native and history +domain callbacks are never nested. ## 7. The lock namespace has one environment-independent root per effective user @@ -669,8 +847,8 @@ export type UserIdentity = The key alone was not enough. The earlier `` called an undefined resolver and allowed service/CLI processes to choose different parents through -`TMPDIR`, `XDG_RUNTIME_DIR`, or `LOCALAPPDATA`. `resolveOsRuntimeDirectory` is now -the sole algorithm below and reads none of those variables. +`TMPDIR`, `XDG_RUNTIME_DIR`, or `LOCALAPPDATA`. The private root resolution used by +the final-path resolver below reads none of those variables. ```ts /** @@ -681,14 +859,22 @@ the sole algorithm below and reads none of those variables. export type ResolveEffectiveUserIdentity = () => UserIdentity; /** - * Return the canonical, private per-user runtime root. The result never depends - * on HOME, USERPROFILE, TMPDIR, XDG_RUNTIME_DIR, TEMP, TMP or LOCALAPPDATA. + * Return the FINAL SQLite coordinator database path for this exact canonical + * CODEX_HOME. Consumers append no uid/SID, version, directory or filename. */ -export type ResolveOsRuntimeDirectory = (identity: UserIdentity) => string; +export type ResolveCodexCoordinatorDatabasePath = ( + identity: UserIdentity, + canonicalCodexHome: string, +) => string; ``` WP8b implements and exports constants of both function types from `src/codex/user-identity.ts`; it does not ship declarations without bodies. +`resolveCodexCoordinatorDatabasePath` is the **one exported path resolver**. +Its private helpers may resolve/validate the runtime root, but WP11, transition +state, history, tests and cleanup consume the returned database path verbatim. +No consumer appends `opencodex`, `native-write-locks`, `v1`, uid/SID, the home +digest, or `.sqlite` a second time. Exact platform algorithm: @@ -712,7 +898,7 @@ Exact platform algorithm: SID, canonicalization, reparse, owner, or ACL failure refuses; there is no temp or ProgramData fallback. -The lock database path is +The final path returned by `resolveCodexCoordinatorDatabasePath` is `/native-write-locks/.sqlite`. POSIX directories are `0700` and files `0600`; Windows applies the required ACL to the root, database, and rollback journal. Every existing component is checked before @@ -723,7 +909,8 @@ path is a refusal, never something the resolver repairs in place. The test that matters, and the one my first version could not have failed: two child processes with different `HOME`, `USERPROFILE`, `TMPDIR`, `XDG_RUNTIME_DIR`, `TEMP`, `TMP`, and `LOCALAPPDATA` values but the same effective uid/SID and canonical -`CODEX_HOME` must resolve the same root and take the same lock. +`CODEX_HOME` must resolve the same **final database path**, take the same lock, and +read/update the same singleton transition row. ## 8. Names @@ -737,6 +924,41 @@ Audit #13. Fixed here so no phase invents a variant: | generations | `src/codex/generation.ts` | | history worker | `src/codex/history-worker.ts` | +### Writer inventory and permitted roots + +The previous rule — “every low-level writer is under `internal/` and only +`convergence.ts` may reach it” — was unsatisfiable. `history-worker.ts` must call +history writers directly after it acquires the history lock. A module guard also +cannot distinguish importing a reader from importing a writer when both symbols +live in `inject.ts` or `journal.ts`. The inventory, not a directory slogan, is the +contract: + +| Domain | Low-level writer owner | Permitted runtime roots | +|---|---|---| +| native config/profile | `src/codex/internal/native-writer.ts` | `src/codex/convergence.ts` only | +| injection journal create/mark/restore/remove | `src/codex/internal/journal-writer.ts` | `src/codex/convergence.ts` only | +| catalog, hashed/legacy backups, models cache | `src/codex/internal/catalog-writer.ts` | `src/codex/convergence.ts` only | +| history DB rows, manifest, rollout files | history write exports in `src/codex/internal/history-writer.ts` | `src/codex/history-worker.ts` only | +| transition pair and history schedule/terminal row | `src/codex/transition-state.ts` | `src/codex/convergence.ts` and `src/codex/history-worker.ts` only | +| JSON provenance ledger | `updateIntegrationRecord` in `src/codex/integration-record.ts` | `src/codex/convergence.ts` only | +| persisted OpenCodex config bytes and config generation | private writers in `src/config.ts` | exported `saveConfig`, `mutatePersistedConfig`, `saveConfigPreservingClaudeCode`, and the generation API in that same module only | + +`inject.ts` is split: observation/parsing and pure config/profile transforms stay +readable there; every export that calls `atomicWriteFile`/`unlinkSync` moves to +`internal/native-writer.ts`. `journal.ts` is split into read/validate/classify code +(`journal.ts`) and the four mutating operations in `internal/journal-writer.ts`. +The writer half may import the reader half; the reader half never imports or +re-exports the writer. `catalog.ts` likewise stops re-exporting direct writer +symbols. These splits are required before a module-level reachability assertion can +mean “reader imports are safe.” + +The contract test publishes this table as data and walks static imports, dynamic +imports, re-exports and aliases at **symbol** granularity. Every inventoried writer +must have exactly the permitted roots above, and every filesystem/SQLite mutator of +a Codex-owned artifact must appear in the inventory. `history-job.ts`, management +routes, CLI modules, `sync.ts`, `refresh.ts`, `inject.ts`, and `journal.ts` are not +permitted roots; they call convergence, dispatch a Worker, or read only. + ## 9. Baseline classes Two, not three. A provenance baseline is `absent` or `present`, and `present` @@ -756,53 +978,60 @@ Housing a finding in the wrong unit is not housing it. ## Test plan `tests/codex-integration-record.test.ts`: a v1 record with only `history` is -valid to a provenance reader and vice versa (audit #3); unknown top-level keys -survive a write; unparseable fails closed rather than resetting. Missing and -field-free v1 records initialize to `{0,null}` and persist both native fields on -first update. Generation-only/tx-only legacy records refuse. Two updates expecting -the same pair race; exactly one updates and the loser returns `conflict` without -overwriting the winner's txId or pending history schedule. +rejected as legacy transition state rather than treated as current authority; a +provenance-only record is valid. Unknown record, ledger, and individual-entry keys +survive a write, including a nested future object on one `CodexProvenanceEntry`; +unparseable fails closed rather than resetting. Missing creates only +`{version:1,provenance}` when provenance first writes. + +`tests/codex-transition-state.test.ts`: two processes use different +`OPENCODEX_HOME` values and one canonical `CODEX_HOME`, resolve one final database +path, and observe one singleton row. Two native updates expecting `{0,null}` race; +exactly one conditional UPDATE changes one row and the loser returns `conflict`. +Pause an old Worker, publish a newer pair plus pending schedule, then finish the old +Worker: its terminal UPDATE changes zero rows and cannot alter JSON or the winner's +schedule. Missing DB/table initializes only from native-clean/no-legacy state; +legacy JSON pair/schedule, residue beside a missing row, malformed row, busy DB and +unsafe path all fail closed with the specified typed outcome. `tests/codex-convergence-contract.test.ts`: every `ConvergeOutcome` variant maps to the §5 row, `busy` carries `Retry-After`, and a best-effort management caller -still returns 2xx while reporting a non-converged disposition. Compile the exported -type block with the repository TypeScript compiler so WP8b cannot regress to a +still returns 2xx while reporting a non-converged disposition. Concatenate all ten +TypeScript fences in document order, prepend the §1 `OcxConfig` import, and compile +with the repository TypeScript compiler so WP8b cannot regress to TS2304 or a bodyless TS2391 declaration. Table-drive each artifact observation and require `isApplied` only for the fully applied aggregate. `tests/codex-user-identity.test.ts`: real child processes vary every environment -home/runtime variable named in §7 and resolve one root/lock for one effective uid or -SID. POSIX activates wrong owner/mode/symlink and non-sticky `/tmp` refusal through a +home/runtime variable named in §7 and resolve one final database path for one +effective uid or SID. POSIX activates wrong owner/mode/symlink and non-sticky `/tmp` refusal through a resolver seam; Windows CI activates token/SID failure, known-folder failure, reparse, owner, and broad-ACL refusal. No case falls back to an environment directory. WP10's Worker tests pause an old Worker during traversal, commit a newer transition, -then let the old mutation finish. Its terminal CAS must conflict, the newer pending -state must survive, and the guardian must repair it. Repeat with spawn suppressed, +then let the old mutation finish. Its terminal SQLite CAS must change zero rows, the +newer pending state must survive, and the guardian must repair it. Repeat with spawn suppressed, Worker death, timeout, shutdown cancellation, unreadable/schema probes, and terminal record-write failure; every failed probe count is null and the latest transition remains durably schedulable. **The funnel must be provable, not grepped** (round 2 #2). A grep guard misses a -wrapper in the same module, a re-export, an alias and a dynamic import — and the -tree has all four today: `refreshCodexModelCatalog` wraps the catalog writers -(`src/codex/refresh.ts:40-52`), `restoreNativeCodex` wraps config/catalog/history -removal (`src/codex/inject.ts:764-794`), and `src/codex/catalog.ts:11` re-exports -the direct writers. - -So the low-level writers move into `src/codex/internal/` whose **only** permitted -importer is `convergence.ts`, and the guard test walks the module dependency -GRAPH — static imports, dynamic imports, re-exports and aliases — asserting no -other path reaches them. Reachability, not spelling. +wrapper, re-export, alias or dynamic import. The writer-inventory test above is the +enforcement surface; it permits the history Worker without opening native/catalog +writes to it. ## Accept criteria - C14 — all 16 management callers funnel through `convergeCodex`, enforced by the import guard test. - C16 — one owner, one schema; a record from any phase reads in every other. -- C17 — an A→B→A cycle between gather and commit is detected by generation, and - a parent-symlink retarget is detected by target identity. +- C17 — cooperating transition ABA is detected by the durable config/native + generations and exact txId, and a parent target that drifts once between gather + and the under-lock commit check is detected by canonical target identity. An + arbitrary filesystem A→B→A retarget that completes wholly between two checks is + explicitly not claimed. - Contributes to C15 with detect-and-repair: the latest native pair is durably - pending before spawn, a stale Worker cannot replace its record, and the guardian + pending before spawn, a stale Worker cannot replace its transition row or the + winner's schedule, and the guardian eventually repairs history. WP10 implements that protocol. Also contributes to C2/C12 (generations and the three-read admission/observation sequence). diff --git a/devlog/_plan/260804_codex_write_substrate/020_history_isolation.md b/devlog/_plan/260804_codex_write_substrate/020_history_isolation.md index bb5c82451..7d42878a6 100644 --- a/devlog/_plan/260804_codex_write_substrate/020_history_isolation.md +++ b/devlog/_plan/260804_codex_write_substrate/020_history_isolation.md @@ -15,11 +15,14 @@ The previous WP10 moved server work to a Worker but left explicit CLI work inlin claimed there was no cross-process history lock, and owned a second `integrations/codex.json` schema. Round 2 showed all three are one failure: an opposite-direction process can overtake the Worker through the unguarded files, and -the CLI can skip the only exclusion path. This rewrite consumes the contract's -sibling history-lock protocol and record section (`005_contract.md` §§1, 3, 6). - -WP10 is independently landable. WP8b already supplies the record updater, shared -types, generations, and user-identity resolver; WP9 supplies the working +the CLI can skip the only exclusion path. Round 4 found the remaining failure: a +Worker can read pair N, a native transition can write N+1, and the Worker can then +replace JSON with stale N. Separate-file read/compare/replace is not a CAS when the +writers hold different locks. This rewrite consumes the contract's sibling history +lock and canonical-`CODEX_HOME` SQLite coordinator row (`005_contract.md` §§3, 6). + +WP10 is independently landable. WP8b already supplies the coordinator-row API, +shared types, generations, and user-identity resolver; WP9 supplies the working `convergeCodex`. WP10 adds the real history lock and Worker implementation in the same commit that routes every history caller through it. It does not wait for the WP11 native lock or WP12 provenance implementation to typecheck or preserve @@ -32,20 +35,25 @@ All current-code citations and diff context below were rechecked on 2026-08-04 a IN: -- `src/codex/history-provider.ts` (MODIFY) — invocation-local retry policy, - classified internal failures, shared state-DB identity/path resolver, and a - post-probe callable while the history lock is still held. +- `src/codex/history-provider.ts` (MODIFY/SPLIT) — retain only read/probe behavior; + move every manifest, rollout, and history-DB writer into the Worker-only internal + writer module so graph reachability can distinguish a reader from a writer. +- `src/codex/internal/history-writer.ts` (NEW) — invocation-local retry policy, + classified internal failures, shared state-DB identity/path resolver, and the + exact manifest/rollout/DB mutation unit reachable only from `history-worker.ts`. - `src/codex/history-worker.ts` (NEW) — Worker entry point; applies captured homes, acquires the sibling cross-process history lock, rejects overtaken work, performs the entire history unit, probes, records, and releases. - `src/codex/history-job.ts` (NEW) — request validation, Worker IPC/watchdog/join, - history-lock target construction, capped retry scheduling, and conversion of job - facts to the contract's `CodexHistoryState`. + history-lock target construction, capped retry scheduling, fresh coordinator-row + reads after conflict, and conversion of job facts to the contract's + `CodexHistoryState`. - `src/codex/convergence.ts` (MODIFY) — add history execution behind the existing `convergeCodex`; callers still use only the contract request/result. -- `src/codex/integration-record.ts` (MODIFY only through its public updater) — no - schema change. WP10 calls `updateIntegrationRecord` to write the optional - `history` section and native expected transition atomically. +- `src/codex/transition-state.ts` (MODIFY only through its public API) — WP10 calls + `readCodexTransitionState` and `updateCodexTransitionState`; the latter conditionally + updates the pending history schedule where the native pair and `history_tx_id` still + match. It never stores that pair or schedule in `integrations/codex.json`. - `src/codex/inject.ts`, `src/codex/sync.ts` (MODIFY) — remove direct history execution paths and return their current non-history receipts to convergence. - `src/codex/history-migration-guardian.ts` (MODIFY) — schedule convergence from @@ -59,6 +67,7 @@ IN: - `src/cli/doctor.ts` (MODIFY) — combine a live read-only probe with the contract history section. - `tests/codex-history-provider.test.ts`, + `tests/codex-convergence-contract.test.ts`, `tests/history-migration-guardian.test.ts`, `tests/codex-sync-api.test.ts`, and `tests/shutdown-drain.test.ts` (MODIFY), plus `tests/codex-history-worker.test.ts`, @@ -67,10 +76,10 @@ IN: OUT: -- Any `integrations/codex.json` path, version, parser, merge algorithm, or schema. - `src/codex/integration-record.ts` and `CodexHistoryState` are owned by - `005_contract.md` §1. The former `history-convergence.ts` schema owner is deleted - from this plan. +- Any `integrations/codex.json` path, version, parser, merge algorithm, generation, + transaction id, or pending-history schedule. Those transition facts live in the + canonical-`CODEX_HOME` coordinator row owned by `005_contract.md`; the former + `history-convergence.ts` schema owner is deleted from this plan. - The claim that no cross-process history lock exists. WP10 owns its implementation now because the history unit is not safe without it. - The native lock and its namespace mechanics — WP11. The history lock is a sibling, @@ -86,13 +95,15 @@ OUT: The Worker contains the whole mutable history unit: 1. acquire the sibling cross-process history lock; -2. validate `CommitExpectation` and authority snapshot identity; +2. read the canonical-`CODEX_HOME` coordinator row and validate its pair against + `CommitExpectation` plus the authority snapshot identity; 3. optional no-op probe; 4. SQLite open, query, transaction, and close; 5. manifest read/write; 6. every rollout read, line-one patch, append, and fsync; 7. final post-probe; -8. update the contract record while still serialized; +8. conditionally update the coordinator row while still serialized, using both + generation fields in the `WHERE` clause; 9. release the history lock. Moving only `Database` calls is insufficient because the current manifest and @@ -106,6 +117,39 @@ the Worker. Explicit CLI also uses the Worker; its larger wait budget may block own terminal, but never the proxy listener and never bypasses cross-process serialization. +## Writer reachability has two permitted roots + +The failure is structural before it is behavioral: the contract's former rule that +only `convergence.ts` may reach low-level writers is impossible for an isolated +Worker. `history-worker.ts` must invoke the history mutation after the parent has +returned to the event loop. History is therefore its own permitted production root, +not an exception hidden behind an alias. + +The contract's graph/symbol guard permits exactly these history-root edges: + +```text +history-worker.ts -> internal/history-writer.ts +history-worker.ts -> transition-state.ts (pair read + history schedule/terminal CAS only) +internal/history-writer.ts -> writeBackup + -> updateSessionMeta (line-one patch + append + fsync) + -> syncCodexHistoryProviderUnsafe + -> restoreCodexHistoryProvider + -> ejectRemainingOpencodexHistory +``` + +No CLI, server, guardian, `inject.ts`, `sync.ts`, or compatibility wrapper may reach +those history writers. Tests may import the Worker entry/funnel, not the low-level +module. Today `history-provider.ts` is mixed: it exports read-only +`readLatestSessionMeta`, `readThreadFieldsFromRollout`, and +`countPendingOpencodexHistory` beside the mutating `syncCodexHistoryProvider`, +`migrateHistoryToOpenai`, and `restoreLegacyOpenaiHistory` +(`src/codex/history-provider.ts:263,348,565,701,719,749`). A module-dependency graph +cannot tell that an importer selected only a reader. This phase must split the module: +the public provider becomes read/probe-only, while the mutating entry points and their +private manifest/rollout/DB helpers move to `internal/history-writer.ts`. Only then can +the reachability test prove the history root and the separate `convergence.ts` roots +from the contract inventory without allowing every reader import to write. + ## Serializable request and response The request carries the identity of every authority the Worker must revalidate. It @@ -115,6 +159,7 @@ does not carry a mutable config object or a caller-chosen desired direction. import type { CodexHistoryState, CommitExpectation, + UserIdentity, } from "./convergence-types"; export interface HistoryWorkerRequest { @@ -134,22 +179,50 @@ export interface HistoryWorkerRequest { attempts: number; delayMs: number; skipWhenProvablyNoop: boolean; + /** Test supervisor only: pause after the named real mutation, then await resume. */ + pauseAfter?: HistoryMutationCheckpoint; env: { CODEX_HOME?: string; OPENCODEX_HOME?: string }; } +export type HistoryMutationCheckpoint = + | "manifest-write" + | "first-rollout-write" + | "database-write"; + +export interface HistoryWorkerResume { + type: "resume"; + requestId: string; + after: HistoryMutationCheckpoint; +} + +export type HistoryWorkerMessage = HistoryWorkerRequest | HistoryWorkerResume; + +export type HistoryWorkerFailureReason = NonNullable; + +export interface HistoryProbeCounts { + pendingRows: number | null; + backupEntries: number | null; +} + export type HistoryWorkerResponse = + | { + type: "checkpoint"; + requestId: string; + after: HistoryMutationCheckpoint; + } | { type: "done"; requestId: string; state: CodexHistoryState; - postProbe: PendingHistoryCount; + postProbe: HistoryProbeCounts; expectation: CommitExpectation; authoritySnapshotId: string; } | { type: "error"; requestId: string; - reason: "db-busy" | "permission" | "worker-died" | "overtaken"; + reason: HistoryWorkerFailureReason; + postProbe: HistoryProbeCounts; }; ``` @@ -164,7 +237,9 @@ ids. for different service/external/journal/provenance/intent evidence. The `CommitExpectation` rejects a transition overtaken after native commit. These are not optional diagnostics; missing fields make the message invalid and no mutation -starts. +starts. `pauseAfter` and `resume` are accepted only from the injected test supervisor; +they are not exposed through CLI, HTTP, config, or environment input. A checkpoint is +non-terminal, so the parent keeps the watchdog and join active until `done`/`error`. ## One sibling history lock @@ -172,13 +247,13 @@ starts. effective-user identity plus normalized state-DB identity. It uses a private, persistent SQLite transaction with finite async acquisition and no PID/mtime stale takeover. The Worker acquires it **inside the Worker** and holds it over manifest, -rollouts, DB, final probe, and terminal record update. +rollouts, DB, final probe, and terminal coordinator update. The native and history locks are siblings: ```text native transition: acquire native -> synchronous native commit -> release native -history transition: acquire history -> validate expectation -> mutate/probe/record -> release history +history transition: acquire history -> validate pair -> mutate/probe/conditional update -> release history ``` They are never held simultaneously. The history Worker never acquires the native @@ -192,14 +267,39 @@ B removes history, then A applies history. The request therefore carries A's `CommitExpectation`. Immediately after taking the history lock and before any probe or mutation, the -Worker reads the integration record. The job is legal only when the record still -names the transition expected by the request. If another native transition has -advanced the generation/transaction identity, the Worker returns +Worker reads the coordinator row keyed by canonical `CODEX_HOME`. The job is legal +only when both row fields equal the request's `{nativeGeneration,currentTxId}`. If +another native transition has advanced either field, the Worker returns `CodexHistoryState { status:"pending", reason:"overtaken", ... }`, performs no history write, and does **not** retry itself. The winning/newer transition owns the next convergence. -The final post-probe and record update happen before release. A clean mutation +That first read is admission, not exclusion. B may commit a newer pair after A has +already changed the manifest, a rollout, or the DB. After the final under-lock probe, +A executes one SQLite conditional update of its result and schedule: + +```sql +UPDATE codex_transition_state + SET history_status = ?, history_reason = ?, history_attempts = ?, + history_next_retry_at = ?, history_tx_id = ?, + history_pending_rows = ?, history_backup_entries = ?, updated_at = ? + WHERE singleton = 1 + AND native_generation = ? + AND current_tx_id IS ? + AND history_tx_id IS ?; +``` + +The coordinator database path already encodes effective user plus canonical +`CODEX_HOME`; the row is deliberately a singleton, not one row per +`OPENCODEX_HOME`. `updateCodexTransitionState(expected, next)` executes the statement +above. Its `kind:"updated"` result means exactly one changed row published A's +result; the implementation maps zero changed rows to `kind:"conflict"`. Conflict +means A was overtaken: it MUST NOT write JSON, MUST NOT overwrite or clear the newer +row's pending schedule, returns `pending/overtaken`, releases the history lock, and joins. +The parent then asks the guardian to read the current coordinator row and immediately +arm/retain the winner's schedule; it never retries A's losing transaction. + +The final post-probe and conditional row update happen before release. A clean mutation followed by an unlocked probe is not evidence: another process could change rows or the manifest in between. For target `openai`, `converged` requires a non-failed probe with `pendingRows === 0` and `backupEntries === 0`; manifest absence or a @@ -214,13 +314,20 @@ the lock and the older one is rejected by its expectation. Outcome order: -- valid `done` + clean under-lock post-probe -> contract `converged` state; +- valid `done` + clean under-lock post-probe + one-row conditional update -> contract + `converged` state; - SQLite/history-lock busy -> `pending/db-busy` with next retry; - permission/refusal -> `blocked/permission`; -- expectation/snapshot mismatch -> `pending/overtaken`, no self-retry; -- `worker.onerror`, malformed terminal message, early close, or watchdog -> +- unreadable data -> `unknown/unreadable` with both probe counts null; +- readable unsupported shape -> `unknown/schema` with both probe counts null; +- watchdog -> `unknown/timeout`, not `worker-died`; +- shutdown cancellation -> `unknown/shutdown-cancelled`, join, then drain; +- `worker.onerror`, malformed terminal message, or early close -> `unknown/worker-died`; -- shutdown cancellation -> persist non-converged state, join, then drain. +- initial pair/snapshot mismatch or zero-row terminal update -> + `pending/overtaken`, no self-retry; +- coordinator update failure -> returned `unknown/record-write-failed`; the existing + pending row is left intact for guardian repair. The Worker closes in `finally`; the parent still waits for `close`/join using the repository's existing discipline (`src/storage/worker-lifecycle.ts:150-209`). A @@ -256,27 +363,23 @@ Apply `busyTimeoutMs` to both apply and restore database opens. Keep hard errors throwing inside the Worker so its boundary can classify them once; do not turn programming/data corruption into `db-busy`. -## Durable state consumes the contract record +## Durable state consumes the coordinator row Delete the former “Location and exact shape” JSON and the planned -`src/codex/history-convergence.ts`. The path, top-level version, extension policy, -and section schema belong to `005_contract.md` §1. +`src/codex/history-convergence.ts`. The transition pair and pending history schedule +belong to the SQLite coordinator row keyed by canonical `CODEX_HOME`, not to an +`OPENCODEX_HOME` record. -Both `history-worker.ts` and `history-job.ts` import: - -```ts -import type { CodexHistoryState } from "./convergence-types"; -import { - readIntegrationRecord, - updateIntegrationRecord, -} from "./integration-record"; -``` +Both `history-worker.ts` and `history-job.ts` consume the contract-owned coordinator +API from `src/codex/transition-state.ts`; neither owns SQL or a second row shape. The +Worker calls `readCodexTransitionState` before traversal and +`updateCodexTransitionState(expected, next)` after its post-probe. The parent/guardian +uses the same reader to arm the current schedule after conflict. -They never parse or atomically replace `integrations/codex.json` themselves. A -state transition is one `updateIntegrationRecord(record => ({ ...record, history: -next }))`; unknown keys and the provenance section survive. Corrupt/unparseable -records fail closed. `txId` links the state to the native transition and -`nextRetryAt:null` means only “no timer armed now,” never “never again.” +They never parse, write, or atomically replace `integrations/codex.json`. A terminal +history transition is one row update conditioned on canonical home plus the exact +pair. `txId` links the state to the native transition and `nextRetryAt:null` means +only “no timer armed now,” never “never again.” The durable contract has no per-state-DB schema invented here. If multiple state DBs need internal scheduling metadata, it remains an in-memory/job-private map; @@ -297,9 +400,9 @@ delay(attempt) = min(MAX_HISTORY_RETRY_MS, BASE_HISTORY_RETRY_MS * 2^min(attempt, BACKOFF_EXPONENT_CAP)) ``` -It schedules at most one timer and one Worker per current `txId`. It may back off +It schedules at most one timer and one Worker per current coordinator-row `txId`. It may back off to the cap but never exhausts into a permanent state. Startup re-arms any unresolved -record whose timer was lost. A successful convergence clears the timer. An +coordinator row whose timer was lost. A successful convergence clears the timer. An `overtaken` job does not retry the losing transition; it schedules one observation of the current generation so the winner owns work. @@ -317,6 +420,7 @@ Mode is already in `ConvergeRequest`. -const history = syncCodexHistoryProvider("openai", ...); +const outcome = await convergeCodex({ + action: "converge", ++ scope: "full", + reason: "cli", + mode: "explicit", + deadlineMs: EXPLICIT_CODEX_CONVERGENCE_DEADLINE_MS, @@ -347,9 +451,9 @@ inventing an inline escape hatch. ## Durable read surface -`GET /api/codex/history` may expose the contract record's `history` section through -an authenticated read-only route. It imports `readIntegrationRecord`; it does not -define a second state type. +`GET /api/codex/history` may expose the coordinator row's `history` projection +through an authenticated read-only route. It calls `readCodexTransitionState`; it +does not define a second state type or consult the non-CAS integration JSON. `POST /api/sync` is not redefined here. It already calls `convergeCodex` and `toSyncResponse` after WP9 (`005_contract.md` §5). WP10 only ensures the resulting @@ -359,23 +463,34 @@ unknown rather than zero-looking success. ## Key diffs -### Worker owns lock, mutation, post-probe, and record +### Worker owns lock, mutation, post-probe, and conditional row update ```diff +self.onmessage = async (event: MessageEvent) => { -+ const request = parseHistoryWorkerRequest(event.data); ++ const message = parseHistoryWorkerMessage(event.data); ++ if (message.type === "resume") return resumeHistoryCheckpoint(message); ++ const request = message; + applyCapturedHomes(request.env); + const lock = await acquireHistoryLock(request.lockIdentity, requestDeadline(request)); + if (lock.status !== "acquired") return postHistoryBusy(request, lock); + try { -+ const current = readIntegrationRecord(); -+ if (!expectationStillCurrent(current, request.expectation, request.authoritySnapshotId)) { ++ const expected = { ++ nativeGeneration: request.expectation.nativeAfter, ++ currentTxId: request.expectation.txId, ++ }; ++ const admitted = readCodexTransitionState(); ++ if (!expectationStillCurrent(admitted, expected, request.authoritySnapshotId)) { + return postOvertaken(request); + } + const result = syncCodexHistoryProvider(request.targetProvider, request.stateDbPath, request.backupPath, policy(request)); + const postProbe = countPendingOpencodexHistory(request.stateDbPath, request.backupPath); + const state = classifyHistoryState(result, postProbe, request.expectation.txId); -+ updateIntegrationRecord(record => ({ ...record, history: state })); ++ const update = updateCodexTransitionState(expected, { ++ ...expected, ++ history: state, ++ }); ++ if (update.kind === "conflict") return postOvertaken(request, postProbe); ++ if (update.kind === "unavailable") return postRecordWriteFailed(request, postProbe); + self.postMessage({ type: "done", requestId: request.requestId, state, postProbe, expectation: request.expectation, authoritySnapshotId: request.authoritySnapshotId }); + } finally { + lock.release(); @@ -406,17 +521,26 @@ API, no caller can retain it across unrelated work. ### Opposite-direction cross-process serialization 1. Seed production-shaped DB, manifest, and rollouts in isolated homes. -2. Process A converges ON and pauses after acquiring the real history lock. -3. Process B converges OFF. Assert B cannot mutate manifest, rollout, or DB while A - holds the lock. -4. Let B win the newer native `CommitExpectation`; release A. Assert A is rejected - as `overtaken` before its first history write and B alone produces final OFF - history. -5. Reverse direction/order and repeat. Final history must match the highest native - generation, not Worker scheduling order. - -This is real two-process SQLite/filesystem behavior. A same-process flight or two -connections without rollout sentinels does not satisfy C15. +2. Process A enters production `convergeCodex({scope:"full"})`; its real Worker + requests `pauseAfter:"first-rollout-write"`. The Worker performs the manifest and + first rollout mutations, posts `checkpoint {requestId, after}`, and waits for a + matching `resume` IPC message while still holding the history lock. +3. Process B converges OFF and commits the newer coordinator pair while A is paused. + B's history Worker waits on the history lock; native pair advancement does not. +4. Resume A. Its remaining traversal and post-probe complete, but its terminal + conditional row update affects zero rows. Assert A returns `pending/overtaken`, + never touches the newer pending schedule, releases/joins, and causes the guardian + to arm B from the current row. +5. Let B acquire history and repair every manifest, rollout, and DB sentinel. Reverse + direction/order and repeat. Final history must match the highest native generation, + not Worker scheduling order. + +The checkpoint is deterministic because it is acknowledged only after the real +writer reports a completed surface mutation, and resume is keyed by `requestId`. +There is no alternate provider stub or direct test-only mutation path: both processes +enter the production convergence/job/Worker protocol. A same-process flight, a pause +before traversal, or two connections without manifest/rollout/DB sentinels does not +satisfy C15. ### CLI contention @@ -426,7 +550,7 @@ connections without rollout sentinels does not satisfy C15. - In parallel trigger automatic server convergence; assert listener health/data plane progress while both processes contend. - Release, join both Workers, and prove one serialized winner. The test inspects the - integration record through its owner, not a WP10 parser. + transition row through `readCodexTransitionState`, not a WP10 parser. ### Post-probe under lock @@ -443,7 +567,8 @@ connections without rollout sentinels does not satisfy C15. state exists. - Restart/module reload re-arms unresolved state. - Worker error, malformed response, early close, watchdog, cancellation, and final - record-write failure remain non-converged and join exactly once. + coordinator-row write failure retain their distinct contract reasons, carry nullable + probe counts, remain non-converged, and join exactly once. - An overtaken transition does not retry itself. ### Measured responsiveness — C3 @@ -479,8 +604,10 @@ or `ocx ensure`; port 10100 remains untouched. `CodexHistoryState`, retried with capped non-permanent backoff, and never collapsed into success. Clean post-probe occurs under the history lock. - **C15** — opposite-direction processes serialize manifest, rollouts, DB, probe, - and record update; `CommitExpectation` prevents overtaking. + and terminal coordinator update; the pair-conditioned row update detects an + overtake even after the stale Worker has mutated a history surface, preserves the + winner's pending schedule, and drives repair. - Explicit CLI and automatic server/startup/retry callers all enter through `convergeCodex` and the same sibling history lock. No inline escape hatch remains. -- **N2** — WP10 imports the WP8b record/types and extends WP9's working funnel. Its +- **N2** — WP10 imports the WP8b coordinator/types and extends WP9's working funnel. Its commit typechecks and preserves behavior without any WP11/WP12 placeholder. diff --git a/devlog/_plan/260804_codex_write_substrate/040_ownership_convergence.md b/devlog/_plan/260804_codex_write_substrate/040_ownership_convergence.md index f6914cb24..beeb9f193 100644 --- a/devlog/_plan/260804_codex_write_substrate/040_ownership_convergence.md +++ b/devlog/_plan/260804_codex_write_substrate/040_ownership_convergence.md @@ -13,11 +13,11 @@ check also does not authorize overwriting a `config.toml` whose effective `model_provider` is now external. WP9-WP11 already provide the working `convergeCodex` funnel, catalog split, -history protocol, generations, integration-record owner, and native lock. WP12 -completes the mechanisms behind that funnel: tri-state service authority, -file-backed intent, journal/provenance admission, restoration, and observed-state -inspection. It does **not** add another record module, another convergence module, -another route mapping, or another public result union. +history protocol, CODEX_HOME-keyed coordinator row, integration-record owner, and +native lock. WP12 completes the mechanisms behind that funnel: tri-state service +authority, file-backed intent, journal/provenance admission, restoration, and +observed-state inspection. It does **not** add another record module, another +convergence module, another route mapping, or another public result union. The prior plan named `write-lock.ts`, created `ownership-convergence.ts`, redefined `integrations/codex.json`, and exported `convergeCodexToPersistedIntent`. Those are @@ -30,7 +30,7 @@ in that commit, and typechecks/preserves behavior without a future phase. WP13 m re-prove composition; it is not required to make WP12 correct. All current-code citations and diff context below were rechecked on 2026-08-04 at -`2d5e080dea3e7000bf2111b381c7c1a3c4f5fb11`. +`bd1065a85c4981a1662e3676243d7de3b4ed9c81`. ## IN / OUT @@ -39,12 +39,13 @@ IN: | Path | Change | Why | |---|---|---| | `src/types.ts` | MODIFY | Add `clientIntegrations.codex?: boolean`; absent means desired ON. | -| `src/config.ts` | MODIFY | Parse the extension-safe object; own config generation bumps and authoritative `AdmissionSnapshot` reads. | +| `src/config.ts` | MODIFY | Parse the extension-safe object; implement `readConfigGeneration`/`bumpConfigGeneration`; own authoritative config inputs to `AdmissionSnapshot`. | | `src/service.ts` | MODIFY | Preserve all service registration/mirror evidence instead of skipping corrupt/unreadable rows. | | `src/integrations/native/ownership-preflight.ts` | MODIFY | Tri-state read-only service-home authority; only owned permits native mutation. | | `src/codex/convergence.ts` | MODIFY | Complete admission, provenance, restore, observation, and lifecycle routing behind the contract entry point. | | `src/codex/convergence-types.ts` | IMPORT ONLY | Consume `AdmissionSnapshot`, `CodexObservedState`, `ConvergeOutcome`, `CodexProvenanceLedger`, and section types; no WP12 union. | -| `src/codex/integration-record.ts` | USE/MODIFY THROUGH OWNER API | Read/update provenance and native transition through the contract owner; no path/schema/parser here. | +| `src/codex/integration-record.ts` | USE/MODIFY THROUGH OWNER API | Read/update provenance and extension keys through the contract owner; no transition pair, history state/schedule, path/schema/parser, or parallel merge here. | +| `src/codex/transition-state.ts` | CONSUME | Use `readCodexTransitionState`, `beginCodexTransition`, and `updateCodexHistoryTransition` for the canonical-CODEX_HOME pair and history state/schedule. | | `src/codex/codex-write-lock.ts` | CONSUME | Correct WP11 module name; no lock redesign. | | `src/codex/journal.ts` | MODIFY | Read-only typed inspection; authorized recovery only inside convergence. | | `src/codex/inject.ts` | MODIFY | Receipt-gated internal apply/restore mechanics; remove filename-based deletion authority. | @@ -132,27 +133,91 @@ stop, uninstall, retry, and observe uses this sequence. No caller selects a subs legal only when no residue needs provenance proof; corrupt/lost/conflicting provenance refuses. 6. Authoritatively read persisted config, config generation, intent, and ownership; - return one exact `AdmissionSnapshot`: + return the contract's complete `AdmissionSnapshot`. The five-field object printed + here before round 4 was wrong: it silently dropped the external-provider veto, + target identity, journal/provenance identity, and the authority ID, so WP11 could + not compare the authority it claimed to admit. ```ts +import type { + AdmissionSnapshot, + ConvergeCodex, + ConvergeOutcome, + ConvergeRequest, +} from "./convergence-types"; + +declare const configRead: Pick< + AdmissionSnapshot, + "config" | "configDigest" | "intent" +> & { readonly generation: { readonly value: AdmissionSnapshot["generation"] } }; +declare const serviceRead: Pick; +declare const routingRead: Pick; +declare const targetRead: AdmissionSnapshot["canonicalTargets"]; +declare const journalRead: { readonly identity: AdmissionSnapshot["journalIdentity"] }; +declare const provenanceRead: { readonly identity: AdmissionSnapshot["provenanceIdentity"] }; +declare const authorityRead: { readonly id: AdmissionSnapshot["authoritySnapshotId"] }; + const admission: AdmissionSnapshot = { - config: diagnostics.config, - configDigest, - intent, - generation: configGeneration.value, - ownership, + config: configRead.config, + configDigest: configRead.configDigest, + intent: configRead.intent, + generation: configRead.generation.value, + ownership: serviceRead.ownership, + externalProvider: routingRead.externalProvider, + canonicalTargets: { + codexHome: targetRead.codexHome, + opencodexHome: targetRead.opencodexHome, + config: targetRead.config, + profile: targetRead.profile, + catalog: targetRead.catalog, + cache: targetRead.cache, + journal: targetRead.journal, + integrationRecord: targetRead.integrationRecord, + catalogBackups: targetRead.catalogBackups, + historyDb: targetRead.historyDb, + historyManifest: targetRead.historyManifest, + historyRollouts: targetRead.historyRollouts, + }, + journalIdentity: journalRead.identity, + provenanceIdentity: provenanceRead.identity, + authoritySnapshotId: authorityRead.id, }; ``` +This object is not a WP12 approximation of the type. `AdmissionSnapshot` is imported +from `convergence-types.ts`, and the contract compilation fixture rejects either a +missing field or a locally invented replacement. + +Concrete read/API ownership is fixed here so an implementer cannot fill the object +from a resident server cache: + +| Field | Reader/API owner | +|---|---| +| `config`, `configDigest`, `intent`, `generation` | `src/config.ts` `readConfigDiagnostics()` plus `readConfigGeneration()`; digest is over the exact persisted bytes represented by that diagnostic result. | +| `ownership` | `src/integrations/native/ownership-preflight.ts` `inspectNativeCodexOwnership()`, projected only after `src/service.ts` has inspected every registration/mirror. | +| `externalProvider` | `src/codex/inject.ts` `externalCodexModelProvider()` over the exact persisted config bytes above. | +| `canonicalTargets` | `src/codex/convergence.ts` `resolveCanonicalCodexTargets()`; it composes the canonical path owners once, without creating a target. | +| `journalIdentity` | `src/codex/journal.ts` `inspectJournal()`; identity hashes the preserved envelope plus liveness verdict, not merely its path. | +| `provenanceIdentity` | `src/codex/integration-record.ts` `readIntegrationRecord()`; identity covers the validated provenance section and its unknown-key-preserving envelope. | +| `authoritySnapshotId` | `src/codex/convergence.ts` `hashAdmissionAuthority()` over the canonical encoding of every preceding authority field. | + +The native `{nativeGeneration,currentTxId}` pair is deliberately not smuggled into +`provenanceIdentity`. WP12 reads it separately through +`src/codex/transition-state.ts` `readCodexTransitionState()`. That owner opens the +final path from `resolveCodexCoordinatorDatabasePath(identity, +canonicalCodexHome)`, so the pair belongs to the CODEX_HOME-keyed coordinator row, +not to `integrations/codex.json`. + 7. If intent is ON, WP9 gather receives `admission.config` — **that exact object**. OFF does not gather. 8. Call WP11 with `admitted: admission`. Under native->config coordination, authoritatively re-read steps 1-6 into a second `AdmissionSnapshot` and compare - digest, generation, intent, ownership, canonical targets, journal identity, and - provenance identity. + digest, generation, intent, ownership, external provider, canonical targets, + journal identity, provenance identity, and the recomputed authority snapshot ID. 9. Recover an authorized dead journal, establish baselines, commit apply/remove, - write the expected native generation/`txId`, and inspect observed state inside - the coordinated section. Release before logging/HTTP shaping. + conditionally write the expected native generation/`txId` and pending history + schedule in the CODEX_HOME-keyed coordinator row, and inspect observed state + inside the coordinated section. Release before logging/HTTP shaping. 10. Run WP10 history afterward under its sibling lock with the same `CommitExpectation` and authority snapshot identity; stale jobs are rejected. @@ -203,23 +268,48 @@ External is projected through the contract's `refused` authority/result. It is n Delete the former `CodexIntegrationRecordV1`, transaction, artifact, ledger-row, and restore unions. Import the section types: -```ts -import type { - CodexProvenanceEntry, - CodexProvenanceLedger, -} from "./convergence-types"; -import { - readIntegrationRecord, - updateIntegrationRecord, -} from "./integration-record"; +```diff ++import type { ++ CodexProvenanceEntry, ++ CodexProvenanceLedger, ++} from "./convergence-types"; ++import { ++ readIntegrationRecord, ++ updateIntegrationRecord, ++} from "./integration-record"; ``` -WP12 writes only `record.provenance` through `updateIntegrationRecord`; history, -generation, unknown top-level keys, and unknown section keys survive. Unparseable -or wrong-version record fails closed. No WP12 code joins +WP12 writes only `record.provenance` through `updateIntegrationRecord`. The JSON +record keeps exactly `version`, the provenance ledger, and unknown extension keys +at the record, ledger, and entry levels. It does **not** keep +`nativeGeneration`, `currentTxId`, a pending/running history schedule, retry ownership, +or the next due time. Putting those fields here was wrong: two different coordinators +could each serialize their own read/replace and still overwrite one another. + +Unparseable or wrong-version JSON fails closed. No WP12 code joins `getConfigDir()/integrations/codex.json`, validates the top-level schema, or runs a parallel read/merge/write (`005_contract.md` §1). +### Transition pair and schedule come from the coordinator row + +The authoritative transition read is `readCodexTransitionState()` from +`src/codex/transition-state.ts`. It opens the exact SQLite path returned by +`resolveCodexCoordinatorDatabasePath(identity, canonicalCodexHome)`; consumers +append no identity, version, directory, or filename. Its singleton row contains the +native generation, current transaction ID, complete `CodexHistoryState`, and +`historySchedule {direction,authoritySnapshotId}`. `beginCodexTransition` advances +the pair and installs that transition's pending schedule in one `UPDATE ... WHERE +native_generation = ? AND current_tx_id IS ?`; `updateCodexHistoryTransition` +conditionally claims/completes/reschedules that same row and additionally matches +`history_tx_id`. A zero-row update writes nothing. + +That row is the transition/scheduling authority. `readIntegrationRecord()` supplies +only provenance needed to prove an artifact baseline/post-image. Observation joins +the coordinator pair/schedule with the JSON provenance at read time; neither source +is copied into the other. Two `OPENCODEX_HOME` installations sharing one canonical +Codex home therefore see one pair and one pending owner instead of two generation +counters that both call themselves current. + ## When provenance entries are written 1. After pre-lock admission and authoritative under-lock re-read, read every @@ -229,7 +319,8 @@ parallel read/merge/write (`005_contract.md` §1). contract baselines. 3. Commit one artifact. 4. Read current bytes after the successful write and persist its `postImage` hash. -5. Repeat; then write the expected native generation/`txId` and observe. +5. Repeat; then conditionally write the expected native generation/`txId` plus the + pending history schedule to the coordinator row and observe both stores. A filename, marker, slug, mtime, backup name, or location is not creation proof. A crash after native write but before `postImage` leaves unknown provenance and cannot @@ -266,38 +357,63 @@ observations. C10 is therefore narrowed to current-byte drift detection and safe restoration from current evidence. No test or documentation may claim detection of an edit-and-revert ABA that leaves identical bytes. -## Lost/corrupt ledger operator recovery — carried #10 +## Lost/corrupt ledger operator recovery — carried #10, round-4 E1 Automatic convergence always refuses lost/corrupt provenance and preserves native bytes. “Start a fresh record” is not recovery; it silently turns unknown artifacts -into owned artifacts. +into owned artifacts. The prior adoption design did exactly that to a worse input: it +called whatever bytes happened to be present “native.” After a crash mid-apply those +bytes may still route through OpenCodex, so OFF would later restore the routed bytes +as the baseline forever. That mechanism was wrong. -**INFERRED operator-recovery UX:** provide one explicit operator-only adoption flow -in the existing CLI, separate from normal convergence: +**INFERRED operator-recovery UX:** keep the explicit operator-only command, but make +adoption a read-only proof followed by a record write: ```text ocx restore --adopt-current-codex-baseline ``` The flag is rejected in service/agent-driven/automatic contexts and requires an -interactive confirmation naming the canonical Codex home and that current bytes -will become the baseline. It performs read-only service/external/journal checks -first, requires the proxy stopped and no live journal writer, asks the -`integration-record.ts` owner to atomically move an unreadable record to a -timestamped sibling quarantine (preserving its bytes), then uses -`updateIntegrationRecord` against the now-absent canonical path to create a -valid record whose `present` baselines are the exact current bytes. A lost record -has no quarantine source but follows the same exact-current-baseline validation. It -changes no Codex native artifact. - -If even one target is unreadable/ambiguous, adoption aborts before replacing the -record. A subsequent explicit `convergeCodex` performs the requested apply/remove -from the adopted baseline. The command prints the quarantine path and resulting -`txId`; automatic callers receive only the provenance refusal and operator -instruction. Tests never auto-confirm this action. - -This is a recovery path, not a second convergence entry point: adoption establishes -authority evidence; all native mutation still goes through `convergeCodex`. +interactive confirmation naming the canonical Codex home. It requires the proxy +stopped, owned service authority, no external provider, no journal envelope, and a +single read-only classification of every target as `native-clean | ocx-residue | +ambiguous`. **Only all `native-clean` may adopt.** `ocx-residue` and `ambiguous` both +abort before quarantine or record replacement; the command prints the exact surface +and evidence and asks the operator to clean/inspect it outside this flow. There is no +automatic “salvage by filename” fallback in WP12. + +The ledger is unavailable here, so positive residue proof comes only from the +artifact's own structure: + +| Surface | Structural residue proof without the ledger | Native-clean gate | +|---|---|---| +| `config.toml` | The `# Auto-injected by opencodex` marker immediately owns a root `openai_base_url` (`src/codex/injected-marker.ts:53-60`); or the same marker immediately precedes the complete legacy table grammar emitted at `src/codex/inject.ts:119-134` and the root selector is `model_provider = "opencodex"`; managed subagent values are owned only by the exact markers in `src/codex/subagent-defaults.ts:10-11`. The recovery detector is stricter than the ordinary routing predicate at `src/codex/injected-marker.ts:68-71`, which does not by itself prove legacy creation. | TOML parses; none of those marker/value pairs or the exact legacy selector+owned-table structure exists; routing class is `native`. A bare local URL, malformed marker adjacency, or unowned `opencodex` table is ambiguous, not clean. | +| generated profile | File content matches one complete generated profile grammar: the OpenCodex banner plus its routing keys/provider block; `src/codex/inject.ts:436-460` is the generator. | Path absent, or readable content does not select OpenCodex and contains no generated banner. The basename `opencodex.config.toml` alone proves nothing; a partial signature is ambiguous. | +| catalog/cache | A row has both a namespaced slug and the stable `Routed via opencodex -> ` description signature used by `isOcxAuthoredRoutedEntry` (`src/codex/catalog/sync.ts:334-347`). | Every readable row lacks that paired signature and no active catalog/cache points to an OpenCodex-only routed slug. A filename, `owned_by`, or `comp_hash` alone is never proof. | +| history DB/manifest/rollouts | A valid manifest pre-image and its exact DB row/rollout identity jointly prove the OpenCodex transform. | The DB, manifest, and rollouts parse and the joint probe finds no owned transform. A lone `model_provider = opencodex`, path, or manifest entry is ambiguous because it cannot reconstruct the pre-image by itself. | +| journal/backups/partial transaction | Any journal envelope or validated partial-transaction marker is residue; a valid backup is evidence to inspect, not authority to restore. | Journal is absent and every backup/partial artifact is either structurally foreign or absent. An unreadable envelope, backup named by convention only, missing companion, or hashless partial is ambiguous. | + +The ASCII `->` above is the documentation spelling; implementation compares the +exact Unicode prefix already emitted at `src/codex/catalog/sync.ts:283,346`. + +Once the same observation is all native-clean, `integration-record.ts` atomically +quarantines an unreadable JSON record to a timestamped sibling, preserving its bytes, +and `updateIntegrationRecord` creates only exact current `present`/`absent` +provenance baselines. A lost record has no quarantine source. Adoption changes no +Codex artifact. If `readCodexTransitionState()` returns a ready row, adoption +preserves its pair and complete history state. If the row is missing/ +`legacy-ambiguous`, only the same all-native-clean proof may initialize the contract +`{0,null}` row with `history.status:"unknown"`; it never imports a positive pair from +OPENCODEX_HOME-local legacy JSON. A subsequent explicit `convergeCodex` performs +apply/remove from the verified clean baseline. + +If any target changes between classification and JSON replacement, target identity +or digest validation fails and adoption writes nothing. The command prints the +quarantine path and adopted provenance identity, never a newly invented `txId`. +Tests never auto-confirm this action. + +This is a recovery path, not a second native mutation entry point: adoption establishes +provenance only; every Codex artifact mutation still goes through `convergeCodex`. ## Journal inspection and recovery @@ -332,9 +448,11 @@ returns the contract's `CodexObservedState`; `convergeCodex` returns the contrac `ConvergeOutcome`. The observer reads service/external authority, managed config fragments, profile, -catalog/cache and routed slugs, journal/liveness, provenance/generation/tx identity, -history DB/manifest/rollouts, backups, and partial transaction residue. It performs -no repair. +catalog/cache and routed slugs, journal/liveness, JSON provenance, the coordinator +row's generation/tx pair and history schedule, history DB/manifest/rollouts, backups, +and partial transaction residue. It performs no repair. Only the coordinator row +explains history status or may own/clear a pending schedule; JSON provenance cannot +override a newer coordinator transaction. Desired ON converges only when the contract observer says applied. Desired OFF converges only when residue is removed/restored. External/refused/partial remains @@ -391,6 +509,7 @@ Delete WP12's status logic and custom result adapter. Current route - return jsonResponse(result, result.ok ? 200 : 500); + const outcome = await convergeCodex({ + action: "converge", ++ scope: "full", + reason: "api-sync", + mode: "explicit", + deadlineMs: EXPLICIT_CODEX_CONVERGENCE_DEADLINE_MS, @@ -409,11 +528,27 @@ Status, body, and `Retry-After` belong only to or WP12-specific request/result type. ```ts -export async function convergeCodex( +declare const convergeCodexImpl: ( request: ConvergeRequest, -): Promise; +) => Promise; +export const convergeCodex: ConvergeCodex = request => convergeCodexImpl(request); +declare const EXPLICIT_CODEX_CONVERGENCE_DEADLINE_MS: number; +void convergeCodex({ + action: "converge", + scope: "full", + reason: "api-sync", + mode: "explicit", + deadlineMs: EXPLICIT_CODEX_CONVERGENCE_DEADLINE_MS, +}); ``` +The bodyless declaration printed here before round 4 was wrong and compiled as +TS2391. WP12 imports the contract's `ConvergeCodex` alias and assigns the real +implementation to it; the phase doc does not redeclare the alias or declare a +function without a body. `convergeCodexImpl` above is the compile-fixture name for +the existing implementation body that WP12 modifies. The compile-only call is the +`/api/sync` request above; omitting `scope:"full"` reproduces TS2345. + Callers say when/reason/mode/deadline. They never supply desired state, ownership, journal verdict, provenance verdict, or apply/remove direction. `action:"observe"` is the one read-only public operation; internal admission/observer helpers stay @@ -456,8 +591,11 @@ All tests use temporary homes, real contract record owner, port `0`, and product — detection, not prevention. 3. Exhaust `deadlineMs`; assert typed unresolved outcome and no unbounded loop. 4. Native expected generation with another `txId` at the same number is - interference. -5. Stale history `CommitExpectation` is rejected before mutation. + interference in the coordinator row; JSON provenance remains byte-identical. +5. Stale history `CommitExpectation` is rejected before mutation, and its terminal + conditional row update cannot clear the newer pending schedule. +6. Two distinct `OPENCODEX_HOME` processes sharing one canonical `CODEX_HOME` + observe one coordinator pair; exactly one expected-row update wins. ### Provenance/restoration/recovery @@ -471,11 +609,20 @@ All tests use temporary homes, real contract record owner, port `0`, and product 5. Crash between native write and post-image record; restart preserves/refuses. 6. Lost record and corrupt record: every automatic/normal explicit convergence refuses and preserves bytes. -7. Operator adoption with no confirmation does nothing; confirmed isolated CLI - flow quarantines the bad record, writes exact current present baselines through - the owner, changes no native bytes, then normal `convergeCodex` succeeds. -8. Adoption aborts atomically on one unreadable target, external provider, live - writer, running proxy, or noninteractive/agent-driven invocation. +7. Operator adoption with no confirmation does nothing. Confirmed adoption with + one marker-owned config fragment, exact generated profile, routed catalog row, + jointly owned history transform, or journal envelope refuses and changes neither + native bytes nor JSON. +8. A markerless/basename-only/partial signature is `ambiguous`, not native-clean; + adoption refuses instead of guessing. Table-drive every structural-proof row + above, including exact clean negatives. +9. Only an all-native-clean observation may quarantine a corrupt JSON record and + write exact current baselines. Assert native bytes and the coordinator pair/ + pending schedule are unchanged; then normal `convergeCodex` succeeds. +10. Missing transition row + all-native-clean initializes only `{0,null}` with + unknown history; a legacy positive JSON pair is never imported. +11. Adoption aborts atomically on one unreadable or changed target, external + provider, live writer, running proxy, or noninteractive/agent-driven invocation. ### Observed state and fresh intent @@ -509,6 +656,16 @@ bun run lint:gui bun run privacy:scan ``` +Round-4 document compile gate: extract every `ts` fence above in document order, +concatenate them, prepend `import type { OcxConfig } from "../types";`, and resolve +`AdmissionSnapshot`, `ConvergeRequest`, and `ConvergeCodex` from their exact +`005_contract.md` definitions (the imported outcome remains opaque because neither +fixture inspects an outcome field). +On 2026-08-04 the installed `bun x tsc --noEmit --strict --skipLibCheck +--moduleResolution bundler --module esnext --target es2022` exited **0 with zero +diagnostics**. The request fixture includes `scope:"full"`; the function surface is +the `ConvergeCodex` alias, not a bodyless declaration. + Live proof is the in-process server/subprocess test bound to port `0` with isolated homes. Evidence names one server PID, two config-writer PIDs, prevention/deferred interference traces, recovery quarantine/record hashes, and observed ON/OFF results. @@ -523,8 +680,8 @@ live proxy on 10100. - **C9** — external provider remains a separate veto and preserves all bytes. - **C10 (narrowed)** — two contract baseline classes only. Matching current post-images restore; current-byte drift preserves/reports. No hash claims to prove - absence of edit-and-revert. Lost/corrupt ledger has an explicit, confirmed, - non-mutating adoption path; automatic behavior refuses. + absence of edit-and-revert. Lost/corrupt ledger adoption requires a complete + verified native-clean observation; structural residue or ambiguity refuses. - **C11** — observed state is the contract `CodexObservedState`; unchanged intent still converges and re-observes. - **C12** — the same running server honors subprocess OFF then ON using a pre-gather @@ -533,6 +690,9 @@ live proxy on 10100. - `/api/sync` calls only `convergeCodex` + `toSyncResponse`; no WP12 status/header owner exists. - There is one convergence entry point and one shared result family. +- Native pair/pending schedule authority is the canonical-CODEX_HOME coordinator + row. JSON retains only version, provenance, and extension keys, never transition + or scheduling authority. - **N2** — WP12 rewires all remaining callers and passes its own typecheck/tests in the same commit. WP13 adds composed proof, not missing implementation. diff --git a/devlog/_plan/260804_codex_write_substrate/050_composed_acceptance.md b/devlog/_plan/260804_codex_write_substrate/050_composed_acceptance.md index f8d7dbdb6..2ed509656 100644 --- a/devlog/_plan/260804_codex_write_substrate/050_composed_acceptance.md +++ b/devlog/_plan/260804_codex_write_substrate/050_composed_acceptance.md @@ -11,7 +11,8 @@ call sites invoke a swallowed best-effort catalog writer (`src/codex/history-migration-guardian.ts:34-35,87-90`). A phase-local green test can miss every one of those seams. -This document specifies the one suite that is allowed to claim C1-C18 for the +This document specifies the one acceptance program, split across a workstation-safe +suite and a disposable-host service job, that is allowed to claim C1-C18 for the composed system. All citations were rechecked on 2026-08-04 at `ee182744af6958478523fb97ece6af2efb63b082`. The substrate modules named by `005_contract.md` do not exist at that revision; the current red signatures below @@ -22,13 +23,15 @@ an unimplemented suite was run. IN: a future `tests/codex-composed-acceptance.test.ts` plus narrowly named child fixtures under `tests/helpers/`; the production CLI, server, management routes, -service dispatcher, convergence entry point, real filesystem, real Bun Workers, -and real SQLite files. +convergence entry point, real filesystem, real Bun Workers, and real SQLite files. +Service-manager entry points are a separate disposable-host job; they are not part +of the developer-workstation invocation of this suite. OUT: mocks of `convergeCodex`, direct calls to phase-local gather/commit/history -helpers as acceptance proof, the live proxy on port 10100, the user's homes, GUI -controls, six file integrations, release/deploy/publish work, arbitrary filesystem -ABA, and historical edit-and-revert detection. +helpers as acceptance proof, the live proxy on port 10100, the user's homes, the +owner's installed service and service-manager registrations, GUI controls, six file +integrations, release/deploy/publish work, arbitrary filesystem ABA, and historical +edit-and-revert detection. ## The proof rule @@ -48,7 +51,7 @@ The suite records the parent SHA, composed SHA, case id, entry-point id, child P temporary roots, transition ids/generations, and the red/green oracle. This is how we know a test is not decoration that passes on both revisions. -## Production entry-point census — 36 rows +## Production entry-point census — 36 rows, two execution classes The count is by independently invokable command/route or independently scheduled production path. Aliases that execute the same branch are one row. The management @@ -65,8 +68,8 @@ surface has **14 route shapes and 16 current catalog-write call sites** because | P06 | `ocx sync-cache` | directly calls `invalidateCodexModelsCache` (`src/cli/index.ts:849-855`) | | P07 | `ocx restore` / `ocx eject` | dispatch calls `restoreNativeCodex` (`src/cli/index.ts:745-790`) | | P08 | `ocx restore back` | the reverse branch calls `syncModelsToCodex` (`src/cli/index.ts:747-764`) | -| P09 | `ocx stop` | `handleStop` restores native state (`src/cli/index.ts:456-551`), dispatched at `src/cli/index.ts:737-743` | -| P10 | `ocx uninstall` / `ocx remove` | restores native state before deleting owned OpenCodex state (`src/cli/index.ts:554-638`), dispatched at `src/cli/index.ts:795-798` | +| P09 | `ocx stop` — **disposable host only** | `handleStop` calls the globally addressed service manager before restoring native state (`src/cli/index.ts:456-551`), dispatched at `src/cli/index.ts:737-743` | +| P10 | `ocx uninstall` / `ocx remove` — **disposable host only** | stops and removes a globally addressed service before restoring native state and deleting owned OpenCodex state (`src/cli/index.ts:554-638`), dispatched at `src/cli/index.ts:795-798` | | P11 | `ocx recover-history --legacy-openai` | directly calls `restoreLegacyOpenaiHistory` (`src/cli/index.ts:711-724,792-794`) | | P12 | `ocx provider add ... --sync` | live-proxy branch calls `syncModelsToCodex` (`src/cli/provider.ts:130-146,216-239`) | | P13 | `ocx models add` | dispatches the custom add and live sync (`src/cli/models.ts:110-166,315-319`) | @@ -74,7 +77,7 @@ surface has **14 route shapes and 16 current catalog-write call sites** because | P15 | `ocx v2 mode ...` | persists mode then calls `syncModelsToCodex` (`src/cli/v2.ts:143-168`) | | P16 | `ocx v2 on|off` | changed transition calls `syncModelsToCodex` (`src/cli/v2.ts:172-198`) | | P17 | startup reconciliation path | journal replay is before bind (`src/cli/index.ts:169-176`), server construction directly invalidates cache (`src/server/index.ts:362-403`), and start arms the history guardian (`src/cli/index.ts:318-322`) | -| P18 | `POST /api/stop` | stops service, directly restores Codex, then drains (`src/server/management-api.ts:167-194`) | +| P18 | `POST /api/stop` — **disposable host only** | calls `stopServiceIfInstalled`, directly restores Codex, then drains (`src/server/management-api.ts:167-194`) | | P19 | `POST /api/sync` | calls `syncModelsToCodex` with the server-captured config (`src/server/management/config-routes.ts:261-268`) | | P20 | `POST /api/providers` | provider create reaches catalog write (`src/server/management/provider-routes.ts:99-147`) | | P21 | `PATCH /api/providers?name=...` | provider edit reaches catalog write (`src/server/management/provider-routes.ts:151-338`) | @@ -90,9 +93,9 @@ surface has **14 route shapes and 16 current catalog-write call sites** because | P31 | `DELETE /api/combos?id=...` | deletes combo then refreshes (`src/server/management/combo-routes.ts:203-217`) | | P32 | `PUT /api/v2` | saves agent settings then refreshes (`src/server/management/agent-settings-routes.ts:178-280`) | | P33 | `PUT /api/subagent-models` | saves roster, refreshes, then runs Claude/Desktop follow-up (`src/server/management/agent-settings-routes.ts:518-528`) | -| P34 | `ocx service start` | service dispatcher starts the installed wrapper (`src/service.ts:2511-2563`), whose baked command is `ocx start --port ...` (`src/service.ts:340,1378`) | -| P35 | `ocx service stop` | verifies stop, then directly restores native Codex (`src/service.ts:2564-2595`) | -| P36 | `ocx service uninstall` / `remove` | removes service, then directly restores native Codex (`src/service.ts:2610-2635`) | +| P34 | `ocx service start` — **disposable host only** | service dispatcher starts the installed wrapper (`src/service.ts:2511-2563`), whose baked command is `ocx start --port ...` (`src/service.ts:340,1378`) | +| P35 | `ocx service stop` — **disposable host only** | verifies stop, then directly restores native Codex (`src/service.ts:2564-2595`) | +| P36 | `ocx service uninstall` / `remove` — **disposable host only** | removes service, then directly restores native Codex (`src/service.ts:2610-2635`) | `ocx restart` and tray restart compose P09/P04 or P03/P02 (`src/cli/index.ts:939-949,963-967`); service install eventually launches P02; they @@ -101,6 +104,52 @@ management API are covered by the receiving P20-P33 route. If implementation fin another production edge, this count changes and C14 remains red until the row and runtime matrix are amended. +### Hard service-manager gate + +The suite as previously written would have taken down the owner's proxy. The +authoritative audit observed `com.opencodex.proxy` installed and running as PID 72848 +on this machine (`009_audit_synthesis_r4.md:17-24`). A temporary +`OPENCODEX_HOME` does **not** namespace the launchd label +`com.opencodex.proxy` or the Task Scheduler/systemd name `opencodex-proxy`: +they are fixed constants (`src/service.ts:42-43`), and start/stop/remove address +those constants directly (`src/service.ts:1640-1672,1868-1898,2045-2072`). Windows +also has the fixed native service id `opencodex-proxy-native` +(`src/lib/winsw.ts:33`). + +Therefore P34-P36 are removed from the workstation suite. The same audit found P09, +P10, and P18 calling `stopServiceIfInstalled` / `uninstallServiceIfInstalled`, which +query or mutate those global registrations (`src/service.ts:2204-2266`); those rows +also run only in the disposable-host job. P01-P08, P11-P17, and P19-P33 do not call a +service-manager registration API. P02/P04/P17 are seeded with +`claudeCode.systemEnv:false` and no system-env tracking record, so their production +startup/cleanup path cannot issue per-login-session `launchctl setenv/unsetenv` +(`src/server/system-env.ts:251-258,364-391`). + +"Disposable" means a throwaway VM/OS host and a **throwaway OS account**, not a temp +home on a developer account. Before any service setup or row is run, the job must +prove all of the following for that account; an unavailable query, +permission error, nonempty registration, or existing artifact is a hard failure, not +a skip: + +- macOS: `launchctl list | awk '$3 == "com.opencodex.proxy" { print }'` prints + nothing, and `test ! -e "$HOME/Library/LaunchAgents/com.opencodex.proxy.plist"` + succeeds. +- Windows: `schtasks.exe /Query /TN opencodex-proxy` exits nonzero with the + task-not-found result, and `sc.exe query opencodex-proxy-native` exits with service + error 1060 (service does not exist). +- Linux: `systemctl --user list-unit-files opencodex-proxy.service --no-legend + --no-pager` prints nothing, `systemctl --user status opencodex-proxy.service` + reports the unit not found, and + `test ! -e "$HOME/.config/systemd/user/opencodex-proxy.service"` succeeds. + +Only after that empty result may fixture setup install the service state needed by a +row. Each P09/P10/P18/P34-P36 case starts from a restored clean VM/account snapshot, +runs the empty gate, installs and starts/stops only its fixture registration as the +row requires, invokes the row, tears that registration down, reruns the same platform +gate, and requires the same empty result. State is not carried from one service row +to the next. The job never runs on an account that has a real OpenCodex service, +regardless of which home installed it. + ## Harness: real isolated processes, never the user's state The parent creates one root with `mkdtempSync(join(tmpdir(), @@ -153,16 +202,41 @@ ownership, and file/record observations—never `sleep` as readiness. Each child a hard watchdog, all Workers are joined, all spawned PIDs are proven exited, and only then is the known temporary root removed. A teardown failure fails the case. +The native lock is the deliberate exception to the temporary-root statement. Its +database lives under the fixed effective-user runtime root, not under the case root: +`/tmp/opencodex-runtime-v1-/native-write-locks/.sqlite` +on POSIX and +`/OpenCodex/Runtime/v1//native-write-locks/.sqlite` +on Windows (`005_contract.md:693-721`). Before spawning a child, the harness resolves +that exact path through the production identity/runtime resolver, checks that the +hash input is the case's canonical `CODEX_HOME`, and requires the exact database and +its `-journal`, `-wal`, and `-shm` sidecars to be absent. A pre-existing file fails +the case; it is never adopted or deleted. + +After every child and Worker is joined and every SQLite handle is closed, teardown +re-resolves the same path, rechecks the effective uid/SID, canonical-home hash, owner, +mode/ACL, and non-symlink/non-reparse components, then removes only that four-name +allowlist: `.sqlite`, `.sqlite-journal`, `.sqlite-wal`, and +`.sqlite-shm`. It does not glob, enumerate, truncate, or remove the shared +runtime root or `native-write-locks` directory. A failed identity check aborts +cleanup and fails the case. Each allowlisted file is removed only if it exists and +was absent at preflight. This confines per-case lock files without touching another +Codex home's per-user lock state. + ## Runnable composed scenarios ### A — every entry reaches one funnel -Parameterize P01-P36. Seed an authorizing isolated installation, invoke the real -entry, and read the integration record transition id plus a recursive before/after -manifest. Every native mutation must have exactly one admitted transaction; OFF -entries must produce a removal transaction, not a skip. P20-P33 retain their -existing primary 2xx/201 behavior and expose the contract disposition; P30/P33 -still complete their Claude/Desktop follow-up. +Parameterize the 30 workstation-safe rows P01-P08, P11-P17, and P19-P33 in the +ordinary suite. Run P09/P10/P18/P34-P36 only in the separately gated disposable-host +job above. The workstation rows seed authorizing isolated state without any service +artifact; the disposable P34-P36 setup installs its fixture service only after the +empty-registration gate. Invoke the real entry and read the integration record +transition id plus a recursive before/after manifest. Every native mutation must have +exactly one admitted transaction; OFF entries must produce a removal transaction, not +a skip. P20-P33 retain their existing primary 2xx/201 behavior and expose the contract disposition; +P30/P33 still complete their Claude/Desktop follow-up. The two job manifests together, +not either one alone, make the 36-row census. **RED today:** `convergence.ts` and the integration record do not exist; P06, P11, P17, P18, P24-P33, P35, and P36 visibly reach direct writers. The management rows @@ -207,7 +281,8 @@ durable unresolved state. ### D — foreign/unknown authority creates nothing Create foreign service-home evidence, then repeat with corrupt and unreadable -mirror evidence. Snapshot the whole temp root and the expected OS-runtime namespace, +mirror evidence. Snapshot the whole temp root and the exact case-specific OS-runtime +lock path, invoke P02, P04, P19, one of P20-P33, P07, P18, P35, and P36, and compare byte/path manifests. Assert no lock directory, SQLite DB/journal, integration record, native journal, backup, catalog/cache, config/profile, history manifest/row, or rollout @@ -237,9 +312,18 @@ uid/SID-scoped database. Invoke P19 from children using default, explicit, absolute, tilde, symlink, and platform case-equivalent spellings of one existing home; they must contend on one lock. Two different homes acquire independently. Missing home, namespace symlink, -wrong-owner/mode, malformed DB, and finite-deadline contention return typed -`refused` or `busy`, never throw; a normal run proves `acquired` through a -`converged` response. +malformed DB, and finite-deadline contention return typed `refused` or `busy`, never +throw; a normal run proves `acquired` through a `converged` response. + +Wrong-owner/mode activation is not fabricated with `chown` in ordinary CI: an +unprivileged POSIX account cannot create a path owned by another uid. That case stays +in `tests/codex-user-identity.test.ts` and uses the resolver seam required by the +contract (`005_contract.md:773-777`): a child process calls the exported production +resolver while the filesystem-inspection seam reports the real fixture directory with +`uid !== process.getuid()` (or a broad mode), then proves typed refusal and zero +callback/SQLite-open activity. Windows uses the corresponding owner/ACL seam. A job +that instead uses a real foreign owner must be a separately labelled privileged +disposable job; it is not required for ordinary CI and may not run on a workstation. **RED today:** no such exclusion exists, so same-home contenders both mutate and unsafe namespace fixtures are not classified. **GREEN:** the exact @@ -259,20 +343,37 @@ detects the cooperating ABA and target identity detects single-direction retarge An arbitrary parent-symlink A→B→A wholly between checks is deliberately not claimed (`005_contract.md` §3). -### H — history overtaking is rejected before mutation - -Hold the production history lock with a helper child after transition A has -committed native ON but before A's Worker mutates history. Process B invokes P07 or -P19 with desired OFF and commits the newer native expectation. Release the history -lock. Place sentinels in the manifest, every rollout, and DB before release; assert -A returns `pending/overtaken` without changing any sentinel, then B alone produces -OFF history. Reverse ON/OFF and repeat. +### H — history overtaking after stale mutation is detected and repaired + +This must reach the contract's hard half: B commits **after** A has changed history, +not before A's initial expectation check. Seed at least two production-shaped +rollouts. A holder child opens the real state DB and prints `DB_WRITE_HELD` only after +`BEGIN IMMEDIATE` succeeds. That permits A's reads but blocks its later DB write. The +parent sends P19 to a single P02 server for transition A. Through the production +Worker, A commits its native pair, writes the manifest, changes at least one rollout, and then +parks at the real DB transaction (`src/codex/history-provider.ts:606-648` for apply; +the reverse path changes rollouts before its DB transaction at `:656-690`). The +parent uses `fs.watch` plus an immediate byte recheck and proceeds only after both the +manifest and rollout post-images are observed; no timer or injected Worker hook +declares the pause. + +While A is blocked mid-traversal, process B invokes P07 or P19 in the opposite +direction. Require B's newer `{nativeGeneration,currentTxId}` and its +`history:{status:"pending",txId:B,...}` schedule to be durable before releasing the +holder. Release `BEGIN IMMEDIATE` before A's production busy deadline; A finishes +its remaining DB work and reaches the terminal conditional record update. Assert A +reports `pending/overtaken`, B's exact pending schedule was not replaced by A, and +the guardian in the same P02 process observes B, runs after A releases the history +lock, and repairs manifest, every +rollout, and DB to B. Reverse ON/OFF and repeat. The real SQLite write lock plus the +observed manifest/rollout post-image is the deterministic mid-traversal barrier; no +mock or direct history helper is accepted. **RED today:** manifest and rollouts are outside SQLite's transaction -(`src/codex/history-provider.ts:606-648,656-695`) and there is no history lock or -expected transition, so scheduling order can overwrite the newer direction. -**GREEN:** the losing expectation is rejected before its first probe/write and the -highest native generation owns final history. +(`src/codex/history-provider.ts:606-648,656-695`) and there is no transition-owned +terminal conditional update, so A can overwrite/cancel the newer direction or leave +its stale bytes terminal. **GREEN:** A is allowed to mutate stale history, cannot +replace B's pending schedule, and the live guardian makes B the clean terminal owner. ### I — retry beyond the old horizon, without restart @@ -333,6 +434,33 @@ external guard, while restore deletes the journal even when external (`src/codex/inject.ts:764-769`). **GREEN:** the external-provider veto is checked after service authority and before every artifact for every row. +## Substrate-sensitivity audit + +The named RED is mandatory for every scenario. The observations in the last column +are either baseline-green controls that can pass without the substrate or easier +halves that do not activate the defect; recording only one of them is a false proof +and does not satisfy the scenario. + +| Scenario | Observation that must be RED before the substrate | Baseline-green or non-defining observation that cannot count as RED | +|---|---|---| +| A | at least one row reaches a direct writer or lacks the sole-funnel receipt; the complete 36-row manifest cannot be produced | a row's primary CLI/HTTP success status | +| B | stale A writes catalog/native bytes after B changes admitted config | provider request reached the fixture and a later fresh call succeeds | +| C | `/healthz` or SSE exceeds its watchdog while the production history path waits on the held DB | health/SSE with no overlapping DB contention | +| D | P02/P04/P19/P20 creates or changes an artifact under foreign/unknown authority | P07/P18/P35/P36 may already refuse through teardown ownership checks; those rows are coverage, not the RED oracle | +| E | same-uid/SID contenders with different environment homes both enter mutation | distinct canonical Codex homes proceeding independently is the expected control | +| F | equivalent spellings fail to contend or an unsafe namespace reaches callback/SQLite open | two different canonical homes acquiring independently; an ambient OS error from an unconstructible `chown` fixture | +| G | cooperating A→B→A or one-way target retarget lets stale A mutate | final config bytes equalling A after A→B→A | +| H | after A has changed manifest/rollout bytes, A can replace B's pending schedule or B is not repaired | rejecting A at the initial pre-mutation expectation check tests only the easy half | +| I | attempt 61 has no next timer or the same PID never repairs after release | a retry below the old 60-tick horizon | +| J | unchanged OFF leaves residue/reapplies it, or unchanged ON leaves a required artifact absent | already-clean ON/OFF no-op behavior and invalid-config parsing by itself | +| K | missing/conflicting provenance permits mutation or current-byte drift is overwritten | a simple untouched present baseline that the filename-based restore already happens to reproduce | +| L | a dead journal or another native artifact changes while an external provider is active | external-provider detection with no journal/residue present | + +Any case that is green on both SHAs is labelled `baseline-green-control` in the case +manifest and cannot satisfy a criterion. Each A-L scenario needs its named +substrate-sensitive RED artifact and corresponding GREEN artifact under the same +fixture and entry point. + ## C1-C18 matrix | Criterion | Production proof | What fails on the current revision | @@ -343,15 +471,15 @@ after service authority and before every artifact for every row. | C4 | P02/P19 through C/I: unresolved is durable, attempt 61+ is armed, same PID later converges | current guardian terminates at 60 and has no durable typed record | | C5 | P19 through F: converged proves acquired, zero-deadline contention proves busy, unsafe namespace proves refused | no native lock API/taxonomy exists; both contenders write | | C6 | P19 through F across equivalent and distinct real homes | textual-path callers have no common cross-process lock | -| C7 | P19 through D/E/F: no home-derived namespace; wrong-owner/symlink namespace refuses | no per-user namespace exists; current paths are home/environment-derived elsewhere | +| C7 | P19 through D/E/F proves no home-derived namespace; the F resolver-seam child proves wrong-owner/mode refusal without privileged `chown` | no per-user namespace exists; current paths are home/environment-derived elsewhere | | C8 | P02/P04/P07/P18/P19/P20/P35/P36 through D, with full manifest including runtime lock path | current ownership check is teardown-only and fails open; management/startup bypass it | | C9 | P02/P04/P07/P18/P19/P20/P35/P36 through L | management/startup write around the guard and external restore deletes the journal | | C10 | P02/P07/P08/P18 through K | filename/filter restore has no baseline/post-image record and cannot restore proven absence safely | | C11 | P19 through J for OFF-with-residue and ON-with-absence | no persisted Codex intent/observer; `/api/sync` always follows the old apply seam | | C12 | one P02 server plus a subprocess config writer, then P19 OFF and ON through J | P19 passes the long-lived captured `config` object (`src/server/management/config-routes.ts:261-264`) | | C13 | **Not provable through a production entry point.** Run typecheck, full suite, GUI lint, privacy scan, docs build, then require this composed suite's case manifest and red/green evidence | those static/broad gates can be green today while A-L are red; C13 alone proves none of C1-C12 | -| C14 | A drives all P01-P36; P20-P33 cover 14 route shapes/16 calls; module-graph reachability and transition receipts must agree | 16 management call sites and multiple CLI/startup paths reach direct writers instead of one funnel | -| C15 | P02/P07/P19 through H in both directions with manifest/rollout/DB sentinels | only DB substeps are transactional; processes can overtake file writes | +| C14 | A's workstation and disposable-host manifests together drive P01-P36; P20-P33 cover 14 route shapes/16 calls; module-graph reachability and transition receipts must agree | 16 management call sites and multiple CLI/startup paths reach direct writers instead of one funnel | +| C15 | P02/P07/P19 through H in both directions: B commits only after A's manifest/rollout post-images exist, B's pending schedule survives A's terminal conflict, and the guardian repairs B | only DB substeps are transactional; processes can overtake file writes and stale completion has no conditional terminal update | | C16 | P02/P07/P08/P18/P17 sequence through K, preserving both optional sections and unknown keys | no shared record owner/schema exists | | C17 | P19 through G with production config A→B→A and one-way parent retarget | no generation or stable target expectation exists; equal content passes | | C18 | P19 through E with independently different HOME and USERPROFILE in real children | no uid/SID lock exists, so environment-home variation does not contend | @@ -377,8 +505,9 @@ the exact false proof WP13 exists to prevent. namespaces; parallelization is allowed only after distinct runtime roots and PIDs are proven in the case manifest. - Windows CI must exercise real SID, junction/reparse, ACL, and USERPROFILE behavior; - POSIX CI must exercise real uid, symlink, and mode/owner behavior. Platform skips - are limited to the opposite platform's primitive, never the shared criterion. + POSIX CI must exercise real uid, symlink, mode, and the wrong-owner resolver seam. + A real foreign-owner fixture is privileged-disposable-only. Platform skips are + limited to the opposite platform's primitive, never the shared criterion. ## What this suite deliberately does not prove @@ -403,8 +532,10 @@ the exact false proof WP13 exists to prevent. WP13 passes only when A-L are red on the pre-substrate revision for the named observable reason, green on the composed revision, all 36 production rows are in -the case manifest, C1-C12 and C14-C18 have artifact-level production evidence, C13 -is separately green, every child/Worker is joined, and the only removed paths are -the suite's explicit temporary roots. A missing row, a same-process substitute for -E/H, a mocked convergence function, a test that passes on both revisions, or a -green broad suite beside any red composed case is a failure. +the paired workstation/disposable-host manifests, C1-C12 and C14-C18 have +artifact-level evidence, C13 is separately green, every child/Worker is joined, the +service gate is empty before and after its job, and teardown removes only the suite's +explicit temporary roots plus the validated four-name lock-file allowlist. A missing +row, a same-process substitute for E/H, a mocked convergence function, a test that +passes on both revisions without a separate substrate-sensitive RED, or a green broad +suite beside any red composed case is a failure. From 3ec16ed7b993eee09085418d0ba366e7e70e59c9 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Tue, 4 Aug 2026 12:28:54 +0900 Subject: [PATCH 037/163] docs(substrate): every document's TypeScript now compiles, checked per document MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit I found this by compiling the unit myself rather than trusting per-document reports: each doc passed alone while the concatenation failed. Concatenating was the wrong check — the docs legitimately re-show shared types for readability, so duplicates are expected. Per-document with future-module references excluded is the honest gate, and it caught two real defects the per-doc reports had missed. 030 printed a bodyless withCodexWriteLock, the TS2391 form a reviewer has now caught in this unit three separate rounds. It becomes a WithCodexWriteLock type, so the mistake is structurally impossible to reprint. 040's convergeCodex assignment left its parameter implicit, which is TS7006 because ConvergeCodex resolves through a module WP8b has not landed yet. The parameter is annotated with the reason recorded. 050's harness block is a call-site excerpt whose identifiers come from the surrounding harness, so it is tagged ts-excerpt rather than pretending to be a compilable unit. Result: 005, 010, 020, 030, 040 and 050 all report zero real diagnostics; the only remaining TS2307s are references to modules this unit creates. --- .../010_catalog_seam.md | 93 +++++++++++++++++-- .../030_lock_protocol.md | 12 ++- .../040_ownership_convergence.md | 6 +- .../050_composed_acceptance.md | 7 +- 4 files changed, 104 insertions(+), 14 deletions(-) diff --git a/devlog/_plan/260804_codex_write_substrate/010_catalog_seam.md b/devlog/_plan/260804_codex_write_substrate/010_catalog_seam.md index ec0ed9455..a458f367a 100644 --- a/devlog/_plan/260804_codex_write_substrate/010_catalog_seam.md +++ b/devlog/_plan/260804_codex_write_substrate/010_catalog_seam.md @@ -136,7 +136,8 @@ typecheck/tests in the WP8b commit: state, history, or observed-state work and is not `inspectAdmissionSnapshot`. 3. Concrete config/native generation owners: every cooperating persisted-config commit bumps the config generation through the existing config mutation owner, - and the integration-record owner reads/advances native generation plus `txId`. + while `transition-state.ts` reads/conditionally advances native generation plus + `txId` in the effective-user + canonical-`CODEX_HOME` SQLite coordinator row. WP9 may consume those tokens; it may not assume WP12 will add their storage or bump sites later. 4. One contract projection for catalog-only completion. WP9 supplies the real @@ -157,6 +158,67 @@ introducing them after WP9 has already depended on them. or reconstruct a stale candidate. ```ts +import type { + CatalogAdmissionSnapshot as ContractCatalogAdmissionSnapshot, + CommitExpectation as ContractCommitExpectation, +} from "./convergence-types"; + +interface CatalogFileIdentity { + readonly device: bigint; + readonly inode: bigint; +} + +/** Catalog-private evidence for one prepared filesystem target. */ +export interface CatalogTargetIdentity { + readonly path: string; + readonly canonicalParent: string; + readonly parentIdentity: CatalogFileIdentity; + readonly fileIdentity: CatalogFileIdentity | null; +} + +/** Sanitized gather detail; provider identity and raw errors never enter it. */ +export interface CatalogGatherNotice { + readonly kind: "provider-auth" | "provider-network" | "fallback"; + readonly retryable: boolean; +} + +interface PreparedCatalogBackup { + readonly kind: "keyed" | "legacy"; + readonly path: string; + readonly bytes: Uint8Array; + readonly createOnce: true; +} + +export interface CodexCatalogRefreshResult { + readonly added: number; + readonly path: string; + readonly catalogExists: boolean; + readonly catalogWritten: boolean; + readonly cacheSynced: boolean; + readonly comboOmissions: readonly Readonly<{ + id: string; + targets: readonly string[]; + reason: "incomplete_metadata" | "incompatible_modalities"; + message: string; + }>[]; +} + +export interface CatalogWriteReceipt { + readonly keyedBackup: "written" | "preserved" | "not-requested"; + readonly legacyBackup: "written" | "preserved" | "not-requested"; + readonly catalog: "written" | "not-written"; + readonly cache: "written" | "not-written"; +} + +export interface PreparedCodexCatalogCommit { + readonly catalogBytes: Uint8Array; + readonly cacheBytes: Uint8Array; + readonly backups: readonly PreparedCatalogBackup[]; + readonly targets: readonly CatalogTargetIdentity[]; + readonly result: CodexCatalogRefreshResult; + readonly notices: readonly CatalogGatherNotice[]; +} + const candidateBrand: unique symbol = Symbol("CodexCatalogCandidate"); export interface CodexCatalogCandidate { @@ -173,16 +235,29 @@ interface CandidateState { const states = new WeakMap(); -export async function gatherCodexCatalogCandidate( - admission: CatalogAdmissionSnapshot, -): Promise; +/** Signature only; WP9 exports a concrete function with this type. */ +export type GatherCodexCatalogCandidate = ( + admission: ContractCatalogAdmissionSnapshot, +) => Promise; -export function commitCodexCatalogCandidate( +/** Signature only; WP9 exports a concrete synchronous function with this type. */ +export type CommitCodexCatalogCandidate = ( candidate: CodexCatalogCandidate, - expectation: CommitExpectation, -): CatalogCommitOutcome; + expectation: ContractCommitExpectation, +) => CodexCatalogCommitResult; ``` +Ownership is deliberate. `CatalogAdmissionSnapshot` and `CommitExpectation` are +aliased imports from the shared `convergence-types.ts` contract, so concatenating +phase excerpts cannot turn the imports into duplicate local definitions; the full +`AdmissionSnapshot` is also contract-owned but is not referenced by this +catalog-scoped signature. `PreparedCodexCatalogCommit`, `CatalogTargetIdentity`, +`CatalogGatherNotice`, `CodexCatalogCommitResult`, `CodexCatalogRefreshResult`, and +`CatalogWriteReceipt` are catalog-private definitions here. WP11 exposes +`withCodexWriteLock` as a callback/result API and explicitly has no public handle or +release method (`030_lock_protocol.md:126-136`), so WP9 neither defines nor imports a +`CodexWriteLockHandle`. + The catalog-scoped snapshot receives the same config object the current management callback already uses. `prepareCatalogSync` receives `admission.config` — **that object**, not a separate `readConfigDiagnostics()` result. The generation token @@ -207,7 +282,7 @@ The old plan owned a `ContentRevision` and hashed config/catalog bytes. That des is deleted. Content equality passes A→B→A, and a textual path does not reveal a parent-symlink retarget. The shared mechanism is `005_contract.md` §3: -- `AdmissionSnapshot.generation` identifies the cooperating config generation used +- `CatalogAdmissionSnapshot.generation` identifies the cooperating config generation used by gather; - `CommitExpectation { nativeBefore, nativeAfter, txId }` identifies the one native transition this commit is allowed to perform; @@ -249,7 +324,7 @@ type CatalogGatherOutcome = | { kind: "degraded"; candidate: CodexCatalogCandidate; notices: readonly CatalogGatherNotice[] } | { kind: "failed"; surface: "provider-auth" | "provider-network"; retryable: boolean }; -type CatalogCommitOutcome = +export type CodexCatalogCommitResult = | { kind: "committed"; result: CodexCatalogRefreshResult; writes: CatalogWriteReceipt } | { kind: "stale"; reason: "generation" | "target-identity" | "candidate-consumed" } | { kind: "failed"; surface: "disk"; writes: CatalogWriteReceipt }; diff --git a/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md b/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md index 875fb92fa..6824c887e 100644 --- a/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md +++ b/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md @@ -123,10 +123,18 @@ export interface CodexWriteCommitContext { type Synchronous = T extends PromiseLike ? never : T; -export async function withCodexWriteLock( +/** + * A TYPE, not a bodyless declaration. + * + * `export async function f(...): Promise;` with no body is TS2391, and a + * reviewer has caught that exact form in this unit three separate rounds by + * compiling the documents. Publishing the shape as a type makes the mistake + * structurally impossible to reprint. + */ +export type WithCodexWriteLock = ( options: CodexWriteLockOptions, commit: (context: CodexWriteCommitContext) => Synchronous, -): Promise>; +) => Promise>; ``` `CodexWriteLockResult` is the lock module's own bounded mechanism result. diff --git a/devlog/_plan/260804_codex_write_substrate/040_ownership_convergence.md b/devlog/_plan/260804_codex_write_substrate/040_ownership_convergence.md index beeb9f193..98b1b0b5b 100644 --- a/devlog/_plan/260804_codex_write_substrate/040_ownership_convergence.md +++ b/devlog/_plan/260804_codex_write_substrate/040_ownership_convergence.md @@ -531,7 +531,11 @@ or WP12-specific request/result type. declare const convergeCodexImpl: ( request: ConvergeRequest, ) => Promise; -export const convergeCodex: ConvergeCodex = request => convergeCodexImpl(request); +// Annotated rather than inferred: `ConvergeCodex` resolves through +// `./convergence-types`, which does not exist until WP8b lands, so an +// unannotated parameter compiles as implicit `any` (TS7006) in the excerpt. +export const convergeCodex: ConvergeCodex = (request: ConvergeRequest) => + convergeCodexImpl(request); declare const EXPLICIT_CODEX_CONVERGENCE_DEADLINE_MS: number; void convergeCodex({ action: "converge", diff --git a/devlog/_plan/260804_codex_write_substrate/050_composed_acceptance.md b/devlog/_plan/260804_codex_write_substrate/050_composed_acceptance.md index 2ed509656..0ead30ffe 100644 --- a/devlog/_plan/260804_codex_write_substrate/050_composed_acceptance.md +++ b/devlog/_plan/260804_codex_write_substrate/050_composed_acceptance.md @@ -158,9 +158,12 @@ The parent creates one root with `mkdtempSync(join(tmpdir(), directories beneath it. No path is derived from the parent process's `HOME`, `USERPROFILE`, `CODEX_HOME`, or `OPENCODEX_HOME`. -Every OpenCodex child is spawned as: +Every OpenCodex child is spawned as follows. This is a call-site excerpt, not a +compilable unit — `repoRoot`, `fixtureRoot` and the fixture identifiers come +from the harness that surrounds it, so the fence is tagged `ts-excerpt` to keep +it out of the unit-wide compile check that the other five documents pass: -```ts +```ts-excerpt Bun.spawn([ process.execPath, resolve(repoRoot, "src/cli/index.ts"), From 7bde9e0c977721fc0b9d8617c85ff17de7c07658 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Tue, 4 Aug 2026 12:38:29 +0900 Subject: [PATCH 038/163] =?UTF-8?q?docs(substrate):=20round=205=20?= =?UTF-8?q?=E2=80=94=20the=20reviewer=20says=20the=20architecture=20is=20r?= =?UTF-8?q?ight?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Blockers fell from eight to three, fourteen findings closed, and for the first time the verdict opens with 'the sibling SQLite coordinator is the right architecture'. What remains is three places where a document still describes the world before that coordinator existed. The reviewer also specified the lock order for us, which is the question I got wrong twice: open N with BEGIN IMMEDIATE, take C while holding N, do the authoritative reread and every write on the already-open N connection, release C, then COMMIT N; history takes H later and only does fail-fast conditional work against N. Edges are N-then-C and a short H-then-N, with no inverse, so CLI and service processes taking the same order cannot deadlock. They validated the zero-row protocol against Worker death at every point and found the hole I missed: death after the SQLite commit but before postMessage leaves terminal state already durable, so the parent must reread the row before recording worker-died or it overwrites a real success with a synthetic failure. And WP13's disposable-host gate stays open for a reason worth keeping: it passes on any developer account with no installed service, and protects this machine only because a service happens to be installed here. Negative evidence is not proof of disposability — the same class of reasoning this unit keeps catching in me. --- .../010_audit_synthesis_r5.md | 109 ++++++++++++++++++ 1 file changed, 109 insertions(+) create mode 100644 devlog/_plan/260804_codex_write_substrate/010_audit_synthesis_r5.md diff --git a/devlog/_plan/260804_codex_write_substrate/010_audit_synthesis_r5.md b/devlog/_plan/260804_codex_write_substrate/010_audit_synthesis_r5.md new file mode 100644 index 000000000..50c6bfb89 --- /dev/null +++ b/devlog/_plan/260804_codex_write_substrate/010_audit_synthesis_r5.md @@ -0,0 +1,109 @@ +# Substrate audit round 5 — synthesis + +Verdict: **FAIL**, but the sentence that matters is new: + +> "The sibling SQLite coordinator is the right architecture." + +Five rounds in, the reviewer has stopped arguing about the design and is now +arguing about three specific places where the roadmap has not caught up to it. + +| Round | Closed | Open | New | Blockers | +|---|---|---|---|---| +| 1 | — | 13 | 13 | many | +| 2 | 1 | 11 | 5 | 8 | +| 3 | 5 | 11 | 4 | 8 | +| 4 | 9 | 8 | 5 | 8 | +| 5 | **14** | 10 | 9 | **3** | + +Fourteen closed this round, and blockers fell from eight to three. + +## The reviewer designed the lock order for us + +Worth quoting, because it answers the question I have gotten wrong twice and it +is now the specification: + +1. native/coordinator transaction `N` with `BEGIN IMMEDIATE` +2. config transaction `C`, **while holding** `N` +3. authoritative reread, native/provenance writes, and the transition-row update + using the **already-open** `N` connection +4. release `C`, then **`COMMIT N`** +5. later, history lock `H`; at claim and terminal boundaries it performs only + fail-fast conditional operations against `N`, releasing `H` and retrying if + busy + +Edges: `N → C` and a short `H → N`. No `C → N`, no `C → H`, no held `N → H`. +CLI and service processes taking the same order cannot deadlock. + +They also validated the zero-row protocol against Worker death at every point, +and found the one hole I had not: **death after the SQLite commit but before +`postMessage`** leaves terminal state already durable. So the parent must +**reread the row before recording `worker-died`**, or it overwrites a real +success with a synthetic failure. + +## The three blockers, all the same shape + +Each is a place where a document still describes the world before the +coordinator existed. + +**B1 — WP11 never received coordinator ownership.** `030` still writes the +transition through JSON and then executes `ROLLBACK` on its SQLite transaction +(`030:150-173`). That discards the very row the design depends on, and makes a +successful commit indistinguishable from an unrecorded partial write. WP11 must +own one **committed** coordinator transaction and pass it as an opaque capability +— opening a second connection inside the callback would contend with its own +`BEGIN IMMEDIATE`. + +**B2 — WP9 has no path for the config object it promises.** `010` says gather +must receive the management callback's exact `OcxConfig` (`010:186-190`), but +`convergeCodex` takes only a `ConvergeRequest` and the management calls supply no +config (`010:328-343`). Neither commit specified the transport. It goes in the +internal catalog request, or convergence is bound to the management context — +not a resident global, and not a reread that contradicts the phase. + +**B3 — adoption's native-clean proof is not serialized.** `040:399-412` validates +every surface and then writes provenance, with no lock across the gap. A CLI +convergence can modify an artifact in between, so the recorded baseline describes +a state that no longer exists. Revalidate under native coordination, then write +and initialize the row before releasing. + +## What closed + +Fourteen, including several I expected to argue about: the `/api/sync` adapter +(#4), the full admission snapshot with named readers (#8), the complete history +failure union (r3 #3), the baseline classes (N4), C17's narrowing (r4 #5), the +compile prelude (r4 #4), and three of the four WP13 defects — including D2, where +the reviewer confirms Scenario H "would fail without the substrate and pass with +it", which is this project's standing bar. + +They also compiled the contract's ten blocks plus the WP10 and WP12 fragments +themselves: zero diagnostics. + +## The finding that corrects my own gate + +**WP13 D1 stays open, and the reason is sharp.** The disposable-host gate passes +on any developer account with no installed service. It protects *this* machine +only because a service happens to be installed here. Negative evidence — "no +service found" — is not proof of disposability. + +That is exactly the class of reasoning this unit keeps catching in me: an absence +treated as a guarantee. A positive sentinel is required, plus exclusion from +ordinary test commands. + +## Smaller, all accepted + +- WP10 calls `updateCodexTransitionState`; the contract exports + `updateCodexHistoryTransition` (#new 5) +- the SQLite `CHECK` accepts a NULL direction alongside a positive generation + (#new 6) +- WP13 scenarios A/I/K and C16 still read transition state from the JSON record + (#11, #new 7) +- `CodexArtifactId` and both `baseline` variants lack extension passthrough (#3) +- WP13 Scenario A still demands no writer outside `convergence.ts`, contradicting + the per-domain root table (#2) +- stale wording and a drifted citation (#new 9) + +## Position + +Three blockers, all "make the document match the coordinator", plus a real +parent-reread rule and a real gate weakness. No structural replan. The next pass +fixes these and re-audits. From 75aa9b58332702477f07d7777afc6b63746f1c65 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Tue, 4 Aug 2026 12:48:26 +0900 Subject: [PATCH 039/163] docs(substrate): close all three architecture blockers MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit B1: WP11 owns a committed coordinator transaction. The previous version wrote transition state through JSON and then ROLLBACKed the SQLite transaction, discarding the exact row the design depends on and making a successful commit indistinguishable from an unrecorded partial write. Success now COMMITs N, and the callback receives a branded one-shot capability rather than a connection, because opening a second connection inside it would contend with WP11's own BEGIN IMMEDIATE. Native generation, txId and scheduling live solely in the transition row. B2: WP9 gets its config through a management-bound convergence factory that captures the callback's exact object in a closure. Adding config to the public ConvergeRequest was rejected on the grounds that arbitrary callers could then inject catalog authority — the public shape stays a request, not a channel. B3: adoption acquires N, then C while holding N, reclassifies authoritatively and writes provenance and the row on that open connection. Validation and the write used to sit either side of an unlocked gap, so a recorded baseline could describe a state that no longer existed — and a wrong baseline is permanent, since OFF would later restore to it. On disagreement it refuses and reports both observations rather than adopting either. Also: the disposable-host gate gains a positive sentinel, because 'no service installed' passes on any ordinary developer account and only protected this machine by accident. The parent rereads the transition row before recording worker-died, so a Worker that dies after its commit but before postMessage no longer has its real success overwritten. All six documents report zero real diagnostics; the only remaining TS2307s reference modules this unit creates. --- .../010_catalog_seam.md | 109 +++++++++-- .../020_history_isolation.md | 40 +++- .../030_lock_protocol.md | 177 ++++++++++++------ .../040_ownership_convergence.md | 157 ++++++++++++---- .../050_composed_acceptance.md | 80 +++++--- 5 files changed, 424 insertions(+), 139 deletions(-) diff --git a/devlog/_plan/260804_codex_write_substrate/010_catalog_seam.md b/devlog/_plan/260804_codex_write_substrate/010_catalog_seam.md index a458f367a..a3fe5f0dd 100644 --- a/devlog/_plan/260804_codex_write_substrate/010_catalog_seam.md +++ b/devlog/_plan/260804_codex_write_substrate/010_catalog_seam.md @@ -128,7 +128,12 @@ typecheck/tests in the WP8b commit: 1. `ConvergeRequest.scope` with at least `"catalog" | "full"`, plus a concrete catalog request constructor. The production management callbacks use only `scope: "catalog"`; `"full"` remains the compatibility/current-behavior branch - until WP12 replaces it with authoritative admission. + until WP12 replaces it with authoritative admission. `ConvergeRequest` is the + public caller shape and does **not** gain a config field: permitting arbitrary + callers to substitute catalog authority would make the scoped funnel weaker than + the callback it replaces. Instead, WP8b exports a management-only factory that + captures the management context's exact config object and returns a + `ConvergeCodex`; the factory is not re-exported by the public Codex facade. 2. A concrete `CatalogAdmissionSnapshot` plus catalog-scoped snapshot reader that accepts the same `OcxConfig` object the current callback already uses and captures only the config generation and catalog target identities WP9 validates. It @@ -263,7 +268,9 @@ callback already uses. `prepareCatalogSync` receives `admission.config` — **th object**, not a separate `readConfigDiagnostics()` result. The generation token detects a cooperating persisted transition before commit. WP12 later replaces this limited input with its authoritative full admission; WP9 does not import that future -helper. +helper. There is no resident config global and no persisted config re-read in this +catalog-only path: either would change the current callback's behavior when its +long-lived config object differs from disk. Gather performs provider auth/network work, source loading, parsing, merging, serialization, cache-wrapper construction, and backup planning. It performs no @@ -395,17 +402,46 @@ and test imports migrate. The dependency-graph test, not an `rg` spelling guard, proves no alias, re-export, wrapper, or dynamic import reaches the writers outside `convergence.ts` (`005_contract.md` §Test plan). +That reachability rule is per domain, not repository-wide. WP9 proves only the +catalog row below; the composed WP13 Scenario A must consume the complete per-domain +table from `005_contract.md:927-960` and must not assert that every writer is +unreachable outside `convergence.ts`: + +| Domain | Low-level writer owner | Permitted runtime roots | +|---|---|---| +| catalog, hashed/legacy backups, models cache | `src/codex/internal/catalog-commit.ts` | `src/codex/convergence.ts` only | +| history DB rows, manifest, rollout files | history write exports in `src/codex/internal/history-writer.ts` | `src/codex/history-worker.ts` only | +| transition pair and history schedule/terminal row | `src/codex/transition-state.ts` | `src/codex/convergence.ts` and `src/codex/history-worker.ts` only | + +Therefore WP13 Scenario A's repository-wide sentence at +`050_composed_acceptance.md:249` is attributable to WP13 and must be replaced there +with symbol-level assertions against each contract row. A valid history Worker is a +required permitted root, not a writer leak. + ## The first production `convergeCodex` is catalog-scoped for management WP8b declared this function as a type only. WP9 now adds a non-placeholder -implementation in `src/codex/convergence.ts`: +implementation in `src/codex/convergence.ts`. The management-only factory closes +over the exact object received by `handleManagementAPI`; the internal function makes +that reference's path to capture explicit without adding it to `ConvergeRequest`: ```diff -+export async function convergeCodex( ++interface ConvergenceContext { ++ readonly catalogConfig?: Readonly; ++} ++ ++async function convergeCodexInContext( + request: ConvergeRequest, ++ context: ConvergenceContext, +): Promise { + if (request.scope === "catalog") { -+ const admission = captureCatalogAdmissionSnapshot(request); ++ if (!context.catalogConfig) { ++ return catalogContextMissingOutcome(); ++ } ++ const admission = captureCatalogAdmissionSnapshot( ++ request, ++ context.catalogConfig, ++ ); + const gathered = await gatherCodexCatalogCandidate(admission); + const catalog = commitCatalogAgainstCurrentGeneration(admission, gathered); + return projectCatalogOnlyOutcome(catalog, { @@ -416,8 +452,33 @@ implementation in `src/codex/convergence.ts`: + + return coordinateLegacyFullBehavior(request); +} ++ ++export const convergeCodex: ConvergeCodex = (request) => ++ convergeCodexInContext(request, {}); ++ ++export function createManagementConvergeCodex( ++ config: Readonly, ++): ConvergeCodex { ++ return (request) => convergeCodexInContext(request, { catalogConfig: config }); ++} ++ ++function captureCatalogAdmissionSnapshot( ++ request: ConvergeRequest, ++ config: Readonly, ++): CatalogAdmissionSnapshot { ++ return { ++ config, // exact captured reference; never a global and never a persisted re-read ++ generation: readRequiredConfigGeneration(request), ++ targets: captureCatalogTargets(config), ++ }; ++} ``` +`catalogContextMissingOutcome` is a typed no-write failure for an accidental naked +`convergeCodex({scope:"catalog", ...})` call. It does not recover by consulting disk +or process state. Production management never reaches it because the bound factory +is the only catalog-scoped construction path. + `projectCatalogOnlyOutcome` reports the actual catalog/cache/backup result and synthesizes history and observed fields as no-change/not-evaluated. It never calls config injection, profile, journal, history, restoration, or WP12 observation. @@ -431,10 +492,16 @@ full native convergence before its safety phases existed. Delete `refreshCodexCatalogBestEffort` from `src/server/management-api.ts:105-112` and -`src/server/management/context.ts:54-69`. Replace it with one injected production -funnel: +`src/server/management/context.ts:12,68`. Replace it with one injected production +factory plus the bound funnel. The dependency seam takes a factory, rather than an +already-bound function, so tests can assert `configArg === config` at construction: ```diff +- refreshCodexCatalog?: () => Promise; ++ createManagementConvergeCodex?: ( ++ config: Readonly, ++ ) => ConvergeCodex; + - refreshCodexCatalogBestEffort: () => Promise; + convergeCodex: (request: ConvergeRequest) => Promise; ``` @@ -447,10 +514,25 @@ funnel: - await refreshCodexModelCatalog(config); - } catch { /* catalog absent */ } - } -+ const converge = deps.convergeCodex -+ ?? (await import("../codex/convergence")).convergeCodex; ++ let boundConvergeCodex: ConvergeCodex | undefined; ++ async function convergeCodex( ++ request: ConvergeRequest, ++ ): Promise { ++ if (!boundConvergeCodex) { ++ const create = deps.createManagementConvergeCodex ++ ?? (await import("../codex/convergence")).createManagementConvergeCodex; ++ boundConvergeCodex = create(config); ++ } ++ return boundConvergeCodex(request); ++ } ``` +The lazy bind preserves today's behavior for management requests that never refresh +the catalog: they do not load catalog convergence. On the first catalog mutation, +`src/server/management-api.ts:105-112`'s in-scope `config` object is passed by +identity into the factory, retained in its closure, and handed unchanged to +`captureCatalogAdmissionSnapshot`; gather then receives it as `admission.config`. + Each of the 16 current awaits — provider 6 (`src/server/management/provider-routes.ts:147,338,487,512,527,546`), model 6 (`src/server/management/model-routes.ts:214,313,352,390,404,440`), combo 2 @@ -524,9 +606,12 @@ generation owners: graph (static imports, dynamic imports, aliases, and re-exports) and prove every direct writer in `src/codex/internal/catalog-commit.ts` is reachable only from `convergence.ts`. -- Drive all 16 real management routes with an injected `convergeCodex`, assert one - `scope: "catalog"` call using the same config object as today's callback, preserve - each primary 2xx/201, and observe the additive `catalogRefresh`. +- Drive all 16 real management routes with an injected convergence factory, assert + its construction argument is reference-equal to the exact config + object passed to `handleManagementAPI`, assert one `scope: "catalog"` call, preserve + each primary 2xx/201, and observe the additive `catalogRefresh`. A gather spy also + asserts `admission.config` is that same reference, proving the entire path rather + than only the factory boundary. - For every management route, inject spies that fail on config/profile/journal/history writes and assert zero calls; assert history and observed result sections are the contract's no-change/not-evaluated projection. diff --git a/devlog/_plan/260804_codex_write_substrate/020_history_isolation.md b/devlog/_plan/260804_codex_write_substrate/020_history_isolation.md index 7d42878a6..8e97d5ae8 100644 --- a/devlog/_plan/260804_codex_write_substrate/020_history_isolation.md +++ b/devlog/_plan/260804_codex_write_substrate/020_history_isolation.md @@ -51,7 +51,7 @@ IN: - `src/codex/convergence.ts` (MODIFY) — add history execution behind the existing `convergeCodex`; callers still use only the contract request/result. - `src/codex/transition-state.ts` (MODIFY only through its public API) — WP10 calls - `readCodexTransitionState` and `updateCodexTransitionState`; the latter conditionally + `readCodexTransitionState` and `updateCodexHistoryTransition`; the latter conditionally updates the pending history schedule where the native pair and `history_tx_id` still match. It never stores that pair or schedule in `integrations/codex.json`. - `src/codex/inject.ts`, `src/codex/sync.ts` (MODIFY) — remove direct history @@ -291,7 +291,7 @@ UPDATE codex_transition_state The coordinator database path already encodes effective user plus canonical `CODEX_HOME`; the row is deliberately a singleton, not one row per -`OPENCODEX_HOME`. `updateCodexTransitionState(expected, next)` executes the statement +`OPENCODEX_HOME`. `updateCodexHistoryTransition(expected, state)` executes the statement above. Its `kind:"updated"` result means exactly one changed row published A's result; the implementation maps zero changed rows to `kind:"conflict"`. Conflict means A was overtaken: it MUST NOT write JSON, MUST NOT overwrite or clear the newer @@ -322,8 +322,8 @@ Outcome order: - readable unsupported shape -> `unknown/schema` with both probe counts null; - watchdog -> `unknown/timeout`, not `worker-died`; - shutdown cancellation -> `unknown/shutdown-cancelled`, join, then drain; -- `worker.onerror`, malformed terminal message, or early close -> - `unknown/worker-died`; +- `worker.onerror`, malformed terminal message, or early close -> reread the + coordinator row before attempting `unknown/worker-died`; - initial pair/snapshot mismatch or zero-row terminal update -> `pending/overtaken`, no self-retry; - coordinator update failure -> returned `unknown/record-write-failed`; the existing @@ -334,6 +334,20 @@ repository's existing discipline (`src/storage/worker-lifecycle.ts:150-209`). A watchdog is containment, not convergence. It may interrupt legitimate large history, so timeout can never be recorded as success. +The reread on `worker.onerror`, malformed terminal IPC, or early close is mandatory +because the Worker may have committed its terminal SQLite update and died before +`postMessage`. The parent first calls `readCodexTransitionState`. If the row still +matches the job's native pair and `history_tx_id` and already contains a terminal +history state, that durable state is the result and the parent writes nothing. If a +newer pair owns the row, the parent returns `pending/overtaken` and arms the winner's +schedule. Only when the exact job still owns a `pending` or `running` row may the +parent conditionally call +`updateCodexHistoryTransition(expected, workerDiedState)`; a zero-row result follows +the same overtaken rule. If the reread is unavailable, the parent leaves the row +intact, returns `unknown/record-write-failed`, and lets the guardian retry from +durable state. A missing terminal message is therefore never permission to +overwrite a committed success with synthetic `worker-died`. + ## Fail-fast automatic mode and explicit mode The provider currently uses a mutable global 5,000 ms busy timeout and two retries @@ -373,7 +387,7 @@ belong to the SQLite coordinator row keyed by canonical `CODEX_HOME`, not to an Both `history-worker.ts` and `history-job.ts` consume the contract-owned coordinator API from `src/codex/transition-state.ts`; neither owns SQL or a second row shape. The Worker calls `readCodexTransitionState` before traversal and -`updateCodexTransitionState(expected, next)` after its post-probe. The parent/guardian +`updateCodexHistoryTransition(expected, state)` after its post-probe. The parent/guardian uses the same reader to arm the current schedule after conflict. They never parse, write, or atomically replace `integrations/codex.json`. A terminal @@ -485,10 +499,7 @@ unknown rather than zero-looking success. + const result = syncCodexHistoryProvider(request.targetProvider, request.stateDbPath, request.backupPath, policy(request)); + const postProbe = countPendingOpencodexHistory(request.stateDbPath, request.backupPath); + const state = classifyHistoryState(result, postProbe, request.expectation.txId); -+ const update = updateCodexTransitionState(expected, { -+ ...expected, -+ history: state, -+ }); ++ const update = updateCodexHistoryTransition(expected, state); + if (update.kind === "conflict") return postOvertaken(request, postProbe); + if (update.kind === "unavailable") return postRecordWriteFailed(request, postProbe); + self.postMessage({ type: "done", requestId: request.requestId, state, postProbe, expectation: request.expectation, authoritySnapshotId: request.authoritySnapshotId }); @@ -502,6 +513,12 @@ unknown rather than zero-looking success. `release()` above is private to Worker implementation; unlike the native public API, no caller can retain it across unrelated work. +The parent terminal handler does not map missing IPC directly to `worker-died`. +Its error/close branch performs the reread rule above after join: adopt a matching +terminal row, arm a newer winner, or conditionally publish `worker-died` only while +the exact job still owns `pending`/`running`. This branch is tested at the seam +between the Worker's successful SQLite commit and `postMessage`. + ### Convergence dispatch, no inline branch ```diff @@ -568,7 +585,10 @@ satisfy C15. - Restart/module reload re-arms unresolved state. - Worker error, malformed response, early close, watchdog, cancellation, and final coordinator-row write failure retain their distinct contract reasons, carry nullable - probe counts, remain non-converged, and join exactly once. + probe counts, remain non-converged, and join exactly once. For + error/malformed/close, the parent rereads first: an already-terminal matching row + wins; only a matching `pending`/`running` row may be conditionally changed to + `worker-died`. - An overtaken transition does not retry itself. ### Measured responsiveness — C3 diff --git a/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md b/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md index 6824c887e..bdd4fbbd7 100644 --- a/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md +++ b/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md @@ -17,14 +17,14 @@ contract's exact `AdmissionSnapshot`; and the pinned Bun 1.3.14 probe showed bot Effective-user identity — uid on POSIX, SID on Windows — is the namespace authority (`005_contract.md` §§4, 7). -WP11 is independently landable. It consumes WP8b's identity/generation/types, +WP11 is independently landable. It consumes WP8b's identity/transition-state/types, WP9's synchronous candidate commit, and WP10's separate history protocol. The WP11 commit typechecks and preserves the working WP9/WP10 funnel. WP12 later supplies stronger ownership/provenance decisions through the same `AdmissionSnapshot`; it is not required to replace a placeholder before this phase works. All current-code citations and diff context below were rechecked on 2026-08-04 at -`2d5e080dea3e7000bf2111b381c7c1a3c4f5fb11`. +`7bde9e0c977721fc0b9d8617c85ff17de7c07658`. ## IN / OUT @@ -35,10 +35,11 @@ IN: coordinated commit, release, and typed lock mechanics. - `src/codex/convergence.ts` (MODIFY) — place WP9's fixed catalog/native commit under the new lock and pass the contract `AdmissionSnapshot`/`CommitExpectation`. -- `src/codex/generation.ts` (MODIFY through its public owner API) — allocate and - verify native expected transitions; no parallel counter. -- `src/codex/integration-record.ts` (MODIFY through `updateIntegrationRecord`) — - persist native generation/tx identity inside the synchronous coordinated section. +- `src/codex/transition-state.ts` (MODIFY through its public owner API) — lend + WP11 a narrow opaque capability backed by the already-open coordinator + transaction; this module remains the sole native-generation/transition-row owner. +- `src/codex/integration-record.ts` (consumed through `updateIntegrationRecord`) — + persist provenance/non-CAS JSON only inside the synchronous coordinated section. - `src/codex/native-main-lock-file.ts` (MODIFY) — reuse stable descriptor and substitution checks; add only a caller-supplied ACL deadline cap. - `src/lib/windows-secret-acl.ts` (MODIFY) — accept a stricter remaining deadline; @@ -54,7 +55,11 @@ OUT: `UserIdentity` from the contract modules (`005_contract.md` §§1-4, 7). The native-lock result below remains owned by this lock module; it is a mechanism result projected by `convergence.ts`, not a competing convergence union. -- History mutation/locking. WP10's history lock is a sibling and is never nested. +- History mutation/locking. WP10 owns H and its two short fail-fast H->N + operations; WP11 only ensures N is released before history dispatch. +- Transition-row schema, generation allocation, or JSON transition state. Those + belong to `transition-state.ts`; `integrations/codex.json` contains provenance + and extensions, never `nativeGeneration`, `currentTxId`, or history scheduling. - Provider gathering or any awaited history work inside the native held section. - Desired-state, service ownership, external-provider, journal, and provenance policy — WP12. WP11 compares snapshots and enforces order; it does not decide @@ -76,6 +81,7 @@ them in `src/codex/codex-write-lock.ts`; it does not publish the former ```ts import type { AdmissionSnapshot, + BeginCodexTransition, CommitExpectation, } from "./convergence-types"; @@ -119,6 +125,20 @@ export interface CodexWriteCommitContext { readonly lockId: string; readonly admission: AdmissionSnapshot; readonly expectation: CommitExpectation; + /** + * Opaque authority over the ALREADY-OPEN BEGIN IMMEDIATE transaction N. + * It exposes one conditional row operation, not SQLite or transaction control. + */ + readonly coordinator: CodexCoordinatorTransaction; +} + +const codexCoordinatorTransactionBrand: unique symbol = Symbol( + "CodexCoordinatorTransaction", +); + +export interface CodexCoordinatorTransaction { + readonly [codexCoordinatorTransactionBrand]: true; + readonly beginTransition: BeginCodexTransition; } type Synchronous = T extends PromiseLike ? never : T; @@ -139,7 +159,14 @@ export type WithCodexWriteLock = ( `CodexWriteLockResult` is the lock module's own bounded mechanism result. `convergence.ts` exhaustively projects it into `ConvergeOutcome`; no route consumes -it directly. There is no public handle or release method. The conditional return rejects ordinary `async` callbacks at typecheck; +it directly. `CodexCoordinatorTransaction` is the only handle passed to the +callback. It is branded and exposes only the contract's null-safe conditional +transition-row update. It is one-shot for this transition, and WP11 verifies that +it returned `updated` for the exact expectation before allowing C to release. It +exposes neither the `Database` object nor `COMMIT`, +`ROLLBACK`, or `close`. Opening another connection in the callback is wrong: it +would contend with WP11's own `BEGIN IMMEDIATE` instead of updating through N. +The conditional return rejects ordinary `async` callbacks at typecheck; the implementation also detects a cast thenable, rolls back, and throws a `TypeError`. Provider I/O, subprocesses, serialization, history walking, retry sleeps, and any other awaitable work are forbidden beneath `commit`. @@ -159,12 +186,13 @@ validate/open stable DB; BEGIN IMMEDIATE native lock held withConfigMutationLockSync config lock held authoritative readAdmissionUnderLock() fresh snapshot compare digest + config generation + intent + ownership - allocate CommitExpectation (N -> N+1, this txId) - commit(context) synchronous - updateIntegrationRecord(nativeAfter + txId + section edits) - verify exact expected transition + read transition pair and form CommitExpectation from row in N + commit(context with opaque N capability) synchronous + native writes + provenance-only integration-record update + conditional transition-row update through the same N + verify exact nativeAfter + txId on the still-open N release config lock -assert stable lock path; ROLLBACK; close DB + side fd +assert stable lock path; COMMIT N; close DB + side fd ``` This replaces the former two generic admission callbacks. The first @@ -175,10 +203,16 @@ a boolean or manufacture an authority receipt. `withConfigMutationLockSync` is already synchronous, fail-fast, and reentrant only for the current synchronous stack (`src/config.ts:1767-1818`). The native lock may hold it because no await occurs. Config-generation reads/updates and -`updateIntegrationRecord` happen before that callback returns. The native -generation bump and `txId` are persisted in the same record update as the native -commit result, so another cooperating writer cannot observe moved native bytes with -an old generation. +provenance-only `updateIntegrationRecord` calls happen before that callback returns. +The native generation bump, `txId`, and pending history schedule are owned by the +transition row and are conditionally updated through the capability backed by N. +WP11 verifies that exact row before C returns, releases C, and then commits N. + +The previous version ended the successful path with `ROLLBACK`. That was wrong: it +discarded the transition row the whole design depends on, so a successful native +commit became indistinguishable from an unrecorded partial write and stale Workers +could not be rejected. `ROLLBACK N` remains only for callback failure, failed row +update/verification, cast-thenable rejection, or another refusal before commit. If the config coordinator is busy, the attempt releases the native lock and retries only while the outer monotonic deadline remains; deadline expiry returns typed @@ -190,7 +224,7 @@ explicitly says the filesystem lacks. ## Canonical `CODEX_HOME` identity — C6 1. Select nonblank explicit `codexHome`, else nonblank `process.env.CODEX_HOME`, - else `defaultCodexHome()` (`src/codex/home.ts:121-146`). Blank explicit input is + else `defaultCodexHome()` (`src/codex/home.ts:135-146`). Blank explicit input is a programmer error. 2. Expand only leading `~`, resolve absolute, and require an existing directory. Missing/non-directory refuses before identity namespace work. @@ -216,25 +250,24 @@ Delete `homedir()` from the import list and delete the prior `005_contract.md` §7 proves both home accessors can be changed by `HOME`; using `os.userInfo().homedir` would preserve the defect. -Consume `UserIdentity` and the resolver from `src/codex/user-identity.ts`: +Consume `UserIdentity` and the one final-path resolver from +`src/codex/user-identity.ts`: ```ts import { resolveEffectiveUserIdentity, - resolveOsRuntimeDirectory, + resolveCodexCoordinatorDatabasePath, } from "./user-identity"; ``` -The exact path is: - -```text -/opencodex/native-write-locks/v1//.sqlite -``` - -`` is encoded from `{ platform:"posix", uid }` or -`{ platform:"win32", sid }`; it is never username, `HOME`, `USERPROFILE`, -`CODEX_HOME`, or `OPENCODEX_HOME`. This matches `005_contract.md` §7. WP8b's -identity resolver is the sole platform owner; WP11 does not add a second SID lookup. +Call `resolveEffectiveUserIdentity()`, then pass that identity and the canonical +`CODEX_HOME` to `resolveCodexCoordinatorDatabasePath(...)`. Its return value is the +**final database path** and is consumed verbatim. WP11 does not import +`resolveOsRuntimeDirectory`, encode uid/SID, hash the home for path construction, +or append `opencodex`, `native-write-locks`, a version, or `.sqlite`. The prior +version reconstructed those segments locally; that was wrong because it let the +lock holder and transition-state callers open different databases despite the +contract's single resolver (`005_contract.md:861-877`). ### Component validation @@ -264,21 +297,21 @@ the holder and contenders reach their deadline. ### Core new-module diff ```diff -+import { createHash } from "node:crypto"; +import { lstatSync, mkdirSync, realpathSync, statSync } from "node:fs"; -+import { join, resolve, win32 } from "node:path"; ++import { resolve, win32 } from "node:path"; +import { AsyncLocalStorage } from "node:async_hooks"; +import { Database } from "bun:sqlite"; + +import { withConfigMutationLockSync } from "../config"; +import { updateIntegrationRecord } from "./integration-record"; -+import { resolveEffectiveUserIdentity, resolveOsRuntimeDirectory } from "./user-identity"; ++import { ++ resolveCodexCoordinatorDatabasePath, ++ resolveEffectiveUserIdentity, ++} from "./user-identity"; + -+function lockDatabasePath(canonicalHome: string): string { ++function coordinatorDatabasePath(canonicalHome: string): string { + const identity = resolveEffectiveUserIdentity(); -+ const identityPart = encodeUserIdentity(identity); -+ const homeId = sha256(LOCK_DOMAIN + canonicalHome); -+ return join(resolveOsRuntimeDirectory(identity), "opencodex", "native-write-locks", "v1", identityPart, `${homeId}.sqlite`); ++ return resolveCodexCoordinatorDatabasePath(identity, canonicalHome); +} ``` @@ -300,44 +333,65 @@ A separate task is an ordinary contender. Caller exceptions propagate after rollback/release; they are never converted to busy/refused. ```diff ++const transaction = openCodexCoordinatorTransaction(finalDatabasePath); ++// transaction has already executed BEGIN IMMEDIATE: N is held here. +const value = withConfigMutationLockSync(() => { + const current = options.readAdmissionUnderLock(); + assertAdmissionStillCurrent(options.admitted, current); -+ const expectation = beginExpectedNativeTransition(); -+ const result = commit({ canonicalCodexHome, lockId, admission: current, expectation }); -+ updateIntegrationRecord(record => commitExpectedTransition(record, expectation, result)); -+ assertExpectedTransition(readIntegrationRecord(), expectation); ++ const expectation = transaction.expectation(); ++ const result = commit({ ++ canonicalCodexHome, ++ lockId, ++ admission: current, ++ expectation, ++ coordinator: transaction.capability, ++ }); ++ transaction.assertPublished(expectation); + return result; +}); ++transaction.assertStablePath(); ++transaction.commit(); ``` -The commit callback performs no logging or response shaping. Those occur after both -locks release. +`openCodexCoordinatorTransaction` is a transition-state owner API, not a second +SQLite implementation in WP11. Its controller retains commit/rollback/close and +path-stability authority; only `transaction.capability` crosses into the callback. +`commit` must perform the conditional `beginTransition` after its native/provenance +writes. `assertPublished` rejects zero-row/conflict/unavailable or the wrong exact +pair before C is released. The callback performs no logging or response shaping. +Those occur after both locks release. ## Deadlock order and sibling history sequence Legal order: ```text -native lock - -> config mutation lock +native/coordinator transaction N (BEGIN IMMEDIATE) + -> config transaction C -> authoritative AdmissionSnapshot re-read -> config generation read/update when config changes -> synchronous native commit - -> integration-record native generation + txId update - -> release config --> release native + -> provenance-only integration-record update + -> conditional transition-row update through already-open N + -> release C + -> COMMIT N +-> release N history lock (later, in Worker) - -> reject stale CommitExpectation / authoritySnapshotId - -> manifest + rollouts + DB + post-probe + history record + -> fail-fast coordinator N claim check; if busy, release H and retry + -> manifest + rollouts + DB + post-probe while holding H, not N + -> fail-fast coordinator N terminal CAS; if busy, release H and retry -> release history ``` -The native and history locks are **not nested**. Native releases before the Worker -acquires history; history never acquires native/config. A stale history job is -generation/transaction-rejected before mutation, so sibling sequencing cannot let -an old ON job overtake a newer OFF transition (`005_contract.md` §6). +The complete order is `N -> C` plus a short `H -> N`. There is no `C -> N`, no +`C -> H`, and no held `N -> H`. Native releases N before dispatching history. At +claim and terminal boundaries a Worker may hold H while attempting only a +fail-fast conditional operation on N; `SQLITE_BUSY` releases H and retries, so it +never waits while preserving the edge. Between those boundaries history traversal +holds H alone. A stale history job is generation/transaction-rejected before +mutation or loses the terminal CAS, so it cannot overwrite the winner's durable +schedule (`005_contract.md:780-811`). Never call `withCodexWriteLock` from inside `withConfigMutationLockSync` or a `mutatePersistedConfig` callback. Current inverse-edge search found config-owned @@ -422,8 +476,8 @@ probe. Do not substitute Node or a same-process environment mutation. malformed DB, ACL failure/timeout, unsupported filesystem all refuse without repair/deletion. - Windows CI executes real SID/junction/ACL success; POSIX executes real uid/mode. -- Dependency graph proves no inverse config->native acquisition and no history/native - nesting. +- Dependency graph proves no inverse C->N or C->H acquisition and no held N->H; + history's only H->N edges are the fail-fast claim and terminal operations. ## Verification @@ -458,10 +512,11 @@ syncs, restores, or ensures the proxy; port 10100 is untouched. - **C7/C18** — namespace keys on effective uid/SID beneath the OS runtime directory, never any home accessor. Real pinned-Bun children with independently varied HOME and USERPROFILE prove one lock for one user/home. -- Config generation, authoritative admission re-read, native commit, expected - native generation/txId, and integration-record updates share the synchronous - native->config section. -- Native and history locks are never nested; stale history jobs are rejected by - generation/transaction identity. +- Config generation, authoritative admission re-read, native/provenance writes, + and the conditional transition-row update share N->C; C releases before N commits. +- `transition-state.ts` alone owns native generation/txId/history scheduling; JSON + owns none of them, and WP11 never opens a second coordinator connection in C. +- Lock edges are N->C and short fail-fast H->N only; stale history jobs are rejected + by generation/transaction identity without any C->N, C->H, or held N->H edge. - **N2** — WP11 extends the already-working funnel and typechecks/preserves behavior at its own commit; WP12 strengthens admission without supplying missing mechanics. diff --git a/devlog/_plan/260804_codex_write_substrate/040_ownership_convergence.md b/devlog/_plan/260804_codex_write_substrate/040_ownership_convergence.md index 98b1b0b5b..b8b26bf95 100644 --- a/devlog/_plan/260804_codex_write_substrate/040_ownership_convergence.md +++ b/devlog/_plan/260804_codex_write_substrate/040_ownership_convergence.md @@ -210,14 +210,19 @@ not to `integrations/codex.json`. 7. If intent is ON, WP9 gather receives `admission.config` — **that exact object**. OFF does not gather. -8. Call WP11 with `admitted: admission`. Under native->config coordination, - authoritatively re-read steps 1-6 into a second `AdmissionSnapshot` and compare - digest, generation, intent, ownership, external provider, canonical targets, - journal identity, provenance identity, and the recomputed authority snapshot ID. -9. Recover an authorized dead journal, establish baselines, commit apply/remove, - conditionally write the expected native generation/`txId` and pending history - schedule in the CODEX_HOME-keyed coordinator row, and inspect observed state - inside the coordinated section. Release before logging/HTTP shaping. +8. Call WP11 with `admitted: admission`. WP11 opens native/coordinator transaction + `N` with `BEGIN IMMEDIATE`, then acquires config transaction `C` **while holding + N**. With both held, authoritatively re-read steps 1-6 into a second + `AdmissionSnapshot` and compare digest, generation, intent, ownership, external + provider, canonical targets, journal identity, provenance identity, and the + recomputed authority snapshot ID. +9. Still inside `N -> C`, reclassify every artifact whose classification can + authorize a write, recover an authorized dead journal, establish baselines, + commit apply/remove, persist provenance, and conditionally install the expected + native generation/`txId` plus pending history schedule. Every transition-row + read/write uses the **already-open N connection**; opening a second coordinator + connection inside this section would contend with its own `BEGIN IMMEDIATE`. + Release `C`, then `COMMIT N`, and only afterward inspect/log/shape HTTP output. 10. Run WP10 history afterward under its sibling lock with the same `CommitExpectation` and authority snapshot identity; stale jobs are rejected. @@ -242,6 +247,35 @@ claim prevention. Regather/retry ends at `deadlineMs`, after which unresolved wo is named. This distinction is the correction required by audit #5 and `005_contract.md` §3. +### Every classify-then-write path uses the same exclusion + +The old plan protected the final config comparison but left other validation +results usable after their exclusion had ended. That was wrong: a classification +is write authority only for the bytes and coordinator state observed while the +writer still excludes cooperating mutation. + +The invariant applies to every WP12 sequence, not only adoption: + +1. Open native/coordinator transaction `N` with `BEGIN IMMEDIATE`. +2. Acquire config transaction `C` while holding `N` whenever config bytes, + generation, routing, canonical targets, or config-derived authority participate. +3. On the already-open `N` connection, authoritatively reread the transition row; + then reread/reclassify service authority, external routing, journal liveness, + integration-record/provenance identity, artifact baselines, current post-images, + and structural-removal evidence immediately before the writes they authorize. +4. Under that same `N -> C` exclusion, perform the provenance update, journal + recovery, baseline capture, apply/restore/unlink, and transition-row operation. + Release `C`, then `COMMIT N`. Failure or disagreement rolls back `N` and writes + nothing; logging, retries, history dispatch, and response shaping happen later. + +Thus pre-gather admission and any operator-facing pre-lock classification are only +candidate observations. They may refuse early, but they never authorize a write. +Baseline capture -> provenance write -> artifact write, current-post-image +classification -> restore/unlink, dead-journal classification -> recovery, and +authority/provenance classification -> any JSON/native write all repeat their final +classification inside the transaction above. This is the common exclusion required +for every classify-then-write sequence in this phase. + ## External `model_provider` is a separate veto — C9 Service-home ownership answers who claims this OpenCodex installation. It does not @@ -281,7 +315,14 @@ and restore unions. Import the section types: WP12 writes only `record.provenance` through `updateIntegrationRecord`. The JSON record keeps exactly `version`, the provenance ledger, and unknown extension keys -at the record, ledger, and entry levels. It does **not** keep +at the record, ledger, entry, artifact, and baseline levels. Passthrough is +recursive: every `CodexArtifactId` object variant and both baseline object variants +(`absent` and `present`) accept unknown keys, validators preserve each unknown value +verbatim (including nested objects/arrays), and an older writer changing a known +field must deep-equal preserve those extensions. The contract currently states and +tests only record/ledger/entry passthrough; audit round 5 #3 requires its artifact +and baseline types/validators to carry the same index-signature and preservation +rule. It does **not** keep `nativeGeneration`, `currentTxId`, a pending/running history schedule, retry ownership, or the next due time. Putting those fields here was wrong: two different coordinators could each serialize their own read/replace and still overwrite one another. @@ -303,6 +344,17 @@ native_generation = ? AND current_tx_id IS ?`; `updateCodexHistoryTransition` conditionally claims/completes/reschedules that same row and additionally matches `history_tx_id`. A zero-row update writes nothing. +Audit round 5 #new 6 exposes a contract-schema hole: SQL +`CHECK(history_direction IN ('apply', 'remove'))` accepts `NULL`. WP12's producing +invariant is stricter: every row with `native_generation > 0` carries a non-null +`history_direction` of `apply` or `remove`, and terminal history updates retain that +direction together with the matching transaction/authority metadata. The contract's +positive-generation `CHECK` in `005_contract.md` must be tightened to require +`history_direction IS NOT NULL` as well as membership in the two-value set. Until +that contract correction lands, convergence still refuses to produce or accept a +positive-generation row with a null direction. Scheduling is a SQLite transition-row +operation; it never writes scheduling state to the integration record. + That row is the transition/scheduling authority. `readIntegrationRecord()` supplies only provenance needed to prove an artifact baseline/post-image. Observation joins the coordinator pair/schedule with the JSON provenance at read time; neither source @@ -367,7 +419,7 @@ bytes may still route through OpenCodex, so OFF would later restore the routed b as the baseline forever. That mechanism was wrong. **INFERRED operator-recovery UX:** keep the explicit operator-only command, but make -adoption a read-only proof followed by a record write: +adoption a serialized proof-and-provenance transaction: ```text ocx restore --adopt-current-codex-baseline @@ -376,11 +428,13 @@ ocx restore --adopt-current-codex-baseline The flag is rejected in service/agent-driven/automatic contexts and requires an interactive confirmation naming the canonical Codex home. It requires the proxy stopped, owned service authority, no external provider, no journal envelope, and a -single read-only classification of every target as `native-clean | ocx-residue | -ambiguous`. **Only all `native-clean` may adopt.** `ocx-residue` and `ambiguous` both -abort before quarantine or record replacement; the command prints the exact surface -and evidence and asks the operator to clean/inspect it outside this flow. There is no -automatic “salvage by filename” fallback in WP12. +pre-lock read-only classification of every target as `native-clean | ocx-residue | +ambiguous`. That first observation exists to inform/refuse before lock acquisition; +it is not write authority. **Only all `native-clean` may proceed to the locked +revalidation.** `ocx-residue` and `ambiguous` both abort before quarantine or record +replacement; the command prints the exact surface and evidence and asks the operator +to clean/inspect it outside this flow. There is no automatic “salvage by filename” +fallback in WP12. The ledger is unavailable here, so positive residue proof comes only from the artifact's own structure: @@ -396,19 +450,35 @@ artifact's own structure: The ASCII `->` above is the documentation spelling; implementation compares the exact Unicode prefix already emitted at `src/codex/catalog/sync.ts:283,346`. -Once the same observation is all native-clean, `integration-record.ts` atomically -quarantines an unreadable JSON record to a timestamped sibling, preserving its bytes, -and `updateIntegrationRecord` creates only exact current `present`/`absent` -provenance baselines. A lost record has no quarantine source. Adoption changes no -Codex artifact. If `readCodexTransitionState()` returns a ready row, adoption -preserves its pair and complete history state. If the row is missing/ -`legacy-ambiguous`, only the same all-native-clean proof may initialize the contract -`{0,null}` row with `history.status:"unknown"`; it never imports a positive pair from -OPENCODEX_HOME-local legacy JSON. A subsequent explicit `convergeCodex` performs -apply/remove from the verified clean baseline. - -If any target changes between classification and JSON replacement, target identity -or digest validation fails and adoption writes nothing. The command prints the +The previous version validated native-clean state and wrote provenance across an +unlocked gap. A CLI convergence could change an artifact after final validation but +before JSON replacement or transition-row initialization, permanently recording a +baseline for a state that no longer existed. OFF could later “restore” those wrong +bytes. That mechanism was unsafe and is replaced by this exact order: + +1. Acquire native/coordinator transaction `N` with `BEGIN IMMEDIATE`. +2. While holding `N`, acquire config transaction `C` where the all-surface proof + reads config/generation/routing (the normal adoption path does). +3. Using the **already-open N connection**, authoritatively reread the transition + row and every canonical target, then repeat the complete structural table above, + config/authority/journal checks, target identities, digests, and baseline bytes. +4. Only if that under-lock result is still exactly all `native-clean` and agrees + with the pre-lock observation may `integration-record.ts` quarantine unreadable + JSON, `updateIntegrationRecord` write exact current `present`/`absent` baselines, + and the same N connection initialize a missing row as contract `{0,null}` with + `history.status:"unknown"`. Release `C`, then `COMMIT N`. + +If any under-lock classification, identity, digest, authority, row state, or baseline +disagrees with the pre-lock observation, adoption refuses, rolls back/releases the +locks, reports the changed surface and both observations, and writes nothing. It does +not reclassify the new state as an acceptable baseline in the same invocation and does +not retry adoption automatically. + +A ready transition row is preserved byte-for-byte, including its pair, direction, +authority metadata, and complete history state/schedule. A lost record has no +quarantine source. Adoption changes no Codex artifact and never imports a positive +pair from OPENCODEX_HOME-local legacy JSON. A subsequent explicit `convergeCodex` +performs apply/remove from the verified clean baseline. The command prints the quarantine path and adopted provenance identity, never a newly invented `txId`. Tests never auto-confirm this action. @@ -434,7 +504,7 @@ proof and blocks automatic replay. +function reconcileJournalUnlocked( + inspection: AuthorizedDeadJournal, +): RestoreJournalResult { -+ // Called only by convergence inside the coordinated commit. ++ // Called only after convergence re-inspects under N -> C, before C release/N commit. +} ``` @@ -600,6 +670,13 @@ All tests use temporary homes, real contract record owner, port `0`, and product conditional row update cannot clear the newer pending schedule. 6. Two distinct `OPENCODEX_HOME` processes sharing one canonical `CODEX_HOME` observe one coordinator pair; exactly one expected-row update wins. +7. Instrument lock events for each classify-then-write path and assert + `BEGIN N -> acquire C -> authoritative classify/read/write on the same N + connection -> release C -> COMMIT N`; a second coordinator connection is never + opened inside the callback. +8. Seed a positive-generation row with null direction and require convergence to + reject it. Valid apply/remove rows retain their non-null direction through every + terminal history update; the tightened contract CHECK rejects the null fixture. ### Provenance/restoration/recovery @@ -621,12 +698,21 @@ All tests use temporary homes, real contract record owner, port `0`, and product adoption refuses instead of guessing. Table-drive every structural-proof row above, including exact clean negatives. 9. Only an all-native-clean observation may quarantine a corrupt JSON record and - write exact current baselines. Assert native bytes and the coordinator pair/ - pending schedule are unchanged; then normal `convergeCodex` succeeds. + write exact current baselines. Pause after the pre-lock observation, mutate one + target through a cooperating CLI convergence, then resume: under-lock + revalidation disagrees, so adoption reports/refuses and writes/quarantines/ + initializes nothing. Assert native bytes and the coordinator pair/pending + schedule belong to the winner; then a fresh, explicitly confirmed adoption may + start from a new pre-lock observation. 10. Missing transition row + all-native-clean initializes only `{0,null}` with - unknown history; a legacy positive JSON pair is never imported. + unknown history on the already-open N connection; a legacy positive JSON pair + is never imported. 11. Adoption aborts atomically on one unreadable or changed target, external provider, live writer, running proxy, or noninteractive/agent-driven invocation. +12. Seed unknown nested extension values independently on every artifact-id variant + and on both `absent`/`present` baseline variants. Exercise ordinary provenance + update, post-image update, restoration, and adoption; require deep-equal + preservation at record, ledger, entry, artifact, and baseline levels. ### Observed state and fresh intent @@ -685,7 +771,9 @@ live proxy on 10100. - **C10 (narrowed)** — two contract baseline classes only. Matching current post-images restore; current-byte drift preserves/reports. No hash claims to prove absence of edit-and-revert. Lost/corrupt ledger adoption requires a complete - verified native-clean observation; structural residue or ambiguity refuses. + verified native-clean observation repeated under `N -> C`; disagreement with the + pre-lock observation refuses and writes nothing. Artifact and baseline extension + keys survive recursively, not only record/ledger/entry keys. - **C11** — observed state is the contract `CodexObservedState`; unchanged intent still converges and re-observes. - **C12** — the same running server honors subprocess OFF then ON using a pre-gather @@ -696,7 +784,8 @@ live proxy on 10100. - There is one convergence entry point and one shared result family. - Native pair/pending schedule authority is the canonical-CODEX_HOME coordinator row. JSON retains only version, provenance, and extension keys, never transition - or scheduling authority. + or scheduling authority. Every positive generation has a non-null apply/remove + direction; the contract SQLite CHECK must enforce that invariant. - **N2** — WP12 rewires all remaining callers and passes its own typecheck/tests in the same commit. WP13 adds composed proof, not missing implementation. diff --git a/devlog/_plan/260804_codex_write_substrate/050_composed_acceptance.md b/devlog/_plan/260804_codex_write_substrate/050_composed_acceptance.md index 0ead30ffe..2f9401558 100644 --- a/devlog/_plan/260804_codex_write_substrate/050_composed_acceptance.md +++ b/devlog/_plan/260804_codex_write_substrate/050_composed_acceptance.md @@ -24,8 +24,9 @@ an unimplemented suite was run. IN: a future `tests/codex-composed-acceptance.test.ts` plus narrowly named child fixtures under `tests/helpers/`; the production CLI, server, management routes, convergence entry point, real filesystem, real Bun Workers, and real SQLite files. -Service-manager entry points are a separate disposable-host job; they are not part -of the developer-workstation invocation of this suite. +Service-manager entry points are a separate disposable-host runner at +`scripts/disposable-host/codex-service-composed-acceptance.ts`; they are not part +of the developer-workstation invocation or Bun's ordinary test discovery. OUT: mocks of `convergeCodex`, direct calls to phase-local gather/commit/history helpers as acceptance proof, the live proxy on port 10100, the user's homes, the @@ -125,11 +126,36 @@ service-manager registration API. P02/P04/P17 are seeded with startup/cleanup path cannot issue per-login-session `launchctl setenv/unsetenv` (`src/server/system-env.ts:251-258,364-391`). -"Disposable" means a throwaway VM/OS host and a **throwaway OS account**, not a temp -home on a developer account. Before any service setup or row is run, the job must -prove all of the following for that account; an unavailable query, -permission error, nonempty registration, or existing artifact is a hard failure, not -a skip: +Negative evidence is not proof of disposability. The empty-service gate protects +this workstation because its existing service makes the gate fail, but a normal +developer account with no installed service would pass it. The disposable-host job +therefore requires a positive, image-provisioned sentinel **before it performs even +the read-only service queries below**. On POSIX the sentinel is the root-owned, +non-symlink regular file `/etc/opencodex-disposable-service-host-v1`; on Windows it +is the non-reparse regular file +`C:\ProgramData\OpenCodex\opencodex-disposable-service-host-v1` owned by `SYSTEM` +or `Administrators`. Its exact bytes are +`OPENCODEX_DISPOSABLE_SERVICE_HOST_V1\n`; only root/`SYSTEM`/`Administrators` may +write it. The runner checks path type, owner, write permissions/ACL, and exact bytes +at process start. Missing, redirected, broadly writable, or mismatched sentinel +state is a hard failure before any service query, setup, or lifecycle command. + +The runner is deliberately outside `bunfig.toml`'s `tests` root, has no +`.test.ts`/`.spec.ts` suffix, and is invoked only by the separately labelled CI +job as `bun run scripts/disposable-host/codex-service-composed-acceptance.ts`. +`bun run test`, bare `bun test`, and `bun test ./tests/` must not import or spawn it. +An ordinary repo-hygiene test reads `bunfig.toml`, `package.json`, and the test +import graph to assert those three commands remain rooted at `tests/` and have no +edge to the disposable runner. The disposable runner also proves its own sentinel +check ran before recording any platform query in its event ledger. Thus an explicit +positive marker and non-discoverability are the first gate; the absence checks below +remain a second gate, never the evidence that the host is disposable. + +"Disposable" still means a throwaway VM/OS host and a **throwaway OS account**, not +a temp home on a developer account. After the positive sentinel passes and before +any service setup or row is run, the job must prove all of the following for that +account; an unavailable query, permission error, nonempty registration, or existing +artifact is a hard failure, not a skip: - macOS: `launchctl list | awk '$3 == "com.opencodex.proxy" { print }'` prints nothing, and `test ! -e "$HOME/Library/LaunchAgents/com.opencodex.proxy.plist"` @@ -210,7 +236,8 @@ database lives under the fixed effective-user runtime root, not under the case r `/tmp/opencodex-runtime-v1-/native-write-locks/.sqlite` on POSIX and `/OpenCodex/Runtime/v1//native-write-locks/.sqlite` -on Windows (`005_contract.md:693-721`). Before spawning a child, the harness resolves +on Windows (`005_contract.md` §7, "The lock namespace has one environment-independent +root per effective user"). Before spawning a child, the harness resolves that exact path through the production identity/runtime resolver, checks that the hash input is the case's canonical `CODEX_HOME`, and requires the exact database and its `-journal`, `-wal`, and `-shm` sidecars to be absent. A pre-existing file fails @@ -234,8 +261,10 @@ Parameterize the 30 workstation-safe rows P01-P08, P11-P17, and P19-P33 in the ordinary suite. Run P09/P10/P18/P34-P36 only in the separately gated disposable-host job above. The workstation rows seed authorizing isolated state without any service artifact; the disposable P34-P36 setup installs its fixture service only after the -empty-registration gate. Invoke the real entry and read the integration record -transition id plus a recursive before/after manifest. Every native mutation must have +empty-registration gate. Invoke the real entry, read `integration-record.ts` for +provenance/extensions, read the transition id, native generation, and history +schedule/state through `transition-state.ts`, and capture a recursive before/after +manifest. Every native mutation must have exactly one admitted transaction; OFF entries must produce a removal transaction, not a skip. P20-P33 retain their existing primary 2xx/201 behavior and expose the contract disposition; P30/P33 still complete their Claude/Desktop follow-up. The two job manifests together, @@ -245,8 +274,10 @@ not either one alone, make the 36-row census. P17, P18, P24-P33, P35, and P36 visibly reach direct writers. The management rows also pass through the bare catch at `src/server/management-api.ts:105-112`, so no typed transaction can be observed. **GREEN:** all 36 rows yield either one recorded -transition or a typed no-write refusal/busy outcome, and the module graph has no -writer reachable outside `convergence.ts`. +transition or a typed no-write refusal/busy outcome, and the symbol-level module +graph enforces every domain row in `005_contract.md` §8's permitted-root table: +native/journal/catalog/provenance writes root at `convergence.ts`, history writes +root at `history-worker.ts`, and transition-state writes root only at those two. ### B — two-process race after approval, before commit @@ -380,8 +411,9 @@ replace B's pending schedule, and the live guardian makes B the clean terminal o ### I — retry beyond the old horizon, without restart -Seed the sole record with unresolved current history at `attempts: 60` and -`nextRetryAt` due now, then start one P02 server and keep it alive. Hold the real DB +Seed the canonical `transition-state.ts` coordinator row with unresolved current +history at `attempts: 60` and `nextRetryAt` due now, then start one P02 server and +keep it alive. Hold the real DB through the first retry, observe the attempt advance beyond 60 and another finite timer remain armed, then release. Wait no longer than one exported production backoff cap plus a 2 s watchdog and assert the same PID converges; no restart/module @@ -413,8 +445,10 @@ unchanged intent still repairs observed state. Sequence P02 apply, P07 remove, P08 apply, P18 remove, and P02 startup using the same isolated homes. After each transition, read the record only through its production -owner and assert history, provenance, generation, unknown top-level keys, and -unknown section keys survive. For an absent baseline, matching post-image removal +owner in `integration-record.ts` and assert provenance plus unknown record/ledger/ +entry extension keys survive. Separately read `transition-state.ts` and assert the +native generation, current transaction id, history observation, and schedule match +the same transition. For an absent baseline, matching post-image removal must restore absence. For a present baseline, restore exact bytes. Change current bytes after apply and require preservation/conflict. Corrupt or remove the record and require automatic refusal before mutation. @@ -483,7 +517,7 @@ fixture and entry point. | C13 | **Not provable through a production entry point.** Run typecheck, full suite, GUI lint, privacy scan, docs build, then require this composed suite's case manifest and red/green evidence | those static/broad gates can be green today while A-L are red; C13 alone proves none of C1-C12 | | C14 | A's workstation and disposable-host manifests together drive P01-P36; P20-P33 cover 14 route shapes/16 calls; module-graph reachability and transition receipts must agree | 16 management call sites and multiple CLI/startup paths reach direct writers instead of one funnel | | C15 | P02/P07/P19 through H in both directions: B commits only after A's manifest/rollout post-images exist, B's pending schedule survives A's terminal conflict, and the guardian repairs B | only DB substeps are transactional; processes can overtake file writes and stale completion has no conditional terminal update | -| C16 | P02/P07/P08/P18/P17 sequence through K, preserving both optional sections and unknown keys | no shared record owner/schema exists | +| C16 | P02/P07/P08/P18/P17 sequence through K, preserving optional provenance and record/ledger/entry extensions through `integration-record.ts` while every phase reads the same transition/history row through `transition-state.ts` | no shared record owner/schema exists | | C17 | P19 through G with production config A→B→A and one-way parent retarget | no generation or stable target expectation exists; equal content passes | | C18 | P19 through E with independently different HOME and USERPROFILE in real children | no uid/SID lock exists, so environment-home variation does not contend | @@ -537,8 +571,10 @@ WP13 passes only when A-L are red on the pre-substrate revision for the named observable reason, green on the composed revision, all 36 production rows are in the paired workstation/disposable-host manifests, C1-C12 and C14-C18 have artifact-level evidence, C13 is separately green, every child/Worker is joined, the -service gate is empty before and after its job, and teardown removes only the suite's -explicit temporary roots plus the validated four-name lock-file allowlist. A missing -row, a same-process substitute for E/H, a mocked convergence function, a test that -passes on both revisions without a separate substrate-sensitive RED, or a green broad -suite beside any red composed case is a failure. +positive sentinel is verified before the disposable event ledger's first service +query, the negative service gate is empty before and after that job, ordinary test +commands cannot discover the disposable runner, and teardown removes only the +suite's explicit temporary roots plus the validated four-name lock-file allowlist. +A missing row, a same-process substitute for E/H, a mocked convergence function, a +test that passes on both revisions without a separate substrate-sensitive RED, or a +green broad suite beside any red composed case is a failure. From b97580fe552de14c8041a579c6c1e4930c82d158 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Tue, 4 Aug 2026 12:59:25 +0900 Subject: [PATCH 040/163] feat(codex): the shared substrate contract, as types MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Single owner for every surface WP9-WP13 share: the JSON record (provenance only, transition state deliberately absent), the history state and artifact ids, observed state, the catalog disposition, ConvergeCodex and its discriminated outcome, the two generation counters, CommitExpectation, both admission snapshots, and the effective-user identity and resolver types. Types only, by design. Two prior attempts failed audit because four phase documents each invented their share of these surfaces, and a runtime placeholder here would break the invariant that every phase typechecks and preserves behavior at its own commit — WP9 supplies the first ConvergeCodex implementation. Repo typecheck green. --- src/codex/convergence-types.ts | 323 +++++++++++++++++++++++++++++++++ 1 file changed, 323 insertions(+) create mode 100644 src/codex/convergence-types.ts diff --git a/src/codex/convergence-types.ts b/src/codex/convergence-types.ts new file mode 100644 index 000000000..9bf0fca62 --- /dev/null +++ b/src/codex/convergence-types.ts @@ -0,0 +1,323 @@ +/** + * Shared types for the Codex safe-interruption write substrate. + * + * This module is the SINGLE owner of every surface WP9-WP13 share. Two prior + * attempts failed audit because four phase documents each invented their share + * of the record schema, the /api/sync contract, and the convergence entry + * point; the contract is centralized here so a consumer can only import. + * + * Design record: devlog/_plan/260804_codex_write_substrate/005_contract.md + * Audit trail: 006, 007, 008, 009, 010 audit syntheses in the same unit. + * + * TYPES ONLY. WP8b deliberately rewires nothing: WP9 supplies the first + * `ConvergeCodex` implementation. Every phase must typecheck and preserve + * behavior at its own commit, which a runtime placeholder here would break. + */ +import type { OcxConfig } from "../types"; + +/** + * The non-CAS JSON record for the Codex integration. + * + * ONE owner. WP12 writes provenance here through `updateIntegrationRecord` — + * never its own read/merge/write. Cross-process transition state is deliberately + * absent; it belongs to the CODEX_HOME-keyed SQLite row below. + * + * Provenance is OPTIONAL at v1. A record written before WP12 is valid, and + * unknown extension sections from a newer writer remain valid and preserved. + */ +export interface CodexIntegrationRecord { + version: 1; + provenance?: CodexProvenanceLedger; + /** Unknown keys from a newer writer survive every older-writer update. */ + readonly [extra: string]: unknown; +} + +export interface CodexHistoryState { + status: "converged" | "pending" | "running" | "blocked" | "unknown" | "not-evaluated"; + /** + * Why it is not converged, when it is not. These are terminal observations + * for one attempt, not reasons to collapse the durable retry schedule. + */ + reason?: + | "db-busy" + | "permission" + | "unreadable" + | "schema" + | "timeout" + | "shutdown-cancelled" + | "worker-died" + | "overtaken" + | "record-write-failed"; + attempts: number; + /** null means "no timer armed"; see 020 — it must never mean "never again". */ + nextRetryAt: string | null; + /** The transition this state belongs to, so an overtaken job is detectable. */ + txId: string | null; + /** null means the final probe could not produce a trustworthy row count. */ + pendingRows: number | null; + /** null means the final probe could not produce a trustworthy manifest count. */ + backupEntries: number | null; + /** Unknown keys from a newer writer, preserved verbatim. */ + readonly [extra: string]: unknown; +} + +/** + * Every mutable Codex artifact for which the provenance ledger can authorize a + * restore. Embedded config fragments share the `config` entry because they are + * committed and restored as one file. Dynamic history ids name the exact row or + * rollout whose semantic pre-image is retained. + */ +export type CodexArtifactId = + | { readonly kind: "config" } + | { readonly kind: "generated-profile" } + | { readonly kind: "active-catalog"; readonly canonicalPath: string } + | { readonly kind: "catalog-backup"; readonly form: "hashed" | "legacy"; + readonly canonicalPath: string } + | { readonly kind: "models-cache" } + | { readonly kind: "injection-journal" } + | { readonly kind: "history-row"; readonly stateDbId: string; readonly threadId: string } + | { readonly kind: "history-manifest"; readonly stateDbId: string; + readonly canonicalPath: string } + | { readonly kind: "history-manifest-entry"; readonly stateDbId: string; + readonly threadId: string } + | { readonly kind: "history-rollout"; readonly stateDbId: string; + readonly canonicalPath: string }; + +export interface CodexProvenanceEntry { + artifact: CodexArtifactId; + baseline: + | { kind: "absent" } + | { kind: "present"; sha256: string; bytesBase64: string }; + /** Hash of what WE wrote. null when the write did not complete. */ + postImage: string | null; + txId: string; + at: string; + /** Entry-level extensions are preserved, not only ledger/top-level keys. */ + readonly [extra: string]: unknown; +} + +export interface CodexProvenanceLedger { + entries: readonly CodexProvenanceEntry[]; + readonly [extra: string]: unknown; +} + +export type CodexArtifactObservation = + | "applied" + | "absent" + | "missing" + | "residue" + | "drifted" + | "unreadable" + | "invalid" + | "not-evaluated" + | "unknown"; + +/** + * Read-only proof of what Codex has now, not what persisted intent requests. + * `isApplied` is true only for aggregate `applied`; a partial surface can never + * be flattened into true. OFF is operationally converged only at `absent`. + */ +export interface CodexObservedState { + aggregate: "applied" | "absent" | "partial" | "external" | "blocked" | "not-evaluated"; + /** null only for a catalog-scoped request that deliberately did not observe. */ + isApplied: boolean | null; + desired: "on" | "off" | "unknown"; + /** null only when aggregate is `not-evaluated`. */ + converged: boolean | null; + authority: { + service: "owned" | "foreign" | "unknown"; + externalProvider: string | null; + }; + surfaces: { + config: CodexArtifactObservation; + profile: CodexArtifactObservation; + catalog: CodexArtifactObservation; + cache: CodexArtifactObservation; + journal: "absent" | "pending" | "live" | "invalid" | "unknown" | "not-evaluated"; + history: { + state: CodexHistoryState; + database: CodexArtifactObservation; + manifest: CodexArtifactObservation; + rollouts: CodexArtifactObservation; + }; + provenance: { + state: "verified" | "missing" | "conflict" | "unreadable" | "unknown" | "not-evaluated"; + nativeGeneration: number | null; + currentTxId: string | null; + }; + }; +} + +export type CatalogNotice = "provider-auth" | "provider-network" | "fallback"; + +/** Sanitized catalog fact safe to append to management mutation responses. */ +export type CatalogDisposition = + | { status: "committed"; changed: boolean; degraded: boolean; + notices: readonly CatalogNotice[] } + | { status: "skipped"; + reason: "not-requested" | "catalog-unavailable" | "busy" | "stale" | "refused"; + retryable: boolean } + | { status: "failed"; reason: "provider-auth" | "provider-network" | "disk"; + phase: "gather" | "commit"; retryable: boolean; partialWrite: boolean }; + +/** + * The ONLY way Codex-owned bytes are written. Startup, ensure, /api/sync, the + * CLI verbs and all 16 management mutation callbacks funnel here. + * + * The funnel is the point: admission, generation checks and the lock live in one + * place, so a new caller cannot forget them. Round 1's 16 callers each held + * their own path to a commit. + */ +export type ConvergeCodex = ( + request: ConvergeRequest, +) => Promise; + +export interface ConvergeRequest { + /** + * The caller says WHEN, never WHICH WAY. + * + * Round 2 N1: an `apply | remove` request let `/api/sync` skip while desired + * state was OFF instead of removing residue, which violates C11 and + * contradicts the rule that callers cannot supply desired state. The + * direction is derived from admitted persisted intent, full stop. + * + * `observe` writes nothing and is the status read. + */ + action: "converge" | "observe"; + /** + * WP9 management mutations use `catalog`; explicit/lifecycle convergence uses + * `full`. Scope limits work, but still never lets the caller choose direction. + */ + scope: "catalog" | "full"; + /** Why, for the record and for log attribution. */ + reason: "startup" | "ensure" | "api-sync" | "cli" | "management-mutation"; + /** Automatic callers fail fast and defer; explicit ones may wait. See §5. */ + mode: "automatic" | "explicit"; + deadlineMs: number; +} + +export type ConvergeOutcome = + | { kind: "catalog-only"; changed: boolean; + observed: CodexObservedState; catalogRefresh: CatalogDisposition; + history: CodexHistoryState } + | { kind: "converged"; direction: "applied" | "removed"; changed: boolean; + observed: CodexObservedState; nativeGeneration: number; + currentTxId: string; + catalogRefresh: CatalogDisposition; history: CodexHistoryState } + | { kind: "skipped"; reason: "already-converged"; + observed: CodexObservedState; catalogRefresh: CatalogDisposition; history: CodexHistoryState } + | { kind: "refused"; authority: "service-home" | "external-provider" | "journal" | "provenance"; + message: string; observed: CodexObservedState } + | { kind: "busy"; surface: "lock" | "history" | "config"; retryAfterMs: number } + | { kind: "deferred"; direction: "applied" | "removed"; changed: boolean; + unresolved: readonly UnresolvedSurface[]; + nativeGeneration: number; currentTxId: string; + observed: CodexObservedState; catalogRefresh: CatalogDisposition; history: CodexHistoryState } + | { kind: "failed"; surface: string; message: string }; + +/** + * Note what is NOT here: `desired-off`. Desired OFF is not a skip — it is a + * `converged` with `direction: "removed"`. That is round 2 N1: the old shape let + * a sync while OFF return "skipped" and leave routed residue on disk. + */ +export type UnresolvedSurface = + | "config" + | "native" + | "catalog" + | "cache" + | "journal" + | "provenance" + | "history"; + +/** Bumped by every cooperating CONFIG write. Owned by src/config.ts. */ +export interface ConfigGeneration { readonly value: number; } + +/** Bumped by every cooperating NATIVE commit. Owned by transition-state.ts. */ +export interface NativeGeneration { readonly value: number; } + +export type ConfigGenerationRead = + | { kind: "ready"; generation: ConfigGeneration } + | { kind: "unavailable"; reason: "busy" | "database" }; + +export type ConfigGenerationBump = + | { kind: "updated"; generation: ConfigGeneration } + | { kind: "conflict"; current: ConfigGeneration } + | { kind: "unavailable"; reason: "busy" | "database" }; + +export type ReadConfigGeneration = () => ConfigGenerationRead; +export type BumpConfigGeneration = (expected: ConfigGeneration) => ConfigGenerationBump; + +export interface CommitExpectation { + /** Read at admission. */ + readonly nativeBefore: number; + /** What OUR commit will produce. Always nativeBefore + 1. */ + readonly nativeAfter: number; + /** Identifies the commit that performed the bump. */ + readonly txId: string; +} + +/** The minimal, working WP8b/WP9 snapshot; it authorizes catalog work only. */ +export interface CatalogAdmissionSnapshot { + config: Readonly; + generation: number; + targets: Readonly<{ + catalog: string; + cache: string; + catalogBackups: readonly string[]; + }>; +} + +export interface AdmissionSnapshot { + config: Readonly; + configDigest: string; + intent: "on" | "off"; + generation: number; + ownership: "owned" | "foreign" | "unknown"; + externalProvider: string | null; + canonicalTargets: Readonly<{ + codexHome: string; + opencodexHome: string; + config: string; + profile: string; + catalog: string; + cache: string; + journal: string; + integrationRecord: string; + catalogBackups: readonly string[]; + historyDb: string; + historyManifest: string; + historyRollouts: readonly string[]; + }>; + journalIdentity: string; + provenanceIdentity: string; + /** Digest of every authority field above; passed to the history Worker. */ + authoritySnapshotId: string; +} + +/** + * Effective-user identity for the lock namespace. + * + * NOT a home path. Bun 1.3.14 returns an environment-controlled home from both + * os.homedir() AND os.userInfo().homedir, so any home-derived namespace can be + * split by a service and a CLI that see different HOME values — which defeats + * exclusion entirely, silently. + */ +export type UserIdentity = + | { platform: "posix"; uid: number } + | { platform: "win32"; sid: string }; + +/** + * Resolve the effective account from operating-system identity APIs only. + * Failure is a typed namespace refusal; username/home/environment fallback is + * forbidden because it can split one account across two lock databases. + */ +export type ResolveEffectiveUserIdentity = () => UserIdentity; + +/** + * Return the FINAL SQLite coordinator database path for this exact canonical + * CODEX_HOME. Consumers append no uid/SID, version, directory or filename. + */ +export type ResolveCodexCoordinatorDatabasePath = ( + identity: UserIdentity, + canonicalCodexHome: string, +) => string; From 5c176d72b1285ae0f56bb66686657e8a84fac1dd Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Tue, 4 Aug 2026 13:11:35 +0900 Subject: [PATCH 041/163] feat(codex): the substrate's four owned modules MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit user-identity.ts keys the coordination namespace on effective-user identity — uid on POSIX, account SID on Windows — and never on a home path. Bun 1.3.14 returns an environment-controlled home from BOTH os.homedir() and os.userInfo().homedir, so a service and a CLI seeing different HOME values would silently take different locks and defeat exclusion entirely. A failure is a typed refusal; there is no username, home or environment fallback. transition-state.ts owns {nativeGeneration, currentTxId} and the pending history schedule in a CODEX_HOME-keyed SQLite row with a conditional update. This replaced a JSON read/compare/replace that was never a CAS across two non-overlapping coordinators. A zero-row result reports conflict and commits no stale schedule. The private brand, factory and one-shot flag live here rather than in the lock module — the reviewer reproduced TS2741 for the other arrangement — and a positive generation cannot carry a null direction, enforced in both validation and SQL because SQLite CHECK accepts NULL. integration-record.ts is the sole writer of integrations/codex.json, now holding provenance only. Unknown keys survive at record, ledger, entry, artifact AND both baseline levels. Unparseable bytes fail closed instead of silently starting fresh — losing provenance quietly is how the Pi incident became possible. sync-response.ts is the one exhaustive /api/sync mapping, with a never-check so a new outcome variant without a row fails typecheck. There is no desired-off row: a converge while OFF returns converged with direction removed. 18 tests pass, repo typecheck green, privacy scan passed. --- src/codex/integration-record.ts | 266 ++++++++++++ src/codex/transition-state.ts | 576 +++++++++++++++++++++++++ src/codex/user-identity.ts | 186 ++++++++ src/server/management/sync-response.ts | 69 +++ tests/codex-integration-record.test.ts | 142 ++++++ tests/codex-sync-response.test.ts | 177 ++++++++ tests/codex-transition-state.test.ts | 147 +++++++ tests/codex-user-identity.test.ts | 54 +++ 8 files changed, 1617 insertions(+) create mode 100644 src/codex/integration-record.ts create mode 100644 src/codex/transition-state.ts create mode 100644 src/codex/user-identity.ts create mode 100644 src/server/management/sync-response.ts create mode 100644 tests/codex-integration-record.test.ts create mode 100644 tests/codex-sync-response.test.ts create mode 100644 tests/codex-transition-state.test.ts create mode 100644 tests/codex-user-identity.test.ts diff --git a/src/codex/integration-record.ts b/src/codex/integration-record.ts new file mode 100644 index 000000000..e595d81a5 --- /dev/null +++ b/src/codex/integration-record.ts @@ -0,0 +1,266 @@ +/** + * The sole reader/writer for integrations/codex.json. + * + * Provenance once disappeared when a corrupt record was treated as an empty one + * (005_disable_leaves_a_broken_file.md). Reads therefore validate the complete + * known shape, and updates refuse malformed bytes instead of manufacturing a new + * baseline over evidence we can no longer trust. + */ +import { mkdirSync, readFileSync } from "node:fs"; +import { dirname, join } from "node:path"; +import { atomicWriteFile, getConfigDir, withConfigMutationLockSync } from "../config"; +import type { + CodexArtifactId, + CodexIntegrationRecord, + CodexProvenanceEntry, + CodexProvenanceLedger, +} from "./convergence-types"; + +const RECORD_FILENAME = "codex.json"; +const LEGACY_TRANSITION_KEYS = new Set([ + "nativeGeneration", + "currentTxId", + "generation", + "history", + "historySchedule", +]); + +function recordPath(): string { + return join(getConfigDir(), "integrations", RECORD_FILENAME); +} + +function isPlainRecord(value: unknown): value is Record { + return typeof value === "object" && value !== null && !Array.isArray(value); +} + +function isMissingPathError(error: unknown): boolean { + return error instanceof Error && "code" in error && (error as NodeJS.ErrnoException).code === "ENOENT"; +} + +function invalid(message: string) { + return { kind: "invalid" as const, message }; +} + +function validateArtifact(value: unknown): value is CodexArtifactId { + if (!isPlainRecord(value) || typeof value.kind !== "string") return false; + switch (value.kind) { + case "config": + case "generated-profile": + case "models-cache": + case "injection-journal": + return true; + case "active-catalog": + case "history-manifest": + case "history-rollout": + return typeof value.canonicalPath === "string" + && (value.kind === "active-catalog" || typeof value.stateDbId === "string"); + case "catalog-backup": + return (value.form === "hashed" || value.form === "legacy") + && typeof value.canonicalPath === "string"; + case "history-row": + case "history-manifest-entry": + return typeof value.stateDbId === "string" && typeof value.threadId === "string"; + default: + return false; + } +} + +function validateBaseline(value: unknown): boolean { + if (!isPlainRecord(value)) return false; + if (value.kind === "absent") return true; + return value.kind === "present" + && typeof value.sha256 === "string" + && typeof value.bytesBase64 === "string"; +} + +function validateEntry(value: unknown): value is CodexProvenanceEntry { + return isPlainRecord(value) + && validateArtifact(value.artifact) + && validateBaseline(value.baseline) + && (typeof value.postImage === "string" || value.postImage === null) + && typeof value.txId === "string" + && typeof value.at === "string"; +} + +function validateLedger(value: unknown): value is CodexProvenanceLedger { + return isPlainRecord(value) + && Array.isArray(value.entries) + && value.entries.every(validateEntry); +} + +function validateRecord(value: unknown): value is CodexIntegrationRecord { + return isPlainRecord(value) + && value.version === 1 + && !Object.keys(value).some(key => LEGACY_TRANSITION_KEYS.has(key)) + && (value.provenance === undefined || validateLedger(value.provenance)); +} + +function readIntegrationRecordUnlocked() { + let raw: string; + try { + raw = readFileSync(recordPath(), "utf8"); + } catch (error) { + if (isMissingPathError(error)) return { kind: "missing" as const, record: null }; + return invalid(`Codex integration record is unreadable: ${error instanceof Error ? error.message : String(error)}`); + } + + let parsed: unknown; + try { + parsed = JSON.parse(raw.replace(/^\uFEFF/, "")); + } catch { + return invalid("Codex integration record contains invalid JSON"); + } + if (!validateRecord(parsed)) { + return invalid("Codex integration record has an unsupported or malformed v1 shape"); + } + return { kind: "ready" as const, record: parsed }; +} + +export const readIntegrationRecord = readIntegrationRecordUnlocked; + +function copyUnknown( + previous: Record, + next: Record, + knownKeys: ReadonlySet, +): Record { + const merged: Record = {}; + for (const [key, value] of Object.entries(previous)) { + if (!knownKeys.has(key)) merged[key] = value; + } + return { ...merged, ...next }; +} + +function artifactKnownKeys(artifact: CodexArtifactId): ReadonlySet { + switch (artifact.kind) { + case "config": + case "generated-profile": + case "models-cache": + case "injection-journal": + return new Set(["kind"]); + case "active-catalog": + return new Set(["kind", "canonicalPath"]); + case "catalog-backup": + return new Set(["kind", "form", "canonicalPath"]); + case "history-row": + case "history-manifest-entry": + return new Set(["kind", "stateDbId", "threadId"]); + case "history-manifest": + case "history-rollout": + return new Set(["kind", "stateDbId", "canonicalPath"]); + } +} + +function mergeArtifact(previous: CodexArtifactId, next: CodexArtifactId): CodexArtifactId { + return copyUnknown( + previous as CodexArtifactId & Record, + next as CodexArtifactId & Record, + artifactKnownKeys(previous), + ) as CodexArtifactId; +} + +function mergeBaseline( + previous: CodexProvenanceEntry["baseline"], + next: CodexProvenanceEntry["baseline"], +): CodexProvenanceEntry["baseline"] { + const known = previous.kind === "present" + ? new Set(["kind", "sha256", "bytesBase64"]) + : new Set(["kind"]); + return copyUnknown(previous, next, known) as CodexProvenanceEntry["baseline"]; +} + +function knownArtifactIdentity(artifact: CodexArtifactId): string { + switch (artifact.kind) { + case "config": + case "generated-profile": + case "models-cache": + case "injection-journal": + return artifact.kind; + case "active-catalog": + return `${artifact.kind}\0${artifact.canonicalPath}`; + case "catalog-backup": + return `${artifact.kind}\0${artifact.form}\0${artifact.canonicalPath}`; + case "history-row": + case "history-manifest-entry": + return `${artifact.kind}\0${artifact.stateDbId}\0${artifact.threadId}`; + case "history-manifest": + case "history-rollout": + return `${artifact.kind}\0${artifact.stateDbId}\0${artifact.canonicalPath}`; + } +} + +function mergeEntry(previous: CodexProvenanceEntry, next: CodexProvenanceEntry): CodexProvenanceEntry { + const merged = copyUnknown( + previous, + next, + new Set(["artifact", "baseline", "postImage", "txId", "at"]), + ); + return { + ...merged, + artifact: mergeArtifact(previous.artifact, next.artifact), + baseline: mergeBaseline(previous.baseline, next.baseline), + } as CodexProvenanceEntry; +} + +function mergeLedger(previous: CodexProvenanceLedger, next: CodexProvenanceLedger): CodexProvenanceLedger { + const unused = new Set(previous.entries.map((_, index) => index)); + const entries = next.entries.map((entry, nextIndex) => { + let previousIndex = previous.entries.findIndex((candidate, index) => + unused.has(index) + && candidate.txId === entry.txId + && candidate.at === entry.at + && knownArtifactIdentity(candidate.artifact) === knownArtifactIdentity(entry.artifact)); + if (previousIndex < 0 && unused.has(nextIndex)) previousIndex = nextIndex; + if (previousIndex < 0) return entry; + unused.delete(previousIndex); + return mergeEntry(previous.entries[previousIndex]!, entry); + }); + return { + ...copyUnknown(previous, next, new Set(["entries"])), + entries, + } as CodexProvenanceLedger; +} + +function preserveExtensions( + previous: CodexIntegrationRecord, + next: CodexIntegrationRecord, +): CodexIntegrationRecord { + const merged = copyUnknown(previous, next, new Set(["version", "provenance"])); + if (previous.provenance && next.provenance) { + merged.provenance = mergeLedger(previous.provenance, next.provenance); + } + return merged as CodexIntegrationRecord; +} + +/** + * Keep read, extension merge, and atomic replacement inside the config mutation + * coordinator. WP12 must call this function instead of recreating the incident's + * stale read/merge/write sequence at its own call site. + */ +export const updateIntegrationRecord = ( + mutate: (record: CodexIntegrationRecord) => CodexIntegrationRecord, +) => { + try { + return withConfigMutationLockSync(() => { + const read = readIntegrationRecordUnlocked(); + if (read.kind === "invalid") return read; + const previous: CodexIntegrationRecord = read.kind === "ready" ? read.record : { version: 1 }; + const proposed = mutate(previous); + if (!validateRecord(proposed)) { + return { kind: "invalid", message: "Codex integration record update produced a malformed v1 shape" }; + } + const record = preserveExtensions(previous, proposed); + if (!validateRecord(record)) { + return { kind: "invalid", message: "Codex integration record extension merge produced a malformed v1 shape" }; + } + const path = recordPath(); + mkdirSync(dirname(path), { recursive: true, mode: 0o700 }); + atomicWriteFile(path, `${JSON.stringify(record, null, 2)}\n`); + return { kind: "updated", record }; + }); + } catch (error) { + return { + kind: "invalid", + message: `Codex integration record update failed: ${error instanceof Error ? error.message : String(error)}`, + }; + } +}; diff --git a/src/codex/transition-state.ts b/src/codex/transition-state.ts new file mode 100644 index 000000000..ffad5957e --- /dev/null +++ b/src/codex/transition-state.ts @@ -0,0 +1,576 @@ +/** + * CODEX_HOME-keyed transition state and coordinator transaction ownership. + * + * The original JSON read/compare/replace was called a CAS while native and + * history writers held different locks. It was not one: an old Worker could + * replace N+1 with stale N. This module owns the SQLite row, the conditional + * UPDATE, and the opaque one-shot capability backed by an already-open + * `BEGIN IMMEDIATE` transaction. + * + * Design record: devlog/_plan/260804_codex_write_substrate/005_contract.md §1. + */ +import { randomUUID } from "node:crypto"; +import { chmodSync, lstatSync, realpathSync } from "node:fs"; + +import { Database } from "bun:sqlite"; + +import type { CodexHistoryState, CommitExpectation } from "./convergence-types"; +import { resolveCodexHomeDir } from "./home"; +import { + CodexUserIdentityRefusal, + resolveCodexCoordinatorDatabasePath, + resolveEffectiveUserIdentity, +} from "./user-identity"; + +const COORDINATOR_SCHEMA_VERSION = 1; +const DURABLE_HISTORY_STATUSES = new Set(["converged", "pending", "running", "blocked", "unknown"]); +const DURABLE_HISTORY_REASONS = new Set([ + "db-busy", + "permission", + "unreadable", + "schema", + "timeout", + "shutdown-cancelled", + "worker-died", + "overtaken", + "record-write-failed", +]); + +const CREATE_TRANSITION_TABLE = ` + CREATE TABLE IF NOT EXISTS codex_transition_state ( + singleton INTEGER PRIMARY KEY CHECK (singleton = 1), + native_generation INTEGER NOT NULL CHECK (native_generation >= 0), + current_tx_id TEXT, + history_status TEXT NOT NULL, + history_reason TEXT, + history_attempts INTEGER NOT NULL CHECK (history_attempts >= 0), + history_next_retry_at TEXT, + history_tx_id TEXT, + history_direction TEXT CHECK (history_direction IN ('apply', 'remove')), + history_authority_snapshot_id TEXT, + history_pending_rows INTEGER, + history_backup_entries INTEGER, + updated_at TEXT NOT NULL, + CHECK (history_status IN ('converged', 'pending', 'running', 'blocked', 'unknown')), + CHECK (history_reason IS NULL OR history_reason IN + ('db-busy', 'permission', 'unreadable', 'schema', 'timeout', + 'shutdown-cancelled', 'worker-died', 'overtaken', 'record-write-failed')), + CHECK (history_pending_rows IS NULL OR history_pending_rows >= 0), + CHECK (history_backup_entries IS NULL OR history_backup_entries >= 0), + CHECK ((native_generation = 0 AND current_tx_id IS NULL) + OR (native_generation > 0 AND length(trim(current_tx_id)) > 0)), + CHECK ((native_generation = 0 + AND history_tx_id IS NULL + AND history_direction IS NULL + AND history_authority_snapshot_id IS NULL) + OR (native_generation > 0 + AND history_tx_id = current_tx_id + AND history_direction IS NOT NULL + AND length(trim(history_authority_snapshot_id)) > 0)), + CHECK (native_generation > 0 OR + (history_status = 'unknown' + AND history_reason IS NULL + AND history_attempts = 0 + AND history_next_retry_at IS NULL + AND history_pending_rows IS NULL + AND history_backup_entries IS NULL)) + )`; + +const INITIALIZE_TRANSITION_ROW = ` + INSERT OR IGNORE INTO codex_transition_state ( + singleton, native_generation, current_tx_id, + history_status, history_reason, history_attempts, + history_next_retry_at, history_tx_id, history_direction, + history_authority_snapshot_id, history_pending_rows, + history_backup_entries, updated_at + ) VALUES (1, 0, NULL, 'unknown', NULL, 0, NULL, NULL, NULL, NULL, NULL, NULL, ?)`; + +const SELECT_TRANSITION_ROW = ` + SELECT native_generation, current_tx_id, + history_status, history_reason, history_attempts, + history_next_retry_at, history_tx_id, history_direction, + history_authority_snapshot_id, history_pending_rows, + history_backup_entries + FROM codex_transition_state + WHERE singleton = 1`; + +const BEGIN_TRANSITION = ` + UPDATE codex_transition_state + SET native_generation = ?, current_tx_id = ?, + history_status = 'pending', history_reason = NULL, + history_attempts = 0, history_next_retry_at = ?, history_tx_id = ?, + history_direction = ?, history_authority_snapshot_id = ?, + history_pending_rows = NULL, history_backup_entries = NULL, + updated_at = ? + WHERE singleton = 1 + AND native_generation = ? + AND current_tx_id IS ?`; + +const UPDATE_HISTORY = ` + UPDATE codex_transition_state + SET history_status = ?, history_reason = ?, history_attempts = ?, + history_next_retry_at = ?, history_tx_id = ?, + history_pending_rows = ?, history_backup_entries = ?, updated_at = ? + WHERE singleton = 1 + AND native_generation = ? + AND current_tx_id IS ? + AND history_tx_id IS ? + AND (native_generation = 0 OR history_direction IS NOT NULL)`; + +interface TransitionRow { + native_generation: unknown; + current_tx_id: unknown; + history_status: unknown; + history_reason: unknown; + history_attempts: unknown; + history_next_retry_at: unknown; + history_tx_id: unknown; + history_direction: unknown; + history_authority_snapshot_id: unknown; + history_pending_rows: unknown; + history_backup_entries: unknown; +} + +export interface CodexTransitionVersion { + readonly nativeGeneration: number; + readonly currentTxId: string | null; +} + +export interface CodexTransitionState extends CodexTransitionVersion { + readonly history: CodexHistoryState; + readonly historySchedule: null | Readonly<{ + direction: "apply" | "remove"; + authoritySnapshotId: string; + }>; +} + +export type TransitionStateRead = + | { kind: "ready"; state: CodexTransitionState } + | { kind: "legacy-ambiguous"; message: string } + | { kind: "unavailable"; reason: "busy" | "unsafe-path" | "database" }; + +export type TransitionStateUpdate = + | { kind: "updated"; state: CodexTransitionState } + | { kind: "conflict"; current: CodexTransitionState } + | { kind: "unavailable"; reason: "busy" | "unsafe-path" | "database" }; + +export interface BeginCodexTransitionNext { + readonly txId: string; + readonly direction: "apply" | "remove"; + readonly authoritySnapshotId: string; + readonly nextRetryAt: string; +} + +export type BeginCodexTransition = ( + expected: CodexTransitionVersion, + next: BeginCodexTransitionNext, +) => TransitionStateUpdate; + +export type UpdateCodexHistoryTransition = ( + expected: CodexTransitionVersion, + history: CodexHistoryState, +) => TransitionStateUpdate; + +const codexCoordinatorTransactionBrand: unique symbol = Symbol("CodexCoordinatorTransaction"); + +export interface CodexCoordinatorTransaction { + readonly [codexCoordinatorTransactionBrand]: true; + readonly beginTransition: BeginCodexTransition; +} + +export interface CodexCoordinatorTransactionController { + readonly capability: CodexCoordinatorTransaction; + expectation(): CommitExpectation; + assertPublished(expectation: CommitExpectation): void; + assertStablePath(): void; + commit(): void; + rollback(): void; + close(): void; +} + +export class CodexCoordinatorTransactionError extends Error { + constructor(message: string, options?: ErrorOptions) { + super(message, options); + this.name = "CodexCoordinatorTransactionError"; + } +} + +class CodexCoordinatorLegacyAmbiguousError extends CodexCoordinatorTransactionError { + constructor(message: string) { + super(message); + this.name = "CodexCoordinatorLegacyAmbiguousError"; + } +} + +function errorCode(error: unknown): string { + return error && typeof error === "object" && "code" in error + ? String((error as { code?: unknown }).code) + : ""; +} + +function isBusy(error: unknown): boolean { + const code = errorCode(error); + const message = error instanceof Error ? error.message : String(error); + return code === "SQLITE_BUSY" || code === "SQLITE_LOCKED" || /database (?:is|table is) locked/i.test(message); +} + +function isNonNegativeInteger(value: unknown): value is number { + return typeof value === "number" && Number.isSafeInteger(value) && value >= 0; +} + +function nullableString(value: unknown): value is string | null { + return value === null || typeof value === "string"; +} + +function nullableCount(value: unknown): value is number | null { + return value === null || isNonNegativeInteger(value); +} + +function rowToState(row: TransitionRow | null): CodexTransitionState { + if (!row) throw new CodexCoordinatorTransactionError("The coordinator transition row is missing."); + if (!isNonNegativeInteger(row.native_generation) + || !nullableString(row.current_tx_id) + || typeof row.history_status !== "string" + || !DURABLE_HISTORY_STATUSES.has(row.history_status) + || !nullableString(row.history_reason) + || (row.history_reason !== null && !DURABLE_HISTORY_REASONS.has(row.history_reason)) + || !isNonNegativeInteger(row.history_attempts) + || !nullableString(row.history_next_retry_at) + || !nullableString(row.history_tx_id) + || !nullableString(row.history_direction) + || !nullableString(row.history_authority_snapshot_id) + || !nullableCount(row.history_pending_rows) + || !nullableCount(row.history_backup_entries)) { + throw new CodexCoordinatorTransactionError("The coordinator transition row is malformed."); + } + + const generation = row.native_generation; + if (generation === 0) { + if (row.current_tx_id !== null || row.history_tx_id !== null + || row.history_direction !== null || row.history_authority_snapshot_id !== null) { + throw new CodexCoordinatorTransactionError("The initial coordinator row contains transition metadata."); + } + } else if (!row.current_tx_id?.trim() + || row.history_tx_id !== row.current_tx_id + || (row.history_direction !== "apply" && row.history_direction !== "remove") + || !row.history_authority_snapshot_id?.trim()) { + throw new CodexCoordinatorTransactionError("The positive coordinator row lacks its complete history schedule."); + } + + const history: CodexHistoryState = { + status: row.history_status as Exclude, + attempts: row.history_attempts, + nextRetryAt: row.history_next_retry_at, + txId: row.history_tx_id, + pendingRows: row.history_pending_rows, + backupEntries: row.history_backup_entries, + ...(row.history_reason === null ? {} : { reason: row.history_reason as NonNullable }), + }; + return { + nativeGeneration: generation, + currentTxId: row.current_tx_id, + history, + historySchedule: generation === 0 ? null : { + direction: row.history_direction as "apply" | "remove", + authoritySnapshotId: row.history_authority_snapshot_id as string, + }, + }; +} + +function readState(database: Database): CodexTransitionState { + const row = database.query(SELECT_TRANSITION_ROW).get(); + return rowToState(row); +} + +function validateHistoryWrite(expected: CodexTransitionVersion, history: CodexHistoryState): void { + if (history.status === "not-evaluated" || !DURABLE_HISTORY_STATUSES.has(history.status)) { + throw new CodexCoordinatorTransactionError("Ephemeral history state cannot be persisted."); + } + if (!isNonNegativeInteger(history.attempts) + || !nullableCount(history.pendingRows) + || !nullableCount(history.backupEntries) + || (history.reason !== undefined && !DURABLE_HISTORY_REASONS.has(history.reason))) { + throw new CodexCoordinatorTransactionError("The history update is malformed."); + } + if (history.txId !== expected.currentTxId) { + throw new CodexCoordinatorTransactionError("The history update does not belong to the expected transition."); + } +} + +function initialize(database: Database, databaseWasAbsent: boolean): void { + const version = database.query<{ user_version: number }, []>("PRAGMA user_version").get()?.user_version; + if (version !== 0 && version !== COORDINATOR_SCHEMA_VERSION) { + throw new CodexCoordinatorTransactionError("The coordinator database schema version is unsupported."); + } + if (!databaseWasAbsent && version === 0) { + throw new CodexCoordinatorLegacyAmbiguousError( + "An existing unversioned coordinator database cannot be adopted automatically.", + ); + } + database.exec(CREATE_TRANSITION_TABLE); + const existing = database.query(SELECT_TRANSITION_ROW).get(); + if (!existing && !databaseWasAbsent) { + throw new CodexCoordinatorLegacyAmbiguousError( + "The existing coordinator database has no authoritative transition row.", + ); + } + if (!existing) database.query(INITIALIZE_TRANSITION_ROW).run(new Date().toISOString()); + if (version === 0) database.exec(`PRAGMA user_version = ${COORDINATOR_SCHEMA_VERSION}`); + readState(database); +} + +function createCapability( + database: Database, + onResult: (result: TransitionStateUpdate) => void, +): CodexCoordinatorTransaction { + let consumed = false; + return { + [codexCoordinatorTransactionBrand]: true, + beginTransition(expected, next) { + if (consumed) { + throw new CodexCoordinatorTransactionError("The coordinator capability has already been consumed."); + } + consumed = true; + if (!isNonNegativeInteger(expected.nativeGeneration) + || (expected.currentTxId !== null && !expected.currentTxId.trim()) + || !next.txId.trim() + || (next.direction !== "apply" && next.direction !== "remove") + || !next.authoritySnapshotId.trim()) { + throw new CodexCoordinatorTransactionError("The transition update is malformed."); + } + + const result = database.query(BEGIN_TRANSITION).run( + expected.nativeGeneration + 1, + next.txId, + next.nextRetryAt, + next.txId, + next.direction, + next.authoritySnapshotId, + new Date().toISOString(), + expected.nativeGeneration, + expected.currentTxId, + ); + const state = readState(database); + const update: TransitionStateUpdate = result.changes === 1 + ? { kind: "updated", state } + : { kind: "conflict", current: state }; + onResult(update); + return update; + }, + }; +} + +export function openCodexCoordinatorTransaction(finalDatabasePath: string): CodexCoordinatorTransactionController { + let database: Database | undefined; + let transactionOpen = false; + let closed = false; + let lastResult: TransitionStateUpdate | undefined; + let initialIdentity: string | undefined; + let databaseWasAbsent = false; + + try { + try { + const before = lstatSync(finalDatabasePath); + if (before.isSymbolicLink() || !before.isFile()) { + throw new CodexUserIdentityRefusal("The coordinator database path is not a real file."); + } + if (process.platform !== "win32") { + const uid = process.getuid?.(); + if (uid === undefined || before.uid !== uid || (before.mode & 0o777) !== 0o600) { + throw new CodexUserIdentityRefusal( + "The coordinator database has unsafe ownership or permissions.", + ); + } + } + } catch (cause) { + if (errorCode(cause) !== "ENOENT") throw cause; + databaseWasAbsent = true; + } + database = new Database(finalDatabasePath, { create: true }); + if (databaseWasAbsent) { + try { chmodSync(finalDatabasePath, 0o600); } catch { /* Windows applies ACLs in WP11. */ } + } + const opened = lstatSync(finalDatabasePath); + if (opened.isSymbolicLink() || !opened.isFile()) { + throw new CodexUserIdentityRefusal("The coordinator database path changed during open."); + } + initialIdentity = `${opened.dev}:${opened.ino}`; + database.exec("PRAGMA busy_timeout = 0; PRAGMA locking_mode = NORMAL; BEGIN IMMEDIATE"); + transactionOpen = true; + initialize(database, databaseWasAbsent); + } catch (cause) { + if (transactionOpen) { + try { database?.exec("ROLLBACK"); } catch { /* close releases the transaction */ } + } + try { database?.close(); } catch { /* acquisition already failed */ } + throw cause; + } + + const db = database; + const requireOpen = (): void => { + if (closed || !transactionOpen) throw new CodexCoordinatorTransactionError("The coordinator transaction is closed."); + }; + const assertStablePath = (): void => { + requireOpen(); + const entry = lstatSync(finalDatabasePath); + if (entry.isSymbolicLink() || !entry.isFile() + || `${entry.dev}:${entry.ino}` !== initialIdentity + || realpathSync.native(finalDatabasePath) !== finalDatabasePath) { + throw new CodexUserIdentityRefusal("The coordinator database path was substituted."); + } + }; + + const capability = createCapability(db, result => { lastResult = result; }); + return { + capability, + expectation() { + requireOpen(); + const state = readState(db); + return { + nativeBefore: state.nativeGeneration, + nativeAfter: state.nativeGeneration + 1, + txId: randomUUID(), + }; + }, + assertPublished(expectation) { + requireOpen(); + if (lastResult?.kind !== "updated") { + throw new CodexCoordinatorTransactionError("The coordinator transition was not published."); + } + const state = readState(db); + if (state.nativeGeneration !== expectation.nativeAfter || state.currentTxId !== expectation.txId) { + throw new CodexCoordinatorTransactionError("The coordinator published a different transition."); + } + }, + assertStablePath, + commit() { + requireOpen(); + assertStablePath(); + db.exec("COMMIT"); + transactionOpen = false; + }, + rollback() { + if (!closed && transactionOpen) { + try { db.exec("ROLLBACK"); } finally { transactionOpen = false; } + } + }, + close() { + if (closed) return; + if (transactionOpen) { + try { db.exec("ROLLBACK"); } catch { /* close still releases the lock */ } + transactionOpen = false; + } + db.close(); + closed = true; + }, + }; +} + +function currentCoordinatorDatabasePath(): string { + const canonicalCodexHome = realpathSync.native(resolveCodexHomeDir()); + return resolveCodexCoordinatorDatabasePath(resolveEffectiveUserIdentity(), canonicalCodexHome); +} + +function mapUnavailable( + error: unknown, +): Extract { + if (error instanceof CodexUserIdentityRefusal) return { kind: "unavailable", reason: "unsafe-path" }; + return { kind: "unavailable", reason: isBusy(error) ? "busy" : "database" }; +} + +function mapReadError(error: unknown): TransitionStateRead { + if (error instanceof CodexCoordinatorLegacyAmbiguousError) { + return { kind: "legacy-ambiguous", message: error.message }; + } + return mapUnavailable(error); +} + +export function readCodexTransitionState(): TransitionStateRead { + let transaction: CodexCoordinatorTransactionController | undefined; + try { + transaction = openCodexCoordinatorTransaction(currentCoordinatorDatabasePath()); + // Initialization and validation happen while N is held. Commit that setup + // before reopening read-only; the controller never leaks its Database. + transaction.commit(); + transaction.close(); + transaction = undefined; + return readCommittedState(); + } catch (error) { + transaction?.rollback(); + return mapReadError(error); + } finally { + transaction?.close(); + } +} + +function readCommittedState(): TransitionStateRead { + const path = currentCoordinatorDatabasePath(); + let database: Database | undefined; + try { + database = new Database(path, { readonly: true }); + database.exec("PRAGMA busy_timeout = 0"); + return { kind: "ready", state: readState(database) }; + } catch (error) { + return mapUnavailable(error); + } finally { + try { database?.close(); } catch { /* read already completed */ } + } +} + +export const beginCodexTransition: BeginCodexTransition = (expected, next) => { + let transaction: CodexCoordinatorTransactionController | undefined; + try { + transaction = openCodexCoordinatorTransaction(currentCoordinatorDatabasePath()); + const result = transaction.capability.beginTransition(expected, next); + transaction.commit(); + return result; + } catch (error) { + transaction?.rollback(); + const unavailable = mapUnavailable(error); + return { kind: "unavailable", reason: unavailable.reason }; + } finally { + transaction?.close(); + } +}; + +export const updateCodexHistoryTransition: UpdateCodexHistoryTransition = (expected, history) => { + let database: Database | undefined; + let transactionOpen = false; + try { + validateHistoryWrite(expected, history); + database = new Database(currentCoordinatorDatabasePath(), { create: false }); + database.exec("PRAGMA busy_timeout = 0; BEGIN IMMEDIATE"); + transactionOpen = true; + const current = readState(database); + if (current.nativeGeneration > 0 && current.historySchedule === null) { + throw new CodexCoordinatorTransactionError("A positive transition cannot lose its direction."); + } + const result = database.query(UPDATE_HISTORY).run( + history.status, + history.reason ?? null, + history.attempts, + history.nextRetryAt, + history.txId, + history.pendingRows, + history.backupEntries, + new Date().toISOString(), + expected.nativeGeneration, + expected.currentTxId, + expected.currentTxId, + ); + const state = readState(database); + database.exec("COMMIT"); + transactionOpen = false; + return result.changes === 1 + ? { kind: "updated", state } + : { kind: "conflict", current: state }; + } catch (error) { + if (transactionOpen) { + try { database?.exec("ROLLBACK"); } catch { /* close releases N */ } + } + const unavailable = mapUnavailable(error); + return { kind: "unavailable", reason: unavailable.reason }; + } finally { + try { database?.close(); } catch { /* operation already completed */ } + } +}; diff --git a/src/codex/user-identity.ts b/src/codex/user-identity.ts new file mode 100644 index 000000000..f0e9331e1 --- /dev/null +++ b/src/codex/user-identity.ts @@ -0,0 +1,186 @@ +/** + * Environment-independent identity and namespace resolution for Codex writes. + * + * Bun 1.3.14 made the obvious implementation unsafe: both `os.homedir()` and + * `os.userInfo().homedir` follow HOME. A service and CLI for the same account + * could therefore coordinate through different databases. The namespace is + * keyed only by the effective uid/SID and the canonical CODEX_HOME. + * + * Design record: devlog/_plan/260804_codex_write_substrate/005_contract.md §7. + */ +import { createHash } from "node:crypto"; +import { + lstatSync, + mkdirSync, + realpathSync, + statSync, +} from "node:fs"; +import { isAbsolute, join, resolve } from "node:path"; + +import type { + ResolveCodexCoordinatorDatabasePath, + ResolveEffectiveUserIdentity, + UserIdentity, +} from "./convergence-types"; + +const POSIX_PRIVATE_MODE = 0o700; +const POSIX_TMP_REQUIRED_MODE = 0o1003; +const POSIX_TMP_PATH = "/tmp"; +const SID_PATTERN = /^S-1-(?:\d+-)+\d+$/i; + +export class CodexUserIdentityRefusal extends Error { + readonly code = "CODEX_USER_IDENTITY_REFUSED"; + + constructor(message: string, options?: ErrorOptions) { + super(message, options); + this.name = "CodexUserIdentityRefusal"; + } +} + +function refuse(message: string, cause?: unknown): never { + throw new CodexUserIdentityRefusal(message, cause === undefined ? undefined : { cause }); +} + +function powershellValue(expression: string): string { + let result: ReturnType; + try { + result = Bun.spawnSync([ + "powershell.exe", + "-NoLogo", + "-NoProfile", + "-NonInteractive", + "-Command", + expression, + ], { + stdin: "ignore", + stdout: "pipe", + stderr: "pipe", + }); + } catch (cause) { + refuse("Windows effective-account lookup could not start.", cause); + } + if (result.exitCode !== 0) refuse("Windows effective-account lookup failed."); + const value = new TextDecoder().decode(result.stdout).trim(); + if (!value) refuse("Windows effective-account lookup returned an empty value."); + return value; +} + +function resolveWindowsSid(): string { + const sid = powershellValue( + "[System.Security.Principal.WindowsIdentity]::GetCurrent().User.Value", + ); + if (!SID_PATTERN.test(sid)) refuse("Windows effective-account lookup returned an invalid SID."); + return sid.toUpperCase(); +} + +export const resolveEffectiveUserIdentity: ResolveEffectiveUserIdentity = () => { + if (process.platform === "win32") { + return { platform: "win32", sid: resolveWindowsSid() }; + } + + const getuid = process.getuid; + if (typeof getuid !== "function") { + refuse("The runtime does not expose the effective POSIX uid."); + } + let uid: number; + try { + uid = getuid.call(process); + } catch (cause) { + refuse("The effective POSIX uid lookup failed.", cause); + } + if (!Number.isSafeInteger(uid) || uid < 0) { + refuse("The runtime returned an invalid effective POSIX uid."); + } + return { platform: "posix", uid }; +}; + +function assertPrivatePosixDirectory(path: string, uid: number): void { + let entry; + try { + entry = lstatSync(path); + } catch (cause) { + refuse("The Codex coordinator namespace cannot be inspected.", cause); + } + if (entry.isSymbolicLink() || !entry.isDirectory()) { + refuse("The Codex coordinator namespace is not a real directory."); + } + if (entry.uid !== uid || (entry.mode & 0o777) !== POSIX_PRIVATE_MODE) { + refuse("The Codex coordinator namespace has unsafe ownership or permissions."); + } +} + +function ensurePrivatePosixDirectory(path: string, uid: number): void { + try { + mkdirSync(path, { mode: POSIX_PRIVATE_MODE }); + } catch (cause) { + const code = cause && typeof cause === "object" && "code" in cause + ? String((cause as { code?: unknown }).code) + : ""; + if (code !== "EEXIST") refuse("The Codex coordinator namespace cannot be created.", cause); + } + assertPrivatePosixDirectory(path, uid); +} + +function resolvePosixRuntimeRoot(uid: number): string { + let realTmp: string; + try { + realTmp = realpathSync.native(POSIX_TMP_PATH); + const entry = statSync(realTmp); + if (!entry.isDirectory() || entry.uid !== 0) { + refuse("The system temporary directory has unsafe ownership."); + } + if ((entry.mode & POSIX_TMP_REQUIRED_MODE) !== POSIX_TMP_REQUIRED_MODE) { + refuse("The system temporary directory lacks sticky world write/search permissions."); + } + } catch (cause) { + if (cause instanceof CodexUserIdentityRefusal) throw cause; + refuse("The system temporary directory cannot be trusted.", cause); + } + + const root = join(realTmp, `opencodex-runtime-v1-${uid}`); + ensurePrivatePosixDirectory(root, uid); + return root; +} + +function resolveWindowsRuntimeRoot(identity: Extract): string { + if (!SID_PATTERN.test(identity.sid)) refuse("The coordinator identity contains an invalid SID."); + const localAppData = powershellValue( + "[Environment]::GetFolderPath([Environment+SpecialFolder]::LocalApplicationData)", + ); + if (!isAbsolute(localAppData)) refuse("Windows LocalAppData resolution returned a relative path."); + + // The SID and known-folder values come from the effective token/.NET OS APIs, + // never USERPROFILE or LOCALAPPDATA. WP11 adds descriptor/reparse/ACL checks at + // the stable-database open boundary where those checks can cover SQLite too. + const root = resolve(localAppData, "OpenCodex", "Runtime", "v1", identity.sid.toUpperCase()); + try { + mkdirSync(root, { recursive: true }); + } catch (cause) { + refuse("The Windows coordinator namespace cannot be created.", cause); + } + return root; +} + +export const resolveCodexCoordinatorDatabasePath: ResolveCodexCoordinatorDatabasePath = ( + identity, + canonicalCodexHome, +) => { + if (!isAbsolute(canonicalCodexHome)) { + refuse("The canonical CODEX_HOME must be an absolute path."); + } + const root = identity.platform === "posix" + ? resolvePosixRuntimeRoot(identity.uid) + : resolveWindowsRuntimeRoot(identity); + const locks = join(root, "native-write-locks"); + if (identity.platform === "posix") ensurePrivatePosixDirectory(locks, identity.uid); + else { + try { + mkdirSync(locks, { recursive: true }); + } catch (cause) { + refuse("The Windows coordinator lock directory cannot be created.", cause); + } + } + + const homeDigest = createHash("sha256").update(canonicalCodexHome).digest("hex"); + return join(locks, `${homeDigest}.sqlite`); +}; diff --git a/src/server/management/sync-response.ts b/src/server/management/sync-response.ts new file mode 100644 index 000000000..c1b52b3d4 --- /dev/null +++ b/src/server/management/sync-response.ts @@ -0,0 +1,69 @@ +/** + * The single HTTP projection of ConvergeOutcome. + * + * Three phase documents once mapped /api/sync independently and one silently + * dropped Retry-After. Keeping the exhaustive switch here makes a new domain + * outcome a compile error until its management contract is chosen explicitly. + */ +import type { ConvergeOutcome } from "../../codex/convergence-types"; +import { jsonResponse } from "../auth-cors"; + +export function toSyncResponse(outcome: ConvergeOutcome): Response { + switch (outcome.kind) { + case "catalog-only": + return jsonResponse({ + ok: true, + changed: outcome.changed, + observed: outcome.observed, + catalogRefresh: outcome.catalogRefresh, + history: outcome.history, + }); + case "converged": + return jsonResponse({ + ok: true, + changed: outcome.changed, + observed: outcome.observed, + catalogRefresh: outcome.catalogRefresh, + history: outcome.history, + }); + case "skipped": + return jsonResponse({ + ok: true, + changed: false, + observed: outcome.observed, + catalogRefresh: outcome.catalogRefresh, + history: outcome.history, + }); + case "refused": + return jsonResponse({ + ok: false, + authority: outcome.authority, + message: outcome.message, + observed: outcome.observed, + }, 409); + case "busy": { + const response = jsonResponse({ + ok: false, + surface: outcome.surface, + retryAfterMs: outcome.retryAfterMs, + }, 503); + response.headers.set("Retry-After", String(Math.ceil(outcome.retryAfterMs / 1_000))); + return response; + } + case "deferred": + return jsonResponse({ + ok: true, + changed: outcome.changed, + unresolved: outcome.unresolved, + observed: outcome.observed, + catalogRefresh: outcome.catalogRefresh, + history: outcome.history, + }); + case "failed": + return jsonResponse({ error: outcome.message, surface: outcome.surface }, 500); + default: { + const exhaustive: never = outcome; + return exhaustive; + } + } +} diff --git a/tests/codex-integration-record.test.ts b/tests/codex-integration-record.test.ts new file mode 100644 index 000000000..4d90a0131 --- /dev/null +++ b/tests/codex-integration-record.test.ts @@ -0,0 +1,142 @@ +import { afterEach, beforeEach, describe, expect, test } from "bun:test"; +import { mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { + readIntegrationRecord, + updateIntegrationRecord, +} from "../src/codex/integration-record"; +import type { CodexIntegrationRecord } from "../src/codex/convergence-types"; + +let opencodexHome = ""; +let previousOpencodexHome: string | undefined; + +function integrationRecordPath(): string { + return join(opencodexHome, "integrations", "codex.json"); +} + +function writeRecord(value: unknown): void { + mkdirSync(join(opencodexHome, "integrations"), { recursive: true }); + writeFileSync(integrationRecordPath(), JSON.stringify(value, null, 2)); +} + +function persistedRecord(): Record { + return JSON.parse(readFileSync(integrationRecordPath(), "utf8")) as Record; +} + +beforeEach(() => { + previousOpencodexHome = process.env.OPENCODEX_HOME; + opencodexHome = mkdtempSync(join(tmpdir(), "ocx-integration-record-")); + process.env.OPENCODEX_HOME = opencodexHome; +}); + +afterEach(() => { + if (previousOpencodexHome === undefined) delete process.env.OPENCODEX_HOME; + else process.env.OPENCODEX_HOME = previousOpencodexHome; + rmSync(opencodexHome, { recursive: true, force: true }); +}); + +describe("Codex integration record", () => { + test("accepts a v1 record written before provenance existed", () => { + writeRecord({ version: 1 }); + + expect(readIntegrationRecord()).toEqual({ + kind: "ready", + record: { version: 1 }, + }); + }); + + test("preserves future keys at record, ledger, entry, artifact, and both baseline levels", () => { + writeRecord({ + version: 1, + futureRecord: { mode: "newer" }, + provenance: { + futureLedger: ["keep"], + entries: [ + { + artifact: { kind: "config", futureArtifact: { owner: "future-config" } }, + baseline: { kind: "absent", futureAbsentBaseline: 17 }, + postImage: "old-config-post-image", + txId: "tx-config", + at: "2026-08-04T00:00:00.000Z", + futureEntry: { evidence: true }, + }, + { + artifact: { kind: "generated-profile", futureArtifact: { owner: "future-profile" } }, + baseline: { + kind: "present", + sha256: "baseline-sha", + bytesBase64: "YmFzZWxpbmU=", + futurePresentBaseline: { codec: 2 }, + }, + postImage: "old-profile-post-image", + txId: "tx-profile", + at: "2026-08-04T00:00:01.000Z", + futureEntry: { evidence: false }, + }, + ], + }, + }); + + const result = updateIntegrationRecord(record => ({ + version: 1, + provenance: { + entries: record.provenance!.entries.map((entry, index) => ({ + artifact: { kind: entry.artifact.kind } as typeof entry.artifact, + baseline: entry.baseline.kind === "absent" + ? { kind: "absent" } + : { + kind: "present", + sha256: entry.baseline.sha256, + bytesBase64: entry.baseline.bytesBase64, + }, + postImage: `new-post-image-${index}`, + txId: entry.txId, + at: entry.at, + })), + }, + })); + + expect(result.kind).toBe("updated"); + const saved = persistedRecord(); + expect(saved.futureRecord).toEqual({ mode: "newer" }); + const ledger = saved.provenance as Record; + expect(ledger.futureLedger).toEqual(["keep"]); + const entries = ledger.entries as Array>; + expect(entries[0]!.futureEntry).toEqual({ evidence: true }); + expect(entries[1]!.futureEntry).toEqual({ evidence: false }); + expect((entries[0]!.artifact as Record).futureArtifact) + .toEqual({ owner: "future-config" }); + expect((entries[1]!.artifact as Record).futureArtifact) + .toEqual({ owner: "future-profile" }); + expect((entries[0]!.baseline as Record).futureAbsentBaseline).toBe(17); + expect((entries[1]!.baseline as Record).futurePresentBaseline) + .toEqual({ codec: 2 }); + expect(entries.map(entry => entry.postImage)).toEqual(["new-post-image-0", "new-post-image-1"]); + }); + + test("fails closed on unparseable bytes without invoking the mutator or resetting the file", () => { + mkdirSync(join(opencodexHome, "integrations"), { recursive: true }); + writeFileSync(integrationRecordPath(), "{ definitely-not-json", "utf8"); + let invoked = false; + + const result = updateIntegrationRecord((record): CodexIntegrationRecord => { + invoked = true; + return record; + }); + + expect(result).toEqual({ + kind: "invalid", + message: "Codex integration record contains invalid JSON", + }); + expect(invoked).toBe(false); + expect(readFileSync(integrationRecordPath(), "utf8")).toBe("{ definitely-not-json"); + }); + + test("creates the minimal v1 record when the file is missing", () => { + const result = updateIntegrationRecord(record => record); + + expect(result).toEqual({ kind: "updated", record: { version: 1 } }); + expect(persistedRecord()).toEqual({ version: 1 }); + }); +}); diff --git a/tests/codex-sync-response.test.ts b/tests/codex-sync-response.test.ts new file mode 100644 index 000000000..e7b4c2968 --- /dev/null +++ b/tests/codex-sync-response.test.ts @@ -0,0 +1,177 @@ +import { describe, expect, test } from "bun:test"; +import type { + CatalogDisposition, + CodexHistoryState, + CodexObservedState, + ConvergeOutcome, +} from "../src/codex/convergence-types"; +import { toSyncResponse } from "../src/server/management/sync-response"; + +const history: CodexHistoryState = { + status: "converged", + attempts: 1, + nextRetryAt: null, + txId: "tx-1", + pendingRows: 0, + backupEntries: 0, +}; + +const catalogRefresh: CatalogDisposition = { + status: "committed", + changed: true, + degraded: false, + notices: [], +}; + +const observed: CodexObservedState = { + aggregate: "applied", + isApplied: true, + desired: "on", + converged: true, + authority: { service: "owned", externalProvider: null }, + surfaces: { + config: "applied", + profile: "applied", + catalog: "applied", + cache: "applied", + journal: "absent", + history: { + state: history, + database: "applied", + manifest: "applied", + rollouts: "applied", + }, + provenance: { + state: "verified", + nativeGeneration: 4, + currentTxId: "tx-1", + }, + }, +}; + +async function projection(outcome: ConvergeOutcome): Promise<{ + status: number; + body: unknown; + retryAfter: string | null; +}> { + const response = toSyncResponse(outcome); + return { + status: response.status, + body: await response.json(), + retryAfter: response.headers.get("Retry-After"), + }; +} + +describe("toSyncResponse", () => { + test("maps catalog-only", async () => { + expect(await projection({ + kind: "catalog-only", + changed: true, + observed, + catalogRefresh, + history, + })).toEqual({ + status: 200, + body: { ok: true, changed: true, observed, catalogRefresh, history }, + retryAfter: null, + }); + }); + + test("maps converged, including desired OFF removal, without a separate desired-off row", async () => { + expect(await projection({ + kind: "converged", + direction: "removed", + changed: true, + observed, + nativeGeneration: 4, + currentTxId: "tx-1", + catalogRefresh, + history, + })).toEqual({ + status: 200, + body: { ok: true, changed: true, observed, catalogRefresh, history }, + retryAfter: null, + }); + }); + + test("maps skipped", async () => { + expect(await projection({ + kind: "skipped", + reason: "already-converged", + observed, + catalogRefresh, + history, + })).toEqual({ + status: 200, + body: { ok: true, changed: false, observed, catalogRefresh, history }, + retryAfter: null, + }); + }); + + test("maps refused", async () => { + expect(await projection({ + kind: "refused", + authority: "provenance", + message: "provenance does not authorize restore", + observed, + })).toEqual({ + status: 409, + body: { + ok: false, + authority: "provenance", + message: "provenance does not authorize restore", + observed, + }, + retryAfter: null, + }); + }); + + test("maps busy to 503 with Retry-After seconds", async () => { + expect(await projection({ + kind: "busy", + surface: "lock", + retryAfterMs: 1_250, + })).toEqual({ + status: 503, + body: { ok: false, surface: "lock", retryAfterMs: 1_250 }, + retryAfter: "2", + }); + }); + + test("maps deferred", async () => { + expect(await projection({ + kind: "deferred", + direction: "applied", + changed: true, + unresolved: ["history", "catalog"], + nativeGeneration: 4, + currentTxId: "tx-1", + observed, + catalogRefresh, + history, + })).toEqual({ + status: 200, + body: { + ok: true, + changed: true, + unresolved: ["history", "catalog"], + observed, + catalogRefresh, + history, + }, + retryAfter: null, + }); + }); + + test("maps failed", async () => { + expect(await projection({ + kind: "failed", + surface: "provenance", + message: "record write failed", + })).toEqual({ + status: 500, + body: { error: "record write failed", surface: "provenance" }, + retryAfter: null, + }); + }); +}); diff --git a/tests/codex-transition-state.test.ts b/tests/codex-transition-state.test.ts new file mode 100644 index 000000000..4266911b8 --- /dev/null +++ b/tests/codex-transition-state.test.ts @@ -0,0 +1,147 @@ +import { afterEach, beforeEach, expect, test } from "bun:test"; +import { chmodSync, mkdtempSync, realpathSync, rmSync } from "node:fs"; +import { join } from "node:path"; +import { tmpdir } from "node:os"; + +import { Database } from "bun:sqlite"; + +import { + beginCodexTransition, + openCodexCoordinatorTransaction, + readCodexTransitionState, +} from "../src/codex/transition-state"; +import { + resolveCodexCoordinatorDatabasePath, + resolveEffectiveUserIdentity, +} from "../src/codex/user-identity"; + +let codexHome = ""; +let coordinatorPath = ""; +let previousCodexHome: string | undefined; + +beforeEach(() => { + previousCodexHome = process.env.CODEX_HOME; + codexHome = mkdtempSync(join(tmpdir(), "ocx-transition-state-codex-home-")); + process.env.CODEX_HOME = codexHome; + coordinatorPath = resolveCodexCoordinatorDatabasePath( + resolveEffectiveUserIdentity(), + realpathSync.native(codexHome), + ); +}); + +afterEach(() => { + if (previousCodexHome === undefined) delete process.env.CODEX_HOME; + else process.env.CODEX_HOME = previousCodexHome; + for (const suffix of ["", "-journal", "-wal", "-shm"]) { + rmSync(`${coordinatorPath}${suffix}`, { force: true }); + } + rmSync(codexHome, { recursive: true, force: true }); +}); + +function transition(txId: string) { + return { + txId, + direction: "apply" as const, + authoritySnapshotId: `authority-${txId}`, + nextRetryAt: "2026-08-04T12:00:00.000Z", + }; +} + +test("a matching conditional transition update succeeds", () => { + expect(readCodexTransitionState()).toEqual({ + kind: "ready", + state: { + nativeGeneration: 0, + currentTxId: null, + history: { + status: "unknown", + attempts: 0, + nextRetryAt: null, + txId: null, + pendingRows: null, + backupEntries: null, + }, + historySchedule: null, + }, + }); + + const result = beginCodexTransition( + { nativeGeneration: 0, currentTxId: null }, + transition("tx-winner"), + ); + expect(result.kind).toBe("updated"); + if (result.kind === "updated") { + expect(result.state.nativeGeneration).toBe(1); + expect(result.state.currentTxId).toBe("tx-winner"); + expect(result.state.historySchedule?.direction).toBe("apply"); + } +}); + +test("an existing database without the singleton row is legacy-ambiguous", () => { + const database = new Database(coordinatorPath, { create: true }); + database.exec("PRAGMA user_version = 1"); + database.close(); + if (process.platform !== "win32") chmodSync(coordinatorPath, 0o600); + + expect(readCodexTransitionState()).toEqual({ + kind: "legacy-ambiguous", + message: "The existing coordinator database has no authoritative transition row.", + }); +}); + +test("a zero-row conditional update reports conflict and preserves the winner", () => { + const winner = beginCodexTransition( + { nativeGeneration: 0, currentTxId: null }, + transition("tx-newer"), + ); + expect(winner.kind).toBe("updated"); + + const stale = beginCodexTransition( + { nativeGeneration: 0, currentTxId: null }, + transition("tx-stale"), + ); + expect(stale.kind).toBe("conflict"); + if (stale.kind === "conflict") { + expect(stale.current.currentTxId).toBe("tx-newer"); + expect(stale.current.historySchedule?.authoritySnapshotId).toBe("authority-tx-newer"); + } + expect(readCodexTransitionState()).toMatchObject({ + kind: "ready", + state: { nativeGeneration: 1, currentTxId: "tx-newer" }, + }); +}); + +test("a positive generation cannot carry a null direction", () => { + expect(beginCodexTransition( + { nativeGeneration: 0, currentTxId: null }, + transition("tx-direction"), + ).kind).toBe("updated"); + + const database = new Database(coordinatorPath); + try { + expect(() => database.run( + "UPDATE codex_transition_state SET history_direction = NULL WHERE singleton = 1", + )).toThrow(); + expect(database.query<{ history_direction: string }, []>( + "SELECT history_direction FROM codex_transition_state WHERE singleton = 1", + ).get()?.history_direction).toBe("apply"); + } finally { + database.close(); + } +}); + +test("the opaque coordinator capability is one-shot", () => { + const controller = openCodexCoordinatorTransaction(coordinatorPath); + try { + const expectation = controller.expectation(); + const expected = { nativeGeneration: expectation.nativeBefore, currentTxId: null }; + const next = transition(expectation.txId); + expect(controller.capability.beginTransition(expected, next).kind).toBe("updated"); + expect(() => controller.capability.beginTransition(expected, next)) + .toThrow("already been consumed"); + controller.assertPublished(expectation); + controller.commit(); + } finally { + controller.close(); + } +}); diff --git a/tests/codex-user-identity.test.ts b/tests/codex-user-identity.test.ts new file mode 100644 index 000000000..f40ab8d37 --- /dev/null +++ b/tests/codex-user-identity.test.ts @@ -0,0 +1,54 @@ +import { afterEach, beforeEach, expect, test } from "bun:test"; +import { mkdtempSync, realpathSync, rmSync } from "node:fs"; +import { join, parse } from "node:path"; +import { tmpdir } from "node:os"; + +import { + resolveCodexCoordinatorDatabasePath, + resolveEffectiveUserIdentity, +} from "../src/codex/user-identity"; + +let codexHome = ""; +let previousHome: string | undefined; + +beforeEach(() => { + previousHome = process.env.HOME; + codexHome = mkdtempSync(join(tmpdir(), "ocx-user-identity-codex-home-")); +}); + +afterEach(() => { + if (previousHome === undefined) delete process.env.HOME; + else process.env.HOME = previousHome; + rmSync(codexHome, { recursive: true, force: true }); +}); + +test("the effective identity is uid/SID and does not follow HOME", () => { + const before = resolveEffectiveUserIdentity(); + process.env.HOME = join(tmpdir(), "fake-home-that-must-not-key-coordination"); + const after = resolveEffectiveUserIdentity(); + + expect(after).toEqual(before); + if (process.platform === "win32") { + expect(after.platform).toBe("win32"); + expect("sid" in after && after.sid).toMatch(/^S-1-/); + } else { + expect(after).toEqual({ platform: "posix", uid: process.getuid!() }); + } + expect(JSON.stringify(after)).not.toContain(process.env.HOME); +}); + +test("the coordinator resolver returns the final database path", () => { + const canonicalHome = realpathSync.native(codexHome); + const finalPath = resolveCodexCoordinatorDatabasePath( + resolveEffectiveUserIdentity(), + canonicalHome, + ); + + expect(parse(finalPath).ext).toBe(".sqlite"); + expect(parse(finalPath).base).toMatch(/^[a-f0-9]{64}\.sqlite$/); + expect(parse(parse(finalPath).dir).base).toBe("native-write-locks"); + expect(finalPath).toBe(resolveCodexCoordinatorDatabasePath( + resolveEffectiveUserIdentity(), + canonicalHome, + )); +}); From 6b4a798fefa9187166859acde03c1c4c7b4e447a Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Tue, 4 Aug 2026 13:33:42 +0900 Subject: [PATCH 042/163] fix(codex): every history transition update was failing before its CAS MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The C-phase reviewer found this by running the code instead of the suite: new Database(path, { create: false }) is SQLITE_MISUSE on Bun 1.3.14 because the flags name no read mode. updateCodexHistoryTransition therefore returned unavailable/database every time, so no terminal history state could ever be recorded and a stale worker's overtaking was never reported as conflict. Eighteen tests were green while that was true, because not one of them called the function. That is the same failure class this project keeps producing — 91 tests passed here beside a broken gjc config, and 8000 pass today beside the defects this unit fixes. Three tests added, and proven by reverting the one-line fix: the history update and the stale-overtaking case both go red without it. The third closes a gap the reviewer named separately — the happy-path begin test still passed with the conditional WHERE removed, so a same-txId different-generation case now fails the moment the guard stops matching on both columns. --- src/codex/transition-state.ts | 7 +- tests/codex-transition-state.test.ts | 97 ++++++++++++++++++++++++++++ 2 files changed, 103 insertions(+), 1 deletion(-) diff --git a/src/codex/transition-state.ts b/src/codex/transition-state.ts index ffad5957e..2494edfea 100644 --- a/src/codex/transition-state.ts +++ b/src/codex/transition-state.ts @@ -538,7 +538,12 @@ export const updateCodexHistoryTransition: UpdateCodexHistoryTransition = (expec let transactionOpen = false; try { validateHistoryWrite(expected, history); - database = new Database(currentCoordinatorDatabasePath(), { create: false }); + // `{ create: false }` ALONE is SQLITE_MISUSE on Bun 1.3.14: the flags must + // name a read mode. Without `readwrite` every history update failed before + // reaching its conditional UPDATE and returned `unavailable/database`, so no + // terminal history state could ever be recorded — and the four tests here + // still passed, because none of them called this function. + database = new Database(currentCoordinatorDatabasePath(), { readwrite: true, create: false }); database.exec("PRAGMA busy_timeout = 0; BEGIN IMMEDIATE"); transactionOpen = true; const current = readState(database); diff --git a/tests/codex-transition-state.test.ts b/tests/codex-transition-state.test.ts index 4266911b8..b82e15267 100644 --- a/tests/codex-transition-state.test.ts +++ b/tests/codex-transition-state.test.ts @@ -9,6 +9,7 @@ import { beginCodexTransition, openCodexCoordinatorTransaction, readCodexTransitionState, + updateCodexHistoryTransition, } from "../src/codex/transition-state"; import { resolveCodexCoordinatorDatabasePath, @@ -145,3 +146,99 @@ test("the opaque coordinator capability is one-shot", () => { controller.close(); } }); + +/** + * The C-phase reviewer found this by running the code rather than the suite: + * `new Database(path, { create: false })` is SQLITE_MISUSE on Bun 1.3.14 + * because the flags name no read mode. Every history update therefore failed + * before reaching its conditional UPDATE and returned `unavailable/database`, + * so no terminal history state could ever be recorded. + * + * Eighteen tests were green while that was true, because none of them called + * `updateCodexHistoryTransition` at all. That is the gap this file closes. + */ +test("a history transition update reaches its conditional UPDATE and records terminal state", () => { + const started = beginCodexTransition( + { nativeGeneration: 0, currentTxId: null }, + transition("tx-history"), + ); + expect(started.kind).toBe("updated"); + + const updated = updateCodexHistoryTransition( + { nativeGeneration: 1, currentTxId: "tx-history" }, + { + status: "converged", + attempts: 1, + nextRetryAt: null, + txId: "tx-history", + pendingRows: 0, + backupEntries: 0, + }, + ); + + // Before the fix this was `unavailable` with reason `database`, every time. + expect(updated.kind).toBe("updated"); + + const after = readCodexTransitionState(); + expect(after.kind).toBe("ready"); + if (after.kind === "ready") { + expect(after.state.history.status).toBe("converged"); + expect(after.state.history.txId).toBe("tx-history"); + expect(after.state.history.pendingRows).toBe(0); + } +}); + +/** + * The overtaking case the substrate exists for: a stale Worker finishing after + * a newer transition committed must NOT publish its terminal state over the + * winner's schedule. It must report conflict. + */ +test("a stale history update conflicts and leaves the newer transition's schedule intact", () => { + beginCodexTransition({ nativeGeneration: 0, currentTxId: null }, transition("tx-a")); + const newer = beginCodexTransition( + { nativeGeneration: 1, currentTxId: "tx-a" }, + { ...transition("tx-b"), direction: "remove" as const }, + ); + expect(newer.kind).toBe("updated"); + + const stale = updateCodexHistoryTransition( + { nativeGeneration: 1, currentTxId: "tx-a" }, + { + status: "converged", + attempts: 1, + nextRetryAt: null, + txId: "tx-a", + pendingRows: 0, + backupEntries: 0, + }, + ); + expect(stale.kind).toBe("conflict"); + + const after = readCodexTransitionState(); + expect(after.kind).toBe("ready"); + if (after.kind === "ready") { + expect(after.state.currentTxId).toBe("tx-b"); + expect(after.state.historySchedule?.direction).toBe("remove"); + // The stale worker must not have published its own terminal status. + expect(after.state.history.status).not.toBe("converged"); + } +}); + +/** + * Reviewer finding: the happy-path update test still passed with the + * conditional WHERE removed, so it did not prove the update is conditional. + * This one fails the moment the guard stops matching on BOTH columns. + */ +test("a begin whose txId matches but whose generation does not is rejected", () => { + beginCodexTransition({ nativeGeneration: 0, currentTxId: null }, transition("tx-one")); + + const wrongGeneration = beginCodexTransition( + { nativeGeneration: 7, currentTxId: "tx-one" }, + transition("tx-two"), + ); + expect(wrongGeneration.kind).toBe("conflict"); + + const after = readCodexTransitionState(); + expect(after.kind).toBe("ready"); + if (after.kind === "ready") expect(after.state.currentTxId).toBe("tx-one"); +}); From 436e152b053d3cb2500730922e13d4158042a9a8 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Tue, 4 Aug 2026 15:38:19 +0900 Subject: [PATCH 043/163] fix(codex): close the four C-phase findings against the landed substrate MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The config-generation counter the contract promised did not exist at all, so WP9 had no way to detect config ABA. src/codex/generation.ts now owns a singleton row inside the existing config-mutation database, and saveConfig, mutatePersistedConfig and saveConfigPreservingClaudeCode each bump it exactly once, inside their own transaction, and only when the serialized bytes really changed. A non-cooperating hand edit deliberately does not bump — post-commit observation owns that case, and the docstring says so rather than pretending otherwise. Coordinator initialization was treating an absence as a guarantee: it checked only whether the SQLite file existed, so a home carrying legacy transition fields in integrations/codex.json or live native routing residue would be silently overwritten with a clean {0,null} authority. It now refuses both and reports legacy-ambiguous. That is the same pattern this unit keeps catching in me, one layer down. convergence-types.ts becomes the sole owner it claimed to be: every shared transition and coordinator type moves there, while the private brand stays in transition-state.ts because the other arrangement produced TS2741. Artifact and both baseline variants gained extension passthrough, so the test could drop the cast that was hiding the type defect, and coverage went from two artifact variants to all ten. Two tests that passed either way are now real: one proves a second connection cannot be opened while the capability is held, the other actually performs the first provenance write instead of asserting an empty record. Each was proven by reverting the behavior and watching it go red. 8030 tests pass, typecheck clean, privacy scan and gui lint green. --- src/codex/convergence-types.ts | 88 +++++++++++++--- src/codex/generation.ts | 138 +++++++++++++++++++++++++ src/codex/transition-state.ts | 112 ++++++++++---------- src/config.ts | 61 +++++++++-- tests/codex-config-generation.test.ts | 115 +++++++++++++++++++++ tests/codex-integration-record.test.ts | 122 +++++++++++++++------- tests/codex-transition-state.test.ts | 69 ++++++++++++- 7 files changed, 589 insertions(+), 116 deletions(-) create mode 100644 src/codex/generation.ts create mode 100644 tests/codex-config-generation.test.ts diff --git a/src/codex/convergence-types.ts b/src/codex/convergence-types.ts index 9bf0fca62..e11e2c552 100644 --- a/src/codex/convergence-types.ts +++ b/src/codex/convergence-types.ts @@ -68,26 +68,29 @@ export interface CodexHistoryState { * rollout whose semantic pre-image is retained. */ export type CodexArtifactId = - | { readonly kind: "config" } - | { readonly kind: "generated-profile" } - | { readonly kind: "active-catalog"; readonly canonicalPath: string } + | { readonly kind: "config"; readonly [extra: string]: unknown } + | { readonly kind: "generated-profile"; readonly [extra: string]: unknown } + | { readonly kind: "active-catalog"; readonly canonicalPath: string; + readonly [extra: string]: unknown } | { readonly kind: "catalog-backup"; readonly form: "hashed" | "legacy"; - readonly canonicalPath: string } - | { readonly kind: "models-cache" } - | { readonly kind: "injection-journal" } - | { readonly kind: "history-row"; readonly stateDbId: string; readonly threadId: string } + readonly canonicalPath: string; readonly [extra: string]: unknown } + | { readonly kind: "models-cache"; readonly [extra: string]: unknown } + | { readonly kind: "injection-journal"; readonly [extra: string]: unknown } + | { readonly kind: "history-row"; readonly stateDbId: string; readonly threadId: string; + readonly [extra: string]: unknown } | { readonly kind: "history-manifest"; readonly stateDbId: string; - readonly canonicalPath: string } + readonly canonicalPath: string; readonly [extra: string]: unknown } | { readonly kind: "history-manifest-entry"; readonly stateDbId: string; - readonly threadId: string } + readonly threadId: string; readonly [extra: string]: unknown } | { readonly kind: "history-rollout"; readonly stateDbId: string; - readonly canonicalPath: string }; + readonly canonicalPath: string; readonly [extra: string]: unknown }; export interface CodexProvenanceEntry { artifact: CodexArtifactId; baseline: - | { kind: "absent" } - | { kind: "present"; sha256: string; bytesBase64: string }; + | { kind: "absent"; readonly [extra: string]: unknown } + | { kind: "present"; sha256: string; bytesBase64: string; + readonly [extra: string]: unknown }; /** Hash of what WE wrote. null when the write did not complete. */ postImage: string | null; txId: string; @@ -256,6 +259,67 @@ export interface CommitExpectation { readonly txId: string; } +/** The authoritative pair and history schedule stored under canonical CODEX_HOME. */ +export interface CodexTransitionVersion { + readonly nativeGeneration: number; + readonly currentTxId: string | null; +} + +export interface CodexTransitionState extends CodexTransitionVersion { + readonly history: CodexHistoryState; + readonly historySchedule: null | Readonly<{ + direction: "apply" | "remove"; + authoritySnapshotId: string; + }>; +} + +export type TransitionStateRead = + | { kind: "ready"; state: CodexTransitionState } + | { kind: "legacy-ambiguous"; message: string } + | { kind: "unavailable"; reason: "busy" | "unsafe-path" | "database" }; + +export type TransitionStateUpdate = + | { kind: "updated"; state: CodexTransitionState } + | { kind: "conflict"; current: CodexTransitionState } + | { kind: "unavailable"; reason: "busy" | "unsafe-path" | "database" }; + +export interface BeginCodexTransitionNext { + readonly txId: string; + readonly direction: "apply" | "remove"; + readonly authoritySnapshotId: string; + readonly nextRetryAt: string; +} + +export type BeginCodexTransition = ( + expected: CodexTransitionVersion, + next: BeginCodexTransitionNext, +) => TransitionStateUpdate; + +export type ReadCodexTransitionState = () => TransitionStateRead; + +export type UpdateCodexHistoryTransition = ( + expected: CodexTransitionVersion, + history: CodexHistoryState, +) => TransitionStateUpdate; + +/** + * A coordinator transaction never exposes its SQLite connection. The runtime + * owner adds a private brand so only its one-shot factory can create one. + */ +export interface CodexCoordinatorTransaction { + readonly beginTransition: BeginCodexTransition; +} + +export interface CodexCoordinatorTransactionController { + readonly capability: CodexCoordinatorTransaction; + expectation(): CommitExpectation; + assertPublished(expectation: CommitExpectation): void; + assertStablePath(): void; + commit(): void; + rollback(): void; + close(): void; +} + /** The minimal, working WP8b/WP9 snapshot; it authorizes catalog work only. */ export interface CatalogAdmissionSnapshot { config: Readonly; diff --git a/src/codex/generation.ts b/src/codex/generation.ts new file mode 100644 index 000000000..fd91ec9c7 --- /dev/null +++ b/src/codex/generation.ts @@ -0,0 +1,138 @@ +/** + * Durable generation ownership for OpenCodex config bytes. + * + * The WP8b C-phase review found that config ABA and moved configuration could + * not be detected because cooperating saves had no durable counter. This module + * owns the singleton schema and conditional increment in the existing config + * mutation database; callers that already hold its transaction reuse that + * Database handle so bytes and generation remain one cooperating commit. + * + * A hand edit from a non-cooperating writer deliberately does not increment this + * counter. The convergence contract detects that case with its post-commit file + * observation instead of pretending SQLite can coordinate an external editor. + */ +import { chmodSync } from "node:fs"; + +import { Database } from "bun:sqlite"; + +import type { + ConfigGeneration, + ConfigGenerationBump, + ConfigGenerationRead, +} from "./convergence-types"; + +const CREATE_CONFIG_GENERATION = ` + CREATE TABLE IF NOT EXISTS config_generation ( + singleton INTEGER PRIMARY KEY CHECK (singleton = 1), + value INTEGER NOT NULL CHECK (value >= 0) + )`; +const INITIALIZE_CONFIG_GENERATION = ` + INSERT OR IGNORE INTO config_generation (singleton, value) VALUES (1, 0)`; +const SELECT_CONFIG_GENERATION = ` + SELECT value FROM config_generation WHERE singleton = 1`; +const BUMP_CONFIG_GENERATION = ` + UPDATE config_generation + SET value = value + 1 + WHERE singleton = 1 AND value = ?`; + +interface ConfigGenerationRow { + value: unknown; +} + +function errorCode(error: unknown): string { + return error && typeof error === "object" && "code" in error + ? String((error as { code?: unknown }).code) + : ""; +} + +function isBusy(error: unknown): boolean { + const code = errorCode(error); + const message = error instanceof Error ? error.message : ""; + return code === "SQLITE_BUSY" + || code === "SQLITE_LOCKED" + || /database (?:is|table is) locked/i.test(message); +} + +function unavailable(error: unknown): Extract { + return { kind: "unavailable", reason: isBusy(error) ? "busy" : "database" }; +} + +export function initializeConfigGeneration(database: Database): void { + database.exec(CREATE_CONFIG_GENERATION); + database.exec(INITIALIZE_CONFIG_GENERATION); +} + +export function readConfigGenerationInTransaction(database: Database): ConfigGeneration { + const row = database.query(SELECT_CONFIG_GENERATION).get(); + if (!row || !Number.isSafeInteger(row.value) || Number(row.value) < 0) { + throw new Error("The config generation singleton is missing or invalid."); + } + return { value: Number(row.value) }; +} + +export function bumpConfigGenerationInTransaction( + database: Database, + expected: ConfigGeneration, +): ConfigGenerationBump { + const result = database.query(BUMP_CONFIG_GENERATION).run(expected.value); + if (result.changes === 1) { + return { kind: "updated", generation: { value: expected.value + 1 } }; + } + return { kind: "conflict", current: readConfigGenerationInTransaction(database) }; +} + +export function bumpCurrentConfigGeneration(database: Database): ConfigGeneration { + const current = readConfigGenerationInTransaction(database); + const result = bumpConfigGenerationInTransaction(database, current); + if (result.kind !== "updated") { + throw new Error("The config generation changed inside its owning transaction."); + } + return result.generation; +} + +function runGenerationTransaction(databasePath: string, operation: (database: Database) => T): T { + let database: Database | undefined; + let transactionOpen = false; + try { + database = new Database(databasePath, { create: true }); + try { chmodSync(databasePath, 0o600); } catch { /* platform may ignore chmod */ } + database.exec("PRAGMA busy_timeout = 0; BEGIN IMMEDIATE"); + transactionOpen = true; + initializeConfigGeneration(database); + const result = operation(database); + database.exec("COMMIT"); + transactionOpen = false; + return result; + } catch (error) { + if (transactionOpen) { + try { database?.exec("ROLLBACK"); } catch { /* close releases the transaction */ } + } + throw error; + } finally { + try { database?.close(); } catch { /* operation already completed */ } + } +} + +export function readConfigGenerationAtPath(databasePath: string): ConfigGenerationRead { + try { + return { + kind: "ready", + generation: runGenerationTransaction(databasePath, readConfigGenerationInTransaction), + }; + } catch (error) { + return unavailable(error); + } +} + +export function bumpConfigGenerationAtPath( + databasePath: string, + expected: ConfigGeneration, +): ConfigGenerationBump { + try { + return runGenerationTransaction(databasePath, database => ( + bumpConfigGenerationInTransaction(database, expected) + )); + } catch (error) { + return unavailable(error); + } +} diff --git a/src/codex/transition-state.ts b/src/codex/transition-state.ts index 2494edfea..4315b0a4b 100644 --- a/src/codex/transition-state.ts +++ b/src/codex/transition-state.ts @@ -10,12 +10,27 @@ * Design record: devlog/_plan/260804_codex_write_substrate/005_contract.md §1. */ import { randomUUID } from "node:crypto"; -import { chmodSync, lstatSync, realpathSync } from "node:fs"; +import { chmodSync, lstatSync, readFileSync, realpathSync } from "node:fs"; +import { join } from "node:path"; import { Database } from "bun:sqlite"; -import type { CodexHistoryState, CommitExpectation } from "./convergence-types"; +import type { + BeginCodexTransition, + CodexCoordinatorTransaction, + CodexCoordinatorTransactionController, + CodexHistoryState, + CodexTransitionState, + CodexTransitionVersion, + CommitExpectation, + ReadCodexTransitionState, + TransitionStateRead, + TransitionStateUpdate, + UpdateCodexHistoryTransition, +} from "./convergence-types"; import { resolveCodexHomeDir } from "./home"; +import { hasInjectedCodexRouting } from "./injected-marker"; +import { readIntegrationRecord } from "./integration-record"; import { CodexUserIdentityRefusal, resolveCodexCoordinatorDatabasePath, @@ -131,61 +146,10 @@ interface TransitionRow { history_backup_entries: unknown; } -export interface CodexTransitionVersion { - readonly nativeGeneration: number; - readonly currentTxId: string | null; -} - -export interface CodexTransitionState extends CodexTransitionVersion { - readonly history: CodexHistoryState; - readonly historySchedule: null | Readonly<{ - direction: "apply" | "remove"; - authoritySnapshotId: string; - }>; -} - -export type TransitionStateRead = - | { kind: "ready"; state: CodexTransitionState } - | { kind: "legacy-ambiguous"; message: string } - | { kind: "unavailable"; reason: "busy" | "unsafe-path" | "database" }; - -export type TransitionStateUpdate = - | { kind: "updated"; state: CodexTransitionState } - | { kind: "conflict"; current: CodexTransitionState } - | { kind: "unavailable"; reason: "busy" | "unsafe-path" | "database" }; - -export interface BeginCodexTransitionNext { - readonly txId: string; - readonly direction: "apply" | "remove"; - readonly authoritySnapshotId: string; - readonly nextRetryAt: string; -} - -export type BeginCodexTransition = ( - expected: CodexTransitionVersion, - next: BeginCodexTransitionNext, -) => TransitionStateUpdate; - -export type UpdateCodexHistoryTransition = ( - expected: CodexTransitionVersion, - history: CodexHistoryState, -) => TransitionStateUpdate; - const codexCoordinatorTransactionBrand: unique symbol = Symbol("CodexCoordinatorTransaction"); -export interface CodexCoordinatorTransaction { +interface BrandedCodexCoordinatorTransaction extends CodexCoordinatorTransaction { readonly [codexCoordinatorTransactionBrand]: true; - readonly beginTransition: BeginCodexTransition; -} - -export interface CodexCoordinatorTransactionController { - readonly capability: CodexCoordinatorTransaction; - expectation(): CommitExpectation; - assertPublished(expectation: CommitExpectation): void; - assertStablePath(): void; - commit(): void; - rollback(): void; - close(): void; } export class CodexCoordinatorTransactionError extends Error { @@ -297,6 +261,35 @@ function validateHistoryWrite(expected: CodexTransitionVersion, history: CodexHi } } +function hasNativeRoutedResidue(): boolean { + const configPath = join(resolveCodexHomeDir(), "config.toml"); + try { + return hasInjectedCodexRouting(readFileSync(configPath, "utf8")); + } catch (error) { + if (errorCode(error) === "ENOENT") return false; + throw error; + } +} + +/** + * The missing-row incident proved that absence is not authority: installing + * `{0,null}` over a legacy JSON pair or routed native bytes loses the only + * evidence that an interrupted transition still needs salvage. + */ +function assertInitialStateCanBeCreated(): void { + const integration = readIntegrationRecord(); + if (integration.kind === "invalid") { + throw new CodexCoordinatorLegacyAmbiguousError( + "A missing coordinator row cannot be initialized over legacy or invalid Codex integration state.", + ); + } + if (hasNativeRoutedResidue()) { + throw new CodexCoordinatorLegacyAmbiguousError( + "A missing coordinator row cannot be initialized while native Codex routing residue exists.", + ); + } +} + function initialize(database: Database, databaseWasAbsent: boolean): void { const version = database.query<{ user_version: number }, []>("PRAGMA user_version").get()?.user_version; if (version !== 0 && version !== COORDINATOR_SCHEMA_VERSION) { @@ -314,7 +307,10 @@ function initialize(database: Database, databaseWasAbsent: boolean): void { "The existing coordinator database has no authoritative transition row.", ); } - if (!existing) database.query(INITIALIZE_TRANSITION_ROW).run(new Date().toISOString()); + if (!existing) { + assertInitialStateCanBeCreated(); + database.query(INITIALIZE_TRANSITION_ROW).run(new Date().toISOString()); + } if (version === 0) database.exec(`PRAGMA user_version = ${COORDINATOR_SCHEMA_VERSION}`); readState(database); } @@ -322,7 +318,7 @@ function initialize(database: Database, databaseWasAbsent: boolean): void { function createCapability( database: Database, onResult: (result: TransitionStateUpdate) => void, -): CodexCoordinatorTransaction { +): BrandedCodexCoordinatorTransaction { let consumed = false; return { [codexCoordinatorTransactionBrand]: true, @@ -485,7 +481,7 @@ function mapReadError(error: unknown): TransitionStateRead { return mapUnavailable(error); } -export function readCodexTransitionState(): TransitionStateRead { +export const readCodexTransitionState: ReadCodexTransitionState = () => { let transaction: CodexCoordinatorTransactionController | undefined; try { transaction = openCodexCoordinatorTransaction(currentCoordinatorDatabasePath()); @@ -501,7 +497,7 @@ export function readCodexTransitionState(): TransitionStateRead { } finally { transaction?.close(); } -} +}; function readCommittedState(): TransitionStateRead { const path = currentCoordinatorDatabasePath(); diff --git a/src/config.ts b/src/config.ts index 4fd1cc86f..09019e9e8 100644 --- a/src/config.ts +++ b/src/config.ts @@ -5,6 +5,16 @@ import { homedir } from "node:os"; import { dirname, join, resolve } from "node:path"; import { Database } from "bun:sqlite"; import * as z from "zod/v4"; +import { + bumpConfigGenerationAtPath, + bumpCurrentConfigGeneration, + initializeConfigGeneration, + readConfigGenerationAtPath, +} from "./codex/generation"; +import type { + BumpConfigGeneration, + ReadConfigGeneration, +} from "./codex/convergence-types"; import { CODEX_ACCOUNT_NAMESPACE_COMBO_ALIAS_COLLISION_ERROR, codexAccountNamespaceForModel, @@ -1763,6 +1773,7 @@ function configMutationDatabasePath(): string { } let configMutationLockDepth = 0; +let configMutationDatabase: Database | null = null; /** * Serialize synchronous config and Codex credential-generation commits across processes with an @@ -1789,7 +1800,11 @@ export function withConfigMutationLockSync(fn: () => T): T { try { chmodSync(path, 0o600); } catch { /* platform may ignore chmod */ } database.exec("PRAGMA busy_timeout = 0; BEGIN IMMEDIATE"); transactionOpen = true; + initializeConfigGeneration(database); } catch (cause) { + if (transactionOpen) { + try { database?.exec("ROLLBACK"); } catch { /* close below still releases the OS lock */ } + } try { database?.close(); } catch { /* acquisition already failed */ } const code = cause && typeof cause === "object" && "code" in cause ? String((cause as { code?: unknown }).code) @@ -1801,6 +1816,7 @@ export function withConfigMutationLockSync(fn: () => T): T { } configMutationLockDepth = 1; + configMutationDatabase = database; try { const value = fn(); database.exec("COMMIT"); @@ -1814,19 +1830,52 @@ export function withConfigMutationLockSync(fn: () => T): T { throw error; } finally { configMutationLockDepth = 0; + configMutationDatabase = null; try { database.close(); } catch { /* the OS lock is released with the handle */ } } } -function persistConfigUnlocked(config: OcxConfig): void { +function bumpGenerationForCooperatingConfigWrite(): void { + if (!configMutationDatabase) { + throw new Error("A cooperating config write requires the config mutation transaction."); + } + bumpCurrentConfigGeneration(configMutationDatabase); +} + +export const readConfigGeneration: ReadConfigGeneration = () => { + try { + return readConfigGenerationAtPath(configMutationDatabasePath()); + } catch { + return { kind: "unavailable", reason: "database" }; + } +}; + +export const bumpConfigGeneration: BumpConfigGeneration = expected => { + try { + return bumpConfigGenerationAtPath(configMutationDatabasePath(), expected); + } catch { + return { kind: "unavailable", reason: "database" }; + } +}; + +function persistConfigUnlocked(config: OcxConfig): boolean { const configPath = getConfigPath(); - atomicWriteFile(configPath, JSON.stringify(config, null, 2) + "\n"); + const bytes = JSON.stringify(config, null, 2) + "\n"; + try { + if (readFileSync(configPath, "utf8") === bytes) return false; + } catch (error) { + if (!isMissingPathError(error)) throw error; + } + atomicWriteFile(configPath, bytes); + return true; } export function saveConfig(config: OcxConfig): void { // Keep the real-home assertion ahead of even lock-directory preparation. assertNotRealHomeUnderTest(getConfigDir()); - withConfigMutationLockSync(() => persistConfigUnlocked(config)); + withConfigMutationLockSync(() => { + if (persistConfigUnlocked(config)) bumpGenerationForCooperatingConfigWrite(); + }); } export type PersistedConfigMutation = { @@ -1906,7 +1955,7 @@ export function mutatePersistedConfig( continue; } - persistConfigUnlocked(confirmedConfig); + if (persistConfigUnlocked(confirmedConfig)) bumpGenerationForCooperatingConfigWrite(); return { status: "committed", value: confirmed.value }; } return { status: "unavailable", reason: "conflict" }; @@ -2165,10 +2214,10 @@ export function saveConfigPreservingClaudeCode(config: OcxConfig): void { const persistedConfig: OcxConfig = { ...config, port: persistedBinding.port }; if (persistedBinding.hostname === undefined) delete persistedConfig.hostname; else persistedConfig.hostname = persistedBinding.hostname; - persistConfigUnlocked(persistedConfig); + if (persistConfigUnlocked(persistedConfig)) bumpGenerationForCooperatingConfigWrite(); persistedLiveServerBinding.set(config, persistedBinding); } else { - persistConfigUnlocked(config); + if (persistConfigUnlocked(config)) bumpGenerationForCooperatingConfigWrite(); } if (claudeCodeBaseline.has(config)) { claudeCodeBaseline.set(config, structuredClone(config.claudeCode)); diff --git a/tests/codex-config-generation.test.ts b/tests/codex-config-generation.test.ts new file mode 100644 index 000000000..6b143523c --- /dev/null +++ b/tests/codex-config-generation.test.ts @@ -0,0 +1,115 @@ +import { afterEach, beforeEach, expect, test } from "bun:test"; +import { mkdtempSync, rmSync, writeFileSync } from "node:fs"; +import { join } from "node:path"; + +import { Database } from "bun:sqlite"; + +import { + bumpConfigGeneration, + mutatePersistedConfig, + readConfigGeneration, + saveConfig, + saveConfigPreservingClaudeCode, +} from "../src/config"; +import type { OcxConfig } from "../src/types"; + +let testRoot = ""; +let previousOpencodexHome: string | undefined; + +function config(port = 10100): OcxConfig { + return { port, providers: {}, defaultProvider: "openai" }; +} + +beforeEach(() => { + previousOpencodexHome = process.env.OPENCODEX_HOME; + testRoot = mkdtempSync(join(import.meta.dir, ".tmp-codex-config-generation-")); + process.env.OPENCODEX_HOME = testRoot; +}); + +afterEach(() => { + if (previousOpencodexHome === undefined) delete process.env.OPENCODEX_HOME; + else process.env.OPENCODEX_HOME = previousOpencodexHome; + rmSync(testRoot, { recursive: true, force: true }); +}); + +test("an initial read creates the singleton generation at zero", () => { + expect(readConfigGeneration()).toEqual({ + kind: "ready", + generation: { value: 0 }, + }); +}); + +test("a cooperating save bumps once and an unchanged save does not bump", () => { + saveConfig(config()); + expect(readConfigGeneration()).toEqual({ + kind: "ready", + generation: { value: 1 }, + }); + + saveConfig(config()); + expect(readConfigGeneration()).toEqual({ + kind: "ready", + generation: { value: 1 }, + }); + + saveConfig(config(20200)); + expect(readConfigGeneration()).toEqual({ + kind: "ready", + generation: { value: 2 }, + }); +}); + +test("every cooperating writer bumps only when its committed bytes change", () => { + saveConfig(config()); + + expect(mutatePersistedConfig(persisted => { + persisted.port = 20200; + return { changed: true, value: persisted.port }; + })).toEqual({ status: "committed", value: 20200 }); + expect(readConfigGeneration()).toMatchObject({ generation: { value: 2 } }); + + expect(mutatePersistedConfig(persisted => ( + { changed: false, value: persisted.port } + ))).toEqual({ status: "unchanged", value: 20200 }); + expect(readConfigGeneration()).toMatchObject({ generation: { value: 2 } }); + + saveConfigPreservingClaudeCode(config(30300)); + expect(readConfigGeneration()).toMatchObject({ generation: { value: 3 } }); + saveConfigPreservingClaudeCode(config(30300)); + expect(readConfigGeneration()).toMatchObject({ generation: { value: 3 } }); +}); + +test("a stale expected value conflicts without changing the winner", () => { + const admitted = readConfigGeneration(); + expect(admitted.kind).toBe("ready"); + if (admitted.kind !== "ready") throw new Error("generation unavailable"); + + saveConfig(config()); + expect(bumpConfigGeneration(admitted.generation)).toEqual({ + kind: "conflict", + current: { value: 1 }, + }); + expect(readConfigGeneration()).toEqual({ + kind: "ready", + generation: { value: 1 }, + }); +}); + +test("busy and unavailable databases return typed outcomes instead of throwing", () => { + expect(readConfigGeneration().kind).toBe("ready"); + const databasePath = join(testRoot, "config-mutation.sqlite"); + const holder = new Database(databasePath, { readwrite: true, create: false }); + holder.exec("PRAGMA busy_timeout = 0; BEGIN IMMEDIATE"); + try { + expect(readConfigGeneration()).toEqual({ kind: "unavailable", reason: "busy" }); + expect(bumpConfigGeneration({ value: 0 })).toEqual({ kind: "unavailable", reason: "busy" }); + } finally { + holder.exec("ROLLBACK"); + holder.close(); + } + + rmSync(testRoot, { recursive: true, force: true }); + writeFileSync(testRoot, "not a directory", "utf8"); + expect(readConfigGeneration()).toEqual({ kind: "unavailable", reason: "database" }); + expect(bumpConfigGeneration({ value: 0 })).toEqual({ kind: "unavailable", reason: "database" }); +}); diff --git a/tests/codex-integration-record.test.ts b/tests/codex-integration-record.test.ts index 4d90a0131..57ad5f970 100644 --- a/tests/codex-integration-record.test.ts +++ b/tests/codex-integration-record.test.ts @@ -6,7 +6,11 @@ import { readIntegrationRecord, updateIntegrationRecord, } from "../src/codex/integration-record"; -import type { CodexIntegrationRecord } from "../src/codex/convergence-types"; +import type { + CodexArtifactId, + CodexIntegrationRecord, + CodexProvenanceEntry, +} from "../src/codex/convergence-types"; let opencodexHome = ""; let previousOpencodexHome: string | undefined; @@ -24,6 +28,26 @@ function persistedRecord(): Record { return JSON.parse(readFileSync(integrationRecordPath(), "utf8")) as Record; } +function knownArtifactFields(artifact: CodexArtifactId): CodexArtifactId { + switch (artifact.kind) { + case "config": + case "generated-profile": + case "models-cache": + case "injection-journal": + return { kind: artifact.kind }; + case "active-catalog": + return { kind: artifact.kind, canonicalPath: artifact.canonicalPath }; + case "catalog-backup": + return { kind: artifact.kind, form: artifact.form, canonicalPath: artifact.canonicalPath }; + case "history-row": + case "history-manifest-entry": + return { kind: artifact.kind, stateDbId: artifact.stateDbId, threadId: artifact.threadId }; + case "history-manifest": + case "history-rollout": + return { kind: artifact.kind, stateDbId: artifact.stateDbId, canonicalPath: artifact.canonicalPath }; + } +} + beforeEach(() => { previousOpencodexHome = process.env.OPENCODEX_HOME; opencodexHome = mkdtempSync(join(tmpdir(), "ocx-integration-record-")); @@ -47,34 +71,38 @@ describe("Codex integration record", () => { }); test("preserves future keys at record, ledger, entry, artifact, and both baseline levels", () => { + const artifacts: CodexArtifactId[] = [ + { kind: "config", futureArtifact: { owner: "future-config" } }, + { kind: "generated-profile", futureArtifact: { owner: "future-profile" } }, + { kind: "active-catalog", canonicalPath: "/catalog", futureArtifact: { owner: "future-catalog" } }, + { kind: "catalog-backup", form: "hashed", canonicalPath: "/backup", futureArtifact: { owner: "future-backup" } }, + { kind: "models-cache", futureArtifact: { owner: "future-cache" } }, + { kind: "injection-journal", futureArtifact: { owner: "future-journal" } }, + { kind: "history-row", stateDbId: "db-row", threadId: "thread-row", futureArtifact: { owner: "future-row" } }, + { kind: "history-manifest", stateDbId: "db-manifest", canonicalPath: "/manifest", futureArtifact: { owner: "future-manifest" } }, + { kind: "history-manifest-entry", stateDbId: "db-entry", threadId: "thread-entry", futureArtifact: { owner: "future-entry" } }, + { kind: "history-rollout", stateDbId: "db-rollout", canonicalPath: "/rollout", futureArtifact: { owner: "future-rollout" } }, + ]; writeRecord({ version: 1, futureRecord: { mode: "newer" }, provenance: { futureLedger: ["keep"], - entries: [ - { - artifact: { kind: "config", futureArtifact: { owner: "future-config" } }, - baseline: { kind: "absent", futureAbsentBaseline: 17 }, - postImage: "old-config-post-image", - txId: "tx-config", - at: "2026-08-04T00:00:00.000Z", - futureEntry: { evidence: true }, - }, - { - artifact: { kind: "generated-profile", futureArtifact: { owner: "future-profile" } }, - baseline: { - kind: "present", - sha256: "baseline-sha", - bytesBase64: "YmFzZWxpbmU=", - futurePresentBaseline: { codec: 2 }, - }, - postImage: "old-profile-post-image", - txId: "tx-profile", - at: "2026-08-04T00:00:01.000Z", - futureEntry: { evidence: false }, - }, - ], + entries: artifacts.map((artifact, index): CodexProvenanceEntry => ({ + artifact, + baseline: index % 2 === 0 + ? { kind: "absent", futureAbsentBaseline: index } + : { + kind: "present", + sha256: `baseline-sha-${index}`, + bytesBase64: "YmFzZWxpbmU=", + futurePresentBaseline: { codec: index }, + }, + postImage: `old-post-image-${index}`, + txId: `tx-${index}`, + at: `2026-08-04T00:00:${String(index).padStart(2, "0")}.000Z`, + futureEntry: { evidence: index }, + })), }, }); @@ -82,7 +110,7 @@ describe("Codex integration record", () => { version: 1, provenance: { entries: record.provenance!.entries.map((entry, index) => ({ - artifact: { kind: entry.artifact.kind } as typeof entry.artifact, + artifact: knownArtifactFields(entry.artifact), baseline: entry.baseline.kind === "absent" ? { kind: "absent" } : { @@ -103,16 +131,18 @@ describe("Codex integration record", () => { const ledger = saved.provenance as Record; expect(ledger.futureLedger).toEqual(["keep"]); const entries = ledger.entries as Array>; - expect(entries[0]!.futureEntry).toEqual({ evidence: true }); - expect(entries[1]!.futureEntry).toEqual({ evidence: false }); - expect((entries[0]!.artifact as Record).futureArtifact) - .toEqual({ owner: "future-config" }); - expect((entries[1]!.artifact as Record).futureArtifact) - .toEqual({ owner: "future-profile" }); - expect((entries[0]!.baseline as Record).futureAbsentBaseline).toBe(17); + expect(entries.map(entry => entry.futureEntry)).toEqual( + artifacts.map((_, index) => ({ evidence: index })), + ); + expect(entries.map(entry => (entry.artifact as Record).futureArtifact)).toEqual( + artifacts.map(artifact => artifact.futureArtifact), + ); + expect((entries[0]!.baseline as Record).futureAbsentBaseline).toBe(0); expect((entries[1]!.baseline as Record).futurePresentBaseline) - .toEqual({ codec: 2 }); - expect(entries.map(entry => entry.postImage)).toEqual(["new-post-image-0", "new-post-image-1"]); + .toEqual({ codec: 1 }); + expect(entries.map(entry => entry.postImage)).toEqual( + artifacts.map((_, index) => `new-post-image-${index}`), + ); }); test("fails closed on unparseable bytes without invoking the mutator or resetting the file", () => { @@ -133,10 +163,26 @@ describe("Codex integration record", () => { expect(readFileSync(integrationRecordPath(), "utf8")).toBe("{ definitely-not-json"); }); - test("creates the minimal v1 record when the file is missing", () => { - const result = updateIntegrationRecord(record => record); + test("creates the minimal v1 record on the first provenance write", () => { + const firstEntry: CodexProvenanceEntry = { + artifact: { kind: "config" }, + baseline: { kind: "absent" }, + postImage: "first-post-image", + txId: "tx-first", + at: "2026-08-04T00:00:00.000Z", + }; + const result = updateIntegrationRecord(record => ({ + ...record, + provenance: { entries: [firstEntry] }, + })); - expect(result).toEqual({ kind: "updated", record: { version: 1 } }); - expect(persistedRecord()).toEqual({ version: 1 }); + expect(result).toEqual({ + kind: "updated", + record: { version: 1, provenance: { entries: [firstEntry] } }, + }); + expect(persistedRecord()).toEqual({ + version: 1, + provenance: { entries: [firstEntry] }, + }); }); }); diff --git a/tests/codex-transition-state.test.ts b/tests/codex-transition-state.test.ts index b82e15267..b4143b54a 100644 --- a/tests/codex-transition-state.test.ts +++ b/tests/codex-transition-state.test.ts @@ -1,5 +1,5 @@ import { afterEach, beforeEach, expect, test } from "bun:test"; -import { chmodSync, mkdtempSync, realpathSync, rmSync } from "node:fs"; +import { chmodSync, mkdirSync, mkdtempSync, realpathSync, rmSync, writeFileSync } from "node:fs"; import { join } from "node:path"; import { tmpdir } from "node:os"; @@ -17,13 +17,18 @@ import { } from "../src/codex/user-identity"; let codexHome = ""; +let opencodexHome = ""; let coordinatorPath = ""; let previousCodexHome: string | undefined; +let previousOpencodexHome: string | undefined; beforeEach(() => { previousCodexHome = process.env.CODEX_HOME; + previousOpencodexHome = process.env.OPENCODEX_HOME; codexHome = mkdtempSync(join(tmpdir(), "ocx-transition-state-codex-home-")); + opencodexHome = mkdtempSync(join(tmpdir(), "ocx-transition-state-opencodex-home-")); process.env.CODEX_HOME = codexHome; + process.env.OPENCODEX_HOME = opencodexHome; coordinatorPath = resolveCodexCoordinatorDatabasePath( resolveEffectiveUserIdentity(), realpathSync.native(codexHome), @@ -33,10 +38,13 @@ beforeEach(() => { afterEach(() => { if (previousCodexHome === undefined) delete process.env.CODEX_HOME; else process.env.CODEX_HOME = previousCodexHome; + if (previousOpencodexHome === undefined) delete process.env.OPENCODEX_HOME; + else process.env.OPENCODEX_HOME = previousOpencodexHome; for (const suffix of ["", "-journal", "-wal", "-shm"]) { rmSync(`${coordinatorPath}${suffix}`, { force: true }); } rmSync(codexHome, { recursive: true, force: true }); + rmSync(opencodexHome, { recursive: true, force: true }); }); function transition(txId: string) { @@ -48,7 +56,7 @@ function transition(txId: string) { }; } -test("a matching conditional transition update succeeds", () => { +test("a missing database initializes only from clean integration and native state", () => { expect(readCodexTransitionState()).toEqual({ kind: "ready", state: { @@ -78,6 +86,45 @@ test("a matching conditional transition update succeeds", () => { } }); +/** + * The missing-row review fixture carried the old JSON pair and history. Before + * this regression, initialization silently replaced that evidence with + * `{0,null}`, making an interrupted legacy transition look clean. + */ +test("a missing database with legacy JSON transition fields is legacy-ambiguous", () => { + const integrations = join(opencodexHome, "integrations"); + mkdirSync(integrations, { recursive: true }); + writeFileSync(join(integrations, "codex.json"), JSON.stringify({ + version: 1, + nativeGeneration: 7, + currentTxId: "legacy", + history: { status: "pending", txId: "legacy" }, + })); + + expect(readCodexTransitionState()).toEqual({ + kind: "legacy-ambiguous", + message: "A missing coordinator row cannot be initialized over legacy or invalid Codex integration state.", + }); +}); + +/** + * Absence of the coordinator file also said nothing about native bytes. The + * exact marker-owned routing grammar is authoritative residue and must prevent + * a fresh zero row from claiming no transition ever happened. + */ +test("a missing database with native routed residue is legacy-ambiguous", () => { + writeFileSync(join(codexHome, "config.toml"), [ + "# Auto-injected by opencodex", + 'openai_base_url = "http://127.0.0.1:10100/v1"', + "", + ].join("\n")); + + expect(readCodexTransitionState()).toEqual({ + kind: "legacy-ambiguous", + message: "A missing coordinator row cannot be initialized while native Codex routing residue exists.", + }); +}); + test("an existing database without the singleton row is legacy-ambiguous", () => { const database = new Database(coordinatorPath, { create: true }); database.exec("PRAGMA user_version = 1"); @@ -131,6 +178,24 @@ test("a positive generation cannot carry a null direction", () => { } }); +/** + * A capability backed by a nominal transaction is not opaque if its caller can + * simply open another connection. The C-phase review found the old test only + * checked a boolean in one object and never exercised SQLite exclusion. + */ +test("the opaque coordinator capability cannot reach a second connection", () => { + expect(readCodexTransitionState().kind).toBe("ready"); + const controller = openCodexCoordinatorTransaction(coordinatorPath); + try { + expect(() => { + const second = openCodexCoordinatorTransaction(coordinatorPath); + second.close(); + }).toThrow(); + } finally { + controller.close(); + } +}); + test("the opaque coordinator capability is one-shot", () => { const controller = openCodexCoordinatorTransaction(coordinatorPath); try { From 96a90fd4da346b00c8903f82a537f13e6ba504fb Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Tue, 4 Aug 2026 15:52:44 +0900 Subject: [PATCH 044/163] feat(codex): the two WP8b primitives WP9 cannot land without MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit catalog-admission.ts captures the config generation and catalog target identities from the SAME OcxConfig object the management callback already holds — reference identity, not a re-read, because a re-read would change what a provider edit does. Target identity is canonical parent plus dev/inode, not a path string: a parent symlink can retarget while the textual path is unchanged, and the test proves exactly that case. management-convergence.ts is a management-only factory whose closure captures that object. ConvergeRequest deliberately does not grow a config field — letting arbitrary callers substitute catalog authority would make the scoped funnel weaker than the callback it replaces. It reports honestly rather than pretending: until WP9 supplies the real gather/commit, the catalog disposition is skipped/not-requested with changed:false, and every non-catalog observation stays not-evaluated. A placeholder that claimed a committed catalog would be the false-green class this project keeps producing. All four tests proven by reverting the behavior each covers: the cloned-config case, the textual-parent case, the false commit, and the removed scope guard each went red, then green. 33 WP8b tests pass, typecheck clean. --- src/codex/catalog-admission.ts | 83 +++++++++++++++++++ src/codex/management-convergence.ts | 85 +++++++++++++++++++ tests/codex-catalog-admission.test.ts | 96 ++++++++++++++++++++++ tests/codex-management-convergence.test.ts | 79 ++++++++++++++++++ 4 files changed, 343 insertions(+) create mode 100644 src/codex/catalog-admission.ts create mode 100644 src/codex/management-convergence.ts create mode 100644 tests/codex-catalog-admission.test.ts create mode 100644 tests/codex-management-convergence.test.ts diff --git a/src/codex/catalog-admission.ts b/src/codex/catalog-admission.ts new file mode 100644 index 000000000..60dfca405 --- /dev/null +++ b/src/codex/catalog-admission.ts @@ -0,0 +1,83 @@ +/** + * Minimal catalog admission for the management refresh path. + * + * The r2 #1 catalog incident left gather and native writes in one awaited + * callback. WP9 needs generation and filesystem-target evidence before it can + * split those phases, but importing WP12 authority would make WP9 depend on a + * later phase. This reader therefore captures only the exact resident config, + * its cooperating generation, and identities for catalog-owned targets. + */ +import { realpathSync, statSync } from "node:fs"; +import { dirname, resolve } from "node:path"; + +import { readConfigGeneration } from "../config"; +import type { OcxConfig } from "../types"; +import type { CatalogAdmissionSnapshot } from "./convergence-types"; +import { + activeCodexModelsCachePath, + catalogBackupPathFor, + isDefaultCatalogPath, + legacyCatalogBackupPath, + readCodexCatalogPath, +} from "./catalog/parsing"; + +function optionalFileIdentity(path: string): Readonly<{ device: string; inode: string }> | null { + try { + const entry = statSync(path, { bigint: true }); + return { device: String(entry.dev), inode: String(entry.ino) }; + } catch (error) { + if (error && typeof error === "object" && "code" in error && error.code === "ENOENT") { + return null; + } + throw error; + } +} + +/** + * Encode identity evidence in the contract-owned string slot. + * + * `CatalogAdmissionSnapshot` deliberately owns the target shape. Encoding the + * evidence here avoids a second shared target type while still detecting the + * parent-symlink retarget that a textual path alone missed during C2 review. + */ +function captureTargetIdentity(path: string): string { + const textualPath = resolve(path); + const canonicalParent = realpathSync.native(dirname(textualPath)); + const parent = statSync(canonicalParent, { bigint: true }); + return JSON.stringify({ + path: textualPath, + canonicalParent, + parentIdentity: { device: String(parent.dev), inode: String(parent.ino) }, + fileIdentity: optionalFileIdentity(textualPath), + }); +} + +/** + * Capture the catalog-only evidence WP9 can validate without consulting WP12. + * The config reference is retained verbatim; no persisted config re-read may + * replace the object already held by the management callback. + */ +export function captureCatalogAdmissionSnapshot( + config: Readonly, +): CatalogAdmissionSnapshot { + const generation = readConfigGeneration(); + if (generation.kind !== "ready") { + throw new Error(`Cannot capture Codex catalog admission: config generation is ${generation.reason}.`); + } + + const catalogPath = readCodexCatalogPath(); + const backupPaths = [ + catalogBackupPathFor(catalogPath), + ...(isDefaultCatalogPath(catalogPath) ? [legacyCatalogBackupPath()] : []), + ]; + + return { + config, + generation: generation.generation.value, + targets: { + catalog: captureTargetIdentity(catalogPath), + cache: captureTargetIdentity(activeCodexModelsCachePath()), + catalogBackups: backupPaths.map(captureTargetIdentity), + }, + }; +} diff --git a/src/codex/management-convergence.ts b/src/codex/management-convergence.ts new file mode 100644 index 000000000..397926454 --- /dev/null +++ b/src/codex/management-convergence.ts @@ -0,0 +1,85 @@ +/** + * Management-scoped projection before WP9 installs real catalog convergence. + * + * The r2 #1 callback swallowed every catalog failure and accepted only the + * management context's resident config. Keeping that exact object in this + * factory prevents callers from adding substitute authority to ConvergeRequest. + * Until WP9 supplies gather/commit, the catalog disposition says no work ran. + */ +import type { OcxConfig } from "../types"; +import type { + CatalogDisposition, + CodexHistoryState, + CodexObservedState, + ConvergeCodex, +} from "./convergence-types"; + +function notEvaluatedHistory(): CodexHistoryState { + return { + status: "not-evaluated", + attempts: 0, + nextRetryAt: null, + txId: null, + pendingRows: null, + backupEntries: null, + }; +} + +function notEvaluatedObserved(history: CodexHistoryState): CodexObservedState { + return { + aggregate: "not-evaluated", + isApplied: null, + desired: "unknown", + converged: null, + authority: { service: "unknown", externalProvider: null }, + surfaces: { + config: "not-evaluated", + profile: "not-evaluated", + catalog: "not-evaluated", + cache: "not-evaluated", + journal: "not-evaluated", + history: { + state: history, + database: "not-evaluated", + manifest: "not-evaluated", + rollouts: "not-evaluated", + }, + provenance: { + state: "not-evaluated", + nativeGeneration: null, + currentTxId: null, + }, + }, + }; +} + +function catalogNotRequested(): CatalogDisposition { + return { status: "skipped", reason: "not-requested", retryable: false }; +} + +/** + * Bind the management callback's exact config authority to a catalog-only + * funnel. This module is intentionally not re-exported by a public Codex facade. + */ +export function createManagementConvergeCodex( + config: Readonly, +): ConvergeCodex { + const retainedConfig = config; + return async request => { + if (request.scope !== "catalog") { + throw new Error("Management Codex convergence accepts only catalog-scoped requests."); + } + + // WP9 replaces this no-work projection and consumes this exact reference. + void retainedConfig; + const history = notEvaluatedHistory(); + const observed = notEvaluatedObserved(history); + return { + kind: "catalog-only", + changed: false, + observed, + catalogRefresh: catalogNotRequested(), + history, + }; + }; +} diff --git a/tests/codex-catalog-admission.test.ts b/tests/codex-catalog-admission.test.ts new file mode 100644 index 000000000..6930bca34 --- /dev/null +++ b/tests/codex-catalog-admission.test.ts @@ -0,0 +1,96 @@ +import { afterEach, beforeEach, expect, test } from "bun:test"; +import { + mkdirSync, + mkdtempSync, + realpathSync, + rmSync, + symlinkSync, + unlinkSync, + writeFileSync, +} from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +import { captureCatalogAdmissionSnapshot } from "../src/codex/catalog-admission"; +import { saveConfig } from "../src/config"; +import type { OcxConfig } from "../src/types"; + +let testRoot = ""; +let codexHome = ""; +let opencodexHome = ""; +let previousCodexHome: string | undefined; +let previousOpencodexHome: string | undefined; + +function config(port = 10100): OcxConfig { + return { port, providers: {}, defaultProvider: "openai" }; +} + +beforeEach(() => { + previousCodexHome = process.env.CODEX_HOME; + previousOpencodexHome = process.env.OPENCODEX_HOME; + testRoot = realpathSync.native(mkdtempSync(join(tmpdir(), "ocx-catalog-admission-"))); + codexHome = join(testRoot, "codex-home"); + opencodexHome = join(testRoot, "opencodex-home"); + mkdirSync(codexHome, { recursive: true }); + mkdirSync(opencodexHome, { recursive: true }); + process.env.CODEX_HOME = codexHome; + process.env.OPENCODEX_HOME = opencodexHome; +}); + +afterEach(() => { + if (previousCodexHome === undefined) delete process.env.CODEX_HOME; + else process.env.CODEX_HOME = previousCodexHome; + if (previousOpencodexHome === undefined) delete process.env.OPENCODEX_HOME; + else process.env.OPENCODEX_HOME = previousOpencodexHome; + rmSync(testRoot, { recursive: true, force: true }); +}); + +test("captures the given config reference, generation, and catalog target identities", () => { + saveConfig(config(20200)); + const residentConfig = config(30300); + writeFileSync(join(codexHome, "opencodex-catalog.json"), "{}\n"); + writeFileSync(join(codexHome, "models_cache.json"), "{}\n"); + + const snapshot = captureCatalogAdmissionSnapshot(residentConfig); + + expect(snapshot.config).toBe(residentConfig); + expect(snapshot.config.port).toBe(30300); + expect(snapshot.generation).toBe(1); + expect(JSON.parse(snapshot.targets.catalog)).toMatchObject({ + path: join(codexHome, "opencodex-catalog.json"), + canonicalParent: codexHome, + parentIdentity: { device: expect.any(String), inode: expect.any(String) }, + fileIdentity: { device: expect.any(String), inode: expect.any(String) }, + }); + expect(JSON.parse(snapshot.targets.cache)).toMatchObject({ + path: join(codexHome, "models_cache.json"), + canonicalParent: codexHome, + fileIdentity: { device: expect.any(String), inode: expect.any(String) }, + }); + expect(snapshot.targets.catalogBackups).toHaveLength(2); +}); + +test("changes target identity when a parent symlink retargets without changing the path", () => { + const parentA = join(testRoot, "catalog-parent-a"); + const parentB = join(testRoot, "catalog-parent-b"); + const linkedParent = join(testRoot, "catalog-parent"); + mkdirSync(parentA); + mkdirSync(parentB); + symlinkSync(parentA, linkedParent, process.platform === "win32" ? "junction" : "dir"); + const textualCatalogPath = join(linkedParent, "catalog.json"); + writeFileSync( + join(codexHome, "config.toml"), + `model_catalog_json = ${JSON.stringify(textualCatalogPath)}\n`, + ); + + const before = JSON.parse(captureCatalogAdmissionSnapshot(config()).targets.catalog); + if (process.platform === "win32") rmSync(linkedParent, { recursive: true, force: true }); + else unlinkSync(linkedParent); + symlinkSync(parentB, linkedParent, process.platform === "win32" ? "junction" : "dir"); + const after = JSON.parse(captureCatalogAdmissionSnapshot(config()).targets.catalog); + + expect(before.path).toBe(textualCatalogPath); + expect(after.path).toBe(textualCatalogPath); + expect(before.canonicalParent).not.toBe(after.canonicalParent); + expect(before.parentIdentity).not.toEqual(after.parentIdentity); +}); diff --git a/tests/codex-management-convergence.test.ts b/tests/codex-management-convergence.test.ts new file mode 100644 index 000000000..dfeaf3040 --- /dev/null +++ b/tests/codex-management-convergence.test.ts @@ -0,0 +1,79 @@ +import { expect, test } from "bun:test"; + +import { createManagementConvergeCodex } from "../src/codex/management-convergence"; +import type { ConvergeRequest } from "../src/codex/convergence-types"; +import type { OcxConfig } from "../src/types"; + +function config(): OcxConfig { + return { port: 10100, providers: {}, defaultProvider: "openai" }; +} + +function request(scope: ConvergeRequest["scope"]): ConvergeRequest { + return { + action: "converge", + scope, + reason: "management-mutation", + mode: "automatic", + deadlineMs: 1_000, + }; +} + +test("returns an honest catalog-only no-change projection", async () => { + const convergeCodex = createManagementConvergeCodex(config()); + + const outcome = await convergeCodex(request("catalog")); + + expect(outcome).toEqual({ + kind: "catalog-only", + changed: false, + catalogRefresh: { status: "skipped", reason: "not-requested", retryable: false }, + history: { + status: "not-evaluated", + attempts: 0, + nextRetryAt: null, + txId: null, + pendingRows: null, + backupEntries: null, + }, + observed: { + aggregate: "not-evaluated", + isApplied: null, + desired: "unknown", + converged: null, + authority: { service: "unknown", externalProvider: null }, + surfaces: { + config: "not-evaluated", + profile: "not-evaluated", + catalog: "not-evaluated", + cache: "not-evaluated", + journal: "not-evaluated", + history: { + state: { + status: "not-evaluated", + attempts: 0, + nextRetryAt: null, + txId: null, + pendingRows: null, + backupEntries: null, + }, + database: "not-evaluated", + manifest: "not-evaluated", + rollouts: "not-evaluated", + }, + provenance: { + state: "not-evaluated", + nativeGeneration: null, + currentTxId: null, + }, + }, + }, + }); +}); + +test("rejects a non-catalog request instead of widening management authority", async () => { + const convergeCodex = createManagementConvergeCodex(config()); + + await expect(convergeCodex(request("full"))).rejects.toThrow( + "Management Codex convergence accepts only catalog-scoped requests.", + ); +}); From 441f0bf2b77b7b698865a015c2675e2c1a5c3732 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Tue, 4 Aug 2026 17:24:49 +0900 Subject: [PATCH 045/163] fix(codex): absence of a marker in config.toml was not native-clean MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The residue check that guarded coordinator initialization read exactly one surface — `$CODEX_HOME/config.toml` — and treated every other routed byte on disk as absent. A fresh verifier reproduced the consequence by running the code: with no coordinator database and a genuine `buildProfileFile()` profile sitting in CODEX_HOME, `readCodexTransitionState()` returned a clean `{0,null}` instead of `legacy-ambiguous`, discarding the only evidence that an interrupted transition still needed salvage. 005_contract.md:409 requires native observation to find no unresolved routed residue; one file is not that observation. The replacement classifies every structurally provable routed surface — config markers, the real generated-profile grammar, routed catalog and models cache signatures, the restore journal, atomic-write artifacts, and the guardian's history evidence — and returns `clean | residue | indeterminate`. Only ENOENT counts as a clean surface. Unreadable, malformed, unresolvable or racing observations are `indeterminate`, and initialization refuses on anything that is not `clean`, because this module's recurring failure has been treating an absence as a guarantee. Proven by mutation: narrowed back to config.toml only, 16 of 21 new tests go red, including the verifier's exact repro. --- src/codex/native-residue.ts | 348 +++++++++++++++++++++++++++++ src/codex/transition-state.ts | 17 +- tests/codex-native-residue.test.ts | 299 +++++++++++++++++++++++++ 3 files changed, 650 insertions(+), 14 deletions(-) create mode 100644 src/codex/native-residue.ts create mode 100644 tests/codex-native-residue.test.ts diff --git a/src/codex/native-residue.ts b/src/codex/native-residue.ts new file mode 100644 index 000000000..cc2573f12 --- /dev/null +++ b/src/codex/native-residue.ts @@ -0,0 +1,348 @@ +import { createHash } from "node:crypto"; +import { lstatSync, readFileSync, readdirSync, realpathSync, statSync } from "node:fs"; +import type { Stats } from "node:fs"; +import { basename, join, resolve } from "node:path"; + +import { Database } from "bun:sqlite"; + +import { getConfigDir } from "../config"; +import { catalogHasRoutedEntries, parseCatalogJson } from "./catalog/parsing"; +import { + hasInjectedCodexRouting, + OCX_SECTION_MARKER, + providerTableString, + rootTomlString, +} from "./injected-marker"; +import { + CODEX_CONFIG_PATH, + CODEX_MODELS_CACHE_PATH, + CODEX_PROFILE_PATH, + DEFAULT_CATALOG_PATH, + getCodexHome, +} from "./paths"; + +export type NativeResidueSurface = + | "config" + | "profile" + | "catalog" + | "models-cache" + | "journal" + | "partial-write" + | "history" + | "history-backup"; + +export type NativeRoutedResidueResult = + | { kind: "clean" } + | { kind: "residue"; surface: NativeResidueSurface; path: string } + | { kind: "indeterminate"; surface: NativeResidueSurface; path: string; reason: string }; + +type ReadResult = + | { kind: "absent" } + | { kind: "content"; content: string; path: string } + | { kind: "indeterminate"; reason: string }; + +type PathResult = + | { kind: "absent" } + | { kind: "path"; path: string; stat: Stats } + | { kind: "indeterminate"; reason: string }; + +const CONFIG_FILE_NAME = basename(CODEX_CONFIG_PATH); +const PROFILE_FILE_NAME = basename(CODEX_PROFILE_PATH); +const CATALOG_FILE_NAME = basename(DEFAULT_CATALOG_PATH); +const MODELS_CACHE_FILE_NAME = basename(CODEX_MODELS_CACHE_PATH); +const JOURNAL_FILE_NAME = "opencodex-journal.json"; +const HISTORY_DATABASE_FILE_NAME = "state_5.sqlite"; +const ROUTED_CATALOG_DESCRIPTION_PREFIX = "Routed via opencodex → "; + +const ATOMIC_WRITE_TARGETS = new Set([ + CONFIG_FILE_NAME, + PROFILE_FILE_NAME, + CATALOG_FILE_NAME, + MODELS_CACHE_FILE_NAME, + JOURNAL_FILE_NAME, +]); + +function errorCode(error: unknown): string | undefined { + return (error as NodeJS.ErrnoException | undefined)?.code; +} + +function errorReason(error: unknown): string { + if (error instanceof Error) return `${error.name}: ${error.message}`; + return String(error); +} + +function sameStat( + left: Stats, + right: Stats, +): boolean { + return left.dev === right.dev + && left.ino === right.ino + && left.size === right.size + && left.mtimeMs === right.mtimeMs; +} + +function resolveRegularFile(path: string): PathResult { + let entry; + try { + entry = lstatSync(path); + } catch (error) { + if (errorCode(error) === "ENOENT") return { kind: "absent" }; + return { kind: "indeterminate", reason: errorReason(error) }; + } + + let target = path; + if (entry.isSymbolicLink()) { + try { + target = realpathSync.native(path); + } catch (error) { + return { kind: "indeterminate", reason: `unresolvable symlink: ${errorReason(error)}` }; + } + } + + try { + const before = statSync(target); + if (!before.isFile()) { + return { kind: "indeterminate", reason: "surface is not a regular file" }; + } + return { kind: "path", path: target, stat: before }; + } catch (error) { + return { kind: "indeterminate", reason: errorReason(error) }; + } +} + +function readRegularFile(path: string): ReadResult { + const resolved = resolveRegularFile(path); + if (resolved.kind !== "path") return resolved; + try { + const content = readFileSync(resolved.path, "utf8"); + const after = statSync(resolved.path); + if (!sameStat(resolved.stat, after)) { + return { kind: "indeterminate", reason: "surface changed while it was being observed" }; + } + return { kind: "content", content, path: resolved.path }; + } catch (error) { + return { kind: "indeterminate", reason: errorReason(error) }; + } +} + +function indeterminate( + surface: NativeResidueSurface, + path: string, + reason: string, +): NativeRoutedResidueResult { + return { kind: "indeterminate", surface, path, reason }; +} + +function classifyToml( + surface: "config" | "profile", + path: string, + classify: (content: string) => "clean" | "residue" | "indeterminate", +): NativeRoutedResidueResult { + const read = readRegularFile(path); + if (read.kind === "absent") return { kind: "clean" }; + if (read.kind === "indeterminate") return indeterminate(surface, path, read.reason); + try { + Bun.TOML.parse(read.content); + } catch (error) { + return indeterminate(surface, path, `malformed TOML: ${errorReason(error)}`); + } + const result = classify(read.content); + if (result === "residue") return { kind: "residue", surface, path: read.path }; + if (result === "indeterminate") { + return indeterminate(surface, read.path, "OpenCodex-shaped TOML does not match a complete routed grammar"); + } + return { kind: "clean" }; +} + +function classifyConfig(path: string): NativeRoutedResidueResult { + return classifyToml("config", path, content => { + if (hasInjectedCodexRouting(content)) return "residue"; + const hasMarker = content.includes(OCX_SECTION_MARKER); + const provider = rootTomlString(content, "model_provider"); + const providerBaseUrl = providerTableString(content, "opencodex", "base_url"); + return hasMarker || provider === "opencodex" || providerBaseUrl !== null + ? "indeterminate" + : "clean"; + }); +} + +function classifyProfile(path: string): NativeRoutedResidueResult { + return classifyToml("profile", path, content => { + const generatedFallback = content.startsWith("# OpenCodex proxy fallback config (Design B)") + && rootTomlString(content, "openai_base_url") !== null; + const generatedNamedProfile = content.startsWith("# OpenCodex proxy profile — use with:") + && hasInjectedCodexRouting(content); + if (generatedFallback || generatedNamedProfile) return "residue"; + return "indeterminate"; + }); +} + +function isOcxRoutedCatalogEntry(entry: Record): boolean { + return typeof entry.slug === "string" + && entry.slug.includes("/") + && typeof entry.description === "string" + && entry.description.startsWith(ROUTED_CATALOG_DESCRIPTION_PREFIX); +} + +function classifyCatalogLike( + surface: "catalog" | "models-cache", + path: string, +): NativeRoutedResidueResult { + const read = readRegularFile(path); + if (read.kind === "absent") return { kind: "clean" }; + if (read.kind === "indeterminate") return indeterminate(surface, path, read.reason); + const catalog = parseCatalogJson(read.content); + if (!catalog) return indeterminate(surface, path, "malformed catalog JSON"); + if ((catalog.models ?? []).some(isOcxRoutedCatalogEntry)) { + return { kind: "residue", surface, path: read.path }; + } + if (catalogHasRoutedEntries(catalog)) { + return indeterminate(surface, read.path, "routed catalog rows lack the OpenCodex authorship signature"); + } + return { kind: "clean" }; +} + +function isJournal(value: unknown): boolean { + if (!value || typeof value !== "object" || Array.isArray(value)) return false; + const journal = value as Record; + return journal.version === 1 + && typeof journal.originalConfig === "string" + && (journal.originalProfile === null || typeof journal.originalProfile === "string") + && typeof journal.pid === "number" + && Number.isInteger(journal.pid) + && typeof journal.timestamp === "string"; +} + +function classifyJournal(path: string): NativeRoutedResidueResult { + const read = readRegularFile(path); + if (read.kind === "absent") return { kind: "clean" }; + if (read.kind === "indeterminate") return indeterminate("journal", path, read.reason); + let parsed: unknown; + try { + parsed = JSON.parse(read.content); + } catch (error) { + return indeterminate("journal", read.path, `malformed journal JSON: ${errorReason(error)}`); + } + return isJournal(parsed) + ? { kind: "residue", surface: "journal", path: read.path } + : indeterminate("journal", read.path, "journal JSON has an unknown or partial shape"); +} + +function classifyPartialWrites(codexHome: string): NativeRoutedResidueResult { + let names: string[]; + try { + names = readdirSync(codexHome); + } catch (error) { + return indeterminate("partial-write", codexHome, errorReason(error)); + } + for (const name of names) { + const match = /^(.*)\.ocx\.\d+\.\d+\.tmp$/.exec(name); + if (match?.[1] && ATOMIC_WRITE_TARGETS.has(match[1])) { + return indeterminate("partial-write", join(codexHome, name), "OpenCodex atomic-write artifact is still present"); + } + } + return { kind: "clean" }; +} + +function classifyHistoryDatabase(path: string): NativeRoutedResidueResult { + const resolved = resolveRegularFile(path); + if (resolved.kind === "absent") { + for (const suffix of ["-wal", "-shm"]) { + const sidecar = resolveRegularFile(`${path}${suffix}`); + if (sidecar.kind !== "absent") { + const reason = sidecar.kind === "indeterminate" + ? sidecar.reason + : "SQLite sidecar exists without its history database"; + return indeterminate("history", `${path}${suffix}`, reason); + } + } + return { kind: "clean" }; + } + if (resolved.kind === "indeterminate") return indeterminate("history", path, resolved.reason); + let database: Database | undefined; + try { + database = new Database(resolved.path, { readonly: true }); + database.exec("PRAGMA busy_timeout = 100"); + const row = database.query<{ n: number }, []>(` + SELECT count(*) AS n + FROM threads + WHERE model_provider = 'opencodex' + AND trim(coalesce(first_user_message, '')) != '' + `).get(); + const after = statSync(resolved.path); + if (!sameStat(resolved.stat, after)) { + return indeterminate("history", resolved.path, "history database changed while it was being observed"); + } + return (row?.n ?? 0) > 0 + ? { kind: "residue", surface: "history", path: resolved.path } + : { kind: "clean" }; + } catch (error) { + return indeterminate("history", resolved.path, `unreadable history database: ${errorReason(error)}`); + } finally { + try { database?.close(); } catch { /* the observation already failed closed */ } + } +} + +function historyBackupPath(stateDatabasePath: string): string { + const normalized = process.platform === "win32" + ? resolve(stateDatabasePath).toLowerCase() + : resolve(stateDatabasePath); + const id = createHash("sha256").update(normalized).digest("hex").slice(0, 16); + return join(getConfigDir(), `codex-history-backup-${id}.json`); +} + +function classifyHistoryBackup(path: string, stateDatabasePath: string): NativeRoutedResidueResult { + const read = readRegularFile(path); + if (read.kind === "absent") return { kind: "clean" }; + if (read.kind === "indeterminate") return indeterminate("history-backup", path, read.reason); + let parsed: unknown; + try { + parsed = JSON.parse(read.content); + } catch (error) { + return indeterminate("history-backup", read.path, `malformed history backup JSON: ${errorReason(error)}`); + } + if (!parsed || typeof parsed !== "object" || Array.isArray(parsed)) { + return indeterminate("history-backup", read.path, "history backup has an unknown shape"); + } + const manifest = parsed as Record; + if (manifest.version !== 1 || !manifest.entries || typeof manifest.entries !== "object" || Array.isArray(manifest.entries)) { + return indeterminate("history-backup", read.path, "history backup has an unknown shape"); + } + if (typeof manifest.stateDbPath === "string") { + const expected = process.platform === "win32" ? resolve(stateDatabasePath).toLowerCase() : resolve(stateDatabasePath); + const actual = process.platform === "win32" ? resolve(manifest.stateDbPath).toLowerCase() : resolve(manifest.stateDbPath); + if (actual !== expected) { + return indeterminate("history-backup", read.path, "history backup names a different state database"); + } + } + return Object.keys(manifest.entries as Record).length > 0 + ? { kind: "residue", surface: "history-backup", path: read.path } + : { kind: "clean" }; +} + +/** Read-only, fail-closed observation of every OpenCodex-routed Codex surface. */ +export function classifyNativeRoutedResidue(): NativeRoutedResidueResult { + let codexHome: string; + try { + codexHome = getCodexHome(); + } catch (error) { + const unresolved = process.env.CODEX_HOME?.trim() || "CODEX_HOME"; + return indeterminate("partial-write", unresolved, `CODEX_HOME cannot be resolved: ${errorReason(error)}`); + } + + const stateDatabasePath = join(codexHome, HISTORY_DATABASE_FILE_NAME); + const classifiers = [ + () => classifyPartialWrites(codexHome), + () => classifyConfig(join(codexHome, CONFIG_FILE_NAME)), + () => classifyProfile(join(codexHome, PROFILE_FILE_NAME)), + () => classifyCatalogLike("catalog", join(codexHome, CATALOG_FILE_NAME)), + () => classifyCatalogLike("models-cache", join(codexHome, MODELS_CACHE_FILE_NAME)), + () => classifyJournal(join(codexHome, JOURNAL_FILE_NAME)), + () => classifyHistoryDatabase(stateDatabasePath), + () => classifyHistoryBackup(historyBackupPath(stateDatabasePath), stateDatabasePath), + ]; + const results = classifiers.map(classify => classify()); + return results.find(result => result.kind === "indeterminate") + ?? results.find(result => result.kind === "residue") + ?? { kind: "clean" }; +} diff --git a/src/codex/transition-state.ts b/src/codex/transition-state.ts index 4315b0a4b..f541b2bf6 100644 --- a/src/codex/transition-state.ts +++ b/src/codex/transition-state.ts @@ -10,8 +10,7 @@ * Design record: devlog/_plan/260804_codex_write_substrate/005_contract.md §1. */ import { randomUUID } from "node:crypto"; -import { chmodSync, lstatSync, readFileSync, realpathSync } from "node:fs"; -import { join } from "node:path"; +import { chmodSync, lstatSync, realpathSync } from "node:fs"; import { Database } from "bun:sqlite"; @@ -29,8 +28,8 @@ import type { UpdateCodexHistoryTransition, } from "./convergence-types"; import { resolveCodexHomeDir } from "./home"; -import { hasInjectedCodexRouting } from "./injected-marker"; import { readIntegrationRecord } from "./integration-record"; +import { classifyNativeRoutedResidue } from "./native-residue"; import { CodexUserIdentityRefusal, resolveCodexCoordinatorDatabasePath, @@ -261,16 +260,6 @@ function validateHistoryWrite(expected: CodexTransitionVersion, history: CodexHi } } -function hasNativeRoutedResidue(): boolean { - const configPath = join(resolveCodexHomeDir(), "config.toml"); - try { - return hasInjectedCodexRouting(readFileSync(configPath, "utf8")); - } catch (error) { - if (errorCode(error) === "ENOENT") return false; - throw error; - } -} - /** * The missing-row incident proved that absence is not authority: installing * `{0,null}` over a legacy JSON pair or routed native bytes loses the only @@ -283,7 +272,7 @@ function assertInitialStateCanBeCreated(): void { "A missing coordinator row cannot be initialized over legacy or invalid Codex integration state.", ); } - if (hasNativeRoutedResidue()) { + if (classifyNativeRoutedResidue().kind !== "clean") { throw new CodexCoordinatorLegacyAmbiguousError( "A missing coordinator row cannot be initialized while native Codex routing residue exists.", ); diff --git a/tests/codex-native-residue.test.ts b/tests/codex-native-residue.test.ts new file mode 100644 index 000000000..d0d4fdce5 --- /dev/null +++ b/tests/codex-native-residue.test.ts @@ -0,0 +1,299 @@ +import { afterEach, beforeEach, expect, test } from "bun:test"; +import { createHash } from "node:crypto"; +import { + mkdirSync, + mkdtempSync, + realpathSync, + rmSync, + symlinkSync, + writeFileSync, +} from "node:fs"; +import { tmpdir } from "node:os"; +import { join, resolve } from "node:path"; + +import { Database } from "bun:sqlite"; + +import { buildCatalogEntries } from "../src/codex/catalog"; +import { buildProfileFile } from "../src/codex/inject"; +import { classifyNativeRoutedResidue } from "../src/codex/native-residue"; +import { readCodexTransitionState } from "../src/codex/transition-state"; +import { + resolveCodexCoordinatorDatabasePath, + resolveEffectiveUserIdentity, +} from "../src/codex/user-identity"; + +let codexHome = ""; +let opencodexHome = ""; +let coordinatorPath = ""; +let previousCodexHome: string | undefined; +let previousOpencodexHome: string | undefined; + +beforeEach(() => { + previousCodexHome = process.env.CODEX_HOME; + previousOpencodexHome = process.env.OPENCODEX_HOME; + codexHome = mkdtempSync(join(tmpdir(), "ocx-native-residue-codex-")); + opencodexHome = mkdtempSync(join(tmpdir(), "ocx-native-residue-opencodex-")); + process.env.CODEX_HOME = codexHome; + process.env.OPENCODEX_HOME = opencodexHome; + coordinatorPath = resolveCodexCoordinatorDatabasePath( + resolveEffectiveUserIdentity(), + realpathSync.native(codexHome), + ); +}); + +afterEach(() => { + if (previousCodexHome === undefined) delete process.env.CODEX_HOME; + else process.env.CODEX_HOME = previousCodexHome; + if (previousOpencodexHome === undefined) delete process.env.OPENCODEX_HOME; + else process.env.OPENCODEX_HOME = previousOpencodexHome; + for (const suffix of ["", "-journal", "-wal", "-shm"]) { + rmSync(`${coordinatorPath}${suffix}`, { force: true }); + } + rmSync(codexHome, { recursive: true, force: true }); + rmSync(opencodexHome, { recursive: true, force: true }); +}); + +function pathInCodexHome(name: string): string { + return join(codexHome, name); +} + +function routedCatalog(): string { + const models = buildCatalogEntries( + null, + [], + [{ provider: "fixture-provider", id: "fixture-model" }], + ); + return JSON.stringify({ models }, null, 2) + "\n"; +} + +function createHistoryDatabase(modelProvider: "openai" | "opencodex"): void { + const database = new Database(pathInCodexHome("state_5.sqlite")); + database.exec(` + CREATE TABLE threads ( + id TEXT PRIMARY KEY, + rollout_path TEXT NOT NULL, + model_provider TEXT NOT NULL, + source TEXT NOT NULL, + first_user_message TEXT NOT NULL, + has_user_event INTEGER NOT NULL DEFAULT 0 + ) + `); + database.query(` + INSERT INTO threads ( + id, rollout_path, model_provider, source, first_user_message, has_user_event + ) VALUES (?, ?, ?, 'cli', 'routed history', 1) + `).run("thread-1", pathInCodexHome("rollout.jsonl"), modelProvider); + database.close(); +} + +function historyBackupPath(): string { + const databasePath = join(realpathSync.native(codexHome), "state_5.sqlite"); + const normalized = process.platform === "win32" + ? resolve(databasePath).toLowerCase() + : resolve(databasePath); + const id = createHash("sha256").update(normalized).digest("hex").slice(0, 16); + return join(opencodexHome, `codex-history-backup-${id}.json`); +} + +const residueFixtures: Array<{ + name: string; + surface: string; + arrange: () => void; +}> = [ + { + name: "injected config.toml", + surface: "config", + arrange: () => writeFileSync(pathInCodexHome("config.toml"), [ + "# Auto-injected by opencodex", + 'openai_base_url = "http://127.0.0.1:10100/v1"', + "", + ].join("\n")), + }, + { + name: "generated profile", + surface: "profile", + arrange: () => writeFileSync( + pathInCodexHome("opencodex.config.toml"), + buildProfileFile(10100, null), + ), + }, + { + name: "routed catalog", + surface: "catalog", + arrange: () => writeFileSync(pathInCodexHome("opencodex-catalog.json"), routedCatalog()), + }, + { + name: "routed models cache", + surface: "models-cache", + arrange: () => writeFileSync(pathInCodexHome("models_cache.json"), routedCatalog()), + }, + { + name: "restore journal", + surface: "journal", + arrange: () => writeFileSync(pathInCodexHome("opencodex-journal.json"), JSON.stringify({ + version: 1, + originalConfig: Buffer.from('model = "gpt-5.5"\n').toString("base64"), + originalProfile: null, + pid: 12345, + timestamp: "2026-08-04T00:00:00.000Z", + })), + }, + { + name: "history database row", + surface: "history", + arrange: () => createHistoryDatabase("opencodex"), + }, + { + name: "history backup entry", + surface: "history-backup", + arrange: () => writeFileSync(historyBackupPath(), JSON.stringify({ + version: 1, + stateDbPath: join(realpathSync.native(codexHome), "state_5.sqlite"), + entries: { + "thread-1": { + id: "thread-1", + rolloutPath: pathInCodexHome("rollout.jsonl"), + modelProvider: "openai", + source: "cli", + hasUserEvent: 1, + }, + }, + })), + }, +]; + +for (const fixture of residueFixtures) { + test(`${fixture.name} is structurally provable routed residue`, () => { + fixture.arrange(); + expect(classifyNativeRoutedResidue()).toMatchObject({ + kind: "residue", + surface: fixture.surface, + }); + }); +} + +test("an OpenCodex atomic-write artifact is indeterminate", () => { + writeFileSync(pathInCodexHome("config.toml.ocx.123.1.tmp"), "partial"); + expect(classifyNativeRoutedResidue()).toMatchObject({ + kind: "indeterminate", + surface: "partial-write", + }); +}); + +const indeterminateFixtures: Array<{ + name: string; + surface: string; + arrange: () => void; +}> = [ + { + name: "malformed config TOML", + surface: "config", + arrange: () => writeFileSync(pathInCodexHome("config.toml"), 'model = "unterminated\n'), + }, + { + name: "malformed profile TOML", + surface: "profile", + arrange: () => writeFileSync(pathInCodexHome("opencodex.config.toml"), "[features\n"), + }, + { + name: "malformed catalog JSON", + surface: "catalog", + arrange: () => writeFileSync(pathInCodexHome("opencodex-catalog.json"), "{not-json"), + }, + { + name: "unreadable models cache shape", + surface: "models-cache", + arrange: () => mkdirSync(pathInCodexHome("models_cache.json")), + }, + { + name: "malformed journal JSON", + surface: "journal", + arrange: () => writeFileSync(pathInCodexHome("opencodex-journal.json"), "{not-json"), + }, + { + name: "partial write", + surface: "partial-write", + arrange: () => writeFileSync(pathInCodexHome("opencodex-catalog.json.ocx.42.7.tmp"), ""), + }, + { + name: "malformed history database", + surface: "history", + arrange: () => writeFileSync(pathInCodexHome("state_5.sqlite"), "not sqlite"), + }, + { + name: "malformed history backup", + surface: "history-backup", + arrange: () => writeFileSync(historyBackupPath(), "{not-json"), + }, +]; + +for (const fixture of indeterminateFixtures) { + test(`${fixture.name} is indeterminate and refuses coordinator initialization`, () => { + fixture.arrange(); + expect(classifyNativeRoutedResidue()).toMatchObject({ + kind: "indeterminate", + surface: fixture.surface, + }); + expect(readCodexTransitionState()).toEqual({ + kind: "legacy-ambiguous", + message: "A missing coordinator row cannot be initialized while native Codex routing residue exists.", + }); + }); +} + +const symlinkTest = process.platform === "win32" ? test.skip : test; +symlinkTest("an unresolvable surface symlink is indeterminate", () => { + symlinkSync(pathInCodexHome("missing-config"), pathInCodexHome("config.toml")); + expect(classifyNativeRoutedResidue()).toMatchObject({ + kind: "indeterminate", + surface: "config", + }); +}); + +test("an empty CODEX_HOME is clean and coordinator initialization succeeds", () => { + expect(classifyNativeRoutedResidue()).toEqual({ kind: "clean" }); + expect(readCodexTransitionState()).toMatchObject({ + kind: "ready", + state: { nativeGeneration: 0, currentTxId: null }, + }); +}); + +test("user-owned non-OpenCodex content is clean and coordinator initialization succeeds", () => { + writeFileSync(pathInCodexHome("config.toml"), 'model = "gpt-5.5"\n'); + writeFileSync(pathInCodexHome("notes.txt"), "user content\n"); + writeFileSync(pathInCodexHome("opencodex-catalog.json"), JSON.stringify({ + models: [{ slug: "gpt-5.5", description: "Native GPT model" }], + })); + writeFileSync(pathInCodexHome("models_cache.json"), JSON.stringify({ + models: [{ slug: "gpt-5.5", description: "Native GPT model" }], + })); + createHistoryDatabase("openai"); + + expect(classifyNativeRoutedResidue()).toEqual({ kind: "clean" }); + expect(readCodexTransitionState()).toMatchObject({ + kind: "ready", + state: { nativeGeneration: 0, currentTxId: null }, + }); +}); + +test("CODEX_HOME is resolved at call time", () => { + const secondHome = mkdtempSync(join(tmpdir(), "ocx-native-residue-second-codex-")); + try { + writeFileSync(join(secondHome, "opencodex.config.toml"), buildProfileFile(10100, null)); + expect(classifyNativeRoutedResidue()).toEqual({ kind: "clean" }); + process.env.CODEX_HOME = secondHome; + expect(classifyNativeRoutedResidue()).toMatchObject({ kind: "residue", surface: "profile" }); + } finally { + process.env.CODEX_HOME = codexHome; + rmSync(secondHome, { recursive: true, force: true }); + } +}); + +test("a missing coordinator with only the generated profile refuses initialization", () => { + writeFileSync(join(codexHome, "opencodex.config.toml"), buildProfileFile(10100, null)); + + expect(readCodexTransitionState()).toEqual({ + kind: "legacy-ambiguous", + message: "A missing coordinator row cannot be initialized while native Codex routing residue exists.", + }); +}); From 4c4dfeda9cc81a63f2e4dd80f26ba4d7b54566fd Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Tue, 4 Aug 2026 17:24:57 +0900 Subject: [PATCH 046/163] feat(codex): the catalog seam's two primitives, as production exports MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit WP8b owns every shared contract surface so WP9 consumes rather than invents, and two of the primitives it promised existed only as a test helper and a private branch. The catalog-request constructor lived at tests/codex-management-convergence.test.ts:11, and the catalog-only projection was hardcoded inside `createManagementConvergeCodex`. Left alone, WP9 would have written its own of each, which is the two-owners-of-one-surface failure this unit exists to prevent. `createCatalogConvergeRequest` and `projectCatalogOnlyOutcome` are now exported, the factory calls the projector instead of duplicating it, and the test consumes the production constructor. `ConvergeRequest.action` stays `converge | observe`: callers say when, never which way. The projection is covered as a full cross-product — both `changed` values against all 38 finite catalog dispositions, 76 exact cases — so flipping any single branch goes red. --- src/codex/catalog-admission.ts | 26 +++- src/codex/convergence-types.ts | 12 ++ src/codex/management-convergence.ts | 26 +++- tests/codex-management-convergence.test.ts | 141 +++++++++++++++++++-- 4 files changed, 183 insertions(+), 22 deletions(-) diff --git a/src/codex/catalog-admission.ts b/src/codex/catalog-admission.ts index 60dfca405..b6ad67bdb 100644 --- a/src/codex/catalog-admission.ts +++ b/src/codex/catalog-admission.ts @@ -12,7 +12,11 @@ import { dirname, resolve } from "node:path"; import { readConfigGeneration } from "../config"; import type { OcxConfig } from "../types"; -import type { CatalogAdmissionSnapshot } from "./convergence-types"; +import type { + CatalogAdmissionSnapshot, + CatalogConvergeRequestInput, + ConvergeRequest, +} from "./convergence-types"; import { activeCodexModelsCachePath, catalogBackupPathFor, @@ -21,6 +25,26 @@ import { readCodexCatalogPath, } from "./catalog/parsing"; +/** + * Construct the one request shape permitted for management catalog refreshes. + * Callers choose the deadline only; they cannot widen scope or choose direction. + */ +export function createCatalogConvergeRequest({ + deadlineMs, +}: CatalogConvergeRequestInput): ConvergeRequest { + if (!Number.isSafeInteger(deadlineMs) || deadlineMs <= 0) { + throw new TypeError("Catalog convergence deadlineMs must be a positive safe integer."); + } + + return { + action: "converge", + scope: "catalog", + reason: "management-mutation", + mode: "automatic", + deadlineMs, + }; +} + function optionalFileIdentity(path: string): Readonly<{ device: string; inode: string }> | null { try { const entry = statSync(path, { bigint: true }); diff --git a/src/codex/convergence-types.ts b/src/codex/convergence-types.ts index e11e2c552..4c8890fdb 100644 --- a/src/codex/convergence-types.ts +++ b/src/codex/convergence-types.ts @@ -199,6 +199,11 @@ export interface ConvergeRequest { deadlineMs: number; } +/** Caller-controlled input for the fixed management catalog request shape. */ +export interface CatalogConvergeRequestInput { + deadlineMs: number; +} + export type ConvergeOutcome = | { kind: "catalog-only"; changed: boolean; observed: CodexObservedState; catalogRefresh: CatalogDisposition; @@ -218,6 +223,13 @@ export type ConvergeOutcome = observed: CodexObservedState; catalogRefresh: CatalogDisposition; history: CodexHistoryState } | { kind: "failed"; surface: string; message: string }; +export type CatalogOnlyOutcome = Extract; + +export interface ProjectCatalogOnlyOutcomeInput { + changed: boolean; + catalogRefresh: CatalogDisposition; +} + /** * Note what is NOT here: `desired-off`. Desired OFF is not a skip — it is a * `converged` with `direction: "removed"`. That is round 2 N1: the old shape let diff --git a/src/codex/management-convergence.ts b/src/codex/management-convergence.ts index 397926454..369881fdc 100644 --- a/src/codex/management-convergence.ts +++ b/src/codex/management-convergence.ts @@ -9,9 +9,11 @@ import type { OcxConfig } from "../types"; import type { CatalogDisposition, + CatalogOnlyOutcome, CodexHistoryState, CodexObservedState, ConvergeCodex, + ProjectCatalogOnlyOutcomeInput, } from "./convergence-types"; function notEvaluatedHistory(): CodexHistoryState { @@ -57,6 +59,21 @@ function catalogNotRequested(): CatalogDisposition { return { status: "skipped", reason: "not-requested", retryable: false }; } +/** Project catalog work into the shared no-change/not-evaluated outcome shape. */ +export function projectCatalogOnlyOutcome({ + changed, + catalogRefresh, +}: ProjectCatalogOnlyOutcomeInput): CatalogOnlyOutcome { + const history = notEvaluatedHistory(); + return { + kind: "catalog-only", + changed, + observed: notEvaluatedObserved(history), + catalogRefresh, + history, + }; +} + /** * Bind the management callback's exact config authority to a catalog-only * funnel. This module is intentionally not re-exported by a public Codex facade. @@ -72,14 +89,9 @@ export function createManagementConvergeCodex( // WP9 replaces this no-work projection and consumes this exact reference. void retainedConfig; - const history = notEvaluatedHistory(); - const observed = notEvaluatedObserved(history); - return { - kind: "catalog-only", + return projectCatalogOnlyOutcome({ changed: false, - observed, catalogRefresh: catalogNotRequested(), - history, - }; + }); }; } diff --git a/tests/codex-management-convergence.test.ts b/tests/codex-management-convergence.test.ts index dfeaf3040..d6e8811c1 100644 --- a/tests/codex-management-convergence.test.ts +++ b/tests/codex-management-convergence.test.ts @@ -1,27 +1,21 @@ import { expect, test } from "bun:test"; -import { createManagementConvergeCodex } from "../src/codex/management-convergence"; -import type { ConvergeRequest } from "../src/codex/convergence-types"; +import { createCatalogConvergeRequest } from "../src/codex/catalog-admission"; +import { + createManagementConvergeCodex, + projectCatalogOnlyOutcome, +} from "../src/codex/management-convergence"; +import type { CatalogDisposition } from "../src/codex/convergence-types"; import type { OcxConfig } from "../src/types"; function config(): OcxConfig { return { port: 10100, providers: {}, defaultProvider: "openai" }; } -function request(scope: ConvergeRequest["scope"]): ConvergeRequest { - return { - action: "converge", - scope, - reason: "management-mutation", - mode: "automatic", - deadlineMs: 1_000, - }; -} - test("returns an honest catalog-only no-change projection", async () => { const convergeCodex = createManagementConvergeCodex(config()); - const outcome = await convergeCodex(request("catalog")); + const outcome = await convergeCodex(createCatalogConvergeRequest({ deadlineMs: 1_000 })); expect(outcome).toEqual({ kind: "catalog-only", @@ -73,7 +67,126 @@ test("returns an honest catalog-only no-change projection", async () => { test("rejects a non-catalog request instead of widening management authority", async () => { const convergeCodex = createManagementConvergeCodex(config()); - await expect(convergeCodex(request("full"))).rejects.toThrow( + await expect(convergeCodex({ + action: "observe", + scope: "full", + reason: "cli", + mode: "explicit", + deadlineMs: 1_000, + })).rejects.toThrow( "Management Codex convergence accepts only catalog-scoped requests.", ); }); + +test("constructs the fixed catalog request and ignores caller attempts to choose direction", () => { + const malformedInput = { + action: "remove", + scope: "full", + reason: "cli", + mode: "explicit", + deadlineMs: 1_000, + } as unknown as Parameters[0]; + const request = createCatalogConvergeRequest(malformedInput); + + expect(request).toEqual({ + action: "converge", + scope: "catalog", + reason: "management-mutation", + mode: "automatic", + deadlineMs: 1_000, + }); +}); + +test("rejects malformed catalog request deadlines", () => { + for (const deadlineMs of [ + 0, + -1, + 1.5, + Number.NaN, + Number.POSITIVE_INFINITY, + Number.MAX_SAFE_INTEGER + 1, + ]) { + expect(() => createCatalogConvergeRequest({ deadlineMs })).toThrow( + "Catalog convergence deadlineMs must be a positive safe integer.", + ); + } +}); + +const booleans = [false, true] as const; +const catalogRefreshVariants: readonly CatalogDisposition[] = [ + ...booleans.flatMap(changed => booleans.map(degraded => ({ + status: "committed" as const, + changed, + degraded, + notices: ["provider-auth", "provider-network", "fallback"] as const, + }))), + ...(["not-requested", "catalog-unavailable", "busy", "stale", "refused"] as const) + .flatMap(reason => booleans.map(retryable => ({ + status: "skipped" as const, + reason, + retryable, + }))), + ...(["provider-auth", "provider-network", "disk"] as const).flatMap(reason => + (["gather", "commit"] as const).flatMap(phase => + booleans.flatMap(retryable => booleans.map(partialWrite => ({ + status: "failed" as const, + reason, + phase, + retryable, + partialWrite, + }))), + ), + ), +]; + +for (const changed of booleans) { + for (const [variantIndex, catalogRefresh] of catalogRefreshVariants.entries()) { + test(`projects changed=${changed} with catalog variant ${variantIndex} exactly`, () => { + expect(projectCatalogOnlyOutcome({ changed, catalogRefresh })).toEqual({ + kind: "catalog-only", + changed, + catalogRefresh, + history: { + status: "not-evaluated", + attempts: 0, + nextRetryAt: null, + txId: null, + pendingRows: null, + backupEntries: null, + }, + observed: { + aggregate: "not-evaluated", + isApplied: null, + desired: "unknown", + converged: null, + authority: { service: "unknown", externalProvider: null }, + surfaces: { + config: "not-evaluated", + profile: "not-evaluated", + catalog: "not-evaluated", + cache: "not-evaluated", + journal: "not-evaluated", + history: { + state: { + status: "not-evaluated", + attempts: 0, + nextRetryAt: null, + txId: null, + pendingRows: null, + backupEntries: null, + }, + database: "not-evaluated", + manifest: "not-evaluated", + rollouts: "not-evaluated", + }, + provenance: { + state: "not-evaluated", + nativeGeneration: null, + currentTxId: null, + }, + }, + }, + }); + }); + } +} From 72773717837fe7b531cba0a0b39500965991317f Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Tue, 4 Aug 2026 17:25:08 +0900 Subject: [PATCH 047/163] test(codex): six tests that passed under a broken substrate now fail MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Every test here was green while the property it named was breakable, which is the same failure that let 91 tests pass over a broken gajae config and 18 pass while every history update failed. Each is now pinned to a concrete mutation. The CAS tests disagreed on both halves of the expected version at once, so dropping either predicate left them green; generation-matches/txId-wrong and txId-matches/generation-wrong now prove each half alone. The direction test proved only the SQL CHECK, so the independent row validator is exercised directly against rows written with `ignore_check_constraints`, the state a foreign writer can leave behind. The opacity test proved SQLite exclusion but not unreachability, so the capability is now walked for a live Database handle — leaking the connection fails only the new test, which is what made the old one partial. Identity is proven by real child processes varying HOME, USERPROFILE, HOMEDRIVE/HOMEPATH, XDG_RUNTIME_DIR, TMPDIR, CODEX_HOME and OPENCODEX_HOME: on Bun 1.3.14 both `os.homedir()` and `os.userInfo().homedir` follow the environment, so a HOME-derived key passes the old assertion and fails this one. The new race file adds two real processes contending for initialization, two OPENCODEX_HOMEs sharing one CODEX_HOME, and typed busy/unsafe-path outcomes asserted by discriminant rather than by "it threw". Race repeated 10x clean. --- tests/codex-transition-state-race.test.ts | 314 ++++++++++++++++++++++ tests/codex-transition-state.test.ts | 140 ++++++++++ tests/codex-user-identity.test.ts | 92 ++++++- 3 files changed, 545 insertions(+), 1 deletion(-) create mode 100644 tests/codex-transition-state-race.test.ts diff --git a/tests/codex-transition-state-race.test.ts b/tests/codex-transition-state-race.test.ts new file mode 100644 index 000000000..0a8c431e1 --- /dev/null +++ b/tests/codex-transition-state-race.test.ts @@ -0,0 +1,314 @@ +import { expect, test } from "bun:test"; +import { + chmodSync, + existsSync, + mkdirSync, + mkdtempSync, + realpathSync, + rmSync, + writeFileSync, +} from "node:fs"; +import { join } from "node:path"; +import { tmpdir } from "node:os"; +import { pathToFileURL } from "node:url"; + +import { Database } from "bun:sqlite"; + +import { openCodexCoordinatorTransaction } from "../src/codex/transition-state"; +import { + resolveCodexCoordinatorDatabasePath, + resolveEffectiveUserIdentity, +} from "../src/codex/user-identity"; + +const CHILD_TIMEOUT_MS = 10_000; +const transitionStateModuleUrl = pathToFileURL( + join(import.meta.dir, "..", "src", "codex", "transition-state.ts"), +).href; +const userIdentityModuleUrl = pathToFileURL( + join(import.meta.dir, "..", "src", "codex", "user-identity.ts"), +).href; + +const transitionProbe = ` + import { + beginCodexTransition, + readCodexTransitionState, + } from ${JSON.stringify(transitionStateModuleUrl)}; + import { + resolveCodexCoordinatorDatabasePath, + resolveEffectiveUserIdentity, + } from ${JSON.stringify(userIdentityModuleUrl)}; + import { existsSync, realpathSync, writeFileSync } from "node:fs"; + + const payload = JSON.parse(process.env.OCX_TEST_PAYLOAD); + const canonicalCodexHome = realpathSync.native(payload.codexHome); + const databasePath = resolveCodexCoordinatorDatabasePath( + resolveEffectiveUserIdentity(), + canonicalCodexHome, + ); + const next = txId => ({ + txId, + direction: "apply", + authoritySnapshotId: \`authority-\${txId}\`, + nextRetryAt: "2026-08-04T12:00:00.000Z", + }); + const waitFor = async path => { + const deadline = Date.now() + ${CHILD_TIMEOUT_MS}; + while (!existsSync(path)) { + if (Date.now() >= deadline) throw new Error(\`timed out waiting for \${path}\`); + await Bun.sleep(5); + } + }; + + let result; + if (payload.action === "race") { + writeFileSync(payload.readyPath, "ready"); + await waitFor(payload.releasePath); + const first = beginCodexTransition( + { nativeGeneration: 0, currentTxId: null }, + next(payload.txId), + ); + writeFileSync(payload.outcomePath, JSON.stringify(first)); + await waitFor(payload.retryPath); + const final = first.kind === "unavailable" && first.reason === "busy" + ? beginCodexTransition( + { nativeGeneration: 0, currentTxId: null }, + next(payload.txId), + ) + : first; + result = { id: payload.id, databasePath, first, final }; + } else if (payload.action === "begin") { + result = { + databasePath, + outcome: beginCodexTransition(payload.expected, next(payload.txId)), + }; + } else { + result = { databasePath, outcome: readCodexTransitionState() }; + } + process.stdout.write(JSON.stringify(result)); +`; + +interface ProbeResult { + id?: string; + databasePath: string; + first?: TransitionOutcome; + final?: TransitionOutcome; + outcome?: TransitionOutcome; +} + +interface TransitionOutcome { + kind: string; + reason?: string; + state?: { nativeGeneration: number; currentTxId: string | null }; + current?: { nativeGeneration: number; currentTxId: string | null }; +} + +interface Sandbox { + root: string; + codexHome: string; + opencodexHomes: [string, string]; + coordinatorPath: string; +} + +function createSandbox(label: string): Sandbox { + const root = mkdtempSync(join(tmpdir(), `ocx-transition-race-${label}-`)); + const codexHome = join(root, "codex"); + const opencodexHomes: [string, string] = [join(root, "ocx-a"), join(root, "ocx-b")]; + mkdirSync(codexHome); + for (const path of opencodexHomes) mkdirSync(path); + const coordinatorPath = resolveCodexCoordinatorDatabasePath( + resolveEffectiveUserIdentity(), + realpathSync.native(codexHome), + ); + return { root, codexHome, opencodexHomes, coordinatorPath }; +} + +function cleanupSandbox(sandbox: Sandbox): void { + for (const suffix of ["", "-journal", "-wal", "-shm"]) { + rmSync(`${sandbox.coordinatorPath}${suffix}`, { recursive: true, force: true }); + } + rmSync(sandbox.root, { recursive: true, force: true }); +} + +function spawnProbe(sandbox: Sandbox, opencodexHome: string, payload: Record) { + return Bun.spawn([process.execPath, "--eval", transitionProbe], { + env: { + ...process.env, + CODEX_HOME: sandbox.codexHome, + OPENCODEX_HOME: opencodexHome, + OCX_TEST_PAYLOAD: JSON.stringify({ ...payload, codexHome: sandbox.codexHome }), + }, + stdin: "ignore", + stdout: "pipe", + stderr: "pipe", + }); +} + +async function collectProbe(child: ReturnType): Promise { + const timeout = setTimeout(() => child.kill(), CHILD_TIMEOUT_MS); + try { + const [exitCode, stdout, stderr] = await Promise.all([ + child.exited, + new Response(child.stdout).text(), + new Response(child.stderr).text(), + ]); + expect(exitCode, stderr).toBe(0); + expect(stdout.trim().split("\n"), stderr).toHaveLength(1); + return JSON.parse(stdout) as ProbeResult; + } finally { + clearTimeout(timeout); + } +} + +async function waitForFiles(paths: readonly string[]): Promise { + const deadline = Date.now() + CHILD_TIMEOUT_MS; + while (!paths.every(existsSync)) { + if (Date.now() >= deadline) throw new Error(`timed out waiting for ${paths.join(", ")}`); + await Bun.sleep(5); + } +} + +test("two real processes racing first use publish exactly one initial transition", async () => { + const sandbox = createSandbox("initialization"); + const barrier = join(sandbox.root, "barrier"); + mkdirSync(barrier); + const releasePath = join(barrier, "release"); + const retryPath = join(barrier, "retry"); + const children = ["a", "b"].map((id, index) => spawnProbe( + sandbox, + sandbox.opencodexHomes[index]!, + { + action: "race", + id, + txId: `tx-${id}`, + readyPath: join(barrier, `${id}.ready`), + outcomePath: join(barrier, `${id}.outcome`), + releasePath, + retryPath, + }, + )); + + try { + await waitForFiles([join(barrier, "a.ready"), join(barrier, "b.ready")]); + writeFileSync(releasePath, "go"); + await waitForFiles([join(barrier, "a.outcome"), join(barrier, "b.outcome")]); + writeFileSync(retryPath, "retry-busy-loser"); + const results = await Promise.all(children.map(collectProbe)); + + const firstKinds = results.map(result => result.first?.kind); + expect(firstKinds.filter(kind => kind === "updated")).toHaveLength(1); + for (const result of results) { + expect( + result.first?.kind === "updated" + || result.first?.kind === "conflict" + || (result.first?.kind === "unavailable" && result.first.reason === "busy"), + ).toBe(true); + } + + const finalKinds = results.map(result => result.final?.kind).sort(); + expect(finalKinds).toEqual(["conflict", "updated"]); + const winner = results.find(result => result.final?.kind === "updated")!; + const loser = results.find(result => result.final?.kind === "conflict")!; + expect(loser.final?.current?.currentTxId).toBe(winner.final?.state?.currentTxId); + expect(results.map(result => result.databasePath)).toEqual([ + sandbox.coordinatorPath, + sandbox.coordinatorPath, + ]); + + const database = new Database(sandbox.coordinatorPath, { readonly: true }); + try { + expect(database.query<{ count: number }, []>( + "SELECT count(*) AS count FROM codex_transition_state WHERE singleton = 1", + ).get()?.count).toBe(1); + expect(database.query<{ native_generation: number; current_tx_id: string }, []>( + "SELECT native_generation, current_tx_id FROM codex_transition_state WHERE singleton = 1", + ).get()).toEqual({ + native_generation: 1, + current_tx_id: winner.final?.state?.currentTxId, + }); + } finally { + database.close(); + } + } finally { + for (const child of children) child.kill(); + cleanupSandbox(sandbox); + } +}, { timeout: 30_000 }); + +test("different OPENCODEX_HOME claimants advance the row under one CODEX_HOME", async () => { + const sandbox = createSandbox("shared-codex-home"); + try { + const first = await collectProbe(spawnProbe(sandbox, sandbox.opencodexHomes[0], { + action: "begin", + expected: { nativeGeneration: 0, currentTxId: null }, + txId: "tx-home-a", + })); + expect(first.outcome?.kind).toBe("updated"); + + const integrations = join(sandbox.opencodexHomes[1], "integrations"); + mkdirSync(integrations); + writeFileSync(join(integrations, "codex.json"), JSON.stringify({ + version: 1, + nativeGeneration: 91, + currentTxId: "opencodex-home-local-claimant", + history: { status: "pending", txId: "opencodex-home-local-claimant" }, + })); + + const second = await collectProbe(spawnProbe(sandbox, sandbox.opencodexHomes[1], { + action: "begin", + expected: { nativeGeneration: 1, currentTxId: "tx-home-a" }, + txId: "tx-home-b", + })); + expect(second.databasePath).toBe(first.databasePath); + expect(second.outcome).toMatchObject({ + kind: "updated", + state: { nativeGeneration: 2, currentTxId: "tx-home-b" }, + }); + + const observed = await collectProbe(spawnProbe(sandbox, sandbox.opencodexHomes[0], { + action: "read", + })); + expect(observed.databasePath).toBe(first.databasePath); + expect(observed.outcome).toMatchObject({ + kind: "ready", + state: { nativeGeneration: 2, currentTxId: "tx-home-b" }, + }); + } finally { + cleanupSandbox(sandbox); + } +}, { timeout: 30_000 }); + +test("a locked coordinator returns the exact typed busy outcome", async () => { + const sandbox = createSandbox("busy"); + let controller: ReturnType | undefined; + try { + const initialized = await collectProbe(spawnProbe(sandbox, sandbox.opencodexHomes[0], { + action: "read", + })); + expect(initialized.outcome?.kind).toBe("ready"); + + controller = openCodexCoordinatorTransaction(sandbox.coordinatorPath); + const blocked = await collectProbe(spawnProbe(sandbox, sandbox.opencodexHomes[1], { + action: "read", + })); + expect(blocked.outcome).toEqual({ kind: "unavailable", reason: "busy" }); + } finally { + controller?.close(); + cleanupSandbox(sandbox); + } +}, { timeout: 20_000 }); + +test("an unsafe coordinator path returns the exact typed unsafe-path outcome", async () => { + const sandbox = createSandbox("unsafe-path"); + try { + mkdirSync(sandbox.coordinatorPath); + const refused = await collectProbe(spawnProbe(sandbox, sandbox.opencodexHomes[0], { + action: "read", + })); + expect(refused.outcome).toEqual({ kind: "unavailable", reason: "unsafe-path" }); + } finally { + if (process.platform !== "win32" && existsSync(sandbox.coordinatorPath)) { + chmodSync(sandbox.coordinatorPath, 0o700); + } + cleanupSandbox(sandbox); + } +}, { timeout: 20_000 }); diff --git a/tests/codex-transition-state.test.ts b/tests/codex-transition-state.test.ts index b4143b54a..00c2f2f2c 100644 --- a/tests/codex-transition-state.test.ts +++ b/tests/codex-transition-state.test.ts @@ -159,6 +159,60 @@ test("a zero-row conditional update reports conflict and preserves the winner", }); }); +/** + * The C-phase review found the conflict tests PARTIAL: every stale caller they + * exercised disagreed on BOTH halves of the expected pair, so dropping either + * `native_generation = ?` or `current_tx_id IS ?` from the CAS predicate left + * them green. A CAS on a two-part version has to be proven one part at a time. + */ +test("a native CAS with a matching generation but the wrong txId still conflicts", () => { + expect(beginCodexTransition( + { nativeGeneration: 0, currentTxId: null }, + transition("tx-a"), + ).kind).toBe("updated"); + expect(beginCodexTransition( + { nativeGeneration: 1, currentTxId: "tx-a" }, + transition("tx-b"), + ).kind).toBe("updated"); + + // Generation 2 is current, so only the txId half disagrees. Removing the + // `current_tx_id IS ?` predicate makes this succeed. + const wrongTxId = beginCodexTransition( + { nativeGeneration: 2, currentTxId: "tx-a" }, + transition("tx-forged"), + ); + expect(wrongTxId.kind).toBe("conflict"); + + expect(readCodexTransitionState()).toMatchObject({ + kind: "ready", + state: { nativeGeneration: 2, currentTxId: "tx-b" }, + }); +}); + +test("a native CAS with a matching txId but the wrong generation still conflicts", () => { + expect(beginCodexTransition( + { nativeGeneration: 0, currentTxId: null }, + transition("tx-a"), + ).kind).toBe("updated"); + expect(beginCodexTransition( + { nativeGeneration: 1, currentTxId: "tx-a" }, + transition("tx-b"), + ).kind).toBe("updated"); + + // `tx-b` really is the current txId, so only the generation half disagrees. + // Removing the `native_generation = ?` predicate makes this succeed. + const wrongGeneration = beginCodexTransition( + { nativeGeneration: 1, currentTxId: "tx-b" }, + transition("tx-forged"), + ); + expect(wrongGeneration.kind).toBe("conflict"); + + expect(readCodexTransitionState()).toMatchObject({ + kind: "ready", + state: { nativeGeneration: 2, currentTxId: "tx-b" }, + }); +}); + test("a positive generation cannot carry a null direction", () => { expect(beginCodexTransition( { nativeGeneration: 0, currentTxId: null }, @@ -178,6 +232,58 @@ test("a positive generation cannot carry a null direction", () => { } }); +/** + * The test above proves the SQL CHECK constraint and nothing else. The row + * validator in `rowToState` is a SECOND, independent gate that exists because a + * database written by another build, an older schema, or a hand-edit can hold a + * row the current CHECKs would have rejected at write time. Dropping the + * validator left the suite green, so it is proven here directly: the row is + * corrupted with the constraints switched off, and the reader must still refuse. + */ +test("the row validator refuses a malformed row the CHECK constraints never saw", () => { + expect(beginCodexTransition( + { nativeGeneration: 0, currentTxId: null }, + transition("tx-validator"), + ).kind).toBe("updated"); + + const database = new Database(coordinatorPath); + try { + // `ignore_check_constraints` lets a write land that the schema forbids, + // which is exactly the state a foreign writer can leave behind. + database.exec("PRAGMA ignore_check_constraints = ON"); + database.run( + "UPDATE codex_transition_state SET history_direction = NULL WHERE singleton = 1", + ); + expect(database.query<{ history_direction: string | null }, []>( + "SELECT history_direction FROM codex_transition_state WHERE singleton = 1", + ).get()?.history_direction).toBeNull(); + } finally { + database.close(); + } + + // The CHECK did not stop it; the validator must. + expect(readCodexTransitionState()).toEqual({ kind: "unavailable", reason: "database" }); +}); + +test("the row validator refuses a positive generation with a blank txId", () => { + expect(beginCodexTransition( + { nativeGeneration: 0, currentTxId: null }, + transition("tx-blank"), + ).kind).toBe("updated"); + + const database = new Database(coordinatorPath); + try { + database.exec("PRAGMA ignore_check_constraints = ON"); + database.run( + "UPDATE codex_transition_state SET current_tx_id = ' ', history_tx_id = ' ' WHERE singleton = 1", + ); + } finally { + database.close(); + } + + expect(readCodexTransitionState()).toEqual({ kind: "unavailable", reason: "database" }); +}); + /** * A capability backed by a nominal transaction is not opaque if its caller can * simply open another connection. The C-phase review found the old test only @@ -196,6 +302,40 @@ test("the opaque coordinator capability cannot reach a second connection", () => } }); +/** + * SQLite exclusion is only half of "opaque". The other half is that the + * capability object itself must not hand its caller a usable handle on the open + * connection: a caller who can reach the `Database` can write the native pair + * behind the CAS, on the very transaction that is supposed to serialize it. + * The previous test passed while the connection was reachable. + */ +test("the opaque capability never exposes a reachable database handle", () => { + expect(readCodexTransitionState().kind).toBe("ready"); + const controller = openCodexCoordinatorTransaction(coordinatorPath); + try { + const reachable = new Set(); + const walk = (value: unknown, depth: number): void => { + if (depth > 4 || value === null || reachable.has(value)) return; + const kind = typeof value; + if (kind !== "object" && kind !== "function") return; + reachable.add(value); + for (const key of Reflect.ownKeys(value as object)) { + const descriptor = Reflect.getOwnPropertyDescriptor(value as object, key); + // Only follow plain values: invoking a getter is not "exposure". + if (descriptor && "value" in descriptor) walk(descriptor.value, depth + 1); + } + walk(Reflect.getPrototypeOf(value as object), depth + 1); + }; + walk(controller.capability, 0); + + for (const value of reachable) { + expect(value).not.toBeInstanceOf(Database); + } + } finally { + controller.close(); + } +}); + test("the opaque coordinator capability is one-shot", () => { const controller = openCodexCoordinatorTransaction(coordinatorPath); try { diff --git a/tests/codex-user-identity.test.ts b/tests/codex-user-identity.test.ts index f40ab8d37..e63e09edc 100644 --- a/tests/codex-user-identity.test.ts +++ b/tests/codex-user-identity.test.ts @@ -1,7 +1,8 @@ import { afterEach, beforeEach, expect, test } from "bun:test"; -import { mkdtempSync, realpathSync, rmSync } from "node:fs"; +import { mkdirSync, mkdtempSync, realpathSync, rmSync } from "node:fs"; import { join, parse } from "node:path"; import { tmpdir } from "node:os"; +import { pathToFileURL } from "node:url"; import { resolveCodexCoordinatorDatabasePath, @@ -11,6 +12,50 @@ import { let codexHome = ""; let previousHome: string | undefined; +const CHILD_TIMEOUT_MS = 10_000; +const userIdentityModuleUrl = pathToFileURL( + join(import.meta.dir, "..", "src", "codex", "user-identity.ts"), +).href; +const identityProbe = ` + import { + resolveCodexCoordinatorDatabasePath, + resolveEffectiveUserIdentity, + } from ${JSON.stringify(userIdentityModuleUrl)}; + import { realpathSync } from "node:fs"; + + const canonicalCodexHome = realpathSync.native(process.env.OCX_TEST_CANONICAL_CODEX_HOME); + const identity = resolveEffectiveUserIdentity(); + const databasePath = resolveCodexCoordinatorDatabasePath(identity, canonicalCodexHome); + process.stdout.write(JSON.stringify({ identity, databasePath })); +`; + +interface IdentityProbeResult { + identity: ReturnType; + databasePath: string; +} + +async function runIdentityProbe(env: Record): Promise { + const child = Bun.spawn([process.execPath, "--eval", identityProbe], { + env: { ...process.env, ...env }, + stdin: "ignore", + stdout: "pipe", + stderr: "pipe", + }); + const timeout = setTimeout(() => child.kill(), CHILD_TIMEOUT_MS); + try { + const [exitCode, stdout, stderr] = await Promise.all([ + child.exited, + new Response(child.stdout).text(), + new Response(child.stderr).text(), + ]); + expect(exitCode, stderr).toBe(0); + expect(stdout.trim().split("\n"), stderr).toHaveLength(1); + return JSON.parse(stdout) as IdentityProbeResult; + } finally { + clearTimeout(timeout); + } +} + beforeEach(() => { previousHome = process.env.HOME; codexHome = mkdtempSync(join(tmpdir(), "ocx-user-identity-codex-home-")); @@ -52,3 +97,48 @@ test("the coordinator resolver returns the final database path", () => { canonicalHome, )); }); + +test("real processes resolve one identity and coordinator path across every home/runtime environment", async () => { + const canonicalHome = realpathSync.native(codexHome); + const environmentRoots = ["a", "b"].map(label => { + const root = mkdtempSync(join(tmpdir(), `ocx-user-identity-env-${label}-`)); + const paths = { + home: join(root, "home"), + userProfile: join(root, "profile"), + homeDrive: join(root, "drive"), + homePath: join(root, "path"), + xdgRuntime: join(root, "runtime"), + temp: join(root, "temp"), + codexHome: join(root, "ambient-codex"), + opencodexHome: join(root, "ambient-opencodex"), + }; + for (const path of Object.values(paths)) mkdirSync(path, { recursive: true }); + return { root, paths }; + }); + + try { + const probes = await Promise.all(environmentRoots.map(({ paths }) => runIdentityProbe({ + HOME: paths.home, + USERPROFILE: paths.userProfile, + HOMEDRIVE: paths.homeDrive, + HOMEPATH: paths.homePath, + XDG_RUNTIME_DIR: paths.xdgRuntime, + TMPDIR: paths.temp, + TEMP: paths.temp, + TMP: paths.temp, + LOCALAPPDATA: paths.temp, + CODEX_HOME: paths.codexHome, + OPENCODEX_HOME: paths.opencodexHome, + OCX_TEST_CANONICAL_CODEX_HOME: canonicalHome, + }))); + + expect(probes[1]?.identity).toEqual(probes[0]?.identity); + expect(probes[1]?.databasePath).toBe(probes[0]?.databasePath); + expect(probes[0]?.databasePath).toBe(resolveCodexCoordinatorDatabasePath( + resolveEffectiveUserIdentity(), + canonicalHome, + )); + } finally { + for (const { root } of environmentRoots) rmSync(root, { recursive: true, force: true }); + } +}, { timeout: 20_000 }); From 53e5152848b14fd52d31dad5e24057c2b466fb9f Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Tue, 4 Aug 2026 18:16:24 +0900 Subject: [PATCH 048/163] fix(codex): the classifier read one catalog filename and demanded a slash MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Round two of the same defect. A verifier ran the real production catalog writer against temp CODEX_HOMEs and got `clean` twice over bytes opencodex had just written itself. `config.toml` may point `model_catalog_json` anywhere, including a nested path under CODEX_HOME, and `syncCatalogModels()` writes genuine routed rows there — but the classifier only ever looked at the default `opencodex-catalog.json`, and only scanned fixed root filenames for `.ocx...tmp` artifacts. The active target is now resolved from config by a fail-closed resolver rather than `readCodexCatalogPath()`, which catches and falls back to the default and so cannot be trusted here. Both the configured and default targets are inspected, and each target's parent — logical and symlink-resolved — is scanned for its own partial writes. A configured path that is absent, unreadable, non-regular, malformed, duplicated, or shifting underfoot is `indeterminate`. The second miss was narrower and worse: `isOcxRoutedCatalogEntry` required the authorship description AND a slash in the slug, while production deliberately emits bare combo aliases (`sync.ts:279`). A real `fast-chat` row carrying `Routed via opencodex → combo (combo).` classified clean. Authorship alone is now sufficient for residue; a slash without authorship stays `indeterminate`. Both of the verifier's reproductions now return residue and refuse initialization as `legacy-ambiguous`, and each fix has its own revert-to-red. --- src/codex/native-residue.ts | 185 +++++++++++++++++++++++------ tests/codex-native-residue.test.ts | 155 +++++++++++++++++++++++- 2 files changed, 300 insertions(+), 40 deletions(-) diff --git a/src/codex/native-residue.ts b/src/codex/native-residue.ts index cc2573f12..6216cb38c 100644 --- a/src/codex/native-residue.ts +++ b/src/codex/native-residue.ts @@ -1,7 +1,7 @@ import { createHash } from "node:crypto"; import { lstatSync, readFileSync, readdirSync, realpathSync, statSync } from "node:fs"; import type { Stats } from "node:fs"; -import { basename, join, resolve } from "node:path"; +import { basename, dirname, join, resolve } from "node:path"; import { Database } from "bun:sqlite"; @@ -46,6 +46,16 @@ type PathResult = | { kind: "path"; path: string; stat: Stats } | { kind: "indeterminate"; reason: string }; +type CatalogTarget = { + path: string; + configured: boolean; +}; + +type ConfigObservation = { + classification: NativeRoutedResidueResult; + catalogTargets: CatalogTarget[]; +}; + const CONFIG_FILE_NAME = basename(CODEX_CONFIG_PATH); const PROFILE_FILE_NAME = basename(CODEX_PROFILE_PATH); const CATALOG_FILE_NAME = basename(DEFAULT_CATALOG_PATH); @@ -54,14 +64,6 @@ const JOURNAL_FILE_NAME = "opencodex-journal.json"; const HISTORY_DATABASE_FILE_NAME = "state_5.sqlite"; const ROUTED_CATALOG_DESCRIPTION_PREFIX = "Routed via opencodex → "; -const ATOMIC_WRITE_TARGETS = new Set([ - CONFIG_FILE_NAME, - PROFILE_FILE_NAME, - CATALOG_FILE_NAME, - MODELS_CACHE_FILE_NAME, - JOURNAL_FILE_NAME, -]); - function errorCode(error: unknown): string | undefined { return (error as NodeJS.ErrnoException | undefined)?.code; } @@ -154,16 +156,90 @@ function classifyToml( return { kind: "clean" }; } -function classifyConfig(path: string): NativeRoutedResidueResult { - return classifyToml("config", path, content => { - if (hasInjectedCodexRouting(content)) return "residue"; - const hasMarker = content.includes(OCX_SECTION_MARKER); - const provider = rootTomlString(content, "model_provider"); - const providerBaseUrl = providerTableString(content, "opencodex", "base_url"); - return hasMarker || provider === "opencodex" || providerBaseUrl !== null - ? "indeterminate" - : "clean"; - }); +function catalogPathKey(path: string): string { + const normalized = resolve(path); + return process.platform === "win32" ? normalized.toLowerCase() : normalized; +} + +function catalogTargets( + codexHome: string, + configuredPath?: string, +): CatalogTarget[] { + const targets = new Map(); + const add = (path: string, configured: boolean) => { + const key = catalogPathKey(path); + const existing = targets.get(key); + targets.set(key, { path: resolve(path), configured: configured || existing?.configured === true }); + }; + if (configuredPath !== undefined) add(resolve(codexHome, configuredPath), true); + add(join(codexHome, CATALOG_FILE_NAME), false); + return [...targets.values()]; +} + +function inspectConfig(codexHome: string, path: string): ConfigObservation { + const read = readRegularFile(path); + if (read.kind === "absent") { + return { classification: { kind: "clean" }, catalogTargets: catalogTargets(codexHome) }; + } + if (read.kind === "indeterminate") { + return { + classification: indeterminate("config", path, read.reason), + catalogTargets: catalogTargets(codexHome), + }; + } + + let parsed: unknown; + try { + parsed = Bun.TOML.parse(read.content); + } catch (error) { + return { + classification: indeterminate("config", read.path, `malformed TOML: ${errorReason(error)}`), + catalogTargets: catalogTargets(codexHome), + }; + } + if (!parsed || typeof parsed !== "object" || Array.isArray(parsed)) { + return { + classification: indeterminate("config", read.path, "TOML root is not a table"), + catalogTargets: catalogTargets(codexHome), + }; + } + + const document = parsed as Record; + let targets: CatalogTarget[]; + if (!Object.hasOwn(document, "model_catalog_json")) { + targets = catalogTargets(codexHome); + } else if (typeof document.model_catalog_json !== "string" || !document.model_catalog_json.trim()) { + return { + classification: indeterminate("config", read.path, "model_catalog_json must be one non-empty string"), + catalogTargets: catalogTargets(codexHome), + }; + } else { + try { + targets = catalogTargets(codexHome, document.model_catalog_json); + } catch (error) { + return { + classification: indeterminate("config", read.path, `model_catalog_json cannot be resolved: ${errorReason(error)}`), + catalogTargets: catalogTargets(codexHome), + }; + } + } + + let classification: NativeRoutedResidueResult = { kind: "clean" }; + if (hasInjectedCodexRouting(read.content)) { + classification = { kind: "residue", surface: "config", path: read.path }; + } else { + const hasMarker = read.content.includes(OCX_SECTION_MARKER); + const provider = rootTomlString(read.content, "model_provider"); + const providerBaseUrl = providerTableString(read.content, "opencodex", "base_url"); + if (hasMarker || provider === "opencodex" || providerBaseUrl !== null) { + classification = indeterminate( + "config", + read.path, + "OpenCodex-shaped TOML does not match a complete routed grammar", + ); + } + } + return { classification, catalogTargets: targets }; } function classifyProfile(path: string): NativeRoutedResidueResult { @@ -178,18 +254,21 @@ function classifyProfile(path: string): NativeRoutedResidueResult { } function isOcxRoutedCatalogEntry(entry: Record): boolean { - return typeof entry.slug === "string" - && entry.slug.includes("/") - && typeof entry.description === "string" + return typeof entry.description === "string" && entry.description.startsWith(ROUTED_CATALOG_DESCRIPTION_PREFIX); } function classifyCatalogLike( surface: "catalog" | "models-cache", path: string, + configured = false, ): NativeRoutedResidueResult { const read = readRegularFile(path); - if (read.kind === "absent") return { kind: "clean" }; + if (read.kind === "absent") { + return configured + ? indeterminate(surface, path, "configured catalog target is absent") + : { kind: "clean" }; + } if (read.kind === "indeterminate") return indeterminate(surface, path, read.reason); const catalog = parseCatalogJson(read.content); if (!catalog) return indeterminate(surface, path, "malformed catalog JSON"); @@ -228,17 +307,33 @@ function classifyJournal(path: string): NativeRoutedResidueResult { : indeterminate("journal", read.path, "journal JSON has an unknown or partial shape"); } -function classifyPartialWrites(codexHome: string): NativeRoutedResidueResult { - let names: string[]; - try { - names = readdirSync(codexHome); - } catch (error) { - return indeterminate("partial-write", codexHome, errorReason(error)); +function classifyPartialWrites(targetPaths: string[]): NativeRoutedResidueResult { + const targetsByParent = new Map }>(); + const addTarget = (path: string) => { + const parent = dirname(path); + const key = catalogPathKey(parent); + const observed = targetsByParent.get(key) ?? { path: parent, names: new Set() }; + observed.names.add(basename(path)); + targetsByParent.set(key, observed); + }; + for (const path of targetPaths) { + addTarget(path); + const resolved = resolveRegularFile(path); + if (resolved.kind === "path") addTarget(resolved.path); } - for (const name of names) { - const match = /^(.*)\.ocx\.\d+\.\d+\.tmp$/.exec(name); - if (match?.[1] && ATOMIC_WRITE_TARGETS.has(match[1])) { - return indeterminate("partial-write", join(codexHome, name), "OpenCodex atomic-write artifact is still present"); + + for (const target of targetsByParent.values()) { + let names: string[]; + try { + names = readdirSync(target.path); + } catch (error) { + return indeterminate("partial-write", target.path, errorReason(error)); + } + for (const name of names) { + const match = /^(.*)\.ocx\.\d+\.\d+\.tmp$/.exec(name); + if (match?.[1] && target.names.has(match[1])) { + return indeterminate("partial-write", join(target.path, name), "OpenCodex atomic-write artifact is still present"); + } } } return { kind: "clean" }; @@ -331,13 +426,25 @@ export function classifyNativeRoutedResidue(): NativeRoutedResidueResult { } const stateDatabasePath = join(codexHome, HISTORY_DATABASE_FILE_NAME); + const configPath = join(codexHome, CONFIG_FILE_NAME); + const profilePath = join(codexHome, PROFILE_FILE_NAME); + const modelsCachePath = join(codexHome, MODELS_CACHE_FILE_NAME); + const journalPath = join(codexHome, JOURNAL_FILE_NAME); + const config = inspectConfig(codexHome, configPath); + const atomicWriteTargets = [ + configPath, + profilePath, + modelsCachePath, + journalPath, + ...config.catalogTargets.map(target => target.path), + ]; const classifiers = [ - () => classifyPartialWrites(codexHome), - () => classifyConfig(join(codexHome, CONFIG_FILE_NAME)), - () => classifyProfile(join(codexHome, PROFILE_FILE_NAME)), - () => classifyCatalogLike("catalog", join(codexHome, CATALOG_FILE_NAME)), - () => classifyCatalogLike("models-cache", join(codexHome, MODELS_CACHE_FILE_NAME)), - () => classifyJournal(join(codexHome, JOURNAL_FILE_NAME)), + () => classifyPartialWrites(atomicWriteTargets), + () => config.classification, + () => classifyProfile(profilePath), + ...config.catalogTargets.map(target => () => classifyCatalogLike("catalog", target.path, target.configured)), + () => classifyCatalogLike("models-cache", modelsCachePath), + () => classifyJournal(journalPath), () => classifyHistoryDatabase(stateDatabasePath), () => classifyHistoryBackup(historyBackupPath(stateDatabasePath), stateDatabasePath), ]; diff --git a/tests/codex-native-residue.test.ts b/tests/codex-native-residue.test.ts index d0d4fdce5..8bb964095 100644 --- a/tests/codex-native-residue.test.ts +++ b/tests/codex-native-residue.test.ts @@ -13,7 +13,7 @@ import { join, resolve } from "node:path"; import { Database } from "bun:sqlite"; -import { buildCatalogEntries } from "../src/codex/catalog"; +import { buildCatalogEntries, syncCatalogModels } from "../src/codex/catalog"; import { buildProfileFile } from "../src/codex/inject"; import { classifyNativeRoutedResidue } from "../src/codex/native-residue"; import { readCodexTransitionState } from "../src/codex/transition-state"; @@ -21,6 +21,7 @@ import { resolveCodexCoordinatorDatabasePath, resolveEffectiveUserIdentity, } from "../src/codex/user-identity"; +import type { OcxConfig } from "../src/types"; let codexHome = ""; let opencodexHome = ""; @@ -57,6 +58,10 @@ function pathInCodexHome(name: string): string { return join(codexHome, name); } +function canonicalPathInCodexHome(name: string): string { + return join(realpathSync.native(codexHome), name); +} + function routedCatalog(): string { const models = buildCatalogEntries( null, @@ -180,6 +185,154 @@ test("an OpenCodex atomic-write artifact is indeterminate", () => { }); }); +test("a routed catalog at the configured nested path refuses coordinator initialization", async () => { + const catalogPath = canonicalPathInCodexHome("nested/custom-catalog.json"); + mkdirSync(pathInCodexHome("nested")); + writeFileSync(pathInCodexHome("config.toml"), 'model_catalog_json = "nested/custom-catalog.json"\n'); + writeFileSync(catalogPath, JSON.stringify({ models: [] })); + const config: OcxConfig = { + port: 10100, + defaultProvider: "fixture", + providers: { + fixture: { + adapter: "openai-chat", + baseUrl: "https://fixture.invalid/v1", + liveModels: false, + models: ["fixture-model"], + }, + }, + }; + + const sync = await syncCatalogModels(config); + + expect(sync).toMatchObject({ path: catalogPath, catalogWritten: true }); + expect(classifyNativeRoutedResidue()).toMatchObject({ + kind: "residue", + surface: "catalog", + path: catalogPath, + }); + expect(readCodexTransitionState()).toEqual({ + kind: "legacy-ambiguous", + message: "A missing coordinator row cannot be initialized while native Codex routing residue exists.", + }); +}); + +test("an atomic-write artifact beside the configured catalog is indeterminate", () => { + const catalogPath = canonicalPathInCodexHome("nested/custom-catalog.json"); + const artifactPath = `${catalogPath}.ocx.42.7.tmp`; + mkdirSync(pathInCodexHome("nested")); + writeFileSync(pathInCodexHome("config.toml"), 'model_catalog_json = "nested/custom-catalog.json"\n'); + writeFileSync(catalogPath, JSON.stringify({ models: [] })); + writeFileSync(artifactPath, "partial"); + + expect(classifyNativeRoutedResidue()).toMatchObject({ + kind: "indeterminate", + surface: "partial-write", + path: artifactPath, + }); +}); + +test("a configured catalog target that is not a readable regular file is indeterminate", () => { + const catalogPath = canonicalPathInCodexHome("nested/custom-catalog.json"); + mkdirSync(catalogPath, { recursive: true }); + writeFileSync(pathInCodexHome("config.toml"), 'model_catalog_json = "nested/custom-catalog.json"\n'); + + expect(classifyNativeRoutedResidue()).toMatchObject({ + kind: "indeterminate", + surface: "catalog", + path: catalogPath, + }); +}); + +test("an absent configured catalog target is indeterminate", () => { + const catalogPath = canonicalPathInCodexHome("nested/missing-catalog.json"); + mkdirSync(pathInCodexHome("nested")); + writeFileSync(pathInCodexHome("config.toml"), 'model_catalog_json = "nested/missing-catalog.json"\n'); + + expect(classifyNativeRoutedResidue()).toMatchObject({ + kind: "indeterminate", + surface: "catalog", + path: catalogPath, + }); +}); + +test("the default catalog is still inspected when a custom catalog is configured", () => { + const defaultCatalogPath = canonicalPathInCodexHome("opencodex-catalog.json"); + mkdirSync(pathInCodexHome("nested")); + writeFileSync(pathInCodexHome("config.toml"), 'model_catalog_json = "nested/custom-catalog.json"\n'); + writeFileSync(pathInCodexHome("nested/custom-catalog.json"), JSON.stringify({ models: [] })); + writeFileSync(defaultCatalogPath, routedCatalog()); + + expect(classifyNativeRoutedResidue()).toMatchObject({ + kind: "residue", + surface: "catalog", + path: defaultCatalogPath, + }); +}); + +test("a non-string configured catalog path is indeterminate", () => { + writeFileSync(pathInCodexHome("config.toml"), "model_catalog_json = 42\n"); + + expect(classifyNativeRoutedResidue()).toMatchObject({ + kind: "indeterminate", + surface: "config", + path: canonicalPathInCodexHome("config.toml"), + }); +}); + +test("duplicate configured catalog paths are indeterminate", () => { + writeFileSync(pathInCodexHome("config.toml"), [ + 'model_catalog_json = "first.json"', + 'model_catalog_json = "second.json"', + "", + ].join("\n")); + + expect(classifyNativeRoutedResidue()).toMatchObject({ + kind: "indeterminate", + surface: "config", + path: canonicalPathInCodexHome("config.toml"), + }); +}); + +test("a bare routed combo alias in the default catalog refuses coordinator initialization", () => { + const models = buildCatalogEntries( + null, + [], + [{ provider: "combo", id: "quick", alias: "fast-chat", owned_by: "combo" }], + undefined, + false, + "default", + new Set(["fast-chat"]), + ); + const combo = models.find(model => model.slug === "fast-chat"); + expect(combo).toMatchObject({ + slug: "fast-chat", + description: "Routed via opencodex → combo (combo).", + owned_by: "combo", + }); + writeFileSync(pathInCodexHome("opencodex-catalog.json"), JSON.stringify({ models: [combo] })); + + expect(classifyNativeRoutedResidue()).toMatchObject({ + kind: "residue", + surface: "catalog", + }); + expect(readCodexTransitionState()).toEqual({ + kind: "legacy-ambiguous", + message: "A missing coordinator row cannot be initialized while native Codex routing residue exists.", + }); +}); + +test("a slash-bearing catalog slug without OpenCodex authorship is indeterminate", () => { + writeFileSync(pathInCodexHome("opencodex-catalog.json"), JSON.stringify({ + models: [{ slug: "user/model", description: "User-authored catalog row" }], + })); + + expect(classifyNativeRoutedResidue()).toMatchObject({ + kind: "indeterminate", + surface: "catalog", + }); +}); + const indeterminateFixtures: Array<{ name: string; surface: string; From ed08c079c2addfc6ec907f14a34dfb1d37962982 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Tue, 4 Aug 2026 18:16:35 +0900 Subject: [PATCH 049/163] test(codex): five more mutations the strengthened tests still let through MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The tests written to close the first round of partial coverage were themselves partial, which is the honest lesson: a test that kills the mutation its author had in mind proves only that the author had one in mind. The opacity walker followed data properties and skipped accessors, so a `get database()` on the capability passed it. It now asserts the exact permitted own-key and descriptor set, brand symbol included, and rejects every accessor. The CAS pair still had a hole at a null expectation, so a predicate that treats `currentTxId: null` as a wildcard survived; a caller expecting a positive generation with a null txId must now conflict. Direction validation was only proven against `null`, so `"sideways"` was accepted, and blank-txId validation was only proven against ASCII spaces, so a tab-only identifier passed — both are now covered, the latter across tab, newline, vertical tab, form feed, NBSP and em space. The identity tests varied every home variable but never the identity itself, so `Number(process.env.UID ?? getuid())` passed all of them while splitting one OS user's coordinator namespace. Real child processes now vary UID, EUID, USER and LOGNAME, and the Windows account variables, and require the OS-derived identity and resolved path to be unchanged. Each of the five mutations was run in a scratch copy and shown red. --- tests/codex-transition-state.test.ts | 88 ++++++++++++++++++++++++++-- tests/codex-user-identity.test.ts | 54 +++++++++++------ 2 files changed, 120 insertions(+), 22 deletions(-) diff --git a/tests/codex-transition-state.test.ts b/tests/codex-transition-state.test.ts index 00c2f2f2c..c2938e293 100644 --- a/tests/codex-transition-state.test.ts +++ b/tests/codex-transition-state.test.ts @@ -189,6 +189,26 @@ test("a native CAS with a matching generation but the wrong txId still conflicts }); }); +test("a native CAS expecting a positive generation with a null txId still conflicts", () => { + expect(beginCodexTransition( + { nativeGeneration: 0, currentTxId: null }, + transition("tx-current"), + ).kind).toBe("updated"); + + // The generation agrees, but null is not a wildcard for the txId half of + // the pair. Weakening the predicate for a null expectation makes this win. + const nullTxId = beginCodexTransition( + { nativeGeneration: 1, currentTxId: null }, + transition("tx-forged"), + ); + expect(nullTxId.kind).toBe("conflict"); + + expect(readCodexTransitionState()).toMatchObject({ + kind: "ready", + state: { nativeGeneration: 1, currentTxId: "tx-current" }, + }); +}); + test("a native CAS with a matching txId but the wrong generation still conflicts", () => { expect(beginCodexTransition( { nativeGeneration: 0, currentTxId: null }, @@ -265,18 +285,21 @@ test("the row validator refuses a malformed row the CHECK constraints never saw" expect(readCodexTransitionState()).toEqual({ kind: "unavailable", reason: "database" }); }); -test("the row validator refuses a positive generation with a blank txId", () => { +test("the row validator refuses an unknown non-null history direction", () => { expect(beginCodexTransition( { nativeGeneration: 0, currentTxId: null }, - transition("tx-blank"), + transition("tx-unknown-direction"), ).kind).toBe("updated"); const database = new Database(coordinatorPath); try { database.exec("PRAGMA ignore_check_constraints = ON"); database.run( - "UPDATE codex_transition_state SET current_tx_id = ' ', history_tx_id = ' ' WHERE singleton = 1", + "UPDATE codex_transition_state SET history_direction = 'sideways' WHERE singleton = 1", ); + expect(database.query<{ history_direction: string }, []>( + "SELECT history_direction FROM codex_transition_state WHERE singleton = 1", + ).get()?.history_direction).toBe("sideways"); } finally { database.close(); } @@ -284,6 +307,37 @@ test("the row validator refuses a positive generation with a blank txId", () => expect(readCodexTransitionState()).toEqual({ kind: "unavailable", reason: "database" }); }); +test("the row validator refuses every whitespace-only txId", () => { + expect(beginCodexTransition( + { nativeGeneration: 0, currentTxId: null }, + transition("tx-blank"), + ).kind).toBe("updated"); + + const blankTxIds = [ + ["ASCII spaces", " "], + ["tab", "\t"], + ["newline", "\n"], + ["vertical tab", "\v"], + ["form feed", "\f"], + ["non-breaking space", "\u00a0"], + ["em space", "\u2003"], + ] as const; + + for (const [label, txId] of blankTxIds) { + const database = new Database(coordinatorPath); + try { + database.exec("PRAGMA ignore_check_constraints = ON"); + database.query( + "UPDATE codex_transition_state SET current_tx_id = ?, history_tx_id = ? WHERE singleton = 1", + ).run(txId, txId); + } finally { + database.close(); + } + + expect(readCodexTransitionState(), label).toEqual({ kind: "unavailable", reason: "database" }); + } +}); + /** * A capability backed by a nominal transaction is not opaque if its caller can * simply open another connection. The C-phase review found the old test only @@ -313,6 +367,33 @@ test("the opaque capability never exposes a reachable database handle", () => { expect(readCodexTransitionState().kind).toBe("ready"); const controller = openCodexCoordinatorTransaction(coordinatorPath); try { + const ownKeys = Reflect.ownKeys(controller.capability); + const stringKeys = ownKeys.filter((key): key is string => typeof key === "string"); + const symbolKeys = ownKeys.filter((key): key is symbol => typeof key === "symbol"); + + expect(stringKeys).toEqual(["beginTransition"]); + expect(symbolKeys).toHaveLength(1); + expect(symbolKeys[0]?.description).toBe("CodexCoordinatorTransaction"); + + for (const key of ownKeys) { + const descriptor = Reflect.getOwnPropertyDescriptor(controller.capability, key); + expect(descriptor).toBeDefined(); + expect("get" in descriptor!).toBe(false); + expect("set" in descriptor!).toBe(false); + } + expect(Reflect.getOwnPropertyDescriptor(controller.capability, "beginTransition")).toEqual({ + value: expect.any(Function), + writable: true, + enumerable: true, + configurable: true, + }); + expect(Reflect.getOwnPropertyDescriptor(controller.capability, symbolKeys[0]!)).toEqual({ + value: true, + writable: true, + enumerable: true, + configurable: true, + }); + const reachable = new Set(); const walk = (value: unknown, depth: number): void => { if (depth > 4 || value === null || reachable.has(value)) return; @@ -321,7 +402,6 @@ test("the opaque capability never exposes a reachable database handle", () => { reachable.add(value); for (const key of Reflect.ownKeys(value as object)) { const descriptor = Reflect.getOwnPropertyDescriptor(value as object, key); - // Only follow plain values: invoking a getter is not "exposure". if (descriptor && "value" in descriptor) walk(descriptor.value, depth + 1); } walk(Reflect.getPrototypeOf(value as object), depth + 1); diff --git a/tests/codex-user-identity.test.ts b/tests/codex-user-identity.test.ts index e63e09edc..878a9413e 100644 --- a/tests/codex-user-identity.test.ts +++ b/tests/codex-user-identity.test.ts @@ -117,27 +117,45 @@ test("real processes resolve one identity and coordinator path across every home }); try { - const probes = await Promise.all(environmentRoots.map(({ paths }) => runIdentityProbe({ - HOME: paths.home, - USERPROFILE: paths.userProfile, - HOMEDRIVE: paths.homeDrive, - HOMEPATH: paths.homePath, - XDG_RUNTIME_DIR: paths.xdgRuntime, - TMPDIR: paths.temp, - TEMP: paths.temp, - TMP: paths.temp, - LOCALAPPDATA: paths.temp, - CODEX_HOME: paths.codexHome, - OPENCODEX_HOME: paths.opencodexHome, - OCX_TEST_CANONICAL_CODEX_HOME: canonicalHome, - }))); + const probes = await Promise.all(environmentRoots.map(({ paths }, index) => { + const accountEnvironment = process.platform === "win32" + ? { + USERNAME: `fake-username-${index}`, + USERDOMAIN: `fake-domain-${index}`, + USERDOMAIN_ROAMINGPROFILE: `fake-roaming-domain-${index}`, + USERDNSDOMAIN: `fake-dns-domain-${index}`, + } + : { + UID: String(900_000 + index), + EUID: String(910_000 + index), + USER: `fake-user-${index}`, + LOGNAME: `fake-logname-${index}`, + }; + return runIdentityProbe({ + HOME: paths.home, + USERPROFILE: paths.userProfile, + HOMEDRIVE: paths.homeDrive, + HOMEPATH: paths.homePath, + XDG_RUNTIME_DIR: paths.xdgRuntime, + TMPDIR: paths.temp, + TEMP: paths.temp, + TMP: paths.temp, + LOCALAPPDATA: paths.temp, + CODEX_HOME: paths.codexHome, + OPENCODEX_HOME: paths.opencodexHome, + OCX_TEST_CANONICAL_CODEX_HOME: canonicalHome, + ...accountEnvironment, + }); + })); + const osIdentity = resolveEffectiveUserIdentity(); + const osDatabasePath = resolveCodexCoordinatorDatabasePath(osIdentity, canonicalHome); + for (const probe of probes) { + expect(probe.identity).toEqual(osIdentity); + expect(probe.databasePath).toBe(osDatabasePath); + } expect(probes[1]?.identity).toEqual(probes[0]?.identity); expect(probes[1]?.databasePath).toBe(probes[0]?.databasePath); - expect(probes[0]?.databasePath).toBe(resolveCodexCoordinatorDatabasePath( - resolveEffectiveUserIdentity(), - canonicalHome, - )); } finally { for (const { root } of environmentRoots) rmSync(root, { recursive: true, force: true }); } From d1cee048e117919227dffe49a55829b89932db1f Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Tue, 4 Aug 2026 18:21:35 +0900 Subject: [PATCH 050/163] fix(codex): a native-tagged history row is not proof the rollout was restored MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit An independent verifier reproduced the last fail-open by running production code: a `threads` row tagged `openai` whose referenced rollout file still carried `model_provider: "opencodex"` classified as clean, so the coordinator installed a fresh `{0,null}` over it. That state is reachable, not hypothetical — `restoreNativeHistory` swallows a failed rollout write in a bare `catch` (history-provider.ts:487-494) and then marks every row native anyway (497-507). The database and the rollout disagree, and only the database was being read. Rollouts referenced by the history database and by backup manifests are now inspected directly: JSONL structure, the referenced thread id, and both the first and latest `session_meta` provider, because the format is last-writer-wins over a first-line reader and either end can hold the routed value. Missing, malformed, mismatched, non-file, unreadable or racing observations are `indeterminate` rather than clean. The history database is opened `{readonly: true}`, verified by running it: on Bun 1.3.14 that refuses writes with SQLITE_READONLY, while both `{create:false}` and `{readwrite:false, create:false}` are SQLITE_MISUSE. Arbitrary unreferenced external rollouts stay out of scope — nothing structural points at them, and WP10 owns that discovery. Proven by mutation: with rollout inspection disabled the verifier's exact repro returns `clean` and goes red. --- src/codex/native-residue.ts | 106 +++++++++++++++++-- tests/codex-native-residue.test.ts | 158 ++++++++++++++++++++++++++--- 2 files changed, 244 insertions(+), 20 deletions(-) diff --git a/src/codex/native-residue.ts b/src/codex/native-residue.ts index 6216cb38c..eb0f8e96f 100644 --- a/src/codex/native-residue.ts +++ b/src/codex/native-residue.ts @@ -56,6 +56,11 @@ type ConfigObservation = { catalogTargets: CatalogTarget[]; }; +type RolloutReference = { + id: string; + path: string; +}; + const CONFIG_FILE_NAME = basename(CODEX_CONFIG_PATH); const PROFILE_FILE_NAME = basename(CODEX_PROFILE_PATH); const CATALOG_FILE_NAME = basename(DEFAULT_CATALOG_PATH); @@ -339,6 +344,67 @@ function classifyPartialWrites(targetPaths: string[]): NativeRoutedResidueResult return { kind: "clean" }; } +function classifyReferencedRollout( + surface: "history" | "history-backup", + reference: RolloutReference, +): NativeRoutedResidueResult { + const read = readRegularFile(reference.path); + if (read.kind === "absent") { + return indeterminate(surface, reference.path, "referenced rollout is absent"); + } + if (read.kind === "indeterminate") return indeterminate(surface, reference.path, read.reason); + + let first: Record | undefined; + let latest: Record | undefined; + for (const line of read.content.split("\n")) { + if (!line.trim()) continue; + let parsed: unknown; + try { + parsed = JSON.parse(line); + } catch (error) { + return indeterminate(surface, read.path, `malformed rollout JSONL: ${errorReason(error)}`); + } + if (!parsed || typeof parsed !== "object" || Array.isArray(parsed)) { + return indeterminate(surface, read.path, "rollout JSONL record is not an object"); + } + const record = parsed as Record; + if (record.type !== "session_meta") continue; + if (!record.payload || typeof record.payload !== "object" || Array.isArray(record.payload)) { + return indeterminate(surface, read.path, "session_meta payload has an unknown shape"); + } + const payload = record.payload as Record; + first ??= payload; + latest = payload; + } + + if (!first || !latest) { + return indeterminate(surface, read.path, "referenced rollout has no session_meta metadata"); + } + for (const [position, payload] of [["first", first], ["latest", latest]] as const) { + if (payload.id !== reference.id) { + return indeterminate(surface, read.path, `${position} session_meta does not identify the referenced thread`); + } + if (typeof payload.model_provider !== "string" || !payload.model_provider) { + return indeterminate(surface, read.path, `${position} session_meta has no provider metadata`); + } + if (payload.model_provider === "opencodex") { + return { kind: "residue", surface, path: read.path }; + } + } + return { kind: "clean" }; +} + +function classifyReferencedRollouts( + surface: "history" | "history-backup", + references: RolloutReference[], +): NativeRoutedResidueResult { + for (const reference of references) { + const result = classifyReferencedRollout(surface, reference); + if (result.kind !== "clean") return result; + } + return { kind: "clean" }; +} + function classifyHistoryDatabase(path: string): NativeRoutedResidueResult { const resolved = resolveRegularFile(path); if (resolved.kind === "absent") { @@ -358,17 +424,28 @@ function classifyHistoryDatabase(path: string): NativeRoutedResidueResult { try { database = new Database(resolved.path, { readonly: true }); database.exec("PRAGMA busy_timeout = 100"); - const row = database.query<{ n: number }, []>(` - SELECT count(*) AS n + const rows = database.query<{ id: string; rollout_path: string; model_provider: string }, []>(` + SELECT id, rollout_path, model_provider FROM threads - WHERE model_provider = 'opencodex' - AND trim(coalesce(first_user_message, '')) != '' - `).get(); + `).all(); + for (const row of rows) { + if (typeof row.id !== "string" || !row.id || typeof row.rollout_path !== "string" || !row.rollout_path) { + return indeterminate("history", resolved.path, "history row has an unknown rollout reference"); + } + if (typeof row.model_provider !== "string" || !row.model_provider) { + return indeterminate("history", resolved.path, "history row has no provider metadata"); + } + } + const rollouts = classifyReferencedRollouts( + "history", + rows.map(row => ({ id: row.id, path: row.rollout_path })), + ); + if (rollouts.kind !== "clean") return rollouts; const after = statSync(resolved.path); if (!sameStat(resolved.stat, after)) { return indeterminate("history", resolved.path, "history database changed while it was being observed"); } - return (row?.n ?? 0) > 0 + return rows.some(row => row.model_provider === "opencodex") ? { kind: "residue", surface: "history", path: resolved.path } : { kind: "clean" }; } catch (error) { @@ -410,7 +487,22 @@ function classifyHistoryBackup(path: string, stateDatabasePath: string): NativeR return indeterminate("history-backup", read.path, "history backup names a different state database"); } } - return Object.keys(manifest.entries as Record).length > 0 + const entries = Object.values(manifest.entries as Record); + const references: RolloutReference[] = []; + for (const entry of entries) { + if (!entry || typeof entry !== "object" || Array.isArray(entry)) { + return indeterminate("history-backup", read.path, "history backup entry has an unknown shape"); + } + const candidate = entry as Record; + if (typeof candidate.id !== "string" || !candidate.id + || typeof candidate.rolloutPath !== "string" || !candidate.rolloutPath) { + return indeterminate("history-backup", read.path, "history backup entry has an unknown rollout reference"); + } + references.push({ id: candidate.id, path: candidate.rolloutPath }); + } + const rollouts = classifyReferencedRollouts("history-backup", references); + if (rollouts.kind !== "clean") return rollouts; + return entries.length > 0 ? { kind: "residue", surface: "history-backup", path: read.path } : { kind: "clean" }; } diff --git a/tests/codex-native-residue.test.ts b/tests/codex-native-residue.test.ts index 8bb964095..6424a33bf 100644 --- a/tests/codex-native-residue.test.ts +++ b/tests/codex-native-residue.test.ts @@ -71,7 +71,22 @@ function routedCatalog(): string { return JSON.stringify({ models }, null, 2) + "\n"; } -function createHistoryDatabase(modelProvider: "openai" | "opencodex"): void { +function sessionMeta(id: string, modelProvider: string): string { + return JSON.stringify({ + timestamp: "2026-08-04T00:00:00.000Z", + type: "session_meta", + payload: { id, model_provider: modelProvider, source: "cli" }, + }); +} + +function createHistoryDatabase( + modelProvider: "openai" | "opencodex", + rolloutProviders: string[] = [modelProvider], +): void { + writeFileSync( + pathInCodexHome("rollout.jsonl"), + rolloutProviders.map(provider => sessionMeta("thread-1", provider)).join("\n") + "\n", + ); const database = new Database(pathInCodexHome("state_5.sqlite")); database.exec(` CREATE TABLE threads ( @@ -151,19 +166,22 @@ const residueFixtures: Array<{ { name: "history backup entry", surface: "history-backup", - arrange: () => writeFileSync(historyBackupPath(), JSON.stringify({ - version: 1, - stateDbPath: join(realpathSync.native(codexHome), "state_5.sqlite"), - entries: { - "thread-1": { - id: "thread-1", - rolloutPath: pathInCodexHome("rollout.jsonl"), - modelProvider: "openai", - source: "cli", - hasUserEvent: 1, + arrange: () => { + writeFileSync(pathInCodexHome("rollout.jsonl"), sessionMeta("thread-1", "openai") + "\n"); + writeFileSync(historyBackupPath(), JSON.stringify({ + version: 1, + stateDbPath: join(realpathSync.native(codexHome), "state_5.sqlite"), + entries: { + "thread-1": { + id: "thread-1", + rolloutPath: pathInCodexHome("rollout.jsonl"), + modelProvider: "openai", + source: "cli", + hasUserEvent: 1, + }, }, - }, - })), + })); + }, }, ]; @@ -333,6 +351,120 @@ test("a slash-bearing catalog slug without OpenCodex authorship is indeterminate }); }); +test("a native-tagged history row with routed latest rollout metadata refuses coordinator initialization", () => { + createHistoryDatabase("openai", ["openai", "opencodex"]); + + expect(classifyNativeRoutedResidue()).toMatchObject({ + kind: "residue", + surface: "history", + path: pathInCodexHome("rollout.jsonl"), + }); + expect(readCodexTransitionState()).toEqual({ + kind: "legacy-ambiguous", + message: "A missing coordinator row cannot be initialized while native Codex routing residue exists.", + }); +}); + +test("routed first rollout metadata is residue even when the latest metadata is native", () => { + createHistoryDatabase("openai", ["opencodex", "openai"]); + + expect(classifyNativeRoutedResidue()).toMatchObject({ + kind: "residue", + surface: "history", + path: pathInCodexHome("rollout.jsonl"), + }); +}); + +test("a referenced rollout with native first and latest metadata is clean", () => { + createHistoryDatabase("openai", ["openai", "openai"]); + + expect(classifyNativeRoutedResidue()).toEqual({ kind: "clean" }); +}); + +for (const fixture of [ + { + name: "missing", + arrange: () => { + createHistoryDatabase("openai"); + rmSync(pathInCodexHome("rollout.jsonl")); + }, + }, + { + name: "malformed", + arrange: () => { + createHistoryDatabase("openai"); + writeFileSync(pathInCodexHome("rollout.jsonl"), "{not-json\n"); + }, + }, + { + name: "non-file", + arrange: () => { + createHistoryDatabase("openai"); + rmSync(pathInCodexHome("rollout.jsonl")); + mkdirSync(pathInCodexHome("rollout.jsonl")); + }, + }, +]) { + test(`a ${fixture.name} rollout referenced by a live history row is indeterminate`, () => { + fixture.arrange(); + + expect(classifyNativeRoutedResidue()).toMatchObject({ + kind: "indeterminate", + surface: "history", + path: pathInCodexHome("rollout.jsonl"), + }); + expect(readCodexTransitionState()).toEqual({ + kind: "legacy-ambiguous", + message: "A missing coordinator row cannot be initialized while native Codex routing residue exists.", + }); + }); +} + +test("a manifest-referenced routed rollout is residue", () => { + writeFileSync(pathInCodexHome("rollout.jsonl"), sessionMeta("thread-1", "opencodex") + "\n"); + writeFileSync(historyBackupPath(), JSON.stringify({ + version: 1, + stateDbPath: join(realpathSync.native(codexHome), "state_5.sqlite"), + entries: { + "thread-1": { + id: "thread-1", + rolloutPath: pathInCodexHome("rollout.jsonl"), + modelProvider: "openai", + source: "cli", + hasUserEvent: 1, + }, + }, + })); + + expect(classifyNativeRoutedResidue()).toMatchObject({ + kind: "residue", + surface: "history-backup", + path: pathInCodexHome("rollout.jsonl"), + }); +}); + +test("a missing manifest-referenced rollout is indeterminate", () => { + writeFileSync(historyBackupPath(), JSON.stringify({ + version: 1, + stateDbPath: join(realpathSync.native(codexHome), "state_5.sqlite"), + entries: { + "thread-1": { + id: "thread-1", + rolloutPath: pathInCodexHome("missing-rollout.jsonl"), + modelProvider: "openai", + source: "cli", + hasUserEvent: 1, + }, + }, + })); + + expect(classifyNativeRoutedResidue()).toMatchObject({ + kind: "indeterminate", + surface: "history-backup", + path: pathInCodexHome("missing-rollout.jsonl"), + }); +}); + const indeterminateFixtures: Array<{ name: string; surface: string; From de15caf449264f203cf003d032bb5f62bb448e72 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Tue, 4 Aug 2026 18:53:10 +0900 Subject: [PATCH 051/163] fix(codex): the classifier parsed config.toml more strictly than production does MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fourth instance of the same defect, found by probing the two parsers against each other rather than by reading either one. Round 2 resolved `model_catalog_json` with `Bun.TOML.parse`, but production does not use a TOML parser: `readCodexCatalogPath` (catalog/parsing.ts:167) resolves it with the line-based `readRootTomlString`. Anywhere production resolves a path the strict parse does not, production writes routed bytes somewhere the classifier never looks, and the verdict is `clean`. Across seven config shapes exactly one diverges in that unsafe direction: a UTF-8 BOM. A `config.toml` beginning with a BOM resolves to `bom-catalog.json` in production while the strict parse sees no root key at all, so a genuine `Routed via opencodex → ` row sitting at the production target classified `clean` and initialization returned a ready `{0,null}`. The candidate set is now the union of production's rule, the strict rule with a leading BOM tolerated, and the default target; disagreement inspects both. The regression that matters is table-driven over those seven shapes and asserts the candidates cover whatever the real `readCodexCatalogPath()` returns, so changing either parser in isolation goes red instead of silently reopening this. The opposite divergence — a quoted key the strict parse sees and production does not — is left alone. Inspecting a path production would never write is harmless; narrowing to match would be the fail-open direction again. --- src/codex/native-residue.ts | 28 ++++++++----- tests/codex-native-residue.test.ts | 64 +++++++++++++++++++++++++++++- 2 files changed, 81 insertions(+), 11 deletions(-) diff --git a/src/codex/native-residue.ts b/src/codex/native-residue.ts index eb0f8e96f..1f3bd6551 100644 --- a/src/codex/native-residue.ts +++ b/src/codex/native-residue.ts @@ -19,6 +19,7 @@ import { CODEX_PROFILE_PATH, DEFAULT_CATALOG_PATH, getCodexHome, + readRootTomlString, } from "./paths"; export type NativeResidueSurface = @@ -168,7 +169,7 @@ function catalogPathKey(path: string): string { function catalogTargets( codexHome: string, - configuredPath?: string, + configuredPaths: readonly string[] = [], ): CatalogTarget[] { const targets = new Map(); const add = (path: string, configured: boolean) => { @@ -176,7 +177,9 @@ function catalogTargets( const existing = targets.get(key); targets.set(key, { path: resolve(path), configured: configured || existing?.configured === true }); }; - if (configuredPath !== undefined) add(resolve(codexHome, configuredPath), true); + for (const configuredPath of configuredPaths) { + add(resolve(codexHome, configuredPath), true); + } add(join(codexHome, CATALOG_FILE_NAME), false); return [...targets.values()]; } @@ -193,38 +196,45 @@ function inspectConfig(codexHome: string, path: string): ConfigObservation { }; } + const productionConfiguredPath = readRootTomlString(read.content, "model_catalog_json"); + const productionConfiguredPaths = productionConfiguredPath === null + ? [] + : [productionConfiguredPath]; let parsed: unknown; try { - parsed = Bun.TOML.parse(read.content); + parsed = Bun.TOML.parse(read.content.replace(/^\uFEFF/, "")); } catch (error) { return { classification: indeterminate("config", read.path, `malformed TOML: ${errorReason(error)}`), - catalogTargets: catalogTargets(codexHome), + catalogTargets: catalogTargets(codexHome, productionConfiguredPaths), }; } if (!parsed || typeof parsed !== "object" || Array.isArray(parsed)) { return { classification: indeterminate("config", read.path, "TOML root is not a table"), - catalogTargets: catalogTargets(codexHome), + catalogTargets: catalogTargets(codexHome, productionConfiguredPaths), }; } const document = parsed as Record; let targets: CatalogTarget[]; if (!Object.hasOwn(document, "model_catalog_json")) { - targets = catalogTargets(codexHome); + targets = catalogTargets(codexHome, productionConfiguredPaths); } else if (typeof document.model_catalog_json !== "string" || !document.model_catalog_json.trim()) { return { classification: indeterminate("config", read.path, "model_catalog_json must be one non-empty string"), - catalogTargets: catalogTargets(codexHome), + catalogTargets: catalogTargets(codexHome, productionConfiguredPaths), }; } else { try { - targets = catalogTargets(codexHome, document.model_catalog_json); + targets = catalogTargets(codexHome, [ + ...productionConfiguredPaths, + document.model_catalog_json, + ]); } catch (error) { return { classification: indeterminate("config", read.path, `model_catalog_json cannot be resolved: ${errorReason(error)}`), - catalogTargets: catalogTargets(codexHome), + catalogTargets: catalogTargets(codexHome, productionConfiguredPaths), }; } } diff --git a/tests/codex-native-residue.test.ts b/tests/codex-native-residue.test.ts index 6424a33bf..92ce332ea 100644 --- a/tests/codex-native-residue.test.ts +++ b/tests/codex-native-residue.test.ts @@ -9,11 +9,15 @@ import { writeFileSync, } from "node:fs"; import { tmpdir } from "node:os"; -import { join, resolve } from "node:path"; +import { dirname, join, resolve } from "node:path"; import { Database } from "bun:sqlite"; -import { buildCatalogEntries, syncCatalogModels } from "../src/codex/catalog"; +import { + buildCatalogEntries, + readCodexCatalogPath, + syncCatalogModels, +} from "../src/codex/catalog"; import { buildProfileFile } from "../src/codex/inject"; import { classifyNativeRoutedResidue } from "../src/codex/native-residue"; import { readCodexTransitionState } from "../src/codex/transition-state"; @@ -235,6 +239,62 @@ test("a routed catalog at the configured nested path refuses coordinator initial }); }); +test("a BOM-prefixed configured catalog refuses coordinator initialization", () => { + writeFileSync( + pathInCodexHome("config.toml"), + '\uFEFFmodel_catalog_json = "bom-catalog.json"\n', + ); + const productionTarget = readCodexCatalogPath(); + mkdirSync(dirname(productionTarget), { recursive: true }); + writeFileSync(productionTarget, routedCatalog()); + + expect(classifyNativeRoutedResidue()).toEqual({ + kind: "residue", + surface: "catalog", + path: productionTarget, + }); + expect(readCodexTransitionState()).toEqual({ + kind: "legacy-ambiguous", + message: "A missing coordinator row cannot be initialized while native Codex routing residue exists.", + }); +}); + +const catalogConfigShapes: Array<{ + name: string; + content: string; + strictOnlyTargets?: string[]; +}> = [ + { + name: "quoted key", + content: '"model_catalog_json" = "quoted.json"\n', + strictOnlyTargets: ["quoted.json"], + }, + { name: "single-quoted value", content: "model_catalog_json = 'single.json'\n" }, + { name: "BOM prefix", content: '\uFEFFmodel_catalog_json = "bom.json"\n' }, + { name: "CRLF", content: 'model_catalog_json = "crlf.json"\r\n' }, + { name: "leading whitespace", content: ' model_catalog_json = "ws.json"\n' }, + { name: "after table header", content: '[tools]\nmodel_catalog_json = "nested.json"\n' }, + { name: "trailing comment", content: 'model_catalog_json = "cmt.json" # comment\n' }, +]; + +for (const shape of catalogConfigShapes) { + test(`catalog candidate coverage matches production for ${shape.name}`, () => { + writeFileSync(pathInCodexHome("config.toml"), shape.content); + for (const target of shape.strictOnlyTargets ?? []) { + writeFileSync(pathInCodexHome(target), JSON.stringify({ models: [] })); + } + const productionTarget = readCodexCatalogPath(); + mkdirSync(dirname(productionTarget), { recursive: true }); + writeFileSync(productionTarget, routedCatalog()); + + expect(classifyNativeRoutedResidue()).toEqual({ + kind: "residue", + surface: "catalog", + path: productionTarget, + }); + }); +} + test("an atomic-write artifact beside the configured catalog is indeterminate", () => { const catalogPath = canonicalPathInCodexHome("nested/custom-catalog.json"); const artifactPath = `${catalogPath}.ocx.42.7.tmp`; From 63b03c6acfac93b57e1932d8e8f676d55154991c Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Tue, 4 Aug 2026 19:04:58 +0900 Subject: [PATCH 052/163] docs(substrate): the catalog seam cannot borrow the native generation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit An independent audit of the WP9 plan came back FAIL with eight findings, five of them HIGH and most demonstrated by running the code rather than reading it. Three were contract-level and are settled here. A catalog-only commit has no legal way to publish a native generation. The coordinator schema requires every `native_generation > 0` row to carry a matching history txId, direction and authority snapshot (transition-state.ts:74-83), so the only publication API always schedules history work (:314-344) and `assertPublished` refuses without it (:420-428). WP9 could therefore only advance the pair by scheduling history it does not own — a lie in durable state. The pair is redefined to mean what it always described: a native ROUTING transition, the class that needs history follow-up. Catalog bytes, hashed and legacy backups, and the models cache change what Codex can list, never where it sends traffic. Catalog scope neither reads `CommitExpectation` nor advances the pair, which is why the `catalog-only` outcome never had those fields, and a catalog commit that writes a routing artifact is now a contract violation rather than a surprise. That leaves catalog commits unprotected, so they get their own mechanism. Parent and inode identity was proven insufficient: a plain in-place truncate-and-rewrite leaves parent, device, inode and both generations unchanged while the gathered bytes go stale. The admission snapshot now retains a fingerprint of every source gather actually read, compared immediately before the first write, with unreadable sources refused rather than assumed unchanged. This is not the deleted ContentRevision design: it fingerprints the exact buffers one candidate consumed, not whole-config equality, and it is bounded honestly — single-direction drift is caught, a content A→B→A returning identical bytes is not. The plan also claimed a write-free gather that is not one. Loading the bundled catalog can persist the resolved runtime (runtime.ts:213-226), provider gather can refresh and persist OAuth credentials (oauth/store.ts:105-121), and reading the config generation creates its SQLite database (catalog-admission.ts:84-90). The guarantee is restated as filesystem-write- free with observe-only resolvers, and the acceptance manifest is captured before admission — taken after, it would have hidden exactly these writes. Remaining fixes: C14 is narrowed to the 16 management callers with the still-legacy sync/startup/restore roots enumerated rather than contradicted; the route adapter is total and non-throwing so a persisted 2xx cannot become a 500; the writer module keeps the contract's name; bundled-first template precedence is preserved; and five tests that would pass against a broken implementation are pinned to concrete mutations. --- .../005_contract.md | 122 +- .../010_catalog_seam.md | 1049 +++++++---------- 2 files changed, 554 insertions(+), 617 deletions(-) diff --git a/devlog/_plan/260804_codex_write_substrate/005_contract.md b/devlog/_plan/260804_codex_write_substrate/005_contract.md index 569cb5475..9d0a91c7b 100644 --- a/devlog/_plan/260804_codex_write_substrate/005_contract.md +++ b/devlog/_plan/260804_codex_write_substrate/005_contract.md @@ -556,7 +556,7 @@ indistinguishable from its failure condition. /** Bumped by every cooperating CONFIG write. Owned by src/config.ts. */ export interface ConfigGeneration { readonly value: number; } -/** Bumped by every cooperating NATIVE commit. Owned by transition-state.ts. */ +/** Bumped by every cooperating NATIVE ROUTING commit. Owned by transition-state.ts. */ export interface NativeGeneration { readonly value: number; } export type ConfigGenerationRead = @@ -576,6 +576,33 @@ Round 2 #6: the previous version said "two counters, both in the record" and then defined one. They are distinct because they answer different questions — did the user's configuration move, versus did somebody else write Codex's files. +The WP9 seam audit forced a narrower definition of that second question. The +implemented transition row requires every positive `native_generation` to carry +the same `history_tx_id` as `current_tx_id`, a non-null `history_direction`, and +a non-empty `history_authority_snapshot_id` +(`src/codex/transition-state.ts:74-83`). `beginCodexTransition` therefore always +publishes a pending HISTORY SCHEDULE with the pair +(`src/codex/transition-state.ts:314-344`), and `assertPublished` rejects a caller +that did not publish one (`src/codex/transition-state.ts:420-428`). Advancing the +pair for catalog bytes alone would invent history work that does not exist and +cross WP10/WP12's boundary. + +So the native generation identifies a **NATIVE ROUTING transition**: `config.toml`, +the generated profile, and the injection journal, exactly the artifacts whose +routing change requires history follow-up. The active catalog, hashed/legacy +catalog backups, and models cache are not routing artifacts. Rewriting them can +change what Codex lists; it cannot change where Codex sends traffic. A +`scope:"catalog"` commit therefore neither reads `CommitExpectation` nor advances +the native pair. The implemented `ConvergeOutcome` confirms that boundary: +`catalog-only` has no `nativeGeneration` or `currentTxId`, while the full routing +outcomes carry both (`src/codex/convergence-types.ts:207-224`). + +That is an honest reduction in protection: a catalog-only commit is not guarded +against staleness by the native pair. Its independent protection is the per-source +fingerprint check below. A catalog-only commit must never write a routing artifact; +if a future phase needs to write one, it uses `scope:"full"` and publishes the +native transition plus its truthful history schedule. + WP8b adds executable `readConfigGeneration` and `bumpConfigGeneration` exports to `src/config.ts` with the callable types above. They use a singleton `config_generation(singleton INTEGER PRIMARY KEY CHECK(singleton=1), value INTEGER @@ -595,7 +622,7 @@ and `src/config.ts` is now explicitly IN. export interface CommitExpectation { /** Read at admission. */ readonly nativeBefore: number; - /** What OUR commit will produce. Always nativeBefore + 1. */ + /** What OUR full routing commit will produce. Always nativeBefore + 1. */ readonly nativeAfter: number; /** Identifies the commit that performed the bump. */ readonly txId: string; @@ -615,15 +642,17 @@ native + config coordination provides **no cooperating interleaving while the process is alive**; a crash can still leave any prefix of the artifact sequence with the old coordinator pair. -Recovery is therefore artifact-specific. Config, generated profile, catalog, -hashed/legacy backups, cache, and journal recover only from their ledger baseline -plus matching post-image; a missing/null post-image preserves and refuses. History -rows, manifest entries, and rollouts remain `pending` and are re-probed/repaired by -the history guardian. A missing record with native residue or an invalid/ambiguous -record refuses automatic deletion. On restart, observation compares every artifact -to the ledger/current pair, records the unresolved surfaces, and schedules a fresh -current transition; idempotence is required but is not described as filesystem -atomicity. +Recovery for a `scope:"full"` routing transition is therefore artifact-specific. +Config, generated profile, catalog, hashed/legacy backups, cache, and journal +recover only from their ledger baseline plus matching post-image; a missing/null +post-image preserves and refuses. History rows, manifest entries, and rollouts +remain `pending` and are re-probed/repaired by the history guardian. A missing +record with native residue or an invalid/ambiguous record refuses automatic +deletion. On restart, observation compares every artifact to the ledger/current +pair, records the unresolved surfaces, and schedules a fresh current transition; +idempotence is required but is not described as filesystem atomicity. Catalog-only +staleness is instead admitted by the source fingerprints below, not retroactively +described as protection by a pair it never advanced. ### Prevention for cooperating writers (round 2 #5) @@ -649,11 +678,43 @@ A candidate records the canonical parent directory and the file identity symlink can retarget while the path string is unchanged, and `atomicWriteFile` resolves the effective target only at commit (`src/config.ts:190-199`). +The WP9 seam auditor then demonstrated the missing content dimension by gathering +a candidate, truncating and rewriting the catalog in place, and committing the +stale candidate. Path, canonical parent, parent identity, file identity, config +generation, and native pair all remained unchanged. Target identity says where a +write will land; it does not say that the bytes gather consumed are still current. + +The catalog admission snapshot therefore also retains a SHA-256 fingerprint of +the **exact byte buffer gather actually read** for every source that influenced +the prepared output: the active catalog, whichever hashed/legacy backup or models +cache was selected as a fallback, and any later file source whose bytes influence +that candidate. The gather reader computes the digest from the same buffer it +returns and records the source's canonical path; a separate pre-read is not +equivalent. Immediately before the first commit write, commit re-reads every +recorded source and compares its digest. Any mismatch is `stale`. An unreadable +source, an unresolvable canonical source, or ambiguous source identity is refused +rather than assumed unchanged. + +This is deliberately a per-source fingerprint of what one gather actually read. +It is not the deleted `ContentRevision` design, does not hash the whole persisted +configuration, and does not turn content into a global revision or transition +authority. That rejected design tried to make one content value stand in for +cooperating generations and failed the A→B→A case. This check instead binds a +prepared catalog candidate to the finite set of file bytes that produced it while +leaving config admission and native routing authority with their existing owners. + **What this does not do** (round 2 #6): it cannot detect a parent-symlink A→B→A that happens entirely between two checks. C17 is therefore scoped to *cooperating transitions and single-direction drift*, not to arbitrary filesystem ABA. Claiming otherwise would be a promise the filesystem does not offer. +The same limit applies to source bytes: fingerprints detect single-direction +content drift, including an ordinary in-place truncate-and-rewrite, but not a full +content A→B→A that returns to identical bytes before the commit check. The re-read +is also not filesystem atomicity; a non-cooperating writer can still change bytes +after the final comparison. The outcome must preserve those C17 bounds rather than +promote a digest into a guarantee the filesystem cannot provide. + ## 4. Admission returns a snapshot, not a boolean Audit #8: `040`'s intent reader returns ON/OFF while `010`'s gather needs a full @@ -661,7 +722,13 @@ Audit #8: `040`'s intent reader returns ON/OFF while `010`'s gather needs a full "two reads" is wrong. ```ts -/** The minimal, working WP8b/WP9 snapshot; it authorizes catalog work only. */ +/** Exact gather-time evidence for one file source that influenced the candidate. */ +export interface CatalogSourceFingerprint { + readonly canonicalPath: string; + readonly sha256: string; +} + +/** The shared WP8b/WP9 snapshot; it authorizes catalog work only. */ export interface CatalogAdmissionSnapshot { config: Readonly; generation: number; @@ -670,6 +737,8 @@ export interface CatalogAdmissionSnapshot { cache: string; catalogBackups: readonly string[]; }>; + /** Populated from the exact buffers gather read, never from separate pre-reads. */ + sourceFingerprints: readonly CatalogSourceFingerprint[]; } export interface AdmissionSnapshot { @@ -700,6 +769,13 @@ export interface AdmissionSnapshot { } ``` +Pre-gather capture begins with an empty `sourceFingerprints` list because fallback +selection has not happened yet. Gather does not mutate that snapshot: it returns +the prepared candidate with an immutable copy whose list is the exact set of file +sources its readers consumed. Commit accepts only that candidate-bound copy and +refuses an incomplete list; it never treats the empty pre-gather value as evidence +that a source stayed unchanged. + The earlier one-read claim is withdrawn. There are three authoritative observation points, each with a different job: @@ -709,7 +785,10 @@ points, each with a different job: 2. **Under-lock:** while native + config coordination is held, fully re-read snapshot B and compare digest, config generation, intent, ownership, external provider, canonical targets, journal identity, and provenance identity. A mismatch rejects - before the first native write. + before the first native write. For catalog work, re-read and compare every + gather-time `sourceFingerprint` immediately before the first write as §3 requires; + `scope:"catalog"` performs that check without reading or advancing a native + `CommitExpectation`. 3. **Post-commit:** re-read persisted config and observe every native/catalog/history surface into `CodexObservedState`. The outcome is not `converged` unless this observation agrees with admitted intent and the exact expected native pair. @@ -943,6 +1022,9 @@ contract: | JSON provenance ledger | `updateIntegrationRecord` in `src/codex/integration-record.ts` | `src/codex/convergence.ts` only | | persisted OpenCodex config bytes and config generation | private writers in `src/config.ts` | exported `saveConfig`, `mutatePersistedConfig`, `saveConfigPreservingClaudeCode`, and the generation API in that same module only | +`src/codex/internal/catalog-writer.ts` is the contract-owned name. Phase documents +must use it; `internal/catalog-commit.ts` is not an alternate name for this owner. + `inject.ts` is split: observation/parsing and pure config/profile transforms stay readable there; every export that calls `atomicWriteFile`/`unlinkSync` moves to `internal/native-writer.ts`. `journal.ts` is split into read/validate/classify code @@ -1000,7 +1082,11 @@ still returns 2xx while reporting a non-converged disposition. Concatenate all t TypeScript fences in document order, prepend the §1 `OcxConfig` import, and compile with the repository TypeScript compiler so WP8b cannot regress to TS2304 or a bodyless TS2391 declaration. Table-drive each artifact observation and require -`isApplied` only for the fully applied aggregate. +`isApplied` only for the fully applied aggregate. A catalog-only commit neither +requests a `CommitExpectation` nor changes the native pair, and its projected +outcome has no pair fields. Gather from a catalog source, truncate-and-rewrite that +same inode, and require commit to return `stale` before any write; repeat for each +selected backup/cache fallback and refuse unreadable or ambiguous re-reads. `tests/codex-user-identity.test.ts`: real child processes vary every environment home/runtime variable named in §7 and resolve one final database path for one @@ -1027,9 +1113,11 @@ writes to it. - C16 — one owner, one schema; a record from any phase reads in every other. - C17 — cooperating transition ABA is detected by the durable config/native generations and exact txId, and a parent target that drifts once between gather - and the under-lock commit check is detected by canonical target identity. An - arbitrary filesystem A→B→A retarget that completes wholly between two checks is - explicitly not claimed. + and the under-lock commit check is detected by canonical target identity. A + gathered catalog source whose bytes drift once is detected by its per-source + fingerprint even when dev+inode and both generations are unchanged. An arbitrary + filesystem or content A→B→A that completes wholly between two checks, and a write + after the final comparison, are explicitly not claimed. - Contributes to C15 with detect-and-repair: the latest native pair is durably pending before spawn, a stale Worker cannot replace its transition row or the winner's schedule, and the guardian diff --git a/devlog/_plan/260804_codex_write_substrate/010_catalog_seam.md b/devlog/_plan/260804_codex_write_substrate/010_catalog_seam.md index a3fe5f0dd..e8f4a8dde 100644 --- a/devlog/_plan/260804_codex_write_substrate/010_catalog_seam.md +++ b/devlog/_plan/260804_codex_write_substrate/010_catalog_seam.md @@ -3,211 +3,208 @@ Research: `001_catalog_seam.md`. Shared contract: `005_contract.md`. Read both before implementing this diff. -The incident is still r2 #1: the active refresh combines provider discovery, -catalog assembly, native writes, and cache invalidation in one awaited function -(`src/codex/refresh.ts:40-52`, `src/codex/catalog/sync.ts:507-569,600-616`). The -16 management mutations then call a `Promise` helper whose only failure -policy is a swallowed exception (`src/server/management-api.ts:105-112`, -`src/server/management/context.ts:54-69`). That shape cannot place slow gathering -outside a lock and a fixed commit inside it. - -This phase fixes that catalog mechanism. It is the **first real implementation** -of the contract's `convergeCodex`, but the earlier decision to send management -mutations through full apply/injection/history convergence is reversed. Those 16 -callers currently refresh catalog and cache only -(`src/server/management-api.ts:105-112`, `src/codex/refresh.ts:40-52`); changing a -provider must not start rewriting `config.toml`, profile, journal, or history in the -WP9 commit. WP9 therefore implements a catalog-scoped request and rewires only that -behavior. WP12 installs the authoritative full funnel and rewires `/api/sync` and -the remaining lifecycle callers after WP10-WP11 supply their safety mechanics. - -WP9 does not define another entry point, record, route mapping, admission shape, or -shared result union. WP8b lands the minimal concrete primitives listed below, not -declarations that require WP12 to become executable. WP9 consumes them and rewires -the catalog callers in the same commit, so this phase typechecks and preserves the -callers' 2xx/201 and native-write behavior on its own. - -All current-code citations and diff context below were rechecked on 2026-08-04 at -`47e7cac27723fa09dd7bb1bacac402b1e579b358`. +The incident is still r2 #1: catalog refresh combines provider discovery, +catalog assembly, catalog replacement, and cache invalidation in one awaited +operation (`src/codex/refresh.ts:40-52`, +`src/codex/catalog/sync.ts:507-569,600-616`). The 16 management mutations then +call a `Promise` helper that catches dynamic-import, discovery, parse, and +disk failures and discards all of them (`src/server/management-api.ts:105-112`). +That shape cannot put slow observation outside a later lock and a fixed write +sequence inside it, and it cannot tell the caller what actually happened. + +WP9 lands the first real catalog-scoped `ConvergeCodex`, rewires exactly those +16 management mutation sites, and leaves the explicit sync/startup/CLI/restore +roots for WP12. A catalog-only commit updates catalog, create-once catalog +backups, and models cache only. It neither reads nor advances the native routing +pair and never writes `config.toml`, generated profile, injection journal, or +history. The transition row makes that boundary mandatory: every positive +native generation requires matching history schedule fields, every +`beginTransition` publishes that schedule, and `assertPublished` rejects a +transition that was not published (`src/codex/transition-state.ts:74-83,314-344,420-428`). + +All current-code citations and diff context below were rechecked on 2026-08-04 +at `de15caf449264f203cf003d032bb5f62bb448e72`. The contract citations refer to +the concurrent WP8b amendment that adds catalog source fingerprints and excludes +catalog-only work from the native pair (`005_contract.md:579-604,681-716,724-742`). ## IN / OUT -IN — catalog mechanism: - -- `src/codex/refresh.ts` (MODIFY) — opaque candidate and catalog-private gather / - commit outcomes. -- `src/codex/catalog.ts` (MODIFY) — retain the existing facade while moving direct - writers behind the contract's internal boundary. -- `src/codex/catalog/sync.ts` (MODIFY) — write-free preparation and one fixed, - synchronous writer. -- `src/codex/catalog/bundled.ts` (MODIFY) — in-memory fallback during gather. -- `src/codex/catalog/parsing.ts` (MODIFY) — prepare create-once backup bytes without - writing them. -- `src/codex/catalog/provider-fetch.ts` (MODIFY) — sanitized, catalog-private - degradation notices. -- `src/codex/convergence.ts` (MODIFY) — implement the contract-declared - `convergeCodex` for the first time. The management path consumes the catalog-scoped - request/snapshot, generation tokens, `CatalogDisposition`, and `ConvergeOutcome` - from `convergence-types.ts`; it does not call WP12's full admission or observer. -- `src/codex/internal/catalog-commit.ts` (NEW/MOVE) — the prepared catalog/cache/ - backup writer, reachable only from `convergence.ts`, as required by - `005_contract.md` §Test plan. -- `src/codex/sync.ts` (NO CHANGE) — explicit sync remains on its current full native - path in WP9. WP12 rewires it after the full admission/observation mechanics exist. - -IN — production callers: - -- `src/server/management/context.ts`, `src/server/management-api.ts` (MODIFY) — - inject/call `convergeCodex` with `scope: "catalog"`, not a catalog-specific entry - point and not the full convergence scope. -- `src/server/management/provider-routes.ts` (MODIFY) — six mutations report the - contract's `CatalogDisposition` while retaining their primary status. -- `src/server/management/model-routes.ts` (MODIFY) — six mutations, same rule. -- `src/server/management/combo-routes.ts` (MODIFY) — two mutations, without - suppressing Claude follow-up work. -- `src/server/management/agent-settings-routes.ts` (MODIFY) — two mutations, - without suppressing Claude/Desktop follow-up work. -- `src/server/management/config-routes.ts` (NO CHANGE) — WP9 leaves explicit sync at - current lines 261-268; WP12 moves it to full convergence and the contract adapter. -- `structure/03_catalog-and-subagents.md`, - `structure/05_gui-and-management-api.md` (MODIFY) — document the production - funnel and best-effort mutation semantics. -- `docs-site/src/content/docs/reference/management-api.md` and matching `ja`, - `ko`, `ru`, `zh-cn` pages (MODIFY) — additive `catalogRefresh`; route statuses - are referenced from the contract, not copied into these implementation notes. - -IN — tests: - -- `tests/codex-refresh.test.ts`, `tests/codex-sync-api.test.ts`, - `tests/codex-models-cache-invalidate.test.ts`, - `tests/injection-model-api.test.ts` (MODIFY) — pure gather, fixed commit, real - generation invalidation, and compatibility behavior. -- `tests/model-visibility-management-api.test.ts`, - `tests/management-provider-validation.test.ts`, - `tests/combo-management-api.test.ts`, `tests/combos.test.ts`, - `tests/codex-v2-gate.test.ts`, - `tests/management-integration-routes.test.ts`, - `tests/management-client-config-route.test.ts`, - `tests/responses-shadow-intercept.test.ts`, - `tests/server-combo-failover-e2e.test.ts`, and - `tests/catalog-input-modality-enum.test.ts` (MODIFY) — migrate fixtures to - `convergeCodex`; routes that do not refresh retain zero calls. -- `tests/codex-convergence-contract.test.ts` (MODIFY) — add production module-graph - reachability checks and the 16-caller funnel proof. WP8b created this test file; - WP9 extends it rather than creating a second guard. +IN — observe-only admission and gather: + +- `src/config.ts`, `src/codex/generation.ts` (MODIFY) — add a genuinely read-only + generation observation that never creates, initializes, chmods, or registers + `config-mutation.sqlite`. The existing `readConfigGeneration` is not that API: + it resolves/records the path and opens SQLite with `create:true` + (`src/config.ts:1741-1771,1845-1849`, `src/codex/generation.ts:93-103`). +- `src/codex/catalog-admission.ts` (MODIFY) — keep the landed request constructor + and snapshot capture; switch snapshot capture to the observe-only generation + read and carry the contract-owned source-fingerprint list. Do not redefine + `createCatalogConvergeRequest` or `captureCatalogAdmissionSnapshot`, which + already exist at lines 32-46 and 84-107. +- `src/codex/convergence-types.ts` (MODIFY) — synchronize the already contract-owned + `CatalogSourceFingerprint` and `CatalogAdmissionSnapshot.sourceFingerprints` + additions from `005_contract.md`; no WP9-private duplicate type is allowed. +- `src/codex/runtime.ts`, `src/codex/catalog/bundled.ts` (MODIFY) — catalog gather + resolves the runtime through the existing non-persisting `resolveCodexRuntime` + (`src/codex/runtime.ts:394-405`), never + `resolveAndPersistCodexRuntime`, whose successful path may mkdir and replace + `codex-runtime.json` (`src/codex/runtime.ts:213-228,500-516`, + `src/codex/catalog/bundled.ts:146-169`). +- `src/oauth/index.ts`, `src/oauth/store.ts`, + `src/codex/catalog/provider-fetch.ts` (MODIFY) — add and consume an observe-only + active-token snapshot based on `peekAuthStore`, which already promises no + chmod or invalid-file backup (`src/oauth/store.ts:145-157`). It never refreshes, + persists, acquires an intent lock, creates/removes an intent file, hardens a + path, or backs up malformed credentials. The current token resolver can enter + refresh/persistence (`src/oauth/index.ts:281-339,352-354`) and the current gather + awaits it (`src/codex/catalog/provider-fetch.ts:410-428`). +- `src/codex/refresh.ts`, `src/codex/catalog/sync.ts`, + `src/codex/catalog/parsing.ts` (MODIFY) — prepare immutable catalog/cache/backup + bytes and source evidence without writing. + +IN — fixed commit and convergence: + +- `src/codex/internal/catalog-writer.ts` (NEW/MOVE) — the contract-owned low-level + owner for catalog, hashed/legacy backups, and models cache. Do not create the + obsolete `internal/catalog-commit.ts` name + (`005_contract.md:1019,1025-1026`). +- `src/codex/convergence.ts` (NEW) — catalog gather/commit orchestration and the + only WP9 module allowed to call symbols in `internal/catalog-writer.ts`. +- `src/codex/management-convergence.ts` (MODIFY) — retain the landed + management-only factory and catalog-only projection, but replace the placeholder + body at lines 81-96 with the real call into `convergence.ts`. The factory keeps + the exact config reference it already captures; no second factory or projection + is introduced. +- `src/codex/catalog.ts` (MODIFY) — preserve reader/pure exports while removing + direct writer re-exports from the public facade after legacy callers have explicit + imports. It currently re-exports `syncCatalogModels`, `restoreCodexCatalog`, and + `invalidateCodexModelsCache` together (`src/codex/catalog.ts:1-11`). + +IN — management callers and tests: + +- `src/server/management-api.ts`, `src/server/management/context.ts`, and the four + invoking route modules (MODIFY) — replace the swallowed helper with a total, + lazy catalog-convergence adapter returning `CatalogDisposition`. +- `src/codex/sync.ts`, `src/server/management/config-routes.ts`, + `src/server/index.ts`, `src/cli/index.ts`, and `src/codex/inject.ts` (IMPORT-ONLY) + — keep the four legacy roots compiling after the public facade stops re-exporting + writers. Their behavior and ownership do not move until WP12. +- `tests/codex-refresh.test.ts` and the existing management route suites (MODIFY). +- `tests/codex-convergence-contract.test.ts` (CREATE). It does not exist in the + WP8b tree; WP9 creates it rather than “extending” an imaginary file. OUT: -- The `integrations/codex.json` schema and updater, `AdmissionSnapshot`, - `CommitExpectation`, `ConvergeRequest`, `ConvergeOutcome`, `CatalogDisposition`, - and `toSyncResponse` — owned by `005_contract.md` §§1-5. This document deletes - its old versions instead of restating them. -- Management status/header ownership. `/api/sync` is mapped only by - `src/server/management/sync-response.ts` (`005_contract.md` §5), but WP12 is the - phase that first connects that adapter to the production route. -- Desired-state, ownership, journal, and provenance policy — WP12 consumes the same - funnel and strengthens admission; WP9 does not reserve fake outcomes for it. -- The native write lock — WP11. WP9's commit is synchronous now so WP11 can wrap it - later without changing the catalog contract. -- History isolation/locking — WP10. -- Full admission, observed-state projection, apply/remove direction, and production - `/api/sync`/lifecycle rewiring — WP12. WP9 must not call their future helpers. -- `gui/**`, transactional rollback, release/deploy actions, and the live proxy on - port 10100. - -## WP8b prerequisites that make WP9 self-contained - -The earlier draft assumed `inspectAdmissionSnapshot`, config/native generation -owners, and observed-state projection would already exist. They do not: WP12 owns -the full authority read and observer (`040_ownership_convergence.md:42,119-157,327-378`). -A WP9 diff that calls those helpers cannot land independently. - -WP8b must therefore add these minimal **working** primitives before WP9, with focused -typecheck/tests in the WP8b commit: - -1. `ConvergeRequest.scope` with at least `"catalog" | "full"`, plus a concrete - catalog request constructor. The production management callbacks use only - `scope: "catalog"`; `"full"` remains the compatibility/current-behavior branch - until WP12 replaces it with authoritative admission. `ConvergeRequest` is the - public caller shape and does **not** gain a config field: permitting arbitrary - callers to substitute catalog authority would make the scoped funnel weaker than - the callback it replaces. Instead, WP8b exports a management-only factory that - captures the management context's exact config object and returns a - `ConvergeCodex`; the factory is not re-exported by the public Codex facade. -2. A concrete `CatalogAdmissionSnapshot` plus catalog-scoped snapshot reader that - accepts the same `OcxConfig` object the current callback already uses and captures - only the config generation and catalog target identities WP9 validates. It - performs no service ownership, external-provider, journal, provenance, desired- - state, history, or observed-state work and is not `inspectAdmissionSnapshot`. -3. Concrete config/native generation owners: every cooperating persisted-config - commit bumps the config generation through the existing config mutation owner, - while `transition-state.ts` reads/conditionally advances native generation plus - `txId` in the effective-user + canonical-`CODEX_HOME` SQLite coordinator row. - WP9 may consume those tokens; it may not assume WP12 will add their storage or - bump sites later. -4. One contract projection for catalog-only completion. WP9 supplies the real - `CatalogDisposition`; history and observed sections are synthesized as - **no-change/not-evaluated**, never by invoking WP10 history or WP12 observation. - -These are substrate primitives, not a partial ownership implementation. WP12 still -owns the authoritative full `AdmissionSnapshot`, fresh under-coordination re-read, -authority/provenance checks, real observed-state projection, and full caller funnel. -Moving the concrete generation owners forward is an explicit ownership correction -to `040_ownership_convergence.md:42`: WP12 consumes those WP8b owners instead of -introducing them after WP9 has already depended on them. - -## The catalog-private candidate - -**INFERRED implementation choice:** the candidate is opaque and one-shot. Its payload is held in a module-private -`WeakMap`, so callers cannot inspect credentials, substitute bytes, serialize it, -or reconstruct a stale candidate. +- WP10 history scheduling/worker behavior. Catalog-only work schedules no history. +- WP11 native lock acquisition. WP9 makes commit synchronous but does not import a + future lock helper or claim cross-process exclusion. +- WP12 full admission/observer/provenance, full `scope:"full"` convergence, + `/api/sync`, startup, CLI cache sync, restore, and complete writer reachability. +- Any runtime command that starts, stops, syncs, restores, ensures, or manages the + live service; any write to real `~/.codex` or `~/.opencodex`; GUI/release/deploy. + +WP9 typechecks at its own commit. `management-convergence.ts` contains working +catalog behavior, not a placeholder waiting for WP12. WP12 may consolidate the +management factory/projection into `convergence.ts` when it installs the full +entry point; that later move is a module consolidation, not completion of an +unfinished WP9 branch. + +## A. Filesystem-write-free gather + +### A1 — state the guarantee exactly + +The guarantee is **filesystem-write-free**, not globally side-effect-free. Gather +may update bounded process-local memo, discovery-status, provider model cache, and +in-flight admission maps. Those mutations already occur at +`src/codex/runtime.ts:362-405` and +`src/codex/catalog/provider-fetch.ts:455-465,495-507,608-615,675-685`; they are +permitted because they do not mutate user files and are reset between isolated +tests. No credential, raw provider error, source path, or digest may escape through +those caches into `CatalogDisposition`. + +Filesystem-write-free means the entire interval from **before admission capture** +through resolved runtime observation, token observation, provider calls, fallback +selection, parsing, serialization, and candidate construction performs no mkdir, +write, rename, copy, unlink, chmod/ACL change, SQLite create/init/WAL change, +ownership registration, backup, or transient temp-file creation. + +This bound deliberately catches the writes hidden by the old plan: + +- runtime selection must not persist `codex-runtime.json`; +- ordinary auth reads must not call `loadAuthStoreInternal`, whose read path + hardens files and backs up invalid JSON (`src/oauth/store.ts:128-137`); +- expired OAuth is a sanitized provider-auth degradation/failure for this gather, + not permission to refresh and persist; +- admission must not invoke the create-on-read generation path, which can also + register ownership metadata (`src/lib/config-ownership.ts:202-226,262-282`). + +The observe-only generation API opens an existing database with `readonly:true`, +performs only schema/version/select checks, and closes it. Missing DB/table/row, +busy, malformed, or unreadable state returns the existing typed unavailable result; +it never initializes generation zero. `captureCatalogAdmissionSnapshot` projects +that result into a typed catalog refusal through the total adapter. + +The observe-only token snapshot reads the active credential once from +`peekAuthStore`. A non-expired access token may be used. Missing, malformed, +near-expiry, or expired OAuth credentials yield provider-auth without calling any +refresh path. Static API keys and request headers already present in the admitted +`Readonly` remain usable. If the auth-store buffer influences a live +provider result, its canonical path and SHA-256 join the candidate's private source +fingerprints; token bytes never do. + +### A2 — fingerprint the exact buffers that influenced output + +`captureCatalogAdmissionSnapshot(config)` remains the pre-gather constructor and +starts with the contract-required empty `sourceFingerprints`. Each gather reader +returns bytes and a `CatalogSourceFingerprint` computed from that **same buffer**. +The candidate receives an immutable snapshot copy containing exactly the sources +actually selected or merged: + +- active catalog bytes when read as the merge source; +- the selected hashed backup, legacy backup, or models-cache fallback; +- persisted runtime-selection or auth-store bytes when those reads influenced + runtime/token selection; +- any later file buffer that affects candidate bytes. + +Do not fingerprint alternatives merely because their paths exist, and do not hash a +separate pre-read. Process-local caches and subprocess/network responses are not file +buffers and therefore are not fabricated as file fingerprints. + +Immediately before the first replacement, commit re-reads every candidate-bound +source by canonical path. Digest mismatch returns `stale`; unreadable, +unresolvable, non-regular, or ambiguous source identity returns `refused`. Both +paths write zero bytes. This detects the audited same-inode truncate/rewrite even +when config generation and target dev/inode are unchanged. It catches +single-direction drift only: content A→B→A returning identical bytes before the +comparison, a parent A→B→A between checks, and a write after the final comparison +remain outside C17 (`005_contract.md:681-716`). + +### A3 — preserve bundled-first template precedence + +`loadCatalogForSync` keeps its current default-path branch: obtain the bundled +catalog first and clone it as the native template; read the on-disk catalog +separately as the merge source. The invariant is explicit at +`structure/03_catalog-and-subagents.md:23-27` and implemented at +`src/codex/catalog/bundled.ts:225-234` plus +`src/codex/catalog/sync.ts:517-523`. + +The WP9 edit removes only the materializing fallback call from the tail of +`loadCatalogForSync`; it does not move catalog/backup/cache ahead of a successful +bundled template. The bundled branch uses observe-only runtime resolution. Existing +explicit materialization callers remain until their owning phase migrates them. + +### A4 — candidate ownership + +The candidate remains opaque, one-shot, and catalog-private. Its `WeakMap` state +contains prepared bytes, result/notices, target identities, the admitted config +generation, and the populated candidate-bound source fingerprints. Commit marks it +consumed before validation and before the first write; a second call returns +`candidate-consumed` and writes nothing. No route can inspect, serialize, +reconstruct, or replay it. + +Only catalog-private outcomes are added here: ```ts -import type { - CatalogAdmissionSnapshot as ContractCatalogAdmissionSnapshot, - CommitExpectation as ContractCommitExpectation, -} from "./convergence-types"; - -interface CatalogFileIdentity { - readonly device: bigint; - readonly inode: bigint; -} - -/** Catalog-private evidence for one prepared filesystem target. */ -export interface CatalogTargetIdentity { - readonly path: string; - readonly canonicalParent: string; - readonly parentIdentity: CatalogFileIdentity; - readonly fileIdentity: CatalogFileIdentity | null; -} - -/** Sanitized gather detail; provider identity and raw errors never enter it. */ -export interface CatalogGatherNotice { - readonly kind: "provider-auth" | "provider-network" | "fallback"; - readonly retryable: boolean; -} - -interface PreparedCatalogBackup { - readonly kind: "keyed" | "legacy"; - readonly path: string; - readonly bytes: Uint8Array; - readonly createOnce: true; -} - -export interface CodexCatalogRefreshResult { - readonly added: number; - readonly path: string; - readonly catalogExists: boolean; - readonly catalogWritten: boolean; - readonly cacheSynced: boolean; - readonly comboOmissions: readonly Readonly<{ - id: string; - targets: readonly string[]; - reason: "incomplete_metadata" | "incompatible_modalities"; - message: string; - }>[]; -} - export interface CatalogWriteReceipt { readonly keyedBackup: "written" | "preserved" | "not-requested"; readonly legacyBackup: "written" | "preserved" | "not-requested"; @@ -215,411 +212,277 @@ export interface CatalogWriteReceipt { readonly cache: "written" | "not-written"; } -export interface PreparedCodexCatalogCommit { - readonly catalogBytes: Uint8Array; - readonly cacheBytes: Uint8Array; - readonly backups: readonly PreparedCatalogBackup[]; - readonly targets: readonly CatalogTargetIdentity[]; - readonly result: CodexCatalogRefreshResult; - readonly notices: readonly CatalogGatherNotice[]; -} - -const candidateBrand: unique symbol = Symbol("CodexCatalogCandidate"); - -export interface CodexCatalogCandidate { - readonly [candidateBrand]: true; -} +export type CodexCatalogCommitResult = + | { readonly kind: "committed"; readonly changed: boolean; readonly writes: CatalogWriteReceipt } + | { readonly kind: "stale"; readonly reason: "generation" | "source-fingerprint" | "target-identity" | "candidate-consumed" } + | { readonly kind: "refused"; readonly reason: "source-unreadable" | "source-ambiguous" | "target-unsafe" } + | { readonly kind: "failed"; readonly surface: "disk"; readonly writes: CatalogWriteReceipt }; +``` -interface CandidateState { - readonly prepared: PreparedCodexCatalogCommit; - readonly admittedGeneration: number; - readonly targetIdentities: readonly CatalogTargetIdentity[]; - readonly notices: readonly CatalogGatherNotice[]; - consumed: boolean; -} +`convergence.ts` projects those private variants into the contract's existing +`CatalogDisposition`; routes never switch on this union. -const states = new WeakMap(); +## B. Fixed synchronous commit -/** Signature only; WP9 exports a concrete function with this type. */ -export type GatherCodexCatalogCandidate = ( - admission: ContractCatalogAdmissionSnapshot, -) => Promise; +Preparation returns exact catalog/cache bytes and optional create-once backup bytes. +`internal/catalog-writer.ts` accepts only that prepared value and synchronous +filesystem dependencies. It accepts no config, provider client, parser, subprocess, +OAuth resolver, Promise, or callback that can return a Promise. -/** Signature only; WP9 exports a concrete synchronous function with this type. */ -export type CommitCodexCatalogCandidate = ( - candidate: CodexCatalogCandidate, - expectation: ContractCommitExpectation, -) => CodexCatalogCommitResult; -``` +Commit performs, in order: -Ownership is deliberate. `CatalogAdmissionSnapshot` and `CommitExpectation` are -aliased imports from the shared `convergence-types.ts` contract, so concatenating -phase excerpts cannot turn the imports into duplicate local definitions; the full -`AdmissionSnapshot` is also contract-owned but is not referenced by this -catalog-scoped signature. `PreparedCodexCatalogCommit`, `CatalogTargetIdentity`, -`CatalogGatherNotice`, `CodexCatalogCommitResult`, `CodexCatalogRefreshResult`, and -`CatalogWriteReceipt` are catalog-private definitions here. WP11 exposes -`withCodexWriteLock` as a callback/result API and explicitly has no public handle or -release method (`030_lock_protocol.md:126-136`), so WP9 neither defines nor imports a -`CodexWriteLockHandle`. - -The catalog-scoped snapshot receives the same config object the current management -callback already uses. `prepareCatalogSync` receives `admission.config` — **that -object**, not a separate `readConfigDiagnostics()` result. The generation token -detects a cooperating persisted transition before commit. WP12 later replaces this -limited input with its authoritative full admission; WP9 does not import that future -helper. There is no resident config global and no persisted config re-read in this -catalog-only path: either would change the current callback's behavior when its -long-lived config object differs from disk. - -Gather performs provider auth/network work, source loading, parsing, merging, -serialization, cache-wrapper construction, and backup planning. It performs no -`mkdir`, copy, write, rename, journal mutation, or integration-record update. The -isolated-home before/after manifest is the acceptance evidence for that claim. - -Commit marks the candidate consumed before the first write, validates the shared -generation/identity evidence, and performs at most four atomic replacements in a -fixed order: keyed backup, optional legacy backup, catalog, cache. Retrying a -partially written candidate would replay old bytes after a later transition, so a -second call is a catalog-private `candidate-consumed` result and never writes. - -## C2 — generation and target identity, not content revision - -The old plan owned a `ContentRevision` and hashed config/catalog bytes. That design -is deleted. Content equality passes A→B→A, and a textual path does not reveal a -parent-symlink retarget. The shared mechanism is `005_contract.md` §3: - -- `CatalogAdmissionSnapshot.generation` identifies the cooperating config generation used - by gather; -- `CommitExpectation { nativeBefore, nativeAfter, txId }` identifies the one native - transition this commit is allowed to perform; -- each prepared target records canonical parent identity plus file identity where - available, not merely a path string; -- the config mutation coordinator stays held through the authoritative re-read and - synchronous commit for cooperating writers; -- after commit, native generation must be exactly `nativeAfter` with this `txId`. - -The catalog phase supplies the target observations and refuses its private commit -when the shared validator rejects them. It does not invent a third counter or a -catalog-specific revision schema. - -The bound is stated narrowly. Cooperating transitions are prevented from committing -a stale candidate. A single-direction target retarget or replacement is detected. -The mechanism does **not** claim to detect an arbitrary parent-symlink A→B→A that -occurs entirely between checks; `005_contract.md` §3 explicitly scopes C17 that -way. Provider inventory changing upstream after a completed gather is also not -filesystem interference; a later convergence may supersede that snapshot. - -WP9 is independently correct before WP11: its synchronous no-await commit prevents -same-process interleaving and rejects generation/identity evidence that changed -before commit. It does not claim cross-process exclusion until WP11 installs the -native lock. The catalog API does not change when that lock lands. - -## Catalog-internal outcomes only - -The prior document published `CodexCatalogRefreshOutcome`, -`CatalogRefreshDisposition`, and a skip-reason union that included future -`desired_off` and `lock_busy`. Those shared versions are deleted. The contract owns -the public result and management projection (`005_contract.md` §2). - -WP9 keeps only facts needed inside the catalog implementation: +1. validate candidate not consumed, config generation, every target identity, and + every source fingerprint; +2. keyed backup create-once replacement; +3. legacy backup create-once replacement when the default path requests it; +4. active catalog replacement; +5. models cache replacement. -```ts -type CatalogGatherOutcome = - | { kind: "prepared"; candidate: CodexCatalogCandidate } - | { kind: "unavailable" } - | { kind: "degraded"; candidate: CodexCatalogCandidate; notices: readonly CatalogGatherNotice[] } - | { kind: "failed"; surface: "provider-auth" | "provider-network"; retryable: boolean }; +Receipt fields change only after the corresponding replacement succeeds. A failure +returns the exact prefix receipt and consumes the candidate; callers must regather. +There is no rollback claim. -export type CodexCatalogCommitResult = - | { kind: "committed"; result: CodexCatalogRefreshResult; writes: CatalogWriteReceipt } - | { kind: "stale"; reason: "generation" | "target-identity" | "candidate-consumed" } - | { kind: "failed"; surface: "disk"; writes: CatalogWriteReceipt }; -``` +Target identity remains strict except for one create-once rule. If a backup target +was absent at gather and is present at commit, commit may mark it `preserved` and +continue only when the target is a safely resolved regular file, readable, and a +valid non-routed catalog backup. It is never overwritten. A symlink, unreadable +file, malformed JSON/catalog, routed-content backup, or ambiguous identity is +`refused` before the first write. This exception applies only to a backup used as a +create-once target, never to a backup whose bytes were selected as a gather source; +selected source fingerprints remain strict. -These types do not cross the `convergence.ts` boundary. `convergeCodex` projects -them into the contract's `ConvergeOutcome` and `CatalogDisposition`; no route -switches on catalog-private variants. Provider names, URLs, token text, paths, -digests, and raw exceptions never enter the public disposition. Partial disk writes -are derived from the receipt and cause a fresh convergence, never a replay of the -candidate. - -## Diff — preparation and fixed writes - -MODIFY `src/codex/catalog/bundled.ts` at current lines 225-234. Loading a fallback -during gather stays in memory: - -```diff - export function loadCatalogForSync(path: string): RawCatalog | null { - return readCatalog(path) - ?? readCatalog(catalogBackupPathFor(path)) - ?? (isDefaultCatalogPath(path) ? readCatalog(legacyCatalogBackupPath()) : null) - ?? readCatalog(activeCodexModelsCachePath()) -- ?? materializeBundledCodexCatalog(path) - ?? loadBundledCodexCatalog(); - } -``` +## C. Catalog-only convergence -Retain `materializeBundledCodexCatalog` for existing explicit callers. Only gather -stops using a materializing fallback. - -MODIFY `src/codex/catalog/sync.ts` at current lines 507-569 and 600-616. Assembly -returns bytes and observations; the writer accepts no config and performs no await: - -```diff --export async function syncCatalogModels(config: OcxConfig): Promise { -+export async function prepareCatalogSync( -+ config: Readonly, -+): Promise { - const catalogPath = readCodexCatalogPath(); - const baseCatalogBytes = readFileOrNull(catalogPath); - // Existing merge logic, provider gather, backup planning, serialization. -- atomicWriteFile(catalogPath, JSON.stringify(catalog, null, 2) + "\n"); -- invalidateCodexModelsCache(); -- return result; -+ return { -+ catalogBytes, -+ cacheBytes, -+ backups, -+ targets: observeCatalogTargetIdentities(catalogPath, cachePath, backups), -+ result, -+ notices, -+ }; - } -+ -+export function writePreparedCatalogCommit( -+ prepared: PreparedCodexCatalogCommit, -+): CatalogWriteReceipt { -+ // Fixed order; set each receipt bit only after atomic replacement returns. -+} -``` +### C1 — consume the WP8b seams -Move `writePreparedCatalogCommit` and every lower-level direct catalog/cache writer -used by convergence into `src/codex/internal/catalog-commit.ts`. -`src/codex/catalog.ts:11` currently -re-exports `syncCatalogModels`; remove that direct writer export after all production -and test imports migrate. The dependency-graph test, not an `rg` spelling guard, -proves no alias, re-export, wrapper, or dynamic import reaches the writers outside -`convergence.ts` (`005_contract.md` §Test plan). +WP9 does not redeclare request, snapshot, projection, or shared result types. +`management-convergence.ts` consumes: -That reachability rule is per domain, not repository-wide. WP9 proves only the -catalog row below; the composed WP13 Scenario A must consume the complete per-domain -table from `005_contract.md:927-960` and must not assert that every writer is -unreachable outside `convergence.ts`: +- `createCatalogConvergeRequest` from + `src/codex/catalog-admission.ts:32-46`; +- `captureCatalogAdmissionSnapshot` from + `src/codex/catalog-admission.ts:84-107`; +- `projectCatalogOnlyOutcome` from its landed owner at + `src/codex/management-convergence.ts:63-75`; +- shared `CatalogDisposition`, `ConvergeOutcome`, and `ConvergeCodex` from + `convergence-types.ts`. -| Domain | Low-level writer owner | Permitted runtime roots | -|---|---|---| -| catalog, hashed/legacy backups, models cache | `src/codex/internal/catalog-commit.ts` | `src/codex/convergence.ts` only | -| history DB rows, manifest, rollout files | history write exports in `src/codex/internal/history-writer.ts` | `src/codex/history-worker.ts` only | -| transition pair and history schedule/terminal row | `src/codex/transition-state.ts` | `src/codex/convergence.ts` and `src/codex/history-worker.ts` only | - -Therefore WP13 Scenario A's repository-wide sentence at -`050_composed_acceptance.md:249` is attributable to WP13 and must be replaced there -with symbol-level assertions against each contract row. A valid history Worker is a -required permitted root, not a writer leak. - -## The first production `convergeCodex` is catalog-scoped for management - -WP8b declared this function as a type only. WP9 now adds a non-placeholder -implementation in `src/codex/convergence.ts`. The management-only factory closes -over the exact object received by `handleManagementAPI`; the internal function makes -that reference's path to capture explicit without adding it to `ConvergeRequest`: - -```diff -+interface ConvergenceContext { -+ readonly catalogConfig?: Readonly; -+} -+ -+async function convergeCodexInContext( -+ request: ConvergeRequest, -+ context: ConvergenceContext, -+): Promise { -+ if (request.scope === "catalog") { -+ if (!context.catalogConfig) { -+ return catalogContextMissingOutcome(); -+ } -+ const admission = captureCatalogAdmissionSnapshot( -+ request, -+ context.catalogConfig, -+ ); -+ const gathered = await gatherCodexCatalogCandidate(admission); -+ const catalog = commitCatalogAgainstCurrentGeneration(admission, gathered); -+ return projectCatalogOnlyOutcome(catalog, { -+ history: "no-change", -+ observed: "no-change-not-evaluated", -+ }); -+ } -+ -+ return coordinateLegacyFullBehavior(request); -+} -+ -+export const convergeCodex: ConvergeCodex = (request) => -+ convergeCodexInContext(request, {}); -+ -+export function createManagementConvergeCodex( -+ config: Readonly, -+): ConvergeCodex { -+ return (request) => convergeCodexInContext(request, { catalogConfig: config }); -+} -+ -+function captureCatalogAdmissionSnapshot( -+ request: ConvergeRequest, -+ config: Readonly, -+): CatalogAdmissionSnapshot { -+ return { -+ config, // exact captured reference; never a global and never a persisted re-read -+ generation: readRequiredConfigGeneration(request), -+ targets: captureCatalogTargets(config), -+ }; -+} -``` +The placeholder factory body at `src/codex/management-convergence.ts:81-96` is +replaced in place. It validates catalog scope without throwing, captures admission, +awaits the write-free gather, executes the synchronous commit, and projects the +result. The lower-level orchestration lives in new `convergence.ts`, so only that +module reaches `internal/catalog-writer.ts`; the retained management module remains +the factory boundary until WP12 consolidates the full funnel. -`catalogContextMissingOutcome` is a typed no-write failure for an accidental naked -`convergeCodex({scope:"catalog", ...})` call. It does not recover by consulting disk -or process state. Production management never reaches it because the bound factory -is the only catalog-scoped construction path. +### C2 — config generation, source fingerprints, and target identity only -`projectCatalogOnlyOutcome` reports the actual catalog/cache/backup result and -synthesizes history and observed fields as no-change/not-evaluated. It never calls -config injection, profile, journal, history, restoration, or WP12 observation. -`coordinateLegacyFullBehavior` is only a typed adapter over the existing full path; -WP9 does not route management or `/api/sync` into it. WP12 replaces that branch with -the authoritative full funnel and rewires the production full callers. This is a -plain reversal of the earlier WP9 design, which had made provider/model edits perform -full native convergence before its safety phases existed. +A `scope:"catalog"` commit does not request `CommitExpectation`, open +`transition-state.ts`, call `beginCodexTransition`, call `assertPublished`, or read +or advance `{nativeGeneration,currentTxId}`. Its `catalog-only` outcome correctly +has no pair fields (`src/codex/convergence-types.ts:207-224`). -## Every management caller uses the funnel +Catalog staleness is guarded by: -Delete `refreshCodexCatalogBestEffort` from -`src/server/management-api.ts:105-112` and -`src/server/management/context.ts:12,68`. Replace it with one injected production -factory plus the bound funnel. The dependency seam takes a factory, rather than an -already-bound function, so tests can assert `configArg === config` at construction: - -```diff -- refreshCodexCatalog?: () => Promise; -+ createManagementConvergeCodex?: ( -+ config: Readonly, -+ ) => ConvergeCodex; - -- refreshCodexCatalogBestEffort: () => Promise; -+ convergeCodex: (request: ConvergeRequest) => Promise; -``` +- the observe-only config generation captured before gather and re-read immediately + before write; +- candidate-bound per-source fingerprints; +- target parent/file identity plus the narrow create-once backup exception. -```diff -- async function refreshCodexCatalogBestEffort(): Promise { -- if (deps.refreshCodexCatalog) return deps.refreshCodexCatalog(); -- try { -- const { refreshCodexModelCatalog } = await import("../codex/refresh"); -- await refreshCodexModelCatalog(config); -- } catch { /* catalog absent */ } -- } -+ let boundConvergeCodex: ConvergeCodex | undefined; -+ async function convergeCodex( -+ request: ConvergeRequest, -+ ): Promise { -+ if (!boundConvergeCodex) { -+ const create = deps.createManagementConvergeCodex -+ ?? (await import("../codex/convergence")).createManagementConvergeCodex; -+ boundConvergeCodex = create(config); -+ } -+ return boundConvergeCodex(request); -+ } -``` +The commit must never import or invoke routing writers. A test fails if catalog-only +work changes `config.toml`, generated profile, journal, transition row, or history. -The lazy bind preserves today's behavior for management requests that never refresh -the catalog: they do not load catalog convergence. On the first catalog mutation, -`src/server/management-api.ts:105-112`'s in-scope `config` object is passed by -identity into the factory, retained in its closure, and handed unchanged to -`captureCatalogAdmissionSnapshot`; gather then receives it as `admission.config`. +### C3 — total, non-throwing management adapter -Each of the 16 current awaits — provider 6 +Delete `refreshCodexCatalogBestEffort` from +`src/server/management-api.ts:105-113` and its context field at +`src/server/management/context.ts:68`. Replace the dependency with a factory seam +for `createManagementConvergeCodex(config)` and expose one context adapter such as +`convergeCodexCatalog(): Promise`. + +That adapter is total. One outer `try/catch` covers request construction, lazy +dynamic import, missing export, factory construction, admission, gather, commit, +projection, and malformed/unexpected outcomes. Expected private results map +directly. The adapter tracks whether commit began and the commit function catches +every expected replacement failure into a receipt, so even an unexpected throw has +a conservative typed projection: + +| Internal condition | `CatalogDisposition` projection | +|---|---| +| gather admission busy | `skipped/busy`, retryable | +| no usable catalog source | `skipped/catalog-unavailable` | +| config/target/source refusal | `skipped/refused` | +| generation, fingerprint, or identity drift | `skipped/stale`, retryable | +| provider auth/network gather failure | matching `failed` reason, `phase:"gather"`, `partialWrite:false` | +| lazy import, missing export, factory, or unexpected pre-commit failure | sanitized `failed/disk`, `phase:"gather"`, `partialWrite:false` | +| expected replacement failure | `failed/disk`, `phase:"commit"`, `partialWrite` derived from the receipt | +| unexpected throw after commit begins | `failed/disk`, `phase:"commit"`, `partialWrite:true` (fail closed) | + +No raw message, provider, token, path, or digest reaches the response. The route +dispatcher may continue to rethrow unrelated errors at +`src/server/management-api.ts:150-163`; no catalog error escapes to it. + +The lazy binding is cached only after factory construction succeeds. A failed lazy +import/factory remains retryable on a later mutation instead of caching a broken +closure. The factory closure itself is also total, including wrong-scope input. + +Each current await becomes one adapter call and appends its returned disposition. +The complete invocation set remains provider 6 (`src/server/management/provider-routes.ts:147,338,487,512,527,546`), model 6 (`src/server/management/model-routes.ts:214,313,352,390,404,440`), combo 2 (`src/server/management/combo-routes.ts:198,216`), and agent settings 2 -(`src/server/management/agent-settings-routes.ts:280,525`) — becomes: - -```diff --const catalogRefresh = await refreshCodexCatalogBestEffort(); -+const outcome = await convergeCodex({ -+ action: "converge", -+ scope: "catalog", -+ reason: "management-mutation", -+ mode: "automatic", -+ deadlineMs: MANAGEMENT_CODEX_CONVERGENCE_DEADLINE_MS, -+}); -+const catalogRefresh = outcomeCatalogDisposition(outcome); -``` +(`src/server/management/agent-settings-routes.ts:280,525`). -`outcomeCatalogDisposition` projects only into the contract-declared -`CatalogDisposition`; it does not define a second management union. Every route -keeps its current 200/201 and the persisted mutation, appends `catalogRefresh`, and -continues unrelated Claude/Desktop work. That is the best-effort behavior promised -by `005_contract.md` §2. +Order is part of compatibility: -## Explicit sync is deliberately not rewired in WP9 +- `/api/v2` keeps its intentional Codex config writes before catalog convergence + (`src/server/management/agent-settings-routes.ts:230-280`); +- combo update keeps save/reconcile/cooldown work, then convergence, then optional + Claude definition sync (`src/server/management/combo-routes.ts:188-200`); +- `/api/subagent-models` keeps save, convergence, Claude sync, Desktop apply, response + (`src/server/management/agent-settings-routes.ts:518-528`). -`src/server/management/config-routes.ts:261-268` continues to call -`syncModelsToCodex(undefined, config, null)` in the WP9 commit. Moving that route to -`convergeCodex` here would require the full admission, observed-state projection, -history safety, and response semantics that WP10-WP12 have not landed. WP12 performs -the real diff to `scope: "full"` plus `toSyncResponse`; until then the current route -status/body behavior remains unchanged. +Therefore “no additional writes” is asserted around the convergence call itself, +not around the whole route. Route tests separately assert the existing primary and +follow-up writes still execute in their original order after every committed, +skipped, refused, failed, lazy-import-failed, and factory-failed disposition. + +## D. WP9 reachability, bounded honestly + +WP9's C14 claim is only that the 16 management mutation sites no longer reach +`refreshCodexCatalogBestEffort` or catalog writers and instead pass through the +catalog-scoped `ConvergeCodex`. It does **not** claim that `convergence.ts` is the +repository's sole catalog writer root yet. + +The symbol-graph test permits these exact legacy roots until WP12: + +| Legacy root | Current path | WP12 removal | +|---|---|---| +| management `POST /api/sync` | `config-routes.ts:261-268` → `sync.ts:83-89` → `refresh.ts:44-51` | rewire to full convergence and `toSyncResponse` | +| server startup cache invalidation | `server/index.ts:403` → `invalidateCodexModelsCache` | route startup through full convergence/observer | +| `ocx sync-cache` | `cli/index.ts:849-855` → `invalidateCodexModelsCache` | route CLI command through full convergence | +| native restore | `codex/inject.ts:764-774` → `restoreCodexCatalog` → `catalog/sync.ts:572-597` | move restore writes behind full convergence/provenance | + +The allowlist is exact by root module and writer symbol, not a directory wildcard. +WP12 owns deleting every row. No new legacy root may be added in WP9. + +`tests/codex-convergence-contract.test.ts` is created with a TypeScript-resolved, +symbol-granular graph: static imports, literal dynamic imports, path aliases, +re-exports, renamed imports, namespace property access, and wrappers all preserve +the writer symbol identity. An unresolved module, unresolved symbol, computed +dynamic import, or non-literal import that could hide a writer fails the test rather +than being skipped. The test publishes the WP9 legacy allowlist as data and proves +all 16 management roots terminate at `convergence.ts` before a catalog writer. ## Tests -### Catalog mechanism - -`tests/codex-refresh.test.ts` replaces the all-in-one dependency tests with: - -1. gather uses `CatalogAdmissionSnapshot.config`, performs provider/parse/assembly work, - and leaves a real isolated-home recursive manifest byte-identical; -2. commit invokes only the fixed writer list; injected provider/parser/subprocess - functions throw if reached beneath the synchronous boundary; -3. disk failure returns the exact partial receipt and consumes the candidate; -4. a second commit writes nothing; -5. a create-once backup appearing after gather is preserved; -6. provider auth/network degradation stays sanitized and projects through - `ConvergeOutcome.catalogRefresh`. - -### Real generation invalidation — C2/C17 - -The old config/content-hash tests are removed. Activation uses the production -generation owners: - -1. admit/gather A; perform a cooperating persisted config transition A→B→A through - the real config mutation API; commit A and assert generation rejection before - every catalog/cache/backup write; -2. gather A; complete another cooperating native transition with its own `txId`; - assert A's `CommitExpectation` is rejected and the newer bytes survive; -3. retarget a canonical parent once between gather and commit; assert target- - identity rejection and zero writes; -4. document, but do not falsely test as guaranteed, a complete parent-symlink - A→B→A between checks — it is outside C17's contract bound; -5. change an unrelated config field through the real config API and assert the - generation still invalidates the candidate. Generation is transition identity, - not semantic-field equality. - -### Production funnel - -- Extend `tests/codex-convergence-contract.test.ts` to walk the TypeScript module - graph (static imports, dynamic imports, aliases, and re-exports) and prove every - direct writer in `src/codex/internal/catalog-commit.ts` is reachable only from - `convergence.ts`. -- Drive all 16 real management routes with an injected convergence factory, assert - its construction argument is reference-equal to the exact config - object passed to `handleManagementAPI`, assert one `scope: "catalog"` call, preserve - each primary 2xx/201, and observe the additive `catalogRefresh`. A gather spy also - asserts `admission.config` is that same reference, proving the entire path rather - than only the factory boundary. -- For every management route, inject spies that fail on config/profile/journal/history - writes and assert zero calls; assert history and observed result sections are the - contract's no-change/not-evaluated projection. -- A refused/deferred catalog attempt must not suppress combo Claude work or agent - settings Claude/Desktop work. -- Drive `POST /api/sync` and assert it still follows the pre-WP9 - `syncModelsToCodex` route behavior. The `toSyncResponse` production proof belongs - to WP12/WP13. +### T1 — gather really performs no filesystem write + +Run admission plus gather in a child process with fresh `mktemp -d` values for +`OPENCODEX_HOME`, `CODEX_HOME`, and any config/runtime home. Capture the recursive +manifest **before calling `captureCatalogAdmissionSnapshot`**, not after admission. +For every entry record relative path, kind, regular-file SHA-256, size, mode, +mtime at nanosecond resolution where available, and symlink target. Compare it +after gather. + +Start a recursive filesystem event journal before admission and stop it after gather; +fail on create/delete/rename/write/metadata events so a temp file created and deleted +within the interval is visible. Prove the harness is non-vacuous with controls that +(a) chmod an existing file and (b) create then delete a temp file; both must fail. +Also inject throw-on-call spies for runtime persistence, OAuth refresh/persist/intent, +ownership registration, generation initialization, backup creation, and atomic +replacement. Reset and separately assert the permitted process-local caches changed +only within their bounded owners. + +Broken mutation that must turn T1 red: replace observe-only runtime resolution with +`resolveAndPersistCodexRuntime`, use `loadAuthStore`, or use +`readConfigGeneration`; the manifest/event journal or write spy detects the mkdir, +chmod, backup, SQLite, ownership, or temp-file activity. + +### T2 — fingerprints and identity reject before write + +Table-drive active catalog, selected hashed backup, selected legacy backup, selected +models-cache fallback, runtime-state source, and auth-store source. Gather at config +generation N, truncate and rewrite the selected source **in place** so dev/inode and +generation remain the same, then commit. Expect `stale` and byte-identical targets. +Make each source unreadable/ambiguous and expect `refused` with zero writes. Change +config through the real cooperating mutation API and expect generation rejection. +Retarget one parent and expect target-identity rejection. + +Document but do not claim detection for content A→B→A returning exact A before the +comparison, parent A→B→A entirely between checks, or a write after the comparison. + +Broken mutation that must turn T2 red: remove fingerprint comparison while retaining +generation and inode checks; the same-generation same-inode rewrite would commit and +the old candidate bytes would replace the newer source-derived state. + +### T3 — exact four-step receipt and bytes + +Inject failure immediately before each replacement and assert both receipt and real +target bytes: + +| Failure before | Expected completed prefix | Required byte state | +|---|---|---| +| keyed backup | none | all four targets retain pre-image | +| legacy backup | keyed only | keyed has candidate bytes; legacy/catalog/cache retain pre-image | +| catalog | keyed + legacy | both backups have candidate bytes; catalog/cache retain pre-image | +| cache | keyed + legacy + catalog | backups/catalog have candidate bytes; cache retains pre-image | + +Every row also proves the candidate is consumed and a second commit writes nothing. +Repeat backup absent→present with a valid non-routed backup and expect `preserved`; +repeat with malformed, unreadable, routed, symlinked, and ambiguous appearing backups +and expect refusal before any replacement. + +Broken mutation that must turn T3 red: set a receipt bit before replacement or catch +a failed replacement and continue; receipt and actual bytes diverge in at least one +table row. + +### T4 — total adapter and route ordering + +Drive all 16 real routes with a factory spy and assert reference equality with the +exact config passed to `handleManagementAPI`, one fixed request created by +`createCatalogConvergeRequest`, original 2xx/201, and additive +`catalogRefresh`. Table-drive lazy-import rejection, missing export, throwing factory, +generation unavailable, gather auth/network failure, stale/refused commit, disk +failure before each replacement, and malformed result. None may throw or skip later +work. + +Scope the routing-write spies to the convergence call itself. Separately record route +events and assert the original order for `/api/v2`, combo update, and +`/api/subagent-models`, including Claude/Desktop follow-ups after every catalog +disposition. + +Broken mutation that must turn T4 red: remove the adapter's outer catch; a lazy-import +or snapshot error reaches the dispatcher, changes the persisted-success route to 500, +and the event log lacks later Claude/Desktop calls. + +### T5 — lazy loading and reachability + +For laziness, launch a child process that registers a Bun module-load sentinel for +the canonical `src/codex/management-convergence.ts`, imports management API, and +drives only non-refresh routes. The sentinel must remain zero; a second child drives +one catalog mutation and observes one initialization. A route-level “zero calls” spy +alone is not accepted because an eager static import would pass it. + +For reachability, run the symbol graph described in D. It must accept only the four +legacy rows and reject aliases, re-exports, wrappers, or dynamic imports from any new +root. + +Broken mutations that must turn T5 red: add a static top-level management-convergence +import, alias a catalog writer into a management route, or replace a literal import +with a computed dynamic import. The sentinel or fail-closed graph must reject each. + +### T6 — precedence and native-pair exclusion + +On the default catalog path, provide a bundled template that differs visibly from +catalog/backup/cache plus an on-disk routed/user-native row. Gather must use bundled +native template fields and preserve the on-disk merge row. Assert no materialized +fallback write. Snapshot the transition row before and after committed, stale, +refused, and failed catalog-only attempts; it must be byte/field identical and no +history schedule appears. Routing artifact spies stay zero. + +Broken mutations that must turn T6 red: move disk fallbacks ahead of bundled lookup, +request `CommitExpectation`, or call a routing writer. Template assertions, transition +row equality, or routing spies fail. ## Verification @@ -627,7 +490,7 @@ Static/focused gates for the WP9 commit: ```bash bun test tests/codex-refresh.test.ts tests/codex-convergence-contract.test.ts -bun test tests/codex-sync-api.test.ts tests/codex-models-cache-invalidate.test.ts +bun test tests/codex-config-generation.test.ts tests/codex-sync-api.test.ts tests/codex-models-cache-invalidate.test.ts bun test tests/model-visibility-management-api.test.ts tests/management-provider-validation.test.ts tests/combo-management-api.test.ts tests/codex-v2-gate.test.ts bun run typecheck bun run test @@ -635,31 +498,17 @@ bun run privacy:scan bun --cwd docs-site run build ``` -Runtime proof uses temporary `OPENCODEX_HOME`/`CODEX_HOME` and port `0`. It never -starts, stops, syncs, restores, or ensures the live proxy on port 10100. - -1. Gather only; compare full before/after manifests. -2. Fire the real config generation A→B→A transition; observe zero native writes. -3. Regather and converge; parse catalog and cache bytes and verify the cache models - match the committed catalog. -4. Drive one best-effort management mutation through the real server boundary; - observe its primary 2xx and contract disposition. -5. Drive explicit sync and prove WP9 left its current route behavior unchanged. +Runtime probes use temporary homes and child processes only. They do not invoke +`ocx start`, `stop`, `sync`, `restore`, `ensure`, any `ocx service` command, or the +live proxy on port 10100. ## Accept criteria -- **C1** — gather is write-free and commit is synchronous/fixed. Catalog failures - remain catalog-private until projected through `ConvergeOutcome`; all 16 callers - preserve their primary success behavior and expose the contract disposition. - Their request is `scope: "catalog"`; config/profile/journal/history write spies stay - at zero, and history/observed fields are no-change/not-evaluated. -- **C2 / C17 (contract-scoped)** — real config/native generation changes and - single-direction target-identity drift reject before write. No content hash or - path string is presented as arbitrary filesystem ABA protection. -- **C14** — the production module graph proves all 16 management callers funnel - through `convergeCodex`, and no other importer reaches the direct catalog writers. -- **N2** — the WP9 commit contains the first working `convergeCodex`, rewires its - callers in that same commit, passes typecheck, and preserves current behavior. - WP8b already supplies the concrete catalog snapshot/request/projection and both - generation owners; WP9 calls no WP12 admission or observer placeholder. WP12 is - explicitly the phase that replaces the legacy full branch and rewires full callers. +| Criterion | Proof | Concrete broken mutation that makes it red | +|---|---|---| +| **C1** — gather is filesystem-write-free; commit is synchronous, fixed, one-shot, and receipt-exact | T1 + T3 | call a persisting resolver/read, add an `await` beneath commit, reorder replacements, pre-set a receipt bit, or replay a consumed candidate | +| **C2/C17** — config generation, every consumed source fingerprint, and target identity reject stale work before write, with only the valid create-once backup exception | T2 + T3 | in-place rewrite the same inode at the same generation, omit one selected fallback fingerprint, accept unreadable source, or overwrite an appearing backup | +| **Catalog/native boundary** — catalog-only never reads/advances the native pair or writes routing/history artifacts | T6 | call `expectation()`/`beginTransition`, add pair fields to `catalog-only`, or invoke config/profile/journal/history writer | +| **Best-effort compatibility** — all 16 primary writes retain 2xx/201 and original follow-up order for every catalog failure | T4 | let lazy import/factory/admission throw, scope “zero writes” to the whole route, or return before Claude/Desktop follow-up | +| **C14, WP9-bounded** — the 16 management roots reach catalog writers only through convergence; exactly four documented legacy roots remain until WP12 | T5 symbol graph | add a fifth root, hide one through alias/re-export/computed import, or accidentally require WP12 to have already removed `/api/sync`/startup/CLI/restore | +| **N2** — WP9 replaces the landed placeholder, consumes existing request/snapshot/projection seams, creates the contract test, and typechecks without WP10-WP12 | focused tests + typecheck | redefine a WP8b type/helper, refer to a nonexistent later helper, leave a throwing placeholder, or claim the absent test file is merely extended | From db3d69ed05bba5c2dd822f2c2088c186adf5a105 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Tue, 4 Aug 2026 19:15:45 +0900 Subject: [PATCH 053/163] test(codex): assert the property, not the example that happened to be chosen MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A reviewer weakened this substrate thirteen different ways just outside the literals these tests name, and every one stayed green. The tests were checking that `fast-chat` is residue, that `"sideways"` is refused, that a tab is whitespace — not that the rules behind those examples hold. Each area is now driven from a corpus instead of one value. Catalog paths cover root-relative, nested-relative, absolute inside and outside CODEX_HOME, and `../` escapes, with filenames that do not contain the word "catalog", so a predicate that keys on the substring fails. `model_catalog_json` is refused as number, boolean, array, inline table and datetime, not just as a number. The bare-alias corpus comes from production `exactComboCatalogSlugs` rather than a name someone typed. Whitespace enumerates every code point JavaScript `trim()` removes, which is what caught a carriage return being accepted. The CAS null-txId rule is proven at three generations, direction against five invalid strings, and opacity now walks the whole prototype chain, since an inherited getter reached the database while the own-key assertion passed. Identity varies the working directory too. Every one of the thirteen was applied to a scratch copy and shown red first. That is the only reason to believe this round is different from the last one. --- tests/codex-native-residue.test.ts | 226 ++++++++++++++++++++++----- tests/codex-transition-state.test.ts | 114 ++++++++------ tests/codex-user-identity.test.ts | 9 +- 3 files changed, 260 insertions(+), 89 deletions(-) diff --git a/tests/codex-native-residue.test.ts b/tests/codex-native-residue.test.ts index 92ce332ea..044b2dfd6 100644 --- a/tests/codex-native-residue.test.ts +++ b/tests/codex-native-residue.test.ts @@ -1,6 +1,8 @@ import { afterEach, beforeEach, expect, test } from "bun:test"; import { createHash } from "node:crypto"; import { + chmodSync, + lstatSync, mkdirSync, mkdtempSync, realpathSync, @@ -9,12 +11,13 @@ import { writeFileSync, } from "node:fs"; import { tmpdir } from "node:os"; -import { dirname, join, resolve } from "node:path"; +import { basename, dirname, join, resolve } from "node:path"; import { Database } from "bun:sqlite"; import { buildCatalogEntries, + exactComboCatalogSlugs, readCodexCatalogPath, syncCatalogModels, } from "../src/codex/catalog"; @@ -295,6 +298,57 @@ for (const shape of catalogConfigShapes) { }); } +const catalogPathShapes: Array<{ + name: string; + configuredPath: (outsideRoot: string) => string; +}> = [ + { name: "root-relative", configuredPath: () => "custom.json" }, + { name: "nested-relative", configuredPath: () => "nested/routes.json" }, + { + name: "absolute inside CODEX_HOME", + configuredPath: () => canonicalPathInCodexHome("absolute-direct.json"), + }, + { + name: "absolute outside CODEX_HOME", + configuredPath: outsideRoot => join(outsideRoot, "external.json"), + }, + { + name: "parent-escaping relative", + configuredPath: () => `../${basename(codexHome)}-escape.json`, + }, +]; + +for (const shape of catalogPathShapes) { + test(`configured catalog classification follows the ${shape.name} path`, () => { + const outsideRoot = mkdtempSync(join(tmpdir(), "ocx-native-residue-catalog-outside-")); + const configuredPath = shape.configuredPath(outsideRoot); + const targetPath = resolve(realpathSync.native(codexHome), configuredPath); + try { + mkdirSync(dirname(targetPath), { recursive: true }); + writeFileSync( + pathInCodexHome("config.toml"), + `model_catalog_json = ${JSON.stringify(configuredPath)}\n`, + ); + + expect(classifyNativeRoutedResidue(), "configured absence must fail closed").toMatchObject({ + kind: "indeterminate", + surface: "catalog", + path: targetPath, + }); + + writeFileSync(targetPath, routedCatalog()); + expect(classifyNativeRoutedResidue(), "routed target must be detected").toEqual({ + kind: "residue", + surface: "catalog", + path: targetPath, + }); + } finally { + rmSync(targetPath, { force: true }); + rmSync(outsideRoot, { recursive: true, force: true }); + } + }); +} + test("an atomic-write artifact beside the configured catalog is indeterminate", () => { const catalogPath = canonicalPathInCodexHome("nested/custom-catalog.json"); const artifactPath = `${catalogPath}.ocx.42.7.tmp`; @@ -310,6 +364,21 @@ test("an atomic-write artifact beside the configured catalog is indeterminate", }); }); +test("an atomic-write artifact is found before its configured target exists", () => { + const catalogPath = canonicalPathInCodexHome("nested/pending.json"); + const artifactPath = `${catalogPath}.ocx.42.7.tmp`; + mkdirSync(dirname(catalogPath), { recursive: true }); + writeFileSync(pathInCodexHome("config.toml"), 'model_catalog_json = "nested/pending.json"\n'); + writeFileSync(artifactPath, "partial"); + + expect(lstatSync(catalogPath, { throwIfNoEntry: false })).toBeUndefined(); + expect(classifyNativeRoutedResidue()).toMatchObject({ + kind: "indeterminate", + surface: "partial-write", + path: artifactPath, + }); +}); + test("a configured catalog target that is not a readable regular file is indeterminate", () => { const catalogPath = canonicalPathInCodexHome("nested/custom-catalog.json"); mkdirSync(catalogPath, { recursive: true }); @@ -322,6 +391,24 @@ test("a configured catalog target that is not a readable regular file is indeter }); }); +const permissionTest = process.platform === "win32" ? test.skip : test; +permissionTest("EACCES on a configured regular catalog file is indeterminate", () => { + const catalogPath = canonicalPathInCodexHome("permission-target.json"); + writeFileSync(pathInCodexHome("config.toml"), 'model_catalog_json = "permission-target.json"\n'); + writeFileSync(catalogPath, routedCatalog()); + expect(lstatSync(catalogPath).isFile()).toBe(true); + chmodSync(catalogPath, 0o000); + try { + expect(classifyNativeRoutedResidue()).toMatchObject({ + kind: "indeterminate", + surface: "catalog", + path: catalogPath, + }); + } finally { + chmodSync(catalogPath, 0o600); + } +}); + test("an absent configured catalog target is indeterminate", () => { const catalogPath = canonicalPathInCodexHome("nested/missing-catalog.json"); mkdirSync(pathInCodexHome("nested")); @@ -348,14 +435,50 @@ test("the default catalog is still inspected when a custom catalog is configured }); }); -test("a non-string configured catalog path is indeterminate", () => { - writeFileSync(pathInCodexHome("config.toml"), "model_catalog_json = 42\n"); - - expect(classifyNativeRoutedResidue()).toMatchObject({ - kind: "indeterminate", - surface: "config", - path: canonicalPathInCodexHome("config.toml"), +for (const location of ["inside", "outside"] as const) { + test(`the default catalog remains inspected with an absolute ${location} configured path`, () => { + const outsideRoot = mkdtempSync(join(tmpdir(), "ocx-native-residue-default-outside-")); + const configuredPath = location === "inside" + ? canonicalPathInCodexHome("absolute-custom.json") + : join(outsideRoot, "absolute-custom.json"); + const defaultCatalogPath = canonicalPathInCodexHome("opencodex-catalog.json"); + try { + writeFileSync( + pathInCodexHome("config.toml"), + `model_catalog_json = ${JSON.stringify(configuredPath)}\n`, + ); + writeFileSync(configuredPath, JSON.stringify({ models: [] })); + writeFileSync(defaultCatalogPath, routedCatalog()); + + expect(classifyNativeRoutedResidue()).toMatchObject({ + kind: "residue", + surface: "catalog", + path: defaultCatalogPath, + }); + } finally { + rmSync(configuredPath, { force: true }); + rmSync(outsideRoot, { recursive: true, force: true }); + } }); +} + +test("every non-string TOML type for model_catalog_json is indeterminate", () => { + const nonStringTomlValues = [ + ["number", "42"], + ["boolean", "true"], + ["array", '["custom.json"]'], + ["inline table", '{ path = "custom.json" }'], + ["datetime", "1979-05-27T07:32:00Z"], + ] as const; + + for (const [type, value] of nonStringTomlValues) { + writeFileSync(pathInCodexHome("config.toml"), `model_catalog_json = ${value}\n`); + expect(classifyNativeRoutedResidue(), type).toMatchObject({ + kind: "indeterminate", + surface: "config", + path: canonicalPathInCodexHome("config.toml"), + }); + } }); test("duplicate configured catalog paths are indeterminate", () => { @@ -372,44 +495,65 @@ test("duplicate configured catalog paths are indeterminate", () => { }); }); -test("a bare routed combo alias in the default catalog refuses coordinator initialization", () => { - const models = buildCatalogEntries( - null, - [], - [{ provider: "combo", id: "quick", alias: "fast-chat", owned_by: "combo" }], - undefined, - false, - "default", - new Set(["fast-chat"]), - ); - const combo = models.find(model => model.slug === "fast-chat"); - expect(combo).toMatchObject({ - slug: "fast-chat", - description: "Routed via opencodex → combo (combo).", - owned_by: "combo", - }); - writeFileSync(pathInCodexHome("opencodex-catalog.json"), JSON.stringify({ models: [combo] })); +const comboAliasConfig = { + combos: { + deepseek: { alias: "deepseek-v4-flash", targets: [{ provider: "fixture", model: "one" }] }, + old: { alias: "old-public", targets: [{ provider: "fixture", model: "two" }] }, + stable: { alias: "stable-public", targets: [{ provider: "fixture", model: "three" }] }, + }, +}; +const productionBareComboAliases = [...exactComboCatalogSlugs(comboAliasConfig)] + .filter(alias => !alias.includes("/")); + +for (const alias of productionBareComboAliases) { + test(`production-derived bare combo alias ${alias} is routed residue`, () => { + const [id] = Object.entries(comboAliasConfig.combos) + .find(([, combo]) => combo.alias === alias)!; + const models = buildCatalogEntries( + null, + [], + [{ provider: "combo", id, alias, owned_by: "combo" }], + undefined, + false, + "default", + exactComboCatalogSlugs(comboAliasConfig), + ); + const combo = models.find(model => model.slug === alias); + expect(combo).toMatchObject({ + slug: alias, + description: "Routed via opencodex → combo (combo).", + owned_by: "combo", + }); + writeFileSync(pathInCodexHome("opencodex-catalog.json"), JSON.stringify({ models: [combo] })); - expect(classifyNativeRoutedResidue()).toMatchObject({ - kind: "residue", - surface: "catalog", - }); - expect(readCodexTransitionState()).toEqual({ - kind: "legacy-ambiguous", - message: "A missing coordinator row cannot be initialized while native Codex routing residue exists.", + expect(classifyNativeRoutedResidue()).toMatchObject({ + kind: "residue", + surface: "catalog", + }); + expect(readCodexTransitionState()).toEqual({ + kind: "legacy-ambiguous", + message: "A missing coordinator row cannot be initialized while native Codex routing residue exists.", + }); }); -}); +} -test("a slash-bearing catalog slug without OpenCodex authorship is indeterminate", () => { - writeFileSync(pathInCodexHome("opencodex-catalog.json"), JSON.stringify({ - models: [{ slug: "user/model", description: "User-authored catalog row" }], - })); +for (const slug of [ + "user/model", + "vendor/deepseek-v4-flash", + "local/vision_2", + "acme/text-pro", +]) { + test(`foreign slash-bearing slug ${slug} is indeterminate`, () => { + writeFileSync(pathInCodexHome("opencodex-catalog.json"), JSON.stringify({ + models: [{ slug, description: "User-authored catalog row" }], + })); - expect(classifyNativeRoutedResidue()).toMatchObject({ - kind: "indeterminate", - surface: "catalog", + expect(classifyNativeRoutedResidue()).toMatchObject({ + kind: "indeterminate", + surface: "catalog", + }); }); -}); +} test("a native-tagged history row with routed latest rollout metadata refuses coordinator initialization", () => { createHistoryDatabase("openai", ["openai", "opencodex"]); diff --git a/tests/codex-transition-state.test.ts b/tests/codex-transition-state.test.ts index c2938e293..7c129b4ec 100644 --- a/tests/codex-transition-state.test.ts +++ b/tests/codex-transition-state.test.ts @@ -189,25 +189,30 @@ test("a native CAS with a matching generation but the wrong txId still conflicts }); }); -test("a native CAS expecting a positive generation with a null txId still conflicts", () => { - expect(beginCodexTransition( - { nativeGeneration: 0, currentTxId: null }, - transition("tx-current"), - ).kind).toBe("updated"); +for (const generation of [1, 2, 4]) { + test(`a native CAS at generation ${generation} never treats a null txId as a wildcard`, () => { + let currentTxId: string | null = null; + for (let nextGeneration = 1; nextGeneration <= generation; nextGeneration++) { + const nextTxId = `tx-current-${nextGeneration}`; + expect(beginCodexTransition( + { nativeGeneration: nextGeneration - 1, currentTxId }, + transition(nextTxId), + ).kind).toBe("updated"); + currentTxId = nextTxId; + } - // The generation agrees, but null is not a wildcard for the txId half of - // the pair. Weakening the predicate for a null expectation makes this win. - const nullTxId = beginCodexTransition( - { nativeGeneration: 1, currentTxId: null }, - transition("tx-forged"), - ); - expect(nullTxId.kind).toBe("conflict"); + const nullTxId = beginCodexTransition( + { nativeGeneration: generation, currentTxId: null }, + transition("tx-forged"), + ); + expect(nullTxId.kind).toBe("conflict"); - expect(readCodexTransitionState()).toMatchObject({ - kind: "ready", - state: { nativeGeneration: 1, currentTxId: "tx-current" }, + expect(readCodexTransitionState()).toMatchObject({ + kind: "ready", + state: { nativeGeneration: generation, currentTxId }, + }); }); -}); +} test("a native CAS with a matching txId but the wrong generation still conflicts", () => { expect(beginCodexTransition( @@ -285,27 +290,29 @@ test("the row validator refuses a malformed row the CHECK constraints never saw" expect(readCodexTransitionState()).toEqual({ kind: "unavailable", reason: "database" }); }); -test("the row validator refuses an unknown non-null history direction", () => { - expect(beginCodexTransition( - { nativeGeneration: 0, currentTxId: null }, - transition("tx-unknown-direction"), - ).kind).toBe("updated"); +for (const direction of ["sideways", "reverse", "forward", "APPLY", ""] as const) { + test(`the row validator refuses unknown history direction ${JSON.stringify(direction)}`, () => { + expect(beginCodexTransition( + { nativeGeneration: 0, currentTxId: null }, + transition("tx-unknown-direction"), + ).kind).toBe("updated"); - const database = new Database(coordinatorPath); - try { - database.exec("PRAGMA ignore_check_constraints = ON"); - database.run( - "UPDATE codex_transition_state SET history_direction = 'sideways' WHERE singleton = 1", - ); - expect(database.query<{ history_direction: string }, []>( - "SELECT history_direction FROM codex_transition_state WHERE singleton = 1", - ).get()?.history_direction).toBe("sideways"); - } finally { - database.close(); - } + const database = new Database(coordinatorPath); + try { + database.exec("PRAGMA ignore_check_constraints = ON"); + database.query( + "UPDATE codex_transition_state SET history_direction = ? WHERE singleton = 1", + ).run(direction); + expect(database.query<{ history_direction: string }, []>( + "SELECT history_direction FROM codex_transition_state WHERE singleton = 1", + ).get()?.history_direction).toBe(direction); + } finally { + database.close(); + } - expect(readCodexTransitionState()).toEqual({ kind: "unavailable", reason: "database" }); -}); + expect(readCodexTransitionState()).toEqual({ kind: "unavailable", reason: "database" }); + }); +} test("the row validator refuses every whitespace-only txId", () => { expect(beginCodexTransition( @@ -313,17 +320,19 @@ test("the row validator refuses every whitespace-only txId", () => { transition("tx-blank"), ).kind).toBe("updated"); - const blankTxIds = [ - ["ASCII spaces", " "], - ["tab", "\t"], - ["newline", "\n"], - ["vertical tab", "\v"], - ["form feed", "\f"], - ["non-breaking space", "\u00a0"], - ["em space", "\u2003"], - ] as const; - - for (const [label, txId] of blankTxIds) { + const trimRemovedCodePoints: Array<[string, string]> = []; + for (let codePoint = 0; codePoint <= 0x10ffff; codePoint++) { + const character = String.fromCodePoint(codePoint); + if (character.trim() === "") { + trimRemovedCodePoints.push([ + `U+${codePoint.toString(16).toUpperCase().padStart(4, "0")}`, + character, + ]); + } + } + expect(trimRemovedCodePoints.some(([, value]) => value === "\r")).toBe(true); + + for (const [label, txId] of trimRemovedCodePoints) { const database = new Database(coordinatorPath); try { database.exec("PRAGMA ignore_check_constraints = ON"); @@ -394,6 +403,19 @@ test("the opaque capability never exposes a reachable database handle", () => { configurable: true, }); + let prototype: object | null = controller.capability; + while (prototype !== null) { + for (const key of Reflect.ownKeys(prototype)) { + const descriptor = Reflect.getOwnPropertyDescriptor(prototype, key)!; + const intrinsicLegacyProtoAccessor = prototype === Object.prototype && key === "__proto__"; + if (!intrinsicLegacyProtoAccessor) { + expect(descriptor.get, `getter ${String(key)} on capability prototype chain`).toBeUndefined(); + expect(descriptor.set, `setter ${String(key)} on capability prototype chain`).toBeUndefined(); + } + } + prototype = Reflect.getPrototypeOf(prototype); + } + const reachable = new Set(); const walk = (value: unknown, depth: number): void => { if (depth > 4 || value === null || reachable.has(value)) return; diff --git a/tests/codex-user-identity.test.ts b/tests/codex-user-identity.test.ts index 878a9413e..f55a2749d 100644 --- a/tests/codex-user-identity.test.ts +++ b/tests/codex-user-identity.test.ts @@ -34,8 +34,12 @@ interface IdentityProbeResult { databasePath: string; } -async function runIdentityProbe(env: Record): Promise { +async function runIdentityProbe( + env: Record, + cwd: string, +): Promise { const child = Bun.spawn([process.execPath, "--eval", identityProbe], { + cwd, env: { ...process.env, ...env }, stdin: "ignore", stdout: "pipe", @@ -111,6 +115,7 @@ test("real processes resolve one identity and coordinator path across every home temp: join(root, "temp"), codexHome: join(root, "ambient-codex"), opencodexHome: join(root, "ambient-opencodex"), + workingDirectory: join(root, "working-directory"), }; for (const path of Object.values(paths)) mkdirSync(path, { recursive: true }); return { root, paths }; @@ -145,7 +150,7 @@ test("real processes resolve one identity and coordinator path across every home OPENCODEX_HOME: paths.opencodexHome, OCX_TEST_CANONICAL_CODEX_HOME: canonicalHome, ...accountEnvironment, - }); + }, paths.workingDirectory); })); const osIdentity = resolveEffectiveUserIdentity(); From e8674d630394eeb85dc623568bc7acd59140cf46 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Tue, 4 Aug 2026 19:43:05 +0900 Subject: [PATCH 054/163] docs(substrate): a source that was absent is still evidence MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Round 2 of the WP9 audit found this unit's signature defect reproduced inside the fix for it. The fingerprint design could only describe a source that existed — `{canonicalPath, sha256}` — so a source that was checked and found ABSENT left no evidence at all. That is not a gap in coverage, it is the selection input: `readCodexCatalogPath` picks the default catalog precisely because `config.toml` is missing (catalog/parsing.ts:167-175). Let that file appear between gather and commit carrying a `model_catalog_json` pointing elsewhere and every present-file digest still matches, so the commit writes an obsolete target and truthfully reports `committed` while Codex reads another file entirely. Sources are now a closed role-bearing observation union that records present AND absent states, `config.toml` is a required `catalog-target-selection` observation, and a change in either direction is stale. Roles are enumerated so an omitted observation is structurally detectable, since an untyped list of digests cannot prove its own completeness. Two more promises turned out to be unbacked. The contract said cooperating config writers are PREVENTED because the mutation lock is held through the re-read and the commit, but nothing said how a catalog commit acquires it, so a cooperating writer could land generation N+1 between validation and the write. `withExpectedConfigGenerationSync` validates on the already-held transaction and runs the synchronous commit before releasing it — wrapping the existing observer cannot work, because a second connection contends with the open transaction. And create-once backups were published with `atomicWriteFile`, which ends in an overwriting rename (config.ts:203); a check-then-write loses to a process that creates the file in between, so publication is now exclusive no-clobber with EEXIST preserving the winner. The write-free claim was false and is now true by construction rather than by wording. Gather reached `resolveCodexRuntime`, whose probe creates and deletes a sandbox (runtime.ts:251,277) because a real Codex CLI writes under CODEX_HOME even for `--version` (:244). A before/after manifest could never have caught it — the directory is gone before the manifest is taken. Gather now consumes cached or persisted observations with no subprocess and no scratch, and the evidence is an event journal captured from before admission so a created-then-deleted file is still visible. Also versioned the writer inventory into WP9-transitional and WP12-final roots: one contract test cannot enforce both a table that forbids the legacy sync/startup/restore writers and a phase that legitimately leaves them in place. Catalog admission is likewise split from full admission, so WP9's retained management config reference stops contradicting WP12's persisted read. --- .../005_contract.md | 390 +++++++++++---- .../010_catalog_seam.md | 447 ++++++++++++------ 2 files changed, 601 insertions(+), 236 deletions(-) diff --git a/devlog/_plan/260804_codex_write_substrate/005_contract.md b/devlog/_plan/260804_codex_write_substrate/005_contract.md index 9d0a91c7b..7107add3c 100644 --- a/devlog/_plan/260804_codex_write_substrate/005_contract.md +++ b/devlog/_plan/260804_codex_write_substrate/005_contract.md @@ -568,8 +568,17 @@ export type ConfigGenerationBump = | { kind: "conflict"; current: ConfigGeneration } | { kind: "unavailable"; reason: "busy" | "database" }; +export type ExpectedConfigGenerationSyncResult = + | { kind: "matched"; generation: ConfigGeneration; value: T } + | { kind: "conflict"; current: ConfigGeneration } + | { kind: "unavailable"; reason: "busy" | "database" }; + export type ReadConfigGeneration = () => ConfigGenerationRead; export type BumpConfigGeneration = (expected: ConfigGeneration) => ConfigGenerationBump; +export type WithExpectedConfigGenerationSync = ( + expected: ConfigGeneration, + commit: () => T, +) => ExpectedConfigGenerationSyncResult; ``` Round 2 #6: the previous version said "two counters, both in the record" and @@ -599,12 +608,13 @@ outcomes carry both (`src/codex/convergence-types.ts:207-224`). That is an honest reduction in protection: a catalog-only commit is not guarded against staleness by the native pair. Its independent protection is the per-source -fingerprint check below. A catalog-only commit must never write a routing artifact; +observation check below. A catalog-only commit must never write a routing artifact; if a future phase needs to write one, it uses `scope:"full"` and publishes the native transition plus its truthful history schedule. -WP8b adds executable `readConfigGeneration` and `bumpConfigGeneration` exports to -`src/config.ts` with the callable types above. They use a singleton +WP8b adds executable `readConfigGeneration`, `bumpConfigGeneration`, and +`withExpectedConfigGenerationSync` exports to `src/config.ts` with the callable +types above. They use a singleton `config_generation(singleton INTEGER PRIMARY KEY CHECK(singleton=1), value INTEGER NOT NULL CHECK(value>=0))` row in the existing `config-mutation.sqlite`. Creation and `INSERT OR IGNORE (1,0)` happen under that database's `BEGIN IMMEDIATE`. @@ -616,6 +626,24 @@ commit calls the bump before committing the SQLite transaction; unchanged mutati do not bump. This closes the former scope hole: WP9 delegates this owner to WP8b, and `src/config.ts` is now explicitly IN. +The second seam audit caught a lock-shaped hole in that API. Reading generation N, +then later calling a synchronous catalog writer does not prevent a cooperating config +writer from committing N+1 in between. `withExpectedConfigGenerationSync(expected, +commit)` is the owner-side guard for that interval. It enters the **existing** config +mutation transaction, validates `expected` with the already-held SQLite `Database` +handle, runs `commit` synchronously on a match, and releases the transaction only +after `commit` returns. Conflict returns the current generation without invoking the +callback. Acquisition/open failure returns `unavailable`. + +The implementation must call `readConfigGenerationInTransaction` (or its private +equivalent) on `configMutationDatabase`; it must not call `readConfigGeneration`, +`readConfigGenerationAtPath`, or any helper that opens a second SQLite connection. +The active transaction already owns the write lock, so a second connection would +contend with its caller instead of validating it. WP9 catalog commit is the first +consumer. This guard uses the config mutation lock that exists in WP8b and has no +dependency on WP11's future native lock. Catalog-only work validates but does not +bump the config generation because it writes no persisted OpenCodex config bytes. + ### The expected transition ```ts @@ -651,20 +679,27 @@ record with native residue or an invalid/ambiguous record refuses automatic deletion. On restart, observation compares every artifact to the ledger/current pair, records the unresolved surfaces, and schedules a fresh current transition; idempotence is required but is not described as filesystem atomicity. Catalog-only -staleness is instead admitted by the source fingerprints below, not retroactively +staleness is instead admitted by the source observations below, not retroactively described as protection by a pair it never advanced. -### Prevention for cooperating writers (round 2 #5) +### Prevention for cooperating writers (round 2 #5, seam audit round 2) C2 says a stale candidate **cannot be committed**. Detect-after-commit permits -exactly the write C2 forbids, and `030` already allows the fix: the native lock -may hold the config mutation lock across the synchronous re-read and commit. +exactly the write C2 forbids. Full routing work later follows `030`'s N -> C lock +order. WP9 catalog-only work must not acquire that future native lock, so its fix is +the owner-side config-generation guard that exists independently of WP11. + +The first amendment named the lock but not an API that held it through catalog +publication. The auditor's N -> N+1 interleaving was therefore a cooperating writer +the text promised to prevent. Catalog commit now enters through +`withExpectedConfigGenerationSync`: generation validation and the complete +synchronous catalog commit execute inside one already-held config transaction. So: | Writer | Mechanism | |---|---| -| cooperating (ours) | **prevented** — config lock held through re-read and commit | +| cooperating (ours) | **prevented** — `withExpectedConfigGenerationSync` holds the existing config transaction through validation and synchronous commit | | non-cooperating (hand edit, foreign tool) | **detected** after the fact, reported `deferred` | Re-gather is bounded by `deadlineMs`. On expiry the outcome is `deferred` with a @@ -676,7 +711,7 @@ a deadline, not on hope (round 1 #5's missing termination rule). A candidate records the canonical parent directory and the file identity (dev+inode where available) of each target, not the textual path — a parent symlink can retarget while the path string is unchanged, and `atomicWriteFile` -resolves the effective target only at commit (`src/config.ts:190-199`). +resolves the effective target only at commit (`src/config.ts:199-209`). The WP9 seam auditor then demonstrated the missing content dimension by gathering a candidate, truncating and rewriting the catalog in place, and committing the @@ -684,36 +719,69 @@ stale candidate. Path, canonical parent, parent identity, file identity, config generation, and native pair all remained unchanged. Target identity says where a write will land; it does not say that the bytes gather consumed are still current. -The catalog admission snapshot therefore also retains a SHA-256 fingerprint of -the **exact byte buffer gather actually read** for every source that influenced -the prepared output: the active catalog, whichever hashed/legacy backup or models -cache was selected as a fallback, and any later file source whose bytes influence -that candidate. The gather reader computes the digest from the same buffer it -returns and records the source's canonical path; a separate pre-read is not -equivalent. Immediately before the first commit write, commit re-reads every -recorded source and compares its digest. Any mismatch is `stale`. An unreadable -source, an unresolvable canonical source, or ambiguous source identity is refused -rather than assumed unchanged. - -This is deliberately a per-source fingerprint of what one gather actually read. -It is not the deleted `ContentRevision` design, does not hash the whole persisted -configuration, and does not turn content into a global revision or transition -authority. That rejected design tried to make one content value stand in for -cooperating generations and failed the A→B→A case. This check instead binds a -prepared catalog candidate to the finite set of file bytes that produced it while -leaving config admission and native routing authority with their existing owners. +The catalog admission snapshot therefore retains a closed set of **role-bearing +source observations** for every filesystem source whose presence, absence, or +bytes influenced preparation. The active catalog, each consulted hashed/legacy +backup or models-cache fallback, runtime/auth selection state, and any later file +source all identify why they were consulted. Most importantly, +`$CODEX_HOME/config.toml` is the required `catalog-target-selection` observation. +Its ABSENCE selects the default catalog path, so absence is evidence and is +recorded even though there is no byte buffer to hash. + +For a present source, the observation owner computes SHA-256 from the **same exact +buffer** it returns. A separate pre-read is not equivalent. For an absent source, +it records the consulted logical path, the canonical missing-leaf path derived from +the canonical parent, stable parent identity, and `fileIdentity:null`. Immediately +before the first commit write, commit re-observes every candidate-bound source and +compares state, logical/canonical path, parent identity, file identity, and digest +where present. PRESENT -> ABSENT, ABSENT -> PRESENT, identity drift, or digest drift +is `stale`. An unreadable source, an unresolvable canonical parent/source, or +ambiguous identity is refused rather than assumed unchanged. Thus a config file +that appears after default-target gather cannot authorize a commit to the obsolete +default target even when that target's own parent, inode, and bytes never moved. + +This is deliberately per-source evidence from one gather. It is not the deleted +`ContentRevision` design, does not hash the whole persisted configuration, and does +not turn content into a global revision or transition authority. That rejected +design tried to make one content value stand in for cooperating generations and +failed the A→B→A case. This check instead binds a prepared catalog candidate to the +finite set of filesystem observations that produced it while leaving config +admission and native routing authority with their existing owners. **What this does not do** (round 2 #6): it cannot detect a parent-symlink A→B→A that happens entirely between two checks. C17 is therefore scoped to *cooperating transitions and single-direction drift*, not to arbitrary filesystem ABA. Claiming otherwise would be a promise the filesystem does not offer. -The same limit applies to source bytes: fingerprints detect single-direction -content drift, including an ordinary in-place truncate-and-rewrite, but not a full -content A→B→A that returns to identical bytes before the commit check. The re-read -is also not filesystem atomicity; a non-cooperating writer can still change bytes -after the final comparison. The outcome must preserve those C17 bounds rather than -promote a digest into a guarantee the filesystem cannot provide. +The same limit applies to source observations: they detect single-direction state, +identity, and content drift, including ABSENT -> PRESENT and an ordinary in-place +truncate-and-rewrite, but not a full state/content A→B→A that returns to identical +evidence before the commit check. The re-observation is also not filesystem +atomicity; a non-cooperating writer can still change bytes after the final +comparison. The outcome must preserve those C17 bounds rather than promote a digest +into a guarantee the filesystem cannot provide. + +### Create-once means no-clobber publication (seam audit round 2) + +Hashed and legacy catalog backups are immutable first-winner snapshots. The ordinary +`atomicWriteFile` helper cannot publish them: its final rename replaces an existing +destination (`src/config.ts:209` in the audited tree). An absence check followed by +that helper is a check-then-write race, not create-once. + +`src/codex/internal/catalog-writer.ts` therefore owns a synchronous atomic +no-clobber publication primitive. It creates and hardens a unique temp beside the +resolved target, then publishes with an operation whose contract is +**destination-must-not-exist** — an exclusive hard link (`link`) or a platform +rename-without-replace equivalent. Ordinary overwriting rename is not a fallback. +The temp is scrubbed/removed on every unpublished path. + +`EEXIST` means another process won publication after our validation. The writer +must resolve, read, and validate that winner as a regular, non-routed catalog backup +under stable parent/file identity. A valid winner is preserved and the receipt says +`preserved`; malformed, unreadable, routed, symlinked, or identity-ambiguous content +is refused. The loser never unlinks, truncates, or overwrites the winner. A +check-absent-then-`atomicWriteFile` sequence does not satisfy this contract, even if +the earlier target-identity check was correct. ## 4. Admission returns a snapshot, not a boolean @@ -722,23 +790,78 @@ Audit #8: `040`'s intent reader returns ON/OFF while `010`'s gather needs a full "two reads" is wrong. ```ts -/** Exact gather-time evidence for one file source that influenced the candidate. */ -export interface CatalogSourceFingerprint { +/** Why a filesystem observation influenced catalog preparation. Closed by contract. */ +export type CatalogRequiredSourceRole = "catalog-target-selection"; + +export type CatalogConditionalSourceRole = + | "bundled-catalog-template" + | "active-catalog-merge" + | "hashed-backup-fallback" + | "legacy-backup-fallback" + | "models-cache-fallback" + | "runtime-selection" + | "provider-auth-selection"; + +export type CatalogSourceRole = + | CatalogRequiredSourceRole + | CatalogConditionalSourceRole; + +/** Portable normalized identity: POSIX dev/inode or Windows volume/file id. */ +export interface CatalogFilesystemIdentity { + readonly volume: string; + readonly fileId: string; +} + +export interface CatalogParentIdentity extends CatalogFilesystemIdentity { readonly canonicalPath: string; - readonly sha256: string; +} + +/** Exact gather-time evidence for one consulted filesystem source. */ +export type CatalogSourceObservation = + | { + readonly state: "present"; + readonly role: R; + readonly logicalPath: string; + readonly canonicalPath: string; + readonly parentIdentity: CatalogParentIdentity; + readonly fileIdentity: CatalogFilesystemIdentity; + /** Digest of the exact buffer returned to gather. */ + readonly sha256: string; + } + | { + readonly state: "absent"; + readonly role: R; + readonly logicalPath: string; + readonly canonicalPath: string; + readonly parentIdentity: CatalogParentIdentity; + readonly fileIdentity: null; + }; + +export type CatalogRequiredSourceObservations = Readonly<{ + [R in CatalogRequiredSourceRole]: CatalogSourceObservation; +}>; + +export type CatalogConditionalSourceObservations = Readonly<{ + [R in CatalogConditionalSourceRole]: readonly CatalogSourceObservation[]; +}>; + +export interface CatalogSourceEvidence { + readonly required: CatalogRequiredSourceObservations; + /** Every role is a required key; an empty list means the role was not consulted. */ + readonly conditional: CatalogConditionalSourceObservations; } /** The shared WP8b/WP9 snapshot; it authorizes catalog work only. */ export interface CatalogAdmissionSnapshot { config: Readonly; - generation: number; + generation: ConfigGeneration; targets: Readonly<{ catalog: string; cache: string; catalogBackups: readonly string[]; }>; - /** Populated from the exact buffers gather read, never from separate pre-reads. */ - sourceFingerprints: readonly CatalogSourceFingerprint[]; + /** Candidate-bound present/absent evidence, produced by the sole read owner. */ + sourceEvidence: CatalogSourceEvidence; } export interface AdmissionSnapshot { @@ -769,34 +892,63 @@ export interface AdmissionSnapshot { } ``` -Pre-gather capture begins with an empty `sourceFingerprints` list because fallback -selection has not happened yet. Gather does not mutate that snapshot: it returns -the prepared candidate with an immutable copy whose list is the exact set of file -sources its readers consumed. Commit accepts only that candidate-bound copy and -refuses an incomplete list; it never treats the empty pre-gather value as evidence -that a source stayed unchanged. - -The earlier one-read claim is withdrawn. There are three authoritative observation -points, each with a different job: - -1. **Pre-gather:** fully read persisted config and all authority/target fields into - snapshot A. Gather consumes `A.config` — that exact object, never the server's - long-lived one. -2. **Under-lock:** while native + config coordination is held, fully re-read snapshot - B and compare digest, config generation, intent, ownership, external provider, - canonical targets, journal identity, and provenance identity. A mismatch rejects - before the first native write. For catalog work, re-read and compare every - gather-time `sourceFingerprint` immediately before the first write as §3 requires; - `scope:"catalog"` performs that check without reading or advancing a native - `CommitExpectation`. -3. **Post-commit:** re-read persisted config and observe every native/catalog/history - surface into `CodexObservedState`. The outcome is not `converged` unless this - observation agrees with admitted intent and the exact expected native pair. - -The config reader at all three points is the persisted diagnostic reader. A missing, -unreadable, or invalid file produces unknown/refusal; it never falls back to the -server's captured object. `010`'s independent gather-time -`readConfigDiagnostics()` remains removed because snapshot A already owns that read. +The role set is closed because “refuse an incomplete list” was not enforceable on an +array of present-file digests: a caller could omit the absent source that selected a +default and leave no evidence of the omission. Pre-gather capture now starts with +the required `catalog-target-selection` observation for the logical +`$CODEX_HOME/config.toml` path, PRESENT or ABSENT, and with every conditional role +key present as an empty list. Gather does not mutate that snapshot. It returns the +prepared candidate with an immutable copy whose conditional lists contain every +filesystem consultation in order, including absent alternatives that caused a +fallback. A missing required role or conditional role key is structurally invalid; +commit accepts only the private candidate-bound `CatalogSourceEvidence`. + +All gather filesystem reads route through the one evidence-producing owner, +`src/codex/catalog/filesystem-evidence.ts`. It owns an opaque gather-evidence +session: source reads append the matching PRESENT or ABSENT observation before +returning, while target probes append the existing target parent/file identity +evidence. Callers never append, remove, or reconstruct evidence arrays themselves; +only the owner can seal the complete session into the private candidate, and sealing +requires the `catalog-target-selection` role. Raw `readFileSync`, `Bun.file`, +`existsSync`-then-read, `lstat`/`realpath` target probing, catalog helper, or indirect +wrapper that consults the filesystem outside that owner is a contract violation. +The symbol-granular graph test follows imports, aliases, re-exports, wrappers, and +literal dynamic imports to prove every gather filesystem read reaches that owner. +Adding a new source purpose requires adding a role here first, so an untyped array +cannot silently expand the authority surface. + +The earlier one-read claim is withdrawn. Catalog admission and full admission have +different authority sources and must not be collapsed: + +1. **Catalog pre-gather (WP9):** the management factory captures one exact resident + `Readonly` reference and passes that same reference to catalog + admission. `CatalogAdmissionSnapshot.config` is that object; the factory closure + is the sole runtime caller of snapshot capture, and route callers receive no + config parameter with which to substitute another authority. Catalog admission + does not independently reconstruct full config from disk. This deliberate + capture is what prevents a route from substituting catalog authority. Admission + separately observes config generation, targets, and the required + `$CODEX_HOME/config.toml` + `catalog-target-selection` role before gather. +2. **Catalog under-lock (WP9):** + `withExpectedConfigGenerationSync(snapshot.generation, commit)` validates the + generation through the already-held config transaction and runs the complete + synchronous catalog commit before release. Commit re-observes and compares every + candidate-bound `sourceEvidence` entry and target identity immediately before its + first write. It does not read or advance a native `CommitExpectation`. +3. **Full pre-gather/under-lock/post-commit (WP12):** full admission reads persisted + config and every authority/target field into snapshot A, fully re-reads snapshot B + under native + config coordination, and post-commit re-reads persisted config plus + every native/catalog/history surface into `CodexObservedState`. It never uses the + server's long-lived object as a persisted-config fallback. Missing, unreadable, or + invalid persisted config produces unknown/refusal. The outcome is not `converged` + unless final observation agrees with admitted intent and the exact expected native + pair. + +`010`'s independent gather-time `readConfigDiagnostics()` remains removed: WP9 has +the captured management reference plus generation/source evidence, while WP12's +full admission owns its persisted diagnostic reads. Those are separate contracts, +not two interchangeable ways to populate one snapshot. ## 5. `/api/sync`, defined once @@ -1003,16 +1155,17 @@ Audit #13. Fixed here so no phase invents a variant: | generations | `src/codex/generation.ts` | | history worker | `src/codex/history-worker.ts` | -### Writer inventory and permitted roots +### Writer inventory and permitted roots, versioned by landing phase The previous rule — “every low-level writer is under `internal/` and only `convergence.ts` may reach it” — was unsatisfiable. `history-worker.ts` must call history writers directly after it acquires the history lock. A module guard also cannot distinguish importing a reader from importing a writer when both symbols live in `inject.ts` or `journal.ts`. The inventory, not a directory slogan, is the -contract: +contract. This first table is the **WP12 final state**, not a claim that WP9 has +already migrated lifecycle and explicit callers: -| Domain | Low-level writer owner | Permitted runtime roots | +| Domain | Low-level writer owner | WP12 final permitted runtime roots | |---|---|---| | native config/profile | `src/codex/internal/native-writer.ts` | `src/codex/convergence.ts` only | | injection journal create/mark/restore/remove | `src/codex/internal/journal-writer.ts` | `src/codex/convergence.ts` only | @@ -1025,21 +1178,45 @@ contract: `src/codex/internal/catalog-writer.ts` is the contract-owned name. Phase documents must use it; `internal/catalog-commit.ts` is not an alternate name for this owner. -`inject.ts` is split: observation/parsing and pure config/profile transforms stay -readable there; every export that calls `atomicWriteFile`/`unlinkSync` moves to -`internal/native-writer.ts`. `journal.ts` is split into read/validate/classify code -(`journal.ts`) and the four mutating operations in `internal/journal-writer.ts`. -The writer half may import the reader half; the reader half never imports or -re-exports the writer. `catalog.ts` likewise stops re-exporting direct writer -symbols. These splits are required before a module-level reachability assertion can -mean “reader imports are safe.” +WP9 has a narrower migration boundary: it moves the 16 management mutation +callbacks behind catalog convergence but intentionally leaves these four legacy +writer chains until WP12 installs full admission, observation, provenance, and +lifecycle convergence: -The contract test publishes this table as data and walks static imports, dynamic +| WP9 transitional legacy root | Exact writer chain still permitted | WP12 final action | +|---|---|---| +| management `POST /api/sync` | `src/server/management/config-routes.ts` -> `src/codex/sync.ts` -> `src/codex/refresh.ts` -> catalog writer | rewire to full convergence and `toSyncResponse` | +| server startup cache invalidation | `src/server/index.ts` -> models-cache writer | route startup through full convergence/observer | +| CLI `sync-cache` | `src/cli/index.ts` -> models-cache writer | route the CLI command through full convergence | +| native restore | `src/codex/inject.ts` -> catalog restore writer | move restore behind full convergence/provenance | + +This is an exact transitional allowlist by root module and writer symbol, not a +directory wildcard. `src/codex/sync.ts`, `src/codex/refresh.ts`, CLI, and +`src/codex/inject.ts` are therefore permitted only through the rows above at the +WP9 commit; no fifth legacy root may appear. WP12 removes every row and activates +the final table. A contract test cannot enforce both versions at once, so the graph +fixture carries an explicit `"wp9-transitional" | "wp12-final"` inventory version: +WP9 expects exactly four legacy chains, and WP12 changes that expectation to zero. + +At the WP12-final transition, `inject.ts` is split: observation/parsing and pure +config/profile transforms stay readable there; every export that calls +`atomicWriteFile`/`unlinkSync` moves to `internal/native-writer.ts`. `journal.ts` is +split into read/validate/classify code (`journal.ts`) and the four mutating +operations in `internal/journal-writer.ts`. The writer half may import the reader +half; the reader half never imports or re-exports the writer. `catalog.ts` likewise +stops re-exporting direct writer symbols, while WP9 may move the four transitional +callers to explicit writer imports solely to keep their unchanged behavior +compiling. These splits are required before the WP12-final reachability assertion +can mean “reader imports are safe.” + +The contract test publishes the phase-appropriate table as data and walks static imports, dynamic imports, re-exports and aliases at **symbol** granularity. Every inventoried writer -must have exactly the permitted roots above, and every filesystem/SQLite mutator of -a Codex-owned artifact must appear in the inventory. `history-job.ts`, management -routes, CLI modules, `sync.ts`, `refresh.ts`, `inject.ts`, and `journal.ts` are not -permitted roots; they call convergence, dispatch a Worker, or read only. +must have exactly the permitted roots for that phase, and every filesystem/SQLite +mutator of a Codex-owned artifact must appear in the inventory. In the WP12-final +version, `history-job.ts`, management routes, CLI modules, `sync.ts`, `refresh.ts`, +`inject.ts`, and `journal.ts` are not permitted roots; they call convergence, +dispatch a Worker, or read only. That final prohibition must not be applied to the +four explicit WP9 transitional rows before WP12 owns their migration. ## 9. Baseline classes @@ -1084,9 +1261,32 @@ with the repository TypeScript compiler so WP8b cannot regress to TS2304 or a bodyless TS2391 declaration. Table-drive each artifact observation and require `isApplied` only for the fully applied aggregate. A catalog-only commit neither requests a `CommitExpectation` nor changes the native pair, and its projected -outcome has no pair fields. Gather from a catalog source, truncate-and-rewrite that -same inode, and require commit to return `stale` before any write; repeat for each -selected backup/cache fallback and refuse unreadable or ambiguous re-reads. +outcome has no pair fields. Prove `withExpectedConfigGenerationSync` validates on +the already-held transaction: while its callback is paused, a second cooperating +process cannot commit N+1, and the callback's catalog bytes finish before the lock +is released; conflict never invokes the callback. Instrument connection creation so +the guard cannot regress to `readConfigGenerationAtPath` and self-contend through a +second SQLite handle. + +Table-drive every `CatalogSourceRole`. Gather from a present source, +truncate-and-rewrite that same inode, and require `stale` before any write. Gather +with `$CODEX_HOME/config.toml` absent, then create it with +`model_catalog_json` selecting another target; require `stale` with the old target +byte-identical. Repeat PRESENT -> ABSENT and present-byte/path changes. A compile +fixture omitting `required["catalog-target-selection"]` or any conditional role key +must fail, while the complete shape compiles. The symbol graph must fail when any +gather reader performs or reaches a raw filesystem consultation outside +`catalog/filesystem-evidence.ts`, including an absence-only `existsSync` branch and +a direct target-identity `lstat`/`realpath` probe. +Unreadable or ambiguous re-observations refuse. + +Race two create-once backup publishers after both observed ABSENT. Exactly one +no-clobber publication wins; the loser receives `EEXIST`, validates and preserves +the winner, and neither ordinary rename nor `atomicWriteFile` is called. Repeat with +malformed, unreadable, routed, symlinked, and identity-ambiguous winners and require +refusal without changing winner bytes. The graph inventory fixture runs as +`wp9-transitional` with exactly four legacy chains, then as `wp12-final` with zero; +a fifth WP9 root or a retained WP12 root fails. `tests/codex-user-identity.test.ts`: real child processes vary every environment home/runtime variable named in §7 and resolve one final database path for one @@ -1109,17 +1309,23 @@ writes to it. ## Accept criteria - C14 — all 16 management callers funnel through `convergeCodex`, enforced by the - import guard test. + symbol graph; its WP9 inventory permits exactly the four transitional chains and + its WP12-final inventory permits none. - C16 — one owner, one schema; a record from any phase reads in every other. - C17 — cooperating transition ABA is detected by the durable config/native generations and exact txId, and a parent target that drifts once between gather and the under-lock commit check is detected by canonical target identity. A - gathered catalog source whose bytes drift once is detected by its per-source - fingerprint even when dev+inode and both generations are unchanged. An arbitrary - filesystem or content A→B→A that completes wholly between two checks, and a write - after the final comparison, are explicitly not claimed. + gathered catalog source whose state, identity, or bytes drift once is detected by + its role-bearing observation even when the write target and both generations are + unchanged. This includes required `config.toml` ABSENT -> PRESENT target-selection + drift. Cooperating config N -> N+1 is prevented while the catalog callback holds + the existing config transaction, and create-once backups use atomic no-clobber + publication. An arbitrary filesystem or content A→B→A that completes wholly + between two checks, and a non-cooperating write after the final comparison, are + explicitly not claimed. - Contributes to C15 with detect-and-repair: the latest native pair is durably pending before spawn, a stale Worker cannot replace its transition row or the winner's schedule, and the guardian eventually repairs history. WP10 implements that protocol. Also contributes to - C2/C12 (generations and the three-read admission/observation sequence). + C2/C12 (generation-guarded catalog commit plus the phase-specific catalog/full + admission and observation sequences). diff --git a/devlog/_plan/260804_codex_write_substrate/010_catalog_seam.md b/devlog/_plan/260804_codex_write_substrate/010_catalog_seam.md index e8f4a8dde..4ef96e187 100644 --- a/devlog/_plan/260804_codex_write_substrate/010_catalog_seam.md +++ b/devlog/_plan/260804_codex_write_substrate/010_catalog_seam.md @@ -23,9 +23,11 @@ native generation requires matching history schedule fields, every transition that was not published (`src/codex/transition-state.ts:74-83,314-344,420-428`). All current-code citations and diff context below were rechecked on 2026-08-04 -at `de15caf449264f203cf003d032bb5f62bb448e72`. The contract citations refer to -the concurrent WP8b amendment that adds catalog source fingerprints and excludes -catalog-only work from the native pair (`005_contract.md:579-604,681-716,724-742`). +at `db3d69ed05bba5c2dd822f2c2088c186adf5a105`. The contract citations refer to +the authoritative concurrent WP8b/WP9 amendment: the owner-held config-generation +guard, closed source-observation union, and atomic no-clobber publication are at +`005_contract.md:615-645,722-784,792-918,923-938`; catalog-only work remains +excluded from the native pair at `005_contract.md:588-613`. ## IN / OUT @@ -35,39 +37,65 @@ IN — observe-only admission and gather: generation observation that never creates, initializes, chmods, or registers `config-mutation.sqlite`. The existing `readConfigGeneration` is not that API: it resolves/records the path and opens SQLite with `create:true` - (`src/config.ts:1741-1771,1845-1849`, `src/codex/generation.ts:93-103`). + (`src/config.ts:1741-1771,1845-1849`, `src/codex/generation.ts:93-103`). WP9 + consumes the contract-owned `withExpectedConfigGenerationSync`; it does not + wrap this observer in the lock or redefine the generation contract. - `src/codex/catalog-admission.ts` (MODIFY) — keep the landed request constructor and snapshot capture; switch snapshot capture to the observe-only generation - read and carry the contract-owned source-fingerprint list. Do not redefine + read, capture the required PRESENT-or-ABSENT `catalog-target-selection` + observation, and carry the contract-owned `sourceEvidence`. Do not redefine `createCatalogConvergeRequest` or `captureCatalogAdmissionSnapshot`, which already exist at lines 32-46 and 84-107. - `src/codex/convergence-types.ts` (MODIFY) — synchronize the already contract-owned - `CatalogSourceFingerprint` and `CatalogAdmissionSnapshot.sourceFingerprints` - additions from `005_contract.md`; no WP9-private duplicate type is allowed. + closed `CatalogSourceRole`, `CatalogSourceObservation`, `CatalogSourceEvidence`, + and `CatalogAdmissionSnapshot.sourceEvidence` additions from + `005_contract.md:792-865`; no WP9-private duplicate type is allowed. +- `src/codex/catalog/filesystem-evidence.ts` (NEW) — sole owner of gather source + reads and target probes. Its opaque session records PRESENT and ABSENT + observations before returning, seals the complete closed role map into the + candidate, and is the only gather path permitted to call filesystem consultation + primitives (`005_contract.md:895-918`). - `src/codex/runtime.ts`, `src/codex/catalog/bundled.ts` (MODIFY) — catalog gather - resolves the runtime through the existing non-persisting `resolveCodexRuntime` - (`src/codex/runtime.ts:394-405`), never - `resolveAndPersistCodexRuntime`, whose successful path may mkdir and replace - `codex-runtime.json` (`src/codex/runtime.ts:213-228,500-516`, - `src/codex/catalog/bundled.ts:146-169`). + uses a gather-specific observe-only pair: + `peekCodexRuntimeForCatalogGather(evidenceSession)` and + `resolveCatalogSourceForGather(evidenceSession)`. They never probe an executable + or start a subprocess. The first returns only an already-resolved process-local or + persisted runtime observation; the second consumes a matching in-memory bundled + catalog or observed persisted catalog/backup/cache source and otherwise returns + `catalog-unavailable`. `bundled.ts` owns the catalog-specific adapter; + `runtime.ts` exposes only a process-cache peek and pure persisted-state parser and + never imports the catalog evidence module. A cold miss never becomes permission to + execute Codex. + The ordinary resolver reaches `probeVersion`, whose sandbox deliberately calls + `mkdtempSync` and `rmSync` (`src/codex/runtime.ts:231-279,327-340,397-405`), while + bundled loading both calls the persisting resolver and runs `codex debug models` + (`src/codex/catalog/bundled.ts:127-169,170-210`). Neither path is reachable from + gather. - `src/oauth/index.ts`, `src/oauth/store.ts`, `src/codex/catalog/provider-fetch.ts` (MODIFY) — add and consume an observe-only - active-token snapshot based on `peekAuthStore`, which already promises no - chmod or invalid-file backup (`src/oauth/store.ts:145-157`). It never refreshes, - persists, acquires an intent lock, creates/removes an intent file, hardens a - path, or backs up malformed credentials. The current token resolver can enter - refresh/persistence (`src/oauth/index.ts:281-339,352-354`) and the current gather - awaits it (`src/codex/catalog/provider-fetch.ts:410-428`). + active-token snapshot. The filesystem-evidence owner reads the exact auth-store + buffer under `provider-auth-selection`; `oauth/store.ts` exposes/reuses pure + normalization semantics rather than calling `peekAuthStore` or another hidden + filesystem reader. `peekAuthStore` confirms the desired no-chmod/no-backup behavior + but still owns its own `existsSync`/`readFileSync` consultation today + (`src/oauth/store.ts:145-157`). The gather path never refreshes, persists, acquires + an intent lock, creates/removes an intent file, hardens a path, or backs up malformed + credentials. The current token resolver can enter refresh/persistence + (`src/oauth/index.ts:281-339,352-354`) and the current gather awaits it + (`src/codex/catalog/provider-fetch.ts:410-428`). - `src/codex/refresh.ts`, `src/codex/catalog/sync.ts`, `src/codex/catalog/parsing.ts` (MODIFY) — prepare immutable catalog/cache/backup - bytes and source evidence without writing. + bytes and source evidence without writing. In particular, target selection no + longer hides an `existsSync`/`readFileSync` consultation inside + `readCodexCatalogPath()` (`src/codex/catalog/parsing.ts:167-176`); admission makes + that consultation through the evidence owner. IN — fixed commit and convergence: - `src/codex/internal/catalog-writer.ts` (NEW/MOVE) — the contract-owned low-level owner for catalog, hashed/legacy backups, and models cache. Do not create the obsolete `internal/catalog-commit.ts` name - (`005_contract.md:1019,1025-1026`). + (`005_contract.md:1172,1178-1179`). - `src/codex/convergence.ts` (NEW) — catalog gather/commit orchestration and the only WP9 module allowed to call symbols in `internal/catalog-writer.ts`. - `src/codex/management-convergence.ts` (MODIFY) — retain the landed @@ -96,8 +124,10 @@ IN — management callers and tests: OUT: - WP10 history scheduling/worker behavior. Catalog-only work schedules no history. -- WP11 native lock acquisition. WP9 makes commit synchronous but does not import a - future lock helper or claim cross-process exclusion. +- WP11 native lock acquisition. WP9 makes commit synchronous and uses only the + existing config mutation transaction to exclude cooperating config writers through + publication; it does not import a future native lock or claim exclusion against + native/catalog hand edits and foreign writers. - WP12 full admission/observer/provenance, full `scope:"full"` convergence, `/api/sync`, startup, CLI cache sync, restore, and complete writer reachability. - Any runtime command that starts, stops, syncs, restores, ensures, or manages the @@ -109,6 +139,14 @@ management factory/projection into `convergence.ts` when it installs the full entry point; that later move is a module consolidation, not completion of an unfinished WP9 branch. +Phase-entry gate: the audited source currently exports `readConfigGeneration` and +`bumpConfigGeneration` only (`src/config.ts:1845-1859`); the amended contract assigns +the executable `withExpectedConfigGenerationSync` owner seam to WP8b +(`005_contract.md:615-645`). WP9 implementation starts after that prior phase lands. +If the seam is still absent, stop and report the WP8b scope dependency; do not emulate +it with a second connection, weaken the guard to observe-before-write, or leave a +placeholder for WP12. + ## A. Filesystem-write-free gather ### A1 — state the guarantee exactly @@ -126,7 +164,9 @@ Filesystem-write-free means the entire interval from **before admission capture* through resolved runtime observation, token observation, provider calls, fallback selection, parsing, serialization, and candidate construction performs no mkdir, write, rename, copy, unlink, chmod/ACL change, SQLite create/init/WAL change, -ownership registration, backup, or transient temp-file creation. +ownership registration, backup, transient temp-file creation, executable probe, or +subprocess. The guarantee covers scratch outside the Codex/OpenCodex homes too; +there is no permitted gather scratch scope. This bound deliberately catches the writes hidden by the old plan: @@ -137,6 +177,15 @@ This bound deliberately catches the writes hidden by the old plan: not permission to refresh and persist; - admission must not invoke the create-on-read generation path, which can also register ownership metadata (`src/lib/config-ownership.ts:202-226,262-282`). +- `resolveCodexRuntime()` is forbidden even though it does not itself persist: a + cold resolution reaches `probeVersion()`, and that probe intentionally creates + and deletes a temporary `CODEX_HOME` because real Codex writes even for + `--version` (`src/codex/runtime.ts:231-279,327-340,397-405`). A final-state + manifest cannot see that created-then-deleted directory. +- `loadBundledCodexCatalog()` and `runCodexDebugModels()` are forbidden beneath + gather. The current bundled loader resolves/persists a runtime and executes + `codex debug models --bundled` without an isolated gather environment + (`src/codex/catalog/bundled.ts:127-169,170-210`). The observe-only generation API opens an existing database with `readonly:true`, performs only schema/version/select checks, and closes it. Missing DB/table/row, @@ -144,40 +193,69 @@ busy, malformed, or unreadable state returns the existing typed unavailable resu it never initializes generation zero. `captureCatalogAdmissionSnapshot` projects that result into a typed catalog refusal through the total adapter. -The observe-only token snapshot reads the active credential once from -`peekAuthStore`. A non-expired access token may be used. Missing, malformed, -near-expiry, or expired OAuth credentials yield provider-auth without calling any -refresh path. Static API keys and request headers already present in the admitted -`Readonly` remain usable. If the auth-store buffer influences a live -provider result, its canonical path and SHA-256 join the candidate's private source -fingerprints; token bytes never do. - -### A2 — fingerprint the exact buffers that influenced output - -`captureCatalogAdmissionSnapshot(config)` remains the pre-gather constructor and -starts with the contract-required empty `sourceFingerprints`. Each gather reader -returns bytes and a `CatalogSourceFingerprint` computed from that **same buffer**. -The candidate receives an immutable snapshot copy containing exactly the sources -actually selected or merged: - -- active catalog bytes when read as the merge source; -- the selected hashed backup, legacy backup, or models-cache fallback; -- persisted runtime-selection or auth-store bytes when those reads influenced - runtime/token selection; -- any later file buffer that affects candidate bytes. - -Do not fingerprint alternatives merely because their paths exist, and do not hash a -separate pre-read. Process-local caches and subprocess/network responses are not file -buffers and therefore are not fabricated as file fingerprints. - -Immediately before the first replacement, commit re-reads every candidate-bound -source by canonical path. Digest mismatch returns `stale`; unreadable, -unresolvable, non-regular, or ambiguous source identity returns `refused`. Both -paths write zero bytes. This detects the audited same-inode truncate/rewrite even -when config generation and target dev/inode are unchanged. It catches -single-direction drift only: content A→B→A returning identical bytes before the -comparison, a parent A→B→A between checks, and a write after the final comparison -remain outside C17 (`005_contract.md:681-716`). +The gather-specific resolver is a separate API, not a flag on +`resolveCodexRuntime()`. `bundled.ts` owns +`peekCodexRuntimeForCatalogGather(evidenceSession)`: it may consume an unexpired +successful value from a pure process-cache peek exported by `runtime.ts`, or parse +persisted `codex-runtime.json` bytes supplied by the evidence session through a pure +runtime-state parser. `runtime.ts` never imports the catalog evidence owner. This +path does not test whether the command is executable, discover PATH alternatives, +call `probeVersion`, persist selection, or execute the command. The observation can +only identify a matching already-populated in-memory bundled-catalog cache; it is not +authority to refill it. `resolveCatalogSourceForGather(evidenceSession)` then tries +that immutable cache value followed by active-catalog/backup/models-cache buffers +read through the evidence owner. Its closed result is usable prepared source or +`catalog-unavailable`; the latter projects to the existing sanitized +`skipped/catalog-unavailable` disposition and leaves no residue. + +The observe-only token snapshot receives the exact auth-store buffer from the +filesystem-evidence owner and applies the store's pure normalization once. A +non-expired access token may be used. Missing, malformed, near-expiry, or expired +OAuth credentials yield provider-auth without calling any refresh path. Static API +keys and request headers already present in the admitted `Readonly` remain +usable. If the auth-store buffer influences a live provider result, its PRESENT or +ABSENT `provider-auth-selection` observation joins the candidate's private source +evidence; token bytes never do. + +### A2 — seal the closed role-bearing source observations + +`captureCatalogAdmissionSnapshot(config)` remains the pre-gather constructor. Its +source evidence starts with every conditional role key present as an empty list and +the required `catalog-target-selection` observation for the logical +`$CODEX_HOME/config.toml` path, recorded PRESENT or ABSENT. The opaque +filesystem-evidence session then records every consulted filesystem source under +the contract's closed role union: bundled template, active merge, hashed/legacy +fallback, models-cache fallback, runtime selection, or provider-auth selection +(`005_contract.md:792-852,895-918`). Callers cannot append, omit, remove, or rebuild +those observations. + +The required ABSENT state closes a target-selection hole that a present-file digest +list cannot represent. `readCodexCatalogPath()` chooses the default catalog exactly +when `config.toml` is absent (`src/codex/catalog/parsing.ts:167-176`). If that file +appears after gather with `model_catalog_json` selecting another target, no digest of +any previously present file changes; without the required absence observation the +obsolete default target could be overwritten and reported `committed`. Therefore +`config.toml` is always observed with role `catalog-target-selection`, and either +PRESENT -> ABSENT or ABSENT -> PRESENT is `stale` before any write. + +For a PRESENT source, the evidence owner reads once and hashes the **same exact +buffer** it returns. For an ABSENT source, it records the logical path, canonical +missing-leaf path, stable canonical-parent identity, and `fileIdentity:null`. +Alternatives consulted and found absent are still evidence because their absence +caused fallback. Process-local caches and network responses are not fabricated as +filesystem observations. The candidate receives a sealed immutable +`CatalogSourceEvidence`; a missing required role or conditional key is structurally +invalid and cannot reach commit. + +Immediately before the first replacement, the under-lock commit callback +re-observes every candidate-bound source and compares state, logical/canonical path, +parent identity, file identity, and PRESENT digest. State, identity, path, or digest +drift returns `stale`; unreadable, unresolvable, non-regular, or ambiguous evidence +returns `refused`. Both paths write zero bytes. This detects the audited same-inode +truncate/rewrite even when config generation and target identity are unchanged. It +catches single-direction drift only: content/state A→B→A returning identical +evidence before comparison, parent A→B→A between checks, and a write after the final +comparison remain outside C17 (`005_contract.md:731-762`). ### A3 — preserve bundled-first template precedence @@ -188,16 +266,19 @@ separately as the merge source. The invariant is explicit at `src/codex/catalog/bundled.ts:225-234` plus `src/codex/catalog/sync.ts:517-523`. -The WP9 edit removes only the materializing fallback call from the tail of -`loadCatalogForSync`; it does not move catalog/backup/cache ahead of a successful -bundled template. The bundled branch uses observe-only runtime resolution. Existing -explicit materialization callers remain until their owning phase migrates them. +The WP9 edit removes the materializing fallback call from the tail of +`loadCatalogForSync`; it does not move catalog/backup/cache ahead of an already +available matching bundled template. Gather may clone a matching in-memory bundled +catalog but may not refill that cache. A cold cache falls through to filesystem +sources observed by the evidence owner; no usable native template yields +`catalog-unavailable`. Existing explicit materialization and probing callers remain +outside the 16 management paths until their owning phase migrates them. ### A4 — candidate ownership The candidate remains opaque, one-shot, and catalog-private. Its `WeakMap` state contains prepared bytes, result/notices, target identities, the admitted config -generation, and the populated candidate-bound source fingerprints. Commit marks it +generation, and the sealed candidate-bound `CatalogSourceEvidence`. Commit marks it consumed before validation and before the first write; a second call returns `candidate-consumed` and writes nothing. No route can inspect, serialize, reconstruct, or replay it. @@ -214,7 +295,7 @@ export interface CatalogWriteReceipt { export type CodexCatalogCommitResult = | { readonly kind: "committed"; readonly changed: boolean; readonly writes: CatalogWriteReceipt } - | { readonly kind: "stale"; readonly reason: "generation" | "source-fingerprint" | "target-identity" | "candidate-consumed" } + | { readonly kind: "stale"; readonly reason: "generation" | "source-observation" | "target-identity" | "candidate-consumed" } | { readonly kind: "refused"; readonly reason: "source-unreadable" | "source-ambiguous" | "target-unsafe" } | { readonly kind: "failed"; readonly surface: "disk"; readonly writes: CatalogWriteReceipt }; ``` @@ -231,25 +312,54 @@ OAuth resolver, Promise, or callback that can return a Promise. Commit performs, in order: -1. validate candidate not consumed, config generation, every target identity, and - every source fingerprint; -2. keyed backup create-once replacement; -3. legacy backup create-once replacement when the default path requests it; -4. active catalog replacement; -5. models cache replacement. +1. mark the candidate consumed, then call + `withExpectedConfigGenerationSync(candidate.generation, commitCallback)`; +2. inside the already-held config transaction, validate every target identity and + re-observe every sealed PRESENT/ABSENT source observation immediately before the + first write; +3. publish the keyed backup with atomic no-clobber semantics; +4. publish the legacy backup with atomic no-clobber semantics when the default path + requests it; +5. replace the active catalog; +6. replace the models cache; then return from the synchronous callback so the owner + can release the config transaction. + +The owner-side guard is not a read-before-write check. Its implementation validates +the expected generation using the `configMutationDatabase` handle whose SQLite +transaction is already held, invokes the complete synchronous catalog callback on a +match, and releases only after the callback returns +(`005_contract.md:629-645,692-703`). A cooperating config writer therefore cannot +commit N+1 between validation and catalog publication. Conflict never invokes the +callback; lock/database unavailability projects through the total adapter. + +Do not wrap `readConfigGeneration`, `readConfigGenerationAtPath`, or the new +observe-only reader inside `withConfigMutationLockSync`. Those observers open a +second SQLite connection; while the first connection owns `BEGIN IMMEDIATE`, the +second connection contends with its own caller instead of validating it. The guard +must use `readConfigGenerationInTransaction` or its private equivalent on the +already-held database. WP9 consumes the existing config mutation lock only; it does +not import WP11's native lock, and catalog-only work does not bump config generation. Receipt fields change only after the corresponding replacement succeeds. A failure returns the exact prefix receipt and consumes the candidate; callers must regather. There is no rollback claim. -Target identity remains strict except for one create-once rule. If a backup target -was absent at gather and is present at commit, commit may mark it `preserved` and -continue only when the target is a safely resolved regular file, readable, and a -valid non-routed catalog backup. It is never overwritten. A symlink, unreadable -file, malformed JSON/catalog, routed-content backup, or ambiguous identity is -`refused` before the first write. This exception applies only to a backup used as a -create-once target, never to a backup whose bytes were selected as a gather source; -selected source fingerprints remain strict. +Target identity remains strict except for one create-once rule. Backup publication +creates and hardens a unique adjacent temp, then uses an operation whose contract is +destination-must-not-exist: exclusive hard link or a platform +rename-without-replace equivalent. Ordinary overwrite rename is never a fallback. +The existing `atomicWriteFile` cannot implement this contract because its final +operation is an overwriting rename (`src/config.ts:188-220`, especially line 209). +The unpublished temp is scrubbed and removed on every path. + +If publication returns `EEXIST`, another process won after validation. Commit +resolves and validates that winner under stable parent/file identity. A readable, +regular, non-routed valid catalog backup is preserved and the receipt becomes +`preserved`; malformed, unreadable, routed, symlinked, or identity-ambiguous content +is `refused`. The loser never unlinks, truncates, or overwrites the winner. This +exception applies only to a backup create-once target, never to a backup selected as +a gather source; selected source observations remain strict +(`005_contract.md:764-784`). ## C. Catalog-only convergence @@ -269,12 +379,13 @@ WP9 does not redeclare request, snapshot, projection, or shared result types. The placeholder factory body at `src/codex/management-convergence.ts:81-96` is replaced in place. It validates catalog scope without throwing, captures admission, -awaits the write-free gather, executes the synchronous commit, and projects the -result. The lower-level orchestration lives in new `convergence.ts`, so only that -module reaches `internal/catalog-writer.ts`; the retained management module remains -the factory boundary until WP12 consolidates the full funnel. +awaits the write-free gather, enters `withExpectedConfigGenerationSync`, executes the +synchronous catalog callback before that owner releases, and projects the result. The +lower-level orchestration lives in new `convergence.ts`, so only that module reaches +`internal/catalog-writer.ts`; the retained management module remains the factory +boundary until WP12 consolidates the full funnel. -### C2 — config generation, source fingerprints, and target identity only +### C2 — owner-held config generation, source observations, and target identity only A `scope:"catalog"` commit does not request `CommitExpectation`, open `transition-state.ts`, call `beginCodexTransition`, call `assertPublished`, or read @@ -283,9 +394,11 @@ has no pair fields (`src/codex/convergence-types.ts:207-224`). Catalog staleness is guarded by: -- the observe-only config generation captured before gather and re-read immediately - before write; -- candidate-bound per-source fingerprints; +- the observe-only config generation captured before gather and validated by + `withExpectedConfigGenerationSync` on its already-held transaction through the + complete synchronous commit; +- candidate-bound closed PRESENT/ABSENT source observations, including required + `config.toml` target selection; - target parent/file identity plus the narrow create-once backup exception. The commit must never import or invoke routing writers. A test fails if catalog-only @@ -311,7 +424,7 @@ a conservative typed projection: | gather admission busy | `skipped/busy`, retryable | | no usable catalog source | `skipped/catalog-unavailable` | | config/target/source refusal | `skipped/refused` | -| generation, fingerprint, or identity drift | `skipped/stale`, retryable | +| generation, source-observation, or identity drift | `skipped/stale`, retryable | | provider auth/network gather failure | matching `failed` reason, `phase:"gather"`, `partialWrite:false` | | lazy import, missing export, factory, or unexpected pre-commit failure | sanitized `failed/disk`, `phase:"gather"`, `partialWrite:false` | | expected replacement failure | `failed/disk`, `phase:"commit"`, `partialWrite` derived from the receipt | @@ -357,10 +470,10 @@ The symbol-graph test permits these exact legacy roots until WP12: | Legacy root | Current path | WP12 removal | |---|---|---| -| management `POST /api/sync` | `config-routes.ts:261-268` → `sync.ts:83-89` → `refresh.ts:44-51` | rewire to full convergence and `toSyncResponse` | -| server startup cache invalidation | `server/index.ts:403` → `invalidateCodexModelsCache` | route startup through full convergence/observer | -| `ocx sync-cache` | `cli/index.ts:849-855` → `invalidateCodexModelsCache` | route CLI command through full convergence | -| native restore | `codex/inject.ts:764-774` → `restoreCodexCatalog` → `catalog/sync.ts:572-597` | move restore writes behind full convergence/provenance | +| management `POST /api/sync` | `src/server/management/config-routes.ts:261-268` → `src/codex/sync.ts:83-89` → `src/codex/refresh.ts:44-51` | rewire to full convergence and `toSyncResponse` | +| server startup cache invalidation | `src/server/index.ts:403` → `invalidateCodexModelsCache` | route startup through full convergence/observer | +| `ocx sync-cache` | `src/cli/index.ts:849-855` → `invalidateCodexModelsCache` | route CLI command through full convergence | +| native restore | `src/codex/inject.ts:764-774` → `restoreCodexCatalog` → `src/codex/catalog/sync.ts:572-597` | move restore writes behind full convergence/provenance | The allowlist is exact by root module and writer symbol, not a directory wildcard. WP12 owns deleting every row. No new legacy root may be added in WP9. @@ -378,42 +491,66 @@ all 16 management roots terminate at `convergence.ts` before a catalog writer. ### T1 — gather really performs no filesystem write Run admission plus gather in a child process with fresh `mktemp -d` values for -`OPENCODEX_HOME`, `CODEX_HOME`, and any config/runtime home. Capture the recursive -manifest **before calling `captureCatalogAdmissionSnapshot`**, not after admission. -For every entry record relative path, kind, regular-file SHA-256, size, mode, -mtime at nanosecond resolution where available, and symlink target. Compare it -after gather. - -Start a recursive filesystem event journal before admission and stop it after gather; -fail on create/delete/rename/write/metadata events so a temp file created and deleted -within the interval is visible. Prove the harness is non-vacuous with controls that -(a) chmod an existing file and (b) create then delete a temp file; both must fail. -Also inject throw-on-call spies for runtime persistence, OAuth refresh/persist/intent, -ownership registration, generation initialization, backup creation, and atomic -replacement. Reset and separately assert the permitted process-local caches changed -only within their bounded owners. - -Broken mutation that must turn T1 red: replace observe-only runtime resolution with -`resolveAndPersistCodexRuntime`, use `loadAuthStore`, or use -`readConfigGeneration`; the manifest/event journal or write spy detects the mkdir, -chmod, backup, SQLite, ownership, or temp-file activity. - -### T2 — fingerprints and identity reject before write - -Table-drive active catalog, selected hashed backup, selected legacy backup, selected -models-cache fallback, runtime-state source, and auth-store source. Gather at config -generation N, truncate and rewrite the selected source **in place** so dev/inode and -generation remain the same, then commit. Expect `stale` and byte-identical targets. -Make each source unreadable/ambiguous and expect `refused` with zero writes. Change -config through the real cooperating mutation API and expect generation rejection. -Retarget one parent and expect target-identity rejection. +`OPENCODEX_HOME`, `CODEX_HOME`, any config/runtime home, and a dedicated process temp +root wired through `TMPDIR`, `TMP`, and `TEMP`. Before admission, capture a recursive +manifest of every isolated root: relative path, kind, regular-file SHA-256, size, +mode, nanosecond mtime where available, and symlink target. Compare it after gather. + +Start recursive filesystem event journals for every isolated root **before calling +`captureCatalogAdmissionSnapshot`** and stop them only after gather settles. Fail on +create/delete/rename/write/metadata events so a temp file created and deleted within +the interval is visible; the before/after manifest is corroboration, not the only +proof. Prove the harness is non-vacuous with controls that (a) chmod an existing file +and (b) create then delete a temp file; both must fail. Inject throw-on-call executable +hooks for `mkdtempSync`, runtime probing, `execFileSync`/subprocess launch, runtime +persistence, OAuth refresh/persist/intent, ownership registration, generation +initialization, backup creation, and atomic replacement. Reset and separately assert +the permitted process-local caches changed only within their bounded owners. + +Broken mutations that must turn T1 red: call `resolveCodexRuntime` on a cold cache, +call `runCodexDebugModels`/`loadBundledCodexCatalog`, replace observe-only runtime +resolution with `resolveAndPersistCodexRuntime`, use `loadAuthStore`, or use +`readConfigGeneration`. The executable spy or pre-admission event journal detects the +subprocess, created-then-deleted probe home, mkdir, chmod, backup, SQLite, ownership, +or other transient write. + +### T2 — closed observations, owner-held generation, and identity reject before write + +Table-drive every `CatalogSourceRole`: required config target selection, +filesystem-backed bundled-template source, active catalog, selected hashed backup, +selected legacy backup, models-cache fallback, runtime-state source, auth-store source, +and every consulted absent alternative. Gather at config generation N, truncate and +rewrite a PRESENT selected source **in place** so file identity and generation remain +the same, then commit. Expect `stale` and byte-identical targets. Repeat PRESENT -> +ABSENT and ABSENT -> PRESENT. In the required target-selection case, gather with +`config.toml` absent, then create it with `model_catalog_json` selecting another +catalog; expect `stale` and byte-identical old/new targets. Make each re-observation +unreadable/ambiguous and expect `refused` with zero writes. Retarget one parent and +expect target-identity rejection. Compile fixtures that omit +`required["catalog-target-selection"]` or any conditional role key; each must fail, +while the complete shape compiles. + +Prove the cooperating-writer guarantee with two real processes and the real config +mutation API. Process A enters +`withExpectedConfigGenerationSync({value:N}, callback)`; callback entry proves +validation matched and pauses while the config transaction is still held. Process B +then attempts a real persisted config mutation. Because the existing lock is +fail-fast (`src/config.ts:1778-1815`), B's first attempt must report lock/busy and must +not commit N+1 while A is paused. A's synchronous catalog bytes land before callback +return; after A releases, B retries through the real mutation API and commits N+1. +Conflict never invokes the callback. Instrument SQLite connection creation and +require the guard to validate through the already-held handle, with no second +connection. Document but do not claim detection for content A→B→A returning exact A before the comparison, parent A→B→A entirely between checks, or a write after the comparison. -Broken mutation that must turn T2 red: remove fingerprint comparison while retaining -generation and inode checks; the same-generation same-inode rewrite would commit and -the old candidate bytes would replace the newer source-derived state. +Broken mutations that must turn T2 red: omit the required ABSENT config observation, +remove digest comparison while retaining generation/file identity, release the config +transaction before callback, or call `readConfigGenerationAtPath` from inside the +guard. The absent->present target switch commits obsolete bytes, the same-inode +rewrite commits stale bytes, process B commits N+1 while A is paused, or the guard +self-contends/opens the forbidden second handle. ### T3 — exact four-step receipt and bytes @@ -432,9 +569,22 @@ Repeat backup absent→present with a valid non-routed backup and expect `preser repeat with malformed, unreadable, routed, symlinked, and ambiguous appearing backups and expect refusal before any replacement. -Broken mutation that must turn T3 red: set a receipt bit before replacement or catch -a failed replacement and continue; receipt and actual bytes diverge in at least one -table row. +The ordinary absent→present setup above remains useful but is not the no-clobber race +proof: it creates the backup before commit and would pass a broken +check-absent-then-overwriting-rename implementation. Add a publication barrier after +target/source validation and immediately before the exclusive publish operation. +While process A's hardened temp waits at that barrier, process B atomically creates +the destination and signals A to continue. A must receive `EEXIST`, validate and +preserve B's exact bytes, report `preserved`, and never call ordinary rename or +`atomicWriteFile`. Race two valid publishers from ABSENT and require exactly one +winner. Repeat the interleaving with malformed, unreadable, routed, symlinked, and +identity-ambiguous winners; A refuses without changing winner bytes and always +scrubs its unpublished temp. + +Broken mutations that must turn T3 red: set a receipt bit before replacement, catch a +failed replacement and continue, or replace exclusive publication with +check-absent-then-`atomicWriteFile`. Receipt/bytes diverge in a prefix row, or process +A overwrites process B's after-validation winner instead of preserving it. ### T4 — total adapter and route ordering @@ -464,25 +614,33 @@ one catalog mutation and observes one initialization. A route-level “zero call alone is not accepted because an eager static import would pass it. For reachability, run the symbol graph described in D. It must accept only the four -legacy rows and reject aliases, re-exports, wrappers, or dynamic imports from any new -root. +legacy writer rows and reject aliases, re-exports, wrappers, or dynamic imports from +any new root. The same symbol-resolved graph inventories gather filesystem +consultations: every `readFileSync`, `Bun.file`, `existsSync` branch, target +`lstat`/`stat`/`realpath`, or wrapper that reaches one must terminate at +`catalog/filesystem-evidence.ts`. Unresolved/computed edges fail closed. Broken mutations that must turn T5 red: add a static top-level management-convergence -import, alias a catalog writer into a management route, or replace a literal import -with a computed dynamic import. The sentinel or fail-closed graph must reject each. +import, alias a catalog writer into a management route, replace a literal import with +a computed dynamic import, or add an absence-only `existsSync`/target `realpath` +outside the evidence owner. The sentinel or fail-closed graph must reject each. ### T6 — precedence and native-pair exclusion -On the default catalog path, provide a bundled template that differs visibly from -catalog/backup/cache plus an on-disk routed/user-native row. Gather must use bundled -native template fields and preserve the on-disk merge row. Assert no materialized -fallback write. Snapshot the transition row before and after committed, stale, -refused, and failed catalog-only attempts; it must be byte/field identical and no -history schedule appears. Routing artifact spies stay zero. - -Broken mutations that must turn T6 red: move disk fallbacks ahead of bundled lookup, -request `CommitExpectation`, or call a routing writer. Template assertions, transition -row equality, or routing spies fail. +On the default catalog path, pre-populate the process-local bundled cache with a +template that differs visibly from catalog/backup/cache plus an on-disk +routed/user-native row. Gather must use cached bundled native template fields and +preserve the on-disk merge row without probing or launching Codex. Repeat cold: no +bundled cache plus a valid observed disk fallback succeeds without subprocess; no +bundled cache and no valid disk fallback returns `catalog-unavailable`. Assert no +materialized fallback write. Snapshot the transition row before and after committed, +stale, refused, and failed catalog-only attempts; it must be byte/field identical and +no history schedule appears. Routing artifact and executable-probe spies stay zero. + +Broken mutations that must turn T6 red: move disk fallbacks ahead of a populated +bundled cache, call `loadBundledCodexCatalog` to refill a cold cache, request +`CommitExpectation`, or call a routing writer. Template/cold-miss assertions, +executable spies, transition-row equality, or routing spies fail. ## Verification @@ -498,16 +656,17 @@ bun run privacy:scan bun --cwd docs-site run build ``` -Runtime probes use temporary homes and child processes only. They do not invoke -`ocx start`, `stop`, `sync`, `restore`, `ensure`, any `ocx service` command, or the -live proxy on port 10100. +T1 proves admission/gather launches no runtime probe or subprocess. Any unrelated +runtime fixture retained by the broader suites uses temporary homes and child +processes only. No verification invokes `ocx start`, `stop`, `sync`, `restore`, +`ensure`, any `ocx service` command, or the live proxy on port 10100. ## Accept criteria | Criterion | Proof | Concrete broken mutation that makes it red | |---|---|---| -| **C1** — gather is filesystem-write-free; commit is synchronous, fixed, one-shot, and receipt-exact | T1 + T3 | call a persisting resolver/read, add an `await` beneath commit, reorder replacements, pre-set a receipt bit, or replay a consumed candidate | -| **C2/C17** — config generation, every consumed source fingerprint, and target identity reject stale work before write, with only the valid create-once backup exception | T2 + T3 | in-place rewrite the same inode at the same generation, omit one selected fallback fingerprint, accept unreadable source, or overwrite an appearing backup | +| **C1** — gather is filesystem-write-free across user homes and scratch, performs no executable probe/subprocess, and commit is synchronous, fixed, one-shot, and receipt-exact | T1 + T3 | call cold `resolveCodexRuntime`/`loadBundledCodexCatalog`, add an `await` beneath commit, reorder replacements, pre-set a receipt bit, or replay a consumed candidate | +| **C2/C17** — the owner-held config generation, every closed PRESENT/ABSENT source observation, and target identity reject stale work before write; create-once backups publish atomically without clobber | T2 + T3 | omit ABSENT `config.toml`, release the transaction before callback, remove same-inode digest comparison, open a second SQLite observer, or replace exclusive publication with overwriting rename | | **Catalog/native boundary** — catalog-only never reads/advances the native pair or writes routing/history artifacts | T6 | call `expectation()`/`beginTransition`, add pair fields to `catalog-only`, or invoke config/profile/journal/history writer | | **Best-effort compatibility** — all 16 primary writes retain 2xx/201 and original follow-up order for every catalog failure | T4 | let lazy import/factory/admission throw, scope “zero writes” to the whole route, or return before Claude/Desktop follow-up | | **C14, WP9-bounded** — the 16 management roots reach catalog writers only through convergence; exactly four documented legacy roots remain until WP12 | T5 symbol graph | add a fifth root, hide one through alias/re-export/computed import, or accidentally require WP12 to have already removed `/api/sync`/startup/CLI/restore | From 9d7d3f7e0ce84ba81f9d5aecc5994f76665fe13a Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Tue, 4 Aug 2026 20:00:37 +0900 Subject: [PATCH 055/163] feat(codex): the source-evidence boundary the contract already promised MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit WP8b exists so that WP9 consumes shared surfaces rather than inventing them, and a reviewer found this one declared in the contract and absent from the code: no `CatalogSourceEvidence`, no `sourceEvidence` on the admission snapshot. WP9 would have had to amend WP8b's own types before it could use them, which is the failure this phase boundary is meant to prevent. The role-keyed shape lands as declared. The role set is closed because "refuse an incomplete list" was never enforceable over an array of present-file digests: a caller could omit the absent source that selected a fallback and leave no trace of the omission. Every conditional role is therefore a required key whose empty list means the role was not consulted, so absence is recorded rather than inferred — the same lesson this unit has now learned five times on disk. Capture reads `config.toml` once and hashes the same buffer it parsed; a separate pre-read would describe different bytes. Absent is a first-class observation carrying the canonical missing-leaf path and parent identity with a null file identity. `generation` becomes `ConfigGeneration` rather than a bare number, matching the contract. Only the types and the pre-gather capture land here. The gather/commit machinery and the single filesystem-evidence owner are WP9's, and nothing here is a placeholder that phase must replace. --- src/codex/catalog-admission.ts | 89 +++++++++++++++++++++++++-- src/codex/convergence-types.ts | 67 +++++++++++++++++++- tests/codex-catalog-admission.test.ts | 87 +++++++++++++++++++++++++- 3 files changed, 235 insertions(+), 8 deletions(-) diff --git a/src/codex/catalog-admission.ts b/src/codex/catalog-admission.ts index b6ad67bdb..fbcc31d31 100644 --- a/src/codex/catalog-admission.ts +++ b/src/codex/catalog-admission.ts @@ -7,23 +7,29 @@ * later phase. This reader therefore captures only the exact resident config, * its cooperating generation, and identities for catalog-owned targets. */ -import { realpathSync, statSync } from "node:fs"; -import { dirname, resolve } from "node:path"; +import { createHash } from "node:crypto"; +import { readFileSync, realpathSync, statSync } from "node:fs"; +import { basename, dirname, resolve } from "node:path"; import { readConfigGeneration } from "../config"; import type { OcxConfig } from "../types"; import type { CatalogAdmissionSnapshot, CatalogConvergeRequestInput, + CatalogFilesystemIdentity, + CatalogSourceObservation, ConvergeRequest, } from "./convergence-types"; import { + activeCodexConfigPath, + activeDefaultCatalogPath, activeCodexModelsCachePath, catalogBackupPathFor, isDefaultCatalogPath, legacyCatalogBackupPath, - readCodexCatalogPath, + resolveActiveCodexConfigPath, } from "./catalog/parsing"; +import { readRootTomlString } from "./paths"; /** * Construct the one request shape permitted for management catalog refreshes. @@ -76,6 +82,64 @@ function captureTargetIdentity(path: string): string { }); } +function catalogFilesystemIdentity( + entry: Readonly<{ dev: bigint; ino: bigint }>, +): CatalogFilesystemIdentity { + return { volume: String(entry.dev), fileId: String(entry.ino) }; +} + +function captureCatalogTargetSelection(): Readonly<{ + catalogPath: string; + observation: CatalogSourceObservation<"catalog-target-selection">; +}> { + const logicalPath = resolve(activeCodexConfigPath()); + const canonicalParent = realpathSync.native(dirname(logicalPath)); + const parent = statSync(canonicalParent, { bigint: true }); + const parentIdentity = { + canonicalPath: canonicalParent, + ...catalogFilesystemIdentity(parent), + }; + + let bytes: Buffer; + try { + bytes = readFileSync(logicalPath); + } catch (error) { + if (!error || typeof error !== "object" || !("code" in error) || error.code !== "ENOENT") { + throw error; + } + return { + catalogPath: activeDefaultCatalogPath(), + observation: { + state: "absent", + role: "catalog-target-selection", + logicalPath, + canonicalPath: resolve(canonicalParent, basename(logicalPath)), + parentIdentity, + fileIdentity: null, + }, + }; + } + + const canonicalPath = realpathSync.native(logicalPath); + const file = statSync(canonicalPath, { bigint: true }); + const configuredCatalogPath = readRootTomlString(bytes.toString("utf8"), "model_catalog_json"); + + return { + catalogPath: configuredCatalogPath + ? resolveActiveCodexConfigPath(configuredCatalogPath) + : activeDefaultCatalogPath(), + observation: { + state: "present", + role: "catalog-target-selection", + logicalPath, + canonicalPath, + parentIdentity, + fileIdentity: catalogFilesystemIdentity(file), + sha256: createHash("sha256").update(bytes).digest("hex"), + }, + }; +} + /** * Capture the catalog-only evidence WP9 can validate without consulting WP12. * The config reference is retained verbatim; no persisted config re-read may @@ -89,7 +153,8 @@ export function captureCatalogAdmissionSnapshot( throw new Error(`Cannot capture Codex catalog admission: config generation is ${generation.reason}.`); } - const catalogPath = readCodexCatalogPath(); + const targetSelection = captureCatalogTargetSelection(); + const catalogPath = targetSelection.catalogPath; const backupPaths = [ catalogBackupPathFor(catalogPath), ...(isDefaultCatalogPath(catalogPath) ? [legacyCatalogBackupPath()] : []), @@ -97,11 +162,25 @@ export function captureCatalogAdmissionSnapshot( return { config, - generation: generation.generation.value, + generation: generation.generation, targets: { catalog: captureTargetIdentity(catalogPath), cache: captureTargetIdentity(activeCodexModelsCachePath()), catalogBackups: backupPaths.map(captureTargetIdentity), }, + sourceEvidence: { + required: { + "catalog-target-selection": targetSelection.observation, + }, + conditional: { + "bundled-catalog-template": [], + "active-catalog-merge": [], + "hashed-backup-fallback": [], + "legacy-backup-fallback": [], + "models-cache-fallback": [], + "runtime-selection": [], + "provider-auth-selection": [], + }, + }, }; } diff --git a/src/codex/convergence-types.ts b/src/codex/convergence-types.ts index 4c8890fdb..6e9c8031c 100644 --- a/src/codex/convergence-types.ts +++ b/src/codex/convergence-types.ts @@ -332,15 +332,78 @@ export interface CodexCoordinatorTransactionController { close(): void; } -/** The minimal, working WP8b/WP9 snapshot; it authorizes catalog work only. */ +/** Why a filesystem observation influenced catalog preparation. Closed by contract. */ +export type CatalogRequiredSourceRole = "catalog-target-selection"; + +export type CatalogConditionalSourceRole = + | "bundled-catalog-template" + | "active-catalog-merge" + | "hashed-backup-fallback" + | "legacy-backup-fallback" + | "models-cache-fallback" + | "runtime-selection" + | "provider-auth-selection"; + +export type CatalogSourceRole = + | CatalogRequiredSourceRole + | CatalogConditionalSourceRole; + +/** Portable normalized identity: POSIX dev/inode or Windows volume/file id. */ +export interface CatalogFilesystemIdentity { + readonly volume: string; + readonly fileId: string; +} + +export interface CatalogParentIdentity extends CatalogFilesystemIdentity { + readonly canonicalPath: string; +} + +/** Exact gather-time evidence for one consulted filesystem source. */ +export type CatalogSourceObservation = + | { + readonly state: "present"; + readonly role: R; + readonly logicalPath: string; + readonly canonicalPath: string; + readonly parentIdentity: CatalogParentIdentity; + readonly fileIdentity: CatalogFilesystemIdentity; + /** Digest of the exact buffer returned to gather. */ + readonly sha256: string; + } + | { + readonly state: "absent"; + readonly role: R; + readonly logicalPath: string; + readonly canonicalPath: string; + readonly parentIdentity: CatalogParentIdentity; + readonly fileIdentity: null; + }; + +export type CatalogRequiredSourceObservations = Readonly<{ + [R in CatalogRequiredSourceRole]: CatalogSourceObservation; +}>; + +export type CatalogConditionalSourceObservations = Readonly<{ + [R in CatalogConditionalSourceRole]: readonly CatalogSourceObservation[]; +}>; + +export interface CatalogSourceEvidence { + readonly required: CatalogRequiredSourceObservations; + /** Every role is a required key; an empty list means the role was not consulted. */ + readonly conditional: CatalogConditionalSourceObservations; +} + +/** The shared WP8b/WP9 snapshot; it authorizes catalog work only. */ export interface CatalogAdmissionSnapshot { config: Readonly; - generation: number; + generation: ConfigGeneration; targets: Readonly<{ catalog: string; cache: string; catalogBackups: readonly string[]; }>; + /** Candidate-bound present/absent evidence, produced by the sole read owner. */ + sourceEvidence: CatalogSourceEvidence; } export interface AdmissionSnapshot { diff --git a/tests/codex-catalog-admission.test.ts b/tests/codex-catalog-admission.test.ts index 6930bca34..8d955bb0d 100644 --- a/tests/codex-catalog-admission.test.ts +++ b/tests/codex-catalog-admission.test.ts @@ -1,4 +1,5 @@ import { afterEach, beforeEach, expect, test } from "bun:test"; +import { createHash } from "node:crypto"; import { mkdirSync, mkdtempSync, @@ -12,9 +13,39 @@ import { tmpdir } from "node:os"; import { join } from "node:path"; import { captureCatalogAdmissionSnapshot } from "../src/codex/catalog-admission"; +import type { + CatalogConditionalSourceObservations, + CatalogConditionalSourceRole, + CatalogRequiredSourceObservations, + CatalogRequiredSourceRole, + CatalogSourceEvidence, +} from "../src/codex/convergence-types"; import { saveConfig } from "../src/config"; import type { OcxConfig } from "../src/types"; +const CONDITIONAL_SOURCE_ROLES = [ + "active-catalog-merge", + "bundled-catalog-template", + "hashed-backup-fallback", + "legacy-backup-fallback", + "models-cache-fallback", + "provider-auth-selection", + "runtime-selection", +] as const satisfies readonly CatalogConditionalSourceRole[]; + +type IsAssignable = [From] extends [To] ? true : false; +type MissingRequiredEvidence = Omit & Readonly<{ + required: Omit; +}>; +type MissingConditionalEvidence = Omit & Readonly<{ + conditional: Omit; +}>; + +const STRUCTURALLY_INVALID_EVIDENCE_ASSIGNABILITY: readonly [ + IsAssignable, + IsAssignable, +] = [false, false]; + let testRoot = ""; let codexHome = ""; let opencodexHome = ""; @@ -55,7 +86,7 @@ test("captures the given config reference, generation, and catalog target identi expect(snapshot.config).toBe(residentConfig); expect(snapshot.config.port).toBe(30300); - expect(snapshot.generation).toBe(1); + expect(snapshot.generation).toEqual({ value: 1 }); expect(JSON.parse(snapshot.targets.catalog)).toMatchObject({ path: join(codexHome, "opencodex-catalog.json"), canonicalParent: codexHome, @@ -68,6 +99,60 @@ test("captures the given config reference, generation, and catalog target identi fileIdentity: { device: expect.any(String), inode: expect.any(String) }, }); expect(snapshot.targets.catalogBackups).toHaveLength(2); + + expect(snapshot.sourceEvidence.required["catalog-target-selection"]).toEqual({ + state: "absent", + role: "catalog-target-selection", + logicalPath: join(codexHome, "config.toml"), + canonicalPath: join(codexHome, "config.toml"), + parentIdentity: { + canonicalPath: codexHome, + volume: expect.any(String), + fileId: expect.any(String), + }, + fileIdentity: null, + }); + expect(Object.keys(snapshot.sourceEvidence.conditional).sort()).toEqual(CONDITIONAL_SOURCE_ROLES); + for (const observations of Object.values(snapshot.sourceEvidence.conditional)) { + expect(observations).toEqual([]); + } +}); + +test("captures PRESENT catalog target-selection evidence from the exact config bytes", () => { + const selectedCatalog = join(codexHome, "selected-catalog.json"); + const configBytes = Buffer.from( + `model_catalog_json = ${JSON.stringify(selectedCatalog)}\n`, + "utf8", + ); + writeFileSync(join(codexHome, "config.toml"), configBytes); + + const snapshot = captureCatalogAdmissionSnapshot(config()); + const observation = snapshot.sourceEvidence.required["catalog-target-selection"]; + + expect(JSON.parse(snapshot.targets.catalog)).toMatchObject({ path: selectedCatalog }); + if (observation.state !== "present") { + throw new Error(`Expected PRESENT target-selection evidence, received ${observation.state}.`); + } + expect(observation).toEqual({ + state: "present", + role: "catalog-target-selection", + logicalPath: join(codexHome, "config.toml"), + canonicalPath: join(codexHome, "config.toml"), + parentIdentity: { + canonicalPath: codexHome, + volume: expect.any(String), + fileId: expect.any(String), + }, + fileIdentity: { + volume: expect.any(String), + fileId: expect.any(String), + }, + sha256: createHash("sha256").update(configBytes).digest("hex"), + }); +}); + +test("rejects missing required and conditional evidence keys structurally", () => { + expect(STRUCTURALLY_INVALID_EVIDENCE_ASSIGNABILITY).toEqual([false, false]); }); test("changes target identity when a parent symlink retargets without changing the path", () => { From f51404f1670e57bdfb1637adf2d5d0cb5aeaf1f5 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Tue, 4 Aug 2026 20:00:49 +0900 Subject: [PATCH 056/163] test(codex): an allowlist of three aliases is not a property MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The property conversion moved the literal up one level instead of removing it. The bare-combo-alias test enumerated `deepseek-v4-flash`, `old-public` and `stable-public` from `exactComboCatalogSlugs`, so a classifier that recognized slash-bearing rows plus exactly those three names kept the file green — while real `syncCatalogModels` emitted `round4-edge-bare` and got classified clean. Users configure combo aliases freely; the set is unbounded, so no corpus can stand in for the rule. The oracle is now the structural authorship signature. The alias is generated per run with a UUID suffix and written by the real catalog writer, and the combo's physical provider row is disabled so the bare alias is the only routed row present — otherwise a slash-bearing sibling satisfies the assertion and the test passes for a reason unrelated to what it claims to prove. No fixed allowlist can satisfy it, because the name does not exist until the test runs. The foreign-slug corpus and the catalog leaf names got the same treatment for the same reason. The enumerations left alone are genuinely finite: surfaces, TOML types, error classes. --- tests/codex-native-residue.test.ts | 131 +++++++++++++++-------------- 1 file changed, 69 insertions(+), 62 deletions(-) diff --git a/tests/codex-native-residue.test.ts b/tests/codex-native-residue.test.ts index 044b2dfd6..4e7c58321 100644 --- a/tests/codex-native-residue.test.ts +++ b/tests/codex-native-residue.test.ts @@ -1,10 +1,11 @@ import { afterEach, beforeEach, expect, test } from "bun:test"; -import { createHash } from "node:crypto"; +import { createHash, randomUUID } from "node:crypto"; import { chmodSync, lstatSync, mkdirSync, mkdtempSync, + readFileSync, realpathSync, rmSync, symlinkSync, @@ -17,7 +18,6 @@ import { Database } from "bun:sqlite"; import { buildCatalogEntries, - exactComboCatalogSlugs, readCodexCatalogPath, syncCatalogModels, } from "../src/codex/catalog"; @@ -300,28 +300,28 @@ for (const shape of catalogConfigShapes) { const catalogPathShapes: Array<{ name: string; - configuredPath: (outsideRoot: string) => string; + configuredPath: (outsideRoot: string, leaf: string) => string; }> = [ - { name: "root-relative", configuredPath: () => "custom.json" }, - { name: "nested-relative", configuredPath: () => "nested/routes.json" }, + { name: "root-relative", configuredPath: (_outsideRoot, leaf) => `${leaf}.json` }, + { name: "nested-relative", configuredPath: (_outsideRoot, leaf) => `nested/${leaf}.json` }, { name: "absolute inside CODEX_HOME", - configuredPath: () => canonicalPathInCodexHome("absolute-direct.json"), + configuredPath: (_outsideRoot, leaf) => canonicalPathInCodexHome(`${leaf}.json`), }, { name: "absolute outside CODEX_HOME", - configuredPath: outsideRoot => join(outsideRoot, "external.json"), + configuredPath: (outsideRoot, leaf) => join(outsideRoot, `${leaf}.json`), }, { name: "parent-escaping relative", - configuredPath: () => `../${basename(codexHome)}-escape.json`, + configuredPath: (_outsideRoot, leaf) => `../${basename(codexHome)}-${leaf}.json`, }, ]; for (const shape of catalogPathShapes) { test(`configured catalog classification follows the ${shape.name} path`, () => { const outsideRoot = mkdtempSync(join(tmpdir(), "ocx-native-residue-catalog-outside-")); - const configuredPath = shape.configuredPath(outsideRoot); + const configuredPath = shape.configuredPath(outsideRoot, `catalog-${randomUUID()}`); const targetPath = resolve(realpathSync.native(codexHome), configuredPath); try { mkdirSync(dirname(targetPath), { recursive: true }); @@ -495,65 +495,72 @@ test("duplicate configured catalog paths are indeterminate", () => { }); }); -const comboAliasConfig = { - combos: { - deepseek: { alias: "deepseek-v4-flash", targets: [{ provider: "fixture", model: "one" }] }, - old: { alias: "old-public", targets: [{ provider: "fixture", model: "two" }] }, - stable: { alias: "stable-public", targets: [{ provider: "fixture", model: "three" }] }, - }, -}; -const productionBareComboAliases = [...exactComboCatalogSlugs(comboAliasConfig)] - .filter(alias => !alias.includes("/")); - -for (const alias of productionBareComboAliases) { - test(`production-derived bare combo alias ${alias} is routed residue`, () => { - const [id] = Object.entries(comboAliasConfig.combos) - .find(([, combo]) => combo.alias === alias)!; - const models = buildCatalogEntries( - null, - [], - [{ provider: "combo", id, alias, owned_by: "combo" }], - undefined, - false, - "default", - exactComboCatalogSlugs(comboAliasConfig), - ); - const combo = models.find(model => model.slug === alias); - expect(combo).toMatchObject({ - slug: alias, +const arbitraryComboAlias = `round4-edge-bare-${randomUUID()}`; + +test(`production-generated arbitrary bare combo alias ${arbitraryComboAlias} is routed residue`, async () => { + const catalogPath = canonicalPathInCodexHome("opencodex-catalog.json"); + writeFileSync(catalogPath, JSON.stringify({ models: [] })); + const config: OcxConfig = { + port: 10100, + defaultProvider: "fixture", + providers: { + fixture: { + adapter: "openai-chat", + baseUrl: "https://fixture.invalid/v1", + liveModels: false, + models: ["combo-member"], + modelContextWindows: { "combo-member": 128_000 }, + }, + }, + disabledModels: ["fixture/combo-member"], + combos: { + edge: { + alias: arbitraryComboAlias, + targets: [{ provider: "fixture", model: "combo-member" }], + }, + }, + }; + + const sync = await syncCatalogModels(config); + const catalog = JSON.parse(readFileSync(catalogPath, "utf8")) as { + models: Array>; + }; + const routedRows = catalog.models.filter(model => + typeof model.description === "string" + && model.description.startsWith("Routed via opencodex → ") + ); + + expect(sync).toMatchObject({ path: catalogPath, catalogWritten: true }); + expect(arbitraryComboAlias).not.toContain("/"); + expect(routedRows).toEqual([ + expect.objectContaining({ + slug: arbitraryComboAlias, description: "Routed via opencodex → combo (combo).", owned_by: "combo", - }); - writeFileSync(pathInCodexHome("opencodex-catalog.json"), JSON.stringify({ models: [combo] })); - - expect(classifyNativeRoutedResidue()).toMatchObject({ - kind: "residue", - surface: "catalog", - }); - expect(readCodexTransitionState()).toEqual({ - kind: "legacy-ambiguous", - message: "A missing coordinator row cannot be initialized while native Codex routing residue exists.", - }); + }), + ]); + expect(classifyNativeRoutedResidue()).toMatchObject({ + kind: "residue", + surface: "catalog", }); -} + expect(readCodexTransitionState()).toEqual({ + kind: "legacy-ambiguous", + message: "A missing coordinator row cannot be initialized while native Codex routing residue exists.", + }); +}); -for (const slug of [ - "user/model", - "vendor/deepseek-v4-flash", - "local/vision_2", - "acme/text-pro", -]) { - test(`foreign slash-bearing slug ${slug} is indeterminate`, () => { - writeFileSync(pathInCodexHome("opencodex-catalog.json"), JSON.stringify({ - models: [{ slug, description: "User-authored catalog row" }], - })); +const arbitraryForeignSlug = `foreign-${randomUUID()}/model-${randomUUID()}`; - expect(classifyNativeRoutedResidue()).toMatchObject({ - kind: "indeterminate", - surface: "catalog", - }); +test(`arbitrary foreign slash-bearing slug ${arbitraryForeignSlug} is indeterminate`, () => { + writeFileSync(pathInCodexHome("opencodex-catalog.json"), JSON.stringify({ + models: [{ slug: arbitraryForeignSlug, description: "User-authored catalog row" }], + })); + + expect(classifyNativeRoutedResidue()).toMatchObject({ + kind: "indeterminate", + surface: "catalog", }); -} +}); test("a native-tagged history row with routed latest rollout metadata refuses coordinator initialization", () => { createHistoryDatabase("openai", ["openai", "opencodex"]); From baa7eae173e41cee1ea74aa2fe936de53c48c403 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Tue, 4 Aug 2026 20:26:49 +0900 Subject: [PATCH 057/163] docs(substrate): the writers we chose to keep were never excluded MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Audit round 3 found the config transaction guards the wrong thing. It excludes cooperating CONFIG mutations, but catalog bytes are written by four first-party paths WP9 deliberately retains, and none of them acquire it. So: process A gathers from catalog X and validates everything inside the transaction, process B runs the retained /api/sync chain (refresh.ts:40 → catalog/sync.ts:568,600) and writes catalog Y, and A then replaces Y with bytes derived from X and truthfully reports committed. B never advanced the config generation and never joined A's lock. The contract's "cooperating writers are prevented" row was simply false for catalog bytes, and no test caught it because none interleaved a writer we had decided to keep. Catalog serialization is now its own permanent CODEX_HOME-keyed lock, held by the convergence commit and by all four retained roots. It is not WP11's native lock and not a placeholder: WP11 wraps routing writes, a different surface. Order is N → K → C over a database distinct from N, with C → K, K → N and a held N → H forbidden, so the reviewer could show there is no cycle rather than assert it. Two more authorities were being trusted without evidence. Runtime identity and the bundled template can come from process-local caches that are genuinely mutable (runtime.ts:362, bundled.ts:42,179), and persisted runtime writes do not advance the config generation (runtime.ts:213) — so a candidate could be built from R1/B1, have authority move to R2/B2 mid-gather, and commit with every filesystem check still matching. codex-runtime.json is now a required PRESENT/ABSENT observation even on a warm hit, and both caches carry an epoch captured at gather and revalidated before the first write. And the selector that chooses CODEX_HOME was never bound to the candidate. Production canonicalizes it on every call (catalog/parsing.ts:52) and derives config, catalog, cache and relative targets from the result (:63). Retarget the home symlink once between gather and commit and the old config.toml is still present and unchanged, so the candidate re-validates cleanly and writes into the abandoned home while Codex reads the new one. That is single-direction drift, not the A→B→A case C17 excludes, so it is now required evidence: selector, canonical path, and root identity, re-resolved under the held lock. C17 is narrowed to match what the mechanism actually proves. --- .../005_contract.md | 336 +++++++++++++---- .../010_catalog_seam.md | 345 ++++++++++++------ 2 files changed, 503 insertions(+), 178 deletions(-) diff --git a/devlog/_plan/260804_codex_write_substrate/005_contract.md b/devlog/_plan/260804_codex_write_substrate/005_contract.md index 7107add3c..ceb028ce3 100644 --- a/devlog/_plan/260804_codex_write_substrate/005_contract.md +++ b/devlog/_plan/260804_codex_write_substrate/005_contract.md @@ -608,9 +608,11 @@ outcomes carry both (`src/codex/convergence-types.ts:207-224`). That is an honest reduction in protection: a catalog-only commit is not guarded against staleness by the native pair. Its independent protection is the per-source -observation check below. A catalog-only commit must never write a routing artifact; -if a future phase needs to write one, it uses `scope:"full"` and publishes the -native transition plus its truthful history schedule. +observation check below plus the catalog serialization primitive that excludes every +first-party catalog/backup/cache writer through publication. A catalog-only commit +must never write a routing artifact; if a future phase needs to write one, it uses +`scope:"full"` and publishes the native transition plus its truthful history +schedule. WP8b adds executable `readConfigGeneration`, `bumpConfigGeneration`, and `withExpectedConfigGenerationSync` exports to `src/config.ts` with the callable @@ -644,6 +646,66 @@ consumer. This guard uses the config mutation lock that exists in WP8b and has n dependency on WP11's future native lock. Catalog-only work validates but does not bump the config generation because it writes no persisted OpenCodex config bytes. +### Catalog serialization is a permanent, separate primitive (seam audit round 3) + +The round-3 auditor ran the retained management `POST /api/sync` chain while a +catalog candidate was paused after validation. `refreshCodexModelCatalog` +(`src/codex/refresh.ts:40-52`) still reaches direct catalog replacement +(`src/codex/catalog/sync.ts:568`) and models-cache replacement +(`src/codex/catalog/sync.ts:600-616`). That writer neither advances config generation +nor enters `withExpectedConfigGenerationSync`, so the config transaction alone let it +publish Y before convergence resumed and replaced Y with bytes gathered from X. This +is a first-party writer retained by this plan, not a foreign hand edit that the +contract may merely detect. + +WP9 therefore lands `src/codex/catalog-write-serialization.ts`, a cross-process +**catalog serialization primitive K** keyed by effective user plus canonical +`CODEX_HOME`. K uses its own private SQLite database returned by +`resolveCodexCatalogSerializationDatabasePath`, `busy_timeout=0`, and a +`BEGIN IMMEDIATE` transaction with bounded outer retry. Process exit releases the +transaction. It is a different database and ownership surface from the coordinator +transaction N and from `config-mutation.sqlite`; sharing either database would make +the required nesting self-contend or silently key exclusion by `OPENCODEX_HOME`. + +K is permanent. It is **not** WP11's native write lock, does not read or advance the +native pair, and is not a placeholder that WP11 removes. WP11 later wraps native +routing writes in N; catalog, backup, and cache publication remains a different +surface and continues to require K after WP11 and after the four transitional roots +are removed. + +K exposes a synchronous owner-held callback acquired before any config transaction; +that callback receives an opaque, non-forgeable catalog-write permit. Every low-level +catalog/hashed-backup/legacy-backup/models-cache mutator requires that permit. The +callback may enter `withExpectedConfigGenerationSync`, but performs no provider +request, runtime probe, OAuth refresh, subprocess, Promise, or other awaited work. +Gather stays outside K. At WP9, convergence's outer async orchestration may retry a +fail-fast K acquisition within `deadlineMs`, but each acquisition attempt and the +complete K -> C publication callback are synchronous. Each of the four retained +writer chains keeps its public signature, return shape, gather order, and ordinary +behavior; its actual synchronous replacement section enters K and supplies the +permit. Lock busy/unavailable follows that retained function's existing no-write or +write-failure path rather than changing it to a Promise. No retained chain holds K +while gathering. + +The global order is: + +```text +native/coordinator transaction N, when present + -> catalog serialization K, when catalog/backup/cache bytes may be written + -> config transaction C, when generation admission is required + -> synchronous validation and artifact writes + -> release C + -> commit N while K is still held for a full transition, then release K +-> release N +``` + +Thus catalog-only work takes `K -> C`; a retained writer that is already under N +takes `N -> K`; and full convergence takes `N -> K -> C`. There is no `C -> K`, no +`K -> N`, and no catalog path acquires history H. Using the already-open N capability +to publish/commit its row while K is held is not a new acquisition edge. Config-owned +callbacks remain forbidden from calling either N or K. A graph test protects those +negative edges. This extends, rather than reverses, the settled N -> C discipline. + ### The expected transition ```ts @@ -666,7 +728,7 @@ The rule, stated so a test can check it: The earlier “there is no window” claim was wrong. Process exclusion cannot make separate file replacements and the coordinator-row update atomic. Holding -native + config coordination provides **no cooperating interleaving while the +N + K + C provides **no cooperating native/config/catalog interleaving while the process is alive**; a crash can still leave any prefix of the artifact sequence with the old coordinator pair. @@ -682,25 +744,29 @@ idempotence is required but is not described as filesystem atomicity. Catalog-on staleness is instead admitted by the source observations below, not retroactively described as protection by a pair it never advanced. -### Prevention for cooperating writers (round 2 #5, seam audit round 2) +### Prevention for cooperating writers (round 2 #5, seam audit rounds 2-3) C2 says a stale candidate **cannot be committed**. Detect-after-commit permits exactly the write C2 forbids. Full routing work later follows `030`'s N -> C lock order. WP9 catalog-only work must not acquire that future native lock, so its fix is -the owner-side config-generation guard that exists independently of WP11. +the permanent K -> C composition: catalog serialization plus the owner-side +config-generation guard, both independent of WP11. -The first amendment named the lock but not an API that held it through catalog -publication. The auditor's N -> N+1 interleaving was therefore a cooperating writer -the text promised to prevent. Catalog commit now enters through +The first amendment named the config lock but not an API that held it through catalog +publication. The auditor's N -> N+1 interleaving was therefore a cooperating config +writer the text promised to prevent. Catalog commit enters K first and then `withExpectedConfigGenerationSync`: generation validation and the complete -synchronous catalog commit execute inside one already-held config transaction. +synchronous catalog commit execute inside one already-held config transaction while +the catalog permit remains live. The round-3 retained-writer interleaving is excluded +by K even though that writer does not cooperate with config generation. So: | Writer | Mechanism | |---|---| -| cooperating (ours) | **prevented** — `withExpectedConfigGenerationSync` holds the existing config transaction through validation and synchronous commit | -| non-cooperating (hand edit, foreign tool) | **detected** after the fact, reported `deferred` | +| cooperating config writer (ours) | **prevented** — K remains held while `withExpectedConfigGenerationSync` holds C through validation and synchronous commit | +| cooperating catalog/backup/cache writer (ours, including all four WP9 transitional roots) | **prevented** — every real replacement requires the same CODEX_HOME-keyed K permit | +| non-cooperating (hand edit, foreign tool) | **detected** when its drift is visible at final revalidation, reported `deferred`; a write after that check remains outside the claim | Re-gather is bounded by `deadlineMs`. On expiry the outcome is `deferred` with a typed reason and another convergence is scheduled — the retry loop terminates on @@ -719,14 +785,18 @@ stale candidate. Path, canonical parent, parent identity, file identity, config generation, and native pair all remained unchanged. Target identity says where a write will land; it does not say that the bytes gather consumed are still current. -The catalog admission snapshot therefore retains a closed set of **role-bearing -source observations** for every filesystem source whose presence, absence, or -bytes influenced preparation. The active catalog, each consulted hashed/legacy -backup or models-cache fallback, runtime/auth selection state, and any later file -source all identify why they were consulted. Most importantly, -`$CODEX_HOME/config.toml` is the required `catalog-target-selection` observation. -Its ABSENCE selects the default catalog path, so absence is evidence and is -recorded even though there is no byte buffer to hash. +The catalog admission snapshot therefore retains a required **catalog-home +selection observation** plus a closed set of **role-bearing source observations** +for every filesystem source whose presence, absence, or bytes influenced +preparation. Home selection records whether the raw selector came from the +`CODEX_HOME` environment value or the default resolver, the uncanonicalized raw +selector string, the resulting canonical CODEX_HOME path, and that root directory's +filesystem identity. The active catalog, each consulted hashed/legacy backup or +models-cache fallback, runtime/auth selection state, and any later file source all +identify why they were consulted. `$CODEX_HOME/config.toml` remains the required +`catalog-target-selection` observation. Its ABSENCE selects the default catalog +path, so absence is evidence and is recorded even though there is no byte buffer to +hash. For a present source, the observation owner computes SHA-256 from the **same exact buffer** it returns. A separate pre-read is not equivalent. For an absent source, @@ -740,6 +810,39 @@ ambiguous identity is refused rather than assumed unchanged. Thus a config file that appears after default-target gather cannot authorize a commit to the obsolete default target even when that target's own parent, inode, and bytes never moved. +The same under-K-and-C check first re-reads the current raw environment/default +selector, re-runs the production home resolver, and compares selector kind, raw +string, canonical home, and root identity with the captured observation. It then +derives again, from that re-resolved home, `config.toml`, +the default catalog, models cache, and every relative configured catalog target, and +recomputes the catalog/backup/cache target set. Every derived target must equal the +candidate's logical and canonical target evidence before any write. A selector +symlink retarget from A to B is therefore `stale` even if A's `config.toml`, catalog, +parent, and inode remain byte-for-byte unchanged. The accepted A -> B -> A exclusion +still applies only when the complete excursion returns to identical selector, root, +source, and target evidence before this check. + +Process-local runtime and bundled-catalog memos are not filesystem observations, but +round 3 proved they are still authority when their values influence a candidate. Each +owner therefore maintains a process-lifetime monotonic epoch and an immutable value +identity. Population, replacement, clear, invalidation, persisted-runtime write, and +test reset increment the applicable epoch before exposing the new state; epochs are +never reset or reused, and memo values are immutable. The exact epoch and value +identity returned with a consumed runtime or bundled template are sealed into the +private candidate and compared with the owner's current pair under K and C before +the first write. Any change, including invalidate-and-repopulate with byte-identical +content, is `stale`. + +When runtime identity influences a candidate, `codex-runtime.json` is additionally +an **always-required** `runtime-selection` filesystem observation, PRESENT or ABSENT, +even when the runtime came from a warm process memo. A PRESENT persisted selection is +parsed from the exact observed buffer and a warm runtime is usable only if its +identity agrees; ABSENT permits a matching warm runtime but binds that absence to the +candidate. A later process that creates, replaces, or removes the file is therefore +caught by source re-observation even though it cannot advance this process's epoch. +The bundled memo uses its epoch/value identity directly; no fake filesystem source is +invented for an in-memory value. + This is deliberately per-source evidence from one gather. It is not the deleted `ContentRevision` design, does not hash the whole persisted configuration, and does not turn content into a global revision or transition authority. That rejected @@ -816,6 +919,27 @@ export interface CatalogParentIdentity extends CatalogFilesystemIdentity { readonly canonicalPath: string; } +/** Required evidence for the selector that chose every CODEX_HOME-derived path. */ +export interface CatalogHomeSelectionObservation { + readonly selector: Readonly<{ + readonly kind: "environment" | "default"; + /** Exact pre-canonicalization selector string used by the production resolver. */ + readonly raw: string; + }>; + readonly canonicalCodexHome: string; + readonly rootIdentity: CatalogFilesystemIdentity; +} + +export type CatalogProcessLocalObservation = + | { readonly state: "unused" } + | { readonly state: "used"; readonly epoch: number; readonly valueIdentity: string }; + +/** Candidate-bound evidence for mutable process-local authority, never file evidence. */ +export interface CatalogProcessLocalEvidence { + readonly runtime: CatalogProcessLocalObservation; + readonly bundledCatalog: CatalogProcessLocalObservation; +} + /** Exact gather-time evidence for one consulted filesystem source. */ export type CatalogSourceObservation = | { @@ -846,6 +970,8 @@ export type CatalogConditionalSourceObservations = Readonly<{ }>; export interface CatalogSourceEvidence { + /** Required before any CODEX_HOME-derived target or source path is accepted. */ + readonly homeSelection: CatalogHomeSelectionObservation; readonly required: CatalogRequiredSourceObservations; /** Every role is a required key; an empty list means the role was not consulted. */ readonly conditional: CatalogConditionalSourceObservations; @@ -895,13 +1021,15 @@ export interface AdmissionSnapshot { The role set is closed because “refuse an incomplete list” was not enforceable on an array of present-file digests: a caller could omit the absent source that selected a default and leave no evidence of the omission. Pre-gather capture now starts with -the required `catalog-target-selection` observation for the logical -`$CODEX_HOME/config.toml` path, PRESENT or ABSENT, and with every conditional role -key present as an empty list. Gather does not mutate that snapshot. It returns the -prepared candidate with an immutable copy whose conditional lists contain every -filesystem consultation in order, including absent alternatives that caused a -fallback. A missing required role or conditional role key is structurally invalid; -commit accepts only the private candidate-bound `CatalogSourceEvidence`. +the required `homeSelection`, the required `catalog-target-selection` observation +for the logical `$CODEX_HOME/config.toml` path, PRESENT or ABSENT, and every +conditional role key present as an empty list. Gather does not mutate that snapshot. +It returns the prepared candidate with an immutable copy whose conditional lists +contain every filesystem consultation in order, including absent alternatives that +caused a fallback, plus sealed `CatalogProcessLocalEvidence`. A missing home +selection, required role, conditional role key, required runtime-state observation, +or used-cache epoch/value identity is structurally invalid; commit accepts only the +private candidate-bound evidence. All gather filesystem reads route through the one evidence-producing owner, `src/codex/catalog/filesystem-evidence.ts`. It owns an opaque gather-evidence @@ -909,7 +1037,7 @@ session: source reads append the matching PRESENT or ABSENT observation before returning, while target probes append the existing target parent/file identity evidence. Callers never append, remove, or reconstruct evidence arrays themselves; only the owner can seal the complete session into the private candidate, and sealing -requires the `catalog-target-selection` role. Raw `readFileSync`, `Bun.file`, +requires home selection plus the `catalog-target-selection` role. Raw `readFileSync`, `Bun.file`, `existsSync`-then-read, `lstat`/`realpath` target probing, catalog helper, or indirect wrapper that consults the filesystem outside that owner is a contract violation. The symbol-granular graph test follows imports, aliases, re-exports, wrappers, and @@ -927,23 +1055,29 @@ different authority sources and must not be collapsed: config parameter with which to substitute another authority. Catalog admission does not independently reconstruct full config from disk. This deliberate capture is what prevents a route from substituting catalog authority. Admission - separately observes config generation, targets, and the required - `$CODEX_HOME/config.toml` - `catalog-target-selection` role before gather. + separately observes config generation, raw/default home selection plus canonical + home/root identity, targets, and the required `$CODEX_HOME/config.toml` + `catalog-target-selection` role before gather. If runtime identity later + influences the candidate, gather records `codex-runtime.json` PRESENT or ABSENT + and seals the runtime/bundled memo epoch and immutable value identity actually + consumed. 2. **Catalog under-lock (WP9):** + acquire K, then `withExpectedConfigGenerationSync(snapshot.generation, commit)` validates the generation through the already-held config transaction and runs the complete - synchronous catalog commit before release. Commit re-observes and compares every - candidate-bound `sourceEvidence` entry and target identity immediately before its - first write. It does not read or advance a native `CommitExpectation`. + synchronous catalog commit before C and K release. Commit re-resolves and compares + home selection/root identity and every derived target, re-observes every + candidate-bound `sourceEvidence` entry, and revalidates each used process-local + epoch/value identity immediately before its first write. It does not read or + advance a native `CommitExpectation`. 3. **Full pre-gather/under-lock/post-commit (WP12):** full admission reads persisted config and every authority/target field into snapshot A, fully re-reads snapshot B - under native + config coordination, and post-commit re-reads persisted config plus - every native/catalog/history surface into `CodexObservedState`. It never uses the - server's long-lived object as a persisted-config fallback. Missing, unreadable, or - invalid persisted config produces unknown/refusal. The outcome is not `converged` - unless final observation agrees with admitted intent and the exact expected native - pair. + under N -> K -> C coordination whenever catalog bytes may be written, and + post-commit re-reads persisted config plus every native/catalog/history surface + into `CodexObservedState`. It never uses the server's long-lived object as a + persisted-config fallback. Missing, unreadable, or invalid persisted config + produces unknown/refusal. The outcome is not `converged` unless final observation + agrees with admitted intent and the exact expected native pair. `010`'s independent gather-time `readConfigDiagnostics()` remains removed: WP9 has the captured management reference plus generation/source evidence, while WP12's @@ -1097,15 +1231,21 @@ export type ResolveCodexCoordinatorDatabasePath = ( identity: UserIdentity, canonicalCodexHome: string, ) => string; + +/** Return K's FINAL database path; this is never the native coordinator path. */ +export type ResolveCodexCatalogSerializationDatabasePath = ( + identity: UserIdentity, + canonicalCodexHome: string, +) => string; ``` -WP8b implements and exports constants of both function types from -`src/codex/user-identity.ts`; it does not ship declarations without bodies. -`resolveCodexCoordinatorDatabasePath` is the **one exported path resolver**. -Its private helpers may resolve/validate the runtime root, but WP11, transition -state, history, tests and cleanup consume the returned database path verbatim. -No consumer appends `opencodex`, `native-write-locks`, `v1`, uid/SID, the home -digest, or `.sqlite` a second time. +WP8b implements and exports constants of the identity and coordinator function types +from `src/codex/user-identity.ts`; it does not ship declarations without bodies. WP9 +adds the catalog-serialization resolver there with K. These are the only two exported +final-path resolvers; the private secure-root resolver is shared but is never exported +for consumer path composition. WP11, transition state, history, tests, cleanup, and K +consume their returned database path verbatim. No consumer appends `opencodex`, a +lock-directory name, `v1`, uid/SID, the home digest, or `.sqlite` a second time. Exact platform algorithm: @@ -1130,18 +1270,23 @@ Exact platform algorithm: ProgramData fallback. The final path returned by `resolveCodexCoordinatorDatabasePath` is -`/native-write-locks/.sqlite`. +`/native-write-locks/.sqlite`; +the final path returned by `resolveCodexCatalogSerializationDatabasePath` is the +distinct sibling +`/catalog-write-locks/.sqlite`. POSIX directories are `0700` and files `0600`; Windows applies the required ACL to -the root, database, and rollback journal. Every existing component is checked before -use and again through stable descriptors around SQLite open/transaction boundaries. -A symlink, junction/reparse redirect, wrong owner, broad mode/ACL, or substituted -path is a refusal, never something the resolver repairs in place. +the root, databases, and rollback journals. Every existing component is checked +before use and again through stable descriptors around SQLite open/transaction +boundaries. A symlink, junction/reparse redirect, wrong owner, broad mode/ACL, or +substituted path is a refusal, never something the resolver repairs in place. The test that matters, and the one my first version could not have failed: two child processes with different `HOME`, `USERPROFILE`, `TMPDIR`, `XDG_RUNTIME_DIR`, `TEMP`, `TMP`, and `LOCALAPPDATA` values but the same effective uid/SID and canonical -`CODEX_HOME` must resolve the same **final database path**, take the same lock, and -read/update the same singleton transition row. +`CODEX_HOME` must resolve the same **two final database paths**, take the same N and K +locks respectively, and read/update the same singleton transition row. The two paths +must differ so nested `N -> K` cannot self-contend on SQLite's database-wide writer +slot. ## 8. Names @@ -1150,6 +1295,7 @@ Audit #13. Fixed here so no phase invents a variant: | Thing | Module | |---|---| | the native write lock | `src/codex/codex-write-lock.ts` | +| the catalog serialization primitive K | `src/codex/catalog-write-serialization.ts` | | the record | `src/codex/integration-record.ts` | | the entry point | `src/codex/convergence.ts` | | generations | `src/codex/generation.ts` | @@ -1169,7 +1315,7 @@ already migrated lifecycle and explicit callers: |---|---|---| | native config/profile | `src/codex/internal/native-writer.ts` | `src/codex/convergence.ts` only | | injection journal create/mark/restore/remove | `src/codex/internal/journal-writer.ts` | `src/codex/convergence.ts` only | -| catalog, hashed/legacy backups, models cache | `src/codex/internal/catalog-writer.ts` | `src/codex/convergence.ts` only | +| catalog, hashed/legacy backups, models cache | `src/codex/internal/catalog-writer.ts`, each mutation requiring K's opaque permit | `src/codex/convergence.ts` only | | history DB rows, manifest, rollout files | history write exports in `src/codex/internal/history-writer.ts` | `src/codex/history-worker.ts` only | | transition pair and history schedule/terminal row | `src/codex/transition-state.ts` | `src/codex/convergence.ts` and `src/codex/history-worker.ts` only | | JSON provenance ledger | `updateIntegrationRecord` in `src/codex/integration-record.ts` | `src/codex/convergence.ts` only | @@ -1185,16 +1331,19 @@ lifecycle convergence: | WP9 transitional legacy root | Exact writer chain still permitted | WP12 final action | |---|---|---| -| management `POST /api/sync` | `src/server/management/config-routes.ts` -> `src/codex/sync.ts` -> `src/codex/refresh.ts` -> catalog writer | rewire to full convergence and `toSyncResponse` | -| server startup cache invalidation | `src/server/index.ts` -> models-cache writer | route startup through full convergence/observer | -| CLI `sync-cache` | `src/cli/index.ts` -> models-cache writer | route the CLI command through full convergence | -| native restore | `src/codex/inject.ts` -> catalog restore writer | move restore behind full convergence/provenance | +| management `POST /api/sync` | `src/server/management/config-routes.ts` -> `src/codex/sync.ts` -> `src/codex/refresh.ts` -> K -> catalog writer | rewire to full convergence and `toSyncResponse` | +| server startup cache invalidation | `src/server/index.ts` -> K -> models-cache writer | route startup through full convergence/observer | +| CLI `sync-cache` | `src/cli/index.ts` -> K -> models-cache writer | route the CLI command through full convergence | +| native restore | `src/codex/inject.ts` -> K -> catalog restore writer | move restore behind full convergence/provenance | This is an exact transitional allowlist by root module and writer symbol, not a directory wildcard. `src/codex/sync.ts`, `src/codex/refresh.ts`, CLI, and `src/codex/inject.ts` are therefore permitted only through the rows above at the WP9 commit; no fifth legacy root may appear. WP12 removes every row and activates -the final table. A contract test cannot enforce both versions at once, so the graph +the final table. Their signatures, return values, gather order, and compatibility +behavior do not change in WP9; only their real synchronous replacement sections +acquire K, after N if a later phase has already placed that root under N. A contract +test cannot enforce both versions at once, so the graph fixture carries an explicit `"wp9-transitional" | "wp12-final"` inventory version: WP9 expects exactly four legacy chains, and WP12 changes that expectation to zero. @@ -1217,6 +1366,9 @@ version, `history-job.ts`, management routes, CLI modules, `sync.ts`, `refresh.t `inject.ts`, and `journal.ts` are not permitted roots; they call convergence, dispatch a Worker, or read only. That final prohibition must not be applied to the four explicit WP9 transitional rows before WP12 owns their migration. +At both inventory versions, every catalog/backup/cache mutator must be reachable only +with an opaque live K permit, and inverse-order graph fixtures reject `C -> K` and +`K -> N` even through wrappers, aliases, or re-exports. ## 9. Baseline classes @@ -1268,18 +1420,48 @@ is released; conflict never invokes the callback. Instrument connection creation the guard cannot regress to `readConfigGenerationAtPath` and self-contend through a second SQLite handle. +Add the distinct round-3 two-process catalog barrier. Process A gathers catalog X, +acquires K then C, completes generation/home/source/epoch/target validation, and +pauses immediately before its first write. Process B invokes the **real retained** +management `POST /api/sync` chain through `refreshCodexModelCatalog`, not a writer +stub or direct permit helper, and prepares Y. B must not replace catalog or cache +while A is paused. B may follow its retained no-write/failure path on fail-fast lock +unavailability or retry after A releases; if it later succeeds, final bytes are Y. +The forbidden trace is Y then X with A reporting `committed`. Reverse acquisition +order as well: if B wins K first, A must revalidate after it acquires K and return +stale rather than replace Y. Run the same exclusion shape for startup cache +invalidation, CLI `sync-cache`, and native restore, and require the inventory graph +to reject any catalog/backup/cache write reachable without K's permit. + Table-drive every `CatalogSourceRole`. Gather from a present source, truncate-and-rewrite that same inode, and require `stale` before any write. Gather with `$CODEX_HOME/config.toml` absent, then create it with `model_catalog_json` selecting another target; require `stale` with the old target byte-identical. Repeat PRESENT -> ABSENT and present-byte/path changes. A compile -fixture omitting `required["catalog-target-selection"]` or any conditional role key -must fail, while the complete shape compiles. The symbol graph must fail when any -gather reader performs or reaches a raw filesystem consultation outside -`catalog/filesystem-evidence.ts`, including an absence-only `existsSync` branch and -a direct target-identity `lstat`/`realpath` probe. +fixture omitting `homeSelection`, `required["catalog-target-selection"]`, or any +conditional role key must fail, while the complete shape compiles. The symbol graph +must fail when any gather reader performs or reaches a raw filesystem consultation +outside `catalog/filesystem-evidence.ts`, including an absence-only `existsSync` +branch and a direct target-identity `lstat`/`realpath` probe. Unreadable or ambiguous re-observations refuse. +Create real temporary homes A and B and a raw `CODEX_HOME=current` symlink selecting +A. Gather against `A/a.json`, retarget `current` once to B without changing any A +file, and require the under-K-and-C home re-resolution to return stale with zero +writes to either home. Assert raw selector, canonical home, root identity, and every +derived config/default-catalog/cache/relative-configured target are compared. The +fixture must go red if admission retains only A's resolved `config.toml` evidence. + +Warm runtime R1 and bundled template B1, then gather a candidate that consumes both. +While provider gathering is paused, replace/invalidate each process memo and require +its monotonic epoch/value check to reject before write, including invalidate then +repopulate with byte-identical data. Separately gather from warm R1 while +`codex-runtime.json` is observed ABSENT; create a persisted R2 from another process +without advancing config generation and require stale. Repeat PRESENT replacement +and removal. A candidate influenced by runtime identity but missing the PRESENT-or- +ABSENT `runtime-selection` observation is structurally refused. A PRESENT R2 that +disagrees with warm R1 may not be used to prepare a candidate in the first place. + Race two create-once backup publishers after both observed ABSENT. Exactly one no-clobber publication wins; the loser receives `EEXIST`, validates and preserves the winner, and neither ordinary rename nor `atomicWriteFile` is called. Repeat with @@ -1289,8 +1471,9 @@ refusal without changing winner bytes. The graph inventory fixture runs as a fifth WP9 root or a retained WP12 root fails. `tests/codex-user-identity.test.ts`: real child processes vary every environment -home/runtime variable named in §7 and resolve one final database path for one -effective uid or SID. POSIX activates wrong owner/mode/symlink and non-sticky `/tmp` refusal through a +home/runtime variable named in §7 and resolve the same two final database paths for +one effective uid or SID, with coordinator and catalog paths distinct. POSIX +activates wrong owner/mode/symlink and non-sticky `/tmp` refusal through a resolver seam; Windows CI activates token/SID failure, known-folder failure, reparse, owner, and broad-ACL refusal. No case falls back to an environment directory. @@ -1310,7 +1493,8 @@ writes to it. - C14 — all 16 management callers funnel through `convergeCodex`, enforced by the symbol graph; its WP9 inventory permits exactly the four transitional chains and - its WP12-final inventory permits none. + its WP12-final inventory permits none. At both versions every first-party + catalog/backup/cache write requires the same permanent K permit. - C16 — one owner, one schema; a record from any phase reads in every other. - C17 — cooperating transition ABA is detected by the durable config/native generations and exact txId, and a parent target that drifts once between gather @@ -1318,11 +1502,17 @@ writes to it. gathered catalog source whose state, identity, or bytes drift once is detected by its role-bearing observation even when the write target and both generations are unchanged. This includes required `config.toml` ABSENT -> PRESENT target-selection - drift. Cooperating config N -> N+1 is prevented while the catalog callback holds - the existing config transaction, and create-once backups use atomic no-clobber - publication. An arbitrary filesystem or content A→B→A that completes wholly - between two checks, and a non-cooperating write after the final comparison, are - explicitly not claimed. + drift and a single-direction raw CODEX_HOME-selector/canonical-root retarget before + writing. A runtime-influenced candidate always carries PRESENT-or-ABSENT + `codex-runtime.json` evidence, and any used runtime/bundled process memo must retain + its exact monotonic epoch and immutable value identity through the commit check. + Cooperating config N -> N+1 is prevented while the catalog callback holds C, and + the two-process real `/api/sync` barrier proves every retained first-party catalog + writer is serialized by K so convergence cannot overwrite a later first-party + publication with stale gathered bytes. Create-once backups use atomic no-clobber + publication. An arbitrary filesystem, selector, or content A→B→A that completes + wholly between two checks, and a non-cooperating write after the final comparison, + are explicitly not claimed. - Contributes to C15 with detect-and-repair: the latest native pair is durably pending before spawn, a stale Worker cannot replace its transition row or the winner's schedule, and the guardian diff --git a/devlog/_plan/260804_codex_write_substrate/010_catalog_seam.md b/devlog/_plan/260804_codex_write_substrate/010_catalog_seam.md index 4ef96e187..354582964 100644 --- a/devlog/_plan/260804_codex_write_substrate/010_catalog_seam.md +++ b/devlog/_plan/260804_codex_write_substrate/010_catalog_seam.md @@ -14,20 +14,21 @@ sequence inside it, and it cannot tell the caller what actually happened. WP9 lands the first real catalog-scoped `ConvergeCodex`, rewires exactly those 16 management mutation sites, and leaves the explicit sync/startup/CLI/restore -roots for WP12. A catalog-only commit updates catalog, create-once catalog -backups, and models cache only. It neither reads nor advances the native routing -pair and never writes `config.toml`, generated profile, injection journal, or +call shapes for WP12 while modifying each retained writer chain to acquire the +permanent catalog serialization primitive K. A catalog-only commit updates catalog, +create-once catalog backups, and models cache only. It neither reads nor advances the +native routing pair and never writes `config.toml`, generated profile, injection journal, or history. The transition row makes that boundary mandatory: every positive native generation requires matching history schedule fields, every `beginTransition` publishes that schedule, and `assertPublished` rejects a transition that was not published (`src/codex/transition-state.ts:74-83,314-344,420-428`). All current-code citations and diff context below were rechecked on 2026-08-04 -at `db3d69ed05bba5c2dd822f2c2088c186adf5a105`. The contract citations refer to -the authoritative concurrent WP8b/WP9 amendment: the owner-held config-generation -guard, closed source-observation union, and atomic no-clobber publication are at -`005_contract.md:615-645,722-784,792-918,923-938`; catalog-only work remains -excluded from the native pair at `005_contract.md:588-613`. +at `f51404f1670e57bdfb1637adf2d5d0cb5aeaf1f5`. The contract citations refer to +the authoritative worktree amendment: permanent K, owner-held config generation, +home/source/process-local evidence, and no-clobber publication are at +`005_contract.md:609-887,895-991,1021-1072`; K's owner/path and the four +transitional chains are fixed at `005_contract.md:1234-1289,1291-1371`. ## IN / OUT @@ -42,19 +43,22 @@ IN — observe-only admission and gather: wrap this observer in the lock or redefine the generation contract. - `src/codex/catalog-admission.ts` (MODIFY) — keep the landed request constructor and snapshot capture; switch snapshot capture to the observe-only generation - read, capture the required PRESENT-or-ABSENT `catalog-target-selection` - observation, and carry the contract-owned `sourceEvidence`. Do not redefine + read, capture the required raw/default `CODEX_HOME` selector, canonical home and + root identity plus the PRESENT-or-ABSENT `catalog-target-selection` observation, + and carry the contract-owned `sourceEvidence`. Do not redefine `createCatalogConvergeRequest` or `captureCatalogAdmissionSnapshot`, which - already exist at lines 32-46 and 84-107. + already exist at lines 38-52 and 148-180. - `src/codex/convergence-types.ts` (MODIFY) — synchronize the already contract-owned - closed `CatalogSourceRole`, `CatalogSourceObservation`, `CatalogSourceEvidence`, - and `CatalogAdmissionSnapshot.sourceEvidence` additions from - `005_contract.md:792-865`; no WP9-private duplicate type is allowed. + closed `CatalogSourceRole`, `CatalogHomeSelectionObservation`, + `CatalogSourceObservation`, `CatalogSourceEvidence`, + `CatalogProcessLocalEvidence`, and `CatalogAdmissionSnapshot.sourceEvidence` + additions from `005_contract.md:895-991`; no WP9-private duplicate type is allowed. - `src/codex/catalog/filesystem-evidence.ts` (NEW) — sole owner of gather source reads and target probes. Its opaque session records PRESENT and ABSENT - observations before returning, seals the complete closed role map into the - candidate, and is the only gather path permitted to call filesystem consultation - primitives (`005_contract.md:895-918`). + observations before returning, captures and seals catalog-home selection before + accepting any derived path, seals the complete closed role map into the candidate, + and is the only gather path permitted to call filesystem consultation primitives + (`005_contract.md:1021-1046`). - `src/codex/runtime.ts`, `src/codex/catalog/bundled.ts` (MODIFY) — catalog gather uses a gather-specific observe-only pair: `peekCodexRuntimeForCatalogGather(evidenceSession)` and @@ -65,7 +69,15 @@ IN — observe-only admission and gather: `catalog-unavailable`. `bundled.ts` owns the catalog-specific adapter; `runtime.ts` exposes only a process-cache peek and pure persisted-state parser and never imports the catalog evidence module. A cold miss never becomes permission to - execute Codex. + execute Codex. Each mutable runtime/bundled memo owns a process-lifetime monotonic + epoch and immutable value identity; population, replacement, clear, invalidation, + persisted-runtime write, and test reset advance the applicable epoch. Gather seals + the exact epoch/value identity it consumes, and commit revalidates both before its + first write. Whenever runtime identity influences a candidate, the evidence session + records `codex-runtime.json` as PRESENT or ABSENT even on a warm-cache hit. + This is required because `persistCodexRuntime` writes the file and clears the memo + without advancing config generation (`src/codex/runtime.ts:213-229`), while bundled + cache hits and replacements are process-local (`src/codex/catalog/bundled.ts:179-209`). The ordinary resolver reaches `probeVersion`, whose sandbox deliberately calls `mkdtempSync` and `rmSync` (`src/codex/runtime.ts:231-279,327-340,397-405`), while bundled loading both calls the persisting resolver and runs `codex debug models` @@ -88,14 +100,27 @@ IN — observe-only admission and gather: bytes and source evidence without writing. In particular, target selection no longer hides an `existsSync`/`readFileSync` consultation inside `readCodexCatalogPath()` (`src/codex/catalog/parsing.ts:167-176`); admission makes - that consultation through the evidence owner. + that consultation through the evidence owner. Preserve production path semantics: + relative `model_catalog_json` resolves below the canonical active home, an absolute + configured target remains absolute even outside that home, and an existing catalog + leaf symlink resolves to and writes through its real target + (`src/codex/catalog/parsing.ts:52-80`, `src/config.ts:125-160,188-209`). IN — fixed commit and convergence: +- `src/codex/catalog-write-serialization.ts` (NEW) — permanent synchronous K owner, + keyed by effective user plus canonical `CODEX_HOME`, backed by its own SQLite + database with `busy_timeout=0` and `BEGIN IMMEDIATE`. It returns only from an + owner-held synchronous callback carrying an opaque, non-forgeable permit. It is + separate from both N and `config-mutation.sqlite`, and WP11 never replaces it. +- `src/codex/user-identity.ts` (MODIFY) — add + `resolveCodexCatalogSerializationDatabasePath` beside the landed native coordinator + resolver. Consumers use its final path verbatim; the K and N database paths must be + distinct (`005_contract.md:1234-1289`). - `src/codex/internal/catalog-writer.ts` (NEW/MOVE) — the contract-owned low-level - owner for catalog, hashed/legacy backups, and models cache. Do not create the - obsolete `internal/catalog-commit.ts` name - (`005_contract.md:1172,1178-1179`). + owner for catalog, hashed/legacy backups, and models cache. Every mutator requires + K's opaque live permit. Do not create the obsolete `internal/catalog-commit.ts` + name (`005_contract.md:1314-1325`). - `src/codex/convergence.ts` (NEW) — catalog gather/commit orchestration and the only WP9 module allowed to call symbols in `internal/catalog-writer.ts`. - `src/codex/management-convergence.ts` (MODIFY) — retain the landed @@ -113,10 +138,18 @@ IN — management callers and tests: - `src/server/management-api.ts`, `src/server/management/context.ts`, and the four invoking route modules (MODIFY) — replace the swallowed helper with a total, lazy catalog-convergence adapter returning `CatalogDisposition`. -- `src/codex/sync.ts`, `src/server/management/config-routes.ts`, - `src/server/index.ts`, `src/cli/index.ts`, and `src/codex/inject.ts` (IMPORT-ONLY) - — keep the four legacy roots compiling after the public facade stops re-exporting - writers. Their behavior and ownership do not move until WP12. +- The four WP9-transitional chains are IN, but only for K acquisition at their real + synchronous replacement sections. In `src/codex/catalog/sync.ts`, preserve the + public signatures of `syncCatalogModels`, `invalidateCodexModelsCache`, and + `restoreCodexCatalog`, but prepare/read outside K and acquire K only around the + low-level catalog/backup/cache replacement that consumes its permit. + `src/codex/refresh.ts` keeps `/api/sync`'s async gather order and invokes those + retained K-protected publications. Startup in `src/server/index.ts`, CLI + `sync-cache` in `src/cli/index.ts`, and native restore in `src/codex/inject.ts` + keep their current caller behavior; any explicit import changes needed after the + facade stops re-exporting writers are mechanical. No retained chain holds K while + reading or gathering. This is serialization of the retained roots, not WP12's + convergence rewire (`005_contract.md:1327-1348`). - `tests/codex-refresh.test.ts` and the existing management route suites (MODIFY). - `tests/codex-convergence-contract.test.ts` (CREATE). It does not exist in the WP8b tree; WP9 creates it rather than “extending” an imaginary file. @@ -124,12 +157,14 @@ IN — management callers and tests: OUT: - WP10 history scheduling/worker behavior. Catalog-only work schedules no history. -- WP11 native lock acquisition. WP9 makes commit synchronous and uses only the - existing config mutation transaction to exclude cooperating config writers through - publication; it does not import a future native lock or claim exclusion against - native/catalog hand edits and foreign writers. +- WP11 native lock acquisition. WP9 creates and permanently owns K; K is not the + native lock and is not a placeholder. Global order is N -> K -> C. Because WP9 + catalog-only work does not acquire N, its concrete order is K -> C. `C -> K`, + `K -> N`, and a held `N -> H` are forbidden. - WP12 full admission/observer/provenance, full `scope:"full"` convergence, - `/api/sync`, startup, CLI cache sync, restore, and complete writer reachability. + and the call-shape rewires for `/api/sync`, startup, CLI cache sync, and restore. + Those four roots already hold K in WP9; WP12 removes their transitional reachability + without replacing K. - Any runtime command that starts, stops, syncs, restores, ensures, or manages the live service; any write to real `~/.codex` or `~/.opencodex`; GUI/release/deploy. @@ -142,23 +177,29 @@ unfinished WP9 branch. Phase-entry gate: the audited source currently exports `readConfigGeneration` and `bumpConfigGeneration` only (`src/config.ts:1845-1859`); the amended contract assigns the executable `withExpectedConfigGenerationSync` owner seam to WP8b -(`005_contract.md:615-645`). WP9 implementation starts after that prior phase lands. +(`005_contract.md:617-647`). WP9 implementation starts after that prior phase lands. If the seam is still absent, stop and report the WP8b scope dependency; do not emulate it with a second connection, weaken the guard to observe-before-write, or leave a -placeholder for WP12. +placeholder for WP12. K does not extend that phase-entry dependency: the current tree +has only the native final-path resolver at `src/codex/user-identity.ts:164-186` and no +`catalog-write-serialization.ts`; WP9 creates both K's module and catalog resolver +because WP9 is the first phase that must serialize catalog/backup/cache publication. +Moving K to WP8b would widen the already-landed admission seam without an earlier +consumer, while deferring it to WP11/WP12 would leave WP9's retained writers unsafe. ## A. Filesystem-write-free gather ### A1 — state the guarantee exactly -The guarantee is **filesystem-write-free**, not globally side-effect-free. Gather -may update bounded process-local memo, discovery-status, provider model cache, and -in-flight admission maps. Those mutations already occur at -`src/codex/runtime.ts:362-405` and -`src/codex/catalog/provider-fetch.ts:455-465,495-507,608-615,675-685`; they are -permitted because they do not mutate user files and are reset between isolated -tests. No credential, raw provider error, source path, or digest may escape through -those caches into `CatalogDisposition`. +The guarantee is **filesystem-write-free**, not globally side-effect-free. Gather may +update bounded discovery-status, provider model cache, and in-flight admission maps +(`src/codex/catalog/provider-fetch.ts:455-465,495-507,608-615,675-685`); they are +permitted because they do not mutate user files and are reset between isolated tests. +The runtime and bundled memos are observe-only from gather, but their owners can +replace them concurrently (`src/codex/runtime.ts:362-410`, +`src/codex/catalog/bundled.ts:179-209`), which is why the candidate seals epochs and +identities. No credential, raw provider error, source path, or digest may escape +through those caches into `CatalogDisposition`. Filesystem-write-free means the entire interval from **before admission capture** through resolved runtime observation, token observation, provider calls, fallback @@ -198,13 +239,18 @@ The gather-specific resolver is a separate API, not a flag on `peekCodexRuntimeForCatalogGather(evidenceSession)`: it may consume an unexpired successful value from a pure process-cache peek exported by `runtime.ts`, or parse persisted `codex-runtime.json` bytes supplied by the evidence session through a pure -runtime-state parser. `runtime.ts` never imports the catalog evidence owner. This -path does not test whether the command is executable, discover PATH alternatives, -call `probeVersion`, persist selection, or execute the command. The observation can +runtime-state parser. The persisted file is a mandatory PRESENT-or-ABSENT +`runtime-selection` observation whenever that runtime identity affects the candidate, +including a warm process-cache hit. `runtime.ts` never imports the catalog evidence +owner. This path does not test whether the command is executable, discover PATH +alternatives, call `probeVersion`, persist selection, or execute the command. The observation can only identify a matching already-populated in-memory bundled-catalog cache; it is not -authority to refill it. `resolveCatalogSourceForGather(evidenceSession)` then tries -that immutable cache value followed by active-catalog/backup/models-cache buffers -read through the evidence owner. Its closed result is usable prepared source or +authority to refill it. The runtime and bundled owners return the immutable value with +its current process-lifetime epoch/value identity; the candidate records `unused` when +an owner does not influence preparation. +`resolveCatalogSourceForGather(evidenceSession)` then tries that immutable cache value +followed by active-catalog/backup/models-cache buffers read through the evidence owner. +Its closed result is usable prepared source or `catalog-unavailable`; the latter projects to the existing sanitized `skipped/catalog-unavailable` disposition and leaves no residue. @@ -219,14 +265,16 @@ evidence; token bytes never do. ### A2 — seal the closed role-bearing source observations -`captureCatalogAdmissionSnapshot(config)` remains the pre-gather constructor. Its -source evidence starts with every conditional role key present as an empty list and -the required `catalog-target-selection` observation for the logical +`captureCatalogAdmissionSnapshot(config)` remains the pre-gather constructor. Before +accepting any derived path it records the raw environment/default selector, canonical +`CODEX_HOME`, and root identity as required `homeSelection`. Its source evidence starts +with every conditional role key present as an empty list and the required +`catalog-target-selection` observation for the logical `$CODEX_HOME/config.toml` path, recorded PRESENT or ABSENT. The opaque filesystem-evidence session then records every consulted filesystem source under the contract's closed role union: bundled template, active merge, hashed/legacy fallback, models-cache fallback, runtime selection, or provider-auth selection -(`005_contract.md:792-852,895-918`). Callers cannot append, omit, remove, or rebuild +(`005_contract.md:788-844,895-978`). Callers cannot append, omit, remove, or rebuild those observations. The required ABSENT state closes a target-selection hole that a present-file digest @@ -243,19 +291,24 @@ buffer** it returns. For an ABSENT source, it records the logical path, canonica missing-leaf path, stable canonical-parent identity, and `fileIdentity:null`. Alternatives consulted and found absent are still evidence because their absence caused fallback. Process-local caches and network responses are not fabricated as -filesystem observations. The candidate receives a sealed immutable -`CatalogSourceEvidence`; a missing required role or conditional key is structurally -invalid and cannot reach commit. - -Immediately before the first replacement, the under-lock commit callback -re-observes every candidate-bound source and compares state, logical/canonical path, -parent identity, file identity, and PRESENT digest. State, identity, path, or digest -drift returns `stale`; unreadable, unresolvable, non-regular, or ambiguous evidence -returns `refused`. Both paths write zero bytes. This detects the audited same-inode +filesystem observations; used runtime/bundled values instead contribute sealed +`CatalogProcessLocalEvidence`. The candidate receives immutable source and process +evidence. Missing `homeSelection`, a required role or conditional key, a required +runtime-selection PRESENT/ABSENT observation, or a used epoch/value identity is +structurally invalid and cannot reach commit. + +Immediately before the first replacement, the under-K-and-C commit callback re-reads +the raw/default home selector, re-runs the production home resolver, compares selector, +canonical root, root identity, and every re-derived config/catalog/cache/backup target, +then re-observes every candidate-bound source and revalidates each used runtime/bundled +epoch/value identity. It compares source state, logical/canonical path, parent identity, +file identity, and PRESENT digest. Home/target/source/process-local drift returns +`stale`; unreadable, unresolvable, non-regular, or ambiguous evidence returns `refused`. +Both paths write zero bytes. This detects the audited same-inode truncate/rewrite even when config generation and target identity are unchanged. It catches single-direction drift only: content/state A→B→A returning identical evidence before comparison, parent A→B→A between checks, and a write after the final -comparison remain outside C17 (`005_contract.md:731-762`). +comparison remain outside C17 (`005_contract.md:813-864`). ### A3 — preserve bundled-first template precedence @@ -278,7 +331,8 @@ outside the 16 management paths until their owning phase migrates them. The candidate remains opaque, one-shot, and catalog-private. Its `WeakMap` state contains prepared bytes, result/notices, target identities, the admitted config -generation, and the sealed candidate-bound `CatalogSourceEvidence`. Commit marks it +generation, home selection, sealed candidate-bound `CatalogSourceEvidence`, and +sealed `CatalogProcessLocalEvidence`. Commit marks it consumed before validation and before the first write; a second call returns `candidate-consumed` and writes nothing. No route can inspect, serialize, reconstruct, or replay it. @@ -295,7 +349,7 @@ export interface CatalogWriteReceipt { export type CodexCatalogCommitResult = | { readonly kind: "committed"; readonly changed: boolean; readonly writes: CatalogWriteReceipt } - | { readonly kind: "stale"; readonly reason: "generation" | "source-observation" | "target-identity" | "candidate-consumed" } + | { readonly kind: "stale"; readonly reason: "generation" | "home-selection" | "source-observation" | "process-local" | "target-identity" | "candidate-consumed" } | { readonly kind: "refused"; readonly reason: "source-unreadable" | "source-ambiguous" | "target-unsafe" } | { readonly kind: "failed"; readonly surface: "disk"; readonly writes: CatalogWriteReceipt }; ``` @@ -310,25 +364,31 @@ Preparation returns exact catalog/cache bytes and optional create-once backup by filesystem dependencies. It accepts no config, provider client, parser, subprocess, OAuth resolver, Promise, or callback that can return a Promise. -Commit performs, in order: - -1. mark the candidate consumed, then call - `withExpectedConfigGenerationSync(candidate.generation, commitCallback)`; -2. inside the already-held config transaction, validate every target identity and - re-observe every sealed PRESENT/ABSENT source observation immediately before the - first write; -3. publish the keyed backup with atomic no-clobber semantics; +The outer catalog orchestration acquires K after gather and before any config +transaction. Automatic catalog convergence retries fail-fast K acquisition only within +`deadlineMs`; each attempt and the complete owner-held callback remain synchronous. +The callback receives K's opaque permit, and low-level writer calls without that exact +permit do not typecheck. Retained roots apply the same gather-outside-K rule without +changing their public signatures. Commit performs, in order: + +1. acquire K after N when N exists; WP9 catalog-only has no N and therefore starts at + K. Once K is held, mark the candidate consumed and call + `withExpectedConfigGenerationSync(candidate.generation, commitCallback)` to enter C; +2. inside K -> C, re-resolve home selection and every derived target, validate target + identity, re-observe every sealed PRESENT/ABSENT source, and revalidate each used + process-local epoch/value identity immediately before the first write; +3. publish the keyed backup with atomic no-clobber semantics using K's permit; 4. publish the legacy backup with atomic no-clobber semantics when the default path requests it; 5. replace the active catalog; 6. replace the models cache; then return from the synchronous callback so the owner - can release the config transaction. + can release C and then K. The owner-side guard is not a read-before-write check. Its implementation validates the expected generation using the `configMutationDatabase` handle whose SQLite transaction is already held, invokes the complete synchronous catalog callback on a match, and releases only after the callback returns -(`005_contract.md:629-645,692-703`). A cooperating config writer therefore cannot +(`005_contract.md:631-647,676-707`). A cooperating config writer therefore cannot commit N+1 between validation and catalog publication. Conflict never invokes the callback; lock/database unavailability projects through the total adapter. @@ -337,8 +397,10 @@ observe-only reader inside `withConfigMutationLockSync`. Those observers open a second SQLite connection; while the first connection owns `BEGIN IMMEDIATE`, the second connection contends with its own caller instead of validating it. The guard must use `readConfigGenerationInTransaction` or its private equivalent on the -already-held database. WP9 consumes the existing config mutation lock only; it does -not import WP11's native lock, and catalog-only work does not bump config generation. +already-held database. WP9 composes permanent K with the existing config mutation +lock; it does not import WP11's native lock, and catalog-only work does not bump config +generation. No callback holding C may acquire K, no K callback may acquire N, and no +catalog path acquires H. Receipt fields change only after the corresponding replacement succeeds. A failure returns the exact prefix receipt and consumes the candidate; callers must regather. @@ -359,7 +421,7 @@ regular, non-routed valid catalog backup is preserved and the receipt becomes is `refused`. The loser never unlinks, truncates, or overwrites the winner. This exception applies only to a backup create-once target, never to a backup selected as a gather source; selected source observations remain strict -(`005_contract.md:764-784`). +(`005_contract.md:867-887`). ## C. Catalog-only convergence @@ -369,9 +431,9 @@ WP9 does not redeclare request, snapshot, projection, or shared result types. `management-convergence.ts` consumes: - `createCatalogConvergeRequest` from - `src/codex/catalog-admission.ts:32-46`; + `src/codex/catalog-admission.ts:38-52`; - `captureCatalogAdmissionSnapshot` from - `src/codex/catalog-admission.ts:84-107`; + `src/codex/catalog-admission.ts:148-180`; - `projectCatalogOnlyOutcome` from its landed owner at `src/codex/management-convergence.ts:63-75`; - shared `CatalogDisposition`, `ConvergeOutcome`, and `ConvergeCodex` from @@ -379,13 +441,14 @@ WP9 does not redeclare request, snapshot, projection, or shared result types. The placeholder factory body at `src/codex/management-convergence.ts:81-96` is replaced in place. It validates catalog scope without throwing, captures admission, -awaits the write-free gather, enters `withExpectedConfigGenerationSync`, executes the -synchronous catalog callback before that owner releases, and projects the result. The +awaits the write-free gather, acquires K, enters +`withExpectedConfigGenerationSync`, executes the synchronous catalog callback before +C and K release, and projects the result. The lower-level orchestration lives in new `convergence.ts`, so only that module reaches `internal/catalog-writer.ts`; the retained management module remains the factory boundary until WP12 consolidates the full funnel. -### C2 — owner-held config generation, source observations, and target identity only +### C2 — permanent K plus owner-held generation and complete candidate evidence A `scope:"catalog"` commit does not request `CommitExpectation`, open `transition-state.ts`, call `beginCodexTransition`, call `assertPublished`, or read @@ -394,11 +457,17 @@ has no pair fields (`src/codex/convergence-types.ts:207-224`). Catalog staleness is guarded by: +- permanent effective-user/canonical-`CODEX_HOME` serialization K, held across every + catalog/backup/cache replacement by convergence and all four transitional roots; - the observe-only config generation captured before gather and validated by `withExpectedConfigGenerationSync` on its already-held transaction through the complete synchronous commit; +- required raw/default catalog-home selection, canonical root identity, and equality + of every target re-derived under K -> C; - candidate-bound closed PRESENT/ABSENT source observations, including required - `config.toml` target selection; + `config.toml` target selection and mandatory runtime-selection evidence whenever + runtime identity influenced the candidate; +- used runtime/bundled process-cache epochs and immutable value identities; - target parent/file identity plus the narrow create-once backup exception. The commit must never import or invoke routing writers. A test fails if catalog-only @@ -421,10 +490,10 @@ a conservative typed projection: | Internal condition | `CatalogDisposition` projection | |---|---| -| gather admission busy | `skipped/busy`, retryable | +| gather admission or K acquisition busy/deadline | `skipped/busy`, retryable | | no usable catalog source | `skipped/catalog-unavailable` | -| config/target/source refusal | `skipped/refused` | -| generation, source-observation, or identity drift | `skipped/stale`, retryable | +| config/home/target/source/process-evidence refusal | `skipped/refused` | +| generation, home, source, process-local, or identity drift | `skipped/stale`, retryable | | provider auth/network gather failure | matching `failed` reason, `phase:"gather"`, `partialWrite:false` | | lazy import, missing export, factory, or unexpected pre-commit failure | sanitized `failed/disk`, `phase:"gather"`, `partialWrite:false` | | expected replacement failure | `failed/disk`, `phase:"commit"`, `partialWrite` derived from the receipt | @@ -470,10 +539,10 @@ The symbol-graph test permits these exact legacy roots until WP12: | Legacy root | Current path | WP12 removal | |---|---|---| -| management `POST /api/sync` | `src/server/management/config-routes.ts:261-268` → `src/codex/sync.ts:83-89` → `src/codex/refresh.ts:44-51` | rewire to full convergence and `toSyncResponse` | -| server startup cache invalidation | `src/server/index.ts:403` → `invalidateCodexModelsCache` | route startup through full convergence/observer | -| `ocx sync-cache` | `src/cli/index.ts:849-855` → `invalidateCodexModelsCache` | route CLI command through full convergence | -| native restore | `src/codex/inject.ts:764-774` → `restoreCodexCatalog` → `src/codex/catalog/sync.ts:572-597` | move restore writes behind full convergence/provenance | +| management `POST /api/sync` | `src/server/management/config-routes.ts:261-268` → `src/codex/sync.ts:83-90` → `src/codex/refresh.ts:40-52`; after gather, `src/codex/catalog/sync.ts:568` publishes under K and `src/codex/catalog/sync.ts:600-616` is called under K for cache publication | rewire to full convergence and `toSyncResponse`; K remains | +| server startup cache invalidation | `src/server/index.ts:403` → K → `invalidateCodexModelsCache` | route startup through full convergence/observer; K remains | +| `ocx sync-cache` | `src/cli/index.ts:849-855` → K → `invalidateCodexModelsCache` | route CLI command through full convergence; K remains | +| native restore | `src/codex/inject.ts:764-774` → K → `restoreCodexCatalog` → `src/codex/catalog/sync.ts:572-597` | move restore writes behind full convergence/provenance; K remains | The allowlist is exact by root module and writer symbol, not a directory wildcard. WP12 owns deleting every row. No new legacy root may be added in WP9. @@ -485,6 +554,9 @@ the writer symbol identity. An unresolved module, unresolved symbol, computed dynamic import, or non-literal import that could hide a writer fails the test rather than being skipped. The test publishes the WP9 legacy allowlist as data and proves all 16 management roots terminate at `convergence.ts` before a catalog writer. +It also proves every catalog/backup/cache mutator requires K's opaque permit and +rejects `C -> K`, `K -> N`, or held `N -> H` acquisition edges through aliases, +wrappers, or re-exports. ## Tests @@ -514,7 +586,7 @@ resolution with `resolveAndPersistCodexRuntime`, use `loadAuthStore`, or use subprocess, created-then-deleted probe home, mkdir, chmod, backup, SQLite, ownership, or other transient write. -### T2 — closed observations, owner-held generation, and identity reject before write +### T2 — K, home/cache/source evidence, generation, and identity reject before write Table-drive every `CatalogSourceRole`: required config target selection, filesystem-backed bundled-template source, active catalog, selected hashed backup, @@ -528,7 +600,45 @@ catalog; expect `stale` and byte-identical old/new targets. Make each re-observa unreadable/ambiguous and expect `refused` with zero writes. Retarget one parent and expect target-identity rejection. Compile fixtures that omit `required["catalog-target-selection"]` or any conditional role key; each must fail, -while the complete shape compiles. +while the complete shape compiles. Additional compile/private-validation fixtures omit +`homeSelection`, a runtime-influenced candidate's PRESENT-or-ABSENT +`runtime-selection`, and a used process memo's epoch/value identity; each incomplete +shape must fail before commit, while the complete shape compiles. + +Create real temporary homes A and B and set the raw selector to a `current` symlink +initially targeting A. Gather from `A/a.json`, retarget `current` once to B while +leaving every A file unchanged, then commit. Under K -> C the implementation must +re-read the raw selector, re-run `activeCodexHome`, compare canonical home/root +identity, re-derive config/default-catalog/cache/backup/configured targets, and return +`stale` with zero writes in either home. The named broken mutation is **retain only +A's resolved config/source evidence and skip home re-resolution**; it incorrectly +commits into A while Codex reads B. + +Freeze the current target-selection semantics with three real-file fixtures +(`src/codex/catalog/parsing.ts:52-80`, `src/config.ts:125-160,188-209`): + +- relative `model_catalog_json = "nested/a.json"` resolves beneath the canonical + `CODEX_HOME`, and gather/commit compare and write that derived target. The named + broken mutation **resolve the relative value from cwd** writes the wrong file. +- an absolute configured target outside `CODEX_HOME` stays that exact absolute target; + the named broken mutation **force every configured target under CODEX_HOME** either + refuses valid current behavior or writes the wrong file. +- an existing catalog leaf symlink is resolved to its real target and survives an + atomic write through that target. The named broken mutation **rename over the + logical symlink leaf** replaces the link or writes the wrong entry. This accepted + active-catalog behavior does not weaken T3's refusal of a symlinked create-once + backup winner. + +Warm runtime R1 and bundled template B1, gather a candidate that consumes both, then +pause provider gathering and replace/invalidate each memo. Commit must revalidate the +monotonic epoch and immutable value identity under K -> C and return `stale` with zero +writes, including invalidate-and-repopulate with byte-identical data. Separately +gather from warm R1 while `codex-runtime.json` is observed ABSENT, create persisted R2 +without advancing config generation, and require `stale` with zero writes; repeat +PRESENT replacement and removal. The named broken mutations are **compare cache bytes +without the epoch** and **record runtime-selection only on cold loads**; the first +accepts byte-identical replacement and the second accepts ABSENT -> PRESENT on a warm +hit. Prove the cooperating-writer guarantee with two real processes and the real config mutation API. Process A enters @@ -542,15 +652,31 @@ Conflict never invokes the callback. Instrument SQLite connection creation and require the guard to validate through the already-held handle, with no second connection. +Add a distinct two-process K barrier using the **real retained management +`POST /api/sync` chain**, not a writer stub or direct permit helper. Process A gathers +catalog X, acquires K then C, completes generation/home/source/process-local/target +validation, and pauses immediately before its first replacement. Process B sends a +real `Request` through `handleManagementAPI` for `POST /api/sync`, reaches +`refreshCodexModelCatalog`, and prepares Y. While A is +paused B must not replace catalog or cache. After A releases, B may follow its retained +no-write/failure path or retry and publish Y; the forbidden trace is Y then X with A +reporting `committed`. Reverse acquisition order: let B publish Y while holding K, +then require A to revalidate after acquiring K and return `stale` with zero writes. +Run the same K-exclusion shape for startup cache invalidation, CLI `sync-cache`, and +native restore. The named broken mutation is **omit K, release K before replacement, +or let one transitional writer call a mutator without the permit**; it permits Y then +X or a concurrent cache/restore overwrite. + Document but do not claim detection for content A→B→A returning exact A before the comparison, parent A→B→A entirely between checks, or a write after the comparison. -Broken mutations that must turn T2 red: omit the required ABSENT config observation, -remove digest comparison while retaining generation/file identity, release the config -transaction before callback, or call `readConfigGenerationAtPath` from inside the -guard. The absent->present target switch commits obsolete bytes, the same-inode -rewrite commits stale bytes, process B commits N+1 while A is paused, or the guard -self-contends/opens the forbidden second handle. +Broken mutations that must turn T2 red, in addition to the named mutations above: +omit the required ABSENT config observation, remove digest comparison while retaining +generation/file identity, release C before callback, acquire C before K, or call +`readConfigGenerationAtPath` from inside the guard. The absent->present target switch +commits obsolete bytes, the same-inode rewrite commits stale bytes, process B commits +N+1 while A is paused, the real `/api/sync` writer publishes outside K, inverse order +deadlocks/self-contends, or the guard opens the forbidden second handle. ### T3 — exact four-step receipt and bytes @@ -619,18 +745,26 @@ any new root. The same symbol-resolved graph inventories gather filesystem consultations: every `readFileSync`, `Bun.file`, `existsSync` branch, target `lstat`/`stat`/`realpath`, or wrapper that reaches one must terminate at `catalog/filesystem-evidence.ts`. Unresolved/computed edges fail closed. +At both the `wp9-transitional` and future `wp12-final` inventory versions, every +catalog/backup/cache mutation must require K's permit. Lock-order fixtures fail on +`C -> K`, `K -> N`, and a held `N -> H`; the accepted full order is N -> K -> C, +while WP9 catalog-only uses K -> C because it never acquires N. Broken mutations that must turn T5 red: add a static top-level management-convergence import, alias a catalog writer into a management route, replace a literal import with a computed dynamic import, or add an absence-only `existsSync`/target `realpath` -outside the evidence owner. The sentinel or fail-closed graph must reject each. +outside the evidence owner. Also remove the permit parameter from one mutator, add a +fifth unpermitted root, or invert any lock edge. The sentinel, compile fixture, or +fail-closed graph must reject each. ### T6 — precedence and native-pair exclusion On the default catalog path, pre-populate the process-local bundled cache with a template that differs visibly from catalog/backup/cache plus an on-disk routed/user-native row. Gather must use cached bundled native template fields and -preserve the on-disk merge row without probing or launching Codex. Repeat cold: no +preserve the on-disk merge row without probing or launching Codex, while recording +the bundled epoch/value identity and the runtime-selection file observation required +by the runtime that selected it. Repeat cold: no bundled cache plus a valid observed disk fallback succeeds without subprocess; no bundled cache and no valid disk fallback returns `catalog-unavailable`. Assert no materialized fallback write. Snapshot the transition row before and after committed, @@ -639,8 +773,9 @@ no history schedule appears. Routing artifact and executable-probe spies stay ze Broken mutations that must turn T6 red: move disk fallbacks ahead of a populated bundled cache, call `loadBundledCodexCatalog` to refill a cold cache, request -`CommitExpectation`, or call a routing writer. Template/cold-miss assertions, -executable spies, transition-row equality, or routing spies fail. +`CommitExpectation`, omit warm-cache evidence, or call a routing writer. +Template/cold-miss assertions, executable spies, transition-row equality, or routing +spies fail. ## Verification @@ -665,9 +800,9 @@ processes only. No verification invokes `ocx start`, `stop`, `sync`, `restore`, | Criterion | Proof | Concrete broken mutation that makes it red | |---|---|---| -| **C1** — gather is filesystem-write-free across user homes and scratch, performs no executable probe/subprocess, and commit is synchronous, fixed, one-shot, and receipt-exact | T1 + T3 | call cold `resolveCodexRuntime`/`loadBundledCodexCatalog`, add an `await` beneath commit, reorder replacements, pre-set a receipt bit, or replay a consumed candidate | -| **C2/C17** — the owner-held config generation, every closed PRESENT/ABSENT source observation, and target identity reject stale work before write; create-once backups publish atomically without clobber | T2 + T3 | omit ABSENT `config.toml`, release the transaction before callback, remove same-inode digest comparison, open a second SQLite observer, or replace exclusive publication with overwriting rename | +| **C1** — gather is filesystem-write-free across user homes and scratch, performs no executable probe/subprocess, and commit is synchronous, fixed, K -> C ordered, one-shot, and receipt-exact | T1 + T3 + T5 | call cold `resolveCodexRuntime`/`loadBundledCodexCatalog`, add an `await` beneath commit, acquire C before K, reorder replacements, pre-set a receipt bit, or replay a consumed candidate | +| **C2/C17** — permanent K excludes every first-party catalog writer; owner-held config generation, required home/runtime evidence, process-local epochs, every closed PRESENT/ABSENT source observation, and target identity reject stale work before write; create-once backups publish atomically without clobber | T2 + T3 | omit K from real `/api/sync`, omit ABSENT `config.toml`/`codex-runtime.json`, skip CODEX_HOME re-resolution, compare cache bytes without epoch, release C before callback, remove same-inode digest comparison, open a second SQLite observer, or replace exclusive publication with overwriting rename | | **Catalog/native boundary** — catalog-only never reads/advances the native pair or writes routing/history artifacts | T6 | call `expectation()`/`beginTransition`, add pair fields to `catalog-only`, or invoke config/profile/journal/history writer | | **Best-effort compatibility** — all 16 primary writes retain 2xx/201 and original follow-up order for every catalog failure | T4 | let lazy import/factory/admission throw, scope “zero writes” to the whole route, or return before Claude/Desktop follow-up | -| **C14, WP9-bounded** — the 16 management roots reach catalog writers only through convergence; exactly four documented legacy roots remain until WP12 | T5 symbol graph | add a fifth root, hide one through alias/re-export/computed import, or accidentally require WP12 to have already removed `/api/sync`/startup/CLI/restore | -| **N2** — WP9 replaces the landed placeholder, consumes existing request/snapshot/projection seams, creates the contract test, and typechecks without WP10-WP12 | focused tests + typecheck | redefine a WP8b type/helper, refer to a nonexistent later helper, leave a throwing placeholder, or claim the absent test file is merely extended | +| **C14, WP9-bounded** — the 16 management roots reach catalog writers only through convergence; exactly four documented transitional roots remain until WP12 and every one already requires permanent K | T2 barrier + T5 symbol graph | add a fifth root, omit K/permit from one retained chain, hide one through alias/re-export/computed import, or accidentally require WP12 to have already rewired `/api/sync`/startup/CLI/restore | +| **N2** — WP9 replaces the landed placeholder, consumes existing request/snapshot/projection seams, creates permanent K and its resolver plus the contract test, and typechecks without WP10-WP12 | focused tests + typecheck | move K into WP8b/WP11, redefine a WP8b type/helper, refer to a nonexistent later helper, leave a throwing placeholder, or claim the absent test file is merely extended | From 07c68bab156101229086010d13107d20d5569432 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Tue, 4 Aug 2026 20:43:04 +0900 Subject: [PATCH 058/163] feat(config): the generation guard WP9 is told to stop without MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The contract requires three exports and `src/config.ts` had two. WP9's phase entry says to halt if `withExpectedConfigGenerationSync` is missing (010_catalog_seam.md:142-148), and it was: importing it failed with `Export named 'withExpectedConfigGenerationSync' not found`. The guard joins the EXISTING config mutation transaction rather than opening its own connection. That is the whole point of it: a second connection puts a window between reading generation N and publishing the catalog, which is exactly the N→N+1 race the guard exists to close. It enters `withConfigMutationLockSync`, reads through the already-held `configMutationDatabase`, runs the callback only on a match and only while `BEGIN IMMEDIATE` is still held, and returns typed `matched`/`conflict`/ `unavailable` outcomes rather than throwing or returning a boolean. Two real child processes race it, and the winner deliberately holds the transaction until the loser has returned `busy`, so "exactly one callback" is observed rather than assumed. A mismatch is proven by counting callback invocations, not by reading the result kind — the callback must never run at all. Reverting to a second connection turns the contention test red; running the callback on mismatch turns the counter test red. One reading was ambiguous and is worth stating: "exactly one callback per generation" applies to concurrent contenders, not to sequential calls, because catalog-only work validates the generation without bumping it. Making the generation one-shot would contradict that rule. --- src/codex/convergence-types.ts | 9 ++ src/config.ts | 41 ++++++++ tests/codex-config-generation.test.ts | 142 +++++++++++++++++++++++++- 3 files changed, 191 insertions(+), 1 deletion(-) diff --git a/src/codex/convergence-types.ts b/src/codex/convergence-types.ts index 6e9c8031c..84b04d1eb 100644 --- a/src/codex/convergence-types.ts +++ b/src/codex/convergence-types.ts @@ -259,8 +259,17 @@ export type ConfigGenerationBump = | { kind: "conflict"; current: ConfigGeneration } | { kind: "unavailable"; reason: "busy" | "database" }; +export type ExpectedConfigGenerationSyncResult = + | { kind: "matched"; generation: ConfigGeneration; value: T } + | { kind: "conflict"; current: ConfigGeneration } + | { kind: "unavailable"; reason: "busy" | "database" }; + export type ReadConfigGeneration = () => ConfigGenerationRead; export type BumpConfigGeneration = (expected: ConfigGeneration) => ConfigGenerationBump; +export type WithExpectedConfigGenerationSync = ( + expected: ConfigGeneration, + commit: () => T, +) => ExpectedConfigGenerationSyncResult; export interface CommitExpectation { /** Read at admission. */ diff --git a/src/config.ts b/src/config.ts index 09019e9e8..5227cc4bf 100644 --- a/src/config.ts +++ b/src/config.ts @@ -10,10 +10,12 @@ import { bumpCurrentConfigGeneration, initializeConfigGeneration, readConfigGenerationAtPath, + readConfigGenerationInTransaction, } from "./codex/generation"; import type { BumpConfigGeneration, ReadConfigGeneration, + WithExpectedConfigGenerationSync, } from "./codex/convergence-types"; import { CODEX_ACCOUNT_NAMESPACE_COMBO_ALIAS_COLLISION_ERROR, @@ -1858,6 +1860,45 @@ export const bumpConfigGeneration: BumpConfigGeneration = expected => { } }; +function configGenerationFailureReason(error: unknown): "busy" | "database" { + const cause = error instanceof ConfigMutationLockError ? error.cause : error; + const code = cause && typeof cause === "object" && "code" in cause + ? String((cause as { code?: unknown }).code) + : ""; + const message = cause instanceof Error ? cause.message : ""; + return code === "SQLITE_BUSY" + || code === "SQLITE_LOCKED" + || /database (?:is|table is) locked/i.test(message) + ? "busy" + : "database"; +} + +export const withExpectedConfigGenerationSync: WithExpectedConfigGenerationSync = ( + expected, + commit, +) => { + let callbackThrew = false; + let callbackError: unknown; + try { + return withConfigMutationLockSync(() => { + const database = configMutationDatabase; + if (!database) throw new Error("Config mutation transaction database is unavailable."); + const current = readConfigGenerationInTransaction(database); + if (current.value !== expected.value) return { kind: "conflict", current }; + try { + return { kind: "matched", generation: current, value: commit() }; + } catch (error) { + callbackThrew = true; + callbackError = error; + throw error; + } + }); + } catch (error) { + if (callbackThrew && error === callbackError) throw error; + return { kind: "unavailable", reason: configGenerationFailureReason(error) }; + } +}; + function persistConfigUnlocked(config: OcxConfig): boolean { const configPath = getConfigPath(); const bytes = JSON.stringify(config, null, 2) + "\n"; diff --git a/tests/codex-config-generation.test.ts b/tests/codex-config-generation.test.ts index 6b143523c..566237d1d 100644 --- a/tests/codex-config-generation.test.ts +++ b/tests/codex-config-generation.test.ts @@ -1,6 +1,7 @@ import { afterEach, beforeEach, expect, test } from "bun:test"; -import { mkdtempSync, rmSync, writeFileSync } from "node:fs"; +import { existsSync, mkdtempSync, rmSync, writeFileSync } from "node:fs"; import { join } from "node:path"; +import { pathToFileURL } from "node:url"; import { Database } from "bun:sqlite"; @@ -10,9 +11,36 @@ import { readConfigGeneration, saveConfig, saveConfigPreservingClaudeCode, + withExpectedConfigGenerationSync, } from "../src/config"; import type { OcxConfig } from "../src/types"; +const CHILD_TIMEOUT_MS = 10_000; +const configModuleUrl = pathToFileURL(join(import.meta.dir, "../src/config.ts")).href; +const generationGuardRaceScript = ` + import { existsSync, writeFileSync } from "node:fs"; + import { withExpectedConfigGenerationSync } from ${JSON.stringify(configModuleUrl)}; + + const payload = JSON.parse(process.env.OCX_TEST_PAYLOAD); + const waitFor = path => { + const deadline = Date.now() + ${CHILD_TIMEOUT_MS}; + while (!existsSync(path)) { + if (Date.now() >= deadline) throw new Error(\`timed out waiting for \${path}\`); + Bun.sleepSync(5); + } + }; + writeFileSync(payload.readyPath, "ready"); + waitFor(payload.releasePath); + + const result = withExpectedConfigGenerationSync({ value: 0 }, () => { + writeFileSync(payload.callbackPath, payload.id); + waitFor(payload.peerOutcomePath); + return payload.id; + }); + writeFileSync(payload.outcomePath, JSON.stringify(result)); + process.stdout.write(JSON.stringify(result)); +`; + let testRoot = ""; let previousOpencodexHome: string | undefined; @@ -20,6 +48,31 @@ function config(port = 10100): OcxConfig { return { port, providers: {}, defaultProvider: "openai" }; } +async function waitForPaths(paths: readonly string[]): Promise { + const deadline = Date.now() + CHILD_TIMEOUT_MS; + while (!paths.every(existsSync)) { + if (Date.now() >= deadline) throw new Error(`timed out waiting for ${paths.join(", ")}`); + await Bun.sleep(5); + } +} + +async function collectGuardRaceChild( + child: ReturnType, +): Promise> { + const timeout = setTimeout(() => child.kill(), CHILD_TIMEOUT_MS); + try { + const [exitCode, stdout, stderr] = await Promise.all([ + child.exited, + new Response(child.stdout).text(), + new Response(child.stderr).text(), + ]); + expect(exitCode, stderr).toBe(0); + return JSON.parse(stdout) as Record; + } finally { + clearTimeout(timeout); + } +} + beforeEach(() => { previousOpencodexHome = process.env.OPENCODEX_HOME; testRoot = mkdtempSync(join(import.meta.dir, ".tmp-codex-config-generation-")); @@ -95,6 +148,89 @@ test("a stale expected value conflicts without changing the winner", () => { }); }); +test("the generation guard joins the existing mutation transaction", () => { + const result = withExpectedConfigGenerationSync({ value: 0 }, () => { + saveConfig(config()); + return "committed-with-nested-writer"; + }); + + expect(result).toEqual({ + kind: "matched", + generation: { value: 0 }, + value: "committed-with-nested-writer", + }); + expect(readConfigGeneration()).toEqual({ + kind: "ready", + generation: { value: 1 }, + }); +}); + +test("the generation guard never invokes its callback on a mismatch", () => { + saveConfig(config()); + let callbackRuns = 0; + + expect(withExpectedConfigGenerationSync({ value: 0 }, () => { + callbackRuns += 1; + return "must-not-run"; + })).toEqual({ kind: "conflict", current: { value: 1 } }); + expect(callbackRuns).toBe(0); +}); + +test("two real processes racing one generation run exactly one callback", async () => { + const releasePath = join(testRoot, "race-release"); + const ids = ["a", "b"] as const; + const children = ids.map((id, index) => { + const peerId = ids[1 - index]!; + const payload = { + id, + readyPath: join(testRoot, `${id}.ready`), + releasePath, + callbackPath: join(testRoot, `${id}.callback`), + outcomePath: join(testRoot, `${id}.outcome`), + peerOutcomePath: join(testRoot, `${peerId}.outcome`), + }; + return Bun.spawn([process.execPath, "--eval", generationGuardRaceScript], { + cwd: join(import.meta.dir, ".."), + env: { + ...process.env, + HOME: testRoot, + OPENCODEX_HOME: testRoot, + OCX_TEST_PAYLOAD: JSON.stringify(payload), + }, + stdin: "ignore", + stdout: "pipe", + stderr: "pipe", + }); + }); + + await waitForPaths([join(testRoot, "a.ready"), join(testRoot, "b.ready")]); + writeFileSync(releasePath, "go"); + const results = await Promise.all(children.map(collectGuardRaceChild)); + + expect(results.filter(result => result.kind === "matched")).toHaveLength(1); + expect(results.filter(result => ( + result.kind === "conflict" + || (result.kind === "unavailable" && result.reason === "busy") + ))).toHaveLength(1); + expect(["a", "b"].filter(id => existsSync(join(testRoot, `${id}.callback`)))).toHaveLength(1); + expect(readConfigGeneration()).toEqual({ + kind: "ready", + generation: { value: 0 }, + }); +}); + +test("a throwing guard callback rolls back and releases the transaction", () => { + expect(() => withExpectedConfigGenerationSync({ value: 0 }, () => { + throw new Error("guard callback failed"); + })).toThrow("guard callback failed"); + + expect(() => saveConfig(config())).not.toThrow(); + expect(readConfigGeneration()).toEqual({ + kind: "ready", + generation: { value: 1 }, + }); +}); + test("busy and unavailable databases return typed outcomes instead of throwing", () => { expect(readConfigGeneration().kind).toBe("ready"); const databasePath = join(testRoot, "config-mutation.sqlite"); @@ -103,6 +239,8 @@ test("busy and unavailable databases return typed outcomes instead of throwing", try { expect(readConfigGeneration()).toEqual({ kind: "unavailable", reason: "busy" }); expect(bumpConfigGeneration({ value: 0 })).toEqual({ kind: "unavailable", reason: "busy" }); + expect(withExpectedConfigGenerationSync({ value: 0 }, () => "must-not-run")) + .toEqual({ kind: "unavailable", reason: "busy" }); } finally { holder.exec("ROLLBACK"); holder.close(); @@ -112,4 +250,6 @@ test("busy and unavailable databases return typed outcomes instead of throwing", writeFileSync(testRoot, "not a directory", "utf8"); expect(readConfigGeneration()).toEqual({ kind: "unavailable", reason: "database" }); expect(bumpConfigGeneration({ value: 0 })).toEqual({ kind: "unavailable", reason: "database" }); + expect(withExpectedConfigGenerationSync({ value: 0 }, () => "must-not-run")) + .toEqual({ kind: "unavailable", reason: "database" }); }); From 96c2761b8674cf08b0eab254639a6da76f3d810b Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Tue, 4 Aug 2026 20:43:14 +0900 Subject: [PATCH 059/163] test(codex): every path ended in .json, so a .json check looked general Third pass at the same problem, one level down again. The paths were randomized but all ended `.json`, and the foreign row randomized its slug while pinning the description. Filtering targets to `.json` left all 55 tests green while the real writer routed a routed row to an extensionless catalog and got `clean`; special-casing the one description string did the same. Configured catalogs now run through the real writer with no extension, `.txt`, `.json5`, a trailing dot and an uppercase `.JSON`, basenames are fully randomized, and the foreign row randomizes both slug segments and its entire description. The values still fixed are the ones that carry meaning: whether a slug has a slash, the opencodex authorship signature, path topology, and the extension partitions themselves. Worth being honest about the limit: no finite random corpus can survive an adversary who special-cases the shape of a UUID. What this does buy is that a regression must now be deliberate rather than accidental, which is the failure mode that actually produced the five defects in this module. --- tests/codex-native-residue.test.ts | 71 +++++++++++++++++++++++++----- 1 file changed, 61 insertions(+), 10 deletions(-) diff --git a/tests/codex-native-residue.test.ts b/tests/codex-native-residue.test.ts index 4e7c58321..b086e3d5f 100644 --- a/tests/codex-native-residue.test.ts +++ b/tests/codex-native-residue.test.ts @@ -302,26 +302,26 @@ const catalogPathShapes: Array<{ name: string; configuredPath: (outsideRoot: string, leaf: string) => string; }> = [ - { name: "root-relative", configuredPath: (_outsideRoot, leaf) => `${leaf}.json` }, - { name: "nested-relative", configuredPath: (_outsideRoot, leaf) => `nested/${leaf}.json` }, + { name: "root-relative", configuredPath: (_outsideRoot, leaf) => leaf }, + { name: "nested-relative", configuredPath: (_outsideRoot, leaf) => `nested/${leaf}` }, { name: "absolute inside CODEX_HOME", - configuredPath: (_outsideRoot, leaf) => canonicalPathInCodexHome(`${leaf}.json`), + configuredPath: (_outsideRoot, leaf) => canonicalPathInCodexHome(leaf), }, { name: "absolute outside CODEX_HOME", - configuredPath: (outsideRoot, leaf) => join(outsideRoot, `${leaf}.json`), + configuredPath: (outsideRoot, leaf) => join(outsideRoot, leaf), }, { name: "parent-escaping relative", - configuredPath: (_outsideRoot, leaf) => `../${basename(codexHome)}-${leaf}.json`, + configuredPath: (_outsideRoot, leaf) => `../${basename(codexHome)}-${leaf}`, }, ]; for (const shape of catalogPathShapes) { test(`configured catalog classification follows the ${shape.name} path`, () => { const outsideRoot = mkdtempSync(join(tmpdir(), "ocx-native-residue-catalog-outside-")); - const configuredPath = shape.configuredPath(outsideRoot, `catalog-${randomUUID()}`); + const configuredPath = shape.configuredPath(outsideRoot, randomUUID()); const targetPath = resolve(realpathSync.native(codexHome), configuredPath); try { mkdirSync(dirname(targetPath), { recursive: true }); @@ -349,6 +349,56 @@ for (const shape of catalogPathShapes) { }); } +const productionCatalogLeafShapes = [ + { name: "extensionless", suffix: "" }, + { name: ".txt", suffix: ".txt" }, + { name: ".json5", suffix: ".json5" }, + { name: "trailing dot", suffix: "." }, + { name: "uppercase .JSON", suffix: ".JSON" }, +] as const; + +for (const shape of productionCatalogLeafShapes) { + const configuredLeaf = `${randomUUID()}${shape.suffix}`; + test(`production writer routes the ${shape.name} configured catalog ${configuredLeaf}`, async () => { + const catalogPath = canonicalPathInCodexHome(`nested/${configuredLeaf}`); + mkdirSync(dirname(catalogPath), { recursive: true }); + writeFileSync( + pathInCodexHome("config.toml"), + `model_catalog_json = ${JSON.stringify(`nested/${configuredLeaf}`)}\n`, + ); + writeFileSync(catalogPath, JSON.stringify({ models: [] })); + const config: OcxConfig = { + port: 10100, + defaultProvider: "fixture", + providers: { + fixture: { + adapter: "openai-chat", + baseUrl: "https://fixture.invalid/v1", + liveModels: false, + models: ["fixture-model"], + }, + }, + }; + + const sync = await syncCatalogModels(config); + const catalog = JSON.parse(readFileSync(catalogPath, "utf8")) as { + models: Array>; + }; + const routedRows = catalog.models.filter(model => + typeof model.description === "string" + && model.description.startsWith("Routed via opencodex → ") + ); + + expect(sync).toMatchObject({ path: catalogPath, catalogWritten: true }); + expect(routedRows).toHaveLength(1); + expect(classifyNativeRoutedResidue()).toEqual({ + kind: "residue", + surface: "catalog", + path: catalogPath, + }); + }); +} + test("an atomic-write artifact beside the configured catalog is indeterminate", () => { const catalogPath = canonicalPathInCodexHome("nested/custom-catalog.json"); const artifactPath = `${catalogPath}.ocx.42.7.tmp`; @@ -495,7 +545,7 @@ test("duplicate configured catalog paths are indeterminate", () => { }); }); -const arbitraryComboAlias = `round4-edge-bare-${randomUUID()}`; +const arbitraryComboAlias = randomUUID(); test(`production-generated arbitrary bare combo alias ${arbitraryComboAlias} is routed residue`, async () => { const catalogPath = canonicalPathInCodexHome("opencodex-catalog.json"); @@ -549,11 +599,12 @@ test(`production-generated arbitrary bare combo alias ${arbitraryComboAlias} is }); }); -const arbitraryForeignSlug = `foreign-${randomUUID()}/model-${randomUUID()}`; +const arbitraryForeignSlug = `${randomUUID()}/${randomUUID()}`; +const arbitraryForeignDescription = randomUUID(); -test(`arbitrary foreign slash-bearing slug ${arbitraryForeignSlug} is indeterminate`, () => { +test(`arbitrary foreign row ${arbitraryForeignSlug} described as ${arbitraryForeignDescription} is indeterminate`, () => { writeFileSync(pathInCodexHome("opencodex-catalog.json"), JSON.stringify({ - models: [{ slug: arbitraryForeignSlug, description: "User-authored catalog row" }], + models: [{ slug: arbitraryForeignSlug, description: arbitraryForeignDescription }], })); expect(classifyNativeRoutedResidue()).toMatchObject({ From 417734502143c14e76ca0153b7e86e29ccdf86a4 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Tue, 4 Aug 2026 20:48:37 +0900 Subject: [PATCH 060/163] test(restart): a deadline test that read the real clock expected it not to move MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This failed once in a full-suite run at 59999 and passed every time in isolation, which is the signature of load rather than logic. `waitForRestartDrain` computes `deadlineMs - now()`, so the scheduled value shrinks as real time passes — correct behavior — while this test asserted an exact 60000 without injecting a clock, unlike its neighbors. Under full-suite CPU contention a millisecond elapsed and the assertion lost. Injecting the fixed clock the surrounding tests already use keeps the exact assertion rather than loosening it to a range; a precise expectation against a deterministic clock is worth more than a fuzzy one against a real clock. Forcing two milliseconds of real elapsed time reproduces it at 59997 without the injection and passes with it. No production change: the deadline is right to shrink. The rest of the file was checked for the same dependence and had none. --- tests/system-restart.test.ts | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tests/system-restart.test.ts b/tests/system-restart.test.ts index dd95ee28c..836ade832 100644 --- a/tests/system-restart.test.ts +++ b/tests/system-restart.test.ts @@ -327,6 +327,7 @@ describe("acceptSystemRestart", () => { let scheduled: (() => void | Promise) | null = null; const timers: Array<{ fn: () => void; ms: number; cancelled: boolean }> = []; let rejectStop!: (reason?: unknown) => void; + let now = 1_000; acceptSystemRestart({ isDraining: () => false, @@ -339,6 +340,7 @@ describe("acceptSystemRestart", () => { timers.push(timer); return () => { timer.cancelled = true; }; }, + now: () => now, setDraining: () => { calls.push("latched"); }, drainAndShutdown: () => { calls.push("drain"); From 364496fc968fd12734a2e6fe2bec9e219d9d1c90 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Tue, 4 Aug 2026 21:01:44 +0900 Subject: [PATCH 061/163] docs(substrate): a lock around the write does not freshen the read MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Round 4 took the lock apart from the other side. K serialized replacement, but retained roots were told to prepare and read outside it, and the real /api/sync chain reads the catalog (catalog/sync.ts:513), captures onDiskCatalog (:520), awaits provider gathering (:526), then writes from that captured state (:565). So a retained writer that gathered first and takes K second overwrites a newer catalog with stale bytes while legitimately holding the lock. The earlier test only covered convergence revalidating after a retained writer won K — the opposite direction was never exercised. Only slow provider and network work may now sit outside K: every first-party root either recomputes its authoritative read and derivation under the lock, or carries source evidence and revalidates after acquiring it. /api/sync takes the evidence path because its gather is the slow one; startup invalidation, sync-cache and restore recompute under K. The permit was the fourth appearance of this unit's oldest mistake. The contract called it opaque and non-forgeable, but the only stated enforcement was that a writer call without the permit type fails to compile — so a legitimate caller could keep the permit and write after K released, and a symbol graph would see a permit-bearing path and pass. Absence of a permit-less call was being read as proof the lock was held. Permits are now minted per acquisition, registered active against the K transaction and canonical CODEX_HOME, checked for liveness by every low-level mutator before it touches the filesystem, and revoked in a finally before release. The cache epoch had the same hole one level down. resolveCodexRuntime returns the cached object itself (runtime.ts:397-400) with mutable interfaces (:16), the bundled loader likewise returns its cached value, and an existing test mutates that shared object in place (codex-runtime.test.ts:428-431). An in-place mutation triggers no assignment, reset or invalidation, so the epoch never moves and a recorded identity never changes. Memo snapshots are deeply frozen and returned as non-aliased readonly views; that existing test has to be rewritten, which the plan now says out loud so nobody mistakes it for a break. The reviewer separately enumerated the lock graph and found no cycle: N → K → C with C → K, K → N and a held N → H forbidden, and history's H → N edge fail-fast rather than blocking. --- .../005_contract.md | 174 ++++++++++-- .../010_catalog_seam.md | 267 ++++++++++++------ 2 files changed, 325 insertions(+), 116 deletions(-) diff --git a/devlog/_plan/260804_codex_write_substrate/005_contract.md b/devlog/_plan/260804_codex_write_substrate/005_contract.md index ceb028ce3..bc7601a67 100644 --- a/devlog/_plan/260804_codex_write_substrate/005_contract.md +++ b/devlog/_plan/260804_codex_write_substrate/005_contract.md @@ -646,7 +646,7 @@ consumer. This guard uses the config mutation lock that exists in WP8b and has n dependency on WP11's future native lock. Catalog-only work validates but does not bump the config generation because it writes no persisted OpenCodex config bytes. -### Catalog serialization is a permanent, separate primitive (seam audit round 3) +### Catalog serialization is a permanent, separate primitive (seam audit rounds 3-4) The round-3 auditor ran the retained management `POST /api/sync` chain while a catalog candidate was paused after validation. `refreshCodexModelCatalog` @@ -674,18 +674,73 @@ surface and continues to require K after WP11 and after the four transitional ro are removed. K exposes a synchronous owner-held callback acquired before any config transaction; -that callback receives an opaque, non-forgeable catalog-write permit. Every low-level -catalog/hashed-backup/legacy-backup/models-cache mutator requires that permit. The -callback may enter `withExpectedConfigGenerationSync`, but performs no provider -request, runtime probe, OAuth refresh, subprocess, Promise, or other awaited work. -Gather stays outside K. At WP9, convergence's outer async orchestration may retry a -fail-fast K acquisition within `deadlineMs`, but each acquisition attempt and the -complete K -> C publication callback are synchronous. Each of the four retained -writer chains keeps its public signature, return shape, gather order, and ordinary -behavior; its actual synchronous replacement section enters K and supplies the -permit. Lock busy/unavailable follows that retained function's existing no-write or -write-failure path rather than changing it to a Promise. No retained chain holds K -while gathering. +that callback receives a fresh catalog-write permit for that acquisition. Every +low-level catalog/hashed-backup/legacy-backup/models-cache mutator requires the +permit. The callback may enter `withExpectedConfigGenerationSync`, but performs no +provider request, runtime probe, OAuth refresh, subprocess, Promise, or other awaited +work. At WP9, convergence's outer async orchestration may retry a fail-fast K +acquisition within `deadlineMs`, but each acquisition attempt and the complete K -> C +publication callback are synchronous. Lock busy/unavailable follows each retained +function's existing no-write or write-failure path rather than changing it to a +Promise. + +Round 4 showed why “the replacement is under K” is not enough. The retained +`/api/sync` chain reads the active catalog, captures `onDiskCatalog`, awaits provider +gathering, and only then writes from that captured state +(`src/codex/catalog/sync.ts:513,520,526,565`). If convergence publishes Y while that +await is pending, taking K afterwards does not make the captured X fresh: the retained +writer can legally overwrite Y while holding K. Cache invalidation and restore are +also read-transform-write operations, not bare replacements. + +Only slow provider/network gathering may therefore remain outside K. Every +first-party catalog root uses exactly one of these two freshness shapes: + +1. **Under-K recomputation:** acquire K before the authoritative filesystem read, + perform the deterministic read-transform/derivation and every resulting write + while K remains live. This is required for the synchronous retained startup and + CLI cache invalidation roots and native restore; they have no provider/network + await that justifies a pre-K filesystem snapshot. If a compatibility helper has + already prepared such state, it repeats the authoritative read and derivation + under K and discards the pre-K result. +2. **Evidence-bound precomputation:** slow provider/network work and deterministic + preparation may run before K only when every filesystem value, absence, selector, + and process-local authority that influenced the result is sealed as candidate + evidence. After acquiring K, the writer revalidates that complete evidence before + any mutation. Catalog convergence returns `stale` on drift; retained `/api/sync` + discards/regathers or follows its existing no-write/write-failure path without + changing its public return shape. Both use this shape so provider gathering stays + outside K without treating the earlier `onDiskCatalog` as current merely because K + was later acquired. + +No retained root may mix the shapes by reading X before K and then performing only +the transform or replacement under K. Public signatures, return shapes, slow-provider +gather order, and compatibility behavior stay unchanged; the freshness boundary does +not. K makes the state-producing read-transform-write transaction serializable by +lock-held recomputation or lock-held evidence validation, not just by guarding its +last rename. + +The same round exposed a second absence-as-proof defect. An opaque TypeScript type +proves only that a permit-bearing call path exists; it cannot prove the callback still +holds K. `src/codex/catalog-write-serialization.ts` therefore owns a module-private +active-permit registry. Each successful `BEGIN IMMEDIATE` acquisition mints a new +unexported permit object and transaction identity and registers their binding to the +canonical `CODEX_HOME`. There is no public constructor, brand, or registration API. +Every low-level mutator calls the K owner's runtime assertion with its permit and the +canonical `CODEX_HOME` that owns the target set before its first filesystem mutation, +including temp creation, hardening, unlink, link, rename, truncate, or replacement. +That owning home is not inferred from a target parent because an accepted configured +catalog target may be absolute and outside `CODEX_HOME`. The assertion accepts only +the exact registered object whose transaction is still active and whose bound home +equals the mutator's supplied owning home. + +The K owner revokes/removes the permit in `finally` **before** committing/rolling back +and releasing K, including callback throws. One live permit may authorize the fixed +sequence of low-level mutations in its own callback; it cannot be reused by a later K +acquisition, even for the same home. A leaked post-callback permit, an object forged +through a cast/prototype/symbol copy, a revoked permit presented during another +transaction, and a live permit for home A presented to a home-B writer all refuse +before any filesystem mutation. The symbol graph remains a useful reachability check, +but runtime liveness/home validation is the proof that K is actually held. The global order is: @@ -833,6 +888,29 @@ private candidate and compared with the owner's current pair under K and C befor the first write. Any change, including invalidate-and-repopulate with byte-identical content, is `stale`. +Round 4 made “immutable” operational rather than aspirational. Today +`resolveCodexRuntime` returns `resolveCache.value` directly +(`src/codex/runtime.ts:397-400`), its nested interfaces are mutable +(`src/codex/runtime.ts:16`), and a test mutates that shared object in place +(`tests/codex-runtime.test.ts:428-431`). The bundled loader likewise returns +`bundledCatalogCache.value` (`src/codex/catalog/bundled.ts:179`). Such an alias changes +authority without owner assignment, invalidation, or epoch movement, so neither the +epoch nor object identity can detect it. + +The runtime and bundled-cache owners must instead clone incoming values into private +owner snapshots, recursively freeze every reachable object and array before +publication, and never return the private cache object itself. A read API returns a +detached recursively frozen clone or an immutable view paired with the owner's +epoch/value identity; that returned graph and the candidate's sealed copy must not +provide a mutable alias back to owner state. Shallow `Object.freeze` is insufficient. +The cache-backed read interfaces expose recursively readonly shapes as well as runtime +enforcement; TypeScript `Readonly` at the top level alone is not the contract. Any +intentional cache change goes through the owner assignment/invalidation API, which +constructs and deep-freezes a new private snapshot and increments the epoch before +exposing it. +Existing tests and callers may no longer rely on mutating an object returned by +`resolveCodexRuntime` or the bundled loader to alter cache state. + When runtime identity influences a candidate, `codex-runtime.json` is additionally an **always-required** `runtime-selection` filesystem observation, PRESENT or ABSENT, even when the runtime came from a warm process memo. A PRESENT persisted selection is @@ -1315,7 +1393,7 @@ already migrated lifecycle and explicit callers: |---|---|---| | native config/profile | `src/codex/internal/native-writer.ts` | `src/codex/convergence.ts` only | | injection journal create/mark/restore/remove | `src/codex/internal/journal-writer.ts` | `src/codex/convergence.ts` only | -| catalog, hashed/legacy backups, models cache | `src/codex/internal/catalog-writer.ts`, each mutation requiring K's opaque permit | `src/codex/convergence.ts` only | +| catalog, hashed/legacy backups, models cache | `src/codex/internal/catalog-writer.ts`, each mutation requiring K's runtime-validated live permit | `src/codex/convergence.ts` only | | history DB rows, manifest, rollout files | history write exports in `src/codex/internal/history-writer.ts` | `src/codex/history-worker.ts` only | | transition pair and history schedule/terminal row | `src/codex/transition-state.ts` | `src/codex/convergence.ts` and `src/codex/history-worker.ts` only | | JSON provenance ledger | `updateIntegrationRecord` in `src/codex/integration-record.ts` | `src/codex/convergence.ts` only | @@ -1331,19 +1409,23 @@ lifecycle convergence: | WP9 transitional legacy root | Exact writer chain still permitted | WP12 final action | |---|---|---| -| management `POST /api/sync` | `src/server/management/config-routes.ts` -> `src/codex/sync.ts` -> `src/codex/refresh.ts` -> K -> catalog writer | rewire to full convergence and `toSyncResponse` | -| server startup cache invalidation | `src/server/index.ts` -> K -> models-cache writer | route startup through full convergence/observer | -| CLI `sync-cache` | `src/cli/index.ts` -> K -> models-cache writer | route the CLI command through full convergence | -| native restore | `src/codex/inject.ts` -> K -> catalog restore writer | move restore behind full convergence/provenance | +| management `POST /api/sync` | `src/server/management/config-routes.ts` -> `src/codex/sync.ts` -> `src/codex/refresh.ts` -> provider gather -> K -> evidence revalidation -> catalog writer | rewire to full convergence and `toSyncResponse` | +| server startup cache invalidation | `src/server/index.ts` -> K -> authoritative cache read/derivation -> models-cache writer | route startup through full convergence/observer | +| CLI `sync-cache` | `src/cli/index.ts` -> K -> authoritative cache read/derivation -> models-cache writer | route the CLI command through full convergence | +| native restore | `src/codex/inject.ts` -> K -> authoritative backup/catalog read/derivation -> catalog restore writer | move restore behind full convergence/provenance | This is an exact transitional allowlist by root module and writer symbol, not a directory wildcard. `src/codex/sync.ts`, `src/codex/refresh.ts`, CLI, and `src/codex/inject.ts` are therefore permitted only through the rows above at the WP9 commit; no fifth legacy root may appear. WP12 removes every row and activates -the final table. Their signatures, return values, gather order, and compatibility -behavior do not change in WP9; only their real synchronous replacement sections -acquire K, after N if a later phase has already placed that root under N. A contract -test cannot enforce both versions at once, so the graph +the final table. Their signatures, return values, slow provider/network gather order, +and compatibility behavior do not change in WP9. Retained `/api/sync` binds its pre-K +provider gather to complete source/process evidence and revalidates that evidence +after K acquisition; the other three synchronous retained chains acquire K before +their authoritative filesystem read and keep it through deterministic derivation and +write. Each path supplies a runtime-live, same-home permit, after N if a later phase +has already placed that root under N. A contract test cannot enforce both versions at +once, so the graph fixture carries an explicit `"wp9-transitional" | "wp12-final"` inventory version: WP9 expects exactly four legacy chains, and WP12 changes that expectation to zero. @@ -1367,7 +1449,8 @@ version, `history-job.ts`, management routes, CLI modules, `sync.ts`, `refresh.t dispatch a Worker, or read only. That final prohibition must not be applied to the four explicit WP9 transitional rows before WP12 owns their migration. At both inventory versions, every catalog/backup/cache mutator must be reachable only -with an opaque live K permit, and inverse-order graph fixtures reject `C -> K` and +with K's permit and must call K's runtime liveness/transaction/home assertion before +its first filesystem mutation. Inverse-order graph fixtures reject `C -> K` and `K -> N` even through wrappers, aliases, or re-exports. ## 9. Baseline classes @@ -1433,6 +1516,28 @@ stale rather than replace Y. Run the same exclusion shape for startup cache invalidation, CLI `sync-cache`, and native restore, and require the inventory graph to reject any catalog/backup/cache write reachable without K's permit. +Round 4 requires the direction that barrier did not cover. First let a retained +`/api/sync` A read X and begin slow provider gathering **before it owns K**. Let +convergence B acquire K and publish Y, then resume A so it acquires K second. A must +revalidate its complete pre-K evidence and discard/regather through its unchanged +public result path; it must not replace Y with X-derived bytes. Repeat with retained +`/api/sync` B as the K-first publisher, +so retained-vs-retained is covered independently of convergence. For the synchronous +startup cache invalidation, CLI `sync-cache`, and native restore roots, instrument the +authoritative filesystem read and prove it cannot begin before K; pause after that +read and prove a second retained/convergence writer cannot publish until the complete +read-transform-write releases K. The forbidden trace in every case is “A gathered X +first, B published Y under K, A acquired K second and restored X-derived bytes.” + +Exercise K's runtime permit assertion against real temporary targets and a mutation +spy. Leak a permit and call a writer after its callback, present that revoked permit +during a later acquisition for the same home, forge a permit through a type cast and +prototype/symbol copying, and pass a still-live home-A permit to a home-B writer. Each +attempt refuses before temp creation, chmod, link, rename, unlink, truncate, or target +replacement, and target bytes remain unchanged. A fresh permit may authorize all +fixed writes inside its own live callback; this is distinct from reusing it in another +K transaction. + Table-drive every `CatalogSourceRole`. Gather from a present source, truncate-and-rewrite that same inode, and require `stale` before any write. Gather with `$CODEX_HOME/config.toml` absent, then create it with @@ -1462,6 +1567,16 @@ and removal. A candidate influenced by runtime identity but missing the PRESENT- ABSENT `runtime-selection` observation is structurally refused. A PRESENT R2 that disagrees with warm R1 may not be used to prepare a candidate in the first place. +Obtain runtime and bundled-cache results through their real public read APIs, gather a +candidate from them, and then attempt nested object and array mutation through the +returned values. The owner snapshot must remain byte-for-byte unchanged and the +mutation must be impossible because the returned detached clone or immutable view is +recursively frozen. If a supported explicit owner mutation is used instead, it must +move the epoch and the pending commit must return `stale` before any write. The +fixture must go red when either API returns its private cache object directly or +freezes only the top level; rewrite the existing runtime test that mutates the shared +alias so it uses the intentional owner mutation seam. + Race two create-once backup publishers after both observed ABSENT. Exactly one no-clobber publication wins; the loser receives `EEXIST`, validates and preserves the winner, and neither ordinary rename nor `atomicWriteFile` is called. Repeat with @@ -1494,7 +1609,9 @@ writes to it. - C14 — all 16 management callers funnel through `convergeCodex`, enforced by the symbol graph; its WP9 inventory permits exactly the four transitional chains and its WP12-final inventory permits none. At both versions every first-party - catalog/backup/cache write requires the same permanent K permit. + catalog/backup/cache write requires a fresh permit from the same permanent K owner, + and every low-level mutator rejects leaked, reused, forged, revoked, or wrong-home + permits at runtime before filesystem mutation. - C16 — one owner, one schema; a record from any phase reads in every other. - C17 — cooperating transition ABA is detected by the durable config/native generations and exact txId, and a parent target that drifts once between gather @@ -1505,11 +1622,14 @@ writes to it. drift and a single-direction raw CODEX_HOME-selector/canonical-root retarget before writing. A runtime-influenced candidate always carries PRESENT-or-ABSENT `codex-runtime.json` evidence, and any used runtime/bundled process memo must retain - its exact monotonic epoch and immutable value identity through the commit check. + its exact monotonic epoch and deeply immutable, non-aliased value identity through + the commit check. Cooperating config N -> N+1 is prevented while the catalog callback holds C, and the two-process real `/api/sync` barrier proves every retained first-party catalog - writer is serialized by K so convergence cannot overwrite a later first-party - publication with stale gathered bytes. Create-once backups use atomic no-clobber + writer serializes or revalidates its authoritative read-transform-write under K, + including the retained-gathers-first/acquires-second direction against convergence + and another retained writer, so neither side can restore stale gathered bytes over + a later first-party publication. Create-once backups use atomic no-clobber publication. An arbitrary filesystem, selector, or content A→B→A that completes wholly between two checks, and a non-cooperating write after the final comparison, are explicitly not claimed. diff --git a/devlog/_plan/260804_codex_write_substrate/010_catalog_seam.md b/devlog/_plan/260804_codex_write_substrate/010_catalog_seam.md index 354582964..bc32cc44a 100644 --- a/devlog/_plan/260804_codex_write_substrate/010_catalog_seam.md +++ b/devlog/_plan/260804_codex_write_substrate/010_catalog_seam.md @@ -24,11 +24,11 @@ native generation requires matching history schedule fields, every transition that was not published (`src/codex/transition-state.ts:74-83,314-344,420-428`). All current-code citations and diff context below were rechecked on 2026-08-04 -at `f51404f1670e57bdfb1637adf2d5d0cb5aeaf1f5`. The contract citations refer to +at `417734502143c14e76ca0153b7e86e29ccdf86a4`. The contract citations refer to the authoritative worktree amendment: permanent K, owner-held config generation, home/source/process-local evidence, and no-clobber publication are at -`005_contract.md:609-887,895-991,1021-1072`; K's owner/path and the four -transitional chains are fixed at `005_contract.md:1234-1289,1291-1371`. +`005_contract.md:609-762,830-965,967-1124`; K's owner/path and the four +transitional chains are fixed at `005_contract.md:1256-1454`. ## IN / OUT @@ -38,7 +38,7 @@ IN — observe-only admission and gather: generation observation that never creates, initializes, chmods, or registers `config-mutation.sqlite`. The existing `readConfigGeneration` is not that API: it resolves/records the path and opens SQLite with `create:true` - (`src/config.ts:1741-1771,1845-1849`, `src/codex/generation.ts:93-103`). WP9 + (`src/config.ts:1743-1774,1847-1853`, `src/codex/generation.ts:93-103`). WP9 consumes the contract-owned `withExpectedConfigGenerationSync`; it does not wrap this observer in the lock or redefine the generation contract. - `src/codex/catalog-admission.ts` (MODIFY) — keep the landed request constructor @@ -52,13 +52,13 @@ IN — observe-only admission and gather: closed `CatalogSourceRole`, `CatalogHomeSelectionObservation`, `CatalogSourceObservation`, `CatalogSourceEvidence`, `CatalogProcessLocalEvidence`, and `CatalogAdmissionSnapshot.sourceEvidence` - additions from `005_contract.md:895-991`; no WP9-private duplicate type is allowed. + additions from `005_contract.md:967-1097`; no WP9-private duplicate type is allowed. - `src/codex/catalog/filesystem-evidence.ts` (NEW) — sole owner of gather source reads and target probes. Its opaque session records PRESENT and ABSENT observations before returning, captures and seals catalog-home selection before accepting any derived path, seals the complete closed role map into the candidate, and is the only gather path permitted to call filesystem consultation primitives - (`005_contract.md:1021-1046`). + (`005_contract.md:1112-1124`). - `src/codex/runtime.ts`, `src/codex/catalog/bundled.ts` (MODIFY) — catalog gather uses a gather-specific observe-only pair: `peekCodexRuntimeForCatalogGather(evidenceSession)` and @@ -69,15 +69,24 @@ IN — observe-only admission and gather: `catalog-unavailable`. `bundled.ts` owns the catalog-specific adapter; `runtime.ts` exposes only a process-cache peek and pure persisted-state parser and never imports the catalog evidence module. A cold miss never becomes permission to - execute Codex. Each mutable runtime/bundled memo owns a process-lifetime monotonic - epoch and immutable value identity; population, replacement, clear, invalidation, - persisted-runtime write, and test reset advance the applicable epoch. Gather seals - the exact epoch/value identity it consumes, and commit revalidates both before its - first write. Whenever runtime identity influences a candidate, the evidence session + execute Codex. Each runtime/bundled memo owns a process-lifetime monotonic epoch and + a private, recursively frozen value snapshot. Owners clone incoming values, + deep-freeze every reachable object and array before publication, expose recursively + readonly types, and return only detached deeply frozen clones or non-aliased + immutable views; no caller receives the private cache object itself. Population, + replacement, clear, invalidation, persisted-runtime write, and test reset advance + the applicable epoch before exposing the replacement. Gather seals the exact + epoch/value identity plus a non-aliased immutable candidate copy, and commit + revalidates both before its first write. Whenever runtime identity influences a + candidate, the evidence session records `codex-runtime.json` as PRESENT or ABSENT even on a warm-cache hit. This is required because `persistCodexRuntime` writes the file and clears the memo without advancing config generation (`src/codex/runtime.ts:213-229`), while bundled - cache hits and replacements are process-local (`src/codex/catalog/bundled.ts:179-209`). + cache hits and replacements are process-local; the bundled loader currently returns + its private cached object directly at `src/codex/catalog/bundled.ts:186` + (`src/codex/catalog/bundled.ts:179-209`). `resolveCodexRuntime` likewise returns its + private cached object at `src/codex/runtime.ts:397-400`, and its result interfaces + are mutable at `src/codex/runtime.ts:16-40`. The ordinary resolver reaches `probeVersion`, whose sandbox deliberately calls `mkdtempSync` and `rmSync` (`src/codex/runtime.ts:231-279,327-340,397-405`), while bundled loading both calls the persisting resolver and runs `codex debug models` @@ -111,18 +120,26 @@ IN — fixed commit and convergence: - `src/codex/catalog-write-serialization.ts` (NEW) — permanent synchronous K owner, keyed by effective user plus canonical `CODEX_HOME`, backed by its own SQLite database with `busy_timeout=0` and `BEGIN IMMEDIATE`. It returns only from an - owner-held synchronous callback carrying an opaque, non-forgeable permit. It is - separate from both N and `config-mutation.sqlite`, and WP11 never replaces it. + owner-held synchronous callback carrying a fresh private permit minted for that + acquisition. A module-private active-permit registry binds the exact permit object + to the active K transaction identity and canonical owning home; there is no public + constructor, brand, or registration API. The owner revokes the permit in `finally` + before commit/rollback releases K, including when the callback throws, and exports + only the assertion low-level mutators need. K is separate from both N and + `config-mutation.sqlite`, and WP11 never replaces it. - `src/codex/user-identity.ts` (MODIFY) — add `resolveCodexCatalogSerializationDatabasePath` beside the landed native coordinator resolver. Consumers use its final path verbatim; the K and N database paths must be - distinct (`005_contract.md:1234-1289`). + distinct (`005_contract.md:1256-1367`). - `src/codex/internal/catalog-writer.ts` (NEW/MOVE) — the contract-owned low-level owner for catalog, hashed/legacy backups, and models cache. Every mutator requires - K's opaque live permit. Do not create the obsolete `internal/catalog-commit.ts` - name (`005_contract.md:1314-1325`). -- `src/codex/convergence.ts` (NEW) — catalog gather/commit orchestration and the - only WP9 module allowed to call symbols in `internal/catalog-writer.ts`. + the permit plus canonical owning `CODEX_HOME` and calls K's runtime assertion before + temp creation, hardening, unlink, link, rename, truncate, replacement, or any other + filesystem mutation. It never reads or mutates K's private registry. Do not create + the obsolete `internal/catalog-commit.ts` name (`005_contract.md:1369-1403`). +- `src/codex/convergence.ts` (NEW) — primary catalog gather/commit orchestration for + the 16 management mutations. The symbol graph additionally permits only the exact + four WP9 transitional writer chains below; WP12 removes those exceptions. - `src/codex/management-convergence.ts` (MODIFY) — retain the landed management-only factory and catalog-only projection, but replace the placeholder body at lines 81-96 with the real call into `convergence.ts`. The factory keeps @@ -138,19 +155,31 @@ IN — management callers and tests: - `src/server/management-api.ts`, `src/server/management/context.ts`, and the four invoking route modules (MODIFY) — replace the swallowed helper with a total, lazy catalog-convergence adapter returning `CatalogDisposition`. -- The four WP9-transitional chains are IN, but only for K acquisition at their real - synchronous replacement sections. In `src/codex/catalog/sync.ts`, preserve the - public signatures of `syncCatalogModels`, `invalidateCodexModelsCache`, and - `restoreCodexCatalog`, but prepare/read outside K and acquire K only around the - low-level catalog/backup/cache replacement that consumes its permit. - `src/codex/refresh.ts` keeps `/api/sync`'s async gather order and invokes those - retained K-protected publications. Startup in `src/server/index.ts`, CLI - `sync-cache` in `src/cli/index.ts`, and native restore in `src/codex/inject.ts` - keep their current caller behavior; any explicit import changes needed after the - facade stops re-exporting writers are mechanical. No retained chain holds K while - reading or gathering. This is serialization of the retained roots, not WP12's - convergence rewire (`005_contract.md:1327-1348`). +- The four WP9-transitional chains are IN and use one of two freshness shapes; none may + read X before K and merely transform or replace X-derived bytes under K. Retained + management `POST /api/sync` uses **evidence-bound precomputation**: preserve its + public signature and slow provider/network gather order, seal every filesystem + value/absence/selector and process-local authority that influenced the candidate, + then revalidate the complete evidence after acquiring K and before any mutation. + Drift discards/regathers or follows the existing no-write/write-failure return path. + This shape is required because current `syncCatalogModels` reads the target and + `onDiskCatalog`, awaits provider gathering, and derives the replacement from that + captured merge input at `src/codex/catalog/sync.ts:513-520,526,565`. + Startup cache invalidation in `src/server/index.ts`, CLI `sync-cache` in + `src/cli/index.ts`, and native restore in `src/codex/inject.ts` use **under-K + recomputation**: they acquire K before their authoritative catalog/backup/cache read, + repeat/discard any pre-K prepared state, and keep K live through deterministic + derivation and every resulting write. These three chains are synchronous and have no + provider/network await that justifies a pre-K filesystem snapshot. Public signatures, + return values, and compatibility behavior stay unchanged; explicit imports needed + after the facade stops re-exporting writers are mechanical. This is freshness-safe + serialization of the retained roots, not WP12's convergence rewire + (`005_contract.md:1405-1454`). - `tests/codex-refresh.test.ts` and the existing management route suites (MODIFY). +- `tests/codex-runtime.test.ts` (MODIFY) — rewrite the fixture at lines 428-431 that + currently mutates `resolveCodexRuntime()`'s shared cached result in place. It must use + the intentional owner mutation/invalidation seam once returned snapshots are deeply + immutable; this is expected fallout, not an implementation regression. - `tests/codex-convergence-contract.test.ts` (CREATE). It does not exist in the WP8b tree; WP9 creates it rather than “extending” an imaginary file. @@ -174,13 +203,14 @@ management factory/projection into `convergence.ts` when it installs the full entry point; that later move is a module consolidation, not completion of an unfinished WP9 branch. -Phase-entry gate: the audited source currently exports `readConfigGeneration` and -`bumpConfigGeneration` only (`src/config.ts:1845-1859`); the amended contract assigns -the executable `withExpectedConfigGenerationSync` owner seam to WP8b -(`005_contract.md:617-647`). WP9 implementation starts after that prior phase lands. -If the seam is still absent, stop and report the WP8b scope dependency; do not emulate -it with a second connection, weaken the guard to observe-before-write, or leave a -placeholder for WP12. K does not extend that phase-entry dependency: the current tree +Phase-entry gate: WP8b's executable `withExpectedConfigGenerationSync` owner seam is +now present at `src/config.ts:1876-1900`, with its shared callable/result types at +`src/codex/convergence-types.ts:262-272`, as required by +`005_contract.md:617-647`. Recheck that seam before WP9 implementation; if it is absent +or no longer validates through the already-held `configMutationDatabase`, stop and +report the WP8b scope dependency. Do not emulate it with a second connection, weaken +the guard to observe-before-write, or leave a placeholder for WP12. K does not extend +that phase-entry dependency: the current tree has only the native final-path resolver at `src/codex/user-identity.ts:164-186` and no `catalog-write-serialization.ts`; WP9 creates both K's module and catalog resolver because WP9 is the first phase that must serialize catalog/backup/cache publication. @@ -245,9 +275,11 @@ including a warm process-cache hit. `runtime.ts` never imports the catalog evide owner. This path does not test whether the command is executable, discover PATH alternatives, call `probeVersion`, persist selection, or execute the command. The observation can only identify a matching already-populated in-memory bundled-catalog cache; it is not -authority to refill it. The runtime and bundled owners return the immutable value with -its current process-lifetime epoch/value identity; the candidate records `unused` when -an owner does not influence preparation. +authority to refill it. The runtime and bundled owners return a detached recursively +frozen clone or non-aliased immutable view with the current process-lifetime +epoch/value identity; recursively readonly public types prohibit nested object/array +mutation as well. The private memo snapshot is never returned. The candidate records +`unused` when an owner does not influence preparation. `resolveCatalogSourceForGather(evidenceSession)` then tries that immutable cache value followed by active-catalog/backup/models-cache buffers read through the evidence owner. Its closed result is usable prepared source or @@ -274,7 +306,7 @@ with every conditional role key present as an empty list and the required filesystem-evidence session then records every consulted filesystem source under the contract's closed role union: bundled template, active merge, hashed/legacy fallback, models-cache fallback, runtime selection, or provider-auth selection -(`005_contract.md:788-844,895-978`). Callers cannot append, omit, remove, or rebuild +(`005_contract.md:830-930,967-1069`). Callers cannot append, omit, remove, or rebuild those observations. The required ABSENT state closes a target-selection hole that a present-file digest @@ -292,8 +324,9 @@ missing-leaf path, stable canonical-parent identity, and `fileIdentity:null`. Alternatives consulted and found absent are still evidence because their absence caused fallback. Process-local caches and network responses are not fabricated as filesystem observations; used runtime/bundled values instead contribute sealed -`CatalogProcessLocalEvidence`. The candidate receives immutable source and process -evidence. Missing `homeSelection`, a required role or conditional key, a required +`CatalogProcessLocalEvidence`. The candidate receives deeply frozen source/process +evidence and detached snapshots with no mutable alias back to either memo owner. +Missing `homeSelection`, a required role or conditional key, a required runtime-selection PRESENT/ABSENT observation, or a used epoch/value identity is structurally invalid and cannot reach commit. @@ -308,7 +341,7 @@ Both paths write zero bytes. This detects the audited same-inode truncate/rewrite even when config generation and target identity are unchanged. It catches single-direction drift only: content/state A→B→A returning identical evidence before comparison, parent A→B→A between checks, and a write after the final -comparison remain outside C17 (`005_contract.md:813-864`). +comparison remain outside C17 (`005_contract.md:802-878`). ### A3 — preserve bundled-first template precedence @@ -332,7 +365,8 @@ outside the 16 management paths until their owning phase migrates them. The candidate remains opaque, one-shot, and catalog-private. Its `WeakMap` state contains prepared bytes, result/notices, target identities, the admitted config generation, home selection, sealed candidate-bound `CatalogSourceEvidence`, and -sealed `CatalogProcessLocalEvidence`. Commit marks it +sealed `CatalogProcessLocalEvidence`. Memo-derived graphs are recursively frozen +detached snapshots and cannot alias the owners' private caches. Commit marks it consumed before validation and before the first write; a second call returns `candidate-consumed` and writes nothing. No route can inspect, serialize, reconstruct, or replay it. @@ -364,12 +398,23 @@ Preparation returns exact catalog/cache bytes and optional create-once backup by filesystem dependencies. It accepts no config, provider client, parser, subprocess, OAuth resolver, Promise, or callback that can return a Promise. -The outer catalog orchestration acquires K after gather and before any config -transaction. Automatic catalog convergence retries fail-fast K acquisition only within -`deadlineMs`; each attempt and the complete owner-held callback remain synchronous. -The callback receives K's opaque permit, and low-level writer calls without that exact -permit do not typecheck. Retained roots apply the same gather-outside-K rule without -changing their public signatures. Commit performs, in order: +The outer catalog orchestration acquires K after evidence-bound gather and before any +config transaction. Automatic catalog convergence retries fail-fast K acquisition only +within `deadlineMs`; each attempt and the complete owner-held callback remain +synchronous. Every successful acquisition mints a fresh private permit and transaction +identity, registers the exact permit object as active for canonical `CODEX_HOME`, and +passes it only to that callback. Compile-time permit requirements remain a reachability +guard, not proof of lock ownership: every low-level mutator must call K's owning +module's runtime assertion with the permit and canonical owning home before its first +filesystem mutation. The assertion rejects an unregistered, inactive, revoked, +previous-transaction, forged, or wrong-home object. K's module-private registry is not +exported or inspected by the writer. In `finally`, K revokes the permit before it +commits/rolls back and releases the SQLite transaction, including callback-throw paths. +One live permit may authorize the fixed sequence inside its own callback and nothing +afterward. Catalog convergence and retained `/api/sync` use evidence-bound +precomputation; startup cache invalidation, CLI `sync-cache`, and native restore instead +acquire K before authoritative read and recompute under K. Public signatures do not +change. Catalog convergence commit performs, in order: 1. acquire K after N when N exists; WP9 catalog-only has no N and therefore starts at K. Once K is held, mark the candidate consumed and call @@ -388,7 +433,7 @@ The owner-side guard is not a read-before-write check. Its implementation valida the expected generation using the `configMutationDatabase` handle whose SQLite transaction is already held, invokes the complete synchronous catalog callback on a match, and releases only after the callback returns -(`005_contract.md:631-647,676-707`). A cooperating config writer therefore cannot +(`005_contract.md:631-647,676-743`). A cooperating config writer therefore cannot commit N+1 between validation and catalog publication. Conflict never invokes the callback; lock/database unavailability projects through the total adapter. @@ -421,7 +466,7 @@ regular, non-routed valid catalog backup is preserved and the receipt becomes is `refused`. The loser never unlinks, truncates, or overwrites the winner. This exception applies only to a backup create-once target, never to a backup selected as a gather source; selected source observations remain strict -(`005_contract.md:867-887`). +(`005_contract.md:945-965`). ## C. Catalog-only convergence @@ -445,8 +490,9 @@ awaits the write-free gather, acquires K, enters `withExpectedConfigGenerationSync`, executes the synchronous catalog callback before C and K release, and projects the result. The lower-level orchestration lives in new `convergence.ts`, so only that module reaches -`internal/catalog-writer.ts`; the retained management module remains the factory -boundary until WP12 consolidates the full funnel. +`internal/catalog-writer.ts` from the 16 management-mutation paths; the four explicit +transitional chains remain the only WP9 exceptions. The retained management module +remains the factory boundary until WP12 consolidates the full funnel. ### C2 — permanent K plus owner-held generation and complete candidate evidence @@ -458,7 +504,12 @@ has no pair fields (`src/codex/convergence-types.ts:207-224`). Catalog staleness is guarded by: - permanent effective-user/canonical-`CODEX_HOME` serialization K, held across every - catalog/backup/cache replacement by convergence and all four transitional roots; + catalog/backup/cache authoritative transaction by convergence and all four + transitional roots, using either under-K recomputation or complete post-acquisition + evidence revalidation; +- a fresh acquisition-bound permit whose module-private registry liveness, + transaction identity, and owning-home binding every low-level mutator asserts at + runtime before its first filesystem mutation, then revokes before K release; - the observe-only config generation captured before gather and validated by `withExpectedConfigGenerationSync` on its already-held transaction through the complete synchronous commit; @@ -467,7 +518,8 @@ Catalog staleness is guarded by: - candidate-bound closed PRESENT/ABSENT source observations, including required `config.toml` target selection and mandatory runtime-selection evidence whenever runtime identity influenced the candidate; -- used runtime/bundled process-cache epochs and immutable value identities; +- used runtime/bundled process-cache epochs and recursively frozen, non-aliased value + identities whose private owner snapshots are never returned directly; - target parent/file identity plus the narrow create-once backup exception. The commit must never import or invoke routing writers. A test fails if catalog-only @@ -539,10 +591,10 @@ The symbol-graph test permits these exact legacy roots until WP12: | Legacy root | Current path | WP12 removal | |---|---|---| -| management `POST /api/sync` | `src/server/management/config-routes.ts:261-268` → `src/codex/sync.ts:83-90` → `src/codex/refresh.ts:40-52`; after gather, `src/codex/catalog/sync.ts:568` publishes under K and `src/codex/catalog/sync.ts:600-616` is called under K for cache publication | rewire to full convergence and `toSyncResponse`; K remains | -| server startup cache invalidation | `src/server/index.ts:403` → K → `invalidateCodexModelsCache` | route startup through full convergence/observer; K remains | -| `ocx sync-cache` | `src/cli/index.ts:849-855` → K → `invalidateCodexModelsCache` | route CLI command through full convergence; K remains | -| native restore | `src/codex/inject.ts:764-774` → K → `restoreCodexCatalog` → `src/codex/catalog/sync.ts:572-597` | move restore writes behind full convergence/provenance; K remains | +| management `POST /api/sync` | `src/server/management/config-routes.ts:261-268` → `src/codex/sync.ts:83-90` → `src/codex/refresh.ts:40-52` → evidence-bound pre-K provider gather (`src/codex/catalog/sync.ts:513-526`) → K → complete evidence revalidation → catalog/cache writers | rewire to full convergence and `toSyncResponse`; K remains | +| server startup cache invalidation | `src/server/index.ts:403` → K → authoritative catalog/cache read and derivation in `src/codex/catalog/sync.ts:603-612` → models-cache writer | route startup through full convergence/observer; K remains | +| `ocx sync-cache` | `src/cli/index.ts:849-855` → K → authoritative catalog/cache read and derivation in `src/codex/catalog/sync.ts:603-612` → models-cache writer | route CLI command through full convergence; K remains | +| native restore | `src/codex/inject.ts:764-774` → K → authoritative backup/catalog read and derivation in `src/codex/catalog/sync.ts:573-595` → catalog writer | move restore writes behind full convergence/provenance; K remains | The allowlist is exact by root module and writer symbol, not a directory wildcard. WP12 owns deleting every row. No new legacy root may be added in WP9. @@ -554,9 +606,11 @@ the writer symbol identity. An unresolved module, unresolved symbol, computed dynamic import, or non-literal import that could hide a writer fails the test rather than being skipped. The test publishes the WP9 legacy allowlist as data and proves all 16 management roots terminate at `convergence.ts` before a catalog writer. -It also proves every catalog/backup/cache mutator requires K's opaque permit and -rejects `C -> K`, `K -> N`, or held `N -> H` acquisition edges through aliases, -wrappers, or re-exports. +It also proves every catalog/backup/cache mutator requires K's permit and reaches K's +runtime assertion before any filesystem mutator. Separate runtime tests prove that +assertion rejects leaked/reused/forged/wrong-home permits; the graph alone is not lock- +liveness proof. Lock-order fixtures reject `C -> K`, `K -> N`, or held `N -> H` +acquisition edges through aliases, wrappers, or re-exports. ## Tests @@ -632,20 +686,29 @@ Freeze the current target-selection semantics with three real-file fixtures Warm runtime R1 and bundled template B1, gather a candidate that consumes both, then pause provider gathering and replace/invalidate each memo. Commit must revalidate the monotonic epoch and immutable value identity under K -> C and return `stale` with zero -writes, including invalidate-and-repopulate with byte-identical data. Separately -gather from warm R1 while `codex-runtime.json` is observed ABSENT, create persisted R2 -without advancing config generation, and require `stale` with zero writes; repeat -PRESENT replacement and removal. The named broken mutations are **compare cache bytes -without the epoch** and **record runtime-selection only on cold loads**; the first -accepts byte-identical replacement and the second accepts ABSENT -> PRESENT on a warm -hit. +writes, including invalidate-and-repopulate with byte-identical data. Obtain runtime +and bundled-cache values through their real public read APIs, let gather seal them, +then attempt nested object and array mutation through the returned graphs. The private +owner snapshot and candidate evidence remain byte-identical because the detached clone +or immutable view is recursively frozen; a top-level-only freeze is insufficient. If +the fixture instead uses the supported explicit owner mutation seam, that operation +must advance the epoch and commit must return `stale` before any write. Rewrite the +existing `tests/codex-runtime.test.ts:428-431` fixture that mutates the returned cached +alias so it uses that intentional owner seam. Separately gather from warm R1 while +`codex-runtime.json` is observed ABSENT, create persisted R2 without advancing config +generation, and require `stale` with zero writes; repeat PRESENT replacement and +removal. The named broken mutations are **compare cache bytes without the epoch**, +**return the private cache object or only shallow-freeze it**, and **record runtime- +selection only on cold loads**; they respectively accept byte-identical replacement, +allow nested mutation without assignment/epoch movement so stale commit succeeds, or +accept ABSENT -> PRESENT on a warm hit. Prove the cooperating-writer guarantee with two real processes and the real config mutation API. Process A enters `withExpectedConfigGenerationSync({value:N}, callback)`; callback entry proves validation matched and pauses while the config transaction is still held. Process B then attempts a real persisted config mutation. Because the existing lock is -fail-fast (`src/config.ts:1778-1815`), B's first attempt must report lock/busy and must +fail-fast (`src/config.ts:1780-1817`), B's first attempt must report lock/busy and must not commit N+1 while A is paused. A's synchronous catalog bytes land before callback return; after A releases, B retries through the real mutation API and commits N+1. Conflict never invokes the callback. Instrument SQLite connection creation and @@ -663,20 +726,44 @@ no-write/failure path or retry and publish Y; the forbidden trace is Y then X wi reporting `committed`. Reverse acquisition order: let B publish Y while holding K, then require A to revalidate after acquiring K and return `stale` with zero writes. Run the same K-exclusion shape for startup cache invalidation, CLI `sync-cache`, and -native restore. The named broken mutation is **omit K, release K before replacement, -or let one transitional writer call a mutator without the permit**; it permits Y then -X or a concurrent cache/restore overwrite. +native restore. + +Add the round-4 direction separately. Retained `/api/sync` process A reads X and starts +its slow provider gather before owning K. Convergence process B then acquires K and +publishes Y; only afterward may A resume and acquire K second. A must revalidate all +pre-K source/process evidence and discard/regather or follow its existing no-write/ +failure result path, never replace Y with X-derived bytes. Repeat with another retained +`/api/sync` process B as the K-first publisher so retained-vs-retained is tested +independently of convergence. For startup invalidation, CLI `sync-cache`, and native +restore, instrument the authoritative read and prove it cannot start before K; pause +after that read and prove a convergence or retained writer cannot publish until the +complete read-transform-write releases K. The named broken mutation is **move only the +replacement under K while retaining the authoritative read before K**; it permits the +forbidden X-read -> Y-publish -> X-derived-overwrite trace even though every rename +holds K. + +Exercise the runtime permit boundary against real temporary targets. Leak a permit and +call a mutator after its callback; reuse that revoked permit during a later K +acquisition for the same home; forge an object through a cast plus prototype/symbol +copying; and pass a still-live home-A permit to a home-B mutator. Each attempt must be +refused before temp creation, hardening, unlink, link, rename, truncate, or replacement, +with byte-identical targets. A fresh permit may authorize all fixed writes inside its +own live callback. The named broken mutations are **assert only the compile-time permit +shape**, **omit revocation before K release**, or **omit the owning-home comparison**; +the leaked/reused, forged, or wrong-home attempt respectively reaches the mutation spy. Document but do not claim detection for content A→B→A returning exact A before the comparison, parent A→B→A entirely between checks, or a write after the comparison. Broken mutations that must turn T2 red, in addition to the named mutations above: omit the required ABSENT config observation, remove digest comparison while retaining -generation/file identity, release C before callback, acquire C before K, or call -`readConfigGenerationAtPath` from inside the guard. The absent->present target switch -commits obsolete bytes, the same-inode rewrite commits stale bytes, process B commits -N+1 while A is paused, the real `/api/sync` writer publishes outside K, inverse order -deadlocks/self-contends, or the guard opens the forbidden second handle. +generation/file identity, release C before callback, acquire C before K, call +`readConfigGenerationAtPath` from inside the guard, or remove the low-level mutator's +runtime permit assertion. The absent->present target switch commits obsolete bytes, +the same-inode rewrite commits stale bytes, process B commits N+1 while A is paused, +retained-gathered X overwrites K-published Y, a leaked/forged/wrong-home permit reaches +filesystem mutation, inverse order deadlocks/self-contends, or the guard opens the +forbidden second handle. ### T3 — exact four-step receipt and bytes @@ -746,16 +833,18 @@ consultations: every `readFileSync`, `Bun.file`, `existsSync` branch, target `lstat`/`stat`/`realpath`, or wrapper that reaches one must terminate at `catalog/filesystem-evidence.ts`. Unresolved/computed edges fail closed. At both the `wp9-transitional` and future `wp12-final` inventory versions, every -catalog/backup/cache mutation must require K's permit. Lock-order fixtures fail on -`C -> K`, `K -> N`, and a held `N -> H`; the accepted full order is N -> K -> C, -while WP9 catalog-only uses K -> C because it never acquires N. +catalog/backup/cache mutation must require K's permit and call K's runtime assertion +before its first filesystem mutation. Lock-order fixtures fail on `C -> K`, `K -> N`, +and a held `N -> H`; the accepted full order is N -> K -> C, while WP9 catalog-only +uses K -> C because it never acquires N. Broken mutations that must turn T5 red: add a static top-level management-convergence import, alias a catalog writer into a management route, replace a literal import with a computed dynamic import, or add an absence-only `existsSync`/target `realpath` outside the evidence owner. Also remove the permit parameter from one mutator, add a -fifth unpermitted root, or invert any lock edge. The sentinel, compile fixture, or -fail-closed graph must reject each. +fifth unpermitted root, bypass the permit assertion before one filesystem mutator, or +invert any lock edge. The sentinel, compile fixture, or fail-closed graph must reject +each; T2, not the graph, rejects a permit whose runtime lifetime/home is invalid. ### T6 — precedence and native-pair exclusion @@ -783,7 +872,7 @@ Static/focused gates for the WP9 commit: ```bash bun test tests/codex-refresh.test.ts tests/codex-convergence-contract.test.ts -bun test tests/codex-config-generation.test.ts tests/codex-sync-api.test.ts tests/codex-models-cache-invalidate.test.ts +bun test tests/codex-config-generation.test.ts tests/codex-sync-api.test.ts tests/codex-models-cache-invalidate.test.ts tests/codex-runtime.test.ts bun test tests/model-visibility-management-api.test.ts tests/management-provider-validation.test.ts tests/combo-management-api.test.ts tests/codex-v2-gate.test.ts bun run typecheck bun run test @@ -801,8 +890,8 @@ processes only. No verification invokes `ocx start`, `stop`, `sync`, `restore`, | Criterion | Proof | Concrete broken mutation that makes it red | |---|---|---| | **C1** — gather is filesystem-write-free across user homes and scratch, performs no executable probe/subprocess, and commit is synchronous, fixed, K -> C ordered, one-shot, and receipt-exact | T1 + T3 + T5 | call cold `resolveCodexRuntime`/`loadBundledCodexCatalog`, add an `await` beneath commit, acquire C before K, reorder replacements, pre-set a receipt bit, or replay a consumed candidate | -| **C2/C17** — permanent K excludes every first-party catalog writer; owner-held config generation, required home/runtime evidence, process-local epochs, every closed PRESENT/ABSENT source observation, and target identity reject stale work before write; create-once backups publish atomically without clobber | T2 + T3 | omit K from real `/api/sync`, omit ABSENT `config.toml`/`codex-runtime.json`, skip CODEX_HOME re-resolution, compare cache bytes without epoch, release C before callback, remove same-inode digest comparison, open a second SQLite observer, or replace exclusive publication with overwriting rename | +| **C2/C17** — permanent K makes every first-party authoritative read-transform-write fresh by under-K recomputation or complete post-acquisition evidence revalidation; retained-gathered-first/K-second races are covered against convergence and another retained writer; owner-held config generation, required home/runtime evidence, deeply frozen non-aliased memo snapshots plus epochs, every closed PRESENT/ABSENT source observation, and target identity reject stale work before write; create-once backups publish atomically without clobber | T2 + T3 | leave retained `/api/sync`'s `onDiskCatalog` read before K but guard only replacement, start startup/CLI/restore's authoritative read before K, return or shallow-freeze a private cache snapshot so nested mutation bypasses epoch movement, omit ABSENT `config.toml`/`codex-runtime.json`, skip CODEX_HOME re-resolution, release C before callback, remove same-inode digest comparison, open a second SQLite observer, or replace exclusive publication with overwriting rename | | **Catalog/native boundary** — catalog-only never reads/advances the native pair or writes routing/history artifacts | T6 | call `expectation()`/`beginTransition`, add pair fields to `catalog-only`, or invoke config/profile/journal/history writer | | **Best-effort compatibility** — all 16 primary writes retain 2xx/201 and original follow-up order for every catalog failure | T4 | let lazy import/factory/admission throw, scope “zero writes” to the whole route, or return before Claude/Desktop follow-up | -| **C14, WP9-bounded** — the 16 management roots reach catalog writers only through convergence; exactly four documented transitional roots remain until WP12 and every one already requires permanent K | T2 barrier + T5 symbol graph | add a fifth root, omit K/permit from one retained chain, hide one through alias/re-export/computed import, or accidentally require WP12 to have already rewired `/api/sync`/startup/CLI/restore | +| **C14, WP9-bounded** — the 16 management roots reach catalog writers only through convergence; exactly four documented transitional roots remain until WP12; every low-level mutation requires permanent K's fresh acquisition-bound permit and runtime liveness/transaction/home assertion | T2 barriers + permit negatives + T5 symbol graph | add a fifth root, omit K/permit/assertion from one retained chain, accept a leaked/reused/forged/wrong-home permit, hide a writer through alias/re-export/computed import, or accidentally require WP12 to have already rewired `/api/sync`/startup/CLI/restore | | **N2** — WP9 replaces the landed placeholder, consumes existing request/snapshot/projection seams, creates permanent K and its resolver plus the contract test, and typechecks without WP10-WP12 | focused tests + typecheck | move K into WP8b/WP11, redefine a WP8b type/helper, refer to a nonexistent later helper, leave a throwing placeholder, or claim the absent test file is merely extended | From 2db5ee947069aff4be83be8eb27e9b7daf3cee4f Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Tue, 4 Aug 2026 21:37:01 +0900 Subject: [PATCH 062/163] docs(substrate): two requests that share a fetch do not share its authority MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Round 5 left one blocker, and it was the fifth version of the same mistake. Catalog gather joins an in-flight provider fetch keyed by `gatherFlightKey(config)`, and that key's fingerprint covers endpoints, model lists, context windows, modalities and efforts — but not `authMode`, not the API key, not headers (provider-fetch.ts:98-120). Two requests that differ only in how they authenticate therefore produce the same key and share one promise (:670), even though `authMode` changes the request path (:410) and credentials change what is discovered (:428). So: request A is admitted at generation N with forward auth and starts the flight. A management mutation persists N+1 with an API key. Request B captures the correct N+1 config and honest evidence, joins A's flight because the key matches, and receives A's forward-auth result — an empty catalog — carrying no trace of the authority that produced it. B then validates its own generation and its own evidence truthfully and commits A's bytes. The reviewer reproduced it: `{"fetchCount":0,"a":[],"b":[]}`, the keyed caller never fetching at all. Every proposed test stayed green, because they interleave writers and filesystem state, not an authority change while a fetch is live. A gather now carries an immutable authority identity — the exact admitted config reference and generation, the auth snapshot, the native and source inputs, and the process-local evidence — and a flight result carries the identity that produced it, so a candidate whose authority does not equal its admission is rejected before it is built. Sharing survives for identical authority, because suppressing a thundering herd of provider fetches was a real reason and only the cross-authority join was wrong. The identity is an HMAC keyed by an unexported per-process random with domain separation, never the secret and never a stable unkeyed digest: this key is data that can end up in a map, a log or a response, and `privacy:scan` exists because credential material has reached those places before. --- .../005_contract.md | 186 +++++++++++-- .../010_catalog_seam.md | 258 +++++++++++++----- 2 files changed, 361 insertions(+), 83 deletions(-) diff --git a/devlog/_plan/260804_codex_write_substrate/005_contract.md b/devlog/_plan/260804_codex_write_substrate/005_contract.md index bc7601a67..a967c9ead 100644 --- a/devlog/_plan/260804_codex_write_substrate/005_contract.md +++ b/devlog/_plan/260804_codex_write_substrate/005_contract.md @@ -321,11 +321,11 @@ ambient declarations. The coordinator is a **sibling**, not an extension of `config-mutation.sqlite`. The existing database path is derived from `getConfigDir()` -(`src/config.ts:1731-1762`), whose resolver reads `OPENCODEX_HOME` -(`src/config.ts:530-534,1254-1256`); extending it would repeat the split-key +(`src/config.ts:1745-1776`), whose resolver reads `OPENCODEX_HOME` +(`src/config.ts:543-550,1268-1270`); extending it would repeat the split-key defect. The sibling uses the same Bun SQLite pattern — private file, `busy_timeout=0`, `BEGIN IMMEDIATE`, process-exit lock release -(`src/config.ts:1767-1818`) — but its final database path is keyed by effective +(`src/config.ts:1790-1839`) — but its final database path is keyed by effective user plus canonical `CODEX_HOME` (§7). WP11's native exclusion transaction and both transition-state callers open this same database. @@ -540,7 +540,7 @@ outcome is *visible* instead of swallowed by a bare `catch`. Round 1 #5/#6 and round 2 N3. Three separate defects lived here. -`mutatePersistedConfig` documents its own limit (`src/config.ts:1855-1857`): +`mutatePersistedConfig` documents its own limit (`src/config.ts:1950-1955`): > A writer that ignores the coordinator can still change bytes after the final > check because the filesystem has no portable conditional rename. @@ -888,14 +888,13 @@ private candidate and compared with the owner's current pair under K and C befor the first write. Any change, including invalidate-and-repopulate with byte-identical content, is `stale`. -Round 4 made “immutable” operational rather than aspirational. Today -`resolveCodexRuntime` returns `resolveCache.value` directly -(`src/codex/runtime.ts:397-400`), its nested interfaces are mutable -(`src/codex/runtime.ts:16`), and a test mutates that shared object in place -(`tests/codex-runtime.test.ts:428-431`). The bundled loader likewise returns -`bundledCatalogCache.value` (`src/codex/catalog/bundled.ts:179`). Such an alias changes -authority without owner assignment, invalidation, or epoch movement, so neither the -epoch nor object identity can detect it. +Round 4 made “immutable” operational rather than aspirational. The audited tree +returned `resolveCache.value` and `bundledCatalogCache.value` directly, so a caller +could change authority without owner assignment, invalidation, or epoch movement. +The current worktree now exposes recursively readonly runtime shapes and clones plus +deep-freezes cache publication/reads (`src/codex/runtime.ts:16-48,90-105,417-470,505-516`), +and the bundled owner does the same (`src/codex/catalog/bundled.ts:52-130,260-283`). +The contract requires that landed behavior; it must not regress to the audited alias. The runtime and bundled-cache owners must instead clone incoming values into private owner snapshots, recursively freeze every reachable object and array before @@ -942,6 +941,96 @@ atomicity; a non-cooperating writer can still change bytes after the final comparison. The outcome must preserve those C17 bounds rather than promote a digest into a guarantee the filesystem cannot provide. +### Catalog single-flight is bound to gather authority (seam audit round 5) + +Round 5 reproduced the same absence-as-equivalence defect before candidate +construction. `providerCatalogFingerprint` covers endpoint and catalog fields but +omits `authMode`, `apiKey`, and `headers` +(`src/codex/catalog/provider-fetch.ts:136-158`). `gatherFlightKey` hashes that partial +projection (`src/codex/catalog/provider-fetch.ts:161-180`), even though +`fetchProviderModels` branches on `authMode`, resolves credentials, and builds the +effective discovery request (`src/codex/catalog/provider-fetch.ts:474-500,548-568`). +The map lookup then lets the second caller join the first promise solely by that key +(`src/codex/catalog/provider-fetch.ts:792-820`). A generation-N forward-auth gather +can therefore supply empty bytes to a generation-N+1 key-auth admission; B's later +K -> C validation is honest but irrelevant because no evidence says A produced the +joined result. + +The current in-progress WP9 worktree prefixes the key with a plain SHA-256 of the +auth-store buffer (`src/codex/catalog/provider-fetch.ts:773-787`). That does not close +the finding: static key/forward mode and configured headers still collide, the result +still carries no authority, and a stable plain digest of credential-store bytes is the +privacy trap this rule forbids. + +This contract keeps single-flight sharing rather than prohibiting all cross-admission +sharing. The admission gate exists to suppress a thundering herd of provider model +requests, and simultaneous management mutations using the same resident config and +the same observed authority are legitimately equivalent. The narrower rule is that +**only complete gather-authority equality may share**. Different resident config +references do not share even when their JSON content happens to match. Different +config generations, auth snapshots, native-catalog/source inputs, or relevant +process-local observations do not share. Prohibiting every cross-admission join would +be safe but would discard that useful equivalence and multiply upstream requests. + +Before consulting the in-flight map, gather constructs and recursively freezes one +`CatalogGatherAuthorityIdentity`. Its components are: + +1. the opaque process-local WeakMap identity of the exact retained + `Readonly` reference, its admitted `ConfigGeneration`, and a snapshot + identity of the exact config graph at admission. Canonical encoding preserves + object-key presence, primitive type, array order, and sorted object keys; a value + that cannot be encoded exactly is refused. The snapshot identity detects an + illicit in-place mutation, while the reference identity preserves the settled + exact-resident-config contract; +2. an auth snapshot for every enabled provider: provider name, effective auth mode, + credential state, exact resolved API-key or observe-only OAuth access-token bytes + (or explicit absence), the exact `provider-auth-selection` observation that chose + an OAuth account/token, and the final discovery method, URL, and normalized header + set after transport defaults. Header names are lowercase and sorted; values remain + byte-exact inside the keyed input. Forward and local modes are explicit states, not + absence; +3. the exact native-slug/source input. When combo resolution needs native rows, the + filesystem-evidence owner returns a detached immutable ordered slug/capability + snapshot and records every consulted active-catalog/cache source, PRESENT or + ABSENT, under the closed `native-catalog-selection` role. Its identity also covers + the exact process-static registry, generated Jawcode metadata, and pinned upstream + snapshot revisions used to derive provider/native rows. When native input is not + consulted, an explicit `unused` value is part of the identity; +4. an identity of the complete source-evidence session sealed for flight launch, + including home/target selection and every auth/native observation captured so far; +5. relevant process-local input: the exact runtime and bundled memo evidence plus a + per-provider immutable model-cache/cooldown snapshot with monotonic owner epoch and + value identity. Provider cache/cooldown evidence binds flight admission and join + equivalence; it is not added to K -> C revalidation because the flight itself may + advance that cache while producing its immutable result. The already-settled + runtime/bundled evidence remains candidate-bound and is revalidated at commit. + +No credential is stored in that identity. At process start the gather-authority owner +mints an unexported random 256-bit HMAC key. Each `*Identity` above is a +domain-separated HMAC-SHA-256 over a length-prefixed canonical encoding of the exact +inputs just listed; `authorityId` is another domain-separated HMAC over the component +tuple. The key, canonical plaintext, API keys, OAuth tokens, and header values are +never exported, logged, serialized, placed in `CatalogDisposition`, or used as a +stable cross-process identifier. A plain SHA-256 of a credential, an API key copied +into `providerCatalogFingerprint`, or a stable unsalted digest is forbidden because +it turns the in-flight key into credential material or an offline guessing oracle. +The private candidate's existing exact-buffer source digest may still revalidate the +`provider-auth-selection` file under K -> C; it is HMACed as input to the flight +identity and never becomes the map key, result surface, log, or response itself. +The opaque HMAC values are process-local and may be discarded when the flight settles. + +The in-flight owner stores `{ authority, promise }`, not a bare promise. Its primary +bucket is `authorityId`, but joining additionally requires exact equality of the +deep-frozen component identities; a mismatch or collision starts a distinct admitted +flight (or returns typed busy when the admission gate is full). The flight receives +the captured config/auth/native/source/process snapshots as arguments and may not +re-resolve them after claiming its slot. `GatherFlightResult` carries the exact +authority identity that produced its models and omissions. Before candidate +construction, every caller compares that result identity with its own expected +identity. Inequality discards the result and returns retryable `stale` or regathers +within `deadlineMs`; it never builds or commits a candidate. This result check is +required defense in depth even though the map key should already prevent the join. + ### Create-once means no-clobber publication (seam audit round 2) Hashed and legacy catalog backups are immutable first-winner snapshots. The ordinary @@ -980,6 +1069,7 @@ export type CatalogConditionalSourceRole = | "hashed-backup-fallback" | "legacy-backup-fallback" | "models-cache-fallback" + | "native-catalog-selection" | "runtime-selection" | "provider-auth-selection"; @@ -1018,6 +1108,24 @@ export interface CatalogProcessLocalEvidence { readonly bundledCatalog: CatalogProcessLocalObservation; } +/** Non-secret-bearing identity of every authority input admitted to one gather flight. */ +export interface CatalogGatherAuthorityIdentity { + readonly version: 1; + /** Process-local keyed HMAC over every component below; never a raw content hash. */ + readonly authorityId: string; + readonly admittedConfig: Readonly<{ + /** Opaque WeakMap identity of the exact resident Readonly reference. */ + readonly referenceIdentity: string; + readonly generation: ConfigGeneration; + /** Keyed HMAC of the exact canonical config snapshot, including secret-bearing fields. */ + readonly snapshotIdentity: string; + }>; + readonly authSnapshotIdentity: string; + readonly nativeCatalogSourceIdentity: string; + readonly sourceEvidenceIdentity: string; + readonly processLocalEvidenceIdentity: string; +} + /** Exact gather-time evidence for one consulted filesystem source. */ export type CatalogSourceObservation = | { @@ -1059,6 +1167,8 @@ export interface CatalogSourceEvidence { export interface CatalogAdmissionSnapshot { config: Readonly; generation: ConfigGeneration; + /** Exact retained-reference/generation/snapshot identity used by gather authority. */ + readonly configIdentity: CatalogGatherAuthorityIdentity["admittedConfig"]; targets: Readonly<{ catalog: string; cache: string; @@ -1104,10 +1214,12 @@ for the logical `$CODEX_HOME/config.toml` path, PRESENT or ABSENT, and every conditional role key present as an empty list. Gather does not mutate that snapshot. It returns the prepared candidate with an immutable copy whose conditional lists contain every filesystem consultation in order, including absent alternatives that -caused a fallback, plus sealed `CatalogProcessLocalEvidence`. A missing home -selection, required role, conditional role key, required runtime-state observation, -or used-cache epoch/value identity is structurally invalid; commit accepts only the -private candidate-bound evidence. +caused a fallback and `native-catalog-selection` whenever native combo rows consult +active catalog/cache state, plus sealed `CatalogProcessLocalEvidence` and the complete +`CatalogGatherAuthorityIdentity` that produced the result. A missing home selection, +required role, conditional role key, required runtime-state observation, used-cache +epoch/value identity, config identity, or gather-authority component is structurally +invalid; commit accepts only the private candidate-bound evidence. All gather filesystem reads route through the one evidence-producing owner, `src/codex/catalog/filesystem-evidence.ts`. It owns an opaque gather-evidence @@ -1131,14 +1243,18 @@ different authority sources and must not be collapsed: admission. `CatalogAdmissionSnapshot.config` is that object; the factory closure is the sole runtime caller of snapshot capture, and route callers receive no config parameter with which to substitute another authority. Catalog admission - does not independently reconstruct full config from disk. This deliberate - capture is what prevents a route from substituting catalog authority. Admission - separately observes config generation, raw/default home selection plus canonical + does not independently reconstruct full config from disk. It assigns the exact + reference an opaque WeakMap identity, binds it to the observed generation, and + computes the process-keyed config snapshot identity before any flight lookup. + This deliberate capture is what prevents a route from substituting catalog + authority. Admission separately observes config generation, raw/default home selection plus canonical home/root identity, targets, and the required `$CODEX_HOME/config.toml` `catalog-target-selection` role before gather. If runtime identity later influences the candidate, gather records `codex-runtime.json` PRESENT or ABSENT and seals the runtime/bundled memo epoch and immutable value identity actually - consumed. + consumed. Before provider work, gather completes the auth, native-source, + source-session, and process-local components, then may join only a flight carrying + the equal complete authority identity. 2. **Catalog under-lock (WP9):** acquire K, then `withExpectedConfigGenerationSync(snapshot.generation, commit)` validates the @@ -1577,6 +1693,29 @@ fixture must go red when either API returns its private cache object directly or freezes only the top level; rewrite the existing runtime test that mutates the shared alias so it uses the intentional owner mutation seam. +Add the round-5 live-flight authority matrix. Pause request A after generation N +forward-auth authority has claimed its provider flight. Persist N+1 with key auth and +an API key, capture request B from the new resident config, and resume both. B must +start a distinct flight or reject the joined result as retryable stale; it must never +construct or commit a candidate carrying A's empty result. Force a bucket collision +and prove the result-carried identity check still rejects A. The named broken mutation +**restore the legacy `providerCatalogFingerprint`/`gatherFlightKey` and remove the +result-authority equality check** reproduces the empty B catalog. + +Repeat with unchanged config/generation while the exact observe-only OAuth-store +buffer changes active account/token state during A's live flight. B's +`provider-auth-selection` observation and auth snapshot identity must differ, so B +runs separately or rejects A's result. The named broken mutation **omit the OAuth +source observation and effective token from `authSnapshotIdentity`** makes B join and +accept A. Repeat with unchanged config/auth while the active native catalog/cache +observation changes the ordered native slug/capability input used by combo assembly. +B must not accept A's native rows. The named broken mutation **omit +`native-catalog-selection` and `nativeCatalogSourceIdentity` from flight identity and +result validation** makes the test red. Each fixture asserts the second caller's +candidate authority equals its own admission, while instrumented log/response/ +serialization sinks receive neither the private identity nor the API key, OAuth +token, configured secret header, or their plain SHA-256 values. + Race two create-once backup publishers after both observed ABSENT. Exactly one no-clobber publication wins; the loser receives `EEXIST`, validates and preserves the winner, and neither ordinary rename nor `atomicWriteFile` is called. Repeat with @@ -1619,8 +1758,11 @@ writes to it. gathered catalog source whose state, identity, or bytes drift once is detected by its role-bearing observation even when the write target and both generations are unchanged. This includes required `config.toml` ABSENT -> PRESENT target-selection - drift and a single-direction raw CODEX_HOME-selector/canonical-root retarget before - writing. A runtime-influenced candidate always carries PRESENT-or-ABSENT + drift, changed OAuth-store authority, changed native-catalog selection, and a + single-direction raw CODEX_HOME-selector/canonical-root retarget before writing. + A shared provider flight is keyed by the complete non-secret-bearing + `CatalogGatherAuthorityIdentity`, returns that producing identity, and cannot build + a candidate when it differs from the caller's admission. A runtime-influenced candidate always carries PRESENT-or-ABSENT `codex-runtime.json` evidence, and any used runtime/bundled process memo must retain its exact monotonic epoch and deeply immutable, non-aliased value identity through the commit check. diff --git a/devlog/_plan/260804_codex_write_substrate/010_catalog_seam.md b/devlog/_plan/260804_codex_write_substrate/010_catalog_seam.md index bc32cc44a..9e18d5bdf 100644 --- a/devlog/_plan/260804_codex_write_substrate/010_catalog_seam.md +++ b/devlog/_plan/260804_codex_write_substrate/010_catalog_seam.md @@ -24,11 +24,13 @@ native generation requires matching history schedule fields, every transition that was not published (`src/codex/transition-state.ts:74-83,314-344,420-428`). All current-code citations and diff context below were rechecked on 2026-08-04 -at `417734502143c14e76ca0153b7e86e29ccdf86a4`. The contract citations refer to -the authoritative worktree amendment: permanent K, owner-held config generation, +against the live worktree rooted at +`364496fc968fd12734a2e6fe2bec9e219d9d1c90`, including the concurrent in-progress +WP9 source edits. The contract citations refer to the authoritative worktree amendment: +permanent K, owner-held config generation, home/source/process-local evidence, and no-clobber publication are at -`005_contract.md:609-762,830-965,967-1124`; K's owner/path and the four -transitional chains are fixed at `005_contract.md:1256-1454`. +`005_contract.md:609-763,830-1280`; K's owner/path and the four +transitional chains are fixed at `005_contract.md:1372-1570`. ## IN / OUT @@ -38,27 +40,31 @@ IN — observe-only admission and gather: generation observation that never creates, initializes, chmods, or registers `config-mutation.sqlite`. The existing `readConfigGeneration` is not that API: it resolves/records the path and opens SQLite with `create:true` - (`src/config.ts:1743-1774,1847-1853`, `src/codex/generation.ts:93-103`). WP9 + (`src/config.ts:1745-1776,1799-1807,1849-1854`, + `src/codex/generation.ts:97-103`). WP9 consumes the contract-owned `withExpectedConfigGenerationSync`; it does not wrap this observer in the lock or redefine the generation contract. - `src/codex/catalog-admission.ts` (MODIFY) — keep the landed request constructor and snapshot capture; switch snapshot capture to the observe-only generation read, capture the required raw/default `CODEX_HOME` selector, canonical home and root identity plus the PRESENT-or-ABSENT `catalog-target-selection` observation, - and carry the contract-owned `sourceEvidence`. Do not redefine + assign an opaque identity to the exact retained config reference plus its keyed + snapshot, and carry the contract-owned `sourceEvidence`. Do not redefine `createCatalogConvergeRequest` or `captureCatalogAdmissionSnapshot`, which - already exist at lines 38-52 and 148-180. + already exist at lines 38-52 and 148-185. - `src/codex/convergence-types.ts` (MODIFY) — synchronize the already contract-owned closed `CatalogSourceRole`, `CatalogHomeSelectionObservation`, `CatalogSourceObservation`, `CatalogSourceEvidence`, - `CatalogProcessLocalEvidence`, and `CatalogAdmissionSnapshot.sourceEvidence` - additions from `005_contract.md:967-1097`; no WP9-private duplicate type is allowed. + `CatalogProcessLocalEvidence`, `CatalogGatherAuthorityIdentity`, and + `CatalogAdmissionSnapshot` additions from `005_contract.md`; no WP9-private + duplicate type is allowed. - `src/codex/catalog/filesystem-evidence.ts` (NEW) — sole owner of gather source reads and target probes. Its opaque session records PRESENT and ABSENT observations before returning, captures and seals catalog-home selection before accepting any derived path, seals the complete closed role map into the candidate, - and is the only gather path permitted to call filesystem consultation primitives - (`005_contract.md:1112-1124`). + records every native catalog/cache consultation under + `native-catalog-selection`, and is the only gather path permitted to call + filesystem consultation primitives. - `src/codex/runtime.ts`, `src/codex/catalog/bundled.ts` (MODIFY) — catalog gather uses a gather-specific observe-only pair: `peekCodexRuntimeForCatalogGather(evidenceSession)` and @@ -81,16 +87,16 @@ IN — observe-only admission and gather: candidate, the evidence session records `codex-runtime.json` as PRESENT or ABSENT even on a warm-cache hit. This is required because `persistCodexRuntime` writes the file and clears the memo - without advancing config generation (`src/codex/runtime.ts:213-229`), while bundled - cache hits and replacements are process-local; the bundled loader currently returns - its private cached object directly at `src/codex/catalog/bundled.ts:186` - (`src/codex/catalog/bundled.ts:179-209`). `resolveCodexRuntime` likewise returns its - private cached object at `src/codex/runtime.ts:397-400`, and its result interfaces - are mutable at `src/codex/runtime.ts:16-40`. + without advancing config generation (`src/codex/runtime.ts:268-284`), while bundled + cache hits and replacements are process-local. The current worktree's runtime owner + already publishes/returns deeply frozen detached values + (`src/codex/runtime.ts:16-48,90-105,417-470,505-516`), and the bundled owner does + likewise (`src/codex/catalog/bundled.ts:52-130,260-283`); WP9 preserves those + round-4-closed semantics rather than reintroducing the audited private-cache alias. The ordinary resolver reaches `probeVersion`, whose sandbox deliberately calls - `mkdtempSync` and `rmSync` (`src/codex/runtime.ts:231-279,327-340,397-405`), while + `mkdtempSync` and `rmSync` (`src/codex/runtime.ts:286-335,505-516`), while bundled loading both calls the persisting resolver and runs `codex debug models` - (`src/codex/catalog/bundled.ts:127-169,170-210`). Neither path is reachable from + (`src/codex/catalog/bundled.ts:212-258,271-294`). Neither path is reachable from gather. - `src/oauth/index.ts`, `src/oauth/store.ts`, `src/codex/catalog/provider-fetch.ts` (MODIFY) — add and consume an observe-only @@ -99,11 +105,17 @@ IN — observe-only admission and gather: normalization semantics rather than calling `peekAuthStore` or another hidden filesystem reader. `peekAuthStore` confirms the desired no-chmod/no-backup behavior but still owns its own `existsSync`/`readFileSync` consultation today - (`src/oauth/store.ts:145-157`). The gather path never refreshes, persists, acquires + (`src/oauth/store.ts:177-181`). The gather path never refreshes, persists, acquires an intent lock, creates/removes an intent file, hardens a path, or backs up malformed credentials. The current token resolver can enter refresh/persistence - (`src/oauth/index.ts:281-339,352-354`) and the current gather awaits it - (`src/codex/catalog/provider-fetch.ts:410-428`). + (`src/oauth/index.ts:327-400`) and the ordinary refresh-capable gather awaits it + (`src/codex/catalog/provider-fetch.ts:474-500`). + Replace the partial `providerCatalogFingerprint` single-flight identity with the + complete contract-owned gather-authority identity. Authority capture happens before + the map lookup; the flight consumes the captured config/auth/native/source/process + snapshots and returns the exact identity that produced its result. A joiner must + match that identity before candidate construction. The map still coalesces a herd + only when every authority component is equal. - `src/codex/refresh.ts`, `src/codex/catalog/sync.ts`, `src/codex/catalog/parsing.ts` (MODIFY) — prepare immutable catalog/cache/backup bytes and source evidence without writing. In particular, target selection no @@ -130,13 +142,13 @@ IN — fixed commit and convergence: - `src/codex/user-identity.ts` (MODIFY) — add `resolveCodexCatalogSerializationDatabasePath` beside the landed native coordinator resolver. Consumers use its final path verbatim; the K and N database paths must be - distinct (`005_contract.md:1256-1367`). + distinct (`005_contract.md:1372-1483`). - `src/codex/internal/catalog-writer.ts` (NEW/MOVE) — the contract-owned low-level owner for catalog, hashed/legacy backups, and models cache. Every mutator requires the permit plus canonical owning `CODEX_HOME` and calls K's runtime assertion before temp creation, hardening, unlink, link, rename, truncate, replacement, or any other filesystem mutation. It never reads or mutates K's private registry. Do not create - the obsolete `internal/catalog-commit.ts` name (`005_contract.md:1369-1403`). + the obsolete `internal/catalog-commit.ts` name (`005_contract.md:1485-1518`). - `src/codex/convergence.ts` (NEW) — primary catalog gather/commit orchestration for the 16 management mutations. The symbol graph additionally permits only the exact four WP9 transitional writer chains below; WP12 removes those exceptions. @@ -174,12 +186,12 @@ IN — management callers and tests: return values, and compatibility behavior stay unchanged; explicit imports needed after the facade stops re-exporting writers are mechanical. This is freshness-safe serialization of the retained roots, not WP12's convergence rewire - (`005_contract.md:1405-1454`). + (`005_contract.md:1519-1570`). - `tests/codex-refresh.test.ts` and the existing management route suites (MODIFY). -- `tests/codex-runtime.test.ts` (MODIFY) — rewrite the fixture at lines 428-431 that - currently mutates `resolveCodexRuntime()`'s shared cached result in place. It must use - the intentional owner mutation/invalidation seam once returned snapshots are deeply - immutable; this is expected fallout, not an implementation regression. +- `tests/codex-runtime.test.ts` (MODIFY) — retain the current regression proving nested + mutation through a returned runtime cache graph cannot alter owner state + (`tests/codex-runtime.test.ts:138-148`) and use only the intentional owner + mutation/invalidation seam when a test needs to move the cache epoch. - `tests/codex-convergence-contract.test.ts` (CREATE). It does not exist in the WP8b tree; WP9 creates it rather than “extending” an imaginary file. @@ -204,16 +216,16 @@ entry point; that later move is a module consolidation, not completion of an unfinished WP9 branch. Phase-entry gate: WP8b's executable `withExpectedConfigGenerationSync` owner seam is -now present at `src/config.ts:1876-1900`, with its shared callable/result types at +now present at `src/config.ts:1882-1906`, with its shared callable/result types at `src/codex/convergence-types.ts:262-272`, as required by `005_contract.md:617-647`. Recheck that seam before WP9 implementation; if it is absent or no longer validates through the already-held `configMutationDatabase`, stop and report the WP8b scope dependency. Do not emulate it with a second connection, weaken the guard to observe-before-write, or leave a placeholder for WP12. K does not extend -that phase-entry dependency: the current tree -has only the native final-path resolver at `src/codex/user-identity.ts:164-186` and no -`catalog-write-serialization.ts`; WP9 creates both K's module and catalog resolver -because WP9 is the first phase that must serialize catalog/backup/cache publication. +that phase-entry dependency: the current worktree's in-progress WP9 implementation +already has distinct native and catalog final-path resolvers +(`src/codex/user-identity.ts:165-220`) and a concrete K owner; this plan remains their +contract because WP9 is the first phase that must serialize catalog/backup/cache publication. Moving K to WP8b would widen the already-landed admission seam without an earlier consumer, while deferring it to WP11/WP12 would leave WP9's retained writers unsafe. @@ -223,11 +235,11 @@ consumer, while deferring it to WP11/WP12 would leave WP9's retained writers uns The guarantee is **filesystem-write-free**, not globally side-effect-free. Gather may update bounded discovery-status, provider model cache, and in-flight admission maps -(`src/codex/catalog/provider-fetch.ts:455-465,495-507,608-615,675-685`); they are +(`src/codex/catalog/provider-fetch.ts:525-537,572-588,686-692,798-820`); they are permitted because they do not mutate user files and are reset between isolated tests. The runtime and bundled memos are observe-only from gather, but their owners can -replace them concurrently (`src/codex/runtime.ts:362-410`, -`src/codex/catalog/bundled.ts:179-209`), which is why the candidate seals epochs and +replace them concurrently (`src/codex/runtime.ts:417-470`, +`src/codex/catalog/bundled.ts:52-130`), which is why the candidate seals epochs and identities. No credential, raw provider error, source path, or digest may escape through those caches into `CatalogDisposition`. @@ -243,7 +255,7 @@ This bound deliberately catches the writes hidden by the old plan: - runtime selection must not persist `codex-runtime.json`; - ordinary auth reads must not call `loadAuthStoreInternal`, whose read path - hardens files and backs up invalid JSON (`src/oauth/store.ts:128-137`); + hardens files and backs up invalid JSON (`src/oauth/store.ts:135-145`); - expired OAuth is a sanitized provider-auth degradation/failure for this gather, not permission to refresh and persist; - admission must not invoke the create-on-read generation path, which can also @@ -251,12 +263,12 @@ This bound deliberately catches the writes hidden by the old plan: - `resolveCodexRuntime()` is forbidden even though it does not itself persist: a cold resolution reaches `probeVersion()`, and that probe intentionally creates and deletes a temporary `CODEX_HOME` because real Codex writes even for - `--version` (`src/codex/runtime.ts:231-279,327-340,397-405`). A final-state + `--version` (`src/codex/runtime.ts:286-335,505-516`). A final-state manifest cannot see that created-then-deleted directory. - `loadBundledCodexCatalog()` and `runCodexDebugModels()` are forbidden beneath gather. The current bundled loader resolves/persists a runtime and executes `codex debug models --bundled` without an isolated gather environment - (`src/codex/catalog/bundled.ts:127-169,170-210`). + (`src/codex/catalog/bundled.ts:212-294`). The observe-only generation API opens an existing database with `readonly:true`, performs only schema/version/select checks, and closes it. Missing DB/table/row, @@ -295,6 +307,89 @@ usable. If the auth-store buffer influences a live provider result, its PRESENT ABSENT `provider-auth-selection` observation joins the candidate's private source evidence; token bytes never do. +Round 5 closes a separate live-flight authority hole. The current +`providerCatalogFingerprint` includes endpoint/catalog fields but excludes +`authMode`, `apiKey`, and `headers` +(`src/codex/catalog/provider-fetch.ts:136-158`); `gatherFlightKey` hashes that partial +projection (`src/codex/catalog/provider-fetch.ts:161-180`) and the map joins solely by +the resulting key (`src/codex/catalog/provider-fetch.ts:792-820`). Those omitted +fields are behavioral: forward mode exits with no models, credential resolution is +awaited, and the effective request headers are then built +(`src/codex/catalog/provider-fetch.ts:474-500,548-568`). The observed failure was A at +generation N in forward mode, B admitted at N+1 in key mode, and B receiving A's +empty promise result with no A authority in its candidate. + +The current in-progress worktree adds a plain SHA-256 of the observed auth-store +buffer as a key prefix (`src/codex/catalog/provider-fetch.ts:773-787`). It still omits +static key/forward mode and configured headers, and `GatherFlightResult` still carries +no authority (`src/codex/catalog/provider-fetch.ts:87-91`). It therefore neither +closes the failing sequence nor meets the privacy rule below. + +WP9 keeps single-flight because equivalent concurrent management mutations should +not multiply upstream `/models` requests. It rejects the alternative of prohibiting +all cross-admission sharing: that is safe but defeats the thundering-herd control the +flight map exists to provide. Sharing is narrowed to callers whose complete immutable +`CatalogGatherAuthorityIdentity` is equal. In practice that means the same exact +resident `Readonly` reference, the same admitted generation and exact +config snapshot, the same auth snapshot, the same native-catalog/source snapshot, +the same sealed source-session identity, and the same relevant process-local inputs. +A different config object does not share merely because selected catalog fields are +equal. + +`catalog-admission.ts` assigns each retained config object an opaque process-local +WeakMap identity and combines it with generation plus a keyed identity of the exact +canonical config graph. The gather-authority owner mints one unexported random +256-bit process key and computes domain-separated HMAC-SHA-256 values over +length-prefixed canonical encodings. Canonical config encoding preserves own-key +presence, primitive type, array order, and sorted object keys; values that cannot be +represented exactly refuse admission. The auth component covers, per enabled +provider, provider name, effective mode, credential state, exact resolved API-key or +observe-only OAuth token bytes (or explicit absence), the exact +`provider-auth-selection` observation, and the final discovery method/URL/normalized +headers after transport defaults. Header names are lowercased and sorted while values +stay byte-exact only inside the HMAC input. + +The native component is explicit `unused` when no combo needs native rows. Otherwise +the evidence owner captures a detached, deeply frozen ordered native slug/capability +snapshot and records every active-catalog/cache consultation, including absence, as +`native-catalog-selection`. The identity covers that exact snapshot and the immutable +revision identities of provider registry data, generated Jawcode metadata, and the +pinned upstream-model snapshot used during assembly. `provider-fetch.ts` consumes +this captured value; it may not call the current hidden native source path after the +flight is keyed. Today that path reaches `nativeOpenAiSlugs()` during combo assembly +(`src/codex/catalog/provider-fetch.ts:880-904`), and the owner can read active catalog +or cache while selecting slugs (`src/codex/catalog/metadata.ts:169-179`). + +Relevant process-local flight input includes the settled runtime/bundled +epoch/value evidence and a per-provider immutable model-cache/cooldown snapshot with +its owner epoch and value identity. The latter controls whether discovery uses fresh, +stale, configured, or network data (`src/codex/catalog/provider-fetch.ts:519-560`), +so omitting it again treats absence of evidence as equality. It binds map admission +but is not added to K -> C revalidation because the producing flight itself may +advance that cache. Runtime/bundled evidence remains candidate-bound and +commit-revalidated exactly as already specified. + +No raw API key, OAuth token, secret header, auth-store buffer, or stable plain digest +is a field of the identity. Only opaque process-keyed HMAC values and the opaque +config-reference token leave the authority owner, and none may be logged, serialized, +or projected into `CatalogDisposition`. Copying `apiKey` into the old fingerprint, +using plain SHA-256, or exporting the HMAC key is forbidden: all three create a +credential-bearing or offline-guessable structure. +The private candidate may retain the existing exact-buffer source digest solely for +K -> C revalidation of `provider-auth-selection`; the flight owner HMACs that +observation as input and never exposes the plain digest as a key, result, log, or +response. + +The in-flight map stores an entry containing the deep-frozen authority plus its +promise, bucketed by `authorityId`; it is no longer `Map>`. +Component equality is checked before joining, so even a forced bucket collision does +not share. The flight receives the captured authority inputs and returns +`{ authority, models, comboOmissions }`. Each caller compares the returned authority +to its own admission before candidate construction. A mismatch discards the result +and returns retryable stale or regathers within `deadlineMs`; no candidate, K permit, +or filesystem write follows. This second check is mandatory defense in depth, not an +optimization delegated to the map key. + ### A2 — seal the closed role-bearing source observations `captureCatalogAdmissionSnapshot(config)` remains the pre-gather constructor. Before @@ -305,9 +400,9 @@ with every conditional role key present as an empty list and the required `$CODEX_HOME/config.toml` path, recorded PRESENT or ABSENT. The opaque filesystem-evidence session then records every consulted filesystem source under the contract's closed role union: bundled template, active merge, hashed/legacy -fallback, models-cache fallback, runtime selection, or provider-auth selection -(`005_contract.md:830-930,967-1069`). Callers cannot append, omit, remove, or rebuild -those observations. +fallback, models-cache fallback, native-catalog selection, runtime selection, or +provider-auth selection. Callers cannot append, omit, remove, or rebuild those +observations. The required ABSENT state closes a target-selection hole that a present-file digest list cannot represent. `readCodexCatalogPath()` chooses the default catalog exactly @@ -341,7 +436,7 @@ Both paths write zero bytes. This detects the audited same-inode truncate/rewrite even when config generation and target identity are unchanged. It catches single-direction drift only: content/state A→B→A returning identical evidence before comparison, parent A→B→A between checks, and a write after the final -comparison remain outside C17 (`005_contract.md:802-878`). +comparison remain outside C17 (`005_contract.md:830-943`). ### A3 — preserve bundled-first template precedence @@ -349,7 +444,7 @@ comparison remain outside C17 (`005_contract.md:802-878`). catalog first and clone it as the native template; read the on-disk catalog separately as the merge source. The invariant is explicit at `structure/03_catalog-and-subagents.md:23-27` and implemented at -`src/codex/catalog/bundled.ts:225-234` plus +`src/codex/catalog/bundled.ts:482-490` plus `src/codex/catalog/sync.ts:517-523`. The WP9 edit removes the materializing fallback call from the tail of @@ -365,11 +460,14 @@ outside the 16 management paths until their owning phase migrates them. The candidate remains opaque, one-shot, and catalog-private. Its `WeakMap` state contains prepared bytes, result/notices, target identities, the admitted config generation, home selection, sealed candidate-bound `CatalogSourceEvidence`, and -sealed `CatalogProcessLocalEvidence`. Memo-derived graphs are recursively frozen -detached snapshots and cannot alias the owners' private caches. Commit marks it -consumed before validation and before the first write; a second call returns -`candidate-consumed` and writes nothing. No route can inspect, serialize, -reconstruct, or replay it. +sealed `CatalogProcessLocalEvidence`, plus the complete deep-frozen +`CatalogGatherAuthorityIdentity` returned by the flight. Candidate construction first +requires exact equality with the caller's expected authority; mismatch returns +retryable stale without constructing the candidate. Memo-derived graphs are +recursively frozen detached snapshots and cannot alias the owners' private caches. +Commit marks a successfully constructed candidate consumed before validation and +before the first write; a second call returns `candidate-consumed` and writes nothing. +No route can inspect, serialize, reconstruct, or replay it. Only catalog-private outcomes are added here: @@ -466,7 +564,7 @@ regular, non-routed valid catalog backup is preserved and the receipt becomes is `refused`. The loser never unlinks, truncates, or overwrites the winner. This exception applies only to a backup create-once target, never to a backup selected as a gather source; selected source observations remain strict -(`005_contract.md:945-965`). +(`005_contract.md:1034-1054`). ## C. Catalog-only convergence @@ -478,7 +576,7 @@ WP9 does not redeclare request, snapshot, projection, or shared result types. - `createCatalogConvergeRequest` from `src/codex/catalog-admission.ts:38-52`; - `captureCatalogAdmissionSnapshot` from - `src/codex/catalog-admission.ts:148-180`; + `src/codex/catalog-admission.ts:148-185`; - `projectCatalogOnlyOutcome` from its landed owner at `src/codex/management-convergence.ts:63-75`; - shared `CatalogDisposition`, `ConvergeOutcome`, and `ConvergeCodex` from @@ -503,6 +601,10 @@ has no pair fields (`src/codex/convergence-types.ts:207-224`). Catalog staleness is guarded by: +- the complete non-secret-bearing gather-authority identity, checked both before + joining a flight and again between the result and candidate construction, so a + shared promise can never erase the admitted config generation/reference, effective + auth, native/source snapshot, or relevant process-local inputs that produced it; - permanent effective-user/canonical-`CODEX_HOME` serialization K, held across every catalog/backup/cache authoritative transaction by convergence and all four transitional roots, using either under-K recomputation or complete post-acquisition @@ -642,10 +744,39 @@ or other transient write. ### T2 — K, home/cache/source evidence, generation, and identity reject before write +Start with the round-5 live-flight authority matrix. In isolated temporary homes, +pause request A after generation N with `authMode:"forward"` has claimed flight F. +Persist N+1 with `authMode:"key"` and an API key, capture B from the new exact config +reference, and resume. B must claim a distinct flight or reject A's result as +retryable stale; B must never construct or commit A's empty result. Force the two +entries into one primary bucket and require the result-carried component comparison to +reject A even when key routing is wrong. The named broken mutation **restore the +legacy `providerCatalogFingerprint`/`gatherFlightKey` and remove result-authority +validation** must reproduce the audited `fetchCount:0, a:[], b:[]` failure. + +Repeat with config and generation unchanged while A is live, but replace the exact +OAuth-store observation so another active account/token is selected. B must capture a +different `provider-auth-selection` and `authSnapshotIdentity`, run separately or +reject A, and never commit A's rows. The named broken mutation **omit OAuth source +observation plus effective token bytes from the auth HMAC input** must turn this test +red. Repeat again with config/auth unchanged while the observed native catalog/cache +changes the ordered slug/capability snapshot used by combo resolution. B must not +accept A's native rows. The named broken mutation **omit +`native-catalog-selection`/`nativeCatalogSourceIdentity` from the map and result +identity** must turn this test red. + +Each row verifies the private identity inside the owner-level unit fixture, while +instrumented production log/response/serialization sinks receive neither that +identity nor the API key, OAuth token, configured secret header, auth-store bytes, or +their plain SHA-256 values. The named privacy mutation +**put `apiKey`, header values, token text, or a stable unkeyed credential digest into +the fingerprint/identity** fails that assertion and `privacy:scan`. + Table-drive every `CatalogSourceRole`: required config target selection, filesystem-backed bundled-template source, active catalog, selected hashed backup, -selected legacy backup, models-cache fallback, runtime-state source, auth-store source, -and every consulted absent alternative. Gather at config generation N, truncate and +selected legacy backup, models-cache fallback, native-catalog source, runtime-state +source, auth-store source, and every consulted absent alternative. Gather at config +generation N, truncate and rewrite a PRESENT selected source **in place** so file identity and generation remain the same, then commit. Expect `stale` and byte-identical targets. Repeat PRESENT -> ABSENT and ABSENT -> PRESENT. In the required target-selection case, gather with @@ -692,9 +823,10 @@ then attempt nested object and array mutation through the returned graphs. The p owner snapshot and candidate evidence remain byte-identical because the detached clone or immutable view is recursively frozen; a top-level-only freeze is insufficient. If the fixture instead uses the supported explicit owner mutation seam, that operation -must advance the epoch and commit must return `stale` before any write. Rewrite the -existing `tests/codex-runtime.test.ts:428-431` fixture that mutates the returned cached -alias so it uses that intentional owner seam. Separately gather from warm R1 while +must advance the epoch and commit must return `stale` before any write. Keep the +current detached/deep-freeze regression at `tests/codex-runtime.test.ts:138-148`; any +fixture that intentionally changes owner state uses the owner seam. Separately gather +from warm R1 while `codex-runtime.json` is observed ABSENT, create persisted R2 without advancing config generation, and require `stale` with zero writes; repeat PRESENT replacement and removal. The named broken mutations are **compare cache bytes without the epoch**, @@ -708,7 +840,7 @@ mutation API. Process A enters `withExpectedConfigGenerationSync({value:N}, callback)`; callback entry proves validation matched and pauses while the config transaction is still held. Process B then attempts a real persisted config mutation. Because the existing lock is -fail-fast (`src/config.ts:1780-1817`), B's first attempt must report lock/busy and must +fail-fast (`src/config.ts:1790-1839`), B's first attempt must report lock/busy and must not commit N+1 while A is paused. A's synchronous catalog bytes land before callback return; after A releases, B retries through the real mutation API and commits N+1. Conflict never invokes the callback. Instrument SQLite connection creation and @@ -831,7 +963,10 @@ legacy writer rows and reject aliases, re-exports, wrappers, or dynamic imports any new root. The same symbol-resolved graph inventories gather filesystem consultations: every `readFileSync`, `Bun.file`, `existsSync` branch, target `lstat`/`stat`/`realpath`, or wrapper that reaches one must terminate at -`catalog/filesystem-evidence.ts`. Unresolved/computed edges fail closed. +`catalog/filesystem-evidence.ts`. That includes native slug selection: direct +`nativeOpenAiSlugs()` reachability from a keyed flight is forbidden unless its +catalog/cache reads consume the authority-captured evidence session. +Unresolved/computed edges fail closed. At both the `wp9-transitional` and future `wp12-final` inventory versions, every catalog/backup/cache mutation must require K's permit and call K's runtime assertion before its first filesystem mutation. Lock-order fixtures fail on `C -> K`, `K -> N`, @@ -841,7 +976,8 @@ uses K -> C because it never acquires N. Broken mutations that must turn T5 red: add a static top-level management-convergence import, alias a catalog writer into a management route, replace a literal import with a computed dynamic import, or add an absence-only `existsSync`/target `realpath` -outside the evidence owner. Also remove the permit parameter from one mutator, add a +outside the evidence owner. Also restore a bare-promise flight map, let the flight +re-resolve auth/native sources after keying, remove the permit parameter from one mutator, add a fifth unpermitted root, bypass the permit assertion before one filesystem mutator, or invert any lock edge. The sentinel, compile fixture, or fail-closed graph must reject each; T2, not the graph, rejects a permit whose runtime lifetime/home is invalid. @@ -890,7 +1026,7 @@ processes only. No verification invokes `ocx start`, `stop`, `sync`, `restore`, | Criterion | Proof | Concrete broken mutation that makes it red | |---|---|---| | **C1** — gather is filesystem-write-free across user homes and scratch, performs no executable probe/subprocess, and commit is synchronous, fixed, K -> C ordered, one-shot, and receipt-exact | T1 + T3 + T5 | call cold `resolveCodexRuntime`/`loadBundledCodexCatalog`, add an `await` beneath commit, acquire C before K, reorder replacements, pre-set a receipt bit, or replay a consumed candidate | -| **C2/C17** — permanent K makes every first-party authoritative read-transform-write fresh by under-K recomputation or complete post-acquisition evidence revalidation; retained-gathered-first/K-second races are covered against convergence and another retained writer; owner-held config generation, required home/runtime evidence, deeply frozen non-aliased memo snapshots plus epochs, every closed PRESENT/ABSENT source observation, and target identity reject stale work before write; create-once backups publish atomically without clobber | T2 + T3 | leave retained `/api/sync`'s `onDiskCatalog` read before K but guard only replacement, start startup/CLI/restore's authoritative read before K, return or shallow-freeze a private cache snapshot so nested mutation bypasses epoch movement, omit ABSENT `config.toml`/`codex-runtime.json`, skip CODEX_HOME re-resolution, release C before callback, remove same-inode digest comparison, open a second SQLite observer, or replace exclusive publication with overwriting rename | +| **C2/C17** — complete gather-authority identity prevents or rejects cross-admission flight reuse across config/auth/native/source/process drift without exposing credentials; permanent K makes every first-party authoritative read-transform-write fresh by under-K recomputation or complete post-acquisition evidence revalidation; retained-gathered-first/K-second races are covered against convergence and another retained writer; owner-held config generation, required home/runtime evidence, deeply frozen non-aliased memo snapshots plus epochs, every closed PRESENT/ABSENT source observation, and target identity reject stale work before write; create-once backups publish atomically without clobber | T2 + T3 | restore partial `gatherFlightKey` plus a bare result, omit OAuth/native authority components, put raw or plain-hashed credentials in the identity, leave retained `/api/sync`'s `onDiskCatalog` read before K but guard only replacement, start startup/CLI/restore's authoritative read before K, return or shallow-freeze a private cache snapshot so nested mutation bypasses epoch movement, omit ABSENT `config.toml`/`codex-runtime.json`, skip CODEX_HOME re-resolution, release C before callback, remove same-inode digest comparison, open a second SQLite observer, or replace exclusive publication with overwriting rename | | **Catalog/native boundary** — catalog-only never reads/advances the native pair or writes routing/history artifacts | T6 | call `expectation()`/`beginTransition`, add pair fields to `catalog-only`, or invoke config/profile/journal/history writer | | **Best-effort compatibility** — all 16 primary writes retain 2xx/201 and original follow-up order for every catalog failure | T4 | let lazy import/factory/admission throw, scope “zero writes” to the whole route, or return before Claude/Desktop follow-up | | **C14, WP9-bounded** — the 16 management roots reach catalog writers only through convergence; exactly four documented transitional roots remain until WP12; every low-level mutation requires permanent K's fresh acquisition-bound permit and runtime liveness/transaction/home assertion | T2 barriers + permit negatives + T5 symbol graph | add a fifth root, omit K/permit/assertion from one retained chain, accept a leaked/reused/forged/wrong-home permit, hide a writer through alias/re-export/computed import, or accidentally require WP12 to have already rewired `/api/sync`/startup/CLI/restore | From 1b32f82efb815b11dcbbde9a0c003c7b6201d817 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Tue, 4 Aug 2026 21:41:22 +0900 Subject: [PATCH 063/163] feat(codex): K, and why an opaque permit type cannot hold a lock MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Round 4 killed the idea that publishing under a lock is enough. The retained `/api/sync` chain reads the active catalog, captures it, awaits provider gathering, and only then writes from that capture (`catalog/sync.ts:513-565`). If convergence publishes Y during that await, acquiring a lock afterwards does not make the captured X fresh — the writer legally overwrites Y while holding it. So K serializes the whole read-transform-write transaction, by recomputing under K or revalidating complete evidence under K, not by guarding a rename. The subtler defect was the permit. An opaque TypeScript type proves a permit-bearing call path EXISTS; it cannot prove the callback still holds K, and a permit leaked past its callback type-checks perfectly. The type is therefore not the mechanism: this module owns a private registry, and every mutator asks at runtime whether its permit is still live for the home it is about to write. There is no exported constructor, brand value, or registration API, so the only way to hold a registered permit is to be inside a live callback. Revocation happens in `finally` before the transaction is released, including when the callback throws, so a mutator racing the release can never find a live permit without a live lock. `busy_timeout = 0` with `BEGIN IMMEDIATE` is deliberate. Contention has to be a typed outcome the caller decides about; blocking would hold K across an unbounded wait. A callback's own error propagates rather than becoming `unavailable`, because telling a caller to retry a deterministic failure is a lie. K gets its own database beside N rather than sharing one: `K -> C` is legal and `N -> K` nests, so one database would self-contend on the very transition it serializes. Four mutations prove the tests: dropping the finally-revoke fails the leak and reuse cases; dropping the home binding fails the cross-home case; `busy_timeout = 5000` turns the fail-fast case into a 5.3-second block; and trusting the type instead of the registry fails four. --- src/codex/catalog-write-serialization.ts | 241 ++++++++++++++++++ src/codex/convergence-types.ts | 12 + src/codex/user-identity.ts | 35 +++ .../codex-catalog-write-serialization.test.ts | 198 ++++++++++++++ 4 files changed, 486 insertions(+) create mode 100644 src/codex/catalog-write-serialization.ts create mode 100644 tests/codex-catalog-write-serialization.test.ts diff --git a/src/codex/catalog-write-serialization.ts b/src/codex/catalog-write-serialization.ts new file mode 100644 index 000000000..f7a8f10bd --- /dev/null +++ b/src/codex/catalog-write-serialization.ts @@ -0,0 +1,241 @@ +/** + * K — the permanent catalog serialization primitive. + * + * Round 4 of the audit killed the idea that "the replacement happens under a + * lock" is enough. The retained `/api/sync` chain reads the active catalog, + * captures it, awaits provider gathering, and only then writes from that + * captured state (`src/codex/catalog/sync.ts:513,520,526,565`). If convergence + * publishes Y while that await is pending, taking a lock afterwards does not + * make the captured X fresh — the writer legally overwrites Y while holding it. + * So K serializes the whole read-transform-write transaction, either by + * recomputing under K or by revalidating complete evidence under K, not by + * guarding the last rename. + * + * The second defect that round found was subtler: an opaque TypeScript permit + * type proves a permit-bearing call path EXISTS. It cannot prove the callback + * still holds K. A leaked permit used after its callback returned type-checks + * perfectly. So the type is not the mechanism — this module owns a private + * active-permit registry, and every mutator asks it at runtime whether the + * permit it holds is still live for the home it is about to write. + * + * K is permanent and is NOT WP11's native write lock. It never reads or + * advances the native pair. Order is `N -> K -> C`; there is no `C -> K` and no + * `K -> N` (`005_contract.md:660-762`). + * + * Design record: devlog/_plan/260804_codex_write_substrate/005_contract.md §3. + */ +import { chmodSync, lstatSync, realpathSync } from "node:fs"; + +import { Database } from "bun:sqlite"; + +import { + CodexUserIdentityRefusal, + resolveCodexCatalogSerializationDatabasePath, + resolveEffectiveUserIdentity, +} from "./user-identity"; + +/** + * Authorization to perform the fixed sequence of catalog mutations inside ONE + * K acquisition. + * + * Deliberately carries no usable field. Holding this object is necessary but + * never sufficient: `assertCatalogWritePermit` is what actually decides, by + * looking the object up in a registry this module alone can write. A forged + * cast, a prototype copy, or a symbol clone produces a value of this type that + * every writer refuses. + */ +export interface CatalogWritePermit { + readonly [catalogWritePermitBrand]: true; +} + +declare const catalogWritePermitBrand: unique symbol; + +export type CatalogSerializationOutcome = + | { kind: "completed"; value: T } + | { kind: "unavailable"; reason: "busy" | "database" | "unsafe-path" }; + +export class CatalogWritePermitRefusal extends Error { + readonly code = "CODEX_CATALOG_WRITE_PERMIT_REFUSED"; + + constructor(message: string) { + super(message); + this.name = "CatalogWritePermitRefusal"; + } +} + +interface PermitRegistration { + /** The exact canonical CODEX_HOME whose artifacts this acquisition may write. */ + readonly canonicalCodexHome: string; + /** Identifies the acquisition, so a permit cannot outlive its transaction. */ + readonly transactionId: string; + live: boolean; +} + +/** + * The registry is a module-private WeakMap keyed by permit IDENTITY. There is + * no exported constructor, brand value, or registration API, so the only way to + * obtain a registered permit is to be inside a live callback. + */ +const activePermits = new WeakMap(); + +let acquisitionCounter = 0; + +function isBusy(error: unknown): boolean { + const code = error && typeof error === "object" && "code" in error + ? String((error as { code?: unknown }).code) + : ""; + const message = error instanceof Error ? error.message : String(error); + return code === "SQLITE_BUSY" || code === "SQLITE_LOCKED" + || /database (?:is|table is) locked/i.test(message); +} + +/** + * The runtime proof every low-level catalog/backup/cache mutator must obtain + * before its FIRST filesystem mutation — including temp creation, hardening, + * unlink, link, rename, truncate, or replacement. + * + * `owningCodexHome` is supplied by the caller rather than inferred from the + * target's parent directory, because an accepted configured catalog target may + * be absolute and outside CODEX_HOME entirely. + */ +export function assertCatalogWritePermit( + permit: CatalogWritePermit, + owningCodexHome: string, +): void { + const registration = activePermits.get(permit as unknown as object); + if (!registration) { + throw new CatalogWritePermitRefusal( + "The catalog write permit was not minted by the serialization owner.", + ); + } + if (!registration.live) { + throw new CatalogWritePermitRefusal( + "The catalog write permit belongs to a released acquisition.", + ); + } + if (registration.canonicalCodexHome !== owningCodexHome) { + throw new CatalogWritePermitRefusal( + "The catalog write permit authorizes a different CODEX_HOME.", + ); + } +} + +/** + * Acquire K for one canonical CODEX_HOME and run `write` while it is held. + * + * Synchronous by contract: the callback performs no provider request, runtime + * probe, OAuth refresh, subprocess, or awaited work. It may enter + * `withExpectedConfigGenerationSync` — that is the `K -> C` edge. + * + * `busy_timeout = 0` with `BEGIN IMMEDIATE` makes contention fail fast and + * typed; the outer async orchestration decides whether to retry within its + * deadline. Blocking here would hold K across an unbounded wait. + */ +export function withCatalogWriteSerialization( + canonicalCodexHome: string, + write: (permit: CatalogWritePermit) => T, +): CatalogSerializationOutcome { + let databasePath: string; + try { + databasePath = resolveCodexCatalogSerializationDatabasePath( + resolveEffectiveUserIdentity(), + canonicalCodexHome, + ); + } catch (error) { + if (error instanceof CodexUserIdentityRefusal) { + return { kind: "unavailable", reason: "unsafe-path" }; + } + return { kind: "unavailable", reason: "database" }; + } + + let database: Database | undefined; + let transactionOpen = false; + let registration: PermitRegistration | undefined; + let permit: CatalogWritePermit | undefined; + + try { + let databaseWasAbsent = false; + try { + const before = lstatSync(databasePath); + if (before.isSymbolicLink() || !before.isFile()) { + return { kind: "unavailable", reason: "unsafe-path" }; + } + if (process.platform !== "win32") { + const uid = process.getuid?.(); + if (uid === undefined || before.uid !== uid || (before.mode & 0o777) !== 0o600) { + return { kind: "unavailable", reason: "unsafe-path" }; + } + } + } catch (cause) { + const code = cause && typeof cause === "object" && "code" in cause + ? String((cause as { code?: unknown }).code) + : ""; + if (code !== "ENOENT") throw cause; + databaseWasAbsent = true; + } + + database = new Database(databasePath, { create: true }); + if (databaseWasAbsent) { + try { chmodSync(databasePath, 0o600); } catch { /* Windows applies ACLs in WP11. */ } + } + const opened = lstatSync(databasePath); + if (opened.isSymbolicLink() || !opened.isFile() + || realpathSync.native(databasePath) !== databasePath) { + return { kind: "unavailable", reason: "unsafe-path" }; + } + + database.exec("PRAGMA busy_timeout = 0; BEGIN IMMEDIATE"); + transactionOpen = true; + + acquisitionCounter += 1; + registration = { + canonicalCodexHome, + transactionId: `${process.pid}:${acquisitionCounter}`, + live: true, + }; + // A bare object: nothing about it is guessable or reconstructable, because + // authority lives in the registry entry rather than in the value. + permit = {} as CatalogWritePermit; + activePermits.set(permit as unknown as object, registration); + + let value: T; + try { + value = write(permit); + } finally { + // Revoke BEFORE the transaction is released, so a mutator racing the + // release can never find a live permit without a live lock. This runs on + // the throwing path too, which is the case a `finally`-less version gets + // wrong. + registration.live = false; + activePermits.delete(permit as unknown as object); + } + + database.exec("COMMIT"); + transactionOpen = false; + return { kind: "completed", value }; + } catch (error) { + if (transactionOpen) { + try { database?.exec("ROLLBACK"); } catch { /* close releases the transaction */ } + transactionOpen = false; + } + if (registration?.live) { + registration.live = false; + if (permit) activePermits.delete(permit as unknown as object); + } + if (error instanceof CodexUserIdentityRefusal) { + return { kind: "unavailable", reason: "unsafe-path" }; + } + if (isBusy(error)) return { kind: "unavailable", reason: "busy" }; + // A callback failure is the caller's error, not a lock outcome: K acquired + // fine. Reporting it as `unavailable` would tell the caller to retry + // something that will fail identically. + throw error; + } finally { + try { database?.close(); } catch { /* acquisition already finished */ } + } +} + +/** Test-only: prove a leaked permit is dead without reaching into the registry. */ +export function isCatalogWritePermitLive(permit: CatalogWritePermit): boolean { + return activePermits.get(permit as unknown as object)?.live === true; +} diff --git a/src/codex/convergence-types.ts b/src/codex/convergence-types.ts index 84b04d1eb..1301f317f 100644 --- a/src/codex/convergence-types.ts +++ b/src/codex/convergence-types.ts @@ -469,3 +469,15 @@ export type ResolveCodexCoordinatorDatabasePath = ( identity: UserIdentity, canonicalCodexHome: string, ) => string; + +/** + * Return K's FINAL database path; this is never the native coordinator path. + * + * Catalog serialization is a separate ownership surface from N. `K -> C` is a + * legal order and `N -> K` nests, so one shared database would self-contend. + * Consumers append no uid/SID, version, directory or filename. + */ +export type ResolveCodexCatalogSerializationDatabasePath = ( + identity: UserIdentity, + canonicalCodexHome: string, +) => string; diff --git a/src/codex/user-identity.ts b/src/codex/user-identity.ts index f0e9331e1..84ecc97f0 100644 --- a/src/codex/user-identity.ts +++ b/src/codex/user-identity.ts @@ -19,6 +19,7 @@ import { isAbsolute, join, resolve } from "node:path"; import type { ResolveCodexCoordinatorDatabasePath, + ResolveCodexCatalogSerializationDatabasePath, ResolveEffectiveUserIdentity, UserIdentity, } from "./convergence-types"; @@ -184,3 +185,37 @@ export const resolveCodexCoordinatorDatabasePath: ResolveCodexCoordinatorDatabas const homeDigest = createHash("sha256").update(canonicalCodexHome).digest("hex"); return join(locks, `${homeDigest}.sqlite`); }; + +/** + * K's FINAL database path. Never the native coordinator path. + * + * Catalog serialization is a different ownership surface from the native + * coordinator N: `K -> C` is a legal order and `N -> K` nests, so sharing one + * database would make the required nesting self-contend. The two live in + * sibling directories under the same per-user runtime root — same identity + * namespace, same environment-independent parent, distinct exclusion. + * + * Consumers use the returned path verbatim and append nothing + * (`005_contract.md:1256-1330`). + */ +export const resolveCodexCatalogSerializationDatabasePath: + ResolveCodexCatalogSerializationDatabasePath = (identity, canonicalCodexHome) => { + if (!isAbsolute(canonicalCodexHome)) { + refuse("The canonical CODEX_HOME must be an absolute path."); + } + const root = identity.platform === "posix" + ? resolvePosixRuntimeRoot(identity.uid) + : resolveWindowsRuntimeRoot(identity); + const locks = join(root, "catalog-write-locks"); + if (identity.platform === "posix") ensurePrivatePosixDirectory(locks, identity.uid); + else { + try { + mkdirSync(locks, { recursive: true }); + } catch (cause) { + refuse("The Windows catalog serialization directory cannot be created.", cause); + } + } + + const homeDigest = createHash("sha256").update(canonicalCodexHome).digest("hex"); + return join(locks, `${homeDigest}.sqlite`); + }; diff --git a/tests/codex-catalog-write-serialization.test.ts b/tests/codex-catalog-write-serialization.test.ts new file mode 100644 index 000000000..1453c8338 --- /dev/null +++ b/tests/codex-catalog-write-serialization.test.ts @@ -0,0 +1,198 @@ +import { afterEach, beforeEach, expect, test } from "bun:test"; +import { mkdtempSync, realpathSync, rmSync, existsSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +import { + assertCatalogWritePermit, + isCatalogWritePermitLive, + withCatalogWriteSerialization, + type CatalogWritePermit, +} from "../src/codex/catalog-write-serialization"; +import { + resolveCodexCatalogSerializationDatabasePath, + resolveCodexCoordinatorDatabasePath, + resolveEffectiveUserIdentity, +} from "../src/codex/user-identity"; + +let codexHome = ""; +let otherHome = ""; +let previousCodexHome: string | undefined; + +beforeEach(() => { + previousCodexHome = process.env.CODEX_HOME; + codexHome = realpathSync.native(mkdtempSync(join(tmpdir(), "ocx-k-home-"))); + otherHome = realpathSync.native(mkdtempSync(join(tmpdir(), "ocx-k-other-"))); + process.env.CODEX_HOME = codexHome; +}); + +afterEach(() => { + if (previousCodexHome === undefined) delete process.env.CODEX_HOME; + else process.env.CODEX_HOME = previousCodexHome; + const identity = resolveEffectiveUserIdentity(); + for (const home of [codexHome, otherHome]) { + const path = resolveCodexCatalogSerializationDatabasePath(identity, home); + for (const suffix of ["", "-journal", "-wal", "-shm"]) { + rmSync(`${path}${suffix}`, { force: true }); + } + } + rmSync(codexHome, { recursive: true, force: true }); + rmSync(otherHome, { recursive: true, force: true }); +}); + +/** + * Sharing one database with the native coordinator would make `N -> K` nest + * onto itself and deadlock the very transition it is meant to serialize. + */ +test("K's database is never the native coordinator's database", () => { + const identity = resolveEffectiveUserIdentity(); + const kPath = resolveCodexCatalogSerializationDatabasePath(identity, codexHome); + const nPath = resolveCodexCoordinatorDatabasePath(identity, codexHome); + + expect(kPath).not.toBe(nPath); + // Same identity namespace, different exclusion surface. + expect(kPath.endsWith(".sqlite")).toBe(true); + expect(kPath).toContain("catalog-write-locks"); + expect(nPath).toContain("native-write-locks"); +}); + +test("two homes take two different K databases", () => { + const identity = resolveEffectiveUserIdentity(); + expect(resolveCodexCatalogSerializationDatabasePath(identity, codexHome)) + .not.toBe(resolveCodexCatalogSerializationDatabasePath(identity, otherHome)); +}); + +test("a callback holding a live permit may write for its own home", () => { + const outcome = withCatalogWriteSerialization(codexHome, (permit) => { + assertCatalogWritePermit(permit, codexHome); + return "published"; + }); + + expect(outcome).toEqual({ kind: "completed", value: "published" }); +}); + +/** + * The defect this registry exists for: an opaque type proves a permit-bearing + * call path exists, never that the callback still holds K. A leaked permit + * type-checks perfectly, so only a runtime lookup can refuse it. + */ +test("a permit leaked out of its callback is refused afterwards", () => { + let leaked: CatalogWritePermit | undefined; + const outcome = withCatalogWriteSerialization(codexHome, (permit) => { + leaked = permit; + expect(isCatalogWritePermitLive(permit)).toBe(true); + return "done"; + }); + expect(outcome.kind).toBe("completed"); + + expect(isCatalogWritePermitLive(leaked!)).toBe(false); + expect(() => assertCatalogWritePermit(leaked!, codexHome)) + .toThrow("was not minted by the serialization owner"); +}); + +test("a permit is revoked even when its callback throws", () => { + let leaked: CatalogWritePermit | undefined; + expect(() => withCatalogWriteSerialization(codexHome, (permit) => { + leaked = permit; + throw new Error("callback exploded"); + })).toThrow("callback exploded"); + + expect(isCatalogWritePermitLive(leaked!)).toBe(false); + expect(() => assertCatalogWritePermit(leaked!, codexHome)).toThrow(); +}); + +/** + * A later acquisition must not resurrect an earlier permit, even for the same + * home: one live permit authorizes the mutations of ITS OWN callback only. + */ +test("a permit cannot be reused by a later acquisition of the same home", () => { + let first: CatalogWritePermit | undefined; + withCatalogWriteSerialization(codexHome, (permit) => { first = permit; }); + + const outcome = withCatalogWriteSerialization(codexHome, (second) => { + expect(second).not.toBe(first); + expect(() => assertCatalogWritePermit(first!, codexHome)).toThrow(); + assertCatalogWritePermit(second, codexHome); + return "second-only"; + }); + expect(outcome).toEqual({ kind: "completed", value: "second-only" }); +}); + +/** + * The owning home is supplied by the caller, not inferred from the target's + * parent, because a configured catalog target may legitimately be absolute and + * outside CODEX_HOME. That makes cross-home confusion possible, so it is + * refused explicitly. + */ +test("a live permit for one home is refused by a writer for another home", () => { + const outcome = withCatalogWriteSerialization(codexHome, (permit) => { + assertCatalogWritePermit(permit, codexHome); + expect(() => assertCatalogWritePermit(permit, otherHome)) + .toThrow("authorizes a different CODEX_HOME"); + return "home-bound"; + }); + expect(outcome).toEqual({ kind: "completed", value: "home-bound" }); +}); + +test("a forged permit shaped like the real one is refused", () => { + withCatalogWriteSerialization(codexHome, (real) => { + const forgedByCast = {} as CatalogWritePermit; + const forgedByPrototype = Object.create( + Object.getPrototypeOf(real) ?? Object.prototype, + ) as CatalogWritePermit; + const forgedBySymbolCopy = { ...(real as object) } as CatalogWritePermit; + + for (const forged of [forgedByCast, forgedByPrototype, forgedBySymbolCopy]) { + expect(() => assertCatalogWritePermit(forged, codexHome)) + .toThrow("was not minted by the serialization owner"); + } + return null; + }); +}); + +/** + * `busy_timeout = 0` is deliberate: contention must be a typed outcome the + * caller can decide about, not an unbounded wait while K is held. + */ +test("a second acquisition during a live callback is typed busy, not blocked", () => { + const outcome = withCatalogWriteSerialization(codexHome, () => { + const nested = withCatalogWriteSerialization(codexHome, () => "should-not-run"); + expect(nested).toEqual({ kind: "unavailable", reason: "busy" }); + return "outer-kept-k"; + }); + + expect(outcome).toEqual({ kind: "completed", value: "outer-kept-k" }); +}); + +test("a different home is not excluded by a live acquisition", () => { + const outcome = withCatalogWriteSerialization(codexHome, () => { + const other = withCatalogWriteSerialization(otherHome, () => "other-home-ran"); + expect(other).toEqual({ kind: "completed", value: "other-home-ran" }); + return "independent"; + }); + + expect(outcome).toEqual({ kind: "completed", value: "independent" }); +}); + +/** + * A callback failure is the caller's error, not a lock outcome. Reporting it as + * `unavailable` would invite a retry of something that fails identically. + */ +test("a callback error propagates rather than becoming a lock outcome", () => { + expect(() => withCatalogWriteSerialization(codexHome, () => { + throw new TypeError("derivation failed"); + })).toThrow(TypeError); + + // K must be released, so the next acquisition succeeds rather than hanging. + expect(withCatalogWriteSerialization(codexHome, () => "recovered")) + .toEqual({ kind: "completed", value: "recovered" }); +}); + +test("K creates its database private to the effective user", () => { + withCatalogWriteSerialization(codexHome, () => null); + const path = resolveCodexCatalogSerializationDatabasePath( + resolveEffectiveUserIdentity(), + codexHome, + ); + expect(existsSync(path)).toBe(true); +}); From e3e43cfa9b18ae95bd646deb1ae3f10340f2d661 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Tue, 4 Aug 2026 21:41:32 +0900 Subject: [PATCH 064/163] feat(config): looking at the generation should not create it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `readConfigGeneration` resolves and records the path and opens SQLite with `create:true` (`config.ts:1743-1774`, `generation.ts:93-103`). Calling it during WP9's gather would create and register `config-mutation.sqlite` as a side effect of merely looking, in a phase whose entire guarantee is that it writes nothing. The observer opens an existing database `readonly`, checks schema and the singleton row, and closes. No creation, initialization, chmod, registration, or lock. It deliberately reuses `ConfigGenerationRead` rather than adding an `absent` variant. A first draft distinguished the missing database, which reads as more informative and is worse: to a caller allowed only to observe, "no file" and "unreadable" mean the same thing — no generation to admit against — and a distinct `absent` invites treating it as a known-good baseline of zero. That is the absence-as-guarantee mistake that produced five wrong-clean verdicts in the residue classifier, and `010_catalog_seam.md:273-276` already settled it: missing DB, table, or row returns the existing typed unavailable and never initializes generation zero. The test asserts the filesystem is unchanged, not just the return value: the database, its home, and its missing parent must all still not exist, and an existing home's mode must be untouched. Falling back to `readConfigGeneration` turns that red. --- src/codex/generation.ts | 51 +++++++++++++++++- src/config.ts | 6 +++ tests/codex-config-generation.test.ts | 78 ++++++++++++++++++++++++++- 3 files changed, 133 insertions(+), 2 deletions(-) diff --git a/src/codex/generation.ts b/src/codex/generation.ts index fd91ec9c7..1b58a2ce5 100644 --- a/src/codex/generation.ts +++ b/src/codex/generation.ts @@ -11,7 +11,7 @@ * counter. The convergence contract detects that case with its post-commit file * observation instead of pretending SQLite can coordinate an external editor. */ -import { chmodSync } from "node:fs"; +import { chmodSync, statSync } from "node:fs"; import { Database } from "bun:sqlite"; @@ -39,6 +39,23 @@ interface ConfigGenerationRow { value: unknown; } +interface SchemaVersionRow { + schema_version: unknown; +} + +/** + * Observation reuses `ConfigGenerationRead` exactly; there is deliberately no + * extra `absent` variant. + * + * A missing database and an unreadable one mean the same thing to a caller that + * is only allowed to LOOK: no generation is available to admit against. Adding + * `absent` would tempt a caller to treat "no file" as a known-good baseline — + * which is the same absence-as-guarantee mistake that produced five wrong-clean + * verdicts in the residue classifier. Only cooperating config writes may create + * and initialize the singleton (`010_catalog_seam.md:273-276`). + */ +export type ConfigGenerationObservation = ConfigGenerationRead; + function errorCode(error: unknown): string { return error && typeof error === "object" && "code" in error ? String((error as { code?: unknown }).code) @@ -124,6 +141,38 @@ export function readConfigGenerationAtPath(databasePath: string): ConfigGenerati } } +/** + * Observe generation state without preparing the mutation database in any way. + * Missing storage is a first-class state: only cooperating config writes have + * authority to create and initialize the generation singleton. + */ +export function observeConfigGenerationAtPath( + databasePath: string, +): ConfigGenerationObservation { + try { + statSync(databasePath); + } catch (error) { + return unavailable(error); + } + + let database: Database | undefined; + try { + database = new Database(databasePath, { readonly: true }); + const schema = database.query("PRAGMA schema_version").get(); + if (!schema || !Number.isSafeInteger(schema.schema_version)) { + throw new Error("The config generation schema version is invalid."); + } + return { + kind: "ready", + generation: readConfigGenerationInTransaction(database), + }; + } catch (error) { + return unavailable(error); + } finally { + try { database?.close(); } catch { /* observation already completed */ } + } +} + export function bumpConfigGenerationAtPath( databasePath: string, expected: ConfigGeneration, diff --git a/src/config.ts b/src/config.ts index 5227cc4bf..bdfa73c5a 100644 --- a/src/config.ts +++ b/src/config.ts @@ -9,8 +9,10 @@ import { bumpConfigGenerationAtPath, bumpCurrentConfigGeneration, initializeConfigGeneration, + observeConfigGenerationAtPath, readConfigGenerationAtPath, readConfigGenerationInTransaction, + type ConfigGenerationObservation, } from "./codex/generation"; import type { BumpConfigGeneration, @@ -1852,6 +1854,10 @@ export const readConfigGeneration: ReadConfigGeneration = () => { } }; +export function observeConfigGeneration(): ConfigGenerationObservation { + return observeConfigGenerationAtPath(join(getConfigDir(), CONFIG_MUTATION_DB_FILENAME)); +} + export const bumpConfigGeneration: BumpConfigGeneration = expected => { try { return bumpConfigGenerationAtPath(configMutationDatabasePath(), expected); diff --git a/tests/codex-config-generation.test.ts b/tests/codex-config-generation.test.ts index 566237d1d..52cbb6d55 100644 --- a/tests/codex-config-generation.test.ts +++ b/tests/codex-config-generation.test.ts @@ -1,5 +1,13 @@ import { afterEach, beforeEach, expect, test } from "bun:test"; -import { existsSync, mkdtempSync, rmSync, writeFileSync } from "node:fs"; +import { + chmodSync, + existsSync, + mkdirSync, + mkdtempSync, + rmSync, + statSync, + writeFileSync, +} from "node:fs"; import { join } from "node:path"; import { pathToFileURL } from "node:url"; @@ -8,6 +16,7 @@ import { Database } from "bun:sqlite"; import { bumpConfigGeneration, mutatePersistedConfig, + observeConfigGeneration, readConfigGeneration, saveConfig, saveConfigPreservingClaudeCode, @@ -42,6 +51,7 @@ const generationGuardRaceScript = ` `; let testRoot = ""; +let previousCodexHome: string | undefined; let previousOpencodexHome: string | undefined; function config(port = 10100): OcxConfig { @@ -74,17 +84,83 @@ async function collectGuardRaceChild( } beforeEach(() => { + previousCodexHome = process.env.CODEX_HOME; previousOpencodexHome = process.env.OPENCODEX_HOME; testRoot = mkdtempSync(join(import.meta.dir, ".tmp-codex-config-generation-")); + process.env.CODEX_HOME = testRoot; process.env.OPENCODEX_HOME = testRoot; }); afterEach(() => { + if (previousCodexHome === undefined) delete process.env.CODEX_HOME; + else process.env.CODEX_HOME = previousCodexHome; if (previousOpencodexHome === undefined) delete process.env.OPENCODEX_HOME; else process.env.OPENCODEX_HOME = previousOpencodexHome; rmSync(testRoot, { recursive: true, force: true }); }); +test("observe-only generation reports a missing database without creating or chmodding paths", () => { + const absentParent = join(testRoot, "missing-parent"); + const absentHome = join(absentParent, "opencodex-home"); + const rootBefore = statSync(testRoot, { bigint: true }); + process.env.CODEX_HOME = absentHome; + process.env.OPENCODEX_HOME = absentHome; + + const absentObservation = observeConfigGeneration(); + expect(existsSync(join(absentHome, "config-mutation.sqlite"))).toBeFalse(); + expect(existsSync(absentHome)).toBeFalse(); + expect(existsSync(absentParent)).toBeFalse(); + // Missing storage reports the ordinary typed unavailable rather than a + // distinct `absent`: a caller that may only observe must not be handed + // something it could mistake for a known-good baseline of zero. + expect(absentObservation).toEqual({ kind: "unavailable", reason: "database" }); + const rootAfter = statSync(testRoot, { bigint: true }); + expect(rootAfter.mode).toBe(rootBefore.mode); + + const existingHome = join(testRoot, "existing-home"); + mkdirSync(existingHome, { mode: 0o751 }); + chmodSync(existingHome, 0o751); + const existingMode = statSync(existingHome).mode & 0o777; + process.env.CODEX_HOME = existingHome; + process.env.OPENCODEX_HOME = existingHome; + + const existingObservation = observeConfigGeneration(); + expect(existsSync(join(existingHome, "config-mutation.sqlite"))).toBeFalse(); + expect(statSync(existingHome).mode & 0o777).toBe(existingMode); + expect(existingObservation).toEqual({ kind: "unavailable", reason: "database" }); +}); + +test("observe-only generation reads an existing value without modifying its database", () => { + saveConfig(config()); + saveConfig(config(20200)); + const databasePath = join(testRoot, "config-mutation.sqlite"); + const before = statSync(databasePath, { bigint: true }); + + expect(observeConfigGeneration()).toEqual({ + kind: "ready", + generation: { value: 2 }, + }); + + const after = statSync(databasePath, { bigint: true }); + expect({ inode: after.ino, mtime: after.mtimeNs, size: after.size }).toEqual({ + inode: before.ino, + mtime: before.mtimeNs, + size: before.size, + }); +}); + +test("observe-only generation returns typed outcomes for malformed and unreadable databases", () => { + const databasePath = join(testRoot, "config-mutation.sqlite"); + writeFileSync(databasePath, "not sqlite", "utf8"); + + expect(observeConfigGeneration()).toEqual({ kind: "unavailable", reason: "database" }); + expect(observeConfigGeneration()).not.toEqual({ kind: "ready", generation: { value: 0 } }); + + rmSync(databasePath); + mkdirSync(databasePath); + expect(observeConfigGeneration()).toEqual({ kind: "unavailable", reason: "database" }); +}); + test("an initial read creates the singleton generation at zero", () => { expect(readConfigGeneration()).toEqual({ kind: "ready", From ea7970a4c0fd530d095d9796f6b63216e24b32ad Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Tue, 4 Aug 2026 21:41:44 +0900 Subject: [PATCH 065/163] feat(codex): gather may look at the runtime, not run it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The ordinary resolver reaches `probeVersion`, whose sandbox calls `mkdtempSync`/`rmSync` (`runtime.ts:231-279,327-340`), and bundled loading both calls the persisting resolver and runs `codex debug models` (`bundled.ts:127-169`). A phase that promises to write nothing cannot reach either, so gather gets its own observe-only pair rather than a flag on the resolver: a cold miss returns the miss, and is never permission to execute Codex. The caches needed fixing to make that guarantee mean anything. Both owners returned their private cached object directly (`bundled.ts:186`, `runtime.ts:397-400`) with mutable result interfaces, so any caller could edit the process-wide cache in place and a candidate sealed against it would describe state that had since changed underneath. Values are now cloned, recursively frozen before publication, and handed out only as detached clones. Each memo carries a process-lifetime monotonic epoch, advanced before the replacement is exposed, on population, replacement, clear, invalidation, persisted-runtime write, and test reset. The epoch exists because `persistCodexRuntime` writes `codex-runtime.json` and clears the memo without advancing the config generation (`runtime.ts:213-229`) — so process-local authority can move while every durable counter stands still, and gather has to be able to notice. Returning the private object again fails the aliasing test; skipping one epoch advance fails that trigger's test. --- src/codex/catalog/bundled.ts | 322 +++++++++++++++++++++++++++++++---- src/codex/runtime.ts | 197 ++++++++++++++++----- tests/codex-runtime.test.ts | 251 ++++++++++++++++++++++++++- 3 files changed, 697 insertions(+), 73 deletions(-) diff --git a/src/codex/catalog/bundled.ts b/src/codex/catalog/bundled.ts index 7003099a7..610380ca0 100644 --- a/src/codex/catalog/bundled.ts +++ b/src/codex/catalog/bundled.ts @@ -34,28 +34,113 @@ import upstreamModelsSnapshot from "../data/upstream-models.json"; import { activeCodexModelsCachePath, catalogBackupPathFor, findNativeTemplate, isDefaultCatalogPath, legacyCatalogBackupPath, parseCatalogJson, readCatalog, readCatalogBackup, readCodexCatalogPath } from "./parsing"; import type { RawCatalog, RawEntry } from "./parsing"; import { codexExecInvocation, isSpawnableCodexCandidate } from "../exec-invocation"; -import { resolveAndPersistCodexRuntime } from "../runtime"; -import type { EffortClampDiagnostic } from "../runtime"; +import { + parsePersistedCodexRuntime, + peekCodexRuntimeProcessCache, + resolveAndPersistCodexRuntime, +} from "../runtime"; +import type { + DeepReadonly, + EffortClampDiagnostic, + ResolvedCodexRuntime, +} from "../runtime"; export { isSpawnableCodexCandidate, codexExecInvocation } from "../exec-invocation"; export const BUNDLED_CATALOG_CACHE_MS = 60_000; -export let bundledCatalogCache: { +export type ReadonlyRawCatalog = DeepReadonly; + +interface BundledCatalogMemo { /** Selected runtime identity; must change when doctor/sync picks a different binary. */ - key: string; - expiresAt: number; - value: RawCatalog | null; -} | null = null; + readonly key: string; + readonly expiresAt: number; + readonly epoch: number; + readonly valueIdentity: string; + readonly value: ReadonlyRawCatalog | null; +} + +export interface BundledCatalogCacheState { + readonly epoch: number; + readonly valueIdentity: string | null; +} + +let bundledCatalogEpoch = 0; +let bundledCatalogCache: BundledCatalogMemo | null = null; + +function cloneAndDeepFreeze(value: T): DeepReadonly { + const clone = (current: unknown): unknown => { + if (Array.isArray(current)) return current.map(clone); + if (current && typeof current === "object") { + const out: Record = {}; + for (const [key, child] of Object.entries(current)) out[key] = clone(child); + return out; + } + return current; + }; + const freeze = (current: unknown): unknown => { + if (!current || typeof current !== "object" || Object.isFrozen(current)) return current; + for (const child of Object.values(current)) freeze(child); + return Object.freeze(current); + }; + return freeze(clone(value)) as DeepReadonly; +} + +function bundledRuntimeKey( + runtime: Pick, + opencodexHome: string = process.env.OPENCODEX_HOME ?? "", +): string { + return [runtime.command, runtime.version ?? "", opencodexHome].join("\0"); +} + +function publishBundledCatalogCache( + key: string, + expiresAt: number, + value: RawCatalog | null, +): void { + const epoch = ++bundledCatalogEpoch; + bundledCatalogCache = { + key, + expiresAt, + epoch, + valueIdentity: `bundled:${epoch}`, + value: value === null ? null : cloneAndDeepFreeze(value), + }; +} + +function clearBundledCatalogCache(): void { + bundledCatalogEpoch += 1; + bundledCatalogCache = null; +} + +export function bundledCatalogCacheState(): Readonly { + return Object.freeze({ + epoch: bundledCatalogEpoch, + valueIdentity: bundledCatalogCache?.valueIdentity ?? null, + }); +} /** Test-only: clear the bundled-catalog cache (owned here; sync.ts calls this instead of assigning the import). */ export function resetBundledCatalogCacheForTests(): void { - bundledCatalogCache = null; + clearBundledCatalogCache(); } /** Drop the process-local bundled catalog memo (e.g. after runtime selection changes). */ export function invalidateBundledCatalogCache(): void { - bundledCatalogCache = null; + clearBundledCatalogCache(); +} + +/** Test-only owner mutation seam; input is cloned and frozen before publication. */ +export function setBundledCatalogCacheForTests( + runtime: Pick, + value: RawCatalog | null, + options: Readonly<{ expiresAt?: number; opencodexHome?: string }> = {}, +): void { + publishBundledCatalogCache( + bundledRuntimeKey(runtime, options.opencodexHome), + options.expiresAt ?? Date.now() + BUNDLED_CATALOG_CACHE_MS, + value, + ); } export type ExecFile = ( @@ -143,7 +228,7 @@ export function runCodexDebugModels( }); } -export function loadBundledCodexCatalog(deps: BundledCatalogDeps = {}): RawCatalog | null { +export function loadBundledCodexCatalog(deps: BundledCatalogDeps = {}): ReadonlyRawCatalog | null { const useCache = !deps.commandCandidates && !deps.execFileSync && !deps.configDir && !deps.env; const execFile = deps.execFileSync ?? (execFileSync as unknown as ExecFile); // Prefer the single resolved runtime so sync/clamp never probe a different binary @@ -168,11 +253,7 @@ export function loadBundledCodexCatalog(deps: BundledCatalogDeps = {}): RawCatal discoverAlternatives: deps.discoverAlternatives ?? false, }); if (useCache) { - cacheKey = [ - resolved.runtime.command, - resolved.runtime.version ?? "", - process.env.OPENCODEX_HOME ?? "", - ].join("\0"); + cacheKey = bundledRuntimeKey(resolved.runtime); } return [resolved.runtime.command]; })(); @@ -183,33 +264,209 @@ export function loadBundledCodexCatalog(deps: BundledCatalogDeps = {}): RawCatal && bundledCatalogCache.key === cacheKey && bundledCatalogCache.expiresAt > Date.now() ) { - return bundledCatalogCache.value; + return bundledCatalogCache.value === null + ? null + : cloneAndDeepFreeze(bundledCatalogCache.value); } for (const command of unique(candidates)) { try { const catalog = parseCatalogJson(runCodexDebugModels(command, execFile, deps)); if (catalog && findNativeTemplate(catalog)) { if (useCache && cacheKey) { - bundledCatalogCache = { - key: cacheKey, - expiresAt: Date.now() + BUNDLED_CATALOG_CACHE_MS, - value: catalog, - }; + publishBundledCatalogCache( + cacheKey, + Date.now() + BUNDLED_CATALOG_CACHE_MS, + catalog, + ); + return cloneAndDeepFreeze(bundledCatalogCache!.value!); } - return catalog; + return cloneAndDeepFreeze(catalog); } } catch { /* try next candidate */ } } if (useCache && cacheKey) { - bundledCatalogCache = { - key: cacheKey, - expiresAt: Date.now() + BUNDLED_CATALOG_CACHE_MS, - value: null, - }; + publishBundledCatalogCache( + cacheKey, + Date.now() + BUNDLED_CATALOG_CACHE_MS, + null, + ); } return null; } +export type CatalogGatherReadableSourceRole = + | "active-catalog-merge" + | "hashed-backup-fallback" + | "legacy-backup-fallback" + | "models-cache-fallback" + | "runtime-selection"; + +/** + * Temporary structural seam for WP9's filesystem-evidence owner. + * The owner maps each closed role to its admitted path and records PRESENT or + * ABSENT before returning the exact bytes. This adapter never reads a path. + */ +export interface CatalogGatherEvidenceSession { + readSource(role: CatalogGatherReadableSourceRole): Uint8Array | null; +} + +export type CatalogGatherProcessLocalObservation = + | Readonly<{ state: "unused" }> + | Readonly<{ state: "used"; epoch: number; valueIdentity: string }>; + +export type CodexRuntimeForCatalogGather = + | Readonly<{ + kind: "available"; + origin: "process-cache" | "persisted"; + runtime: DeepReadonly; + processLocal: CatalogGatherProcessLocalObservation; + }> + | Readonly<{ + kind: "runtime-unavailable"; + processLocal: Readonly<{ state: "unused" }>; + }>; + +export type CatalogSourceForGather = + | Readonly<{ + kind: "available"; + source: + | "bundled-catalog-template" + | "active-catalog-merge" + | "hashed-backup-fallback" + | "legacy-backup-fallback" + | "models-cache-fallback"; + catalog: ReadonlyRawCatalog; + processLocal: Readonly<{ + runtime: CatalogGatherProcessLocalObservation; + bundledCatalog: CatalogGatherProcessLocalObservation; + }>; + }> + | Readonly<{ + kind: "catalog-unavailable"; + processLocal: Readonly<{ + runtime: Readonly<{ state: "unused" }>; + bundledCatalog: Readonly<{ state: "unused" }>; + }>; + }>; + +const UNUSED_PROCESS_LOCAL = Object.freeze({ state: "unused" as const }); + +function sameRuntimeIdentity( + left: Pick, + right: Pick, +): boolean { + return left.command === right.command && (left.version ?? null) === (right.version ?? null); +} + +/** + * Observe an already-resolved runtime only. The evidence owner supplies the + * exact persisted bytes (or observed absence); this path never probes or writes. + */ +export function peekCodexRuntimeForCatalogGather( + evidenceSession: CatalogGatherEvidenceSession, +): CodexRuntimeForCatalogGather { + const persistedBytes = evidenceSession.readSource("runtime-selection"); + const persisted = persistedBytes === null + ? null + : parsePersistedCodexRuntime(persistedBytes); + const processMemo = peekCodexRuntimeProcessCache(); + + if (processMemo.kind === "available") { + const runtime = processMemo.value.runtime; + if (persistedBytes === null || (persisted && sameRuntimeIdentity(runtime, { + command: persisted.command, + version: persisted.selectedVersion ?? null, + }))) { + return cloneAndDeepFreeze({ + kind: "available" as const, + origin: "process-cache" as const, + runtime, + processLocal: { + state: "used" as const, + epoch: processMemo.epoch, + valueIdentity: processMemo.valueIdentity, + }, + }); + } + } + + if (persisted) { + return cloneAndDeepFreeze({ + kind: "available" as const, + origin: "persisted" as const, + runtime: { + command: persisted.command, + version: persisted.selectedVersion ?? null, + source: persisted.source, + }, + processLocal: UNUSED_PROCESS_LOCAL, + }); + } + + return Object.freeze({ + kind: "runtime-unavailable" as const, + processLocal: UNUSED_PROCESS_LOCAL, + }); +} + +/** + * Resolve only already-observed catalog sources. A cold process cache falls + * through to evidence-owned persisted sources and never becomes probe authority. + */ +export function resolveCatalogSourceForGather( + evidenceSession: CatalogGatherEvidenceSession, +): CatalogSourceForGather { + const bundledMemo = bundledCatalogCache; + if (bundledMemo?.value && bundledMemo.expiresAt > Date.now()) { + const runtime = peekCodexRuntimeForCatalogGather(evidenceSession); + if (runtime.kind === "available" && bundledMemo.key === bundledRuntimeKey(runtime.runtime)) { + return cloneAndDeepFreeze({ + kind: "available" as const, + source: "bundled-catalog-template" as const, + catalog: bundledMemo.value, + processLocal: { + runtime: runtime.processLocal, + bundledCatalog: { + state: "used" as const, + epoch: bundledMemo.epoch, + valueIdentity: bundledMemo.valueIdentity, + }, + }, + }); + } + } + + const roles = [ + "active-catalog-merge", + "hashed-backup-fallback", + "legacy-backup-fallback", + "models-cache-fallback", + ] as const; + for (const role of roles) { + const bytes = evidenceSession.readSource(role); + if (bytes === null) continue; + const catalog = parseCatalogJson(Buffer.from(bytes).toString("utf8")); + if (!catalog || !findNativeTemplate(catalog)) continue; + return cloneAndDeepFreeze({ + kind: "available" as const, + source: role, + catalog, + processLocal: { + runtime: UNUSED_PROCESS_LOCAL, + bundledCatalog: UNUSED_PROCESS_LOCAL, + }, + }); + } + + return Object.freeze({ + kind: "catalog-unavailable" as const, + processLocal: Object.freeze({ + runtime: UNUSED_PROCESS_LOCAL, + bundledCatalog: UNUSED_PROCESS_LOCAL, + }), + }); +} + export function materializeBundledCodexCatalog(path: string, deps: BundledCatalogDeps = {}): RawCatalog | null { const catalog = loadBundledCodexCatalog(deps); if (!catalog) return null; @@ -219,7 +476,7 @@ export function materializeBundledCodexCatalog(path: string, deps: BundledCatalo } catch { return null; } - return catalog; + return JSON.parse(JSON.stringify(catalog)) as RawCatalog; } export function loadCatalogForSync(path: string): RawCatalog | null { @@ -236,16 +493,17 @@ export function loadCatalogForSync(path: string): RawCatalog | null { export function readCurrentCatalogOrCache(): RawCatalog | null { const path = readCodexCatalogPath(); - return (isDefaultCatalogPath(path) ? loadBundledCodexCatalog() : null) - ?? readCatalog(path) - ?? readCatalog(activeCodexModelsCachePath()); + const bundled = isDefaultCatalogPath(path) ? loadBundledCodexCatalog() : null; + if (bundled) return JSON.parse(JSON.stringify(bundled)) as RawCatalog; + return readCatalog(path) ?? readCatalog(activeCodexModelsCachePath()); } export function loadCatalogTemplate(): RawEntry | null { const catalogPath = readCodexCatalogPath(); + const bundled = loadBundledCodexCatalog(); const native = findNativeTemplate(readCatalog(catalogPath)) ?? findNativeTemplate(readCatalogBackup(catalogPath)) ?? findNativeTemplate(readCatalog(activeCodexModelsCachePath())) - ?? findNativeTemplate(loadBundledCodexCatalog()); + ?? findNativeTemplate(bundled ? JSON.parse(JSON.stringify(bundled)) as RawCatalog : null); return native ? JSON.parse(JSON.stringify(native)) : null; } diff --git a/src/codex/runtime.ts b/src/codex/runtime.ts index 6978da569..9680cc4db 100644 --- a/src/codex/runtime.ts +++ b/src/codex/runtime.ts @@ -13,32 +13,38 @@ export type CodexRuntimeSource = | "path" | "fallback"; +export type DeepReadonly = + T extends (...args: never[]) => unknown ? T + : T extends readonly (infer U)[] ? readonly DeepReadonly[] + : T extends object ? { readonly [K in keyof T]: DeepReadonly } + : T; + export interface ResolvedCodexRuntime { - command: string; - version: string | null; - source: CodexRuntimeSource; + readonly command: string; + readonly version: string | null; + readonly source: CodexRuntimeSource; } export interface RuntimeProbeFailure { - command: string; - source: CodexRuntimeSource; - reason: string; + readonly command: string; + readonly source: CodexRuntimeSource; + readonly reason: string; } export interface EffortClampDiagnostic { - runtimePath: string; - runtimeVersion: string | null; - removedEfforts: string[]; - affectedModels: string[]; + readonly runtimePath: string; + readonly runtimeVersion: string | null; + readonly removedEfforts: readonly string[]; + readonly affectedModels: readonly string[]; } export interface ResolveCodexRuntimeResult { - runtime: ResolvedCodexRuntime; - failures: RuntimeProbeFailure[]; - replacedConfigured?: { from: ResolvedCodexRuntime; reason: string }; - newerAvailable?: ResolvedCodexRuntime; + readonly runtime: ResolvedCodexRuntime; + readonly failures: readonly RuntimeProbeFailure[]; + readonly replacedConfigured?: Readonly<{ from: ResolvedCodexRuntime; reason: string }>; + readonly newerAvailable?: ResolvedCodexRuntime; /** Set when the selected runtime could not be written to codex-runtime.json. */ - persistError?: string; + readonly persistError?: string; } export type RuntimeExecFile = ( @@ -70,17 +76,43 @@ export interface ResolveCodexRuntimeDeps { discoverAlternatives?: boolean; } -interface PersistedRuntimeState { - version: 1; - command: string; - source: CodexRuntimeSource; - selectedVersion: string | null; - updatedAt: string; +export interface PersistedCodexRuntimeState { + readonly version: 1; + readonly command: string; + readonly source: CodexRuntimeSource; + readonly selectedVersion?: string | null; + readonly updatedAt: string; } const PERSIST_FILE = "codex-runtime.json"; const CLAMP_PERSIST_FILE = "codex-runtime-clamp.json"; +function cloneAndDeepFreeze(value: T): DeepReadonly { + const clone = (current: unknown): unknown => { + if (Array.isArray(current)) return current.map(clone); + if (current && typeof current === "object") { + const out: Record = {}; + for (const [key, child] of Object.entries(current)) out[key] = clone(child); + return out; + } + return current; + }; + const freeze = (current: unknown): unknown => { + if (!current || typeof current !== "object" || Object.isFrozen(current)) return current; + for (const child of Object.values(current)) freeze(child); + return Object.freeze(current); + }; + return freeze(clone(value)) as DeepReadonly; +} + +function isCodexRuntimeSource(value: unknown): value is CodexRuntimeSource { + return value === "environment" + || value === "configured" + || value === "shim" + || value === "path" + || value === "fallback"; +} + export function codexRuntimeStatePath(configDir: string = getConfigDir()): string { return join(configDir, PERSIST_FILE); } @@ -94,9 +126,16 @@ interface PersistedClampState extends EffortClampDiagnostic { updatedAt: string; } +interface LoadedEffortClampDiagnostic { + runtimePath: string; + runtimeVersion: string | null; + removedEfforts: string[]; + affectedModels: string[]; +} + export function loadLastEffortClamp( deps: ResolveCodexRuntimeDeps = {}, -): EffortClampDiagnostic | null { +): LoadedEffortClampDiagnostic | null { const configDir = deps.configDir ?? getConfigDir(); const read = deps.readFileSync ?? ((path, encoding) => readFileSync(path, encoding)); try { @@ -196,15 +235,31 @@ export function compareCodexVersions(a: string | null, b: string | null): number return 0; } +/** Parse already-observed runtime-selection bytes without consulting the filesystem. */ +export function parsePersistedCodexRuntime( + bytes: string | Uint8Array, +): DeepReadonly | null { + try { + const text = typeof bytes === "string" ? bytes : Buffer.from(bytes).toString("utf8"); + const raw = JSON.parse(text) as Partial; + if (raw.version !== 1 || typeof raw.command !== "string" || !raw.command.trim()) return null; + if (!isCodexRuntimeSource(raw.source) || typeof raw.updatedAt !== "string") return null; + if (raw.selectedVersion !== undefined + && raw.selectedVersion !== null + && typeof raw.selectedVersion !== "string") return null; + return cloneAndDeepFreeze(raw as PersistedCodexRuntimeState); + } catch { + return null; + } +} + export function loadPersistedCodexRuntime( deps: ResolveCodexRuntimeDeps = {}, -): PersistedRuntimeState | null { +): DeepReadonly | null { const configDir = deps.configDir ?? getConfigDir(); const read = deps.readFileSync ?? ((path, encoding) => readFileSync(path, encoding)); try { - const raw = JSON.parse(read(codexRuntimeStatePath(configDir), "utf8")) as PersistedRuntimeState; - if (raw?.version !== 1 || typeof raw.command !== "string" || !raw.command.trim()) return null; - return raw; + return parsePersistedCodexRuntime(read(codexRuntimeStatePath(configDir), "utf8")); } catch { return null; } @@ -216,16 +271,16 @@ export function persistCodexRuntime( ): void { const configDir = deps.configDir ?? getConfigDir(); mkdirSync(configDir, { recursive: true, mode: 0o700 }); - const payload: PersistedRuntimeState = { + const payload: PersistedCodexRuntimeState = { version: 1, command: runtime.command, source: runtime.source, selectedVersion: runtime.version, updatedAt: new Date((deps.now ?? Date.now)()).toISOString(), }; + // Invalidate process authority before the persisted replacement is visible. + clearCodexRuntimeResolveCache(); atomicWriteFile(codexRuntimeStatePath(configDir), `${JSON.stringify(payload, null, 2)}\n`); - // Same-process consumers must re-resolve; catalog cache is keyed by runtime identity. - resolveCache = null; } function probeVersion( @@ -360,15 +415,68 @@ export function effortClampAppliesToRuntime( } const RESOLVE_CACHE_MS = 15_000; -let resolveCache: { key: string; at: number; value: ResolveCodexRuntimeResult } | null = null; +interface ResolveCacheMemo { + readonly key: string; + readonly at: number; + readonly epoch: number; + readonly valueIdentity: string; + readonly value: DeepReadonly; +} + +export type CodexRuntimeProcessCachePeek = + | Readonly<{ + kind: "available"; + epoch: number; + valueIdentity: string; + value: DeepReadonly; + }> + | Readonly<{ kind: "unavailable"; epoch: number }>; + +let resolveCacheEpoch = 0; +let resolveCache: ResolveCacheMemo | null = null; + +function publishResolveCache(key: string, at: number, value: ResolveCodexRuntimeResult): void { + const epoch = ++resolveCacheEpoch; + resolveCache = { + key, + at, + epoch, + valueIdentity: `runtime:${epoch}`, + value: cloneAndDeepFreeze(value), + }; +} + +function clearResolveCache(): void { + resolveCacheEpoch += 1; + resolveCache = null; +} + +/** Clear process-local runtime authority without resolving a replacement. */ +export function clearCodexRuntimeResolveCache(): void { + clearResolveCache(); +} + +/** Observe only an unexpired successful process memo; never resolves or probes. */ +export function peekCodexRuntimeProcessCache(): CodexRuntimeProcessCachePeek { + const memo = resolveCache; + if (!memo || Date.now() - memo.at >= RESOLVE_CACHE_MS) { + return Object.freeze({ kind: "unavailable" as const, epoch: resolveCacheEpoch }); + } + return Object.freeze({ + kind: "available" as const, + epoch: memo.epoch, + valueIdentity: memo.valueIdentity, + value: cloneAndDeepFreeze(memo.value), + }); +} function persistedRuntimeCacheStamp(deps: ResolveCodexRuntimeDeps): string { // Include on-disk selection so doctor --fix in another process busts this memo. const configDir = deps.configDir ?? getConfigDir(); const read = deps.readFileSync ?? ((path, encoding) => readFileSync(path, encoding)); try { - const raw = JSON.parse(read(codexRuntimeStatePath(configDir), "utf8")) as PersistedRuntimeState; - if (raw?.version !== 1 || typeof raw.command !== "string") return ""; + const raw = parsePersistedCodexRuntime(read(codexRuntimeStatePath(configDir), "utf8")); + if (!raw) return ""; return `${raw.command}|${raw.selectedVersion ?? ""}|${raw.updatedAt ?? ""}`; } catch { return ""; @@ -397,17 +505,30 @@ function resolveCacheKey(deps: ResolveCodexRuntimeDeps): string | null { export function resolveCodexRuntime(deps: ResolveCodexRuntimeDeps = {}): ResolveCodexRuntimeResult { const cacheKey = resolveCacheKey(deps); if (cacheKey && resolveCache && resolveCache.key === cacheKey && Date.now() - resolveCache.at < RESOLVE_CACHE_MS) { - return resolveCache.value; + return cloneAndDeepFreeze(resolveCache.value); } const result = resolveCodexRuntimeUncached(deps); - if (cacheKey) resolveCache = { key: cacheKey, at: Date.now(), value: result }; - return result; + if (cacheKey) { + publishResolveCache(cacheKey, Date.now(), result); + return cloneAndDeepFreeze(resolveCache!.value); + } + return cloneAndDeepFreeze(result); } /** Test-only: drop the short-lived process resolve cache. */ export function resetCodexRuntimeResolveCacheForTests(): void { - resolveCache = null; + clearCodexRuntimeResolveCache(); +} + +/** Test-only owner mutation seam; input is cloned and frozen before publication. */ +export function setCodexRuntimeResolveCacheForTests( + value: ResolveCodexRuntimeResult, + deps: ResolveCodexRuntimeDeps = {}, +): void { + const key = resolveCacheKey(deps); + if (!key) throw new TypeError("Injected runtime dependencies cannot populate the process memo."); + publishResolveCache(key, Date.now(), value); } function resolveCodexRuntimeUncached(deps: ResolveCodexRuntimeDeps = {}): ResolveCodexRuntimeResult { @@ -468,7 +589,7 @@ function resolveCodexRuntimeUncached(deps: ResolveCodexRuntimeDeps = {}): Resolv replacedConfigured = { from: { command: persisted.command, - version: persisted.selectedVersion, + version: persisted.selectedVersion ?? null, source: "configured", }, reason: failures.find(item => sameRuntimeCommand(item.command, persisted.command))?.reason @@ -518,7 +639,7 @@ export function resolveAndPersistCodexRuntime( const message = error instanceof Error ? error.message : String(error); const persistError = redactUserPath(redactSecretString(message)).slice(0, 200); console.warn(`[opencodex] Failed to persist Codex runtime selection: ${persistError}`); - return { ...result, persistError }; + return cloneAndDeepFreeze({ ...result, persistError }); } } return result; diff --git a/tests/codex-runtime.test.ts b/tests/codex-runtime.test.ts index bf48f1977..4c37be2d8 100644 --- a/tests/codex-runtime.test.ts +++ b/tests/codex-runtime.test.ts @@ -1,26 +1,59 @@ import { describe, expect, test } from "bun:test"; -import { existsSync, mkdtempSync, readFileSync, writeFileSync } from "node:fs"; +import { chmodSync, existsSync, mkdtempSync, readFileSync, writeFileSync } from "node:fs"; import { tmpdir } from "node:os"; import { join, dirname } from "node:path"; import { + clearCodexRuntimeResolveCache, compareCodexVersions, displayCodexRuntimePath, effortClampAppliesToRuntime, loadLastEffortClamp, loadPersistedCodexRuntime, parseCodexVersionOutput, + peekCodexRuntimeProcessCache, persistCodexRuntime, persistEffortClamp, resolveAndPersistCodexRuntime, resolveCodexRuntime, resetCodexRuntimeResolveCacheForTests, + setCodexRuntimeResolveCacheForTests, type RuntimeExecFile, } from "../src/codex/runtime"; +import { + bundledCatalogCacheState, + invalidateBundledCatalogCache, + peekCodexRuntimeForCatalogGather, + resetBundledCatalogCacheForTests, + resolveCatalogSourceForGather, + setBundledCatalogCacheForTests, + type CatalogGatherEvidenceSession, +} from "../src/codex/catalog/bundled"; function tempConfigDir(): string { return mkdtempSync(join(tmpdir(), "ocx-runtime-")); } +function persistedRuntimeBytes( + command: string, + version: string | null = "0.145.0", +): Uint8Array { + return Buffer.from(JSON.stringify({ + version: 1, + command, + source: "environment", + selectedVersion: version, + updatedAt: "2026-08-04T00:00:00.000Z", + })); +} + +function gatherEvidence( + sources: Partial[0], Uint8Array>> = {}, +): CatalogGatherEvidenceSession { + return { + readSource: role => sources[role] ?? null, + }; +} + describe("parseCodexVersionOutput / compareCodexVersions", () => { test("parses dotted and prerelease versions", () => { expect(parseCodexVersionOutput("codex-cli 0.133.0")).toBe("0.133.0"); @@ -36,6 +69,216 @@ describe("parseCodexVersionOutput / compareCodexVersions", () => { }); }); +describe("observe-only Codex catalog gather caches", () => { + test("cold gather returns typed misses without spawning or probing an executable", () => { + const home = tempConfigDir(); + const launcher = process.platform === "win32" + ? join(home, "codex.cmd") + : join(home, "codex"); + const spawnLog = join(home, "spawn.log"); + if (process.platform === "win32") { + writeFileSync(launcher, [ + "@echo off", + `echo spawn>>"${spawnLog}"`, + "echo codex-cli 0.145.0", + "", + ].join("\r\n")); + } else { + writeFileSync(launcher, [ + "#!/bin/sh", + `printf '%s\\n' spawn >> '${spawnLog}'`, + "printf '%s\\n' 'codex-cli 0.145.0'", + "", + ].join("\n")); + chmodSync(launcher, 0o755); + } + + const previousHome = process.env.OPENCODEX_HOME; + const previousCli = process.env.CODEX_CLI_PATH; + const previousPath = process.env.PATH; + process.env.OPENCODEX_HOME = home; + process.env.CODEX_CLI_PATH = launcher; + process.env.PATH = ""; + resetCodexRuntimeResolveCacheForTests(); + resetBundledCatalogCacheForTests(); + + try { + const evidence = gatherEvidence(); + expect(peekCodexRuntimeForCatalogGather(evidence)).toEqual({ + kind: "runtime-unavailable", + processLocal: { state: "unused" }, + }); + expect(resolveCatalogSourceForGather(evidence)).toEqual({ + kind: "catalog-unavailable", + processLocal: { + runtime: { state: "unused" }, + bundledCatalog: { state: "unused" }, + }, + }); + expect(existsSync(spawnLog) ? readFileSync(spawnLog, "utf8").trim().split("\n").length : 0).toBe(0); + } finally { + if (previousHome === undefined) delete process.env.OPENCODEX_HOME; + else process.env.OPENCODEX_HOME = previousHome; + if (previousCli === undefined) delete process.env.CODEX_CLI_PATH; + else process.env.CODEX_CLI_PATH = previousCli; + if (previousPath === undefined) delete process.env.PATH; + else process.env.PATH = previousPath; + resetCodexRuntimeResolveCacheForTests(); + resetBundledCatalogCacheForTests(); + } + }); + + test("runtime cache reads are detached and recursively frozen", () => { + const deps = { env: { PATH: "" }, discoverAlternatives: false }; + resetCodexRuntimeResolveCacheForTests(); + setCodexRuntimeResolveCacheForTests({ + runtime: { command: "codex", version: null, source: "fallback" }, + failures: [{ command: "missing", source: "path", reason: "not found" }], + }, deps); + + const first = resolveCodexRuntime(deps); + const second = resolveCodexRuntime(deps); + expect(first).not.toBe(second); + expect(Object.isFrozen(first)).toBe(true); + expect(Object.isFrozen(first.runtime)).toBe(true); + expect(Object.isFrozen(first.failures)).toBe(true); + expect(Object.isFrozen(first.failures[0])).toBe(true); + expect(() => { + (first.failures as Array<{ reason: string }>)[0]!.reason = "mutated"; + }).toThrow(); + expect(resolveCodexRuntime(deps).failures[0]?.reason).toBe("not found"); + }); + + test("bundled gather reads are detached and recursively frozen", () => { + const runtime = { command: "/tmp/codex", version: "0.145.0", source: "environment" as const }; + resetBundledCatalogCacheForTests(); + setBundledCatalogCacheForTests(runtime, { + models: [{ + slug: "gpt-5.5", + base_instructions: "private", + supported_reasoning_levels: [{ effort: "medium", description: "medium" }], + }], + }); + const evidence = gatherEvidence({ + "runtime-selection": persistedRuntimeBytes(runtime.command, runtime.version), + }); + + const first = resolveCatalogSourceForGather(evidence); + const second = resolveCatalogSourceForGather(evidence); + expect(first.kind).toBe("available"); + expect(second.kind).toBe("available"); + if (first.kind !== "available" || second.kind !== "available") return; + expect(first.catalog).not.toBe(second.catalog); + expect(Object.isFrozen(first.catalog)).toBe(true); + expect(Object.isFrozen(first.catalog.models)).toBe(true); + expect(Object.isFrozen(first.catalog.models?.[0])).toBe(true); + expect(Object.isFrozen(first.catalog.models?.[0]?.supported_reasoning_levels)).toBe(true); + expect(() => { + (first.catalog.models as Array>)[0]!.base_instructions = "mutated"; + }).toThrow(); + expect(() => { + (first.catalog.models?.[0]?.supported_reasoning_levels as unknown[]).push({ effort: "high" }); + }).toThrow(); + expect(second.catalog.models?.[0]?.base_instructions).toBe("private"); + expect(second.catalog.models?.[0]?.supported_reasoning_levels).toHaveLength(1); + }); + + test("gather consumes a persisted runtime observation and observed disk fallback", () => { + resetCodexRuntimeResolveCacheForTests(); + resetBundledCatalogCacheForTests(); + const evidence = gatherEvidence({ + "runtime-selection": persistedRuntimeBytes("/tmp/persisted-codex"), + "active-catalog-merge": Buffer.from(JSON.stringify({ + models: [{ slug: "gpt-5.5", base_instructions: "observed" }], + })), + }); + + const runtime = peekCodexRuntimeForCatalogGather(evidence); + expect(runtime.kind).toBe("available"); + if (runtime.kind === "available") { + expect(runtime.origin).toBe("persisted"); + expect(runtime.runtime.command).toBe("/tmp/persisted-codex"); + expect(runtime.processLocal).toEqual({ state: "unused" }); + } + + const source = resolveCatalogSourceForGather(evidence); + expect(source.kind).toBe("available"); + if (source.kind === "available") { + expect(source.source).toBe("active-catalog-merge"); + expect(source.catalog.models?.[0]?.base_instructions).toBe("observed"); + expect(source.processLocal).toEqual({ + runtime: { state: "unused" }, + bundledCatalog: { state: "unused" }, + }); + } + }); + + test("runtime epoch advances on population and replacement", () => { + const deps = { env: { PATH: "" }, discoverAlternatives: false }; + resetCodexRuntimeResolveCacheForTests(); + const beforePopulation = peekCodexRuntimeProcessCache().epoch; + setCodexRuntimeResolveCacheForTests({ + runtime: { command: "codex-a", version: "1.0.0", source: "path" }, + failures: [], + }, deps); + const afterPopulation = peekCodexRuntimeProcessCache().epoch; + expect(afterPopulation).toBeGreaterThan(beforePopulation); + + setCodexRuntimeResolveCacheForTests({ + runtime: { command: "codex-b", version: "2.0.0", source: "path" }, + failures: [], + }, deps); + expect(peekCodexRuntimeProcessCache().epoch).toBeGreaterThan(afterPopulation); + }); + + test("bundled epoch advances on population, replacement, and negative-cache publication", () => { + const runtime = { command: "/tmp/codex", version: "0.145.0", source: "environment" as const }; + resetBundledCatalogCacheForTests(); + const beforePopulation = bundledCatalogCacheState().epoch; + setBundledCatalogCacheForTests(runtime, { models: [{ slug: "first" }] }); + const afterPopulation = bundledCatalogCacheState().epoch; + expect(afterPopulation).toBeGreaterThan(beforePopulation); + + setBundledCatalogCacheForTests(runtime, { models: [{ slug: "replacement" }] }); + const afterReplacement = bundledCatalogCacheState().epoch; + expect(afterReplacement).toBeGreaterThan(afterPopulation); + + setBundledCatalogCacheForTests(runtime, null); + expect(bundledCatalogCacheState().epoch).toBeGreaterThan(afterReplacement); + }); + + test("runtime epoch advances on clear", () => { + const before = peekCodexRuntimeProcessCache().epoch; + clearCodexRuntimeResolveCache(); + expect(peekCodexRuntimeProcessCache().epoch).toBeGreaterThan(before); + }); + + test("bundled epoch advances on invalidation", () => { + const before = bundledCatalogCacheState().epoch; + invalidateBundledCatalogCache(); + expect(bundledCatalogCacheState().epoch).toBeGreaterThan(before); + }); + + test("runtime epoch advances on persisted-runtime write", () => { + const before = peekCodexRuntimeProcessCache().epoch; + persistCodexRuntime({ + command: "/tmp/codex", + version: "0.145.0", + source: "configured", + }, { configDir: tempConfigDir() }); + expect(peekCodexRuntimeProcessCache().epoch).toBeGreaterThan(before); + }); + + test("both epochs advance on test reset", () => { + const runtimeBefore = peekCodexRuntimeProcessCache().epoch; + const bundledBefore = bundledCatalogCacheState().epoch; + resetCodexRuntimeResolveCacheForTests(); + resetBundledCatalogCacheForTests(); + expect(peekCodexRuntimeProcessCache().epoch).toBeGreaterThan(runtimeBefore); + expect(bundledCatalogCacheState().epoch).toBeGreaterThan(bundledBefore); + }); +}); + describe("resolveCodexRuntime", () => { test("CODEX_CLI_PATH overrides all other sources when valid", () => { const configDir = tempConfigDir(); @@ -427,8 +670,10 @@ describe("resolveCodexRuntime", () => { try { const cached = resolveCodexRuntime(deps); expect(cached.runtime.source).toBe("fallback"); - cached.runtime.source = "environment"; - (cached.runtime as { version?: string | null }).version = undefined; + setCodexRuntimeResolveCacheForTests({ + runtime: { command: "codex", version: null, source: "environment" }, + failures: cached.failures, + }, deps); const before = readFileSync(statePath, "utf8"); resolveAndPersistCodexRuntime(deps); From cdb0c8faf13a0fe9e7e8c7b167b2548c947bdbc4 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Tue, 4 Aug 2026 21:41:58 +0900 Subject: [PATCH 066/163] feat(oauth): preparing a catalog should not rotate the user's credentials MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The token resolver can enter refresh and persistence (`oauth/index.ts:281-339,352-354`) and catalog gather awaited it (`provider-fetch.ts:410-428`). So merely preparing a catalog could refresh credentials and write to the auth store — a durable, user-visible side effect from an operation that is supposed to observe. Gather now takes an observe-only snapshot: no refresh, no persist, no intent lock, no intent file created or removed, no path hardening, no backup of a malformed store. An expired token is reported as expired instead of triggering a rotation. `store.ts` exposes its normalization as a pure function over an already-read buffer, and `peekAuthStore` reuses it, so there is one parsing rule rather than two that can drift. The bytes come from the filesystem-evidence owner under `provider-auth-selection` rather than from another hidden reader, which is the point: a second reader is a second observation, and a candidate sealed against one cannot be validated against the other. Non-WP9 callers keep the refreshing path unchanged, and a refreshing gather can never share an in-flight request with an observing one. The test asserts filesystem invariants rather than token values: the store stays byte-identical with unchanged mode, mtime and inode, the directory listing is unchanged so an unexpected new file fails, and the refresh path records zero calls. Restoring the refreshing resolver turns it red. --- src/codex/catalog/provider-fetch.ts | 162 ++++++++++++++++-- src/oauth/index.ts | 64 ++++++- src/oauth/store.ts | 37 +++- tests/catalog-oauth-observation.test.ts | 215 ++++++++++++++++++++++++ 4 files changed, 458 insertions(+), 20 deletions(-) create mode 100644 tests/catalog-oauth-observation.test.ts diff --git a/src/codex/catalog/provider-fetch.ts b/src/codex/catalog/provider-fetch.ts index 5422e1db5..f22e56029 100644 --- a/src/codex/catalog/provider-fetch.ts +++ b/src/codex/catalog/provider-fetch.ts @@ -2,7 +2,7 @@ import { execFileSync } from "node:child_process"; import { createHash } from "node:crypto"; import { copyFileSync, existsSync, mkdirSync, readFileSync, realpathSync } from "node:fs"; import { delimiter, dirname, join, resolve } from "node:path"; -import { atomicWriteFile, expandUserPath, getConfigDir, websocketsEnabled } from "../../config"; +import { atomicWriteFile, expandUserPath, getConfigDir, resolveEnvValue, websocketsEnabled } from "../../config"; import { CODEX_CONFIG_PATH, CODEX_MODELS_CACHE_PATH, DEFAULT_CATALOG_PATH, readRootTomlString, resolveCodexConfigPath } from "../paths"; import { clearModelCache, @@ -18,7 +18,12 @@ import { setCached, type ProviderModelDiscoveryFailure, } from "../model-cache"; -import { buildModelsRequest, resolveModelsAuthToken } from "../../oauth"; +import { + buildModelsRequest, + observeActiveOAuthAccessToken, + resolveModelsAuthToken, + type OAuthActiveTokenObservation, +} from "../../oauth"; import type { OcxConfig, OcxProviderConfig } from "../../types"; import { modelInList } from "../../types"; import { CODEX_REASONING_LEVELS, codexEffortRank, configuredReasoningEfforts, modelRecordValue, sanitizeCodexReasoningEfforts } from "../../reasoning-effort"; @@ -64,11 +69,44 @@ import type { ComboCatalogOmission } from "./aggregation"; /** Concurrent gatherRoutedModels callers with the same catalog identity share one live discovery. * Keyed by gatherFlightKey so a different config cannot join or evict the wrong flight. */ +export interface CatalogGatherProviderAuthEvidence { + /** Exact auth-store bytes already read and recorded by the filesystem-evidence owner. */ + readonly authStoreBuffer: Uint8Array | null; +} + +export interface CatalogGatherProviderAuthOutcome { + readonly provider: string; + readonly state: OAuthActiveTokenObservation["kind"]; +} + +export interface GatherRoutedModelsOptions { + comboOmissions?: ComboCatalogOmission[]; + providerAuthOutcomes?: CatalogGatherProviderAuthOutcome[]; +} + interface GatherFlightResult { models: CatalogModel[]; comboOmissions: ComboCatalogOmission[]; + providerAuthOutcomes: CatalogGatherProviderAuthOutcome[]; } +interface ModelsAuthResolution { + readonly apiKey: string | undefined; + readonly observed: boolean; + readonly oauthApiBaseUrl?: string; +} + +type ModelsAuthResolver = + | { readonly kind: "refreshing" } + | { + readonly kind: "observed"; + readonly resolve: (name: string, provider: OcxProviderConfig) => ModelsAuthResolution; + }; + +type ModelsAuthResolverFactory = ( + outcomes: CatalogGatherProviderAuthOutcome[], +) => ModelsAuthResolver; + const gatherInflight = new Map>(); const MAX_CONCURRENT_CATALOG_GATHERS = 8; const gatherGate = createAdmissionGate("catalog_gathers", MAX_CONCURRENT_CATALOG_GATHERS); @@ -407,7 +445,39 @@ function boundedOwnedBy(value: unknown): string | undefined { return value; } -export async function fetchProviderModels(name: string, prov: OcxProviderConfig, ttlMs: number, contextCap?: number): Promise { +const refreshingModelsAuthResolver: ModelsAuthResolver = { kind: "refreshing" }; + +function observedModelsAuthResolver( + authStoreBuffer: Uint8Array | null, + outcomes: CatalogGatherProviderAuthOutcome[], +): ModelsAuthResolver { + return { + kind: "observed", + resolve(name, provider) { + if (provider.authMode === "forward") return { apiKey: undefined, observed: true }; + if (provider.authMode !== "oauth") { + return { apiKey: resolveEnvValue(provider.apiKey), observed: true }; + } + + const observation = observeActiveOAuthAccessToken(name, authStoreBuffer); + outcomes.push({ provider: name, state: observation.kind }); + if (observation.kind !== "available") return { apiKey: undefined, observed: true }; + return { + apiKey: observation.snapshot.accessToken, + observed: true, + ...(observation.snapshot.apiBaseUrl ? { oauthApiBaseUrl: observation.snapshot.apiBaseUrl } : {}), + }; + }, + }; +} + +async function fetchProviderModelsWithAuth( + name: string, + prov: OcxProviderConfig, + ttlMs: number, + contextCap: number | undefined, + resolveAuth: ModelsAuthResolver, +): Promise { if (prov.authMode === "forward") return []; // ChatGPT backend has no /models const seedVertexDefault = prov.adapter === "google" && prov.googleMode === "vertex" @@ -425,7 +495,10 @@ export async function fetchProviderModels(name: string, prov: OcxProviderConfig, clearProviderDiscoveryStatus(name); return configured; } - const apiKey = await resolveModelsAuthToken(name, prov); + const auth: ModelsAuthResolution = resolveAuth.kind === "refreshing" + ? { apiKey: await resolveModelsAuthToken(name, prov), observed: false } + : resolveAuth.resolve(name, prov); + const apiKey = auth.apiKey; // A configured default is a real callable selector and must remain discoverable when a // compatible provider's live /models request fails (issue #308). Keep this separate from the // explicit static list: `liveModels: false` + empty `models[]` intentionally publishes zero @@ -487,7 +560,12 @@ export async function fetchProviderModels(name: string, prov: OcxProviderConfig, return stale ? withVertexDefaultSeed(applyConfigHintsToCachedModels(name, prov, stale, contextCap)) : failedDiscoveryConfigured; } const discovery = resolveProviderModelDiscovery(name, prov); - const { url, headers } = buildModelsRequest(prov, apiKey, name); + const { url, headers } = buildModelsRequest( + prov, + apiKey, + name, + auth.observed ? { oauthApiBaseUrl: auth.oauthApiBaseUrl } : undefined, + ); const urlClass = new URL(url).hostname.endsWith("aiplatform.googleapis.com") ? "vertex-aiplatform" : "provider-models"; @@ -633,6 +711,15 @@ export async function fetchProviderModels(name: string, prov: OcxProviderConfig, } } +export async function fetchProviderModels( + name: string, + prov: OcxProviderConfig, + ttlMs: number, + contextCap?: number, +): Promise { + return fetchProviderModelsWithAuth(name, prov, ttlMs, contextCap, refreshingModelsAuthResolver); +} + export function shouldExposeProviderModel(providerName: string, modelId: string): boolean { if (providerName === "opencode-free") return modelId === "big-pickle" || modelId.endsWith("-free"); return true; @@ -669,35 +756,78 @@ export function filterCatalogVisibleModels( export async function gatherRoutedModels( config: OcxConfig, - options?: { comboOmissions?: ComboCatalogOmission[] }, + options?: GatherRoutedModelsOptions, +): Promise { + return gatherRoutedModelsWithAuth( + config, + `refreshing:${gatherFlightKey(config)}`, + () => refreshingModelsAuthResolver, + options, + ); +} + +/** + * Catalog-gather model discovery using only auth-store bytes already captured by the + * filesystem-evidence owner. This entry point never reaches the refreshing resolver. + */ +export async function gatherRoutedModelsForCatalogGather( + config: OcxConfig, + evidence: CatalogGatherProviderAuthEvidence, + options?: GatherRoutedModelsOptions, +): Promise { + const authStoreBuffer = evidence.authStoreBuffer === null + ? null + : Uint8Array.from(evidence.authStoreBuffer); + const authIdentity = authStoreBuffer === null + ? "absent" + : createHash("sha256").update(authStoreBuffer).digest("hex"); + return gatherRoutedModelsWithAuth( + config, + `observed:${authIdentity}:${gatherFlightKey(config)}`, + outcomes => observedModelsAuthResolver(authStoreBuffer, outcomes), + options, + ); +} + +async function gatherRoutedModelsWithAuth( + config: OcxConfig, + key: string, + createAuthResolver: ModelsAuthResolverFactory, + options?: GatherRoutedModelsOptions, ): Promise { - const key = gatherFlightKey(config); let promise = gatherInflight.get(key); if (!promise) { const lease = gatherGate.tryAcquire(); if (!lease) throw new CatalogGatherBusyError(); // Claim the slot synchronously before any await so same-key callers join this flight. // Distinct keys keep their own entries — a second config must not evict the first. - const flight = gatherRoutedModelsUncached(config).finally(() => { + const flight = gatherRoutedModelsUncached(config, createAuthResolver).finally(() => { if (gatherInflight.get(key) === flight) gatherInflight.delete(key); lease.release(); }); gatherInflight.set(key, flight); promise = flight; } - const { models, comboOmissions } = await promise; + const { models, comboOmissions, providerAuthOutcomes } = await promise; if (options?.comboOmissions) { options.comboOmissions.length = 0; options.comboOmissions.push(...comboOmissions); } + if (options?.providerAuthOutcomes) { + options.providerAuthOutcomes.length = 0; + options.providerAuthOutcomes.push(...providerAuthOutcomes); + } return models; } async function gatherRoutedModelsUncached( config: OcxConfig, + createAuthResolver: ModelsAuthResolverFactory, ): Promise { // Flight-local list: joiners copy from the resolved promise, not a process-global last write. const localOmissions: ComboCatalogOmission[] = []; + const localProviderAuthOutcomes: CatalogGatherProviderAuthOutcome[] = []; + const resolveAuth = createAuthResolver(localProviderAuthOutcomes); const ttlMs = config.modelCacheTtlMs ?? DEFAULT_MODEL_CACHE_TTL_MS; // Persisted provider entries can predate newer registry fields (noVisionModels, // modelInputModalities, ...). The ROUTER merges registry seeds at request time @@ -713,7 +843,13 @@ async function gatherRoutedModelsUncached( return [name, enriched]; }); const lists = await Promise.all( - activeProviders.map(([name, prov]) => fetchProviderModels(name, prov, ttlMs, providerContextCap(config, name))), + activeProviders.map(([name, prov]) => fetchProviderModelsWithAuth( + name, + prov, + ttlMs, + providerContextCap(config, name), + resolveAuth, + )), ); const apiAugmented = augmentRoutedModelsWithRegistryOpenAiApiRows(lists.flat(), config); const all = augmentRoutedModelsWithJawcodeMetadata(apiAugmented, activeProviders.map(([name]) => name), config.providers, config) @@ -811,7 +947,11 @@ async function gatherRoutedModelsUncached( // Custom rows override discovered rows that encode to the same Codex-facing slug. const customKeys = new Set(customModels.map(c => routedSlug(c.provider, c.id))); const deduped = all.filter(m => !customKeys.has(routedSlug(m.provider, m.id))); - return { models: [...deduped, ...customModels], comboOmissions: localOmissions }; + return { + models: [...deduped, ...customModels], + comboOmissions: localOmissions, + providerAuthOutcomes: localProviderAuthOutcomes, + }; } export function augmentRoutedModelsWithRegistryOpenAiApiRows( diff --git a/src/oauth/index.ts b/src/oauth/index.ts index ef777b55e..aa2a8278f 100644 --- a/src/oauth/index.ts +++ b/src/oauth/index.ts @@ -4,7 +4,7 @@ import type { OcxConfig, OcxProviderConfig, RefreshPolicy } from "../types"; import { loadConfig, resolveEnvValue, saveConfig } from "../config"; import { maskEmail } from "../lib/privacy"; import { KiroTokenRefreshError, environmentKiroRoutingMetadata, loginKiro, refreshKiroToken, settleKiroLoginTransaction } from "./kiro"; -import { getAccountCredential, getAccountSet, removeAccount, saveAccountCredential, saveCredential, setActiveAccount, getCredential, credentialGeneration, createOAuthRefreshIntentLock, mergeAccountCredential, markAccountNeedsReauthIfGeneration, readOAuthRefreshIntent, writeOAuthRefreshIntent, markOAuthRefreshIntentStaleOwner, clearOAuthRefreshIntent, OAuthMutationBusyError } from "./store"; +import { getAccountCredential, getAccountSet, removeAccount, saveAccountCredential, saveCredential, setActiveAccount, getCredential, credentialGeneration, createOAuthRefreshIntentLock, mergeAccountCredential, markAccountNeedsReauthIfGeneration, readOAuthRefreshIntent, writeOAuthRefreshIntent, markOAuthRefreshIntentStaleOwner, clearOAuthRefreshIntent, normalizeAuthStoreBuffer, OAuthMutationBusyError } from "./store"; import { loginXai, refreshXaiToken, XAI_LOCAL_CLI_DETACH_WARNING, XaiTokenRequestError } from "./xai"; import { ANTHROPIC_OAUTH_BETA, AnthropicTokenError, loginAnthropic, refreshAnthropicToken } from "./anthropic"; import { loginKimi, refreshKimiToken } from "./kimi"; @@ -56,6 +56,20 @@ export interface OAuthAccessSnapshot { kiro?: Pick; } +export interface ObservedOAuthAccessSnapshot extends OAuthAccessSnapshot { + /** Allowlisted provider API origin consumed by GitHub Copilot model discovery. */ + apiBaseUrl?: string; +} + +export type OAuthActiveTokenObservation = + | { readonly kind: "available"; readonly snapshot: ObservedOAuthAccessSnapshot } + | { readonly kind: "missing" } + | { readonly kind: "malformed" } + | { readonly kind: "needs-reauth" } + | { readonly kind: "expired" } + | { readonly kind: "near-expiry" } + | { readonly kind: "unsupported" }; + const MAX_OAUTH_TOKEN_REFRESH_FLIGHTS = 32; const OAUTH_TOKEN_REFRESH_FLIGHT_STALE_MS = 120_000; interface OAuthRefreshFlightEvidence { flightId: string; dispatched: boolean } @@ -278,6 +292,38 @@ function accessSnapshot(provider: string, accountId: string, cred: OAuthCredenti }; } +/** + * Observe the active OAuth token from an auth-store buffer supplied by its filesystem owner. + * Missing, malformed, reauth-required, and expiring credentials are typed no-token outcomes; + * this path never refreshes, locks, hardens, backs up, or persists credentials. + */ +export function observeActiveOAuthAccessToken( + provider: string, + authStoreBuffer: Uint8Array | null, + now = Date.now(), +): OAuthActiveTokenObservation { + const authStore = normalizeAuthStoreBuffer(authStoreBuffer); + if (authStore.kind === "absent") return { kind: "missing" }; + if (authStore.kind === "malformed") return { kind: "malformed" }; + if (!isOAuthProvider(provider)) return { kind: "unsupported" }; + + const accountSet = authStore.store[provider]; + const account = accountSet?.accounts.find(candidate => candidate.id === accountSet.activeAccountId); + if (!account) return { kind: "missing" }; + if (account.needsReauth) return { kind: "needs-reauth" }; + if (account.credential.expires <= now) return { kind: "expired" }; + if (account.credential.expires <= now + REFRESH_SKEW_MS) return { kind: "near-expiry" }; + + const apiBaseUrl = validateCopilotApiBaseUrl(account.credential.apiBaseUrl); + return { + kind: "available", + snapshot: { + ...accessSnapshot(provider, account.id, account.credential), + ...(apiBaseUrl ? { apiBaseUrl } : {}), + }, + }; +} + async function resolveAccessSnapshotForAccount( provider: string, accountId: string, @@ -583,13 +629,25 @@ function modelDiscoveryTransportSeed(providerName: string, prov: OcxProviderConf * Everyone else uses the OpenAI-style `/models` + Bearer with a `{ data: [{ id, owned_by? }] }` * response. */ -export function buildModelsRequest(prov: OcxProviderConfig, apiKey: string | undefined, providerName = ""): { url: string; headers: Record } { +export interface ModelsRequestObservedAuth { + readonly oauthApiBaseUrl?: string; +} + +export function buildModelsRequest( + prov: OcxProviderConfig, + apiKey: string | undefined, + providerName = "", + observedAuth?: ModelsRequestObservedAuth, +): { url: string; headers: Record } { const transportSeed = modelDiscoveryTransportSeed(providerName, prov); + const copilotApiBaseUrl = observedAuth === undefined + ? (providerName === "github-copilot" ? getOAuthCredentialApiBaseUrl(providerName) : undefined) + : observedAuth.oauthApiBaseUrl; const effectiveProvider = resolveProviderTransport( providerName, transportSeed, undefined, - providerName === "github-copilot" ? getOAuthCredentialApiBaseUrl(providerName) : undefined, + copilotApiBaseUrl, ); const headers: Record = { ...(effectiveProvider.headers ?? {}) }; const discoveryUrl = (defaultUrl: string): string => resolveProviderModelDiscoveryUrl( diff --git a/src/oauth/store.ts b/src/oauth/store.ts index 084019c4a..a018de097 100644 --- a/src/oauth/store.ts +++ b/src/oauth/store.ts @@ -29,7 +29,14 @@ import { import { validateCopilotApiBaseUrl } from "./github-copilot"; import type { OAuthCredentialSource, OAuthCredentials, ProviderAccount, ProviderAccountSet } from "./types"; -type AuthStore = Record; +export type AuthStore = Record; + +export type AuthStoreBufferSnapshot = + | { readonly kind: "ready"; readonly store: AuthStore } + | { readonly kind: "absent" } + | { readonly kind: "malformed" }; + +const authStoreDecoder = new TextDecoder("utf-8", { fatal: true }); let lastReconciledGeneration = 0; let liveOAuthAccountKeys = new Set(); @@ -142,6 +149,27 @@ export function loadAuthStore(): AuthStore { return loadAuthStoreInternal().store; } +/** + * Pure normalization for auth-store bytes already read by another owner. + * This function performs no filesystem consultation, hardening, backup, or persistence. + */ +export function normalizeAuthStoreBuffer(buffer: Uint8Array | null): AuthStoreBufferSnapshot { + if (buffer === null) return { kind: "absent" }; + try { + const parsed: unknown = JSON.parse(authStoreDecoder.decode(buffer)); + if (!parsed || typeof parsed !== "object" || Array.isArray(parsed)) { + return { kind: "malformed" }; + } + const { store } = normalizeAuthStore(parsed); + if (Object.keys(parsed).length > 0 && Object.keys(store).length === 0) { + return { kind: "malformed" }; + } + return { kind: "ready", store }; + } catch { + return { kind: "malformed" }; + } +} + /** * Observe-only auth store read for diagnostics (`ocx doctor` / status). * Does not chmod paths or backup invalid JSON — corrupt files are treated as empty. @@ -149,11 +177,8 @@ export function loadAuthStore(): AuthStore { export function peekAuthStore(): AuthStore { const path = getAuthStorePath(); if (!existsSync(path)) return {}; - try { - return normalizeAuthStore(JSON.parse(readFileSync(path, "utf-8"))).store; - } catch { - return {}; - } + const snapshot = normalizeAuthStoreBuffer(readFileSync(path)); + return snapshot.kind === "ready" ? snapshot.store : {}; } function persist(store: AuthStore): void { diff --git a/tests/catalog-oauth-observation.test.ts b/tests/catalog-oauth-observation.test.ts new file mode 100644 index 000000000..fa1fa144e --- /dev/null +++ b/tests/catalog-oauth-observation.test.ts @@ -0,0 +1,215 @@ +import { afterEach, beforeEach, describe, expect, test } from "bun:test"; +import { + chmodSync, + existsSync, + mkdtempSync, + mkdirSync, + readFileSync, + readdirSync, + rmSync, + statSync, + writeFileSync, +} from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { + observeActiveOAuthAccessToken, + OAUTH_PROVIDERS, +} from "../src/oauth"; +import { + gatherRoutedModelsForCatalogGather, + type CatalogGatherProviderAuthOutcome, +} from "../src/codex/catalog/provider-fetch"; +import { clearModelCache } from "../src/codex/model-cache"; +import { getAuthRefreshIntentPath } from "../src/oauth/store"; +import type { OcxConfig, OcxProviderConfig } from "../src/types"; + +interface FileSnapshot { + readonly bytes: Buffer; + readonly inode: bigint; + readonly mode: bigint; + readonly mtimeNs: bigint; +} + +const originalHome = process.env.HOME; +const originalOpencodexHome = process.env.OPENCODEX_HOME; +const originalCodexHome = process.env.CODEX_HOME; +const originalKimiRefresh = OAUTH_PROVIDERS.kimi!.refresh; + +let root: string; +let opencodexHome: string; + +function authStoreBytes(expires: number): Buffer { + return Buffer.from(JSON.stringify({ + kimi: { + activeAccountId: "active", + accounts: [{ + id: "active", + credential: { + access: "fixture-a", + refresh: "fixture-r", + expires, + }, + }], + }, + }) + "\n"); +} + +function snapshotFile(path: string): FileSnapshot { + const stat = statSync(path, { bigint: true }); + return { + bytes: readFileSync(path), + inode: stat.ino, + mode: stat.mode, + mtimeNs: stat.mtimeNs, + }; +} + +function expectFileUnchanged(path: string, before: FileSnapshot): void { + const after = snapshotFile(path); + expect(Buffer.compare(after.bytes, before.bytes)).toBe(0); + expect({ inode: after.inode, mode: after.mode, mtimeNs: after.mtimeNs }).toEqual({ + inode: before.inode, + mode: before.mode, + mtimeNs: before.mtimeNs, + }); +} + +function liveKimiProvider(onFetch: () => void): OcxProviderConfig { + return { + ...structuredClone(OAUTH_PROVIDERS.kimi!.providerConfig), + liveModels: true, + models: ["k3"], + fetch: async () => { + onFetch(); + return new Response(JSON.stringify({ data: [{ id: "k3" }] }), { + status: 200, + headers: { "content-type": "application/json" }, + }); + }, + }; +} + +async function runCatalogGather( + authStoreBuffer: Uint8Array | null, + onFetch: () => void, +): Promise<{ rows: Awaited>; outcomes: CatalogGatherProviderAuthOutcome[] }> { + const config: OcxConfig = { providers: { kimi: liveKimiProvider(onFetch) } }; + const outcomes: CatalogGatherProviderAuthOutcome[] = []; + const rows = await gatherRoutedModelsForCatalogGather( + config, + { authStoreBuffer }, + { providerAuthOutcomes: outcomes }, + ); + return { rows, outcomes }; +} + +beforeEach(() => { + root = mkdtempSync(join(tmpdir(), "ocx-catalog-auth-observe-")); + opencodexHome = join(root, "opencodex"); + mkdirSync(opencodexHome, { recursive: true, mode: 0o700 }); + process.env.HOME = join(root, "home"); + process.env.OPENCODEX_HOME = opencodexHome; + process.env.CODEX_HOME = join(root, "codex"); + clearModelCache(); +}); + +afterEach(() => { + OAUTH_PROVIDERS.kimi!.refresh = originalKimiRefresh; + clearModelCache(); + if (originalHome === undefined) delete process.env.HOME; + else process.env.HOME = originalHome; + if (originalOpencodexHome === undefined) delete process.env.OPENCODEX_HOME; + else process.env.OPENCODEX_HOME = originalOpencodexHome; + if (originalCodexHome === undefined) delete process.env.CODEX_HOME; + else process.env.CODEX_HOME = originalCodexHome; + rmSync(root, { recursive: true, force: true }); +}); + +describe("catalog gather OAuth observation", () => { + test("expired active token stays typed and gather does not refresh or touch the auth store", async () => { + const now = Date.now(); + const authPath = join(opencodexHome, "auth.json"); + writeFileSync(authPath, authStoreBytes(now - 1), { mode: 0o600 }); + chmodSync(authPath, 0o644); + + // A pre-existing intent marker proves observe-only gather neither removes nor rewrites it. + const intentPath = getAuthRefreshIntentPath("kimi", "active"); + writeFileSync(intentPath, "{\"version\":1,\"sentinel\":true}\n", { mode: 0o600 }); + + const authBefore = snapshotFile(authPath); + const intentBefore = snapshotFile(intentPath); + const listingBefore = readdirSync(opencodexHome).sort(); + const observedBuffer = readFileSync(authPath); + let refreshCalls = 0; + let outboundCalls = 0; + OAUTH_PROVIDERS.kimi!.refresh = async () => { + refreshCalls += 1; + return { access: "replacement-a", refresh: "replacement-r", expires: now + 3_600_000 }; + }; + + expect(observeActiveOAuthAccessToken("kimi", observedBuffer, now).kind).toBe("expired"); + const { rows, outcomes } = await runCatalogGather(observedBuffer, () => { outboundCalls += 1; }); + + expect(rows.map(row => row.id)).toEqual(["k3"]); + expect(refreshCalls).toBe(0); + expect(outboundCalls).toBe(0); + expect(outcomes).toEqual([{ provider: "kimi", state: "expired" }]); + expectFileUnchanged(authPath, authBefore); + expectFileUnchanged(intentPath, intentBefore); + expect(readdirSync(opencodexHome).sort()).toEqual(listingBefore); + expect(readdirSync(opencodexHome).some(name => name.startsWith("auth.json.invalid-"))).toBe(false); + expect(existsSync(`${authPath}.pre-multiauth`)).toBe(false); + }); + + test("unparseable auth-store bytes are typed malformed and never backed up or rewritten", async () => { + const authPath = join(opencodexHome, "auth.json"); + writeFileSync(authPath, "{unparseable\n", { mode: 0o644 }); + const before = snapshotFile(authPath); + const listingBefore = readdirSync(opencodexHome).sort(); + const observedBuffer = readFileSync(authPath); + let refreshCalls = 0; + let outboundCalls = 0; + OAUTH_PROVIDERS.kimi!.refresh = async () => { + refreshCalls += 1; + return { access: "replacement-a", refresh: "replacement-r", expires: Date.now() + 3_600_000 }; + }; + + expect(observeActiveOAuthAccessToken("kimi", observedBuffer).kind).toBe("malformed"); + const { rows, outcomes } = await runCatalogGather(observedBuffer, () => { outboundCalls += 1; }); + + expect(rows.map(row => row.id)).toEqual(["k3"]); + expect(outcomes).toEqual([{ provider: "kimi", state: "malformed" }]); + expect(refreshCalls).toBe(0); + expect(outboundCalls).toBe(0); + expectFileUnchanged(authPath, before); + expect(readdirSync(opencodexHome).sort()).toEqual(listingBefore); + expect(readdirSync(opencodexHome).some(name => name.startsWith("auth.json.invalid-"))).toBe(false); + expect(existsSync(`${authPath}.pre-multiauth`)).toBe(false); + }); + + test("available observed token permits live discovery without entering refresh", async () => { + const now = Date.now(); + const authPath = join(opencodexHome, "auth.json"); + writeFileSync(authPath, authStoreBytes(now + 3_600_000), { mode: 0o644 }); + const before = snapshotFile(authPath); + const listingBefore = readdirSync(opencodexHome).sort(); + const observedBuffer = readFileSync(authPath); + let refreshCalls = 0; + let outboundCalls = 0; + OAUTH_PROVIDERS.kimi!.refresh = async () => { + refreshCalls += 1; + return { access: "replacement-a", refresh: "replacement-r", expires: now + 3_600_000 }; + }; + + expect(observeActiveOAuthAccessToken("kimi", observedBuffer, now).kind).toBe("available"); + const { rows, outcomes } = await runCatalogGather(observedBuffer, () => { outboundCalls += 1; }); + + expect(rows.map(row => row.id)).toEqual(["k3"]); + expect(outcomes).toEqual([{ provider: "kimi", state: "available" }]); + expect(refreshCalls).toBe(0); + expect(outboundCalls).toBe(1); + expectFileUnchanged(authPath, before); + expect(readdirSync(opencodexHome).sort()).toEqual(listingBefore); + }); +}); From 1923a2c0a7af233cc8cd1dfad7c340362e4964e0 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Tue, 4 Aug 2026 21:42:11 +0900 Subject: [PATCH 067/163] test(codex): a fail-fast lock has no guaranteed first-round winner The two-process initialization race required exactly one `updated` on the FIRST attempt. But the coordinator sets `busy_timeout = 0` on purpose, so under load both contenders can lose their first attempt to SQLITE_BUSY and settle it on the retry the test already performs. The assertion was measuring scheduler luck, and it failed twice in full-suite runs while passing in isolation. The invariant worth asserting is that there are never two winners, so the first round is now bounded above rather than pinned. Nothing is loosened about the outcome: `finalKinds` still requires exactly one `updated` and one `conflict`, the loser must observe the winner's txId, and the row is still checked to be a single `{1, winner}`. Five consecutive runs clean. --- tests/codex-transition-state-race.test.ts | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/tests/codex-transition-state-race.test.ts b/tests/codex-transition-state-race.test.ts index 0a8c431e1..2c049fc03 100644 --- a/tests/codex-transition-state-race.test.ts +++ b/tests/codex-transition-state-race.test.ts @@ -195,7 +195,13 @@ test("two real processes racing first use publish exactly one initial transition const results = await Promise.all(children.map(collectProbe)); const firstKinds = results.map(result => result.first?.kind); - expect(firstKinds.filter(kind => kind === "updated")).toHaveLength(1); + // AT MOST one first-attempt winner, not exactly one. The coordinator uses + // `busy_timeout = 0` deliberately, so under load both contenders can lose + // their first attempt to SQLITE_BUSY and resolve it on the retry below. + // Demanding a first-round winner asserts scheduler luck; the invariant that + // actually matters — never two winners — is the filter being <= 1, and the + // terminal state is pinned exactly by `finalKinds`. + expect(firstKinds.filter(kind => kind === "updated").length).toBeLessThanOrEqual(1); for (const result of results) { expect( result.first?.kind === "updated" From ba7083f6c9beb52ecbb4bd1347170a9797f04e1b Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Tue, 4 Aug 2026 22:06:11 +0900 Subject: [PATCH 068/163] feat(codex): the catalog writers, behind a permit they must present first MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four mutators — active catalog, hashed backup, legacy backup, models cache — now sit behind K's permit. Each one's first executable statement is the assertion, before target resolution, temp creation, hardening, link, rename, truncate, or unlink. That ordering is the whole point, and it is easy to get subtly wrong. A writer that validates just before its final rename looks guarded and reads fine, but it has already created and hardened a temp file next to a target it was never authorized to touch. Moving the assertion into the rename callback leaves the final directory listing identical after cleanup, so the test records the individual effects instead: with the late assertion it observes temp, harden, truncate and unlink where it expected none. The owning home comes from the caller rather than the target's parent directory, because an accepted configured catalog target may be absolute and outside CODEX_HOME entirely — inferring it there would authorize by location what was never authorized by ownership. Backups publish create-once through a hardened temp plus `linkSync`, whose EEXIST is the no-clobber: an existing backup is preserved and the result says `preserved` rather than lying about having written. Deleting the destination first turns both backup tests red. Replacement delegates to the existing `atomicWriteFile` rather than hand-rolling a second temp-and-rename discipline. Known gap, left to convergence deliberately: section B also wants an EEXIST winner validated as regular, non-routed, readable and identity-stable. This API has no evidence contract to do that with, so it guarantees atomic no-clobber and reports which happened; the stronger validation belongs where the evidence is. --- src/codex/internal/catalog-writer.ts | 201 ++++++++++++++++++++ tests/codex-catalog-writer.test.ts | 264 +++++++++++++++++++++++++++ 2 files changed, 465 insertions(+) create mode 100644 src/codex/internal/catalog-writer.ts create mode 100644 tests/codex-catalog-writer.test.ts diff --git a/src/codex/internal/catalog-writer.ts b/src/codex/internal/catalog-writer.ts new file mode 100644 index 000000000..4c409c151 --- /dev/null +++ b/src/codex/internal/catalog-writer.ts @@ -0,0 +1,201 @@ +import { chmodSync, linkSync, truncateSync, unlinkSync, writeFileSync } from "node:fs"; + +import { + AtomicWriteResidualTempError, + AtomicWriteSecretResidualError, + atomicWriteFile, + resolveWriteTarget, + type AtomicWriteIO, +} from "../../config"; +import { + assertCatalogWritePermit, + type CatalogWritePermit, +} from "../catalog-write-serialization"; +import { + forgetEphemeralSecretPath, + hardenSecretPath, +} from "../../lib/windows-secret-acl"; + +export interface PreparedCatalogFileWrite { + readonly path: string; + readonly content: string; +} + +export type CatalogBackupPublication = "written" | "preserved"; + +export interface CatalogBackupWriteIO { + readonly resolveTarget: (path: string) => string; + readonly write: (path: string, content: string) => void; + readonly harden: (path: string) => void; + /** Must fail with EEXIST rather than replacing an existing destination. */ + readonly publishNoReplace: (source: string, destination: string) => void; + readonly truncate: (path: string) => void; + readonly unlink: (path: string) => void; +} + +let backupTempSequence = 0; + +function isMissingPathError(error: unknown): boolean { + return (error as NodeJS.ErrnoException | undefined)?.code === "ENOENT"; +} + +function isExistingPathError(error: unknown): boolean { + return (error as NodeJS.ErrnoException | undefined)?.code === "EEXIST"; +} + +function defaultBackupWriteIO(destinationPath: string): CatalogBackupWriteIO { + return { + resolveTarget: resolveWriteTarget, + write: (path, content) => writeFileSync(path, content, { encoding: "utf8", mode: 0o600 }), + harden: path => { + try { chmodSync(path, 0o600); } catch { /* platform may ignore chmod */ } + if (process.platform === "win32") { + hardenSecretPath(path, { required: true, timeoutMemoKey: destinationPath }); + } + }, + publishNoReplace: linkSync, + truncate: path => truncateSync(path, 0), + unlink: unlinkSync, + }; +} + +function removePublishedTemp(tempPath: string, io: CatalogBackupWriteIO): void { + try { + io.unlink(tempPath); + forgetEphemeralSecretPath(tempPath); + return; + } catch (firstError) { + if (isMissingPathError(firstError)) { + forgetEphemeralSecretPath(tempPath); + return; + } + try { + io.unlink(tempPath); + forgetEphemeralSecretPath(tempPath); + return; + } catch (retryError) { + if (isMissingPathError(retryError)) { + forgetEphemeralSecretPath(tempPath); + return; + } + throw new AtomicWriteResidualTempError(tempPath, true, { cause: retryError }); + } + } +} + +function scrubAndRemoveUnpublishedTemp( + tempPath: string, + hardened: boolean, + io: CatalogBackupWriteIO, + cause: unknown, +): void { + let scrubbed = false; + try { + io.truncate(tempPath); + scrubbed = true; + } catch (error) { + if (isMissingPathError(error)) scrubbed = true; + else { + try { + io.write(tempPath, ""); + scrubbed = true; + } catch { /* removal may still succeed */ } + } + } + + let removed = false; + try { + io.unlink(tempPath); + removed = true; + } catch (error) { + if (isMissingPathError(error)) removed = true; + else { + try { + io.unlink(tempPath); + removed = true; + } catch (retryError) { + if (isMissingPathError(retryError)) removed = true; + } + } + } + + if (!removed && !scrubbed) { + throw new AtomicWriteSecretResidualError(tempPath, { cause }); + } + if (!removed && !hardened) { + try { + io.harden(tempPath); + hardened = true; + } catch { /* zero-byte residual is reported honestly */ } + } + if (removed) forgetEphemeralSecretPath(tempPath); + if (!removed) throw new AtomicWriteResidualTempError(tempPath, hardened, { cause }); +} + +function publishCatalogBackup( + prepared: PreparedCatalogFileWrite, + suppliedIo?: CatalogBackupWriteIO, +): CatalogBackupPublication { + const io = suppliedIo ?? defaultBackupWriteIO(prepared.path); + const target = io.resolveTarget(prepared.path); + const tempPath = `${target}.ocx.${process.pid}.backup.${++backupTempSequence}.tmp`; + let hardened = false; + + try { + io.write(tempPath, prepared.content); + io.harden(tempPath); + hardened = true; + io.publishNoReplace(tempPath, target); + } catch (error) { + scrubAndRemoveUnpublishedTemp(tempPath, hardened, io, error); + if (isExistingPathError(error)) return "preserved"; + throw error; + } + + removePublishedTemp(tempPath, io); + return "written"; +} + +/** Replace the active catalog with the caller's already-prepared bytes. */ +export function replaceActiveCodexCatalog( + permit: CatalogWritePermit, + owningCodexHome: string, + prepared: PreparedCatalogFileWrite, + io?: AtomicWriteIO, +): void { + assertCatalogWritePermit(permit, owningCodexHome); + atomicWriteFile(prepared.path, prepared.content, io); +} + +/** Atomically publish the catalog-path-keyed immutable backup without clobbering. */ +export function publishHashedCodexCatalogBackup( + permit: CatalogWritePermit, + owningCodexHome: string, + prepared: PreparedCatalogFileWrite, + io?: CatalogBackupWriteIO, +): CatalogBackupPublication { + assertCatalogWritePermit(permit, owningCodexHome); + return publishCatalogBackup(prepared, io); +} + +/** Atomically publish the legacy immutable backup without clobbering. */ +export function publishLegacyCodexCatalogBackup( + permit: CatalogWritePermit, + owningCodexHome: string, + prepared: PreparedCatalogFileWrite, + io?: CatalogBackupWriteIO, +): CatalogBackupPublication { + assertCatalogWritePermit(permit, owningCodexHome); + return publishCatalogBackup(prepared, io); +} + +/** Replace Codex's models cache with the caller's already-prepared bytes. */ +export function replaceCodexModelsCache( + permit: CatalogWritePermit, + owningCodexHome: string, + prepared: PreparedCatalogFileWrite, + io?: AtomicWriteIO, +): void { + assertCatalogWritePermit(permit, owningCodexHome); + atomicWriteFile(prepared.path, prepared.content, io); +} diff --git a/tests/codex-catalog-writer.test.ts b/tests/codex-catalog-writer.test.ts new file mode 100644 index 000000000..261fd1942 --- /dev/null +++ b/tests/codex-catalog-writer.test.ts @@ -0,0 +1,264 @@ +import { afterEach, beforeEach, expect, test } from "bun:test"; +import { + chmodSync, + existsSync, + linkSync, + mkdirSync, + mkdtempSync, + readFileSync, + readdirSync, + realpathSync, + renameSync, + rmSync, + statSync, + truncateSync, + unlinkSync, + writeFileSync, +} from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +import type { AtomicWriteIO } from "../src/config"; +import { + type CatalogWritePermit, + CatalogWritePermitRefusal, + withCatalogWriteSerialization, +} from "../src/codex/catalog-write-serialization"; +import { + resolveCodexCatalogSerializationDatabasePath, + resolveEffectiveUserIdentity, +} from "../src/codex/user-identity"; +import { + type CatalogBackupWriteIO, + type PreparedCatalogFileWrite, + publishHashedCodexCatalogBackup, + publishLegacyCodexCatalogBackup, + replaceActiveCodexCatalog, + replaceCodexModelsCache, +} from "../src/codex/internal/catalog-writer"; + +interface MutatorCase { + readonly name: string; + readonly invoke: ( + permit: CatalogWritePermit, + owningCodexHome: string, + prepared: PreparedCatalogFileWrite, + effects: string[], + ) => unknown; +} + +let testRoot = ""; +let codexHome = ""; +let otherCodexHome = ""; +let targetDir = ""; +let previousCodexHome: string | undefined; +let previousOpenCodexHome: string | undefined; + +function atomicIo(effects: string[]): AtomicWriteIO { + return { + write(path, content) { + effects.push(`temp:${path}`); + writeFileSync(path, content, { encoding: "utf8", mode: 0o600 }); + }, + harden(path) { + effects.push(`harden:${path}`); + chmodSync(path, 0o600); + }, + rename(source, destination) { + effects.push(`rename:${source}->${destination}`); + expect(existsSync(source)).toBe(true); + renameSync(source, destination); + }, + truncate(path) { + effects.push(`truncate:${path}`); + truncateSync(path, 0); + }, + unlink(path) { + effects.push(`unlink:${path}`); + unlinkSync(path); + }, + }; +} + +function backupIo(effects: string[]): CatalogBackupWriteIO { + return { + resolveTarget: path => path, + write(path, content) { + effects.push(`temp:${path}`); + writeFileSync(path, content, { encoding: "utf8", mode: 0o600 }); + }, + harden(path) { + effects.push(`harden:${path}`); + chmodSync(path, 0o600); + }, + publishNoReplace(source, destination) { + effects.push(`publish:${source}->${destination}`); + expect(existsSync(source)).toBe(true); + linkSync(source, destination); + }, + truncate(path) { + effects.push(`truncate:${path}`); + truncateSync(path, 0); + }, + unlink(path) { + effects.push(`unlink:${path}`); + unlinkSync(path); + }, + }; +} + +const mutators: readonly MutatorCase[] = [ + { + name: "active catalog replacement", + invoke: (permit, home, prepared, effects) => + replaceActiveCodexCatalog(permit, home, prepared, atomicIo(effects)), + }, + { + name: "hashed backup publication", + invoke: (permit, home, prepared, effects) => + publishHashedCodexCatalogBackup(permit, home, prepared, backupIo(effects)), + }, + { + name: "legacy backup publication", + invoke: (permit, home, prepared, effects) => + publishLegacyCodexCatalogBackup(permit, home, prepared, backupIo(effects)), + }, + { + name: "models cache replacement", + invoke: (permit, home, prepared, effects) => + replaceCodexModelsCache(permit, home, prepared, atomicIo(effects)), + }, +] as const; + +function directorySnapshot(): string[] { + return readdirSync(targetDir).sort(); +} + +function expectRefusedBeforeFilesystemEffect( + mutator: MutatorCase, + permit: CatalogWritePermit, + owningHome = codexHome, +): void { + const before = directorySnapshot(); + const effects: string[] = []; + const prepared = { + path: join(targetDir, `${mutator.name.replaceAll(" ", "-")}.json`), + content: "new bytes\n", + }; + + expect(() => mutator.invoke(permit, owningHome, prepared, effects)) + .toThrow(CatalogWritePermitRefusal); + expect(effects).toEqual([]); + expect(directorySnapshot()).toEqual(before); +} + +function withLivePermit(callback: (permit: CatalogWritePermit) => T): T { + const outcome = withCatalogWriteSerialization(codexHome, callback); + expect(outcome.kind).toBe("completed"); + if (outcome.kind !== "completed") throw new Error(`K unavailable: ${outcome.reason}`); + return outcome.value; +} + +beforeEach(() => { + previousCodexHome = process.env.CODEX_HOME; + previousOpenCodexHome = process.env.OPENCODEX_HOME; + testRoot = realpathSync.native(mkdtempSync(join(tmpdir(), "ocx-catalog-writer-"))); + codexHome = join(testRoot, "codex-home"); + otherCodexHome = join(testRoot, "other-codex-home"); + targetDir = join(testRoot, "external-catalog-targets"); + for (const path of [codexHome, otherCodexHome, targetDir, join(testRoot, "opencodex-home")]) { + mkdirSync(path, { recursive: true }); + } + process.env.CODEX_HOME = codexHome; + process.env.OPENCODEX_HOME = join(testRoot, "opencodex-home"); +}); + +afterEach(() => { + if (previousCodexHome === undefined) delete process.env.CODEX_HOME; + else process.env.CODEX_HOME = previousCodexHome; + if (previousOpenCodexHome === undefined) delete process.env.OPENCODEX_HOME; + else process.env.OPENCODEX_HOME = previousOpenCodexHome; + + const identity = resolveEffectiveUserIdentity(); + for (const home of [codexHome, otherCodexHome]) { + const databasePath = resolveCodexCatalogSerializationDatabasePath(identity, home); + for (const suffix of ["", "-journal", "-wal", "-shm"]) { + rmSync(`${databasePath}${suffix}`, { force: true }); + } + } + rmSync(testRoot, { recursive: true, force: true }); +}); + +test("every mutator refuses a missing or forged permit before temp creation", () => { + for (const mutator of mutators) { + expectRefusedBeforeFilesystemEffect( + mutator, + undefined as unknown as CatalogWritePermit, + ); + expectRefusedBeforeFilesystemEffect(mutator, {} as CatalogWritePermit); + } +}); + +test("every mutator refuses leaked and revoked permits before temp creation", () => { + let leaked: CatalogWritePermit | undefined; + withLivePermit((permit) => { + leaked = permit; + }); + + let revoked: CatalogWritePermit | undefined; + expect(() => withCatalogWriteSerialization(codexHome, (permit) => { + revoked = permit; + throw new Error("revoke this acquisition"); + })).toThrow("revoke this acquisition"); + + for (const mutator of mutators) { + expectRefusedBeforeFilesystemEffect(mutator, leaked!); + expectRefusedBeforeFilesystemEffect(mutator, revoked!); + } +}); + +test("every mutator refuses a live permit bound to a different home before temp creation", () => { + withLivePermit((permit) => { + for (const mutator of mutators) { + expectRefusedBeforeFilesystemEffect(mutator, permit, otherCodexHome); + } + }); +}); + +for (const mutator of mutators) { + test(`${mutator.name} writes prepared bytes atomically with the right live permit`, () => { + const path = join(targetDir, `${mutator.name.replaceAll(" ", "-")}.json`); + const isBackup = mutator.name.includes("backup"); + if (!isBackup) writeFileSync(path, "old bytes\n", { mode: 0o600 }); + const effects: string[] = []; + + const result = withLivePermit((permit) => + mutator.invoke(permit, codexHome, { path, content: "new bytes\n" }, effects) + ); + + expect(readFileSync(path, "utf8")).toBe("new bytes\n"); + expect(statSync(path).mode & 0o777).toBe(0o600); + expect(readdirSync(targetDir).filter(name => name.endsWith(".tmp"))).toEqual([]); + expect(effects.some(effect => effect.startsWith("temp:"))).toBe(true); + expect(effects.some(effect => effect.startsWith(isBackup ? "publish:" : "rename:"))).toBe(true); + if (isBackup) expect(result).toBe("written"); + }); +} + +for (const [name, publish] of [ + ["hashed", publishHashedCodexCatalogBackup], + ["legacy", publishLegacyCodexCatalogBackup], +] as const) { + test(`${name} create-once backup preserves an existing winner byte-for-byte`, () => { + const path = join(targetDir, `${name}.backup.json`); + writeFileSync(path, "first winner\n", { mode: 0o600 }); + + const result = withLivePermit((permit) => + publish(permit, codexHome, { path, content: "late contender\n" }) + ); + + expect(result).toBe("preserved"); + expect(readFileSync(path, "utf8")).toBe("first winner\n"); + expect(readdirSync(targetDir).filter(entry => entry.endsWith(".tmp"))).toEqual([]); + }); +} From 2becc771977afb112fc8db45ed878fb67625c1a5 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Tue, 4 Aug 2026 22:06:26 +0900 Subject: [PATCH 069/163] feat(codex): one owner for every byte gather reads, and for every absence MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Gather now reads sources and probes targets through a single owner that records PRESENT or ABSENT before returning anything. Callers cannot append, remove, or rebuild the evidence themselves; only the owner seals a session, and sealing without the required `catalog-target-selection` observation is refused. Recording absence is the part that carries the weight. A source that was missing is why a fallback was chosen, so omitting it leaves a candidate that cannot explain itself and cannot be revalidated under K. Every conditional role is a key whose empty list means "not consulted" — the same distinction this unit has had to relearn five times on disk. Home selection is captured and sealed before any derived path is accepted, because a path derived from a home that had not been pinned yet is evidence of nothing. Admission switches to the observe-only generation read, so preparing a catalog no longer creates the mutation database as a side effect of looking at it. This also retires two placeholder interfaces. The runtime/bundled and OAuth leaves were built in parallel against assumed shapes because this owner did not exist yet; both now import the real types and re-export them so their callers and tests are unaffected, and the owner's `authStoreBuffer` reads through the same observation-first path rather than being a second reader of the same file. Zero-write is asserted rather than assumed: the test compares the full recursive listing plus content hash, mode, nanosecond mtime, inode, device, size and symlink target across populated, empty and nonexistent homes. Returning bytes without recording the observation, or sealing without the required role, each turn red. One ambiguity worth naming: the contract forbids filesystem consultation outside this owner, but production `defaultCodexHome()` consults the filesystem itself, including WSL discovery. The owner calls that production resolver rather than duplicating its behavior, so "outside the owner" is read as the gather call path. Relocating the resolver would be a larger, separate change. --- src/codex/catalog-admission.ts | 217 ++++++++-------- src/codex/catalog/bundled.ts | 24 +- src/codex/catalog/filesystem-evidence.ts | 302 +++++++++++++++++++++++ src/codex/catalog/provider-fetch.ts | 8 +- src/codex/convergence-types.ts | 44 ++++ tests/codex-catalog-admission.test.ts | 40 ++- tests/codex-filesystem-evidence.test.ts | 242 ++++++++++++++++++ 7 files changed, 741 insertions(+), 136 deletions(-) create mode 100644 src/codex/catalog/filesystem-evidence.ts create mode 100644 tests/codex-filesystem-evidence.test.ts diff --git a/src/codex/catalog-admission.ts b/src/codex/catalog-admission.ts index fbcc31d31..3b6f74ff9 100644 --- a/src/codex/catalog-admission.ts +++ b/src/codex/catalog-admission.ts @@ -7,29 +7,31 @@ * later phase. This reader therefore captures only the exact resident config, * its cooperating generation, and identities for catalog-owned targets. */ -import { createHash } from "node:crypto"; -import { readFileSync, realpathSync, statSync } from "node:fs"; -import { basename, dirname, resolve } from "node:path"; +import { createHmac, randomBytes } from "node:crypto"; +import { join, resolve } from "node:path"; -import { readConfigGeneration } from "../config"; +import { observeConfigGeneration } from "../config"; import type { OcxConfig } from "../types"; import type { CatalogAdmissionSnapshot, CatalogConvergeRequestInput, - CatalogFilesystemIdentity, - CatalogSourceObservation, + ConfigGeneration, ConvergeRequest, } from "./convergence-types"; import { - activeCodexConfigPath, - activeDefaultCatalogPath, - activeCodexModelsCachePath, catalogBackupPathFor, - isDefaultCatalogPath, legacyCatalogBackupPath, - resolveActiveCodexConfigPath, + samePath, } from "./catalog/parsing"; import { readRootTomlString } from "./paths"; +import { + acceptCatalogGatherSourcePath, + captureAndSealCatalogHomeSelection, + captureCatalogGatherTargetIdentity, + createCatalogGatherEvidenceSession, + readCatalogGatherSource, + sealCatalogGatherEvidenceSession, +} from "./catalog/filesystem-evidence"; /** * Construct the one request shape permitted for management catalog refreshes. @@ -51,93 +53,81 @@ export function createCatalogConvergeRequest({ }; } -function optionalFileIdentity(path: string): Readonly<{ device: string; inode: string }> | null { - try { - const entry = statSync(path, { bigint: true }); - return { device: String(entry.dev), inode: String(entry.ino) }; - } catch (error) { - if (error && typeof error === "object" && "code" in error && error.code === "ENOENT") { - return null; - } - throw error; - } -} - -/** - * Encode identity evidence in the contract-owned string slot. - * - * `CatalogAdmissionSnapshot` deliberately owns the target shape. Encoding the - * evidence here avoids a second shared target type while still detecting the - * parent-symlink retarget that a textual path alone missed during C2 review. - */ -function captureTargetIdentity(path: string): string { - const textualPath = resolve(path); - const canonicalParent = realpathSync.native(dirname(textualPath)); - const parent = statSync(canonicalParent, { bigint: true }); - return JSON.stringify({ - path: textualPath, - canonicalParent, - parentIdentity: { device: String(parent.dev), inode: String(parent.ino) }, - fileIdentity: optionalFileIdentity(textualPath), - }); -} +const CONFIG_IDENTITY_KEY = randomBytes(32); +const configReferenceIdentities = new WeakMap(); +let nextConfigReferenceIdentity = 0; -function catalogFilesystemIdentity( - entry: Readonly<{ dev: bigint; ino: bigint }>, -): CatalogFilesystemIdentity { - return { volume: String(entry.dev), fileId: String(entry.ino) }; +function encodeLengthPrefixed(value: string): string { + return `${Buffer.byteLength(value, "utf8")}:${value}`; } -function captureCatalogTargetSelection(): Readonly<{ - catalogPath: string; - observation: CatalogSourceObservation<"catalog-target-selection">; -}> { - const logicalPath = resolve(activeCodexConfigPath()); - const canonicalParent = realpathSync.native(dirname(logicalPath)); - const parent = statSync(canonicalParent, { bigint: true }); - const parentIdentity = { - canonicalPath: canonicalParent, - ...catalogFilesystemIdentity(parent), - }; +function canonicalConfigEncoding(value: unknown, ancestors = new Set()): string { + if (value === null) return "null"; + switch (typeof value) { + case "undefined": return "undefined"; + case "boolean": return value ? "boolean:1" : "boolean:0"; + case "string": return `string:${encodeLengthPrefixed(value)}`; + case "number": { + if (!Number.isFinite(value)) throw new TypeError("Catalog config identity cannot encode a non-finite number."); + return `number:${Object.is(value, -0) ? "-0" : String(value)}`; + } + case "bigint": + case "function": + case "symbol": + throw new TypeError(`Catalog config identity cannot encode ${typeof value}.`); + case "object": break; + } - let bytes: Buffer; + if (ancestors.has(value)) throw new TypeError("Catalog config identity cannot encode a cyclic graph."); + if (Object.getOwnPropertySymbols(value).length > 0) { + throw new TypeError("Catalog config identity cannot encode symbol keys."); + } + ancestors.add(value); try { - bytes = readFileSync(logicalPath); - } catch (error) { - if (!error || typeof error !== "object" || !("code" in error) || error.code !== "ENOENT") { - throw error; + if (Array.isArray(value)) { + const items = Array.from({ length: value.length }, (_, index) => ( + Object.hasOwn(value, index) + ? `item:${canonicalConfigEncoding(value[index], ancestors)}` + : "hole" + )); + return `array:${value.length}:${items.map(encodeLengthPrefixed).join("")}`; } - return { - catalogPath: activeDefaultCatalogPath(), - observation: { - state: "absent", - role: "catalog-target-selection", - logicalPath, - canonicalPath: resolve(canonicalParent, basename(logicalPath)), - parentIdentity, - fileIdentity: null, - }, - }; + const keys = Object.keys(value).sort(); + const entries = keys.map(key => { + const descriptor = Object.getOwnPropertyDescriptor(value, key); + if (!descriptor || !("value" in descriptor)) { + throw new TypeError("Catalog config identity cannot encode accessor properties."); + } + return `${encodeLengthPrefixed(key)}${encodeLengthPrefixed(canonicalConfigEncoding(descriptor.value, ancestors))}`; + }); + return `object:${keys.length}:${entries.join("")}`; + } finally { + ancestors.delete(value); } +} - const canonicalPath = realpathSync.native(logicalPath); - const file = statSync(canonicalPath, { bigint: true }); - const configuredCatalogPath = readRootTomlString(bytes.toString("utf8"), "model_catalog_json"); +function keyedConfigIdentity(domain: string, payload: string): string { + return createHmac("sha256", CONFIG_IDENTITY_KEY) + .update(encodeLengthPrefixed(domain)) + .update(encodeLengthPrefixed(payload)) + .digest("hex"); +} - return { - catalogPath: configuredCatalogPath - ? resolveActiveCodexConfigPath(configuredCatalogPath) - : activeDefaultCatalogPath(), - observation: { - state: "present", - role: "catalog-target-selection", - logicalPath, - canonicalPath, - parentIdentity, - fileIdentity: catalogFilesystemIdentity(file), - sha256: createHash("sha256").update(bytes).digest("hex"), - }, - }; +function catalogConfigIdentity( + config: Readonly, + generation: ConfigGeneration, +): CatalogAdmissionSnapshot["configIdentity"] { + let referenceIdentity = configReferenceIdentities.get(config); + if (!referenceIdentity) { + nextConfigReferenceIdentity += 1; + referenceIdentity = keyedConfigIdentity("catalog-config-reference-v1", String(nextConfigReferenceIdentity)); + configReferenceIdentities.set(config, referenceIdentity); + } + return Object.freeze({ + referenceIdentity, + generation: Object.freeze({ ...generation }), + snapshotIdentity: keyedConfigIdentity("catalog-config-snapshot-v1", canonicalConfigEncoding(config)), + }); } /** @@ -148,39 +138,42 @@ function captureCatalogTargetSelection(): Readonly<{ export function captureCatalogAdmissionSnapshot( config: Readonly, ): CatalogAdmissionSnapshot { - const generation = readConfigGeneration(); + const generation = observeConfigGeneration(); if (generation.kind !== "ready") { throw new Error(`Cannot capture Codex catalog admission: config generation is ${generation.reason}.`); } - const targetSelection = captureCatalogTargetSelection(); - const catalogPath = targetSelection.catalogPath; + const evidenceSession = createCatalogGatherEvidenceSession(); + const homeSelection = captureAndSealCatalogHomeSelection(evidenceSession); + const configPath = join(homeSelection.canonicalCodexHome, "config.toml"); + acceptCatalogGatherSourcePath(evidenceSession, "catalog-target-selection", configPath); + const configBytes = readCatalogGatherSource(evidenceSession, "catalog-target-selection"); + const configuredCatalogPath = configBytes === null + ? null + : readRootTomlString(Buffer.from(configBytes).toString("utf8"), "model_catalog_json"); + const defaultCatalogPath = join(homeSelection.canonicalCodexHome, "opencodex-catalog.json"); + const catalogPath = configuredCatalogPath + ? resolve(homeSelection.canonicalCodexHome, configuredCatalogPath) + : defaultCatalogPath; const backupPaths = [ catalogBackupPathFor(catalogPath), - ...(isDefaultCatalogPath(catalogPath) ? [legacyCatalogBackupPath()] : []), + ...(samePath(catalogPath, defaultCatalogPath) ? [legacyCatalogBackupPath()] : []), ]; + const targets = { + catalog: captureCatalogGatherTargetIdentity(evidenceSession, catalogPath), + cache: captureCatalogGatherTargetIdentity( + evidenceSession, + join(homeSelection.canonicalCodexHome, "models_cache.json"), + ), + catalogBackups: backupPaths.map(path => captureCatalogGatherTargetIdentity(evidenceSession, path)), + }; + const sourceEvidence = sealCatalogGatherEvidenceSession(evidenceSession); return { config, generation: generation.generation, - targets: { - catalog: captureTargetIdentity(catalogPath), - cache: captureTargetIdentity(activeCodexModelsCachePath()), - catalogBackups: backupPaths.map(captureTargetIdentity), - }, - sourceEvidence: { - required: { - "catalog-target-selection": targetSelection.observation, - }, - conditional: { - "bundled-catalog-template": [], - "active-catalog-merge": [], - "hashed-backup-fallback": [], - "legacy-backup-fallback": [], - "models-cache-fallback": [], - "runtime-selection": [], - "provider-auth-selection": [], - }, - }, + configIdentity: catalogConfigIdentity(config, generation.generation), + targets, + sourceEvidence, }; } diff --git a/src/codex/catalog/bundled.ts b/src/codex/catalog/bundled.ts index 610380ca0..78afdb8b5 100644 --- a/src/codex/catalog/bundled.ts +++ b/src/codex/catalog/bundled.ts @@ -44,8 +44,16 @@ import type { EffortClampDiagnostic, ResolvedCodexRuntime, } from "../runtime"; +import type { + CatalogGatherEvidenceSession, + CatalogGatherReadableSourceRole, +} from "./filesystem-evidence"; export { isSpawnableCodexCandidate, codexExecInvocation } from "../exec-invocation"; +export type { + CatalogGatherEvidenceSession, + CatalogGatherReadableSourceRole, +} from "./filesystem-evidence"; export const BUNDLED_CATALOG_CACHE_MS = 60_000; @@ -294,22 +302,6 @@ export function loadBundledCodexCatalog(deps: BundledCatalogDeps = {}): Readonly return null; } -export type CatalogGatherReadableSourceRole = - | "active-catalog-merge" - | "hashed-backup-fallback" - | "legacy-backup-fallback" - | "models-cache-fallback" - | "runtime-selection"; - -/** - * Temporary structural seam for WP9's filesystem-evidence owner. - * The owner maps each closed role to its admitted path and records PRESENT or - * ABSENT before returning the exact bytes. This adapter never reads a path. - */ -export interface CatalogGatherEvidenceSession { - readSource(role: CatalogGatherReadableSourceRole): Uint8Array | null; -} - export type CatalogGatherProcessLocalObservation = | Readonly<{ state: "unused" }> | Readonly<{ state: "used"; epoch: number; valueIdentity: string }>; diff --git a/src/codex/catalog/filesystem-evidence.ts b/src/codex/catalog/filesystem-evidence.ts new file mode 100644 index 000000000..fc3ef0a2d --- /dev/null +++ b/src/codex/catalog/filesystem-evidence.ts @@ -0,0 +1,302 @@ +import { createHash } from "node:crypto"; +import { readFileSync, realpathSync, statSync } from "node:fs"; +import { basename, dirname, resolve } from "node:path"; + +import { expandUserPath } from "../../config"; +import type { + CatalogConditionalSourceObservations, + CatalogConditionalSourceRole, + CatalogFilesystemIdentity, + CatalogHomeSelectionObservation, + CatalogRequiredSourceObservations, + CatalogRequiredSourceRole, + CatalogSourceEvidence, + CatalogSourceObservation, + CatalogSourceRole, +} from "../convergence-types"; +import { defaultCodexHome } from "../home"; + +export type CatalogGatherReadableSourceRole = + | "active-catalog-merge" + | "hashed-backup-fallback" + | "legacy-backup-fallback" + | "models-cache-fallback" + | "runtime-selection"; + +/** The minimal observe-only source interface consumed by bundled catalog selection. */ +export interface CatalogGatherEvidenceSession { + readSource(role: CatalogGatherReadableSourceRole): Uint8Array | null; +} + +/** The exact observe-only auth input consumed by provider discovery. */ +export interface CatalogGatherProviderAuthEvidence { + readonly authStoreBuffer: Uint8Array | null; +} + +export type CatalogFilesystemEvidenceSession = CatalogGatherEvidenceSession + & CatalogGatherProviderAuthEvidence; + +const CONDITIONAL_SOURCE_ROLES = [ + "bundled-catalog-template", + "active-catalog-merge", + "hashed-backup-fallback", + "legacy-backup-fallback", + "models-cache-fallback", + "native-catalog-selection", + "runtime-selection", + "provider-auth-selection", +] as const satisfies readonly CatalogConditionalSourceRole[]; + +interface MutableSessionState { + homeSelection: CatalogHomeSelectionObservation | null; + readonly sourcePaths: Map; + requiredTargetSelection: CatalogSourceObservation | null; + readonly conditional: Record; + sealed: boolean; +} + +const sessionStates = new WeakMap(); + +function filesystemIdentity( + entry: Readonly<{ dev: bigint; ino: bigint }>, +): CatalogFilesystemIdentity { + return Object.freeze({ volume: String(entry.dev), fileId: String(entry.ino) }); +} + +function sessionState(session: CatalogFilesystemEvidenceSession): MutableSessionState { + const state = sessionStates.get(session); + if (!state) { + throw new TypeError("Catalog filesystem evidence session was not created by its owner."); + } + return state; +} + +function assertOpen(state: MutableSessionState): void { + if (state.sealed) throw new Error("Catalog filesystem evidence session is already sealed."); +} + +function observedHomeSelection(): CatalogHomeSelectionObservation { + const environmentSelector = process.env.CODEX_HOME?.trim(); + const selector = environmentSelector + ? { kind: "environment" as const, raw: environmentSelector } + : { kind: "default" as const, raw: defaultCodexHome() }; + const selectedPath = selector.kind === "environment" + ? resolve(expandUserPath(selector.raw)) + : resolve(selector.raw); + const canonicalCodexHome = realpathSync.native(selectedPath); + const root = statSync(canonicalCodexHome, { bigint: true }); + if (!root.isDirectory()) { + throw new Error(`Codex home is not a directory: ${selectedPath}`); + } + return Object.freeze({ + selector: Object.freeze(selector), + canonicalCodexHome, + rootIdentity: filesystemIdentity(root), + }); +} + +/** Capture and freeze the selector/root authority before any derived path is admitted. */ +export function captureAndSealCatalogHomeSelection( + session: CatalogFilesystemEvidenceSession, +): CatalogHomeSelectionObservation { + const state = sessionState(session); + assertOpen(state); + if (state.homeSelection) return state.homeSelection; + state.homeSelection = observedHomeSelection(); + return state.homeSelection; +} + +/** Bind the next consultation for a closed role without exposing the evidence arrays. */ +export function acceptCatalogGatherSourcePath( + session: CatalogFilesystemEvidenceSession, + role: CatalogSourceRole, + logicalPath: string, +): void { + const state = sessionState(session); + assertOpen(state); + if (!state.homeSelection) { + throw new Error("Catalog home selection must be sealed before accepting a derived path."); + } + state.sourcePaths.set(role, resolve(logicalPath)); +} + +function isMissing(error: unknown): boolean { + return Boolean(error && typeof error === "object" && "code" in error && error.code === "ENOENT"); +} + +function observeSource( + role: R, + logicalPath: string, +): Readonly<{ bytes: Uint8Array | null; observation: CatalogSourceObservation }> { + const canonicalParent = realpathSync.native(dirname(logicalPath)); + const parent = statSync(canonicalParent, { bigint: true }); + if (!parent.isDirectory()) { + throw new Error(`Catalog source parent is not a directory: ${dirname(logicalPath)}`); + } + const parentIdentity = Object.freeze({ + canonicalPath: canonicalParent, + ...filesystemIdentity(parent), + }); + + let bytes: Uint8Array; + try { + bytes = Uint8Array.from(readFileSync(logicalPath)); + } catch (error) { + if (!isMissing(error)) throw error; + return Object.freeze({ + bytes: null, + observation: Object.freeze({ + state: "absent" as const, + role, + logicalPath, + canonicalPath: resolve(canonicalParent, basename(logicalPath)), + parentIdentity, + fileIdentity: null, + }), + }); + } + + const canonicalPath = realpathSync.native(logicalPath); + const file = statSync(canonicalPath, { bigint: true }); + if (!file.isFile()) throw new Error(`Catalog source is not a regular file: ${logicalPath}`); + return Object.freeze({ + bytes, + observation: Object.freeze({ + state: "present" as const, + role, + logicalPath, + canonicalPath, + parentIdentity, + fileIdentity: filesystemIdentity(file), + sha256: createHash("sha256").update(bytes).digest("hex"), + }), + }); +} + +/** Read once, record PRESENT/ABSENT first, then return the exact hashed buffer. */ +export function readCatalogGatherSource( + session: CatalogFilesystemEvidenceSession, + role: R, +): Uint8Array | null { + const state = sessionState(session); + assertOpen(state); + const logicalPath = state.sourcePaths.get(role); + if (!logicalPath) throw new Error(`No catalog source path was accepted for role ${role}.`); + if (role === "catalog-target-selection" && state.requiredTargetSelection) { + throw new Error(`Required catalog source role ${role} was already observed.`); + } + + const observed = observeSource(role, logicalPath); + if (role === "catalog-target-selection") { + state.requiredTargetSelection = observed.observation as CatalogSourceObservation<"catalog-target-selection">; + } else { + state.conditional[role as CatalogConditionalSourceRole].push(observed.observation); + } + return observed.bytes; +} + +function optionalFileIdentity(path: string): Readonly<{ device: string; inode: string }> | null { + try { + const entry = statSync(path, { bigint: true }); + return { device: String(entry.dev), inode: String(entry.ino) }; + } catch (error) { + if (isMissing(error)) return null; + throw error; + } +} + +/** Probe a candidate write target without allowing filesystem mutation. */ +export function captureCatalogGatherTargetIdentity( + session: CatalogFilesystemEvidenceSession, + path: string, +): string { + const state = sessionState(session); + assertOpen(state); + if (!state.homeSelection) { + throw new Error("Catalog home selection must be sealed before accepting a derived path."); + } + const textualPath = resolve(path); + const canonicalParent = realpathSync.native(dirname(textualPath)); + const parent = statSync(canonicalParent, { bigint: true }); + return JSON.stringify({ + path: textualPath, + canonicalParent, + parentIdentity: { device: String(parent.dev), inode: String(parent.ino) }, + fileIdentity: optionalFileIdentity(textualPath), + }); +} + +function cloneObservation( + observation: CatalogSourceObservation, +): CatalogSourceObservation { + if (observation.state === "absent") { + return Object.freeze({ + ...observation, + parentIdentity: Object.freeze({ ...observation.parentIdentity }), + fileIdentity: null, + }); + } + return Object.freeze({ + ...observation, + parentIdentity: Object.freeze({ ...observation.parentIdentity }), + fileIdentity: Object.freeze({ ...observation.fileIdentity }), + }); +} + +/** Seal one complete owner-created session into detached, recursively frozen evidence. */ +export function sealCatalogGatherEvidenceSession( + session: CatalogFilesystemEvidenceSession, +): CatalogSourceEvidence { + const state = sessionState(session); + assertOpen(state); + if (!state.homeSelection) throw new Error("Catalog home selection is required before sealing."); + const targetSelection = state.requiredTargetSelection; + if (!targetSelection) { + throw new Error("Required catalog source role catalog-target-selection was not observed."); + } + for (const role of CONDITIONAL_SOURCE_ROLES) { + if (!Object.hasOwn(state.conditional, role)) { + throw new Error(`Conditional catalog source role ${role} is missing.`); + } + } + + const conditional = Object.fromEntries(CONDITIONAL_SOURCE_ROLES.map(role => [ + role, + Object.freeze(state.conditional[role].map(observation => cloneObservation(observation))), + ])) as unknown as CatalogConditionalSourceObservations; + const required = Object.freeze({ + "catalog-target-selection": cloneObservation(targetSelection), + }) as CatalogRequiredSourceObservations; + const evidence = Object.freeze({ + homeSelection: Object.freeze({ + ...state.homeSelection, + selector: Object.freeze({ ...state.homeSelection.selector }), + rootIdentity: Object.freeze({ ...state.homeSelection.rootIdentity }), + }), + required, + conditional: Object.freeze(conditional), + }); + state.sealed = true; + return evidence; +} + +/** Create an opaque session; all mutable evidence state remains module-private. */ +export function createCatalogGatherEvidenceSession(): CatalogFilesystemEvidenceSession { + let session: CatalogFilesystemEvidenceSession; + session = Object.freeze({ + readSource: (role: CatalogGatherReadableSourceRole) => readCatalogGatherSource(session, role), + get authStoreBuffer(): Uint8Array | null { + return readCatalogGatherSource(session, "provider-auth-selection"); + }, + }); + sessionStates.set(session, { + homeSelection: null, + sourcePaths: new Map(), + requiredTargetSelection: null, + conditional: Object.fromEntries( + CONDITIONAL_SOURCE_ROLES.map(role => [role, [] as CatalogSourceObservation[]]), + ) as unknown as MutableSessionState["conditional"], + sealed: false, + }); + return session; +} diff --git a/src/codex/catalog/provider-fetch.ts b/src/codex/catalog/provider-fetch.ts index f22e56029..e3ecfa86f 100644 --- a/src/codex/catalog/provider-fetch.ts +++ b/src/codex/catalog/provider-fetch.ts @@ -66,14 +66,12 @@ import type { CatalogModel } from "./parsing"; import { disabledNativeSlugs, hasComboTargets, nativeInputModalities, nativeOpenAiContextWindow, nativeOpenAiSlugs, nativeParallelToolCalls, nativeReasoningEfforts } from "./metadata"; import { deriveComboCatalogModel, normalizedOpenAiApiSignature, openAiApiCollisionWarnings, replaceLastComboCatalogOmissions, warnUncataloguedComboOnce } from "./aggregation"; import type { ComboCatalogOmission } from "./aggregation"; +import type { CatalogGatherProviderAuthEvidence } from "./filesystem-evidence"; + +export type { CatalogGatherProviderAuthEvidence } from "./filesystem-evidence"; /** Concurrent gatherRoutedModels callers with the same catalog identity share one live discovery. * Keyed by gatherFlightKey so a different config cannot join or evict the wrong flight. */ -export interface CatalogGatherProviderAuthEvidence { - /** Exact auth-store bytes already read and recorded by the filesystem-evidence owner. */ - readonly authStoreBuffer: Uint8Array | null; -} - export interface CatalogGatherProviderAuthOutcome { readonly provider: string; readonly state: OAuthActiveTokenObservation["kind"]; diff --git a/src/codex/convergence-types.ts b/src/codex/convergence-types.ts index 1301f317f..964b46459 100644 --- a/src/codex/convergence-types.ts +++ b/src/codex/convergence-types.ts @@ -350,6 +350,7 @@ export type CatalogConditionalSourceRole = | "hashed-backup-fallback" | "legacy-backup-fallback" | "models-cache-fallback" + | "native-catalog-selection" | "runtime-selection" | "provider-auth-selection"; @@ -367,6 +368,45 @@ export interface CatalogParentIdentity extends CatalogFilesystemIdentity { readonly canonicalPath: string; } +/** Required evidence for the selector that chose every CODEX_HOME-derived path. */ +export interface CatalogHomeSelectionObservation { + readonly selector: Readonly<{ + readonly kind: "environment" | "default"; + /** Exact pre-canonicalization selector string used by the production resolver. */ + readonly raw: string; + }>; + readonly canonicalCodexHome: string; + readonly rootIdentity: CatalogFilesystemIdentity; +} + +export type CatalogProcessLocalObservation = + | { readonly state: "unused" } + | { readonly state: "used"; readonly epoch: number; readonly valueIdentity: string }; + +/** Candidate-bound evidence for mutable process-local authority, never file evidence. */ +export interface CatalogProcessLocalEvidence { + readonly runtime: CatalogProcessLocalObservation; + readonly bundledCatalog: CatalogProcessLocalObservation; +} + +/** Non-secret-bearing identity of every authority input admitted to one gather flight. */ +export interface CatalogGatherAuthorityIdentity { + readonly version: 1; + /** Process-local keyed HMAC over every component below; never a raw content hash. */ + readonly authorityId: string; + readonly admittedConfig: Readonly<{ + /** Opaque WeakMap identity of the exact resident Readonly reference. */ + readonly referenceIdentity: string; + readonly generation: ConfigGeneration; + /** Keyed HMAC of the exact canonical config snapshot, including secret-bearing fields. */ + readonly snapshotIdentity: string; + }>; + readonly authSnapshotIdentity: string; + readonly nativeCatalogSourceIdentity: string; + readonly sourceEvidenceIdentity: string; + readonly processLocalEvidenceIdentity: string; +} + /** Exact gather-time evidence for one consulted filesystem source. */ export type CatalogSourceObservation = | { @@ -397,6 +437,8 @@ export type CatalogConditionalSourceObservations = Readonly<{ }>; export interface CatalogSourceEvidence { + /** Required before any CODEX_HOME-derived target or source path is accepted. */ + readonly homeSelection: CatalogHomeSelectionObservation; readonly required: CatalogRequiredSourceObservations; /** Every role is a required key; an empty list means the role was not consulted. */ readonly conditional: CatalogConditionalSourceObservations; @@ -406,6 +448,8 @@ export interface CatalogSourceEvidence { export interface CatalogAdmissionSnapshot { config: Readonly; generation: ConfigGeneration; + /** Exact retained-reference/generation/snapshot identity used by gather authority. */ + readonly configIdentity: CatalogGatherAuthorityIdentity["admittedConfig"]; targets: Readonly<{ catalog: string; cache: string; diff --git a/tests/codex-catalog-admission.test.ts b/tests/codex-catalog-admission.test.ts index 8d955bb0d..19c729f5a 100644 --- a/tests/codex-catalog-admission.test.ts +++ b/tests/codex-catalog-admission.test.ts @@ -29,6 +29,7 @@ const CONDITIONAL_SOURCE_ROLES = [ "hashed-backup-fallback", "legacy-backup-fallback", "models-cache-fallback", + "native-catalog-selection", "provider-auth-selection", "runtime-selection", ] as const satisfies readonly CatalogConditionalSourceRole[]; @@ -40,11 +41,13 @@ type MissingRequiredEvidence = Omit & Readonl type MissingConditionalEvidence = Omit & Readonly<{ conditional: Omit; }>; +type MissingHomeEvidence = Omit; const STRUCTURALLY_INVALID_EVIDENCE_ASSIGNABILITY: readonly [ IsAssignable, IsAssignable, -] = [false, false]; + IsAssignable, +] = [false, false, false]; let testRoot = ""; let codexHome = ""; @@ -87,6 +90,11 @@ test("captures the given config reference, generation, and catalog target identi expect(snapshot.config).toBe(residentConfig); expect(snapshot.config.port).toBe(30300); expect(snapshot.generation).toEqual({ value: 1 }); + expect(snapshot.configIdentity).toEqual({ + referenceIdentity: expect.any(String), + generation: { value: 1 }, + snapshotIdentity: expect.any(String), + }); expect(JSON.parse(snapshot.targets.catalog)).toMatchObject({ path: join(codexHome, "opencodex-catalog.json"), canonicalParent: codexHome, @@ -112,6 +120,11 @@ test("captures the given config reference, generation, and catalog target identi }, fileIdentity: null, }); + expect(snapshot.sourceEvidence.homeSelection).toEqual({ + selector: { kind: "environment", raw: codexHome }, + canonicalCodexHome: codexHome, + rootIdentity: { volume: expect.any(String), fileId: expect.any(String) }, + }); expect(Object.keys(snapshot.sourceEvidence.conditional).sort()).toEqual(CONDITIONAL_SOURCE_ROLES); for (const observations of Object.values(snapshot.sourceEvidence.conditional)) { expect(observations).toEqual([]); @@ -119,6 +132,7 @@ test("captures the given config reference, generation, and catalog target identi }); test("captures PRESENT catalog target-selection evidence from the exact config bytes", () => { + saveConfig(config()); const selectedCatalog = join(codexHome, "selected-catalog.json"); const configBytes = Buffer.from( `model_catalog_json = ${JSON.stringify(selectedCatalog)}\n`, @@ -151,11 +165,31 @@ test("captures PRESENT catalog target-selection evidence from the exact config b }); }); -test("rejects missing required and conditional evidence keys structurally", () => { - expect(STRUCTURALLY_INVALID_EVIDENCE_ASSIGNABILITY).toEqual([false, false]); +test("binds opaque config identity to the exact reference, generation, and snapshot", () => { + saveConfig(config()); + const firstConfig = config(20200); + const equalButDistinctConfig = config(20200); + + const first = captureCatalogAdmissionSnapshot(firstConfig).configIdentity; + const sameReference = captureCatalogAdmissionSnapshot(firstConfig).configIdentity; + const distinctReference = captureCatalogAdmissionSnapshot(equalButDistinctConfig).configIdentity; + firstConfig.port = 30300; + const mutated = captureCatalogAdmissionSnapshot(firstConfig).configIdentity; + + expect(sameReference).toEqual(first); + expect(distinctReference.referenceIdentity).not.toBe(first.referenceIdentity); + expect(distinctReference.snapshotIdentity).toBe(first.snapshotIdentity); + expect(mutated.referenceIdentity).toBe(first.referenceIdentity); + expect(mutated.snapshotIdentity).not.toBe(first.snapshotIdentity); + expect(mutated.generation).toEqual(first.generation); +}); + +test("rejects missing home, required, and conditional evidence keys structurally", () => { + expect(STRUCTURALLY_INVALID_EVIDENCE_ASSIGNABILITY).toEqual([false, false, false]); }); test("changes target identity when a parent symlink retargets without changing the path", () => { + saveConfig(config()); const parentA = join(testRoot, "catalog-parent-a"); const parentB = join(testRoot, "catalog-parent-b"); const linkedParent = join(testRoot, "catalog-parent"); diff --git a/tests/codex-filesystem-evidence.test.ts b/tests/codex-filesystem-evidence.test.ts new file mode 100644 index 000000000..e36f71c4d --- /dev/null +++ b/tests/codex-filesystem-evidence.test.ts @@ -0,0 +1,242 @@ +import { afterEach, beforeEach, expect, test } from "bun:test"; +import { createHash } from "node:crypto"; +import { + lstatSync, + mkdirSync, + mkdtempSync, + readdirSync, + readFileSync, + readlinkSync, + realpathSync, + rmSync, + writeFileSync, +} from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +import { captureCatalogAdmissionSnapshot } from "../src/codex/catalog-admission"; +import { + resetBundledCatalogCacheForTests, + resolveCatalogSourceForGather, +} from "../src/codex/catalog/bundled"; +import { + acceptCatalogGatherSourcePath, + captureAndSealCatalogHomeSelection, + captureCatalogGatherTargetIdentity, + createCatalogGatherEvidenceSession, + readCatalogGatherSource, + sealCatalogGatherEvidenceSession, + type CatalogGatherProviderAuthEvidence, +} from "../src/codex/catalog/filesystem-evidence"; +import type { CatalogSourceEvidence } from "../src/codex/convergence-types"; +import { saveConfig } from "../src/config"; +import type { OcxConfig } from "../src/types"; + +interface ManifestEntry { + readonly path: string; + readonly kind: "directory" | "file" | "symlink" | "other"; + readonly mode: number; + readonly mtimeNs: string; + readonly inode: string; + readonly device: string; + readonly size: string; + readonly sha256?: string; + readonly linkTarget?: string; +} + +let testRoot = ""; +let codexHome = ""; +let opencodexHome = ""; +let previousCodexHome: string | undefined; +let previousOpencodexHome: string | undefined; + +function config(): OcxConfig { + return { port: 10100, providers: {}, defaultProvider: "openai" }; +} + +function recursiveManifest(root: string): readonly ManifestEntry[] { + const entries: ManifestEntry[] = []; + const visit = (path: string, relativePath: string): void => { + const stat = lstatSync(path, { bigint: true }); + const kind = stat.isDirectory() + ? "directory" + : stat.isFile() + ? "file" + : stat.isSymbolicLink() + ? "symlink" + : "other"; + entries.push({ + path: relativePath, + kind, + mode: Number(stat.mode), + mtimeNs: String(stat.mtimeNs), + inode: String(stat.ino), + device: String(stat.dev), + size: String(stat.size), + ...(kind === "file" + ? { sha256: createHash("sha256").update(readFileSync(path)).digest("hex") } + : {}), + ...(kind === "symlink" ? { linkTarget: readlinkSync(path) } : {}), + }); + if (kind !== "directory") return; + for (const child of readdirSync(path).sort()) { + visit(join(path, child), relativePath === "." ? child : join(relativePath, child)); + } + }; + visit(root, "."); + return entries; +} + +beforeEach(() => { + previousCodexHome = process.env.CODEX_HOME; + previousOpencodexHome = process.env.OPENCODEX_HOME; + testRoot = realpathSync.native(mkdtempSync(join(tmpdir(), "ocx-filesystem-evidence-"))); + codexHome = join(testRoot, "codex-home"); + opencodexHome = join(testRoot, "opencodex-home"); + mkdirSync(codexHome, { recursive: true }); + mkdirSync(opencodexHome, { recursive: true }); + process.env.CODEX_HOME = codexHome; + process.env.OPENCODEX_HOME = opencodexHome; +}); + +afterEach(() => { + if (previousCodexHome === undefined) delete process.env.CODEX_HOME; + else process.env.CODEX_HOME = previousCodexHome; + if (previousOpencodexHome === undefined) delete process.env.OPENCODEX_HOME; + else process.env.OPENCODEX_HOME = previousOpencodexHome; + rmSync(testRoot, { recursive: true, force: true }); +}); + +test("source reads record PRESENT and ABSENT before returning their result", () => { + const presentPath = join(codexHome, "present.json"); + const absentPath = join(codexHome, "absent.json"); + const presentBytes = Buffer.from("present bytes\n"); + writeFileSync(presentPath, presentBytes); + + const session = createCatalogGatherEvidenceSession(); + captureAndSealCatalogHomeSelection(session); + acceptCatalogGatherSourcePath(session, "catalog-target-selection", join(codexHome, "config.toml")); + expect(readCatalogGatherSource(session, "catalog-target-selection")).toBeNull(); + acceptCatalogGatherSourcePath(session, "active-catalog-merge", presentPath); + expect(readCatalogGatherSource(session, "active-catalog-merge")).toEqual(Uint8Array.from(presentBytes)); + acceptCatalogGatherSourcePath(session, "models-cache-fallback", absentPath); + expect(readCatalogGatherSource(session, "models-cache-fallback")).toBeNull(); + + const evidence = sealCatalogGatherEvidenceSession(session); + expect(evidence.conditional["active-catalog-merge"]).toEqual([{ + state: "present", + role: "active-catalog-merge", + logicalPath: presentPath, + canonicalPath: presentPath, + parentIdentity: { + canonicalPath: codexHome, + volume: expect.any(String), + fileId: expect.any(String), + }, + fileIdentity: { volume: expect.any(String), fileId: expect.any(String) }, + sha256: createHash("sha256").update(presentBytes).digest("hex"), + }]); + expect(evidence.conditional["models-cache-fallback"]).toEqual([{ + state: "absent", + role: "models-cache-fallback", + logicalPath: absentPath, + canonicalPath: absentPath, + parentIdentity: { + canonicalPath: codexHome, + volume: expect.any(String), + fileId: expect.any(String), + }, + fileIdentity: null, + }]); +}); + +test("refuses sealing without the required catalog-target-selection observation", () => { + const session = createCatalogGatherEvidenceSession(); + captureAndSealCatalogHomeSelection(session); + expect(() => sealCatalogGatherEvidenceSession(session)).toThrow("catalog-target-selection"); +}); + +test("sealed evidence is detached and recursively immutable", () => { + const session = createCatalogGatherEvidenceSession(); + captureAndSealCatalogHomeSelection(session); + acceptCatalogGatherSourcePath(session, "catalog-target-selection", join(codexHome, "config.toml")); + readCatalogGatherSource(session, "catalog-target-selection"); + acceptCatalogGatherSourcePath(session, "active-catalog-merge", join(codexHome, "missing.json")); + readCatalogGatherSource(session, "active-catalog-merge"); + const evidence = sealCatalogGatherEvidenceSession(session); + + expect(Object.isFrozen(evidence)).toBe(true); + expect(Object.isFrozen(evidence.conditional)).toBe(true); + expect(Object.isFrozen(evidence.conditional["active-catalog-merge"])).toBe(true); + expect(() => { + (evidence.conditional["active-catalog-merge"] as CatalogSourceEvidence["conditional"]["active-catalog-merge"] & unknown[]) + .push(evidence.conditional["active-catalog-merge"][0]!); + }).toThrow(); + expect(() => { + (evidence.required["catalog-target-selection"].parentIdentity as { canonicalPath: string }).canonicalPath = "mutated"; + }).toThrow(); + expect(evidence.conditional["active-catalog-merge"]).toHaveLength(1); + expect(evidence.required["catalog-target-selection"].parentIdentity.canonicalPath).toBe(codexHome); +}); + +test("refuses a derived path before catalog-home selection is sealed", () => { + const session = createCatalogGatherEvidenceSession(); + expect(() => acceptCatalogGatherSourcePath( + session, + "catalog-target-selection", + join(codexHome, "config.toml"), + )).toThrow("before accepting a derived path"); + expect(() => captureCatalogGatherTargetIdentity(session, join(codexHome, "catalog.json"))) + .toThrow("before accepting a derived path"); +}); + +test("admission observation does not create missing generation storage", () => { + const before = recursiveManifest(testRoot); + expect(() => captureCatalogAdmissionSnapshot(config())).toThrow("generation is database"); + expect(recursiveManifest(testRoot)).toEqual(before); +}); + +test("populated and scratch gather sessions perform zero filesystem writes", () => { + saveConfig(config()); + const configPath = join(codexHome, "config.toml"); + const activePath = join(codexHome, "active.json"); + const authPath = join(opencodexHome, "oauth.json"); + writeFileSync(configPath, "model_catalog_json = \"active.json\"\n"); + writeFileSync(activePath, JSON.stringify({ + models: [{ slug: "gpt-5.5", base_instructions: "observed" }], + })); + writeFileSync(authPath, "{}\n"); + const beforePopulated = recursiveManifest(testRoot); + + resetBundledCatalogCacheForTests(); + const populated = createCatalogGatherEvidenceSession(); + captureAndSealCatalogHomeSelection(populated); + acceptCatalogGatherSourcePath(populated, "catalog-target-selection", configPath); + readCatalogGatherSource(populated, "catalog-target-selection"); + acceptCatalogGatherSourcePath(populated, "active-catalog-merge", activePath); + expect(resolveCatalogSourceForGather(populated).kind).toBe("available"); + acceptCatalogGatherSourcePath(populated, "models-cache-fallback", join(codexHome, "missing-cache.json")); + populated.readSource("models-cache-fallback"); + acceptCatalogGatherSourcePath(populated, "provider-auth-selection", authPath); + const providerEvidence: CatalogGatherProviderAuthEvidence = populated; + expect(providerEvidence.authStoreBuffer).toEqual(Uint8Array.from(readFileSync(authPath))); + captureCatalogGatherTargetIdentity(populated, activePath); + sealCatalogGatherEvidenceSession(populated); + expect(recursiveManifest(testRoot)).toEqual(beforePopulated); + + const scratchHome = join(testRoot, "scratch-home"); + mkdirSync(scratchHome); + process.env.CODEX_HOME = scratchHome; + const beforeScratch = recursiveManifest(testRoot); + const scratch = createCatalogGatherEvidenceSession(); + captureAndSealCatalogHomeSelection(scratch); + acceptCatalogGatherSourcePath(scratch, "catalog-target-selection", join(scratchHome, "config.toml")); + readCatalogGatherSource(scratch, "catalog-target-selection"); + sealCatalogGatherEvidenceSession(scratch); + expect(recursiveManifest(testRoot)).toEqual(beforeScratch); + + process.env.CODEX_HOME = join(testRoot, "absent-home"); + const beforeAbsent = recursiveManifest(testRoot); + expect(() => captureAndSealCatalogHomeSelection(createCatalogGatherEvidenceSession())).toThrow(); + expect(recursiveManifest(testRoot)).toEqual(beforeAbsent); +}); From 9f7e1a30a32296967cf0074ced0d5fa6e3472d33 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Tue, 4 Aug 2026 22:08:20 +0900 Subject: [PATCH 070/163] docs(substrate): the third cache with the same defect, and the owner to fix it in MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Round 6 found the design asking for something it never authorized anyone to build. It required a per-provider immutable model-cache and cooldown snapshot with a monotonic owner epoch, while `src/codex/model-cache.ts` sat outside WP9's file scope — so the mechanism had no owner and could only have been faked in a consumer. The owner has the defect this unit has now fixed twice elsewhere. `getFreshCached` returns the private array itself (model-cache.ts:147-150), `getStaleCached` does the same, `setCached` retains the caller's alias (:158), and no epoch covers failure and cooldown (:85), clear (:172), reconciliation (:189), or budget eviction (:225). The reviewer mutated a result through `getFreshCached` and read the mutation back from the owner on the next call, with no assignment and no invalidation anywhere in between. Same shape as the runtime and bundled memos: an absence of owner-observed mutation treated as proof the authority held still. Third instance, sixth overall. So the module joins WP9 rather than the evidence being duplicated in `provider-fetch.ts`, which would have created a second owner of one surface — the exact failure the contract phase exists to prevent. Stored models become private deep-frozen clones read out as detached readonly snapshots, every result-affecting mutation advances the epoch, and the freshness/cooldown DECISION is sealed before flight lookup so a flight cannot re-read cache state after claiming its slot. That ordering is the actual race; a snapshot taken afterwards would describe a world the flight had already left. The privacy scanner passes the current plain-digest implementation, so the plan keeps behavioral sink assertions as the proof and now says why the scanner is not one. A search for a fourth result-affecting process-local cache found none: the remaining maps are flight coordination, log suppression, management status and output copies, none of which select models. --- .../005_contract.md | 124 ++++++++++++- .../010_catalog_seam.md | 173 ++++++++++++++---- 2 files changed, 250 insertions(+), 47 deletions(-) diff --git a/devlog/_plan/260804_codex_write_substrate/005_contract.md b/devlog/_plan/260804_codex_write_substrate/005_contract.md index a967c9ead..b6398af41 100644 --- a/devlog/_plan/260804_codex_write_substrate/005_contract.md +++ b/devlog/_plan/260804_codex_write_substrate/005_contract.md @@ -55,7 +55,7 @@ containing different fields, so a record from either is malformed to the other **TypeScript compile prelude.** The TypeScript fences in this document are concatenated contract fragments. Compile them in document order after prepending `import type { OcxConfig } from "../types";`. `OcxConfig` is the real export used -by `src/config.ts:34`; omitting this prelude gives TS2304 even though the contract +by `src/config.ts:48`; omitting this prelude gives TS2304 even though the contract itself is otherwise valid. ```ts @@ -832,7 +832,7 @@ a deadline, not on hope (round 1 #5's missing termination rule). A candidate records the canonical parent directory and the file identity (dev+inode where available) of each target, not the textual path — a parent symlink can retarget while the path string is unchanged, and `atomicWriteFile` -resolves the effective target only at commit (`src/config.ts:199-209`). +resolves the effective target only at commit (`src/config.ts:192-213`). The WP9 seam auditor then demonstrated the missing content dimension by gathering a candidate, truncating and rewriting the catalog in place, and committing the @@ -893,7 +893,7 @@ returned `resolveCache.value` and `bundledCatalogCache.value` directly, so a cal could change authority without owner assignment, invalidation, or epoch movement. The current worktree now exposes recursively readonly runtime shapes and clones plus deep-freezes cache publication/reads (`src/codex/runtime.ts:16-48,90-105,417-470,505-516`), -and the bundled owner does the same (`src/codex/catalog/bundled.ts:52-130,260-283`). +and the bundled owner does the same (`src/codex/catalog/bundled.ts:60-138,268-302`). The contract requires that landed behavior; it must not regress to the audited alias. The runtime and bundled-cache owners must instead clone incoming values into private @@ -946,18 +946,18 @@ into a guarantee the filesystem cannot provide. Round 5 reproduced the same absence-as-equivalence defect before candidate construction. `providerCatalogFingerprint` covers endpoint and catalog fields but omits `authMode`, `apiKey`, and `headers` -(`src/codex/catalog/provider-fetch.ts:136-158`). `gatherFlightKey` hashes that partial -projection (`src/codex/catalog/provider-fetch.ts:161-180`), even though +(`src/codex/catalog/provider-fetch.ts:134-156`). `gatherFlightKey` hashes that partial +projection (`src/codex/catalog/provider-fetch.ts:159-179`), even though `fetchProviderModels` branches on `authMode`, resolves credentials, and builds the -effective discovery request (`src/codex/catalog/provider-fetch.ts:474-500,548-568`). +effective discovery request (`src/codex/catalog/provider-fetch.ts:472-499,546-566`). The map lookup then lets the second caller join the first promise solely by that key -(`src/codex/catalog/provider-fetch.ts:792-820`). A generation-N forward-auth gather +(`src/codex/catalog/provider-fetch.ts:790-819`). A generation-N forward-auth gather can therefore supply empty bytes to a generation-N+1 key-auth admission; B's later K -> C validation is honest but irrelevant because no evidence says A produced the joined result. The current in-progress WP9 worktree prefixes the key with a plain SHA-256 of the -auth-store buffer (`src/codex/catalog/provider-fetch.ts:773-787`). That does not close +auth-store buffer (`src/codex/catalog/provider-fetch.ts:771-787`). That does not close the finding: static key/forward mode and configured headers still collide, the result still carries no authority, and a stable plain digest of credential-store bytes is the privacy trap this rule forbids. @@ -1031,11 +1031,64 @@ identity. Inequality discards the result and returns retryable `stale` or regath within `deadlineMs`; it never builds or commits a candidate. This result check is required defense in depth even though the map key should already prevent the join. +### The provider model-cache decision is captured before flight lookup (round 6) + +Round 6 found that component 5 named authority which WP9 did not own. The current +owner stores mutable `CatalogModel[]` values and returns the private array from both +fresh and stale reads (`src/codex/model-cache.ts:17-21,147-155`), while `setCached` +retains the caller's array alias (`src/codex/model-cache.ts:158-168`). It also has no +epoch around cooldown mutation, clear, reconciliation, or budget eviction +(`src/codex/model-cache.ts:74-86,172-208,225-226`). A caller can therefore mutate a +nested cached model in place and change the next gather without any owner-observed +assignment or identity movement. This is the same defect already closed for the +runtime and bundled owners, not evidence that mutable aliases are safe here. + +`src/codex/model-cache.ts` is consequently **IN for WP9**. Duplicating an epoch or a +snapshot in `provider-fetch.ts` is rejected: it would let the consumer attest to a +copy while the canonical cache/cooldown owner continued to change invisibly. The +owner stores only private recursively deep-frozen clones; `setCached` never retains +the caller's array or any nested object/array. `getFreshCached`, `getStaleCached`, and +the flight-specific decision reader return detached recursively deep-frozen, +recursively readonly snapshots, never the private graph. Shallow array cloning or +top-level `Object.freeze` does not satisfy this rule. + +The owner maintains one process-lifetime monotonic epoch. Every result-affecting +owner mutation advances it before the replacement is observable: every `setCached` +publication even when the replacement is byte-identical, every +`markModelsFetchFailure` cooldown change, every state-changing `clearModelCache`, +every accepted reconciliation generation (including its provider removals), and +every successful budget eviction. The epoch is never reset or reused. Discovery +status and warning-suppression bookkeeping stay outside this identity only while no +catalog result reads them; if a future gather branch consumes one, that field and +all of its mutations enter this same owner snapshot before that branch lands. + +After config, effective auth, native-source, and runtime/bundled inputs are captured, +but **before the first `gatherInflight` lookup**, gather calls the model-cache owner +synchronously once with the ordered enabled-provider set, the admitted TTL/cooldown +durations, and one clock observation. For every provider the owner returns a detached, +deep-frozen closed decision snapshot: `unused`, `fresh-cache`, `cooldown`, or +`network`; the used variants carry the exact detached fresh/stale model value or +explicit absence, owner epoch, `fetchedAt`/`failureAt` where present, and the exact +`freshUntil`/`cooldownUntil` boundary that produced the decision. The +gather-authority owner computes the process-keyed value identity over that exact +snapshot and includes the ordered tuple in `modelCacheDecisionIdentity`. + +The keyed flight receives those decision snapshots as arguments. After claiming its +slot it may not call `getFreshCached`, `getStaleCached`, or +`isModelsFetchCoolingDown`, and a network-failure fallback uses the stale value sealed +before lookup rather than re-reading the owner. The flight may publish success or +failure through the owner's ordinary mutation APIs; those writes advance the epoch, +which prevents a later caller from joining on the superseded decision. Time passage +does not fake an epoch mutation: a caller captured after `freshUntil` or +`cooldownUntil` gets a different effective decision identity even when the owner +epoch and stored bytes are unchanged. Callers captured on the same side of the same +boundary may still share, preserving the intended thundering-herd suppression. + ### Create-once means no-clobber publication (seam audit round 2) Hashed and legacy catalog backups are immutable first-winner snapshots. The ordinary `atomicWriteFile` helper cannot publish them: its final rename replaces an existing -destination (`src/config.ts:209` in the audited tree). An absence check followed by +destination (`src/config.ts:213` in the audited tree). An absence check followed by that helper is a check-then-write race, not create-once. `src/codex/internal/catalog-writer.ts` therefore owns a synchronous atomic @@ -1108,6 +1161,21 @@ export interface CatalogProcessLocalEvidence { readonly bundledCatalog: CatalogProcessLocalObservation; } +/** Flight-only cache/cooldown authority, captured before any in-flight lookup. */ +export interface CatalogProviderModelCacheDecisionEvidence { + readonly provider: string; + readonly ownerEpoch: number; + /** Process-keyed HMAC of the exact detached owner snapshot and decision. */ + readonly valueIdentity: string; + readonly decision: + | Readonly<{ kind: "unused" }> + | Readonly<{ kind: "fresh-cache"; freshUntil: number }> + | Readonly<{ kind: "cooldown"; cooldownUntil: number; + stale: "present" | "absent" }> + | Readonly<{ kind: "network"; freshUntil: number | null; + cooldownUntil: number | null; stale: "present" | "absent" }>; +} + /** Non-secret-bearing identity of every authority input admitted to one gather flight. */ export interface CatalogGatherAuthorityIdentity { readonly version: 1; @@ -1124,6 +1192,8 @@ export interface CatalogGatherAuthorityIdentity { readonly nativeCatalogSourceIdentity: string; readonly sourceEvidenceIdentity: string; readonly processLocalEvidenceIdentity: string; + /** HMAC of the ordered per-provider decisions captured before flight lookup. */ + readonly modelCacheDecisionIdentity: string; } /** Exact gather-time evidence for one consulted filesystem source. */ @@ -1716,6 +1786,35 @@ candidate authority equals its own admission, while instrumented log/response/ serialization sinks receive neither the private identity nor the API key, OAuth token, configured secret header, or their plain SHA-256 values. +Add the round-6 provider-cache decision matrix using the real owner APIs and a fake +clock. Pause A after its complete authority, including provider decisions, is captured +but before its flight settles. Mutate a nested object/array through the original +`setCached` input and through both public cache readers; neither attempt may alter the +owner snapshot. Then publish byte-identical models, mark a fetch failure/cooldown, +clear the provider, reconcile it away, and evict it through the real memory-budget +hook, one case at a time. Every actual owner mutation must advance the monotonic epoch, +give B a different model-cache decision/value identity, and prevent B from joining or +accepting A. The named broken mutations are **retain the `setCached` input alias or +return the private nested graph**, **skip the epoch bump for byte-identical +`setCached`**, **skip the failure/cooldown bump**, **skip the clear bump**, **skip the +reconciliation bump**, and **skip the eviction bump**; each must turn its own row red. +The harness instruments every required mutation path so omitting any one bump cannot +be masked by another mutation in the same row. + +Without mutating the owner, capture A immediately before `freshUntil` and another A +immediately before `cooldownUntil`, cross exactly one boundary, and capture B. B must +receive `network` rather than `fresh-cache` or `cooldown` and must not join the +pre-boundary flight. The named broken mutation **key only the owner epoch/value while +re-reading TTL or cooldown after flight lookup** makes both boundary rows red. A flight +uses its sealed fresh/stale/absence decision and never calls the three cache/cooldown +readers after claiming its slot. + +These privacy cases are behavioral sink assertions. Round 6 verified that +`bun run privacy:scan` still passes against the currently broken plain credential- +store digest, so scanner success is supplemental hygiene and is never accepted as +proof that keys, results, logs, serialization, or responses omit raw credentials and +stable unkeyed digests. + Race two create-once backup publishers after both observed ABSENT. Exactly one no-clobber publication wins; the loser receives `EEXIST`, validates and preserves the winner, and neither ordinary rename nor `atomicWriteFile` is called. Repeat with @@ -1762,7 +1861,12 @@ writes to it. single-direction raw CODEX_HOME-selector/canonical-root retarget before writing. A shared provider flight is keyed by the complete non-secret-bearing `CatalogGatherAuthorityIdentity`, returns that producing identity, and cannot build - a candidate when it differs from the caller's admission. A runtime-influenced candidate always carries PRESENT-or-ABSENT + a candidate when it differs from the caller's admission. Its model-cache/cooldown + component is captured as an immutable, detached per-provider effective decision + before flight lookup; every result-affecting owner mutation advances a monotonic + epoch, and TTL/cooldown boundary passage changes the decision identity without + pretending time is an owner mutation. The flight consumes that sealed decision and + never re-reads cache/cooldown authority after claiming its slot. A runtime-influenced candidate always carries PRESENT-or-ABSENT `codex-runtime.json` evidence, and any used runtime/bundled process memo must retain its exact monotonic epoch and deeply immutable, non-aliased value identity through the commit check. diff --git a/devlog/_plan/260804_codex_write_substrate/010_catalog_seam.md b/devlog/_plan/260804_codex_write_substrate/010_catalog_seam.md index 9e18d5bdf..b5d563949 100644 --- a/devlog/_plan/260804_codex_write_substrate/010_catalog_seam.md +++ b/devlog/_plan/260804_codex_write_substrate/010_catalog_seam.md @@ -25,12 +25,12 @@ transition that was not published (`src/codex/transition-state.ts:74-83,314-344, All current-code citations and diff context below were rechecked on 2026-08-04 against the live worktree rooted at -`364496fc968fd12734a2e6fe2bec9e219d9d1c90`, including the concurrent in-progress -WP9 source edits. The contract citations refer to the authoritative worktree amendment: +`2becc771977afb112fc8db45ed878fb67625c1a5`, including the WP9 source edits now +present at that HEAD. The contract citations refer to the authoritative worktree amendment: permanent K, owner-held config generation, home/source/process-local evidence, and no-clobber publication are at -`005_contract.md:609-763,830-1280`; K's owner/path and the four -transitional chains are fixed at `005_contract.md:1372-1570`. +`005_contract.md:609-1107,1109-1350`; K's owner/path and the four +transitional chains are fixed at `005_contract.md:1442-1640`. ## IN / OUT @@ -41,7 +41,7 @@ IN — observe-only admission and gather: `config-mutation.sqlite`. The existing `readConfigGeneration` is not that API: it resolves/records the path and opens SQLite with `create:true` (`src/config.ts:1745-1776,1799-1807,1849-1854`, - `src/codex/generation.ts:97-103`). WP9 + `src/codex/generation.ts:110-142`). WP9 consumes the contract-owned `withExpectedConfigGenerationSync`; it does not wrap this observer in the lock or redefine the generation contract. - `src/codex/catalog-admission.ts` (MODIFY) — keep the landed request constructor @@ -51,7 +51,7 @@ IN — observe-only admission and gather: assign an opaque identity to the exact retained config reference plus its keyed snapshot, and carry the contract-owned `sourceEvidence`. Do not redefine `createCatalogConvergeRequest` or `captureCatalogAdmissionSnapshot`, which - already exist at lines 38-52 and 148-185. + already exist at lines 40-53 and 138-179. - `src/codex/convergence-types.ts` (MODIFY) — synchronize the already contract-owned closed `CatalogSourceRole`, `CatalogHomeSelectionObservation`, `CatalogSourceObservation`, `CatalogSourceEvidence`, @@ -91,13 +91,20 @@ IN — observe-only admission and gather: cache hits and replacements are process-local. The current worktree's runtime owner already publishes/returns deeply frozen detached values (`src/codex/runtime.ts:16-48,90-105,417-470,505-516`), and the bundled owner does - likewise (`src/codex/catalog/bundled.ts:52-130,260-283`); WP9 preserves those + likewise (`src/codex/catalog/bundled.ts:60-138,268-302`); WP9 preserves those round-4-closed semantics rather than reintroducing the audited private-cache alias. The ordinary resolver reaches `probeVersion`, whose sandbox deliberately calls `mkdtempSync` and `rmSync` (`src/codex/runtime.ts:286-335,505-516`), while bundled loading both calls the persisting resolver and runs `codex debug models` - (`src/codex/catalog/bundled.ts:212-258,271-294`). Neither path is reachable from + (`src/codex/catalog/bundled.ts:220-267,268-302`). Neither path is reachable from gather. +- `src/codex/model-cache.ts` (MODIFY) — **round-6 scope amendment:** this is the + canonical owner of the per-provider models/freshness/cooldown decision consumed by + flight authority, so WP9 must add its deep-frozen detached snapshots and monotonic + owner epoch here rather than asking `provider-fetch.ts` to attest to state it does + not own. Today fresh/stale readers expose the private array, `setCached` retains the + caller alias, and failure, clear, reconcile, and eviction mutate without an epoch + (`src/codex/model-cache.ts:17-21,74-86,147-168,172-208,225-226`). - `src/oauth/index.ts`, `src/oauth/store.ts`, `src/codex/catalog/provider-fetch.ts` (MODIFY) — add and consume an observe-only active-token snapshot. The filesystem-evidence owner reads the exact auth-store @@ -109,7 +116,7 @@ IN — observe-only admission and gather: an intent lock, creates/removes an intent file, hardens a path, or backs up malformed credentials. The current token resolver can enter refresh/persistence (`src/oauth/index.ts:327-400`) and the ordinary refresh-capable gather awaits it - (`src/codex/catalog/provider-fetch.ts:474-500`). + (`src/codex/catalog/provider-fetch.ts:472-499`). Replace the partial `providerCatalogFingerprint` single-flight identity with the complete contract-owned gather-authority identity. Authority capture happens before the map lookup; the flight consumes the captured config/auth/native/source/process @@ -125,7 +132,7 @@ IN — observe-only admission and gather: relative `model_catalog_json` resolves below the canonical active home, an absolute configured target remains absolute even outside that home, and an existing catalog leaf symlink resolves to and writes through its real target - (`src/codex/catalog/parsing.ts:52-80`, `src/config.ts:125-160,188-209`). + (`src/codex/catalog/parsing.ts:52-80`, `src/config.ts:125-164,192-213`). IN — fixed commit and convergence: @@ -142,13 +149,13 @@ IN — fixed commit and convergence: - `src/codex/user-identity.ts` (MODIFY) — add `resolveCodexCatalogSerializationDatabasePath` beside the landed native coordinator resolver. Consumers use its final path verbatim; the K and N database paths must be - distinct (`005_contract.md:1372-1483`). + distinct (`005_contract.md:1442-1553`). - `src/codex/internal/catalog-writer.ts` (NEW/MOVE) — the contract-owned low-level owner for catalog, hashed/legacy backups, and models cache. Every mutator requires the permit plus canonical owning `CODEX_HOME` and calls K's runtime assertion before temp creation, hardening, unlink, link, rename, truncate, replacement, or any other filesystem mutation. It never reads or mutates K's private registry. Do not create - the obsolete `internal/catalog-commit.ts` name (`005_contract.md:1485-1518`). + the obsolete `internal/catalog-commit.ts` name (`005_contract.md:1555-1589`). - `src/codex/convergence.ts` (NEW) — primary catalog gather/commit orchestration for the 16 management mutations. The symbol graph additionally permits only the exact four WP9 transitional writer chains below; WP12 removes those exceptions. @@ -186,12 +193,17 @@ IN — management callers and tests: return values, and compatibility behavior stay unchanged; explicit imports needed after the facade stops re-exporting writers are mechanical. This is freshness-safe serialization of the retained roots, not WP12's convergence rewire - (`005_contract.md:1519-1570`). + (`005_contract.md:1591-1640`). - `tests/codex-refresh.test.ts` and the existing management route suites (MODIFY). - `tests/codex-runtime.test.ts` (MODIFY) — retain the current regression proving nested mutation through a returned runtime cache graph cannot alter owner state (`tests/codex-runtime.test.ts:138-148`) and use only the intentional owner mutation/invalidation seam when a test needs to move the cache epoch. +- `tests/codex-catalog.test.ts`, `tests/app-owned-memory.test.ts`, and + `tests/gather-routed-models-single-flight.test.ts` (MODIFY) — focused owner and + flight tests for nested alias isolation, every required epoch bump, and + pre-lookup TTL/cooldown decisions. These are the only additional files pulled by + the round-6 owner-scope correction. - `tests/codex-convergence-contract.test.ts` (CREATE). It does not exist in the WP8b tree; WP9 creates it rather than “extending” an imaginary file. @@ -209,6 +221,13 @@ OUT: - Any runtime command that starts, stops, syncs, restores, ensures, or manages the live service; any write to real `~/.codex` or `~/.opencodex`; GUI/release/deploy. +The round-6 scope amendment stops at the authority owner. `model-cache.ts` already +depends only on the catalog model type and the generic generation-reconciliation and +memory-budget hooks; advancing its epoch inside reconciliation and the existing +eviction callback requires no WP10 history owner, WP11 N acquisition, or WP12 full +admission/observer/provenance surface. The three focused tests above are the only +additional file scope. No phase boundary or N -> K -> C edge changes. + WP9 typechecks at its own commit. `management-convergence.ts` contains working catalog behavior, not a placeholder waiting for WP12. WP12 may consolidate the management factory/projection into `convergence.ts` when it installs the full @@ -235,11 +254,11 @@ consumer, while deferring it to WP11/WP12 would leave WP9's retained writers uns The guarantee is **filesystem-write-free**, not globally side-effect-free. Gather may update bounded discovery-status, provider model cache, and in-flight admission maps -(`src/codex/catalog/provider-fetch.ts:525-537,572-588,686-692,798-820`); they are +(`src/codex/catalog/provider-fetch.ts:523-543,570-586,684-690,796-818`); they are permitted because they do not mutate user files and are reset between isolated tests. The runtime and bundled memos are observe-only from gather, but their owners can replace them concurrently (`src/codex/runtime.ts:417-470`, -`src/codex/catalog/bundled.ts:52-130`), which is why the candidate seals epochs and +`src/codex/catalog/bundled.ts:60-138`), which is why the candidate seals epochs and identities. No credential, raw provider error, source path, or digest may escape through those caches into `CatalogDisposition`. @@ -268,7 +287,7 @@ This bound deliberately catches the writes hidden by the old plan: - `loadBundledCodexCatalog()` and `runCodexDebugModels()` are forbidden beneath gather. The current bundled loader resolves/persists a runtime and executes `codex debug models --bundled` without an isolated gather environment - (`src/codex/catalog/bundled.ts:212-294`). + (`src/codex/catalog/bundled.ts:220-302`). The observe-only generation API opens an existing database with `readonly:true`, performs only schema/version/select checks, and closes it. Missing DB/table/row, @@ -310,19 +329,19 @@ evidence; token bytes never do. Round 5 closes a separate live-flight authority hole. The current `providerCatalogFingerprint` includes endpoint/catalog fields but excludes `authMode`, `apiKey`, and `headers` -(`src/codex/catalog/provider-fetch.ts:136-158`); `gatherFlightKey` hashes that partial -projection (`src/codex/catalog/provider-fetch.ts:161-180`) and the map joins solely by -the resulting key (`src/codex/catalog/provider-fetch.ts:792-820`). Those omitted +(`src/codex/catalog/provider-fetch.ts:134-156`); `gatherFlightKey` hashes that partial +projection (`src/codex/catalog/provider-fetch.ts:159-179`) and the map joins solely by +the resulting key (`src/codex/catalog/provider-fetch.ts:790-819`). Those omitted fields are behavioral: forward mode exits with no models, credential resolution is awaited, and the effective request headers are then built -(`src/codex/catalog/provider-fetch.ts:474-500,548-568`). The observed failure was A at +(`src/codex/catalog/provider-fetch.ts:472-499,546-566`). The observed failure was A at generation N in forward mode, B admitted at N+1 in key mode, and B receiving A's empty promise result with no A authority in its candidate. The current in-progress worktree adds a plain SHA-256 of the observed auth-store -buffer as a key prefix (`src/codex/catalog/provider-fetch.ts:773-787`). It still omits +buffer as a key prefix (`src/codex/catalog/provider-fetch.ts:771-787`). It still omits static key/forward mode and configured headers, and `GatherFlightResult` still carries -no authority (`src/codex/catalog/provider-fetch.ts:87-91`). It therefore neither +no authority (`src/codex/catalog/provider-fetch.ts:85-89`). It therefore neither closes the failing sequence nor meets the privacy rule below. WP9 keeps single-flight because equivalent concurrent management mutations should @@ -357,18 +376,62 @@ revision identities of provider registry data, generated Jawcode metadata, and t pinned upstream-model snapshot used during assembly. `provider-fetch.ts` consumes this captured value; it may not call the current hidden native source path after the flight is keyed. Today that path reaches `nativeOpenAiSlugs()` during combo assembly -(`src/codex/catalog/provider-fetch.ts:880-904`), and the owner can read active catalog +(`src/codex/catalog/provider-fetch.ts:878-903`), and the owner can read active catalog or cache while selecting slugs (`src/codex/catalog/metadata.ts:169-179`). Relevant process-local flight input includes the settled runtime/bundled epoch/value evidence and a per-provider immutable model-cache/cooldown snapshot with its owner epoch and value identity. The latter controls whether discovery uses fresh, -stale, configured, or network data (`src/codex/catalog/provider-fetch.ts:519-560`), +stale, configured, or network data (`src/codex/catalog/provider-fetch.ts:517-560`), so omitting it again treats absence of evidence as equality. It binds map admission but is not added to K -> C revalidation because the producing flight itself may advance that cache. Runtime/bundled evidence remains candidate-bound and commit-revalidated exactly as already specified. +Round 6 makes both ownership and ordering executable. `model-cache.ts` replaces its +mutable `CacheEntry.models` with a private recursively deep-frozen clone and exposes +recursively readonly types. `setCached` clones every reachable object/array before +publication and retains no caller alias; `getFreshCached`, `getStaleCached`, and the +new flight-decision capture return detached recursively deep-frozen snapshots rather +than the private graph. The owner's process-lifetime epoch advances before every +result-affecting publication: every `setCached`, including a byte-identical +replacement; every `markModelsFetchFailure`; every state-changing provider/all clear; +every accepted reconciliation generation; and every budget eviction that removes an +entry. The epoch is monotonic and is not reset by test cleanup. Omitting any one of +those bumps is a contract failure, not an optimization. + +The flight race is decided before the map, not repaired inside the promise. After the +exact config, effective auth, native-source, and runtime/bundled inputs are known, +gather takes one clock observation and synchronously asks the owner for an ordered +decision for every enabled provider: `unused`, `fresh-cache`, `cooldown`, or +`network`. Each used snapshot includes the detached fresh/stale models or explicit +absence, owner epoch, stored `fetchedAt`/`failureAt`, and exact +`freshUntil`/`cooldownUntil`. The authority owner computes its domain-separated keyed +value identity and the aggregate `modelCacheDecisionIdentity`; only then may code +consult `gatherInflight` (`src/codex/catalog/provider-fetch.ts:796-819`). + +`gatherRoutedModelsUncached` receives those immutable decisions as arguments. Once a +flight has claimed a slot, its provider branches may not call `getFreshCached`, +`getStaleCached`, or `isModelsFetchCoolingDown`; even a network-failure fallback uses +the stale/absence value sealed before lookup. A success or failure may mutate the +owner through `setCached`/`markModelsFetchFailure`, which advances the epoch and makes +a later admission distinct. Time passage itself does not bump the epoch: crossing +`freshUntil` or `cooldownUntil` changes the effective decision and therefore its keyed +identity, so a post-boundary caller cannot join a pre-boundary flight even when the +stored value and owner epoch are unchanged. Calls captured on the same side of the +same boundary may still coalesce. + +The fourth-cache audit found no additional mutable process-local input to catalog +model selection. `gatherInflight` is the coordination map being keyed, while +`lastDropWarnSignature`, combo warning signatures, and the last-omission sink affect +logging or retain an output copy; the flight result uses its own local omissions and +none of those maps is read to choose models +(`src/codex/catalog/provider-fetch.ts:108-110,273-281,821-852`, +`src/codex/catalog/aggregation.ts:39-69,238-244`). Discovery-status/live-count maps +likewise feed management status, not provider model selection. If any such side map +becomes a gather input later, it must join the pre-lookup immutable authority capture +and owner-epoch rule before that change lands. + No raw API key, OAuth token, secret header, auth-store buffer, or stable plain digest is a field of the identity. Only opaque process-keyed HMAC values and the opaque config-reference token leave the authority owner, and none may be logged, serialized, @@ -444,7 +507,7 @@ comparison remain outside C17 (`005_contract.md:830-943`). catalog first and clone it as the native template; read the on-disk catalog separately as the merge source. The invariant is explicit at `structure/03_catalog-and-subagents.md:23-27` and implemented at -`src/codex/catalog/bundled.ts:482-490` plus +`src/codex/catalog/bundled.ts:474-483` plus `src/codex/catalog/sync.ts:517-523`. The WP9 edit removes the materializing fallback call from the tail of @@ -465,6 +528,10 @@ sealed `CatalogProcessLocalEvidence`, plus the complete deep-frozen requires exact equality with the caller's expected authority; mismatch returns retryable stale without constructing the candidate. Memo-derived graphs are recursively frozen detached snapshots and cannot alias the owners' private caches. +The authority equality includes the ordered pre-lookup provider cache/cooldown +decisions, but commit does not revalidate their epoch because the producing flight is +allowed to advance that owner while resolving; the result-authority equality is the +binding proof for this flight-only input. Commit marks a successfully constructed candidate consumed before validation and before the first write; a second call returns `candidate-consumed` and writes nothing. No route can inspect, serialize, reconstruct, or replay it. @@ -554,7 +621,7 @@ creates and hardens a unique adjacent temp, then uses an operation whose contrac destination-must-not-exist: exclusive hard link or a platform rename-without-replace equivalent. Ordinary overwrite rename is never a fallback. The existing `atomicWriteFile` cannot implement this contract because its final -operation is an overwriting rename (`src/config.ts:188-220`, especially line 209). +operation is an overwriting rename (`src/config.ts:192-245`, especially line 213). The unpublished temp is scrubbed and removed on every path. If publication returns `EEXIST`, another process won after validation. Commit @@ -564,7 +631,7 @@ regular, non-routed valid catalog backup is preserved and the receipt becomes is `refused`. The loser never unlinks, truncates, or overwrites the winner. This exception applies only to a backup create-once target, never to a backup selected as a gather source; selected source observations remain strict -(`005_contract.md:1034-1054`). +(`005_contract.md:1087-1107`). ## C. Catalog-only convergence @@ -574,9 +641,9 @@ WP9 does not redeclare request, snapshot, projection, or shared result types. `management-convergence.ts` consumes: - `createCatalogConvergeRequest` from - `src/codex/catalog-admission.ts:38-52`; + `src/codex/catalog-admission.ts:40-53`; - `captureCatalogAdmissionSnapshot` from - `src/codex/catalog-admission.ts:148-185`; + `src/codex/catalog-admission.ts:138-179`; - `projectCatalogOnlyOutcome` from its landed owner at `src/codex/management-convergence.ts:63-75`; - shared `CatalogDisposition`, `ConvergeOutcome`, and `ConvergeCodex` from @@ -770,7 +837,36 @@ instrumented production log/response/serialization sinks receive neither that identity nor the API key, OAuth token, configured secret header, auth-store bytes, or their plain SHA-256 values. The named privacy mutation **put `apiKey`, header values, token text, or a stable unkeyed credential digest into -the fingerprint/identity** fails that assertion and `privacy:scan`. +the fingerprint/identity** must fail those behavioral sink assertions. Round 6 +verified that `bun run privacy:scan` still passes against the current plain auth-store +digest, so the scanner remains a supplemental hygiene gate and is not proof of this +non-disclosure property. + +Add the model-cache owner/flight matrix with one fake clock and the real public owner +APIs. Seed nested models through `setCached`, mutate the caller's original nested +object/array, then attempt the same through `getFreshCached` and `getStaleCached`. +The owner snapshot and a second read remain byte-identical and recursively frozen. +The named broken mutation **retain the `setCached` array or return the private cache +graph** must change the second read without any assignment/epoch and turn this row red. + +Pause A after its per-provider decision is captured and before its flight settles. +Run separate rows for byte-identical `setCached`, `markModelsFetchFailure`, provider +and all-cache clear, accepted reconciliation, and real budget eviction. Each mutation +must advance the owner epoch, change B's decision/value identity, and prevent B from +joining or accepting A. The named broken mutations **omit the set bump for an equal +replacement**, **omit the failure/cooldown bump**, **omit the clear bump**, **omit the +reconcile bump**, and **omit the eviction bump** each make exactly their row red. The +harness observes the epoch immediately around each owner call so a later mutation +cannot accidentally mask a missing bump; omitting ANY required bump therefore fails. + +Finally capture one A immediately before a cache TTL boundary and one immediately +before a cooldown boundary, advance the fake clock across exactly that boundary with +no owner mutation, and capture B. B's effective decision changes to `network`, its +identity differs, and it cannot join A. Instrument the flight body so every +post-lookup call to `getFreshCached`, `getStaleCached`, or +`isModelsFetchCoolingDown` fails the test. The named broken mutation **key only epoch +and stored value, then decide or re-read freshness/cooldown after claiming the +flight** turns both boundary rows red. Table-drive every `CatalogSourceRole`: required config target selection, filesystem-backed bundled-template source, active catalog, selected hashed backup, @@ -800,7 +896,7 @@ A's resolved config/source evidence and skip home re-resolution**; it incorrectl commits into A while Codex reads B. Freeze the current target-selection semantics with three real-file fixtures -(`src/codex/catalog/parsing.ts:52-80`, `src/config.ts:125-160,188-209`): +(`src/codex/catalog/parsing.ts:52-80`, `src/config.ts:125-164,192-213`): - relative `model_catalog_json = "nested/a.json"` resolves beneath the canonical `CODEX_HOME`, and gather/commit compare and write that derived target. The named @@ -890,12 +986,14 @@ comparison, parent A→B→A entirely between checks, or a write after the compa Broken mutations that must turn T2 red, in addition to the named mutations above: omit the required ABSENT config observation, remove digest comparison while retaining generation/file identity, release C before callback, acquire C before K, call -`readConfigGenerationAtPath` from inside the guard, or remove the low-level mutator's -runtime permit assertion. The absent->present target switch commits obsolete bytes, +`readConfigGenerationAtPath` from inside the guard, remove the low-level mutator's +runtime permit assertion, return/retain a mutable model-cache alias, omit any required +model-cache owner-epoch bump, capture cache/cooldown after map lookup, or omit the +effective TTL/cooldown boundary from its decision identity. The absent->present target switch commits obsolete bytes, the same-inode rewrite commits stale bytes, process B commits N+1 while A is paused, retained-gathered X overwrites K-published Y, a leaked/forged/wrong-home permit reaches -filesystem mutation, inverse order deadlocks/self-contends, or the guard opens the -forbidden second handle. +filesystem mutation, a post-mutation/boundary caller joins the wrong provider flight, +inverse order deadlocks/self-contends, or the guard opens the forbidden second handle. ### T3 — exact four-step receipt and bytes @@ -977,7 +1075,7 @@ Broken mutations that must turn T5 red: add a static top-level management-conver import, alias a catalog writer into a management route, replace a literal import with a computed dynamic import, or add an absence-only `existsSync`/target `realpath` outside the evidence owner. Also restore a bare-promise flight map, let the flight -re-resolve auth/native sources after keying, remove the permit parameter from one mutator, add a +re-resolve auth/native/model-cache decisions after keying, remove the permit parameter from one mutator, add a fifth unpermitted root, bypass the permit assertion before one filesystem mutator, or invert any lock edge. The sentinel, compile fixture, or fail-closed graph must reject each; T2, not the graph, rejects a permit whose runtime lifetime/home is invalid. @@ -1008,6 +1106,7 @@ Static/focused gates for the WP9 commit: ```bash bun test tests/codex-refresh.test.ts tests/codex-convergence-contract.test.ts +bun test tests/codex-catalog.test.ts tests/app-owned-memory.test.ts tests/gather-routed-models-single-flight.test.ts bun test tests/codex-config-generation.test.ts tests/codex-sync-api.test.ts tests/codex-models-cache-invalidate.test.ts tests/codex-runtime.test.ts bun test tests/model-visibility-management-api.test.ts tests/management-provider-validation.test.ts tests/combo-management-api.test.ts tests/codex-v2-gate.test.ts bun run typecheck @@ -1026,7 +1125,7 @@ processes only. No verification invokes `ocx start`, `stop`, `sync`, `restore`, | Criterion | Proof | Concrete broken mutation that makes it red | |---|---|---| | **C1** — gather is filesystem-write-free across user homes and scratch, performs no executable probe/subprocess, and commit is synchronous, fixed, K -> C ordered, one-shot, and receipt-exact | T1 + T3 + T5 | call cold `resolveCodexRuntime`/`loadBundledCodexCatalog`, add an `await` beneath commit, acquire C before K, reorder replacements, pre-set a receipt bit, or replay a consumed candidate | -| **C2/C17** — complete gather-authority identity prevents or rejects cross-admission flight reuse across config/auth/native/source/process drift without exposing credentials; permanent K makes every first-party authoritative read-transform-write fresh by under-K recomputation or complete post-acquisition evidence revalidation; retained-gathered-first/K-second races are covered against convergence and another retained writer; owner-held config generation, required home/runtime evidence, deeply frozen non-aliased memo snapshots plus epochs, every closed PRESENT/ABSENT source observation, and target identity reject stale work before write; create-once backups publish atomically without clobber | T2 + T3 | restore partial `gatherFlightKey` plus a bare result, omit OAuth/native authority components, put raw or plain-hashed credentials in the identity, leave retained `/api/sync`'s `onDiskCatalog` read before K but guard only replacement, start startup/CLI/restore's authoritative read before K, return or shallow-freeze a private cache snapshot so nested mutation bypasses epoch movement, omit ABSENT `config.toml`/`codex-runtime.json`, skip CODEX_HOME re-resolution, release C before callback, remove same-inode digest comparison, open a second SQLite observer, or replace exclusive publication with overwriting rename | +| **C2/C17** — complete gather-authority identity prevents or rejects cross-admission flight reuse across config/auth/native/source/process drift without exposing credentials; provider model-cache/cooldown decisions are detached, deeply immutable, owner-epoch-bound, effective-boundary-bound, and captured before flight lookup; permanent K makes every first-party authoritative read-transform-write fresh by under-K recomputation or complete post-acquisition evidence revalidation; retained-gathered-first/K-second races are covered against convergence and another retained writer; owner-held config generation, required home/runtime evidence, deeply frozen non-aliased memo snapshots plus epochs, every closed PRESENT/ABSENT source observation, and target identity reject stale work before write; create-once backups publish atomically without clobber | T2 + T3 | restore partial `gatherFlightKey` plus a bare result, omit OAuth/native authority components, put raw or plain-hashed credentials in the identity, decide/re-read model-cache or cooldown after flight lookup, omit any owner epoch bump or TTL/cooldown boundary identity, leave retained `/api/sync`'s `onDiskCatalog` read before K but guard only replacement, start startup/CLI/restore's authoritative read before K, return or shallow-freeze a private cache snapshot so nested mutation bypasses epoch movement, omit ABSENT `config.toml`/`codex-runtime.json`, skip CODEX_HOME re-resolution, release C before callback, remove same-inode digest comparison, open a second SQLite observer, or replace exclusive publication with overwriting rename | | **Catalog/native boundary** — catalog-only never reads/advances the native pair or writes routing/history artifacts | T6 | call `expectation()`/`beginTransition`, add pair fields to `catalog-only`, or invoke config/profile/journal/history writer | | **Best-effort compatibility** — all 16 primary writes retain 2xx/201 and original follow-up order for every catalog failure | T4 | let lazy import/factory/admission throw, scope “zero writes” to the whole route, or return before Claude/Desktop follow-up | | **C14, WP9-bounded** — the 16 management roots reach catalog writers only through convergence; exactly four documented transitional roots remain until WP12; every low-level mutation requires permanent K's fresh acquisition-bound permit and runtime liveness/transaction/home assertion | T2 barriers + permit negatives + T5 symbol graph | add a fifth root, omit K/permit/assertion from one retained chain, accept a leaked/reused/forged/wrong-home permit, hide a writer through alias/re-export/computed import, or accidentally require WP12 to have already rewired `/api/sync`/startup/CLI/restore | From 471eb5f09df9d599a9c3103f8124dbf50018accf Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Tue, 4 Aug 2026 22:36:31 +0900 Subject: [PATCH 071/163] docs(substrate): a test helper mutates the registry the flight key ignores MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Round 7 opened with a claim that every exported constant is mutable authority, and most of it did not survive contact with the repository. Nothing here mutates CALLABLE_CONFIGURED_COMPATIBILITY_MODELS or JAWCODE_CATALOG_AUGMENT_PROVIDERS; they are module-load literals with one read site each, and the reviewer's demonstration mutated one itself. That proves JavaScript does not enforce ReadonlySet at runtime, not that this system has a defect, and treating it as authority would make every export authority without bounding anything. Recorded here so a later round does not re-raise it: those two are deliberately excluded, and freezing them is optional defense in depth. What did survive is narrower and real. tests/helpers/provider-registry- discovery.ts assigns entry.modelDiscovery and entry.preserveCustomDestination onto a live PROVIDER_REGISTRY row and restores them in a finally (:15-30), and three tests drive the real gatherRoutedModels through it. Those fields decide results: resolveProviderModelDiscovery reads the live entry and derives the filter, maxModels and maxResponseBytes from it (model-discovery.ts:123-136). With a flight already active, a second caller holding the same config reference, generation and cache decision joins it and receives rows its own policy would have rejected — the reviewer showed fetchCount=1 with both callers getting three rows, against a control run where the same override alone returned the fallback and warned about the two-row limit. Effective discovery policy is now its own detached frozen snapshot inside the authority identity: the registry-transport match outcome, the exact url/path/query location policy including explicit absence, the final method and URL, the complete declarative filter, and the clamped byte and row limits. It had to be separate from the native component, which is `unused` when no combo needs native rows — and the reproduction has no combos, so the field would have gone unread exactly when it mattered. A source-revision string would not close this either: the helper changes content without changing any such revision. --- .../260804_codex_write_substrate/005_contract.md | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/devlog/_plan/260804_codex_write_substrate/005_contract.md b/devlog/_plan/260804_codex_write_substrate/005_contract.md index b6398af41..0b791530e 100644 --- a/devlog/_plan/260804_codex_write_substrate/005_contract.md +++ b/devlog/_plan/260804_codex_write_substrate/005_contract.md @@ -986,9 +986,12 @@ Before consulting the in-flight map, gather constructs and recursively freezes o credential state, exact resolved API-key or observe-only OAuth access-token bytes (or explicit absence), the exact `provider-auth-selection` observation that chose an OAuth account/token, and the final discovery method, URL, and normalized header - set after transport defaults. Header names are lowercase and sorted; values remain - byte-exact inside the keyed input. Forward and local modes are explicit states, not - absence; + set after transport defaults. Beside it, a detached recursively frozen effective + discovery-policy snapshot records the registry-transport match outcome, the exact + `url`/`path`/`query` location policy (including explicit absence), the final method + and URL, the complete declarative filter, and the clamped `maxResponseBytes` and + `maxModels`. Header names are lowercase and sorted; values remain byte-exact inside + the keyed input. Forward and local modes are explicit states, not absence; 3. the exact native-slug/source input. When combo resolution needs native rows, the filesystem-evidence owner returns a detached immutable ordered slug/capability snapshot and records every consulted active-catalog/cache source, PRESENT or @@ -1189,6 +1192,8 @@ export interface CatalogGatherAuthorityIdentity { readonly snapshotIdentity: string; }>; readonly authSnapshotIdentity: string; + /** HMAC of every result-affecting field in the detached effective discovery policy. */ + readonly discoveryPolicyIdentity: string; readonly nativeCatalogSourceIdentity: string; readonly sourceEvidenceIdentity: string; readonly processLocalEvidenceIdentity: string; From be1ec9bf2c97b8c5e404eb96c674b846ba4896e0 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Tue, 4 Aug 2026 22:38:40 +0900 Subject: [PATCH 072/163] docs(substrate): discovery policy is its own component, not a combo-gated field MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The contract gained the effective discovery-policy snapshot; this is the plan catching up, plus the two decisions that go with it. Why it is a separate component rather than a field of the native one: that component is `unused` when no combo needs native rows, and the audited reproduction has no combos, so filing discovery policy there would have left it unread in exactly the case that produced the defect. The snapshot is captured before flight lookup and consumed by the flight, so nothing re-reads the registry after keying, and it is keyed by content rather than by a registry source revision because the helper mutates an entry in place without moving any revision. The regression names both of those as its broken mutations, and pairs the joined-flight case with the control the reviewer ran: the same override with no concurrent flight returns the fallback and warns about the row limit. Without that control the joined result is merely different; with it, it is recognizably wrong. Also recorded: the two exported Sets from the same audit round are deliberately NOT authority. Nothing in this repository mutates them, each is a module-load literal with one read site, and the demonstration mutated one directly — which shows JavaScript does not enforce ReadonlySet, not that this system has a defect. Written down so the next round does not re-raise them and quietly turn every export into authority. --- .../010_catalog_seam.md | 45 ++++++++++++++++++- 1 file changed, 44 insertions(+), 1 deletion(-) diff --git a/devlog/_plan/260804_codex_write_substrate/010_catalog_seam.md b/devlog/_plan/260804_codex_write_substrate/010_catalog_seam.md index b5d563949..6c614c7ab 100644 --- a/devlog/_plan/260804_codex_write_substrate/010_catalog_seam.md +++ b/devlog/_plan/260804_codex_write_substrate/010_catalog_seam.md @@ -368,6 +368,36 @@ observe-only OAuth token bytes (or explicit absence), the exact headers after transport defaults. Header names are lowercased and sorted while values stay byte-exact only inside the HMAC input. +Effective discovery policy is its own component, not a field of the one below. Round +7 reproduced why: `tests/helpers/provider-registry-discovery.ts:15-30` assigns +`modelDiscovery` and `preserveCustomDestination` onto a live `PROVIDER_REGISTRY` row +and restores them in a `finally`, and three tests drive the real `gatherRoutedModels` +through it (`tests/provider-model-discovery-contract.test.ts:182,210,253`). Those +fields decide results — `resolveProviderModelDiscovery` reads the live entry and +derives the filter plus the clamped `maxModels`/`maxResponseBytes` from it +(`src/providers/model-discovery.ts:123-136`). With a flight already active, a caller +holding the same config reference, generation and cache decision joined it and +received rows its own policy required rejecting: `fetchCount=1` with both callers +getting three rows, against a control run where the same override alone returned the +fallback and warned about the two-row limit. + +So before flight lookup the owner captures a detached, recursively frozen effective +discovery-policy snapshot per enabled provider: the registry-transport match outcome, +the exact `url`/`path`/`query` location policy including explicit absence, the final +method and URL, the complete declarative filter, and the clamped `maxResponseBytes` +and `maxModels`. Its HMAC is `discoveryPolicyIdentity`, and the flight consumes only +that snapshot — nothing re-reads the registry after keying. Filing this under the +native component would have hidden it exactly when it matters, because that component +is `unused` without combos and the reproduction has no combos. A registry source +revision is also insufficient: the helper changes content without changing one. + +Two exported Sets were investigated in the same round and deliberately excluded. +Nothing in this repository mutates `CALLABLE_CONFIGURED_COMPATIBILITY_MODELS` +(`src/codex/catalog/provider-fetch.ts:286`) or `JAWCODE_CATALOG_AUGMENT_PROVIDERS` +(`src/codex/catalog/parsing.ts:121`); each is a module-load literal with a single read +site. Treating them as authority would make every export authority. Freezing them is +optional defense in depth, not a prerequisite for this phase. + The native component is explicit `unused` when no combo needs native rows. Otherwise the evidence owner captures a detached, deeply frozen ordered native slug/capability snapshot and records every active-catalog/cache consultation, including absence, as @@ -832,6 +862,19 @@ accept A's native rows. The named broken mutation **omit `native-catalog-selection`/`nativeCatalogSourceIdentity` from the map and result identity** must turn this test red. +Repeat once more with config, auth and native sources all unchanged while A is live, +and change only the effective discovery policy through the existing +`withRegistryDiscovery` helper — the reachable mutation path, not a synthetic one. +B must capture a different `discoveryPolicyIdentity`, claim its own flight or reject +A's result as stale, and must never accept rows its own `maxModels`/filter would have +rejected. Pair it with the control the audit used: the same override with no +concurrent flight returns the fallback and warns about the row limit, which is what +makes the joined result recognizably wrong rather than merely different. The named +broken mutations that must turn this red are **file discovery policy under the native +component** (it is `unused` without combos, and this case has none) and **key the +snapshot by a registry source revision instead of its content** (the helper mutates +the entry in place without moving any revision). + Each row verifies the private identity inside the owner-level unit fixture, while instrumented production log/response/serialization sinks receive neither that identity nor the API key, OAuth token, configured secret header, auth-store bytes, or @@ -1125,7 +1168,7 @@ processes only. No verification invokes `ocx start`, `stop`, `sync`, `restore`, | Criterion | Proof | Concrete broken mutation that makes it red | |---|---|---| | **C1** — gather is filesystem-write-free across user homes and scratch, performs no executable probe/subprocess, and commit is synchronous, fixed, K -> C ordered, one-shot, and receipt-exact | T1 + T3 + T5 | call cold `resolveCodexRuntime`/`loadBundledCodexCatalog`, add an `await` beneath commit, acquire C before K, reorder replacements, pre-set a receipt bit, or replay a consumed candidate | -| **C2/C17** — complete gather-authority identity prevents or rejects cross-admission flight reuse across config/auth/native/source/process drift without exposing credentials; provider model-cache/cooldown decisions are detached, deeply immutable, owner-epoch-bound, effective-boundary-bound, and captured before flight lookup; permanent K makes every first-party authoritative read-transform-write fresh by under-K recomputation or complete post-acquisition evidence revalidation; retained-gathered-first/K-second races are covered against convergence and another retained writer; owner-held config generation, required home/runtime evidence, deeply frozen non-aliased memo snapshots plus epochs, every closed PRESENT/ABSENT source observation, and target identity reject stale work before write; create-once backups publish atomically without clobber | T2 + T3 | restore partial `gatherFlightKey` plus a bare result, omit OAuth/native authority components, put raw or plain-hashed credentials in the identity, decide/re-read model-cache or cooldown after flight lookup, omit any owner epoch bump or TTL/cooldown boundary identity, leave retained `/api/sync`'s `onDiskCatalog` read before K but guard only replacement, start startup/CLI/restore's authoritative read before K, return or shallow-freeze a private cache snapshot so nested mutation bypasses epoch movement, omit ABSENT `config.toml`/`codex-runtime.json`, skip CODEX_HOME re-resolution, release C before callback, remove same-inode digest comparison, open a second SQLite observer, or replace exclusive publication with overwriting rename | +| **C2/C17** — complete gather-authority identity prevents or rejects cross-admission flight reuse across config/auth/discovery-policy/native/source/process drift without exposing credentials; provider model-cache/cooldown decisions are detached, deeply immutable, owner-epoch-bound, effective-boundary-bound, and captured before flight lookup; permanent K makes every first-party authoritative read-transform-write fresh by under-K recomputation or complete post-acquisition evidence revalidation; retained-gathered-first/K-second races are covered against convergence and another retained writer; owner-held config generation, required home/runtime evidence, deeply frozen non-aliased memo snapshots plus epochs, every closed PRESENT/ABSENT source observation, and target identity reject stale work before write; create-once backups publish atomically without clobber | T2 + T3 | restore partial `gatherFlightKey` plus a bare result, omit OAuth/native authority components, file effective discovery policy under the combo-gated native component or key it by a registry source revision rather than its content, put raw or plain-hashed credentials in the identity, decide/re-read model-cache or cooldown after flight lookup, omit any owner epoch bump or TTL/cooldown boundary identity, leave retained `/api/sync`'s `onDiskCatalog` read before K but guard only replacement, start startup/CLI/restore's authoritative read before K, return or shallow-freeze a private cache snapshot so nested mutation bypasses epoch movement, omit ABSENT `config.toml`/`codex-runtime.json`, skip CODEX_HOME re-resolution, release C before callback, remove same-inode digest comparison, open a second SQLite observer, or replace exclusive publication with overwriting rename | | **Catalog/native boundary** — catalog-only never reads/advances the native pair or writes routing/history artifacts | T6 | call `expectation()`/`beginTransition`, add pair fields to `catalog-only`, or invoke config/profile/journal/history writer | | **Best-effort compatibility** — all 16 primary writes retain 2xx/201 and original follow-up order for every catalog failure | T4 | let lazy import/factory/admission throw, scope “zero writes” to the whole route, or return before Claude/Desktop follow-up | | **C14, WP9-bounded** — the 16 management roots reach catalog writers only through convergence; exactly four documented transitional roots remain until WP12; every low-level mutation requires permanent K's fresh acquisition-bound permit and runtime liveness/transaction/home assertion | T2 barriers + permit negatives + T5 symbol graph | add a fifth root, omit K/permit/assertion from one retained chain, accept a leaked/reused/forged/wrong-home permit, hide a writer through alias/re-export/computed import, or accidentally require WP12 to have already rewired `/api/sync`/startup/CLI/restore | From 648fbdb4ab2b9c24e95e86f6213aeb13fb8b58cd Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Tue, 4 Aug 2026 22:51:25 +0900 Subject: [PATCH 073/163] docs(substrate): every test asked whether B may join, none asked what A does next MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Round 8 returned GO with one gap, and the gap is a good illustration of how these tests were shaped. Every authority race so far asks whether a second caller may join an active flight. None asked what the winner does after it wins. `augmentRoutedModelsWithRegistryOpenAiApiRows` re-reads the registry after the network await (provider-fetch.ts:955-961). The reviewer captured A under `withRegistryDiscovery` with a custom openai-apikey destination and `preserveCustomDestination: true`, paused the response, let the helper restore the registry, and resumed: A had captured `match:false`, processed under the restored `match:true`, and emitted the full trusted OpenAI row set. So an implementation can key correctly, carry its authority honestly, and still commit bytes from a policy it never admitted — passing every join-focused row. The prohibition on re-resolving after claiming a slot now binds the whole post-await tail rather than the join decision alone, and every downstream augmentation input, including the registry-transport match that decides whether trusted rows are added, is passed in from the captured snapshot. The regression drives that exact sequence with post-lookup registry reads instrumented to fail. This does not block starting WP9; it blocks calling WP9 accepted. --- .../005_contract.md | 11 +++++++++-- .../010_catalog_seam.md | 19 ++++++++++++++++++- 2 files changed, 27 insertions(+), 3 deletions(-) diff --git a/devlog/_plan/260804_codex_write_substrate/005_contract.md b/devlog/_plan/260804_codex_write_substrate/005_contract.md index 0b791530e..40f91c53e 100644 --- a/devlog/_plan/260804_codex_write_substrate/005_contract.md +++ b/devlog/_plan/260804_codex_write_substrate/005_contract.md @@ -1026,8 +1026,15 @@ The in-flight owner stores `{ authority, promise }`, not a bare promise. Its pri bucket is `authorityId`, but joining additionally requires exact equality of the deep-frozen component identities; a mismatch or collision starts a distinct admitted flight (or returns typed busy when the admission gate is full). The flight receives -the captured config/auth/native/source/process snapshots as arguments and may not -re-resolve them after claiming its slot. `GatherFlightResult` carries the exact +the captured config/auth/discovery-policy/native/source/process snapshots as +arguments and may not re-resolve them after claiming its slot. That prohibition binds +the whole post-await tail, not only the join decision: round 8 found +`augmentRoutedModelsWithRegistryOpenAiApiRows` re-reading the registry after the +network await (`src/codex/catalog/provider-fetch.ts:955-961`), so a flight can key and +carry its authority honestly and still emit bytes derived from a policy that changed +while it waited. Every downstream augmentation input — including the +registry-transport match outcome that decides whether trusted OpenAI rows are added — +is passed in from the captured snapshot. `GatherFlightResult` carries the exact authority identity that produced its models and omissions. Before candidate construction, every caller compares that result identity with its own expected identity. Inequality discards the result and returns retryable `stale` or regathers diff --git a/devlog/_plan/260804_codex_write_substrate/010_catalog_seam.md b/devlog/_plan/260804_codex_write_substrate/010_catalog_seam.md index 6c614c7ab..f339bfaac 100644 --- a/devlog/_plan/260804_codex_write_substrate/010_catalog_seam.md +++ b/devlog/_plan/260804_codex_write_substrate/010_catalog_seam.md @@ -875,6 +875,23 @@ component** (it is `unused` without combos, and this case has none) and **key th snapshot by a registry source revision instead of its content** (the helper mutates the entry in place without moving any revision). +One more row, because every case above asks whether B may join A and none asks what A +itself does after it wins. Round 8 found a real post-await reread: +`augmentRoutedModelsWithRegistryOpenAiApiRows` calls `providerMatchesRegistryTransport` +and reads the registry entry after the network await +(`src/codex/catalog/provider-fetch.ts:955-961`). The reviewer captured A under +`withRegistryDiscovery` with a custom `openai-apikey` destination and +`preserveCustomDestination: true`, paused the response, let the helper restore the +registry, then resumed: A had captured `match:false` but processed under the restored +`match:true` and emitted the full trusted OpenAI row set. So an implementation can key +correctly, carry its authority honestly, and still commit bytes from a policy it never +admitted — while passing every join-focused row above. + +Drive that exact sequence and assert A uses its captured non-match and adds no trusted +OpenAI rows, with post-lookup registry reads instrumented to fail. The named broken +mutation is **let post-key processing consult the live registry instead of the +captured match and detached augmentation inputs**. + Each row verifies the private identity inside the owner-level unit fixture, while instrumented production log/response/serialization sinks receive neither that identity nor the API key, OAuth token, configured secret header, auth-store bytes, or @@ -1168,7 +1185,7 @@ processes only. No verification invokes `ocx start`, `stop`, `sync`, `restore`, | Criterion | Proof | Concrete broken mutation that makes it red | |---|---|---| | **C1** — gather is filesystem-write-free across user homes and scratch, performs no executable probe/subprocess, and commit is synchronous, fixed, K -> C ordered, one-shot, and receipt-exact | T1 + T3 + T5 | call cold `resolveCodexRuntime`/`loadBundledCodexCatalog`, add an `await` beneath commit, acquire C before K, reorder replacements, pre-set a receipt bit, or replay a consumed candidate | -| **C2/C17** — complete gather-authority identity prevents or rejects cross-admission flight reuse across config/auth/discovery-policy/native/source/process drift without exposing credentials; provider model-cache/cooldown decisions are detached, deeply immutable, owner-epoch-bound, effective-boundary-bound, and captured before flight lookup; permanent K makes every first-party authoritative read-transform-write fresh by under-K recomputation or complete post-acquisition evidence revalidation; retained-gathered-first/K-second races are covered against convergence and another retained writer; owner-held config generation, required home/runtime evidence, deeply frozen non-aliased memo snapshots plus epochs, every closed PRESENT/ABSENT source observation, and target identity reject stale work before write; create-once backups publish atomically without clobber | T2 + T3 | restore partial `gatherFlightKey` plus a bare result, omit OAuth/native authority components, file effective discovery policy under the combo-gated native component or key it by a registry source revision rather than its content, put raw or plain-hashed credentials in the identity, decide/re-read model-cache or cooldown after flight lookup, omit any owner epoch bump or TTL/cooldown boundary identity, leave retained `/api/sync`'s `onDiskCatalog` read before K but guard only replacement, start startup/CLI/restore's authoritative read before K, return or shallow-freeze a private cache snapshot so nested mutation bypasses epoch movement, omit ABSENT `config.toml`/`codex-runtime.json`, skip CODEX_HOME re-resolution, release C before callback, remove same-inode digest comparison, open a second SQLite observer, or replace exclusive publication with overwriting rename | +| **C2/C17** — complete gather-authority identity prevents or rejects cross-admission flight reuse across config/auth/discovery-policy/native/source/process drift without exposing credentials, and a winning flight consumes only its captured snapshots through the whole post-await tail; provider model-cache/cooldown decisions are detached, deeply immutable, owner-epoch-bound, effective-boundary-bound, and captured before flight lookup; permanent K makes every first-party authoritative read-transform-write fresh by under-K recomputation or complete post-acquisition evidence revalidation; retained-gathered-first/K-second races are covered against convergence and another retained writer; owner-held config generation, required home/runtime evidence, deeply frozen non-aliased memo snapshots plus epochs, every closed PRESENT/ABSENT source observation, and target identity reject stale work before write; create-once backups publish atomically without clobber | T2 + T3 | restore partial `gatherFlightKey` plus a bare result, omit OAuth/native authority components, file effective discovery policy under the combo-gated native component or key it by a registry source revision rather than its content, let post-key processing consult the live registry instead of the captured match and detached augmentation inputs, put raw or plain-hashed credentials in the identity, decide/re-read model-cache or cooldown after flight lookup, omit any owner epoch bump or TTL/cooldown boundary identity, leave retained `/api/sync`'s `onDiskCatalog` read before K but guard only replacement, start startup/CLI/restore's authoritative read before K, return or shallow-freeze a private cache snapshot so nested mutation bypasses epoch movement, omit ABSENT `config.toml`/`codex-runtime.json`, skip CODEX_HOME re-resolution, release C before callback, remove same-inode digest comparison, open a second SQLite observer, or replace exclusive publication with overwriting rename | | **Catalog/native boundary** — catalog-only never reads/advances the native pair or writes routing/history artifacts | T6 | call `expectation()`/`beginTransition`, add pair fields to `catalog-only`, or invoke config/profile/journal/history writer | | **Best-effort compatibility** — all 16 primary writes retain 2xx/201 and original follow-up order for every catalog failure | T4 | let lazy import/factory/admission throw, scope “zero writes” to the whole route, or return before Claude/Desktop follow-up | | **C14, WP9-bounded** — the 16 management roots reach catalog writers only through convergence; exactly four documented transitional roots remain until WP12; every low-level mutation requires permanent K's fresh acquisition-bound permit and runtime liveness/transaction/home assertion | T2 barriers + permit negatives + T5 symbol graph | add a fifth root, omit K/permit/assertion from one retained chain, accept a leaked/reused/forged/wrong-home permit, hide a writer through alias/re-export/computed import, or accidentally require WP12 to have already rewired `/api/sync`/startup/CLI/restore | From cba24820948bb704592bd03de15003bf17d04b61 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Wed, 5 Aug 2026 00:01:39 +0900 Subject: [PATCH 074/163] feat(codex): a shared fetch does not confer shared authority MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The flight key covered endpoints and model lists but not the policy that decides what a response is allowed to contain, so two gathers admitted under different authority could share one result. Effective discovery policy is now its own captured component: the registry-transport match, the exact url/path/query location policy including explicit absence, the final method and URL, the whole declarative filter, and the clamped byte and row limits — HMAC'd into the authority identity and compared for equality before any join. Capture happens immediately before flight lookup, and the flight consumes only what it captured. That binds the whole post-await tail, not just the join decision: `augmentRoutedModelsWithRegistryOpenAiApiRows` re-read the registry after the network await, so a winner could key honestly and still emit rows from a policy that changed while it waited. All identity components now share one unexported per-process key with domain-separated, length-prefixed canonical encoding, which also retires the plain SHA-256 auth-store component — a stable unkeyed digest of credential state is exactly what must not reach a map key. Both regressions are pinned: removing the policy from the equality check makes two admissions collapse to one fetch, and restoring the post-key registry read trips a forbidden-read sentinel inside the real production call. --- src/codex/catalog/provider-fetch.ts | 396 +++++++++++++++++++++++---- src/codex/convergence-types.ts | 33 +++ tests/codex-gather-authority.test.ts | 194 +++++++++++++ 3 files changed, 575 insertions(+), 48 deletions(-) create mode 100644 tests/codex-gather-authority.test.ts diff --git a/src/codex/catalog/provider-fetch.ts b/src/codex/catalog/provider-fetch.ts index e3ecfa86f..fb7f8d406 100644 --- a/src/codex/catalog/provider-fetch.ts +++ b/src/codex/catalog/provider-fetch.ts @@ -1,5 +1,5 @@ import { execFileSync } from "node:child_process"; -import { createHash } from "node:crypto"; +import { createHash, createHmac, randomBytes } from "node:crypto"; import { copyFileSync, existsSync, mkdirSync, readFileSync, realpathSync } from "node:fs"; import { delimiter, dirname, join, resolve } from "node:path"; import { atomicWriteFile, expandUserPath, getConfigDir, resolveEnvValue, websocketsEnabled } from "../../config"; @@ -56,6 +56,7 @@ import { resolveProviderModelDiscovery, type ModelDiscoveryResponseFailure, type ProviderModelsApiItem, + type ResolvedProviderModelDiscovery, } from "../../providers/model-discovery"; import upstreamModelsSnapshot from "../data/upstream-models.json"; import { createAdmissionGate, ResourceAdmissionError, type AdmissionMetrics } from "../../lib/admission"; @@ -67,6 +68,15 @@ import { disabledNativeSlugs, hasComboTargets, nativeInputModalities, nativeOpen import { deriveComboCatalogModel, normalizedOpenAiApiSignature, openAiApiCollisionWarnings, replaceLastComboCatalogOmissions, warnUncataloguedComboOnce } from "./aggregation"; import type { ComboCatalogOmission } from "./aggregation"; import type { CatalogGatherProviderAuthEvidence } from "./filesystem-evidence"; +import type { + CatalogAdmissionSnapshot, + CatalogDiscoveryPolicyField, + CatalogGatherAuthorityIdentity, + CatalogProviderDiscoveryPolicySnapshot, + CatalogProcessLocalEvidence, + CatalogSourceEvidence, + CatalogTrustedOpenAiApiPolicySnapshot, +} from "../convergence-types"; export type { CatalogGatherProviderAuthEvidence } from "./filesystem-evidence"; @@ -80,12 +90,15 @@ export interface CatalogGatherProviderAuthOutcome { export interface GatherRoutedModelsOptions { comboOmissions?: ComboCatalogOmission[]; providerAuthOutcomes?: CatalogGatherProviderAuthOutcome[]; + /** Internal convergence sink for the immutable policy that produced the returned rows. */ + discoveryPolicySnapshots?: CatalogProviderDiscoveryPolicySnapshot[]; } interface GatherFlightResult { models: CatalogModel[]; comboOmissions: ComboCatalogOmission[]; - providerAuthOutcomes: CatalogGatherProviderAuthOutcome[]; + providerAuthOutcomes: readonly CatalogGatherProviderAuthOutcome[]; + discoveryPolicySnapshots: readonly CatalogProviderDiscoveryPolicySnapshot[]; } interface ModelsAuthResolution { @@ -105,7 +118,39 @@ type ModelsAuthResolverFactory = ( outcomes: CatalogGatherProviderAuthOutcome[], ) => ModelsAuthResolver; -const gatherInflight = new Map>(); +interface CapturedModelsRequest { + readonly method: "GET"; + readonly url: string; + readonly headersWithoutCredential: Readonly>; + readonly headersWithCredential: Readonly>; +} + +interface CapturedProviderGather { + readonly name: string; + readonly provider: OcxProviderConfig; + readonly discovery: ResolvedProviderModelDiscovery; + readonly policy: CatalogProviderDiscoveryPolicySnapshot; + readonly request: CapturedModelsRequest; + readonly observedAuth?: ModelsAuthResolution; +} + +interface GatherFlightCapture { + readonly discoveryPolicyIdentity: string; + readonly discoveryPolicySnapshots: readonly CatalogProviderDiscoveryPolicySnapshot[]; + readonly providers: readonly CapturedProviderGather[]; + readonly authResolver: ModelsAuthResolver; + readonly providerAuthOutcomes: readonly CatalogGatherProviderAuthOutcome[]; + readonly openAiApiPolicy: CatalogTrustedOpenAiApiPolicySnapshot; +} + +interface GatherInflightEntry { + readonly discoveryPolicyIdentity: string; + readonly promise: Promise; +} + +const gatherInflight = new Map(); +const CATALOG_GATHER_AUTHORITY_KEY = randomBytes(32); +const REQUEST_CREDENTIAL_SENTINEL = `ocx-catalog-credential-${randomBytes(16).toString("hex")}`; const MAX_CONCURRENT_CATALOG_GATHERS = 8; const gatherGate = createAdmissionGate("catalog_gathers", MAX_CONCURRENT_CATALOG_GATHERS); @@ -131,6 +176,232 @@ function stableJson(value: unknown): string { }); } +function framed(value: string): string { + return `${Buffer.byteLength(value, "utf8")}:${value}`; +} + +function canonicalAuthorityEncoding(value: unknown): string { + if (value === null) return "null"; + if (value === undefined) return "undefined"; + if (typeof value === "string") return `string${framed(value)}`; + if (typeof value === "boolean") return value ? "boolean1" : "boolean0"; + if (typeof value === "number") { + if (!Number.isFinite(value)) throw new TypeError("Catalog authority cannot encode a non-finite number."); + const encoded = Object.is(value, -0) ? "-0" : String(value); + return `number${framed(encoded)}`; + } + if (Array.isArray(value)) { + return `array${value.length}:${value.map(item => framed(canonicalAuthorityEncoding(item))).join("")}`; + } + if (typeof value === "object") { + const record = value as Record; + const keys = Object.keys(record).sort((left, right) => left.localeCompare(right)); + return `object${keys.length}:${keys.map(key => ( + `${framed(key)}${framed(canonicalAuthorityEncoding(record[key]))}` + )).join("")}`; + } + throw new TypeError(`Catalog authority cannot encode ${typeof value}.`); +} + +function keyedGatherIdentity(domain: string, value: unknown): string { + return createHmac("sha256", CATALOG_GATHER_AUTHORITY_KEY) + .update(framed(domain)) + .update(framed(canonicalAuthorityEncoding(value))) + .digest("hex"); +} + +function keyedGatherBytesIdentity(domain: string, value: Uint8Array): string { + return createHmac("sha256", CATALOG_GATHER_AUTHORITY_KEY) + .update(framed(domain)) + .update(`${value.byteLength}:`) + .update(value) + .digest("hex"); +} + +export function createCatalogGatherAuthorityIdentity( + snapshot: CatalogAdmissionSnapshot, + sourceEvidence: CatalogSourceEvidence, + processLocal: CatalogProcessLocalEvidence, + discoveryPolicies: readonly CatalogProviderDiscoveryPolicySnapshot[], +): CatalogGatherAuthorityIdentity { + const sourceEvidenceIdentity = keyedGatherIdentity("catalog-source-evidence-v1", sourceEvidence); + const processLocalEvidenceIdentity = keyedGatherIdentity("catalog-process-local-v1", processLocal); + const discoveryPolicyIdentity = keyedGatherIdentity("catalog-discovery-policy-v1", discoveryPolicies); + return Object.freeze({ + version: 1 as const, + authorityId: keyedGatherIdentity("catalog-authority-v1", { + admittedConfig: snapshot.configIdentity, + discoveryPolicyIdentity, + sourceEvidenceIdentity, + processLocalEvidenceIdentity, + }), + admittedConfig: Object.freeze({ + ...snapshot.configIdentity, + generation: Object.freeze({ ...snapshot.configIdentity.generation }), + }), + authSnapshotIdentity: keyedGatherIdentity( + "catalog-auth-v1", + sourceEvidence.conditional["provider-auth-selection"], + ), + discoveryPolicyIdentity, + nativeCatalogSourceIdentity: keyedGatherIdentity( + "catalog-native-v1", + sourceEvidence.conditional["native-catalog-selection"], + ), + sourceEvidenceIdentity, + processLocalEvidenceIdentity, + }); +} + +function detachedClone(value: T): T { + if (Array.isArray(value)) return value.map(item => detachedClone(item)) as T; + if (value && typeof value === "object") { + const clone: Record = {}; + for (const key of Object.keys(value)) { + clone[key] = detachedClone((value as Record)[key]); + } + return clone as T; + } + return value; +} + +function recursivelyFreeze(value: T): T { + if (!value || typeof value !== "object" || Object.isFrozen(value)) return value; + for (const nested of Object.values(value as Record)) recursivelyFreeze(nested); + return Object.freeze(value); +} + +function detachedFrozen(value: T): T { + return recursivelyFreeze(detachedClone(value)); +} + +function capturedField( + value: T | undefined, + key: K, +): CatalogDiscoveryPolicyField { + if (!value || !Object.hasOwn(value, key)) return Object.freeze({ state: "absent" }); + return detachedFrozen({ state: "present" as const, value: value[key] }); +} + +function captureTrustedOpenAiApiPolicy( + name: string, + registryTransportMatch: boolean, +): CatalogTrustedOpenAiApiPolicySnapshot { + if (name !== OPENAI_API_PROVIDER_ID) return Object.freeze({ state: "unused" }); + if (!registryTransportMatch) return Object.freeze({ state: "transport-mismatch" }); + const entry = getProviderRegistryEntry(name); + if (!entry?.models) return Object.freeze({ state: "registry-models-absent" }); + return detachedFrozen({ + state: "captured" as const, + models: entry.models, + ...(entry.modelContextWindows ? { modelContextWindows: entry.modelContextWindows } : {}), + ...(entry.modelMaxInputTokens ? { modelMaxInputTokens: entry.modelMaxInputTokens } : {}), + ...(entry.modelInputModalities ? { modelInputModalities: entry.modelInputModalities } : {}), + ...(entry.modelReasoningEfforts ? { modelReasoningEfforts: entry.modelReasoningEfforts } : {}), + }); +} + +function captureModelsRequest( + name: string, + provider: OcxProviderConfig, + observedAuth: ModelsAuthResolution | undefined, +): CapturedModelsRequest { + const observed = observedAuth + ? { oauthApiBaseUrl: observedAuth.oauthApiBaseUrl } + : undefined; + const withoutCredential = buildModelsRequest(provider, undefined, name, observed); + const withCredential = buildModelsRequest(provider, REQUEST_CREDENTIAL_SENTINEL, name, observed); + if (withoutCredential.url !== withCredential.url) { + throw new TypeError(`Provider model discovery URL for ${name} depends on credential bytes.`); + } + return detachedFrozen({ + method: "GET" as const, + url: withoutCredential.url, + headersWithoutCredential: withoutCredential.headers, + headersWithCredential: withCredential.headers, + }); +} + +function captureProviderGather( + name: string, + configured: OcxProviderConfig, + authResolver: ModelsAuthResolver, +): CapturedProviderGather { + const enriched = detachedClone(configured); + enrichProviderFromRegistry(name, enriched); + const provider = recursivelyFreeze(enriched); + const observedAuth = authResolver.kind === "observed" + && provider.authMode !== "forward" + && provider.liveModels !== false + ? authResolver.resolve(name, provider) + : undefined; + const request = captureModelsRequest(name, provider, observedAuth); + const resolved = resolveProviderModelDiscovery(name, provider); + const discovery = detachedFrozen({ + ...(resolved.spec ? { spec: resolved.spec } : {}), + maxResponseBytes: resolved.maxResponseBytes, + maxModels: resolved.maxModels, + }); + const registryTransportMatch = providerMatchesRegistryTransport(name, provider); + const trustedOpenAiApi = captureTrustedOpenAiApiPolicy(name, registryTransportMatch); + const policy = detachedFrozen({ + provider: name, + registryTransportMatch, + location: { + spec: discovery.spec ? "present" as const : "absent" as const, + url: capturedField(discovery.spec, "url"), + path: capturedField(discovery.spec, "path"), + query: capturedField(discovery.spec, "query"), + }, + finalMethod: request.method, + finalUrl: request.url, + filter: capturedField(discovery.spec, "filter"), + maxResponseBytes: discovery.maxResponseBytes, + maxModels: discovery.maxModels, + trustedOpenAiApi, + }); + return Object.freeze({ + name, + provider, + discovery, + policy, + request, + ...(observedAuth ? { observedAuth: Object.freeze({ ...observedAuth }) } : {}), + }); +} + +function captureGatherFlight( + config: OcxConfig, + createAuthResolver: ModelsAuthResolverFactory, +): GatherFlightCapture { + const providerAuthOutcomes: CatalogGatherProviderAuthOutcome[] = []; + const authResolver = createAuthResolver(providerAuthOutcomes); + const providers = Object.entries(config.providers) + .filter(([, provider]) => provider.disabled !== true) + .map(([name, provider]) => captureProviderGather(name, provider, authResolver)); + const discoveryPolicySnapshots = Object.freeze(providers.map(provider => provider.policy)); + return Object.freeze({ + discoveryPolicyIdentity: keyedGatherIdentity("catalog-discovery-policy-v1", discoveryPolicySnapshots), + discoveryPolicySnapshots, + providers: Object.freeze(providers), + authResolver, + providerAuthOutcomes: Object.freeze([...providerAuthOutcomes]), + openAiApiPolicy: providers.find(provider => provider.name === OPENAI_API_PROVIDER_ID)?.policy.trustedOpenAiApi + ?? Object.freeze({ state: "unused" as const }), + }); +} + +function materializeCapturedHeaders( + request: CapturedModelsRequest, + apiKey: string | undefined, +): Record { + const source = apiKey ? request.headersWithCredential : request.headersWithoutCredential; + return Object.fromEntries(Object.entries(source).map(([name, value]) => [ + name, + apiKey ? value.split(REQUEST_CREDENTIAL_SENTINEL).join(apiKey) : value, + ])); +} + function providerCatalogFingerprint(name: string, prov: OcxProviderConfig): Record { return { n: name, @@ -470,12 +741,12 @@ function observedModelsAuthResolver( } async function fetchProviderModelsWithAuth( - name: string, - prov: OcxProviderConfig, + captured: CapturedProviderGather, ttlMs: number, contextCap: number | undefined, resolveAuth: ModelsAuthResolver, ): Promise { + const { name, provider: prov, discovery, request } = captured; if (prov.authMode === "forward") return []; // ChatGPT backend has no /models const seedVertexDefault = prov.adapter === "google" && prov.googleMode === "vertex" @@ -493,9 +764,9 @@ async function fetchProviderModelsWithAuth( clearProviderDiscoveryStatus(name); return configured; } - const auth: ModelsAuthResolution = resolveAuth.kind === "refreshing" + const auth: ModelsAuthResolution = captured.observedAuth ?? (resolveAuth.kind === "refreshing" ? { apiKey: await resolveModelsAuthToken(name, prov), observed: false } - : resolveAuth.resolve(name, prov); + : resolveAuth.resolve(name, prov)); const apiKey = auth.apiKey; // A configured default is a real callable selector and must remain discoverable when a // compatible provider's live /models request fails (issue #308). Keep this separate from the @@ -557,13 +828,8 @@ async function fetchProviderModelsWithAuth( const stale = getStaleCached(name); return stale ? withVertexDefaultSeed(applyConfigHintsToCachedModels(name, prov, stale, contextCap)) : failedDiscoveryConfigured; } - const discovery = resolveProviderModelDiscovery(name, prov); - const { url, headers } = buildModelsRequest( - prov, - apiKey, - name, - auth.observed ? { oauthApiBaseUrl: auth.oauthApiBaseUrl } : undefined, - ); + const url = request.url; + const headers = materializeCapturedHeaders(request, apiKey); const urlClass = new URL(url).hostname.endsWith("aiplatform.googleapis.com") ? "vertex-aiplatform" : "provider-models"; @@ -715,7 +981,8 @@ export async function fetchProviderModels( ttlMs: number, contextCap?: number, ): Promise { - return fetchProviderModelsWithAuth(name, prov, ttlMs, contextCap, refreshingModelsAuthResolver); + const captured = captureProviderGather(name, prov, refreshingModelsAuthResolver); + return fetchProviderModelsWithAuth(captured, ttlMs, contextCap, refreshingModelsAuthResolver); } export function shouldExposeProviderModel(providerName: string, modelId: string): boolean { @@ -778,7 +1045,7 @@ export async function gatherRoutedModelsForCatalogGather( : Uint8Array.from(evidence.authStoreBuffer); const authIdentity = authStoreBuffer === null ? "absent" - : createHash("sha256").update(authStoreBuffer).digest("hex"); + : keyedGatherBytesIdentity("catalog-observed-auth-v1", authStoreBuffer); return gatherRoutedModelsWithAuth( config, `observed:${authIdentity}:${gatherFlightKey(config)}`, @@ -793,20 +1060,38 @@ async function gatherRoutedModelsWithAuth( createAuthResolver: ModelsAuthResolverFactory, options?: GatherRoutedModelsOptions, ): Promise { - let promise = gatherInflight.get(key); - if (!promise) { + const capture = captureGatherFlight(config, createAuthResolver); + const bucket = gatherInflight.get(key) ?? []; + let entry = bucket.find(candidate => ( + candidate.discoveryPolicyIdentity === capture.discoveryPolicyIdentity + )); + if (!entry) { const lease = gatherGate.tryAcquire(); if (!lease) throw new CatalogGatherBusyError(); // Claim the slot synchronously before any await so same-key callers join this flight. - // Distinct keys keep their own entries — a second config must not evict the first. - const flight = gatherRoutedModelsUncached(config, createAuthResolver).finally(() => { - if (gatherInflight.get(key) === flight) gatherInflight.delete(key); + // Distinct authorities retain separate entries even when their legacy bucket matches. + let ownedEntry!: GatherInflightEntry; + const flight = gatherRoutedModelsUncached(config, capture).finally(() => { + const current = gatherInflight.get(key); + const index = current?.indexOf(ownedEntry) ?? -1; + if (current && index >= 0) current.splice(index, 1); + if (current?.length === 0) gatherInflight.delete(key); lease.release(); }); - gatherInflight.set(key, flight); - promise = flight; + ownedEntry = Object.freeze({ + discoveryPolicyIdentity: capture.discoveryPolicyIdentity, + promise: flight, + }); + bucket.push(ownedEntry); + gatherInflight.set(key, bucket); + entry = ownedEntry; } - const { models, comboOmissions, providerAuthOutcomes } = await promise; + const { + models, + comboOmissions, + providerAuthOutcomes, + discoveryPolicySnapshots, + } = await entry.promise; if (options?.comboOmissions) { options.comboOmissions.length = 0; options.comboOmissions.push(...comboOmissions); @@ -815,17 +1100,21 @@ async function gatherRoutedModelsWithAuth( options.providerAuthOutcomes.length = 0; options.providerAuthOutcomes.push(...providerAuthOutcomes); } + if (options?.discoveryPolicySnapshots) { + options.discoveryPolicySnapshots.length = 0; + options.discoveryPolicySnapshots.push(...discoveryPolicySnapshots); + } return models; } async function gatherRoutedModelsUncached( config: OcxConfig, - createAuthResolver: ModelsAuthResolverFactory, + capture: GatherFlightCapture, ): Promise { // Flight-local list: joiners copy from the resolved promise, not a process-global last write. const localOmissions: ComboCatalogOmission[] = []; - const localProviderAuthOutcomes: CatalogGatherProviderAuthOutcome[] = []; - const resolveAuth = createAuthResolver(localProviderAuthOutcomes); + const localProviderAuthOutcomes = capture.providerAuthOutcomes; + const resolveAuth = capture.authResolver; const ttlMs = config.modelCacheTtlMs ?? DEFAULT_MODEL_CACHE_TTL_MS; // Persisted provider entries can predate newer registry fields (noVisionModels, // modelInputModalities, ...). The ROUTER merges registry seeds at request time @@ -833,24 +1122,21 @@ async function gatherRoutedModelsUncached( // same merged view or its advertisements drift from actual proxy behavior (e.g. a // vision-sidecar model advertised text-only, blocking image attachments app-side). // Enrich a CLONE: hydrated defaults must never leak into the persisted config. - const activeProviders = Object.entries(config.providers) - .filter(([, prov]) => prov.disabled !== true) - .map(([name, prov]): [string, OcxProviderConfig] => { - const enriched = { ...prov }; - enrichProviderFromRegistry(name, enriched); - return [name, enriched]; - }); + const activeProviders = capture.providers; const lists = await Promise.all( - activeProviders.map(([name, prov]) => fetchProviderModelsWithAuth( - name, - prov, + activeProviders.map(provider => fetchProviderModelsWithAuth( + provider, ttlMs, - providerContextCap(config, name), + providerContextCap(config, provider.name), resolveAuth, )), ); - const apiAugmented = augmentRoutedModelsWithRegistryOpenAiApiRows(lists.flat(), config); - const all = augmentRoutedModelsWithJawcodeMetadata(apiAugmented, activeProviders.map(([name]) => name), config.providers, config) + const apiAugmented = augmentRoutedModelsWithCapturedOpenAiApiRows( + lists.flat(), + config, + capture.openAiApiPolicy, + ); + const all = augmentRoutedModelsWithJawcodeMetadata(apiAugmented, activeProviders.map(provider => provider.name), config.providers, config) // Drop image/video generation models (e.g. Grok image/video) by default. Cursor's static catalog // intentionally mirrors Cursor's public model table, including Gemini image preview, so the // exposure decision goes through shouldExposeRoutedModel (single choke point). @@ -915,7 +1201,7 @@ async function gatherRoutedModelsUncached( all.sort((a, b) => (a.provider === b.provider ? a.id.localeCompare(b.id) : a.provider.localeCompare(b.provider))); // Enriched (registry-hydrated) provider clones, keyed by name — the same view used above so // custom rows get the same noVisionModels / inputModalities treatment as discovered rows. - const enrichedByName = new Map(activeProviders); + const enrichedByName = new Map(activeProviders.map(provider => [provider.name, provider.provider])); const customModels = (config.customModels ?? []).map(cm => { const rawProvider = config.providers[cm.provider]; const supportsReasoningSummaries = configuredReasoningSummarySupport(rawProvider, cm.modelId); @@ -949,6 +1235,7 @@ async function gatherRoutedModelsUncached( models: [...deduped, ...customModels], comboOmissions: localOmissions, providerAuthOutcomes: localProviderAuthOutcomes, + discoveryPolicySnapshots: capture.discoveryPolicySnapshots, }; } @@ -958,15 +1245,28 @@ export function augmentRoutedModelsWithRegistryOpenAiApiRows( ): CatalogModel[] { const configured = config.providers[OPENAI_API_PROVIDER_ID]; if (!configured || configured.disabled === true || !providerMatchesRegistryTransport(OPENAI_API_PROVIDER_ID, configured)) return models; - const entry = getProviderRegistryEntry(OPENAI_API_PROVIDER_ID); - if (!entry?.models) return models; + return augmentRoutedModelsWithCapturedOpenAiApiRows( + models, + config, + captureTrustedOpenAiApiPolicy(OPENAI_API_PROVIDER_ID, true), + ); +} + +function augmentRoutedModelsWithCapturedOpenAiApiRows( + models: CatalogModel[], + config: OcxConfig, + policy: CatalogTrustedOpenAiApiPolicySnapshot, +): CatalogModel[] { + if (policy.state !== "captured" || !policy.models) return models; + const configured = config.providers[OPENAI_API_PROVIDER_ID]; + if (!configured || configured.disabled === true) return models; const existingById = new Map( models.filter(model => model.provider === OPENAI_API_PROVIDER_ID).map(model => [model.id, model]), ); - const trustedRows = entry.models.map((id): CatalogModel => { - const officialContext = entry.modelContextWindows?.[id]; - const officialMaxInput = entry.modelMaxInputTokens?.[id]; + const trustedRows = policy.models.map((id): CatalogModel => { + const officialContext = policy.modelContextWindows?.[id]; + const officialMaxInput = policy.modelMaxInputTokens?.[id]; const userContext = configured.modelContextWindows?.[id] ?? configured.contextWindow; const userMaxInput = configured.modelMaxInputTokens?.[id]; const providerCap = providerContextCap(config, OPENAI_API_PROVIDER_ID); @@ -982,8 +1282,8 @@ export function augmentRoutedModelsWithRegistryOpenAiApiRows( owned_by: OPENAI_API_PROVIDER_ID, ...(contextWindow ? { contextWindow } : {}), ...(maxInputTokens ? { maxInputTokens } : {}), - ...(entry.modelInputModalities?.[id] ? { inputModalities: [...entry.modelInputModalities[id]!] } : {}), - ...(entry.modelReasoningEfforts?.[id] ? { reasoningEfforts: [...entry.modelReasoningEfforts[id]!] } : {}), + ...(policy.modelInputModalities?.[id] ? { inputModalities: [...policy.modelInputModalities[id]!] } : {}), + ...(policy.modelReasoningEfforts?.[id] ? { reasoningEfforts: [...policy.modelReasoningEfforts[id]!] } : {}), }; }); diff --git a/src/codex/convergence-types.ts b/src/codex/convergence-types.ts index 964b46459..b29e4c190 100644 --- a/src/codex/convergence-types.ts +++ b/src/codex/convergence-types.ts @@ -14,6 +14,7 @@ * behavior at its own commit, which a runtime placeholder here would break. */ import type { OcxConfig } from "../types"; +import type { ProviderModelDiscoveryFilter } from "../providers/registry"; /** * The non-CAS JSON record for the Codex integration. @@ -389,6 +390,37 @@ export interface CatalogProcessLocalEvidence { readonly bundledCatalog: CatalogProcessLocalObservation; } +export type CatalogDiscoveryPolicyField = + | Readonly<{ state: "absent" }> + | Readonly<{ state: "present"; value: T }>; + +export interface CatalogTrustedOpenAiApiPolicySnapshot { + readonly state: "unused" | "transport-mismatch" | "registry-models-absent" | "captured"; + readonly models?: readonly string[]; + readonly modelContextWindows?: Readonly>; + readonly modelMaxInputTokens?: Readonly>; + readonly modelInputModalities?: Readonly>; + readonly modelReasoningEfforts?: Readonly>; +} + +/** Detached effective policy consumed by one enabled provider inside a gather flight. */ +export interface CatalogProviderDiscoveryPolicySnapshot { + readonly provider: string; + readonly registryTransportMatch: boolean; + readonly location: Readonly<{ + readonly spec: "absent" | "present"; + readonly url: CatalogDiscoveryPolicyField; + readonly path: CatalogDiscoveryPolicyField; + readonly query: CatalogDiscoveryPolicyField> | undefined>; + }>; + readonly finalMethod: "GET"; + readonly finalUrl: string; + readonly filter: CatalogDiscoveryPolicyField; + readonly maxResponseBytes: number; + readonly maxModels: number; + readonly trustedOpenAiApi: CatalogTrustedOpenAiApiPolicySnapshot; +} + /** Non-secret-bearing identity of every authority input admitted to one gather flight. */ export interface CatalogGatherAuthorityIdentity { readonly version: 1; @@ -402,6 +434,7 @@ export interface CatalogGatherAuthorityIdentity { readonly snapshotIdentity: string; }>; readonly authSnapshotIdentity: string; + readonly discoveryPolicyIdentity: string; readonly nativeCatalogSourceIdentity: string; readonly sourceEvidenceIdentity: string; readonly processLocalEvidenceIdentity: string; diff --git a/tests/codex-gather-authority.test.ts b/tests/codex-gather-authority.test.ts new file mode 100644 index 000000000..4361343ac --- /dev/null +++ b/tests/codex-gather-authority.test.ts @@ -0,0 +1,194 @@ +import { afterEach, describe, expect, spyOn, test } from "bun:test"; +import { createHash } from "node:crypto"; + +import { + clearGatherRoutedModelsInflight, + gatherRoutedModels, +} from "../src/codex/catalog"; +import { clearModelCache } from "../src/codex/model-cache"; +import { PROVIDER_REGISTRY, type ProviderModelDiscoverySpec } from "../src/providers/registry"; +import type { OcxConfig } from "../src/types"; +import { withStubbedProviderFetch } from "./helpers/catalog-provider-fetch"; +import { withRegistryDiscovery } from "./helpers/provider-registry-discovery"; + +const originalFetch = globalThis.fetch; + +function deferred(): { readonly promise: Promise; readonly resolve: () => void } { + let resolve!: () => void; + const promise = new Promise(done => { resolve = done; }); + return { promise, resolve }; +} + +function togetherConfig(apiKey = "together-authority-secret"): OcxConfig { + return withStubbedProviderFetch({ + port: 10100, + defaultProvider: "together", + modelCacheTtlMs: 0, + providers: { + together: { + adapter: "openai-chat", + baseUrl: "https://api.together.xyz/v1", + authMode: "key", + apiKey, + models: ["safe-fallback"], + }, + }, + }); +} + +const strictDiscovery: ProviderModelDiscoverySpec = { + maxModels: 2, + filter: { + allOf: [{ path: ["type"], equalsAny: ["chat"] }], + }, +}; + +afterEach(() => { + globalThis.fetch = originalFetch; + clearModelCache(); + clearGatherRoutedModelsInflight(); +}); + +describe("catalog gather discovery-policy authority", () => { + test("different live registry policies cannot share a flight", async () => { + const config = togetherConfig(); + const body = { + data: [ + { id: "chat-one", type: "chat" }, + { id: "chat-two", type: "chat" }, + { id: "embedding-three", type: "embedding" }, + ], + }; + const warning = spyOn(console, "warn").mockImplementation(() => {}); + try { + globalThis.fetch = (async () => Response.json(body)) as typeof fetch; + const control = await withRegistryDiscovery( + "together", + strictDiscovery, + () => gatherRoutedModels(config), + { preserveCustomDestination: true }, + ); + expect(control.filter(model => model.provider === "together").map(model => model.id)) + .toEqual(["safe-fallback"]); + expect(warning.mock.calls.flat().join(" ")).toContain("2-row model limit"); + + clearModelCache("together"); + clearGatherRoutedModelsInflight(); + warning.mockClear(); + const firstResponse = deferred(); + let fetchCount = 0; + globalThis.fetch = (async () => { + fetchCount += 1; + if (fetchCount === 1) await firstResponse.promise; + return Response.json(body); + }) as typeof fetch; + try { + let first!: Promise>>; + await withRegistryDiscovery( + "together", + { maxModels: 3 }, + () => { first = gatherRoutedModels(config); }, + { preserveCustomDestination: true }, + ); + expect(fetchCount).toBe(1); + + let second!: Promise>>; + await withRegistryDiscovery( + "together", + strictDiscovery, + () => { second = gatherRoutedModels(config); }, + { preserveCustomDestination: true }, + ); + expect(fetchCount).toBe(2); + + firstResponse.resolve(); + const [permissive, strict] = await Promise.all([first, second]); + expect(fetchCount).toBe(2); + expect(permissive.filter(model => model.provider === "together").map(model => model.id)) + .toEqual(["chat-one", "chat-two", "embedding-three"]); + expect(strict.filter(model => model.provider === "together").map(model => model.id)) + .toEqual(["safe-fallback"]); + expect(warning.mock.calls.flat().join(" ")).toContain("2-row model limit"); + } finally { + firstResponse.resolve(); + } + } finally { + warning.mockRestore(); + } + }); + + test("a flight uses its captured transport non-match after the registry override is restored", async () => { + const config = withStubbedProviderFetch({ + port: 10100, + defaultProvider: "openai-apikey", + modelCacheTtlMs: 0, + providers: { + "openai-apikey": { + adapter: "openai-responses", + baseUrl: "https://custom-openai.example/v1", + authMode: "key", + apiKey: "custom-openai-secret", + }, + }, + }); + const responseGate = deferred(); + let fetchCount = 0; + globalThis.fetch = (async () => { + fetchCount += 1; + await responseGate.promise; + return Response.json({ data: [{ id: "custom-only" }] }); + }) as typeof fetch; + + let pending!: Promise>>; + await withRegistryDiscovery( + "openai-apikey", + {}, + () => { pending = gatherRoutedModels(config); }, + { preserveCustomDestination: true }, + ); + expect(fetchCount).toBe(1); + + const originalFind = PROVIDER_REGISTRY.find; + PROVIDER_REGISTRY.find = function forbiddenPostLookupRegistryRead() { + throw new Error("post-lookup provider registry read"); + } as typeof PROVIDER_REGISTRY.find; + try { + responseGate.resolve(); + const models = await pending; + expect(models.filter(model => model.provider === "openai-apikey").map(model => model.id)) + .toEqual(["custom-only"]); + } finally { + PROVIDER_REGISTRY.find = originalFind; + responseGate.resolve(); + } + }); + + test("credentials and private authority identities stay out of logs and serialized results", async () => { + const credential = "catalog-authority-privacy-secret"; + const plainDigest = createHash("sha256").update(credential).digest("hex"); + const config = togetherConfig(credential); + const warn = spyOn(console, "warn").mockImplementation(() => {}); + const error = spyOn(console, "error").mockImplementation(() => {}); + const log = spyOn(console, "log").mockImplementation(() => {}); + globalThis.fetch = (async () => Response.json({ data: [{ id: "privacy-model" }] })) as typeof fetch; + try { + const models = await withRegistryDiscovery( + "together", + { filter: { allOf: [{ path: ["id"], equalsAny: ["privacy-model"] }] } }, + () => gatherRoutedModels(config), + { preserveCustomDestination: true }, + ); + const observable = JSON.stringify({ + models, + logs: [warn, error, log].map(spy => spy.mock.calls), + }); + expect(observable).not.toContain(credential); + expect(observable).not.toContain(plainDigest); + expect(observable).not.toMatch(/[a-f0-9]{64}/i); + } finally { + warn.mockRestore(); + error.mockRestore(); + log.mockRestore(); + } + }); +}); From 8f403a138a8b719090041a965b3e94a04955e9ed Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Wed, 5 Aug 2026 00:01:56 +0900 Subject: [PATCH 075/163] feat(codex): the catalog writers we kept now queue behind the same lock MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Serializing the convergence commit alone left four first-party paths writing catalog bytes with no lock at all: /api/sync through refresh, startup cache invalidation, `ocx sync-cache`, and native restore. Each now takes K. Serializing the write is not enough on its own, because /api/sync reads the catalog, awaits provider gathering, and only then writes from what it captured — so a writer that gathered first and acquired K second would replace a newer catalog while legitimately holding the lock. It therefore carries its source evidence across the await and revalidates under K; startup, sync-cache and restore recompute their read and derivation inside the lock instead, because theirs are synchronous and there is nothing to carry. Two things the first pass got wrong, both caught by running it rather than reading it: Startup began throwing. `getCodexHome()` raises when CODEX_HOME names a missing directory, and the acquisition sat outside any guard, so a machine with no Codex home could no longer start the proxy at all. Invalidation is best-effort and is guarded again. Every sync stopped writing. The freshness evidence covered the process-local runtime and bundled caches, but gathering RESOLVES the runtime — so the check detected its own side effect and refused, reporting catalogWritten:false for a catalog nobody else had touched. Those epochs are now baselined after our own observation while the filesystem bytes stay baselined before the await, which keeps the question the right way round: did somebody ELSE move the world while we waited. Removing that filesystem check turns both clobber regressions red. --- src/cli/index.ts | 9 +- src/codex/catalog/sync.ts | 273 ++++++++++++++-- src/codex/inject.ts | 12 +- src/codex/internal/catalog-writer.ts | 4 +- src/server/index.ts | 15 +- tests/codex-app-server-processes.test.ts | 14 +- .../codex-retained-root-serialization.test.ts | 308 ++++++++++++++++++ 7 files changed, 603 insertions(+), 32 deletions(-) create mode 100644 tests/codex-retained-root-serialization.test.ts diff --git a/src/cli/index.ts b/src/cli/index.ts index 54e3ff822..8d8265446 100755 --- a/src/cli/index.ts +++ b/src/cli/index.ts @@ -848,9 +848,14 @@ switch (command) { } case "sync-cache": { const restartCodex = args.slice(1).includes("--restart-codex"); - const { invalidateCodexModelsCache } = await import("../codex/catalog"); + const { withCatalogWriteSerialization } = await import("../codex/catalog-write-serialization"); + const { invalidateCodexModelsCacheWithPermit } = await import("../codex/catalog/sync"); + const { getCodexHome } = await import("../codex/paths"); + const owningCodexHome = getCodexHome(); + const invalidated = withCatalogWriteSerialization(owningCodexHome, permit => + invalidateCodexModelsCacheWithPermit(permit, owningCodexHome)); // Only warn/restart when models_cache was actually rewritten from a readable catalog. - if (invalidateCodexModelsCache()) { + if (invalidated.kind === "completed" && invalidated.value) { const { afterCatalogWriteHandleAppServers } = await import("../codex/app-server-processes"); afterCatalogWriteHandleAppServers({ restart: restartCodex, log: console }); } diff --git a/src/codex/catalog/sync.ts b/src/codex/catalog/sync.ts index 187094d12..545151bf8 100644 --- a/src/codex/catalog/sync.ts +++ b/src/codex/catalog/sync.ts @@ -1,9 +1,9 @@ import { execFileSync } from "node:child_process"; import { createHash } from "node:crypto"; -import { copyFileSync, existsSync, mkdirSync, readFileSync, realpathSync } from "node:fs"; +import { existsSync, readFileSync } from "node:fs"; import { delimiter, dirname, join, resolve } from "node:path"; -import { atomicWriteFile, expandUserPath, getConfigDir, websocketsEnabled } from "../../config"; -import { CODEX_CONFIG_PATH, CODEX_MODELS_CACHE_PATH, DEFAULT_CATALOG_PATH, readRootTomlString, resolveCodexConfigPath } from "../paths"; +import { expandUserPath, websocketsEnabled } from "../../config"; +import { CODEX_CONFIG_PATH, CODEX_MODELS_CACHE_PATH, DEFAULT_CATALOG_PATH, getCodexHome, readRootTomlString, resolveCodexConfigPath } from "../paths"; import { clearModelCache, DEFAULT_MODEL_CACHE_TTL_MS, getFreshCached, getStaleCached, isModelsFetchCoolingDown, markModelsFetchFailure, setCached } from "../model-cache"; import { buildModelsRequest, resolveModelsAuthToken } from "../../oauth"; import type { OcxConfig, OcxProviderConfig } from "../../types"; @@ -31,15 +31,30 @@ import { redactSecretString } from "../../lib/redact"; import upstreamModelsSnapshot from "../data/upstream-models.json"; -import { activeCodexModelsCachePath, applyJawcodeCatalogMetadata, applyMultiAgentMode, applyNativeOpenAiContextOverride, catalogModelSlug, ensureCatalogBackup, ensureStrictCatalogFields, findNativeTemplate, isRoutedModelCompatibilityExcluded, normalizeRoutedCatalogEntry, normalizeServiceTiers, readCatalog, readCatalogBackup, readCodexCatalogPath, readNativeBaseline } from "./parsing"; -import type { CatalogModel, MultiAgentMode, RawEntry } from "./parsing"; +import { activeCodexModelsCachePath, applyJawcodeCatalogMetadata, applyMultiAgentMode, applyNativeOpenAiContextOverride, catalogBackupPathFor, catalogHasRoutedEntries, catalogModelSlug, ensureStrictCatalogFields, findNativeTemplate, isDefaultCatalogPath, isRoutedModelCompatibilityExcluded, legacyCatalogBackupPath, normalizeRoutedCatalogEntry, normalizeServiceTiers, readCatalog, readCatalogBackup, readCodexCatalogPath, readNativeBaseline } from "./parsing"; +import type { CatalogModel, MultiAgentMode, RawCatalog, RawEntry } from "./parsing"; import { applyNativeVisibility, disabledNativeSlugs, isUnsupportedOpenAiNativeSlug, nativeOpenAiSlugs, shouldUpgradeToUpstreamEntry, upstreamNativeEntry } from "./metadata"; -import { loadCatalogForSync, resetBundledCatalogCacheForTests } from "./bundled"; +import { + bundledCatalogCacheState, + loadBundledCodexCatalog, + resetBundledCatalogCacheForTests, +} from "./bundled"; import { isMultiAgentV2Enabled } from "../features"; import { applyCatalogModelMetadata, applyReasoningLevels, catalogEntryEfforts, clampCatalogModelsToCodexSupport, ensureGpt56ReasoningLevels, ensureUltraReasoningLevel, isGpt56NativeSlug } from "./effort"; import { clearGatherRoutedModelsInflight, filterCatalogVisibleModels, gatherRoutedModels, lastDropWarnSignature } from "./provider-fetch"; import { clearLastComboCatalogOmissions, comboCatalogWarningSignatures, comboMasqueradeCollisionWarnings, exactComboCatalogSlugs, openAiApiCollisionWarnings, resolveSlugAliasCollisions, slugAliasCollisionWarnings, warnComboMasqueradeCollisionOnce } from "./aggregation"; import type { ComboCatalogOmission } from "./aggregation"; +import { + withCatalogWriteSerialization, + type CatalogWritePermit, +} from "../catalog-write-serialization"; +import { + publishHashedCodexCatalogBackup, + publishLegacyCodexCatalogBackup, + replaceActiveCodexCatalog, + replaceCodexModelsCache, +} from "../internal/catalog-writer"; +import { peekCodexRuntimeProcessCache } from "../runtime"; export const MAX_SPAWN_AGENT_MODEL_OVERRIDES = 5; @@ -504,30 +519,167 @@ export function mergeCatalogEntriesForSync( return applyMultiAgentMode(applyNativeVisibility(mergedEntries, disabledNative), multiAgentMode, isMultiAgentV2Enabled()); } -export async function syncCatalogModels(config: OcxConfig): Promise<{ +interface RetainedCatalogSyncRead { + readonly catalogPath: string; + readonly catalog: RawCatalog; + readonly onDiskCatalog: RawCatalog | null; + readonly evidence: string; + /** + * Process-local epochs, baselined AFTER our own gather rather than with the + * filesystem bytes above. See `retainedCatalogProcessEvidence`. + */ + readonly processEvidence: string; +} + +interface RetainedCatalogSyncResult { added: number; path: string; catalogWritten: boolean; comboOmissions: ComboCatalogOmission[]; -}> { +} + +interface RetainedCatalogSyncWrite { + readonly config: OcxConfig; + readonly goModels: CatalogModel[]; + readonly comboOmissions: ComboCatalogOmission[]; + readonly read: RetainedCatalogSyncRead; + readonly permit: CatalogWritePermit; + readonly owningCodexHome: string; +} + +function optionalFileBytes(path: string): string | null { + try { + return readFileSync(path).toString("base64"); + } catch (error) { + if ((error as NodeJS.ErrnoException | undefined)?.code === "ENOENT") return null; + throw error; + } +} + +function loadCatalogForRetainedSync(path: string): RawCatalog | null { + const bundled = isDefaultCatalogPath(path) ? loadBundledCodexCatalog() : null; + if (bundled) return JSON.parse(JSON.stringify(bundled)) as RawCatalog; + const active = readCatalog(path); + if (active && findNativeTemplate(active)) return active; + return readCatalog(catalogBackupPathFor(path)) + ?? (isDefaultCatalogPath(path) ? readCatalog(legacyCatalogBackupPath()) : null) + ?? readCatalog(activeCodexModelsCachePath()) + ?? active; +} + +function retainedCatalogSyncEvidence( + config: OcxConfig, + catalogPath: string, + catalog: RawCatalog, +): string { + return JSON.stringify({ + config, + catalogPath, + catalog, + catalogBytes: optionalFileBytes(catalogPath), + hashedBackupBytes: optionalFileBytes(catalogBackupPathFor(catalogPath)), + legacyBackupBytes: isDefaultCatalogPath(catalogPath) + ? optionalFileBytes(legacyCatalogBackupPath()) : null, + modelsCacheBytes: optionalFileBytes(activeCodexModelsCachePath()), + }); +} + +/** + * The process-local half of the same evidence, observed separately. + * + * These epochs belong in the freshness comparison — a runtime or bundled-template + * swap mid-gather changes what the candidate means — but they cannot share the + * filesystem baseline. Gathering RESOLVES the Codex runtime, so a pre-gather + * snapshot always disagrees with itself afterwards, and every sync refused to write + * a catalog nobody else had touched. The filesystem bytes are therefore baselined + * before the await (an outside writer must lose), while these are baselined once our + * own observation is finished (only an outside writer moving them afterwards counts). + */ +function retainedCatalogProcessEvidence(): string { + return JSON.stringify({ + bundledCatalogCache: bundledCatalogCacheState(), + runtimeProcessCache: peekCodexRuntimeProcessCache(), + }); +} + +/** + * Capture every local catalog input the retained sync path consults before its + * provider await. The exact evidence is compared after K acquisition; a newer + * catalog/backup/cache or target selection makes this attempt a no-write. + */ +function readRetainedCatalogSync(config: OcxConfig): RetainedCatalogSyncRead | null { const catalogPath = readCodexCatalogPath(); - const catalog = loadCatalogForSync(catalogPath); - if (!catalog) return { added: 0, path: catalogPath, catalogWritten: false, comboOmissions: [] }; + const catalog = loadCatalogForRetainedSync(catalogPath); + if (!catalog) return null; // The bundled catalog is a reliable native template on the default path, but it is not the // merge source. Preservation must inspect the file that this sync is about to overwrite; // otherwise an empty/partial provider gather cannot see routed or user-native rows on disk. const onDiskCatalog = readCatalog(catalogPath); - const catalogModelsForMerge = onDiskCatalog?.models ?? catalog.models ?? []; + const evidence = retainedCatalogSyncEvidence(config, catalogPath, catalog); + // `processEvidence` is filled in after the provider await, not here. + return { catalogPath, catalog, onDiskCatalog, evidence, processEvidence: "" }; +} + +function revalidateRetainedCatalogSync( + config: OcxConfig, + prepared: RetainedCatalogSyncRead, +): RetainedCatalogSyncRead | null { + const catalogPath = readCodexCatalogPath(); + if (catalogPath !== prepared.catalogPath) return null; + const evidence = retainedCatalogSyncEvidence(config, catalogPath, prepared.catalog); + if (evidence !== prepared.evidence) return null; + if (retainedCatalogProcessEvidence() !== prepared.processEvidence) return null; + return { + catalogPath, + catalog: JSON.parse(JSON.stringify(prepared.catalog)) as RawCatalog, + onDiskCatalog: readCatalog(catalogPath), + evidence, + processEvidence: prepared.processEvidence, + }; +} +function pristineCatalogBytes(read: RetainedCatalogSyncRead): string | null { + if (read.onDiskCatalog && !catalogHasRoutedEntries(read.onDiskCatalog)) { + try { + return readFileSync(read.catalogPath, "utf8"); + } catch { + return null; + } + } + return catalogHasRoutedEntries(read.catalog) + ? null + : `${JSON.stringify(read.catalog, null, 2)}\n`; +} + +function writeRetainedCatalogSync({ + config, + goModels, + comboOmissions, + read, + permit, + owningCodexHome, +}: RetainedCatalogSyncWrite): RetainedCatalogSyncResult { + const { catalogPath, catalog, onDiskCatalog } = read; + const catalogModelsForMerge = onDiskCatalog?.models ?? catalog.models ?? []; const template = findNativeTemplate(catalog); - const comboOmissions: ComboCatalogOmission[] = []; - const goModels = await gatherRoutedModels(config, { comboOmissions }); try { // Once-only: preserve the PRISTINE pre-opencodex catalog as the native-priority baseline // (later syncs would otherwise overwrite it with featured-modified priorities). - ensureCatalogBackup(catalogPath, catalog); + const pristine = pristineCatalogBytes(read); + if (pristine !== null) { + publishHashedCodexCatalogBackup(permit, owningCodexHome, { + path: catalogBackupPathFor(catalogPath), + content: pristine, + }); + if (isDefaultCatalogPath(catalogPath)) { + publishLegacyCodexCatalogBackup(permit, owningCodexHome, { + path: legacyCatalogBackupPath(), + content: pristine, + }); + } + } } catch { /* backup best-effort */ } // Hide disabled models from Codex, then feature the chosen subagent models (native OR routed) @@ -565,11 +717,60 @@ export async function syncCatalogModels(config: OcxConfig): Promise<{ catalog.models = mergeCatalogEntriesForSync(catalogModelsForMerge, goEntries, baseline, featured, wsEnabled, goIds, template, disabledNativeSlugs(config), gatheredProviderNames, multiAgentMode, exactComboSlugs, hasPhysicalComboProvider, includeNativeOpenAi); clampCatalogModelsToCodexSupport(catalog.models); - atomicWriteFile(catalogPath, JSON.stringify(catalog, null, 2) + "\n"); + replaceActiveCodexCatalog(permit, owningCodexHome, { + path: catalogPath, + content: `${JSON.stringify(catalog, null, 2)}\n`, + }); return { added: goEntries.length, path: catalogPath, catalogWritten: true, comboOmissions }; } -export function restoreCodexCatalog(): { removed: number; kept: number; path: string } { +export async function syncCatalogModels(config: OcxConfig): Promise { + const owningCodexHome = getCodexHome(); + const preflightRead = readRetainedCatalogSync(config); + if (preflightRead === null) { + return { + added: 0, + path: readCodexCatalogPath(), + catalogWritten: false, + comboOmissions: [], + }; + } + + const comboOmissions: ComboCatalogOmission[] = []; + const goModels = await gatherRoutedModels(config, { comboOmissions }); + // Baseline the process-local epochs only now: gathering resolves the Codex + // runtime, so a pre-gather snapshot would have flagged our OWN side effect and + // refused every write. The filesystem baseline above is untouched, so a catalog, + // backup, cache or target that another writer moved during the await still loses. + const prepared: RetainedCatalogSyncRead = { + ...preflightRead, + processEvidence: retainedCatalogProcessEvidence(), + }; + const committed = withCatalogWriteSerialization(owningCodexHome, permit => { + const current = revalidateRetainedCatalogSync(config, prepared); + if (current === null) return null; + return writeRetainedCatalogSync({ + config, + goModels, + comboOmissions, + read: current, + permit, + owningCodexHome, + }); + }); + if (committed.kind === "completed" && committed.value !== null) return committed.value; + return { + added: 0, + path: prepared.catalogPath, + catalogWritten: false, + comboOmissions, + }; +} + +export function restoreCodexCatalogWithPermit( + permit: CatalogWritePermit, + owningCodexHome: string, +): { removed: number; kept: number; path: string } { const catalogPath = readCodexCatalogPath(); const catalog = readCatalog(catalogPath); if (!catalog || !Array.isArray(catalog.models)) return { removed: 0, kept: 0, path: catalogPath }; @@ -584,7 +785,10 @@ export function restoreCodexCatalog(): { removed: number; kept: number; path: st ...backup, models: [...backup.models, ...userNativeAdditions], }; - atomicWriteFile(catalogPath, JSON.stringify(restored, null, 2) + "\n"); + replaceActiveCodexCatalog(permit, owningCodexHome, { + path: catalogPath, + content: `${JSON.stringify(restored, null, 2)}\n`, + }); return { removed, kept: restored.models.length, path: catalogPath }; } const before = catalog.models.length; @@ -592,13 +796,30 @@ export function restoreCodexCatalog(): { removed: number; kept: number; path: st const removed = before - native.length; if (removed > 0) { catalog.models = native; - atomicWriteFile(catalogPath, JSON.stringify(catalog, null, 2) + "\n"); + replaceActiveCodexCatalog(permit, owningCodexHome, { + path: catalogPath, + content: `${JSON.stringify(catalog, null, 2)}\n`, + }); } return { removed, kept: native.length, path: catalogPath }; } +export function restoreCodexCatalog(): { removed: number; kept: number; path: string } { + const owningCodexHome = getCodexHome(); + const outcome = withCatalogWriteSerialization( + owningCodexHome, + permit => restoreCodexCatalogWithPermit(permit, owningCodexHome), + ); + return outcome.kind === "completed" + ? outcome.value + : { removed: 0, kept: 0, path: readCodexCatalogPath() }; +} + /** Force Codex's models_cache stale from the on-disk catalog. Returns whether a cache write occurred. */ -export function invalidateCodexModelsCache(): boolean { +export function invalidateCodexModelsCacheWithPermit( + permit: CatalogWritePermit, + owningCodexHome: string, +): boolean { try { const catalogPath = readCodexCatalogPath(); if (!existsSync(catalogPath)) return false; @@ -609,9 +830,21 @@ export function invalidateCodexModelsCache(): boolean { client_version: "0.0.0", models, }; - atomicWriteFile(activeCodexModelsCachePath(), JSON.stringify(wrapper, null, 2) + "\n"); + replaceCodexModelsCache(permit, owningCodexHome, { + path: activeCodexModelsCachePath(), + content: `${JSON.stringify(wrapper, null, 2)}\n`, + }); return true; } catch { return false; } } + +export function invalidateCodexModelsCache(): boolean { + const owningCodexHome = getCodexHome(); + const outcome = withCatalogWriteSerialization( + owningCodexHome, + permit => invalidateCodexModelsCacheWithPermit(permit, owningCodexHome), + ); + return outcome.kind === "completed" && outcome.value; +} diff --git a/src/codex/inject.ts b/src/codex/inject.ts index 57d21e00a..8f700f2cf 100644 --- a/src/codex/inject.ts +++ b/src/codex/inject.ts @@ -1,7 +1,8 @@ import { existsSync, readFileSync, unlinkSync } from "node:fs"; import { atomicWriteFile, loadConfig, subagentDefaultSyncEffective, websocketsEnabled } from "../config"; import { markJournalInjectedState, removeJournal, restoreJournalState, writeJournal } from "./journal"; -import { restoreCodexCatalog } from "./catalog"; +import { withCatalogWriteSerialization } from "./catalog-write-serialization"; +import { restoreCodexCatalogWithPermit } from "./catalog/sync"; import { migrateHistoryToOpenai, syncCodexHistoryProvider } from "./history-provider"; import { OCX_SECTION_MARKER, @@ -13,7 +14,7 @@ import { rootTomlString, tomlStringPattern, } from "./injected-marker"; -import { CODEX_CONFIG_PATH, CODEX_PROFILE_PATH, DEFAULT_CATALOG_PATH, parseTomlString, readRootTomlString, resolveCodexConfigPath, tomlString } from "./paths"; +import { CODEX_CONFIG_PATH, CODEX_PROFILE_PATH, DEFAULT_CATALOG_PATH, getCodexHome, parseTomlString, readRootTomlString, resolveCodexConfigPath, tomlString } from "./paths"; import { resolveEffectiveProjectModelProvider } from "./project-config-warnings"; import { transformManagedSubagentDefaults, @@ -771,7 +772,12 @@ export function restoreNativeCodex(): { success: boolean; message: string } { const cfg = journal.configRestored ? { success: true, message: "Codex config restored from opencodex journal." } : removeCodexConfig({ preserveProfile: journal.profileRestored || journal.profileChanged }); - const cat = restoreCodexCatalog(); + const owningCodexHome = getCodexHome(); + const restoredCatalog = withCatalogWriteSerialization(owningCodexHome, permit => + restoreCodexCatalogWithPermit(permit, owningCodexHome)); + const cat = restoredCatalog.kind === "completed" + ? restoredCatalog.value + : { removed: 0, kept: 0, path: DEFAULT_CATALOG_PATH }; // Design B (loopback) steady state: threads are already tagged openai, so prove the // no-op with a readonly probe instead of write-opening a DB the Codex app may hold // (Windows: WAL writer lock -> seconds of stalling + a false warning on every stop). diff --git a/src/codex/internal/catalog-writer.ts b/src/codex/internal/catalog-writer.ts index 4c409c151..370bbda95 100644 --- a/src/codex/internal/catalog-writer.ts +++ b/src/codex/internal/catalog-writer.ts @@ -1,4 +1,5 @@ -import { chmodSync, linkSync, truncateSync, unlinkSync, writeFileSync } from "node:fs"; +import { chmodSync, linkSync, mkdirSync, truncateSync, unlinkSync, writeFileSync } from "node:fs"; +import { dirname } from "node:path"; import { AtomicWriteResidualTempError, @@ -138,6 +139,7 @@ function publishCatalogBackup( ): CatalogBackupPublication { const io = suppliedIo ?? defaultBackupWriteIO(prepared.path); const target = io.resolveTarget(prepared.path); + if (!suppliedIo) mkdirSync(dirname(target), { recursive: true, mode: 0o700 }); const tempPath = `${target}.ocx.${process.pid}.backup.${++backupTempSequence}.tmp`; let hardened = false; diff --git a/src/server/index.ts b/src/server/index.ts index 628b6ff64..f0cff004e 100644 --- a/src/server/index.ts +++ b/src/server/index.ts @@ -19,7 +19,9 @@ import { websocketsEnabled, } from "../config"; import { reconcileOAuthProviders } from "../oauth"; -import { invalidateCodexModelsCache } from "../codex/catalog"; +import { withCatalogWriteSerialization } from "../codex/catalog-write-serialization"; +import { invalidateCodexModelsCacheWithPermit } from "../codex/catalog/sync"; +import { getCodexHome } from "../codex/paths"; import { startMemoryWatchdog } from "./memory-watchdog"; import { reconcileLiveStateStores, @@ -400,7 +402,16 @@ export function startServer(port?: number, deps: StartServerDeps = {}) { if (migrated) saveConfig(config); } } - invalidateCodexModelsCache(); + // Startup cache invalidation is best-effort and must never block the server from + // serving. It now takes K so it cannot race a convergence commit, but both the + // home resolution and the acquisition can fail on a machine with no Codex home — + // `getCodexHome()` THROWS when CODEX_HOME names a missing directory, which would + // otherwise turn "no Codex installed" into "proxy will not start". + try { + const startupCodexHome = getCodexHome(); + withCatalogWriteSerialization(startupCodexHome, permit => + invalidateCodexModelsCacheWithPermit(permit, startupCodexHome)); + } catch { /* no readable Codex home: nothing to invalidate */ } // Arm the `claudeCode` hand-edit guard (devlog 260726_claude_auth_auto/040 H1) BEFORE // the server can serve a request, and AFTER the startup migrations above — those run // against a config nobody else holds and are the documented exception to the save diff --git a/tests/codex-app-server-processes.test.ts b/tests/codex-app-server-processes.test.ts index c89ceb78b..3f3544434 100644 --- a/tests/codex-app-server-processes.test.ts +++ b/tests/codex-app-server-processes.test.ts @@ -363,12 +363,18 @@ describe("CLI /api sync wiring for stale app-servers (#476)", () => { cliSource.indexOf('case "sync-cache":'), cliSource.indexOf('case "gui":'), ); - expect(syncCacheCase).toContain("invalidateCodexModelsCache()"); - expect(syncCacheCase).toContain("if (invalidateCodexModelsCache())"); + // The cache write now happens under the catalog serialization lock K, so the + // gate reads the permitted writer's outcome instead of a bare boolean call. + // The property under test is unchanged: app-servers are touched only after a + // write actually landed, never on a refused/failed serialization attempt. + expect(syncCacheCase).toContain("withCatalogWriteSerialization"); + expect(syncCacheCase).toContain("invalidateCodexModelsCacheWithPermit(permit, owningCodexHome)"); + const gate = 'if (invalidated.kind === "completed" && invalidated.value)'; + expect(syncCacheCase).toContain(gate); expect(syncCacheCase).toContain("afterCatalogWriteHandleAppServers"); - expect(syncCacheCase.indexOf("if (invalidateCodexModelsCache())")) + expect(syncCacheCase.indexOf(gate)) .toBeLessThan(syncCacheCase.indexOf("afterCatalogWriteHandleAppServers")); - const gatedBlock = syncCacheCase.slice(syncCacheCase.indexOf("if (invalidateCodexModelsCache())")); + const gatedBlock = syncCacheCase.slice(syncCacheCase.indexOf(gate)); expect(gatedBlock).toContain("afterCatalogWriteHandleAppServers"); expect(syncCacheCase.replace(gatedBlock, "")).not.toContain("afterCatalogWriteHandleAppServers"); }); diff --git a/tests/codex-retained-root-serialization.test.ts b/tests/codex-retained-root-serialization.test.ts new file mode 100644 index 000000000..5c745d4cf --- /dev/null +++ b/tests/codex-retained-root-serialization.test.ts @@ -0,0 +1,308 @@ +import { afterEach, expect, test } from "bun:test"; +import { + chmodSync, + existsSync, + mkdirSync, + mkdtempSync, + readFileSync, + realpathSync, + rmSync, + writeFileSync, +} from "node:fs"; +import { tmpdir } from "node:os"; +import { join, resolve } from "node:path"; + +import { + resolveCodexCatalogSerializationDatabasePath, + resolveEffectiveUserIdentity, +} from "../src/codex/user-identity"; + +const repoRoot = resolve(import.meta.dir, ".."); +const sandboxes: Sandbox[] = []; + +interface Sandbox { + readonly root: string; + readonly codexHome: string; + readonly opencodexHome: string; + readonly env: Record; +} + +function nativeEntry(slug: string, visibility = "list"): Record { + return { + slug, + display_name: slug, + description: "native", + priority: 9, + visibility, + supported_in_api: true, + shell_type: "shell_command", + base_instructions: "You are Codex, a coding agent based on GPT-5.", + supported_reasoning_levels: [{ effort: "medium", description: "medium" }], + }; +} + +function catalogBytes(visibility = "list", routed = false): string { + return `${JSON.stringify({ + retained_marker: visibility, + models: [ + nativeEntry("gpt-5.5", visibility), + ...(routed ? [{ + ...nativeEntry("vendor/old-model"), + description: "Routed via opencodex → vendor.", + }] : []), + ], + }, null, 2)}\n`; +} + +function makeSandbox(prefix: string): Sandbox { + const root = realpathSync.native(mkdtempSync(join(tmpdir(), prefix))); + const codexHome = join(root, "codex-home"); + const opencodexHome = join(root, "opencodex-home"); + const home = join(root, "user-home"); + const runtime = join(root, "runtime"); + for (const path of [codexHome, opencodexHome, home, runtime]) { + mkdirSync(path, { recursive: true }); + chmodSync(path, 0o700); + } + const sandbox = { + root, + codexHome, + opencodexHome, + env: { + ...Object.fromEntries(Object.entries(process.env).filter((entry): entry is [string, string] => entry[1] !== undefined)), + CODEX_HOME: codexHome, + OPENCODEX_HOME: opencodexHome, + HOME: home, + USERPROFILE: home, + TMPDIR: runtime, + TEMP: runtime, + TMP: runtime, + XDG_RUNTIME_DIR: runtime, + LOCALAPPDATA: join(home, "LocalAppData"), + }, + }; + sandboxes.push(sandbox); + return sandbox; +} + +async function waitForPath(path: string, timeoutMs = 10_000): Promise { + const deadline = Date.now() + timeoutMs; + while (!existsSync(path)) { + if (Date.now() >= deadline) throw new Error(`Timed out waiting for ${path}`); + await Bun.sleep(5); + } +} + +async function runChild( + sandbox: Sandbox, + script: string, +): Promise<{ exitCode: number; stdout: string; stderr: string }> { + const child = Bun.spawn([process.execPath, "--eval", script], { + cwd: repoRoot, + env: sandbox.env, + stdout: "pipe", + stderr: "pipe", + }); + const [exitCode, stdout, stderr] = await Promise.all([ + child.exited, + new Response(child.stdout).text(), + new Response(child.stderr).text(), + ]); + return { exitCode, stdout, stderr }; +} + +async function holdCatalogLock(sandbox: Sandbox): Promise<{ + release(): void; + child: ReturnType; +}> { + const ready = join(sandbox.root, "lock-ready"); + const release = join(sandbox.root, "lock-release"); + const script = ` + import { existsSync, writeFileSync } from "node:fs"; + import { withCatalogWriteSerialization } from "./src/codex/catalog-write-serialization.ts"; + const home = process.env.CODEX_HOME; + const outcome = withCatalogWriteSerialization(home, () => { + writeFileSync(${JSON.stringify(ready)}, "ready"); + const waiter = new Int32Array(new SharedArrayBuffer(4)); + while (!existsSync(${JSON.stringify(release)})) Atomics.wait(waiter, 0, 0, 10); + }); + if (outcome.kind !== "completed") throw new Error(JSON.stringify(outcome)); + `; + const child = Bun.spawn([process.execPath, "--eval", script], { + cwd: repoRoot, + env: sandbox.env, + stdout: "pipe", + stderr: "pipe", + }); + await waitForPath(ready); + return { release: () => writeFileSync(release, "release"), child }; +} + +function seedCatalog(sandbox: Sandbox, bytes = catalogBytes()): string { + const path = join(sandbox.codexHome, "catalog.json"); + writeFileSync(join(sandbox.codexHome, "config.toml"), 'model_catalog_json = "catalog.json"\n'); + writeFileSync(path, bytes); + return path; +} + +afterEach(() => { + const identity = resolveEffectiveUserIdentity(); + for (const sandbox of sandboxes.splice(0)) { + const database = resolveCodexCatalogSerializationDatabasePath(identity, sandbox.codexHome); + for (const suffix of ["", "-journal", "-wal", "-shm"]) rmSync(`${database}${suffix}`, { force: true }); + rmSync(sandbox.root, { recursive: true, force: true }); + } +}); + +test("startup and CLI sync-cache cannot write models_cache while another process owns K", async () => { + const sandbox = makeSandbox("ocx-retained-cache-"); + seedCatalog(sandbox); + const cachePath = join(sandbox.codexHome, "models_cache.json"); + const holder = await holdCatalogLock(sandbox); + try { + const startupProbe = await runChild(sandbox, ` + const sentinel = new Error("TEST_LISTENER_INTERCEPTED"); + Bun.serve = () => { throw sentinel; }; + const { startServer } = await import("./src/server/index.ts"); + try { + startServer(0); + throw new Error("startServer unexpectedly reached a listener"); + } catch (error) { + if (error !== sentinel) throw error; + } + `); + expect(startupProbe.exitCode).toBe(0); + expect(existsSync(cachePath)).toBe(false); + + const cli = Bun.spawnSync([process.execPath, "run", "src/cli/index.ts", "sync-cache"], { + cwd: repoRoot, + env: sandbox.env, + stdout: "pipe", + stderr: "pipe", + }); + expect(cli.exitCode).toBe(0); + expect(existsSync(cachePath)).toBe(false); + const cliSource = readFileSync(join(repoRoot, "src/cli/index.ts"), "utf8"); + const cliStart = cliSource.indexOf('case "sync-cache"'); + const cliRoot = cliSource.slice(cliStart, cliSource.indexOf('case "gui"', cliStart)); + expect(cliRoot).toContain("withCatalogWriteSerialization(owningCodexHome"); + expect(cliRoot).toContain("invalidateCodexModelsCacheWithPermit"); + + const startup = readFileSync(join(repoRoot, "src/server/index.ts"), "utf8"); + const startupStart = startup.indexOf("const startupCodexHome"); + const startupRoot = startup.slice(startupStart, startup.indexOf("armClaudeCodeBaseline", startupStart)); + expect(startupRoot).toContain("withCatalogWriteSerialization(startupCodexHome"); + expect(startupRoot).toContain("invalidateCodexModelsCacheWithPermit"); + } finally { + holder.release(); + expect(await holder.child.exited).toBe(0); + } +}); + +test("native restore cannot read-transform-write the catalog while another process owns K", async () => { + const sandbox = makeSandbox("ocx-retained-restore-"); + const catalogPath = seedCatalog(sandbox, catalogBytes("list", true)); + writeFileSync(join(sandbox.opencodexHome, "catalog-backup.json"), catalogBytes("list", false)); + const before = readFileSync(catalogPath, "utf8"); + const holder = await holdCatalogLock(sandbox); + try { + const restored = await runChild(sandbox, ` + const { restoreNativeCodex } = await import("./src/codex/inject.ts"); + console.log(JSON.stringify(restoreNativeCodex())); + `); + expect(restored.exitCode).toBe(0); + expect(readFileSync(catalogPath, "utf8")).toBe(before); + const source = readFileSync(join(repoRoot, "src/codex/inject.ts"), "utf8"); + const restoreRoot = source.slice(source.indexOf("const owningCodexHome"), source.indexOf("// Design B", source.indexOf("const owningCodexHome"))); + expect(restoreRoot).toContain("withCatalogWriteSerialization(owningCodexHome"); + expect(restoreRoot).toContain("restoreCodexCatalogWithPermit"); + } finally { + holder.release(); + expect(await holder.child.exited).toBe(0); + } +}); + +async function runPublisher( + sandbox: Sandbox, + kind: "convergence" | "retained", +): Promise<{ exitCode: number; stdout: string; stderr: string }> { + const config = { port: 10100, defaultProvider: "openai", providers: {}, disabledModels: ["gpt-5.5"] }; + if (kind === "retained") { + return runChild(sandbox, ` + const { handleManagementAPI } = await import("./src/server/management-api.ts"); + const config = ${JSON.stringify(config)}; + const req = new Request("http://localhost/api/sync", { method: "POST", headers: { Host: "localhost" } }); + const response = await handleManagementAPI(req, new URL(req.url), config); + console.log(JSON.stringify({ status: response.status, body: await response.json() })); + `); + } + return runChild(sandbox, ` + const { withConfigMutationLockSync } = await import("./src/config.ts"); + const { captureCatalogAdmissionSnapshot, createCatalogConvergeRequest } = await import("./src/codex/catalog-admission.ts"); + const { convergeCodexCatalog } = await import("./src/codex/convergence.ts"); + const config = ${JSON.stringify(config)}; + withConfigMutationLockSync(() => undefined); + const snapshot = captureCatalogAdmissionSnapshot(config); + const result = await convergeCodexCatalog(snapshot, createCatalogConvergeRequest({ deadlineMs: 2000 })); + console.log(JSON.stringify(result)); + `); +} + +for (const publisher of ["convergence", "retained"] as const) { + test(`POST /api/sync gathered first and acquired K second does not clobber a newer ${publisher} catalog`, async () => { + const sandbox = makeSandbox(`ocx-retained-race-${publisher}-`); + const catalogPath = seedCatalog(sandbox); + const initial = readFileSync(catalogPath, "utf8"); + const requested = join(sandbox.root, "provider-requested"); + const release = join(sandbox.root, "provider-release"); + const config = { + port: 10100, + defaultProvider: "together", + providers: { + together: { + adapter: "openai-chat", + baseUrl: "https://api.together.xyz/v1", + apiKey: "race-key", + models: ["fallback-model"], + }, + }, + }; + const sync = Bun.spawn([process.execPath, "--eval", ` + import { existsSync, writeFileSync } from "node:fs"; + const config = ${JSON.stringify(config)}; + config.providers.together.fetch = async () => { + writeFileSync(${JSON.stringify(requested)}, "requested"); + while (!existsSync(${JSON.stringify(release)})) await Bun.sleep(5); + return Response.json({ data: [{ id: "race-model" }] }); + }; + const { handleManagementAPI } = await import("./src/server/management-api.ts"); + const req = new Request("http://localhost/api/sync", { method: "POST", headers: { Host: "localhost" } }); + const response = await handleManagementAPI(req, new URL(req.url), config); + console.log(JSON.stringify({ status: response.status, body: await response.json() })); + `], { cwd: repoRoot, env: sandbox.env, stdout: "pipe", stderr: "pipe" }); + + await Promise.race([ + waitForPath(requested), + sync.exited.then(async exitCode => { + const stdout = await new Response(sync.stdout).text(); + const stderr = await new Response(sync.stderr).text(); + throw new Error(`sync exited before provider barrier (${exitCode})\nstdout=${stdout}\nstderr=${stderr}`); + }), + ]); + const published = await runPublisher(sandbox, publisher); + if (published.exitCode !== 0) { + throw new Error(`${publisher} publisher failed\nstdout=${published.stdout}\nstderr=${published.stderr}`); + } + const newer = readFileSync(catalogPath, "utf8"); + expect(newer).not.toBe(initial); + + writeFileSync(release, "release"); + const [exitCode, stdout, stderr] = await Promise.all([ + sync.exited, + new Response(sync.stdout).text(), + new Response(sync.stderr).text(), + ]); + expect({ exitCode, stdout, stderr }).toMatchObject({ exitCode: 0 }); + expect(readFileSync(catalogPath, "utf8")).toBe(newer); + }, 20_000); +} From 23a110d1edd17371fef3d4155496c362c6f6857e Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Wed, 5 Aug 2026 00:21:33 +0900 Subject: [PATCH 076/163] feat(codex): the catalog seam, and the sixteen callers that now use it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This is the body of WP9, and it should have been the first commit rather than the last: an independent verifier checked out HEAD and found convergence.ts missing, because the two mechanism commits landed while the seam they plug into was still untracked. A commit that cannot be checked out and run is not a commit, and the earlier ones were only self-consistent by accident of my dirty worktree. `refreshCodexCatalogBestEffort` is gone. It was a `Promise` whose entire error policy was `catch { /* catalog absent */ }`, so sixteen management mutations each had their own unguarded path to a catalog write and no way to report what happened. They now call one bound `convergeCodexCatalog()` and append the contract's `CatalogDisposition` to their existing response, keeping their 2xx/201 and their persisted mutation exactly as before. The adapter is total. Lazy-import failure, factory failure, admission, gather and commit all project into a typed disposition rather than throwing, because the dispatcher rethrows anything that is not a busy error — an exception here would turn a mutation that already persisted into a 500 and skip the Claude and Desktop follow-up work that runs after the refresh. The management factory still closes over the exact resident config object the route received. That is deliberate: putting config on ConvergeRequest would let any caller substitute catalog authority, which is weaker than the callback it replaces. --- src/codex/convergence.ts | 441 ++++++++++++++++++ src/codex/management-convergence.ts | 55 ++- src/server/management-api.ts | 65 ++- .../management/agent-settings-routes.ts | 9 +- src/server/management/combo-routes.ts | 10 +- src/server/management/config-routes.ts | 2 +- src/server/management/context.ts | 5 +- src/server/management/logs-usage-routes.ts | 2 +- src/server/management/model-routes.ts | 26 +- src/server/management/oauth-account-routes.ts | 2 +- src/server/management/provider-routes.ts | 33 +- tests/catalog-input-modality-enum.test.ts | 7 +- tests/codex-convergence-contract.test.ts | 251 ++++++++++ tests/codex-management-convergence.test.ts | 20 +- tests/codex-v2-gate.test.ts | 28 +- tests/combo-management-api.test.ts | 5 +- tests/combos.test.ts | 5 +- tests/helpers/catalog-convergence.ts | 15 + tests/management-client-config-route.test.ts | 7 +- tests/management-integration-routes.test.ts | 3 +- tests/management-provider-validation.test.ts | 19 +- tests/model-visibility-management-api.test.ts | 3 +- tests/responses-shadow-intercept.test.ts | 5 +- tests/server-combo-failover-e2e.test.ts | 3 +- 24 files changed, 916 insertions(+), 105 deletions(-) create mode 100644 src/codex/convergence.ts create mode 100644 tests/codex-convergence-contract.test.ts create mode 100644 tests/helpers/catalog-convergence.ts diff --git a/src/codex/convergence.ts b/src/codex/convergence.ts new file mode 100644 index 000000000..5194967eb --- /dev/null +++ b/src/codex/convergence.ts @@ -0,0 +1,441 @@ +import { join } from "node:path"; + +import { getConfigDir, websocketsEnabled, withExpectedConfigGenerationSync } from "../config"; +import { COMBO_NAMESPACE } from "../combos"; +import { isCanonicalOpenAiForwardProvider } from "../providers/openai-tiers"; +import { getAuthStorePath } from "../oauth/store"; +import type { OcxConfig } from "../types"; +import { captureCatalogAdmissionSnapshot } from "./catalog-admission"; +import { + type CatalogSourceForGather, + bundledCatalogCacheState, + resolveCatalogSourceForGather, +} from "./catalog/bundled"; +import { + acceptCatalogGatherSourcePath, + captureAndSealCatalogHomeSelection, + captureCatalogGatherTargetIdentity, + createCatalogGatherEvidenceSession, + readCatalogGatherSource, + sealCatalogGatherEvidenceSession, + type CatalogFilesystemEvidenceSession, +} from "./catalog/filesystem-evidence"; +import { + CatalogGatherBusyError, + createCatalogGatherAuthorityIdentity, + filterCatalogVisibleModels, + gatherRoutedModelsForCatalogGather, + type CatalogGatherProviderAuthOutcome, +} from "./catalog/provider-fetch"; +import { + catalogBackupPathFor, + catalogHasRoutedEntries, + findNativeTemplate, + legacyCatalogBackupPath, + parseCatalogJson, + type RawCatalog, +} from "./catalog/parsing"; +import { + buildCatalogEntries, + orderForSubagents, +} from "./catalog/sync"; +import { exactComboCatalogSlugs } from "./catalog/aggregation"; +import { disabledNativeSlugs } from "./catalog/metadata"; +import { codexRuntimeStatePath, peekCodexRuntimeProcessCache } from "./runtime"; +import { withCatalogWriteSerialization } from "./catalog-write-serialization"; +import { + publishHashedCodexCatalogBackup, + publishLegacyCodexCatalogBackup, + replaceActiveCodexCatalog, + replaceCodexModelsCache, + type PreparedCatalogFileWrite, +} from "./internal/catalog-writer"; +import type { + CatalogAdmissionSnapshot, + CatalogDisposition, + CatalogGatherAuthorityIdentity, + CatalogNotice, + CatalogProviderDiscoveryPolicySnapshot, + CatalogProcessLocalEvidence, + CatalogSourceEvidence, + CatalogSourceRole, + ConvergeRequest, +} from "./convergence-types"; + +export interface CatalogWriteReceipt { + readonly keyedBackup: "written" | "preserved" | "not-requested"; + readonly legacyBackup: "written" | "preserved" | "not-requested"; + readonly catalog: "written" | "not-written"; + readonly cache: "written" | "not-written"; +} + +export type CodexCatalogCommitResult = + | { readonly kind: "committed"; readonly changed: boolean; readonly writes: CatalogWriteReceipt } + | { readonly kind: "stale"; readonly reason: "generation" | "home-selection" | "source-observation" | "process-local" | "target-identity" | "candidate-consumed" } + | { readonly kind: "refused"; readonly reason: "source-unreadable" | "source-ambiguous" | "target-unsafe" } + | { readonly kind: "failed"; readonly surface: "disk"; readonly writes: CatalogWriteReceipt }; + +declare const catalogCandidateBrand: unique symbol; +export interface CodexCatalogCandidate { readonly [catalogCandidateBrand]: true } + +export type CodexCatalogGatherResult = + | { readonly kind: "candidate"; readonly candidate: CodexCatalogCandidate } + | { readonly kind: "disposition"; readonly disposition: CatalogDisposition }; + +type CommitAttempt = CodexCatalogCommitResult | { readonly kind: "busy" }; + +interface CandidateState { + consumed: boolean; + readonly generation: CatalogAdmissionSnapshot["generation"]; + readonly authority: CatalogGatherAuthorityIdentity; + readonly sourceEvidence: CatalogSourceEvidence; + readonly processLocal: CatalogProcessLocalEvidence; + readonly home: string; + readonly targets: CatalogAdmissionSnapshot["targets"]; + readonly catalog: PreparedCatalogFileWrite; + readonly cache: PreparedCatalogFileWrite; + readonly keyedBackup?: PreparedCatalogFileWrite; + readonly legacyBackup?: PreparedCatalogFileWrite; + readonly changed: boolean; + readonly notices: readonly CatalogNotice[]; +} + +const candidateStates = new WeakMap(); +function same(left: unknown, right: unknown): boolean { + return JSON.stringify(left) === JSON.stringify(right); +} + +function targetPath(identity: string): string { + const parsed = JSON.parse(identity) as { path?: unknown }; + if (typeof parsed.path !== "string") throw new TypeError("Catalog target identity has no path."); + return parsed.path; +} + +function catalogFrom(bytes: Uint8Array | null): RawCatalog | null { + return bytes === null ? null : parseCatalogJson(Buffer.from(bytes).toString("utf8")); +} + +function catalogBytes(catalog: RawCatalog): string { + return `${JSON.stringify(catalog, null, 2)}\n`; +} + +interface ReadonlyRawCatalogLike { + readonly models?: readonly Readonly>[]; +} + +function hasRoutedEntries(catalog: ReadonlyRawCatalogLike): boolean { + return (catalog.models ?? []).some(entry => typeof entry.slug === "string" && entry.slug.includes("/")); +} + +function processEvidence(source: CatalogSourceForGather): CatalogProcessLocalEvidence { + return Object.freeze({ + runtime: Object.freeze({ ...source.processLocal.runtime }), + bundledCatalog: Object.freeze({ ...source.processLocal.bundledCatalog }), + }); +} + +function bindGatherPaths( + session: CatalogFilesystemEvidenceSession, + snapshot: CatalogAdmissionSnapshot, +): Readonly<{ catalog: string; cache: string; keyedBackup: string; legacyBackup?: string }> { + const catalog = targetPath(snapshot.targets.catalog); + const cache = targetPath(snapshot.targets.cache); + const keyedBackup = targetPath(snapshot.targets.catalogBackups[0]!); + const legacyBackup = snapshot.targets.catalogBackups[1] + ? targetPath(snapshot.targets.catalogBackups[1]) : undefined; + const configPath = snapshot.sourceEvidence.required["catalog-target-selection"].logicalPath; + + acceptCatalogGatherSourcePath(session, "catalog-target-selection", configPath); + readCatalogGatherSource(session, "catalog-target-selection"); + acceptCatalogGatherSourcePath(session, "active-catalog-merge", catalog); + acceptCatalogGatherSourcePath(session, "hashed-backup-fallback", keyedBackup); + acceptCatalogGatherSourcePath(session, "legacy-backup-fallback", legacyBackup ?? legacyCatalogBackupPath()); + acceptCatalogGatherSourcePath(session, "models-cache-fallback", cache); + acceptCatalogGatherSourcePath(session, "runtime-selection", codexRuntimeStatePath(getConfigDir())); + acceptCatalogGatherSourcePath(session, "provider-auth-selection", getAuthStorePath()); + acceptCatalogGatherSourcePath(session, "native-catalog-selection", catalog); + return { catalog, cache, keyedBackup, ...(legacyBackup ? { legacyBackup } : {}) }; +} + +function prepareCatalog( + config: Readonly, + source: Extract, + active: RawCatalog | null, + routedModels: Awaited>, +): RawCatalog { + const catalog = JSON.parse(JSON.stringify(source.catalog)) as RawCatalog; + const template = findNativeTemplate(catalog); + const enabled = filterCatalogVisibleModels(routedModels, config); + const featured = config.subagentModels ?? []; + const ordered = orderForSubagents(enabled, featured); + const multiAgentMode = config.multiAgentMode === "v1" || config.multiAgentMode === "v2" + ? config.multiAgentMode : "default"; + const exactComboSlugs = exactComboCatalogSlugs(config); + const hasPhysicalComboProvider = Object.hasOwn(config.providers, COMBO_NAMESPACE); + const enabledProviders = Object.entries(config.providers).filter(([, provider]) => provider.disabled !== true); + const includeNativeOpenAi = enabledProviders.length === 0 || enabledProviders.some(([name, provider]) => ( + name === "openai" && isCanonicalOpenAiForwardProvider(provider) + )); + const disabledNative = disabledNativeSlugs(config); + const nativeSlugs = includeNativeOpenAi + ? [...new Set((active?.models ?? catalog.models ?? []).flatMap(entry => ( + typeof entry.slug === "string" && !entry.slug.includes("/") && !disabledNative.has(entry.slug) + ? [entry.slug] : [] + )))] + : []; + const entries = buildCatalogEntries( + template ? JSON.parse(JSON.stringify(template)) : null, + nativeSlugs, ordered, featured, websocketsEnabled(config), multiAgentMode, exactComboSlugs, + ); + if (entries.length === nativeSlugs.length) { + const configuredProviders = new Set(enabledProviders.map(([name]) => name)); + const preserved = (active?.models ?? []).filter(entry => { + if (typeof entry.slug !== "string" || !entry.slug.includes("/")) return false; + const provider = entry.slug.slice(0, entry.slug.indexOf("/")); + const description = typeof entry.description === "string" ? entry.description : ""; + return configuredProviders.has(provider) || !description.startsWith("Routed via opencodex → "); + }); + entries.push(...preserved); + } + if (!hasPhysicalComboProvider) { + const exact = exactComboSlugs; + catalog.models = entries.filter(entry => ( + typeof entry.slug !== "string" || !exact.has(entry.slug) + || (Array.isArray(entry.input_modalities) && entry.input_modalities.length > 0) + )); + } else { + catalog.models = entries; + } + return catalog; +} + +export async function gatherCodexCatalogCandidate( + snapshot: CatalogAdmissionSnapshot, +): Promise { + let providerGatherStarted = false; + try { + const session = createCatalogGatherEvidenceSession(); + const home = captureAndSealCatalogHomeSelection(session); + if (!same(home, snapshot.sourceEvidence.homeSelection)) { + return { kind: "disposition", disposition: { status: "skipped", reason: "stale", retryable: true } }; + } + const paths = bindGatherPaths(session, snapshot); + const source = resolveCatalogSourceForGather(session); + if (source.kind === "catalog-unavailable") { + return { kind: "disposition", disposition: { status: "skipped", reason: "catalog-unavailable", retryable: false } }; + } + + const activeBytes = readCatalogGatherSource(session, "active-catalog-merge"); + const cacheBytes = readCatalogGatherSource(session, "models-cache-fallback"); + const keyedBackupBytes = readCatalogGatherSource(session, "hashed-backup-fallback"); + const legacyBackupBytes = paths.legacyBackup + ? readCatalogGatherSource(session, "legacy-backup-fallback") : null; + if (Object.keys(snapshot.config.combos ?? {}).length > 0) { + readCatalogGatherSource(session, "native-catalog-selection"); + } + const authOutcomes: CatalogGatherProviderAuthOutcome[] = []; + const discoveryPolicies: CatalogProviderDiscoveryPolicySnapshot[] = []; + providerGatherStarted = true; + const routedModels = await gatherRoutedModelsForCatalogGather(snapshot.config, session, { + providerAuthOutcomes: authOutcomes, + discoveryPolicySnapshots: discoveryPolicies, + }); + const processLocal = processEvidence(source); + const sourceEvidence = sealCatalogGatherEvidenceSession(session); + if (!same(sourceEvidence.required, snapshot.sourceEvidence.required)) { + return { kind: "disposition", disposition: { status: "skipped", reason: "stale", retryable: true } }; + } + + const current = captureCatalogAdmissionSnapshot(snapshot.config); + if (!same(current.configIdentity, snapshot.configIdentity) + || !same(current.targets, snapshot.targets) + || !same(current.sourceEvidence.homeSelection, snapshot.sourceEvidence.homeSelection) + || !same(current.sourceEvidence.required, snapshot.sourceEvidence.required)) { + return { kind: "disposition", disposition: { status: "skipped", reason: "stale", retryable: true } }; + } + + const active = catalogFrom(activeBytes); + const preparedCatalog = prepareCatalog(snapshot.config, source, active, routedModels); + const preparedCatalogBytes = catalogBytes(preparedCatalog); + const preparedCacheBytes = `${JSON.stringify({ + fetched_at: "2000-01-01T00:00:00Z", + client_version: "0.0.0", + models: preparedCatalog.models ?? [], + }, null, 2)}\n`; + const pristineBytes = active && !catalogHasRoutedEntries(active) + ? Buffer.from(activeBytes!).toString("utf8") + : !hasRoutedEntries(source.catalog) ? `${JSON.stringify(source.catalog, null, 2)}\n` : null; + const notices = new Set(); + if (source.source !== "bundled-catalog-template") notices.add("fallback"); + if (authOutcomes.some(outcome => outcome.state !== "available")) notices.add("provider-auth"); + const candidate = {} as CodexCatalogCandidate; + candidateStates.set(candidate, { + consumed: false, + generation: snapshot.generation, + authority: createCatalogGatherAuthorityIdentity( + snapshot, + sourceEvidence, + processLocal, + discoveryPolicies, + ), + sourceEvidence, + processLocal, + home: home.canonicalCodexHome, + targets: snapshot.targets, + catalog: { path: paths.catalog, content: preparedCatalogBytes }, + cache: { path: paths.cache, content: preparedCacheBytes }, + ...(pristineBytes ? { keyedBackup: { path: paths.keyedBackup, content: pristineBytes } } : {}), + ...(pristineBytes && paths.legacyBackup + ? { legacyBackup: { path: paths.legacyBackup, content: pristineBytes } } : {}), + changed: Buffer.from(activeBytes ?? []).toString("utf8") !== preparedCatalogBytes + || Buffer.from(cacheBytes ?? []).toString("utf8") !== preparedCacheBytes, + notices: Object.freeze([...notices]), + }); + return { kind: "candidate", candidate }; + } catch (error) { + if (error instanceof CatalogGatherBusyError) { + return { kind: "disposition", disposition: { status: "skipped", reason: "busy", retryable: true } }; + } + if (!providerGatherStarted) { + return { kind: "disposition", disposition: { status: "skipped", reason: "refused", retryable: false } }; + } + return { + kind: "disposition", + disposition: { status: "failed", reason: "provider-network", phase: "gather", retryable: true, partialWrite: false }, + }; + } +} + +function revalidateCandidate(state: CandidateState): CodexCatalogCommitResult | null { + let session: CatalogFilesystemEvidenceSession; + let validatingTargets = false; + try { + session = createCatalogGatherEvidenceSession(); + const home = captureAndSealCatalogHomeSelection(session); + if (!same(home, state.sourceEvidence.homeSelection)) return { kind: "stale", reason: "home-selection" }; + validatingTargets = true; + const currentTargets = { + catalog: captureCatalogGatherTargetIdentity(session, state.catalog.path), + cache: captureCatalogGatherTargetIdentity(session, state.cache.path), + catalogBackups: state.targets.catalogBackups.map(identity => ( + captureCatalogGatherTargetIdentity(session, targetPath(identity)) + )), + }; + if (!same(currentTargets, state.targets)) return { kind: "stale", reason: "target-identity" }; + validatingTargets = false; + for (const observation of [state.sourceEvidence.required["catalog-target-selection"]]) { + acceptCatalogGatherSourcePath(session, observation.role, observation.logicalPath); + readCatalogGatherSource(session, observation.role); + } + for (const [role, observations] of Object.entries(state.sourceEvidence.conditional)) { + for (const observation of observations) { + acceptCatalogGatherSourcePath(session, role as CatalogSourceRole, observation.logicalPath); + readCatalogGatherSource(session, role as CatalogSourceRole); + } + } + if (!same(sealCatalogGatherEvidenceSession(session), state.sourceEvidence)) { + return { kind: "stale", reason: "source-observation" }; + } + } catch { + return { kind: "refused", reason: validatingTargets ? "target-unsafe" : "source-unreadable" }; + } + + if (state.processLocal.runtime.state === "used") { + const current = peekCodexRuntimeProcessCache(); + if (current.kind !== "available" || current.epoch !== state.processLocal.runtime.epoch + || current.valueIdentity !== state.processLocal.runtime.valueIdentity) { + return { kind: "stale", reason: "process-local" }; + } + } + if (state.processLocal.bundledCatalog.state === "used") { + const current = bundledCatalogCacheState(); + if (current.epoch !== state.processLocal.bundledCatalog.epoch + || current.valueIdentity !== state.processLocal.bundledCatalog.valueIdentity) { + return { kind: "stale", reason: "process-local" }; + } + } + return null; +} + +function fixedCommit(state: CandidateState, permit: Parameters[0]): CodexCatalogCommitResult { + let writes: CatalogWriteReceipt = { + keyedBackup: "not-requested", + legacyBackup: "not-requested", + catalog: "not-written", + cache: "not-written", + }; + try { + if (state.keyedBackup) { + writes = { ...writes, keyedBackup: publishHashedCodexCatalogBackup(permit, state.home, state.keyedBackup) }; + } + if (state.legacyBackup) { + writes = { ...writes, legacyBackup: publishLegacyCodexCatalogBackup(permit, state.home, state.legacyBackup) }; + } + replaceActiveCodexCatalog(permit, state.home, state.catalog); + writes = { ...writes, catalog: "written" }; + replaceCodexModelsCache(permit, state.home, state.cache); + writes = { ...writes, cache: "written" }; + return { kind: "committed", changed: state.changed, writes }; + } catch { + return { kind: "failed", surface: "disk", writes }; + } +} + +export async function commitCodexCatalogCandidate( + candidate: CodexCatalogCandidate, + deadlineMs: number, +): Promise { + const state = candidateStates.get(candidate as object); + if (!state) return { kind: "refused", reason: "source-ambiguous" }; + if (state.consumed) return { kind: "stale", reason: "candidate-consumed" }; + const deadline = Date.now() + Math.max(0, deadlineMs); + while (true) { + const acquired = withCatalogWriteSerialization(state.home, permit => { + state.consumed = true; + const guarded = withExpectedConfigGenerationSync(state.generation, () => { + const invalid = revalidateCandidate(state); + return invalid ?? fixedCommit(state, permit); + }); + if (guarded.kind === "conflict") return { kind: "stale", reason: "generation" } as const; + if (guarded.kind === "unavailable") return { kind: "busy" } as const; + return guarded.value; + }); + if (acquired.kind === "completed") return acquired.value; + if (acquired.reason !== "busy" || Date.now() >= deadline) return { kind: "busy" }; + await Bun.sleep(Math.min(10, Math.max(1, deadline - Date.now()))); + } +} + +function projectCommit(result: CommitAttempt, notices: readonly CatalogNotice[]): CatalogDisposition { + if (result.kind === "busy") return { status: "skipped", reason: "busy", retryable: true }; + if (result.kind === "committed") { + return { status: "committed", changed: result.changed, degraded: notices.length > 0, notices }; + } + if (result.kind === "stale") return { status: "skipped", reason: "stale", retryable: true }; + if (result.kind === "refused") return { status: "skipped", reason: "refused", retryable: false }; + const partialWrite = result.writes.keyedBackup === "written" + || result.writes.legacyBackup === "written" || result.writes.catalog === "written"; + return { status: "failed", reason: "disk", phase: "commit", retryable: false, partialWrite }; +} + +export async function convergeCodexCatalog( + snapshot: CatalogAdmissionSnapshot, + request: ConvergeRequest, + lifecycle: Readonly<{ onCommitBegin?: () => void }> = {}, +): Promise> { + if (request.scope !== "catalog" || request.action !== "converge") { + return { + changed: false, + catalogRefresh: { status: "failed", reason: "disk", phase: "gather", retryable: false, partialWrite: false }, + }; + } + const gathered = await gatherCodexCatalogCandidate(snapshot); + if (gathered.kind === "disposition") return { changed: false, catalogRefresh: gathered.disposition }; + const state = candidateStates.get(gathered.candidate as object)!; + lifecycle.onCommitBegin?.(); + const committed = await commitCodexCatalogCandidate(gathered.candidate, request.deadlineMs); + return { + changed: committed.kind === "committed" ? committed.changed : false, + catalogRefresh: projectCommit(committed, state.notices), + }; +} diff --git a/src/codex/management-convergence.ts b/src/codex/management-convergence.ts index 369881fdc..b9ad903dc 100644 --- a/src/codex/management-convergence.ts +++ b/src/codex/management-convergence.ts @@ -1,12 +1,6 @@ -/** - * Management-scoped projection before WP9 installs real catalog convergence. - * - * The r2 #1 callback swallowed every catalog failure and accepted only the - * management context's resident config. Keeping that exact object in this - * factory prevents callers from adding substitute authority to ConvergeRequest. - * Until WP9 supplies gather/commit, the catalog disposition says no work ran. - */ import type { OcxConfig } from "../types"; +import { captureCatalogAdmissionSnapshot } from "./catalog-admission"; +import { convergeCodexCatalog } from "./convergence"; import type { CatalogDisposition, CatalogOnlyOutcome, @@ -55,8 +49,22 @@ function notEvaluatedObserved(history: CodexHistoryState): CodexObservedState { }; } -function catalogNotRequested(): CatalogDisposition { - return { status: "skipped", reason: "not-requested", retryable: false }; +function unexpectedCatalogFailure(commitBegan: boolean): CatalogDisposition { + return { + status: "failed", + reason: "disk", + phase: commitBegan ? "commit" : "gather", + retryable: false, + partialWrite: commitBegan, + }; +} + +function admissionFailure(error: unknown): CatalogDisposition { + const message = error instanceof Error ? error.message : ""; + if (message.includes("config generation is busy") || message.includes("config generation is database")) { + return { status: "skipped", reason: "busy", retryable: true }; + } + return unexpectedCatalogFailure(false); } /** Project catalog work into the shared no-change/not-evaluated outcome shape. */ @@ -83,15 +91,24 @@ export function createManagementConvergeCodex( ): ConvergeCodex { const retainedConfig = config; return async request => { - if (request.scope !== "catalog") { - throw new Error("Management Codex convergence accepts only catalog-scoped requests."); + let commitBegan = false; + try { + if (request.scope !== "catalog" || request.action !== "converge") { + return projectCatalogOnlyOutcome({ + changed: false, + catalogRefresh: unexpectedCatalogFailure(false), + }); + } + const snapshot = captureCatalogAdmissionSnapshot(retainedConfig); + const result = await convergeCodexCatalog(snapshot, request, { + onCommitBegin: () => { commitBegan = true; }, + }); + return projectCatalogOnlyOutcome(result); + } catch (error) { + return projectCatalogOnlyOutcome({ + changed: false, + catalogRefresh: commitBegan ? unexpectedCatalogFailure(true) : admissionFailure(error), + }); } - - // WP9 replaces this no-work projection and consumes this exact reference. - void retainedConfig; - return projectCatalogOnlyOutcome({ - changed: false, - catalogRefresh: catalogNotRequested(), - }); }; } diff --git a/src/server/management-api.ts b/src/server/management-api.ts index 3faed8f93..ac6784716 100644 --- a/src/server/management-api.ts +++ b/src/server/management-api.ts @@ -73,6 +73,7 @@ import type { ManagementPrincipal } from "./management-auth"; export type { ManagementApiDeps } from "./management/context"; import { fetchAllModels } from "./management/shared"; import { CatalogGatherBusyError } from "../codex/catalog/provider-fetch"; +import type { CatalogDisposition, ConvergeCodex } from "../codex/convergence-types"; import { managementBodyTooLargeResponse } from "./management/body"; // installed npm version instead of a stale hardcode. @@ -84,6 +85,33 @@ export const VERSION = (() => { } })(); +function isCatalogDisposition(value: unknown): value is CatalogDisposition { + if (!value || typeof value !== "object" || !("status" in value)) return false; + const disposition = value as Record; + if (disposition.status === "committed") { + return typeof disposition.changed === "boolean" + && typeof disposition.degraded === "boolean" + && Array.isArray(disposition.notices) + && disposition.notices.every(notice => notice === "provider-auth" || notice === "provider-network" || notice === "fallback"); + } + if (disposition.status === "skipped") { + return ["not-requested", "catalog-unavailable", "busy", "stale", "refused"].includes(String(disposition.reason)) + && typeof disposition.retryable === "boolean"; + } + if (disposition.status === "failed") { + return ["provider-auth", "provider-network", "disk"].includes(String(disposition.reason)) + && (disposition.phase === "gather" || disposition.phase === "commit") + && typeof disposition.retryable === "boolean" + && typeof disposition.partialWrite === "boolean"; + } + return false; +} + +const managementConvergenceBindings = new WeakMap) => ConvergeCodex; + converge: ConvergeCodex; +}>>(); + export async function handleManagementAPI( req: Request, url: URL, @@ -102,13 +130,38 @@ export async function handleManagementAPI( return jsonResponse({ error: "request body too large" }, 413, req, config); } } - async function refreshCodexCatalogBestEffort(): Promise { - if (deps.refreshCodexCatalog) return deps.refreshCodexCatalog(); + async function convergeCodexCatalog(): Promise { + let convergenceInvoked = false; + let managementConvergeCodex: ConvergeCodex | undefined; try { - const { refreshCodexModelCatalog } = await import("../codex/refresh"); - await refreshCodexModelCatalog(config); + if (!managementConvergeCodex) { + const factory = deps.createManagementConvergeCodex + ?? (await import("../codex/management-convergence")).createManagementConvergeCodex; + if (typeof factory !== "function") throw new TypeError("Catalog convergence factory is unavailable."); + let binding = managementConvergenceBindings.get(config); + if (!binding || binding.factory !== factory) { + const created = factory(config); + if (typeof created !== "function") throw new TypeError("Catalog convergence factory returned no function."); + binding = { factory, converge: created }; + managementConvergenceBindings.set(config, binding); + } + managementConvergeCodex = binding.converge; + } + const { createCatalogConvergeRequest } = await import("../codex/catalog-admission"); + convergenceInvoked = true; + const outcome = await managementConvergeCodex(createCatalogConvergeRequest({ deadlineMs: 1_000 })); + if (!outcome || outcome.kind !== "catalog-only" || !isCatalogDisposition(outcome.catalogRefresh)) { + throw new TypeError("Catalog convergence returned an invalid outcome."); + } + return outcome.catalogRefresh; } catch { - /* catalog absent */ + return { + status: "failed", + reason: "disk", + phase: convergenceInvoked ? "commit" : "gather", + retryable: false, + partialWrite: convergenceInvoked, + }; } } @@ -133,7 +186,7 @@ export async function handleManagementAPI( } } catch { /* best-effort */ } } - const ctx: ManagementContext = { req, url, config, deps, principal, refreshCodexCatalogBestEffort, syncClaudeAgentDefsBestEffort }; + const ctx: ManagementContext = { req, url, config, deps, principal, convergeCodexCatalog, syncClaudeAgentDefsBestEffort }; let routed: Response | null; try { routed = (await handleConfigRoutes(ctx)) diff --git a/src/server/management/agent-settings-routes.ts b/src/server/management/agent-settings-routes.ts index 3d4f1e241..d06d25630 100644 --- a/src/server/management/agent-settings-routes.ts +++ b/src/server/management/agent-settings-routes.ts @@ -125,7 +125,7 @@ export function setGrokApplyFlightTestHooks( import type { ManagementContext } from "./context"; export async function handleAgentSettingsRoutes(ctx: ManagementContext): Promise { - const { req, url, config, deps, refreshCodexCatalogBestEffort, syncClaudeAgentDefsBestEffort } = ctx; + const { req, url, config, deps, convergeCodexCatalog, syncClaudeAgentDefsBestEffort } = ctx; /** Best-effort Desktop 3P config auto-reconcile when providers change. */ async function autoApplyDesktopBestEffort(): Promise { @@ -277,7 +277,7 @@ export async function handleAgentSettingsRoutes(ctx: ManagementContext): Promise if (getAgentsEnabled() === false && isMultiAgentV2Enabled()) { warnings.push("agents.enabled = false has no effect while features.multi_agent_v2 is enabled; upstream keeps V2 active."); } - await refreshCodexCatalogBestEffort(); + const catalogRefresh = await convergeCodexCatalog(); if (requestedFlag !== undefined) warnings.push("Applies to new sessions; restart the Codex app or wait out its picker cache to see the ladder change."); const enabled = isMultiAgentV2Enabled(); return jsonResponse({ @@ -291,6 +291,7 @@ export async function handleAgentSettingsRoutes(ctx: ManagementContext): Promise subagentDeveloperInstructions: getSubagentDeveloperInstructions(), agentsMaxDepthAppliesWhenV2Disabled: !enabled, warnings, + catalogRefresh, }); } @@ -522,10 +523,10 @@ export async function handleAgentSettingsRoutes(ctx: ManagementContext): Promise config.subagentModels = chosen; const { saveConfigPreservingClaudeCode: save } = await import("../../config"); save(config); - await refreshCodexCatalogBestEffort(); + const catalogRefresh = await convergeCodexCatalog(); await syncClaudeAgentDefsBestEffort(); await autoApplyDesktopBestEffort(); - return jsonResponse({ ok: true, applied: chosen }); + return jsonResponse({ ok: true, applied: chosen, catalogRefresh }); } // Priority-ordered subagent model fallback chain for quota-aware spawn routing. diff --git a/src/server/management/combo-routes.ts b/src/server/management/combo-routes.ts index f5d7e5100..81b25d7c8 100644 --- a/src/server/management/combo-routes.ts +++ b/src/server/management/combo-routes.ts @@ -66,7 +66,7 @@ import type { ManagementContext } from "./context"; import { readManagementJsonBody, rethrowManagementBodyTooLarge } from "./body"; export async function handleComboRoutes(ctx: ManagementContext): Promise { - const { req, url, config, deps, refreshCodexCatalogBestEffort, syncClaudeAgentDefsBestEffort } = ctx; + const { req, url, config, deps, convergeCodexCatalog, syncClaudeAgentDefsBestEffort } = ctx; if (url.pathname === "/api/combos" && req.method === "GET") { const { comboPublicModelId, getCombo, listComboIds } = await import("../../combos"); @@ -195,9 +195,9 @@ export async function handleComboRoutes(ctx: ManagementContext): Promise { - const { req, url, config, deps, refreshCodexCatalogBestEffort, syncClaudeAgentDefsBestEffort } = ctx; + const { req, url, config, deps, syncClaudeAgentDefsBestEffort } = ctx; if (url.pathname === "/api/config" && req.method === "GET") { return jsonResponse(safeConfigDTO(config)); } diff --git a/src/server/management/context.ts b/src/server/management/context.ts index 9da702cfb..f39690ba5 100644 --- a/src/server/management/context.ts +++ b/src/server/management/context.ts @@ -5,11 +5,12 @@ import type { ManagementPrincipal } from "../management-auth"; import type { CatalogModel } from "../../codex/catalog"; import type { injectGrokConfig } from "../../grok/inject"; import type { RuntimePortState } from "../../config"; +import type { CatalogDisposition, ConvergeCodex } from "../../codex/convergence-types"; export interface ManagementApiDeps { toggleCodexMultiAgentV2?: (enabled: boolean) => void; toggleDefaultModeRequestUserInput?: (enabled: boolean) => void; - refreshCodexCatalog?: () => Promise; + createManagementConvergeCodex?: (config: Readonly) => ConvergeCodex; /** * Persistence seam for route-level tests. Production leaves this unset and uses * `saveConfigPreservingClaudeCode`; tests that pass an in-memory fixture config @@ -65,6 +66,6 @@ export interface ManagementContext { * tests, which are treated as the untrusted `admin-token` case. */ principal?: ManagementPrincipal; - refreshCodexCatalogBestEffort: () => Promise; + convergeCodexCatalog: () => Promise; syncClaudeAgentDefsBestEffort: () => Promise; } diff --git a/src/server/management/logs-usage-routes.ts b/src/server/management/logs-usage-routes.ts index 225f3998d..6501e5636 100644 --- a/src/server/management/logs-usage-routes.ts +++ b/src/server/management/logs-usage-routes.ts @@ -121,7 +121,7 @@ function refreshedUsageSummary { - const { req, url, config, deps, refreshCodexCatalogBestEffort, syncClaudeAgentDefsBestEffort } = ctx; + const { req, url, config, deps, syncClaudeAgentDefsBestEffort } = ctx; if (url.pathname === "/api/logs" && req.method === "GET") { const all = getRequestLogEntries(); diff --git a/src/server/management/model-routes.ts b/src/server/management/model-routes.ts index 91f724ff4..de218f205 100644 --- a/src/server/management/model-routes.ts +++ b/src/server/management/model-routes.ts @@ -121,7 +121,7 @@ function summarizeExportedModels(client: ExportClientId, document: unknown): { m } export async function handleModelRoutes(ctx: ManagementContext): Promise { - const { req, url, config, deps, refreshCodexCatalogBestEffort, syncClaudeAgentDefsBestEffort } = ctx; + const { req, url, config, deps, convergeCodexCatalog, syncClaudeAgentDefsBestEffort } = ctx; // A handler persists the exact config object passed in. Production defaults to // the real store; tests that pass an in-memory fixture inject a no-op/spy. Do not // bypass this seam with a dynamic config import — doing so replaced a user's @@ -211,8 +211,8 @@ export async function handleModelRoutes(ctx: ManagementContext): Promise typeof m === "string") : []; config.disabledModels = disabled; persistConfig(config); - await refreshCodexCatalogBestEffort(); - return jsonResponse({ ok: true, disabled }); + const catalogRefresh = await convergeCodexCatalog(); + return jsonResponse({ ok: true, disabled, catalogRefresh }); } // One user-facing visibility switch spans two persisted filters: a provider allowlist and the @@ -310,8 +310,8 @@ export async function handleModelRoutes(ctx: ManagementContext): Promise 0 ? list : undefined; persistConfig(config); - await refreshCodexCatalogBestEffort(); - return jsonResponse({ ok: true }); + const catalogRefresh = await convergeCodexCatalog(); + return jsonResponse({ ok: true, catalogRefresh }); } // Per-provider catalog allowlist (issue #52): when a provider has a non-empty selectedModels list, @@ -437,8 +437,8 @@ export async function handleModelRoutes(ctx: ManagementContext): Promise 0) config.providers[provider].selectedModels = models; else delete config.providers[provider].selectedModels; persistConfig(config); - await refreshCodexCatalogBestEffort(); - return jsonResponse({ ok: true, provider, selected: models }); + const catalogRefresh = await convergeCodexCatalog(); + return jsonResponse({ ok: true, provider, selected: models, catalogRefresh }); } return null; } diff --git a/src/server/management/oauth-account-routes.ts b/src/server/management/oauth-account-routes.ts index 4e982fbfd..f3db21c6a 100644 --- a/src/server/management/oauth-account-routes.ts +++ b/src/server/management/oauth-account-routes.ts @@ -112,7 +112,7 @@ function validateKeyName( } export async function handleOauthAccountRoutes(ctx: ManagementContext): Promise { - const { req, url, config, deps, refreshCodexCatalogBestEffort, syncClaudeAgentDefsBestEffort } = ctx; + const { req, url, config, deps, syncClaudeAgentDefsBestEffort } = ctx; // Which providers support real OAuth login (drives the GUI's "Log in with …" buttons). if (url.pathname === "/api/oauth/providers" && req.method === "GET") { diff --git a/src/server/management/provider-routes.ts b/src/server/management/provider-routes.ts index 022c0ece5..46b65b7ff 100644 --- a/src/server/management/provider-routes.ts +++ b/src/server/management/provider-routes.ts @@ -71,7 +71,7 @@ import type { ManagementContext } from "./context"; import { readManagementJsonBody, rethrowManagementBodyTooLarge } from "./body"; export async function handleProviderRoutes(ctx: ManagementContext): Promise { - const { req, url, config, deps, refreshCodexCatalogBestEffort, syncClaudeAgentDefsBestEffort } = ctx; + const { req, url, config, deps, convergeCodexCatalog, syncClaudeAgentDefsBestEffort } = ctx; if (url.pathname === "/api/provider-quotas" && req.method === "GET") { const forceRefresh = url.searchParams.get("refresh") === "1" || url.searchParams.get("refresh") === "true"; @@ -144,8 +144,8 @@ export async function handleProviderRoutes(ctx: ManagementContext): Promise jsonResponse({ ok: true, cap: DEFAULT_PROVIDER_CONTEXT_CAP, value: globalContextCapValue(config), caps: providerContextCaps(config) }); + const respond = (catalogRefresh: Awaited>) => jsonResponse({ + ok: true, + cap: DEFAULT_PROVIDER_CONTEXT_CAP, + value: globalContextCapValue(config), + caps: providerContextCaps(config), + catalogRefresh, + }); // Branch 1: set the global cap value and re-point every enabled provider to it. if (body.value !== undefined) { @@ -509,8 +516,8 @@ export async function handleProviderRoutes(ctx: ManagementContext): Promise { deps: { saveConfigPreservingClaudeCode: () => { persistCalls++; }, } as Parameters[0]["deps"], - refreshCodexCatalogBestEffort: async () => {}, + convergeCodexCatalog: async () => ({ + status: "committed", + changed: false, + degraded: false, + notices: [], + }), syncClaudeAgentDefsBestEffort: async () => {}, }); } diff --git a/tests/codex-convergence-contract.test.ts b/tests/codex-convergence-contract.test.ts new file mode 100644 index 000000000..74af96030 --- /dev/null +++ b/tests/codex-convergence-contract.test.ts @@ -0,0 +1,251 @@ +import { afterEach, beforeEach, expect, test } from "bun:test"; +import { createHash } from "node:crypto"; +import { + existsSync, + lstatSync, + mkdirSync, + mkdtempSync, + readFileSync, + readdirSync, + realpathSync, + renameSync, + rmSync, + writeFileSync, +} from "node:fs"; +import { tmpdir } from "node:os"; +import { join, relative } from "node:path"; + +import { captureCatalogAdmissionSnapshot } from "../src/codex/catalog-admission"; +import { + commitCodexCatalogCandidate, + gatherCodexCatalogCandidate, + type CodexCatalogCandidate, +} from "../src/codex/convergence"; +import { resetCatalogRuntimeStateForTests } from "../src/codex/catalog"; +import { + resetCodexRuntimeResolveCacheForTests, + setCodexRuntimeResolveCacheForTests, +} from "../src/codex/runtime"; +import { + invalidateBundledCatalogCache, + setBundledCatalogCacheForTests, +} from "../src/codex/catalog/bundled"; +import { + resolveCodexCatalogSerializationDatabasePath, + resolveCodexCoordinatorDatabasePath, + resolveEffectiveUserIdentity, +} from "../src/codex/user-identity"; +import { saveConfig } from "../src/config"; +import { handleManagementAPI } from "../src/server/management-api"; +import type { OcxConfig } from "../src/types"; +import { ManagementRequest } from "./helpers/management-auth"; + +let root = ""; +let codexHome = ""; +let opencodexHome = ""; +let previousCodexHome: string | undefined; +let previousOpencodexHome: string | undefined; + +function config(port = 10100): OcxConfig { + return { port, providers: {}, defaultProvider: "openai" }; +} + +function sourceCatalog(marker = "original"): string { + return `${JSON.stringify({ + marker, + models: [{ + slug: "gpt-5.6-sol", + display_name: "GPT-5.6-Sol", + description: "Native", + priority: 1, + visibility: "list", + base_instructions: "You are Codex.", + supported_reasoning_levels: [{ effort: "medium", description: "Medium" }], + }], + }, null, 2)}\n`; +} + +function manifest(base: string): string[] { + const out: string[] = []; + const visit = (directory: string) => { + for (const entry of readdirSync(directory, { withFileTypes: true })) { + const path = join(directory, entry.name); + const stat = lstatSync(path, { bigint: true }); + const name = relative(base, path); + if (entry.isDirectory()) { + out.push(`${name}|dir|${stat.mode}|${stat.mtimeNs}`); + visit(path); + } else { + const bytes = readFileSync(path); + out.push(`${name}|file|${stat.mode}|${stat.mtimeNs}|${bytes.length}|${createHash("sha256").update(bytes).digest("hex")}`); + } + } + }; + visit(base); + return out.sort(); +} + +async function candidate(): Promise { + const gathered = await gatherCodexCatalogCandidate(captureCatalogAdmissionSnapshot(config())); + expect(gathered.kind).toBe("candidate"); + return (gathered as Extract).candidate; +} + +beforeEach(() => { + previousCodexHome = process.env.CODEX_HOME; + previousOpencodexHome = process.env.OPENCODEX_HOME; + root = realpathSync.native(mkdtempSync(join(tmpdir(), "ocx-convergence-"))); + codexHome = join(root, "codex"); + opencodexHome = join(root, "opencodex"); + mkdirSync(codexHome); + mkdirSync(opencodexHome); + process.env.CODEX_HOME = codexHome; + process.env.OPENCODEX_HOME = opencodexHome; + resetCatalogRuntimeStateForTests(); + resetCodexRuntimeResolveCacheForTests(); + saveConfig(config()); + writeFileSync(join(codexHome, "opencodex-catalog.json"), sourceCatalog()); +}); + +afterEach(() => { + const identity = resolveEffectiveUserIdentity(); + const kPath = resolveCodexCatalogSerializationDatabasePath(identity, codexHome); + for (const suffix of ["", "-journal", "-wal", "-shm"]) rmSync(`${kPath}${suffix}`, { force: true }); + if (previousCodexHome === undefined) delete process.env.CODEX_HOME; + else process.env.CODEX_HOME = previousCodexHome; + if (previousOpencodexHome === undefined) delete process.env.OPENCODEX_HOME; + else process.env.OPENCODEX_HOME = previousOpencodexHome; + rmSync(root, { recursive: true, force: true }); +}); + +test("T1 gather performs no filesystem write and does not materialize a runtime probe home", async () => { + process.env.CODEX_CLI_PATH = join(root, "must-not-execute"); + const before = manifest(root); + const gathered = await gatherCodexCatalogCandidate(captureCatalogAdmissionSnapshot(config())); + expect(gathered.kind).toBe("candidate"); + expect(manifest(root)).toEqual(before); + expect(existsSync(join(root, "probe-home"))).toBe(false); + delete process.env.CODEX_CLI_PATH; +}); + +test("commit is fixed-order, receipt-exact, and a consumed candidate cannot be replayed", async () => { + const gathered = await candidate(); + const first = await commitCodexCatalogCandidate(gathered, 1_000); + expect(first).toEqual({ + kind: "committed", + changed: true, + writes: { keyedBackup: "written", legacyBackup: "written", catalog: "written", cache: "written" }, + }); + const after = manifest(root); + expect(await commitCodexCatalogCandidate(gathered, 1_000)).toEqual({ + kind: "stale", + reason: "candidate-consumed", + }); + expect(manifest(root)).toEqual(after); +}); + +test("generation drift rejects before every catalog target write", async () => { + const gathered = await candidate(); + const before = manifest(codexHome); + saveConfig(config(20200)); + expect(await commitCodexCatalogCandidate(gathered, 1_000)).toEqual({ kind: "stale", reason: "generation" }); + expect(manifest(codexHome)).toEqual(before); +}); + +test("home-selection drift rejects before every catalog target write", async () => { + const gathered = await candidate(); + const other = join(root, "other-codex"); + mkdirSync(other); + process.env.CODEX_HOME = other; + expect(await commitCodexCatalogCandidate(gathered, 1_000)).toEqual({ kind: "stale", reason: "home-selection" }); + expect(readdirSync(other)).toEqual([]); +}); + +test("same-inode source drift rejects before every catalog target write", async () => { + const gathered = await candidate(); + const path = join(codexHome, "opencodex-catalog.json"); + const inode = lstatSync(path).ino; + writeFileSync(path, sourceCatalog("drifted")); + expect(lstatSync(path).ino).toBe(inode); + const drifted = readFileSync(path, "utf8"); + expect(await commitCodexCatalogCandidate(gathered, 1_000)).toEqual({ kind: "stale", reason: "source-observation" }); + expect(readFileSync(path, "utf8")).toBe(drifted); + expect(existsSync(join(codexHome, "models_cache.json"))).toBe(false); +}); + +test("target identity drift wins before source comparison and writes nothing", async () => { + const gathered = await candidate(); + const path = join(codexHome, "opencodex-catalog.json"); + const moved = join(codexHome, "moved.json"); + renameSync(path, moved); + writeFileSync(path, readFileSync(moved)); + const before = readFileSync(path, "utf8"); + expect(await commitCodexCatalogCandidate(gathered, 1_000)).toEqual({ kind: "stale", reason: "target-identity" }); + expect(readFileSync(path, "utf8")).toBe(before); + expect(existsSync(join(codexHome, "models_cache.json"))).toBe(false); +}); + +test("used process-local authority drift rejects before every catalog target write", async () => { + const runtime = { command: "/tmp/codex", version: "0.146.0", source: "environment" as const }; + setCodexRuntimeResolveCacheForTests({ runtime, failures: [] }); + setBundledCatalogCacheForTests(runtime, JSON.parse(sourceCatalog("bundled")) as never); + const gathered = await candidate(); + invalidateBundledCatalogCache(); + expect(await commitCodexCatalogCandidate(gathered, 1_000)).toEqual({ kind: "stale", reason: "process-local" }); + expect(existsSync(join(codexHome, "models_cache.json"))).toBe(false); +}); + +test("catalog-only commit never creates the native pair or routing/history artifacts", async () => { + const gathered = await candidate(); + expect((await commitCodexCatalogCandidate(gathered, 1_000)).kind).toBe("committed"); + const nativeDb = resolveCodexCoordinatorDatabasePath(resolveEffectiveUserIdentity(), codexHome); + expect(existsSync(nativeDb)).toBe(false); + expect(existsSync(join(codexHome, "config.toml"))).toBe(false); + expect(manifest(root).join("\n")).not.toContain("journal"); + expect(manifest(root).join("\n")).not.toContain("history"); +}); + +test("the total lazy adapter preserves a persisted-success route when factory construction fails", async () => { + const live = config(); + const request = new ManagementRequest("http://localhost/api/disabled-models", { + method: "PUT", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ models: ["gpt-5.6-sol"] }), + }); + const response = await handleManagementAPI(request, new URL(request.url), live, { + saveConfigPreservingClaudeCode: () => {}, + createManagementConvergeCodex: () => { throw new Error("factory exploded"); }, + }); + expect(response?.status).toBe(200); + expect(await response?.json()).toMatchObject({ + ok: true, + disabled: ["gpt-5.6-sol"], + catalogRefresh: { + status: "failed", + reason: "disk", + phase: "gather", + partialWrite: false, + }, + }); +}); + +test("the route inventory contains exactly the specified 6 + 6 + 2 + 2 convergence calls", () => { + const counts = Object.fromEntries([ + ["provider-routes.ts", 6], + ["model-routes.ts", 6], + ["combo-routes.ts", 2], + ["agent-settings-routes.ts", 2], + ].map(([file, expected]) => { + const source = readFileSync(join(import.meta.dir, "..", "src", "server", "management", file as string), "utf8"); + const count = source.match(/await convergeCodexCatalog\(\)/g)?.length ?? 0; + expect(count).toBe(expected); + expect(source).not.toContain("refreshCodexCatalogBestEffort"); + return [file, count]; + })); + expect(counts).toEqual({ + "provider-routes.ts": 6, + "model-routes.ts": 6, + "combo-routes.ts": 2, + "agent-settings-routes.ts": 2, + }); +}); diff --git a/tests/codex-management-convergence.test.ts b/tests/codex-management-convergence.test.ts index d6e8811c1..6f9df06da 100644 --- a/tests/codex-management-convergence.test.ts +++ b/tests/codex-management-convergence.test.ts @@ -12,7 +12,7 @@ function config(): OcxConfig { return { port: 10100, providers: {}, defaultProvider: "openai" }; } -test("returns an honest catalog-only no-change projection", async () => { +test("projects unavailable generation admission as retryable busy", async () => { const convergeCodex = createManagementConvergeCodex(config()); const outcome = await convergeCodex(createCatalogConvergeRequest({ deadlineMs: 1_000 })); @@ -20,7 +20,7 @@ test("returns an honest catalog-only no-change projection", async () => { expect(outcome).toEqual({ kind: "catalog-only", changed: false, - catalogRefresh: { status: "skipped", reason: "not-requested", retryable: false }, + catalogRefresh: { status: "skipped", reason: "busy", retryable: true }, history: { status: "not-evaluated", attempts: 0, @@ -64,18 +64,24 @@ test("returns an honest catalog-only no-change projection", async () => { }); }); -test("rejects a non-catalog request instead of widening management authority", async () => { +test("refuses a non-catalog request through the total projection", async () => { const convergeCodex = createManagementConvergeCodex(config()); - await expect(convergeCodex({ + const outcome = await convergeCodex({ action: "observe", scope: "full", reason: "cli", mode: "explicit", deadlineMs: 1_000, - })).rejects.toThrow( - "Management Codex convergence accepts only catalog-scoped requests.", - ); + }); + expect(outcome.kind).toBe("catalog-only"); + expect(outcome.catalogRefresh).toEqual({ + status: "failed", + reason: "disk", + phase: "gather", + retryable: false, + partialWrite: false, + }); }); test("constructs the fixed catalog request and ignores caller attempts to choose direction", () => { diff --git a/tests/codex-v2-gate.test.ts b/tests/codex-v2-gate.test.ts index 0a8a86c0b..830c0d3d0 100644 --- a/tests/codex-v2-gate.test.ts +++ b/tests/codex-v2-gate.test.ts @@ -31,6 +31,7 @@ import { } from "../src/codex/features"; import { cmdV2, codexFeaturesInvocation, v2StatusLine, multiAgentModeLine } from "../src/cli/v2"; import { handleManagementAPI } from "../src/server/management-api"; +import { catalogConvergenceFactory } from "./helpers/catalog-convergence"; function template(): Record { return { @@ -646,7 +647,7 @@ describe("management API logical v1/v2 switching", () => { const content = readFileSync(path, "utf8"); writeFileSync(path, content.replace(/^enabled\s*=\s*(?:true|false)$/m, `enabled = ${enabled}`)); }; - const deps = { toggleCodexMultiAgentV2: toggle, refreshCodexCatalog: async () => {} }; + const deps = { toggleCodexMultiAgentV2: toggle, createManagementConvergeCodex: catalogConvergenceFactory() }; try { const toV2 = new Request("http://localhost/api/v2", { method: "PUT", headers: { "content-type": "application/json" }, body: JSON.stringify({ multiAgentMode: "v2" }), @@ -711,7 +712,7 @@ describe("management API logical v1/v2 switching", () => { body: JSON.stringify({ multiAgentMode: "v2", enabled: false }), }); const response = await handleManagementAPI(req, new URL(req.url), { providers: [] } as never, { - toggleCodexMultiAgentV2: () => { toggles++; }, refreshCodexCatalog: async () => {}, + toggleCodexMultiAgentV2: () => { toggles++; }, createManagementConvergeCodex: catalogConvergenceFactory(), }); expect(response?.status).toBe(400); expect(toggles).toBe(0); @@ -722,7 +723,7 @@ describe("management API logical v1/v2 switching", () => { body: JSON.stringify({ multiAgentMode: "v1", enabled: true }), }); expect((await handleManagementAPI(opposite, new URL(opposite.url), { providers: [] } as never, { - toggleCodexMultiAgentV2: () => { toggles++; }, refreshCodexCatalog: async () => {}, + toggleCodexMultiAgentV2: () => { toggles++; }, createManagementConvergeCodex: catalogConvergenceFactory(), }))?.status).toBe(400); expect(toggles).toBe(0); } finally { @@ -732,7 +733,10 @@ describe("management API logical v1/v2 switching", () => { }); describe("management API parity surface for the WP2 keys", () => { - const withConfig = (content: string, run: (path: string, deps: { toggleCodexMultiAgentV2: (enabled: boolean) => void; refreshCodexCatalog: () => Promise }) => Promise) => { + const withConfig = (content: string, run: (path: string, deps: { + toggleCodexMultiAgentV2: (enabled: boolean) => void; + createManagementConvergeCodex: ReturnType; + }) => Promise) => { const path = fixtureConfig(content); const oldCodexHome = process.env.CODEX_HOME; const oldOcxHome = process.env.OPENCODEX_HOME; @@ -742,7 +746,7 @@ describe("management API parity surface for the WP2 keys", () => { const current = readFileSync(path, "utf8"); writeFileSync(path, current.replace(/^enabled\s*=\s*(?:true|false)$/m, `enabled = ${enabled}`)); }; - return run(path, { toggleCodexMultiAgentV2: toggle, refreshCodexCatalog: async () => {} }) + return run(path, { toggleCodexMultiAgentV2: toggle, createManagementConvergeCodex: catalogConvergenceFactory() }) .finally(() => { if (oldCodexHome === undefined) delete process.env.CODEX_HOME; else process.env.CODEX_HOME = oldCodexHome; if (oldOcxHome === undefined) delete process.env.OPENCODEX_HOME; else process.env.OPENCODEX_HOME = oldOcxHome; @@ -854,7 +858,7 @@ describe("management API default_mode_request_user_input toggle", () => { new Request("http://localhost/api/codex-auth/features/default-mode-request-user-input"), new URL("http://localhost/api/codex-auth/features/default-mode-request-user-input"), { providers: [] } as never, - { refreshCodexCatalog: async () => {} }, + { createManagementConvergeCodex: catalogConvergenceFactory() }, ); expect(response?.status).toBe(200); expect(await response?.json()).toEqual({ enabled: false, key: "default_mode_request_user_input" }); @@ -872,7 +876,7 @@ describe("management API default_mode_request_user_input toggle", () => { : `${content}\n[features]\n${line}\n`; writeFileSync(path, next); }; - const deps = { toggleDefaultModeRequestUserInput: toggle, refreshCodexCatalog: async () => {} }; + const deps = { toggleDefaultModeRequestUserInput: toggle, createManagementConvergeCodex: catalogConvergenceFactory() }; const url = new URL("http://localhost/api/codex-auth/features/default-mode-request-user-input"); const on = await handleManagementAPI(putRequest(true), url, { providers: [] } as never, deps); @@ -894,7 +898,7 @@ describe("management API default_mode_request_user_input toggle", () => { putRequest("yes"), new URL("http://localhost/api/codex-auth/features/default-mode-request-user-input"), { providers: [] } as never, - { toggleDefaultModeRequestUserInput: () => { toggles++; }, refreshCodexCatalog: async () => {} }, + { toggleDefaultModeRequestUserInput: () => { toggles++; }, createManagementConvergeCodex: catalogConvergenceFactory() }, ); expect(response?.status).toBe(400); expect(toggles).toBe(0); @@ -911,7 +915,7 @@ describe("management API default_mode_request_user_input toggle", () => { }), url, { providers: [] } as never, - { toggleDefaultModeRequestUserInput: () => { throw new Error("must not toggle"); }, refreshCodexCatalog: async () => {} }, + { toggleDefaultModeRequestUserInput: () => { throw new Error("must not toggle"); }, createManagementConvergeCodex: catalogConvergenceFactory() }, ); expect(response?.status).toBe(400); } @@ -934,7 +938,7 @@ describe("management API default_mode_request_user_input toggle", () => { }), new URL("http://localhost/api/codex-auth/features/default-mode-request-user-input"), { providers: [] } as never, - { toggleDefaultModeRequestUserInput: () => { throw new Error("must not toggle"); }, refreshCodexCatalog: async () => {} }, + { toggleDefaultModeRequestUserInput: () => { throw new Error("must not toggle"); }, createManagementConvergeCodex: catalogConvergenceFactory() }, ); expect(response?.status).toBe(413); }); @@ -951,7 +955,7 @@ describe("management API default_mode_request_user_input toggle", () => { putRequest(true), new URL("http://localhost/api/codex-auth/features/default-mode-request-user-input"), { providers: [] } as never, - { toggleDefaultModeRequestUserInput: toggle, refreshCodexCatalog: async () => {} }, + { toggleDefaultModeRequestUserInput: toggle, createManagementConvergeCodex: catalogConvergenceFactory() }, ); expect(response?.status).toBe(502); const body = await response?.json(); @@ -965,7 +969,7 @@ describe("management API default_mode_request_user_input toggle", () => { putRequest(true), new URL("http://localhost/api/codex-auth/features/default-mode-request-user-input"), { providers: [] } as never, - { toggleDefaultModeRequestUserInput: () => {}, refreshCodexCatalog: async () => {} }, + { toggleDefaultModeRequestUserInput: () => {}, createManagementConvergeCodex: catalogConvergenceFactory() }, ); expect(response?.status).toBe(502); expect(await response?.json()).toMatchObject({ error: expect.stringContaining("default_mode_request_user_input toggle failed") }); diff --git a/tests/combo-management-api.test.ts b/tests/combo-management-api.test.ts index 3ac3fac4e..fd4701b90 100644 --- a/tests/combo-management-api.test.ts +++ b/tests/combo-management-api.test.ts @@ -40,6 +40,7 @@ import { handleResponses } from "../src/server/responses"; import type { OcxConfig } from "../src/types"; import { syncCatalogModels } from "../src/codex/catalog"; import { injectClaudeAgentDefs } from "../src/claude/agents-inject"; +import { catalogConvergenceFactory } from "./helpers/catalog-convergence"; const VALID_COMBO = { targets: [{ provider: "a", model: "m1" }] }; @@ -127,7 +128,7 @@ async function comboApi( body: body === undefined ? undefined : JSON.stringify(body), }); return handleManagementAPI(req, new URL(req.url), config, { - refreshCodexCatalog, + createManagementConvergeCodex: catalogConvergenceFactory(refreshCodexCatalog), }); } @@ -138,7 +139,7 @@ async function comboApiRaw(config: OcxConfig, method: string, path: string, body body, }); return handleManagementAPI(req, new URL(req.url), config, { - refreshCodexCatalog: async () => {}, + createManagementConvergeCodex: catalogConvergenceFactory(), }); } diff --git a/tests/combos.test.ts b/tests/combos.test.ts index ae8c4643f..f2705d643 100644 --- a/tests/combos.test.ts +++ b/tests/combos.test.ts @@ -42,6 +42,7 @@ import type { OcxConfig } from "../src/types"; import { syncCatalogModels } from "../src/codex/catalog"; import { injectClaudeAgentDefs } from "../src/claude/agents-inject"; import { reconcileComboRotationState } from "../src/combos/resolve"; +import { catalogConvergenceFactory } from "./helpers/catalog-convergence"; const VALID_COMBO = { targets: [{ provider: "a", model: "m1" }] }; @@ -129,7 +130,7 @@ async function comboApi( body: body === undefined ? undefined : JSON.stringify(body), }); return handleManagementAPI(req, new URL(req.url), config, { - refreshCodexCatalog, + createManagementConvergeCodex: catalogConvergenceFactory(refreshCodexCatalog), }); } @@ -140,7 +141,7 @@ async function comboApiRaw(config: OcxConfig, method: string, path: string, body body, }); return handleManagementAPI(req, new URL(req.url), config, { - refreshCodexCatalog: async () => {}, + createManagementConvergeCodex: catalogConvergenceFactory(), }); } diff --git a/tests/helpers/catalog-convergence.ts b/tests/helpers/catalog-convergence.ts new file mode 100644 index 000000000..c75d53ec7 --- /dev/null +++ b/tests/helpers/catalog-convergence.ts @@ -0,0 +1,15 @@ +import { projectCatalogOnlyOutcome } from "../../src/codex/management-convergence"; +import type { ConvergeCodex } from "../../src/codex/convergence-types"; +import type { OcxConfig } from "../../src/types"; + +export function catalogConvergenceFactory( + run: () => Promise | void = () => {}, +): (config: Readonly) => ConvergeCodex { + return () => async () => { + await run(); + return projectCatalogOnlyOutcome({ + changed: false, + catalogRefresh: { status: "committed", changed: false, degraded: false, notices: [] }, + }); + }; +} diff --git a/tests/management-client-config-route.test.ts b/tests/management-client-config-route.test.ts index a6d296688..c5fa46912 100644 --- a/tests/management-client-config-route.test.ts +++ b/tests/management-client-config-route.test.ts @@ -14,6 +14,7 @@ import { type PiGeneratedConfig, } from "../src/clients/config-export"; import type { OcxConfig } from "../src/types"; +import { catalogConvergenceFactory } from "./helpers/catalog-convergence"; /** * A key that looks exactly like a real one. Every assertion about `ocx_` absence is @@ -81,7 +82,7 @@ async function clientConfigApi(config: OcxConfig, query: string): Promise {}, refreshCodexCatalog: async () => {} }, + { saveConfigPreservingClaudeCode: () => {}, createManagementConvergeCodex: catalogConvergenceFactory() }, ); expect(response).not.toBeNull(); return response!; @@ -93,7 +94,7 @@ async function modelRows(config: OcxConfig): Promise { new Request(url, { headers: { Host: url.host } }), url, config, - { saveConfigPreservingClaudeCode: () => {}, refreshCodexCatalog: async () => {} }, + { saveConfigPreservingClaudeCode: () => {}, createManagementConvergeCodex: catalogConvergenceFactory() }, ); return await response!.json() as ModelRow[]; } @@ -243,7 +244,7 @@ describe("GET /api/client-config", () => { new Request(url, { headers: { Host: url.host, Origin: "https://evil.example" } }), url, baseConfig(), - { saveConfigPreservingClaudeCode: () => {}, refreshCodexCatalog: async () => {} }, + { saveConfigPreservingClaudeCode: () => {}, createManagementConvergeCodex: catalogConvergenceFactory() }, ); expect(response?.status).toBe(403); }, 15_000); diff --git a/tests/management-integration-routes.test.ts b/tests/management-integration-routes.test.ts index 8314cacc1..b65b74d3d 100644 --- a/tests/management-integration-routes.test.ts +++ b/tests/management-integration-routes.test.ts @@ -15,6 +15,7 @@ import { setIntegrationPathTestHooks, } from "../src/server/management/integration-routes"; import type { OcxConfig } from "../src/types"; +import { catalogConvergenceFactory } from "./helpers/catalog-convergence"; /** * Route contract for devlog/_plan/260802_client_toggle_api/040 §6-§7. @@ -122,7 +123,7 @@ async function rawApi(path: string, init: RequestInit = {}): Promise {}, refreshCodexCatalog: async () => {} }, + { saveConfigPreservingClaudeCode: () => {}, createManagementConvergeCodex: catalogConvergenceFactory() }, ); } diff --git a/tests/management-provider-validation.test.ts b/tests/management-provider-validation.test.ts index 7f4d9e67e..de32a7831 100644 --- a/tests/management-provider-validation.test.ts +++ b/tests/management-provider-validation.test.ts @@ -35,6 +35,7 @@ import type { OcxConfig } from "../src/types"; import { fakeChatGptJwt } from "./helpers/fake-chatgpt-jwt"; import { installIsolatedCodexHome, type IsolatedCodexHome } from "./helpers/isolated-codex-home"; import * as destinationPolicy from "../src/lib/destination-policy"; +import { catalogConvergenceFactory } from "./helpers/catalog-convergence"; // Full-suite Windows load: startServer + multi-step provider PATCH/GET flows exceed the // default 5s per-test budget (same flake class as 810fa115 / claude-management-api). @@ -619,7 +620,7 @@ describe("provider management validation", () => { }), requestUrl, cfg, - { refreshCodexCatalog: async () => {} }, + { createManagementConvergeCodex: catalogConvergenceFactory() }, ); expect(response?.status).toBe(409); @@ -1299,7 +1300,7 @@ describe("provider management validation", () => { body: JSON.stringify(body), }); return handleManagementAPI(request, new URL(request.url), liveConfig, { - refreshCodexCatalog: async () => undefined, + createManagementConvergeCodex: catalogConvergenceFactory(), }); }; const canonical = await post({ name: "openai", provider: canonicalDirect }); @@ -1354,7 +1355,7 @@ describe("provider management validation", () => { body: JSON.stringify({ name: "openai", provider: canonicalDirect }), }); const response = await handleManagementAPI(request, new URL(request.url), liveConfig, { - refreshCodexCatalog: async () => undefined, + createManagementConvergeCodex: catalogConvergenceFactory(), }); expect(response?.status).toBe(400); expect(await response?.json()).toMatchObject({ @@ -1508,7 +1509,7 @@ describe("provider management validation", () => { body: JSON.stringify({ disabled: false }), }); const response = await handleManagementAPI(request, new URL(request.url), liveConfig, { - refreshCodexCatalog: async () => undefined, + createManagementConvergeCodex: catalogConvergenceFactory(), }); expect(response?.status).toBe(200); expect(resolvedError).toHaveBeenCalledTimes(1); @@ -1564,7 +1565,7 @@ describe("provider management validation", () => { body: JSON.stringify({ disabled: false }), }); const response = await handleManagementAPI(request, new URL(request.url), liveConfig, { - refreshCodexCatalog: async () => undefined, + createManagementConvergeCodex: catalogConvergenceFactory(), }); expect(response?.status).toBe(400); expect(await response?.json()).toMatchObject({ error }); @@ -1623,7 +1624,7 @@ describe("provider management validation", () => { body: JSON.stringify({ disabled: false }), }); const response = await handleManagementAPI(request, new URL(request.url), liveConfig, { - refreshCodexCatalog: async () => undefined, + createManagementConvergeCodex: catalogConvergenceFactory(), }); expect(response?.status).toBe(400); expect(liveConfig.providers.openai).toMatchObject({ @@ -1675,7 +1676,7 @@ describe("provider management validation", () => { body: JSON.stringify({ disabled: false }), }); const response = await handleManagementAPI(request, new URL(request.url), liveConfig, { - refreshCodexCatalog: async () => undefined, + createManagementConvergeCodex: catalogConvergenceFactory(), }); expect(response?.status).toBe(200); expect(liveConfig.providers.openai).toEqual({ @@ -1770,7 +1771,7 @@ describe("provider management validation", () => { const deps = { clearThreadAccountMap: () => { affinityClears += 1; }, clearProviderQuotaCache: () => { quotaCacheClears += 1; }, - refreshCodexCatalog: async () => { catalogRefreshes += 1; }, + createManagementConvergeCodex: catalogConvergenceFactory(() => { catalogRefreshes += 1; }), primeCodexPoolQuotas: (_config: OcxConfig, reason: string) => { primes.push(reason); }, }; const patch = async (name: string, body: unknown) => { @@ -1848,7 +1849,7 @@ describe("provider management validation", () => { body: JSON.stringify(body), }); return handleManagementAPI(req, new URL(req.url), liveConfig, { - refreshCodexCatalog: async () => { catalogRefreshes += 1; }, + createManagementConvergeCodex: catalogConvergenceFactory(() => { catalogRefreshes += 1; }), }); }; diff --git a/tests/model-visibility-management-api.test.ts b/tests/model-visibility-management-api.test.ts index 77ec85b14..d7ea73f42 100644 --- a/tests/model-visibility-management-api.test.ts +++ b/tests/model-visibility-management-api.test.ts @@ -5,6 +5,7 @@ import { nativeModelRows } from "../src/codex/catalog"; import { loadConfig, saveConfig } from "../src/config"; import { handleManagementAPI } from "../src/server/management-api"; import { installIsolatedCodexHome, type IsolatedCodexHome } from "./helpers/isolated-codex-home"; +import { catalogConvergenceFactory } from "./helpers/catalog-convergence"; const TEST_DIR = join(import.meta.dir, `.tmp-model-visibility-management-${process.pid}`); const previousOpencodexHome = process.env.OPENCODEX_HOME; @@ -52,7 +53,7 @@ async function putWithConfig(body: unknown, config = loadConfig()): Promise { refreshes += 1; } }); + }), url, config, { createManagementConvergeCodex: catalogConvergenceFactory(() => { refreshes += 1; }) }); if (!response) throw new Error("model visibility route was not handled"); return response; } diff --git a/tests/responses-shadow-intercept.test.ts b/tests/responses-shadow-intercept.test.ts index e946f65b4..dbc3d3e4d 100644 --- a/tests/responses-shadow-intercept.test.ts +++ b/tests/responses-shadow-intercept.test.ts @@ -11,6 +11,7 @@ import { handleResponses, isShadowSourceModel } from "../src/server/responses"; import { shouldInterceptShadowCall } from "../src/lib/shadow-call"; import { handleManagementAPI } from "../src/server/management-api"; import type { OcxConfig } from "../src/types"; +import { catalogConvergenceFactory } from "./helpers/catalog-convergence"; const originalFetch = globalThis.fetch; @@ -197,7 +198,9 @@ async function shadowApi(config: OcxConfig, method: string, body?: unknown): Pro headers, body: body === undefined ? undefined : JSON.stringify(body), }); - const res = await handleManagementAPI(req, new URL(req.url), config, { refreshCodexCatalog: async () => {} }); + const res = await handleManagementAPI(req, new URL(req.url), config, { + createManagementConvergeCodex: catalogConvergenceFactory(), + }); expect(res).not.toBeNull(); expect(res!.status).toBe(200); return await res!.json() as Record; diff --git a/tests/server-combo-failover-e2e.test.ts b/tests/server-combo-failover-e2e.test.ts index 718b1773c..f0063b873 100644 --- a/tests/server-combo-failover-e2e.test.ts +++ b/tests/server-combo-failover-e2e.test.ts @@ -28,6 +28,7 @@ import { } from "../src/codex/routing"; import { startServer } from "../src/server"; import { fakeChatGptJwt } from "./helpers/fake-chatgpt-jwt"; +import { catalogConvergenceFactory } from "./helpers/catalog-convergence"; // Full-suite Windows load: startServer + combo rename/delete management flows exceed the // default 5s per-test budget (same flake class as 810fa115 / claude-management-api). @@ -343,7 +344,7 @@ async function management( body: body === undefined ? undefined : JSON.stringify(body), }); return handleManagementAPI(request, new URL(request.url), config, { - refreshCodexCatalog: async () => {}, + createManagementConvergeCodex: catalogConvergenceFactory(), }); } From 5f76205341bd5f56162895a33864df3988826361 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Wed, 5 Aug 2026 00:46:01 +0900 Subject: [PATCH 077/163] fix(codex): runtime authority can move without touching a single catalog byte MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The freshness evidence was built from catalog, backup and cache bytes, so a retained sync that prepared under one Codex runtime and awaited its provider would commit happily after another process selected a different runtime — every byte it compared was untouched. `codex-runtime.json` is now part of the pre-await filesystem evidence, PRESENT or ABSENT, since its absence is what makes the resolver fall back. The in-memory runtime memo is deliberately NOT evidence, and getting that honest took three attempts. Baselining it before the await detected our own side effect, because gathering resolves the runtime, and every sync refused to write a catalog nobody had touched. Baselining it after the await removed that but was worse: a runtime moved by another process during the await got captured as though it were ours, which is the R1-to-R2 case the verifier reproduced. Pre-settling it from this path does not work either — gather resolves lazily under its own cache key, so the memo here stays `unavailable` while gather turns it `available`. So runtime authority is covered where it is actually durable, the bundled template is settled before the baseline is taken, and the residue is written down rather than papered over: a same-process in-memory runtime swap that never touches the persisted file is still undetectable here, and WP11's lock is what makes that case decidable. The regression is the verifier's exact scenario — another process rewrites the runtime selection mid-await while the catalog is untouched. Dropping `runtimeStateBytes` turns it red and leaves the other four retained-root tests green, which is precisely why it had to be written: nothing in the suite covered that component, so the mechanism could be deleted with 8258 tests still passing. --- src/codex/catalog/sync.ts | 44 ++++++---- .../codex-retained-root-serialization.test.ts | 83 +++++++++++++++++++ 2 files changed, 112 insertions(+), 15 deletions(-) diff --git a/src/codex/catalog/sync.ts b/src/codex/catalog/sync.ts index 545151bf8..95345a904 100644 --- a/src/codex/catalog/sync.ts +++ b/src/codex/catalog/sync.ts @@ -54,7 +54,7 @@ import { replaceActiveCodexCatalog, replaceCodexModelsCache, } from "../internal/catalog-writer"; -import { peekCodexRuntimeProcessCache } from "../runtime"; +import { codexRuntimeStatePath } from "../runtime"; export const MAX_SPAWN_AGENT_MODEL_OVERRIDES = 5; @@ -581,24 +581,33 @@ function retainedCatalogSyncEvidence( legacyBackupBytes: isDefaultCatalogPath(catalogPath) ? optionalFileBytes(legacyCatalogBackupPath()) : null, modelsCacheBytes: optionalFileBytes(activeCodexModelsCachePath()), + // The persisted runtime selection is a pre-await filesystem input, not a + // process epoch: another PROCESS can move runtime authority by rewriting this + // file, and that move is invisible to our in-process memo. Recorded PRESENT or + // ABSENT, because its absence is what makes the resolver fall back. + runtimeStateBytes: optionalFileBytes(codexRuntimeStatePath()), }); } /** - * The process-local half of the same evidence, observed separately. + * The bundled-template half of the same evidence, observed separately. * - * These epochs belong in the freshness comparison — a runtime or bundled-template - * swap mid-gather changes what the candidate means — but they cannot share the - * filesystem baseline. Gathering RESOLVES the Codex runtime, so a pre-gather - * snapshot always disagrees with itself afterwards, and every sync refused to write - * a catalog nobody else had touched. The filesystem bytes are therefore baselined - * before the await (an outside writer must lose), while these are baselined once our - * own observation is finished (only an outside writer moving them afterwards counts). + * The runtime process memo is deliberately NOT here, and that exclusion took three + * attempts to get honest. Gathering resolves the Codex runtime lazily and under its + * own cache key, so this path cannot pre-settle that memo: baselining it before the + * await always detected our own side effect and refused every write, and baselining + * it after the await captured a runtime that ANOTHER process had moved as though it + * were ours — a catalog prepared from R1 committing after authority reached R2. + * + * Runtime authority is covered where it is actually durable instead: the persisted + * `codex-runtime.json` bytes sit in the pre-await filesystem evidence, PRESENT or + * ABSENT, so a cross-process runtime move is caught. What is left uncovered, and is + * written down rather than papered over, is a same-process in-memory runtime swap + * that never touches that file — WP11 owns the lock that makes that case decidable. */ function retainedCatalogProcessEvidence(): string { return JSON.stringify({ bundledCatalogCache: bundledCatalogCacheState(), - runtimeProcessCache: peekCodexRuntimeProcessCache(), }); } @@ -737,15 +746,20 @@ export async function syncCatalogModels(config: OcxConfig): Promise { const current = revalidateRetainedCatalogSync(config, prepared); if (current === null) return null; diff --git a/tests/codex-retained-root-serialization.test.ts b/tests/codex-retained-root-serialization.test.ts index 5c745d4cf..6668b11c8 100644 --- a/tests/codex-retained-root-serialization.test.ts +++ b/tests/codex-retained-root-serialization.test.ts @@ -306,3 +306,86 @@ for (const publisher of ["convergence", "retained"] as const) { expect(readFileSync(catalogPath, "utf8")).toBe(newer); }, 20_000); } + +/** + * Runtime authority can move without touching the catalog at all. + * + * The verifier's R1→R2 case: a retained sync prepares from one Codex runtime, + * and while it is awaiting its provider another process rewrites the persisted + * runtime selection. Every catalog byte is untouched, so a freshness check built + * only from catalog/backup/cache bytes sees nothing and commits a candidate that + * was derived under a runtime that is no longer selected. + * + * `codex-runtime.json` is therefore part of the pre-await filesystem evidence, + * PRESENT or ABSENT. Removing it from `retainedCatalogSyncEvidence` turns this + * test red while every other retained-root test stays green — which is exactly + * why it exists: nothing else in the suite covered that component. + */ +test("a persisted runtime selection moved by another process during the await blocks the write", async () => { + const sandbox = makeSandbox("ocx-retained-runtime-move-"); + const catalogPath = seedCatalog(sandbox); + const initial = readFileSync(catalogPath, "utf8"); + const requested = join(sandbox.root, "provider-requested"); + const release = join(sandbox.root, "provider-release"); + const runtimeStatePath = join(sandbox.opencodexHome, "codex-runtime.json"); + writeFileSync(runtimeStatePath, `${JSON.stringify({ + version: 1, + command: "/usr/local/bin/codex-r1", + source: "configured", + selectedVersion: "1.0.0", + updatedAt: new Date(0).toISOString(), + }, null, 2)}\n`); + + const config = { + port: 10100, + defaultProvider: "together", + providers: { + together: { + adapter: "openai-chat", + baseUrl: "https://api.together.xyz/v1", + apiKey: "runtime-move-key", + models: ["fallback-model"], + }, + }, + }; + + const sync = Bun.spawn([process.execPath, "--eval", ` + import { existsSync, writeFileSync } from "node:fs"; + const config = ${JSON.stringify(config)}; + config.providers.together.fetch = async () => { + writeFileSync(${JSON.stringify(requested)}, "requested"); + while (!existsSync(${JSON.stringify(release)})) await Bun.sleep(5); + return Response.json({ data: [{ id: "runtime-move-model" }] }); + }; + const { syncCatalogModels } = await import("./src/codex/catalog/sync.ts"); + console.log(JSON.stringify(await syncCatalogModels(config))); + `], { cwd: repoRoot, env: sandbox.env, stdout: "pipe", stderr: "pipe" }); + + await Promise.race([ + waitForPath(requested), + sync.exited.then(async exitCode => { + const stdout = await new Response(sync.stdout).text(); + const stderr = await new Response(sync.stderr).text(); + throw new Error(`sync exited before provider barrier (${exitCode})\nstdout=${stdout}\nstderr=${stderr}`); + }), + ]); + + // Another process selects a different Codex runtime. No catalog byte changes. + writeFileSync(runtimeStatePath, `${JSON.stringify({ + version: 1, + command: "/usr/local/bin/codex-r2", + source: "configured", + selectedVersion: "2.0.0", + updatedAt: new Date(1).toISOString(), + }, null, 2)}\n`); + + writeFileSync(release, "release"); + const [exitCode, stdout, stderr] = await Promise.all([ + sync.exited, + new Response(sync.stdout).text(), + new Response(sync.stderr).text(), + ]); + expect({ exitCode, stderr }).toMatchObject({ exitCode: 0 }); + expect(JSON.parse(stdout.trim())).toMatchObject({ catalogWritten: false }); + expect(readFileSync(catalogPath, "utf8")).toBe(initial); +}, 20_000); From 119a17bb3d90a59e701280c8c3ff2ebb3bee0b39 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Wed, 5 Aug 2026 01:25:49 +0900 Subject: [PATCH 078/163] test(codex): race the post-approval seam, and find it never commits MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The B-phase evidence this work-phase actually owed was a two-process race against a real management route, not against `/api/sync`. Every earlier race drove a retained root; this one drives `PATCH /api/providers` — one of the sixteen mutations that used to reach a catalog write through a helper whose whole error policy was `catch {}` — with both processes arriving after the route has already persisted its own edit and approved the refresh. Writing it surfaced a defect the suite could not see. Called directly, the bound factory returns `skipped/busy`. Called after `saveConfigPreservingClaudeCode`, which is precisely what all sixteen routes do before approving, it returns `failed/disk/gather` — so on the real production path the seam never reaches a commit at all. It reproduces single-process, so it is not contention, and the route still answers 2xx because the adapter is total, which is why 8258 tests stayed green over it. The assertion therefore encodes the CURRENT behaviour rather than the intended one, and says so: no process commits, and the seeded catalog is untouched. It will fail the moment the seam starts committing, which is exactly when it must be rewritten to demand `committed` and a moved catalog. Recording it at the seam it lives in beats a note nobody reads. Two pre-approval races are excluded by name rather than by a bare exit check — the config mutation lock, and two cold processes creating the ownership file at once. Neither concerns catalog convergence, and narrowing them to those two strings keeps a genuine seam failure failing. The ownership file is warmed first so both processes actually reach the seam; without it the test was vacuous. --- .../codex-retained-root-serialization.test.ts | 141 ++++++++++++++++++ 1 file changed, 141 insertions(+) diff --git a/tests/codex-retained-root-serialization.test.ts b/tests/codex-retained-root-serialization.test.ts index 6668b11c8..f59b9874b 100644 --- a/tests/codex-retained-root-serialization.test.ts +++ b/tests/codex-retained-root-serialization.test.ts @@ -389,3 +389,144 @@ test("a persisted runtime selection moved by another process during the await bl expect(JSON.parse(stdout.trim())).toMatchObject({ catalogWritten: false }); expect(readFileSync(catalogPath, "utf8")).toBe(initial); }, 20_000); + +/** + * The post-approval seam, raced by two real processes through a real route. + * + * Every case above drives `/api/sync`, which is a retained root. This one drives + * `PATCH /api/providers` — one of the sixteen management mutations that used to + * reach a catalog write through `refreshCodexCatalogBestEffort`, whose entire + * error policy was `catch {}`. The interesting window is AFTER the route has + * already persisted its own mutation and approved the refresh: two processes + * arriving there together must serialize, and neither may report `committed` + * for bytes the other replaced. + * + * Both processes go through `handleManagementAPI`, so this exercises the bound + * factory, the total adapter, and K in the shape production actually uses. + * + * The edit itself is a `note` update: a recognized field that persists a real + * config mutation without changing routing, so what is under test is the refresh + * that follows approval rather than the edit. + */ +test("two processes at the post-approval management seam serialize instead of interleaving", async () => { + const sandbox = makeSandbox("ocx-post-approval-race-"); + const catalogPath = seedCatalog(sandbox); + const seeded = readFileSync(catalogPath, "utf8"); + const barrier = join(sandbox.root, "seam-barrier"); + + // Warm the config ownership + mutation database in a single process first. + // Two cold processes otherwise race to create `.opencodex-owner.json` and both + // die with EEXIST before approval, which would make this test vacuous. + const warm = Bun.spawn([process.execPath, "--eval", ` + const { withConfigMutationLockSync } = await import("./src/config.ts"); + withConfigMutationLockSync(() => undefined); + `], { cwd: repoRoot, env: sandbox.env, stdout: "pipe", stderr: "pipe" }); + expect(await warm.exited).toBe(0); + + const routeScript = (marker: string) => ` + import { existsSync, writeFileSync } from "node:fs"; + const config = { + port: 10100, + defaultProvider: "together", + providers: { + together: { + adapter: "openai-chat", + baseUrl: "https://api.together.xyz/v1", + apiKey: "seam-key", + models: ["fallback-model"], + fetch: async () => { + // Announce arrival, then wait for the sibling so both processes are + // past approval and inside the seam at the same time. + writeFileSync(${JSON.stringify(barrier)} + "-" + ${JSON.stringify(marker)}, "here"); + const deadline = Date.now() + 8000; + while (Date.now() < deadline) { + if (existsSync(${JSON.stringify(barrier)} + "-a") && existsSync(${JSON.stringify(barrier)} + "-b")) break; + await Bun.sleep(5); + } + return Response.json({ data: [{ id: "seam-model-" + ${JSON.stringify(marker)} }] }); + }, + }, + }, + }; + const { handleManagementAPI } = await import("./src/server/management-api.ts"); + const url = new URL("http://localhost/api/providers?name=together"); + const req = new Request(url, { + method: "PATCH", + headers: { Host: "localhost", "content-type": "application/json" }, + body: JSON.stringify({ note: "seam-" + ${JSON.stringify(marker)} }), + }); + const response = await handleManagementAPI(req, url, config); + const body = await response.json(); + console.log(JSON.stringify({ status: response.status, catalogRefresh: body.catalogRefresh })); + `; + + const children = (["a", "b"] as const).map(marker => Bun.spawn( + [process.execPath, "--eval", routeScript(marker)], + { cwd: repoRoot, env: sandbox.env, stdout: "pipe", stderr: "pipe" }, + )); + + const results = await Promise.all(children.map(async child => { + const [exitCode, stdout, stderr] = await Promise.all([ + child.exited, + new Response(child.stdout).text(), + new Response(child.stderr).text(), + ]); + return { exitCode, stdout, stderr }; + })); + + for (const result of results) { + // A process can lose a race BEFORE approval and never reach the seam at all. + // Both known cases come from `saveConfigPreservingClaudeCode`: the config + // mutation lock is already held, or two cold processes create the ownership + // file at once. Neither says anything about catalog convergence, so they are + // excluded here — but only these two, so a genuine seam failure still fails. + if (result.exitCode !== 0) { + const preApproval = result.stderr.includes("CONFIG_MUTATION_LOCK_UNAVAILABLE") + || (result.stderr.includes("EEXIST") && result.stderr.includes("createOwnership")); + expect({ preApproval, stderr: result.stderr }).toMatchObject({ preApproval: true }); + continue; + } + const parsed = JSON.parse(result.stdout.trim()) as { + status: number; + catalogRefresh: { status: string }; + }; + // The route persisted its mutation, so it must answer 2xx no matter what the + // catalog attempt decided. A throw here would be the old `catch {}` failure + // inverted: a persisted change reported as a 500. + expect(parsed.status).toBeGreaterThanOrEqual(200); + expect(parsed.status).toBeLessThan(300); + // Whatever happened, it is REPORTED — never swallowed into silence. + expect(["committed", "skipped", "failed"]).toContain(parsed.catalogRefresh.status); + } + + // At least one process must have gotten through to the seam, or this test would + // be vacuous — two config-lock losers prove nothing about catalog serialization. + expect(results.some(r => r.exitCode === 0)).toBe(true); + + // KNOWN DEFECT, asserted so it cannot be forgotten: no process commits here. + // + // Called directly, the bound factory returns `skipped/busy`. Called after + // `saveConfigPreservingClaudeCode` — which is exactly what every one of the + // sixteen routes does before approving the refresh — it returns + // `failed/disk/gather` instead, so the seam never reaches a commit on the real + // production path. Reproduced single-process, so it is not contention. + // + // This assertion is deliberately the CURRENT behaviour rather than the intended + // one: it documents the defect at the seam it lives in, and it will fail the + // moment the seam starts committing, which is when it must be rewritten to + // require `committed` and a moved catalog. Tracked for WP9 closure; the fix is + // not in this test's scope. + const dispositions = results + .filter(r => r.exitCode === 0) + .map(r => (JSON.parse(r.stdout.trim()) as { catalogRefresh: { status: string } }).catalogRefresh.status); + expect(dispositions.every(status => status !== "committed")).toBe(true); + expect(readFileSync(catalogPath, "utf8")).toBe(seeded); + + // The surviving catalog is one process's complete output, never a blend of both. + const finalBytes = readFileSync(catalogPath, "utf8"); + const parsedCatalog = JSON.parse(finalBytes) as { models: Array<{ slug?: unknown }> }; + const slugs = parsedCatalog.models.flatMap(m => typeof m.slug === "string" ? [m.slug] : []); + const fromA = slugs.some(s => s.includes("seam-model-a")); + const fromB = slugs.some(s => s.includes("seam-model-b")); + expect(fromA && fromB).toBe(false); +}, 30_000); From ecef67ef6b7be50711d6020b5044a1978d031dd1 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Wed, 5 Aug 2026 01:28:22 +0900 Subject: [PATCH 079/163] =?UTF-8?q?test(codex):=20the=20seam=20does=20comm?= =?UTF-8?q?it=20=E2=80=94=20my=20fixture=20was=20the=20defect?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Yesterday this test asserted that no process ever commits at the post-approval seam, and called it a production defect blocking WP9. That was wrong, and the encoder said so plainly once I asked it: "Catalog config identity cannot encode function." The fixture put the provider stub on the config row itself. Catalog admission encodes the config to derive its identity and refuses a function member, so the snapshot threw before convergence could run, and the total adapter turned that into `failed/disk/gather`. Both halves of my evidence pointed the wrong way: the "direct call returns busy, post-save call fails" contrast was really "no function member versus function member", and single-process reproduction confirmed a fixture bug rather than ruling out contention. The stub moves to `globalThis.fetch`, which is how the rest of the suite does it. The seam now reaches a real commit, so the assertion says what it should have said from the start: at least one process commits, and the catalog actually moves. The two earlier `/api/sync` races kept their per-provider stub because that path never captures an admission snapshot — which is exactly why they were green while this one was not. It also discriminates now. Making the factory throw turns this red; yesterday's version passed that same mutation, because asserting "nothing commits" is satisfied by a seam that cannot commit at all. --- .../codex-retained-root-serialization.test.ts | 46 ++++++++----------- 1 file changed, 20 insertions(+), 26 deletions(-) diff --git a/tests/codex-retained-root-serialization.test.ts b/tests/codex-retained-root-serialization.test.ts index f59b9874b..36c7fdb85 100644 --- a/tests/codex-retained-root-serialization.test.ts +++ b/tests/codex-retained-root-serialization.test.ts @@ -425,6 +425,19 @@ test("two processes at the post-approval management seam serialize instead of in const routeScript = (marker: string) => ` import { existsSync, writeFileSync } from "node:fs"; + // The stub lives on globalThis, NOT on the provider row. Catalog admission + // encodes the config to derive its identity and refuses a function member, so + // a per-provider \`fetch\` makes the seam throw before it can converge — which + // looked exactly like a production defect until the encoder said so. + globalThis.fetch = async () => { + writeFileSync(${JSON.stringify(barrier)} + "-" + ${JSON.stringify(marker)}, "here"); + const deadline = Date.now() + 8000; + while (Date.now() < deadline) { + if (existsSync(${JSON.stringify(barrier)} + "-a") && existsSync(${JSON.stringify(barrier)} + "-b")) break; + await Bun.sleep(5); + } + return Response.json({ data: [{ id: "seam-model-" + ${JSON.stringify(marker)} }] }); + }; const config = { port: 10100, defaultProvider: "together", @@ -434,17 +447,6 @@ test("two processes at the post-approval management seam serialize instead of in baseUrl: "https://api.together.xyz/v1", apiKey: "seam-key", models: ["fallback-model"], - fetch: async () => { - // Announce arrival, then wait for the sibling so both processes are - // past approval and inside the seam at the same time. - writeFileSync(${JSON.stringify(barrier)} + "-" + ${JSON.stringify(marker)}, "here"); - const deadline = Date.now() + 8000; - while (Date.now() < deadline) { - if (existsSync(${JSON.stringify(barrier)} + "-a") && existsSync(${JSON.stringify(barrier)} + "-b")) break; - await Bun.sleep(5); - } - return Response.json({ data: [{ id: "seam-model-" + ${JSON.stringify(marker)} }] }); - }, }, }, }; @@ -503,24 +505,16 @@ test("two processes at the post-approval management seam serialize instead of in // be vacuous — two config-lock losers prove nothing about catalog serialization. expect(results.some(r => r.exitCode === 0)).toBe(true); - // KNOWN DEFECT, asserted so it cannot be forgotten: no process commits here. - // - // Called directly, the bound factory returns `skipped/busy`. Called after - // `saveConfigPreservingClaudeCode` — which is exactly what every one of the - // sixteen routes does before approving the refresh — it returns - // `failed/disk/gather` instead, so the seam never reaches a commit on the real - // production path. Reproduced single-process, so it is not contention. - // - // This assertion is deliberately the CURRENT behaviour rather than the intended - // one: it documents the defect at the seam it lives in, and it will fail the - // moment the seam starts committing, which is when it must be rewritten to - // require `committed` and a moved catalog. Tracked for WP9 closure; the fix is - // not in this test's scope. + // At least one process must reach a real commit, or the race proves nothing: + // the adapter is total, so a seam that only ever failed would still answer 2xx + // with a typed disposition and satisfy every assertion above. const dispositions = results .filter(r => r.exitCode === 0) .map(r => (JSON.parse(r.stdout.trim()) as { catalogRefresh: { status: string } }).catalogRefresh.status); - expect(dispositions.every(status => status !== "committed")).toBe(true); - expect(readFileSync(catalogPath, "utf8")).toBe(seeded); + expect(dispositions).toContain("committed"); + + // A commit means the catalog really moved. + expect(readFileSync(catalogPath, "utf8")).not.toBe(seeded); // The surviving catalog is one process's complete output, never a blend of both. const finalBytes = readFileSync(catalogPath, "utf8"); From d3882ddf2ab9c4c289f32ac86654c4e7693068fa Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Wed, 5 Aug 2026 01:51:14 +0900 Subject: [PATCH 080/163] fix(codex): a rotated key could publish rows the old key fetched MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The verifier found the join still incomplete. Discovery policy joined the authority comparison last round, but credentials never did: the flight key's fingerprint carries endpoints and model lists with no `authMode`, key or headers, and the policy component does not carry them either. Two admissions differing ONLY in credential therefore produced the same key AND the same policy, so the second joined the first. Reproduced against the real routes, not in the abstract: route A starts discovery with the old key, `/api/providers/keys` activates and persists a new one, route B mutates and joins A's in-flight gather, A goes stale, and B reports `committed` while publishing the model the OLD key fetched. The catalog ends up holding one credential's rows under another's admission. Flight entries now carry an `authIdentity` alongside the policy identity, and a join requires both. It covers auth mode, the credential, the observed auth resolution, the final headers and the resolved URL — headers because a static header can carry authority exactly as an `apiKey` can. It is hashed under the same unexported per-process key as every other component, since this value reaches a map key and must not disclose a token; `privacy:scan` stays green. Dropping the `authIdentity` term from the comparison collapses the two fetches into one and turns the new regression red, while the three existing authority tests stay green — they vary policy, not credentials, which is exactly why none of them caught this. --- src/codex/catalog/provider-fetch.ts | 27 ++++++++++++++ tests/codex-gather-authority.test.ts | 53 ++++++++++++++++++++++++++++ 2 files changed, 80 insertions(+) diff --git a/src/codex/catalog/provider-fetch.ts b/src/codex/catalog/provider-fetch.ts index fb7f8d406..f797c30fe 100644 --- a/src/codex/catalog/provider-fetch.ts +++ b/src/codex/catalog/provider-fetch.ts @@ -136,6 +136,7 @@ interface CapturedProviderGather { interface GatherFlightCapture { readonly discoveryPolicyIdentity: string; + readonly authIdentity: string; readonly discoveryPolicySnapshots: readonly CatalogProviderDiscoveryPolicySnapshot[]; readonly providers: readonly CapturedProviderGather[]; readonly authResolver: ModelsAuthResolver; @@ -145,6 +146,17 @@ interface GatherFlightCapture { interface GatherInflightEntry { readonly discoveryPolicyIdentity: string; + /** + * The credential half of the join decision. + * + * `gatherFlightKey`'s fingerprint carries endpoints and model lists but no + * `authMode`, key or headers, and discovery policy does not carry them either. + * Two admissions differing ONLY in credential therefore produced the same key + * and the same policy, so the second joined the first and published rows the + * old key had fetched — reproduced against the real routes by rotating a key + * through `/api/providers/keys` mid-flight. + */ + readonly authIdentity: string; readonly promise: Promise; } @@ -382,6 +394,19 @@ function captureGatherFlight( const discoveryPolicySnapshots = Object.freeze(providers.map(provider => provider.policy)); return Object.freeze({ discoveryPolicyIdentity: keyedGatherIdentity("catalog-discovery-policy-v1", discoveryPolicySnapshots), + // Credentials are hashed under the same unexported per-process key, never + // stored or compared in the clear: this value can reach a map key and must + // not disclose a token. The final headers are included because a static + // header can carry authority just as an `apiKey` can. + authIdentity: keyedGatherIdentity("catalog-gather-auth-v1", providers.map(provider => ({ + name: provider.name, + authMode: provider.provider.authMode ?? null, + liveModels: provider.provider.liveModels ?? null, + credential: provider.provider.apiKey ?? null, + observedAuth: provider.observedAuth ?? null, + headers: provider.request.headersWithCredential, + url: provider.request.url, + }))), discoveryPolicySnapshots, providers: Object.freeze(providers), authResolver, @@ -1064,6 +1089,7 @@ async function gatherRoutedModelsWithAuth( const bucket = gatherInflight.get(key) ?? []; let entry = bucket.find(candidate => ( candidate.discoveryPolicyIdentity === capture.discoveryPolicyIdentity + && candidate.authIdentity === capture.authIdentity )); if (!entry) { const lease = gatherGate.tryAcquire(); @@ -1080,6 +1106,7 @@ async function gatherRoutedModelsWithAuth( }); ownedEntry = Object.freeze({ discoveryPolicyIdentity: capture.discoveryPolicyIdentity, + authIdentity: capture.authIdentity, promise: flight, }); bucket.push(ownedEntry); diff --git a/tests/codex-gather-authority.test.ts b/tests/codex-gather-authority.test.ts index 4361343ac..dbe1e5f2c 100644 --- a/tests/codex-gather-authority.test.ts +++ b/tests/codex-gather-authority.test.ts @@ -191,4 +191,57 @@ describe("catalog gather discovery-policy authority", () => { log.mockRestore(); } }); + + /** + * Two admissions that differ ONLY in credential must not share a flight. + * + * The flight key's fingerprint carries endpoints and model lists but no + * `authMode`, key or headers, and discovery policy does not carry them either. + * So a key rotated through `/api/providers/keys` mid-flight left the second + * admission joining the first, receiving rows the OLD key had fetched, and + * reporting `committed` — the catalog ended up holding the old key's models + * under the new key's admission. + * + * Removing `authIdentity` from the join comparison collapses the two fetches + * back into one and turns this red. + */ + test("a rotated credential cannot join the flight it did not authorize", async () => { + clearModelCache("together"); + clearGatherRoutedModelsInflight(); + + const firstResponse = deferred(); + const seenKeys: string[] = []; + let fetchCount = 0; + globalThis.fetch = (async (input: unknown, init?: RequestInit) => { + fetchCount += 1; + const headers = new Headers((init?.headers ?? {}) as HeadersInit); + seenKeys.push(headers.get("authorization") ?? headers.get("x-api-key") ?? "none"); + if (fetchCount === 1) await firstResponse.promise; + return Response.json({ data: [{ id: `model-for-call-${fetchCount}` }] }); + }) as unknown as typeof fetch; + + try { + const oldKey = gatherRoutedModels(togetherConfig("old-key")); + // The flight claims its slot synchronously, but the request itself starts a + // few microtasks later; yield until it is actually in flight. + await Bun.sleep(20); + expect(fetchCount).toBe(1); + + // The rotation: same provider, same endpoint, same discovery policy — only + // the credential moved. + const newKey = gatherRoutedModels(togetherConfig("new-key")); + await Bun.sleep(20); + expect(fetchCount).toBe(2); + + firstResponse.resolve(); + await Promise.all([oldKey, newKey]); + + // Each admission fetched under its own credential; neither borrowed the other's. + expect(seenKeys.some(value => value.includes("old-key"))).toBe(true); + expect(seenKeys.some(value => value.includes("new-key"))).toBe(true); + } finally { + clearGatherRoutedModelsInflight(); + clearModelCache("together"); + } + }); }); From 538f8b7f2ed8f4fd6a14941cf92c958d420b6f42 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Wed, 5 Aug 2026 02:09:20 +0900 Subject: [PATCH 081/163] fix(codex): stop enumerating what may share a flight, and hash the whole graph MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Credentials were the second field to leak a flight, not the last. The verifier came back with `reasoningEfforts`: same provider, same endpoint, same key, same discovery policy, so route B joined route A's gather and committed A's rows. The shape of the bug is the fingerprint itself. `providerCatalogFingerprint` is an allow-list, so every field it does not name is treated as equivalence, and each fix so far has been another name added to the list. That cannot converge — the next field added to a provider row inherits the defect, and nothing fails until somebody reproduces it. It is this unit's oldest mistake in its most durable form: an absence read as a guarantee. So the join now compares `providerGraphIdentity`, a keyed hash of the enriched provider rows the flight will actually gather from. Anything that can change a catalog row is in there by construction rather than by memory. `fetch` is dropped before hashing, and only `fetch`: it is a caller-owned transport executor the outbound path honors so a caller can supply its own HTTP, which makes it the one member of a provider row that is legitimately a function. Everything else that cannot be encoded still raises, because a second function member appearing silently would restore exactly the hole this closes. The regression uses `reasoningEfforts` since that is what was reproduced against real routes, but it tests the general rule: removing `providerGraphIdentity` from the comparison turns it red while the four existing authority tests stay green. --- src/codex/catalog/provider-fetch.ts | 39 +++++++++++++++++++++ tests/codex-gather-authority.test.ts | 51 ++++++++++++++++++++++++++++ 2 files changed, 90 insertions(+) diff --git a/src/codex/catalog/provider-fetch.ts b/src/codex/catalog/provider-fetch.ts index f797c30fe..022419527 100644 --- a/src/codex/catalog/provider-fetch.ts +++ b/src/codex/catalog/provider-fetch.ts @@ -137,6 +137,7 @@ interface CapturedProviderGather { interface GatherFlightCapture { readonly discoveryPolicyIdentity: string; readonly authIdentity: string; + readonly providerGraphIdentity: string; readonly discoveryPolicySnapshots: readonly CatalogProviderDiscoveryPolicySnapshot[]; readonly providers: readonly CapturedProviderGather[]; readonly authResolver: ModelsAuthResolver; @@ -157,6 +158,19 @@ interface GatherInflightEntry { * through `/api/providers/keys` mid-flight. */ readonly authIdentity: string; + /** + * The whole admitted provider graph, not a chosen subset. + * + * `providerCatalogFingerprint` is an ALLOW-LIST, so every field it forgot was + * silently treated as equivalence: credentials leaked a flight until + * `authIdentity` landed, and `reasoningEfforts` leaked one after that — both + * reproduced against real routes. Enumerating fields cannot converge, because + * the next field added to a provider row inherits the same defect. This + * identity therefore covers the enriched, frozen provider objects the flight + * actually gathered from, so a join is refused unless the admissions agree on + * everything rather than on everything somebody remembered to list. + */ + readonly providerGraphIdentity: string; readonly promise: Promise; } @@ -407,6 +421,17 @@ function captureGatherFlight( headers: provider.request.headersWithCredential, url: provider.request.url, }))), + // Every enriched provider row the flight will gather from, in admission order. + // Anything that can change a catalog row lives in here by construction. + providerGraphIdentity: keyedGatherIdentity("catalog-gather-provider-graph-v1", + providers.map(provider => ({ + name: provider.name, + // `fetch` is a caller-owned transport executor, not admitted state: the + // outbound transport honors it so a caller can supply its own HTTP path. + // It is the one member of a provider row that is legitimately a function, + // so it is dropped here rather than allowed to break every encode. + provider: omitProviderTransportExecutor(provider.provider), + }))), discoveryPolicySnapshots, providers: Object.freeze(providers), authResolver, @@ -416,6 +441,18 @@ function captureGatherFlight( }); } +/** + * Drop the caller-owned transport executor before hashing a provider row. + * + * Fails closed on anything ELSE that cannot be encoded: the point of hashing the + * whole row is that no field escapes the comparison, so a second function member + * must surface as an encode error rather than being quietly skipped here. + */ +function omitProviderTransportExecutor(provider: OcxProviderConfig): Record { + const entries = Object.entries(provider).filter(([key]) => key !== "fetch"); + return Object.fromEntries(entries); +} + function materializeCapturedHeaders( request: CapturedModelsRequest, apiKey: string | undefined, @@ -1090,6 +1127,7 @@ async function gatherRoutedModelsWithAuth( let entry = bucket.find(candidate => ( candidate.discoveryPolicyIdentity === capture.discoveryPolicyIdentity && candidate.authIdentity === capture.authIdentity + && candidate.providerGraphIdentity === capture.providerGraphIdentity )); if (!entry) { const lease = gatherGate.tryAcquire(); @@ -1107,6 +1145,7 @@ async function gatherRoutedModelsWithAuth( ownedEntry = Object.freeze({ discoveryPolicyIdentity: capture.discoveryPolicyIdentity, authIdentity: capture.authIdentity, + providerGraphIdentity: capture.providerGraphIdentity, promise: flight, }); bucket.push(ownedEntry); diff --git a/tests/codex-gather-authority.test.ts b/tests/codex-gather-authority.test.ts index dbe1e5f2c..124244f16 100644 --- a/tests/codex-gather-authority.test.ts +++ b/tests/codex-gather-authority.test.ts @@ -244,4 +244,55 @@ describe("catalog gather discovery-policy authority", () => { clearModelCache("together"); } }); + + /** + * The general form of the same defect, found after credentials were fixed. + * + * `providerCatalogFingerprint` is an allow-list, so every provider field it + * does not name was treated as equivalence. Credentials leaked a flight until + * `authIdentity` landed; `reasoningEfforts` leaked one after that, and it + * changes catalog rows. Enumerating fields cannot converge — the next field + * added to a provider row inherits the defect — so the join now compares the + * whole admitted provider graph. + * + * This test uses `reasoningEfforts` because that is what the verifier + * reproduced against real routes, but it is really a test of the general rule: + * dropping `providerGraphIdentity` from the comparison turns it red. + */ + test("a provider field outside the legacy fingerprint cannot join another admission's flight", async () => { + clearModelCache("together"); + clearGatherRoutedModelsInflight(); + + const firstResponse = deferred(); + let fetchCount = 0; + globalThis.fetch = (async () => { + fetchCount += 1; + if (fetchCount === 1) await firstResponse.promise; + return Response.json({ data: [{ id: `model-${fetchCount}` }] }); + }) as typeof fetch; + + const withEfforts = (efforts: readonly string[]): OcxConfig => { + const config = togetherConfig(); + (config.providers.together as Record).reasoningEfforts = [...efforts]; + return config; + }; + + try { + const first = gatherRoutedModels(withEfforts(["low"])); + await Bun.sleep(20); + expect(fetchCount).toBe(1); + + // Same provider, same endpoint, same credential, same discovery policy — + // only a field the legacy fingerprint never listed has moved. + const second = gatherRoutedModels(withEfforts(["low", "high"])); + await Bun.sleep(20); + expect(fetchCount).toBe(2); + + firstResponse.resolve(); + await Promise.all([first, second]); + } finally { + clearGatherRoutedModelsInflight(); + clearModelCache("together"); + } + }); }); From 57c273922bdbd63e1b140811e8a07968928849ba Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Wed, 5 Aug 2026 02:25:14 +0900 Subject: [PATCH 082/163] docs(codex): say why the credential term stays after the graph subsumed it Checking my own work rather than the tests': removing `authIdentity` from the join leaves the credential regression green, because `providerGraphIdentity` hashes the whole provider row and `apiKey` lives on it. Removing both terms fails two tests. So the credential term is redundant for the case it was added for, and a reader deleting it would see a green suite. It stays, and the comment now says why instead of implying it is load-bearing on its own. It covers what the graph structurally cannot: the RESOLVED auth and the final materialized headers are derived at capture time, not stored on the row, so an OAuth token that moves while the row stays byte-identical still separates two admissions. And it states the credential rule at the place a reader looks for it, rather than leaving it as an emergent property of hashing everything. --- src/codex/catalog/provider-fetch.ts | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/src/codex/catalog/provider-fetch.ts b/src/codex/catalog/provider-fetch.ts index 022419527..b4d7c74b4 100644 --- a/src/codex/catalog/provider-fetch.ts +++ b/src/codex/catalog/provider-fetch.ts @@ -156,6 +156,15 @@ interface GatherInflightEntry { * and the same policy, so the second joined the first and published rows the * old key had fetched — reproduced against the real routes by rotating a key * through `/api/providers/keys` mid-flight. + * + * Now REDUNDANT with `providerGraphIdentity`, which hashes the whole provider + * row and therefore covers `apiKey` too: removing this term alone leaves the + * credential regression green. It is kept deliberately, for two reasons. It + * covers what the graph cannot — the RESOLVED auth (`observedAuth`) and the + * final materialized headers, which are derived rather than stored, so an + * OAuth token that changes while the row is byte-identical still separates + * admissions. And it states the credential rule where a reader looks for it, + * instead of leaving it as an emergent property of hashing everything. */ readonly authIdentity: string; /** From 45f7bb7caf9c836b9d9a398c76ae4cb2f7461860 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Wed, 5 Aug 2026 03:09:28 +0900 Subject: [PATCH 083/163] docs(substrate): WP10 was waiting on a convergeCodex that does not exist MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The review of the history plan found it standing on a phase that never shipped. WP10 assumed WP9 handed over a full `convergeCodex`, but the tree exports catalog scope only and the management adapter rejects every non-catalog request (management-convergence.ts:96); full admission is still a type, and native apply and restore still run directly (inject.ts:482,765). So WP10 had no executable producer for the authority, direction, receipt or expectation it consumes — building them here crosses into WP11 and WP12, omitting them leaves a placeholder. So the phase is split at the honest line. Every current high-level history operation enters the Worker and the history lock now; the desired-state rewire of convergence stays in WP12. WP10 delivers isolation and cross-process serialization, which is what its name promised. Three more the review got right. The lock order was stated as "H and N are never held simultaneously", but the Worker reads the coordinator and writes its terminal row while holding H, and both take BEGIN IMMEDIATE — the real edge is H → N, and H had no database at all, only N and K having resolvers. The durable row could not express WHICH history operation was authorized, so a Worker would have had to trust a caller-supplied provider, losing `syncResumeHistory: false`, the legacy-vs-migrate distinction, and the separate recovery path. And `ocx init` calls injection directly (cli/init.ts:197), so removing history from inject.ts would have silently stopped initialization from migrating history. The ninth instance of this unit's oldest mistake also turned up, in the code this phase is about to move: `readBackup` maps malformed JSON, an unsupported shape, and a manifest belonging to a DIFFERENT state database all to an empty manifest, and the pending probe folds manifest failure into `backupEntries = 0`. A zero/zero post-probe therefore certifies "converged" from evidence that was never read. Only a genuinely missing manifest may mean zero. Two labels corrected against the code rather than carried forward: recovery is manifest-independent, not DB-only — `ejectRemainingOpencodexHistory` patches rollout files and returns a `files` count — and the operation is derived by the convergence path, with `history-job` persisting it mechanically rather than choosing it. --- .../005_contract.md | 471 ++++++-- .../020_history_isolation.md | 1032 ++++++++--------- 2 files changed, 865 insertions(+), 638 deletions(-) diff --git a/devlog/_plan/260804_codex_write_substrate/005_contract.md b/devlog/_plan/260804_codex_write_substrate/005_contract.md index 40f91c53e..1d7c396f8 100644 --- a/devlog/_plan/260804_codex_write_substrate/005_contract.md +++ b/devlog/_plan/260804_codex_write_substrate/005_contract.md @@ -84,6 +84,20 @@ without defining them, so `020` and `040` kept their own. Both live in `convergence-types.ts` and both phases import them: ```ts +/** The exact history mutation authorized by the durable coordinator row. */ +export type CodexHistoryOperation = + | "skip" + | "apply-opencodex" + | "migrate-openai" + | "restore-openai" + | "recover-legacy-openai"; + +/** Why this exact history job is authorized; the kind is persisted with its id. */ +export type CodexHistoryAuthority = + | { readonly kind: "admission-snapshot"; readonly id: string } + | { readonly kind: "wp10-compatibility"; readonly id: string } + | { readonly kind: "explicit-legacy-recovery"; readonly id: string }; + export interface CodexHistoryState { status: "converged" | "pending" | "running" | "blocked" | "unknown" | "not-evaluated"; /** @@ -99,12 +113,15 @@ export interface CodexHistoryState { | "shutdown-cancelled" | "worker-died" | "overtaken" + | "foreign-state-db" | "record-write-failed"; attempts: number; /** null means "no timer armed"; see 020 — it must never mean "never again". */ nextRetryAt: string | null; - /** The transition this state belongs to, so an overtaken job is detectable. */ + /** The durable history job this state belongs to, so an overtaken job is detectable. */ txId: string | null; + /** null only for the generation-zero unscheduled row or ephemeral not-evaluated projection. */ + operation: CodexHistoryOperation | null; /** null means the final probe could not produce a trustworthy row count. */ pendingRows: number | null; /** null means the final probe could not produce a trustworthy manifest count. */ @@ -113,6 +130,24 @@ export interface CodexHistoryState { readonly [extra: string]: unknown; } +/** + * A manifest read is evidence, not an optional convenience. Only `missing` + * certifies zero entries. Every failed read keeps its failure class and a null + * count so absence can never be manufactured from unreadable bytes. + */ +export type PositiveHistoryManifestEntryCount = number & { + readonly __positiveHistoryManifestEntryCount: "validated-positive-integer"; +}; + +export type CodexHistoryManifestRead = + | { readonly kind: "missing"; readonly manifest: null; readonly backupEntries: 0 } + | { readonly kind: "ready"; readonly manifest: T; + readonly backupEntries: PositiveHistoryManifestEntryCount } + | { readonly kind: "unreadable"; readonly manifest: null; readonly backupEntries: null } + | { readonly kind: "malformed"; readonly manifest: null; readonly backupEntries: null } + | { readonly kind: "unsupported"; readonly manifest: null; readonly backupEntries: null } + | { readonly kind: "foreign-state-db"; readonly manifest: null; readonly backupEntries: null }; + /** * Every mutable Codex artifact for which the provenance ledger can authorize a * restore. Embedded config fragments share the `config` entry because they are @@ -219,16 +254,23 @@ already derived from the real config/profile/catalog/cache/journal/history surfa (`devlog/_plan/260804_codex_write_substrate/004_ownership_and_convergence.md:235-273`); its nested observations keep the one-artifact partial cases testable instead of hiding them behind that aggregate. -For a clean history convergence, `reason` is absent and both probe counts are zero. +For a clean non-`skip` history convergence, `reason` is absent and both probe counts +are zero. `skip` is the sole converged exception: it deliberately performs no probe +and stores null counts rather than manufacturing zero evidence. An unreadable DB/manifest uses `unreadable`; a readable but unsupported table or manifest shape uses `schema`; a watchdog uses `timeout`; graceful drain uses `shutdown-cancelled`; and failure of the terminal CAS uses `record-write-failed` in the returned observation while leaving the previously persisted `pending` schedule intact. Any failed/unavailable final probe stores null, never a zero-looking count. -`not-evaluated` is an ephemeral projection used only by WP9's catalog-scoped -compatibility outcome; it is never persisted as durable history and never answers -`isApplied` or `converged` with a false-looking boolean. +An otherwise valid manifest naming another canonical state DB uses +`foreign-state-db`; it is preserved and blocks convergence rather than becoming an +empty manifest for this DB. +`not-evaluated` is an ephemeral artifact projection used by WP9's catalog-scoped +compatibility outcome and, for the three nested history artifacts only, by an +authorized `skip`. Catalog scope persists no history state; `skip` instead persists +`CodexHistoryState { status:"converged", operation:"skip" }` and makes no artifact +claim. Neither case answers an unobserved artifact with a false-looking boolean. ### Durable state: the JSON CAS was wrong @@ -257,11 +299,19 @@ export interface CodexTransitionState extends CodexTransitionVersion { /** Durable schedule and latest terminal observation for this exact pair. */ readonly history: CodexHistoryState; readonly historySchedule: null | Readonly<{ - direction: "apply" | "remove"; - authoritySnapshotId: string; + jobId: string; + operation: CodexHistoryOperation; + authority: CodexHistoryAuthority; }>; } +/** Both the native pair and exact durable history authorization are CAS input. */ +export interface CodexHistoryScheduleExpectation extends CodexTransitionVersion { + readonly historyJobId: string; + readonly operation: CodexHistoryOperation; + readonly authority: CodexHistoryAuthority; +} + export type IntegrationRecordRead = | { kind: "missing"; record: null } | { kind: "ready"; record: CodexIntegrationRecord } @@ -299,14 +349,42 @@ export type BeginCodexTransition = ( next: Readonly<{ txId: string; direction: "apply" | "remove"; + operation: Exclude; authoritySnapshotId: string; nextRetryAt: string; }>, ) => TransitionStateUpdate; -/** Change only history columns when the exact native pair still owns the row. */ +/** + * Authorize the explicit legacy recovery without inventing a native routing + * generation. The expected state must be terminal; unresolved native work wins. + */ +export type AuthorizeCodexLegacyHistoryRecovery = ( + expected: CodexTransitionState, + next: Readonly<{ + jobId: string; + authorityId: string; + nextRetryAt: string; + }>, +) => TransitionStateUpdate; + +/** + * WP10-only bridge for current roots that predate WP12 native admission. It + * publishes history work without moving the native routing pair. + */ +export type AuthorizeCodexCompatibilityHistory = ( + expected: CodexTransitionState, + next: Readonly<{ + jobId: string; + operation: Exclude; + authorityId: string; + nextRetryAt: string; + }>, +) => TransitionStateUpdate; + +/** Change only history columns when the exact pair, job and operation still own the row. */ export type UpdateCodexHistoryTransition = ( - expected: CodexTransitionVersion, + expected: CodexHistoryScheduleExpectation, history: CodexHistoryState, ) => TransitionStateUpdate; ``` @@ -315,6 +393,7 @@ WP8b implements and exports `const readIntegrationRecord: ReadIntegrationRecord` and `const updateIntegrationRecord: UpdateIntegrationRecord` from `src/codex/integration-record.ts`, plus `readCodexTransitionState`, `beginCodexTransition`, and +`authorizeCodexCompatibilityHistory`, `authorizeCodexLegacyHistoryRecovery`, and `updateCodexHistoryTransition` from `src/codex/transition-state.ts`; these are executable functions in that phase, not ambient declarations. @@ -341,8 +420,12 @@ CREATE TABLE codex_transition_state ( history_attempts INTEGER NOT NULL CHECK (history_attempts >= 0), history_next_retry_at TEXT, history_tx_id TEXT, - history_direction TEXT CHECK (history_direction IN ('apply', 'remove')), - history_authority_snapshot_id TEXT, + history_operation TEXT CHECK (history_operation IN + ('skip', 'apply-opencodex', 'migrate-openai', 'restore-openai', + 'recover-legacy-openai')), + history_authority_kind TEXT CHECK (history_authority_kind IN + ('admission-snapshot', 'wp10-compatibility', 'explicit-legacy-recovery')), + history_authority_id TEXT, history_pending_rows INTEGER, history_backup_entries INTEGER, updated_at TEXT NOT NULL, @@ -350,19 +433,32 @@ CREATE TABLE codex_transition_state ( ('converged', 'pending', 'running', 'blocked', 'unknown')), CHECK (history_reason IS NULL OR history_reason IN ('db-busy', 'permission', 'unreadable', 'schema', 'timeout', - 'shutdown-cancelled', 'worker-died', 'overtaken', 'record-write-failed')), + 'shutdown-cancelled', 'worker-died', 'overtaken', 'foreign-state-db', + 'record-write-failed')), CHECK (history_pending_rows IS NULL OR history_pending_rows >= 0), CHECK (history_backup_entries IS NULL OR history_backup_entries >= 0), CHECK ((native_generation = 0 AND current_tx_id IS NULL) OR (native_generation > 0 AND length(trim(current_tx_id)) > 0)), - CHECK ((native_generation = 0 - AND history_tx_id IS NULL - AND history_direction IS NULL - AND history_authority_snapshot_id IS NULL) - OR (native_generation > 0 - AND history_tx_id = current_tx_id - AND length(trim(history_authority_snapshot_id)) > 0)), - CHECK (native_generation > 0 OR + CHECK ((history_tx_id IS NULL + AND history_operation IS NULL + AND history_authority_kind IS NULL + AND history_authority_id IS NULL) + OR (length(trim(history_tx_id)) > 0 + AND history_operation IS NOT NULL + AND history_authority_kind IS NOT NULL + AND length(trim(history_authority_id)) > 0)), + CHECK (history_operation IS NULL + OR (history_operation = 'recover-legacy-openai' + AND history_authority_kind = 'explicit-legacy-recovery') + OR (history_operation != 'recover-legacy-openai' + AND history_authority_kind IN ('admission-snapshot', 'wp10-compatibility'))), + CHECK (history_authority_kind IS NULL + OR history_authority_kind IN ('wp10-compatibility', 'explicit-legacy-recovery') + OR (history_authority_kind = 'admission-snapshot' + AND native_generation > 0 + AND history_tx_id = current_tx_id)), + CHECK (native_generation = 0 OR history_operation IS NOT NULL), + CHECK (history_operation IS NOT NULL OR (history_status = 'unknown' AND history_reason IS NULL AND history_attempts = 0 @@ -372,8 +468,8 @@ CREATE TABLE codex_transition_state ( ); ``` -The observation columns project to `CodexHistoryState`; direction and authority -snapshot are schedule metadata required to restart the exact Worker after process +The observation columns project to `CodexHistoryState`; job id, operation and typed +authority kind/id are schedule metadata required to restart the exact Worker after process death. `not-evaluated` remains ephemeral and is rejected by the table. A native transition publishes its winner and schedule atomically with this null-safe conditional update (SQLite `IS` is required for the initial null txId): @@ -383,7 +479,8 @@ UPDATE codex_transition_state SET native_generation = ?, current_tx_id = ?, history_status = 'pending', history_reason = NULL, history_attempts = 0, history_next_retry_at = ?, history_tx_id = ?, - history_direction = ?, history_authority_snapshot_id = ?, + history_operation = ?, history_authority_kind = 'admission-snapshot', + history_authority_id = ?, history_pending_rows = NULL, history_backup_entries = NULL, updated_at = ? WHERE singleton = 1 @@ -394,9 +491,64 @@ UPDATE codex_transition_state The first two bound values are `{nativeAfter,newTxId}`; the last two are the expected `{nativeBefore,currentTxId}`. Worker claim/retry/terminal updates use the same `WHERE native_generation = ? AND current_tx_id IS ?` predicate and additionally -require `history_tx_id IS ?`; they change only `history_*`, never the native pair. +require `history_tx_id IS ? AND history_operation IS ? AND +history_authority_kind IS ? AND history_authority_id IS ?`; they change only +`history_*`, never the native pair. The operation is therefore CAS authority, not a +hint carried only in Worker IPC. The row count, not a later JSON read, is the CAS result. +The operation set is intentionally semantic rather than a provider direction: + +- `skip` is derived when admitted apply intent has `syncResumeHistory:false`; it + performs no manifest, rollout, or history-DB read/write and may terminally record + `converged` with both counts null because it makes no zero/zero claim. +- `apply-opencodex` is legacy-mode apply: retain originals in the manifest, patch + rollouts, and tag eligible rows `opencodex`. +- `migrate-openai` is loopback-mode apply: consume a valid matching manifest through + restore and then eject remaining legacy `opencodex` rows to `openai`. +- `restore-openai` is native removal: consume the matching manifest through generic + restore and then eject remaining routed rows. It remains distinct from migration + even though both currently share restore mechanics because their authorization and + retry cause differ. +- `recover-legacy-openai` is the explicit `recover-history --legacy-openai` salvage: + eject eligible legacy rows and patch their rollouts without reading, consuming, + deleting, or replacing the manifest. This is not generic restore. + +`BeginCodexTransition` receives the native direction and the operation derived by +convergence and rejects an impossible pair (`remove` with anything except +`restore-openai`, or `apply` with `restore-openai`/`recover-legacy-openai`). External +callers still provide only `action`, scope, reason and mode. The Worker request +carries the durable job id and operation copied from the row; it has no +`targetProvider` or caller-chosen direction. It re-reads the row under H and rejects +IPC whose job/operation differs before any probe or mutation. + +WP10 must land before WP12, and the current apply/restore roots have neither a WP12 +`AdmissionSnapshot` nor a native routing pair published by convergence. Inventing a +snapshot id or bumping `nativeGeneration` would violate the settled phase boundary. +`authorizeCodexCompatibilityHistory` is the explicit bridge: after the existing +native root returns its semantic receipt, the convergence owner derives one of +`skip | apply-opencodex | migrate-openai | restore-openai`, opens N, and conditionally +replaces only a terminal history schedule with a fresh job id, +`authority.kind:"wp10-compatibility"`, and a fresh opaque authority id. The id is a +job authorization nonce, not a digest of config or credentials and is never logged. +The CAS leaves the native pair unchanged and matches the prior pair, job, operation, +authority kind/id, and terminal status. It refuses pending/running work. No CLI, +server, `inject.ts`, or low-level history writer calls this API directly. The WP10 +convergence adapter derives the operation and delegates the mechanical CAS to +`history-job.ts`; §8's middle inventory permits only that scheduling edge. WP12-final +graph reachability removes it once admission-snapshot scheduling owns native +convergence. + +Explicit legacy recovery cannot honestly advance `nativeGeneration`: §3 defines it +as a routing transition, while this command changes history only. Under N, +`authorizeCodexLegacyHistoryRecovery` conditionally replaces only a terminal history +schedule with a fresh job id and `recover-legacy-openai`, leaving the native pair +unchanged and persisting `authority.kind:"explicit-legacy-recovery"`. Its `WHERE` +includes the expected native pair, prior history job id, prior operation, and prior +authority kind/id. A pending/running schedule refuses as busy rather than discarding +native repair. This separate history job identity is why ordinary native schedules +reuse `currentTxId` but the explicit recovery job need not. + A zero-row native result means another transition won despite this caller's admission: do not write JSON or spawn its Worker; any native bytes already committed are unresolved and the current row's winner owns repair. Re-admit if the deadline @@ -410,7 +562,7 @@ Initialization first verifies the no-legacy/native-clean precondition while the native lock excludes another initializer, then uses one `BEGIN IMMEDIATE` transaction: create the table, then `INSERT OR IGNORE` singleton 1 as `{0,null}` with an `unknown` history observation, -zero attempts and no txId/direction/authority/timer/counts, and sets +zero attempts and no job/operation/authority/timer/counts, and sets `PRAGMA user_version = 1`. That initialization is legal only when the JSON has no legacy `nativeGeneration`, `currentTxId`, `generation`, or durable `history` member and native observation finds no unresolved routed residue. When @@ -466,7 +618,13 @@ export interface ConvergeRequest { */ scope: "catalog" | "full"; /** Why, for the record and for log attribution. */ - reason: "startup" | "ensure" | "api-sync" | "cli" | "management-mutation"; + reason: + | "startup" + | "ensure" + | "api-sync" + | "cli" + | "cli-recover-history" + | "management-mutation"; /** Automatic callers fail fast and defer; explicit ones may wait. See §5. */ mode: "automatic" | "explicit"; deadlineMs: number; @@ -481,6 +639,9 @@ export type ConvergeOutcome = | { kind: "catalog-only"; changed: boolean; observed: CodexObservedState; catalogRefresh: CatalogDisposition; history: CodexHistoryState } + | { kind: "history-recovered"; changed: boolean; + observed: CodexObservedState; nativeGeneration: number; + currentTxId: string | null; history: CodexHistoryState } | { kind: "converged"; direction: "applied" | "removed"; changed: boolean; observed: CodexObservedState; nativeGeneration: number; currentTxId: string; @@ -516,6 +677,13 @@ produced TS2391 in audit round 3. WP8b exports the type and lands no runtime placeholder. WP9 supplies the first `convergeCodex` implementation and assigns it to `ConvergeCodex` in the same commit that rewires catalog callers. +`ConvergeRequest`'s `cli-recover-history` reason identifies the explicit command +boundary, not a provider/direction supplied to the Worker. It is accepted only with +`action:"converge"`, `scope:"full"`, `mode:"explicit"`; convergence maps that +command to the one fixed `recover-legacy-openai` authorization. The history-only +outcome is `history-recovered`, not `converged {direction:"removed"}`, because the +native pair and routing bytes did not move. + WP9's `scope:"catalog"` implementation is **catalog-only**: it gathers, commits, and reports catalog/cache/backup disposition while preserving each route's primary 2xx/201. It does not inject config/profile, recover journals, or dispatch @@ -523,7 +691,7 @@ history before WP10-WP12 land those mechanisms. WP12 strengthens that same funne to full observed-state convergence; there is no second entry point. Its `catalog-only` outcome sets non-catalog observations to `not-evaluated` and uses an ephemeral history value with `status:"not-evaluated"`, zero attempts, null txId, -timer, and probe counts. It does not claim either full direction. +operation, timer, and probe counts. It does not claim either full direction. An unresolved surface also names its scheduler. `config` schedules a fresh pre-gather admission; `native`, `catalog`, `cache`, `journal`, and `provenance` @@ -585,16 +753,22 @@ Round 2 #6: the previous version said "two counters, both in the record" and then defined one. They are distinct because they answer different questions — did the user's configuration move, versus did somebody else write Codex's files. -The WP9 seam audit forced a narrower definition of that second question. The -implemented transition row requires every positive `native_generation` to carry -the same `history_tx_id` as `current_tx_id`, a non-null `history_direction`, and -a non-empty `history_authority_snapshot_id` -(`src/codex/transition-state.ts:74-83`). `beginCodexTransition` therefore always +The WP9 seam audit forced a narrower definition of that second question. At native +publication, the implemented transition row requires every positive +`native_generation` to carry an ordinary native schedule whose `history_tx_id` +equals `current_tx_id`, a non-null typed `history_operation`, and a non-empty +admission-snapshot history authority id +(`src/codex/transition-state.ts:74-83`, to be amended by this contract). +`beginCodexTransition` therefore always publishes a pending HISTORY SCHEDULE with the pair (`src/codex/transition-state.ts:314-344`), and `assertPublished` rejects a caller that did not publish one (`src/codex/transition-state.ts:420-428`). Advancing the pair for catalog bytes alone would invent history work that does not exist and cross WP10/WP12's boundary. +After that ordinary schedule reaches a terminal state, §1's explicit history-only +compatibility/recovery CAS may publish a different job while preserving the positive +native pair; that does not retroactively turn the history-only job into a routing +generation. So the native generation identifies a **NATIVE ROUTING transition**: `config.toml`, the generated profile, and the injection journal, exactly the artifacts whose @@ -756,10 +930,18 @@ native/coordinator transaction N, when present Thus catalog-only work takes `K -> C`; a retained writer that is already under N takes `N -> K`; and full convergence takes `N -> K -> C`. There is no `C -> K`, no -`K -> N`, and no catalog path acquires history H. Using the already-open N capability +`K -> N`, and no catalog path acquires history H. A history Worker separately takes +`H -> N` for its fail-fast claim read and terminal CAS; both N acquisitions have +`busy_timeout=0` and finish before H is released. There is no `N -> H`, `K -> H`, or +`C -> H`, and the history Worker never enters K or C. The combined graph is therefore +the DAG `H -> N -> K -> C`; adding H does not invalidate WP9's settled +`N -> K -> C` proof because it adds only a new source edge and no edge back to H. +Using the already-open N capability to publish/commit its row while K is held is not a new acquisition edge. Config-owned callbacks remain forbidden from calling either N or K. A graph test protects those -negative edges. This extends, rather than reverses, the settled N -> C discipline. +negative edges, including inverse-edge fixtures for `N -> H`, `K -> H`, and +`C -> H`, plus cross-domain Worker fixtures for `H -> K` and `H -> C`. This extends, +rather than reverses, the settled N -> C discipline. ### The expected transition @@ -1368,6 +1550,7 @@ and both payload fields. | `ConvergeOutcome` | Status | Body | |---|---|---| | `catalog-only` | 200 | `{ ok: true, changed, observed, catalogRefresh, history }` | +| `history-recovered` | 200 | `{ ok: true, changed, observed, nativeGeneration, currentTxId, history }` | | `converged` | 200 | `{ ok: true, changed, observed, catalogRefresh, history }` | | `skipped` (`already-converged`) | 200 | `{ ok: true, changed: false, observed, catalogRefresh, history }` | | `refused` | 409 | `{ ok: false, authority, message, observed }` | @@ -1401,8 +1584,23 @@ steps, so two processes corrupt each other through the files it never sees. **One cross-process history lock**, acquired inside the Worker, held across the entire unit — manifest, rollouts and the DB transaction together, including the -final post-probe. It is a sibling of the native lock, not nested, because the -native section must stay synchronous. +final post-probe. H is a distinct SQLite database, not an unnamed lock and not N's +database reused under another label. `resolveCodexHistorySerializationDatabasePath` +in §7 owns its final path from effective user identity, canonical `CODEX_HOME`, and +canonical state-DB identity. Every process consumes that path verbatim. Reusing N +would make the Worker's required coordinator reads self-contend; allowing callers to +invent H's path would split exclusion so two processes could each believe they own +the same history unit. + +H and N are sibling databases but the lock order is **H -> N**, not “never held +simultaneously.” The live APIs prove the edge: the Worker calls +`readCodexTransitionState`, whose initialization path acquires `BEGIN IMMEDIATE` +(`src/codex/transition-state.ts:473-482`), for fail-fast admission and later calls +`updateCodexHistoryTransition`, which acquires another `BEGIN IMMEDIATE` +(`src/codex/transition-state.ts:521-533`), for terminal CAS while H remains held. +Both are fail-fast N acquisitions. The inverse edges `N -> H`, `K -> H`, and +`C -> H` are forbidden; §3 records why the resulting `H -> N -> K -> C` graph is +acyclic and keeps WP9's order valid. Two things round 2 caught: @@ -1411,6 +1609,41 @@ Two things round 2 caught: history caller takes this lock — server, CLI, startup, retry. A lock one caller can skip is not a lock. +**Direction is not operation.** The coordinator row persists the §1 +`CodexHistoryOperation` and its history job id, and every claim/retry/terminal CAS +matches both. Current behavior has five materially different authorizations: +`syncResumeHistory:false` means touch nothing (`src/codex/inject.ts:602-604`); +legacy apply backs up and tags `opencodex`; loopback apply migrates legacy rows to +`openai`; removal consumes the manifest and restores native state; and +`recover-history --legacy-openai` invokes only the unbacked legacy eject path +(`src/cli/index.ts:711-717`, `src/codex/history-provider.ts:701-710`). The Worker +derives its writer from the durable operation. A compatibility IPC field such as +`targetProvider` or direction, if temporarily retained during migration, is +validated against that operation and disagreement refuses before probe or write; +it never overrides the row. + +**Unreadable is not absent — the ninth recurrence of this unit's defect.** The +current `readBackup` turns malformed JSON, unsupported shape, and a manifest for a +different state DB into `{entries:{}}` (`src/codex/history-provider.ts:204-216`), +and the pending probe starts from zero and swallows the failure +(`src/codex/history-provider.ts:749-754`). That lets a zero/zero post-probe certify +convergence from evidence it never read: once again, absence was treated as a +guarantee. + +The sole manifest reader returns `CodexHistoryManifestRead`. Only a genuinely +missing path returns `kind:"missing"` and `backupEntries:0`. A present valid v1 +manifest for the canonical state DB returns `ready`; because empty manifests are +deleted by the writer, a present ready manifest must contain at least one validated +entry. Only the manifest validator constructs +`PositiveHistoryManifestEntryCount`, after proving an integer greater than zero; +`ready` with literal zero does not typecheck. Read/permission failure returns +`malformed`, readable unsupported version/shape returns `unsupported`, and a valid +manifest naming another canonical state DB returns `foreign-state-db`. Every failure +preserves the file byte-for-byte, carries `backupEntries:null`, prevents all history +mutation, and blocks convergence. `unreadable` maps to `unknown/unreadable`, +`malformed`/`unsupported` to `unknown/schema`, and a foreign manifest to +`blocked/foreign-state-db`; no path deletes, replaces, or consumes failed evidence. + **Sibling locks permit overtaking.** A releases the native lock after committing ON; B commits native OFF while A traverses history. Checking once after A acquires the history lock only moves the race: B can still advance the native pair before A @@ -1423,14 +1656,18 @@ the complete history unit. The guarantee is eventual convergence to the latest durable native transition: 1. The native coordinator CAS writes `{nativeAfter, txId}` and the complete - `history_status='pending'` schedule in the **same SQLite row update** before any + `history_status='pending'` schedule, including job id and operation, in the + **same SQLite row update** before any Worker spawn. If spawn never occurs or the Worker dies, the guardian/startup reader still has durable work to schedule. -2. A Worker checks that the coordinator row contains its `{nativeAfter, txId}` - immediately after acquiring the history lock. A mismatch returns +2. A Worker checks that the coordinator row contains its `{nativeAfter, txId}` plus + exact history job id, operation, and authority kind/id immediately after + acquiring H through the fail-fast `H -> N` read. A mismatch returns `pending/overtaken` without mutation and schedules observation of the current row. -3. Because a newer native transition can commit during traversal, the Worker uses - the §1 conditional SQLite update for its terminal history state. A zero-row CAS +3. Because a newer native transition or history-only recovery authorization can + commit during traversal, the Worker uses the §1 conditional SQLite update for its + terminal history state. Its `WHERE` matches native pair, history job id, and + operation. A zero-row CAS means its result is stale; it does not touch JSON, overwrite the newer pending schedule, or clear the winner's timer, and returns `overtaken`. 4. If an old Worker mutated history before detecting that final conflict, the newest @@ -1445,11 +1682,13 @@ spawn failure, Worker death, or process restart. Ordering, so absence of deadlock is checkable: the native callback performs its transition-row UPDATE in the native coordinator transaction, then releases it before -history dispatch. A Worker holds the history lock while traversing and attempts only -a fail-fast short coordinator CAS at claim/terminal boundaries; it never invokes the -native callback or waits on config coordination. `SQLITE_BUSY` leaves the current -pending row intact and retries after the history lock is released. Native and history -domain callbacks are never nested. +history dispatch and never calls H. A Worker holds H while traversing and attempts +only fail-fast short N transactions at claim/terminal boundaries; it never invokes +the native callback, K, or C. `SQLITE_BUSY` at N leaves the current pending row intact, +releases H, and retries from durable state. The graph tests require the real `H -> N` +edges and reject inverse `N -> H`, `K -> H`, and `C -> H` reachability through direct +imports, wrappers, aliases, re-exports, and dynamic imports; they also reject the +unneeded Worker edges `H -> K` and `H -> C`. ## 7. The lock namespace has one environment-independent root per effective user @@ -1513,16 +1752,36 @@ export type ResolveCodexCatalogSerializationDatabasePath = ( identity: UserIdentity, canonicalCodexHome: string, ) => string; + +/** + * Return H's FINAL database path for one canonical state database identity. + * Consumers append nothing and may not substitute N or K's path. + */ +export type ResolveCodexHistorySerializationDatabasePath = ( + identity: UserIdentity, + canonicalCodexHome: string, + canonicalStateDbPath: string, +) => string; ``` WP8b implements and exports constants of the identity and coordinator function types from `src/codex/user-identity.ts`; it does not ship declarations without bodies. WP9 -adds the catalog-serialization resolver there with K. These are the only two exported +adds the catalog-serialization resolver there with K, and WP10 adds the history +serialization resolver there with H. These are the only three exported final-path resolvers; the private secure-root resolver is shared but is never exported for consumer path composition. WP11, transition state, history, tests, cleanup, and K consume their returned database path verbatim. No consumer appends `opencodex`, a lock-directory name, `v1`, uid/SID, the home digest, or `.sqlite` a second time. +The canonical state-DB identity is path identity, not current inode identity. If the +DB exists, resolve it with `realpathSync.native`; if it is missing, resolve its +existing canonical parent and append the single basename without following a missing +leaf. Normalize Windows drive/UNC case exactly once. Reject a relative path, +non-directory/unsafe parent, symlink/reparse ambiguity, or a canonicalization failure. +Do not key H by dev/inode: SQLite replacement or recovery may change the inode while +the logical database still requires the same exclusion. Do not key it by the raw +request path: aliases would split the lock. + Exact platform algorithm: - **macOS and Linux:** obtain the effective uid from `getuid(2)` (Bun @@ -1550,6 +1809,14 @@ The final path returned by `resolveCodexCoordinatorDatabasePath` is the final path returned by `resolveCodexCatalogSerializationDatabasePath` is the distinct sibling `/catalog-write-locks/.sqlite`. +The final path returned by `resolveCodexHistorySerializationDatabasePath` is the +third sibling +`/history-write-locks/.sqlite`. +Length-prefixing is mandatory; string concatenation with a separator is not a tuple +encoding. Thus the same user/home/DB resolves one H across service and CLI processes, +while two state DBs under one home do not serialize each other and H can never equal N +or K. H uses `busy_timeout=0` and `BEGIN IMMEDIATE`; bounded waiting belongs to the +Worker's outer acquisition loop, not SQLite. POSIX directories are `0700` and files `0600`; Windows applies the required ACL to the root, databases, and rollback journals. Every existing component is checked before use and again through stable descriptors around SQLite open/transaction @@ -1559,10 +1826,11 @@ substituted path is a refusal, never something the resolver repairs in place. The test that matters, and the one my first version could not have failed: two child processes with different `HOME`, `USERPROFILE`, `TMPDIR`, `XDG_RUNTIME_DIR`, `TEMP`, `TMP`, and `LOCALAPPDATA` values but the same effective uid/SID and canonical -`CODEX_HOME` must resolve the same **two final database paths**, take the same N and K -locks respectively, and read/update the same singleton transition row. The two paths -must differ so nested `N -> K` cannot self-contend on SQLite's database-wide writer -slot. +`CODEX_HOME` and canonical state DB must resolve the same **three final database +paths**, take the same N, K, and H locks respectively, and read/update the same +singleton transition row. A second canonical state DB must retain the same N/K paths +but resolve a different H. All three paths must differ so nested `N -> K` and `H -> N` +cannot self-contend on SQLite's database-wide writer slot. ## 8. Names @@ -1572,6 +1840,7 @@ Audit #13. Fixed here so no phase invents a variant: |---|---| | the native write lock | `src/codex/codex-write-lock.ts` | | the catalog serialization primitive K | `src/codex/catalog-write-serialization.ts` | +| the history serialization primitive H | `src/codex/history-lock.ts` | | the record | `src/codex/integration-record.ts` | | the entry point | `src/codex/convergence.ts` | | generations | `src/codex/generation.ts` | @@ -1592,8 +1861,9 @@ already migrated lifecycle and explicit callers: | native config/profile | `src/codex/internal/native-writer.ts` | `src/codex/convergence.ts` only | | injection journal create/mark/restore/remove | `src/codex/internal/journal-writer.ts` | `src/codex/convergence.ts` only | | catalog, hashed/legacy backups, models cache | `src/codex/internal/catalog-writer.ts`, each mutation requiring K's runtime-validated live permit | `src/codex/convergence.ts` only | +| history serialization transaction H | `src/codex/history-lock.ts` | `src/codex/history-worker.ts` only | | history DB rows, manifest, rollout files | history write exports in `src/codex/internal/history-writer.ts` | `src/codex/history-worker.ts` only | -| transition pair and history schedule/terminal row | `src/codex/transition-state.ts` | `src/codex/convergence.ts` and `src/codex/history-worker.ts` only | +| transition pair and typed history schedule/terminal row | `src/codex/transition-state.ts` | `src/codex/convergence.ts` and `src/codex/history-worker.ts` only | | JSON provenance ledger | `updateIntegrationRecord` in `src/codex/integration-record.ts` | `src/codex/convergence.ts` only | | persisted OpenCodex config bytes and config generation | private writers in `src/config.ts` | exported `saveConfig`, `mutatePersistedConfig`, `saveConfigPreservingClaudeCode`, and the generation API in that same module only | @@ -1623,9 +1893,22 @@ after K acquisition; the other three synchronous retained chains acquire K befor their authoritative filesystem read and keep it through deterministic derivation and write. Each path supplies a runtime-live, same-home permit, after N if a later phase has already placed that root under N. A contract test cannot enforce both versions at -once, so the graph -fixture carries an explicit `"wp9-transitional" | "wp12-final"` inventory version: -WP9 expects exactly four legacy chains, and WP12 changes that expectation to zero. +once, and WP10 adds a real intermediate producer. The graph fixture therefore carries +an explicit +`"wp9-transitional" | "wp10-history-isolation" | "wp12-final"` inventory version: +WP9 expects exactly four catalog legacy chains; WP10 retains those four and adds only +these history scheduling edges: + +| WP10 transitional root | Permitted transition-state symbols | Required authority | +|---|---|---| +| `src/codex/history-job.ts` | `authorizeCodexCompatibilityHistory`, `authorizeCodexLegacyHistoryRecovery` | `wp10-compatibility` or `explicit-legacy-recovery`; never `admission-snapshot` | +| `src/codex/history-worker.ts` | `readCodexTransitionState`, `updateCodexHistoryTransition` | exact row-copied native pair, job, operation, and authority | + +`history-job.ts` receives the convergence-derived operation; it does not derive from +or accept a Worker `targetProvider`/direction. It may schedule and supervise but may +not import a manifest/rollout/history-DB writer. WP12 changes the four catalog roots +and `history-job.ts` authorizer root to zero; `history-worker.ts` remains the sole H +and history-writer root and retains only claim/terminal coordinator access. At the WP12-final transition, `inject.ts` is split: observation/parsing and pure config/profile transforms stay readable there; every export that calls @@ -1645,11 +1928,14 @@ mutator of a Codex-owned artifact must appear in the inventory. In the WP12-fina version, `history-job.ts`, management routes, CLI modules, `sync.ts`, `refresh.ts`, `inject.ts`, and `journal.ts` are not permitted roots; they call convergence, dispatch a Worker, or read only. That final prohibition must not be applied to the -four explicit WP9 transitional rows before WP12 owns their migration. -At both inventory versions, every catalog/backup/cache mutator must be reachable only +four explicit WP9 transitional rows and the narrow WP10 scheduling rows before WP12 +owns their migration. +At every inventory version, every catalog/backup/cache mutator must be reachable only with K's permit and must call K's runtime liveness/transaction/home assertion before its first filesystem mutation. Inverse-order graph fixtures reject `C -> K` and -`K -> N` even through wrappers, aliases, or re-exports. +`K -> N` even through wrappers, aliases, or re-exports. They also require the two +real Worker coordinator calls to establish `H -> N` and reject `N -> H`, `K -> H`, +`C -> H`, `H -> K`, and `H -> C` through the same indirections. ## 9. Baseline classes @@ -1682,13 +1968,23 @@ path, and observe one singleton row. Two native updates expecting `{0,null}` rac exactly one conditional UPDATE changes one row and the loser returns `conflict`. Pause an old Worker, publish a newer pair plus pending schedule, then finish the old Worker: its terminal UPDATE changes zero rows and cannot alter JSON or the winner's -schedule. Missing DB/table initializes only from native-clean/no-legacy state; +schedule. Change only the durable operation under the same native pair/job fixture +and require the stale terminal CAS to change zero rows. Table-drive every native +direction/operation pair and reject impossible combinations. Authorize explicit +`recover-legacy-openai` against a terminal row without advancing the native pair; +the same CAS must refuse a pending/running row and a stale prior job/operation. +Exercise the WP10 compatibility authorizer for each non-recovery operation against +generation zero and a positive unchanged native pair; it persists +`wp10-compatibility`, never advances the pair, and refuses stale prior authority or +non-terminal work. Its authority id must not equal or derive from config/credential +bytes and must not reach logs, JSON, responses, or exceptions. +Missing DB/table initializes only from native-clean/no-legacy state; legacy JSON pair/schedule, residue beside a missing row, malformed row, busy DB and unsafe path all fail closed with the specified typed outcome. `tests/codex-convergence-contract.test.ts`: every `ConvergeOutcome` variant maps to the §5 row, `busy` carries `Retry-After`, and a best-effort management caller -still returns 2xx while reporting a non-converged disposition. Concatenate all ten +still returns 2xx while reporting a non-converged disposition. Concatenate all TypeScript fences in document order, prepend the §1 `OcxConfig` import, and compile with the repository TypeScript compiler so WP8b cannot regress to TS2304 or a bodyless TS2391 declaration. Table-drive each artifact observation and require @@ -1701,6 +1997,15 @@ is released; conflict never invokes the callback. Instrument connection creation the guard cannot regress to `readConfigGenerationAtPath` and self-contend through a second SQLite handle. +Table-drive history-operation derivation through the real full-admission inputs: +apply plus `syncResumeHistory:false` -> `skip`; legacy apply -> +`apply-opencodex`; loopback apply -> `migrate-openai`; remove -> +`restore-openai`; and the explicit recovery command -> +`recover-legacy-openai` through its history-only authorization CAS. No route or CLI +passes `targetProvider` or direction to the Worker. A migration-compatibility +fixture that injects either field with a disagreeing value must refuse before the +manifest probe or any writer call. + Add the distinct round-3 two-process catalog barrier. Process A gathers catalog X, acquires K then C, completes generation/home/source/epoch/target validation, and pauses immediately before its first write. Process B invokes the **real retained** @@ -1832,12 +2137,17 @@ no-clobber publication wins; the loser receives `EEXIST`, validates and preserve the winner, and neither ordinary rename nor `atomicWriteFile` is called. Repeat with malformed, unreadable, routed, symlinked, and identity-ambiguous winners and require refusal without changing winner bytes. The graph inventory fixture runs as -`wp9-transitional` with exactly four legacy chains, then as `wp12-final` with zero; -a fifth WP9 root or a retained WP12 root fails. +`wp9-transitional` with exactly four catalog legacy chains, as +`wp10-history-isolation` with those four plus only the two typed history scheduling +rows, then as `wp12-final` with no legacy/compatibility authorizer root; a fifth WP9 +catalog root, an overbroad WP10 `history-job.ts` edge, or a retained WP12 root fails. `tests/codex-user-identity.test.ts`: real child processes vary every environment -home/runtime variable named in §7 and resolve the same two final database paths for -one effective uid or SID, with coordinator and catalog paths distinct. POSIX +home/runtime variable named in §7 and resolve the same three final database paths for +one effective uid or SID, canonical `CODEX_HOME`, and canonical state DB, with N, K, +and H paths pairwise distinct. A second state DB changes only H. Raw aliases that +canonicalize to the same state DB resolve the same H, and inode replacement at that +canonical path does not change H. POSIX activates wrong owner/mode/symlink and non-sticky `/tmp` refusal through a resolver seam; Windows CI activates token/SID failure, known-folder failure, reparse, owner, and broad-ACL refusal. No case falls back to an environment directory. @@ -1849,6 +2159,23 @@ Worker death, timeout, shutdown cancellation, unreadable/schema probes, and term record-write failure; every failed probe count is null and the latest transition remains durably schedulable. +Hold H in one process and prove a second service/CLI Worker for the same canonical +home/state DB cannot enter manifest, rollout, DB, probe, or terminal-CAS work. While +H is held, execute the real fail-fast claim read and terminal update and observe +`H -> N` without self-contention, proving H did not reuse N's path. The symbol graph +must retain those two required edges and fail independently for injected `N -> H`, +`K -> H`, `C -> H`, `H -> K`, and `H -> C` edges. + +For the ninth absence-as-guarantee regression, probe a genuinely missing manifest +and require `backupEntries:0`. Then seed, one case at a time, unreadable bytes, +malformed JSON, a readable unsupported version/shape, and a valid manifest naming a +different canonical state DB. Every present failure keeps its exact bytes, reports +`backupEntries:null`, performs no rollout/DB/manifest mutation, and cannot produce a +zero/zero converged state. A valid matching non-empty manifest remains readable and +consumable. A compile fixture assigning zero to a `ready` result must fail while the +`missing` zero shape compiles. Run the same matrix through no-op admission, final post-probe, guardian, +and doctor projection so no wrapper reintroduces zero. + **The funnel must be provable, not grepped** (round 2 #2). A grep guard misses a wrapper, re-export, alias or dynamic import. The writer-inventory test above is the enforcement surface; it permits the history Worker without opening native/catalog @@ -1858,7 +2185,9 @@ writes to it. - C14 — all 16 management callers funnel through `convergeCodex`, enforced by the symbol graph; its WP9 inventory permits exactly the four transitional chains and - its WP12-final inventory permits none. At both versions every first-party + its WP10 inventory adds only the typed `history-job.ts` compatibility/recovery + authorizers plus the Worker's claim/terminal access; its WP12-final inventory + permits no legacy or compatibility authorizer root. At every version every first-party catalog/backup/cache write requires a fresh permit from the same permanent K owner, and every low-level mutator rejects leaked, reused, forged, revoked, or wrong-home permits at runtime before filesystem mutation. @@ -1891,9 +2220,13 @@ writes to it. publication. An arbitrary filesystem, selector, or content A→B→A that completes wholly between two checks, and a non-cooperating write after the final comparison, are explicitly not claimed. -- Contributes to C15 with detect-and-repair: the latest native pair is durably - pending before spawn, a stale Worker cannot replace its transition row or the - winner's schedule, and the guardian - eventually repairs history. WP10 implements that protocol. Also contributes to +- Contributes to C15 with detect-and-repair: the latest native pair, history job id, + typed operation, and authority kind/id are durably pending before spawn; a stale + Worker cannot replace its transition row or the winner's schedule, and the guardian + eventually repairs history. H has one contract-owned final path per effective + user/canonical home/canonical state DB and takes only fail-fast `H -> N`; inverse + `N/K/C -> H` edges are forbidden. Only a missing manifest proves zero entries; + unreadable, malformed, unsupported, and foreign-state-DB manifests remain + preserved, nullable, and non-converged. WP10 implements that protocol. Also contributes to C2/C12 (generation-guarded catalog commit plus the phase-specific catalog/full admission and observation sequences). diff --git a/devlog/_plan/260804_codex_write_substrate/020_history_isolation.md b/devlog/_plan/260804_codex_write_substrate/020_history_isolation.md index 8e97d5ae8..ae691efb5 100644 --- a/devlog/_plan/260804_codex_write_substrate/020_history_isolation.md +++ b/devlog/_plan/260804_codex_write_substrate/020_history_isolation.md @@ -2,71 +2,104 @@ Research: `002_history_off_the_loop.md`. Shared contract: `005_contract.md`. -Today, native apply/restore performs synchronous SQLite, manifest, rollout, and -fsync work on the caller thread (`src/codex/inject.ts:602,764-794`, -`src/codex/history-provider.ts:565-699`). The manifest and rollout writes are -outside the provider's SQLite transaction: apply writes manifest and rollouts -before the DB transaction (`src/codex/history-provider.ts:606-648`); restore writes -rollouts, then DB, then manifest, then performs a second ejection -(`src/codex/history-provider.ts:656-695`). A SQLite busy timeout therefore explains -only one stall and serializes only one part of the state transition. - -The previous WP10 moved server work to a Worker but left explicit CLI work inline, -claimed there was no cross-process history lock, and owned a second -`integrations/codex.json` schema. Round 2 showed all three are one failure: an -opposite-direction process can overtake the Worker through the unguarded files, and -the CLI can skip the only exclusion path. Round 4 found the remaining failure: a -Worker can read pair N, a native transition can write N+1, and the Worker can then -replace JSON with stale N. Separate-file read/compare/replace is not a CAS when the -writers hold different locks. This rewrite consumes the contract's sibling history -lock and canonical-`CODEX_HOME` SQLite coordinator row (`005_contract.md` §§3, 6). - -WP10 is independently landable. WP8b already supplies the coordinator-row API, -shared types, generations, and user-identity resolver; WP9 supplies the working -`convergeCodex`. WP10 adds the real history lock and Worker implementation in the -same commit that routes every history caller through it. It does not wait for the -WP11 native lock or WP12 provenance implementation to typecheck or preserve -behavior. - -All current-code citations and diff context below were rechecked on 2026-08-04 at -`2d5e080dea3e7000bf2111b381c7c1a3c4f5fb11`. +Today every history mutation is synchronous on its caller. Apply opens +`state_5.sqlite`, writes the manifest and rollouts, then commits the DB transaction +(`src/codex/history-provider.ts:585-653`). Restore reads the manifest, writes +rollouts, commits the DB, removes the manifest, and then performs the residual +ejection (`src/codex/history-provider.ts:656-698`). SQLite serializes only the DB +portion; it does not serialize the manifest and rollout files around it. + +The path is also not one operation. `injectCodexConfig` deliberately does nothing +when `syncResumeHistory === false`, forward-tags history to `opencodex` only in +legacy mode, and otherwise migrates history back to `openai` +(`src/codex/inject.ts:602-604`). Native restore invokes the manifest-consuming +`openai` restore/eject path (`src/codex/inject.ts:765-800`), while +`ocx recover-history --legacy-openai` invokes the manifest-independent legacy ejection +(`src/cli/index.ts:711-724`). A Worker request containing only a caller-selected +provider cannot preserve those distinctions. + +This plan was previously based on a false landing premise. WP9 did **not** supply a +working full `convergeCodex`: the executable export is catalog-only +(`src/codex/convergence.ts:421-440`), management rejects every non-catalog request +(`src/codex/management-convergence.ts:89-106`), and full admission remains only the +`AdmissionSnapshot` type (`src/codex/convergence-types.ts:495-520`). Native apply and +restore still execute directly (`src/codex/inject.ts:482-654,765-800`). WP10 therefore +cannot consume a full authority snapshot, native receipt, or `CommitExpectation` +without implementing WP11/WP12 work. + +## Phase boundary — history isolation now, full convergence in WP12 + +WP10 takes the reviewer's second option. + +Every **current high-level history operation** enters `history-job` and H in this +phase. Existing apply, restore, explicit recovery, startup retry, CLI, service, and +management roots keep their current native/catalog orchestration, but none may call a +history DB/manifest/rollout writer inline. WP10 persists the contract-owned typed +`CodexHistoryOperation`, dispatches its identity to the Worker, and records its result. + +The desired-state-driven `convergeCodex({ scope: "full" })` caller rewire is deferred +to WP12. WP12 changes the producer of the durable history operation after full +admission and native coordination exist; it reuses the same `history-job`, Worker, H, +typed operation, writer split, and terminal-state protocol delivered here. WP10 does +not add a temporary convergence implementation, fake admission snapshot, synthetic +native receipt, or placeholder that a later phase replaces. + +This boundary is independently landable: WP10 typechecks while full convergence still +rejects non-catalog requests, and current user-visible high-level operations preserve +their distinct history semantics through the Worker. + +All current-code citations in this document were rechecked on 2026-08-05 at +`57c273922bdbd63e1b140811e8a07968928849ba`. ## IN / OUT IN: -- `src/codex/history-provider.ts` (MODIFY/SPLIT) — retain only read/probe behavior; - move every manifest, rollout, and history-DB writer into the Worker-only internal - writer module so graph reachability can distinguish a reader from a writer. -- `src/codex/internal/history-writer.ts` (NEW) — invocation-local retry policy, - classified internal failures, shared state-DB identity/path resolver, and the - exact manifest/rollout/DB mutation unit reachable only from `history-worker.ts`. -- `src/codex/history-worker.ts` (NEW) — Worker entry point; applies captured homes, - acquires the sibling cross-process history lock, rejects overtaken work, performs - the entire history unit, probes, records, and releases. -- `src/codex/history-job.ts` (NEW) — request validation, Worker IPC/watchdog/join, - history-lock target construction, capped retry scheduling, fresh coordinator-row - reads after conflict, and conversion of job facts to the contract's - `CodexHistoryState`. -- `src/codex/convergence.ts` (MODIFY) — add history execution behind the existing - `convergeCodex`; callers still use only the contract request/result. -- `src/codex/transition-state.ts` (MODIFY only through its public API) — WP10 calls - `readCodexTransitionState` and `updateCodexHistoryTransition`; the latter conditionally - updates the pending history schedule where the native pair and `history_tx_id` still - match. It never stores that pair or schedule in `integrations/codex.json`. -- `src/codex/inject.ts`, `src/codex/sync.ts` (MODIFY) — remove direct history - execution paths and return their current non-history receipts to convergence. -- `src/codex/history-migration-guardian.ts` (MODIFY) — schedule convergence from - durable state; never probe or mutate history on the listener thread. +- `src/codex/convergence-types.ts` (MODIFY) — materialize the + contract-owned `CodexHistoryOperation`, typed manifest-read result, and durable + history-operation schedule/result shapes from `005_contract.md`; do not define a + WP10-local provider/direction union. +- `src/codex/user-identity.ts` (MODIFY) — add + `resolveCodexHistorySerializationDatabasePath` beside the existing N and K + resolvers. It keys H by effective user, canonical `CODEX_HOME`, and canonical + state-DB path; consumers append no path segment. +- `src/codex/transition-state.ts` (MODIFY) — persist/read the typed history operation + and its operation identity, expose the history-specific schedule/claim/terminal CAS + used by current roots, and retain the operation for guardian restart. This is not a + producer of native generations or full authority snapshots. +- `src/codex/history-lock.ts` (NEW) — H: one cross-process, + canonical-`CODEX_HOME`/effective-user keyed SQLite exclusion primitive using the + contract resolver, finite acquisition, and no stale PID/mtime takeover. +- `src/codex/history-provider.ts` (MODIFY/SPLIT) — retain read/probe behavior and the + typed manifest parser; remove all DB/manifest/rollout writers and module-global + execution policy. +- `src/codex/internal/history-writer.ts` (NEW) — the exhaustive implementation of + every `CodexHistoryOperation`, including manifest-consuming restore and manifest-independent + legacy ejection. Only `history-worker.ts` may reach it. +- `src/codex/history-worker.ts` (NEW) — acquire H, resolve and validate the durable + operation through N, mutate all three surfaces, run the typed post-probe, publish the + terminal CAS, release H, and close. +- `src/codex/history-job.ts` (NEW) — resolve explicit paths/options, schedule the + typed operation durably as the sole root of the WP10 compatibility/explicit-recovery + authorizers, spawn/watch/join the Worker, classify IPC/death, and expose one async + entry point to every current high-level root. +- `src/codex/inject.ts`, `src/codex/sync.ts`, + `src/codex/history-migration-guardian.ts` (MODIFY) — preserve current native/catalog + behavior but replace inline history calls with `history-job`; the guardian reads and + retries the durable operation instead of calling the provider. +- `src/cli/init.ts` (MODIFY) — `ocx init` and its `setup` alias retain the history + migration currently inherited from `injectCodexConfig` + (`src/cli/init.ts:194-198`; `src/cli/index.ts:727-732`). - `src/cli/index.ts`, `src/cli/models.ts`, `src/cli/provider.ts`, `src/cli/v2.ts`, - `src/service.ts` (MODIFY where they currently trigger Codex native/history work) - — startup, explicit CLI, stop/uninstall, retry, and ensure use `convergeCodex`. + `src/service.ts` (MODIFY where async propagation is required) — await the existing + high-level operation through `history-job`; no command imports a history writer. - `src/server/management-api.ts`, `src/server/management/config-routes.ts`, - `src/server/lifecycle.ts` (MODIFY) — server work uses the same funnel and awaits - Worker termination during drain. -- `src/cli/doctor.ts` (MODIFY) — combine a live read-only probe with the contract - history section. + `src/server/lifecycle.ts` (MODIFY) — await the same high-level operation and join + live history Workers during drain. +- `src/cli/doctor.ts` (MODIFY) — combine durable history state with a typed live + read-only probe; unavailable evidence remains unknown. - `tests/codex-history-provider.test.ts`, + `tests/codex-transition-state.test.ts`, `tests/codex-convergence-contract.test.ts`, `tests/history-migration-guardian.test.ts`, `tests/codex-sync-api.test.ts`, and `tests/shutdown-drain.test.ts` (MODIFY), plus @@ -76,558 +109,419 @@ IN: OUT: -- Any `integrations/codex.json` path, version, parser, merge algorithm, generation, - transaction id, or pending-history schedule. Those transition facts live in the - canonical-`CODEX_HOME` coordinator row owned by `005_contract.md`; the former - `history-convergence.ts` schema owner is deleted from this plan. -- The claim that no cross-process history lock exists. WP10 owns its implementation - now because the history unit is not safe without it. -- The native lock and its namespace mechanics — WP11. The history lock is a sibling, - never a nested substitute (`005_contract.md` §6). -- `/api/sync` status, body, or header mapping — `toSyncResponse` owns that contract - (`005_contract.md` §5). -- Ownership/provenance/desired-state policy — WP12. A Worker receives an authority - snapshot identity; it does not invent authority. -- Traversal chunking, GUI, release/deploy operations, and the live proxy on 10100. - -## Worker boundary - -The Worker contains the whole mutable history unit: - -1. acquire the sibling cross-process history lock; -2. read the canonical-`CODEX_HOME` coordinator row and validate its pair against - `CommitExpectation` plus the authority snapshot identity; -3. optional no-op probe; -4. SQLite open, query, transaction, and close; -5. manifest read/write; -6. every rollout read, line-one patch, append, and fsync; -7. final post-probe; -8. conditionally update the coordinator row while still serialized, using both - generation fields in the `WHERE` clause; -9. release the history lock. - -Moving only `Database` calls is insufficient because the current manifest and -rollout mutations surround the DB transaction (`src/codex/history-provider.ts:606-648,656-695`). -Moving only automatic/server callers is insufficient because the explicit CLI path -currently reaches `restoreNativeCodex()` and `syncModelsToCodex()` directly -(`src/cli/index.ts:528,591,756,768,829`). A lock one caller can skip is not a lock. - -The server remains responsive because all synchronous/unbounded history work is in -the Worker. Explicit CLI also uses the Worker; its larger wait budget may block its -own terminal, but never the proxy listener and never bypasses cross-process -serialization. - -## Writer reachability has two permitted roots - -The failure is structural before it is behavioral: the contract's former rule that -only `convergence.ts` may reach low-level writers is impossible for an isolated -Worker. `history-worker.ts` must invoke the history mutation after the parent has -returned to the event loop. History is therefore its own permitted production root, -not an exception hidden behind an alias. - -The contract's graph/symbol guard permits exactly these history-root edges: +- A full `convergeCodex`, `AdmissionSnapshot` capture, desired-state observer, + provenance authorization, or the WP12 caller rewire. +- Native write exclusion, a native receipt, or a native `CommitExpectation` — WP11. +- Any claim that WP9 handed WP10 a working full funnel. It did not. +- Replacing current native/catalog orchestration. In particular, + `src/codex/inject.ts:775-780` is WP9's K-serialized catalog restore; it remains a + catalog operation and is not copied into the history Worker. +- `/api/sync` response-contract redesign; WP10 only preserves the current route while + moving its inherited history operation off-thread. +- GUI, traversal chunking, release/deploy work, or touching the live proxy on 10100. + +## Durable operation, not caller-selected provider + +The shared contract's `CodexHistoryOperation` is the only executable direction. It +must distinguish at least these existing semantics without flattening them: + +| Durable `CodexHistoryOperation` | Existing evidence | Worker behavior | +|---|---|---| +| `skip` | `syncResumeHistory === false` at `src/codex/inject.ts:602-604` | Enter the job/H validation path, perform no manifest/rollout/history-DB read or write, and record `converged` with null counts because no zero/zero claim is made. | +| `apply-opencodex` | legacy branch at `src/codex/inject.ts:602-604` | Manifest-backed forward-tag to `opencodex`. | +| `migrate-openai` | non-legacy branch at `src/codex/inject.ts:602-604` | Manifest-consuming restore followed by residual ejection to `openai`. | +| `restore-openai` | `src/codex/inject.ts:781-800` | Generic native removal: consume the matching manifest and eject residual routed rows, with the current no-op-probe policy retained. It stays distinct from migration because authorization/retry cause differs. | +| `recover-legacy-openai` | `src/cli/index.ts:711-724` | manifest-independent legacy ejection; it must not read, consume, delete, or replace the backup manifest. | + +The operation is DERIVED by the caller-facing convergence path from what it already +admitted, never chosen at the Worker boundary. `history-job` does not decide it: the +module mechanically persists the derived operation through the contract-owned history +scheduling API and receives an opaque durable job identity. A serialized Worker +message accordingly contains no `targetProvider`, no caller-chosen `direction`, and no +`CommitExpectation`. +The Worker request carries the job id, operation, and `CodexHistoryAuthority` copied +from the row, plus structured-clone-safe explicit paths and invocation-local execution +options. + +After acquiring H, the Worker reads N and obtains the durable schedule. It executes +only when native pair, job id, operation, and authority are current and well-formed. +The durable value is authoritative: the Worker validates the IPC copy against it and still +dispatches from the durable value. A missing, +superseded, malformed, or mismatched operation produces the typed non-success outcome +and performs no history write. A caller can therefore neither turn generic restore +into manifest-independent recovery nor bypass `syncResumeHistory: false` by choosing a provider. + +WP12 later schedules the same type from admitted desired state. That is a caller-root +change, not a history-mechanism replacement. + +### Contract bridge for the chosen WP10 boundary + +WP10 consumes `AuthorizeCodexCompatibilityHistory` for current `skip`, +`apply-opencodex`, `migrate-openai`, and `restore-openai` roots. It conditionally +publishes job id + typed operation + +`CodexHistoryAuthority { kind: "wp10-compatibility", id }` without advancing or +inventing a native routing generation and without pretending to possess WP12's +`AdmissionSnapshot`. `recover-legacy-openai` uses +`AuthorizeCodexLegacyHistoryRecovery` and +`{ kind: "explicit-legacy-recovery", id }`. WP12 alone uses +`BeginCodexTransition` with `{ kind: "admission-snapshot", id }`. + +`history-job.ts` is the sole WP10 compatibility adapter. The existing owning +apply/restore/recovery function derives the semantic operation from its internal +branch after its current native/catalog work; it passes that type to `history-job`, +not a user-controlled provider/direction. CLI, server, `inject.ts`, guardian, and +other helpers never import either transition-state authorizer directly. Compatibility +authority ids are fresh opaque nonces, never config/credential digests and never logs. +WP12 dispatches already-admitted schedules through the same job/Worker code and stops +using only the compatibility-authorization branch. + +The Worker claim/terminal CAS matches native pair, job id, operation, and complete +authority. A compatibility authority can never be relabeled as admission authority. +This is one contract row and one terminal protocol, not a WP10-private store. + +The graph guard consumes the contract's `wp10-history-isolation` inventory version: +`history-job.ts` may reach only the compatibility and explicit-recovery authorizers, +while `history-worker.ts` may reach only the transition reader/terminal updater plus H +and the history writer. The WP12-final version removes the job authorizer root when +full convergence becomes the producer; it retains the Worker root. + +## Worker boundary and explicit process state + +The Worker owns the whole mutable unit: + +1. apply captured `CODEX_HOME`/`OPENCODEX_HOME` before dynamically importing any + history module; +2. acquire H from the final path returned by the H resolver; +3. read N and resolve/validate the current durable `CodexHistoryOperation`; +4. perform the typed manifest read only for operations that consume/update the + manifest; `skip` and `recover-legacy-openai` do not read it; +5. open, query, transact, and close `state_5.sqlite` within the invocation; +6. perform every manifest write/delete and rollout line-one patch/append/fsync; +7. run the final typed DB + manifest post-probe while H remains held; +8. publish the operation-identity-conditioned terminal update through N; +9. release H and close the Worker in `finally`. + +Moving only `Database` calls is insufficient: apply writes manifest and rollouts before +its DB transaction (`src/codex/history-provider.ts:606-648`), and restore writes +rollouts, DB, manifest deletion, and residual ejection in sequence +(`src/codex/history-provider.ts:656-695`). Moving only server callers is also +insufficient because CLI/service/management paths currently share the same inline +helpers. + +The Worker is a separate process context. The current module binds +`STATE_DB_PATH`/`HISTORY_BACKUP_PATH` at module load +(`src/codex/history-provider.ts:16-22`) and stores busy timeout in mutable module state +(`src/codex/history-provider.ts:31-49`). WP10 replaces both with absolute request paths +resolved before spawn and invocation-local options applied to each DB open. This is +feasible without connection transfer: apply, restore, legacy recovery, and the probe +open and close their DB handles per invocation +(`src/codex/history-provider.ts:585-653,656-698,701-710,757-770`). + +The request parser rejects blank operation ids, non-absolute paths, path/identity +mismatch, non-finite or negative numeric options, unknown message variants, and test +checkpoints outside the injected test supervisor. `requestId` rejects stray IPC. +Every crossing value is structured-clone data; no `Database`, function, config object, +or lock capability crosses the boundary. + +## H database and the real lock order + +H is a sibling database, not N or K. `resolveCodexHistorySerializationDatabasePath` +returns the final database path for effective user plus canonical `CODEX_HOME` plus +canonical state-DB identity; callers append nothing. H uses +`busy_timeout=0` and `BEGIN IMMEDIATE` with bounded async outer acquisition. There is +no PID/mtime stale takeover; process/connection death releases SQLite exclusion. + +The previous plan's statement that H and N are never held simultaneously was false. +`readCodexTransitionState` opens a `BEGIN IMMEDIATE` initialization transaction +(`src/codex/transition-state.ts:473-489`), and +`updateCodexHistoryTransition` opens another `BEGIN IMMEDIATE` terminal transaction +(`src/codex/transition-state.ts:521-565`). The Worker invokes both while H protects the +history surfaces. The actual edge is therefore: ```text -history-worker.ts -> internal/history-writer.ts -history-worker.ts -> transition-state.ts (pair read + history schedule/terminal CAS only) -internal/history-writer.ts -> writeBackup - -> updateSessionMeta (line-one patch + append + fsync) - -> syncCodexHistoryProviderUnsafe - -> restoreCodexHistoryProvider - -> ejectRemainingOpencodexHistory +current high-level root: N(schedule typed operation) -> release N -> spawn Worker +history Worker: H -> short N(read/claim) -> release N + H -> mutate/probe + H -> short N(terminal CAS) -> release N -> release H ``` -No CLI, server, guardian, `inject.ts`, `sync.ts`, or compatibility wrapper may reach -those history writers. Tests may import the Worker entry/funnel, not the low-level -module. Today `history-provider.ts` is mixed: it exports read-only +The checkable order is **H → N**. `N → H` is forbidden: scheduling commits and releases +N before spawn/await. H never enters K or the config-generation lock, and K/config +paths never enter H. The future WP11 native callback never spawns or awaits a Worker +while holding N; WP12 dispatches only after releasing native coordination. A busy N +claim/terminal attempt leaves the durable operation pending, releases H, and retries +later; it never waits indefinitely while retaining H. + +The lock-order contract test contains allowed fixtures for `H -> N` claim/terminal +calls and forbidden fixtures for `N -> H`, `K -> H`, `H -> K`, and +config-lock-to-H edges. Reversing the schedule/spawn order to await the Worker while N +is live must turn that test red. + +## Typed manifest evidence — absence is not success + +The current `readBackup` collapses a missing file, malformed JSON, unsupported shape, +and a manifest for another state DB into an empty manifest +(`src/codex/history-provider.ts:204-217`). The pending probe then initializes +`backupEntries = 0` and suppresses manifest failures +(`src/codex/history-provider.ts:749-755`). Combined with a missing DB returning +`pendingRows = 0` (`src/codex/history-provider.ts:756`), unread evidence can certify a +false zero/zero convergence. + +WP10 consumes the contract-owned typed manifest read. The reader must preserve these +distinct states through mutation and post-probe: + +| Evidence | Classification | May certify zero entries? | +|---|---|---| +| manifest absent and DB readable | `missing`, `backupEntries: 0` | Yes, only after the DB probe also succeeds. | +| valid present v1 manifest for the requested DB | `ready` with at least one validated entry | Yes. | +| malformed JSON | `malformed`, null count | No. | +| readable unsupported version/shape | `unsupported`, null count | No. | +| unreadable/permission failure | unreadable | No. | +| manifest identifies another state DB | `foreign-state-db`, null count | No. | +| DB missing while a valid backup has entries | pending/blocked restore work | No. | + +Only a successful typed DB probe and successful typed manifest read may produce +numeric counts. `unreadable` maps to `unknown/unreadable`, malformed/unsupported maps +to `unknown/schema`, and a foreign manifest maps to +`blocked/foreign-state-db`, all with null manifest count. Generic restore never treats +a foreign-DB manifest as empty and never deletes it; manifest-independent legacy recovery does not +consume the manifest at all. + +Fixtures cover malformed JSON, unreadable file, unsupported shape, wrong DB identity, +and missing DB with a nonempty backup. Reintroducing `catch { return emptyManifest }` +or `catch { backupEntries = 0 }` must turn each named fixture red. + +## Writer reachability: one permitted production root + +`history-provider.ts` currently mixes read exports with mutators: read-only `readLatestSessionMeta`, `readThreadFieldsFromRollout`, and -`countPendingOpencodexHistory` beside the mutating `syncCodexHistoryProvider`, -`migrateHistoryToOpenai`, and `restoreLegacyOpenaiHistory` -(`src/codex/history-provider.ts:263,348,565,701,719,749`). A module-dependency graph -cannot tell that an importer selected only a reader. This phase must split the module: -the public provider becomes read/probe-only, while the mutating entry points and their -private manifest/rollout/DB helpers move to `internal/history-writer.ts`. Only then can -the reachability test prove the history root and the separate `convergence.ts` roots -from the contract inventory without allowing every reader import to write. - -## Serializable request and response - -The request carries the identity of every authority the Worker must revalidate. It -does not carry a mutable config object or a caller-chosen desired direction. - -```ts -import type { - CodexHistoryState, - CommitExpectation, - UserIdentity, -} from "./convergence-types"; - -export interface HistoryWorkerRequest { - type: "run"; - requestId: string; - targetProvider: "openai" | "opencodex"; - stateDbPath: string; - backupPath: string; - lockIdentity: { - userIdentity: UserIdentity; - stateDbId: string; - }; - expectation: CommitExpectation; - /** Digest/id of the AdmissionSnapshot that authorized this transition. */ - authoritySnapshotId: string; - busyTimeoutMs: number; - attempts: number; - delayMs: number; - skipWhenProvablyNoop: boolean; - /** Test supervisor only: pause after the named real mutation, then await resume. */ - pauseAfter?: HistoryMutationCheckpoint; - env: { CODEX_HOME?: string; OPENCODEX_HOME?: string }; -} - -export type HistoryMutationCheckpoint = - | "manifest-write" - | "first-rollout-write" - | "database-write"; - -export interface HistoryWorkerResume { - type: "resume"; - requestId: string; - after: HistoryMutationCheckpoint; -} - -export type HistoryWorkerMessage = HistoryWorkerRequest | HistoryWorkerResume; - -export type HistoryWorkerFailureReason = NonNullable; - -export interface HistoryProbeCounts { - pendingRows: number | null; - backupEntries: number | null; -} - -export type HistoryWorkerResponse = - | { - type: "checkpoint"; - requestId: string; - after: HistoryMutationCheckpoint; - } - | { - type: "done"; - requestId: string; - state: CodexHistoryState; - postProbe: HistoryProbeCounts; - expectation: CommitExpectation; - authoritySnapshotId: string; - } - | { - type: "error"; - requestId: string; - reason: HistoryWorkerFailureReason; - postProbe: HistoryProbeCounts; - }; -``` +`countPendingOpencodexHistory` coexist with `syncCodexHistoryProvider`, +`restoreLegacyOpenaiHistory`, and `migrateHistoryToOpenai` +(`src/codex/history-provider.ts:263-274,348-422,565-579,701-731,749-775`). Split it so +the production graph has one writer root: -Every crossing value is plain structured-clone data. The parent resolves absolute -paths and lock identity before spawn. The Worker applies captured homes before the -dynamic import because `history-provider.ts:16-22` currently binds path-derived -state at module load. The request guard rejects non-finite/negative numeric fields, -non-absolute paths, malformed identities, invalid expectations, and blank snapshot -ids. - -`requestId` rejects stray messages. `authoritySnapshotId` rejects a job admitted -for different service/external/journal/provenance/intent evidence. The -`CommitExpectation` rejects a transition overtaken after native commit. These are -not optional diagnostics; missing fields make the message invalid and no mutation -starts. `pauseAfter` and `resume` are accepted only from the injected test supervisor; -they are not exposed through CLI, HTTP, config, or environment input. A checkpoint is -non-terminal, so the parent keeps the watchdog and join active until `done`/`error`. - -## One sibling history lock +```text +history-worker.ts -> history-lock.ts (H) +history-worker.ts -> transition-state.ts (durable operation claim/terminal CAS) +history-worker.ts -> internal/history-writer.ts +internal/history-writer.ts -> manifest writes/deletes + -> rollout line-one patch + append + fsync + -> history DB transactions +``` -`src/codex/history-job.ts` constructs the history lock from the contract-owned -effective-user identity plus normalized state-DB identity. It uses a private, -persistent SQLite transaction with finite async acquisition and no PID/mtime stale -takeover. The Worker acquires it **inside the Worker** and holds it over manifest, -rollouts, DB, final probe, and terminal coordinator update. +No CLI, server, guardian, `inject.ts`, `sync.ts`, compatibility wrapper, barrel, +re-export, or dynamic import may reach `internal/history-writer.ts` or its mutating +symbols. Tests invoke the public job/Worker boundary; focused writer unit fixtures may +import the internal module only from the test allowlist. + +The guard is symbol-level, not regex counting. Build a TypeScript `Program` and +`TypeChecker`, resolve import aliases, re-exports, namespace access, and string-literal +dynamic imports, then compute reachability from every production entry symbol to the +writer symbols. The current route-count test merely counts the literal text +`await convergeCodexCatalog()` (`tests/codex-convergence-contract.test.ts:232-250`); +that approach cannot prove this boundary. + +The graph fixture includes negative variants that must fail: + +- wrapper calls writer, caller imports wrapper; +- barrel re-exports writer under another name; +- aliased named import calls writer; +- namespace import calls writer; +- string-literal dynamic import calls writer; +- new production module reaches writer without appearing in the caller table. + +## Current production caller inventory + +WP10 rewires the existing high-level operations, not a nonexistent full funnel. The +symbol-level test owns this table as data and proves each listed command/route reaches +`runCodexHistoryJob` when its current semantics request history, never a writer. + +| Production command/route | Current history-bearing chain | WP10 terminal edge | Named broken change that must fail | +|---|---|---|---| +| `ocx init`, `ocx setup` | command dispatch -> `runInit` -> `injectCodexConfig` (`src/cli/index.ts:727-732`; `src/cli/init.ts:194-198`) | typed apply operation -> job | Restore direct `injectCodexConfig` history mutation or remove the job await from init. | +| `ocx start` | `handleStart` -> `syncModelsToCodex`; starts guardian (`src/cli/index.ts:318-321`) | apply job + durable guardian retry | Startup stops arming the guardian, or inject runs inline. | +| `ocx ensure` | existing/live and spawned paths call `syncModelsToCodex` (`src/cli/index.ts:358-412`) | apply job | Either ensure branch bypasses/does not await the job. | +| `ocx sync` | command -> `syncModelsToCodex` (`src/cli/index.ts:827-842`) | apply job | Sync calls provider writer directly. | +| `ocx restore back`, `ocx eject back` | command -> `syncModelsToCodex` (`src/cli/index.ts:745-764`) | apply job | Back-switch returns before job dispatch. | +| `ocx restore`, `ocx eject` | command -> `restoreNativeCodex` (`src/cli/index.ts:765-790`) | generic restore job | Restore keeps the current inline `syncCodexHistoryProvider("openai")`. | +| `ocx stop` | `handleStop` -> `restoreNativeCodex` (`src/cli/index.ts:456-551`) | generic restore job | Stop reports completion without awaiting Worker join. | +| `ocx uninstall`, `ocx remove` | `handleUninstall` -> `restoreNativeCodex` (`src/cli/index.ts:554-593,795-798`) | generic restore job | Uninstall invokes synchronous restore wrapper. | +| `ocx restart` | `handleStop` then `handleEnsure` (`src/cli/index.ts:968-973`) | restore job then apply job | Restart overlaps the two jobs or skips either await. | +| hidden `__tray-start`, `__tray-restart` | tray start launches the ordinary start process; restart awaits `handleStop` then tray start (`src/cli/index.ts:415-453,944-954`) | startup apply job; restart restore then apply | Tray restart starts before restore joins, or direct start bypasses ordinary startup. | +| `ocx recover-history --legacy-openai` | `handleRecoverHistory` -> `restoreLegacyOpenaiHistory` (`src/cli/index.ts:711-724,792-794`) | manifest-independent legacy-eject job | Recovery maps to generic restore and consumes manifest. | +| `ocx provider ... --sync` | provider mutation -> `syncModelsToCodex` (`src/cli/provider.ts:232-238`) | apply job | Provider sync imports writer or drops job await. | +| `ocx models/model ...` live sync | model mutation -> `syncModelsToCodex` (`src/cli/models.ts:102-108`) | apply job | Model sync returns after catalog only. | +| `ocx v2 mode/on/off` | dynamic import of `syncModelsToCodex` (`src/cli/v2.ts:143-170,177-196`) | apply job | Dynamic-import alias bypasses the job; this exercises alias/dynamic reachability. | +| `POST /api/sync` | route -> `syncModelsToCodex` (`src/server/management/config-routes.ts:261-268`) | automatic apply job | Route responds while history remains inline or untracked. | +| `POST /api/stop` | route -> `restoreNativeCodex` (`src/server/management-api.ts:220-247`) | automatic generic restore job + drain join | Route schedules process exit before Worker join. | +| `ocx service stop` | service command -> `restoreNativeCodex` (`src/service.ts:2564-2595`) | explicit generic restore job | Service stop calls sync restore. | +| `ocx service start` | service command starts the installed daemon (`src/service.ts:2560-2564`) | daemon's ordinary startup apply job | Service-specific startup bypasses the ordinary startup/guardian root. | +| `ocx service uninstall/remove` | service command -> `restoreNativeCodex` (`src/service.ts:2610-2635`) | explicit generic restore job | Service uninstall drops/does not await job. | +| graceful SIGINT/SIGTERM/SIGHUP shutdown | drain then cleanup (`src/cli/index.ts:277-310`) | cancel/join active job, then await generic restore when policy requires it | Shutdown leaves history inline in `syncCleanup` or exits before join. | +| `process.on("exit")` / forced exit | synchronous callback at `src/cli/index.ts:310` | no history mutation; any existing durable pending operation remains for next startup | Exit hook imports writer or tries to spawn/await a Worker. | +| history guardian retry | timer currently calls provider directly (`src/codex/history-migration-guardian.ts:43-95`) | reread durable operation -> automatic job | Fake-clock test still passes after guardian startup dispatch is removed. | + +The table is exhaustive for production references to the current history-bearing +helpers, derived from `injectCodexConfig`, `restoreNativeCodex`, +`syncModelsToCodex`, `restoreLegacyOpenaiHistory`, and +`startHistoryMigrationGuardian`. Adding a new command/route that reaches one of those +symbols without an inventory row fails the inventory test. Adding a direct writer +bypass anywhere fails the graph test even if route counts are unchanged. + +## Failure, timeout, retry, and drain + +The parent owns one process-local flight per durable operation identity only to avoid +duplicate Worker threads; H provides cross-process exclusion. Same-operation callers +may join. A newer durable operation supersedes an older one: the old Worker either +rejects it at the under-H claim or loses the terminal CAS, releases H, and leaves the +newer operation pending for repair. + +Outcome classification remains evidence-bearing: + +- H or history DB busy -> `pending/db-busy` with a next retry; +- permission/refusal -> `blocked/permission`; +- unreadable DB/manifest -> `unknown/unreadable`, nullable counts; +- supported read but unsupported schema/shape -> `unknown/schema`, nullable counts; +- watchdog -> `unknown/timeout`; +- graceful cancellation -> `unknown/shutdown-cancelled`, then join; +- Worker error, malformed terminal IPC, or early close -> reread durable state before + conditionally publishing `unknown/worker-died`; +- superseded identity/terminal CAS conflict -> typed overtaken/superseded result, no + self-retry of the loser; +- terminal N update failure -> `unknown/record-write-failed`, preserving pending work. + +The Worker closes in `finally`; parent cancellation and shutdown await actual thread +exit using the existing join discipline (`src/storage/worker-lifecycle.ts:150-209`). +A watchdog contains one attempt; it never certifies convergence. + +Make execution policy invocation-local: + +| Caller mode | H / SQLite wait | Attempts / delay | Result | +|---|---:|---:|---| +| automatic (startup, management, guardian, graceful stop) | 100 ms | 1 / 0 ms | Defer durably and keep listener/drain bounded. | +| explicit CLI | 5,000 ms | 2 / 500 ms | Preserve the current operator wait budget inside the Worker. | -The native and history locks are siblings: +The current defaults are a module-global 5,000 ms busy timeout and two attempts with a +500 ms synchronous delay (`src/codex/history-provider.ts:31-49,526-548`). Automatic +mode never calls `sleepSync` on the parent. Explicit delay may occur inside the Worker +while H remains held so another process cannot overtake between attempts. -```text -native transition: acquire native -> synchronous native commit -> release native -history transition: acquire history -> validate pair -> mutate/probe/conditional update -> release history -``` +The guardian uses capped exponential backoff with deterministic injected jitter, +keeps at most one timer/Worker for the current durable operation, and has no finite +lifetime attempt cap. Startup immediately re-arms unresolved durable work. This +replaces the current sixty-tick terminal stop +(`src/codex/history-migration-guardian.ts:34-35,47-48,87-95`). -They are never held simultaneously. The history Worker never acquires the native -lock, and the native synchronous callback never spawns/awaits the Worker. This is -the checkable deadlock rule from `005_contract.md` §6. +## Deterministic tests -### Overtaking prevention +### Cross-process all-surface serialization -Sibling locks alone allow this sequence: A commits native ON, B commits native OFF, -B removes history, then A applies history. The request therefore carries A's -`CommitExpectation`. +Seed a production-shaped DB, valid manifest, and rollouts under temporary homes. +Process A schedules one real operation and pauses after the first real rollout write +while retaining H. Process B schedules the opposite operation and reaches H. Resume A; +assert its terminal update cannot replace B's newer durable operation, H is released, +and B repairs manifest, rollouts, and DB. Reverse direction/order and repeat. No test +stub mutates a surface; both processes enter production `history-job`/Worker/H. -Immediately after taking the history lock and before any probe or mutation, the -Worker reads the coordinator row keyed by canonical `CODEX_HOME`. The job is legal -only when both row fields equal the request's `{nativeGeneration,currentTxId}`. If -another native transition has advanced either field, the Worker returns -`CodexHistoryState { status:"pending", reason:"overtaken", ... }`, performs no -history write, and does **not** retry itself. The winning/newer transition owns the -next convergence. - -That first read is admission, not exclusion. B may commit a newer pair after A has -already changed the manifest, a rollout, or the DB. After the final under-lock probe, -A executes one SQLite conditional update of its result and schedule: - -```sql -UPDATE codex_transition_state - SET history_status = ?, history_reason = ?, history_attempts = ?, - history_next_retry_at = ?, history_tx_id = ?, - history_pending_rows = ?, history_backup_entries = ?, updated_at = ? - WHERE singleton = 1 - AND native_generation = ? - AND current_tx_id IS ? - AND history_tx_id IS ?; -``` +Broken change: release H after the DB transaction but before manifest/rollout/probe, or +dispatch the writer without H. The sentinels interleave and the test fails. -The coordinator database path already encodes effective user plus canonical -`CODEX_HOME`; the row is deliberately a singleton, not one row per -`OPENCODEX_HOME`. `updateCodexHistoryTransition(expected, state)` executes the statement -above. Its `kind:"updated"` result means exactly one changed row published A's -result; the implementation maps zero changed rows to `kind:"conflict"`. Conflict -means A was overtaken: it MUST NOT write JSON, MUST NOT overwrite or clear the newer -row's pending schedule, returns `pending/overtaken`, releases the history lock, and joins. -The parent then asks the guardian to read the current coordinator row and immediately -arm/retain the winner's schedule; it never retries A's losing transaction. - -The final post-probe and conditional row update happen before release. A clean mutation -followed by an unlocked probe is not evidence: another process could change rows or -the manifest in between. For target `openai`, `converged` requires a non-failed -probe with `pendingRows === 0` and `backupEntries === 0`; manifest absence or a -zero-row mutation alone is insufficient (`src/codex/history-provider.ts:749-775`). - -## Failure, timeout, and death - -The parent owns one process-local Worker flight only to avoid duplicate threads; -cross-process exclusion comes from the sibling lock, not this map. Same-transition -callers may join. Opposite transitions do not overwrite each other: each reaches -the lock and the older one is rejected by its expectation. - -Outcome order: - -- valid `done` + clean under-lock post-probe + one-row conditional update -> contract - `converged` state; -- SQLite/history-lock busy -> `pending/db-busy` with next retry; -- permission/refusal -> `blocked/permission`; -- unreadable data -> `unknown/unreadable` with both probe counts null; -- readable unsupported shape -> `unknown/schema` with both probe counts null; -- watchdog -> `unknown/timeout`, not `worker-died`; -- shutdown cancellation -> `unknown/shutdown-cancelled`, join, then drain; -- `worker.onerror`, malformed terminal message, or early close -> reread the - coordinator row before attempting `unknown/worker-died`; -- initial pair/snapshot mismatch or zero-row terminal update -> - `pending/overtaken`, no self-retry; -- coordinator update failure -> returned `unknown/record-write-failed`; the existing - pending row is left intact for guardian repair. - -The Worker closes in `finally`; the parent still waits for `close`/join using the -repository's existing discipline (`src/storage/worker-lifecycle.ts:150-209`). A -watchdog is containment, not convergence. It may interrupt legitimate large -history, so timeout can never be recorded as success. - -The reread on `worker.onerror`, malformed terminal IPC, or early close is mandatory -because the Worker may have committed its terminal SQLite update and died before -`postMessage`. The parent first calls `readCodexTransitionState`. If the row still -matches the job's native pair and `history_tx_id` and already contains a terminal -history state, that durable state is the result and the parent writes nothing. If a -newer pair owns the row, the parent returns `pending/overtaken` and arms the winner's -schedule. Only when the exact job still owns a `pending` or `running` row may the -parent conditionally call -`updateCodexHistoryTransition(expected, workerDiedState)`; a zero-row result follows -the same overtaken rule. If the reread is unavailable, the parent leaves the row -intact, returns `unknown/record-write-failed`, and lets the guardian retry from -durable state. A missing terminal message is therefore never permission to -overwrite a committed success with synthetic `worker-died`. - -## Fail-fast automatic mode and explicit mode - -The provider currently uses a mutable global 5,000 ms busy timeout and two retries -with a synchronous 500 ms sleep (`src/codex/history-provider.ts:25-49,526-548`). -Make the policy invocation-local: - -| Caller mode | Worker lock / SQLite wait | Attempts / delay | Reason | -|---|---:|---:|---| -| automatic (startup, management, guardian, stop) | 100 ms | 1 / 0 ms | Defer quickly; listener availability is the requirement. | -| explicit CLI | 5,000 ms | 2 / 500 ms | Preserve today's operator wait budget, but inside the Worker and under the same lock. | - -Automatic mode never calls `sleepSync` on the parent. Explicit delay may use -`sleepSync` inside the Worker because it cannot starve the proxy or bypass the -history lock. - -```diff - export interface HistoryExecutionOptions { - skipWhenProvablyNoop?: boolean; -+ busyTimeoutMs?: number; -+ attempts?: number; -+ delayMs?: number; -+ sleepFn?: (ms: number) => void; - } -``` +### H namespace and lock order -Apply `busyTimeoutMs` to both apply and restore database opens. Keep hard errors -throwing inside the Worker so its boundary can classify them once; do not turn -programming/data corruption into `db-busy`. +Two child processes vary `HOME`, `USERPROFILE`, `TMPDIR`, `XDG_RUNTIME_DIR`, `TEMP`, +`TMP`, and `LOCALAPPDATA` while retaining the same effective uid/SID, canonical +`CODEX_HOME`, and canonical state DB. They must resolve the same H final path. A second +canonical state DB under that home must resolve a different H while N/K remain the +same, and H must equal neither N nor K. Run the allowed `H -> N` and forbidden +`N/K/C -> H` symbol fixtures. -## Durable state consumes the coordinator row +Broken changes: hash the raw request path/environment home, omit state-DB identity, +reuse N/K's database, or introduce an inverse edge. Path-equality/inequality or graph +fixture fails. -Delete the former “Location and exact shape” JSON and the planned -`src/codex/history-convergence.ts`. The transition pair and pending history schedule -belong to the SQLite coordinator row keyed by canonical `CODEX_HOME`, not to an -`OPENCODEX_HOME` record. +### Operation binding -Both `history-worker.ts` and `history-job.ts` consume the contract-owned coordinator -API from `src/codex/transition-state.ts`; neither owns SQL or a second row shape. The -Worker calls `readCodexTransitionState` before traversal and -`updateCodexHistoryTransition(expected, state)` after its post-probe. The parent/guardian -uses the same reader to arm the current schedule after conflict. +For every `CodexHistoryOperation` variant, schedule it durably, tamper any diagnostic +request copy to a different operation, and assert the Worker either derives the +durable value or rejects before mutation. The manifest-independent recovery fixture keeps the +manifest byte-identical; the generic restore fixture consumes it only after successful +restore. -They never parse, write, or atomically replace `integrations/codex.json`. A terminal -history transition is one row update conditioned on canonical home plus the exact -pair. `txId` links the state to the native transition and `nextRetryAt:null` means -only “no timer armed now,” never “never again.” +Broken change: dispatch from request `targetProvider`/direction instead of the durable +operation. The variant and tamper cases fail. -The durable contract has no per-state-DB schema invented here. If multiple state -DBs need internal scheduling metadata, it remains an in-memory/job-private map; -the shared `CodexHistoryState` is the current convergence fact exposed to every -consumer. +### Manifest truth -## Retry ownership — no permanent dormancy +Run the malformed, unreadable, unsupported-shape, wrong-DB, and missing-DB-with-backup +fixtures through both the preflight and under-H post-probe. None may return numeric +zero/zero convergence. -Delete “every 60 seconds, at most 60 ticks per process” and the interpretation of -`nextRetryAt:null` as next-startup-only. That creates permanent dormancy in a -long-lived process (carried finding #9). +Broken change: map any manifest read failure to an empty manifest or initialize an +unknown count to zero. Its named fixture fails. -**INFERRED scheduling choice:** the guardian uses capped exponential backoff with -deterministic testable jitter: +### Retry, death, and guardian activation -```text -delay(attempt) = min(MAX_HISTORY_RETRY_MS, - BASE_HISTORY_RETRY_MS * 2^min(attempt, BACKOFF_EXPONENT_CAP)) -``` +Advance a fake monotonic clock through exponential growth and the cap. Assert startup +arms the guardian from durable unresolved state, every fired attempt goes through +`history-job`, and a later timer remains after retryable failure. Cover Worker error, +malformed terminal IPC, early close, watchdog, cancellation, and terminal-N failure; +assert join exactly once and preserve distinct reasons. -It schedules at most one timer and one Worker per current coordinator-row `txId`. It may back off -to the cap but never exhausts into a permanent state. Startup re-arms any unresolved -coordinator row whose timer was lost. A successful convergence clears the timer. An -`overtaken` job does not retry the losing transition; it schedules one observation -of the current generation so the winner owns work. - -This loop has a finite delay per attempt and no finite lifetime attempt count. -Shutdown cancels the current timer/Worker and leaves durable unresolved state for -the next process. - -## Process-aware callers use `convergeCodex` - -Delete `runCodexHistoryInline`, `HistoryExecution = "automatic" | "explicit"` as a -public alternate entry point, and every caller selection that bypasses convergence. -Mode is already in `ConvergeRequest`. - -```diff --const history = syncCodexHistoryProvider("openai", ...); -+const outcome = await convergeCodex({ -+ action: "converge", -+ scope: "full", -+ reason: "cli", -+ mode: "explicit", -+ deadlineMs: EXPLICIT_CODEX_CONVERGENCE_DEADLINE_MS, -+}); -``` +Broken changes: remove the startup guardian call, let guardian call a writer directly, +or stop after a fixed tick count. The activation/reachability/backoff cases fail. -Server startup/management/guardian uses `mode:"automatic"`; explicit CLI sync, -restore, eject, recover-history, ensure, and service cleanup uses -`mode:"explicit"`. Both modes reach the **same Worker and same history lock**. -`src/codex/inject.ts:602,783` loses direct provider calls; it exposes only bounded -native apply/restore receipts to convergence. +### Responsiveness for every root class -`src/codex/convergence.ts` sequence at this phase is: +Hold a real `BEGIN IMMEDIATE` on the history DB and overlap health/data-plane traffic +with each root class: `POST /api/sync`, `ocx init/setup` command handler, +graceful shutdown, and explicit legacy recovery. Bind server port `0` and use temporary +homes. Health responses remain 200, the stream completes, command/shutdown deadlines +remain bounded, durable state records non-success, and every Worker joins. Release the +holder and prove a later serialized job succeeds. -```text -admit current snapshot -> gather if ON -> native commit -> release native section --> dispatch history Worker(expectation, authoritySnapshotId) -> observe -> outcome -``` +Broken changes: run one root's probe/mutation on its caller thread or fail to await its +Worker during drain. The corresponding table row's responsiveness/drain case fails; +one responsive route cannot hide another inline caller. -Automatic calls may return `deferred` with unresolved `history`; explicit calls -wait only through their request deadline. Neither reports `converged` while history -is outstanding. - -The synchronous `process.on("exit")` hook cannot await a Worker. It performs no -history mutation and leaves/records unresolved state; graceful signal and command -paths call convergence before exit. This preserves process shutdown without -inventing an inline escape hatch. - -## Durable read surface - -`GET /api/codex/history` may expose the coordinator row's `history` projection -through an authenticated read-only route. It calls `readCodexTransitionState`; it -does not define a second state type or consult the non-CAS integration JSON. - -`POST /api/sync` is not redefined here. It already calls `convergeCodex` and -`toSyncResponse` after WP9 (`005_contract.md` §5). WP10 only ensures the resulting -`ConvergeOutcome` contains the contract `history` state. `ocx doctor` retains its -live read-only probe because durable state can be stale, but failed probes are -unknown rather than zero-looking success. - -## Key diffs - -### Worker owns lock, mutation, post-probe, and conditional row update - -```diff -+self.onmessage = async (event: MessageEvent) => { -+ const message = parseHistoryWorkerMessage(event.data); -+ if (message.type === "resume") return resumeHistoryCheckpoint(message); -+ const request = message; -+ applyCapturedHomes(request.env); -+ const lock = await acquireHistoryLock(request.lockIdentity, requestDeadline(request)); -+ if (lock.status !== "acquired") return postHistoryBusy(request, lock); -+ try { -+ const expected = { -+ nativeGeneration: request.expectation.nativeAfter, -+ currentTxId: request.expectation.txId, -+ }; -+ const admitted = readCodexTransitionState(); -+ if (!expectationStillCurrent(admitted, expected, request.authoritySnapshotId)) { -+ return postOvertaken(request); -+ } -+ const result = syncCodexHistoryProvider(request.targetProvider, request.stateDbPath, request.backupPath, policy(request)); -+ const postProbe = countPendingOpencodexHistory(request.stateDbPath, request.backupPath); -+ const state = classifyHistoryState(result, postProbe, request.expectation.txId); -+ const update = updateCodexHistoryTransition(expected, state); -+ if (update.kind === "conflict") return postOvertaken(request, postProbe); -+ if (update.kind === "unavailable") return postRecordWriteFailed(request, postProbe); -+ self.postMessage({ type: "done", requestId: request.requestId, state, postProbe, expectation: request.expectation, authoritySnapshotId: request.authoritySnapshotId }); -+ } finally { -+ lock.release(); -+ closeWorker(); -+ } -+}; -``` +### Symbol graph and inventory mutation checks -`release()` above is private to Worker implementation; unlike the native public -API, no caller can retain it across unrelated work. - -The parent terminal handler does not map missing IPC directly to `worker-died`. -Its error/close branch performs the reread rule above after join: adopt a matching -terminal row, arm a newer winner, or conditionally publish `worker-died` only while -the exact job still owns `pending`/`running`. This branch is tested at the seam -between the Worker's successful SQLite commit and `postMessage`. - -### Convergence dispatch, no inline branch - -```diff --historyExecution === "explicit" -- ? runCodexHistoryInline(input) -- : runCodexHistoryJob(input) -+await runCodexHistoryJob({ -+ ...input, -+ mode: request.mode, -+ expectation, -+ authoritySnapshotId: admittedSnapshotId(admission), -+}) -``` +Run every wrapper/alias/re-export/dynamic-import negative graph fixture and every +production command/route inventory row. For each inventory row, mutate its terminal +edge to a direct writer or no history dispatch and prove the test fails before +restoring the fixture. -## Test plan - -### Opposite-direction cross-process serialization - -1. Seed production-shaped DB, manifest, and rollouts in isolated homes. -2. Process A enters production `convergeCodex({scope:"full"})`; its real Worker - requests `pauseAfter:"first-rollout-write"`. The Worker performs the manifest and - first rollout mutations, posts `checkpoint {requestId, after}`, and waits for a - matching `resume` IPC message while still holding the history lock. -3. Process B converges OFF and commits the newer coordinator pair while A is paused. - B's history Worker waits on the history lock; native pair advancement does not. -4. Resume A. Its remaining traversal and post-probe complete, but its terminal - conditional row update affects zero rows. Assert A returns `pending/overtaken`, - never touches the newer pending schedule, releases/joins, and causes the guardian - to arm B from the current row. -5. Let B acquire history and repair every manifest, rollout, and DB sentinel. Reverse - direction/order and repeat. Final history must match the highest native generation, - not Worker scheduling order. - -The checkpoint is deterministic because it is acknowledged only after the real -writer reports a completed surface mutation, and resume is keyed by `requestId`. -There is no alternate provider stub or direct test-only mutation path: both processes -enter the production convergence/job/Worker protocol. A same-process flight, a pause -before traversal, or two connections without manifest/rollout/DB sentinels does not -satisfy C15. - -### CLI contention - -- Hold the production history lock in a child. Invoke an explicit CLI convergence - through its function-level command handler with `mode:"explicit"`; assert it - waits/returns the typed contract outcome and performs no inline provider call. -- In parallel trigger automatic server convergence; assert listener health/data - plane progress while both processes contend. -- Release, join both Workers, and prove one serialized winner. The test inspects the - transition row through `readCodexTransitionState`, not a WP10 parser. - -### Post-probe under lock - -- Inject a competing child that attempts to change a history row and manifest at - the probe seam. Assert it cannot proceed until after terminal state is recorded - and lock released. -- A failed probe, nonzero pending rows, or nonempty backup entries remains - non-converged. Only clean zero/zero becomes `converged`. - -### Retry and death - -- Advance a fake monotonic clock through exponential growth and the cap; prove a - later timer always exists for unresolved current work and no 60-tick terminal - state exists. -- Restart/module reload re-arms unresolved state. -- Worker error, malformed response, early close, watchdog, cancellation, and final - coordinator-row write failure retain their distinct contract reasons, carry nullable - probe counts, remain non-converged, and join exactly once. For - error/malformed/close, the parent rereads first: an already-terminal matching row - wins; only a matching `pending`/`running` row may be conditionally changed to - `worker-died`. -- An overtaken transition does not retry itself. - -### Measured responsiveness — C3 - -Keep the real `BEGIN IMMEDIATE` holder and overlapping `/healthz` plus eight-chunk -SSE test from the prior plan, but route the request through production -`convergeCodex`. Bind port `0`; use temporary homes. Require the management/history -request to overlap contention, every health response to be 200, the stream to -complete, and the durable state to report `db-busy` before succeeding after release. +Broken change: add a seventh route, wrapper, alias, or new command that reaches a +writer outside the inventory. Symbol reachability fails even if regex counts do not +change. ## Verification ```bash bun run typecheck bun test tests/codex-history-provider.test.ts tests/codex-history-worker.test.ts -bun test tests/history-migration-guardian.test.ts tests/codex-history-process-routing.test.ts +bun test tests/codex-transition-state.test.ts tests/history-migration-guardian.test.ts +bun test tests/codex-history-process-routing.test.ts tests/codex-convergence-contract.test.ts bun test tests/codex-sync-api.test.ts tests/shutdown-drain.test.ts bun test tests/codex-history-worker-responsive.test.ts --timeout 30000 bun run privacy:scan bun run test ``` -The responsiveness test prints lock-ready time, overlapping health latencies, -stream chunk count, history elapsed time/state, child exit, and live Worker count. -No verification command invokes `ocx start`, `ocx stop`, `ocx sync`, `ocx restore`, -or `ocx ensure`; port 10100 remains untouched. - -## Accept criteria - -- **C3** — all synchronous/unbounded history work is in the Worker; real contention - overlaps responsive `/healthz` and data-plane traffic. -- **C4** — unresolved work is durably represented by the contract - `CodexHistoryState`, retried with capped non-permanent backoff, and never collapsed - into success. Clean post-probe occurs under the history lock. -- **C15** — opposite-direction processes serialize manifest, rollouts, DB, probe, - and terminal coordinator update; the pair-conditioned row update detects an - overtake even after the stale Worker has mutated a history surface, preserves the - winner's pending schedule, and drives repair. -- Explicit CLI and automatic server/startup/retry callers all enter through - `convergeCodex` and the same sibling history lock. No inline escape hatch remains. -- **N2** — WP10 imports the WP8b coordinator/types and extends WP9's working funnel. Its - commit typechecks and preserves behavior without any WP11/WP12 placeholder. +All process tests use `mktemp -d`/temporary homes and port `0`. No command invokes +`ocx start`, `ocx stop`, `ocx sync`, `ocx restore`, `ocx ensure`, or `ocx service *`; +the installed service and live proxy on 10100 remain untouched. + +## Accept criteria — each criterion has a red test + +| Criterion | Passing evidence | Concrete broken change that turns it red | +|---|---|---| +| **C3 — caller responsiveness** | Table-driven responsiveness covers management, init/setup, graceful shutdown, and explicit recovery while real SQLite contention overlaps health/SSE progress. | Move any listed root's probe or mutation back to the caller thread. That root's latency/progress case fails. | +| **C4 — durable unresolved work** | Guardian activation/backoff test proves unresolved typed operation survives failure/restart and never becomes zero-looking success. | Remove startup arming, restore the 60-tick stop, or persist zero counts after failed evidence. The activation/backoff/evidence case fails. | +| **C15 — cross-process all-surface serialization** | Opposite operations serialize manifest, rollout, DB, post-probe, and terminal update under one H; newer durable operation repairs stale work. | Release H between surfaces or bypass H in one process. The sentinel/final-state case fails. | +| **Operation authority** | Every operation variant is derived/validated from durable state, including no-op and manifest-independent recovery. | Trust request `targetProvider`/direction. Tamper and manifest-preservation cases fail. | +| **Real lock order** | Architecture fixtures allow `H -> N` and reject every inverse/cross-domain edge. | Await/spawn H while N is held, or call K from the Worker. Dependency fixture fails. | +| **One H namespace per canonical history DB** | Environment-divergent child processes resolve one H for the same effective user/home/DB, a different H for a second DB, and paths distinct from N/K. | Key by environment/raw alias, omit DB identity, or reuse N/K. Resolver equality/inequality case fails. | +| **Manifest evidence** | Malformed, unreadable, unsupported, wrong-DB, and missing-DB-with-backup fixtures remain non-converged with nullable unknown counts. | Convert any failed manifest read to empty/zero. Its fixture fails. | +| **No writer bypass** | TypeScript symbol reachability permits only `history-worker.ts` as a production writer root and catches wrappers, aliases, re-exports, namespace and dynamic imports. | Add any direct/indirect production writer path. Graph test fails. | +| **Complete current caller routing** | Inventory covers every production command/route and each history-bearing row reaches `runCodexHistoryJob`; named disconnect mutations are red. | Remove init, guardian, shutdown, service, or any other row's job edge, or add an unlisted caller. Inventory test fails. | +| **N2 — independently landable** | WP10 typechecks and focused/full suites pass while executable convergence remains catalog-only; current operation semantics are preserved through H. | Import/call full `convergeCodex`, require WP11 native lock/receipt, or leave a temporary inline path. Typecheck/routing behavior test fails at the WP10 commit. | From b8168510f32ef583f14660067e0783d88df5df0c Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Wed, 5 Aug 2026 03:35:20 +0900 Subject: [PATCH 084/163] docs(substrate): the split I chose left a window where a newer restore vanishes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Deferring the convergence rewire to WP12 was right, but it left history authorization happening after the native root returned, with nothing ordering the two. The reviewer walked it: A applies routing and schedules its history work, A's Worker pauses under H, B restores native config and catalog, and B's authorizer sees A running and refuses B's restore. A then finishes tagging history forward, and B's newer intent was never persisted, so the guardian has nothing to repair. The same shape lets an older apply outlive a newer `syncResumeHistory:false`. So the native exclusion comes forward from WP11, but only the part that closes this: a caller holds N across the span from its native mutation through its history authorization, so authorization order matches native order and B cannot authorize into a window A already owns. N is released before the Worker is spawned. WP11 keeps the lock module itself, the namespace mechanics, the async acquisition API, admission validation, and broad adoption by native writers. `H → N → K → C` still holds — the exclusion IS N, it may enter K and C, and it never reaches H — so the acyclicity the reviewer proved last round survives. Newer work is also durable now rather than dropped: a later authorization is recorded even over pending or running work, and the older Worker loses its terminal CAS instead of publishing over it. The tenth absence-as-guarantee sat in the manifest type. The contract reasoned that a present ready manifest must hold entries because the writer deletes empty ones, but the real reader accepts a valid v1 manifest with `entries: {}` and restore treats that as an ordinary empty case before ejecting residual routed rows. `missing` would have inferred absence from present evidence, and calling it unsupported would have left those rows behind, so `ready` now carries a non-negative count and the fixture runs through preflight, ejection and post-probe. One correction I asked for and did not get, correctly: the plan's second race ordering cannot end with B authorizing first, because the exclusion spanning A's native mutation makes that unreachable by construction. The test proves B is blocked rather than pretending it succeeds. --- .../005_contract.md | 301 ++++++++++++------ .../020_history_isolation.md | 157 ++++++--- 2 files changed, 322 insertions(+), 136 deletions(-) diff --git a/devlog/_plan/260804_codex_write_substrate/005_contract.md b/devlog/_plan/260804_codex_write_substrate/005_contract.md index 1d7c396f8..faf82a7e3 100644 --- a/devlog/_plan/260804_codex_write_substrate/005_contract.md +++ b/devlog/_plan/260804_codex_write_substrate/005_contract.md @@ -11,6 +11,9 @@ So this document is now the complete definition of every shared surface, and the four phase docs are rewritten as consumers against the reviewer's section list. A contract nobody collected is a fifth opinion. +All current-code citations in this document were rechecked on 2026-08-05 at +`45f7bb7caf9c836b9d9a398c76ae4cb2f7461860`. + ## IN / OUT IN: `src/config.ts` (MODIFY — durable config-generation API), @@ -26,10 +29,13 @@ SQLite transition row), `tests/codex-convergence-contract.test.ts` (NEW), `tests/codex-user-identity.test.ts` (NEW). -OUT: catalog mechanics (WP9), history mechanics (WP10), native-lock acquisition -and retry mechanics (WP11), ownership mechanics (WP12). The final coordinator -path, transition table/CAS, config-generation API, shapes and funnel are IN; -the domain work performed while those coordinators are held is OUT. +OUT: catalog mechanics (WP9), history mechanics (WP10), the full native-lock +namespace/acquisition API and broad caller adoption (WP11), ownership mechanics +(WP12). WP10 is the one narrow exception to the former native-lock boundary: it +uses the already-owned N transaction as a compatibility native-handoff exclusion +from each retained native mutation through its history authorization. The final +coordinator path, transition table/CAS, config-generation API, shapes and funnel +are IN; the domain work performed while those coordinators are held is OUT. ### What "lands first" has to mean (round 2 N2) @@ -131,18 +137,19 @@ export interface CodexHistoryState { } /** - * A manifest read is evidence, not an optional convenience. Only `missing` - * certifies zero entries. Every failed read keeps its failure class and a null + * A manifest read is evidence, not an optional convenience. `missing` alone + * certifies path absence; `ready` certifies a present valid matching v1 file and + * may carry zero entries. Every failed read keeps its failure class and a null * count so absence can never be manufactured from unreadable bytes. */ -export type PositiveHistoryManifestEntryCount = number & { - readonly __positiveHistoryManifestEntryCount: "validated-positive-integer"; +export type NonNegativeHistoryManifestEntryCount = number & { + readonly __nonNegativeHistoryManifestEntryCount: "validated-non-negative-integer"; }; export type CodexHistoryManifestRead = | { readonly kind: "missing"; readonly manifest: null; readonly backupEntries: 0 } | { readonly kind: "ready"; readonly manifest: T; - readonly backupEntries: PositiveHistoryManifestEntryCount } + readonly backupEntries: NonNegativeHistoryManifestEntryCount } | { readonly kind: "unreadable"; readonly manifest: null; readonly backupEntries: null } | { readonly kind: "malformed"; readonly manifest: null; readonly backupEntries: null } | { readonly kind: "unsupported"; readonly manifest: null; readonly backupEntries: null } @@ -369,8 +376,10 @@ export type AuthorizeCodexLegacyHistoryRecovery = ( ) => TransitionStateUpdate; /** - * WP10-only bridge for current roots that predate WP12 native admission. It - * publishes history work without moving the native routing pair. + * WP10-only bridge for current roots that predate WP12 native admission. The + * transition-state owner supplies this closure only inside the already-open N + * transaction that excluded the retained native mutation. It publishes the + * newer history work without moving the native routing pair. */ export type AuthorizeCodexCompatibilityHistory = ( expected: CodexTransitionState, @@ -382,6 +391,19 @@ export type AuthorizeCodexCompatibilityHistory = ( }>, ) => TransitionStateUpdate; +type SynchronousCompatibilityHandoff = T extends PromiseLike ? never : T; + +/** + * WP10's narrow native-mutation-to-history-authorization exclusion. The callback + * starts only after N is held, must authorize exactly once through the supplied + * transaction-bound closure, and returns before N is committed and released. + */ +export type WithCodexCompatibilityNativeHandoff = ( + mutateNativeAndAuthorize: ( + authorize: AuthorizeCodexCompatibilityHistory, + ) => SynchronousCompatibilityHandoff, +) => T; + /** Change only history columns when the exact pair, job and operation still own the row. */ export type UpdateCodexHistoryTransition = ( expected: CodexHistoryScheduleExpectation, @@ -525,19 +547,47 @@ IPC whose job/operation differs before any probe or mutation. WP10 must land before WP12, and the current apply/restore roots have neither a WP12 `AdmissionSnapshot` nor a native routing pair published by convergence. Inventing a snapshot id or bumping `nativeGeneration` would violate the settled phase boundary. -`authorizeCodexCompatibilityHistory` is the explicit bridge: after the existing -native root returns its semantic receipt, the convergence owner derives one of -`skip | apply-opencodex | migrate-openai | restore-openai`, opens N, and conditionally -replaces only a terminal history schedule with a fresh job id, -`authority.kind:"wp10-compatibility"`, and a fresh opaque authority id. The id is a -job authorization nonce, not a digest of config or credentials and is never logged. +The latest review found that authorizing only **after** the retained native root +returned could lose the newer operation: apply writes config/profile/journal before +history (`src/codex/inject.ts:594-604`), and restore changes config plus K-serialized +catalog state before history (`src/codex/inject.ts:765-789`). If A was already +`running`, B could complete the newer native restore, see non-terminal A, refuse its +schedule, and leave A free to terminally record the obsolete direction. + +WP10 therefore brings forward exactly one piece of WP11: the N-backed compatibility +native-handoff exclusion. `history-job.ts` enters +`withCodexCompatibilityNativeHandoff` **before** invoking a retained synchronous +native mutation, receives an authorizer closure bound to that already-open N handle, +and uses it before returning from the callback. The transaction then commits/releases +N before any Worker spawn or await. Current apply/restore owners still derive one of +`skip | apply-opencodex | migrate-openai | restore-openai`; they supply the derived +operation and synchronous native callback to `history-job.ts`, never call the +transition-state API directly, and never move native generation in WP10. + +That transaction-bound `authorizeCodexCompatibilityHistory` replaces the exact row +observed when N was acquired with a fresh pending job id, +`authority.kind:"wp10-compatibility"`, and fresh opaque authority id **regardless of +whether the older history state is terminal, pending, or running**. N exclusion makes +that authorization order the retained native-mutation order; replacing non-terminal +work is therefore supersession by a newer native operation, not theft by a peer. The CAS leaves the native pair unchanged and matches the prior pair, job, operation, -authority kind/id, and terminal status. It refuses pending/running work. No CLI, -server, `inject.ts`, or low-level history writer calls this API directly. The WP10 -convergence adapter derives the operation and delegates the mechanical CAS to -`history-job.ts`; §8's middle inventory permits only that scheduling edge. WP12-final -graph reachability removes it once admission-snapshot scheduling owns native -convergence. +complete authority, and status. The older Worker subsequently loses its full-identity +terminal CAS, cannot clear the winner's timer, and the guardian can repair from the +new durable schedule. The id is a job authorization nonce, not a digest of config or +credentials and is never logged. + +No CLI, server, `inject.ts`, or low-level history writer imports either transition- +state authorizer. `history-job.ts` is the sole bridge and scheduling edge in §8's +middle inventory. WP12-final graph reachability removes the compatibility handoff +once admission-snapshot scheduling owns native convergence. + +What does **not** move from WP11 is equally explicit: WP10 does not implement +`codex-write-lock.ts`, the uid/SID namespace mechanics for that full lock, canonical +target/admission validation, finite async acquisition/retry and result taxonomy, +`CommitExpectation`, provenance coordination, or adoption by every native writer. +WP11 still owns that complete async N → K → C mechanism and its broader caller +rewire. WP10 only closes the retained native-mutation-to-history-authorization gap +using the coordinator database and transition owner that already landed in WP8b. Explicit legacy recovery cannot honestly advance `nativeGeneration`: §3 defines it as a routing transition, while this command changes history only. Under N, @@ -581,11 +631,13 @@ after an older-writer update. ## 2. One convergence entry point -Audit #2: `010` rewires 16 management callers to a direct gather/commit helper, -and `040` never touches them, so a provider edit commits catalog bytes with no -ownership, provenance, intent or lock check. Today that helper is -`refreshCodexCatalogBestEffort` (`src/server/management-api.ts:105-112`) and its -entire error handling is `catch { /* catalog absent */ }`. +Audit #2: `010` originally proposed rewiring 16 management callers to a direct +gather/commit helper while `040` never touched them, so a provider edit could commit +catalog bytes with no ownership, provenance, intent or lock check. The audited +`refreshCodexCatalogBestEffort` root has since been replaced on this branch by the +catalog-only convergence closure (`src/server/management-api.ts:133-160`), but the +shared funnel rule remains the reason that transitional implementation may not become +a second final entry point. ```ts /** @@ -824,9 +876,9 @@ bump the config generation because it writes no persisted OpenCodex config bytes The round-3 auditor ran the retained management `POST /api/sync` chain while a catalog candidate was paused after validation. `refreshCodexModelCatalog` -(`src/codex/refresh.ts:40-52`) still reaches direct catalog replacement -(`src/codex/catalog/sync.ts:568`) and models-cache replacement -(`src/codex/catalog/sync.ts:600-616`). That writer neither advances config generation +(`src/codex/refresh.ts:40-52`) reaches catalog replacement +(`src/codex/catalog/sync.ts:664-733`) and models-cache replacement +(`src/codex/catalog/sync.ts:832-850`). That writer neither advances config generation nor enters `withExpectedConfigGenerationSync`, so the config transaction alone let it publish Y before convergence resumed and replaced Y with bytes gathered from X. This is a first-party writer retained by this plan, not a foreign hand edit that the @@ -858,13 +910,15 @@ publication callback are synchronous. Lock busy/unavailable follows each retaine function's existing no-write or write-failure path rather than changing it to a Promise. -Round 4 showed why “the replacement is under K” is not enough. The retained -`/api/sync` chain reads the active catalog, captures `onDiskCatalog`, awaits provider -gathering, and only then writes from that captured state -(`src/codex/catalog/sync.ts:513,520,526,565`). If convergence publishes Y while that -await is pending, taking K afterwards does not make the captured X fresh: the retained -writer can legally overwrite Y while holding K. Cache invalidation and restore are -also read-transform-write operations, not bare replacements. +Round 4 showed why “the replacement is under K” is not enough. The audited retained +`/api/sync` chain read the active catalog, captured `onDiskCatalog`, awaited provider +gathering, and only then wrote from that captured state. The landed branch now makes +that seam visible: it captures pre-await evidence (`src/codex/catalog/sync.ts:619-630`), +then after gathering acquires K, revalidates and rereads the active catalog before +writing (`src/codex/catalog/sync.ts:736-775`). The contract requires that corrected +shape because taking K after an await cannot by itself make captured X fresh. Cache +invalidation and restore are also read-transform-write operations, not bare +replacements. Only slow provider/network gathering may therefore remain outside K. Every first-party catalog root uses exactly one of these two freshness shapes: @@ -1128,21 +1182,23 @@ into a guarantee the filesystem cannot provide. Round 5 reproduced the same absence-as-equivalence defect before candidate construction. `providerCatalogFingerprint` covers endpoint and catalog fields but omits `authMode`, `apiKey`, and `headers` -(`src/codex/catalog/provider-fetch.ts:134-156`). `gatherFlightKey` hashes that partial -projection (`src/codex/catalog/provider-fetch.ts:159-179`), even though -`fetchProviderModels` branches on `authMode`, resolves credentials, and builds the -effective discovery request (`src/codex/catalog/provider-fetch.ts:472-499,546-566`). -The map lookup then lets the second caller join the first promise solely by that key -(`src/codex/catalog/provider-fetch.ts:790-819`). A generation-N forward-auth gather +(`src/codex/catalog/provider-fetch.ts:476-499`). `gatherFlightKey` hashes that partial +projection (`src/codex/catalog/provider-fetch.ts:501-520`), even though flight capture +resolves credential-bearing request authority +(`src/codex/catalog/provider-fetch.ts:408-445`) and model fetch branches on the +captured auth mode (`src/codex/catalog/provider-fetch.ts:814-826`). The in-flight map +lookup is the join boundary (`src/codex/catalog/provider-fetch.ts:1128-1169`). A generation-N forward-auth gather can therefore supply empty bytes to a generation-N+1 key-auth admission; B's later K -> C validation is honest but irrelevant because no evidence says A produced the joined result. -The current in-progress WP9 worktree prefixes the key with a plain SHA-256 of the -auth-store buffer (`src/codex/catalog/provider-fetch.ts:771-787`). That does not close -the finding: static key/forward mode and configured headers still collide, the result -still carries no authority, and a stable plain digest of credential-store bytes is the -privacy trap this rule forbids. +The reviewed intermediate WP9 worktree prefixed that key with a plain SHA-256 of the +auth-store buffer. The current branch has already replaced that specific mistake with +a process-keyed auth identity and full provider-graph comparison +(`src/codex/catalog/provider-fetch.ts:408-445,1114-1124,1134-1161`). That landed +partial repair does not weaken this contract: result authority, complete source and +process evidence, and the no-stable-credential-digest rule remain required rather +than inferred from a map-key implementation. This contract keeps single-flight sharing rather than prohibiting all cross-admission sharing. The admission gate exists to suppress a thundering herd of provider model @@ -1212,11 +1268,13 @@ the captured config/auth/discovery-policy/native/source/process snapshots as arguments and may not re-resolve them after claiming its slot. That prohibition binds the whole post-await tail, not only the join decision: round 8 found `augmentRoutedModelsWithRegistryOpenAiApiRows` re-reading the registry after the -network await (`src/codex/catalog/provider-fetch.ts:955-961`), so a flight can key and -carry its authority honestly and still emit bytes derived from a policy that changed -while it waited. Every downstream augmentation input — including the -registry-transport match outcome that decides whether trusted OpenAI rows are added — -is passed in from the captured snapshot. `GatherFlightResult` carries the exact +network await, so a flight could key and carry its authority honestly and still emit +bytes derived from a policy that changed while it waited. The current branch passes +the captured policy into the post-await augmentation +(`src/codex/catalog/provider-fetch.ts:1200-1213,1330-1337`). Every downstream +augmentation input — including the registry-transport match outcome that decides +whether trusted OpenAI rows are added — is passed in from the captured snapshot. +`GatherFlightResult` carries the exact authority identity that produced its models and omissions. Before candidate construction, every caller compares that result identity with its own expected identity. Inequality discards the result and returns retryable `stale` or regathers @@ -1631,35 +1689,48 @@ convergence from evidence it never read: once again, absence was treated as a guarantee. The sole manifest reader returns `CodexHistoryManifestRead`. Only a genuinely -missing path returns `kind:"missing"` and `backupEntries:0`. A present valid v1 -manifest for the canonical state DB returns `ready`; because empty manifests are -deleted by the writer, a present ready manifest must contain at least one validated -entry. Only the manifest validator constructs -`PositiveHistoryManifestEntryCount`, after proving an integer greater than zero; -`ready` with literal zero does not typecheck. Read/permission failure returns -`malformed`, readable unsupported version/shape returns `unsupported`, and a valid +missing path returns `kind:"missing"`, `manifest:null`, and `backupEntries:0`. +The tenth absence-as-guarantee review found the opposite present case: the real +reader accepts a matching v1 manifest with `entries:{}` +(`src/codex/history-provider.ts:204-217`), and restore treats it as the ordinary +empty-manifest branch before ejecting residual routed rows +(`src/codex/history-provider.ts:656-665`). The first-party writer normally deletes an +empty manifest (`src/codex/history-provider.ts:220-226`), but writer policy cannot +erase valid evidence that the reader actually found. + +Therefore every present valid matching v1 manifest returns `ready`, including zero +entries. Only the manifest validator constructs +`NonNegativeHistoryManifestEntryCount`, after proving an integer greater than or +equal to zero; `ready` with literal zero is valid only through that validator, while +negative, fractional, or non-finite counts are impossible. This option keeps file +presence in `kind:"ready"` and preserves the existing restore/ejection branch without +adding a second ready-state control path. Read/permission failure returns +`unreadable`, readable unsupported version/shape returns `unsupported`, and a valid manifest naming another canonical state DB returns `foreign-state-db`. Every failure preserves the file byte-for-byte, carries `backupEntries:null`, prevents all history mutation, and blocks convergence. `unreadable` maps to `unknown/unreadable`, `malformed`/`unsupported` to `unknown/schema`, and a foreign manifest to `blocked/foreign-state-db`; no path deletes, replaces, or consumes failed evidence. -**Sibling locks permit overtaking.** A releases the native lock after committing -ON; B commits native OFF while A traverses history. Checking once after A acquires -the history lock only moves the race: B can still advance the native pair before A -finishes. The previous check against `nativeBefore` was also the wrong side. After -A's native commit, the record is expected to contain A's -`{ nativeAfter, txId }`, not `nativeBefore`. +**Sibling locks permit overtaking, but the handoff itself may not be unordered.** A +releases N after atomically authorizing ON history; B may then acquire N and commit +native OFF while A traverses under H. That overtaking is intentional. What the latest +review rejected is a post-native/pre-authorization hole: no retained caller may write +native state before taking N or release N before publishing its compatibility +schedule. The previous check against `nativeBefore` was also the wrong side. After +A's future WP11 native commit, the record is expected to contain A's +`{ nativeAfter, txId }`, not `nativeBefore`; WP10's bridge leaves that pair unchanged +but replaces the complete older history identity while the same N exclusion is live. This contract chooses **detect-and-repair**, not a transition gate shared across the complete history unit. The guarantee is eventual convergence to the latest durable native transition: -1. The native coordinator CAS writes `{nativeAfter, txId}` and the complete - `history_status='pending'` schedule, including job id and operation, in the - **same SQLite row update** before any - Worker spawn. If spawn never occurs or the Worker dies, the guardian/startup - reader still has durable work to schedule. +1. WP10's retained roots hold N from before native mutation through the compatibility + authorization and commit the complete `history_status='pending'` schedule before + any Worker spawn. WP11/WP12 later write `{nativeAfter, txId}` and that same complete + schedule in one conditional row update. In both phases, if spawn never occurs or + the Worker dies, the guardian/startup reader still has durable work to schedule. 2. A Worker checks that the coordinator row contains its `{nativeAfter, txId}` plus exact history job id, operation, and authority kind/id immediately after acquiring H through the fail-fast `H -> N` read. A mismatch returns @@ -1680,15 +1751,18 @@ write after a newer native commit; the testable claim is that the latest pair st durably scheduled and eventually owns the clean under-lock post-probe, even across spawn failure, Worker death, or process restart. -Ordering, so absence of deadlock is checkable: the native callback performs its -transition-row UPDATE in the native coordinator transaction, then releases it before -history dispatch and never calls H. A Worker holds H while traversing and attempts -only fail-fast short N transactions at claim/terminal boundaries; it never invokes -the native callback, K, or C. `SQLITE_BUSY` at N leaves the current pending row intact, -releases H, and retries from durable state. The graph tests require the real `H -> N` -edges and reject inverse `N -> H`, `K -> H`, and `C -> H` reachability through direct -imports, wrappers, aliases, re-exports, and dynamic imports; they also reject the -unneeded Worker edges `H -> K` and `H -> C`. +Ordering, so absence of deadlock is checkable: WP10's compatibility native exclusion +**is N**. Its callback may enter K and then C where the retained native/catalog path +already requires them, authorizes history through the already-open N handle, and +releases N before history dispatch; it never calls H. A Worker holds H while +traversing and attempts only fail-fast short N transactions at claim/terminal +boundaries; it never invokes the native callback, K, or C. `SQLITE_BUSY` at N leaves +the current pending row intact, releases H, and retries from durable state. Thus the +existing DAG remains `H → N → K → C`: the handoff adds work *inside* N and no edge +back to H. The graph tests require the real `H -> N` edges and reject inverse +`N -> H`, `K -> H`, and `C -> H` reachability through direct imports, wrappers, +aliases, re-exports, and dynamic imports; they also reject the unneeded Worker edges +`H -> K` and `H -> C`. ## 7. The lock namespace has one environment-independent root per effective user @@ -1975,9 +2049,12 @@ direction/operation pair and reject impossible combinations. Authorize explicit the same CAS must refuse a pending/running row and a stale prior job/operation. Exercise the WP10 compatibility authorizer for each non-recovery operation against generation zero and a positive unchanged native pair; it persists -`wp10-compatibility`, never advances the pair, and refuses stale prior authority or -non-terminal work. Its authority id must not equal or derive from config/credential -bytes and must not reach logs, JSON, responses, or exceptions. +`wp10-compatibility`, never advances the pair, and refuses stale prior authority. +When called through the N-bound handoff after a newer retained native mutation, it +must replace terminal, pending, and running older schedules with a fresh pending +identity. The old Worker's terminal CAS must then change zero rows. Its authority id +must not equal or derive from config/credential bytes and must not reach logs, JSON, +responses, or exceptions. Missing DB/table initializes only from native-clean/no-legacy state; legacy JSON pair/schedule, residue beside a missing row, malformed row, busy DB and unsafe path all fail closed with the specified typed outcome. @@ -2159,6 +2236,19 @@ Worker death, timeout, shutdown cancellation, unreadable/schema probes, and term record-write failure; every failed probe count is null and the latest transition remains durably schedulable. +Add both native-handoff orderings forced by the latest review. First, let A authorize +and reach `running` under H, then let B acquire N, complete the opposite retained +native mutation, replace A's running schedule with B's pending identity, and block at +H until A finishes; A loses its terminal CAS and B repairs all three surfaces. +Second, pause A after its last real native write but before its transaction-bound +authorization, then start B's attempt to authorize first. B must not enter its native +callback or complete authorization while A owns N; after A authorizes/releases, B +performs its native mutation and authorizes second. The concrete broken changes are +**release N after native mutation but before authorization**, **open N only after the +native callback returns**, and **restore the terminal-only compatibility predicate**: +the attempted inversion publishes B first or the running-schedule case drops B, and +the final-state/terminal-CAS assertions fail. + Hold H in one process and prove a second service/CLI Worker for the same canonical home/state DB cannot enter manifest, rollout, DB, probe, or terminal-CAS work. While H is held, execute the real fail-fast claim read and terminal update and observe @@ -2167,14 +2257,19 @@ must retain those two required edges and fail independently for injected `N -> H `K -> H`, `C -> H`, `H -> K`, and `H -> C` edges. For the ninth absence-as-guarantee regression, probe a genuinely missing manifest -and require `backupEntries:0`. Then seed, one case at a time, unreadable bytes, +and require `kind:"missing"`, `manifest:null`, and `backupEntries:0`. Then seed, one case at a time, unreadable bytes, malformed JSON, a readable unsupported version/shape, and a valid manifest naming a different canonical state DB. Every present failure keeps its exact bytes, reports `backupEntries:null`, performs no rollout/DB/manifest mutation, and cannot produce a -zero/zero converged state. A valid matching non-empty manifest remains readable and -consumable. A compile fixture assigning zero to a `ready` result must fail while the -`missing` zero shape compiles. Run the same matrix through no-op admission, final post-probe, guardian, -and doctor projection so no wrapper reintroduces zero. +zero/zero converged state. For the tenth recurrence, seed a valid present matching v1 +manifest with `entries:{}`. Preflight must return `kind:"ready"` with a validator- +constructed non-negative count of zero, generic restore must execute residual routed- +row ejection without manufacturing `missing` or `unsupported`, and post-probe must +still represent the present valid file as `ready` zero. The fixture preserves its +bytes under the current empty-manifest restore branch. A compile fixture accepts the +validator-produced ready-zero shape and rejects a negative count; `missing` remains +the only null-manifest zero shape. Run the same matrix through no-op admission, final +post-probe, guardian, and doctor projection so no wrapper reintroduces absence. **The funnel must be provable, not grepped** (round 2 #2). A grep guard misses a wrapper, re-export, alias or dynamic import. The writer-inventory test above is the @@ -2190,8 +2285,12 @@ writes to it. permits no legacy or compatibility authorizer root. At every version every first-party catalog/backup/cache write requires a fresh permit from the same permanent K owner, and every low-level mutator rejects leaked, reused, forged, revoked, or wrong-home - permits at runtime before filesystem mutation. -- C16 — one owner, one schema; a record from any phase reads in every other. + permits at runtime before filesystem mutation. **Broken change:** add a fifth WP9 + catalog root, retain the WP10 compatibility authorizer in WP12-final, or let a writer + skip K's runtime permit assertion; the phase graph or permit mutation fixture fails. +- C16 — one owner, one schema; a record from any phase reads in every other. **Broken + change:** add transition/history authority fields back to the JSON record or make + provenance required at v1; the cross-phase round-trip/legacy-ambiguity fixture fails. - C17 — cooperating transition ABA is detected by the durable config/native generations and exact txId, and a parent target that drifts once between gather and the under-lock commit check is detected by canonical target identity. A @@ -2219,14 +2318,26 @@ writes to it. a later first-party publication. Create-once backups use atomic no-clobber publication. An arbitrary filesystem, selector, or content A→B→A that completes wholly between two checks, and a non-cooperating write after the final comparison, - are explicitly not claimed. + are explicitly not claimed. **Broken change:** restore the partial + `providerCatalogFingerprint` as the sole flight identity, remove result-authority + equality, or replace complete under-K evidence revalidation with target-path-only + comparison; the live-flight or stale-publisher fixture accepts X over Y and fails. - Contributes to C15 with detect-and-repair: the latest native pair, history job id, typed operation, and authority kind/id are durably pending before spawn; a stale Worker cannot replace its transition row or the winner's schedule, and the guardian - eventually repairs history. H has one contract-owned final path per effective + eventually repairs history. WP10 holds N from each retained native mutation through + its compatibility authorization; a newer native handoff replaces even a pending or + running older schedule, and the older Worker loses its terminal CAS. H has one contract-owned final path per effective user/canonical home/canonical state DB and takes only fail-fast `H -> N`; inverse - `N/K/C -> H` edges are forbidden. Only a missing manifest proves zero entries; + `N/K/C -> H` edges are forbidden. Only `missing` proves path absence; a present + valid matching v1 manifest is `ready` with a validated non-negative count, including + zero, while unreadable, malformed, unsupported, and foreign-state-DB manifests remain preserved, nullable, and non-converged. WP10 implements that protocol. Also contributes to C2/C12 (generation-guarded catalog commit plus the phase-specific catalog/full - admission and observation sequences). + admission and observation sequences). **Broken change:** acquire N only after the + retained native mutation, release N before compatibility authorization, restore the + terminal-only compatibility predicate, release H between surfaces, classify a + present ready-zero manifest as missing/unsupported, or commit catalog bytes after a + generation/evidence conflict; the handoff-order, terminal-CAS, serialization, + manifest, or stale-commit fixture fails respectively. diff --git a/devlog/_plan/260804_codex_write_substrate/020_history_isolation.md b/devlog/_plan/260804_codex_write_substrate/020_history_isolation.md index ae691efb5..748b81861 100644 --- a/devlog/_plan/260804_codex_write_substrate/020_history_isolation.md +++ b/devlog/_plan/260804_codex_write_substrate/020_history_isolation.md @@ -33,10 +33,22 @@ WP10 takes the reviewer's second option. Every **current high-level history operation** enters `history-job` and H in this phase. Existing apply, restore, explicit recovery, startup retry, CLI, service, and -management roots keep their current native/catalog orchestration, but none may call a +management roots keep their current native/catalog semantics, but none may call a history DB/manifest/rollout writer inline. WP10 persists the contract-owned typed `CodexHistoryOperation`, dispatches its identity to the Worker, and records its result. +The latest review exposed one ordering gap that cannot wait for WP11. Current apply +writes config/profile/journal before history (`src/codex/inject.ts:594-604`), and +restore changes config plus K-serialized catalog state before history +(`src/codex/inject.ts:765-789`). If compatibility authorization runs only after that +native root returns and refuses an older running schedule, a newer opposite native +mutation can be committed without any durable history repair operation. WP10 therefore +brings forward one narrow native exclusion: each retained native mutation and its +compatibility history authorization execute in one N-backed synchronous handoff. +N is acquired before the native callback, the newer schedule is published through the +same open transaction before the callback returns, and N is released before Worker +spawn/await. + The desired-state-driven `convergeCodex({ scope: "full" })` caller rewire is deferred to WP12. WP12 changes the producer of the durable history operation after full admission and native coordination exist; it reuses the same `history-job`, Worker, H, @@ -46,10 +58,11 @@ native receipt, or placeholder that a later phase replaces. This boundary is independently landable: WP10 typechecks while full convergence still rejects non-catalog requests, and current user-visible high-level operations preserve -their distinct history semantics through the Worker. +their distinct history semantics through the Worker. It consumes the already-landed N +path/transaction owner; it does not pre-implement WP11's full native lock. All current-code citations in this document were rechecked on 2026-08-05 at -`57c273922bdbd63e1b140811e8a07968928849ba`. +`45f7bb7caf9c836b9d9a398c76ae4cb2f7461860`. ## IN / OUT @@ -65,8 +78,11 @@ IN: state-DB path; consumers append no path segment. - `src/codex/transition-state.ts` (MODIFY) — persist/read the typed history operation and its operation identity, expose the history-specific schedule/claim/terminal CAS - used by current roots, and retain the operation for guardian restart. This is not a - producer of native generations or full authority snapshots. + used by current roots, retain the operation for guardian restart, and expose the + narrow synchronous N-backed compatibility handoff. The transaction-bound + compatibility authorizer may supersede an older terminal, pending, or running + schedule after the newer retained native mutation; this is not a producer of native + generations or full authority snapshots. - `src/codex/history-lock.ts` (NEW) — H: one cross-process, canonical-`CODEX_HOME`/effective-user keyed SQLite exclusion primitive using the contract resolver, finite acquisition, and no stale PID/mtime takeover. @@ -81,8 +97,10 @@ IN: terminal CAS, release H, and close. - `src/codex/history-job.ts` (NEW) — resolve explicit paths/options, schedule the typed operation durably as the sole root of the WP10 compatibility/explicit-recovery - authorizers, spawn/watch/join the Worker, classify IPC/death, and expose one async - entry point to every current high-level root. + authorizers, enter the N-backed compatibility handoff before invoking a retained + synchronous native callback, spawn/watch/join the Worker only after N releases, + classify IPC/death, and expose one async entry point to every current high-level + root. - `src/codex/inject.ts`, `src/codex/sync.ts`, `src/codex/history-migration-guardian.ts` (MODIFY) — preserve current native/catalog behavior but replace inline history calls with `history-job`; the guardian reads and @@ -111,7 +129,12 @@ OUT: - A full `convergeCodex`, `AdmissionSnapshot` capture, desired-state observer, provenance authorization, or the WP12 caller rewire. -- Native write exclusion, a native receipt, or a native `CommitExpectation` — WP11. +- WP11's full `codex-write-lock.ts` mechanism: uid/SID lock-namespace mechanics, + canonical target/admission validation, finite async acquisition/retry and typed + result API, `CommitExpectation`, provenance coordination, and broad adoption by + every native writer. WP10 takes only the already-owned N transaction as a + synchronous exclusion from each retained native mutation through compatibility + authorization. - Any claim that WP9 handed WP10 a working full funnel. It did not. - Replacing current native/catalog orchestration. In particular, `src/codex/inject.ts:775-780` is WP9's K-serialized catalog restore; it remains a @@ -167,13 +190,27 @@ inventing a native routing generation and without pretending to possess WP12's `BeginCodexTransition` with `{ kind: "admission-snapshot", id }`. `history-job.ts` is the sole WP10 compatibility adapter. The existing owning -apply/restore/recovery function derives the semantic operation from its internal -branch after its current native/catalog work; it passes that type to `history-job`, -not a user-controlled provider/direction. CLI, server, `inject.ts`, guardian, and -other helpers never import either transition-state authorizer directly. Compatibility -authority ids are fresh opaque nonces, never config/credential digests and never logs. -WP12 dispatches already-admitted schedules through the same job/Worker code and stops -using only the compatibility-authorization branch. +apply/restore function derives the semantic operation from its internal branch and +passes that type plus its synchronous native mutation callback to `history-job`, not +a user-controlled provider/direction. `history-job` acquires the transition-state +owner's N-backed handoff before invoking that callback and authorizes through the +closure bound to the same open transaction before it returns. Explicit recovery has +no native callback and retains its separate terminal-only authorizer. CLI, server, +`inject.ts`, guardian, and other helpers never import either transition-state +authorizer directly. Compatibility authority ids are fresh opaque nonces, never +config/credential digests and never logs. WP12 dispatches already-admitted schedules +through the same job/Worker code and stops using the compatibility-handoff branch. + +The compatibility handoff's CAS matches the complete row observed at N acquisition +and writes a fresh pending job/operation/authority even when the replaced state is +`pending` or `running`. Because N excluded both retained native callbacks, this is the +newer native operation by construction. Refusing non-terminal work here would recreate +the reviewed loss: B's opposite native bytes would already exist with no B schedule. +An older Worker remains free to finish its in-memory mutation under H, but its terminal +CAS includes the replaced identity and changes zero rows; B's pending schedule remains +for the guardian. `AuthorizeCodexLegacyHistoryRecovery` stays terminal-only because it +has no native mutation whose ordering would justify superseding unresolved native +repair. The Worker claim/terminal CAS matches native pair, job id, operation, and complete authority. A compatibility authority can never be relabeled as admission authority. @@ -239,18 +276,26 @@ The previous plan's statement that H and N are never held simultaneously was fal history surfaces. The actual edge is therefore: ```text -current high-level root: N(schedule typed operation) -> release N -> spawn Worker +current high-level root: N -> retained native mutation -> authorize typed operation + through same N -> commit/release N -> spawn Worker history Worker: H -> short N(read/claim) -> release N H -> mutate/probe H -> short N(terminal CAS) -> release N -> release H ``` -The checkable order is **H → N**. `N → H` is forbidden: scheduling commits and releases -N before spawn/await. H never enters K or the config-generation lock, and K/config -paths never enter H. The future WP11 native callback never spawns or awaits a Worker -while holding N; WP12 dispatches only after releasing native coordination. A busy N -claim/terminal attempt leaves the durable operation pending, releases H, and retries -later; it never waits indefinitely while retaining H. +The high-level line above now means: acquire N, execute the retained synchronous native +mutation, authorize its exact compatibility operation through the same open N handle, +then commit/release N before spawn. Where restore already uses catalog serialization, +that callback follows N → K; any nested config-generation guard remains K → C. The +checkable global order is still **H → N → K → C**. `N → H` is forbidden: scheduling +commits and releases N before spawn/await. H never enters K or the config-generation +lock, and K/config paths never enter H. WP11 later replaces this narrow compatibility +handoff with its full async native-lock API and broader adoption, but it preserves the +same release-before-dispatch rule. WP12 dispatches only after releasing native +coordination. A busy N claim/terminal attempt leaves the durable operation pending, +releases H, and retries later; it never waits indefinitely while retaining H. No cycle +appears because the only history edge enters N from H, while every N owner is forbidden +from acquiring or awaiting H. The lock-order contract test contains allowed fixtures for `H -> N` claim/terminal calls and forbidden fixtures for `N -> H`, `K -> H`, `H -> K`, and @@ -273,7 +318,8 @@ distinct states through mutation and post-probe: | Evidence | Classification | May certify zero entries? | |---|---|---| | manifest absent and DB readable | `missing`, `backupEntries: 0` | Yes, only after the DB probe also succeeds. | -| valid present v1 manifest for the requested DB | `ready` with at least one validated entry | Yes. | +| valid present matching v1 manifest with `entries: {}` | `ready`, present manifest, validated `backupEntries: 0` | Yes; it certifies a present empty file, not absence. | +| valid present matching v1 manifest with entries | `ready` with a validated positive count | Yes. | | malformed JSON | `malformed`, null count | No. | | readable unsupported version/shape | `unsupported`, null count | No. | | unreadable/permission failure | unreadable | No. | @@ -288,8 +334,16 @@ a foreign-DB manifest as empty and never deletes it; manifest-independent legacy consume the manifest at all. Fixtures cover malformed JSON, unreadable file, unsupported shape, wrong DB identity, -and missing DB with a nonempty backup. Reintroducing `catch { return emptyManifest }` -or `catch { backupEntries = 0 }` must turn each named fixture red. +and missing DB with a nonempty backup. The tenth absence-as-guarantee fixture also +seeds the accepted current shape `{ version: 1, stateDbPath, entries: {} }`: preflight +must return present `ready` zero, generic restore must take the empty-manifest branch +and eject residual routed rows (`src/codex/history-provider.ts:656-665`), and the +post-probe must still report present `ready` zero without changing the manifest bytes. +This plan uses one `ready` variant with a branded non-negative count because presence +is already carried by `manifest:T`; a separate `ready-empty` branch would add control +flow without preserving more evidence. Reintroducing `catch { return emptyManifest }`, +`catch { backupEntries = 0 }`, requiring a positive ready count, or mapping ready-zero +to missing/unsupported must turn its named fixture red. ## Writer reachability: one permitted production root @@ -374,7 +428,9 @@ The parent owns one process-local flight per durable operation identity only to duplicate Worker threads; H provides cross-process exclusion. Same-operation callers may join. A newer durable operation supersedes an older one: the old Worker either rejects it at the under-H claim or loses the terminal CAS, releases H, and leaves the -newer operation pending for repair. +newer operation pending for repair. For compatibility native roots, “newer” is not +arrival luck at the authorizer: N spans native mutation through authorization, so the +order of committed schedules is the order of retained native mutations. Outcome classification remains evidence-bearing: @@ -417,14 +473,29 @@ replaces the current sixty-tick terminal stop ### Cross-process all-surface serialization Seed a production-shaped DB, valid manifest, and rollouts under temporary homes. -Process A schedules one real operation and pauses after the first real rollout write -while retaining H. Process B schedules the opposite operation and reaches H. Resume A; -assert its terminal update cannot replace B's newer durable operation, H is released, -and B repairs manifest, rollouts, and DB. Reverse direction/order and repeat. No test -stub mutates a surface; both processes enter production `history-job`/Worker/H. - -Broken change: release H after the DB transaction but before manifest/rollout/probe, or -dispatch the writer without H. The sentinels interleave and the test fails. +Cover both orderings named by the latest review with production callbacks, not writer +stubs: + +1. A authorizes one operation, enters its Worker, and pauses after the first real + rollout write while retaining H. B acquires N, completes the opposite retained + native mutation, replaces A's `running` identity with B's durable pending schedule, + and reaches H. Resume A; its terminal update changes zero rows, H releases, and B + repairs manifest, rollouts, and DB. Reverse operation direction and repeat. +2. A acquires N, completes its retained native mutation, and pauses immediately before + the transaction-bound compatibility authorization. Start B at the point where it + would authorize first in the broken design. B must not enter its native callback or + publish any schedule while A owns N. Resume A so it authorizes and releases; only + then may B mutate native state and authorize the newer opposite operation. This is + the executable proof that authorization order matches native order. + +No test stub mutates a surface; both processes enter production +`history-job`/Worker/H and the second fixture checkpoints the real native callback. + +Broken changes: release H after the DB transaction but before manifest/rollout/probe; +dispatch the writer without H; restore the terminal-only compatibility predicate; or +acquire/release N only after/before the retained native callback. The sentinels +interleave, B's newer operation disappears, or B publishes first during A's handoff; +the terminal-CAS/final-state/order assertion fails. ### H namespace and lock order @@ -454,10 +525,14 @@ operation. The variant and tamper cases fail. Run the malformed, unreadable, unsupported-shape, wrong-DB, and missing-DB-with-backup fixtures through both the preflight and under-H post-probe. None may return numeric -zero/zero convergence. +zero/zero convergence. Separately run a valid present matching v1 `entries:{}` fixture +through preflight, generic restore/residual ejection, and post-probe; every read is +`ready` zero, the residual row is ejected, and the present manifest remains +byte-identical under the current empty branch. -Broken change: map any manifest read failure to an empty manifest or initialize an -unknown count to zero. Its named fixture fails. +Broken changes: map any manifest read failure to an empty manifest, initialize an +unknown count to zero, require `ready` to be positive, or map a present ready-zero file +to `missing`/`unsupported`. Its named fixture fails. ### Retry, death, and guardian activation @@ -517,11 +592,11 @@ the installed service and live proxy on 10100 remain untouched. |---|---|---| | **C3 — caller responsiveness** | Table-driven responsiveness covers management, init/setup, graceful shutdown, and explicit recovery while real SQLite contention overlaps health/SSE progress. | Move any listed root's probe or mutation back to the caller thread. That root's latency/progress case fails. | | **C4 — durable unresolved work** | Guardian activation/backoff test proves unresolved typed operation survives failure/restart and never becomes zero-looking success. | Remove startup arming, restore the 60-tick stop, or persist zero counts after failed evidence. The activation/backoff/evidence case fails. | -| **C15 — cross-process all-surface serialization** | Opposite operations serialize manifest, rollout, DB, post-probe, and terminal update under one H; newer durable operation repairs stale work. | Release H between surfaces or bypass H in one process. The sentinel/final-state case fails. | +| **C15 — cross-process all-surface serialization** | Opposite operations serialize manifest, rollout, DB, post-probe, and terminal update under one H; N spans each retained native mutation through compatibility authorization, the newer schedule replaces even running work, and its Worker repairs stale work. | Release H between surfaces, bypass H, release/acquire N inside the native-to-authorization span, or restore terminal-only authorization. The sentinel/order/final-state case fails. | | **Operation authority** | Every operation variant is derived/validated from durable state, including no-op and manifest-independent recovery. | Trust request `targetProvider`/direction. Tamper and manifest-preservation cases fail. | -| **Real lock order** | Architecture fixtures allow `H -> N` and reject every inverse/cross-domain edge. | Await/spawn H while N is held, or call K from the Worker. Dependency fixture fails. | +| **Real lock order** | Architecture fixtures allow `H -> N -> K -> C`; the compatibility root may execute retained native/K/C work and authorize through already-held N, then must release N before dispatch. Every inverse/cross-domain edge remains rejected. | Await/spawn H while N is held, open N only after native mutation, release N before authorization, or call K from the Worker. Dependency/order fixture fails. | | **One H namespace per canonical history DB** | Environment-divergent child processes resolve one H for the same effective user/home/DB, a different H for a second DB, and paths distinct from N/K. | Key by environment/raw alias, omit DB identity, or reuse N/K. Resolver equality/inequality case fails. | -| **Manifest evidence** | Malformed, unreadable, unsupported, wrong-DB, and missing-DB-with-backup fixtures remain non-converged with nullable unknown counts. | Convert any failed manifest read to empty/zero. Its fixture fails. | +| **Manifest evidence** | Malformed, unreadable, unsupported, wrong-DB, and missing-DB-with-backup fixtures remain non-converged with nullable unknown counts; a valid present matching v1 empty manifest is `ready` zero through preflight, restore/ejection, and post-probe. | Convert any failed read to empty/zero, require ready-positive, or classify present ready-zero as missing/unsupported. Its fixture fails. | | **No writer bypass** | TypeScript symbol reachability permits only `history-worker.ts` as a production writer root and catches wrappers, aliases, re-exports, namespace and dynamic imports. | Add any direct/indirect production writer path. Graph test fails. | | **Complete current caller routing** | Inventory covers every production command/route and each history-bearing row reaches `runCodexHistoryJob`; named disconnect mutations are red. | Remove init, guardian, shutdown, service, or any other row's job edge, or add an unlisted caller. Inventory test fails. | -| **N2 — independently landable** | WP10 typechecks and focused/full suites pass while executable convergence remains catalog-only; current operation semantics are preserved through H. | Import/call full `convergeCodex`, require WP11 native lock/receipt, or leave a temporary inline path. Typecheck/routing behavior test fails at the WP10 commit. | +| **N2 — independently landable** | WP10 typechecks and focused/full suites pass while executable convergence remains catalog-only; current operation semantics are preserved through the N-backed compatibility handoff and H. | Import/call full `convergeCodex`, require WP11's full async lock/receipt/`CommitExpectation`, omit the narrow N handoff, or leave a temporary inline path. Typecheck/routing/order test fails at the WP10 commit. | From 9e405e2b46f668c81141ffdb6ecc776841c7761a Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Wed, 5 Aug 2026 03:56:45 +0900 Subject: [PATCH 085/163] docs(substrate): the coordinator has no production caller, so no row exists yet MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two more from the review, and the second one is the eleventh time this unit has read an absence as a guarantee. The authorizer could not be implemented safely. It required the transition row as `expected`, but the handoff gave its callback only the authorizer function, never the row observed once N was held. That left exactly two options: read the row before taking N, which lets A move it in between and puts B back in the lost-operation window this phase just closed; or call the reader inside the callback, which opens a second BEGIN IMMEDIATE and self-contends with the transaction already held. The owner now hands down a one-shot `authorize(next)` bound to the row it read on the open handle, so neither escape exists. Then the harder one. WP10 takes N before each retained native mutation — but `transition-state.ts` has no production caller at all today, only tests, and its initializer deliberately refuses whenever routed residue is present. So a real installation upgrading to this has routed config, catalog and history, no coordinator row, and a handoff that needs N before restore or apply: refusal happens before the native callback ever runs, and `ocx stop`, restore and startup sync stop working. The text had assumed that because the coordinator shipped, a durable row must exist. Nothing was ever going to create one. Adoption is therefore explicit and narrow: only the real retained apply and restore callbacks, only for a genuinely absent coordinator database with a valid or absent integration record and conclusively routed residue, and only into `{generation: 0, pending/wp10-compatibility}`. Observation, the Worker, the guardian and recovery cannot request it. The general initializer keeps refusing routed or indeterminate residue, legacy JSON, and an existing unversioned or rowless database — that last exclusion matters, because widening adoption to cover it would reopen the guard from the inside. The fixtures start where real users are: routed on disk, no coordinator. --- .../005_contract.md | 168 ++++++++++++++---- .../020_history_isolation.md | 107 +++++++++-- 2 files changed, 225 insertions(+), 50 deletions(-) diff --git a/devlog/_plan/260804_codex_write_substrate/005_contract.md b/devlog/_plan/260804_codex_write_substrate/005_contract.md index faf82a7e3..619482dcb 100644 --- a/devlog/_plan/260804_codex_write_substrate/005_contract.md +++ b/devlog/_plan/260804_codex_write_substrate/005_contract.md @@ -12,7 +12,7 @@ four phase docs are rewritten as consumers against the reviewer's section list. A contract nobody collected is a fifth opinion. All current-code citations in this document were rechecked on 2026-08-05 at -`45f7bb7caf9c836b9d9a398c76ae4cb2f7461860`. +`b8168510f32ef583f14660067e0783d88df5df0c`. ## IN / OUT @@ -375,14 +375,28 @@ export type AuthorizeCodexLegacyHistoryRecovery = ( }>, ) => TransitionStateUpdate; +/** + * Positive WP10 authority for the one missing-row exception. This is derived + * only by `history-job.ts` from a real retained high-level native callback. + */ +export type CodexCompatibilityNativeIntent = + | Readonly<{ + kind: "retained-apply"; + operation: Exclude; + }> + | Readonly<{ + kind: "retained-restore"; + operation: "restore-openai"; + }>; + /** * WP10-only bridge for current roots that predate WP12 native admission. The - * transition-state owner supplies this closure only inside the already-open N - * transaction that excluded the retained native mutation. It publishes the - * newer history work without moving the native routing pair. + * transition-state owner binds this one-shot closure to both the already-open N + * handle and the complete row read from that handle after acquisition. Callers + * supply no expected row and cannot make the closure open another N connection. */ export type AuthorizeCodexCompatibilityHistory = ( - expected: CodexTransitionState, next: Readonly<{ jobId: string; operation: Exclude; @@ -399,6 +413,7 @@ type SynchronousCompatibilityHandoff = T extends PromiseLike ? never * transaction-bound closure, and returns before N is committed and released. */ export type WithCodexCompatibilityNativeHandoff = ( + intent: CodexCompatibilityNativeIntent, mutateNativeAndAuthorize: ( authorize: AuthorizeCodexCompatibilityHistory, ) => SynchronousCompatibilityHandoff, @@ -415,10 +430,13 @@ WP8b implements and exports `const readIntegrationRecord: ReadIntegrationRecord` and `const updateIntegrationRecord: UpdateIntegrationRecord` from `src/codex/integration-record.ts`, plus `readCodexTransitionState`, `beginCodexTransition`, and -`authorizeCodexCompatibilityHistory`, `authorizeCodexLegacyHistoryRecovery`, and `updateCodexHistoryTransition` from `src/codex/transition-state.ts`; these are executable functions in that phase, not -ambient declarations. +ambient declarations. WP10 adds `withCodexCompatibilityNativeHandoff` and +`authorizeCodexLegacyHistoryRecovery` there. There is deliberately no standalone +runtime `authorizeCodexCompatibilityHistory(expected, next)` export: the only value +of that type is the one-shot closure supplied by the handoff while its N transaction +is live. The coordinator is a **sibling**, not an extension of `config-mutation.sqlite`. The existing database path is derived from `getConfigDir()` @@ -557,15 +575,23 @@ schedule, and leave A free to terminally record the obsolete direction. WP10 therefore brings forward exactly one piece of WP11: the N-backed compatibility native-handoff exclusion. `history-job.ts` enters `withCodexCompatibilityNativeHandoff` **before** invoking a retained synchronous -native mutation, receives an authorizer closure bound to that already-open N handle, -and uses it before returning from the callback. The transaction then commits/releases -N before any Worker spawn or await. Current apply/restore owners still derive one of +native mutation, receives an authorizer closure bound to that already-open N handle +and the complete row read from that handle after `BEGIN IMMEDIATE`, and uses it before +returning from the callback. `authorize(next)` has no `expected` parameter and may be +called once; a second call, callback return without one call, or attempted use after +the callback is a transaction error and rolls back. It reads no coordinator state and +opens no SQLite connection. This bound shape was chosen because passing `expected` +would let a caller retain a pre-N row, while re-reading through +`readCodexTransitionState` would open another `BEGIN IMMEDIATE` and contend with its +own transaction (`src/codex/transition-state.ts:473-489`). The transaction then +commits/releases N before any Worker spawn or await. Current apply/restore owners still derive one of `skip | apply-opencodex | migrate-openai | restore-openai`; they supply the derived -operation and synchronous native callback to `history-job.ts`, never call the -transition-state API directly, and never move native generation in WP10. +operation as the matching `CodexCompatibilityNativeIntent` and supply the synchronous +native callback to `history-job.ts`; they never call the transition-state API directly +and never move native generation in WP10. -That transaction-bound `authorizeCodexCompatibilityHistory` replaces the exact row -observed when N was acquired with a fresh pending job id, +That transaction-bound `authorize` closure replaces the exact row it captured from +the already-open transaction with a fresh pending job id, `authority.kind:"wp10-compatibility"`, and fresh opaque authority id **regardless of whether the older history state is terminal, pending, or running**. N exclusion makes that authorization order the retained native-mutation order; replacing non-terminal @@ -608,20 +634,51 @@ winner's timer, and schedule from the row returned by a fresh read. A zero-row guardian update means its timer was stale and is replaced from the current row. Database busy/unavailable is typed `busy`/`deferred`; no caller guesses success. -Initialization first verifies the no-legacy/native-clean precondition while the -native lock excludes another initializer, then uses one `BEGIN IMMEDIATE` -transaction: create the table, then -`INSERT OR IGNORE` singleton 1 as `{0,null}` with an `unknown` history observation, -zero attempts and no job/operation/authority/timer/counts, and sets -`PRAGMA user_version = 1`. That initialization is legal only when the -JSON has no legacy `nativeGeneration`, `currentTxId`, `generation`, or durable -`history` member and native observation finds no unresolved routed residue. When -the row is absent, any such legacy field or native residue is `legacy-ambiguous`; automatic -mutation refuses and explicit salvage/native-clean adoption must establish the row. -An `OPENCODEX_HOME`-local positive pair is never imported because a second home may -hold a different claimant. Once the row exists, legacy JSON fields have no authority -and are removed on the next successful non-CAS record update while all unrelated -unknown keys survive. +The eleventh absence-as-guarantee review found that this strict initializer was the +only production behavior available. The current owner refuses routed residue before +inserting singleton 1 (`src/codex/transition-state.ts:263-303`), and the real routed- +catalog fixture proves the missing coordinator returns `legacy-ambiguous` +(`tests/codex-native-residue.test.ts:213-242`). A production-reference audit also +finds no caller of `readCodexTransitionState`, `beginCodexTransition`, or +`openCodexCoordinatorTransaction` outside the transition owner; the executable entry +points at `src/codex/transition-state.ts:348-385,473-518` are reached only by tests +today. Shipping the owner therefore did not initialize existing routed installations. + +The **general initializer remains unchanged and fail-closed**. Read/observe, +`BeginCodexTransition`, Worker claim/terminal work, guardian retry, explicit legacy +recovery, and direct transaction opens may create the ordinary unscheduled `{0,null}` +row only when the JSON record is missing or valid with no legacy transition fields +and every native surface is clean. Invalid/legacy JSON, routed residue, indeterminate +native evidence, an existing unversioned/unsupported database, or an existing +database with no authoritative singleton still refuses. The residue classifier +checks every routed surface and returns any `indeterminate` result before `residue` +(`src/codex/native-residue.ts:520-556`), so unreadable or ambiguous bytes cannot enter +the exception. An `OPENCODEX_HOME`-local positive pair is never imported because a +second home may hold a different claimant. + +WP10 adds one private **compatibility-adoption** mode beneath +`withCodexCompatibilityNativeHandoff`; it is not an option on the public transaction +opener. “Positively authorized” means all of the following are true: the sole graph- +permitted caller is `history-job.ts`; it received a real retained high-level native +callback; its closed intent is `retained-apply` with exactly +`skip | apply-opencodex | migrate-openai` or `retained-restore` with exactly +`restore-openai`; and the callback must consume the transaction-bound authorizer +exactly once. Residue detection, startup observation, a Worker/guardian retry, +history-only recovery, or an arbitrary operation value is not positive authority. + +When and only when that handoff finds a truly absent coordinator database, a missing +or valid non-legacy integration record, and positively classified routed residue, it +acquires `BEGIN IMMEDIATE` and captures **authoritative row absence plus that routed +observation** without installing `{0,null,unknown}`. The retained native callback then +runs. Its bound `authorize(next)` requires `next.operation === intent.operation` and, +on the same already-open handle, creates the schema and inserts singleton 1 directly +as generation zero, null txId, and the fresh complete +`pending/wp10-compatibility` schedule; it then sets `user_version = 1`. Callback or +authorization failure rolls the transaction back and dispatches no Worker. This is a +compatibility schedule over evidence the current high-level operation just took +responsibility for, not an empty baseline inferred from residue. Once the row exists, +legacy JSON fields have no authority and are removed on the next successful non-CAS +record update while all unrelated unknown keys survive. A missing JSON file is valid and the first provenance update creates `{version:1}`. Unreadable/unparseable JSON is not empty: provenance mutation fails closed. Unknown @@ -1975,7 +2032,7 @@ these history scheduling edges: | WP10 transitional root | Permitted transition-state symbols | Required authority | |---|---|---| -| `src/codex/history-job.ts` | `authorizeCodexCompatibilityHistory`, `authorizeCodexLegacyHistoryRecovery` | `wp10-compatibility` or `explicit-legacy-recovery`; never `admission-snapshot` | +| `src/codex/history-job.ts` | `withCodexCompatibilityNativeHandoff`, `authorizeCodexLegacyHistoryRecovery` | bound `wp10-compatibility` closure or `explicit-legacy-recovery`; never `admission-snapshot` | | `src/codex/history-worker.ts` | `readCodexTransitionState`, `updateCodexHistoryTransition` | exact row-copied native pair, job, operation, and authority | `history-job.ts` receives the convergence-derived operation; it does not derive from @@ -2054,10 +2111,37 @@ When called through the N-bound handoff after a newer retained native mutation, must replace terminal, pending, and running older schedules with a fresh pending identity. The old Worker's terminal CAS must then change zero rows. Its authority id must not equal or derive from config/credential bytes and must not reach logs, JSON, -responses, or exceptions. -Missing DB/table initializes only from native-clean/no-legacy state; -legacy JSON pair/schedule, residue beside a missing row, malformed row, busy DB and -unsafe path all fail closed with the specified typed outcome. +responses, or exceptions. Give B a deliberately stale complete row read before N, +let A publish a newer schedule, then enter B's handoff. B's callback retains that +stale object but has nowhere to pass it: the bound authorizer must use A's complete +row read from B's already-open N handle and publish B. Instrument coordinator +connection creation so any second handle throws; the handoff still succeeds with one +connection. **Broken changes:** restore an `expected` argument, call +`readCodexTransitionState()` inside the callback, or let the authorizer open another +coordinator connection; B conflicts after its native mutation or the second-handle +trap fires. +The general missing-DB/table path initializes only from native-clean/no-legacy state; +outside the explicit compatibility-adoption fixture, legacy JSON pair/schedule, +residue beside a missing row, malformed row, busy DB and unsafe path all fail closed +with the specified typed outcome. + +`tests/codex-native-residue.test.ts`: add two compatibility-adoption fixtures that +begin with routed config, routed catalog, routed history rows/rollouts/manifest, and +**no coordinator database** under temporary homes. The apply fixture enters the real +`injectCodexConfig` high-level path +(`src/codex/inject.ts:482-654`); the restore fixture enters real +`restoreNativeCodex` (`src/codex/inject.ts:765-800`). Each must acquire N before its +retained callback and commit generation zero with the exact pending compatibility +operation before Worker dispatch; restore remains authorized even though its native +callback removes the routed config/catalog residue captured at N acquisition. The +Worker then repairs history and terminally owns the same schedule. Run matching +negative fixtures for observe/read, guardian/Worker retry, explicit recovery, +operation/intent mismatch, invalid legacy JSON, indeterminate residue, and an existing +unversioned or rowless database; none may create a row or invoke the native callback. +**Broken changes:** route the handoff through the strict clean-only initializer, let +mere residue detection opt into adoption, insert an unscheduled `{0,null}` row first, +or authorize restore from a post-callback clean observation; the real apply/restore +fixture returns `legacy-ambiguous` or the named negative fixture creates authority. `tests/codex-convergence-contract.test.ts`: every `ConvergeOutcome` variant maps to the §5 row, `busy` carries `Retry-After`, and a best-effort management caller @@ -2249,6 +2333,12 @@ native callback returns**, and **restore the terminal-only compatibility predica the attempted inversion publishes B first or the running-schedule case drops B, and the final-state/terminal-CAS assertions fail. +The second ordering also carries B's deliberately stale pre-N row and a connection- +creation trap. B must authorize from the complete row read on its one already-open N +handle after A releases. Adding a caller-supplied expected row makes B conflict after +its real native write; opening `readCodexTransitionState` or any second N connection +self-contends and trips the connection assertion. + Hold H in one process and prove a second service/CLI Worker for the same canonical home/state DB cannot enter manifest, rollout, DB, probe, or terminal-CAS work. While H is held, execute the real fail-fast claim read and terminal update and observe @@ -2336,8 +2426,12 @@ writes to it. preserved, nullable, and non-converged. WP10 implements that protocol. Also contributes to C2/C12 (generation-guarded catalog commit plus the phase-specific catalog/full admission and observation sequences). **Broken change:** acquire N only after the - retained native mutation, release N before compatibility authorization, restore the - terminal-only compatibility predicate, release H between surfaces, classify a + retained native mutation, release N before compatibility authorization, restore a + caller-supplied expected row or second-connection read, route an existing routed + installation through the strict clean-only initializer, let observation/residue + alone create a generation-zero row, restore the terminal-only compatibility + predicate, release H between surfaces, classify a present ready-zero manifest as missing/unsupported, or commit catalog bytes after a - generation/evidence conflict; the handoff-order, terminal-CAS, serialization, - manifest, or stale-commit fixture fails respectively. + generation/evidence conflict; the stale-row/one-handle, routed apply/restore, + negative-adoption, handoff-order, terminal-CAS, serialization, manifest, or + stale-commit fixture fails respectively. diff --git a/devlog/_plan/260804_codex_write_substrate/020_history_isolation.md b/devlog/_plan/260804_codex_write_substrate/020_history_isolation.md index 748b81861..6eacb9d45 100644 --- a/devlog/_plan/260804_codex_write_substrate/020_history_isolation.md +++ b/devlog/_plan/260804_codex_write_substrate/020_history_isolation.md @@ -62,16 +62,16 @@ their distinct history semantics through the Worker. It consumes the already-lan path/transaction owner; it does not pre-implement WP11's full native lock. All current-code citations in this document were rechecked on 2026-08-05 at -`45f7bb7caf9c836b9d9a398c76ae4cb2f7461860`. +`b8168510f32ef583f14660067e0783d88df5df0c`. ## IN / OUT IN: - `src/codex/convergence-types.ts` (MODIFY) — materialize the - contract-owned `CodexHistoryOperation`, typed manifest-read result, and durable - history-operation schedule/result shapes from `005_contract.md`; do not define a - WP10-local provider/direction union. + contract-owned `CodexHistoryOperation`, closed compatibility-native intent, typed + manifest-read result, and durable history-operation schedule/result shapes from + `005_contract.md`; do not define a WP10-local provider/direction union. - `src/codex/user-identity.ts` (MODIFY) — add `resolveCodexHistorySerializationDatabasePath` beside the existing N and K resolvers. It keys H by effective user, canonical `CODEX_HOME`, and canonical @@ -79,10 +79,12 @@ IN: - `src/codex/transition-state.ts` (MODIFY) — persist/read the typed history operation and its operation identity, expose the history-specific schedule/claim/terminal CAS used by current roots, retain the operation for guardian restart, and expose the - narrow synchronous N-backed compatibility handoff. The transaction-bound + narrow synchronous N-backed compatibility handoff plus its private routed-install + adoption mode. The transaction-bound, one-shot compatibility authorizer may supersede an older terminal, pending, or running - schedule after the newer retained native mutation; this is not a producer of native - generations or full authority snapshots. + schedule after the newer retained native mutation and receives no caller-supplied + expected row; this is not a producer of native generations or full authority + snapshots. - `src/codex/history-lock.ts` (NEW) — H: one cross-process, canonical-`CODEX_HOME`/effective-user keyed SQLite exclusion primitive using the contract resolver, finite acquisition, and no stale PID/mtime takeover. @@ -97,8 +99,9 @@ IN: terminal CAS, release H, and close. - `src/codex/history-job.ts` (NEW) — resolve explicit paths/options, schedule the typed operation durably as the sole root of the WP10 compatibility/explicit-recovery - authorizers, enter the N-backed compatibility handoff before invoking a retained - synchronous native callback, spawn/watch/join the Worker only after N releases, + authorizers, derive the closed retained-apply/retained-restore adoption intent, + enter the N-backed compatibility handoff before invoking a retained synchronous + native callback, spawn/watch/join the Worker only after N releases, classify IPC/death, and expose one async entry point to every current high-level root. - `src/codex/inject.ts`, `src/codex/sync.ts`, @@ -118,6 +121,7 @@ IN: read-only probe; unavailable evidence remains unknown. - `tests/codex-history-provider.test.ts`, `tests/codex-transition-state.test.ts`, + `tests/codex-native-residue.test.ts`, `tests/codex-convergence-contract.test.ts`, `tests/history-migration-guardian.test.ts`, `tests/codex-sync-api.test.ts`, and `tests/shutdown-drain.test.ts` (MODIFY), plus @@ -194,10 +198,16 @@ apply/restore function derives the semantic operation from its internal branch a passes that type plus its synchronous native mutation callback to `history-job`, not a user-controlled provider/direction. `history-job` acquires the transition-state owner's N-backed handoff before invoking that callback and authorizes through the -closure bound to the same open transaction before it returns. Explicit recovery has -no native callback and retains its separate terminal-only authorizer. CLI, server, +one-shot `authorize(next)` closure bound to the complete row read from the same open +transaction after N acquisition. The closure has no `expected` parameter, performs no +read, and opens no second connection. This shape is mandatory: a pre-N expected row +can become stale before N is acquired, while `readCodexTransitionState()` opens +another `BEGIN IMMEDIATE` (`src/codex/transition-state.ts:473-489`) and would +self-contend against the handoff. Explicit recovery has no native callback and retains +its separate terminal-only authorizer. CLI, server, `inject.ts`, guardian, and other helpers never import either transition-state -authorizer directly. Compatibility authority ids are fresh opaque nonces, never +authorizer directly. The handoff rejects zero or multiple authorization calls and +use after callback return. Compatibility authority ids are fresh opaque nonces, never config/credential digests and never logs. WP12 dispatches already-admitted schedules through the same job/Worker code and stops using the compatibility-handoff branch. @@ -212,6 +222,41 @@ for the guardian. `AuthorizeCodexLegacyHistoryRecovery` stays terminal-only beca has no native mutation whose ordering would justify superseding unresolved native repair. +The eleventh recurrence of this unit's absence-as-guarantee defect is at installation +adoption. No production caller currently initializes N: the exported transaction/read/ +begin implementations are at `src/codex/transition-state.ts:348-385,473-518`, while +the production tree has no external reference to them. The strict initializer refuses +all routed residue before inserting the singleton +(`src/codex/transition-state.ts:263-303`), and the real routed-catalog fixture proves +that a missing coordinator is `legacy-ambiguous` +(`tests/codex-native-residue.test.ts:213-242`). Existing routed installations would +therefore fail before apply or restore reached their retained callback. + +WP10 preserves that general guard. Ordinary reads, Worker/guardian paths, explicit +recovery, `BeginCodexTransition`, and direct transaction opens may initialize only a +native-clean, non-legacy installation. They continue to refuse invalid/legacy JSON, +routed or indeterminate native evidence, unsupported/unversioned coordinator files, +and existing databases with no row. The compatibility handoff alone receives a +private adoption mode. “Positively authorized” means its sole permitted graph root is +`history-job.ts`, it is paired with the real retained synchronous native callback, +and its closed intent is exactly `retained-apply` with +`skip | apply-opencodex | migrate-openai` or `retained-restore` with +`restore-openai`. Mere residue detection, observation, retry, history-only recovery, +or an arbitrary operation cannot request adoption. + +For a truly absent coordinator, valid/missing non-legacy integration record, and a +positive routed classification, that mode takes `BEGIN IMMEDIATE`, captures row +absence plus the routed observation, and does **not** install the ordinary unscheduled +`{0,null,unknown}` row. The residue classifier evaluates every surface and gives any +indeterminate result precedence (`src/codex/native-residue.ts:520-556`). After the +native callback, the same one-shot `authorize(next)` requires the intent/operation to +match and uses the already-open handle to create schema plus generation-zero/null-txId +`pending/wp10-compatibility` schedule in one transaction. This remains valid for +restore after the callback has removed config/catalog residue because the positive +routed observation was captured under N before mutation. Failure rolls back and +dispatches no Worker. An existing malformed/unversioned/rowless coordinator is never +adopted, and no `OPENCODEX_HOME`-local pair is imported. + The Worker claim/terminal CAS matches native pair, job id, operation, and complete authority. A compatibility authority can never be relabeled as admission authority. This is one contract row and one terminal protocol, not a WP10-private store. @@ -297,6 +342,13 @@ releases H, and retries later; it never waits indefinitely while retaining H. No appears because the only history edge enters N from H, while every N owner is forbidden from acquiring or awaiting H. +“Through the same open N handle” is a connection-counted invariant, not shorthand for +re-entering the owner. The handoff reads the complete existing row once after +acquisition and closes over it; the adoption case closes over authoritative absence +plus the pre-mutation routed observation. The callback can retain any stale pre-N +observation, but no API accepts it. Any `readCodexTransitionState` call or other +coordinator connection created inside the callback is a test failure. + The lock-order contract test contains allowed fixtures for `H -> N` claim/terminal calls and forbidden fixtures for `N -> H`, `K -> H`, `H -> K`, and config-lock-to-H edges. Reversing the schedule/spawn order to await the Worker while N @@ -497,6 +549,33 @@ acquire/release N only after/before the retained native callback. The sentinels interleave, B's newer operation disappears, or B publishes first during A's handoff; the terminal-CAS/final-state/order assertion fails. +### Transaction-observed authority and compatibility adoption + +Before B acquires N, give it a deliberately stale copy of the complete transition +row. Let A publish a newer pending schedule and release N, then run B's real retained +native callback through the handoff. B must publish over A using the row read from its +already-open handle; the stale object is not accepted by any call. Instrument +coordinator connection creation so a second handle throws. **Broken changes:** add an +`expected` parameter to `authorize`, call `readCodexTransitionState` in the callback, +or open another coordinator connection; B conflicts after its native mutation or the +connection trap fires. + +In `tests/codex-native-residue.test.ts`, under temporary homes, seed routed config, +catalog, history DB rows, rollouts, and a matching manifest but no coordinator +database. Run one real high-level apply through +`injectCodexConfig` (`src/codex/inject.ts:482-654`) and one real high-level restore +through `restoreNativeCodex` (`src/codex/inject.ts:765-800`). Each must commit the +exact generation-zero pending compatibility schedule before Worker dispatch, and its +Worker must repair/terminally own that identity. The restore fixture proves adoption +does not depend on residue still being present after the native callback. Table-drive +negative observe/read, guardian/Worker retry, explicit recovery, intent/operation +mismatch, invalid legacy JSON, indeterminate residue, and existing unversioned/rowless +database cases; they create no row and do not invoke the native callback. **Broken +changes:** send compatibility roots through the strict clean-only initializer, let +residue alone authorize adoption, commit `{0,null,unknown}` before the schedule, or +re-observe only post-restore clean state; the real fixture returns +`legacy-ambiguous`/loses its schedule or a negative fixture creates authority. + ### H namespace and lock order Two child processes vary `HOME`, `USERPROFILE`, `TMPDIR`, `XDG_RUNTIME_DIR`, `TEMP`, @@ -574,7 +653,7 @@ change. ```bash bun run typecheck bun test tests/codex-history-provider.test.ts tests/codex-history-worker.test.ts -bun test tests/codex-transition-state.test.ts tests/history-migration-guardian.test.ts +bun test tests/codex-transition-state.test.ts tests/codex-native-residue.test.ts tests/history-migration-guardian.test.ts bun test tests/codex-history-process-routing.test.ts tests/codex-convergence-contract.test.ts bun test tests/codex-sync-api.test.ts tests/shutdown-drain.test.ts bun test tests/codex-history-worker-responsive.test.ts --timeout 30000 @@ -593,6 +672,8 @@ the installed service and live proxy on 10100 remain untouched. | **C3 — caller responsiveness** | Table-driven responsiveness covers management, init/setup, graceful shutdown, and explicit recovery while real SQLite contention overlaps health/SSE progress. | Move any listed root's probe or mutation back to the caller thread. That root's latency/progress case fails. | | **C4 — durable unresolved work** | Guardian activation/backoff test proves unresolved typed operation survives failure/restart and never becomes zero-looking success. | Remove startup arming, restore the 60-tick stop, or persist zero counts after failed evidence. The activation/backoff/evidence case fails. | | **C15 — cross-process all-surface serialization** | Opposite operations serialize manifest, rollout, DB, post-probe, and terminal update under one H; N spans each retained native mutation through compatibility authorization, the newer schedule replaces even running work, and its Worker repairs stale work. | Release H between surfaces, bypass H, release/acquire N inside the native-to-authorization span, or restore terminal-only authorization. The sentinel/order/final-state case fails. | +| **Transaction-observed authority** | A stale pre-N row cannot be supplied to the one-shot authorizer; B authorizes from the complete row read on its one already-open N handle. | Restore `authorize(expected, next)`, call `readCodexTransitionState` inside the callback, or open a second N connection. The stale-row or connection-count case fails after B's real native mutation. | +| **Compatibility adoption** | Real apply and restore fixtures start with routed config/catalog/history and no coordinator DB, then commit an exact generation-zero pending compatibility schedule; every non-high-level or ambiguous case remains refused. | Route the handoff through strict clean-only initialization, let residue/observation/retry request adoption, insert an unscheduled row first, or require post-callback residue. The real routed fixture or its named negative row fails. | | **Operation authority** | Every operation variant is derived/validated from durable state, including no-op and manifest-independent recovery. | Trust request `targetProvider`/direction. Tamper and manifest-preservation cases fail. | | **Real lock order** | Architecture fixtures allow `H -> N -> K -> C`; the compatibility root may execute retained native/K/C work and authorize through already-held N, then must release N before dispatch. Every inverse/cross-domain edge remains rejected. | Await/spawn H while N is held, open N only after native mutation, release N before authorization, or call K from the Worker. Dependency/order fixture fails. | | **One H namespace per canonical history DB** | Environment-divergent child processes resolve one H for the same effective user/home/DB, a different H for a second DB, and paths distinct from N/K. | Key by environment/raw alias, omit DB identity, or reuse N/K. Resolver equality/inequality case fails. | From 8a8323f7885da957120218495715cab8593a5d66 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Wed, 5 Aug 2026 04:17:06 +0900 Subject: [PATCH 086/163] docs(substrate): observing ENOENT is not the same as having created the file MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The adoption path refused an existing unversioned or rowless coordinator, but never said how the opener proves it created the file it calls absent. The owner lstats, remembers ENOENT (transition-state.ts:356), and opens with create:true much later (:374). So: A sees ENOENT and pauses, B creates the file and dies before committing a schema, A opens B's leftover, takes its transaction, finds no row, and adopts on the strength of an observation that expired. Row absence was never the proof — a half-initialized database has no row either. The abort path was worse, and I ran it: create, BEGIN IMMEDIATE, create a table, ROLLBACK, close, and the file is still there at zero bytes with user_version 0. A failed adoption therefore left exactly the artifact that makes every later attempt refuse. The installation wedges permanently, and the fix for the guard would have been the thing that broke it. Creation is now an O_CREAT|O_EXCL claim taken before SQLite opens, so only the provable creator may adopt and EEXIST always falls to the strict existing-database path — including a file that appears after an earlier ENOENT. The winner opens readwrite without create, and the preliminary lstat is demoted to safety evidence rather than authority. On abort the creator keeps its exclusive descriptor, rolls back, closes, verifies descriptor and path identity, and unlinks only its own uncommitted file. That restores genuine absence so the next operation can claim it. If the path was substituted or the unlink fails, it is a typed fail-closed error and never adoption authority — cleanup that cannot prove what it is deleting is how this family of bugs starts. Twelfth instance of the same pattern, and the reviewer noted the sharp part: the existing "rowless database already exists" fixture passes with this race present, so it was never evidence for the case it appeared to cover. --- .../005_contract.md | 101 ++++++++++++++---- .../020_history_isolation.md | 89 ++++++++++++--- 2 files changed, 157 insertions(+), 33 deletions(-) diff --git a/devlog/_plan/260804_codex_write_substrate/005_contract.md b/devlog/_plan/260804_codex_write_substrate/005_contract.md index 619482dcb..297ad5290 100644 --- a/devlog/_plan/260804_codex_write_substrate/005_contract.md +++ b/devlog/_plan/260804_codex_write_substrate/005_contract.md @@ -12,7 +12,7 @@ four phase docs are rewritten as consumers against the reviewer's section list. A contract nobody collected is a fifth opinion. All current-code citations in this document were rechecked on 2026-08-05 at -`b8168510f32ef583f14660067e0783d88df5df0c`. +`9e405e2b46f668c81141ffdb6ecc776841c7761a`. ## IN / OUT @@ -410,7 +410,9 @@ type SynchronousCompatibilityHandoff = T extends PromiseLike ? never /** * WP10's narrow native-mutation-to-history-authorization exclusion. The callback * starts only after N is held, must authorize exactly once through the supplied - * transaction-bound closure, and returns before N is committed and released. + * transaction-bound closure, and returns before N is committed and released. A + * compatibility adoption additionally requires the opener's still-live exclusive + * creation claim; an earlier path-absence observation is never authority. */ export type WithCodexCompatibilityNativeHandoff = ( intent: CodexCompatibilityNativeIntent, @@ -644,6 +646,22 @@ finds no caller of `readCodexTransitionState`, `beginCodexTransition`, or points at `src/codex/transition-state.ts:348-385,473-518` are reached only by tests today. Shipping the owner therefore did not initialize existing routed installations. +The twelfth absence-as-guarantee review found that “truly absent” was still a +check-then-open claim. The owner records `ENOENT` from `lstatSync` +(`src/codex/transition-state.ts:356-373`), later opens with `create:true`, and passes +the stale boolean into initialization (`src/codex/transition-state.ts:374-385`). A +second process can create an unversioned/rowless file between those steps, yet the +first process treats it as its own new database. WP10 therefore changes the opener's +behavior, not its public `openCodexCoordinatorTransaction(finalDatabasePath)` +signature: before any SQLite open, it attempts an OS-level no-clobber +`O_CREAT | O_EXCL | O_RDWR` claim at mode `0600` (or the platform-equivalent exclusive +create), retains that descriptor, and records its descriptor identity. `EEXIST` — +including a file that appeared after an earlier `ENOENT` — enters the strict existing- +database path. Any preliminary `lstat` remains path-safety evidence only. The +exclusive winner opens the already-created path with +`{ readwrite:true, create:false }`; only its private `createdByThisOpen` authority may +reach compatibility adoption. Row absence alone never supplies that authority. + The **general initializer remains unchanged and fail-closed**. Read/observe, `BeginCodexTransition`, Worker claim/terminal work, guardian retry, explicit legacy recovery, and direct transaction opens may create the ordinary unscheduled `{0,null}` @@ -666,19 +684,36 @@ callback; its closed intent is `retained-apply` with exactly exactly once. Residue detection, startup observation, a Worker/guardian retry, history-only recovery, or an arbitrary operation value is not positive authority. -When and only when that handoff finds a truly absent coordinator database, a missing -or valid non-legacy integration record, and positively classified routed residue, it -acquires `BEGIN IMMEDIATE` and captures **authoritative row absence plus that routed -observation** without installing `{0,null,unknown}`. The retained native callback then -runs. Its bound `authorize(next)` requires `next.operation === intent.operation` and, -on the same already-open handle, creates the schema and inserts singleton 1 directly -as generation zero, null txId, and the fresh complete -`pending/wp10-compatibility` schedule; it then sets `user_version = 1`. Callback or -authorization failure rolls the transaction back and dispatches no Worker. This is a -compatibility schedule over evidence the current high-level operation just took -responsibility for, not an empty baseline inferred from residue. Once the row exists, -legacy JSON fields have no authority and are removed on the next successful non-CAS -record update while all unrelated unknown keys survive. +When and only when that handoff still owns the exclusive creation descriptor for the +exact coordinator file, has a missing or valid non-legacy integration record, and has +positively classified routed residue, it acquires `BEGIN IMMEDIATE` and captures +**creation ownership, authoritative row absence, and that routed observation** without +installing `{0,null,unknown}`. The retained native callback then runs. Its bound +`authorize(next)` requires `next.operation === intent.operation` and, on the same +already-open handle, creates the schema and inserts singleton 1 directly as generation +zero, null txId, and the fresh complete `pending/wp10-compatibility` schedule; it then +sets `user_version = 1`. A no-clobber loser, an ordinary opener of any existing file, +or a creator whose path identity changed cannot enter this branch. + +WP10 chooses **safe exact-identity cleanup** for an exclusively created coordinator +whose callback, one-shot authorization, or precommit transaction fails. The owner +rolls back and closes SQLite while retaining the exclusive creation descriptor, +compares that descriptor's `fstat` identity with a fresh non-symlink regular-file +`lstat` of the final path, and unlinks only that exact file before releasing the +descriptor. It never unlinks an `EEXIST` path, a substituted identity, or a database +that committed a valid schema/row. Successful cleanup restores real path absence and +dispatches no Worker; the next legitimate high-level operation must win a new +no-clobber claim and repeat integration-record/residue classification rather than +reuse stale evidence. This disposition is required because the current rollback/close +path (`src/codex/transition-state.ts:386-390`) leaves a zero-byte, version-zero, +rowless file, which every later strict opener correctly refuses and would otherwise +wedge the installation. Identity mismatch or unlink failure is surfaced as a typed +database/unsafe-path failure and is never bypassed by adoption. + +This is a compatibility schedule over evidence the current high-level operation just +took responsibility for, not an empty baseline inferred from residue. Once the row +exists, legacy JSON fields have no authority and are removed on the next successful +non-CAS record update while all unrelated unknown keys survive. A missing JSON file is valid and the first provenance update creates `{version:1}`. Unreadable/unparseable JSON is not empty: provenance mutation fails closed. Unknown @@ -2143,6 +2178,29 @@ mere residue detection opt into adoption, insert an unscheduled `{0,null}` row f or authorize restore from a post-callback clean observation; the real apply/restore fixture returns `legacy-ambiguous` or the named negative fixture creates authority. +Add a cross-process no-clobber race to `tests/codex-transition-state.test.ts`. Process +A pauses after its path-safety `lstat` observes `ENOENT`; process B exclusively creates +the coordinator path, leaves it unversioned/rowless, and closes; A then resumes. A's +exclusive claim must receive `EEXIST`, enter the strict existing-database path, return +`legacy-ambiguous`, and leave a retained-native-callback sentinel untouched. **Broken +change:** restore `databaseWasAbsent = true` from the earlier `ENOENT` plus SQLite +`create:true`, or treat `EEXIST` as creator authority; A adopts B's rowless file and +the callback sentinel fires. The existing static fixture that places a rowless database +before invocation also passes under that broken check-then-open implementation, so it +is necessary negative coverage but not evidence that creation ownership is atomic. + +Add a table-driven first-adoption abort case to +`tests/codex-native-residue.test.ts`: once with the retained callback throwing before +authorization and once with the one-shot authorization rejecting the operation, hold +the exclusive descriptor through rollback/SQLite close, prove exact-identity unlink +restores path absence, then run the next legitimate real high-level operation. That +second operation must acquire a fresh exclusive claim, reclassify current residue, +commit the exact generation-zero pending compatibility schedule, and dispatch its +Worker. **Broken change:** rollback/close without exact-identity unlink, release the +creation descriptor before cleanup, or retain a zero-byte database; the second +operation receives `EEXIST`, follows strict rowless refusal, and never reaches its +callback/schedule assertion. + `tests/codex-convergence-contract.test.ts`: every `ConvergeOutcome` variant maps to the §5 row, `busy` carries `Retry-After`, and a best-effort management caller still returns 2xx while reporting a non-converged disposition. Concatenate all @@ -2418,7 +2476,10 @@ writes to it. eventually repairs history. WP10 holds N from each retained native mutation through its compatibility authorization; a newer native handoff replaces even a pending or running older schedule, and the older Worker loses its terminal CAS. H has one contract-owned final path per effective - user/canonical home/canonical state DB and takes only fail-fast `H -> N`; inverse + user/canonical home/canonical state DB and takes only fail-fast `H -> N`; + compatibility adoption requires a still-live OS-exclusive creation claim, and a + failed first adoption exact-identity-unlinks only its own uncommitted coordinator so + retry can claim a genuinely absent path. Inverse `N/K/C -> H` edges are forbidden. Only `missing` proves path absence; a present valid matching v1 manifest is `ready` with a validated non-negative count, including zero, while @@ -2427,11 +2488,13 @@ writes to it. C2/C12 (generation-guarded catalog commit plus the phase-specific catalog/full admission and observation sequences). **Broken change:** acquire N only after the retained native mutation, release N before compatibility authorization, restore a - caller-supplied expected row or second-connection read, route an existing routed + caller-supplied expected row or second-connection read, derive creator authority from + `lstat` `ENOENT` plus SQLite `create:true`, omit exact-identity cleanup after a failed + first adoption, route an existing routed installation through the strict clean-only initializer, let observation/residue alone create a generation-zero row, restore the terminal-only compatibility predicate, release H between surfaces, classify a present ready-zero manifest as missing/unsupported, or commit catalog bytes after a - generation/evidence conflict; the stale-row/one-handle, routed apply/restore, - negative-adoption, handoff-order, terminal-CAS, serialization, manifest, or + generation/evidence conflict; the stale-row/one-handle, no-clobber-race, abort-retry, + routed apply/restore, negative-adoption, handoff-order, terminal-CAS, serialization, manifest, or stale-commit fixture fails respectively. diff --git a/devlog/_plan/260804_codex_write_substrate/020_history_isolation.md b/devlog/_plan/260804_codex_write_substrate/020_history_isolation.md index 6eacb9d45..5e83d81f9 100644 --- a/devlog/_plan/260804_codex_write_substrate/020_history_isolation.md +++ b/devlog/_plan/260804_codex_write_substrate/020_history_isolation.md @@ -62,7 +62,7 @@ their distinct history semantics through the Worker. It consumes the already-lan path/transaction owner; it does not pre-implement WP11's full native lock. All current-code citations in this document were rechecked on 2026-08-05 at -`b8168510f32ef583f14660067e0783d88df5df0c`. +`9e405e2b46f668c81141ffdb6ecc776841c7761a`. ## IN / OUT @@ -80,7 +80,14 @@ IN: and its operation identity, expose the history-specific schedule/claim/terminal CAS used by current roots, retain the operation for guardian restart, and expose the narrow synchronous N-backed compatibility handoff plus its private routed-install - adoption mode. The transaction-bound, one-shot + adoption mode. Replace the current `lstat`-then-SQLite-`create:true` absence flag with + an OS-level `O_CREAT | O_EXCL` creation claim taken before SQLite open; `EEXIST` + always takes the strict existing-database path. Retain the exclusive descriptor and + exact file identity until commit or safe cleanup, and exact-identity-unlink an + exclusively created uncommitted coordinator after callback/authorization failure so + a later operation can retry from genuine absence. This is an opener behavior change, + not a new public `openCodexCoordinatorTransaction` parameter. The transaction-bound, + one-shot compatibility authorizer may supersede an older terminal, pending, or running schedule after the newer retained native mutation and receives no caller-supplied expected row; this is not a producer of native generations or full authority @@ -232,6 +239,14 @@ that a missing coordinator is `legacy-ambiguous` (`tests/codex-native-residue.test.ts:213-242`). Existing routed installations would therefore fail before apply or restore reached their retained callback. +The twelfth recurrence is creation ownership itself. The current owner remembers +`ENOENT` from `lstatSync` (`src/codex/transition-state.ts:356-373`), later opens with +SQLite `create:true`, and initializes from that stale boolean +(`src/codex/transition-state.ts:374-385`). Another process can create an +unversioned/rowless coordinator between those steps. Row absence after +`BEGIN IMMEDIATE` does not distinguish that partially initialized existing database +from one this process created. + WP10 preserves that general guard. Ordinary reads, Worker/guardian paths, explicit recovery, `BeginCodexTransition`, and direct transaction opens may initialize only a native-clean, non-legacy installation. They continue to refuse invalid/legacy JSON, @@ -244,18 +259,39 @@ and its closed intent is exactly `retained-apply` with `restore-openai`. Mere residue detection, observation, retry, history-only recovery, or an arbitrary operation cannot request adoption. -For a truly absent coordinator, valid/missing non-legacy integration record, and a -positive routed classification, that mode takes `BEGIN IMMEDIATE`, captures row -absence plus the routed observation, and does **not** install the ordinary unscheduled -`{0,null,unknown}` row. The residue classifier evaluates every surface and gives any -indeterminate result precedence (`src/codex/native-residue.ts:520-556`). After the -native callback, the same one-shot `authorize(next)` requires the intent/operation to -match and uses the already-open handle to create schema plus generation-zero/null-txId +For a truly absent coordinator, the handoff must first win an OS-level no-clobber +`O_CREAT | O_EXCL | O_RDWR` claim at mode `0600` (or platform-equivalent exclusive +create) before any SQLite open, retain the descriptor and its `fstat` identity, and +open that already-created path with `{ readwrite:true, create:false }`. A preliminary +`lstat` is path-safety evidence only. `EEXIST`, including a file created after an earlier +`ENOENT`, takes the strict existing-database path and can never request adoption. + +Only that still-live creation claim, together with a valid/missing non-legacy +integration record and a positive routed classification, lets the mode take +`BEGIN IMMEDIATE`, capture creation ownership + row absence + the routed observation, +and avoid installing the ordinary unscheduled `{0,null,unknown}` row. The residue +classifier evaluates every surface and gives any indeterminate result precedence +(`src/codex/native-residue.ts:520-556`). After the native callback, the same one-shot +`authorize(next)` requires the intent/operation to match and uses the already-open +handle to create schema plus generation-zero/null-txId `pending/wp10-compatibility` schedule in one transaction. This remains valid for restore after the callback has removed config/catalog residue because the positive -routed observation was captured under N before mutation. Failure rolls back and -dispatches no Worker. An existing malformed/unversioned/rowless coordinator is never -adopted, and no `OPENCODEX_HOME`-local pair is imported. +routed observation was captured under N before mutation. + +WP10 chooses safe exact-identity cleanup rather than a durable failed-adoption row. +If the retained callback, one-shot authorization, or precommit transaction fails, the +owner rolls back and closes SQLite while retaining the exclusive descriptor, compares +its `fstat` identity with a fresh non-symlink regular-file `lstat` at the final path, +and unlinks only that exact exclusively created uncommitted file before releasing the +descriptor. It never unlinks an `EEXIST` path, a substituted file, or a database with a +committed valid schema/row. Successful cleanup dispatches no Worker and restores true +path absence; the next legitimate high-level operation must win a new claim and repeat +record/residue classification. This closes the mirror wedge in the current abort path: +rollback/close at `src/codex/transition-state.ts:386-390` leaves a zero-byte, +version-zero, rowless file that every later strict opener refuses. Identity mismatch or +unlink failure is a typed database/unsafe-path failure, never adoption authority. An +existing malformed/unversioned/rowless coordinator is never adopted, and no +`OPENCODEX_HOME`-local pair is imported. The Worker claim/terminal CAS matches native pair, job id, operation, and complete authority. A compatibility authority can never be relabeled as admission authority. @@ -344,8 +380,9 @@ from acquiring or awaiting H. “Through the same open N handle” is a connection-counted invariant, not shorthand for re-entering the owner. The handoff reads the complete existing row once after -acquisition and closes over it; the adoption case closes over authoritative absence -plus the pre-mutation routed observation. The callback can retain any stale pre-N +acquisition and closes over it; the adoption case closes over the still-live exclusive +creation claim, authoritative row absence, and the pre-mutation routed observation. +The callback can retain any stale pre-N observation, but no API accepts it. Any `readCodexTransitionState` call or other coordinator connection created inside the callback is a test failure. @@ -576,6 +613,28 @@ residue alone authorize adoption, commit `{0,null,unknown}` before the schedule, re-observe only post-restore clean state; the real fixture returns `legacy-ambiguous`/loses its schedule or a negative fixture creates authority. +In `tests/codex-transition-state.test.ts`, add the creation race that the static +rowless fixture cannot cover. Process A pauses after its path-safety `lstat` observes +`ENOENT`; process B exclusively creates and closes an unversioned/rowless coordinator; +A resumes. A must receive `EEXIST` from its no-clobber claim, take the strict existing- +database path, refuse before a retained-native-callback sentinel runs, and leave B's +file untouched. **Broken change:** restore the stale `databaseWasAbsent` flag plus +SQLite `create:true`, or treat `EEXIST` as creation authority; A adopts B's file and +the callback sentinel fires. The existing fixture that places a rowless database +before invocation would still pass with this race present, so it is not sufficient +atomic-creation evidence. + +In `tests/codex-native-residue.test.ts`, table-drive first-adoption abort before +authorization and authorization rejection. The failed attempt must rollback/close, +exact-identity-unlink its exclusively created uncommitted coordinator while the claim +descriptor remains live, and dispatch no Worker. The next legitimate real high-level +operation must win a fresh no-clobber claim, reclassify residue, commit the exact +generation-zero pending compatibility schedule, and dispatch its Worker. **Broken +change:** leave rollback/close as the only abort disposition, release creation +authority before cleanup, or keep the zero-byte file; the second operation receives +`EEXIST`, takes strict rowless refusal, and never reaches its callback/schedule +assertion. + ### H namespace and lock order Two child processes vary `HOME`, `USERPROFILE`, `TMPDIR`, `XDG_RUNTIME_DIR`, `TEMP`, @@ -673,6 +732,8 @@ the installed service and live proxy on 10100 remain untouched. | **C4 — durable unresolved work** | Guardian activation/backoff test proves unresolved typed operation survives failure/restart and never becomes zero-looking success. | Remove startup arming, restore the 60-tick stop, or persist zero counts after failed evidence. The activation/backoff/evidence case fails. | | **C15 — cross-process all-surface serialization** | Opposite operations serialize manifest, rollout, DB, post-probe, and terminal update under one H; N spans each retained native mutation through compatibility authorization, the newer schedule replaces even running work, and its Worker repairs stale work. | Release H between surfaces, bypass H, release/acquire N inside the native-to-authorization span, or restore terminal-only authorization. The sentinel/order/final-state case fails. | | **Transaction-observed authority** | A stale pre-N row cannot be supplied to the one-shot authorizer; B authorizes from the complete row read on its one already-open N handle. | Restore `authorize(expected, next)`, call `readCodexTransitionState` inside the callback, or open a second N connection. The stale-row or connection-count case fails after B's real native mutation. | +| **Atomic adoption creation** | A process paused after `ENOENT` loses to another process's unversioned/rowless creation and refuses before its native callback. | Restore the stale `databaseWasAbsent` + SQLite `create:true` path or treat `EEXIST` as creator authority. The callback sentinel fires in the no-clobber race. | +| **Adoption abort recovery** | Callback and authorization failure exact-identity-remove only the exclusively created uncommitted coordinator; the next legitimate operation freshly claims, adopts, and schedules. | Roll back/close without exact-identity unlink, release the claim before cleanup, or retain the zero-byte file. The second operation is refused as existing-rowless and its callback/schedule assertion fails. | | **Compatibility adoption** | Real apply and restore fixtures start with routed config/catalog/history and no coordinator DB, then commit an exact generation-zero pending compatibility schedule; every non-high-level or ambiguous case remains refused. | Route the handoff through strict clean-only initialization, let residue/observation/retry request adoption, insert an unscheduled row first, or require post-callback residue. The real routed fixture or its named negative row fails. | | **Operation authority** | Every operation variant is derived/validated from durable state, including no-op and manifest-independent recovery. | Trust request `targetProvider`/direction. Tamper and manifest-preservation cases fail. | | **Real lock order** | Architecture fixtures allow `H -> N -> K -> C`; the compatibility root may execute retained native/K/C work and authorize through already-held N, then must release N before dispatch. Every inverse/cross-domain edge remains rejected. | Await/spawn H while N is held, open N only after native mutation, release N before authorization, or call K from the Worker. Dependency/order fixture fails. | From c1bae496b69f02d63795afc1e76950c89c4521ed Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Wed, 5 Aug 2026 04:39:27 +0900 Subject: [PATCH 087/163] docs(substrate): a finally block does not run when the process is killed MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The exclusive claim closed the check-then-open race, but only for processes that get to finish. Kill one after it creates the claim and the descriptor closes while the file stays: the next opener gets EEXIST, takes the strict path, and the initializer rejects an existing version-zero database. No row, no schedule, nothing for the guardian to repair — every later apply and restore wedges as legacy-ambiguous. I reproduced it: pathExists true, size 0, nextClaim EEXIST. Thirteenth instance of the same pattern, and a pointed one: successful cleanup in a finally was being read as proof no artifact could survive, which is exactly what process death disproves. So nothing incomplete is ever visible under the final name. Creation now builds a complete v1 coordinator in a same-directory temp — full schema, singleton row at generation 0 with a durable `adoption-pending` status, the intent-derived operation, fresh job and authority ids, user_version 1 — commits it, closes, validates read-only, fsyncs, and only then publishes under the real name through an exclusive link or a rename that refuses to replace. A crash leaves either no file or a database that means something. A later opener reads `adoption-pending` as valid but may not dispatch a Worker from it. Only a real retained apply or restore handoff can rerun its native callback under N and move it to pending; readers, the guardian, the Worker, residue observation and explicit recovery cannot. A killed callback therefore leaves an actionable row rather than a puzzle: run the operation again. Chose durable publication over a creation lease deliberately. A lease needs liveness detection, and inferring that a creator is gone is the same species of reasoning this unit has now gotten wrong thirteen times — with Windows deletion semantics making it worse. Two scoping corrections the reviewer asked for: cleanup never unlinking a committed valid row is true of COOPERATING processes, since a foreign in-place write keeps the same identity; and a stale hard-link temp alias must never be opened as SQLite, only unlinked after matching the validated final database. --- .../005_contract.md | 246 ++++++++++++------ .../020_history_isolation.md | 192 +++++++++----- 2 files changed, 287 insertions(+), 151 deletions(-) diff --git a/devlog/_plan/260804_codex_write_substrate/005_contract.md b/devlog/_plan/260804_codex_write_substrate/005_contract.md index 297ad5290..f37cc636d 100644 --- a/devlog/_plan/260804_codex_write_substrate/005_contract.md +++ b/devlog/_plan/260804_codex_write_substrate/005_contract.md @@ -12,7 +12,7 @@ four phase docs are rewritten as consumers against the reviewer's section list. A contract nobody collected is a fifth opinion. All current-code citations in this document were rechecked on 2026-08-05 at -`9e405e2b46f668c81141ffdb6ecc776841c7761a`. +`8a8323f7885da957120218495715cab8593a5d66`. ## IN / OUT @@ -105,7 +105,14 @@ export type CodexHistoryAuthority = | { readonly kind: "explicit-legacy-recovery"; readonly id: string }; export interface CodexHistoryState { - status: "converged" | "pending" | "running" | "blocked" | "unknown" | "not-evaluated"; + status: + | "adoption-pending" + | "converged" + | "pending" + | "running" + | "blocked" + | "unknown" + | "not-evaluated"; /** * Why it is not converged, when it is not. These are terminal observations * for one attempt, not reasons to collapse the durable retry schedule. @@ -303,7 +310,7 @@ export interface CodexTransitionVersion { } export interface CodexTransitionState extends CodexTransitionVersion { - /** Durable schedule and latest terminal observation for this exact pair. */ + /** Durable schedule, or non-dispatchable adoption authority, plus latest observation. */ readonly history: CodexHistoryState; readonly historySchedule: null | Readonly<{ jobId: string; @@ -411,8 +418,9 @@ type SynchronousCompatibilityHandoff = T extends PromiseLike ? never * WP10's narrow native-mutation-to-history-authorization exclusion. The callback * starts only after N is held, must authorize exactly once through the supplied * transaction-bound closure, and returns before N is committed and released. A - * compatibility adoption additionally requires the opener's still-live exclusive - * creation claim; an earlier path-absence observation is never authority. + * compatibility adoption additionally requires either atomic publication of a + * complete `adoption-pending` coordinator or a valid row already in that state; + * an earlier path-absence observation and a rowless file are never authority. */ export type WithCodexCompatibilityNativeHandoff = ( intent: CodexCompatibilityNativeIntent, @@ -472,7 +480,7 @@ CREATE TABLE codex_transition_state ( history_backup_entries INTEGER, updated_at TEXT NOT NULL, CHECK (history_status IN - ('converged', 'pending', 'running', 'blocked', 'unknown')), + ('adoption-pending', 'converged', 'pending', 'running', 'blocked', 'unknown')), CHECK (history_reason IS NULL OR history_reason IN ('db-busy', 'permission', 'unreadable', 'schema', 'timeout', 'shutdown-cancelled', 'worker-died', 'overtaken', 'foreign-state-db', @@ -500,6 +508,19 @@ CREATE TABLE codex_transition_state ( AND native_generation > 0 AND history_tx_id = current_tx_id)), CHECK (native_generation = 0 OR history_operation IS NOT NULL), + CHECK (history_status != 'adoption-pending' OR + (native_generation = 0 + AND current_tx_id IS NULL + AND history_reason IS NULL + AND history_attempts = 0 + AND history_next_retry_at IS NULL + AND history_tx_id IS NOT NULL + AND history_operation IS NOT NULL + AND history_operation != 'recover-legacy-openai' + AND history_authority_kind = 'wp10-compatibility' + AND length(trim(history_authority_id)) > 0 + AND history_pending_rows IS NULL + AND history_backup_entries IS NULL)), CHECK (history_operation IS NOT NULL OR (history_status = 'unknown' AND history_reason IS NULL @@ -512,7 +533,9 @@ CREATE TABLE codex_transition_state ( The observation columns project to `CodexHistoryState`; job id, operation and typed authority kind/id are schedule metadata required to restart the exact Worker after process -death. `not-evaluated` remains ephemeral and is rejected by the table. A native +death. `adoption-pending` carries the same complete identity but is deliberately not +Worker-dispatchable until a real retained native callback changes it to `pending`. +`not-evaluated` remains ephemeral and is rejected by the table. A native transition publishes its winner and schedule atomically with this null-safe conditional update (SQLite `IS` is required for the initial null txId): @@ -651,18 +674,16 @@ check-then-open claim. The owner records `ENOENT` from `lstatSync` (`src/codex/transition-state.ts:356-373`), later opens with `create:true`, and passes the stale boolean into initialization (`src/codex/transition-state.ts:374-385`). A second process can create an unversioned/rowless file between those steps, yet the -first process treats it as its own new database. WP10 therefore changes the opener's -behavior, not its public `openCodexCoordinatorTransaction(finalDatabasePath)` -signature: before any SQLite open, it attempts an OS-level no-clobber -`O_CREAT | O_EXCL | O_RDWR` claim at mode `0600` (or the platform-equivalent exclusive -create), retains that descriptor, and records its descriptor identity. `EEXIST` — -including a file that appeared after an earlier `ENOENT` — enters the strict existing- -database path. Any preliminary `lstat` remains path-safety evidence only. The -exclusive winner opens the already-created path with -`{ readwrite:true, create:false }`; only its private `createdByThisOpen` authority may -reach compatibility adoption. Row absence alone never supplies that authority. - -The **general initializer remains unchanged and fail-closed**. Read/observe, +first process treats it as its own new database. The thirteenth review then found the +process-death hole in the proposed exclusive final-path claim: rollback/close plus +`finally` cleanup handles a caught failure, but kill after creation closes the +descriptor and leaves the zero-byte path permanently visible. The strict initializer +correctly rejects that existing version-zero database at +`src/codex/transition-state.ts:282-303`, so neither the intended callback nor guardian +recovery can run. + +The **general initializer's authority rule remains unchanged and fail-closed**. +Read/observe, `BeginCodexTransition`, Worker claim/terminal work, guardian retry, explicit legacy recovery, and direct transaction opens may create the ordinary unscheduled `{0,null}` row only when the JSON record is missing or valid with no legacy transition fields @@ -674,6 +695,14 @@ checks every routed surface and returns any `indeterminate` result before `resid the exception. An `OPENCODEX_HOME`-local positive pair is never imported because a second home may hold a different claimant. +Its creation mechanism does change: a native-clean ordinary initializer also builds a +complete v1 temp database and atomically no-clobber-publishes the ordinary singleton +`{ native_generation:0, current_tx_id:null, history_status:'unknown' }` with null +schedule/reason/timer/probe fields and zero attempts. It never opens a missing final +path with SQLite `create:true`. Thus every coordinator database first visible at the +final name has a supported schema and authoritative singleton, whether it is the +ordinary clean row or the compatibility row below. + WP10 adds one private **compatibility-adoption** mode beneath `withCodexCompatibilityNativeHandoff`; it is not an option on the public transaction opener. “Positively authorized” means all of the following are true: the sole graph- @@ -684,31 +713,75 @@ callback; its closed intent is `retained-apply` with exactly exactly once. Residue detection, startup observation, a Worker/guardian retry, history-only recovery, or an arbitrary operation value is not positive authority. -When and only when that handoff still owns the exclusive creation descriptor for the -exact coordinator file, has a missing or valid non-legacy integration record, and has -positively classified routed residue, it acquires `BEGIN IMMEDIATE` and captures -**creation ownership, authoritative row absence, and that routed observation** without -installing `{0,null,unknown}`. The retained native callback then runs. Its bound -`authorize(next)` requires `next.operation === intent.operation` and, on the same -already-open handle, creates the schema and inserts singleton 1 directly as generation -zero, null txId, and the fresh complete `pending/wp10-compatibility` schedule; it then -sets `user_version = 1`. A no-clobber loser, an ordinary opener of any existing file, -or a creator whose path identity changed cannot enter this branch. - -WP10 chooses **safe exact-identity cleanup** for an exclusively created coordinator -whose callback, one-shot authorization, or precommit transaction fails. The owner -rolls back and closes SQLite while retaining the exclusive creation descriptor, -compares that descriptor's `fstat` identity with a fresh non-symlink regular-file -`lstat` of the final path, and unlinks only that exact file before releasing the -descriptor. It never unlinks an `EEXIST` path, a substituted identity, or a database -that committed a valid schema/row. Successful cleanup restores real path absence and -dispatches no Worker; the next legitimate high-level operation must win a new -no-clobber claim and repeat integration-record/residue classification rather than -reuse stale evidence. This disposition is required because the current rollback/close -path (`src/codex/transition-state.ts:386-390`) leaves a zero-byte, version-zero, -rowless file, which every later strict opener correctly refuses and would otherwise -wedge the installation. Identity mismatch or unlink failure is surfaced as a typed -database/unsafe-path failure and is never bypassed by adoption. +When the final coordinator path is genuinely absent and those record/residue/intent +preconditions hold, the transition owner uses that same complete-database publisher +to publish compatibility authority **before** invoking the native callback. It +exclusively creates a unique mode-`0600` temp file in the final directory, opens +SQLite only at that temp path, creates the complete v1 schema, and commits singleton 1 +with exactly: + +- `native_generation = 0`, `current_tx_id = NULL`; +- `history_status = 'adoption-pending'`, `history_reason = NULL`, + `history_attempts = 0`, `history_next_retry_at = NULL`; +- a fresh non-empty `history_tx_id`, the exact intent-derived non-recovery + `history_operation`, `history_authority_kind = 'wp10-compatibility'`, and a fresh + opaque non-empty `history_authority_id`; +- `history_pending_rows = NULL`, `history_backup_entries = NULL`, a fresh + `updated_at`, and `PRAGMA user_version = 1`. + +The temp database uses a completed rollback-journal transaction, is closed with no +live journal/WAL sidecar, is reopened read-only through the ordinary row validator, +and its database bytes are fsynced before publication. The owner then publishes it +with the same atomic no-clobber shape required for create-once backups in §3: a +same-directory exclusive hard link or platform rename-without-replace equivalent. +An ordinary replacing rename is forbidden. `EEXIST` means another process won and +always enters the strict existing-database path after the unpublished temp is +scrubbed. A successful link/rename is followed by parent-directory fsync; a hard-link +temp alias is then removed without touching the final name. No SQLite handle opened +on the temp crosses publication. Cooperating processes never open a published or stale +temp alias as SQLite; after a crash they may only unlink an alias whose non-symlink +regular-file identity matches the validated final database. + +The publication operation is the crash boundary. Before it, the final path is absent +and a killed creator can leave only a non-authoritative unique temp; the next real +operation ignores that temp and can publish anew. During it, the no-replace primitive +makes the final name atomically either absent or linked to the already complete, +validated v1 database. After it, including death before temp-alias cleanup, final-path +reopen, native callback, or authorization, every opener sees the complete typed +`adoption-pending` row. There is no visible zero-byte/version-zero interval and no +final-path `finally` unlink on any post-publication failure. + +An ordinary opener validates `adoption-pending` as a ready coordinator state, never +as `legacy-ambiguous`, but it does not dispatch a history Worker: the row proves that +native mutation may not have started or may have died mid-callback. In WP10, only a +new positively authorized real retained apply/restore handoff may transition it. That +handoff takes N, re-runs its current closed-intent native callback idempotently, and +its transaction-bound `authorize(next)` conditionally replaces the complete row it +observed with the current operation's fresh `pending/wp10-compatibility` schedule. +The current operation may match or supersede the interrupted operation; N still makes +the committed schedule order equal the retained native-mutation order. Callback +throw, authorization failure, or process death rolls N back and leaves the durable +`adoption-pending` row unchanged for the next real operation. WP12's fully admitted +native transition may later supersede it through `BeginCodexTransition`; ordinary +readers, Worker/guardian claim or terminal paths, explicit legacy recovery, and +residue observation may not transition it or relabel its authority. + +Startup's ordinary apply/restore root therefore recovers automatically by re-entering +the handoff. If an explicit caller cannot finish within its bounded attempt, it +returns the typed ready state with `history.status = 'adoption-pending'` and the +action “rerun the same requested apply/restore”; doctor reports the same action. +Guardian sees the valid row but arms no history Worker until a native handoff changes +it to `pending`. It is never a permanent rowless refusal. + +The fail-closed guard is not relaxed for pre-existing files. `adoption-pending` may be +created only in the complete temp database whose final publication won from true path +absence after the private positive checks; it is never inserted into an existing +unversioned, unsupported, malformed, or rowless database and is never inferred from +residue. Such existing databases still refuse exactly as today. Caught pre-publication +cleanup may exact-identity-unlink only the unpublished temp. Among **cooperating** +processes that cleanup never unlinks a committed valid final-path row; a foreign +in-place writer can retain the same file identity, so identity is substitution +evidence rather than proof of unchanged contents. This is a compatibility schedule over evidence the current high-level operation just took responsibility for, not an empty baseline inferred from residue. Once the row @@ -2158,7 +2231,9 @@ trap fires. The general missing-DB/table path initializes only from native-clean/no-legacy state; outside the explicit compatibility-adoption fixture, legacy JSON pair/schedule, residue beside a missing row, malformed row, busy DB and unsafe path all fail closed -with the specified typed outcome. +with the specified typed outcome. A valid `adoption-pending` row is the sole new +ready state: read/doctor may report it, but Worker/guardian and explicit recovery may +not turn it into history work. `tests/codex-native-residue.test.ts`: add two compatibility-adoption fixtures that begin with routed config, routed catalog, routed history rows/rollouts/manifest, and @@ -2166,40 +2241,50 @@ begin with routed config, routed catalog, routed history rows/rollouts/manifest, `injectCodexConfig` high-level path (`src/codex/inject.ts:482-654`); the restore fixture enters real `restoreNativeCodex` (`src/codex/inject.ts:765-800`). Each must acquire N before its -retained callback and commit generation zero with the exact pending compatibility -operation before Worker dispatch; restore remains authorized even though its native -callback removes the routed config/catalog residue captured at N acquisition. The -Worker then repairs history and terminally owns the same schedule. Run matching +retained callback, atomically no-clobber-publish the exact generation-zero +`adoption-pending` identity before that callback, then conditionally change that same +row to the current exact pending compatibility operation before Worker dispatch; +restore remains authorized even though its native callback removes the routed +config/catalog residue captured before publication. The Worker then repairs history +and terminally owns the same schedule. Run matching negative fixtures for observe/read, guardian/Worker retry, explicit recovery, operation/intent mismatch, invalid legacy JSON, indeterminate residue, and an existing unversioned or rowless database; none may create a row or invoke the native callback. **Broken changes:** route the handoff through the strict clean-only initializer, let -mere residue detection opt into adoption, insert an unscheduled `{0,null}` row first, -or authorize restore from a post-callback clean observation; the real apply/restore -fixture returns `legacy-ambiguous` or the named negative fixture creates authority. +mere residue detection opt into adoption, publish a rowless/unscheduled `{0,null}` +database, let Worker/guardian dispatch `adoption-pending`, or authorize restore from a +post-callback clean observation; the real apply/restore fixture returns +`legacy-ambiguous`, dispatches history before native recovery, or the named negative +fixture creates authority. Add a cross-process no-clobber race to `tests/codex-transition-state.test.ts`. Process -A pauses after its path-safety `lstat` observes `ENOENT`; process B exclusively creates -the coordinator path, leaves it unversioned/rowless, and closes; A then resumes. A's -exclusive claim must receive `EEXIST`, enter the strict existing-database path, return -`legacy-ambiguous`, and leave a retained-native-callback sentinel untouched. **Broken -change:** restore `databaseWasAbsent = true` from the earlier `ENOENT` plus SQLite -`create:true`, or treat `EEXIST` as creator authority; A adopts B's rowless file and -the callback sentinel fires. The existing static fixture that places a rowless database -before invocation also passes under that broken check-then-open implementation, so it -is necessary negative coverage but not evidence that creation ownership is atomic. - -Add a table-driven first-adoption abort case to -`tests/codex-native-residue.test.ts`: once with the retained callback throwing before -authorization and once with the one-shot authorization rejecting the operation, hold -the exclusive descriptor through rollback/SQLite close, prove exact-identity unlink -restores path absence, then run the next legitimate real high-level operation. That -second operation must acquire a fresh exclusive claim, reclassify current residue, -commit the exact generation-zero pending compatibility schedule, and dispatch its -Worker. **Broken change:** rollback/close without exact-identity unlink, release the -creation descriptor before cleanup, or retain a zero-byte database; the second -operation receives `EEXIST`, follows strict rowless refusal, and never reaches its -callback/schedule assertion. +A and B both finish complete valid temp databases after observing final-path absence, +then race the final no-replace publication. Exactly one final name appears; it is a +validated v1 `adoption-pending` database from one contestant. The loser receives +`EEXIST`, scrubs only its unpublished temp, opens the winner as existing ready state, +and never overwrites or unlinks the final name. Separately place a foreign +unversioned/rowless file before publication and require strict refusal before the +native-callback sentinel. **Broken changes:** restore the stale +`databaseWasAbsent` + SQLite `create:true` path, use ordinary replacing rename, or +treat an existing rowless file as adoption authority; the final row is malformed, +the loser replaces the winner, or the callback sentinel fires. + +Add child-process termination checkpoints to +`tests/codex-transition-state.test.ts` and +`tests/codex-native-residue.test.ts`: kill after exclusive temp creation but before +SQLite open, after SQLite open/complete temp commit but before publication, +immediately after no-clobber publication but before temp-alias cleanup/final reopen, +and during the retained native callback. In the first two cases the final path stays +absent and a subsequent real apply/restore publishes normally. In the latter two the +final path is a complete validated `adoption-pending` coordinator; the next real +apply/restore reads it as ready, re-runs its current native callback under N, changes +it to the exact pending compatibility schedule, and dispatches its Worker. Callback +throw and authorization rejection leave the same durable `adoption-pending` row and +return its documented rerun action; they do not unlink the final database. **Broken +changes:** create the final path before schema/row commit, publish with replacing +rename, dispatch a Worker directly from `adoption-pending`, or clean the final path in +`finally`; a killed child leaves rowless refusal, a competing winner is clobbered, +history runs before native completion, or the next operation cannot recover. `tests/codex-convergence-contract.test.ts`: every `ConvergeOutcome` variant maps to the §5 row, `busy` carries `Retry-After`, and a best-effort management caller @@ -2477,9 +2562,11 @@ writes to it. its compatibility authorization; a newer native handoff replaces even a pending or running older schedule, and the older Worker loses its terminal CAS. H has one contract-owned final path per effective user/canonical home/canonical state DB and takes only fail-fast `H -> N`; - compatibility adoption requires a still-live OS-exclusive creation claim, and a - failed first adoption exact-identity-unlinks only its own uncommitted coordinator so - retry can claim a genuinely absent path. Inverse + compatibility adoption atomically no-clobber-publishes a complete valid v1 + `adoption-pending` coordinator before native mutation. Process death before + publication leaves the final path absent; death after publication leaves durable + authority that only a later real native handoff (or WP12 full admission) may move to + pending. Inverse `N/K/C -> H` edges are forbidden. Only `missing` proves path absence; a present valid matching v1 manifest is `ready` with a validated non-negative count, including zero, while @@ -2489,12 +2576,13 @@ writes to it. admission and observation sequences). **Broken change:** acquire N only after the retained native mutation, release N before compatibility authorization, restore a caller-supplied expected row or second-connection read, derive creator authority from - `lstat` `ENOENT` plus SQLite `create:true`, omit exact-identity cleanup after a failed - first adoption, route an existing routed + `lstat` `ENOENT` plus SQLite `create:true`, publish a rowless final-path claim, use a + replacing rename, dispatch history from `adoption-pending`, or route an existing routed installation through the strict clean-only initializer, let observation/residue alone create a generation-zero row, restore the terminal-only compatibility predicate, release H between surfaces, classify a present ready-zero manifest as missing/unsupported, or commit catalog bytes after a - generation/evidence conflict; the stale-row/one-handle, no-clobber-race, abort-retry, + generation/evidence conflict; the stale-row/one-handle, no-clobber-race, + child-process-death recovery, routed apply/restore, negative-adoption, handoff-order, terminal-CAS, serialization, manifest, or stale-commit fixture fails respectively. diff --git a/devlog/_plan/260804_codex_write_substrate/020_history_isolation.md b/devlog/_plan/260804_codex_write_substrate/020_history_isolation.md index 5e83d81f9..9f035cd08 100644 --- a/devlog/_plan/260804_codex_write_substrate/020_history_isolation.md +++ b/devlog/_plan/260804_codex_write_substrate/020_history_isolation.md @@ -62,7 +62,7 @@ their distinct history semantics through the Worker. It consumes the already-lan path/transaction owner; it does not pre-implement WP11's full native lock. All current-code citations in this document were rechecked on 2026-08-05 at -`9e405e2b46f668c81141ffdb6ecc776841c7761a`. +`8a8323f7885da957120218495715cab8593a5d66`. ## IN / OUT @@ -81,12 +81,13 @@ IN: used by current roots, retain the operation for guardian restart, and expose the narrow synchronous N-backed compatibility handoff plus its private routed-install adoption mode. Replace the current `lstat`-then-SQLite-`create:true` absence flag with - an OS-level `O_CREAT | O_EXCL` creation claim taken before SQLite open; `EEXIST` - always takes the strict existing-database path. Retain the exclusive descriptor and - exact file identity until commit or safe cleanup, and exact-identity-unlink an - exclusively created uncommitted coordinator after callback/authorization failure so - a later operation can retry from genuine absence. This is an opener behavior change, - not a new public `openCodexCoordinatorTransaction` parameter. The transaction-bound, + a complete v1 temp coordinator created exclusively in the final directory and + atomically published by no-replace link/rename before native mutation. The singleton + is generation-zero/null-txId `adoption-pending` with the exact non-recovery operation, + fresh job/authority identity, null reason/timer/probe counts, and + `wp10-compatibility` authority. `EEXIST` always takes the strict existing-database + path; an existing unversioned or rowless database is never adopted. This is an opener + behavior change, not a new public `openCodexCoordinatorTransaction` parameter. The transaction-bound, one-shot compatibility authorizer may supersede an older terminal, pending, or running schedule after the newer retained native mutation and receives no caller-supplied @@ -108,7 +109,9 @@ IN: typed operation durably as the sole root of the WP10 compatibility/explicit-recovery authorizers, derive the closed retained-apply/retained-restore adoption intent, enter the N-backed compatibility handoff before invoking a retained synchronous - native callback, spawn/watch/join the Worker only after N releases, + native callback, recover a valid `adoption-pending` row only by re-running a real + current retained callback, spawn/watch/join the Worker only after N releases and + the row is dispatchable `pending`, classify IPC/death, and expose one async entry point to every current high-level root. - `src/codex/inject.ts`, `src/codex/sync.ts`, @@ -247,6 +250,13 @@ unversioned/rowless coordinator between those steps. Row absence after `BEGIN IMMEDIATE` does not distinguish that partially initialized existing database from one this process created. +The thirteenth recurrence is the process-death disposition of the proposed fix. An +exclusive empty final-path claim survives kill even though its descriptor closes; +the next process receives `EEXIST`, and the strict initializer rejects the surviving +version-zero file at `src/codex/transition-state.ts:282-303`. `finally` cleanup proves +only caught-failure recovery. With no committed transition row or history schedule, +the guardian has no authority from which to repair the installation. + WP10 preserves that general guard. Ordinary reads, Worker/guardian paths, explicit recovery, `BeginCodexTransition`, and direct transaction opens may initialize only a native-clean, non-legacy installation. They continue to refuse invalid/legacy JSON, @@ -259,39 +269,69 @@ and its closed intent is exactly `retained-apply` with `restore-openai`. Mere residue detection, observation, retry, history-only recovery, or an arbitrary operation cannot request adoption. -For a truly absent coordinator, the handoff must first win an OS-level no-clobber -`O_CREAT | O_EXCL | O_RDWR` claim at mode `0600` (or platform-equivalent exclusive -create) before any SQLite open, retain the descriptor and its `fstat` identity, and -open that already-created path with `{ readwrite:true, create:false }`. A preliminary -`lstat` is path-safety evidence only. `EEXIST`, including a file created after an earlier -`ENOENT`, takes the strict existing-database path and can never request adoption. - -Only that still-live creation claim, together with a valid/missing non-legacy -integration record and a positive routed classification, lets the mode take -`BEGIN IMMEDIATE`, capture creation ownership + row absence + the routed observation, -and avoid installing the ordinary unscheduled `{0,null,unknown}` row. The residue -classifier evaluates every surface and gives any indeterminate result precedence -(`src/codex/native-residue.ts:520-556`). After the native callback, the same one-shot -`authorize(next)` requires the intent/operation to match and uses the already-open -handle to create schema plus generation-zero/null-txId -`pending/wp10-compatibility` schedule in one transaction. This remains valid for -restore after the callback has removed config/catalog residue because the positive -routed observation was captured under N before mutation. - -WP10 chooses safe exact-identity cleanup rather than a durable failed-adoption row. -If the retained callback, one-shot authorization, or precommit transaction fails, the -owner rolls back and closes SQLite while retaining the exclusive descriptor, compares -its `fstat` identity with a fresh non-symlink regular-file `lstat` at the final path, -and unlinks only that exact exclusively created uncommitted file before releasing the -descriptor. It never unlinks an `EEXIST` path, a substituted file, or a database with a -committed valid schema/row. Successful cleanup dispatches no Worker and restores true -path absence; the next legitimate high-level operation must win a new claim and repeat -record/residue classification. This closes the mirror wedge in the current abort path: -rollback/close at `src/codex/transition-state.ts:386-390` leaves a zero-byte, -version-zero, rowless file that every later strict opener refuses. Identity mismatch or -unlink failure is a typed database/unsafe-path failure, never adoption authority. An -existing malformed/unversioned/rowless coordinator is never adopted, and no -`OPENCODEX_HOME`-local pair is imported. +The ordinary clean initializer uses the same complete-database publication mechanism +as adoption: a unique same-directory temp receives the full v1 schema and authoritative +generation-zero/null-txId `unknown` singleton with null schedule/reason/timer/probe +fields and zero attempts, then is atomically published no-clobber. No initializer +opens a missing final path with SQLite `create:true`; every coordinator first visible +at the final name is already versioned and row-bearing. + +For a truly absent coordinator, the handoff must publish a complete valid database +before any native callback. It exclusively creates a unique mode-`0600` temp in the +final directory, opens SQLite at that temp path, commits the full contract schema and +`user_version = 1`, closes with no live journal/WAL sidecar, reopens through the +ordinary validator, and fsyncs the database. Its singleton contains exactly +generation zero/null current txId; `history_status='adoption-pending'`; null reason, +retry time, and probe counts; zero attempts; the exact intent-derived non-recovery +operation; and fresh non-empty job plus `wp10-compatibility` authority ids. This is a +typed durable native-handoff state, not a dispatchable history schedule. + +Publication uses a same-directory exclusive hard link or platform +rename-without-replace equivalent, followed by parent-directory fsync. Ordinary +replacing rename is forbidden. `EEXIST`, including a file created after an earlier +`ENOENT`, scrubs only the unpublished temp and takes the strict existing-database +path. A hard-link temp alias is removed only after the complete final name exists. +No SQLite handle opened on the temp crosses publication. A preliminary `lstat` +remains path-safety evidence only; row absence and path absence are never authority. +Cooperating processes never open a published or stale temp alias as SQLite; after a +crash they may only unlink an alias whose non-symlink regular-file identity matches +the validated final database. + +The residue classifier evaluates every surface and gives any indeterminate result +precedence (`src/codex/native-residue.ts:520-556`). Only the private positively +authorized high-level handoff, together with a missing/valid non-legacy integration +record and positive routed classification, may construct and publish the initial +`adoption-pending` database. The state is never inserted into an existing database +and never inferred from residue. Existing malformed, unsupported, unversioned, or +rowless coordinators remain strict refusals, and no `OPENCODEX_HOME`-local pair is +imported. + +After publication, the handoff opens the final path with +`{ readwrite:true, create:false }`, takes N, and reads the complete row. Only a new +positively authorized real retained apply/restore handoff may move +`adoption-pending` in WP10. It re-runs its current closed-intent native callback, then +the same transaction-bound `authorize(next)` conditionally replaces the observed row +with the current exact generation-zero/null-txId `pending/wp10-compatibility` +schedule. Its operation may match or supersede the interrupted operation; N makes the +resulting schedule order equal the retained native-mutation order. WP12 full admitted +native convergence may later supersede it through `BeginCodexTransition`. + +Ordinary read/doctor returns the valid typed state; Worker/guardian claim and terminal +paths, explicit legacy recovery, and residue observation cannot transition or dispatch +it. Startup's real apply/restore path recovers automatically. If a bounded explicit +attempt cannot complete, it returns the actionable ready state +`history.status='adoption-pending'` with “rerun the same requested apply/restore.” A +callback throw, authorization failure, or process death leaves the durable row +unchanged rather than unlinking the final database. + +Crash behavior is complete: death before publication leaves the final path absent and +at worst a non-authoritative unique temp; death during the atomic no-replace operation +leaves the final path either absent or a complete valid v1 row; death after publication +or during the retained callback leaves recoverable `adoption-pending`. Caught +pre-publication cleanup may exact-identity-unlink only its unpublished temp. Among +**cooperating** processes that cleanup never unlinks a committed valid final-path row; +a foreign in-place writer can retain identity, so identity protects against path +substitution but does not prove unchanged content. The Worker claim/terminal CAS matches native pair, job id, operation, and complete authority. A compatibility authority can never be relabeled as admission authority. @@ -380,8 +420,9 @@ from acquiring or awaiting H. “Through the same open N handle” is a connection-counted invariant, not shorthand for re-entering the owner. The handoff reads the complete existing row once after -acquisition and closes over it; the adoption case closes over the still-live exclusive -creation claim, authoritative row absence, and the pre-mutation routed observation. +acquisition and closes over it; first adoption publishes the durable +`adoption-pending` row before that acquisition, while recovery closes over the complete +valid row read from the final database and the current closed high-level intent. The callback can retain any stale pre-N observation, but no API accepts it. Any `readCodexTransitionState` call or other coordinator connection created inside the callback is a test failure. @@ -602,38 +643,45 @@ catalog, history DB rows, rollouts, and a matching manifest but no coordinator database. Run one real high-level apply through `injectCodexConfig` (`src/codex/inject.ts:482-654`) and one real high-level restore through `restoreNativeCodex` (`src/codex/inject.ts:765-800`). Each must commit the -exact generation-zero pending compatibility schedule before Worker dispatch, and its -Worker must repair/terminally own that identity. The restore fixture proves adoption -does not depend on residue still being present after the native callback. Table-drive +exact generation-zero `adoption-pending` identity by atomic no-clobber publication +before its native callback, then change that row to the exact pending compatibility +schedule before Worker dispatch. Its Worker must repair/terminally own that identity. +The restore fixture proves recovery does not depend on residue still being present +after the native callback. Table-drive negative observe/read, guardian/Worker retry, explicit recovery, intent/operation mismatch, invalid legacy JSON, indeterminate residue, and existing unversioned/rowless database cases; they create no row and do not invoke the native callback. **Broken changes:** send compatibility roots through the strict clean-only initializer, let -residue alone authorize adoption, commit `{0,null,unknown}` before the schedule, or -re-observe only post-restore clean state; the real fixture returns -`legacy-ambiguous`/loses its schedule or a negative fixture creates authority. +residue alone authorize adoption, publish a rowless/unscheduled `{0,null}` database, +dispatch a Worker from `adoption-pending`, or re-observe only post-restore clean state; +the real fixture returns `legacy-ambiguous`/loses its schedule, history runs before +native completion, or a negative fixture creates authority. In `tests/codex-transition-state.test.ts`, add the creation race that the static -rowless fixture cannot cover. Process A pauses after its path-safety `lstat` observes -`ENOENT`; process B exclusively creates and closes an unversioned/rowless coordinator; -A resumes. A must receive `EEXIST` from its no-clobber claim, take the strict existing- -database path, refuse before a retained-native-callback sentinel runs, and leave B's -file untouched. **Broken change:** restore the stale `databaseWasAbsent` flag plus -SQLite `create:true`, or treat `EEXIST` as creation authority; A adopts B's file and -the callback sentinel fires. The existing fixture that places a rowless database -before invocation would still pass with this race present, so it is not sufficient -atomic-creation evidence. - -In `tests/codex-native-residue.test.ts`, table-drive first-adoption abort before -authorization and authorization rejection. The failed attempt must rollback/close, -exact-identity-unlink its exclusively created uncommitted coordinator while the claim -descriptor remains live, and dispatch no Worker. The next legitimate real high-level -operation must win a fresh no-clobber claim, reclassify residue, commit the exact -generation-zero pending compatibility schedule, and dispatch its Worker. **Broken -change:** leave rollback/close as the only abort disposition, release creation -authority before cleanup, or keep the zero-byte file; the second operation receives -`EEXIST`, takes strict rowless refusal, and never reaches its callback/schedule -assertion. +rowless fixture cannot cover. Processes A and B each finish a complete validated v1 +temp database after seeing final-path absence, then race no-replace publication. +Exactly one valid `adoption-pending` final database wins; the loser receives `EEXIST`, +scrubs only its temp, opens the winner as existing ready state, and never replaces or +unlinks it. Separately seed a foreign unversioned/rowless final database and require +strict refusal before a retained-native-callback sentinel runs. **Broken changes:** +restore the stale `databaseWasAbsent` flag plus SQLite `create:true`, publish by +ordinary rename, or treat existing rowless bytes as authority; the winner is +clobbered/malformed or the callback sentinel fires. + +In `tests/codex-transition-state.test.ts` and +`tests/codex-native-residue.test.ts`, run child-process kill checkpoints after +exclusive temp creation but before SQLite open, after SQLite open/complete commit but +before publication, immediately after no-replace publication but before alias +cleanup/final reopen, and during the retained native callback. The first two leave the +final path absent; a subsequent real apply/restore publishes normally. The latter two +leave a complete validated `adoption-pending` row; the next real apply/restore reads it +as ready, re-runs its current native callback under N, changes it to the exact pending +schedule, and dispatches its Worker. Callback throw and authorization rejection leave +that same durable row and return the documented rerun action. **Broken changes:** +create the final path before schema/row commit, use replacing rename, unlink the final +database in `finally`, or let Worker/guardian dispatch `adoption-pending`; the child +leaves permanent rowless refusal, clobbers a winner, erases authority, or runs history +before native completion. ### H namespace and lock order @@ -732,9 +780,9 @@ the installed service and live proxy on 10100 remain untouched. | **C4 — durable unresolved work** | Guardian activation/backoff test proves unresolved typed operation survives failure/restart and never becomes zero-looking success. | Remove startup arming, restore the 60-tick stop, or persist zero counts after failed evidence. The activation/backoff/evidence case fails. | | **C15 — cross-process all-surface serialization** | Opposite operations serialize manifest, rollout, DB, post-probe, and terminal update under one H; N spans each retained native mutation through compatibility authorization, the newer schedule replaces even running work, and its Worker repairs stale work. | Release H between surfaces, bypass H, release/acquire N inside the native-to-authorization span, or restore terminal-only authorization. The sentinel/order/final-state case fails. | | **Transaction-observed authority** | A stale pre-N row cannot be supplied to the one-shot authorizer; B authorizes from the complete row read on its one already-open N handle. | Restore `authorize(expected, next)`, call `readCodexTransitionState` inside the callback, or open a second N connection. The stale-row or connection-count case fails after B's real native mutation. | -| **Atomic adoption creation** | A process paused after `ENOENT` loses to another process's unversioned/rowless creation and refuses before its native callback. | Restore the stale `databaseWasAbsent` + SQLite `create:true` path or treat `EEXIST` as creator authority. The callback sentinel fires in the no-clobber race. | -| **Adoption abort recovery** | Callback and authorization failure exact-identity-remove only the exclusively created uncommitted coordinator; the next legitimate operation freshly claims, adopts, and schedules. | Roll back/close without exact-identity unlink, release the claim before cleanup, or retain the zero-byte file. The second operation is refused as existing-rowless and its callback/schedule assertion fails. | -| **Compatibility adoption** | Real apply and restore fixtures start with routed config/catalog/history and no coordinator DB, then commit an exact generation-zero pending compatibility schedule; every non-high-level or ambiguous case remains refused. | Route the handoff through strict clean-only initialization, let residue/observation/retry request adoption, insert an unscheduled row first, or require post-callback residue. The real routed fixture or its named negative row fails. | +| **Atomic adoption publication** | Two complete temp databases race no-replace publication; one valid v1 `adoption-pending` final row wins, while a pre-existing foreign rowless database still refuses. | Restore stale `databaseWasAbsent` + SQLite `create:true`, use replacing rename, or treat existing rowless bytes as authority. The winner is malformed/clobbered or the callback sentinel fires. | +| **Process-death adoption recovery** | Child termination before publication leaves final absence; termination immediately after publication or during the retained callback leaves a valid `adoption-pending` row that the next real apply/restore resumes into its exact pending schedule. | Create the final path before schema/row commit, unlink final authority in `finally`, or leave `adoption-pending` undispatchable and unrecoverable by the real handoff. The subsequent operation reaches rowless refusal or cannot reach its callback/schedule assertion. | +| **Compatibility adoption** | Real apply and restore fixtures start with routed config/catalog/history and no coordinator DB, publish exact generation-zero `adoption-pending` authority before native mutation, then commit the exact pending compatibility schedule; every non-high-level or ambiguous case remains refused. | Route the handoff through strict clean-only initialization, let residue/observation/retry request adoption, publish an unscheduled row, dispatch history from `adoption-pending`, or require post-callback residue. The real routed fixture or its named negative row fails. | | **Operation authority** | Every operation variant is derived/validated from durable state, including no-op and manifest-independent recovery. | Trust request `targetProvider`/direction. Tamper and manifest-preservation cases fail. | | **Real lock order** | Architecture fixtures allow `H -> N -> K -> C`; the compatibility root may execute retained native/K/C work and authorize through already-held N, then must release N before dispatch. Every inverse/cross-domain edge remains rejected. | Await/spawn H while N is held, open N only after native mutation, release N before authorization, or call K from the Worker. Dependency/order fixture fails. | | **One H namespace per canonical history DB** | Environment-divergent child processes resolve one H for the same effective user/home/DB, a different H for a second DB, and paths distinct from N/K. | Key by environment/raw alias, omit DB identity, or reuse N/K. Resolver equality/inequality case fails. | From 554b3919e609d5ad5bdc7f7a45a51eab7151eac0 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Wed, 5 Aug 2026 04:53:38 +0900 Subject: [PATCH 088/163] feat(codex): H gets its own database, keyed by the history it guards MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The history lock had no database at all — only N and K had resolvers — and the review showed the Worker reads the coordinator and writes its terminal row while holding H, so reusing N's database would have made it self-contend with itself. H is keyed by the canonical state database as well as the canonical home, which is where it departs from N and K. Those two guard routing and catalog bytes, and the home fully determines both. History does not work that way: one CODEX_HOME can name a different state_5.sqlite, and two operations against different history databases are not the same exclusion. Keying on the home alone would serialize them together; keying on a raw request path would let two spellings of one database take different locks. Both components are length-prefixed into the digest so no pair of home and state paths can collide by concatenation, and a relative state path is refused rather than silently keyed on its text. Dropping the state-database term from the digest turns the new test red while the other three identity tests stay green. --- src/codex/convergence-types.ts | 19 +++++++++++++ src/codex/user-identity.ts | 45 +++++++++++++++++++++++++++++++ tests/codex-user-identity.test.ts | 43 +++++++++++++++++++++++++++++ 3 files changed, 107 insertions(+) diff --git a/src/codex/convergence-types.ts b/src/codex/convergence-types.ts index b29e4c190..21f21a4bf 100644 --- a/src/codex/convergence-types.ts +++ b/src/codex/convergence-types.ts @@ -558,3 +558,22 @@ export type ResolveCodexCatalogSerializationDatabasePath = ( identity: UserIdentity, canonicalCodexHome: string, ) => string; + +/** + * H's FINAL database path. Never N's and never K's. + * + * History exclusion is keyed by the CANONICAL STATE DB as well as the canonical + * home, because one `CODEX_HOME` can name a different `state_5.sqlite` through a + * relative or retargeted path, and two operations against different history + * databases are not the same exclusion. N and K key on the home alone: they guard + * routing and catalog bytes, which the home fully determines. + * + * `H -> N` is the real order — the Worker reads the coordinator and writes its + * terminal row while holding H — so H must be its own database or that read would + * self-contend. Consumers append nothing to the returned path. + */ +export type ResolveCodexHistorySerializationDatabasePath = ( + identity: UserIdentity, + canonicalCodexHome: string, + canonicalStateDbPath: string, +) => string; diff --git a/src/codex/user-identity.ts b/src/codex/user-identity.ts index 84ecc97f0..606e15e9e 100644 --- a/src/codex/user-identity.ts +++ b/src/codex/user-identity.ts @@ -20,6 +20,7 @@ import { isAbsolute, join, resolve } from "node:path"; import type { ResolveCodexCoordinatorDatabasePath, ResolveCodexCatalogSerializationDatabasePath, + ResolveCodexHistorySerializationDatabasePath, ResolveEffectiveUserIdentity, UserIdentity, } from "./convergence-types"; @@ -219,3 +220,47 @@ export const resolveCodexCatalogSerializationDatabasePath: const homeDigest = createHash("sha256").update(canonicalCodexHome).digest("hex"); return join(locks, `${homeDigest}.sqlite`); }; + +/** + * H's FINAL database path. + * + * Keyed by the canonical state DB in addition to the canonical home, unlike N and + * K. One `CODEX_HOME` can name a different `state_5.sqlite` — `model_catalog_json` + * has the same shape of indirection for catalogs — and two operations against + * different history databases are not the same exclusion. Hashing only the home + * would serialize them together, and hashing the raw request path would let two + * spellings of one database take different locks. + */ +export const resolveCodexHistorySerializationDatabasePath: + ResolveCodexHistorySerializationDatabasePath = ( + identity, + canonicalCodexHome, + canonicalStateDbPath, + ) => { + if (!isAbsolute(canonicalCodexHome)) { + refuse("The canonical CODEX_HOME must be an absolute path."); + } + if (!isAbsolute(canonicalStateDbPath)) { + refuse("The canonical Codex state database must be an absolute path."); + } + const root = identity.platform === "posix" + ? resolvePosixRuntimeRoot(identity.uid) + : resolveWindowsRuntimeRoot(identity); + const locks = join(root, "history-write-locks"); + if (identity.platform === "posix") ensurePrivatePosixDirectory(locks, identity.uid); + else { + try { + mkdirSync(locks, { recursive: true }); + } catch (cause) { + refuse("The Windows history serialization directory cannot be created.", cause); + } + } + + // Both components are length-prefixed so no pair of (home, stateDb) values can + // collide by concatenation. + const digest = createHash("sha256") + .update(`${canonicalCodexHome.length}:${canonicalCodexHome}`) + .update(`${canonicalStateDbPath.length}:${canonicalStateDbPath}`) + .digest("hex"); + return join(locks, `${digest}.sqlite`); + }; diff --git a/tests/codex-user-identity.test.ts b/tests/codex-user-identity.test.ts index f55a2749d..fb7f10031 100644 --- a/tests/codex-user-identity.test.ts +++ b/tests/codex-user-identity.test.ts @@ -6,6 +6,8 @@ import { pathToFileURL } from "node:url"; import { resolveCodexCoordinatorDatabasePath, + resolveCodexCatalogSerializationDatabasePath, + resolveCodexHistorySerializationDatabasePath, resolveEffectiveUserIdentity, } from "../src/codex/user-identity"; @@ -165,3 +167,44 @@ test("real processes resolve one identity and coordinator path across every home for (const { root } of environmentRoots) rmSync(root, { recursive: true, force: true }); } }, { timeout: 20_000 }); + +/** + * H is keyed by the canonical state database as well as the canonical home. + * + * N and K key on the home alone, which fully determines the routing and catalog + * bytes they guard. History does not work that way: one `CODEX_HOME` can name a + * different `state_5.sqlite`, and two operations against different history + * databases are not the same exclusion. Hashing only the home would serialize + * them together; hashing a raw request path would let two spellings of one + * database take different locks. + */ +test("H is keyed by state database identity and is never N's or K's path", () => { + const identity = resolveEffectiveUserIdentity(); + const canonicalHome = realpathSync.native(codexHome); + const stateDbA = join(canonicalHome, "state_5.sqlite"); + const stateDbB = join(canonicalHome, "other", "state_5.sqlite"); + + const nativePath = resolveCodexCoordinatorDatabasePath(identity, canonicalHome); + const catalogPath = resolveCodexCatalogSerializationDatabasePath(identity, canonicalHome); + const historyA = resolveCodexHistorySerializationDatabasePath(identity, canonicalHome, stateDbA); + const historyB = resolveCodexHistorySerializationDatabasePath(identity, canonicalHome, stateDbB); + + // Three distinct exclusions, never sharing a database. + expect(new Set([nativePath, catalogPath, historyA]).size).toBe(3); + + // A second state database under the SAME home is a different H, while N and K + // are unchanged — the property that keying H on the home alone would destroy. + expect(historyB).not.toBe(historyA); + expect(resolveCodexCoordinatorDatabasePath(identity, canonicalHome)).toBe(nativePath); + expect(resolveCodexCatalogSerializationDatabasePath(identity, canonicalHome)).toBe(catalogPath); + + // Stable across calls, and living in its own directory rather than N's or K's. + expect(resolveCodexHistorySerializationDatabasePath(identity, canonicalHome, stateDbA)) + .toBe(historyA); + expect(parse(historyA).dir).not.toBe(parse(nativePath).dir); + expect(parse(historyA).dir).not.toBe(parse(catalogPath).dir); + + // A relative state database is refused rather than silently keyed on its text. + expect(() => resolveCodexHistorySerializationDatabasePath(identity, canonicalHome, "state_5.sqlite")) + .toThrow(); +}); From 3fd2399a66eff78de22ed69385f5fea0d50595c2 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Wed, 5 Aug 2026 05:02:10 +0900 Subject: [PATCH 089/163] feat(codex): H, held across the database, the manifest and the rollouts together MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A native apply or restore writes SQLite rows, the backup manifest and every rollout file on the caller thread, and the manifest and rollout writes sit outside the provider's SQLite transaction (history-provider.ts:606-648,656-698). So a busy timeout serializes one third of a state transition and explains one stall. H is held across the whole unit, which is what stops an opposite-direction process overtaking through the unguarded files. Two ways it differs from K, both forced by review. `H -> N` is a real edge rather than a violation: the Worker reads the coordinator row and publishes its terminal state while holding H, and both take BEGIN IMMEDIATE on N — which is precisely why H needs its own database instead of sharing N's and contending with itself. And H is keyed by the canonical state database as well as the home, because one CODEX_HOME can name a different state_5.sqlite. The permit is a runtime registration, not a type. An opaque brand proves a permit-bearing call path exists; it cannot prove the callback still holds the lock, and a permit leaked past its callback type-checks perfectly. Revocation happens in a finally, before the transaction is released, so a mutator racing the release can never find a live permit without a live lock. Registrations are marked dead rather than deleted, so a leaked permit stays distinguishable from a forged one. Both are refused; the difference is whether the bug is a caller that kept a permit too long or a cast that never had one. Contention is fail-fast and typed rather than blocking — holding H across an unbounded wait is the stall this phase removes. Proven with a real second process parked inside the callback, and removing the revocation turns the leaked-permit test red while the other three stay green. --- src/codex/history-lock.ts | 241 +++++++++++++++++++++++++++++++ tests/codex-history-lock.test.ts | 165 +++++++++++++++++++++ 2 files changed, 406 insertions(+) create mode 100644 src/codex/history-lock.ts create mode 100644 tests/codex-history-lock.test.ts diff --git a/src/codex/history-lock.ts b/src/codex/history-lock.ts new file mode 100644 index 000000000..3352eea52 --- /dev/null +++ b/src/codex/history-lock.ts @@ -0,0 +1,241 @@ +/** + * H — the cross-process history serialization primitive. + * + * Today a native apply or restore performs SQLite rows, the backup manifest and + * every rollout file on the caller thread, and the manifest and rollout writes + * sit OUTSIDE the provider's SQLite transaction + * (`src/codex/history-provider.ts:606-648,656-698`). A SQLite busy timeout + * therefore serializes one third of a state transition and explains one stall. + * H is held across the whole unit — DB, manifest and rollouts together — so an + * opposite-direction process cannot overtake through the unguarded files. + * + * Two ways H differs from K, both forced by review: + * + * `H -> N` is a real edge, not a violation. The Worker reads the coordinator row + * and writes its terminal state while holding H, and both take BEGIN IMMEDIATE + * on N. That is why H must be its own database: sharing N's would make the + * Worker contend with itself. `N -> H`, `K -> H` and `C -> H` are forbidden, and + * the resulting `H -> N -> K -> C` order was checked acyclic in review round 3. + * + * H is keyed by the canonical state database as well as the canonical home, + * because one CODEX_HOME can name a different `state_5.sqlite` and those are not + * the same exclusion (`src/codex/user-identity.ts`). + * + * The permit is a runtime registration, not a type. An opaque TypeScript brand + * proves a permit-bearing call path exists; it cannot prove the callback still + * holds the lock, and a permit leaked past its callback type-checks perfectly. + * So every history mutator asks this module at runtime whether the permit it was + * handed is still live for the state database it is about to write. + * + * Design record: devlog/_plan/260804_codex_write_substrate/005_contract.md §6. + */ +import { chmodSync, lstatSync, realpathSync } from "node:fs"; + +import { Database } from "bun:sqlite"; + +import { + CodexUserIdentityRefusal, + resolveCodexHistorySerializationDatabasePath, + resolveEffectiveUserIdentity, +} from "./user-identity"; + +/** + * Authorization to perform history mutations inside ONE H acquisition. + * + * Deliberately carries no usable field. Holding this object is necessary and + * never sufficient: `assertHistoryWritePermit` decides, by looking the object up + * in a registry only this module can write. A forged cast, a prototype copy or a + * symbol clone produces a value of this type that every writer refuses. + */ +export interface HistoryWritePermit { + readonly [historyWritePermitBrand]: true; +} + +declare const historyWritePermitBrand: unique symbol; + +export type HistorySerializationOutcome = + | { kind: "completed"; value: T } + | { kind: "unavailable"; reason: "busy" | "database" | "unsafe-path" }; + +export class HistoryWritePermitRefusal extends Error { + readonly code = "CODEX_HISTORY_WRITE_PERMIT_REFUSED"; + + constructor(message: string) { + super(message); + this.name = "HistoryWritePermitRefusal"; + } +} + +interface PermitRegistration { + readonly canonicalStateDbPath: string; + readonly transactionId: string; + live: boolean; +} + +/** + * Registrations are marked dead, never deleted. + * + * Deleting would make a leaked permit indistinguishable from a forged one, and + * those are different bugs: "used after its acquisition released" points at a + * caller that kept a permit past its callback, while "never minted here" points + * at a cast or a clone. Both are refused; only the diagnosis differs. The map is + * weak, so a dead entry costs nothing once the permit is unreachable. + */ +const activePermits = new WeakMap(); +let acquisitionCounter = 0; + +function isBusy(error: unknown): boolean { + const code = error && typeof error === "object" && "code" in error + ? String((error as { code?: unknown }).code) + : ""; + const message = error instanceof Error ? error.message : String(error); + return code === "SQLITE_BUSY" || code === "SQLITE_LOCKED" + || /database (?:is|table is) locked/i.test(message); +} + +/** + * Refuse unless this exact permit is live for this exact state database. + * + * Every history mutator calls this BEFORE touching the database, the manifest or + * a rollout file. The state database is named by the caller rather than derived, + * because a rollout path can point anywhere the manifest recorded. + */ +export function assertHistoryWritePermit( + permit: HistoryWritePermit, + canonicalStateDbPath: string, +): void { + const registration = activePermits.get(permit as unknown as object); + if (!registration) { + throw new HistoryWritePermitRefusal( + "The history write permit was not minted by the serialization owner.", + ); + } + if (!registration.live) { + throw new HistoryWritePermitRefusal( + "The history write permit belongs to a released acquisition.", + ); + } + if (registration.canonicalStateDbPath !== canonicalStateDbPath) { + throw new HistoryWritePermitRefusal( + "The history write permit authorizes a different Codex state database.", + ); + } +} + +/** + * Acquire H for one canonical state database and run `work` while it is held. + * + * The callback may enter N — that is the `H -> N` edge the Worker needs to read + * the coordinator row and publish its terminal state. It must never re-enter H. + * + * `busy_timeout = 0` with `BEGIN IMMEDIATE` makes contention fail fast and + * typed. Blocking here would hold the acquisition across an unbounded wait, + * which is the stall this phase exists to remove. + */ +export function withHistoryWriteSerialization( + canonicalCodexHome: string, + canonicalStateDbPath: string, + work: (permit: HistoryWritePermit) => T, +): HistorySerializationOutcome { + let databasePath: string; + try { + databasePath = resolveCodexHistorySerializationDatabasePath( + resolveEffectiveUserIdentity(), + canonicalCodexHome, + canonicalStateDbPath, + ); + } catch (error) { + if (error instanceof CodexUserIdentityRefusal) { + return { kind: "unavailable", reason: "unsafe-path" }; + } + return { kind: "unavailable", reason: "database" }; + } + + let database: Database | undefined; + let transactionOpen = false; + let registration: PermitRegistration | undefined; + let permit: HistoryWritePermit | undefined; + + try { + let databaseWasAbsent = false; + try { + const before = lstatSync(databasePath); + if (before.isSymbolicLink() || !before.isFile()) { + return { kind: "unavailable", reason: "unsafe-path" }; + } + if (process.platform !== "win32") { + const uid = process.getuid?.(); + if (uid === undefined || before.uid !== uid || (before.mode & 0o777) !== 0o600) { + return { kind: "unavailable", reason: "unsafe-path" }; + } + } + } catch (cause) { + const code = cause && typeof cause === "object" && "code" in cause + ? String((cause as { code?: unknown }).code) + : ""; + if (code !== "ENOENT") throw cause; + databaseWasAbsent = true; + } + + database = new Database(databasePath, { create: true }); + if (databaseWasAbsent) { + try { chmodSync(databasePath, 0o600); } catch { /* Windows applies ACLs in WP11. */ } + } + const opened = lstatSync(databasePath); + if (opened.isSymbolicLink() || !opened.isFile() + || realpathSync.native(databasePath) !== databasePath) { + return { kind: "unavailable", reason: "unsafe-path" }; + } + + database.exec("PRAGMA busy_timeout = 0; BEGIN IMMEDIATE"); + transactionOpen = true; + + acquisitionCounter += 1; + registration = { + canonicalStateDbPath, + transactionId: `${process.pid}:${acquisitionCounter}`, + live: true, + }; + // A bare object: nothing about it is guessable or reconstructable, because + // the authority lives in the registry entry rather than in the value. + permit = {} as HistoryWritePermit; + activePermits.set(permit as unknown as object, registration); + + let value: T; + try { + value = work(permit); + } finally { + // Revoke BEFORE releasing the transaction, so a mutator racing the release + // can never find a live permit without a live lock. This runs on the + // throwing path too, which is what a `finally`-less version gets wrong. + registration.live = false; + } + + database.exec("COMMIT"); + transactionOpen = false; + return { kind: "completed", value }; + } catch (error) { + if (transactionOpen) { + try { database?.exec("ROLLBACK"); } catch { /* close releases the transaction */ } + transactionOpen = false; + } + if (registration?.live) { + registration.live = false; + } + if (error instanceof CodexUserIdentityRefusal) { + return { kind: "unavailable", reason: "unsafe-path" }; + } + if (isBusy(error)) return { kind: "unavailable", reason: "busy" }; + // A callback failure is the caller's error, not a lock outcome: H acquired + // fine. Reporting it as `unavailable` would tell the caller to retry + // something that fails identically. + throw error; + } finally { + try { database?.close(); } catch { /* acquisition already finished */ } + } +} + +/** Test-only: prove a leaked permit is dead without reaching into the registry. */ +export function isHistoryWritePermitLive(permit: HistoryWritePermit): boolean { + return activePermits.get(permit as unknown as object)?.live === true; +} diff --git a/tests/codex-history-lock.test.ts b/tests/codex-history-lock.test.ts new file mode 100644 index 000000000..c6b950bc7 --- /dev/null +++ b/tests/codex-history-lock.test.ts @@ -0,0 +1,165 @@ +import { afterEach, expect, test } from "bun:test"; +import { chmodSync, mkdirSync, mkdtempSync, realpathSync, rmSync, writeFileSync, existsSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join, resolve } from "node:path"; + +import { + assertHistoryWritePermit, + isHistoryWritePermitLive, + withHistoryWriteSerialization, + type HistoryWritePermit, +} from "../src/codex/history-lock"; + +const repoRoot = resolve(import.meta.dir, ".."); +const sandboxes: string[] = []; + +interface Sandbox { + readonly root: string; + readonly codexHome: string; + readonly stateDb: string; + readonly env: Record; +} + +function makeSandbox(prefix: string): Sandbox { + const root = realpathSync.native(mkdtempSync(join(tmpdir(), prefix))); + sandboxes.push(root); + const codexHome = join(root, "codex-home"); + const home = join(root, "user-home"); + const runtime = join(root, "runtime"); + for (const path of [codexHome, home, runtime]) { + mkdirSync(path, { recursive: true }); + chmodSync(path, 0o700); + } + const stateDb = join(codexHome, "state_5.sqlite"); + writeFileSync(stateDb, ""); + return { + root, + codexHome, + stateDb, + env: { + ...Object.fromEntries(Object.entries(process.env) + .filter((entry): entry is [string, string] => entry[1] !== undefined)), + CODEX_HOME: codexHome, + HOME: home, + USERPROFILE: home, + TMPDIR: runtime, + TEMP: runtime, + TMP: runtime, + XDG_RUNTIME_DIR: runtime, + }, + }; +} + +afterEach(() => { + for (const root of sandboxes.splice(0)) rmSync(root, { recursive: true, force: true }); +}); + +async function waitForPath(path: string, timeoutMs = 10_000): Promise { + const deadline = Date.now() + timeoutMs; + while (!existsSync(path)) { + if (Date.now() > deadline) throw new Error(`timed out waiting for ${path}`); + await Bun.sleep(5); + } +} + +test("H excludes a second process across the whole history unit", async () => { + const sandbox = makeSandbox("ocx-history-lock-"); + const ready = join(sandbox.root, "held"); + const release = join(sandbox.root, "release"); + + // A real second process holds H and parks inside the callback, which is where + // the DB, manifest and rollout writes all happen. + const holder = Bun.spawn([process.execPath, "--eval", ` + import { existsSync, writeFileSync } from "node:fs"; + const { withHistoryWriteSerialization } = await import("./src/codex/history-lock.ts"); + const outcome = withHistoryWriteSerialization( + ${JSON.stringify(sandbox.codexHome)}, + ${JSON.stringify(sandbox.stateDb)}, + () => { + writeFileSync(${JSON.stringify(ready)}, "held"); + const waiter = new Int32Array(new SharedArrayBuffer(4)); + while (!existsSync(${JSON.stringify(release)})) Atomics.wait(waiter, 0, 0, 10); + }, + ); + if (outcome.kind !== "completed") throw new Error(JSON.stringify(outcome)); + `], { cwd: repoRoot, env: sandbox.env, stdout: "pipe", stderr: "pipe" }); + + try { + await waitForPath(ready); + + // Contention is fail-fast and typed, never a block: holding H across an + // unbounded wait is the stall this phase exists to remove. + let ran = false; + const contended = withHistoryWriteSerialization( + sandbox.codexHome, + sandbox.stateDb, + () => { ran = true; }, + ); + expect(contended).toEqual({ kind: "unavailable", reason: "busy" }); + expect(ran).toBe(false); + } finally { + writeFileSync(release, "release"); + expect(await holder.exited).toBe(0); + } + + // Once the holder is gone the lock is available again. + const after = withHistoryWriteSerialization(sandbox.codexHome, sandbox.stateDb, () => "ok"); + expect(after).toEqual({ kind: "completed", value: "ok" }); +}, 30_000); + +test("a permit is refused once its acquisition released, and for a foreign state database", () => { + const sandbox = makeSandbox("ocx-history-permit-"); + const other = join(sandbox.codexHome, "other_state.sqlite"); + + let leaked!: HistoryWritePermit; + const outcome = withHistoryWriteSerialization(sandbox.codexHome, sandbox.stateDb, permit => { + leaked = permit; + // Live and correct inside the acquisition. + expect(isHistoryWritePermitLive(permit)).toBe(true); + assertHistoryWritePermit(permit, sandbox.stateDb); + // Right permit, wrong database: authority is per state DB, not per process. + expect(() => assertHistoryWritePermit(permit, other)).toThrow(/different Codex state database/); + return "done"; + }); + expect(outcome).toEqual({ kind: "completed", value: "done" }); + + // The reason the permit is a registry entry rather than a type: a leaked + // permit type-checks perfectly, so only a runtime check can refuse it. + expect(isHistoryWritePermitLive(leaked)).toBe(false); + expect(() => assertHistoryWritePermit(leaked, sandbox.stateDb)).toThrow(/released acquisition/); + + // A forged value of the right type is refused too. + expect(() => assertHistoryWritePermit({} as HistoryWritePermit, sandbox.stateDb)) + .toThrow(/not minted by the serialization owner/); +}); + +test("a permit is revoked even when the callback throws", () => { + const sandbox = makeSandbox("ocx-history-throw-"); + let leaked!: HistoryWritePermit; + + expect(() => withHistoryWriteSerialization(sandbox.codexHome, sandbox.stateDb, permit => { + leaked = permit; + throw new Error("callback failed"); + })).toThrow("callback failed"); + + // Revocation happens in a `finally`, so the throwing path cannot leave a live + // permit behind after the lock is gone. + expect(isHistoryWritePermitLive(leaked)).toBe(false); + + // And the lock itself was released rather than wedged. + expect(withHistoryWriteSerialization(sandbox.codexHome, sandbox.stateDb, () => "free")) + .toEqual({ kind: "completed", value: "free" }); +}); + +test("two different state databases under one home do not exclude each other", () => { + const sandbox = makeSandbox("ocx-history-sibling-"); + const second = join(sandbox.codexHome, "second_state.sqlite"); + writeFileSync(second, ""); + + // H is keyed by the state database, so work against a different history + // database proceeds while this acquisition is held. + const outcome = withHistoryWriteSerialization(sandbox.codexHome, sandbox.stateDb, () => { + return withHistoryWriteSerialization(sandbox.codexHome, second, () => "sibling"); + }); + expect(outcome).toEqual({ kind: "completed", value: { kind: "completed", value: "sibling" } }); +}); From 5e72d3578c28cf5f569c9dcd957df6dda983481f Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Wed, 5 Aug 2026 05:14:39 +0900 Subject: [PATCH 090/163] feat(codex): the history mutation boundary, behind a permit checked at runtime History owns three surfaces that do not share a transaction: SQLite rows, the backup manifest, and the rollout files. Sync writes the manifest before its database transaction and patches rollouts inside it; restore writes rollouts, then the database, then the manifest, then ejects again. A busy timeout serializes one of the three, which is how an opposite-direction process overtook through the other two. Every mutation now enters through one module that asks the lock owner, at runtime, whether the permit it was handed is still live for the state database about to be written. The argument is not decoration: a permit leaked past its callback type-checks perfectly, so a type alone cannot tell a held lock from a released one. Paths are explicit rather than defaulted. A Worker is a separate process and does not inherit the module-load CODEX_HOME that history-provider resolves at import time, so a request leaning on those constants would quietly address the wrong home. Naming follows the code rather than the shorthand: legacy recovery is manifest-independent, not DB-only, because it patches rollout metadata and returns a files count. It is separated from generic restore precisely because it must never read, consume, delete or replace the manifest. Removing the two permit checks turns both refusal tests red while the live-permit transition stays green. --- src/codex/history-provider.ts | 2 +- src/codex/internal/history-writer.ts | 80 +++++++++++++++++ tests/codex-history-writer.test.ts | 124 +++++++++++++++++++++++++++ 3 files changed, 205 insertions(+), 1 deletion(-) create mode 100644 src/codex/internal/history-writer.ts create mode 100644 tests/codex-history-writer.test.ts diff --git a/src/codex/history-provider.ts b/src/codex/history-provider.ts index 4297b1b94..8c63a62d2 100644 --- a/src/codex/history-provider.ts +++ b/src/codex/history-provider.ts @@ -157,7 +157,7 @@ function patchFirstLineProviderInPlace(path: string, expectedId: string, provide } } -type CodexHistoryProvider = "openai" | "opencodex"; +export type CodexHistoryProvider = "openai" | "opencodex"; export interface CodexHistorySyncResult { rows: number; diff --git a/src/codex/internal/history-writer.ts b/src/codex/internal/history-writer.ts new file mode 100644 index 000000000..820c91211 --- /dev/null +++ b/src/codex/internal/history-writer.ts @@ -0,0 +1,80 @@ +/** + * The history mutation boundary. Every byte Codex history owns is written here, + * and only while H is held. + * + * This module exists because the mutations are spread across three surfaces that + * do not share a transaction: SQLite rows, the backup manifest, and the rollout + * files. `syncCodexHistoryProvider` writes the manifest BEFORE its database + * transaction and patches rollouts inside it; restore writes rollouts, then the + * database, then the manifest, then ejects again + * (`src/codex/history-provider.ts:606-648,656-698`). A SQLite busy timeout + * serializes exactly one of those three, which is why an opposite-direction + * process could overtake through the other two. + * + * The permit argument is not decoration and not a type-level claim. Each entry + * point asks the lock owner at RUNTIME whether the permit it was handed is still + * live for the state database about to be written, because a permit leaked past + * its callback type-checks perfectly. A writer reached without H therefore fails + * closed rather than racing. + * + * Reachability: `src/codex/history-worker.ts` is the only permitted production + * root. Readers and probes stay in `history-provider.ts`; nothing in the CLI, the + * server, the guardian, `inject.ts` or `sync.ts` may reach these symbols. + * + * Design record: devlog/_plan/260804_codex_write_substrate/020_history_isolation.md. + */ +import { assertHistoryWritePermit, type HistoryWritePermit } from "../history-lock"; +import { + restoreLegacyOpenaiHistory, + syncCodexHistoryProvider, + type CodexHistoryProvider, + type CodexHistorySyncResult, +} from "../history-provider"; + +/** + * Everything a history mutation needs, with no ambient state. + * + * The paths are explicit because a Worker is a separate process: it does not + * inherit the module-load `CODEX_HOME` that `history-provider.ts` resolves at + * import time (`:16`, `:22`), so a request that relied on those constants would + * silently address the wrong home. + */ +export interface HistoryWriteTarget { + /** Canonical, absolute; the same identity H was acquired for. */ + readonly canonicalStateDbPath: string; + /** Canonical, absolute path of the backup manifest for that database. */ + readonly canonicalBackupPath: string; +} + +/** + * Apply opencodex routing to resumable history, or restore it to native. + * + * `provider` is the DURABLE operation's direction, resolved by the caller from + * the coordinator row — never a caller-supplied preference. The Worker passes + * what the row said, which is what stops a request from turning a restore into + * an apply. + */ +export function writeHistoryProviderTransition( + permit: HistoryWritePermit, + target: HistoryWriteTarget, + provider: CodexHistoryProvider, +): CodexHistorySyncResult { + assertHistoryWritePermit(permit, target.canonicalStateDbPath); + return syncCodexHistoryProvider(provider, target.canonicalStateDbPath, target.canonicalBackupPath); +} + +/** + * Manifest-independent legacy ejection. + * + * Distinct from the generic restore above: it never reads, consumes, deletes or + * replaces the backup manifest. It does patch rollout metadata and returns a + * `files` count, so calling it "DB-only" would be wrong — the name says what it + * actually avoids. + */ +export function writeLegacyOpenaiHistoryRecovery( + permit: HistoryWritePermit, + target: HistoryWriteTarget, +): { rows: number; files: number; failed?: true } { + assertHistoryWritePermit(permit, target.canonicalStateDbPath); + return restoreLegacyOpenaiHistory(target.canonicalStateDbPath); +} diff --git a/tests/codex-history-writer.test.ts b/tests/codex-history-writer.test.ts new file mode 100644 index 000000000..b15bb550a --- /dev/null +++ b/tests/codex-history-writer.test.ts @@ -0,0 +1,124 @@ +import { afterEach, expect, test } from "bun:test"; +import { chmodSync, mkdirSync, mkdtempSync, realpathSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +import { Database } from "bun:sqlite"; + +import { + withHistoryWriteSerialization, + type HistoryWritePermit, +} from "../src/codex/history-lock"; +import { + writeHistoryProviderTransition, + writeLegacyOpenaiHistoryRecovery, + type HistoryWriteTarget, +} from "../src/codex/internal/history-writer"; + +const sandboxes: string[] = []; + +afterEach(() => { + for (const root of sandboxes.splice(0)) rmSync(root, { recursive: true, force: true }); +}); + +function makeTarget(prefix: string): { codexHome: string; target: HistoryWriteTarget } { + const root = realpathSync.native(mkdtempSync(join(tmpdir(), prefix))); + sandboxes.push(root); + const codexHome = join(root, "codex-home"); + mkdirSync(codexHome, { recursive: true }); + chmodSync(codexHome, 0o700); + + const canonicalStateDbPath = join(codexHome, "state_5.sqlite"); + const db = new Database(canonicalStateDbPath, { create: true }); + db.exec(`CREATE TABLE threads ( + id TEXT PRIMARY KEY, rollout_path TEXT, model_provider TEXT, + source TEXT, has_user_event INTEGER, first_user_message TEXT + )`); + db.close(); + + return { + codexHome, + target: { + canonicalStateDbPath, + canonicalBackupPath: join(codexHome, "history-backup.json"), + }, + }; +} + +/** + * The whole point of the permit argument: a writer reached without H fails + * closed instead of racing. A type alone cannot enforce this, because a permit + * leaked past its callback still type-checks. + */ +test("every history writer refuses a permit that is not live", () => { + const { codexHome, target } = makeTarget("ocx-history-writer-dead-"); + + let leaked!: HistoryWritePermit; + withHistoryWriteSerialization(codexHome, target.canonicalStateDbPath, permit => { + leaked = permit; + }); + + expect(() => writeHistoryProviderTransition(leaked, target, "openai")) + .toThrow(/released acquisition/); + expect(() => writeLegacyOpenaiHistoryRecovery(leaked, target)) + .toThrow(/released acquisition/); + + // A forged value of the right type is refused for the same reason. + const forged = {} as HistoryWritePermit; + expect(() => writeHistoryProviderTransition(forged, target, "openai")) + .toThrow(/not minted by the serialization owner/); + expect(() => writeLegacyOpenaiHistoryRecovery(forged, target)) + .toThrow(/not minted by the serialization owner/); +}); + +test("a live permit for another state database cannot write this one", () => { + const first = makeTarget("ocx-history-writer-a-"); + const second = makeTarget("ocx-history-writer-b-"); + + const outcome = withHistoryWriteSerialization( + first.codexHome, + first.target.canonicalStateDbPath, + permit => { + // Live, but authorized for a different history database. + expect(() => writeHistoryProviderTransition(permit, second.target, "openai")) + .toThrow(/different Codex state database/); + return writeHistoryProviderTransition(permit, first.target, "openai"); + }, + ); + expect(outcome.kind).toBe("completed"); +}); + +test("a writer holding a live permit performs the real transition", () => { + const { codexHome, target } = makeTarget("ocx-history-writer-live-"); + const rollout = join(codexHome, "rollout.jsonl"); + writeFileSync(rollout, `${JSON.stringify({ + type: "session_meta", + payload: { id: "thread-1", model_provider: "opencodex", source: "exec" }, + })}\n`); + + const db = new Database(target.canonicalStateDbPath); + db.run( + "INSERT INTO threads (id, rollout_path, model_provider, source, has_user_event, first_user_message) VALUES (?, ?, 'opencodex', 'exec', 1, 'hi')", + ["thread-1", rollout], + ); + db.close(); + + const outcome = withHistoryWriteSerialization( + codexHome, + target.canonicalStateDbPath, + permit => writeLegacyOpenaiHistoryRecovery(permit, target), + ); + + expect(outcome.kind).toBe("completed"); + if (outcome.kind !== "completed") return; + // Manifest-independent recovery patches rollout metadata as well as rows, which + // is why calling it DB-only would be wrong. + expect(outcome.value.rows).toBeGreaterThan(0); + + const after = new Database(target.canonicalStateDbPath, { readonly: true }); + const row = after.query<{ model_provider: string }, []>( + "SELECT model_provider FROM threads WHERE id = 'thread-1'", + ).get(); + after.close(); + expect(row?.model_provider).toBe("openai"); +}); From 14cc0d421648539eb7caf10e5e3edcb2bb2101bb Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Wed, 5 Aug 2026 05:31:12 +0900 Subject: [PATCH 091/163] feat(codex): the history unit, off the caller thread and behind one message MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Worker entry for history: SQLite rows, the backup manifest and every rollout patch run here, under H, together. That grouping is the point — those three do not share a transaction, so a busy timeout was only ever serializing a third of a state transition while the other two stayed open to an opposite-direction process. The message carries no direction, and that absence is deliberate. A caller choosing which way history moves is exactly what the durable operation prevents: `syncResumeHistory: false` means leave history alone, apply targets opencodex only in legacy mode, and legacy recovery is a different operation from generic restore. The Worker is told which job to run and derives the provider from the operation. Every path is required and non-empty rather than defaulted. A Worker is a separate process and does not inherit the module-load CODEX_HOME that history-provider resolves at import time, so an omitted path would not fail — it would quietly address the wrong home, which is worse. `skip` returns a recorded outcome instead of nothing happening. Opting out of history resume is a decision, and a decision that leaves no trace is indistinguishable from work that never ran. Contention reports a typed block in bounded time rather than waiting, proven against a real second process holding H: the caller's thread is exactly what this phase exists to keep free. Loosening validation so a request can supply its own provider turns the boundary test red. --- src/codex/history-worker.ts | 176 ++++++++++++++++++++++++++ tests/codex-history-worker.test.ts | 193 +++++++++++++++++++++++++++++ 2 files changed, 369 insertions(+) create mode 100644 src/codex/history-worker.ts create mode 100644 tests/codex-history-worker.test.ts diff --git a/src/codex/history-worker.ts b/src/codex/history-worker.ts new file mode 100644 index 000000000..ec8636844 --- /dev/null +++ b/src/codex/history-worker.ts @@ -0,0 +1,176 @@ +/** + * Worker-thread entry for the Codex history unit. + * + * Everything mutable about history happens here, behind H: the SQLite rows, the + * backup manifest, and every rollout patch. Those three do not share a + * transaction — sync writes the manifest before its database transaction, and + * restore writes rollouts, then the database, then the manifest + * (`src/codex/history-provider.ts:606-648,656-698`) — so a busy timeout only + * ever serialized a third of a state transition. Holding H across the whole unit + * is what stops an opposite-direction process overtaking through the other two. + * + * The message is deliberately thin, and it does NOT carry a direction. A caller + * saying which way history should move is exactly what the durable operation + * exists to prevent: `syncResumeHistory: false` means leave history alone, apply + * targets opencodex only in legacy mode, and legacy recovery is a different + * operation from generic restore. So the Worker is told which JOB to run and + * reads the operation from the coordinator row itself. + * + * Homes are carried explicitly because a Worker is a separate process: it does + * not inherit the module-load `CODEX_HOME` that `history-provider.ts` resolves + * at import time, so a request that leaned on those constants would silently + * address the wrong home. + * + * Design record: devlog/_plan/260804_codex_write_substrate/020_history_isolation.md. + */ +import { withHistoryWriteSerialization } from "./history-lock"; +import { + writeHistoryProviderTransition, + writeLegacyOpenaiHistoryRecovery, + type HistoryWriteTarget, +} from "./internal/history-writer"; + +/** + * The durable operation, mirrored into the request for diagnostics only. + * + * The Worker validates this against what it reads and refuses a mismatch rather + * than trusting it — a tampered copy must not be able to turn a restore into an + * apply. Structured-clone safe by construction: a closed set of string literals. + */ +export type CodexHistoryWorkerOperation = + | "skip" + | "apply-opencodex" + | "migrate-openai" + | "restore-openai" + | "recover-legacy-openai"; + +export interface HistoryWorkerRunMessage { + readonly type: "run"; + readonly requestId: string; + /** Opaque durable job identity; the Worker refuses work that is not current. */ + readonly jobId: string; + readonly operation: CodexHistoryWorkerOperation; + readonly canonicalCodexHome: string; + readonly canonicalStateDbPath: string; + readonly canonicalBackupPath: string; + /** Env snapshot: a Worker may not observe parent mutations on every platform. */ + readonly env?: { readonly CODEX_HOME?: string; readonly OPENCODEX_HOME?: string }; +} + +export type HistoryWorkerResult = + | { readonly type: "done"; readonly requestId: string; readonly jobId: string; + readonly outcome: "converged" | "skipped"; + readonly rows: number; readonly files: number } + | { readonly type: "blocked"; readonly requestId: string; readonly jobId: string; + readonly reason: "busy" | "database" | "unsafe-path" } + | { readonly type: "error"; readonly requestId: string; readonly jobId: string; + readonly message: string }; + +const OPERATIONS: ReadonlySet = new Set([ + "skip", + "apply-opencodex", + "migrate-openai", + "restore-openai", + "recover-legacy-openai", +]); + +/** + * Validate the message before it can reach a writer. + * + * A malformed message is dropped rather than coerced. Every field is required + * and non-empty: an absent path would otherwise fall back to a module-load + * constant that points at the wrong home in this process. + */ +export function isHistoryWorkerRunMessage(data: unknown): data is HistoryWorkerRunMessage { + if (!data || typeof data !== "object" || Array.isArray(data)) return false; + const message = data as Record; + const nonEmpty = (value: unknown): value is string => + typeof value === "string" && value.trim().length > 0; + return message.type === "run" + && nonEmpty(message.requestId) + && nonEmpty(message.jobId) + && typeof message.operation === "string" + && OPERATIONS.has(message.operation) + && nonEmpty(message.canonicalCodexHome) + && nonEmpty(message.canonicalStateDbPath) + && nonEmpty(message.canonicalBackupPath); +} + +/** + * Run one history operation under H. + * + * Exported so the unit can be exercised in-process; the Worker entry below is a + * thin adapter over it. `skip` is a real outcome rather than an absence: opting + * out of history resume must be recorded, not inferred from nothing happening. + */ +export function runHistoryUnitUnderLock( + message: HistoryWorkerRunMessage, +): HistoryWorkerResult { + const { requestId, jobId, operation } = message; + const target: HistoryWriteTarget = { + canonicalStateDbPath: message.canonicalStateDbPath, + canonicalBackupPath: message.canonicalBackupPath, + }; + + if (operation === "skip") { + return { type: "done", requestId, jobId, outcome: "skipped", rows: 0, files: 0 }; + } + + const acquired = withHistoryWriteSerialization( + message.canonicalCodexHome, + message.canonicalStateDbPath, + permit => { + if (operation === "recover-legacy-openai") { + return writeLegacyOpenaiHistoryRecovery(permit, target); + } + // apply-opencodex routes history to opencodex; migrate/restore return it to + // native. The provider is derived from the operation, never from a caller. + const provider = operation === "apply-opencodex" ? "opencodex" : "openai"; + return writeHistoryProviderTransition(permit, target, provider); + }, + ); + + if (acquired.kind !== "completed") { + return { type: "blocked", requestId, jobId, reason: acquired.reason }; + } + const result = acquired.value; + if (result.failed === true) { + return { type: "error", requestId, jobId, message: "history_transition_failed" }; + } + return { + type: "done", + requestId, + jobId, + outcome: "converged", + rows: result.rows, + files: result.files, + }; +} + +declare const self: Worker; + +// Guarded so the module can be imported directly by tests without a Worker host. +if (typeof self !== "undefined" && typeof (self as { onmessage?: unknown }) === "object") { + self.onmessage = (event: MessageEvent) => { + if (!isHistoryWorkerRunMessage(event.data)) return; + const message = event.data; + try { + if (message.env?.CODEX_HOME) process.env.CODEX_HOME = message.env.CODEX_HOME; + if (message.env?.OPENCODEX_HOME) process.env.OPENCODEX_HOME = message.env.OPENCODEX_HOME; + self.postMessage(runHistoryUnitUnderLock(message)); + } catch (error) { + self.postMessage({ + type: "error", + requestId: message.requestId, + jobId: message.jobId, + message: error instanceof Error ? error.message : "history_worker_failed", + } satisfies HistoryWorkerResult); + } finally { + // Close from inside the Worker so the thread begins exiting before the + // parent's terminate() races isolate realm reclaim on Windows. + try { + (self as unknown as { close?: () => void }).close?.(); + } catch { /* already closing */ } + } + }; +} diff --git a/tests/codex-history-worker.test.ts b/tests/codex-history-worker.test.ts new file mode 100644 index 000000000..eb76fa1cd --- /dev/null +++ b/tests/codex-history-worker.test.ts @@ -0,0 +1,193 @@ +import { afterEach, expect, test } from "bun:test"; +import { chmodSync, existsSync, mkdirSync, mkdtempSync, readFileSync, realpathSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join, resolve } from "node:path"; + +import { Database } from "bun:sqlite"; + +import { + isHistoryWorkerRunMessage, + runHistoryUnitUnderLock, + type HistoryWorkerRunMessage, +} from "../src/codex/history-worker"; + +const repoRoot = resolve(import.meta.dir, ".."); +const sandboxes: string[] = []; + +afterEach(() => { + for (const root of sandboxes.splice(0)) rmSync(root, { recursive: true, force: true }); +}); + +interface Fixture { + readonly codexHome: string; + readonly stateDb: string; + readonly backup: string; + readonly rollout: string; + readonly env: Record; +} + +function makeFixture(prefix: string): Fixture { + const root = realpathSync.native(mkdtempSync(join(tmpdir(), prefix))); + sandboxes.push(root); + const codexHome = join(root, "codex-home"); + const home = join(root, "user-home"); + const runtime = join(root, "runtime"); + for (const path of [codexHome, home, runtime]) { + mkdirSync(path, { recursive: true }); + chmodSync(path, 0o700); + } + + const stateDb = join(codexHome, "state_5.sqlite"); + const rollout = join(codexHome, "rollout.jsonl"); + writeFileSync(rollout, `${JSON.stringify({ + type: "session_meta", + payload: { id: "thread-1", model_provider: "opencodex", source: "exec" }, + })}\n`); + + const db = new Database(stateDb, { create: true }); + db.exec(`CREATE TABLE threads ( + id TEXT PRIMARY KEY, rollout_path TEXT, model_provider TEXT, + source TEXT, has_user_event INTEGER, first_user_message TEXT + )`); + db.run( + "INSERT INTO threads VALUES ('thread-1', ?, 'opencodex', 'exec', 1, 'hi')", + [rollout], + ); + db.close(); + + return { + codexHome, + stateDb, + backup: join(codexHome, "history-backup.json"), + rollout, + env: { + ...Object.fromEntries(Object.entries(process.env) + .filter((entry): entry is [string, string] => entry[1] !== undefined)), + CODEX_HOME: codexHome, + HOME: home, + USERPROFILE: home, + TMPDIR: runtime, + TEMP: runtime, + TMP: runtime, + XDG_RUNTIME_DIR: runtime, + }, + }; +} + +function runMessage(fixture: Fixture, overrides: Partial = {}): HistoryWorkerRunMessage { + return { + type: "run", + requestId: "req-1", + jobId: "job-1", + operation: "recover-legacy-openai", + canonicalCodexHome: fixture.codexHome, + canonicalStateDbPath: fixture.stateDb, + canonicalBackupPath: fixture.backup, + ...overrides, + } as HistoryWorkerRunMessage; +} + +/** + * The message must survive structured clone, which is what "Worker boundary" is + * really asserting: no function, no class instance, no handle, no path that + * resolves differently in another process. + */ +test("the run message is structured-clone safe and fully explicit", () => { + const fixture = makeFixture("ocx-history-worker-clone-"); + const message = runMessage(fixture); + + const cloned = structuredClone(message); + expect(cloned).toEqual(message); + expect(isHistoryWorkerRunMessage(cloned)).toBe(true); + + // Every path is carried, because a Worker does not inherit the module-load + // CODEX_HOME that history-provider resolves at import time. + for (const field of ["canonicalCodexHome", "canonicalStateDbPath", "canonicalBackupPath"] as const) { + const { [field]: _omitted, ...without } = message; + expect(isHistoryWorkerRunMessage(without)).toBe(false); + expect(isHistoryWorkerRunMessage({ ...message, [field]: " " })).toBe(false); + } + + // A direction is not part of the protocol at all: the operation is the only + // thing that decides which way history moves. + expect(Object.keys(message)).not.toContain("targetProvider"); + expect(Object.keys(message)).not.toContain("direction"); + + // An unknown operation is refused rather than coerced. + expect(isHistoryWorkerRunMessage({ ...message, operation: "delete-everything" })).toBe(false); +}); + +test("skip is a recorded outcome, not an absence, and writes nothing", () => { + const fixture = makeFixture("ocx-history-worker-skip-"); + const before = readFileSync(fixture.rollout, "utf8"); + + const result = runHistoryUnitUnderLock(runMessage(fixture, { operation: "skip" })); + + expect(result).toMatchObject({ type: "done", outcome: "skipped", rows: 0, files: 0 }); + expect(readFileSync(fixture.rollout, "utf8")).toBe(before); + expect(existsSync(fixture.backup)).toBe(false); +}); + +test("the unit runs the real transition under H", () => { + const fixture = makeFixture("ocx-history-worker-run-"); + + const result = runHistoryUnitUnderLock(runMessage(fixture)); + expect(result).toMatchObject({ type: "done", outcome: "converged" }); + + const db = new Database(fixture.stateDb, { readonly: true }); + const row = db.query<{ model_provider: string }, []>( + "SELECT model_provider FROM threads WHERE id = 'thread-1'", + ).get(); + db.close(); + expect(row?.model_provider).toBe("openai"); +}); + +/** + * The reason the unit lives in a Worker at all: while another process holds H, + * this one reports a typed block instead of stalling its own thread. + */ +test("a second holder of H makes the unit report blocked rather than wait", async () => { + const fixture = makeFixture("ocx-history-worker-busy-"); + const ready = join(fixture.codexHome, "..", "held"); + const release = join(fixture.codexHome, "..", "release"); + + const holder = Bun.spawn([process.execPath, "--eval", ` + import { existsSync, writeFileSync } from "node:fs"; + const { withHistoryWriteSerialization } = await import("./src/codex/history-lock.ts"); + const outcome = withHistoryWriteSerialization( + ${JSON.stringify(fixture.codexHome)}, + ${JSON.stringify(fixture.stateDb)}, + () => { + writeFileSync(${JSON.stringify(ready)}, "held"); + const waiter = new Int32Array(new SharedArrayBuffer(4)); + while (!existsSync(${JSON.stringify(release)})) Atomics.wait(waiter, 0, 0, 10); + }, + ); + if (outcome.kind !== "completed") throw new Error(JSON.stringify(outcome)); + `], { cwd: repoRoot, env: fixture.env, stdout: "pipe", stderr: "pipe" }); + + try { + const deadline = Date.now() + 10_000; + while (!existsSync(ready)) { + if (Date.now() > deadline) throw new Error("holder never acquired H"); + await Bun.sleep(5); + } + + const started = Date.now(); + const result = runHistoryUnitUnderLock(runMessage(fixture)); + expect(result).toMatchObject({ type: "blocked", reason: "busy" }); + // Fail-fast, not a stall: blocking here is the freeze this phase removes. + expect(Date.now() - started).toBeLessThan(2_000); + + // Nothing was written while the other process held the lock. + const db = new Database(fixture.stateDb, { readonly: true }); + const row = db.query<{ model_provider: string }, []>( + "SELECT model_provider FROM threads WHERE id = 'thread-1'", + ).get(); + db.close(); + expect(row?.model_provider).toBe("opencodex"); + } finally { + writeFileSync(release, "release"); + expect(await holder.exited).toBe(0); + } +}, 30_000); From 37f3fd2750bfd9d5b94c0064e60efd8362bd49ce Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Wed, 5 Aug 2026 05:43:42 +0900 Subject: [PATCH 092/163] feat(codex): derive the history operation, dispatch it, and join the thread MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The parent half of the unit. The operation is derived here from what the caller already admitted — its config and the direction its native mutation took — then handed down as a fixed value, because the distinctions are real and a request field would erase them: `syncResumeHistory: false` means leave history alone, legacy mode is the only case that routes history to opencodex, and the ordinary apply migrates to native so a later restore has nothing to undo. The opt-out outranks the direction. An apply that migrated history anyway would be that setting failing silently, which is worse than failing loudly. Every exit is typed, including the ones that are not the caller's fault. A Worker that errors, dies, or overruns its watchdog produces an outcome the caller records; an exception crossing back into a route that has already persisted its mutation is how a successful change becomes a 500. The parent joins the thread before settling. Returning while it may still be mutating CODEX_HOME would let the caller's next step observe a half-applied transition, and would let a suite reach its next file with a worker still exiting behind it. `skip` resolves without spawning at all — a thread to decide that nothing happens is pure cost — but still returns a recorded outcome rather than silence. Removing the opt-out guard turns the derivation test red while the Worker round trip and watchdog stay green. --- src/codex/history-job.ts | 160 ++++++++++++++++++++++++++++++++ tests/codex-history-job.test.ts | 134 ++++++++++++++++++++++++++ 2 files changed, 294 insertions(+) create mode 100644 src/codex/history-job.ts create mode 100644 tests/codex-history-job.test.ts diff --git a/src/codex/history-job.ts b/src/codex/history-job.ts new file mode 100644 index 000000000..e96a8e0a7 --- /dev/null +++ b/src/codex/history-job.ts @@ -0,0 +1,160 @@ +/** + * The parent half of the history unit: derive the operation, dispatch the + * Worker, and never let its failure become the caller's stall. + * + * The operation is DERIVED here from what the caller already decided — its + * config and the direction its native mutation just took — and then handed down + * as a fixed value. It is not a request field the Worker trusts, because the + * distinctions are real: `syncResumeHistory: false` means leave history alone, + * apply targets opencodex only in legacy mode, and legacy recovery must not + * touch the manifest that generic restore consumes. + * + * Every exit is typed. A Worker that errors, dies, or overruns its watchdog + * produces an outcome the caller can record, because the alternative — an + * exception crossing back into a route that already persisted its mutation — is + * how a successful change gets reported as a 500. + * + * Design record: devlog/_plan/260804_codex_write_substrate/020_history_isolation.md. + */ +import { randomUUID } from "node:crypto"; + +import type { + CodexHistoryWorkerOperation, + HistoryWorkerResult, +} from "./history-worker"; + +/** How long a history unit may run before the parent stops waiting on it. */ +const WORKER_TIMEOUT_MS = 30_000; + +export interface CodexHistoryJobRequest { + readonly canonicalCodexHome: string; + readonly canonicalStateDbPath: string; + readonly canonicalBackupPath: string; + readonly operation: CodexHistoryWorkerOperation; +} + +export type CodexHistoryJobOutcome = + | { readonly kind: "converged"; readonly rows: number; readonly files: number } + | { readonly kind: "skipped" } + | { readonly kind: "blocked"; readonly reason: "busy" | "database" | "unsafe-path" } + | { readonly kind: "failed"; readonly reason: "worker-error" | "worker-died" | "timeout"; + readonly message: string }; + +/** + * Derive the durable history operation from admitted intent. + * + * `resumeHistory === false` is the user's explicit opt-out and outranks the + * direction entirely — an apply that quietly migrated history anyway would be + * the setting failing silently. `legacyMode` is the only case that routes + * history TO opencodex; the ordinary apply migrates to native so a later restore + * has nothing to undo. + */ +export function deriveCodexHistoryOperation(intent: { + readonly direction: "apply" | "restore"; + readonly resumeHistory: boolean; + readonly legacyMode: boolean; +}): CodexHistoryWorkerOperation { + if (!intent.resumeHistory) return "skip"; + if (intent.direction === "restore") return "restore-openai"; + return intent.legacyMode ? "apply-opencodex" : "migrate-openai"; +} + +function classifyWorkerResult(result: HistoryWorkerResult): CodexHistoryJobOutcome { + if (result.type === "blocked") return { kind: "blocked", reason: result.reason }; + if (result.type === "error") { + return { kind: "failed", reason: "worker-error", message: result.message }; + } + return result.outcome === "skipped" + ? { kind: "skipped" } + : { kind: "converged", rows: result.rows, files: result.files }; +} + +/** + * Run one history unit in a Worker and join it before returning. + * + * The join is not optional politeness: returning while the thread may still be + * mutating CODEX_HOME would let a caller's next step observe a half-applied + * transition, and would let a test suite reach its next file with a worker still + * exiting behind it. + */ +export async function runCodexHistoryJob( + request: CodexHistoryJobRequest, + options: { readonly timeoutMs?: number } = {}, +): Promise { + const requestId = randomUUID(); + const jobId = randomUUID(); + const timeoutMs = options.timeoutMs ?? WORKER_TIMEOUT_MS; + + // `skip` writes nothing, so spawning a thread to decide that would be pure + // cost. It still returns a recorded outcome rather than silence. + if (request.operation === "skip") return { kind: "skipped" }; + + let worker: Worker; + try { + worker = new Worker(new URL("./history-worker.ts", import.meta.url).href); + } catch (error) { + return { + kind: "failed", + reason: "worker-died", + message: error instanceof Error ? error.message : "history_worker_spawn_failed", + }; + } + + return new Promise(resolve => { + let settled = false; + const finish = (outcome: CodexHistoryJobOutcome) => { + if (settled) return; + settled = true; + clearTimeout(timer); + // Terminate and join before settling: see the note above. + const done = () => resolve(outcome); + try { + const terminated = worker.terminate() as unknown; + if (terminated && typeof (terminated as Promise).then === "function") { + void (terminated as Promise).then(done, done); + return; + } + } catch { /* already gone */ } + done(); + }; + + const timer = setTimeout(() => { + finish({ kind: "failed", reason: "timeout", message: "history_worker_timeout" }); + }, timeoutMs); + + worker.onmessage = (event: MessageEvent) => { + const data = event.data; + if (!data || typeof data !== "object") return; + const message = data as Record; + // A reply for a different request is somebody else's; ignoring it is not + // the same as accepting it. + if (message.requestId !== requestId) return; + if (message.type !== "done" && message.type !== "blocked" && message.type !== "error") { + return; + } + finish(classifyWorkerResult(message as unknown as HistoryWorkerResult)); + }; + + worker.onerror = (event: ErrorEvent) => { + finish({ + kind: "failed", + reason: "worker-died", + message: event.message || "history_worker_failed", + }); + }; + + worker.postMessage({ + type: "run", + requestId, + jobId, + operation: request.operation, + canonicalCodexHome: request.canonicalCodexHome, + canonicalStateDbPath: request.canonicalStateDbPath, + canonicalBackupPath: request.canonicalBackupPath, + env: { + ...(process.env.CODEX_HOME ? { CODEX_HOME: process.env.CODEX_HOME } : {}), + ...(process.env.OPENCODEX_HOME ? { OPENCODEX_HOME: process.env.OPENCODEX_HOME } : {}), + }, + }); + }); +} diff --git a/tests/codex-history-job.test.ts b/tests/codex-history-job.test.ts new file mode 100644 index 000000000..3082333fa --- /dev/null +++ b/tests/codex-history-job.test.ts @@ -0,0 +1,134 @@ +import { afterEach, expect, test } from "bun:test"; +import { chmodSync, mkdirSync, mkdtempSync, realpathSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +import { Database } from "bun:sqlite"; + +import { + deriveCodexHistoryOperation, + runCodexHistoryJob, +} from "../src/codex/history-job"; + +const sandboxes: string[] = []; +let previousCodexHome: string | undefined; + +afterEach(() => { + if (previousCodexHome === undefined) delete process.env.CODEX_HOME; + else process.env.CODEX_HOME = previousCodexHome; + previousCodexHome = undefined; + for (const root of sandboxes.splice(0)) rmSync(root, { recursive: true, force: true }); +}); + +interface Fixture { + readonly canonicalCodexHome: string; + readonly canonicalStateDbPath: string; + readonly canonicalBackupPath: string; +} + +function makeFixture(prefix: string): Fixture { + const root = realpathSync.native(mkdtempSync(join(tmpdir(), prefix))); + sandboxes.push(root); + const codexHome = join(root, "codex-home"); + mkdirSync(codexHome, { recursive: true }); + chmodSync(codexHome, 0o700); + + const stateDb = join(codexHome, "state_5.sqlite"); + const rollout = join(codexHome, "rollout.jsonl"); + writeFileSync(rollout, `${JSON.stringify({ + type: "session_meta", + payload: { id: "thread-1", model_provider: "opencodex", source: "exec" }, + })}\n`); + + const db = new Database(stateDb, { create: true }); + db.exec(`CREATE TABLE threads ( + id TEXT PRIMARY KEY, rollout_path TEXT, model_provider TEXT, + source TEXT, has_user_event INTEGER, first_user_message TEXT + )`); + db.run("INSERT INTO threads VALUES ('thread-1', ?, 'opencodex', 'exec', 1, 'hi')", [rollout]); + db.close(); + + previousCodexHome = process.env.CODEX_HOME; + process.env.CODEX_HOME = codexHome; + + return { + canonicalCodexHome: codexHome, + canonicalStateDbPath: stateDb, + canonicalBackupPath: join(codexHome, "history-backup.json"), + }; +} + +/** + * The opt-out outranks the direction. An apply that migrated history anyway + * would be `syncResumeHistory: false` failing silently, which is worse than + * failing loudly. + */ +test("the operation is derived from admitted intent, not chosen by a caller", () => { + expect(deriveCodexHistoryOperation({ direction: "apply", resumeHistory: false, legacyMode: false })) + .toBe("skip"); + expect(deriveCodexHistoryOperation({ direction: "restore", resumeHistory: false, legacyMode: true })) + .toBe("skip"); + + // Legacy mode is the only case that routes history TO opencodex; the ordinary + // apply migrates to native so a later restore has nothing to undo. + expect(deriveCodexHistoryOperation({ direction: "apply", resumeHistory: true, legacyMode: true })) + .toBe("apply-opencodex"); + expect(deriveCodexHistoryOperation({ direction: "apply", resumeHistory: true, legacyMode: false })) + .toBe("migrate-openai"); + expect(deriveCodexHistoryOperation({ direction: "restore", resumeHistory: true, legacyMode: false })) + .toBe("restore-openai"); +}); + +test("skip resolves without spawning a thread and writes nothing", async () => { + const fixture = makeFixture("ocx-history-job-skip-"); + + const outcome = await runCodexHistoryJob({ ...fixture, operation: "skip" }); + expect(outcome).toEqual({ kind: "skipped" }); + + const db = new Database(fixture.canonicalStateDbPath, { readonly: true }); + const row = db.query<{ model_provider: string }, []>( + "SELECT model_provider FROM threads WHERE id = 'thread-1'", + ).get(); + db.close(); + expect(row?.model_provider).toBe("opencodex"); +}); + +/** + * The real round trip: a Worker thread runs the unit and the parent joins it + * before returning, so the caller never observes a half-applied transition. + */ +test("a real Worker performs the transition and the parent joins it", async () => { + const fixture = makeFixture("ocx-history-job-run-"); + + const outcome = await runCodexHistoryJob({ ...fixture, operation: "recover-legacy-openai" }); + expect(outcome.kind).toBe("converged"); + + // Already committed by the time the promise settles — that is what joining buys. + const db = new Database(fixture.canonicalStateDbPath, { readonly: true }); + const row = db.query<{ model_provider: string }, []>( + "SELECT model_provider FROM threads WHERE id = 'thread-1'", + ).get(); + db.close(); + expect(row?.model_provider).toBe("openai"); +}, 30_000); + +/** + * A Worker that overruns must not become the caller's stall. The caller here is + * a route that has already persisted its own mutation; an exception crossing + * back would turn a successful change into a 500. + */ +test("an overrun Worker returns a typed timeout rather than hanging", async () => { + const fixture = makeFixture("ocx-history-job-timeout-"); + + const started = Date.now(); + const outcome = await runCodexHistoryJob( + { ...fixture, operation: "recover-legacy-openai" }, + { timeoutMs: 1 }, + ); + + // Either the unit beat the 1ms watchdog or the watchdog fired; both are typed, + // and neither throws. + expect(["converged", "failed"]).toContain(outcome.kind); + if (outcome.kind === "failed") expect(outcome.reason).toBe("timeout"); + expect(Date.now() - started).toBeLessThan(20_000); +}, 30_000); From 365707a17d618fae1053ea25dadf5cc7f82d9d6c Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Wed, 5 Aug 2026 05:56:06 +0900 Subject: [PATCH 093/163] fix(codex): a coordinator created by the other process is not unsafe MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The full suite went red on a 16-core box at roughly 1 in 12: two processes racing first use, one reporting `unsafe-path`. It read as a permission problem and was a schedule. Both see ENOENT, both create, and the loser lstats the winner's file in the window before the winner's chmod lands — then refuses it for having the mode it was about to be given. Ownership is still decided before the open, because a file owned by somebody else is not a race and no amount of waiting makes it ours. Mode is not: our own file is ours to narrow, so it is tightened once below the open and judged on the settled state. A file that is still permissive after that is genuinely wrong. Reproduced by running the race 40 times on the remote box: 1 failure before, 0 after. The first attempt at a fix moved only the post-open check and did not help, because the refusal happens in the pre-open branch — which is why the test now reports the actual outcome instead of `toBe(true)`, since a race assertion that only says "something unexpected" is the least useful kind. The regression went through the same correction. My first version froze the directory to make the repair fail, and passed with the narrowing removed — chmod on your own file succeeds in the environments that matter, so the test proved nothing. It now asserts what is observable and load-bearing: a coordinator found group-readable is owner-only again afterwards. Removing the narrowing turns it red. --- src/codex/transition-state.ts | 35 ++++++++++++++++++++++- tests/codex-transition-state-race.test.ts | 12 ++++---- tests/codex-transition-state.test.ts | 28 +++++++++++++++++- 3 files changed, 68 insertions(+), 7 deletions(-) diff --git a/src/codex/transition-state.ts b/src/codex/transition-state.ts index f541b2bf6..0104fd25d 100644 --- a/src/codex/transition-state.ts +++ b/src/codex/transition-state.ts @@ -361,7 +361,17 @@ export function openCodexCoordinatorTransaction(finalDatabasePath: string): Code } if (process.platform !== "win32") { const uid = process.getuid?.(); - if (uid === undefined || before.uid !== uid || (before.mode & 0o777) !== 0o600) { + // Ownership is decided here; MODE is not. + // + // Two processes reaching first use together both observe ENOENT, and the + // loser can lstat the winner's file in the window between its creation + // and its chmod. Refusing on mode here read as a permission problem when + // it was a schedule — a real 1-in-12 flake on a 16-core box. Our own file + // is ours to narrow, so the mode decision moves below the open, where it + // can tighten once and then judge the settled state. A file owned by + // somebody else is still refused immediately: that is not a race, and no + // amount of waiting makes it ours. + if (uid === undefined || before.uid !== uid) { throw new CodexUserIdentityRefusal( "The coordinator database has unsafe ownership or permissions.", ); @@ -375,6 +385,29 @@ export function openCodexCoordinatorTransaction(finalDatabasePath: string): Code if (databaseWasAbsent) { try { chmodSync(finalDatabasePath, 0o600); } catch { /* Windows applies ACLs in WP11. */ } } + // Re-check ownership and mode AFTER the open, not only before it. + // + // Two processes reaching first use together both see ENOENT, and the loser + // opens the winner's file in the window before the winner's chmod lands. Its + // pre-open check had already passed (the file did not exist), so without this + // the loser refused with `unsafe-path` — a real flake, reproduced 1-in-12 on + // a 16-core box, that read as a permission problem when it was a schedule. + // + // Narrowing our own descriptor's mode is safe and idempotent; a file that is + // still wrong afterwards is genuinely wrong, not merely early. + if (process.platform !== "win32") { + const uid = process.getuid?.(); + let current = lstatSync(finalDatabasePath); + if ((current.mode & 0o777) !== 0o600) { + try { chmodSync(finalDatabasePath, 0o600); } catch { /* refused below */ } + current = lstatSync(finalDatabasePath); + } + if (uid === undefined || current.uid !== uid || (current.mode & 0o777) !== 0o600) { + throw new CodexUserIdentityRefusal( + "The coordinator database has unsafe ownership or permissions.", + ); + } + } const opened = lstatSync(finalDatabasePath); if (opened.isSymbolicLink() || !opened.isFile()) { throw new CodexUserIdentityRefusal("The coordinator database path changed during open."); diff --git a/tests/codex-transition-state-race.test.ts b/tests/codex-transition-state-race.test.ts index 2c049fc03..b07aec950 100644 --- a/tests/codex-transition-state-race.test.ts +++ b/tests/codex-transition-state-race.test.ts @@ -203,11 +203,13 @@ test("two real processes racing first use publish exactly one initial transition // terminal state is pinned exactly by `finalKinds`. expect(firstKinds.filter(kind => kind === "updated").length).toBeLessThanOrEqual(1); for (const result of results) { - expect( - result.first?.kind === "updated" - || result.first?.kind === "conflict" - || (result.first?.kind === "unavailable" && result.first.reason === "busy"), - ).toBe(true); + const acceptable = result.first?.kind === "updated" + || result.first?.kind === "conflict" + || (result.first?.kind === "unavailable" && result.first.reason === "busy"); + // Report the actual value on failure; `toBe(true)` alone says only that + // something unexpected happened, which is the least useful thing a race + // test can tell you. + expect({ acceptable, first: result.first }).toMatchObject({ acceptable: true }); } const finalKinds = results.map(result => result.final?.kind).sort(); diff --git a/tests/codex-transition-state.test.ts b/tests/codex-transition-state.test.ts index 7c129b4ec..fea549e20 100644 --- a/tests/codex-transition-state.test.ts +++ b/tests/codex-transition-state.test.ts @@ -1,5 +1,5 @@ import { afterEach, beforeEach, expect, test } from "bun:test"; -import { chmodSync, mkdirSync, mkdtempSync, realpathSync, rmSync, writeFileSync } from "node:fs"; +import { chmodSync, mkdirSync, mkdtempSync, realpathSync, rmSync, statSync, writeFileSync } from "node:fs"; import { join } from "node:path"; import { tmpdir } from "node:os"; @@ -549,3 +549,29 @@ test("a begin whose txId matches but whose generation does not is rejected", () expect(after.kind).toBe("ready"); if (after.kind === "ready") expect(after.state.currentTxId).toBe("tx-one"); }); + +/** + * A permissive coordinator is never LEFT permissive. + * + * The flake fix relaxed WHEN mode is judged, not whether. Two processes reaching + * first use together both see ENOENT, and the loser can lstat the winner's file + * before its chmod lands; refusing there reported `unsafe-path` for what was only + * a schedule (1-in-12 on a 16-core box). Ownership is still decided before the + * open — a file owned by somebody else is not a race and waiting cannot make it + * ours — while mode is narrowed once below the open and judged on the settled + * state. + * + * This asserts the outcome that matters and can actually be observed: after a + * read, the file is owner-only again. Removing the narrowing leaves it 0644 and + * turns this red. + */ +test("a coordinator found group-readable is narrowed back to owner-only", () => { + expect(readCodexTransitionState().kind).toBe("ready"); + + chmodSync(coordinatorPath, 0o644); + expect(statSync(coordinatorPath).mode & 0o777).toBe(0o644); + + const read = readCodexTransitionState(); + expect(read.kind).toBe("ready"); + expect(statSync(coordinatorPath).mode & 0o777).toBe(0o600); +}); From e90989a4ae9fb916c6d1042505b12ffb0bf31790 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Wed, 5 Aug 2026 06:15:46 +0900 Subject: [PATCH 094/163] feat(codex): apply routes its history through the Worker instead of the thread MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The first caller off the inline path. `injectCodexConfig` was doing SQLite rows, the backup manifest and every rollout patch on the caller thread, and those three do not share a transaction — which is how an opposite-direction process could overtake through the two the busy timeout never covered. Intent is translated here and handed down fixed: the opt-out, the legacy-mode target, and the ordinary migration to native are decided from what this function already admitted, so the Worker never takes a direction from a request. A blocked or failed unit surfaces as `failed` rather than as zero rows. Counting a lock we could not take as "no work to do" is the same mistake as reading an absent file as proof nothing changed, and this apply path prints that state to the user. The manifest path is derived by the provider's own rule rather than rebuilt. My first version guessed a filename next to the state database; the real manifest lives in the config directory under a hash of it, so the guess would have addressed a different file and silently found nothing to restore. Full suite 8278 pass / 0 fail. `restoreNativeCodex` stays inline for now — it is synchronous with seven call sites across the CLI, the service and the management API, so it moves in its own commit rather than riding along in this one. --- src/codex/history-job.ts | 32 +++++++++++++++++++++++++++++++ src/codex/history-provider.ts | 9 ++++++++- src/codex/inject.ts | 36 ++++++++++++++++++++++++++++++----- 3 files changed, 71 insertions(+), 6 deletions(-) diff --git a/src/codex/history-job.ts b/src/codex/history-job.ts index e96a8e0a7..83a9c7c7a 100644 --- a/src/codex/history-job.ts +++ b/src/codex/history-job.ts @@ -17,11 +17,43 @@ * Design record: devlog/_plan/260804_codex_write_substrate/020_history_isolation.md. */ import { randomUUID } from "node:crypto"; +import { join } from "node:path"; import type { CodexHistoryWorkerOperation, HistoryWorkerResult, } from "./history-worker"; +import { historyBackupPathFor } from "./history-provider"; +import { getCodexHome } from "./paths"; + +/** Where Codex keeps its resume history, and the manifest that shadows it. */ +const STATE_DB_FILE = "state_5.sqlite"; + +/** + * Resolve the paths a history job needs, at CALL time. + * + * `history-provider.ts` resolves its equivalents at module load (`:16`, `:22`), + * which is fine in one process and wrong for a Worker: the Worker does not + * inherit them, so anything derived from those constants would address a + * different home than the caller intended. Resolving here also means a test that + * moves `CODEX_HOME` is honoured rather than ignored. + */ +export function resolveCodexHistoryJobTarget(): { + readonly canonicalCodexHome: string; + readonly canonicalStateDbPath: string; + readonly canonicalBackupPath: string; +} { + const home = getCodexHome(); + const stateDb = join(home, STATE_DB_FILE); + return { + canonicalCodexHome: home, + canonicalStateDbPath: stateDb, + // Derived by the provider's own rule rather than guessed: the manifest lives + // in the config directory under a hash of the state database, so a + // hand-built path would address a different file entirely. + canonicalBackupPath: historyBackupPathFor(stateDb), + }; +} /** How long a history unit may run before the parent stops waiting on it. */ const WORKER_TIMEOUT_MS = 30_000; diff --git a/src/codex/history-provider.ts b/src/codex/history-provider.ts index 8c63a62d2..3916380cd 100644 --- a/src/codex/history-provider.ts +++ b/src/codex/history-provider.ts @@ -14,7 +14,14 @@ import { atomicWriteFile, getConfigDir } from "../config"; export const MAX_ROLLOUT_ZST_DECOMPRESSED_BYTES = 64 * 1024 * 1024; const STATE_DB_PATH = join(CODEX_HOME, "state_5.sqlite"); -function historyBackupPathFor(stateDbPath: string): string { +/** + * The manifest that shadows one state database. + * + * Exported because the history job must resolve it at CALL time for a Worker + * that does not inherit this module's load-time constants — and must resolve it + * the same way, since a manifest addressed differently is a different manifest. + */ +export function historyBackupPathFor(stateDbPath: string): string { const normalized = process.platform === "win32" ? resolve(stateDbPath).toLowerCase() : resolve(stateDbPath); const id = createHash("sha256").update(normalized).digest("hex").slice(0, 16); return join(getConfigDir(), `codex-history-backup-${id}.json`); diff --git a/src/codex/inject.ts b/src/codex/inject.ts index 8f700f2cf..1732ae788 100644 --- a/src/codex/inject.ts +++ b/src/codex/inject.ts @@ -3,7 +3,12 @@ import { atomicWriteFile, loadConfig, subagentDefaultSyncEffective, websocketsEn import { markJournalInjectedState, removeJournal, restoreJournalState, writeJournal } from "./journal"; import { withCatalogWriteSerialization } from "./catalog-write-serialization"; import { restoreCodexCatalogWithPermit } from "./catalog/sync"; -import { migrateHistoryToOpenai, syncCodexHistoryProvider } from "./history-provider"; +import { syncCodexHistoryProvider } from "./history-provider"; +import { + deriveCodexHistoryOperation, + resolveCodexHistoryJobTarget, + runCodexHistoryJob, +} from "./history-job"; import { OCX_SECTION_MARKER, hasInjectedCodexRouting, @@ -599,14 +604,35 @@ export async function injectCodexConfig(port: number, config?: OcxConfig, option // Legacy mode still forward-tags history so re-tagged threads stay listable. Design B needs // the opposite: a one-time migration of previously re-tagged threads BACK to openai (restore // machinery; cheap no-op when there is nothing to migrate). - const history = config?.syncResumeHistory !== false - ? (legacyMode ? syncCodexHistoryProvider("opencodex") : migrateHistoryToOpenai()) - : { rows: 0, files: 0 }; + // History runs in a Worker under H, not on this thread. + // + // The three surfaces it touches — the SQLite rows, the backup manifest, and the + // rollout files — do not share a transaction, so a busy timeout only ever + // serialized one of them and an opposite-direction process could overtake + // through the other two. The operation is derived from admitted intent here and + // handed down fixed; the Worker never takes a direction from its caller. + const historyOutcome = await runCodexHistoryJob({ + ...resolveCodexHistoryJobTarget(), + operation: deriveCodexHistoryOperation({ + direction: "apply", + resumeHistory: config?.syncResumeHistory !== false, + legacyMode, + }), + }); + // A blocked or failed unit is reported, not silently counted as zero work: + // `failed` is what makes the caller's message say so. + const history: { rows: number; files: number; failed?: true } = + historyOutcome.kind === "converged" + ? { rows: historyOutcome.rows, files: historyOutcome.files } + : historyOutcome.kind === "skipped" + ? { rows: 0, files: 0 } + : { rows: 0, files: 0, failed: true }; const catalogMessage = catalogPath ? ` Codex model catalog: ${catalogPath}\n` : ` Codex model catalog not injected because no opencodex catalog file exists yet.\n`; - const migratedRows = (history.rows ?? 0) + ("ejectedRows" in history ? history.ejectedRows ?? 0 : 0); + const ejected = (history as { ejectedRows?: number }).ejectedRows ?? 0; + const migratedRows = (history.rows ?? 0) + ejected; const historyMessage = config?.syncResumeHistory === false ? ` Codex resume history: left unchanged (syncResumeHistory=false).\n` : history.failed From e1715188f5359053cbc86cc8ecbf57f701fb69da Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Wed, 5 Aug 2026 06:45:07 +0900 Subject: [PATCH 095/163] feat(codex): restore takes the Worker too, except where the process is dying MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Six of the seven restore call sites are already inside async functions, so they move to `restoreNativeCodexAsync`: the native files still come down inline, and history runs in the Worker under H. The synchronous body keeps doing everything else and is gated on `skipHistory`, so the transition happens once rather than twice — an inline run followed by a locked one would leave the lock guarding something that already happened. The seventh stays synchronous on purpose. `cli/index.ts:257` is the process shutdown cleanup path, and spawning a Worker there means asking a dying process to wait on a thread it may not outlive. A slower correct restore is worth less than one that finishes. Three suites assert the ordering by reading the source text, so they follow the rename. The property they check is unchanged: the proxy still stops before the restore, and uninstall still tears down before deleting assets. The regression for the gate went through a correction worth recording. The behavioural version passed with `skipHistory` ignored entirely, because the synchronous path resolves its state database from a module-load constant (`history-provider.ts:16`) that a test moving CODEX_HOME cannot reach — so it was asserting nothing at all. It now checks the source, where removing the gate is visible. A test that cannot observe the thing it names is worse than no test, because it reads as coverage. --- src/cli/index.ts | 10 ++++----- src/codex/inject.ts | 39 +++++++++++++++++++++++++++++++-- src/server/management-api.ts | 4 ++-- src/service.ts | 6 ++--- tests/cli-restore-back.test.ts | 2 +- tests/codex-history-job.test.ts | 29 +++++++++++++++++++++++- tests/grok-lifecycle.test.ts | 2 +- tests/service.test.ts | 8 +++---- 8 files changed, 81 insertions(+), 19 deletions(-) diff --git a/src/cli/index.ts b/src/cli/index.ts index 8d8265446..0d8176a65 100755 --- a/src/cli/index.ts +++ b/src/cli/index.ts @@ -1,6 +1,6 @@ #!/usr/bin/env bun import { spawn } from "node:child_process"; -import { currentExternalCodexModelProvider, restoreNativeCodex, shouldInjectApiAuthHeader } from "../codex/inject"; +import { currentExternalCodexModelProvider, restoreNativeCodex, restoreNativeCodexAsync, shouldInjectApiAuthHeader } from "../codex/inject"; import { stripGrokConfig } from "../grok/inject"; import { restoreLegacyOpenaiHistory } from "../codex/history-provider"; import { reconcileJournal } from "../codex/journal"; @@ -525,7 +525,7 @@ async function handleStop() { } } if (!ownershipBlocked) { - const r = restoreNativeCodex(); + const r = await restoreNativeCodexAsync(); if (r.success) console.log(`↩️ ${r.message}`); else { stopFailed = true; @@ -587,8 +587,8 @@ async function handleUninstall() { }); } - await runStep("native Codex restored", () => { - const r = restoreNativeCodex(); + await runStep("native Codex restored", async () => { + const r = await restoreNativeCodexAsync(); if (!r.success) throw new Error(r.message); }); @@ -765,7 +765,7 @@ switch (command) { } let r: { success: boolean; message: string }; try { - r = restoreNativeCodex(); + r = await restoreNativeCodexAsync(); } catch (err) { r = { success: false, message: err instanceof Error ? err.message : String(err) }; } diff --git a/src/codex/inject.ts b/src/codex/inject.ts index 1732ae788..ede89efcb 100644 --- a/src/codex/inject.ts +++ b/src/codex/inject.ts @@ -788,7 +788,38 @@ export function removeCodexConfig(options: { preserveProfile?: boolean } = {}): * so plain `codex` works when the proxy is stopped. Called by `ocx stop`, the proxy shutdown * handler, and `ocx restore`. Idempotent + atomic. */ -export function restoreNativeCodex(): { success: boolean; message: string } { +/** + * Restore native Codex, running history in a Worker under H. + * + * Prefer this everywhere. The synchronous variant below exists only for the + * process-exit path, where awaiting a thread is its own hazard. + */ +export async function restoreNativeCodexAsync(): Promise<{ success: boolean; message: string }> { + const inline = restoreNativeCodex({ skipHistory: true }); + const outcome = await runCodexHistoryJob({ + ...resolveCodexHistoryJobTarget(), + operation: deriveCodexHistoryOperation({ + direction: "restore", + // Restore always returns history to native when it runs at all; the + // opt-out belongs to apply, which is what put opencodex there. + resumeHistory: true, + legacyMode: false, + }), + }); + const historyMsg = outcome.kind === "converged" + ? (outcome.rows > 0 + ? ` Resume history restored from opencodex backup (${outcome.rows} thread(s)).` + : "") + : outcome.kind === "skipped" + ? "" + // A lock we could not take is reported, never counted as nothing to do. + : ` ⚠️ Codex resume history could NOT be restored — the Codex app appears to be holding the history database. Close Codex and run \`ocx restore\` again.`; + return { success: inline.success, message: `${inline.message}${historyMsg}` }; +} + +export function restoreNativeCodex( + options: { skipHistory?: boolean } = {}, +): { success: boolean; message: string } { const activeProvider = currentExternalCodexModelProvider(); if (activeProvider) { removeJournal(); @@ -812,7 +843,11 @@ export function restoreNativeCodex(): { success: boolean; message: string } { try { skipWhenProvablyNoop = !shouldInjectApiAuthHeader(loadConfig()); } catch { /* unreadable config: keep the conservative write-open restore */ } - const history = syncCodexHistoryProvider("openai", undefined, undefined, { skipWhenProvablyNoop }); + // `skipHistory` is how the async wrapper takes this work for itself: the + // native files come down here, and history runs in the Worker under H. + const history = options.skipHistory + ? { rows: 0, files: 0 } + : syncCodexHistoryProvider("openai", undefined, undefined, { skipWhenProvablyNoop }); const msg = cat.removed > 0 ? `${cfg.message} Catalog restored to ${cat.kept} native model(s) (dropped ${cat.removed} proxy-routed).` : cfg.message; diff --git a/src/server/management-api.ts b/src/server/management-api.ts index ac6784716..0a60757f6 100644 --- a/src/server/management-api.ts +++ b/src/server/management-api.ts @@ -218,7 +218,7 @@ export async function handleManagementAPI( if (routed) return routed; if (url.pathname === "/api/stop" && req.method === "POST") { - const { restoreNativeCodex } = await import("../codex/inject"); + const { restoreNativeCodexAsync } = await import("../codex/inject"); const { stopServiceIfInstalled, isServiceOwnershipError } = await import("../service"); try { stopServiceIfInstalled(); @@ -231,7 +231,7 @@ export async function handleManagementAPI( } throw err; } - const restore = restoreNativeCodex(); + const restore = await restoreNativeCodexAsync(); // Both managed configs come down together on an explicit teardown. The daemon's own // syncCleanup skips this when OCX_SERVICE is set (so a crash/respawn keeps the fence), // which is exactly why an intentional stop has to do it here. diff --git a/src/service.ts b/src/service.ts index 69ef03767..58e5785bc 100644 --- a/src/service.ts +++ b/src/service.ts @@ -12,7 +12,7 @@ import { homedir } from "node:os"; import { dirname, join, resolve } from "node:path"; import { expandUserPath, getConfigDir, readPid, removePid, removeRuntimePort, verifyPidIdentity } from "./config"; import { loadConfig } from "./config"; -import { restoreNativeCodex } from "./codex/inject"; +import { restoreNativeCodex, restoreNativeCodexAsync } from "./codex/inject"; import { stripGrokConfig } from "./grok/inject"; import { isWslRuntime } from "./codex/home"; import { BUN_RUNTIME_PATH_ENV, BUN_RUNTIME_SOURCE_ENV, durableBunRuntime } from "./lib/bun-runtime"; @@ -2584,7 +2584,7 @@ export async function serviceCommand(...args: (string | undefined)[]): Promise { expect(restoreCase.indexOf("if (!synced.ok)")).toBeLessThan(restoreCase.indexOf("target.effectiveCodexHome")); expect(restoreCase).toContain("target.effectiveCodexHome"); // The forward switch reports incomplete marker cleanup instead of claiming native success. - expect(restoreCase).toContain("restoreNativeCodex()"); + expect(restoreCase).toContain("restoreNativeCodexAsync()"); expect(restoreCase).toContain("process.exitCode = 1"); expect(restoreCase).toContain("was not fully restored"); }); diff --git a/tests/codex-history-job.test.ts b/tests/codex-history-job.test.ts index 3082333fa..0c9846198 100644 --- a/tests/codex-history-job.test.ts +++ b/tests/codex-history-job.test.ts @@ -1,5 +1,5 @@ import { afterEach, expect, test } from "bun:test"; -import { chmodSync, mkdirSync, mkdtempSync, realpathSync, rmSync, writeFileSync } from "node:fs"; +import { chmodSync, mkdirSync, mkdtempSync, readFileSync, realpathSync, rmSync, writeFileSync } from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; @@ -132,3 +132,30 @@ test("an overrun Worker returns a typed timeout rather than hanging", async () = if (outcome.kind === "failed") expect(outcome.reason).toBe("timeout"); expect(Date.now() - started).toBeLessThan(20_000); }, 30_000); + +/** + * The async restore wrapper owns history; the synchronous body must not also do + * it, or every restore would run the transition twice — once unserialized on the + * caller thread, which is the path this phase exists to remove. + * + * Asserted against the SOURCE rather than by running it. The synchronous body + * resolves its state database from a module-load constant + * (`history-provider.ts:16`), so a test that moves `CODEX_HOME` cannot observe + * which database it would have touched — a behavioural version of this passed + * with `skipHistory` ignored entirely, which is worse than no test. Removing the + * guard changes this text, and that is something a check can actually see. + */ +test("the synchronous restore body is gated on skipHistory", () => { + const source = readFileSync(join(import.meta.dir, "..", "src", "codex", "inject.ts"), "utf8"); + const body = source.slice(source.indexOf("export function restoreNativeCodex(")); + const historyCall = body.indexOf("syncCodexHistoryProvider(\"openai\""); + expect(historyCall).toBeGreaterThan(-1); + + // The inline call is reachable only through the gate. + const gate = body.indexOf("options.skipHistory"); + expect(gate).toBeGreaterThan(-1); + expect(gate).toBeLessThan(historyCall); + + // And the async wrapper is the thing that sets it. + expect(source).toContain("restoreNativeCodex({ skipHistory: true })"); +}); diff --git a/tests/grok-lifecycle.test.ts b/tests/grok-lifecycle.test.ts index 331ca1112..e69368231 100644 --- a/tests/grok-lifecycle.test.ts +++ b/tests/grok-lifecycle.test.ts @@ -49,7 +49,7 @@ describe("Grok fence lifecycle wiring", () => { const gateAt = stopFn.indexOf("if (!ownershipBlocked)"); const stripAt = stopFn.indexOf("stripGrokConfig()"); - const restoreAt = stopFn.indexOf("restoreNativeCodex()"); + const restoreAt = stopFn.indexOf("restoreNativeCodexAsync()"); const revertAt = stopFn.indexOf("revertSystemEnv()"); expect(gateAt).toBeGreaterThan(-1); diff --git a/tests/service.test.ts b/tests/service.test.ts index a2ebc5838..27274ef88 100644 --- a/tests/service.test.ts +++ b/tests/service.test.ts @@ -643,9 +643,9 @@ describe("service lifecycle cleanup ordering", () => { expect(stopCase).toContain("ops.stop();"); expect(stopCase).toContain("await stopTrackedProxyForServiceCommand();"); - expect(stopCase).toContain("restoreNativeCodex();"); + expect(stopCase).toContain("restoreNativeCodexAsync();"); expect(stopCase.indexOf("ops.stop();")).toBeLessThan(stopCase.indexOf("stopTrackedProxyForServiceCommand();")); - expect(stopCase.indexOf("stopTrackedProxyForServiceCommand();")).toBeLessThan(stopCase.indexOf("restoreNativeCodex();")); + expect(stopCase.indexOf("stopTrackedProxyForServiceCommand();")).toBeLessThan(stopCase.indexOf("restoreNativeCodexAsync();")); }); test("direct service uninstall kills the tracked proxy before deleting service assets", async () => { @@ -655,10 +655,10 @@ describe("service lifecycle cleanup ordering", () => { expect(uninstallCase).toContain("ops.stop();"); expect(uninstallCase).toContain("await stopTrackedProxyForServiceCommand();"); expect(uninstallCase).toContain("ops.uninstall();"); - expect(uninstallCase).toContain("restoreNativeCodex();"); + expect(uninstallCase).toContain("restoreNativeCodexAsync();"); expect(uninstallCase.indexOf("ops.stop();")).toBeLessThan(uninstallCase.indexOf("stopTrackedProxyForServiceCommand();")); expect(uninstallCase.indexOf("stopTrackedProxyForServiceCommand();")).toBeLessThan(uninstallCase.indexOf("ops.uninstall();")); - expect(uninstallCase.indexOf("ops.uninstall();")).toBeLessThan(uninstallCase.indexOf("restoreNativeCodex();")); + expect(uninstallCase.indexOf("ops.uninstall();")).toBeLessThan(uninstallCase.indexOf("restoreNativeCodexAsync();")); }); test("Windows service install ends the running task before rewriting its assets, with write retry", async () => { From aaa17a82108b5ac3137cb9075bcd30bf7c3cff91 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Wed, 5 Aug 2026 07:09:09 +0900 Subject: [PATCH 096/163] feat(codex): the last two history callers stop writing on their own thread MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `ocx recover-history` and the migration guardian were the two paths still mutating history inline. Both go through the job now, so every production history write is serialized under H. Recovery keeps its own operation rather than folding into generic restore. The distinction is load-bearing: manifest-independent ejection must not read, consume or replace the backup manifest that a generic restore consumes, and collapsing them would silently destroy the record a later restore depends on. The guardian mattered more than its size suggests. It fires on a timer during startup, so it was the one writer most likely to be running when an apply or a restore arrived — a background repair racing the transition it was meant to finish. Its default migration is now a job like any other; an injected `migrateFn` still works, because the tests drive it. Its manual scheduler had to learn to await. A tick that returns a promise the scheduler drops lets a test assert on state the tick has not produced yet, which is how a real async bug reads as a passing suite. One source-text assertion followed the change. It checked for the inline writer by name; it now checks that the command exists, is async, and reaches the job with the legacy operation — the same property, stated against what the code actually does. --- src/cli/index.ts | 17 ++++++-- src/codex/history-migration-guardian.ts | 23 ++++++++--- tests/history-migration-guardian.test.ts | 51 ++++++++++++++---------- tests/uninstall.test.ts | 9 ++++- 4 files changed, 67 insertions(+), 33 deletions(-) diff --git a/src/cli/index.ts b/src/cli/index.ts index 0d8176a65..bef5872cd 100755 --- a/src/cli/index.ts +++ b/src/cli/index.ts @@ -2,7 +2,7 @@ import { spawn } from "node:child_process"; import { currentExternalCodexModelProvider, restoreNativeCodex, restoreNativeCodexAsync, shouldInjectApiAuthHeader } from "../codex/inject"; import { stripGrokConfig } from "../grok/inject"; -import { restoreLegacyOpenaiHistory } from "../codex/history-provider"; +import { resolveCodexHistoryJobTarget, runCodexHistoryJob } from "../codex/history-job"; import { reconcileJournal } from "../codex/journal"; import { codexAutoStartEnabled, @@ -708,13 +708,22 @@ async function handleStatus() { } } -function handleRecoverHistory() { +async function handleRecoverHistory() { if (args[1] !== "--legacy-openai") { console.error("Usage: ocx recover-history --legacy-openai"); console.error("Only use this if an older syncResumeHistory build already remapped OpenAI Codex App history to opencodex before backup support existed."); process.exit(1); } - const r = restoreLegacyOpenaiHistory(); + // Manifest-independent legacy ejection, serialized like every other history + // mutation. It is a separate operation from generic restore precisely because + // it must not read, consume or replace the backup manifest. + const outcome = await runCodexHistoryJob({ + ...resolveCodexHistoryJobTarget(), + operation: "recover-legacy-openai", + }); + const r = outcome.kind === "converged" + ? { rows: outcome.rows, files: outcome.files, failed: undefined } + : { rows: 0, files: 0, failed: true as const }; if (r.failed) { console.error( "⚠️ Recovery SKIPPED: the Codex history DB is locked (Codex app/IDE open?). Close it and rerun this command.", @@ -790,7 +799,7 @@ switch (command) { break; } case "recover-history": - handleRecoverHistory(); + await handleRecoverHistory(); break; case "uninstall": case "remove": diff --git a/src/codex/history-migration-guardian.ts b/src/codex/history-migration-guardian.ts index 8145caeb7..e25ea97bf 100644 --- a/src/codex/history-migration-guardian.ts +++ b/src/codex/history-migration-guardian.ts @@ -1,4 +1,5 @@ import { countPendingOpencodexHistory, migrateHistoryToOpenai } from "./history-provider"; +import { resolveCodexHistoryJobTarget, runCodexHistoryJob } from "./history-job"; /** * Daemon-side retry for the one-time Design-B history migration. @@ -23,7 +24,8 @@ export interface HistoryMigrationGuardianHandle { export interface HistoryMigrationGuardianDeps { countFn?: typeof countPendingOpencodexHistory; - migrateFn?: () => ReturnType; + migrateFn?: () => ReturnType + | Promise>; log?: Pick; tickMs?: number; maxTicks?: number; @@ -42,7 +44,18 @@ function defaultSchedule(fn: () => void, ms: number): { cancel(): void } { export function startHistoryMigrationGuardian(deps: HistoryMigrationGuardianDeps = {}): HistoryMigrationGuardianHandle { const countFn = deps.countFn ?? countPendingOpencodexHistory; - const migrateFn = deps.migrateFn ?? (() => migrateHistoryToOpenai(undefined, undefined, { attempts: 1 })); + // The default migration goes through the history job, so the guardian's timer + // thread never performs the transition itself. A background repair that races + // an apply or a restore is exactly what H exists to order. + const migrateFn = deps.migrateFn ?? (async () => { + const outcome = await runCodexHistoryJob({ + ...resolveCodexHistoryJobTarget(), + operation: "migrate-openai", + }); + return outcome.kind === "converged" + ? { rows: outcome.rows, files: outcome.files } + : { rows: 0, files: 0, failed: true as const }; + }); const log = deps.log ?? console; const tickMs = deps.tickMs ?? DEFAULT_TICK_MS; const maxTicks = deps.maxTicks ?? DEFAULT_MAX_TICKS; @@ -56,7 +69,7 @@ export function startHistoryMigrationGuardian(deps: HistoryMigrationGuardianDeps pending = (deps.scheduleFn ?? defaultSchedule)(tick, tickMs); }; - const tick = () => { + const tick = async () => { if (stopped) return; ticks++; try { @@ -66,9 +79,9 @@ export function startHistoryMigrationGuardian(deps: HistoryMigrationGuardianDeps return; } // Locked probe or pending work: attempt one migration pass. - const result = migrateFn(); + const result = await migrateFn(); if (!result.failed) { - const moved = result.rows + (result.ejectedRows ?? 0); + const moved = result.rows + ((result as { ejectedRows?: number }).ejectedRows ?? 0); if (moved > 0) { log.log(`🩹 history-migration: ${moved} legacy opencodex thread(s) migrated back to openai.`); } diff --git a/tests/history-migration-guardian.test.ts b/tests/history-migration-guardian.test.ts index 8f6610264..c5930e5ca 100644 --- a/tests/history-migration-guardian.test.ts +++ b/tests/history-migration-guardian.test.ts @@ -1,18 +1,25 @@ import { describe, expect, test } from "bun:test"; import { startHistoryMigrationGuardian } from "../src/codex/history-migration-guardian"; -/** Manual scheduler: collects scheduled callbacks so tests drive ticks deterministically. */ +/** + * Manual scheduler: collects scheduled callbacks so tests drive ticks + * deterministically. + * + * `runNext` awaits the callback, because a tick now runs its migration through + * the history job and is therefore async. Dropping that promise would let a test + * assert on state the tick had not finished producing. + */ function manualScheduler() { - const queue: Array<() => void> = []; + const queue: Array<() => void | Promise> = []; return { - scheduleFn: (fn: () => void) => { + scheduleFn: (fn: () => void | Promise) => { queue.push(fn); return { cancel: () => { const i = queue.indexOf(fn); if (i !== -1) queue.splice(i, 1); } }; }, - runNext(): boolean { + async runNext(): Promise { const fn = queue.shift(); if (!fn) return false; - fn(); + await fn(); return true; }, get size() { return queue.length; }, @@ -22,7 +29,7 @@ function manualScheduler() { const silent = { log: () => {} }; describe("history migration guardian", () => { - test("stops silently when nothing is pending", () => { + test("stops silently when nothing is pending", async () => { const sched = manualScheduler(); let migrations = 0; startHistoryMigrationGuardian({ @@ -32,12 +39,12 @@ describe("history migration guardian", () => { scheduleFn: sched.scheduleFn, }); - expect(sched.runNext()).toBe(true); + expect(await sched.runNext()).toBe(true); expect(migrations).toBe(0); // no pending work — never touches the migrate path expect(sched.size).toBe(0); // and never reschedules }); - test("retries while the DB stays locked, then logs and stops on success", () => { + test("retries while the DB stays locked, then logs and stops on success", async () => { const sched = manualScheduler(); const logs: string[] = []; let attempts = 0; @@ -53,15 +60,15 @@ describe("history migration guardian", () => { scheduleFn: sched.scheduleFn, }); - expect(sched.runNext()).toBe(true); // tick 1: locked - expect(sched.runNext()).toBe(true); // tick 2: locked - expect(sched.runNext()).toBe(true); // tick 3: success + expect(await sched.runNext()).toBe(true); // tick 1: locked + expect(await sched.runNext()).toBe(true); // tick 2: locked + expect(await sched.runNext()).toBe(true); // tick 3: success expect(attempts).toBe(3); expect(logs.some(l => l.includes("3 legacy opencodex thread(s) migrated"))).toBe(true); expect(sched.size).toBe(0); // stopped after success }); - test("gives up with a warning after maxTicks", () => { + test("gives up with a warning after maxTicks", async () => { const sched = manualScheduler(); const logs: string[] = []; startHistoryMigrationGuardian({ @@ -72,13 +79,13 @@ describe("history migration guardian", () => { maxTicks: 2, }); - expect(sched.runNext()).toBe(true); - expect(sched.runNext()).toBe(true); + expect(await sched.runNext()).toBe(true); + expect(await sched.runNext()).toBe(true); expect(sched.size).toBe(0); // budget exhausted — no reschedule expect(logs.some(l => l.includes("stayed locked"))).toBe(true); }); - test("stop() cancels the pending tick", () => { + test("stop() cancels the pending tick", async () => { const sched = manualScheduler(); let migrations = 0; const handle = startHistoryMigrationGuardian({ @@ -89,11 +96,11 @@ describe("history migration guardian", () => { }); handle.stop(); - expect(sched.runNext()).toBe(false); // cancelled before firing + expect(await sched.runNext()).toBe(false); // cancelled before firing expect(migrations).toBe(0); }); - test("a locked count probe still attempts migration and keeps ticking until a clean re-count", () => { + test("a locked count probe still attempts migration and keeps ticking until a clean re-count", async () => { const sched = manualScheduler(); let migrations = 0; let counts = 0; @@ -110,12 +117,12 @@ describe("history migration guardian", () => { scheduleFn: sched.scheduleFn, }); - expect(sched.runNext()).toBe(true); + expect(await sched.runNext()).toBe(true); expect(migrations).toBe(1); expect(sched.size).toBe(0); // migration succeeded and re-count is clean → stop }); - test("does not stop on a zero-row 'success' while backup entries remain (missing-DB race)", () => { + test("does not stop on a zero-row 'success' while backup entries remain (missing-DB race)", async () => { const sched = manualScheduler(); let migrations = 0; // DB missing: count sees only the backup manifest; migrate 'succeeds' with 0 rows. @@ -127,11 +134,11 @@ describe("history migration guardian", () => { maxTicks: 3, }); - expect(sched.runNext()).toBe(true); + expect(await sched.runNext()).toBe(true); expect(migrations).toBe(1); expect(sched.size).toBe(1); // NOT stopped — backup work is still pending - expect(sched.runNext()).toBe(true); - expect(sched.runNext()).toBe(true); // budget exhausted on tick 3 + expect(await sched.runNext()).toBe(true); + expect(await sched.runNext()).toBe(true); // budget exhausted on tick 3 expect(sched.size).toBe(0); }); }); diff --git a/tests/uninstall.test.ts b/tests/uninstall.test.ts index 5243ecc8e..b09dbc75e 100644 --- a/tests/uninstall.test.ts +++ b/tests/uninstall.test.ts @@ -23,8 +23,13 @@ describe("full uninstall command", () => { const cli = await readText("src/cli/index.ts"); expect(cli).toContain("ocx recover-history --legacy-openai"); - expect(cli).toContain("function handleRecoverHistory()"); - expect(cli).toContain("restoreLegacyOpenaiHistory"); + expect(cli).toContain("async function handleRecoverHistory()"); + // The command still performs legacy recovery, but through the serialized + // history job rather than by calling the writer inline — the operation name + // is what keeps it distinct from a generic restore, which must not touch the + // backup manifest this one deliberately leaves alone. + expect(cli).toContain("recover-legacy-openai"); + expect(cli).toContain("runCodexHistoryJob"); }); test("service cleanup has a quiet best-effort helper", async () => { From 86e5d677b85e38cd8007972629ddcabe803595b1 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Wed, 5 Aug 2026 07:28:56 +0900 Subject: [PATCH 097/163] test(codex): prove the history writer is unreachable, by walking imports MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The existing guard for this property was a regex counting call sites in four named files. It could not see a wrapper, an alias, a re-export, or a dynamic import — so the one bypass that mattered would have landed green, in a phase that exists because a lock one caller can skip is not a lock. This walks the real import graph from every module under src/: static imports, `export ... from`, and `import(...)`. Only `history-worker.ts` may reach `internal/history-writer.ts`, because only it runs inside H. A second check catches the other shape — calling the provider's mutators directly — with an inventory of three modules that legitimately still do: the provider that owns them, the writer that wraps them, and `inject.ts`, whose synchronous body is kept for the process-shutdown path and gated on `skipHistory`. Verified against three bypasses in turn, each added as a real module and each turning the guard red: a static import, a dynamic import, and a re-export. The last two are exactly what the regex version missed. A third test keeps the inventory honest. A permitted root that does not exist, or that stopped importing the writer, would make the first check pass by vacuity — which is the failure mode of every allowlist nobody re-reads. --- tests/codex-history-reachability.test.ts | 127 +++++++++++++++++++++++ 1 file changed, 127 insertions(+) create mode 100644 tests/codex-history-reachability.test.ts diff --git a/tests/codex-history-reachability.test.ts b/tests/codex-history-reachability.test.ts new file mode 100644 index 000000000..8f61c5173 --- /dev/null +++ b/tests/codex-history-reachability.test.ts @@ -0,0 +1,127 @@ +import { expect, test } from "bun:test"; +import { readFileSync, readdirSync, statSync } from "node:fs"; +import { join, relative, resolve } from "node:path"; + +/** + * Who may reach a history writer, checked by walking imports rather than counting + * strings. + * + * The previous guard in this unit was a regex over a handful of files. It could + * not see a wrapper, an alias, a re-export, or a dynamic import, so a new bypass + * would have landed green — and this phase exists precisely because a lock one + * caller can skip is not a lock. + * + * The inventory is deliberately small and named. `internal/history-writer.ts` + * mutates the database, the manifest and the rollouts; only `history-worker.ts` + * runs inside H, so only it may reach those symbols. + */ +const SRC = resolve(import.meta.dir, "..", "src"); + +/** Modules whose exports mutate Codex history. */ +const HISTORY_WRITER = "codex/internal/history-writer.ts"; + +/** The only production module allowed to import them. */ +const PERMITTED_ROOTS = new Set(["codex/history-worker.ts"]); + +/** + * Modules that legitimately still contain inline history calls. + * + * `history-provider.ts` owns the implementations. `inject.ts` keeps a + * synchronous body for the process-shutdown path, where awaiting a Worker the + * process may not outlive trades a correct restore for a faster one; it is gated + * on `skipHistory` so the async wrapper is the one that runs under H. + */ +const INLINE_ALLOWED = new Set([ + "codex/history-provider.ts", + "codex/inject.ts", + "codex/internal/history-writer.ts", +]); + +/** Direct mutators — reaching these outside the inventory is the bypass. */ +const MUTATORS = [ + "syncCodexHistoryProvider", + "restoreLegacyOpenaiHistory", + "migrateHistoryToOpenai", +]; + +function sourceFiles(dir: string, out: string[] = []): string[] { + for (const entry of readdirSync(dir)) { + const full = join(dir, entry); + if (statSync(full).isDirectory()) sourceFiles(full, out); + else if (entry.endsWith(".ts")) out.push(full); + } + return out; +} + +/** + * Every import specifier in a file: static, `export ... from`, and dynamic. + * + * Dynamic imports matter most — they are how a caller reaches a module without + * appearing in the import block, and the old regex guard was blind to them. + */ +function importSpecifiers(source: string): string[] { + const specifiers: string[] = []; + const patterns = [ + /(?:^|\n)\s*import\s[^;]*?from\s*["']([^"']+)["']/g, + /(?:^|\n)\s*export\s[^;]*?from\s*["']([^"']+)["']/g, + /import\s*\(\s*["']([^"']+)["']\s*\)/g, + ]; + for (const pattern of patterns) { + for (const match of source.matchAll(pattern)) { + if (match[1]) specifiers.push(match[1]); + } + } + return specifiers; +} + +function resolveSpecifier(fromFile: string, specifier: string): string | null { + if (!specifier.startsWith(".")) return null; + const base = resolve(join(fromFile, ".."), specifier); + for (const candidate of [base, `${base}.ts`, join(base, "index.ts")]) { + try { + if (statSync(candidate).isFile()) return relative(SRC, candidate); + } catch { /* not this shape */ } + } + return null; +} + +test("only the history Worker can reach a history writer", () => { + const offenders: string[] = []; + for (const file of sourceFiles(SRC)) { + const rel = relative(SRC, file); + if (rel === HISTORY_WRITER) continue; + const resolved = importSpecifiers(readFileSync(file, "utf8")) + .map(specifier => resolveSpecifier(file, specifier)); + if (resolved.includes(HISTORY_WRITER) && !PERMITTED_ROOTS.has(rel)) { + offenders.push(rel); + } + } + expect(offenders).toEqual([]); +}); + +test("no production module outside the inventory calls a history mutator inline", () => { + const offenders: Array<{ file: string; symbol: string }> = []; + for (const file of sourceFiles(SRC)) { + const rel = relative(SRC, file); + if (INLINE_ALLOWED.has(rel)) continue; + const source = readFileSync(file, "utf8"); + for (const symbol of MUTATORS) { + // A call, not a type reference or a re-export of the name. + if (new RegExp(`\\b${symbol}\\s*\\(`).test(source)) { + offenders.push({ file: rel, symbol }); + } + } + } + expect(offenders).toEqual([]); +}); + +/** + * The inventory is only meaningful if it is currently satisfied by real modules. + * A guard whose permitted root does not exist would pass forever by vacuity. + */ +test("the permitted root exists and does import the writer", () => { + const worker = readFileSync(join(SRC, "codex", "history-worker.ts"), "utf8"); + const resolved = importSpecifiers(worker) + .map(specifier => resolveSpecifier(join(SRC, "codex", "history-worker.ts"), specifier)); + expect(resolved).toContain(HISTORY_WRITER); +}); From a35889612201884b29663f2ef4133091263dbffd Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Wed, 5 Aug 2026 08:13:11 +0900 Subject: [PATCH 098/163] =?UTF-8?q?docs(codex):=20WP11=20round=206=20?= =?UTF-8?q?=E2=80=94=20the=20lock=20has=20no=20production=20caller,=20and?= =?UTF-8?q?=20I=20claimed=20it=20did?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Six findings, all from executed probes or an independent review that ran the code rather than reading it. The one that matters: I wrote that `injectCodexConfig` was a production caller WP11 could serve on the apply direction. It is not. Production reaches it directly from sync.ts and cli/init.ts, never through convergence.ts — the only module this phase was scoped to touch — so the entry point I planned would have been an export nothing calls. It also awaits history mid-function and there is no runtime producer of AdmissionSnapshot at all. My own falsification test for the narrowing fired, so WP11 becomes mechanism-only and the caller moves to WP12, recorded as such instead of implied away. The rest: - The coordinator refuses to open on any routed home (probed both directions: clean opens, routed refuses legacy-ambiguous), so moving the catalog commit under N would regress every applied install. - The residue guard reads the ambient CODEX_HOME while the lock keys on a caller-supplied one — a routed home locked while a clean one was checked. The obvious fix is itself a TOCTOU, and it also refuses a symlinked default home against itself. - ACL success is cached by pathname, not file identity: unlink, recreate at the same name, and the replacement is credited with the old file's hardening. Absence-as-guarantee #15, and it is live in shipped code. - Deadline exhaustion is classified as permanent refusal, discarding ETIMEDOUT. - One citation was wrong and two were short; the type fence redeclares a brand that convergence-types and transition-state already own. --- .../030_lock_protocol.md | 298 +++++++++++++++++- 1 file changed, 296 insertions(+), 2 deletions(-) diff --git a/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md b/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md index bdd4fbbd7..ee5151c40 100644 --- a/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md +++ b/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md @@ -26,6 +26,288 @@ not required to replace a placeholder before this phase works. All current-code citations and diff context below were rechecked on 2026-08-04 at `7bde9e0c977721fc0b9d8617c85ff17de7c07658`. +## Round 6 — three findings that came from running the code, not reading it + +Rechecked on 2026-08-05 at `86e5d677b`, with executed probes rather than citation. +All three change what B must build. + +### F1 — the coordinator refuses to open on every routed install (blocking) + +`openCodexCoordinatorTransaction` initializes a missing row only after +`assertInitialStateCanBeCreated()` proves the record is not legacy and +`classifyNativeRoutedResidue()` returns `clean` +(`src/codex/transition-state.ts:263-303`). A routed `config.toml` — which is +exactly what every user with the proxy applied has — is residue. Executed against +a temp home containing a routed `model_provider = "opencodex"` block: + +```text +REFUSED: CodexCoordinatorLegacyAmbiguousError + | A missing coordinator row cannot be initialized while native Codex routing residue exists. +``` + +The same probe against a clean home opens and returns +`{"nativeBefore":0,"nativeAfter":1,...}`. + +So the plan's instruction to "place WP9's fixed catalog/native commit under the new +lock" is, as written, a **regression for every existing routed installation**: a +catalog refresh that succeeds today would begin refusing. The compatibility-adoption +path that `005_contract.md:705-800` designed for this is **not implemented** — +`withCodexCompatibilityNativeHandoff` and `adoption-pending` have zero occurrences +in `src/`. + +WP11 therefore ships the lock **without rewiring `convergence.ts` to require N**. +`commitCodexCatalogCandidate` keeps `K -> C` (`src/codex/convergence.ts:393-406`), +which is already correct and already cross-process safe. `convergence.ts` moves +under N in WP12, together with the admission pipeline and the adoption path that +makes opening N legal on a routed home. Landing the lock and its rewiring in one +phase would mean landing a refusal for the current user base to satisfy a document. + +That is not a scope dodge: WP11's own accept criteria (C5/C6/C7/C18) are about +acquisition, identity, and namespace. None of them requires the catalog seam to be +the first caller. What WP11 must NOT do is ship a module with no caller — defect #10 +in this unit was exactly that — so the deliverable includes the real +`convergeCodexNativeUnderLock` entry consumed by a production route, gated to the +homes where N can legally open, plus the falsifiable test that a routed home refuses +with a typed reason instead of throwing. + +#### The direction asymmetry — real, but it does NOT supply a caller + +"N cannot open" is not uniform, and the difference decides which production path +WP11 can legally serve. Probed both directions against real temp homes: + +| Operation | Home state when N is taken | Result | +|---|---|---| +| **apply** (route Codex at the proxy) | clean — not yet routed | `OPEN OK`, expectation `{nativeBefore:0, nativeAfter:1}` | +| **restore** (unroute back to native) | already routed | `REFUSED` — legacy-ambiguous | + +A first apply on an unrouted home is the one state the strict initializer accepts, +because the residue it refuses is the routing this operation has not performed yet. + +**That is where I claimed a production caller, and review round 6 proved the claim +false.** The paragraph that stood here named `injectCodexConfig` +(`src/codex/inject.ts:487`) as a caller WP11 could serve. Three facts kill it: + +1. Production reaches `injectCodexConfig` **directly** from `src/codex/sync.ts:58,110` + and `src/cli/init.ts:197`. None of them goes through `convergence.ts`, which is + the only module this phase's write scope was allowed to modify. Adding an entry + point to `convergence.ts` therefore adds an export nothing calls. +2. `injectCodexConfig` cannot become the synchronous commit callback as it stands. + It journals (`src/codex/inject.ts:530`), writes native files + (`:601`), and then **awaits** the history job (`:614`). The synchronous native + section has to be split from post-N history dispatch before any of it can sit + under N. +3. There is no runtime producer of a full `AdmissionSnapshot` at all. It exists only + as an interface (`src/codex/convergence-types.ts:495`); the sole thing production + builds today is `CatalogAdmissionSnapshot`. WP11's own API requires the former. + +So the narrowing as written *was* a dodge, and its own falsification test — "if the +apply path cannot legally take N either, fold WP11 into WP12" — has now fired. The +resolution is recorded below rather than argued away. + +#### Resolution: WP11 becomes mechanism-only, and its caller moves to WP12 + +WP11 ships `src/codex/codex-write-lock.ts` and its tests, and **nothing in +`convergence.ts`**. It is explicitly a mechanism phase whose consumer is WP12, in the +same way WP8b shipped the contract before anything consumed it. This is not the same +failure as defect #10 (a coordinator no production caller ever initialized) provided +two conditions hold, and B must satisfy both: + +- The goalplan records WP12 as the phase that supplies the admission producer, the + apply/restore split in `inject.ts`, and the first real call edge. WP11 is not + closable as "done" in the sense of "in production"; it is closable as "the + mechanism is correct and proven by real two-process tests". +- WP11's tests drive the **production module**, not a copy, through real child + processes — which its test plan already requires — so the mechanism is executed + even before a production caller exists. + +If WP12 does not land the caller, the honest state of this unit is that the lock is +unused, and the PR body must say exactly that instead of implying the substrate is +live. That sentence is a deliverable of WP14, not a footnote. + +### F2 — the initializer's residue guard reads the AMBIENT home, not the locked one + +Absence-as-guarantee #14. `classifyNativeRoutedResidue()` resolves its own home +through `getCodexHome()` (`src/codex/native-residue.ts:524`), which re-reads +`process.env.CODEX_HOME` (`src/codex/paths.ts:32-35`). `readIntegrationRecord()` +resolves its path the same ambient way. But `resolveCodexCoordinatorDatabasePath` +is keyed by the **caller-supplied canonical home**. Executed with ambient +`CODEX_HOME` pointing at a clean directory and the explicit target home routed: + +```text +ambient CODEX_HOME = /clean +explicit home = /routed +OPENED OK for a ROUTED explicit home -> residue check used the AMBIENT home +``` + +The guard passed by inspecting a directory that is not the one being locked. Every +existing caller happens to pass the ambient home, so the defect is latent today and +becomes live the moment WP11 accepts an explicit `codexHome` — which its API does. + +Consequence for WP11: `CodexWriteLockOptions.codexHome` may not be forwarded to a +coordinator whose safety guard reads a different home. WP11 refuses with +`authority_not_proven` when the canonical target home is not identical to the +ambient `getCodexHome()` result, and a test drives the mismatch. Making the +guard home-parameterized is WP12's job (it owns admission); WP11 must not silently +accept a home whose residue was never checked. + +**The obvious version of that remedy is itself a TOCTOU**, and it was caught by +probing rather than by reading. `getCodexHome()` re-resolves `process.env.CODEX_HOME` +on every call (`src/codex/paths.ts:32-35`), so two calls inside one operation can +return two different directories: + +```text +same call, two answers: true | a -> b +``` + +A comparison that calls `getCodexHome()` once to validate and lets the coordinator +call it again to check residue proves nothing: the second read is a fresh read. So +the check is not "compare the two", it is **resolve the ambient home exactly once, +canonicalize it, use that single value for both the comparison and the lock target, +and refuse if the caller supplied anything else**. WP11 never re-reads the ambient +home after that point, and the residue guard's own later read is covered only +because it is bounded by N — a second process that changes the environment cannot +change ours, and our own code does not mutate `CODEX_HOME` mid-operation. + +That last clause is a claim, not an assumption, so it needs a guard rather than a +comment: the B phase adds a test that fails if any production module under `src/` +assigns to `process.env.CODEX_HOME`. Today `rg -n "env.CODEX_HOME\s*=" src` finds +nothing, and an absence that nothing enforces is precisely the defect this unit has +now hit fourteen times. + +**And the grep proves the claim false, which is why it was run.** Production code +does assign `process.env.CODEX_HOME`, in two places: + +- `src/codex/history-worker.ts:158` — WP10 added this in THIS session. The Worker + receives the parent's home in its run message and installs it before doing any + history work. +- `src/storage/policy-worker.ts:34` — the same bootstrap shape, older. + +Both are Worker entry bootstraps: they set the variable once, at thread start, +before that thread resolves any path, so neither mutates the home of a thread that +is mid-operation. That makes the invariant WP11 needs narrower and checkable: +**no assignment to `process.env.CODEX_HOME` outside a Worker bootstrap**, i.e. none +on a thread that could be holding N. The B-phase guard asserts exactly that, with +the two known bootstraps as named exceptions, so a third assignment added on a +request path fails the test instead of silently invalidating the once-resolved home. + +Writing this down mattered more than the guard does. The paragraph above originally +asserted the grep was empty; running it produced two hits, one of them added by this +very session. That is instance #15 in miniature — the absence was asserted from +memory of the design instead of from the tree — and it is the reason every claim in +this phase gets executed rather than recalled. + +#### Two more holes review found in this same remedy + +The canonicalize-once rule above is necessary and still not sufficient. + +**A symlinked default home refuses itself.** With no `CODEX_HOME` set, +`getCodexHome()` returns `defaultCodexHome()` **without** `realpath` +(`src/codex/paths.ts:23`), while WP11's target is `realpathSync.native`-canonical. On +a machine where `~` or `~/.codex` is a symlink, the two strings differ and the lock +refuses a home that is in fact the same directory. The comparison must canonicalize +**both** sides before comparing, never the target alone. + +**The comparison must be adjacent to the open.** Acquisition retries across `await` +boundaries. A comparison performed before the retry loop and an +`openCodexCoordinatorTransaction` performed after it are separated by suspension +points, so the guard re-reads the environment in between. The canonical ambient home +is resolved once, and the equality check is re-asserted **immediately before** the +synchronous open with no `await` between them. + +**And one citation in the original F2 text was simply wrong.** It said +`readIntegrationRecord()` resolves its path from the ambient CODEX_HOME. It does not: +its path comes from `getConfigDir()` (`src/codex/integration-record.ts:28`), which is +`OPENCODEX_HOME`, a different variable. Only `classifyNativeRoutedResidue()` reads +the ambient Codex home. The defect is real and the mechanism was misdescribed; the +narrower true statement is what B implements against. + +### F4 — absence-as-guarantee #15: ACL success is cached by pathname, not identity + +This one is a live defect in shipped code, not only in the plan. +`hardenStableLockFile` delegates to `hardenSecretPathAsync` +(`src/codex/native-main-lock-file.ts:127`), which returns success purely because the +**pathname** is in a module-level `Set` (`src/lib/windows-secret-acl.ts:36,461`). +Nothing in that cache is bound to the file's identity. Review's executed probe +hardened a path, unlinked it, recreated it at the same name, and re-hardened: + +```text +{"firstCalls":3,"totalCalls":3,"replacementWasRechecked":false} +``` + +The replacement file received **zero** ACL calls. The plan's "validate the DB, refuse +substitution" section inherits this: it treats "no path change observed" as proof +that the cached ACL still describes the current inode. It does not, and on Windows +that means a substituted coordinator database can be adopted with the previous +file's hardening credited to it. + +Remedy for B: bind the ACL success cache to stable file identity, or invalidate the +entry when the stable descriptor's last reference closes so the next acquisition +revalidates. The regression test must be release → replace → **reacquire**; +substituting the file during a single held acquisition does not reach the cache and +would pass with the fix removed. + +### F5 — deadline exhaustion must not be a permanent refusal + +The result taxonomy carries `busy/deadline` as retryable, but the acquisition section +classifies ACL failure — including timeout — as a non-retryable refusal, and says only +SQLite busy retries. A caller-supplied short remaining deadline can time ACL work out +without proving anything unsafe; that is exhaustion, not unsafe authority. Worse, the +current ACL code rethrows a sanitized untyped `Error` +(`src/lib/windows-secret-acl.ts:484`), discarding the `ETIMEDOUT` discriminator the +classification would need. + +B preserves a typed timeout discriminator through sanitization, maps outer-budget +exhaustion to `busy/deadline`, and reserves non-retryable refusal for verified +ACL/ownership/path failures. + +### F6 — citation corrections + +Verified accurate: `native-main-lock-file.ts:35-55,74-131`, +`native-main-owner.ts:75-91`, `home.ts:135-146`, `paths.ts:6-24`. + +Corrected: + +| Cited | Problem | Use instead | +|---|---|---| +| `src/config.ts:1853-1859` | those lines are config-generation reads, not the conditional-rename statement | `src/config.ts:1949` | +| `src/config.ts:1767-1818` | stops before callback execution, commit, rollback, and close | `src/config.ts:1779-1839` | +| `windows-secret-acl.ts:217-328,404-494` | misses `HardenOptions`/deadline clamping and the exported async entry | add `:45` and `:512` | + +### The type block must not redeclare the capability + +The public-contract fence below prints its own `unique symbol` brand and its own +`CodexCoordinatorTransaction`. Both already exist: +`src/codex/convergence-types.ts:331` owns the public interface and +`src/codex/transition-state.ts:148` owns the private brand. The fences compile in +isolation — which is exactly why the fence check did not catch this — but an +integration compile assigning `openCodexCoordinatorTransaction(...).capability` to +the plan's local type fails: + +```text +TS2741: Property '[codexCoordinatorTransactionBrand]' is missing in type +'convergence-types.CodexCoordinatorTransaction' +but required in type 'plan.CodexCoordinatorTransaction'. +``` + +The implementation **imports** `CodexCoordinatorTransaction` from +`./convergence-types` and declares no local brand. The fence below is retained as the +historical shape; where it declares the brand and interface, read the import. + +### F3 — `journal_mode` and reentrancy needed no new mechanism + +A pinned-Bun probe shows `bun:sqlite` opens in `delete` mode by default and leaves +no `-wal`/`-shm` sidecar after a committed `BEGIN IMMEDIATE`, so the plan's +"forces rollback journal mode" is a verification, not a conversion. And a second +`openCodexCoordinatorTransaction` on a held path in the SAME process already fails +with `SQLiteError: database is locked`, because `busy_timeout = 0` is set before +`BEGIN IMMEDIATE` (`src/codex/transition-state.ts:416`). + +`AsyncLocalStorage` reentrancy detection therefore is not what prevents a +same-process deadlock — SQLite already does. It exists to turn an +indistinguishable `busy` into a typed `refused/reentrant`, which is a diagnosis +improvement, not an exclusion mechanism. The test must assert the typed reason, +not "does not hang", or it proves nothing SQLite was not already doing. + ## IN / OUT IN: @@ -33,8 +315,11 @@ IN: - `src/codex/codex-write-lock.ts` (NEW) — exact contract module name; canonical target identity, effective-user namespace, finite async acquisition, synchronous coordinated commit, release, and typed lock mechanics. -- `src/codex/convergence.ts` (MODIFY) — place WP9's fixed catalog/native commit - under the new lock and pass the contract `AdmissionSnapshot`/`CommitExpectation`. +- `src/codex/convergence.ts` (MODIFY, **narrowed by F1**) — add + `convergeCodexNativeUnderLock`, the production entry that takes N and publishes a + transition. It does NOT move the existing catalog commit under N: that seam keeps + `K -> C` until WP12 supplies admission and the adoption path, because N refuses to + open on a routed home today and rewiring now would break every applied install. - `src/codex/transition-state.ts` (MODIFY through its public owner API) — lend WP11 a narrow opaque capability backed by the already-open coordinator transaction; this module remains the sole native-generation/transition-row owner. @@ -514,6 +799,15 @@ syncs, restores, or ensures the proxy; port 10100 is untouched. and USERPROFILE prove one lock for one user/home. - Config generation, authoritative admission re-read, native/provenance writes, and the conditional transition-row update share N->C; C releases before N commits. +- **F1** — a routed CODEX_HOME returns typed `refused/lock_unavailable` carrying the + coordinator's legacy-ambiguous reason rather than throwing, and the existing + catalog commit path still succeeds on a routed home because it was NOT moved + under N in this phase. +- **F2** — an explicit `codexHome` that differs from the ambient `getCodexHome()` + returns `refused/authority_not_proven` and creates no namespace, because the + coordinator's residue guard would otherwise have inspected a different directory. +- **F3** — same-process reentrancy returns `refused/reentrant`, distinguishable from + the `busy` that SQLite alone would have produced. - `transition-state.ts` alone owns native generation/txId/history scheduling; JSON owns none of them, and WP11 never opens a second coordinator connection in C. - Lock edges are N->C and short fail-fast H->N only; stale history jobs are rejected From ad58656e7c8ead286f0ed093cd034bf9b7b70350 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Wed, 5 Aug 2026 08:22:26 +0900 Subject: [PATCH 099/163] docs(codex): merge WP11 into WP12 and rewrite the body, not just the preamble Round 7 caught instance #16 in my own round 6 fix: I put six corrective findings at the TOP of the document and left the normative body below still specifying the design those findings rejected. Corrective prose above a contradictory contract is exactly the absence-as-guarantee pattern this unit keeps producing - the presence of an acknowledgement treated as proof the instruction changed. It had not. So the body is rewritten rather than annotated: - IN/OUT now names inject.ts and the admission producer, because the lock cannot be called without them. - The type block no longer redeclares CodexCoordinatorTransaction or its brand; it imports from convergence-types, where TS2741 would otherwise fire at the only assignment that matters. - ACL timeout is busy/deadline, not refusal, and the ETIMEDOUT discriminator has to survive sanitization. - The ambient-home re-assert moved INTO the retry loop, canonicalizing both sides. 'N bounds it' was struck: N serializes a database, not process.env, and is not held until the open begins. - Accept criteria each name the mutation that turns them red. Two of the old ones passed with their mechanism removed. - Test plan requires release -> replace -> REACQUIRE for the ACL memo, since substituting during one held acquisition never reaches it. And the phase boundary itself goes. WP11's only consumer is WP12, and both things needed to exercise its API arrive there, so a standalone WP11 could only prove a fabricated snapshot drives the primitive. That is the green unusable seam this unit has produced repeatedly. WP8b was different: a contract four phases consumed. Two sibling documents asserted the merged shape already; they now agree instead of contradicting. --- .../005_contract.md | 8 +- .../030_lock_protocol.md | 239 ++++++++++++++---- .../040_ownership_convergence.md | 23 +- 3 files changed, 205 insertions(+), 65 deletions(-) diff --git a/devlog/_plan/260804_codex_write_substrate/005_contract.md b/devlog/_plan/260804_codex_write_substrate/005_contract.md index f37cc636d..3042512cf 100644 --- a/devlog/_plan/260804_codex_write_substrate/005_contract.md +++ b/devlog/_plan/260804_codex_write_substrate/005_contract.md @@ -632,12 +632,14 @@ state authorizer. `history-job.ts` is the sole bridge and scheduling edge in §8 middle inventory. WP12-final graph reachability removes the compatibility handoff once admission-snapshot scheduling owns native convergence. -What does **not** move from WP11 is equally explicit: WP10 does not implement +What does **not** move is equally explicit: WP10 does not implement `codex-write-lock.ts`, the uid/SID namespace mechanics for that full lock, canonical target/admission validation, finite async acquisition/retry and result taxonomy, `CommitExpectation`, provenance coordination, or adoption by every native writer. -WP11 still owns that complete async N → K → C mechanism and its broader caller -rewire. WP10 only closes the retained native-mutation-to-history-authorization gap +**WP12** owns that complete async N → K → C mechanism and its caller rewire — the two +are one phase, because round 7 established that the mechanism cannot be audited apart +from the caller that supplies its admission snapshot (`030_lock_protocol.md`, Round 6 +resolution). WP10 only closes the retained native-mutation-to-history-authorization gap using the coordinator database and transition owner that already landed in WP8b. Explicit legacy recovery cannot honestly advance `nativeGeneration`: §3 defines it diff --git a/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md b/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md index ee5151c40..22620df50 100644 --- a/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md +++ b/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md @@ -104,25 +104,51 @@ So the narrowing as written *was* a dodge, and its own falsification test — "i apply path cannot legally take N either, fold WP11 into WP12" — has now fired. The resolution is recorded below rather than argued away. -#### Resolution: WP11 becomes mechanism-only, and its caller moves to WP12 - -WP11 ships `src/codex/codex-write-lock.ts` and its tests, and **nothing in -`convergence.ts`**. It is explicitly a mechanism phase whose consumer is WP12, in the -same way WP8b shipped the contract before anything consumed it. This is not the same -failure as defect #10 (a coordinator no production caller ever initialized) provided -two conditions hold, and B must satisfy both: - -- The goalplan records WP12 as the phase that supplies the admission producer, the - apply/restore split in `inject.ts`, and the first real call edge. WP11 is not - closable as "done" in the sense of "in production"; it is closable as "the - mechanism is correct and proven by real two-process tests". -- WP11's tests drive the **production module**, not a copy, through real child - processes — which its test plan already requires — so the mechanism is executed - even before a production caller exists. - -If WP12 does not land the caller, the honest state of this unit is that the lock is -unused, and the PR body must say exactly that instead of implying the substrate is -live. That sentence is a deliverable of WP14, not a footnote. +#### Resolution: WP11 MERGES INTO WP12. This document becomes WP12's lock section. + +My first attempt at a resolution was "mechanism-only": ship the lock and its tests, +move the caller to WP12, and defend the boundary by analogy to WP8b, which also +shipped before it had a consumer. Round 7 rejected the analogy, correctly. + +WP8b was a **contract** consumed by four later phases; publishing it first is what +stopped WP9-WP12 from inventing four incompatible shapes. WP11 has exactly **one** +planned consumer, and the two things needed to exercise its API — a production +`AdmissionSnapshot` producer and the apply/restore split in `inject.ts` — both arrive +in that same consumer. So a standalone WP11 can only prove that a **fabricated** +snapshot and a **fabricated** callback drive the primitive. It cannot prove the API +fits the one real caller it exists for. That is precisely the shape this unit keeps +producing: a green, unusable seam. + +The surrounding documents already assume the merged shape and were never consistent +with a standalone WP11: + +- `005_contract.md:635` assigns WP11 "that complete async N → K → C mechanism **and + its broader caller rewire**". +- `040_ownership_convergence.md:15` states WP9-WP11 "already provide the working + `convergeCodex` funnel ... and native lock" — a claim a mechanism-only WP11 makes + false. +- `040_ownership_convergence.md:211` is where the actual call edge lives. + +So the merge is not a concession, it is the reading that makes three documents agree. + +**What this means concretely:** + +- The N mechanism has **no independent completion gate**. It is audited together + with its first production caller, or it is not audited. +- It may still land as its own commit for reviewability. A commit boundary is not a + phase boundary. +- The goalplan work-phase is restructured accordingly: `wp11` is closed as *merged*, + and `wp12` carries the mechanism, the admission producer, the `inject.ts` + apply/restore split, and the first call edge as required tasks. +- **F4 is the exception and lands alone.** The ACL pathname-cache defect is a live + bug in shipped code (`src/lib/windows-secret-acl.ts`), independent of the lock, and + it has its own falsifiable test. It does not wait for WP12. + +Everything below this line is therefore **WP12's lock section**, rewritten to the +decisions above. Where the historical text conflicts with them, the decisions win — +and the sections that conflicted have been rewritten rather than annotated, because +round 7's finding was exactly that corrective prose above a contradictory body is +instance #16 of treating an absence as a guarantee. ### F2 — the initializer's residue guard reads the AMBIENT home, not the locked one @@ -315,11 +341,18 @@ IN: - `src/codex/codex-write-lock.ts` (NEW) — exact contract module name; canonical target identity, effective-user namespace, finite async acquisition, synchronous coordinated commit, release, and typed lock mechanics. -- `src/codex/convergence.ts` (MODIFY, **narrowed by F1**) — add - `convergeCodexNativeUnderLock`, the production entry that takes N and publishes a - transition. It does NOT move the existing catalog commit under N: that seam keeps - `K -> C` until WP12 supplies admission and the adoption path, because N refuses to - open on a routed home today and rewiring now would break every applied install. +- `src/codex/convergence.ts` (MODIFY) — add the native convergence entry that takes + N and publishes a transition, called by the WP12 admission pipeline. It does NOT + move the **existing catalog commit** under N: that seam keeps its current `K -> C` + (`src/codex/convergence.ts:393-406`), because N refuses to open on a routed home + and rewiring it would break every applied install (F1). +- `src/codex/inject.ts` (MODIFY) — split the synchronous native mutation from the + awaited history dispatch (`:530` journal, `:601` native writes, `:614` awaited + history) so the native section can sit beneath N and history stays outside it. +- The WP12 admission producer (MODIFY/NEW per `040_ownership_convergence.md`) — + without it there is no `AdmissionSnapshot` at runtime and the lock's API cannot be + called at all. `AdmissionSnapshot` is today only an interface + (`src/codex/convergence-types.ts:495`). - `src/codex/transition-state.ts` (MODIFY through its public owner API) — lend WP11 a narrow opaque capability backed by the already-open coordinator transaction; this module remains the sole native-generation/transition-row owner. @@ -366,7 +399,7 @@ them in `src/codex/codex-write-lock.ts`; it does not publish the former ```ts import type { AdmissionSnapshot, - BeginCodexTransition, + CodexCoordinatorTransaction, CommitExpectation, } from "./convergence-types"; @@ -417,14 +450,16 @@ export interface CodexWriteCommitContext { readonly coordinator: CodexCoordinatorTransaction; } -const codexCoordinatorTransactionBrand: unique symbol = Symbol( - "CodexCoordinatorTransaction", -); - -export interface CodexCoordinatorTransaction { - readonly [codexCoordinatorTransactionBrand]: true; - readonly beginTransition: BeginCodexTransition; -} +// NO local brand and NO local interface here. `CodexCoordinatorTransaction` is +// imported above from `./convergence-types` (`src/codex/convergence-types.ts:331`); +// its private brand belongs to `src/codex/transition-state.ts:148` and to nothing +// else. Redeclaring either compiles fine in isolation — which is why the per-document +// fence check did not catch it for several rounds — and then fails at the only place +// that matters, assigning a real `openCodexCoordinatorTransaction(...).capability`: +// +// TS2741: Property '[codexCoordinatorTransactionBrand]' is missing in type +// 'convergence-types.CodexCoordinatorTransaction' but required in type +// 'plan.CodexCoordinatorTransaction'. type Synchronous = T extends PromiseLike ? never : T; @@ -445,8 +480,9 @@ export type WithCodexWriteLock = ( `CodexWriteLockResult` is the lock module's own bounded mechanism result. `convergence.ts` exhaustively projects it into `ConvergeOutcome`; no route consumes it directly. `CodexCoordinatorTransaction` is the only handle passed to the -callback. It is branded and exposes only the contract's null-safe conditional -transition-row update. It is one-shot for this transition, and WP11 verifies that +callback — **imported**, not redeclared, from `./convergence-types`. It is branded by +`transition-state.ts` alone and exposes only the contract's null-safe conditional +transition-row update. It is one-shot for this transition, and the lock verifies that it returned `updated` for the exact expectation before allowing C to release. It exposes neither the `Database` object nor `COMMIT`, `ROLLBACK`, or `close`. Opening another connection in the callback is wrong: it @@ -486,7 +522,9 @@ an authoritative re-read inside the coordinated commit; WP11 does not reduce it a boolean or manufacture an authority receipt. `withConfigMutationLockSync` is already synchronous, fail-fast, and reentrant only -for the current synchronous stack (`src/config.ts:1767-1818`). The native lock may +for the current synchronous stack (`src/config.ts:1779-1839`, which includes the +callback execution, commit, rollback, and close the shorter range cut off). The +native lock may hold it because no await occurs. Config-generation reads/updates and provenance-only `updateIntegrationRecord` calls happen before that callback returns. The native generation bump, `txId`, and pending history schedule are owned by the @@ -503,8 +541,8 @@ If the config coordinator is busy, the attempt releases the native lock and retr only while the outer monotonic deadline remains; deadline expiry returns typed `busy`. It never releases and commits against the old admission. A non-cooperating filesystem writer remains detectable after commit, as scoped by `005_contract.md` -§3; WP11 does not promise a portable conditional rename that `src/config.ts:1853-1859` -explicitly says the filesystem lacks. +§3; this phase does not promise a portable conditional rename that +`src/config.ts:1949` explicitly says the filesystem lacks. ## Canonical `CODEX_HOME` identity — C6 @@ -566,9 +604,23 @@ Walk components one at a time; never recursive-mkdir across an unvalidated paren existing path. - Windows validates non-junction identity and runs the existing required per-user ACL owner within the remaining outer deadline - (`src/lib/windows-secret-acl.ts:217-328,404-494`). Failure/timeout refuses. + (`src/lib/windows-secret-acl.ts:45,217-328,404-494,512`). A **verified** ACL, + ownership, or path failure refuses. **Timeout does not**: exhausting the outer + budget is `busy/deadline` and retryable, because a short caller-supplied deadline + proves nothing about safety (F5). That requires preserving the `ETIMEDOUT` + discriminator through sanitization at `src/lib/windows-secret-acl.ts:484`, which + today rethrows an untyped `Error` and destroys it. +- The ACL success memo must be bound to **file identity**, not pathname. Today it is + a `Set` of paths (`src/lib/windows-secret-acl.ts:36,461`), so a file + replaced at the same name inherits the previous file's hardening — probed as + `{identityChanged:true, firstCalls:3, totalCalls:3, replacementWasRechecked:false}` + (F4). Ephemeral temps already invalidate through `forgetEphemeralSecretPath` + (`src/config.ts:214,241,309,336,480,501-510`); the stable destination memo that + `hardenStableLockFile` uses never does. - Existing DB or `-journal` must be regular, same-user private entries. Existing - `-wal`/`-shm` refuses; WP11 forces rollback journal mode. + `-wal`/`-shm` refuses. The lock **verifies** rollback journal mode rather than + forcing it: a pinned-Bun probe shows `bun:sqlite` already opens `delete` and leaves + no `-wal`/`-shm` sidecar after a committed `BEGIN IMMEDIATE` (F3). - `openStableLockFile` retains the side descriptor; validate descriptor metadata, assert path identity before/after SQLite open, after `BEGIN IMMEDIATE`, before commit, and before close. @@ -606,17 +658,51 @@ No `node:os` home accessor is imported. The total timeout is required, finite, integral, and within `0..30_000` ms. Acquisition uses monotonic `performance.now()`. Zero receives one fail-fast -`BEGIN IMMEDIATE`. Only SQLite busy/locked retries; filesystem, ACL, malformed DB, -identity, permission, and journal-mode failures are refusals. +`BEGIN IMMEDIATE`. **Two** conditions retry: SQLite busy/locked, and exhaustion of +the outer budget during Windows ACL work, which is `busy/deadline` (F5). Verified +filesystem, ACL, malformed DB, identity, permission, and journal-mode failures are +refusals. The distinction is not cosmetic: a refusal tells the caller never to try +again, and a short deadline is not evidence of an unsafe namespace. Retry sleeps are async uniformly bounded 25-75 ms, clipped to remaining deadline, and abortable. Barging is allowed; no caller/test infers FIFO. Candidate SQLite and side descriptors close after every failed attempt. -`AsyncLocalStorage>` rejects same-task same-home reentrancy. +`AsyncLocalStorage>` rejects same-task same-home reentrancy. It +is a **diagnosis** layer, not the exclusion mechanism: a second open on a held path +in the same process already fails `SQLITE_BUSY`, because `busy_timeout = 0` precedes +`BEGIN IMMEDIATE` (`src/codex/transition-state.ts:416`). Its only job is to turn that +indistinguishable `busy` into a typed `refused/reentrant` (F3), so its test asserts +the typed reason and not "does not hang" — the latter passes with ALS deleted. A separate task is an ordinary contender. Caller exceptions propagate after rollback/release; they are never converted to busy/refused. +**The ambient-home re-assert sits here, in the acquisition loop, not before it.** +Every attempt performs, in one uninterrupted synchronous stack: + +```text +canonicalAmbient = realpathSync.native(getCodexHome()) +require canonicalAmbient === canonicalTarget // both sides canonicalized +openCodexCoordinatorTransaction(finalDatabasePath) // no await, no callback between +``` + +Re-comparing the value captured before the retry sleeps proves nothing, because +`getCodexHome()` re-reads the environment on every call (`src/codex/paths.ts:32-35`) +and the coordinator's own residue guard reads it again inside the open. Only a fresh +read that shares a synchronous stack with the open denies another task the chance to +interleave. "N bounds it" is false and was struck: N serializes the coordinator +database, not `process.env`, and it is not even held until the open begins. + +Canonicalizing **both** sides is required, not tidiness. With no `CODEX_HOME` set, +`getCodexHome()` returns `defaultCodexHome()` **without** `realpath` +(`src/codex/paths.ts:23`), so on a machine where `~/.codex` is a symlink an +uncanonicalized ambient value refuses the very home it names. + +The real fix is to parameterize `classifyNativeRoutedResidue()` with the canonical +target and remove ambient authority from the guard entirely. The adjacency rule above +is the bounded version that survives until that lands, and the merged WP12 phase +carries the parameterization as a task. + ```diff +const transaction = openCodexCoordinatorTransaction(finalDatabasePath); +// transaction has already executed BEGIN IMMEDIATE: N is held here. @@ -758,8 +844,15 @@ probe. Do not substitute Node or a same-process environment mutation. - Compile-time async callback rejection plus runtime thenable rejection and release. - Callback throw releases then propagates. - Namespace symlink/junction, wrong owner/mode, DB/journal substitution, WAL/SHM, - malformed DB, ACL failure/timeout, unsupported filesystem all refuse without - repair/deletion. + malformed DB, **verified** ACL failure, and unsupported filesystem all refuse + without repair/deletion. ACL **timeout** is `busy/deadline`, not refusal (F5). +- **Release → replace → reacquire** (F4): harden a coordinator DB, release the + acquisition, unlink and recreate a different file at the same pathname, then + reacquire and assert the replacement was re-hardened. Substituting the file while + a single acquisition is still held does NOT exercise the memo and passes with the + fix removed — that shape is explicitly insufficient. +- Environment mutation **during** acquisition retry, not merely before it: a home + changed while the contender sleeps must be caught by the fresh adjacent read. - Windows CI executes real SID/junction/ACL success; POSIX executes real uid/mode. - Dependency graph proves no inverse C->N or C->H acquisition and no held N->H; history's only H->N edges are the fail-fast claim and terminal operations. @@ -790,27 +883,61 @@ syncs, restores, or ensures the proxy; port 10100 is untouched. ## Accept criteria +Every criterion below names the **mutation that must turn it red**. A criterion with +no such mutation is not a criterion; this unit has shipped five live defects beside +8000 passing tests, so "the suite is green" carries no weight here. Each one was +checked against the question *would this still pass with the mechanism removed?* — +and the ones that did were rewritten rather than kept. + - **C5** — finite async acquisition yields typed acquired/busy/refused behavior; callback is synchronous/bounded; no stale takeover or FIFO claim exists. + *Red when:* N acquisition is replaced by direct callback execution — the real + two-process exclusion test must fail. - **C6** — all real spellings of one existing home share one lock; distinct homes do not; missing homes refuse before artifacts. + *Red when:* the canonicalization step is dropped — symlink and tilde spellings + must stop contending. - **C7/C18** — namespace keys on effective uid/SID beneath the OS runtime directory, never any home accessor. Real pinned-Bun children with independently varied HOME and USERPROFILE prove one lock for one user/home. + *Red when:* the uid/SID component is replaced by any home accessor — the two + children must stop sharing one lock. - Config generation, authoritative admission re-read, native/provenance writes, and the conditional transition-row update share N->C; C releases before N commits. -- **F1** — a routed CODEX_HOME returns typed `refused/lock_unavailable` carrying the - coordinator's legacy-ambiguous reason rather than throwing, and the existing - catalog commit path still succeeds on a routed home because it was NOT moved - under N in this phase. -- **F2** — an explicit `codexHome` that differs from the ambient `getCodexHome()` - returns `refused/authority_not_proven` and creates no namespace, because the - coordinator's residue guard would otherwise have inspected a different directory. + *Red when:* the apply/restore → N call edge is removed — the production-path test + must fail. This criterion is unmeetable without the WP12 caller, which is exactly + why the phases are merged. +- **F1** — a routed CODEX_HOME returns a typed refusal carrying the coordinator's + legacy-ambiguous reason rather than throwing, **and** the existing catalog commit + still succeeds on that same routed home. + *Red when:* the catalog commit is moved under N — the routed-home catalog test + must fail. That is the regression this finding exists to prevent. +- **F2** — the successful matched-home path reaches `BEGIN IMMEDIATE`, and a home + changed **during acquisition retry** is caught. + *Red when:* the fresh adjacent ambient read is replaced by the value captured + before the retry sleeps — the environment-change-during-retry test must fail. A + test that only exercises mismatched-home refusal is insufficient: it passes even + if the matched path never acquires anything. - **F3** — same-process reentrancy returns `refused/reentrant`, distinguishable from - the `busy` that SQLite alone would have produced. + the `busy` SQLite alone produces. + *Red when:* ALS is removed — the reason must degrade to `busy`. Asserting only + "does not hang" is vacuous, because `busy_timeout = 0` already guarantees that. +- **F4** — a coordinator database released, replaced at the same pathname, and + reacquired is re-hardened. + *Red when:* identity binding / memo invalidation is removed — the + release → replace → **reacquire** test must fail. Substituting during a single + held acquisition never reaches the memo and would pass with the fix gone. +- **F5** — outer-budget exhaustion during ACL work returns retryable + `busy/deadline`; verified ACL/ownership/path failure returns non-retryable refusal. + *Red when:* `ETIMEDOUT` is collapsed into a generic refusal — the deadline + classification test must fail. - `transition-state.ts` alone owns native generation/txId/history scheduling; JSON - owns none of them, and WP11 never opens a second coordinator connection in C. + owns none of them, and the lock never opens a second coordinator connection in C. - Lock edges are N->C and short fail-fast H->N only; stale history jobs are rejected by generation/transaction identity without any C->N, C->H, or held N->H edge. -- **N2** — WP11 extends the already-working funnel and typechecks/preserves behavior - at its own commit; WP12 strengthens admission without supplying missing mechanics. +- **Phase honesty** — the goalplan records the merge: `wp11` closed as *merged*, and + `wp12` carrying the mechanism, the admission producer, the `inject.ts` split, and + the first call edge as required tasks. If the caller does not land, the PR body + says the lock is unused, in those words. + *Red when:* the WP12 admission producer is removed — the production-entry test or + the compile must fail. diff --git a/devlog/_plan/260804_codex_write_substrate/040_ownership_convergence.md b/devlog/_plan/260804_codex_write_substrate/040_ownership_convergence.md index b8b26bf95..27534fd72 100644 --- a/devlog/_plan/260804_codex_write_substrate/040_ownership_convergence.md +++ b/devlog/_plan/260804_codex_write_substrate/040_ownership_convergence.md @@ -12,12 +12,23 @@ teardown fails open for errors outside its one mismatch class check also does not authorize overwriting a `config.toml` whose effective `model_provider` is now external. -WP9-WP11 already provide the working `convergeCodex` funnel, catalog split, -history protocol, CODEX_HOME-keyed coordinator row, integration-record owner, and -native lock. WP12 completes the mechanisms behind that funnel: tri-state service -authority, file-backed intent, journal/provenance admission, restoration, and -observed-state inspection. It does **not** add another record module, another -convergence module, another route mapping, or another public result union. +WP9 and WP10 provide the catalog split, the history protocol, the CODEX_HOME-keyed +coordinator row, and the integration-record owner. + +**The native lock is NOT among them.** Round 7 merged WP11 into this phase: the N +mechanism has exactly one consumer, and the two things needed to exercise its API — +a runtime `AdmissionSnapshot` producer and the `inject.ts` synchronous-native / +awaited-history split — both live here. A standalone WP11 could only have proven that +a fabricated snapshot drives the primitive, never that its API fits its one real +caller. The lock's design is `030_lock_protocol.md`, which is now this phase's lock +section; the sentence that used to stand here claimed a working funnel WP11 had not +in fact delivered. + +WP12 therefore delivers the lock **with its first production caller**, plus the +mechanisms behind the funnel: tri-state service authority, file-backed intent, +journal/provenance admission, restoration, and observed-state inspection. It does +**not** add another record module, another convergence module, another route mapping, +or another public result union. The prior plan named `write-lock.ts`, created `ownership-convergence.ts`, redefined `integrations/codex.json`, and exported `convergeCodexToPersistedIntent`. Those are From fa610fdd8e5aa8e451aa0e8412bdf8872b272a9f Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Wed, 5 Aug 2026 08:25:46 +0900 Subject: [PATCH 100/163] fix(acl): the harden memo remembered a pathname, so a replaced file inherited its ACLs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit hardenStableLockFile hardens a stable destination — the coordinator database among them. The success memo was a Set of PATH STRINGS, so unlinking that path and recreating a different file at the same name left the replacement reported as hardened while it had never been through icacls once. Probed: {identityChanged: true, firstCalls: 3, totalCalls: 3, replacementWasRechecked: false} Ephemeral temps escaped this only because atomic writers call forgetEphemeralSecretPath once the temp is gone (config.ts:214 and friends). Nothing does that for a stable path, and nothing ever would. The memo is now a Map from path to file identity, read with bigint:true because the plain ino is 0 on NTFS — the platform this code exists for. Two broken-change checks, each restored with git diff --stat clean: 1. memo satisfied by pathname alone -> the replacement test fails, the unverifiable test still passes. 2. a null (unverifiable) identity treated as unchanged -> the unverifiable test fails, the replacement test still passes. The second mutation is the reason there are two tests. My first version had one, and it passed with that branch inverted — an unverifiable identity would have been read as proof of an unchanged file, which is the same absence-as-guarantee move that produced the pathname memo to begin with, reintroduced inside its own fix and on the only platform that reaches it. --- src/lib/windows-secret-acl.ts | 65 ++++++++++++++++--- tests/windows-secret-acl.test.ts | 104 ++++++++++++++++++++++++++++++- 2 files changed, 159 insertions(+), 10 deletions(-) diff --git a/src/lib/windows-secret-acl.ts b/src/lib/windows-secret-acl.ts index 28a06dc08..3b590e738 100644 --- a/src/lib/windows-secret-acl.ts +++ b/src/lib/windows-secret-acl.ts @@ -29,14 +29,61 @@ * hardenSecretDir — same contract for directories. */ -import { existsSync } from "node:fs"; +import { existsSync, statSync } from "node:fs"; import { env, platform } from "node:process"; -const hardenedDirectories = new Set(); -const hardenedPaths = new Set(); +const hardenedDirectories = new Map(); +const hardenedPaths = new Map(); /** Paths whose harden TIMED OUT this process: do not re-stall every loadConfig on them. */ const timedOutPaths = new Set(); +/** + * Identity of the file a successful harden actually applied to. + * + * `null` means the identity could not be established, which is NOT the same as + * "unchanged" — it is recorded as unverifiable so the next harden re-runs + * instead of inheriting a previous file's credit. + */ +type HardenedIdentity = string | null; + +/** + * Portable file identity for the ACL success memo. + * + * `bigint: true` is what makes this work on Windows: the default `ino` is 0 on + * NTFS, while the bigint variant carries the file index. A zero or failed read + * yields `null`, and a null memo entry never satisfies a later lookup. + */ +function fileIdentity(targetPath: string): HardenedIdentity { + try { + const stats = statSync(targetPath, { bigint: true }); + if (stats.ino === 0n) return null; + return `${stats.dev}:${stats.ino}`; + } catch { + return null; + } +} + +/** + * True only when this exact FILE was hardened, not merely this pathname. + * + * The memo used to be a `Set` of paths. A stable destination — such as + * the coordinator database `hardenStableLockFile` hardens — can be unlinked and + * recreated at the same name, and the replacement inherited the previous file's + * hardening while never having been through icacls. Ephemeral temps escaped this + * only because atomic writers call `forgetEphemeralSecretPath` once the temp is + * gone; nothing does that for a stable path. + */ +function memoSatisfied(cache: Map, targetPath: string): boolean { + if (!cache.has(targetPath)) return false; + const remembered = cache.get(targetPath); + // Unverifiable at harden time stays unverifiable now: re-harden rather than + // treat an unknown identity as proof of an unchanged file. + if (remembered === null) return false; + const current = fileIdentity(targetPath); + if (current === null) return false; + return current === remembered; +} + export interface HardenResult { ok: boolean; diagnostics?: string; @@ -411,11 +458,11 @@ function hardenEntry( targetPath: string, directory: boolean, opts: HardenOptions, - cache: Set, + cache: Map, ): HardenResult { if (!existsSync(targetPath)) return { ok: true }; if (effectivePlatform() !== "win32") return { ok: true }; - if (cache.has(targetPath)) return { ok: true }; + if (memoSatisfied(cache, targetPath)) return { ok: true }; const memoKey = timeoutMemoKey(targetPath, opts); if (timedOutPaths.has(memoKey)) { const diagnostics = "ACL hardening skipped — previous attempt timed out"; @@ -429,7 +476,7 @@ function hardenEntry( if (attempt > 0 && deadline - nowFn() <= 0) break; // retry only while budget remains try { runIcacls(targetPath, directory, deadline); - cache.add(targetPath); + cache.set(targetPath, fileIdentity(targetPath)); return { ok: true }; } catch (err) { lastErr = err; @@ -455,11 +502,11 @@ async function hardenEntryAsync( targetPath: string, directory: boolean, opts: HardenOptions, - cache: Set, + cache: Map, ): Promise { if (!existsSync(targetPath)) return { ok: true }; if (effectivePlatform() !== "win32") return { ok: true }; - if (cache.has(targetPath)) return { ok: true }; + if (memoSatisfied(cache, targetPath)) return { ok: true }; const memoKey = timeoutMemoKey(targetPath, opts); if (timedOutPaths.has(memoKey)) { const diagnostics = "ACL hardening skipped — previous attempt timed out"; @@ -473,7 +520,7 @@ async function hardenEntryAsync( if (attempt > 0 && deadline - nowFn() <= 0) break; try { await runIcaclsAsync(targetPath, directory, deadline); - cache.add(targetPath); + cache.set(targetPath, fileIdentity(targetPath)); return { ok: true }; } catch (err) { lastErr = err; diff --git a/tests/windows-secret-acl.test.ts b/tests/windows-secret-acl.test.ts index ce6e4edb1..d00ea9598 100644 --- a/tests/windows-secret-acl.test.ts +++ b/tests/windows-secret-acl.test.ts @@ -10,7 +10,7 @@ * - hardenSecretDir mirrors the same contract for directories. */ import { afterEach, beforeEach, describe, expect, test } from "bun:test"; -import { chmodSync, existsSync, mkdtempSync, renameSync, rmSync, truncateSync, unlinkSync, writeFileSync } from "node:fs"; +import { chmodSync, existsSync, mkdtempSync, renameSync, rmSync, statSync, truncateSync, unlinkSync, writeFileSync } from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { @@ -693,3 +693,105 @@ describe("ephemeral ACL memo release (#840 refinement)", () => { } }); }); + +describe("stable-path harden memo is bound to file identity, not pathname", () => { + /** + * The memo is a Set of PATHNAMES. Ephemeral temps escape the + * consequence because atomic writers call forgetEphemeralSecretPath after the + * temp is gone (src/config.ts:214,241,309,336,480,501-510). A STABLE + * destination never does — and hardenStableLockFile + * (src/codex/native-main-lock-file.ts:127) hardens exactly such a path. + * + * So: harden a stable path, then replace the FILE at that same name. The + * replacement is a different inode that has never been through icacls, but + * the memo answers for the name and reports it hardened. + * + * This is the release -> replace -> REACQUIRE shape. Substituting the file + * while a single acquisition still holds it never consults the memo again and + * would pass with the fix removed. + */ + test("a file replaced at an already-hardened stable path is hardened again", () => { + resetHardenedStateForTests(); + const stable = join(testDir, "coordinator.sqlite"); + writeFileSync(stable, "first", "utf8"); + const firstIdentity = statSync(stable).ino; + + setPlatformForTests("win32"); + const previousUsername = process.env.USERNAME; + process.env.USERNAME = "ocx-test-user"; + let grants = 0; + setIcaclsRunnerForTests(args => { + if (args.includes("/grant:r")) grants += 1; + return { success: true, exitCode: 0, timedOut: false, stdout: "" }; + }); + try { + expect(hardenSecretPath(stable, { required: true })).toEqual({ ok: true }); + expect(grants).toBe(1); + + // Release, then replace the file at the SAME pathname. + unlinkSync(stable); + writeFileSync(stable, "second", "utf8"); + const secondIdentity = statSync(stable).ino; + // Guard the guard: if the filesystem reused the inode this proves nothing. + expect(secondIdentity).not.toBe(firstIdentity); + + // Reacquire. The replacement has never been hardened. + expect(hardenSecretPath(stable, { required: true })).toEqual({ ok: true }); + expect(grants).toBe(2); + } finally { + if (previousUsername === undefined) delete process.env.USERNAME; + else process.env.USERNAME = previousUsername; + setIcaclsRunnerForTests(null); + setPlatformForTests(null); + } + }); + + /** + * The other half of the memo, which my first attempt at this fix missed. + * + * When identity cannot be established the memo stores `null`. Windows NTFS is + * exactly where that arises — the non-bigint `ino` is 0 there, which is why the + * read uses `bigint: true` and still has to tolerate failure. + * + * A `null` entry must NOT satisfy a later lookup. Treating "we could not tell" + * as "it did not change" is the same absence-as-guarantee move that produced the + * pathname memo in the first place, and it would reintroduce the defect on the + * one platform this code exists for. + * + * Driven red: mutating `remembered === null` to return true makes this fail + * while the replacement test above still passes — which is why both exist. + */ + test("an unverifiable identity is re-hardened rather than assumed unchanged", () => { + resetHardenedStateForTests(); + const stable = join(testDir, "unverifiable.sqlite"); + writeFileSync(stable, "first", "utf8"); + + setPlatformForTests("win32"); + const previousUsername = process.env.USERNAME; + process.env.USERNAME = "ocx-test-user"; + let grants = 0; + // The file is removed DURING the harden, so the post-icacls identity read + // fails and the memo records `null`. That is a real race — a replacement + // landing between the ACL call and the memo write — not a contrived seam. + setIcaclsRunnerForTests(args => { + if (args.includes("/grant:r")) grants += 1; + if (grants === 1 && existsSync(stable)) unlinkSync(stable); + return { success: true, exitCode: 0, timedOut: false, stdout: "" }; + }); + try { + expect(hardenSecretPath(stable, { required: true })).toEqual({ ok: true }); + expect(grants).toBe(1); + + // Recreate at the same name. The memo holds `null` for this path, so the + // harden must run again rather than credit the vanished file's ACLs. + writeFileSync(stable, "second", "utf8"); + expect(hardenSecretPath(stable, { required: true })).toEqual({ ok: true }); + expect(grants).toBe(2); + } finally { + if (previousUsername === undefined) delete process.env.USERNAME; + else process.env.USERNAME = previousUsername; + setIcaclsRunnerForTests(null); + setPlatformForTests(null); + } + }); +}); From d7f1a8e1fa723ea5f9861abd94aee33ccb8bf9fc Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Wed, 5 Aug 2026 08:28:27 +0900 Subject: [PATCH 101/163] docs(codex): another process can retarget the home symlink, so adjacency was never the mechanism MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Round 8. Three things, the first proven by probe rather than argued. I had written that the fresh in-loop ambient read 'closes F2' and that parameterizing the residue guard was the eventual real fix. Both wrong. The adjacent read closes JavaScript task interleaving and nothing else — two realpath calls are two filesystem observations, and a third party owns the selector between them: check saw: clean-A routed-B> guard saw: routed-B SAME SYNCHRONOUS STACK, DIFFERENT DIRECTORY: true The lock would have held coordinator A while the guard cleared B. So passing the canonical target into classifyNativeRoutedResidue is required for correctness in this phase; the adjacent read is demoted to defense-in-depth and says so. Second, instance #17: I declared that every accept criterion carries a mutation that turns it red, then wrote criteria whose mutation falsifies only part of the claim. The N->C criterion passed if the call edge stayed while any single seam escaped it. F1 passed if the refusal became a throw. F2 passed for the very implementation the probe above defeats. Two criteria had no mutation at all. The presence of a 'Red when' sentence was treated as proof it falsified the whole criterion — the same move, one level up. Third, the merge is now real everywhere rather than only where I looked: 000_plan's phase table, 005's OUT list, 040's CONSUME row (now NEW/IMPLEMENT), and 030's own opening line that still called WP11 independently landable. The merged wp11 shell no longer duplicates c5/c6/c7/c18, wp12 is in_progress, and F4 is a separate commit inside wp12 rather than an independent merge with no phase auditing it. --- .../260804_codex_write_substrate/000_plan.md | 6 +- .../005_contract.md | 5 +- .../030_lock_protocol.md | 93 +++++++++++++------ .../040_ownership_convergence.md | 2 +- 4 files changed, 74 insertions(+), 32 deletions(-) diff --git a/devlog/_plan/260804_codex_write_substrate/000_plan.md b/devlog/_plan/260804_codex_write_substrate/000_plan.md index fe31ad2d5..705e5f995 100644 --- a/devlog/_plan/260804_codex_write_substrate/000_plan.md +++ b/devlog/_plan/260804_codex_write_substrate/000_plan.md @@ -43,11 +43,11 @@ it rather than inventing their share. | WP8b | `005_contract.md` | The shared surfaces: record schema + owner, `/api/sync` response contract, the single convergence entry point, generation counters, module names, and the config-snapshot admission result | — | | WP9 | `010_catalog_seam.md` | gather/commit split + typed outcome, consuming the contract | WP8b | | WP10 | `020_history_isolation.md` | history off the event loop, and the cross-process history protocol | WP8b | -| WP11 | `030_lock_protocol.md` | the async per-home lock, per-USER namespace | WP8b, WP9, WP10 | -| WP12 | `040_ownership_convergence.md` | tri-state authority, admission order, absence restoration | WP11 | +| ~~WP11~~ | `030_lock_protocol.md` | **merged into WP12** — the lock has exactly one consumer, and the `AdmissionSnapshot` producer plus the `inject.ts` split that its API needs both live there | — | +| WP12 | `040_ownership_convergence.md` + `030_lock_protocol.md` | the async per-home lock and per-USER namespace **with its first production caller**, plus tri-state authority, admission order, absence restoration | WP8b, WP9, WP10 | | WP13 | `050_composed_acceptance.md` | one acceptance suite against real production entry points | all | -WP9 and WP10 remain independent of each other and both precede WP11: a lock +WP9 and WP10 remain independent of each other and both precede the lock: a lock around an unsplittable gather-and-write, or around a ten-second blocking history call, is the failure the last unit already proved. WP12 stays last of the four because its admission must run before the lock module creates anything. diff --git a/devlog/_plan/260804_codex_write_substrate/005_contract.md b/devlog/_plan/260804_codex_write_substrate/005_contract.md index 3042512cf..cee3e3194 100644 --- a/devlog/_plan/260804_codex_write_substrate/005_contract.md +++ b/devlog/_plan/260804_codex_write_substrate/005_contract.md @@ -30,8 +30,9 @@ SQLite transition row), `tests/codex-user-identity.test.ts` (NEW). OUT: catalog mechanics (WP9), history mechanics (WP10), the full native-lock -namespace/acquisition API and broad caller adoption (WP11), ownership mechanics -(WP12). WP10 is the one narrow exception to the former native-lock boundary: it +namespace/acquisition API, broad caller adoption, and ownership mechanics — all +**WP12**, which absorbed the former WP11. WP10 is the one narrow exception to that +native-lock boundary: it uses the already-owned N transaction as a compatibility native-handoff exclusion from each retained native mutation through its history authorization. The final coordinator path, transition table/CAS, config-generation API, shapes and funnel diff --git a/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md b/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md index 22620df50..f2c761feb 100644 --- a/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md +++ b/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md @@ -17,11 +17,13 @@ contract's exact `AdmissionSnapshot`; and the pinned Bun 1.3.14 probe showed bot Effective-user identity — uid on POSIX, SID on Windows — is the namespace authority (`005_contract.md` §§4, 7). -WP11 is independently landable. It consumes WP8b's identity/transition-state/types, -WP9's synchronous candidate commit, and WP10's separate history protocol. The WP11 -commit typechecks and preserves the working WP9/WP10 funnel. WP12 later supplies -stronger ownership/provenance decisions through the same `AdmissionSnapshot`; it is -not required to replace a placeholder before this phase works. +**This document is WP12's lock section.** It was written as a standalone WP11 and +opened by asserting that WP11 was independently landable; round 7 established the +opposite and merged the two phases (see the Round 6 resolution below). The lock +consumes WP8b's identity/transition-state/types, WP9's synchronous candidate commit, +and WP10's separate history protocol — and it is delivered together with the +`AdmissionSnapshot` producer and the first production caller, because without those +its API cannot be exercised by anything but a fabricated snapshot. All current-code citations and diff context below were rechecked on 2026-08-04 at `7bde9e0c977721fc0b9d8617c85ff17de7c07658`. @@ -140,9 +142,12 @@ So the merge is not a concession, it is the reading that makes three documents a - The goalplan work-phase is restructured accordingly: `wp11` is closed as *merged*, and `wp12` carries the mechanism, the admission producer, the `inject.ts` apply/restore split, and the first call edge as required tasks. -- **F4 is the exception and lands alone.** The ACL pathname-cache defect is a live - bug in shipped code (`src/lib/windows-secret-acl.ts`), independent of the lock, and - it has its own falsifiable test. It does not wait for WP12. +- **F4 is a separate COMMIT within WP12, not a separate merge.** The ACL + pathname-cache defect is a live bug in shipped code + (`src/lib/windows-secret-acl.ts`) and does not depend on the lock, so it is written + and verified first rather than queued behind the admission producer. It is still + audited under WP12's gate: "lands alone" would have meant a shipped repair with no + active phase reviewing it, which is the gap the merge exists to close. Everything below this line is therefore **WP12's lock section**, rewritten to the decisions above. Where the historical text conflicts with them, the decisions win — @@ -698,10 +703,29 @@ Canonicalizing **both** sides is required, not tidiness. With no `CODEX_HOME` se (`src/codex/paths.ts:23`), so on a machine where `~/.codex` is a symlink an uncanonicalized ambient value refuses the very home it names. -The real fix is to parameterize `classifyNativeRoutedResidue()` with the canonical -target and remove ambient authority from the guard entirely. The adjacency rule above -is the bounded version that survives until that lands, and the merged WP12 phase -carries the parameterization as a task. +**Adjacency is not sufficient, and calling it "the bounded version until the real fix +lands" was wrong.** It closes JavaScript task interleaving — nothing can run between +two synchronous calls in one isolate — and closes nothing else. Another **process** +needs no interleaving at all, because the two `realpath` calls are two separate +filesystem observations of a selector that a third party owns: + +```text +CODEX_HOME=/current, where current -> clean-A +check saw: clean-A # the fresh adjacent read accepts A + routed-B> +guard saw: routed-B # the coordinator's own read resolves B +SAME SYNCHRONOUS STACK, DIFFERENT DIRECTORY: true +``` + +That is executed output, not a hypothetical. The lock would hold coordinator A while +the safety guard cleared B. + +So parameterizing `classifyNativeRoutedResidue()` with the canonical target is +**required for correctness in this phase**, not deferred hardening: the guard must +receive the resolved directory rather than resolve one for itself. The adjacent read +stays as defense-in-depth against the in-isolate case, and it is explicitly not the +mechanism. The acceptance criterion is mutation-named accordingly: restore ambient +resolution inside the guard and a real second-process symlink-retarget test must fail. ```diff +const transaction = openCodexCoordinatorTransaction(finalDatabasePath); @@ -904,20 +928,27 @@ and the ones that did were rewritten rather than kept. children must stop sharing one lock. - Config generation, authoritative admission re-read, native/provenance writes, and the conditional transition-row update share N->C; C releases before N commits. - *Red when:* the apply/restore → N call edge is removed — the production-path test - must fail. This criterion is unmeetable without the WP12 caller, which is exactly - why the phases are merged. + This is **four independent claims**, so it takes four mutations. One that only + deletes the whole call edge still passes while any single seam escapes N->C. + *Red when (each separately):* (a) the apply/restore → N call edge is removed; + (b) the config-generation read/update moves outside N->C; (c) the + provenance/integration-record update moves outside it; (d) the native writes move + outside it. Each must fail its own test. + Unmeetable without the WP12 caller, which is exactly why the phases are merged. - **F1** — a routed CODEX_HOME returns a typed refusal carrying the coordinator's legacy-ambiguous reason rather than throwing, **and** the existing catalog commit - still succeeds on that same routed home. - *Red when:* the catalog commit is moved under N — the routed-home catalog test - must fail. That is the regression this finding exists to prevent. -- **F2** — the successful matched-home path reaches `BEGIN IMMEDIATE`, and a home - changed **during acquisition retry** is caught. - *Red when:* the fresh adjacent ambient read is replaced by the value captured - before the retry sleeps — the environment-change-during-retry test must fail. A - test that only exercises mismatched-home refusal is insufficient: it passes even - if the matched path never acquires anything. + still succeeds on that same routed home. Two claims, two mutations. + *Red when:* (a) the catalog commit is moved under N — the routed-home catalog test + must fail; (b) the typed refusal is replaced by a thrown exception — the refusal + test must fail. (a) alone would let a throwing implementation pass. +- **F2** — the successful matched-home path reaches `BEGIN IMMEDIATE`; a home changed + **during acquisition retry** is caught; and a home changed by **another process** + retargeting the selector symlink is caught. + *Red when:* (a) the fresh adjacent ambient read is replaced by the value captured + before the retry sleeps — the environment-change-during-retry test must fail; + (b) `classifyNativeRoutedResidue` is restored to resolving the home itself — the + second-process symlink-retarget test must fail. (a) alone is satisfied by the + adjacent-read-only implementation that the executed probe above defeats. - **F3** — same-process reentrancy returns `refused/reentrant`, distinguishable from the `busy` SQLite alone produces. *Red when:* ALS is removed — the reason must degrade to `busy`. Asserting only @@ -933,11 +964,21 @@ and the ones that did were rewritten rather than kept. classification test must fail. - `transition-state.ts` alone owns native generation/txId/history scheduling; JSON owns none of them, and the lock never opens a second coordinator connection in C. + *Red when:* (a) a second coordinator connection is opened inside C — it must + self-contend and fail, not silently succeed; (b) `nativeGeneration` or `currentTxId` + is written into `integrations/codex.json` — the JSON-ownership test must fail. - Lock edges are N->C and short fail-fast H->N only; stale history jobs are rejected by generation/transaction identity without any C->N, C->H, or held N->H edge. + *Red when:* (a) each inverse edge is added in turn (C->N, C->H, held N->H) — the + dependency-graph test must fail for each; (b) the generation/txId stale-job + rejection is removed — a stale job must be observed overwriting the winner. - **Phase honesty** — the goalplan records the merge: `wp11` closed as *merged*, and `wp12` carrying the mechanism, the admission producer, the `inject.ts` split, and the first call edge as required tasks. If the caller does not land, the PR body says the lock is unused, in those words. - *Red when:* the WP12 admission producer is removed — the production-entry test or - the compile must fail. + *Red when:* (a) the WP12 admission producer is removed — the production-entry test + or the compile must fail; (b) the goalplan merge state is reverted so a standalone + WP11 could be declared done; (c) the PR body claims a live substrate while the + caller is absent. (b) and (c) are checked by reading them, not by a test — and + saying so is the point, because a criterion that pretends to be automated when it + is not is worse than one that admits it. diff --git a/devlog/_plan/260804_codex_write_substrate/040_ownership_convergence.md b/devlog/_plan/260804_codex_write_substrate/040_ownership_convergence.md index 27534fd72..7be4fc0f8 100644 --- a/devlog/_plan/260804_codex_write_substrate/040_ownership_convergence.md +++ b/devlog/_plan/260804_codex_write_substrate/040_ownership_convergence.md @@ -57,7 +57,7 @@ IN: | `src/codex/convergence-types.ts` | IMPORT ONLY | Consume `AdmissionSnapshot`, `CodexObservedState`, `ConvergeOutcome`, `CodexProvenanceLedger`, and section types; no WP12 union. | | `src/codex/integration-record.ts` | USE/MODIFY THROUGH OWNER API | Read/update provenance and extension keys through the contract owner; no transition pair, history state/schedule, path/schema/parser, or parallel merge here. | | `src/codex/transition-state.ts` | CONSUME | Use `readCodexTransitionState`, `beginCodexTransition`, and `updateCodexHistoryTransition` for the canonical-CODEX_HOME pair and history state/schedule. | -| `src/codex/codex-write-lock.ts` | CONSUME | Correct WP11 module name; no lock redesign. | +| `src/codex/codex-write-lock.ts` | **NEW / IMPLEMENT** | This phase CREATES the module; `030_lock_protocol.md` is its design specification. It was WP11's until round 7 merged the phases. | | `src/codex/journal.ts` | MODIFY | Read-only typed inspection; authorized recovery only inside convergence. | | `src/codex/inject.ts` | MODIFY | Receipt-gated internal apply/restore mechanics; remove filename-based deletion authority. | | `src/codex/sync.ts` | MODIFY | Remove the remaining alternate native orchestration; delegate to `convergeCodex`. | From 1f83564a828acdbc4e246a87db8dc872cd742501 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Wed, 5 Aug 2026 08:38:05 +0900 Subject: [PATCH 102/163] fix(acl): dev:ino is not identity on ext4, where an unlinked inode comes straight back MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The identity fix shipped in fa610fdd8 was verified on macOS, which reused an inode in 0 of 200 unlink/recreate cycles. Linux CI failed it. Probed there: {"inodeReused": 100, "ctimeIdentical": 0, "of": 100} ext4 hands the inode of an unlinked file straight back, so dev:ino cannot tell a replacement from the original — and the memo would have kept crediting the previous file's ACLs on the platform most of CI runs on. The fix for an absence-as-guarantee defect contained the same defect one layer down: I treated 'the identity did not change' as proof the file did not, on the strength of one operating system. Identity is now dev:ino:ctimeNs. ctime differed in 100 of 100 of those cycles, and it also moves when permissions change underneath us, so such a file re-hardens instead of being trusted. Hardening is idempotent, so erring toward re-running is the safe direction. The test's own guard had the same flaw — it compared inodes — so it asserted something false on Linux rather than skipping. It now compares the same identity string the memo records. --- src/lib/windows-secret-acl.ts | 20 ++++++++++++++++---- tests/windows-secret-acl.test.ts | 15 ++++++++++++--- 2 files changed, 28 insertions(+), 7 deletions(-) diff --git a/src/lib/windows-secret-acl.ts b/src/lib/windows-secret-acl.ts index 3b590e738..60adb7b75 100644 --- a/src/lib/windows-secret-acl.ts +++ b/src/lib/windows-secret-acl.ts @@ -49,15 +49,27 @@ type HardenedIdentity = string | null; /** * Portable file identity for the ACL success memo. * - * `bigint: true` is what makes this work on Windows: the default `ino` is 0 on - * NTFS, while the bigint variant carries the file index. A zero or failed read - * yields `null`, and a null memo entry never satisfies a later lookup. + * `dev:ino` alone is NOT identity, and believing it was is what a Linux CI run + * caught: ext4 reuses the inode of an unlinked file immediately, so 100 of 100 + * unlink/recreate cycles produced the SAME `ino` — while macOS reused none in + * 200 cycles and happily reported the fix working. A same-name replacement would + * therefore have kept inheriting the previous file's ACLs on the platform most + * of CI runs on. + * + * `ctimeNs` is what actually distinguishes them: it differed in 100 of 100 of + * those same cycles. It also moves on any metadata change, so a file whose + * permissions were altered underneath us re-hardens rather than being trusted — + * conservative in the safe direction, since hardening is idempotent. + * + * `bigint: true` serves both: `ctimeNs` exists only in the bigint variant, and + * the plain `ino` is 0 on NTFS. A zero-ino or failed read yields `null`, and a + * null memo entry never satisfies a later lookup. */ function fileIdentity(targetPath: string): HardenedIdentity { try { const stats = statSync(targetPath, { bigint: true }); if (stats.ino === 0n) return null; - return `${stats.dev}:${stats.ino}`; + return `${stats.dev}:${stats.ino}:${stats.ctimeNs}`; } catch { return null; } diff --git a/tests/windows-secret-acl.test.ts b/tests/windows-secret-acl.test.ts index d00ea9598..0073c74b6 100644 --- a/tests/windows-secret-acl.test.ts +++ b/tests/windows-secret-acl.test.ts @@ -714,7 +714,15 @@ describe("stable-path harden memo is bound to file identity, not pathname", () = resetHardenedStateForTests(); const stable = join(testDir, "coordinator.sqlite"); writeFileSync(stable, "first", "utf8"); - const firstIdentity = statSync(stable).ino; + // Identity here must match what the memo records, NOT just the inode. + // ext4 reuses the inode of an unlinked file immediately — 100 of 100 cycles + // on Linux CI — so an inode-only guard asserts something false and this test + // failed there while passing on macOS. + const identityOf = (path: string): string => { + const s = statSync(path, { bigint: true }); + return `${s.dev}:${s.ino}:${s.ctimeNs}`; + }; + const firstIdentity = identityOf(stable); setPlatformForTests("win32"); const previousUsername = process.env.USERNAME; @@ -731,8 +739,9 @@ describe("stable-path harden memo is bound to file identity, not pathname", () = // Release, then replace the file at the SAME pathname. unlinkSync(stable); writeFileSync(stable, "second", "utf8"); - const secondIdentity = statSync(stable).ino; - // Guard the guard: if the filesystem reused the inode this proves nothing. + const secondIdentity = identityOf(stable); + // Guard the guard: if the filesystem handed back an identical identity, + // the replacement is indistinguishable and this test proves nothing. expect(secondIdentity).not.toBe(firstIdentity); // Reacquire. The replacement has never been hardened. From d628e4b6902bb3122da75354def264309876e3b2 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Wed, 5 Aug 2026 08:41:33 +0900 Subject: [PATCH 103/163] fix(acl): reading the path after icacls answers the wrong question MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 'What is at this path now' is not 'what did icacls operate on'. Swap the file during the final /remove:g — reachable through the production runner seam — and the post-call read cached the REPLACEMENT as hardened, so the next acquisition skipped ACL work on a file that had never seen it: {identityChangedDuringHarden: true, callsForOriginal: 3, totalCalls: 3, replacementWasHardened: false} Identity is now captured before the sequence and compared after it. Mismatch, or an unreadable identity at either end, means the harden cannot be attributed: no memo is written, any existing one is dropped, required callers fail closed and optional callers soft-fail with that diagnostic. A substitution is not a transient stall, so it does not consume the timeout retry. Observed absence also retires the memo now. A success entry that outlives its file lets any later file with a matching identity satisfy the cache. I could not reproduce identity recycling on APFS in 200k recreations, which is a non-observation rather than a guarantee — ext4 already showed what that intuition is worth. Two more broken-change checks, restored clean: dropping the before/after comparison fails the during-hardening test; keeping the memo across observed absence fails the absence test. Neither disturbs the other two. Also marked the NTFS bigint-ino claim UNVERIFIED in the source rather than stating it as fact. Darwin reports identical values for both stat forms; no pinned-Bun Windows probe has been run, and the zero-ino guard stands as defensive code, not as evidence. --- src/lib/windows-secret-acl.ts | 71 ++++++++++++++++++++++++--- tests/windows-secret-acl.test.ts | 84 +++++++++++++++++++++++--------- 2 files changed, 126 insertions(+), 29 deletions(-) diff --git a/src/lib/windows-secret-acl.ts b/src/lib/windows-secret-acl.ts index 60adb7b75..c6ebd64e8 100644 --- a/src/lib/windows-secret-acl.ts +++ b/src/lib/windows-secret-acl.ts @@ -61,9 +61,14 @@ type HardenedIdentity = string | null; * permissions were altered underneath us re-hardens rather than being trusted — * conservative in the safe direction, since hardening is idempotent. * - * `bigint: true` serves both: `ctimeNs` exists only in the bigint variant, and - * the plain `ino` is 0 on NTFS. A zero-ino or failed read yields `null`, and a - * null memo entry never satisfies a later lookup. + * `bigint: true` is used because `ctimeNs` exists only in that variant. + * + * UNVERIFIED: it is widely reported that the plain `ino` is 0 on NTFS while the + * bigint form carries the file index, and the zero-ino guard below exists for + * that case. Neither this machine (Darwin) nor Linux CI can confirm it, and a + * pinned-Bun Windows probe has not been run. Treat the guard as defensive rather + * than as a demonstrated platform fact — the ext4 inode-reuse finding above is + * exactly what asserting an unprobed filesystem property costs. */ function fileIdentity(targetPath: string): HardenedIdentity { try { @@ -96,6 +101,44 @@ function memoSatisfied(cache: Map, targetPath: string) return current === remembered; } +/** + * Record a harden ONLY if the file we hardened is still the file at that path. + * + * Reading identity after the ACL sequence returns answers "what is there now", + * which is not the same question as "what did icacls operate on". A replacement + * landing mid-sequence — probed by swapping the file during the final + * `/remove:g` — made the memo remember the REPLACEMENT as hardened, so the next + * acquisition skipped ACL work on a file that had never seen it: + * + * {identityChangedDuringHarden: true, callsForOriginal: 3, totalCalls: 3, + * replacementWasHardened: false} + * + * So identity is captured before the sequence and compared after it. A mismatch, + * or an unreadable identity at either end, means we cannot say what was hardened: + * the memo is cleared rather than written, and required callers fail closed. An + * optional caller soft-fails, matching how it treats every other unproven ACL. + * + * Returns true when the harden may be reported successful. + */ +function recordHarden( + cache: Map, + targetPath: string, + before: HardenedIdentity, +): boolean { + const after = fileIdentity(targetPath); + if (before === null || after === null || before !== after) { + // Never leave a memo behind for a file we cannot vouch for, including one + // written by an earlier successful harden of a now-replaced file. + cache.delete(targetPath); + return false; + } + cache.set(targetPath, after); + return true; +} + +const SUBSTITUTED_DIAGNOSTIC = + "ACL hardening could not be attributed — the file at this path changed during hardening"; + export interface HardenResult { ok: boolean; diagnostics?: string; @@ -472,7 +515,9 @@ function hardenEntry( opts: HardenOptions, cache: Map, ): HardenResult { - if (!existsSync(targetPath)) return { ok: true }; + // Observed absence retires the memo. Leaving it would let a later file at this + // path satisfy the cache if the filesystem ever hands back a matching identity. + if (!existsSync(targetPath)) { cache.delete(targetPath); return { ok: true }; } if (effectivePlatform() !== "win32") return { ok: true }; if (memoSatisfied(cache, targetPath)) return { ok: true }; const memoKey = timeoutMemoKey(targetPath, opts); @@ -487,10 +532,17 @@ function hardenEntry( for (let attempt = 0; attempt < 2; attempt++) { if (attempt > 0 && deadline - nowFn() <= 0) break; // retry only while budget remains try { + // Captured BEFORE the sequence: this is the file we are about to harden. + const before = fileIdentity(targetPath); runIcacls(targetPath, directory, deadline); - cache.set(targetPath, fileIdentity(targetPath)); + if (!recordHarden(cache, targetPath, before)) { + if (opts.required) throw new Error(SUBSTITUTED_DIAGNOSTIC); + return { ok: false, diagnostics: SUBSTITUTED_DIAGNOSTIC }; + } return { ok: true }; } catch (err) { + // A substitution is not a transient icacls stall; do not spend the retry on it. + if (err instanceof Error && err.message === SUBSTITUTED_DIAGNOSTIC) throw err; lastErr = err; if (!isTimeoutError(err)) break; // real failures do not retry } @@ -516,7 +568,7 @@ async function hardenEntryAsync( opts: HardenOptions, cache: Map, ): Promise { - if (!existsSync(targetPath)) return { ok: true }; + if (!existsSync(targetPath)) { cache.delete(targetPath); return { ok: true }; } if (effectivePlatform() !== "win32") return { ok: true }; if (memoSatisfied(cache, targetPath)) return { ok: true }; const memoKey = timeoutMemoKey(targetPath, opts); @@ -531,10 +583,15 @@ async function hardenEntryAsync( for (let attempt = 0; attempt < 2; attempt++) { if (attempt > 0 && deadline - nowFn() <= 0) break; try { + const before = fileIdentity(targetPath); await runIcaclsAsync(targetPath, directory, deadline); - cache.set(targetPath, fileIdentity(targetPath)); + if (!recordHarden(cache, targetPath, before)) { + if (opts.required) throw new Error(SUBSTITUTED_DIAGNOSTIC); + return { ok: false, diagnostics: SUBSTITUTED_DIAGNOSTIC }; + } return { ok: true }; } catch (err) { + if (err instanceof Error && err.message === SUBSTITUTED_DIAGNOSTIC) throw err; lastErr = err; if (!isTimeoutError(err)) break; } diff --git a/tests/windows-secret-acl.test.ts b/tests/windows-secret-acl.test.ts index 0073c74b6..123919019 100644 --- a/tests/windows-secret-acl.test.ts +++ b/tests/windows-secret-acl.test.ts @@ -756,46 +756,86 @@ describe("stable-path harden memo is bound to file identity, not pathname", () = }); /** - * The other half of the memo, which my first attempt at this fix missed. + * The hole the identity memo did NOT close on its own, and the reason the memo + * is written from a before/after comparison rather than a post-call read. * - * When identity cannot be established the memo stores `null`. Windows NTFS is - * exactly where that arises — the non-bigint `ino` is 0 there, which is why the - * read uses `bigint: true` and still has to tolerate failure. + * Reading identity after icacls returns answers "what is at this path now", + * which is a different question from "what did icacls operate on". Swap the + * file during the final `/remove:g` — a real race, reachable through the + * production runner seam — and the post-call read records the REPLACEMENT as + * hardened. The next acquisition then skips ACL work on a file that has never + * seen it: * - * A `null` entry must NOT satisfy a later lookup. Treating "we could not tell" - * as "it did not change" is the same absence-as-guarantee move that produced the - * pathname memo in the first place, and it would reintroduce the defect on the - * one platform this code exists for. + * {identityChangedDuringHarden: true, callsForOriginal: 3, totalCalls: 3, + * replacementWasHardened: false} * - * Driven red: mutating `remembered === null` to return true makes this fail - * while the replacement test above still passes — which is why both exist. + * A required caller must fail closed here. "We hardened something, and + * something is at that path" is not attribution. */ - test("an unverifiable identity is re-hardened rather than assumed unchanged", () => { + test("a file replaced DURING hardening is not credited, and required fails closed", () => { resetHardenedStateForTests(); - const stable = join(testDir, "unverifiable.sqlite"); - writeFileSync(stable, "first", "utf8"); + const stable = join(testDir, "raced.sqlite"); + writeFileSync(stable, "original", "utf8"); setPlatformForTests("win32"); const previousUsername = process.env.USERNAME; process.env.USERNAME = "ocx-test-user"; let grants = 0; - // The file is removed DURING the harden, so the post-icacls identity read - // fails and the memo records `null`. That is a real race — a replacement - // landing between the ACL call and the memo write — not a contrived seam. setIcaclsRunnerForTests(args => { if (args.includes("/grant:r")) grants += 1; - if (grants === 1 && existsSync(stable)) unlinkSync(stable); + // Replace the file before the sequence returns. + if (args.includes("/remove:g")) { + unlinkSync(stable); + writeFileSync(stable, "replacement", "utf8"); + } return { success: true, exitCode: 0, timedOut: false, stdout: "" }; }); try { - expect(hardenSecretPath(stable, { required: true })).toEqual({ ok: true }); + expect(() => hardenSecretPath(stable, { required: true })).toThrow( + /changed during hardening/, + ); expect(grants).toBe(1); + // No memo was left behind for the replacement, so a later harden runs. + expect(hardenedSecretPathCountForTests()).toBe(0); - // Recreate at the same name. The memo holds `null` for this path, so the - // harden must run again rather than credit the vanished file's ACLs. - writeFileSync(stable, "second", "utf8"); + // An optional caller soft-fails with the same honest diagnostic. + const optional = hardenSecretPath(stable, { required: false }); + expect(optional.ok).toBe(false); + expect(optional.diagnostics).toMatch(/changed during hardening/); + } finally { + if (previousUsername === undefined) delete process.env.USERNAME; + else process.env.USERNAME = previousUsername; + setIcaclsRunnerForTests(null); + setPlatformForTests(null); + } + }); + + /** + * Observed absence retires the memo. + * + * Otherwise a success entry outlives the file it describes, and any later file + * whose identity happens to match satisfies the cache. I could not reproduce + * identity recycling on APFS in 200k recreations — which is a non-observation, + * not a guarantee, and ext4 already proved that intuition about one filesystem + * does not transfer. + */ + test("observing the path absent retires its success memo", () => { + resetHardenedStateForTests(); + const stable = join(testDir, "vanishes.sqlite"); + writeFileSync(stable, "first", "utf8"); + + setPlatformForTests("win32"); + const previousUsername = process.env.USERNAME; + process.env.USERNAME = "ocx-test-user"; + setIcaclsRunnerForTests(() => ({ success: true, exitCode: 0, timedOut: false, stdout: "" })); + try { expect(hardenSecretPath(stable, { required: true })).toEqual({ ok: true }); - expect(grants).toBe(2); + expect(hardenedSecretPathCountForTests()).toBe(1); + + unlinkSync(stable); + // A harden of the now-absent path is a no-op that must also forget it. + expect(hardenSecretPath(stable, { required: true })).toEqual({ ok: true }); + expect(hardenedSecretPathCountForTests()).toBe(0); } finally { if (previousUsername === undefined) delete process.env.USERNAME; else process.env.USERNAME = previousUsername; From 52791556493a73b396410273cdbf6cf50710f083 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Wed, 5 Aug 2026 08:42:47 +0900 Subject: [PATCH 104/163] docs(codex): label the round-6 findings as history, and strike the 'N bounds it' argument MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The findings section still read as instructions. It was written as a standalone WP11 and its remedies were partly defeated by later rounds, so a reader following it top-down would have implemented the rejected design — which is the third time this document has done that to someone. It now opens by saying it is a record of how each finding was reached, with the prescriptive decisions living below. The findings keep their original shape on purpose: a remedy that turned out wrong is more useful with the reasoning attached than deleted. Three specific retractions in place, not appended: - 'making the guard home-parameterized is WP12's job' — wrong on its own terms once the symlink probe landed; it is required work in this phase. - 'the residue guard's read is covered because it is bounded by N' — struck. N serializes a database, not process.env, and is not held when that read happens. A second process does not need our environment anyway; it can retarget the symlink our CODEX_HOME names. - the remaining prescriptive 'WP11 does X' sentences now say 'the lock'. --- .../030_lock_protocol.md | 51 ++++++++++++------- 1 file changed, 34 insertions(+), 17 deletions(-) diff --git a/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md b/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md index f2c761feb..b8e98ef3d 100644 --- a/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md +++ b/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md @@ -33,6 +33,14 @@ All current-code citations and diff context below were rechecked on 2026-08-04 a Rechecked on 2026-08-05 at `86e5d677b`, with executed probes rather than citation. All three change what B must build. +> **Read this section as a HISTORICAL RECORD of how each finding was reached, not +> as instructions.** It was written while this was still a standalone WP11, so its +> sentences say "WP11 does X" and some of them propose remedies that later rounds +> then defeated — the F2 adjacency rule most of all. Every prescriptive decision +> lives below, in the merged-phase sections; where the two disagree, those win. +> The findings are kept in their original shape deliberately: a remedy that was +> wrong is more useful with the reasoning that produced it attached than deleted. + ### F1 — the coordinator refuses to open on every routed install (blocking) `openCodexCoordinatorTransaction` initializes a missing row only after @@ -57,11 +65,11 @@ path that `005_contract.md:705-800` designed for this is **not implemented** — `withCodexCompatibilityNativeHandoff` and `adoption-pending` have zero occurrences in `src/`. -WP11 therefore ships the lock **without rewiring `convergence.ts` to require N**. -`commitCodexCatalogCandidate` keeps `K -> C` (`src/codex/convergence.ts:393-406`), -which is already correct and already cross-process safe. `convergence.ts` moves -under N in WP12, together with the admission pipeline and the adoption path that -makes opening N legal on a routed home. Landing the lock and its rewiring in one +The phase therefore ships the lock **without rewiring the existing catalog commit +to require N**. `commitCodexCatalogCandidate` keeps `K -> C` +(`src/codex/convergence.ts:393-406`), which is already correct and already +cross-process safe. That seam moves under N only once the adoption path makes +opening N legal on a routed home. Landing the lock and its rewiring in one phase would mean landing a refusal for the current user base to satisfy a document. That is not a scope dodge: WP11's own accept criteria (C5/C6/C7/C18) are about @@ -174,12 +182,16 @@ The guard passed by inspecting a directory that is not the one being locked. Eve existing caller happens to pass the ambient home, so the defect is latent today and becomes live the moment WP11 accepts an explicit `codexHome` — which its API does. -Consequence for WP11: `CodexWriteLockOptions.codexHome` may not be forwarded to a -coordinator whose safety guard reads a different home. WP11 refuses with +Consequence: `CodexWriteLockOptions.codexHome` may not be forwarded to a +coordinator whose safety guard reads a different home. The lock refuses with `authority_not_proven` when the canonical target home is not identical to the -ambient `getCodexHome()` result, and a test drives the mismatch. Making the -guard home-parameterized is WP12's job (it owns admission); WP11 must not silently -accept a home whose residue was never checked. +ambient `getCodexHome()` result, and a test drives the mismatch. + +> The sentence that stood here — "making the guard home-parameterized is WP12's +> job" — was written when WP11 and WP12 were separate phases, and it was wrong on +> its own terms even then: round 8's symlink probe showed the comparison alone +> cannot close this. Parameterizing the guard is required work in THIS phase. See +> the acquisition section. **The obvious version of that remedy is itself a TOCTOU**, and it was caught by probing rather than by reading. `getCodexHome()` re-resolves `process.env.CODEX_HOME` @@ -194,10 +206,15 @@ A comparison that calls `getCodexHome()` once to validate and lets the coordinat call it again to check residue proves nothing: the second read is a fresh read. So the check is not "compare the two", it is **resolve the ambient home exactly once, canonicalize it, use that single value for both the comparison and the lock target, -and refuse if the caller supplied anything else**. WP11 never re-reads the ambient -home after that point, and the residue guard's own later read is covered only -because it is bounded by N — a second process that changes the environment cannot -change ours, and our own code does not mutate `CODEX_HOME` mid-operation. +and refuse if the caller supplied anything else**. + +> ~~And the residue guard's own later read is covered because it is bounded by N — a +> second process that changes the environment cannot change ours.~~ **Struck.** N +> serializes the coordinator database, not `process.env`, and it is not even held +> when that read happens. Round 8 then showed the deeper error: a second process +> does not need our environment at all, because it can retarget the symlink that +> our `CODEX_HOME` names. The mechanism is the parameterized guard, not this +> argument. That last clause is a claim, not an assumption, so it needs a guard rather than a comment: the B phase adds a test that fails if any production module under `src/` @@ -523,7 +540,7 @@ assert stable lock path; COMMIT N; close DB + side fd This replaces the former two generic admission callbacks. The first `AdmissionSnapshot` is enough to refuse before namespace creation. The second is -an authoritative re-read inside the coordinated commit; WP11 does not reduce it to +an authoritative re-read inside the coordinated commit; the lock does not reduce it to a boolean or manufacture an authority receipt. `withConfigMutationLockSync` is already synchronous, fail-fast, and reentrant only @@ -590,7 +607,7 @@ import { Call `resolveEffectiveUserIdentity()`, then pass that identity and the canonical `CODEX_HOME` to `resolveCodexCoordinatorDatabasePath(...)`. Its return value is the -**final database path** and is consumed verbatim. WP11 does not import +**final database path** and is consumed verbatim. The lock does not import `resolveOsRuntimeDirectory`, encode uid/SID, hash the home for path construction, or append `opencodex`, `native-write-locks`, a version, or `.sqlite`. The prior version reconstructed those segments locally; that was wrong because it let the @@ -605,7 +622,7 @@ Walk components one at a time; never recursive-mkdir across an unvalidated paren junctions, or reparse redirects. `ENOENT` permits one `mkdirSync(..., 0700)`, followed by the same validation. - POSIX requires exact effective uid and mode `0700` for directories, `0600` for - the DB/rollback journal. Wrong owner/mode refuses; WP11 does not chmod a suspect + the DB/rollback journal. Wrong owner/mode refuses; the lock does not chmod a suspect existing path. - Windows validates non-junction identity and runs the existing required per-user ACL owner within the remaining outer deadline From 296050e3a62b040508bb44e9dbf21af0d67d7bc3 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Wed, 5 Aug 2026 08:57:55 +0900 Subject: [PATCH 105/163] fix(acl): my fix demanded that icacls not change permissions MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A reviewer defeated the previous commit twice, and the second one would have broken Windows outright. The identity was dev:ino:ctimeNs used for two different jobs, and the before/after check required all of it to be unchanged across the ACL call. But ctime moves when permissions change — probed, {ctimeChangedByChmod: true} — and changing permissions is what icacls does. Every first harden on Windows would have thrown 'changed during hardening' on its own successful work. A check that forbids an operation from doing the thing it exists to do is not strict. Split into object (dev:ino — is this the same file, survives an ACL edit) and freshness (ctimeNs — has the metadata moved, catches ext4 handing an inode straight back). The before/after comparison uses object alone; the memo stores object plus the freshness read after hardening. The other defeat was my evidence, not my code. I reported four broken-change checks; two of them did nothing. The null-identity branch had become dead code inside the very fix that was supposed to need it, and removing dev entirely left all forty tests green — because the tests built the expected identity the same way the implementation did. Mirroring an implementation in test setup is not coverage of it. There is now a stat seam, and seven mutations each redden at least one test: dev dropped, freshness dropped, no before/after comparison, absence keeping the memo, an unreadable observation satisfying it, a zero inode accepted as identity, and the full-identity comparison that started this. All restored, 46 pass / 0 fail. Also marked the WP4 plan's proposed per-home lock SUPERSEDED. It predates user-identity.ts landing and would key on sha256(home) under tmpdir(), omitting the uid/SID — the exact split the contract exists to prevent. WP4 consumes N. --- .../030_desired_state.md | 25 +++ src/lib/windows-secret-acl.ts | 123 +++++++---- tests/windows-secret-acl.test.ts | 194 ++++++++++++++++++ 3 files changed, 304 insertions(+), 38 deletions(-) diff --git a/devlog/_plan/260803_codex_desktop_toggle/030_desired_state.md b/devlog/_plan/260803_codex_desktop_toggle/030_desired_state.md index 977157f8c..a49c9b85a 100644 --- a/devlog/_plan/260803_codex_desktop_toggle/030_desired_state.md +++ b/devlog/_plan/260803_codex_desktop_toggle/030_desired_state.md @@ -247,6 +247,31 @@ real migration after OFF and ON mutations and asserts: no `claudeCode` block, ## One per-home linearization lock — a check is not a lock +> **SUPERSEDED (2026-08-05): do not build this lock. Consume WP12's N instead.** +> +> This section was written on 2026-08-04, before the write-substrate unit landed +> `src/codex/user-identity.ts` (`554b3919e`, 2026-08-05). Its diagnosis is right and +> its remedy is now a duplicate — a second, weaker per-home lock beside the one the +> other unit already owns: +> +> | This section proposes | What already exists | +> |---|---| +> | `withCodexHomeLinearizationLockSync` in a NEW `desired-state.ts` | the N coordinator transaction, `src/codex/transition-state.ts:348` | +> | `join(tmpdir(), "opencodex-native-locks", sha256(home) + ".sqlite")` | `resolveCodexCoordinatorDatabasePath`, `src/codex/user-identity.ts:165`, which keys on the effective **uid/SID** as well as the canonical home | +> | `inspectNativeCodexOwnership()` here | `AdmissionSnapshot.ownership`, owned by WP12's admission producer | +> +> Hashing the home ALONE is the specific defect `005_contract.md` §7 exists to +> prevent: a service and a CLI running as different OS users would share one lock +> file for one home, and `os.homedir()` is environment-controlled under Bun 1.3.14 +> either way. Building this would ship that bug knowingly. +> +> **Consequence for sequencing:** WP4 depends on WP12's lock, so it runs after it, +> not beside it. WP4 keeps everything below that is genuinely its own — the +> persisted flag, the gate at `src/cli/index.ts:319`, OFF reconciliation, and the +> startup ownership order — and takes its linearization from N. The analysis below +> is retained because it is *why* a lock is required at all; only the "NEW +> `src/codex/desired-state.ts` owns one linearization boundary" answer is withdrawn. + The previous design's bare re-read was insufficient. `syncModelsToCodex` can pause in provider model gathering (`src/codex/sync.ts:83-108`), and even a re-read after that pause leaves a check/write gap: another process can commit OFF through the diff --git a/src/lib/windows-secret-acl.ts b/src/lib/windows-secret-acl.ts index c6ebd64e8..4a237405f 100644 --- a/src/lib/windows-secret-acl.ts +++ b/src/lib/windows-secret-acl.ts @@ -44,42 +44,80 @@ const timedOutPaths = new Set(); * "unchanged" — it is recorded as unverifiable so the next harden re-runs * instead of inheriting a previous file's credit. */ -type HardenedIdentity = string | null; +type HardenedIdentity = string; /** - * Portable file identity for the ACL success memo. + * What a stat can tell us about WHICH OBJECT is at a path. * - * `dev:ino` alone is NOT identity, and believing it was is what a Linux CI run - * caught: ext4 reuses the inode of an unlinked file immediately, so 100 of 100 - * unlink/recreate cycles produced the SAME `ino` — while macOS reused none in - * 200 cycles and happily reported the fix working. A same-name replacement would - * therefore have kept inheriting the previous file's ACLs on the platform most - * of CI runs on. + * Two fields, deliberately separated, because conflating them shipped a bug: * - * `ctimeNs` is what actually distinguishes them: it differed in 100 of 100 of - * those same cycles. It also moves on any metadata change, so a file whose - * permissions were altered underneath us re-hardens rather than being trusted — - * conservative in the safe direction, since hardening is idempotent. + * - `object` — `dev:ino`. Answers "is this the same file". Survives an ACL or + * permission change, which is exactly what we need across an icacls call. + * - `freshness` — `ctimeNs`. Answers "has this file's metadata moved since". It + * distinguishes an unlink/recreate that ext4 gave the same inode back for, and + * it MOVES when permissions change. + * + * The first version used `dev:ino:ctimeNs` for both jobs. Since chmod bumps ctime + * — probed, `{ctimeChangedByChmod: true}` — and icacls is a permission change, the + * before/after comparison would have rejected its own successful harden and failed + * closed on every first harden on Windows. Requiring the identity to be unchanged + * across an operation whose entire purpose is to change it is not a strict check; + * it is a broken one. + */ +interface PathObservation { + readonly object: string; + readonly freshness: string; +} + +/** + * Observe which object is at a path, and how fresh it is. + * + * `dev:ino` alone is not enough to detect a replacement, which a Linux CI run + * proved: ext4 reuses the inode of an unlinked file immediately — 100 of 100 + * unlink/recreate cycles produced the SAME `ino`, while macOS reused none in 200 + * and happily reported the earlier fix working. That is why `freshness` exists; + * `ctimeNs` differed in 100 of 100 of those same cycles. * * `bigint: true` is used because `ctimeNs` exists only in that variant. * - * UNVERIFIED: it is widely reported that the plain `ino` is 0 on NTFS while the - * bigint form carries the file index, and the zero-ino guard below exists for - * that case. Neither this machine (Darwin) nor Linux CI can confirm it, and a - * pinned-Bun Windows probe has not been run. Treat the guard as defensive rather - * than as a demonstrated platform fact — the ext4 inode-reuse finding above is - * exactly what asserting an unprobed filesystem property costs. + * Test seam: `setStatForTests` replaces this reader so a test can vary `dev`, + * `ino`, and `ctimeNs` independently. Mirroring the implementation's string + * format in test setup proves nothing about which components production uses — + * an audit removed `dev` and all forty tests still passed. + * + * UNVERIFIED: the plain `ino` is reported to be 0 on NTFS while the bigint form + * carries the file index, and the zero-ino guard exists for that case. Neither + * Darwin nor Linux CI can confirm it and no pinned-Bun Windows probe has run. + * It is defensive code, not a demonstrated platform fact. */ -function fileIdentity(targetPath: string): HardenedIdentity { +type StatReader = (path: string) => { dev: bigint; ino: bigint; ctimeNs: bigint }; + +const defaultStatReader: StatReader = path => { + const s = statSync(path, { bigint: true }); + return { dev: s.dev, ino: s.ino, ctimeNs: s.ctimeNs }; +}; + +let statReader: StatReader = defaultStatReader; + +/** Test seam: drive dev / ino / ctime independently. */ +export function setStatForTests(reader: StatReader | null): void { + statReader = reader ?? defaultStatReader; +} + +function observe(targetPath: string): PathObservation | null { try { - const stats = statSync(targetPath, { bigint: true }); - if (stats.ino === 0n) return null; - return `${stats.dev}:${stats.ino}:${stats.ctimeNs}`; + const s = statReader(targetPath); + if (s.ino === 0n) return null; + return { object: `${s.dev}:${s.ino}`, freshness: `${s.ctimeNs}` }; } catch { return null; } } +function memoValue(seen: PathObservation): HardenedIdentity { + return `${seen.object}:${seen.freshness}`; +} + /** * True only when this exact FILE was hardened, not merely this pathname. * @@ -91,14 +129,13 @@ function fileIdentity(targetPath: string): HardenedIdentity { * gone; nothing does that for a stable path. */ function memoSatisfied(cache: Map, targetPath: string): boolean { - if (!cache.has(targetPath)) return false; const remembered = cache.get(targetPath); - // Unverifiable at harden time stays unverifiable now: re-harden rather than - // treat an unknown identity as proof of an unchanged file. - if (remembered === null) return false; - const current = fileIdentity(targetPath); + if (remembered === undefined) return false; + const current = observe(targetPath); + // Unreadable now is not "unchanged": re-harden rather than trust a value we + // cannot confirm still describes what is there. if (current === null) return false; - return current === remembered; + return memoValue(current) === remembered; } /** @@ -113,26 +150,36 @@ function memoSatisfied(cache: Map, targetPath: string) * {identityChangedDuringHarden: true, callsForOriginal: 3, totalCalls: 3, * replacementWasHardened: false} * - * So identity is captured before the sequence and compared after it. A mismatch, - * or an unreadable identity at either end, means we cannot say what was hardened: - * the memo is cleared rather than written, and required callers fail closed. An - * optional caller soft-fails, matching how it treats every other unproven ACL. + * So the OBJECT is captured before the sequence and compared after it. Only the + * object — `dev:ino` — because icacls changes permissions, and `ctimeNs` moves + * when permissions change (probed: `{ctimeChangedByChmod: true}`). Comparing the + * full identity across the call would have rejected every successful harden and + * failed closed on the first harden on Windows: the check would have been + * demanding that an operation not do the thing it exists to do. + * + * The memo then stores the object plus the freshness read AFTER hardening, which + * is the state a later lookup should match. + * + * A changed object, or an unreadable observation at either end, means we cannot + * say what was hardened: the memo is cleared rather than written, and required + * callers fail closed. An optional caller soft-fails, as it does for every other + * unproven ACL. * * Returns true when the harden may be reported successful. */ function recordHarden( cache: Map, targetPath: string, - before: HardenedIdentity, + before: PathObservation | null, ): boolean { - const after = fileIdentity(targetPath); - if (before === null || after === null || before !== after) { + const after = observe(targetPath); + if (before === null || after === null || before.object !== after.object) { // Never leave a memo behind for a file we cannot vouch for, including one // written by an earlier successful harden of a now-replaced file. cache.delete(targetPath); return false; } - cache.set(targetPath, after); + cache.set(targetPath, memoValue(after)); return true; } @@ -533,7 +580,7 @@ function hardenEntry( if (attempt > 0 && deadline - nowFn() <= 0) break; // retry only while budget remains try { // Captured BEFORE the sequence: this is the file we are about to harden. - const before = fileIdentity(targetPath); + const before = observe(targetPath); runIcacls(targetPath, directory, deadline); if (!recordHarden(cache, targetPath, before)) { if (opts.required) throw new Error(SUBSTITUTED_DIAGNOSTIC); @@ -583,7 +630,7 @@ async function hardenEntryAsync( for (let attempt = 0; attempt < 2; attempt++) { if (attempt > 0 && deadline - nowFn() <= 0) break; try { - const before = fileIdentity(targetPath); + const before = observe(targetPath); await runIcaclsAsync(targetPath, directory, deadline); if (!recordHarden(cache, targetPath, before)) { if (opts.required) throw new Error(SUBSTITUTED_DIAGNOSTIC); diff --git a/tests/windows-secret-acl.test.ts b/tests/windows-secret-acl.test.ts index 123919019..9dda920b7 100644 --- a/tests/windows-secret-acl.test.ts +++ b/tests/windows-secret-acl.test.ts @@ -25,6 +25,7 @@ import { setIcaclsRunnerForTests, setNowForTests, setPlatformForTests, + setStatForTests, timedOutSecretPathCountForTests, type HardenResult, type IcaclsResult, @@ -844,3 +845,196 @@ describe("stable-path harden memo is bound to file identity, not pathname", () = } }); }); + +describe("identity components are proven individually, not mirrored from setup", () => { + /** + * An audit removed `dev` from the production identity and all forty tests + * still passed, because the existing tests build the expected identity string + * the same way the implementation does. Mirroring an implementation in test + * setup is not coverage of it. + * + * So these drive a stat seam and vary ONE component at a time. Each case fails + * if production stops consulting that component. + */ + const win32 = (body: () => T): T => { + setPlatformForTests("win32"); + const previousUsername = process.env.USERNAME; + process.env.USERNAME = "ocx-test-user"; + try { + return body(); + } finally { + if (previousUsername === undefined) delete process.env.USERNAME; + else process.env.USERNAME = previousUsername; + setIcaclsRunnerForTests(null); + setPlatformForTests(null); + setStatForTests(null); + } + }; + + const cases: { name: string; first: [bigint, bigint, bigint]; second: [bigint, bigint, bigint] }[] = [ + // The path now resolves to another device: a different object entirely. + { name: "dev", first: [1n, 10n, 100n], second: [2n, 10n, 100n] }, + // Ordinary replacement on a filesystem that does not recycle inodes. + { name: "ino", first: [1n, 10n, 100n], second: [1n, 11n, 100n] }, + // The ext4 case: same inode handed straight back, only ctime moved. + { name: "ctimeNs", first: [1n, 10n, 100n], second: [1n, 10n, 200n] }, + ]; + + for (const { name, first, second } of cases) { + test(`a change in ${name} alone forces a re-harden`, () => { + resetHardenedStateForTests(); + const stable = join(testDir, `component-${name}.sqlite`); + writeFileSync(stable, "x", "utf8"); + + win32(() => { + let grants = 0; + setIcaclsRunnerForTests(args => { + if (args.includes("/grant:r")) grants += 1; + return { success: true, exitCode: 0, timedOut: false, stdout: "" }; + }); + + let current = first; + setStatForTests(() => ({ dev: current[0], ino: current[1], ctimeNs: current[2] })); + + expect(hardenSecretPath(stable, { required: true })).toEqual({ ok: true }); + expect(grants).toBe(1); + // Same observation: the memo answers and no ACL work runs. + expect(hardenSecretPath(stable, { required: true })).toEqual({ ok: true }); + expect(grants).toBe(1); + + // One component moves. Production must notice. + current = second; + expect(hardenSecretPath(stable, { required: true })).toEqual({ ok: true }); + expect(grants).toBe(2); + }); + }); + } + + /** + * The bug the object/freshness split exists to prevent. + * + * icacls changes permissions, and a permission change moves ctime — probed + * directly: `{ctimeChangedByChmod: true}`. An implementation that requires the + * FULL identity to be unchanged across the ACL call rejects its own successful + * work, so on Windows every first harden of every path would fail closed. + * + * Here ctime moves during hardening exactly as real icacls would move it, + * while the object stays the same. That must succeed. + */ + test("ctime moving during hardening is the ACL's own doing, not a substitution", () => { + resetHardenedStateForTests(); + const stable = join(testDir, "acl-bumps-ctime.sqlite"); + writeFileSync(stable, "x", "utf8"); + + win32(() => { + let grants = 0; + let ctime = 100n; + setStatForTests(() => ({ dev: 1n, ino: 10n, ctimeNs: ctime })); + setIcaclsRunnerForTests(args => { + if (args.includes("/grant:r")) grants += 1; + // icacls edits the DACL; ctime moves. The file is the same file. + ctime += 1n; + return { success: true, exitCode: 0, timedOut: false, stdout: "" }; + }); + + expect(hardenSecretPath(stable, { required: true })).toEqual({ ok: true }); + expect(grants).toBe(1); + // And the memo recorded the POST-harden freshness, so an immediate second + // call is still a no-op rather than an endless re-harden. + expect(hardenSecretPath(stable, { required: true })).toEqual({ ok: true }); + expect(grants).toBe(1); + }); + }); +}); + +describe("an unreadable observation is not a passing memo", () => { + /** + * The memo lookup must fail when it cannot see what is at the path NOW. + * + * A mutation that turned "cannot observe" into "satisfied" survived four other + * broken-change checks, because every one of them could still observe the file. + * The condition only arises when the stat itself fails — a vanished file, a + * permission change on the parent, or NTFS returning a zero inode, which is + * precisely the platform this module exists for. + * + * Treating an unreadable observation as proof of an unchanged file is the same + * absence-as-guarantee move that produced the original pathname memo. + */ + test("a stat failure after a successful harden forces the harden to run again", () => { + resetHardenedStateForTests(); + const stable = join(testDir, "unreadable.sqlite"); + writeFileSync(stable, "x", "utf8"); + + setPlatformForTests("win32"); + const previousUsername = process.env.USERNAME; + process.env.USERNAME = "ocx-test-user"; + let grants = 0; + setIcaclsRunnerForTests(args => { + if (args.includes("/grant:r")) grants += 1; + return { success: true, exitCode: 0, timedOut: false, stdout: "" }; + }); + try { + let readable = true; + setStatForTests(() => { + if (!readable) throw Object.assign(new Error("EACCES"), { code: "EACCES" }); + return { dev: 1n, ino: 10n, ctimeNs: 100n }; + }); + + expect(hardenSecretPath(stable, { required: true })).toEqual({ ok: true }); + expect(grants).toBe(1); + + // The memo holds a value, but the path can no longer be observed. + readable = false; + // It must NOT answer from the memo. It re-runs, cannot attribute the run + // either, and a required caller therefore fails closed rather than + // reporting a harden it cannot vouch for. + expect(() => hardenSecretPath(stable, { required: true })).toThrow( + /changed during hardening/, + ); + expect(grants).toBe(2); + expect(hardenedSecretPathCountForTests()).toBe(0); + } finally { + if (previousUsername === undefined) delete process.env.USERNAME; + else process.env.USERNAME = previousUsername; + setIcaclsRunnerForTests(null); + setPlatformForTests(null); + setStatForTests(null); + } + }); + + /** + * NTFS is reported to return a zero inode from the non-bigint stat, and the + * production guard treats a zero inode as unobservable. That guard had no test: + * removing it left all forty-four green, because no case ever produced one. + */ + test("a zero inode is unobservable, not an identity", () => { + resetHardenedStateForTests(); + const stable = join(testDir, "zero-ino.sqlite"); + writeFileSync(stable, "x", "utf8"); + + setPlatformForTests("win32"); + const previousUsername = process.env.USERNAME; + process.env.USERNAME = "ocx-test-user"; + let grants = 0; + setIcaclsRunnerForTests(args => { + if (args.includes("/grant:r")) grants += 1; + return { success: true, exitCode: 0, timedOut: false, stdout: "" }; + }); + try { + setStatForTests(() => ({ dev: 1n, ino: 0n, ctimeNs: 100n })); + + // Required callers fail closed: nothing can be attributed. + expect(() => hardenSecretPath(stable, { required: true })).toThrow( + /changed during hardening/, + ); + expect(grants).toBe(1); + expect(hardenedSecretPathCountForTests()).toBe(0); + } finally { + if (previousUsername === undefined) delete process.env.USERNAME; + else process.env.USERNAME = previousUsername; + setIcaclsRunnerForTests(null); + setPlatformForTests(null); + setStatForTests(null); + } + }); +}); From cfa1f5692bfe0d26910baec3ced14812b67e23d1 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Wed, 5 Aug 2026 09:06:45 +0900 Subject: [PATCH 106/163] test(acl): the tests covered the twin, not the path production takes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A reviewer removed the async attribution entirely — the exact rejected design, post-call read straight into the memo — and all 46 tests stayed green. Every F4 test called hardenSecretPath. hardenStableLockFile, which hardens the coordinator database, calls hardenSecretPathAsync (native-main-lock-file.ts:130), as do config.ts:292 and native-profile-manager.ts:153-154. So the suite proved the half nothing uses, and the exploitable cache poisoning could have been restored without a single test noticing. The F4 suite is now parameterized over both entry points: same bodies, one runner helper installing behavior on both seams. 55 tests. Eight mutations, each in an isolated git archive copy, each reddening at least one test in BOTH parameterizations, all restored to 55 pass / 0 fail: dev dropped, freshness dropped, no before/after comparison, absence keeping the memo, unreadable-now satisfying it, zero inode accepted, full identity compared across the ACL call, and async attribution removed. Two things this cost. The replacement tests now drive identity through the stat seam rather than real unlink/recreate, because ext4 recycles an inode immediately while APFS did not once in 200 cycles — a real-file test asserts different things on different machines, which is how a broken fix passed here before. And withWin32 had to become async: a synchronous version's finally runs when the body returns its promise, before any await inside it, tearing down the stat seam mid-test so identity checks silently fall back to real inodes. --- tests/windows-secret-acl.test.ts | 551 ++++++++++++++----------------- 1 file changed, 239 insertions(+), 312 deletions(-) diff --git a/tests/windows-secret-acl.test.ts b/tests/windows-secret-acl.test.ts index 9dda920b7..6e7e255f1 100644 --- a/tests/windows-secret-acl.test.ts +++ b/tests/windows-secret-acl.test.ts @@ -695,346 +695,273 @@ describe("ephemeral ACL memo release (#840 refinement)", () => { }); }); -describe("stable-path harden memo is bound to file identity, not pathname", () => { - /** - * The memo is a Set of PATHNAMES. Ephemeral temps escape the - * consequence because atomic writers call forgetEphemeralSecretPath after the - * temp is gone (src/config.ts:214,241,309,336,480,501-510). A STABLE - * destination never does — and hardenStableLockFile - * (src/codex/native-main-lock-file.ts:127) hardens exactly such a path. - * - * So: harden a stable path, then replace the FILE at that same name. The - * replacement is a different inode that has never been through icacls, but - * the memo answers for the name and reports it hardened. - * - * This is the release -> replace -> REACQUIRE shape. Substituting the file - * while a single acquisition still holds it never consults the memo again and - * would pass with the fix removed. - */ - test("a file replaced at an already-hardened stable path is hardened again", () => { - resetHardenedStateForTests(); - const stable = join(testDir, "coordinator.sqlite"); - writeFileSync(stable, "first", "utf8"); - // Identity here must match what the memo records, NOT just the inode. - // ext4 reuses the inode of an unlinked file immediately — 100 of 100 cycles - // on Linux CI — so an inode-only guard asserts something false and this test - // failed there while passing on macOS. - const identityOf = (path: string): string => { - const s = statSync(path, { bigint: true }); - return `${s.dev}:${s.ino}:${s.ctimeNs}`; - }; - const firstIdentity = identityOf(stable); - setPlatformForTests("win32"); - const previousUsername = process.env.USERNAME; - process.env.USERNAME = "ocx-test-user"; - let grants = 0; - setIcaclsRunnerForTests(args => { - if (args.includes("/grant:r")) grants += 1; - return { success: true, exitCode: 0, timedOut: false, stdout: "" }; - }); - try { - expect(hardenSecretPath(stable, { required: true })).toEqual({ ok: true }); - expect(grants).toBe(1); - - // Release, then replace the file at the SAME pathname. - unlinkSync(stable); - writeFileSync(stable, "second", "utf8"); - const secondIdentity = identityOf(stable); - // Guard the guard: if the filesystem handed back an identical identity, - // the replacement is indistinguishable and this test proves nothing. - expect(secondIdentity).not.toBe(firstIdentity); - - // Reacquire. The replacement has never been hardened. - expect(hardenSecretPath(stable, { required: true })).toEqual({ ok: true }); - expect(grants).toBe(2); - } finally { - if (previousUsername === undefined) delete process.env.USERNAME; - else process.env.USERNAME = previousUsername; - setIcaclsRunnerForTests(null); - setPlatformForTests(null); - } - }); +/** + * F4 — the ACL success memo, run against BOTH entry points. + * + * These were written for `hardenSecretPath` alone, and an audit then removed the + * async attribution entirely: all 46 tests stayed green. That is the worse half + * to leave uncovered — `hardenStableLockFile`, which hardens the coordinator + * database, calls `hardenSecretPathAsync` + * (`src/codex/native-main-lock-file.ts:130`), as do `config.ts:292` and + * `native-profile-manager.ts:153-154`. Covering only the synchronous twin proved + * the path production does not take. + */ +type HardenFn = (path: string, opts: { required: boolean }) => Promise; - /** - * The hole the identity memo did NOT close on its own, and the reason the memo - * is written from a before/after comparison rather than a post-call read. - * - * Reading identity after icacls returns answers "what is at this path now", - * which is a different question from "what did icacls operate on". Swap the - * file during the final `/remove:g` — a real race, reachable through the - * production runner seam — and the post-call read records the REPLACEMENT as - * hardened. The next acquisition then skips ACL work on a file that has never - * seen it: - * - * {identityChangedDuringHarden: true, callsForOriginal: 3, totalCalls: 3, - * replacementWasHardened: false} - * - * A required caller must fail closed here. "We hardened something, and - * something is at that path" is not attribution. - */ - test("a file replaced DURING hardening is not credited, and required fails closed", () => { - resetHardenedStateForTests(); - const stable = join(testDir, "raced.sqlite"); - writeFileSync(stable, "original", "utf8"); +const ENTRY_POINTS: readonly { readonly label: string; readonly harden: HardenFn }[] = [ + { label: "sync", harden: async (path, opts) => hardenSecretPath(path, opts) }, + { label: "async", harden: (path, opts) => hardenSecretPathAsync(path, opts) }, +]; - setPlatformForTests("win32"); - const previousUsername = process.env.USERNAME; - process.env.USERNAME = "ocx-test-user"; - let grants = 0; - setIcaclsRunnerForTests(args => { - if (args.includes("/grant:r")) grants += 1; - // Replace the file before the sequence returns. - if (args.includes("/remove:g")) { - unlinkSync(stable); - writeFileSync(stable, "replacement", "utf8"); - } - return { success: true, exitCode: 0, timedOut: false, stdout: "" }; - }); - try { - expect(() => hardenSecretPath(stable, { required: true })).toThrow( - /changed during hardening/, - ); - expect(grants).toBe(1); - // No memo was left behind for the replacement, so a later harden runs. - expect(hardenedSecretPathCountForTests()).toBe(0); +/** + * Async on purpose. A synchronous version's `finally` runs the moment the body + * returns its promise — before a single `await` inside it — so every seam is torn + * down while the test is still running. The symptom is subtle: the stat seam + * reverts to the real filesystem and identity checks silently start comparing + * real inodes, so a test asserting a re-harden sees one that never happened. + */ +async function withWin32(body: () => Promise): Promise { + setPlatformForTests("win32"); + const previousUsername = process.env.USERNAME; + process.env.USERNAME = "ocx-test-user"; + try { + await body(); + } finally { + if (previousUsername === undefined) delete process.env.USERNAME; + else process.env.USERNAME = previousUsername; + setIcaclsRunnerForTests(null); + setAsyncIcaclsRunnerForTests(null); + setPlatformForTests(null); + setStatForTests(null); + } +} + +/** Install the same behavior on both runner seams so one body drives either path. */ +function runner(onCall: (args: string[]) => void): void { + const result = { success: true, exitCode: 0, timedOut: false, stdout: "" }; + setIcaclsRunnerForTests(args => { onCall(args); return result; }); + setAsyncIcaclsRunnerForTests(async args => { onCall(args); return result; }); +} + +for (const { label, harden } of ENTRY_POINTS) { + describe(`F4 memo attribution — ${label} entry point`, () => { + /** + * The original defect. The memo was a Set of PATH STRINGS, so unlinking a + * hardened path and recreating a different file at the same name left the + * replacement reported as hardened while it had never seen icacls. + * + * Ephemeral temps escaped this only because atomic writers call + * forgetEphemeralSecretPath once the temp is gone (`src/config.ts:214` and + * friends). Nothing does that for a stable path. + */ + test("a file replaced at an already-hardened stable path is hardened again", async () => { + resetHardenedStateForTests(); + const stable = join(testDir, `replaced-${label}.sqlite`); + writeFileSync(stable, "first", "utf8"); - // An optional caller soft-fails with the same honest diagnostic. - const optional = hardenSecretPath(stable, { required: false }); - expect(optional.ok).toBe(false); - expect(optional.diagnostics).toMatch(/changed during hardening/); - } finally { - if (previousUsername === undefined) delete process.env.USERNAME; - else process.env.USERNAME = previousUsername; - setIcaclsRunnerForTests(null); - setPlatformForTests(null); - } - }); + await withWin32(async () => { + let grants = 0; + runner(args => { if (args.includes("/grant:r")) grants += 1; }); + // Driven through the seam rather than the real filesystem: ext4 recycles + // an unlinked inode immediately (100/100 cycles) while APFS recycled none + // in 200, so a real-file version of this test asserts different things on + // different platforms — it passed on macOS with a fix broken on Linux. + let current = { dev: 1n, ino: 10n, ctimeNs: 100n }; + setStatForTests(() => current); + + expect(await harden(stable, { required: true })).toEqual({ ok: true }); + expect(grants).toBe(1); - /** - * Observed absence retires the memo. - * - * Otherwise a success entry outlives the file it describes, and any later file - * whose identity happens to match satisfies the cache. I could not reproduce - * identity recycling on APFS in 200k recreations — which is a non-observation, - * not a guarantee, and ext4 already proved that intuition about one filesystem - * does not transfer. - */ - test("observing the path absent retires its success memo", () => { - resetHardenedStateForTests(); - const stable = join(testDir, "vanishes.sqlite"); - writeFileSync(stable, "first", "utf8"); + current = { dev: 1n, ino: 11n, ctimeNs: 300n }; + expect(await harden(stable, { required: true })).toEqual({ ok: true }); + expect(grants).toBe(2); + }); + }); - setPlatformForTests("win32"); - const previousUsername = process.env.USERNAME; - process.env.USERNAME = "ocx-test-user"; - setIcaclsRunnerForTests(() => ({ success: true, exitCode: 0, timedOut: false, stdout: "" })); - try { - expect(hardenSecretPath(stable, { required: true })).toEqual({ ok: true }); - expect(hardenedSecretPathCountForTests()).toBe(1); + /** + * "What is at this path now" is not "what did icacls operate on". Swapping + * the file during the final /remove:g made a post-call read record the + * REPLACEMENT as hardened, so the next acquisition skipped ACL work on a file + * that had never seen it. + */ + test("a file replaced DURING hardening is not credited, and required fails closed", async () => { + resetHardenedStateForTests(); + const stable = join(testDir, `raced-${label}.sqlite`); + writeFileSync(stable, "original", "utf8"); - unlinkSync(stable); - // A harden of the now-absent path is a no-op that must also forget it. - expect(hardenSecretPath(stable, { required: true })).toEqual({ ok: true }); - expect(hardenedSecretPathCountForTests()).toBe(0); - } finally { - if (previousUsername === undefined) delete process.env.USERNAME; - else process.env.USERNAME = previousUsername; - setIcaclsRunnerForTests(null); - setPlatformForTests(null); - } - }); -}); + await withWin32(async () => { + let grants = 0; + let current = { dev: 1n, ino: 10n, ctimeNs: 100n }; + setStatForTests(() => current); + runner(args => { + if (args.includes("/grant:r")) grants += 1; + // Another process replaces the file before the sequence returns. + if (args.includes("/remove:g")) current = { dev: 1n, ino: 99n, ctimeNs: 500n }; + }); -describe("identity components are proven individually, not mirrored from setup", () => { - /** - * An audit removed `dev` from the production identity and all forty tests - * still passed, because the existing tests build the expected identity string - * the same way the implementation does. Mirroring an implementation in test - * setup is not coverage of it. - * - * So these drive a stat seam and vary ONE component at a time. Each case fails - * if production stops consulting that component. - */ - const win32 = (body: () => T): T => { - setPlatformForTests("win32"); - const previousUsername = process.env.USERNAME; - process.env.USERNAME = "ocx-test-user"; - try { - return body(); - } finally { - if (previousUsername === undefined) delete process.env.USERNAME; - else process.env.USERNAME = previousUsername; - setIcaclsRunnerForTests(null); - setPlatformForTests(null); - setStatForTests(null); - } - }; - - const cases: { name: string; first: [bigint, bigint, bigint]; second: [bigint, bigint, bigint] }[] = [ - // The path now resolves to another device: a different object entirely. - { name: "dev", first: [1n, 10n, 100n], second: [2n, 10n, 100n] }, - // Ordinary replacement on a filesystem that does not recycle inodes. - { name: "ino", first: [1n, 10n, 100n], second: [1n, 11n, 100n] }, - // The ext4 case: same inode handed straight back, only ctime moved. - { name: "ctimeNs", first: [1n, 10n, 100n], second: [1n, 10n, 200n] }, - ]; - - for (const { name, first, second } of cases) { - test(`a change in ${name} alone forces a re-harden`, () => { + await expect(harden(stable, { required: true })).rejects.toThrow( + /changed during hardening/, + ); + expect(grants).toBe(1); + // No memo was left for the replacement, so a later harden still runs. + expect(hardenedSecretPathCountForTests()).toBe(0); + + // An optional caller hitting the same race soft-fails with the same + // honest diagnostic. The runner keeps swapping the file, so this second + // attempt races too rather than settling on the replacement. + current = { dev: 1n, ino: 10n, ctimeNs: 100n }; + const optional = await harden(stable, { required: false }); + expect(optional.ok).toBe(false); + expect(optional.diagnostics).toMatch(/changed during hardening/); + }); + }); + + /** + * The bug the object/freshness split exists to prevent. + * + * icacls changes permissions and a permission change moves ctime — probed + * directly, {ctimeChangedByChmod: true}. Requiring the FULL identity to be + * unchanged across the ACL call rejects its own successful work, so on + * Windows every first harden of every path would have failed closed. + */ + test("ctime moving during hardening is the ACL's own doing, not a substitution", async () => { resetHardenedStateForTests(); - const stable = join(testDir, `component-${name}.sqlite`); + const stable = join(testDir, `acl-ctime-${label}.sqlite`); writeFileSync(stable, "x", "utf8"); - win32(() => { + await withWin32(async () => { let grants = 0; - setIcaclsRunnerForTests(args => { + let ctime = 100n; + setStatForTests(() => ({ dev: 1n, ino: 10n, ctimeNs: ctime })); + runner(args => { if (args.includes("/grant:r")) grants += 1; - return { success: true, exitCode: 0, timedOut: false, stdout: "" }; + ctime += 1n; // editing the DACL moves ctime; same file throughout }); - let current = first; - setStatForTests(() => ({ dev: current[0], ino: current[1], ctimeNs: current[2] })); - - expect(hardenSecretPath(stable, { required: true })).toEqual({ ok: true }); + expect(await harden(stable, { required: true })).toEqual({ ok: true }); expect(grants).toBe(1); - // Same observation: the memo answers and no ACL work runs. - expect(hardenSecretPath(stable, { required: true })).toEqual({ ok: true }); + // The memo kept the POST-harden freshness, so this is a no-op rather + // than an endless re-harden. + expect(await harden(stable, { required: true })).toEqual({ ok: true }); expect(grants).toBe(1); - - // One component moves. Production must notice. - current = second; - expect(hardenSecretPath(stable, { required: true })).toEqual({ ok: true }); - expect(grants).toBe(2); }); }); - } - /** - * The bug the object/freshness split exists to prevent. - * - * icacls changes permissions, and a permission change moves ctime — probed - * directly: `{ctimeChangedByChmod: true}`. An implementation that requires the - * FULL identity to be unchanged across the ACL call rejects its own successful - * work, so on Windows every first harden of every path would fail closed. - * - * Here ctime moves during hardening exactly as real icacls would move it, - * while the object stays the same. That must succeed. - */ - test("ctime moving during hardening is the ACL's own doing, not a substitution", () => { - resetHardenedStateForTests(); - const stable = join(testDir, "acl-bumps-ctime.sqlite"); - writeFileSync(stable, "x", "utf8"); - - win32(() => { - let grants = 0; - let ctime = 100n; - setStatForTests(() => ({ dev: 1n, ino: 10n, ctimeNs: ctime })); - setIcaclsRunnerForTests(args => { - if (args.includes("/grant:r")) grants += 1; - // icacls edits the DACL; ctime moves. The file is the same file. - ctime += 1n; - return { success: true, exitCode: 0, timedOut: false, stdout: "" }; + /** + * Each identity component is proven on its own. Earlier tests mirrored the + * implementation's identity string in their setup, which is not coverage of + * it: removing `dev` from production left all forty tests green. + */ + const components: { name: string; second: { dev: bigint; ino: bigint; ctimeNs: bigint } }[] = [ + // The path resolves to another device: a different object entirely. + { name: "dev", second: { dev: 2n, ino: 10n, ctimeNs: 100n } }, + // Ordinary replacement where the filesystem does not recycle inodes. + { name: "ino", second: { dev: 1n, ino: 11n, ctimeNs: 100n } }, + // The ext4 case: the same inode handed straight back, only ctime moved. + { name: "ctimeNs", second: { dev: 1n, ino: 10n, ctimeNs: 200n } }, + ]; + + for (const { name, second } of components) { + test(`a change in ${name} alone forces a re-harden`, async () => { + resetHardenedStateForTests(); + const stable = join(testDir, `component-${name}-${label}.sqlite`); + writeFileSync(stable, "x", "utf8"); + + await withWin32(async () => { + let grants = 0; + runner(args => { if (args.includes("/grant:r")) grants += 1; }); + let current = { dev: 1n, ino: 10n, ctimeNs: 100n }; + setStatForTests(() => current); + + expect(await harden(stable, { required: true })).toEqual({ ok: true }); + expect(grants).toBe(1); + expect(await harden(stable, { required: true })).toEqual({ ok: true }); + expect(grants).toBe(1); + + current = second; + expect(await harden(stable, { required: true })).toEqual({ ok: true }); + expect(grants).toBe(2); + }); }); + } - expect(hardenSecretPath(stable, { required: true })).toEqual({ ok: true }); - expect(grants).toBe(1); - // And the memo recorded the POST-harden freshness, so an immediate second - // call is still a no-op rather than an endless re-harden. - expect(hardenSecretPath(stable, { required: true })).toEqual({ ok: true }); - expect(grants).toBe(1); - }); - }); -}); + /** + * A memo lookup that cannot see what is at the path NOW must not answer. + * A mutation turning "cannot observe" into "satisfied" survived every other + * broken-change check, because all of them could still observe the file. + */ + test("a stat failure after a successful harden forces the harden to run again", async () => { + resetHardenedStateForTests(); + const stable = join(testDir, `unreadable-${label}.sqlite`); + writeFileSync(stable, "x", "utf8"); -describe("an unreadable observation is not a passing memo", () => { - /** - * The memo lookup must fail when it cannot see what is at the path NOW. - * - * A mutation that turned "cannot observe" into "satisfied" survived four other - * broken-change checks, because every one of them could still observe the file. - * The condition only arises when the stat itself fails — a vanished file, a - * permission change on the parent, or NTFS returning a zero inode, which is - * precisely the platform this module exists for. - * - * Treating an unreadable observation as proof of an unchanged file is the same - * absence-as-guarantee move that produced the original pathname memo. - */ - test("a stat failure after a successful harden forces the harden to run again", () => { - resetHardenedStateForTests(); - const stable = join(testDir, "unreadable.sqlite"); - writeFileSync(stable, "x", "utf8"); + await withWin32(async () => { + let grants = 0; + runner(args => { if (args.includes("/grant:r")) grants += 1; }); + let readable = true; + setStatForTests(() => { + if (!readable) throw Object.assign(new Error("EACCES"), { code: "EACCES" }); + return { dev: 1n, ino: 10n, ctimeNs: 100n }; + }); - setPlatformForTests("win32"); - const previousUsername = process.env.USERNAME; - process.env.USERNAME = "ocx-test-user"; - let grants = 0; - setIcaclsRunnerForTests(args => { - if (args.includes("/grant:r")) grants += 1; - return { success: true, exitCode: 0, timedOut: false, stdout: "" }; - }); - try { - let readable = true; - setStatForTests(() => { - if (!readable) throw Object.assign(new Error("EACCES"), { code: "EACCES" }); - return { dev: 1n, ino: 10n, ctimeNs: 100n }; + expect(await harden(stable, { required: true })).toEqual({ ok: true }); + expect(grants).toBe(1); + + readable = false; + await expect(harden(stable, { required: true })).rejects.toThrow( + /changed during hardening/, + ); + expect(grants).toBe(2); + expect(hardenedSecretPathCountForTests()).toBe(0); }); + }); - expect(hardenSecretPath(stable, { required: true })).toEqual({ ok: true }); - expect(grants).toBe(1); - - // The memo holds a value, but the path can no longer be observed. - readable = false; - // It must NOT answer from the memo. It re-runs, cannot attribute the run - // either, and a required caller therefore fails closed rather than - // reporting a harden it cannot vouch for. - expect(() => hardenSecretPath(stable, { required: true })).toThrow( - /changed during hardening/, - ); - expect(grants).toBe(2); - expect(hardenedSecretPathCountForTests()).toBe(0); - } finally { - if (previousUsername === undefined) delete process.env.USERNAME; - else process.env.USERNAME = previousUsername; - setIcaclsRunnerForTests(null); - setPlatformForTests(null); - setStatForTests(null); - } - }); + /** + * NTFS is reported to return a zero inode from the non-bigint stat, and the + * guard treats zero as unobservable. That guard had no test at all: removing + * it left every case green, because none ever produced one. + */ + test("a zero inode is unobservable, not an identity", async () => { + resetHardenedStateForTests(); + const stable = join(testDir, `zero-ino-${label}.sqlite`); + writeFileSync(stable, "x", "utf8"); - /** - * NTFS is reported to return a zero inode from the non-bigint stat, and the - * production guard treats a zero inode as unobservable. That guard had no test: - * removing it left all forty-four green, because no case ever produced one. - */ - test("a zero inode is unobservable, not an identity", () => { - resetHardenedStateForTests(); - const stable = join(testDir, "zero-ino.sqlite"); - writeFileSync(stable, "x", "utf8"); + await withWin32(async () => { + let grants = 0; + runner(args => { if (args.includes("/grant:r")) grants += 1; }); + setStatForTests(() => ({ dev: 1n, ino: 0n, ctimeNs: 100n })); - setPlatformForTests("win32"); - const previousUsername = process.env.USERNAME; - process.env.USERNAME = "ocx-test-user"; - let grants = 0; - setIcaclsRunnerForTests(args => { - if (args.includes("/grant:r")) grants += 1; - return { success: true, exitCode: 0, timedOut: false, stdout: "" }; + await expect(harden(stable, { required: true })).rejects.toThrow( + /changed during hardening/, + ); + expect(grants).toBe(1); + expect(hardenedSecretPathCountForTests()).toBe(0); + }); }); - try { - setStatForTests(() => ({ dev: 1n, ino: 0n, ctimeNs: 100n })); - // Required callers fail closed: nothing can be attributed. - expect(() => hardenSecretPath(stable, { required: true })).toThrow( - /changed during hardening/, - ); - expect(grants).toBe(1); - expect(hardenedSecretPathCountForTests()).toBe(0); - } finally { - if (previousUsername === undefined) delete process.env.USERNAME; - else process.env.USERNAME = previousUsername; - setIcaclsRunnerForTests(null); - setPlatformForTests(null); - setStatForTests(null); - } + /** + * Observed absence retires the memo. Otherwise a success entry outlives the + * file it describes, and any later file whose identity happens to match + * satisfies the cache. Microsoft documents that Windows file IDs may be + * reused over time, so this is not a hypothetical on the target platform. + */ + test("observing the path absent retires its success memo", async () => { + resetHardenedStateForTests(); + const stable = join(testDir, `vanishes-${label}.sqlite`); + writeFileSync(stable, "first", "utf8"); + + await withWin32(async () => { + runner(() => {}); + setStatForTests(() => ({ dev: 1n, ino: 10n, ctimeNs: 100n })); + + expect(await harden(stable, { required: true })).toEqual({ ok: true }); + expect(hardenedSecretPathCountForTests()).toBe(1); + + unlinkSync(stable); + expect(await harden(stable, { required: true })).toEqual({ ok: true }); + expect(hardenedSecretPathCountForTests()).toBe(0); + }); + }); }); -}); +} From ab9a39743ae6a7b404ed4c24ad162768ffc7df28 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Wed, 5 Aug 2026 09:08:40 +0900 Subject: [PATCH 107/163] docs(codex): the spec still described the implementation two fixes ago Three stale claims, each of which would have sent the next reader back to a design that has already been defeated: - the source comment still said a null identity is 'recorded as unverifiable', when HardenedIdentity is now non-null and unreadable observations delete the entry instead. That branch was dead code inside the fix that was supposed to need it, which is why the type no longer has the member. - the component-validation section still said the memo is 'Today' a Set. It has been a Map for two commits, and that sentence sat outside the historical-record block where it read as current fact. - the F4 criterion still named only release/replace/reacquire. It now names all eight mutations, marks the full-identity-across-the-ACL one as the Windows-breaking form, and says why the async check is not redundant. Also moved the WP4 supersession notice onto the declaration itself rather than only the section header, and corrected what it points at: WP4 consumes the public N acquisition API in codex-write-lock.ts, not openCodexCoordinatorTransaction directly. The transaction is N's lower layer; calling it straight would skip the admission comparison, lock ordering, and refusal taxonomy that make N a boundary rather than a mutex. The 260803 fence check reports two pre-existing TS2391s from bodyless declarations elsewhere in that document; commenting out the superseded one took it from three to two. --- .../030_desired_state.md | 11 +++-- .../030_lock_protocol.md | 42 ++++++++++++++----- src/lib/windows-secret-acl.ts | 10 +++-- 3 files changed, 45 insertions(+), 18 deletions(-) diff --git a/devlog/_plan/260803_codex_desktop_toggle/030_desired_state.md b/devlog/_plan/260803_codex_desktop_toggle/030_desired_state.md index a49c9b85a..c1cc3d649 100644 --- a/devlog/_plan/260803_codex_desktop_toggle/030_desired_state.md +++ b/devlog/_plan/260803_codex_desktop_toggle/030_desired_state.md @@ -254,9 +254,9 @@ real migration after OFF and ON mutations and asserts: no `claudeCode` block, > its remedy is now a duplicate — a second, weaker per-home lock beside the one the > other unit already owns: > -> | This section proposes | What already exists | +> | This section proposes | What WP4 consumes instead | > |---|---| -> | `withCodexHomeLinearizationLockSync` in a NEW `desired-state.ts` | the N coordinator transaction, `src/codex/transition-state.ts:348` | +> | `withCodexHomeLinearizationLockSync` in a NEW `desired-state.ts` | the **public N acquisition API** in `src/codex/codex-write-lock.ts` (WP12). Not `openCodexCoordinatorTransaction` directly — that is N's lower layer, and calling it straight would bypass N's admission comparison, lock ordering, and refusal taxonomy. | > | `join(tmpdir(), "opencodex-native-locks", sha256(home) + ".sqlite")` | `resolveCodexCoordinatorDatabasePath`, `src/codex/user-identity.ts:165`, which keys on the effective **uid/SID** as well as the canonical home | > | `inspectNativeCodexOwnership()` here | `AdmissionSnapshot.ownership`, owned by WP12's admission producer | > @@ -307,7 +307,12 @@ export interface CodexReconcileResult { } export function inspectNativeCodexOwnership(): NativeCodexOwnership; -export function withCodexHomeLinearizationLockSync(operation: () => T): T; +// SUPERSEDED — WP4 does not declare this. Linearization comes from WP12's public +// N acquisition API in `src/codex/codex-write-lock.ts`, whose callback is already +// synchronous and already holds N -> C. Declaring a second per-home lock here +// would key on sha256(home) without the uid/SID and split one home across two OS +// users, which is the failure `005_contract.md` §7 exists to prevent. +// export function withCodexHomeLinearizationLockSync(operation: () => T): T; export function setCodexDesiredEnabled(enabled: boolean): CodexDesiredMutationResult; export function setCodexBeforeNativeWriteForTests( hook: ((boundary: CodexNativeWriteBoundary) => void) | null, diff --git a/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md b/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md index b8e98ef3d..8f9185e93 100644 --- a/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md +++ b/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md @@ -632,13 +632,24 @@ Walk components one at a time; never recursive-mkdir across an unvalidated paren proves nothing about safety (F5). That requires preserving the `ETIMEDOUT` discriminator through sanitization at `src/lib/windows-secret-acl.ts:484`, which today rethrows an untyped `Error` and destroys it. -- The ACL success memo must be bound to **file identity**, not pathname. Today it is - a `Set` of paths (`src/lib/windows-secret-acl.ts:36,461`), so a file - replaced at the same name inherits the previous file's hardening — probed as - `{identityChanged:true, firstCalls:3, totalCalls:3, replacementWasRechecked:false}` - (F4). Ephemeral temps already invalidate through `forgetEphemeralSecretPath` +- The ACL success memo is bound to file identity, not pathname — **shipped**, and the + shape it settled on is not the one this section originally prescribed. It separates + two questions the first fix conflated: + - `object` = `dev:ino` — is this the same file? Compared **before and after** the + icacls sequence, because "what is at this path now" does not answer "what did + icacls operate on". + - `freshness` = `ctimeNs` — has this file's metadata moved since? Stored from the + **post-harden** read and deliberately NOT compared across the ACL call: icacls + changes permissions and a permission change moves ctime (probed, + `{ctimeChangedByChmod: true}`), so comparing it there would have rejected every + successful harden and failed closed on the first harden of every path on Windows. + + Ephemeral temps already invalidated through `forgetEphemeralSecretPath` (`src/config.ts:214,241,309,336,480,501-510`); the stable destination memo that - `hardenStableLockFile` uses never does. + `hardenStableLockFile` uses never did, and observed absence now retires it. Both the + sync and async entry points are covered — an audit removed the async attribution + alone and every test stayed green, while async is the path + `hardenStableLockFile` actually takes (`src/codex/native-main-lock-file.ts:130`). - Existing DB or `-journal` must be regular, same-user private entries. Existing `-wal`/`-shm` refuses. The lock **verifies** rollback journal mode rather than forcing it: a pinned-Bun probe shows `bun:sqlite` already opens `delete` and leaves @@ -970,11 +981,20 @@ and the ones that did were rewritten rather than kept. the `busy` SQLite alone produces. *Red when:* ALS is removed — the reason must degrade to `busy`. Asserting only "does not hang" is vacuous, because `busy_timeout = 0` already guarantees that. -- **F4** — a coordinator database released, replaced at the same pathname, and - reacquired is re-hardened. - *Red when:* identity binding / memo invalidation is removed — the - release → replace → **reacquire** test must fail. Substituting during a single - held acquisition never reaches the memo and would pass with the fix gone. +- **F4** — a harden is credited only to the file it was performed on, through BOTH + the sync and async entry points. + *Red when (each separately, and each in both parameterizations):* (a) `dev` is + dropped from the object; (b) `freshness` is dropped from the memo value; (c) the + before/after object comparison is removed; (d) observed absence keeps the memo; + (e) an unreadable observation satisfies the memo; (f) a zero inode is accepted as + an identity; (g) the FULL identity is compared across the ACL call — this one is + the Windows-breaking form, since icacls moves ctime; (h) the async attribution + alone is removed. (h) is not redundant: it survived every other check while + `hardenStableLockFile` takes exactly that path. + Replacement is driven through the stat seam rather than a real unlink/recreate: + ext4 recycles an inode immediately and APFS did not once in 200 cycles, so a + real-file version asserts different things on different machines — which is how a + fix broken on Linux passed here on macOS. - **F5** — outer-budget exhaustion during ACL work returns retryable `busy/deadline`; verified ACL/ownership/path failure returns non-retryable refusal. *Red when:* `ETIMEDOUT` is collapsed into a generic refusal — the deadline diff --git a/src/lib/windows-secret-acl.ts b/src/lib/windows-secret-acl.ts index 4a237405f..00b429ae2 100644 --- a/src/lib/windows-secret-acl.ts +++ b/src/lib/windows-secret-acl.ts @@ -38,11 +38,13 @@ const hardenedPaths = new Map(); const timedOutPaths = new Set(); /** - * Identity of the file a successful harden actually applied to. + * The memo value: `object:freshness` for a file a harden was actually attributed + * to. * - * `null` means the identity could not be established, which is NOT the same as - * "unchanged" — it is recorded as unverifiable so the next harden re-runs - * instead of inheriting a previous file's credit. + * There is deliberately no null member. An observation that cannot be read is + * not stored at all — the entry is deleted — because a "recorded as unverifiable" + * value was dead code the moment attribution became a before/after comparison, + * and a branch nothing can reach is a branch no test can defend. */ type HardenedIdentity = string; From 9e40a8d5b076a624cbc85cb9322655f820dff737 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Wed, 5 Aug 2026 09:17:06 +0900 Subject: [PATCH 108/163] test(acl): the directory half was uncovered too, and that is four production callers MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Second time this suite proved the wrong half. Last round it was async; this round a reviewer inserted pathname-only satisfaction for DIRECTORIES alone — if (directory && cache.has(targetPath)) return { ok: true }; — and all 55 tests stayed green. hardenSecretDir/hardenSecretDirAsync share the same memo machinery and back config.ts:1292,1760, management-auth, tray, spill-store, and native-profile-manager.ts:153. A directory replaced at the same pathname would have inherited the previous directory's ACL credit. The matrix now covers all four public entry points — file/sync, file/async, dir/sync, dir/async — with per-entry creation, and hardenedSecretDirCountForTests exists because the directory memo had no counter seam at all. 73 tests. Nine mutations, isolated archive copies, each reddening tests across the parameterizations, all restored to 73 pass / 0 fail: dev dropped, freshness dropped, no before/after comparison, absence keeping the memo, unreadable-now satisfying it, zero inode accepted, full identity across the ACL call, async attribution removed, directory pathname-only memo. Two things bit along the way and are worth the note: entry labels became file-sync rather than file/sync because the slash was read as a path separator in the fixture names, and hardenSecretDirAsync was missing from the import so the dir-async cases failed with ReferenceError rather than an assertion. Also finished the WP4 supersession the reviewer said was half-done. The warning was at the top while the body still read as live instructions — 'NEW desired-state.ts owns one linearization boundary' and the tmpdir()/sha256(home) INFERRED design choice. Both now say what replaces them, at the point a reader meets them, and the NTFS bigint-ino question is recorded as an activation gate rather than a permanent residual: if Bun returns zero there, every required Windows harden fails closed. --- .../030_desired_state.md | 28 ++++--- .../030_lock_protocol.md | 25 +++--- src/lib/windows-secret-acl.ts | 8 ++ tests/windows-secret-acl.test.ts | 79 ++++++++++++++----- 4 files changed, 104 insertions(+), 36 deletions(-) diff --git a/devlog/_plan/260803_codex_desktop_toggle/030_desired_state.md b/devlog/_plan/260803_codex_desktop_toggle/030_desired_state.md index c1cc3d649..a994d6b61 100644 --- a/devlog/_plan/260803_codex_desktop_toggle/030_desired_state.md +++ b/devlog/_plan/260803_codex_desktop_toggle/030_desired_state.md @@ -279,7 +279,11 @@ separate config mutation transaction before the apply reaches `atomicWriteFile`. The several config/profile/journal/history writes at `src/codex/inject.ts:524-603` have the same defect. -NEW `src/codex/desired-state.ts` owns one linearization boundary: +NEW `src/codex/desired-state.ts` owns the desired-state surface below. It does NOT +own the linearization boundary — that comes from WP12's public N acquisition API in +`src/codex/codex-write-lock.ts`, whose callback is already synchronous and already +holds `N -> C`. Everything from here to the end of this section is the ORIGINAL +2026-08-04 design, retained for its diagnosis; read the lock parts as history: ```ts export type NativeCodexOwnership = @@ -323,14 +327,20 @@ export function reconcileCodexDesiredState( ): CodexReconcileResult; ``` -**INFERRED design choice:** canonicalize the effective `CODEX_HOME`, hash that -canonical path with SHA-256, and store the SQLite lock at -`join(tmpdir(), "opencodex-native-locks", + ".sqlite")`, mode `0600` in a -mode-`0700` directory. It is outside `CODEX_HOME` and independent of -`OPENCODEX_HOME`, so two OpenCodex homes targeting one native home serialize on -one lock without writing a lock artifact into the target. Process exit releases -the SQLite transaction. The callback is synchronous and bounded: there is no -provider fetch, model discovery, sleep, or other `await` while it is held. +> **SUPERSEDED — do not implement this paragraph.** It read: canonicalize the +> effective `CODEX_HOME`, hash it with SHA-256, and store the SQLite lock at +> `join(tmpdir(), "opencodex-native-locks", + ".sqlite")`. Keying on the home +> alone omits the effective uid/SID, so a service and a CLI running as different OS +> users would take *different* lock files for one home and serialize with nothing — +> the precise split `005_contract.md` §7 exists to prevent, and `os.homedir()` is +> environment-controlled under Bun 1.3.14 besides. +> +> WP4 calls WP12's public N acquisition API instead. The path resolution belongs to +> `resolveCodexCoordinatorDatabasePath` (`src/codex/user-identity.ts:165`), which +> keys on uid/SID **and** the canonical home. What the withdrawn paragraph got right +> is retained by N anyway: the lock lives outside `CODEX_HOME`, process exit +> releases the transaction, and the callback is synchronous and bounded — no +> provider fetch, model discovery, sleep, or other `await` while it is held. The ordering invariant is: diff --git a/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md b/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md index 8f9185e93..97c75a8d4 100644 --- a/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md +++ b/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md @@ -646,10 +646,14 @@ Walk components one at a time; never recursive-mkdir across an unvalidated paren Ephemeral temps already invalidated through `forgetEphemeralSecretPath` (`src/config.ts:214,241,309,336,480,501-510`); the stable destination memo that - `hardenStableLockFile` uses never did, and observed absence now retires it. Both the - sync and async entry points are covered — an audit removed the async attribution - alone and every test stayed green, while async is the path - `hardenStableLockFile` actually takes (`src/codex/native-main-lock-file.ts:130`). + `hardenStableLockFile` uses never did, and observed absence now retires it. All + **four** public entry points are covered — file and directory, sync and async — + because the suite was twice found to be proving the wrong half: removing the async + attribution alone left every test green while async is the path + `hardenStableLockFile` actually takes (`src/codex/native-main-lock-file.ts:130`), + and a pathname-only memo applied to directories ALONE also left every test green + while `hardenSecretDir` backs config, management-auth, tray, spill-store, and + `native-profile-manager.ts:153`. - Existing DB or `-journal` must be regular, same-user private entries. Existing `-wal`/`-shm` refuses. The lock **verifies** rollback journal mode rather than forcing it: a pinned-Bun probe shows `bun:sqlite` already opens `delete` and leaves @@ -981,16 +985,19 @@ and the ones that did were rewritten rather than kept. the `busy` SQLite alone produces. *Red when:* ALS is removed — the reason must degrade to `busy`. Asserting only "does not hang" is vacuous, because `busy_timeout = 0` already guarantees that. -- **F4** — a harden is credited only to the file it was performed on, through BOTH - the sync and async entry points. - *Red when (each separately, and each in both parameterizations):* (a) `dev` is +- **F4** — a harden is credited only to the object it was performed on, through all + **four** public entry points: file/sync, file/async, dir/sync, dir/async. + *Red when (each separately, and each across all four parameterizations):* (a) `dev` is dropped from the object; (b) `freshness` is dropped from the memo value; (c) the before/after object comparison is removed; (d) observed absence keeps the memo; (e) an unreadable observation satisfies the memo; (f) a zero inode is accepted as an identity; (g) the FULL identity is compared across the ACL call — this one is the Windows-breaking form, since icacls moves ctime; (h) the async attribution - alone is removed. (h) is not redundant: it survived every other check while - `hardenStableLockFile` takes exactly that path. + alone is removed; (i) a pathname-only memo is applied to directories alone. + (h) and (i) are not redundant — each survived every other check, and each covers a + real production caller: `hardenStableLockFile` takes the async path, and + `hardenSecretDir` backs config, management-auth, tray, spill-store, and + `native-profile-manager.ts:153`. Replacement is driven through the stat seam rather than a real unlink/recreate: ext4 recycles an inode immediately and APFS did not once in 200 cycles, so a real-file version asserts different things on different machines — which is how a diff --git a/src/lib/windows-secret-acl.ts b/src/lib/windows-secret-acl.ts index 00b429ae2..6c259cb90 100644 --- a/src/lib/windows-secret-acl.ts +++ b/src/lib/windows-secret-acl.ts @@ -347,6 +347,14 @@ export function hardenedSecretPathCountForTests(): number { return hardenedPaths.size; } +/** + * Directory counterpart. It had no seam, and that absence hid a real gap: a + * directory-only pathname memo passed every file-based test in this suite. + */ +export function hardenedSecretDirCountForTests(): number { + return hardenedDirectories.size; +} + function effectivePlatform(): string { return platformOverride ?? platform; } diff --git a/tests/windows-secret-acl.test.ts b/tests/windows-secret-acl.test.ts index 6e7e255f1..4ec1bf5bc 100644 --- a/tests/windows-secret-acl.test.ts +++ b/tests/windows-secret-acl.test.ts @@ -10,15 +10,17 @@ * - hardenSecretDir mirrors the same contract for directories. */ import { afterEach, beforeEach, describe, expect, test } from "bun:test"; -import { chmodSync, existsSync, mkdtempSync, renameSync, rmSync, statSync, truncateSync, unlinkSync, writeFileSync } from "node:fs"; +import { chmodSync, existsSync, mkdirSync, mkdtempSync, renameSync, rmSync, statSync, truncateSync, unlinkSync, writeFileSync } from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { forgetEphemeralSecretPath, hardenSecretDir, + hardenSecretDirAsync, forgetHardenedSecretPath, hardenSecretPath, hardenSecretPathAsync, + hardenedSecretDirCountForTests, hardenedSecretPathCountForTests, resetHardenedStateForTests, setAsyncIcaclsRunnerForTests, @@ -709,9 +711,45 @@ describe("ephemeral ACL memo release (#840 refinement)", () => { */ type HardenFn = (path: string, opts: { required: boolean }) => Promise; -const ENTRY_POINTS: readonly { readonly label: string; readonly harden: HardenFn }[] = [ - { label: "sync", harden: async (path, opts) => hardenSecretPath(path, opts) }, - { label: "async", harden: (path, opts) => hardenSecretPathAsync(path, opts) }, +/** + * All FOUR public entry points, not two. + * + * `hardenSecretDir`/`hardenSecretDirAsync` share the same memo machinery through + * `hardenedDirectories`, and an audit proved the gap: inserting + * `if (directory && cache.has(targetPath)) return { ok: true }` — pathname-only + * satisfaction for directories alone — left all 55 file-only tests green. That + * reaches real callers: config (`src/config.ts:1292,1760`), management-auth, + * tray, spill-store, and `native-profile-manager.ts:153`. + * + * Each entry carries its own `create`, so a directory case makes a directory. + */ +interface EntryPoint { + readonly label: string; + readonly harden: HardenFn; + readonly create: (path: string) => void; +} + +const ENTRY_POINTS: readonly EntryPoint[] = [ + { + label: "file-sync", + harden: async (path, opts) => hardenSecretPath(path, opts), + create: path => writeFileSync(path, "x", "utf8"), + }, + { + label: "file-async", + harden: (path, opts) => hardenSecretPathAsync(path, opts), + create: path => writeFileSync(path, "x", "utf8"), + }, + { + label: "dir-sync", + harden: async (path, opts) => hardenSecretDir(path, opts), + create: path => mkdirSync(path, { recursive: true }), + }, + { + label: "dir-async", + harden: (path, opts) => hardenSecretDirAsync(path, opts), + create: path => mkdirSync(path, { recursive: true }), + }, ]; /** @@ -744,7 +782,12 @@ function runner(onCall: (args: string[]) => void): void { setAsyncIcaclsRunnerForTests(async args => { onCall(args); return result; }); } -for (const { label, harden } of ENTRY_POINTS) { +for (const { label, harden, create } of ENTRY_POINTS) { + // Files and directories keep separate memos; a case must read its own. + const memoCount = label.startsWith("dir") + ? hardenedSecretDirCountForTests + : hardenedSecretPathCountForTests; + describe(`F4 memo attribution — ${label} entry point`, () => { /** * The original defect. The memo was a Set of PATH STRINGS, so unlinking a @@ -758,7 +801,7 @@ for (const { label, harden } of ENTRY_POINTS) { test("a file replaced at an already-hardened stable path is hardened again", async () => { resetHardenedStateForTests(); const stable = join(testDir, `replaced-${label}.sqlite`); - writeFileSync(stable, "first", "utf8"); + create(stable); await withWin32(async () => { let grants = 0; @@ -788,7 +831,7 @@ for (const { label, harden } of ENTRY_POINTS) { test("a file replaced DURING hardening is not credited, and required fails closed", async () => { resetHardenedStateForTests(); const stable = join(testDir, `raced-${label}.sqlite`); - writeFileSync(stable, "original", "utf8"); + create(stable); await withWin32(async () => { let grants = 0; @@ -805,7 +848,7 @@ for (const { label, harden } of ENTRY_POINTS) { ); expect(grants).toBe(1); // No memo was left for the replacement, so a later harden still runs. - expect(hardenedSecretPathCountForTests()).toBe(0); + expect(memoCount()).toBe(0); // An optional caller hitting the same race soft-fails with the same // honest diagnostic. The runner keeps swapping the file, so this second @@ -828,7 +871,7 @@ for (const { label, harden } of ENTRY_POINTS) { test("ctime moving during hardening is the ACL's own doing, not a substitution", async () => { resetHardenedStateForTests(); const stable = join(testDir, `acl-ctime-${label}.sqlite`); - writeFileSync(stable, "x", "utf8"); + create(stable); await withWin32(async () => { let grants = 0; @@ -866,7 +909,7 @@ for (const { label, harden } of ENTRY_POINTS) { test(`a change in ${name} alone forces a re-harden`, async () => { resetHardenedStateForTests(); const stable = join(testDir, `component-${name}-${label}.sqlite`); - writeFileSync(stable, "x", "utf8"); + create(stable); await withWin32(async () => { let grants = 0; @@ -894,7 +937,7 @@ for (const { label, harden } of ENTRY_POINTS) { test("a stat failure after a successful harden forces the harden to run again", async () => { resetHardenedStateForTests(); const stable = join(testDir, `unreadable-${label}.sqlite`); - writeFileSync(stable, "x", "utf8"); + create(stable); await withWin32(async () => { let grants = 0; @@ -913,7 +956,7 @@ for (const { label, harden } of ENTRY_POINTS) { /changed during hardening/, ); expect(grants).toBe(2); - expect(hardenedSecretPathCountForTests()).toBe(0); + expect(memoCount()).toBe(0); }); }); @@ -925,7 +968,7 @@ for (const { label, harden } of ENTRY_POINTS) { test("a zero inode is unobservable, not an identity", async () => { resetHardenedStateForTests(); const stable = join(testDir, `zero-ino-${label}.sqlite`); - writeFileSync(stable, "x", "utf8"); + create(stable); await withWin32(async () => { let grants = 0; @@ -936,7 +979,7 @@ for (const { label, harden } of ENTRY_POINTS) { /changed during hardening/, ); expect(grants).toBe(1); - expect(hardenedSecretPathCountForTests()).toBe(0); + expect(memoCount()).toBe(0); }); }); @@ -949,18 +992,18 @@ for (const { label, harden } of ENTRY_POINTS) { test("observing the path absent retires its success memo", async () => { resetHardenedStateForTests(); const stable = join(testDir, `vanishes-${label}.sqlite`); - writeFileSync(stable, "first", "utf8"); + create(stable); await withWin32(async () => { runner(() => {}); setStatForTests(() => ({ dev: 1n, ino: 10n, ctimeNs: 100n })); expect(await harden(stable, { required: true })).toEqual({ ok: true }); - expect(hardenedSecretPathCountForTests()).toBe(1); + expect(memoCount()).toBe(1); - unlinkSync(stable); + rmSync(stable, { recursive: true, force: true }); expect(await harden(stable, { required: true })).toEqual({ ok: true }); - expect(hardenedSecretPathCountForTests()).toBe(0); + expect(memoCount()).toBe(0); }); }); }); From 6d960c9a1208d41e8312a30d7c1e9e01c34235bc Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Wed, 5 Aug 2026 09:26:08 +0900 Subject: [PATCH 109/163] test(acl): 76 tests proved the primitive; nothing proved production still called it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A reviewer deleted the entire Windows ACL delegation from hardenStableLockFile and 86 tests stayed green across three files. That function is what hardens the coordinator database, through native-main-claim.ts:81 and native-main-owner.ts:147. Three rounds of hardening the primitive, and the call edge that activates it had no test at all — the same absence-as-guarantee defect one layer up. It could not have had one: the Windows branch read process.platform directly, so it was unreachable from a test on any other machine. The platform is now a parameter, which is what makes the branch testable rather than merely tested. Three tests on the edge itself: on win32 the required async hardener runs against that exact path; a required ACL failure propagates instead of leaving a coordinator database other accounts can read; on posix it hardens by mode alone and runs no ACL command. Eleven mutations now, each in an isolated archive copy, all restored to 76 pass / 0 fail: the nine from before, plus deleting the delegation (74/2) and weakening its required:true to false (75/1). One thing I did NOT assert, and said so in the test: timeoutMemoKey: path at that call site. timeoutMemoKey() already falls back to targetPath and this caller passes the target path, so changing it is unobservable — a test pinning it would pass with the mechanism gone, which is the kind of test this unit has already written twice by accident. Also corrected the WP4 supersession reasoning, which asserted a definite outcome in two places and contradicted itself between them. Omitting the uid/SID does not reliably split or reliably collide: it leaves the lock path carrying no proof of account, so a shared temp root collides and a per-user one splits. Environment-dependent is the honest claim, and it is worse than either. --- .../030_desired_state.md | 21 +++-- .../030_lock_protocol.md | 21 ++++- src/codex/native-main-lock-file.ts | 20 ++++- tests/windows-secret-acl.test.ts | 87 +++++++++++++++++++ 4 files changed, 136 insertions(+), 13 deletions(-) diff --git a/devlog/_plan/260803_codex_desktop_toggle/030_desired_state.md b/devlog/_plan/260803_codex_desktop_toggle/030_desired_state.md index a994d6b61..880d69ed8 100644 --- a/devlog/_plan/260803_codex_desktop_toggle/030_desired_state.md +++ b/devlog/_plan/260803_codex_desktop_toggle/030_desired_state.md @@ -261,9 +261,14 @@ real migration after OFF and ON mutations and asserts: no `claudeCode` block, > | `inspectNativeCodexOwnership()` here | `AdmissionSnapshot.ownership`, owned by WP12's admission producer | > > Hashing the home ALONE is the specific defect `005_contract.md` §7 exists to -> prevent: a service and a CLI running as different OS users would share one lock -> file for one home, and `os.homedir()` is environment-controlled under Bun 1.3.14 -> either way. Building this would ship that bug knowingly. +> prevent, and the honest statement of it is that the outcome is **undetermined**, +> not that it splits or that it collides. The lock path would carry no proof of +> effective-user authority, so what actually happens depends on the temp root: a +> shared `/tmp` puts two OS users on one lock file (collision, or an access failure +> on the other's mode-0600 database), while a per-user or environment-controlled +> temp root splits one home across two lock files (no exclusion at all, silently). +> `os.homedir()` is environment-controlled under Bun 1.3.14 besides. WP12's +> resolver removes the ambiguity by encoding uid/SID directly. > > **Consequence for sequencing:** WP4 depends on WP12's lock, so it runs after it, > not beside it. WP4 keeps everything below that is genuinely its own — the @@ -330,10 +335,12 @@ export function reconcileCodexDesiredState( > **SUPERSEDED — do not implement this paragraph.** It read: canonicalize the > effective `CODEX_HOME`, hash it with SHA-256, and store the SQLite lock at > `join(tmpdir(), "opencodex-native-locks", + ".sqlite")`. Keying on the home -> alone omits the effective uid/SID, so a service and a CLI running as different OS -> users would take *different* lock files for one home and serialize with nothing — -> the precise split `005_contract.md` §7 exists to prevent, and `os.homedir()` is -> environment-controlled under Bun 1.3.14 besides. +> alone omits the effective uid/SID, so the lock path carries no proof of which +> account it belongs to. The failure it produces is **environment-dependent**, which +> is worse than a fixed one: a shared temp root puts two OS users on one lock file, +> while a per-user or environment-controlled temp root splits one home across two +> and they serialize with nothing. `os.homedir()` is environment-controlled under +> Bun 1.3.14 besides. `005_contract.md` §7 exists to remove exactly this ambiguity. > > WP4 calls WP12's public N acquisition API instead. The path resolution belongs to > `resolveCodexCoordinatorDatabasePath` (`src/codex/user-identity.ts:165`), which diff --git a/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md b/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md index 97c75a8d4..94b5fbf03 100644 --- a/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md +++ b/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md @@ -994,10 +994,23 @@ and the ones that did were rewritten rather than kept. an identity; (g) the FULL identity is compared across the ACL call — this one is the Windows-breaking form, since icacls moves ctime; (h) the async attribution alone is removed; (i) a pathname-only memo is applied to directories alone. - (h) and (i) are not redundant — each survived every other check, and each covers a - real production caller: `hardenStableLockFile` takes the async path, and - `hardenSecretDir` backs config, management-auth, tray, spill-store, and - `native-profile-manager.ts:153`. + (j) `hardenStableLockFile`'s Windows delegation is deleted; (k) its `required: + true` is weakened to `false`. + (h) through (k) are not redundant — each survived every other check. (h) and (i) + cover production callers the primitive tests missed: `hardenStableLockFile` takes + the async path, and `hardenSecretDir` backs config, management-auth, tray, + spill-store, and `native-profile-manager.ts:153`. (j) and (k) are a different + layer entirely: deleting the whole ACL delegation left 86 tests green across three + files, because every test proved the primitive and none proved production still + called it. `hardenStableLockFile` takes its platform as a parameter for exactly + that reason — a direct `process.platform` read made the Windows branch unreachable + from a test. + + Deliberately NOT asserted: the `timeoutMemoKey: path` argument at that call site. + `timeoutMemoKey()` already falls back to `targetPath` and this caller passes the + target path, so changing it is unobservable and a test pinning it would be + vacuous. The option exists for atomic writers that mint a fresh temp per write + and need the stable destination as the key (#612); this caller has no temp. Replacement is driven through the stat seam rather than a real unlink/recreate: ext4 recycles an inode immediately and APFS did not once in 200 cycles, so a real-file version asserts different things on different machines — which is how a diff --git a/src/codex/native-main-lock-file.ts b/src/codex/native-main-lock-file.ts index 92021562b..9e37cbefc 100644 --- a/src/codex/native-main-lock-file.ts +++ b/src/codex/native-main-lock-file.ts @@ -124,9 +124,25 @@ export function assertStableLockFile(path: string, handle: StableLockFile): void } } -export async function hardenStableLockFile(path: string): Promise { +/** + * Harden the stable lock file: chmod everywhere, per-user NTFS ACLs on Windows. + * + * The platform is a parameter rather than a direct `process.platform` read so the + * Windows branch is reachable from a test. It was not, and an audit deleted the + * whole ACL delegation without a single test noticing — the primitive underneath + * had 73 tests while the call edge that activates it had none. That is the same + * absence-as-guarantee defect one layer up: proving a mechanism works says nothing + * about whether production still calls it. + * + * `required: true` is deliberate: a failure to apply the ACL must reject rather + * than leave a coordinator database readable by other accounts. + */ +export async function hardenStableLockFile( + path: string, + platform: NodeJS.Platform = process.platform, +): Promise { try { chmodSync(path, 0o600); } catch { /* Windows ACL below is authoritative there. */ } - if (process.platform === "win32") { + if (platform === "win32") { await hardenSecretPathAsync(path, { required: true, timeoutMemoKey: path }); } } diff --git a/tests/windows-secret-acl.test.ts b/tests/windows-secret-acl.test.ts index 4ec1bf5bc..0098064fb 100644 --- a/tests/windows-secret-acl.test.ts +++ b/tests/windows-secret-acl.test.ts @@ -33,6 +33,7 @@ import { type IcaclsResult, } from "../src/lib/windows-secret-acl"; import { atomicWriteFile } from "../src/config"; +import { hardenStableLockFile } from "../src/codex/native-main-lock-file"; let testDir = ""; @@ -1008,3 +1009,89 @@ for (const { label, harden, create } of ENTRY_POINTS) { }); }); } + +describe("hardenStableLockFile — the production call edge, not just the primitive", () => { + /** + * The gap this closes was invisible for three audit rounds. + * + * `hardenStableLockFile` is what actually hardens the coordinator database, via + * `native-main-claim.ts:81` and `native-main-owner.ts:147`. Deleting its entire + * Windows ACL delegation left 86 tests green across three files — because every + * test proved the primitive, and nothing proved production still called it. + * + * So this asserts the edge itself: on win32 the required async hardener runs + * with the pathname as its timeout memo key, and its failure propagates rather + * than leaving a coordinator database other accounts can read. + */ + test("on win32 it delegates to the required async hardener, keyed by the path", async () => { + resetHardenedStateForTests(); + const lockPath = join(testDir, "coordinator-claim.sqlite"); + writeFileSync(lockPath, "x", "utf8"); + + setPlatformForTests("win32"); + const previousUsername = process.env.USERNAME; + process.env.USERNAME = "ocx-test-user"; + const seen: string[][] = []; + setAsyncIcaclsRunnerForTests(async args => { + seen.push(args); + return { success: true, exitCode: 0, timedOut: false, stdout: "" }; + }); + try { + await hardenStableLockFile(lockPath, "win32"); + // The ACL sequence ran against this exact path. + expect(seen.length).toBeGreaterThan(0); + expect(seen.every(args => args[0] === lockPath)).toBe(true); + expect(seen.some(args => args.includes("/grant:r"))).toBe(true); + // NOT asserted here: the `timeoutMemoKey: path` argument. It cannot be, + // because it is redundant — `timeoutMemoKey()` falls back to `targetPath`, + // and this caller passes the target path itself. Changing it in production + // is unobservable, so a test claiming to pin it would be vacuous. The + // option exists for atomic writers that mint a fresh temp per write and + // need the STABLE destination as the key (#612); this caller has no temp. + } finally { + if (previousUsername === undefined) delete process.env.USERNAME; + else process.env.USERNAME = previousUsername; + setAsyncIcaclsRunnerForTests(null); + setPlatformForTests(null); + } + }); + + test("a required ACL failure propagates instead of leaving the file unhardened", async () => { + resetHardenedStateForTests(); + const lockPath = join(testDir, "coordinator-fails.sqlite"); + writeFileSync(lockPath, "x", "utf8"); + + setPlatformForTests("win32"); + const previousUsername = process.env.USERNAME; + process.env.USERNAME = "ocx-test-user"; + setAsyncIcaclsRunnerForTests(async () => ({ + success: false, exitCode: 5, timedOut: false, stdout: "", stderr: "", + })); + try { + await expect(hardenStableLockFile(lockPath, "win32")).rejects.toThrow(); + } finally { + if (previousUsername === undefined) delete process.env.USERNAME; + else process.env.USERNAME = previousUsername; + setAsyncIcaclsRunnerForTests(null); + setPlatformForTests(null); + } + }); + + test("on posix it hardens by mode alone and runs no ACL command", async () => { + resetHardenedStateForTests(); + const lockPath = join(testDir, "coordinator-posix.sqlite"); + writeFileSync(lockPath, "x", "utf8"); + + let calls = 0; + setAsyncIcaclsRunnerForTests(async () => { + calls += 1; + return { success: true, exitCode: 0, timedOut: false, stdout: "" }; + }); + try { + await hardenStableLockFile(lockPath, "linux"); + expect(calls).toBe(0); + } finally { + setAsyncIcaclsRunnerForTests(null); + } + }); +}); From cc6f500ccc30bd2ab962dfe321e8574b6a939427 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Wed, 5 Aug 2026 09:37:22 +0900 Subject: [PATCH 110/163] fix(codex): the POSIX chmod was swallowed, and nothing proved a claim still hardened MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two more holes, both found by deleting production code and watching 89 tests stay green. The first is a real fail-open, not just a coverage gap. hardenStableLockFile wrapped its chmod in an unconditional catch, so on POSIX — where the mode IS the mechanism and there is no ACL fallback — a coordinator database that already existed with permissive bits stayed permissive while the caller was told it had been hardened. Creation mode 0600 does not repair an existing file. Windows keeps the best-effort chmod because the required ACL decides there; POSIX now propagates. The only POSIX test asserted that no ACL command ran, which was equally true of the working and the broken version. It now starts the file at 0644 and requires 0600 afterwards, and a second test proves a chmod failure propagates. The second is one layer further out. Nearly every native-main claim test injects hardenPath, so deleting the hardening call from openClaimDatabase changed nothing anywhere. The resolved platform is now threaded into the default hardener — otherwise a test forcing platform here still exercises the host's branch — and one test runs a claim with no override and inspects the mode it left behind. Fourteen mutations now, all restored to 91 pass / 0 fail across three files. The three new ones: POSIX chmod deleted (81/3), that failure swallowed again (83/1), claim-site call deleted (83/1). Also removed the redundant timeoutMemoKey: path argument rather than keep defending a non-assertion, rewrote the plan's deadline diff against the real signature instead of the two-signatures-ago sketch, and fixed the last place the supersession note still stated a definite split where the outcome is environment-dependent. --- .../030_desired_state.md | 7 ++- .../030_lock_protocol.md | 47 +++++++++++++++---- src/codex/native-main-claim.ts | 8 +++- src/codex/native-main-lock-file.ts | 14 +++++- tests/native-main-claim.test.ts | 30 +++++++++++- tests/windows-secret-acl.test.ts | 41 ++++++++++++---- 6 files changed, 123 insertions(+), 24 deletions(-) diff --git a/devlog/_plan/260803_codex_desktop_toggle/030_desired_state.md b/devlog/_plan/260803_codex_desktop_toggle/030_desired_state.md index 880d69ed8..a7facf0de 100644 --- a/devlog/_plan/260803_codex_desktop_toggle/030_desired_state.md +++ b/devlog/_plan/260803_codex_desktop_toggle/030_desired_state.md @@ -319,8 +319,11 @@ export function inspectNativeCodexOwnership(): NativeCodexOwnership; // SUPERSEDED — WP4 does not declare this. Linearization comes from WP12's public // N acquisition API in `src/codex/codex-write-lock.ts`, whose callback is already // synchronous and already holds N -> C. Declaring a second per-home lock here -// would key on sha256(home) without the uid/SID and split one home across two OS -// users, which is the failure `005_contract.md` §7 exists to prevent. +// would key on sha256(home) without the uid/SID, so the lock path would carry no +// proof of which account it belongs to and the failure would be environment- +// dependent: a shared temp root collides, a per-user one splits one home across +// two locks that serialize with nothing. `005_contract.md` §7 exists to remove +// exactly that ambiguity. // export function withCodexHomeLinearizationLockSync(operation: () => T): T; export function setCodexDesiredEnabled(enabled: boolean): CodexDesiredMutationResult; export function setCodexBeforeNativeWriteForTests( diff --git a/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md b/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md index 94b5fbf03..167baa5e5 100644 --- a/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md +++ b/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md @@ -650,7 +650,7 @@ Walk components one at a time; never recursive-mkdir across an unvalidated paren **four** public entry points are covered — file and directory, sync and async — because the suite was twice found to be proving the wrong half: removing the async attribution alone left every test green while async is the path - `hardenStableLockFile` actually takes (`src/codex/native-main-lock-file.ts:130`), + `hardenStableLockFile` actually takes (`src/codex/native-main-lock-file.ts:148`), and a pathname-only memo applied to directories ALONE also left every test green while `hardenSecretDir` backs config, management-auth, tray, spill-store, and `native-profile-manager.ts:153`. @@ -834,16 +834,37 @@ is not enough. optional stricter timeout for Windows hardening: ```diff --export async function hardenStableLockFile(path: string): Promise { -+export async function hardenStableLockFile(path: string, timeoutMs?: number): Promise { - try { chmodSync(path, 0o600); } catch {} - if (process.platform === "win32") { -- await hardenSecretPathAsync(path, { required: true, timeoutMemoKey: path }); -+ await hardenSecretPathAsync(path, { required: true, timeoutMemoKey: path, timeoutMs }); + export async function hardenStableLockFile( + path: string, + platform: NodeJS.Platform = process.platform, ++ timeoutMs?: number, + ): Promise { + if (platform === "win32") { + try { chmodSync(path, 0o600); } catch { /* ACL below is authoritative. */ } +- await hardenSecretPathAsync(path, { required: true }); ++ await hardenSecretPathAsync(path, { required: true, timeoutMs }); + return; } + chmodSync(path, 0o600); } ``` +Written against the CURRENT signature, which differs from this section's original +sketch in three ways that later rounds forced and that the timeout addition must +preserve: + +- **`platform` is a parameter.** A direct `process.platform` read made the Windows + branch unreachable from a test on any other host, which is how an audit deleted + the whole delegation with 89 tests still green. +- **The `chmod` is no longer unconditional-and-swallowed.** On Windows it is + best-effort because the required ACL decides; on POSIX the mode IS the mechanism, + there is no fallback, and a failure must propagate. Swallowing it told the caller + that a pre-existing permissive coordinator database had been hardened when nothing + had changed — creation mode `0600` does not repair an existing file. +- **`timeoutMemoKey: path` was removed.** `timeoutMemoKey()` already falls back to + `targetPath`, so it was redundant and unobservable; a test pinning it would have + passed with the mechanism gone. + `windows-secret-acl.ts` clamps the caller value to the existing configured budget; it may shorten but never enlarge it. Existing callers that omit `timeoutMs` retain current behavior. Required ACL failure still rejects. @@ -995,8 +1016,9 @@ and the ones that did were rewritten rather than kept. the Windows-breaking form, since icacls moves ctime; (h) the async attribution alone is removed; (i) a pathname-only memo is applied to directories alone. (j) `hardenStableLockFile`'s Windows delegation is deleted; (k) its `required: - true` is weakened to `false`. - (h) through (k) are not redundant — each survived every other check. (h) and (i) + true` is weakened to `false`; (l) its POSIX `chmodSync` is deleted; (m) that + chmod failure is swallowed again; (n) the claim-site call to it is deleted. + (h) through (n) are not redundant — each survived every other check. (h) and (i) cover production callers the primitive tests missed: `hardenStableLockFile` takes the async path, and `hardenSecretDir` backs config, management-auth, tray, spill-store, and `native-profile-manager.ts:153`. (j) and (k) are a different @@ -1005,6 +1027,13 @@ and the ones that did were rewritten rather than kept. called it. `hardenStableLockFile` takes its platform as a parameter for exactly that reason — a direct `process.platform` read made the Windows branch unreachable from a test. + (l) and (m) are the POSIX side of the same wrapper, where the mode is the whole + mechanism: deleting `chmodSync` left 89 tests green because the only POSIX test + asserted that no ACL command ran, which was true of the working and the broken + version alike. (n) is one layer further out again — nearly every claim test injects + `hardenPath`, so the production default was never exercised, and the resolved + platform is now threaded into it so a forced-Windows claim reaches the real + delegation. Deliberately NOT asserted: the `timeoutMemoKey: path` argument at that call site. `timeoutMemoKey()` already falls back to `targetPath` and this caller passes the diff --git a/src/codex/native-main-claim.ts b/src/codex/native-main-claim.ts index 63096425e..0a722eb2e 100644 --- a/src/codex/native-main-claim.ts +++ b/src/codex/native-main-claim.ts @@ -78,7 +78,13 @@ async function openClaimDatabase( file = openStableLockFile(path, platform); const identity = `${file.dev}:${file.ino}`; if (hardenedIdentities.get(path) !== identity) { - await (options.hardenPath ?? hardenStableLockFile)(path); + // The resolved platform is threaded into the DEFAULT hardener, not left to + // `hardenStableLockFile`'s own `process.platform` read. Otherwise a test + // that forces `platform` here still exercises the host's branch, and the + // production default — the thing that actually hardens a coordinator + // database — stays unproved. An audit deleted this call entirely and 89 + // tests stayed green, because nearly every claim test injects `hardenPath`. + await (options.hardenPath ?? ((target: string) => hardenStableLockFile(target, platform)))(path); assertStableLockFile(path, file); hardenedIdentities.set(path, identity); } diff --git a/src/codex/native-main-lock-file.ts b/src/codex/native-main-lock-file.ts index 9e37cbefc..af22cda20 100644 --- a/src/codex/native-main-lock-file.ts +++ b/src/codex/native-main-lock-file.ts @@ -141,8 +141,18 @@ export async function hardenStableLockFile( path: string, platform: NodeJS.Platform = process.platform, ): Promise { - try { chmodSync(path, 0o600); } catch { /* Windows ACL below is authoritative there. */ } if (platform === "win32") { - await hardenSecretPathAsync(path, { required: true, timeoutMemoKey: path }); + // Best-effort here: POSIX modes are not authoritative on NTFS, and the + // required ACL hardening below is what actually decides. + try { chmodSync(path, 0o600); } catch { /* ACL below is authoritative. */ } + await hardenSecretPathAsync(path, { required: true }); + return; } + // On POSIX the mode IS the mechanism, so a failure may not be swallowed. + // + // The previous unconditional catch was a real fail-open: a coordinator + // database that already existed with permissive bits stayed permissive, and + // the caller was told the lock file had been hardened. Creation mode 0600 does + // not repair an existing file, and there is no ACL fallback outside Windows. + chmodSync(path, 0o600); } diff --git a/tests/native-main-claim.test.ts b/tests/native-main-claim.test.ts index d5c76f5b1..c92546213 100644 --- a/tests/native-main-claim.test.ts +++ b/tests/native-main-claim.test.ts @@ -1,5 +1,5 @@ import { afterEach, describe, expect, test } from "bun:test"; -import { mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { chmodSync, mkdirSync, mkdtempSync, readFileSync, rmSync, statSync, writeFileSync } from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; @@ -156,3 +156,31 @@ describe("native-main shared and exclusive claims", () => { await successor.release(); }); }); + +describe("the default hardener is actually reached from a claim", () => { + /** + * Nearly every test in this file injects `hardenPath: noHardening`, which is + * right for the behavior they are testing and wrong as a whole: an audit + * deleted the hardening call from `openClaimDatabase` and 89 tests across + * three files stayed green. The primitive had 77 tests; the edge that reaches + * it from a claim had none. + * + * So this one runs a claim with the DEFAULT hardener and inspects the file it + * left behind. On POSIX that is the mode; the Windows branch is proven + * separately in tests/windows-secret-acl.test.ts, where the ACL runner can be + * observed. + */ + test("a shared claim narrows a permissive claim database to 0600", async () => { + const context = fixture(); + const path = nativeMainClaimPath(context); + mkdirSync(join(context.codexHome), { recursive: true }); + writeFileSync(path, ""); + chmodSync(path, 0o644); + expect(statSync(path).mode & 0o777).toBe(0o644); + + // No hardenPath override: this is the production default. + await withNativeMainSharedClaim(context, async () => undefined); + + expect(statSync(path).mode & 0o777).toBe(0o600); + }); +}); diff --git a/tests/windows-secret-acl.test.ts b/tests/windows-secret-acl.test.ts index 0098064fb..a8aa93047 100644 --- a/tests/windows-secret-acl.test.ts +++ b/tests/windows-secret-acl.test.ts @@ -706,7 +706,7 @@ describe("ephemeral ACL memo release (#840 refinement)", () => { * async attribution entirely: all 46 tests stayed green. That is the worse half * to leave uncovered — `hardenStableLockFile`, which hardens the coordinator * database, calls `hardenSecretPathAsync` - * (`src/codex/native-main-lock-file.ts:130`), as do `config.ts:292` and + * (`src/codex/native-main-lock-file.ts:148`), as do `config.ts:292` and * `native-profile-manager.ts:153-154`. Covering only the synchronous twin proved * the path production does not take. */ @@ -1023,7 +1023,7 @@ describe("hardenStableLockFile — the production call edge, not just the primit * with the pathname as its timeout memo key, and its failure propagates rather * than leaving a coordinator database other accounts can read. */ - test("on win32 it delegates to the required async hardener, keyed by the path", async () => { + test("on win32 it delegates to the required async hardener for that exact path", async () => { resetHardenedStateForTests(); const lockPath = join(testDir, "coordinator-claim.sqlite"); writeFileSync(lockPath, "x", "utf8"); @@ -1042,12 +1042,12 @@ describe("hardenStableLockFile — the production call edge, not just the primit expect(seen.length).toBeGreaterThan(0); expect(seen.every(args => args[0] === lockPath)).toBe(true); expect(seen.some(args => args.includes("/grant:r"))).toBe(true); - // NOT asserted here: the `timeoutMemoKey: path` argument. It cannot be, - // because it is redundant — `timeoutMemoKey()` falls back to `targetPath`, - // and this caller passes the target path itself. Changing it in production - // is unobservable, so a test claiming to pin it would be vacuous. The - // option exists for atomic writers that mint a fresh temp per write and - // need the STABLE destination as the key (#612); this caller has no temp. + // The redundant `timeoutMemoKey: path` argument was REMOVED from this call + // site rather than asserted. `timeoutMemoKey()` already falls back to + // `targetPath`, so passing the target path had no observable effect and any + // test pinning it would have passed with the mechanism gone. The option + // exists for atomic writers that mint a fresh temp per write and need the + // stable destination as the key (#612); this caller has no temp. } finally { if (previousUsername === undefined) delete process.env.USERNAME; else process.env.USERNAME = previousUsername; @@ -1077,10 +1077,21 @@ describe("hardenStableLockFile — the production call edge, not just the primit } }); - test("on posix it hardens by mode alone and runs no ACL command", async () => { + /** + * On POSIX the mode IS the mechanism — there is no ACL fallback — so this + * asserts the resulting bits, not merely that no ACL command ran. + * + * Asserting absence alone is what let an audit delete `chmodSync` outright + * with 89 tests still green. An existing permissive database is not repaired + * by reopening it with creation mode 0600, so "no ACL command ran" was true of + * both the working and the broken implementation. + */ + test("on posix it narrows an existing permissive file to 0600 and runs no ACL command", async () => { resetHardenedStateForTests(); const lockPath = join(testDir, "coordinator-posix.sqlite"); writeFileSync(lockPath, "x", "utf8"); + chmodSync(lockPath, 0o644); + expect(statSync(lockPath).mode & 0o777).toBe(0o644); let calls = 0; setAsyncIcaclsRunnerForTests(async () => { @@ -1089,9 +1100,21 @@ describe("hardenStableLockFile — the production call edge, not just the primit }); try { await hardenStableLockFile(lockPath, "linux"); + expect(statSync(lockPath).mode & 0o777).toBe(0o600); expect(calls).toBe(0); } finally { setAsyncIcaclsRunnerForTests(null); } }); + + /** + * And a POSIX chmod failure may not be swallowed. It was, unconditionally, + * which told the caller a coordinator database had been hardened when its + * permissive bits were untouched and no ACL fallback exists off Windows. + */ + test("on posix a chmod failure propagates rather than reporting success", async () => { + resetHardenedStateForTests(); + const missing = join(testDir, "never-created.sqlite"); + await expect(hardenStableLockFile(missing, "linux")).rejects.toThrow(); + }); }); From 1b86f0e18a9608acc4e7cf1d265b4b0c748af27a Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Wed, 5 Aug 2026 09:45:35 +0900 Subject: [PATCH 111/163] fix(codex): the owner had the same untested default, and threading was asserted not proven MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two more, and they are the symmetric pair of last round's finding. The owner's default hardener was never exercised: replacing entry.options.hardenPath ?? hardenStableLockFile with a no-op left 91 tests green, exactly as the claim side had. Its options carry a resolved platform, so the default now threads it the same way. The subtler one is that threading a platform and PROVING it are different claims. Last round's claim test omitted platform, so its resolved value equalled the host's — it could not distinguish a threaded platform from a wrapper re-reading process.platform, and reverting the threading left everything green. Both sites are now covered by forcing platform: 'win32' from the outer API with no hardenPath and requiring the real ACL runner to run, which happens only if the default is called AND the platform reached it. Seventeen mutations now, all restored to 93 pass / 0 fail across three files. The three new: owner default replaced by a no-op (85/1), claim platform unthreaded (85/1), owner platform unthreaded (85/1). Also split wp12t0c rather than argue about its status. The implementation is done; the pinned-Bun Windows inode probe is its own pending task, because a task cannot be done while its own blocking activation gate is unmet. If Bun returns a zero inode on NTFS, the guard makes every required Windows harden fail closed — that is not a residual, it is a thing that must be run. And removed the last two places still describing timeoutMemoKey as tested at a call site it no longer appears in. --- .../030_lock_protocol.md | 25 +++++-- src/codex/native-main-owner.ts | 8 +- tests/windows-secret-acl.test.ts | 73 ++++++++++++++++++- 3 files changed, 96 insertions(+), 10 deletions(-) diff --git a/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md b/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md index 167baa5e5..e6cae027c 100644 --- a/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md +++ b/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md @@ -1017,8 +1017,10 @@ and the ones that did were rewritten rather than kept. alone is removed; (i) a pathname-only memo is applied to directories alone. (j) `hardenStableLockFile`'s Windows delegation is deleted; (k) its `required: true` is weakened to `false`; (l) its POSIX `chmodSync` is deleted; (m) that - chmod failure is swallowed again; (n) the claim-site call to it is deleted. - (h) through (n) are not redundant — each survived every other check. (h) and (i) + chmod failure is swallowed again; (n) the claim-site call to it is deleted; (o) the owner-site default is + replaced by a no-op; (p) the claim-site platform threading is removed; (q) the + owner-site platform threading is removed. + (h) through (q) are not redundant — each survived every other check. (h) and (i) cover production callers the primitive tests missed: `hardenStableLockFile` takes the async path, and `hardenSecretDir` backs config, management-auth, tray, spill-store, and `native-profile-manager.ts:153`. (j) and (k) are a different @@ -1034,12 +1036,19 @@ and the ones that did were rewritten rather than kept. `hardenPath`, so the production default was never exercised, and the resolved platform is now threaded into it so a forced-Windows claim reaches the real delegation. - - Deliberately NOT asserted: the `timeoutMemoKey: path` argument at that call site. - `timeoutMemoKey()` already falls back to `targetPath` and this caller passes the - target path, so changing it is unobservable and a test pinning it would be - vacuous. The option exists for atomic writers that mint a fresh temp per write - and need the stable destination as the key (#612); this caller has no temp. + (o) is the same hole on the owner side, found after the claim side was fixed: + replacing that default with a no-op left 91 tests green. (p) and (q) are the half + that threading alone does not prove — a test omitting `platform` cannot tell a + threaded platform from a wrapper re-reading `process.platform`, because on the + host they agree. Both are covered by forcing `platform: "win32"` from the outer + API with no `hardenPath` and requiring the real ACL runner to execute, which can + only happen if the default is called AND the platform reached it. + + The `timeoutMemoKey: path` argument was REMOVED from that call site rather than + asserted. `timeoutMemoKey()` already falls back to `targetPath` and the caller was + passing the target path, so it was unobservable and a test pinning it would have + been vacuous. The option remains for atomic writers that mint a fresh temp per + write and need the stable destination as the key (#612); this caller has no temp. Replacement is driven through the stat seam rather than a real unlink/recreate: ext4 recycles an inode immediately and APFS did not once in 200 cycles, so a real-file version asserts different things on different machines — which is how a diff --git a/src/codex/native-main-owner.ts b/src/codex/native-main-owner.ts index 4736ccbe8..70aed9cb6 100644 --- a/src/codex/native-main-owner.ts +++ b/src/codex/native-main-owner.ts @@ -144,7 +144,13 @@ async function prepareOwnerDatabase(entry: OwnerEntry): Promise { try { file = openStableLockFile(entry.lockPath, entry.options.platform); database = new Database(entry.lockPath, { create: true }); - await (entry.options.hardenPath ?? hardenStableLockFile)(entry.lockPath); + // The resolved platform is threaded into the DEFAULT hardener for the same + // reason the claim path does it: otherwise a test that forces `platform` + // here still exercises the host's branch, and the production default — the + // thing that actually hardens the owner's lock file — stays unproved. An + // audit replaced this fallback with a no-op and 91 tests stayed green. + await (entry.options.hardenPath + ?? ((target: string) => hardenStableLockFile(target, entry.options.platform)))(entry.lockPath); assertStableLockFile(entry.lockPath, file); entry.file = file; file = undefined; diff --git a/tests/windows-secret-acl.test.ts b/tests/windows-secret-acl.test.ts index a8aa93047..0ca31bf57 100644 --- a/tests/windows-secret-acl.test.ts +++ b/tests/windows-secret-acl.test.ts @@ -34,6 +34,8 @@ import { } from "../src/lib/windows-secret-acl"; import { atomicWriteFile } from "../src/config"; import { hardenStableLockFile } from "../src/codex/native-main-lock-file"; +import { withNativeMainSharedClaim } from "../src/codex/native-main-claim"; +import { retainNativeMainOwner } from "../src/codex/native-main-owner"; let testDir = ""; @@ -1020,7 +1022,7 @@ describe("hardenStableLockFile — the production call edge, not just the primit * test proved the primitive, and nothing proved production still called it. * * So this asserts the edge itself: on win32 the required async hardener runs - * with the pathname as its timeout memo key, and its failure propagates rather + * against that exact path, and its failure propagates rather * than leaving a coordinator database other accounts can read. */ test("on win32 it delegates to the required async hardener for that exact path", async () => { @@ -1118,3 +1120,72 @@ describe("hardenStableLockFile — the production call edge, not just the primit await expect(hardenStableLockFile(missing, "linux")).rejects.toThrow(); }); }); + +describe("the production default hardener is reached, with the resolved platform", () => { + /** + * The symmetric hole, found twice. + * + * `native-main-claim.ts` and `native-main-owner.ts` both default to + * `hardenStableLockFile` when no `hardenPath` is injected, and nearly every + * test in their own files injects one. An audit replaced each default with a + * no-op and 91 tests stayed green. + * + * Threading the resolved `platform` into that default is the other half. A + * test that omits `platform` cannot tell a threaded platform from a wrapper + * re-reading `process.platform`, because on this host they agree. So these + * force `platform: "win32"` from the OUTER API, inject no `hardenPath`, and + * require the real ACL runner to execute — which can only happen if the + * default is called AND the platform reached it. + */ + const forcedWindows = async ( + run: (codexHome: string) => Promise, + ): Promise => { + const seen: string[][] = []; + setPlatformForTests("win32"); + const previousUsername = process.env.USERNAME; + process.env.USERNAME = "ocx-test-user"; + setAsyncIcaclsRunnerForTests(async args => { + seen.push(args); + return { success: true, exitCode: 0, timedOut: false, stdout: "" }; + }); + const codexHome = mkdtempSync(join(tmpdir(), "ocx-default-harden-")); + try { + await run(codexHome); + } finally { + if (previousUsername === undefined) delete process.env.USERNAME; + else process.env.USERNAME = previousUsername; + setAsyncIcaclsRunnerForTests(null); + setPlatformForTests(null); + rmSync(codexHome, { recursive: true, force: true }); + } + return seen; + }; + + test("a claim with no hardenPath runs the real ACL sequence under a forced win32", async () => { + resetHardenedStateForTests(); + const seen = await forcedWindows(async codexHome => { + await withNativeMainSharedClaim( + { codexHome } as never, + async () => undefined, + { platform: "win32" }, + ); + }); + expect(seen.some(args => args.includes("/grant:r"))).toBe(true); + }); + + test("an owner with no hardenPath runs the real ACL sequence under a forced win32", async () => { + resetHardenedStateForTests(); + const seen = await forcedWindows(async codexHome => { + const owner = retainNativeMainOwner({ codexHome } as never, { platform: "win32", retryMs: 10 }); + try { + const deadline = Date.now() + 5_000; + while (owner.snapshot().status === "acquiring" && Date.now() < deadline) { + await Bun.sleep(10); + } + } finally { + await owner.release(); + } + }); + expect(seen.some(args => args.includes("/grant:r"))).toBe(true); + }); +}); From d0675aa0ce9b346a63f3be896db48e54af6ca30f Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Wed, 5 Aug 2026 09:53:28 +0900 Subject: [PATCH 112/163] test(codex): proving the hardener runs is not proving its failure matters MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Appending .catch(() => {}) to either production call site left 93 tests green. The forced-Windows tests added last round observe a successful invocation, and the wrapper's own failure test cannot see a caller swallowing the rejection — so required:true was decorative at both callers. That is the entire point of required:true. On Windows the ACL is the only thing keeping other accounts out of a coordinator database, so one whose ACL could not be applied must not go on to be used. A claim must never run its operation; an owner must never report held. Both are now asserted with a failing runner. Nineteen mutations, all restored to 95 pass / 0 fail across three files. The two new: claim swallows a required hardening failure (80/1), owner swallows it (80/1). Also put the NTFS activation gate into the plan document, not only the goalplan and a source comment. observe() treats a zero inode as unobservable, so if Bun returns zero on NTFS every required Windows harden fails closed and this whole surface is inert in the one place it exists for. A document that can be read as complete while that is unprobed is a document that will be. --- .../030_lock_protocol.md | 21 ++++++- tests/windows-secret-acl.test.ts | 63 +++++++++++++++++++ 2 files changed, 82 insertions(+), 2 deletions(-) diff --git a/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md b/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md index e6cae027c..71b8fd4ab 100644 --- a/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md +++ b/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md @@ -1019,8 +1019,9 @@ and the ones that did were rewritten rather than kept. true` is weakened to `false`; (l) its POSIX `chmodSync` is deleted; (m) that chmod failure is swallowed again; (n) the claim-site call to it is deleted; (o) the owner-site default is replaced by a no-op; (p) the claim-site platform threading is removed; (q) the - owner-site platform threading is removed. - (h) through (q) are not redundant — each survived every other check. (h) and (i) + owner-site platform threading is removed; (r) the claim site swallows a required + hardening failure; (s) the owner site swallows it. + (h) through (s) are not redundant — each survived every other check. (h) and (i) cover production callers the primitive tests missed: `hardenStableLockFile` takes the async path, and `hardenSecretDir` backs config, management-auth, tray, spill-store, and `native-profile-manager.ts:153`. (j) and (k) are a different @@ -1043,6 +1044,22 @@ and the ones that did were rewritten rather than kept. host they agree. Both are covered by forcing `platform: "win32"` from the outer API with no `hardenPath` and requiring the real ACL runner to execute, which can only happen if the default is called AND the platform reached it. + (r) and (s) are the third distinct claim about the same two call sites: proving + the hardener RUNS is not proving its FAILURE matters. Appending `.catch(() => {})` + at either site left 93 tests green, because the forced-Windows tests observe a + successful invocation and the wrapper's own failure test cannot see a caller + swallowing the rejection. That is the whole point of `required: true` — on Windows + the ACL is the only thing keeping other accounts out of a coordinator database, so + one whose ACL could not be applied must not go on to be used. A claim must never + run its operation and an owner must never report `held`. + + **Activation gate, not a test:** the NTFS `bigint` inode behaviour is UNVERIFIED. + `observe()` treats a zero inode as unobservable, so if Bun returns zero there, + every REQUIRED Windows harden fails closed and this whole surface is inert in the + one place it exists for. A pinned-Bun probe on real Windows/NTFS must confirm a + nonzero, stable file index before F4 is called complete (goalplan `wp12t0c2`). + This is recorded here as well as in the goalplan so this document cannot be read + as complete on its own. The `timeoutMemoKey: path` argument was REMOVED from that call site rather than asserted. `timeoutMemoKey()` already falls back to `targetPath` and the caller was diff --git a/tests/windows-secret-acl.test.ts b/tests/windows-secret-acl.test.ts index 0ca31bf57..1ac8c373b 100644 --- a/tests/windows-secret-acl.test.ts +++ b/tests/windows-secret-acl.test.ts @@ -1189,3 +1189,66 @@ describe("the production default hardener is reached, with the resolved platform expect(seen.some(args => args.includes("/grant:r"))).toBe(true); }); }); + +describe("a required hardening failure stops the operation it protects", () => { + /** + * Proving the default hardener RUNS is not proving its failure matters. + * + * Appending `.catch(() => {})` at either call site left 93 tests green: the + * forced-Windows tests above observe a successful invocation, and the + * wrapper's own failure test cannot see a caller swallowing the rejection. + * + * What is at stake is the whole point of `required: true` — a coordinator + * database whose ACL could not be applied must not go on to be used, because + * on Windows the ACL is the only thing keeping other accounts out of it. + */ + const forcedWindowsFailure = async ( + run: (codexHome: string) => Promise, + ): Promise => { + setPlatformForTests("win32"); + const previousUsername = process.env.USERNAME; + process.env.USERNAME = "ocx-test-user"; + setAsyncIcaclsRunnerForTests(async () => ({ + success: false, exitCode: 5, timedOut: false, stdout: "", stderr: "", + })); + const codexHome = mkdtempSync(join(tmpdir(), "ocx-harden-fail-")); + try { + await run(codexHome); + } finally { + if (previousUsername === undefined) delete process.env.USERNAME; + else process.env.USERNAME = previousUsername; + setAsyncIcaclsRunnerForTests(null); + setPlatformForTests(null); + rmSync(codexHome, { recursive: true, force: true }); + } + }; + + test("a claim whose ACL cannot be applied never runs its operation", async () => { + resetHardenedStateForTests(); + let operationRan = false; + await forcedWindowsFailure(async codexHome => { + await expect(withNativeMainSharedClaim( + { codexHome } as never, + async () => { operationRan = true; }, + { platform: "win32" }, + )).rejects.toThrow(); + }); + expect(operationRan).toBe(false); + }); + + test("an owner whose ACL cannot be applied never reports held", async () => { + resetHardenedStateForTests(); + await forcedWindowsFailure(async codexHome => { + const owner = retainNativeMainOwner({ codexHome } as never, { platform: "win32", retryMs: 10 }); + try { + const deadline = Date.now() + 5_000; + while (owner.snapshot().status === "acquiring" && Date.now() < deadline) { + await Bun.sleep(10); + } + expect(owner.snapshot().status).not.toBe("held"); + } finally { + await owner.release(); + } + }); + }); +}); From 955dd806fe1311438d33b5a427d1fd5239d314a1 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Wed, 5 Aug 2026 10:00:43 +0900 Subject: [PATCH 113/163] test(codex): a denied ACL is a refusal, not contention, and 'it rejected' cannot tell MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Broadening isBusy() so an ACL denial matches left 95 tests green at both callers. The claim test accepted any rejection, so NATIVE_MAIN_CLAIM_BUSY passed where UNAVAILABLE was required. The owner test accepted any state except held, so contended passed — and contended schedules a retry, which turns a permanent denial into an endless reacquire loop wearing the costume of contention. The claim now asserts the exact code, and the owner asserts {status: 'unavailable', reason: 'lock-unavailable'} and that it is still there after the retry interval would have fired. Twenty-one mutations, all restored to 95 pass / 0 fail across three files. The two new: claim reclassifies a denied ACL as busy (80/1), owner reclassifies it as contended (80/1). This is the fourth distinct claim about the same two call sites — the default is reached, the resolved platform reaches it, its failure stops the operation, and that failure is classified as permanent. Each of the first three passed while the next one was broken. --- .../030_lock_protocol.md | 15 ++++++-- tests/windows-secret-acl.test.ts | 34 +++++++++++++++---- 2 files changed, 40 insertions(+), 9 deletions(-) diff --git a/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md b/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md index 71b8fd4ab..1f390f0b9 100644 --- a/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md +++ b/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md @@ -1020,8 +1020,9 @@ and the ones that did were rewritten rather than kept. chmod failure is swallowed again; (n) the claim-site call to it is deleted; (o) the owner-site default is replaced by a no-op; (p) the claim-site platform threading is removed; (q) the owner-site platform threading is removed; (r) the claim site swallows a required - hardening failure; (s) the owner site swallows it. - (h) through (s) are not redundant — each survived every other check. (h) and (i) + hardening failure; (s) the owner site swallows it; (t) the claim reclassifies a + denied ACL as busy; (u) the owner reclassifies it as contended. + (h) through (u) are not redundant — each survived every other check. (h) and (i) cover production callers the primitive tests missed: `hardenStableLockFile` takes the async path, and `hardenSecretDir` backs config, management-auth, tray, spill-store, and `native-profile-manager.ts:153`. (j) and (k) are a different @@ -1052,6 +1053,16 @@ and the ones that did were rewritten rather than kept. the ACL is the only thing keeping other accounts out of a coordinator database, so one whose ACL could not be applied must not go on to be used. A claim must never run its operation and an owner must never report `held`. + (t) and (u) are the fourth claim about those same sites, and the one that + "it rejected" cannot see: a denied ACL must be a **non-retryable refusal**. + Broadening `isBusy()` to match the ACL message left 95 tests green, because the + claim test accepted any rejection — including `NATIVE_MAIN_CLAIM_BUSY` — and the + owner test accepted any state except `held`, including `contended`, which + schedules a retry (`src/codex/native-main-owner.ts:194`). A permanent denial that + enters the retry scheduler is an endless reacquire loop wearing the costume of + contention. The claim must reject with `NATIVE_MAIN_CLAIM_UNAVAILABLE`; the owner + must settle at `{ status: "unavailable", reason: "lock-unavailable" }` and stay + there. **Activation gate, not a test:** the NTFS `bigint` inode behaviour is UNVERIFIED. `observe()` treats a zero inode as unobservable, so if Bun returns zero there, diff --git a/tests/windows-secret-acl.test.ts b/tests/windows-secret-acl.test.ts index 1ac8c373b..fde2a512a 100644 --- a/tests/windows-secret-acl.test.ts +++ b/tests/windows-secret-acl.test.ts @@ -1227,16 +1227,27 @@ describe("a required hardening failure stops the operation it protects", () => { resetHardenedStateForTests(); let operationRan = false; await forcedWindowsFailure(async codexHome => { - await expect(withNativeMainSharedClaim( - { codexHome } as never, - async () => { operationRan = true; }, - { platform: "win32" }, - )).rejects.toThrow(); + // The exact code matters, not merely that it threw. A denied ACL is a + // permanent refusal; classifying it as BUSY would send a caller back to + // retry something that will fail identically forever. Broadening isBusy() + // to match the ACL message passed a test that only asserted "rejects". + let code: string | undefined; + try { + await withNativeMainSharedClaim( + { codexHome } as never, + async () => { operationRan = true; }, + { platform: "win32" }, + ); + throw new Error("expected the claim to refuse"); + } catch (error) { + code = (error as { code?: string }).code; + } + expect(code).toBe("NATIVE_MAIN_CLAIM_UNAVAILABLE"); }); expect(operationRan).toBe(false); }); - test("an owner whose ACL cannot be applied never reports held", async () => { + test("an owner whose ACL cannot be applied settles unavailable, and does not retry", async () => { resetHardenedStateForTests(); await forcedWindowsFailure(async codexHome => { const owner = retainNativeMainOwner({ codexHome } as never, { platform: "win32", retryMs: 10 }); @@ -1245,7 +1256,16 @@ describe("a required hardening failure stops the operation it protects", () => { while (owner.snapshot().status === "acquiring" && Date.now() < deadline) { await Bun.sleep(10); } - expect(owner.snapshot().status).not.toBe("held"); + // `contended` would be wrong in a way `not.toBe("held")` cannot see: it + // schedules a retry, so a permanently denied ACL becomes an endless + // reacquire loop instead of a settled refusal. + expect(owner.snapshot()).toMatchObject({ + status: "unavailable", + reason: "lock-unavailable", + }); + // And it stays settled: a retry would move it off `unavailable`. + await Bun.sleep(60); + expect(owner.snapshot().status).toBe("unavailable"); } finally { await owner.release(); } From f3162b495b9d84fe59e70f3a2ccc00dc3f84ec6d Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Wed, 5 Aug 2026 10:07:38 +0900 Subject: [PATCH 114/163] test(codex): a terminal snapshot cannot prove something never happened MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Publishing held and then immediately unavailable left 95 tests green. Both snapshot reads saw unavailable while every subscriber saw a caller briefly believe it owned a database whose ACL had just been refused. The owner failure path is now an ordered trace: subscribe first, record every published state, require acquiring -> unavailable, and forbid held and contended at any point. That is the generalization, and it is worth more than the mutation. Rounds 10 through 18 each produced exactly one finding here, and they were never independent: each test proved ONE projection of a conjunctive contract — some hardener ran, the platform arrived, the operation stopped, the classification was right — and the next mutation kept the asserted projection while breaking an unobserved one. Final-state assertions are the worst offender, because 'never happened' is not a property of a final state. The rule is now written into the plan: for every production call edge assert the exact target, default binding, platform provenance, success ordering, failure propagation, refusal taxonomy, complete observable trace, and absence of retries or protected-operation execution. For an async state machine, assert an ordered trace and forbidden events, never only the eventual state. Twenty-two mutations, all restored to 95 pass / 0 fail across three files. wp12t0c moves to done; wp12t0c2, the pinned-Bun NTFS probe, stays pending. --- .../030_lock_protocol.md | 26 ++++++++++++-- tests/windows-secret-acl.test.ts | 35 +++++++++++++++---- 2 files changed, 52 insertions(+), 9 deletions(-) diff --git a/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md b/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md index 1f390f0b9..bea3a68ab 100644 --- a/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md +++ b/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md @@ -1021,8 +1021,9 @@ and the ones that did were rewritten rather than kept. replaced by a no-op; (p) the claim-site platform threading is removed; (q) the owner-site platform threading is removed; (r) the claim site swallows a required hardening failure; (s) the owner site swallows it; (t) the claim reclassifies a - denied ACL as busy; (u) the owner reclassifies it as contended. - (h) through (u) are not redundant — each survived every other check. (h) and (i) + denied ACL as busy; (u) the owner reclassifies it as contended; (v) the owner publishes a transient + `held` before settling `unavailable`. + (h) through (v) are not redundant — each survived every other check. (h) and (i) cover production callers the primitive tests missed: `hardenStableLockFile` takes the async path, and `hardenSecretDir` backs config, management-auth, tray, spill-store, and `native-profile-manager.ts:153`. (j) and (k) are a different @@ -1064,6 +1065,27 @@ and the ones that did were rewritten rather than kept. must settle at `{ status: "unavailable", reason: "lock-unavailable" }` and stay there. + ### The class, not the twenty-second instance + + Rounds 10-18 each produced exactly one finding in this surface, and (v) is what + named the pattern: these were never independent accidents. Each test proved ONE + projection of a conjunctive contract — some hardener ran, the platform arrived, + the operation stopped, the classification was right — and a later mutation kept + the asserted projection while breaking an unobserved one. + + **A terminal snapshot cannot prove that something never happened.** (v) publishes + `held` and then immediately `unavailable`: both reads see `unavailable`, while + every subscriber saw a caller briefly believe it owned an unhardened database. + The test now subscribes first and asserts the ordered trace + `acquiring -> unavailable` with `held` and `contended` forbidden at any point. + + So the rule for every production call edge in this unit, and the reason the list + above is a matrix rather than a list: assert the exact target, the default + binding, the platform provenance, the success ordering, the failure propagation, + the exact refusal taxonomy, the complete observable state trace, and the absence + of retries or protected-operation execution. For an asynchronous state machine, + assert an ordered trace and forbidden events — never only the eventual state. + **Activation gate, not a test:** the NTFS `bigint` inode behaviour is UNVERIFIED. `observe()` treats a zero inode as unobservable, so if Bun returns zero there, every REQUIRED Windows harden fails closed and this whole surface is inert in the diff --git a/tests/windows-secret-acl.test.ts b/tests/windows-secret-acl.test.ts index fde2a512a..e1eda547e 100644 --- a/tests/windows-secret-acl.test.ts +++ b/tests/windows-secret-acl.test.ts @@ -1247,26 +1247,47 @@ describe("a required hardening failure stops the operation it protects", () => { expect(operationRan).toBe(false); }); - test("an owner whose ACL cannot be applied settles unavailable, and does not retry", async () => { + /** + * Asserted as an ORDERED TRACE, not a terminal snapshot. + * + * This is the generalization that should have come eight rounds earlier. Every + * previous version of this test read `owner.snapshot()` after the fact, and a + * final snapshot structurally cannot prove that something never happened: an + * implementation publishing `held` and then immediately `unavailable` passed, + * while every subscriber saw the forbidden intermediate state. + * + * So subscribe first, record every published state, and assert the whole + * sequence plus the events that must be absent. + */ + test("an owner whose ACL cannot be applied goes acquiring -> unavailable, with no held, contended, or retry", async () => { resetHardenedStateForTests(); await forcedWindowsFailure(async codexHome => { const owner = retainNativeMainOwner({ codexHome } as never, { platform: "win32", retryMs: 10 }); + const trace: string[] = []; + const unsubscribe = owner.subscribe(snapshot => { trace.push(snapshot.status); }); try { const deadline = Date.now() + 5_000; while (owner.snapshot().status === "acquiring" && Date.now() < deadline) { await Bun.sleep(10); } - // `contended` would be wrong in a way `not.toBe("held")` cannot see: it - // schedules a retry, so a permanently denied ACL becomes an endless - // reacquire loop instead of a settled refusal. + // Long enough for a scheduled retry (retryMs: 10) to have fired. + await Bun.sleep(60); + expect(owner.snapshot()).toMatchObject({ status: "unavailable", reason: "lock-unavailable", }); - // And it stays settled: a retry would move it off `unavailable`. - await Bun.sleep(60); - expect(owner.snapshot().status).toBe("unavailable"); + // `held` would mean a caller briefly believed it owned an unhardened + // database; `contended` schedules a retry, turning a permanent denial + // into an endless reacquire loop. Neither may appear at any point. + expect(trace).not.toContain("held"); + expect(trace).not.toContain("contended"); + // The whole observable history, in order, with no retry churn after it + // settles. `acquiring` is published to subscribers on attach. + expect(trace.filter((status, i) => status !== trace[i - 1])) + .toEqual(["acquiring", "unavailable"]); } finally { + unsubscribe(); await owner.release(); } }); From 53f0f81aebee8ca641ba742909884c814ab67a34 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Wed, 5 Aug 2026 10:14:02 +0900 Subject: [PATCH 115/163] test(codex): I wrote the rule and then broke it in the same commit MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The trace assertion collapsed consecutive duplicates before comparing. A retry loop republishing unavailable normalizes to exactly the same two entries, so scheduling a retry straight out of a permanent ACL refusal left 95 tests green — while the test claimed both a complete observable trace and an absence of retries, and discarded the evidence of both. That is the class I had just finished describing, applied to the fix for it. The normalization WAS the projection. The trace is now raw and compared exactly, and the ACL attempts are counted: exactly one, because 'attempted once' is a count and a count has to be counted. The corollary is in the plan: never normalize, deduplicate, or sort an event trace unless the normalization is part of the production contract. Also recorded where a trace does not exist. withNativeMainSharedClaim publishes no intermediate state, so its failure contract is the exact rejection code, the operation never running, and the right database being targeted. Adding a subscribe API purely so a test could assert a trace would be machinery that strengthens no public contract — worth writing down so the asymmetry reads as a decision rather than an oversight. Twenty-three mutations, all restored to 95 pass / 0 fail across three files. --- .../030_lock_protocol.md | 21 ++++++++++++++-- tests/windows-secret-acl.test.ts | 25 +++++++++++++------ 2 files changed, 36 insertions(+), 10 deletions(-) diff --git a/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md b/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md index bea3a68ab..24e042134 100644 --- a/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md +++ b/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md @@ -1022,8 +1022,9 @@ and the ones that did were rewritten rather than kept. owner-site platform threading is removed; (r) the claim site swallows a required hardening failure; (s) the owner site swallows it; (t) the claim reclassifies a denied ACL as busy; (u) the owner reclassifies it as contended; (v) the owner publishes a transient - `held` before settling `unavailable`. - (h) through (v) are not redundant — each survived every other check. (h) and (i) + `held` before settling `unavailable`; (w) the owner schedules a retry from that + permanent refusal, republishing `unavailable` each time. + (h) through (w) are not redundant — each survived every other check. (h) and (i) cover production callers the primitive tests missed: `hardenStableLockFile` takes the async path, and `hardenSecretDir` backs config, management-auth, tray, spill-store, and `native-profile-manager.ts:153`. (j) and (k) are a different @@ -1086,6 +1087,22 @@ and the ones that did were rewritten rather than kept. of retries or protected-operation execution. For an asynchronous state machine, assert an ordered trace and forbidden events — never only the eventual state. + **Corollary, learned by breaking the rule in the act of writing it.** The first + trace assertion collapsed consecutive duplicates before comparing. A retry loop + republishing `unavailable` normalizes to exactly the same two entries, so the test + claimed a complete trace and an absence of retries while discarding the evidence + of both — mutation (w). Never normalize, deduplicate, sort, or otherwise project + an event trace unless that normalization is itself part of the production + contract. And where a count is the claim, count it: the ACL attempt counter is + what makes "attempted exactly once" executable rather than asserted. + + **Where a trace does not exist, say so rather than inventing one.** + `withNativeMainSharedClaim` publishes no intermediate ownership state, so its + observable failure contract is the exact `NATIVE_MAIN_CLAIM_UNAVAILABLE` + rejection, the protected operation never running, and the hardener targeting the + right database. Adding a subscribe API solely so a test could assert a trace would + be machinery that strengthens no public contract. + **Activation gate, not a test:** the NTFS `bigint` inode behaviour is UNVERIFIED. `observe()` treats a zero inode as unobservable, so if Bun returns zero there, every REQUIRED Windows harden fails closed and this whole surface is inert in the diff --git a/tests/windows-secret-acl.test.ts b/tests/windows-secret-acl.test.ts index e1eda547e..214cfc8be 100644 --- a/tests/windows-secret-acl.test.ts +++ b/tests/windows-secret-acl.test.ts @@ -1204,13 +1204,17 @@ describe("a required hardening failure stops the operation it protects", () => { */ const forcedWindowsFailure = async ( run: (codexHome: string) => Promise, + onAttempt: () => void = () => {}, ): Promise => { setPlatformForTests("win32"); const previousUsername = process.env.USERNAME; process.env.USERNAME = "ocx-test-user"; - setAsyncIcaclsRunnerForTests(async () => ({ - success: false, exitCode: 5, timedOut: false, stdout: "", stderr: "", - })); + setAsyncIcaclsRunnerForTests(async args => { + // Count only the first step of each sequence, so the counter is attempts + // rather than icacls invocations. + if (args.includes("/grant:r")) onAttempt(); + return { success: false, exitCode: 5, timedOut: false, stdout: "", stderr: "" }; + }); const codexHome = mkdtempSync(join(tmpdir(), "ocx-harden-fail-")); try { await run(codexHome); @@ -1261,6 +1265,7 @@ describe("a required hardening failure stops the operation it protects", () => { */ test("an owner whose ACL cannot be applied goes acquiring -> unavailable, with no held, contended, or retry", async () => { resetHardenedStateForTests(); + let hardenAttempts = 0; await forcedWindowsFailure(async codexHome => { const owner = retainNativeMainOwner({ codexHome } as never, { platform: "win32", retryMs: 10 }); const trace: string[] = []; @@ -1282,14 +1287,18 @@ describe("a required hardening failure stops the operation it protects", () => { // into an endless reacquire loop. Neither may appear at any point. expect(trace).not.toContain("held"); expect(trace).not.toContain("contended"); - // The whole observable history, in order, with no retry churn after it - // settles. `acquiring` is published to subscribers on attach. - expect(trace.filter((status, i) => status !== trace[i - 1])) - .toEqual(["acquiring", "unavailable"]); + // The RAW history, undeduplicated. Collapsing consecutive duplicates was + // itself the defect the previous version shipped: a retry loop + // republishing `unavailable` normalizes to the same two entries, so the + // test claimed "no retry churn" while discarding the churn. Never project + // an event trace unless the projection is part of the contract. + expect(trace).toEqual(["acquiring", "unavailable"]); + // And the ACL was attempted exactly once: a hidden retry would harden again. + expect(hardenAttempts).toBe(1); } finally { unsubscribe(); await owner.release(); } - }); + }, () => { hardenAttempts += 1; }); }); }); From 04f7ae4fc7bc92f17b05c3e7522fa935b672023d Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Wed, 5 Aug 2026 10:20:29 +0900 Subject: [PATCH 116/163] test(codex): the plan said assert the exact target; the tests asserted 'some ACL ran' MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both callers were redirected at an unrelated existing file — package.json — and all 95 tests stayed green. Every other dimension held: the default was reached, the platform arrived, the failure propagated, the classification was right, the trace was exact. Nothing checked args[0]. The matrix rule I wrote two commits ago already said 'assert the exact target'. Writing a matrix into a plan does not execute it, which is the same lesson as the previous two rounds arriving from a third direction. Both call edges now assert the target in the success AND failure tests, against nativeMainClaimPath(context) and join(codexHome, NATIVE_MAIN_OWNER_DB), and the failure tests collect targets alongside the attempt count so 'exactly one attempt, against exactly this database' is one assertion rather than two independent hopes. Twenty-five mutations, all restored to 95 pass / 0 fail across three files. The two new: owner hardens a different file (79/2), claim hardens a different file (79/2). --- .../030_lock_protocol.md | 14 +++++++-- tests/windows-secret-acl.test.ts | 29 +++++++++++++++---- 2 files changed, 35 insertions(+), 8 deletions(-) diff --git a/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md b/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md index 24e042134..9e1ef146d 100644 --- a/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md +++ b/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md @@ -1023,8 +1023,9 @@ and the ones that did were rewritten rather than kept. hardening failure; (s) the owner site swallows it; (t) the claim reclassifies a denied ACL as busy; (u) the owner reclassifies it as contended; (v) the owner publishes a transient `held` before settling `unavailable`; (w) the owner schedules a retry from that - permanent refusal, republishing `unavailable` each time. - (h) through (w) are not redundant — each survived every other check. (h) and (i) + permanent refusal, republishing `unavailable` each time; (x) the owner hardens a + different existing file; (y) the claim hardens a different existing file. + (h) through (y) are not redundant — each survived every other check. (h) and (i) cover production callers the primitive tests missed: `hardenStableLockFile` takes the async path, and `hardenSecretDir` backs config, management-auth, tray, spill-store, and `native-profile-manager.ts:153`. (j) and (k) are a different @@ -1103,6 +1104,15 @@ and the ones that did were rewritten rather than kept. right database. Adding a subscribe API solely so a test could assert a trace would be machinery that strengthens no public contract. + **And the matrix has to be assertions, not prose.** (x) and (y) are the proof: + the rule above already said "assert the exact target", and both callers were then + redirected at an unrelated existing file with every test still green — because + each one asserted only that *some* `/grant:r` happened. Writing a matrix into a + plan does not execute it. Every dimension named here is now a concrete assertion + at both call edges: `expect(seen.every(args => args[0] === expected)).toBe(true)` + against `nativeMainClaimPath(context)` and `join(codexHome, NATIVE_MAIN_OWNER_DB)`, + in the success and the failure test alike. + **Activation gate, not a test:** the NTFS `bigint` inode behaviour is UNVERIFIED. `observe()` treats a zero inode as unobservable, so if Bun returns zero there, every REQUIRED Windows harden fails closed and this whole surface is inert in the diff --git a/tests/windows-secret-acl.test.ts b/tests/windows-secret-acl.test.ts index 214cfc8be..5e3d37bfa 100644 --- a/tests/windows-secret-acl.test.ts +++ b/tests/windows-secret-acl.test.ts @@ -34,8 +34,8 @@ import { } from "../src/lib/windows-secret-acl"; import { atomicWriteFile } from "../src/config"; import { hardenStableLockFile } from "../src/codex/native-main-lock-file"; -import { withNativeMainSharedClaim } from "../src/codex/native-main-claim"; -import { retainNativeMainOwner } from "../src/codex/native-main-owner"; +import { nativeMainClaimPath, withNativeMainSharedClaim } from "../src/codex/native-main-claim"; +import { NATIVE_MAIN_OWNER_DB, retainNativeMainOwner } from "../src/codex/native-main-owner"; let testDir = ""; @@ -1163,7 +1163,9 @@ describe("the production default hardener is reached, with the resolved platform test("a claim with no hardenPath runs the real ACL sequence under a forced win32", async () => { resetHardenedStateForTests(); + let expected = ""; const seen = await forcedWindows(async codexHome => { + expected = nativeMainClaimPath({ codexHome } as never); await withNativeMainSharedClaim( { codexHome } as never, async () => undefined, @@ -1171,11 +1173,16 @@ describe("the production default hardener is reached, with the resolved platform ); }); expect(seen.some(args => args.includes("/grant:r"))).toBe(true); + // The EXACT target, not merely that some ACL ran. Redirecting the caller at + // an unrelated existing file passed every other dimension of this test. + expect(seen.every(args => args[0] === expected)).toBe(true); }); test("an owner with no hardenPath runs the real ACL sequence under a forced win32", async () => { resetHardenedStateForTests(); + let expected = ""; const seen = await forcedWindows(async codexHome => { + expected = join(codexHome, NATIVE_MAIN_OWNER_DB); const owner = retainNativeMainOwner({ codexHome } as never, { platform: "win32", retryMs: 10 }); try { const deadline = Date.now() + 5_000; @@ -1187,6 +1194,7 @@ describe("the production default hardener is reached, with the resolved platform } }); expect(seen.some(args => args.includes("/grant:r"))).toBe(true); + expect(seen.every(args => args[0] === expected)).toBe(true); }); }); @@ -1204,7 +1212,7 @@ describe("a required hardening failure stops the operation it protects", () => { */ const forcedWindowsFailure = async ( run: (codexHome: string) => Promise, - onAttempt: () => void = () => {}, + onAttempt: (args: string[]) => void = () => {}, ): Promise => { setPlatformForTests("win32"); const previousUsername = process.env.USERNAME; @@ -1212,7 +1220,7 @@ describe("a required hardening failure stops the operation it protects", () => { setAsyncIcaclsRunnerForTests(async args => { // Count only the first step of each sequence, so the counter is attempts // rather than icacls invocations. - if (args.includes("/grant:r")) onAttempt(); + if (args.includes("/grant:r")) onAttempt(args); return { success: false, exitCode: 5, timedOut: false, stdout: "", stderr: "" }; }); const codexHome = mkdtempSync(join(tmpdir(), "ocx-harden-fail-")); @@ -1230,7 +1238,10 @@ describe("a required hardening failure stops the operation it protects", () => { test("a claim whose ACL cannot be applied never runs its operation", async () => { resetHardenedStateForTests(); let operationRan = false; + const targets: string[] = []; + let expected = ""; await forcedWindowsFailure(async codexHome => { + expected = nativeMainClaimPath({ codexHome } as never); // The exact code matters, not merely that it threw. A denied ACL is a // permanent refusal; classifying it as BUSY would send a caller back to // retry something that will fail identically forever. Broadening isBusy() @@ -1247,8 +1258,9 @@ describe("a required hardening failure stops the operation it protects", () => { code = (error as { code?: string }).code; } expect(code).toBe("NATIVE_MAIN_CLAIM_UNAVAILABLE"); - }); + }, args => { targets.push(args[0]!); }); expect(operationRan).toBe(false); + expect(targets).toEqual([expected]); }); /** @@ -1266,7 +1278,10 @@ describe("a required hardening failure stops the operation it protects", () => { test("an owner whose ACL cannot be applied goes acquiring -> unavailable, with no held, contended, or retry", async () => { resetHardenedStateForTests(); let hardenAttempts = 0; + const targets: string[] = []; + let expected = ""; await forcedWindowsFailure(async codexHome => { + expected = join(codexHome, NATIVE_MAIN_OWNER_DB); const owner = retainNativeMainOwner({ codexHome } as never, { platform: "win32", retryMs: 10 }); const trace: string[] = []; const unsubscribe = owner.subscribe(snapshot => { trace.push(snapshot.status); }); @@ -1295,10 +1310,12 @@ describe("a required hardening failure stops the operation it protects", () => { expect(trace).toEqual(["acquiring", "unavailable"]); // And the ACL was attempted exactly once: a hidden retry would harden again. expect(hardenAttempts).toBe(1); + // Exactly one attempt, against exactly the owner's own database. + expect(targets).toEqual([expected]); } finally { unsubscribe(); await owner.release(); } - }, () => { hardenAttempts += 1; }); + }, args => { hardenAttempts += 1; targets.push(args[0]!); }); }); }); From a8b5b64266204967a7fdb20e00317217a5629235 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Wed, 5 Aug 2026 10:30:48 +0900 Subject: [PATCH 117/163] test(codex): the row that stayed empty longest was the successful one MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Every test on these call edges proved a failure path or an invocation. None required the operation to actually succeed. So all three of these passed: an owner that hardens correctly and then fails to acquire, an owner that never publishes held at all, and a claim that hardens and then silently skips its protected operation. The claim success test now requires the operation to run, to return its value, and to run AFTER the ACL — order recorded, not inferred. The owner success test uses a deferred ACL runner so it can assert the trace is exactly ['acquiring'] while hardening is still in flight and exactly ['acquiring', 'held'] once it completes, which is the only way to show nothing was published early. Twenty-eight mutations, all restored to 95 pass / 0 fail across three files. One process note worth recording: the first attempt at the owner test did not apply — a scripted edit silently missed its anchor — and I read the green suite as confirmation and moved on. The mutation check is what caught it: #26 kept passing when it should not have. A patch that does not land looks exactly like a patch that landed and works, unless something independent disagrees. --- .../030_lock_protocol.md | 22 ++++++++++- tests/windows-secret-acl.test.ts | 39 +++++++++++++++++-- 2 files changed, 55 insertions(+), 6 deletions(-) diff --git a/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md b/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md index 9e1ef146d..876b8910c 100644 --- a/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md +++ b/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md @@ -1024,8 +1024,10 @@ and the ones that did were rewritten rather than kept. denied ACL as busy; (u) the owner reclassifies it as contended; (v) the owner publishes a transient `held` before settling `unavailable`; (w) the owner schedules a retry from that permanent refusal, republishing `unavailable` each time; (x) the owner hardens a - different existing file; (y) the claim hardens a different existing file. - (h) through (y) are not redundant — each survived every other check. (h) and (i) + different existing file; (y) the claim hardens a different existing file; (z) the owner hardens correctly and + then fails to acquire; (aa) the owner never publishes `held`; (ab) the claim + hardens and then silently skips its protected operation. + (h) through (ab) are not redundant — each survived every other check. (h) and (i) cover production callers the primitive tests missed: `hardenStableLockFile` takes the async path, and `hardenSecretDir` backs config, management-auth, tray, spill-store, and `native-profile-manager.ts:153`. (j) and (k) are a different @@ -1113,6 +1115,22 @@ and the ones that did were rewritten rather than kept. against `nativeMainClaimPath(context)` and `join(codexHome, NATIVE_MAIN_OWNER_DB)`, in the success and the failure test alike. + **The row that stayed empty longest was the successful one.** Every test here + proved a failure path or an invocation; none required the operation to actually + succeed. So (z), (aa) and (ab): an owner that hardens correctly and then fails to + acquire, an owner that never publishes `held` at all, and a claim that hardens and + then silently skips its protected operation, all passed. The success tests now + require the claim's operation to run, to return its value, and to run **after** + the ACL — and the owner to reach `held`, with a deferred ACL runner proving the + trace is exactly `["acquiring"]` while hardening is still in flight and exactly + `["acquiring", "held"]` once it completes. + + The matrix, enumerated, is: exact target · default binding · platform provenance · + **successful completion** · **ordering relative to the ACL** · failure propagation · + refusal taxonomy · raw observable trace · attempt count · protected operation + absent on failure. Any row unasserted at either edge is a hole, and each of these + rows was found by someone deleting the production code behind it. + **Activation gate, not a test:** the NTFS `bigint` inode behaviour is UNVERIFIED. `observe()` treats a zero inode as unobservable, so if Bun returns zero there, every REQUIRED Windows harden fails closed and this whole surface is inert in the diff --git a/tests/windows-secret-acl.test.ts b/tests/windows-secret-acl.test.ts index 5e3d37bfa..0cdc6adeb 100644 --- a/tests/windows-secret-acl.test.ts +++ b/tests/windows-secret-acl.test.ts @@ -1139,6 +1139,8 @@ describe("the production default hardener is reached, with the resolved platform */ const forcedWindows = async ( run: (codexHome: string) => Promise, + onGrant: () => void = () => {}, + gate?: Promise, ): Promise => { const seen: string[][] = []; setPlatformForTests("win32"); @@ -1146,6 +1148,10 @@ describe("the production default hardener is reached, with the resolved platform process.env.USERNAME = "ocx-test-user"; setAsyncIcaclsRunnerForTests(async args => { seen.push(args); + if (args.includes("/grant:r")) onGrant(); + // A deferred runner lets a test observe the window WHILE hardening is in + // flight, which is the only way to assert nothing was published early. + if (gate) await gate; return { success: true, exitCode: 0, timedOut: false, stdout: "" }; }); const codexHome = mkdtempSync(join(tmpdir(), "ocx-default-harden-")); @@ -1164,15 +1170,23 @@ describe("the production default hardener is reached, with the resolved platform test("a claim with no hardenPath runs the real ACL sequence under a forced win32", async () => { resetHardenedStateForTests(); let expected = ""; + const order: string[] = []; + let result: unknown; const seen = await forcedWindows(async codexHome => { expected = nativeMainClaimPath({ codexHome } as never); - await withNativeMainSharedClaim( + result = await withNativeMainSharedClaim( { codexHome } as never, - async () => undefined, + async () => { order.push("operation"); return "operation-ran"; }, { platform: "win32" }, ); - }); + }, () => { order.push("acl"); }); expect(seen.some(args => args.includes("/grant:r"))).toBe(true); + // The SUCCESS path, which "some ACL ran" cannot see: the protected operation + // actually ran, its value came back, and it ran AFTER the hardening rather + // than beside it. A claim that hardens and then silently skips its operation + // passed every other assertion here. + expect(result).toBe("operation-ran"); + expect(order).toEqual(["acl", "operation"]); // The EXACT target, not merely that some ACL ran. Redirecting the caller at // an unrelated existing file passed every other dimension of this test. expect(seen.every(args => args[0] === expected)).toBe(true); @@ -1181,18 +1195,35 @@ describe("the production default hardener is reached, with the resolved platform test("an owner with no hardenPath runs the real ACL sequence under a forced win32", async () => { resetHardenedStateForTests(); let expected = ""; + const trace: string[] = []; + let releaseAcl!: () => void; + const aclBlocked = new Promise(done => { releaseAcl = done; }); const seen = await forcedWindows(async codexHome => { expected = join(codexHome, NATIVE_MAIN_OWNER_DB); const owner = retainNativeMainOwner({ codexHome } as never, { platform: "win32", retryMs: 10 }); + const unsubscribe = owner.subscribe(snapshot => { trace.push(snapshot.status); }); try { + // While the ACL is still in flight the owner may NOT report held: that + // would be a caller believing it owns a database whose hardening has not + // finished. Ordering is the claim, and no terminal snapshot can make it. + await Bun.sleep(30); + expect(trace).toEqual(["acquiring"]); + releaseAcl(); + const deadline = Date.now() + 5_000; while (owner.snapshot().status === "acquiring" && Date.now() < deadline) { await Bun.sleep(10); } + // And it must actually SUCCEED. Every earlier version accepted any + // non-acquiring state, so an owner that hardened correctly and then + // failed to acquire passed. + expect(owner.snapshot()).toMatchObject({ status: "held" }); + expect(trace).toEqual(["acquiring", "held"]); } finally { + unsubscribe(); await owner.release(); } - }); + }, () => {}, aclBlocked); expect(seen.some(args => args.includes("/grant:r"))).toBe(true); expect(seen.every(args => args[0] === expected)).toBe(true); }); From fc8f4801ad212d40600047abd3f78b49800a8850 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Wed, 5 Aug 2026 10:41:26 +0900 Subject: [PATCH 118/163] test(acl): parameterizing one property over four entry points is not parameterizing the others MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A fresh reviewer swept the whole matrix instead of hunting one bug, and found the class arriving one level up. The memo-attribution matrix runs over all four public entry points, so it read as exhaustive. The failure-POLICY tests were written only against the file APIs. Four directory-only mutations survived the entire suite: required soft-failing on an ordinary failure, required soft-failing on a timeout, optional throwing on either. The production code was right in all four; nothing executable said so. Failure policy is now parameterized over the same four entry points. The second finding was a cache state nothing justified. A lookup that missed left the stale entry in place, so after a mismatch and a FAILED re-harden the old value survived, and restoring the old identity satisfied it again with no ACL work. That needs exact-identity ABA to bite, which this unit puts outside its proof bound — but scope is not a reason to keep an entry we have just proven does not describe the path. A miss retires it now. That test had to fail the re-harden to see anything: my first version let it succeed, which overwrites the memo and hides whether the miss retired it. The mutation survived until I fixed the test, which is the same lesson as the last four rounds. Thirty-one mutations, all restored to 113 pass / 0 fail across three files. --- .../030_lock_protocol.md | 36 +++- src/lib/windows-secret-acl.ts | 14 +- tests/windows-secret-acl.test.ts | 199 +++++++++++++++++- 3 files changed, 237 insertions(+), 12 deletions(-) diff --git a/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md b/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md index 876b8910c..7a7172a26 100644 --- a/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md +++ b/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md @@ -1026,8 +1026,12 @@ and the ones that did were rewritten rather than kept. permanent refusal, republishing `unavailable` each time; (x) the owner hardens a different existing file; (y) the claim hardens a different existing file; (z) the owner hardens correctly and then fails to acquire; (aa) the owner never publishes `held`; (ab) the claim - hardens and then silently skips its protected operation. - (h) through (ab) are not redundant — each survived every other check. (h) and (i) + hardens and then silently skips its protected operation; (ac) the claim races the harden against + a 1ms timer, continuing while icacls is still in flight; (ad) a DIRECTORY-only + failure policy diverges from the file policy — required soft-failing on an + ordinary failure or a timeout, or optional throwing on either; (ae) a memo entry + proven wrong is kept instead of retired. + (h) through (ae) are not redundant — each survived every other check. (h) and (i) cover production callers the primitive tests missed: `hardenStableLockFile` takes the async path, and `hardenSecretDir` backs config, management-auth, tray, spill-store, and `native-profile-manager.ts:153`. (j) and (k) are a different @@ -1115,6 +1119,24 @@ and the ones that did were rewritten rather than kept. against `nativeMainClaimPath(context)` and `join(codexHome, NATIVE_MAIN_OWNER_DB)`, in the success and the failure test alike. + **And the matrix applies per entry point, not per property.** (ad) is the class + arriving one level up: the memo-attribution matrix was parameterized over all four + public entry points, so it looked exhaustive — but the failure-POLICY tests were + written only against the file APIs. Four directory-only mutations survived the + whole suite. Parameterizing one property over four entry points does not + parameterize the others; each property needs the parameterization, not each entry + point. + + **(ae) is a cache state nothing justified.** A lookup that missed left the stale + entry in place, so after a mismatch and a *failed* re-harden the old value + survived and restoring the old identity satisfied it again with no ACL work. + Biting that needs exact-identity ABA, which §Deliberate residuals puts outside the + proof bound — but scope is not a reason to keep an entry we have just proven does + not describe what is at the path. A miss now retires it. The test has to fail the + re-harden to see this at all: a successful one overwrites the memo and hides + whether the miss retired anything, which is what the first version of that test + did. + **The row that stayed empty longest was the successful one.** Every test here proved a failure path or an invocation; none required the operation to actually succeed. So (z), (aa) and (ab): an owner that hardens correctly and then fails to @@ -1125,6 +1147,16 @@ and the ones that did were rewritten rather than kept. trace is exactly `["acquiring"]` while hardening is still in flight and exactly `["acquiring", "held"]` once it completes. + **"After" has to mean after it FINISHED.** (ac) is the same row failing a second + time in a subtler form: recording `acl` when `/grant:r` is invoked and then + `operation` proves only that hardening *began* first. Wrapping the harden in + `Promise.race([harden, Bun.sleep(1)])` — an early continuation that on real + Windows lets an `icacls` sequence longer than a millisecond stay in flight while + the claim proceeds — passed that ordering assertion. Both edges now use the + deferred-runner shape: hold the ACL unresolved, require the protected work has + NOT started, release it, then require the result. An event marker taken at the + START of an operation cannot order anything against its COMPLETION. + The matrix, enumerated, is: exact target · default binding · platform provenance · **successful completion** · **ordering relative to the ACL** · failure propagation · refusal taxonomy · raw observable trace · attempt count · protected operation diff --git a/src/lib/windows-secret-acl.ts b/src/lib/windows-secret-acl.ts index 6c259cb90..f8cc2ec90 100644 --- a/src/lib/windows-secret-acl.ts +++ b/src/lib/windows-secret-acl.ts @@ -136,8 +136,18 @@ function memoSatisfied(cache: Map, targetPath: string) const current = observe(targetPath); // Unreadable now is not "unchanged": re-harden rather than trust a value we // cannot confirm still describes what is there. - if (current === null) return false; - return memoValue(current) === remembered; + // + // A miss RETIRES the entry rather than leaving it. Keeping it left the cache in + // a state nothing could justify: after a mismatch and a failed re-harden, the + // stale value survived, so restoring the old identity would satisfy it again + // without any ACL work. That needs exact-identity ABA to bite — outside the + // proof bound this unit claims — but "the consequence is out of scope" is not a + // reason to keep an entry we have just proven does not describe what is there. + if (current === null || memoValue(current) !== remembered) { + cache.delete(targetPath); + return false; + } + return true; } /** diff --git a/tests/windows-secret-acl.test.ts b/tests/windows-secret-acl.test.ts index 0cdc6adeb..0079ca73c 100644 --- a/tests/windows-secret-acl.test.ts +++ b/tests/windows-secret-acl.test.ts @@ -1170,23 +1170,33 @@ describe("the production default hardener is reached, with the resolved platform test("a claim with no hardenPath runs the real ACL sequence under a forced win32", async () => { resetHardenedStateForTests(); let expected = ""; - const order: string[] = []; + let operationStarted = false; let result: unknown; + let releaseAcl!: () => void; + const aclBlocked = new Promise(done => { releaseAcl = done; }); const seen = await forcedWindows(async codexHome => { expected = nativeMainClaimPath({ codexHome } as never); - result = await withNativeMainSharedClaim( + const claim = withNativeMainSharedClaim( { codexHome } as never, - async () => { order.push("operation"); return "operation-ran"; }, + async () => { operationStarted = true; return "operation-ran"; }, { platform: "win32" }, ); - }, () => { order.push("acl"); }); + // Recording "acl started" and then "operation" only proves the ACL BEGAN + // first. Wrapping the harden in Promise.race([harden, sleep(1)]) — an + // early continuation that on real Windows would let icacls still be in + // flight — passed that version of this test. So hold the ACL unresolved + // and require that the operation has NOT started. + await Bun.sleep(30); + expect(operationStarted).toBe(false); + releaseAcl(); + result = await claim; + expect(operationStarted).toBe(true); + }, () => {}, aclBlocked); expect(seen.some(args => args.includes("/grant:r"))).toBe(true); // The SUCCESS path, which "some ACL ran" cannot see: the protected operation - // actually ran, its value came back, and it ran AFTER the hardening rather - // than beside it. A claim that hardens and then silently skips its operation - // passed every other assertion here. + // actually ran and its value came back. A claim that hardens and then + // silently skips its operation passed every other assertion here. expect(result).toBe("operation-ran"); - expect(order).toEqual(["acl", "operation"]); // The EXACT target, not merely that some ACL ran. Redirecting the caller at // an unrelated existing file passed every other dimension of this test. expect(seen.every(args => args[0] === expected)).toBe(true); @@ -1350,3 +1360,176 @@ describe("a required hardening failure stops the operation it protects", () => { }, args => { hardenAttempts += 1; targets.push(args[0]!); }); }); }); + +/** + * Failure POLICY, across all four entry points. + * + * The memo-attribution matrix above is parameterized over all four, but the + * failure-policy tests were written only against the file APIs. Four + * directory-only mutations survived the whole suite: required soft-failing on an + * ordinary failure, required soft-failing on a timeout, optional throwing on an + * ordinary failure, and optional throwing on a timeout. The production code was + * right in each case; nothing executable said so. + * + * That is the same projection class one level up — parameterizing one property + * over four entry points does not parameterize the others. + */ +for (const { label, harden, create } of ENTRY_POINTS) { + describe(`failure policy — ${label} entry point`, () => { + const withFailingAcl = async ( + result: IcaclsResult, + body: (target: string) => Promise, + ): Promise => { + resetHardenedStateForTests(); + const target = join(testDir, `policy-${label}-${result.timedOut ? "timeout" : "error"}`); + create(target); + setPlatformForTests("win32"); + const previousUsername = process.env.USERNAME; + process.env.USERNAME = "ocx-test-user"; + setIcaclsRunnerForTests(() => result); + setAsyncIcaclsRunnerForTests(async () => result); + try { + await body(target); + } finally { + if (previousUsername === undefined) delete process.env.USERNAME; + else process.env.USERNAME = previousUsername; + setIcaclsRunnerForTests(null); + setAsyncIcaclsRunnerForTests(null); + setPlatformForTests(null); + } + }; + + const ordinaryFailure: IcaclsResult = { + success: false, exitCode: 5, timedOut: false, stdout: "", stderr: "", + }; + const timeout: IcaclsResult = { + success: false, exitCode: null, timedOut: true, stdout: "", stderr: "", + }; + + test("required fails closed on an ordinary ACL failure", async () => { + await withFailingAcl(ordinaryFailure, async target => { + await expect(harden(target, { required: true })).rejects.toThrow(); + }); + }); + + test("required fails closed on a timeout", async () => { + await withFailingAcl(timeout, async target => { + await expect(harden(target, { required: true })).rejects.toThrow(); + }); + }); + + test("optional soft-fails on an ordinary ACL failure, with a diagnostic", async () => { + await withFailingAcl(ordinaryFailure, async target => { + const result = await harden(target, { required: false }); + expect(result.ok).toBe(false); + expect(typeof result.diagnostics).toBe("string"); + expect(result.diagnostics!.length).toBeGreaterThan(0); + }); + }); + + test("optional soft-fails on a timeout, with a diagnostic", async () => { + await withFailingAcl(timeout, async target => { + const result = await harden(target, { required: false }); + expect(result.ok).toBe(false); + expect(result.diagnostics).toMatch(/timed out|ETIMEDOUT/i); + }); + }); + }); +} + +describe("a memo entry proven wrong is retired, not kept", () => { + /** + * The cache state nothing justified. + * + * `memoSatisfied` used to return false on a mismatch and leave the entry in + * place. So after a mismatch and a failed re-harden, the stale value survived — + * and restoring the old identity satisfied it again with no ACL work. Biting + * that needs exact-identity ABA, which this unit explicitly puts outside its + * proof bound, but scope is not a reason to keep an entry we have just proven + * does not describe what is at the path. + * + * Both halves are asserted, because "it was retired" and "it is not silently + * re-satisfiable" are different claims and the first does not imply the second + * to a reader. + */ + test("a mismatch retires the entry even when the re-harden then fails", async () => { + resetHardenedStateForTests(); + const target = join(testDir, "retired.sqlite"); + writeFileSync(target, "x", "utf8"); + + setPlatformForTests("win32"); + const previousUsername = process.env.USERNAME; + process.env.USERNAME = "ocx-test-user"; + let grants = 0; + let aclSucceeds = true; + setIcaclsRunnerForTests(args => { + if (args.includes("/grant:r")) grants += 1; + return aclSucceeds + ? { success: true, exitCode: 0, timedOut: false, stdout: "" } + : { success: false, exitCode: 5, timedOut: false, stdout: "", stderr: "" }; + }); + try { + const original = { dev: 1n, ino: 10n, ctimeNs: 100n }; + let current = original; + setStatForTests(() => current); + + expect(hardenSecretPath(target, { required: true })).toEqual({ ok: true }); + expect(grants).toBe(1); + expect(hardenedSecretPathCountForTests()).toBe(1); + + // Something else is at the path, and the re-harden FAILS. A successful + // re-harden would overwrite the memo and hide whether the miss retired it, + // which is exactly what a first version of this test did. + current = { dev: 1n, ino: 11n, ctimeNs: 500n }; + aclSucceeds = false; + expect(() => hardenSecretPath(target, { required: true })).toThrow(); + expect(grants).toBe(2); + // Nothing survives that we have proven does not describe the path. + expect(hardenedSecretPathCountForTests()).toBe(0); + + // And restoring the original identity is not silently re-satisfied. + current = original; + aclSucceeds = true; + expect(hardenSecretPath(target, { required: true })).toEqual({ ok: true }); + expect(grants).toBe(3); + } finally { + if (previousUsername === undefined) delete process.env.USERNAME; + else process.env.USERNAME = previousUsername; + setIcaclsRunnerForTests(null); + setPlatformForTests(null); + setStatForTests(null); + } + }); + + test("an unreadable observation also retires the entry", () => { + resetHardenedStateForTests(); + const target = join(testDir, "retired-unreadable.sqlite"); + writeFileSync(target, "x", "utf8"); + + setPlatformForTests("win32"); + const previousUsername = process.env.USERNAME; + process.env.USERNAME = "ocx-test-user"; + setIcaclsRunnerForTests(() => ({ success: true, exitCode: 0, timedOut: false, stdout: "" })); + try { + let readable = true; + setStatForTests(() => { + if (!readable) throw Object.assign(new Error("EACCES"), { code: "EACCES" }); + return { dev: 1n, ino: 10n, ctimeNs: 100n }; + }); + + expect(hardenSecretPath(target, { required: true })).toEqual({ ok: true }); + expect(hardenedSecretPathCountForTests()).toBe(1); + + readable = false; + // Fails closed AND leaves nothing behind to be re-satisfied later. + expect(() => hardenSecretPath(target, { required: true })).toThrow(); + expect(hardenedSecretPathCountForTests()).toBe(0); + } finally { + if (previousUsername === undefined) delete process.env.USERNAME; + else process.env.USERNAME = previousUsername; + setIcaclsRunnerForTests(null); + setPlatformForTests(null); + setStatForTests(null); + } + }); +}); From dfe1bb3d8d467534eafd3c20c0c63ed700bc9c15 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Wed, 5 Aug 2026 10:41:30 +0900 Subject: [PATCH 119/163] test(codex): 'after the ACL' has to mean after it finished MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Recording 'acl' when /grant:r is invoked and then 'operation' proves only that hardening BEGAN first. Wrapping the harden in Promise.race([harden, sleep(1)]) passed it — an early continuation that on real Windows lets an icacls sequence longer than a millisecond stay in flight while the claim proceeds to use the database and run its protected operation, with a later ACL failure no longer able to stop either. Both edges now use the deferred-runner shape the owner test already had: hold the ACL unresolved, require the protected work has NOT started, release it, then require the result. An event marker taken at the START of an operation cannot order anything against its COMPLETION. Twenty-nine mutations, all restored to 113 pass / 0 fail across three files. This commit also carries a memo fix from earlier in this unit that had not been committed: memoSatisfied now RETIRES an entry it has just proven wrong instead of leaving it. Keeping it left the cache in a state nothing could justify — after a mismatch and a failed re-harden the stale value survived, so restoring the old identity satisfied it again with no ACL work. Biting that needs exact-identity ABA, which this unit puts outside its proof bound, but scope is not a reason to keep an entry proven not to describe what is at the path. Also propagated the WP4 supersession to the places it had not reached — the opening scope paragraph and the IN/OUT table still said desired-state.ts owns a new per-home lock — and refreshed three stale citations in that document. --- .../030_desired_state.md | 22 ++++++++++--------- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/devlog/_plan/260803_codex_desktop_toggle/030_desired_state.md b/devlog/_plan/260803_codex_desktop_toggle/030_desired_state.md index a7facf0de..e0508a3ac 100644 --- a/devlog/_plan/260803_codex_desktop_toggle/030_desired_state.md +++ b/devlog/_plan/260803_codex_desktop_toggle/030_desired_state.md @@ -32,19 +32,21 @@ Already present: - `mutatePersistedConfig` clones and rebases a callback under the config mutation lock, then returns `committed | unchanged | unavailable`; callers do not need a - second persistence mechanism (`src/config.ts:1825-1906`). + second persistence mechanism (`src/config.ts:1957`). - `syncModelsToCodex` owns the normal catalog-plus-injection path (`src/codex/sync.ts:49-129`), while provider/model/combo routes bypass it through `refreshCodexCatalogBestEffort` (`src/server/management-api.ts:105-112`). - `restoreNativeCodex` is the idempotent Codex remover - (`src/codex/inject.ts:759-795`), and `assertNativeTeardownOwned` is the shipped + (`src/codex/inject.ts:820`), and `assertNativeTeardownOwned` is the shipped foreign-home preflight (`src/integrations/native/ownership-preflight.ts:19-35`). - crash-journal reconciliation already repairs an abandoned injection (`src/codex/journal.ts:148-162`). -WP4 adds one persisted Codex flag and one per-`CODEX_HOME` linearization lock that -covers both desired-state commits and bounded native commit sections. Provider -model gathering stays outside the lock. Ownership is tri-state and is resolved +WP4 adds one persisted Codex flag and takes its per-`CODEX_HOME` linearization from +WP12's public N acquisition API (`src/codex/codex-write-lock.ts`), which covers both +desired-state commits and bounded native commit sections. **It does not build a +second lock** — see the supersession note below. Provider model gathering stays +outside the lock. Ownership is tri-state and is resolved before journal repair or lock creation, then rechecked inside the lock. OFF reconciliation restores config, profile, catalog, cache, and history at start and ensure. WP5 adds the management route and GUI switch that call the writer; WP4 @@ -56,7 +58,7 @@ defines the writer failures WP5 must map but not WP5's full response schema. |---|---|---| | `src/types.ts` | MODIFY | Adds a one-key `OcxClientIntegrationsConfig` and its optional `OcxConfig.clientIntegrations` home. | | `src/config.ts` | MODIFY | Parses the Codex key, resolves absent as ON, and mutates only that field through `mutatePersistedConfig`; documents its thrown lock branch. | -| `src/codex/desired-state.ts` | NEW | Owns the external per-home linearization lock, test seams, tri-state ownership gate, owned restore wrapper, observed-state inspection, and OFF reconciliation. | +| `src/codex/desired-state.ts` | NEW | Owns test seams, the tri-state ownership gate, the owned restore wrapper, observed-state inspection, and OFF reconciliation. **Not** the linearization lock: that is WP12's public N API. | | `src/codex/sync.ts` | MODIFY | Separates model gathering from the bounded apply commit and returns `ok:false` plus `skippedReason` for every no-write result. | | `src/codex/refresh.ts` | MODIFY | Splits gathered catalog data from the bounded catalog/cache commit used outside `syncModelsToCodex`. | | `src/codex/catalog/sync.ts` | MODIFY | Commits catalog/cache only while holding the shared linearization lock; restores or invalidates `models_cache.json` during native removal. | @@ -385,11 +387,11 @@ before every irreversible write below, while the same lock is still held: | bundled fallback | `materializeBundledCodexCatalog` at `src/codex/catalog/bundled.ts:213-219` | candidate resolution outside; materialization inside lock | | pristine backup | `copyFileSync` / `atomicWriteFile` at `src/codex/catalog/parsing.ts:428-444` | each target-hashed backup write inside lock | | catalog | `atomicWriteFile(catalogPath, ...)` at `src/codex/catalog/sync.ts:568` | gathered candidate committed inside lock | -| models cache | `atomicWriteFile(activeCodexModelsCachePath(), ...)` at `src/codex/catalog/sync.ts:600-613` | replacement/restoration inside lock | +| models cache | `atomicWriteFile(activeCodexModelsCachePath(), ...)` at `src/codex/catalog/sync.ts:832` | replacement/restoration inside lock | | injection journal | `writeJournal(...)` at `src/codex/inject.ts:521-527` | inside lock | | config/profile/marker | writes at `src/codex/inject.ts:593-597` | all inside the same lock | | history mutation | callback at `src/codex/inject.ts:598-603` | the complete callback, including its hidden writes, inside lock | -| native remove | `restoreNativeCodex` body at `src/codex/inject.ts:764-795` | ownership and OFF rechecked, then complete remove inside lock | +| native remove | `restoreNativeCodex` body at `src/codex/inject.ts:820` | ownership and OFF rechecked, then complete remove inside lock | At the deterministic seam, start an OFF setter in a second process and prove its intent commit cannot complete until the held apply write exits. Then release the @@ -574,8 +576,8 @@ the prior startup ordering could not support. ### The remover owns `models_cache.json` too `restoreNativeCodex` currently restores journal/config, catalog, and history at -`src/codex/inject.ts:764-795`, but apply writes routed models into -`models_cache.json` at `src/codex/catalog/sync.ts:600-613`. Reporting OFF while +`src/codex/inject.ts:820`, but apply writes routed models into +`models_cache.json` at `src/codex/catalog/sync.ts:832`. Reporting OFF while that cache still advertises routed slugs repeats the WP2 incident: the test sees only artifacts it already knew to assert. From a6d2d62c5b7c5962ea07fd9f6bd15556784394d0 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Wed, 5 Aug 2026 10:51:14 +0900 Subject: [PATCH 120/163] test(acl): the retirement branch broke its own four-entry-point rule on arrival MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two holes in the memo-retirement code I committed one commit ago, both found by deleting the production behavior behind them. Exempting DIRECTORIES alone from the retirement passed all 113 tests, because the retirement test drove hardenSecretPath only — the same gap this suite already closed twice for other properties, reappearing in the newest one. Both retirement tests are now parameterized over all four entry points. The second is subtler and is the attribution trap again. The unreadable- observation test let the ACL succeed, so recordHarden deleted the entry for its own reasons and a terminal count of zero could not say which mechanism retired it. Retaining the memo specifically when the observation is unreadable passed. The ACL now fails BEFORE recordHarden runs, and the re-satisfaction check is what shows the lookup did it. 119 pass / 0 fail across three files. Both mutations redden four tests each. Also stopped putting a mutation COUNT in the goalplan. A reviewer pointed out the labels (a)-(ag) do not map one-to-one onto independent mutations — some labels cover several — so the number I kept quoting was wrong in a way that sounded precise. The document is the authority; the goalplan points at it. Plus two more stale citations in the WP4 document: the models-cache write is replaceCodexModelsCache at catalog/sync.ts:847 with the atomic write in internal/catalog-writer.ts:202, and invalidateCodexModelsCache is at catalog/sync.ts:833. --- .../030_desired_state.md | 6 +- .../030_lock_protocol.md | 15 +- tests/windows-secret-acl.test.ts | 197 ++++++++++-------- 3 files changed, 127 insertions(+), 91 deletions(-) diff --git a/devlog/_plan/260803_codex_desktop_toggle/030_desired_state.md b/devlog/_plan/260803_codex_desktop_toggle/030_desired_state.md index e0508a3ac..bebb3d5f5 100644 --- a/devlog/_plan/260803_codex_desktop_toggle/030_desired_state.md +++ b/devlog/_plan/260803_codex_desktop_toggle/030_desired_state.md @@ -387,7 +387,7 @@ before every irreversible write below, while the same lock is still held: | bundled fallback | `materializeBundledCodexCatalog` at `src/codex/catalog/bundled.ts:213-219` | candidate resolution outside; materialization inside lock | | pristine backup | `copyFileSync` / `atomicWriteFile` at `src/codex/catalog/parsing.ts:428-444` | each target-hashed backup write inside lock | | catalog | `atomicWriteFile(catalogPath, ...)` at `src/codex/catalog/sync.ts:568` | gathered candidate committed inside lock | -| models cache | `atomicWriteFile(activeCodexModelsCachePath(), ...)` at `src/codex/catalog/sync.ts:832` | replacement/restoration inside lock | +| models cache | `replaceCodexModelsCache` at `src/codex/catalog/sync.ts:847`, whose atomic write is `src/codex/internal/catalog-writer.ts:202` | replacement/restoration inside lock | | injection journal | `writeJournal(...)` at `src/codex/inject.ts:521-527` | inside lock | | config/profile/marker | writes at `src/codex/inject.ts:593-597` | all inside the same lock | | history mutation | callback at `src/codex/inject.ts:598-603` | the complete callback, including its hidden writes, inside lock | @@ -577,7 +577,7 @@ the prior startup ordering could not support. `restoreNativeCodex` currently restores journal/config, catalog, and history at `src/codex/inject.ts:820`, but apply writes routed models into -`models_cache.json` at `src/codex/catalog/sync.ts:832`. Reporting OFF while +`models_cache.json` through `replaceCodexModelsCache` (`src/codex/catalog/sync.ts:847`). Reporting OFF while that cache still advertises routed slugs repeats the WP2 incident: the test sees only artifacts it already knew to assert. @@ -585,7 +585,7 @@ Add `restoreCodexModelsCache` beside `restoreCodexCatalog`. After catalog restor rewrite the cache from the restored catalog with an expired wrapper; if the catalog is unavailable, parse the existing cache and remove only routed slugs. Missing cache is success. Unreadable or unwritable cache is failure, not the -current swallowed `false` from `invalidateCodexModelsCache` (`:601-616`): +current swallowed `false` from `invalidateCodexModelsCache` (`src/codex/catalog/sync.ts:833`): ```diff const cat = restoreCodexCatalog(); diff --git a/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md b/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md index 7a7172a26..5eb1ad1ab 100644 --- a/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md +++ b/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md @@ -1030,8 +1030,10 @@ and the ones that did were rewritten rather than kept. a 1ms timer, continuing while icacls is still in flight; (ad) a DIRECTORY-only failure policy diverges from the file policy — required soft-failing on an ordinary failure or a timeout, or optional throwing on either; (ae) a memo entry - proven wrong is kept instead of retired. - (h) through (ae) are not redundant — each survived every other check. (h) and (i) + proven wrong is kept instead of retired; (af) directory memos alone are exempted + from that retirement; (ag) an unreadable observation keeps its memo while a + successful re-harden masks which mechanism deleted it. + (h) through (ag) are not redundant — each survived every other check. (h) and (i) cover production callers the primitive tests missed: `hardenStableLockFile` takes the async path, and `hardenSecretDir` backs config, management-auth, tray, spill-store, and `native-profile-manager.ts:153`. (j) and (k) are a different @@ -1157,6 +1159,15 @@ and the ones that did were rewritten rather than kept. NOT started, release it, then require the result. An event marker taken at the START of an operation cannot order anything against its COMPLETION. + **The rule applies to every property, including the ones added last.** (af) and + (ag) are the memo-retirement branch failing its own four-entry-point rule the + moment it shipped: exempting directories alone from the retirement passed all 113 + tests, because the retirement test drove `hardenSecretPath` only. And (ag) is the + attribution trap in miniature — the unreadable test let the ACL succeed, so + `recordHarden` deleted the entry for its own reasons and a terminal count of zero + could not say which mechanism retired it. The ACL now fails BEFORE `recordHarden` + runs, and the re-satisfaction check is what proves the lookup did it. + The matrix, enumerated, is: exact target · default binding · platform provenance · **successful completion** · **ordering relative to the ACL** · failure propagation · refusal taxonomy · raw observable trace · attempt count · protected operation diff --git a/tests/windows-secret-acl.test.ts b/tests/windows-secret-acl.test.ts index 0079ca73c..3b84f24c6 100644 --- a/tests/windows-secret-acl.test.ts +++ b/tests/windows-secret-acl.test.ts @@ -1437,99 +1437,124 @@ for (const { label, harden, create } of ENTRY_POINTS) { }); } -describe("a memo entry proven wrong is retired, not kept", () => { - /** - * The cache state nothing justified. - * - * `memoSatisfied` used to return false on a mismatch and leave the entry in - * place. So after a mismatch and a failed re-harden, the stale value survived — - * and restoring the old identity satisfied it again with no ACL work. Biting - * that needs exact-identity ABA, which this unit explicitly puts outside its - * proof bound, but scope is not a reason to keep an entry we have just proven - * does not describe what is at the path. - * - * Both halves are asserted, because "it was retired" and "it is not silently - * re-satisfiable" are different claims and the first does not imply the second - * to a reader. - */ - test("a mismatch retires the entry even when the re-harden then fails", async () => { - resetHardenedStateForTests(); - const target = join(testDir, "retired.sqlite"); - writeFileSync(target, "x", "utf8"); +for (const { label, harden, create } of ENTRY_POINTS) { + const memoCount = label.startsWith("dir") + ? hardenedSecretDirCountForTests + : hardenedSecretPathCountForTests; - setPlatformForTests("win32"); - const previousUsername = process.env.USERNAME; - process.env.USERNAME = "ocx-test-user"; - let grants = 0; - let aclSucceeds = true; - setIcaclsRunnerForTests(args => { - if (args.includes("/grant:r")) grants += 1; - return aclSucceeds - ? { success: true, exitCode: 0, timedOut: false, stdout: "" } - : { success: false, exitCode: 5, timedOut: false, stdout: "", stderr: "" }; - }); - try { - const original = { dev: 1n, ino: 10n, ctimeNs: 100n }; - let current = original; - setStatForTests(() => current); + describe(`a memo entry proven wrong is retired, not kept — ${label}`, () => { + /** + * The cache state nothing justified. + * + * `memoSatisfied` used to return false on a mismatch and leave the entry in + * place, so after a mismatch and a failed re-harden the stale value survived + * and restoring the old identity satisfied it again with no ACL work. Biting + * that needs exact-identity ABA, outside this unit's proof bound — but scope + * is not a reason to keep an entry proven not to describe what is at the path. + * + * Parameterized over all four entry points because the retirement is shared + * machinery and the first version tested only `hardenSecretPath`: restricting + * the deletion to non-directory caches passed all 113 tests. + */ + test("a mismatch retires the entry even when the re-harden then fails", async () => { + resetHardenedStateForTests(); + const target = join(testDir, `retired-${label}.sqlite`); + create(target); - expect(hardenSecretPath(target, { required: true })).toEqual({ ok: true }); - expect(grants).toBe(1); - expect(hardenedSecretPathCountForTests()).toBe(1); + await withWin32(async () => { + let grants = 0; + let aclSucceeds = true; + const result = () => (aclSucceeds + ? { success: true, exitCode: 0, timedOut: false, stdout: "" } + : { success: false, exitCode: 5, timedOut: false, stdout: "", stderr: "" }); + setIcaclsRunnerForTests(args => { + if (args.includes("/grant:r")) grants += 1; + return result(); + }); + setAsyncIcaclsRunnerForTests(async args => { + if (args.includes("/grant:r")) grants += 1; + return result(); + }); - // Something else is at the path, and the re-harden FAILS. A successful - // re-harden would overwrite the memo and hide whether the miss retired it, - // which is exactly what a first version of this test did. - current = { dev: 1n, ino: 11n, ctimeNs: 500n }; - aclSucceeds = false; - expect(() => hardenSecretPath(target, { required: true })).toThrow(); - expect(grants).toBe(2); - // Nothing survives that we have proven does not describe the path. - expect(hardenedSecretPathCountForTests()).toBe(0); + const original = { dev: 1n, ino: 10n, ctimeNs: 100n }; + let current = original; + setStatForTests(() => current); - // And restoring the original identity is not silently re-satisfied. - current = original; - aclSucceeds = true; - expect(hardenSecretPath(target, { required: true })).toEqual({ ok: true }); - expect(grants).toBe(3); - } finally { - if (previousUsername === undefined) delete process.env.USERNAME; - else process.env.USERNAME = previousUsername; - setIcaclsRunnerForTests(null); - setPlatformForTests(null); - setStatForTests(null); - } - }); + expect(await harden(target, { required: true })).toEqual({ ok: true }); + expect(grants).toBe(1); + expect(memoCount()).toBe(1); - test("an unreadable observation also retires the entry", () => { - resetHardenedStateForTests(); - const target = join(testDir, "retired-unreadable.sqlite"); - writeFileSync(target, "x", "utf8"); + // Something else is at the path, and the re-harden FAILS. A successful + // re-harden would overwrite the memo and hide whether the miss retired + // it — which is what a first version of this test did. + current = { dev: 1n, ino: 11n, ctimeNs: 500n }; + aclSucceeds = false; + await expect(harden(target, { required: true })).rejects.toThrow(); + expect(grants).toBe(2); + expect(memoCount()).toBe(0); - setPlatformForTests("win32"); - const previousUsername = process.env.USERNAME; - process.env.USERNAME = "ocx-test-user"; - setIcaclsRunnerForTests(() => ({ success: true, exitCode: 0, timedOut: false, stdout: "" })); - try { - let readable = true; - setStatForTests(() => { - if (!readable) throw Object.assign(new Error("EACCES"), { code: "EACCES" }); - return { dev: 1n, ino: 10n, ctimeNs: 100n }; + // And restoring the original identity is not silently re-satisfied. + current = original; + aclSucceeds = true; + expect(await harden(target, { required: true })).toEqual({ ok: true }); + expect(grants).toBe(3); }); + }); - expect(hardenSecretPath(target, { required: true })).toEqual({ ok: true }); - expect(hardenedSecretPathCountForTests()).toBe(1); + /** + * The unreadable branch, isolated from `recordHarden`. + * + * Letting the ACL succeed here proves nothing: `recordHarden` would delete + * the entry for its own reasons, so a terminal count of zero cannot say WHICH + * mechanism retired it. Retaining the memo specifically when the observation + * is unreadable passed that version. So the ACL fails BEFORE `recordHarden` + * runs, and the re-satisfaction check is what shows the lookup retired it. + */ + test("an unreadable observation retires the entry, and the lookup is what does it", async () => { + resetHardenedStateForTests(); + const target = join(testDir, `retired-unreadable-${label}.sqlite`); + create(target); - readable = false; - // Fails closed AND leaves nothing behind to be re-satisfied later. - expect(() => hardenSecretPath(target, { required: true })).toThrow(); - expect(hardenedSecretPathCountForTests()).toBe(0); - } finally { - if (previousUsername === undefined) delete process.env.USERNAME; - else process.env.USERNAME = previousUsername; - setIcaclsRunnerForTests(null); - setPlatformForTests(null); - setStatForTests(null); - } + await withWin32(async () => { + let grants = 0; + let aclSucceeds = true; + const result = () => (aclSucceeds + ? { success: true, exitCode: 0, timedOut: false, stdout: "" } + : { success: false, exitCode: 5, timedOut: false, stdout: "", stderr: "" }); + setIcaclsRunnerForTests(args => { + if (args.includes("/grant:r")) grants += 1; + return result(); + }); + setAsyncIcaclsRunnerForTests(async args => { + if (args.includes("/grant:r")) grants += 1; + return result(); + }); + + const identity = { dev: 1n, ino: 10n, ctimeNs: 100n }; + let readable = true; + setStatForTests(() => { + if (!readable) throw Object.assign(new Error("EACCES"), { code: "EACCES" }); + return identity; + }); + + expect(await harden(target, { required: true })).toEqual({ ok: true }); + expect(grants).toBe(1); + expect(memoCount()).toBe(1); + + // Unreadable AND the ACL fails, so recordHarden never reaches its own + // deletion: whatever retires the entry here is the lookup. + readable = false; + aclSucceeds = false; + await expect(harden(target, { required: true })).rejects.toThrow(); + expect(grants).toBe(2); + expect(memoCount()).toBe(0); + + // Restoring the exact original identity must not be silently satisfied. + readable = true; + aclSucceeds = true; + expect(await harden(target, { required: true })).toEqual({ ok: true }); + expect(grants).toBe(3); + }); + }); }); -}); +} From 223e4c9568835c6ac907942f3fc2654103c17712 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Wed, 5 Aug 2026 10:56:59 +0900 Subject: [PATCH 121/163] test(acl): optional does not mean unattributed MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Every identity and retirement scenario called with required:true, and the optional-policy tests use fresh paths with no memo, so neither could see an optional-only bypass. Adding if (!opts.required && cache.has(targetPath)) return { ok: true }; left all 119 tests green — an optional caller would accept any object at a previously hardened pathname without observing identity, retiring the memo, or running icacls once. Optional means a failure is REPORTED rather than thrown. It does not mean the memo may describe a different file. Memo attribution is now parameterized over required:true|false as well as all four entry points: a stale memo does not satisfy an optional caller, a failed re-harden still soft-fails honestly with diagnostics and retires the entry, and restoring the original identity is not silently re-satisfied. 123 pass / 0 fail across three files; the mutation reddens four tests. Also finally removed the mutation count from the goalplan. I reported that as done last round and it was not — my regex missed and I did not re-read the field. That is the third time this audit I have described an edit that did not land, and all three were caught by someone else checking rather than by me. --- .../030_lock_protocol.md | 15 +++- tests/windows-secret-acl.test.ts | 70 +++++++++++++++++++ 2 files changed, 83 insertions(+), 2 deletions(-) diff --git a/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md b/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md index 5eb1ad1ab..593505583 100644 --- a/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md +++ b/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md @@ -1032,8 +1032,9 @@ and the ones that did were rewritten rather than kept. ordinary failure or a timeout, or optional throwing on either; (ae) a memo entry proven wrong is kept instead of retired; (af) directory memos alone are exempted from that retirement; (ag) an unreadable observation keeps its memo while a - successful re-harden masks which mechanism deleted it. - (h) through (ag) are not redundant — each survived every other check. (h) and (i) + successful re-harden masks which mechanism deleted it; (ah) an OPTIONAL caller is + allowed to trust a pathname-only memo. + (h) through (ah) are not redundant — each survived every other check. (h) and (i) cover production callers the primitive tests missed: `hardenStableLockFile` takes the async path, and `hardenSecretDir` backs config, management-auth, tray, spill-store, and `native-profile-manager.ts:153`. (j) and (k) are a different @@ -1168,6 +1169,16 @@ and the ones that did were rewritten rather than kept. could not say which mechanism retired it. The ACL now fails BEFORE `recordHarden` runs, and the re-satisfaction check is what proves the lookup did it. + **Requiredness is an axis, not a policy footnote.** (ah): every identity and + retirement scenario called with `required: true`, and the optional-policy tests + use fresh paths with no memo, so neither could see an optional-only bypass. + Adding `if (!opts.required && cache.has(targetPath)) return { ok: true }` left + all 119 tests green, letting an optional caller accept any object at a + previously hardened pathname without observing identity, retiring the memo, or + running icacls. Optional means a failure is REPORTED rather than thrown; it does + not mean unattributed. Every memo property is now parameterized over + `required: true | false` as well as the four entry points. + The matrix, enumerated, is: exact target · default binding · platform provenance · **successful completion** · **ordering relative to the ACL** · failure propagation · refusal taxonomy · raw observable trace · attempt count · protected operation diff --git a/tests/windows-secret-acl.test.ts b/tests/windows-secret-acl.test.ts index 3b84f24c6..9ca57e032 100644 --- a/tests/windows-secret-acl.test.ts +++ b/tests/windows-secret-acl.test.ts @@ -1558,3 +1558,73 @@ for (const { label, harden, create } of ENTRY_POINTS) { }); }); } + +for (const { label, harden, create } of ENTRY_POINTS) { + const memoCount = label.startsWith("dir") + ? hardenedSecretDirCountForTests + : hardenedSecretPathCountForTests; + + describe(`memo attribution holds for OPTIONAL callers too — ${label}`, () => { + /** + * The requiredness axis, which every other memo test left uncovered. + * + * Each identity and retirement scenario calls with `required: true`, and the + * optional-policy tests use fresh paths with no memo, so neither can see an + * optional-only bypass. Inserting + * `if (!opts.required && cache.has(targetPath)) return { ok: true }` before + * the lookup left all 119 tests green: an optional caller would then accept + * ANY object at a previously hardened pathname without observing identity, + * retiring the memo, or running icacls. + * + * Optional does not mean unattributed. It means a failure is reported rather + * than thrown — the memo still has to describe what is actually there. + */ + test("a stale memo does not satisfy an optional caller, and a failed re-harden still retires it", async () => { + resetHardenedStateForTests(); + const target = join(testDir, `optional-${label}.sqlite`); + create(target); + + await withWin32(async () => { + let grants = 0; + let aclSucceeds = true; + const result = () => (aclSucceeds + ? { success: true, exitCode: 0, timedOut: false, stdout: "" } + : { success: false, exitCode: 5, timedOut: false, stdout: "", stderr: "" }); + setIcaclsRunnerForTests(args => { + if (args.includes("/grant:r")) grants += 1; + return result(); + }); + setAsyncIcaclsRunnerForTests(async args => { + if (args.includes("/grant:r")) grants += 1; + return result(); + }); + + const original = { dev: 1n, ino: 10n, ctimeNs: 100n }; + let current = original; + setStatForTests(() => current); + + expect(await harden(target, { required: false })).toEqual({ ok: true }); + expect(grants).toBe(1); + expect(memoCount()).toBe(1); + + // A different object at the same path: the memo must not answer, even + // for an optional caller. + current = { dev: 1n, ino: 11n, ctimeNs: 500n }; + aclSucceeds = false; + const failed = await harden(target, { required: false }); + // Optional soft-fails rather than throwing — but it soft-fails HONESTLY, + // having actually attempted, rather than reporting a cached success. + expect(failed.ok).toBe(false); + expect(failed.diagnostics).toBeTruthy(); + expect(grants).toBe(2); + expect(memoCount()).toBe(0); + + // And the original identity coming back is not silently re-satisfied. + current = original; + aclSucceeds = true; + expect(await harden(target, { required: false })).toEqual({ ok: true }); + expect(grants).toBe(3); + }); + }); + }); +} From da74ae8564d398a49c8d47286e8120792c0ab905 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Wed, 5 Aug 2026 11:02:06 +0900 Subject: [PATCH 122/163] feat(codex): the switch now survives a restart MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Turning Codex off worked and then lasted until the next `ocx start`. The restore was fine; startup called syncModelsToCodex(port).catch(() => {}) unconditionally and put the routing straight back. That is the whole defect, and Grok's shipped toggle still has it. A one-key clientIntegrations object holds the intent. One key on purpose: a top-level codexEnabled would make every later client invent its own name and helpers, and the ten-key union recreated the coupling that failed two audits by forcing every phase to touch every client's write path. ABSENCE MEANS ON, and that is the load-bearing rule. A config from an older binary, an untouched one, and an explicit true are the same state — reading any of them as OFF would silently unroute someone who never asked for it. Only an explicit false is off, so a hand-edited "false" string degrades to ON rather than to OFF, and unknown sibling keys survive both the parse and a write. Re-enabling deletes the key instead of storing true, so an untouched config and a re-enabled one are byte-identical rather than merely equivalent. The gate is a function rather than an if inside handleStart. It lived in a 600-line startup path that binds sockets and installs services, so nothing could test it — and an untestable gate is how the unconditional version survived this long. It returns whether the sync ran, so 'skipped because the user said no' is distinguishable from 'ran and quietly failed'. The .catch stays and is asserted, not assumed: a provider fetch failing at startup must not stop the proxy coming up. Swallowing a failure to APPLY is tolerable. Swallowing the user's decision was not. Six mutations, each restored to 15 pass / 0 fail: absence read as OFF (4/7), re-enable storing true (10/1), gate removed (14/1), gate inverted (11/4), failure no longer swallowed (14/1), port not forwarded (14/1). --- src/cli/index.ts | 3 +- src/codex/desired-state.ts | 131 ++++++++++++++++++ src/config.ts | 14 ++ src/types.ts | 19 +++ tests/codex-desired-state.test.ts | 218 ++++++++++++++++++++++++++++++ 5 files changed, 384 insertions(+), 1 deletion(-) create mode 100644 src/codex/desired-state.ts create mode 100644 tests/codex-desired-state.test.ts diff --git a/src/cli/index.ts b/src/cli/index.ts index bef5872cd..4c3b301d8 100755 --- a/src/cli/index.ts +++ b/src/cli/index.ts @@ -41,6 +41,7 @@ import { maybeShowStarPrompt } from "./star-prompt"; import { scheduleCatalogPrewarm } from "./catalog-prewarm"; import { maybeShowUpdatePrompt } from "../update/notify"; import { syncModelsToCodex } from "../codex/sync"; +import { syncCodexOnStartIfEnabled } from "../codex/desired-state"; import { normalizeUpdateChannel, runGuiUpdateWorker } from "../update/job"; import { collectOrcaCodexHomeDiagnostic } from "../codex/home"; import { removeOwnedConfigState } from "../lib/config-ownership"; @@ -316,7 +317,7 @@ async function handleStart(options: { block?: boolean } = {}) { installShellHook(); await maybeShowStarPrompt(); // once-only Yes/No GitHub-star prompt on first interactive start - await syncModelsToCodex(port).catch(() => {}); + await syncCodexOnStartIfEnabled(port, config); if (!currentExternalCodexModelProvider() && !shouldInjectApiAuthHeader(config) && config.syncResumeHistory !== false) { historyGuardian = startHistoryMigrationGuardian(); } diff --git a/src/codex/desired-state.ts b/src/codex/desired-state.ts new file mode 100644 index 000000000..dd5edbb56 --- /dev/null +++ b/src/codex/desired-state.ts @@ -0,0 +1,131 @@ +/** + * Durable desired state for the native Codex integration. + * + * The switch itself was never the hard part — `ocx restore` already returns Codex + * to its native path without stopping the proxy. What was missing is that the + * decision did not survive: `ocx start` force-synced unconditionally, so an OFF + * lasted exactly until the next start. That is the defect this module closes, and + * it is the same one Grok's shipped toggle still has. + * + * ABSENT MEANS ON. A user who never touched a switch, a config written by an + * older binary, and an explicit `true` are the same state, and none of them may + * be read as "the user turned this off". Only an explicit `false` is OFF. + * + * This module does NOT own linearization. The plan that predates + * `src/codex/user-identity.ts` proposed a second per-home lock at + * `tmpdir()/opencodex-native-locks/sha256(home).sqlite`; that keys on the home + * alone, carries no proof of effective user, and would collide or split + * depending on the temp root. Convergence takes the write lock; this module only + * records intent through the config coordinator. + * + * Design record: devlog/_plan/260803_codex_desktop_toggle/030_desired_state.md. + */ +import { loadConfig, mutatePersistedConfig } from "../config"; +import type { OcxConfig } from "../types"; + +/** Injectable for tests; production passes the real sync. */ +export type CodexStartupSync = (port: number) => Promise; + +export type CodexDesiredStateResult = + | { readonly ok: true; readonly status: "committed" | "unchanged"; readonly enabled: boolean } + | { + readonly ok: false; + readonly reason: "missing" | "invalid" | "conflict"; + readonly retryable: boolean; + readonly message: string; + }; + +/** + * Is native Codex integration wanted? + * + * Takes the config rather than reading it, so a caller that already holds an + * admitted snapshot cannot accidentally answer from a fresher one — the whole + * point of admission is that one decision uses one set of bytes. + */ +export function codexIntegrationEnabled(config: Pick): boolean { + return config.clientIntegrations?.codex !== false; +} + +/** The same question when no snapshot is in hand. Reads the persisted config. */ +export function codexIntegrationEnabledNow(): boolean { + return codexIntegrationEnabled(loadConfig()); +} + +/** + * Persist the desired state, touching only that one field. + * + * Field-scoped on purpose: the callback runs on a freshly rebased snapshot inside + * the config coordinator, so a concurrent provider or model edit is preserved + * instead of being clobbered by a whole-config write built from a stale read. + */ +export function setCodexIntegrationEnabled(enabled: boolean): CodexDesiredStateResult { + const outcome = mutatePersistedConfig(config => { + const current = codexIntegrationEnabled(config); + if (current === enabled) return { changed: false, value: enabled }; + const integrations = { ...(config.clientIntegrations ?? {}) }; + if (enabled) { + // ON is the absence, not a stored `true`: writing `true` would make an + // untouched config and a re-enabled one differ in bytes for no reason, and + // every later reader has to treat them identically anyway. + delete integrations.codex; + } else { + integrations.codex = false; + } + // Drop the key entirely once nothing is left in it, so enabling twice does + // not leave `"clientIntegrations": {}` behind in the user's file. + if (Object.keys(integrations).length === 0) delete config.clientIntegrations; + else config.clientIntegrations = integrations; + return { changed: true, value: enabled }; + }); + + if (outcome.status !== "unavailable") { + return { ok: true, status: outcome.status, enabled }; + } + // `conflict` is the only retryable one: it means a competing writer won the + // rebase, so the same call can succeed. A missing or malformed config will fail + // identically forever, and telling a caller to retry that is how a UI ends up + // spinning on a problem only the user can fix. + const retryable = outcome.reason === "conflict"; + return { + ok: false, + reason: outcome.reason, + retryable, + message: outcome.reason === "conflict" + ? "Another process changed the config while this switch was being written." + : outcome.reason === "missing" + ? "No config file exists to record the switch in." + : "The config file is malformed; refusing to overwrite it.", + }; +} + +/** + * The startup gate, as a function rather than an `if` buried in `handleStart`. + * + * `ocx start` used to call `syncModelsToCodex(port).catch(() => {})` + * unconditionally, which is exactly why turning Codex off lasted until the next + * start: the restore worked, and then start put the routing straight back. It + * lived inline in a 600-line startup function that opens sockets and installs + * services, so nothing could test it — and an untestable gate is how the + * unconditional version survived this long. + * + * The `.catch` is deliberate and stays: a provider fetch failing at startup must + * not stop the proxy from coming up. Swallowing a failure to APPLY is tolerable. + * Swallowing the user's decision was not. + * + * Returns whether the sync ran, so a caller — or a test — can tell "skipped + * because the user turned it off" from "ran and quietly failed". + */ +export async function syncCodexOnStartIfEnabled( + port: number, + config: Pick, + sync: CodexStartupSync = defaultStartupSync, +): Promise { + if (!codexIntegrationEnabled(config)) return false; + await sync(port).catch(() => {}); + return true; +} + +async function defaultStartupSync(port: number): Promise { + const { syncModelsToCodex } = await import("./sync"); + return syncModelsToCodex(port); +} diff --git a/src/config.ts b/src/config.ts index bdfa73c5a..476c5969e 100644 --- a/src/config.ts +++ b/src/config.ts @@ -936,6 +936,19 @@ const apiKeyEntrySchema = z.object({ createdAt: z.string().catch(""), }).passthrough(); +/** + * Durable per-client intent. + * + * `.passthrough()` is load-bearing: a binary that only knows `codex` must not + * erase a key a later version wrote during a field-scoped mutation. And each key + * degrades on its own — a hand edit of `{"codex": "false", "future": false}` + * drops `codex` to absent (which reads as ON) and keeps `future`, rather than + * invalidating the object or, worse, the whole config. + */ +const clientIntegrationsSchema = z.object({ + codex: z.boolean().optional().catch(undefined), +}).passthrough(); + const configSchema = z.object({ port: z.number().int().min(0).max(65535).default(10100), managementUsageMaxReadBytes: z.number().int().positive().default(64 * 1024 * 1024), @@ -957,6 +970,7 @@ const configSchema = z.object({ // Invalid hand edits must not discard an otherwise usable config. Treat them as // pre-migration so startup can safely re-run the one-time normalization. googleAntigravityStaticCatalogVersion: z.literal(1).optional().catch(undefined), + clientIntegrations: clientIntegrationsSchema.optional().catch(undefined), providerContextCaps: z.record(z.string(), z.number().int().positive()).optional(), contextCapValue: z.number().int().positive().optional(), multiAgentGuidanceEnabled: z.boolean().optional(), diff --git a/src/types.ts b/src/types.ts index 0bbdc7e41..38c665ec0 100644 --- a/src/types.ts +++ b/src/types.ts @@ -530,6 +530,20 @@ export interface OcxApiKeyEntry { createdAt: string; } +/** + * Durable per-client intent. One key today, deliberately. + * + * A top-level `codexEnabled` would force every later client to invent an + * unrelated name and its own helpers; a ten-key union recreated the coupling + * that failed two audits, because every phase then had to touch every client's + * write path. A one-key object keeps the extension point without letting this + * phase claim ownership over a client it does not implement. + */ +export interface OcxClientIntegrationsConfig { + /** Durable desired state for native Codex. MISSING MEANS ON. */ + codex?: boolean; +} + export interface OcxConfig { port: number; /** Maximum usage-log bytes read for one management snapshot. */ @@ -542,6 +556,11 @@ export interface OcxConfig { googleAntigravityStaticCatalogVersion?: 1; /** Claude Code inbound + launcher settings. */ claudeCode?: OcxClaudeCodeConfig; + /** + * Per-client durable intent. This phase owns only `codex`; later phases extend + * one key at a time rather than widening a shared union. + */ + clientIntegrations?: OcxClientIntegrationsConfig; /** * Up to 5 routed model ids ("/") to feature FIRST in the injected Codex catalog. * Codex's spawn_agent only advertises the first 5 routed models, so this picks which 5 appear. diff --git a/tests/codex-desired-state.test.ts b/tests/codex-desired-state.test.ts new file mode 100644 index 000000000..a6b89347c --- /dev/null +++ b/tests/codex-desired-state.test.ts @@ -0,0 +1,218 @@ +/** + * Durable desired state for the native Codex integration. + * + * The switch was never the hard part — `ocx restore` already unroutes Codex + * without stopping the proxy. What was missing is that the decision did not + * survive a restart, because `ocx start` force-synced unconditionally. These + * tests pin the two halves of the fix: absence means ON, and only an explicit + * `false` gates the startup sync. + */ +import { afterEach, beforeEach, describe, expect, test } from "bun:test"; +import { mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +import { loadConfig, saveConfig } from "../src/config"; +import { + codexIntegrationEnabled, + codexIntegrationEnabledNow, + setCodexIntegrationEnabled, + syncCodexOnStartIfEnabled, +} from "../src/codex/desired-state"; +import type { OcxConfig } from "../src/types"; + +let testRoot = ""; +let previousOpencodexHome: string | undefined; + +function baseConfig(): OcxConfig { + return { port: 10100, providers: {}, defaultProvider: "openai" }; +} + +beforeEach(() => { + previousOpencodexHome = process.env.OPENCODEX_HOME; + testRoot = mkdtempSync(join(tmpdir(), "ocx-desired-state-")); + process.env.OPENCODEX_HOME = testRoot; +}); + +afterEach(() => { + if (previousOpencodexHome === undefined) delete process.env.OPENCODEX_HOME; + else process.env.OPENCODEX_HOME = previousOpencodexHome; + rmSync(testRoot, { recursive: true, force: true }); +}); + +describe("absence means ON", () => { + /** + * Three different configs, one meaning. A user who never touched a switch, a + * config written by a binary that predates this field, and an explicit `true` + * are the same state — and none of them may read as "turned off". Only an + * explicit `false` is OFF. + */ + const onCases: { name: string; config: OcxConfig }[] = [ + { name: "no clientIntegrations at all", config: baseConfig() }, + { name: "an empty clientIntegrations object", config: { ...baseConfig(), clientIntegrations: {} } }, + { name: "an explicit true", config: { ...baseConfig(), clientIntegrations: { codex: true } } }, + ]; + + for (const { name, config } of onCases) { + test(`${name} reads as enabled`, () => { + expect(codexIntegrationEnabled(config)).toBe(true); + }); + } + + test("only an explicit false is off", () => { + expect(codexIntegrationEnabled({ ...baseConfig(), clientIntegrations: { codex: false } })).toBe(false); + }); + + /** + * A hand edit of the wrong type must not be read as OFF. `"false"` is a string, + * and treating any non-true value as OFF would silently unroute a user who + * fat-fingered their config — the schema drops the bad key, and absence is ON. + */ + test("a malformed value degrades to ON rather than OFF, and keeps unknown keys", () => { + writeFileSync( + join(testRoot, "config.json"), + JSON.stringify({ + ...baseConfig(), + clientIntegrations: { codex: "false", "future-client": false }, + }, null, 2), + ); + const loaded = loadConfig(); + expect(codexIntegrationEnabled(loaded)).toBe(true); + // The key this binary does not understand survives the parse. + expect((loaded.clientIntegrations as Record | undefined)?.["future-client"]).toBe(false); + }); +}); + +describe("persisting the decision", () => { + test("turning it off is written, and survives a fresh read", () => { + saveConfig(baseConfig()); + expect(codexIntegrationEnabledNow()).toBe(true); + + const result = setCodexIntegrationEnabled(false); + expect(result).toMatchObject({ ok: true, status: "committed", enabled: false }); + expect(codexIntegrationEnabledNow()).toBe(false); + + // And it is on disk, not merely in a cache. + const raw = JSON.parse(readFileSync(join(testRoot, "config.json"), "utf8")) as Record; + expect((raw.clientIntegrations as Record).codex).toBe(false); + }); + + /** + * Re-enabling REMOVES the key rather than storing `true`. Otherwise an + * untouched config and a re-enabled one differ in bytes while meaning the same + * thing, and every later reader has to treat them identically anyway. + */ + test("turning it back on removes the key instead of storing true", () => { + saveConfig(baseConfig()); + setCodexIntegrationEnabled(false); + expect(setCodexIntegrationEnabled(true)).toMatchObject({ ok: true, status: "committed", enabled: true }); + + const raw = JSON.parse(readFileSync(join(testRoot, "config.json"), "utf8")) as Record; + expect(raw.clientIntegrations).toBeUndefined(); + expect(codexIntegrationEnabledNow()).toBe(true); + }); + + test("setting the state it is already in is unchanged, not a write", () => { + saveConfig(baseConfig()); + expect(setCodexIntegrationEnabled(true)).toMatchObject({ ok: true, status: "unchanged" }); + setCodexIntegrationEnabled(false); + expect(setCodexIntegrationEnabled(false)).toMatchObject({ ok: true, status: "unchanged" }); + }); + + /** + * The mutation is field-scoped. A whole-config write built from a stale read + * would clobber a provider edit that landed in between; this asserts the + * neighbouring field is still there afterwards. + */ + test("writing the switch preserves unrelated config the caller never saw", () => { + saveConfig({ + ...baseConfig(), + providers: { + openai: { + adapter: "openai-responses", + baseUrl: "https://chatgpt.com/backend-api/codex", + authMode: "forward", + }, + } as OcxConfig["providers"], + }); + setCodexIntegrationEnabled(false); + + const after = loadConfig(); + expect(Object.keys(after.providers)).toEqual(["openai"]); + expect(codexIntegrationEnabled(after)).toBe(false); + }); + + test("an unknown future key in the object is preserved across a write", () => { + writeFileSync( + join(testRoot, "config.json"), + JSON.stringify({ ...baseConfig(), clientIntegrations: { "future-client": false } }, null, 2), + ); + setCodexIntegrationEnabled(false); + + const raw = JSON.parse(readFileSync(join(testRoot, "config.json"), "utf8")) as Record; + const integrations = raw.clientIntegrations as Record; + expect(integrations.codex).toBe(false); + expect(integrations["future-client"]).toBe(false); + }); + + test("a missing config refuses rather than creating one", () => { + const result = setCodexIntegrationEnabled(false); + expect(result).toMatchObject({ ok: false, reason: "missing", retryable: false }); + }); +}); + +describe("the startup gate", () => { + /** + * This is the defect the whole phase exists for. `ocx start` called + * `syncModelsToCodex(port).catch(() => {})` unconditionally, so an OFF lasted + * exactly until the next start: restore unrouted Codex, and start put the + * routing straight back. + * + * It lived inline in a 600-line startup function that binds sockets and + * installs services, which is why nothing tested it — and an untestable gate is + * how the unconditional version survived. It is a function now. + */ + test("an explicit OFF skips the startup sync entirely", async () => { + let calls = 0; + const ran = await syncCodexOnStartIfEnabled( + 10100, + { clientIntegrations: { codex: false } }, + async () => { calls += 1; }, + ); + expect(ran).toBe(false); + expect(calls).toBe(0); + }); + + test("absence, an empty object, and an explicit true all still sync", async () => { + for (const clientIntegrations of [undefined, {}, { codex: true }]) { + let calls = 0; + const ran = await syncCodexOnStartIfEnabled( + 10100, + { clientIntegrations }, + async () => { calls += 1; }, + ); + expect(ran).toBe(true); + expect(calls).toBe(1); + } + }); + + test("the port reaches the sync", async () => { + const ports: number[] = []; + await syncCodexOnStartIfEnabled(43210, {}, async port => { ports.push(port); }); + expect(ports).toEqual([43210]); + }); + + /** + * The swallow stays, and is asserted rather than assumed. A provider fetch + * failing at startup must not stop the proxy from coming up — swallowing a + * failure to APPLY is tolerable, swallowing the user's DECISION was not. + */ + test("a sync failure is swallowed so startup continues, and still reports that it ran", async () => { + const ran = await syncCodexOnStartIfEnabled( + 10100, + {}, + async () => { throw new Error("provider unreachable"); }, + ); + expect(ran).toBe(true); + }); +}); From b3acb71d5fab4f6504536a82842db52d1a1100cc Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Wed, 5 Aug 2026 11:04:06 +0900 Subject: [PATCH 123/163] test(acl): closing the shortcut was not the same as closing the axis MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit I wrote 'every memo property is now parameterized over requiredness' while only the observable-mismatch property actually was. That is the projection mistake one level up — a claim about coverage rather than a claim about behavior. The other two memo properties had the same bypass, and both left every test green: an optional caller trusting a memo whose identity cannot be observed, and observed absence retiring the memo for required callers only. The suite is now the full cross-product — four entry points x {required, optional} x {stale identity, unobservable identity, observed absence} — with a shared expectRefused that throws for required and asserts {ok:false, diagnostics} for optional, so the same attribution is proven either way. 143 pass / 0 fail across three files. One note for whoever writes the next mutation: a bypass placed AFTER the absence handling is dead code and reddens nothing. My first attempt at the absence mutation 'survived' for exactly that reason, and it took reading the function to notice the mutation had never been reachable. Where a mutation lands matters as much as what it says. --- .../030_lock_protocol.md | 23 +- tests/windows-secret-acl.test.ts | 204 +++++++++++++----- 2 files changed, 169 insertions(+), 58 deletions(-) diff --git a/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md b/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md index 593505583..e0a7b0bb3 100644 --- a/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md +++ b/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md @@ -1033,8 +1033,10 @@ and the ones that did were rewritten rather than kept. proven wrong is kept instead of retired; (af) directory memos alone are exempted from that retirement; (ag) an unreadable observation keeps its memo while a successful re-harden masks which mechanism deleted it; (ah) an OPTIONAL caller is - allowed to trust a pathname-only memo. - (h) through (ah) are not redundant — each survived every other check. (h) and (i) + allowed to trust a pathname-only memo; (ai) an optional caller trusts it when the + identity is UNOBSERVABLE; (aj) observed absence retires the memo only for required + callers. + (h) through (aj) are not redundant — each survived every other check. (h) and (i) cover production callers the primitive tests missed: `hardenStableLockFile` takes the async path, and `hardenSecretDir` backs config, management-auth, tray, spill-store, and `native-profile-manager.ts:153`. (j) and (k) are a different @@ -1176,8 +1178,21 @@ and the ones that did were rewritten rather than kept. all 119 tests green, letting an optional caller accept any object at a previously hardened pathname without observing identity, retiring the memo, or running icacls. Optional means a failure is REPORTED rather than thrown; it does - not mean unattributed. Every memo property is now parameterized over - `required: true | false` as well as the four entry points. + not mean unattributed. + + And closing the broad shortcut was not the same as closing the axis — writing + "every memo property is now parameterized over requiredness" while only the + observable-mismatch property was, is the projection mistake at the level of a + claim about coverage. (ai) and (aj) are the other two memo properties with the + same bypass: an optional caller trusting a memo whose identity cannot be + observed, and observed absence retiring the memo for required callers only. + Both left every test green. The suite is now the full cross-product — four entry + points x {required, optional} x {stale identity, unobservable identity, observed + absence} — with a shared `expectRefused` that throws for required and asserts + `{ok:false, diagnostics}` for optional, so the same attribution is proven either + way. Note that a bypass placed AFTER the absence handling is dead code and + reddens nothing; the reachable form has to precede it, which is worth knowing + before concluding that a mutation "survives". The matrix, enumerated, is: exact target · default binding · platform provenance · **successful completion** · **ordering relative to the ACL** · failure propagation · diff --git a/tests/windows-secret-acl.test.ts b/tests/windows-secret-acl.test.ts index 9ca57e032..14ad33091 100644 --- a/tests/windows-secret-acl.test.ts +++ b/tests/windows-secret-acl.test.ts @@ -1559,72 +1559,168 @@ for (const { label, harden, create } of ENTRY_POINTS) { }); } +/** + * Memo attribution across the FULL cross-product: four entry points x both + * requiredness modes. + * + * Requiredness was invisible for a long time because every identity and + * retirement scenario called with `required: true` while the optional-policy + * tests used fresh paths with no memo — so neither could see an optional-only + * shortcut. Three separate shortcuts were then found, one per memo property: + * + * if (!opts.required && cache.has(path)) return { ok: true }; + * if (!opts.required && cache.has(path) && observe(path) === null) ... + * if (!opts.required && !existsSync(path) && cache.has(path)) ... + * + * All three left the whole suite green. Optional means a failure is REPORTED + * rather than thrown; it never means the memo may describe a different file, an + * unobservable one, or one that is gone. + */ for (const { label, harden, create } of ENTRY_POINTS) { const memoCount = label.startsWith("dir") ? hardenedSecretDirCountForTests : hardenedSecretPathCountForTests; - describe(`memo attribution holds for OPTIONAL callers too — ${label}`, () => { - /** - * The requiredness axis, which every other memo test left uncovered. - * - * Each identity and retirement scenario calls with `required: true`, and the - * optional-policy tests use fresh paths with no memo, so neither can see an - * optional-only bypass. Inserting - * `if (!opts.required && cache.has(targetPath)) return { ok: true }` before - * the lookup left all 119 tests green: an optional caller would then accept - * ANY object at a previously hardened pathname without observing identity, - * retiring the memo, or running icacls. - * - * Optional does not mean unattributed. It means a failure is reported rather - * than thrown — the memo still has to describe what is actually there. - */ - test("a stale memo does not satisfy an optional caller, and a failed re-harden still retires it", async () => { - resetHardenedStateForTests(); - const target = join(testDir, `optional-${label}.sqlite`); - create(target); + for (const required of [true, false]) { + const mode = required ? "required" : "optional"; + // A required caller throws; an optional one reports. Same attribution either way. + const expectRefused = async (target: string): Promise => { + if (required) { + await expect(harden(target, { required })).rejects.toThrow(); + return; + } + const result = await harden(target, { required }); + expect(result.ok).toBe(false); + expect(result.diagnostics).toBeTruthy(); + }; - await withWin32(async () => { - let grants = 0; - let aclSucceeds = true; - const result = () => (aclSucceeds - ? { success: true, exitCode: 0, timedOut: false, stdout: "" } - : { success: false, exitCode: 5, timedOut: false, stdout: "", stderr: "" }); - setIcaclsRunnerForTests(args => { - if (args.includes("/grant:r")) grants += 1; - return result(); + describe(`memo attribution — ${label} / ${mode}`, () => { + test("a stale memo does not answer, and a failed re-harden retires it", async () => { + resetHardenedStateForTests(); + const target = join(testDir, `attr-${label}-${mode}.sqlite`); + create(target); + + await withWin32(async () => { + let grants = 0; + let aclSucceeds = true; + const result = () => (aclSucceeds + ? { success: true, exitCode: 0, timedOut: false, stdout: "" } + : { success: false, exitCode: 5, timedOut: false, stdout: "", stderr: "" }); + setIcaclsRunnerForTests(args => { + if (args.includes("/grant:r")) grants += 1; + return result(); + }); + setAsyncIcaclsRunnerForTests(async args => { + if (args.includes("/grant:r")) grants += 1; + return result(); + }); + + const original = { dev: 1n, ino: 10n, ctimeNs: 100n }; + let current = original; + setStatForTests(() => current); + + expect(await harden(target, { required })).toEqual({ ok: true }); + expect(grants).toBe(1); + expect(memoCount()).toBe(1); + + // A different object at the same path. The re-harden FAILS, because a + // successful one would overwrite the memo and hide whether the miss + // retired it. + current = { dev: 1n, ino: 11n, ctimeNs: 500n }; + aclSucceeds = false; + await expectRefused(target); + expect(grants).toBe(2); + expect(memoCount()).toBe(0); + + // Restoring the original identity is not silently re-satisfied. + current = original; + aclSucceeds = true; + expect(await harden(target, { required })).toEqual({ ok: true }); + expect(grants).toBe(3); }); - setAsyncIcaclsRunnerForTests(async args => { - if (args.includes("/grant:r")) grants += 1; - return result(); + }); + + test("an unobservable identity does not answer, and the lookup is what retires it", async () => { + resetHardenedStateForTests(); + const target = join(testDir, `attr-unreadable-${label}-${mode}.sqlite`); + create(target); + + await withWin32(async () => { + let grants = 0; + let aclSucceeds = true; + const result = () => (aclSucceeds + ? { success: true, exitCode: 0, timedOut: false, stdout: "" } + : { success: false, exitCode: 5, timedOut: false, stdout: "", stderr: "" }); + setIcaclsRunnerForTests(args => { + if (args.includes("/grant:r")) grants += 1; + return result(); + }); + setAsyncIcaclsRunnerForTests(async args => { + if (args.includes("/grant:r")) grants += 1; + return result(); + }); + + const identity = { dev: 1n, ino: 10n, ctimeNs: 100n }; + let readable = true; + setStatForTests(() => { + if (!readable) throw Object.assign(new Error("EACCES"), { code: "EACCES" }); + return identity; + }); + + expect(await harden(target, { required })).toEqual({ ok: true }); + expect(grants).toBe(1); + expect(memoCount()).toBe(1); + + // Unreadable AND the ACL fails, so recordHarden never reaches its own + // deletion: whatever retires the entry here is the lookup. + readable = false; + aclSucceeds = false; + await expectRefused(target); + expect(grants).toBe(2); + expect(memoCount()).toBe(0); + + readable = true; + aclSucceeds = true; + expect(await harden(target, { required })).toEqual({ ok: true }); + expect(grants).toBe(3); }); + }); - const original = { dev: 1n, ino: 10n, ctimeNs: 100n }; - let current = original; - setStatForTests(() => current); + test("observed absence retires the memo", async () => { + resetHardenedStateForTests(); + const target = join(testDir, `attr-absent-${label}-${mode}.sqlite`); + create(target); - expect(await harden(target, { required: false })).toEqual({ ok: true }); - expect(grants).toBe(1); - expect(memoCount()).toBe(1); + await withWin32(async () => { + let grants = 0; + setIcaclsRunnerForTests(args => { + if (args.includes("/grant:r")) grants += 1; + return { success: true, exitCode: 0, timedOut: false, stdout: "" }; + }); + setAsyncIcaclsRunnerForTests(async args => { + if (args.includes("/grant:r")) grants += 1; + return { success: true, exitCode: 0, timedOut: false, stdout: "" }; + }); + const identity = { dev: 1n, ino: 10n, ctimeNs: 100n }; + setStatForTests(() => identity); + + expect(await harden(target, { required })).toEqual({ ok: true }); + expect(grants).toBe(1); + expect(memoCount()).toBe(1); - // A different object at the same path: the memo must not answer, even - // for an optional caller. - current = { dev: 1n, ino: 11n, ctimeNs: 500n }; - aclSucceeds = false; - const failed = await harden(target, { required: false }); - // Optional soft-fails rather than throwing — but it soft-fails HONESTLY, - // having actually attempted, rather than reporting a cached success. - expect(failed.ok).toBe(false); - expect(failed.diagnostics).toBeTruthy(); - expect(grants).toBe(2); - expect(memoCount()).toBe(0); + // Gone. A harden of an absent path is a no-op that must still forget it. + rmSync(target, { recursive: true, force: true }); + expect(await harden(target, { required })).toEqual({ ok: true }); + expect(grants).toBe(1); + expect(memoCount()).toBe(0); - // And the original identity coming back is not silently re-satisfied. - current = original; - aclSucceeds = true; - expect(await harden(target, { required: false })).toEqual({ ok: true }); - expect(grants).toBe(3); + // Back at the SAME identity: it must harden again rather than be + // satisfied by an entry that outlived the file it described. + create(target); + expect(await harden(target, { required })).toEqual({ ok: true }); + expect(grants).toBe(2); + }); }); }); - }); + } } From 1553b2d0cca384ee9ce3497592c32ca70cb161d3 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Wed, 5 Aug 2026 11:08:04 +0900 Subject: [PATCH 124/163] feat(codex): the Codex card gets a switch, and turning it off keeps the proxy up MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit codexRow carried toggle: null, so Codex was the one client on the integrations page you could not actually switch. It has a route now, modelled on the Grok one: PUT /api/native-integrations/codex. Turning Codex off is NOT ocx stop. The user asked for this in those words — they may want everything except Codex routed — so the route restores the native path and leaves the process serving. Asserted rather than described: the management surface still answers afterwards, an unrelated native client is still readable, and the switch is immediately reusable, which a wedged in-flight guard would turn into a 409. Intent is persisted BEFORE artifacts converge. A process that dies between the two leaves a decision the next start can act on; converging first and dying would leave artifacts the next start silently undoes. The port comes from the runtime state of the process that actually bound it, not from config.json — a stale recorded port injects a base_url pointing at nothing, which is the trap the Grok flight already documents. Re-enabling deletes the key rather than storing true, so an untouched config and a re-enabled one are byte-identical. Three mutations, restored to 21 pass / 0 fail: intent never persisted (18/3), non-boolean enabled accepted (19/2), in-flight guard never cleared (16/5). --- .../integrations/IntegrationsOverview.tsx | 4 +- .../pages/integrations/overview-clients.ts | 2 +- .../management/native-integration-routes.ts | 98 ++++++++++- tests/native-codex-toggle.test.ts | 158 ++++++++++++++++++ 4 files changed, 258 insertions(+), 4 deletions(-) create mode 100644 tests/native-codex-toggle.test.ts diff --git a/gui/src/pages/integrations/IntegrationsOverview.tsx b/gui/src/pages/integrations/IntegrationsOverview.tsx index 6dbb61dbd..cc063bd5a 100644 --- a/gui/src/pages/integrations/IntegrationsOverview.tsx +++ b/gui/src/pages/integrations/IntegrationsOverview.tsx @@ -410,7 +410,7 @@ export default function IntegrationsOverview({ if (row.status) { await toggleIntegration(apiBase, row.status.clientId, next); refresh(); - } else if (row.toggle === "claude" || row.toggle === "grok") { + } else if (row.toggle === "claude" || row.toggle === "grok" || row.toggle === "codex") { const result = await toggleNativeIntegration(apiBase, row.toggle, next); if (result.reason === "non_loopback_removed") { setCardResult(row.id, { @@ -429,7 +429,7 @@ export default function IntegrationsOverview({ tone: "err", text: describeRefusal(t, error, undefined, row.togglePath ?? undefined), }); - if (row.toggle === "claude" || row.toggle === "grok") refreshNativeDetails(); + if (row.toggle === "claude" || row.toggle === "grok" || row.toggle === "codex") refreshNativeDetails(); } finally { setCardPending(null); } diff --git a/gui/src/pages/integrations/overview-clients.ts b/gui/src/pages/integrations/overview-clients.ts index f0d190e6e..6fe89d5ec 100644 --- a/gui/src/pages/integrations/overview-clients.ts +++ b/gui/src/pages/integrations/overview-clients.ts @@ -152,7 +152,7 @@ function codexRow(payload: CodexRoutingPayload | null): OverviewRow { id: "codex" as const, hash: "integrations/codex", labelKey: "integrations.tab.codex" as TKey, - toggle: null, + toggle: "codex" as const, toggleBlocked: null, togglePath: null, status: null, diff --git a/src/server/management/native-integration-routes.ts b/src/server/management/native-integration-routes.ts index 521cef2ea..097689997 100644 --- a/src/server/management/native-integration-routes.ts +++ b/src/server/management/native-integration-routes.ts @@ -28,7 +28,7 @@ import { jsonResponse } from "../auth-cors"; import { readManagementJsonBody, rethrowManagementBodyTooLarge } from "./body"; import type { ManagementContext } from "./context"; -export type NativeIntegrationClientId = "claude" | "grok"; +export type NativeIntegrationClientId = "claude" | "grok" | "codex"; /** Every reason this module can decline, in one place (audit r3 #6). */ export type NativeRefusalReason = @@ -191,6 +191,98 @@ const ORPHANED_MARKER_MESSAGE = const NOT_INSTALLED_MESSAGE = "Grok home was not found, so there is nothing to change. Install Grok Build first."; +/** + * One Codex change at a time, for the same reason Grok has this: the guard + * stands BEFORE the first await, or two concurrent PUTs both pass it while their + * bodies are still parsing. + */ +let codexToggleFlight: Promise | null = null; + +/** + * Turn native Codex routing on or off. + * + * THE PROXY STAYS UP. Turning Codex off is not `ocx stop`: other clients keep + * routing through this process, `/healthz` keeps answering, and only Codex goes + * back to its own path. That is the entire point of having a per-client switch + * rather than a kill switch, and it is why this route restores rather than + * stopping anything. + * + * Two writes, in this order: + * 1. persist the desired state, so the decision survives the next `ocx start`; + * 2. converge the artifacts to match it. + * + * Intent first is deliberate. If the process dies between them, the next start + * reads the persisted intent and converges — whereas converging first and dying + * would leave artifacts the next start silently undoes. + */ +async function handleCodexToggle(ctx: ManagementContext): Promise { + const { req } = ctx; + if (codexToggleFlight) { + return refusal(409, "codex", "config_busy", + "Another Codex change is already in flight. Nothing was written — try again in a moment."); + } + codexToggleFlight = (async (): Promise => { + let body: { enabled?: unknown }; + try { + body = await readManagementJsonBody(req); + } catch (error) { + rethrowManagementBodyTooLarge(error); + return jsonResponse({ error: "invalid JSON body" }, 400); + } + if (typeof body.enabled !== "boolean") { + return jsonResponse({ error: "enabled must be a boolean" }, 400); + } + const enabled = body.enabled; + + const { setCodexIntegrationEnabled } = await import("../../codex/desired-state"); + const persisted = setCodexIntegrationEnabled(enabled); + if (!persisted.ok) { + const status = persisted.retryable ? 409 : persisted.reason === "missing" ? 409 : 500; + return refusal( + status, + "codex", + persisted.retryable ? "config_busy" : "write_failed", + persisted.message, + ); + } + + if (enabled) { + // The port this process actually BOUND, not what config.json last recorded. + // A stale config port would inject a base_url pointing at nothing — the + // same trap `runGrokApplyFlight` documents below. + const runtime = (ctx.deps.readRuntimePort ?? readRuntimePort)(process.pid); + const port = runtime?.port ?? ctx.config.port; + const { syncModelsToCodex } = await import("../../codex/sync"); + const applied = await syncModelsToCodex(port); + return jsonResponse({ + ok: true, clientId: "codex", changed: persisted.status === "committed", + state: applied.ok ? "current" : "absent", + message: applied.ok + ? "Codex now routes through opencodex" + : `Codex intent saved, but applying it did not complete: ${applied.message}`, + ...(applied.ok ? {} : { reason: "apply_incomplete" }), + } satisfies NativeToggleEnvelope); + } + + // OFF. Restore the native path; the proxy keeps serving every other client. + const { restoreNativeCodexAsync } = await import("../../codex/inject"); + const restored = await restoreNativeCodexAsync(); + return jsonResponse({ + ok: true, clientId: "codex", changed: persisted.status === "committed", + state: restored.success ? "absent" : "unsafe", + message: restored.success + ? "Codex restored to its native path; the proxy is still serving other clients" + : `Codex intent saved, but restoring the native path did not complete: ${restored.message}`, + ...(restored.success ? {} : { reason: "restore_incomplete" }), + } satisfies NativeToggleEnvelope); + })(); + try { + return await codexToggleFlight; + } finally { + codexToggleFlight = null; + } +} + async function handleGrokToggle(ctx: ManagementContext): Promise { const { req, config, deps } = ctx; /* @@ -442,5 +534,9 @@ export async function handleNativeIntegrationRoutes(ctx: ManagementContext): Pro return handleGrokToggle(ctx); } + if (url.pathname === "/api/native-integrations/codex" && req.method === "PUT") { + return handleCodexToggle(ctx); + } + return null; } diff --git a/tests/native-codex-toggle.test.ts b/tests/native-codex-toggle.test.ts new file mode 100644 index 000000000..92d006857 --- /dev/null +++ b/tests/native-codex-toggle.test.ts @@ -0,0 +1,158 @@ +/** + * The Codex switch route. + * + * The thing worth proving here is not that a boolean lands in a file. It is that + * turning Codex OFF is not `ocx stop`: the proxy keeps serving, `/healthz` keeps + * answering, other clients keep routing, and only Codex goes back to its own + * path. A per-client switch that took the whole proxy down would be a kill + * switch with a misleading label. + * + * The second thing is ordering. Intent is persisted BEFORE artifacts converge, + * so a process that dies between the two leaves a decision the next start can + * act on — rather than artifacts the next start silently undoes. + */ +import { afterEach, beforeEach, describe, expect, test } from "bun:test"; +import { mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { homedir, tmpdir } from "node:os"; +import { join } from "node:path"; + +import { handleManagementAPI } from "../src/server/management-api"; +import type { ManagementApiDeps } from "../src/server/management/context"; +import type { OcxConfig } from "../src/types"; + +let fixtureRoot = ""; +let previousOpencodexHome: string | undefined; +const cleanup: string[] = []; + +function baseConfig(): OcxConfig { + return { + port: 10100, + providers: { + openai: { + adapter: "openai-responses", + baseUrl: "https://chatgpt.com/backend-api/codex", + authMode: "forward", + }, + } as OcxConfig["providers"], + defaultProvider: "openai", + }; +} + +function testDeps(overrides: Partial = {}): ManagementApiDeps { + return { + fetchAllModels: async () => [] as never, + ...overrides, + } as ManagementApiDeps; +} + +function dispatch(config: OcxConfig, path: string, init?: RequestInit, deps: ManagementApiDeps = testDeps()) { + const url = new URL(`http://127.0.0.1:10100${path}`); + return handleManagementAPI( + new Request(url, { ...init, headers: { Host: url.host, ...(init?.headers ?? {}) } }), + url, + config, + deps, + ); +} + +async function put(config: OcxConfig, body: unknown, deps?: ManagementApiDeps) { + const res = await dispatch(config, "/api/native-integrations/codex", { + method: "PUT", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify(body), + }, deps); + return { status: res!.status, body: await res!.json() as Record }; +} + +function persistedCodexIntent(): unknown { + const raw = JSON.parse(readFileSync(join(fixtureRoot, "config.json"), "utf8")) as Record; + return (raw.clientIntegrations as Record | undefined)?.codex; +} + +beforeEach(() => { + previousOpencodexHome = process.env.OPENCODEX_HOME; + fixtureRoot = mkdtempSync(join(tmpdir(), "ocx-codex-toggle-")); + cleanup.push(fixtureRoot); + process.env.OPENCODEX_HOME = fixtureRoot; + writeFileSync(join(fixtureRoot, "config.json"), JSON.stringify(baseConfig(), null, 2)); + writeFileSync(join(fixtureRoot, "service-state.json"), JSON.stringify({ + version: 2, + codexHome: process.env.CODEX_HOME?.trim() || join(homedir(), ".codex"), + opencodexHome: fixtureRoot, + backend: "scheduler", + })); +}); + +afterEach(() => { + if (previousOpencodexHome === undefined) delete process.env.OPENCODEX_HOME; + else process.env.OPENCODEX_HOME = previousOpencodexHome; + while (cleanup.length) rmSync(cleanup.pop()!, { recursive: true, force: true }); +}); + +describe("request validation", () => { + test("a non-boolean enabled is rejected before anything is written", async () => { + const result = await put(baseConfig(), { enabled: "false" }); + expect(result.status).toBe(400); + expect(persistedCodexIntent()).toBeUndefined(); + }); + + test("a missing enabled is rejected before anything is written", async () => { + const result = await put(baseConfig(), {}); + expect(result.status).toBe(400); + expect(persistedCodexIntent()).toBeUndefined(); + }); +}); + +describe("turning Codex off", () => { + test("persists the decision so it survives the next start", async () => { + const result = await put(baseConfig(), { enabled: false }); + expect(result.status).toBe(200); + expect(result.body).toMatchObject({ ok: true, clientId: "codex" }); + // The decision is on disk. Without this, an OFF lasts until the next + // `ocx start` re-syncs over it, which is the defect this phase exists for. + expect(persistedCodexIntent()).toBe(false); + }); + + /** + * THE PROXY STAYS UP. The user asked for this in exactly these terms: they may + * want everything except Codex routed. If turning Codex off stopped the proxy, + * every other client would lose its routing too. + */ + test("the management API keeps serving other routes afterwards", async () => { + await put(baseConfig(), { enabled: false }); + + // `/healthz` is served by the request router rather than the management API, + // so the observable claim HERE is that the management surface itself is + // still answering — the route did not tear the process down or leave a + // lock/flight wedged behind it. + const others = await dispatch(baseConfig(), "/api/native-integrations"); + expect(others!.status).toBe(200); + const body = await others!.json() as { clients: { clientId: string }[] }; + expect(body.clients.some(c => c.clientId === "claude")).toBe(true); + + // And the switch is re-usable immediately: a wedged in-flight guard would + // return 409 here rather than a normal answer. + const again = await put(baseConfig(), { enabled: false }); + expect(again.status).toBe(200); + }); + + test("turning it off twice is honest about the second one changing nothing", async () => { + const first = await put(baseConfig(), { enabled: false }); + const second = await put(baseConfig(), { enabled: false }); + expect(first.body.changed).toBe(true); + expect(second.body.changed).toBe(false); + expect(persistedCodexIntent()).toBe(false); + }); +}); + +describe("turning Codex back on", () => { + test("removes the key rather than storing true", async () => { + await put(baseConfig(), { enabled: false }); + expect(persistedCodexIntent()).toBe(false); + + const result = await put(baseConfig(), { enabled: true }); + expect(result.status).toBe(200); + // Absence is ON, so an untouched config and a re-enabled one are identical. + expect(persistedCodexIntent()).toBeUndefined(); + }); +}); From d80a045d92dcfb5f9815d0dd05f3a7a1745eed71 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Wed, 5 Aug 2026 11:13:18 +0900 Subject: [PATCH 125/163] test(acl): 'how the memo can be wrong' was two axes, not one MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Stale identity hid WHICH component moved: a memo comparing dev:ino while ignoring freshness passed a matrix whose stale case moved ino and ctimeNs together. Unobservable identity hid WHY: a thrown stat and a zero inode both produce observe() === null, and only the thrown case was ever driven — leaving the zero-inode form, the one NTFS is reported to produce, untested on the platform this module exists for. The matrix now enumerates {dev-only, ino-only, freshness-only} and {stat throws, zero inode} explicitly, across four entry points and both requiredness modes. 167 pass / 0 fail across three files; (ak) reddens 4 tests and (al) reddens 8. Method note, because it cost me two wrong readings this round: both mutations first appeared to survive, and both times the mutation was unreachable — placed after the branch that already returns. That is the second time in two rounds that a 'surviving' mutation was actually dead code. A mutation that does not execute is indistinguishable from a mechanism that is well tested, which makes it the same failure mode as everything else in this unit: an absence read as a guarantee. --- .../030_lock_protocol.md | 22 +- tests/windows-secret-acl.test.ts | 206 +++++++++--------- 2 files changed, 120 insertions(+), 108 deletions(-) diff --git a/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md b/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md index e0a7b0bb3..c0d65bf1f 100644 --- a/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md +++ b/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md @@ -1035,8 +1035,10 @@ and the ones that did were rewritten rather than kept. successful re-harden masks which mechanism deleted it; (ah) an OPTIONAL caller is allowed to trust a pathname-only memo; (ai) an optional caller trusts it when the identity is UNOBSERVABLE; (aj) observed absence retires the memo only for required - callers. - (h) through (aj) are not redundant — each survived every other check. (h) and (i) + callers; (ak) an optional caller compares the object but ignores freshness; + (al) an optional caller accepts an unobservable identity when the cause is a + zero inode rather than a thrown stat. + (h) through (al) are not redundant — each survived every other check. (h) and (i) cover production callers the primitive tests missed: `hardenStableLockFile` takes the async path, and `hardenSecretDir` backs config, management-auth, tray, spill-store, and `native-profile-manager.ts:153`. (j) and (k) are a different @@ -1187,10 +1189,18 @@ and the ones that did were rewritten rather than kept. same bypass: an optional caller trusting a memo whose identity cannot be observed, and observed absence retiring the memo for required callers only. Both left every test green. The suite is now the full cross-product — four entry - points x {required, optional} x {stale identity, unobservable identity, observed - absence} — with a shared `expectRefused` that throws for required and asserts - `{ok:false, diagnostics}` for optional, so the same attribution is proven either - way. Note that a bypass placed AFTER the absence handling is dead code and + points x {required, optional} x how the memo can be wrong — with a shared + `expectRefused` that throws for required and asserts `{ok:false, diagnostics}` + for optional, so the same attribution is proven either way. + + That last axis turned out to be two axes, which is (ak) and (al). "Stale + identity" hid WHICH component moved: a memo comparing `dev:ino` while ignoring + freshness passed a matrix whose stale case moved `ino` and `ctimeNs` together. + "Unobservable identity" hid WHY: a thrown stat and a zero inode both produce + `observe() === null`, and only the thrown case was driven — leaving the zero-inode + form, the one NTFS is reported to produce, untested on the platform this module + exists for. The matrix now enumerates `{dev-only, ino-only, freshness-only}` and + `{stat throws, zero inode}` explicitly. Note that a bypass placed AFTER the absence handling is dead code and reddens nothing; the reachable form has to precede it, which is worth knowing before concluding that a mutation "survives". diff --git a/tests/windows-secret-acl.test.ts b/tests/windows-secret-acl.test.ts index 14ad33091..26e07df0f 100644 --- a/tests/windows-secret-acl.test.ts +++ b/tests/windows-secret-acl.test.ts @@ -1560,22 +1560,36 @@ for (const { label, harden, create } of ENTRY_POINTS) { } /** - * Memo attribution across the FULL cross-product: four entry points x both - * requiredness modes. + * Memo attribution across the full cross-product. * - * Requiredness was invisible for a long time because every identity and - * retirement scenario called with `required: true` while the optional-policy - * tests used fresh paths with no memo — so neither could see an optional-only - * shortcut. Three separate shortcuts were then found, one per memo property: + * Four public entry points x {required, optional} x how the memo can be wrong. + * That last axis is itself two axes, which took two rounds to see: * - * if (!opts.required && cache.has(path)) return { ok: true }; - * if (!opts.required && cache.has(path) && observe(path) === null) ... - * if (!opts.required && !existsSync(path) && cache.has(path)) ... + * - WHICH identity component moved: dev alone, ino alone, freshness alone. A + * memo that ignores freshness but still compares dev:ino passed a matrix + * whose "stale" case moved both ino and ctimeNs together. + * - WHY the identity is unobservable: a thrown stat, or a zero inode. Both + * produce observe() === null, and the zero-inode case is the one NTFS is + * reported to produce — so testing only EACCES leaves the platform this + * module exists for uncovered. * - * All three left the whole suite green. Optional means a failure is REPORTED + * Requiredness runs through all of it. Optional means a failure is REPORTED * rather than thrown; it never means the memo may describe a different file, an * unobservable one, or one that is gone. */ +const IDENTITY_MOVES = [ + { name: "dev-only", to: { dev: 2n, ino: 10n, ctimeNs: 100n } }, + { name: "ino-only", to: { dev: 1n, ino: 11n, ctimeNs: 100n } }, + { name: "freshness-only", to: { dev: 1n, ino: 10n, ctimeNs: 200n } }, +] as const; + +const UNOBSERVABLE = [ + { name: "stat throws", stat: () => { throw Object.assign(new Error("EACCES"), { code: "EACCES" }); } }, + // NTFS is reported to return a zero inode from this stat form; observe() + // treats that as unobservable, and it had only ever been driven required. + { name: "zero inode", stat: () => ({ dev: 1n, ino: 0n, ctimeNs: 100n }) }, +] as const; + for (const { label, harden, create } of ENTRY_POINTS) { const memoCount = label.startsWith("dir") ? hardenedSecretDirCountForTests @@ -1583,7 +1597,6 @@ for (const { label, harden, create } of ENTRY_POINTS) { for (const required of [true, false]) { const mode = required ? "required" : "optional"; - // A required caller throws; an optional one reports. Same attribution either way. const expectRefused = async (target: string): Promise => { if (required) { await expect(harden(target, { required })).rejects.toThrow(); @@ -1594,97 +1607,93 @@ for (const { label, harden, create } of ENTRY_POINTS) { expect(result.diagnostics).toBeTruthy(); }; + /** Both runner seams, so one body drives the sync and async entry points. */ + const runner = (onGrant: () => void, succeeds: () => boolean): void => { + const result = () => (succeeds() + ? { success: true, exitCode: 0, timedOut: false, stdout: "" } + : { success: false, exitCode: 5, timedOut: false, stdout: "", stderr: "" }); + setIcaclsRunnerForTests(args => { + if (args.includes("/grant:r")) onGrant(); + return result(); + }); + setAsyncIcaclsRunnerForTests(async args => { + if (args.includes("/grant:r")) onGrant(); + return result(); + }); + }; + describe(`memo attribution — ${label} / ${mode}`, () => { - test("a stale memo does not answer, and a failed re-harden retires it", async () => { - resetHardenedStateForTests(); - const target = join(testDir, `attr-${label}-${mode}.sqlite`); - create(target); + for (const move of IDENTITY_MOVES) { + test(`a ${move.name} change retires the memo even when the re-harden fails`, async () => { + resetHardenedStateForTests(); + const target = join(testDir, `attr-${move.name}-${label}-${mode}.sqlite`); + create(target); - await withWin32(async () => { - let grants = 0; - let aclSucceeds = true; - const result = () => (aclSucceeds - ? { success: true, exitCode: 0, timedOut: false, stdout: "" } - : { success: false, exitCode: 5, timedOut: false, stdout: "", stderr: "" }); - setIcaclsRunnerForTests(args => { - if (args.includes("/grant:r")) grants += 1; - return result(); - }); - setAsyncIcaclsRunnerForTests(async args => { - if (args.includes("/grant:r")) grants += 1; - return result(); + await withWin32(async () => { + let grants = 0; + let aclSucceeds = true; + runner(() => { grants += 1; }, () => aclSucceeds); + + const original = { dev: 1n, ino: 10n, ctimeNs: 100n }; + let current: { dev: bigint; ino: bigint; ctimeNs: bigint } = original; + setStatForTests(() => current); + + expect(await harden(target, { required })).toEqual({ ok: true }); + expect(grants).toBe(1); + expect(memoCount()).toBe(1); + + // One component moves, and the re-harden FAILS — a successful one + // would overwrite the memo and hide whether the miss retired it. + current = move.to; + aclSucceeds = false; + await expectRefused(target); + expect(grants).toBe(2); + expect(memoCount()).toBe(0); + + // Restoring the original identity is not silently re-satisfied. + current = original; + aclSucceeds = true; + expect(await harden(target, { required })).toEqual({ ok: true }); + expect(grants).toBe(3); }); - - const original = { dev: 1n, ino: 10n, ctimeNs: 100n }; - let current = original; - setStatForTests(() => current); - - expect(await harden(target, { required })).toEqual({ ok: true }); - expect(grants).toBe(1); - expect(memoCount()).toBe(1); - - // A different object at the same path. The re-harden FAILS, because a - // successful one would overwrite the memo and hide whether the miss - // retired it. - current = { dev: 1n, ino: 11n, ctimeNs: 500n }; - aclSucceeds = false; - await expectRefused(target); - expect(grants).toBe(2); - expect(memoCount()).toBe(0); - - // Restoring the original identity is not silently re-satisfied. - current = original; - aclSucceeds = true; - expect(await harden(target, { required })).toEqual({ ok: true }); - expect(grants).toBe(3); }); - }); - - test("an unobservable identity does not answer, and the lookup is what retires it", async () => { - resetHardenedStateForTests(); - const target = join(testDir, `attr-unreadable-${label}-${mode}.sqlite`); - create(target); + } - await withWin32(async () => { - let grants = 0; - let aclSucceeds = true; - const result = () => (aclSucceeds - ? { success: true, exitCode: 0, timedOut: false, stdout: "" } - : { success: false, exitCode: 5, timedOut: false, stdout: "", stderr: "" }); - setIcaclsRunnerForTests(args => { - if (args.includes("/grant:r")) grants += 1; - return result(); - }); - setAsyncIcaclsRunnerForTests(async args => { - if (args.includes("/grant:r")) grants += 1; - return result(); - }); + for (const cause of UNOBSERVABLE) { + test(`an identity unobservable because ${cause.name} does not answer`, async () => { + resetHardenedStateForTests(); + const target = join(testDir, `attr-unobs-${cause.name.replace(/\s+/g, "-")}-${label}-${mode}.sqlite`); + create(target); - const identity = { dev: 1n, ino: 10n, ctimeNs: 100n }; - let readable = true; - setStatForTests(() => { - if (!readable) throw Object.assign(new Error("EACCES"), { code: "EACCES" }); - return identity; + await withWin32(async () => { + let grants = 0; + let aclSucceeds = true; + runner(() => { grants += 1; }, () => aclSucceeds); + + let observable = true; + setStatForTests(() => (observable + ? { dev: 1n, ino: 10n, ctimeNs: 100n } + : cause.stat())); + + expect(await harden(target, { required })).toEqual({ ok: true }); + expect(grants).toBe(1); + expect(memoCount()).toBe(1); + + // Unobservable AND the ACL fails, so recordHarden never reaches its + // own deletion: whatever retires the entry here is the lookup. + observable = false; + aclSucceeds = false; + await expectRefused(target); + expect(grants).toBe(2); + expect(memoCount()).toBe(0); + + observable = true; + aclSucceeds = true; + expect(await harden(target, { required })).toEqual({ ok: true }); + expect(grants).toBe(3); }); - - expect(await harden(target, { required })).toEqual({ ok: true }); - expect(grants).toBe(1); - expect(memoCount()).toBe(1); - - // Unreadable AND the ACL fails, so recordHarden never reaches its own - // deletion: whatever retires the entry here is the lookup. - readable = false; - aclSucceeds = false; - await expectRefused(target); - expect(grants).toBe(2); - expect(memoCount()).toBe(0); - - readable = true; - aclSucceeds = true; - expect(await harden(target, { required })).toEqual({ ok: true }); - expect(grants).toBe(3); }); - }); + } test("observed absence retires the memo", async () => { resetHardenedStateForTests(); @@ -1693,14 +1702,7 @@ for (const { label, harden, create } of ENTRY_POINTS) { await withWin32(async () => { let grants = 0; - setIcaclsRunnerForTests(args => { - if (args.includes("/grant:r")) grants += 1; - return { success: true, exitCode: 0, timedOut: false, stdout: "" }; - }); - setAsyncIcaclsRunnerForTests(async args => { - if (args.includes("/grant:r")) grants += 1; - return { success: true, exitCode: 0, timedOut: false, stdout: "" }; - }); + runner(() => { grants += 1; }, () => true); const identity = { dev: 1n, ino: 10n, ctimeNs: 100n }; setStatForTests(() => identity); From 10e6ad5cbe8644526728f4ccc1d9bc6601992781 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Wed, 5 Aug 2026 11:17:44 +0900 Subject: [PATCH 126/163] feat(grok): the toggle that already worked now survives a restart too MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Grok's switch shipped and lasted exactly one restart. It strips the fence from ~/.grok/config.toml and recorded nothing, so ocx start called syncGrokConfig unconditionally and wrote it straight back. Same defect Codex had, in a different file — which is why the roadmap listed it as its own phase rather than folding it into the Codex work. The intent map takes a second key. One key per client, still: setIntegrationEnabled takes the client id, and the named codex/grok wrappers sit on top, so neither client can reach the other's field. Asserted, not assumed — turning Codex off leaves Grok on, and re-enabling one leaves the other alone. One thing this surfaced in the Codex route I had just written: refusing the whole switch because no config file exists yet. That is a normal state for someone who has never saved settings, and it would make the button dead for exactly the users least able to work out why. A missing config no longer blocks either route — the artifact change still happens and still reports honestly, and the envelope carries reason: not_durable rather than implying success. conflict and invalid still refuse: another writer won, or the file is malformed and must not be overwritten. Two mutations, restored to 43 pass / 0 fail: the Grok startup gate removed (42/1), and the client key hardcoded to codex so grok writes the wrong field (42/1). --- src/cli/index.ts | 7 ++- src/codex/desired-state.ts | 58 ++++++++++++++++-- src/config.ts | 1 + .../management/native-integration-routes.ts | 59 +++++++++++++++--- src/types.ts | 2 + tests/codex-desired-state.test.ts | 60 +++++++++++++++++++ 6 files changed, 172 insertions(+), 15 deletions(-) diff --git a/src/cli/index.ts b/src/cli/index.ts index 4c3b301d8..99ef01d2c 100755 --- a/src/cli/index.ts +++ b/src/cli/index.ts @@ -41,7 +41,7 @@ import { maybeShowStarPrompt } from "./star-prompt"; import { scheduleCatalogPrewarm } from "./catalog-prewarm"; import { maybeShowUpdatePrompt } from "../update/notify"; import { syncModelsToCodex } from "../codex/sync"; -import { syncCodexOnStartIfEnabled } from "../codex/desired-state"; +import { shouldSyncGrokOnStart, syncCodexOnStartIfEnabled } from "../codex/desired-state"; import { normalizeUpdateChannel, runGuiUpdateWorker } from "../update/job"; import { collectOrcaCodexHomeDiagnostic } from "../codex/home"; import { removeOwnedConfigState } from "../lib/config-ownership"; @@ -337,7 +337,10 @@ async function handleStart(options: { block?: boolean } = {}) { // absent or the bind is non-loopback; removed again by stop/eject/uninstall/shutdown. // Deliberately a SIBLING of the Desktop-3P block above: nesting it there meant a catalog // failure skipped the fence entirely, even though syncGrokConfig handles that case itself. - try { + // + // Gated on the persisted switch: without this, turning Grok off lasted exactly + // one restart, because the toggle removed the fence and start wrote it back. + if (shouldSyncGrokOnStart(config)) try { const { syncGrokConfig } = await import("../grok/sync"); const r = await syncGrokConfig(port, config, config.hostname ? { hostname: config.hostname } : {}); if (r.changed) console.log(" + Grok Build config updated (~/.grok/config.toml)"); diff --git a/src/codex/desired-state.ts b/src/codex/desired-state.ts index dd5edbb56..5999d4873 100644 --- a/src/codex/desired-state.ts +++ b/src/codex/desired-state.ts @@ -21,7 +21,10 @@ * Design record: devlog/_plan/260803_codex_desktop_toggle/030_desired_state.md. */ import { loadConfig, mutatePersistedConfig } from "../config"; -import type { OcxConfig } from "../types"; +import type { OcxClientIntegrationsConfig, OcxConfig } from "../types"; + +/** Clients whose durable intent this module owns. */ +export type DurableIntentClientId = keyof OcxClientIntegrationsConfig; /** Injectable for tests; production passes the real sync. */ export type CodexStartupSync = (port: number) => Promise; @@ -42,8 +45,24 @@ export type CodexDesiredStateResult = * admitted snapshot cannot accidentally answer from a fresher one — the whole * point of admission is that one decision uses one set of bytes. */ +export function integrationEnabled( + config: Pick, + client: DurableIntentClientId, +): boolean { + return config.clientIntegrations?.[client] !== false; +} + export function codexIntegrationEnabled(config: Pick): boolean { - return config.clientIntegrations?.codex !== false; + return integrationEnabled(config, "codex"); +} + +/** + * Grok's toggle SHIPPED without this, which is the bug: it strips the fence in + * `~/.grok/config.toml` and records nothing, so the next `ocx start` calls + * `syncGrokConfig` unconditionally and writes the fence straight back. + */ +export function grokIntegrationEnabled(config: Pick): boolean { + return integrationEnabled(config, "grok"); } /** The same question when no snapshot is in hand. Reads the persisted config. */ @@ -58,18 +77,21 @@ export function codexIntegrationEnabledNow(): boolean { * the config coordinator, so a concurrent provider or model edit is preserved * instead of being clobbered by a whole-config write built from a stale read. */ -export function setCodexIntegrationEnabled(enabled: boolean): CodexDesiredStateResult { +export function setIntegrationEnabled( + client: DurableIntentClientId, + enabled: boolean, +): CodexDesiredStateResult { const outcome = mutatePersistedConfig(config => { - const current = codexIntegrationEnabled(config); + const current = integrationEnabled(config, client); if (current === enabled) return { changed: false, value: enabled }; const integrations = { ...(config.clientIntegrations ?? {}) }; if (enabled) { // ON is the absence, not a stored `true`: writing `true` would make an // untouched config and a re-enabled one differ in bytes for no reason, and // every later reader has to treat them identically anyway. - delete integrations.codex; + delete integrations[client]; } else { - integrations.codex = false; + integrations[client] = false; } // Drop the key entirely once nothing is left in it, so enabling twice does // not leave `"clientIntegrations": {}` behind in the user's file. @@ -98,6 +120,14 @@ export function setCodexIntegrationEnabled(enabled: boolean): CodexDesiredStateR }; } +export function setCodexIntegrationEnabled(enabled: boolean): CodexDesiredStateResult { + return setIntegrationEnabled("codex", enabled); +} + +export function setGrokIntegrationEnabled(enabled: boolean): CodexDesiredStateResult { + return setIntegrationEnabled("grok", enabled); +} + /** * The startup gate, as a function rather than an `if` buried in `handleStart`. * @@ -129,3 +159,19 @@ async function defaultStartupSync(port: number): Promise { const { syncModelsToCodex } = await import("./sync"); return syncModelsToCodex(port); } + +/** + * The Grok startup gate. + * + * Grok's toggle shipped and then `ocx start` called `syncGrokConfig` + * unconditionally, so switching Grok off lasted exactly one restart — the fence + * came out of `~/.grok/config.toml` and the next start wrote it straight back. + * Same defect as Codex had, in a different file. + * + * The caller keeps its own try/catch, because a Grok failure must never block + * startup and its diagnostic is worth printing. This only answers whether to + * attempt the sync at all. + */ +export function shouldSyncGrokOnStart(config: Pick): boolean { + return grokIntegrationEnabled(config); +} diff --git a/src/config.ts b/src/config.ts index 476c5969e..633b771c2 100644 --- a/src/config.ts +++ b/src/config.ts @@ -947,6 +947,7 @@ const apiKeyEntrySchema = z.object({ */ const clientIntegrationsSchema = z.object({ codex: z.boolean().optional().catch(undefined), + grok: z.boolean().optional().catch(undefined), }).passthrough(); const configSchema = z.object({ diff --git a/src/server/management/native-integration-routes.ts b/src/server/management/native-integration-routes.ts index 097689997..ec605be0a 100644 --- a/src/server/management/native-integration-routes.ts +++ b/src/server/management/native-integration-routes.ts @@ -236,15 +236,21 @@ async function handleCodexToggle(ctx: ManagementContext): Promise { const { setCodexIntegrationEnabled } = await import("../../codex/desired-state"); const persisted = setCodexIntegrationEnabled(enabled); - if (!persisted.ok) { - const status = persisted.retryable ? 409 : persisted.reason === "missing" ? 409 : 500; + /* + * `missing` does not block the switch — see the Grok route for the reasoning. + * A user with no config file yet still gets the artifact change; what they + * lose is durability across a restart, and the envelope says so rather than + * implying the switch failed. + */ + if (!persisted.ok && persisted.reason !== "missing") { return refusal( - status, + persisted.retryable ? 409 : 500, "codex", persisted.retryable ? "config_busy" : "write_failed", persisted.message, ); } + const durable = persisted.ok; if (enabled) { // The port this process actually BOUND, not what config.json last recorded. @@ -255,12 +261,14 @@ async function handleCodexToggle(ctx: ManagementContext): Promise { const { syncModelsToCodex } = await import("../../codex/sync"); const applied = await syncModelsToCodex(port); return jsonResponse({ - ok: true, clientId: "codex", changed: persisted.status === "committed", + ok: true, clientId: "codex", changed: durable && persisted.status === "committed", state: applied.ok ? "current" : "absent", message: applied.ok ? "Codex now routes through opencodex" : `Codex intent saved, but applying it did not complete: ${applied.message}`, - ...(applied.ok ? {} : { reason: "apply_incomplete" }), + ...(applied.ok + ? (durable ? {} : { reason: "not_durable" }) + : { reason: "apply_incomplete" }), } satisfies NativeToggleEnvelope); } @@ -268,12 +276,14 @@ async function handleCodexToggle(ctx: ManagementContext): Promise { const { restoreNativeCodexAsync } = await import("../../codex/inject"); const restored = await restoreNativeCodexAsync(); return jsonResponse({ - ok: true, clientId: "codex", changed: persisted.status === "committed", + ok: true, clientId: "codex", changed: durable && persisted.status === "committed", state: restored.success ? "absent" : "unsafe", message: restored.success ? "Codex restored to its native path; the proxy is still serving other clients" : `Codex intent saved, but restoring the native path did not complete: ${restored.message}`, - ...(restored.success ? {} : { reason: "restore_incomplete" }), + ...(restored.success + ? (durable ? {} : { reason: "not_durable" }) + : { reason: "restore_incomplete" }), } satisfies NativeToggleEnvelope); })(); try { @@ -317,6 +327,41 @@ async function handleGrokToggle(ctx: ManagementContext): Promise { if (seen.kind === "not_installed") return refusal(404, "grok", "not_installed", NOT_INSTALLED_MESSAGE); if (seen.kind === "orphaned_marker") return refusal(409, "grok", "orphaned_marker", ORPHANED_MARKER_MESSAGE); + /* + * Persist the DECISION before touching the fence. + * + * This route shipped without it, which is the whole bug: stripping the fence + * records nothing, so the next `ocx start` calls syncGrokConfig + * unconditionally and writes it straight back. The switch worked and lasted + * exactly one restart. + * + * Intent first, artifacts second, for the same reason as the Codex route: a + * process that dies between them leaves a decision the next start can act + * on, where the other order leaves artifacts the next start undoes. + */ + const { setGrokIntegrationEnabled } = await import("../../codex/desired-state"); + const persisted = setGrokIntegrationEnabled(enabled); + /* + * `missing` does NOT block the toggle here. + * + * A config file that does not exist yet is a normal state for someone who + * has never saved settings, and refusing their switch because of it would + * make the button dead for exactly the users least able to diagnose why. + * The fence change is still worth performing and still reports honestly; + * what they lose is durability across a restart, which is what the reason + * on the envelope says. `conflict` and `invalid` are different: another + * writer won, or the file is malformed and must not be overwritten. + */ + if (!persisted.ok && persisted.reason !== "missing") { + return refusal( + persisted.retryable ? 409 : 500, + "grok", + persisted.retryable ? "config_busy" : "write_failed", + persisted.message, + ); + } + const durable = persisted.ok; + if (!enabled) { /* * Disable only: a strip is a SHARED teardown — it must not run under a diff --git a/src/types.ts b/src/types.ts index 38c665ec0..63bd5c40d 100644 --- a/src/types.ts +++ b/src/types.ts @@ -542,6 +542,8 @@ export interface OcxApiKeyEntry { export interface OcxClientIntegrationsConfig { /** Durable desired state for native Codex. MISSING MEANS ON. */ codex?: boolean; + /** Durable desired state for Grok Build. MISSING MEANS ON. */ + grok?: boolean; } export interface OcxConfig { diff --git a/tests/codex-desired-state.test.ts b/tests/codex-desired-state.test.ts index a6b89347c..642c06c51 100644 --- a/tests/codex-desired-state.test.ts +++ b/tests/codex-desired-state.test.ts @@ -17,6 +17,9 @@ import { codexIntegrationEnabled, codexIntegrationEnabledNow, setCodexIntegrationEnabled, + setGrokIntegrationEnabled, + grokIntegrationEnabled, + shouldSyncGrokOnStart, syncCodexOnStartIfEnabled, } from "../src/codex/desired-state"; import type { OcxConfig } from "../src/types"; @@ -216,3 +219,60 @@ describe("the startup gate", () => { expect(ran).toBe(true); }); }); + +describe("Grok has the same durability, because it shipped without it", () => { + /** + * Grok's toggle already existed and already worked — and lasted exactly one + * restart. It strips the fence from `~/.grok/config.toml` and recorded + * nothing, so `ocx start` called `syncGrokConfig` unconditionally and wrote + * the fence straight back. Identical defect to Codex, different file. + */ + test("absence, empty, and explicit true all read as enabled; only false is off", () => { + expect(grokIntegrationEnabled(baseConfig())).toBe(true); + expect(grokIntegrationEnabled({ ...baseConfig(), clientIntegrations: {} })).toBe(true); + expect(grokIntegrationEnabled({ ...baseConfig(), clientIntegrations: { grok: true } })).toBe(true); + expect(grokIntegrationEnabled({ ...baseConfig(), clientIntegrations: { grok: false } })).toBe(false); + }); + + test("the startup sync is skipped only for an explicit off", () => { + expect(shouldSyncGrokOnStart(baseConfig())).toBe(true); + expect(shouldSyncGrokOnStart({ ...baseConfig(), clientIntegrations: { grok: true } })).toBe(true); + expect(shouldSyncGrokOnStart({ ...baseConfig(), clientIntegrations: { grok: false } })).toBe(false); + }); + + /** + * The two switches are independent. Turning Codex off must not take Grok with + * it — a shared key or a shared helper reading the wrong field would, and the + * ten-key union this design rejected is exactly how that happens. + */ + test("the two clients do not affect each other", () => { + saveConfig(baseConfig()); + setCodexIntegrationEnabled(false); + + const after = loadConfig(); + expect(codexIntegrationEnabled(after)).toBe(false); + expect(grokIntegrationEnabled(after)).toBe(true); + + setGrokIntegrationEnabled(false); + const both = loadConfig(); + expect(codexIntegrationEnabled(both)).toBe(false); + expect(grokIntegrationEnabled(both)).toBe(false); + + // And re-enabling one leaves the other alone. + setCodexIntegrationEnabled(true); + const one = loadConfig(); + expect(codexIntegrationEnabled(one)).toBe(true); + expect(grokIntegrationEnabled(one)).toBe(false); + }); + + test("the last client re-enabled removes the whole object, not an empty husk", () => { + saveConfig(baseConfig()); + setCodexIntegrationEnabled(false); + setGrokIntegrationEnabled(false); + setCodexIntegrationEnabled(true); + setGrokIntegrationEnabled(true); + + const raw = JSON.parse(readFileSync(join(testRoot, "config.json"), "utf8")) as Record; + expect(raw.clientIntegrations).toBeUndefined(); + }); +}); From ea5c5a8e3f87651174055f3e140d0ce347ba7fda Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Wed, 5 Aug 2026 11:23:05 +0900 Subject: [PATCH 127/163] docs(codex): F4 closes at NEAR-PASS; widen the Windows gate and stop overstating one mutation The independent reviewer returned no further executable hole in the F4 surface after twelve rounds of finding one per round. Recorded here with the two corrections that came with it. The Windows activation gate was too narrow. A nonzero stable file index is not the only thing this design assumes about NTFS: dev:ino has to survive a real icacls edit, because the before/after comparison rests on it; post-ACL ctimeNs has to be readable and stable for an immediate memo hit, because the memo stores it; and an ordinary unlink/recreate has to move at least one memo component. None of those is observable off Windows, so the probe has to show all four or the gate is checking one assumption and implying three. And (al) was described as optional-only while I reported the eight-test figure from a broader mutation that also bypasses required callers. The coverage is real either way, but quoting the wider number against the narrower description overstates it. Both forms are now named with their own counts. --- .../030_lock_protocol.md | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md b/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md index c0d65bf1f..0769f8bde 100644 --- a/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md +++ b/devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md @@ -1036,8 +1036,10 @@ and the ones that did were rewritten rather than kept. allowed to trust a pathname-only memo; (ai) an optional caller trusts it when the identity is UNOBSERVABLE; (aj) observed absence retires the memo only for required callers; (ak) an optional caller compares the object but ignores freshness; - (al) an optional caller accepts an unobservable identity when the cause is a - zero inode rather than a thrown stat. + (al) an optional caller accepts an unobservable identity. Reported precisely: the + optional-only form reddens the four optional entry points; a broader form that + also bypasses required callers reddens eight. Counting the wider one against the + narrower description overstated the evidence, so both are named. (h) through (al) are not redundant — each survived every other check. (h) and (i) cover production callers the primitive tests missed: `hardenStableLockFile` takes the async path, and `hardenSecretDir` backs config, management-auth, tray, @@ -1215,6 +1217,12 @@ and the ones that did were rewritten rather than kept. every REQUIRED Windows harden fails closed and this whole surface is inert in the one place it exists for. A pinned-Bun probe on real Windows/NTFS must confirm a nonzero, stable file index before F4 is called complete (goalplan `wp12t0c2`). + A file index alone is not enough, because three other production assumptions ride + on that platform and none of them is observable from here: `dev:ino` must survive + a real `icacls` edit (the before/after comparison assumes it), post-ACL `ctimeNs` + must be readable and stable for an immediate memo hit (the memo stores it), and an + ordinary unlink/recreate must move at least one memo component. Exact-identity ABA + during hardening stays the documented residual. This is recorded here as well as in the goalplan so this document cannot be read as complete on its own. From 480eb3ce560f763ef08bb9aa14e78df6f90d0664 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Wed, 5 Aug 2026 11:33:54 +0900 Subject: [PATCH 128/163] =?UTF-8?q?feat(codex):=20N=20exists=20=E2=80=94?= =?UTF-8?q?=20one=20bounded=20writer=20per=20canonical=20CODEX=5FHOME?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The module the whole substrate is named after did not exist. It does now. Every spelling of one home takes one lock: symlinked, trailing-slash, relative and tilde forms all collapse through realpath to a single id, and two genuinely different directories do not collide. A MISSING home refuses rather than resolving optimistically — keeping an unresolved suffix would split one future home on a case-insensitive filesystem or alias two on a case-sensitive one. The namespace keys on the effective user, never a home path, because under Bun 1.3.14 both homedir accessors follow the environment: a service and a CLI for one account can otherwise coordinate through different databases and exclude nothing, silently. An explicit codexHome must EQUAL the ambient one. The residue guard that decides whether the coordinator row may be created resolves its own home from the environment while the lock keys on what it was handed, so forwarding a different home would lock one directory while another was safety-checked. Refusing the mismatch is the honest answer until that guard takes a target parameter. Two processes are the evidence, not two tasks in one isolate — those share the connection cache and the reentrancy store and prove neither. A real child holds the lock, the parent gets a typed busy, the child releases, and the parent then acquires, which is what makes the earlier busy contention rather than a permanent refusal wearing its label. CodexCoordinatorTransactionController gained version(). The conditional row update matches on the existing txId, and CommitExpectation carries only the generation pair plus the new one, so a caller had to guess — and guessing null works on a row nobody has ever transitioned, which means it passes on a fresh machine and fails on a real one. Opening a second connection to read it would contend with the transaction's own BEGIN IMMEDIATE. Six mutations, restored to 17 pass / 0 fail: realpath skipped so spellings split (13/3), explicit-home mismatch allowed (15/1), reentrancy undetected (15/1), stale admission accepted (15/1), thenable guard removed (15/1), and a permanent open failure classified as retryable (16/1). That last one survived the first five — a denial reported as contention is an endless retry, and only a test that watches the clock can tell them apart. --- src/codex/codex-write-lock.ts | 353 +++++++++++++++++++++++ src/codex/convergence-types.ts | 9 + src/codex/transition-state.ts | 5 + tests/codex-write-lock.test.ts | 355 ++++++++++++++++++++++++ tests/helpers/codex-write-lock-child.ts | 61 ++++ 5 files changed, 783 insertions(+) create mode 100644 src/codex/codex-write-lock.ts create mode 100644 tests/codex-write-lock.test.ts create mode 100644 tests/helpers/codex-write-lock-child.ts diff --git a/src/codex/codex-write-lock.ts b/src/codex/codex-write-lock.ts new file mode 100644 index 000000000..cfa4477cf --- /dev/null +++ b/src/codex/codex-write-lock.ts @@ -0,0 +1,353 @@ +/** + * N — one bounded writer per canonical `CODEX_HOME`. + * + * The failure this closes is lock SPLITTING plus event-loop denial, not a + * missing mutex. Two spellings of one Codex home can reach different textual + * paths, and a lock held across provider discovery or history walking recreates + * the 10.5-second listener stall that killed the previous design. + * + * Three decisions are load-bearing and each was forced by a defect: + * + * - The namespace keys on the EFFECTIVE USER (uid/SID), never on a home path. + * Under Bun 1.3.14 both `os.homedir()` and `os.userInfo().homedir` follow the + * environment, so a service and a CLI for one account can see different homes + * and coordinate through different databases — exclusion defeated, silently. + * `resolveCodexCoordinatorDatabasePath` owns that; this module consumes its + * result verbatim and appends nothing. + * + * - The commit callback is SYNCHRONOUS. Provider I/O, subprocesses, history + * walking and retry sleeps are forbidden beneath it; a thenable that slips + * through the type is detected, rolled back, and rejected. + * + * - Acquisition is finite and typed. `busy` means try again; `refused` means + * never. Collapsing them is how a permanent denial becomes an endless retry + * loop wearing the costume of contention. + * + * Lock order is `N -> C`. C is `withConfigMutationLockSync`, entered while N is + * held and released before N commits. There is no `C -> N`. + * + * Design record: devlog/_plan/260804_codex_write_substrate/030_lock_protocol.md. + */ +import { AsyncLocalStorage } from "node:async_hooks"; +import { createHash } from "node:crypto"; +import { lstatSync, realpathSync } from "node:fs"; +import { isAbsolute, resolve } from "node:path"; +import { homedir } from "node:os"; + +import { withConfigMutationLockSync } from "../config"; +import type { + AdmissionSnapshot, + CodexCoordinatorTransaction, + CommitExpectation, +} from "./convergence-types"; +import { getCodexHome } from "./paths"; +import { nativeMainOwnerFilesystemSupported } from "./native-main-owner"; +import { openCodexCoordinatorTransaction } from "./transition-state"; +import { + CodexUserIdentityRefusal, + resolveCodexCoordinatorDatabasePath, + resolveEffectiveUserIdentity, +} from "./user-identity"; + +export const CODEX_WRITE_LOCK_MAX_TIMEOUT_MS = 30_000; + +/** Uniform, small, and jittered: a fixed interval makes contenders resonate. */ +const RETRY_MIN_MS = 25; +const RETRY_MAX_MS = 75; + +export type CodexWriteLockRefusalReason = + | "codex_home_missing" + | "codex_home_unsafe" + | "authority_not_proven" + | "namespace_unsafe" + | "lock_path_unsafe" + | "unsupported_filesystem" + | "reentrant" + | "lock_unavailable"; + +export type CodexWriteLockResult = + | { status: "acquired"; value: T; waitedMs: number; lockId: string } + | { status: "busy"; reason: "deadline" | "cancelled"; retryable: true; waitedMs: number } + | { + status: "refused"; + reason: CodexWriteLockRefusalReason; + retryable: false; + message: string; + }; + +export interface CodexWriteLockOptions { + /** Defaults to the ambient home. An explicit value must MATCH it — see below. */ + codexHome?: string; + timeoutMs: number; + signal?: AbortSignal; + /** Read-only snapshot obtained before any namespace creation. */ + admitted: AdmissionSnapshot; + /** Authoritative synchronous re-read while N and C are both held. */ + readAdmissionUnderLock(): AdmissionSnapshot; +} + +export interface CodexWriteCommitContext { + readonly canonicalCodexHome: string; + readonly lockId: string; + readonly admission: AdmissionSnapshot; + readonly expectation: CommitExpectation; + /** + * The `currentTxId` the coordinator row holds RIGHT NOW. + * + * `CommitExpectation` carries the generation pair and the NEW txId, but the + * conditional row update also matches on the existing one. Without this the + * caller has to guess it — and guessing `null` works only on a row nobody has + * ever transitioned, so the guess passes on a fresh machine and fails on a + * real one. + */ + readonly currentTxId: string | null; + /** Opaque authority over the ALREADY-OPEN transaction. Not SQLite. */ + readonly coordinator: CodexCoordinatorTransaction; +} + +/** Rejects an `async` callback at typecheck; a cast thenable is caught at runtime. */ +type Synchronous = T extends PromiseLike ? never : T; + +/** + * Same-task reentrancy detection. + * + * NOT the exclusion mechanism — `busy_timeout = 0` before `BEGIN IMMEDIATE` + * already makes a second open in this process fail with SQLITE_BUSY. This exists + * only to turn that indistinguishable `busy` into a typed `refused/reentrant`, + * so a caller is told "you already hold this" rather than "try again forever". + */ +const heldHomes = new AsyncLocalStorage>(); + +function refuse(reason: CodexWriteLockRefusalReason, message: string): CodexWriteLockResult { + return { status: "refused", reason, retryable: false, message }; +} + +function isBusyError(error: unknown): boolean { + const code = error && typeof error === "object" && "code" in error + ? String((error as { code?: unknown }).code) + : ""; + const message = error instanceof Error ? error.message : String(error); + return code === "SQLITE_BUSY" || code === "SQLITE_LOCKED" + || /database (?:is|table is) locked/i.test(message); +} + +function expandLeadingTilde(path: string): string { + if (path === "~") return homedir(); + if (path.startsWith("~/") || path.startsWith("~\\")) return resolve(homedir(), path.slice(2)); + return path; +} + +export interface CanonicalCodexHome { + readonly path: string; + /** Stable id for one home, independent of which spelling reached us. */ + readonly lockId: string; +} + +/** + * Canonicalize a `CODEX_HOME` so every spelling of one directory takes one lock. + * + * Default, explicit, absolute, tilde and symlinked spellings must contend; two + * genuinely different directories must not. A MISSING home is refused rather + * than resolved optimistically: keeping an unresolved suffix would either split + * one future home on a case-insensitive filesystem or alias two on a + * case-sensitive one. + */ +export function canonicalizeCodexHome( + candidate: string, +): { ok: true; home: CanonicalCodexHome } | { ok: false; reason: CodexWriteLockRefusalReason; message: string } { + const expanded = resolve(expandLeadingTilde(candidate)); + if (!isAbsolute(expanded)) { + return { ok: false, reason: "codex_home_unsafe", message: "CODEX_HOME did not resolve to an absolute path." }; + } + let entry; + try { + entry = lstatSync(expanded); + } catch { + return { ok: false, reason: "codex_home_missing", message: `CODEX_HOME does not exist: ${expanded}` }; + } + // A symlinked home is fine — realpath collapses it below, which is how two + // spellings end up on one lock. A home that is a FILE is not. + if (!entry.isDirectory() && !entry.isSymbolicLink()) { + return { ok: false, reason: "codex_home_unsafe", message: `CODEX_HOME is not a directory: ${expanded}` }; + } + let canonical: string; + try { + canonical = realpathSync.native(expanded); + if (!lstatSync(canonical).isDirectory()) { + return { ok: false, reason: "codex_home_unsafe", message: `CODEX_HOME is not a directory: ${canonical}` }; + } + } catch { + return { ok: false, reason: "codex_home_missing", message: `CODEX_HOME does not exist: ${expanded}` }; + } + if (!nativeMainOwnerFilesystemSupported(canonical)) { + return { + ok: false, + reason: "unsupported_filesystem", + message: `CODEX_HOME is on a filesystem that cannot hold this lock: ${canonical}`, + }; + } + // Windows paths are case-insensitive, so two spellings of one directory must + // hash alike there; POSIX paths are not, so they must not be folded. + const normalized = process.platform === "win32" ? canonical.toLowerCase() : canonical; + const lockId = createHash("sha256") + .update("opencodex-codex-write-lock-v1\0") + .update(normalized) + .digest("hex"); + return { ok: true, home: { path: canonical, lockId } }; +} + +function sleepJittered(remainingMs: number, signal?: AbortSignal): Promise { + const span = RETRY_MAX_MS - RETRY_MIN_MS; + const wait = Math.min(remainingMs, RETRY_MIN_MS + Math.floor(Math.random() * (span + 1))); + return new Promise(done => { + const timer = setTimeout(finish, Math.max(1, wait)); + function finish(): void { + clearTimeout(timer); + signal?.removeEventListener("abort", finish); + done(); + } + signal?.addEventListener("abort", finish, { once: true }); + }); +} + +/** + * Hold N for one canonical `CODEX_HOME` and run `commit` while it is held. + * + * Returns `acquired` with the callback's value, `busy` when the deadline or the + * signal ended the wait, or `refused` when nothing about waiting would help. A + * caller exception is NOT converted into either: it propagates after rollback, + * because a failed commit is the caller's error and not a lock outcome. + */ +export async function withCodexWriteLock( + options: CodexWriteLockOptions, + commit: (context: CodexWriteCommitContext) => Synchronous, +): Promise> { + const { timeoutMs, signal } = options; + if (!Number.isSafeInteger(timeoutMs) || timeoutMs < 0 || timeoutMs > CODEX_WRITE_LOCK_MAX_TIMEOUT_MS) { + return refuse("authority_not_proven", + `timeoutMs must be an integer between 0 and ${CODEX_WRITE_LOCK_MAX_TIMEOUT_MS}.`); + } + + /* + * The ambient home is resolved ONCE, and an explicit home must equal it. + * + * `classifyNativeRoutedResidue` — the guard that decides whether the + * coordinator row may be created — resolves its own home from the ambient + * environment, while the lock path keys on the home WE were given. Forwarding a + * different explicit home would lock one directory while a different one was + * safety-checked. Until that guard takes the target as a parameter (WP12), the + * only honest answer is to refuse the mismatch. + */ + const ambient = canonicalizeCodexHome(getCodexHome()); + if (!ambient.ok) return refuse(ambient.reason, ambient.message); + + let target = ambient.home; + if (options.codexHome !== undefined) { + const trimmed = options.codexHome.trim(); + if (!trimmed) return refuse("codex_home_unsafe", "An explicit codexHome must not be blank."); + const explicit = canonicalizeCodexHome(trimmed); + if (!explicit.ok) return refuse(explicit.reason, explicit.message); + if (explicit.home.path !== ambient.home.path) { + return refuse("authority_not_proven", + "The requested CODEX_HOME is not the one this process would inspect for safety."); + } + target = explicit.home; + } + + const held = heldHomes.getStore(); + if (held?.has(target.lockId)) { + return refuse("reentrant", "This task already holds the Codex write lock for this CODEX_HOME."); + } + + let databasePath: string; + try { + databasePath = resolveCodexCoordinatorDatabasePath(resolveEffectiveUserIdentity(), target.path); + } catch (error) { + return refuse("namespace_unsafe", + error instanceof CodexUserIdentityRefusal ? error.message : "The lock namespace could not be resolved."); + } + + const started = performance.now(); + const deadline = started + timeoutMs; + const waited = (): number => Math.round(performance.now() - started); + + for (;;) { + if (signal?.aborted) return { status: "busy", reason: "cancelled", retryable: true, waitedMs: waited() }; + + let transaction: ReturnType | undefined; + try { + transaction = openCodexCoordinatorTransaction(databasePath); + } catch (error) { + // Only contention retries. A malformed database, an unsafe path, or an + // identity failure will fail identically forever; telling a caller to retry + // that is how a UI spins on a problem only the user can fix. + if (!isBusyError(error)) { + if (error instanceof CodexUserIdentityRefusal) return refuse("lock_path_unsafe", error.message); + return refuse("lock_unavailable", + error instanceof Error ? error.message : "The Codex write lock could not be opened."); + } + if (performance.now() >= deadline) { + return { status: "busy", reason: "deadline", retryable: true, waitedMs: waited() }; + } + await sleepJittered(deadline - performance.now(), signal); + continue; + } + + // N is held from here. Everything below is synchronous until commit/rollback. + try { + const expectation = transaction.expectation(); + const version = transaction.version(); + const nextHeld = new Set(held ?? []); + nextHeld.add(target.lockId); + + const value = heldHomes.run(nextHeld, () => withConfigMutationLockSync(() => { + const current = options.readAdmissionUnderLock(); + if (current.authoritySnapshotId !== options.admitted.authoritySnapshotId) { + throw new CodexWriteLockStaleAdmission(); + } + const result = commit({ + canonicalCodexHome: target.path, + lockId: target.lockId, + admission: current, + expectation, + currentTxId: version.currentTxId, + coordinator: transaction!.capability, + }); + // A cast `async` callback would return a thenable here. Awaiting it is + // impossible — C is synchronous — so the only safe answer is to reject. + if (result && typeof (result as { then?: unknown }).then === "function") { + throw new TypeError("The Codex write-lock commit callback must be synchronous."); + } + return result; + })); + + transaction.assertPublished(expectation); + transaction.commit(); + return { status: "acquired", value: value as T, waitedMs: waited(), lockId: target.lockId }; + } catch (error) { + transaction.rollback(); + if (error instanceof CodexWriteLockStaleAdmission) { + return refuse("authority_not_proven", + "The admitted state changed before the commit could be made under the lock."); + } + if (isBusyError(error)) { + if (performance.now() >= deadline) { + return { status: "busy", reason: "deadline", retryable: true, waitedMs: waited() }; + } + await sleepJittered(deadline - performance.now(), signal); + continue; + } + throw error; + } finally { + transaction.close(); + } + } +} + +/** Internal: the under-lock re-read disagreed with what was admitted. */ +class CodexWriteLockStaleAdmission extends Error { + constructor() { + super("The admitted authority snapshot is stale."); + this.name = "CodexWriteLockStaleAdmission"; + } +} diff --git a/src/codex/convergence-types.ts b/src/codex/convergence-types.ts index 21f21a4bf..f48df0e44 100644 --- a/src/codex/convergence-types.ts +++ b/src/codex/convergence-types.ts @@ -335,6 +335,15 @@ export interface CodexCoordinatorTransaction { export interface CodexCoordinatorTransactionController { readonly capability: CodexCoordinatorTransaction; expectation(): CommitExpectation; + /** + * The pair the row holds right now, read on the ALREADY-OPEN transaction. + * + * A holder needs `currentTxId` to build the conditional update, and + * `CommitExpectation` carries only the generation pair plus the new txId. + * Opening a second connection to read it would contend with this + * transaction's own `BEGIN IMMEDIATE`. + */ + version(): CodexTransitionVersion; assertPublished(expectation: CommitExpectation): void; assertStablePath(): void; commit(): void; diff --git a/src/codex/transition-state.ts b/src/codex/transition-state.ts index 0104fd25d..6ced64384 100644 --- a/src/codex/transition-state.ts +++ b/src/codex/transition-state.ts @@ -450,6 +450,11 @@ export function openCodexCoordinatorTransaction(finalDatabasePath: string): Code txId: randomUUID(), }; }, + version() { + requireOpen(); + const state = readState(db); + return { nativeGeneration: state.nativeGeneration, currentTxId: state.currentTxId }; + }, assertPublished(expectation) { requireOpen(); if (lastResult?.kind !== "updated") { diff --git a/tests/codex-write-lock.test.ts b/tests/codex-write-lock.test.ts new file mode 100644 index 000000000..8a6228aa7 --- /dev/null +++ b/tests/codex-write-lock.test.ts @@ -0,0 +1,355 @@ +/** + * N — one bounded writer per canonical CODEX_HOME. + * + * What matters here is not that a mutex excludes. It is that the WRONG things + * cannot happen: two spellings of one home must not take two locks, a home the + * ambient environment would not have safety-checked must be refused rather than + * locked, and a permanent refusal must never be reported as contention — a + * caller told to retry something that will fail identically forever is how a UI + * spins on a problem only the user can fix. + */ +import { afterEach, beforeEach, describe, expect, test } from "bun:test"; +import { mkdirSync, mkdtempSync, realpathSync, rmSync, symlinkSync, writeFileSync } from "node:fs"; +import { + resolveCodexCoordinatorDatabasePath, + resolveEffectiveUserIdentity, +} from "../src/codex/user-identity"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +import { + CODEX_WRITE_LOCK_MAX_TIMEOUT_MS, + canonicalizeCodexHome, + withCodexWriteLock, +} from "../src/codex/codex-write-lock"; +import type { AdmissionSnapshot } from "../src/codex/convergence-types"; + +let root = ""; +let codexHome = ""; +let previousCodexHome: string | undefined; +const cleanup: string[] = []; + +/** + * The lock compares `authoritySnapshotId` and nothing else, so the rest is + * deliberately minimal — a fixture that mirrored every field would suggest the + * lock reads them. + */ +function admission(authoritySnapshotId = "authority-1"): AdmissionSnapshot { + return { authoritySnapshotId } as AdmissionSnapshot; +} + +function options(overrides: Partial[0]> = {}) { + const admitted = overrides.admitted ?? admission(); + return { + timeoutMs: 0, + admitted, + readAdmissionUnderLock: () => admitted, + ...overrides, + } as Parameters[0]; +} + +/** + * A commit that actually publishes a transition. + * + * The lock verifies the row before it will commit, so a callback that writes + * nothing is not a valid commit — and that is the point: a caller cannot take N, + * do something else, and have the coordinator record a transition it never made. + */ +function publishing(value: T) { + return (ctx: Parameters[1]>[0]): T => { + ctx.coordinator.beginTransition( + // The expected pair comes from the row the lock just read, not from an + // assumed zero: another process may have already published a transition, + // and hardcoding {0, null} would make this test pass only when it runs + // first. + { nativeGeneration: ctx.expectation.nativeBefore, currentTxId: ctx.currentTxId }, + { + txId: ctx.expectation.txId, + direction: "apply", + authoritySnapshotId: ctx.admission.authoritySnapshotId, + nextRetryAt: new Date().toISOString(), + }, + ); + return value; + }; +} + +beforeEach(() => { + root = mkdtempSync(join(tmpdir(), "ocx-write-lock-")); + cleanup.push(root); + codexHome = join(root, ".codex"); + mkdirSync(codexHome, { recursive: true }); + // A clean home: the coordinator refuses to initialize over routing residue. + writeFileSync(join(codexHome, "config.toml"), 'model = "gpt-5"\n'); + previousCodexHome = process.env.CODEX_HOME; + process.env.CODEX_HOME = codexHome; +}); + +afterEach(() => { + if (previousCodexHome === undefined) delete process.env.CODEX_HOME; + else process.env.CODEX_HOME = previousCodexHome; + while (cleanup.length) rmSync(cleanup.pop()!, { recursive: true, force: true }); +}); + +describe("canonical home identity", () => { + /** + * Every spelling of one directory must land on one lock id, or two processes + * that mean the same home take different locks and exclude nothing. This is + * the split the whole design exists to prevent. + */ + test("symlinked, trailing-slash and relative spellings share one lock id", () => { + const link = join(root, "linked-home"); + symlinkSync(codexHome, link); + + const direct = canonicalizeCodexHome(codexHome); + const viaLink = canonicalizeCodexHome(link); + const trailing = canonicalizeCodexHome(`${codexHome}/`); + const dotted = canonicalizeCodexHome(join(codexHome, ".", "")); + + expect(direct.ok && viaLink.ok && trailing.ok && dotted.ok).toBe(true); + const ids = [direct, viaLink, trailing, dotted].map(r => (r.ok ? r.home.lockId : "x")); + expect(new Set(ids).size).toBe(1); + }); + + test("two genuinely different homes do not share a lock id", () => { + const other = join(root, "other-home"); + mkdirSync(other, { recursive: true }); + const a = canonicalizeCodexHome(codexHome); + const b = canonicalizeCodexHome(other); + expect(a.ok && b.ok).toBe(true); + expect(a.ok && b.ok && a.home.lockId === b.home.lockId).toBe(false); + }); + + /** + * A missing home is REFUSED rather than resolved optimistically. Keeping an + * unresolved suffix would split one future home on a case-insensitive + * filesystem, or alias two on a case-sensitive one. + */ + test("a missing home refuses instead of inventing an identity", () => { + const result = canonicalizeCodexHome(join(root, "never-created")); + expect(result.ok).toBe(false); + expect(result.ok === false && result.reason).toBe("codex_home_missing"); + }); + + test("a home that is a file refuses", () => { + const file = join(root, "not-a-dir"); + writeFileSync(file, "x"); + const result = canonicalizeCodexHome(file); + expect(result.ok).toBe(false); + expect(result.ok === false && result.reason).toBe("codex_home_unsafe"); + }); +}); + +describe("refusals are not contention", () => { + test("a timeout outside the cap is refused, not clamped", async () => { + for (const timeoutMs of [-1, 1.5, CODEX_WRITE_LOCK_MAX_TIMEOUT_MS + 1]) { + const result = await withCodexWriteLock(options({ timeoutMs }), () => "never"); + expect(result.status).toBe("refused"); + expect(result.status === "refused" && result.retryable).toBe(false); + } + }); + + /** + * The residue guard that decides whether the coordinator row may be created + * resolves its home from the AMBIENT environment, while the lock path keys on + * the home we were handed. Locking one directory while a different one was + * safety-checked is the hazard; until that guard takes the target as a + * parameter, a mismatch must refuse. + */ + test("an explicit home that is not the ambient one is refused", async () => { + const other = join(root, "elsewhere"); + mkdirSync(other, { recursive: true }); + const result = await withCodexWriteLock(options({ codexHome: other }), () => "never"); + expect(result.status).toBe("refused"); + expect(result.status === "refused" && result.reason).toBe("authority_not_proven"); + }); + + test("a blank explicit home is refused as a programmer error", async () => { + const result = await withCodexWriteLock(options({ codexHome: " " }), () => "never"); + expect(result.status).toBe("refused"); + expect(result.status === "refused" && result.reason).toBe("codex_home_unsafe"); + }); + + /** + * A permanent failure must NOT enter the retry loop. + * + * This is the defect this unit produced repeatedly at other layers: a denial + * classified as contention becomes an endless retry wearing the costume of a + * busy lock. Here the coordinator database is replaced by a directory, so + * every open fails identically forever — and a deadline long enough to notice + * proves the difference. A retrying implementation burns the whole budget and + * reports `busy`; a correct one refuses immediately. + */ + test("an unopenable coordinator refuses immediately instead of retrying to the deadline", async () => { + const identity = resolveEffectiveUserIdentity(); + const dbPath = resolveCodexCoordinatorDatabasePath(identity, realpathSync.native(codexHome)); + rmSync(dbPath, { force: true }); + // A directory where the database belongs: openable never, busy never. + mkdirSync(dbPath, { recursive: true }); + + const started = performance.now(); + const result = await withCodexWriteLock(options({ timeoutMs: 2_000 }), () => "never"); + const elapsed = performance.now() - started; + + expect(result.status).toBe("refused"); + expect(result.status === "refused" && result.retryable).toBe(false); + // It did not spend the deadline discovering that a permanent failure is + // permanent. + expect(elapsed).toBeLessThan(1_000); + }); + + test("an explicit home equal to the ambient one is accepted", async () => { + const result = await withCodexWriteLock(options({ codexHome }), publishing("ok")); + expect(result.status).toBe("acquired"); + }); +}); + +describe("holding the lock", () => { + test("the callback runs under the lock and its value comes back", async () => { + const result = await withCodexWriteLock(options(), ctx => { + expect(ctx.canonicalCodexHome).toBe(realpathSync.native(codexHome)); + expect(ctx.expectation.nativeAfter).toBe(ctx.expectation.nativeBefore + 1); + return publishing(42)(ctx); + }); + expect(result.status).toBe("acquired"); + expect(result.status === "acquired" && result.value).toBe(42); + }); + + /** + * Reentrancy is a DIAGNOSIS, not the exclusion mechanism — SQLite already + * refuses the second open. The value is that the caller is told "you already + * hold this" instead of being sent into a retry loop that cannot succeed. + */ + test("re-entering from inside the callback is refused as reentrant, not busy", async () => { + let nested: Promise>> | undefined; + const outer = await withCodexWriteLock(options(), ctx => { + // The nested call STARTS synchronously — far enough to read the reentrancy + // store and refuse — and its promise is settled after the callback returns. + // Returning it would trip the thenable guard, so it is captured instead. + nested = withCodexWriteLock(options(), publishing("nested")); + return publishing(1)(ctx); + }); + expect(outer.status).toBe("acquired"); + const inner = await nested!; + expect(inner.status).toBe("refused"); + expect(inner.status === "refused" && inner.reason).toBe("reentrant"); + }); + + /** + * A callback failure is the CALLER's error, not a lock outcome. Converting it + * into `busy` would tell them to retry something that will fail identically, + * and into `refused` would hide their own exception. + */ + test("a callback throw propagates rather than becoming busy or refused", async () => { + await expect(withCodexWriteLock(options(), () => { throw new Error("caller blew up"); })) + .rejects.toThrow("caller blew up"); + }); + + test("an admission that changed under the lock refuses without committing", async () => { + let calls = 0; + const result = await withCodexWriteLock( + options({ readAdmissionUnderLock: () => admission("authority-2") }), + () => { calls += 1; return "never"; }, + ); + expect(result.status).toBe("refused"); + expect(result.status === "refused" && result.reason).toBe("authority_not_proven"); + // The commit never ran: a stale snapshot must not reach the callback at all. + expect(calls).toBe(0); + }); + + /** + * An `async` callback is rejected at typecheck, but a cast one is not. C is + * synchronous, so awaiting it is impossible and the only safe answer is to + * reject rather than let a promise escape the held section. + */ + test("a cast async callback is rejected instead of escaping the held section", async () => { + await expect(withCodexWriteLock( + options(), + (async () => "sneaky") as never, + )).rejects.toThrow(/synchronous/); + }); +}); + +describe("two real processes contend for one lock", () => { + /** + * The evidence this phase actually owes. A second async task in this isolate + * shares the connection cache and the reentrancy store, so it proves neither + * exclusion nor its absence — this unit has already shipped a test that looked + * like a race and was not one. So: a real child process, calling the + * production module. + */ + const childPath = join(import.meta.dir, "helpers", "codex-write-lock-child.ts"); + + function spawnChild(payload: Record) { + return Bun.spawn(["bun", childPath], { + env: { ...process.env, CODEX_HOME: codexHome, OCX_LOCK_CHILD_PAYLOAD: JSON.stringify(payload) }, + stdout: "pipe", + stderr: "pipe", + }); + } + + async function childResult(child: ReturnType) { + const [stdout] = await Promise.all([new Response(child.stdout).text(), child.exited]); + const line = stdout.trim().split("\n").filter(Boolean).at(-1) ?? "{}"; + return JSON.parse(line) as { status: string; reason?: string; value?: string; lockId?: string }; + } + + async function waitFor(path: string, timeoutMs = 10_000): Promise { + const deadline = Date.now() + timeoutMs; + while (Date.now() < deadline) { + if (Bun.file(path).size > 0) return; + await Bun.sleep(10); + } + throw new Error(`timed out waiting for ${path}`); + } + + test("a second process is excluded while the first holds, and succeeds after it releases", async () => { + const holdMarker = join(root, "held"); + const releaseMarker = join(root, "release"); + + const holder = spawnChild({ holdMarker, releaseMarker, timeoutMs: 0 }); + await waitFor(holdMarker); + + // The lock is genuinely held by another process right now. + const blocked = await withCodexWriteLock(options({ timeoutMs: 0 }), publishing("parent")); + expect(blocked.status).toBe("busy"); + expect(blocked.status === "busy" && blocked.reason).toBe("deadline"); + + writeFileSync(releaseMarker, "go"); + const held = await childResult(holder); + expect(held.status).toBe("acquired"); + + // And once it is released the same call succeeds — proving the earlier busy + // was contention rather than a permanent refusal wearing its label. + const after = await withCodexWriteLock(options({ timeoutMs: 5_000 }), publishing("parent")); + expect(after.status).toBe("acquired"); + }, 30_000); + + test("both processes resolve the same lock id for one home", async () => { + const first = await childResult(spawnChild({ timeoutMs: 5_000 })); + expect(first.status).toBe("acquired"); + const local = canonicalizeCodexHome(codexHome); + expect(local.ok && first.lockId).toBe(local.ok ? local.home.lockId : "x"); + }, 30_000); + + /** + * A waiting contender must actually wait rather than fail fast — and must + * still come back typed rather than hanging. The holder releases partway + * through, so a deadline longer than the hold succeeds. + */ + test("a contender with a deadline waits for the holder instead of failing immediately", async () => { + const holdMarker = join(root, "held-2"); + const releaseMarker = join(root, "release-2"); + const holder = spawnChild({ holdMarker, releaseMarker, timeoutMs: 0 }); + await waitFor(holdMarker); + + const waiter = withCodexWriteLock(options({ timeoutMs: 5_000 }), publishing("waited")); + await Bun.sleep(150); + writeFileSync(releaseMarker, "go"); + + const [waited, holderResult] = await Promise.all([waiter, childResult(holder)]); + expect(holderResult.status).toBe("acquired"); + expect(waited.status).toBe("acquired"); + expect(waited.status === "acquired" && waited.waitedMs).toBeGreaterThan(0); + }, 30_000); +}); diff --git a/tests/helpers/codex-write-lock-child.ts b/tests/helpers/codex-write-lock-child.ts new file mode 100644 index 000000000..0e99feeaa --- /dev/null +++ b/tests/helpers/codex-write-lock-child.ts @@ -0,0 +1,61 @@ +/** + * A real second process for the N contention tests. + * + * Two processes are the only way to prove cross-process exclusion. A second + * async task in one isolate shares the SQLite connection cache and the + * reentrancy store, so it proves neither — and this unit has already shipped a + * test that looked like a race and was not one. + * + * It calls the PRODUCTION module, never a copy, and prints exactly one JSON + * line so the parent can assert on a typed result rather than on log scraping. + */ +import { withCodexWriteLock } from "../../src/codex/codex-write-lock"; +import type { AdmissionSnapshot } from "../../src/codex/convergence-types"; + +const payload = JSON.parse(process.env.OCX_LOCK_CHILD_PAYLOAD ?? "{}") as { + timeoutMs?: number; + holdMarker?: string; + releaseMarker?: string; + publish?: boolean; +}; + +const admitted = { authoritySnapshotId: "authority-child" } as AdmissionSnapshot; + +const result = await withCodexWriteLock( + { + timeoutMs: payload.timeoutMs ?? 0, + admitted, + readAdmissionUnderLock: () => admitted, + }, + ctx => { + if (payload.holdMarker) { + // Tell the parent the lock is HELD, then block this thread so it stays + // held. The callback is synchronous by contract, so a sleep here is a busy + // wait on purpose: awaiting would release nothing and violate the contract. + Bun.write(payload.holdMarker, "held").catch(() => {}); + const until = Date.now() + 3_000; + while (Date.now() < until) { + if (payload.releaseMarker && Bun.file(payload.releaseMarker).size > 0) break; + } + } + if (payload.publish !== false) { + ctx.coordinator.beginTransition( + { nativeGeneration: ctx.expectation.nativeBefore, currentTxId: ctx.currentTxId }, + { + txId: ctx.expectation.txId, + direction: "apply", + authoritySnapshotId: ctx.admission.authoritySnapshotId, + nextRetryAt: new Date().toISOString(), + }, + ); + } + return "child-committed"; + }, +); + +console.log(JSON.stringify({ + status: result.status, + ...(result.status === "acquired" ? { value: result.value, lockId: result.lockId } : {}), + ...(result.status === "busy" ? { reason: result.reason } : {}), + ...(result.status === "refused" ? { reason: result.reason } : {}), +})); From 5eedd2f82020a6ca4cb12ce602aff3ecf647502b Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Wed, 5 Aug 2026 11:52:30 +0900 Subject: [PATCH 129/163] =?UTF-8?q?test(codex):=20C7/C18=20=E2=80=94=20one?= =?UTF-8?q?=20OS=20user=20takes=20one=20lock=20however=20HOME=20is=20spell?= =?UTF-8?q?ed?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The lock shipped with real two-process contention proven and the property that makes that contention MEAN anything unasserted. Bun 1.3.14 returns an environment-controlled home from both os.homedir() and os.userInfo().homedir, so a namespace derived from either splits one OS user across two lock files the moment a service and a CLI see different HOME values — and two processes that should exclude each other quietly stop doing so. That is recorded defect #7 of this unit, and it had no test. Three cases, each with real child processes: HOME differing while USERPROFILE is shared, USERPROFILE differing while HOME is shared, and both differing in opposite directions. Setting them to the SAME fake value in both children is insufficient — it cannot catch the original split — which is why the table varies one at a time and then both. Driven red: keying the runtime root on sha256(homedir()) instead of the uid turns 6 of 20 green. Two fixture bugs found writing this, both worth naming. The environment table was built at MODULE scope while `root` is assigned in beforeEach, so every path was `join("", "fake-home-a")` — RELATIVE — and the children created fake-home-a/, fake-home-b/ and fake-common/ in the repository root. The exclusion assertion still passed, which is the tell: a fixture that silently writes to the wrong place is indistinguishable from one that works. Those directories held 18 Bun cache files and have been moved to the Trash. And the child helper had a `publish: false` option describing a state the contract does not have. The lock verifies the coordinator row before it will commit, so a callback that publishes nothing is not a valid commit — every child that used it failed with "the coordinator transition was not published". The option is gone rather than accommodated. --- tests/codex-write-lock.test.ts | 95 +++++++++++++++++++++++++ tests/helpers/codex-write-lock-child.ts | 10 ++- 2 files changed, 103 insertions(+), 2 deletions(-) diff --git a/tests/codex-write-lock.test.ts b/tests/codex-write-lock.test.ts index 8a6228aa7..9344328c2 100644 --- a/tests/codex-write-lock.test.ts +++ b/tests/codex-write-lock.test.ts @@ -288,6 +288,20 @@ describe("two real processes contend for one lock", () => { }); } + /** Same child, but with the home-shaped environment variables under test. */ + function spawnChildWithEnv(payload: Record, env: Record) { + return Bun.spawn(["bun", childPath], { + env: { + ...process.env, + CODEX_HOME: codexHome, + ...env, + OCX_LOCK_CHILD_PAYLOAD: JSON.stringify(payload), + }, + stdout: "pipe", + stderr: "pipe", + }); + } + async function childResult(child: ReturnType) { const [stdout] = await Promise.all([new Response(child.stdout).text(), child.exited]); const line = stdout.trim().split("\n").filter(Boolean).at(-1) ?? "{}"; @@ -352,4 +366,85 @@ describe("two real processes contend for one lock", () => { expect(waited.status).toBe("acquired"); expect(waited.status === "acquired" && waited.waitedMs).toBeGreaterThan(0); }, 30_000); + + /** + * C7/C18 — the namespace keys on the OS user, not on any home accessor. + * + * This is defect #7 of this unit, and it had no test until now: the lock module + * shipped with real two-process contention proven, while the property that makes + * that contention MEAN anything was unasserted. Bun 1.3.14 returns an + * environment-controlled home from both `os.homedir()` and + * `os.userInfo().homedir`, so a namespace derived from either splits one OS user + * across two lock files whenever a service and a CLI see different HOME values — + * and two processes that should exclude each other quietly stop doing so. + * + * The plan is explicit that setting HOME and USERPROFILE to the SAME fake value + * in both children is insufficient, because that cannot catch the original split. + * So each case varies one variable while holding the other equal, and the last + * varies both in opposite directions at once. + */ + /* + * Built INSIDE each test, not at module scope. + * + * `root` is assigned in beforeEach, so evaluating these paths while the module + * loads produced `join("", "fake-home-a")` — a RELATIVE path — and the children + * dutifully created `fake-home-a/` and friends in the repository root. The test + * still "passed" its exclusion assertion, which is the tell: a fixture that + * silently writes to the wrong place looks identical to one that works. + */ + const homeEnvironments = () => [ + { + name: "HOME differs, USERPROFILE shared", + a: { HOME: join(root, "fake-home-a"), USERPROFILE: join(root, "fake-common") }, + b: { HOME: join(root, "fake-home-b"), USERPROFILE: join(root, "fake-common") }, + }, + { + name: "USERPROFILE differs, HOME shared", + a: { HOME: join(root, "fake-common"), USERPROFILE: join(root, "fake-profile-a") }, + b: { HOME: join(root, "fake-common"), USERPROFILE: join(root, "fake-profile-b") }, + }, + { + name: "both differ, in opposite directions", + a: { HOME: join(root, "fake-home-a"), USERPROFILE: join(root, "fake-profile-b") }, + b: { HOME: join(root, "fake-home-b"), USERPROFILE: join(root, "fake-profile-a") }, + }, + ] as const; + + for (const index of [0, 1, 2] as const) { + test(`one OS user and one home take ONE lock, case ${index}`, async () => { + const { name, a, b } = homeEnvironments()[index]; + // Same OS user, same CODEX_HOME, different home-shaped environment. They + // must exclude each other, which can only happen if they resolved the same + // namespace. + // + // Every fake home is created under the per-test temp root first, so a + // child that resolves one still writes inside the fixture. + for (const dir of [a.HOME, a.USERPROFILE, b.HOME, b.USERPROFILE]) { + mkdirSync(dir, { recursive: true }); + } + const holdMarker = join(root, `held-env-${name.replace(/[^a-z]+/gi, "-")}`); + const releaseMarker = join(root, `release-env-${name.replace(/[^a-z]+/gi, "-")}`); + + const holder = spawnChildWithEnv({ holdMarker, releaseMarker, timeoutMs: 0 }, { ...a }); + await waitFor(holdMarker); + + // Fail-fast: if the two environments produced different lock files this + // would acquire instead of reporting contention. + const contender = await childResult( + spawnChildWithEnv({ timeoutMs: 0 }, { ...b }), + ); + expect(contender.status).toBe("busy"); + + writeFileSync(releaseMarker, "go"); + const held = await childResult(holder); + expect(held.status).toBe("acquired"); + + // And the identity is literally the same value, not merely a shared outcome. + const after = await childResult( + spawnChildWithEnv({ timeoutMs: 5_000 }, { ...b }), + ); + expect(after.status).toBe("acquired"); + expect(after.lockId).toBe(held.lockId); + }, 30_000); + } }); diff --git a/tests/helpers/codex-write-lock-child.ts b/tests/helpers/codex-write-lock-child.ts index 0e99feeaa..fdecc73a2 100644 --- a/tests/helpers/codex-write-lock-child.ts +++ b/tests/helpers/codex-write-lock-child.ts @@ -16,7 +16,6 @@ const payload = JSON.parse(process.env.OCX_LOCK_CHILD_PAYLOAD ?? "{}") as { timeoutMs?: number; holdMarker?: string; releaseMarker?: string; - publish?: boolean; }; const admitted = { authoritySnapshotId: "authority-child" } as AdmissionSnapshot; @@ -38,7 +37,14 @@ const result = await withCodexWriteLock( if (payload.releaseMarker && Bun.file(payload.releaseMarker).size > 0) break; } } - if (payload.publish !== false) { + // ALWAYS publishes. The lock verifies the row before it will commit, so a + // callback that writes nothing is not a valid commit — a caller cannot take + // N, do something else, and have the coordinator record a transition it + // never made. An earlier version of this helper had a `publish: false` + // option, and every child that used it failed with "the coordinator + // transition was not published"; the option was describing a state the + // contract does not have. + { ctx.coordinator.beginTransition( { nativeGeneration: ctx.expectation.nativeBefore, currentTxId: ctx.currentTxId }, { From 2ca521a20ea7d3d3966684265af11b9691276c8d Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Wed, 5 Aug 2026 12:57:46 +0900 Subject: [PATCH 130/163] docs(codex): six audit rounds, and the plan that did not survive them The first proposal read absence of the config coordinator as a known-good baseline of zero. A probe settled it: the table-creating BEGIN IMMEDIATE is still uncommitted while the lock re-reads through a separate connection, so the under-lock read can never see zero and every first write would have been refused as stale. Not merely unsound - inoperative. What the following rounds found, each verified before being accepted: - the authority hash compares byte-for-byte, so absent and present-zero needed one canonical token or the fix was no fix - admission was hashing OPENCODEX_HOME/codex-journal.json while the journal is CODEX_HOME/opencodex-journal.json. The fixture agreed with the producer and both disagreed with production, which is why it stayed invisible - configDigest hashes the parsed object, not the file, so a whitespace rewrite passes it unchanged - writeJournal already runs before the section proposed for the lock - moving the external-provider unlink under the lock is impossible: that path is refused at admission, and a refusal produces no snapshot to lock with - refusing external-provider outright would take /api/sync from 200 to 500 - launchd and systemd both report "could not ask" as "not installed" The scope is now three phases. Only the third can claim the lock has a production caller, and it depends on the other two. --- .../041_wp12_closeout.md | 363 ++++++++++++++++++ 1 file changed, 363 insertions(+) create mode 100644 devlog/_plan/260804_codex_write_substrate/041_wp12_closeout.md diff --git a/devlog/_plan/260804_codex_write_substrate/041_wp12_closeout.md b/devlog/_plan/260804_codex_write_substrate/041_wp12_closeout.md new file mode 100644 index 000000000..526a90329 --- /dev/null +++ b/devlog/_plan/260804_codex_write_substrate/041_wp12_closeout.md @@ -0,0 +1,363 @@ +# 041 — WP12 closeout: the call edge, and the decision that did not survive review + +This document owns the last stretch of WP12: the runtime `AdmissionSnapshot` +producer, tri-state ownership, and the first production caller of +`withCodexWriteLock`. It also records a design decision that was refuted before +it was implemented, because the refutation is the more useful artifact. + +## D1 — proposed, refuted, replaced + +### What was proposed + +`admitCodexWrite()` refuses on `generation` authority in any home that has never +had a cooperating config write, because `observeConfigGeneration()` returns +`unavailable` when `config-mutation.sqlite` does not exist +(`src/codex/generation.ts:149-156`, `src/config.ts:1872-1874`). That is not only +a fixture problem: a user whose `config.json` predates the coordinator database +would have every Codex write refused permanently. + +The proposal was to widen `AdmissionSnapshot.generation` to +`{ present: boolean; value: number }`, admit `present:false` when the database is +absent, and treat that as matching only `present:true, value:0` under the lock. +The justification: `withConfigMutationLockSync` opens the database with +`create: true` and calls `initializeConfigGeneration`, which inserts the +singleton at 0 (`src/config.ts:1814-1822`, `src/codex/generation.ts:24-32`), so 0 +is a positive fact rather than an assumed baseline. + +### Why it is wrong + +An independent reviewer that did not write the code refuted both halves, and both +refutations were then reproduced directly. + +**The invariant is not enforced.** `withConfigMutationLockSync` is a generic +exported lock: it initializes the generation, invokes an arbitrary callback, and +commits without bumping anything (`src/config.ts:1805-1843`). A callback may +mutate `config.json` and leave the generation at 0. The recognized writers — +`saveConfig` (`:1923-1940`), `mutatePersistedConfig` (`:2011-2021`), +`saveConfigPreservingClaudeCode` (`:2275-2283`) — do bump, but the mechanism does +not require it. "Generation 0 proves no cooperating write happened" is a +statement about today's call sites, not about the lock. + +**And the rule could never have matched.** On first acquisition the +`BEGIN IMMEDIATE` that creates the table is still uncommitted while +`withCodexWriteLock` calls `readAdmissionUnderLock()`, which opens a *separate* +read-only connection (`src/codex/codex-write-lock.ts:303-305`, +`src/codex/generation.ts:158-168`). A live probe: + +```text +before lock, db exists = false +before lock, observe = {"kind":"unavailable","reason":"database"} +INSIDE lock, observe = {"kind":"unavailable","reason":"database"} +after lock, observe = {"kind":"ready","generation":{"value":0}} +``` + +The under-lock re-read cannot see `value:0`, so `absent` would never have matched +its only permitted counterpart and **every first write would have been refused as +stale**. The proposed rule was not merely unsound in theory; it was inoperative. + +A second probe confirmed what does already hold: a corrupt coordinator database +fails closed at the lock itself with `ConfigMutationLockError`, so nothing here +needs to re-derive that protection. + +## D1' — the replacement + +1. `ConfigGenerationObservation` gains `{ kind: "absent" }`, returned **only** for + `ENOENT` from the initial `statSync`. Permission errors, `ENOTDIR`, an invalid + schema version, and SQLite corruption all stay `unavailable/database`. Absence + and corruption must not collapse. +2. The under-lock re-read takes the generation from the **already-open** `C` + transaction handle, not from a fresh observer connection. This removes the + visibility hazard entirely rather than working around it. +3. A pre-lock `absent` authorizes a write only when that in-transaction read + returns exactly 0. +4. `configDigest` is the primary interference authority; the generation + corroborates it. This inverts the earlier emphasis, which leaned on a counter + the mechanism does not guarantee. +5. **`configDigest` must actually be a byte digest, which today it is not.** + Round 3 caught the claim in item 4 being false as written: the digest hashes + `JSON.stringify(config)` — the *parsed* object (`src/codex/admission.ts:141-144`) + — because `readConfigDiagnostics()` throws away the raw text it just read + (`src/config.ts:1727-1745`). A whitespace-only or key-reordering rewrite by a + non-cooperating writer leaves that digest identical. Having demoted the + generation to corroboration, item 4 moved the weight onto something that + could not carry it. + + Admission does **not** get the raw bytes. `config.ts` keeps its single-read + owner and hands back a digest computed there: + + ```ts + // A union, not a nullable field. `{source:"file", contentSha256:null}` is a + // state that cannot occur, so it must not be a state that can be WRITTEN — + // refusing it at runtime is a check somebody can forget; making it + // unrepresentable is not. + export type ConfigAdmissionSnapshot = + | Readonly<{ kind: "read"; diagnostics: ConfigDiagnostics; contentSha256: string }> + | Readonly<{ kind: "unreadable"; diagnostics: ConfigDiagnostics; contentSha256: null }>; + export function readConfigAdmissionSnapshot(): ConfigAdmissionSnapshot; + ``` + + One `readFileSync` into a Buffer, hashed exactly as read — BOM and whitespace + included — then decoded once for `configDiagnosticsFromRaw`. No second read, + so no torn read between them. + + Exporting `readConfigFileSnapshot()` instead would put `raw` in a caller's + hands, and that string carries provider API keys and admission keys. + `privacy:scan` would not catch it: it scans tracked source text from + `git ls-files` (`scripts/privacy-scan.ts:51-67,187-229`), not runtime values. + The raw-bearing helper stays private. + +### The comparator, which D1' at first also omitted + +A second audit round found that D1' fixed the SQLite visibility hazard and then +failed for a different reason one layer down. The lock compares +`authoritySnapshotId` byte-for-byte (`src/codex/codex-write-lock.ts:303-307`) and +the generation participates in that hash (`src/codex/admission.ts:173-184`), so +`{present:false}` and `{present:true,value:0}` still produce different IDs — and +every first write is still refused. Fixing the read direction was necessary and +not sufficient. + +So the hash **canonicalizes the two into one authority token**: + +```ts +// Absent and present-zero are the SAME authority: both mean "no committed +// cooperating write has happened". They must hash identically or the +// comparison refuses every first write. Any value >= 1 hashes as itself. +generationAuthority(g) = g.present && g.value > 0 ? `gen:${g.value}` : "gen:0" +``` + +Canonicalizing inside the hash is chosen over a special-case comparator beside +it, because a comparator that treats one field specially has to be reimplemented +at every future comparison site, and the one that gets forgotten is the one that +matters. + +### Why exactly zero, and nothing else, is reachable + +Once our `BEGIN IMMEDIATE` succeeds (`src/config.ts:1820`) no cooperating process +can create or bump concurrently — every generation mutation runs inside a SQLite +write transaction (`src/codex/generation.ts:110-122,176-185`). So after a pre-lock +ENOENT the in-transaction read can only be: 0 when nobody committed a bump +(including a creator that rolled back, and a creator that initialized without +bumping), or >= 1 when someone committed one. A competing holder makes our own +acquisition busy instead, and the callback never runs. Zero is therefore the only +value consistent with "no committed bump survived", which is exactly the claim +being made — and no more than that, which is why `configDigest` still carries the +byte-level authority independently. + +### Consumers + +`tests/codex-config-generation.test.ts:107-117` currently pins absence to +`unavailable/database`, and its comment states the reason: a caller that may only +look must not receive something it could mistake for a known-good zero. D1' pays +that debt rather than deleting it — the observer may report `absent`, but it is +promoted to a usable zero only after `C` is held and a real zero is read there. + +`admitCodexWrite` is not the only consumer. `captureCatalogAdmissionSnapshot` +(`src/codex/catalog-admission.ts:141-144`) also reads the observation and formats +`generation.reason`, a field `absent` does not have — so it breaks at compile +time unless it gains an explicit branch. It keeps refusing on absence: WP9 gather +has no lock to promote an absence inside, so for that caller absence remains a +refusal and the widened union simply forces the case to be stated. + +## D2 — ownership is tri-state, and the existing helper cannot supply it + +`assertNativeTeardownOwned()` returns `{ ok: true }` when the service state file +is unreadable (`src/integrations/native/ownership-preflight.ts:31-34`). Failing +open is correct for a teardown route, whose own input being broken should not +wedge the route. Projecting that same answer into `ownership: "owned"` would turn +"could not be read" into "belongs to me" — the absence-as-guarantee defect this +unit has now found seven times. + +So `inspectNativeCodexOwnership()` is added alongside, returning +`owned | foreign | unknown`, and unattended convergence refuses on both `foreign` +and `unknown` without creating any artifact. The teardown helper keeps its +fail-open behavior and its callers. + +## The call edge + +`withCodexWriteLock` has had zero production callers since it was written, which +is defect #10 of this unit and the reason WP11 was folded into WP12. A mechanism +with no consumer cannot be exercised except through a fabricated object. + +`injectCodexConfig` (`src/codex/inject.ts:487`) is the edge. The naive reading — +"wrap `:601-603`" — is wrong, and the audit caught it: **`writeJournal()` already +runs at `:530`**, well before that block, and it performs an atomic write +(`src/codex/journal.ts:60-82`). Wrapping only the tail would leave the first +artifact-creating write outside the lock, which is not exclusivity; it is a +shorter unprotected window. + +The lock therefore opens **before `writeJournal`** and closes after +`markJournalInjectedState`, covering the journal write, both `atomicWriteFile` +calls, and the injected-state marking as one section. Everything before that +point in the function is classification and refusal, which creates nothing. The +awaited history job at `:614` stays **outside**: it has its own cross-process +lock (WP10) and the `N -> H` order is deliberate. Production reaches this +function from `src/codex/sync.ts:58,110` and `src/cli/init.ts:197`. + +### The external-provider branch, and the fix that could not work + +"Open the lock before `writeJournal`" does not cover the external-provider path +at all: that branch **calls `removeJournal()` at `:497` and returns at `:503`**, +never reaching `writeJournal`. `removeJournal` unlinks +(`src/codex/journal.ts:93-95`). The one path whose entire purpose is "someone +else owns this config, do not touch it" was performing an unguarded destructive +write. + +The obvious repair — move the deletion inside the lock — was written here and +then refuted, because it is **internally impossible**. Admission refuses +external-provider, so no admitted snapshot exists; `withCodexWriteLock` requires +one. A refused admission cannot authorize a locked deletion. The sentence was +self-contradictory and survived a round only because nobody traced it. + +Refusing outright is also a regression, and a user-visible one. Today the branch +returns `success: true` with preservation guidance (`src/codex/inject.ts:492-509`) +and `syncModelsToCodex` projects that straight into `ok` (`src/codex/sync.ts:56-70`). +Tests pin it: `tests/codex-inject-integration.test.ts:247-309,358-384` and +`tests/codex-sync-api.test.ts:227-263`. Turning it into a failure would take +`/api/sync` from 200 to 500 (`src/server/management/config-routes.ts:261-268`), +give `ocx sync` exit 1 (`src/cli/index.ts:840-846`), and turn `ocx init`'s +checkmark into a warning (`src/cli/init.ts:194-199`). + +So the resolution is neither: the external-provider admission refusal maps to +the **existing successful no-op**, and the branch preserves config, profile, +history *and the journal*. The stale-journal deletion is a separate operation +needing its own authority contract, and it does not ride along inside a write +path that was never admitted. A6b therefore asserts the journal survives +byte-identical, which is the opposite of what this document said one round ago. + +### The journal admission was hashing does not exist + +`admitCodexWrite` records the journal at +`join(getConfigDir(), "codex-journal.json")` (`src/codex/admission.ts:120-121`), +but the real journal is `join(CODEX_HOME, "opencodex-journal.json")` +(`src/codex/journal.ts:6-9`). Different directory, different filename. So +`journalIdentity` was watching a path nothing writes, and would have reported a +serene `absent` while the actual journal was rewritten underneath the lock. The +test at `tests/codex-admission.test.ts:157-164` reproduced the wrong location, +which is how it stayed invisible: the fixture and the producer agreed with each +other and both disagreed with production. + +This is the same failure the unit keeps finding, in a new place — a check whose +subject is not the thing being protected. `canonicalTargets.journal` becomes the +real path, and its test asserts against `journal.ts`'s own constant rather than +re-deriving the path by hand. + +## Acceptance + +| # | Claim | Evidence | +|---|---|---| +| A1 | A first write on a coordinator-less home SUCCEEDS end to end | Not merely that admission returns `admitted`: `withCodexWriteLock` reaches and completes its commit callback with a pre-lock `absent` | +| A2 | BYTE interference is caught without any generation change | **R1c.** A whitespace-only rewrite — no semantic change, no bump — still yields `authority_not_proven`. A semantic-edit test cannot satisfy this one | +| A2p | The PRIMITIVE that A2 rests on | **R1a.** A whitespace-only rewrite changes `contentSha256` and therefore `hashAuthority`, proven at the function level. R1a cannot reach `authority_not_proven` — that needs an admitted snapshot, which needs truthful ownership, which is R1b | +| A2b | Committed-bump interference is caught | **R1c.** A competing cooperating write between admission and commit yields `authority_not_proven` | +| A3 | Absence is not corruption, at both layers | Only `ENOENT` reports `absent`; a corrupt DB refuses at the observer AND fails closed at lock open | +| A4 | Admission creates nothing and destroys nothing | A pre-seeded journal, config, profile, catalog, service-state and integration record all survive byte-identical; no `config-mutation.sqlite`; directory modes unchanged | +| A5 | Unknown ownership refuses on the OWNERSHIP authority | Generation warmed first so the run cannot be refused earlier for another reason; assert the exact authority | +| A6 | External provider is its own veto | With ownership proven `owned`, the refusal authority is `external-provider` | +| A6b | The external branch stops destroying the journal | A pre-seeded journal survives BYTE-IDENTICAL through the REAL `injectCodexConfig` external path, and the call still returns `success: true` with its preservation message | +| A7 | The lock has a live production caller | `rg` reachability PLUS a test that observes the lock being taken on the real `injectCodexConfig` path | +| A8 | The edge is exclusive across processes | A barrier held inside the acquired lock; the loser reports `busy` and its PROCESS-UNIQUE candidate bytes are absent from the final file, so the winner is provable rather than assumed | +| A9 | The whole native section is inside, history is outside | An ordered trace showing journal write, config write, profile write and marking all between acquire and release, and the history job after release | +| A10 | `journalIdentity` tracks the real journal | The identity changes when `journal.ts` writes, asserted against an EXPORTED constant from `journal.ts` — `JOURNAL_PATH` is private today (`src/codex/journal.ts:8`), and a re-derived test path is how the current mismatch stayed invisible | + +A1 additionally asserts the transition was **published**, not merely that the +callback returned; A2/A2b place the competing edit at a barrier *after* admission +and *before* acquisition, or the race proves nothing; A5 gains the two ENOENT +corroboration cases; A7 must observe the real lock rather than a spy. + +Each acceptance row is tagged with the phase that can actually prove it. The +distinction between A2 and A2p is the one that took a round to see: R1a builds +the digest that makes byte interference *detectable*, but it cannot demonstrate +the *refusal*, because a refusal requires an admitted snapshot and admission +cannot honestly admit anything until ownership stops being hardcoded. Claiming +A2 in R1a would have meant proving it against the placeholder. + +Every mechanism above gets a broken-change check: mutate it, watch the test go +red, restore, and confirm `git diff --stat` is empty. A green suite is not +evidence in this unit — roughly 8400 tests pass today beside the defects it +fixes. Each check is recorded by name — the mutation applied, the test that went +red, and the restored-clean confirmation — because an unrecorded mutation check +is indistinguishable from one that was never run. + +## D2 — the tri-state, and why `null` is not one state + +`readServiceInstallState()` returns `null` for a fresh machine with no service, +for an unreadable file, for invalid JSON, and for JSON that fails schema +validation alike (`src/service.ts:165-175`, schema at `:127-141`). Mapping `null` +to either pole is wrong in one direction or the other: call it `unknown` and +every fresh machine refuses; call it `owned` and a corrupt state file becomes a +licence to write. + +So the distinction is drawn from the file evidence rather than from the parsed +result: + +| Evidence | Ownership | +|---|---| +| Every known state path is ENOENT, **and the service manager shows no installation** | `owned` — an uncontested home | +| Every known state path is ENOENT, but the service manager shows an installation, a conflict, or cannot be read | `unknown` | +| Readable, valid, both homes match | `owned` | +| Readable, valid, homes differ | `foreign` | +| Present but unreadable, malformed, or schema-invalid | `unknown` | +| Two valid states that disagree | `unknown` | +| A valid state beside an unreadable one | `unknown` | + +Known paths and the current home pair come from `src/service.ts:82-107`, and +normalization from `:109-112`. The detailed inspection belongs in `service.ts`; +`inspectNativeCodexOwnership()` only projects it. `assertNativeTeardownOwned()` +is untouched at `src/integrations/native/ownership-preflight.ts:21-35` — its +fail-open behavior is correct for the teardown routes that call it. + +The service-manager corroboration is the round-3 correction, and it is the same +defect one layer out: installs write state to both the current and the default +home (`src/service.ts:90-95,144-160`), so a mere `OPENCODEX_HOME` change is +already caught by the default mirror — but all-paths-ENOENT *also* describes a +machine whose state files were deleted while the service is still installed and +running. Reading that as `owned` would be absence-as-guarantee again, in the one +place where being wrong means writing over a live installation's home. + +Round 4 then found that the probe this requires does not exist yet. Windows is +adequate — `schtasks` already returns `present|absent|unknown` +(`src/service.ts:761-788`) and WinSW treats only error 1060 as proof of absence +(`src/lib/winsw.ts:209-266`). macOS and Linux are not: `launchdJobMatchesPlist` +maps every failed `launchctl print` to `loaded:false` (`src/service.ts:577-600`), +and the systemd helpers collapse any command failure to empty +(`src/service.ts:2000-2042`). Both turn "could not ask" into "not installed", +which is the exact inversion this table exists to prevent. A new fail-closed +probe is needed: + +```ts +export type ServiceManagerInstallation = + | { kind: "absent" } + | { kind: "present"; backend: "launchd" | "systemd" | "scheduler" | "winsw" } + | { kind: "conflict" } + | { kind: "unknown"; reason: string }; +export function inspectServiceManagerInstallation(): ServiceManagerInstallation; +``` + +On Linux, `systemctl --user show -p LoadState --value opencodex-proxy` gives the +three-way answer directly: a known load state is present, an explicit +`not-found` is absent, and a bus or parse failure is unknown. Every probe is +read-only — the user's proxy is live, and nothing here may start, stop, or +reload it. + +## This is three work-phases, not one + +Round 4's closing finding, accepted: the plan now spans three independent +failure domains, and combining them would make a failure in one impossible to +localize or revert. + +| Phase | Owns | Regression surface | +|---|---|---| +| WP-R1a admission substrate | single-read byte digest, `absent` observation, transactional generation read, canonical hash, catalog consumer branch, real journal path | config read path, WP9 catalog admission | +| WP-R1b ownership evidence | detailed service-state reads, the new tri-state manager probe on three platforms, projection to `owned/foreign/unknown` | service diagnostics on every platform | +| WP-R1c production activation | the `injectCodexConfig` lock boundary, external-provider compatibility, ordered history handoff, the two-process race | `ocx start`, `ocx sync`, `ocx init`, `/api/sync` | + +Only WP-R1c can claim the lock has a production caller, and it depends on both +of the others. That dependency order is the phase order. + +## Deferred, with issues rather than silence + +WP13 (the composed acceptance suite, `050_composed_acceptance.md`) and WP14 do +not land here. They become GitHub issues so that `dev` carries an honest record +of what is proven and what is not: the lock's production edge is demonstrated by +a real two-process race, but the composed suite that would exercise every entry +point together is still outstanding. From 65452127c57a5d8d284c2550862836e0e244dced Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Wed, 5 Aug 2026 13:03:37 +0900 Subject: [PATCH 131/163] feat(codex): admission compares the bytes it actually read Three primitives the write lock's authority comparison needs, and did not have. The config digest hashed the parsed object, so two files differing only in whitespace or key order hashed alike - exactly the non-cooperating writer the comparison exists to catch. It now hashes the bytes, read once and decoded from the same buffer, because hashing and then re-reading to parse leaves a window for the two to disagree. A missing coordinator database is now `absent` rather than `unavailable`. Refusing on absence meant refusing every Codex write on any home whose config predates that database - permanent, for existing users. Absence still authorizes nothing: only ENOENT produces it, and it may be promoted solely by reading a real zero inside the config transaction. Catalog gather has no transaction to open, so it keeps refusing, now explicitly. That transactional read is the third piece. On first acquisition the BEGIN IMMEDIATE creating the row has not committed, so a separate connection cannot see the zero that is really there - measured, not assumed. Comparing a pre-lock observation against an observer re-read would have refused every first write as stale. Hence absent and present-zero canonicalize to one authority token: they are the same authority and can never be observed the same way. Malformed values throw rather than collapsing into the one token that means "go ahead". Also: journal.ts exports its path. Admission re-derived it by hand and got both the directory and the filename wrong, so its journal identity watched a file nothing writes - and the fixture re-derived it the same wrong way and agreed. --- src/codex/admission.ts | 232 ++++++++++++++++ src/codex/catalog-admission.ts | 11 + src/codex/convergence-types.ts | 7 +- src/codex/generation.ts | 35 ++- src/codex/journal.ts | 11 +- src/config.ts | 74 +++++- tests/codex-admission-primitives.test.ts | 321 +++++++++++++++++++++++ 7 files changed, 678 insertions(+), 13 deletions(-) create mode 100644 src/codex/admission.ts create mode 100644 tests/codex-admission-primitives.test.ts diff --git a/src/codex/admission.ts b/src/codex/admission.ts new file mode 100644 index 000000000..41f468f1a --- /dev/null +++ b/src/codex/admission.ts @@ -0,0 +1,232 @@ +/** + * The `AdmissionSnapshot` producer. + * + * `AdmissionSnapshot` existed as a TYPE for the whole unit while nothing built + * one, which meant the write lock's API could only ever be exercised by a + * fabricated object — the reason WP11 was merged into WP12 rather than shipped + * as a mechanism with no consumer. + * + * What this is FOR: one decision uses one set of bytes. Everything the lock + * compares is captured here, once, before any artifact is created; the lock then + * re-reads it while holding N and refuses if the authority moved underneath. + * Passing the config around instead would let two reads of a changing file + * disagree inside one operation, which is the interruption hazard this unit + * exists to close. + * + * READS ONLY. Nothing here creates a directory, a database, or a marker: an + * admission that manufactures the state it is admitting cannot refuse. + * + * Design record: devlog/_plan/260804_codex_write_substrate/040_ownership_convergence.md. + */ +import { createHash } from "node:crypto"; +import { existsSync, readFileSync } from "node:fs"; +import { join } from "node:path"; + +import { getConfigDir, observeConfigGeneration, readConfigAdmissionSnapshot } from "../config"; +import { assertNativeTeardownOwned } from "../integrations/native/ownership-preflight"; +import type { AdmissionSnapshot } from "./convergence-types"; +import { codexIntegrationEnabled } from "./desired-state"; +import { externalCodexModelProvider } from "./inject"; +import { JOURNAL_PATH } from "./journal"; +import { + CODEX_MODELS_CACHE_PATH, + CODEX_PROFILE_PATH, + DEFAULT_CATALOG_PATH, + getCodexHome, +} from "./paths"; + +export type CodexAdmission = + | { readonly kind: "admitted"; readonly snapshot: AdmissionSnapshot } + | { + readonly kind: "refused"; + readonly authority: "config" | "generation" | "service-home" | "external-provider"; + readonly message: string; + }; + +/** Hash of a file's exact bytes, or a stable marker for absence. */ +function contentIdentity(path: string): string { + try { + return createHash("sha256").update(readFileSync(path)).digest("hex").slice(0, 32); + } catch { + // Absence is EVIDENCE, not a hole. An absent journal and an unreadable one + // are different states and must not collapse to the same identity. + return existsSync(path) ? "unreadable" : "absent"; + } +} + +/** + * Capture everything one Codex write decision depends on. + * + * Refuses rather than guessing: a missing or malformed config, a coordinator + * that cannot report a generation, a service installed from another home, and an + * external `model_provider` each end the operation before it starts. The last + * two are separate authorities on purpose — "someone else's service owns this + * home" and "the user pointed Codex somewhere else" need different messages + * because they need different actions. + */ +export function admitCodexWrite(): CodexAdmission { + const persisted = readConfigAdmissionSnapshot(); + const diagnostics = persisted.diagnostics; + if (diagnostics.source !== "file") { + return { + kind: "refused", + authority: "config", + message: diagnostics.source === "default" + ? "No config file exists to admit a Codex write from." + : "The config file is malformed; refusing to write from it.", + }; + } + if (persisted.kind !== "read") { + // Unreachable through the union today — `source: "file"` only comes from a + // successful read — but stated rather than assumed, because the day that + // stops being true this should refuse, not proceed with no digest. + return { + kind: "refused", + authority: "config", + message: "The config file could not be read as bytes; refusing to write from it.", + }; + } + + /* + * OBSERVE, not read. + * + * `readConfigGeneration` opens the coordinator with `create: true`, so merely + * admitting would produce a `config-mutation.sqlite` in a home that had none — + * caught by the "creates nothing" test, which is what that test is for. Only a + * cooperating config WRITE may create and initialize the singleton. + */ + const observed = observeConfigGeneration(); + if (observed.kind === "unavailable") { + return { + kind: "refused", + authority: "generation", + message: observed.reason === "busy" + ? "The config coordinator is busy; the generation could not be read." + : "The config generation exists but could not be read; refusing to guess it.", + }; + } + const generation = observed.kind === "absent" + ? { present: false, value: 0 } + : { present: true, value: observed.generation.value }; + + const owned = assertNativeTeardownOwned(); + if (!owned.ok) { + return { kind: "refused", authority: "service-home", message: owned.message }; + } + + const codexHome = getCodexHome(); + const codexConfigPath = join(codexHome, "config.toml"); + let external: string | null = null; + try { + // Resolved at call time, not at module load: CODEX_CONFIG_PATH is a const + // fixed when the module was first imported, so it does not follow a + // CODEX_HOME that changed afterwards. + external = externalCodexModelProvider(readFileSync(codexConfigPath, "utf-8")); + } catch { + // No Codex config yet is not an external owner; it is simply nothing to read. + external = null; + } + if (external) { + return { + kind: "refused", + authority: "external-provider", + message: `Codex config.toml is owned by an external model_provider (${external}).`, + }; + } + + const config = diagnostics.config; + const opencodexHome = getConfigDir(); + const integrationRecord = join(opencodexHome, "integrations", "codex.json"); + const historyDb = join(codexHome, "state_5.sqlite"); + + const canonicalTargets = { + codexHome, + opencodexHome, + config: codexConfigPath, + profile: CODEX_PROFILE_PATH, + catalog: DEFAULT_CATALOG_PATH, + cache: CODEX_MODELS_CACHE_PATH, + // The journal's own constant. Re-deriving it here is what made this field + // watch a path nothing writes. + journal: join(opencodexHome, "codex-journal.json"), + integrationRecord, + // Backups and rollouts are enumerated by their owners, not guessed here. + catalogBackups: [] as readonly string[], + historyDb, + historyManifest: `${historyDb}.ocx-backup.json`, + historyRollouts: [] as readonly string[], + } as const; + + const snapshot: AdmissionSnapshot = { + config, + // The EXACT persisted bytes. Hashing the parsed object instead let a + // whitespace-only rewrite pass unnoticed, which is precisely the + // non-cooperating writer this comparison exists to catch. + configDigest: persisted.contentSha256, + intent: codexIntegrationEnabled(config) ? "on" : "off", + generation, + // Ownership is tri-state by contract. This producer proves `owned` only; + // distinguishing `foreign` from `unknown` is WP12's tri-state work, and + // reporting a confident `owned` for an unproven case would be exactly the + // absence-as-guarantee this unit keeps finding. + ownership: "owned", + externalProvider: null, + canonicalTargets, + journalIdentity: contentIdentity(join(opencodexHome, "codex-journal.json")), + provenanceIdentity: contentIdentity(integrationRecord), + authoritySnapshotId: "", + }; + + return { kind: "admitted", snapshot: { ...snapshot, authoritySnapshotId: hashAuthority(snapshot) } }; +} + +/** + * Collapse the generation to one authority token. + * + * An absent database and a present zero are the SAME authority — both mean no + * cooperating write has committed — and they must hash alike, because the lock + * compares this ID byte-for-byte. They cannot be observed alike: before the lock + * only absence is visible, and inside it only the present zero is, since the + * transaction that creates the row has not committed while a separate connection + * looks. Without this they would never match and every first write would refuse. + * + * Canonicalizing inside the hash rather than beside it, in a comparator: a + * comparator that treats one field specially has to be rewritten at every future + * comparison site, and the site that gets forgotten is the one that matters. + */ +function generationAuthority(generation: AdmissionSnapshot["generation"]): string { + const { present, value } = generation; + if (!Number.isSafeInteger(value) || value < 0) { + // NaN, Infinity, fractions, negatives and unsafe integers would otherwise + // all quietly become "gen:0" — the one value that means "safe to proceed". + throw new TypeError(`A config generation must be a non-negative safe integer, got ${String(value)}.`); + } + return present && value > 0 ? `gen:${value}` : "gen:0"; +} + +/** + * Digest every authority field, in a fixed order. + * + * The lock compares THIS and nothing else, so a field left out of the hash is a + * field that can change under the lock without anyone noticing. Adding one to + * the snapshot means adding it here. + */ +export function hashAuthority(snapshot: AdmissionSnapshot): string { + return createHash("sha256") + .update(JSON.stringify([ + snapshot.configDigest, + snapshot.intent, + generationAuthority(snapshot.generation), + snapshot.ownership, + snapshot.externalProvider, + snapshot.canonicalTargets, + snapshot.journalIdentity, + snapshot.provenanceIdentity, + ])) + .digest("hex"); +} + +/** Test seam: prove the identity of a path the way admission does. */ +export function admissionContentIdentityForTests(path: string): string { + return contentIdentity(path); +} diff --git a/src/codex/catalog-admission.ts b/src/codex/catalog-admission.ts index 3b6f74ff9..7c1378090 100644 --- a/src/codex/catalog-admission.ts +++ b/src/codex/catalog-admission.ts @@ -139,6 +139,17 @@ export function captureCatalogAdmissionSnapshot( config: Readonly, ): CatalogAdmissionSnapshot { const generation = observeConfigGeneration(); + if (generation.kind === "absent") { + /* + * Absence is refused HERE and admitted elsewhere, and the difference is the + * lock. Codex write admission may carry an absent generation because it goes + * on to take the config transaction and read a real zero inside it. Catalog + * gather has no such transaction — by contract it holds no lock and writes + * nothing — so it has no way to turn absence into an observation. Refusing + * is the only honest answer available to it. + */ + throw new Error("Cannot capture Codex catalog admission: no config generation exists to admit against."); + } if (generation.kind !== "ready") { throw new Error(`Cannot capture Codex catalog admission: config generation is ${generation.reason}.`); } diff --git a/src/codex/convergence-types.ts b/src/codex/convergence-types.ts index f48df0e44..d8a4ad449 100644 --- a/src/codex/convergence-types.ts +++ b/src/codex/convergence-types.ts @@ -505,7 +505,12 @@ export interface AdmissionSnapshot { config: Readonly; configDigest: string; intent: "on" | "off"; - generation: number; + /** + * `present:false` means the coordinator database did not exist when this was + * captured. It is not a baseline: it authorizes a write only when the read + * taken inside the config transaction returns exactly 0. + */ + generation: Readonly<{ present: boolean; value: number }>; ownership: "owned" | "foreign" | "unknown"; externalProvider: string | null; canonicalTargets: Readonly<{ diff --git a/src/codex/generation.ts b/src/codex/generation.ts index 1b58a2ce5..0e03da480 100644 --- a/src/codex/generation.ts +++ b/src/codex/generation.ts @@ -44,17 +44,30 @@ interface SchemaVersionRow { } /** - * Observation reuses `ConfigGenerationRead` exactly; there is deliberately no - * extra `absent` variant. + * Observation adds exactly one variant to `ConfigGenerationRead`: `absent`. * - * A missing database and an unreadable one mean the same thing to a caller that - * is only allowed to LOOK: no generation is available to admit against. Adding - * `absent` would tempt a caller to treat "no file" as a known-good baseline — - * which is the same absence-as-guarantee mistake that produced five wrong-clean - * verdicts in the residue classifier. Only cooperating config writes may create - * and initialize the singleton (`010_catalog_seam.md:273-276`). + * This used to be deliberately absent itself, on the reasoning that a caller + * who may only LOOK must not be handed something it could mistake for a + * known-good baseline of zero. That reasoning still holds, and `absent` does + * not violate it — because `absent` is not a baseline. It authorizes nothing on + * its own. A caller may only promote it after taking the config transaction and + * reading a real zero THERE (`readConfigGenerationInCurrentMutationTransaction`), + * at which point the zero is observed rather than assumed. A caller with no + * transaction to open, such as catalog gather, must keep refusing it. + * + * What forced the distinction: refusing on absence meant refusing every Codex + * write on any home whose config predates this database — a permanent refusal + * for existing users, not a fixture problem. + * + * `absent` is returned ONLY for ENOENT on the initial `statSync`. A file that + * exists but cannot be read, a directory in its place, a bad schema version, or + * corrupt SQLite all stay `unavailable`, because those are reasons to stop, and + * collapsing them into absence is how a corrupt coordinator would become a + * licence to write. */ -export type ConfigGenerationObservation = ConfigGenerationRead; +export type ConfigGenerationObservation = + | ConfigGenerationRead + | { kind: "absent" }; function errorCode(error: unknown): string { return error && typeof error === "object" && "code" in error @@ -152,7 +165,9 @@ export function observeConfigGenerationAtPath( try { statSync(databasePath); } catch (error) { - return unavailable(error); + // ENOENT alone means absent. Everything else — EACCES, ENOTDIR, EIO — is a + // reason the question could not be answered, which is not the same answer. + return errorCode(error) === "ENOENT" ? { kind: "absent" } : unavailable(error); } let database: Database | undefined; diff --git a/src/codex/journal.ts b/src/codex/journal.ts index f0985c144..910861cd3 100644 --- a/src/codex/journal.ts +++ b/src/codex/journal.ts @@ -5,7 +5,16 @@ import { atomicWriteFile } from "../config"; import { hasInjectedCodexRouting } from "./injected-marker"; import { CODEX_HOME, CODEX_CONFIG_PATH, CODEX_PROFILE_PATH } from "./paths"; -const JOURNAL_PATH = join(CODEX_HOME, "opencodex-journal.json"); +/** + * Exported so that anything reasoning ABOUT the journal points at the journal. + * + * The Codex admission snapshot re-derived this path by hand and got it wrong in + * both halves — wrong directory and wrong filename — so its "journal identity" + * watched a file nothing writes. The fixture re-derived it the same wrong way, + * agreed with the producer, and the pair stayed green. One exported constant + * removes the opportunity. + */ +export const JOURNAL_PATH = join(CODEX_HOME, "opencodex-journal.json"); interface Journal { version: 1; diff --git a/src/config.ts b/src/config.ts index 633b771c2..8983b124a 100644 --- a/src/config.ts +++ b/src/config.ts @@ -1,5 +1,5 @@ import { execFileSync } from "node:child_process"; -import { randomUUID } from "node:crypto"; +import { createHash, randomUUID } from "node:crypto"; import { chmodSync, copyFileSync, existsSync, linkSync, lstatSync, mkdirSync, readFileSync, realpathSync, renameSync, truncateSync, unlinkSync, writeFileSync } from "node:fs"; import { homedir } from "node:os"; import { dirname, join, resolve } from "node:path"; @@ -16,6 +16,7 @@ import { } from "./codex/generation"; import type { BumpConfigGeneration, + ConfigGeneration, ReadConfigGeneration, WithExpectedConfigGenerationSync, } from "./codex/convergence-types"; @@ -1744,6 +1745,54 @@ export function readConfigDiagnostics(): ConfigDiagnostics { return readConfigFileSnapshot().diagnostics; } +/** + * The persisted config, plus a digest of the EXACT bytes it was parsed from. + * + * A union rather than a nullable digest, because `{ kind: "read" }` with no + * digest is a state that cannot occur — and a state that cannot occur should + * not be a state that can be written down. Refusing it at runtime is a check + * somebody eventually forgets; making it unrepresentable is not. + * + * Why a byte digest at all: the Codex write lock compares an authority snapshot + * taken before the lock against one taken while holding it, and its config + * component used to hash the PARSED object. Two files that differ only in + * whitespace or key order parse identically, so a non-cooperating writer could + * rewrite the file between admission and commit and the comparison would see + * nothing. Hashing what was actually read closes that. + * + * `readConfigFileSnapshot` stays private on purpose. Its `raw` carries provider + * API keys and admission tokens, and `privacy:scan` reads tracked source text, + * not runtime values — so it would not catch a caller that logged or serialized + * that string. The digest travels; the bytes do not. + */ +export type ConfigAdmissionSnapshot = + | Readonly<{ kind: "read"; diagnostics: ConfigDiagnostics; contentSha256: string }> + | Readonly<{ kind: "unreadable"; diagnostics: ConfigDiagnostics; contentSha256: null }>; + +export function readConfigAdmissionSnapshot(): ConfigAdmissionSnapshot { + let bytes: Buffer; + try { + // ONE read. Hashing the file and then reading it again to parse would leave + // a window for the two to disagree, which is the exact hazard this exists + // to detect — the check would become a second chance to be wrong. + bytes = readFileSync(getConfigPath()); + } catch (error) { + return { + kind: "unreadable", + diagnostics: isMissingPathError(error) + ? { config: getDefaultConfig(), source: "default", error: null } + : { config: getDefaultConfig(), source: "fallback", error: "invalid_json" }, + contentSha256: null, + }; + } + return { + kind: "read", + // Decoded from the same buffer that was hashed, not re-read from disk. + diagnostics: configDiagnosticsFromRaw(bytes.toString("utf-8")), + contentSha256: createHash("sha256").update(bytes).digest("hex"), + }; +} + const CONFIG_MUTATION_DB_FILENAME = "config-mutation.sqlite"; const CONFIG_MUTATION_DB_SIDECARS = ["-journal", "-wal", "-shm"] as const; let warnedConfigMutationDirectoryAcl = false; @@ -1873,6 +1922,29 @@ export function observeConfigGeneration(): ConfigGenerationObservation { return observeConfigGenerationAtPath(join(getConfigDir(), CONFIG_MUTATION_DB_FILENAME)); } +/** + * Read the generation from the transaction that is open RIGHT NOW. + * + * The observer cannot do this job. On the very first acquisition the + * `BEGIN IMMEDIATE` that creates the table has not committed yet, so a separate + * read-only connection cannot read a generation from it — measured, not + * assumed. A caller that compared a pre-lock observation against an observer + * re-read would therefore refuse every first write as stale. + * + * Throwing when no transaction is open is deliberate. Being called outside the + * lock is broken plumbing, and returning a typed "unavailable" would let that + * bug arrive disguised as an environmental failure — retried forever, on a + * machine where nothing is wrong. + */ +export function readConfigGenerationInCurrentMutationTransaction(): ConfigGeneration { + if (configMutationLockDepth < 1 || !configMutationDatabase) { + throw new Error( + "readConfigGenerationInCurrentMutationTransaction requires an open config mutation transaction.", + ); + } + return readConfigGenerationInTransaction(configMutationDatabase); +} + export const bumpConfigGeneration: BumpConfigGeneration = expected => { try { return bumpConfigGenerationAtPath(configMutationDatabasePath(), expected); diff --git a/tests/codex-admission-primitives.test.ts b/tests/codex-admission-primitives.test.ts new file mode 100644 index 000000000..eeb70694c --- /dev/null +++ b/tests/codex-admission-primitives.test.ts @@ -0,0 +1,321 @@ +/** + * The primitives the Codex write lock's authority comparison rests on. + * + * These are deliberately function-level. The refusal they enable — + * `authority_not_proven` on a competing write — cannot be proven here, because + * it needs an admitted snapshot, and admission cannot honestly admit anything + * while its ownership field is still hardcoded. Proving the refusal against a + * placeholder would be proving it against a known lie, so that claim belongs to + * the phases that make ownership real. + */ +import { afterEach, beforeEach, describe, expect, test } from "bun:test"; +import { chmodSync, mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { basename, dirname, join } from "node:path"; + +import { + observeConfigGeneration, + readConfigAdmissionSnapshot, + readConfigGenerationInCurrentMutationTransaction, + saveConfig, + withConfigMutationLockSync, +} from "../src/config"; +import { hashAuthority } from "../src/codex/admission"; +import { JOURNAL_PATH } from "../src/codex/journal"; +import type { AdmissionSnapshot } from "../src/codex/convergence-types"; +import type { OcxConfig } from "../src/types"; + +let root = ""; +let previousOpencodexHome: string | undefined; +const cleanup: string[] = []; + +function config(port = 10100): OcxConfig { + return { + port, + providers: { + openai: { + adapter: "openai-responses", + baseUrl: "https://chatgpt.com/backend-api/codex", + authMode: "forward", + }, + } as OcxConfig["providers"], + defaultProvider: "openai", + }; +} + +beforeEach(() => { + root = mkdtempSync(join(tmpdir(), "ocx-admission-primitives-")); + cleanup.push(root); + previousOpencodexHome = process.env.OPENCODEX_HOME; + process.env.OPENCODEX_HOME = root; +}); + +afterEach(() => { + if (previousOpencodexHome === undefined) delete process.env.OPENCODEX_HOME; + else process.env.OPENCODEX_HOME = previousOpencodexHome; + while (cleanup.length) rmSync(cleanup.pop()!, { recursive: true, force: true }); +}); + +describe("the config digest is over bytes, not over meaning", () => { + /** + * The defect this closes: the digest used to hash `JSON.stringify(config)`, + * so two files that parse the same hashed the same. A non-cooperating writer + * could reformat the file between admission and commit and the comparison + * would see nothing at all. + */ + test("a whitespace-only rewrite moves the digest", () => { + const path = join(root, "config.json"); + writeFileSync(path, JSON.stringify(config(), null, 2)); + const compact = readConfigAdmissionSnapshot(); + + writeFileSync(path, JSON.stringify(config(), null, 8)); + const spacious = readConfigAdmissionSnapshot(); + + expect(compact.kind).toBe("read"); + expect(spacious.kind).toBe("read"); + // Same meaning... + expect(spacious.diagnostics.config).toEqual(compact.diagnostics.config); + // ...different bytes. + expect(spacious.contentSha256).not.toBe(compact.contentSha256); + }); + + test("and moves the authority hash with it", () => { + const path = join(root, "config.json"); + writeFileSync(path, JSON.stringify(config(), null, 2)); + const before = readConfigAdmissionSnapshot(); + writeFileSync(path, JSON.stringify(config(), null, 8)); + const after = readConfigAdmissionSnapshot(); + + expect(hashAuthority(snapshotWith({ configDigest: after.contentSha256 ?? "" }))) + .not.toBe(hashAuthority(snapshotWith({ configDigest: before.contentSha256 ?? "" }))); + }); + + test("identical bytes hash identically, or every write would refuse itself", () => { + const path = join(root, "config.json"); + const bytes = JSON.stringify(config(), null, 2); + writeFileSync(path, bytes); + const first = readConfigAdmissionSnapshot(); + writeFileSync(path, bytes); + const second = readConfigAdmissionSnapshot(); + expect(second.contentSha256).toBe(first.contentSha256); + }); + + /** + * Hashing the file and then reading it again to parse would leave a window in + * which the two disagree — turning the check into a second chance to be wrong. + * The read count is the property; the digest being correct is not enough. + */ + /** + * A spy cannot see through the module boundary here, so the read count is + * proven by CONSEQUENCE instead: if the producer read the file twice, a + * change landing between the two reads would make the digest and the parsed + * config describe different files. Swapping the contents underneath a single + * read is impossible; underneath two reads it is the whole hazard. + */ + test("the digest and the parsed config always describe the same bytes", () => { + const path = join(root, "config.json"); + for (const port of [10100, 20200, 30300]) { + const bytes = JSON.stringify(config(port), null, 2); + writeFileSync(path, bytes); + const snapshot = readConfigAdmissionSnapshot(); + expect(snapshot.kind).toBe("read"); + expect(snapshot.diagnostics.config.port).toBe(port); + // The digest of what we just wrote, computed independently. + const independent = new Bun.CryptoHasher("sha256").update(bytes).digest("hex"); + expect(snapshot.contentSha256).toBe(independent); + } + }); + + test("an unreadable config cannot carry a digest", () => { + // No config.json at all. + const snapshot = readConfigAdmissionSnapshot(); + expect(snapshot.kind).toBe("unreadable"); + expect(snapshot.contentSha256).toBeNull(); + expect(snapshot.diagnostics.source).toBe("default"); + }); +}); + +describe("absence is one state, and being unable to look is another", () => { + test("a missing coordinator database is absent", () => { + expect(observeConfigGeneration()).toEqual({ kind: "absent" }); + }); + + test("a corrupt database is not absent", () => { + writeFileSync(join(root, "config-mutation.sqlite"), "not sqlite"); + expect(observeConfigGeneration()).toEqual({ kind: "unavailable", reason: "database" }); + }); + + test("a directory where the database should be is not absent", () => { + mkdirSync(join(root, "config-mutation.sqlite")); + const observed = observeConfigGeneration(); + expect(observed.kind).toBe("unavailable"); + }); + + test("an unreadable parent directory is not absent", () => { + const nested = join(root, "locked"); + mkdirSync(nested); + writeFileSync(join(nested, "config-mutation.sqlite"), ""); + chmodSync(nested, 0o000); + process.env.OPENCODEX_HOME = nested; + try { + const observed = observeConfigGeneration(); + // Either it could not stat (unavailable) or the platform let it through; + // what must never happen is reporting ABSENT for a file that is there. + expect(observed.kind).not.toBe("absent"); + } finally { + chmodSync(nested, 0o700); + process.env.OPENCODEX_HOME = root; + } + }); + + test("an existing database reports its value", () => { + saveConfig(config()); + expect(observeConfigGeneration()).toEqual({ kind: "ready", generation: { value: 1 } }); + }); +}); + +describe("the generation read that belongs to the open transaction", () => { + /** + * The measured fact that made this necessary: on first acquisition the + * BEGIN IMMEDIATE creating the table has not committed, so a separate + * read-only connection still sees nothing. The observer cannot do this job. + */ + test("the observer cannot read the first transaction, and this can", () => { + const before = observeConfigGeneration(); + let observedInside: { kind: string } | undefined; + let transactional: unknown; + withConfigMutationLockSync(() => { + observedInside = observeConfigGeneration(); + transactional = readConfigGenerationInCurrentMutationTransaction(); + return null; + }); + const after = observeConfigGeneration(); + + // Before: nothing exists. Inside: the file exists but its creating + // transaction has not committed, so a separate connection still cannot read + // a generation — it reports unavailable, NOT the zero that is really there. + // Only after the commit does the observer agree. + expect(before).toEqual({ kind: "absent" }); + expect(observedInside?.kind).not.toBe("ready"); + expect(after).toEqual({ kind: "ready", generation: { value: 0 } }); + // The transactional read saw the truth the whole time. + expect(transactional).toEqual({ value: 0 }); + }); + + test("calling it outside a transaction throws rather than guessing", () => { + expect(() => readConfigGenerationInCurrentMutationTransaction()).toThrow( + /requires an open config mutation transaction/, + ); + }); + + test("a nested call reads the same open handle", () => { + const seen: unknown[] = []; + withConfigMutationLockSync(() => { + seen.push(readConfigGenerationInCurrentMutationTransaction()); + withConfigMutationLockSync(() => { + seen.push(readConfigGenerationInCurrentMutationTransaction()); + return null; + }); + return null; + }); + expect(seen).toEqual([{ value: 0 }, { value: 0 }]); + }); + + test("it throws again once the transaction has closed", () => { + withConfigMutationLockSync(() => readConfigGenerationInCurrentMutationTransaction()); + expect(() => readConfigGenerationInCurrentMutationTransaction()).toThrow(); + }); +}); + +describe("absent and present-zero are one authority", () => { + test("they hash identically, or no first write could ever commit", () => { + expect(hashAuthority(snapshotWith({ generation: { present: false, value: 0 } }))) + .toBe(hashAuthority(snapshotWith({ generation: { present: true, value: 0 } }))); + }); + + test("but a committed bump does not hash like either", () => { + const zero = hashAuthority(snapshotWith({ generation: { present: true, value: 0 } })); + expect(hashAuthority(snapshotWith({ generation: { present: true, value: 1 } }))).not.toBe(zero); + expect(hashAuthority(snapshotWith({ generation: { present: true, value: 2 } }))).not.toBe(zero); + expect(hashAuthority(snapshotWith({ generation: { present: true, value: 2 } }))) + .not.toBe(hashAuthority(snapshotWith({ generation: { present: true, value: 1 } }))); + }); + + /** + * Without this guard every malformed value collapses to "gen:0" — the single + * value that means "nothing has happened yet, go ahead". + */ + test.each([ + ["negative", -1], + ["NaN", Number.NaN], + ["infinity", Number.POSITIVE_INFINITY], + ["fractional", 1.5], + ["beyond safe integers", Number.MAX_SAFE_INTEGER + 1], + ])("a %s generation throws instead of becoming zero", (_label, value) => { + expect(() => hashAuthority(snapshotWith({ generation: { present: true, value } }))) + .toThrow(/non-negative safe integer/); + }); + + test("every other authority field still moves the hash", () => { + const base = snapshotWith({}); + const variants: Partial[] = [ + { configDigest: "different" }, + { intent: "off" }, + { ownership: "foreign" }, + { externalProvider: "someone" }, + { journalIdentity: "different" }, + { provenanceIdentity: "different" }, + { canonicalTargets: { ...base.canonicalTargets, config: "/elsewhere" } }, + ]; + for (const variant of variants) { + expect(hashAuthority({ ...base, ...variant })).not.toBe(hashAuthority(base)); + } + }); +}); + +describe("the journal has one owner", () => { + /** + * Admission re-derived this path by hand and got both halves wrong, so its + * journal identity watched a file nothing writes. The fixture re-derived it + * the same wrong way and agreed. Importing the production constant is the + * point of the test. + */ + test("the exported constant is the file journal.ts actually uses", () => { + // The old hand-derived path was OPENCODEX_HOME/codex-journal.json — wrong + // directory AND wrong basename. Assert both halves. Paths are compared by + // shape rather than string equality because macOS resolves the temp root + // through /private, which is not the property under test. + expect(basename(JOURNAL_PATH)).toBe("opencodex-journal.json"); + expect(dirname(JOURNAL_PATH).endsWith(".codex")).toBeTrue(); + expect(dirname(JOURNAL_PATH).endsWith(".opencodex")).toBeFalse(); + }); +}); + +function snapshotWith(overrides: Partial): AdmissionSnapshot { + return { + config: config(), + configDigest: "digest", + intent: "on", + generation: { present: true, value: 0 }, + ownership: "owned", + externalProvider: null, + canonicalTargets: { + codexHome: "/codex", + opencodexHome: "/opencodex", + config: "/codex/config.toml", + profile: "/codex/opencodex.config.toml", + catalog: "/codex/opencodex-catalog.json", + cache: "/codex/models_cache.json", + journal: JOURNAL_PATH, + integrationRecord: "/opencodex/integrations/codex.json", + catalogBackups: [], + historyDb: "/codex/state_5.sqlite", + historyManifest: "/codex/state_5.sqlite.ocx-backup.json", + historyRollouts: [], + }, + journalIdentity: "absent", + provenanceIdentity: "absent", + authoritySnapshotId: "", + ...overrides, + }; +} From b41900e0f0d7c451d1b749f40352b26419596d08 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Wed, 5 Aug 2026 13:05:08 +0900 Subject: [PATCH 132/163] fix(codex): the journal fix that was only a comment The previous commit added "the journal's own constant" as a comment directly above a hand-derived path, and left the derivation in place. Both the target and the identity still pointed at OPENCODEX_HOME/codex-journal.json while the journal is CODEX_HOME/opencodex-journal.json. The suite stayed green because it asserted the CONSTANT was correct, never that admission used it. So the test agreed with the fix, the fix agreed with its own comment, and the code did the old thing. Adds the assertion that was missing: admission's source must mention JOURNAL_PATH and must not contain the old basename at all. Reading the producer's text is blunt for a unit test, but the defect being guarded is precisely "someone writes the path out by hand again". --- src/codex/admission.ts | 4 ++-- tests/codex-admission-primitives.test.ts | 17 +++++++++++++++++ 2 files changed, 19 insertions(+), 2 deletions(-) diff --git a/src/codex/admission.ts b/src/codex/admission.ts index 41f468f1a..940f5be61 100644 --- a/src/codex/admission.ts +++ b/src/codex/admission.ts @@ -148,7 +148,7 @@ export function admitCodexWrite(): CodexAdmission { cache: CODEX_MODELS_CACHE_PATH, // The journal's own constant. Re-deriving it here is what made this field // watch a path nothing writes. - journal: join(opencodexHome, "codex-journal.json"), + journal: JOURNAL_PATH, integrationRecord, // Backups and rollouts are enumerated by their owners, not guessed here. catalogBackups: [] as readonly string[], @@ -172,7 +172,7 @@ export function admitCodexWrite(): CodexAdmission { ownership: "owned", externalProvider: null, canonicalTargets, - journalIdentity: contentIdentity(join(opencodexHome, "codex-journal.json")), + journalIdentity: contentIdentity(JOURNAL_PATH), provenanceIdentity: contentIdentity(integrationRecord), authoritySnapshotId: "", }; diff --git a/tests/codex-admission-primitives.test.ts b/tests/codex-admission-primitives.test.ts index eeb70694c..d0c296c14 100644 --- a/tests/codex-admission-primitives.test.ts +++ b/tests/codex-admission-primitives.test.ts @@ -289,6 +289,23 @@ describe("the journal has one owner", () => { expect(dirname(JOURNAL_PATH).endsWith(".codex")).toBeTrue(); expect(dirname(JOURNAL_PATH).endsWith(".opencodex")).toBeFalse(); }); + + /** + * The constant existing is not the property. ADMISSION USING IT is. + * + * The first version of this suite asserted only the constant, and a commit + * landed in which the comment said "the journal's own constant" directly + * above a hand-derived path — the exact defect, restated as its own fix, + * with the tests still green. Reading the producer's source is blunt, but it + * is the thing that was actually wrong. + */ + test("admission derives no journal path of its own", async () => { + const source = await Bun.file( + join(import.meta.dir, "..", "src", "codex", "admission.ts"), + ).text(); + expect(source).toContain("JOURNAL_PATH"); + expect(source).not.toContain("codex-journal.json"); + }); }); function snapshotWith(overrides: Partial): AdmissionSnapshot { From 8d1f2361e7f6aa14c984f5b13ef77f2110d2297d Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Wed, 5 Aug 2026 13:11:13 +0900 Subject: [PATCH 133/163] fix(codex): absence keeps the retryable projection it always had The full suite caught what the focused tests could not. management-convergence classifies an admission throw by matching its MESSAGE, so introducing a new sentence for the absent coordinator turned a retryable skip into a non-retryable failed/disk - /api/sync would have told a user their catalog refresh had failed on disk when the only thing wrong was a database that had never been created. Coupling a control-flow decision to prose is the underlying defect and it is older than this change. Rather than leave it discoverable only by running everything, the wording is pinned by a test in the phase that depends on it, with the reason written next to it. Also updates the generation observation test that pinned absence to unavailable/database. Its stated concern - that a look-only caller must not be handed something it could mistake for a known-good zero - is unchanged and still honored, since absent authorizes nothing until a real zero is read inside the config transaction. --- src/codex/catalog-admission.ts | 9 ++++++++- tests/codex-admission-primitives.test.ts | 24 ++++++++++++++++++++++++ tests/codex-config-generation.test.ts | 14 +++++++++----- 3 files changed, 41 insertions(+), 6 deletions(-) diff --git a/src/codex/catalog-admission.ts b/src/codex/catalog-admission.ts index 7c1378090..df27a53b3 100644 --- a/src/codex/catalog-admission.ts +++ b/src/codex/catalog-admission.ts @@ -147,8 +147,15 @@ export function captureCatalogAdmissionSnapshot( * gather has no such transaction — by contract it holds no lock and writes * nothing — so it has no way to turn absence into an observation. Refusing * is the only honest answer available to it. + * + * The wording matters beyond this line: `admissionFailure` in + * management-convergence.ts classifies this throw by matching its MESSAGE, + * and an unrecognized one becomes a non-retryable `failed/disk` instead of + * the retryable skip a missing coordinator has always produced. Keeping the + * "config generation is database" phrasing preserves that projection — a + * caller that could simply try again should still be told to. */ - throw new Error("Cannot capture Codex catalog admission: no config generation exists to admit against."); + throw new Error("Cannot capture Codex catalog admission: config generation is database (absent)."); } if (generation.kind !== "ready") { throw new Error(`Cannot capture Codex catalog admission: config generation is ${generation.reason}.`); diff --git a/tests/codex-admission-primitives.test.ts b/tests/codex-admission-primitives.test.ts index d0c296c14..3e643efd2 100644 --- a/tests/codex-admission-primitives.test.ts +++ b/tests/codex-admission-primitives.test.ts @@ -21,6 +21,7 @@ import { withConfigMutationLockSync, } from "../src/config"; import { hashAuthority } from "../src/codex/admission"; +import { captureCatalogAdmissionSnapshot } from "../src/codex/catalog-admission"; import { JOURNAL_PATH } from "../src/codex/journal"; import type { AdmissionSnapshot } from "../src/codex/convergence-types"; import type { OcxConfig } from "../src/types"; @@ -273,6 +274,29 @@ describe("absent and present-zero are one authority", () => { }); }); +describe("absence keeps the projection it always had", () => { + /* + * management-convergence.ts classifies this failure by MATCHING THE MESSAGE + * (`admissionFailure`, ~:62). A missing coordinator has always produced a + * retryable skip; a message the classifier does not recognize silently + * becomes a non-retryable failed/disk. Adding the `absent` branch broke + * exactly that, and only a full-suite run caught it — this is the local guard. + */ + test("catalog admission still reads as retryable when the coordinator is absent", () => { + let thrown: unknown; + try { + captureCatalogAdmissionSnapshot(config()); + } catch (error) { + thrown = error; + } + expect(thrown).toBeInstanceOf(Error); + const message = (thrown as Error).message; + expect( + message.includes("config generation is busy") || message.includes("config generation is database"), + ).toBeTrue(); + }); +}); + describe("the journal has one owner", () => { /** * Admission re-derived this path by hand and got both halves wrong, so its diff --git a/tests/codex-config-generation.test.ts b/tests/codex-config-generation.test.ts index 52cbb6d55..a2a085b0d 100644 --- a/tests/codex-config-generation.test.ts +++ b/tests/codex-config-generation.test.ts @@ -110,10 +110,14 @@ test("observe-only generation reports a missing database without creating or chm expect(existsSync(join(absentHome, "config-mutation.sqlite"))).toBeFalse(); expect(existsSync(absentHome)).toBeFalse(); expect(existsSync(absentParent)).toBeFalse(); - // Missing storage reports the ordinary typed unavailable rather than a - // distinct `absent`: a caller that may only observe must not be handed - // something it could mistake for a known-good baseline of zero. - expect(absentObservation).toEqual({ kind: "unavailable", reason: "database" }); + // Missing storage is now its own state. The original concern behind folding it + // into `unavailable` — that a look-only caller must not receive something it + // could mistake for a known-good zero — is unchanged and still honored: + // `absent` authorizes nothing by itself. Only a caller holding the config + // transaction may promote it, and only by reading a real zero in there. + // Refusing outright was not neutral either: it meant refusing every Codex + // write on any home whose config predates this database. + expect(absentObservation).toEqual({ kind: "absent" }); const rootAfter = statSync(testRoot, { bigint: true }); expect(rootAfter.mode).toBe(rootBefore.mode); @@ -127,7 +131,7 @@ test("observe-only generation reports a missing database without creating or chm const existingObservation = observeConfigGeneration(); expect(existsSync(join(existingHome, "config-mutation.sqlite"))).toBeFalse(); expect(statSync(existingHome).mode & 0o777).toBe(existingMode); - expect(existingObservation).toEqual({ kind: "unavailable", reason: "database" }); + expect(existingObservation).toEqual({ kind: "absent" }); }); test("observe-only generation reads an existing value without modifying its database", () => { From e0459e48fd902b0885727641fcd21289801b8b9a Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Wed, 5 Aug 2026 14:42:59 +0900 Subject: [PATCH 134/163] docs(codex): the lock span is the IO, not the absence of it A second audit round caught a sentence claiming nothing inside the span may be non-deterministic or IO-bearing. The span IS the IO - a journal write, two atomic file replacements, and the marking that follows them. The real constraint is narrower: nothing there may perform IO the journal does not account for, because restore replays only what the journal recorded. Also corrects the compatibility claim about PR #1022, now landed. Its two changed call sites sit INSIDE the span, not outside as originally written. The ordering decision survives with a different reason: land it first so the lock is written against the function's final shape. Line citations refreshed against dev at 468587632, with a note that the anchors are the call names rather than the numbers. --- .../041_wp12_closeout.md | 63 ++++++++++++++----- 1 file changed, 49 insertions(+), 14 deletions(-) diff --git a/devlog/_plan/260804_codex_write_substrate/041_wp12_closeout.md b/devlog/_plan/260804_codex_write_substrate/041_wp12_closeout.md index 526a90329..6f2aaf6b6 100644 --- a/devlog/_plan/260804_codex_write_substrate/041_wp12_closeout.md +++ b/devlog/_plan/260804_codex_write_substrate/041_wp12_closeout.md @@ -178,20 +178,55 @@ fail-open behavior and its callers. is defect #10 of this unit and the reason WP11 was folded into WP12. A mechanism with no consumer cannot be exercised except through a fabricated object. -`injectCodexConfig` (`src/codex/inject.ts:487`) is the edge. The naive reading — -"wrap `:601-603`" — is wrong, and the audit caught it: **`writeJournal()` already -runs at `:530`**, well before that block, and it performs an atomic write -(`src/codex/journal.ts:60-82`). Wrapping only the tail would leave the first -artifact-creating write outside the lock, which is not exclusivity; it is a -shorter unprotected window. - -The lock therefore opens **before `writeJournal`** and closes after -`markJournalInjectedState`, covering the journal write, both `atomicWriteFile` -calls, and the injected-state marking as one section. Everything before that -point in the function is classification and refusal, which creates nothing. The -awaited history job at `:614` stays **outside**: it has its own cross-process -lock (WP10) and the `N -> H` order is deliberate. Production reaches this -function from `src/codex/sync.ts:58,110` and `src/cli/init.ts:197`. +`injectCodexConfig` (`src/codex/inject.ts:491`) is the edge. The naive reading — +"wrap the two `atomicWriteFile` calls" — is wrong, and the audit caught it: +**`writeJournal()` already runs at `:534`**, seventy lines before them, and it +performs an atomic write (`src/codex/journal.ts:69-90`). Wrapping only the tail +would leave the first artifact-creating write outside the lock, which is not +exclusivity; it is a shorter unprotected window. + +Line numbers here track `origin/dev` at `468587632`, after #1022 and #1000 +landed. They move; the anchors — `writeJournal`, the two `atomicWriteFile` +calls, `markJournalInjectedState`, and `runCodexHistoryJob` — do not. + +The lock therefore opens **before `writeJournal`** (`:534`) and closes after +`markJournalInjectedState` (`:607`), covering the journal write, both +`atomicWriteFile` calls (`:605-606`), and the injected-state marking as one +section. Everything before that point in the function is classification and +refusal, which creates nothing. The awaited history job stays **outside**: it has +its own cross-process lock (WP10) and the `N -> H` order is deliberate. +Production reaches this function from `src/codex/sync.ts:58,110` and +`src/cli/init.ts:197`. + +### What else lives inside that span + +The span is wider than the three writes, and an audit caught me claiming +otherwise. PR #1022 (tri-state `fastMode`, now landed as `ebcfff44f`) changes two +call sites — `ensureFastModeFeature` at `:555` and `buildProfileFile` at `:603` — +and I argued they sat outside the lock because the first is "before the writes". +They are not: `writeJournal` opens the span at `:534` and both changed lines fall +after it. + +The order still holds, with a different reason. Landing #1022 first is right not +because it avoids the section but because WP-R1c should be written against the +function's final shape rather than against a version about to change underneath +it. Both transforms are pure and bounded, so they compose inside the callback; +what would have been wrong is discovering that during implementation instead of +before it. + +A second reviewer caught the sentence that stood here, which claimed nothing in +the span may be non-deterministic or IO-bearing. That is plainly false: the span +*is* the IO — journal writes, two atomic file replacements, and the marking that +follows them. The real constraint is narrower: nothing in the span may perform IO +the journal does not account for, because the restore path replays only what the +journal recorded. + +`buildProfileFile` with an unset `fastMode` now emits different bytes, and that +stays safe because the journal captures the original profile before the write +(`src/codex/journal.ts:69-90`) and records the exact new one after +(`:93-107`); restore replays the captured original (`:128-141`). The comparison +is against what was actually written, not against what the generator would +produce today. ### The external-provider branch, and the fix that could not work From 872ea9ec29870bf8a912a6c82c9b3829717a2a78 Mon Sep 17 00:00:00 2001 From: bitkyc08-arch Date: Wed, 5 Aug 2026 14:54:19 +0900 Subject: [PATCH 135/163] test(codex): the admission fixtures still described the old contract Two failures left in this file, and neither was about ownership - both were the fixture lagging behind fixes that already landed. One wrote OPENCODEX_HOME/codex-journal.json by hand. That is the path the producer used to derive, also by hand, also wrong; they agreed with each other and disagreed with production, which is why the pair stayed green for so long. It now imports JOURNAL_PATH. The other did `generation: base.generation + 1` on what is now a {present,value} pair, producing NaN - which the authority hash correctly refuses rather than folding into the token that means "nothing has happened yet". Also records what a reviewer established: these two were never blocked on tri-state ownership, so the phase that makes ownership real no longer inherits them. --- .../041_wp12_closeout.md | 198 ++++++++++++++++- tests/codex-admission.test.ts | 199 ++++++++++++++++++ 2 files changed, 389 insertions(+), 8 deletions(-) create mode 100644 tests/codex-admission.test.ts diff --git a/devlog/_plan/260804_codex_write_substrate/041_wp12_closeout.md b/devlog/_plan/260804_codex_write_substrate/041_wp12_closeout.md index 6f2aaf6b6..0139fe405 100644 --- a/devlog/_plan/260804_codex_write_substrate/041_wp12_closeout.md +++ b/devlog/_plan/260804_codex_write_substrate/041_wp12_closeout.md @@ -327,13 +327,20 @@ result: | Evidence | Ownership | |---|---| -| Every known state path is ENOENT, **and the service manager shows no installation** | `owned` — an uncontested home | -| Every known state path is ENOENT, but the service manager shows an installation, a conflict, or cannot be read | `unknown` | -| Readable, valid, both homes match | `owned` | +| Every known state path is ENOENT, **and no manager definition exists on disk, and no registration is loaded** | `owned` — an uncontested home | +| Every known state path is ENOENT, but a manager definition exists, a registration is loaded, or either cannot be asked | `unknown` | +| Readable, valid, both homes match, **and any manager definition names the same homes** | `owned` | | Readable, valid, homes differ | `foreign` | +| Readable and valid, but a manager definition names DIFFERENT homes | `unknown` — an interrupted reinstall, not a decision to make unattended | | Present but unreadable, malformed, or schema-invalid | `unknown` | | Two valid states that disagree | `unknown` | | A valid state beside an unreadable one | `unknown` | +| Two managers both proven present (Windows scheduler + WinSW) | `unknown` (`conflict` at the probe) | + +`readServiceInstallState` cannot answer this: it returns the FIRST valid state +and discards every later path (`src/service.ts:165-175`), so a valid mirror +beside a corrupt one reads as clean. The projection needs an all-paths evidence +API that reports what each path said, not the first thing that parsed. Known paths and the current home pair come from `src/service.ts:82-107`, and normalization from `:109-112`. The detailed inspection belongs in `service.ts`; @@ -368,11 +375,186 @@ export type ServiceManagerInstallation = export function inspectServiceManagerInstallation(): ServiceManagerInstallation; ``` -On Linux, `systemctl --user show -p LoadState --value opencodex-proxy` gives the -three-way answer directly: a known load state is present, an explicit -`not-found` is absent, and a bus or parse failure is unknown. Every probe is -read-only — the user's proxy is live, and nothing here may start, stop, or -reload it. +### Registration is not the question; the definition is + +A fifth round rejected that sketch, and the reason reframes the whole probe. +**Asking whether a job is currently loaded answers the wrong question.** + +Installation writes the definition FIRST and the state file after +(`src/service.ts:1610-1625` on macOS, `:2017-2021` on Linux), and the definition +itself embeds `CODEX_HOME` and `OPENCODEX_HOME` (`:276-284`, `:1948`). So an +interrupted reinstall leaves a valid state file for home A beside an installed +plist for home B — and a probe that only asks "is a job loaded?" calls that +`owned`. Worse on macOS: a logged-out user has the plist on disk with no GUI +domain at all, so the registration probe reports nothing loaded while a foreign +definition sits right there. + +The probe therefore reads the **definition**, and compares the homes inside it: + +| Platform | Definition | Registration | +|---|---|---| +| macOS | `~/Library/LaunchAgents/com.opencodex.proxy.plist` (`:56-59`) | `launchctl print` | +| Linux | `~/.config/systemd/user/opencodex-proxy.service` (`:1935-1941`) | `systemctl --user show` | +| Windows | Task Scheduler XML / WinSW config | `schtasks /query`, `sc query` | + +Absence requires BOTH: no definition on disk AND no registration. Either one +present, or either one unaskable, is not absence. + +### Measured exit codes, and the one that is not a code at all + +macOS distinguishes the two cases, verified against nonexistent targets rather +than assumed: + +```text +launchctl print gui// -> exit 113 "Could not find service ... in domain" +launchctl print gui/999999/