From 0e5aa001eb8691448156572c5bee5a914b904512 Mon Sep 17 00:00:00 2001 From: mintaka Date: Fri, 4 Sep 2026 15:43:32 -0400 Subject: [PATCH 1/9] =?UTF-8?q?docs(platform):=20design=20self-host=20stac?= =?UTF-8?q?k=20supervision=20=E2=80=94=20keep=20pgid,=20decline=20Quadlet?= =?UTF-8?q?=20(RIG-3239)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Decides how the long-lived self-host stack services (compass-server, compass-runner, containerized postgres, OTel collector) are supervised. Ruling (Matt fork — OQ-1): KEEP the hand-rolled DL-183/DL-262 pgid mechanism as the single built-in supervision model; do NOT adopt per-service Podman Quadlet units. Quadlet is Linux/systemd-only, so the pgid path survives for the embedded + non-systemd tiers regardless — adopting Quadlet means two behaviorally-equivalent supervision models, the imperative cold sequence would need oneshot pre-units + sdnotify re-plumbing, and per-service units displace the DL-259-named `compass-stack up` verb. Optionally ship a documented thin systemd USER-unit wrapper (Type=oneshot + RemainAfterExit=yes) for boot-start/logout survival — systemd wrapping the supervisor, never replacing it. Docker-socket declined (daemon model vs the rootless/no-daemon invariant). The record was red-teamed by a design-critic pass before submission; its findings are folded: the "one model everywhere including macOS" claim corrected to "one portable model with a named unbuilt darwin start-time-seam port" (pgidfile.go's /proc reader is Linux-only, `up` refuses on darwin today); the crux argued on maintenance cost not slice size (DL-259 pins the whole self-host tier to Linux); the oneshot wrapper's latent failure modes (status-lie under RemainAfterExit, stop-what-you-didn't-start, PATH/linger/After) folded into OQ-4 + the T1 unit checklist; and the crash-recovery fork reshaped into OQ-3 as three options — accept the gap (v1), a blocking `compass-stack up --supervise` under Type=exec/Restart=on-failure (systemd whole-stack recovery with one model), or reopen Quadlet. Three load-bearing Matt forks ride to the design gate: OQ-1 (adopt Quadlet vs keep), OQ-2 (ship the T1 wrapper at all), OQ-3 (crash-recovery posture). Ledger: adds DL-328 (Infrastructure & CI section). Status stays Draft — freezes Active on merge. Refs RIG-3239 Co-authored-by: Matt Wilkinson --- docs/designs/DECISIONS.md | 1 + .../compass-stack-supervision/design.md | 496 ++++++++++++++++++ 2 files changed, 497 insertions(+) create mode 100644 docs/designs/platform/compass-stack-supervision/design.md diff --git a/docs/designs/DECISIONS.md b/docs/designs/DECISIONS.md index 889af981..fbab6533 100644 --- a/docs/designs/DECISIONS.md +++ b/docs/designs/DECISIONS.md @@ -427,6 +427,7 @@ check enforces the mechanical half. Full rationale: | DL-299 | The `:vX.Y.Z` agent image is minted by digest re-tag (`skopeo copy`) of the newest already-published per-push `:git-` artifact at-or-before the release sha (resolved by walking first-parent ancestors for a GHCR-resolvable tag, never a path-based `git log`) — never a second ~90m closure build; `:latest` stays owned exclusively by the per-push publish job; the mint shares the `publish-agent-image` concurrency group under `queue: max` + `cancel-in-progress: false` so a non-superseding release mint is serialized, never dropped | Active (Matt, 2026-08-25) | [unified release lane](infra/release/compass-unified-release-lane/design.md) | | DL-300 | `publish-agent-image.yml` retires as a FILE with its DUTY relocated into the unified lane's per-push `publish-image` job (verbatim `publish.sh`, same closure-paths gate, same serialize/immutability/coherence/least-privilege/off-hot-path posture) via two-PR staging (land the fold, observe one green per-push publish, then delete) — accepted by the agent-image zone owner (compass-managed), who amend their record with a superseded-by pointer | Active (Matt, 2026-08-25) | [unified release lane](infra/release/compass-unified-release-lane/design.md) | | DL-301 | release-please authenticates with a scoped GitHub App installation token (contents + pull-requests, installation-scoped, per-run) so the release-please-opened Release PR receives `pull_request` CI and branch protection is a real merge gate — a bare `GITHUB_TOKEN`-opened PR gets none under GitHub's recursion guard | Active (Matt, 2026-08-25) | [unified release lane](infra/release/compass-unified-release-lane/design.md) | +| DL-328 | Self-host stack supervision KEEPS the hand-rolled DL-183/DL-262 pgid mechanism as the SINGLE built-in supervision model; per-service Podman Quadlet units are NOT adopted (Linux/systemd-only, so the pgid path survives regardless — two behaviorally-equivalent models vs one; the imperative cold sequence would need oneshot pre-units + sdnotify re-plumbing; per-service units displace the DL-259-named `compass-stack up` verb). A DOCUMENTED thin systemd USER-unit wrapper (`Type=oneshot`+`RemainAfterExit=yes`, ExecStart=`compass-stack up`/ExecStop=`compass-stack down`) MAY ship in the self-host doc for boot-start/logout survival — systemd wrapping the supervisor, never replacing it. Docker-socket declined at the stack layer (daemon model vs the rootless/no-daemon hard requirement; no per-container keep-id equivalent). CRASH RECOVERY is an accepted v1 gap with a named follow-up (blocking `compass-stack up --supervise` under a `Type=exec`/`Restart=on-failure` unit — systemd whole-stack restart with one model), left to Matt (OQ-3); adopt-Quadlet-vs-keep is OQ-1 | Active (Matt, 2026-09-04) | [stack supervision §Approach](platform/compass-stack-supervision/design.md#approach) | ## Test harness diff --git a/docs/designs/platform/compass-stack-supervision/design.md b/docs/designs/platform/compass-stack-supervision/design.md new file mode 100644 index 00000000..28d8e56b --- /dev/null +++ b/docs/designs/platform/compass-stack-supervision/design.md @@ -0,0 +1,496 @@ +# Compass self-host stack supervision: Quadlet vs hand-rolled pgid + +Status: Draft +Issue: RIG-3239 + +## Problem / Intent + +Decide the supervision mechanism for the long-lived self-host stack services +(compass-server, compass-runner, the containerized postgres, the bundled OTel +collector): adopt Podman **Quadlet** (systemd-native declarative unit files) or +keep the hand-rolled **DL-183/DL-262 pgid** mechanism `compass-stack` ships +today, with the Docker-socket engine recorded as considered-and-declined at the +stack layer. Scope is bring-up order, teardown, restart policy, crash recovery, +and rootless posture of the stack services ONLY — the per-session runner +backend is frozen out of scope (RIG-3070: podman permanent for self-host, +microVM behind the seam; see +`ui/compass-native-embedded-revival/design.md:71-74`), as is the macOS embedded +runner backend (sibling RIG-3238 Apple-container record). + +## Approach + +**Recommended ruling (Matt fork — see Open Questions OQ-1): keep the +hand-rolled DL-183/DL-262 pgid supervision as the SINGLE built-in mechanism; +do not adopt per-service Quadlet units. Optionally ship a documented THIN +systemd user-unit wrapper around the existing `compass-stack up`/`down` for +self-hosters who want boot-start and logout survival — systemd wrapping the +supervisor, never replacing it.** Docker-socket is declined at the stack layer, +mirroring its per-session-runner rejection. + +### What exists today (grounded) + +The supervisor is `compass-stack` (`go/cmd/compass-stack/main.go:8-9`): + +> ```go +> // - up: bring the embedded stack to Ready (or attach to a live one) and +> // return once ready; the children keep running (up does NOT block). +> ``` + +The bring-up is an ordered, partly IMPERATIVE chain +(`go/internal/stack/stack.go:89-93`): + +> ```go +> // Cold sequence (devenv.nix:122-143): private postgres up+reachable → TLS anchor +> // (expiry-aware) → compass-server → poll GetServerInfo readiness → runner token +> // (idempotent 0600) → agent image present → compass-runner (token via env). On +> // any step failure the children started so far are drained and the lock +> // released, so no half-started stack leaks. +> ``` + +Cross-process teardown is the DL-183 pgid record +(`go/internal/stack/pgidfile.go:14-18`): + +> ```go +> // pgidFileName is the state-dir record of the child process groups a successful +> // up spawned, beside stack.lock / stack.lock.guard. A fresh down (which holds no +> // in-memory Process handle for a stack a prior up spawned) reads it to learn +> // which groups to signal. It is removed on a fully successful teardown. +> ``` + +with the start-time identity token closing the pid-recycle window +(`pgidfile.go:71-74`): + +> ```go +> // the process-group id (== the child's pid, set via Setpgid at spawn) + +> // the group leader's start time as read at spawn. StartTime is the identity +> // token — it turns the down-side check from "does a group with this pgid +> // exist" (which a recycled pid passes falsely) into "does a group with this +> ``` + +DL-262 extended the record to a v2 kind-tagged discriminated union for the +containerized children (`pgidfile.go:23-26`): + +> ```go +> // v2 (this build) grows the entry line into a kind-tagged discriminated union +> // (proc / ctr, see pgidEntry): the container-backed postgres of S4 has no +> // process-group teardown identity, so it is recorded and torn down by container +> // name instead. +> ``` + +and the down side (`go/internal/stack/downdetached.go:62-68`) reads the record, +identity-checks each group, SIGTERMs in reverse start order with bounded +SIGKILL escalation, and confirms per component: + +> ```go +> // reads the persisted pgid record, identity-checks each recorded group, SIGTERMs +> // the live ones in reverse start order with bounded SIGKILL escalation, and +> // confirms teardown per component by the channel each has (server/postgres by +> // socket quiescence, the socketless runner by group-ESRCH). Only pgids read from +> // this stack's own state-dir file are ever signaled, and each group's identity +> // (pgid + leader start-time token) is re-verified immediately before every +> // signal. +> ``` + +The mechanism is already Linux-anchored: the start-time reader is +`/proc`-backed (`pgidfile.go:333-336`): + +> ```go +> // The wired implementation reads /proc//stat, which exists +> // only on Linux — and the embedded stack is Linux/podman-only at runtime anyway +> ``` + +The substrate invariant is rootless podman +(`go/internal/runtime/podman.go:23-27`): + +> ```go +> // PodmanCLI, its rootless-podman-CLI implementation. Rootless is a hard +> // requirement (design: architecture-lineage): no daemon, no root, no rootful fallback. +> // Containers run with --userns=keep-id:uid=,gid= so the +> // invoking host user is mapped to the baked agent uid; files the agent writes +> // in a bind-mount still map back to the invoking user on the host. +> ``` + +and the stack's own postgres container couples the keep-id user mapping to the +frozen DSN contract +(`go/internal/stack/adapters/postgres_container.go:44-48`): + +> ```go +> // role a user-less DSN connects as so the frozen S4 DSN (host= port=

+> // dbname=compass sslmode=disable — no user=) authenticates: pgx resolves a +> // user-less DSN to the OS user, and under --userns=keep-id the container +> // runs as that same host user, so the createdb superuser must be it too. +> ``` + +**What today's mechanism does NOT do: crash recovery.** `Up` returns once the +stack is Ready and nothing supervises afterward — health is a probe-on-demand, +not a monitor loop (`stack.go:198-200`): + +> ```go +> // Health probes current readiness by asking the server over the socket. An +> // answering probe is Ready (or Attached, for a stack that never spawned); a +> // failing probe is Failed with the probe error as the detail. +> ``` + +A crashed compass-server stays down until the operator reruns `compass-stack +up`. This is the one genuine capability Quadlet/systemd would add. + +### The Quadlet option, factually + +Quadlet is podman's systemd generator: `.container`/`.pod`/`.volume`/`.network` +unit files generated into regular systemd services, with `[Unit]` dependency +translation (`After=`/`Requires=` between Quadlet units), systemd `Restart=` +policy, sdnotify readiness (`Notify=true` → `--sdnotify container`), and +boot-start via a generator-applied `[Install] WantedBy=` section. Rootless is +supported as systemd USER units: unit files under +`~/.config/containers/systemd/` (and the other rootless search paths), run in +the user's session — explicitly NOT via `User=` in a system unit ("Quadlet +units do not support running as a non-root user by defining the User, Group, or +DynamicUser systemd options. If you want to run a rootless Quadlet, you will +need to create the user and add the unit file to one of the above rootless unit +search paths"). Quadlet requires cgroup v2. Source: + (read +this session). So the rootless/no-daemon substrate invariant DOES hold under +Quadlet user units — podman stays daemonless; systemd (already pid 1) is the +supervisor, not a container daemon. + +### Why keep the hand-rolled mechanism (the recommendation) + +1. **Two supervision models vs one — the crux (a maintenance-cost argument, + not a slice-size one).** Quadlet needs systemd, so it covers only + self-host-on-systemd-Linux. Two other tiers fall outside it and keep the + DL-183 mechanism regardless: the embedded front door (DL-319 dual-mode app, + `DECISIONS.md:298`: the app "spawns/supervises a LOCAL stack via rootless + podman on the user's own machine (macOS via podman machine, Linux native)") + cannot ride systemd on macOS at all, and dev/devenv + non-systemd Linux + hosts also can't. Note the Quadlet-covered slice is NOT marginal — by + DL-259's own wording (`DECISIONS.md:292`, "a KVM-capable **Linux** + machine") the whole named self-host tier IS Linux — so the argument is not + "Quadlet only helps a corner"; it is that adopting Quadlet means + maintaining, testing, and keeping behaviorally-equivalent TWO supervision + models (unit-file bring-up/teardown semantics AND the pgid record) for the + same four services, permanently, because the DL-183 path cannot retire while + embedded and non-systemd tiers exist. One supervision model beats two. + **Caveat (a cost the KEEP ruling also owns):** the pgid mechanism is not + literally "works everywhere" today — its start-time identity token reads + `/proc//stat`, Linux-only, and on a non-Linux unix "this reader fails + and up refuses" (`pgidfile.go:333-338`), so on macOS the runner + server run + as darwin HOST processes (embedded-revival OQ-7, + `ui/compass-native-embedded-revival/design.md:911-912`) that the current + record cannot identity-token. The mechanism is PORTABLE to darwin (setpgid / + SIGTERM / `kill(-pgid)` all exist; a `kern.proc` start-time read is a bounded + swap behind the existing `readStartTime` var seam), but that port is unbuilt + work the DL-319 / RIG-3238 embedded-macOS lane owns. This darwin-port cost + exists under EITHER ruling (Quadlet can't run on macOS at all, so it never + removes it) — which is exactly why it does not flip the decision: it is a + cost of shipping embedded macOS, not a cost of choosing pgid over Quadlet. +2. **DL-183/DL-262 are load-bearing, frozen, and tested.** `DECISIONS.md:281` + (DL-183, Active) and `DECISIONS.md:295` (DL-262, Active) freeze the record + format, the identity-token verify-before-signal discipline, and the v1/v2 + cross-version grammar guard; the invariants are non-negotiable in the frozen + teardown record + (`ui/compass-stack-cross-process-teardown/design.md:299-301`): + + > ```text + > - **Only the exact persisted pgids are ever signaled.** Never a pattern kill, + > never a scan of the process table, never a pgid not read from this stack's + > own state-dir file. + > ``` + + Retiring a working, tested mechanism for a declarative-standard win that + only lands on one platform slice is churn without a capability gain the + wrapper (below) doesn't also deliver. +3. **The bring-up chain is imperative, not declarative.** The cold sequence + (`stack.go:89-93`, quoted above) interleaves containers with host-binary + children AND side-effect steps: TLS anchor generation (expiry-aware), runner + token minting (idempotent 0600), agent-image presence, readiness polls. + Under Quadlet, compass-server/compass-runner are host binaries — plain + systemd user services, not `.container` units — so "Quadlet" is really + "Quadlet for 2 containers + 2 plain user units + oneshot pre-units for the + TLS/token/image steps + sdnotify readiness plumbing in the server". That is + a full re-plumb of `spawnChain`, not a swap of supervisors. +4. **DL-259 compatibility cuts both ways.** DL-259 (`DECISIONS.md:292`) froze: + + > ```text + > The self-host stack stays a host-level bring-up on a KVM-capable Linux + > machine (`compass-stack up`; microVM D3 hard-fail consumed, no + > compose/Swarm packaging) + > ``` + + Quadlet unit files are systemd-native host-level configuration, NOT + compose/Swarm packaging — adopting them would not violate DL-259's letter. + But DL-259's operative clause names **`compass-stack up`** as the bring-up + surface; per-service Quadlet units would displace that entry point (the + operator's verbs become `systemctl --user start/stop`), which is a DL-259 + amendment, not a compatible extension. The thin wrapper below keeps + `compass-stack up` as the bring-up verb and is squarely inside DL-259. + +### The middle path: a documented thin systemd wrapper (recommended add-on) + +A single systemd USER unit that wraps the existing supervisor verbatim — +`Type=oneshot` + `RemainAfterExit=yes`, `ExecStart=compass-stack up`, +`ExecStop=compass-stack down` — shipped as documentation in the self-host doc +(DL-259's install surface), not as installed machinery. It delivers the real +operational wins systemd offers (start at boot via `loginctl enable-linger` + +`WantedBy=default.target`, survive logout, one `systemctl --user` status verb) +with ZERO code change and ZERO second supervision model: DL-183/DL-262 remain +the sole bring-up/teardown mechanism, systemd merely invokes it. +[INFERENCE] The unit shape is standard systemd; the exact unit text is Plan +task T1's deliverable and must be smoke-tested on a real systemd host before +the doc ships. What the wrapper does NOT deliver: per-child restart-on-crash +(systemd sees one oneshot, not four services) — carried as OQ-3. **And a +caveat the T1 doc MUST carry:** with `RemainAfterExit=yes` systemd reports the +unit `active` from the instant `up` returns until an explicit stop, so a child +that crashes afterward is invisible to systemd — the unit's `active` state +means only "up-once-succeeded", NOT "stack healthy". The status source of +truth stays `compass-stack status`, never `systemctl --user status`; the fuller +wrapper failure modes are OQ-4. + +### Alternatives considered + +- **Podman Quadlet per-service units (declined, pending Matt — OQ-1).** What + it is: `.container` units for postgres/collector, plain systemd user units + for compass-server/compass-runner, `After=`/`Requires=` ordering, `Restart=` + crash recovery, sdnotify readiness. Why it loses: Linux/systemd-only, so the + DL-183 mechanism survives anyway (two models); retires a tested frozen + mechanism; requires re-plumbing the imperative cold sequence into oneshot + pre-units; displaces the DL-259-named `compass-stack up` entry point. What + it would win: per-service `Restart=` crash recovery, boot-start, journald + logs, a declarative standard operators already know. +- **Hand-rolled DL-183/DL-262 pgid supervision (recommended keep).** What it + is: the shipped mechanism quoted above. Why it wins: ONE PORTABLE model + across the tiers — native today on self-host + embedded Linux, and portable + to embedded macOS via a bounded `readStartTime` seam swap the RIG-3238 lane + owns (NOT running there today: `pgidfile.go:333-338` `up` refuses on non-Linux + unix); frozen, tested, load-bearing; teardown-safety invariants proven. Known + gap: no crash recovery (OQ-3) and no boot-start (covered by the wrapper). +- **Blocking `compass-stack up --supervise` under a `Type=exec` unit + (surfaced by the red-team — OQ-3).** What it is: a NEW blocking foreground + mode (up-to-Ready, then watch children / poll Health, exit non-zero on a + child death, teardown on signal) paired with a `Type=exec`/`Type=notify` + systemd unit using `Restart=on-failure` + `RestartSec=`. Division of labor: + the stack keeps its single supervision model (DL-183 spawn order, + identity-token teardown, drain), systemd supplies ONLY whole-unit + restart/backoff. What it wins: systemd-grade crash recovery with ONE model + and no per-service Quadlet — and it fixes the oneshot wrapper's status-lie + (a dead stack is `failed`, not fake-`active`) and stop-what-you-didn't-start + edge (systemd owns the pid it spawned). Cost: a real code change (blocking + mode + child-exit watching, where `up` abandons children by design today), + so unlike T1 it is not free. This is the strongest competitor to the drafted + accept-the-gap posture — routed to Matt as OQ-3's second option, not decided + here. +- **Non-systemd process supervisors (s6 / runit / supervisord) — declined.** + Dominated by the same two-models argument (none covers embedded macOS or + non-systemd-agnostic dev) plus a new third-party runtime dependency the + self-host tier does not otherwise carry. +- **`compass-stack up` as a thin facade over `systemctl` (declined).** Keep the + DL-259 verb but have it generate + `systemctl --user` Quadlet units under the + hood. Rejected: it still stands up the full two-model machinery (the pgid + path survives for embedded/non-systemd) while adding a generation layer, and + it inherits every imperative-cold-sequence re-plumb of full Quadlet — the + facade hides the second model without removing it. +- **Docker-socket engine (declined).** A daemon-model engine violates the + substrate's rootless/no-daemon hard requirement (`podman.go:23-24`, quoted + above), and docker's userns remapping is daemon-global with no + `--userns=keep-id:uid=` per-container equivalent — the exact grounds the + embedded-revival record rejected it on for the runner and the macOS runtime + (`ui/compass-native-embedded-revival/design.md:963-966`): + + > ```text + > (`podman.go:25-27`), on rootless podman which is "a hard requirement … no + > daemon, no root, no rootful fallback" (`podman.go:24`). Docker's userns + > remapping is DAEMON-GLOBAL (`daemon.json` `userns-remap`), not a per-container + > keep-id, so that contract cannot be reproduced on docker. + > ``` + + At the stack layer the same holds: the postgres container's DSN + authentication depends on keep-id (`postgres_container.go:44-48`, quoted + above), and DL-262's teardown grammar is podman-verbed (`downdetached.go` + `podman stop`/`rm -f`). Declined for the same "other engines" reasons, + recorded here for completeness. + +## Plan + +The recommended ruling is keep-hand-rolled, so the plan is deliberately +minimal: no supervision code changes. The only implementation work is the +documented systemd wrapper (T1) and the explicit close-out (T2). If Matt rules +for Quadlet adoption instead (OQ-1), this plan is void and a new impl plan is +drafted under that ruling — the tasks below are NOT a Quadlet migration plan. + +### Global Constraints + +- Platform floor for the wrapper: Linux with systemd user sessions + cgroup v2 + (Quadlet/systemd requirement); the wrapper is optional documentation, never + a prerequisite — `compass-stack up` alone remains fully supported. +- Rootless invariant: everything runs as the invoking user; no root units, no + daemon, no rootful fallback (`go/internal/runtime/podman.go:23-24`). +- DL-259: `compass-stack up` stays the bring-up verb; no compose/Swarm + packaging; the self-host doc + flake + preflight remain the install surface. +- DL-183 teardown-safety invariants survive untouched: only exact persisted + identities are ever signaled; verify-before-signal; bounded escalation; + survivor-rewrite on partial failure + (`ui/compass-stack-cross-process-teardown/design.md:299-301`). +- ID allocation + freeze order (ledger-collision guard): this record claims + DL-328. Main's ledger tail is DL-324; DL-325/DL-326/DL-327 are claimed by + unlanded PRs (#804/#859 on DL-325, #859 on DL-326, #868 on DL-327), so DL-328 + is the first free number as of this writing. The driver MUST re-grep main's + then-current ledger tail immediately before landing and take the next free id + if DL-328 is taken. The row cites its own decision self-containedly (no + cross-cite to an unlanded number), so merge order cannot falsify it. + +### T1 — Documented systemd user-unit wrapper in the self-host doc + +Add a "run at boot under systemd" section to the self-host doc (DL-259's +install surface) containing a complete, copy-pasteable systemd USER unit that +wraps the existing CLI, plus the `loginctl enable-linger ` step and the +`systemctl --user enable --now` / `status` / `stop` verbs. Documentation only; +no shipped unit file, no code change. + +Interfaces: + +- Consumes: `compass-stack up` / `compass-stack down` exactly as shipped + (`go/cmd/compass-stack/main.go:8-10` — up returns at Ready and does not + block; down is the DL-183 cross-process teardown), which is what makes + `Type=oneshot` + `RemainAfterExit=yes` the correct service type. +- Produces: a documented unit of the shape + `[Service] Type=oneshot / RemainAfterExit=yes / ExecStart=/compass-stack up --state-dir

/ ExecStop=/compass-stack down --state-dir / TimeoutStopSec=90` + with `[Unit] After=network.target`, `[Install] WantedBy=default.target`, and + an explicit `Environment=PATH=` (or `ExecSearchPath=`), placed at + `~/.config/systemd/user/compass-stack.service`. The unit content MUST carry + every OQ-4 item (explicit `--state-dir`, absolute paths + PATH, `After=`, + `TimeoutStopSec>=90`) and the doc MUST carry the linger-mandatory step, the + `journalctl --user -u compass-stack` log note, and the status-truth caveat + (`compass-stack status`, not `systemctl status`). +- Test cycle: smoke-test the documented unit on a real systemd host — + `systemctl --user start` reaches Ready (verify `compass-stack status`), + `systemctl --user stop` tears down (verify the `stack.pgids` file is gone + and sockets are dark), a reboot with linger enabled brings the stack back, + AND the negative case: kill a child post-start and confirm systemd still + reports `active` (proving the OQ-4 status-truth caveat the doc states). The + doc ships only after this cycle passes. + +### T2 — Close-out + +Record the ruling's consequence honestly: no supervision code changes; RIG-3239 +closes as a decision record + doc task. The main agent owns the DECISIONS.md +ledger row referencing this record. + +Interfaces: + +- Consumes: this record (frozen on merge) and Matt's OQ-1 ruling. +- Produces: the RIG-3239 close-out state; no code artifacts. +- Test cycle: none (no code); markdownlint on the record is the gate. + +## Tasks + +- [ ] T1 — self-host doc gains the systemd user-unit wrapper section + (copy-pasteable unit + linger + verbs), smoke-tested on a systemd host. +- [ ] T2 — close-out: ruling recorded, no code changes, ledger row deferred to + the main agent. + +## Open Questions + +### OQ-1 [load-bearing, Matt fork] — adopt Quadlet, or keep the hand-rolled supervision? + +The central fork. **Recommendation: keep DL-183/DL-262 as the single +supervision mechanism; do not adopt per-service Quadlet units; add the T1 +documented systemd wrapper for boot-start.** Grounds: Quadlet is +Linux/systemd-only so the pgid path survives regardless (two models vs one); +DL-183/DL-262 are frozen, tested, and load-bearing; the cold sequence is +imperative (TLS anchor, token mint, readiness polls) and would need oneshot +pre-units + sdnotify re-plumbing; per-service Quadlet displaces the +DL-259-named `compass-stack up` entry point. Quadlet's genuine wins +(per-service `Restart=` crash recovery, journald, a standard operators know) +are real — if Matt weighs crash recovery heavily, the fork reopens (see OQ-3). +Matt rules. + +### OQ-2 [load-bearing, Matt fork] — is the T1 wrapper wanted at all? + +The wrapper is an add-on, not a dependency of the ruling. If Matt prefers the +self-host doc stay minimal ("run `compass-stack up` in a tmux/session and be +done"), T1 drops and RIG-3239 closes as a pure decision record. +**Recommendation: ship T1** — boot-start/logout-survival is the most common +real self-host ask and costs one doc section. + +### OQ-3 [load-bearing, Matt fork] — crash recovery: accept the gap, build a blocking `--supervise`, or reopen Quadlet? + +Neither today's mechanism nor the T1 oneshot wrapper restarts a crashed child: +`Health` is probe-on-demand (`stack.go:198-200`), and systemd sees the oneshot +wrapper as one unit that stays `active` after `up` returns, not four services. +Per-service `Restart=` is exactly what full Quadlet adoption would buy. Three +options, in ascending code cost: + +- **(a) Accept the gap (T1 oneshot wrapper as drafted).** A crashed stack + surfaces legibly via `compass-stack status` / the client's connection + failure, and the always-on-VPS steady state (DL-319) makes silent long-lived + crashes operator-visible. Zero code. **Recommendation for v1.** +- **(b) Build a blocking `compass-stack up --supervise` under a `Type=exec` + (or `Type=notify`) unit with `Restart=on-failure`.** The supervisor blocks + after Ready, watches its children, and exits non-zero on a child death; + systemd restarts the whole unit. This adds systemd-grade WHOLE-STACK crash + recovery with ONE supervision model (DL-183 spawn/teardown unchanged; systemd + supplies only restart/backoff) and simultaneously fixes the oneshot wrapper's + status-lie and stop-what-you-didn't-start edges (OQ-4). Whole-stack (not + per-service) restart is arguably the RIGHT granularity anyway — the cold + sequence's ordering means a restarted postgres needs the server's readiness + re-verified. Cost: a real code change (a blocking mode + child-exit watching, + where `up` abandons children by design today). +- **(c) Reopen Quadlet (OQ-1).** Per-service `Restart=`, at the cost of two + supervision models — the OQ-1 tradeoff. + +**Recommendation: (a) for v1, with (b) as the named follow-up if field reports +show crash recovery matters** — (b) dominates the originally-drafted +in-supervisor-loop shape and delivers recovery WITHOUT a second model, so it, +not Quadlet, is the escalation path. If Matt rates crash recovery as a LAUNCH +requirement, choose (b) now (or, if per-service granularity is required, (c) +reopens OQ-1). Matt rules. + +### OQ-4 [non-load-bearing] — wrapper failure modes + unit content, unverified until smoke-tested + +The T1 oneshot unit shape wrapping a returns-at-Ready CLI is standard systemd +practice but was NOT executed this session. The T1 doc MUST address these, +each grounded, and the T1 smoke cycle resolves them (non-load-bearing: none +changes the ruling, all are doc-completeness for the executor): + +- **Status lies (the RemainAfterExit trap).** With `RemainAfterExit=yes` the + unit reads `active` from the instant `up` returns until an explicit stop, so + a child crashing later is invisible to systemd. The doc MUST state the status + source of truth is `compass-stack status`, and that the unit's `active` means + only "up-once-succeeded". (Option (b) in OQ-3 eliminates this: a `Type=exec` + unit tracks a live main pid, so a dead stack is `failed`.) +- **Stop-what-you-didn't-start.** `up` attaches to an already-live stack rather + than failing (`stack.go:131-146` `upLocked` probe→attach), so a systemd + `ExecStart` succeeds against a stack a manual `up`/dev session started; a + later `stop`/unit-failure/shutdown then runs `ExecStop=compass-stack down`, + which tears down whatever the shared state-dir's pgid record names + (`downdetached.go:62-68` — per-state-dir identity-safe, but not per-invoker). + The unit MUST pin an explicit `--state-dir` so the systemd stack and any + interactive stack are distinct unless deliberately shared. +- **Environment + boot order.** systemd user units inherit no login-shell PATH, + so `ExecStart` needs an absolute `compass-stack` path AND the stack's own + `LookPath` children resolve (podman; the DL-321 PATH-threaded sidecars) — set + an explicit `Environment=PATH=` or `ExecSearchPath=`. Add `After=network.target` + (the TLS door bind); podman needs no daemon so no socket dependency + (`podman.go:23-24`). `loginctl enable-linger` is MANDATORY, not optional — + rootless podman hard-requires `XDG_RUNTIME_DIR`, which exists only in a + lingering or logged-in user session (`defaultRuntimeDir` falls back to + `/run` when it is unset, `main.go:284-290`, but podman itself + still needs the real runtime dir). +- **`TimeoutStopSec`.** `ExecStop=compass-stack down` can take up to the 80s + worst-case drain (`downdetached.go:15-24`: "sum to 65s; on the escalation + path … 15 + (30+5) + (10+5) + (10+5) = 80s"), so set `TimeoutStopSec=` >= 90s + or systemd SIGKILLs mid-teardown. + +### Resolved within this record + +- Rootless under Quadlet: confirmed — rootless Quadlet runs as systemd USER + units from the rootless unit search paths (podman stays daemonless); it is + explicitly not achievable via `User=` in a system unit. Source: + . + Moot under the recommended keep ruling, load-bearing only if OQ-1 flips. +- DL-259 squaring: Quadlet units are host-level systemd config, not + compose/Swarm packaging — no letter violation; but per-service units would + displace the `compass-stack up` verb DL-259 names, so full adoption would + need a DL-259 amendment. The T1 wrapper keeps the verb and needs none. +- Docker-socket: declined at the stack layer (daemon model vs the + rootless/no-daemon hard requirement, no per-container keep-id equivalent), + mirroring the embedded-revival OQ-9 rejection — see Alternatives. From b5acd4d53a5e9dfef483edc944ebd9c6e53e7c23 Mon Sep 17 00:00:00 2001 From: mintaka Date: Fri, 4 Sep 2026 16:57:37 -0400 Subject: [PATCH 2/9] docs(platform): re-author stack supervision to Matt's ruling + fold red-team (RIG-3239) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Re-author the self-host stack supervision record against Matt's RIG-3258 ruling and fold the design-critic red-team on the result. The original draft weighed Quadlet-vs-pgid and proposed accepting the crash-recovery gap for v1; Matt ruled: ship a one-command constant-on service on ALL platforms with auto-start at reboot and REAL crash recovery. Ruling folded into the record: - Keep the DL-183/DL-262 pgid mechanism as the SINGLE cross-platform model; decline Quadlet on a third independent ground (Linux/systemd-only — structurally cannot meet the all-platforms bar). - Whole-stack crash recovery = a blocking `compass-stack up --supervise` foreground mode wrapped by the platform-native OS supervisor's restart policy (systemd `Type=exec` on Linux, launchd `KeepAlive` on macOS). - One-command `compass-stack service install`/`uninstall` writing + enabling the native unit; the OS supervisor supplies only restart/backoff/boot; the pgid mechanism stays the sole bring-up/teardown and status truth stays `compass-stack status`. Red-team corrections folded (core direction survived every attack): - T1: the supervise loop is NOT a free consumer of the stack core. Name the bounded core change it forces — `Process.Wait` is single-caller (a fan-in Wait plus drainChildren's Wait double-calls exec.Cmd.Wait), so the loop takes sole Wait ownership; container children's Wait is 120s-bounded (supervise polls their liveness instead of a restart-storm every ~2min); Wait's cancel path group-SIGKILLs (teardown on a non-cancelable ctx, the upLocked WithoutCancel pattern); a partial drain orphans a survivor across restart (pre-spawn record-consuming cleanup). - T2: pin `KillMode=mixed` / `AbandonProcessGroup=true` so the OS does not parallel-kill the children before the DL-183 ordered drain; pin `RestartSec`/`ThrottleInterval` + an explicit start-limit posture; drop the per-user-manager-ignored `After=network.target` (loopback bind); make the `down` verb unit-aware so it stops through the unit rather than being undone by a restart. - T3: the darwin identity reader is a TWO-site swap (spawn-side pgidfile seam AND down-side groupsignal reader) sharing one uint64 timeval encoding, with the existing mirror-test extended so they cannot drift. - OQ-5: fold the launchd crashloop cost into option (a) — an experimental label does not stop a KeepAlive crashloop, so (a) carries an install-time preflight + a bring-up-failure self-limit. Routed to Matt as RIG-3261. Ledger-impact: DL-328 rewritten to the ruled design (keep pgid, decline Quadlet, ship --supervise + service install), Status Active. Spec-impact: none Refs RIG-3239 Refs RIG-3261 Co-authored-by: Matt Wilkinson --- docs/designs/DECISIONS.md | 2 +- .../compass-stack-supervision/design.md | 889 ++++++++++++------ 2 files changed, 598 insertions(+), 293 deletions(-) diff --git a/docs/designs/DECISIONS.md b/docs/designs/DECISIONS.md index fbab6533..2d6683be 100644 --- a/docs/designs/DECISIONS.md +++ b/docs/designs/DECISIONS.md @@ -427,7 +427,7 @@ check enforces the mechanical half. Full rationale: | DL-299 | The `:vX.Y.Z` agent image is minted by digest re-tag (`skopeo copy`) of the newest already-published per-push `:git-` artifact at-or-before the release sha (resolved by walking first-parent ancestors for a GHCR-resolvable tag, never a path-based `git log`) — never a second ~90m closure build; `:latest` stays owned exclusively by the per-push publish job; the mint shares the `publish-agent-image` concurrency group under `queue: max` + `cancel-in-progress: false` so a non-superseding release mint is serialized, never dropped | Active (Matt, 2026-08-25) | [unified release lane](infra/release/compass-unified-release-lane/design.md) | | DL-300 | `publish-agent-image.yml` retires as a FILE with its DUTY relocated into the unified lane's per-push `publish-image` job (verbatim `publish.sh`, same closure-paths gate, same serialize/immutability/coherence/least-privilege/off-hot-path posture) via two-PR staging (land the fold, observe one green per-push publish, then delete) — accepted by the agent-image zone owner (compass-managed), who amend their record with a superseded-by pointer | Active (Matt, 2026-08-25) | [unified release lane](infra/release/compass-unified-release-lane/design.md) | | DL-301 | release-please authenticates with a scoped GitHub App installation token (contents + pull-requests, installation-scoped, per-run) so the release-please-opened Release PR receives `pull_request` CI and branch protection is a real merge gate — a bare `GITHUB_TOKEN`-opened PR gets none under GitHub's recursion guard | Active (Matt, 2026-08-25) | [unified release lane](infra/release/compass-unified-release-lane/design.md) | -| DL-328 | Self-host stack supervision KEEPS the hand-rolled DL-183/DL-262 pgid mechanism as the SINGLE built-in supervision model; per-service Podman Quadlet units are NOT adopted (Linux/systemd-only, so the pgid path survives regardless — two behaviorally-equivalent models vs one; the imperative cold sequence would need oneshot pre-units + sdnotify re-plumbing; per-service units displace the DL-259-named `compass-stack up` verb). A DOCUMENTED thin systemd USER-unit wrapper (`Type=oneshot`+`RemainAfterExit=yes`, ExecStart=`compass-stack up`/ExecStop=`compass-stack down`) MAY ship in the self-host doc for boot-start/logout survival — systemd wrapping the supervisor, never replacing it. Docker-socket declined at the stack layer (daemon model vs the rootless/no-daemon hard requirement; no per-container keep-id equivalent). CRASH RECOVERY is an accepted v1 gap with a named follow-up (blocking `compass-stack up --supervise` under a `Type=exec`/`Restart=on-failure` unit — systemd whole-stack restart with one model), left to Matt (OQ-3); adopt-Quadlet-vs-keep is OQ-1 | Active (Matt, 2026-09-04) | [stack supervision §Approach](platform/compass-stack-supervision/design.md#approach) | +| DL-328 | Self-host stack supervision KEEPS the hand-rolled DL-183/DL-262 pgid mechanism as the SINGLE cross-platform supervision model; per-service Podman Quadlet units are NOT adopted (Linux/systemd-only, so Quadlet structurally fails the ruled all-platforms bar AND the pgid path survives regardless — two models vs one; the imperative cold sequence would need oneshot pre-units + sdnotify re-plumbing; per-service units displace the DL-259-named `compass-stack up` verb). Whole-stack CRASH RECOVERY ships as a blocking `compass-stack up --supervise` foreground mode (up-to-Ready, watch children, non-zero exit on child death, teardown on signal) wrapped by the platform-native OS supervisor's restart policy, plus a one-command `compass-stack service install`/`uninstall` that writes + enables the native unit — systemd USER unit (`Type=exec`, `Restart=on-failure`, `TimeoutStopSec>=90`) on Linux, launchd LaunchAgent (`RunAtLoad`, `KeepAlive={SuccessfulExit=false}`, `ExitTimeOut>=90`) on macOS — auto-start at reboot + restart-on-crash; the OS supervisor supplies ONLY restart/backoff/boot-start (DL-183 spawn/teardown unchanged), status truth stays `compass-stack status`. Docker-socket declined at the stack layer (daemon model vs the rootless/no-daemon hard requirement; no per-container keep-id equivalent) | Active (Matt, 2026-09-04) | [stack supervision §Approach](platform/compass-stack-supervision/design.md#approach) | ## Test harness diff --git a/docs/designs/platform/compass-stack-supervision/design.md b/docs/designs/platform/compass-stack-supervision/design.md index 28d8e56b..e054b2a0 100644 --- a/docs/designs/platform/compass-stack-supervision/design.md +++ b/docs/designs/platform/compass-stack-supervision/design.md @@ -1,31 +1,42 @@ -# Compass self-host stack supervision: Quadlet vs hand-rolled pgid +# Compass self-host stack supervision: constant-on cross-platform service Status: Draft Issue: RIG-3239 ## Problem / Intent -Decide the supervision mechanism for the long-lived self-host stack services -(compass-server, compass-runner, the containerized postgres, the bundled OTel -collector): adopt Podman **Quadlet** (systemd-native declarative unit files) or -keep the hand-rolled **DL-183/DL-262 pgid** mechanism `compass-stack` ships -today, with the Docker-socket engine recorded as considered-and-declined at the -stack layer. Scope is bring-up order, teardown, restart policy, crash recovery, -and rootless posture of the stack services ONLY — the per-session runner -backend is frozen out of scope (RIG-3070: podman permanent for self-host, -microVM behind the seam; see -`ui/compass-native-embedded-revival/design.md:71-74`), as is the macOS embedded -runner backend (sibling RIG-3238 Apple-container record). +Decide how the long-lived self-host stack services (compass-server, +compass-runner, the containerized postgres, the bundled OTel collector) run +as a **constant-on, auto-restarting service on all platforms** — Linux and +macOS — with one-command install, auto-start at reboot, and whole-stack +crash recovery. The supervision-mechanism fork (Podman **Quadlet** vs the +hand-rolled **DL-183/DL-262 pgid** mechanism `compass-stack` ships today) is +part of this decision, with the Docker-socket engine recorded as +considered-and-declined at the stack layer. Scope is bring-up order, +teardown, restart policy, crash recovery, boot-start, and rootless posture +of the stack services ONLY — the per-session runner backend is frozen out of +scope (RIG-3070: podman permanent for self-host, microVM behind the seam; +see `ui/compass-native-embedded-revival/design.md:71-74`), as is the macOS +*embedded runner* backend (sibling RIG-3238 Apple-container record). One +COUPLING crosses that fence and is grounded below: "constant-on service on +macOS" shares the runner-on-darwin / VM-topology unknowns that RIG-3238 +carries as its OQ-12 and the embedded-revival record carries as its OQ-7. ## Approach -**Recommended ruling (Matt fork — see Open Questions OQ-1): keep the -hand-rolled DL-183/DL-262 pgid supervision as the SINGLE built-in mechanism; -do not adopt per-service Quadlet units. Optionally ship a documented THIN -systemd user-unit wrapper around the existing `compass-stack up`/`down` for -self-hosters who want boot-start and logout survival — systemd wrapping the -supervisor, never replacing it.** Docker-socket is declined at the stack layer, -mirroring its per-session-runner rejection. +**Ruled (Matt, 2026-09-04 — the OQ-1/OQ-2/OQ-3 forks are decided, see +Resolved decisions): keep the hand-rolled DL-183/DL-262 pgid supervision as +the SINGLE cross-platform supervision model; do not adopt per-service +Quadlet units (Quadlet is Linux/systemd-only and structurally cannot meet +the all-platforms bar). Build REAL whole-stack crash recovery as a blocking +`compass-stack up --supervise` foreground mode, and ship a one-command +`compass-stack service install` that writes + enables the platform-native +supervisor unit — a systemd USER unit on Linux, a launchd LaunchAgent on +macOS — with auto-start at reboot and restart-on-crash. The OS supervisor +supplies ONLY restart/backoff/boot-start; DL-183 spawn order and +identity-token teardown remain the sole bring-up/teardown mechanism.** +Docker-socket is declined at the stack layer, mirroring its +per-session-runner rejection. ### What exists today (grounded) @@ -132,7 +143,8 @@ not a monitor loop (`stack.go:198-200`): > ``` A crashed compass-server stays down until the operator reruns `compass-stack -up`. This is the one genuine capability Quadlet/systemd would add. +up`. Matt's ruling rejects accepting that gap ("a VPS doesn't fix a crash"): +the supervise mode below closes it. ### The Quadlet option, factually @@ -153,41 +165,28 @@ this session). So the rootless/no-daemon substrate invariant DOES hold under Quadlet user units — podman stays daemonless; systemd (already pid 1) is the supervisor, not a container daemon. -### Why keep the hand-rolled mechanism (the recommendation) - -1. **Two supervision models vs one — the crux (a maintenance-cost argument, - not a slice-size one).** Quadlet needs systemd, so it covers only - self-host-on-systemd-Linux. Two other tiers fall outside it and keep the - DL-183 mechanism regardless: the embedded front door (DL-319 dual-mode app, +### Why pgid stays the single model (ruled) + +1. **The all-platforms bar — Quadlet structurally fails it.** Matt's ruling + requires a constant-on service "on all platforms": Linux AND macOS. + Quadlet needs systemd, and macOS has no systemd — so Quadlet cannot even + express the macOS half of the requirement, while the pgid mechanism is + portable (setpgid / SIGTERM / `kill(-pgid)` all exist on darwin; the + start-time identity read is a bounded per-OS swap behind the existing + `readStartTime` var seam, `pgidfile.go:331-338` — T3 below). This is a + second, independent reason on top of the two below. +2. **Two supervision models vs one — the maintenance crux.** Even on Linux, + adopting Quadlet means maintaining, testing, and keeping + behaviorally-equivalent TWO supervision models (unit-file + bring-up/teardown semantics AND the pgid record) for the same four + services, permanently: the embedded front door (DL-319 dual-mode app, `DECISIONS.md:298`: the app "spawns/supervises a LOCAL stack via rootless - podman on the user's own machine (macOS via podman machine, Linux native)") - cannot ride systemd on macOS at all, and dev/devenv + non-systemd Linux - hosts also can't. Note the Quadlet-covered slice is NOT marginal — by - DL-259's own wording (`DECISIONS.md:292`, "a KVM-capable **Linux** - machine") the whole named self-host tier IS Linux — so the argument is not - "Quadlet only helps a corner"; it is that adopting Quadlet means - maintaining, testing, and keeping behaviorally-equivalent TWO supervision - models (unit-file bring-up/teardown semantics AND the pgid record) for the - same four services, permanently, because the DL-183 path cannot retire while - embedded and non-systemd tiers exist. One supervision model beats two. - **Caveat (a cost the KEEP ruling also owns):** the pgid mechanism is not - literally "works everywhere" today — its start-time identity token reads - `/proc//stat`, Linux-only, and on a non-Linux unix "this reader fails - and up refuses" (`pgidfile.go:333-338`), so on macOS the runner + server run - as darwin HOST processes (embedded-revival OQ-7, - `ui/compass-native-embedded-revival/design.md:911-912`) that the current - record cannot identity-token. The mechanism is PORTABLE to darwin (setpgid / - SIGTERM / `kill(-pgid)` all exist; a `kern.proc` start-time read is a bounded - swap behind the existing `readStartTime` var seam), but that port is unbuilt - work the DL-319 / RIG-3238 embedded-macOS lane owns. This darwin-port cost - exists under EITHER ruling (Quadlet can't run on macOS at all, so it never - removes it) — which is exactly why it does not flip the decision: it is a - cost of shipping embedded macOS, not a cost of choosing pgid over Quadlet. -2. **DL-183/DL-262 are load-bearing, frozen, and tested.** `DECISIONS.md:281` - (DL-183, Active) and `DECISIONS.md:295` (DL-262, Active) freeze the record - format, the identity-token verify-before-signal discipline, and the v1/v2 - cross-version grammar guard; the invariants are non-negotiable in the frozen - teardown record + podman on the user's own machine (macOS via podman machine, Linux + native)") cannot ride systemd on macOS at all, and dev/devenv + + non-systemd Linux hosts also can't — so the DL-183 path cannot retire + while those tiers exist. One supervision model beats two. DL-183/DL-262 + are load-bearing, frozen, and tested (`DECISIONS.md:281`, `:295`); the + invariants are non-negotiable in the frozen teardown record (`ui/compass-stack-cross-process-teardown/design.md:299-301`): > ```text @@ -196,98 +195,197 @@ supervisor, not a container daemon. > own state-dir file. > ``` - Retiring a working, tested mechanism for a declarative-standard win that - only lands on one platform slice is churn without a capability gain the - wrapper (below) doesn't also deliver. 3. **The bring-up chain is imperative, not declarative.** The cold sequence (`stack.go:89-93`, quoted above) interleaves containers with host-binary - children AND side-effect steps: TLS anchor generation (expiry-aware), runner - token minting (idempotent 0600), agent-image presence, readiness polls. - Under Quadlet, compass-server/compass-runner are host binaries — plain - systemd user services, not `.container` units — so "Quadlet" is really - "Quadlet for 2 containers + 2 plain user units + oneshot pre-units for the - TLS/token/image steps + sdnotify readiness plumbing in the server". That is - a full re-plumb of `spawnChain`, not a swap of supervisors. -4. **DL-259 compatibility cuts both ways.** DL-259 (`DECISIONS.md:292`) froze: - - > ```text - > The self-host stack stays a host-level bring-up on a KVM-capable Linux - > machine (`compass-stack up`; microVM D3 hard-fail consumed, no - > compose/Swarm packaging) - > ``` - - Quadlet unit files are systemd-native host-level configuration, NOT - compose/Swarm packaging — adopting them would not violate DL-259's letter. - But DL-259's operative clause names **`compass-stack up`** as the bring-up - surface; per-service Quadlet units would displace that entry point (the - operator's verbs become `systemctl --user start/stop`), which is a DL-259 - amendment, not a compatible extension. The thin wrapper below keeps - `compass-stack up` as the bring-up verb and is squarely inside DL-259. - -### The middle path: a documented thin systemd wrapper (recommended add-on) - -A single systemd USER unit that wraps the existing supervisor verbatim — -`Type=oneshot` + `RemainAfterExit=yes`, `ExecStart=compass-stack up`, -`ExecStop=compass-stack down` — shipped as documentation in the self-host doc -(DL-259's install surface), not as installed machinery. It delivers the real -operational wins systemd offers (start at boot via `loginctl enable-linger` + -`WantedBy=default.target`, survive logout, one `systemctl --user` status verb) -with ZERO code change and ZERO second supervision model: DL-183/DL-262 remain -the sole bring-up/teardown mechanism, systemd merely invokes it. -[INFERENCE] The unit shape is standard systemd; the exact unit text is Plan -task T1's deliverable and must be smoke-tested on a real systemd host before -the doc ships. What the wrapper does NOT deliver: per-child restart-on-crash -(systemd sees one oneshot, not four services) — carried as OQ-3. **And a -caveat the T1 doc MUST carry:** with `RemainAfterExit=yes` systemd reports the -unit `active` from the instant `up` returns until an explicit stop, so a child -that crashes afterward is invisible to systemd — the unit's `active` state -means only "up-once-succeeded", NOT "stack healthy". The status source of -truth stays `compass-stack status`, never `systemctl --user status`; the fuller -wrapper failure modes are OQ-4. + children AND side-effect steps: TLS anchor generation (expiry-aware), + runner token minting (idempotent 0600), agent-image presence, readiness + polls. Under Quadlet, compass-server/compass-runner are host binaries — + plain systemd user services, not `.container` units — so "Quadlet" is + really "Quadlet for 2 containers + 2 plain user units + oneshot + pre-units for the TLS/token/image steps + sdnotify readiness plumbing in + the server". That is a full re-plumb of `spawnChain`, not a swap of + supervisors. And DL-259's operative clause (`DECISIONS.md:292`) names + **`compass-stack up`** as the bring-up surface; per-service Quadlet units + would displace that entry point (the operator's verbs become + `systemctl --user start/stop`), a DL-259 amendment. The service verb + below keeps `compass-stack` as the verb surface and is squarely inside + DL-259 (a systemd/launchd unit is host-level configuration, not + compose/Swarm packaging). + +### The ruled design: `--supervise` + `service install` + +Two additions, one supervision model. + +**1. Blocking `compass-stack up --supervise` — whole-stack crash recovery.** +A new foreground mode: run the existing `Up` to Ready, then BLOCK — watch +the spawned children and exit non-zero when any child dies; tear the stack +down on SIGTERM/SIGINT and exit zero. It builds on the existing handle +contract — every child exposes `Wait(ctx)` / `Signal(sig)` / `Pid()` +(`go/internal/stack/deps.go:136-138`) and `Health` is an on-demand probe +(`stack.go:198-211`) — but the supervise loop is NOT a free consumer of the +core: `Process.Wait` is single-caller, the container children's `Wait` is +120s-bounded, `Wait`'s cancellation path group-SIGKILLs, and a partial +drain orphans a survivor across restart. T1 below specifies the bounded +core change these force (sole `Wait` ownership, container liveness-poll, +non-cancelable teardown ctx, pre-spawn record cleanup). Division of labor: +the stack keeps its single supervision model (DL-183 spawn order, +identity-token teardown, drain); the OS supervisor wraps the ONE blocking +process and supplies only `Restart=on-failure` + backoff (systemd +`Type=exec`; launchd `KeepAlive`). Whole-stack (not per-service) restart is +the right granularity: the cold sequence's ordering means a restarted +postgres needs the server's readiness re-verified anyway. This also +eliminates the two failure modes a returns-at-Ready oneshot wrapper has: +the status-lie (a `Type=exec` unit tracks a live main pid, so a dead stack +reads `failed`, never fake-`active`) and stop-what-you-didn't-start (the OS +supervisor owns exactly the pid it spawned). An ATTACHED supervise (a live +stack already answered the probe, `stack.go:131-146`) owns no child +handles, so the loop degrades to Health-polling with the same exit contract +(exiting only after N consecutive failures, T1). + +**2. One-command `compass-stack service install` — the constant-on service.** +A new verb pair extending the `up|down|status|preflight` surface +(`go/cmd/compass-stack/main.go:8-13`): `service install` platform-detects +and renders + installs + enables the native unit; `service uninstall` +stops, disables, and removes it. + +- **Linux (systemd user unit)** at + `~/.config/systemd/user/compass-stack.service`: `Type=exec`, + `ExecStart=/compass-stack up --supervise --state-dir `, + `Restart=on-failure` + pinned `RestartSec=5`, `TimeoutStopSec=90` (the + DL-262 teardown drain worst case is ~80s, `downdetached.go:15-24`), + `KillMode=mixed` + an explicit start-limit posture (so the ordered drain + and the crashloop ceiling are chosen, not left to defaults), explicit + `Environment=PATH=` (user units inherit no login-shell PATH; podman + the + DL-321 PATH-threaded sidecars must resolve), `[Install] + WantedBy=default.target`; NO `After=network.target` (a system-manager + unit the per-user manager ignores; the server binds loopback only); install + checks/advises `loginctl enable-linger` (mandatory — rootless podman + requires a real `XDG_RUNTIME_DIR`, which exists only in a lingering or + logged-in session). +- **macOS (launchd LaunchAgent)** at + `~/Library/LaunchAgents/com.rigelbuild.compass-stack.plist`: + `ProgramArguments` = the same absolute `up --supervise` invocation, + `RunAtLoad=true`, `KeepAlive={SuccessfulExit=false}` (restart on crash, + stay down on clean exit), `ExitTimeOut=90` (launchd's SIGTERM→SIGKILL + grace, sized like `TimeoutStopSec`), pinned `ThrottleInterval=` + + `AbandonProcessGroup=true` (let the ordered drain finish before launchd + group-kills), explicit `EnvironmentVariables` PATH; installed + started + via `launchctl bootstrap gui/$UID` and enabled for boot. + +Stopping the unit sends SIGTERM to the supervise process, which runs the +normal DL-183 teardown — one code path, no `ExecStop` verb split — provided +the unit template does NOT let the OS pre-empt it: systemd's default +`KillMode=control-group` and launchd's default group teardown would SIGKILL +the children in parallel before the ordered drain, so the templates pin +`KillMode=mixed` / `AbandonProcessGroup=true` (T2). The operator's +`compass-stack down` verb also becomes unit-aware so it stops through the +unit rather than being undone by a restart (T2). Status truth stays +`compass-stack status` (the unit's state is process-liveness, not stack +health); the T2 checklist (former OQ-4, below) is the unit content gate. + +### The macOS runtime path (grounded) + +Matt has committed macOS to scope ("support this on macOS as well"), so +macOS-support is DECIDED; what needed grounding is the implementation path, +because **the stack currently refuses to run on darwin**: the whole +`internal/stack` + `cmd/compass-stack` tree is `//go:build unix` +(`pgidfile.go:1`, `main.go:1` — it COMPILES on darwin), but the start-time +identity reader is `/proc//stat`, Linux-only, and "on a non-Linux unix +this reader fails and up refuses" (`pgidfile.go:331-338`). Two candidate +paths were examined: + +- **Native-darwin supervise (chosen).** The refusal lifts by giving the + DL-183 identity token a darwin reader — `sysctl KERN_PROC` via + `golang.org/x/sys/unix` (already in the module, `go/go.mod:96`) reading + the `KinfoProc` start `timeval`. It is a bounded but TWO-site swap (T3 + below): the token is read both at spawn (`readStartTime`, + `pgidfile.go:331-338`) and independently at teardown + (`readGroupLeaderStartTime`, `groupsignal.go:93-98`), and + `GroupSignaller.Alive` compares them for `uint64` equality — so both + darwin readers must share ONE encoding (the token is a `uint64`, not an + opaque string; Linux clock-ticks vs darwin timeval never compare ACROSS + OSes, but the two readers on one OS must agree). With that swap the + darwin topology is the one the embedded-revival record already names + (OQ-7, `ui/compass-native-embedded-revival/design.md:911-912`): + compass-server + compass-runner run as darwin HOST processes (proc + entries, group-signal teardown works — setpgid/kill(-pgid) exist on + darwin), while postgres + agent containers run inside the podman-machine + Linux VM driven by the host podman CLI (ctr entries, `podman stop`/`rm` + teardown unchanged). +- **In-VM supervise (declined).** Run the whole `compass-stack` inside the + podman-machine VM and have launchd drive it over `podman machine ssh`. + Declined: it puts compass-server inside a VM the client must then reach + across a VM boundary, adds a second machine-lifecycle layer under the + unit (launchd → machine → stack), forfeits the DL-319 darwin-host-process + topology the embedded lane is building toward, and dies the moment the + RIG-3238 apple-container lane removes podman-machine. + +**What the native path does NOT resolve — the RIG-3238 coupling.** The +darwin start-time reader unblocks spawn/teardown identity, but a WORKING +macOS stack end-to-end still depends on unknowns the sibling lanes own: +embedded-revival OQ-7 (the DL-260 postgres DSN and the agent sockets are +AF_UNIX bind-mounts, and AF_UNIX does not cross the virtiofs/VM boundary — +unvalidated) and RIG-3238's OQ-12 ("the runner cannot run natively on +darwin once the podman-machine Linux VM is gone", +`docs/designs/platform/apple-container-macos-runner/design.md:713-733`, +which also notes the runner's podman host-capability preflight has no +darwin answer yet). Those are stack-TOPOLOGY unknowns, not supervision +unknowns — this record ships the supervision mechanics for macOS (T2 +launchd template + T3 darwin identity reader) and leaves the topology +validation with the lane that owns it. Whether macOS `service install` +GA-gates on that lane is OQ-5 (the one remaining fork, below). (The +`apple-container-macos-runner/design.md:713-733` line range cited above +resolves only once that sibling RIG-3238 record lands on main — PR #869 is +not yet merged; the citation is a forward reference, not a +same-tree anchor.) ### Alternatives considered -- **Podman Quadlet per-service units (declined, pending Matt — OQ-1).** What - it is: `.container` units for postgres/collector, plain systemd user units - for compass-server/compass-runner, `After=`/`Requires=` ordering, `Restart=` - crash recovery, sdnotify readiness. Why it loses: Linux/systemd-only, so the - DL-183 mechanism survives anyway (two models); retires a tested frozen - mechanism; requires re-plumbing the imperative cold sequence into oneshot - pre-units; displaces the DL-259-named `compass-stack up` entry point. What - it would win: per-service `Restart=` crash recovery, boot-start, journald - logs, a declarative standard operators already know. -- **Hand-rolled DL-183/DL-262 pgid supervision (recommended keep).** What it - is: the shipped mechanism quoted above. Why it wins: ONE PORTABLE model - across the tiers — native today on self-host + embedded Linux, and portable - to embedded macOS via a bounded `readStartTime` seam swap the RIG-3238 lane - owns (NOT running there today: `pgidfile.go:333-338` `up` refuses on non-Linux - unix); frozen, tested, load-bearing; teardown-safety invariants proven. Known - gap: no crash recovery (OQ-3) and no boot-start (covered by the wrapper). -- **Blocking `compass-stack up --supervise` under a `Type=exec` unit - (surfaced by the red-team — OQ-3).** What it is: a NEW blocking foreground - mode (up-to-Ready, then watch children / poll Health, exit non-zero on a - child death, teardown on signal) paired with a `Type=exec`/`Type=notify` - systemd unit using `Restart=on-failure` + `RestartSec=`. Division of labor: - the stack keeps its single supervision model (DL-183 spawn order, - identity-token teardown, drain), systemd supplies ONLY whole-unit - restart/backoff. What it wins: systemd-grade crash recovery with ONE model - and no per-service Quadlet — and it fixes the oneshot wrapper's status-lie - (a dead stack is `failed`, not fake-`active`) and stop-what-you-didn't-start - edge (systemd owns the pid it spawned). Cost: a real code change (blocking - mode + child-exit watching, where `up` abandons children by design today), - so unlike T1 it is not free. This is the strongest competitor to the drafted - accept-the-gap posture — routed to Matt as OQ-3's second option, not decided - here. +- **Podman Quadlet per-service units (declined — ruled).** What it is: + `.container` units for postgres/collector, plain systemd user units for + compass-server/compass-runner, `After=`/`Requires=` ordering, `Restart=` + crash recovery, sdnotify readiness. Why it loses: Linux/systemd-only, so + it structurally cannot meet the ruled all-platforms bar (no systemd on + macOS) AND the DL-183 mechanism survives anyway (two models); retires a + tested frozen mechanism; requires re-plumbing the imperative cold + sequence into oneshot pre-units; displaces the DL-259-named + `compass-stack up` entry point. What it would win: per-service `Restart=` + crash recovery, boot-start, journald logs, a declarative standard + operators already know — all of which the ruled `--supervise` + + `service install` design delivers at whole-stack granularity with one + model. +- **Accept the crash-recovery gap for v1 (declined — ruled).** The + originally-drafted posture: zero code, a crashed stack surfaces via + `compass-stack status` / client connection failure, and the always-on-VPS + steady state (DL-319) makes silent crashes operator-visible. Matt + rejected it verbatim ("a VPS doesn't fix a crash … opt 1 reasoning makes + no sense"): visibility is not recovery, and the macOS half of the + requirement never had a VPS assumption to lean on. +- **Documented-only thin systemd oneshot wrapper (declined — ruled).** The + originally-drafted add-on: a copy-pasteable `Type=oneshot` + + `RemainAfterExit=yes` user unit (`ExecStart=compass-stack up`, + `ExecStop=compass-stack down`) shipped as a doc section, Linux-only, no + code change. Matt upgraded it to a first-class product feature ("ship + with a one command way … on all platforms"). It also carries two edges + the ruled design eliminates: the status-lie (with `RemainAfterExit=yes` + the unit reads `active` from the instant `up` returns, so a later child + crash is invisible to systemd) and stop-what-you-didn't-start (`up` + attaches to an already-live stack rather than failing, + `stack.go:131-146`, so the unit's `ExecStop` could tear down a stack a + manual session started). - **Non-systemd process supervisors (s6 / runit / supervisord) — declined.** Dominated by the same two-models argument (none covers embedded macOS or non-systemd-agnostic dev) plus a new third-party runtime dependency the self-host tier does not otherwise carry. -- **`compass-stack up` as a thin facade over `systemctl` (declined).** Keep the - DL-259 verb but have it generate + `systemctl --user` Quadlet units under the - hood. Rejected: it still stands up the full two-model machinery (the pgid - path survives for embedded/non-systemd) while adding a generation layer, and - it inherits every imperative-cold-sequence re-plumb of full Quadlet — the - facade hides the second model without removing it. +- **`compass-stack up` as a thin facade over `systemctl` (declined).** Keep + the DL-259 verb but have it generate + `systemctl --user` Quadlet units + under the hood. Rejected: it still stands up the full two-model machinery + (the pgid path survives for embedded/non-systemd) while adding a + generation layer, and it inherits every imperative-cold-sequence re-plumb + of full Quadlet — the facade hides the second model without removing it. + (Note `service install` is NOT this facade: it renders a unit that wraps + the pgid supervisor; it never generates per-service units or delegates + bring-up to systemd.) - **Docker-socket engine (declined).** A daemon-model engine violates the substrate's rootless/no-daemon hard requirement (`podman.go:23-24`, quoted above), and docker's userns remapping is daemon-global with no @@ -310,187 +408,394 @@ wrapper failure modes are OQ-4. ## Plan -The recommended ruling is keep-hand-rolled, so the plan is deliberately -minimal: no supervision code changes. The only implementation work is the -documented systemd wrapper (T1) and the explicit close-out (T2). If Matt rules -for Quadlet adoption instead (OQ-1), this plan is void and a new impl plan is -drafted under that ruling — the tasks below are NOT a Quadlet migration plan. - ### Global Constraints -- Platform floor for the wrapper: Linux with systemd user sessions + cgroup v2 - (Quadlet/systemd requirement); the wrapper is optional documentation, never - a prerequisite — `compass-stack up` alone remains fully supported. -- Rootless invariant: everything runs as the invoking user; no root units, no - daemon, no rootful fallback (`go/internal/runtime/podman.go:23-24`). -- DL-259: `compass-stack up` stays the bring-up verb; no compose/Swarm - packaging; the self-host doc + flake + preflight remain the install surface. +- ONE supervision model only: DL-183/DL-262 pgid spawn + identity-token + teardown, wrapped by `--supervise`. The OS supervisor (systemd/launchd) + supplies ONLY restart/backoff/boot-start — never bring-up order, never + teardown, never per-service units. Never two models. +- Cross-platform: Linux (systemd user unit) AND macOS (launchd + LaunchAgent). All platforms is the ruled bar; a Linux-only deliverable + does not close this record. +- Rootless invariant: everything runs as the invoking user; no root units, + no daemon, no rootful fallback (`go/internal/runtime/podman.go:23-24`). +- DL-259 preserved: `compass-stack` stays the verb surface; no + compose/Swarm packaging — a systemd/launchd unit is host-level + configuration, not a packaging format; the self-host doc + flake + + preflight remain the install surface. - DL-183 teardown-safety invariants survive untouched: only exact persisted identities are ever signaled; verify-before-signal; bounded escalation; survivor-rewrite on partial failure (`ui/compass-stack-cross-process-teardown/design.md:299-301`). +- Stop grace: `TimeoutStopSec` (systemd) / `ExitTimeOut` (launchd) >= 90s — + the DL-262 teardown drain worst case is ~80s + (`downdetached.go:15-24`: "15 + (30+5) + (10+5) + (10+5) = 80s"). +- Status truth stays `compass-stack status`: the OS unit expresses + process-liveness of the supervise pid, never stack health semantics. - ID allocation + freeze order (ledger-collision guard): this record claims DL-328. Main's ledger tail is DL-324; DL-325/DL-326/DL-327 are claimed by - unlanded PRs (#804/#859 on DL-325, #859 on DL-326, #868 on DL-327), so DL-328 - is the first free number as of this writing. The driver MUST re-grep main's - then-current ledger tail immediately before landing and take the next free id - if DL-328 is taken. The row cites its own decision self-containedly (no - cross-cite to an unlanded number), so merge order cannot falsify it. + unlanded PRs (#804/#859 on DL-325, #859 on DL-326, #868 on DL-327), so + DL-328 is the first free number as of this writing. The driver MUST + re-grep main's then-current ledger tail immediately before landing and + take the next free id if DL-328 is taken. The row cites its own decision + self-containedly (no cross-cite to an unlanded number), so merge order + cannot falsify it. + +### T1 — Blocking `compass-stack up --supervise` mode + +Add the foreground supervise mode: `up --supervise` runs the existing `Up` +to Ready, then blocks watching the stack; a child death exits non-zero +(after draining the survivors); SIGTERM/SIGINT runs `Down` and exits zero. + +**T1 requires a bounded change to the stack core — NOT "the core +unchanged" (folded from the design-critic red-team, three interacting +contract facts).** + +1. **`Process.Wait` is single-caller, so a fan-in Wait plus drain + double-Waits.** Each `Wait` spawns its own `go cmd.Wait()` + (`adapters/process.go:132-136`), and `exec.Cmd.Wait` errors on a second + call; the core's contract is explicitly one sequential caller + ("drainChildren: Signal then Wait per child", `process.go:84-88`). A + supervise fan-in holding a `Wait` on EVERY child, then calling + `drainChildren` (which Signals-then-Waits each survivor), double-Waits + every survivor → spurious errors corrupt the joined drain result and can + report a clean drain as failed, which leaves the pgid record in place + (`stack.go:185-190` removes it only on a nil drain). So the supervise + loop takes **sole `Wait` ownership**: it holds the one waiter per child + and, on a death, drives the reverse-order teardown by `Signal` + + awaiting its OWN already-held wait results, never a second + `drainChildren` Wait. (Equivalent: make the handle's `Wait` + idempotent/shared — one waiter goroutine, result fanned out over a closed + channel. Either is a real core change; "unchanged" is dropped.) +2. **The container children's `Wait` is silently bounded to 120s** — + `containerProcess.Wait` → `podmanExec.wait` → `fireAndCheck` under + `context.WithTimeout(ctx, containerCommandTimeout=120s)` + (`postgres_container.go:20-25,271-278,323-327`); that budget was sized + for one-shot drain commands. A supervise fan-in Waiting on a HEALTHY + postgres gets a deadline error after 120s → the loop reads it as a child + death → drains the whole healthy stack and exits non-zero → OS supervisor + restarts → a guaranteed whole-stack restart every ~2 minutes. So + supervise does NOT Wait container children through the bounded adapter: + it supervises them by liveness poll (`podman container exists` / Health) + on the same interval, treating only a real exit / "no such container" as + death, OR via an unbounded `podman wait` path that re-invokes on timeout. +3. **Context discipline (else SIGTERM hard-kills the group before graceful + teardown).** `Process.Wait`'s cancellation path escalates to + `syscall.Kill(-pid, SIGKILL)` of the whole group (`process.go:152-169`). + If the fan-in Waits thread the CLI's `signal.NotifyContext` ctx, SIGTERM + cancels every in-flight Wait and hard-kills every child in parallel + BEFORE the DL-183 reverse-order SIGTERM drain runs. So the fan-in Waits + run on a background / non-cancelable ctx, and on signal the loop runs + `Down(context.WithoutCancel(ctx))` — the exact pattern `upLocked` already + uses for its failure drain (`stack.go:153`) — optionally with a fresh + deadline under the 90s unit stop grace. +4. **Pre-spawn survivor cleanup (else the restart loop systematizes an + orphan leak).** If a child-death drain partially fails, supervise exits + non-zero with the pgid record still describing the survivor; the + restarting `up` never consults that record — `upLocked` probes the socket + then spawns (`stack.go:136-155`), and the first `recordChild` REWRITES + the whole file to reflect only the new children (`stack.go:377-400`), + destroying the old survivor's teardown identity (nothing will ever signal + it — DL-183 forbids signaling any pgid not read from the current record), + and a half-dead old server still holding the socket can wedge the new + spawn into a bind-fail restart loop. So supervise mode gains a pre-spawn + step: before (re)spawning, run the `downdetached` record-consuming + cleanup against any existing `stack.pgids` (identity-checked, so a + post-reboot stale record is a safe no-op, `groupsignal.go:60-70`), or + refuse to spawn while a record with a live identity-checked entry exists. + +Division of labor is unchanged: the stack keeps its single supervision +model (DL-183 spawn order, identity-token teardown, drain); the OS +supervisor wraps the ONE blocking process and supplies only +`Restart=on-failure` + backoff (systemd `Type=exec`; launchd `KeepAlive`). +Whole-stack (not per-service) restart is the right granularity: the cold +sequence's ordering means a restarted postgres needs the server's readiness +re-verified anyway (the crashloop ceiling is the OS supervisor's +start-limit / throttle, pinned in the T2 templates). This also eliminates +the two failure modes a returns-at-Ready oneshot wrapper has: the +status-lie (a `Type=exec` unit tracks a live main pid, so a dead stack +reads `failed`, never fake-`active`) and stop-what-you-didn't-start (the OS +supervisor owns exactly the pid it spawned). -### T1 — Documented systemd user-unit wrapper in the self-host doc +Interfaces: -Add a "run at boot under systemd" section to the self-host doc (DL-259's -install surface) containing a complete, copy-pasteable systemd USER unit that -wraps the existing CLI, plus the `loginctl enable-linger ` step and the -`systemctl --user enable --now` / `status` / `stop` verbs. Documentation only; -no shipped unit file, no code change. +- Consumes: `stack.Up` / `Stack.Down` (`stack.go:94,183`), the per-child + `Wait(ctx)` process-handle contract (`deps.go:136-138`), `Stack.Health` + (`stack.go:198-211`) for the attached-stack degradation and container + liveness, and the `downdetached` record-consuming cleanup for the + pre-spawn step — plus the ONE core change above (sole-Wait-ownership or an + idempotent/shared `Wait`). +- Produces: a `--supervise` flag on the `up` verb in + `go/cmd/compass-stack/main.go` plus a supervise loop in `internal/stack` + (a method on `*Stack`, e.g. `Supervise(ctx) error`): fan-in the owned + process children's held `Wait` results on a non-cancelable ctx, poll + container-child + attached liveness on an interval; first death → drive + the reverse-order teardown from the held results and return non-nil (CLI + exits non-zero, OS supervisor restarts); ctx cancel → `Down` + (`context.WithoutCancel`), return nil on clean teardown; pre-spawn + consume any stale record. An attached stack (no owned children, + `stack.go:131-146`) supervises by polling `Health`, exiting non-zero only + after N consecutive probe failures (a single transient probe error must + not tear the stack — the restart then takes ownership of the manual + stack; interval + N stated in the impl). +- Test cycle (red → green, existing stub harness + `internal/stack/harness_test.go`): (1) a stubbed child's `Wait` returning + early causes `Supervise` to drain survivors and return non-nil — with NO + double-Wait error on the survivors; (2) ctx cancel causes a clean `Down` + (pgid record removed) and nil, with the drain running on a + non-cancelable ctx (no group SIGKILL before graceful teardown); (3) + attached supervise exits non-nil only after N consecutive health-probe + failures, not one; (4) a partial-drain child death → restart → the + surviving old child is torn down by the pre-spawn cleanup, not orphaned. + Plus a process-level smoke on Linux: `up --supervise`, `kill` a child, + assert non-zero exit and a clean survivor teardown. + +### T2 — `compass-stack service install` / `uninstall` + unit templates + +Add the `service` verb pair: platform-detect (`runtime.GOOS`), render the +unit from an embedded template with the resolved absolute binary path, +state dir, and flags, install it, enable + start it; `uninstall` stops, +disables, and removes it. Both idempotent. Interfaces: -- Consumes: `compass-stack up` / `compass-stack down` exactly as shipped - (`go/cmd/compass-stack/main.go:8-10` — up returns at Ready and does not - block; down is the DL-183 cross-process teardown), which is what makes - `Type=oneshot` + `RemainAfterExit=yes` the correct service type. -- Produces: a documented unit of the shape - `[Service] Type=oneshot / RemainAfterExit=yes / ExecStart=/compass-stack up --state-dir / ExecStop=/compass-stack down --state-dir / TimeoutStopSec=90` - with `[Unit] After=network.target`, `[Install] WantedBy=default.target`, and - an explicit `Environment=PATH=` (or `ExecSearchPath=`), placed at - `~/.config/systemd/user/compass-stack.service`. The unit content MUST carry - every OQ-4 item (explicit `--state-dir`, absolute paths + PATH, `After=`, - `TimeoutStopSec>=90`) and the doc MUST carry the linger-mandatory step, the - `journalctl --user -u compass-stack` log note, and the status-truth caveat - (`compass-stack status`, not `systemctl status`). -- Test cycle: smoke-test the documented unit on a real systemd host — - `systemctl --user start` reaches Ready (verify `compass-stack status`), - `systemctl --user stop` tears down (verify the `stack.pgids` file is gone - and sockets are dark), a reboot with linger enabled brings the stack back, - AND the negative case: kill a child post-start and confirm systemd still - reports `active` (proving the OQ-4 status-truth caveat the doc states). The - doc ships only after this cycle passes. - -### T2 — Close-out - -Record the ruling's consequence honestly: no supervision code changes; RIG-3239 -closes as a decision record + doc task. The main agent owns the DECISIONS.md -ledger row referencing this record. +- Consumes: the T1 `--supervise` mode; the CLI's existing flag/config + resolution (`main.go` — the rendered unit pins the SAME resolved + `--state-dir` and flags so the service stack and an interactive stack + are distinct unless deliberately shared). +- Produces: `service install` / `service uninstall` verbs extending the + `up|down|status|preflight` dispatch (`main.go:6`); two embedded unit + templates — + - systemd user unit → `~/.config/systemd/user/compass-stack.service`: + `Type=exec`, `ExecStart= up --supervise …`, `Restart=on-failure`, + `RestartSec=5`, `TimeoutStopSec=90`, `KillMode=mixed` (folded from the + red-team: the default `KillMode=control-group` SIGTERMs EVERY process in + the unit cgroup — server, runner, the dev-path postgres wrapper, and + depending on rootless podman's cgroup placement conmon — in PARALLEL, + bypassing the DL-183 reverse-order graceful teardown; `mixed` sends + SIGTERM to the main pid only and reserves the group SIGKILL for the + `TimeoutStopSec` deadline, composing correctly with the 90s grace), + explicit start-limit posture (`StartLimitIntervalSec=`/`StartLimitBurst=` + stated, not left to the 5-in-10s default — the intended circuit breaker + is chosen, not an accident of defaults), `Environment=PATH=…`, + `WantedBy=default.target`; NO `After=network.target` (it is a + system-manager unit the per-user manager silently ignores, and the + server binds loopback only, `main.go:30-31` — no network-up ordering + needed); install runs `systemctl --user daemon-reload` + `enable --now` + and checks/advises `loginctl enable-linger` (mandatory for boot-start: + rootless podman requires a real `XDG_RUNTIME_DIR`; the CLI's own + fallback `defaultRuntimeDir` is not a substitute for podman's, + `main.go:284-292`). + - launchd plist → `~/Library/LaunchAgents/com.rigelbuild.compass-stack.plist`: + `ProgramArguments` = ` up --supervise …`, `RunAtLoad=true`, + `KeepAlive={SuccessfulExit=false}`, `ExitTimeOut=90`, + `ThrottleInterval=` stated (launchd's restart floor, the macOS analogue + of `RestartSec`), `AbandonProcessGroup=true` (folded from the red-team: + launchd's default teardown SIGKILLs remaining process-group members at + job stop, which would race the DL-183 graceful drain — the plist must + let the children survive long enough for the ordered reverse teardown), + `EnvironmentVariables` PATH; install runs + `launchctl bootstrap gui/$UID` then `enable`. +- Operator stop-truth — the `down` verb vs the installed unit (folded from + the red-team): running `compass-stack down` against a service-supervised + stack kills the children out from under the supervise process, which + observes child death, drains, exits non-zero — and the OS supervisor + RESTARTS the stack the operator just stopped, so `down` becomes a + no-op-with-extra-steps and the only real stop verbs become + `systemctl --user stop` / `launchctl bootout`, contradicting DL-259's + one-verb surface. So `service install` makes `down` unit-aware: when an + installed unit is active, `down` stops it THROUGH the unit + (`systemctl --user stop` / `launchctl bootout`) so the supervise process + performs the teardown and exits ZERO (no restart), preserving + `compass-stack` as the stop surface. This is a fold, not a Matt fork. +- Unit-content gate: every item of the T2 checklist (former OQ-4, below) — + explicit `--state-dir`, absolute paths + PATH, `KillMode=mixed` / + `AbandonProcessGroup=true` ordered-teardown knobs, pinned `RestartSec` / + `ThrottleInterval` + start-limit posture, stop grace >= 90s, log routing + (journald / launchd `StandardErrorPath`), the linger step, the + status-truth doc caveat. +- Test cycle: unit tests on the template rendering (golden files: absolute + paths, the >= 90s stop grace, the pinned state dir, `KillMode=mixed` / + `AbandonProcessGroup=true`). Smoke on a real systemd host: install → unit + active + `compass-stack status` Ready → kill a child → unit enters + `failed`/restarts and the stack comes back → `systemctl --user stop` + produces the ORDERED reverse drain (runner exits before server, not a + parallel cgroup massacre — the `KillMode=mixed` assertion) with the pgid + record gone and sockets dark → `compass-stack down` while the unit is + active stops it through the unit and exits zero (no restart) → uninstall + removes the unit. The launchd leg smokes the same cycle on a darwin host + (install → running → crash-kill → KeepAlive restart → stop → uninstall); + it lands with T3 and is gated by OQ-5's end-to-end caveat. + +### T3 — darwin start-time identity reader + +Give the DL-183 identity token a darwin implementation so `up` no longer +refuses on macOS. **This is TWO swaps in two packages, not one** (folded +from the design-critic red-team): the token is read at spawn by the core's +`readStartTime` seam (`pgidfile.go:331-338`) AND, independently, at +teardown by the down-side identity check's own deliberately-duplicated +reader `readGroupLeaderStartTime` (`adapters/groupsignal.go:93-98`, which +"duplicates the core's parser ... because the parenthesized-comm gotcha is +the same on both sides"). `GroupSignaller.Alive` compares the two for +`uint64` equality (`groupsignal.go:90`, `got == startTime`), so on a single +OS the spawn-side and down-side darwin readers MUST produce the identical +encoding — the "encodings never need to agree" property holds ACROSS OSes +(Linux clock-ticks vs darwin timeval never compare) but is FALSE across the +two readers on one OS. A darwin `up` whose token the darwin `down` cannot +match would silently skip every live process child at teardown. So T3 adds +a darwin reader to BOTH seams sharing ONE encoding: `sysctl KERN_PROC` via +`golang.org/x/sys/unix` (already a module dependency, `go/go.mod:96`) +reading `KinfoProc`'s start `timeval`, packed as `sec*1e6 + usec` into the +`uint64` the pipeline already carries (`pgidEntry.StartTime uint64`, +`pgidfile.go:80-88`; `Alive`'s `startTime uint64`; serialized decimal — +correcting the record's earlier "opaque equality string": it is a `uint64`, +which darwin must pack a timeval into). Split each reader into +`_linux.go`/`_darwin.go` build-tagged files behind the unchanged seams. Interfaces: -- Consumes: this record (frozen on merge) and Matt's OQ-1 ruling. -- Produces: the RIG-3239 close-out state; no code artifacts. -- Test cycle: none (no code); markdownlint on the record is the gate. +- Consumes: the `readStartTime` var seam (`pgidfile.go:331-338`) AND the + down-side `readGroupLeaderStartTime` (`groupsignal.go:93-108`); the pgid + record grammar unchanged (proc entries carry whatever `uint64` the host's + reader produced at spawn). +- Produces: darwin readers for BOTH seams (four build-tagged files: + `readstarttime_{linux,darwin}.go` + the groupsignal `_{linux,darwin}.go` + split), sharing one darwin timeval-packing encoding; removal of the + "refuses on non-Linux unix" behavior (the seam comments update from + test-seam to cross-OS seam); no CLI surface change. +- Test cycle: the existing seam-stubbed tests keep passing unchanged; the + existing spawn-vs-down mirror-test pair (`groupsignal_test.go:77-84`, + which exists precisely because the duplication is load-bearing) EXTENDS + to the darwin readers so the two encodings cannot drift; a new + darwin-tagged unit test reads the test process's own start time twice + (stable, non-empty) and verifies a dead/mismatched pid fails the identity + check. Runs on the DL-263 darwin CI leg. The full macOS stack-up smoke is + OQ-5-gated (topology, not supervision). + +### T4 — Docs, ledger, close-out + +Ship the operator surface and record the ruling. + +Interfaces: + +- Consumes: T1-T3 landed; this record (frozen on merge). +- Produces: the self-host doc (DL-259's install surface) gains a "run as a + service" section documenting `compass-stack service install` for both + platforms, the linger prerequisite, the log locations, and the + status-truth caveat (`compass-stack status`, never + `systemctl --user status` / `launchctl print` alone); the DECISIONS.md + DL-328 row (rewritten to this ruling — the coordinator lands it with the + record); RIG-3239 close-out. +- Test cycle: markdownlint on the record + doc; the doc's command sequence + is walked once verbatim on a Linux host as part of the T2 smoke. ## Tasks -- [ ] T1 — self-host doc gains the systemd user-unit wrapper section - (copy-pasteable unit + linger + verbs), smoke-tested on a systemd host. -- [ ] T2 — close-out: ruling recorded, no code changes, ledger row deferred to - the main agent. +- [ ] T1 — `up --supervise` blocking mode: supervise loop on the stack core + (child-`Wait` fan-in, drain + non-zero on child death, signal-driven + `Down`), harness unit tests + Linux process smoke. +- [ ] T2 — `service install`/`uninstall` verbs + embedded systemd/launchd + unit templates, rendered against the T2 unit-content checklist; + golden-file tests + systemd-host smoke (launchd smoke with T3). +- [ ] T3 — darwin `readStartTime` (`sysctl KERN_PROC` via x/sys) behind the + existing var seam; darwin-tagged unit test on the DL-263 CI leg. +- [ ] T4 — self-host doc "run as a service" section (both platforms + + status-truth caveat), DL-328 ledger row, close-out. ## Open Questions -### OQ-1 [load-bearing, Matt fork] — adopt Quadlet, or keep the hand-rolled supervision? - -The central fork. **Recommendation: keep DL-183/DL-262 as the single -supervision mechanism; do not adopt per-service Quadlet units; add the T1 -documented systemd wrapper for boot-start.** Grounds: Quadlet is -Linux/systemd-only so the pgid path survives regardless (two models vs one); -DL-183/DL-262 are frozen, tested, and load-bearing; the cold sequence is -imperative (TLS anchor, token mint, readiness polls) and would need oneshot -pre-units + sdnotify re-plumbing; per-service Quadlet displaces the -DL-259-named `compass-stack up` entry point. Quadlet's genuine wins -(per-service `Restart=` crash recovery, journald, a standard operators know) -are real — if Matt weighs crash recovery heavily, the fork reopens (see OQ-3). -Matt rules. - -### OQ-2 [load-bearing, Matt fork] — is the T1 wrapper wanted at all? - -The wrapper is an add-on, not a dependency of the ruling. If Matt prefers the -self-host doc stay minimal ("run `compass-stack up` in a tmux/session and be -done"), T1 drops and RIG-3239 closes as a pure decision record. -**Recommendation: ship T1** — boot-start/logout-survival is the most common -real self-host ask and costs one doc section. - -### OQ-3 [load-bearing, Matt fork] — crash recovery: accept the gap, build a blocking `--supervise`, or reopen Quadlet? - -Neither today's mechanism nor the T1 oneshot wrapper restarts a crashed child: -`Health` is probe-on-demand (`stack.go:198-200`), and systemd sees the oneshot -wrapper as one unit that stays `active` after `up` returns, not four services. -Per-service `Restart=` is exactly what full Quadlet adoption would buy. Three -options, in ascending code cost: - -- **(a) Accept the gap (T1 oneshot wrapper as drafted).** A crashed stack - surfaces legibly via `compass-stack status` / the client's connection - failure, and the always-on-VPS steady state (DL-319) makes silent long-lived - crashes operator-visible. Zero code. **Recommendation for v1.** -- **(b) Build a blocking `compass-stack up --supervise` under a `Type=exec` - (or `Type=notify`) unit with `Restart=on-failure`.** The supervisor blocks - after Ready, watches its children, and exits non-zero on a child death; - systemd restarts the whole unit. This adds systemd-grade WHOLE-STACK crash - recovery with ONE supervision model (DL-183 spawn/teardown unchanged; systemd - supplies only restart/backoff) and simultaneously fixes the oneshot wrapper's - status-lie and stop-what-you-didn't-start edges (OQ-4). Whole-stack (not - per-service) restart is arguably the RIGHT granularity anyway — the cold - sequence's ordering means a restarted postgres needs the server's readiness - re-verified. Cost: a real code change (a blocking mode + child-exit watching, - where `up` abandons children by design today). -- **(c) Reopen Quadlet (OQ-1).** Per-service `Restart=`, at the cost of two - supervision models — the OQ-1 tradeoff. - -**Recommendation: (a) for v1, with (b) as the named follow-up if field reports -show crash recovery matters** — (b) dominates the originally-drafted -in-supervisor-loop shape and delivers recovery WITHOUT a second model, so it, -not Quadlet, is the escalation path. If Matt rates crash recovery as a LAUNCH -requirement, choose (b) now (or, if per-service granularity is required, (c) -reopens OQ-1). Matt rules. - -### OQ-4 [non-load-bearing] — wrapper failure modes + unit content, unverified until smoke-tested - -The T1 oneshot unit shape wrapping a returns-at-Ready CLI is standard systemd -practice but was NOT executed this session. The T1 doc MUST address these, -each grounded, and the T1 smoke cycle resolves them (non-load-bearing: none -changes the ruling, all are doc-completeness for the executor): - -- **Status lies (the RemainAfterExit trap).** With `RemainAfterExit=yes` the - unit reads `active` from the instant `up` returns until an explicit stop, so - a child crashing later is invisible to systemd. The doc MUST state the status - source of truth is `compass-stack status`, and that the unit's `active` means - only "up-once-succeeded". (Option (b) in OQ-3 eliminates this: a `Type=exec` - unit tracks a live main pid, so a dead stack is `failed`.) -- **Stop-what-you-didn't-start.** `up` attaches to an already-live stack rather - than failing (`stack.go:131-146` `upLocked` probe→attach), so a systemd - `ExecStart` succeeds against a stack a manual `up`/dev session started; a - later `stop`/unit-failure/shutdown then runs `ExecStop=compass-stack down`, - which tears down whatever the shared state-dir's pgid record names - (`downdetached.go:62-68` — per-state-dir identity-safe, but not per-invoker). - The unit MUST pin an explicit `--state-dir` so the systemd stack and any - interactive stack are distinct unless deliberately shared. -- **Environment + boot order.** systemd user units inherit no login-shell PATH, - so `ExecStart` needs an absolute `compass-stack` path AND the stack's own - `LookPath` children resolve (podman; the DL-321 PATH-threaded sidecars) — set - an explicit `Environment=PATH=` or `ExecSearchPath=`. Add `After=network.target` - (the TLS door bind); podman needs no daemon so no socket dependency - (`podman.go:23-24`). `loginctl enable-linger` is MANDATORY, not optional — - rootless podman hard-requires `XDG_RUNTIME_DIR`, which exists only in a - lingering or logged-in user session (`defaultRuntimeDir` falls back to - `/run` when it is unset, `main.go:284-290`, but podman itself - still needs the real runtime dir). -- **`TimeoutStopSec`.** `ExecStop=compass-stack down` can take up to the 80s - worst-case drain (`downdetached.go:15-24`: "sum to 65s; on the escalation - path … 15 + (30+5) + (10+5) + (10+5) = 80s"), so set `TimeoutStopSec=` >= 90s - or systemd SIGKILLs mid-teardown. - -### Resolved within this record - +### OQ-4 [non-load-bearing] — service-unit content checklist (the T2 gate) + +The unit shapes are standard systemd/launchd practice but were NOT executed +this session; the T2 smoke cycle resolves them. Each rendered unit MUST +carry, and the T2 tests MUST assert (non-load-bearing: none changes the +ruling, all are content-completeness for the executor): + +- **Pinned identity.** Explicit `--state-dir` in the rendered invocation, so + the service stack and any interactive stack are distinct unless + deliberately shared — `up` attaches to an already-live stack rather than + failing (`stack.go:131-146`), so an unpinned unit could adopt (and later + tear down) a stack a manual session started. +- **Environment + boot order.** systemd user units and launchd agents + inherit no login-shell PATH, so the `ExecStart`/`ProgramArguments` binary + path is absolute AND an explicit PATH is set (the stack's own `LookPath` + children — podman, the DL-321 sidecars — must resolve). Linux adds + `After=network.target` (the TLS door bind); podman needs no daemon so no + socket dependency (`podman.go:23-24`). `loginctl enable-linger` is + MANDATORY on Linux — rootless podman hard-requires a real + `XDG_RUNTIME_DIR`, which exists only in a lingering or logged-in session + (`defaultRuntimeDir` falls back to `/run` when unset, + `main.go:284-292`, but podman itself still needs the real runtime dir). + launchd covers boot via `RunAtLoad` with no linger analogue needed. +- **Stop grace.** Teardown can take the ~80s worst-case drain + (`downdetached.go:15-24`), so `TimeoutStopSec` / `ExitTimeOut` >= 90s or + the OS supervisor SIGKILLs mid-teardown. +- **Log + status routing.** Linux: journald via the unit + (`journalctl --user -u compass-stack`); macOS: explicit + `StandardOutPath`/`StandardErrorPath`. Both docs state the status source + of truth is `compass-stack status` — the unit's state is supervise-pid + liveness only. + +### OQ-5 [load-bearing, Matt fork] — macOS service: ship against podman-machine now, or gate GA on RIG-3238? + +The supervision mechanics for macOS are shippable in this record (T3 darwin +identity reader — a bounded seam swap; T2 launchd template), but a WORKING +macOS stack end-to-end still hangs on topology unknowns owned by the +sibling lanes: embedded-revival OQ-7 (the postgres DSN + agent sockets are +AF_UNIX bind-mounts, unvalidated across the podman-machine virtiofs/VM +boundary, `ui/compass-native-embedded-revival/design.md:909-931`) and +RIG-3238's OQ-12 (runner-on-darwin once podman-machine goes, +`platform/apple-container-macos-runner/design.md:713-733`). The fork: (a) +ship T2/T3's macOS support now, labeled experimental until the topology +validates, or (b) hold the macOS half of `service install` behind the +RIG-3238 lane's resolution and ship Linux-only first. **The cost option (a) +must carry (folded from the design-critic red-team):** an "experimental" +doc label does NOT stop a crashloop — on a mac where the socket topology +cannot work, `up --supervise` never reaches Ready, and launchd's +`KeepAlive={SuccessfulExit=false}` (which cannot tell a never-Ready +bring-up failure from a post-Ready crash) restarts it every ~10s forever, +burning CPU + podman-machine churn. So option (a) is only safe bundled with +a bounded-crashloop guard: the darwin `service install` runs an +install-time preflight (podman machine reachable + one probe cycle) and +refuses with a legible error until it passes, AND/OR the supervise loop +self-disables after N consecutive bring-up failures (a start-failure +backoff launchd's `SuccessfulExit` key cannot express). **Recommendation: +(a) with that preflight + self-limit** — build and land the darwin +mechanics now (they are small, testable on the DL-263 CI leg, and required +under EVERY macOS outcome, including apple-container), gate only the "macOS +supported" doc claim + GA smoke on the sibling lane's socket-topology +validation, and let Linux ship independently either way. This is dependency +ordering only — macOS-in-scope is ruled, not open. + +### Resolved decisions + +- **OQ-1 (Quadlet vs pgid) — RULED: keep pgid, decline Quadlet** (Matt, + 2026-09-04). The all-platforms requirement adds a third independent + ground: Quadlet is Linux/systemd-only and cannot express the macOS half + at all. See Approach. +- **OQ-2 (ship the wrapper?) — RULED: yes, upgraded** (Matt, 2026-09-04): + not an optional Linux-only doc snippet but a first-class one-command + `compass-stack service install` on all platforms, auto-start at reboot. + Verbatim: "we need to ship with a one command way to stand up the stack + as a constant-on service on all platforms". +- **OQ-3 (crash recovery) — RULED: build it** (Matt, 2026-09-04): the + accept-the-gap posture is rejected ("a VPS doesn't fix a crash … opt 1 + reasoning makes no sense"); the blocking `up --supervise` under the OS + supervisor's restart policy is the design (option (b) of the original + fork), keeping one supervision model. - Rootless under Quadlet: confirmed — rootless Quadlet runs as systemd USER units from the rootless unit search paths (podman stays daemonless); it is explicitly not achievable via `User=` in a system unit. Source: . - Moot under the recommended keep ruling, load-bearing only if OQ-1 flips. -- DL-259 squaring: Quadlet units are host-level systemd config, not - compose/Swarm packaging — no letter violation; but per-service units would - displace the `compass-stack up` verb DL-259 names, so full adoption would - need a DL-259 amendment. The T1 wrapper keeps the verb and needs none. + Moot under the keep ruling; recorded for the declined alternative. +- DL-259 squaring: systemd/launchd units are host-level configuration, not + compose/Swarm packaging — no letter violation; per-service Quadlet units + would have displaced the `compass-stack up` verb DL-259 names (an + amendment), but the ruled `service install` keeps `compass-stack` as the + verb surface and needs none. - Docker-socket: declined at the stack layer (daemon model vs the rootless/no-daemon hard requirement, no per-container keep-id equivalent), mirroring the embedded-revival OQ-9 rejection — see Alternatives. +- macOS in scope: ruled by Matt ("how do we support this on macOS as + well?") — only the dependency ordering (OQ-5) remains open, never + whether macOS is supported. From 215576ca9edfa0a68c2e1452bb8b59a83e14835a Mon Sep 17 00:00:00 2001 From: mintaka Date: Fri, 4 Sep 2026 18:27:08 -0400 Subject: [PATCH 3/9] =?UTF-8?q?docs(platform):=20fold=20PR=20#872=20review?= =?UTF-8?q?=20=E2=80=94=20lock-lifetime=20fork=20+=20consistency=20(RIG-32?= =?UTF-8?q?39)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fold the single-review-of-record (ReviewStackSupervision872) on the stack-supervision design: 1 high, 5 medium, 4 low. All load-bearing Go citations were re-verified at the PR head before folding. High (lock-lifetime self-deadlock) — a genuine load-bearing fork on the Matt-ruled DL-183 interlock, not a coordinator ruling. `up --supervise` is the first caller to hold the state-dir lockfile with a live pid for the stack's whole lifetime, so `DownDetached`'s live-holder guard (`downdetached.go:74-84`) permanently refuses `down` and T1's pre-spawn sweep self-refuses. Folded the two factual defects it exposed (the false T2 stop-truth premise; the T1 item-4 ordering) and captured the mechanism as OQ-6 (release-at-Ready vs mode-token guard), recommendation (b), routed to Matt via RIG-3261 alongside OQ-5. Added a Global Constraints lock-lifetime invariant. Medium — OQ-4 `After=network.target` contradiction removed to match the ruled position; `main.go:30-31` miscite corrected to `main.go:42-45` (`defaultListenAddr`); Approach reason-#1 and OQ-5 darwin-port sites widened from one-seam to two-seam; T1 container-supervision fork ruled (liveness poll via `ContainerController.Exists`, unbounded-`podman wait` rejected) and the Interfaces `Health`/`Exists` roles split; `### Resolved decisions` promoted to `##` so the Open-Questions freeze scope holds only OQ-4/OQ-5/OQ-6. Low — OQ-5/OQ-6 routed to RIG-3261 in the headings; `golang.org/x/sys` promotion indirect→direct noted with the `go.mod`/`go.sum` delta added to T3 Produces; DL-328 row relocated from `## Infrastructure & CI` to `## Desktop shell` (placement only, row text byte-identical); garbled "non-systemd-agnostic" Alternatives prose fixed. Additive review-fix commit atop the PR head; no `--supervise` mechanism was ruled — OQ-6 is Matt's. Ledger-impact: DL-328 relocated within DECISIONS.md (no text/status change) Refs RIG-3239 Co-authored-by: Matt Wilkinson --- docs/designs/DECISIONS.md | 2 +- .../compass-stack-supervision/design.md | 150 ++++++++++++++---- 2 files changed, 117 insertions(+), 35 deletions(-) diff --git a/docs/designs/DECISIONS.md b/docs/designs/DECISIONS.md index 2d6683be..c3457126 100644 --- a/docs/designs/DECISIONS.md +++ b/docs/designs/DECISIONS.md @@ -310,6 +310,7 @@ check enforces the mechanical half. Full rationale: | DL-319 | The Compass native app is DUAL-MODE again: `mode="embedded"` returns as the low-friction onboarding / local-dev front door — the app spawns/supervises a LOCAL stack via rootless podman on the user's own machine (macOS via podman machine, Linux native; Windows/WSL deferred) — ADDED ALONGSIDE the fully-surviving client mode, which stays first-class and is the RECOMMENDED steady-state for real self-host (always-on VPS/EC2 running `compass-stack up`, reached over TLS). Rationale is the trust model (DL-318): single-tenant operator-own-code needs no KVM isolation; podman is a permanent supported tier (microVM recommended where the host has KVM). Supersedes DL-235 (the "client is the ONLY mode" thesis — the exclusivity dies, the client surface survives whole) and the app-never-spawns half of DL-236 (whose standalone-`compass-stack` half stays Active and load-bearing); restores the dual-mode SHAPE of DL-106 by citation, and partial-supersedes DL-259's KVM-floor clause by citation (funnel entry 2's podman tier runs on a KVM-absent VPS) | Active (Matt, 2026-09-01) | [embedded-revival §A1](ui/compass-native-embedded-revival/design.md#a1--mode-selection-contract-dual-mode-returns-embedded-is-the-onboarding-default) | | DL-320 | app.toml is dual-mode: absent → embedded (the zero-config onboarding default returns); `mode="embedded"` accepts no server_url/ca_cert; `mode="client"` keeps the built contract (https-only server_url required, optional ca_cert, keychain-first bearer per DL-109); the `--mode`/`$COMPASS_APP_MODE` override returns (flag > env > file > default). Graduation embedded→client is a config edit documented in the self-host doc, not an in-app flow. Supersedes DL-237 | Active (Matt, 2026-09-01) | [embedded-revival §A1](ui/compass-native-embedded-revival/design.md#a1--mode-selection-contract-dual-mode-returns-embedded-is-the-onboarding-default) | | DL-321 | The app bundle carries embedded's sidecars again — `bin/{compass-app,compass-stack,compass-server,compass-runner}` + dist, PATH-threaded (DL-215's mechanism restored by citation) — but NO postgres tooling and no `compass-postgres` sidecar: the embedded stack's postgres is the DL-260 stock `postgres:18` container via rootless podman (the CLI's own default), leaving rootless podman (plus podman machine on macOS) as the packaged embedded mode's sole container prerequisite. DL-217 STAYS superseded. macOS stages the same four binaries in `Contents/MacOS/`. Supersedes DL-238 | Active (Matt, 2026-09-01) | [embedded-revival §A4](ui/compass-native-embedded-revival/design.md#a4--bundle-re-scope-sidecars-return-postgres-stays-a-container) | +| DL-328 | Self-host stack supervision KEEPS the hand-rolled DL-183/DL-262 pgid mechanism as the SINGLE cross-platform supervision model; per-service Podman Quadlet units are NOT adopted (Linux/systemd-only, so Quadlet structurally fails the ruled all-platforms bar AND the pgid path survives regardless — two models vs one; the imperative cold sequence would need oneshot pre-units + sdnotify re-plumbing; per-service units displace the DL-259-named `compass-stack up` verb). Whole-stack CRASH RECOVERY ships as a blocking `compass-stack up --supervise` foreground mode (up-to-Ready, watch children, non-zero exit on child death, teardown on signal) wrapped by the platform-native OS supervisor's restart policy, plus a one-command `compass-stack service install`/`uninstall` that writes + enables the native unit — systemd USER unit (`Type=exec`, `Restart=on-failure`, `TimeoutStopSec>=90`) on Linux, launchd LaunchAgent (`RunAtLoad`, `KeepAlive={SuccessfulExit=false}`, `ExitTimeOut>=90`) on macOS — auto-start at reboot + restart-on-crash; the OS supervisor supplies ONLY restart/backoff/boot-start (DL-183 spawn/teardown unchanged), status truth stays `compass-stack status`. Docker-socket declined at the stack layer (daemon model vs the rootless/no-daemon hard requirement; no per-container keep-id equivalent) | Active (Matt, 2026-09-04) | [stack supervision §Approach](platform/compass-stack-supervision/design.md#approach) | ## Agent batteries @@ -427,7 +428,6 @@ check enforces the mechanical half. Full rationale: | DL-299 | The `:vX.Y.Z` agent image is minted by digest re-tag (`skopeo copy`) of the newest already-published per-push `:git-` artifact at-or-before the release sha (resolved by walking first-parent ancestors for a GHCR-resolvable tag, never a path-based `git log`) — never a second ~90m closure build; `:latest` stays owned exclusively by the per-push publish job; the mint shares the `publish-agent-image` concurrency group under `queue: max` + `cancel-in-progress: false` so a non-superseding release mint is serialized, never dropped | Active (Matt, 2026-08-25) | [unified release lane](infra/release/compass-unified-release-lane/design.md) | | DL-300 | `publish-agent-image.yml` retires as a FILE with its DUTY relocated into the unified lane's per-push `publish-image` job (verbatim `publish.sh`, same closure-paths gate, same serialize/immutability/coherence/least-privilege/off-hot-path posture) via two-PR staging (land the fold, observe one green per-push publish, then delete) — accepted by the agent-image zone owner (compass-managed), who amend their record with a superseded-by pointer | Active (Matt, 2026-08-25) | [unified release lane](infra/release/compass-unified-release-lane/design.md) | | DL-301 | release-please authenticates with a scoped GitHub App installation token (contents + pull-requests, installation-scoped, per-run) so the release-please-opened Release PR receives `pull_request` CI and branch protection is a real merge gate — a bare `GITHUB_TOKEN`-opened PR gets none under GitHub's recursion guard | Active (Matt, 2026-08-25) | [unified release lane](infra/release/compass-unified-release-lane/design.md) | -| DL-328 | Self-host stack supervision KEEPS the hand-rolled DL-183/DL-262 pgid mechanism as the SINGLE cross-platform supervision model; per-service Podman Quadlet units are NOT adopted (Linux/systemd-only, so Quadlet structurally fails the ruled all-platforms bar AND the pgid path survives regardless — two models vs one; the imperative cold sequence would need oneshot pre-units + sdnotify re-plumbing; per-service units displace the DL-259-named `compass-stack up` verb). Whole-stack CRASH RECOVERY ships as a blocking `compass-stack up --supervise` foreground mode (up-to-Ready, watch children, non-zero exit on child death, teardown on signal) wrapped by the platform-native OS supervisor's restart policy, plus a one-command `compass-stack service install`/`uninstall` that writes + enables the native unit — systemd USER unit (`Type=exec`, `Restart=on-failure`, `TimeoutStopSec>=90`) on Linux, launchd LaunchAgent (`RunAtLoad`, `KeepAlive={SuccessfulExit=false}`, `ExitTimeOut>=90`) on macOS — auto-start at reboot + restart-on-crash; the OS supervisor supplies ONLY restart/backoff/boot-start (DL-183 spawn/teardown unchanged), status truth stays `compass-stack status`. Docker-socket declined at the stack layer (daemon model vs the rootless/no-daemon hard requirement; no per-container keep-id equivalent) | Active (Matt, 2026-09-04) | [stack supervision §Approach](platform/compass-stack-supervision/design.md#approach) | ## Test harness diff --git a/docs/designs/platform/compass-stack-supervision/design.md b/docs/designs/platform/compass-stack-supervision/design.md index e054b2a0..2e631b04 100644 --- a/docs/designs/platform/compass-stack-supervision/design.md +++ b/docs/designs/platform/compass-stack-supervision/design.md @@ -172,8 +172,10 @@ supervisor, not a container daemon. Quadlet needs systemd, and macOS has no systemd — so Quadlet cannot even express the macOS half of the requirement, while the pgid mechanism is portable (setpgid / SIGTERM / `kill(-pgid)` all exist on darwin; the - start-time identity read is a bounded per-OS swap behind the existing - `readStartTime` var seam, `pgidfile.go:331-338` — T3 below). This is a + start-time identity read is a bounded per-OS swap at TWO seams — the + spawn-side `readStartTime` (`pgidfile.go:331-338`) and the down-side + `readGroupLeaderStartTime` (`adapters/groupsignal.go:93-98`), which must + share one encoding — T3 below). This is a second, independent reason on top of the two below. 2. **Two supervision models vs one — the maintenance crux.** Even on Linux, adopting Quadlet means maintaining, testing, and keeping @@ -296,7 +298,8 @@ paths were examined: - **Native-darwin supervise (chosen).** The refusal lifts by giving the DL-183 identity token a darwin reader — `sysctl KERN_PROC` via - `golang.org/x/sys/unix` (already in the module, `go/go.mod:96`) reading + `golang.org/x/sys/unix` (already in the module graph as an indirect + dependency, `go/go.mod:96`, promoted to direct by this import) reading the `KinfoProc` start `timeval`. It is a bounded but TWO-site swap (T3 below): the token is read both at spawn (`readStartTime`, `pgidfile.go:331-338`) and independently at teardown @@ -375,7 +378,7 @@ same-tree anchor.) manual session started). - **Non-systemd process supervisors (s6 / runit / supervisord) — declined.** Dominated by the same two-models argument (none covers embedded macOS or - non-systemd-agnostic dev) plus a new third-party runtime dependency the + the non-systemd dev/devenv tier) plus a new third-party runtime dependency the self-host tier does not otherwise carry. - **`compass-stack up` as a thin facade over `systemctl` (declined).** Keep the DL-259 verb but have it generate + `systemctl --user` Quadlet units @@ -427,6 +430,15 @@ same-tree anchor.) identities are ever signaled; verify-before-signal; bounded escalation; survivor-rewrite on partial failure (`ui/compass-stack-cross-process-teardown/design.md:299-301`). +- Lock lifetime under `--supervise` (OQ-6, Matt fork): `up --supervise` is + the first caller to hold the state-dir lockfile with a LIVE pid for the + stack's whole lifetime, voiding the "up always exits, so the lock holder + is dead in every teardown" invariant `DownDetached`'s live-holder guard + relies on (`pgidfile.go:90-94`, `downdetached.go:74-84`). Whichever OQ-6 + option lands, `compass-stack down` MUST remain a real stop of a + supervised stack (DL-259) and the T1 pre-spawn sweep MUST run before the + lock is acquired (else it self-refuses). No `--supervise` deliverable + ships a wedged `down`. - Stop grace: `TimeoutStopSec` (systemd) / `ExitTimeOut` (launchd) >= 90s — the DL-262 teardown drain worst case is ~80s (`downdetached.go:15-24`: "15 + (30+5) + (10+5) + (10+5) = 80s"). @@ -476,9 +488,14 @@ contract facts).** death → drains the whole healthy stack and exits non-zero → OS supervisor restarts → a guaranteed whole-stack restart every ~2 minutes. So supervise does NOT Wait container children through the bounded adapter: - it supervises them by liveness poll (`podman container exists` / Health) - on the same interval, treating only a real exit / "no such container" as - death, OR via an unbounded `podman wait` path that re-invokes on timeout. + it supervises them by liveness POLL — `ContainerController.Exists` + (`deps.go:191-192`) on the same interval as the attached-Health poll, + treating only a real exit / "no such container" (N consecutive) as + death. The alternative — an unbounded `podman wait` path that re-invokes + on timeout — is REJECTED: it would mutate the adapter's `Wait` timeout + contract that `DownDetached`'s bounded drain also depends on, coupling + supervision to teardown; the poll adds a loop over an existing seam and + changes no contract. 3. **Context discipline (else SIGTERM hard-kills the group before graceful teardown).** `Process.Wait`'s cancellation path escalates to `syscall.Kill(-pid, SIGKILL)` of the whole group (`process.go:152-169`). @@ -499,10 +516,16 @@ contract facts).** it — DL-183 forbids signaling any pgid not read from the current record), and a half-dead old server still holding the socket can wedge the new spawn into a bind-fail restart loop. So supervise mode gains a pre-spawn - step: before (re)spawning, run the `downdetached` record-consuming + step: BEFORE acquiring the state-dir lock (else `DownDetached` observes + the supervise process's own live lockfile pid and returns + `ErrStackStarting` against itself — the same OQ-6 live-holder guard — and + the sweep silently never runs), run the `downdetached` record-consuming cleanup against any existing `stack.pgids` (identity-checked, so a post-reboot stale record is a safe no-op, `groupsignal.go:60-70`), or refuse to spawn while a record with a live identity-checked entry exists. + (Under OQ-6 option (b) the guard could instead learn to exempt the + caller's own pid; the before-lock ordering is the option-independent + fix.) Division of labor is unchanged: the stack keeps its single supervision model (DL-183 spawn order, identity-token teardown, drain); the OS @@ -521,10 +544,14 @@ Interfaces: - Consumes: `stack.Up` / `Stack.Down` (`stack.go:94,183`), the per-child `Wait(ctx)` process-handle contract (`deps.go:136-138`), `Stack.Health` - (`stack.go:198-211`) for the attached-stack degradation and container - liveness, and the `downdetached` record-consuming cleanup for the - pre-spawn step — plus the ONE core change above (sole-Wait-ownership or an - idempotent/shared `Wait`). + (`stack.go:201-211`) for ATTACHED-stack degradation only (it is a single + server-socket probe — `s.deps.Prober.Probe`, `stack.go:202` — nothing + more; a dead postgres behind a still-answering server reads Ready), + `ContainerController.Exists` (`deps.go:191-192`) for container-child + liveness (widening that seam's documented role from teardown-only to + teardown-plus-supervision), and the `downdetached` record-consuming + cleanup for the pre-spawn step — plus the ONE core change above + (sole-Wait-ownership or an idempotent/shared `Wait`). - Produces: a `--supervise` flag on the `up` verb in `go/cmd/compass-stack/main.go` plus a supervise loop in `internal/stack` (a method on `*Stack`, e.g. `Supervise(ctx) error`): fan-in the owned @@ -580,7 +607,7 @@ Interfaces: is chosen, not an accident of defaults), `Environment=PATH=…`, `WantedBy=default.target`; NO `After=network.target` (it is a system-manager unit the per-user manager silently ignores, and the - server binds loopback only, `main.go:30-31` — no network-up ordering + server binds loopback only, `main.go:42-45` — no network-up ordering needed); install runs `systemctl --user daemon-reload` + `enable --now` and checks/advises `loginctl enable-linger` (mandatory for boot-start: rootless podman requires a real `XDG_RUNTIME_DIR`; the CLI's own @@ -597,17 +624,28 @@ Interfaces: `EnvironmentVariables` PATH; install runs `launchctl bootstrap gui/$UID` then `enable`. - Operator stop-truth — the `down` verb vs the installed unit (folded from - the red-team): running `compass-stack down` against a service-supervised - stack kills the children out from under the supervise process, which - observes child death, drains, exits non-zero — and the OS supervisor - RESTARTS the stack the operator just stopped, so `down` becomes a - no-op-with-extra-steps and the only real stop verbs become - `systemctl --user stop` / `launchctl bootout`, contradicting DL-259's - one-verb surface. So `service install` makes `down` unit-aware: when an - installed unit is active, `down` stops it THROUGH the unit - (`systemctl --user stop` / `launchctl bootout`) so the supervise process - performs the teardown and exits ZERO (no restart), preserving - `compass-stack` as the stop surface. This is a fold, not a Matt fork. + the red-team, and corrected for the OQ-6 lock-lifetime fork): a naive + `compass-stack down` against a service-supervised stack does NOT reach the + children — because `up --supervise` holds the state-dir lockfile with a + LIVE pid for its whole lifetime, `DownDetached` step 1 hits the + live-holder guard and returns `ErrStackStarting` + (`downdetached.go:74-84`), rendered as "a stack is starting; retry once it + is up" (`main.go:406-409`) — a guard that never clears while supervise + runs. OQ-6 (below) decides how `down` gets a real path to a supervised + stack: option (b)'s mode-token guard makes a supervising holder a valid + `down` target that is signalled (it tears down and exits ZERO, no + restart); option (a) releases the lock at Ready so `down`'s guard passes + and it reads the record. On top of whichever OQ-6 lands, `service install` + makes `down` UNIT-AWARE: when an installed unit is active, `down` stops it + THROUGH the unit (`systemctl --user stop` / `launchctl bootout`) so the + supervise process performs the ordered teardown and exits ZERO (no + restart from the OS supervisor), preserving `compass-stack` as the single + stop surface DL-259 names — rather than the operator having to know + `systemctl --user stop` / `launchctl bootout`. The unit-aware `down` is a + fold; the underlying lock-lifetime mechanism it rides on is OQ-6, a Matt + fork. For a bare (unit-less) `up --supervise` — dev/devenv, and T1's + standalone Linux smoke before `service install` exists — `down` works + once OQ-6 is ruled and not before; T1's smoke asserts the OQ-6 behavior. - Unit-content gate: every item of the T2 checklist (former OQ-4, below) — explicit `--state-dir`, absolute paths + PATH, `KillMode=mixed` / `AbandonProcessGroup=true` ordered-teardown knobs, pinned `RestartSec` / @@ -644,7 +682,9 @@ encoding — the "encodings never need to agree" property holds ACROSS OSes two readers on one OS. A darwin `up` whose token the darwin `down` cannot match would silently skip every live process child at teardown. So T3 adds a darwin reader to BOTH seams sharing ONE encoding: `sysctl KERN_PROC` via -`golang.org/x/sys/unix` (already a module dependency, `go/go.mod:96`) +`golang.org/x/sys/unix` (already in the module graph as an indirect +dependency, `go/go.mod:96`, promoted to direct by this import — no new +module) reading `KinfoProc`'s start `timeval`, packed as `sec*1e6 + usec` into the `uint64` the pipeline already carries (`pgidEntry.StartTime uint64`, `pgidfile.go:80-88`; `Alive`'s `startTime uint64`; serialized decimal — @@ -660,9 +700,11 @@ Interfaces: reader produced at spawn). - Produces: darwin readers for BOTH seams (four build-tagged files: `readstarttime_{linux,darwin}.go` + the groupsignal `_{linux,darwin}.go` - split), sharing one darwin timeval-packing encoding; removal of the - "refuses on non-Linux unix" behavior (the seam comments update from - test-seam to cross-OS seam); no CLI surface change. + split), sharing one darwin timeval-packing encoding; the `go/go.mod` + (and `go.sum`) delta from `go mod tidy` promoting `golang.org/x/sys` out + of the indirect block; removal of the "refuses on non-Linux unix" + behavior (the seam comments update from test-seam to cross-OS seam); no + CLI surface change. - Test cycle: the existing seam-stubbed tests keep passing unchanged; the existing spawn-vs-down mirror-test pair (`groupsignal_test.go:77-84`, which exists precisely because the duplication is load-bearing) EXTENDS @@ -719,9 +761,11 @@ ruling, all are content-completeness for the executor): - **Environment + boot order.** systemd user units and launchd agents inherit no login-shell PATH, so the `ExecStart`/`ProgramArguments` binary path is absolute AND an explicit PATH is set (the stack's own `LookPath` - children — podman, the DL-321 sidecars — must resolve). Linux adds - `After=network.target` (the TLS door bind); podman needs no daemon so no - socket dependency (`podman.go:23-24`). `loginctl enable-linger` is + children — podman, the DL-321 sidecars — must resolve). NO + `After=network.target` (it is a system-manager unit the per-user manager + silently ignores, and the server binds loopback only, + `main.go:42-45` — no network-up ordering needed); podman needs no daemon + so no socket dependency (`podman.go:23-24`). `loginctl enable-linger` is MANDATORY on Linux — rootless podman hard-requires a real `XDG_RUNTIME_DIR`, which exists only in a lingering or logged-in session (`defaultRuntimeDir` falls back to `/run` when unset, @@ -736,10 +780,10 @@ ruling, all are content-completeness for the executor): of truth is `compass-stack status` — the unit's state is supervise-pid liveness only. -### OQ-5 [load-bearing, Matt fork] — macOS service: ship against podman-machine now, or gate GA on RIG-3238? +### OQ-5 [load-bearing, Matt fork — tracked RIG-3261] — macOS service: ship against podman-machine now, or gate GA on RIG-3238? The supervision mechanics for macOS are shippable in this record (T3 darwin -identity reader — a bounded seam swap; T2 launchd template), but a WORKING +identity reader — a bounded two-seam swap; T2 launchd template), but a WORKING macOS stack end-to-end still hangs on topology unknowns owned by the sibling lanes: embedded-revival OQ-7 (the postgres DSN + agent sockets are AF_UNIX bind-mounts, unvalidated across the podman-machine virtiofs/VM @@ -767,7 +811,45 @@ supported" doc claim + GA smoke on the sibling lane's socket-topology validation, and let Linux ship independently either way. This is dependency ordering only — macOS-in-scope is ruled, not open. -### Resolved decisions +### OQ-6 [load-bearing, Matt fork — tracked RIG-3261] — `--supervise` lock lifetime: release the lock at Ready, or teach the down-guard a supervising holder? + +`up --supervise` is the FIRST caller to hold the state-dir lockfile with a +LIVE pid for the stack's whole lifetime, voiding the invariant the teardown +path relies on: "up always exits after a successful spawn, so the writer +pid is dead in every linger teardown" (`pgidfile.go:90-94`). `upLocked` +returns a spawning `*Stack` still HOLDING the lock (`stack.go:151`; released +only on the attach path or by `Down`), the lockfile carries `os.Getpid()` +(`lockfile.go:124`), and `DownDetached` step 1 refuses outright when that +holder is live — `if live { return ErrStackStarting }` +(`downdetached.go:74-84`), rendered as "a stack is starting; retry once it +is up" (`main.go:406-409`). So while a bare `up --supervise` runs, +`compass-stack down` is permanently wedged behind that guard and never +reaches the record. The fork — how the supervise process reconciles holding +the lock for its lifetime with `down` staying a real stop verb (DL-259): + +- **(a) Release the O_EXCL lockfile once Ready is reached.** Restores the + linger-ownership shape `DownDetached` already expects (a dead/absent + holder → guard passes → `down` reads the record and signals the + children); the supervise process's teardown authority then comes from its + held child handles, not the lock. Cost: forfeits the mutual exclusion the + live lock gives — a second `up` could spawn a DUPLICATE stack while the + first supervises, a regression to the DL-183 single-owner model. +- **(b) Teach `DownDetached`'s live-holder guard a mode/state token.** The + lockfile records whether the holder is a mid-bring-up `up` (refuse, as + today) or a supervise process parked at Ready (a VALID `down` target that + is signalled, not refused). Preserves single-owner mutual exclusion. Cost: + a real semantic change to the frozen DL-183 interlock — the guard, the + lockfile format, and their invariants. + +**Recommendation: (b)** — it keeps DL-183's single-owner guarantee, which +(a) trades away, and makes `down` a first-class stop of a supervised stack +rather than leaning on T2's install-time `down`-unit-awareness to mask a +wedged guard. T1 item 4 and the T2 operator-stop-truth bullet are written +against (b); if Matt rules (a), both fold to the release-at-Ready shape. +This modifies a Matt-ruled safety contract (DL-183), so it is his call, not +a coordinator fold. + +## Resolved decisions - **OQ-1 (Quadlet vs pgid) — RULED: keep pgid, decline Quadlet** (Matt, 2026-09-04). The all-platforms requirement adds a third independent From 6d1861ec5649667f5b504dd801df4a23273df03e Mon Sep 17 00:00:00 2001 From: mintaka Date: Fri, 4 Sep 2026 18:53:46 -0400 Subject: [PATCH 4/9] =?UTF-8?q?docs(platform):=20fold=20PR=20#872=20re-rev?= =?UTF-8?q?iew=20=E2=80=94=20Approach/T-cross-ref=20OQ-6=20consistency=20(?= =?UTF-8?q?RIG-3239)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ReviewStackSupervision872r2 confirmed all 10 prior findings closed and the freeze gate PASS, but flagged three new mediums of the summary-vs-detail drift class: the prior fold corrected the OQ-6 lock-lifetime premise at its detail sites (T1 item 4, T2 stop-truth, Global Constraints) but left two upstream Approach summaries and one T2->T1 cross-reference carrying the pre-fold, now-falsified claims. Fold reconciles them: - Approach :340 counted "the one remaining fork" (OQ-5) while Open Questions now holds two load-bearing forks; names both OQ-5 and OQ-6. - Approach :283-285 justified unit-aware `down` by a "undone by a restart" rationale the T2 fold already retracted; rewritten to the live-holder-guard model and flags the OQ-6 dependency. - T2 stop-truth :648 delegated the OQ-6 behavior to "T1's smoke", which had no such leg; added T1 test-cycle item (5) (`down` against a supervised stack exits ZERO, never `ErrStackStarting`) and repointed the cross-ref. - L1: T4 T3 tasks row still singular ("readStartTime"); widened to both darwin seams (`readStartTime` + `readGroupLeaderStartTime`, one shared encoding), matching the two-seam prose. L2 (two ragged line-wraps, render-correct, no dropped text) deferred — not churning a freeze-ready record for cosmetics. Additive fold child off the bookmark tip; no DECISIONS.md change (DL-328 unchanged). Gates: markdownlint-cli2 0 errors (192 files); design-ledger-gate OK (292 rows / 122 headers). Refs RIG-3239 Co-authored-by: Matt Wilkinson --- .../compass-stack-supervision/design.md | 23 ++++++++++++------- 1 file changed, 15 insertions(+), 8 deletions(-) diff --git a/docs/designs/platform/compass-stack-supervision/design.md b/docs/designs/platform/compass-stack-supervision/design.md index 2e631b04..2860b3f7 100644 --- a/docs/designs/platform/compass-stack-supervision/design.md +++ b/docs/designs/platform/compass-stack-supervision/design.md @@ -281,7 +281,8 @@ the unit template does NOT let the OS pre-empt it: systemd's default the children in parallel before the ordered drain, so the templates pin `KillMode=mixed` / `AbandonProcessGroup=true` (T2). The operator's `compass-stack down` verb also becomes unit-aware so it stops through the -unit rather than being undone by a restart (T2). Status truth stays +unit (T2) — but a bare `up --supervise` wedges `down` behind the +live-holder guard until OQ-6 is ruled (below). Status truth stays `compass-stack status` (the unit's state is process-liveness, not stack health); the T2 checklist (former OQ-4, below) is the unit content gate. @@ -336,7 +337,8 @@ darwin answer yet). Those are stack-TOPOLOGY unknowns, not supervision unknowns — this record ships the supervision mechanics for macOS (T2 launchd template + T3 darwin identity reader) and leaves the topology validation with the lane that owns it. Whether macOS `service install` -GA-gates on that lane is OQ-5 (the one remaining fork, below). (The +GA-gates on that lane is OQ-5 (one of two remaining forks, below — the +other is OQ-6, the `--supervise` lock lifetime). (The `apple-container-macos-runner/design.md:713-733` line range cited above resolves only once that sibling RIG-3238 record lands on main — PR #869 is not yet merged; the citation is a forward reference, not a @@ -573,9 +575,13 @@ Interfaces: non-cancelable ctx (no group SIGKILL before graceful teardown); (3) attached supervise exits non-nil only after N consecutive health-probe failures, not one; (4) a partial-drain child death → restart → the - surviving old child is torn down by the pre-spawn cleanup, not orphaned. - Plus a process-level smoke on Linux: `up --supervise`, `kill` a child, - assert non-zero exit and a clean survivor teardown. + surviving old child is torn down by the pre-spawn cleanup, not orphaned; + (5) with `up --supervise` running, `compass-stack down` performs a real + stop (per the ruled OQ-6 option) and the supervise process exits ZERO, + never `ErrStackStarting` — the OQ-6 behavior T2's stop-truth delegates + here; it lands with the OQ-6 ruling. Plus a process-level smoke on Linux: + `up --supervise`, `kill` a child, assert non-zero exit and a clean + survivor teardown. ### T2 — `compass-stack service install` / `uninstall` + unit templates @@ -645,7 +651,7 @@ Interfaces: fold; the underlying lock-lifetime mechanism it rides on is OQ-6, a Matt fork. For a bare (unit-less) `up --supervise` — dev/devenv, and T1's standalone Linux smoke before `service install` exists — `down` works - once OQ-6 is ruled and not before; T1's smoke asserts the OQ-6 behavior. + once OQ-6 is ruled and not before; T1's test cycle item (5) covers it. - Unit-content gate: every item of the T2 checklist (former OQ-4, below) — explicit `--state-dir`, absolute paths + PATH, `KillMode=mixed` / `AbandonProcessGroup=true` ordered-teardown knobs, pinned `RestartSec` / @@ -739,8 +745,9 @@ Interfaces: - [ ] T2 — `service install`/`uninstall` verbs + embedded systemd/launchd unit templates, rendered against the T2 unit-content checklist; golden-file tests + systemd-host smoke (launchd smoke with T3). -- [ ] T3 — darwin `readStartTime` (`sysctl KERN_PROC` via x/sys) behind the - existing var seam; darwin-tagged unit test on the DL-263 CI leg. +- [ ] T3 — darwin start-time readers at BOTH seams (`readStartTime` + + `readGroupLeaderStartTime`, `sysctl KERN_PROC` via x/sys, one shared + encoding); darwin-tagged unit test on the DL-263 CI leg. - [ ] T4 — self-host doc "run as a service" section (both platforms + status-truth caveat), DL-328 ledger row, close-out. From 93933c5bdf8b82308d820a92ca47d7d3e0bdfa27 Mon Sep 17 00:00:00 2001 From: mintaka Date: Fri, 4 Sep 2026 19:18:46 -0400 Subject: [PATCH 5/9] =?UTF-8?q?docs(platform):=20fold=20PR=20#872=20re-rev?= =?UTF-8?q?iew=20r3=20=E2=80=94=20item=20(5)=20option-discrimination=20+?= =?UTF-8?q?=20arity=20(RIG-3239)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ReviewStackSupervision872r3 confirmed all 3 prior mediums + L1 genuinely closed at both detail and summary sites, but caught two live gating mediums of the recurring summary-vs-detail drift class in the freeze candidate: - M1 (new, from the r2 fold): T1 test-cycle item (5) stated an unconditional "exits ZERO", true only under OQ-6 option (b), contradicting the option-discriminating T2 stop-truth bullet and T1's own child-death non-zero exit contract. Rewrote item (5) to be option-agnostic — never wedges on ErrStackStarting and the stop lands with the ruling; under (b) the supervising holder is signalled and exits ZERO, under (a) the lock is released so down signals the children and the child-death path applies. Anchored it in-process to the OQ-6 counterpart of downdetached_test.go:238 (verified at source). Being option-agnostic, it no longer belongs in the OQ-6 (b)-dependent inventory (:854), closing M1's second site. - M2 (pre-existing, survived all three folds): the bolded "three interacting contract facts" summary counted a four-item list; item 4 (pre-spawn survivor cleanup) is the most cross-referenced fact (Global Constraint + OQ-6-(b) site), so the miscount is load-bearing. Fixed the count to four. - L1 (folded into M1's edit): item (5) now disambiguates the in-process harness assertion from the Linux process smoke. Gates: markdownlint-cli2 0 errors (192 files); design-ledger-gate OK (292 rows / 122 headers). Diff: design.md only, 11 insertions / 7 deletions. Ledger-impact: none (DL-328 cell unchanged) Refs RIG-3239 Co-authored-by: Matt Wilkinson --- .../compass-stack-supervision/design.md | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/docs/designs/platform/compass-stack-supervision/design.md b/docs/designs/platform/compass-stack-supervision/design.md index 2860b3f7..5ded8387 100644 --- a/docs/designs/platform/compass-stack-supervision/design.md +++ b/docs/designs/platform/compass-stack-supervision/design.md @@ -462,7 +462,7 @@ to Ready, then blocks watching the stack; a child death exits non-zero (after draining the survivors); SIGTERM/SIGINT runs `Down` and exits zero. **T1 requires a bounded change to the stack core — NOT "the core -unchanged" (folded from the design-critic red-team, three interacting +unchanged" (folded from the design-critic red-team, four interacting contract facts).** 1. **`Process.Wait` is single-caller, so a fan-in Wait plus drain @@ -576,12 +576,16 @@ Interfaces: attached supervise exits non-nil only after N consecutive health-probe failures, not one; (4) a partial-drain child death → restart → the surviving old child is torn down by the pre-spawn cleanup, not orphaned; - (5) with `up --supervise` running, `compass-stack down` performs a real - stop (per the ruled OQ-6 option) and the supervise process exits ZERO, - never `ErrStackStarting` — the OQ-6 behavior T2's stop-truth delegates - here; it lands with the OQ-6 ruling. Plus a process-level smoke on Linux: - `up --supervise`, `kill` a child, assert non-zero exit and a clean - survivor teardown. + (5) `compass-stack down` against a supervised (Ready) stack never wedges + on `ErrStackStarting` and the stop lands with the OQ-6 ruling — asserted + in-process against a stubbed Ready-supervised lock holder (the OQ-6 + counterpart to `downdetached_test.go:238`, which today asserts refusal + for an up-in-flight holder): under (b) `DownDetached` signals the + supervising holder and it exits ZERO; under (a) the lock is already + released so `down` signals the children directly and the supervise loop's + child-death path applies. This is the OQ-6 behavior T2's stop-truth + delegates here. Plus a process-level smoke on Linux: `up --supervise`, + `kill` a child, assert non-zero exit and a clean survivor teardown. ### T2 — `compass-stack service install` / `uninstall` + unit templates From 3b1c04b0304a697ae56ff6a9d73f3da2bec7c83f Mon Sep 17 00:00:00 2001 From: mintaka Date: Fri, 4 Sep 2026 19:42:49 -0400 Subject: [PATCH 6/9] =?UTF-8?q?docs(platform):=20fold=20PR=20#872=20re-rev?= =?UTF-8?q?iew=20r4=20=E2=80=94=20OQ-6=20(b)-dependent=20inventory=20+=20r?= =?UTF-8?q?eason=20ordinal=20(RIG-3239)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ReviewStackSupervision872r4 confirmed the r3 fold clean (M1 item-(5) option-discrimination, M2 arity 3->4, L1 harness-vs-process all genuinely closed at detail AND summary sites; no fold-introduced contradiction; freeze gate PASS; new downdetached_test.go:238 citation verified reachable at source), and caught two pre-existing instances of the summary-vs-detail drift class the fold never touched: - Medium (:858-859): the OQ-6 Recommendation's "if Matt rules (a), rewrite these" inventory named T1 item 4 and the T2 stop-truth bullet as (b)-dependent; neither is. T1 item 4's own parenthetical (:528-530) names (b) as the DECLINED alternative and calls the before-lock sweep "the option-independent fix", and Global Constraint :439-442 mandates the sweep "Whichever OQ-6 option lands" as a hard MUST — so the inventory instructed an executor who receives ruling (a) to fold away a requirement that survives (a) unchanged. The T2 bullet (:645-648) is option-discriminating and already carries both (a)/(b) branches. Second-order: the r3 fold made item (5) option-agnostic too, so the record now has ZERO genuinely (b)-dependent sites -> rewrote the sentence to state every T1/T2 site is option-agnostic and a ruling of (a) needs only deletion of moot (b) branches, not a rewrite (per the reviewer's suggested fix; NOT re-listed, which would re-falsify the M1 fix). - Low (:178-179): the first "keep pgid" reason called itself a "second, independent reason on top of the two below" (1 on top of 2 = third); the Resolved-decisions row for the same argument (:867) already says "third". Aligned to "third". Both folded coordinator-direct (judgment-dense frozen-record consistency). Gates: markdownlint-cli2 0 errors (192 files); design-ledger-gate OK (292 rows / 122 headers). Diff: design.md only, 9 ins / 3 del. Refs RIG-3239 Ledger-impact: none (DECISIONS.md unchanged) Co-authored-by: Matt Wilkinson --- .../platform/compass-stack-supervision/design.md | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/docs/designs/platform/compass-stack-supervision/design.md b/docs/designs/platform/compass-stack-supervision/design.md index 5ded8387..9428d456 100644 --- a/docs/designs/platform/compass-stack-supervision/design.md +++ b/docs/designs/platform/compass-stack-supervision/design.md @@ -176,7 +176,7 @@ supervisor, not a container daemon. spawn-side `readStartTime` (`pgidfile.go:331-338`) and the down-side `readGroupLeaderStartTime` (`adapters/groupsignal.go:93-98`), which must share one encoding — T3 below). This is a - second, independent reason on top of the two below. + third, independent reason on top of the two below. 2. **Two supervision models vs one — the maintenance crux.** Even on Linux, adopting Quadlet means maintaining, testing, and keeping behaviorally-equivalent TWO supervision models (unit-file @@ -855,8 +855,14 @@ the lock for its lifetime with `down` staying a real stop verb (DL-259): **Recommendation: (b)** — it keeps DL-183's single-owner guarantee, which (a) trades away, and makes `down` a first-class stop of a supervised stack rather than leaning on T2's install-time `down`-unit-awareness to mask a -wedged guard. T1 item 4 and the T2 operator-stop-truth bullet are written -against (b); if Matt rules (a), both fold to the release-at-Ready shape. +wedged guard. Every T1/T2 site is already written to be +option-agnostic: T1 item 4's before-lock sweep is the +option-independent fix (Global Constraint — the sweep MUST run before +the lock is acquired whichever OQ-6 option lands), the T2 +operator-stop-truth bullet carries both (a)/(b) branches explicitly, +and T1 test-cycle item (5) states its (a) and (b) shapes — so a +ruling of (a) needs no rewrite, only deletion of the now-moot (b) +branches. This modifies a Matt-ruled safety contract (DL-183), so it is his call, not a coordinator fold. From 75fe8465751c4b4ebb01f58e7d1a531529e494e5 Mon Sep 17 00:00:00 2001 From: mintaka Date: Sat, 5 Sep 2026 19:45:39 -0400 Subject: [PATCH 7/9] =?UTF-8?q?docs(platform):=20freeze=20stack-supervisio?= =?UTF-8?q?n=20=E2=80=94=20fold=20OQ-5/OQ-6=20rulings=20(RIG-3239)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Matt ruled both remaining forks on RIG-3261/RIG-3239: - OQ-6 → (b): teach `DownDetached`'s live-holder guard a mode/state token — a supervise process parked at Ready is a valid `down` target (signalled, exits zero), a mid-bring-up `up` is still refused. Preserves DL-183 single-owner mutual exclusion that option (a) would have traded away. - OQ-5 → ship macOS supervision now, runtime-agnostic: the launchd LaunchAgent supervises the `compass-stack` host process independent of the runtime inside it (podman today, apple-container later), so macOS supervision is NOT gated on the RIG-3238 backend choice. The OQ-5-as-filed podman-machine-vs-RIG-3238 fork dissolves; the residual macOS GA gate is the stack-topology validation owned by the sibling lane, not a supervision fork. Both folded into Resolved decisions; the (a)/(b) fork prose collapses to the ruled (b) across Approach / Global Constraints / T1 / T2 / T3; Open Questions now holds only OQ-4 (non-load-bearing content checklist). Status: Draft → Active. Ledger-impact: none (DL-328 already captures the supervision decision; the OQ-6 guard-token is the internal mechanism DL-328's "DL-183 spawn/teardown unchanged" already anticipates). Spec-impact: none. Refs RIG-3239 Co-authored-by: Matt Wilkinson --- .../compass-stack-supervision/design.md | 192 +++++++----------- 1 file changed, 75 insertions(+), 117 deletions(-) diff --git a/docs/designs/platform/compass-stack-supervision/design.md b/docs/designs/platform/compass-stack-supervision/design.md index 9428d456..bf0da53d 100644 --- a/docs/designs/platform/compass-stack-supervision/design.md +++ b/docs/designs/platform/compass-stack-supervision/design.md @@ -1,6 +1,6 @@ # Compass self-host stack supervision: constant-on cross-platform service -Status: Draft +Status: Active (Matt, 2026-09-05) Issue: RIG-3239 ## Problem / Intent @@ -281,8 +281,9 @@ the unit template does NOT let the OS pre-empt it: systemd's default the children in parallel before the ordered drain, so the templates pin `KillMode=mixed` / `AbandonProcessGroup=true` (T2). The operator's `compass-stack down` verb also becomes unit-aware so it stops through the -unit (T2) — but a bare `up --supervise` wedges `down` behind the -live-holder guard until OQ-6 is ruled (below). Status truth stays +unit (T2) — for a bare `up --supervise`, OQ-6→(b)'s mode-token guard +makes the supervising holder a valid `down` target (Resolved +decisions). Status truth stays `compass-stack status` (the unit's state is process-liveness, not stack health); the T2 checklist (former OQ-4, below) is the unit content gate. @@ -336,13 +337,19 @@ which also notes the runner's podman host-capability preflight has no darwin answer yet). Those are stack-TOPOLOGY unknowns, not supervision unknowns — this record ships the supervision mechanics for macOS (T2 launchd template + T3 darwin identity reader) and leaves the topology -validation with the lane that owns it. Whether macOS `service install` -GA-gates on that lane is OQ-5 (one of two remaining forks, below — the -other is OQ-6, the `--supervise` lock lifetime). (The -`apple-container-macos-runner/design.md:713-733` line range cited above -resolves only once that sibling RIG-3238 record lands on main — PR #869 is -not yet merged; the citation is a forward reference, not a -same-tree anchor.) +validation with the lane that owns it. Per Matt's RIG-3239 ruling the +launchd LaunchAgent supervises the `compass-stack` host process +independent of what runtime runs inside it (podman today, +apple-container later), so macOS supervision is NOT gated on the +RIG-3238 backend choice — this record ships the macOS mechanics now +(OQ-5, Resolved decisions). The only remaining macOS gate is the +"macOS supported" GA doc-claim + end-to-end smoke, which ride the +stack-TOPOLOGY validation owned by the sibling lane (embedded-revival +OQ-7's AF_UNIX-across-the-VM-boundary question) — dependency ordering +on another lane, not a supervision fork here. (The +`apple-container-macos-runner/design.md:713-733` citation resolves +only once that sibling RIG-3238 record lands on main — PR #869 is not +yet merged; it is a forward reference, not a same-tree anchor.) ### Alternatives considered @@ -432,12 +439,12 @@ same-tree anchor.) identities are ever signaled; verify-before-signal; bounded escalation; survivor-rewrite on partial failure (`ui/compass-stack-cross-process-teardown/design.md:299-301`). -- Lock lifetime under `--supervise` (OQ-6, Matt fork): `up --supervise` is +- Lock lifetime under `--supervise` (OQ-6 → (b), ruled): `up --supervise` is the first caller to hold the state-dir lockfile with a LIVE pid for the stack's whole lifetime, voiding the "up always exits, so the lock holder is dead in every teardown" invariant `DownDetached`'s live-holder guard - relies on (`pgidfile.go:90-94`, `downdetached.go:74-84`). Whichever OQ-6 - option lands, `compass-stack down` MUST remain a real stop of a + relies on (`pgidfile.go:90-94`, `downdetached.go:74-84`). Under the ruled + (b) mode-token guard, `compass-stack down` MUST remain a real stop of a supervised stack (DL-259) and the T1 pre-spawn sweep MUST run before the lock is acquired (else it self-refuses). No `--supervise` deliverable ships a wedged `down`. @@ -577,14 +584,13 @@ Interfaces: failures, not one; (4) a partial-drain child death → restart → the surviving old child is torn down by the pre-spawn cleanup, not orphaned; (5) `compass-stack down` against a supervised (Ready) stack never wedges - on `ErrStackStarting` and the stop lands with the OQ-6 ruling — asserted - in-process against a stubbed Ready-supervised lock holder (the OQ-6 - counterpart to `downdetached_test.go:238`, which today asserts refusal - for an up-in-flight holder): under (b) `DownDetached` signals the - supervising holder and it exits ZERO; under (a) the lock is already - released so `down` signals the children directly and the supervise loop's - child-death path applies. This is the OQ-6 behavior T2's stop-truth - delegates here. Plus a process-level smoke on Linux: `up --supervise`, + on `ErrStackStarting` and the stop lands under the ruled OQ-6 (b) — + asserted in-process against a stubbed Ready-supervised lock holder (the + OQ-6 counterpart to `downdetached_test.go:238`, which today asserts + refusal for an up-in-flight holder): the (b) mode-token guard makes + `DownDetached` signal the supervising holder and it exits ZERO. This is + the OQ-6 behavior T2's stop-truth delegates here. Plus a process-level + smoke on Linux: `up --supervise`, `kill` a child, assert non-zero exit and a clean survivor teardown. ### T2 — `compass-stack service install` / `uninstall` + unit templates @@ -634,28 +640,25 @@ Interfaces: `EnvironmentVariables` PATH; install runs `launchctl bootstrap gui/$UID` then `enable`. - Operator stop-truth — the `down` verb vs the installed unit (folded from - the red-team, and corrected for the OQ-6 lock-lifetime fork): a naive + the red-team, resolved under the ruled OQ-6 (b) lock-lifetime): a naive `compass-stack down` against a service-supervised stack does NOT reach the children — because `up --supervise` holds the state-dir lockfile with a LIVE pid for its whole lifetime, `DownDetached` step 1 hits the live-holder guard and returns `ErrStackStarting` (`downdetached.go:74-84`), rendered as "a stack is starting; retry once it is up" (`main.go:406-409`) — a guard that never clears while supervise - runs. OQ-6 (below) decides how `down` gets a real path to a supervised - stack: option (b)'s mode-token guard makes a supervising holder a valid - `down` target that is signalled (it tears down and exits ZERO, no - restart); option (a) releases the lock at Ready so `down`'s guard passes - and it reads the record. On top of whichever OQ-6 lands, `service install` - makes `down` UNIT-AWARE: when an installed unit is active, `down` stops it - THROUGH the unit (`systemctl --user stop` / `launchctl bootout`) so the - supervise process performs the ordered teardown and exits ZERO (no - restart from the OS supervisor), preserving `compass-stack` as the single - stop surface DL-259 names — rather than the operator having to know - `systemctl --user stop` / `launchctl bootout`. The unit-aware `down` is a - fold; the underlying lock-lifetime mechanism it rides on is OQ-6, a Matt - fork. For a bare (unit-less) `up --supervise` — dev/devenv, and T1's - standalone Linux smoke before `service install` exists — `down` works - once OQ-6 is ruled and not before; T1's test cycle item (5) covers it. + runs. OQ-6 → (b) resolves it: the mode-token guard makes a supervising + holder parked at Ready a valid `down` target that is signalled (it tears + down and exits ZERO, no restart), while a mid-bring-up `up` is still + refused. On top of that, `service install` makes `down` UNIT-AWARE: when + an installed unit is active, `down` stops it THROUGH the unit + (`systemctl --user stop` / `launchctl bootout`) so the supervise process + performs the ordered teardown and exits ZERO (no restart from the OS + supervisor), preserving `compass-stack` as the single stop surface DL-259 + names — rather than the operator having to know `systemctl --user stop` / + `launchctl bootout`. For a bare (unit-less) `up --supervise` — dev/devenv, + and T1's standalone Linux smoke before `service install` exists — the (b) + guard gives `down` a real path; T1's test cycle item (5) covers it. - Unit-content gate: every item of the T2 checklist (former OQ-4, below) — explicit `--state-dir`, absolute paths + PATH, `KillMode=mixed` / `AbandonProcessGroup=true` ordered-teardown knobs, pinned `RestartSec` / @@ -673,7 +676,8 @@ Interfaces: active stops it through the unit and exits zero (no restart) → uninstall removes the unit. The launchd leg smokes the same cycle on a darwin host (install → running → crash-kill → KeepAlive restart → stop → uninstall); - it lands with T3 and is gated by OQ-5's end-to-end caveat. + it lands with T3; the full end-to-end macOS GA smoke rides the + stack-topology validation lane, not supervision (OQ-5, Resolved). ### T3 — darwin start-time identity reader @@ -721,8 +725,8 @@ Interfaces: to the darwin readers so the two encodings cannot drift; a new darwin-tagged unit test reads the test process's own start time twice (stable, non-empty) and verifies a dead/mismatched pid fails the identity - check. Runs on the DL-263 darwin CI leg. The full macOS stack-up smoke is - OQ-5-gated (topology, not supervision). + check. Runs on the DL-263 darwin CI leg. The full macOS stack-up smoke + rides the topology-validation lane, not this supervision record (OQ-5, Resolved). ### T4 — Docs, ledger, close-out @@ -791,81 +795,6 @@ ruling, all are content-completeness for the executor): of truth is `compass-stack status` — the unit's state is supervise-pid liveness only. -### OQ-5 [load-bearing, Matt fork — tracked RIG-3261] — macOS service: ship against podman-machine now, or gate GA on RIG-3238? - -The supervision mechanics for macOS are shippable in this record (T3 darwin -identity reader — a bounded two-seam swap; T2 launchd template), but a WORKING -macOS stack end-to-end still hangs on topology unknowns owned by the -sibling lanes: embedded-revival OQ-7 (the postgres DSN + agent sockets are -AF_UNIX bind-mounts, unvalidated across the podman-machine virtiofs/VM -boundary, `ui/compass-native-embedded-revival/design.md:909-931`) and -RIG-3238's OQ-12 (runner-on-darwin once podman-machine goes, -`platform/apple-container-macos-runner/design.md:713-733`). The fork: (a) -ship T2/T3's macOS support now, labeled experimental until the topology -validates, or (b) hold the macOS half of `service install` behind the -RIG-3238 lane's resolution and ship Linux-only first. **The cost option (a) -must carry (folded from the design-critic red-team):** an "experimental" -doc label does NOT stop a crashloop — on a mac where the socket topology -cannot work, `up --supervise` never reaches Ready, and launchd's -`KeepAlive={SuccessfulExit=false}` (which cannot tell a never-Ready -bring-up failure from a post-Ready crash) restarts it every ~10s forever, -burning CPU + podman-machine churn. So option (a) is only safe bundled with -a bounded-crashloop guard: the darwin `service install` runs an -install-time preflight (podman machine reachable + one probe cycle) and -refuses with a legible error until it passes, AND/OR the supervise loop -self-disables after N consecutive bring-up failures (a start-failure -backoff launchd's `SuccessfulExit` key cannot express). **Recommendation: -(a) with that preflight + self-limit** — build and land the darwin -mechanics now (they are small, testable on the DL-263 CI leg, and required -under EVERY macOS outcome, including apple-container), gate only the "macOS -supported" doc claim + GA smoke on the sibling lane's socket-topology -validation, and let Linux ship independently either way. This is dependency -ordering only — macOS-in-scope is ruled, not open. - -### OQ-6 [load-bearing, Matt fork — tracked RIG-3261] — `--supervise` lock lifetime: release the lock at Ready, or teach the down-guard a supervising holder? - -`up --supervise` is the FIRST caller to hold the state-dir lockfile with a -LIVE pid for the stack's whole lifetime, voiding the invariant the teardown -path relies on: "up always exits after a successful spawn, so the writer -pid is dead in every linger teardown" (`pgidfile.go:90-94`). `upLocked` -returns a spawning `*Stack` still HOLDING the lock (`stack.go:151`; released -only on the attach path or by `Down`), the lockfile carries `os.Getpid()` -(`lockfile.go:124`), and `DownDetached` step 1 refuses outright when that -holder is live — `if live { return ErrStackStarting }` -(`downdetached.go:74-84`), rendered as "a stack is starting; retry once it -is up" (`main.go:406-409`). So while a bare `up --supervise` runs, -`compass-stack down` is permanently wedged behind that guard and never -reaches the record. The fork — how the supervise process reconciles holding -the lock for its lifetime with `down` staying a real stop verb (DL-259): - -- **(a) Release the O_EXCL lockfile once Ready is reached.** Restores the - linger-ownership shape `DownDetached` already expects (a dead/absent - holder → guard passes → `down` reads the record and signals the - children); the supervise process's teardown authority then comes from its - held child handles, not the lock. Cost: forfeits the mutual exclusion the - live lock gives — a second `up` could spawn a DUPLICATE stack while the - first supervises, a regression to the DL-183 single-owner model. -- **(b) Teach `DownDetached`'s live-holder guard a mode/state token.** The - lockfile records whether the holder is a mid-bring-up `up` (refuse, as - today) or a supervise process parked at Ready (a VALID `down` target that - is signalled, not refused). Preserves single-owner mutual exclusion. Cost: - a real semantic change to the frozen DL-183 interlock — the guard, the - lockfile format, and their invariants. - -**Recommendation: (b)** — it keeps DL-183's single-owner guarantee, which -(a) trades away, and makes `down` a first-class stop of a supervised stack -rather than leaning on T2's install-time `down`-unit-awareness to mask a -wedged guard. Every T1/T2 site is already written to be -option-agnostic: T1 item 4's before-lock sweep is the -option-independent fix (Global Constraint — the sweep MUST run before -the lock is acquired whichever OQ-6 option lands), the T2 -operator-stop-truth bullet carries both (a)/(b) branches explicitly, -and T1 test-cycle item (5) states its (a) and (b) shapes — so a -ruling of (a) needs no rewrite, only deletion of the now-moot (b) -branches. -This modifies a Matt-ruled safety contract (DL-183), so it is his call, not -a coordinator fold. - ## Resolved decisions - **OQ-1 (Quadlet vs pgid) — RULED: keep pgid, decline Quadlet** (Matt, @@ -896,5 +825,34 @@ a coordinator fold. rootless/no-daemon hard requirement, no per-container keep-id equivalent), mirroring the embedded-revival OQ-9 rejection — see Alternatives. - macOS in scope: ruled by Matt ("how do we support this on macOS as - well?") — only the dependency ordering (OQ-5) remains open, never - whether macOS is supported. + well?") — macOS is supported, and the dependency ordering (OQ-5) is now + ruled below. +- **OQ-5 (macOS service dependency ordering) — RULED: ship supervision now, + runtime-agnostic** (Matt, RIG-3239, 2026-09-05): "we'd use a launchd + service for the stack, unrelated to it running on podman/apple + container." The launchd LaunchAgent supervises the `compass-stack` host + process independent of the runtime inside it, so macOS supervision is NOT + gated on the RIG-3238 podman-vs-apple-container backend choice — the + OQ-5-as-filed "ship against podman-machine now vs gate GA on RIG-3238" + fork dissolves. This record ships the macOS supervision mechanics now (T2 + launchd template + T3 darwin identity reader) WITH the darwin + `service install` install-time preflight + N-consecutive-failure + self-limit crashloop guard (from the red-team — a never-Ready mac still + KeepAlive-restarts every ~10s otherwise). The only residual macOS gate is + the "macOS supported" GA doc-claim + end-to-end smoke, which ride the + stack-TOPOLOGY validation owned by the sibling lane (embedded-revival + OQ-7's AF_UNIX-across-the-VM-boundary question) — dependency ordering on + another lane, not a supervision fork here. +- **OQ-6 (`--supervise` lock lifetime) — RULED: (b), teach the down-guard a + mode/state token** (Matt, RIG-3261, 2026-09-05): `DownDetached`'s + live-holder guard learns whether the lock holder is a mid-bring-up `up` + (refuse, as today — `ErrStackStarting`) or a supervise process parked at + Ready (a VALID `down` target that is signalled, tears down, and exits + ZERO). Preserves DL-183's single-owner mutual exclusion that option (a), + releasing the lock at Ready, would have traded away. Cost accepted: a + real semantic change to the DL-183 down-guard — the guard's refuse-vs-signal + decision and the lockfile mode/state token — landed by T1 (the before-lock + pre-spawn sweep stays the option-independent fix), the teardown signaling + itself (pgid record, verify-before-signal, reverse-order SIGTERM) + unchanged, with the `downdetached_test.go:238` counterpart asserting the + Ready-holder signal-not-refuse path. From 38de52152929a7b3a4f7dcab9420cfbb3e232e14 Mon Sep 17 00:00:00 2001 From: mintaka Date: Sat, 5 Sep 2026 19:57:03 -0400 Subject: [PATCH 8/9] docs(platform): renumber supervision ledger row DL-328 -> DL-329 (RIG-3239) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit sync-before-submit fix: rebased onto current main, whose ledger tail advanced to DL-328 (gateway_credentials encryption) since this branch was cut. The record's own ID-allocation guard mandates taking the next free id when the claimed number is taken — DL-329. Renumbered the DECISIONS.md row + the record's ID-allocation prose + the two T4 self-references; folded the runtime-agnostic macOS + OQ-6 mode-token clauses into the DL-329 row. gateway-creds DL-328 untouched. Ledger-impact: renumber only (DL-328 -> DL-329); no decision content change. Spec-impact: none. Refs RIG-3239 Co-authored-by: Matt Wilkinson --- docs/designs/DECISIONS.md | 2 +- .../platform/compass-stack-supervision/design.md | 16 ++++++++-------- 2 files changed, 9 insertions(+), 9 deletions(-) diff --git a/docs/designs/DECISIONS.md b/docs/designs/DECISIONS.md index c3457126..a89d7baf 100644 --- a/docs/designs/DECISIONS.md +++ b/docs/designs/DECISIONS.md @@ -310,7 +310,7 @@ check enforces the mechanical half. Full rationale: | DL-319 | The Compass native app is DUAL-MODE again: `mode="embedded"` returns as the low-friction onboarding / local-dev front door — the app spawns/supervises a LOCAL stack via rootless podman on the user's own machine (macOS via podman machine, Linux native; Windows/WSL deferred) — ADDED ALONGSIDE the fully-surviving client mode, which stays first-class and is the RECOMMENDED steady-state for real self-host (always-on VPS/EC2 running `compass-stack up`, reached over TLS). Rationale is the trust model (DL-318): single-tenant operator-own-code needs no KVM isolation; podman is a permanent supported tier (microVM recommended where the host has KVM). Supersedes DL-235 (the "client is the ONLY mode" thesis — the exclusivity dies, the client surface survives whole) and the app-never-spawns half of DL-236 (whose standalone-`compass-stack` half stays Active and load-bearing); restores the dual-mode SHAPE of DL-106 by citation, and partial-supersedes DL-259's KVM-floor clause by citation (funnel entry 2's podman tier runs on a KVM-absent VPS) | Active (Matt, 2026-09-01) | [embedded-revival §A1](ui/compass-native-embedded-revival/design.md#a1--mode-selection-contract-dual-mode-returns-embedded-is-the-onboarding-default) | | DL-320 | app.toml is dual-mode: absent → embedded (the zero-config onboarding default returns); `mode="embedded"` accepts no server_url/ca_cert; `mode="client"` keeps the built contract (https-only server_url required, optional ca_cert, keychain-first bearer per DL-109); the `--mode`/`$COMPASS_APP_MODE` override returns (flag > env > file > default). Graduation embedded→client is a config edit documented in the self-host doc, not an in-app flow. Supersedes DL-237 | Active (Matt, 2026-09-01) | [embedded-revival §A1](ui/compass-native-embedded-revival/design.md#a1--mode-selection-contract-dual-mode-returns-embedded-is-the-onboarding-default) | | DL-321 | The app bundle carries embedded's sidecars again — `bin/{compass-app,compass-stack,compass-server,compass-runner}` + dist, PATH-threaded (DL-215's mechanism restored by citation) — but NO postgres tooling and no `compass-postgres` sidecar: the embedded stack's postgres is the DL-260 stock `postgres:18` container via rootless podman (the CLI's own default), leaving rootless podman (plus podman machine on macOS) as the packaged embedded mode's sole container prerequisite. DL-217 STAYS superseded. macOS stages the same four binaries in `Contents/MacOS/`. Supersedes DL-238 | Active (Matt, 2026-09-01) | [embedded-revival §A4](ui/compass-native-embedded-revival/design.md#a4--bundle-re-scope-sidecars-return-postgres-stays-a-container) | -| DL-328 | Self-host stack supervision KEEPS the hand-rolled DL-183/DL-262 pgid mechanism as the SINGLE cross-platform supervision model; per-service Podman Quadlet units are NOT adopted (Linux/systemd-only, so Quadlet structurally fails the ruled all-platforms bar AND the pgid path survives regardless — two models vs one; the imperative cold sequence would need oneshot pre-units + sdnotify re-plumbing; per-service units displace the DL-259-named `compass-stack up` verb). Whole-stack CRASH RECOVERY ships as a blocking `compass-stack up --supervise` foreground mode (up-to-Ready, watch children, non-zero exit on child death, teardown on signal) wrapped by the platform-native OS supervisor's restart policy, plus a one-command `compass-stack service install`/`uninstall` that writes + enables the native unit — systemd USER unit (`Type=exec`, `Restart=on-failure`, `TimeoutStopSec>=90`) on Linux, launchd LaunchAgent (`RunAtLoad`, `KeepAlive={SuccessfulExit=false}`, `ExitTimeOut>=90`) on macOS — auto-start at reboot + restart-on-crash; the OS supervisor supplies ONLY restart/backoff/boot-start (DL-183 spawn/teardown unchanged), status truth stays `compass-stack status`. Docker-socket declined at the stack layer (daemon model vs the rootless/no-daemon hard requirement; no per-container keep-id equivalent) | Active (Matt, 2026-09-04) | [stack supervision §Approach](platform/compass-stack-supervision/design.md#approach) | +| DL-329 | Self-host stack supervision KEEPS the hand-rolled DL-183/DL-262 pgid mechanism as the SINGLE cross-platform supervision model; per-service Podman Quadlet units are NOT adopted (Linux/systemd-only, so Quadlet structurally fails the ruled all-platforms bar AND the pgid path survives regardless — two models vs one; the imperative cold sequence would need oneshot pre-units + sdnotify re-plumbing; per-service units displace the DL-259-named `compass-stack up` verb). Whole-stack CRASH RECOVERY ships as a blocking `compass-stack up --supervise` foreground mode (up-to-Ready, watch children, non-zero exit on child death, teardown on signal) wrapped by the platform-native OS supervisor's restart policy, plus a one-command `compass-stack service install`/`uninstall` that writes + enables the native unit — systemd USER unit (`Type=exec`, `Restart=on-failure`, `TimeoutStopSec>=90`) on Linux, launchd LaunchAgent (`RunAtLoad`, `KeepAlive={SuccessfulExit=false}`, `ExitTimeOut>=90`) on macOS — auto-start at reboot + restart-on-crash; the OS supervisor supplies ONLY restart/backoff/boot-start (DL-183 spawn/teardown unchanged), status truth stays `compass-stack status`. macOS supervision is RUNTIME-AGNOSTIC (the launchd LaunchAgent supervises the `compass-stack` host process independent of the podman/apple-container runtime inside it), so it ships now and is NOT gated on the RIG-3238 backend choice (Matt, RIG-3239). The `--supervise` lock lifetime teaches `DownDetached`'s live-holder guard a mode/state token — a supervise process parked at Ready is a valid `down` target (signalled, exits zero), a mid-bring-up `up` is still refused — preserving DL-183 single-owner mutual exclusion (OQ-6 → (b), Matt, RIG-3261). Docker-socket declined at the stack layer (daemon model vs the rootless/no-daemon hard requirement; no per-container keep-id equivalent) | Active (Matt, 2026-09-05) | [stack supervision §Approach](platform/compass-stack-supervision/design.md#approach) | ## Agent batteries diff --git a/docs/designs/platform/compass-stack-supervision/design.md b/docs/designs/platform/compass-stack-supervision/design.md index bf0da53d..796ebaed 100644 --- a/docs/designs/platform/compass-stack-supervision/design.md +++ b/docs/designs/platform/compass-stack-supervision/design.md @@ -453,12 +453,12 @@ yet merged; it is a forward reference, not a same-tree anchor.) (`downdetached.go:15-24`: "15 + (30+5) + (10+5) + (10+5) = 80s"). - Status truth stays `compass-stack status`: the OS unit expresses process-liveness of the supervise pid, never stack health semantics. -- ID allocation + freeze order (ledger-collision guard): this record claims - DL-328. Main's ledger tail is DL-324; DL-325/DL-326/DL-327 are claimed by - unlanded PRs (#804/#859 on DL-325, #859 on DL-326, #868 on DL-327), so - DL-328 is the first free number as of this writing. The driver MUST - re-grep main's then-current ledger tail immediately before landing and - take the next free id if DL-328 is taken. The row cites its own decision +- ID allocation + freeze order (ledger-collision guard): this record lands as + DL-329. At freeze, main's ledger tail was DL-328 (DL-325 runner trust-split, + DL-326 session-volume clone, DL-327 token-subject, DL-328 gateway-creds + encryption all landed), so DL-329 is the first free number — the record's + originally-claimed DL-328 was taken by the gateway-creds row, and the driver + took the next free id per this guard. The row cites its own decision self-containedly (no cross-cite to an unlanded number), so merge order cannot falsify it. @@ -740,7 +740,7 @@ Interfaces: platforms, the linger prerequisite, the log locations, and the status-truth caveat (`compass-stack status`, never `systemctl --user status` / `launchctl print` alone); the DECISIONS.md - DL-328 row (rewritten to this ruling — the coordinator lands it with the + DL-329 row (rewritten to this ruling — the coordinator lands it with the record); RIG-3239 close-out. - Test cycle: markdownlint on the record + doc; the doc's command sequence is walked once verbatim on a Linux host as part of the T2 smoke. @@ -757,7 +757,7 @@ Interfaces: `readGroupLeaderStartTime`, `sysctl KERN_PROC` via x/sys, one shared encoding); darwin-tagged unit test on the DL-263 CI leg. - [ ] T4 — self-host doc "run as a service" section (both platforms + - status-truth caveat), DL-328 ledger row, close-out. + status-truth caveat), DL-329 ledger row, close-out. ## Open Questions From b62f0213b86ebf0198e1e24029a40456b30fa8d5 Mon Sep 17 00:00:00 2001 From: mintaka Date: Sat, 5 Sep 2026 21:04:31 -0400 Subject: [PATCH 9/9] docs(platform): close review findings on the RIG-3261 fold (RIG-3239) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review of the freeze fold (0H/0M gate) returned 2M/2L — the recurring summary-vs-detail drift class, this time inverted: the fold ADDED a new load-bearing deliverable to the OQ-5 Resolved summary with no detail site to back it. Closed: - M1 (crashloop guard): the fold folded the red-team's darwin install-time preflight + never-Ready self-limit into the OQ-5 Resolved bullet as shipped-now, but it existed at NO detail site and contradicted the Division-of-labor paragraph (which said the OS start-limit IS the ceiling). Landed the guard at every detail site: T1 Produces + test cycle (the never-Ready BRING-UP self-limit, distinct from the attached-probe N-consecutive degradation); T2 Produces (service install runs the install-time preflight and refuses on failure) + unit-content gate + test cycle (preflight-refusal unit test); T1/T2 Tasks bullets. Reconciled the Division-of-labor paragraph: the OS start-limit is the ceiling for POST-Ready crash restarts, the crashloop guard bounds the never-Ready path (throttle alone would KeepAlive-loop a misconfigured host). - M2 (provenance): the OQ-5 ruling was tagged "(Matt, RIG-3239, 2026-09-05)" but RIG-3239 has zero comments — the ruling is a genuine 2026-09-05 session directive. Retagged both sites (OQ-5 Resolved bullet + the Approach prose) to "session directive". Mirrored the directive as a RIG-3239 comment so the cite is durably resolvable. - L1 (stale cross-ref): the OQ-12 forward-reference cited apple-container-macos-runner/design.md:713-733 + a "cannot run natively on darwin" quote that no longer exists post-#869-reorientation (OQ-12 was ruled the opposite way — runner IS host-side darwin-native). Dropped the stale line-range + quote for a range-free reference reflecting the ruled state; fixed the Problem/Intent echo too. Verified: markdownlint 0, design-ledger-gate OK (296 rows), provenance-leftover + stale-cite scans clean, guard referenced at 17 sites. Re-review confirmed M2 + L1 closed but M1 only PARTIALLY — the crashloop guard was physically present at all detail sites, but the never-Ready self-limit MECHANISM was unimplementable as specified. Three new mediums + three lows, now closed: - M-1 (self-limit reachability + counter): the self-limit lived in `Supervise`, but a never-Ready `Up` returns nil `*Stack` and never enters it; and N-consecutive failures span OS-supervisor RESTARTS (separate processes), so an in-process counter resets each loop. Relocated the self-limit to the CLI supervise wrapper (`main.go`) that owns the `Up` retry, with the failure counter PERSISTED in the state dir (reset on Ready); corrected T1 test item (6) to name the CLI-level loop (the internal/stack harness can't exercise it). - M-2 (terminal-exit knob): no unit-template knob expressed the "terminal non-restart exit" — systemd `Restart=on-failure` / launchd `KeepAlive={SuccessfulExit=false}` restart on ANY non-zero. Pinned the reserved code (78/EX_CONFIG): systemd `RestartPreventExitStatus=78`; launchd has no per-code exemption so the wrapper self-`bootout`s its agent on the terminal path. Added both to the templates, the unit-content gate, and the T2 test cycle; corrected the OQ-4 "both rendered/asserted by T2" claim to name half 2's correct owners (behavior T1, exit-knob T2). - M-3 (cross-platform preflight): the install-preflight was macOS-only ("podman machine reachable") for a cross-platform verb; on Linux podman is native. Restated per-platform (darwin: podman machine; linux: rootless podman + real XDG_RUNTIME_DIR) + one probe cycle on both; bound it to REUSE the existing `preflight` verb rather than a divergent check; made the T2 refusal test cover the linux path too (the all-platforms bar forbids a macOS-only close). - L-1: qualified the Approach "crashloop ceiling" phrasing to POST-Ready. L-2: restored a range-free forward-ref caveat (the #869 record isn't merged yet). L-3: aligned the DL-329 ledger provenance tag with the record (session directive, mirrored on RIG-3239). Scoped re-review of the M-1/M-2/M-3 interdiff returned FREEZABLE (0H/0M/3L). Closed two of the three non-gating lows (the third, a cosmetic soft-wrap, is left as-is per the reviewer's explicit non-gating note): - L-1 (preflight-reuse citation scope): the M-3 fix bound half 1 to "REUSING the existing `preflight` verb", but `runPreflight` (`cmd/compass-stack/preflight.go:86-110`) runs `checkKVM()` + the microVM-trio floors (`hostcheck.go:42-46`) that gate the microVM RUNNER, not the supervised stack — and darwin has no `/dev/kvm`, so as-cited it would refuse install on every Mac. Softened to reuse the verb's per-check machinery with a supervision-scoped check set (`internal/preflight/preflight.go:28-34` `MachineReady`, darwin-only/linux-absent), explicitly excluding the KVM/microVM floors; swept the OQ-4 summary site to match. - L-3 (launchd never-Ready coverage): T1 test item (6) asserted only the systemd-shaped terminal exit. Extended it per-platform — linux exits the reserved code (78/EX_CONFIG), darwin additionally invokes the self-`bootout`/`disable` seam (stubbed at the CLI boundary, asserted called once with the agent label), since launchd keys on zero-vs-non-zero with no per-code exemption. Verified: new cites resolve at head (`MachineReady`, `runPreflight`, `MicroVMFloors`); markdownlint 0; design-ledger-gate OK (296 rows). A second scoped re-review of that interdiff confirmed both closures (no new summary/detail split: all 12 preflight mentions enumerated, the two that assert the reuse mechanism both swept; the launchd mechanism agrees across all four sites) and returned FREEZABLE 0H/0M/3L. Those three lows are now dispositioned as fixes rather than deferrals: - The darwin `MachineReady` seam has no adapter wired at head and its absence is SILENT, not failing (`internal/preflight/preflight.go:117` guards `GOOS == "darwin" && MachineReady != nil`; the sole wiring site `compass-app/embedded.go:379-383` omits it, pending embedded-revival T-6). Following the citation as written would have produced an install-time preflight that silently passes on a mac with a dead podman machine — inert on exactly the platform this record just cleared to ship. Added the obligation to the T2 Produces darwin item: T2 either wires a darwin machine probe itself or the darwin half of guard 1 is inert until T-6. - The unit-content gate listed "launchd self-`bootout`" alongside the systemd knob as if both were renderable unit content; they are not symmetric (the launchd path is wrapper runtime behavior with nothing to render). Marked it cross-referenced rather than rendered, pointing at OQ-4's ownership split. - Reflowed the soft-wrap stub tails the two fixes left in the T1 item (6), T2 Produces, and OQ-4 paragraphs. Verified: the new `preflight.go:117` + `embedded.go:379-383` cites resolve at head; markdownlint 0; design-ledger-gate OK (296 rows). Ledger-impact: none. Spec-impact: none. Refs RIG-3239 Co-authored-by: Matt Wilkinson --- docs/designs/DECISIONS.md | 2 +- .../compass-stack-supervision/design.md | 149 ++++++++++++++---- 2 files changed, 117 insertions(+), 34 deletions(-) diff --git a/docs/designs/DECISIONS.md b/docs/designs/DECISIONS.md index a89d7baf..1f774f44 100644 --- a/docs/designs/DECISIONS.md +++ b/docs/designs/DECISIONS.md @@ -310,7 +310,7 @@ check enforces the mechanical half. Full rationale: | DL-319 | The Compass native app is DUAL-MODE again: `mode="embedded"` returns as the low-friction onboarding / local-dev front door — the app spawns/supervises a LOCAL stack via rootless podman on the user's own machine (macOS via podman machine, Linux native; Windows/WSL deferred) — ADDED ALONGSIDE the fully-surviving client mode, which stays first-class and is the RECOMMENDED steady-state for real self-host (always-on VPS/EC2 running `compass-stack up`, reached over TLS). Rationale is the trust model (DL-318): single-tenant operator-own-code needs no KVM isolation; podman is a permanent supported tier (microVM recommended where the host has KVM). Supersedes DL-235 (the "client is the ONLY mode" thesis — the exclusivity dies, the client surface survives whole) and the app-never-spawns half of DL-236 (whose standalone-`compass-stack` half stays Active and load-bearing); restores the dual-mode SHAPE of DL-106 by citation, and partial-supersedes DL-259's KVM-floor clause by citation (funnel entry 2's podman tier runs on a KVM-absent VPS) | Active (Matt, 2026-09-01) | [embedded-revival §A1](ui/compass-native-embedded-revival/design.md#a1--mode-selection-contract-dual-mode-returns-embedded-is-the-onboarding-default) | | DL-320 | app.toml is dual-mode: absent → embedded (the zero-config onboarding default returns); `mode="embedded"` accepts no server_url/ca_cert; `mode="client"` keeps the built contract (https-only server_url required, optional ca_cert, keychain-first bearer per DL-109); the `--mode`/`$COMPASS_APP_MODE` override returns (flag > env > file > default). Graduation embedded→client is a config edit documented in the self-host doc, not an in-app flow. Supersedes DL-237 | Active (Matt, 2026-09-01) | [embedded-revival §A1](ui/compass-native-embedded-revival/design.md#a1--mode-selection-contract-dual-mode-returns-embedded-is-the-onboarding-default) | | DL-321 | The app bundle carries embedded's sidecars again — `bin/{compass-app,compass-stack,compass-server,compass-runner}` + dist, PATH-threaded (DL-215's mechanism restored by citation) — but NO postgres tooling and no `compass-postgres` sidecar: the embedded stack's postgres is the DL-260 stock `postgres:18` container via rootless podman (the CLI's own default), leaving rootless podman (plus podman machine on macOS) as the packaged embedded mode's sole container prerequisite. DL-217 STAYS superseded. macOS stages the same four binaries in `Contents/MacOS/`. Supersedes DL-238 | Active (Matt, 2026-09-01) | [embedded-revival §A4](ui/compass-native-embedded-revival/design.md#a4--bundle-re-scope-sidecars-return-postgres-stays-a-container) | -| DL-329 | Self-host stack supervision KEEPS the hand-rolled DL-183/DL-262 pgid mechanism as the SINGLE cross-platform supervision model; per-service Podman Quadlet units are NOT adopted (Linux/systemd-only, so Quadlet structurally fails the ruled all-platforms bar AND the pgid path survives regardless — two models vs one; the imperative cold sequence would need oneshot pre-units + sdnotify re-plumbing; per-service units displace the DL-259-named `compass-stack up` verb). Whole-stack CRASH RECOVERY ships as a blocking `compass-stack up --supervise` foreground mode (up-to-Ready, watch children, non-zero exit on child death, teardown on signal) wrapped by the platform-native OS supervisor's restart policy, plus a one-command `compass-stack service install`/`uninstall` that writes + enables the native unit — systemd USER unit (`Type=exec`, `Restart=on-failure`, `TimeoutStopSec>=90`) on Linux, launchd LaunchAgent (`RunAtLoad`, `KeepAlive={SuccessfulExit=false}`, `ExitTimeOut>=90`) on macOS — auto-start at reboot + restart-on-crash; the OS supervisor supplies ONLY restart/backoff/boot-start (DL-183 spawn/teardown unchanged), status truth stays `compass-stack status`. macOS supervision is RUNTIME-AGNOSTIC (the launchd LaunchAgent supervises the `compass-stack` host process independent of the podman/apple-container runtime inside it), so it ships now and is NOT gated on the RIG-3238 backend choice (Matt, RIG-3239). The `--supervise` lock lifetime teaches `DownDetached`'s live-holder guard a mode/state token — a supervise process parked at Ready is a valid `down` target (signalled, exits zero), a mid-bring-up `up` is still refused — preserving DL-183 single-owner mutual exclusion (OQ-6 → (b), Matt, RIG-3261). Docker-socket declined at the stack layer (daemon model vs the rootless/no-daemon hard requirement; no per-container keep-id equivalent) | Active (Matt, 2026-09-05) | [stack supervision §Approach](platform/compass-stack-supervision/design.md#approach) | +| DL-329 | Self-host stack supervision KEEPS the hand-rolled DL-183/DL-262 pgid mechanism as the SINGLE cross-platform supervision model; per-service Podman Quadlet units are NOT adopted (Linux/systemd-only, so Quadlet structurally fails the ruled all-platforms bar AND the pgid path survives regardless — two models vs one; the imperative cold sequence would need oneshot pre-units + sdnotify re-plumbing; per-service units displace the DL-259-named `compass-stack up` verb). Whole-stack CRASH RECOVERY ships as a blocking `compass-stack up --supervise` foreground mode (up-to-Ready, watch children, non-zero exit on child death, teardown on signal) wrapped by the platform-native OS supervisor's restart policy, plus a one-command `compass-stack service install`/`uninstall` that writes + enables the native unit — systemd USER unit (`Type=exec`, `Restart=on-failure`, `TimeoutStopSec>=90`) on Linux, launchd LaunchAgent (`RunAtLoad`, `KeepAlive={SuccessfulExit=false}`, `ExitTimeOut>=90`) on macOS — auto-start at reboot + restart-on-crash; the OS supervisor supplies ONLY restart/backoff/boot-start (DL-183 spawn/teardown unchanged), status truth stays `compass-stack status`. macOS supervision is RUNTIME-AGNOSTIC (the launchd LaunchAgent supervises the `compass-stack` host process independent of the podman/apple-container runtime inside it), so it ships now and is NOT gated on the RIG-3238 backend choice (Matt, session directive, 2026-09-05; mirrored on RIG-3239). The `--supervise` lock lifetime teaches `DownDetached`'s live-holder guard a mode/state token — a supervise process parked at Ready is a valid `down` target (signalled, exits zero), a mid-bring-up `up` is still refused — preserving DL-183 single-owner mutual exclusion (OQ-6 → (b), Matt, RIG-3261). Docker-socket declined at the stack layer (daemon model vs the rootless/no-daemon hard requirement; no per-container keep-id equivalent) | Active (Matt, 2026-09-05) | [stack supervision §Approach](platform/compass-stack-supervision/design.md#approach) | ## Agent batteries diff --git a/docs/designs/platform/compass-stack-supervision/design.md b/docs/designs/platform/compass-stack-supervision/design.md index 796ebaed..aef80f53 100644 --- a/docs/designs/platform/compass-stack-supervision/design.md +++ b/docs/designs/platform/compass-stack-supervision/design.md @@ -19,8 +19,9 @@ scope (RIG-3070: podman permanent for self-host, microVM behind the seam; see `ui/compass-native-embedded-revival/design.md:71-74`), as is the macOS *embedded runner* backend (sibling RIG-3238 Apple-container record). One COUPLING crosses that fence and is grounded below: "constant-on service on -macOS" shares the runner-on-darwin / VM-topology unknowns that RIG-3238 -carries as its OQ-12 and the embedded-revival record carries as its OQ-7. +macOS" touches the runner-on-darwin / VM-topology question RIG-3238 carries as +its OQ-12 (now ruled host-side) and the embedded-revival record carries as its +OQ-7. ## Approach @@ -256,7 +257,9 @@ stops, disables, and removes it. `Restart=on-failure` + pinned `RestartSec=5`, `TimeoutStopSec=90` (the DL-262 teardown drain worst case is ~80s, `downdetached.go:15-24`), `KillMode=mixed` + an explicit start-limit posture (so the ordered drain - and the crashloop ceiling are chosen, not left to defaults), explicit + and the POST-Ready crash-restart ceiling are chosen, not left to defaults; + the never-Ready path is bounded separately by the crashloop guard, OQ-4), + explicit `Environment=PATH=` (user units inherit no login-shell PATH; podman + the DL-321 PATH-threaded sidecars must resolve), `[Install] WantedBy=default.target`; NO `After=network.target` (a system-manager @@ -330,15 +333,16 @@ darwin start-time reader unblocks spawn/teardown identity, but a WORKING macOS stack end-to-end still depends on unknowns the sibling lanes own: embedded-revival OQ-7 (the DL-260 postgres DSN and the agent sockets are AF_UNIX bind-mounts, and AF_UNIX does not cross the virtiofs/VM boundary — -unvalidated) and RIG-3238's OQ-12 ("the runner cannot run natively on -darwin once the podman-machine Linux VM is gone", -`docs/designs/platform/apple-container-macos-runner/design.md:713-733`, -which also notes the runner's podman host-capability preflight has no -darwin answer yet). Those are stack-TOPOLOGY unknowns, not supervision +unvalidated) and RIG-3238's OQ-12, the runner-on-darwin / VM-topology +question (`docs/designs/platform/apple-container-macos-runner/design.md` — a +forward reference: that sibling RIG-3238 record lands with PR #869, not yet +merged), now RULED host-side: the runner runs natively on darwin and drives the +backend host-side, which if anything strengthens this record's +runtime-agnostic position. Those are stack-TOPOLOGY unknowns, not supervision unknowns — this record ships the supervision mechanics for macOS (T2 launchd template + T3 darwin identity reader) and leaves the topology -validation with the lane that owns it. Per Matt's RIG-3239 ruling the -launchd LaunchAgent supervises the `compass-stack` host process +validation with the lane that owns it. Per Matt's 2026-09-05 session directive +the launchd LaunchAgent supervises the `compass-stack` host process independent of what runtime runs inside it (podman today, apple-container later), so macOS supervision is NOT gated on the RIG-3238 backend choice — this record ships the macOS mechanics now @@ -346,10 +350,7 @@ RIG-3238 backend choice — this record ships the macOS mechanics now "macOS supported" GA doc-claim + end-to-end smoke, which ride the stack-TOPOLOGY validation owned by the sibling lane (embedded-revival OQ-7's AF_UNIX-across-the-VM-boundary question) — dependency ordering -on another lane, not a supervision fork here. (The -`apple-container-macos-runner/design.md:713-733` citation resolves -only once that sibling RIG-3238 record lands on main — PR #869 is not -yet merged; it is a forward reference, not a same-tree anchor.) +on another lane, not a supervision fork here. ### Alternatives considered @@ -542,10 +543,13 @@ supervisor wraps the ONE blocking process and supplies only `Restart=on-failure` + backoff (systemd `Type=exec`; launchd `KeepAlive`). Whole-stack (not per-service) restart is the right granularity: the cold sequence's ordering means a restarted postgres needs the server's readiness -re-verified anyway (the crashloop ceiling is the OS supervisor's -start-limit / throttle, pinned in the T2 templates). This also eliminates -the two failure modes a returns-at-Ready oneshot wrapper has: the -status-lie (a `Type=exec` unit tracks a live main pid, so a dead stack +re-verified anyway. The OS supervisor's start-limit / throttle (pinned in the +T2 templates) is the ceiling for POST-Ready crash restarts; a stack that +NEVER reaches Ready is bounded separately by the crashloop guard (T2's +install-time preflight refusal + T1's never-Ready bring-up self-limit), since +throttle alone would KeepAlive-loop a misconfigured host indefinitely. This +also eliminates the two failure modes a returns-at-Ready oneshot wrapper has: +the status-lie (a `Type=exec` unit tracks a live main pid, so a dead stack reads `failed`, never fake-`active`) and stop-what-you-didn't-start (the OS supervisor owns exactly the pid it spawned). @@ -573,7 +577,19 @@ Interfaces: `stack.go:131-146`) supervises by polling `Health`, exiting non-zero only after N consecutive probe failures (a single transient probe error must not tear the stack — the restart then takes ownership of the manual - stack; interval + N stated in the impl). + stack; interval + N stated in the impl). Separately — and OUTSIDE + `Supervise`, because a never-Ready `Up` returns a nil `*Stack` and never + enters it — the never-Ready BRING-UP self-limit lives in the CLI supervise + wrapper (`go/cmd/compass-stack/main.go`): it owns the `Up` call, and on an + `Up` that fails to reach Ready it increments a failure counter PERSISTED in + the state dir (beside `stack.lock`), since the OS supervisor's restart — a + fresh process — is the actual retry driver, so an in-process counter would + reset every loop. On the Nth consecutive persisted failure the wrapper exits + TERMINALLY (the reserved non-restart code, see T2) so a misconfigured host + does not KeepAlive-loop every ~10s; any `Up` that reaches Ready resets the + counter to zero and hands the `*Stack` to `Supervise(ctx)`. The persistence + file, its reset-on-Ready rule, and N are stated in the impl — the never-Ready + crashloop guard, half 2 (half 1 = T2's install-time preflight). - Test cycle (red → green, existing stub harness `internal/stack/harness_test.go`): (1) a stubbed child's `Wait` returning early causes `Supervise` to drain survivors and return non-nil — with NO @@ -589,9 +605,18 @@ Interfaces: OQ-6 counterpart to `downdetached_test.go:238`, which today asserts refusal for an up-in-flight holder): the (b) mode-token guard makes `DownDetached` signal the supervising holder and it exits ZERO. This is - the OQ-6 behavior T2's stop-truth delegates here. Plus a process-level - smoke on Linux: `up --supervise`, - `kill` a child, assert non-zero exit and a clean survivor teardown. + the OQ-6 behavior T2's stop-truth delegates here. (6) the CLI supervise + wrapper, given a stubbed `Up` that fails to reach Ready N consecutive times + across simulated restarts (the persisted counter surviving each), takes the + terminal non-restart path — on linux exiting with the reserved code + (EX_CONFIG/78), on darwin additionally invoking the self-`bootout`/`disable` + seam (stubbed at the CLI boundary, asserted called once with the agent + label) before exit, since launchd keys on zero-vs-non-zero with no per-code + exemption — NOT the restart-inviting non-zero; a single failure followed by a + Ready resets the counter. (This tests the CLI-level loop, not `Supervise` — + the `internal/stack` harness cannot exercise a CLI-owned loop.) Plus a + process-level smoke on Linux: `up --supervise`, `kill` a child, assert + non-zero exit and a clean survivor teardown. ### T2 — `compass-stack service install` / `uninstall` + unit templates @@ -607,8 +632,27 @@ Interfaces: `--state-dir` and flags so the service stack and an interactive stack are distinct unless deliberately shared). - Produces: `service install` / `service uninstall` verbs extending the - `up|down|status|preflight` dispatch (`main.go:6`); two embedded unit - templates — + `up|down|status|preflight` dispatch (`main.go:6`). `service install` runs the + host-prerequisite check by reusing the `preflight` verb's per-check machinery + (`internal/hostcheck` Result/Decide + the print/exit surface), not a divergent + second check, but with a SUPERVISION-scoped check set — NOT the full + `runPreflight`, which also runs the KVM + microVM-trio floors + (`preflight.go:86-110`, `hostcheck.go:42-46`) that gate the microVM RUNNER, + not the supervised stack (DL-319's podman tier is KVM-absent, and darwin has + no `/dev/kvm` at all — running those would refuse install on every supported + podman host). The supervision subset, per platform: on darwin the podman + machine reachable (`internal/preflight/preflight.go:28-34` `MachineReady`, + darwin-only, linux-absent) — the adapter behind that seam lands in + embedded-revival T-6, and a nil `MachineReady` leaves the check ABSENT rather + than failing (`preflight.go:117`; the sole wiring site, + `compass-app/embedded.go:379-383`, omits it today), so T2 either wires a + darwin machine probe itself or the darwin half of guard 1 is inert until T-6 + lands; on linux rootless podman + a real `XDG_RUNTIME_DIR` (the linger + prerequisite below), plus one probe cycle on both. It REFUSES with a legible + error until the check passes — so a misconfigured host fails at install, not + in an invisible KeepAlive restart loop (the never-Ready crashloop guard, + half 1; the supervise-loop self-limit in T1 is half 2). On success it + renders + enables one of two embedded unit templates — - systemd user unit → `~/.config/systemd/user/compass-stack.service`: `Type=exec`, `ExecStart= up --supervise …`, `Restart=on-failure`, `RestartSec=5`, `TimeoutStopSec=90`, `KillMode=mixed` (folded from the @@ -620,7 +664,12 @@ Interfaces: `TimeoutStopSec` deadline, composing correctly with the 90s grace), explicit start-limit posture (`StartLimitIntervalSec=`/`StartLimitBurst=` stated, not left to the 5-in-10s default — the intended circuit breaker - is chosen, not an accident of defaults), `Environment=PATH=…`, + is chosen, not an accident of defaults), + `RestartPreventExitStatus=78` (the reserved terminal code, EX_CONFIG, the + never-Ready self-limit exits with — `Restart=on-failure` restarts on ANY + non-zero otherwise, so without this the terminal exit is indistinguishable + from a child-death non-zero and gets restarted; half 2's exit needs this + knob to actually stay down), `Environment=PATH=…`, `WantedBy=default.target`; NO `After=network.target` (it is a system-manager unit the per-user manager silently ignores, and the server binds loopback only, `main.go:42-45` — no network-up ordering @@ -637,7 +686,11 @@ Interfaces: launchd's default teardown SIGKILLs remaining process-group members at job stop, which would race the DL-183 graceful drain — the plist must let the children survive long enough for the ordered reverse teardown), - `EnvironmentVariables` PATH; install runs + `EnvironmentVariables` PATH. launchd's `KeepAlive={SuccessfulExit=false}` + keys on zero-vs-non-zero with NO per-code exemption, so the never-Ready + terminal exit (half 2) cannot ride an exit code here: on the Nth failure + the supervise wrapper `launchctl bootout`s / `disable`s its own agent + before exiting, so launchd does not re-launch it. install runs `launchctl bootstrap gui/$UID` then `enable`. - Operator stop-truth — the `down` verb vs the installed unit (folded from the red-team, resolved under the ruled OQ-6 (b) lock-lifetime): a naive @@ -664,10 +717,18 @@ Interfaces: `AbandonProcessGroup=true` ordered-teardown knobs, pinned `RestartSec` / `ThrottleInterval` + start-limit posture, stop grace >= 90s, log routing (journald / launchd `StandardErrorPath`), the linger step, the - status-truth doc caveat. + crashloop guard (install-time preflight refusal + the terminal-exit knob + the never-Ready self-limit rides: systemd `RestartPreventExitStatus=78`; on + launchd there is no renderable knob — the wrapper's self-`bootout` covers + that path, T1-owned, see OQ-4), the status-truth doc caveat. - Test cycle: unit tests on the template rendering (golden files: absolute paths, the >= 90s stop grace, the pinned state dir, `KillMode=mixed` / - `AbandonProcessGroup=true`). Smoke on a real systemd host: install → unit + `AbandonProcessGroup=true`, the systemd `RestartPreventExitStatus=78` knob) + plus a unit test that `service install` REFUSES + with the legible error when the install-time preflight fails — on BOTH + platforms (darwin: podman machine unreachable; linux: rootless podman / + `XDG_RUNTIME_DIR` missing), since the all-platforms bar forbids a + macOS-only close. Smoke on a real systemd host: install → unit active + `compass-stack status` Ready → kill a child → unit enters `failed`/restarts and the stack comes back → `systemctl --user stop` produces the ORDERED reverse drain (runner exits before server, not a @@ -749,10 +810,12 @@ Interfaces: - [ ] T1 — `up --supervise` blocking mode: supervise loop on the stack core (child-`Wait` fan-in, drain + non-zero on child death, signal-driven - `Down`), harness unit tests + Linux process smoke. -- [ ] T2 — `service install`/`uninstall` verbs + embedded systemd/launchd - unit templates, rendered against the T2 unit-content checklist; - golden-file tests + systemd-host smoke (launchd smoke with T3). + `Down`, N-consecutive-bring-up-failure self-limit), harness unit tests + + Linux process smoke. +- [ ] T2 — `service install`/`uninstall` verbs (with install-time preflight) + + embedded systemd/launchd unit templates, rendered against the T2 + unit-content checklist; golden-file tests + systemd-host smoke (launchd + smoke with T3). - [ ] T3 — darwin start-time readers at BOTH seams (`readStartTime` + `readGroupLeaderStartTime`, `sysctl KERN_PROC` via x/sys, one shared encoding); darwin-tagged unit test on the DL-263 CI leg. @@ -794,6 +857,26 @@ ruling, all are content-completeness for the executor): `StandardOutPath`/`StandardErrorPath`. Both docs state the status source of truth is `compass-stack status` — the unit's state is supervise-pid liveness only. +- **Crashloop guard (never-Ready bring-up).** `KeepAlive={SuccessfulExit=false}` + / systemd `Restart=on-failure` restart a stack that never reaches Ready + (podman machine unreachable on darwin, rootless-podman/`XDG_RUNTIME_DIR` + missing on linux, missing dep) every throttle interval indefinitely. The + `ThrottleInterval` / start-limit is the ceiling for POST-Ready crash restarts + only; it does not stop a never-Ready loop. Two guards cover it: (1) half 1 — + `service install` runs the per-platform install-time preflight (reusing the + `preflight` verb's per-check machinery with a supervision-scoped check set — + not the KVM/microVM-floor `runPreflight`) and refuses with a legible error + until it passes, so a misconfigured host fails at install, not in an + invisible restart loop (T2-rendered + T2-tested on both platforms); (2) + half 2 — the CLI supervise wrapper self-limits after N consecutive + never-Ready `Up` failures across OS-supervisor restarts (counter persisted + in the state dir; distinct from the attached-probe N-consecutive Health + degradation — this counts failures to REACH Ready), exiting with a reserved + terminal code the unit exempts from restart (systemd + `RestartPreventExitStatus=78`; launchd has no per-code + exemption, so the wrapper self-`bootout`s its agent on that path). Half 2's + behavior is T1-owned + T1-tested; the exit-exemption knob is T2-rendered. + Interval + N stated in the impl. ## Resolved decisions @@ -828,7 +911,7 @@ ruling, all are content-completeness for the executor): well?") — macOS is supported, and the dependency ordering (OQ-5) is now ruled below. - **OQ-5 (macOS service dependency ordering) — RULED: ship supervision now, - runtime-agnostic** (Matt, RIG-3239, 2026-09-05): "we'd use a launchd + runtime-agnostic** (Matt, session directive, 2026-09-05): "we'd use a launchd service for the stack, unrelated to it running on podman/apple container." The launchd LaunchAgent supervises the `compass-stack` host process independent of the runtime inside it, so macOS supervision is NOT