fix(service-cluster,cli): multi-node gate fails closed when unregistered, and mounts on every boot route (#14114)

* fix(service-cluster,cli): multi-node gate fails closed when unregistered; mount it on every boot route

Half 1 — the default direction: checkMultiNodeAllowed with no registered
gate now refuses a DECLARED multi-node topology (requested > 1) instead of
silently allowing a licensed capability; undeclared / single-replica input
keeps the historical allow, and a registered gate's verdicts are unchanged.

Half 2 — sink the mount: new mountMultiNodeGateFromHost in service-cluster
imports the distribution's gate-carrier packages through the boot surface's
host-anchored importer, so registration no longer depends on one app config
file executing; os serve calls it before consulting the gate on the
cluster-driver route (declared domain:cli half).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016ZC5rNQj3WEet5HAmmAkMs

* chore(changeset): ADR-0087 disposition for the gate default change

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016ZC5rNQj3WEet5HAmmAkMs

* test(service-cluster): pin what a no-gate denial actually does at boot (#14116)

The gate's module doc promised "the caller downgrades to single-node — never
bricks". Measured on #14116 that is false for the only deployment shape that
can reach the new fail-closed refusal at all: the refusal's trigger
(`requested > 1`) is the same operator declaration the split-brain guard keys
off, so the in-process fallback serve leaves behind is then refused and the
boot stops.

Pins the whole chain from the real pieces rather than restating prose:
`defineCluster({})` resolves `memory` (what Runtime builds when serve leaves
clusterConfig unset), and the guard throws for that driver at replicas 3 while
staying quiet at 1 — the genuine downgrade case, kept distinct so the two are
never conflated again. The prose in `multi-node-gate.ts`,
`multi-node-gate-mount.ts`, `serve.ts` and the changeset is corrected to match,
including the operator-facing boot warning, which said "downgrading to
single-node" while the process was about to refuse.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Jack Zhuang <50353452+hotlong@users.noreply.github.com>

Author: 3 people

.changeset/multi-node-gate-fail-closed-mount.md

@@ -0,0 +1,55 @@
+---
+"@objectstack/service-cluster": minor
+"@objectstack/cli": patch
+---
+
+fix(service-cluster,cli): the multi-node gate fails closed when unregistered, and is mounted on every boot route (#13537)
+
+**BREAKING behaviour narrowing on a licensed capability, shipped as `minor`
+under the repo's launch-window convention for breaking changes.**
+
+Multi-node clustering is a paid capability (maintainer ruling 2026-08-31,
+recorded on #13537). Two defects together made its authorization gate
+unenforceable by construction — measured on a real thin-extension EE
+deployment, a `maxNodes: 1` trial license booted 3 replicas with full cluster
+coordination and no warning (cloud#1752):
+
+- `checkMultiNodeAllowed` **defaulted to ALLOW when no gate was registered**,
+  so every boot route that skipped the one config file wiring the gate ran an
+  unlicensed cluster silently.
+- `registerMultiNodeGate` was reachable from exactly **one** mount point (the
+  EE app config, cloud repo), which the thin-extension and `OS_ARTIFACT_URL`
+  artifact-direct boot routes never execute.
+
+Both halves change:
+
+- **Fail-closed default** (`@objectstack/service-cluster`): with no gate
+  registered, a DECLARED multi-node topology (`requested > 1`) is now
+  **refused** — `os serve` drops the remote driver and warns loudly.
+  ⛔ Read the boot outcome precisely: with a multi-node topology declared, the
+  in-process fallback then trips the split-brain guard and the boot is
+  **REFUSED**, not quietly degraded (measured on #14116; the guard's trigger
+  and this default's trigger are the same declaration). The refusal is the
+  correct outcome — N replicas on per-process locks is the silent split-brain
+  that guard exists to stop — but it is a refusal, and an operator upgrading
+  into this default must be told so. An undeclared or single-replica count (`OS_CLUSTER_REPLICAS`
+  unset, `1`, or meaningless) keeps the historical allow: it declares no
+  multi-node topology, so there is nothing to gate. A registered gate's
+  verdicts are byte-identical to before — entitled deployments are untouched.
+  New exports: `hasMultiNodeGate()`, `MULTI_NODE_NO_GATE_REASON`.
+- **Route-independent mounting** (`mountMultiNodeGateFromHost`, new): the boot
+  surface about to consult the gate hands over its host-anchored importer and
+  the helper loads the distribution packages that carry the gate
+  (`MULTI_NODE_GATE_CARRIER_PACKAGES`), so registration no longer depends on
+  one app config file executing. `os serve` now calls it before the consult
+  (`@objectstack/cli`), best-effort: with no distribution installed nothing
+  mounts and the fail-closed default answers.
+
+**Migration.** A deployment that ran `OS_CLUSTER_DRIVER` (non-memory) with
+`OS_CLUSTER_REPLICAS > 1` and **no** registered gate was running an
+unlicensed multi-node topology on the old fail-open default; it now downgrades
+to single-node at boot and logs the refusal. Deploy a distribution that
+registers the gate (at module load of a carrier package, so every boot route
+mounts it), or remove the multi-node declaration.
+
+<!-- adr-0087: not-required (no-migration-prescription) A runtime default-direction change on the multi-node authorization gate: no spec key is removed, renamed or re-shaped, so there is no tombstone and nothing mechanical for `objectstack migrate meta` to rewrite. The channel that reaches an affected operator is the boot-time refusal itself (`os serve` logs `MULTI_NODE_NO_GATE_REASON` with the remedy, and a declared multi-node topology then stops the boot at the split-brain guard rather than degrading silently); whether to deploy a gate-registering distribution or drop the multi-node declaration is a deployment decision no migration entry can perform. -->

packages/cli/src/commands/serve.ts

@@ -2422,8 +2422,19 @@ export default class Serve extends Command {
       const __clusterDriver = process.env.OS_CLUSTER_DRIVER?.trim();
       if (__clusterDriver && __clusterDriver !== 'memory') {
         // Multi-node authorization gate (open mechanism): a distribution (e.g.
-        // an EE license) may deny multi-node. On denial, downgrade to
-        // single-node rather than fail — multi-node is an add-on, never brick.
+        // an EE license) may deny multi-node. On denial this file drops the
+        // remote driver and leaves `clusterConfig` unset, so the Runtime falls
+        // back to the in-process driver.
+        //
+        // ⛔ That is NOT the same as "the boot survives", and this comment used
+        // to say it was ("never brick"). Measured on #14116: when the operator
+        // ALSO declared a multi-node topology (`OS_CLUSTER_REPLICAS > 1` /
+        // `OS_EXPECT_MULTI_NODE=true`), the in-process fallback then trips the
+        // split-brain guard in `ClusterServicePlugin.init` and the boot is
+        // REFUSED. The refusal is correct — N replicas on per-process locks is
+        // exactly the silent corruption that guard exists to stop — but a
+        // denial and a declared topology together mean refuse, not degrade.
+        // The warning below says so.
         // Dynamic, non-literal specifier so the CLI does not statically depend
         // on the cluster package (mirrors the remote-driver import below).
         //
@@ -2443,8 +2454,32 @@ export default class Serve extends Command {
           checkMultiNodeAllowed: (requested?: number) => MultiNodeGateVerdict;
           /** Optional: an app on a pre-#13330 `service-cluster` does not have it. */
           listClusterDrivers?: () => string[];
+          // Optional: an app may pin an older service-cluster that predates
+          // the mount helper (#13537); `?.` below keeps that boot walking.
+          mountMultiNodeGateFromHost?: (
+            importer: (specifier: string) => Promise<unknown>,
+          ) => Promise<unknown>;
         };
-        const { checkMultiNodeAllowed } = __clusterModule;
+        const { checkMultiNodeAllowed, mountMultiNodeGateFromHost } = __clusterModule;
+        // [#13537] Mount the distribution's gate on EVERY boot route, BEFORE
+        // consulting it. Registration used to depend on one app config file
+        // executing (the EE config calling `registerMultiNodeGate`), so the
+        // thin-extension and artifact-direct routes booted with no gate at
+        // all — and the gate then defaulted to allow. The helper imports the
+        // gate-carrying distribution packages through this file's own
+        // host-anchored importer (passed as a value, so every carrier load
+        // resolves from the served app per #4719 — same guarantee as the
+        // `importFromHost` call above, just exercised inside the package that
+        // owns the carrier list). Best-effort: with no distribution installed
+        // nothing mounts, and the gate's fail-closed default answers below.
+        //
+        // ⚠️ Resolved against #13330's namespace read on main: the mount helper
+        // is destructured from THE SAME `__clusterModule`, so the instance that
+        // registers is provably the instance `checkMultiNodeAllowed` and
+        // `listClusterDrivers` are read from. Re-importing the package for the
+        // mount would have re-opened the split this file just closed.
+        try { await mountMultiNodeGateFromHost?.(importFromHost); }
+        catch { /* never brick the boot for an add-on — the check below fails closed */ }
         // Ask the gate about the topology the operator actually DECLARED.
         // Calling zero-arg leaves `requested` undefined, which a cap-aware gate
         // has nothing to clamp against — so the licensed-overflow verdict was
@@ -2468,7 +2503,11 @@ export default class Serve extends Command {
         if (!__gate.allowed) {
           console.warn(
             `[cluster] multi-node not authorized (${__gate.reason ?? 'denied'}) — ` +
-            `downgrading to single-node (in-memory cluster). Remove OS_CLUSTER_DRIVER to silence.`,
+            `falling back to the in-process cluster driver. If this deployment ALSO ` +
+            `declared a multi-node topology (OS_CLUSTER_REPLICAS>1 / OS_EXPECT_MULTI_NODE=true), ` +
+            `the split-brain guard refuses the boot rather than run replicas on per-process ` +
+            `locks — drop the declaration to run single-node, or license the capability. ` +
+            `Remove OS_CLUSTER_DRIVER to silence.`,
           );
         } else {
           // Licensed-overflow advisory: the cluster IS entitled to run, it just

packages/services/service-cluster/src/index.ts

@@ -77,8 +77,20 @@ export type {
 export {
     registerMultiNodeGate,
     checkMultiNodeAllowed,
+    hasMultiNodeGate,
+    MULTI_NODE_NO_GATE_REASON,
     __resetMultiNodeGate,
     type MultiNodeGate,
     type MultiNodeVerdict,
     type ResolvedMultiNodeVerdict,
 } from './multi-node-gate.js';
+
+// [#13537] Route-independent gate mounting: a boot surface hands its
+// host-anchored importer over so the distribution's gate is mounted on EVERY
+// boot route, not only where one app config file executes.
+export {
+    mountMultiNodeGateFromHost,
+    MULTI_NODE_GATE_CARRIER_PACKAGES,
+    type MultiNodeGateMountAttempt,
+    type MultiNodeGateMountReading,
+} from './multi-node-gate-mount.js';

packages/services/service-cluster/src/multi-node-gate-mount.test.ts

@@ -0,0 +1,105 @@
+// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
+import { describe, it, expect, afterEach } from 'vitest';
+import { PLATFORM_PLUGIN_WIRED_RUNTIMES } from '@objectstack/spec';
+import {
+    registerMultiNodeGate,
+    checkMultiNodeAllowed,
+    __resetMultiNodeGate,
+} from './multi-node-gate.js';
+import {
+    mountMultiNodeGateFromHost,
+    MULTI_NODE_GATE_CARRIER_PACKAGES,
+} from './multi-node-gate-mount.js';
+
+afterEach(() => __resetMultiNodeGate());
+
+/** An always-allowing gate, standing in for a distribution's licence check. */
+const FAKE_GATE = { allowMultiNode: () => ({ allowed: true, reason: 'fake licence' }) };
+
+describe('mountMultiNodeGateFromHost', () => {
+    it('registers via a carrier whose module load registers, and stops there', async () => {
+        const imported: string[] = [];
+        const reading = await mountMultiNodeGateFromHost(async (specifier) => {
+            imported.push(specifier);
+            // The carrier contract: registration is a SIDE EFFECT of module
+            // load. The fake registers on first import, like a real carrier
+            // whose module scope calls `registerMultiNodeGate`.
+            registerMultiNodeGate(FAKE_GATE);
+            return {};
+        });
+        expect(reading).toEqual({
+            alreadyRegistered: false,
+            registered: true,
+            attempts: [{ package: MULTI_NODE_GATE_CARRIER_PACKAGES[0], outcome: 'registered' }],
+        });
+        // Mount done after the first carrier — the second is never imported.
+        expect(imported).toEqual([MULTI_NODE_GATE_CARRIER_PACKAGES[0]]);
+        // And the mounted gate is the one the consult now reads.
+        expect(checkMultiNodeAllowed(3)).toMatchObject({ allowed: true, reason: 'fake licence' });
+    });
+
+    it('reports every carrier unavailable when none resolves, and stays unregistered', async () => {
+        const reading = await mountMultiNodeGateFromHost(async (specifier) => {
+            throw new Error(`Cannot find package '${specifier}'`);
+        });
+        expect(reading.alreadyRegistered).toBe(false);
+        expect(reading.registered).toBe(false);
+        expect(reading.attempts).toEqual(
+            MULTI_NODE_GATE_CARRIER_PACKAGES.map((pkg) => ({
+                package: pkg,
+                outcome: 'unavailable',
+                error: `Cannot find package '${pkg}'`,
+            })),
+        );
+        // The open-core outcome: nothing mounted, so the fail-closed default
+        // answers the consult that follows (#13537).
+        expect(checkMultiNodeAllowed(3).allowed).toBe(false);
+    });
+
+    it('never throws: a rejecting importer becomes an `unavailable` attempt', async () => {
+        await expect(
+            mountMultiNodeGateFromHost(async () => {
+                throw 'not-an-Error'; // eslint-disable-line no-throw-literal
+            }),
+        ).resolves.toMatchObject({
+            registered: false,
+            attempts: expect.arrayContaining([
+                expect.objectContaining({ outcome: 'unavailable', error: 'not-an-Error' }),
+            ]),
+        });
+    });
+
+    it('records a carrier that loads without registering (the #13330 split, or an old carrier)', async () => {
+        const reading = await mountMultiNodeGateFromHost(async () => ({}));
+        expect(reading.registered).toBe(false);
+        expect(reading.attempts).toEqual(
+            MULTI_NODE_GATE_CARRIER_PACKAGES.map((pkg) => ({
+                package: pkg,
+                outcome: 'loaded-without-gate',
+            })),
+        );
+    });
+
+    it('does not import anything when a gate is already registered', async () => {
+        registerMultiNodeGate(FAKE_GATE);
+        const imported: string[] = [];
+        const reading = await mountMultiNodeGateFromHost(async (specifier) => {
+            imported.push(specifier);
+            return {};
+        });
+        expect(reading).toEqual({ alreadyRegistered: true, registered: true, attempts: [] });
+        expect(imported).toEqual([]);
+    });
+
+    it('names only real, roster-declared distribution runtimes as carriers', () => {
+        // Drift guard (#10921): every carrier must be a package the spec
+        // roster declares as a real out-of-repo `plugins[]`-wired runtime —
+        // a fabricated name would sit here looking identical and simply never
+        // resolve. The list itself is owned by the mount module (the roster
+        // is provenance, not a resolution registry, by its own contract).
+        for (const pkg of MULTI_NODE_GATE_CARRIER_PACKAGES) {
+            expect(PLATFORM_PLUGIN_WIRED_RUNTIMES[pkg]).toBeDefined();
+        }
+        expect(MULTI_NODE_GATE_CARRIER_PACKAGES.length).toBeGreaterThan(0);
+    });
+});