From 6c28b1b75b990921911ee36d03f76e582803b7bd Mon Sep 17 00:00:00 2001 From: Dat Nguyen Date: Tue, 25 Aug 2026 09:55:10 +0000 Subject: [PATCH 01/45] docs: add the team hub design and threat model Reverses three documented non-goals - no daemon, no signal bus or inbox, no raw transcript upload - and records what each one cost. Covers component boundaries, wire protocol, identity, storage, the inbox, the web UI, failure behavior, a phased plan, and the threat model behind all of it. Four decisions are recorded with the losing argument kept rather than dropped: D-1 the push is the sharing decision. Sync is opt-in per repo, pushed run events are team-readable, briefs stay private to the sender until shared. D-1b the acceptor names the graph. An inbox message reaches an agent only as pre-fenced data, never as the prompt that chose the task. D-3 SQLite is the hub's truth; JSONL is a derived export. Revision 1 had it backwards and claimed append-only was a physical property of a file, which is false - sed -i disproves it. The laptop's event log is unchanged. D-4 no central conversation store, on measured evidence: 61% of 80 sampled transcripts on the author's machine carry a credential shape scan.ts already recognises, and that is a floor, not an estimate. Nothing under src/ is built yet. Three questions in section 14 need answering before phase 1, one of which - where the brief encryption key is wrapped - has no option that is both unattended-restartable and safe from a root operator. Co-Authored-By: Claude Opus 5 --- docs/hub-design.md | 952 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 952 insertions(+) create mode 100644 docs/hub-design.md diff --git a/docs/hub-design.md b/docs/hub-design.md new file mode 100644 index 0000000..c779861 --- /dev/null +++ b/docs/hub-design.md @@ -0,0 +1,952 @@ +# The team hub — design and threat model + +Status: **proposal, not built.** Nothing in `src/hub/` exists yet. +Author decisions locked before this document was written are in [§1](#1-what-is-locked). +Open questions for the author: [§14](#14-open-questions-for-the-author). + +**Revision 2.** Two questions were put to an adversarial pair of reviews: should the hub +hold a database, and should it hold full conversations? The answers went in opposite +directions and both changed this document. Storage is now SQLite-as-truth +([§6](#6-hub-storage), decision D-3) — revision 1 was wrong, and the specific error is +recorded rather than quietly fixed. Central conversation storage is refused +([§18](#18-decision-record-d-3-and-d-4), decision D-4), on measured evidence from a real +transcript corpus. Three defects the pro-database review found in revision 1's briefs are +fixed: owner-curated brief depth ([§7.2](#72-brief-depth-is-the-owners-choice)), +redaction-on-read ([§7.3](#73-redaction-on-read)), and the admission that this design's +top-ranked threat is uninvestigable with the data it retains +([§9.6](#96-what-this-design-cannot-investigate)). Federated search replaces the central +corpus as the answer to deep search ([§17](#17-federated-search)). + +--- + +## 0. What stops being true + +Three sentences the project can say today, and will not be able to say after this ships: + +| Today | After the hub | +| --- | --- | +| "Not a workflow server. No daemon, no web UI, no cloud." | There is a daemon and a web UI. | +| "No signal bus, inbox, or daemon… an inbox that starts an agent on someone else's laptop is a different product with a much harder threat model." | This is that product. | +| Content leaves a machine only through a print-once, expiring link a human chose to send. | Content leaves on a schedule, to a box, and stays there. | + +That third one is the real change, and the second one is the warning being cashed. The +README already reasoned its way to *not* building this and named exactly why. Building it +is legitimate — it is the author's project and the author's call — but the "much harder +threat model" has to be actually built, not waived by the phrase "approval-gated." Most of +[§9](#9-threat-model) exists because approval-gating answers a question the attack does not ask. + +**What does not change:** loomgraph still makes zero model calls, still shells out to the +agent CLI you installed, and **the hub never runs an agent.** It stores and routes. Agents +run on member machines, under the member's own sandbox, started by the member. + +--- + +## 1. What is locked + +Decided by the author before design; not relitigated here. + +1. **The inbox is approval-gated.** An inbound message never executes anything. It queues. + A human runs `lg inbox accept ` for anything to happen. No auto-dispatch in v1. +2. **The hub ingests run events and distilled extractive briefs only.** No raw transcripts, + under any flag. `src/handoff/scan.ts` is a hard gate on ingest. +3. **One central hub** on a shared server, per-member bearer tokens, each member's CLI + talks to it. + +Locked decision 3 is the single largest risk multiplier in this design: it converts +per-laptop compromise into team-wide compromise, and creates aggregation leaks no +per-brief scanner can see ([§9.4](#94-what-the-scanner-stops-buying-under-retention)). It stays, +so the rest of the architecture compensates: hold as little as possible, never execute on +the hub, and put the real anti-injection defense on the *consuming* machine. + +--- + +## 2. Shape, in one paragraph + +The hub does to the team what `events.jsonl` already does to a run: a server-stamped, +append-only record of what happened, greppable via `lg-hub export --jsonl` and queryable via +SQL. The client's existing event log doubles as the push outbox, so there is no queue and no +spool directory. The existing pure-renderer pattern doubles as the web UI, so there is no +build step and no client JS. `src/handoff/scan.ts` sits in front of everything that ingests +text, and now in front of everything that serves it too ([§7.3](#73-redaction-on-read)). +Only the hub's own storage is a new idiom; the rest is the existing three pointed at a +network. + +--- + +## 3. Component boundaries + +**A third binary, `lg-hub`.** Not folded into `lg` — a long-running daemon would poison +`lg`'s "you run it and it exits" contract. Not folded into `lg-handoff` — that subtree's +import purity is load-bearing. + +``` +src/hub/ the lg-hub bin -> dist/hub/cli.js + cli.ts commander wiring; the only argv reader + server.ts node:http binding; deliberately dumb (see §12 on testing) + handlers.ts (WireRequest, deps) -> WireResponse; where all behavior lives + auth.ts token hashing, member resolution, revocation + storage.ts HubStore: node:sqlite, WAL, one transaction per batch (§6) + inbox.ts message lifecycle transitions + feed.ts feed partitioning and cursor logic + ui/*.ts pure (data) -> html renderers +src/team/ client side + transport.ts the injected Fetch seam (mirrors handoff's Exec seam) + sync.ts cursor logic over events.jsonl + fence.ts untrusted-content fencing <- security-critical, see §8.3 +src/commands/ new thin files: enroll.ts, sync.ts, inbox.ts, wired into src/cli.ts +``` + +**Import rule.** `src/hub/` may import `src/core/` and may import `src/handoff/scan.ts` +one-way. `src/handoff/` still imports nothing outward, so AGENTS.md's rule holds as +written. **Do not copy the scanner.** The `buildEnclavePushArgs`-exists-twice precedent is +for a 20-line argv builder; a security gate must never fork. AGENTS.md needs a line saying +which direction the new arrow points. + +Note what this costs: `src/handoff/types.ts:1-8` keeps the subtree extractable "once a team +fabric exists outside this repo." The fabric now exists *inside* it, so extraction later +means the hub depends on the extracted sibling. Acceptable, but the comment should be +updated rather than left to quietly become false. + +**Exit codes.** `lg`'s team verbs reuse its namespace — sync/inbox failure is `2`, never +`3` or `4`, which stay budget and paused. `lg-hub` gets its own small namespace, per the +`lg-handoff` precedent: `0` clean exit, `1` config or usage, `2` fatal runtime (port bind, +corrupt data dir). + +--- + +## 4. Wire protocol + +HTTP/1.1 + JSON on `node:http`. Client uses global `fetch` (Node ≥ 22) behind the seam. +Zero new dependencies. `Authorization: Bearer ` on everything except `/v1/health`. + +| Endpoint | Method | Notes | +| --- | --- | --- | +| `/v1/health` | GET | unauthenticated; `{ok, version}` | +| `/v1/events` | POST | `{runId, streamId, graphName, state, events[]}`, ordered by `seq` | +| `/v1/briefs` | POST | the four bundle files inline as strings | +| `/v1/feed?after=&limit=50` | GET | newest-first page + `nextCursor`; keyset, see [§6.2](#62-pagination-is-keyset-not-byte-offsets) | +| `/v1/runs/:member/:runId` | GET | stored state + events | +| `/v1/inbox` | POST | send; schema in [§8](#8-the-inbox) | +| `/v1/inbox?state=queued` | GET | addressee is always the authenticated member | +| `/v1/inbox/:id/transition` | POST | `{to, runId?}` | +| `/v1/admin/members` | POST | enroll; admin token only | + +**Idempotency uses natural keys, not an `Idempotency-Key` header.** Events already carry +`(runId, seq)` from `src/core/events.ts`. The hub keys them `(member, streamId, runId, seq)` +where `member` comes from the token and **never** from the body. A per-run high-water mark +acks-and-drops anything at or below it. Same seq with different content is `409` plus a +visible feed item — silence about divergence is worse than noise. Briefs are keyed by +`sha256(handoff.md)`. Inbox messages carry a client `crypto.randomUUID()`. + +**Reserve `streamId` in phase 1 even though nothing reads it yet.** It is a random id +minted at `run_started`. Without it, `(runId, seq)` assumes one machine and one history per +run forever; a wiped `.loomgraph`, a copied repo directory, or any future multi-machine +resume produces same-key-different-content and the 409 policy fires noise exactly when the +user is already confused. Reserving the field now is free. Retrofitting it after real data +exists is not. + +**The outbox already exists — do not build one.** `.loomgraph/runs//events.jsonl` is +append-only and unbuffered by hard rule, which is the definition of a durable outbox. Sync +is a cursor over it: + +- `.loomgraph/sync/.cursor` holds the last acked seq, written temp-then-rename. +- `lg run` / `lg resume` hook the **existing** `onEvent` callback already threaded through + `EngineDeps` and used in `src/commands/run.ts`. Batch every 10 events or 5 s, 1500 ms + timeout, and **any failure is one line on stderr and nothing else.** A hub outage cannot + affect a run, its checkpoints, or its exit code. +- `lg sync [runId]` replays from the cursor. **This is the only path that must be correct.** + The live push is best-effort sugar over it. + +The cursor advances only on a 2xx naming `highWaterSeq`, so a cut connection just resends. + +**Ordering is by hub `receivedAt`, never by client `ts`.** Client clocks skew; the feed is +served from the hub's own arrival order. Client timestamps are displayed and labeled as +reported, and no cursor is ever derived from one ([§6.2](#62-pagination-is-keyset-not-byte-offsets)). + +--- + +## 5. Identity and auth + +Enrollment is admin-mediated and print-once, matching the enclave share-link aesthetic the +project already lives with: + +``` +# on the hub host +$ lg-hub member add alice +lgt_a1b2c3d4.<32 bytes base64url> # printed once, never recoverable + +# on alice's machine +$ lg enroll https://hub.internal lgt_a1b2c3d4.xxxx +wrote ~/.config/loomgraph/hub.json (0600) +``` + +The hub stores only `{member, keyId, tokenHash: sha256(secret), createdAt}`, appended to +`members.jsonl`. `LOOMGRAPH_HUB_URL` / `LOOMGRAPH_HUB_TOKEN` override the file, the same +pattern as `ENCLAVE_TOKEN`. + +**Attribution is server-side, always.** Every stored record gets `member` stamped from the +token's keyId. `HandoffMeta.createdBy` — currently `userInfo().username` in +`src/handoff/commands.ts` — is displayed as *"claims created-by"* at most. Never trust a +client-supplied owner field; that is forgery vector A5. + +**Revocation** appends `{revoked: keyId, ts}` to `members.jsonl`, replayed at startup and +on SIGHUP. A ten-person member file is trivially small. + +**Add a scan rule for the hub token shape before the first token is minted.** This is +non-optional and easy to forget. Members will paste tokens into shells and configs; agents +will read those shells; `lg-handoff pack` will faithfully distil a session quoting one. +The pipeline is *designed* to republish exactly this, and `SCAN_RULES` in +`src/handoff/scan.ts` has no rule for a shape that does not exist yet. Choose the `lgt_` +prefix, add the rule in the same commit. + +**Transport.** `lg-hub serve` refuses to bind a non-loopback address unless +`--behind-tls-proxy` is passed, and says why. Deploy behind Caddy, or on a WireGuard or +Tailscale interface. Bearer tokens over plaintext LAN HTTP are precisely the credential +class `scan.ts`'s `auth-header` rule exists to catch; the project should not ship the +vulnerability its own scanner names. + +--- + +## 6. Hub storage + +**SQLite (WAL, `node:sqlite`) is the hub's truth. JSONL is a derived export.** + +Revision 1 said the opposite, and was wrong in a way worth recording rather than silently +correcting. + +### 6.1 Why revision 1 was wrong + +Revision 1 did not choose JSONL *over* SQLite. It chose **both**: JSONL as truth, plus a +SQLite index, plus `lg-hub reindex`, plus a phase-4 test proving the rebuild was +byte-identical. That is two storage engines, two write paths, and a consistency proof +between them — assembled to avoid one engine that ships inside Node 22. Simplicity was the +stated goal and was not what the design delivered. + +Three specific errors: + +- **"Append-only is a physical property" was false.** Nothing physically prevents `sed -i` + on a `.jsonl` file. Append-only-ness of a file is discipline too. SQLite enforces it + *harder*, because the prohibition can be declared: + `CREATE TRIGGER … BEFORE UPDATE ON events BEGIN SELECT RAISE(ABORT,'append-only'); END;` +- **Tamper-evidence against the hub operator was zero in both designs.** An operator with + root rewrites a JSONL line as easily as a row. The real mechanism is a hash chain, which + revision 1 did not have; it is now a column. +- **`EventLog.read` skipping unparseable lines is correct on a laptop and wrong as server + truth.** `src/core/events.ts` states the intent plainly — "A torn or corrupt line must + not take down the audit trail" — which on a laptop is graceful degradation. As the + server's only copy it means a torn line silently deletes an event from history, and a + rebuild bakes the loss in. Loud corruption beats silent loss. + +**What does not change: the laptop.** `.loomgraph/runs//events.jsonl` stays exactly +as it is, unbuffered and append-only. AGENTS.md's invariant is about the run log, and the +run log is where the greppable-log property actually lives; revision 1 mistakenly read that +rule as binding on a component that did not exist when it was written. The hub is a +different component with a different job. **AGENTS.md needs one added line saying so**, and +saying that the hub's greppable artifact is a derived export, not its truth. + +### 6.2 Pagination is keyset, not byte offsets + +Revision 1's cursor was `base64({day, offset})` — a byte offset into a day-partitioned +file, handed to clients who hold it indefinitely. That is a public API made of the wrong +material: + +- `reindex`, the design's own recovery mechanism, invalidated every outstanding cursor + unless the rebuild was bit-perfect — which is precisely why that test had to exist. The + escape hatch and the pagination scheme were at war. +- Tombstoning ([§11](#11-deletion--decision-d-2-revised)) shifts offsets, and compaction was + rejected, so clients would page through tombstones forever. +- **A wrong byte offset is undetectable.** The client lands mid-line, or skips items, or + repeats them, and nothing errors. + +Cursors are now keyset over `(received_at, rowid)`, which survives rebuilds, retention +purges, schema evolution and reordering. + +### 6.3 Schema + +```sql +PRAGMA journal_mode=WAL; +PRAGMA foreign_keys=ON; + +-- the verbatim client line is kept in `json`, so the export in §6.4 is lossless +CREATE TABLE events ( + member TEXT NOT NULL, stream_id TEXT NOT NULL, run_id TEXT NOT NULL, seq INTEGER NOT NULL, + received_at TEXT NOT NULL, kind TEXT NOT NULL, node_id TEXT, + json TEXT NOT NULL CHECK (json_valid(json)), + prev_hash BLOB, row_hash BLOB NOT NULL, + PRIMARY KEY (member, stream_id, run_id, seq) +) WITHOUT ROWID; +CREATE INDEX events_feed ON events(received_at); + +CREATE TABLE runs ( + member TEXT NOT NULL, run_id TEXT NOT NULL, stream_id TEXT NOT NULL, + graph_name TEXT, state_json TEXT, high_water_seq INTEGER NOT NULL, updated_at TEXT NOT NULL, + PRIMARY KEY (member, run_id)); + +CREATE TABLE briefs ( + brief_id TEXT PRIMARY KEY, member TEXT NOT NULL, sha256 TEXT UNIQUE NOT NULL, + received_at TEXT NOT NULL, expires_at TEXT, revoked_at TEXT, + key_id TEXT REFERENCES item_keys(key_id), -- null only if encryption is off + handoff_md BLOB, meta_json BLOB, html BLOB); +CREATE TABLE brief_files (brief_id TEXT, path TEXT, PRIMARY KEY (brief_id, path)); +CREATE TABLE brief_shares (brief_id TEXT, grantee TEXT, granted_at TEXT, revoked_at TEXT, + PRIMARY KEY (brief_id, grantee)); + +CREATE TABLE inbox ( + id TEXT PRIMARY KEY, from_member TEXT NOT NULL, to_member TEXT NOT NULL, + subject TEXT, body TEXT, re_json TEXT, proposed_graph TEXT, + state TEXT NOT NULL, created_at TEXT NOT NULL); +CREATE TABLE inbox_history (id TEXT, to_state TEXT, ts TEXT, by TEXT, run_id TEXT); + +CREATE TABLE members ( + key_id TEXT PRIMARY KEY, member TEXT NOT NULL, token_hash TEXT NOT NULL, + scopes TEXT NOT NULL, created_at TEXT NOT NULL, revoked_at TEXT); +CREATE TABLE sessions (sid TEXT PRIMARY KEY, member TEXT, expires_at TEXT); +CREATE TABLE read_marks (member TEXT, kind TEXT, ref TEXT, read_at TEXT, + PRIMARY KEY (member, kind, ref)); +CREATE TABLE access_log (ts TEXT, member TEXT, action TEXT, ref TEXT); +CREATE TABLE item_keys (key_id TEXT PRIMARY KEY, wrapped_key BLOB NOT NULL); + +CREATE VIRTUAL TABLE search USING fts5(member, kind, ref, text); + +CREATE TRIGGER events_no_update BEFORE UPDATE ON events + BEGIN SELECT RAISE(ABORT, 'events is append-only'); END; +CREATE TRIGGER events_no_delete BEFORE DELETE ON events + BEGIN SELECT RAISE(ABORT, 'events is append-only'); END; +``` + +Everything revision 1 hand-rolled becomes a declared constraint: the high-water mark is a +primary key, 409-on-divergence is `INSERT OR IGNORE` plus a compare-on-conflict, the feed +is an index, receipts are columns, the members-file replay is a table, the §11 tombstone is +`revoked_at`. One ingest batch is one transaction — it happened or it did not — replacing +revision 1's four-file interleaving that had to be reasoned about by hand. + +`row_hash = sha256(prev_hash || json)`, with the chain head published to members +periodically. This is the tamper-evidence revision 1 claimed from file semantics and did +not actually have. + +### 6.4 The greppable artifact survives as an export + +`lg-hub export --jsonl` emits exactly revision 1's directory layout — `events.jsonl` per +run, one JSON object per line — reconstructed from the `json` column, which holds the +verbatim client line. The grep audience loses nothing; the query audience gains +`lg metrics`, full-text search, unread state, threading, revocable share grants and +multi-day range queries, none of which are reachable by walking files. + +**The law for AGENTS.md, inverted from revision 1:** the hub's database is truth; JSONL is +a rebuildable export. The laptop's `events.jsonl` is untouched and remains append-only. + +### 6.5 Operations + +- **Backup** is `VACUUM INTO 'snap.db'` — one statement, consistent. Revision 1's + "`rsync` the data directory" was a live copy of dozens of files mid-write. +- **Recovery** at 2am: `PRAGMA integrity_check`, then restore the last snapshot and replay + from members' local cursors, which are the real durable outbox ([§4](#4-wire-protocol)) + and are unaffected by hub state. +- **Migrations** are cheap because the event payload is an opaque verbatim `json` column; + new client fields need no `ALTER TABLE`. Only hub-side projections migrate. +- **Postgres is not warranted.** One process, ten members at most, embedded synchronous + access, zero-dependency ethos. Revisit at multiple hub nodes or roughly fifty members; + arguing for it now would only discredit the SQLite case. + +## 7. Visibility — decision D-1 + +**The two memos disagreed here, and this is the resolution.** + +The architecture memo said every member reads everything; right-sized for a small team. +The threat memo said private-by-default with explicit scoped sharing, because +`lg-handoff` is private-only and *refuses* `--visibility org`, and because a flat pool +means one leaked token or one XSS drains the whole team's briefs. + +**Resolution: the push is the sharing decision.** + +- Nothing reaches the hub that a member did not push. Sync is **opt-in per repository** + (`lg sync --enable` writes `.loomgraph/hub.json`), never on by default, never global. + A member who never enables sync is invisible to the hub, and that must stay true. +- **Run events, once pushed, are team-readable.** This is the monitoring feature the author + asked for, and enabling sync on a repo is the consent act. Making pushed runs private + would make the feature pointless. +- **Briefs are private to the sender until explicitly shared** to named members, revocably. + A brief is quoted session content — a different asset class from a status table. +- **An inbox message is readable only by its sender and its addressee.** No broadcast. + +So the threat memo wins on briefs and inboxes; the architecture memo wins on run events; +and the granularity that makes both defensible is per-repo opt-in rather than per-item +prompting. What was given up: a member cannot enable sync on a repo and then hide one +embarrassing run in it. Retraction ([§11](#11-deletion--decision-d-2-revised)) is the answer to +that, not per-run visibility flags. + +**Token scopes** (`ingest` / `read` / `admin`) are separate from this and should land by +phase 3. A CI token that pushes events should not be able to read everyone's briefs or send +inbox messages. + +### 7.1 What "conversations" means here — decision D-4 + +The author asked whether the hub should hold a database to share **all conversations** +between members. The database half is [§6](#6-hub-storage); this half is refused, and the +evidence is in [§18](#18-decision-record-d-3-and-d-4). The short form: on the author's own +machine, 61% of real agent transcripts contain a credential shape the existing scanner +already recognises, which is a floor rather than an estimate. Centralising transcripts +means roughly six in ten uploads carrying a known credential shape, permanently, on one +shared box, readable by everyone with a token. + +The counter-design — per-session opt-in, encryption at rest, short retention, access +logging, redaction-on-read — was argued well and defeated by its own requirement: server-side +search, redaction and rendering all need the hub to hold decryptable plaintext, so it +mitigates every threat except A4 while materially raising A4's payoff. Its author's summary +of the position was "I chose the honeypot." On a shared server, that is the wrong choice. + +**What is kept from that argument is [§7.2](#72-brief-depth-is-the-owners-choice), +[§7.3](#73-redaction-on-read), [§9.6](#96-what-this-design-cannot-investigate) and +[§17](#17-federated-search)** — because the objection that briefs are too thin was correct +even though the proposed remedy was not. + +### 7.2 Brief depth is the owner's choice + +Revision 1 inherited `lg-handoff`'s fixed extraction: `firstTurn(user)`, `lastTurn(assistant)`, +`lastTurn(user)`, plus a file list. A sixty-turn session becomes three quoted turns, and the +load-bearing one — Done — is **the agent's summary of its own work**, which the README's own +failure-mode section teaches you to distrust: Claude Code returns `subtype: "success"` with +`is_error: true` on a lapsed session, and a sandboxed verifier can report PASS having read +nothing. The brief keeps the claim and discards the evidence, then tells the reader to +verify every claim against the repo. + +That makes a fixed-shape brief the worst point on the curve: most of the retention risk, +little of the value. The fix is not more content by default — it is letting the person who +was there choose: + +```bash +lg-handoff pack claude --turns 12-31,44 --include-tool-result 27 --session-file +``` + +Still extractive, still no model call, still scanned, still owner-curated. What changes is +that the dead end at turn 23 — "we tried patching `disburse.ts` first and it broke +reconciliation" — can be carried, because that sentence is worth more to the next person +than the summary is. `--turns` without an explicit list keeps today's default. + +The turns and tool-result blocks a reader can request are bounded by what the reader +already extracts; **this does not loosen "the readers drop, they do not carry."** Adding a +field to `DistilledSession` still means deciding it is safe to publish. + +### 7.3 Redaction-on-read + +Scanning only at ingest means a rule added later protects nothing already stored. The +`lgt_` token rule from [§5](#5-identity-and-auth) is the worked example: any token that +leaked before that rule existed is exposed for as long as the store keeps it. + +So stored content is also served through `scanText` + `rewritePaths` masking **at egress**, +on every read path — API, web UI and export. Consequences worth stating: every rule added +in future retroactively protects all history; a finding at read time is logged and surfaced +to the owner rather than silently masked; and the ingest gate stays exactly as it is, since +egress masking is a second layer and not a replacement for refusing to store a secret. + +Cost: reads are no longer a straight file copy. At this data volume that is not a +performance question. + +--- + +## 8. The inbox + +### 8.1 Message schema + +``` +{ v: 1, id: uuid, from: , to: { member: "bob" }, + subject: string, body: string, + re: { member, runId } | { briefId } | null, + proposedGraph: { source: , vars: {...} } | null, + createdAt, state, history: [{to, ts, by, runId?}] } +``` + +**Addressing is person-only.** A repo has no owner who can approve; a run has no inbox. +Both exist only as the optional `re:` context reference. Repo-addressing is the feature +that quietly turns this into a dispatch system, so it is deliberately absent. + +**Ingest gates**, in this order, fail-closed, mirroring `pushCommand`: `scanText` over +`subject`, `body`, `proposedGraph.source` and every var value — reject with masked findings +on any hit; then `parseGraph` on any `proposedGraph`, rejecting invalid graphs at send time +so an acceptor never receives an unrunnable request. Validation stays loud, per AGENTS.md. + +**Lifecycle:** `queued → seen → accepted | declined | expired`, then +`accepted → done | failed`, reported by the acceptor's own sync. Only the addressee's token +may transition its own messages. + +### 8.2 What `accept` actually does — decision D-1b + +The memos disagreed here too. The architecture memo had `accept` write the sender's graph +via `saveGraphSource` and enter the normal `runCommand` path. The threat memo said a +message must never be able to name the task, because that is attack A2 with a green light. + +**Resolution: the acceptor names the graph. The message is only ever data inside it.** + +``` +$ lg inbox show 7f2a # mandatory reading step; see §8.3 +$ lg inbox accept 7f2a --graph ./graphs/triage.yaml +``` + +`--graph` points at a **local file the acceptor already has and trusts.** The message body +is exposed to that graph only as `{{inbox.body}}`, which is materialized pre-fenced +([§8.3](#83-fencing-is-the-load-bearing-control)). The sender's `proposedGraph` is inert by +default; running it requires `--use-proposed-graph`, which prints the full YAML plus +`renderPlan` and requires typing the message id to confirm. There is no flag that skips +`show`, and **there must never be an `--auto-accept`, a trusted-sender bypass, or an +accept triggered by an event.** + +`--cwd` is always the acceptor's. A message may name a repo *remote* as a suggestion and +can never name a local path. + +Inbox-sourced runs default to the most restricted sandbox available and never inherit +`workspace-write` or `bypass`. A message cannot name its own execution mode — sender-supplied +capability is the whole attack with permission attached. + +Progress flows back to the sender through ordinary event sync. No new mechanism. + +### 8.3 Fencing is the load-bearing control + +**This is the most important section in this document.** + +An inbox message is untrusted input authored by someone else's agent, which may itself have +been steered by a web page, a dependency README, or a PR body it read. Approval-gating is a +boolean on *ingestion*; the exploit is in *interpretation*. A human clicking accept is +saying "this looks like real work from a colleague," not auditing an instruction set they +were never shown as an instruction set. Habits decay into muscle memory within a week. + +So `src/team/fence.ts` wraps every inbox-sourced value before any agent CLI sees it: + +- an explicit, un-spoofable delimiter, with delimiter-lookalikes in the body neutralized; +- a preamble stating the content is untrusted third-party data and instructions inside it + are not to be followed; +- every line prefixed, reusing the discipline in `src/handoff/render.ts` — whose `quote()` + exists so "no line of transcript can break out of the quote," and which deliberately + ships **no markdown engine** because an inline-link parser is a way to smuggle + `javascript:` into a page. The same reasoning applies verbatim to inbox content. + +`lg inbox show` renders with the same fence: sender, source run, timestamp, and the entire +body — untruncated — inside a visible quarantine frame labeled *"untrusted message from +<member>; loomgraph did not write this and cannot vouch for it."* No link activation, +no markdown, escape everything. + +**Say plainly, in the README and in `show`'s own output, that fencing is mitigation and not +proof.** Nothing at the prompt layer is a hard boundary against a determined injection. +Fencing lowers the odds, the restricted sandbox bounds the blast radius, and the human is +the last check — the same posture the scanner section already takes. + +--- + +## 9. Threat model + +### 9.1 New trust boundaries + +Today there are two: transcript → readers (the narrowing boundary in +`src/handoff/readers/*.ts`), and bundle → enclave (scan, then constraints, then spawn, in +`pushCommand`). The hub adds: + +- **B1 member → hub.** Every run now has a network side effect, on a schedule, not per + human decision. +- **B2 hub → member.** Entirely new direction. `lg-handoff` has "No pull" as a design + point; this deletes it. +- **B3 member ↔ member, transitively.** Any teammate can author input to my machine. Since + teammates run agents, this is really: **anything any teammate's agent ever read** can + author input to my machine. +- **B4 browser ↔ hub.** A client class with cookies, a DOM, and adversarial text to render. +- **B5 storage at rest.** Aggregated team content, long-lived, on one box. A new asset class. +- **B6 hub operator and co-tenants.** Rooting my laptop gets you my sessions. Rooting the + hub gets you the team's. +- **B7 token custody.** A new secret on N machines — and one the handoff pipeline is built + to accidentally republish ([§5](#5-identity-and-auth)). + +### 9.2 Attack paths, ranked + +**A1 — prompt-injected teammate → my inbox → my agent. (High × Critical; not close.)** +Teammate's agent reads a poisoned page, is instructed to send a hub message, the message +queues, I accept because it reads like plausible colleague work, my agent consumes it as +instructions. Sender authenticated, transport intact, human approved: **every planned +control passes and the attack still lands.** Mitigated only by [§8.3](#83-fencing-is-the-load-bearing-control) +fencing + acceptor-named graph + restricted sandbox. This is what the README's warning was about. + +**A2 — malicious accept. (High × Critical.)** A1's mechanism restated, because +approval-gating is designed as the defense against it and does not defend against it. +Accept gates whether a message enters the workflow; it says nothing about what the message +says once in. Approval authorizes the topic; the payload is in the details. + +**A3 — compromised member token. (Medium × High.)** Possession equals identity, replayable +until noticed. Grants reads per [§7](#7-visibility--decision-d-1) plus forged messages to +every other member — feeding A1 from an authenticated sender, which clears reputation +checks. Uniquely here, the token can leak *through loomgraph's own handoff pipeline*. + +**A4 — compromised hub. (Low × Catastrophic.)** Read everything ingested, impersonate +anyone, inject into every inbox with no injection needed, rewrite the log. The mitigation is +not "trust the hub" — it is that the hub holds as little as possible and cannot itself +execute, which is why A1's real defense lives on the consuming machine. + +**A5 — replay or forgery of events. (Medium × Medium-High.)** Forged `run_finished`, +resurrected states, spoofed run ids. Corrupts the shared record and anything keyed off it. +Countered by server-side attribution and the natural-key high-water mark. + +**A6 — XSS from brief content. (Medium-High × High.)** Briefs are arbitrary quoted model +and user text by construction, rendered in an authenticated origin holding the team's data. + +**A7 — scanner miss, retained forever.** See below. + +### 9.3 Why the scanner still earns its place + +Keep it as a mandatory, **server-side, non-bypassable** ingest gate. It genuinely catches +URL-embedded credentials, `Authorization` headers, vendor-prefixed keys, JWTs, and +`TOKEN=`-style assignments, and — the part that matters most — `scanBundleDir` **fails +closed** via `UNREADABLE_RULE`, so "clean" means "looked and found nothing." Re-run it on +the server even when the client claims clean. + +### 9.4 What the scanner stops buying under retention + +Retention inverts the cost of a false negative. Under handoff a miss sat behind a link that +expired in 7 days and could be revoked. On the hub it sits indefinitely, readable by +everyone [§7](#7-visibility--decision-d-1) admits. Every named gap — AWS secret access +keys, header-less PEM bodies, hex client secrets, non-home absolute paths — becomes +permanent team-wide exposure instead of a week-long single-recipient one. + +And aggregation creates leak shapes that are in no single brief, which a line-oriented +single-file scanner structurally cannot see: + +- **The hub token itself**, until the rule from [§5](#5-identity-and-auth) exists. +- **Cross-brief correlation** — a hostname here, a username there, a ticket scheme in a + third. Individually beneath notice; together, a map of the team's infrastructure. +- **The org graph.** `meta.json` carries `createdBy`, `createdAt`, and + `repo.remote/sha/branch`. Across a team that is an accurate timestamped record of who + touched what in which private repo. No rule flags it, and it is exactly what a departing + employee or an attacker wants. +- **`files.txt` as a source-tree map.** The union of `filesTouched` sketches private + codebases' structure. + +Mitigation is minimization, not more rules: ingest the least identity that works, make +`repo.remote` and `files.txt` opt-in, and be able to actually delete. + +### 9.5 Web UI surface + +Requirements, all v1: contextual escaping on every interpolated value (the existing +`escapeHtml` in both renderers); **no markdown engine**, matching +`src/handoff/render.ts`'s stated rationale; no `innerHTML` on any brief-derived value; +strict CSP `default-src 'none'` with no inline script, so a missed escape cannot execute; +``, already present in the handoff page; +`X-Frame-Options: DENY` and `frame-ancestors 'none'` against clickjacking. + +Note `svg` is in `ENCLAVE_ALLOWED_EXTENSIONS` and SVG is an XSS vector (inline ` + + +`; From 02cb20f6dad2cb1e6908175e33974d6fba5ae867 Mon Sep 17 00:00:00 2001 From: Dat Date: Mon, 21 Sep 2026 08:52:49 +0700 Subject: [PATCH 35/45] fix(team): sanitise synced event lines so secrets never reach the hub `syncRun` pushed `.loomgraph/runs//events.jsonl` lines verbatim. The projection (`projectState`) is a genuine allowlist, but the event stream bypassed it entirely, so the hub received raw `node_finished.error` (the claude and codex adapters fold the agent's full result text and stderr into it), raw `run_finished.error`, `run_started.cwd` (home dir + username), the INTERPOLATED `human_requested.question` (`{{vars.x}}` and `{{nodes.x.output}}` already substituted) and the verbatim `human_resolved.answer`. The hub's `events` table has no-update/no-delete triggers, so a leak there is permanent and visible to every read-scoped member. Four fixes: - `EVENT_DATA_ALLOWLIST` + `sanitizeEventLine` in `src/team/sync.ts`: every event kind names its publishable `data` fields, each marked `pass` (engine identifiers, enums, numbers) or `text` (run through the same rewrite/mask/cap the projection applies). Unnamed fields are dropped; an unclassifiable line is dropped rather than passed through. Wired into `buildBatch`, not `syncRun`, because `buildBatch` is the one choke point both `lg sync` and the live `LiveBatcher` go through. The local log stays raw - sanitising at push time preserves "export reproduces ingested lines byte for byte". - `hostname` threaded through as a REQUIRED field of the new `ProjectionIdentity`. `rewritePaths` had always accepted it, no caller on the sync path supplied it, and the machine hostname published unrewritten for the whole of phase 1. Two separate opts declarations are how it went missing, so `ProjectionOpts` is now an alias rather than a second type. - `rejectControlInText` in `src/hub/wire.ts`: a node error is routinely a multi-line stack trace, and refusing the newline 400d the whole batch. The cursor only advances on a 2xx, so that run could never sync again. Tab, LF and CR are allowed in error text only; identity strings keep `rejectControl`. - `stripControl` in `src/team/project.ts`, running FIRST in `safeText`: many CLIs colour stderr, and an ESC would wedge sync the same way. It also defeats the masker - `sk-ant-api03[0m-XXXX` matches nothing, and stripping after the mask would reassemble the secret in clear. Order is strip -> rewrite -> mask -> cap and is pinned by a regression test; any other arrangement is exploitable. The root cause was that no test asserted anything about synced event content. `src/team/sync-redaction.test.ts` now does, covering each event kind's fields, the ordering property, and the wedge cases. 759 tests pass, typecheck clean. Co-Authored-By: Claude Opus 5 (1M context) --- src/commands/resume.ts | 4 +- src/commands/run.ts | 4 +- src/commands/sync.test.ts | 42 ++++- src/commands/sync.ts | 4 +- src/hub/wire.test.ts | 77 ++++++++ src/hub/wire.ts | 28 ++- src/team/batch.test.ts | 11 +- src/team/project.test.ts | 166 ++++++++++++++++- src/team/project.ts | 90 +++++++-- src/team/sync-redaction.test.ts | 315 ++++++++++++++++++++++++++++++++ src/team/sync.test.ts | 7 +- src/team/sync.ts | 146 +++++++++++++-- 12 files changed, 850 insertions(+), 44 deletions(-) create mode 100644 src/team/sync-redaction.test.ts diff --git a/src/commands/resume.ts b/src/commands/resume.ts index 53c1b77..6f41cfd 100644 --- a/src/commands/resume.ts +++ b/src/commands/resume.ts @@ -1,4 +1,4 @@ -import { homedir, userInfo } from "node:os"; +import { homedir, hostname, userInfo } from "node:os"; import { defaultRegistry } from "../adapters/registry.js"; import { execute, readySet } from "../core/engine.js"; import { parseGraph } from "../core/graph.js"; @@ -80,7 +80,7 @@ export async function resumeCommand( ctx: { runId, store, - opts: { home, username: userInfo().username, repoRoot: cwd }, + opts: { home, username: userInfo().username, repoRoot: cwd, hostname: hostname() }, } satisfies BatchCtx, }); diff --git a/src/commands/run.ts b/src/commands/run.ts index f7c1937..47d0f27 100644 --- a/src/commands/run.ts +++ b/src/commands/run.ts @@ -1,5 +1,5 @@ import { readFileSync } from "node:fs"; -import { homedir, userInfo } from "node:os"; +import { homedir, hostname, userInfo } from "node:os"; import { defaultRegistry } from "../adapters/registry.js"; import { execute, makeRunId, newRunState } from "../core/engine.js"; import { parseGraph } from "../core/graph.js"; @@ -73,7 +73,7 @@ export async function runCommand(file: string, options: RunOptions, deps: RunCom ctx: { runId, store, - opts: { home, username: userInfo().username, repoRoot: cwd }, + opts: { home, username: userInfo().username, repoRoot: cwd, hostname: hostname() }, } satisfies BatchCtx, }); diff --git a/src/commands/sync.test.ts b/src/commands/sync.test.ts index 813006f..bc83ae3 100644 --- a/src/commands/sync.test.ts +++ b/src/commands/sync.test.ts @@ -1,6 +1,6 @@ import { describe, expect, it, beforeEach, afterEach, vi } from "vitest"; import { mkdtempSync, readFileSync, rmSync } from "node:fs"; -import { tmpdir } from "node:os"; +import { hostname, tmpdir } from "node:os"; import { join } from "node:path"; import { EventLog } from "../core/events.js"; import { CheckpointStore } from "../core/store.js"; @@ -175,4 +175,42 @@ describe("lg sync --all", () => { expect(outs).toContain("synced 3 runs"); expect(outs.filter((o) => o.startsWith("synced run-"))).toHaveLength(3); }); -}); \ No newline at end of file +}); +describe("lg sync supplies the machine identity", () => { + it("16. the command threads os.hostname() through, so the hostname is rewritten out of a published error", async () => { + // BUG 1: `rewritePaths` had always accepted a `hostname`, but nothing on + // the sync path could supply one - `ProjectionOpts` had no such field - so + // the machine hostname published untouched out of the ONE channel that is + // otherwise a real allowlist. This test uses the real `hostname()` and + // passes NO hostname option, so it fails again the moment the production + // call site stops supplying it. + const host = hostname(); + const runId = "run-host"; + const state = makeState(runId); + state.nodes.a = { + nodeId: "a", + status: "failed", + startedAt: "2026-08-25T00:00:00.000Z", + endedAt: "2026-08-25T00:00:01.000Z", + attempts: 1, + output: null, + error: `ssh ${host}: connection refused`, + costUsd: 0, + }; + new CheckpointStore(runsDir(cwd)).save(state); + new EventLog(runsDir(cwd)).append(runId, { kind: "run_started", data: {} }); + + let pushed: EventBatch | null = null; + const fetch: Fetch = async (_url, init) => { + pushed = JSON.parse(init.body ?? "null") as EventBatch; + return { status: 200, json: async () => ({ highWaterSeq: 0 }) }; + }; + + const code = await syncCommand({ env: ENV, home, cwd, username: "alice", runId, f: fetch }); + + expect(code).toBe(0); + const error = pushed!.state.nodes.a!.error; + expect(error).not.toContain(host); + expect(error).toContain("${HOSTNAME}"); + }); +}); diff --git a/src/commands/sync.ts b/src/commands/sync.ts index 9086a69..9308583 100644 --- a/src/commands/sync.ts +++ b/src/commands/sync.ts @@ -1,5 +1,5 @@ import { mkdirSync, writeFileSync } from "node:fs"; -import { homedir, userInfo } from "node:os"; +import { homedir, hostname, userInfo } from "node:os"; import { join } from "node:path"; import type { CheckpointStore } from "../core/store.js"; import { syncRun, type ProjectionOpts } from "../team/sync.js"; @@ -23,6 +23,7 @@ export interface SyncOptions { cwd?: string; home?: string; username?: string; + hostname?: string; env?: NodeJS.ProcessEnv; f?: Fetch; } @@ -79,6 +80,7 @@ export async function syncCommand(opts: SyncOptions = {}): Promise { home, username: opts.username ?? userInfo().username, repoRoot: cwd, + hostname: opts.hostname ?? hostname(), }, }; diff --git a/src/hub/wire.test.ts b/src/hub/wire.test.ts index 42031e5..b4759f6 100644 --- a/src/hub/wire.test.ts +++ b/src/hub/wire.test.ts @@ -544,6 +544,83 @@ describe("eventBatchSchema", () => { }); }); + /** + * A node error is routinely a stack trace or a multi-line stderr dump. Refusing the + * whitespace controls inside one 400s the WHOLE batch, and because the sync cursor only + * advances on a 2xx the same batch is retried forever - that run can never sync again. + * Tab, newline and carriage return must therefore pass; everything the check exists to + * stop must still be stopped. + */ + describe("CLASS 9: control characters in a node error", () => { + const stackTrace = + "Error: boom\n at run (/app/src/core/engine.ts:12:5)\n at main (/app/src/cli.ts:3:1)"; + + it("accepts a node error carrying a multi-line stack trace", () => { + const result = eventBatchSchema.safeParse( + withNode(node({ status: "failed", error: stackTrace })), + ); + expect(result.success).toBe(true); + }); + + it("preserves the newlines in an accepted multi-line error byte-for-byte", () => { + const result = eventBatchSchema.safeParse( + withNode(node({ status: "failed", error: stackTrace })), + ); + expect(result.success).toBe(true); + if (!result.success) return; + expect(result.data.state.nodes["n1"]?.error).toBe(stackTrace); + }); + + it.each([ + ["tab", "command failed:\texit status 1"], + ["carriage return", "downloading...\rdownload failed"], + ["CRLF", "line one\r\nline two"], + ])("accepts a node error containing a %s", (_label, error) => { + expect( + eventBatchSchema.safeParse(withNode(node({ status: "failed", error }))).success, + ).toBe(true); + }); + + it.each([ + ["NUL", "boom\u0000truncated"], + ["ESC / ANSI colour sequence", "\u001b[31mboom\u001b[0m"], + ["ESC / OSC terminal title sequence", "boom\u001b]0;pwned\u0007"], + ["BEL", "boom\u0007"], + ["SOH", "boom\u0001"], + ["vertical tab", "boom\u000bmore"], + ["form feed", "boom\u000cmore"], + ["DEL", "boom\u007f"], + ])("still rejects a node error containing %s", (_label, error) => { + expect( + eventBatchSchema.safeParse(withNode(node({ status: "failed", error }))).success, + ).toBe(false); + }); + + it("still rejects an error made only of newlines, which is empty after trimming", () => { + expect( + eventBatchSchema.safeParse(withNode(node({ status: "failed", error: "\n\n" }))).success, + ).toBe(false); + }); + + /** + * The widening is scoped to `error` alone. Identity strings keep the original refusal - + * a newline or tab in a run id, graph name or cwd is still a delimiter-injection shape + * and must stay a 400. + */ + it.each(["runId", "graphName", "cwd"] as const)( + "still rejects a newline in state.%s", + (field) => { + const state = baseState(); + const batch = baseBatch({ state: { ...state, [field]: `${state[field]}\nx` } }); + expect(eventBatchSchema.safeParse(batch).success).toBe(false); + }, + ); + + it("still rejects a tab in the top-level streamId", () => { + expect(eventBatchSchema.safeParse(baseBatch({ streamId: "s\t1" })).success).toBe(false); + }); + }); + describe("must remain accepted", () => { it("accepts an event seq gap, e.g. [0,5]", () => { const events = [0, 5].map((s) => rawLine.replace('"seq":0', `"seq":${s}`)); diff --git a/src/hub/wire.ts b/src/hub/wire.ts index 5ed0f3b..b124642 100644 --- a/src/hub/wire.ts +++ b/src/hub/wire.ts @@ -128,6 +128,27 @@ function rejectControl(s: string): boolean { return /[\u0000-\u001f\u007f]/.test(s); } +/** + * `rejectControl` minus the three whitespace controls that legitimately occur INSIDE error + * text: tab (U+0009), line feed (U+000A) and carriage return (U+000D). + * + * A node error is routinely a stack trace or a multi-line stderr dump. Refusing a newline + * there 400s the WHOLE batch, and the sync cursor only advances on a 2xx + * (`src/team/sync.ts` - `syncRun` returns `{ok:false}` without writing the cursor), so the + * same batch is retried forever and that run can never sync again. Ordinary error output + * must not be a denial-of-sync. + * + * Everything the original check exists to stop is still stopped: NUL, BEL, ESC - so ANSI + * colour and OSC terminal-title sequences cannot ride in on an error string - vertical tab, + * form feed, every other C0 code point, and DEL. Identity strings (run id, stream id, graph + * name, cwd) keep using `rejectControl` unchanged: a tab or newline there is still a + * delimiter-injection shape with no legitimate producer. Do not point identityString at + * this function. + */ +function rejectControlInText(s: string): boolean { + return /[\u0000-\u0008\u000b\u000c\u000e-\u001f\u007f]/.test(s); +} + /** A node id the engine's graph parser (src/core/graph.ts) would accept. */ const nodeIdString = z .string() @@ -152,11 +173,14 @@ const projectedNodeSchema = z startedAt: isoInstant, endedAt: isoInstant.nullable(), attempts: z.number().int().min(1).max(MAX_ATTEMPTS), + // `rejectControlInText`, not `rejectControl`: see its comment. An error of only + // whitespace still fails, because `trim()` strips the tabs and newlines now allowed. error: z .string() .nullable() - .refine((s) => s === null || (s.trim() !== "" && !rejectControl(s)), { - message: "error must be null or a non-empty string", + .refine((s) => s === null || (s.trim() !== "" && !rejectControlInText(s)), { + message: + "error must be null or a non-empty string whose only control characters are tab, newline or carriage return", }), costUsd: z.number().nonnegative(), }) diff --git a/src/team/batch.test.ts b/src/team/batch.test.ts index c56ee10..e341a06 100644 --- a/src/team/batch.test.ts +++ b/src/team/batch.test.ts @@ -98,7 +98,16 @@ function seedRun(dir: string, runId: string): { store: CheckpointStore; log: Eve } function makeCtx(store: CheckpointStore, runId = "fixed-run"): BatchCtx { - return { runId, store, opts: { home: "/home/alice", username: "alice", repoRoot: "/repo" } }; + return { + runId, + store, + opts: { + home: "/home/alice", + username: "alice", + repoRoot: "/repo", + hostname: "alice-laptop.local", + }, + }; } function okFetchCalls(): { f: Fetch; calls: { count: number; batches: EventBatch[] } } { diff --git a/src/team/project.test.ts b/src/team/project.test.ts index b2a7744..bd0eea1 100644 --- a/src/team/project.test.ts +++ b/src/team/project.test.ts @@ -11,9 +11,14 @@ const shaped = (prefix: string, body: string): string => prefix + body; import { describe, expect, it } from "vitest"; import type { NodeResult, RunState } from "../core/types.js"; import { eventBatchSchema, type EventBatch } from "../hub/wire.js"; -import { projectState } from "./project.js"; +import { projectState, safeText } from "./project.js"; -const OPTS = { home: "/home/alice", username: "alice", repoRoot: "/home/alice/work/repo" }; +const OPTS = { + home: "/home/alice", + username: "alice", + repoRoot: "/home/alice/work/repo", + hostname: "alice-laptop.local", +}; function baseState(): RunState { return { @@ -290,6 +295,131 @@ describe("projectState", () => { expect(projected.nodes.a!.error).toBe("missing ${HOME}/.config/loomgraph/hub.json"); }); + it("an ANSI-coloured error is stripped clean and passes the hub's wire schema", () => { + // Many CLIs colour stderr by default. The hub still rejects ESC (U+001B) in + // a node error - deliberately, so a colour code or an OSC terminal-title + // sequence cannot ride in - so an uncleaned error would 400 the whole batch + // and wedge that run's sync forever, exactly as a newline used to. + const state = baseState(); + state.nodes.a = node("a", "failed", { + error: "\u001b[31mbuild failed\u001b[0m in \u001b]0;title\u0007module", + }); + + const projected = projectState(state, OPTS); + + expect(projected.nodes.a!.error).toBe("build failed in module"); + + const batch: EventBatch = { + runId: projected.runId, + streamId: state.streamId, + graphName: projected.graphName, + state: projected, + events: [], + }; + expect(eventBatchSchema.safeParse(batch).success).toBe(true); + }); + + it("a multi-line stack trace survives intact - tab, newline and carriage return are kept", () => { + const trace = "Error: boom\n\tat run (/home/alice/work/repo/src/a.ts:1:1)\r\n\tat main"; + const state = baseState(); + state.nodes.a = node("a", "failed", { error: trace }); + + const projected = projectState(state, OPTS); + + expect(projected.nodes.a!.error).toBe( + "Error: boom\n\tat run (${REPO_ROOT}/src/a.ts:1:1)\r\n\tat main", + ); + + const batch: EventBatch = { + runId: projected.runId, + streamId: state.streamId, + graphName: projected.graphName, + state: projected, + events: [], + }; + expect(eventBatchSchema.safeParse(batch).success).toBe(true); + }); + + it("a NUL, a bell, a DEL and a vertical tab are removed while the text around them survives", () => { + const state = baseState(); + state.nodes.a = node("a", "failed", { error: "a\u0000b\u0007c\u007fd\u000b" }); + + const projected = projectState(state, OPTS); + + expect(projected.nodes.a!.error).toBe("abcd"); + }); + + it("control characters are stripped BEFORE masking, so an escape spliced into a secret cannot evade the masker", () => { + // A colouriser can emit an escape in the middle of a token. Stripping after + // masking would hand the wire a reassembled, UNMASKED secret; stripping + // first means the masker sees the contiguous token it has a rule for. + const secret = shaped("sk-ant-", "api03-0000VERYFAKE0000VERYFAKE0000"); + const spliced = `${secret.slice(0, 12)}\u001b[0m${secret.slice(12)}`; + const state = baseState(); + state.nodes.a = node("a", "failed", { error: `key ${spliced}` }); + + const projected = projectState(state, OPTS); + + expect(projected.nodes.a!.error).toBe("key sk-a..."); + expect(projected.nodes.a!.error).not.toContain("api03"); + }); + + it("the full chain runs in order: rewrite, then mask, then cap", () => { + // ORDER REGRESSION, pinned as one assertion because control stripping now + // sits in this chain and makes it easy to disturb. Capping before masking + // would truncate the key below its rule's `{16,}` tail and publish real + // characters; masking before rewriting would leave the home path intact. + const secret = shaped("sk-ant-", "api03-0000VERYFAKE0000VERYFAKE0000"); + const filler = "y".repeat(190); + const state = baseState(); + state.nodes.a = node("a", "failed", { + error: `\u001b[31m/home/alice/work/repo/a.ts ${secret} ${filler}\u001b[0m`, + }); + + const projected = projectState(state, OPTS); + const error = projected.nodes.a!.error!; + + expect(error).not.toContain("\u001b"); + // rewrite ran: the repo root became a placeholder + expect(error).toContain("${REPO_ROOT}/a.ts"); + // mask ran, and ran before the cap: the key is 7 characters, not a fragment + expect(error).toContain("sk-a..."); + expect(error).not.toContain("api03"); + // cap ran last, on the already-rewritten, already-masked, already-stripped text + expect(error).toHaveLength(201); + expect(error.endsWith("…")).toBe(true); + }); + + it("masking runs BEFORE capping, so a secret straddling the 200-char cap cannot be published as a fragment", () => { + // ORDER REGRESSION. Capping first would truncate this key below its rule's + // `{16,}` tail, the mask would then fail to match, and the surviving + // prefix would publish real key material. Do not reorder rewrite/mask/cap. + const secret = shaped("sk-ant-", "api03-0000VERYFAKE0000VERYFAKE0000"); + const state = baseState(); + state.nodes.a = node("a", "failed", { error: `${"x".repeat(190)} ${secret}` }); + + const projected = projectState(state, OPTS); + + expect(projected.nodes.a!.error).toContain("sk-a..."); + expect(projected.nodes.a!.error).not.toContain("api03"); + expect(projected.nodes.a!.error).not.toContain(secret.slice(0, 20)); + }); + + it("d. the machine hostname is rewritten out of a projected node error", () => { + // BUG 1: `rewritePaths` has always had a hostname rule, but `ProjectionOpts` + // had no `hostname` field, so no caller could supply one and the rule never + // fired - a leak in the one channel that IS an allowlist. + const state = baseState(); + state.nodes.a = node("a", "failed", { + error: "ssh alice-laptop.local: connection refused (short form alice-laptop too)", + }); + + const projected = projectState(state, OPTS); + + expect(projected.nodes.a!.error).not.toContain("alice-laptop"); + expect(projected.nodes.a!.error).toContain("${HOSTNAME}"); + }); + it("a repo-root path in a node error is rewritten to the REPO_ROOT placeholder", () => { const state = baseState(); state.nodes.a = node("a", "failed", { @@ -367,4 +497,34 @@ describe("projectState", () => { const parsed = eventBatchSchema.safeParse(batch); expect(parsed.success).toBe(true); }); -}); \ No newline at end of file +}); +describe("safeText strips control characters the wire schema refuses", () => { + // The hub accepts \t \n \r inside a node error (multi-line stack traces are + // legitimate) but still refuses the rest of C0, DEL and ESC, so ANSI colour + // and OSC terminal-title sequences cannot ride in. Many CLI tools colour + // their stderr by default, so an unsanitised ESC would 400 the batch and + // wedge that run's sync permanently - the same failure mode as the newline + // bug, reached by a different route. Stripping belongs here, producer-side. + it("removes ANSI colour codes so the result passes the wire schema", () => { + const out = safeText("\u001b[31mbuild failed\u001b[0m", OPTS); + expect(out).not.toMatch(/\u001b/); + expect(out).toContain("build failed"); + }); + + it("removes OSC, BEL, NUL, VT, FF and DEL", () => { + const out = safeText("a\u0000b\u0007c\u000bd\u000ce\u007ff", OPTS); + expect(out).toBe("abcdef"); + }); + + it("PRESERVES tab, newline and carriage return", () => { + const out = safeText("line1\nline2\tcol\r\nline3", OPTS); + expect(out).toBe("line1\nline2\tcol\r\nline3"); + }); + + it("strips before capping, so the cap still bounds the final text", () => { + const noisy = `${"\u001b[31m".repeat(200)}${"x".repeat(300)}`; + const out = safeText(noisy, OPTS); + expect(out).not.toMatch(/\u001b/); + expect((out ?? "").length).toBeLessThanOrEqual(201); + }); +}); diff --git a/src/team/project.ts b/src/team/project.ts index 2f10c4b..3929f47 100644 --- a/src/team/project.ts +++ b/src/team/project.ts @@ -2,6 +2,23 @@ import type { RunState } from "../core/types.js"; import type { ProjectedState, ProjectedNode } from "../hub/wire.js"; import { SCAN_RULES, rewritePaths } from "../handoff/scan.js"; +/** + * The machine facts every published string is rewritten against. Mirrors + * `rewritePaths`' own opts (`src/handoff/scan.ts`) rather than a narrower + * `(state, home, repoRoot)` form, because `rewritePaths` skips a protection + * whenever the field it needs is empty - a narrower signature invites a caller + * to pass `""` and silently disable one. `hostname` is REQUIRED for exactly + * that reason: it was optional on `rewritePaths`, no caller on the sync path + * ever supplied it, and the machine hostname published unrewritten for the + * whole of phase 1. Do not make it optional again. + */ +export interface ProjectionIdentity { + home: string; + username: string; + repoRoot: string; + hostname: string; +} + /** * Ceiling on a published node error. 200 is the number `claude.ts:33` already * truncates stdout to, so it matches the largest thing the adapters @@ -33,14 +50,65 @@ function maskSecrets(text: string): string { return out; } -/** Sanitise a node error for publication: paths rewritten, secrets masked, length capped. */ -function safeError( +/** + * Remove control characters the hub's wire schema refuses, keeping the three + * that legitimately appear in error text. + * + * `projectedNodeSchema.error` permits only TAB, LF and CR; the rest of C0, DEL + * and ESC stay refused so ANSI colour and OSC terminal-title sequences cannot + * ride in. Many CLI tools colour their stderr by default, so an unsanitised ESC + * would 400 the batch and wedge that run's sync PERMANENTLY - the same failure + * as the newline bug, reached by a different route. The producer strips, and the + * hub keeps refusing: validation must never refuse a shape the engine can + * legitimately produce, and the engine must not produce one it refuses. + * + * ORDER MATTERS - this runs FIRST, before rewrite and mask. An ESC spliced into + * a secret or a path defeats their patterns, and stripping afterwards would + * reassemble the original in clear. Full order: strip -> rewrite -> mask -> cap. + * Do not reorder: a masked token is already `first4 + "..."`, so capping cannot + * reveal a fragment, but any other arrangement is exploitable. + */ +function stripControl(s: string): string { + return ( + s + // OSC: ESC ] ... terminated by BEL or ST. Must run before CSI so a title + // sequence is consumed whole rather than leaving its payload behind. + .replace(/\u001b\][^\u0007\u001b]*(?:\u0007|\u001b\\)?/g, "") + // CSI: ESC [ params intermediates final. Removing only the ESC byte would + // leave "[31m" in the text - and, worse, leave a spliced secret still + // unmatchable by the masker. + .replace(/\u001b\[[0-9;?]*[ -/]*[@-~]?/g, "") + // Any other two-byte escape. + .replace(/\u001b[@-_]?/g, "") + // Remaining C0 and DEL, keeping TAB, LF and CR. + .replace(/[\u0000-\u0008\u000b\u000c\u000e-\u001f\u007f]/g, "") + ); +} + +/** + * Sanitise a published string: paths rewritten, secrets masked, length capped. + * + * ORDER IS LOAD-BEARING - rewrite, then mask, then cap. Capping first would let + * a secret straddling the 200th character be truncated below its rule's + * `{16,}` tail, so the mask would no longer match and the surviving prefix + * would publish real key material. Do not reorder these three lines. + * + * Exported because `buildBatch` sanitises event `data` with the SAME function. + * A second implementation over there would drift from this one; there must be + * exactly one definition of "safe to publish" on the sync path. + */ +export function safeText( error: string | null, - opts: { home: string; username: string; repoRoot: string }, + opts: ProjectionIdentity, ): string | null { if (error === null) return null; - let out = rewritePaths(error, opts); + // Strip FIRST. An ESC spliced into the middle of a secret or an absolute path + // breaks the masker's and the rewriter's patterns; stripping afterwards would + // then reassemble the original in clear. Removing the noise before either one + // runs is what makes them see the real shape. + let out = stripControl(error); + out = rewritePaths(out, opts); out = maskSecrets(out); if (out.length > MAX_ERROR_LENGTH) { @@ -57,17 +125,9 @@ function safeError( * followed by deletes - so a future content-carrying field added to `RunState` cannot * silently start publishing itself. * - * The signature mirrors `rewritePaths`' own opts (`{ home, username, repoRoot }`) rather than - * the narrower `(state, home, repoRoot)` form, because `rewritePaths` rewrites the - * `/home/`, `/Users/` and `C:\Users\` shapes and skips all of them when - * `username` is empty. A narrower signature invites a caller to pass `""` and silently - * disable one of its three protections. Mirroring the opts shape keeps one vocabulary across - * both functions and loses nothing. Do not "restore" the narrower form. + * `opts` is `ProjectionIdentity` - see its doc comment for why every field is required. */ -export function projectState( - state: RunState, - opts: { home: string; username: string; repoRoot: string }, -): ProjectedState { +export function projectState(state: RunState, opts: ProjectionIdentity): ProjectedState { const nodes: Record = {}; for (const [id, node] of Object.entries(state.nodes)) { nodes[id] = { @@ -76,7 +136,7 @@ export function projectState( startedAt: node.startedAt, endedAt: node.endedAt, attempts: node.attempts, - error: safeError(node.error, opts), + error: safeText(node.error, opts), costUsd: node.costUsd, }; } diff --git a/src/team/sync-redaction.test.ts b/src/team/sync-redaction.test.ts new file mode 100644 index 0000000..8d95ab8 --- /dev/null +++ b/src/team/sync-redaction.test.ts @@ -0,0 +1,315 @@ +import { describe, expect, it, beforeEach, afterEach } from "vitest"; +import { mkdtempSync, readFileSync, rmSync } from "node:fs"; +import { join } from "node:path"; +import { tmpdir } from "node:os"; +import { EventLog } from "../core/events.js"; +import { CheckpointStore } from "../core/store.js"; +import { execute, newRunState } from "../core/engine.js"; +import { parseGraph } from "../core/graph.js"; +import type { Adapter, AdapterInput, AdapterOutput } from "../adapters/types.js"; +import type { RunState } from "../core/types.js"; +import { HubStore } from "../hub/storage.js"; +import { handle, type HandlerDeps, type WireRequest } from "../hub/handlers.js"; +import type { EventBatch } from "../hub/wire.js"; +import { buildBatch, sanitizeEventLine, syncRun, type ProjectionOpts } from "./sync.js"; +import type { Fetch, HubConfig } from "./transport.js"; + +/** + * THE EVENT STREAM IS A PUBLISHED CHANNEL. `src/team/project.test.ts` proves the + * *projection* is an allowlist; nothing proved anything about the raw event + * lines that ride alongside it in the same push, which is exactly how they came + * to carry unmasked errors, un-rewritten paths and interpolated var values. + * These tests pin the push, not the projection: the local log stays raw and the + * wire does not. + */ + +const FROZEN = "2026-08-25T00:00:00.000Z"; + +/** An anthropic-key shape, so `SCAN_RULES` has a rule that must fire on it. */ +const SECRET = "sk-ant-api03-LEAKLEAKLEAKLEAK1234"; +const HOME = "/home/alice"; +const HOME_PATH = `${HOME}/.config/loomgraph/hub.json`; +const HOSTNAME = "alice-laptop.local"; + +let tmp: string; +let cwd: string; +let eventRoot: string; +let store: CheckpointStore; +let log: EventLog; +let opts: ProjectionOpts; + +beforeEach(() => { + tmp = mkdtempSync(join(tmpdir(), "loomgraph-redact-")); + cwd = join(tmp, "repo"); + eventRoot = join(cwd, ".loomgraph", "runs"); + store = new CheckpointStore(eventRoot); + log = new EventLog(eventRoot); + opts = { home: HOME, username: "alice", repoRoot: cwd, hostname: HOSTNAME }; +}); + +afterEach(() => { + rmSync(tmp, { recursive: true, force: true }); +}); + +function stub(name: string, out: AdapterOutput): Adapter { + return { name, run: async (_input: AdapterInput) => out }; +} + +/** The real ingest handler, so a sanitized line still has to pass the wire schema. */ +function hub(): { f: Fetch; cfg: HubConfig; pushed: EventBatch[]; statuses: number[] } { + const hubStore = HubStore.open(":memory:", { now: () => FROZEN }); + const token = hubStore.addMember("alice", ["ingest"]).token; + const deps: HandlerDeps = { store: hubStore, now: () => FROZEN, version: "test-v" }; + const pushed: EventBatch[] = []; + const statuses: number[] = []; + const f: Fetch = async (url, init) => { + const body = JSON.parse(init.body ?? "null") as EventBatch; + const req: WireRequest = { + method: init.method, + path: new URL(url).pathname, + query: {}, + headers: { authorization: init.headers.authorization }, + body, + }; + const res = handle(req, deps); + pushed.push(body); + statuses.push(res.status); + return { status: res.status, json: async () => res.body }; + }; + return { f, cfg: { url: "http://hub.test", token }, pushed, statuses }; +} + +async function push(runId: string, state: RunState): Promise<{ wire: string; statuses: number[] }> { + const { f, cfg, pushed, statuses } = hub(); + const result = await syncRun({ f, cfg, cwd, eventRoot, runId, state, opts, timeoutMs: 5000 }); + expect(result.ok).toBe(true); + return { wire: JSON.stringify(pushed), statuses }; +} + +function localLog(runId: string): string { + return readFileSync(join(eventRoot, runId, "events.jsonl"), "utf8"); +} + +const FAILING_GRAPH = ` +name: leaky +budget: { maxUsd: 10, maxWallClockSec: 600, maxNodeRuns: 20 } +nodes: + boom: { type: command, run: "true" } +edges: + - { from: boom, to: END } +`; + +const HUMAN_GRAPH = ` +name: ask +budget: { maxUsd: 10, maxWallClockSec: 600, maxNodeRuns: 20 } +nodes: + ask: { type: human, question: "ship with {{vars.token}}?" } +edges: + - { from: ask, to: END } +`; + +describe("what a sync publishes", () => { + it("a. a node error carrying a secret and an absolute home path reaches the hub with neither", async () => { + const runId = "run-leak"; + const graph = parseGraph(FAILING_GRAPH); + const state = newRunState(graph, { runId, cwd }); + const failed = await execute(graph, state, { + store, + log, + registry: { + command: stub("command", { + ok: false, + text: "", + costUsd: 0, + raw: null, + error: `command exited with code 1: could not read ${HOME_PATH}, key ${SECRET}`, + }), + }, + sleep: async () => {}, + }); + expect(failed.status).toBe("failed"); + + const { wire, statuses } = await push(runId, failed); + + expect(statuses).toEqual([200]); + expect(wire).not.toContain(SECRET); + expect(wire).not.toContain("sk-ant-api03-LEAK"); + expect(wire).not.toContain(HOME_PATH); + expect(wire).not.toContain("/home/alice"); + // The event is still published - sanitized, not dropped. + expect(wire).toContain("node_finished"); + expect(wire).toContain("could not read ${HOME}/.config/loomgraph/hub.json"); + expect(wire).toContain("sk-a..."); + }); + + it("b. a human question that interpolated a secret var does not reach the hub", async () => { + const runId = "run-ask"; + const graph = parseGraph(HUMAN_GRAPH); + const state = newRunState(graph, { runId, cwd, vars: { token: SECRET } }); + const paused = await execute(graph, state, { store, log, registry: {}, sleep: async () => {} }); + expect(paused.status).toBe("paused"); + + const answered = await execute(graph, paused, { + store, + log, + registry: {}, + sleep: async () => {}, + humanAnswers: { ask: `approved, reused ${SECRET}` }, + }); + + const { wire, statuses } = await push(runId, answered); + + expect(statuses).toEqual([200]); + expect(wire).not.toContain(SECRET); + expect(wire).toContain("human_requested"); + expect(wire).toContain("human_resolved"); + // Both the interpolated question and the typed answer survive, masked. + expect(wire).toContain("ship with sk-a...?"); + expect(wire).toContain("approved, reused sk-a..."); + }); + + it("c. the local events.jsonl keeps the raw values - sanitising is a push-time transform only", async () => { + const runId = "run-local"; + const graph = parseGraph(FAILING_GRAPH); + const state = newRunState(graph, { runId, cwd }); + const failed = await execute(graph, state, { + store, + log, + registry: { + command: stub("command", { + ok: false, + text: "", + costUsd: 0, + raw: null, + error: `could not read ${HOME_PATH}, key ${SECRET}`, + }), + }, + sleep: async () => {}, + }); + + const before = localLog(runId); + expect(before).toContain(SECRET); + expect(before).toContain(HOME_PATH); + + await push(runId, failed); + + // Byte-identical after the push: sync writes only the cursor. + expect(localLog(runId)).toBe(before); + expect(localLog(runId)).toContain(SECRET); + expect(localLog(runId)).toContain(HOME_PATH); + }); +}); + +describe("sanitizeEventLine", () => { + const line = (kind: string, data: Record, nodeId?: string): string => + JSON.stringify({ + ts: "2026-08-25T00:00:00.000Z", + runId: "r", + seq: 0, + kind, + ...(nodeId === undefined ? {} : { nodeId }), + data, + }); + + it("rewrites the raw cwd run_started publishes, which the projection rewrote and this did not", () => { + const out = sanitizeEventLine( + line("run_started", { graph: "g", resumed: false, cwd: `${HOME}/work/repo`, streamId: "s" }), + { ...opts, repoRoot: "/nowhere" }, + ); + const data = (JSON.parse(out!) as { data: Record }).data; + expect(data.cwd).toBe("${HOME}/work/repo"); + expect(data).toEqual({ graph: "g", resumed: false, cwd: "${HOME}/work/repo", streamId: "s" }); + }); + + it("drops a data field nobody classified, rather than publishing it", () => { + // The whole point of the allowlist: a field added to an event's `data` + // upstream must NOT start publishing itself just because it exists. + const out = sanitizeEventLine( + line("node_finished", { status: "failed", attempts: 1, costUsd: 0, error: null, stdout: SECRET }), + opts, + ); + expect(out).not.toContain(SECRET); + expect(JSON.parse(out!)).toEqual({ + ts: "2026-08-25T00:00:00.000Z", + runId: "r", + seq: 0, + kind: "node_finished", + data: { status: "failed", attempts: 1, costUsd: 0, error: null }, + }); + }); + + it("drops an unclassifiable line entirely: bad JSON, a non-object, or an unknown kind", () => { + expect(sanitizeEventLine("not json", opts)).toBeNull(); + expect(sanitizeEventLine("[1,2,3]", opts)).toBeNull(); + expect(sanitizeEventLine("null", opts)).toBeNull(); + expect(sanitizeEventLine(line("teleported", { secret: SECRET }), opts)).toBeNull(); + }); + + it("drops a text field whose type changed under us instead of publishing it unsanitised", () => { + const out = sanitizeEventLine(line("human_requested", { question: { raw: SECRET } }, "ask"), opts); + expect(out).not.toContain(SECRET); + expect((JSON.parse(out!) as { data: Record }).data).toEqual({}); + }); + + it("leaves a line with nothing to sanitise byte-identical to its local log line", () => { + const source = line("node_started", { attempt: 1, type: "command" }, "boom"); + expect(sanitizeEventLine(source, opts)).toBe(source); + }); + + it("buildBatch is the choke point: the lines it emits are the sanitised ones", () => { + const graph = parseGraph(FAILING_GRAPH); + const state = newRunState(graph, { runId: "run-b", cwd }); + const batch = buildBatch(state, opts, [ + line("node_finished", { status: "failed", attempts: 1, costUsd: 0, error: SECRET }, "boom"), + "this line cannot be classified", + ]); + expect(batch.events).toHaveLength(1); + expect(batch.events[0]).not.toContain(SECRET); + expect(batch.events[0]).toContain("sk-a..."); + }); +}); + +describe("control characters on the event path", () => { + it("an ANSI-coloured node error goes out clean on the EVENT line too, and the hub accepts the batch", async () => { + // The event path reuses `safeText`, so it inherits control stripping. This + // pins that: an ESC on an event line is not schema-checked by the hub (the + // event `data` record is deliberately permissive), so nothing else would + // catch a regression here - the colour codes would just silently publish. + const runId = "run-ansi"; + const graph = parseGraph(FAILING_GRAPH); + const state = newRunState(graph, { runId, cwd }); + const failed = await execute(graph, state, { + store, + log, + registry: { + command: stub("command", { + ok: false, + text: "", + costUsd: 0, + raw: null, + error: `\u001b[31mbuild failed\u001b[0m reading ${HOME_PATH}\n\tkey ${SECRET}`, + }), + }, + sleep: async () => {}, + }); + + const { wire, statuses } = await push(runId, failed); + + expect(statuses).toEqual([200]); + // Both forms: a raw ESC, and the `\\u001b` text JSON.stringify would escape + // it to. Asserting only the raw char would pass vacuously. + expect(wire).not.toContain("\u001b"); + expect(wire).not.toContain("\\u001b"); + expect(wire).not.toContain("[31m"); + expect(wire).not.toContain(SECRET); + expect(wire).not.toContain(HOME_PATH); + // The newline is legitimate and survives, JSON-escaped, on the event line. + expect(wire).toContain("build failed reading ${HOME}/.config/loomgraph/hub.json"); + expect(wire).toContain("sk-a..."); + + // ...and the escape is still in the local log, where JSON.stringify wrote + // it as the six characters `\\u001b`. + expect(localLog(runId)).toContain("\\u001b[31m"); + expect(localLog(runId)).toContain(SECRET); + }); +}); diff --git a/src/team/sync.test.ts b/src/team/sync.test.ts index 33a5018..2a6c48a 100644 --- a/src/team/sync.test.ts +++ b/src/team/sync.test.ts @@ -27,7 +27,12 @@ import type { Fetch, HubConfig } from "./transport.js"; const FROZEN = "2026-08-25T00:00:00.000Z"; -const OPTS: ProjectionOpts = { home: "/home/alice", username: "alice", repoRoot: "/work/repo" }; +const OPTS: ProjectionOpts = { + home: "/home/alice", + username: "alice", + repoRoot: "/work/repo", + hostname: "alice-laptop.local", +}; const CFG: HubConfig = { url: "http://hub.test", token: "lgt_00000000.FAKEfake0000FAKEfake0000" }; diff --git a/src/team/sync.ts b/src/team/sync.ts index 369560c..f3b9955 100644 --- a/src/team/sync.ts +++ b/src/team/sync.ts @@ -1,20 +1,21 @@ import { mkdirSync, readFileSync, renameSync, writeFileSync } from "node:fs"; import { join } from "node:path"; -import { EventLog } from "../core/events.js"; +import { EventLog, type LgEventKind } from "../core/events.js"; import type { RunState } from "../core/types.js"; import type { EventBatch } from "../hub/wire.js"; -import { projectState } from "./project.js"; +import { projectState, safeText, type ProjectionIdentity } from "./project.js"; import { postEvents, type Fetch, type HubConfig } from "./transport.js"; /** At most this many event lines per request. */ const BATCH_LIMIT = 500; -/** The projection opts `projectState` requires; mirrored so callers name it once. */ -export interface ProjectionOpts { - home: string; - username: string; - repoRoot: string; -} +/** + * The identity `projectState` and `sanitizeEventLine` both need. An alias, not + * a second declaration: two copies would let one gain a field the other lacks, + * which is precisely how `hostname` came to be missing here while + * `rewritePaths` had accepted it all along. + */ +export type ProjectionOpts = ProjectionIdentity; /** What `readCursor` returns: the highest event seq the hub has acked. */ export interface Cursor { @@ -82,20 +83,134 @@ export function pendingLines(lines: string[], ackedSeq: number): string[] { return pending; } +/** + * WHICH `data` FIELDS EACH EVENT KIND MAY PUBLISH, AND HOW. + * + * The same hand-written allowlist discipline `projectState` uses, for the same + * reason and with the same rule: a future content-carrying field must not + * silently start publishing itself. Every field is named here; anything not + * named is DROPPED from the pushed line. Never replace this with a generic walk + * over `data`, and never add a field without deciding which column it belongs + * in. + * + * "pass" the value is an engine- or graph-derived identifier, enum, number + * or boolean. It carries no operator content, and the projection + * already publishes the same class of fact (node ids, graph name, + * costs, budgets). Copied as-is. + * "text" the value is operator- or environment-derived text: an adapter + * error (which can be a whole agent result or a raw stderr dump), an + * absolute cwd, an INTERPOLATED human question (`{{vars.x}}` and + * `{{nodes.x.output}}` already substituted), or a typed answer. Run + * through `safeText` - the same rewrite/mask/cap the projection + * applies to a node error. + * + * `budget_exceeded.reason` is engine-generated and could be "pass"; it is + * "text" because it costs nothing and a string that reaches the wire should + * have gone through the sanitiser unless there is a reason it cannot. + */ +const EVENT_DATA_ALLOWLIST: Record> = { + run_started: { graph: "pass", resumed: "pass", cwd: "text", streamId: "pass" }, + node_started: { attempt: "pass", type: "pass" }, + node_finished: { status: "pass", attempts: "pass", costUsd: "pass", error: "text" }, + edge_crossed: { from: "pass", to: "pass", when: "pass" }, + budget_checked: { spent: "pass", budget: "pass", ready: "pass" }, + budget_exceeded: { reason: "text", spent: "pass", budget: "pass" }, + human_requested: { question: "text" }, + human_resolved: { answer: "text" }, + run_finished: { status: "pass", error: "text", spent: "pass" }, +}; + +function isEventKind(value: unknown): value is LgEventKind { + return typeof value === "string" && value in EVENT_DATA_ALLOWLIST; +} + +/** + * SANITISE ONE EVENT LINE FOR THE WIRE. THE LOCAL LOG IS NEVER TOUCHED. + * + * `.loomgraph/runs//events.jsonl` stays raw and complete - that is the + * author's own debugging record and it must keep full fidelity. This transform + * runs at PUSH time, on the copy that crosses to the hub, so `lg-hub export` + * still reproduces the INGESTED lines byte for byte; those lines simply stop + * carrying secrets. + * + * The event object is rebuilt field by field rather than mutated, for the same + * reason `projectState` is: no spread, no delete, no unknown key riding along. + * + * Returns null when the line cannot be classified - unparseable, not an object, + * or an unrecognised kind. An unclassifiable line is DROPPED, never passed + * through: a kind this function does not know is a kind whose `data` nobody has + * reviewed. Dropping costs an audit line; passing through costs a leak. + */ +export function sanitizeEventLine(line: string, opts: ProjectionOpts): string | null { + let parsed: unknown; + try { + parsed = JSON.parse(line); + } catch { + return null; + } + if (parsed === null || typeof parsed !== "object" || Array.isArray(parsed)) return null; + + const event = parsed as Record; + if (!isEventKind(event.kind)) return null; + + const fields = EVENT_DATA_ALLOWLIST[event.kind]; + const raw = event.data; + const source: Record = + raw !== null && typeof raw === "object" && !Array.isArray(raw) + ? (raw as Record) + : {}; + + const data: Record = {}; + for (const [key, handling] of Object.entries(fields)) { + if (!(key in source)) continue; + const value = source[key]; + if (handling === "pass") { + data[key] = value; + continue; + } + // "text": a value that is neither a string nor null is a field whose shape + // changed under us, so it is dropped rather than published unsanitised. + if (value === null) data[key] = null; + else if (typeof value === "string") data[key] = safeText(value, opts); + } + + // Same key order the engine writes (`src/core/events.ts`), so an event with + // nothing to sanitise serialises byte-identically to its local log line. + return JSON.stringify({ + ts: event.ts, + runId: event.runId, + seq: event.seq, + kind: event.kind, + ...(typeof event.nodeId === "string" ? { nodeId: event.nodeId } : {}), + data, + }); +} + /** * Assemble one push: `runId`, `streamId` and `graphName` come from the loaded - * RunState, `state` is the projected projection, and `events` are the verbatim - * lines chosen by `pendingLines`. The projection is recomputed per batch so the - * `updatedAt`/`seq` the hub hears tracks the state that was current for that - * window. + * RunState, `state` is the projected projection, and `events` are the lines + * chosen by `pendingLines`, each sanitised for the wire. The projection is + * recomputed per batch so the `updatedAt`/`seq` the hub hears tracks the state + * that was current for that window. + * + * SANITISING HAPPENS HERE, not in `syncRun`, because `buildBatch` is the single + * choke point both push paths go through: `lg sync` and the live `LiveBatcher` + * in `./batch.ts`. Moving it up into `syncRun` would leave the live stream + * publishing raw lines. */ export function buildBatch(state: RunState, opts: ProjectionOpts, lines: string[]): EventBatch { + const events: string[] = []; + for (const line of lines) { + const safe = sanitizeEventLine(line, opts); + if (safe !== null) events.push(safe); + } + return { runId: state.runId, streamId: state.streamId, graphName: state.graphName, state: projectState(state, opts), - events: lines, + events, }; } @@ -116,8 +231,9 @@ export type SyncResult = { ok: true; ackedSeq: number } | { ok: false; error: st /** * Push a whole run to the hub in windows of at most 500 lines. Local events are - * read with `EventLog.read` - the only file sync writes is the cursor under - * `.loomgraph/sync/`; nothing under `runs/` is ever touched. + * read with `EventLog.read` and sanitised by `buildBatch` on the way out - the + * only file sync writes is the cursor under `.loomgraph/sync/`; nothing under + * `runs/` is ever touched, and the log on disk keeps its raw values. * * The cursor advances ONLY on a 2xx naming `highWaterSeq`. Any `{ok:false}` * leaves the cursor exactly as it was, so a cut mid-batch resends the same From 06f7135375435c944b8b12844b656121f68fd3b6 Mon Sep 17 00:00:00 2001 From: Dat Date: Mon, 21 Sep 2026 08:54:31 +0700 Subject: [PATCH 36/45] docs(readme): correct four stale claims about install, the hub store and the web UI - `npm i -g loomgraph` does not work: the package is not published, and the registry returns Not found. Replaced with clone + build + `npm link`, and the other two binaries (`lg-handoff`, `lg-hub`) are now named. - "the run's event lines verbatim" was the claim that caused unsafe deployment specs to be written. The section described only the state projection and omitted the raw event channel entirely. It now documents the per-kind `data` allowlist, names the fields that carry operator text, and states that sanitising happens at push time so the local log stays raw and `lg-hub export` still reproduces ingested lines byte for byte. - "No web UI - that is phase 4" is false: it shipped in fb12c2d and is ON by default. It gets its own section, including the caveat that it keeps the bearer token in `localStorage` - harmless on the loopback default, not harmless on a plaintext `http://` origin behind `--behind-tls-proxy`. Also corrected in "What this is not", where the UI is not read-only: it can add and revoke members. - "a single SQLite database" understates the on-disk footprint. WAL mode means three files, and copying `hub.db` alone while the server runs silently drops every committed write still in the WAL. Co-Authored-By: Claude Opus 5 (1M context) --- README.md | 60 +++++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 47 insertions(+), 13 deletions(-) diff --git a/README.md b/README.md index 5a7612a..6c3e8cd 100644 --- a/README.md +++ b/README.md @@ -19,11 +19,15 @@ It does not call a model itself. Your agent CLIs are the runtime. ## Install +Not published to npm. Build it from a clone and link the binaries: + ```bash -npm i -g loomgraph +git clone https://github.com/datj9/loomgraph && cd loomgraph +npm install && npm run build +npm link # or: npm pack && npm i -g ./loomgraph-0.1.0.tgz ``` -Requires Node >= 22. The binary is `lg`. +Requires Node >= 22. Three binaries land on your PATH: `lg`, `lg-handoff` and `lg-hub`. ## 60-second quickstart @@ -495,7 +499,7 @@ names because that stdout shape has not yet been captured from a real invocation ## The hub -The third binary is `lg-hub`: an HTTP API in front of a single SQLite database. It +The third binary is `lg-hub`: an HTTP API in front of one SQLite database. It stores what members push and serves reads out of that store - it never runs an agent. Agents run on the member's own machine, started by the member; the hub has no way to start one, and that absence is the design. A daemon that can only store and route is a @@ -508,9 +512,14 @@ On the hub host: ```bash lg-hub init # create the data dir and hub.db lg-hub member add alice # prints alice's token once - write it down -lg-hub serve # binds 127.0.0.1:8369 +lg-hub serve # binds 127.0.0.1:8369, web UI on the same origin ``` +That database runs in WAL mode, so the hub's state on disk is **three** files, not +one: `hub.db`, `hub.db-wal` and `hub.db-shm`. Copying `hub.db` alone while the +server is running gives you a backup missing every committed write still in the +WAL. Use `VACUUM INTO` (or stop the service first). + On a member machine: ```bash @@ -528,15 +537,28 @@ lines to stdout for grepping, and `lg-hub export --out ` to write one ### What the hub receives -A sync pushes two things: the run's event lines verbatim - the same JSONL that sits -under `.loomgraph/runs//events.jsonl` - and a projection of the run state. The -projection is where content stops. It is built field by field, never as a filtered -copy of the full state, so there is no field a `vars` value or a node `output` could -ride in on: +A sync pushes two things, and **both** are filtered on the way out: + +**The run state projection.** Built field by field, never as a filtered copy of the +full state, so there is no field a `vars` value or a node `output` could ride in on: - `vars` reach the hub as key names only. - node `output` never reaches the hub. -- node `error` is path-rewritten, secret-masked, and capped at 200 characters. +- node `error` is control-stripped, path-rewritten, secret-masked, and capped at 200 + characters. + +**The run's event lines.** These are *not* pushed verbatim. Each line is rebuilt +against a per-kind allowlist of `data` fields before it leaves the machine; a field +not on the list is dropped, and a line whose `kind` the allowlist does not know is +dropped whole. Fields carrying operator or environment text - `node_finished.error` +and `run_finished.error` (an adapter folds the agent's full result text and stderr +into these), `run_started.cwd`, the interpolated `human_requested.question`, and +`human_resolved.answer` - go through the same strip/rewrite/mask/cap as a node error. + +The filtering happens at **push** time, not at emission. Your local +`.loomgraph/runs//events.jsonl` keeps its raw values for debugging; only the +copy crossing to the hub is sanitised. `lg-hub export` still reproduces the *ingested* +lines byte for byte - those lines simply no longer carry secrets. Same error before and after: @@ -558,6 +580,19 @@ like an SSH key, and `member revoke ` is the off switch. exactly the credential shape this project's own scanner has a rule for, so a bind that would put the token on the wire without TLS is an error rather than an option. +### The web UI ships, and it is on by default + +`lg-hub serve` serves a self-contained web UI on the same origin as the API - one +embedded HTML document, no build step, no external requests. It renders runs, the +activity feed and the member roster using `textContent` only, so an untrusted +transcript cannot inject markup. Pass `--no-ui` for an API-only bind. + +It authenticates with a bearer token you paste, and **keeps that token in +`localStorage`**. On the default loopback bind that is fine. Behind +`--behind-tls-proxy` on a plaintext `http://` origin it is not: the token sits in +browser storage on an origin anyone on that network can impersonate. Terminate TLS in +front of it, or run `--no-ui`. + ### Two caveats, stated up front **Phase 1 does not mask on egress.** The projection is the only gate; whatever does @@ -583,7 +618,6 @@ Both of these were run end to end against the built binary with a live hub: ### What phase 1 does not ship - No inbox - that is phase 3. -- No web UI - that is phase 4. - No briefs on the hub, no encryption at rest, and no redaction on read - all phase 2. Nothing in phase 1 is encrypted. - No full transcripts, ever. The handoff refusal stands unchanged: a transcript is a @@ -594,8 +628,8 @@ Both of these were run end to end against the built binary with a live hub: - **Not a model, and not an SDK for one.** loomgraph makes zero API calls of its own and has no LLM SDK dependency. - **Not a replacement for your agent CLI.** It shells out to the CLI you already installed and authenticated. - **Not a workflow server.** A daemon ships in phase 1 - `lg-hub` - but it stores and - routes, and never runs an agent. No web UI (phase 4), no cloud, no plugin system in - v0.1. + routes, and never runs an agent. It ships a web UI over its own store - reads, plus member add/revoke; no + cloud, and no plugin system in v0.1. `lg report --publish` does not change that: it writes a static file and shells out to the `enclave` cli the same way a node shells out to `claude`. If `enclave` is not installed the From d47f4dca03009ea383e7c5b856dc837844e8d0a0 Mon Sep 17 00:00:00 2001 From: Dat Date: Mon, 21 Sep 2026 08:57:50 +0700 Subject: [PATCH 37/45] fix(sync): gate manual syncs on the repo opt-in, not just the live batcher `repoSyncEnabled` was consulted in exactly one place - `src/team/batch.ts`, the live batcher. `lg sync ` and `lg sync --all` never called it, so a repo that had never run `lg sync --enable` could still push every run it had. The opt-in read as a per-repo consent control while only gating one of the two push paths. That matters more than a normal flag bug because the hub cannot take it back: the `events` table carries no-update/no-delete triggers, so a run pushed out of a repo nobody meant to share is permanent and visible to every read-scoped member. Removing it means hand-dropping a trigger on the production database. `syncCommand` now refuses with exit 1 and a message naming `lg sync --enable`. The check runs BEFORE `loadHubConfig`: when neither the opt-in nor the enrollment exists, pointing at `lg enroll` would send the operator to configure a hub this repo still would not push to. Tests 17-21 cover a single run, `--all`, precedence over the missing enrollment, a `hub.json` whose `sync` is `"true"` rather than `true`, and the enable-then-sync sequence. Existing push tests now seed the opt-in, which is the behaviour change made visible. 764 tests pass, typecheck clean. Co-Authored-By: Claude Opus 5 (1M context) --- README.md | 5 +++ src/commands/sync.test.ts | 82 ++++++++++++++++++++++++++++++++++++++- src/commands/sync.ts | 31 ++++++++++++++- 3 files changed, 115 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 6c3e8cd..58f311b 100644 --- a/README.md +++ b/README.md @@ -530,6 +530,11 @@ lg sync # push one run lg sync --all # or every run under .loomgraph/runs/ ``` +`lg sync --enable` is a hard gate, not a hint: without `.loomgraph/hub.json` in +the repo, `lg sync ` and `lg sync --all` both refuse and push nothing. The +hub cannot delete an event once ingested, so opting in has to be a deliberate act +per repo rather than something a forgotten flag decides for you. + The rest of the hub-facing surface: `lg-hub member revoke ` and `lg-hub member ls` for the roster, `lg-hub export --jsonl` to print the raw stored lines to stdout for grepping, and `lg-hub export --out ` to write one diff --git a/src/commands/sync.test.ts b/src/commands/sync.test.ts index bc83ae3..3dc6c34 100644 --- a/src/commands/sync.test.ts +++ b/src/commands/sync.test.ts @@ -1,5 +1,5 @@ import { describe, expect, it, beforeEach, afterEach, vi } from "vitest"; -import { mkdtempSync, readFileSync, rmSync } from "node:fs"; +import { mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; import { hostname, tmpdir } from "node:os"; import { join } from "node:path"; import { EventLog } from "../core/events.js"; @@ -48,6 +48,12 @@ function optInPath(): string { return join(cwd, ".loomgraph", "hub.json"); } +/** Opt the temp repo in, the same way `lg sync --enable` does. */ +function seedOptIn(): void { + mkdirSync(join(cwd, ".loomgraph"), { recursive: true }); + writeFileSync(optInPath(), '{"sync":true}\n', { encoding: "utf8" }); +} + function runAwareFetch(failRun?: string): { fetch: Fetch; calls: { count: number } } { const calls = { count: 0 }; const fetch: Fetch = async (_url, init) => { @@ -115,12 +121,14 @@ describe("lg sync usage errors", () => { it("10. hub not configured -> 1", async () => { seedRun("run-a"); + seedOptIn(); const code = await syncCommand({ ...base(), env: {}, home: join(tmp, "absent-home") }); expect(code).toBe(1); }); it("11. an unknown runId -> 1", async () => { seedRun("run-a"); + seedOptIn(); const code = await syncCommand({ ...base(), runId: "ghost" }); expect(code).toBe(1); }); @@ -129,6 +137,7 @@ describe("lg sync usage errors", () => { describe("lg sync ", () => { it("12. a successful single-run sync -> 0", async () => { seedRun("run-a"); + seedOptIn(); const { fetch, calls } = runAwareFetch(); const code = await syncCommand({ ...base(), runId: "run-a", f: fetch }); expect(code).toBe(0); @@ -137,6 +146,7 @@ describe("lg sync ", () => { it("13. a failing single-run sync -> 2", async () => { seedRun("run-a"); + seedOptIn(); const { fetch, calls } = runAwareFetch("run-a"); const code = await syncCommand({ ...base(), runId: "run-a", f: fetch }); expect(code).toBe(2); @@ -149,6 +159,7 @@ describe("lg sync --all", () => { seedRun("run-a"); seedRun("run-b"); seedRun("run-c"); + seedOptIn(); const { fetch, calls } = runAwareFetch("run-b"); const { outs, errs } = captureConsole(); @@ -165,6 +176,7 @@ describe("lg sync --all", () => { seedRun("run-a"); seedRun("run-b"); seedRun("run-c"); + seedOptIn(); const { fetch, calls } = runAwareFetch(); const { outs } = captureConsole(); @@ -199,6 +211,7 @@ describe("lg sync supplies the machine identity", () => { }; new CheckpointStore(runsDir(cwd)).save(state); new EventLog(runsDir(cwd)).append(runId, { kind: "run_started", data: {} }); + seedOptIn(); let pushed: EventBatch | null = null; const fetch: Fetch = async (_url, init) => { @@ -214,3 +227,70 @@ describe("lg sync supplies the machine identity", () => { expect(error).toContain("${HOSTNAME}"); }); }); + +describe("lg sync honours the repo opt-in", () => { + // BUG 5: `repoSyncEnabled` gated only the live batcher (`src/team/batch.ts`). + // `lg sync ` and `lg sync --all` never consulted it, so a repo that + // had never run `lg sync --enable` could still push every run it had. The + // hub's `events` table has no-update/no-delete triggers, which makes "I + // forgot this repo was not opted in" permanent and visible to every + // read-scoped member. The opt-in must gate every push path, not one of them. + + it("17. a run id in a repo that never opted in -> 1, nothing is pushed, and the message names --enable", async () => { + seedRun("run-a"); + const { fetch, calls } = runAwareFetch(); + const { errs } = captureConsole(); + + const code = await syncCommand({ ...base(), runId: "run-a", f: fetch }); + + expect(code).toBe(1); + expect(calls.count).toBe(0); + expect(errs.some((e) => e.includes("lg sync --enable"))).toBe(true); + }); + + it("18. --all in a repo that never opted in -> 1 and pushes nothing, even with runs present", async () => { + seedRun("run-a"); + seedRun("run-b"); + const { fetch, calls } = runAwareFetch(); + captureConsole(); + + const code = await syncCommand({ ...base(), all: true, f: fetch }); + + expect(code).toBe(1); + expect(calls.count).toBe(0); + }); + + it("19. the gate is checked BEFORE the hub config, so an un-opted repo reports the opt-in rather than the enrollment", async () => { + // Both conditions hold at once. The opt-in is the local consent decision + // and the more specific fix, so it is what the operator is told about. + seedRun("run-a"); + const { errs } = captureConsole(); + + const code = await syncCommand({ ...base(), env: {}, home: join(tmp, "absent-home"), runId: "run-a" }); + + expect(code).toBe(1); + expect(errs.some((e) => e.includes("lg sync --enable"))).toBe(true); + expect(errs.some((e) => e.includes("lg enroll"))).toBe(false); + }); + + it("20. a hub.json whose sync flag is not exactly true does not count as opting in", async () => { + seedRun("run-a"); + mkdirSync(join(cwd, ".loomgraph"), { recursive: true }); + writeFileSync(optInPath(), '{"sync":"true"}\n', { encoding: "utf8" }); + const { fetch, calls } = runAwareFetch(); + captureConsole(); + + expect(await syncCommand({ ...base(), runId: "run-a", f: fetch })).toBe(1); + expect(calls.count).toBe(0); + }); + + it("21. --enable then sync works in one sequence", async () => { + seedRun("run-a"); + const { fetch, calls } = runAwareFetch(); + captureConsole(); + + expect(await syncCommand({ ...base(), enable: true })).toBe(0); + expect(await syncCommand({ ...base(), runId: "run-a", f: fetch })).toBe(0); + expect(calls.count).toBe(1); + }); +}); diff --git a/src/commands/sync.ts b/src/commands/sync.ts index 9308583..780b85d 100644 --- a/src/commands/sync.ts +++ b/src/commands/sync.ts @@ -3,14 +3,20 @@ import { homedir, hostname, userInfo } from "node:os"; import { join } from "node:path"; import type { CheckpointStore } from "../core/store.js"; import { syncRun, type ProjectionOpts } from "../team/sync.js"; -import { loadHubConfig, type Fetch, type HubConfig } from "../team/transport.js"; +import { + loadHubConfig, + repoSyncEnabled, + type Fetch, + type HubConfig, +} from "../team/transport.js"; import { openStore, runsDir } from "./context.js"; /** * EXIT CODES - `lg sync` owns 0, 1 and 2, and nothing else: * 0 everything synced (or `--enable` succeeded) * 1 usage error: no runId and no --all and no --enable, a runId and --all - * together, an unknown runId, or the hub is not configured + * together, an unknown runId, the repo has not opted in, or the hub is + * not configured * 2 at least one sync failed * Never 3 or 4 - those are budget-exceeded and paused, and they belong to * `lg run`. @@ -62,6 +68,27 @@ export async function syncCommand(opts: SyncOptions = {}): Promise { return 1; } + // THE REPO OPT-IN GATES EVERY PUSH PATH, NOT JUST THE LIVE ONE. + // + // `repoSyncEnabled` used to be consulted only by the live batcher + // (`src/team/batch.ts`), so `lg sync ` and `lg sync --all` would + // happily push a repo that had never run `lg sync --enable`. The hub's + // `events` table carries no-update/no-delete triggers: a run pushed out of a + // repo nobody meant to share cannot be retracted, and every read-scoped + // member can see it. "I forgot this repo was not opted in" has to be + // impossible, not merely documented. + // + // Checked BEFORE `loadHubConfig` deliberately. When neither the opt-in nor + // the enrollment exists, the opt-in is the more specific problem and the one + // the operator is deciding about; naming `lg enroll` first would send them + // to configure a hub this repo still would not push to. + if (!repoSyncEnabled(cwd)) { + console.error( + `hub sync is not enabled for ${cwd} - run: lg sync --enable (this repo has never opted in)`, + ); + return 1; + } + const home = opts.home ?? homedir(); const cfg = loadHubConfig(opts.env ?? process.env, home); if (cfg === null) { From 7348068ea7702194c7d9d6943b8b98e81f129e58 Mon Sep 17 00:00:00 2001 From: Dat Date: Mon, 21 Sep 2026 09:06:26 +0700 Subject: [PATCH 38/45] feat(deploy): add hub provisioning, backup and NetBird ACL scripts Everything needed to stand a team hub up on a WireGuard mesh, with no deployment's addresses baked in: they live in `deploy/hub.env` (gitignored; `hub.env.example` is the template). A REQUIRED variable aborts the script by name rather than defaulting, because a provisioning script that guesses an IP converges the wrong network - and this repo is public, so shipping one operator's mesh map as a default would be both wrong and rude. - `install-hub.sh` - idempotent provisioning. Installs from a LOCAL tarball, never the registry: loomgraph is not published, so `npm i -g loomgraph` cannot work. Preflight refuses to proceed if the mesh interface is missing, the hub IP is not assigned to it, or the port is held by something else. - `lg-hub.service` - the unit, now a template: `@MESH_IP@` and `@HUB_PORT@` are substituted at install time so the installed file carries concrete values (`systemctl cat` showing a variable would hide the bind address, which is the one thing an operator needs to read). The three ExecStart flags each have a comment block; `--no-ui` is deliberate, since the UI keeps its bearer token in localStorage and there is no browser use case on this bind. - `backup-hub.sh` - `VACUUM INTO` plus a chain verify, with SQLITE_BUSY retry. Copying `hub.db` alone is not a backup: WAL mode means three files. - `netbird-acl.sh` - converges the ACL model (members reach the hub on one TCP port and nothing else, operator keeps mesh SSH, `Default` All -> All disabled but never deleted, for one-call rollback). Dry-run by default, typed confirmation before the lockout step, `--verify` with positive criteria and `--verify --from-member` for the negative ones. The API token goes to curl via `--config -` on stdin, never argv. The sandbox peer and the stale-policy cleanup are both optional and skip cleanly when unconfigured. Nothing here has been run against live infrastructure. Every step needs root and is the operator's to run. Co-Authored-By: Claude Opus 5 (1M context) --- .gitignore | 4 + deploy/backup-hub.sh | 321 +++++++++++ deploy/hub.env.example | 70 +++ deploy/install-hub.sh | 476 ++++++++++++++++ deploy/lg-hub.service | 86 +++ deploy/netbird-acl.sh | 1233 ++++++++++++++++++++++++++++++++++++++++ 6 files changed, 2190 insertions(+) create mode 100755 deploy/backup-hub.sh create mode 100644 deploy/hub.env.example create mode 100755 deploy/install-hub.sh create mode 100644 deploy/lg-hub.service create mode 100755 deploy/netbird-acl.sh diff --git a/.gitignore b/.gitignore index fb5a710..2608014 100644 --- a/.gitignore +++ b/.gitignore @@ -11,3 +11,7 @@ dist/ # Never commit either - the share url grants read access to the artifact. handoff-bundle/ SHARE-URL.txt + +# deploy/hub.env holds one deployment's real addresses: mesh IPs, the control +# plane URL, SSH fallbacks. Copy deploy/hub.env.example and fill it in locally. +deploy/hub.env diff --git a/deploy/backup-hub.sh b/deploy/backup-hub.sh new file mode 100755 index 0000000..0c40f5a --- /dev/null +++ b/deploy/backup-hub.sh @@ -0,0 +1,321 @@ +#!/usr/bin/env bash +# +# backup-hub.sh - WAL-safe, verified backup of the loomgraph hub database. +# +# hub.db runs PRAGMA journal_mode=WAL (src/hub/storage.ts), so it has -wal and +# -shm sidecars and a plain `cp hub.db` produces an INCONSISTENT snapshot: +# committed pages that still live only in the WAL are silently lost. This +# script uses `VACUUM INTO`, which takes a read transaction against the live +# database and writes a fully-checkpointed, self-contained copy. +# +# The store is also a hash chain - row_hash = sha256(prev_hash || json), with +# the head in chain_head (src/hub/storage.ts) - so a torn or partial copy can +# open cleanly and still be corrupt. Every copy is therefore verified before it +# is kept: +# (a) the copy opens, +# (b) PRAGMA integrity_check returns ok, +# (c) the hash chain verifies end to end, from the 32-zero-byte genesis to +# the value stored in chain_head. +# Any failure renames the copy to *.rejected and exits non-zero. +# +# This runs against a LIVE server, and storage.ts sets no busy_timeout on any +# connection, so a lock conflict surfaces immediately as SQLITE_BUSY. The +# snapshot therefore sets its own busy_timeout and retries with exponential +# backoff, and gives up with exit code 3 rather than emitting a partial copy. +# +# Every database access runs as the lghub service user: node:sqlite creates +# hub.db-wal / hub.db-shm on demand, and root-owned sidecars would lock the +# service out of its own store. +# +# Restore is a documented MANUAL procedure, deliberately not a flag here. +# +# Exit codes: 0 ok | 1 failure or failed verification | 2 usage | 3 could not +# acquire a lock (retryable; safe for cron to treat as "try again later"). +# +# Usage: sudo ./backup-hub.sh +# Overrides (env): LGHUB_DATA_DIR, LGHUB_BACKUP_DIR, LGHUB_RETAIN, LGHUB_USER, +# LGHUB_BUSY_TIMEOUT_MS, LGHUB_BUSY_RETRIES + +set -euo pipefail + +readonly DATA_DIR="${LGHUB_DATA_DIR:-/var/lib/lghub}" +readonly BACKUP_DIR="${LGHUB_BACKUP_DIR:-/var/backups/lghub}" +readonly SERVICE_USER="${LGHUB_USER:-lghub}" +readonly RETAIN="${LGHUB_RETAIN:-14}" +readonly DB_PATH="${DATA_DIR}/hub.db" +# storage.ts sets no busy_timeout anywhere, so a lock conflict with the running +# server surfaces instantly as SQLITE_BUSY. This script runs against a live +# server by design, so it sets its own timeout and retries with backoff. +readonly BUSY_TIMEOUT_MS="${LGHUB_BUSY_TIMEOUT_MS:-10000}" +readonly BUSY_RETRIES="${LGHUB_BUSY_RETRIES:-5}" +# Exit codes: 0 ok, 1 failed/verification failed, 2 usage, 3 could not acquire +# a lock (retryable - safe for a cron job to treat as "try again later"). +readonly EXIT_BUSY=3 + +log() { printf 'backup-hub: %s\n' "$*"; } +die() { printf 'backup-hub: FATAL %s\n' "$*" >&2; exit 1; } + +WORK_DIR="" +VERIFIER_PATH="" +cleanup() { + if [ -n "$WORK_DIR" ] && [ -d "$WORK_DIR" ]; then + rm -rf -- "$WORK_DIR" + fi +} +trap cleanup EXIT + +# Run a command as the hub service user so any SQLite sidecar files touched +# during the read stay lghub-owned. Running the read as root can leave a +# root-owned -wal/-shm behind and lock the service out of its own database. +run_as_hub() { + if [ "$(id -un)" = "$SERVICE_USER" ]; then + "$@" + elif [ "$(id -u)" -eq 0 ]; then + runuser -u "$SERVICE_USER" -- "$@" + else + die "must run as root or as ${SERVICE_USER} (current user: $(id -un))" + fi +} + +preflight() { + local cmd + for cmd in node find sort; do + command -v "$cmd" >/dev/null 2>&1 || die "missing required command: ${cmd}" + done + if [ "$(id -un)" != "$SERVICE_USER" ] && [ "$(id -u)" -ne 0 ]; then + die "must run as root or as ${SERVICE_USER} (current user: $(id -un))" + fi + [ -f "$DB_PATH" ] || die "database not found: ${DB_PATH}" + if ! [[ "$RETAIN" =~ ^[0-9]+$ ]] || [ "$RETAIN" -lt 1 ]; then + die "LGHUB_RETAIN must be a positive integer, got: ${RETAIN}" + fi + if ! [[ "$BUSY_RETRIES" =~ ^[0-9]+$ ]] || [ "$BUSY_RETRIES" -lt 1 ]; then + die "LGHUB_BUSY_RETRIES must be a positive integer, got: ${BUSY_RETRIES}" + fi + if ! [[ "$BUSY_TIMEOUT_MS" =~ ^[0-9]+$ ]]; then + die "LGHUB_BUSY_TIMEOUT_MS must be a non-negative integer, got: ${BUSY_TIMEOUT_MS}" + fi + if [ ! -d "$BACKUP_DIR" ]; then + if [ "$(id -u)" -eq 0 ]; then + install -d -o "$SERVICE_USER" -g "$SERVICE_USER" -m 0750 "$BACKUP_DIR" + log "created ${BACKUP_DIR} (0750 ${SERVICE_USER}:${SERVICE_USER})" + else + die "backup directory does not exist and cannot be created as a non-root user: ${BACKUP_DIR}" + fi + fi +} + +# Sets WORK_DIR and VERIFIER_PATH. Must NOT be called in a command +# substitution: the subshell would discard WORK_DIR and leak the temp dir past +# the EXIT trap. +write_verifier() { + WORK_DIR="$(mktemp -d)" + # World-readable so the unprivileged service user can read the script when + # this runs under runuser. It contains no secrets. + chmod 0755 "$WORK_DIR" + VERIFIER_PATH="${WORK_DIR}/vacuum-and-verify.mjs" + cat >"$VERIFIER_PATH" <<'VERIFIER_EOF' +// VACUUM INTO a live WAL database, then verify the copy: it opens, passes +// PRAGMA integrity_check, and its hash chain is continuous end to end. +// +// Chain construction copied from src/hub/storage.ts: +// genesis = 32 zero bytes (chain_head seed) +// row_hash = sha256(prev_hash || json) json is the client line, utf8, verbatim +// chain_head.head = row_hash of the most recently inserted event +// Insertion order is rowid order: `events` is an ordinary rowid table and +// triggers forbid UPDATE and DELETE, so rowids are append-only. +import { DatabaseSync } from "node:sqlite"; +import { createHash } from "node:crypto"; +import { rmSync } from "node:fs"; + +const [srcPath, dstPath, busyTimeoutMsArg, retriesArg] = process.argv.slice(2); +if (!srcPath || !dstPath) { + console.error("usage: vacuum-and-verify.mjs [busyTimeoutMs] [retries]"); + process.exit(2); +} +const busyTimeoutMs = Number(busyTimeoutMsArg ?? 10000); +const maxAttempts = Number(retriesArg ?? 5); + +function fail(message) { + console.error(`backup-hub: VERIFY FAILED: ${message}`); + process.exit(1); +} + +function toBuffer(value, what) { + if (value === null || value === undefined) fail(`${what} is NULL`); + return Buffer.from(value); +} + +/** + * src/hub/storage.ts sets no busy_timeout on any connection, so a lock + * conflict surfaces immediately as SQLITE_BUSY rather than waiting. This + * script runs against a live server by design, so it must expect that and + * retry rather than emit a partial or missing snapshot. + * + * Observed shape from node:sqlite: code ERR_SQLITE_ERROR, errcode 5, + * errstr "database is locked". 261 = SQLITE_BUSY_SNAPSHOT, 517 = + * SQLITE_BUSY_TIMEOUT. + */ +function isBusy(err) { + const code = err?.errcode; + if (code === 5 || code === 261 || code === 517) return true; + return /database is locked|database table is locked/i.test(String(err?.message ?? "")); +} + +/** Synchronous sleep: this script is deliberately straight-line. */ +function sleepSync(ms) { + Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, ms); +} + +// 1. Snapshot. VACUUM INTO reads the live database under a read transaction +// and writes a checkpointed, standalone copy - no -wal/-shm needed. +// It either completes or throws; a throw leaves a partial file behind, +// which is removed before the next attempt so no torn copy can survive. +let snapshotted = false; +for (let attempt = 1; attempt <= maxAttempts && !snapshotted; attempt += 1) { + let src; + try { + src = new DatabaseSync(srcPath, { readOnly: true }); + src.exec(`PRAGMA busy_timeout = ${Math.trunc(busyTimeoutMs)}`); + src.exec(`VACUUM INTO '${dstPath.replace(/'/g, "''")}'`); + snapshotted = true; + } catch (err) { + rmSync(dstPath, { force: true }); + if (!isBusy(err)) { + console.error( + `backup-hub: VACUUM INTO failed: ${err instanceof Error ? err.message : String(err)}`, + ); + process.exit(1); + } + if (attempt === maxAttempts) { + console.error( + `backup-hub: BUSY: could not acquire a read lock on ${srcPath} after ${maxAttempts} ` + + `attempt(s) with a ${busyTimeoutMs}ms busy_timeout each. No backup was produced. ` + + `Something is holding a long write lock - check the hub server and any manual ` + + `lg-hub command.`, + ); + process.exit(3); + } + const backoffMs = Math.min(1000 * 2 ** (attempt - 1), 16000); + console.error( + `backup-hub: SQLITE_BUSY on attempt ${attempt}/${maxAttempts}; retrying in ${backoffMs}ms`, + ); + sleepSync(backoffMs); + } finally { + if (src !== undefined) src.close(); + } +} + +// 2. The copy opens, and 3. integrity_check / chain continuity. +let dst; +try { + dst = new DatabaseSync(dstPath, { readOnly: true }); + + const integrity = dst.prepare("PRAGMA integrity_check").all(); + const verdict = integrity.map((r) => String(r.integrity_check)).join("; "); + if (integrity.length !== 1 || verdict !== "ok") { + fail(`PRAGMA integrity_check returned: ${verdict}`); + } + + const genesis = Buffer.alloc(32); + let expected = genesis; + let count = 0; + const rows = dst + .prepare("SELECT rowid AS rid, json, prev_hash, row_hash FROM events ORDER BY rowid") + .iterate(); + for (const row of rows) { + const prev = toBuffer(row.prev_hash, `events.prev_hash at rowid ${row.rid}`); + const stored = toBuffer(row.row_hash, `events.row_hash at rowid ${row.rid}`); + if (!prev.equals(expected)) { + fail( + `chain break at rowid ${row.rid}: stored prev_hash ${prev.toString("hex")} ` + + `!= previous row_hash ${expected.toString("hex")}`, + ); + } + const computed = createHash("sha256").update(expected).update(String(row.json), "utf8").digest(); + if (!computed.equals(stored)) { + fail( + `chain break at rowid ${row.rid}: stored row_hash ${stored.toString("hex")} ` + + `!= sha256(prev_hash || json) ${computed.toString("hex")}`, + ); + } + expected = computed; + count += 1; + } + + const headRow = dst.prepare("SELECT head FROM chain_head WHERE id=1").get(); + if (headRow === undefined) fail("chain_head has no row with id=1"); + const head = toBuffer(headRow.head, "chain_head.head"); + if (!head.equals(expected)) { + fail( + `chain_head ${head.toString("hex")} does not match the last row hash ` + + `${expected.toString("hex")} (${count} event(s) walked)`, + ); + } + + console.log( + `backup-hub: verified - integrity_check ok, ${count} event(s), chain head ${head.toString("hex")}`, + ); +} catch (err) { + console.error(`backup-hub: verification error: ${err instanceof Error ? err.message : String(err)}`); + process.exit(1); +} finally { + if (dst !== undefined) dst.close(); +} +VERIFIER_EOF + chmod 0644 "$VERIFIER_PATH" +} + +prune() { + local stale + # Names are UTC timestamps, so lexical sort is chronological. + stale="$(find "$BACKUP_DIR" -maxdepth 1 -type f -name 'hub-*.db' -printf '%f\n' | + sort -r | tail -n "+$((RETAIN + 1))" || true)" + if [ -z "$stale" ]; then + log "retention: ${RETAIN} copies kept, nothing to prune" + return 0 + fi + local name + while IFS= read -r name; do + [ -n "$name" ] || continue + rm -f -- "${BACKUP_DIR}/${name}" + log "pruned ${name}" + done <<<"$stale" +} + +main() { + preflight + + local stamp out + write_verifier + stamp="$(date -u +%Y%m%dT%H%M%SZ)" + out="${BACKUP_DIR}/hub-${stamp}.db" + + if [ -e "$out" ]; then + die "backup target already exists: ${out}" + fi + + log "snapshotting ${DB_PATH} -> ${out}" + local rc=0 + run_as_hub node --disable-warning=ExperimentalWarning \ + "$VERIFIER_PATH" "$DB_PATH" "$out" "$BUSY_TIMEOUT_MS" "$BUSY_RETRIES" || rc=$? + if [ "$rc" -ne 0 ]; then + if [ "$rc" -eq "$EXIT_BUSY" ]; then + # The verifier already removed any partial file before giving up. + printf 'backup-hub: FATAL could not acquire a lock on %s; NO backup was produced. Retry later.\n' \ + "$DB_PATH" >&2 + exit "$EXIT_BUSY" + fi + if [ -e "$out" ]; then + mv -- "$out" "${out}.rejected" + die "backup verification failed; copy kept for inspection at ${out}.rejected (it is NOT a usable backup)" + fi + die "backup failed before a copy was produced" + fi + + chmod 0640 "$out" + log "backup complete: ${out} ($(du -h -- "$out" | cut -f1))" + prune +} + +main "$@" diff --git a/deploy/hub.env.example b/deploy/hub.env.example new file mode 100644 index 0000000..0019141 --- /dev/null +++ b/deploy/hub.env.example @@ -0,0 +1,70 @@ +# deploy/hub.env - YOUR deployment's real values. COPY, DO NOT EDIT THIS FILE. +# +# cp deploy/hub.env.example deploy/hub.env # then fill it in +# +# `deploy/hub.env` is gitignored on purpose. None of these are secrets in the +# credential sense - they are addresses - but a public repo does not need a map +# of your mesh, and the scripts refuse to run on someone else's defaults. +# +# Every script under deploy/ sources this file if it exists, then falls back to +# the environment. A variable marked REQUIRED aborts the script when unset, with +# a message naming it - that is deliberate: a deployment script that guesses an +# IP address converges the wrong network. + +# --- NetBird control plane ------------------------------------------------- + +# REQUIRED. Base URL of your NetBird management API, including /api. +# Self-hosted: https://netbird.example.com/api +# NetBird cloud: https://api.netbird.io/api +NETBIRD_API= + +# Keychain service name the token is stored under (macOS). The token itself is +# NEVER put in this file - netbird-acl.sh reads it from the Keychain or an +# `nbtoken` helper and passes it to curl on stdin. +NETBIRD_KEYCHAIN_SERVICE=netbird-pat + +# --- Peers ----------------------------------------------------------------- + +# REQUIRED. Mesh IP of the peer that runs lg-hub. +LOOMGRAPH_HUB_IP= + +# Display name for that peer in log output. Cosmetic only. +LOOMGRAPH_HUB_PEER_NAME=the hub peer + +# Mesh IP of a peer the operator keeps SSH to besides the hub. Leave empty if +# you have none: the sandbox group and its policy are then skipped entirely. +LOOMGRAPH_SANDBOX_IP= + +# REQUIRED for `--verify --from-member`. Comma-separated mesh IPs that a member +# peer must NOT be able to reach. These are the negative acceptance criteria. +LOOMGRAPH_MAC_IPS= + +# --- Hub service ----------------------------------------------------------- + +LOOMGRAPH_HUB_PORT=8369 +LOOMGRAPH_HEALTH_PATH=/v1/health + +# Regex the /v1/health BODY must match. Leave empty until you have seen the real +# payload once - the built-in check is a heuristic, and a status code is not a +# health check here (the hub serves its UI for any non-/v1 GET, so /healthz +# returns 200 HTML even when the API is dead). +LOOMGRAPH_HEALTH_EXPECT= + +# --- Lockout fallbacks ----------------------------------------------------- + +# Optional but strongly recommended: a host:port that still reaches the hub +# WITHOUT the mesh, used in the warning printed before the All -> All policy is +# disabled. Empty means the warning says you have no fallback - which may be +# true, and you should know it before continuing. +LOOMGRAPH_HUB_PUBLIC_SSH= + +# Optional. `user@host` for the by-hand post-apply check of mesh SSH. +LOOMGRAPH_SANDBOX_SSH= +LOOMGRAPH_HUB_SSH= + +# --- One-off cleanup ------------------------------------------------------- + +# Optional. Exact name of a stale policy to delete during --apply. Empty skips +# that step. NetBird names auto-created policies +# "Temporary access policy for peer ". +NETBIRD_DEAD_POLICY= diff --git a/deploy/install-hub.sh b/deploy/install-hub.sh new file mode 100755 index 0000000..ac99312 --- /dev/null +++ b/deploy/install-hub.sh @@ -0,0 +1,476 @@ +#!/usr/bin/env bash +# +# install-hub.sh - provision the loomgraph hub on the mesh host. +# +# Idempotent: safe to re-run. A second run against an already-provisioned host +# changes nothing and reports "no changes". +# +# Assumes node >= 22.13 and npm are ALREADY installed (developed against node +# v22.22.1, npm 10.9.4, systemd 257, Ubuntu 25.04). This script never installs +# or upgrades node. +# +# Refuses to proceed if: +# - node < 22.13 +# - port 8369 is already bound by something that is not lg-hub.service +# - the mesh interface is absent, or LOOMGRAPH_HUB_IP is not assigned to it +# +# PACKAGE SOURCE - read this before running. +# +# loomgraph is NOT published to the public npm registry: fetching +# https://registry.npmjs.org/loomgraph returns "Not found". `npm i -g loomgraph` +# (as the README currently documents) cannot work, so this script installs from +# a LOCAL artifact by default and never falls back to the registry for the +# loomgraph package itself. +# +# Build the artifact on a machine with the repo checked out: +# +# npm ci +# npm run build # tsup -> dist/ ; the bins point at dist/, so this +# # is mandatory, `npm pack` will not build for you +# npm pack # produces loomgraph-.tgz in the repo root +# scp loomgraph-.tgz :/tmp/ +# +# Then, on the host: +# +# sudo LOOMGRAPH_PACKAGE=/tmp/loomgraph-.tgz ./install-hub.sh +# +# With no LOOMGRAPH_PACKAGE set, the script looks for loomgraph-.tgz +# beside itself, in the repo root, and in the current directory, then falls back +# to a built repo working tree (one containing dist/hub/cli.js). If it finds +# none of those it exits with instructions rather than letting npm emit a bare +# 404. Whichever source is used, its version must equal the pinned version. +# +# Note: loomgraph's own runtime dependencies (commander, execa, yaml, zod) still +# come from the registry, so the host needs network access to it. +# +# Usage: sudo LOOMGRAPH_PACKAGE=/tmp/loomgraph-0.1.0.tgz ./install-hub.sh +# Overrides (env): LOOMGRAPH_VERSION, LOOMGRAPH_PACKAGE + +set -euo pipefail + +SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)" + +readonly SERVICE_NAME="lg-hub.service" +readonly SERVICE_USER="lghub" +readonly SERVICE_GROUP="lghub" +readonly DATA_DIR="/var/lib/lghub" +readonly UNIT_SRC="${SCRIPT_DIR}/lg-hub.service" +readonly UNIT_DST="/etc/systemd/system/${SERVICE_NAME}" +readonly BIN_PATH="/usr/bin/lg-hub" +# Deployment addresses come from deploy/hub.env (gitignored; copy +# deploy/hub.env.example). Nothing host-specific is baked into this script - a +# provisioning script that defaults to someone else's mesh IP binds the wrong +# address and the unit fails to start with a confusing error. +ENV_FILE="${LOOMGRAPH_HUB_ENV:-${SCRIPT_DIR}/hub.env}" +if [ -f "$ENV_FILE" ]; then + set -a + # shellcheck source=/dev/null + . "$ENV_FILE" + set +a +fi + +readonly MESH_IFACE="${LOOMGRAPH_MESH_IFACE:-wt0}" +readonly MESH_IP="${LOOMGRAPH_HUB_IP:-}" +readonly HUB_PORT="${LOOMGRAPH_HUB_PORT:-8369}" +readonly MIN_NODE_MAJOR=22 +readonly MIN_NODE_MINOR=13 + +# Pinned version fallback. The repo's package.json wins when this script is run +# from a checkout; never @latest either way. +DEFAULT_VERSION="0.1.0" + +CHANGED=0 +# Set by resolve_package_source. A global rather than a command substitution so +# that a failure inside it exits the script instead of a subshell. +PACKAGE_SOURCE="" + +log() { printf 'install-hub: %s\n' "$*"; } +step() { printf 'install-hub: [changed] %s\n' "$*"; CHANGED=$((CHANGED + 1)); } +die() { printf 'install-hub: FATAL %s\n' "$*" >&2; exit 1; } + +# --- preconditions ---------------------------------------------------------- + +require_root() { + if [ "$(id -u)" -ne 0 ]; then + die "must run as root (try: sudo $0)" + fi +} + +require_commands() { + local missing=() + local cmd + for cmd in node npm tar ss ip systemctl useradd groupadd install; do + if ! command -v "$cmd" >/dev/null 2>&1; then + missing+=("$cmd") + fi + done + if [ "${#missing[@]}" -gt 0 ]; then + die "missing required commands: ${missing[*]}" + fi + if ! command -v runuser >/dev/null 2>&1 && ! command -v sudo >/dev/null 2>&1; then + die "need runuser or sudo to drop privileges to ${SERVICE_USER}; neither is installed" + fi +} + +# Never run lg-hub as root. node:sqlite creates hub.db-wal and hub.db-shm +# alongside the database on first write; if root creates them, they end up +# root-owned and the service - which runs as lghub under ProtectSystem=strict - +# can no longer write its own store. Every lg-hub invocation goes through here. +run_as_hub() { + if command -v runuser >/dev/null 2>&1; then + runuser -u "$SERVICE_USER" -- "$@" + else + sudo -u "$SERVICE_USER" -- "$@" + fi +} + +check_node_version() { + local raw major minor + raw="$(node -v)" # e.g. v22.22.1 + raw="${raw#v}" + major="${raw%%.*}" + minor="${raw#*.}" + minor="${minor%%.*}" + if ! [[ "$major" =~ ^[0-9]+$ && "$minor" =~ ^[0-9]+$ ]]; then + die "could not parse node version from 'node -v' output: $(node -v)" + fi + if [ "$major" -lt "$MIN_NODE_MAJOR" ] || + { [ "$major" -eq "$MIN_NODE_MAJOR" ] && [ "$minor" -lt "$MIN_NODE_MINOR" ]; }; then + die "node $(node -v) is too old; loomgraph requires >= ${MIN_NODE_MAJOR}.${MIN_NODE_MINOR} (node:sqlite is only usable unflagged from 22.13). This script does not install node." + fi + log "node $(node -v) satisfies >= ${MIN_NODE_MAJOR}.${MIN_NODE_MINOR}" +} + +check_config() { + if [ -z "$MESH_IP" ]; then + die "LOOMGRAPH_HUB_IP is not set (the mesh address lg-hub binds). Copy deploy/hub.env.example to ${ENV_FILE} and fill it in, or export it for this run." + fi +} + +check_mesh() { + if ! ip -o link show dev "$MESH_IFACE" >/dev/null 2>&1; then + die "interface ${MESH_IFACE} does not exist - NetBird is not up. The unit binds ${MESH_IP} and will not start without it." + fi + if ! ip -o -4 addr show dev "$MESH_IFACE" | grep -Fq " ${MESH_IP}/"; then + die "${MESH_IP} is not assigned to ${MESH_IFACE}. Check 'ip -4 addr show dev ${MESH_IFACE}' and the NetBird peer configuration." + fi + log "${MESH_IP} is assigned to ${MESH_IFACE}" +} + +check_port_free() { + local holders + holders="$(ss -ltnH "sport = :${HUB_PORT}" 2>/dev/null || true)" + if [ -z "$holders" ]; then + log "port ${HUB_PORT} is free" + return 0 + fi + # Our own service already listening is the expected state on a re-run. + if systemctl is-active --quiet "$SERVICE_NAME"; then + log "port ${HUB_PORT} is held by ${SERVICE_NAME} (already provisioned)" + return 0 + fi + printf '%s\n' "$holders" >&2 + die "port ${HUB_PORT} is already bound by a process that is not ${SERVICE_NAME} (listeners above). Refusing to provision over it." +} + +# --- provisioning steps ----------------------------------------------------- + +resolve_version() { + local pkg_json="${SCRIPT_DIR}/../package.json" + if [ -n "${LOOMGRAPH_VERSION:-}" ]; then + printf '%s' "$LOOMGRAPH_VERSION" + return 0 + fi + if [ -f "$pkg_json" ]; then + local v + v="$(node -e 'const p=require(process.argv[1]); process.stdout.write(String(p.version ?? ""))' "$pkg_json")" + if [ -n "$v" ]; then + printf '%s' "$v" + return 0 + fi + fi + printf '%s' "$DEFAULT_VERSION" +} + +installed_version() { + local root + root="$(npm root -g 2>/dev/null || true)" + if [ -z "$root" ] || [ ! -f "${root}/loomgraph/package.json" ]; then + return 0 + fi + node -e 'const p=require(process.argv[1]); process.stdout.write(String(p.version ?? ""))' \ + "${root}/loomgraph/package.json" 2>/dev/null || true +} + +ensure_user() { + if ! getent group "$SERVICE_GROUP" >/dev/null 2>&1; then + groupadd --system "$SERVICE_GROUP" + step "created system group ${SERVICE_GROUP}" + fi + if ! getent passwd "$SERVICE_USER" >/dev/null 2>&1; then + useradd --system \ + --gid "$SERVICE_GROUP" \ + --home-dir "$DATA_DIR" \ + --no-create-home \ + --shell /usr/sbin/nologin \ + --comment "loomgraph hub service account" \ + "$SERVICE_USER" + step "created system user ${SERVICE_USER} (no login shell)" + fi +} + +ensure_data_dir() { + if [ ! -d "$DATA_DIR" ]; then + install -d -o "$SERVICE_USER" -g "$SERVICE_GROUP" -m 0750 "$DATA_DIR" + step "created ${DATA_DIR} (0750 ${SERVICE_USER}:${SERVICE_GROUP})" + return 0 + fi + local mode owner + mode="$(stat -c '%a' "$DATA_DIR")" + owner="$(stat -c '%U:%G' "$DATA_DIR")" + if [ "$mode" != "750" ]; then + chmod 0750 "$DATA_DIR" + step "fixed ${DATA_DIR} mode ${mode} -> 750" + fi + if [ "$owner" != "${SERVICE_USER}:${SERVICE_GROUP}" ]; then + chown -R "${SERVICE_USER}:${SERVICE_GROUP}" "$DATA_DIR" + step "fixed ${DATA_DIR} owner ${owner} -> ${SERVICE_USER}:${SERVICE_GROUP}" + fi +} + +# Read the version out of a packed tarball without unpacking it to disk. +tarball_version() { + tar -xzOf "$1" package/package.json 2>/dev/null | + node -e 'let s="";process.stdin.on("data",(d)=>{s+=d;}).on("end",()=>{try{process.stdout.write(String(JSON.parse(s).version ?? ""));}catch{}});' +} + +# Sets PACKAGE_SOURCE. Never resolves to the public registry on its own: +# loomgraph is not published there. +resolve_package_source() { + local want="$1" candidate found + PACKAGE_SOURCE="" + + if [ -n "${LOOMGRAPH_PACKAGE:-}" ]; then + if [ -e "$LOOMGRAPH_PACKAGE" ]; then + PACKAGE_SOURCE="$LOOMGRAPH_PACKAGE" + log "package source: ${PACKAGE_SOURCE} (from LOOMGRAPH_PACKAGE)" + return 0 + fi + case "$LOOMGRAPH_PACKAGE" in + /* | ./* | ../* | *.tgz | *.tar.gz) + die "LOOMGRAPH_PACKAGE points at a path that does not exist: ${LOOMGRAPH_PACKAGE}" + ;; + esac + # Not a path: treat as an npm spec. Opt-in only, and only meaningful + # against a private registry. + log "WARNING: LOOMGRAPH_PACKAGE='${LOOMGRAPH_PACKAGE}' is not an existing path, so it is" + log " being passed to npm as a package spec. loomgraph is NOT on the public" + log " registry, so this only works against a private/mirrored one." + PACKAGE_SOURCE="$LOOMGRAPH_PACKAGE" + return 0 + fi + + for candidate in \ + "${SCRIPT_DIR}/loomgraph-${want}.tgz" \ + "${SCRIPT_DIR}/../loomgraph-${want}.tgz" \ + "./loomgraph-${want}.tgz"; do + if [ -f "$candidate" ]; then + PACKAGE_SOURCE="$candidate" + log "package source: ${PACKAGE_SOURCE} (auto-discovered tarball)" + return 0 + fi + done + + # A built working tree is acceptable; an unbuilt one is not - the bins point + # at dist/ and npm will not run the build for us (there is no prepare script). + if [ -f "${SCRIPT_DIR}/../package.json" ]; then + found="$(cd -- "${SCRIPT_DIR}/.." && pwd)" + if [ -f "${found}/dist/hub/cli.js" ]; then + PACKAGE_SOURCE="$found" + log "package source: ${PACKAGE_SOURCE} (built repo working tree)" + return 0 + fi + die "found a repo working tree at ${found} but ${found}/dist/hub/cli.js is missing - run 'npm ci && npm run build' there first, or build a tarball with 'npm pack' and pass LOOMGRAPH_PACKAGE=/path/to/loomgraph-${want}.tgz" + fi + + die "no loomgraph package to install. loomgraph is NOT published to npm, so there is nothing to fetch. On a machine with the repo: 'npm ci && npm run build && npm pack' produces loomgraph-${want}.tgz; copy it to this host and re-run with LOOMGRAPH_PACKAGE=/path/to/loomgraph-${want}.tgz" +} + +ensure_package() { + local want have src_version + want="$1" + have="$(installed_version)" + if [ "$have" = "$want" ]; then + log "loomgraph ${want} already installed globally" + return 0 + fi + + resolve_package_source "$want" + + # The pin has to mean something: check the artifact really is the version we + # think we are installing, before npm touches the system. + if [ -f "$PACKAGE_SOURCE" ]; then + src_version="$(tarball_version "$PACKAGE_SOURCE")" + if [ -z "$src_version" ]; then + die "could not read a version from ${PACKAGE_SOURCE}; is it an 'npm pack' tarball?" + fi + if [ "$src_version" != "$want" ]; then + die "version mismatch: ${PACKAGE_SOURCE} contains loomgraph ${src_version}, but the pinned version is ${want}. Set LOOMGRAPH_VERSION=${src_version} if that tarball is what you mean to install." + fi + elif [ -d "$PACKAGE_SOURCE" ]; then + src_version="$(node -e 'const p=require(process.argv[1]); process.stdout.write(String(p.version ?? ""))' "${PACKAGE_SOURCE}/package.json")" + if [ "$src_version" != "$want" ]; then + die "version mismatch: ${PACKAGE_SOURCE} is loomgraph ${src_version}, but the pinned version is ${want}" + fi + fi + + log "installing loomgraph ${want} from ${PACKAGE_SOURCE} (pinned; never @latest)" + npm install -g --no-fund --no-audit -- "$PACKAGE_SOURCE" + have="$(installed_version)" + if [ "$have" != "$want" ]; then + die "after installing ${PACKAGE_SOURCE} the global loomgraph reports version '${have:-}', expected '${want}'" + fi + step "installed loomgraph ${want} globally from ${PACKAGE_SOURCE}" +} + +ensure_bin_path() { + local prefix real + prefix="$(npm prefix -g)" + real="${prefix}/bin/lg-hub" + if [ ! -x "$real" ]; then + die "npm reports global prefix ${prefix} but ${real} is missing or not executable" + fi + if [ "$real" = "$BIN_PATH" ]; then + log "${BIN_PATH} provided directly by the npm global prefix" + return 0 + fi + if [ -L "$BIN_PATH" ] && [ "$(readlink -f "$BIN_PATH")" = "$(readlink -f "$real")" ]; then + log "${BIN_PATH} already links to ${real}" + return 0 + fi + if [ -e "$BIN_PATH" ] && [ ! -L "$BIN_PATH" ]; then + die "${BIN_PATH} exists and is not a symlink; refusing to replace it. The unit's ExecStart expects ${BIN_PATH}." + fi + ln -sfn "$real" "$BIN_PATH" + step "linked ${BIN_PATH} -> ${real}" +} + +ensure_db() { + if [ -f "${DATA_DIR}/hub.db" ]; then + log "${DATA_DIR}/hub.db already exists" + check_store_ownership + return 0 + fi + # --data-dir is passed explicitly: resolveDataDir() would otherwise fall back + # to $HOME/.local/share/loomgraph-hub, which is not where the unit looks. + run_as_hub "$BIN_PATH" init --data-dir "$DATA_DIR" + if [ ! -f "${DATA_DIR}/hub.db" ]; then + die "'lg-hub init' completed but ${DATA_DIR}/hub.db was not created" + fi + step "initialised ${DATA_DIR}/hub.db as ${SERVICE_USER}" + check_store_ownership +} + +# A root-owned hub.db-wal or hub.db-shm means somebody ran lg-hub as root. The +# service cannot write through it, so surface it rather than let the unit fail +# with an opaque SQLITE_CANTOPEN later. +check_store_ownership() { + local f owner stray=0 + for f in "${DATA_DIR}/hub.db" "${DATA_DIR}/hub.db-wal" "${DATA_DIR}/hub.db-shm"; do + [ -e "$f" ] || continue + owner="$(stat -c '%U' "$f")" + if [ "$owner" != "$SERVICE_USER" ]; then + printf 'install-hub: %s is owned by %s, expected %s\n' "$f" "$owner" "$SERVICE_USER" >&2 + stray=1 + fi + done + if [ "$stray" -eq 1 ]; then + die "store files are not owned by ${SERVICE_USER} (see above). Something ran lg-hub as root. Stop the service, 'chown -R ${SERVICE_USER}:${SERVICE_GROUP} ${DATA_DIR}', and re-run." + fi +} + +ensure_unit() { + if [ ! -f "$UNIT_SRC" ]; then + die "unit file not found next to this script: ${UNIT_SRC}" + fi + + # The unit ships as a TEMPLATE: @MESH_IP@ and @HUB_PORT@ are substituted here + # so the installed file carries concrete values. systemd could expand an + # EnvironmentFile instead, but then `systemctl cat` shows a variable and the + # operator has to go find what it resolved to - exactly the wrong trade when + # the value being hidden is the bind address. + local rendered + rendered="$(mktemp)" + # shellcheck disable=SC2064 # expand now: the path must survive this function + trap "rm -f '${rendered}'" RETURN + sed -e "s|@MESH_IP@|${MESH_IP}|g" -e "s|@HUB_PORT@|${HUB_PORT}|g" "$UNIT_SRC" > "$rendered" + if grep -q '@MESH_IP@\|@HUB_PORT@' "$rendered"; then + die "unit template still contains an unsubstituted placeholder after rendering" + fi + + local unit_changed=0 + if ! cmp -s "$rendered" "$UNIT_DST"; then + install -o root -g root -m 0644 "$rendered" "$UNIT_DST" + systemctl daemon-reload + step "installed ${UNIT_DST}" + unit_changed=1 + else + log "${UNIT_DST} already up to date" + fi + + if ! systemctl is-enabled --quiet "$SERVICE_NAME" 2>/dev/null; then + systemctl enable "$SERVICE_NAME" + step "enabled ${SERVICE_NAME}" + fi + + if ! systemctl is-active --quiet "$SERVICE_NAME"; then + systemctl start "$SERVICE_NAME" + step "started ${SERVICE_NAME}" + elif [ "$unit_changed" -eq 1 ] || [ "$CHANGED" -gt 0 ]; then + systemctl restart "$SERVICE_NAME" + step "restarted ${SERVICE_NAME} (unit or package changed)" + else + log "${SERVICE_NAME} already running with the current unit and package" + fi +} + +report() { + if [ "$CHANGED" -eq 0 ]; then + log "no changes - host already provisioned" + else + log "${CHANGED} change(s) applied" + fi + log "verify with:" + log " systemctl status ${SERVICE_NAME}" + log " ss -tln | grep ${HUB_PORT} # expect ${MESH_IP}:${HUB_PORT}, nothing on 0.0.0.0:${HUB_PORT}" + log "" + log "the web UI is disabled (--no-ui in the unit); the JSON API is the only surface." + log "NEVER run lg-hub as root - it would leave root-owned hub.db-wal/-shm and lock" + log "the service out of its own store. Add a member like this:" + log " runuser -u ${SERVICE_USER} -- ${BIN_PATH} member add --data-dir ${DATA_DIR}" + log "note: the store sets no busy_timeout, so a member/export command issued while" + log "the server is mid-ingest can fail with SQLITE_BUSY. Re-run it if it does." +} + +main() { + require_root + require_commands + check_config + check_node_version + check_mesh + check_port_free + + local version + version="$(resolve_version)" + log "pinned loomgraph version: ${version}" + + ensure_user + ensure_data_dir + ensure_package "$version" + ensure_bin_path + ensure_db + ensure_unit + report +} + +main "$@" diff --git a/deploy/lg-hub.service b/deploy/lg-hub.service new file mode 100644 index 0000000..4d21ce3 --- /dev/null +++ b/deploy/lg-hub.service @@ -0,0 +1,86 @@ +[Unit] +Description=loomgraph team hub (lg-hub) +After=network-online.target netbird.service +Wants=network-online.target netbird.service + +[Service] +Type=simple +User=lghub +Group=lghub + +# DATA DIRECTORY - must be explicit. +# resolveDataDir() (src/hub/serve.ts) falls back to $HOME/.local/share/ +# loomgraph-hub when neither --data-dir nor LOOMGRAPH_HUB_DIR is set. That +# default is broken for this unit: lghub is a system account whose home IS +# /var/lib/lghub, and ProtectHome=yes makes the usual home paths unreadable +# anyway. --data-dir on ExecStart below is authoritative; this env var is a +# backstop so any lg-hub process started in this unit's environment lands on +# the same store even if ExecStart is later edited. +Environment=LOOMGRAPH_HUB_DIR=/var/lib/lghub + +# --------------------------------------------------------------------------- +# READ THIS BEFORE CHANGING --behind-tls-proxy OR THE BIND ADDRESS. +# +# There is NO TLS proxy in front of this service. Nothing terminates TLS for +# it, and nothing is planned to. The flag's name is misleading; it is passed +# for the reason below, not because a proxy exists. +# +# @MESH_IP@ - rendered by install-hub.sh from LOOMGRAPH_HUB_IP - is a NetBird +# (WireGuard) mesh address. Every packet between mesh peers is already +# encrypted by WireGuard (ChaCha20-Poly1305), and the +# address is not routable from the public internet - public exposure on this +# host is limited to 22/80/443, and 8369 is bound to the mesh address only. +# That is what protects the bearer token on the wire. +# +# --behind-tls-proxy affects exactly ONE thing: the startup bind check in +# refuseBind() (src/hub/server.ts), which otherwise refuses any non-loopback +# bind on the grounds that a bearer token would cross plaintext HTTP. It is +# read once at startup (src/hub/serve.ts) and never again. There is NO +# request-time behaviour attached to it: the server does not read +# X-Forwarded-For, X-Real-IP, or any other proxy header, and does not derive +# identity, address, or scheme from request headers. Passing the flag +# therefore cannot introduce a header-spoofing vector. +# +# THIS REASONING DEPENDS ON THE MESH. If NetBird is removed, or this service +# is rebound to a routable address, the justification is void and the flag +# MUST be removed (and the transport decision re-made) before that change +# ships. +# --------------------------------------------------------------------------- + +# --------------------------------------------------------------------------- +# WHY --no-ui. +# +# `lg-hub serve` serves a web UI on the SAME ORIGIN as the JSON API by default +# (src/hub/serve.ts). That UI stores the member bearer token in localStorage +# (src/hub/ui.ts: TOKEN_KEY = "lg-hub-token") against an http:// origin, and +# the API it drives includes POST /v1/members, which mints a NEW member token +# for any caller holding an admin-scoped token (src/hub/handlers.ts). +# +# Persisting a bearer token in localStorage on a plaintext origin, next to a +# token-minting endpoint, is a materially larger blast radius than the JSON API +# alone - any XSS or hostile page reachable on that origin reads the token and +# can mint more. The hub has no browser-facing use case here, so the UI is +# switched off rather than accepted. Re-enabling it means re-doing this +# assessment, not just deleting a flag. +# --------------------------------------------------------------------------- + +# Invoke the lg-hub bin (dist/hub/cli.js), never dist/hub/serve.js directly: +# cli.js installs a process warning filter before the node:sqlite import graph +# is evaluated. Bypassing it reintroduces an ExperimentalWarning on every start. +# The host and port below are substituted by install-hub.sh, from +# LOOMGRAPH_HUB_IP and LOOMGRAPH_HUB_PORT in deploy/hub.env, when it renders this +# template into /etc/systemd/system/. Editing the installed unit by hand works, +# but the next install-hub.sh run overwrites it - change deploy/hub.env instead. +ExecStart=/usr/bin/lg-hub serve --host @MESH_IP@ --port @HUB_PORT@ --behind-tls-proxy --data-dir /var/lib/lghub --no-ui + +Restart=on-failure +RestartSec=5 + +NoNewPrivileges=yes +ProtectSystem=strict +ProtectHome=yes +PrivateTmp=yes +ReadWritePaths=/var/lib/lghub + +[Install] +WantedBy=multi-user.target diff --git a/deploy/netbird-acl.sh b/deploy/netbird-acl.sh new file mode 100755 index 0000000..7521272 --- /dev/null +++ b/deploy/netbird-acl.sh @@ -0,0 +1,1233 @@ +#!/usr/bin/env bash +# +# netbird-acl.sh - converge NetBird access control to the loomgraph hub model: +# members reach the hub on one TCP port and nothing else, the operator keeps +# mesh SSH, and the default All -> All policy is disabled (never deleted). +# +# Default mode is DRY-RUN: nothing is written without an explicit --apply. +# +# deploy/netbird-acl.sh dry-run, print every API call it would make +# deploy/netbird-acl.sh --apply converge (idempotent, re-runnable) +# deploy/netbird-acl.sh --verify read-only check of the converged state +# +# The management API token is read from the macOS Keychain (or an `nbtoken` +# helper) and is never printed, logged, or passed on a command line. +# +set -euo pipefail + +# ---------------------------------------------------------------- configuration + +# This script ships with NO deployment addresses baked in. Yours live in +# deploy/hub.env (gitignored); copy deploy/hub.env.example and fill it in. A +# script that falls back to someone else's mesh IP converges the wrong network, +# so the REQUIRED variables abort instead of defaulting. +SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)" +ENV_FILE="${LOOMGRAPH_HUB_ENV:-$SCRIPT_DIR/hub.env}" +if [ -f "$ENV_FILE" ]; then + # `set -a` exports every assignment so the values reach this script's own + # parameter expansions below and any child process that wants them. + set -a + # shellcheck source=/dev/null + . "$ENV_FILE" + set +a +fi + +# Named here rather than inlined at each use so the error says WHICH file to +# edit. `die` is defined further down; this runs after it, from main's preamble. +require_var() { + local name="$1" value="$2" what="$3" + if [ -z "$value" ]; then + die "$name is not set ($what). Set it in $ENV_FILE - copy deploy/hub.env.example to start." + fi +} + +API_BASE="${NETBIRD_API:-}" +KEYCHAIN_SERVICE="${NETBIRD_KEYCHAIN_SERVICE:-netbird-pat}" + +HUB_PEER_IP="${LOOMGRAPH_HUB_IP:-}" +HUB_PEER_NAME="${LOOMGRAPH_HUB_PEER_NAME:-the hub peer}" +SANDBOX_PEER_IP="${LOOMGRAPH_SANDBOX_IP:-}" +IFS=',' read -r -a MAC_PEER_IPS <<< "${LOOMGRAPH_MAC_IPS:-}" +HUB_PORT="${LOOMGRAPH_HUB_PORT:-8369}" +SSH_PORT="22" + +# The hub serves its web UI for ANY non-/v1 GET, so a status-code probe on a +# path like /healthz returns 200 with an HTML page even when the API is dead. +# Probe the real route and assert on the BODY. +HEALTH_PATH="${LOOMGRAPH_HEALTH_PATH:-/v1/health}" +HEALTH_EXPECT="${LOOMGRAPH_HEALTH_EXPECT:-}" +PROBE_CONNECT_TIMEOUT="${LOOMGRAPH_PROBE_CONNECT_TIMEOUT:-5}" +PROBE_TIMEOUT="${LOOMGRAPH_PROBE_TIMEOUT:-10}" +# Ports a colleague peer must NOT reach on the personal MacBooks. +BLOCKED_PROBE_PORTS=(22 80 443 "$HUB_PORT") + +GROUP_HUB="hub" +GROUP_MEMBERS="loomgraph-members" +GROUP_SANDBOX="sandbox" +GROUP_MACS="personal macbooks" +GROUP_CLIENTS="clients" +GROUP_ALL="All" + +POLICY_MEMBER_HUB="loomgraph-hub-access" +POLICY_OP_HUB="loomgraph-operator-hub-access" +POLICY_OP_HUB_SSH="loomgraph-operator-hub-ssh" +POLICY_OP_SANDBOX_SSH="loomgraph-operator-sandbox-ssh" +POLICY_DEFAULT="Default" +# Optional one-off cleanup. Empty means "no stale policy to delete" and step 5 +# becomes a no-op rather than matching a policy name that is not yours. +POLICY_DEAD="${NETBIRD_DEAD_POLICY:-}" + +# A route to the hub that does NOT depend on the mesh, printed in the lockout +# warning. Empty is a legitimate answer - and one worth seeing spelled out +# before the All -> All policy goes away. +PUBLIC_FALLBACK_SSH="${LOOMGRAPH_HUB_PUBLIC_SSH:-}" +SANDBOX_SSH="${LOOMGRAPH_SANDBOX_SSH:-}" +HUB_SSH="${LOOMGRAPH_HUB_SSH:-}" + +# ---------------------------------------------------------------- mode + output + +MODE="dry-run" +ASSUME_YES="no" +FROM_MEMBER="no" +DO_PROBE="yes" +FAIL_COUNT=0 +PASS_COUNT=0 + +usage() { + cat <<'USAGE' +Usage: netbird-acl.sh [--apply | --verify | --dry-run] [options] + + (no flag) DRY-RUN. Prints every API call that --apply would make. Default. + --apply Perform the changes. Requires the self-lockout guard to pass. + --verify Read-only. Checks the converged state against the acceptance + criteria, PASS/FAIL each. Exit 1 on any FAIL. + + --assume-yes Skip the interactive confirmation before disabling "Default". + Required when --apply runs without a TTY. + --from-member Run --verify from a loomgraph-members peer: adds the negative + reachability probes (the MacBooks must be UNREACHABLE from here). + Meaningless from the operator Mac - it would prove nothing. + --no-probe --verify checks the API state only; skip all network probes. + +Configuration: + Read from deploy/hub.env (override the path with LOOMGRAPH_HUB_ENV), then + from the environment. Copy deploy/hub.env.example and fill it in - nothing + deployment-specific is baked into this script. + + NETBIRD_API REQUIRED. Management API base URL, incl. /api + NETBIRD_TOKEN API token (else `nbtoken`, else macOS Keychain) + NETBIRD_KEYCHAIN_SERVICE Keychain service name, default netbird-pat + LOOMGRAPH_HUB_IP REQUIRED. Hub peer mesh IP + LOOMGRAPH_HUB_PEER_NAME display name for that peer in output + LOOMGRAPH_SANDBOX_IP second operator-SSH peer; empty skips it + LOOMGRAPH_MAC_IPS REQUIRED for --from-member. Comma-separated mesh + IPs that a member peer must NOT reach + LOOMGRAPH_HUB_PORT hub service port, default 8369 + LOOMGRAPH_HEALTH_PATH hub health route, default /v1/health + LOOMGRAPH_HEALTH_EXPECT regex the health body must match (overrides the + built-in healthy-body heuristic) + LOOMGRAPH_HUB_PUBLIC_SSH non-mesh fallback host:port for the lockout warning + NETBIRD_DEAD_POLICY exact name of a stale policy to delete; empty skips +USAGE +} + +log() { printf '%s\n' "$*"; } +info() { printf ' %s\n' "$*"; } +warn() { printf 'WARN %s\n' "$*" >&2; } +die() { printf 'ERROR %s\n' "$*" >&2; exit 1; } +step() { printf '\n== %s\n' "$*"; } +pass() { PASS_COUNT=$((PASS_COUNT + 1)); printf ' PASS %s\n' "$*"; } +fail() { FAIL_COUNT=$((FAIL_COUNT + 1)); printf ' FAIL %s\n' "$*"; } + +while [ "$#" -gt 0 ]; do + case "$1" in + --apply) MODE="apply" ;; + --verify) MODE="verify" ;; + --dry-run) MODE="dry-run" ;; + --assume-yes) ASSUME_YES="yes" ;; + --from-member) FROM_MEMBER="yes" ;; + --no-probe) DO_PROBE="no" ;; + -h|--help) usage; exit 0 ;; + *) usage >&2; die "unknown argument: $1" ;; + esac + shift +done + +# ---------------------------------------------------------------- prerequisites + +for required in curl python3; do + command -v "$required" >/dev/null 2>&1 || die "required command not found: $required" +done + +WORKDIR="$(mktemp -d "${TMPDIR:-/tmp}/netbird-acl.XXXXXX")" +chmod 700 "$WORKDIR" +cleanup() { + case "$WORKDIR" in + */netbird-acl.*) rm -r -f -- "$WORKDIR" ;; + esac +} +trap cleanup EXIT INT TERM + +GROUPS_JSON="$WORKDIR/groups.json" +POLICIES_JSON="$WORKDIR/policies.json" +PEERS_JSON="$WORKDIR/peers.json" +NBJSON="$WORKDIR/nbjson.py" + +# ---------------------------------------------------------------- json helper +# +# A single read-only python helper. It never touches the network and never sees +# the token; it only parses the JSON curl already fetched and builds request +# bodies with correct escaping. + +cat > "$NBJSON" <<'PY' +"""Read-only JSON helpers for netbird-acl.sh. No network, no secrets.""" +import json +import re +import sys + + +def load(path): + with open(path, encoding="utf-8") as handle: + return json.load(handle) + + +def group_by_name(groups, name): + for group in groups: + if group.get("name") == name: + return group + return None + + +def group_peer_ids(group): + ids = [] + for peer in group.get("peers") or []: + if isinstance(peer, dict): + ids.append(peer.get("id", "")) + else: + ids.append(str(peer)) + return [i for i in ids if i] + + +def rule_group_ids(rule, key): + ids = [] + for item in rule.get(key) or []: + if isinstance(item, dict): + ids.append(item.get("id", "")) + else: + ids.append(str(item)) + return [i for i in ids if i] + + +def policy_by_name(policies, name): + found = [p for p in policies if p.get("name") == name] + if len(found) > 1: + sys.stderr.write("WARN %d policies named %r; using the first\n" % (len(found), name)) + return found[0] if found else None + + +def cmd_group_id(argv): + group = group_by_name(load(argv[0]), argv[1]) + if not group: + return 1 + sys.stdout.write(group.get("id", "")) + return 0 + + +def cmd_group_peers(argv): + group = group_by_name(load(argv[0]), argv[1]) + if not group: + return 1 + for peer_id in group_peer_ids(group): + print(peer_id) + return 0 + + +def cmd_group_peer_count(argv): + group = group_by_name(load(argv[0]), argv[1]) + if not group: + return 1 + print(len(group_peer_ids(group))) + return 0 + + +def cmd_peer_id(argv): + for peer in load(argv[0]): + if peer.get("ip") == argv[1]: + sys.stdout.write(peer.get("id", "")) + return 0 + return 1 + + +def cmd_peer_name(argv): + for peer in load(argv[0]): + if peer.get("ip") == argv[1]: + sys.stdout.write(peer.get("name", "")) + return 0 + return 1 + + +def cmd_peer_ip(argv): + for peer in load(argv[0]): + if peer.get("id") == argv[1]: + sys.stdout.write(peer.get("ip", "")) + return 0 + return 1 + + +def cmd_groups_of_peer(argv): + for group in load(argv[0]): + if argv[1] in group_peer_ids(group): + print(group.get("name", "")) + return 0 + + +def cmd_policy_id(argv): + policy = policy_by_name(load(argv[0]), argv[1]) + if not policy: + return 1 + sys.stdout.write(policy.get("id", "")) + return 0 + + +def cmd_policy_enabled(argv): + policy = policy_by_name(load(argv[0]), argv[1]) + if not policy: + return 1 + return 0 if policy.get("enabled") else 2 + + +def cmd_policy_matches(argv): + """policy-matches + + Exit 0 only when the named policy is enabled AND carries an enabled accept + rule that is exactly src -> dst on that protocol/port set/direction. + """ + path, name, src, dst, proto, ports_csv, bidir = argv[:7] + policy = policy_by_name(load(path), name) + if not policy or not policy.get("enabled"): + return 1 + want_ports = sorted(p for p in ports_csv.split(",") if p) + want_bidir = bidir.lower() == "true" + for rule in policy.get("rules") or []: + if not rule.get("enabled"): + continue + if rule.get("action", "accept") != "accept": + continue + if rule_group_ids(rule, "sources") != [src]: + continue + if rule_group_ids(rule, "destinations") != [dst]: + continue + if rule.get("protocol") != proto: + continue + if sorted(str(p) for p in (rule.get("ports") or [])) != want_ports: + continue + if rule.get("port_ranges"): + continue + if bool(rule.get("bidirectional")) != want_bidir: + continue + return 0 + return 1 + + +def cmd_rules_referencing(argv): + """rules-referencing + + -> policy|rule|enabled|role|protocol|ports|direction, one line per rule. + """ + path, group_id = argv[:2] + for policy in load(path): + for rule in policy.get("rules") or []: + roles = [] + if group_id in rule_group_ids(rule, "sources"): + roles.append("source") + if group_id in rule_group_ids(rule, "destinations"): + roles.append("destination") + if not roles: + continue + enabled = bool(policy.get("enabled")) and bool(rule.get("enabled")) + print("|".join([ + policy.get("name", ""), + rule.get("name", ""), + "enabled" if enabled else "disabled", + "+".join(roles), + str(rule.get("protocol", "")), + ",".join(str(p) for p in (rule.get("ports") or [])), + "bidirectional" if rule.get("bidirectional") else "unidirectional", + ])) + return 0 + + +def cmd_dead_policy_safe(argv): + """Exit 0 only when every rule of the named policy has no sources and no destinations.""" + policy = policy_by_name(load(argv[0]), argv[1]) + if not policy: + return 1 + for rule in policy.get("rules") or []: + if rule_group_ids(rule, "sources") or rule_group_ids(rule, "destinations"): + return 2 + return 0 + + +def cmd_mk_group(argv): + print(json.dumps({"name": argv[0], "peers": [p for p in argv[1:] if p]}, sort_keys=True)) + return 0 + + +def cmd_mk_policy(argv): + """mk-policy """ + name, description, src, dst, proto, ports_csv, bidir = argv[:7] + rule = { + "name": name, + "description": description, + "enabled": True, + "action": "accept", + "bidirectional": bidir.lower() == "true", + "protocol": proto, + "sources": [src], + "destinations": [dst], + } + ports = [p for p in ports_csv.split(",") if p] + if ports: + rule["ports"] = ports + print(json.dumps({ + "name": name, + "description": description, + "enabled": True, + "sourcePostureChecks": [], + "rules": [rule], + }, sort_keys=True)) + return 0 + + +def cmd_mk_disable(argv): + """Build the PUT body that disables a policy, preserving every rule verbatim. + + GET returns sources/destinations as expanded group objects; PUT wants bare + group ids. Only the top-level `enabled` flag changes - same as the toggle in + the dashboard, so the policy can be re-enabled with one more PUT. + """ + policy = policy_by_name(load(argv[0]), argv[1]) + if not policy: + return 1 + rules = [] + for rule in policy.get("rules") or []: + new_rule = { + "id": rule.get("id"), + "name": rule.get("name", ""), + "description": rule.get("description", ""), + "enabled": bool(rule.get("enabled")), + "action": rule.get("action", "accept"), + "bidirectional": bool(rule.get("bidirectional")), + "protocol": rule.get("protocol", "all"), + "sources": rule_group_ids(rule, "sources"), + "destinations": rule_group_ids(rule, "destinations"), + } + if rule.get("ports"): + new_rule["ports"] = [str(p) for p in rule["ports"]] + if rule.get("port_ranges"): + new_rule["port_ranges"] = rule["port_ranges"] + rules.append({k: v for k, v in new_rule.items() if v is not None}) + print(json.dumps({ + "name": policy.get("name", ""), + "description": policy.get("description", ""), + "enabled": False, + "sourcePostureChecks": policy.get("source_posture_checks") or [], + "rules": rules, + }, sort_keys=True)) + return 0 + + +HEALTHY_TOKENS = {"ok", "up", "true", "pass", "passing", "healthy", "serving", "alive", "ready"} + + +def cmd_health_verdict(argv): + """health-verdict [expect-regex] -> VERDICT:detail on stdout. + + The hub serves its web UI for any non-/v1 GET, so an HTML body means the API + did not answer even when the status code was 200. Judge the body, never the + status code alone. + """ + with open(argv[0], "rb") as handle: + raw = handle.read() + text = raw.decode("utf-8", "replace").strip() + expect = argv[1] if len(argv) > 1 else "" + summary = " ".join(text.split())[:200] + + if not text: + print("EMPTY:no response body") + return 0 + lowered = text.lower() + if lowered.startswith(" [args...]\n") + sys.exit(64) + sys.exit(COMMANDS[sys.argv[1]](sys.argv[2:])) +PY + +nbj() { python3 "$NBJSON" "$@"; } + +# ---------------------------------------------------------------- api plumbing + +TOKEN="" +API_STATUS="" +API_BODY="" + +load_token() { + if [ -n "${NETBIRD_TOKEN:-}" ]; then + TOKEN="$NETBIRD_TOKEN" + elif command -v nbtoken >/dev/null 2>&1; then + TOKEN="$(nbtoken)" || die "the nbtoken helper failed; check it or set NETBIRD_TOKEN" + elif command -v security >/dev/null 2>&1; then + TOKEN="$(security find-generic-password -a "$USER" -s "$KEYCHAIN_SERVICE" -w 2>/dev/null)" || + die "no Keychain item for service \"$KEYCHAIN_SERVICE\" / account \"$USER\"; add it or set NETBIRD_TOKEN" + else + die "no token source: set NETBIRD_TOKEN, install nbtoken, or store the PAT in the Keychain" + fi + [ -n "$TOKEN" ] || die "empty NetBird token from the configured source" +} + +# api_call [body-file] -> response body on stdout, status in API_STATUS. +# +# The token reaches curl through a config file on stdin, so it never appears in +# the process table, in `ps`, or in any shell history. +api_call() { + local method="$1" path="$2" body_file="${3:-}" + local out status rc + out="$WORKDIR/response.json" + set +e + status="$( + { + printf 'url = "%s%s"\n' "$API_BASE" "$path" + printf 'request = "%s"\n' "$method" + printf 'header = "Authorization: Token %s"\n' "$TOKEN" + printf 'header = "Accept: application/json"\n' + printf 'silent\n' + printf 'show-error\n' + printf 'output = "%s"\n' "$out" + printf 'write-out = "%%{http_code}"\n' + if [ -n "$body_file" ]; then + printf 'header = "Content-Type: application/json"\n' + printf 'data-binary = "@%s"\n' "$body_file" + fi + } | curl --config - + )" + rc="$?" + set -e + [ "$rc" -eq 0 ] || die "curl failed (exit $rc) on $method $path" + API_STATUS="$status" + if [ -f "$out" ]; then + cat "$out" + rm -f -- "$out" + fi +} + +api_get() { + local path="$1" dest="$2" + api_call GET "$path" > "$dest" + case "$API_STATUS" in + 2*) ;; + *) die "GET $path returned HTTP $API_STATUS: $(head -c 400 "$dest")" ;; + esac +} + +# api_write +# +# Dry-run: prints the exact call and body, changes nothing. +# Apply: performs the call and leaves the response body in API_BODY. +api_write() { + local method="$1" path="$2" body_file="$3" description="$4" + API_BODY="" + if [ "$MODE" != "apply" ]; then + printf ' DRY-RUN %s %s%s\n' "$method" "$API_BASE" "$path" + printf ' %s\n' "$description" + if [ -n "$body_file" ]; then + printf ' body: %s\n' "$(nbj compact "$body_file")" + fi + return 0 + fi + API_BODY="$(api_call "$method" "$path" "$body_file")" + case "$API_STATUS" in + 2*) info "$method $path -> HTTP $API_STATUS ($description)" ;; + *) die "$method $path returned HTTP $API_STATUS: $(printf '%s' "$API_BODY" | head -c 400)" ;; + esac +} + +refresh_state() { + api_get "/groups" "$GROUPS_JSON" + api_get "/policies" "$POLICIES_JSON" + api_get "/peers" "$PEERS_JSON" +} + +# ---------------------------------------------------------------- lookups + +peer_id_for_ip() { + local ip="$1" id + if ! id="$(nbj peer-id "$PEERS_JSON" "$ip")"; then + die "no NetBird peer found with mesh IP $ip" + fi + printf '%s' "$id" +} + +group_id_or_empty() { + local name="$1" id + if id="$(nbj group-id "$GROUPS_JSON" "$name")"; then + printf '%s' "$id" + fi +} + +require_group_id() { + local name="$1" id + id="$(group_id_or_empty "$name")" + [ -n "$id" ] || die "expected group \"$name\" to exist but it does not" + printf '%s' "$id" +} + +slug() { printf '%s' "$1" | tr -c 'a-zA-Z0-9' '-'; } + +# write_body -> path of the file holding the generated JSON +write_body() { + local name="$1" + shift + local path="$WORKDIR/body-$name.json" + "$@" > "$path" + printf '%s' "$path" +} + +# ---------------------------------------------------------------- convergence + +# ensure_group [peer-id...] -> id in ENSURE_GROUP_ID +ENSURE_GROUP_ID="" +ensure_group() { + local name="$1" + shift + local existing body + existing="$(group_id_or_empty "$name")" + if [ -n "$existing" ]; then + info "group \"$name\" already exists ($existing) - no change" + ENSURE_GROUP_ID="$existing" + return 0 + fi + body="$(write_body "group-$(slug "$name")" nbj mk-group "$name" "$@")" + api_write POST "/groups" "$body" "create group \"$name\"" + if [ "$MODE" != "apply" ]; then + ENSURE_GROUP_ID="" + return 0 + fi + printf '%s' "$API_BODY" > "$WORKDIR/created-group.json" + ENSURE_GROUP_ID="$(nbj id-of "$WORKDIR/created-group.json")" + [ -n "$ENSURE_GROUP_ID" ] || die "group \"$name\" was created but the API returned no id" + info "group \"$name\" created ($ENSURE_GROUP_ID)" +} + +# ensure_policy +ensure_policy() { + local name="$1" description="$2" src="$3" dst="$4" proto="$5" ports="$6" bidir="$7" + local direction="unidirectional" + [ "$bidir" = "false" ] || direction="bidirectional" + if nbj policy-matches "$POLICIES_JSON" "$name" "$src" "$dst" "$proto" "$ports" "$bidir"; then + info "policy \"$name\" already matches and is enabled - no change" + return 0 + fi + if nbj policy-id "$POLICIES_JSON" "$name" > /dev/null; then + warn "policy \"$name\" exists but does not match the target rule." + die "refusing to rewrite \"$name\" automatically - inspect it in the dashboard, fix or delete it, then re-run" + fi + local body + body="$(write_body "policy-$(slug "$name")" \ + nbj mk-policy "$name" "$description" "$src" "$dst" "$proto" "$ports" "$bidir")" + api_write POST "/policies" "$body" "create policy \"$name\" ($proto/$ports, $direction)" +} + +# ---------------------------------------------------------------- lockout guard + +# Evaluated against the live policy list. Every path the operator needs after +# "Default" goes away must already exist and be enabled. +operator_path_ok() { + local macs="$1" hub="$2" sandbox="$3" ok=0 + if ! nbj policy-matches "$POLICIES_JSON" "$POLICY_OP_HUB" "$macs" "$hub" tcp "$HUB_PORT" false; then + warn "missing or disabled: \"$POLICY_OP_HUB\" ($GROUP_MACS -> $GROUP_HUB tcp/$HUB_PORT)" + ok=1 + fi + if ! nbj policy-matches "$POLICIES_JSON" "$POLICY_OP_HUB_SSH" "$macs" "$hub" tcp "$SSH_PORT" false; then + warn "missing or disabled: \"$POLICY_OP_HUB_SSH\" ($GROUP_MACS -> $GROUP_HUB tcp/$SSH_PORT)" + ok=1 + fi + if ! nbj policy-matches "$POLICIES_JSON" "$POLICY_OP_SANDBOX_SSH" "$macs" "$sandbox" tcp "$SSH_PORT" false; then + warn "missing or disabled: \"$POLICY_OP_SANDBOX_SSH\" ($GROUP_MACS -> $GROUP_SANDBOX tcp/$SSH_PORT)" + ok=1 + fi + return "$ok" +} + +lockout_warning() { + printf '\n' + printf ' ************************************************************\n' + printf ' * ABOUT TO DISABLE THE "Default" All -> All POLICY\n' + printf ' *\n' + printf ' * If the new policies are insufficient you WILL lose mesh\n' + printf ' * access to %s. Fallbacks, in order:\n' "$HUB_PEER_NAME" + printf ' *\n' + if [ -n "$PUBLIC_FALLBACK_SSH" ]; then + printf ' * 1. public SSH to %s, off the mesh\n' "$PUBLIC_FALLBACK_SSH" + else + printf ' * 1. NONE CONFIGURED - LOOMGRAPH_HUB_PUBLIC_SSH is empty, so\n' + printf ' * this script knows of no way back in without the mesh.\n' + fi + printf ' * 2. your provider console - know the login BEFORE continuing\n' + printf ' *\n' + printf ' * Run this from a shell that does NOT depend on the mesh.\n' + printf ' * "Default" is disabled, never deleted: re-enable it with a\n' + printf ' * single PUT if anything goes wrong.\n' + printf ' ************************************************************\n' + printf '\n' +} + +confirm_disable() { + if [ "$ASSUME_YES" = "yes" ]; then + info "confirmation skipped (--assume-yes)" + return 0 + fi + if [ ! -t 0 ]; then + die "refusing to disable \"$POLICY_DEFAULT\" without a TTY; re-run with --assume-yes if you are sure" + fi + local answer="" + printf 'Type DISABLE to continue, anything else aborts: ' + read -r answer + [ "$answer" = "DISABLE" ] || die "aborted by operator; nothing was disabled" +} + +# ---------------------------------------------------------------- converge flow + +# Refuse to proceed if a colleague peer also sits in "clients" (which carries +# the 0.0.0.0/0 exit node) or in "personal macbooks". Membership is fixed by +# hand, not silently rewritten here. +check_member_membership() { + local member_peer groups_of leaked=0 + if ! nbj group-id "$GROUPS_JSON" "$GROUP_MEMBERS" > /dev/null; then + return 0 + fi + while IFS= read -r member_peer; do + [ -n "$member_peer" ] || continue + groups_of="$(nbj groups-of-peer "$GROUPS_JSON" "$member_peer" | tr '\n' ' ')" + case " $groups_of " in + *" $GROUP_CLIENTS "*|*" $GROUP_MACS "*) + warn "peer $member_peer is in \"$GROUP_MEMBERS\" and also in: $groups_of" + leaked=1 + ;; + esac + done < <(nbj group-peers "$GROUPS_JSON" "$GROUP_MEMBERS") + [ "$leaked" -eq 0 ] || + die "a $GROUP_MEMBERS peer also belongs to $GROUP_CLIENTS or $GROUP_MACS; fix group membership first" +} + +converge() { + local hub_peer sandbox_peer + local hub_group members_group sandbox_group macs_group + + hub_peer="$(peer_id_for_ip "$HUB_PEER_IP")" + # The sandbox peer is OPTIONAL. Spec 00 listed only `hub` and + # `loomgraph-members`; the sandbox group exists because "operator retains mesh + # SSH to a second box" is unsatisfiable without a destination group holding + # it. A deployment with no such box sets LOOMGRAPH_SANDBOX_IP empty and gets + # neither the group nor its policy - not an empty group that matches nothing. + sandbox_peer="" + if [ -n "$SANDBOX_PEER_IP" ]; then + sandbox_peer="$(peer_id_for_ip "$SANDBOX_PEER_IP")" + fi + + step "Step 1 - groups" + ensure_group "$GROUP_HUB" "$hub_peer"; hub_group="$ENSURE_GROUP_ID" + ensure_group "$GROUP_MEMBERS"; members_group="$ENSURE_GROUP_ID" + sandbox_group="" + if [ -n "$sandbox_peer" ]; then + ensure_group "$GROUP_SANDBOX" "$sandbox_peer"; sandbox_group="$ENSURE_GROUP_ID" + else + info "LOOMGRAPH_SANDBOX_IP is empty - skipping the \"$GROUP_SANDBOX\" group" + fi + macs_group="$(require_group_id "$GROUP_MACS")" + check_member_membership + + step "Step 2 - member policy ($GROUP_MEMBERS -> $GROUP_HUB tcp/$HUB_PORT, unidirectional)" + ensure_policy "$POLICY_MEMBER_HUB" \ + "loomgraph colleagues reach the hub on tcp/$HUB_PORT and nothing else" \ + "$members_group" "$hub_group" tcp "$HUB_PORT" false + + step "Step 3 - operator policies (enumerated BEFORE Default is touched)" + ensure_policy "$POLICY_OP_HUB" \ + "operator Macs reach the loomgraph hub port" \ + "$macs_group" "$hub_group" tcp "$HUB_PORT" false + ensure_policy "$POLICY_OP_HUB_SSH" \ + "operator Macs administer the hub over the mesh" \ + "$macs_group" "$hub_group" tcp "$SSH_PORT" false + if [ -n "$sandbox_group" ]; then + ensure_policy "$POLICY_OP_SANDBOX_SSH" \ + "operator Macs keep mesh SSH to the sandbox peer" \ + "$macs_group" "$sandbox_group" tcp "$SSH_PORT" false + else + info "no sandbox peer configured - skipping \"$POLICY_OP_SANDBOX_SSH\"" + fi + + step "Step 4 - self-lockout guard, then DISABLE \"$POLICY_DEFAULT\"" + if [ "$MODE" = "apply" ]; then + # Re-read live state: the guard must judge what the API actually has, not + # what this run intended to create. + refresh_state + hub_group="$(require_group_id "$GROUP_HUB")" + sandbox_group="$(require_group_id "$GROUP_SANDBOX")" + macs_group="$(require_group_id "$GROUP_MACS")" + if ! operator_path_ok "$macs_group" "$hub_group" "$sandbox_group"; then + die "self-lockout guard FAILED: operator access policies are not in place. \"$POLICY_DEFAULT\" left ENABLED." + fi + info "self-lockout guard passed: operator keeps hub tcp/$HUB_PORT, hub tcp/$SSH_PORT, sandbox tcp/$SSH_PORT" + else + info "in --apply the guard re-reads live state here and refuses to disable" + info "\"$POLICY_DEFAULT\" unless the three operator policies above exist and are enabled." + fi + + local default_id="" + if default_id="$(nbj policy-id "$POLICIES_JSON" "$POLICY_DEFAULT")"; then + if nbj policy-enabled "$POLICIES_JSON" "$POLICY_DEFAULT"; then + lockout_warning + if [ "$MODE" = "apply" ]; then + confirm_disable + fi + local body + body="$(write_body "disable-default" nbj mk-disable "$POLICIES_JSON" "$POLICY_DEFAULT")" + api_write PUT "/policies/$default_id" "$body" \ + "DISABLE (never delete) policy \"$POLICY_DEFAULT\"" + info "rollback: PUT /policies/$default_id with the same body and enabled=true" + else + info "\"$POLICY_DEFAULT\" is already disabled - no change" + fi + else + warn "policy \"$POLICY_DEFAULT\" not found - it should exist, disabled, as a one-call rollback" + fi + + step "Step 5 - delete the stale auto-created policy, if one was named" + local dead_id="" + if [ -z "$POLICY_DEAD" ]; then + info "NETBIRD_DEAD_POLICY is empty - nothing to delete, skipping" + elif dead_id="$(nbj policy-id "$POLICIES_JSON" "$POLICY_DEAD")"; then + if nbj dead-policy-safe "$POLICIES_JSON" "$POLICY_DEAD"; then + api_write DELETE "/policies/$dead_id" "" "delete dead policy \"$POLICY_DEAD\"" + else + warn "\"$POLICY_DEAD\" has non-empty sources or destinations - NOT deleting it" + fi + else + info "dead policy \"$POLICY_DEAD\" not present - no change" + fi + info "the PEER behind that policy is left alone; delete it by hand if it is not in use" + + if [ "$MODE" = "apply" ]; then + printf '\nApply complete. Re-run with --verify to check the acceptance criteria.\n' + else + printf '\nDRY-RUN complete. Nothing was changed. Re-run with --apply to converge.\n' + fi +} + +# ---------------------------------------------------------------- verify flow + +# ---------------------------------------------------------------- probes +# +# Network probes, not NetBird API calls. They never carry the token. +# +# PROBE_RC curl exit code: 0 = an HTTP response came back, 7 = could not +# connect, 28 = timed out, 52/56 = TCP connected then died. +# PROBE_CODE HTTP status (0 when no response). +# PROBE_BODY first 2 KiB of the body. +PROBE_RC=0 +PROBE_CODE="" +PROBE_BODY_FILE="" + +http_probe() { + local host="$1" port="$2" path="$3" + PROBE_BODY_FILE="$WORKDIR/probe-body" + : > "$PROBE_BODY_FILE" + set +e + PROBE_CODE="$( + curl --silent \ + --connect-timeout "$PROBE_CONNECT_TIMEOUT" \ + --max-time "$PROBE_TIMEOUT" \ + --output "$PROBE_BODY_FILE" \ + --write-out '%{http_code}' \ + "http://$host:$port$path" 2>/dev/null + )" + PROBE_RC="$?" + set -e +} + +# A peer that should be blocked must give us no HTTP response at all. curl exit +# 7 (refused/unreachable) or 28 (timeout, the usual NetBird drop) is the pass +# signal; anything that completed a TCP handshake is a FAIL. +probe_is_blocked() { + case "$PROBE_RC" in + 7|28) return 0 ;; + *) return 1 ;; + esac +} + +probe_outcome_text() { + case "$PROBE_RC" in + 0) printf 'HTTP %s received' "$PROBE_CODE" ;; + 7) printf 'connection refused or filtered (curl 7)' ;; + 28) printf 'timed out (curl 28) - consistent with an ACL drop' ;; + 52) printf 'TCP CONNECTED then empty reply (curl 52)' ;; + 56) printf 'TCP CONNECTED then reset (curl 56)' ;; + 35) printf 'TCP CONNECTED, TLS handshake attempted (curl 35)' ;; + *) printf 'curl exit %s' "$PROBE_RC" ;; + esac +} + +# Positive probe: the hub API itself must answer on the real route, and the +# BODY must look healthy. A 200 with an HTML page is the web UI answering for a +# non-/v1 path - that is a dead API, not a pass. +verify_hub_health() { + local verdict kind detail + http_probe "$HUB_PEER_IP" "$HUB_PORT" "$HEALTH_PATH" + if [ "$PROBE_RC" -ne 0 ]; then + fail "hub $HUB_PEER_IP:$HUB_PORT$HEALTH_PATH did not answer: $(probe_outcome_text)" + return 0 + fi + verdict="$(nbj health-verdict "$PROBE_BODY_FILE" "$HEALTH_EXPECT")" + kind="${verdict%%:*}" + detail="${verdict#*:}" + case "$kind" in + HEALTHY) + pass "hub $HEALTH_PATH answered HTTP $PROBE_CODE and the body is healthy: $detail" + ;; + JSON_NO_STATUS) + # A JSON body proves the API answered, not the UI - but the schema is not + # one we recognise, so surface it instead of silently passing it as healthy. + pass "hub $HEALTH_PATH answered HTTP $PROBE_CODE with a non-HTML JSON body ($detail)" + info "check that body by eye, or pin it with LOOMGRAPH_HEALTH_EXPECT=" + ;; + HTML) + fail "hub $HEALTH_PATH returned HTTP $PROBE_CODE but the body is the WEB UI, not the API: $detail" + ;; + *) + fail "hub $HEALTH_PATH returned HTTP $PROBE_CODE with an unhealthy body [$kind]: $detail" + ;; + esac +} + +# Negative probes. Only meaningful when run FROM a loomgraph-members peer. +verify_member_isolation() { + local mac port blocked_all=0 + + http_probe "$HUB_PEER_IP" "$HUB_PORT" "$HEALTH_PATH" + if [ "$PROBE_RC" -eq 0 ]; then + pass "member peer reaches the hub on $HUB_PEER_IP:$HUB_PORT (HTTP $PROBE_CODE)" + else + fail "member peer cannot reach the hub on $HUB_PEER_IP:$HUB_PORT: $(probe_outcome_text)" + fi + + # Same host, a port the policy does not open: must be blocked. + http_probe "$HUB_PEER_IP" "$SSH_PORT" "/" + if probe_is_blocked; then + pass "member peer is blocked from $HUB_PEER_IP:$SSH_PORT ($(probe_outcome_text))" + else + fail "member peer REACHED $HUB_PEER_IP:$SSH_PORT - the policy is wider than tcp/$HUB_PORT ($(probe_outcome_text))" + fi + + for mac in "${MAC_PEER_IPS[@]}"; do + for port in "${BLOCKED_PROBE_PORTS[@]}"; do + http_probe "$mac" "$port" "/" + if probe_is_blocked; then + info "blocked as expected: $mac:$port ($(probe_outcome_text))" + else + fail "member peer REACHED MacBook $mac:$port - $(probe_outcome_text)" + blocked_all=1 + fi + done + done + if [ "$blocked_all" -eq 0 ]; then + pass "member peer cannot reach either MacBook on ports ${BLOCKED_PROBE_PORTS[*]}" + fi +} + +verify_members_reach_only_hub() { + local members_group="$1" extra=0 line policy_name enabled_flag + while IFS= read -r line; do + [ -n "$line" ] || continue + policy_name="${line%%|*}" + enabled_flag="$(printf '%s' "$line" | cut -d'|' -f3)" + if [ "$enabled_flag" != "enabled" ]; then + continue + fi + if [ "$policy_name" = "$POLICY_MEMBER_HUB" ]; then + continue + fi + fail "extra enabled rule touches $GROUP_MEMBERS: $line" + extra=1 + done < <(nbj rules-referencing "$POLICIES_JSON" "$members_group") + if [ "$extra" -eq 0 ]; then + pass "no other enabled rule references $GROUP_MEMBERS" + fi +} + +verify_members_group_hygiene() { + local stray=0 member_peer member_ip groups_of gname + while IFS= read -r member_peer; do + [ -n "$member_peer" ] || continue + member_ip="$(nbj peer-ip "$PEERS_JSON" "$member_peer" || printf 'unknown-ip')" + groups_of="$(nbj groups-of-peer "$GROUPS_JSON" "$member_peer")" + while IFS= read -r gname; do + [ -n "$gname" ] || continue + case "$gname" in + "$GROUP_ALL"|"$GROUP_MEMBERS") ;; + *) + fail "member peer $member_ip is also in group \"$gname\" (leak path around the ACL)" + stray=1 + ;; + esac + done <<< "$groups_of" + done < <(nbj group-peers "$GROUPS_JSON" "$GROUP_MEMBERS") + if [ "$stray" -eq 0 ]; then + pass "every $GROUP_MEMBERS peer sits only in $GROUP_ALL and $GROUP_MEMBERS" + fi +} + +verify() { + local hub_peer sandbox_peer + local hub_group members_group sandbox_group macs_group + + hub_peer="$(peer_id_for_ip "$HUB_PEER_IP")" + sandbox_peer="" + if [ -n "$SANDBOX_PEER_IP" ]; then + sandbox_peer="$(peer_id_for_ip "$SANDBOX_PEER_IP")" + fi + + step "Acceptance criteria" + + hub_group="$(group_id_or_empty "$GROUP_HUB")" + if [ -z "$hub_group" ]; then + fail "group \"$GROUP_HUB\" exists" + else + local hub_peers hub_count + hub_peers="$(nbj group-peers "$GROUPS_JSON" "$GROUP_HUB" | tr '\n' ' ')" + hub_count="$(nbj group-peer-count "$GROUPS_JSON" "$GROUP_HUB")" + if [ "$hub_count" = "1" ] && [ "$hub_peers" = "$hub_peer " ]; then + pass "group \"$GROUP_HUB\" holds exactly $HUB_PEER_NAME ($HUB_PEER_IP)" + else + fail "group \"$GROUP_HUB\" should hold only $HUB_PEER_NAME ($HUB_PEER_IP); holds $hub_count peer(s)" + fi + fi + + members_group="$(group_id_or_empty "$GROUP_MEMBERS")" + if [ -z "$members_group" ]; then + fail "group \"$GROUP_MEMBERS\" exists" + else + pass "group \"$GROUP_MEMBERS\" exists" + fi + + sandbox_group="" + if [ -z "$SANDBOX_PEER_IP" ]; then + info "LOOMGRAPH_SANDBOX_IP is empty - sandbox criteria not checked" + else + sandbox_group="$(group_id_or_empty "$GROUP_SANDBOX")" + if [ -z "$sandbox_group" ]; then + fail "group \"$GROUP_SANDBOX\" exists (required for operator mesh SSH to $SANDBOX_PEER_IP)" + else + local sandbox_peers + sandbox_peers="$(nbj group-peers "$GROUPS_JSON" "$GROUP_SANDBOX" | tr '\n' ' ')" + if [ "$sandbox_peers" = "$sandbox_peer " ]; then + pass "group \"$GROUP_SANDBOX\" holds exactly the sandbox peer ($SANDBOX_PEER_IP)" + else + fail "group \"$GROUP_SANDBOX\" should hold only the sandbox peer ($SANDBOX_PEER_IP)" + fi + fi + fi + + macs_group="$(group_id_or_empty "$GROUP_MACS")" + if [ -z "$macs_group" ]; then + fail "group \"$GROUP_MACS\" exists" + fi + + # A loomgraph-members peer reaches the hub IP on the hub port, and nothing else. + if [ -n "$members_group" ] && [ -n "$hub_group" ] && + nbj policy-matches "$POLICIES_JSON" "$POLICY_MEMBER_HUB" \ + "$members_group" "$hub_group" tcp "$HUB_PORT" false; then + pass "\"$POLICY_MEMBER_HUB\" enabled: $GROUP_MEMBERS -> $GROUP_HUB tcp/$HUB_PORT, unidirectional" + else + fail "\"$POLICY_MEMBER_HUB\" missing, disabled, or not exactly $GROUP_MEMBERS -> $GROUP_HUB tcp/$HUB_PORT unidirectional" + fi + + # A loomgraph-members peer reaches nothing else, including both MacBooks. + if [ -n "$members_group" ]; then + verify_members_reach_only_hub "$members_group" + verify_members_group_hygiene + fi + + # Operator retains mesh SSH to the hub (and the sandbox peer, when configured). + if [ -n "$macs_group" ] && [ -n "$hub_group" ] && [ -n "$sandbox_group" ] && + operator_path_ok "$macs_group" "$hub_group" "$sandbox_group"; then + pass "operator retains hub tcp/$HUB_PORT, hub tcp/$SSH_PORT and sandbox tcp/$SSH_PORT" + else + fail "operator access policies are incomplete (see warnings above)" + fi + + # Default is disabled, not deleted. + if nbj policy-id "$POLICIES_JSON" "$POLICY_DEFAULT" > /dev/null; then + if nbj policy-enabled "$POLICIES_JSON" "$POLICY_DEFAULT"; then + fail "\"$POLICY_DEFAULT\" is still ENABLED (All -> All, every port)" + else + pass "\"$POLICY_DEFAULT\" is present and disabled" + fi + else + fail "\"$POLICY_DEFAULT\" has been DELETED - it must remain, disabled, for one-call rollback" + fi + + # The stale auto-created policy is gone, when one was named. + if [ -z "$POLICY_DEAD" ]; then + info "NETBIRD_DEAD_POLICY is empty - stale-policy criterion not checked" + elif nbj policy-id "$POLICIES_JSON" "$POLICY_DEAD" > /dev/null; then + fail "stale policy \"$POLICY_DEAD\" still exists" + else + pass "stale policy \"$POLICY_DEAD\" is gone" + fi + + if [ "$DO_PROBE" = "yes" ]; then + step "Reachability probes" + if [ "$FROM_MEMBER" = "yes" ]; then + info "running from a $GROUP_MEMBERS peer: the MacBooks must be unreachable from here" + verify_member_isolation + else + verify_hub_health + info "the negative criteria (a member peer reaching nothing but the hub) cannot be" + info "proven from this machine. Re-run on a $GROUP_MEMBERS peer: netbird-acl.sh --verify --from-member" + fi + else + info "network probes skipped (--no-probe): the API state above is all that was checked" + fi + + printf '\n%s passed, %s failed\n' "$PASS_COUNT" "$FAIL_COUNT" + if [ "$FAIL_COUNT" -ne 0 ]; then + printf 'Acceptance: FAIL\n' + return 1 + fi + printf 'Acceptance (API side): PASS\n' + printf '\nStill to confirm by hand from the operator machine:\n' + printf ' netbird status --detail | grep -E "Peers count|Status:"\n' + if [ -n "$HUB_SSH" ]; then + printf ' ssh %s hostname # expect the hub host\n' "$HUB_SSH" + fi + if [ -n "$SANDBOX_SSH" ]; then + printf ' ssh %s hostname # expect the sandbox host\n' "$SANDBOX_SSH" + fi + printf ' curl -s http://%s:%s%s # read the BODY\n' "$HUB_PEER_IP" "$HUB_PORT" "$HEALTH_PATH" + printf 'And, from a %s peer once one has been enrolled:\n' "$GROUP_MEMBERS" + printf ' netbird-acl.sh --verify --from-member\n' +} + +# ---------------------------------------------------------------- main + +main() { + # Checked here, not at assignment, so --help still works without a hub.env and + # the message can name the file to edit. + require_var NETBIRD_API "$API_BASE" "NetBird management API base URL, including /api" + require_var LOOMGRAPH_HUB_IP "$HUB_PEER_IP" "mesh IP of the peer running lg-hub" + if [ "$FROM_MEMBER" = "yes" ] && [ -z "${MAC_PEER_IPS[0]:-}" ]; then + die "LOOMGRAPH_MAC_IPS is not set (the peers a member must NOT reach). --from-member proves nothing without them. Set it in $ENV_FILE." + fi + + case "$MODE" in + dry-run) log "MODE: DRY-RUN (no changes; pass --apply to converge)" ;; + apply) log "MODE: APPLY" ;; + verify) log "MODE: VERIFY (read-only)" ;; + esac + log "API: $API_BASE" + + load_token + refresh_state + + if [ "$MODE" = "verify" ]; then + verify + else + converge + fi +} + +main From 213c05eaf66ccc08de55eb3e401e3fa505faa512 Mon Sep 17 00:00:00 2001 From: Dat Date: Mon, 21 Sep 2026 09:06:26 +0700 Subject: [PATCH 39/45] docs(hub): add member onboarding and operator runbook Two audiences, two documents, both written against `deploy/hub.env` variables rather than any one deployment's addresses. `hub-onboarding.md` is what a colleague reads before `lg enroll`. It leads with the property that actually governs the decision - anything reaching the hub is readable by every member and can never be deleted, because the events table aborts UPDATE and DELETE by trigger - and then states plainly what each of the two channels publishes. Both are filtered now, so the page says so; what it refuses to imply is that filtering is a proof. The masking rules recognise shapes someone thought of, and an organisation's own token format is probably not one of them. The test it asks the reader to apply is "if the masker missed this, would I mind the whole team reading it, forever". `hub-operations.md` is the runbook. Its goal is that someone who did not build the hub can restore it from a backup and prove the restored database is intact using nothing but that page. The restore and chain-verify procedure was tested end to end, including against a live 226 KB WAL. Co-Authored-By: Claude Opus 5 (1M context) --- docs/hub-onboarding.md | 333 +++++++++++++++ docs/hub-operations.md | 915 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 1248 insertions(+) create mode 100644 docs/hub-onboarding.md create mode 100644 docs/hub-operations.md diff --git a/docs/hub-onboarding.md b/docs/hub-onboarding.md new file mode 100644 index 0000000..3218c3b --- /dev/null +++ b/docs/hub-onboarding.md @@ -0,0 +1,333 @@ +# Hub onboarding — read this before you enroll + +You are about to join the loomgraph team hub. Read all of it before you run +`lg enroll`. + +## The one thing you must understand first + +**Anything that reaches the hub is readable by every other member and can never +be deleted.** The events table aborts every UPDATE and DELETE by database +trigger, so a secret that lands there cannot be retracted by any supported +means. Rotating the secret is the remedy; erasing it is not available. + +Both things a sync pushes are filtered before they leave your machine — the run +state projection and the event lines alike. That filtering is an **allowlist of +shapes somebody thought of**, not a proof. It reliably removes what it +recognises: absolute paths, your username, your hostname, ANSI escapes, and +secrets matching known patterns, with everything capped at 200 characters. It +cannot recognise a credential format its rules do not describe. + +So the question to ask is not "is it masked" but "if the masker missed this, +would I mind the whole team reading it, forever". The rest of this page gives +you what you need to answer that per repository. + +## Claude Code still runs on your machine, under your credentials + +loomgraph does not run agents. `lg run` starts the CLI you already installed — +`claude`, `codex`, `opencode` — as a child process on your own laptop, signed in +as you. + +The hub never starts an agent. It has no code path that could. It stores what +members push and serves reads back out of that store. That absence is the design, +not a missing feature: a daemon that can only store and route is a store, not a +scheduler. + +Consequences worth being explicit about: + +- Nobody shares an account. You use your own Claude Code subscription or API key. +- Nobody's token or session is copied to the hub or to another member. +- If the hub is down, your runs are unaffected. A dead hub changes neither a run's + exit code nor any node's outcome. Only the sync fails. + +## Where the hub lives + +One VPS on the team's WireGuard mesh, on port `8369`. It is reachable only over +that mesh — there is no public route to the port. You will be given a peer that +can reach exactly this one address and port, and nothing else on the mesh. + +Your operator gives you the address; this page writes it as `$HUB_IP`. Set it +once and the commands below paste as written: + +```bash +export HUB_IP= +``` + +The hub is one SQLite database behind an HTTP API. That is the whole thing. + +## What a sync actually pushes + +Two things, filtered by the same rules but assembled differently. + +| | What it is | Filtered? | +| --- | --- | --- | +| **The projection** | A summary of run state, assembled field by field | **Yes** — hand-written allowlist of fields | +| **The event lines** | Your run's JSONL, rebuilt field by field against a per-kind allowlist | **Yes** — same strip/rewrite/mask/cap on every text field | + +Your local `.loomgraph/runs//events.jsonl` keeps its **raw** values — that +is your debugging record and it is never rewritten. The filtering happens at push +time, on the copy that crosses to the hub. + +### The projection + +Built field by field, so there is no field a variable value or a node's output +could ride in on: + +- `vars` appear as key names only, never values. +- Node output is absent — there is no field for it. +- Node errors are control-stripped, path-rewritten (`/Users/you/work/repo/run.sh` + becomes `${REPO_ROOT}/run.sh`, and your username and hostname likewise), have + known secret shapes cut to their first four characters, and are capped at 200 + characters. + +### The event lines + +Each line is rebuilt against a list naming, per event kind, exactly which `data` +fields may be published. A field not on the list is dropped. A line whose kind +the list does not know is dropped whole, rather than passed through. + +Fields carrying operator or environment text go through the same +strip → rewrite → mask → cap the projection applies: + +| Event | Field | Treatment | +| --- | --- | --- | +| `run_started` | `data.cwd` | Path-rewritten: your home directory and username are replaced | +| `node_finished` | `data.error` | Stripped, path-rewritten, masked, capped at 200 characters | +| `run_finished` | `data.error` | Same | +| `human_requested` | `data.question` | Interpolated first (see below), then sanitised | +| `human_resolved` | `data.answer` | Whatever the reviewer typed, sanitised | + +Everything else in an event — node ids, statuses, attempt counts, costs, budget +numbers, edge names — is published as-is. Those are engine- and graph-derived +identifiers, the same class of fact the projection already publishes. + +### Why "node output never reaches the hub" needs a footnote + +For a node that **succeeds**, it holds outright: no event carries the output +text, and the projection has no field for it. + +For a node that **fails**, the adapters fold output into the error string: + +- **claude**: when the CLI reports `is_error`, the error becomes + `claude run reported is_error (): `. +- **claude and codex**: on a non-zero exit, the process's full trimmed **stderr** + is appended to the error. +- **command / verifier nodes**: an unmet `expect` puts the expected string into + the error. + +That error is sanitised and capped at 200 characters before it is pushed, in both +channels. But "sanitised" means the masker's rules ran — not that the first 200 +characters of your agent's failure output are safe to share by construction. +Assume a teammate will read them. + +### Variable values have one door + +A `human` node's question is a template resolved exactly like an agent prompt. A +question written as `Approve deploy with token {{vars.api_key}}?` is interpolated +*before* the `human_requested` event is written, so the value is in the text that +gets sanitised. If the value matches a known secret shape it is masked. If it does +not — an internal token format, a customer identifier, a connection string the +rules do not describe — it publishes. Do not interpolate a secret into a question. + +### Masking is an allowlist, not a proof + +The rules recognise the shapes they describe and nothing else. During development +a canary shaped `AKIA` plus 18 more characters passed through unmasked, because +the rule matches exactly 20 characters (`\bAKIA[0-9A-Z]{16}\b`). The canary was +malformed rather than the rule being wrong — which is the point: the rules are a +filter for shapes someone thought of, and your organisation's own token format is +probably not one of them. + +### Every member reads everything + +There is no per-member filtering on reads. Any token with the `read` scope lists +every member's runs and fetches any member's events by naming that member in the +URL. "Team-readable" means the whole team, not your own runs. + +### Nothing is encrypted, and nothing can be deleted + +`hub.db` is a plaintext SQLite file on the host's disk. No encryption at rest, no +per-item keys, and no masking on egress — whatever reached the hub is served back +exactly as stored. + +The `events` table carries triggers that abort every UPDATE and DELETE. A secret +that lands there cannot be removed without dismantling the store's integrity +guarantee by hand. Treat a leak into the hub as permanent, and rotate the secret +rather than hoping to erase it. + +### What this means in practice + +Before you enable sync on a repository, ask: + +1. If the masker missed something in my build's stderr, would every teammate + reading it be a problem? That is the actual test — not whether masking exists. +2. Do my scripts, tests, or agent prompts print a token, connection string, or + customer identifier on failure? Fix that first — it is worth fixing regardless + of the hub. +3. Are my variable *names* sensitive on their own? Both channels publish names. +4. Do any of my human-review questions interpolate a variable that holds a secret? + Rewrite the question. + +If the answer to any of these is uncomfortable, do not enable sync for that +repository. The decision is yours, and it is per repository. + +## Your token is your identity + +`lg-hub member add ` prints a token like `lgt_1a2b3c4d.`. +The hub stores only a SHA-256 hash of the secret half, so: + +- It is printed **once**. It cannot be reprinted, recovered, or looked up. Not by + you, not by the operator, not from the database. +- Possession equals identity. Anyone holding that string *is* you to the hub — + they can push runs under your name and read everything every member pushed. + There is no second factor and no device binding. + +So: + +- **Store it in your OS keychain** (macOS Keychain, `secret-tool` on Linux). Not in + a dotfile you back up, not in a note app, not in a shared drive. +- **Never put it in chat, email, a ticket, or a screenshot.** It must be delivered + to you over a channel where the value can be destroyed after use, and you should + destroy it there once you have enrolled. +- **If you suspect anyone else has seen it, say so immediately.** The operator runs + `lg-hub member revoke ` and issues a new one. Revocation takes effect on + the next request — there is no session cache to wait out. + +### Keep the token out of your shell history + +`lg enroll ` puts the token in the command line, so it lands in +`~/.zsh_history` or `~/.bash_history`, in your shell's process list while it runs, +and in any terminal recording. Prefer the environment variables, which `lg` reads +before it looks at the config file — set both or neither: + +```bash +export LOOMGRAPH_HUB_URL=http://$HUB_IP:8369 +export LOOMGRAPH_HUB_TOKEN=$(security find-generic-password -s loomgraph-hub -w) # macOS +``` + +If you do run `lg enroll` with the token as an argument, prefix the command with a +space if your shell is configured to skip such lines, and scrub the entry +afterwards. + +`lg enroll` writes `~/.config/loomgraph/hub.json` with mode 0600. That file is a +credential — treat it like `~/.ssh/id_ed25519`: not in a dotfiles repository, not +synced to a cloud drive, not copied to a second machine (ask for a second token +instead, so the two can be revoked independently). + +## Enrolling + +Steps 1 to 4 are the operator's; you do 5 and 6. + +1. The operator creates a NetBird setup key scoped to the `loomgraph-members` + group, with a short expiry and single use. +2. You install the NetBird client and join with that key. +3. The operator verifies from their own machine that your peer can reach + `$HUB_IP:8369` and nothing else on the mesh. +4. The operator runs `lg-hub member add ` and delivers the token out-of-band. +5. You configure the identity — either the environment variables above, or: + + ```bash + lg enroll http://$HUB_IP:8369 lgt_1a2b3c4d. + ``` + + This writes `~/.config/loomgraph/hub.json` (mode 0600) holding the url and + token. It is your machine identity, written once for the whole machine — not + per repository. + +6. You opt a repository in, deliberately, one at a time: + + ```bash + cd ~/work/some-repo + lg sync --enable + ``` + + This writes `.loomgraph/hub.json` containing exactly `{"sync":true}`. It never + contains a token. Two files share the name `hub.json` and never share a job: the + one in your home directory is *who you are*, the one in the repository is *this + repository consents to be synced*. + +### The opt-in only gates the automatic push — read this carefully + +`lg sync --enable` controls one thing: whether `lg run` pushes events live while a +run is in progress. In a repository that has not opted in, `lg run` pushes nothing. + +**It does not gate a manual sync.** `lg sync ` and `lg sync --all` never +check the opt-in file. If you are enrolled, running `lg sync --all` inside *any* +repository pushes that repository's local runs to the hub, opted in or not. + +So the real rule is: **once you are enrolled, do not run `lg sync` in a repository +you have not decided to publish.** The flag protects you from pushing by accident +during a run; it does not protect you from pushing on purpose in the wrong +directory. + +## Daily use + +```bash +lg run examples/hello.yaml # runs locally; live-pushes only if this repo is opted in +lg sync # push one run explicitly — no opt-in check +lg sync --all # push every run under .loomgraph/runs/ — no opt-in check +``` + +When a repository is opted in, `lg run` pushes events to the hub as the run +proceeds, in batches. If the hub is unreachable you get one line on stderr — +`hub sync unavailable: N batches not pushed (run )` — and the run continues +and finishes normally. Failed batches are not retried automatically. Your events +are already durable in `.loomgraph/runs//events.jsonl`, so once the hub is +back, `lg sync ` pushes what was missed. Re-pushing is safe: the hub +deduplicates by run and sequence number. + +`lg sync` exit codes: `0` everything synced, `1` a usage error or the hub is not +configured, `2` at least one run failed to sync. + +## There is a web UI, and it is on by default + +The hub serves a browser UI on the same origin as the API — +`http://$HUB_IP:8369/`. It has no login of its own: you paste your member +token into it, and it keeps that token in the browser's `localStorage` and sends it +as a bearer header. (Some project documentation still says a UI is a later phase. +That text is out of date; the UI ships.) + +What that means for you: + +- Pasting your token into the UI stores it, in clear, in the browser profile of + whatever machine you used. Any other page you open from that origin, and anyone + with access to that browser profile, can read it. The origin is plain `http://`, + protected by the WireGuard mesh rather than TLS. +- Prefer a browser profile you control, and use the UI's logout, which clears the + stored token. +- Do not paste your token into a shared or kiosk machine's browser. If you do, + treat the token as exposed and ask for it to be revoked. + +## loomgraph does not share sessions + +To save you asking: there is no way to hand someone your live session. + +- No transcript upload. Full transcripts are never published, to the hub or + anywhere else — a transcript is a credential dump. +- No session transplant. Nothing writes into another person's home directory and no + adapter resumes someone else's session id. +- No cross-CLI replay. + +What you can hand over is understanding, not state: `lg-handoff pack` distills a +session into a brief — goal, files, claims, open questions, the exact commit — +quoting turns verbatim with no model summarising anything, and publishes it behind +a private, expiring link. + +If you actually need two people in one live session, the answer is **tmux over mesh +SSH**: one person hosts the session on their machine, the other attaches over the +mesh. loomgraph is not involved and will not be. + +## If something goes wrong + +| Symptom | What it means | What to do | +| --- | --- | --- | +| `hub not configured - run: lg enroll ` | No `~/.config/loomgraph/hub.json` and no `LOOMGRAPH_HUB_URL`/`LOOMGRAPH_HUB_TOKEN` set | Re-run `lg enroll`, or export both variables | +| `nothing to sync - pass a run id, or --all` | You ran `lg sync` with no argument | Pass a run id or `--all` | +| Sync fails with 401 | Token revoked, mistyped, or truncated | Ask the operator whether it was revoked; if not, request a new one | +| Sync fails with 403 | Your token lacks the scope for that call | Ask the operator to re-issue with the right scopes | +| Sync fails with a connection error | Mesh down, or the hub is down | `netbird status`; then tell the operator | +| `lg run` printed `hub sync unavailable` | Hub was unreachable during the run | The run is fine. Re-run `lg sync ` later | +| Nothing syncs during a run and there is no error | The repository is not opted in | `lg sync --enable` — after re-reading "What a sync actually pushes" | +| **You pushed a secret** | It is in the event table, unmasked and undeletable | Tell the operator now, and **rotate the secret**. Do not wait for it to be removed — it cannot be | + +Operator-side procedures — backup, restore, revocation, upgrades — are in +[hub-operations.md](./hub-operations.md). diff --git a/docs/hub-operations.md b/docs/hub-operations.md new file mode 100644 index 0000000..bdcca83 --- /dev/null +++ b/docs/hub-operations.md @@ -0,0 +1,915 @@ +# Hub operations runbook + +Operator-facing. Everything here is run on the hub host as an administrator. +Member-facing material is in [hub-onboarding.md](./hub-onboarding.md). + +The goal of this document is that someone who did not build the hub can restore +it from a backup and prove the restored database is intact, using nothing but +this page. + +## The facts you need + +This runbook is written against **your** deployment's addresses, which live in +`deploy/hub.env` (copy `deploy/hub.env.example`). Nothing here hardcodes them. +Export them once per shell and every command below pastes as written: + +```bash +set -a; . /path/to/loomgraph/deploy/hub.env; set +a +HUB_IP="$LOOMGRAPH_HUB_IP" +``` + +| Thing | Value | +| --- | --- | +| Host | the mesh peer running `lg-hub` (developed against Ubuntu 25.04) | +| Public address | `$HUB_PUBLIC_IP` — the non-mesh fallback route, if you have one | +| Mesh address | `$HUB_IP` (WireGuard, interface `wt0` under NetBird) | +| Hub listener | `$HUB_IP:8369` — mesh only, never `0.0.0.0` | +| Service user | `lghub` | +| Data directory | `/var/lib/lghub` (mode 0750, owned `lghub:lghub`) | +| Database | `/var/lib/lghub/hub.db` plus `hub.db-wal` and `hub.db-shm` | +| systemd unit | `lg-hub.service` | +| Binary | `/usr/bin/lg-hub` (which is `dist/hub/cli.js`) | +| Health endpoint | `GET /v1/health` | + +The hub never runs an agent. Agents run on each member's own machine under that +member's own credentials. A hub outage is a sync outage and nothing else. + +Commands below are written with `sudo`; run them as root or via `sudo` as shown. + +### The data directory is not optional on the command line + +`lg-hub` resolves its data directory in this order: `--data-dir`, then the +`LOOMGRAPH_HUB_DIR` environment variable, then `~/.local/share/loomgraph-hub`. +There is no "current" database it can find on its own. + +**Every `lg-hub` command you type by hand must name the data directory +explicitly.** If you forget, the command silently creates a second, empty +database in the invoking user's home directory and succeeds against it — +`member ls` prints nothing, `export` prints nothing, and neither is an error. +That is the single most common way to waste an hour here. + +Throughout this document, commands are written as: + +```bash +sudo -u lghub lg-hub --data-dir /var/lib/lghub +``` + +**Run every command that opens the database as `lghub`, never as root.** Opening a +WAL database creates `hub.db-wal` and `hub.db-shm` beside it, owned by whoever +opened it. A root-owned sidecar left behind by a careless `sqlite3` or `lg-hub` +invocation stops the service from writing, and the failure surfaces later as +unexplained ingest errors. + +Confirm which directory the running service actually uses before trusting +anything else on this page: + +```bash +systemctl cat lg-hub.service | grep -E 'ExecStart|Environment' +``` + +## Service + +### Start, stop, restart, status + +```bash +sudo systemctl start lg-hub +sudo systemctl stop lg-hub +sudo systemctl restart lg-hub +systemctl status lg-hub +sudo systemctl enable lg-hub # start at boot +``` + +### Logs + +The service logs to stdout and stderr, so everything lands in the journal. + +```bash +journalctl -u lg-hub -n 100 --no-pager # last 100 lines +journalctl -u lg-hub -f # follow +journalctl -u lg-hub --since "1 hour ago" +journalctl -u lg-hub -p err --no-pager # errors only +``` + +### What a healthy start looks like + +``` +lg-hub serving on http://$HUB_IP:8369 +web UI: http://$HUB_IP:8369/ (paste a token to connect) +``` + +`systemctl status lg-hub` shows `active (running)`, and: + +```bash +ss -tln | grep 8369 +# LISTEN 0 511 $HUB_IP:8369 0.0.0.0:* +``` + +The listener must be on `$HUB_IP`. Anything on `0.0.0.0:8369` means the +unit's `--host` is wrong and the hub is answering on the public interface — stop +the service and fix the unit before doing anything else. + +Reachability, from the host itself or any mesh peer: + +```bash +curl -s http://$HUB_IP:8369/v1/health +# {"ok":true,"version":"0.1.0"} +``` + +Use `/v1/health`, not `/healthz`. `/healthz` does not exist as an API route, and +because the web UI is served for any non-`/v1` GET, requesting it returns HTTP +200 with an HTML page. A `/healthz` check that only looks at the status code +passes even when the API is broken. + +### What a bind failure looks like + +**Refused bind (no transport flag).** `lg-hub serve` refuses any non-loopback +host unless `--behind-tls-proxy` is passed. If that flag is missing from the +unit: + +``` +refusing to bind $HUB_IP: a bearer token over plaintext non-loopback HTTP +would expose the hub's credentials. Pass --behind-tls-proxy if a trusted TLS +proxy terminates the connection in front of this address. +``` + +The process exits 1 and systemd reports `status=1/FAILURE`. Fix: restore the flag +in the unit. There is no TLS proxy in this deployment and none is expected — the +flag is passed because the bind is on a WireGuard mesh address unreachable from +the public internet, and it changes exactly one thing, the startup bind check. It +affects no request-time behaviour: the server derives no identity, address or +scheme from request headers. If NetBird is ever removed, or this service is ever +rebound to a routable address, that reasoning is void and the flag must be +removed. + +**Port already in use.** + +``` +lg-hub fatal: listen EADDRINUSE: address already in use $HUB_IP:8369 +``` + +Exit code 2. Find the holder with `sudo ss -tlnp | grep 8369` — usually a previous +instance that systemd did not reap, or a hand-started `lg-hub serve`. + +**Invalid port.** `invalid --port: `, exit 1. + +**`wt0` not up yet.** This does *not* produce a bind failure. The host has +`net.ipv4.ip_nonlocal_bind=1`, so the socket binds to `$HUB_IP` even +before NetBird assigns it. The service starts clean and looks healthy; it is +simply unreachable until the mesh is up. See "wt0 down at boot" below. + +## Membership + +### Add a member + +Do this only after the member's NetBird peer exists, is in the +`loomgraph-members` group, and has been verified to reach `$HUB_IP:8369` +and nothing else. Network access first, hub token second — never the reverse. + +```bash +sudo -u lghub lg-hub member add alice --data-dir /var/lib/lghub +# lgt_1a2b3c4d. +# The token above is shown once and cannot be recovered. Store it somewhere safe. +``` + +Default scopes are `ingest,read`. Override with `--scopes`: + +```bash +sudo -u lghub lg-hub member add alice --scopes ingest,read --data-dir /var/lib/lghub +``` + +The scopes that exist and what each permits: + +| Scope | Grants | +| --- | --- | +| `ingest` | `POST /v1/events` — push runs | +| `read` | `GET /v1/feed`, `GET /v1/runs`, `GET /v1/runs//` — read every member's runs | +| `admin` | `GET/POST /v1/members`, `POST /v1/members//revoke` — list members, **mint new member tokens over HTTP**, revoke | + +Grant `admin` to a person, not to a workstation, and only when you mean it: an +`admin` token can mint further tokens over the API without touching this host. + +The printed token goes to stdout; the warning goes to stderr. Deliver the token +out-of-band, over a channel where the value can be destroyed afterwards — never +chat, email, a ticket, or a shared document. The store keeps only a SHA-256 hash +of the secret, so a lost token cannot be recovered, only revoked and reissued. + +Tell the recipient two things when you hand it over: + +- `lg enroll ` puts the token in argv, so it lands in their shell + history and process list. The alternative is the `LOOMGRAPH_HUB_URL` and + `LOOMGRAPH_HUB_TOKEN` environment variables, which `lg` reads in preference to + the config file — both must be set or neither is used. +- Pasting the token into the web UI stores it in that browser's `localStorage`, in + clear, on a plain `http://` origin. + +### List members + +```bash +sudo -u lghub lg-hub member ls --data-dir /var/lib/lghub +# 1a2b3c4d alice +# 5e6f7a8b bob revoked +``` + +Columns are tab-separated: key id, member name, and the literal `revoked` when the +token has been revoked. Revoked rows are never removed — the roster is a history, +not a current-state list. + +### Revoke a member + +```bash +sudo -u lghub lg-hub member revoke 1a2b3c4d --data-dir /var/lib/lghub +# revoked 1a2b3c4d +``` + +Exit 1 with `no active member with key id ` if the key id is unknown or +already revoked. Revocation takes effect on the next request — every request +re-resolves the token against the members table, and there is no session cache or +token TTL to wait out. A revoked token is refused with 401 immediately, and its +syncs stop. + +Revocation is the immediate response to any suspected token exposure. It costs +nothing: issue a new token and the member re-runs `lg enroll`. + +### Offboarding checklist + +A person leaving needs **both** halves removed. Revoking the hub token leaves them +on the mesh; removing the NetBird peer leaves a valid token that works again the +moment they get back on the mesh by any other route. Do both, in this order, and +record the date. + +1. **Revoke the hub token.** + `sudo -u lghub lg-hub member revoke --data-dir /var/lib/lghub` +2. **Confirm the revocation landed.** + `sudo -u lghub lg-hub member ls --data-dir /var/lib/lghub` — the row must show + `revoked`. +3. **Revoke every other token that person holds.** Check `member ls` for more than + one row with their name — a second machine means a second key id. +4. **Delete their NetBird peer** in the NetBird console (or via the API), and + remove it from the `loomgraph-members` group. Deleting the peer is the + authoritative step; group removal alone leaves a peer that a later policy + change could re-admit. +5. **Expire or delete any unused setup key** that was issued for them. +6. **Verify they are gone from the mesh:** the peer no longer appears in + `netbird status --detail` output from the operator machine, and a probe of + `$HUB_IP:8369` from their machine fails at the network layer. +7. **Leave their data alone.** Events they pushed stay in the store. The events + table is append-only, enforced by database triggers that abort any UPDATE or + DELETE. There is no supported way to remove a member's history in phase 1, and + attempting it breaks the hash chain. Say this out loud during offboarding so + nobody is surprised later. + +## Audit and export + +Export is lossless: the stored line is the line the client sent, byte for byte, +never re-encoded. This was verified end to end against a live hub. + +```bash +# every stored line to stdout, one per line, for grepping +sudo -u lghub lg-hub export --jsonl --data-dir /var/lib/lghub > /tmp/all-events.jsonl + +# one file per run: /runs///events.jsonl +sudo -u lghub lg-hub export --out /tmp/hub-export --data-dir /var/lib/lghub +``` + +Exactly one of `--jsonl` and `--out` is required; passing both or neither exits 1 +with `export requires exactly one of --jsonl or --out`. + +Which to use: `--jsonl` is a flat stream with nothing added, so member and run +identity are not representable in it — use it to grep for content. `--out` carries +identity structurally in the path, which is why it needs no envelope — use it when +you care about who ran what. + +Examples: + +```bash +# every failed node across every member +sudo -u lghub lg-hub export --jsonl --data-dir /var/lib/lghub | grep '"kind":"node_failed"' + +# what one member ran +sudo -u lghub lg-hub export --out /tmp/hub-export --data-dir /var/lib/lghub +ls /tmp/hub-export/runs/alice/ +``` + +### What is actually in those lines + +Treat an export as sensitive material, not as a log file. The event lines are +stored exactly as the client sent them — the masking, path rewriting and +200-character cap that members may have been told about apply **only** to the +separate run-state projection, never to the event stream. Concretely, the lines +contain: + +- `run_started.data.cwd` — the member's absolute repository path, un-rewritten, + including their username and home directory layout. +- `node_finished.data.error` and `run_finished.data.error` — the raw error: no + masking, no path rewriting, no length cap. When a `claude` node fails with + `is_error`, this is the agent's entire result text; on a non-zero exit, both the + `claude` and `codex` adapters append the process's full stderr. +- `human_requested.data.question` — the question *after* template interpolation, so + any `{{vars.*}}` or `{{nodes.*.output}}` values are substituted into it. +- `human_resolved.data.answer` — whatever the reviewer typed. + +Write exports somewhere only you can read, and delete them when you are done: + +```bash +rm -rf /tmp/hub-export /tmp/all-events.jsonl +``` + +### Every read-scoped member already sees all of this + +There is no per-member filtering on any read route. A token with the `read` scope +lists every member's runs and fetches any member's events by naming that member in +the URL. The export gives you nothing the team cannot already read; its value is +that it is greppable and offline, not that it is privileged. + +### When a member reports a leaked secret + +The events table carries triggers that abort every UPDATE and DELETE. **There is +no supported way to remove a line from the store.** Deleting one would break the +hash chain even if you dismantled the triggers by hand, which would destroy the +one mechanism that tells you later whether the store was altered. + +So the response is rotation, not redaction: + +1. **Rotate the leaked credential itself**, immediately. This is the only step that + actually reduces exposure. +2. Find the blast radius so the rotation is complete: + `sudo -u lghub lg-hub export --jsonl --data-dir /var/lib/lghub | grep -c ''` + — and note which members and runs it appears in with `--out`. +3. Tell the team the value is in the store, is readable by every member, and is + permanent. They need to know it was not quietly cleaned up. +4. Record it. The hub's value is that its history is intact and known; a leak that + is documented is survivable, one that is silently assumed-deleted is not. +5. Fix the source — the script, test, or prompt that printed the secret into a + failing node's output. + +Do not offer to "remove it from the database". You cannot, and saying you will +leaves the team with a false belief about where their secret is. + +## The hash chain + +Every event row carries `prev_hash` and `row_hash`, where +`row_hash = sha256(prev_hash ‖ json)` and `json` is the client's line verbatim. +The chain starts at 32 zero bytes and its current end is stored in the +`chain_head` table (single row, `id = 1`). Rows are chained in insertion order, +which is `rowid` order. + +This is what makes a partial or torn restore detectable. Verifying it is a step in +both the backup and the restore procedure, and it is not optional. + +**There is no `lg-hub` subcommand that verifies the chain.** Use the script below. +Install it once on the host: + +```bash +sudo tee /usr/local/sbin/lg-hub-verify-chain.mjs >/dev/null <<'NODE_EOF' +import { DatabaseSync } from "node:sqlite"; +import { createHash } from "node:crypto"; + +const dbPath = process.argv[2]; +if (dbPath === undefined) { + console.error("usage: node lg-hub-verify-chain.mjs "); + process.exit(2); +} + +const GENESIS = Buffer.alloc(32); +const db = new DatabaseSync(dbPath); + +let prev = GENESIS; +let count = 0; +let failed = false; + +for (const row of db + .prepare("SELECT rowid AS rid, prev_hash, row_hash, json FROM events ORDER BY rowid") + .iterate()) { + const storedPrev = row.prev_hash === null ? GENESIS : Buffer.from(row.prev_hash); + if (!storedPrev.equals(prev)) { + console.error(`BROKEN LINK at rowid ${row.rid}`); + console.error(` expected prev_hash ${prev.toString("hex")}`); + console.error(` stored prev_hash ${storedPrev.toString("hex")}`); + failed = true; + break; + } + const expected = createHash("sha256").update(prev).update(row.json, "utf8").digest(); + const stored = Buffer.from(row.row_hash); + if (!expected.equals(stored)) { + console.error(`CONTENT MISMATCH at rowid ${row.rid}`); + console.error(` expected row_hash ${expected.toString("hex")}`); + console.error(` stored row_hash ${stored.toString("hex")}`); + failed = true; + break; + } + prev = expected; + count += 1; +} + +if (!failed) { + const head = db.prepare("SELECT head FROM chain_head WHERE id=1").get(); + if (head === undefined) { + console.error("NO CHAIN HEAD: chain_head has no row with id=1"); + failed = true; + } else { + const storedHead = Buffer.from(head.head); + if (!storedHead.equals(prev)) { + console.error("HEAD MISMATCH: last row_hash is not the stored chain head"); + console.error(` computed ${prev.toString("hex")}`); + console.error(` stored ${storedHead.toString("hex")}`); + failed = true; + } + } +} + +db.close(); + +if (failed) { + console.error(`chain verification FAILED after ${count} verified events`); + process.exit(1); +} +console.log(`chain OK: ${count} events, head ${prev.toString("hex")}`); +NODE_EOF +sudo chmod 0755 /usr/local/sbin/lg-hub-verify-chain.mjs +``` + +Run it against a **snapshot**, never the live database — the live head moves while +you read, which produces a spurious HEAD MISMATCH: + +```bash +sudo -u lghub node --no-warnings /usr/local/sbin/lg-hub-verify-chain.mjs \ + /var/backups/lghub/hub-20260921T030000Z.db +# chain OK: 41207 events, head 9f2c... +``` + +Exit codes: `0` chain verifies, `1` chain does not verify, `2` usage error. + +There are three distinct failures, and they mean different things: + +| Output | Meaning | +| --- | --- | +| `BROKEN LINK at rowid N` | Rows are missing from the middle, or were reordered. The file is not a faithful copy. | +| `CONTENT MISMATCH at rowid N` | A stored line or its hash was altered. Corruption or tampering. | +| `HEAD MISMATCH` | Every row links correctly, but the tail is short of the recorded head — rows are missing from the end. Typical of a snapshot taken mid-write, or a truncated copy. | + +## Backup + +### The WAL constraint — read before writing any backup script + +`hub.db` runs `PRAGMA journal_mode=WAL`, so committed data can live in +`hub.db-wal` rather than in `hub.db` itself. **`cp hub.db backup.db` produces a +file that is missing recent commits and may be internally inconsistent.** Copying +all three files with `cp` is no better: they are copied at different instants and +the set will not agree. + +Take backups with `VACUUM INTO`, which writes a fully-checkpointed, consistent, +single-file copy from a live database without stopping the service. + +### Taking a backup + +```bash +STAMP=$(date -u +%Y%m%dT%H%M%SZ) +sudo install -d -o lghub -g lghub -m 0750 /var/backups/lghub +sudo -u lghub sqlite3 /var/lib/lghub/hub.db \ + "VACUUM INTO '/var/backups/lghub/hub-${STAMP}.db'" +``` + +Install the CLI if it is missing: `sudo apt-get install -y sqlite3`. If you cannot +install it, node does the same thing — `node:sqlite` is built in: + +```bash +sudo -u lghub node --no-warnings -e \ + 'const {DatabaseSync}=require("node:sqlite"); + const db=new DatabaseSync(process.argv[1]); + db.exec("VACUUM INTO \x27"+process.argv[2]+"\x27"); + db.close();' \ + /var/lib/lghub/hub.db "/var/backups/lghub/hub-${STAMP}.db" +``` + +(The destination path is interpolated into SQL, so keep backup paths free of +quote characters. The `sqlite3` form above is the preferred one.) + +### Verifying a backup — a backup you have not verified is not a backup + +Three checks, in this order. All three must pass. + +```bash +B=/var/backups/lghub/hub-${STAMP}.db + +# 1. the file opens and its pages are structurally sound +sudo -u lghub sqlite3 "$B" "PRAGMA integrity_check;" # must print: ok + +# 2. it is the schema version this hub expects +sudo -u lghub sqlite3 "$B" "PRAGMA user_version;" # must print: 1 + +# 3. the hash chain is continuous end to end +sudo -u lghub node --no-warnings /usr/local/sbin/lg-hub-verify-chain.mjs "$B" +``` + +A backup that fails any of the three is deleted, and the backup is retaken. If it +fails twice in a row, treat it as an incident: stop and run the same three checks +against a fresh snapshot of the live database. + +Prove the verification actually works, once, when you set this up: copy a good +backup, truncate the copy (`truncate -s -4096 /tmp/broken.db`), and confirm the +chain script exits non-zero on it. A verification step nobody has seen fail is a +verification step nobody should trust. + +### Schedule and retention + +- **Daily**, off-peak, via a systemd timer or cron running as `lghub`. +- **Destination** `/var/backups/lghub/`, mode 0750, owned `lghub:lghub`. These + files are plaintext copies of the entire store — same sensitivity as `hub.db` + itself. +- **Retention**: keep 14 daily copies, plus one monthly copy for 12 months. Prune + the rest. With one hub and a 48 GB disk this is cheap; see "Disk growth". +- **Off-host copy**: at least one verified copy must live somewhere other than + this VPS, or a lost VPS is a lost history. The copy is unencrypted — encrypt it + in transit and at rest wherever it lands. + +```bash +# prune everything but the 14 newest +ls -1t /var/backups/lghub/hub-*.db | tail -n +15 | xargs -r sudo rm -- +``` + +## Restore + +This is the procedure the runbook exists for. Read it through before starting. + +You need: a backup file that passes all three verification checks, root on the +host, and a maintenance window — syncs fail while the service is down, which does +not affect anyone's runs. + +### 1. Pick and verify the backup *before* touching the live database + +```bash +B=/var/backups/lghub/hub-20260921T030000Z.db +sudo -u lghub sqlite3 "$B" "PRAGMA integrity_check;" # ok +sudo -u lghub sqlite3 "$B" "PRAGMA user_version;" # 1 +sudo -u lghub node --no-warnings /usr/local/sbin/lg-hub-verify-chain.mjs "$B" +``` + +If any check fails, stop and pick an older backup. Never restore an unverified +file over a live one. + +### 2. Stop the service + +```bash +sudo systemctl stop lg-hub +systemctl is-active lg-hub # inactive +sudo ss -tlnp | grep 8369 # nothing +``` + +The process must be gone, not just idle. A live writer during a restore produces +exactly the silent corruption this procedure exists to avoid. + +### 3. Move the current database aside — do not delete it + +```bash +STAMP=$(date -u +%Y%m%dT%H%M%SZ) +sudo install -d -o lghub -g lghub -m 0750 /var/lib/lghub/preserved +sudo mv /var/lib/lghub/hub.db /var/lib/lghub/preserved/hub.db.${STAMP} +sudo mv /var/lib/lghub/hub.db-wal /var/lib/lghub/preserved/hub.db-wal.${STAMP} 2>/dev/null || true +sudo mv /var/lib/lghub/hub.db-shm /var/lib/lghub/preserved/hub.db-shm.${STAMP} 2>/dev/null || true +``` + +The `-wal` and `-shm` sidecars **must** be moved out too. A stale WAL left beside +a restored database is applied on the next open and will corrupt it. + +### 4. Put the backup in place + +```bash +sudo cp "$B" /var/lib/lghub/hub.db +sudo chown lghub:lghub /var/lib/lghub/hub.db +sudo chmod 0600 /var/lib/lghub/hub.db +ls -l /var/lib/lghub/ +``` + +There must be no `hub.db-wal` or `hub.db-shm` in the directory at this point. A +`VACUUM INTO` output is a single self-contained file. + +### 5. Verify the restored file in place, before starting the service + +```bash +sudo -u lghub sqlite3 /var/lib/lghub/hub.db "PRAGMA integrity_check;" # ok +sudo -u lghub sqlite3 /var/lib/lghub/hub.db "PRAGMA user_version;" # 1 +sudo -u lghub node --no-warnings /usr/local/sbin/lg-hub-verify-chain.mjs \ + /var/lib/lghub/hub.db +# chain OK: events, head +``` + +Record the event count and head hex in your incident notes. If the chain does not +verify here, go to "When the chain does not verify" below and do not start the +service. + +### 6. Start the service and check it + +```bash +sudo systemctl start lg-hub +systemctl status lg-hub +journalctl -u lg-hub -n 20 --no-pager # expect: lg-hub serving on http://$HUB_IP:8369 +ss -tln | grep 8369 # expect: $HUB_IP:8369 +curl -s http://$HUB_IP:8369/v1/health +``` + +### 7. Verify the data came back + +```bash +sudo -u lghub lg-hub member ls --data-dir /var/lib/lghub +sudo -u lghub lg-hub export --jsonl --data-dir /var/lib/lghub | wc -l +``` + +The roster must list the members you expect, and the line count must match the +event count the chain script reported in step 5. If the roster is empty, you are +almost certainly looking at a different data directory — re-read "The data +directory is not optional on the command line". + +### 8. Re-verify the chain once the service has taken traffic + +```bash +STAMP=$(date -u +%Y%m%dT%H%M%SZ) +sudo -u lghub sqlite3 /var/lib/lghub/hub.db \ + "VACUUM INTO '/var/backups/lghub/post-restore-${STAMP}.db'" +sudo -u lghub node --no-warnings /usr/local/sbin/lg-hub-verify-chain.mjs \ + /var/backups/lghub/post-restore-${STAMP}.db +``` + +### 9. Tell the members + +Anything pushed after the backup was taken is gone from the hub but still exists +on each member's machine. Ask every member to run `lg sync --all` in each opted-in +repository. Re-pushing is safe and idempotent: the hub deduplicates by +`(member, streamId, runId, seq)` and stores a given line once. Their local +`.loomgraph/sync/.cursor` may claim a higher acked sequence than the hub +now holds; deleting that file makes `lg sync` resend the run from the start, which +is the correct recovery. + +Member tokens survive a restore — they are rows in the restored database. If you +restored to a database from *before* a member was added, that member's token is no +longer known to the hub and they will get 401. Add them again, which mints a new +token, and deliver it out-of-band. + +### 10. Clean up + +Once the hub has been healthy for a day, remove `/var/lib/lghub/preserved/`. Not +before — the preserved copy is your only path back if the restore turns out to be +the wrong choice. + +### When the chain does not verify + +Do not start the service, and do not attempt to repair the file. The events table +carries triggers that abort every UPDATE and DELETE, so in-place repair is not +possible by design, and there is no repair tool. + +Work through these in order. + +1. **`HEAD MISMATCH` only, every row links.** The copy is internally consistent + but its tail is short of the recorded head — rows are missing from the end. The + data present is trustworthy and the loss is bounded to the end of the stream. + Prefer an older backup that verifies cleanly. If this is the only copy you + have, it is usable: the missing tail is recoverable because members hold the + authoritative event logs locally, and step 9's `lg sync --all` re-pushes it. + Record explicitly in your notes that you accepted a head mismatch, what the + computed and stored heads were, and when. +2. **`BROKEN LINK` or `CONTENT MISMATCH`.** The file is corrupt or has been + altered. Discard it. Move to the next-older backup and start again at step 1. + Do not put a file in this state into service — the chain is the only mechanism + that would tell you later that something was wrong. +3. **No backup verifies.** Check the live database you preserved in step 3: it may + still be intact and the restore may not have been necessary. + `sudo -u lghub sqlite3 /var/lib/lghub/preserved/hub.db. "PRAGMA integrity_check;"` + then run the chain script on it. If it verifies, put it back using steps 4 to 8. +4. **Nothing verifies anywhere — start clean.** Accept that the hub's history is + gone and rebuild it from the members, who hold every event locally: + + ```bash + sudo systemctl stop lg-hub + sudo mv /var/lib/lghub/hub.db \ + /var/lib/lghub/preserved/hub.db.corrupt.$(date -u +%Y%m%dT%H%M%SZ) + sudo rm -f /var/lib/lghub/hub.db-wal /var/lib/lghub/hub.db-shm + sudo -u lghub lg-hub init --data-dir /var/lib/lghub + sudo systemctl start lg-hub + ``` + + A fresh database has **no members**. Every token ever issued is now invalid. You + must re-add every member (`lg-hub member add`) and redeliver every token + out-of-band, and each member must re-run `lg enroll` and then `lg sync --all`. + Keep the corrupt file — it is evidence, and losing a chain is worth + understanding. + +Whatever the outcome, write down what you did. The chain's value is that a break +is visible; that value is only realised if the break and the response are on +record. + +## Disk growth + +Events are append-only. Nothing prunes them, and **phase 1 ships no pruning, +retention, or archival command at all**. The database only grows. + +Measure it: + +```bash +du -sh /var/lib/lghub /var/backups/lghub +ls -l /var/lib/lghub/hub.db +df -h /var/lib + +# how many events, and how they arrive over time +sudo -u lghub sqlite3 /var/lib/lghub/hub.db "SELECT count(*) FROM events;" +sudo -u lghub sqlite3 /var/lib/lghub/hub.db \ + "SELECT substr(received_at,1,10) AS day, count(*) FROM events + GROUP BY day ORDER BY day DESC LIMIT 14;" + +# largest contributors +sudo -u lghub sqlite3 /var/lib/lghub/hub.db \ + "SELECT member, count(*) FROM events GROUP BY member ORDER BY 2 DESC;" +``` + +What to watch: + +- **`df -h /var/lib` below 20% free** — act. The disk is 48 GB and event lines are + small (hundreds of bytes), so a small team will not approach this for a long + time; a runaway graph emitting events in a loop will. +- **Backups grow with the database.** Fourteen daily copies means roughly fifteen + times the live size on disk. Count `/var/backups/lghub` when you size headroom. +- **A day an order of magnitude above the trailing average** — look at the + per-member counts and ask that member what they ran. + +If you need space back, the only supported lever in phase 1 is backup retention: +prune older backups, or move them off-host. Deleting events is not supported and +breaks the chain. + +## Upgrades + +`PRAGMA user_version = 1` is the migration anchor. `HubStore` runs the full schema +only when it opens a database whose `user_version` is `0`; a database already at +`1` is opened as-is and never re-schema'd. A future release that changes the +schema will bump this number, and that bump is how you tell whether an upgrade +touched the database. + +The procedure, in order. Do not reorder it. + +1. **Back up and verify.** Take a `VACUUM INTO` snapshot and run all three checks + from "Verifying a backup". Record the current `user_version` and the chain + head. Do not proceed on an unverified backup. + + ```bash + sudo -u lghub sqlite3 /var/lib/lghub/hub.db "PRAGMA user_version;" + ``` + +2. **Read the release notes** for the version you are moving to. If it changes the + schema, it says so and gives the new `user_version`. +3. **Stop the service.** `sudo systemctl stop lg-hub` +4. **Upgrade the binary to a pinned version**, never `@latest`: + + ```bash + sudo npm install -g loomgraph@ + lg-hub --version + ``` + +5. **Start the service and verify it.** `sudo systemctl start lg-hub`, then + `systemctl status lg-hub`, then the journal lines and `ss -tln` from "What a + healthy start looks like", then `curl -s http://$HUB_IP:8369/v1/health` + and check the version in the response. +6. **Verify the chain.** Snapshot again with `VACUUM INTO` and run the chain + script. The event count must be at least what it was before, and the chain must + verify. + + ```bash + sudo -u lghub sqlite3 /var/lib/lghub/hub.db "PRAGMA user_version;" + ``` + + If `user_version` changed and the release notes did not say it would, stop: + restore the pre-upgrade backup and downgrade the binary to the pinned version + you came from. +7. **Smoke test with a real member.** Have someone run `lg sync ` and + confirm it returns `synced (acked seq N)`. + +Rolling back is the restore procedure above plus `npm install -g` of the previous +pinned version. Downgrade the binary before restoring the database, so the older +binary never opens a newer schema. + +## Failure modes + +### Hub down + +**What breaks:** nothing on the member side except syncing. Verified end to end +against a live hub: killing the hub mid-run changed neither the run's exit code +nor any node's outcome. Members' agents run on their own machines and never +consult the hub to decide anything. + +**What a member sees:** one stderr line per run — +`hub sync unavailable: N batches not pushed (run )` — and the run finishes +normally. A manual `lg sync` fails with exit 2 and a per-run message. + +**Do syncs retry?** No. There is no automatic retry, no queue, and no backoff. A +failed batch is dropped rather than re-buffered, because the events are already +durable in the member's `.loomgraph/runs//events.jsonl`. Catch-up is +manual: once the hub is back, members run `lg sync ` or `lg sync --all`, +which re-sends from their local cursor. Re-sending is free — ingest is idempotent +by `(member, streamId, runId, seq)`. + +**Operator response:** + +```bash +systemctl status lg-hub +journalctl -u lg-hub -n 100 --no-pager +df -h /var/lib # rule out a full disk first +sudo ss -tlnp | grep 8369 # rule out a stale process +sudo systemctl restart lg-hub +curl -s http://$HUB_IP:8369/v1/health +``` + +Then tell members to run `lg sync --all` in each opted-in repository. + +### `wt0` down at boot + +`net.ipv4.ip_nonlocal_bind=1` is set on this host, so `lg-hub` binds +`$HUB_IP:8369` successfully even when NetBird has not yet assigned the +address. The service starts clean and the journal shows a normal startup. It is +simply unreachable until the mesh comes up. The unit orders itself after +`netbird.service` and `network-online.target`, which usually makes this invisible, +but ordering is not a guarantee that NetBird has finished negotiating. + +**Symptom:** service `active (running)`, `ss -tln` shows the listener, but members +cannot reach it and `curl` from another mesh peer times out. + +**Diagnose, on the hub host:** + +```bash +systemctl status netbird +ip addr show wt0 # must show $HUB_IP +sudo netbird status --detail +curl -s http://$HUB_IP:8369/v1/health # from the host itself +``` + +If `curl` on the host succeeds but mesh peers cannot reach it, the hub is fine and +the problem is NetBird or the ACL policy — check that `loomgraph-hub-access` +(`loomgraph-members -> hub`, tcp/8369) is enabled and that the member's peer is in +`loomgraph-members`. + +**Fix:** `sudo systemctl restart netbird`, wait for `wt0` to carry the address, +then confirm. Restarting `lg-hub` is not required — the socket is already bound — +but it is harmless. + +**Do not** work around this by binding a different address. A bind on a routable +address puts a bearer token on the public internet in plaintext. + +### Disk full + +SQLite cannot write when the filesystem is full. Ingest fails with +`SQLITE_FULL: database or disk is full`; the ingest transaction rolls back, so +nothing partial is committed and the hash chain is not damaged. Members see their +syncs fail; their local logs are unaffected. Reads may still work. + +**Diagnose:** + +```bash +df -h /var/lib /var/backups +du -sh /var/lib/lghub /var/backups/lghub +journalctl -u lg-hub -p err -n 50 --no-pager +``` + +**Recover, in this order:** + +1. **Free space from backups first** — they are reproducible, the live database is + not. Prune old copies or move them off-host: + `ls -1t /var/backups/lghub/hub-*.db | tail -n +8 | xargs -r sudo rm --` +2. **Clear other consumers** — `sudo journalctl --vacuum-size=200M`, and remove any + export directories left in `/tmp`. +3. **Do not delete events** to reclaim space. The table is append-only and the + triggers will refuse; forcing it would break the chain. +4. **Restart the service** once there is headroom, and confirm a member can sync. +5. **Take a fresh backup and verify the chain** — a full disk during a write is + exactly the condition worth proving you came through cleanly. +6. **Fix the cause.** If growth is organic, grow the volume. If one member's graph + emitted events in a loop, fix the graph. + +## Known gaps in phase 1 + +State these plainly when someone asks; none of them are bugs, all of them are +scope. + +- **No chain-verification subcommand.** `lg-hub` has no `verify`. The script in + this document is the procedure, and any backup script must implement the same + check. +- **No backup or restore subcommand.** Backups are `VACUUM INTO`; restore is the + manual procedure above. +- **No pruning, retention, or archival.** The store only grows. +- **No filtering of the event lines.** Only the run-state projection is masked, + path-rewritten and capped. The event stream is stored verbatim, and it carries + raw errors (including agent output and stderr from failed nodes), un-rewritten + absolute paths, and interpolated variable values inside human-review questions. + See "What is actually in those lines". +- **No egress masking.** Whatever reaches the hub is served back as stored. +- **No per-member read filtering.** Any `read` token sees every member's runs and + events. +- **No encryption at rest.** `hub.db` and every backup are plaintext SQLite. +- **No deletion of anything.** The events table is append-only, enforced by + triggers. A leaked secret is permanent; rotate it. +- **The per-repo opt-in does not gate manual syncs.** `.loomgraph/hub.json` + (`lg sync --enable`) is checked only by the live batcher during `lg run`. + `lg sync ` and `lg sync --all` push regardless. Say this when you explain + the opt-in to a member, or they will over-trust it. +- **A web UI ships and is on by default.** `lg-hub serve` serves a self-contained + HTML page for any non-`/v1` GET, on the same origin as the API; `--no-ui` + disables it and serves the JSON API only. The README's roadmap still lists a UI + as phase 4 — the code is ahead of that text. The page has no login of its own: a + member pastes a token, which the browser keeps in `localStorage` in clear on a + plain `http://` origin. It grants no API access a token holder did not already + have, but it does create a second place tokens come to rest. Consider `--no-ui` + if nobody uses it. From f1696d04ffdc79432bd9622bdc7b92120074c476 Mon Sep 17 00:00:00 2001 From: Dat Date: Mon, 21 Sep 2026 09:50:23 +0700 Subject: [PATCH 40/45] fix(deploy): quote hub.env values that contain spaces `hub.env` is sourced by bash, so `LOOMGRAPH_HUB_PEER_NAME=the hub peer` makes the shell try to run `hub` as a command, and the template shipped with exactly that as its default. The failure surfaces as `hub.env: line N: : command not found` before any script logic runs, which points at the wrong place entirely. Quoted the default, noted the rule at the top of the file, and flagged it on NETBIRD_DEAD_POLICY, whose real value is a NetBird-generated sentence. Co-Authored-By: Claude Opus 5 (1M context) --- deploy/hub.env.example | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/deploy/hub.env.example b/deploy/hub.env.example index 0019141..d69643e 100644 --- a/deploy/hub.env.example +++ b/deploy/hub.env.example @@ -2,6 +2,9 @@ # # cp deploy/hub.env.example deploy/hub.env # then fill it in # +# This file is SOURCED by bash (`set -a; . hub.env`). Quote any value containing +# a space - `NAME=two words` makes the shell try to run `words` as a command. +# # `deploy/hub.env` is gitignored on purpose. None of these are secrets in the # credential sense - they are addresses - but a public repo does not need a map # of your mesh, and the scripts refuse to run on someone else's defaults. @@ -29,7 +32,7 @@ NETBIRD_KEYCHAIN_SERVICE=netbird-pat LOOMGRAPH_HUB_IP= # Display name for that peer in log output. Cosmetic only. -LOOMGRAPH_HUB_PEER_NAME=the hub peer +LOOMGRAPH_HUB_PEER_NAME="the hub peer" # Mesh IP of a peer the operator keeps SSH to besides the hub. Leave empty if # you have none: the sandbox group and its policy are then skipped entirely. @@ -66,5 +69,5 @@ LOOMGRAPH_HUB_SSH= # Optional. Exact name of a stale policy to delete during --apply. Empty skips # that step. NetBird names auto-created policies -# "Temporary access policy for peer ". +# "Temporary access policy for peer ", so this value needs quoting. NETBIRD_DEAD_POLICY= From 7f085e1127fc908fe66c67d7b1bd0a788a57187b Mon Sep 17 00:00:00 2001 From: Dat Date: Mon, 21 Sep 2026 10:12:58 +0700 Subject: [PATCH 41/45] chore: ignore npm pack output `npm pack` is a documented step in the hub install path (loomgraph is not published, so install-hub.sh consumes a local tarball). The artifact it drops in the repo root should never be a candidate for commit. Co-Authored-By: Claude Opus 5 (1M context) --- .gitignore | 3 +++ 1 file changed, 3 insertions(+) diff --git a/.gitignore b/.gitignore index 2608014..c3d916e 100644 --- a/.gitignore +++ b/.gitignore @@ -15,3 +15,6 @@ SHARE-URL.txt # deploy/hub.env holds one deployment's real addresses: mesh IPs, the control # plane URL, SSH fallbacks. Copy deploy/hub.env.example and fill it in locally. deploy/hub.env + +# npm pack output: the install artifact deploy/install-hub.sh consumes. +*.tgz From 4ae6cb91323330cc75dfe3bf17db2380e7b1d70e Mon Sep 17 00:00:00 2001 From: Dat Date: Mon, 21 Sep 2026 10:21:54 +0700 Subject: [PATCH 42/45] docs(deploy): say which regex flavour LOOMGRAPH_HEALTH_EXPECT uses The health-body check runs through Python's `re.search`, so a POSIX class like `[[:space:]]` is not a character class - it is a nested set. It fails to match and only emits a FutureWarning, which reads as "the hub is unhealthy" when the body was fine all along. Documented the flavour, and included the real payload and a working pattern so the next person copies one that works. Co-Authored-By: Claude Opus 5 (1M context) --- deploy/hub.env.example | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/deploy/hub.env.example b/deploy/hub.env.example index d69643e..f653505 100644 --- a/deploy/hub.env.example +++ b/deploy/hub.env.example @@ -51,6 +51,12 @@ LOOMGRAPH_HEALTH_PATH=/v1/health # payload once - the built-in check is a heuristic, and a status code is not a # health check here (the hub serves its UI for any non-/v1 GET, so /healthz # returns 200 HTML even when the API is dead). +# +# PYTHON `re` SYNTAX, not POSIX ERE - the check runs through re.search. A POSIX +# class like [[:space:]] is not a character class there, it is a nested set, and +# it fails to match while only emitting a FutureWarning. Use \s, \d, \w. +# A real payload looks like {"ok":true,"version":"0.1.0"}, so: +# LOOMGRAPH_HEALTH_EXPECT='"ok"\s*:\s*true' LOOMGRAPH_HEALTH_EXPECT= # --- Lockout fallbacks ----------------------------------------------------- From 5b70969e08fd978e177ffaec263c5899691cc704 Mon Sep 17 00:00:00 2001 From: Dat Date: Mon, 21 Sep 2026 10:44:14 +0700 Subject: [PATCH 43/45] feat(deploy): add enroll-member.sh for the NetBird half of onboarding A member needs two independent grants and loomgraph has no NetBird coupling at all - nothing in `src/` references the mesh - so the network half was a manual console step with no record of how it is meant to be done. This script is that record. - `--new ` mints a one-off setup key whose `auto_groups` puts the new peer straight into `loomgraph-members`, so there is no window where a peer is on the mesh but ungrouped, and no second step to forget. Keys are single-use and expire in 24h by default: a setup key with no end date is a credential sitting in somebody's chat history. - `--peer ` adds an existing peer by name, hostname, mesh IP or id. The group PUT replaces the peer list, so the current members are read back and the new id appended - sending just the new one would silently evict everybody already in the group. - `--list` shows who is in the group, with connection state. Dry-run by default, like `netbird-acl.sh`, and the API token reaches curl via a `--config` file on stdin rather than argv. On success it prints the colleague's own instructions, including the two things that reliably go wrong: `netbird up` silently drops its flags when the client is already connected (so `netbird down` comes first, and there is no `netbird set`), and sync is opt-in per repository and does nothing until `lg sync --enable` is run in that repo. It points them at docs/hub-onboarding.md before that step rather than after. Co-Authored-By: Claude Opus 5 (1M context) --- deploy/enroll-member.sh | 289 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 289 insertions(+) create mode 100755 deploy/enroll-member.sh diff --git a/deploy/enroll-member.sh b/deploy/enroll-member.sh new file mode 100755 index 0000000..87949d3 --- /dev/null +++ b/deploy/enroll-member.sh @@ -0,0 +1,289 @@ +#!/usr/bin/env bash +# +# enroll-member.sh - the NetBird half of adding a colleague to the hub. +# +# A member needs TWO independent grants, and this script does only the first: +# +# 1. NETWORK - their peer is in the loomgraph-members group, so packets to +# the hub port are allowed by the ACL. THIS SCRIPT. +# 2. IDENTITY - a hub token, so the hub answers them. +# `lg-hub member add `, run on the hub host. +# +# Either alone is useless: a token without mesh access cannot reach the port, +# and mesh access without a token gets a 401. Revoking someone means undoing +# both - `lg-hub member revoke ` AND removing their peer from the group. +# +# Default mode is DRY-RUN. Nothing is written without --apply. +# +# deploy/enroll-member.sh --new alice dry-run a setup key +# deploy/enroll-member.sh --new alice --apply create it, print it once +# deploy/enroll-member.sh --peer alices-mbp --apply add an EXISTING peer +# deploy/enroll-member.sh --list show the group's members +# +# The management API token is read from NETBIRD_TOKEN, an `nbtoken` helper, or +# the macOS Keychain, and is never printed, logged, or passed on a command line. +# +set -euo pipefail + +SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)" +ENV_FILE="${LOOMGRAPH_HUB_ENV:-$SCRIPT_DIR/hub.env}" +if [ -f "$ENV_FILE" ]; then + set -a + # shellcheck source=/dev/null + . "$ENV_FILE" + set +a +fi + +API_BASE="${NETBIRD_API:-}" +KEYCHAIN_SERVICE="${NETBIRD_KEYCHAIN_SERVICE:-netbird-pat}" +GROUP_MEMBERS="${LOOMGRAPH_MEMBERS_GROUP:-loomgraph-members}" +HUB_IP="${LOOMGRAPH_HUB_IP:-}" +HUB_PORT="${LOOMGRAPH_HUB_PORT:-8369}" + +# A setup key that never expires is a credential with no end date sitting in +# somebody's chat history. One-off plus a short window means a leaked key is +# useless by the time anyone finds it. +KEY_EXPIRY_SECONDS="${LOOMGRAPH_SETUP_KEY_EXPIRY:-86400}" + +MODE="dry-run" +ACTION="" +TARGET="" +TOKEN="" + +log() { printf '%s\n' "$*"; } +info() { printf ' %s\n' "$*"; } +die() { printf 'ERROR %s\n' "$*" >&2; exit 1; } +step() { printf '\n== %s\n' "$*"; } + +usage() { + cat <<'USAGE' +Usage: enroll-member.sh (--new | --peer | --list) [--apply] + + --new Create a one-off setup key that auto-joins the new peer to + the members group. Use for a colleague with no peer yet. + The key is printed ONCE and cannot be recovered. + --peer Add an EXISTING peer to the members group, by peer name, + hostname or mesh IP. Use when they are already on the mesh. + --list Print the members group's current peers. Read-only. + --apply Actually write. Without it, everything is a dry-run. + +Configuration comes from deploy/hub.env - see hub.env.example. +USAGE +} + +while [ "$#" -gt 0 ]; do + case "$1" in + --new) ACTION="new"; TARGET="${2:-}"; [ -n "$TARGET" ] || die "--new needs a name"; shift ;; + --peer) ACTION="peer"; TARGET="${2:-}"; [ -n "$TARGET" ] || die "--peer needs a name or IP"; shift ;; + --list) ACTION="list" ;; + --apply) MODE="apply" ;; + -h|--help) usage; exit 0 ;; + *) usage >&2; die "unknown argument: $1" ;; + esac + shift +done + +[ -n "$ACTION" ] || { usage >&2; die "pick one of --new, --peer or --list"; } +[ -n "$API_BASE" ] || die "NETBIRD_API is not set. Fill in $ENV_FILE (copy hub.env.example)." + +load_token() { + if [ -n "${NETBIRD_TOKEN:-}" ]; then + TOKEN="$NETBIRD_TOKEN" + elif command -v nbtoken >/dev/null 2>&1; then + TOKEN="$(nbtoken)" || die "the nbtoken helper failed; check it or set NETBIRD_TOKEN" + elif command -v security >/dev/null 2>&1; then + TOKEN="$(security find-generic-password -a "$USER" -s "$KEYCHAIN_SERVICE" -w 2>/dev/null)" || + die "no Keychain item for service \"$KEYCHAIN_SERVICE\" / account \"$USER\"; add it or set NETBIRD_TOKEN" + else + die "no token source: set NETBIRD_TOKEN, install nbtoken, or store the PAT in the Keychain" + fi + [ -n "$TOKEN" ] || die "the API token resolved to an empty string" +} + +# The token goes to curl through a --config file on STDIN. Never as -H on the +# command line: argv is world-readable in `ps` for the life of the process. +api() { + local method="$1" path="$2" body="${3:-}" + { + printf 'url = "%s%s"\n' "$API_BASE" "$path" + printf 'header = "Authorization: Token %s"\n' "$TOKEN" + printf 'header = "Content-Type: application/json"\n' + printf 'request = "%s"\n' "$method" + printf 'silent\nshow-error\nfail\n' + if [ -n "$body" ]; then + printf 'data-binary = "%s"\n' "@-" + fi + } > "$CURL_CONFIG" + + if [ -n "$body" ]; then + printf '%s' "$body" | curl --config "$CURL_CONFIG" || + die "$method $path failed" + else + curl --config "$CURL_CONFIG" < /dev/null || die "$method $path failed" + fi +} + +WORK_DIR="$(mktemp -d "${TMPDIR:-/tmp}/enroll-member.XXXXXX")" +CURL_CONFIG="$WORK_DIR/curl.conf" +trap 'rm -rf "$WORK_DIR"' EXIT + +group_id() { + api GET /groups > "$WORK_DIR/groups.json" + GROUP_NAME="$GROUP_MEMBERS" python3 - "$WORK_DIR/groups.json" <<'PY' +import json, os, sys +want = os.environ["GROUP_NAME"] +for g in json.load(open(sys.argv[1])): + if g["name"] == want: + print(g["id"]); raise SystemExit(0) +raise SystemExit(1) +PY +} + +main() { + load_token + log "API: $API_BASE" + + local gid + gid="$(group_id)" || die "group \"$GROUP_MEMBERS\" does not exist - run netbird-acl.sh first" + + case "$ACTION" in + list) + step "peers in \"$GROUP_MEMBERS\" ($gid)" + api GET /peers > "$WORK_DIR/peers.json" + GROUP_ID="$gid" python3 - "$WORK_DIR/groups.json" "$WORK_DIR/peers.json" <<'PY' +import json, os, sys +gid = os.environ["GROUP_ID"] +groups = json.load(open(sys.argv[1])) +peers = {p["id"]: p for p in json.load(open(sys.argv[2]))} +members = next(g for g in groups if g["id"] == gid).get("peers") or [] +if not members: + print(" (none yet)") +for m in members: + pid = m["id"] if isinstance(m, dict) else m + p = peers.get(pid, {}) + print(" %-24s %-16s %s" % (p.get("name", pid), p.get("ip", "?"), + "connected" if p.get("connected") else "offline")) +PY + ;; + + new) + step "one-off setup key \"$TARGET\", auto-joining \"$GROUP_MEMBERS\"" + local body + body="$(GROUP_ID="$gid" NAME="$TARGET" EXPIRY="$KEY_EXPIRY_SECONDS" python3 -c ' +import json, os +print(json.dumps({ + "name": os.environ["NAME"], + "type": "one-off", + "expires_in": int(os.environ["EXPIRY"]), + "usage_limit": 1, + "ephemeral": False, + "auto_groups": [os.environ["GROUP_ID"]], +}))')" + if [ "$MODE" != "apply" ]; then + info "DRY-RUN POST ${API_BASE}/setup-keys" + info "body: $body" + info "re-run with --apply to create it" + return 0 + fi + api POST /setup-keys "$body" > "$WORK_DIR/key.json" + local key + key="$(python3 -c 'import json,sys; print(json.load(open(sys.argv[1]))["key"])' "$WORK_DIR/key.json")" + print_member_instructions "$TARGET" "$key" + ;; + + peer) + step "add existing peer \"$TARGET\" to \"$GROUP_MEMBERS\"" + api GET /peers > "$WORK_DIR/peers.json" + local pid + pid="$(REF="$TARGET" python3 - "$WORK_DIR/peers.json" <<'PY' +import json, os, sys +ref = os.environ["REF"] +hits = [p for p in json.load(open(sys.argv[1])) + if ref in (p.get("name"), p.get("hostname"), p.get("ip"), p.get("id"))] +if len(hits) != 1: + raise SystemExit(1) +print(hits[0]["id"]) +PY +)" || die "no single peer matched \"$TARGET\" - check the name with: netbird-acl.sh --verify, or the NetBird console" + + # A group PUT REPLACES the peer list, so the existing members are read + # and sent back with the new one appended. Sending just the new id would + # silently evict everyone already in the group. + local body + body="$(GROUP_ID="$gid" NEW_PEER="$pid" NAME="$GROUP_MEMBERS" python3 - "$WORK_DIR/groups.json" <<'PY' +import json, os, sys +gid, new = os.environ["GROUP_ID"], os.environ["NEW_PEER"] +g = next(x for x in json.load(open(sys.argv[1])) if x["id"] == gid) +peers = [p["id"] if isinstance(p, dict) else p for p in (g.get("peers") or [])] +if new not in peers: + peers.append(new) +print(json.dumps({"name": os.environ["NAME"], "peers": peers})) +PY +)" + if [ "$MODE" != "apply" ]; then + info "DRY-RUN PUT ${API_BASE}/groups/${gid}" + info "body: $body" + info "re-run with --apply" + return 0 + fi + api PUT "/groups/${gid}" "$body" > /dev/null + info "peer \"$TARGET\" ($pid) is now in \"$GROUP_MEMBERS\"" + printf '\nThey still need a hub token. On the hub host:\n' + printf ' sudo -u lghub lg-hub member add --data-dir /var/lib/lghub\n' + ;; + esac +} + +print_member_instructions() { + local name="$1" key="$2" + cat < + +5. Opt in PER REPOSITORY. Nothing syncs until they do this, in each repo: + + lg sync --enable + + Have them read docs/hub-onboarding.md BEFORE step 5, not after. What + reaches the hub is readable by every member and can never be deleted. + +---------------------------------------------------------------------------- + +Then, from THEIR machine, prove the ACL holds: + + deploy/enroll-member.sh --list + deploy/netbird-acl.sh --verify --from-member + +The second one is the only way to check the negative criteria: a member peer +must reach the hub on tcp/${HUB_PORT} and nothing else, including your MacBooks. +It proves nothing from the operator machine, which is in "personal macbooks" +and is supposed to reach everything. +EOF +} + +main From 82b44b668d774d1e75eda696f8720a00d3076903 Mon Sep 17 00:00:00 2001 From: Dat Date: Mon, 21 Sep 2026 10:52:53 +0700 Subject: [PATCH 44/45] docs(hub): document the NetBird half of adding a member The runbook's Membership section predated `enroll-member.sh` and described only the hub token, with a one-line "do this after the peer exists" that never said how to make the peer exist. That left the network half as console tribal knowledge - the half that has to happen FIRST, since a token issued before its owner can reach the hub is a credential waiting in a chat window for a network change to make it live. Now four numbered steps: add the peer (one-off setup key with auto_groups, or an existing peer by name/IP), add the hub member, hand over all three artefacts together with the onboarding doc BEFORE `lg sync --enable`, then verify with `--verify --from-member` from their machine - the only place the negative criteria can be proven. Also records two things that cost time: an empty members group right after issuing a setup key is normal, because `auto_groups` applies at join time rather than at creation; and `netbird up` silently drops its flags when the client is already connected. Co-Authored-By: Claude Opus 5 (1M context) --- docs/hub-operations.md | 76 +++++++++++++++++++++++++++++++++++++++--- 1 file changed, 72 insertions(+), 4 deletions(-) diff --git a/docs/hub-operations.md b/docs/hub-operations.md index bdcca83..6514a6c 100644 --- a/docs/hub-operations.md +++ b/docs/hub-operations.md @@ -159,11 +159,52 @@ simply unreachable until the mesh is up. See "wt0 down at boot" below. ## Membership -### Add a member +Every member needs **two independent grants**, and neither is useful alone. A +token without mesh access cannot reach the port; mesh access without a token +gets a 401. -Do this only after the member's NetBird peer exists, is in the -`loomgraph-members` group, and has been verified to reach `$HUB_IP:8369` -and nothing else. Network access first, hub token second — never the reverse. +| Layer | Grants | Tool | +| --- | --- | --- | +| Network | Their peer may send packets to `$HUB_IP:8369` | `deploy/enroll-member.sh` (operator machine) | +| Identity | The hub accepts and answers their requests | `lg-hub member add` (hub host) | + +Do the network half **first**. A token that exists before its owner can reach +the hub is a credential sitting in a chat window waiting for a network change to +make it live. + +### 1. Add the NetBird peer + +For a colleague with no peer yet, mint a one-off setup key that auto-joins the +members group. Dry-run first — every write in these scripts is opt-in: + +```bash +deploy/enroll-member.sh --new hoang.luong # dry-run, prints the API call +deploy/enroll-member.sh --new hoang.luong --apply # creates it, prints the key ONCE +``` + +The key is single-use and expires in 24h (`LOOMGRAPH_SETUP_KEY_EXPIRY`). Its +`auto_groups` puts the peer into `loomgraph-members` at join time, so there is no +window in which a peer is on the mesh but ungrouped, and no second step to +forget. + +For someone already on the mesh, add their existing peer instead: + +```bash +deploy/enroll-member.sh --peer hoangs-mbp --apply # by name, hostname, mesh IP or id +``` + +Check who is in the group at any time: + +```bash +deploy/enroll-member.sh --list +``` + +An empty group right after issuing a setup key is normal — `auto_groups` applies +when the peer actually connects, not when the key is created. + +### 2. Add the hub member + +Do this only after the peer exists and is in the `loomgraph-members` group. ```bash sudo -u lghub lg-hub member add alice --data-dir /var/lib/lghub @@ -202,6 +243,33 @@ Tell the recipient two things when you hand it over: - Pasting the token into the web UI stores it in that browser's `localStorage`, in clear, on a plain `http://` origin. +### 3. Hand it over + +Send all three together, and point them at +[hub-onboarding.md](./hub-onboarding.md) **before** they run `lg sync --enable`, +not after — that is the irreversible step. + +1. `docs/hub-onboarding.md` +2. The setup key +3. The hub token + +Their own command sequence is printed by `enroll-member.sh --apply`. The step +that reliably goes wrong is `netbird up`: it silently ignores its flags when the +client is already connected, printing "Already connected" and dropping them. +`netbird down` first. There is no `netbird set`. + +### 4. Verify, from their machine + +```bash +deploy/enroll-member.sh --list # operator: their peer appears +deploy/netbird-acl.sh --verify --from-member # THEIR machine +``` + +`--from-member` is the only way to prove the negative criteria — that a member +reaches the hub on tcp/8369 and nothing else, including the operator MacBooks. It +proves nothing from the operator machine, which is in `personal macbooks` and is +supposed to reach everything. + ### List members ```bash From 9c885fd13efaec31ad5499d2ca9f86973c05c889 Mon Sep 17 00:00:00 2001 From: Dat Date: Mon, 21 Sep 2026 10:54:20 +0700 Subject: [PATCH 45/45] docs: add a member quickstart, the doc you hand a new colleague `hub-onboarding.md` explains what syncing shares and why the decision is irreversible. It deliberately never says which commands to run, so until now there was nothing to hand someone that got them from "nothing installed" to "pushed a run" - that lived in an operator's head and in the block `enroll-member.sh` prints. `member-quickstart.md` is that page. Six numbered steps, the three artefacts to ask the operator for (two of which are issued once), and a troubleshooting section made entirely of diagnostics that lie in this stack: `netbird up` dropping its flags while connected, `/healthz` returning 200 HTML while the API is dead, `nc -z` missing NetBird's userspace SSH listener, and `dig` bypassing macOS scoped resolvers. Each of those has already cost someone an afternoon. Step 5 gates on reading `hub-onboarding.md` rather than restating it - the two pages answer different questions and should not drift into one. Cross-linked from the README's hub section, from the runbook's hand-over step (now four artefacts, not three), and from the script's printed block. Co-Authored-By: Claude Opus 5 (1M context) --- README.md | 8 ++ deploy/enroll-member.sh | 6 +- docs/hub-operations.md | 16 +-- docs/member-quickstart.md | 203 ++++++++++++++++++++++++++++++++++++++ 4 files changed, 225 insertions(+), 8 deletions(-) create mode 100644 docs/member-quickstart.md diff --git a/README.md b/README.md index 58f311b..f82dc78 100644 --- a/README.md +++ b/README.md @@ -520,6 +520,14 @@ one: `hub.db`, `hub.db-wal` and `hub.db-shm`. Copying `hub.db` alone while the server is running gives you a backup missing every committed write still in the WAL. Use `VACUUM INTO` (or stop the service first). +Onboarding a colleague is two grants, not one — a NetBird peer in the members +group, and a hub token. `deploy/enroll-member.sh` does the first, +`lg-hub member add` the second, and +[docs/hub-operations.md](docs/hub-operations.md) is the operator runbook for +both. Hand the new member [docs/member-quickstart.md](docs/member-quickstart.md) +(the commands) and [docs/hub-onboarding.md](docs/hub-onboarding.md) (what +syncing shares). + On a member machine: ```bash diff --git a/deploy/enroll-member.sh b/deploy/enroll-member.sh index 87949d3..a17d725 100755 --- a/deploy/enroll-member.sh +++ b/deploy/enroll-member.sh @@ -247,6 +247,10 @@ destroyed afterwards, and delete it there once they are connected. --- give them this --------------------------------------------------------- +0. Send them docs/member-quickstart.md - it is these steps, in full, with the + troubleshooting for the ones that reliably go wrong. The rest of this block + is the short form. + 1. Install NetBird: https://docs.netbird.io/how-to/installation 2. Join the mesh. 'netbird up' SILENTLY IGNORES these flags if the client is @@ -269,7 +273,7 @@ destroyed afterwards, and delete it there once they are connected. lg sync --enable - Have them read docs/hub-onboarding.md BEFORE step 5, not after. What + Have them read docs/hub-onboarding.md BEFORE this step, not after. What reaches the hub is readable by every member and can never be deleted. ---------------------------------------------------------------------------- diff --git a/docs/hub-operations.md b/docs/hub-operations.md index 6514a6c..c813760 100644 --- a/docs/hub-operations.md +++ b/docs/hub-operations.md @@ -245,15 +245,17 @@ Tell the recipient two things when you hand it over: ### 3. Hand it over -Send all three together, and point them at -[hub-onboarding.md](./hub-onboarding.md) **before** they run `lg sync --enable`, -not after — that is the irreversible step. +Send all four together: -1. `docs/hub-onboarding.md` -2. The setup key -3. The hub token +1. [member-quickstart.md](./member-quickstart.md) — every command they run, in + order, with troubleshooting +2. [hub-onboarding.md](./hub-onboarding.md) — what syncing actually shares. + Point at this one explicitly: it gates step 5 of the quickstart, and it is + the only irreversible decision in the process +3. The setup key — single use, 24h +4. The hub token — printed once -Their own command sequence is printed by `enroll-member.sh --apply`. The step +Their own command sequence is also printed by `enroll-member.sh --apply`. The step that reliably goes wrong is `netbird up`: it silently ignores its flags when the client is already connected, printing "Already connected" and dropping them. `netbird down` first. There is no `netbird set`. diff --git a/docs/member-quickstart.md b/docs/member-quickstart.md new file mode 100644 index 0000000..1507a27 --- /dev/null +++ b/docs/member-quickstart.md @@ -0,0 +1,203 @@ +# Member quickstart — join the mesh, join the hub + +For a new member. Start to finish in about fifteen minutes, most of it waiting +for installers. + +Read [hub-onboarding.md](./hub-onboarding.md) **before step 5**. This page tells +you which commands to run; that one tells you what you are agreeing to share, and +it is the part you cannot undo. + +## What you are joining + +Two separate systems, and you need both. + +| | What it is | What it gives you | +| --- | --- | --- | +| **NetBird** | A WireGuard mesh VPN | A private address that can reach the hub. Nothing else on the mesh. | +| **loomgraph hub** | An HTTP API over SQLite | Somewhere to push your run telemetry so the team can see it | + +The hub is not on the public internet. There is no URL you can open from a café +without the mesh — that is deliberate, and it is what protects your token. + +**The hub never runs an agent.** `lg run` starts `claude`, `codex` or `opencode` +on *your* laptop, as *you*, under your own subscription. Nobody shares an +account, no token or session of yours is copied anywhere, and if the hub is down +your runs are unaffected — only the sync fails. + +## What your operator gives you + +Three things. Ask for all three before you start; two of them can only be issued +once. + +| | Looks like | Notes | +| --- | --- | --- | +| Management URL | `https://netbird.example.com` | NetBird control plane | +| Setup key | `A1B2C3D4-...` | **Single use, expires in 24h.** Not recoverable. | +| Hub token | `lgt_1a2b3c4d.` | **Printed once.** Not recoverable. | +| Hub address | `http://100.x.y.z:8369` | A mesh address; useless off the mesh | + +Delete the setup key and the token from wherever they were sent to you as soon +as you have finished step 4. + +## 1. Install NetBird + + — packages for macOS, Linux, +Windows, iOS and Android. + +## 2. Join the mesh + +```bash +netbird down +netbird up --setup-key --management-url +``` + +`netbird down` first is not optional. **`netbird up` silently ignores its flags +when the client is already connected** — it prints `Already connected`, drops +your setup key and management URL, and leaves you attached to whatever it was +using before. There is no `netbird set`. + +Check it worked: + +```bash +netbird status --detail | grep -E "Management|Signal|Peers count" +``` + +You should see your own mesh IP and a peer count above zero. + +## 3. Confirm you can reach the hub + +```bash +curl -s http:///v1/health +# {"ok":true,"version":"0.1.0"} +``` + +That exact body is the test. A 200 alone proves nothing — see +[Troubleshooting](#troubleshooting). + +You are expected to reach this one address on this one port and **nothing else** +on the mesh. That is not a restriction aimed at you; it is what lets the team run +a shared hub on a network that also has people's laptops on it. + +## 4. Install loomgraph and enroll + +loomgraph is not on npm. Build it from the repo: + +```bash +git clone && cd loomgraph +npm install && npm run build +npm link +``` + +Then store your identity: + +```bash +lg enroll http:// +``` + +That writes `~/.config/loomgraph/hub.json`, mode 0600. + +**Your token goes into your shell history this way.** If you would rather it did +not, skip `lg enroll` and export both of these instead — `lg` prefers them over +the config file, and both must be set or neither is used: + +```bash +export LOOMGRAPH_HUB_URL=http:// +export LOOMGRAPH_HUB_TOKEN= +``` + +Either way, put the token in your OS keychain as the master copy — macOS +Keychain, `secret-tool` on Linux. Not a dotfile you back up, not a note app, not +a shared drive. + +**Possession equals identity.** Anyone holding that string *is* you to the hub. +There is no second factor and no device binding. If you think anyone else has +seen it, say so immediately — revocation is instant and costs nothing. + +## 5. Read the onboarding doc, then opt in per repository + +Stop here and read [hub-onboarding.md](./hub-onboarding.md). It is short, and it +is the only part of this process you cannot reverse. + +The short version: **anything that reaches the hub is readable by every member +and can never be deleted.** The events table aborts UPDATE and DELETE by database +trigger. Filtering runs before anything leaves your machine — paths, usernames, +hostnames and known secret shapes are removed, and everything is capped at 200 +characters — but it is an allowlist of shapes someone thought of, not a proof. +Your organisation's own token format is probably not one of them. + +Nothing syncs until you opt in, and you opt in **per repository**: + +```bash +cd ~/work/some-repo +lg sync --enable # writes .loomgraph/hub.json +``` + +Without that file, `lg sync` refuses and pushes nothing. Enable it only on +repositories where you would be comfortable with the whole team reading your +build's stderr, forever. + +## 6. Run something and push it + +```bash +lg run examples/hello.yaml # three shell commands, no model calls, zero cost +lg ls # find the run id +lg sync # push that one run +lg sync --all # or every run in this repo +``` + +Runs also stream to the hub live as they execute, once the repo is opted in. + +## Day-to-day + +| Command | What it does | +| --- | --- | +| `lg run ` | Execute a graph | +| `lg ls` | List your runs | +| `lg status ` | Node and budget status | +| `lg resume ` | Continue from the last checkpoint | +| `lg events ` | The raw local JSONL trail | +| `lg report ` | Self-contained HTML report | +| `lg sync ` / `--all` | Push to the hub | + +Your local `.loomgraph/runs//events.jsonl` keeps **raw** values — that is +your debugging record, and it is never rewritten. Only the copy crossing to the +hub is filtered. + +## Troubleshooting + +**`curl` to the hub hangs or refuses.** Check `netbird status` first. If the +client is connected but the hub is unreachable, your peer may not be in the +members group yet — ask your operator to run `deploy/enroll-member.sh --list`. + +**A 200 from the hub that is not JSON.** If the hub is running its web UI, any +non-`/v1` path returns HTML with a 200. `/healthz` looks healthy even when the +API is dead. Always probe `/v1/health` and read the body. + +**`nc -z 22` says the port is closed, but `ssh` works.** Expected. NetBird +clients run their SSH server in userspace netstack, so it is not a host socket +and a raw port probe cannot see it. Test with the real client, not a probe. + +**`dig` says `.netbird.selfhosted` does not resolve.** On macOS `dig` bypasses +scoped resolvers. Use `dscacheutil -q host -a name ` instead. + +**`lg sync` says the repo is not enabled.** You have not run `lg sync --enable` +in *that* repository. It is per repo, deliberately. + +**`lg sync` says the hub is not configured.** `lg enroll` has not run, or your +two environment variables are half-set — both `LOOMGRAPH_HUB_URL` and +`LOOMGRAPH_HUB_TOKEN`, or neither. + +**401 from the hub.** Your token has been revoked, or you pasted it wrong. There +is no session cache to wait out; every request re-resolves the token. + +## Verify the network boundary yourself + +Once you are on the mesh, from *your* machine: + +```bash +deploy/netbird-acl.sh --verify --from-member +``` + +This checks the part nobody else can: that your peer reaches the hub on its one +port and is blocked from everything else, including the operator's laptops. If +anything there FAILs, tell your operator before you sync anything.