Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
6c28b1b
docs: add the team hub design and threat model
datj9 Aug 25, 2026
00ea918
docs: add the hub implementation plan and fix what review found
datj9 Aug 25, 2026
4b3db20
feat: add a scan rule for hub tokens
datj9 Aug 25, 2026
748be01
chore: require node 22.13 for node:sqlite
datj9 Aug 25, 2026
8402e49
docs: declare the projected state type with the wire vocabulary
datj9 Aug 25, 2026
95ac109
docs: make the projected state single-sourced and unambiguous on the …
datj9 Aug 25, 2026
4aa9cb5
docs: scope the storage law to the component that owns it
datj9 Aug 25, 2026
69bff0d
feat: add the hub wire types
datj9 Aug 25, 2026
d0e810c
fix: tolerate unknown fields on the wire but not in the projection
datj9 Aug 25, 2026
ba79115
refactor: tag the ingest result union so a conflict cannot be missed
datj9 Aug 25, 2026
38b0dcc
fix: refuse a node whose map key and nodeId disagree
datj9 Aug 25, 2026
fca1326
docs: define the high-water mark for a batch that carries no events
datj9 Aug 25, 2026
8e365ab
fix: constrain the wire so an incoherent batch cannot be stored
datj9 Aug 25, 2026
49a9146
refactor: type the run row status as a run status
datj9 Aug 25, 2026
054afd2
fix: reject wire values that cannot describe a real run
datj9 Aug 25, 2026
d239293
feat: add the hub store
datj9 Aug 25, 2026
ec5fc48
docs: correct the index ddl, the minting boundary and the feed deriva…
datj9 Aug 25, 2026
45057d5
feat: add hub auth
datj9 Aug 25, 2026
64d05f6
fix: agree on one alphabet for hub token key ids
datj9 Aug 25, 2026
3ac79b0
feat: add the hub http handlers
datj9 Aug 25, 2026
5dc3117
fix: report the error the handler actually found
datj9 Aug 25, 2026
d4c7b1a
feat: add the lg-hub binary
datj9 Aug 25, 2026
6b7d39f
fix: export the stored event line verbatim
datj9 Aug 25, 2026
69efac6
feat: stamp a stream id on every run
datj9 Aug 25, 2026
787f16f
feat: project run state before it leaves the machine
datj9 Aug 25, 2026
dd92810
fix: mask the node error before it leaves the machine
datj9 Aug 25, 2026
505d092
feat: add the client transport and sync cursor
datj9 Aug 25, 2026
b2e19fe
fix: bound the hub request regardless of the transport
datj9 Aug 25, 2026
854a975
feat: add lg enroll and lg sync
datj9 Aug 25, 2026
17a1a42
feat: push events as a run produces them
datj9 Aug 25, 2026
ca7aa60
test: prove the cursor never passes the last ack
datj9 Aug 26, 2026
a6d7b11
docs: describe the hub in the readme
datj9 Aug 26, 2026
7d46de7
test: assemble hub token fixtures at runtime
datj9 Aug 26, 2026
7c4b734
Merge remote-tracking branch 'origin/main' into feat/hub-phase1
datj9 Aug 27, 2026
fb12c2d
feat(hub): serve a web UI and expose runs/members over HTTP
datj9 Sep 17, 2026
02cb20f
fix(team): sanitise synced event lines so secrets never reach the hub
datj9 Sep 21, 2026
06f7135
docs(readme): correct four stale claims about install, the hub store …
datj9 Sep 21, 2026
d47f4dc
fix(sync): gate manual syncs on the repo opt-in, not just the live ba…
datj9 Sep 21, 2026
7348068
feat(deploy): add hub provisioning, backup and NetBird ACL scripts
datj9 Sep 21, 2026
213c05e
docs(hub): add member onboarding and operator runbook
datj9 Sep 21, 2026
f1696d0
fix(deploy): quote hub.env values that contain spaces
datj9 Sep 21, 2026
7f085e1
chore: ignore npm pack output
datj9 Sep 21, 2026
4ae6cb9
docs(deploy): say which regex flavour LOOMGRAPH_HEALTH_EXPECT uses
datj9 Sep 21, 2026
5b70969
feat(deploy): add enroll-member.sh for the NetBird half of onboarding
datj9 Sep 21, 2026
82b44b6
docs(hub): document the NetBird half of adding a member
datj9 Sep 21, 2026
9c885fd
docs: add a member quickstart, the doc you hand a new colleague
datj9 Sep 21, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -11,3 +11,10 @@ dist/
# Never commit either - the share url grants read access to the artifact.
handoff-bundle/
SHARE-URL.txt

# deploy/hub.env holds one deployment's real addresses: mesh IPs, the control
# plane URL, SSH fallbacks. Copy deploy/hub.env.example and fill it in locally.
deploy/hub.env

# npm pack output: the install artifact deploy/install-hub.sh consumes.
*.tgz
16 changes: 12 additions & 4 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,8 @@ node dist/cli.js run examples/hello.yaml # smoke test, zero cost
- **Tests never spawn a real agent CLI.** No test may execute `claude`, `codex`, `opencode`, `enclave`, or `git`, and no test may make a network request. Adapter tests parse fixture strings; engine tests inject stub adapters through the registry argument; `src/handoff/` routes every spawn - including `opencode export` and `enclave push` - through an injected `Exec` seam that tests replace with a fake.
- **Never invent cost numbers.** If a CLI does not report a price, record `0`. Do not derive cost from a token count and a price table anywhere in this codebase.
- **Checkpoint after every edge crossing.** Not at the end of a batch, not at the end of the run. `CheckpointStore.save` writes a temp file and renames it; never write `state.json` in place.
- **The event log is append-only and unbuffered.** A killed process must leave a readable JSONL log. Do not add buffering or rewrite past lines.
- **The event log is append-only and unbuffered.** This rule governs `.loomgraph/runs/<id>/events.jsonl` on the machine that ran the graph. A killed process must leave a readable JSONL log. Do not add buffering or rewrite past lines.
- **The hub's truth is SQLite.** `lg-hub export --jsonl` is a derived, rebuildable artifact. Never make a JSONL file on the hub authoritative, never make the laptop's log a database.
- **Graph validation stays loud.** Unknown node ids, cycles, missing budgets and unknown adapters throw with the offending node id in the message. Do not downgrade a validation error to a warning.
- **No LLM SDK dependency.** The agent CLIs are the runtime. Adding an API client to `dependencies` is out of scope for this project.
- **Adapter output is a contract.** Every adapter returns `{ ok, text, costUsd, raw, error }`. Cost is recorded even when the run failed, because budget accounting depends on it.
Expand All @@ -36,13 +37,20 @@ src/core/ types, store, events, graph, budget, engine (no CLI concerns)
src/adapters/ one file per executor, plus the registry
src/commands/ CLI command implementations and pure renderers
src/handoff/ the `lg-handoff` bin: session readers, secret scanner, brief renderer
src/hub/ the `lg-hub` bin: HTTP API and SQLite database
src/team/ client-side code for the team hub
examples/ graph files that must stay valid (`lg validate`)
```

Nothing under `src/handoff/` may import from `src/core/` or `src/adapters/`. The subtree
owns its own enclave helpers so it stays extractable into a sibling package with a `git mv`.
That is why `buildEnclavePushArgs` exists twice and neither copy should be deduplicated
into a shared module.
owns its own enclave helpers, so it stays extractable into a sibling package with a `git mv`;
once extracted, the hub depends on that sibling package, not on this repo. That is why
`buildEnclavePushArgs` exists twice and neither copy should be deduplicated into a shared
module.

`src/hub/` and `src/team/` may import from `src/core/` and `src/handoff/scan.ts`; the arrow
never reverses. Do not fork the scanner - the `buildEnclavePushArgs`-exists-twice precedent
covers argv builders, not security gates.

Inside `src/handoff/`, one file per job, and the data flows one way:

Expand Down
157 changes: 151 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,11 +19,15 @@ It does not call a model itself. Your agent CLIs are the runtime.

## Install

Not published to npm. Build it from a clone and link the binaries:

```bash
npm i -g loomgraph
git clone https://github.com/datj9/loomgraph && cd loomgraph
npm install && npm run build
npm link # or: npm pack && npm i -g ./loomgraph-0.1.0.tgz
```

Requires Node >= 22. The binary is `lg`.
Requires Node >= 22. Three binaries land on your PATH: `lg`, `lg-handoff` and `lg-hub`.

## 60-second quickstart

Expand Down Expand Up @@ -486,18 +490,159 @@ always means "looked at and found nothing".
headings. There is no model call, so `pack` works offline and cannot invent a claim.
- **`private` visibility only.** A transcript is production data. `--visibility org` and
`public` are refused.
- **No signal bus, inbox, or daemon.** That would contradict "Not a workflow server"
below, and an inbox that starts an agent on someone else's laptop is a different product
with a much harder threat model.
- **No inbox.** An inbox that starts an agent on someone else's laptop is a different
product with a much harder threat model, and phase 1 ships no inbox - that is phase 3.
There is a daemon now, and the Hub section below is precise about what it does.

Known rough edge: the `enclave share create --json` parser accepts several plausible field
names because that stdout shape has not yet been captured from a real invocation.

## The hub

The third binary is `lg-hub`: an HTTP API in front of one SQLite database. It
stores what members push and serves reads out of that store - it never runs an agent.
Agents run on the member's own machine, started by the member; the hub has no way to
start one, and that absence is the design. A daemon that can only store and route is a
store, not a scheduler.

### Set one up

On the hub host:

```bash
lg-hub init # create the data dir and hub.db
lg-hub member add alice # prints alice's token once - write it down
lg-hub serve # binds 127.0.0.1:8369, web UI on the same origin
```

That database runs in WAL mode, so the hub's state on disk is **three** files, not
one: `hub.db`, `hub.db-wal` and `hub.db-shm`. Copying `hub.db` alone while the
server is running gives you a backup missing every committed write still in the
WAL. Use `VACUUM INTO` (or stop the service first).

Onboarding a colleague is two grants, not one — a NetBird peer in the members
group, and a hub token. `deploy/enroll-member.sh` does the first,
`lg-hub member add` the second, and
[docs/hub-operations.md](docs/hub-operations.md) is the operator runbook for
both. Hand the new member [docs/member-quickstart.md](docs/member-quickstart.md)
(the commands) and [docs/hub-onboarding.md](docs/hub-onboarding.md) (what
syncing shares).

On a member machine:

```bash
lg enroll http://10.0.0.5:8369 <token> # identity lives in ~/.config/loomgraph/hub.json
lg sync --enable # opt this repo in: .loomgraph/hub.json
lg run examples/hello.yaml
lg sync <runId> # push one run
lg sync --all # or every run under .loomgraph/runs/
```

`lg sync --enable` is a hard gate, not a hint: without `.loomgraph/hub.json` in
the repo, `lg sync <runId>` and `lg sync --all` both refuse and push nothing. The
hub cannot delete an event once ingested, so opting in has to be a deliberate act
per repo rather than something a forgotten flag decides for you.

The rest of the hub-facing surface: `lg-hub member revoke <keyId>` and
`lg-hub member ls` for the roster, `lg-hub export --jsonl` to print the raw stored
lines to stdout for grepping, and `lg-hub export --out <dir>` to write one
`runs/<member>/<runId>/events.jsonl` per run.

### What the hub receives

A sync pushes two things, and **both** are filtered on the way out:

**The run state projection.** Built field by field, never as a filtered copy of the
full state, so there is no field a `vars` value or a node `output` could ride in on:

- `vars` reach the hub as key names only.
- node `output` never reaches the hub.
- node `error` is control-stripped, path-rewritten, secret-masked, and capped at 200
characters.

**The run's event lines.** These are *not* pushed verbatim. Each line is rebuilt
against a per-kind allowlist of `data` fields before it leaves the machine; a field
not on the list is dropped, and a line whose `kind` the allowlist does not know is
dropped whole. Fields carrying operator or environment text - `node_finished.error`
and `run_finished.error` (an adapter folds the agent's full result text and stderr
into these), `run_started.cwd`, the interpolated `human_requested.question`, and
`human_resolved.answer` - go through the same strip/rewrite/mask/cap as a node error.

The filtering happens at **push** time, not at emission. Your local
`.loomgraph/runs/<runId>/events.jsonl` keeps its raw values for debugging; only the
copy crossing to the hub is sanitised. `lg-hub export` still reproduces the *ingested*
lines byte for byte - those lines simply no longer carry secrets.

Same error before and after:

```
in : command exited with code 1: /home/alice/work/repo/run.sh: token sk-ant-api03-… rejected
out: command exited with code 1: ${REPO_ROOT}/run.sh: token sk-a... rejected
```

### Tokens are possession-equals-identity

Anyone holding a member token is that member to the hub. `member add` prints a token
once and the store keeps only its hash, so it can never be printed again - treat it
like an SSH key, and `member revoke <keyId>` is the off switch.

### `serve` refuses a non-loopback bind

`lg-hub serve` binds `127.0.0.1` by default and refuses any other host unless
`--behind-tls-proxy` is passed. A bearer token over plaintext non-loopback HTTP is
exactly the credential shape this project's own scanner has a rule for, so a bind that
would put the token on the wire without TLS is an error rather than an option.

### The web UI ships, and it is on by default

`lg-hub serve` serves a self-contained web UI on the same origin as the API - one
embedded HTML document, no build step, no external requests. It renders runs, the
activity feed and the member roster using `textContent` only, so an untrusted
transcript cannot inject markup. Pass `--no-ui` for an API-only bind.

It authenticates with a bearer token you paste, and **keeps that token in
`localStorage`**. On the default loopback bind that is fine. Behind
`--behind-tls-proxy` on a plaintext `http://` origin it is not: the token sits in
browser storage on an origin anyone on that network can impersonate. Terminate TLS in
front of it, or run `--no-ui`.

### Two caveats, stated up front

**Phase 1 does not mask on egress.** The projection is the only gate; whatever does
reach the hub is served back as stored. A repository whose var *key names* are
themselves sensitive should keep sync disabled - redaction-on-read is phase 2.

**The masking is an allowlist, not a proof.** Error masking reuses `lg-handoff`'s
`SCAN_RULES`, so it catches the shapes those rules know and nothing else. During
development a test canary shaped `AKIA` plus 18 more characters passed through
unmasked, because the rule is `\bAKIA[0-9A-Z]{16}\b` - exactly 20 characters. The
canary was malformed rather than the rule being wrong, but that is exactly the point: a
shape the rules do not cover reaches a team-readable field unmasked.

### Verified against a live hub

Both of these were run end to end against the built binary with a live hub:

- **The export is byte-identical to the local log.** `lg-hub export` reproduces the
ingested lines exactly; it does not re-encode them.
- **A dead hub changes neither the exit code nor the node outcomes.** The hub was
killed mid-exercise; the run was unaffected.

### What phase 1 does not ship

- No inbox - that is phase 3.
- No briefs on the hub, no encryption at rest, and no redaction on read - all phase 2.
Nothing in phase 1 is encrypted.
- No full transcripts, ever. The handoff refusal stands unchanged: a transcript is a
credential dump, and syncing to the hub does not soften that.

## What this is not

- **Not a model, and not an SDK for one.** loomgraph makes zero API calls of its own and has no LLM SDK dependency.
- **Not a replacement for your agent CLI.** It shells out to the CLI you already installed and authenticated.
- **Not a workflow server.** No daemon, no web UI, no cloud, no plugin system in v0.1.
- **Not a workflow server.** A daemon ships in phase 1 - `lg-hub` - but it stores and
routes, and never runs an agent. It ships a web UI over its own store - reads, plus member add/revoke; no
cloud, and no plugin system in v0.1.

`lg report --publish` does not change that: it writes a static file and shells out to the
`enclave` cli the same way a node shells out to `claude`. If `enclave` is not installed the
Expand Down
Loading
Loading