Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ jobs:
strategy:
fail-fast: false
matrix:
node: [20, 22]
node: [22, 24]
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
Expand Down
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,4 @@ dist/
*.log
.claude/
.hermes/
.devkit/
2 changes: 1 addition & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ npm run typecheck && npm test && npm run build
node dist/cli.js run examples/hello.yaml
```

Node >= 20 is required.
Node >= 22 is required (`execa` v10 uses `Set.prototype.union`, which lands in Node 22).

## Working on a change

Expand Down
43 changes: 37 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ It does not call a model itself. Your agent CLIs are the runtime.
npm i -g loomgraph
```

Requires Node >= 20. The binary is `lg`.
Requires Node >= 22. The binary is `lg`.

## 60-second quickstart

Expand Down Expand Up @@ -118,7 +118,7 @@ edges:
to: END
```

Templates resolve against run state: `{{vars.ticket}}` (or the shorthand `{{ticket}}`) and `{{nodes.<id>.output}}`. An unresolvable reference is an error, not an empty string.
Templates resolve against run state: `{{vars.ticket}}` (or the shorthand `{{ticket}}`) and `{{nodes.<id>.output}}`. An unresolvable reference is an error, not an empty string and never a passthrough — which is why node ids are restricted to `[A-Za-z0-9_-]`, 1 to 64 characters. A dot would collide with the reference syntax itself, so `lg validate` rejects it rather than letting `{{nodes.my.node.output}}` mean nothing at run time.

Check a graph before running it — `lg validate` catches unknown node ids, cycles, missing budgets, and bad adapters:

Expand Down Expand Up @@ -171,16 +171,27 @@ lg resume <runId> --answer approve="ship it"

Every node accepts `retries` (default 0), `timeoutSec` (default 900), and `cwd`.

A `command` node also accepts two optional assertions, because a shell command that exits 0
having done nothing is not a passing check: `expectNonEmpty: true` fails the node when the
command wrote no output, and `expect: "<literal>"` fails it when that substring is absent
from stdout. `npm run lint --if-present` in a repo with no lint script is the case these
exist for.

## Budgets

Three ceilings, all enforced *before* each dispatch batch, all recorded in the checkpoint:
Three ceilings, all enforced *before* each dispatch batch **and once more before a run is
allowed to finish successfully**, all recorded in the checkpoint:

- `maxUsd` — summed from what the adapters actually report.
- `maxWallClockSec` — measured from the run's creation, so it survives a resume.
- `maxNodeRuns` — counts every attempt, retries included.

Hitting a ceiling stops the run with status `failed`, a `budget_exceeded` event naming the ceiling, and exit code 3. Nothing further is dispatched.

A ceiling breached by the final batch fails the run too. A node that already finished keeps
its result — the run fails, the work does not unwind — so `lg status` still shows what was
done and exactly how far over the line it went.

Cost numbers are never invented. Claude Code reports `total_cost_usd` and that number is used as-is; adapters that report no price record exactly `0.0000`, and `lg status` says so.

## Audit trail
Expand All @@ -200,11 +211,26 @@ Event kinds: `run_started`, `node_started`, `node_finished`, `edge_crossed`, `bu

| Adapter | Command it runs | Status |
| --- | --- | --- |
| `claude` | `claude -p <prompt> --output-format json --permission-mode acceptEdits --max-turns <n>` | Tested against Claude Code 2.1.232 |
| `claude` | `claude -p <prompt> --output-format json --permission-mode acceptEdits --max-turns <n>` | Tested against Claude Code 2.1.232 and the array-form json output of 3.x |
| `codex` | `codex exec <prompt> --json --skip-git-repo-check --sandbox read-only -C <cwd>` | Tested against codex-cli 0.145.0 |
| `opencode` | `opencode run <prompt>` | **Experimental — never executed against a real binary.** The parser is unit-tested; the invocation is not. |
| `opencode` | `opencode run --format json [-m <model>] <prompt>` | Tested against opencode 1.18.17 |

Cost reporting differs by CLI: Claude Code reports `total_cost_usd`, and OpenCode reports a price per step under `--format json` — which is the only reason this adapter uses that format, since the default one prints prose and no price at all. Codex reports nothing, and loomgraph records `0` for it rather than estimating from a price table. Wall-clock and node-run ceilings still apply either way.

Cost reporting differs by CLI: Claude Code reports `total_cost_usd`; Codex and OpenCode report nothing, and loomgraph records `0` rather than estimating from a price table. Wall-clock and node-run ceilings still apply to them.
### Choosing a model

An `agent` or `verifier` node may name the model it wants, passed straight through to the CLI:

```yaml
review:
type: verifier
adapter: opencode
model: "opencode-go/deepseek-v4-flash"
prompt: "Review the diff. Reply PASS or FAIL."
pass: "PASS"
```

Omit it and the CLI's own resolution decides, which is not always what the config says: with no `-m`, opencode ignored a configured `model` and fell through to a provider with no credentials. `OPENCODE_MODEL` is ignored — the flag is the only way. A `command` or `human` node that declares a model is a validation error rather than a silently ignored key.

### Environment

Expand All @@ -230,6 +256,7 @@ Both agent adapters close stdin before spawning. Codex otherwise prints `Reading
| `lg status <runId>` | Per-node table plus the budget line |
| `lg ls` | Every run with status and cost |
| `lg validate <graph.yaml>` | Exit 0 if valid, else exit 1 with the specific error |
| `lg report <runId> [--out path] [--publish] [--title t] [--visibility private\|org]` | Render the run to a self-contained html file; `--publish` hosts it with the `enclave` cli |
| `lg events <runId> [--kind K]` | The JSONL audit trail, filterable |

Exit codes: `0` success, `1` validation or usage error, `2` run failed, `3` budget exceeded, `4` paused awaiting a human.
Expand All @@ -240,6 +267,10 @@ Exit codes: `0` success, `1` validation or usage error, `2` run failed, `3` budg
- **Not a replacement for your agent CLI.** It shells out to the CLI you already installed and authenticated.
- **Not a workflow server.** No daemon, no web UI, no cloud, no plugin system in v0.1.

`lg report --publish` does not change that: it writes a static file and shells out to the
`enclave` cli the same way a node shells out to `claude`. If `enclave` is not installed the
report is still written, and nothing is uploaded.

Concurrency caveat: fan-out nodes in v0.1 share one working directory. If two branches edit the same files, they will collide. Per-node git worktrees are phase 2.

## Roadmap
Expand Down
5 changes: 4 additions & 1 deletion examples/parallel-verify.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -20,16 +20,19 @@ nodes:
type: command
run: "npm run test --if-present"
timeoutSec: 600
expectNonEmpty: true

lint:
type: command
run: "npm run lint --if-present"
run: "npm run typecheck"
timeoutSec: 600
expectNonEmpty: true

typecheck:
type: command
run: "npm run typecheck --if-present"
timeoutSec: 600
expectNonEmpty: true

merge:
type: command
Expand Down
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@
"author": "Dat Nguyen",
"license": "MIT",
"engines": {
"node": ">=20"
"node": ">=22"
},
"scripts": {
"build": "tsup",
Expand Down
69 changes: 69 additions & 0 deletions src/adapters/claude.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,14 @@ const MAX_TURNS = `{"type":"result","subtype":"error_max_turns","result":"","tot
// Note `subtype` is "success" while `is_error` is true - trusting subtype alone
// makes an auth failure look like a completed agent run.
const AUTH_FAILURE = `{"type":"result","subtype":"success","is_error":true,"result":"Failed to authenticate: OAuth session expired and could not be refreshed","terminal_reason":"api_error","total_cost_usd":0,"num_turns":1}`;
// Captured from Claude Code 3.x: stdout is now a JSON array of message
// objects whose last element carries the run result.
const SUCCESS_ARRAY = `[
{"type":"system","subtype":"init","cwd":"/tmp/x","session_id":"SID"},
{"type":"assistant","message":{"role":"assistant","content":[{"type":"text","text":"BANANA"}]}},
{"type":"rate_limit_event","session_id":"SID"},
{"type":"result","subtype":"success","is_error":false,"num_turns":1,"stop_reason":"end_turn","total_cost_usd":0.2642395,"result":"BANANA","usage":{"input_tokens":4,"output_tokens":5}}
]`;

describe("buildClaudeArgs", () => {
it("builds the verified non-interactive argv", () => {
Expand All @@ -32,6 +40,23 @@ describe("buildClaudeArgs", () => {
"acceptEdits",
]);
});

it("appends --model when a model is given", () => {
expect(buildClaudeArgs("hi", undefined, "claude-opus-5")).toEqual([
"-p",
"hi",
"--output-format",
"json",
"--permission-mode",
"acceptEdits",
"--model",
"claude-opus-5",
]);
});

it("omits --model when no model is given", () => {
expect(buildClaudeArgs("hi", 8)).not.toContain("--model");
});
});

describe("parseClaudeJson", () => {
Expand Down Expand Up @@ -78,4 +103,48 @@ describe("parseClaudeJson", () => {
const out = parseClaudeJson(AUTH_FAILURE);
expect(out.text).toMatch(/Failed to authenticate/);
});

it("extracts text and cost from the array-form result element", () => {
const out = parseClaudeJson(SUCCESS_ARRAY);
expect(out.ok).toBe(true);
expect(out.text).toBe("BANANA");
expect(out.costUsd).toBe(0.2642395);
expect(out.error).toBeNull();
});

it("records the cost from the array even when the result reports an error", () => {
const out = parseClaudeJson(`[
{"type":"system","subtype":"init","cwd":"/tmp/x","session_id":"SID"},
{"type":"result","subtype":"error_max_turns","is_error":false,"total_cost_usd":0.11,"result":""}
]`);
expect(out.ok).toBe(false);
expect(out.text).toBe("");
expect(out.costUsd).toBe(0.11);
expect(out.error).toBe("claude run ended with subtype error_max_turns");
});

it("fails with a named error when the array carries no result element", () => {
const out = parseClaudeJson('[{"type":"system","subtype":"init"},{"type":"assistant"}]');
expect(out.ok).toBe(false);
expect(out.text).toBe("");
expect(out.costUsd).toBe(0);
expect(out.error).toBe("could not parse claude json output: no result element in array");
});

it("takes the last result element when an array carries more than one", () => {
const out = parseClaudeJson(`[
{"type":"system","subtype":"init"},
{"type":"result","subtype":"error_max_turns","is_error":false,"total_cost_usd":0.01,"result":""},
{"type":"result","subtype":"success","is_error":false,"total_cost_usd":0.99,"result":"second"}
]`);
expect(out.ok).toBe(true);
expect(out.text).toBe("second");
expect(out.costUsd).toBe(0.99);
});

it("keeps the whole array as raw", () => {
const out = parseClaudeJson(SUCCESS_ARRAY);
expect(Array.isArray(out.raw)).toBe(true);
expect((out.raw as unknown[]).length).toBe(4);
});
});
51 changes: 38 additions & 13 deletions src/adapters/claude.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,15 @@ import type { Adapter, AdapterInput, AdapterOutput } from "./types.js";
*
* stdout is a single JSON object with `subtype`, `result`, `session_id`,
* `num_turns` and `total_cost_usd`.
*
* Claude Code 3.x emits a JSON *array* of message objects whose last
* `type: "result"` element carries the run result, so the parser finds that
* element before reading `subtype` and `total_cost_usd`.
*/
export function buildClaudeArgs(prompt: string, maxTurns?: number): string[] {
export function buildClaudeArgs(prompt: string, maxTurns?: number, model?: string): string[] {
const args = ["-p", prompt, "--output-format", "json", "--permission-mode", "acceptEdits"];
if (maxTurns !== undefined) args.push("--max-turns", String(maxTurns));
if (model !== undefined) args.push("--model", model);
return args;
}

Expand All @@ -22,9 +27,9 @@ export function parseClaudeJson(stdout: string): AdapterOutput {
} catch {
return {
ok: false,
text: stdout,
text: "",
costUsd: 0,
raw: stdout,
raw: null,
error: `could not parse claude json output: ${stdout.slice(0, 200)}`,
};
}
Expand All @@ -33,30 +38,50 @@ export function parseClaudeJson(stdout: string): AdapterOutput {
return { ok: false, text: stdout, costUsd: 0, raw: parsed, error: "could not parse claude json output: not an object" };
}

const obj = parsed as Record<string, unknown>;
const cost = typeof obj.total_cost_usd === "number" ? obj.total_cost_usd : 0;
const text = typeof obj.result === "string" ? obj.result : "";
const subtype = typeof obj.subtype === "string" ? obj.subtype : "unknown";
// Claude Code 3.x emits an array of messages; the run result is the last
// element with `type: "result"`. The legacy single-object form is `result`
// itself.
let result: Record<string, unknown>;
if (Array.isArray(parsed)) {
const items = parsed as unknown[];
let found: Record<string, unknown> | undefined;
for (const item of items) {
if (typeof item === "object" && item !== null && (item as Record<string, unknown>).type === "result") {
found = item as Record<string, unknown>;
}
}
if (found === undefined) {
return { ok: false, text: "", costUsd: 0, raw: parsed, error: "could not parse claude json output: no result element in array" };
}
result = found;
} else {
result = parsed as Record<string, unknown>;
}

// Cost is harvested even on failure - budget accounting depends on it.
const costUsd = typeof result.total_cost_usd === "number" ? result.total_cost_usd : 0;
const text = typeof result.result === "string" ? result.result : "";

// Claude Code can report `subtype: "success"` while `is_error` is true - an
// expired OAuth session comes back exactly that way. Trusting subtype alone
// makes an auth failure look like a completed agent run, so check both.
if (obj.is_error === true) {
const reason = typeof obj.terminal_reason === "string" ? obj.terminal_reason : "is_error";
if (result.is_error === true) {
const reason = typeof result.subtype === "string" ? result.subtype : "is_error";
return {
ok: false,
text,
costUsd: cost,
costUsd,
raw: parsed,
error: `claude run reported is_error (${reason}): ${text || "no message"}`,
};
}

const subtype = typeof result.subtype === "string" ? result.subtype : "unknown";
if (subtype !== "success") {
return { ok: false, text, costUsd: cost, raw: parsed, error: `claude run ended with subtype ${subtype}` };
return { ok: false, text, costUsd, raw: parsed, error: `claude run ended with subtype ${subtype}` };
}

return { ok: true, text, costUsd: cost, raw: parsed, error: null };
return { ok: true, text, costUsd, raw: parsed, error: null };
}

export class ClaudeAdapter implements Adapter {
Expand All @@ -65,7 +90,7 @@ export class ClaudeAdapter implements Adapter {
constructor(private readonly bin = "claude") {}

async run(input: AdapterInput): Promise<AdapterOutput> {
const args = buildClaudeArgs(input.prompt, input.maxTurns);
const args = buildClaudeArgs(input.prompt, input.maxTurns, input.model);
const result = await execa(this.bin, args, {
cwd: input.cwd,
timeout: input.timeoutSec * 1000,
Expand Down
10 changes: 10 additions & 0 deletions src/adapters/codex.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,16 @@ describe("buildCodexArgs", () => {
it("keeps read-only as the default policy", () => {
expect(buildCodexArgs("x", "/repo")).toEqual(buildCodexArgs("x", "/repo", "read-only"));
});

it("appends --model when a model is given", () => {
const args = buildCodexArgs("x", "/repo", "read-only", "gpt-5.6-sol");
expect(args).toContain("--model");
expect(args[args.indexOf("--model") + 1]).toBe("gpt-5.6-sol");
});

it("omits --model when no model is given", () => {
expect(buildCodexArgs("x", "/repo")).not.toContain("--model");
});
});

describe("parseCodexJsonl", () => {
Expand Down
12 changes: 9 additions & 3 deletions src/adapters/codex.ts
Original file line number Diff line number Diff line change
Expand Up @@ -29,10 +29,16 @@ import type { Adapter, AdapterInput, AdapterOutput } from "./types.js";
*/
export type CodexSandbox = "read-only" | "workspace-write" | "bypass";

export function buildCodexArgs(prompt: string, cwd: string, sandbox: CodexSandbox = "read-only"): string[] {
export function buildCodexArgs(
prompt: string,
cwd: string,
sandbox: CodexSandbox = "read-only",
model?: string,
): string[] {
const policy =
sandbox === "bypass" ? ["--dangerously-bypass-approvals-and-sandbox"] : ["--sandbox", sandbox];
return ["exec", prompt, "--json", "--skip-git-repo-check", ...policy, "-C", cwd];
const chosen = model === undefined ? [] : ["--model", model];
return ["exec", prompt, "--json", "--skip-git-repo-check", ...policy, ...chosen, "-C", cwd];
}

/** Pull agent message text out of the several event shapes codex has shipped. */
Expand Down Expand Up @@ -128,7 +134,7 @@ export class CodexAdapter implements Adapter {
) {}

async run(input: AdapterInput): Promise<AdapterOutput> {
const result = await execa(this.bin, buildCodexArgs(input.prompt, input.cwd, this.sandbox), {
const result = await execa(this.bin, buildCodexArgs(input.prompt, input.cwd, this.sandbox, input.model), {
cwd: input.cwd,
timeout: input.timeoutSec * 1000,
reject: false,
Expand Down
Loading
Loading