Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 2 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,8 +62,8 @@ judgments (not lab benchmarks):
Runs on macOS and Linux (Ubuntu and common CI/test distros); Windows is the
exception.

**Measured agent surface** (same P2 fixture flow, Docker ARM64, 17 Jul 2026 —
point-in-time): Headless warm **147** est. tokens vs Selenium **410** /
**Measured agent surface** (same P2 fixture flow, Docker ARM64, 27 Aug 2026 —
point-in-time): Headless warm **174** est. tokens vs Selenium **410** /
Puppeteer **499**. Full method and limits:
[BENCHMARK.md](apps/headless/docs/BENCHMARK.md).

Expand Down Expand Up @@ -236,7 +236,6 @@ published GHCR image under Docker Desktop:
docker run --shm-size=1g ghcr.io/lockintime/headless:latest headless --version
```


### npm / npx

JavaScript-based agent harnesses can run the verified launcher without a
Expand Down
20 changes: 10 additions & 10 deletions apps/headless/docs/BENCHMARK.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,26 +5,26 @@ record through the `Continue` transition, and save a final screenshot.

## Current snapshot

Five fresh containers were run for each case on 12 August 2026, on Apple
Five fresh containers were run for each case on 27 August 2026, on Apple
Silicon with Docker Linux ARM64. The table reports the median of each metric.
These remain point-in-time measurements; repeat the benchmark before using
them to compare a change. The generated
[`results.json`](../../../packages/benchmark-results/results.json) preserves
all 20 raw samples and the aggregation provenance.

| Workflow | Estimated tokens | Wall time | CPU time | Peak memory |
| --- | ---: | ---: | ---: | ---: |
| Headless, cold | 218 | 3,484 ms | 2,028 ms | 368 MiB |
| Headless, warm | 174 | 3,248 ms | 1,379 ms | 366 MiB |
| Selenium with Python | 410 | 2,880 ms | 2,010 ms | 363 MiB |
| Puppeteer | 499 | 2,402 ms | 1,860 ms | 358 MiB |
| Workflow | Estimated tokens | Wall time | CPU time | Peak memory |
| -------------------- | ---------------: | --------: | -------: | ----------: |
| Headless, cold | 218 | 3,413 ms | 1,794 ms | 381 MiB |
| Headless, warm | 174 | 3,239 ms | 1,218 ms | 379 MiB |
| Selenium with Python | 410 | 2,828 ms | 1,788 ms | 378 MiB |
| Puppeteer | 499 | 2,400 ms | 1,849 ms | 367 MiB |

Estimated tokens are `ceil(workflow source bytes / 4)`. They compare the agent
workflow surface, not billed LLM tokens, tool schemas, prompts, or responses.

Headless has the smallest measured agent surface: the warm workflow uses about
58% fewer estimated tokens than Selenium and 65% fewer than Puppeteer. Its
median CPU time is about 31% lower than Selenium and 26% lower than Puppeteer.
median CPU time is about 32% lower than Selenium and 34% lower than Puppeteer.
Puppeteer is fastest and has the lowest median peak memory; Headless does not
lead those dimensions. The reusable P2 flow command reduces orchestration work
for real agent-driven repeats, but this benchmark retains the comparable
Expand All @@ -38,9 +38,9 @@ lookup.

## Method

Each workflow uses Chromium 151 and FFmpeg 5.1 to produce the same two
Each workflow uses Chromium 151.0.7922.173 and FFmpeg 5.1.9 to produce the same two
artifacts: an MP4 that tours both pages and a final viewport PNG. Selenium 4.8.3
uses ChromeDriver 151; Puppeteer Core is 22.15.0. All waits use page load or an
uses ChromeDriver 151.0.7922.173; Puppeteer Core is 22.15.0. All waits use page load or an
explicit URL condition. Every measured run gets a fresh container. The warm
Headless case starts its host and session before timing; the cold case
includes them.
Expand Down
2 changes: 2 additions & 0 deletions apps/web/lib/repository-content.d.mts
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
export type BenchmarkWorkflow = {
case: string;
label: string;
workflow: string;
name: string;
variant: string;
Expand All @@ -19,6 +20,7 @@ export type BenchmarkWorkflow = {

export type BenchmarkContent = {
sectionLabel: string;
methodDate: string;
headline: string;
summary: string;
proofs: Array<{
Expand Down
53 changes: 53 additions & 0 deletions apps/web/lib/repository-content.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -165,12 +165,19 @@ export function loadBenchmarkContent() {
year: "numeric",
timeZone: "UTC",
}).format(generatedAt);
const methodDate = new Intl.DateTimeFormat("en-GB", {
day: "numeric",
month: "long",
year: "numeric",
timeZone: "UTC",
}).format(generatedAt);

const workflows = BENCHMARK_CASES.map((caseName) => {
const entry = cases.get(caseName);
const presentation = PRESENTATION[caseName];
return {
case: caseName,
label: entry.label,
workflow: entry.label.replace(",", ""),
...presentation,
tokens: entry.median.estimatedTokens,
Expand All @@ -189,6 +196,7 @@ export function loadBenchmarkContent() {

benchmarkCache = {
sectionLabel: `P2 benchmark / ${date}`,
methodDate,
headline:
warm.estimatedTokens ===
Math.min(...workflows.map((workflow) => workflow.tokens))
Expand Down Expand Up @@ -431,6 +439,51 @@ export function loadDocumentationContent() {
export function validateRepositoryContent() {
const benchmark = loadBenchmarkContent();
const documentation = loadDocumentationContent();
const readme = readRepositoryFile("README.md");
const method = readRepositoryFile("apps/headless/docs/BENCHMARK.md");
const measuredClaim = readme.match(
/\*\*Measured agent surface\*\*[\s\S]*?Docker ARM64, ([0-9]{1,2} [A-Z][a-z]{2} [0-9]{4})[\s\S]*?Headless warm \*\*(\d+)\*\* est\. tokens vs Selenium \*\*(\d+)\*\* \/\s*Puppeteer \*\*(\d+)\*\*/,
);
if (!measuredClaim) fail("README measured agent-surface claim is missing");
const workflows = new Map(
benchmark.workflows.map((workflow) => [workflow.case, workflow]),
);
const expectedClaim = [
benchmark.sectionLabel.replace("P2 benchmark / ", ""),
String(workflows.get("headless-warm").tokens),
String(workflows.get("selenium").tokens),
String(workflows.get("puppeteer").tokens),
];
if (
JSON.stringify(measuredClaim.slice(1)) !== JSON.stringify(expectedClaim)
) {
fail(
"README measured agent-surface claim does not match generated results",
);
}
if (!method.includes(`for each case on ${benchmark.methodDate},`)) {
fail("benchmark method date does not match generated results");
}
for (const workflow of benchmark.workflows) {
const row = method
.split("\n")
.find((line) => line.startsWith(`| ${workflow.label}`));
if (!row) fail(`benchmark method is missing row: ${workflow.label}`);
const cells = row
.split("|")
.slice(1, -1)
.map((cell) => cell.trim());
const expectedCells = [
workflow.label,
workflow.tokens.toLocaleString("en-US"),
`${workflow.wallMs.toLocaleString("en-US")} ms`,
`${workflow.cpuMs.toLocaleString("en-US")} ms`,
`${workflow.memoryMiB.toLocaleString("en-US")} MiB`,
];
if (JSON.stringify(cells) !== JSON.stringify(expectedCells)) {
fail(`benchmark method row is stale: ${workflow.label}`);
}
}
return {
benchmarkCases: benchmark.workflows.length,
commandGroups: documentation.commandGroups.length,
Expand Down
56 changes: 29 additions & 27 deletions docs/ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,12 +81,13 @@ of them:
Honest snapshot, so newcomers know what is real:

**Working and verified**

- 39-verb JSON protocol over a `0600` Unix socket with peer-UID checks
(`apps/headless/Sources/HeadlessProtocol/`).
- Two engines behind one CLI: macOS `WKWebView` app, Linux sandboxed Chromium
over the DevTools fd-3/4 pipe (no debug port).
- Progressive context pruning (`inspect --context
summary|outline|text|actions|full`, `--task`, `--within @rN`, `--budget`)
summary|outline|text|actions|full`, `--task`, `--within @rN`, `--budget`)
with a measured **94.5 % token reduction** on the 120-section fixture.
- Evidence capture: PNG/JPG/PDF screenshots, viewport/section series, MP4/MOV/
WebM/GIF recordings, visual diffs, flows, QA reports.
Expand All @@ -107,6 +108,7 @@ Honest snapshot, so newcomers know what is real:
command reference, and a Docker sandbox wrapper.

**Not yet real**

- ~~No CI on pull requests or `main`~~ — PR CI landed (`.github/workflows/ci.yml`,
backlog §D1/§D3). Correctness fixes in §A are still outstanding, and the
macOS E2E is nightly/label-gated rather than a per-PR gate.
Expand All @@ -121,7 +123,7 @@ Honest snapshot, so newcomers know what is real:
three places each and will drift; the site has no deploy pipeline.
- Windows is not supported.
- A list of real code defects (thread-safety on shutdown, oversized `qa
report` responses, `@eN` ref invalidation surprises, host code duplication)
report` responses, `@eN` ref invalidation surprises, host code duplication)
— all catalogued in the [improvements backlog](roadmap/improvements-backlog.md).

---
Expand Down Expand Up @@ -152,23 +154,23 @@ codebase's existing behavior because they are the product:

## 4. Platform support

| Platform | Status today | Target |
| --- | --- | --- |
| **macOS 13+** (Apple Silicon) | Universal signed release pipeline awaiting tag | Signed + notarized, Homebrew, universal binary |
| **macOS Intel** | Universal release pipeline awaiting tag | Universal binary in release CI |
| **Linux** (Debian/Ubuntu, non-Snap Chromium) | Verified installer and GHCR pipeline awaiting tag | curl installer, published Docker image (GHCR), apt guidance |
| **Linux other distros** | Works where a non-Snap Chromium exists | Documented candidate paths per distro family |
| **Windows 10/11** | Not supported | **Stretch goal (Phase W)** — Chromium host ported; not required for "done". See [architecture decisions §6](roadmap/architecture-decisions.md). |
| **Any OS via Docker** | Works (build locally) | `docker run ghcr.io/…/headless` one-liner, including as the practical Windows answer (WSL2/Docker Desktop) until Phase W lands |
| Platform | Status today | Target |
| -------------------------------------------- | ------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- |
| **macOS 13+** (Apple Silicon) | Universal signed release pipeline awaiting tag | Signed + notarized, Homebrew, universal binary |
| **macOS Intel** | Universal release pipeline awaiting tag | Universal binary in release CI |
| **Linux** (Debian/Ubuntu, non-Snap Chromium) | Verified installer and GHCR pipeline awaiting tag | curl installer, published Docker image (GHCR), apt guidance |
| **Linux other distros** | Works where a non-Snap Chromium exists | Documented candidate paths per distro family |
| **Windows 10/11** | Not supported | **Stretch goal (Phase W)** — Chromium host ported; not required for "done". See [architecture decisions §6](roadmap/architecture-decisions.md). |
| **Any OS via Docker** | Works (build locally) | `docker run ghcr.io/…/headless` one-liner, including as the practical Windows answer (WSL2/Docker Desktop) until Phase W lands |

Agent-harness support (the other axis of "platform"):

| Harness | Today | Target |
| --- | --- | --- |
| MCP clients (Claude Code, Cursor, Codex, …) | `headless-mcp` stdio server | unchanged core; per-client setup docs + `.mcp.json` example |
| Claude Code | manual skill pointer | root `CLAUDE.md` + discoverable skill |
| Codex / OpenCode / Amp / others | `AGENTS.md` convention | root `AGENTS.md` (done in this change) |
| Plain shell agents | CLI + `headless capabilities` | unchanged; capabilities doc kept machine-checked |
| Harness | Today | Target |
| ------------------------------------------- | ----------------------------- | ----------------------------------------------------------- |
| MCP clients (Claude Code, Cursor, Codex, …) | `headless-mcp` stdio server | unchanged core; per-client setup docs + `.mcp.json` example |
| Claude Code | manual skill pointer | root `CLAUDE.md` + discoverable skill |
| Codex / OpenCode / Amp / others | `AGENTS.md` convention | root `AGENTS.md` (done in this change) |
| Plain shell agents | CLI + `headless capabilities` | unchanged; capabilities doc kept machine-checked |

---

Expand All @@ -184,7 +186,7 @@ every item carries file/line detail.
Write down what the product is, what must not change, and everything that is
wrong. Add agent rule files so every harness can work on this repo.

*Exit test:* this document set is merged; `AGENTS.md`/`CLAUDE.md` exist at
_Exit test:_ this document set is merged; `AGENTS.md`/`CLAUDE.md` exist at
root.

### Phase 1 — Trust the build (CI + correctness)
Expand All @@ -202,7 +204,7 @@ known races exist.
`ELEMENT_NOT_FOUND` (backlog §A7).
- Web app gets `next build` + eslint in the same CI (backlog §D3).

*Exit test:* a PR cannot merge with failing tests; the E2E suites pass on CI
_Exit test:_ a PR cannot merge with failing tests; the E2E suites pass on CI
runners, not just laptops; the known-crash list in the backlog §A is empty.

### Phase 2 — One host, written once (deduplication refactor)
Expand All @@ -216,7 +218,7 @@ errors end-to-end (backlog §B).
This is also the **prerequisite for Windows**: after it, a Windows port is one
new engine + one new transport backend, not a third copy of everything.

*Exit test:* adding a hypothetical new verb touches one dispatch site; the
_Exit test:_ adding a hypothetical new verb touches one dispatch site; the
error-code mapping is a typed enum, not string matching; the capability matrix
(clipboard, network mock, PDF fidelity, …) is generated from code and asserted
in tests.
Expand All @@ -237,7 +239,7 @@ in tests.
- **npm wrapper** (`npx headless-browser` style) that downloads the platform
binary — the cheapest path into JS-centric agent stacks. (Backlog §E.)

*Exit test:* a new user on a clean macOS or Linux machine gets from zero to
_Exit test:_ a new user on a clean macOS or Linux machine gets from zero to
`headless start` + first `visit` in under two minutes without touching a
compiler, and without a Gatekeeper override on macOS.

Expand All @@ -258,7 +260,7 @@ Make Headless the obvious choice inside every harness:
implemented so `fill` can type literal `--json`; response pagination for
large reports remains (backlog §G).

*Exit test:* a fresh Claude Code, Cursor, and Codex session can each discover
_Exit test:_ a fresh Claude Code, Cursor, and Codex session can each discover
and drive Headless with zero manual prompting beyond repo checkout.

### Phase 5 — Website and docs as a product surface
Expand All @@ -269,11 +271,11 @@ and drive Headless with zero manual prompting beyond repo checkout.
generated from the CLI) (backlog §F).
- Add the missing pages: install, security model, MCP setup, command
reference, changelog, platform matrix.
- Re-run the benchmark with the task-aware inspect flow before quoting any
token number — the site currently markets `--task` while quoting pre-`--task`
measurements (backlog §F4).
- Keep the generated benchmark current with the task-aware inspect flow before
quoting any token number. The site imports the generated medians and carries
their point-in-time caveat (backlog §F4).

*Exit test:* site deploys on merge; every number and command on it is
_Exit test:_ site deploys on merge; every number and command on it is
generated or test-asserted; the "stale benchmark" warning is gone because the
benchmark is current.

Expand All @@ -295,7 +297,7 @@ Phase 2's engine/transport split. Shape of the work (detailed in
- Until then, the documented Windows answer is Docker Desktop/WSL2 with the
published image (Phase 3 dependency).

*Exit test:* the Linux E2E scenario passes on a Windows runner with the
_Exit test:_ the Linux E2E scenario passes on a Windows runner with the
Chromium engine; `winget install headless` works.

---
Expand All @@ -315,7 +317,7 @@ of the following hold:
returns an explicit capability error; the matrix is generated and asserted.
4. **Agent-native:** any MCP-capable harness and any AGENTS.md-reading harness
can drive Headless from a fresh checkout with no human glue; `headless
capabilities` is machine-accurate.
capabilities` is machine-accurate.
5. **Evidence current:** benchmark re-run on the shipping workflow; QA
evidence regenerated for the release; website deployed and drift-free.
6. **Docs:** this roadmap's Phases 1–5 checked off, with Windows either
Expand Down
6 changes: 4 additions & 2 deletions docs/roadmap/improvements-backlog.md
Original file line number Diff line number Diff line change
Expand Up @@ -431,9 +431,11 @@ Owner-decided scope: package managers, no hosted service.
matrix, the README's comparison table (strongest positioning content, not
on site). Plus `robots.txt`, `sitemap`, OG metadata, per-page `metadata`,
404 page.
- **F4. Stale-benchmark honesty:** ([#50](https://github.com/LockInTime/headless/issues/50)) site quotes pre-`--task` numbers while
- **F4. Stale-benchmark honesty:** ([#50](https://github.com/LockInTime/headless/issues/50)) ~~site quotes pre-`--task` numbers while
marketing `--task`, and drops BENCHMARK.md's re-run warning; headline says
"Measured, not claimed." Fix by refresh (D5) or by carrying the caveat.
"Measured, not claimed." Fix by refresh (D5) or by carrying the caveat.~~
**Done:** a new five-repeat, task-aware snapshot was generated on 27 August 2026. The website imports its medians and date, preserves the point-in-time
warning, and web lint rejects README claims that drift from generated data.
- **F5. Dead weight:** ([#51](https://github.com/LockInTime/headless/issues/51)) `components/ui/side-rays.tsx` + `ogl` dep (unused),
`public/scan-dashboard.png` (unreferenced), leftover shadcn `.dark` block in
`globals.css`, 8 unused button variants; two WebGL stacks (~700 KB) for
Expand Down
Loading