[rgarcia]

Summary

Sets up kernel/benchmarks, a home for benchmarks of everyday web tasks across agents, browser tools, and browser configurations. It uses Harbor as the runtime and Harbor's Kernel environment for browsers.

The pieces:

• Suite (task set): ClawBench, pinned to an upstream TIGER-AI-Lab/ClawBench commit.
• Toolset (what the agent drives the browser with): kernel-mcp-local, which builds kernel/kernel-mcp-server at any ref.
• Arm (agent + model + toolset + browser config): codex-kernel-mcp, codex-kernel-mcp-site-config, claude-kernel-mcp.
• Configurator (optional per-task browser settings): site-config.
  1. An LLM infers the task's primary site.
  2. Kernel's config registry looks it up.
  3. An LLM translates the registry's proxy recipe into a Kernel proxy, created once per recipe and reused by name.
  4. The registry's guidance becomes "Site guidance" in the instruction.
• Benchmark (a suite plus its arms): kernel-mcp-clawbench, runnable with gh workflow run kernel-mcp-clawbench.yml.

How it works

• bench run generates tasks with the suite, then for each arm copies them, resolves browser settings, and applies the toolset. It then runs harbor run for every arm in parallel.
• jobs/run-<id>.json is written before Harbor starts and updated as each arm finishes. It records each arm's job directory, exit status, and per-task browser settings, steering, and provenance. A run that hits GitHub's six-hour limit still gets a report.
• Per-task browser settings go to environment/kernel-config.json. harness.environment:KernelTaskEnvironment passes them to the Kernel provider as browser_create_overrides.
• Harbor is pinned to kernel/harbor@5e85bc1a, which adds full create params and overrides to the Kernel provider.
• ClawBench uses the browser Harbor creates for the trial instead of creating and deleting its own. ClawBench's tool wiring is removed, so toolsets supply the tools.
• kernel-mcp-local changes three things:
  • The image entrypoint starts Redis and the server, so tasks need no setup patching.
  • The agent gets a stdio MCP entry and instructions to read its session ID from /run/kernel-mcp/session-id.
  • The verifier records kernel_mcp_valid.
• report/ is the TypeScript summary and Braintrust publisher moved from kernel-mcp-server. The candidate/baseline comparison and the browser-lifecycle span were removed.
• Trials are retried on transient Kernel API and agent-install errors only. A Docker build failure is no longer retried five times on new paid VMs.

Validation

• ruff check, ruff format --check, and pytest (6 tests) pass. The report suite passes bun test (19), tsc, and prettier.
• Local live run with the same steps as the workflow (codex, 2 tasks × 2 arms, real Kernel browsers): 4/4 trials graded, 0 infrastructure errors, kernel_mcp_valid = 1 on all four, and no browsers left over.
  • v2-1134 (Red Cross): both arms passed. The registry has no recommendation for that site, so the configurator kept the stealth default.
  • v2-091 (Indeed): the stealth-only arm was stopped by Cloudflare's "Additional Verification Required" page. The site-config arm used a US ISP proxy plus the registry's guidance, reached the application, and stopped at phone verification.
• After the review fixes, v2-1134 on both codex arms: 2/2 reward 1.0, kernel_mcp_valid 1.0, no exceptions.
• An adversarial review by a separate agent covered simplicity, plan adherence, correctness, and README readiness; its must-fix and should-fix findings are addressed.
• The full 129-task suite has not been run from this repo yet.