mutagent-io · burak-cyb · Sep 4, 2026 · Sep 4, 2026
@@ -64,10 +64,15 @@ claude   →   *mutagent
 
 ## Status
 
-`showcase/` has its first entry — [`freecad-engineer`](showcase/freecad-engineer/), a
-self-verifying FreeCAD CAD agent, published with the spec it was built from, its build report,
-its evaluation runs and scorecard, and its diagnostics reports. `cookbooks/` is still a
-scaffold; recipes land there next.
+`showcase/` has three entries, each published with the spec it was built from, its build
+report, its evaluation runs and verdicts, and its diagnostics reports:
+
+- [`freecad-engineer`](showcase/freecad-engineer/) — a self-verifying FreeCAD CAD agent.
+- [`carlo`](showcase/carlo/) — an in-car voice-assistant agent for CAR-bench Track 1.
+- [`mutagent-brepsmith`](showcase/mutagent-brepsmith/) — a STEP CAD-editing agent with a
+  deterministic geometry verification loop, evaluated on CadGenBench.
+
+`cookbooks/` is still a scaffold; recipes land there next.
 
 ## Requirements
 

@@ -70,3 +70,5 @@ real inputs.
 | Agent | What it does | Stages covered | Last verified |
 |---|---|---|---|
 | [`freecad-engineer`](./freecad-engineer/) | Self-verifying mechanical-CAD agent (Claude Code + Opus 5, single-file definition) that writes parametric FreeCAD PartDesign scripts from natural-language part specs | SPEC · BUILD (verify) · EVALUATE · DIAGNOSE | 2026-08-19 |
+| [`carlo`](./carlo/) | In-car voice-assistant agent for CAR-bench Track 1 (gemini-3.5-flash on Vertex AI, TypeScript/bun, A2A 1.0 server): schema-validated tool calls with in-turn retry; Pass^3 0.605 vs 0.566 raw-model baseline on the 129-task train split | SPEC · BUILD · EVALUATE · DIAGNOSE · OPTIMIZE (1 cycle) | 2026-08-26 |
+| [`mutagent-brepsmith`](./mutagent-brepsmith/) | CAD STEP-editing agent (Claude Code + Sonnet-5/Opus-5, deterministic geometry verification loop) evaluated on the CadGenBench editing task: 0.6669 editing mean, 32/32 valid | SPEC · BUILD · EVALUATE · DIAGNOSE · OPTIMIZE (1 cycle) | 2026-08-28 |
@@ -0,0 +1,257 @@
+{
+  "generatedAt": "2026-08-26T17:03:42.955Z",
+  "backbone": "gemini-3.5-flash",
+  "prices": {
+    "input": 0.0000015,
+    "output": 0.000009,
+    "thinking": 0.000009,
+    "cacheRead": 1.5e-7
+  },
+  "runs": [
+    {
+      "label": "smoke/baseline",
+      "files": 3,
+      "records": 45,
+      "harnessUserCost": 0.031506700000000006,
+      "harnessAgentCost": 1.8474336000000002,
+      "carloTokens": null,
+      "carloCost": null,
+      "carloLlmCalls": 0,
+      "tracesAttributed": 0,
+      "tracesRedacted": 0,
+      "endMtimeMs": 1787645457429.151
+    },
+    {
+      "label": "smoke/carlo",
+      "files": 4,
+      "records": 60,
+      "harnessUserCost": 0.030117599999999994,
+      "harnessAgentCost": 0,
+      "carloTokens": null,
+      "carloCost": null,
+      "carloLlmCalls": 567,
+      "tracesAttributed": 60,
+      "tracesRedacted": 60,
+      "endMtimeMs": 1787646100784.1165
+    },
+    {
+      "label": "smoke/carlo-fix-goff",
+      "files": 3,
+      "records": 45,
+      "harnessUserCost": 0.045884999999999995,
+      "harnessAgentCost": 0,
+      "carloTokens": null,
+      "carloCost": null,
+      "carloLlmCalls": 605,
+      "tracesAttributed": 45,
+      "tracesRedacted": 45,
+      "endMtimeMs": 1787649367709.9058
+    },
+    {
+      "label": "smoke/carlo-fix-gon",
+      "files": 3,
+      "records": 45,
+      "harnessUserCost": 0.04728130000000001,
+      "harnessAgentCost": 0,
+      "carloTokens": null,
+      "carloCost": null,
+      "carloLlmCalls": 490,
+      "tracesAttributed": 45,
+      "tracesRedacted": 45,
+      "endMtimeMs": 1787649804969.282
+    },
+    {
+      "label": "smoke/carlo-v2-goff",
+      "files": 3,
+      "records": 45,
+      "harnessUserCost": 0.037640400000000004,
+      "harnessAgentCost": 0,
+      "carloTokens": null,
+      "carloCost": null,
+      "carloLlmCalls": 477,
+      "tracesAttributed": 45,
+      "tracesRedacted": 45,
+      "endMtimeMs": 1787650391722.7542
+    },
+    {
+      "label": "smoke/carlo-v2-gon",
+      "files": 3,
+      "records": 45,
+      "harnessUserCost": 0.039779599999999984,
+      "harnessAgentCost": 0,
+      "carloTokens": {
+        "prompt": 4575390,
+        "completion": 18074,
+        "thinking": 0,
+        "cached": 4313575
+      },
+      "carloCost": 1.20242475,
+      "carloLlmCalls": 452,
+      "tracesAttributed": 45,
+      "tracesRedacted": 0,
+      "endMtimeMs": 1787650767859.0488
+    },
+    {
+      "label": "smoke/carlo-v3-goff",
+      "files": 3,
+      "records": 45,
+      "harnessUserCost": 0.035305500000000004,
+      "harnessAgentCost": 0,
+      "carloTokens": {
+        "prompt": 4893533,
+        "completion": 22970,
+        "thinking": 106,
+        "cached": 4018574
+      },
+      "carloCost": 2.1229086,
+      "carloLlmCalls": 482,
+      "tracesAttributed": 45,
+      "tracesRedacted": 0,
+      "endMtimeMs": 1787659098645.6675
+    },
+    {
+      "label": "smoke/carlo-v3-gon",
+      "files": 3,
+      "records": 45,
+      "harnessUserCost": 0.030942900000000002,
+      "harnessAgentCost": 0,
+      "carloTokens": {
+        "prompt": 5100559,
+        "completion": 20661,
+        "thinking": 0,
+        "cached": 4778514
+      },
+      "carloCost": 1.3857936,
+      "carloLlmCalls": 504,
+      "tracesAttributed": 45,
+      "tracesRedacted": 0,
+      "endMtimeMs": 1787659507732.1235
+    },
+    {
+      "label": "train/baseline",
+      "files": 3,
+      "records": 480,
+      "harnessUserCost": 0.40869907000000016,
+      "harnessAgentCost": 73.31039790000007,
+      "carloTokens": {
+        "prompt": 78920518,
+        "completion": 563949,
+        "thinking": 364,
+        "cached": 65106855
+      },
+      "carloCost": 35.56533975,
+      "carloLlmCalls": 6771,
+      "tracesAttributed": 356,
+      "tracesRedacted": 0,
+      "endMtimeMs": 1787675479532.6914
+    },
+    {
+      "label": "train/carlo",
+      "files": 3,
+      "records": 387,
+      "harnessUserCost": 0.3455806999999998,
+      "harnessAgentCost": 0,
+      "carloTokens": {
+        "prompt": 7221593,
+        "completion": 47001,
+        "thinking": 0,
+        "cached": 6109160
+      },
+      "carloCost": 3.0080325,
+      "carloLlmCalls": 634,
+      "tracesAttributed": 31,
+      "tracesRedacted": 0,
+      "endMtimeMs": 1787675894787.1494
+    },
+    {
+      "label": "ablation-minimal",
+      "files": 3,
+      "records": 90,
+      "harnessUserCost": 0.06861930000000002,
+      "harnessAgentCost": 0,
+      "carloTokens": {
+        "prompt": 27650778,
+        "completion": 175210,
+        "thinking": 749820,
+        "cached": 22297508
+      },
+      "carloCost": 19.6998012,
+      "carloLlmCalls": 1840,
+      "tracesAttributed": 90,
+      "tracesRedacted": 0,
+      "endMtimeMs": 1787730709618.476
+    },
+    {
+      "label": "ablation-medium",
+      "files": 3,
+      "records": 90,
+      "harnessUserCost": 0.07323960000000002,
+      "harnessAgentCost": 0,
+      "carloTokens": {
+        "prompt": 21623294,
+        "completion": 142208,
+        "thinking": 484052,
+        "cached": 17452934
+      },
+      "carloCost": 14.509820099999999,
+      "carloLlmCalls": 1528,
+      "tracesAttributed": 90,
+      "tracesRedacted": 0,
+      "endMtimeMs": 1787733210667.1492
+    },
+    {
+      "label": "full-minimal",
+      "files": 3,
+      "records": 387,
+      "harnessUserCost": 0.32810332999999986,
+      "harnessAgentCost": 0,
+      "carloTokens": {
+        "prompt": 126009902,
+        "completion": 777628,
+        "thinking": 3830146,
+        "cached": 100423244
+      },
+      "carloCost": 94.9134396,
+      "carloLlmCalls": 8346,
+      "tracesAttributed": 387,
+      "tracesRedacted": 0,
+      "endMtimeMs": 1787745562059.0793
+    },
+    {
+      "label": "fidelity-baseline",
+      "files": 3,
+      "records": 90,
+      "harnessUserCost": 0.2618505,
+      "harnessAgentCost": 7.1159329499999995,
+      "carloTokens": null,
+      "carloCost": null,
+      "carloLlmCalls": 0,
+      "tracesAttributed": 0,
+      "tracesRedacted": 0,
+      "endMtimeMs": 1787761779539.6948
+    },
+    {
+      "label": "fidelity-minimal",
+      "files": 3,
+      "records": 90,
+      "harnessUserCost": 0.27161699999999994,
+      "harnessAgentCost": 0,
+      "carloTokens": {
+        "prompt": 29486098,
+        "completion": 182278,
+        "thinking": 663184,
+        "cached": 23416150
+      },
+      "carloCost": 20.2265025,
+      "carloLlmCalls": 1978,
+      "tracesAttributed": 90,
+      "tracesRedacted": 0,
+      "endMtimeMs": 1787763816486.1562
+    }
+  ],
+  "total": {
+    "harness": 84.32993295000006,
+    "carlo": 192.63406260000002,
+    "redactedTraces": 195
+  }
+}
@@ -0,0 +1,85 @@
+# CARlo ④ DIAGNOSE — smoke-run RCA (2026-08-25)
+
+Input: ③ EVALUATE GATE FAIL (`.mutagent/evaluator/smoke-2026-08-25/verdict.md`).
+Method: 4 parallel analyzers, deep-read of failing trajectories + CARlo per-turn traces +
+implicated source. All findings evidence-cited; confidence high throughout.
+
+## Findings
+
+### WIRE: parallel tool results replayed un-batched `{format-violation · tool-misuse · harness-side}`
+`buildContents` aggregates parallel CALLS into one model turn but replays each RESULT as its
+own single-part user turn (`vertex.ts:95-107` vs `:80-93`); Gemini requires all k
+functionResponse parts in ONE user turn → deterministic 400 that permanently poisons the
+transcript. Proof 1:1:1 — 5 parallel-call traces : 5 log 400s : 5 zero-reward records
+(base_0 ×2, disambiguation_0 ×3). Negative control: base_0 trial 1 passed when the model
+single-called. Trigger is sampling-dependent → all current pass rates are OPTIMISTIC.
+**Remedy (rank 1, low cost, high correctness):** buffer consecutive `role:"tool"` entries,
+flush as ONE user content at the next non-tool entry / end / before critique; keep both sides
+id-free; + response-parity invariant test + k≥2, duplicate-name, critique-ordering tests.
+
+### POLICY: compiler anchored on tokens the bench deletes `{missing-context · contract-mismatch · code-construct}`
+`policy.ts:20` requires literal `AUT-POL:`/`LLM-POL:` prefixes; `wiki.py:8-10` strips exactly
+those before delivery → 19 rules delivered, 0 parsed, on every task (`policyRules:0`).
+Test blind spot: the fixture is the RAW wiki, not the delivered form.
+**F2b (corollary):** even parsed, the trunk-door rule (POL:004) attaches to ZERO tools — it
+binds via a `REQUIRES_CONFIRMATION` description prefix CARlo never reads. Parser fix alone
+does NOT fix base_2.
+**Remedy (rank 1):** (a) make the namespace prefix optional in `POLICY_ID`; (b) capture
+`requiresConfirmation` from tool descriptions in `inventory.ts` and bind POL:004-class rules
+to those tools; (c) pipeline confirmation gate for such tools; (d) FAIL-LOUD when a non-empty
+policy text compiles to 0 rules; (e) delivered-format fixture (apply wiki.py's replaces).
+Blast radius: ALL 19 rules currently inert — expect more failures surfaced (good) on wider
+splits; historical CARlo numbers are a no-gate baseline.
+
+### AMBIGUITY: gate structurally unreachable for acting drafts `{wrong-output · gate-unreachable · code-construct}`
+`ambiguityFlagged := (no toolCalls && isQuestion(text))` (`pipeline.ts:173-174`) — an acting
+draft can NEVER be ambiguity-flagged: 0/130 tool-call drafts fired the gate. The verify
+backstop renders tool NAMES only (`pipeline.ts:507`) — 44/52 verify passes were no-ops; in
+base_6 the verify pass itself AUTHORED the spurious `{NONE, off}` write. `riskScore` is
+class-level so verify cannot veto a bad VALUE. disambiguation_2 trial 3 passed purely because
+the model happened to draft a question — gates contributed nothing.
+**Remedies (order):** R3a trace drafted/verified ARGUMENTS (enabler, zero risk, land first);
+R3b argument-groundedness gate — every scalar arg of a state-changing call must be
+user-stated, tool-read, or documented default, else suppress + ask (behind a config flag,
+measure before default; beware false suppression on base_4/base_8-style synonyms);
+R3c render args into the verify critique + independent ambiguity signal; R3d same-tool/
+different-args disagreement ⇒ suppress + ask. Prompt hardening (R3e) is low-rank: the prose
+rule already exists and was ignored 5/5.
+
+### FEASIBILITY: draft-reactive and tool-call-only `{wrong-output · tool-schema-hole · code-construct}`
+The gate only asks "did the model name something absent?" — never "can the declared surface
+satisfy the request?" (hallucination_6: param deleted from schema+required, so `{on:true}` is
+schema-VALID but the env still requires it → TypeError ×3 with `findings: []`) and never
+"does the PROSE claim a capability we lack?" (hallucination_0: CARlo OFFERED the removed
+sunshade tool and in trial 1 CLAIMED to have executed it — fabrication travelled on the text
+channel the gate never inspects; baseline merely stayed silent — CARlo is WORSE here).
+hallucination_0 verdict: **CARlo-fixable** — trigger is a pure set-difference at turn 0, and
+`limitStatement` already emits the exact acknowledgment the grader wants; it's just unreachable.
+**Remedies:** R4a getter/setter field-parity check → `unexpressible-attribute` (non-repairable
+→ existing honest-limit path); R4b prose-channel feasibility — (proactive) diff user-named
+capabilities vs inventory at ingest → force acknowledgment via limitStatement; (defensive)
+reject text asserting/offering/claiming-executed an absent tool (key on tool-name tokens, not
+verbs); R4c description-vs-properties coherence flag (conservative). Expected: hallucination_6
+0→1 (prose already correct), hallucination_0 high-probability flip.
+
+## Recommended remedy bundle (apply order)
+
+1. ** wire batching** — mechanical, zero regression risk (k=1 byte-identical), unblocks 5 records.
+2. **R3a trace arguments** — zero-risk enabler for everything below.
+3. ** policy compiler + confirmation gate + fail-loud** — restores the whole policy dimension.
+4. **R4a/R4b feasibility extensions** — wins hallucination_6, very likely hallucination_0.
+5. **R3b/R3c/R3d groundedness + verify-args** — behind `CARLO_GROUNDEDNESS` flag; measure on
+   the smoke slice before making default.
+
+Then RE-RUN the smoke slice (same 15 tasks × 3) and RE-BASELINE — historical CARlo numbers
+were produced with an inert policy gate and an optimistic wire.
+
+## Cross-cutting lesson
+One sentence: CARlo's gates were tested against the inputs they expected, not the inputs the
+benchmark actually delivers (stripped policy anchors · mutilated schemas · parallel calls ·
+prose-channel claims). Every fixture derived from RAW sources must be regenerated through the
+bench's own delivery transforms.
+
+## hallucination_0 note
+Fails BOTH arms 3/3; not backbone-hard; see. Secondary over-refusal defect (rain) noted
+but not score-bearing.