phase5(stage0): orchestrator + reporter + pre-run engineering

AGENTS.md

@@ -33,7 +33,7 @@ data/  results/             # gitignored (benchmarks, JSONL traces)
 ```powershell
 .venv\Scripts\Activate.ps1          # activate venv
 pip install -e ".[dev]"             # install (first run downloads torch - slow)
-pytest -q                           # run tests (expect 130 passing; data-dependent tests skip without clones)
+pytest -q                           # run tests (expect 158 passing; data-dependent tests skip without clones)
 python scripts/check_parser.py      # parser spot-check (offline)
 python scripts/pilot_score_dist.py  # scorer pilot (offline)
 powershell -ExecutionPolicy Bypass -File scripts/clone_benchmarks.ps1

CHANGELOG.md

@@ -8,6 +8,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ## [Unreleased]
 
 ### Added
+- Phase 5 Stage 0 (pre-run engineering): runners now gate **every** proposed call and record per-call decision/S/latency in `gate_events`; ablation modes A1–A3 (`--ablation semantic-only|rule-only|raw-request`) wired through the middleware with trace tags; exact McNemar + continuity variant and seeded bootstrap CI in `eval/stats.py`; τ×δ sweep grid with escalate share (`sweep_grid`); token/cost tracking (`LLMClient.usage`, `eval/cost.py`); `scripts/run_phase5.py` orchestrator (resume, `--jobs`, retries, `--max-calls`, freeze lock/manifest, `--check`); `scripts/report_phase5.py` reporter (headline, breakdowns, stats, Pareto, latency, utility proxies, cost); smoke runs verified (5 cases/condition). 158 tests green.
 - B2 MCPTox contract subset (65 contracts: filesystem path-scope, mail, GitHub, process/shell, DB/Prisma, misc) + best-effort world-state seeding from the trusted request (`world_state.seed_from_request`); coverage frozen at 8.1% distinct / 37.2% availability-weighted (`configs/b2_coverage.json`). Gated reruns: B2 blocks only the out-of-scope `write_file` on MCPTox — uncontracted poisoned registrations are invisible to per-tool contracts (`docs/experiments/03_b2_toolgate.md`).
 - Error taxonomy + rule fixes (`docs/experiments/04_gate2_gated_eval.md`): run-time contracts are now logged into gate traces (`parser_backend` + `contract` context), `scripts/error_taxonomy.py` classifies residual attacks; rules gained destructive-file keywords, file path-scope enforcement, and a shell-operator guard for command-like parameters. v2 at τ=0.75: all-benchmark ASR 0.44 → 0.30, MCPTox attack-influenced 11 → 8, pilot FPR unchanged at 4%; 117 tests green.
 - Threshold sweep / Pareto: `eval/sweep.py::sweep_cases` re-decides cached traces at every τ (case-level, escalate counts as block) + `scripts/sweep_gated.py` CLI joining ungated ground truth, gated traces, and the labeled pilot FPR; knee at τ=0.75 (pilot FPR 4%, InjecAgent ASR 0.17, MCPTox 0.53) — `docs/experiments/04_gate2_gated_eval.md`.

roadmap.md

@@ -84,14 +84,14 @@
 - Budget: ≈20.7k agent calls + ≈13.8k parser calls (ours + ablations) on `gpt-4o-mini` (est. $15–35, ~4–8 h with 3-way file-level parallelism).
 
 **Stage 0 — pre-run engineering (must land before the freeze):**
-- [ ] Gate every proposed call (user + attacker), not only attack-relevant ones — per-call decision/latency in `gate_events` (FPR proxy + full latency samples)
-- [ ] Wire ablations A1–A3 (`--ablation semantic-only|rule-only|raw-request`), ablation tag in trace metadata
-- [ ] Exact McNemar (+ continuity-corrected variant) and seeded bootstrap CI (10k) in `eval/stats.py`, with tests
-- [ ] τ×δ sweep grid (τ ∈ {0.40–0.80 step 0.05} × δ ∈ {0.05, 0.10, 0.15}) + escalate share per cell
-- [ ] Token/cost logging captured per run
-- [ ] `scripts/run_phase5.py` orchestrator: resume, `--jobs` parallelism, retry/backoff, `--max-calls` guard, lock/manifest, `--check` integrity mode
-- [ ] `scripts/report_phase5.py` reporter: tables, breakdowns (risk/tool/split/setting), stratification, CIs/McNemar, τ×δ Pareto, latency/utility, cost-vs-ASR
-- [ ] Smoke runs (`--limit 5`) green for every condition before the freeze
+- [x] Gate every proposed call (user + attacker), not only attack-relevant ones — per-call decision/latency in `gate_events` (FPR proxy + full latency samples)
+- [x] Wire ablations A1–A3 (`--ablation semantic-only|rule-only|raw-request`), ablation tag in trace metadata
+- [x] Exact McNemar (+ continuity-corrected variant) and seeded bootstrap CI (10k) in `eval/stats.py`, with tests
+- [x] τ×δ sweep grid (τ ∈ {0.40–0.80 step 0.05} × δ ∈ {0.05, 0.10, 0.15}) + escalate share per cell
+- [x] Token/cost logging captured per run
+- [x] `scripts/run_phase5.py` orchestrator: resume, `--jobs` parallelism, retry/backoff, `--max-calls` guard, lock/manifest, `--check` integrity mode
+- [x] `scripts/report_phase5.py` reporter: tables, breakdowns (risk/tool/split/setting), stratification, CIs/McNemar, τ×δ Pareto, latency/utility, cost-vs-ASR
+- [x] Smoke runs (5 cases per condition) green before the freeze — events/latency/cost/ablation paths verified
 
 **Stage 1 — freeze + full runs:**
 - [ ] Write `configs/phase5_lock.json` + `results/phase5/manifest.json` (requires clean tree)