Constructed-adversarial accuracy benchmark (blocked-recall = 1.0)

[pengfei-threemoonslab]

Summary

The first published, non-circular verdict-accuracy result — the "verdict provably right" north-star artifact. It's the half real-history mining can't supply: W25 showed real merged PRs almost never contain a must_block capability change (9 decided / 241 PRs), so the benchmark's positives come from the repo's bundled fixtures.

The key property is non-circularity: each fixture was built to be a specific case, so its label is the fixture's documented design intent (external ground truth) — not a post-hoc opinion about what the engine returned. Scoring the engine's verdict against those definitional labels is a legitimate measurement, not grading-its-own-homework.

Result (7-case curated set)

label \ verdict	allow	review	insufficient_evidence	block
safe_to_merge	2	0	0	0
needs_human	0	1	1	0
must_block	0	0	0	3

Metric	Value
blocked_recall	1.0 (3/3 known-unsafe → blocked)
benign_escalation_rate	0.0 (0/2 known-safe → allow)
needs_human_caught	1.0 (2/2 → review / insufficient_evidence, never auto-passed)

A clean confusion matrix: the gate blocks what's known-unsafe, lets through what's known-safe, and routes the ambiguous to a human.

How it fits

This is one of three strata the W25 finding called for. The committed mined runs supply the other two — the negative control (the 226 trigger-skips) and the real-history extraction-coverage (insufficient_evidence) rate. Together: blocked-recall from constructed positives + noise/coverage from mined reality.

What's here

• benchmark/miner/constructed.py — runs the curated bundled fixtures via fixture run, records each verdict as a score-able row, reusing the same confusion-matrix code as the mined corpus.
• CLI python -m benchmark.miner constructed to (re)generate; committed results/constructed.{jsonl,csv,labels.csv}.
• Tests: a fast committed-data pin of the headline, plus a live-engine regression (test_live_engine_still_produces_the_constructed_verdicts) that re-runs the fixtures so a change regressing a blocked verdict fails in CI, not silently in the data file.

Type

• Check or risk-model change
• Input adapter change
• CLI or GitHub Action behavior
• Report, schema, or SARIF output
• Documentation only (benchmark tooling + tests over bundled fixtures; no src/ change)

Verification

CI is authoritative for python -m ruff check ., python -m compileall -q src tests, and python -m pytest.

• Full miner suite (test_miner + test_miner_labels + test_miner_corpus + test_miner_constructed, 34 tests) green; ruff clean; git diff --check clean on the new artifacts.
• The live-engine regression re-runs all 7 fixtures and confirms the committed corpus matches.

Release-readiness notes

• No user-code import added to default scan paths
• No network access added to default scan paths (runs bundled fixtures locally)
• New or changed check IDs documented (n/a)
• Report/schema changes are additive or documented (n/a — benchmark-local)