calibration: backfill the corpus from the 10k+ historical review ledger (labels + firings; raw context phase 2)

[JSONbored]

Problem

The calibration corpus (#8101/#8104 fired events + human overrides) only accumulates from the moment those writers shipped — audit_events had 0 calibration.* rows and a young signal history on 2026-07-22, while the ledger holds 10k+ historical reviews whose decisions and human outcomes are already durably recorded (review_targets/review_audit, orb_pr_outcomes, agent.action.close/agent.action.merge events, reversal_reopened/reversal_reverted events). Backfilling turns the backtest/trend system from "starts learning today" into "starts with years of ground truth."

What is honestly reconstructable (verified against the live schema, 2026-07-23)

Corpus ingredient	Backfillable?	Source
Human verdict labels (confirmed/reversed)	Yes	terminal PR outcomes + the reversal events recordReversalSignals already writes; a decision later reversed ⇒ reversed, a terminal decision left standing ⇒ confirmed
Rule firings per blocker code	Mostly	review_audit/decision_json blocker lists where recorded; coverage varies by era of the ledger schema
metadata.confidence	Partial	present where decision_json carried per-finding confidence (AI-review-era rows); absent for older deterministic-only rows
Raw prompt context (issueText/prTitle/prBody/diff)	Re-fetchable (phase 2)	the PRs/issues still exist on GitHub for public repos — a bounded, rate-limit-aware re-fetch job can reconstruct them
modelResponseText	No — permanently	never stored before the #8139 capture; not reproducible (no deterministic model output)

Net effect: threshold backtests (#8138) and the precision/accuracy trends (#8113/#8140) gain instant deep history; logic-replay backtests (#8139) can only accumulate forward — no backfill can manufacture the model responses.

Requirements

⚠️ Required pattern. Pure-core/thin-IO split exactly like scripts/backtest-corpus-export-core.ts / backtest-corpus-export.ts. The transform (ledger rows → synthesized RuleFiredEvent/HumanOverrideEvent pairs) is the tested pure core; the CLI does the D1 reads/writes.

• Synthesized events must be idempotent and distinguishable: dedupe on (eventType, targetKey, provenance) and tag metadata.backfilled: true so trend/backtest consumers can include or exclude them explicitly, and re-runs never double-write.
• Dry-run mode is the default (--apply opts in), printing per-rule counts of what would be written.
• Never fabricate what isn't there: a row without a recorded blocker list yields NO fired events (skip + count), never a guess. modelResponseText is never synthesized.
• Phase 2 (separate flag, separate PR if large): the GitHub re-fetch of raw context for public-repo rows, bounded and resumable.

Deliverables

• scripts/backfill-calibration-corpus-core.ts (pure transform + dedupe/provenance logic, fully unit-tested)
• scripts/backfill-calibration-corpus.ts (thin IO: D1 read of the ledger eras, dry-run/apply, summary output)
• A dry-run report from the real DB pasted into this issue before any --apply

Test Coverage Requirements

scripts/** sits outside coverage.include, so Codecov does not gate this patch — the pure core still requires full unit tests per house convention (both label directions, reversal precedence, dedupe/idempotency, the skip-don't-guess paths).

Links & Resources

• Epic: verifiable pre-merge backtest for ORB/AMS deterministic rules #8082 (epic), calibration: wire linked_issue_scope_mismatch into the shared signal-tracking module (fired + reversal) #8101/calibration: wire recordRuleFired/recordHumanOverride for the remaining isConfiguredGateBlocker codes (excludes linked_issue_scope_mismatch, covered by #8101) #8104 (the live writers whose event shapes this synthesizes), Feed rule/AI-judgment calibration data into an accuracy trend view #8113/Scaffold the advisory backtest CI workflow + threshold-only comparison, and persist each run's result #8138/calibration: compute REGRESSED-verdict track-record statistics from ORB's persisted backtest results #8140 (the consumers that gain history)
• src/review/signal-tracking-wire.ts (the event shapes to match), src/services/public-accuracy-trend.ts's loadReversalDayRows (the reversal-derivation SQL precedent)

Boundaries

Backfilled events are additive, tagged, and reversible (deletable by the backfilled tag). No GitHub re-fetch in phase 1. modelResponseText is out of scope permanently.

maintainer-only — one-shot data migration against the production ledger.

[linear-code]

JSO-5663

[JSONbored]

Ground truth pinned before building (2026-07-23): the ledger of record is the Cloudflare D1 (verified live): review_audit 126,351, orb_pr_outcomes 8,705, review_targets 3,233 — so this backfill's source AND destination are both the D1; the selfhosted Postgres path stays a documented non-goal (self-host operators' corpora accumulate in their own Postgres via the same schema; a pg-driver variant is future work if ever needed). Self-hosted AMS is unaffected — miner nodes' local ledgers never feed this corpus.

Scope correction from the same check: reversal_reopened/reversal_reverted events = 0 in the ledger. Phase 1 can therefore synthesize firings + confirmed labels only (decisions that stood) — genuinely useful for precision baselines and the trend view, but the positive class has no D1-derivable rows. Historical reversed labels move to phase 2 alongside the raw-context re-fetch: derive them from GitHub ground truth (a closed-by-ORB PR later merged; a merged PR later reverted), which needs the API pass anyway.

[JSONbored]

Era analysis + proposed phase-1 transform (2026-07-23) — needs one mapping decision before code.

What the D1 ledger actually holds, by granularity:

Slice	Rows	Per-rule blocker codes?	Confidence?	Outcome label derivable?
review_audit gate_decision	256 (mostly [shadow])	No — one coarse reason (ci_red, rejected, auto_merge…) in the summary string	No	Yes (pair with pr_outcome/reviewed disposition)
review_targets.decision_json	3,233	No — decision-level verdict (merge/comment/close)	Yes — decision-level confidence	Yes
reviewed/pr_outcome rows	~3.5k / ~1.3k	—	—	The label source (merged/closed/manual)

So the optimistic "firings per blocker code" from this issue's table doesn't exist historically at rule granularity — no fabrication-free way to synthesize e.g. a duplicate_pr_risk firing from a coarse reason string.

What IS clean and valuable — the AI-review confidence corpus: each review_targets decision with a verdict + confidence, labeled by the human outcome (merged after a would-close ⇒ the decision was wrong; stood ⇒ confirmed). That is precisely the corpus shape the threshold backtest (#8138) consumes for DEFAULT_AI_REVIEW_CLOSE_CONFIDENCE-family rules — thousands of decided cases for the highest-traffic threshold, on day one, including a REAL positive class (would-close decisions the human merged anyway — no dependence on the empty reversal_* event types).

The decision I won't make unilaterally: synthesized fired events must carry a ruleId. Mapping a decision-level historical verdict onto today's rule ids (ai_consensus_defect/ai_review_split) is semantically close but not identical (those are finding-level codes; the ledger's confidence is decision-level). Options:

• (a) Map onto the existing AI-review rule ids — corpus immediately usable by the shipped threshold backtest; slight semantic stretch, permanently marked metadata.backfilled: true + metadata.provenance: "decision_level".
• (b) A dedicated historical rule id (e.g. ai_review_decision_legacy) — zero semantic stretch, but nothing consumes it until KNOWN_THRESHOLDS grows a mapping row, so the backfill's value arrives only with a follow-up wiring change.

Recommendation: (a), with the provenance tags making exclusion trivial if it ever muddies a comparison. Say which, and phase 1 gets built against it (dry-run first, report here, then --apply).