You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
#8105 (the Phase-2 merge-gating decision) explicitly requires real production data: how often a REGRESSED verdict fired, whether it was ever wrong, how often it was right. #8138 persists each backtest run's BacktestComparison result durably (via recordAuditEvent, from ORB's live review pipeline — not a CI job, see #8142), but nothing yet aggregates those individual records into the summary a maintainer actually needs to make that call.
Requirements
Add a pure function (e.g. computeRegressedVerdictTrackRecord) that takes an array of historical BacktestComparison results (calibration: Pareto-floor comparator between two BacktestScoreReports #8086) and returns a summary: total runs, count/rate of "regressed" verdicts, broken down per ruleId (a Map<string, { total, regressed, improved, unchanged }> or equivalent).
The pure aggregation function must be fully testable against synthetic fixture data now — it does not need to wait for real production data to exist, since the aggregation logic itself doesn't depend on how much real data has accumulated. Cover: zero runs, all-clean runs, some-regressed runs, and the per-ruleId breakdown with more than one ruleId present.
Deliverables
The pure aggregation function + its type, in packages/loopover-engine/src/calibration/ alongside the other calibration primitives.
Tests covering the cases above.
A thin CLI wrapper reading real persisted data and printing the summary (no dedicated test required for the wrapper itself, provided it contains no logic beyond the read + the function call + printing — mirrors this epic's established pure-core/thin-IO exemption).
Test Coverage Requirements
99%+ patch coverage (branch-counted) on the pure aggregation function.
Expected Outcome
Once #8138 has been running for a while, a maintainer can run one command and get the real track-record summary #8105 needs to actually decide from, instead of manually reading through individual PR comments one at a time.
Context
#8105 (the Phase-2 merge-gating decision) explicitly requires real production data: how often a REGRESSED verdict fired, whether it was ever wrong, how often it was right. #8138 persists each backtest run's
BacktestComparisonresult durably (viarecordAuditEvent, from ORB's live review pipeline — not a CI job, see #8142), but nothing yet aggregates those individual records into the summary a maintainer actually needs to make that call.Requirements
computeRegressedVerdictTrackRecord) that takes an array of historicalBacktestComparisonresults (calibration: Pareto-floor comparator between two BacktestScoreReports #8086) and returns a summary: total runs, count/rate of"regressed"verdicts, broken down perruleId(aMap<string, { total, regressed, improved, unchanged }>or equivalent).audit_eventsrows Scaffold the advisory backtest CI workflow + threshold-only comparison, and persist each run's result #8138 wrote (via the same event type it uses,THRESHOLD_BACKTEST_EVENT_TYPEinsrc/services/threshold-backtest-run.ts) and calls the pure function, printing the summary.Deliverables
packages/loopover-engine/src/calibration/alongside the other calibration primitives.Test Coverage Requirements
99%+ patch coverage (branch-counted) on the pure aggregation function.
Expected Outcome
Once #8138 has been running for a while, a maintainer can run one command and get the real track-record summary #8105 needs to actually decide from, instead of manually reading through individual PR comments one at a time.
Links & Resources