Skip to content

Decide + implement whether a REGRESSED backtest verdict should ever gate a merge (Phase 2) #8105

Description

@JSONbored

Problem

#8138 shipped the backtest check as advisory-only, deliberately — a "regressed" verdict from compareBacktestScores (#8086) is posted as a report but never blocks a merge. This issue is the tracked follow-up that decides whether that should ever change, once #8138 has run in production and starts producing real reports.

Area

Policy decision (touches the gate's own merge/close authority — trust/safety-critical) plus, if the decision is "yes," the actual gate-wiring change inside ORB's live review pipeline (not a separate CI job — #8138 runs entirely inside src/queue/processors.ts's request handling, see #8142).

Proposal

  • Once Scaffold the advisory backtest CI workflow + threshold-only comparison, and persist each run's result #8138 has run in production for a real stretch (enough PRs touching a backtested rule or AI-judgment surface to have actual regressed/improved/unchanged verdicts on record, not zero data), run calibration: compute REGRESSED-verdict track-record statistics from ORB's persisted backtest results #8140's track-record tool and review the real numbers: how often did a regressed verdict fire, was it ever wrong (a false regression a human would have approved anyway), how often was it right.
  • Decide: never gate (stays advisory forever); gate only maintainer-authored rule PRs; gate all rule-touching PRs including contributor ones; or a softer disposition (e.g. hold-for-review rather than a hard block, mirroring aiReviewLowConfidenceDisposition's existing hold_for_review option in src/rules/advisory.ts).
  • If the decision is anything other than "never," implement it as an actual gate-behavior change in ORB's live pipeline, following whichever pattern the decision calls for.

Deliverables

  • A written decision, with the real track-record evidence it was based on.
  • If the decision changes gate behavior: the implementation.

Resources

Boundaries

Do not implement any merge-blocking before #8138 has real production data to decide from — this issue cannot be meaningfully picked up until #8138 has shipped and run for a while.

maintainer-only — a real trust/safety-critical policy decision, not a build task for contributors.

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.roadmapOn the Wave-2 agent-layer roadmap board (project 9)

Projects

Status
Todo
Status
Todo

Relationships

None yet

Development

No branches or pull requests

Issue actions