You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
#8138 shipped the backtest check as advisory-only, deliberately — a "regressed" verdict from compareBacktestScores (#8086) is posted as a report but never blocks a merge. This issue is the tracked follow-up that decides whether that should ever change, once #8138 has run in production and starts producing real reports.
Area
Policy decision (touches the gate's own merge/close authority — trust/safety-critical) plus, if the decision is "yes," the actual gate-wiring change inside ORB's live review pipeline (not a separate CI job — #8138 runs entirely inside src/queue/processors.ts's request handling, see #8142).
Decide: never gate (stays advisory forever); gate only maintainer-authored rule PRs; gate all rule-touching PRs including contributor ones; or a softer disposition (e.g. hold-for-review rather than a hard block, mirroring aiReviewLowConfidenceDisposition's existing hold_for_review option in src/rules/advisory.ts).
If the decision is anything other than "never," implement it as an actual gate-behavior change in ORB's live pipeline, following whichever pattern the decision calls for.
Deliverables
A written decision, with the real track-record evidence it was based on.
If the decision changes gate behavior: the implementation.
src/review/auto-tune.ts (the existing tightening-only precedent, if the decision leans toward automated gating)
src/rules/advisory.ts (aiReviewLowConfidenceDisposition's hold_for_review option, a precedent for a softer-than-hard-block disposition)
Boundaries
Do not implement any merge-blocking before #8138 has real production data to decide from — this issue cannot be meaningfully picked up until #8138 has shipped and run for a while.
maintainer-only — a real trust/safety-critical policy decision, not a build task for contributors.
Problem
#8138 shipped the backtest check as advisory-only, deliberately — a
"regressed"verdict fromcompareBacktestScores(#8086) is posted as a report but never blocks a merge. This issue is the tracked follow-up that decides whether that should ever change, once #8138 has run in production and starts producing real reports.Area
Policy decision (touches the gate's own merge/close authority — trust/safety-critical) plus, if the decision is "yes," the actual gate-wiring change inside ORB's live review pipeline (not a separate CI job — #8138 runs entirely inside
src/queue/processors.ts's request handling, see #8142).Proposal
aiReviewLowConfidenceDisposition's existinghold_for_reviewoption insrc/rules/advisory.ts).Deliverables
Resources
src/review/auto-tune.ts(the existing tightening-only precedent, if the decision leans toward automated gating)src/rules/advisory.ts(aiReviewLowConfidenceDisposition'shold_for_reviewoption, a precedent for a softer-than-hard-block disposition)Boundaries
Do not implement any merge-blocking before #8138 has real production data to decide from — this issue cannot be meaningfully picked up until #8138 has shipped and run for a while.
maintainer-only — a real trust/safety-critical policy decision, not a build task for contributors.