feat(review): backtest confidence-threshold changes against real signal history in ORB's live gate

[JSONbored]

Summary

• ORB's live review pipeline now detects PRs that change a known AI-judgment confidence threshold (LINKED_ISSUE_SATISFACTION_CONFIDENCE_FLOOR, DEFAULT_AI_REVIEW_CLOSE_CONFIDENCE), replays the change against real recorded rule-fired/human-override history, and folds a precision/recall comparison into the existing unified PR comment.
• Runs entirely inline in the Worker request path — reuses ORB's existing diff access, D1 access, and comment builder instead of adding a new CI workflow or a separate bot comment.
• Never blocks the gate; comment-only, matching Scaffold the advisory backtest CI workflow + threshold-only comparison, and persist each run's result #8138's original scope.

Architecture note (differs from #8138's literal text)

#8138 as originally scoped described a standalone CI workflow (.github/workflows/*.yml) shelling out to wrangler d1 execute. During implementation that approach was replaced with an ORB-native one: the same detection/backtest/render logic now lives in src/services/threshold-backtest.ts (pure) and src/services/threshold-backtest-run.ts (D1 orchestration via the existing SignalStore), called directly from src/queue/processors.ts and threaded into the unified comment via unified-comment-bridge.ts → unified-comment.ts. This avoids a second CI job and a second bot comment on an already-slow CI pipeline, and keeps threshold backtesting behind the same live gate infra everything else uses. All functional requirements from the issue (detect the two named constants, backtest against real history, render a comparison, persist the structured result, never block merge) are preserved.

Test plan

• npm run typecheck — clean
• New/changed suites: threshold-backtest.test.ts, threshold-backtest-run.test.ts, threshold-backtest-advisory-wiring.test.ts, backtest-threshold-engine.test.ts — 100% coverage (statements/branches/functions/lines) on both new src/services/ files
• packages/loopover-engine internal suite — 655/655 passing (barrel-export additions don't regress anything)
• npm run test:ci

Closes #8138

[superagent-security]

Superagent didn't find any vulnerabilities or security issues in this PR.

[cloudflare-workers-and-pages]

Deploying with    Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Preview URL	Updated (UTC)
✅ Deployment successful! View logs	loopover-ui	2ba9f33	Commit Preview URL Branch Preview URL	Jul 23 2026, 02:12 AM

[codecov]

Codecov Report

❌ Patch coverage is 93.05556% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 92.02%. Comparing base (66a026f) to head (2ba9f33).
⚠️ Report is 1 commits behind head on main.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
src/review/unified-comment.ts	25.00%	1 Missing and 2 partials ⚠️
src/queue/processors.ts	91.66%	0 Missing and 1 partial ⚠️
src/review/unified-comment-bridge.ts	0.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #8142      +/-   ##
==========================================
- Coverage   92.03%   92.02%   -0.01%     
==========================================
  Files         760      763       +3     
  Lines       77424    77496      +72     
  Branches    23399    23420      +21     
==========================================
+ Hits        71255    71318      +63     
- Misses       5061     5062       +1     
- Partials     1108     1116       +8     

Flag	Coverage Δ
shard-1	53.66% <47.22%> (-3.44%)	⬇️
shard-2	54.72% <73.61%> (+0.38%)	⬆️
shard-3	54.09% <73.61%> (+3.78%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...pover-engine/src/calibration/backtest-threshold.ts	100.00% <100.00%> (ø)
src/services/threshold-backtest-run.ts	100.00% <100.00%> (ø)
src/services/threshold-backtest.ts	100.00% <100.00%> (ø)
src/queue/processors.ts	95.75% <91.66%> (-0.02%)	⬇️
src/review/unified-comment-bridge.ts	99.25% <0.00%> (-0.37%)	⬇️
src/review/unified-comment.ts	98.47% <25.00%> (-1.14%)	⬇️

... and 1 file with indirect coverage changes

[loopover-orb]

Caution

🛑 LoopOver review result - fixes required

_{Review updated: 2026-07-23 02:25:23 UTC}

12 files · 1 AI reviewer · no blockers · CI failing · unstable

🛑 Suggested Action - Manual Review

Review summary
This PR adds a threshold-backtest advisory (#8138) that detects changes to two known confidence-threshold constants in a PR's own diff, replays the change against real recorded rule-fired/human-override history via the existing SignalStore, and renders the precision/recall comparison as a new collapsible section in the unified comment — never blocking the gate. The pure/IO split (threshold-backtest.ts vs threshold-backtest-run.ts vs the engine's backtest-threshold.ts) mirrors the existing linked-issue-satisfaction pattern, wiring into processors.ts/unified-comment-bridge.ts/unified-comment.ts is additive and gated behind a watched-file cheap-skip, and every I/O path (SignalStore read, D1 write, the whole advisory) fails open/silent so a broken backtest never affects the review pass. The diff-based `detectChangedThresholds` regex approach is a real design tradeoff (only catches the constant when the diff hunk shows a full `export const NAME = value;` line, e.g. missed if reformatted across lines) but is honestly documented and tested for its ambiguous-edit cases.

Nits — 6 non-blocking

• src/queue/processors.ts:7412 uses console.error directly for the failure path — verify this matches the logger convention used by the sibling runLinkedIssueSatisfactionForAdvisory failure path rather than a project-wide structured logger.
• detectChangedThresholds (src/services/threshold-backtest.ts) relies on a single-line `export const NAME = value;` regex match per diff line — a constant whose declaration is reformatted (e.g. wrapped, or with a type annotation) across the change would silently produce no detection; worth a one-line doc caveat if not already covered.
• THRESHOLD_BACKTEST_WATCHED_PATHS (processors.ts:7382) is a hardcoded two-path allowlist that must be kept in sync by hand whenever KNOWN_THRESHOLDS' source files change — consider deriving it or at least colocating a comment pointing at KNOWN_THRESHOLDS so the two don't drift.
• codecov/patch is failing at 93.05% vs the 99% target; given BASE BRANCH STATUS shows this branch is 1 commit behind default, part of the shortfall may be attributable to drift rather than this diff — but worth confirming the uncovered lines are inside this PR's own new files before merging.
• Add a short doc-comment cross-reference between THRESHOLD_BACKTEST_WATCHED_PATHS and KNOWN_THRESHOLDS so future additions to one are prompted to check the other (src/queue/processors.ts:7382, src/services/threshold-backtest.ts KNOWN_THRESHOLDS).
• PR author also opened the linked issue — Link an issue that was opened by a different contributor, or provide a rationale for why this self-authored issue represents genuine discovery work.

CI checks failing

• codecov/patch — 93.05% of diff hit (target 99.00%)

Decision drivers

• ✅ Code review — No blockers (1 reviewer)
• ✅ Gate result — Passing (No configured blocker found.)

Context & advisory signals — never blocks the verdict

Signal	Result	Evidence
Linked issue	✅ Linked	#8138
Related work	✅ No active overlap found	No same-issue or scoped active PR overlap found.
Change scope	✅ 20/20	Low review scope from cached public metadata (1 linked issue).
Validation posture	✅ 25/25	PR body includes validation/test evidence.
Contributor workload	✅ 10/10	Author activity: 14 registered-repo PR(s), 14 merged, 229 issue(s).
Contributor context	✅ Confirmed Gittensor contributor	JSONbored; Gittensor profile; 14 PR(s), 229 issue(s).
Improvement	✅ Minor	risk: clean · value: minor · LLM: moderate

Linked issue satisfaction

Partially addressed
The PR implements the core detection/backtest/render/persist logic for the two named threshold constants and covers it with tests, satisfying the substantive analytical intent, but it explicitly deviates from the issue's specific deliverable of a standalone CI workflow (`.github/workflows/*.yml`) with the mandated trigger/paths/draft-skip/concurrency shape and a CLI-style script mirroring export-d

Review context

• Author: JSONbored
• Role context: owner (maintainer lane)
• Public audience mode: oss maintainer
• Lane context: Repository is configured for direct PR review.
• Public profile languages: Python, TypeScript, Ruby, Go, JavaScript, MDX, Shell, Solidity
• Official Gittensor activity: 14 PR(s), 229 issue(s).
• PR-specific overlap: none found.

Contributor next steps

• Start here: Treat this as maintainer-lane context rather than normal contributor-lane activity.

Signal definitions

• Related work = same linked issue, overlapping active PRs, or title/path similarity.
• Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
• Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
• Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
• Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.

🧪 Chat with LoopOver

Ask LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.

• @loopover ask <question> answers contribution-quality Q&A with source citations and freshness.
• @loopover chat <question> answers in natural prose from cached decision-pack facts via local inference (maintainer/collaborator; read-only).
• A plain-language @loopover mention with a real question is routed to the closest matching read-only command automatically — no exact syntax required.

Full command reference: https://loopover.ai/docs/loopover-commands

🧪 Experimental — new and may change.

_{🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed}

---

💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.

• Re-run LoopOver review