Skip to content

feat(review): backtest-gated autonomous loosening of the satisfaction confidence floor#8163

Merged
JSONbored merged 1 commit into
mainfrom
feat/backtest-gated-loosening
Jul 23, 2026
Merged

feat(review): backtest-gated autonomous loosening of the satisfaction confidence floor#8163
JSONbored merged 1 commit into
mainfrom
feat/backtest-gated-loosening

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

Summary

Advances #8121 (the epic stays open for the decomposed remainder).

Scope

  • The PR title follows type(scope): short summary Conventional Commit format, for example fix(api): restore profile access checks.
  • This PR is focused and does not mix unrelated backend, UI, MCP, docs, dependency, and deploy changes.
  • This follows CONTRIBUTING.md and does not reintroduce GitHub Pages, VitePress, site/, or CNAME.
  • I linked a currently open issue this PR resolves (e.g. Closes #123) — a linked open issue is required for every contributor PR.

Validation

  • git diff --check
  • npm run actionlint
  • npm run typecheck
  • npm run test:coverage locally; codecov/patch requires ≥99% coverage of the lines AND branches you changed (aim for 100% on your diff so CI variance does not fail near the threshold). Global coverage is a non-blocking trend with a loose 90% backstop, not the gate.
  • npm run test:workers
  • npm run build:mcp
  • npm run test:mcp-pack
  • npm run ui:openapi:check
  • npm run ui:lint
  • npm run ui:typecheck
  • npm run ui:test
  • npm run ui:build
  • npm audit --audit-level=moderate
  • New or changed behavior has unit/integration tests for new branches, fallback paths, and sanitizer boundaries

If any required check was skipped, explain why:

  • Full npm run test:ci ran green end-to-end (1110 test files) on the pre-rebase head; the rebase onto feat(api): weekly rule-calibration trend endpoint over fired/override history and backtest runs #8156's merge resolved one textual conflict in src/api/routes.ts (both branches append internal routes at the same anchor — both kept), re-verified with a clean tsc --noEmit plus both internal-route suites (14 tests). New-module coverage: 100% line AND branch on both satisfaction-floor-loosening.ts (6 tests) and satisfaction-floor-loosening-run.ts (12 tests, incl. the mocked-evaluator defense-in-depth case and the best-effort audit failure path); floor threading covered in both the pure suite (3 new tests) and the run suite (3 new tests); route auth/flag-gate/response pinned in its own suite. cf-typegen regenerated and committed for the new var.

Safety

  • No secrets, wallet details, hotkeys, coldkeys, user PATs, private keys, raw trust scores, private rankings, or private maintainer evidence are exposed.
  • Public GitHub text stays sanitized, low-noise, and does not imply compensation guarantees or optimization tactics.
  • Auth, cookie, CORS, GitHub App, Cloudflare, or session changes include negative-path tests.
  • API/OpenAPI/MCP behavior is updated and tested where needed.
  • UI changes use live API data or real empty/error/loading states, not production mock/demo fallbacks.
  • Visible UI changes include a UI Evidence section below with JPG/JPEG or PNG screenshots arranged as organized, captioned, clickable thumbnails. SVG screenshots are not used as review evidence. Review-only screenshots or recordings are not committed to the repository.
  • Public docs/changelogs are updated where needed; changelogs are only edited for release-prep PRs.

This is trust-critical by nature; the containment is layered: env flag off by default → evaluator's own gates → write-path direction/bound re-checks → validated read (a corrupted override row can never tighten the floor or pass the minimum, and reads fail safe to the shipped default) → every apply audited with full evidence. A stored-override read failure degrades to the shipped constant, never blocks the review pass. Negative paths tested: 401 without token, 404 with flag off, DB-error fail-safe.

UI Evidence

Not applicable — no UI change.

Notes

… confidence floor (#8121 narrow start)

auto-tune.ts's OverridePayload states the historical rule: a loosening
recommendation never carries a payload, because autonomous loosening is the
regression risk the loop exists to avoid. The #8082 backtest primitives are
exactly the missing risk measurement -- this ships the epic's approved narrow
start for its named first candidate, LINKED_ISSUE_SATISFACTION_CONFIDENCE_FLOOR.

evaluateSatisfactionFloorLoosening (pure) proposes the SMALLEST candidate step
below the current floor that scores strictly improved on the visible split AND
non-regressed on the deterministic held-out split (#8087's fixed-seed split,
so repeated runs can never fish for a lucky partition), never below a hard
0.3 safety minimum, never on a small corpus. The run layer applies a cleared
proposal by writing a system_flags override (the same operational-flag table
the circuit breakers use) plus a calibration.satisfaction_floor_loosened
audit event, with independent direction/bound re-checks at the write path.
The live floor resolves inside runLoopOverLinkedIssueSatisfaction; the pure
parse takes the floor as a defaulted parameter, byte-identical when absent.

Everything is gated on SATISFACTION_FLOOR_AUTOTUNE_ENABLED (wrangler var,
false by default): the override read, the apply path, and the manual trigger
route POST /v1/internal/calibration/loosen-satisfaction-floor all 404/no-op
until the operator opts in, and flipping the flag off instantly restores the
shipped floor. 100% line+branch coverage on both new modules; floor threading
covered in the pure and run suites.

Advances #8121.
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@JSONbored JSONbored self-assigned this Jul 23, 2026
@JSONbored
JSONbored merged commit ba93111 into main Jul 23, 2026
4 checks passed
@JSONbored
JSONbored deleted the feat/backtest-gated-loosening branch July 23, 2026 07:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant