Skip to content

fix(libsy): warn when the scorer cannot reach a picker's non-default tier - #394

Open
WUKUNTAI-0211 wants to merge 3 commits into
NVIDIA-NeMo:mainfrom
WUKUNTAI-0211:fix/264-capable-first-scorer-gate
Open

fix(libsy): warn when the scorer cannot reach a picker's non-default tier#394
WUKUNTAI-0211 wants to merge 3 commits into
NVIDIA-NeMo:mainfrom
WUKUNTAI-0211:fix/264-capable-first-scorer-gate

Conversation

@WUKUNTAI-0211

@WUKUNTAI-0211 WUKUNTAI-0211 commented Aug 13, 2026

Copy link
Copy Markdown

What

Warns at route construction when a stage_router route pairs a picker with a confidence_threshold the scorer can never reach in the direction that leaves the default tier. No routing behaviour changes.

pub fn max_efficient_confidence() -> f64 { confidence_for_units(1.0) }  // 0.4621…
pub fn max_capable_confidence()   -> f64 { confidence_for_units(2.0) }  // 0.7616…

pub fn scorer_cannot_leave_default(mode, confidence_threshold) -> bool

Why

Closes #264

score_signal sums one positive axis against one negative dimension, then squashes with tanh. Each direction has a hard ceiling below 1.0:

direction units available ceiling
efficient production_intensity only, bounded to 1.0 by ratio 0.4621
capable severity at its HARD_SEVERITY cap, plus one of spinning / exploring (they partition the not-producing case) 0.7616

Above its ceiling the gate never opens, so the scorer can only ever confirm the default tier. The reporter measured 439 consecutive turns at capable_first + 0.5 with zero turns routed to the efficient tier and zero upstream errors. Nothing in the logs or in routing_decisions.stage_router separates "signals were weak this turn" from "this branch cannot fire at all".

efficient_first + 1.0 is the mirror case, and the [0.0, 1.0] range check accepts it today.

How tested

  • cargo test --workspace green (23 test binaries, 0 failed)
  • cargo clippy --workspace --all-targets clean
  • cargo fmt --all --check clean
  • uv run ruff check . clean
  • uv run mypy switchyard clean
  • uv run pytest tests/ --ignore=tests/e2e green (135 passed)
  • Manual smoke: captured the log for all four combinations — capable_first + 0.5 and efficient_first + 1.0 warn; 0.45 and 0.5 respectively stay silent.

tests/e2e/test_closed_book_proxy_integration.py fails locally, but it fails identically on a clean checkout of main — a pre-existing network-dependent failure, unrelated to this change.

Seven unit tests added. The two that pin the ceilings derive them from the dimension set rather than asserting literals, so they fail if the signal set ever changes.

Checklist

  • Unit tests added for the bug fix.
  • New public symbols re-exported from crates/libsy/src/lib.rs.
  • Commits signed off per the DCO.
  • Python-side checklist items are not applicable — Rust-only change, no customer-facing surface change.

Notes for reviewers

Which of the issue's three options this takes. Option 1 (lower the default) no longer applies: confidence_threshold is a required TOML field with no default, and the Python paths the issue cites were removed when routing moved to Rust. Option 3 (add a second negative signal) changes routing behaviour and needs recalibration. This is option 2 — happy to re-scope if you prefer option 3.

Warn, not reject. The hard overrides (should_escalate / should_deescalate) run before the scorer and still fire, and the judge still resolves turns, so the route is not inert end to end — only the scorer is. Rejecting at load would break running deployments, including the confidence_threshold = 1.0 fixture in config.rs.

Placement. The guard lives in libsy so it covers the TOML server path and the PyO3 bindings from one place, and the ceilings sit next to the SCORE_GAIN / SIGNAL_UNIT / HARD_SEVERITY constants they are derived from. Deployments on capable_first will see this alongside the existing "capable_first is experimental" warning in config.rs; collapsing the pair is an option if it reads as noise.

The capable ceiling depends on the severity table. It assumes HARD_SEVERITY is the strongest severity the scorer sees, which is what that constant's doc comment already states (critical is intercepted by the override). If a pattern with a severity between 0.7 and 1.0 is ever added to ERROR_PATTERNS, max_capable_confidence moves and its test fails — intentionally.

Docs left alone. #264 also flags four lines in docs/routing_algorithms/stage_router_routing.md. #288 is already rewriting that page, so touching it here would only conflict. A follow-up can point it at these helpers once that lands.

Summary by CodeRabbit

  • New Features

    • Added confidence-limit utilities for identifying efficient and capable maximum thresholds.
    • Added detection for thresholds that cannot move a picker beyond its default tier.
  • Bug Fixes

    • Warnings now identify unreachable thresholds, the affected mode, and recommended values.
    • Routing preserves hard overrides and fall-through behavior for undecided turns.
  • Tests

    • Added coverage for confidence ceilings, mode-specific thresholds, and picker behavior at supported limits.

Signed-off-by: WUKUNTAI <WUKUNTAI-0211@users.noreply.github.com>
@WUKUNTAI-0211
WUKUNTAI-0211 requested a review from a team as a code owner August 13, 2026 02:59
@coderabbitai

coderabbitai Bot commented Aug 13, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: c5439412-2825-425c-9910-b11107eba2c2

📥 Commits

Reviewing files that changed from the base of the PR and between f86c6db and 1380c9a.

📒 Files selected for processing (1)
  • crates/libsy/src/algorithms/util/stage.rs
🚧 Files skipped from review as they are similar to previous changes (1)
  • crates/libsy/src/algorithms/util/stage.rs

Walkthrough

The change adds scorer confidence ceilings and detects thresholds that cannot select the non-default tier. Route construction warns about unreachable thresholds while preserving existing routing behavior. The crate publicly re-exports both confidence helpers. Tests cover ceiling calculations and picker outcomes.

Changes

Stage routing threshold diagnostics

Layer / File(s) Summary
Confidence ceilings and threshold detection
crates/libsy/src/algorithms/util/stage.rs
Added efficient-side and capable-side confidence helpers. Added mode-aware detection for unreachable thresholds. Tests cover ceiling values, inclusive boundaries, and end-to-end tier selection.
Route construction diagnostics and exports
crates/libsy/src/algorithms/stage.rs, crates/libsy/src/lib.rs
Route construction warns when capable_first or efficient_first cannot select the opposite tier. Construction still proceeds. Both confidence helpers are publicly re-exported.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Mergeability Score: ⚪ Minimal · up to 1380c

This adds route-construction warnings for unreachable scorer thresholds without changing routing behavior; the reported checks pass, and no actionable merge-blocking risk remains beyond normal review.

Poem

I’m a rabbit guarding each stage,
With ceiling marks upon the page.
When thresholds block a tier from view,
A warning tells the route what’s true.
The tests hop past each boundary bright—
And routing stays on course tonight!

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly describes the main change: warning when a scorer cannot reach a picker's non-default tier.
Linked Issues check ✅ Passed The PR addresses issue #264 by warning when configured thresholds exceed reachable scorer confidence while preserving routing behavior.
Out of Scope Changes check ✅ Passed The changes are limited to scorer ceiling helpers, route-construction warnings, exports, and related tests.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/libsy/src/algorithms/stage.rs`:
- Around line 170-177: The error message in pick_tier’s confidence-threshold
warning overstates fall-through behavior by saying every turn is affected.
Update the message to limit the claim to scorer-only turns, while preserving the
existing explanation of the judge or capable-tier fallback and the threshold
guidance.

In `@crates/libsy/src/algorithms/util/stage.rs`:
- Around line 355-357: Update scorer_cannot_leave_default to account for
PickerMode::EfficientFirst when confidence_threshold is 1.0, reflecting that
finite tanh scores cannot satisfy the inclusive pick_tier gate; alternatively,
narrow and rename the public helper to explicitly cover only
PickerMode::CapableFirst. Update the related test covering Lines 715-725 to
assert the corrected contract.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 44ee6220-b2c9-43db-853e-963c61a17194

📥 Commits

Reviewing files that changed from the base of the PR and between b256d93 and a49b307.

📒 Files selected for processing (3)
  • crates/libsy/src/algorithms/stage.rs
  • crates/libsy/src/algorithms/util/stage.rs
  • crates/libsy/src/lib.rs

Comment thread crates/libsy/src/algorithms/stage.rs Outdated
Comment thread crates/libsy/src/algorithms/util/stage.rs
Signed-off-by: WUKUNTAI <WUKUNTAI-0211@users.noreply.github.com>
@WUKUNTAI-0211 WUKUNTAI-0211 changed the title fix(libsy): warn when capable_first cannot reach the efficient tier fix(libsy): warn when the scorer cannot reach a picker's non-default tier Aug 13, 2026
@WUKUNTAI-0211

Copy link
Copy Markdown
Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Aug 13, 2026

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/libsy/src/algorithms/util/stage.rs`:
- Around line 360-378: Update max_capable_confidence and the shared
signal-normalization path used by dimensions_from_signal so the capable ceiling
is derived from the exact normalized value accepted by the inclusive pick_tier
gate, making the strongest capable signal attainable. Add an end-to-end test at
max_capable_confidence covering the capable-threshold boundary, distinct from
efficient_first_at_its_ceiling_never_picks_capable.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 0353e5c9-5a04-4ec2-91c9-d22861e2c40b

📥 Commits

Reviewing files that changed from the base of the PR and between a49b307 and f86c6db.

📒 Files selected for processing (3)
  • crates/libsy/src/algorithms/stage.rs
  • crates/libsy/src/algorithms/util/stage.rs
  • crates/libsy/src/lib.rs
🚧 Files skipped from review as they are similar to previous changes (1)
  • crates/libsy/src/algorithms/stage.rs

Comment thread crates/libsy/src/algorithms/util/stage.rs
…carry

Signed-off-by: WUKUNTAI <WUKUNTAI-0211@users.noreply.github.com>
@WUKUNTAI-0211

Copy link
Copy Markdown
Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Aug 13, 2026

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] stage_router: capable_first can't offload at the default confidence_threshold=0.5

1 participant