Skip to content

Synthesized positional scores inflate retrieval confidence once code search returns hits #243

Description

@NathanTarbert

Found by the review of PR #242 (Phase 2, code search). Not fixed there because it changes confidence behaviour on every ticket and wants measuring rather than guessing.

parseSnippets synthesizes a positional score for text-format results — score: Math.max(0.5, 1 - i * 0.05) (pathfinder.ts) — because the MCP text format carries no numeric relevance. That was tolerable while only search-docs produced them. Now search-code produces its own independent list, so the top code hit gets 1.0 regardless of how marginal it is, and min_score defaults to 0.3.

Two consumers read those numbers:

  • confidence.heuristicScoretopScore * 0.4 + avgScore * 0.4 + min(count * 0.03, 0.2)
  • generator.assessConfidenceavg + min(count * 0.05, 0.15)

So retrieval confidence is inflated for any question that returns any code hit, including irrelevant ones. That is the opposite of what the groundedness work in this module was for.

PR #242 caps the merged list at defaultLimit, which fixes the count half — the prompt and the count bonus no longer double. The topScore = 1.0 half is untouched.

Worth deciding rather than patching

Options, roughly in order of how much they'd need measuring:

  1. Scale synthesized code scores into a lower band than docs, so a positional rank cannot present as a perfect match.
  2. Have the confidence heuristics ignore synthesized scores entirely and read only real relevance numbers, falling back to count when none exist.
  3. Ask whether heuristicScore should read retrieval scores at all now that groundedness assesses the actual response text.

Option 3 is the one I'd want to think about — the heuristic predates the groundedness gate, and "how good were the search results" is a weaker signal than "is the answer traceable to them", which we now compute directly.

Whichever way it goes, this is exactly what the Phase 1 harness (#241 / CPK-8076) exists to measure, so it should land after that.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: aiAI response pipeline, groundedness, confidenceroadmapTracked on the Outpost roadmaproadmap: nextRoadmap horizon: after launch path clears

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions