Found by the review of PR #242 (Phase 2, code search). Not fixed there because it changes confidence behaviour on every ticket and wants measuring rather than guessing.
parseSnippets synthesizes a positional score for text-format results — score: Math.max(0.5, 1 - i * 0.05) (pathfinder.ts) — because the MCP text format carries no numeric relevance. That was tolerable while only search-docs produced them. Now search-code produces its own independent list, so the top code hit gets 1.0 regardless of how marginal it is, and min_score defaults to 0.3.
Two consumers read those numbers:
confidence.heuristicScore — topScore * 0.4 + avgScore * 0.4 + min(count * 0.03, 0.2)
generator.assessConfidence — avg + min(count * 0.05, 0.15)
So retrieval confidence is inflated for any question that returns any code hit, including irrelevant ones. That is the opposite of what the groundedness work in this module was for.
PR #242 caps the merged list at defaultLimit, which fixes the count half — the prompt and the count bonus no longer double. The topScore = 1.0 half is untouched.
Worth deciding rather than patching
Options, roughly in order of how much they'd need measuring:
- Scale synthesized code scores into a lower band than docs, so a positional rank cannot present as a perfect match.
- Have the confidence heuristics ignore synthesized scores entirely and read only real relevance numbers, falling back to count when none exist.
- Ask whether
heuristicScore should read retrieval scores at all now that groundedness assesses the actual response text.
Option 3 is the one I'd want to think about — the heuristic predates the groundedness gate, and "how good were the search results" is a weaker signal than "is the answer traceable to them", which we now compute directly.
Whichever way it goes, this is exactly what the Phase 1 harness (#241 / CPK-8076) exists to measure, so it should land after that.
Found by the review of PR #242 (Phase 2, code search). Not fixed there because it changes confidence behaviour on every ticket and wants measuring rather than guessing.
parseSnippetssynthesizes a positional score for text-format results —score: Math.max(0.5, 1 - i * 0.05)(pathfinder.ts) — because the MCP text format carries no numeric relevance. That was tolerable while onlysearch-docsproduced them. Nowsearch-codeproduces its own independent list, so the top code hit gets1.0regardless of how marginal it is, andmin_scoredefaults to0.3.Two consumers read those numbers:
confidence.heuristicScore—topScore * 0.4 + avgScore * 0.4 + min(count * 0.03, 0.2)generator.assessConfidence—avg + min(count * 0.05, 0.15)So retrieval confidence is inflated for any question that returns any code hit, including irrelevant ones. That is the opposite of what the groundedness work in this module was for.
PR #242 caps the merged list at
defaultLimit, which fixes the count half — the prompt and the count bonus no longer double. ThetopScore = 1.0half is untouched.Worth deciding rather than patching
Options, roughly in order of how much they'd need measuring:
heuristicScoreshould read retrieval scores at all now that groundedness assesses the actual response text.Option 3 is the one I'd want to think about — the heuristic predates the groundedness gate, and "how good were the search results" is a weaker signal than "is the answer traceable to them", which we now compute directly.
Whichever way it goes, this is exactly what the Phase 1 harness (#241 / CPK-8076) exists to measure, so it should land after that.