fix: give each query lexeme its own BM25 candidate budget

[nuemaan]

Summary

• The FTS prefilter from perf: bound classic BM25 candidates with Postgres FTS #252 bounded candidates with one global ts_rank_cd ordering. That ordering scores term density inside a chunk and ignores corpus-wide rarity, while BM25 weights rare terms heavily, so the chunk BM25 ranks first can be the first one truncated.
• Each query lexeme now draws from its own share of the budget through a lateral join, so a lexeme matching few chunks always contributes them. _MIN_CANDIDATES_PER_LEXEME keeps many-lexeme queries from dividing the budget into slivers.
• Adds a warning when the candidate pool saturates. The debug line reported candidates == limit whether the corpus held exactly that many or far more, so silent truncation was indistinguishable from a healthy query.
• Classic route only. Map-nav default path untouched. No API, worker, deployment, or migration impact.
• Resolves Classic BM25 FTS prefilter can truncate the top BM25 match in large namespaces #278, follows up on Optimize BM25 retrieval candidate loading with Postgres FTS prefilter #195 and perf: bound classic BM25 candidates with Postgres FTS #252.

Measured on 5001 chunks where 5000 densely repeat a common term and one holds a rare term, querying data zebra:

ts_rank_cd position of the rare chunk:   5001 of 5001
rank_rows_by_bm25 position:                 1 of 5001

candidate pool, global ordering, limit 2000:  2000 rows, rare chunk absent
candidate pool, per-lexeme budget:            1001 rows, rare chunk present

The per-lexeme pool is smaller and still keeps the match BM25 wants. Cost is one GIN probe per lexeme rather than one overall, and _MAX_FTS_QUERY_TOKENS already caps that at 50.

Verification

• uv run pytest apps/api/tests/unit/test_bm25_channel_tsquery.py apps/api/tests/contract/test_bm25_fts_prefilter_contract.py — 16 passed against real Postgres 16
• uv run pytest packages/shared-python/shared/tests/test_retrieval_search_channels.py — 4 passed
• uvx ruff check packages/shared-python apps/api/tests — clean
• Reverted the channel change and re-ran the new regression test to confirm it fails without the fix: AssertionError: assert 'common-2' == 'rare-zebra'

test_content_channel_uses_bounded_or_fts_after_scope_filters asserted on the literal LIMIT :fts_candidate_limit as a position marker for the candidate bound. That clause is now LIMIT lb.per_lexeme_limit, so the assertions were repointed. The intent from #244, that scope filters land ahead of any candidate bound so exclusions cannot consume the budget, is unchanged and still asserted.

Not tested: production-scale corpora, and lateral-probe cost at the 50 lexeme ceiling on a large namespace. Both need a dataset I do not have locally.

Deployment Notes

• No new environment variables. RETRIEVAL_POSTGRES_FTS_CANDIDATE_LIMIT keeps its meaning as the total budget, now spent per lexeme.
• No database migrations.
• Backwards compatible. The full-scan fallback is untouched, so rolling back is a code-only revert.

Checklist

• Tests were added or updated when behavior changed
• Public docs, examples, or OpenAPI contracts were updated when needed
• Database migrations are idempotent and safe to deploy
• Logs, errors, and validation paths avoid leaking secrets or user data
• The pull request description explains any breaking or user-visible change

[nuemaan]

Heads up that CI here is sitting at action_required rather than a pass or fail, so the required checks never ran. Fork PRs need a maintainer to approve the workflow run, and a fresh push does not clear it. That is what mergeable_state: blocked reflects, not a failing build.

Rebased onto current main just now. No conflicts, channels.py was untouched upstream. Re-ran locally against Postgres 16 after the rebase: 16 passed on the api unit and contract tests, 4 passed on the shared channel tests, ruff clean.

Worth separating the two things regardless of what happens to this PR: the truncation described in #278 is live on main today. If you would rather fix it differently, or fold it into other retrieval work, that is fine by me. The repro in the issue stands on its own.

[suguanYang]

Thanks for the clear repro — I re-ran the two prefilters locally on Postgres 16 (simple FTS + rank_rows_by_bm25) and the rare-term bug is real.

Their case still holds. 2000 dense data chunks + 1 zebra, query data zebra, limit 2000:

ts_rank_cd of rare-zebra:     2001 of 2001
BM25 on full corpus:          1st (11.71 vs 3.45)

global pool:      2000 rows, rare-zebra absent → BM25 top is common-*
per-lexeme pool:  1001 rows, rare-zebra present → BM25 top is rare-zebra

So this is a real fix for that pattern. I still would not merge it as a general recall improvement — the same setup shows a regression on covering / multi-term hits, which global ts_rank_cd keeps and this change drops.

Regression 1 (deterministic). 400 dense-only chunks per term + one chunk that contains all six query terms once. Query one two three four five six, limit 2000 → per_lexeme = 333:

ts_rank_cd of cover-6:        1 of 2401  (0.60 vs 0.50 for a dense single-term chunk)
BM25 on full corpus:          1st (10.30)

global pool:      cover-6 present  → BM25 top is cover-6 (15.08)
per-lexeme pool:  cover-6 absent   → BM25 never sees it; top is one-*

Each term’s slice is filled by denser single-term rows. Combined cover-density still ranks the covering chunk first, which is why current main keeps it.

Regression 2 (common 2-term shape, tie-break dependent). 2000 data filler N + 2000 analysis filler N + 1 data analysis, query data analysis:

ts_rank_cd of cover-both:     1 of 4001  (0.20 vs 0.10)
BM25 on full corpus:          1st (4.33)

global pool:      cover-both present → BM25 top is cover-both
per-lexeme pool:  cover-both absent  if it loses the per-term ORDER BY
                  (same ts_rank_cd 0.10 as the other 2000 hits, limit 1000)

If that chunk happens to be first in the posting list it survives; if it is later, it does not. Production chunks are not inserted first.

I also saw the unsaturated-budget issue: 1500 foo + 400 bar = 1900 matches, under the 2000 cap. Global returned all 1900; per-lexeme returned 1400 (1000+400). In that corpus the BM25 winner did not change, so I am not hanging the review on top-k there — but the old path would not have truncated at all.

Could you add a contract test for the covering chunk (regression 1 is the clean one), and keep a global coverage slice rather than spending the whole budget per lexeme? Rare-term starvation is the actual #278 failure; a hybrid (global ts_rank_cd slice + per-lexeme floor) would fix that without dropping the chunk the combined ranking already puts first. The saturation warning is useful either way.