two queue filtered search with max effort

[hailangx]

Summary

• Introduces a new two-queue search algorithm for graph-based vector search that decouples exploration from
  filter evaluation, improving recall for low-selectivity filtered queries
• Adds a callback-based filter (GarnetFilterProvider) to diskann-garnet alongside the existing bitmap filter, enabling
  per-candidate FFI filter evaluation from Garnet/C#

Motivation

The existing beta-filtered search works well when filters are moderately selective, but struggles with low-selectivity
filters where most candidates are rejected. In those cases, the search converges prematurely because pruning is based
on distance to filtered results that haven't been found yet. The two-queue approach keeps exploration broad until
enough filtered results are accumulated.

Design

Two-Queue Search (two_queue_search.rs)

• Maintains two separate queues: a min-heap (candidates) for exploration ordered by distance, and a max-heap
  (filtered_results) for filter-passing neighbors
• All graph neighbors are explored regardless of filter status; only filter-passing nodes are added to filtered_results
• Convergence: terminates when filtered_results has enough results and the closest unexplored candidate is farther than
  the worst filtered result
• Supports a max_candidates hop limit as a safety cap and result_size_factor (default 10) to control result queue
  capacity (k * result_size_factor)
• Reports termination reason via TwoQueueTermination enum (Exhausted, MaxCandidates, Converged, FilterTerminated)

[Copilot]

Pull request overview

Adds a new “two-queue” filtered graph search path intended to improve recall for low-selectivity filtered queries by decoupling exploration from filter acceptance, and extends the Garnet FFI to support per-candidate callback-based filtering (in addition to bitmap filtering). Also wires the new search mode into the benchmark tooling/config.

Changes:

• Introduces TwoQueueSearch (DiskANN) + scratch support (two heaps: exploration candidates + filtered results) and exports it from diskann::graph::search.
• Adds Garnet callback filtering via GarnetFilterProvider and a unified GarnetFilter enum to select bitmap vs callback filtering.
• Adds a benchmark phase + benchmark-core search wrapper for running two-queue filtered search experiments.

Reviewed changes

Copilot reviewed 20 out of 20 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
diskann/src/neighbor/queue.rs	Adds “unbounded” variants of not-visited queue traversal helpers.
diskann/src/graph/search/two_queue_search.rs	New two-queue filtered search implementation + termination reporting.
diskann/src/graph/search/scratch.rs	Extends SearchScratch with candidates + filtered_results heaps and adds new_two_queue.
diskann/src/graph/search/mod.rs	Registers/exports the new two_queue_search module and public types.
diskann/src/graph/search/diverse_search.rs	Updates manual SearchScratch { ... } initializer to include new fields.
diskann/src/graph/config/defaults.rs	Adds RESULT_SIZE_FACTOR default for two-queue result heap sizing.
diskann-garnet/src/test_utils.rs	Updates test callbacks to include a no-op filter callback.
diskann-garnet/src/lib.rs	Extends FFI (create_index, search_*) to support callback-based filtering and max effort.
diskann-garnet/src/labels.rs	Adds GarnetFilterProvider and GarnetFilter enum.
diskann-garnet/src/garnet.rs	Extends callback bundle (Callbacks) with FilterCandidateCallback.
diskann-garnet/src/ffi_tests.rs	Updates FFI tests for new create_index signature.
diskann-garnet/src/ffi_recall_tests.rs	Updates recall tests for new create_index signature.
diskann-garnet/src/dyn_index.rs	Routes callback filtering to TwoQueueSearch; keeps bitmap filtering via beta-filter path.
diskann-benchmark/src/inputs/async_.rs	Adds TopkTwoQueueFilter phase config schema.
diskann-benchmark/src/backend/index/spherical.rs	Adds execution path for two-queue filtered benchmark phase (spherical backend).
diskann-benchmark/src/backend/index/search/knn.rs	Adds Knn runner integration for the benchmark-core TwoQueue searcher.
diskann-benchmark/src/backend/index/benchmarks.rs	Adds generic backend execution path for TopkTwoQueueFilter.
diskann-benchmark/example/async-two-queue-filter-ground-truth-small.json	Adds an example benchmark input for two-queue filtered search.
diskann-benchmark-core/src/search/graph/two_queue.rs	Adds benchmark-core TwoQueue search wrapper built on diskann::TwoQueueSearch.
diskann-benchmark-core/src/search/graph/mod.rs	Exports the new benchmark-core TwoQueue searcher.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

create_index gained a new filter_callback parameter, which changes the exported C ABI for an existing symbol. Any external callers not updated will pass the wrong arguments and can crash/UB. Consider providing a versioned entry point (e.g., create_index_v2) and keeping the old signature delegating to a default/no-op filter, or otherwise ensuring backward compatibility for existing FFI consumers.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 27.55556% with 326 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.25%. Comparing base (8126b0f) to head (0cbda48).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
diskann/src/graph/search/two_queue_search.rs	0.00%	164 Missing ⚠️
...skann-benchmark-core/src/search/graph/two_queue.rs	0.00%	59 Missing ⚠️
diskann-benchmark/src/backend/index/benchmarks.rs	0.00%	30 Missing ⚠️
diskann-garnet/src/lib.rs	48.78%	21 Missing ⚠️
diskann-benchmark/src/backend/index/search/knn.rs	0.00%	14 Missing ⚠️
diskann-garnet/src/labels.rs	0.00%	12 Missing ⚠️
diskann-benchmark/src/inputs/async_.rs	0.00%	11 Missing ⚠️
diskann-garnet/src/dyn_index.rs	37.50%	10 Missing ⚠️
diskann-garnet/src/test_utils.rs	25.00%	3 Missing ⚠️
diskann/src/neighbor/queue.rs	96.22%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #929      +/-   ##
==========================================
+ Coverage   89.44%   90.25%   +0.81%     
==========================================
  Files         449      451       +2     
  Lines       83779    84208     +429     
==========================================
+ Hits        74932    76004    +1072     
+ Misses       8847     8204     -643     

Flag	Coverage Δ
miri	90.25% <27.55%> (+0.81%)	⬆️
unittests	90.21% <27.55%> (+0.93%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
diskann-benchmark-core/src/search/graph/mod.rs	100.00% <ø> (ø)
diskann-benchmark/src/backend/index/spherical.rs	100.00% <ø> (ø)
diskann-garnet/src/garnet.rs	97.69% <100.00%> (+0.04%)	⬆️
diskann/src/graph/search/scratch.rs	98.68% <100.00%> (+0.46%)	⬆️
diskann/src/neighbor/queue.rs	98.18% <96.22%> (-0.15%)	⬇️
diskann-garnet/src/test_utils.rs	97.31% <25.00%> (-2.00%)	⬇️
diskann-garnet/src/dyn_index.rs	58.02% <37.50%> (-6.77%)	⬇️
diskann-benchmark/src/inputs/async_.rs	36.98% <0.00%> (-0.75%)	⬇️
diskann-garnet/src/labels.rs	90.32% <0.00%> (-9.68%)	⬇️
diskann-benchmark/src/backend/index/search/knn.rs	62.82% <0.00%> (-13.75%)	⬇️
... and 4 more

... and 40 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[hildebrandmw]

Thanks Haiyang. There was some mention of fusing predicate checks with element retrieval. Is that something that's still planned?

[hildebrandmw]

I'm a little worried about what adding this universally will do for compile times. Ideally, we'd have a more focused way to add extensions like this that don't encroach on our hard-won compile-time reduction efforts. It would require some backend shuffling, but I think there is a world where search phases behave more like plugins rather than enums, so we can target specific monomorphizations instead of forcing this on all instances.

What've you observed in terms of compile time differences here?

[hildebrandmw]

Shouldn't this live in two_queue? That's what it's meant to control, right? Things that live here should be focused on graph building confiugration. To that end, I'm not sure what FILTER_BETA is doing here either, but at least for this PR, can this constant be moved nearer to what it's meant to influence?

[hailangx]

:) I think we wanted the defaults to be in a single place. The **FILTER_BETA ** was moved here pervious comments request last time.

I would prefer we have a central place for defaults, if you think here should be all graph build related defaults, then we might just create another defaults for all search related.

[hildebrandmw]

This adds two BinaryHeaps for all users of SearchScratch, just for use in two-queue algorithm. Can we instead create these two queues manually in the place where they are needed instead of putting it in a central location? Really, SearchScratch needs to get pared down.

[hildebrandmw]

Can these be removed? It doesn't look like they are used?

[hildebrandmw]

Are there invariants that exist among these fields? If so - they shouldn't be public and this struct should have a dedicated constructor.

[hildebrandmw]

It doesn't look like Knn::search_l() is used at all in the two-queue method (outside of a size hint). If that's the case, then this shouldn't wrap Knn and instead capture just the parameters that actually affect the algorithm.

[hildebrandmw]

What about QueryVisitDecision::Terminate?