[QST] CAGRA self-search QPS seems low — is this expected on L4 with SINGLE_CTA/search_width=1?

[marcelo-cjl]

What is your question?

I benchmarked CAGRA search with a small self-search program and measured a QPS that seems lower than I expected from the cuVS ANN benchmarks. Is ~44k QPS a reasonable number for these parameters on an L4, or is something off? And which parameters would have the biggest impact on QPS?

Reproduction

Minimal reproducer (self-contained script that builds cuVS from source and runs a tiny C++ benchmark using cuvs::neighbors::cagra::build / cuvs::neighbors::cagra::search):

https://github.com/marcelo-cjl/cuvs/blob/dc50c8e0598438008186a64e3dc3523cbbe57145/cpp/bench/cuvs_cagra_self_search/cuvs_cagra_self_search.sh

Environment:

• Machine: AWS g6.4xlarge
• GPU: NVIDIA L4 23 GB
• Driver: 560.35.05
• cuVS: built from source at commit dc50c8e (Release, -DBUILD_TESTS=OFF -DBUILD_C_LIBRARY=OFF -DBUILD_CAGRA_HNSWLIB=OFF -DBUILD_MG_ALGOS=OFF -DCUVS_COMPILE_DYNAMIC_ONLY=ON, CMAKE_CUDA_ARCHITECTURES=native)

Dataset: "self-search" — the query set is the training set itself (Cohere, 1,000,000 rows x 768 dims, float). self_recall_at_1 is the fraction of queries where the top-1 label equals the query's own row id.

Index / search parameters:

Parameter	Value
dtype	float
metric	L2Expanded
graph_degree	64
intermediate_graph_degree	128
topk	1
itopk_size	32
search_width	1
max_iterations	0
search_algo	SINGLE_CTA

Observed result:

stage	rows	dim	search_ms	qps	self_recall_at_1
search	1,000,000	768	22,882.8	43,700.9	0.999338

search_ms is measured with CUDA events around a single cuvs::neighbors::cagra::search call (1M queries in one batch); qps = rows * 1000 / search_ms.

Expected behavior

I was expecting higher throughput based on the cuVS ANN benchmarks. Specifically:

1. Is ~44k QPS reasonable for SINGLE_CTA, search_width=1, topk=1, 1M x 768 float on an L4, or does it indicate a configuration problem?
2. Which knobs have the biggest impact on QPS (e.g., multi-CTA search algorithm, larger search_width/itopk_size, half precision, query batch size)?
3. Is there an official way to run this exact workload through cuvs-bench so I can compare against a reference number?