[BUG] Blackwell RTX 4500 OpenSearch integration: CUDA Error 700

[maestromadman]

Describe the bug

Intermittent CUDA error 700: illegal memory access during CAGRA builds on Blackwell (sm_120, RTX PRO 4500). The underlying error is cudaErrorInvalidDevice: invalid device ordinal, thrown in the IVF-PQ k-means step of knn-graph construction:

GpuIndexCagra::trainEx → CuvsCagra::train → cuvs::neighbors::cagra::build
 → cagra::detail::build_knn_graph → ivf_pq::detail::train_per_subset
 → cuvs::cluster::kmeans::detail::countLabels (kmeans_common.cuh:112)
 → cub::DeviceHistogram::HistogramEven(...)
 Reason = cudaErrorInvalidDevice: invalid device ordinal

Non-deterministic: anywhere from ~1-3% of GPU segment builds fail, across segment sizes from ~31k to ~299k vectors. Does not reproduce on A10G (sm_86).

Steps/Code to reproduce bug

Repeated cuvs::neighbors::cagra::build calls over segments of a 10M × 768-dim float32 dataset (via OpenSearch remote-vector-index-builder); a small fraction crash as above. Full environment setup: linked here

Expected behavior (what should happen)

CAGRA builds complete on the GPU on Blackwell (sm_120) without illegal memory access, as they do on Ampere (sm_86).

Environment details (please complete the following information):

• Environment location: Cloud (AWS EC2 g7.2xlarge)
• Method of cuVS install: Docker — libcuvs 25.08 in opensearchproject/remote-vector-index-builder (CUDA 12.9, nvidia/cuda:12.9.0-base-ubuntu24.04). Reproduces on both a from-source rebuild and the prebuilt NGC api-latest image.
• GPU: RTX PRO 4500 Blackwell, sm_120, driver 595.91.07 · CUDA 12.9 (nvcc 12.9.86)

Additional context

Appears sm_120-specific in libcuvs 25.08 (reproduces on two independent CUDA-12.9 images, so not a CUDA-version issue). Illegal access is in cuVS's IVF-PQ k-means histogram path, reached via FAISS's GpuIndexCagra wrapper.