Skip to content

vector-backends.md

Latest commit

 

History

History
171 lines (135 loc) · 7.07 KB

vector-backends.md

File metadata and controls

171 lines (135 loc) · 7.07 KB
Raw

Vector retrieval backends (Step 2.2)

How the five VectorStore implementations in rag-backends share a contract while exposing very different native filter / index APIs.

Overview

Step 2.1 fixed the read-layer contract: every backend takes the same (ctx, vector, top_k, corpus_ids, filters) tuple and returns list[ChunkRef]. Step 2.2 widens backend coverage on top of that contract:

Backend Filter native shape Index variant control
pgvector parameterised SQL (WHERE …) ivfflat / hnsw DDL switch
Qdrant Filter(must / must_not / should) + FieldCondition HnswConfigDiff + ScalarQuantization
Weaviate v4 Filter.by_property(...) + & / | / ~ Configure.VectorIndex.{flat, hnsw, hnsw+pq}
Pinecone MongoDB-style operator dict ($eq / $in / $and / …) Managed — hint is advisory
Elasticsearch Query-DSL bool.filter clauses dense_vector.index_options.type=hnsw

Each backend lives in packages/backends/src/rag_backends/vector/:

  • pgvector.py, qdrant.py — required dependencies, ship with the base install (Step 1.1 + Step 2.1).
  • weaviate.py, pinecone.py, elasticsearch.py — optional extras ([weaviate] / [pinecone] / [elasticsearch]) so the default install stays light. SDK import is deferred to __init__ — the module imports fine without the extra installed; constructing the class raises a clear ImportError pointing at the extra.

Usage

from rag_backends.vector.weaviate import WeaviateVectorStore
from rag_core.types import IndexHint

store = WeaviateVectorStore(http_host="localhost")
await store.initialize(
    dimension=1536,
    hint=IndexHint(
        estimated_size=2_000_000,
        recall_target=0.95,
        latency_target_ms=50.0,
    ),
)

Every other call site is identical to pgvector / Qdrant — the VectorStore ABC enforces a single API. Read more at reference/backends.md.

Internals

IndexHint → variant selection

_index_hint.select_index_variant(hint) centralises the ADR-0009 §1 table. Every backend initialize() calls into it so the bucket thresholds live in one place:

size < 100K                        → "flat"
size < 1M and recall ≥ 0.99        → "ivfflat"
size < 10M                          → "hnsw"
size ≥ 10M                          → "ivf_pq"
hint is None                        → "hnsw"     # production default

Each backend then maps the variant onto its own primitives. Mismatches between intent and what the backend can offer (e.g. Pinecone-managed indexes, or Weaviate having no IVF-flat) are resolved by the closest available analogue and logged at initialize() time — never silently converted at query time.

FilterExpr translation

The Step 2.1 FilterExpr AST is the universal predicate language. Each backend ships a translator module that turns the AST into native filter form:

Translator Output type
_filter_sql.translate() (where_sql, params) for pgvector
_filter_qdrant.translate() qdrant_client.models.Filter
_filter_weaviate.translate() weaviate.classes.query._Filters
_filter_pinecone.translate() dict (Pinecone operator format)
_filter_elasticsearch.translate() dict (ES query-DSL clause)

Three contracts every translator must honour:

  1. Fail loud on unsupported fields. _check_field() raises RetrievalError rather than silently dropping a predicate. Losing an ACL clause silently is a security bug; losing a model filter silently is a correctness bug. This is the same rule the Step 2.1 translators established.
  2. Empty AnyIn / Or matches nothing. Each backend encodes a contradiction in its native dialect (Pinecone: $in: []; ES: match_none; Weaviate: equal("__never__")). Treating them as vacuously true is the more dangerous default — empty inputs usually come from a policy producing an empty allow-list, which should return zero results, not everything.
  3. Agree with the reference evaluate(). rag_core.filter.evaluate is the canonical oracle. Backend translators MUST produce results equivalent to running evaluate(expr, chunk_attrs) over every chunk — the Noop store enforces this in conformance tests, and the integration suite covers the real translators.

Tenant isolation patterns

Backend Primary primitive Defense-in-depth
pgvector WHERE tenant_id = $1 composite PK (tenant_id, chunk_id)
Qdrant FieldCondition(tenant_id) composite UUID5 over (tenant_id, chunk_id)
Weaviate Filter.by_property("tenant_id").equal(...) composite UUID5; delete filters by tenant_id
Pinecone namespace=tenant_id + explicit $eq filter
Elasticsearch {"term": {"tenant_id": …}} composite _id = f"{tenant_id}::{chunk_id}"

Pinecone is the only backend that has a first-class multi-tenancy primitive — namespaces — so we use it. The other four enforce tenant scope at the filter layer plus a composite identifier that prevents cross-tenant collisions on chunk_id.

Extension points

Adding another vector backend

  1. Create rag_backends/vector/<name>.py subclassing VectorStore.
  2. Create _filter_<name>.py and have it implement translate(expr) → native.
  3. Add <name> to [project.optional-dependencies] in packages/backends/pyproject.toml.
  4. Import the SDK lazily inside __init__ (and any initialize()-time helpers). Raise a clear ImportError pointing at the extra.
  5. Wire IndexHint via _index_hint.select_index_variant(hint) — never re-implement the bucket table.
  6. Conformance: the Step 1.1b SPI ABC already provides 14 conformance tests via tests/contract/test_vector_store.py. Add a live integration suite under tests/integration/test_<name>.py with pytest.importorskip("<sdk>") at the top and a reachability fixture in tests/integration/conftest.py.
  7. Filter-translator unit tests live under tests/backends/ and run without the SDK installed (Weaviate stubs the SDK at sys.modules; Pinecone / ES return plain dicts).

Supporting an additional FilterExpr field

When a backend stores a new metadata field that the policy engine could push down (e.g. document_id, trust_level):

  1. Add the field to the storage write path (bulk_index) so it's actually indexed.
  2. Add it to the backend's _SUPPORTED_FIELDS allowlist in the translator.
  3. Add a translation case if the native dialect requires a non-default operator (e.g. Pinecone's acl_labels $in vs scalar $eq).
  4. Extend the conformance test in tests/contract/test_vector_store.py so every backend is held to the same semantic bar.

References