Skip to content

caching.md

Latest commit

 

History

History
218 lines (165 loc) · 8.97 KB

caching.md

File metadata and controls

218 lines (165 loc) · 8.97 KB
Raw

Caching — three caches, three invalidation rules

Overview

AgentContextOS ships three distinct cache SPIs, each with a different cache key and a different invalidation contract. They commonly share a backend (Redis) but the SPIs themselves are separate because conflating them produces either stale answers or low hit rates.

Cache Key Invalidation trigger
EmbeddingCache (model_id, model_version, text_hash) Embedder model or version change
RetrievalCache (plan_hash, corpus_version) Document add / update / delete in queried corpus
AnswerCache (plan_hash, corpus_version, policy_version) Either of the above + any policy / PII rule change

A single "semantic cache" that ignores these distinctions produces stale results (when corpus changes invalidate at the wrong granularity) or a < 5% hit rate (when it invalidates too aggressively).

Step 4.1 implemented these as a tiered L0/L1/L2 cache. The production implementation lives in rag-cache (TieredRetrievalCache / TieredAnswerCache, in-memory L0 + similarity L2) with the Redis L1 tier in rag-backends. See tiered-cache.md and ADR-0021. This document describes the SPI contract the tiers implement.

Why it exists

The V1.0 plan described a single "L0/L1/L2 semantic cache" in Step 4.1. Three observations forced the split:

  1. Embedding cache reuse across re-indexes. Re-chunking a corpus (e.g., new chunker version) should keep the embedding cache hot — text hasn't changed. A single cache that keys on the query plan would miss.
  2. Retrieval cache invalidates on corpus mutation. Adding one document to a 10M-doc corpus shouldn't dump the entire answer cache. A single cache that keyed on policy would.
  3. Answer cache invalidates on policy changes. A tenant rotating their PII policy must invalidate cached answers (which may contain text affected by the new policy) but should preserve retrieval and embedding caches.

Usage

All three SPIs take ctx: RequestContext as their first positional argument (enforced by tests/contract/spi_signature.py). The remaining parameters are keyword-only to keep call sites readable.

EmbeddingCache

from rag_core.spi import EmbeddingCache

cache: EmbeddingCache = ...

cached = await cache.get(
    ctx,
    model_id="bge-large-en-v1.5",
    model_version="1.5.0",
    text_hash=h,
)
if cached is None:
    emb = await embedder.embed(ctx, text, chunk_id)
    await cache.put(
        ctx,
        model_id="bge-large-en-v1.5",
        model_version="1.5.0",
        text_hash=h,
        value=emb,
    )

Hit rate target: ≥ 99% on re-ingest of identical corpora.

Agent-loop reuse semantics (Step 2.11 finding)

The Step 2.11 AgentLoopV0 integrates the EmbeddingCache between the understander and the router so the per-iter vector lookup avoids redundant embedder calls. The 50-query harness (report, gap-list) measured 0% hit rate on the decomposable bucket (queries that fan out to genuinely different sub-queries) because text-hash equality is too strict for semantically-related-but-textually-distinct sub-queries.

Fix landed (G-01 ✅, ADR-0010). rag-retrieval ships SemanticEmbeddingCache — a wrapper above any existing EmbeddingCache implementation that adds a get_semantic method. get_semantic scans an LRU-bounded per-tenant token-set index using Jaccard similarity, returning the most-similar cached entry when similarity ≥ threshold (default 0.5). The text-hash exact-match cache stays as the inner layer (called first on every lookup). AgentLoopV0 detects the wrapper automatically and consults get_semantic after an exact-match miss. Harness rerun showed decomposable-bucket hit rate 0% → 43.3% and overall 28.2% → 52.7%.

v0 uses linear-scan Jaccard, which scales to ~10⁴ entries per tenant. ADR-0010 documents the migration path to a vector-indexed variant for larger tenants — the get_semantic signature is stable across the swap.

RetrievalCache

from rag_core.spi import RetrievalCache

cache: RetrievalCache = ...

cached = await cache.get(ctx, plan_hash=plan.hash(), corpus_version=corpus.version)
if cached is None:
    refs = await retrieve_pipeline(ctx, plan)
    await cache.put(
        ctx, plan_hash=plan.hash(), corpus_version=corpus.version, value=refs
    )

RetrievalCache.invalidate_corpus(ctx, corpus_id=...) is wired into the write path: every successful bulk_index / bulk_delete on a corpus calls it after bumping corpus_version. Backends with no scan support return 0 from invalidate_corpus and rely on the version bump alone.

Hit rate target: ≥ 30% on production query mix (Pareto-distributed).

AnswerCache

from rag_core.spi import AnswerCache

cache: AnswerCache = ...

cached = await cache.get(
    ctx,
    plan_hash=plan.hash(),
    corpus_version=corpus.version,
    policy_version=current_policy_version(ctx),
)
if cached is None:
    answer_bytes = serialize(await llm_pipeline(ctx, plan))
    await cache.put(
        ctx,
        plan_hash=plan.hash(),
        corpus_version=corpus.version,
        policy_version=current_policy_version(ctx),
        value=answer_bytes,
    )

AnswerCache stores opaque bytes — the answer envelope (text + citations + trace metadata) is still firming up in Phase 3, so the SPI doesn't constrain it. Callers handle their own serialization.

AnswerCache.invalidate_policy(ctx, policy_version=...) is called by the control plane on any policy rotation.

Hit rate target: ≥ 15% on production query mix.

Internals

Single Redis backend, three keyspaces

The default implementation puts all three in one Redis instance under prefixes rag:emb:, rag:ret:, rag:ans:. Memory budgets are tunable per cache. Eviction is LRU within each prefix.

corpus_version

A monotonically incrementing integer per corpus, bumped by the write path (IndexBackend.bulk_index) on any successful upsert / delete. Read by the retrieval pipeline as part of the cache key.

policy_version

A hash of the active PolicyEngine configuration for the tenant. Bumped by the control plane on any policy change. Stale answers are invalidated transparently on the next lookup.

Cross-tenant safety

Every cache key is implicitly tenant-scoped via ctx.tenant_id baked into plan_hash. A red-team probe in tests/redteam/test_cross_tenant_cache.py asserts no key from tenant A can resolve to tenant B's value, across every Step 4.1 implementation (in-memory, tiered, semantic).

Sizing

  • Embedding cache: largest. ~1 KB per entry × millions of entries.
  • Retrieval cache: medium. ~10 KB per entry (top-K ChunkRefs).
  • Answer cache: smallest. ~5 KB per entry (text + citations).

Default sizing in rag.yaml:

cache:
  embedding:
    provider: redis
    max_memory_mb: 4096
    ttl_seconds: 0       # never expire (invalidated on model change)
  retrieval:
    provider: redis
    max_memory_mb: 1024
    ttl_seconds: 3600    # belt-and-braces
  answer:
    provider: redis
    max_memory_mb: 512
    ttl_seconds: 900

Extension points

The canonical SPI shapes:

class EmbeddingCache(ABC):
    async def get(self, ctx, *, model_id: str, model_version: str, text_hash: str) -> Embedding | None: ...
    async def put(self, ctx, *, model_id: str, model_version: str, text_hash: str, value: Embedding, ttl_seconds: int | None = None) -> None: ...
    async def invalidate_model(self, ctx, *, model_id: str, model_version: str) -> int: ...

class RetrievalCache(ABC):
    async def get(self, ctx, *, plan_hash: str, corpus_version: int) -> list[ChunkRef] | None: ...
    async def put(self, ctx, *, plan_hash: str, corpus_version: int, value: list[ChunkRef], ttl_seconds: int | None = None) -> None: ...
    async def invalidate_corpus(self, ctx, *, corpus_id: str) -> int: ...

class AnswerCache(ABC):
    async def get(self, ctx, *, plan_hash: str, corpus_version: int, policy_version: str) -> bytes | None: ...
    async def put(self, ctx, *, plan_hash: str, corpus_version: int, policy_version: str, value: bytes, ttl_seconds: int | None = None) -> None: ...
    async def invalidate_policy(self, ctx, *, policy_version: str) -> int: ...

Plug in alternative backends (Valkey, Memcached, in-memory LRU) without changing consumers. Reference noop implementations live in rag_core.spi.noop and are exercised by tests/contract/test_*_cache.py.

Related

  • ADR-0008 — plan_hash is the cache key for retrieval + answer caches.
  • ADR-0021 + tiered-cache.md — the Step 4.1 L0/L1/L2 production implementation.
  • reference/cache.md — rag-cache public API.
  • TRACKER.md Steps 1.1e (SPI split) + 4.1 (production implementation ✅).
  • performance.md — hit-rate targets and observability.