feat(backends): storage backend plugins — pgvector, Qdrant, Redis, S3 (Step 1.1)

Makefile

@@ -4,7 +4,7 @@ PYTEST        := uv run pytest
 RUFF          := uv run ruff
 MYPY          := uv run mypy
 
-.PHONY: help bootstrap lint fmt test test-contract test-logs schemas config-schema config-validate dev dev-full dev-down dev-wait dev-seed dev-reset docker clean audit infra-init infra-validate infra-plan helm-lint helm-template
+.PHONY: help bootstrap lint fmt test test-contract test-logs test-integration test-backends schemas config-schema config-validate dev dev-full dev-down dev-wait dev-seed dev-reset docker clean audit infra-init infra-validate infra-plan helm-lint helm-template
 
 help: ## Show this help
 	@grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) \
@@ -54,6 +54,12 @@ test-contract: ## Run SPI conformance suites only
 test-logs: ## Run logging schema + PII + registry gates
 	$(PYTEST) tests/logs/ -v
 
+test-integration: ## Run integration tests (requires: make dev + make dev-wait)
+	$(PYTEST) tests/integration/ -v -m integration
+
+test-backends: ## Run backend unit tests (LocalFileStorage — no services required)
+	$(PYTEST) tests/integration/test_local_storage.py -v
+
 # ---------------------------------------------------------------------------
 # Schemas
 # ---------------------------------------------------------------------------
@@ -65,9 +71,9 @@ schemas: ## Generate JSON Schema + Proto artifacts → dist/schemas/
 # ---------------------------------------------------------------------------
 # Local dev stack
 # ---------------------------------------------------------------------------
-dev: ## Start core local stack (postgres, redis, qdrant)
+dev: ## Start core local stack (postgres, redis, qdrant, minio)
 	docker compose --profile core up -d
-	@echo "\033[32m✓ Core stack running. Postgres=5432, Redis=6379, Qdrant=6333\033[0m"
+	@echo "\033[32m✓ Core stack running. Postgres=5432, Redis=6379, Qdrant=6333, MinIO=9000\033[0m"
 
 dev-full: ## Start core + full observability stack
 	docker compose --profile core --profile observability up -d

TRACKER.md

@@ -10,7 +10,7 @@
 
 **Last updated:** 2026-05-24
 **Current phase:** Phase 1 — Ingestion + Knowledge Store
-**Next action:** Phase 1 Step 1.1 — Storage backends
+**Next action:** Phase 1 Step 1.2 — Connectors framework
 
 ---
 
@@ -30,14 +30,14 @@
 | Phase | Title | Steps | ✅ Done | Remaining |
 |-------|-------|------:|-------:|----------:|
 | 0 | Foundation | 13 | **13** | 0 |
-| 1 | Ingestion + Knowledge Store | 10 | 0 | 10 |
+| 1 | Ingestion + Knowledge Store | 10 | **1** | 9 |
 | 2 | Retrieval Engine | 10 | 0 | 10 |
 | 3 | Gateway & Agent Runtime | 11 | 0 | 11 |
 | 4 | Reliability | 6 | 0 | 6 |
 | 5 | Eval & Observability | 7 | 0 | 7 |
 | 6 | Governance & Tenancy | 10 | 0 | 10 |
 | 7 | Pilot, Harden, GA | 10 | 0 | 10 |
-| **Total** | | **77** | **13** | **64** |
+| **Total** | | **77** | **14** | **63** |
 
 ---
 
@@ -65,7 +65,7 @@
 
 | Step | Title | Status | Branch | PR | Key Deliverables |
 |------|-------|--------|--------|----|-----------------|
-| 1.1 | Storage backends | ⏳ | — | — | Postgres/pgvector plugin, Qdrant plugin, Redis plugin, S3/GCS/Azure blob plugin; all pass VectorStore conformance suite |
+| 1.1 | Storage backends | 🚧 | `build/phase-1/step-1.1-storage-backends` | — | `rag-backends` package: `PgVectorStore` (asyncpg + pgvector, ivfflat), `QdrantVectorStore` (query_points API), `RedisCache`, `S3Storage` (aioboto3, MinIO-compatible), `LocalFileStorage`; integration tests (skip-if-no-service); MinIO added to dev stack; `task test-integration` + `task test-backends`; ADR-0004 |
 | 1.2 | Connectors framework | ⏳ | — | — | `Connector` SPI implementation; built-in: filesystem, S3, GCS; crawler base class |
 | 1.3 | Document parsers | ⏳ | — | — | PDF, DOCX, PPTX, XLSX, HTML, Markdown, plain text, JSON, CSV, YAML parsers; MIME detection |
 | 1.4 | OCR pipeline | ⏳ | — | — | Tesseract + PaddleOCR plugins; image region extraction; confidence scoring |

Taskfile.yml

@@ -41,6 +41,11 @@ tasks:
       - "{{.RUFF}} format --check ."
       - "{{.MYPY}} packages/ apps/gateway/"
 
+  lint:backends:
+    desc: mypy --strict on the backends package only
+    cmds:
+      - "{{.MYPY}} packages/backends/"
+
   lint:logging:
     desc: "RAG001 — ban logging.getLogger() outside rag_core.logging"
     cmds:
@@ -90,6 +95,16 @@ tasks:
     cmds:
       - "{{.PYTEST}} tests/eval/ -v"
 
+  test-integration:
+    desc: "Run integration tests (requires: task dev + task dev-wait)"
+    cmds:
+      - "{{.PYTEST}} tests/integration/ -v -m integration"
+
+  test-backends:
+    desc: "Run backend unit tests (LocalFileStorage — no services required)"
+    cmds:
+      - "{{.PYTEST}} tests/integration/test_local_storage.py -v"
+
   # ---------------------------------------------------------------------------
   # Schemas
   # env: sets PYTHONPATH cross-platform (Task handles Windows vs Unix syntax)

docker-compose.yml

@@ -45,6 +45,40 @@ services:
       timeout: 5s
       retries: 5
 
+  # MinIO — S3-compatible object storage for local dev + integration tests
+  minio:
+    image: minio/minio:RELEASE.2024-07-16T23-46-41Z
+    profiles: [core]
+    command: server /data --console-address ":9001"
+    environment:
+      MINIO_ROOT_USER: minioadmin
+      MINIO_ROOT_PASSWORD: minioadmin
+    ports:
+      - "9000:9000"   # S3 API
+      - "9001:9001"   # Web console
+    volumes:
+      - minio_data:/data
+    healthcheck:
+      test: ["CMD", "mc", "ready", "local"]
+      interval: 10s
+      timeout: 5s
+      retries: 5
+
+  # MinIO client — creates the default bucket on first start
+  minio-init:
+    image: minio/mc:RELEASE.2024-07-11T18-01-28Z
+    profiles: [core]
+    depends_on:
+      minio:
+        condition: service_started
+    entrypoint: >
+      /bin/sh -c "
+        until mc alias set local http://minio:9000 minioadmin minioadmin; do sleep 2; done;
+        mc mb --ignore-existing local/rag-dev;
+        mc mb --ignore-existing local/rag-integration-test;
+        echo 'MinIO buckets ready.';
+      "
+
   elasticsearch:
     image: elasticsearch:8.14.3
     profiles: [core]
@@ -127,3 +161,4 @@ volumes:
   qdrant_data:
   es_data:
   grafana_data:
+  minio_data:

docs/README.md

@@ -8,12 +8,14 @@
 | [high-level-architecture.svg](architecture/high-level-architecture.svg) | Layered architecture diagram (SVG) |
 | [eval-skeleton.md](architecture/eval-skeleton.md) | Eval framework architecture: golden-set schema, metric functions, RAGAS spike, `ragctl eval` CLI, extension points |
 | [iac.md](architecture/iac.md) | IaC overview: Terraform module design, Helm chart structure, dev/prod environments, extension points |
+| [storage-backends.md](architecture/storage-backends.md) | Storage backend architecture: PgVector, Qdrant, Redis, S3/MinIO, tenant isolation, integration test strategy |
 
 ## reference/
 
 | File | Description |
 |------|-------------|
 | [ragctl.md](reference/ragctl.md) | Full `ragctl` command reference — public usage, internals, extension points |
+| [backends.md](reference/backends.md) | `rag-backends` reference — PgVectorStore, QdrantVectorStore, RedisCache, S3Storage, LocalFileStorage |
 
 ## guides/
 
@@ -29,6 +31,7 @@
 | [ADR-0001-monorepo-and-tech-stack.md](adr/ADR-0001-monorepo-and-tech-stack.md) | Decision: uv workspaces, Pydantic v2, Taskfile, Python 3.12+ |
 | [ADR-0002-eval-framework.md](adr/ADR-0002-eval-framework.md) | Decision: pure-Python Tier 1 metrics always-on; RAGAS as optional Tier 2 |
 | [ADR-0003-iac-kubernetes-native.md](adr/ADR-0003-iac-kubernetes-native.md) | Decision: Kubernetes-native Terraform modules over cloud-provider-specific RDS/ElastiCache |
+| [ADR-0004-storage-backends.md](adr/ADR-0004-storage-backends.md) | Decision: single `rag-backends` package, MinIO for S3-compatible dev storage, graceful integration test skip |
 
 ## research/
 

docs/adr/ADR-0004-storage-backends.md

@@ -0,0 +1,85 @@
+# ADR-0004 — Storage Backend Architecture
+
+**Status:** Accepted  
+**Date:** 2026-05-24  
+**Deciders:** Core team  
+**Step:** Phase 1 Step 1.1
+
+---
+
+## Context
+
+Phase 1 requires real persistent storage backends to replace the in-memory noop
+implementations used in Phase 0.  Four categories of backends are needed:
+
+| Category | Use case |
+|----------|----------|
+| Dense vector store | ANN retrieval for embeddings |
+| Cache | Semantic cache, result cache |
+| Object storage | Raw document blobs, chunk payloads |
+
+Multiple vendors are needed to avoid lock-in.
+
+---
+
+## Decision
+
+### 1. Single `rag-backends` workspace package
+
+All real backend implementations live in `packages/backends/` (`rag-backends`).
+This keeps `rag-core` clean and the SPI boundary stable.  Consumer packages
+only import from `rag-core.spi` (ABCs) — `rag-backends` is swapped in at
+the composition root.
+
+### 2. VectorStore backends: PgVectorStore and QdrantVectorStore
+
+- **PgVectorStore** — asyncpg + pgvector; `ivfflat` cosine index;
+  `rag_vector_store` table separate from application tables.
+- **QdrantVectorStore** — single collection with payload-based tenant filter;
+  `query_points` API (qdrant-client ≥ 1.10).
+
+Both have an `initialize()` method (idempotent DDL / collection creation).
+
+### 3. Corpus ID gap in Embedding
+
+`rag_core.types.Embedding` does not carry `corpus_id`.  The VectorStore SPI
+takes `corpus_ids` as a filter parameter, but backends store `corpus_id = ""`
+for now and skip corpus filtering when the list is empty.
+
+**Resolution:** Phase 1.8 (embedder pipeline) will extend `Embedding` with
+`corpus_id`.  Backends will be updated at that point.  The gap is tracked in
+[TRACKER.md](../../TRACKER.md).
+
+### 4. Cache backend: RedisCache
+
+`RedisCache` implements the `Cache` SPI using `redis.asyncio`.  It uses a
+configurable key prefix (`"rag:"` by default) to namespace keys and optional
+TTL on every `set()` call.
+
+### 5. Object storage: S3Storage + LocalFileStorage
+
+- **S3Storage** — aioboto3; works with AWS S3 and MinIO.
+- **LocalFileStorage** — aiofiles; path-traversal protected; ideal for dev and
+  testing without cloud credentials.
+
+**MinIO** is added to the local dev stack (`task dev`) as an S3-compatible
+backend on port 9000.  Two buckets are created automatically: `rag-dev` and
+`rag-integration-test`.
+
+### 6. Integration tests skip gracefully
+
+All integration tests check backend reachability in a session-scoped fixture
+and call `pytest.skip()` if the service is not running.  This keeps the
+standard CI passing without Docker and lets engineers run `task test-integration`
+against the local stack.
+
+---
+
+## Consequences
+
+- **Positive:** Real persistence available from Phase 1.1; all tests self-skip
+  without services; SPI boundary unchanged.
+- **Negative:** `corpus_id` filtering in vector stores is a stub until Phase
+  1.8 — callers must pass `corpus_ids=[]` or accept unfiltered results.
+- **Neutral:** numpy added as a dependency for pgvector's asyncpg codec;
+  acceptable given it is already ubiquitous in the ML stack.

docs/architecture/storage-backends.md

@@ -0,0 +1,92 @@
+# Storage Backends Architecture
+
+## Overview
+
+Phase 1 Step 1.1 introduces the first real backend implementations.  Each
+backend is a concrete class that implements a `rag-core` SPI ABC.  The SPI
+boundary is unchanged — callers never import from `rag_backends` directly;
+they depend on the ABC and receive a backend instance via dependency injection.
+
+```
+┌─────────────────────────────────────────────┐
+│          Application / Gateway              │
+│  injects concrete backend at startup        │
+└───────────┬─────────────────────────────────┘
+            │ uses
+            ▼
+┌──────────────────────┐
+│  rag-core SPI ABCs   │  VectorStore, Cache, Storage
+└──────────┬───────────┘
+           │ implements
+           ▼
+┌──────────────────────────────────────────────────────────┐
+│  rag-backends                                             │
+│                                                           │
+│  vector/pgvector.py  →  PgVectorStore  →  Postgres       │
+│  vector/qdrant.py    →  QdrantVectorStore → Qdrant       │
+│  cache/redis.py      →  RedisCache     →  Redis          │
+│  storage/s3.py       →  S3Storage      →  S3 / MinIO     │
+│  storage/local.py    →  LocalFileStorage → filesystem    │
+└──────────────────────────────────────────────────────────┘
+```
+
+## Tenant isolation strategy
+
+| Backend | Isolation mechanism |
+|---------|---------------------|
+| PgVectorStore | `WHERE tenant_id = $1` on every query + delete |
+| QdrantVectorStore | Payload filter `tenant_id == X` on every search + delete |
+| RedisCache | Key prefix by caller convention (no server-side isolation) |
+| S3Storage | Object key prefix (caller convention) |
+
+## Initialization pattern
+
+Both VectorStore backends expose an async `initialize()` method.  This method
+is idempotent (uses `IF NOT EXISTS` / checks collection existence) and must be
+called once at application startup before the first read/write.
+
+```python
+# at startup
+pg_store = PgVectorStore(dsn=..., dimension=1536)
+await pg_store.initialize()
+
+# inject into pipeline
+pipeline = IngestionPipeline(vector_store=pg_store, ...)
+```
+
+## Local dev stack
+
+The `task dev` command (`docker compose --profile core up -d`) now starts
+MinIO alongside Postgres, Redis, and Qdrant.  A `minio-init` sidecar creates
+the default buckets on first launch.
+
+| Service | Port | Default credentials |
+|---------|------|---------------------|
+| MinIO S3 API | 9000 | minioadmin / minioadmin |
+| MinIO console | 9001 | minioadmin / minioadmin |
+
+## Integration tests
+
+Integration tests live in `tests/integration/` and are marked
+`@pytest.mark.integration`.  Each test file has a session-scoped fixture that
+calls `pytest.skip()` if the required service is not reachable.
+
+```bash
+task dev          # start services
+task dev-wait     # wait for health checks to pass
+task test-integration   # run all integration tests
+```
+
+The standard CI pipeline does not run integration tests (no Docker services).
+A separate CI workflow will be added in Phase 4 to run them against a real
+stack.
+
+## Extension points
+
+To add a new VectorStore backend:
+
+1. Create `packages/backends/src/rag_backends/vector/<name>.py`
+2. Implement `VectorStore` ABC (upsert, query, delete, health)
+3. Add an `initialize()` method for schema/collection setup
+4. Add an integration test in `tests/integration/test_<name>.py`
+5. Export from `rag_backends.vector.__init__` and `rag_backends.__init__`