feat(infra): IaC foundation — Terraform modules + Helm chart skeleton (Step 0.9)

.github/workflows/ci.yml

@@ -22,6 +22,7 @@ jobs:
       pull-requests: read
     outputs:
       python: ${{ steps.filter.outputs.python }}
+      infra: ${{ steps.filter.outputs.infra }}
     steps:
       - uses: actions/checkout@v4
       - uses: dorny/paths-filter@v4
@@ -34,6 +35,9 @@ jobs:
               - 'tests/**'
               - 'pyproject.toml'
               - 'uv.lock'
+            infra:
+              - 'infra/terraform/**'
+              - 'infra/helm/**'
 
   # ---------------------------------------------------------------------------
   # Lint + type-check + unit tests (Ubuntu + macOS + Windows)
@@ -116,6 +120,45 @@ jobs:
       - run: uv sync --all-packages
       - run: uv run pytest tests/logs/ -v
 
+  # ---------------------------------------------------------------------------
+  # IaC validation — terraform validate + helm lint (no live cluster needed)
+  # ---------------------------------------------------------------------------
+  iac:
+    name: IaC validate (terraform + helm)
+    needs: changes
+    if: needs.changes.outputs.infra == 'true'
+    runs-on: ubuntu-22.04
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Setup Terraform
+        uses: hashicorp/setup-terraform@v3
+        with:
+          terraform_version: "~1.9"
+
+      - name: Setup Helm
+        uses: azure/setup-helm@v4
+        with:
+          version: "latest"
+
+      - name: Terraform validate — dev
+        working-directory: infra/terraform/environments/dev
+        run: |
+          terraform init -backend=false
+          terraform validate
+
+      - name: Terraform validate — prod
+        working-directory: infra/terraform/environments/prod
+        run: |
+          terraform init -backend=false
+          terraform validate
+
+      - name: Helm lint — rag-platform
+        run: helm lint infra/helm/rag-platform/
+
+      - name: Helm template — dry run
+        run: helm template rag-platform infra/helm/rag-platform/ > /dev/null
+
   # ---------------------------------------------------------------------------
   # Dependency CVE audit
   # ---------------------------------------------------------------------------
@@ -140,7 +183,7 @@ jobs:
   # ---------------------------------------------------------------------------
   ci-pass:
     name: CI passed
-    needs: [lint-test, secrets-scan, log-gates, audit]
+    needs: [lint-test, secrets-scan, log-gates, audit, iac]
     if: always()
     runs-on: ubuntu-22.04
     steps:

Makefile

@@ -4,7 +4,7 @@ PYTEST        := uv run pytest
 RUFF          := uv run ruff
 MYPY          := uv run mypy
 
-.PHONY: help bootstrap lint fmt test test-contract test-logs schemas config-schema config-validate dev dev-full dev-down dev-wait dev-seed dev-reset docker clean audit
+.PHONY: help bootstrap lint fmt test test-contract test-logs schemas config-schema config-validate dev dev-full dev-down dev-wait dev-seed dev-reset docker clean audit infra-init infra-validate infra-plan helm-lint helm-template
 
 help: ## Show this help
 	@grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) \
@@ -97,6 +97,25 @@ docker: ## Build all Docker images
 audit: ## Run pip-audit for dependency CVEs
 	uv run pip-audit
 
+# ---------------------------------------------------------------------------
+# IaC — Terraform + Helm
+# ---------------------------------------------------------------------------
+infra-init: ## terraform init for the dev environment
+	terraform -chdir=infra/terraform/environments/dev init
+
+infra-validate: ## terraform validate for dev + prod environments
+	terraform -chdir=infra/terraform/environments/dev validate
+	terraform -chdir=infra/terraform/environments/prod validate
+
+infra-plan: ## terraform plan for the dev environment
+	terraform -chdir=infra/terraform/environments/dev plan
+
+helm-lint: ## Lint the rag-platform Helm chart
+	helm lint infra/helm/rag-platform/
+
+helm-template: ## Render Helm templates (dry run)
+	helm template rag-platform infra/helm/rag-platform/
+
 # ---------------------------------------------------------------------------
 # Cleanup
 # ---------------------------------------------------------------------------

TRACKER.md

@@ -8,9 +8,9 @@
 > Use `Taskfile.yml` for task targets (cross-platform), `Makefile` for Unix convenience.
 > CI matrix always includes `ubuntu-22.04`, `macos-14`, and `windows-latest`.
 
-**Last updated:** 2026-05-23
+**Last updated:** 2026-05-24
 **Current phase:** Phase 0 — Foundation
-**Next action:** Phase 0 Step 0.9 — IaC foundation
+**Next action:** Phase 0 Step 0.10 — ragctl CLI scaffold
 
 ---
 
@@ -29,15 +29,15 @@
 
 | Phase | Title | Steps | ✅ Done | Remaining |
 |-------|-------|------:|-------:|----------:|
-| 0 | Foundation | 13 | **11** | 2 |
+| 0 | Foundation | 13 | **12** | 1 |
 | 1 | Ingestion + Knowledge Store | 10 | 0 | 10 |
 | 2 | Retrieval Engine | 10 | 0 | 10 |
 | 3 | Gateway & Agent Runtime | 11 | 0 | 11 |
 | 4 | Reliability | 6 | 0 | 6 |
 | 5 | Eval & Observability | 7 | 0 | 7 |
 | 6 | Governance & Tenancy | 10 | 0 | 10 |
 | 7 | Pilot, Harden, GA | 10 | 0 | 10 |
-| **Total** | | **77** | **11** | **66** |
+| **Total** | | **77** | **12** | **65** |
 
 ---
 
@@ -56,7 +56,7 @@
 | 0.7b | Structured logging foundation | ✅ | `build/phase-0/step-0.7b-structured-logging` | — | Shared logger package (`packages/core/src/rag_core/logging.py`), event registry (`events.py`), RAG001 pre-commit hook + ruff T201 rule, schema+PII CI gates in `tests/logs/` (50 tests; log-gates job now blocking). **Follow-up (0.7b-observability-package):** extracted `packages/observability/` (`rag-observability`) with full 7-field JSON schema (ts, level, service, module, msg, env, version), `set_log_context()` contextvar-based context manager, `_context.py`; `rag_core.logging` and `rag_core.events` reduced to backwards-compat shims (391 tests pass). |
 | 0.7c | Audit log skeleton | ✅ | `build/phase-0/step-0.7c-audit-log-skeleton` | — | `AuditStore` SPI (append/events/verify_chain), `NoopAuditStore` (SHA-256 hash chain), `AuditWriter` facade (store + structured log), 14 conformance tests |
 | 0.8 | Eval skeleton | ✅ | `build/phase-0/step-0.8-eval-skeleton` | — | `rag_core.eval` domain types (GoldenSample, EvalMetrics, EvalReport), `rag_config.eval` metric functions (recall@k, MRR, citation_precision), RagasAdapter spike, `ragctl eval run/show`, 5-sample golden JSONL fixture, `tests/eval/` harness (39 tests) |
-| 0.9 | IaC foundation | ⏳ | — | — | Terraform modules for core backends, Helm chart skeleton, `infra/` layout |
+| 0.9 | IaC foundation | ✅ | `build/phase-0/step-0.9-iac-foundation` | — | Terraform modules for Postgres/pgvector, Redis, Qdrant, Elasticsearch (Kubernetes-native, Helm provider); `rag-platform` Helm chart (Deployment, Service, ConfigMap, ServiceAccount, HPA, PDB, Ingress); dev + prod environments; `task infra:*` + `task helm:*` targets; ADR-0003 |
 | 0.10 | `ragctl` CLI scaffold | ⏳ | — | — | `packages/ragctl/` CLI (Typer), `ragctl ingest/query/eval/logs/traces/config` top-level commands, shell completion |
 
 ---

Taskfile.yml

@@ -156,6 +156,37 @@ tasks:
     cmds:
       - uv run pip-audit
 
+  # ---------------------------------------------------------------------------
+  # IaC — Terraform + Helm
+  # ---------------------------------------------------------------------------
+  infra:init:
+    desc: terraform init for the dev environment
+    dir: infra/terraform/environments/dev
+    cmds:
+      - terraform init
+
+  infra:validate:
+    desc: terraform validate for dev + prod environments
+    cmds:
+      - terraform -chdir=infra/terraform/environments/dev validate
+      - terraform -chdir=infra/terraform/environments/prod validate
+
+  infra:plan:
+    desc: terraform plan for the dev environment (requires live cluster)
+    dir: infra/terraform/environments/dev
+    cmds:
+      - terraform plan
+
+  helm:lint:
+    desc: Lint the rag-platform Helm chart
+    cmds:
+      - helm lint infra/helm/rag-platform/
+
+  helm:template:
+    desc: Render Helm templates to stdout (dry run)
+    cmds:
+      - helm template rag-platform infra/helm/rag-platform/
+
   # ---------------------------------------------------------------------------
   # Cleanup — platform-specific commands via platforms: key
   # ---------------------------------------------------------------------------

docs/README.md

@@ -7,13 +7,15 @@
 | [RAG-Platform-HLD.md](architecture/RAG-Platform-HLD.md) | High-Level Design: problem statement, layered architecture, pluggable backends, `rag.yaml` contract, deployment topologies, KPIs, risks, glossary |
 | [high-level-architecture.svg](architecture/high-level-architecture.svg) | Layered architecture diagram (SVG) |
 | [eval-skeleton.md](architecture/eval-skeleton.md) | Eval framework architecture: golden-set schema, metric functions, RAGAS spike, `ragctl eval` CLI, extension points |
+| [iac.md](architecture/iac.md) | IaC overview: Terraform module design, Helm chart structure, dev/prod environments, extension points |
 
 ## adr/
 
 | File | Description |
 |------|-------------|
 | [ADR-0001-monorepo-and-tech-stack.md](adr/ADR-0001-monorepo-and-tech-stack.md) | Decision: uv workspaces, Pydantic v2, Taskfile, Python 3.12+ |
 | [ADR-0002-eval-framework.md](adr/ADR-0002-eval-framework.md) | Decision: pure-Python Tier 1 metrics always-on; RAGAS as optional Tier 2 |
+| [ADR-0003-iac-kubernetes-native.md](adr/ADR-0003-iac-kubernetes-native.md) | Decision: Kubernetes-native Terraform modules over cloud-provider-specific RDS/ElastiCache |
 
 ## research/
 

docs/adr/ADR-0003-iac-kubernetes-native.md

@@ -0,0 +1,48 @@
+# ADR-0003 — Kubernetes-native IaC (Terraform + Helm providers)
+
+**Status:** Accepted  
+**Date:** 2026-05-24  
+**Deciders:** Core team
+
+---
+
+## Context
+
+Phase 0 requires an IaC foundation that provisions the four core backends (Postgres/pgvector, Redis, Qdrant, Elasticsearch) and a gateway deployment template.
+
+Two approaches were considered:
+
+1. **Cloud-provider-specific modules** — AWS Terraform modules (RDS, ElastiCache, EC2/ECS for Qdrant). Concrete, well-tested, but locks the platform to AWS for Phase 0.
+2. **Kubernetes-native modules** — Terraform `helm` + `kubernetes` providers wrapping official Helm charts. Works on any conformant K8s cluster (local kind, EKS, GKE, AKS, bare-metal).
+
+---
+
+## Decision
+
+Use **Kubernetes-native Terraform modules** (option 2) for Phase 0.
+
+Each backend module wraps a mature, actively-maintained Helm chart:
+- `bitnami/postgresql` (pgvector image override)
+- `bitnami/redis`
+- `qdrant/qdrant`
+- `bitnami/elasticsearch`
+
+The gateway is deployed via a first-party `rag-platform` Helm chart.
+
+---
+
+## Rationale
+
+- **No cloud lock-in at Phase 0.** Design partners may run on GKE or AKS; committing to AWS-specific resources before Phase 1 would create unnecessary switching cost.
+- **Works locally.** Developers can run `terraform apply` against `docker-desktop` or `kind` — no cloud account required. This is consistent with the project's Docker Compose-first local dev story.
+- **Gradual escape hatch.** When a design partner needs managed RDS instead of in-cluster Postgres, they set `enable_postgres = false` and point the gateway at an external DSN. No Terraform module rewrite needed.
+- **Helm chart reuse.** The same Bitnami charts are the de facto standard for K8s backend deployments. Wrapping them in Terraform gives declarative lifecycle management without reimplementing health checks, PVCs, or RBAC.
+
+---
+
+## Consequences
+
+- **Requires a running K8s cluster** for `terraform plan/apply`. `terraform validate` is cluster-free and runs in CI.
+- **In-cluster Postgres is not HA at Phase 0.** The Bitnami chart supports replication; the prod environment variables expose `master_replicas` and `data_replicas` for Elasticsearch, and `replicas` for Qdrant. Postgres primary-standby replication will be configured in Phase 4 (Reliability).
+- **State management** — dev uses local state; prod requires an S3 backend to be configured before first apply. The `backend "s3"` block in `environments/prod/main.tf` has commented-out placeholders.
+- **Cloud-specific modules deferred** — AWS RDS, ElastiCache, and managed Qdrant Cloud integrations are Phase 1+ concerns and will live under `infra/terraform/modules/aws/` when introduced.

docs/architecture/iac.md

@@ -0,0 +1,133 @@
+# Infrastructure as Code (IaC)
+
+## Overview
+
+AgentContextOS uses **Terraform** to provision all platform backends and a **Helm chart** to deploy the gateway service on Kubernetes.
+Both tools target Kubernetes as the common runtime so the same configuration works on local clusters (kind, Docker Desktop), EKS, GKE, and AKS without modification.
+
+See [ADR-0003](../adr/ADR-0003-iac-kubernetes-native.md) for the decision to use Kubernetes-native IaC rather than cloud-provider-specific modules.
+
+---
+
+## Directory layout
+
+```
+infra/
+├── terraform/
+│   ├── versions.tf                        # provider version constraints
+│   ├── variables.tf                       # root module inputs
+│   ├── outputs.tf                         # root module outputs (hostnames, secrets)
+│   ├── main.tf                            # wires backend modules + namespace
+│   ├── modules/
+│   │   ├── postgres/                      # pgvector via Bitnami chart
+│   │   ├── redis/                         # Redis via Bitnami chart
+│   │   ├── qdrant/                        # Qdrant via official chart
+│   │   └── elasticsearch/                 # Elasticsearch via Bitnami chart
+│   ├── environments/
+│   │   ├── dev/                           # local cluster, minimal resources
+│   │   └── prod/                          # production cluster, HA sizing
+│   └── .gitignore                         # excludes state, .terraform/, tfvars
+└── helm/
+    └── rag-platform/
+        ├── Chart.yaml
+        ├── values.yaml                    # default values
+        ├── values.dev.yaml                # dev overrides (single replica, small resources)
+        └── templates/
+            ├── deployment.yaml            # rag-gateway Deployment
+            ├── service.yaml
+            ├── configmap.yaml             # rag.yaml config
+            ├── serviceaccount.yaml
+            ├── hpa.yaml                   # HorizontalPodAutoscaler
+            ├── pdb.yaml                   # PodDisruptionBudget
+            ├── ingress.yaml
+            └── NOTES.txt
+```
+
+---
+
+## Terraform modules
+
+Each module wraps a Helm chart deployment so backends are managed declaratively alongside application code.
+All four modules share the same contract: `namespace`, `environment`, `storage_size`, and `resources` inputs; `host`, `port`, and (where applicable) `secret_name` outputs.
+
+| Module | Helm chart | Purpose |
+|--------|-----------|---------|
+| `postgres` | `bitnami/postgresql` | pgvector store for documents, chunks, embeddings, audit events |
+| `redis` | `bitnami/redis` | Semantic cache L1, rate-limit counters, session state |
+| `qdrant` | `qdrant/qdrant` | Primary vector store for Phase 1 retrieval |
+| `elasticsearch` | `bitnami/elasticsearch` | BM25 keyword retrieval for Phase 2 hybrid search |
+
+### Switching to managed cloud services (Phase 1+)
+
+In production it is common to replace in-cluster stateful backends with managed equivalents (RDS, ElastiCache, Qdrant Cloud, OpenSearch).
+To do this: set `enable_<backend> = false` in the environment `main.tf` and provide the external connection string via environment variables or a Kubernetes Secret.
+The Helm chart reads backends via `RAG_POSTGRES_HOST`, `RAG_REDIS_HOST`, etc., so no application code changes are needed.
+
+---
+
+## Helm chart
+
+The `rag-platform` chart deploys the `rag-gateway` FastAPI service.
+
+Key design points:
+
+- **ConfigMap-mounted config** — `rag.yaml` is rendered from Helm values and mounted at `/app/config/rag.yaml`. The Deployment `checksum/config` annotation triggers a rolling restart on config changes.
+- **Secret references** — database credentials are read from a Kubernetes Secret (provisioned by the Terraform postgres module). The chart never hardcodes credentials.
+- **HPA + PDB** — enabled by default in prod values; disabled in `values.dev.yaml`.
+- **Pod anti-affinity** — prefers scheduling replicas on separate nodes to survive single-node failures.
+- **Read-only root filesystem** — only `/tmp` is writable (emptyDir volume).
+
+### Usage
+
+```bash
+# Dev (single replica, local backends)
+helm upgrade --install rag-platform infra/helm/rag-platform \
+  -f infra/helm/rag-platform/values.yaml \
+  -f infra/helm/rag-platform/values.dev.yaml \
+  --namespace rag-dev --create-namespace
+
+# Lint
+task helm:lint
+
+# Dry run (render templates)
+task helm:template
+```
+
+---
+
+## Environments
+
+### dev
+
+- State stored locally (`terraform.tfstate`) — not shared.
+- Minimal resource requests (CPU 100m, memory 128 Mi for most backends).
+- Single Qdrant replica, 5 Gi storage per backend.
+- Targets `docker-desktop` Kubernetes context by default; override via `kube_context` variable.
+
+### prod
+
+- Remote state backend (`s3` block) — fill in bucket/key before first `terraform init`.
+- Production sizing: 3 Qdrant replicas, 100–200 Gi storage.
+- Elasticsearch security enabled, Redis auth can be toggled on.
+- Cluster credentials passed as variables (never hardcoded).
+
+---
+
+## Task targets
+
+| Target | Description |
+|--------|-------------|
+| `task infra:init` | `terraform init` for the dev environment |
+| `task infra:validate` | `terraform validate` for dev + prod |
+| `task infra:plan` | `terraform plan` for dev (requires a running cluster) |
+| `task helm:lint` | `helm lint` the rag-platform chart |
+| `task helm:template` | Render templates to stdout (dry run) |
+
+---
+
+## Extension points
+
+- **New backend module** — add a directory under `infra/terraform/modules/`, follow the `variables.tf` / `main.tf` / `outputs.tf` convention, and wire it in `main.tf` with a `count = var.enable_<backend> ? 1 : 0` guard.
+- **New environment** — copy `environments/dev/` to `environments/staging/`, update the backend block and sizing variables.
+- **Additional Helm templates** — add files under `templates/`. Use `_helpers.tpl` macros for consistent label generation.
+- **ServiceMonitor** — once Prometheus Operator is deployed, enable `metrics.serviceMonitor.enabled` in the Qdrant module values to wire up scraping automatically.