feat: complete data-ml integration for Postgres

.github/workflows/validate.yml

@@ -6,11 +6,45 @@ on:
     branches: [ main ]
 
 jobs:
-  readme:
+  validate:
     runs-on: ubuntu-latest
+    services:
+      postgres:
+        image: postgres:16
+        env:
+          POSTGRES_USER: aurekai
+          POSTGRES_PASSWORD: aurekai
+          POSTGRES_DB: aurekai
+        ports:
+          - 5432:5432
+        options: >-
+          --health-cmd "pg_isready -U aurekai -d aurekai"
+          --health-interval 10s
+          --health-timeout 5s
+          --health-retries 10
+    env:
+      DATABASE_URL: postgresql://aurekai:aurekai@localhost:5432/aurekai
     steps:
       - uses: actions/checkout@v4
-      - name: Ensure scaffold files exist
+
+      - name: Ensure required files exist
         run: |
           test -f README.md
           test -f LICENSE
+          test -f .release-versions.env
+          test -f scripts/init-db.sh
+          test -f tests/validate-schemas.sh
+          test -f tests/validate-scripts.sh
+
+      - name: Install dependencies
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y postgresql-client
+          python3 -m pip install --upgrade pip
+          python3 -m pip install -r requirements.txt
+
+      - name: Validate schemas
+        run: bash tests/validate-schemas.sh "$DATABASE_URL"
+
+      - name: Validate scripts
+        run: bash tests/validate-scripts.sh "$DATABASE_URL"

.release-versions.env

@@ -0,0 +1,4 @@
+AUREKAI_VERSION=0.8.0-alpha.4
+AKAI_PACKAGE_VERSION=0.8.0-alpha.4
+AUREKAI_MANIFEST_SCHEMA=aurekai.deploy.v1
+HELM_CHART_VERSION=0.8.1

README.md

@@ -4,20 +4,55 @@
 
 # aurekai-postgres
 
-Aurekai integration surface for Postgres.
+Aurekai integration surface for PostgreSQL - robust, transactional, multi-tenant data-ml storage for model memory, SAE dictionaries, semantic cache, manifests, and proof bundles.
 
-Status: planned
-Type: data
+Status: active
+Type: data-ml
+
+## Overview
+
+`aurekai-postgres` provides PostgreSQL DDL schemas and operational scripts to persist and query Aurekai artifacts with production-grade guarantees: ACID transactions, row-level security compatibility, indexing, and JSONB-native querying.
 
 ## Core Template Set
 
-- doctor-deep
-- manifest-verify
-- model-memory-pack
-- sae-audit
-- semantic-cache-bench
-- proof-bundle-export
-- release-gate
+| Template | Description |
+|---|---|
+| `doctor-deep` | Connection checks, schema integrity, and optional `akai doctor --deep` |
+| `manifest-verify` | Validate `aurekai.deploy.v1` manifest record in Postgres |
+| `model-memory-pack` | Insert `.akmodel` / `.bfmodel` / `.akfpqx` / `.bffpqx` metadata + payload |
+| `sae-audit` | Audit SAE dictionary records by extension/operator/layer |
+| `semantic-cache-bench` | Benchmark cache write/read throughput on Postgres |
+| `proof-bundle-export` | Export proof bundle rows to `out/proof-bundle.json` |
+| `release-gate` | Verify populated tables and release constraints before publish |
+
+## Quick Start
+
+```bash
+# 1) Start local postgres (optional)
+docker compose -f examples/docker-compose.postgres.yaml up -d
+
+# 2) Export connection string
+export DATABASE_URL=postgresql://aurekai:aurekai@localhost:5432/aurekai
+
+# 3) Initialize schema
+bash scripts/init-db.sh "$DATABASE_URL"
+
+# 4) Run deep checks
+bash scripts/doctor-deep.sh "$DATABASE_URL"
+
+# 5) Run release gate
+bash scripts/release-gate.sh "$DATABASE_URL"
+```
+
+## Directory Layout
+
+```
+sql/                   PostgreSQL schema files
+scripts/               Operational template scripts
+examples/              Local compose + sample payloads
+tests/                 Validation scripts for CI/local
+docs/                  Quickstart and reference docs
+```
 
 ## Canonical References
 

docs/quickstart.md

@@ -0,0 +1,40 @@
+# Quickstart
+
+## 1) Start PostgreSQL
+
+Use Docker for local development:
+
+```bash
+docker compose -f examples/docker-compose.postgres.yaml up -d
+```
+
+## 2) Set DATABASE_URL
+
+```bash
+export DATABASE_URL=postgresql://aurekai:aurekai@localhost:5432/aurekai
+```
+
+## 3) Install Python dependency for benchmark scripts
+
+```bash
+python3 -m pip install -r requirements.txt
+```
+
+## 4) Initialize schema
+
+```bash
+bash scripts/init-db.sh "$DATABASE_URL"
+```
+
+## 5) Run diagnostics
+
+```bash
+bash scripts/doctor-deep.sh "$DATABASE_URL"
+```
+
+## 6) Run full validation
+
+```bash
+bash tests/validate-schemas.sh "$DATABASE_URL"
+bash tests/validate-scripts.sh "$DATABASE_URL"
+```

docs/schema.md

@@ -0,0 +1,21 @@
+# Schema Reference
+
+## Tables
+
+1. `model_memory`: metadata + optional payload for `.akmodel`, `.bfmodel`, `.akfpqx`, `.bffpqx`
+2. `sae_dictionaries`: metadata + optional payload for `.aksae`, `.bfsae`
+3. `semantic_cache`: key/value cache with optional embedding byte payload
+4. `manifests`: release manifests (`aurekai.deploy.v1`) with JSONB raw payload
+5. `proof_bundles`: proof outputs and status for recipe runs
+
+## Postgres Types Used
+
+1. `BYTEA` for binary artifacts and embeddings
+2. `JSONB` for metadata/raw payload and proof records
+3. `TIMESTAMPTZ` for UTC event timestamps
+
+## Indexing Strategy
+
+1. B-tree indexes on high-cardinality lookup fields (`artifact_name`, `operator_id`, `version`, `status`)
+2. GIN indexes on JSONB columns for fast metadata and payload filtering
+3. Unique constraint on `manifests(name, version)`

docs/scripts.md

@@ -0,0 +1,12 @@
+# Scripts Reference
+
+Each script accepts `DATABASE_URL` as first positional arg, or via environment variable.
+
+1. `scripts/init-db.sh`: apply all schema files
+2. `scripts/doctor-deep.sh`: verify connectivity, required tables, row counts, optional `akai doctor --deep`
+3. `scripts/manifest-verify.sh`: ensure specific manifest exists
+4. `scripts/model-memory-pack.sh`: insert binary artifact into `model_memory`
+5. `scripts/sae-audit.sh`: summarize SAE records and checksum health
+6. `scripts/semantic-cache-bench.sh`: benchmark cache write/read operations
+7. `scripts/proof-bundle-export.sh`: export `proof_bundles` rows to JSON
+8. `scripts/release-gate.sh`: enforce release readiness checks

examples/docker-compose.postgres.yaml

@@ -0,0 +1,15 @@
+version: '3.9'
+services:
+  postgres:
+    image: postgres:16
+    environment:
+      POSTGRES_USER: aurekai
+      POSTGRES_PASSWORD: aurekai
+      POSTGRES_DB: aurekai
+    ports:
+      - "5432:5432"
+    healthcheck:
+      test: ["CMD-SHELL", "pg_isready -U aurekai -d aurekai"]
+      interval: 5s
+      timeout: 5s
+      retries: 20

examples/sample-akmodel.json

@@ -0,0 +1,9 @@
+{
+  "name": "my-operator-v1",
+  "ext": ".akmodel",
+  "operator_id": "akai_run_operator",
+  "version": "0.8.0-alpha.4",
+  "schema_version": "aurekai.deploy.v1",
+  "size_bytes": 204800,
+  "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
+}

examples/sample-aurekai.manifest.json

@@ -0,0 +1,11 @@
+{
+  "schema_version": "aurekai.deploy.v1",
+  "name": "aurekai",
+  "version": "0.8.0-alpha.4",
+  "operator_count": 89,
+  "runtime_ref": "git:1ba1f19",
+  "helm_chart_ver": "0.8.1",
+  "pypi_ver": "0.8.0a3",
+  "npm_ver": "0.8.0-alpha.4",
+  "jsr_ver": "0.8.0-alpha.3"
+}

examples/sample-db-init.sh

@@ -0,0 +1,48 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REPO_ROOT="$SCRIPT_DIR/.."
+DB_URL="${DATABASE_URL:-postgresql://aurekai:aurekai@localhost:5432/aurekai}"
+
+echo "Using DATABASE_URL=$DB_URL"
+bash "$REPO_ROOT/scripts/init-db.sh" "$DB_URL"
+
+python3 - <<PYEOF
+import json
+import os
+import psycopg
+
+db_url = os.environ.get("DATABASE_URL", "postgresql://aurekai:aurekai@localhost:5432/aurekai")
+manifest = json.load(open("$REPO_ROOT/examples/sample-aurekai.manifest.json", "r", encoding="utf-8"))
+with psycopg.connect(db_url, autocommit=True) as conn:
+    with conn.cursor() as cur:
+        cur.execute(
+            """
+            INSERT INTO manifests (schema_version, name, version, operator_count, runtime_ref, helm_chart_ver, pypi_ver, npm_ver, jsr_ver, raw_json)
+            VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s::jsonb)
+            ON CONFLICT (name, version) DO NOTHING
+            """,
+            (
+                manifest["schema_version"],
+                manifest["name"],
+                manifest["version"],
+                manifest["operator_count"],
+                manifest["runtime_ref"],
+                manifest["helm_chart_ver"],
+                manifest["pypi_ver"],
+                manifest["npm_ver"],
+                manifest["jsr_ver"],
+                json.dumps(manifest),
+            ),
+        )
+        cur.execute("INSERT INTO sae_dictionaries (artifact_name, artifact_ext, operator_id, layer_index, feature_count, size_bytes, sha256) VALUES (%s,%s,%s,%s,%s,%s,%s)",
+            ("layer-0-features.aksae", ".aksae", "akai_sae_encode", 0, 512, 4096, "abc123def456abc123def456abc123def456abc123def456abc123def456abc123"))
+        cur.execute("INSERT INTO proof_bundles (recipe_name, status, operator_id, proof_json) VALUES (%s,%s,%s,%s::jsonb)",
+            ("bench.recipe.json", "ok", "akai_run_operator", json.dumps({"status": "ok", "proof": "generated"})))
+        cur.execute("INSERT INTO model_memory (artifact_name, artifact_ext, size_bytes, sha256) VALUES (%s,%s,%s,%s)",
+            ("demo.akmodel", ".akmodel", 1024, "deadbeef"))
+PYEOF
+
+bash "$REPO_ROOT/scripts/semantic-cache-bench.sh" "$DB_URL" 100
+bash "$REPO_ROOT/scripts/release-gate.sh" "$DB_URL"

requirements.txt

@@ -0,0 +1 @@
+psycopg[binary]>=3.2,<3.4

scripts/common.sh

@@ -0,0 +1,12 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+DATABASE_URL="${1:-${DATABASE_URL:-}}"
+if [ -z "$DATABASE_URL" ]; then
+    echo "DATABASE_URL not provided. Pass as first arg or export DATABASE_URL." >&2
+    exit 1
+fi
+
+psql_exec() {
+    psql "$DATABASE_URL" -v ON_ERROR_STOP=1 "$@"
+}

scripts/doctor-deep.sh

@@ -0,0 +1,29 @@
+#!/usr/bin/env bash
+# Usage: bash scripts/doctor-deep.sh <database_url>
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+source "$SCRIPT_DIR/common.sh" "${1:-}"
+
+echo "[doctor-deep] Checking Postgres connectivity"
+psql_exec -t -A -c "SELECT 'ok'" | grep -q ok
+
+echo "[doctor-deep] Checking required tables"
+for tbl in model_memory sae_dictionaries semantic_cache manifests proof_bundles; do
+    count=$(psql_exec -t -A -c "SELECT count(*) FROM pg_tables WHERE schemaname='public' AND tablename='$tbl';")
+    if [ "$count" != "1" ]; then
+        echo "[doctor-deep] Missing table: $tbl" >&2
+        exit 1
+    fi
+    rows=$(psql_exec -t -A -c "SELECT count(*) FROM $tbl;")
+    echo "  $tbl: $rows rows"
+done
+
+if command -v akai >/dev/null 2>&1; then
+    echo "[doctor-deep] Running akai doctor --deep"
+    akai doctor --deep
+else
+    echo "[doctor-deep] akai not installed; skipping akai doctor --deep"
+fi
+
+echo "[doctor-deep] PASS"

scripts/init-db.sh

@@ -0,0 +1,20 @@
+#!/usr/bin/env bash
+# Usage: bash scripts/init-db.sh <database_url>
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+source "$SCRIPT_DIR/common.sh" "${1:-}"
+
+echo "[init-db] Applying schemas to $DATABASE_URL"
+for schema in \
+    schema-model-memory.sql \
+    schema-sae.sql \
+    schema-semantic-cache.sql \
+    schema-manifests.sql \
+    schema-proof-bundles.sql; do
+    echo "  Applying $schema"
+    psql_exec -f "$SCRIPT_DIR/../sql/$schema" >/dev/null
+done
+
+echo "[init-db] Tables:"
+psql_exec -t -A -c "SELECT tablename FROM pg_tables WHERE schemaname='public' ORDER BY tablename;"

scripts/manifest-verify.sh

@@ -0,0 +1,17 @@
+#!/usr/bin/env bash
+# Usage: bash scripts/manifest-verify.sh <database_url> <name> <version>
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+source "$SCRIPT_DIR/common.sh" "${1:-}"
+NAME="${2:?manifest name required}"
+VERSION="${3:?manifest version required}"
+
+count=$(psql_exec -t -A -c "SELECT count(*) FROM manifests WHERE name='$NAME' AND version='$VERSION' AND schema_version='aurekai.deploy.v1';")
+if [ "$count" -lt 1 ]; then
+    echo "[manifest-verify] FAIL: manifest not found" >&2
+    exit 1
+fi
+
+psql_exec -P pager=off -c "SELECT id, name, version, schema_version, operator_count, created_at FROM manifests WHERE name='$NAME' AND version='$VERSION';"
+echo "[manifest-verify] PASS"