feat(llm): llama.cpp come motore predefinito per estrazione ed embedding

README.md

@@ -129,7 +129,7 @@ Data flows through three main paths:
 | **Vector Store** | Redis for AI (RedisSearch + RedisJSON) | latest |
 | **LLM Inference** | Ollama (local, no API key) | latest |
 | **LLM Model** | Llama 3 | latest |
-| **Embedding Model** | nomic-embed-text (768 dim) | latest |
+| **Embedding Model** | Qwen3-Embedding-0.6B (1024 dim) via llama.cpp | Q8_0 |
 | **REST API** | FastAPI + uvicorn | 0.115+ |
 | **Data Models** | Pydantic v2 + pydantic-settings | 2.7+ |
 | **Multi-Agent** | Microsoft Agent Framework (MAF) | latest |
@@ -837,15 +837,19 @@ cp .env.example .env
 | --- | --- | --- |
 | `REDIS_URL` | `redis://redis:6379` | Connection URL (use `localhost` for local dev) |
 | `REDIS_INDEX_NAME` | `kg_vectors` | Vector index name |
-| `REDIS_VECTOR_DIM` | `768` | Vector dimension (depends on embedding model) |
+| `REDIS_VECTOR_DIM` | `1024` | Vector dimension — **must** match the embedding model (1024 Qwen3-Embedding, 768 nomic-embed-text). Cambiarlo impone il rebuild dell'indice |
 
 ### Ollama
 
 | Variable | Default | Description |
 | --- | --- | --- |
 | `OLLAMA_BASE_URL` | `http://ollama:11434` | Ollama URL (use `localhost` for local dev) |
 | `OLLAMA_LLM_MODEL` | `llama3` | Text generation model |
-| `OLLAMA_EMBEDDING_MODEL` | `nomic-embed-text` | Embedding model (768 dim) |
+| `OLLAMA_EMBEDDING_MODEL` | `nomic-embed-text` | Embedding model (768 dim), solo con `EMBEDDING_PROVIDER=ollama` |
+| `KG_LLM_PROVIDER` | `llamacpp` | Estrazione: `llamacpp` / `ollama` / `anthropic` |
+| `EMBEDDING_PROVIDER` | `llamacpp` | Embedding: `llamacpp` / `ollama` |
+| `LLAMACPP_BASE_URL` | `http://localhost:8080` | llama-swap, chat compatibile OpenAI |
+| `LLAMACPP_EMBEDDING_BASE_URL` | `http://localhost:8081` | Server embedding dedicato |
 
 ### Chunking
 

docker-compose.ghcr.yml

@@ -122,9 +122,21 @@ services:
       NEO4J_USER: "${NEO4J_USER:-neo4j}"
       NEO4J_PASSWORD: "${NEO4J_PASSWORD:-password}"
       REDIS_URL: "redis://redis:6379"
+      # Default engine: self-hosted llama.cpp. REDIS_VECTOR_DIM must match the
+      # embedding model's width (1024 for Qwen3-Embedding-0.6B).
+      KG_LLM_PROVIDER: "${KG_LLM_PROVIDER:-llamacpp}"
+      EMBEDDING_PROVIDER: "${EMBEDDING_PROVIDER:-llamacpp}"
+      REDIS_VECTOR_DIM: "${REDIS_VECTOR_DIM:-1024}"
+      LLAMACPP_BASE_URL: "${LLAMACPP_BASE_URL:-http://host.docker.internal:8080}"
+      LLAMACPP_LLM_MODEL: "${LLAMACPP_LLM_MODEL:-qwen3.5-9b}"
+      LLAMACPP_EMBEDDING_BASE_URL: "${LLAMACPP_EMBEDDING_BASE_URL:-http://host.docker.internal:8081}"
+      LLAMACPP_EMBEDDING_MODEL: "${LLAMACPP_EMBEDDING_MODEL:-qwen3-embedding}"
+      # Only used with KG_LLM_PROVIDER/EMBEDDING_PROVIDER=ollama.
       OLLAMA_BASE_URL: "${OLLAMA_BASE_URL:-http://host.docker.internal:11434}"
       OLLAMA_LLM_MODEL: "${OLLAMA_LLM_MODEL:-llama3}"
       OLLAMA_EMBEDDING_MODEL: "${OLLAMA_EMBEDDING_MODEL:-nomic-embed-text}"
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
     depends_on:
       neo4j:
         condition: service_healthy

docker-compose.yml

@@ -119,11 +119,18 @@ services:
       dockerfile: Dockerfile
     ports:
       - "8000:8000"
-    # OLLAMA_BASE_URL is read from .env:
+    # Default engine: the self-hosted llama.cpp server on the host, reached via
+    # host.docker.internal — chat on :8080 (llama-swap), embeddings on :8081.
+    # Both are read from .env (LLAMACPP_BASE_URL / LLAMACPP_EMBEDDING_BASE_URL).
+    #
+    # OLLAMA_BASE_URL only matters with KG_LLM_PROVIDER/EMBEDDING_PROVIDER=ollama:
     #   default prod  → http://host.docker.internal:11434  (Ollama on host)
     #   profile gpu   → http://ollama-gpu:11434
     #   profile cpu   → http://ollama-cpu:11434
     env_file: .env
+    # host.docker.internal is not resolvable on Linux without this.
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
     depends_on:
       neo4j:
         condition: service_healthy

knowledge-graph-api/.env.example

@@ -7,13 +7,40 @@ NEO4J_DATABASE=neo4j
 # Redis for AI (Vector Store)
 REDIS_URL=redis://redis:6379
 REDIS_INDEX_NAME=kg_vectors
-REDIS_VECTOR_DIM=768
+# DEVE combaciare con l'ampiezza del modello di embedding scelto:
+#   llamacpp + Qwen3-Embedding-0.6B → 1024   (misurato)
+#   ollama   + nomic-embed-text     →  768
+# Cambiarlo invalida un indice esistente: l'avvio si interrompe con l'istruzione
+# di FT.DROPINDEX, e va poi rieseguita l'ingestione di TUTTI i documenti.
+REDIS_VECTOR_DIM=1024
 
-# LLM provider for extraction stage: "ollama" (default) or "anthropic".
-# Embeddings always go to Ollama regardless of this setting.
-KG_LLM_PROVIDER=ollama
+# Provider LLM per la fase di estrazione:
+#   "llamacpp" (default) → server di inferenza self-hosted
+#   "ollama"             → comportamento legacy
+#   "anthropic"          → Claude, opzionale
+KG_LLM_PROVIDER=llamacpp
 
-# Ollama (Inference API locale)
+# Provider degli embedding, scelto separatamente dall'estrazione: e' l'unica
+# scelta non reversibile a costo zero, perche' dimensiona l'indice vettoriale.
+# "llamacpp" (default) oppure "ollama".
+EMBEDDING_PROVIDER=llamacpp
+
+# llama.cpp / llama-swap — compatibile OpenAI, self-hosted.
+# Due server distinti di proposito: l'embedding deve rispondere in millisecondi
+# e non puo' stare dietro a uno swap di modello che dura minuti.
+LLAMACPP_BASE_URL=http://host.docker.internal:8080
+LLAMACPP_LLM_MODEL=qwen3.5-9b
+# Vuoto → usa LLAMACPP_LLM_MODEL
+LLAMACPP_EXTRACTION_MODEL=
+LLAMACPP_EMBEDDING_BASE_URL=http://host.docker.internal:8081
+LLAMACPP_EMBEDDING_MODEL=qwen3-embedding
+# Solo se llama-server e' avviato con --api-key
+LLAMACPP_API_KEY=
+# Alto di proposito: una richiesta che arriva durante uno swap attende il
+# caricamento, e il modello batch da 17,7 GB su disco rotante impiega ~2 minuti.
+LLAMACPP_TIMEOUT_SECONDS=600
+
+# Ollama (Inference API locale) — usato solo con KG_LLM_PROVIDER/EMBEDDING_PROVIDER=ollama
 OLLAMA_BASE_URL=http://ollama:11434
 OLLAMA_LLM_MODEL=llama3
 OLLAMA_EMBEDDING_MODEL=nomic-embed-text

knowledge-graph-api/README.md

@@ -30,13 +30,48 @@ Handles document ingestion pipeline, Redis vector store, Neo4j graph database an
 | Data models | Pydantic v2 + pydantic-settings |
 | Graph DB | Neo4j 5.18 (async driver) |
 | Vector store | Redis Stack (RedisSearch + RedisJSON) |
-| LLM / Embedding | Ollama (llama3 + nomic-embed-text) |
+| LLM / Embedding | llama.cpp self-hosted (qwen3.5-9b + Qwen3-Embedding-0.6B, 1024 dim). Ollama e Claude selezionabili via `KG_LLM_PROVIDER` / `EMBEDDING_PROVIDER` |
 | Logging | structlog (JSON in prod, console in dev) |
 | Testing | pytest + pytest-asyncio + pytest-mock |
 | Linting | ruff |
 
 ---
 
+## ⚠️ Migrazione: gli embedding passano a 1024 dimensioni
+
+Il provider predefinito è ora il **server di inferenza llama.cpp self-hosted**.
+Per l'estrazione il cambio è trasparente. Per gli **embedding no**:
+
+| | prima | ora |
+| --- | --- | --- |
+| provider | Ollama | llama.cpp |
+| modello | `nomic-embed-text` | `Qwen3-Embedding-0.6B-Q8_0` |
+| dimensioni | 768 | **1024** |
+
+Vettori prodotti da modelli diversi non sono confrontabili, e l'indice Redis è
+dimensionato sull'ampiezza del modello. **Un'installazione con dati esistenti
+va reindicizzata:**
+
+```bash
+redis-cli FT.DROPINDEX kg_vectors DD    # DD cancella anche i documenti
+# poi rieseguire l'ingestione di tutti i documenti
+```
+
+Non serve ricordarselo: `RedisVectorStore.create_index` confronta la larghezza
+dell'indice vivo con `REDIS_VECTOR_DIM` e **si rifiuta di partire** su
+disallineamento, stampando il comando qui sopra. Prima non lo faceva — la
+creazione era un no-op silenzioso e il disallineamento emergeva come query che
+non restituivano nulla.
+
+Per restare su Ollama senza reindicizzare, basta dichiararlo:
+
+```bash
+EMBEDDING_PROVIDER=ollama
+REDIS_VECTOR_DIM=768
+```
+
+---
+
 ## Prerequisites
 
 - Python 3.11+
@@ -144,7 +179,8 @@ Document
   [1] File Routing        → MIME type detection (PDF / DOCX / TXT)
   [2] Content Extraction  → raw text + page count
   [3] Text Chunking       → 1024-char chunks, 128-char overlap
-  [4] Embedding           → 768-D vectors via Ollama (nomic-embed-text)
+  [4] Embedding           → 1024-D vectors via llama.cpp (Qwen3-Embedding-0.6B),
+                            l'intero batch in una sola richiesta
   [5] Deduplication       → SHA-256 hash, skip existing chunks
   [6] Entity Extraction   → LLM extracts entities and relations
   [7] Vector Storage      → upsert into Redis (RedisSearch)

knowledge-graph-api/config/settings.py

@@ -15,13 +15,40 @@ class Settings(BaseSettings):
     # Redis (Vector Store)
     REDIS_URL: str = "redis://localhost:6379"
     REDIS_INDEX_NAME: str = "kg_vectors"
-    REDIS_VECTOR_DIM: int = 768
+    # Must match the output width of EMBEDDING_PROVIDER's model:
+    #   llamacpp + Qwen3-Embedding-0.6B → 1024  (measured, not declared)
+    #   ollama   + nomic-embed-text     →  768
+    # Changing this invalidates an existing index — see RedisVectorStore.create_index,
+    # which refuses to run against an index built at a different width.
+    REDIS_VECTOR_DIM: int = 1024
 
     # LLM provider selection for the extraction stage.
-    # "ollama" (default) keeps the legacy local-inference behaviour.
-    # "anthropic" routes extraction to Claude (Haiku by default).
-    # Embeddings always go to Ollama regardless of this setting.
-    KG_LLM_PROVIDER: str = "ollama"
+    # "llamacpp" (default) → the self-hosted llama.cpp inference server.
+    # "ollama"             → the legacy local-inference behaviour.
+    # "anthropic"          → Claude (Haiku by default), opt-in.
+    KG_LLM_PROVIDER: str = "llamacpp"
+
+    # Embeddings are selected separately from extraction: they are the one
+    # thing that cannot be swapped freely, because the vector index is sized
+    # to the model's output width.
+    # "llamacpp" (default) or "ollama".
+    EMBEDDING_PROVIDER: str = "llamacpp"
+
+    # llama.cpp / llama-swap — OpenAI-compatible, self-hosted.
+    # Chat and embeddings are two separate servers on purpose: embeddings must
+    # answer in milliseconds and cannot sit behind a model swap that takes
+    # minutes.
+    LLAMACPP_BASE_URL: str = "http://localhost:8080"
+    LLAMACPP_LLM_MODEL: str = "qwen3.5-9b"
+    # Empty → falls back to LLAMACPP_LLM_MODEL.
+    LLAMACPP_EXTRACTION_MODEL: str = ""
+    LLAMACPP_EMBEDDING_BASE_URL: str = "http://localhost:8081"
+    LLAMACPP_EMBEDDING_MODEL: str = "qwen3-embedding"
+    # Set only if llama-server was started with --api-key.
+    LLAMACPP_API_KEY: str = ""
+    # A request arriving mid-swap waits for the load to finish; the 17.7 GB
+    # batch model needs ~2 minutes from spinning disk just to reach VRAM.
+    LLAMACPP_TIMEOUT_SECONDS: float = 600.0
 
     # Ollama (Inference API locale)
     OLLAMA_BASE_URL: str = "http://localhost:11434"

knowledge-graph-api/pipeline/embedder.py

@@ -1,4 +1,17 @@
-"""Embedding generation via Ollama HTTP API."""
+"""Embedding generation.
+
+Two backends, selected by ``EMBEDDING_PROVIDER``:
+
+* ``llamacpp`` (default) — the self-hosted llama.cpp embedding server, an
+  OpenAI-compatible ``/v1/embeddings`` endpoint that accepts a whole batch in
+  one request.
+* ``ollama`` — the legacy path, one HTTP round-trip per text.
+
+Embeddings are configured separately from the extraction LLM
+(``KG_LLM_PROVIDER``) because they are the one choice that cannot be changed
+freely: the Redis index is sized to the model's output width, so switching
+provider or model means rebuilding it.
+"""
 
 import httpx
 from tenacity import retry, stop_after_attempt, wait_exponential
@@ -8,27 +21,97 @@
 
 
 class Embedder:
-    """Generates embeddings using Ollama's nomic-embed-text model."""
+    """Generates embeddings from the configured provider."""
 
     def __init__(self) -> None:
-        self.base_url = settings.OLLAMA_BASE_URL
-        self.model = settings.OLLAMA_EMBEDDING_MODEL
+        self.provider = settings.EMBEDDING_PROVIDER.lower().strip()
+        if self.provider == "llamacpp":
+            self.base_url = settings.LLAMACPP_EMBEDDING_BASE_URL
+            self.model = settings.LLAMACPP_EMBEDDING_MODEL
+        elif self.provider == "ollama":
+            self.base_url = settings.OLLAMA_BASE_URL
+            self.model = settings.OLLAMA_EMBEDDING_MODEL
+        else:
+            raise ValueError(
+                f"Unknown EMBEDDING_PROVIDER={settings.EMBEDDING_PROVIDER!r} "
+                "(supported: 'llamacpp', 'ollama')"
+            )
 
     async def embed(self, texts: list[str]) -> list[list[float]]:
-        """Embed a list of texts sequentially.
+        """Embed a list of texts.
 
         Args:
             texts: Texts to embed.
 
         Returns:
-            List of 768-dimensional vectors.
+            One vector per input text, in the same order, each of
+            ``settings.REDIS_VECTOR_DIM`` dimensions.
         """
+        if not texts:
+            return []
+        if self.provider == "llamacpp":
+            # One request for the whole batch: the server processes them in a
+            # single forward pass, which is far cheaper than a round-trip per
+            # text (measured: 64 texts / 1142 tokens in 0.58 s).
+            data = await self._post_embeddings(texts)
+            # Parsing is deliberately outside the retry: a malformed or
+            # truncated response is a contract violation, not a transient
+            # fault, and retrying it just delays the error by ~15 seconds.
+            return self._parse_embeddings(data, texts)
         vectors: list[list[float]] = []
         for text in texts:
             vec = await self._embed_single(text)
             vectors.append(vec)
         return vectors
 
+    @retry(
+        stop=stop_after_attempt(settings.MAX_RETRIES),
+        wait=wait_exponential(multiplier=1, min=1, max=30),
+        reraise=True,
+    )
+    async def _post_embeddings(self, texts: list[str]) -> dict:
+        """POST a batch to the OpenAI-compatible ``/v1/embeddings`` endpoint."""
+        headers = (
+            {"Authorization": f"Bearer {settings.LLAMACPP_API_KEY}"}
+            if settings.LLAMACPP_API_KEY
+            else None
+        )
+        async with httpx.AsyncClient(timeout=settings.LLAMACPP_TIMEOUT_SECONDS) as client:
+            response = await client.post(
+                f"{self.base_url.rstrip('/')}/v1/embeddings",
+                headers=headers,
+                json={"model": self.model, "input": texts},
+            )
+            response.raise_for_status()
+            return response.json()
+
+    def _parse_embeddings(self, data: dict, texts: list[str]) -> list[list[float]]:
+        """Validate the response and return one vector per input, in order."""
+        try:
+            rows = data["data"]
+        except (KeyError, TypeError) as exc:
+            raise RuntimeError(f"unexpected embeddings response shape: {exc}") from exc
+
+        if len(rows) != len(texts):
+            raise RuntimeError(
+                f"embeddings endpoint returned {len(rows)} vectors for {len(texts)} inputs"
+            )
+        # The OpenAI shape carries an explicit index; sort by it rather than
+        # trusting positional order. A silently permuted batch would attach
+        # every vector to the wrong chunk — a corruption nothing downstream
+        # would catch.
+        rows = sorted(rows, key=lambda r: r.get("index", 0))
+        vectors = [list(row["embedding"]) for row in rows]
+
+        logger.debug(
+            "embeddings_generated",
+            provider=self.provider,
+            model=self.model,
+            n=len(vectors),
+            dim=len(vectors[0]) if vectors else 0,
+        )
+        return vectors
+
     @retry(
         stop=stop_after_attempt(settings.MAX_RETRIES),
         wait=wait_exponential(multiplier=1, min=1, max=30),

knowledge-graph-api/pipeline/llm/__init__.py

@@ -2,12 +2,15 @@
 
 The extraction stage of the pipeline calls an LLM to convert text chunks
 into structured JSON. The provider is pluggable so the same pipeline can
-run against a local Ollama model, a hosted Anthropic Claude model, or
-future providers without touching the call sites.
+run against the self-hosted llama.cpp server (the default), a local Ollama
+model, or a hosted Anthropic Claude model, without touching the call sites.
 
-Embeddings are NOT covered by this abstraction — they remain wired to
-``OLLAMA_EMBEDDING_MODEL`` via ``pipeline.embedder`` because the vector
-index is sized to that model's output (768 dims for ``nomic-embed-text``).
+Embeddings are NOT covered by this abstraction: they are selected on their
+own by ``EMBEDDING_PROVIDER`` in ``pipeline.embedder``. The split is
+deliberate — extraction can be swapped freely, embeddings cannot, because
+the Redis index is sized to the model's output width (1024 for
+``Qwen3-Embedding-0.6B``, 768 for ``nomic-embed-text``) and changing it
+means rebuilding the index and re-ingesting every document.
 """
 
 from __future__ import annotations

knowledge-graph-api/pipeline/llm/factory.py

@@ -5,18 +5,21 @@
 from config.settings import settings
 from pipeline.llm.anthropic_provider import AnthropicProvider
 from pipeline.llm.base import LLMProvider
+from pipeline.llm.llamacpp_provider import LlamaCppProvider
 from pipeline.llm.ollama_provider import OllamaProvider
 
 
 def get_llm_provider() -> LLMProvider:
-    """Return the provider selected by ``KG_LLM_PROVIDER`` (default: ollama)."""
+    """Return the provider selected by ``KG_LLM_PROVIDER`` (default: llamacpp)."""
 
     choice = settings.KG_LLM_PROVIDER.lower().strip()
+    if choice == "llamacpp":
+        return LlamaCppProvider()
     if choice == "anthropic":
         return AnthropicProvider()
     if choice == "ollama":
         return OllamaProvider()
     raise ValueError(
         f"Unknown KG_LLM_PROVIDER={settings.KG_LLM_PROVIDER!r} "
-        "(supported: 'ollama', 'anthropic')"
+        "(supported: 'llamacpp', 'ollama', 'anthropic')"
     )