Validator: Answer relevance custom LLM judge

backend/README.md

@@ -193,7 +193,7 @@ Set the resulting digest as `AUTH_TOKEN` in your `.env` / `.env.test`.
 
 ## Multi-tenant API Key Configuration
 
-Ban List and Topic Relevance Config APIs use `X-API-KEY` auth instead of bearer token auth.
+Ban List and LLM Prompt Config APIs use `X-API-KEY` auth instead of bearer token auth.
 
 Required environment variables:
 - `KAAPI_AUTH_URL`: Base URL of the Kaapi auth service used to verify API keys.
@@ -203,7 +203,7 @@ At runtime, the backend calls:
 - `GET {KAAPI_AUTH_URL}/apikeys/verify`
 - Header: `X-API-KEY: <token>`
 
-If verification succeeds, tenant's scope (`organization_id`, `project_id`) is resolved from the auth response and applied to tenant-scoped CRUD operations (for example Ban Lists and Topic Relevance Configs).
+If verification succeeds, tenant's scope (`organization_id`, `project_id`) is resolved from the auth response and applied to tenant-scoped CRUD operations (for example Ban Lists and LLM Prompt Configs).
 
 ## Guardrails AI Setup
 

backend/app/alembic/versions/008_added_llm_validator_prompt.py

@@ -0,0 +1,104 @@
+"""Added llm_validator_prompt: rename topic_relevance to llm_prompt, add validator_name, rename configuration to llm_prompt
+
+Revision ID: 008
+Revises: 007
+Create Date: 2026-05-08 00:00:00.000000
+
+"""
+
+from typing import Sequence, Union
+
+import sqlalchemy as sa
+from alembic import op
+
+revision: str = "008"
+down_revision = "007"
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    # Rename table
+    op.rename_table("topic_relevance", "llm_prompt")
+
+    # Rename indexes created by migration 006
+    op.execute(
+        "ALTER INDEX idx_topic_relevance_organization RENAME TO idx_llm_prompt_organization"
+    )
+    op.execute(
+        "ALTER INDEX idx_topic_relevance_project RENAME TO idx_llm_prompt_project"
+    )
+    op.execute(
+        "ALTER INDEX idx_topic_relevance_prompt_schema_version "
+        "RENAME TO idx_llm_prompt_prompt_schema_version"
+    )
+    op.execute(
+        "ALTER INDEX idx_topic_relevance_is_active RENAME TO idx_llm_prompt_is_active"
+    )
+
+    # Add validator_name column (server_default backfills existing rows as topic_relevance)
+    op.add_column(
+        "llm_prompt",
+        sa.Column(
+            "validator_name",
+            sa.String(),
+            nullable=False,
+            server_default="topic_relevance",
+        ),
+    )
+    # Drop server_default so future rows must supply validator_name explicitly
+    op.alter_column("llm_prompt", "validator_name", server_default=None)
+
+    # Rename configuration → llm_prompt column
+    op.alter_column("llm_prompt", "configuration", new_column_name="llm_prompt")
+
+    # Replace unique constraint to include validator_name and use new column name
+    op.drop_constraint(
+        "uq_topic_relevance_config_org_project_prompt",
+        "llm_prompt",
+        type_="unique",
+    )
+    op.create_unique_constraint(
+        "uq_llm_prompt_config",
+        "llm_prompt",
+        [
+            "organization_id",
+            "project_id",
+            "validator_name",
+            "prompt_schema_version",
+            "llm_prompt",
+        ],
+    )
+
+    op.create_index("idx_llm_prompt_validator_name", "llm_prompt", ["validator_name"])
+
+
+def downgrade() -> None:
+    op.drop_index("idx_llm_prompt_validator_name", table_name="llm_prompt")
+
+    op.drop_constraint("uq_llm_prompt_config", "llm_prompt", type_="unique")
+    op.create_unique_constraint(
+        "uq_topic_relevance_config_org_project_prompt",
+        "llm_prompt",
+        ["organization_id", "project_id", "prompt_schema_version", "llm_prompt"],
+    )
+
+    op.alter_column("llm_prompt", "llm_prompt", new_column_name="configuration")
+
+    op.drop_column("llm_prompt", "validator_name")
+
+    op.execute(
+        "ALTER INDEX idx_llm_prompt_is_active RENAME TO idx_topic_relevance_is_active"
+    )
+    op.execute(
+        "ALTER INDEX idx_llm_prompt_prompt_schema_version "
+        "RENAME TO idx_topic_relevance_prompt_schema_version"
+    )
+    op.execute(
+        "ALTER INDEX idx_llm_prompt_project RENAME TO idx_topic_relevance_project"
+    )
+    op.execute(
+        "ALTER INDEX idx_llm_prompt_organization RENAME TO idx_topic_relevance_organization"
+    )
+
+    op.rename_table("llm_prompt", "topic_relevance")

backend/app/alembic/versions/009_add_output_text_to_request_log.py

@@ -0,0 +1,28 @@
+"""Add output_text to request_log
+
+Revision ID: 009
+Revises: 008
+Create Date: 2026-05-21 00:00:00.000000
+
+"""
+
+from typing import Sequence, Union
+
+import sqlalchemy as sa
+from alembic import op
+
+revision: str = "009"
+down_revision = "008"
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    op.add_column(
+        "request_log",
+        sa.Column("output_text", sa.String(), nullable=True),
+    )
+
+
+def downgrade() -> None:
+    op.drop_column("request_log", "output_text")

backend/app/api/API_USAGE.md

@@ -6,7 +6,7 @@ This guide explains how to use the current API surface for:
 - Runtime validator discovery
 - Guardrail execution
 - Ban list CRUD for multi-tenant projects
-- Topic relevance config CRUD for multi-tenant projects
+- LLM prompt config CRUD for multi-tenant projects (`topic_relevance` and `answer_relevance_custom_llm`)
 
 ## Base URL and Version
 
@@ -24,7 +24,7 @@ This API currently uses two auth modes:
    - Used by validator config and guardrails endpoints.
    - The server validates your plaintext bearer token against a SHA-256 digest stored in `AUTH_TOKEN`.
 2. multi-tenant API key auth (`X-API-KEY: <token>`)
-   - Used by ban list and topic relevance config endpoints.
+   - Used by ban list and LLM prompt config endpoints.
    - The API key is verified against `KAAPI_AUTH_URL` and resolves tenant's scope (`organization_id`, `project_id`).
 
 Notes:
@@ -184,8 +184,8 @@ Request fields:
 Important:
 - Runtime validators use `on_fail`.
 - If you pass objects from config APIs, server normalization supports `on_fail_action` and strips non-runtime fields.
-- For `topic_relevance`, pass `topic_relevance_config_id` only.
-- The API resolves `configuration` + `prompt_schema_version` in `guardrails.py` before validator execution, so the validator always executes with both values.
+- For `topic_relevance`, pass `topic_relevance_config_id` only. The API resolves `configuration` + `prompt_schema_version` in `guardrails.py` before validator execution.
+- For `answer_relevance_custom_llm`, `input` must be a JSON string `{"query": "...", "answer": "..."}`. Pass `custom_prompt_id` to use a stored tenant prompt, or omit to use the built-in default prompt.
 
 Example:
 
@@ -342,86 +342,107 @@ curl -X DELETE "http://localhost:8001/api/v1/guardrails/ban_lists/<ban_list_id>"
   -H "X-API-KEY: <api-key>"
 ```
 
-## 6) Topic Relevance Config APIs (multi-tenant)
+## 6) LLM Prompt Config APIs (multi-tenant)
 
-These endpoints manage tenant-scoped topic relevance presets and use `X-API-KEY` auth.
+These endpoints manage tenant-scoped LLM prompt configs for the `topic_relevance` and `answer_relevance_custom_llm` validators. They use `X-API-KEY` auth.
 
 Base path:
-- `/api/v1/guardrails/topic_relevance_configs`
+- `/api/v1/guardrails/llm_prompt_configs`
 
-## 6.1 Create topic relevance config
+The `validator_name` field determines which validator the config applies to:
+- `"topic_relevance"` — a scope description used as the LLM topic guard prompt. No placeholder requirements.
+- `"answer_relevance_custom_llm"` — a custom evaluation prompt. Must contain `{query}` and `{answer}` placeholders.
+
+## 6.1 Create LLM prompt config
 
 Endpoint:
-- `POST /api/v1/guardrails/topic_relevance_configs/`
+- `POST /api/v1/guardrails/llm_prompt_configs/`
 
-Example:
+Example (topic relevance):
 
 ```bash
-curl -X POST "http://localhost:8001/api/v1/guardrails/topic_relevance_configs/" \
+curl -X POST "http://localhost:8001/api/v1/guardrails/llm_prompt_configs/" \
   -H "X-API-KEY: <api-key>" \
   -H "Content-Type: application/json" \
   -d '{
+    "validator_name": "topic_relevance",
     "name": "Maternal Health Scope",
     "description": "Topic guard for maternal health support bot",
     "prompt_schema_version": 1,
-    "configuration": "Pregnancy care: Questions about prenatal care, ANC visits, nutrition, supplements, danger signs. Postpartum care: Questions about recovery after delivery, breastfeeding, and mother health checks."
+    "llm_prompt": "Pregnancy care: Questions about prenatal care, ANC visits, nutrition, supplements, danger signs. Postpartum care: Questions about recovery after delivery, breastfeeding, and mother health checks."
+  }'
+```
+
+Example (answer relevance):
+
+```bash
+curl -X POST "http://localhost:8001/api/v1/guardrails/llm_prompt_configs/" \
+  -H "X-API-KEY: <api-key>" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "validator_name": "answer_relevance_custom_llm",
+    "name": "Maternal Health Relevance",
+    "description": "Checks if LLM answer addresses a maternal health query",
+    "llm_prompt": "You are evaluating a maternal health assistant.\nQuery: {query}\nAnswer: {answer}\n\nDoes the answer directly address the maternal health query with accurate information?\nAnswer only YES or NO."
   }'
 ```
 
-## 6.2 List topic relevance configs
+## 6.2 List LLM prompt configs
 
 Endpoint:
-- `GET /api/v1/guardrails/topic_relevance_configs/?offset=0&limit=20`
+- `GET /api/v1/guardrails/llm_prompt_configs/?offset=0&limit=20`
+
+Optional filter:
+- `validator_name=topic_relevance|answer_relevance_custom_llm`
 
 Example:
 
 ```bash
-curl -X GET "http://localhost:8001/api/v1/guardrails/topic_relevance_configs/?offset=0&limit=20" \
+curl -X GET "http://localhost:8001/api/v1/guardrails/llm_prompt_configs/?validator_name=topic_relevance&offset=0&limit=20" \
   -H "X-API-KEY: <api-key>"
 ```
 
-## 6.3 Get topic relevance config by id
+## 6.3 Get LLM prompt config by id
 
 Endpoint:
-- `GET /api/v1/guardrails/topic_relevance_configs/{id}`
+- `GET /api/v1/guardrails/llm_prompt_configs/{id}`
 
 Example:
 
 ```bash
-curl -X GET "http://localhost:8001/api/v1/guardrails/topic_relevance_configs/<topic_relevance_config_id>" \
+curl -X GET "http://localhost:8001/api/v1/guardrails/llm_prompt_configs/<config_id>" \
   -H "X-API-KEY: <api-key>"
 ```
 
-## 6.4 Update topic relevance config
+## 6.4 Update LLM prompt config
 
 Endpoint:
-- `PATCH /api/v1/guardrails/topic_relevance_configs/{id}`
+- `PATCH /api/v1/guardrails/llm_prompt_configs/{id}`
 
 Example:
 
 ```bash
-curl -X PATCH "http://localhost:8001/api/v1/guardrails/topic_relevance_configs/<topic_relevance_config_id>" \
+curl -X PATCH "http://localhost:8001/api/v1/guardrails/llm_prompt_configs/<config_id>" \
   -H "X-API-KEY: <api-key>" \
   -H "Content-Type: application/json" \
   -d '{
-    "prompt_schema_version": 1,
-    "configuration": "Pregnancy care: Updated scope definition"
+    "llm_prompt": "Pregnancy care: Updated scope definition"
   }'
 ```
 
-## 6.5 Delete topic relevance config
+## 6.5 Delete LLM prompt config
 
 Endpoint:
-- `DELETE /api/v1/guardrails/topic_relevance_configs/{id}`
+- `DELETE /api/v1/guardrails/llm_prompt_configs/{id}`
 
 Example:
 
 ```bash
-curl -X DELETE "http://localhost:8001/api/v1/guardrails/topic_relevance_configs/<topic_relevance_config_id>" \
+curl -X DELETE "http://localhost:8001/api/v1/guardrails/llm_prompt_configs/<config_id>" \
   -H "X-API-KEY: <api-key>"
 ```
 
-## 7) End-to-End Usage Pattern
+## 8) End-to-End Usage Pattern
 
 Recommended request flow:
 1. Create/update validator configs via `/guardrails/validators/configs`.
@@ -430,16 +451,17 @@ Recommended request flow:
 4. Use `safe_text` as downstream text.
 5. If `rephrase_needed=true`, ask user to rephrase.
 6. For `ban_list` validators without inline `banned_words`, create/manage a ban list first and pass `ban_list_id`.
-7. For `topic_relevance`, create/manage a topic relevance config and pass `topic_relevance_config_id` at runtime. The server resolves the configuration string internally.
+7. For `topic_relevance`, create/manage an LLM prompt config (`validator_name: "topic_relevance"`) and pass `topic_relevance_config_id` at runtime. The server resolves `llm_prompt` and `prompt_schema_version` internally.
+8. For `answer_relevance_custom_llm`, format `input` as `{"query": "...", "answer": "..."}`. Optionally create an LLM prompt config (`validator_name: "answer_relevance_custom_llm"`) and pass `custom_prompt_id`. If no `custom_prompt_id` is given, the built-in default prompt is used.
 
-## 8) Common Errors
+## 9) Common Errors
 
 - `401 Missing Authorization header`
   - Add `Authorization: Bearer <token>`.
 - `401 Invalid authorization token`
   - Verify plaintext token matches server-side hash.
 - `401 Missing X-API-KEY header`
-  - Add `X-API-KEY: <api-key>` for ban list and topic relevance config endpoints.
+  - Add `X-API-KEY: <api-key>` for ban list and LLM prompt config endpoints.
 - `401 Invalid API key`
   - Verify the API key is valid in the upstream Kaapi auth service.
 - `Invalid request_id`
@@ -448,10 +470,10 @@ Recommended request flow:
   - Type+stage is unique per organization/project scope.
 - `Validator not found`
   - Confirm `id`, `organization_id`, and `project_id` match.
-- `Topic relevance preset not found`
-  - Confirm topic relevance config `id` exists within your tenant scope.
+- `LLM prompt config not found`
+  - Confirm the LLM prompt config `id` exists within your tenant scope.
 
-## 9) Current Validator Types
+## 10) Current Validator Types
 
 From `validators.json`:
 - `uli_slur_match`
@@ -463,6 +485,7 @@ From `validators.json`:
 - `llamaguard_7b`
 - `profanity_free`
 - `nsfw_text`
+- `answer_relevance_custom_llm`
 
 Source of truth:
 - `backend/app/core/validators/validators.json`

backend/app/api/docs/guardrails/run_guardrails.md

@@ -6,8 +6,9 @@ Behavior notes:
 - The endpoint always saves a `request_log` entry for the run.
 - Validator logs are also saved; with `suppress_pass_logs=true`, only fail-case validator logs are persisted. Otherwise, all validator logs are added.
 - For `ban_list`, `ban_list_id` can be resolved to `banned_words` from tenant ban list configs.
-- For `topic_relevance`, `topic_relevance_config_id` is required and is resolved to `configuration` + `prompt_schema_version` from tenant topic relevance configs. Requires `OPENAI_API_KEY` to be configured; returns a validation failure with an explicit error if missing.
+- For `topic_relevance`, `topic_relevance_config_id` is required and is resolved to `llm_prompt` + `prompt_schema_version` from tenant LLM prompt configs. Requires `OPENAI_API_KEY` to be configured; returns a validation failure with an explicit error if missing.
 - For `llm_critic`, `OPENAI_API_KEY` must be configured; returns `success=false` with an explicit error if missing.
+- For `answer_relevance_custom_llm`, `input` must be a JSON string `{"query": "...", "answer": "..."}`. Pass `custom_prompt_id` to use a tenant-stored prompt template, or `prompt_template` inline. Requires `OPENAI_API_KEY`.
 - For `llamaguard_7b`, `policies` accepts human-readable policy names (see table below). If omitted, all policies are enforced by default.
 
   | `policies` value            | Policy enforced                  |