diff --git a/TRACKER.md b/TRACKER.md index 60ae7f9..62f51a9 100644 --- a/TRACKER.md +++ b/TRACKER.md @@ -14,12 +14,13 @@ | | | |---|---| | **Last updated** | 2026-06-10 | -| **Current phase** | Phase 7 — Pilot, Harden, GA (**7 / 10 steps**) — Phases 0–6 ✅ complete | -| **Overall** | **81 / 84 steps** — Phases 0–6 complete | -| **Next action** | **Step 7.8 — Support, SLAs, on-call**: support tiers (Community/Pro/Business/Enterprise); SLA targets; incident response + postmortem template; per-alert runbooks; public status page; PagerDuty tie-in. | +| **Current phase** | Phase 7 — Pilot, Harden, GA (**8 / 10 steps**) — Phases 0–6 ✅ complete | +| **Overall** | **82 / 84 steps** — Phases 0–6 complete | +| **Next action** | **Step 7.9 — Billing & metering**: per-tenant metering (queries / docs / storage / reranker / LLM tokens); usage-export API; Stripe metered billing behind a seam; invoice generation; tenant usage dashboard. | **Recently shipped** +- **7.8** ✅ Support, SLAs & on-call — the support model + ops runbooks, **grounded in the platform's own signals**. Four support tiers + per-tier SLA targets (matching `marketplace/pricing.yaml`) in [`support-sla.md`](docs/guides/support-sla.md); a new `docs/runbooks/` area with **one runbook per alert type** (`alerts.md` — acl/tenant escape · `breaker.opened` · `drift.detected` · cost anomaly · `quota.exceeded` · injection spike · gateway degraded · latency/availability SLO · `eval.regression` · ingest), each naming the exact event / `/v1/status/*` / SLO that raises it; an incident-response process + a blameless postmortem template; the status page reads **measured** health (`/v1/status/health` + Grafana SLOs), not manual toggles. **Paging reuses the Step 3.9 webhook system** (PagerDuty as a subscriber — no new code). Deferred: hosted status-page deploy, the PagerDuty account, staffing. [ADR-0050](docs/adr/ADR-0050-support-sla-oncall.md) [#181](https://github.com/officialCodeWork/AgentContextOS/pull/181) - **7.7** ✅ Packaging & distribution — **one-command install per channel** + **one tag fans out to every channel**. New `rag-platform` PyPI **meta-package** (`packaging/rag-platform/`) pins the workspace component dists for a reproducible `pip install rag-platform`; the existing `agentcontextos` SDK scope is kept (no rename). A new `release.yml` publishes PyPI (OIDC Trusted Publishing) + npm (`@agentcontextos/sdk`, provenance) + the **admin-ui image** (new `apps/admin-ui/Dockerfile`, cosign + SBOM) + the Helm chart → **OCI** (cosign), alongside the existing `docker.yml` (gateway image, cosign + SBOM) and `release-airgap.yml` — all keyed off the same `vX.Y.Z` tag. Deferred: Go/Java/.NET SDK publish, worker/eval-runner images (single-process today). [ADR-0049](docs/adr/ADR-0049-packaging-distribution.md) [#180](https://github.com/officialCodeWork/AgentContextOS/pull/180) - **7.6** ✅ Marketplace listings — listing artifacts for AWS / Azure / GCP under a new `marketplace/` (approval + procurement stay an external per-cloud process, like the 7.3 pentest). **One canonical `pricing.yaml`** (tiers + usage-metered dimensions whose ids match the Step 7.9 metering signals) + **one shared `listing-content.md`** mapped to all three clouds, so listings can't drift and prices reconcile with metering. Each per-cloud spec maps the model to that cloud's billing primitives and **reuses existing delivery** (Helm / AMI / air-gap bundle / GHCR — no new packaging); per-cloud submission checklists + a procurement acceptance test in the guide. [ADR-0048](docs/adr/ADR-0048-marketplace-listings.md) [#179](https://github.com/officialCodeWork/AgentContextOS/pull/179) - **7.5** ✅ Documentation site — a **Docusaurus 3** app (`website/`) that serves the repo `docs/` tree **in place** (single source of truth; `markdown.format: 'detect'` renders hand-written `.md` as CommonMark). The **REST API reference** (`docs/reference/rest-api.md`) is **generated + drift-gated** from `dist/openapi.json` by `scripts/gen_api_reference.py` (`task docs:api`) — never hand-edited. Doc honesty is a `tests/docs/` suite (every `ragctl` / `/v1/` reference in a quickstart must be real) + `lychee` offline link-check + `codespell`, wired into a new `docs` CI workflow. Deferred: Algolia search, versioned docs, the deploy workflow. [ADR-0047](docs/adr/ADR-0047-documentation-site.md) [#178](https://github.com/officialCodeWork/AgentContextOS/pull/178) @@ -812,7 +813,7 @@ New ground — the only prior crypto was HMAC signing. The V1 plan calls for en | 7.5 | Documentation site | ✅ | [#178](https://github.com/officialCodeWork/AgentContextOS/pull/178) — Docusaurus app (`website/`) serving `docs/` in place; **generated + drift-gated REST API reference** from `dist/openapi.json`; `tests/docs/` quickstart-honesty suite + `lychee`/`codespell`; `docs` CI workflow; `task docs:api`/`docs:build` | | 7.6 | Marketplace listings | ✅ | [#179](https://github.com/officialCodeWork/AgentContextOS/pull/179) — `marketplace/` listing artifacts (AWS/Azure/GCP); one canonical `pricing.yaml` (dims = 7.9 metering signals) + shared listing copy mapped to all three; reuse Helm/AMI/airgap/GHCR delivery; submission checklists + procurement test; approval = external process | | 7.7 | Packaging & distribution | ✅ | [#180](https://github.com/officialCodeWork/AgentContextOS/pull/180) — `rag-platform` PyPI meta-package; `release.yml` fans one tag out to PyPI/npm/GHCR(admin-ui)/Helm-OCI (cosign-keyless + SPDX SBOM + OIDC/provenance) alongside `docker.yml`+`release-airgap.yml`; `apps/admin-ui/Dockerfile`; channels matrix + guide | -| 7.8 | Support & SLA | ⏳ | Support tiers defined; SLA dashboards; PagerDuty integration; runbooks | +| 7.8 | Support & SLA | ✅ | [#181](https://github.com/officialCodeWork/AgentContextOS/pull/181) — support tiers + SLA targets (= `pricing.yaml`); `docs/runbooks/` with **a runbook per alert type** (tied to real events / `/v1/status/*` / SLOs); incident-response + postmortem template; status page from measured health; PagerDuty via the Step 3.9 webhook system (no new code) | | 7.9 | Billing integration | ⏳ | Stripe metered billing; usage export API; invoice generation | | 7.10 | GA cutover | ⏳ | `main` tag `v1.0.0`; release notes; all Phase 7 exit gates passed | @@ -875,6 +876,15 @@ New ground — the only prior crypto was HMAC signing. The V1 plan calls for en - **Signed + SBOM, keyless.** cosign keyless (GitHub OIDC) for images + chart, SPDX SBOM per image, OIDC Trusted Publishing for PyPI, `--provenance` for npm, cosign-over-`SHA256SUMS` for the air-gap bundle — no long-lived registry tokens. - **Scope:** the meta-package + `release.yml` + the admin-ui Dockerfile + a channels-matrix `packaging/README.md` + the guide. **Deferred:** publishing the generated Go/Java/.NET SDKs; separate `worker`/`eval-runner` images (single gateway process today; `core` is a library, not an image); the one-time trusted-publisher / GHCR setup + first publish (external). [ADR-0049](docs/adr/ADR-0049-packaging-distribution.md), [guides/packaging-distribution.md](docs/guides/packaging-distribution.md). +### 7.8 — Support, SLAs & on-call ✅ [#181](https://github.com/officialCodeWork/AgentContextOS/pull/181) + +- The support + ops layer, **grounded in the signals the platform already emits** rather than a new alerting subsystem. +- **Support tiers + SLA** ([`support-sla.md`](docs/guides/support-sla.md)): Community / Pro / Business / Enterprise with response targets; per-tier availability + latency SLAs that **match `marketplace/pricing.yaml`** (sold = promised = measured); the follow-the-sun on-call + escalation. +- **A runbook for every alert type** (`docs/runbooks/alerts.md`): acl/tenant escape · `breaker.opened` · `drift.detected` · cost anomaly · `quota.exceeded` · injection spike · gateway degraded · latency/availability SLO · `eval.regression` · ingest failures — each names the exact event / `/v1/status/*` surface / SLO that raises it, a diagnosis path, a mitigation (often the existing self-healing), and an escalation. +- **Incident response + postmortem** (`incident-response.md` + `postmortem-template.md`): severity ladder, IC/Comms/Ops roles, the detect→resolve flow, and a **status page that reflects measured health** (`/v1/status/health` + Grafana SLOs, Step 3.11/5.6e) — not manual toggles. +- **Paging reuses the Step 3.9 webhook system** — PagerDuty is a subscriber for `audit.policy_violation` / `drift.detected` / `eval.regression` (Events API v2), no new code. +- **Scope:** the guide + the `docs/runbooks/` area + this ADR. No code/`dist` change. **Deferred:** the hosted status-page deploy, the PagerDuty account + routing key, staffing the rotation (people process). [ADR-0050](docs/adr/ADR-0050-support-sla-oncall.md), [runbooks/](docs/runbooks/README.md). + --- ## PR & Branch History @@ -1029,6 +1039,7 @@ Complete log of every PR. Routine Dependabot bumps are grouped; everything else | [#178](https://github.com/officialCodeWork/AgentContextOS/pull/178) | 2026-06-10 | feat(docs): Docusaurus documentation site + generated API reference + doc-tests (Step 7.5) | | [#179](https://github.com/officialCodeWork/AgentContextOS/pull/179) | 2026-06-10 | docs(marketplace): AWS/Azure/GCP listing artifacts + canonical pricing model (Step 7.6) | | [#180](https://github.com/officialCodeWork/AgentContextOS/pull/180) | 2026-06-10 | feat(release): rag-platform PyPI meta + release.yml (PyPI/npm/Helm-OCI/admin-ui image) (Step 7.7) | +| [#181](https://github.com/officialCodeWork/AgentContextOS/pull/181) | 2026-06-10 | docs(support): support tiers + SLA + per-alert runbooks + incident response (Step 7.8) | | #78–#80, #116–#118 | Open | Dependabot bumps — awaiting merge | | #81 | Closed | Dependabot bump — superseded | diff --git a/docs/README.md b/docs/README.md index 6d836ab..eb6aa0c 100644 --- a/docs/README.md +++ b/docs/README.md @@ -135,6 +135,7 @@ | [documentation-site.md](guides/documentation-site.md) | The documentation site (Step 7.5): the Docusaurus app under `website/` that serves `docs/` in place; running/building it; the generated REST API reference; the CI gates (API-ref drift · quickstart doc-tests · spell-check · `lychee` link-check · site build); adding docs | | [marketplace-listings.md](guides/marketplace-listings.md) | Cloud marketplace listings (Step 7.6): what's in-repo (`marketplace/` — one canonical `pricing.yaml` + shared listing copy + per-cloud AWS/Azure/GCP specs) vs the external approval/procurement process; reusing the Helm/AMI/airgap/GHCR delivery; the submission checklists + the procurement acceptance test | | [packaging-distribution.md](guides/packaging-distribution.md) | Packaging & distribution (Step 7.7): the one-command install per channel (PyPI `rag-platform`, npm `@agentcontextos/sdk`, GHCR images, Helm OCI, air-gap); how one `vX.Y.Z` tag fans out across `docker.yml` + `release.yml` + `release-airgap.yml`; cosign-keyless + SPDX SBOM + OIDC/provenance signing; verification commands | +| [support-sla.md](guides/support-sla.md) | Support, SLAs & on-call (Step 7.8): the four support tiers + response targets; SLA targets per tier (matching `marketplace/pricing.yaml`); the follow-the-sun on-call + escalation; wiring platform alerts to **PagerDuty** via the Step 3.9 webhook system (no new code) | | [curl-quickstart.md](guides/curl-quickstart.md) | 🥈 Curl-able RAG (Step 3.1): 5-minute walkthrough from `curl` to gateway response, including ingest, query, generate, OpenAPI | | [grpcurl-quickstart.md](guides/grpcurl-quickstart.md) | gRPC quickstart (Step 3.2): 5-minute walkthrough using `grpcurl` against the in-process server — health check, list corpora, server-streaming query, structured errors | | [mcp-quickstart.md](guides/mcp-quickstart.md) | MCP quickstart (Step 3.3): 5-minute walkthrough — `ragctl mcp-query`, running the stdio server, mounting `@ragplatform/mcp` in Claude Desktop, the three tools, error shape | @@ -202,6 +203,7 @@ broken, and what to fix before committing to the next phase. | [ADR-0046-design-partner-pilots.md](adr/ADR-0046-design-partner-pilots.md) | Decision (Step 7.4): ship the pilot **machine** in-repo (runbook + templates + per-vertical kits + a `ragctl pilot` KPI reader) while the partner **relationships** stay an external GTM deliverable (like the 7.3 pentest); each success criterion (quality / latency / integration / security) maps to a platform signal (eval / feedback / drift / cost / metrics / compliance) so KPIs are **pulled, not self-reported**; sliced 7.4a–d, leading with customer-support/KB, the framework vertical-extensible; no new package / core type / governed SPI call / `dist` change — the report reads existing status endpoints; rejected a pure-external motion, self-reported KPIs, a `rag-pilot` package, one generic kit | | [ADR-0047-documentation-site.md](adr/ADR-0047-documentation-site.md) | Decision (Step 7.5): publish docs via **Docusaurus** sourcing the repo `docs/` tree **in place** (single source of truth, no copy); `markdown.format: 'detect'` so hand-written `.md` renders as CommonMark; the **REST API reference is generated + drift-gated** from `dist/openapi.json` (a committed file, not a build-time plugin); doc honesty is a `tests/docs/` suite (every `ragctl` / `/v1/` reference is real) + `lychee` + `codespell`; rejected MkDocs, copying `docs/`, an OpenAPI plugin | | [ADR-0049-packaging-distribution.md](adr/ADR-0049-packaging-distribution.md) | Decision (Step 7.7): a `rag-platform` PyPI **meta-package** pins the workspace component dists (one-command server install); keep the existing `agentcontextos` SDK scope (no rename); **one `vX.Y.Z` tag fans out** to PyPI/npm/GHCR/Helm-OCI/air-gap across `docker.yml` + a new `release.yml` + `release-airgap.yml`; cosign-keyless + SPDX SBOM + OIDC/provenance everywhere; deferred Go/Java/.NET publish + worker/eval-runner images; rejected a `ragplatform` rename + one mega-workflow | +| [ADR-0050-support-sla-oncall.md](adr/ADR-0050-support-sla-oncall.md) | Decision (Step 7.8): **runbooks grounded in the platform's own signals** — one per alert type, each naming the exact event / `/v1/status/*` / SLO that raises it; **paging reuses the Step 3.9 webhook system** (PagerDuty as a subscriber, no new code) + Grafana SLOs; the status page reflects **measured** health (not manual toggles); SLA tiers match `marketplace/pricing.yaml`; rejected a new alerting subsystem, generic runbooks, a separate SLA datastore | | [ADR-0048-marketplace-listings.md](adr/ADR-0048-marketplace-listings.md) | Decision (Step 7.6): one canonical `marketplace/pricing.yaml` (metered dims = the Step 7.9 signals) + one shared listing copy mapped to all three clouds, so listings can't drift and prices reconcile with metering; **reuse existing delivery** (Helm / AMI / airgap / GHCR) — the per-cloud Marketplace wrappers are thin shells; approval + procurement are documented external process, not a faked status; rejected per-cloud pricing, custom packaging, encoding a "live" status | | [ADR-0042-compliance-posture.md](adr/ADR-0042-compliance-posture.md) | Decision (Step 6.10, Phase-6 capstone): add the three compliance pieces on top of the controls the platform already ships (audit/ACL/PII/BYOK/SSO/quotas). New `rag-compliance` package (config-free, like rag-feedback/rag-drift): `RetentionEnforcer` drives tenant-scoped `purge_*`; `compliance_posture`/`residency_ok` are pure. **Retention is a capability on the existing stores, not a new SPI** — non-abstract `purge_before`/`purge_tenant` (default no-op) on Feedback/Provenance, with `dry_run` in the SPI so a preview counts-without-deleting uniformly (ProvenanceStore has no `list`). **Audit is never purged in place** (the hash chain would break) — audit retention is the 6.6b WORM export; `audit_days` is advisory. **Right-to-erasure is always-on, tenant-self-service, two-flag** — `POST /v1/compliance/erase` erases the *calling* tenant's data (scope from the principal, never the body), dry-run by default, delete needs `dry_run=false` AND `confirm=true`. **Residency = declared per tenant + enforced at ingest** (`tenants[].data_region` vs `cfg.compliance.region` → 403), opt-in, a single-deployment assertion not multi-region routing. **The SOC 2 / GDPR mapping is a doc backed by a live posture** (`GET /v1/status/compliance` reports which controls are on, so the mapping is checkable). Deferred: subject-level (vs tenant-level) erasure, an admin retention-sweep endpoint, multi-region routing, automated audit-evidence bundles; rejected purging the audit chain, a `Purgeable` SPI mixin, a static doc with no live backing | | [ADR-0041-airgap-bundle.md](adr/ADR-0041-airgap-bundle.md) | Decision (Step 6.9): ship the platform as one signed, self-contained offline bundle (all runtime images + Helm chart + config + installer). Integrity reuses the WORM-export pattern (6.6b): a standard `SHA256SUMS` whose hash is pinned as `manifest.content_hash` is the **hard gate** — verifiable with nothing but `sha256sum`, no network/cosign — and a cosign signature **over `SHA256SUMS`** adds authenticity; the *same* `SHA256SUMS` drives the Python verifier and the standalone shell installer so they can't diverge. The shell/pwsh `install.{sh,ps1}` (shipped inside the bundle) need only docker+helm (air-gap hosts lack uv/the workspace); `ragctl airgap` holds the typed/tested build+verify logic (pure core separated from a stubbable docker/helm/cosign subprocess seam; `--dry-run` = a verifiable bundle minus image blobs, so the path is testable with no Docker). Digest-pinned manifest-driven image set (`infra/airgap/images.txt` + the chart-derived gateway image); key-based cosign is the air-gap recommendation (keyless needs Rekor + an identity policy), keyless is the connected-release path (`release-airgap.yml` on tags). Deferred: ctr/podman load, registry re-tag/push, multi-arch selection, bundling backend charts, TUF-rooted offline keyless verify; rejected `oras`/OCI (no registry to pull from in an air-gap), a second HMAC scheme (cosign already the signer), a pure-shell build (would escape mypy/tests) | @@ -237,6 +239,18 @@ holds the fill-in templates, per-vertical kits, and published case studies. | [customer-support/README.md](pilots/customer-support/README.md) | Customer-support / internal-KB pilot kit (Step 7.4b) — sample corpus (PII handbook + product FAQ + a planted injection probe), domain-calibrated success criteria (deflection), the `ragctl pilot` seed-and-demo flow, the PII + injection security demonstration | | [customer-support/case-study.md](pilots/customer-support/case-study.md) | Worked case study (Step 7.4d) — the framework run end-to-end on the kit; real `ragctl pilot report` KPIs (satisfaction +0.733 · 0/5 drift · cost ok → PASS) + the PII + injection security demonstration | +## runbooks/ + +Operational runbooks (Step 7.8) — one per alert the platform can raise, plus the +incident-response process and the postmortem template. + +| File | Description | +|------|-------------| +| [README.md](runbooks/README.md) | Runbooks index — the alert → runbook table + the Step 7.8 acceptance checklist | +| [alerts.md](runbooks/alerts.md) | One runbook per alert type (acl/tenant escape · `breaker.opened` · `drift.detected` · cost anomaly · `quota.exceeded` · injection spike · gateway degraded · latency/availability SLO · `eval.regression` · ingest failures), each tied to the platform event / `/v1/status/*` signal that raises it | +| [incident-response.md](runbooks/incident-response.md) | Severity ladder · IC/Comms/Ops roles · detect→declare→mitigate→communicate→resolve→postmortem flow · the status page (measured from `/v1/status/health` + Grafana SLOs) | +| [postmortem-template.md](runbooks/postmortem-template.md) | Blameless postmortem template (summary · impact · timeline · root cause · action items) | + ## research/ | File | Description | diff --git a/docs/adr/ADR-0050-support-sla-oncall.md b/docs/adr/ADR-0050-support-sla-oncall.md new file mode 100644 index 0000000..f1ee0f7 --- /dev/null +++ b/docs/adr/ADR-0050-support-sla-oncall.md @@ -0,0 +1,60 @@ +# ADR-0050 — Support, SLAs & on-call: runbooks grounded in the platform's own signals + +**Status:** Accepted +**Date:** 2026-06-10 +**Step:** 7.8 — Support, SLAs, on-call (Phase 7 — Pilot, Harden, GA) +**Related:** [guides/support-sla.md](../guides/support-sla.md), [runbooks/](../runbooks/README.md), [3.9 outbound webhooks](ADR-0018-outbound-webhooks.md), [3.11 status & metrics](ADR-0020-status-transports.md), [5.6e Grafana dashboards](../guides/grafana-dashboards.md), [7.6 marketplace pricing](ADR-0048-marketplace-listings.md), [planning/phases/phase-7-pilot-ga.md](../../planning/phases/phase-7-pilot-ga.md) + +## Context + +Step 7.8 wants support tiers, SLA targets, an on-call rotation, incident response + +postmortem process, runbooks for **every alert type**, and a public status page. +The platform already emits the signals these need: structured events +(`drift.detected`, `breaker.opened`, `quota.exceeded`, `audit.policy_violation`, +`injection.blocked`, `gateway.*.degraded`, …), outbound webhooks (3.9), live status +endpoints + health (3.11), Grafana SLO dashboards (5.6e), and the eval gate (5.3). +The question is whether to build a new alerting/paging subsystem or compose what +exists. + +## Decision + +**1. Runbooks are grounded in real signals, one per alert type.** `docs/runbooks/alerts.md` +has a section per alert that names the exact event / status endpoint / SLO that +raises it, a diagnosis path using the existing `/v1/status/*` surfaces, a mitigation +(often the existing self-healing — breaker half-open, degrade-open, rebaseline), and +an escalation. No generic prose — every runbook ties to a signal the platform +actually emits. + +**2. Paging reuses the webhook system — no new code.** PagerDuty is a webhook +subscriber (Step 3.9 Events API v2 target) for `audit.policy_violation` / +`drift.detected` / `eval.regression`; threshold/SLO alerts come from Grafana on the +3.11 metrics. We do not build a bespoke alerting pipeline. + +**3. The status page reflects measured signals.** Component health derives from +`GET /v1/status/health` + the Grafana SLO rules, not manual toggles (except Comms +updates during a declared incident) — same "checkable, not asserted" posture as the +compliance posture (6.10). + +**4. SLA tiers are consistent with pricing.** The availability/latency targets match +`marketplace/pricing.yaml` (Step 7.6), so what's sold equals what's promised equals +what's measured. + +## Consequences + +- New `docs/runbooks/` (index + per-alert runbooks + incident-response + postmortem + template), `docs/guides/support-sla.md`, and this ADR. +- Every alert the platform can raise has a runbook; on-call follows the link, then + escalates per the support guide. +- **Deferred:** the hosted status-page deployment (the component map + probe targets + are specified; hosting is operational), the actual PagerDuty account + routing key, + and staffing the rotation (a people process, not code). + +## Alternatives considered + +- **Build a new alerting/paging subsystem.** Rejected — the webhook system (3.9) + + Grafana (5.6e) already deliver events + SLO rules; PagerDuty is just a subscriber. +- **Generic, signal-agnostic runbooks.** Rejected — a runbook is only useful if it + names the exact signal and the exact `/v1/status/*` surface to check; grounding + them in the real events is the whole point. +- **A separate SLA datastore.** Rejected — availability/latency are already measured + (3.11) and the SLA targets live in `pricing.yaml`; one source each. diff --git a/docs/guides/support-sla.md b/docs/guides/support-sla.md new file mode 100644 index 0000000..4bba22f --- /dev/null +++ b/docs/guides/support-sla.md @@ -0,0 +1,61 @@ +# Guide: support tiers, SLAs & on-call (Step 7.8) + +The support model, the SLA commitments per tier, the on-call rotation, and how +platform alerts page a human. + +## Support tiers + +| Tier | Channels | Response target (first response) | Hours | +|------|----------|----------------------------------|-------| +| **Community** | GitHub issues / discussions | best-effort | — | +| **Pro** | Email + portal | P1 4h · P2 1 business day | Business hours | +| **Business** | Email + portal + chat | P1 1h · P2 4h | 24×5 | +| **Enterprise** | + dedicated Slack + named TAM | P1 30m · P2 2h | 24×7 | + +Severity: **P1** service down / data risk · **P2** major feature degraded · **P3** +minor · **P4** question. The pricing for each tier is in +[`marketplace/pricing.yaml`](../../marketplace/pricing.yaml) (single source). + +## SLA targets + +| Tier | Availability (monthly) | Latency | Recovery | +|------|------------------------|---------|----------| +| Pro | 99.5% | P95 ≤ 1s | best-effort | +| Business | 99.9% | P95 ≤ 800 ms | RPO 24h / RTO 4h | +| Enterprise | 99.95% | P95 ≤ 800 ms | custom RPO/RTO | + +Latency is the served-path target; the gateway **overhead** budget (p99 ≤ 30 ms, +Step 4.6) is monitored separately. Availability + latency are measured from +`GET /v1/status/health` + `gateway.request_duration_ms` (Step 3.11) and surfaced on +the [status page](../runbooks/incident-response.md#status-page). + +## On-call + +- **Rotation** staffed across three timezone regions (follow-the-sun) so P1s are + always covered within the tier response target. +- **Primary → secondary → engineering-lead** escalation; the secondary is paged if + the primary doesn't ack within 10 minutes. +- Every alert links to a [runbook](../runbooks/README.md); the on-call's first job + is to follow it, then escalate if the mitigation doesn't hold. + +## Paging — wiring alerts to PagerDuty + +The platform already emits structured events and **outbound webhooks** (Step 3.9). +PagerDuty is wired as a webhook subscriber (no new code), using the **Events API +v2**: + +1. Register a webhook subscription pointing at the PagerDuty Events API v2 URL with + your routing key (`POST /v1/webhooks/subscriptions`). +2. Subscribe it to the alerting event types: `drift.detected`, + `audit.policy_violation`, `eval.regression` (the Step 3.9 catalogue). +3. Map event → PagerDuty severity: + +| Event | PD severity | Runbook | +|-------|-------------|---------| +| `audit.policy_violation` | critical | [acl / tenant escape](../runbooks/alerts.md#aclegress_violation--tenant-escape) | +| `drift.detected` | warning | [retrieval drift](../runbooks/alerts.md#driftdetected) | +| `eval.regression` | warning | [eval regression](../runbooks/alerts.md#evalregression) | + +Threshold/SLO alerts (latency, availability, error rate, cost spikes) are raised by +the metrics backend (Prometheus/Grafana, Step 5.6e) against the same runbooks. See +[runbooks/](../runbooks/README.md) and [ADR-0050](../adr/ADR-0050-support-sla-oncall.md). diff --git a/docs/runbooks/README.md b/docs/runbooks/README.md new file mode 100644 index 0000000..d95ff53 --- /dev/null +++ b/docs/runbooks/README.md @@ -0,0 +1,35 @@ +# runbooks — operational runbooks (Step 7.8) + +One runbook per alert the platform can raise, plus the incident-response process +and the postmortem template. Every alert (a structured event, a webhook, or a +Grafana SLO rule) links here; the on-call follows the runbook, then escalates per +[support-sla.md](../guides/support-sla.md#on-call). + +## Alert → runbook + +| Alert (event / signal) | Severity | Runbook | +|------------------------|----------|---------| +| `audit.policy_violation` (ACL / tenant escape) | P1 | [alerts.md](alerts.md#aclegress_violation--tenant-escape) | +| `breaker.opened` (backend down) | P2 | [alerts.md](alerts.md#breakeropened) | +| `drift.detected` (retrieval quality) | P3 | [alerts.md](alerts.md#driftdetected) | +| Cost spike (`/v1/status/cost` = elevated) | P2 | [alerts.md](alerts.md#cost-anomaly-elevated) | +| `quota.exceeded` (tenant throttled) | P3 | [alerts.md](alerts.md#quotaexceeded) | +| `injection.blocked` spike | P2 | [alerts.md](alerts.md#injectionblocked-spike) | +| Gateway degraded (`gateway.*.degraded`) | P2 | [alerts.md](alerts.md#gateway-degraded) | +| Latency / availability SLO breach | P1/P2 | [alerts.md](alerts.md#latency--availability-slo-breach) | +| `eval.regression` | P3 | [alerts.md](alerts.md#evalregression) | +| Ingest failures | P3 | [alerts.md](alerts.md#ingest-failures) | + +## Process + +- [incident-response.md](incident-response.md) — severity ladder, roles, comms, the + status page, the timeline. +- [postmortem-template.md](postmortem-template.md) — the blameless postmortem. + +## Acceptance (Step 7.8) + +- A runbook exists for **every alert type** (the table above) — each maps the alert + to the platform signal that raises it, a diagnosis path, a mitigation, and an + escalation. +- On-call staffed across timezones (follow-the-sun); see [support-sla.md](../guides/support-sla.md#on-call). +- Status page live with historical uptime (see incident-response.md#status-page). diff --git a/docs/runbooks/alerts.md b/docs/runbooks/alerts.md new file mode 100644 index 0000000..cdc3a01 --- /dev/null +++ b/docs/runbooks/alerts.md @@ -0,0 +1,83 @@ +# Alert runbooks + +One section per alert. Each maps the alert to the platform signal that raises it, +a diagnosis path, a mitigation, and when to escalate. Severity + paging are in +[support-sla.md](../guides/support-sla.md). + +--- + +## acl.egress_violation / tenant escape + +- **Signal:** `audit.policy_violation` / `acl.egress_violation` event; red-team gate failing. +- **Severity:** P1 — potential cross-tenant / cross-label data exposure. +- **Diagnose:** `GET /v1/audit` (tenant-scoped) for the offending request; confirm whether the egress verifier (6.4) caught it (defense-in-depth) or it escaped. +- **Mitigate:** if escaped, disable the affected tenant's traffic; verify `cfg.acl.enabled` + `verify_egress`; check the principal's `acl_labels`. +- **Escalate:** security lead immediately; treat as an incident (postmortem required). + +## breaker.opened + +- **Signal:** `breaker.opened` event; `GET /v1/status/breakers` shows an open breaker. +- **Severity:** P2 — a backend is down; the fallback chain is degrading responses. +- **Diagnose:** identify the backend (vector/keyword/graph/embedder/reranker/llm); check its health + the gateway's degraded shape. +- **Mitigate:** the breaker auto-probes (half-open); if the backend is healthy, force-close via `POST /v1/status/breakers`. Restore the backend. +- **Escalate:** if multiple breakers open or answers are empty, page secondary. + +## drift.detected + +- **Signal:** `drift.detected` event (transition-edge) / `GET /v1/status/drift`. +- **Severity:** P3 — retrieval quality is degrading, not an outage. +- **Diagnose:** which monitor (query distribution / embedding PSI / retrieval score / citation-CTR / faithfulness); compare reference vs current. +- **Mitigate:** investigate corpus/model changes; if the new normal is legitimate, `POST /v1/status/drift/{metric}/rebaseline`. +- **Escalate:** if faithfulness drift coincides with negative feedback, involve the retrieval owner. + +## Cost anomaly (elevated) + +- **Signal:** `GET /v1/status/cost` reports `elevated` (ratio + z-score gates, Step 5.6c). +- **Severity:** P2 — runaway spend. +- **Diagnose:** identify the tenant; check `GET /v1/status/quotas` + recent query volume/token usage. +- **Mitigate:** tighten the tenant's quota (`cfg.quotas`); confirm no abuse / runaway agent loop. +- **Escalate:** billing owner if spend will breach the tenant's plan. + +## quota.exceeded + +- **Signal:** `quota.exceeded` event; tenant receiving 429s. +- **Severity:** P3 — expected throttling, unless a legitimate tenant is blocked. +- **Diagnose:** which dimension (QPS / tokens / cost / queries / storage) via `GET /v1/status/quotas`. +- **Mitigate:** raise the tenant's limit if legitimate; otherwise it's working as intended. + +## injection.blocked spike + +- **Signal:** elevated rate of `injection.blocked` events. +- **Severity:** P2 — possible attack, or a corpus poisoned with injection payloads. +- **Diagnose:** inspect which corpus/source the blocked chunks came from; confirm the guard dropped them before the LLM. +- **Mitigate:** quarantine the source; re-run the red-team gate; tune the detector threshold if false-positive-driven. +- **Escalate:** security lead if sustained or targeted. + +## Gateway degraded + +- **Signal:** `gateway.cache.degraded` / `gateway.rerank.degraded` / `gateway.answer.failed` logs. +- **Severity:** P2 — graceful degradation (no 5xx), but quality/latency impacted. +- **Diagnose:** which stage degraded (cache miss-through / reranker→retrieval-only / LLM answer failure); check the underlying backend. +- **Mitigate:** restore the backend; degradation is automatic and self-healing once it recovers. + +## Latency / availability SLO breach + +- **Signal:** Grafana SLO rule on `gateway.request_duration_ms` / `GET /v1/status/health` (Step 3.11/5.6e). +- **Severity:** P1 if availability < tier SLA; P2 if latency only. +- **Diagnose:** check breakers (open?), backend latency, cache hit rate, load. +- **Mitigate:** scale the bottleneck; engage breakers/fallback; update the status page. +- **Escalate:** declare an incident if a tier SLA is at risk. + +## eval.regression + +- **Signal:** `eval.regression` event / the CI eval-gate failing (Step 5.3). +- **Severity:** P3 — a change dropped recall/MRR/faithfulness below the baseline delta. +- **Diagnose:** the gate's PR comment + the regression bisector (Step 5.6d) to find the first bad commit. +- **Mitigate:** revert/fix the offending change; update the baseline only if the drop is intentional + reviewed. + +## Ingest failures + +- **Signal:** ingest run summary errors; `POST /v1/ingest/document` failures. +- **Severity:** P3 — new content isn't indexed; serving is unaffected. +- **Diagnose:** per-document error isolation in the ingest run summary; check parser/embedder/store health. +- **Mitigate:** re-run the failed documents; fix the failing stage; resumable crawls resume from the watermark. diff --git a/docs/runbooks/incident-response.md b/docs/runbooks/incident-response.md new file mode 100644 index 0000000..e5ac091 --- /dev/null +++ b/docs/runbooks/incident-response.md @@ -0,0 +1,49 @@ +# Incident response + +How we run an incident from detection to postmortem. + +## Severity + +| Sev | Definition | Examples | +|-----|-----------|----------| +| **P1** | Service down or data at risk | availability < SLA, ACL/tenant-escape violation, total retrieval failure | +| **P2** | Major degradation | a backend down (breaker open), cost runaway, latency SLO breach | +| **P3** | Minor / quality | drift, eval regression, ingest failures, expected throttling | +| **P4** | Question / cosmetic | docs, non-urgent requests | + +## Roles + +- **Incident Commander (IC)** — owns the incident, makes decisions, delegates. The + paged on-call is IC until handed off. +- **Comms** — owns status-page updates + customer/stakeholder comms. +- **Ops/Eng** — execute the runbook mitigation. + +For a P1, IC and Comms are distinct people. + +## Flow + +1. **Detect** — an alert pages on-call (PagerDuty via the webhook/Grafana wiring) or + a report arrives. +2. **Declare** — open an incident channel; set severity; IC takes command. +3. **Mitigate** — follow the [alert runbook](alerts.md); prioritise restoring + service over root-causing. +4. **Communicate** — post an initial status-page update within 15 min for P1/P2, + then on a regular cadence (P1: every 30 min). +5. **Resolve** — confirm the signal is healthy (`/v1/status/health`, the relevant + `/v1/status/*`, the Grafana SLO); close the incident. +6. **Postmortem** — within 5 business days, blameless, using the + [template](postmortem-template.md). Every action item gets an owner + a date. + +## Status page + +A public status page reports component health + historical uptime: + +- **Components:** Gateway API · Retrieval · Ingest · Admin console. +- **Source of truth:** `GET /v1/status/health` (wiring + per-route error rate, Step + 3.11) + the Grafana SLO rules (Step 5.6e) — the page reflects measured signals, + not manual toggles, except during a declared incident when Comms posts updates. +- **Historical uptime:** computed from the availability series backing the tier SLAs + (see [support-sla.md](../guides/support-sla.md#sla-targets)). +- **Implementation:** an external status-page service (e.g. an Upptime-style repo or + a hosted provider) polling `/healthz` + `/v1/status/health`. The probe targets + + component map are the config artifact; the hosting is operational (deferred). diff --git a/docs/runbooks/postmortem-template.md b/docs/runbooks/postmortem-template.md new file mode 100644 index 0000000..7d89d6e --- /dev/null +++ b/docs/runbooks/postmortem-template.md @@ -0,0 +1,55 @@ +# Postmortem — `` + +> Blameless. Focus on systems and signals, not individuals. Copy this file to +> `docs/runbooks/postmortems/YYYY-MM-DD-.md` and fill it in within 5 business +> days of resolution. + +| Field | Value | +|-------|-------| +| Date | `YYYY-MM-DD` | +| Severity | `P1 / P2 / P3` | +| Duration | `` | +| Incident Commander | `` | +| Authors | `` | + +## Summary + +`<2–3 sentences: what happened, who was affected, how it was resolved.>` + +## Impact + +- Tenants affected: `` +- SLA impact: `` +- Data impact: `` + +## Detection + +- Alert / signal that fired: `` (``) +- Time to detect: `` + +## Timeline + +| Time (UTC) | Event | +|------------|-------| +| `HH:MM` | `` | +| `HH:MM` | `` | +| `HH:MM` | `` | +| `HH:MM` | `` | + +## Root cause + +`` + +## What went well / what didn't + +- Went well: `<…>` +- Didn't: `<…>` + +## Action items + +| Action | Owner | Due | Issue | +|--------|-------|-----|-------| +| `` | `` | `YYYY-MM-DD` | `<#link>` | + +> Quality regressions found here should also become an eval golden-set case or a +> drift threshold change (close the loop, per the Step 5.x signals).