docs(support): support tiers + SLA + per-alert runbooks + incident response (Step 7.8)

TRACKER.md

@@ -14,12 +14,13 @@
 | | |
 |---|---|
 | **Last updated** | 2026-06-10 |
-| **Current phase** | Phase 7 — Pilot, Harden, GA (**7 / 10 steps**) — Phases 0–6 ✅ complete |
-| **Overall** | **81 / 84 steps** — Phases 0–6 complete |
-| **Next action** | **Step 7.8 — Support, SLAs, on-call**: support tiers (Community/Pro/Business/Enterprise); SLA targets; incident response + postmortem template; per-alert runbooks; public status page; PagerDuty tie-in. |
+| **Current phase** | Phase 7 — Pilot, Harden, GA (**8 / 10 steps**) — Phases 0–6 ✅ complete |
+| **Overall** | **82 / 84 steps** — Phases 0–6 complete |
+| **Next action** | **Step 7.9 — Billing & metering**: per-tenant metering (queries / docs / storage / reranker / LLM tokens); usage-export API; Stripe metered billing behind a seam; invoice generation; tenant usage dashboard. |
 
 **Recently shipped**
 
+- **7.8** ✅ Support, SLAs & on-call — the support model + ops runbooks, **grounded in the platform's own signals**.  Four support tiers + per-tier SLA targets (matching `marketplace/pricing.yaml`) in [`support-sla.md`](docs/guides/support-sla.md); a new `docs/runbooks/` area with **one runbook per alert type** (`alerts.md` — acl/tenant escape · `breaker.opened` · `drift.detected` · cost anomaly · `quota.exceeded` · injection spike · gateway degraded · latency/availability SLO · `eval.regression` · ingest), each naming the exact event / `/v1/status/*` / SLO that raises it; an incident-response process + a blameless postmortem template; the status page reads **measured** health (`/v1/status/health` + Grafana SLOs), not manual toggles.  **Paging reuses the Step 3.9 webhook system** (PagerDuty as a subscriber — no new code).  Deferred: hosted status-page deploy, the PagerDuty account, staffing.  [ADR-0050](docs/adr/ADR-0050-support-sla-oncall.md)  [#181](https://github.com/officialCodeWork/AgentContextOS/pull/181)
 - **7.7** ✅ Packaging & distribution — **one-command install per channel** + **one tag fans out to every channel**.  New `rag-platform` PyPI **meta-package** (`packaging/rag-platform/`) pins the workspace component dists for a reproducible `pip install rag-platform`; the existing `agentcontextos` SDK scope is kept (no rename).  A new `release.yml` publishes PyPI (OIDC Trusted Publishing) + npm (`@agentcontextos/sdk`, provenance) + the **admin-ui image** (new `apps/admin-ui/Dockerfile`, cosign + SBOM) + the Helm chart → **OCI** (cosign), alongside the existing `docker.yml` (gateway image, cosign + SBOM) and `release-airgap.yml` — all keyed off the same `vX.Y.Z` tag.  Deferred: Go/Java/.NET SDK publish, worker/eval-runner images (single-process today).  [ADR-0049](docs/adr/ADR-0049-packaging-distribution.md)  [#180](https://github.com/officialCodeWork/AgentContextOS/pull/180)
 - **7.6** ✅ Marketplace listings — listing artifacts for AWS / Azure / GCP under a new `marketplace/` (approval + procurement stay an external per-cloud process, like the 7.3 pentest).  **One canonical `pricing.yaml`** (tiers + usage-metered dimensions whose ids match the Step 7.9 metering signals) + **one shared `listing-content.md`** mapped to all three clouds, so listings can't drift and prices reconcile with metering.  Each per-cloud spec maps the model to that cloud's billing primitives and **reuses existing delivery** (Helm / AMI / air-gap bundle / GHCR — no new packaging); per-cloud submission checklists + a procurement acceptance test in the guide.  [ADR-0048](docs/adr/ADR-0048-marketplace-listings.md)  [#179](https://github.com/officialCodeWork/AgentContextOS/pull/179)
 - **7.5** ✅ Documentation site — a **Docusaurus 3** app (`website/`) that serves the repo `docs/` tree **in place** (single source of truth; `markdown.format: 'detect'` renders hand-written `.md` as CommonMark).  The **REST API reference** (`docs/reference/rest-api.md`) is **generated + drift-gated** from `dist/openapi.json` by `scripts/gen_api_reference.py` (`task docs:api`) — never hand-edited.  Doc honesty is a `tests/docs/` suite (every `ragctl` / `/v1/` reference in a quickstart must be real) + `lychee` offline link-check + `codespell`, wired into a new `docs` CI workflow.  Deferred: Algolia search, versioned docs, the deploy workflow.  [ADR-0047](docs/adr/ADR-0047-documentation-site.md)  [#178](https://github.com/officialCodeWork/AgentContextOS/pull/178)
@@ -812,7 +813,7 @@ New ground — the only prior crypto was HMAC signing.  The V1 plan calls for en
 | 7.5 | Documentation site | ✅ | [#178](https://github.com/officialCodeWork/AgentContextOS/pull/178) — Docusaurus app (`website/`) serving `docs/` in place; **generated + drift-gated REST API reference** from `dist/openapi.json`; `tests/docs/` quickstart-honesty suite + `lychee`/`codespell`; `docs` CI workflow; `task docs:api`/`docs:build` |
 | 7.6 | Marketplace listings | ✅ | [#179](https://github.com/officialCodeWork/AgentContextOS/pull/179) — `marketplace/` listing artifacts (AWS/Azure/GCP); one canonical `pricing.yaml` (dims = 7.9 metering signals) + shared listing copy mapped to all three; reuse Helm/AMI/airgap/GHCR delivery; submission checklists + procurement test; approval = external process |
 | 7.7 | Packaging & distribution | ✅ | [#180](https://github.com/officialCodeWork/AgentContextOS/pull/180) — `rag-platform` PyPI meta-package; `release.yml` fans one tag out to PyPI/npm/GHCR(admin-ui)/Helm-OCI (cosign-keyless + SPDX SBOM + OIDC/provenance) alongside `docker.yml`+`release-airgap.yml`; `apps/admin-ui/Dockerfile`; channels matrix + guide |
-| 7.8 | Support & SLA | ⏳ | Support tiers defined; SLA dashboards; PagerDuty integration; runbooks |
+| 7.8 | Support & SLA | ✅ | [#181](https://github.com/officialCodeWork/AgentContextOS/pull/181) — support tiers + SLA targets (= `pricing.yaml`); `docs/runbooks/` with **a runbook per alert type** (tied to real events / `/v1/status/*` / SLOs); incident-response + postmortem template; status page from measured health; PagerDuty via the Step 3.9 webhook system (no new code) |
 | 7.9 | Billing integration | ⏳ | Stripe metered billing; usage export API; invoice generation |
 | 7.10 | GA cutover | ⏳ | `main` tag `v1.0.0`; release notes; all Phase 7 exit gates passed |
 
@@ -875,6 +876,15 @@ New ground — the only prior crypto was HMAC signing.  The V1 plan calls for en
 - **Signed + SBOM, keyless.**  cosign keyless (GitHub OIDC) for images + chart, SPDX SBOM per image, OIDC Trusted Publishing for PyPI, `--provenance` for npm, cosign-over-`SHA256SUMS` for the air-gap bundle — no long-lived registry tokens.
 - **Scope:** the meta-package + `release.yml` + the admin-ui Dockerfile + a channels-matrix `packaging/README.md` + the guide.  **Deferred:** publishing the generated Go/Java/.NET SDKs; separate `worker`/`eval-runner` images (single gateway process today; `core` is a library, not an image); the one-time trusted-publisher / GHCR setup + first publish (external).  [ADR-0049](docs/adr/ADR-0049-packaging-distribution.md), [guides/packaging-distribution.md](docs/guides/packaging-distribution.md).
 
+### 7.8 — Support, SLAs & on-call ✅ [#181](https://github.com/officialCodeWork/AgentContextOS/pull/181)
+
+- The support + ops layer, **grounded in the signals the platform already emits** rather than a new alerting subsystem.
+- **Support tiers + SLA** ([`support-sla.md`](docs/guides/support-sla.md)): Community / Pro / Business / Enterprise with response targets; per-tier availability + latency SLAs that **match `marketplace/pricing.yaml`** (sold = promised = measured); the follow-the-sun on-call + escalation.
+- **A runbook for every alert type** (`docs/runbooks/alerts.md`): acl/tenant escape · `breaker.opened` · `drift.detected` · cost anomaly · `quota.exceeded` · injection spike · gateway degraded · latency/availability SLO · `eval.regression` · ingest failures — each names the exact event / `/v1/status/*` surface / SLO that raises it, a diagnosis path, a mitigation (often the existing self-healing), and an escalation.
+- **Incident response + postmortem** (`incident-response.md` + `postmortem-template.md`): severity ladder, IC/Comms/Ops roles, the detect→resolve flow, and a **status page that reflects measured health** (`/v1/status/health` + Grafana SLOs, Step 3.11/5.6e) — not manual toggles.
+- **Paging reuses the Step 3.9 webhook system** — PagerDuty is a subscriber for `audit.policy_violation` / `drift.detected` / `eval.regression` (Events API v2), no new code.
+- **Scope:** the guide + the `docs/runbooks/` area + this ADR.  No code/`dist` change.  **Deferred:** the hosted status-page deploy, the PagerDuty account + routing key, staffing the rotation (people process).  [ADR-0050](docs/adr/ADR-0050-support-sla-oncall.md), [runbooks/](docs/runbooks/README.md).
+
 ---
 
 ## PR & Branch History
@@ -1029,6 +1039,7 @@ Complete log of every PR. Routine Dependabot bumps are grouped; everything else
 | [#178](https://github.com/officialCodeWork/AgentContextOS/pull/178) | 2026-06-10 | feat(docs): Docusaurus documentation site + generated API reference + doc-tests (Step 7.5) |
 | [#179](https://github.com/officialCodeWork/AgentContextOS/pull/179) | 2026-06-10 | docs(marketplace): AWS/Azure/GCP listing artifacts + canonical pricing model (Step 7.6) |
 | [#180](https://github.com/officialCodeWork/AgentContextOS/pull/180) | 2026-06-10 | feat(release): rag-platform PyPI meta + release.yml (PyPI/npm/Helm-OCI/admin-ui image) (Step 7.7) |
+| [#181](https://github.com/officialCodeWork/AgentContextOS/pull/181) | 2026-06-10 | docs(support): support tiers + SLA + per-alert runbooks + incident response (Step 7.8) |
 | #78–#80, #116–#118 | Open | Dependabot bumps — awaiting merge |
 | #81 | Closed | Dependabot bump — superseded |
 

docs/README.md

@@ -135,6 +135,7 @@
 | [documentation-site.md](guides/documentation-site.md) | The documentation site (Step 7.5): the Docusaurus app under `website/` that serves `docs/` in place; running/building it; the generated REST API reference; the CI gates (API-ref drift · quickstart doc-tests · spell-check · `lychee` link-check · site build); adding docs |
 | [marketplace-listings.md](guides/marketplace-listings.md) | Cloud marketplace listings (Step 7.6): what's in-repo (`marketplace/` — one canonical `pricing.yaml` + shared listing copy + per-cloud AWS/Azure/GCP specs) vs the external approval/procurement process; reusing the Helm/AMI/airgap/GHCR delivery; the submission checklists + the procurement acceptance test |
 | [packaging-distribution.md](guides/packaging-distribution.md) | Packaging & distribution (Step 7.7): the one-command install per channel (PyPI `rag-platform`, npm `@agentcontextos/sdk`, GHCR images, Helm OCI, air-gap); how one `vX.Y.Z` tag fans out across `docker.yml` + `release.yml` + `release-airgap.yml`; cosign-keyless + SPDX SBOM + OIDC/provenance signing; verification commands |
+| [support-sla.md](guides/support-sla.md) | Support, SLAs & on-call (Step 7.8): the four support tiers + response targets; SLA targets per tier (matching `marketplace/pricing.yaml`); the follow-the-sun on-call + escalation; wiring platform alerts to **PagerDuty** via the Step 3.9 webhook system (no new code) |
 | [curl-quickstart.md](guides/curl-quickstart.md) | 🥈 Curl-able RAG (Step 3.1): 5-minute walkthrough from `curl` to gateway response, including ingest, query, generate, OpenAPI |
 | [grpcurl-quickstart.md](guides/grpcurl-quickstart.md) | gRPC quickstart (Step 3.2): 5-minute walkthrough using `grpcurl` against the in-process server — health check, list corpora, server-streaming query, structured errors |
 | [mcp-quickstart.md](guides/mcp-quickstart.md) | MCP quickstart (Step 3.3): 5-minute walkthrough — `ragctl mcp-query`, running the stdio server, mounting `@ragplatform/mcp` in Claude Desktop, the three tools, error shape |
@@ -202,6 +203,7 @@ broken, and what to fix before committing to the next phase.
 | [ADR-0046-design-partner-pilots.md](adr/ADR-0046-design-partner-pilots.md) | Decision (Step 7.4): ship the pilot **machine** in-repo (runbook + templates + per-vertical kits + a `ragctl pilot` KPI reader) while the partner **relationships** stay an external GTM deliverable (like the 7.3 pentest); each success criterion (quality / latency / integration / security) maps to a platform signal (eval / feedback / drift / cost / metrics / compliance) so KPIs are **pulled, not self-reported**; sliced 7.4a–d, leading with customer-support/KB, the framework vertical-extensible; no new package / core type / governed SPI call / `dist` change — the report reads existing status endpoints; rejected a pure-external motion, self-reported KPIs, a `rag-pilot` package, one generic kit |
 | [ADR-0047-documentation-site.md](adr/ADR-0047-documentation-site.md) | Decision (Step 7.5): publish docs via **Docusaurus** sourcing the repo `docs/` tree **in place** (single source of truth, no copy); `markdown.format: 'detect'` so hand-written `.md` renders as CommonMark; the **REST API reference is generated + drift-gated** from `dist/openapi.json` (a committed file, not a build-time plugin); doc honesty is a `tests/docs/` suite (every `ragctl` / `/v1/` reference is real) + `lychee` + `codespell`; rejected MkDocs, copying `docs/`, an OpenAPI plugin |
 | [ADR-0049-packaging-distribution.md](adr/ADR-0049-packaging-distribution.md) | Decision (Step 7.7): a `rag-platform` PyPI **meta-package** pins the workspace component dists (one-command server install); keep the existing `agentcontextos` SDK scope (no rename); **one `vX.Y.Z` tag fans out** to PyPI/npm/GHCR/Helm-OCI/air-gap across `docker.yml` + a new `release.yml` + `release-airgap.yml`; cosign-keyless + SPDX SBOM + OIDC/provenance everywhere; deferred Go/Java/.NET publish + worker/eval-runner images; rejected a `ragplatform` rename + one mega-workflow |
+| [ADR-0050-support-sla-oncall.md](adr/ADR-0050-support-sla-oncall.md) | Decision (Step 7.8): **runbooks grounded in the platform's own signals** — one per alert type, each naming the exact event / `/v1/status/*` / SLO that raises it; **paging reuses the Step 3.9 webhook system** (PagerDuty as a subscriber, no new code) + Grafana SLOs; the status page reflects **measured** health (not manual toggles); SLA tiers match `marketplace/pricing.yaml`; rejected a new alerting subsystem, generic runbooks, a separate SLA datastore |
 | [ADR-0048-marketplace-listings.md](adr/ADR-0048-marketplace-listings.md) | Decision (Step 7.6): one canonical `marketplace/pricing.yaml` (metered dims = the Step 7.9 signals) + one shared listing copy mapped to all three clouds, so listings can't drift and prices reconcile with metering; **reuse existing delivery** (Helm / AMI / airgap / GHCR) — the per-cloud Marketplace wrappers are thin shells; approval + procurement are documented external process, not a faked status; rejected per-cloud pricing, custom packaging, encoding a "live" status |
 | [ADR-0042-compliance-posture.md](adr/ADR-0042-compliance-posture.md) | Decision (Step 6.10, Phase-6 capstone): add the three compliance pieces on top of the controls the platform already ships (audit/ACL/PII/BYOK/SSO/quotas). New `rag-compliance` package (config-free, like rag-feedback/rag-drift): `RetentionEnforcer` drives tenant-scoped `purge_*`; `compliance_posture`/`residency_ok` are pure. **Retention is a capability on the existing stores, not a new SPI** — non-abstract `purge_before`/`purge_tenant` (default no-op) on Feedback/Provenance, with `dry_run` in the SPI so a preview counts-without-deleting uniformly (ProvenanceStore has no `list`). **Audit is never purged in place** (the hash chain would break) — audit retention is the 6.6b WORM export; `audit_days` is advisory. **Right-to-erasure is always-on, tenant-self-service, two-flag** — `POST /v1/compliance/erase` erases the *calling* tenant's data (scope from the principal, never the body), dry-run by default, delete needs `dry_run=false` AND `confirm=true`. **Residency = declared per tenant + enforced at ingest** (`tenants[].data_region` vs `cfg.compliance.region` → 403), opt-in, a single-deployment assertion not multi-region routing. **The SOC 2 / GDPR mapping is a doc backed by a live posture** (`GET /v1/status/compliance` reports which controls are on, so the mapping is checkable). Deferred: subject-level (vs tenant-level) erasure, an admin retention-sweep endpoint, multi-region routing, automated audit-evidence bundles; rejected purging the audit chain, a `Purgeable` SPI mixin, a static doc with no live backing |
 | [ADR-0041-airgap-bundle.md](adr/ADR-0041-airgap-bundle.md) | Decision (Step 6.9): ship the platform as one signed, self-contained offline bundle (all runtime images + Helm chart + config + installer). Integrity reuses the WORM-export pattern (6.6b): a standard `SHA256SUMS` whose hash is pinned as `manifest.content_hash` is the **hard gate** — verifiable with nothing but `sha256sum`, no network/cosign — and a cosign signature **over `SHA256SUMS`** adds authenticity; the *same* `SHA256SUMS` drives the Python verifier and the standalone shell installer so they can't diverge. The shell/pwsh `install.{sh,ps1}` (shipped inside the bundle) need only docker+helm (air-gap hosts lack uv/the workspace); `ragctl airgap` holds the typed/tested build+verify logic (pure core separated from a stubbable docker/helm/cosign subprocess seam; `--dry-run` = a verifiable bundle minus image blobs, so the path is testable with no Docker). Digest-pinned manifest-driven image set (`infra/airgap/images.txt` + the chart-derived gateway image); key-based cosign is the air-gap recommendation (keyless needs Rekor + an identity policy), keyless is the connected-release path (`release-airgap.yml` on tags). Deferred: ctr/podman load, registry re-tag/push, multi-arch selection, bundling backend charts, TUF-rooted offline keyless verify; rejected `oras`/OCI (no registry to pull from in an air-gap), a second HMAC scheme (cosign already the signer), a pure-shell build (would escape mypy/tests) |
@@ -237,6 +239,18 @@ holds the fill-in templates, per-vertical kits, and published case studies.
 | [customer-support/README.md](pilots/customer-support/README.md) | Customer-support / internal-KB pilot kit (Step 7.4b) — sample corpus (PII handbook + product FAQ + a planted injection probe), domain-calibrated success criteria (deflection), the `ragctl pilot` seed-and-demo flow, the PII + injection security demonstration |
 | [customer-support/case-study.md](pilots/customer-support/case-study.md) | Worked case study (Step 7.4d) — the framework run end-to-end on the kit; real `ragctl pilot report` KPIs (satisfaction +0.733 · 0/5 drift · cost ok → PASS) + the PII + injection security demonstration |
 
+## runbooks/
+
+Operational runbooks (Step 7.8) — one per alert the platform can raise, plus the
+incident-response process and the postmortem template.
+
+| File | Description |
+|------|-------------|
+| [README.md](runbooks/README.md) | Runbooks index — the alert → runbook table + the Step 7.8 acceptance checklist |
+| [alerts.md](runbooks/alerts.md) | One runbook per alert type (acl/tenant escape · `breaker.opened` · `drift.detected` · cost anomaly · `quota.exceeded` · injection spike · gateway degraded · latency/availability SLO · `eval.regression` · ingest failures), each tied to the platform event / `/v1/status/*` signal that raises it |
+| [incident-response.md](runbooks/incident-response.md) | Severity ladder · IC/Comms/Ops roles · detect→declare→mitigate→communicate→resolve→postmortem flow · the status page (measured from `/v1/status/health` + Grafana SLOs) |
+| [postmortem-template.md](runbooks/postmortem-template.md) | Blameless postmortem template (summary · impact · timeline · root cause · action items) |
+
 ## research/
 
 | File | Description |