diff --git a/TRACKER.md b/TRACKER.md index 62f51a9..27800c3 100644 --- a/TRACKER.md +++ b/TRACKER.md @@ -14,12 +14,13 @@ | | | |---|---| | **Last updated** | 2026-06-10 | -| **Current phase** | Phase 7 — Pilot, Harden, GA (**8 / 10 steps**) — Phases 0–6 ✅ complete | -| **Overall** | **82 / 84 steps** — Phases 0–6 complete | -| **Next action** | **Step 7.9 — Billing & metering**: per-tenant metering (queries / docs / storage / reranker / LLM tokens); usage-export API; Stripe metered billing behind a seam; invoice generation; tenant usage dashboard. | +| **Current phase** | Phase 7 — Pilot, Harden, GA (**9 / 10 steps**) — Phases 0–6 ✅ complete | +| **Overall** | **83 / 84 steps** — Phases 0–6 complete | +| **Next action** | **Step 7.10 — GA cutover** (the final step): version bump to v1.0.0; CHANGELOG / release notes; GA readiness checklist (all Phase 7 exit gates); final security-review + eval sign-off; cut the tag/release. | **Recently shipped** +- **7.9** ✅ Billing & metering — per-tenant usage accounting, **mirroring the 5.6c cost pattern** (dataclasses in `rag-observability`, the gateway wraps a Pydantic response → **no `dist/schemas` churn**). New `rag_observability.billing`: a `UsageMeter` (observe-only per-tenant, per-dimension counters: queries / docs / storage / reranker / LLM tokens) fed O(1) from `record_request_usage` independent of quotas; `GET /v1/billing/usage` serves the per-tenant dashboard; pure `generate_invoice` + `PricingModel`/`load_pricing` price an invoice **from `marketplace/pricing.yaml`** (a price change is config) and **reconcile with metered usage ±0.5% by construction**; a `BillingProvider` Protocol seam for Stripe / marketplace metering (degrade-open). Inert by default (`cfg.billing`); `ragctl billing` drives meter→invoice→reconcile; **no governed SPI call** (policy-coverage linter unchanged); only `dist/openapi` + `dist/rag.schema` regenerate. Deferred: the concrete Stripe/marketplace adapters, the docs/reranker/storage feed points, an admin usage card. [ADR-0051](docs/adr/ADR-0051-billing-metering.md) [#182](https://github.com/officialCodeWork/AgentContextOS/pull/182) - **7.8** ✅ Support, SLAs & on-call — the support model + ops runbooks, **grounded in the platform's own signals**. Four support tiers + per-tier SLA targets (matching `marketplace/pricing.yaml`) in [`support-sla.md`](docs/guides/support-sla.md); a new `docs/runbooks/` area with **one runbook per alert type** (`alerts.md` — acl/tenant escape · `breaker.opened` · `drift.detected` · cost anomaly · `quota.exceeded` · injection spike · gateway degraded · latency/availability SLO · `eval.regression` · ingest), each naming the exact event / `/v1/status/*` / SLO that raises it; an incident-response process + a blameless postmortem template; the status page reads **measured** health (`/v1/status/health` + Grafana SLOs), not manual toggles. **Paging reuses the Step 3.9 webhook system** (PagerDuty as a subscriber — no new code). Deferred: hosted status-page deploy, the PagerDuty account, staffing. [ADR-0050](docs/adr/ADR-0050-support-sla-oncall.md) [#181](https://github.com/officialCodeWork/AgentContextOS/pull/181) - **7.7** ✅ Packaging & distribution — **one-command install per channel** + **one tag fans out to every channel**. New `rag-platform` PyPI **meta-package** (`packaging/rag-platform/`) pins the workspace component dists for a reproducible `pip install rag-platform`; the existing `agentcontextos` SDK scope is kept (no rename). A new `release.yml` publishes PyPI (OIDC Trusted Publishing) + npm (`@agentcontextos/sdk`, provenance) + the **admin-ui image** (new `apps/admin-ui/Dockerfile`, cosign + SBOM) + the Helm chart → **OCI** (cosign), alongside the existing `docker.yml` (gateway image, cosign + SBOM) and `release-airgap.yml` — all keyed off the same `vX.Y.Z` tag. Deferred: Go/Java/.NET SDK publish, worker/eval-runner images (single-process today). [ADR-0049](docs/adr/ADR-0049-packaging-distribution.md) [#180](https://github.com/officialCodeWork/AgentContextOS/pull/180) - **7.6** ✅ Marketplace listings — listing artifacts for AWS / Azure / GCP under a new `marketplace/` (approval + procurement stay an external per-cloud process, like the 7.3 pentest). **One canonical `pricing.yaml`** (tiers + usage-metered dimensions whose ids match the Step 7.9 metering signals) + **one shared `listing-content.md`** mapped to all three clouds, so listings can't drift and prices reconcile with metering. Each per-cloud spec maps the model to that cloud's billing primitives and **reuses existing delivery** (Helm / AMI / air-gap bundle / GHCR — no new packaging); per-cloud submission checklists + a procurement acceptance test in the guide. [ADR-0048](docs/adr/ADR-0048-marketplace-listings.md) [#179](https://github.com/officialCodeWork/AgentContextOS/pull/179) @@ -814,7 +815,7 @@ New ground — the only prior crypto was HMAC signing. The V1 plan calls for en | 7.6 | Marketplace listings | ✅ | [#179](https://github.com/officialCodeWork/AgentContextOS/pull/179) — `marketplace/` listing artifacts (AWS/Azure/GCP); one canonical `pricing.yaml` (dims = 7.9 metering signals) + shared listing copy mapped to all three; reuse Helm/AMI/airgap/GHCR delivery; submission checklists + procurement test; approval = external process | | 7.7 | Packaging & distribution | ✅ | [#180](https://github.com/officialCodeWork/AgentContextOS/pull/180) — `rag-platform` PyPI meta-package; `release.yml` fans one tag out to PyPI/npm/GHCR(admin-ui)/Helm-OCI (cosign-keyless + SPDX SBOM + OIDC/provenance) alongside `docker.yml`+`release-airgap.yml`; `apps/admin-ui/Dockerfile`; channels matrix + guide | | 7.8 | Support & SLA | ✅ | [#181](https://github.com/officialCodeWork/AgentContextOS/pull/181) — support tiers + SLA targets (= `pricing.yaml`); `docs/runbooks/` with **a runbook per alert type** (tied to real events / `/v1/status/*` / SLOs); incident-response + postmortem template; status page from measured health; PagerDuty via the Step 3.9 webhook system (no new code) | -| 7.9 | Billing integration | ⏳ | Stripe metered billing; usage export API; invoice generation | +| 7.9 | Billing integration | ✅ | [#182](https://github.com/officialCodeWork/AgentContextOS/pull/182) — `rag_observability.billing`: observe-only `UsageMeter` (5 dimensions) + `GET /v1/billing/usage`; pure `generate_invoice` priced from `pricing.yaml` (reconciles ±0.5%); `BillingProvider` Stripe/marketplace seam; `cfg.billing`; `ragctl billing`; mirrors 5.6c (no `dist/schemas` churn) | | 7.10 | GA cutover | ⏳ | `main` tag `v1.0.0`; release notes; all Phase 7 exit gates passed | ### 7.1 — Load + chaos testing ✅ [#164](https://github.com/officialCodeWork/AgentContextOS/pull/164) @@ -885,6 +886,15 @@ New ground — the only prior crypto was HMAC signing. The V1 plan calls for en - **Paging reuses the Step 3.9 webhook system** — PagerDuty is a subscriber for `audit.policy_violation` / `drift.detected` / `eval.regression` (Events API v2), no new code. - **Scope:** the guide + the `docs/runbooks/` area + this ADR. No code/`dist` change. **Deferred:** the hosted status-page deploy, the PagerDuty account + routing key, staffing the rotation (people process). [ADR-0050](docs/adr/ADR-0050-support-sla-oncall.md), [runbooks/](docs/runbooks/README.md). +### 7.9 — Billing & metering ✅ [#182](https://github.com/officialCodeWork/AgentContextOS/pull/182) + +- Per-tenant usage accounting for money — a *third* concern beside cost-anomaly detection (5.6c) and quota enforcement (4.5), **mirroring the 5.6c shape** to avoid churn. +- **Dataclasses in `rag-observability`, no `dist/schemas` churn.** `rag_observability.billing` holds a `UsageMeter` (thread-safe per-tenant, per-dimension counters — `queries` / `docs_ingested` / `storage_gb` / `reranker_calls` / `llm_tokens`), `UsageSnapshot`, and the pricing/invoice types; the gateway wraps the snapshot in a gateway-local Pydantic `BillingUsageResponse` (like `CostStatusResponse`). Only `dist/openapi` (the new endpoint) + `dist/rag.schema` (the new `cfg.billing` block) regenerate. +- **Observe-only metering; enforcement stays quotas.** Fed O(1) from `record_request_usage` (queries + tokens), independently of quotas, **making no governed SPI call** (the policy-coverage linter needs no entry — same as cost/drift/feedback). `GET /v1/billing/usage` serves the per-tenant usage dashboard; inert by default (`cfg.billing`). +- **Invoicing is offline, pure, priced from config.** `generate_invoice(snapshot, pricing, tier)` is pure; `PricingModel`/`load_pricing` read `marketplace/pricing.yaml`, so a price change is config (the shared 7.6/7.9 bar). Each metered line item's quantity equals the metered overage, so **bills reconcile with usage ±0.5% by construction** (the acceptance) — `ragctl billing` prints the invoice + the reconciliation check. +- **External billers are a Protocol seam.** `BillingProvider.report_usage` (Stripe usage-records / marketplace metering APIs) with a `NoopBillingProvider` default, called degrade-open — no `rag_core.spi` ceremony. +- **Scope:** the billing module + the usage endpoint + `cfg.billing` + wiring + `ragctl billing` + tests (unit reconciliation + gateway endpoint + CLI). All gates green (ruff, mypy --strict 330 files, config/openapi-drift regenerated). **Deferred:** the concrete Stripe/marketplace adapters, the docs/reranker/storage feed points, an admin usage card, subject-level invoicing. [ADR-0051](docs/adr/ADR-0051-billing-metering.md), [reference/billing.md](docs/reference/billing.md). + --- ## PR & Branch History @@ -1040,6 +1050,7 @@ Complete log of every PR. Routine Dependabot bumps are grouped; everything else | [#179](https://github.com/officialCodeWork/AgentContextOS/pull/179) | 2026-06-10 | docs(marketplace): AWS/Azure/GCP listing artifacts + canonical pricing model (Step 7.6) | | [#180](https://github.com/officialCodeWork/AgentContextOS/pull/180) | 2026-06-10 | feat(release): rag-platform PyPI meta + release.yml (PyPI/npm/Helm-OCI/admin-ui image) (Step 7.7) | | [#181](https://github.com/officialCodeWork/AgentContextOS/pull/181) | 2026-06-10 | docs(support): support tiers + SLA + per-alert runbooks + incident response (Step 7.8) | +| [#182](https://github.com/officialCodeWork/AgentContextOS/pull/182) | 2026-06-10 | feat(billing): per-tenant usage metering + usage API + invoice generation (Step 7.9) | | #78–#80, #116–#118 | Open | Dependabot bumps — awaiting merge | | #81 | Closed | Dependabot bump — superseded | diff --git a/apps/admin-ui/design/claude-design-prompt-v2-api-coverage.md b/apps/admin-ui/design/claude-design-prompt-v2-api-coverage.md new file mode 100644 index 0000000..f76d878 --- /dev/null +++ b/apps/admin-ui/design/claude-design-prompt-v2-api-coverage.md @@ -0,0 +1,296 @@ +# Claude Design prompt — Admin UI **v2: full API coverage** + +> Companion to [`claude-design-prompt.md`](claude-design-prompt.md) (the original 9-page brief). +> This v2 brief covers **every gateway API function that does not yet have a UI**. Paste the +> **PROMPT** block into Claude.ai (artifacts) to generate the new pages as one navigable React +> mockup that matches the *already-shipped* design system. Grounded 1:1 in the real gateway +> response models (Phase 5–7), so the mockup maps straight onto the Next.js 14 implementation. + +--- + +## Why this exists (gap analysis — for me, not the prompt) + +The console already ships 13 pages. Cross-referencing the nav against the gateway's **51 REST +endpoints**, these API functions have **no UI yet**. Each becomes a page (or an action on an +existing page) below. + +| # | New surface | API function(s) with no UI | Priority | +|---|-------------|----------------------------|----------| +| 1 | **Usage & Billing** | `GET /v1/billing/usage` (+ client-side invoice from `pricing.yaml`) | **Now** — current branch (Step 7.9) | +| 2 | **Query Playground** | `POST /v1/query`, `POST /v1/retrieve`, `POST /v1/feedback` | High — demo-able core flow | +| 3 | **Ingest** | `POST /v1/ingest/document` (multipart) | High — completes the write path | +| 4 | **Chat + Embeddings** | `POST /v1/chat/completions` (SSE), `GET /v1/models`, `POST /v1/embeddings` | Medium | +| 5 | **Agent Runs** | `POST /v1/agent` (SSE event stream) | Medium | +| 6 | **Compliance** | `GET /v1/status/compliance`, `POST /v1/compliance/erase` | High — enterprise buyer surface | +| 7 | **Identity (SSO & SCIM)** | `GET /v1/status/sso`, `/scim/v2/Users`, `/scim/v2/Groups` | Medium | +| 8 | **Tenants → live** | `GET /v1/status/tenant` (wire the existing *Preview* page) | Medium | +| — | **Audit (enhance)** | add `GET /v1/audit/verify` + `POST /v1/audit/export` actions | Low | + +Already covered (no work): `/v1/corpora`, `/v1/webhooks/*`, `/v1/audit`, `/v1/query/{id}/trace`, +and the whole `/v1/status/{health,metrics,logs,breakers,quotas,drift,feedback,cost,experiments}` +family (Dashboard / Live Status / Metrics / Logs / Trace). **API Keys** stays *Preview* — no +backend endpoint exists yet. + +--- + +## PROMPT + +You are extending an **existing, shipped operator console** for **AgentContextOS**, a +multi-tenant RAG (Retrieval-Augmented Generation) platform. Build the **new pages below** as a +**single self-contained React artifact** with **Tailwind CSS**, **light + dark mode**, and +realistic inline mock data (no backend calls — simulate everything client-side). + +**Critical: match the existing design system exactly** — these pages must look like they were +always part of the app. Do not invent a new visual language. + +### Established design system (reproduce it faithfully) + +- **Aesthetic:** Vercel / Linear / Stripe operator dashboard. Neutral slate/zinc palette, + information-dense, scannable. shadcn/ui look — `rounded-lg` cards, subtle 1px borders + (`hsl(var(--border))`), soft shadows, `text-sm` default, `tabular-nums` for numbers, + **monospace for all IDs / hashes / secrets / trace ids**. Geist Sans + Geist Mono. lucide + icons. Visible focus rings, AA contrast in both themes. +- **App shell (already exists — render it so the new pages live inside it):** + - **Left sidebar** (collapsible): "AgentContextOS" + "Admin" badge. Nav is grouped. The + current groups are **Overview · Observability · Knowledge · Delivery · Governance · System**. + Add the new pages into nav as specified per-page below (introduce two new groups, + **Operate** and **Billing**). Active route highlighted; each item has a lucide icon. + - **Top bar:** tenant switcher (`acme`, `globex`, `initech`), cosmetic global search, a + light/dark toggle, and a principal menu (`svc-admin`). The **active tenant scopes every + page** — billing, usage, compliance, identity all read "the current tenant". + - **Content area:** a `PageHeader` (title + one-line description, optional status badge, + primary action top-right) then the page body, max-width ~1400px. +- **Shared primitives to reuse (same names/variants as the app):** `Card`, `Button` + (default/secondary/outline/ghost/destructive; sizes sm/md/icon), `Badge` (secondary, outline, + success=emerald, error=red, warning=amber, info=blue, violet), `StatusDot` (emerald=up, + amber=degraded, red=down, muted=unknown, optional ping), `Table` (muted header, divide-y rows, + hover), `EmptyState` (icon+title+desc+CTA), `ErrorState` (red, retry), `Skeleton` (pulse), + `Dialog` (modal), `Sheet` (right drawer), `Dropdown`, `Tooltip`, `Toast`, `ChipInput`, + `Sparkline` + `Bar` (inline SVG charts — **no charting library**). +- **Every data view must show four designed states** — `ready`, `loading` (skeletons), + `empty` (designed empty state), `error` (retry) — and provide a small toggle so I can review + each, exactly like the existing pages do. +- **Status semantics everywhere:** emerald = ok/up/healthy/positive, amber = degraded/warning, + red = error/down/blocked, muted = unknown/disabled. Relative timestamps with absolute on hover. + +Now build these pages. Use believable, varied mock data (several rows, mixed statuses, a few +long values to test truncation). Make the nav actually switch between all of them. + +--- + +### Page 1 — Usage & Billing *(new group "Billing"; icon: receipt/credit-card)* + +Per-tenant metered usage for the current billing period + a client-computed invoice preview. +Maps to `GET /v1/billing/usage`. Metering is **observe-only** — surface a small note that usage +is metered passively and **never blocks requests**. + +- **Top strip:** current tenant, billing period (e.g. "Jun 1 – Jun 30, 2026"), a **tier badge** + (`free` / `pro` / `enterprise`), and a **period-to-date total** (big number, currency). +- **Usage cards — one per billing dimension** (these are the exact metered dimensions): + `queries`, `docs_ingested`, `storage_gb`, `reranker_calls`, `llm_tokens`. Each card: dimension + label, **accrued quantity** (big, `tabular-nums`), the tier **included allotment** as a thin + progress bar (used vs included), and **overage** highlighted amber when used > included. Add a + tiny `Sparkline` of the last ~14 days (mock). +- **Invoice preview** (computed client-side from usage × pricing): a table of **line items** — + `{ description, dimension, quantity, unit_price, amount }` — starting with a `" plan + (monthly base)"` row, then one **overage** row per dimension that exceeded its inclusion + (`" overage ()"`), and a bold **Total**. Pricing is data: tiers have a + `monthly_base` + per-dimension `inclusions`; metered rates have `{ unit, unit_size, price }` + (e.g. queries priced per `1k`, llm_tokens per `1M`). Show currency (USD). +- **Actions (cosmetic):** "Download invoice (PDF)", "Export usage (CSV)", and a "Billing + provider: Stripe — connected / not connected" row (the platform reports usage to an external + biller behind a seam). +- **Data shapes:** + - usage: `{ tenant_id, usage: { queries: 128400, docs_ingested: 920, storage_gb: 14.6, + reranker_calls: 41200, llm_tokens: 8_900_000 } }` + - invoice: `{ tenant_id, tier_id, currency: "USD", line_items: [{ description, dimension, + quantity, unit_price, amount }], total }` + +### Page 2 — Query Playground *(new group "Operate"; icon: search/play)* + +Run a real RAG query and inspect the result. Maps to `POST /v1/query` and `POST /v1/retrieve`. + +- **Left control column:** a query **textarea**; a **corpus** multiselect; a **top-k** slider + (1–50); toggles for **Rerank** and **Generate answer**; an "Advanced" disclosure with ACL + labels (`ChipInput`) and metadata filters. Two buttons: **Run query** (full pipeline) and + **Retrieve only** (no rerank/pack/LLM). +- **Right results column:** + - **Answer panel** (only when "Generate answer" on): the generated answer with **inline + citation markers** `[1] [2]` that map to chunks; a footer with 👍 / 👎 (submits + `POST /v1/feedback`, signal `thumbs_up`/`thumbs_down`), **Copy answer** (implicit `copy` + signal), and **Regenerate** (implicit `regenerate` signal). + - **Ranked chunks list:** each row = rank #, **score** (badge), source corpus + doc title, + a snippet (expandable), and chunk id (mono, copyable). Clicking a chunk opens a right + `Sheet` with full chunk text + metadata. + - **Timings strip:** a compact horizontal bar breaking down latency by stage (understanding → + retrieve → rerank → pack → generate, in ms) plus **cache hit** badges and total ms. + - A **"View full trace →"** link that deep-links to the existing Query Trace page for this + `request_id`. +- **States:** empty = "Run a query to see results"; loading = streaming/skeleton chunks; error + = designed error with retry. +- **Data shapes:** request `{ query, corpus_ids: [], top_k, rerank, generate_answer, filters }`; + response `{ request_id, answer?, citations: [{ marker, chunk_id }], chunks: [{ chunk_id, + score, corpus_id, doc_title, snippet, metadata }], metadata: { timings_ms: { understanding, + retrieve, rerank, pack, generate }, cache_hits: { embedding, retrieval, answer }, total_ms } }`. + +### Page 3 — Ingest *(group "Knowledge"; icon: upload)* + +Push a document into a corpus. Maps to `POST /v1/ingest/document` (multipart). + +- **Drop zone** (drag-and-drop + file picker) accepting PDF/DOCX/PPTX/XLSX/MD/HTML/TXT/CSV/JSON. +- **Form:** target **corpus** select, optional **source URI**, principal (read-only = + current principal). **Ingest** button. +- **Upload queue:** one row per file with a live **status** progression + `pending → ingested / failed` (`StatusDot` + badge), and on success the returned + **chunk_count** and **embedding_count**; on failure an error reason. A success row links to the + corpus ("View in Corpora →"). Note in the UI that the pipeline is *parse → chunk → enrich → + PII-redact → embed → index* and that **PII is redacted before embedding**. +- **States:** empty = "Drop files to ingest"; loading = per-file progress; error = per-file. +- **Data shape:** result `{ status: "pending"|"ingested"|"failed", document_id, corpus_id, + chunk_count, embedding_count, error? }`. + +### Page 4 — Chat + Embeddings *(group "Operate"; icon: message-square)* + +OpenAI-compatible playground. Maps to `POST /v1/chat/completions` (streaming), `GET /v1/models`, +`POST /v1/embeddings`. Use **tabs**: **Chat** · **Embeddings** · **Models**. + +- **Chat tab:** a chat thread (user/assistant bubbles); a composer; a right rail with **model** + picker (from Models), **temperature** slider, **RAG corpus** selector, and a **stream tokens** + toggle. Assistant replies **stream in token-by-token** (simulate with a typing effect) and show + **citation chips** beneath when RAG context was used. Show a per-message **usage** footer + (prompt/completion tokens). +- **Embeddings tab:** a textarea, a model picker, **Embed** button → show vector **dimension**, + **L2 norm**, and a preview of the first ~12 values (mono grid); support multiple inputs + (one vector card each) and a usage/token count. +- **Models tab:** a small table of available models — `{ id, type: "embedder"|"llm", context, + dimension? }` — with a copy-id button. +- **States** on each tab (empty/loading/error). + +### Page 5 — Agent Runs *(group "Operate"; icon: bot/workflow)* + +Drive the iterative agent and watch it think. Maps to `POST /v1/agent` (Server-Sent Events). + +- **Top:** a **goal** input, optional corpus + max-steps, and a **Run** button. A run header + shows status (`running` / `completed` / `failed`), elapsed time, and step count. +- **Live event timeline** (the core of the page): render the streamed `AgentEvent` frames as a + vertical timeline — `run_started`, `phase_changed` (phase chip: plan / retrieve / reason / + answer), `tool_started` / `tool_completed` (collapsible card with tool name, args JSON, + result), `answer_delta` (streaming answer block that grows), `checkpoint_saved` (small marker), + `run_completed` / `run_failed`. Use the typing effect for `answer_delta`. Distinct icons + + colors per event type. +- **Past runs** (left list or below): mock prior runs (goal, status, duration, steps) — click to + replay its timeline. +- **States:** empty = "Start a run"; loading = "waiting for first event…"; error = failed run + with the failure frame surfaced. + +### Page 6 — Compliance *(group "Governance"; icon: shield-check)* + +Live compliance posture + GDPR right-to-erasure. Maps to `GET /v1/status/compliance` and +`POST /v1/compliance/erase`. This is an **enterprise-buyer surface** — make it reassuring and +auditable, with SOC 2 / GDPR framing. + +- **Posture grid:** one card/row per control with an on/off **Badge** and a one-line "what it + means", deep-linking to the page that governs it. Controls (all booleans from the API): + `audit`, `acl`, `pii`, `byok` (BYOK encryption), `sso`, `scim`, `quotas`, `guard` + (hallucination guard), `breakers`, `retention`, `residency`. Plus a **data region** badge + (e.g. `eu-west-1`) and **retention window** (`retention_days`). Add a small "Posture score" + summary (N of M controls enabled). +- **Right-to-erasure tool** (a clearly separated **Danger zone** card): tenant (current), + a **Dry run** that previews counts — `{ feedback_records_erased, provenance_records_erased, + dry_run: true }` — then a **Delete** path gated behind a typed confirm dialog (requires an + explicit `confirm` + a second toggle), styled destructive. Explain that the audit chain is + **never purged in place** (its retention is the WORM export). +- **Data shape:** posture `{ audit, acl, pii, byok, sso, scim, quotas, guard, breakers, + retention, residency: booleans, region, retention_days }`; erasure result as above. + +### Page 7 — Identity (SSO & SCIM) *(group "Governance"; icon: users/fingerprint)* + +Federated auth + directory provisioning. Maps to `GET /v1/status/sso`, `/scim/v2/Users`, +`/scim/v2/Groups`. Two sections (or tabs): **SSO** and **SCIM Directory**. + +- **SSO section:** per-tenant card — **protocol** badge (`OIDC` / `SAML`), **configured** + status, issuer / entity-id (mono, read-only), and a "Test login" (cosmetic). A top banner shows + global `sso_enabled` / `scim_enabled`. +- **SCIM Directory section** (read-only — provisioned by the IdP): + - **Users table:** `userName`, `displayName`, `emails`, **active** (StatusDot), `groups` + (chips), `externalId` (mono). Row → `Sheet` with the full SCIM user. + - **Groups table:** `displayName`, member count, `externalId`. Row → members list. + - A **SCIM endpoint** info card: base URL `/scim/v2`, and a **per-tenant bearer token** shown + with the **reveal-once** pattern (`Copy` + "you won't see this again", masked as `••••` + afterwards) — reuse the webhook-secret reveal flow. +- **Data shapes:** sso `{ sso_enabled, scim_enabled, tenant: { protocol, configured, issuer } }`; + scim user `{ id, userName, displayName, emails: [], active, externalId, groups: [] }`; scim + group `{ id, displayName, members: [] }`. + +### Page 8 — Tenants → wire the live detail *(group "Governance"; existing page)* + +The Tenants page is currently a **Preview** stub. Keep the list + "New tenant" as Preview, but +add a **live detail drawer** backed by `GET /v1/status/tenant`: when a tenant row is clicked, +show a `Sheet` with the **resolved logical-tenancy settings** — `namespace`, `pii_action` +(allow/redact/mask/block badge), `acl_labels` (chips), `dedicated_index` (bool), and +`physical_index` (mono). Drop the "Preview" ribbon from this detail (it's real now); keep it on +the create flow. + +### Enhancement — Audit Log actions *(existing page)* + +On the existing Audit Log page add two header actions: **Verify chain** (`GET /v1/audit/verify` +→ toast/badge with `ok` + event count, green/red) and **Export (WORM)** (`POST /v1/audit/export` +→ a dialog showing `content_hash` (mono), `signed_at`, and `signature` with a Copy button, plus +"chain_verified ✓"). Make clear the export is a **tamper-evident, content-hashed, HMAC-signed** +bundle. + +### Enhancement — Dashboard tiles *(existing page)* + +Add three stat tiles that deep-link to the new pages so they're reachable from Overview: +**Period-to-date spend** (→ Billing), **Compliance posture** (e.g. "9/11 controls" → Compliance), +and **Drift** ("5/5 monitors ok" → already on Live Status). Match the existing stat-card style. + +--- + +### Interaction + quality bar (same as the shipped app) + +- Real **modals, drawers/sheets, dropdowns, confirmation dialogs, toasts, tabs, badges, toggles, + tooltips, streaming text** — shadcn/ui style, keyboard accessible. +- Consistent spacing + date formatting (relative + absolute on hover), monospace for every + id/hash/secret/token, `tabular-nums` for all metrics. +- The nav must actually switch between **all** new pages plus the existing ones in the single + artifact, with the new **Operate** and **Billing** groups slotted in. +- Honor `prefers-reduced-motion` for the streaming/typing effects. + +**Deliverable:** one polished, navigable React artifact covering the eight surfaces above +(+ the two enhancements), each with its loading / empty / error states, visually indistinguishable +from the existing console. + +--- + +## Notes for implementation (for me, not the design prompt) + +**Live-wire order (matches build priority):** + +1. **Usage & Billing** — current branch (Step 7.9). `GET /v1/billing/usage` → + `BillingUsageResponse { tenant_id, usage: dict[dimension→float] }`. Invoice is **pure + + client-side** from `marketplace/pricing.yaml` via the `generate_invoice` shape in + `packages/observability/src/rag_observability/billing.py` (`PricingModel` / `Invoice` / + `InvoiceLineItem`). Metering is observe-only — never gate requests on it. +2. **Query Playground** + **Ingest** — wire `POST /v1/query` / `/v1/retrieve` / `/v1/feedback` + and `POST /v1/ingest/document` (multipart). Deep-link the trace to the existing `/trace?id=`. +3. **Compliance** — `GET /v1/status/compliance` (posture) + `POST /v1/compliance/erase` + (dry-run default; `confirm=true` to delete). High buyer value. +4. **Tenants live** — `GET /v1/status/tenant`; **Identity** — `GET /v1/status/sso` + + `/scim/v2/{Users,Groups}` (SCIM is per-tenant **bearer**, separate from the JWT/header auth). +5. **Chat / Embeddings / Agent** — `POST /v1/chat/completions` (SSE), `GET /v1/models`, + `POST /v1/embeddings`, `POST /v1/agent` (SSE). Streaming surfaces — reuse the SSE handling. + +**Conventions to keep:** header-driven identity (active tenant → `X-Tenant-Id`, fixed +`X-Principal-Id`); hybrid **live → seed fallback** with a visible "Demo data" badge so every page +renders in CI / screenshots; exact wire types come from `dist/openapi.json` / +`@agentcontextos/sdk`, not hand-typed. Add each new route to +`src/components/shell/nav.ts` and a `src/lib/api.ts` function with a seed fallback in +`src/lib/mock.ts`, mirroring the existing pattern. + +**Stays Preview:** API Keys (no `/v1/api-keys` backend endpoint exists yet). SCIM **writes** +(create/update users & groups) are IdP-driven — the console view is read-only. + +**Suggested slicing into sub-step PRs** (one complete, verified vertical each): `7.9a` Billing · +`7.9b` Query Playground + Ingest · `7.9c` Compliance + Tenants-live · `7.9d` Identity (SSO/SCIM) · +`7.9e` Chat/Embeddings/Agent · `7.9f` Audit verify/export + Dashboard tiles. diff --git a/apps/gateway/src/rag_gateway/app.py b/apps/gateway/src/rag_gateway/app.py index 3c51c7c..38dc6e4 100644 --- a/apps/gateway/src/rag_gateway/app.py +++ b/apps/gateway/src/rag_gateway/app.py @@ -420,6 +420,7 @@ def build_app( feedback_store: Any | None = None, drift_registry: Any | None = None, cost_tracker: Any | None = None, + usage_meter: Any | None = None, experiment_tracker: Any | None = None, shadow_runner: Any | None = None, ab_router: Any | None = None, @@ -760,6 +761,10 @@ def build_app( # ``GET /v1/status/cost``. ``None`` in the plain ``build_app`` (inert); # the config-driven wiring builds it from ``cfg.cost``. Observe-only. app.state.cost_tracker = cost_tracker + # Per-tenant usage meter (Step 7.9) — fed observe-only from the request paths; + # serves ``GET /v1/billing/usage``. ``None`` in plain ``build_app`` (inert); + # ``build_app_from_config`` builds it from ``cfg.billing``. + app.state.usage_meter = usage_meter # A/B experiment sample collector (Step 5.7) — per-(experiment, variant) # metric windows fed by the shadow / A/B-routing paths and analyzed on demand diff --git a/apps/gateway/src/rag_gateway/quota_guard.py b/apps/gateway/src/rag_gateway/quota_guard.py index e65b7f7..71b1180 100644 --- a/apps/gateway/src/rag_gateway/quota_guard.py +++ b/apps/gateway/src/rag_gateway/quota_guard.py @@ -21,6 +21,7 @@ from fastapi import Request from rag_core.errors import QuotaExceededError, RateLimitError from rag_core.types import RequestContext +from rag_observability.billing import BillingDimension def _quota_enabled(request: Request) -> bool: @@ -86,6 +87,12 @@ async def record_request_usage(request: Request, ctx: RequestContext, *, tokens: quotas** — a deployment can watch spend spikes without enabling enforcement. Both feeds are observe-only and skip non-positive token counts. """ + meter = getattr(request.app.state, "usage_meter", None) + if meter is not None: + meter.observe(str(ctx.tenant_id), BillingDimension.queries, 1) + if tokens > 0: + meter.observe(str(ctx.tenant_id), BillingDimension.llm_tokens, tokens) + if tokens > 0: tracker = getattr(request.app.state, "cost_tracker", None) if tracker is not None: diff --git a/apps/gateway/src/rag_gateway/status.py b/apps/gateway/src/rag_gateway/status.py index 97088cf..127cb22 100644 --- a/apps/gateway/src/rag_gateway/status.py +++ b/apps/gateway/src/rag_gateway/status.py @@ -291,6 +291,22 @@ class QuotaActionResponse(BaseModel): reset: bool +class BillingUsageResponse(BaseModel): + """One tenant's accrued metered usage for the current billing period (Step 7.9). + + Mirrors :class:`~rag_observability.billing.UsageSnapshot`. Per-tenant — the + tenant comes from the ``tenant_id`` query param or the ``X-Tenant-Id`` header. + ``usage`` maps each billing dimension (queries / docs_ingested / storage_gb / + reranker_calls / llm_tokens) to its accrued quantity; empty when no usage meter + is wired. Read-only — metering never mutates a request. + """ + + model_config = {"frozen": True} + + tenant_id: str + usage: dict[str, float] = Field(default_factory=dict) + + class CostStatusResponse(BaseModel): """One tenant's recent-spend anomaly verdict (Step 5.6c). @@ -848,6 +864,21 @@ async def status_cost(request: Request, tenant_id: str | None = None) -> CostSta snap = tracker.snapshot(tid) return CostStatusResponse(**dataclasses.asdict(snap)) + @router.get("/v1/billing/usage", response_model=BillingUsageResponse) + async def billing_usage(request: Request, tenant_id: str | None = None) -> BillingUsageResponse: + """One tenant's accrued metered usage for the current billing period (Step 7.9). + + Tenant resolves from the ``tenant_id`` query param, else the + ``X-Tenant-Id`` header, else ``"default"``. Empty when no usage meter is + wired (billing disabled). Read-only — metering never rejects a request. + """ + tid = tenant_id or request.headers.get("X-Tenant-Id") or "default" + meter = getattr(request.app.state, "usage_meter", None) + if meter is None: + return BillingUsageResponse(tenant_id=tid) + snap = meter.snapshot(tid) + return BillingUsageResponse(**dataclasses.asdict(snap)) + @router.get("/v1/status/experiments", response_model=ExperimentsStatusResponse) async def status_experiments(request: Request) -> ExperimentsStatusResponse: """A/B experiment comparisons — control vs candidate per experiment (Step 5.7). diff --git a/apps/gateway/src/rag_gateway/wiring.py b/apps/gateway/src/rag_gateway/wiring.py index 4025fc8..955dd22 100644 --- a/apps/gateway/src/rag_gateway/wiring.py +++ b/apps/gateway/src/rag_gateway/wiring.py @@ -65,6 +65,7 @@ ABExperimentTracker, CostTracker, TraceCollector, + UsageMeter, install_trace_collector, register_platform_metrics, ) @@ -602,6 +603,18 @@ def build_cost_tracker_from_config(cfg: RagConfig) -> CostTracker | None: ) +def build_usage_meter_from_config(cfg: RagConfig) -> UsageMeter | None: + """Build the per-tenant usage meter from ``cfg.billing`` (Step 7.9). + + Returns ``None`` when billing is disabled. Observe-only — the meter feeds + ``GET /v1/billing/usage`` and never affects a request. + """ + bc = cfg.billing + if not bc.enabled: + return None + return UsageMeter(max_tenants=bc.max_tenants) + + _PLATFORM_METER = "rag_gateway.platform" _platform_metrics_registered: set[str] = set() @@ -814,6 +827,11 @@ def build_app_from_config(cfg: RagConfig, **overrides: Any) -> FastAPI: if "cost_tracker" not in overrides: overrides["cost_tracker"] = build_cost_tracker_from_config(cfg) + # Per-tenant usage meter (Step 7.9) — observe-only metering from ``cfg.billing`` + # feeding ``GET /v1/billing/usage``. Disabled by config yields ``None``. + if "usage_meter" not in overrides: + overrides["usage_meter"] = build_usage_meter_from_config(cfg) + # A/B experiments (Step 5.7) — opt-in sample collector from ``cfg.experiments``. if "experiment_tracker" not in overrides: overrides["experiment_tracker"] = build_experiment_tracker_from_config(cfg) diff --git a/apps/gateway/tests/test_billing.py b/apps/gateway/tests/test_billing.py new file mode 100644 index 0000000..d200e00 --- /dev/null +++ b/apps/gateway/tests/test_billing.py @@ -0,0 +1,45 @@ +"""Tests for the per-tenant usage dashboard (Step 7.9). + +Drives the gateway with a wired usage meter: the inert default (empty), the +per-tenant usage surface, header-based tenant resolution, and isolation. +""" + +from __future__ import annotations + +from fastapi.testclient import TestClient +from rag_gateway import build_app +from rag_observability.billing import BillingDimension, UsageMeter + + +def _meter() -> UsageMeter: + m = UsageMeter() + m.observe("acme", BillingDimension.queries, 1234) + m.observe("acme", BillingDimension.llm_tokens, 50_000) + return m + + +def test_disabled_is_empty() -> None: + client = TestClient(build_app()) + body = client.get("/v1/billing/usage").json() + assert body["tenant_id"] == "default" + assert body["usage"] == {} + + +def test_usage_reflects_meter() -> None: + client = TestClient(build_app(usage_meter=_meter())) + body = client.get("/v1/billing/usage?tenant_id=acme").json() + assert body["usage"]["queries"] == 1234 + assert body["usage"]["llm_tokens"] == 50000 + + +def test_tenant_via_header() -> None: + client = TestClient(build_app(usage_meter=_meter())) + body = client.get("/v1/billing/usage", headers={"X-Tenant-Id": "acme"}).json() + assert body["tenant_id"] == "acme" + assert body["usage"]["queries"] == 1234 + + +def test_unknown_tenant_is_empty() -> None: + client = TestClient(build_app(usage_meter=_meter())) + body = client.get("/v1/billing/usage?tenant_id=globex").json() + assert body["usage"] == {} diff --git a/dist/openapi.json b/dist/openapi.json index 82f9b45..7f9d4c3 100644 --- a/dist/openapi.json +++ b/dist/openapi.json @@ -486,6 +486,27 @@ "title": "AuditVerifyResponse", "type": "object" }, + "BillingUsageResponse": { + "description": "One tenant's accrued metered usage for the current billing period (Step 7.9).\n\nMirrors :class:`~rag_observability.billing.UsageSnapshot`. Per-tenant — the\ntenant comes from the ``tenant_id`` query param or the ``X-Tenant-Id`` header.\n``usage`` maps each billing dimension (queries / docs_ingested / storage_gb /\nreranker_calls / llm_tokens) to its accrued quantity; empty when no usage meter\nis wired. Read-only — metering never mutates a request.", + "properties": { + "tenant_id": { + "title": "Tenant Id", + "type": "string" + }, + "usage": { + "additionalProperties": { + "type": "number" + }, + "title": "Usage", + "type": "object" + } + }, + "required": [ + "tenant_id" + ], + "title": "BillingUsageResponse", + "type": "object" + }, "BlobRef": { "description": "Reference to a blob stored in the ``Storage`` SPI rather than inline.\n\nUsed by chunks whose text exceeds the inline-storage threshold (see\nADR-0007 tiered storage). Callers must hydrate via ``Storage.get(uri)``\nonly when the text is actually needed.", "properties": { @@ -5659,6 +5680,56 @@ ] } }, + "/v1/billing/usage": { + "get": { + "description": "One tenant's accrued metered usage for the current billing period (Step 7.9).\n\nTenant resolves from the ``tenant_id`` query param, else the\n``X-Tenant-Id`` header, else ``\"default\"``. Empty when no usage meter is\nwired (billing disabled). Read-only — metering never rejects a request.", + "operationId": "billing_usage_v1_billing_usage_get", + "parameters": [ + { + "in": "query", + "name": "tenant_id", + "required": false, + "schema": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "title": "Tenant Id" + } + } + ], + "responses": { + "200": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/BillingUsageResponse" + } + } + }, + "description": "Successful Response" + }, + "422": { + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/HTTPValidationError" + } + } + }, + "description": "Validation Error" + } + }, + "summary": "Billing Usage", + "tags": [ + "status" + ] + } + }, "/v1/chat/completions": { "post": { "description": "RAG chat completion: retrieve → inject context → generate.\n\nSee ``docs/reference/openai-compat.md`` (``#post-v1chatcompletions``).", diff --git a/dist/openapi.yaml b/dist/openapi.yaml index 794db52..1f3695a 100644 --- a/dist/openapi.yaml +++ b/dist/openapi.yaml @@ -434,6 +434,35 @@ components: - ok title: AuditVerifyResponse type: object + BillingUsageResponse: + description: 'One tenant''s accrued metered usage for the current billing period + (Step 7.9). + + + Mirrors :class:`~rag_observability.billing.UsageSnapshot`. Per-tenant — the + + tenant comes from the ``tenant_id`` query param or the ``X-Tenant-Id`` header. + + ``usage`` maps each billing dimension (queries / docs_ingested / storage_gb + / + + reranker_calls / llm_tokens) to its accrued quantity; empty when no usage + meter + + is wired. Read-only — metering never mutates a request.' + properties: + tenant_id: + title: Tenant Id + type: string + usage: + additionalProperties: + type: number + title: Usage + type: object + required: + - tenant_id + title: BillingUsageResponse + type: object BlobRef: description: 'Reference to a blob stored in the ``Storage`` SPI rather than inline. @@ -4739,6 +4768,43 @@ paths: summary: Whole-log hash-chain integrity check (Step 6.6) tags: - audit + /v1/billing/usage: + get: + description: 'One tenant''s accrued metered usage for the current billing period + (Step 7.9). + + + Tenant resolves from the ``tenant_id`` query param, else the + + ``X-Tenant-Id`` header, else ``"default"``. Empty when no usage meter is + + wired (billing disabled). Read-only — metering never rejects a request.' + operationId: billing_usage_v1_billing_usage_get + parameters: + - in: query + name: tenant_id + required: false + schema: + anyOf: + - type: string + - type: 'null' + title: Tenant Id + responses: + '200': + content: + application/json: + schema: + $ref: '#/components/schemas/BillingUsageResponse' + description: Successful Response + '422': + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' + description: Validation Error + summary: Billing Usage + tags: + - status /v1/chat/completions: post: description: 'RAG chat completion: retrieve → inject context → generate. diff --git a/dist/rag.schema.json b/dist/rag.schema.json index 107c3c2..771f8ed 100644 --- a/dist/rag.schema.json +++ b/dist/rag.schema.json @@ -113,6 +113,30 @@ "title": "BackendsConfig", "type": "object" }, + "BillingConfig": { + "additionalProperties": false, + "description": "Per-tenant usage metering knobs (Step 7.9).\n\nWhen ``enabled`` the gateway feeds a per-tenant usage meter from the request\npaths (queries + LLM tokens) and serves ``GET /v1/billing/usage``.\n**Observe-only** \u2014 metering never rejects or mutates a request (the quota\nenforcer is the enforcement path). Invoicing is offline (``ragctl billing``),\npriced from ``marketplace/pricing.yaml`` so a price change is config, not code.\n\n* ``default_tier`` \u2014 the subscription tier an invoice is generated against.\n* ``max_tenants`` \u2014 cap on the per-tenant meter map (bounds memory).", + "properties": { + "enabled": { + "default": false, + "title": "Enabled", + "type": "boolean" + }, + "default_tier": { + "default": "pro", + "title": "Default Tier", + "type": "string" + }, + "max_tenants": { + "default": 10000, + "minimum": 1, + "title": "Max Tenants", + "type": "integer" + } + }, + "title": "BillingConfig", + "type": "object" + }, "BreakerConfig": { "additionalProperties": false, "description": "Per-backend circuit-breaker knobs (Step 4.4).\n\nWhen ``enabled`` the gateway wraps each retrieval backend (vector / keyword /\ngraph) in a :class:`~rag_breaker.breaker.CircuitBreaker` so a backend that\nstarts failing is isolated: calls *fail fast* instead of stalling behind its\ntimeout, and ``HybridRetriever`` fuses whatever sources remain healthy.\n\nOn by default \u2014 like the behaviour-neutral fallback chain (and unlike the\nguard): a CLOSED breaker is pure pass-through, and failing fast on a genuinely\ndown backend is strictly better than hanging. One shared config drives every\nbackend's breaker; operators watch state and force-close via the status API.\n\n* ``failure_threshold`` \u2014 consecutive failures that trip a breaker OPEN.\n* ``reset_timeout_s`` \u2014 seconds OPEN before a single trial call is admitted\n (HALF_OPEN).\n* ``half_open_max_calls`` \u2014 trial calls admitted while HALF_OPEN (the rest\n fail fast, bounding load on a recovering backend).\n* ``success_threshold`` \u2014 trial successes needed to close the breaker.", @@ -2132,6 +2156,9 @@ "cost": { "$ref": "#/$defs/CostConfig" }, + "billing": { + "$ref": "#/$defs/BillingConfig" + }, "experiments": { "$ref": "#/$defs/ExperimentsConfig" } diff --git a/dist/rag.schema.yaml b/dist/rag.schema.yaml index 60ee222..60955f3 100644 --- a/dist/rag.schema.yaml +++ b/dist/rag.schema.yaml @@ -153,6 +153,41 @@ $defs: $ref: '#/$defs/TelemetryConfig' title: BackendsConfig type: object + BillingConfig: + additionalProperties: false + description: 'Per-tenant usage metering knobs (Step 7.9). + + + When ``enabled`` the gateway feeds a per-tenant usage meter from the request + + paths (queries + LLM tokens) and serves ``GET /v1/billing/usage``. + + **Observe-only** — metering never rejects or mutates a request (the quota + + enforcer is the enforcement path). Invoicing is offline (``ragctl billing``), + + priced from ``marketplace/pricing.yaml`` so a price change is config, not code. + + + * ``default_tier`` — the subscription tier an invoice is generated against. + + * ``max_tenants`` — cap on the per-tenant meter map (bounds memory).' + properties: + enabled: + default: false + title: Enabled + type: boolean + default_tier: + default: pro + title: Default Tier + type: string + max_tenants: + default: 10000 + minimum: 1 + title: Max Tenants + type: integer + title: BillingConfig + type: object BreakerConfig: additionalProperties: false description: "Per-backend circuit-breaker knobs (Step 4.4).\n\nWhen ``enabled``\ @@ -2062,6 +2097,8 @@ properties: $ref: '#/$defs/DriftConfig' cost: $ref: '#/$defs/CostConfig' + billing: + $ref: '#/$defs/BillingConfig' experiments: $ref: '#/$defs/ExperimentsConfig' title: RagConfig diff --git a/docs/README.md b/docs/README.md index eb6aa0c..060f900 100644 --- a/docs/README.md +++ b/docs/README.md @@ -83,6 +83,7 @@ | [drift.md](reference/drift.md) | Drift monitors (Step 5.5) — `rag_drift.population_stability_index` / `DriftMonitor` / `DriftMonitorRegistry`; the `DriftSnapshot` / `DriftReport` / `DriftMetric` / `DriftMethod` / `DriftStatus` core types; the five-monitor table (metric → method → signal → drift condition); `GET /v1/status/drift` + `POST /v1/status/drift/{metric}/rebaseline`; `cfg.drift`; the `drift.detected` event + webhook; `ragctl drift`; PSI rule-of-thumb; extension points | | [feedback.md](reference/feedback.md) | Online feedback (Step 5.4) — `rag_feedback.FeedbackRecorder` / `aggregate_feedback` / `score_for_signal` / `kind_for_signal`; the `FeedbackRecord` / `FeedbackStats` / `FeedbackKind` / `FeedbackSignal` core types + `FeedbackStore` SPI (`NoopFeedbackStore`); `FeedbackRequest` / `FeedbackAck` wire types; the signal→score/kind table; `POST /v1/feedback` (body identity, PII-redacted comments, degrade-open ack) + `GET /v1/status/feedback` dashboard; `cfg.feedback`; `feedback.recorded` event; `ragctl feedback`; extension points | | [cost.md](reference/cost.md) | Cost anomaly (Step 5.6c) — `rag_observability.CostTracker` (`observe` / `snapshot`) + the `CostSnapshot` dataclass (`status` ok/elevated/insufficient_data, `ratio`, `z_score`, token + micro-dollar means); the two-gate scale-free detection rule; `GET /v1/status/cost` (`CostStatusResponse`); `cfg.cost`; fed from `record_request_usage` independent of quotas; the admin **Cost anomaly** card; extension points | +| [billing.md](reference/billing.md) | Billing & metering (Step 7.9) — `rag_observability.billing`: `UsageMeter` (observe-only per-tenant, per-dimension counters) + `BillingDimension` + `UsageSnapshot`; pure `generate_invoice` + `PricingModel` / `load_pricing` (priced from `marketplace/pricing.yaml`, reconciles ±0.5% by construction); the `BillingProvider` Stripe/marketplace seam; `GET /v1/billing/usage` (`BillingUsageResponse`, no `dist/schemas` churn — mirrors 5.6c); `cfg.billing`; `ragctl billing`; extension points | | [experiments.md](reference/experiments.md) | A/B testing & shadow mode (Step 5.7) — the pure `rag_config.eval.analyze_ab_experiment` (lift + normal-approx Welch CI + significance, stdlib `statistics.NormalDist`) → `ABAnalysisResult` (`rag_core.eval`); `rag_observability.ABExperimentTracker` (bounded per-`(experiment, variant)` sample windows, decoupled from the analyzer); `GET /v1/status/experiments` (gateway composes the two); `cfg.experiments` (opt-in); `ragctl experiments`; the 5.7a–d slice plan + extension points | | [eval-harness.md](reference/eval-harness.md) | Offline eval harness + CI gate (Steps 5.2–5.3) — the five metrics (`recall_at_k` / `mrr` / `ndcg_at_k` / `lexical_faithfulness` / `citation_precision`) + `render_html_report` in `rag_config.eval`; `EvalReport` / `EvalMetrics` / `GroupEvalMetrics` (`by_domain` / `by_difficulty`); the `eval/golden_set_v0` harness + `generate.py` (500-query / 5-domain committed golden set, drift-gated); `python -m eval.golden_set_v0.harness` (`--domain` / `--queries` / `--k` / `--check`); `ragctl eval run --html` / `eval show`; **the gate** (`python -m eval.golden_set_v0.gate`, committed `EvalBaseline`, floor + regression-delta, sticky PR comment); **the regression bisector** (Step 5.6d — `python -m eval.golden_set_v0.bisect`, binary-search `good..bad` in throwaway git worktrees to find the first commit that dropped a metric; pure `bisect_commits` → `BisectResult`); thresholds; extension points | | [rag-observability.md](reference/rag-observability.md) | `rag-observability` — structured logger, event registry, `AsyncTelemetrySink` (bounded, drop-on-overflow) | @@ -204,6 +205,7 @@ broken, and what to fix before committing to the next phase. | [ADR-0047-documentation-site.md](adr/ADR-0047-documentation-site.md) | Decision (Step 7.5): publish docs via **Docusaurus** sourcing the repo `docs/` tree **in place** (single source of truth, no copy); `markdown.format: 'detect'` so hand-written `.md` renders as CommonMark; the **REST API reference is generated + drift-gated** from `dist/openapi.json` (a committed file, not a build-time plugin); doc honesty is a `tests/docs/` suite (every `ragctl` / `/v1/` reference is real) + `lychee` + `codespell`; rejected MkDocs, copying `docs/`, an OpenAPI plugin | | [ADR-0049-packaging-distribution.md](adr/ADR-0049-packaging-distribution.md) | Decision (Step 7.7): a `rag-platform` PyPI **meta-package** pins the workspace component dists (one-command server install); keep the existing `agentcontextos` SDK scope (no rename); **one `vX.Y.Z` tag fans out** to PyPI/npm/GHCR/Helm-OCI/air-gap across `docker.yml` + a new `release.yml` + `release-airgap.yml`; cosign-keyless + SPDX SBOM + OIDC/provenance everywhere; deferred Go/Java/.NET publish + worker/eval-runner images; rejected a `ragplatform` rename + one mega-workflow | | [ADR-0050-support-sla-oncall.md](adr/ADR-0050-support-sla-oncall.md) | Decision (Step 7.8): **runbooks grounded in the platform's own signals** — one per alert type, each naming the exact event / `/v1/status/*` / SLO that raises it; **paging reuses the Step 3.9 webhook system** (PagerDuty as a subscriber, no new code) + Grafana SLOs; the status page reflects **measured** health (not manual toggles); SLA tiers match `marketplace/pricing.yaml`; rejected a new alerting subsystem, generic runbooks, a separate SLA datastore | +| [ADR-0051-billing-metering.md](adr/ADR-0051-billing-metering.md) | Decision (Step 7.9): **mirror the 5.6c cost pattern** — dataclasses in `rag-observability`, gateway wraps in a Pydantic response (**no `dist/schemas` churn**); metering is **observe-only** (no governed SPI call, fed from `record_request_usage` independent of quotas); invoicing is offline + pure + **priced from config** (`marketplace/pricing.yaml`, reconciles ±0.5% by construction); Stripe/marketplace via a `BillingProvider` Protocol seam; rejected a new `rag-billing` package + core types, metering-through-quotas, hard-coded prices | | [ADR-0048-marketplace-listings.md](adr/ADR-0048-marketplace-listings.md) | Decision (Step 7.6): one canonical `marketplace/pricing.yaml` (metered dims = the Step 7.9 signals) + one shared listing copy mapped to all three clouds, so listings can't drift and prices reconcile with metering; **reuse existing delivery** (Helm / AMI / airgap / GHCR) — the per-cloud Marketplace wrappers are thin shells; approval + procurement are documented external process, not a faked status; rejected per-cloud pricing, custom packaging, encoding a "live" status | | [ADR-0042-compliance-posture.md](adr/ADR-0042-compliance-posture.md) | Decision (Step 6.10, Phase-6 capstone): add the three compliance pieces on top of the controls the platform already ships (audit/ACL/PII/BYOK/SSO/quotas). New `rag-compliance` package (config-free, like rag-feedback/rag-drift): `RetentionEnforcer` drives tenant-scoped `purge_*`; `compliance_posture`/`residency_ok` are pure. **Retention is a capability on the existing stores, not a new SPI** — non-abstract `purge_before`/`purge_tenant` (default no-op) on Feedback/Provenance, with `dry_run` in the SPI so a preview counts-without-deleting uniformly (ProvenanceStore has no `list`). **Audit is never purged in place** (the hash chain would break) — audit retention is the 6.6b WORM export; `audit_days` is advisory. **Right-to-erasure is always-on, tenant-self-service, two-flag** — `POST /v1/compliance/erase` erases the *calling* tenant's data (scope from the principal, never the body), dry-run by default, delete needs `dry_run=false` AND `confirm=true`. **Residency = declared per tenant + enforced at ingest** (`tenants[].data_region` vs `cfg.compliance.region` → 403), opt-in, a single-deployment assertion not multi-region routing. **The SOC 2 / GDPR mapping is a doc backed by a live posture** (`GET /v1/status/compliance` reports which controls are on, so the mapping is checkable). Deferred: subject-level (vs tenant-level) erasure, an admin retention-sweep endpoint, multi-region routing, automated audit-evidence bundles; rejected purging the audit chain, a `Purgeable` SPI mixin, a static doc with no live backing | | [ADR-0041-airgap-bundle.md](adr/ADR-0041-airgap-bundle.md) | Decision (Step 6.9): ship the platform as one signed, self-contained offline bundle (all runtime images + Helm chart + config + installer). Integrity reuses the WORM-export pattern (6.6b): a standard `SHA256SUMS` whose hash is pinned as `manifest.content_hash` is the **hard gate** — verifiable with nothing but `sha256sum`, no network/cosign — and a cosign signature **over `SHA256SUMS`** adds authenticity; the *same* `SHA256SUMS` drives the Python verifier and the standalone shell installer so they can't diverge. The shell/pwsh `install.{sh,ps1}` (shipped inside the bundle) need only docker+helm (air-gap hosts lack uv/the workspace); `ragctl airgap` holds the typed/tested build+verify logic (pure core separated from a stubbable docker/helm/cosign subprocess seam; `--dry-run` = a verifiable bundle minus image blobs, so the path is testable with no Docker). Digest-pinned manifest-driven image set (`infra/airgap/images.txt` + the chart-derived gateway image); key-based cosign is the air-gap recommendation (keyless needs Rekor + an identity policy), keyless is the connected-release path (`release-airgap.yml` on tags). Deferred: ctr/podman load, registry re-tag/push, multi-arch selection, bundling backend charts, TUF-rooted offline keyless verify; rejected `oras`/OCI (no registry to pull from in an air-gap), a second HMAC scheme (cosign already the signer), a pure-shell build (would escape mypy/tests) | diff --git a/docs/adr/ADR-0051-billing-metering.md b/docs/adr/ADR-0051-billing-metering.md new file mode 100644 index 0000000..056db57 --- /dev/null +++ b/docs/adr/ADR-0051-billing-metering.md @@ -0,0 +1,67 @@ +# ADR-0051 — Billing & metering: dataclasses in observability, observe-only, invoice from config + +**Status:** Accepted +**Date:** 2026-06-10 +**Step:** 7.9 — Billing & metering (Phase 7 — Pilot, Harden, GA) +**Related:** [reference/billing.md](../reference/billing.md), [5.6c cost anomaly](ADR-0031-cost-anomaly.md), [4.5 quotas](ADR-0024-quotas-rate-limiting.md), [7.6 marketplace pricing](ADR-0048-marketplace-listings.md), [planning/phases/phase-7-pilot-ga.md](../../planning/phases/phase-7-pilot-ga.md) + +## Context + +Step 7.9 wants per-tenant metering (queries / docs / storage / reranker / LLM +tokens), a usage-export API, Stripe + marketplace billing, invoicing, and a tenant +usage dashboard — with the acceptance "bills reconcile with metered usage within +±0.5%" and "pricing changes deployable via config." The platform already meters +tokens for cost-anomaly (5.6c) and enforces caps via quotas (4.5); billing is a +*third*, distinct concern: usage accounting for money. + +## Decision + +**1. Mirror the 5.6c cost pattern — dataclasses in `rag-observability`.** The +`UsageMeter` + `UsageSnapshot` + pricing/invoice types are plain dataclasses in +`rag_observability.billing`; the gateway wraps the snapshot in a gateway-local +Pydantic `BillingUsageResponse`. So there is **no `rag-core` type / `dist/schemas` +churn** — only `dist/openapi.*` gains the `GET /v1/billing/usage` endpoint. + +**2. Metering is observe-only; enforcement stays the quota enforcer.** The meter is +fed O(1) from `record_request_usage` (queries + tokens), independently of quotas +(a deployment can meter without enforcing). It **makes no governed SPI call**, so +the PolicyEngine coverage linter needs no new entry — same posture as cost / drift +/ feedback. Inert by default (`cfg.billing.enabled = false`). + +**3. Invoicing is offline and priced from config.** `generate_invoice` is a **pure** +function (usage + `PricingModel` → `Invoice`); the pricing is data loadable from +`marketplace/pricing.yaml`, so a price change is config, not code (the 7.9 "pricing +deployable via config" bar, shared with the 7.6 listings). Invoicing isn't a +hot-path gateway concern, so it lives in `ragctl billing`. Reconciliation holds **by +construction** — each metered line item's quantity equals the metered overage, so +the bill matches usage exactly (well inside ±0.5%). + +**4. External billers are a Protocol seam.** `BillingProvider.report_usage` (a +Stripe usage-records adapter, or the AWS/Azure/GCP marketplace metering APIs) is a +package-local Protocol with a `NoopBillingProvider` default, called degrade-open — +no `rag_core.spi` contract-suite ceremony for an external-integration seam (same +reasoning as the injection detector being package-local). + +## Consequences + +- New `rag_observability.billing` (meter + pricing + pure invoice + provider seam), + the `GET /v1/billing/usage` endpoint + `cfg.billing` + `build_usage_meter_from_config`, + a `ragctl billing` demo, and tests (unit reconciliation + gateway endpoint + CLI). +- Only `dist/openapi.*` + `dist/rag.schema.*` (the new `cfg.billing` block) + regenerate; `dist/schemas/` (rag_core) is untouched. +- **Deferred:** the concrete Stripe + marketplace `BillingProvider` adapters + (behind the seam); wiring `docs_ingested` / `reranker_calls` / `storage_gb` feed + points (the meter supports them; only queries + tokens are fed today); an admin + usage dashboard card; subject-level (vs tenant-level) invoicing. + +## Alternatives considered + +- **A new `rag-billing` package + `rag-core` types.** Rejected — mirroring 5.6c + (dataclasses in observability) avoids `dist/schemas` churn + a contract suite for + what is read-side accounting, not a governed SPI. +- **Meter through the quota enforcer.** Rejected — quotas *enforce* (and can reject); + billing *meters* (never rejects) and must run even with quotas off. They're fed + independently from the same `record_request_usage` choke point, like cost. +- **Bill from hard-coded prices.** Rejected — "pricing deployable via config" means + the invoice generator takes a `PricingModel` loaded from `pricing.yaml`, the same + source the marketplace listings map to. diff --git a/docs/reference/billing.md b/docs/reference/billing.md new file mode 100644 index 0000000..a9886ff --- /dev/null +++ b/docs/reference/billing.md @@ -0,0 +1,82 @@ +# Billing & metering (Step 7.9) + +Per-tenant usage metering, the usage-export API, and invoice generation — +`rag_observability.billing`. Mirrors the Step 5.6c cost-anomaly design: plain +dataclasses in `rag-observability` (the gateway wraps them in a Pydantic +response), so there's **no `rag-core` type / `dist/schemas` churn**. + +## Overview + +| Piece | What | +|-------|------| +| `UsageMeter` | Thread-safe per-tenant, per-dimension counters; fed O(1) from the request paths | +| `BillingDimension` | `queries` · `docs_ingested` · `storage_gb` · `reranker_calls` · `llm_tokens` | +| `UsageSnapshot` | One tenant's accrued usage (dimension → quantity) | +| `PricingModel` / `load_pricing` | Tiers + metered rates, loadable from `marketplace/pricing.yaml` | +| `generate_invoice` | **Pure** usage + pricing → `Invoice`; line-item quantities equal metered overage exactly | +| `BillingProvider` | Seam for reporting usage to Stripe / a marketplace metering API (degrade-open) | + +## Usage + +### Metering (gateway) + +`UsageMeter` is **observe-only** — it never rejects or mutates a request. It's fed +from `record_request_usage` (queries + LLM tokens) and serves the usage API: + +```bash +curl 'localhost:8000/v1/billing/usage?tenant_id=acme' +# {"tenant_id": "acme", "usage": {"queries": 1234, "llm_tokens": 50000}} +``` + +Empty (`{}`) when `cfg.billing.enabled` is false (the default). Enable it: + +```yaml +# rag.yaml +billing: + enabled: true + default_tier: pro + max_tenants: 10000 +``` + +### Invoicing (offline) + +Invoicing is offline (not a hot-path concern) and **priced from config**, so a +price change is a `pricing.yaml` edit, not code: + +```bash +ragctl billing --tenant acme --tier pro +# meters synthetic usage → invoice line items + total + a reconciliation check +``` + +`generate_invoice(snapshot, pricing, tier_id)` is pure: a base-fee line plus one +line per dimension whose usage exceeds the tier's inclusion, billed at the metered +rate. Because each metered line item's quantity **equals the metered overage**, the +bill reconciles with usage exactly (well within the ±0.5% acceptance bar). + +## Internals + +- **No `dist/schemas` churn** — the gateway's `BillingUsageResponse` is a + gateway-local Pydantic model wrapping the dataclass via `dataclasses.asdict` + (same pattern as `CostStatusResponse`). Only `dist/openapi.*` gains the endpoint. +- **No governed SPI call** — metering is observe-only (like cost / drift / feedback), + so the PolicyEngine coverage linter needs no new entry. Enforcement is the quota + enforcer (Step 4.5); billing only *meters*. +- **Fed independently** of quotas — a deployment can meter usage without enabling + quota enforcement. + +## Extension points + +- **Report to an external biller.** Implement `BillingProvider.report_usage` (a + Stripe usage-records adapter, or the AWS/Azure/GCP marketplace metering APIs) and + call it degrade-open from the gateway. The default `NoopBillingProvider` records + nothing (the usage API still serves). +- **More feed points.** `docs_ingested` (ingest path), `reranker_calls` (rerank + stage), and `storage_gb` (a gauge) are supported by the meter; wire additional + `observe` / `set_gauge` calls at those sites. + +## See also + +- [marketplace/pricing.yaml](../../marketplace/pricing.yaml) — the canonical pricing the dimensions map to +- [reference/cost.md](cost.md) — the cost-anomaly sibling (5.6c) this mirrors +- [reference/quotas.md](quota.md) — the enforcement path (4.5) +- [ADR-0051](../adr/ADR-0051-billing-metering.md) diff --git a/docs/reference/rest-api.md b/docs/reference/rest-api.md index ff39455..1024546 100644 --- a/docs/reference/rest-api.md +++ b/docs/reference/rest-api.md @@ -534,6 +534,27 @@ Delete User ### status +#### `GET /v1/billing/usage` + +Billing Usage + +One tenant's accrued metered usage for the current billing period (Step 7.9). + +Tenant resolves from the ``tenant_id`` query param, else the +``X-Tenant-Id`` header, else ``"default"``. Empty when no usage meter is +wired (billing disabled). Read-only — metering never rejects a request. + +| Name | In | Required | Type | Description | +|------|----|:--------:|------|-------------| +| `tenant_id` | query | no | — | | + +**Responses** + +| Status | Schema | Description | +|--------|--------|-------------| +| 200 | `BillingUsageResponse` | Successful Response | +| 422 | `HTTPValidationError` | Validation Error | + #### `GET /v1/connectors/status` Connectors Status diff --git a/packages/config/src/rag_config/schema.py b/packages/config/src/rag_config/schema.py index d37483a..f56c6e6 100644 --- a/packages/config/src/rag_config/schema.py +++ b/packages/config/src/rag_config/schema.py @@ -1089,6 +1089,24 @@ class CostConfig(_StrictBase): dollars_per_1k_tokens: Annotated[float, Field(ge=0.0)] = 0.0 +class BillingConfig(_StrictBase): + """Per-tenant usage metering knobs (Step 7.9). + + When ``enabled`` the gateway feeds a per-tenant usage meter from the request + paths (queries + LLM tokens) and serves ``GET /v1/billing/usage``. + **Observe-only** — metering never rejects or mutates a request (the quota + enforcer is the enforcement path). Invoicing is offline (``ragctl billing``), + priced from ``marketplace/pricing.yaml`` so a price change is config, not code. + + * ``default_tier`` — the subscription tier an invoice is generated against. + * ``max_tenants`` — cap on the per-tenant meter map (bounds memory). + """ + + enabled: bool = False + default_tier: str = "pro" + max_tenants: Annotated[int, Field(ge=1)] = 10_000 + + class CandidateConfig(_StrictBase): """Fusion weights for a candidate retriever (Steps 5.7b / 5.7c). @@ -1188,6 +1206,7 @@ class RagConfig(_StrictBase): feedback: FeedbackConfig = Field(default_factory=FeedbackConfig) drift: DriftConfig = Field(default_factory=DriftConfig) cost: CostConfig = Field(default_factory=CostConfig) + billing: BillingConfig = Field(default_factory=BillingConfig) experiments: ExperimentsConfig = Field(default_factory=ExperimentsConfig) @field_validator("tenants") diff --git a/packages/observability/src/rag_observability/__init__.py b/packages/observability/src/rag_observability/__init__.py index 0082cca..718c524 100644 --- a/packages/observability/src/rag_observability/__init__.py +++ b/packages/observability/src/rag_observability/__init__.py @@ -1,6 +1,20 @@ """AgentContextOS observability — logging, events, context, async telemetry sink.""" from rag_observability.async_sink import AsyncTelemetrySink, SinkStats +from rag_observability.billing import ( + BillingDimension, + BillingProvider, + Invoice, + InvoiceLineItem, + MeteredRate, + NoopBillingProvider, + PricingModel, + PricingTier, + UsageMeter, + UsageSnapshot, + generate_invoice, + load_pricing, +) from rag_observability.cost import CostSnapshot, CostTracker from rag_observability.experiments import ABExperimentTracker from rag_observability.logbuffer import ( @@ -26,22 +40,34 @@ __all__ = [ "ABExperimentTracker", "AsyncTelemetrySink", + "BillingDimension", + "BillingProvider", "BufferedLogRecord", "CostSnapshot", "CostTracker", "CounterSample", "GaugeSample", "HistogramSample", + "Invoice", + "InvoiceLineItem", "LogTail", + "MeteredRate", "MetricsCollector", "MetricsSnapshot", + "NoopBillingProvider", + "PricingModel", + "PricingTier", "RingBufferLogHandler", "SinkStats", "TraceCollector", + "UsageMeter", + "UsageSnapshot", "attach_log_tail", + "generate_invoice", "get_log_tail", "get_metrics_collector", "install_trace_collector", + "load_pricing", "register_platform_metrics", "reset_log_tail", "reset_metrics_collector", diff --git a/packages/observability/src/rag_observability/billing.py b/packages/observability/src/rag_observability/billing.py new file mode 100644 index 0000000..903af5c --- /dev/null +++ b/packages/observability/src/rag_observability/billing.py @@ -0,0 +1,244 @@ +"""Per-tenant usage metering + invoice generation (Step 7.9). + +Mirrors the Step 5.6c cost-anomaly design: plain dataclasses in +``rag-observability`` (the gateway wraps them in a Pydantic response), so there is +**no ``rag-core`` type / ``dist/schemas`` churn**. Metering is **observe-only** — +it never rejects or mutates a request (the quota enforcer is the enforcement path). + +* ``UsageMeter`` — thread-safe per-tenant, per-dimension counters fed O(1) from the + request paths; ``snapshot`` powers ``GET /v1/billing/usage``. +* ``generate_invoice`` — a **pure** function mapping a usage snapshot + a + ``PricingModel`` to an ``Invoice``; line-item quantities equal the metered + quantities by construction, so bills reconcile with usage exactly. +* ``PricingModel`` is data (loadable from ``marketplace/pricing.yaml``), so a + pricing change is config, not code. +""" + +from __future__ import annotations + +import threading +from collections import defaultdict +from dataclasses import dataclass, field +from enum import StrEnum +from typing import Protocol + + +class BillingDimension(StrEnum): + """The metered usage dimensions (ids match ``marketplace/pricing.yaml``).""" + + queries = "queries" + docs_ingested = "docs_ingested" + storage_gb = "storage_gb" + reranker_calls = "reranker_calls" + llm_tokens = "llm_tokens" + + +@dataclass(frozen=True) +class UsageSnapshot: + """One tenant's accrued usage for the current billing period.""" + + tenant_id: str + usage: dict[str, float] = field(default_factory=dict) + + def quantity(self, dimension: BillingDimension) -> float: + return self.usage.get(dimension.value, 0.0) + + +class UsageMeter: + """Bounded per-tenant, per-dimension usage counters (thread-safe). + + ``observe`` accumulates a counter dimension (queries / docs / tokens / reranker + calls); ``set_gauge`` records a level dimension (storage). ``max_tenants`` caps + the map so an unbounded tenant space can't grow memory without bound. + """ + + def __init__(self, *, max_tenants: int = 10_000) -> None: + self._max_tenants = max(1, max_tenants) + self._counters: dict[str, dict[str, float]] = defaultdict(dict) + self._lock = threading.Lock() + + def observe(self, tenant_id: str, dimension: BillingDimension, quantity: float = 1.0) -> None: + """Accumulate ``quantity`` for a counter dimension (non-positive ignored).""" + if quantity <= 0: + return + with self._lock: + if tenant_id not in self._counters and len(self._counters) >= self._max_tenants: + return + row = self._counters[tenant_id] + row[dimension.value] = row.get(dimension.value, 0.0) + float(quantity) + + def set_gauge(self, tenant_id: str, dimension: BillingDimension, level: float) -> None: + """Record the current level of a gauge dimension (e.g. storage GB).""" + if level < 0: + return + with self._lock: + if tenant_id not in self._counters and len(self._counters) >= self._max_tenants: + return + self._counters[tenant_id][dimension.value] = float(level) + + def snapshot(self, tenant_id: str) -> UsageSnapshot: + with self._lock: + usage = dict(self._counters.get(tenant_id, {})) + return UsageSnapshot(tenant_id=tenant_id, usage=usage) + + def snapshot_all(self) -> list[UsageSnapshot]: + with self._lock: + tenants = list(self._counters) + return [self.snapshot(t) for t in tenants] + + def reset(self, tenant_id: str) -> None: + """Clear a tenant's counters (call after closing a billing period).""" + with self._lock: + self._counters.pop(tenant_id, None) + + +@dataclass(frozen=True) +class MeteredRate: + """A usage-metered price: ``price`` per ``unit_size`` of a dimension.""" + + dimension: BillingDimension + unit: str + unit_size: float + price: float + + +@dataclass(frozen=True) +class PricingTier: + """A subscription tier: a monthly base fee + per-dimension inclusions.""" + + tier_id: str + monthly_base: float + inclusions: dict[str, float] = field(default_factory=dict) + + +@dataclass(frozen=True) +class PricingModel: + """Tiers + metered rates — the data behind every invoice (config, not code).""" + + currency: str + tiers: dict[str, PricingTier] + rates: dict[str, MeteredRate] + + +def load_pricing(data: dict[str, object]) -> PricingModel: + """Parse a ``marketplace/pricing.yaml``-shaped dict into a ``PricingModel``.""" + tiers: dict[str, PricingTier] = {} + raw_tiers = data.get("tiers") + if isinstance(raw_tiers, list): + for t in raw_tiers: + if not isinstance(t, dict): + continue + base = t.get("monthly_base", 0) + included = dict(t.get("included") or {}) + tiers[str(t["id"])] = PricingTier( + tier_id=str(t["id"]), + monthly_base=float(base) if isinstance(base, (int, float)) else 0.0, + inclusions={str(k): float(v) for k, v in included.items()}, + ) + rates: dict[str, MeteredRate] = {} + raw_dims = data.get("metered_dimensions") + if isinstance(raw_dims, list): + for d in raw_dims: + if not isinstance(d, dict): + continue + dim = BillingDimension(str(d["id"])) + unit = str(d.get("unit", "")) + rates[dim.value] = MeteredRate( + dimension=dim, + unit=unit, + unit_size=_unit_size(unit), + price=float(d.get("price", 0.0)), + ) + return PricingModel(currency=str(data.get("currency", "USD")), tiers=tiers, rates=rates) + + +def _unit_size(unit: str) -> float: + """Map a pricing unit string ('1k queries', '1M tokens', 'GB-month') to a size.""" + u = unit.lower() + if u.startswith("1m") or "million" in u: + return 1_000_000.0 + if u.startswith("1k") or "thousand" in u: + return 1_000.0 + return 1.0 + + +@dataclass(frozen=True) +class InvoiceLineItem: + description: str + dimension: str | None + quantity: float + unit_price: float + amount: float + + +@dataclass(frozen=True) +class Invoice: + tenant_id: str + tier_id: str + currency: str + line_items: list[InvoiceLineItem] + total: float + + def metered_quantity(self, dimension: BillingDimension) -> float: + """Total billed quantity for a dimension across line items (for reconciliation).""" + return sum(li.quantity for li in self.line_items if li.dimension == dimension.value) + + +def generate_invoice(snapshot: UsageSnapshot, pricing: PricingModel, tier_id: str) -> Invoice: + """Map usage + pricing → an invoice. Pure; metered line-item quantities equal the + metered overage exactly, so the bill reconciles with usage by construction.""" + tier = pricing.tiers.get(tier_id) or PricingTier(tier_id=tier_id, monthly_base=0.0) + items: list[InvoiceLineItem] = [ + InvoiceLineItem( + description=f"{tier_id} plan (monthly base)", + dimension=None, + quantity=1.0, + unit_price=tier.monthly_base, + amount=round(tier.monthly_base, 2), + ) + ] + for dim_value, used in sorted(snapshot.usage.items()): + rate = pricing.rates.get(dim_value) + if rate is None: + continue + included = tier.inclusions.get(dim_value, 0.0) + overage = max(0.0, used - included) + if overage <= 0: + continue + units = overage / rate.unit_size + items.append( + InvoiceLineItem( + description=f"{dim_value} overage ({rate.unit})", + dimension=dim_value, + quantity=overage, + unit_price=rate.price, + amount=round(units * rate.price, 2), + ) + ) + total = round(sum(li.amount for li in items), 2) + return Invoice( + tenant_id=snapshot.tenant_id, + tier_id=tier_id, + currency=pricing.currency, + line_items=items, + total=total, + ) + + +class BillingProvider(Protocol): + """Seam for reporting metered usage to an external biller (Stripe / marketplace). + + Production impls (e.g. a Stripe usage-records adapter, or the AWS/Azure/GCP + marketplace metering APIs) live behind this Protocol and are called + degrade-open from the gateway; the default is the no-op below. + """ + + def report_usage(self, snapshot: UsageSnapshot) -> None: ... + + +class NoopBillingProvider: + """Default provider — records nothing (metering still serves the usage API).""" + + def report_usage(self, snapshot: UsageSnapshot) -> None: + """Discard the snapshot — no external biller is wired.""" + return None diff --git a/packages/observability/tests/test_billing.py b/packages/observability/tests/test_billing.py new file mode 100644 index 0000000..a0d0cf1 --- /dev/null +++ b/packages/observability/tests/test_billing.py @@ -0,0 +1,82 @@ +"""Tests for per-tenant metering + invoice generation (Step 7.9).""" + +from __future__ import annotations + +from rag_observability.billing import ( + BillingDimension, + UsageMeter, + generate_invoice, + load_pricing, +) + +PRICING = { + "currency": "USD", + "tiers": [ + {"id": "pro", "monthly_base": 499, "included": {"queries": 1_000_000, "storage_gb": 50}}, + {"id": "business", "monthly_base": 2500, "included": {"queries": 10_000_000}}, + ], + "metered_dimensions": [ + {"id": "queries", "unit": "1k queries", "price": 0.50}, + {"id": "storage_gb", "unit": "GB-month", "price": 0.25}, + {"id": "llm_tokens", "unit": "1M tokens", "price": 5.0}, + ], +} + + +def test_meter_observe_and_snapshot() -> None: + m = UsageMeter() + for _ in range(3): + m.observe("acme", BillingDimension.queries) + m.observe("acme", BillingDimension.llm_tokens, 1500) + m.observe("acme", BillingDimension.queries, 0) # non-positive ignored + m.set_gauge("acme", BillingDimension.storage_gb, 75.0) + snap = m.snapshot("acme") + assert snap.quantity(BillingDimension.queries) == 3 + assert snap.quantity(BillingDimension.llm_tokens) == 1500 + assert snap.quantity(BillingDimension.storage_gb) == 75.0 + + +def test_meter_max_tenants_cap() -> None: + m = UsageMeter(max_tenants=2) + for t in ("a", "b", "c"): + m.observe(t, BillingDimension.queries) + assert len(m.snapshot_all()) == 2 # third tenant dropped + + +def test_invoice_base_only_when_within_inclusions() -> None: + pricing = load_pricing(PRICING) + m = UsageMeter() + m.observe("acme", BillingDimension.queries, 500_000) # under Pro's 1M inclusion + inv = generate_invoice(m.snapshot("acme"), pricing, "pro") + assert inv.total == 499.0 + assert len(inv.line_items) == 1 # base fee only + + +def test_invoice_overage_and_reconciliation() -> None: + pricing = load_pricing(PRICING) + m = UsageMeter() + m.observe("acme", BillingDimension.queries, 1_400_000) # 400k over Pro's 1M + m.observe("acme", BillingDimension.llm_tokens, 3_000_000) # 3M tokens, no inclusion + m.set_gauge("acme", BillingDimension.storage_gb, 80.0) # 30 over Pro's 50 + snap = m.snapshot("acme") + inv = generate_invoice(snap, pricing, "pro") + + # base 499 + queries 400k/1k*0.50=200 + storage 30*0.25=7.5 + tokens 3M/1M*5=15 + assert inv.total == round(499 + 200 + 7.5 + 15, 2) + + # bills reconcile with metered usage within ±0.5% (exact, by construction) + for dim, included in ( + (BillingDimension.queries, 1_000_000), + (BillingDimension.storage_gb, 50), + (BillingDimension.llm_tokens, 0), + ): + metered_overage = snap.quantity(dim) - included + billed = inv.metered_quantity(dim) + assert abs(billed - metered_overage) <= 0.005 * max(metered_overage, 1) + + +def test_pricing_unit_sizes() -> None: + pricing = load_pricing(PRICING) + assert pricing.rates["queries"].unit_size == 1_000.0 + assert pricing.rates["llm_tokens"].unit_size == 1_000_000.0 + assert pricing.rates["storage_gb"].unit_size == 1.0 diff --git a/packages/ragctl/src/ragctl/main.py b/packages/ragctl/src/ragctl/main.py index d827335..4c56e3e 100644 --- a/packages/ragctl/src/ragctl/main.py +++ b/packages/ragctl/src/ragctl/main.py @@ -5713,6 +5713,78 @@ async def _run() -> None: asyncio.run(_run()) +@app.command("billing") +def billing( + tenant: str = typer.Option("acme", "--tenant", "-t", help="Tenant id."), + tier: str = typer.Option("pro", "--tier", help="Pricing tier for the invoice."), + queries: int = typer.Option(1_400_000, "--queries", help="Queries to meter."), + tokens: int = typer.Option(3_000_000, "--tokens", help="LLM tokens to meter."), + storage: float = typer.Option(80.0, "--storage-gb", help="Storage GB to meter."), +) -> None: + """Show the Step 7.9 metering -> invoice flow: meter usage, then bill it. + + Meters synthetic usage into a ``UsageMeter``, loads a default pricing model, + generates an invoice, and reconciles billed overage against metered usage + (the +/-0.5% acceptance bar). No external services. + + Example:: + + ragctl billing --tenant acme --tier pro + """ + from rag_observability.billing import ( + BillingDimension, + UsageMeter, + generate_invoice, + load_pricing, + ) + + pricing_data: dict[str, object] = { + "currency": "USD", + "tiers": [ + { + "id": "pro", + "monthly_base": 499, + "included": {"queries": 1_000_000, "storage_gb": 50}, + }, + {"id": "business", "monthly_base": 2500, "included": {"queries": 10_000_000}}, + ], + "metered_dimensions": [ + {"id": "queries", "unit": "1k queries", "price": 0.50}, + {"id": "storage_gb", "unit": "GB-month", "price": 0.25}, + {"id": "llm_tokens", "unit": "1M tokens", "price": 5.0}, + ], + } + pricing = load_pricing(pricing_data) + meter = UsageMeter() + meter.observe(tenant, BillingDimension.queries, queries) + meter.observe(tenant, BillingDimension.llm_tokens, tokens) + meter.set_gauge(tenant, BillingDimension.storage_gb, storage) + snap = meter.snapshot(tenant) + inv = generate_invoice(snap, pricing, tier) + + typer.echo(f"\nmetered usage — tenant={tenant}") + for dim, qty in sorted(snap.usage.items()): + typer.echo(f" {dim:<16} {qty:>14,.0f}") + typer.echo(f"\ninvoice — tenant={tenant} tier={tier}") + typer.echo("─" * 64) + for li in inv.line_items: + typer.echo(f" {li.description:<34} qty={li.quantity:>13,.0f} ${li.amount:>10,.2f}") + typer.echo("─" * 64) + typer.echo(f" {'TOTAL':<38} {inv.currency:>8} ${inv.total:>10,.2f}") + + inclusions = pricing.tiers[tier].inclusions if tier in pricing.tiers else {} + worst = 0.0 + for dim in (BillingDimension.queries, BillingDimension.llm_tokens, BillingDimension.storage_gb): + metered_overage = max(0.0, snap.quantity(dim) - inclusions.get(dim.value, 0.0)) + billed = inv.metered_quantity(dim) + denom = max(metered_overage, 1.0) + worst = max(worst, abs(billed - metered_overage) / denom) + verdict = "ok" if worst <= 0.005 else "MISMATCH" + typer.echo( + f" reconciliation: billed overage vs metered usage [{verdict}, max drift {worst:.4%}]" + ) + + def main() -> None: app() diff --git a/packages/ragctl/tests/test_billing.py b/packages/ragctl/tests/test_billing.py new file mode 100644 index 0000000..39f51ca --- /dev/null +++ b/packages/ragctl/tests/test_billing.py @@ -0,0 +1,42 @@ +"""Tests for ``ragctl billing`` — Step 7.9. + +Meters synthetic usage, generates an invoice, and reconciles billed overage +against metered usage. No infrastructure. +""" + +from __future__ import annotations + +from ragctl.main import app +from typer.testing import CliRunner + +runner = CliRunner() + + +def test_billing_invoice_and_reconciliation() -> None: + result = runner.invoke(app, ["billing", "--tenant", "acme", "--tier", "pro"]) + assert result.exit_code == 0, result.output + assert "invoice — tenant=acme tier=pro" in result.output + assert "TOTAL" in result.output + # base 499 + queries 400k/1k*0.5=200 + storage 30*0.25=7.5 + tokens 3M/1M*5=15 + assert "721.50" in result.output + assert "reconciliation" in result.output + assert "[ok," in result.output + + +def test_billing_within_inclusions_is_base_only() -> None: + result = runner.invoke( + app, + [ + "billing", + "--tier", + "business", + "--queries", + "1000", + "--tokens", + "0", + "--storage-gb", + "0", + ], + ) + assert result.exit_code == 0, result.output + assert "2,500.00" in result.output # business base, no overage