diff --git a/README.md b/README.md index 6eb172b..6f3f08e 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,10 @@ +

+ + + Reversi logo + +

+ # Reversi **Reversibility, approvals, policy and tamper-evident evidence for every tool an AI agent can touch.** @@ -6,20 +13,27 @@ Agents write to real systems: orders, payments, tickets, email. When a run goes answers: *what did it change, why, and how do we undo it?* Reversi sits between an agent and its tools and answers all three. -- **Tool manifests** declare how reversible each tool is, what undoes it, and which permission it needs. +- **Tool manifests** declare how reversible each tool is, what undoes it, and which permission it needs. They live in a + versioned, signable registry (drafts, approval, deprecation) and can be drafted from an MCP server's own tool list. - **Write-ahead journal**: intent is recorded *before* a tool runs; if journaling fails, the call doesn't run. -- **Saga rollback**: newest first, idempotent, dependency-aware, with retries and verification. It never claims an +- **Saga rollback**: newest first, idempotent, dependency-aware, with retries and read-back verification. It never claims an irreversible step was undone, and unknown outcomes are reconciled by a human, not guessed. - **Hash-chained ledger** on files, SQLite or PostgreSQL: edits, deletions and reordering are detectable. Arguments and - results are hashed, never stored; credentials in free text are redacted. -- **Policy as code**: allow/deny/approve rules, rate limits, environments, tests (`reversi policy test`), plus content - checks for credentials, hidden payloads, injection phrasing and exfiltration sequences. -- **Approvals**: queued, bound to the exact arguments, single use, expiring, persisted, with webhook notification. -- **MCP gateway**: put it in front of any MCP server (stdio or Streamable HTTP) to journal, enforce, approve and roll - back without changing the agent. Operator tools live on a separate authenticated endpoint. -- **Console, API and metrics**: roles (agent, auditor, approver, operator, admin), key rotation, OIDC, Prometheus, - kill switch, signed evidence bundles and auditor reports. -- **Connectors** for filesystem, GitHub, Slack and Stripe with working compensations. + results are hashed, never stored; credentials in free text are redacted. Heads can be anchored outside the system. +- **Policy as code**: allow/deny/approve rules, rate limits, environments, risk scoring, tests (`reversi policy test`), plus content + checks for credentials, hidden payloads, injection phrasing and exfiltration sequences. Simulate a policy change against + captured history before you publish it. +- **Approvals**: queued, bound to the exact arguments, single use, expiring, persisted; quorum, group routing, escalation, + delegation and break-glass; notified over Slack, Teams, email, PagerDuty, Opsgenie or a signed webhook. +- **MCP gateway**: put it in front of any MCP server (stdio or Streamable HTTP). With no manifests it starts in observe mode and + shows what to classify; then enforce, advise or observe, with schema validation, timeouts, retries and read-backs. Operator + tools live on a separate authenticated endpoint. +- **Console, API and metrics**: a server-rendered console (off by default; open on localhost, or login) to decide approvals, run + rollbacks and edit manifests and policy; roles (agent, auditor, approver, operator, admin), key rotation, OIDC, Prometheus, + OpenTelemetry, kill switches, signed evidence bundles and auditor reports. +- **GitOps**: `reversi.config.yaml`, `reversi apply` to make a registry match a directory in git, hot reload, retention and legal holds. +- **Connectors** for fourteen systems (filesystem, GitHub, GitLab, Slack, Teams, Linear, Jira, Stripe, Google Calendar, Postgres, + S3, Cloud Storage, Blob Storage, Kubernetes) with working compensations and a certification suite. ## Try it @@ -37,25 +51,33 @@ Node 22.13+ is required. For the gateway, see [examples/gateway](examples/gatewa | Path | What | |---|---| | `packages/reversi` | Core: manifests, ledger, engine, policy, content checks, approvals, evidence, stores, auth (`@multimindlab/reversi`) | -| `packages/cli` | `reversi` command line | +| `packages/cli` | `reversi` command line (init, doctor, lint, test, apply, inspect, rollback, approvals, simulation) | | `packages/server` | HTTP API, operator console and metrics (Hono) | | `packages/gateway` | MCP gateway (`reversi-gateway`) | -| `packages/connectors` | Filesystem, GitHub, Slack, Stripe connectors and `reversi-connectors` | -| `packages/sdk-py` | Python client and offline verifier, stdlib only (`reversi-sdk`) | +| `packages/connectors` | Fourteen connectors and `reversi-connectors` | +| `packages/sdk-py` | Python client, offline verifier, Pydantic bindings and framework adapters (`reversi-sdk`) | | `viewer/` | Single-file trace viewer | | `examples/` | TypeScript, Python and gateway examples | | `docs/` | Mintlify docs (`docs/docs.json` is the navigation) | -| `deploy/`, `Dockerfile` | Docker image and a Compose stack on PostgreSQL | +| `deploy/`, `Dockerfile` | Docker image, a Compose stack on PostgreSQL, and a Helm chart (`deploy/helm/reversi`) | | `schemas/`, `conformance/` | JSON Schemas and cross-language test fixtures | ## Honest status Built and tested with unit and integration tests, including a live PostgreSQL run and real stdio processes. -**Not exercised here:** the connectors against live GitHub, Slack and Stripe (they are tested against fakes of those -APIs), the Dockerfile, Compose stack and GitHub workflows (written, not executed). **Not built:** OpenTelemetry traces, -risk scoring, retention tooling, framework adapters, a hosted service. Reversi enforces what manifests declare; it -cannot discover what a tool really does, content checks are heuristics, and the ledger is tamper-*evident*, not -tamper-proof. Read [limits](docs/limits.mdx) and the [threat model](docs/threat-model.mdx) before relying on it. + +**Built but not exercised against real systems:** the fourteen connectors (tested against fakes of each API, except the local +filesystem one), notification channels and OpenTelemetry export (tested against local fake receivers), the Python framework +adapters (tested against fakes of each framework), the Dockerfile, the Compose stack, the **Helm chart (never installed on a +cluster; `helm` was not available to lint it)** and the GitHub workflows (written, not run). + +**Not built:** a Terraform provider, a Kubernetes operator (use `reversi apply` and the Helm chart), OIDC browser redirect login +for the console (login takes a pasted API key or access token), a hosted control plane, Go and Java SDKs, clustering with run +hand-off, gateway passthrough of MCP resources, prompts and sampling, and an external security review. + +Reversi enforces what manifests declare; it cannot discover what a tool really does, content checks are heuristics, and the +ledger is tamper-*evident*, not tamper-proof. Read [limits](docs/limits.mdx), the [roadmap](docs/roadmap.mdx) and the +[threat model](docs/threat-model.mdx) before relying on it. ## Docs diff --git a/ROADMAP.md b/ROADMAP.md index 0083ea8..928c692 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -1,6 +1,14 @@ # Roadmap -See [docs/roadmap.mdx](docs/roadmap.mdx) for what is built and what is not. In short: the engine, policy, approvals, -content checks, signed evidence, three stores, the server with console, the MCP gateway, four connectors, the CLI and the -Python SDK are built. Not built: OpenTelemetry traces, risk scoring and delegated approval, retention tooling, more -connectors, framework adapters, a hosted service. +See [docs/roadmap.mdx](docs/roadmap.mdx) for the full list of what is built, what is built but not yet verified against real +systems, and what is not built. In short: + +- **Built:** the engine, manifest registry, policy (with simulation), approvals (quorum, delegation, break-glass), content + checks, signed and anchored evidence, three stores, the server and console, the MCP gateway, the CLI and `reversi apply`, + notifications, OpenTelemetry, retention and legal holds, fourteen connectors, Python framework adapters, Docker, Compose and + a Helm chart. +- **Built, not verified against real systems:** the connectors, notification channels, framework adapters, Docker files, the + Helm chart (never installed on a cluster) and the GitHub workflows. +- **Not built:** a Terraform provider, a Kubernetes operator, OIDC browser redirect login for the console, a hosted control + plane, Go and Java SDKs, clustering with run hand-off, gateway passthrough of MCP resources, prompts and sampling, and an + external security review. diff --git a/SECURITY.md b/SECURITY.md index 080e678..e86d142 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -1,13 +1,16 @@ # Security policy -Reversi is a security-relevant project. Please report vulnerabilities privately, not in a public -issue. +Reversi is a security-relevant project. Please report vulnerabilities privately, not in a public issue. -**Report to:** security contact to be set before the repository is made public (replace this line -with an address or a GitHub private vulnerability reporting link). +**Report a vulnerability:** use GitHub's private vulnerability reporting. On the repository page open the +**Security** tab, then **Report a vulnerability** +(). Only the maintainers can see the report. -Please include affected version, a description, and reproduction steps. We aim to acknowledge -reports within a few working days. +For a hardening question or an idea that is not sensitive (a missing check, a documentation gap), a normal public +issue is fine. If you are not sure which this is, use the private route. + +Please include the affected version, a description, and reproduction steps. We aim to acknowledge reports within a few +working days. ## Scope notes diff --git a/conformance/manifest-cases.json b/conformance/manifest-cases.json index 0f0b1c1..4c7998c 100644 --- a/conformance/manifest-cases.json +++ b/conformance/manifest-cases.json @@ -11,7 +11,9 @@ "effect": "read_only", "permission": "orders:read", "approval": "never", - "requires": [] + "requires": [], + "version": 1, + "status": "approved" } }, { @@ -26,7 +28,9 @@ "effect": "idempotent", "permission": "flags:write", "approval": "never", - "requires": [] + "requires": [], + "version": 1, + "status": "approved" } }, { @@ -49,13 +53,17 @@ "permission": "pay:write", "approval": "never", "requires": [], + "version": 1, + "status": "approved", "compensation": { "tool": "refund", "argument_map": { "charge_id": "$result.charge_id", "amt": "$args.amount" }, - "requires_approval": false + "requires_approval": false, + "kind": "compensation", + "residue": [] } } }, @@ -76,10 +84,14 @@ "permission": "docs:write", "approval": "never", "requires": [], + "version": 1, + "status": "approved", "compensation": { "tool": "delete_draft", "argument_map": {}, - "requires_approval": false + "requires_approval": false, + "kind": "compensation", + "residue": [] } } }, @@ -95,7 +107,9 @@ "effect": "irreversible", "permission": "mail:send", "approval": "always", - "requires": [] + "requires": [], + "version": 1, + "status": "approved" } }, { @@ -111,7 +125,9 @@ "effect": "irreversible", "permission": "mail:send", "approval": "always", - "requires": [] + "requires": [], + "version": 1, + "status": "approved" } }, { @@ -131,7 +147,9 @@ "approval": "never", "requires": [ "a" - ] + ], + "version": 1, + "status": "approved" } }, { @@ -299,10 +317,14 @@ "permission": "pay:write", "approval": "never", "requires": [], + "version": 1, + "status": "approved", "compensation": { "tool": "refund", "argument_map": {}, - "requires_approval": false + "requires_approval": false, + "kind": "compensation", + "residue": [] } } }, @@ -315,6 +337,374 @@ }, "valid": false }, + { + "input": { + "tool": "pay.create", + "effect": "compensable", + "permission": "pay:write", + "version": 3, + "status": "deprecated", + "replaced_by": "pay.create_v2", + "owner": "payments", + "oncall": "pager:pay", + "risk": { + "data_class": "financial", + "blast_radius": "high", + "environment": "production" + }, + "timeout_ms": "30s", + "idempotency": { + "key": "{{args.order_id}}:{{args.amount}}", + "scope": "actor" + }, + "audit": { + "retention_days": 400, + "evidence_export": "required", + "redaction": [ + "$args.client_secret", + "$result.client_secret" + ] + }, + "approval": { + "roles": [ + "finance-approver" + ], + "quorum": 2, + "timeout": "15m", + "on_timeout": "escalate", + "escalate_to": [ + "cfo" + ] + }, + "requires": [ + "a", + { + "tool": "b", + "status": "succeeded" + } + ], + "schema_hash": "abc123", + "compensation": { + "tool": "pay.cancel", + "argument_map": {}, + "kind": "compensation", + "residue": [ + "customer may get a notification" + ], + "timeout_ms": "10s", + "verification": { + "tool": "pay.get", + "arguments": { + "id": "$result.id" + }, + "expect": { + "status": "canceled" + }, + "retries": 4, + "backoff_ms": "2s" + } + } + }, + "valid": true, + "normalized": { + "tool": "pay.create", + "effect": "compensable", + "permission": "pay:write", + "approval": "always", + "requires": [ + "a", + "b" + ], + "version": 3, + "status": "deprecated", + "approval_rules": { + "roles": [ + "finance-approver" + ], + "quorum": 2, + "timeout_ms": 900000, + "on_timeout": "escalate", + "escalate_to": [ + "cfo" + ] + }, + "replaced_by": "pay.create_v2", + "owner": "payments", + "oncall": "pager:pay", + "schema_hash": "abc123", + "risk": { + "data_class": "financial", + "blast_radius": "high", + "environment": "production" + }, + "timeout_ms": 30000, + "idempotency": { + "key": "{{args.order_id}}:{{args.amount}}", + "scope": "actor" + }, + "audit": { + "retention_days": 400, + "evidence_export": "required", + "redaction": [ + "$args.client_secret", + "$result.client_secret" + ] + }, + "compensation": { + "tool": "pay.cancel", + "argument_map": {}, + "requires_approval": false, + "kind": "compensation", + "residue": [ + "customer may get a notification" + ], + "verification": { + "tool": "pay.get", + "arguments": { + "id": "$result.id" + }, + "expect": { + "status": "canceled" + }, + "retries": 4, + "backoff_ms": 2000 + }, + "timeout_ms": 10000 + } + } + }, + { + "input": { + "tool": "x", + "effect": "reversible", + "permission": "p", + "compensation": { + "tool": "y", + "argument_map": {}, + "kind": "inverse" + } + }, + "valid": true, + "normalized": { + "tool": "x", + "effect": "reversible", + "permission": "p", + "approval": "never", + "requires": [], + "version": 1, + "status": "approved", + "compensation": { + "tool": "y", + "argument_map": {}, + "requires_approval": false, + "kind": "inverse", + "residue": [] + } + } + }, + { + "input": { + "tool": "x", + "effect": "reversible", + "permission": "p", + "compensation": { + "tool": "y", + "argument_map": {}, + "kind": "inverse", + "residue": [ + "something" + ] + } + }, + "valid": false + }, + { + "input": { + "tool": "x", + "effect": "read_only", + "permission": "p", + "retry": { + "attempts": 4, + "base_ms": 100, + "max_ms": "2s" + } + }, + "valid": true, + "normalized": { + "tool": "x", + "effect": "read_only", + "permission": "p", + "approval": "never", + "requires": [], + "version": 1, + "status": "approved", + "retry": { + "attempts": 4, + "base_ms": 100, + "max_ms": 2000 + } + } + }, + { + "input": { + "tool": "x", + "effect": "compensable", + "permission": "p", + "retry": { + "attempts": 4 + }, + "compensation": { + "tool": "y", + "argument_map": {} + } + }, + "valid": false + }, + { + "input": { + "tool": "x", + "effect": "read_only", + "permission": "p", + "retry": { + "attempts": 0 + } + }, + "valid": false + }, + { + "input": { + "tool": "x", + "effect": "read_only", + "permission": "p", + "idempotency": { + "key": "no template" + } + }, + "valid": false + }, + { + "input": { + "tool": "x", + "effect": "read_only", + "permission": "p", + "idempotency": { + "key": "{{args.id}}", + "scope": "galaxy" + } + }, + "valid": false + }, + { + "input": { + "tool": "x", + "effect": "read_only", + "permission": "p", + "status": "retired" + }, + "valid": false + }, + { + "input": { + "tool": "x", + "effect": "read_only", + "permission": "p", + "version": 0 + }, + "valid": false + }, + { + "input": { + "tool": "x", + "effect": "read_only", + "permission": "p", + "risk": { + "blast_radius": "enormous" + } + }, + "valid": false + }, + { + "input": { + "tool": "x", + "effect": "read_only", + "permission": "p", + "timeout_ms": "soon" + }, + "valid": false + }, + { + "input": { + "tool": "x", + "effect": "read_only", + "permission": "p", + "approval": { + "quorum": 3 + } + }, + "valid": false + }, + { + "input": { + "tool": "x", + "effect": "read_only", + "permission": "p", + "approval": { + "roles": [ + "a" + ], + "quorum": 11 + } + }, + "valid": false + }, + { + "input": { + "tool": "x", + "effect": "read_only", + "permission": "p", + "audit": { + "redaction": [ + "client_secret" + ] + } + }, + "valid": false + }, + { + "input": { + "tool": "x", + "effect": "reversible", + "permission": "p", + "compensation": { + "tool": "y", + "argument_map": {}, + "verification": { + "tool": "g", + "expect": {} + } + } + }, + "valid": false + }, + { + "input": { + "tool": "x", + "effect": "reversible", + "permission": "p", + "compensation": { + "tool": "y", + "argument_map": {}, + "verification": { + "tool": "g", + "expect": { + "a": 1 + }, + "retries": 99 + } + } + }, + "valid": false + }, { "input": [], "valid": false diff --git a/deploy/docker-compose.yml b/deploy/docker-compose.yml index b598464..2346652 100644 --- a/deploy/docker-compose.yml +++ b/deploy/docker-compose.yml @@ -2,7 +2,7 @@ # node packages/connectors/dist/main.js manifests fs --format yaml > examples/gateway/manifests/fs.yaml (after npm run build) # cp deploy/keys.example.json deploy/keys.json # then replace the placeholder hashes (see the comments in the file) # POSTGRES_PASSWORD=$(openssl rand -hex 16) docker compose -f deploy/docker-compose.yml up --build -# The console is at http://127.0.0.1:7421/console . Put TLS in front of it for anything beyond localhost. +# The console is off by default; this file turns on the login console at http://127.0.0.1:7421/console (paste an API key). Put TLS in front of it for anything beyond localhost. services: postgres: image: postgres:16 @@ -30,6 +30,7 @@ services: REVERSI_STORE: postgres REVERSI_DATABASE_URL: postgres://reversi:${POSTGRES_PASSWORD}@postgres:5432/reversi REVERSI_LOG: json + REVERSI_CONSOLE: login volumes: - ./keys.json:/config/keys.json:ro - ../examples/gateway/manifests:/config/manifests:ro diff --git a/deploy/helm/reversi/.helmignore b/deploy/helm/reversi/.helmignore new file mode 100644 index 0000000..5357942 --- /dev/null +++ b/deploy/helm/reversi/.helmignore @@ -0,0 +1,2 @@ +.git +*.swp diff --git a/deploy/helm/reversi/Chart.yaml b/deploy/helm/reversi/Chart.yaml new file mode 100644 index 0000000..f28e658 --- /dev/null +++ b/deploy/helm/reversi/Chart.yaml @@ -0,0 +1,6 @@ +apiVersion: v2 +name: reversi +description: "Reversi server (and optionally the MCP gateway): journals, polices, approves and rolls back AI-agent tool calls" +type: application +version: 0.1.0 +appVersion: "0.1.0" diff --git a/deploy/helm/reversi/templates/_helpers.tpl b/deploy/helm/reversi/templates/_helpers.tpl new file mode 100644 index 0000000..8e2230a --- /dev/null +++ b/deploy/helm/reversi/templates/_helpers.tpl @@ -0,0 +1,50 @@ +{{- define "reversi.name" -}}{{ .Release.Name }}-reversi{{- end -}} +{{- define "reversi.labels" -}} +app.kubernetes.io/name: reversi +app.kubernetes.io/instance: {{ .Release.Name }} +app.kubernetes.io/version: {{ .Chart.AppVersion | quote }} +{{- end -}} +{{- define "reversi.manifestsCM" -}}{{ if .Values.manifests.existingConfigMap }}{{ .Values.manifests.existingConfigMap }}{{ else }}{{ include "reversi.name" . }}-manifests{{ end }}{{- end -}} +{{- define "reversi.policyCM" -}}{{ if .Values.policy.existingConfigMap }}{{ .Values.policy.existingConfigMap }}{{ else }}{{ include "reversi.name" . }}-policy{{ end }}{{- end -}} +{{- define "reversi.env" -}} +- { name: PORT, value: {{ .Values.server.port | quote }} } +- { name: HOST, value: "0.0.0.0" } +- { name: REVERSI_ENV, value: {{ .Values.server.environment | quote }} } +- { name: REVERSI_STORE, value: {{ .Values.store.kind | quote }} } +- { name: REVERSI_CONSOLE, value: {{ .Values.server.console | quote }} } +{{- if .Values.server.logJson }} +- { name: REVERSI_LOG, value: json } +{{- end }} +{{- if .Values.server.secureCookies }} +- { name: REVERSI_SECURE_COOKIES, value: "1" } +{{- end }} +{{- if .Values.registry.persistence.enabled }} +- { name: REVERSI_MANIFESTS, value: /state/manifests } +- { name: REVERSI_POLICY, value: /state/policy } +{{- else }} +{{- if or .Values.manifests.existingConfigMap .Values.manifests.inline }} +- { name: REVERSI_MANIFESTS, value: /config/manifests } +{{- end }} +{{- if or .Values.policy.existingConfigMap .Values.policy.inline }} +- { name: REVERSI_POLICY, value: /config/policy/policy.yaml } +{{- end }} +{{- end }} +{{- if .Values.keys.existingSecret }} +- { name: REVERSI_KEYS, value: /config/keys/{{ .Values.keys.key }} } +{{- end }} +{{- if eq .Values.store.kind "postgres" }} +- name: REVERSI_DATABASE_URL + valueFrom: { secretKeyRef: { name: {{ required "store.databaseUrlSecret.name is required for the postgres store" .Values.store.databaseUrlSecret.name }}, key: {{ .Values.store.databaseUrlSecret.key }} } } +{{- end }} +{{- if .Values.signing.existingSecret }} +- { name: REVERSI_SIGNING_KEY, value: /config/signing/signing.pem } +{{- end }} +{{- if .Values.oidc.enabled }} +- { name: REVERSI_OIDC_ISSUER, value: {{ .Values.oidc.issuer | quote }} } +- { name: REVERSI_OIDC_AUDIENCE, value: {{ .Values.oidc.audience | quote }} } +- { name: REVERSI_OIDC_ROLE_MAP, value: {{ .Values.oidc.roleMap | toJson | quote }} } +{{- end }} +{{- range .Values.extraEnv }} +- {{ toJson . }} +{{- end }} +{{- end -}} diff --git a/deploy/helm/reversi/templates/configmaps.yaml b/deploy/helm/reversi/templates/configmaps.yaml new file mode 100644 index 0000000..5e27edb --- /dev/null +++ b/deploy/helm/reversi/templates/configmaps.yaml @@ -0,0 +1,23 @@ +{{- if and .Values.manifests.inline (not .Values.manifests.existingConfigMap) }} +apiVersion: v1 +kind: ConfigMap +metadata: + name: {{ include "reversi.name" . }}-manifests + labels: {{- include "reversi.labels" . | nindent 4 }} +data: +{{- range $name, $text := .Values.manifests.inline }} + {{ $name }}: | +{{ $text | indent 4 }} +{{- end }} +--- +{{- end }} +{{- if and .Values.policy.inline (not .Values.policy.existingConfigMap) }} +apiVersion: v1 +kind: ConfigMap +metadata: + name: {{ include "reversi.name" . }}-policy + labels: {{- include "reversi.labels" . | nindent 4 }} +data: + policy.yaml: | +{{ .Values.policy.inline | indent 4 }} +{{- end }} diff --git a/deploy/helm/reversi/templates/deployment.yaml b/deploy/helm/reversi/templates/deployment.yaml new file mode 100644 index 0000000..b836a59 --- /dev/null +++ b/deploy/helm/reversi/templates/deployment.yaml @@ -0,0 +1,128 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: {{ include "reversi.name" . }} + labels: {{- include "reversi.labels" . | nindent 4 }} +spec: + replicas: {{ .Values.replicaCount }} + selector: + matchLabels: + app.kubernetes.io/name: reversi + app.kubernetes.io/instance: {{ .Release.Name }} + template: + metadata: + labels: {{- include "reversi.labels" . | nindent 8 }} + {{- with .Values.podAnnotations }} + annotations: {{- toYaml . | nindent 8 }} + {{- end }} + spec: + serviceAccountName: {{ if .Values.serviceAccount.create }}{{ include "reversi.name" . }}{{ else }}default{{ end }} + securityContext: { runAsNonRoot: true, runAsUser: 1000, fsGroup: 1000, seccompProfile: { type: RuntimeDefault } } + {{- if .Values.registry.persistence.enabled }} + initContainers: + - name: init-state + image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}" + command: ["sh", "-c", "mkdir -p /state/manifests /state/policy"] + securityContext: { allowPrivilegeEscalation: false, readOnlyRootFilesystem: true, capabilities: { drop: [ALL] } } + volumeMounts: [{ name: state, mountPath: /state }] + {{- end }} + containers: + - name: server + image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}" + imagePullPolicy: {{ .Values.image.pullPolicy }} + securityContext: { allowPrivilegeEscalation: false, readOnlyRootFilesystem: true, capabilities: { drop: [ALL] } } + env: {{- include "reversi.env" . | nindent 12 }} + - { name: REVERSI_DATA, value: /data } + ports: [{ name: http, containerPort: {{ .Values.server.port }} }] + readinessProbe: { httpGet: { path: /readyz, port: http }, periodSeconds: 10 } + livenessProbe: { httpGet: { path: /healthz, port: http }, periodSeconds: 20, failureThreshold: 3 } + {{- with .Values.resources }} + resources: {{- toYaml . | nindent 12 }} + {{- end }} + volumeMounts: + - { name: data, mountPath: /data } + - { name: tmp, mountPath: /tmp } + {{- if .Values.registry.persistence.enabled }} + - { name: state, mountPath: /state } + {{- end }} + {{- if and (not .Values.registry.persistence.enabled) (or .Values.manifests.existingConfigMap .Values.manifests.inline) }} + - { name: manifests, mountPath: /config/manifests, readOnly: true } + {{- end }} + {{- if or .Values.policy.existingConfigMap .Values.policy.inline }} + - { name: policy, mountPath: /config/policy, readOnly: true } + {{- end }} + {{- if .Values.keys.existingSecret }} + - { name: keys, mountPath: /config/keys, readOnly: true } + {{- end }} + {{- if .Values.signing.existingSecret }} + - { name: signing, mountPath: /config/signing, readOnly: true } + {{- end }} + {{- if .Values.gateway.enabled }} + - name: gateway + image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}" + imagePullPolicy: {{ .Values.image.pullPolicy }} + command: ["node", "packages/gateway/dist/main.js", "--config", "/config/gateway/gateway.yaml", "--console", {{ .Values.gateway.console | quote }}] + securityContext: { allowPrivilegeEscalation: false, readOnlyRootFilesystem: true, capabilities: { drop: [ALL] } } + env: {{- include "reversi.env" . | nindent 12 }} + - { name: REVERSI_DATA, value: /data } + ports: [{ name: mcp, containerPort: {{ .Values.gateway.port }} }] + readinessProbe: { httpGet: { path: /readyz, port: mcp }, periodSeconds: 10 } + volumeMounts: + - { name: data, mountPath: /data } + - { name: tmp, mountPath: /tmp } + - { name: gateway, mountPath: /config/gateway, readOnly: true } + {{- if .Values.registry.persistence.enabled }} + - { name: state, mountPath: /state } + {{- end }} + {{- if or .Values.manifests.existingConfigMap .Values.manifests.inline }} + - { name: manifests, mountPath: /config/manifests, readOnly: true } + {{- end }} + {{- if or .Values.policy.existingConfigMap .Values.policy.inline }} + - { name: policy, mountPath: /config/policy, readOnly: true } + {{- end }} + {{- if .Values.keys.existingSecret }} + - { name: keys, mountPath: /config/keys, readOnly: true } + {{- end }} + {{- end }} + volumes: + - name: tmp + emptyDir: {} + {{- if .Values.registry.persistence.enabled }} + - name: state + persistentVolumeClaim: { claimName: {{ include "reversi.name" . }}-state } + {{- end }} + - name: data + {{- if and .Values.store.persistence.enabled (ne .Values.store.kind "postgres") }} + persistentVolumeClaim: { claimName: {{ include "reversi.name" . }}-data } + {{- else }} + emptyDir: {} + {{- end }} + {{- if or .Values.manifests.existingConfigMap .Values.manifests.inline }} + - name: manifests + configMap: { name: {{ include "reversi.manifestsCM" . }} } + {{- end }} + {{- if or .Values.policy.existingConfigMap .Values.policy.inline }} + - name: policy + configMap: { name: {{ include "reversi.policyCM" . }} } + {{- end }} + {{- if .Values.keys.existingSecret }} + - name: keys + secret: { secretName: {{ .Values.keys.existingSecret }} } + {{- end }} + {{- if .Values.signing.existingSecret }} + - name: signing + secret: { secretName: {{ .Values.signing.existingSecret }} } + {{- end }} + {{- if .Values.gateway.enabled }} + - name: gateway + configMap: { name: {{ required "gateway.existingConfigMap is required when the gateway is enabled" .Values.gateway.existingConfigMap }} } + {{- end }} + {{- with .Values.nodeSelector }} + nodeSelector: {{- toYaml . | nindent 8 }} + {{- end }} + {{- with .Values.tolerations }} + tolerations: {{- toYaml . | nindent 8 }} + {{- end }} + {{- with .Values.affinity }} + affinity: {{- toYaml . | nindent 8 }} + {{- end }} diff --git a/deploy/helm/reversi/templates/ingress.yaml b/deploy/helm/reversi/templates/ingress.yaml new file mode 100644 index 0000000..55b5e8f --- /dev/null +++ b/deploy/helm/reversi/templates/ingress.yaml @@ -0,0 +1,28 @@ +{{- if .Values.ingress.enabled }} +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: {{ include "reversi.name" . }} + labels: {{- include "reversi.labels" . | nindent 4 }} + {{- with .Values.ingress.annotations }} + annotations: {{- toYaml . | nindent 4 }} + {{- end }} +spec: + {{- if .Values.ingress.className }} + ingressClassName: {{ .Values.ingress.className }} + {{- end }} + {{- if .Values.ingress.tls.enabled }} + tls: + - hosts: [{{ .Values.ingress.host | quote }}] + {{- if .Values.ingress.tls.secretName }} + secretName: {{ .Values.ingress.tls.secretName }} + {{- end }} + {{- end }} + rules: + - host: {{ .Values.ingress.host | quote }} + http: + paths: + - path: / + pathType: Prefix + backend: { service: { name: {{ include "reversi.name" . }}, port: { name: http } } } +{{- end }} diff --git a/deploy/helm/reversi/templates/pvc.yaml b/deploy/helm/reversi/templates/pvc.yaml new file mode 100644 index 0000000..fbdc686 --- /dev/null +++ b/deploy/helm/reversi/templates/pvc.yaml @@ -0,0 +1,27 @@ +{{- if and .Values.store.persistence.enabled (ne .Values.store.kind "postgres") }} +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: {{ include "reversi.name" . }}-data + labels: {{- include "reversi.labels" . | nindent 4 }} +spec: + accessModes: [ReadWriteOnce] + {{- if .Values.store.persistence.storageClass }} + storageClassName: {{ .Values.store.persistence.storageClass }} + {{- end }} + resources: { requests: { storage: {{ .Values.store.persistence.size }} } } +{{- end }} +--- +{{- if .Values.registry.persistence.enabled }} +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: {{ include "reversi.name" . }}-state + labels: {{- include "reversi.labels" . | nindent 4 }} +spec: + accessModes: [ReadWriteOnce] + {{- if .Values.registry.persistence.storageClass }} + storageClassName: {{ .Values.registry.persistence.storageClass }} + {{- end }} + resources: { requests: { storage: {{ .Values.registry.persistence.size }} } } +{{- end }} diff --git a/deploy/helm/reversi/templates/service.yaml b/deploy/helm/reversi/templates/service.yaml new file mode 100644 index 0000000..c4361c5 --- /dev/null +++ b/deploy/helm/reversi/templates/service.yaml @@ -0,0 +1,14 @@ +apiVersion: v1 +kind: Service +metadata: + name: {{ include "reversi.name" . }} + labels: {{- include "reversi.labels" . | nindent 4 }} +spec: + selector: + app.kubernetes.io/name: reversi + app.kubernetes.io/instance: {{ .Release.Name }} + ports: + - { name: http, port: {{ .Values.server.port }}, targetPort: http } + {{- if .Values.gateway.enabled }} + - { name: mcp, port: {{ .Values.gateway.port }}, targetPort: mcp } + {{- end }} diff --git a/deploy/helm/reversi/templates/serviceaccount.yaml b/deploy/helm/reversi/templates/serviceaccount.yaml new file mode 100644 index 0000000..35d02a9 --- /dev/null +++ b/deploy/helm/reversi/templates/serviceaccount.yaml @@ -0,0 +1,11 @@ +{{- if .Values.serviceAccount.create }} +apiVersion: v1 +kind: ServiceAccount +metadata: + name: {{ include "reversi.name" . }} + labels: {{- include "reversi.labels" . | nindent 4 }} + {{- with .Values.serviceAccount.annotations }} + annotations: {{- toYaml . | nindent 4 }} + {{- end }} +automountServiceAccountToken: false +{{- end }} diff --git a/deploy/helm/reversi/templates/validate.yaml b/deploy/helm/reversi/templates/validate.yaml new file mode 100644 index 0000000..75f314a --- /dev/null +++ b/deploy/helm/reversi/templates/validate.yaml @@ -0,0 +1,4 @@ +{{- if eq .Values.server.console "open" }}{{ fail "server.console=open has no login and is refused on a cluster: use off or login" }}{{- end }} +{{- if and (gt (int .Values.replicaCount) 1) (ne .Values.store.kind "postgres") }}{{ fail "more than one replica needs store.kind=postgres" }}{{- end }} +{{- if and (eq .Values.server.console "login") (not .Values.keys.existingSecret) (not .Values.oidc.enabled) }}{{ fail "server.console=login needs keys.existingSecret or oidc.enabled to log in with" }}{{- end }} +{{- if and .Values.registry.persistence.enabled (gt (int .Values.replicaCount) 1) }}{{ fail "registry.persistence needs replicaCount 1 (the volume is ReadWriteOnce)" }}{{- end }} diff --git a/deploy/helm/reversi/values.yaml b/deploy/helm/reversi/values.yaml new file mode 100644 index 0000000..d332ae0 --- /dev/null +++ b/deploy/helm/reversi/values.yaml @@ -0,0 +1,74 @@ +# The image is built from the repository's Dockerfile; push it to your registry and set it here. +image: + repository: reversi + tag: "0.1.0" + pullPolicy: IfNotPresent + +replicaCount: 1 # more than one needs the sqlite-free stores: set store.kind to postgres + +server: + port: 7421 + # off | login. The open console is for a laptop and is refused on a cluster (the pod listens on all addresses). + console: off + environment: production + logJson: true + secureCookies: true + +# Config files are mounted at /config. Give them inline, or point at existing ConfigMap/Secret objects. +manifests: + # A registry folder is edited from the console and `reversi apply`; plain files are read-only. + existingConfigMap: "" # a ConfigMap with manifest YAML/JSON files; if empty, `inline` below is used + inline: {} # { "orders.yaml": "" } +# A writable, versioned registry on a PersistentVolume: lets the console editors and `reversi apply --server` publish manifests +# and policy at runtime. Needs replicaCount 1 (ReadWriteOnce). When enabled it replaces the ConfigMap sources above. +registry: + persistence: + enabled: false + size: 1Gi + storageClass: "" +policy: + existingConfigMap: "" + inline: "" # policy.yaml text +keys: + existingSecret: "" # a Secret with key `keys.json` (API keys as hashes; see deploy/keys.example.json) + key: keys.json + +store: + kind: postgres # files | sqlite | postgres + # postgres: the URL is read from this existing Secret (never put it in values) + databaseUrlSecret: { name: "", key: DATABASE_URL } + persistence: # files / sqlite only + enabled: false + size: 5Gi + storageClass: "" + +oidc: + enabled: false + issuer: "" + audience: "" + roleMap: {} # { "reversi-ops": "operator" } + +signing: + existingSecret: "" # a Secret holding an Ed25519 key `signing.pem`, for signed evidence + +extraEnv: [] # additional environment variables, e.g. REVERSI_CAPTURE: "1" + +gateway: + enabled: false + port: 7611 + existingConfigMap: "" # a ConfigMap with gateway.yaml (upstreams, mode, listen, auth) + console: off + +ingress: + enabled: false + className: "" + host: reversi.example.com + tls: { enabled: true, secretName: "" } + annotations: {} + +serviceAccount: { create: true, annotations: {} } +resources: {} +podAnnotations: {} +nodeSelector: {} +tolerations: [] +affinity: {} diff --git a/docs/api-reference/admin.mdx b/docs/api-reference/admin.mdx index 919fe63..72d61b7 100644 --- a/docs/api-reference/admin.mdx +++ b/docs/api-reference/admin.mdx @@ -1,35 +1,197 @@ --- title: Approvals, evidence and admin endpoints -description: Decide approvals, download signed bundles, check status, flip the kill switch. +description: Decide approvals, download signed bundles, flip kill switches, manage manifests, policy, keys, holds and retention. --- +Examples use `$AUDIT_KEY`, `$APPROVER_KEY`, `$OPS_KEY` and `$ADMIN_KEY` for credentials of that role. + ## Approvals ### `GET /v1/approvals?status=&run_id=` (auditor+) -`{"approvals":[{"id","run_id","step","kind","tool","reason","triggers":[…],"arg_keys":[…],"args_hash","status","requested_ms","expires_ms","decided_by","decided_ms","decision_note"}]}`. -`status`: `pending`, `approved`, `denied`, `consumed`, `expired`. Only argument **names** are stored. + +```bash +curl -s -H "Authorization: Bearer $AUDIT_KEY" 'http://127.0.0.1:7421/v1/approvals?status=pending' +``` +```json +{"approvals":[{"id":"apr_5f0b3a2e-…","run_id":"demo-2","step":null,"kind":"call","tool":"notes.purge_all","reason":"cleanup", + "triggers":["policy `no-destructive-names`: destructive name","manifest requires approval"],"arg_keys":["older_than"], + "args_hash":"0271c5d0…","status":"pending","requested_ms":1792171988196,"expires_ms":1792175588196,"decided_by":null, + "decided_ms":null,"decision_note":null,"requested_by":"agent1","required_roles":[],"quorum":1,"decision_count":0, + "decisions":[],"on_timeout":"deny","escalate_to":[],"escalated":false,"ttl_ms":3600000}]} +``` + +`status`: `pending`, `approved`, `denied`, `consumed`, `expired`. Only argument **names** are stored. `required_roles` (roles or +groups that may decide; empty means any approver), `quorum`, `decisions`, `on_timeout` and `escalated` come from the manifest +and policy ([approvals](/concepts/approvals)). A decided request also has `break_glass` when overridden that way. ### `POST /v1/approvals/{id}/approve` and `/deny` (approver+) -Optional body `{"note":"…"}`. Returns the updated record. 409 if it is no longer pending, 404 if unknown. + +```bash +curl -s -X POST -H "Authorization: Bearer $APPROVER_KEY" -H 'content-type: application/json' \ + http://127.0.0.1:7421/v1/approvals/$ID/approve -d '{"note":"checked with the owner"}' +``` + +Optional body `{"note":"…"}`. Returns the updated record (with quorum, status stays `pending` until enough distinct approvers +agree). Errors: 404 unknown; 403 you are not among the required roles or groups, the request was made by your own run, or you already voted; 409 no longer +pending (`approval … is approved, not pending`). The caller's roles and groups are taken from the credential. + +### `POST /v1/approvals/{id}/break-glass` (admin) + +```bash +curl -s -X POST -H "Authorization: Bearer $ADMIN_KEY" -H 'content-type: application/json' \ + http://127.0.0.1:7421/v1/approvals/$ID/break-glass \ + -d '{"reason":"prod incident 42, owner unreachable","evidence":"INC-42"}' +``` + +Emergency override of a pending request. `reason` needs at least 10 characters (400 otherwise) and `evidence` a ticket or +incident reference. The record becomes `approved` with `break_glass: {by, reason, evidence, ms}` and the console shows a break-glass badge on it. + +### Delegations + +`GET /v1/delegations` (auditor+), `POST /v1/delegations` (approver+), `DELETE /v1/delegations/{id}` (the delegator or an admin). +Body of the POST: `{"to":"bob","hours":24,"reason":"on leave","groups":["finance"]}`; `from` defaults to you and only an admin may +set another. → 201 with the delegation. + +`reversi-server` serves these routes. The same delegations are visible in the [console](/console) (Approvals page) and `reversi delegate`. ## Evidence ### `GET /v1/runs/{id}/bundle` and `GET /v1/ledger/bundle` (auditor+) + A signed [evidence bundle](/concepts/evidence). 404 unless the server has `REVERSI_SIGNING_KEY`. +```bash +curl -s -H "Authorization: Bearer $AUDIT_KEY" http://127.0.0.1:7421/v1/runs/demo-1/bundle > bundle.json +reversi verify-bundle bundle.json +``` + ### `GET /v1/evidence/key` (auditor+) -`{"alg":"ed25519","key_id":"…","public_key_pem":"…"}`. + +```json +{"alg":"ed25519","key_id":"a1533ad58f5fb547","public_key_pem":"-----BEGIN PUBLIC KEY-----\n…\n-----END PUBLIC KEY-----\n"} +``` + +### `GET /v1/ledger/verify` (auditor+) + +`{"ok":true,"records":3,"head":"e3c2c625…"}` or `{"ok":false,"seq":7,"error":"…"}`. ## Status and kill switch ### `GET /v1/status` (auditor+) -`{"halted": null | "reason","environment","runs","pending_approvals"}`. + +```json +{"halted":null,"environment":"default","runs":1,"pending_approvals":0} +``` ### `POST /v1/halt` (admin) -`{"reason":"incident 42"}`. Every new call is blocked until resumed; rollbacks still work. The caller's name is appended to the reason. -### `POST /v1/resume` (admin) +```bash +curl -s -X POST -H "Authorization: Bearer $ADMIN_KEY" -H 'content-type: application/json' \ + http://127.0.0.1:7421/v1/halt -d '{"reason":"incident 42","tool":"notes.*"}' +``` +```json +{"halted":null,"halt":{"id":"halt_f3c9bd6f-…","reason":"incident 42 (by root)","scope":{"tool":"notes.*"},"by":"root","ts_ms":1792171503842}} +``` + +`reason` is required. Optional `tool`, `actor`, `tenant` and `environment` (globs) narrow the scope; with none, every new call is +blocked. `halted` in the answer is the global halt message (null for a scoped halt). Rollbacks still work. The caller's name is +appended to the reason. + +### `GET /v1/halts` (auditor+) and `POST /v1/resume` (admin) + +`GET` → `{"halts":[…]}`. `POST /v1/resume` with `{"id":"halt_…"}` lifts one; with no body it lifts all → `{"halted":null,"lifted":1}`. + +## Legal holds, retention, anchors + +| Route | Role | Notes | +|---|---|---| +| `GET /v1/holds?all=1` | auditor | `{"holds":[{id, scope, reason, by, ts_ms}]}`; `all=1` includes released | +| `POST /v1/holds` | admin | `{"run_id"?, "tenant"?, "actor"?, "tool"?, "reason"}` at least one scope field; 201 | +| `DELETE /v1/holds/{id}` | admin | Release. 404 if not active | +| `GET /v1/retention/plan` | auditor | `{"config":{…},"runs":[…]}`, which runs are due and which are held. 404 without a `retention:` section in the policy | +| `POST /v1/retention/apply` | admin | `{"dry_run"?: true, "archive_dir"?: "…"}`. Deletes run state, approvals and captures past retention; never the ledger | +| `POST /v1/anchor` | admin | Write the ledger head to the configured sinks → `{"anchor","delivered","errors"}`; 404 with no sinks, 502 if a sink failed | + +See [retention](/operations/retention) and [evidence](/concepts/evidence). + +## Manifest and policy registry + +These are the routes `reversi apply --server` uses. They need a server started on a registry folder (and a policy versions +folder for the policy routes), else 404 `no manifest registry folder is configured on this server`. See [GitOps](/operations/gitops). + +### `GET /v1/registry` (operator+) + +```json +{"manifests":[{"tool":"notes.read","version":3,"status":"approved","checksum":"289e5d91…","manifest":{…}}], + "policy":{"version":1,"checksum":"263fab7a…","history":[…]}} +``` + +`policy` is null without a policy versions folder. + +### `POST /v1/registry/manifests` + +```bash +curl -s -X POST -H "Authorization: Bearer $ADMIN_KEY" -H 'content-type: application/json' \ + http://127.0.0.1:7421/v1/registry/manifests \ + -d '{"manifest":{"tool":"notes.purge_all","effect":"irreversible","permission":"notes:write"},"status":"approved"}' +``` +```json +{"tool":"notes.purge_all","version":1,"checksum":"d8f3d60c…"} +``` + +`status` `"draft"` needs **operator**; anything else publishes as `approved` and needs **admin**, then reloads the running +configuration. 400 for an invalid manifest (the message lists the problems). Optional `note`. New versions are signed if the +server has a signing key. + +### `POST /v1/registry/manifests/{tool}/approve` and `/deprecate` (admin) + +`approve` body `{"version"?: n, "four_eyes"?: true}` promotes a draft (the latest by default) to a new approved version; with +`four_eyes` the author cannot be the approver. `deprecate` body `{"replaced_by"?: "tool"}`. Both → `{"tool","version"}` and reload. + +### `POST /v1/registry/policy` and `POST /v1/registry/policy/{version}/activate` (admin) + +`{"text":"","note":"…","activate":true}` validates and stores a new version (activating it unless `activate` is +false) → 201 with the version record. `activate` makes an older version current, which is how you roll back → `{"active":1}`. +If the saved configuration fails to load the answer is 409 `saved, but the new configuration did not load: …` and the previous +one stays active. + +### `POST /v1/reload` (admin) + +Re-read manifests and policy now. + +```json +{"changed":false,"tools":3,"policy_checksum":"b9927db4…"} +``` + +`error` is present, with `changed: false`, if the new configuration was rejected. + +## API keys + +Need `REVERSI_KEYS` to point at a writable file (404 otherwise). Admin only. + +```bash +curl -s -X POST -H "Authorization: Bearer $ADMIN_KEY" -H 'content-type: application/json' \ + http://127.0.0.1:7421/v1/keys -d '{"name":"tmp","role":"auditor","expires_in_days":1}' +``` +```json +{"name":"tmp","roles":["auditor"],"key":"rv_9a8f0653…","note":"this is the only time the key is shown"} +``` + +- `GET /v1/keys` lists keys by `index` with name, roles, groups, tenant, expiry, `disabled`, `allow_observe`, `max_permissions` and a 12-character `fingerprint` (never the key). +- `POST /v1/keys` body: `name`, `role` or `roles`, optional `groups`, `tenant`, `max_permissions`, `expires_in_days`. Only the SHA-256 is stored. +- `PATCH /v1/keys/{index}` with `{"disabled": true}` and/or `{"expires_at": "" | null}`. +- `DELETE /v1/keys/{index}`. + +Changes take effect immediately. + +## Slack callback + +`POST /integrations/slack/actions` approves or denies from the buttons on a Slack notification, verifying Slack's signature and +a 5-minute timestamp window. `reversi-server` enables it when `REVERSI_SLACK_SIGNING_SECRET` and `REVERSI_SLACK_USERS` (a JSON file mapping Slack user ids to +names, roles and groups) are set, and otherwise answers `404 Slack approvals are not enabled`. A mapped user needs the +`approver` role or higher to decide ([authentication](/operations/authentication#slack-approvals-user-mapping)). ## Operational -`GET /healthz`, `GET /readyz`, `GET /metrics`: see [monitoring](/operations/monitoring). +`GET /healthz` → `{"ok":true}`. `GET /readyz` → `{"ok":true,"ledger_head":"…","halted":null}`, 503 if the store cannot be read. +`GET /metrics` (auditor, or open with `REVERSI_METRICS_PUBLIC=1`): see [monitoring](/operations/monitoring). diff --git a/docs/api-reference/operator.mdx b/docs/api-reference/operator.mdx index d3a1f1e..b2ac9fc 100644 --- a/docs/api-reference/operator.mdx +++ b/docs/api-reference/operator.mdx @@ -1,39 +1,222 @@ --- title: Operator endpoints -description: Plan, compensate, resolve, export, verify, report. Auditor reads; operator acts. +description: Plan, roll back, resolve, export, report, search, preview and simulate. Auditor reads; operator acts. --- -## `GET /v1/runs/{id}/plan` (auditor+) -Simulated rollback: `{"run_id","actions":[{"step","tool","kind","compensation_tool?","args?","idempotency_key?","blocked_by?","note?"}],"fully_reversible"}`. -`kind`: `compensate`, `skip`, `defer`, `unrecoverable`, `unresolved`, `unmanifested`. +Examples use `$AUDIT_KEY` (auditor), `$APPROVER_KEY` and `$OPS_KEY` (operator). -## `POST /v1/runs/{id}/approvals` (approver+) +## Rollback + +### `GET /v1/runs/{id}/plan` (auditor+) + +Simulated rollback. Executes nothing. + +```bash +curl -s -H "Authorization: Bearer $AUDIT_KEY" http://127.0.0.1:7421/v1/runs/demo-1/plan +``` ```json -{ "tool": "send_email", "kind": "call", "count": 1 } +{"run_id":"demo-1","actions":[{"step":0,"tool":"notes.write","kind":"compensate","compensation_tool":"notes.restore", + "args":{"name":"a","backup":"b1"},"idempotency_key":"1631984a…","needs_approval":false,"compensation_kind":"inverse","verified":true}], + "fully_reversible":true} +``` + +`kind`: `compensate`, `skip`, `defer` (waits for `blocked_by` steps), `unrecoverable`, `unresolved`, `unmanifested`. Other fields: +`compensation_tool`, `args`, `idempotency_key`, `needs_approval`, `compensation_kind` (`inverse` or `compensation`), `residue`, +`verified` (the undo is read back), `note`. + +### `POST /v1/runs/{id}/rollback` (operator+) + +Without `"confirm": true` this is a dry run: + +```bash +curl -s -X POST -H "Authorization: Bearer $OPS_KEY" -H 'content-type: application/json' \ + http://127.0.0.1:7421/v1/runs/demo-1/rollback -d '{}' +``` +```json +{"dry_run":true,"plan":{…},"executable":true} +``` + +With `{"confirm": true}` (and optionally `"approve_compensations": true`, which also approves compensations that need approval) +the server runs the rollback in the background and answers **202**: + +```json +{"run_id":"demo-1","running":true,"started_ms":1792171502758,"by":"ops"} +``` + +It needs a compensator: start the server with `REVERSI_COMPENSATOR_CMD` (404 `this server has no compensator configured` otherwise). +To run manifests' declared read-backs, also set `REVERSI_READER_CMD`: a command that receives `{"tool","arguments"}` as JSON +on stdin and prints the tool's JSON result. +Compensations are attempted 3 times with backoff and the same idempotency key. A second start while one runs is a 409. + +### `GET /v1/runs/{id}/rollback` (auditor+) + +Progress and result of the last rollback this server started for the run (404 if none): + +```json +{"run_id":"demo-1","running":false,"started_ms":1792171502758,"by":"ops","finished_ms":1792171502791, + "report":{"run_id":"demo-1","compensated":[0],"skipped":[],"failed":[],"unrecoverable":[],"unresolved":[], + "unmanifested":[],"deferred":[],"awaiting_approval":[],"unverified":[0],"complete":false}} ``` -Standing pre-approval for a tool in this run, any arguments, `count` uses. For compensations use `"kind": "compensation"` -and the **compensation** tool's name. To decide a queued request instead, see [admin](/api-reference/admin). -## `POST /v1/runs/{id}/steps/{n}/compensation` (operator+) -Journals `compensating` first, then returns what to run: -`{"decision":"go","call":{"tool","args","idempotency_key",…}}`, or `defer`, `awaiting_approval`, `not_actionable`, `nothing_to_do`. +`complete` is false in this example because the manifest declares a read-back and the server was started without +`REVERSI_READER_CMD`, so it could not call the tool to verify it and step 0 is `unverified`. With `REVERSI_READER_CMD` set and +the read-back matching, `unverified` is empty and `complete` is true. A failure to run at all appears as `error`. The +[gateway](/gateway) does the read-back through the upstream. -## `POST /v1/runs/{id}/steps/{n}/compensation/outcome` (operator+) -`{"status":"compensated"}` or `{"status":"failed","error":"…"}`. +### Step by step: `…/steps/{n}/compensation` (operator+) + +For callers that run the undo themselves. `POST /v1/runs/{id}/steps/{n}/compensation` journals `compensating` first, then says what to run: + +```json +{"decision":"go","step":0,"call":{"run_id":"demo-3","step":0,"tool":"notes.restore","original_tool":"notes.write", + "args":{"name":"b","backup":"b9"},"idempotency_key":"18784249…"}} +``` + +Other decisions: `defer`, `awaiting_approval`, `not_actionable` (with `kind` and `note`, e.g. `irreversible: cannot be undone, needs human follow-up`), `nothing_to_do`. + +Report what happened with `POST /v1/runs/{id}/steps/{n}/compensation/outcome`: + +```json +{"status":"compensated"} +{"status":"failed","error":"…"} +``` +→ `{"ok":true}` + +### `POST /v1/runs/{id}/steps/{n}/resolve` (operator+) -## `POST /v1/runs/{id}/steps/{n}/resolve` (operator+) Settle a step whose outcome was unknown, after checking downstream: + +```bash +curl -s -X POST -H "Authorization: Bearer $OPS_KEY" -H 'content-type: application/json' \ + http://127.0.0.1:7421/v1/runs/demo-2/steps/0/resolve \ + -d '{"status":"succeeded","note":"found the order in the admin UI"}' +``` ```json -{ "status": "succeeded", "result": { "id": 7 }, "note": "found order 7 in the admin UI" } +{"step":0,"status":"succeeded"} ``` -`note` is required and is stored in the ledger with the caller's name. `status` is `succeeded` or `failed` (with optional `error`). -Returns `{"step","status"}`. -## `GET /v1/runs/{id}/export` (auditor+) -The run's ledger records as `application/x-ndjson`. +`note` is required and is stored in the ledger with the caller's name. `status` is `succeeded` (optional `result`, so the step +can be rolled back) or `failed` (optional `error`). + +### `POST /v1/runs/{id}/approvals` (approver+) + +```json +{ "tool": "send_email", "kind": "call", "count": 1 } +``` + +Standing pre-approval for a tool in this run, any arguments, `count` uses (1 to 100). For compensations use `"kind": "compensation"` +and the **compensation** tool's name. → `201 {"ok":true}`. To decide a queued request instead, see [admin](/api-reference/admin). + +## Evidence and search + +### `GET /v1/runs/{id}/export` (auditor+) + +The run's ledger records as `application/x-ndjson`, one JSON object per line (`seq`, `ts_ms`, `run_id`, `step`, `actor`, +`tool`, `payload_hash`, `reason`, `event`, `detail`, `prev_hash`, `hash`). + +### `GET /v1/runs/{id}/report`, `GET /v1/runs/{id}/report.pdf` (auditor+) + +A self-contained auditor report as HTML, or as `application/pdf`. + +### `GET /v1/report` (auditor+) + +The period report: what agents did, what was blocked or would have been, approvals, rollbacks. + +```bash +curl -s -H "Authorization: Bearer $AUDIT_KEY" 'http://127.0.0.1:7421/v1/report?since=7d&format=md' +``` + +`since` is `7d` (default), `12h`, `30m` or an ISO date. `format` is `json` (default), `md` or `html`. The JSON has `runs`, `calls`, +`actors`, `tools`, `by_effect`, `outcomes`, `blocked`, `would_block`, `unclassified_tools`, `approvals`, `rollback`, +`irreversible_executed`, `needs_attention`, `budget_warnings` and `days`. + +### `GET /v1/search/runs` (auditor+) + +```bash +curl -s -H "Authorization: Bearer $AUDIT_KEY" \ + 'http://127.0.0.1:7421/v1/search/runs?actor=agent1&status=rolled_back&since=7d' +``` +```json +{"runs":[{"run_id":"demo-1","actor":"agent1","mode":"enforce","started_ms":1792171472466,"steps":1,"tools":["notes.write"], + "status":"rolled_back","unresolved":[],"failed_compensations":0}]} +``` + +Filters: `actor`, `tenant`, `tool`, `status` (`needs_attention`, `rolled_back`, `complete`, `blocked_only`, `empty`), `mode`, `since`, +`until`, `text` (part of the run id), `limit`, `after` (a cursor). + +### `GET /v1/search/records` (auditor+) + +Ledger records. Filters: `run`, `actor`, `tool`, `event` (one or several, comma separated), `text`, `since`, `until`, `limit`, `after_seq`. +→ `{"records":[…],"next_seq":1}`. + +## Preview and simulation + +### `GET /v1/impact` (auditor+) + +What a call would do, without making it. + +```bash +curl -s -H "Authorization: Bearer $AUDIT_KEY" \ + 'http://127.0.0.1:7421/v1/impact?tool=notes.write&args=%7B%22name%22:%22a%22%7D' +``` +```json +{"tool":"notes.write","known":true,"status":"approved","effect":"reversible","owner":"you@example.com", + "risk":{"data_class":"internal","blast_radius":"low"},"verdict":"allow","reasons":[], + "risk_score":{"score":20,"factors":["effect reversible +15","data class internal +5"]}, + "compensation":{"tool":"notes.restore","kind":"inverse","residue":[],"verified":true,"needs_approval":false}, + "requires":[],"rollback_outlook":["undo by calling notes.restore","the undo is read back and verified"],"warnings":[]} +``` + +`tool` is required; `args` is URL-encoded JSON; `actor` is optional. A tool with no manifest answers `"known": false`, +`"verdict": "block"`. + +### `POST /v1/speculate` (auditor+) + +Compare alternative plans without running them. + +```bash +curl -s -X POST -H "Authorization: Bearer $AUDIT_KEY" -H 'content-type: application/json' \ + http://127.0.0.1:7421/v1/speculate \ + -d '{"plans":[{"name":"a","calls":[{"tool":"notes.write","args":{"name":"x"}}]}]}' +``` + +Optional `policy` (YAML text) and `actor`. Each plan comes back with per-call verdicts, `approvals_needed`, `max_risk`, +`total_risk`, `irreversible`, `residue`, `fully_reversible`, its own `rollback` plan and a `rank`. Results the manifests had to +synthesize are marked in `notes`. + +### `POST /v1/simulate` (operator+) + +Replay captured calls through a candidate policy. Needs capture on (404 `capture is off` otherwise). + +```bash +curl -s -X POST -H "Authorization: Bearer $OPS_KEY" -H 'content-type: application/json' \ + http://127.0.0.1:7421/v1/simulate -d '{"since":"1d"}' +``` +```json +{"calls":1,"runs":1,"unchanged":1,"newly_blocked":[],"newly_needs_approval":[],"newly_allowed":[],"notes":[]} +``` + +Body: `policy` (YAML text; default is the running policy), `since` (default `7d`), `run_id`. + +### `POST /v1/runs/{id}/replay` (operator+) + +Replay one captured run, once as recorded and once with changes. Needs capture on. + +```json +{"changes":[{"path":"name","value":"b"}],"policy":"…optional YAML…"} +``` + +A change is `{path, value}` with optional `step` and `tool` to narrow it. → `baseline`, `variant`, `differences`, +`baseline_plan`, `variant_plan`. + +## Status and inventory + +### `GET /v1/alerts` (auditor+) + +`{"alerts":[…]}`: what needs attention right now, computed from the ledger, approvals and upstream health (severity, title, +detail, `ts_ms`, `run_id`). -## `GET /v1/runs/{id}/report` (auditor+) -A self-contained HTML auditor report. +### `GET /v1/connectors` (auditor+) -## `GET /v1/ledger/verify` (auditor+) -`{"ok":true,"records":42,"head":"…"}` or `{"ok":false,"seq":7,"error":"…"}`. +`{"connectors":[{"name","kind","healthy","tools","classified","unclassified","credential?","detail?"}]}`. Filled by the gateway +console; the standalone server returns an empty list. diff --git a/docs/api-reference/overview.mdx b/docs/api-reference/overview.mdx index 51c7e4d..ca26097 100644 --- a/docs/api-reference/overview.mdx +++ b/docs/api-reference/overview.mdx @@ -1,6 +1,6 @@ --- title: API overview -description: HTTP API conventions, roles and status codes. +description: HTTP API conventions, roles, status codes and an index of every route. --- Start the server (see [deployment](/operations/deployment) for every option): @@ -10,24 +10,65 @@ REVERSI_MANIFESTS=./tools.yaml REVERSI_KEYS=./keys.json node packages/server/dis ``` All `/v1` routes need `Authorization: Bearer `. `GET /healthz` and `GET /readyz` are open; -`GET /metrics` needs an auditor credential. [Roles](/operations/authentication) gate each route: +`GET /metrics` needs an auditor credential unless `REVERSI_METRICS_PUBLIC=1`. [Roles](/operations/authentication) gate each route. +Request and response bodies are JSON, except where noted. Bodies are capped at 1 MB. + +Examples in these pages use `$KEY`, `$AGENT_KEY`, `$OPS_KEY` and so on for credentials, and `http://127.0.0.1:7421` for the +server. They were run against a local server. | Plane | Role needed | Routes | |---|---|---| | Agent | `agent` | [runs](/api-reference/runs): open a run, `intents`, `outcome` | -| Read | `auditor`+ | plan, export, report, bundle, ledger verify, status, list approvals | -| Approve | `approver`+ | approve, deny, pre-approve | -| Operate | `operator`+ | compensation, resolve ([operator](/api-reference/operator)) | -| Admin | `admin` | halt, resume ([admin](/api-reference/admin)) | +| Read | `auditor`+ | run state, plan, export, reports, search, impact, alerts, status, approvals, ledger verify, bundles | +| Approve | `approver`+ | approve, deny, pre-approve, delegations | +| Operate | `operator`+ | compensation, resolve, rollback, simulate, replay, registry read ([operator](/api-reference/operator)) | +| Admin | `admin` | halt, resume, break-glass, publish or approve manifests and policy, keys, holds, retention, anchor, reload ([admin](/api-reference/admin)) | + +Roles are cumulative: each includes the ones before it. Agents see only the runs they opened. | Status | Meaning | |---|---| -| 400 | Invalid body or id | +| 400 | Invalid body, id or query | | 401 | Missing or invalid credential | | 403 | Role or permission cap forbids it | -| 404 | Unknown run (also for another agent's run), approval, or a feature that is not enabled | -| 409 | State conflict (duplicate run id, outcome already reported, approval already decided) | +| 404 | Unknown run (also for another agent's run), approval, or a feature that is not enabled on this server | +| 409 | State conflict (duplicate run id, outcome already reported, approval already decided, a saved configuration that did not load) | | 413 | Body over 1 MB | +| 502 | Anchor delivery failed | | 503 | Ledger unavailable; the request was refused (fail closed) | -Policy refusals are **not** errors: `POST …/intents` returns 200 with `"decision": "block"` and `reasons`. +Errors are `{"error": "message"}`. Policy refusals are **not** errors: `POST …/intents` returns 200 with `"decision": "block"` and `reasons`. + +**Features that are not enabled** answer 404 with a message saying why, for example `no retention settings`, `capture is off`, +`no manifest registry folder is configured on this server`, `signed bundles are not enabled on this server`. +On the standalone `reversi-server`, `/v1/connectors` returns an empty list (the gateway console fills it in), and +`/integrations/slack/actions` answers 404 unless `REVERSI_SLACK_SIGNING_SECRET` and `REVERSI_SLACK_USERS` are set. + +## Index + +| Route | Role | Page | +|---|---|---| +| `POST /v1/runs` | agent | [runs](/api-reference/runs) | +| `GET /v1/runs`, `GET /v1/runs/{id}` | agent (own), auditor+ (all) | [runs](/api-reference/runs) | +| `POST /v1/runs/{id}/intents`, `POST /v1/runs/{id}/steps/{n}/outcome` | agent (own run) | [runs](/api-reference/runs) | +| `GET /v1/runs/{id}/plan` | auditor | [operator](/api-reference/operator) | +| `POST /v1/runs/{id}/rollback`, `GET /v1/runs/{id}/rollback` | operator, auditor | [operator](/api-reference/operator) | +| `POST /v1/runs/{id}/steps/{n}/compensation`, `…/compensation/outcome`, `…/resolve` | operator | [operator](/api-reference/operator) | +| `POST /v1/runs/{id}/approvals` | approver | [operator](/api-reference/operator) | +| `GET /v1/runs/{id}/export`, `…/report`, `…/report.pdf` | auditor | [operator](/api-reference/operator) | +| `GET /v1/search/runs`, `GET /v1/search/records` | auditor | [operator](/api-reference/operator) | +| `GET /v1/impact`, `POST /v1/speculate` | auditor | [operator](/api-reference/operator) | +| `POST /v1/simulate`, `POST /v1/runs/{id}/replay` | operator | [operator](/api-reference/operator) | +| `GET /v1/report`, `GET /v1/alerts`, `GET /v1/connectors` | auditor | [operator](/api-reference/operator) | +| `GET /v1/approvals`, `POST /v1/approvals/{id}/approve`, `…/deny` | auditor, approver | [admin](/api-reference/admin) | +| `POST /v1/approvals/{id}/break-glass` | admin | [admin](/api-reference/admin) | +| `GET/POST /v1/delegations`, `DELETE /v1/delegations/{id}` | auditor, approver | [admin](/api-reference/admin) | +| `GET /v1/runs/{id}/bundle`, `GET /v1/ledger/bundle`, `GET /v1/evidence/key`, `GET /v1/ledger/verify` | auditor | [admin](/api-reference/admin) | +| `GET /v1/status`, `GET /v1/halts` | auditor | [admin](/api-reference/admin) | +| `POST /v1/halt`, `POST /v1/resume`, `POST /v1/reload` | admin | [admin](/api-reference/admin) | +| `GET/POST /v1/holds`, `DELETE /v1/holds/{id}`, `GET /v1/retention/plan`, `POST /v1/retention/apply`, `POST /v1/anchor` | auditor read, admin write | [admin](/api-reference/admin) | +| `GET /v1/registry` | operator | [admin](/api-reference/admin) | +| `POST /v1/registry/manifests`, `…/{tool}/approve`, `…/{tool}/deprecate`, `POST /v1/registry/policy`, `…/policy/{version}/activate` | operator (drafts), admin | [admin](/api-reference/admin) | +| `GET/POST /v1/keys`, `PATCH/DELETE /v1/keys/{index}` | admin | [admin](/api-reference/admin) | +| `POST /integrations/slack/actions` | Slack signature | [admin](/api-reference/admin) | +| `GET /healthz`, `/readyz`, `/metrics` | open, open, auditor | [monitoring](/operations/monitoring) | diff --git a/docs/api-reference/runs.mdx b/docs/api-reference/runs.mdx index 358bc2f..aff7bbd 100644 --- a/docs/api-reference/runs.mdx +++ b/docs/api-reference/runs.mdx @@ -3,38 +3,86 @@ title: Agent endpoints description: Open runs, request calls, report outcomes. Agent role. --- -Agents see only runs they opened (another agent's run is a 404). Auditors and above can read any run. +Agents see only runs they opened (another agent's run is a 404). Auditors and above can read any run. Journaling on a run +(`intents`, `outcome`) is only ever done by the principal that opened it, whatever other roles the key holds. ## `POST /v1/runs` +```bash +curl -s -X POST http://127.0.0.1:7421/v1/runs \ + -H "Authorization: Bearer $AGENT_KEY" -H 'content-type: application/json' \ + -d '{"permissions":["notes:read","notes:write"],"run_id":"demo-1","reason":"docs example"}' +``` ```json -{ "permissions": ["orders:write"], "run_id": "optional", "mode": "enforce", "reason": "why this run exists" } +{"run_id":"demo-1","mode":"enforce","scope_seal":"336f0ddd8ee69f57…"} ``` -→ `201 {"run_id","mode","scope_seal"}`. Permissions must fit within the key's `max_permissions`. + +| Field | Meaning | +|---|---| +| `permissions` | Required. The run's scope. Must fit within the key's `max_permissions` (else 403) | +| `run_id` | Optional (generated if omitted). Up to 128 characters: letters, digits, `.`, `_`, `-`, starting with a letter or digit. 400 otherwise, 409 if it exists | +| `mode` | `enforce` (default), `observe` or `advise`. The last two let violating calls run, so a key needs `allow_observe` (else 403) | +| `reason` | Optional free text recorded on the run | + +→ `201`. Mode and scope are sealed at this point and cannot be widened later. ## `POST /v1/runs/{id}/intents` Authorize a call and journal its intent. Execute the tool only on `allow` or `would_block`. +```bash +curl -s -X POST http://127.0.0.1:7421/v1/runs/demo-1/intents \ + -H "Authorization: Bearer $AGENT_KEY" -H 'content-type: application/json' \ + -d '{"tool":"notes.write","args":{"name":"a"},"reason":"save note"}' +``` ```json -{ "tool": "create_order", "args": {"sku": "BOOK-1"}, "reason": "customer asked", "depends_on": [0] } +{"decision":"allow","step":0} ``` -→ `{"decision":"allow","step":3}` · `{"decision":"would_block","step":3,"reasons":[…]}` · `{"decision":"block","reasons":[…]}` + +`tool`, `args` (any JSON, hashed and never stored) and `reason` are required; `depends_on` is an optional list of earlier step +numbers this call needs. + +→ `{"decision":"allow","step":3}` · `{"decision":"would_block","step":3,"reasons":[…]}` (observe or advise mode) · `{"decision":"block","reasons":[…]}` A call waiting for approval is a `block` whose reason names the request: -`approval required: request apr_… is pending an operator decision; retry after it is approved`. After an approver decides it, -send the same intent again ([approvals](/concepts/approvals)). Violations also come from [policy](/concepts/policy), -[content checks](/concepts/content-checks) and the kill switch. + +```json +{"decision":"block","reasons":["approval required: request apr_5f0b3a2e-… is pending an operator decision; retry after it is approved"]} +``` + +After an approver decides it, send the same intent again ([approvals](/concepts/approvals)). A different set of arguments needs +a new approval. Violations also come from [policy](/concepts/policy), [content checks](/concepts/content-checks) and the kill switch. ## `POST /v1/runs/{id}/steps/{n}/outcome` +```bash +curl -s -X POST http://127.0.0.1:7421/v1/runs/demo-1/steps/0/outcome \ + -H "Authorization: Bearer $AGENT_KEY" -H 'content-type: application/json' \ + -d '{"status":"succeeded","result":{"backup_id":"b1"}}' +``` +```json +{"step":0,"status":"succeeded"} +``` + ```json { "status": "succeeded", "result": {"order_id": "ord_1"} } { "status": "failed", "error": "…" } { "status": "uncertain", "error": "timeout" } ``` + `result` is used only to build the compensation arguments named in the manifest. It is hashed, not stored in the ledger. +Reporting an outcome twice is a 409. ## `GET /v1/runs`, `GET /v1/runs/{id}` -Run state, including `unresolved` step numbers. +```bash +curl -s -H "Authorization: Bearer $AUDIT_KEY" http://127.0.0.1:7421/v1/runs +curl -s -H "Authorization: Bearer $AUDIT_KEY" http://127.0.0.1:7421/v1/runs/demo-1 +``` +```json +{"runs":["demo-1"]} +``` + +The run: `run_id`, `actor`, `mode`, `scope` (permissions and seal), `created_ms`, `steps[]` (tool, reason, effect, status, +dependencies, compensation tool, handle, error, manifest version, risk score), and `unresolved`, the step numbers whose outcome +is unknown. For a list with filters use [search](/api-reference/operator). diff --git a/docs/cli.mdx b/docs/cli.mdx index b1e0590..11391b3 100644 --- a/docs/cli.mdx +++ b/docs/cli.mdx @@ -1,59 +1,213 @@ --- title: CLI -description: Validate manifests, inspect and roll back runs, decide approvals, test policy, sign evidence. +description: Set up a project, lint and test manifests, publish and apply them, inspect and roll back runs, decide approvals, simulate policy, sign evidence. --- ```bash npm install -g @multimindlab/reversi-cli +reversi --help ``` -**Storage options** (every command that reads runs or the ledger): `-d, --data ` (default `./reversi-data`), -`--store files|sqlite|postgres` (default `REVERSI_STORE` or `files`), `--database-url ` (default -`REVERSI_DATABASE_URL`). See [storage](/operations/storage). Commands marked **m** also need `-m, --manifests `. +Every command and option below comes from `reversi --help`; that output is the authority if a later version differs. -## Inspect and undo +## Shared options + +| Option | Meaning | +|---|---| +| `--config ` (global) | Project file. Default: `reversi.config.yaml` in this or a parent folder, or `REVERSI_CONFIG`. See [GitOps](/operations/gitops) | +| `-d, --data ` | Data directory. Default: the config file, `REVERSI_DATA`, or `./reversi-data` | +| `--store files\|sqlite\|postgres` | Default: the config file, `REVERSI_STORE`, or `files` | +| `--database-url ` | PostgreSQL URL. Default: `REVERSI_DATABASE_URL` | +| `-m, --manifests ` | A file, a folder, a registry folder or an https URL. Default: the config file or `REVERSI_MANIFESTS` | + +These four are accepted by every command that reads runs, approvals or the ledger (`-m` by most of them, not by `verify`, +`export`, `bundle` or `approvals`). See [storage](/operations/storage). `--json` gives machine-readable output where listed. + +## Exit codes + +| Code | Meaning | +|---|---| +| `0` | Success | +| `1` | Error, or a failed check (`verify`, `lint`, `test`, `doctor`, `manifests verify`, `verify-bundle`, `anchor verify`, a failed `anchor publish`) | +| `2` | Not fully done: `plan` says not fully reversible, `rollback` incomplete, `simulate` found newly blocked calls, or `apply --dry-run --detailed-exitcode` found changes | +| `64` | Usage error (unknown option or command) | + +## Set up and check | Command | Does | |---|---| +| `reversi init [dir] [--store files\|sqlite] [--force]` | Create `reversi.config.yaml`, `manifests/example.yaml`, `policy.yaml`, `tests/manifests.test.yaml`, a `.gitignore` entry and a data store (default `sqlite`). Never overwrites without `--force` | +| `reversi show-config` | Print the project configuration found, with paths resolved | +| `reversi doctor [--policy ] [--send-test] [--gateway-config ] [--upstream-cmd [--upstream-arg ]... \| --upstream-url ]` | Check Node version, config, manifests, lint, data store, ledger chain, signing key, policy and its tests, notification channels (the `notifications:` of the project file), writable data directory, and optionally an upstream. `--send-test` sends a test message through every channel. Exit 1 on failures | | `reversi validate ` | Check manifests, including that every compensation and required tool exists | -| `reversi verify [ledger]` | Verify the hash chain of a file, or of the ledger in the store. Exit 1 if broken | -| `reversi runs` **m** | List runs; flags unresolved steps | -| `reversi plan [--json]` **m** | Dry run of the rollback. Exit 2 if not fully reversible | -| `reversi rollback ` **m** | Undo. `--compensator-cmd ` (JSON call on stdin, exit 0 = undone), `--verify-cmd `, `--retries `, `--approve-compensations`, `--timeout-ms`, `--json`. Exit 2 if incomplete | -| `reversi resolve ` **m** | Settle a step whose outcome was unknown: `--status succeeded\|failed`, `--note ` (required), `--result `, `--actor` | +| `reversi lint [path] [--strict] [--json]` plus the upstream options of `doctor` | Find unsafe, incomplete or contradictory manifests (missing owner, missing data class, no timeout, ...). With an upstream, also compare them with its real tools. `--strict` exits 1 on warnings too | +| `reversi test [fixtures...] [--policy ]` | Run manifest contract tests and the policy's own tests, touching no system. Reports tools no test covers | +| `reversi suggest-manifest [-m ] [-o ] [--server ]` plus the upstream options | Connect to an MCP server and draft a manifest for every tool that has none. Prints the drafts, or writes one YAML file each to `-o`. Drafts need a person's approval | + + +`validate`, `lint`, `test`, `doctor`, `plan` and `suggest-manifest` accept a manifest file, a folder of files or a versioned registry folder (`index.json` plus `tools/`); for a registry they check the current non-draft version of each tool. + + +```text +$ reversi init proj && cd proj && reversi test +ok writing a note can be undone, and the undo is read back +ok a failed write leaves nothing to undo +ok policy: ordinary tools are allowed +ok policy: drop needs approval +not covered by any test: notes.restore +4 passed +``` + +```text +$ reversi lint --strict; echo $? +warning L011 notes.restore: mutating tool has no owner: nobody is named to ask when it misbehaves +warning L012 notes.restore: no risk.data_class: policy and retention cannot treat sensitive data differently +info L022 notes.restore: no timeout_ms: a hung call is only bounded by the gateway default +0 errors, 2 warnings, 1 notes +1 +``` + +## Manifest registry and policy versions + +A registry keeps every version of every manifest, with author, approver, checksum and an optional signature. The same idea +applies to the policy. See [GitOps](/operations/gitops) for the whole workflow. + +| Command | Does | +|---|---| +| `reversi manifests publish [--status draft\|approved] [--registry ] [--by ] [--key ] [--note ]` | Add a file or folder of manifests as new versions. Default status `draft`: someone else approves | +| `reversi manifests approve [--version ] [--four-eyes] ...` | Promote a draft to approved as a new, signed version. `--four-eyes`: the author may not approve their own draft | +| `reversi manifests deprecate [--replaced-by ] ...` | Keep the tool working and rolling back, but flag new calls | +| `reversi manifests rollback ...` | Publish an old version again as the newest | +| `reversi manifests list \| history \| diff \| check ` | Inspect versions. `check` equals `validate` | +| `reversi manifests verify [--trust ]` | Recompute every version's checksum, and with `--trust` check each is signed by that key. Exit 1 on failure | +| `reversi policy check ` / `reversi policy test [cases]` | Validate a policy file; run its inline `tests:` and an optional extra file of cases. Exit 1 on failure | +| `reversi policy publish [--dir ] [--by ] [--note ] [--no-activate]` | Validate, test, store as a new version and (by default) activate it | +| `reversi policy history \| diff \| activate [--dir ]` | Inspect versions; `activate` an older one to roll back | +| `reversi apply ...` | Make a registry (and policy) match a directory kept in git. Only differences are published; **nothing is deleted**. See below | + +`--registry` and `--dir` default to the `manifests:` and `policy:` settings of the project file. Policy versions are not signed; +manifest versions are when you pass `--key`. + +### `reversi apply ` + +```text +$ reversi apply state --dry-run --detailed-exitcode; echo "exit $?" ++ manifest notes.read ++ manifest notes.write ++ manifest notes.restore ++ policy policy +4 changes (dry run: nothing applied) +exit 2 +``` + +| Option | Meaning | +|---|---| +| `--registry `, `--policy-dir ` | Local target. Default `manifests:` and `policy:` from the config. The policy folder must already exist | +| `--server ` and `--token ` (or `REVERSI_TOKEN`) | Apply to a running server instead. The token must be an admin key to change anything; an operator key can read and dry-run | +| `--status approved\|draft` | Status of new versions. Default `approved`: the reviewed directory is the source of truth | +| `--prune` | **Deprecate** tools that are in the target but not in the directory. Never deletes | +| `--dry-run` | Show what would change, change nothing | +| `--detailed-exitcode` | Exit 2 when there are changes. It acts with `--dry-run` only, for CI drift checks | +| `--by`, `--key`, `--note` | Who is applying (default your user name), signing key for new versions, note on each version | + +Layout, local versus server mode and a CI example are in [GitOps](/operations/gitops). + +## Run the gateway and console + +| Command | Does | +|---|---| +| `reversi gateway [args...]` | Run the [MCP gateway](/gateway) with the same options as `reversi-gateway`. Arguments are passed through; `reversi gateway --help` prints the gateway's usage | +| `reversi console [-m] [-d] [--policy ] [--port ] [--role auditor\|approver\|operator]` | Open the [console](/console) on `http://localhost:/console` (default 7421) in open mode: no login, loopback only, never admin. Stop with Ctrl-C. Manifest and policy editing and rollback execution are not wired in this mode | + +## Inspect + +| Command | Does | +|---|---| +| `reversi runs list [--actor] [--tenant] [--tool] [--status needs_attention\|rolled_back\|complete\|blocked_only\|empty] [--mode enforce\|observe\|advise] [--since] [--until] [--text] [--limit ] [--json]` | List runs, newest last. `--since` takes `7d`, `12h`, `30m` or a date | +| `reversi runs show [--json]` | Steps, rollback outlook, approvals and the event trail | +| `reversi actions list [--run] [--tool] [--actor] [--status] [--since] [--limit] [--json]` | Individual calls across runs | +| `reversi actions show [--json]` | One call: state, ledger records, and with capture on its redacted arguments | +| `reversi search [--run] [--actor] [--tool] [--event a,b] [--text] [--since] [--until] [--limit] [--json]` | Search ledger records | +| `reversi plan [--json]` | Dry run of the rollback. Exit 2 if not fully reversible | | `reversi export [--run ]` | Print ledger records as JSON Lines | +| `reversi verify [ledger]` | Verify the hash chain of a JSON Lines file, or of the ledger in the store. Exit 1 if broken | + +```text +$ reversi runs list -d data -m registry +demo-1 actor=agent1 mode=enforce steps=1 rolled_back +``` -## Approvals +## Undo | Command | Does | |---|---| -| `reversi approvals list [--status pending]` | Show requests (shared with the gateway and server through the store) | -| `reversi approvals approve [--note] [--as ]` | Approve a pending request | -| `reversi approvals deny [--note] [--as ]` | Deny it | +| `reversi rollback [--compensator-cmd ] [--verify-cmd ] [--reader-cmd ] [--approve-compensations] [--retries ] [--timeout-ms ] [--json]` | Undo a run. The compensator gets the call as JSON on stdin; exit 0 means undone. `--verify-cmd` gets the same JSON afterwards; exit 0 confirms the undo. `--reader-cmd` runs a manifest's declared read-back: it gets `{"tool","arguments"}` as JSON on stdin and prints the result as JSON. Without it those steps are listed as "undone but not verified". Defaults: 1 attempt, 30000 ms. Exit 2 if incomplete | +| `reversi resolve --status succeeded\|failed --note [--result ] [--actor ]` | Settle a step whose outcome was unknown, after checking the downstream system. The note is required and recorded. Pass `--result` so a `succeeded` step can be rolled back | + +## Approvals, delegation, break-glass + +| Command | Does | +|---|---| +| `reversi approvals list [--status pending\|approved\|denied\|consumed\|expired]` | Show requests (shared with the gateway and server through the store) | +| `reversi approvals approve ` / `deny ` `[--note] [--as ] [--groups a,b] [--roles r,...]` | Decide. `--groups` and `--roles` say what you decide as, for requests routed to groups or roles (default role `operator`). The decision goes through the engine: it is journaled in the ledger, delegations apply and notification channels from the project file are told | +| `reversi delegate add --from --to --hours --reason [--group ]...` | Hand approval authority to someone for a while | +| `reversi delegate list` / `reversi delegate revoke ` | Inspect and end delegations | +| `reversi break-glass --reason --evidence [--by ]` | Emergency override of a pending approval. Reason of at least 10 characters and a ticket or incident reference; recorded as break-glass | Over `files` storage the CLI and a running server share `approvals.json`; for several machines use SQLite on a shared -volume or PostgreSQL. +volume or PostgreSQL. See [approvals](/concepts/approvals). -## Policy and kill switch +## Kill switches | Command | Does | |---|---| -| `reversi policy check ` | Validate a policy file | -| `reversi policy test [cases]` | Run inline `tests:` and an optional extra file of cases. Exit 1 on failure | -| `reversi kill-switch on\|off ` | Create or remove the file named by `REVERSI_KILL_FILE`. While it exists, new calls are blocked | +| `reversi halt add [--tool ] [--actor ] [--tenant ] [--environment ] [--by ]` | Block new calls everywhere, or only those matching the scope. Shared by every process on the same store | +| `reversi halt list` / `reversi halt lift [id]` | List halts; lift one, or all | +| `reversi kill-switch on\|off ` | Create or remove the file named by `REVERSI_KILL_FILE` (or `kill_switch_file:`). While it exists, every gateway and server using it blocks new calls | -## Evidence +## What-if and simulation + +`capture`, `simulate` and `replay` need [capture](/concepts/simulation) turned on (`capture: { enabled: true }` in the project file, or `REVERSI_CAPTURE=1` with optional `REVERSI_CAPTURE_TTL_DAYS`). `speculate` and `impact` do not. | Command | Does | |---|---| -| `reversi keygen ` | Write `.key` (private, mode 600) and `.pub`. Never overwrites | -| `reversi bundle --key --out [--run ]` | Signed evidence bundle | -| `reversi verify-bundle [--pub ]` | Verify a bundle; `--pub` checks against a key you trust | -| `reversi report --out ` **m** | Self-contained HTML report | +| `reversi capture list [--run] [--tool] [--since] [--json]` | Captured, redacted calls | +| `reversi capture purge [--run ]` | Delete expired captures, or all for a run | +| `reversi simulate --policy [--since 7d] [--run ] [--json]` | Replay captured calls through a candidate policy; report newly blocked, newly needing approval, newly allowed. Exit 2 if any call is newly blocked | +| `reversi replay --set ... [--policy ] [--json]` | Replay one captured run once as recorded and once with changes, e.g. `--set 1.amount=9000`, `--set tool:pay.charge/amount=9000`, `--set amount=9000` | +| `reversi speculate [--policy ] [--actor ] [--json]` | Compare alternative plans without running them: decisions, approvals, risk, residue, rollback | +| `reversi impact [--args ] [--policy ] [--actor ] [--json]` | What approving a call would do and what could be undone | + +```text +$ reversi impact notes.write --args '{"name":"a"}' +notes.write: reversible +decision now: allow +data class internal, blast radius low, owner you@example.com +rollback: undo by calling notes.restore +rollback: the undo is read back and verified +``` -## Exit codes +## Data lifecycle + +| Command | Does | +|---|---| +| `reversi retention plan [--days ]` | Which runs are due for deletion, and which are held | +| `reversi retention apply [--days ] [--dry-run] [--archive-dir ] [--key ]` | Delete run state, approvals and captures past retention. The ledger itself is never deleted. `--archive-dir` writes a signed bundle per run first | +| `reversi hold add [--run] [--tenant] [--actor] [--tool] --reason [--by ]` | Legal hold: stop retention deleting matching runs | +| `reversi hold list [--all]` / `reversi hold release ` | List holds (`--all` includes released); release one | -`0` ok, `1` error or failed check, `2` rollback or plan incomplete / not fully reversible, `64` usage error. +See [retention](/operations/retention). + +## Evidence + +| Command | Does | +|---|---| +| `reversi keygen ` | Write `.key` (private, mode 600) and `.pub`. Never overwrites | +| `reversi bundle --key --out [--run ]` | Signed evidence bundle of the ledger or one run | +| `reversi verify-bundle [--pub ]` | Verify a bundle. Without `--pub` the signature only proves the bundle is self-consistent | +| `reversi report [run_id] --out ` | With a run id: a self-contained report, HTML or PDF by extension (`--out` required). Without: the period report, `--since 7d`, `--format md\|html\|json` (or inferred from `--out`); printed to the terminal when `--out` is omitted | +| `reversi anchor publish [--file ] [--webhook ] [--secret ] [--key ]` | Write the current ledger head to the `anchor:` sinks in the project file, or to a file or webhook if you pass one. Exit 1 if a sink failed | +| `reversi anchor verify [--pub ]` | Check every anchor in a file against the ledger | ## The compensation call diff --git a/docs/concepts/approvals.mdx b/docs/concepts/approvals.mdx index 880f1dc..ac7eff3 100644 --- a/docs/concepts/approvals.mdx +++ b/docs/concepts/approvals.mdx @@ -1,6 +1,6 @@ --- title: Approvals -description: Human decisions bound to the exact call, used once, expiring, and visible to everyone who needs them. +description: Human decisions bound to the exact call, used once, expiring, routed to the right people, and recorded. --- A call needs approval when the manifest says `approval: always` (always true for irreversible tools), when a @@ -12,32 +12,170 @@ Compensations can also require approval (`compensation.requires_approval`). 1. The agent makes the call. Reversi queues an **approval request** and blocks the call. The message names the request id: `approval required: request apr_… is pending an operator decision; retry after it is approved`. 2. An approver decides it: in the [console](/console), with `reversi approvals approve `, over the - [HTTP API](/api-reference/admin), or through the gateway's operator endpoint. A webhook can tell them first. -3. The agent retries the **same call**. The approval is consumed, the call runs, and the ledger records `approved`. + [HTTP API](/api-reference/admin), through the gateway's operator endpoint, or from a Slack button. A + [notification](/operations/notifications) can tell them first. +3. The agent retries the **same call** with the same arguments. The approval is consumed, the call runs, and the ledger + records `approved`. + +```bash +reversi approvals list -d ./reversi-data +# apr_4442c0ff-… pending run=r1 call payments.refund args=[amount,order] why: policy `big-refund`: large refund; manifest requires approval +reversi approvals approve apr_4442c0ff-… -d ./reversi-data --as fiona --groups finance --note ok +``` ## What an approval is bound to - **The arguments.** A call approval carries the hash of the arguments it was requested for. Approve a refund of 5 and a retry for 500 is a different call and needs its own approval. -- **One use.** After the call runs the approval is `consumed`. -- **A time limit.** Pending and approved requests expire (default one hour). -- **A decision record.** Who decided, when, and the note, in the approval store **and in the ledger** (an `approved` record for approvals, a `blocked` record for denials, with the approver as actor). -- **Claimed atomically.** Consuming or deciding a request is a compare-and-swap in the store, so two processes sharing a database cannot both use one approval. -- **A denial sticks** for that call, with the reason shown to the agent. +- **One use.** After the call runs the approval is `consumed`. Running the same call again queues a new request. +- **A time limit.** Pending and approved requests expire (default one hour, `approval.ttl` in + [`reversi.config.yaml`](/cli), or the timeout set by the manifest or rule). +- **A decision record.** Who decided, when, the basis (role, group or delegation) and the note, in the approval store. + The ledger also gets a record, whichever way the decision is made (CLI, server, gateway or console): `approved` for + approvals, `blocked` for denials, with the approver as actor. +- **Claimed atomically.** Consuming or deciding a request is a compare-and-swap in the store, so two processes sharing a + database cannot both use one approval or both cast the last vote. +- **A denial sticks** for that call (same arguments, same step), with the reason shown to the agent. A request stores the names of the arguments, never their values, so approvals do not become a second copy of sensitive data. -## Pre-approval +`reversi approvals approve|deny` goes through the engine, like the server and console do: the decision is journaled in the +ledger (`approved`, or `blocked` for a denial, with the person given by `--as` as actor), delegations apply, and +[notification channels](/operations/notifications) from the project file are told (`approval_decided`). + +## Who may approve: routing and quorum + +By default any `approver` or above may decide. Name who must decide on the manifest or on a `require_approval` rule. + +```yaml tools.yaml +- tool: payments.refund + effect: reversible + permission: payments:write + approval: + roles: [finance] # Reversi roles or group names + quorum: 2 # distinct approvers needed, 1 to 10 + timeout: 15m + on_timeout: escalate # deny (default) or escalate + escalate_to: [admin] +``` + +```yaml policy.yaml +rules: + - id: big-refund + match: { tool: "payments.refund" } + when: [{ path: $args.amount, op: gt, value: 500 }] + action: require_approval + approvers: [finance] + quorum: 2 + approval_timeout: 30m + on_timeout: escalate + escalate_to: [admin] + reason: large refund +``` + +- **Roles and groups.** An approver qualifies if they hold a listed role, or belong to a listed group. Groups come from + `groups` on an [API key](/operations/authentication) or, for OIDC, from the values of the role claim. +- **Several requirements on one call** (manifest plus rules) are merged: any listed role may approve, the largest + quorum wins, the shortest timeout wins, and `deny` on timeout wins over `escalate`. +- **Quorum.** Each approver counts once. One denial ends the request. A quorum above 1 needs `roles`. While votes are + missing the request stays `pending` and the message says `needs 2 approvers from finance`. +- **No self-approval.** Whoever owns the run that asked cannot decide the request. +- **Deciding as a group member.** Over HTTP and in the console your key's groups are used. With the CLI you state them: + `--groups finance` and `--roles operator` (default `operator`). A person who is not in the group is refused: + `this request needs an approver from: finance`. The CLI trusts the caller, so use it on the host that owns the data. +- **The open console** acts as one role and has no groups, so it can decide only requests that name that role or no one. + +## Escalation on timeout + +With `on_timeout: escalate` an unanswered request does not simply expire. When its time is up it is widened once to +`escalate_to` (or left as is if none is named), given a fresh time limit, an `escalated` ledger record is written and an +`approval_escalated` notification is sent. If it expires again it is `expired`. The check happens when the queue is read, +so it is only as prompt as the alert interval, console views or API reads. + +With `on_timeout: deny` (the default) the request becomes `expired`. The agent's next retry is told +`approval … expired unanswered and was denied; the call must be requested again`, and a retry after that queues a new one. + +## Delegation + +An approver on leave can hand their authority to someone else for a while. + +```bash +reversi delegate add --from fred --to dan --hours 24 --reason "on leave" --group finance -m tools.yaml -d ./reversi-data +reversi delegate list -m tools.yaml -d ./reversi-data +reversi delegate revoke dlg_… -m tools.yaml -d ./reversi-data +``` + +- **Who and how long.** `--from`, `--to`, a reason (required), and an end time at most 90 days away. You cannot delegate to + yourself. +- **Scope.** `--group` (repeatable) limits it to requests routed to those groups. **Without it the delegation covers + every routed request**, so always pass `--group`. +- **What happens.** The delegate approves as themselves; the decision records `on_behalf_of` and the basis + `delegated by fred (finance)`. It counts as fred's vote: fred cannot vote on the same request afterwards, and the + delegate cannot vote twice. The delegate still needs the `approver` role to reach the approve endpoint. +- **Ends** at its expiry or when revoked. +- **Where.** The CLI above, the Approvals page of the console (own authority, hours and reason), or the + `/v1/delegations` endpoints, which `reversi-server` serves ([admin API](/api-reference/admin)). Delegations apply + everywhere a decision is made, including `reversi approvals approve --as `. -An operator can grant standing approval for a tool in a run (`POST /v1/runs/{id}/approvals`, `count` uses, any -arguments). Use it for compensations you know you will want to run, not for agent calls. +## Break-glass + +An admin can force a pending request through when the usual approvers are unreachable. It skips routing and quorum, and +it is loud on purpose. + +```bash +reversi break-glass apr_… --reason "payments outage, customer escalated" --evidence INC-77 --by root -m tools.yaml -d ./reversi-data +``` + +```bash +curl -X POST -H "Authorization: Bearer $ADMIN_KEY" -H 'content-type: application/json' \ + -d '{"reason":"customer is waiting on the line","evidence":"INC-42"}' \ + http://localhost:7421/v1/approvals/apr_…/break-glass +``` + +- **Admin only** over HTTP and in the console (other roles get `403`). A reason of at least 10 characters and a pointer to + evidence (ticket or incident) are required (`400` otherwise). The CLI has no login, so it accepts `--by` as given. +- **Recorded.** The approval keeps `break_glass` with who, why, the evidence reference and the time. The ledger gets an + `approved` record whose detail starts `BREAK-GLASS: … evidence: INC-77`. +- **Notified.** A critical `break_glass` notification goes out, and a critical `break_glass` [alert](/operations/monitoring) + shows in the console. + +## What the approver sees + +The Approvals page lists each pending request with why approval is needed (rule and manifest), who may decide, the +argument names and hash, the expiry and quorum progress. Each tool links to an **impact preview**, also at `GET /v1/impact`: +what the manifest and policy say about the tool, the verdict, the risk score and how the call could be undone. Previews +are about the tool, not the argument values, which are never stored. ## Notifications -Set `REVERSI_APPROVAL_WEBHOOK` to an http(s) URL. Each new request is POSTed as JSON with a `text` field, so a Slack or -Teams incoming-webhook URL works as is, plus the structured request. Set `REVERSI_CONSOLE_URL` to link the console in -the message. Delivery is best effort with a 5 s timeout and never changes the decision. +With [`notifications:`](/operations/notifications) configured, Reversi sends `approval_requested`, `approval_progress` +(a quorum vote that is not the last), `approval_decided`, `approval_escalated` and `break_glass`. A Slack channel with +`interactive: true` adds Approve and Deny buttons. + +The older single `REVERSI_APPROVAL_WEBHOOK` (or `approval.webhook`) still works: each new request is POSTed as JSON with a +`text` field, so a Slack or Teams incoming-webhook URL works as is. Set `REVERSI_CONSOLE_URL` to link the console. +Delivery is best effort with a 5 s timeout and never changes the decision. It goes through the +[egress rules](/operations/retention), like channels do; a refused URL is logged as `approval webhook failed`. + +### Approving from Slack + +The Slack buttons post to `/integrations/slack/actions`, which verifies Slack's signature (`v0`, five-minute window) and +maps the Slack user id to a Reversi person. `reversi-server` enables it when `REVERSI_SLACK_SIGNING_SECRET` and +`REVERSI_SLACK_USERS` (a JSON file mapping Slack user ids to name, roles and groups) are set; see +[authentication](/operations/authentication#slack-approvals-user-mapping). In code: + +```ts +createApp({ /* … */ slack: { signing_secret, users: { U123: { name: "fiona", roles: ["approver"], groups: ["finance"] } } } }); +``` + +An unmapped user, or a mapped user without the `approver` role or higher, gets "You are not set up as an approver in +Reversi." The same routing, quorum and self-approval rules apply. Checked here with locally signed requests, not with real Slack. + +## Pre-approval + +An operator can grant standing approval for a tool in a run (`POST /v1/runs/{id}/approvals`, `count` from 1 to 100, +any arguments). Use it for compensations you know you will want to run, not for agent calls. ## Who can decide @@ -46,4 +184,4 @@ The `approver` role and up. Agents cannot approve anything, including their own ## Storage Requests live in the data store (`approvals.json` for files, a table for SQLite and PostgreSQL), so the CLI, server -and gateway see the same queue and a restart does not lose it. +and gateway see the same queue and a restart does not lose it. Delegations live in the same store. diff --git a/docs/concepts/evidence.mdx b/docs/concepts/evidence.mdx index b222ea8..4be2cea 100644 --- a/docs/concepts/evidence.mdx +++ b/docs/concepts/evidence.mdx @@ -1,6 +1,6 @@ --- title: Evidence -description: Signed bundles, auditor reports and independent verification. +description: Signed bundles, anchored ledger heads, run and period reports, and independent verification. --- ## Signed bundles @@ -13,6 +13,7 @@ reversi keygen ./signing # writes signing.key (private) and reversi bundle -d ./reversi-data --key signing.key --out ledger.json reversi bundle -d ./reversi-data --key signing.key --out run.json --run run_123 reversi verify-bundle ledger.json --pub signing.pub +# ok: 9 records, hash chain verified, key 855d22c17ca4a717 (trusted) ``` `verify-bundle` recomputes every record hash, checks the chain where it applies and checks the signature. @@ -23,23 +24,101 @@ reversi verify-bundle ledger.json --pub signing.pub - **`--pub`** checks the signature against a key you already trust. Without it, a valid signature only proves the bundle is self-consistent: anyone could re-sign an edited bundle with their own key. -The server serves bundles (`GET /v1/runs/{id}/bundle`, `/v1/ledger/bundle`) when started with `REVERSI_SIGNING_KEY`, and -publishes the public key at `/v1/evidence/key`. Give auditors the public key out of band. +The server serves bundles (`GET /v1/runs/{id}/bundle`, `/v1/ledger/bundle`) when started with `REVERSI_SIGNING_KEY` +(or `signing_key:` in the project file), and publishes the public key at `/v1/evidence/key`. Give auditors the public key +out of band. The same key signs anchors and retention archives. ## Anchoring A hash chain detects edits, deletions and reordering. It cannot detect truncation of the tail or a rewrite of the whole -file by someone who controls it. Publish the ledger head (`reversi verify` prints it; every bundle contains it) somewhere -the operator cannot rewrite: a ticket, a signed commit, an external timestamp service. +file by someone who controls it. **Anchoring** publishes the ledger head somewhere the operator cannot quietly rewrite, so +that later you can prove the history is unchanged. -## Auditor report +### What is anchored + +One JSON line per anchor: the hash of the last record (`head`), how many records the ledger had (`records`), a timestamp, +the data `region` if set under [`egress`](/operations/retention), and, when a signing key is configured, the `key_id` and +an Ed25519 `signature`. + +```json +{"format":"reversi-anchor","version":1,"ts_ms":1792171387301,"head":"cdb044b2…","records":9,"key_id":"855d22c17ca4a717","signature":"LLHxL16j…"} +``` + +### Sinks and schedule + +```yaml reversi.config.yaml +signing_key: ./signing.key +anchor: + sinks: + - { type: file, path: /mnt/worm/anchors.jsonl } # append-only file on other storage + - { type: webhook, url: "https://audit.example.com/anchors", secret: "${ANCHOR_SECRET}" } + interval: 1h +``` + +- **File** sinks append one line per anchor. **Webhook** sinks POST the JSON; with `secret` the body is signed in the + `x-reversi-signature: sha256=` header, which you check like a [notification webhook](/operations/notifications). + Webhooks go through the [egress guard](/operations/retention) and time out after 10 s. +- The server and gateway anchor **once at start and then every `interval`** (default one hour). A failing sink is logged + and does not lose the anchor for the other sinks. +- On demand: `reversi anchor publish [--file ] [--webhook ] [--secret ] [--key ]` (it uses the sinks + in the config when none are given, and exits 1 if any sink failed), or `POST /v1/anchor` (admin). + +There is no scheduler for reports or bundles. Anchors are the scheduled evidence; run `reversi report` and +`reversi bundle` from cron if you want files on a timetable. + +### Verifying + +```bash +reversi anchor verify anchors.jsonl --pub signing.pub -d ./reversi-data +# 2026-10-16T17:23:07Z 9 records ok (signature trusted) +``` + +Every anchor in the file is checked against the ledger: the ledger itself must verify, the signature must match the key you +trust (with `--pub`; an unsigned anchor is then rejected), the ledger must still have at least that many records, and the +record at that position must still have the anchored hash. A shortened ledger fails with +`the ledger has 5 records but was anchored at 9: it has been truncated`; a rewritten one with +`history was rewritten`. Exit code 1 if any anchor fails. Keep the anchor file where the ledger's operator cannot edit it, +otherwise it proves nothing. + +## Run report ```bash reversi report run_123 -m tools.yaml -d ./reversi-data --out run_123.html +reversi report run_123 -m tools.yaml -d ./reversi-data --out run_123.pdf +``` + +The extension picks the format. Both hold who ran what, step statuses, the rollback outlook (what stands, what cannot be +undone and why) and the full event trail with record hashes. The PDF also states the data region and the ledger head at +generation. The PDF is a plain, dependency-free A4 document (monospaced text) that any viewer opens and text tools can read. +Over HTTP: `GET /v1/runs/{id}/report` (HTML) and `GET /v1/runs/{id}/report.pdf`; the [console](/console) links both. + +## Period report + +What the agents did over a window, for a weekly review or an auditor. + +```bash +reversi report --since 7d --out week.md # md (default), html or json by extension or --format +reversi report --since 2026-10-01 --format json +``` + +It covers calls, runs, outcomes, calls by tool, actor and reversibility class, what was **blocked** and what **would have +been** blocked in observe and advise modes, tools called without a manifest, approvals (requested, approved, denied, +expired, escalated, median time to decision), rollback results and verification rate, irreversible actions that ran, runs +needing attention, budgets near their limit, and calls per day. The server serves it at +`GET /v1/report?since=7d&format=md|html|json` (auditor and up). + +## Retention archives + +When [retention](/operations/retention) deletes a run's working data, it can first write a signed bundle of that run's +ledger records to an archive folder (`reversi retention apply --archive-dir ./arch --key signing.key`). A manifest with +`audit.evidence_export: required` makes that mandatory. The archive is an ordinary single-run bundle: + +```bash +reversi verify-bundle arch/r1.json --pub signing.pub +# ok: 9 records, record hashes verified (single run), key 855d22c17ca4a717 (trusted) ``` -A self-contained HTML page: who ran what, step statuses, the rollback outlook (what stands, what cannot be undone and -why) and the full event trail. The server and [console](/console) serve the same report. +The ledger itself is never purged; the deletion is recorded in it as a `purged` event. ## Offline verification without Node diff --git a/docs/concepts/manifests.mdx b/docs/concepts/manifests.mdx index 6068605..5583159 100644 --- a/docs/concepts/manifests.mdx +++ b/docs/concepts/manifests.mdx @@ -1,19 +1,87 @@ --- title: Tool manifests -description: How a tool declares its reversibility, its undo, and the permission it needs. +description: How a tool declares its reversibility, its undo, who owns it and who must approve it, plus the registry, lint and contract tests that keep manifests honest. --- A manifest is one JSON or YAML object per tool (or an array of them). The machine-readable schema is [`schemas/manifest.schema.json`](https://github.com/multimindlab/reversi/blob/main/schemas/manifest.schema.json). +Unknown fields are rejected at every level, so a typo is an error, not a silently ignored setting. + +```yaml manifests/payments.yaml +- tool: stripe.get_charge + description: Read one charge. + effect: read_only + permission: payments:read + +- tool: stripe.create_charge + description: Charge a card. + effect: compensable + permission: payments:write + owner: payments-team@example.com + oncall: payments-oncall + risk: { data_class: financial, blast_radius: high, environment: prod } + timeout_ms: 30s + idempotency: { key: "charge-{{args.order_id}}", scope: global } + audit: + retention_days: 2555 + evidence_export: required + redaction: [$args.card_token] + approval: + roles: [finance] + quorum: 2 + timeout: 15m + on_timeout: escalate + escalate_to: [cfo] + compensation: + tool: stripe.refund + kind: compensation + residue: [the charge still appears on the customer's statement] + requires_approval: false + timeout_ms: 20s + argument_map: + charge_id: $result.id + amount: $args.amount + verification: + tool: stripe.get_charge + arguments: { charge_id: $result.id } + expect: { refunded: true } + retries: 3 + backoff_ms: 2s + +- tool: stripe.refund + description: Refund a charge. + effect: idempotent + permission: payments:write + retry: { attempts: 3, base_ms: 200ms, max_ms: 5s } +``` + +`reversi validate manifests` accepts this file (`ok: 3 tool manifests valid`). + +## Fields | Field | Meaning | |---|---| -| `tool` | Tool name | -| `effect` | `read_only`, `idempotent`, `reversible`, `compensable`, `irreversible` | -| `permission` | Permission the run's scope must contain | -| `approval` | `never` (default) or `always`. Forced to `always` for irreversible tools | +| `tool` | Tool name: letters, digits and `_ . : / -`, up to 128 characters. Required | +| `effect` | `read_only`, `idempotent`, `reversible`, `compensable`, `irreversible`. Required | +| `permission` | Permission the run's scope must contain. No whitespace. Required | +| `description` | Free text. Reviewers read it; lint warns when a mutating tool has none | +| `approval` | `never` (default), `always`, or an object (see [Approval](#approval)). Forced to `always` for irreversible tools | | `compensation` | Required for `reversible` and `compensable`; forbidden otherwise | -| `requires` | Tools that must have succeeded earlier in the run. Also creates rollback ordering | +| `requires` | Tools that must have succeeded earlier in the run. Also creates rollback ordering. Entries may be a name or `{ tool, status: succeeded }` | +| `version` | Positive integer, default 1. A registry increments it on every publish | +| `status` | `draft`, `approved` (default) or `deprecated` | +| `replaced_by` | For a deprecated tool, the tool to use instead. Must be registered | +| `owner`, `oncall` | Who to ask when it misbehaves. Free text | +| `risk` | `data_class` (free text such as `pii`, `financial`), `blast_radius` (`low`, `medium`, `high`, `critical`), `environment` (recorded only) | +| `timeout_ms` | Longest the gateway waits for the upstream. A number of milliseconds or a duration | +| `retry` | `attempts` (1 to 10, default 3), `base_ms` (default 200), `max_ms` (default 5000). Only for `read_only` and `idempotent` tools | +| `idempotency` | `key` template and `scope` (see [Duplicate calls](#duplicate-calls)) | +| `audit` | `retention_days`, `evidence_export` (`required` or `optional`), `redaction` (paths) | +| `schema_hash` | Hash of the upstream tool's input schema when the manifest was written (see [Schema drift](#schema-drift)) | + +Durations are a number of milliseconds or a string with a unit: `500ms`, `30s`, `15m`, `2h`, `1d`. That applies to +`timeout_ms`, `retry.base_ms`, `retry.max_ms`, `approval.timeout`, `compensation.timeout_ms` and +`verification.backoff_ms`. ## Effects @@ -23,31 +91,249 @@ A manifest is one JSON or YAML object per tool (or an array of them). The machin - **compensable**: a counteraction exists but leaves residue (a refund still appears on a statement). - **irreversible**: no safe undo (an email). Always needs approval; rollback reports it as not undone. -## Compensation arguments +## Compensation -`argument_map` builds the undo call's arguments from references: +| Field | Meaning | +|---|---| +| `tool` | The tool that undoes this one. Not the tool itself | +| `argument_map` | Builds the undo call's arguments from references (below) | +| `kind` | `compensation` (default): makes amends, may leave residue. `inverse`: restores the exact prior state | +| `residue` | What the undo cannot take back, in plain words. An `inverse` cannot declare residue | +| `requires_approval` | A human must approve running this compensation. Default false | +| `timeout_ms` | Recorded on the step for the party that runs the undo | +| `verification` | Read back the result of the undo before it counts as done (below) | + +`argument_map` values are references: - `$args.path`: from the original call's arguments - `$result.path`: from what the tool returned Arguments are **not stored in the ledger**. Reversi keeps only the values named by `argument_map`, in run state, so it can build the undo later. If a referenced value is missing when the tool returns, the step is marked -*unrecoverable* rather than guessed. +*unrecoverable* rather than guessed. The plan reports `kind` and `residue` per step, so a reviewer sees what a rollback +will leave behind before running it. + +### Verification + +An API can answer "ok" and do nothing. `verification` calls a read-only tool after the undo and compares the answer: + +- `tool`: must be registered with effect `read_only` +- `arguments`: references, resolved from the original call and result +- `expect`: dotted path to expected value, deep equal. At least one entry. A path into an array uses the index (`items.0.id`) +- `retries` (1 to 10, default 3) and `backoff_ms` (default 1000, doubled between attempts), because undo is often eventually consistent + +If every attempt mismatches, the compensation fails with the first mismatch in the ledger. See +[Rollback](/concepts/rollback#verification) for what happens when nothing can run the read-back. + +## Approval + +`approval: always` makes a human approve each call. The object form says who and how many: ```yaml -- tool: charge_card - effect: compensable - permission: payments:write - requires: [create_order] - compensation: - tool: refund - requires_approval: false - argument_map: - charge_id: $result.charge_id - amount: $args.amount +approval: + roles: [finance] # an approver must hold one of these roles or groups + quorum: 2 # 1 to 10; a quorum above 1 needs roles + timeout: 15m + on_timeout: escalate # deny (default) or escalate + escalate_to: [cfo] ``` +`required: false` in the object form means `never`. Internally the manifest keeps `approval: always` plus the rules. +Rules from the manifest and from [policy](/concepts/policy#approvers-and-quorum) are merged for one call: any listed role +may approve, the largest quorum wins, the shortest timeout wins. See [approvals](/concepts/approvals). + `approval` is deliberately separate from `effect`. A reversible tool can still need a human (large amounts), and irreversible tools always do. + +## Duplicate calls + +```yaml +idempotency: { key: "charge-{{args.order_id}}", scope: global } +``` + +The key is a template over call arguments and needs at least one `{{args.field}}`. Two calls of the same tool with the +same rendered key are duplicates: the second is refused while the first still stands (in flight, succeeded, uncertain or +being compensated). `scope` is `run` (default), `actor` or `global`. If an argument is missing or not a plain value the +call is refused rather than guessed. Once the first call is undone or failed, the key is free again. + +This is separate from the idempotency key Reversi gives each compensation, which stays the same across retries so the +downstream API can deduplicate. + +## Lifecycle + +| Status | Behavior | +|---|---| +| `draft` | Not enforceable: calls are blocked (`the manifest ... is a draft`). A registry skips drafts when it loads | +| `approved` | Enforced. The default when `status` is omitted | +| `deprecated` | Still runs and still rolls back. Each call is noted `manifest deprecated; use ` | + +## What the validator enforces + +- `reversible` and `compensable` tools declare a compensation; the other effects must not. +- `irreversible` tools cannot set `approval: never`. +- A tool cannot require itself or compensate itself. `requires` has no duplicates. +- Every `argument_map`, `verification.arguments` and `audit.redaction` entry is a reference such as `$args.id` or `$result.charge_id`. +- `kind: inverse` cannot list residue. +- `retry` is only allowed on `read_only` and `idempotent` tools: retrying anything else could repeat its effect. +- `verification.expect` is not empty. `approval.quorum` is 1 to 10. +- Once all manifests are loaded: the compensation tool, every required tool, the verification tool and `replaced_by` must exist, and the verification tool must be `read_only`. A duplicate tool name is an error. + +## Schema drift + +`schema_hash` is a short hash of the upstream tool's input schema. When the gateway sees a different hash it adds a +violation (`the tool's input schema changed since its manifest was approved: review the manifest again`), so in enforce +mode the call is blocked until someone reviews and re-approves the manifest. `reversi suggest-manifest` fills it in. + +## Registry: versions, signing, trust + +A registry is a folder of immutable versions. Every publish makes a new version; a draft becomes enforceable only when +someone approves it, and the approval is itself a new version. + +```bash +reversi keygen keys/mf # keys/mf.key (private) and keys/mf.pub +reversi manifests publish manifests --registry reg --by alice --key keys/mf.key --note "first draft" +# stripe.create_charge v1 draft 40301964dd00 +reversi manifests approve stripe.create_charge --registry reg --by bob --key keys/mf.key --four-eyes +# stripe.create_charge v2 approved by bob +reversi manifests history stripe.create_charge --registry reg +# v1 draft ... by alice 40301964dd00 signed - first draft +# v2 approved ... by alice, approved by bob 5804deda419f signed - approved from v1 +``` + +| Command | Does | +|---|---| +| `reversi manifests publish ` | Add a file or folder as new versions. `--status draft` (default) or `approved` | +| `reversi manifests approve ` | Promote a draft. `--version `, `--four-eyes` (the author may not approve their own draft) | +| `reversi manifests deprecate ` | Needs an approved version. `--replaced-by ` | +| `reversi manifests rollback ` | Publish an old version again as the newest, history intact | +| `reversi manifests history `, `diff `, `list` | Inspect | +| `reversi manifests verify [--trust ]` | Check every checksum, and every signature against a public key. Exit 1 on a problem | +| `reversi manifests check ` | Validate plain manifest files (same as `reversi validate`) | + +All writing commands take `--registry ` (default: `manifests:` in `reversi.config.yaml`), `--by`, `--key` and `--note`. + +- **Checksums.** Each version's checksum is in `index.json` and is checked on load. A file edited outside the workflow is refused. +- **Signing.** With `--key`, each version carries an Ed25519 signature over `reversi-manifest:@:` and the key id. +- **Trusted key.** Set `manifest_public_key` in the project config (or `REVERSI_MANIFEST_PUBLIC_KEY`) to a PEM public key. Every version must then be signed by it, or the registry does not load. Setting it for manifests that are not in a registry is an error. +- **Remote registry.** `manifests:` may be an `https://` URL serving `index.json` and `tools//v.json` as static files. The newest non-draft version of each tool is fetched, its checksum (and signature, with a trusted key) is checked, and the project's `egress:` rules apply (see the [gateway](/gateway) page). Tested against a local static server; not run against a hosted one. +- **HTTP.** A server with a registry folder exposes `GET /v1/registry` and `POST /v1/registry/manifests`, `.../{tool}/approve`, `.../{tool}/deprecate`. Drafts need the operator role; approving or publishing as approved needs admin. A change reloads the engine. + +`approve`, `deprecate` and `rollback` keep a manifest's `approval:` object (roles, quorum, timeout, `escalate_to`) as you wrote it, and so do +`reversi apply` and the console editor; a registry holding such a manifest loads everywhere. Two things to know: + +- `manifests publish --status` overrides a `status:` in the file: the flag (default `draft`) decides. +- A draft is not enforced and is simply absent from the loaded registry, so a call to a tool that has only a draft reports + "no manifest", not "is a draft". + +## Lint + +```bash +reversi lint manifests +# warning L011 stripe.refund: mutating tool has no owner: nobody is named to ask when it misbehaves +# warning L012 stripe.refund: no risk.data_class: policy and retention cannot treat sensitive data differently +# info L022 stripe.refund: no timeout_ms: a hung call is only bounded by the gateway default +# 0 errors, 2 warnings, 1 notes +``` + +Lint finds manifests that are valid but unsafe, incomplete or contradictory. It exits 1 on errors, and on warnings +with `--strict`. `--json` prints the findings. With `--upstream-cmd`, `--upstream-url` or `--gateway-config` it also checks the +manifests against the real tools' input schemas. + +| Code | Level | Finding | +|---|---|---| +| L010 | warning | Mutating tool has no description | +| L011 | warning | Mutating tool has no owner | +| L012 | warning | No `risk.data_class` | +| L013 | warning | Compensable but no `residue` declared | +| L014 | warning | The compensation tool is not idempotent, but rollback retries it | +| L015 | warning | Irreversible with no `blast_radius` and no approvers | +| L016 | info | The undo is not verified | +| L017 | error | `argument_map` or verification reads an argument the input schema lacks (with an upstream) | +| L019 | warning | A read-only tool shares a permission with tools that change things | +| L020 | error | The idempotency key uses an argument the input schema lacks (with an upstream) | +| L021 | info | Irreversible without `audit.evidence_export: required` | +| L022 | info | Mutating tool with no `timeout_ms` | +| L023 | error | `requires` forms a cycle | +| L024 | warning | Draft manifest | +| L027 | info | No `schema_hash` (with an upstream) | +| L030 | warning | Declared reversible but the compensation lists residue | +| L031 | warning | Declared compensable but the compensation is an exact `inverse` | +| L032 | info | Compensation takes no arguments from the call | +| L033 | warning | A redaction path names an argument the input schema lacks (with an upstream) | +| L034 | warning | No upstream tool has this name (with an upstream) | +| L035 | warning | The upstream offers a tool with no manifest | + +## Contract tests + +`reversi test` runs manifest contract tests without touching any system: for a fake call and a fake result, does the +manifest produce the undo arguments, verification and plan you expect? It reads every `*.yaml`, `*.test.yaml` or `*.json` +under `tests/` next to the config, or the files you pass, and also runs the policy's own tests. + +```yaml tests/payments.test.yaml +- name: a charge is refunded with the right arguments, and the read-back is checked + tool: stripe.create_charge + args: { order_id: o1, amount: 2500 } + result: { id: ch_1 } + expect_plan: compensate + expect_handle: { charge_id: ch_1, amount: 2500 } + expect_verification: { charge_id: ch_1 } + verify_with: { refunded: true } + expect_verified: true +- name: a failed charge leaves nothing to undo + tool: stripe.create_charge + args: { order_id: o1, amount: 2500 } + fail: card declined +- name: the same order twice is refused + tool: stripe.create_charge + args: { order_id: o1, amount: 2500 } + result: { id: ch_1 } + expect_duplicate: true +``` + +```bash +reversi test tests/payments.test.yaml -m manifests +# ok a charge is refunded with the right arguments, and the read-back is checked +# ok a failed charge leaves nothing to undo +# ok the same order twice is refused +# not covered by any test: stripe.refund +# 3 passed +``` + +Case fields: `tool`, `args`, `result` (the fake success), `fail` (a message: the call fails), `after` (tools to run first; +default is the manifest's `requires`), `expect_plan` (`compensate`, `skip` or `unrecoverable`), `expect_handle` +(undo arguments), `expect_verification` (verification arguments), `verify_with` plus `expect_verified` (would this read-back +be accepted), `expect_duplicate`, and `expect_blocked` (a substring of the reason the call must be refused for). Exit 1 on any +failure. Tools without a test are listed as not covered. + +## Draft manifests from a running server + +```bash +reversi suggest-manifest --upstream-cmd node --upstream-arg connectors/dist/main.js \ + --upstream-arg serve --upstream-arg fs --upstream-arg --root --upstream-arg ./sandbox -o drafts +# drafts/fs.write_file.yaml medium +# ... +# 9 drafted, 0 skipped (already have manifests) +``` + +It connects to an MCP server and drafts a manifest for each tool that has none, from names, descriptions, input schemas and the +server's annotations (`readOnlyHint`, `destructiveHint`, `idempotentHint`) and the `mcp-compensate` convention. +Every draft has `status: draft`, a confidence (`low` or `medium` in the run above), comments saying what was guessed, a `schema_hash`, and a +`TODO` residue where the undo needs describing. Annotations are the server's own claims, so a draft grants nothing until a +person reviews and approves it. Over the gateway the same thing is `reversi.suggest_manifest`. + +## Python and JSON Schema + +The JSON Schema and the Python SDK mirror the TypeScript validator. `reversi_sdk.validate_manifest` returns the normalized +manifest and a list of errors, and is tested against the TypeScript implementation on shared fixtures: + +```python +from reversi_sdk import validate_manifest +manifest, errors = validate_manifest({"tool": "stripe.refund", "effect": "idempotent", + "permission": "payments:write", "timeout_ms": "30s"}) +# manifest["timeout_ms"] == 30000, errors == [] +``` + +Optional pydantic v2 models (`reversi_sdk.pydantic_models.ToolManifest`) run the same validator first, so they cannot disagree +with it. Manifests are read from JSON in Python; convert YAML first. diff --git a/docs/concepts/policy.mdx b/docs/concepts/policy.mdx index 9985626..3635eb6 100644 --- a/docs/concepts/policy.mdx +++ b/docs/concepts/policy.mdx @@ -1,6 +1,6 @@ --- title: Policy as code -description: Rules, limits and tests that add restrictions on top of manifests and scopes. +description: Rules, limits, budgets, freezes, time windows, risk scoring, approvers, versions and tests that add restrictions on top of manifests and scopes. --- A policy file adds rules to what manifests and scopes already enforce. It can **deny** a call, **require @@ -26,47 +26,263 @@ limits: match: { effect: [reversible, compensable] } max_calls: 50 window_seconds: 60 - per: run # run | actor | global + per: run # run | actor | tenant | global tests: - name: small refund is fine input: { tool: stripe.create_refund, environment: prod, args: { amount: 100 } } expect: allow ``` +The top-level keys are `version`, `default`, `rules`, `limits`, `budgets`, `freezes`, `risk`, `retention`, `content` and +`tests`. Anything else is an error. Rule and limit ids must be unique. + ## Matching -`match` fields (all optional, all must hold): `tool`, `permission`, `actor`, `environment` (globs: `*` any -run of characters, `?` one), and `effect` (an exact effect class or a list). `when` conditions then look at the call -arguments with `$args.path.to.value`: +`match` fields (all optional, all must hold): `tool`, `permission`, `actor`, `environment`, `tenant`, `data_class`, +`blast_radius` (globs: `*` any run of characters, `?` one; each may be a list), and `effect` (an exact effect class or a +list). `data_class` and `blast_radius` come from the tool's manifest `risk`; a call to a tool without them does not match a +rule that names them. + +`when` conditions then look at the call arguments with `$args.path.to.value`: `eq`, `ne`, `gt`, `gte`, `lt`, `lte`, `in`, `not_in`, `matches` (regular expression), `exists`, `contains`. -A comparison on a missing or mistyped value is simply false, so a rule never fires by accident on bad input. +Numbers and numeric strings both count as numbers, so an upstream that coerces `"5000"` cannot slip under `gt 1000`. A +comparison on a **missing** value is false. For `deny` and `require_approval` rules a value that is present but of the wrong +type for the operator (text under `gt`, a number under `matches`) counts as a match, so a restriction cannot be dodged by +sending the wrong type. `allow` rules do not do this. + +### Tenants + +`tenant` matches the tenant of the run. The server sets it from the `tenant` on the API key that opens the run. Runs created by +the gateway carry no tenant, so a rule with `tenant:` never matches there. + +```yaml +rules: + - id: pii-tools + match: { data_class: [pii, financial], tenant: "acme-*" } + action: require_approval +``` ## Precedence -`deny` beats `require_approval` beats `allow`. Every matching rule is evaluated and reported. Limits are sliding -windows counted per run, per actor or globally; only calls that actually started count. +`deny` beats `require_approval` beats `allow`. Every matching rule is evaluated and reported. With `default: deny`, a call +that no `allow` rule matches is denied (`no allow rule matches this call (default deny)`). + +## Limits + +Sliding windows counted `per` run, actor, tenant or globally. Only calls that actually started count: a call that was +blocked or is waiting for approval does not use up the window. Counters live in the data store, so every process sharing it +shares the counts. + +## Budgets + +A limit counts calls. A budget can also add up an argument: + +```yaml +budgets: + - id: daily-charge-total + match: { tool: stripe.create_charge } + metric: sum # calls | sum + path: $args.amount # required for sum + max: 1000000 + window_seconds: 86400 + per: tenant # run | actor | tenant | global (default run) + action: require_approval # deny (default) | require_approval + warn_at: 0.8 # default 0.8 +``` + +A call that would take the total above `max` is denied or needs approval, and the reason says how much was used. Between +`warn_at` and `max` the call goes through and the ledger notes `budget daily-charge-total at 85% (...)`. A call whose +argument is not a number adds 0. -## Environments +## Time windows -`environment` comes from `REVERSI_ENV` (or `--environment`). Use one policy file with rules per environment, so -production can be stricter than staging. +A rule applies only inside its window: + +```yaml +rules: + - id: weekend-writes + match: { effect: [reversible, compensable, irreversible], environment: prod } + window: { days: [sat, sun], timezone: Europe/London } + action: deny + reason: weekend change restriction + - id: out-of-hours + match: { environment: prod } + window: { days: [mon, tue, wed, thu, fri], between: "09:00-17:00", timezone: America/New_York, outside: true } + action: require_approval +``` + +`days` is any of `sun`..`sat` (default every day), `between` is `HH:MM-HH:MM` and may wrap midnight (`22:00-06:00`), `timezone` +is an IANA name (default `UTC`), and `outside: true` inverts the window. Checked with a fixed clock, the first rule denies on +a Saturday and allows on a Friday; the second asks for approval at 19:00 New York time and not at 11:00. + +## Freezes + +```yaml +freezes: + - id: quarter-end + from: "2026-12-30T00:00:00Z" + to: "2027-01-02T00:00:00Z" + reason: quarter-end freeze + match: { environment: prod } # optional: tool, effect, environment +``` + +During the period a matching call is denied: `freeze quarter-end: quarter-end freeze (until 2027-01-02T00:00:00Z)`. Without +`match`, every call that is not `read_only` is frozen, which includes tools with no manifest. `to` must be after `from`. + +## Risk scoring + +A transparent 0 to 100 score from the call's effect, the manifest's `blast_radius` and `data_class`, the environment, an +argument and the actor. Every factor is listed, so a reviewer sees why a call scored what it did. Add a `risk` section to turn +it on: + +```yaml +risk: + weights: + effect: { read_only: 0, idempotent: 5, reversible: 15, compensable: 35, irreversible: 60 } + amount: + path: $args.amount + bands: + - { over: 10000, points: 10 } + - { over: 100000, points: 25 } # the highest band that is exceeded applies + actors: { new-agent: 20 } + thresholds: { approve: 50, deny: 90 } +``` + +At or above `approve` the call needs approval; at or above `deny` it is refused. `approve` cannot exceed `deny`. The score is +clamped to 0..100 and recorded with the step and in the intent record: +`risk 60 (effect compensable +35, amount over 100000 +25)`. A `$args.amount` call of 200000 by `new-agent` in `prod` scores +`effect compensable +35, environment prod +10, amount over 100000 +25, actor new-agent +20` = 90, which is denied. + +Default weights, used for anything you do not override: + +| Factor | Points | +|---|---| +| effect | `read_only` 0, `idempotent` 5, `reversible` 15, `compensable` 30, `irreversible` 60 | +| blast radius | `low` 0, `medium` 10, `high` 20, `critical` 35 | +| data class | `public` 0, `internal` 5, `pii` 15, `financial` 20, `phi` 25, `secret` 25 | +| environment | `prod` 10, `production` 10 | +| no manifest | 40 (`weights.unclassified`) | + + +A map you give under `weights` replaces the default map for that key. If you set only `effect: { compensable: 35 }`, +`irreversible` scores 0. List every class you care about, as above. + + +## Approvers and quorum + +A `require_approval` rule can say who may approve and how many must: + +```yaml +rules: + - id: big-charges + match: { tool: "stripe.create_charge", environment: prod } + when: [{ path: $args.amount, op: gt, value: 50000 }] + action: require_approval + reason: large charge + approvers: [finance] # roles or groups; empty means any approver + quorum: 2 # 1 to 10 + approval_timeout: 15m + on_timeout: escalate # deny (default) | escalate + escalate_to: [cfo] +``` + +`approvers`, `quorum` and `approval_timeout` are only valid on `require_approval` rules. When several rules and the manifest's +own [approval settings](/concepts/manifests#approval) apply to one call, they are merged: any listed role may approve, the +largest quorum wins, the shortest timeout wins. See [approvals](/concepts/approvals). + +## Retention and content + +Two more sections live in the same file. + +```yaml +retention: + default_days: 90 + rules: + - { match: { data_class: financial }, days: 2555 } # match: tool, tenant, environment, data_class +content: + secrets: deny + pii: require_approval +``` + +`retention` says how long run state, approval requests and captured arguments are kept; a run is kept for the longest of the +default, the matching rules and each tool's manifest `audit.retention_days`. The ledger itself is never purged. See +[retention](/operations/retention). `content` configures the checks in [content checks](/concepts/content-checks) +(`secrets`, `encoded_payloads`, `injection`, `pii`: `flag`, `require_approval` or `deny`; plus `cumulative` and +`max_string_bytes`). + +## Modes + +Policy decisions are the same in every mode; the mode decides what happens to a violation. + +| Mode | Violation | +|---|---| +| `enforce` (default) | The call is refused and a `blocked` record names the rule | +| `observe` | Journaled as `would_block`, and the call proceeds | +| `advise` | Like `observe`, and the agent's result gets an extra text block: `Reversi advisory (nothing was blocked): ...` (gateway) | + +The [gateway](/gateway) sets the mode with `--mode` or `mode:` in its config. Use `observe` or `advise` first, read the +`would_block` records or `reversi report` (with no run id, the period report), then switch to `enforce`. The [gateway](/gateway) turns a block into an MCP +error result the agent can read. + +## More than one policy source + +The engine merges the policy file with the gateway's `allow` and `deny` lists, which it compiles into rules +(`gateway-deny-0`, `gateway-allow`). Rules, limits, budgets, freezes and tests are concatenated, so ids must stay unique +(a clash is an error). If any source says `default: deny`, the result is `deny`: a gateway `allow` list therefore makes the +merged policy an allowlist. `risk`, `retention` and `content` come from the first source that has one, which is the policy file. ## Test it ```bash reversi policy check policy.yaml +# ok: 3 rules, 1 limits, default allow, content checks configured reversi policy test policy.yaml [more-cases.yaml] +# ok small charge in prod is allowed +# ok big charge needs approval +# 2 passed ``` -`policy test` runs the inline `tests:` and any extra file of cases, and exits 1 on a failure. Put it in CI. -A case may name the `rule` that must have fired. +`policy test` runs the inline `tests:` and any extra file of cases, and exits 1 on a failure. Put it in CI. A case may name the +`rule` that must have fired, which also covers limits, budgets and freezes by id. `input` takes `tool`, `effect`, `permission`, +`actor`, `environment`, `tenant`, `data_class`, `blast_radius` and `args`; the defaults are actor `agent`, environment `dev` and +empty `args`. Each case starts with fresh counters, so limits and budgets are tested on a single call, and windows and +freezes use the real clock. An extra file is a list of the same cases: + +```yaml more-cases.yaml +- name: one charge over the daily budget asks for approval + input: { tool: stripe.create_charge, effect: compensable, environment: dev, args: { amount: 2000000 } } + expect: require_approval + rule: daily-charge-total +- name: acme financial data needs approval + input: { tool: crm.export, effect: reversible, environment: dev, tenant: acme-eu, data_class: financial } + expect: require_approval + rule: pii-tools +``` -## What a block looks like +## Versioned policies -In enforce mode a violation journals a `blocked` record naming the rule, and the call is refused. In observe mode it -is journaled as `would_block` and the call proceeds. The [gateway](/gateway) turns a block into an MCP error result -the agent can read. +Point `policy:` at a folder and each change becomes a version with an author, a note, a checksum and a record of how many of its +own tests passed. Publishing validates the policy and runs its tests; a policy whose tests fail is refused. + +```bash +reversi policy publish policy.yaml --dir pol --by alice --note "initial" +# policy v1 2 tests passed active ac3740b71b29 +reversi policy publish policy2.yaml --dir pol --by alice --note "lower threshold" --no-activate +# policy v2 2 tests passed 66d6ed8eb871 +reversi policy history --dir pol +# v1 *active* ... by alice 2 tests ac3740b71b29 - initial +# v2 ... by alice 2 tests 66d6ed8eb871 - lower threshold +reversi policy diff 1 2 --dir pol +# - when: [{ path: $args.amount, op: gt, value: 50000 }] +# + when: [{ path: $args.amount, op: gt, value: 20000 }] +reversi policy activate 2 --dir pol # also how you roll back: activate an older version +# policy v2 is now active +``` -Combining files: the gateway compiles its `allow` and `deny` lists into rules and merges them with your policy file. +`--dir` defaults to `policy:` in `reversi.config.yaml`. The engine loads the active version, and a running engine picks up a new one on reload +(`POST /v1/reload`; the server and gateway also watch the files unless `reload: { watch: false }` is set). Each version's file is checked against its checksum when read, so a policy edited outside the history is refused. Over +HTTP an admin can `POST /v1/registry/policy` (`text`, `note`, `activate`) and `POST /v1/registry/policy/{version}/activate`. +The [console](/console) policy page validates, runs the tests and can +[simulate the draft against captured history](/concepts/simulation#policy-simulation) before you publish. diff --git a/docs/concepts/rollback.mdx b/docs/concepts/rollback.mdx index b5ab435..9abd9ab 100644 --- a/docs/concepts/rollback.mdx +++ b/docs/concepts/rollback.mdx @@ -1,6 +1,6 @@ --- title: Rollback -description: Saga rollback with dependency ordering, retries, verification and honest reporting. +description: Saga rollback with dependency ordering, retries, verification, residue reporting and honest reporting. --- Rollback undoes a run's standing steps with their declared compensations: **newest first**, one at a time, journaling @@ -12,39 +12,154 @@ Rollback undoes a run's standing steps with their declared compensations: **newe `requires` and from `depends_on` on a call. A stuck step is reported as `deferred`; everything independent is still undone. - Irreversible steps are reported as **cannot be undone**. Reversi never claims otherwise. - Steps whose outcome was never reported (`intended`, `uncertain`) are **unresolved**: Reversi will not guess. See - [reconciliation](#reconciliation). + [uncertain steps](#uncertain-steps). - Calls with no manifest (observe mode only) are reported as `unmanifested`. +- A step whose compensation arguments could not be built (a `$result` value was missing) is `unrecoverable`. - Re-running a rollback is safe: compensated steps are skipped, failed ones retried with the **same idempotency key**, which you pass to the downstream API so retries deduplicate. +- A halt or kill switch stops new calls only. Rollbacks keep working. `reversi plan ` (or `GET /v1/runs/{id}/plan`, or `reversi.plan_rollback` over MCP) is a dry run. It exits 2 when the run is not fully reversible. -## Retries and verification +## What the plan tells you before you run it + +```bash +reversi plan run_v --json +``` + +```json +{ + "run_id": "run_v", + "actions": [ + { + "step": 0, "tool": "stripe.create_charge", "kind": "compensate", + "compensation_tool": "stripe.refund", + "args": { "amount": 700, "charge_id": "ch_o9" }, + "idempotency_key": "e5ecbb51d5597b6dfb2e2914a0921be1efca3ef93919a06466d0f3d777a343b3", + "needs_approval": false, + "residue": ["the charge still appears on the customer's statement"], + "compensation_kind": "compensation", + "verified": true + } + ], + "fully_reversible": true +} +``` + +`kind` is one of `compensate`, `skip` (reads and idempotent tools need no undo), `unrecoverable`, `unresolved`, +`unmanifested` or `defer` (with `blocked_by`). For a `compensate` step the plan shows: + +- **`residue`**: what the undo cannot take back, from the manifest. The plain-text `reversi plan` output does not print it; use + `--json` or `GET /v1/runs/{id}/plan`. +- **`compensation_kind`**: `inverse` (restores exact state) or `compensation` (makes amends). +- **`verified`**: the undo will be read back before it counts as done. +- **`needs_approval`**: the compensation waits for an [approval](/concepts/approvals). + +`fully_reversible` is true when every standing step is `compensate` or `skip`, assuming each compensation succeeds. Residue +does not make it false: a refund that leaves a statement line is still a refund. Read `residue` to decide whether that is +acceptable. + +## Retries ```bash reversi rollback run_123 -m tools.yaml --compensator-cmd ./undo.sh --retries 4 --verify-cmd ./check-undone.sh ``` -- `--retries n` retries a failing compensation with exponential backoff (and jitter) and the same idempotency key. A +- `--retries n` (1 to 10) retries a failing compensation with exponential backoff (and jitter) and the same idempotency key. A step that still fails is `compensation_failed`, with the error and the attempt count. - `--verify-cmd` runs after each successful compensation with the same JSON on stdin. Exit 0 confirms the undo is visible downstream; anything else makes the compensation fail. This catches APIs that say "ok" and do nothing. -In code: `run.rollback({ compensate, retry: { attempts, base_ms, max_ms }, verify })`. +In code: `run.rollback({ compensate, retry: { attempts, base_ms, max_ms }, verify, verifier })`. + +## Verification -## Reconciliation +A manifest can declare its own [read-back](/concepts/manifests#verification): after the undo, call a read-only tool and require +the answer to match. Reversi does it for you: -A timeout or crash leaves a step whose effect is unknown. An operator checks the downstream system and settles it: +1. The compensation runs (with retries, if you asked). +2. The verification tool is called with the arguments resolved from the original call and result. +3. The answer is compared with `expect`. On a mismatch it waits `backoff_ms`, doubling each time, and tries again, up to `retries`. +4. A match journals `verified` and the step becomes `compensated`. No match journals `verification_failed` with the first + mismatch (`` `status` is "succeeded", expected "canceled" ``) and the step becomes `compensation_failed`. + +If the verification's arguments cannot be resolved, that is `verification_failed` too (`cannot verify: arguments +unavailable`): Reversi does not skip a check it cannot make. + +**Unverified.** If nothing is available to run the read-back, the step is still `compensated`, but Reversi journals `unverified` +(`no verifier is available to run the declared read-back`), lists the step under `unverified` in the report, and the report is +**not complete**. It will not call an unchecked undo a checked one. + +| Where you roll back | Runs the declared read-back? | +|---|---| +| Gateway `reversi.execute_rollback` | Yes, through the upstream (3 attempts, 250 ms base for the compensation) | +| Your code | Yes, if you pass `verifier` to `run.rollback` | +| `reversi rollback --reader-cmd ` | Yes. The command gets `{"tool","arguments"}` as JSON on stdin and prints the tool's JSON result. Without `--reader-cmd` a manifest with a `verification` ends `unverified`, exit 2. `--verify-cmd` is a separate check and does not satisfy it | +| `reversi-server` (`POST /v1/runs/{id}/rollback`, console) | Yes, when started with `REVERSI_READER_CMD`, which works like `--reader-cmd`. Without it, `unverified` | ```bash -reversi resolve run_123 4 -m tools.yaml --status succeeded --result '{"id":7}' \ - --note "found order 7 in the admin UI" --actor dana +reversi rollback run_123 -m tools.yaml --compensator-cmd ./undo.sh --reader-cmd ./read.sh +``` + +If the reader's answer does not match `expect`, the step fails with `verification failed: did not show the expected +state after the undo` (exit 2). If the reader exits non-zero or prints something that is not JSON, that counts as a failed +read. Without `--reader-cmd` the text output names the steps: + +```text +undone but not verified (declared read-back not run; pass --reader-cmd): 0 +rollback INCOMPLETE ``` -The note is required and stored in the ledger with the actor. Passing `--result` lets the step be rolled back. Over -HTTP: `POST /v1/runs/{id}/steps/{n}/resolve`; in the [console](/console), the **Resolve** form on the run page. -In code, `run.reconcile(probe)` asks a probe function about each unresolved step and settles those it can answer. +`--json` shows `"unverified": [0]`. Run again and the step, already `compensated`, is skipped, so the second report says +complete. Check the `unverified` event in `reversi export --run ` if that matters to you. + +## The rollback report + +`rollback` (CLI `--json`, `run.rollback`, `reversi.execute_rollback`) returns: + +| Field | Meaning | +|---|---| +| `compensated` | Steps undone | +| `skipped` | Steps that need no undo | +| `failed` | `{ step, error }`: the compensation or its verification failed, after retries | +| `unrecoverable` | Irreversible, or the compensation arguments are unavailable | +| `unresolved` | Outcome unknown (`intended`, `uncertain`) | +| `unmanifested` | No manifest, so the effect is unknown | +| `deferred` | `{ step, blocked_by }`: a dependent step is still standing | +| `awaiting_approval` | The compensation needs an approval that is pending or denied | +| `unverified` | Undone, but the declared read-back could not be run | +| `complete` | True only if `failed`, `unrecoverable`, `unresolved`, `unmanifested`, `deferred`, `awaiting_approval` and `unverified` are all empty | + +## Uncertain steps + +A timeout or crash leaves a step whose effect is unknown: the gateway journals an upstream timeout or transport error as +`uncertain`, and a step journaled `intended` that never completed is unknown too. The plan says so and the rollback refuses to +guess: + +```text +reversi plan run_timeout + step 0 stripe.create_charge unresolved outcome unknown: reconcile with the downstream system + NOT fully reversible: see above +``` + +The command exits 2. + +An operator checks the downstream system and settles it: + +```bash +reversi resolve run_timeout 0 --status succeeded --result '{"id":"ch_o50"}' \ + --note "found ch_o50 in the Stripe dashboard" --actor dana +# step 0 is now succeeded +reversi plan run_timeout +# step 0 stripe.create_charge compensate -> stripe.refund +``` + +The note is required and stored in the ledger with the actor. Passing `--result` lets the step be rolled back: without it the +compensation arguments that come from `$result` are missing and the step is `unrecoverable`. `--status failed` settles a +step that never took effect. Over HTTP: `POST /v1/runs/{id}/steps/{n}/resolve`; in the [console](/console), the **Resolve** form +on the run page. In code, `run.reconcile(probe)` asks a probe function about each unresolved step and settles those it can +answer, leaving the rest. ## Compensation approval @@ -52,12 +167,52 @@ A compensation with `requires_approval: true` waits for an [approval](/concepts/ snapshotted when the step runs, so editing a manifest later cannot change what a past step's rollback needs. `--approve-compensations` (CLI) or `approve_compensations: true` (gateway) approves them in one go: an operator decision. +## Duplicate protection: idempotency keys + +Two kinds, easy to mix up: + +- **Forward calls.** A manifest's [`idempotency`](/concepts/manifests#duplicate-calls) key makes Reversi refuse a repeat of + the same logical call (same order id, say) while the first still stands. +- **Compensations.** Each step's undo has a key derived from the run, the step and the tool. It is identical on every retry and + every re-run, and is in the compensation call as `idempotency_key`. The gateway also sends it to the upstream in the + MCP call metadata. Pass it to the downstream API so a retry after a lost response does nothing twice. + +## Stopping new calls: kill switches and scoped halts + +A halt blocks new calls with `kill switch: ` while rollbacks continue. It can be narrowed. + +```bash +reversi halt add "investigating duplicate charges" --tool 'stripe.*' --actor billing-agent --by dana +reversi halt list +# halt_e0d44609-... investigating duplicate charges scope={"tool":"stripe.*","actor":"billing-agent"} by dana +reversi halt lift # one id, or all +# lifted 1 +``` + +- A scope is any of `tool`, `actor`, `tenant` and `environment` (globs). No scope halts everything. A call must match every field named. +- Halts live in the data store, so every process sharing it sees them at once. +- A file works for everything at once: `reversi kill-switch on ` creates the file named by `REVERSI_KILL_FILE` (or `kill_switch_file:` + in the config), and every gateway and server watching it blocks new calls until `kill-switch off`. +- Over HTTP an admin uses `POST /v1/halt` (body: `reason` and optional scope) and `POST /v1/resume`; `GET /v1/halts` lists them. + The MCP tools `reversi.halt` and `reversi.resume` are unscoped. + +## Continuing a run + +An agent that reconnects (a new process, a dropped connection) can name its run to continue it, so approvals and the rollback +plan stay together. Over HTTP send `Reversi-Run-Id: `; over stdio pass `--run-id ` to `reversi-gateway`. If the run +exists it is continued, not restarted; otherwise it is created under that id. + +- Only the principal that opened a run may continue it. Another key gets an error (`belongs to another principal`). +- The run keeps the mode and scope it was opened with. +- The id must be a valid run id; a bad one is a 400. + ## Who runs the compensation | Way | How | |---|---| | CLI | `--compensator-cmd` receives the call as JSON on stdin | -| Gateway | `reversi.execute_rollback` on the operator endpoint calls the compensation tool on the upstream | +| Gateway | `reversi.execute_rollback` on the operator endpoint calls the compensation tool on the upstream. Without `confirm=true` it is a dry run | +| Server | `POST /v1/runs/{id}/rollback` with `confirm: true`, when `REVERSI_COMPENSATOR_CMD` is set. Starts a job (202); `GET /v1/runs/{id}/rollback` shows its progress | | Your code | `run.rollback({ compensate })`, or `beginCompensation` / `completeCompensation` for remote workers | | Connectors | `reversi-connectors compensate` is a ready-made `--compensator-cmd` | diff --git a/docs/concepts/simulation.mdx b/docs/concepts/simulation.mdx new file mode 100644 index 0000000..b738103 --- /dev/null +++ b/docs/concepts/simulation.mdx @@ -0,0 +1,216 @@ +--- +title: Simulation and capture +description: Try a policy on last week's calls, replay a run with one argument changed, preview what approving a call does, and compare plans before running any of them. +--- + +Four tools answer "what would happen" without touching a real system. They run calls through a throwaway engine: an +in-memory ledger and run state, your real manifests and policy, and nothing executed anywhere. The real ledger and runs are +not changed. + +| Question | Tool | Needs capture | +|---|---|---| +| What would this new policy have done to last week's calls? | [Policy simulation](#policy-simulation) | Yes | +| What if this one argument had been different? | [Replay](#replay-a-run-with-a-change) | Yes | +| What happens if I approve this call, and can it be undone? | [Impact preview](#impact-preview) | No | +| Which of these plans is safest? | [Speculative execution](#speculative-execution) | No | + +## Capture + +The ledger never stores arguments, only hashes. Replay and simulation need the arguments, so keeping them is a separate, +explicit choice. **Capture is off by default.** Turn it on in `reversi.config.yaml`: + +```yaml +capture: + enabled: true + ttl: 7d # how long a record lives (default 7 days) + redact_pii: true # default true +``` + +or with `REVERSI_CAPTURE=1` (and `REVERSI_CAPTURE_TTL_DAYS=14`) in the environment of the gateway or server. `reversi doctor` +reports whether it is on. The gateway and the server both record when it is on. + +**What is stored**, per call: run id and step (no step for a call that was blocked before it became one), tool, actor, tenant, +environment, the arguments, the result once the call finishes, the decision (`allow`, `would_block`, `block`) with its +reasons, the outcome (`succeeded`, `failed`, `uncertain`), the manifest version, the risk score, and when it was captured and +when it expires. + +**Redaction happens before anything is written:** + +- credentials in any string (tokens, keys) become `[REDACTED:github_token]` and the like; +- personal data (emails, phone numbers, card numbers, SSNs, IBANs, with checksum validation) becomes `[PII:email]` and the + like, unless `redact_pii: false`; +- every path in the manifest's [`audit.redaction`](/concepts/manifests#fields) (`$args.card_token`, `$result.secret`) becomes `[REDACTED]`; +- strings over 20,000 characters are cut, lists are cut to 200 items and nesting to 12 levels. + +**Expiry.** A record past its `expires` time is no longer listed or used, and `reversi capture purge` deletes expired records +(or everything for one run with `--run`). A [retention](/operations/retention) run purges captures along with the run state. + +```bash +reversi capture list +# 2026-10-16T17:22:38Z run_demo#0 stripe.create_charge allow succeeded args={"order_id":"o1","amount":2500,"card_token":"[REDACTED]","email":"[PII:email]"} +# 2026-10-16T17:22:38Z run_demo stripe.create_charge block args={"order_id":"o2","amount":80000,"card_token":"[REDACTED]","email":"[PII:email]"} +reversi capture list --run run_demo --tool stripe.create_charge --since 1d --json +reversi capture purge --run run_demo +# removed 1 captured calls +``` + +Because values are redacted, a rule that reads inside a redacted value can decide differently in a simulation than it did +live. The report says so with a note. + +## Policy simulation + +Replay captured calls through a candidate policy and see what changes. + +```bash +reversi simulate --policy strict.yaml --since 7d # default period is 7d; --run for one run; --json +# 3 captured calls in 1 runs: 0 unchanged +# +# newly blocked (3) +# run_demo#0 stripe.create_charge allow -> block policy `no-big`: charges over 10.00 are paused +# run_demo stripe.create_charge needs_approval -> block policy `no-big`: charges over 10.00 are paused +# run_demo stripe.create_charge needs_approval -> block policy `no-big`: charges over 10.00 are paused +# note: some captured arguments were redacted, so rules that look inside those values may behave differently than they did live +``` + +Each call is compared with what was decided at the time: `allow`, `needs_approval` or `block`. Calls that now differ are +listed under `newly_blocked`, `newly_needs_approval` and `newly_allowed`. The command exits 2 when something would be newly +blocked, so it fits in CI before a policy change. A run that was `needs_approval` and is now `block` counts as newly blocked. + + +The sandbox sees the manifests and the candidate policy, not the live state around the original call. A call the history +blocked for another reason (a halt, a missing prerequisite, a content check) can show up as `newly allowed`. Read those rows +against the reason that was recorded. + + +Where it is available: + +| Where | How | +|---|---| +| CLI | `reversi simulate --policy ` | +| HTTP | `POST /v1/simulate` with `{ "policy": "", "run_id": "...", "since": "7d" }`. Operator role. Without `policy` it uses the active one. 404 if capture is off, 400 for an invalid policy | +| Console | The **Policy** page: **Simulate against captured history** (last 7 days) beside **Validate and run its tests** | + +## Replay a run with a change + +Replay one captured run twice in the sandbox, as recorded and with your changes, and compare each decision and the rollback +plan. + +```bash +reversi replay run_demo --set 0.amount=90000 +# run run_demo: 3 captured calls +# 0 stripe.create_charge as recorded: allow with your changes: needs_approval (approval required: simulated) +# 1 stripe.create_charge as recorded: needs_approval with your changes: needs_approval (approval required: simulated) +# 2 stripe.create_charge as recorded: needs_approval with your changes: needs_approval (approval required: simulated) +# 1 decision would change +# rollback as recorded: fully reversible; with changes: fully reversible +``` + +`--set` is repeatable and takes `path=value`, with the value as JSON (a plain word is a string): + +| Form | Applies to | +|---|---| +| `amount=9000` | Every call | +| `1.amount=9000` | Step 1 only | +| `tool:stripe.create_charge/amount=100` | Every call to that tool | + +`--policy ` also tries a different policy in the same replay. The JSON report has `baseline`, `variant`, `differences`, +`baseline_plan` and `variant_plan`. + +| Where | How | +|---|---| +| CLI | `reversi replay --set ... [--policy ...] [--json]` | +| HTTP | `POST /v1/runs/{id}/replay` with `{ "policy": "", "changes": [{ "step": 1, "tool": "...", "path": "$args.qty", "value": 50 }] }`. Operator role. 404 for a run with no captures | +| Console | **Replay with a change** on a run's page (when capture is on): step, tool, argument path, new value. One change per submit | + +## Impact preview + +What would approving this call do, and what could be undone afterwards? It reads manifests and policy only, so it works with +capture off. + +```bash +reversi impact stripe.create_charge --args '{"order_id":"o9","amount":90000}' +# stripe.create_charge: compensable +# decision now: needs_approval (approval required: simulated) +# risk 80: effect compensable +30, blast radius high +20, data class financial +20, environment prod +10 +# data class financial, blast radius high, owner payments-team@example.com +# rollback: undo by calling stripe.refund +# rollback: but the charge still appears on the customer's statement +# rollback: the undo is read back and verified +reversi impact nope.tool +# nope.tool: no manifest +# decision now: block (no manifest is registered for this tool) +# rollback: unknown: without a manifest Reversi cannot say whether this can be undone +# warning: unclassified tool +``` + +The preview gives the decision now (`allow`, `needs_approval`, `block`) with the reasons, the risk score and its factors, owner +and risk labels, the tools that must run first, who must approve and how many, and the rollback outlook: the undo tool, its +residue, whether it is verified, whether it needs approval. Warnings cover irreversible effects, draft or deprecated manifests +and an unverified undo. Prerequisites in `requires` are assumed satisfied, so the answer is about this call. + +| Where | How | +|---|---| +| CLI | `reversi impact [--args ] [--actor ] [--policy ] [--json]` | +| HTTP | `GET /v1/impact?tool=...&args=&actor=...`. Auditor role | +| Console | **Impact preview** page (`/console/impact`), linked from pending approvals and manifest pages | +| Gateway | `reversi.preview_impact` with `{ tool, arguments }`, on the agent endpoint too, so an agent can ask before it acts | + +## Speculative execution + +Compare alternative plans before running any of them. Each plan is a list of calls; each is run in the sandbox and scored. + +```yaml plans.yaml +plans: + - name: charge 900.00 + calls: + - { tool: stripe.create_charge, args: { order_id: o7, amount: 90000 }, result: { id: ch_7 } } + - name: charge 9.00 + calls: + - { tool: stripe.create_charge, args: { order_id: o8, amount: 900 }, result: { id: ch_8 } } + - name: just look + calls: + - { tool: stripe.get_charge, args: { charge_id: ch_1 } } +``` + +```bash +reversi speculate plans.yaml --policy policy.yaml [--actor billing-agent] [--json] +# charge 9.00 <- recommended +# can run to the end; approvals needed: 0; highest risk: 0; fully reversible +# leaves behind: the charge still appears on the customer's statement +# just look +# can run to the end; approvals needed: 0; highest risk: 0; fully reversible +# note: some results were synthesized from the manifests, so the rollback outlook assumes compensation arguments are available +# charge 900.00 +# can run to the end; approvals needed: 1; highest risk: 0; fully reversible +# leaves behind: the charge still appears on the customer's statement +``` + +For each plan you get: the first call that would be refused (`blocked_at`), how many approvals it needs, the highest and total +risk score, the irreversible tools, the residue a rollback would leave, whether it is fully reversible, and the rollback plan. +Plans are ranked, lower is safer: 10,000 if the plan is blocked, plus 10 per approval, plus the highest risk score, plus 50 +per irreversible call, plus 25 if it is not fully reversible. `recommended` is the best plan that can run to the end. +A `result` you give is used to build compensation arguments; without one the result is synthesized from the manifest and the plan +notes it. Risk scores appear when the policy has a [`risk`](/concepts/policy#risk-scoring) section (here it has none, so they are 0). + +| Where | How | +|---|---| +| CLI | `reversi speculate ` | +| HTTP | `POST /v1/speculate` with `{ "plans": [{ "name": "...", "calls": [{ "tool": "...", "args": {}, "result": {} }] }], "policy": "", "actor": "..." }`. Auditor role | +| Console, gateway | No page and no `reversi.*` tool for it | + +## Availability at a glance + +| | CLI | `/v1` endpoint | Console | Gateway tool | +|---|---|---|---|---| +| Capture list and purge | `reversi capture list`, `purge` | none | none | none | +| Policy simulation | `reversi simulate` | `POST /v1/simulate` | Policy page | none | +| Replay | `reversi replay` | `POST /v1/runs/{id}/replay` | Run page | none | +| Impact preview | `reversi impact` | `GET /v1/impact` | Impact preview | `reversi.preview_impact` | +| Speculation | `reversi speculate` | `POST /v1/speculate` | none | none | + +## Limits + +- Simulations use the manifests and policy as loaded now, with the real clock for time windows and freezes. +- A sandbox run has every permission in the manifests, so scope problems do not show. +- Limit and budget counters start empty in the sandbox and count only the replayed calls, so a limit that fired live because of other traffic may not fire in a replay. +- Only calls captured while capture was on, and not yet expired, can be simulated or replayed. diff --git a/docs/connectors.mdx b/docs/connectors.mdx index 852b911..58b26f7 100644 --- a/docs/connectors.mdx +++ b/docs/connectors.mdx @@ -1,9 +1,9 @@ --- title: Connectors -description: Ready-made tools for the filesystem, GitHub, Slack and Stripe, each with a Reversi manifest and a compensation that really undoes it. +description: Ready-made tools for the filesystem, GitHub, GitLab, Slack, Teams, Linear, Jira, Stripe, Google Calendar, Postgres, S3, Cloud Storage, Blob Storage and Kubernetes, each with a Reversi manifest, a compensation that really undoes it, and a certification suite. --- -`@multimindlab/reversi-connectors` ships four connectors. Every tool has a [manifest](/concepts/manifests), and every +`@multimindlab/reversi-connectors` ships fourteen connectors. Every tool has a [manifest](/concepts/manifests), and every compensation named in a manifest is itself a tool in the same package that accepts exactly the arguments the `argument_map` produces. Rollback therefore works end to end, with no glue code. @@ -34,16 +34,34 @@ retried rollback never repeats a side effect. | Command | Purpose | |---|---| -| `serve [--root dir] [--backup-dir dir]` | MCP server on stdio | +| `serve [--root dir] [--backup-dir dir]` | MCP server on stdio | | `manifests [--format json\|yaml]` | Print manifests | | `compensate [--root dir] [--backup-dir dir]` | Run one compensation call from stdin | +| `certify ` | Print the twelve certification checks and where the test suite enforces them | -| Connector | Environment variable | +`` is one of `fs`, `github`, `slack`, `stripe`, `gitlab`, `teams`, `linear`, `jira`, `gcalendar`, `postgres`, `aws`, `gcp`, `azure`, `kubernetes`. + +| Connector | Environment variables | |---|---| | `fs` | none; `--root` is the only directory it can touch | | `github` | `GITHUB_TOKEN` | | `slack` | `SLACK_BOT_TOKEN` | | `stripe` | `STRIPE_SECRET_KEY` | +| `gitlab` | `GITLAB_TOKEN`; optional `GITLAB_BASE_URL` for a self-hosted instance | +| `teams` | `TEAMS_TOKEN` (a Microsoft Graph access token) | +| `linear` | `LINEAR_API_KEY` | +| `jira` | `JIRA_BASE_URL`, `JIRA_EMAIL`, `JIRA_API_TOKEN` | +| `gcalendar` | `GOOGLE_CALENDAR_TOKEN` | +| `postgres` | `POSTGRES_URL` | +| `aws` | `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_REGION`; optional `AWS_SESSION_TOKEN` | +| `gcp` | `GCP_ACCESS_TOKEN` | +| `azure` | `AZURE_STORAGE_TOKEN`, `AZURE_STORAGE_ACCOUNT` | +| `kubernetes` | `KUBE_API_SERVER`, `KUBE_TOKEN`; optional `KUBE_CA_CERT` (a path to the cluster CA, PEM) | + +Credentials are read from the environment only; a flag such as `--token` is not accepted. `compensate` enables every +connector whose main credential variable is set (`GITHUB_TOKEN`, `SLACK_BOT_TOKEN`, `STRIPE_SECRET_KEY`, +`GITLAB_TOKEN`, `TEAMS_TOKEN`, `LINEAR_API_KEY`, `JIRA_API_TOKEN`, `GOOGLE_CALENDAR_TOKEN`, `POSTGRES_URL`, +`AWS_ACCESS_KEY_ID`, `GCP_ACCESS_TOKEN`, `AZURE_STORAGE_TOKEN`, `KUBE_API_SERVER`). Secrets are redacted from every error message. @@ -53,6 +71,7 @@ In code: import { allConnectors, compensatorFor } from "@multimindlab/reversi-connectors"; const connectors = allConnectors({ fs: { root: "./work" }, github: { token: process.env.GITHUB_TOKEN! } }); +// also: gitlab, teams, linear, jira, gcalendar, postgres, aws, gcp, azure, kubernetes const registry = Registry.from(connectors.flatMap((c) => c.manifests)); // ... run.call({ tool, args, reason, execute: (a) => connectors[0].tools[tool](a, {}) }) await run.rollback({ compensate: compensatorFor(connectors) }); @@ -142,3 +161,288 @@ Limits: and confirm them outside the agent. - A deleted Stripe customer cannot be restored, and its saved payment methods are gone. Rollback only deletes customers the run created. + +## GitLab (`gitlab`) + +Calls `https://gitlab.com/api/v4`. For a self-hosted instance set `GITLAB_BASE_URL` (or `baseUrl` in code) to its root; `/api/v4` is added. The token goes in the `PRIVATE-TOKEN` header. `project` is a numeric ID or a full path such as `group/project`. + +| Tool | Effect | Permission | Undone by | +|---|---|---|---| +| `gitlab.get_issue` | read_only | `gitlab:issues:read` | | +| `gitlab.create_issue` | reversible | `gitlab:issues:write` | `gitlab.close_issue` | +| `gitlab.add_note` | reversible | `gitlab:issues:write` | `gitlab.delete_note` (404 is success) | +| `gitlab.create_branch` | reversible | `gitlab:branch:write` | `gitlab.delete_branch` (404 is success) | +| `gitlab.close_issue`, `gitlab.delete_note` | idempotent | `gitlab:issues:write` | | +| `gitlab.delete_branch` | idempotent | `gitlab:branch:delete` | | + +Required token scopes: `read_api` for `get_issue`; `api` for everything else. The user behind the token needs at least the Developer role in the project. + +Limits: + +- The issue is closed, not deleted (as with GitHub): it stays visible in the project. +- `gitlab.delete_branch` is idempotent so it can run as a compensation; an agent that calls it itself is not undone. Keep `gitlab:branch:delete` out of the agent's scope. Protected branches cannot be deleted without the Maintainer role. + +## Microsoft Teams (`teams`) + +Calls Microsoft Graph, `https://graph.microsoft.com/v1.0`, with a bearer token in `TEAMS_TOKEN`. Obtain the token yourself (the connector does not run an OAuth flow). + +| Tool | Graph request | Effect | Permission | Undone by | +|---|---|---|---|---| +| `teams.list_channels` | `GET /teams/{team-id}/channels` | read_only | `teams:channels:read` | | +| `teams.post_message` | `POST /teams/{team-id}/channels/{channel-id}/messages` | reversible | `teams:messages:write` | `teams.delete_message` | +| `teams.delete_message` | `POST /teams/{team-id}/channels/{channel-id}/messages/{message-id}/softDelete` | idempotent | `teams:messages:delete` | | + +Microsoft Graph permissions, as we read the Graph reference (confirm against the current reference for your tenant, since Microsoft changes these): + +| Tool | Graph permission | +|---|---| +| `teams.list_channels` | `Channel.ReadBasic.All` (delegated or application) | +| `teams.post_message` | `ChannelMessage.Send`, delegated only. Application permissions cannot post to a channel except for data import. | +| `teams.delete_message` | `ChannelMessage.ReadWrite`, delegated. The signed-in user must be allowed to delete that message. | + +Limits: + +- Posting needs a signed-in user (delegated token). A service principal with only application permissions cannot use `teams.post_message`. +- `softDelete` marks the message deleted; members still see a "message deleted" placeholder in the channel, and the message remains in compliance exports. It is not a purge. +- People may have read or replied to the message before it is deleted. +- The Graph endpoints and permissions above are implemented from Microsoft's documentation and tested against a fake; they have not been run against a live tenant. + +## Linear (`linear`) + +One GraphQL endpoint, `https://api.linear.app/graphql`. A personal API key is sent as the whole `Authorization` value (no `Bearer ` prefix). To use an OAuth token instead, pass it prefixed with `Bearer `. + +| Tool | GraphQL operation | Effect | Permission | Undone by | +|---|---|---|---|---| +| `linear.get_issue` | `issue(id:)` | read_only | `linear:issues:read` | | +| `linear.create_issue` | `issueCreate` | reversible | `linear:issues:write` | `linear.archive_issue` | +| `linear.add_comment` | `commentCreate` | reversible | `linear:comments:write` | `linear.delete_comment` | +| `linear.archive_issue` | `issueArchive` | idempotent | `linear:issues:write` | | +| `linear.delete_comment` | `commentDelete` | idempotent | `linear:comments:write` | | + +OAuth scopes: `read` for `get_issue`; `write` (or `issues:create` and `comments:create` for the create calls) for the rest. A personal API key has the user's full access. + +Limits: + +- An archived issue still exists; it is hidden from boards and can be restored in the Linear UI. Linear sends notifications when an issue is created, and archiving does not recall them. +- GraphQL errors come back with HTTP 200 or 400; both are treated as failures. "Entity not found" counts as success for the two compensations, since the goal (the thing is gone) is met. + +## Jira (`jira`) + +Jira Cloud REST API v3 at `JIRA_BASE_URL` (such as `https://yourco.atlassian.net`), with basic authentication: your account email and an API token. Descriptions and comments are sent as Atlassian Document Format paragraphs. + +| Tool | Effect | Permission | Undone by | +|---|---|---|---| +| `jira.get_issue` | read_only | `jira:issues:read` | | +| `jira.create_issue` | reversible | `jira:issues:write` | `jira.delete_issue` | +| `jira.add_comment` | reversible | `jira:comments:write` | `jira.delete_comment` | +| `jira.transition_issue` | compensable | `jira:issues:transition` | `jira.restore_status` | +| `jira.delete_issue` | idempotent | `jira:issues:delete` | | +| `jira.delete_comment` | idempotent | `jira:comments:write` | | +| `jira.restore_status` | idempotent | `jira:issues:transition` | | + +`jira.transition_issue` is not reversible by itself. It reads the issue's current status first and returns `previous_status` and `previous_status_id`; rollback calls `jira.restore_status`, which finds a workflow transition that leads to that status and runs it. A transition can be given as an ID, a transition name, or the name of the status it leads to. + +Jira project permissions the account needs: Browse Projects, Create Issues, Add Comments, Transition Issues, Delete Issues (for the `create_issue` compensation), Delete Own Comments (for the `add_comment` compensation). With OAuth 3LO instead of basic auth the scopes are `read:jira-work` and `write:jira-work`. + +Limits: + +- Deleting an issue cannot be undone and the issue key number is not reused. `jira.delete_issue` is idempotent only so it can run as a compensation; keep `jira:issues:delete` out of the agent's scope. Jira refuses to delete an issue that has subtasks. +- If the workflow has no transition back to the previous status, `jira.restore_status` fails and the rollback reports the step as failed. The issue history shows both transitions, and workflow post-functions (assignee changes, resolutions, notifications) run each time. + +## Google Calendar (`gcalendar`) + +Calls `https://www.googleapis.com/calendar/v3` with a bearer token in `GOOGLE_CALENDAR_TOKEN`. `calendar_id` defaults to `primary`. Every write passes `sendUpdates=none`, so no invitation or change emails go out. + +| Tool | Effect | Permission | Undone by | +|---|---|---|---| +| `gcalendar.list_events` | read_only | `gcalendar:events:read` | | +| `gcalendar.create_event` | reversible | `gcalendar:events:write` | `gcalendar.delete_event` (404 and 410 are success) | +| `gcalendar.update_event` | reversible | `gcalendar:events:write` | `gcalendar.restore_event` | +| `gcalendar.delete_event` | idempotent | `gcalendar:events:delete` | | +| `gcalendar.restore_event` | idempotent | `gcalendar:events:write` | | + +`gcalendar.update_event` reads the event first and returns `previous`, holding the old value (or `null` if the field was empty) of exactly the fields it changed. `gcalendar.restore_event` patches those values back; a `null` removes the field. Fields: `summary`, `description`, `location`, `start`, `end`. + +OAuth scopes: `https://www.googleapis.com/auth/calendar.events.readonly` for `list_events`; `https://www.googleapis.com/auth/calendar.events` for the rest. + +Limits: + +- Attendees are not managed by this connector, and nothing is emailed. If you need invitations, do that outside the agent. +- A deleted event goes to the calendar's trash for a while but is not restored by rollback of `create_event`; it is deleted. +- Someone editing the event between `update_event` and rollback has those edits to the same fields overwritten by the restore. + +## Postgres (`postgres`) + +Connects with `POSTGRES_URL` through the `pg` package. Every tool takes parameters only: + +- Table, schema and column names must match `^[A-Za-z_][A-Za-z0-9_]{0,62}$` and are always double-quoted. +- Every value is a bound parameter (`$1`, `$2`, ...). No value is ever written into SQL text. The tests assert the exact SQL and parameters. + +| Tool | Effect | Permission | Undone by | +|---|---|---|---| +| `postgres.query` | read_only | `postgres:rows:read` | | +| `postgres.insert_row` | reversible | `postgres:rows:write` | `postgres.delete_row` (by the primary key from `RETURNING`) | +| `postgres.update_row` | reversible | `postgres:rows:write` | `postgres.restore_row` | +| `postgres.delete_row` | idempotent | `postgres:rows:delete` | | +| `postgres.restore_row` | idempotent | `postgres:rows:write` | | + +How each works: + +- `insert_row` runs `INSERT INTO "t" ("a","b") VALUES ($1,$2) RETURNING "id"` (the key column is `id` unless `key_column` says otherwise) and returns the key. +- `update_row` runs one transaction: `BEGIN`, `SELECT` of the old values of the columns being set with `FOR UPDATE`, `UPDATE`, `COMMIT`. It returns the old values; `restore_row` sets them back. +- `query` takes one `SELECT` or `WITH` statement with `$n` placeholders and `values`. It runs between `BEGIN READ ONLY` and `ROLLBACK`, so a write hidden in a `WITH` is refused by the server. Statements containing `;` are rejected up front because a multi-statement string sent without parameters would run outside the guard. At most 1000 rows come back. +- Objects and arrays are bound as JSON text (for `json` and `jsonb` columns). Binary columns are not supported. +- A server error carries its SQLSTATE (`Postgres error 42501: ...`) and the transaction is rolled back. A lost connection after a write was sent is `EffectUncertainError`. + +Database privileges: `SELECT` for `query`; `INSERT` for `insert_row`; `UPDATE` for `update_row` and `restore_row`; `DELETE` for `delete_row`. Use a dedicated role. + +Limits: + +- `READ ONLY` stops writes to tables, not every side effect: `nextval()`, advisory locks and functions from extensions still run. Grant the role only what it needs. +- Rollback of `insert_row` deletes the row, but sequences stay advanced (gaps in `serial` ids), triggers and `ON DELETE CASCADE` rules run, and rows other transactions linked to it meanwhile are affected. +- `restore_row` writes the old values back, so `updated_at`-style triggers fire again and any change by others to those columns since is overwritten. +- `postgres.delete_row` deletes data for good. It is idempotent so it can run as a compensation; keep `postgres:rows:delete` out of the agent's scope. + +## Amazon S3 (`aws`) + +Signs requests with AWS Signature Version 4, implemented here with `node:crypto` (no AWS SDK). Credentials: `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_REGION`, optional `AWS_SESSION_TOKEN`. Requests go to `https://.s3..amazonaws.com` (path-style for bucket names containing dots, or against `endpoint` for MinIO and similar). Content is text by default, or base64 with `encoding: "base64"`; objects up to 5 MiB. + +| Tool | Effect | Permission | IAM actions | Undone by | +|---|---|---|---|---| +| `aws.get_object` | read_only | `aws:s3:read` | `s3:GetObject` | | +| `aws.list_objects` | read_only | `aws:s3:read` | `s3:ListBucket` | | +| `aws.put_object` | reversible | `aws:s3:write` | `s3:GetObject`, `s3:PutObject`, `s3:ListBucket` | `aws.restore_object` | +| `aws.delete_object` | reversible | `aws:s3:write` | `s3:GetObject`, `s3:PutObject`, `s3:DeleteObject`, `s3:ListBucket` | `aws.restore_object` | +| `aws.restore_object` | idempotent | `aws:s3:write` | `s3:GetObject`, `s3:PutObject`, `s3:DeleteObject` | | + +How the undo works: before `put_object` overwrites or `delete_object` deletes an existing key, the connector copies it server-side to `.reversi-backups/` in the same bucket and returns `backup_key` and `existed`. `restore_object` copies the backup back over the key, or deletes the key when there was no previous object (`backup_key` is `null`). Without `s3:ListBucket` S3 answers 403 instead of 404 for a missing key, so grant it. + +Limits: + +- Backup objects are kept (that is what makes the restore repeatable). Add a lifecycle rule that expires `.reversi-backups/`. +- Copies use `CopyObject`, which is limited to 5 GiB per object; larger objects fail. +- There is no precondition on S3 writes here, so a writer racing between the backup and the write can have its change restored over. The Cloud Storage and Blob Storage connectors guard this with generation and ETag preconditions. +- Bucket event notifications, replication and versioning react to every write, copy and delete the connector makes. +- `.reversi-backups/` keys cannot be written or deleted through the tools and are hidden from `list_objects`. + +## Google Cloud Storage (`gcp`) + +Calls the Cloud Storage JSON API at `https://storage.googleapis.com` with a bearer token in `GCP_ACCESS_TOKEN`. Same shape as S3, with preconditions: the connector reads the object's generation, copies that generation to `.reversi-backups/` with `rewriteTo` (looping until `done`, so large objects work), and then writes or deletes with `ifGenerationMatch` set to that generation (`0` for a new object). If someone changed the object in between, the write is refused with 412 and nothing is overwritten. + +| Tool | Effect | Permission | Undone by | +|---|---|---|---| +| `gcp.get_object`, `gcp.list_objects` | read_only | `gcp:storage:read` | | +| `gcp.put_object`, `gcp.delete_object` | reversible | `gcp:storage:write` | `gcp.restore_object` | +| `gcp.restore_object` | idempotent | `gcp:storage:write` | | + +Required: OAuth scope `https://www.googleapis.com/auth/devstorage.read_write` (`devstorage.read_only` for reads only) and the IAM permissions `storage.objects.get`, `storage.objects.list`, `storage.objects.create`, `storage.objects.delete`. The role `roles/storage.objectAdmin` covers them. + +Limits: as for S3, backup objects remain (add a lifecycle rule); object versioning and Pub/Sub notifications react to every write; `restore_object` overwrites whatever is at the key now. + +## Azure Blob Storage (`azure`) + +Calls `https://.blob.core.windows.net` (service version `2023-11-03`) with a Microsoft Entra bearer token in `AZURE_STORAGE_TOKEN` (audience `https://storage.azure.com/`). Same shape as S3, with ETag preconditions: the blob is copied synchronously to `.reversi-backups/` only if its ETag is unchanged (`x-ms-source-if-match`), then written with `If-Match` (or `If-None-Match: *` for a new blob) or deleted with `If-Match`. + +| Tool | Effect | Permission | Undone by | +|---|---|---|---| +| `azure.get_blob`, `azure.list_blobs` | read_only | `azure:blob:read` | | +| `azure.put_blob`, `azure.delete_blob` | reversible | `azure:blob:write` | `azure.restore_blob` | +| `azure.restore_blob` | idempotent | `azure:blob:write` | | + +Required Azure roles: Storage Blob Data Reader for reads; Storage Blob Data Contributor for writes and deletes. + +Limits: the synchronous copy (`Copy Blob From URL`) is limited to 256 MiB; larger blobs fail. Backup blobs remain (add a lifecycle management rule). Soft delete, versioning and Event Grid events on the account react to each operation. Only block blobs are written. + +## Kubernetes (`kubernetes`) + +Calls the Kubernetes REST API at `KUBE_API_SERVER` with a bearer token in `KUBE_TOKEN`. With `KUBE_CA_CERT` (a path to the cluster CA in PEM) requests go through `node:https` with that CA; without it the connector uses the runtime's `fetch`. Namespaces must be DNS-1123 labels and names DNS-1123 subdomains, so they cannot contain a path or a query. + +| Tool | Effect | Permission | RBAC verbs | Undone by | +|---|---|---|---|---| +| `kubernetes.get_deployment` | read_only | `kubernetes:deployments:read` | `get` on `deployments.apps` | | +| `kubernetes.list_pods` | read_only | `kubernetes:pods:read` | `list` on `pods` | | +| `kubernetes.scale_deployment` | reversible | `kubernetes:deployments:scale` | `get`, `patch` on `deployments.apps/scale` | `kubernetes.restore_replicas` | +| `kubernetes.create_configmap` | reversible | `kubernetes:configmaps:write` | `create` on `configmaps` | `kubernetes.delete_configmap` | +| `kubernetes.restore_replicas` | idempotent | `kubernetes:deployments:scale` | `patch` on `deployments.apps/scale` | | +| `kubernetes.delete_configmap` | idempotent | `kubernetes:configmaps:delete` | `delete` on `configmaps` | | +| `kubernetes.rollout_restart` | irreversible, approval always | `kubernetes:deployments:restart` | `patch` on `deployments.apps` | nothing | +| `kubernetes.delete_pod` | irreversible, approval always | `kubernetes:pods:delete` | `delete` on `pods` | nothing | + +- `scale_deployment` reads the current replica count through the scale subresource, scales with a merge patch, and returns `previous_replicas`; rollback sets it back. +- `create_configmap` fails with 409 if the name exists, so rollback never deletes an object it did not create. Rollback deletes with a UID precondition, so a different ConfigMap that later took the same name is left alone (the 409 is treated as success). +- `rollout_restart` stamps the pod template annotation `kubectl.kubernetes.io/restartedAt`, like `kubectl rollout restart`. Pods are replaced, so it is declared irreversible and always needs approval. `delete_pod` likewise. + +Limits: a HorizontalPodAutoscaler may scale back on its own; `restore_replicas` sets the saved count even if someone changed it since; pods that read a ConfigMap you created may have acted on it before it was deleted. The `KUBE_CA_CERT` transport is tested against a local HTTP server, not a TLS handshake with a real CA. + +## Certification + +Every connector is run through a twelve-check certification suite in `npm test`. The suite drives each tool against a fake backend with fault injection, using a real Reversi engine for the rollback checks. A connector that fails any check fails the test run. Each check records `pass`, `fail` or `not_applicable`, and `not_applicable` must carry a reason. + +| Check | What it verifies | +|---|---| +| Forward success | Every tool does its job; read-only tools change nothing and send no write | +| Forward failure | A rejected call is an error naming the HTTP status, with the credential scrubbed, and the backend is unchanged | +| Duplicate call | The same call with the same idempotency key a second time has no further effect (applies to idempotent tools and to tools that send the key, such as Stripe writes) | +| Retry | A lost connection on a write is `EffectUncertainError`, a 5xx is an error naming the status, and running the call again succeeds | +| Compensation success | A real engine rolls the step back; the backend is back in its prior state (or the documented residue); the compensation is a registered idempotent tool that takes exactly the arguments the manifest maps | +| Compensation failure | A 500 during the undo leaves the step `compensation_failed` and the rollback incomplete; after the fault clears, a retry completes | +| Verification failure | A `verify` callback that returns false makes the step report as failed and not compensated | +| Permission denial | A 401 or 403 is an error naming the status, without the credential | +| Timeout | A backend that never answers ends in `EffectUncertainError` for a write and a plain error for a read | +| Redaction | The credential is in no error message, returned JSON, request URL or request body | +| Dependency block | With a `requires` edge, rolling back while the later step stands defers the earlier one | +| Manifest drift | A changed, added or removed tool input schema is detected against stored hashes | + +Results, generated from the test run (the test suite fails if this table or the reasons below drift from the actual results): + +| Connector | Forward success | Forward failure | Duplicate call | Retry | Compensation success | Compensation failure | Verification failure | Permission denial | Timeout | Redaction | Dependency block | Manifest drift | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| fs | pass | pass | pass | n/a | pass | pass | pass | n/a | n/a | n/a | n/a | pass | +| github | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | n/a | pass | +| slack | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | n/a | pass | +| stripe | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | n/a | pass | +| gitlab | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | n/a | pass | +| teams | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | n/a | pass | +| linear | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | n/a | pass | +| jira | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | n/a | pass | +| gcalendar | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | n/a | pass | +| postgres | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | n/a | pass | +| aws | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | n/a | pass | +| gcp | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | n/a | pass | +| azure | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | n/a | pass | +| kubernetes | pass | pass | pass | pass | pass | pass | pass | pass | pass | pass | n/a | pass | + +Why a check is `n/a`: + +- fs, Retry: the local filesystem has no network transport that can fail transiently; errors are definite +- fs, Permission denial: access control is the operating system's; the connector confines paths to its root instead (covered in filesystem.test.ts) +- fs, Timeout: local file operations have no network wait to time out +- fs, Redaction: the connector holds no credential +- All connectors, Dependency block: no tool manifest declares `requires` + +The dependency block check is implemented and runs as soon as a manifest declares `requires`; the suite's self-test proves it with a synthetic connector that does. No shipped connector declares one yet, so for them it has not been exercised. + +Run `reversi-connectors certify all` for the checklist, and see `packages/connectors/test/certify.test.ts`, `test/certification.ts` and `test/fixtures/.ts` for the suite. `test/certification.selftest.test.ts` shows the suite failing a deliberately broken connector (credential leak, lost uncertainty, an undo that does nothing, no timeout). + +### Manifest drift + +`connectorSchemaHashes(connector)` returns a SHA-256 of each tool's input schema (canonical JSON, so key order does not matter). Store it as a baseline; `detectManifestDrift(manifests, inputSchemas, baseline)` later returns the tool names that were `added`, `removed` or `changed`: + +```ts +import { connectorSchemaHashes, detectManifestDrift } from "@multimindlab/reversi-connectors"; + +const baseline = connectorSchemaHashes(connector); // store this +// later, after upgrading the package: +const drift = detectManifestDrift(connector.manifests, connector.inputSchemas, baseline); +if (drift.changed.length) throw new Error(`tool schemas changed: ${drift.changed.join(", ")}`); +``` + +The package does not store or compare a baseline for you yet; this is the building block for a gateway or CI step. + +## Honest limits of the certification + +- **Everything is tested against fakes, not live services.** Each fake is written from the vendor's API documentation. If a real service behaves differently from its fake, the suite cannot tell. The first run against a real account is the real test; start with a throwaway project, bucket or database. +- **An HTTP 5xx on a write is reported as an HTTP error naming the status, not as `EffectUncertainError`.** The server answered, so the connector treats the call as failed. Only a lost connection or a timeout is uncertain. For a gateway timeout (502, 504) the write may in fact have been applied; check before retrying. +- **Compensations that leave residue:** GitHub and GitLab issues are closed, not deleted; Stripe payment intents are canceled, not removed; Linear issues are archived; Teams messages become soft-deleted placeholders; S3, Cloud Storage and Blob Storage keep `.reversi-backups/` copies; the filesystem keeps its backup directory; Postgres sequences stay advanced; a Jira rollback leaves two transitions in the issue history. Where a fake backend does not return to its exact prior state after rollback, the fixture overrides the comparison in code (`restored`, or a fingerprint that ignores backups), so the difference is stated rather than hidden. Residue the fakes do not model (Jira history, Postgres sequences, notifications) is documented here but not tested. +- **Irreversible tools are certified only for what they can be:** Stripe refunds, `kubernetes.rollout_restart` and `kubernetes.delete_pod` are checked for forward behaviour, failure, retry, permission, timeout and redaction. They have no compensation, so rollback reports them unrecoverable. +- **Concurrent writers.** The restore tools put back what the connector saw. Someone else changing the same thing between the call and the rollback is overwritten, except where a precondition guards it (Cloud Storage generations, Blob Storage ETags, Kubernetes UIDs, Postgres row locks during `update_row`). diff --git a/docs/console.mdx b/docs/console.mdx index 93188fb..7a162db 100644 --- a/docs/console.mdx +++ b/docs/console.mdx @@ -1,28 +1,138 @@ --- title: Operator console -description: See what an agent changed, decide approvals, plan rollbacks and download evidence in a browser. +description: See what an agent changed, decide approvals, run rollbacks, edit manifests and policy, and download evidence in a browser. --- -The server serves a console at `/console`: server-rendered pages, no client-side JavaScript. Sign in with an API key or OIDC -access token that has a console role (`auditor`, `approver`, `operator` or `admin`). Agent credentials are refused. +The console is a set of server-rendered pages at `/console`, with no client-side JavaScript. The server, the gateway (on its +own port) and `reversi console` can all serve it. + +## Modes + +The console is **off by default** and serves nothing until you choose a mode. + +| Mode | Login | Who may use it | Turn on with | +|---|---|---|---| +| `off` (default) | none | Nobody. `/console` answers 404 with a hint | nothing | +| `open` | none | Your own browser on this machine. Fixed role (default `operator`, or `auditor` / `approver`), **never admin** | `REVERSI_CONSOLE=open`, `reversi console`, gateway `--console open` | +| `login` | paste an API key or an OIDC access token | Anyone with an `auditor`, `approver`, `operator` or `admin` credential. Agent credentials are refused | `REVERSI_CONSOLE=login`, gateway `--console login` | + +```bash +reversi console # open mode on http://localhost:7421/console as operator +reversi console --role auditor --port 7500 # read-only +REVERSI_CONSOLE=login REVERSI_KEYS=./keys.json node packages/server/dist/main.js +``` + +The same setting can live in `reversi.config.yaml` (`console: { mode: login }`; for `open`, `role:`), and for the server +`REVERSI_CONSOLE_ROLE=auditor|approver|operator` picks the open-mode role. + +**Open mode is for a laptop.** It has no login, so it is fenced: + +- Only requests whose `Host` is `localhost`, `127.0.0.1` or `::1` are answered, any `Origin` must equal the host, and `Sec-Fetch-Site` must be `same-origin` or `none`. Anything else gets a 403 page (tests cover foreign hosts and origins). +- The server and the gateway **refuse to start** in open mode if they listen on a non-loopback address (`HOST` for the server, `listen.host` for the gateway; the Docker image sets `HOST=0.0.0.0`, so use `login` there). They also refuse the `admin` role. +- It is labelled "open console, no login" in the header. + +**Login mode** takes an API key or OIDC *access token* pasted into a form. There is no browser redirect to an identity +provider (OIDC redirect login is not built). The session is a signed cookie, re-checked against the key ring on every +request, so disabling or demoting a key ends the session at once. An OIDC session never outlives its token. + +## Pages | Page | For | |---|---| -| **Runs** | Filter by actor; "needs attention" shows runs with unresolved, failed-compensation or unrecoverable steps | -| **Run detail** | Steps and statuses, the rollback plan as a dry run, the full event trail, pending approvals, the auditor report, a signed bundle, and a **Resolve** form for unknown outcomes (operator) | -| **Approvals** | Pending queue with approve and deny forms, and recent decisions. Shows why approval is needed and the argument names, never values | -| **Tools** | Every manifest with effect, permission, approval and compensation, plus tools called without a manifest | -| **Evidence** | Ledger verification, signed whole-ledger bundle, and the kill switch (admin) | +| **Runs** | Filter by actor, tenant, tool; "needs attention" shows runs with unresolved, failed-compensation or unrecoverable steps. Link to compare two runs | +| **Run detail** | Steps and statuses with risk scores, the rollback plan (dry run), the full event trail, pending approvals, verification of the undo, report and bundle links, and the actions below | +| **Approvals** | Pending queue and recent decisions, with approve, deny, break-glass and delegation | +| **Alerts** | Live list of what needs attention: severity, what, since when, which run | +| **Manifests** | Enforced manifests with version, owner, data class, undo and lint findings; a manifest editor | +| **Policy** | The active policy, its own tests, a rule builder, an editor with simulation, and versions | +| **Impact** | Preview a call to a tool: decision now, risk score, approvers, rollback outlook | +| **Tools** | Inventory: manifests, tools the upstream offers now, use counts, which are unclassified | +| **Connectors** | Each upstream: kind, health, tools, how many have manifests. Filled by the gateway | +| **Compare** | Two runs side by side | +| **Evidence** | Ledger verification, signed whole-ledger bundle, kill switches | +| **Settings** | Roles, API keys, legal holds, retention, notification channels, integrations | + +### Run detail + +- **Auditor report** (HTML and PDF) and a **signed evidence bundle** (needs a signing key). **Replay with a change** appears when capture is on: change one argument and see which decisions would differ, in a sandbox that executes nothing. +- **Resolve** (operator): for a step stuck at `intended` or `uncertain`, choose `succeeded` or `failed` and write how you checked. Recorded with your name. +- **Rollback plan**: each action, the compensation tool, whether it needs approval, residue, and whether the undo is read back. +- **Run the rollback / Retry the rollback** (operator): shown when a compensator is configured. It runs in the background, the page refreshes every 2 s, and the result lists compensated, failed and unverified steps. The checkbox "also approve compensations that need approval" works only if you hold the approver role too. Without a compensator the page says how to run it elsewhere. +- **Escalate to on-call** (operator): shown when a compensation failed or a step is unrecoverable. Writes an `escalated` record and sends a critical notification. + +Where a compensator comes from: + +| Console served by | Executes rollbacks? | +|---|---| +| Server | Yes, if `REVERSI_COMPENSATOR_CMD` is set (a command that gets the call as JSON on stdin; exit 0 = undone). Declared read-backs run through `REVERSI_READER_CMD` (a command that gets `{"tool","arguments"}` as JSON on stdin and prints the result as JSON). Without it those steps stay `unverified` and the report says incomplete | +| Gateway | Yes, always: compensations and read-backs are called on the upstream | +| `reversi console` | No. It shows the plan; use `reversi rollback` | + +### Approvals + +Each row shows why approval is needed, the **who may decide** column (the required roles or groups, "any approver", the quorum +progress such as `1/2`, an `escalated` badge) and argument **names** only, never values. Approve and deny take a note. +A requester cannot approve their own request, and the server applies the same role, group and quorum rules as the API. +**Break-glass** (admin) overrides a pending request with a reason of at least 10 characters and an evidence reference such as +a ticket; it is marked `break-glass` on the approvals page and in the decision note. **Delegation** hands your approval authority to someone for 1 to 2160 hours; +the delegate decides as themselves on your behalf. It appears only when the engine services are present. + +### Manifests + +Needs a **registry folder** (created by `reversi manifests publish`, or by `reversi apply`) as the `manifests:` setting. With +plain YAML files the console shows manifests but they are **read-only**, and the page says so. With a registry: + +| Action | Role | +|---|---| +| New manifest (form builder or YAML), validate, see the diff against the current version | operator | +| Save as **draft** (not enforced) | operator | +| Draft from MCP tools (paste a `tools/list`, or the connected upstream's tools are prefilled) | operator | +| Run contract tests (the `reversi test` format) | any console role | +| **Publish as approved**, **approve** a draft (optionally "the author may not approve it"), **deprecate**, **restore** an older version | admin | + +Every save is a new version; history shows the author, approver and signature. An approve or restore reloads the running +configuration, and if the new configuration does not load the page says so and the previous one stays active. Drafts from +MCP tools come with a confidence level and notes; annotations are the server's claims and only ever produce drafts. + +A manifest's `approval:` object (roles, quorum, timeout, `escalate_to`) is kept as written by save, approve, deprecate and restore. + +### Policy + +Needs a **policy versions folder** (`reversi policy publish`) as the `policy:` setting; a single policy file is shown +read-only. The rule builder (tool, effect, environment, tenant, data class, one argument condition, approver groups, quorum) +adds a rule to the editor. Any console role can validate (runs the policy's own tests, shows a diff against the active +version) and **simulate** against the last 7 days of captured calls (needs capture on): newly blocked, newly needing +approval, newly allowed. **Publish and activate**, publish without activating, and **activate** an older version (which is how +you roll back) need admin. Versions list the author, tests passed and note, with diffs between neighbours. + +### Settings + +| Section | Notes | +|---|---| +| Roles | What each role may do | +| Users and API keys | Needs the server started with `REVERSI_KEYS` pointing at a writable file. Admin can create (the key is shown once), disable, enable and remove. Not available in the gateway console or with OIDC-only sign in | +| Legal holds | List; admin can place and release ([retention](/operations/retention)) | +| Retention | Needs a `retention:` section in the policy. Admin can dry-run or apply | +| Notification channels | Names; admin can send a test ([notifications](/operations/notifications)) | +| Integrations | Slack approvals, OpenTelemetry, anchoring, evidence signing and egress status | + +### What needs which role + +| Role | Gets | +|---|---| +| `auditor` | Everything read-only, contract tests, policy validate and simulate, report and bundle downloads | +| `approver` | Plus approve, deny, delegate | +| `operator` | Plus resolve, run or retry a rollback, escalate, replay, draft and save manifests, import from MCP tools | +| `admin` | Plus publish approved manifests, approve or deprecate or restore, publish or activate policy, kill switches, break-glass, keys, holds, retention, test notifications | -The console shows the rollback plan but never **executes** compensations: it has no connection to the systems they act on. Run -`reversi rollback`, the operator API, or `reversi.execute_rollback` on the gateway. +Buttons the role cannot use are hidden and the POST is refused anyway. ## Security - The session cookie is signed (HMAC-SHA-256), `HttpOnly`, `SameSite=Strict`, scoped to `/console`, and `Secure` with `REVERSI_SECURE_COOKIES=1`. It lasts 8 hours and ends on restart unless you set `REVERSI_SESSION_SECRET`. -- Every action is a POST with a per-session CSRF token. -- All values are HTML-escaped and responses carry a strict `Content-Security-Policy` (`default-src 'none'`). -- Roles are checked on every page; an auditor sees no approve buttons and the POST is refused anyway. +- Every action is a POST with a per-session CSRF token. The login form body is capped at 16 KB. +- All values are HTML-escaped and responses carry a strict `Content-Security-Policy` (`default-src 'none'`, `form-action 'self'`). +- Roles are checked on every page. Terminate TLS in front of the server. Do not expose the console to the internet without a reverse proxy and network controls. diff --git a/docs/docs.json b/docs/docs.json index 8da6f0e..f573dd6 100644 --- a/docs/docs.json +++ b/docs/docs.json @@ -7,6 +7,11 @@ "light": "#6E9BFF", "dark": "#1F5FD6" }, + "logo": { + "light": "/logo/light.png", + "dark": "/logo/dark.png" + }, + "favicon": "/logo/favicon.png", "navigation": { "groups": [ { @@ -14,7 +19,10 @@ "pages": [ "index", "quickstart", - "testing" + "testing", + "testing/adoption", + "testing/governance", + "testing/operations" ] }, { @@ -25,6 +33,7 @@ "concepts/ledger", "concepts/policy", "concepts/approvals", + "concepts/simulation", "concepts/content-checks", "concepts/evidence", "concepts/security-model" @@ -35,6 +44,7 @@ "pages": [ "gateway", "connectors", + "integrations", "console", "cli", "sdks/typescript", @@ -46,9 +56,12 @@ "group": "Operate", "pages": [ "operations/deployment", + "operations/gitops", "operations/storage", "operations/authentication", - "operations/monitoring" + "operations/monitoring", + "operations/notifications", + "operations/retention" ] }, { diff --git a/docs/gateway.mdx b/docs/gateway.mdx index 7f93455..584cbc4 100644 --- a/docs/gateway.mdx +++ b/docs/gateway.mdx @@ -5,36 +5,162 @@ description: Journal, police, approve and roll back any MCP server's tool calls, `reversi-gateway` is an MCP server that fronts one or more upstream MCP servers. Point your MCP client (Claude Code, Cursor, VS Code, your own agent) at the gateway instead of the upstream. Every `tools/call` goes through the Reversi -engine before it is forwarded. +engine before it is forwarded. The same program runs as `reversi gateway ...`. ```bash +# Day one: no manifests. Everything runs and is journaled; nothing is blocked. +reversi-gateway --upstream-cmd npx --upstream-arg -y --upstream-arg my-mcp-server + +# Later: enforce with manifests and a policy. reversi-gateway --manifests ./manifests --policy ./policy.yaml \ - --upstream-cmd npx --upstream-arg -y --upstream-arg my-mcp-server # stdio, enforcing -reversi-gateway --config gateway.yaml --manifests ./manifests # anything more + --upstream-cmd npx --upstream-arg -y --upstream-arg my-mcp-server +reversi-gateway --config gateway.yaml --manifests ./manifests # anything more +``` + +## Observe first + +You do not need manifests to start. With no manifests the gateway starts in **observe** mode unless you set a mode: + +```text +reversi-gateway: no manifests yet, so starting in observe mode: everything runs and is journaled. Run `reversi suggest-manifest` to draft manifests for what the upstream offers. +``` + +- Manifests come from `--manifests`, `REVERSI_MANIFESTS`, `manifests:` in `reversi.config.yaml`, or `./manifests` if that folder exists. +- An explicit `--mode` or `mode:` always wins. Without one: observe when no manifests loaded, `enforce` otherwise. +- The mode is read at start. After you add manifests, restart to leave the automatic observe mode. + +On start the gateway lists the upstream's tools and prints a **classification banner** on stderr. Output from a real run +against the filesystem connector with no manifests: + +```text +Reversi gateway running in observe mode. + +WARN fs.read_file - no manifest; treated as a change that could not be undone +WARN fs.write_file - no manifest; treated as a change that could not be undone +... + +0 classified, 0 unclassified reads, 9 unclassified changes, 0 destructive without a manifest. + +9 tools have no manifest. They run (nothing is blocked) and are journaled as unclassified. Draft manifests with: reversi suggest-manifest. ``` +| Line | Meaning | +|---|---| +| `OK` | A manifest covers the tool (effect and status shown) | +| `NOTE` | No manifest, but the server's annotations say it is read-only. Unverified | +| `WARN` | No manifest. Treated as a change that could not be undone | +| `BLOCK` | No manifest and the server says it is destructive | + +The same data is available any time as the `reversi.classification` tool, and in the [console](/console) under Tools and +Connectors. To turn the picture into manifests, run `reversi suggest-manifest --upstream-cmd ...` (or the +`reversi.suggest_manifest` tool), review the drafts, then publish them ([manifests](/concepts/manifests), +[GitOps](/operations/gitops)). + +## Modes + +| Mode | A call that breaks a rule | +|---|---| +| `enforce` | Blocked. The model gets the reason as an `isError` result | +| `advise` | Runs. The result carries one extra text block: `Reversi advisory (nothing was blocked): `. The record is journaled as `would_block` | +| `observe` | Runs and is journaled as `would_block`. No text is added to the result | + +In `enforce` mode a tool with no manifest is blocked. `unclassified_reads: observe` relaxes that for one case only: a tool +without a manifest whose server annotations say `readOnlyHint: true` runs, in a separate shadow observe run named +`-reads`, journaled with the note "unclassified read: the server says it is read-only; this is unverified". +Unclassified changes stay blocked. The default is `block`. + ## What happens to a call -1. **Manifest and scope.** A tool with no manifest, or whose permission is outside the session's scope, is blocked. -2. **Policy, content checks, kill switch** apply ([policy](/concepts/policy), [content checks](/concepts/content-checks)). +1. **Manifest and scope.** A tool with no manifest (in enforce mode), a draft manifest, or a permission outside the session's scope is blocked. +2. **Schema checks** (below), then **policy, content checks, kill switch** ([policy](/concepts/policy), [content checks](/concepts/content-checks)). 3. **Approval** if required: the call is blocked with the approval id, and the agent retries after an approver decides ([approvals](/concepts/approvals)). 4. **Journal the intent** (write-ahead), **forward** to the upstream, **journal the outcome**. The upstream result is returned unchanged. An `isError` result is journaled as `failed`; a transport error or timeout as `uncertain`. -A blocked call returns an MCP result with `isError: true` and the reason in the text, so the model can read it. Compensation -arguments (`$result.*`) resolve from `structuredContent`, or from a single text block holding JSON. +A blocked call returns an MCP result with `isError: true` and `Blocked by Reversi: ` in the text, so the model can +read it. Compensation arguments (`$result.*`) resolve from `structuredContent`, or from a single text block holding JSON. MCP has no field for "why", and every ledger record needs one. Clients may send it in `_meta["reversi/reason"]`; otherwise the record says `MCP tools/call `. -**One session is one run**, with the actor taken from the credential (or `actor` in the config when unauthenticated). +**One session is one run**, with the actor taken from the credential (or `actor` in the config when unauthenticated). A client +that reconnects can continue its run: over HTTP send `Reversi-Run-Id: `, over stdio pass `--run-id `. Only the +principal that opened a run may continue it. + +## Schema validation + +With `validate_schemas: true` (the default) the gateway compares each call with the schemas the upstream publishes in `tools/list`. + +| What | Checked against | If it does not match | +|---|---|---| +| Arguments | the tool's `inputSchema` | A violation, like a policy breach: `arguments do not match the tool's input schema: ...` (first five problems). Blocked in `enforce`, `would_block` in `observe` and `advise` | +| Result | the tool's `outputSchema`, when the result has `structuredContent` | The call has already happened, so it is journaled as `succeeded` with a note `result does not match the tool's output schema: ...`. The result is **not** used to build compensation arguments, so that step cannot be rolled back automatically | + +The checker covers `type`, `enum`, `const`, `required`, `properties`, `additionalProperties`, `items`, numeric and string +bounds, `pattern`, `multipleOf`, `allOf`, `anyOf` and `oneOf`. It is a small checker, not a full JSON Schema validator: +`$ref` and `if/then/else` are not followed. + +The MCP SDK's own client would throw away a result that breaks the output schema even though the call took effect. The +gateway turns that client-side validation off on purpose and does the check itself, so a mismatch is recorded instead of +losing the result. + +## MCP annotations are hints + +Servers may annotate tools (`readOnlyHint`, `destructiveHint`, `idempotentHint`, `openWorldHint`). Reversi reads them for the +banner and to seed **draft** manifests (`reversi suggest-manifest`, console import). They never grant anything: a draft +manifest is not enforceable until a person approves it, and an annotation alone never makes a mutating tool pass in +`enforce` mode (only `unclassified_reads: observe` lets a read-only claim through, and only as an unverified shadow run). + +### Schema drift + +A manifest drafted from an upstream tool carries `schema_hash`, a hash of the tool's input schema. If the upstream later +changes that schema, calls are flagged with `the tool's input schema changed since its manifest was approved: review the +manifest again`. That is a violation (blocked in `enforce`) until someone reviews the tool and publishes a manifest with the new hash. + +## Timeout and retry + +Set per tool in its manifest: + +```yaml +- tool: crm.lookup + effect: read_only + permission: crm:read + timeout_ms: 10s + retry: { attempts: 3, base_ms: 200, max_ms: 5s } +``` + +- `timeout_ms`: the gateway stops waiting after this long and journals the step as `uncertain` (the upstream may still have acted). +- `retry` repeats a failed or timed-out forward with exponential backoff. It is accepted **only** for `read_only` and `idempotent` tools; the manifest is rejected otherwise, and the gateway ignores retries for any other effect. Retrying a change could repeat it. +- Defaults when `retry` is given: 3 attempts, 200 ms base, 5 s cap (1 to 10 attempts). + +## Verification during rollback + +A manifest can declare a read-back (`compensation.verification`) such as "call `notes.read` and expect `restored: true`". +When the gateway executes a rollback it runs that read-back **through the upstream** (the verifying tool is an ordinary +upstream tool), retrying with backoff, and records `verified`, `verification_failed` or `unverified`. The same applies to +rollbacks started from the gateway console. See [rollback](/concepts/rollback). + +## Hot reload + +Manifests, the policy and the manifest public key are re-read while the gateway runs. A configuration that fails to load is +rejected and the previous one keeps serving. A successful change is written to the ledger as a `reloaded` record. + +| Trigger | Notes | +|---|---| +| File watch | On by default, polling every 2 s. `reload: { watch: false, interval: 10s }` in `reversi.config.yaml` | +| `SIGHUP` | `kill -HUP ` | +| `reversi.reload` | Admin only, operator endpoint | + +The gateway's own config file (upstreams, `listen`, `auth`, `allow`, `deny`) is read at start; restart to change it. Mode is also fixed at start. ## Config ```yaml gateway.yaml -mode: enforce # observe: journal and never block (violations become would_block) -actor: dev-laptop # for unauthenticated sessions +mode: enforce # enforce | advise | observe +unclassified_reads: block # block | observe (see above) +validate_schemas: true +actor: dev-laptop # for unauthenticated sessions (default mcp-client) permissions: [fs:read, fs:write] # scope of each session; default: every permission in the manifests upstreams: - name: github @@ -46,6 +172,7 @@ upstreams: allow: ["fs.*", "github.get_*"] # if set, only these may be called (and listed) deny: ["fs.remove_dir"] listen: { host: 127.0.0.1, port: 7431 } # omit to speak MCP on stdio +console: { mode: off } # off | open | login, with role for open resource_url: https://gateway.example.com # public URL, for OAuth protected-resource metadata auth: keys: ./keys.json @@ -54,11 +181,36 @@ allowed_hosts: [gateway.example.com] allowed_origins: [] ``` +Unknown fields are rejected, with every error reported at once. + +| Field | Default | Meaning | +|---|---|---| +| `mode` | `enforce` (`observe` if no manifests and none named) | See [modes](#modes) | +| `unclassified_reads` | `block` | `observe` lets server-claimed read-only tools without a manifest run as shadow observe runs | +| `validate_schemas` | `true` | Check arguments and structured results against the upstream's schemas | +| `console` | `{ mode: off }` | The [console](/console) on the gateway's own port. `mode`: `off`, `open`, `login`; `role` (open mode only): `auditor`, `approver`, `operator` | +| `actor` | `mcp-client` | Principal for unauthenticated sessions | +| `permissions` | all in the manifests | Scope of every session (capped by a key's `max_permissions`) | +| `upstreams[]` | required | `name`, then exactly one of `command` (+ `args`, `env`, `cwd`) or `url` (+ `headers`); `namespace` | +| `allow`, `deny` | empty | Tool-name globs | +| `listen` | stdio | `host` (default `127.0.0.1`) and `port` | +| `resource_url`, `allowed_hosts`, `allowed_origins` | derived | See below | +| `auth.keys`, `auth.oidc` | none | API key file; OIDC `issuer`, `audience`, `role_map`, optional `role_claim`, `actor_claim`, `jwks_uri` | + +Command-line flags (`reversi-gateway --help`): `--config`, `--manifests`, `--policy`, `--data`, `--store`, `--database-url`, +`--environment`, `--signing-key`, `--console`, `--console-role`, and for one upstream without a config file +`--upstream-cmd` (+ repeatable `--upstream-arg`) or `--upstream-url`, `--mode`, `--actor`, `--run-id`. Everything else +(notifications, capture, egress, anchors, reload) comes from `reversi.config.yaml`, found in the current folder or above +(or `REVERSI_CONFIG`); see [GitOps](/operations/gitops). + - **Several upstreams** are namespaced `.` by default (`namespace: false` per upstream to keep names). Manifests use the exposed names. Two upstreams exposing the same name without namespacing is an error. - `${NAME}` in `args`, `env` and `headers` expands from the environment, so secrets stay out of the file. A stdio upstream does **not** inherit the gateway's environment, only what you list. - `allow`/`deny` globs are compiled into policy rules, so they also show up as `blocked` records. +- **Egress check.** Before connecting, every `url` upstream is checked against the project's `egress:` rules (host allowlist, + `block_private`, and cloud metadata addresses, which are always refused). A refused upstream stops the gateway at start + with `upstream : `. Stdio upstreams are not network destinations and are not checked. ## Endpoints (HTTP mode) @@ -66,43 +218,66 @@ allowed_origins: [] |---|---|---| | `/mcp` | `agent` role | The agent endpoint: upstream tools plus read-only `reversi.*` tools | | `/operator/mcp` | `auditor` and up | The operator endpoint: only `reversi.*` tools, limited by role | -| `/healthz`, `/readyz` | anyone | Liveness; readiness pings every upstream | +| `/console/*` | per console mode | The [console](/console), only if `console` is `open` or `login` | +| `/healthz`, `/readyz` | anyone | Liveness; readiness pings every upstream (cached 2 s) and answers `{"ok","upstreams":[{"name","ok"}],"sessions"}`, 503 if any is down | | `/.well-known/oauth-protected-resource` | anyone | Points clients at the OIDC issuer | A 401 carries `WWW-Authenticate: Bearer resource_metadata="…"`. Tokens must have the configured `audience`. Sessions belong to one principal: using another's session id is a 403. `Host` is validated against `allowed_hosts` (default: loopback only when bound to loopback) and any `Origin` header is rejected unless allowed, which stops a web page driving a local gateway (DNS rebinding). The gateway **refuses to listen beyond loopback without authentication**, and never serves the operator endpoint without it. +Idle MCP sessions are closed after 30 minutes. + +### Console on the gateway's port + +The [console](/console) can be served by the gateway itself, on the same port and with the same off/open/login switch: +`--console open|login` (or `console:` in the gateway config or `reversi.config.yaml`). It needs HTTP mode (`listen`). + +- `open`: no login, only requests addressed to a loopback host from the same origin. The gateway refuses to start if `listen.host` is not loopback or if the role is `admin`. Default role `operator`; `--console-role` picks `auditor` or `approver`. +- `login`: paste an API key or OIDC access token; needs `auth.keys` or `auth.oidc`. +- Rollbacks started in this console run the compensations **on the upstream** with the idempotency key, and verification read-backs go through the upstream too. The Connectors page lists each upstream with its health and how many tools have manifests. Manifest and policy editing works only when `manifests:` / `policy:` point at a registry or policy-versions folder ([GitOps](/operations/gitops)). ## `reversi.*` tools On the **agent** endpoint (your own runs only): -`reversi.list_runs`, `reversi.get_run`, `reversi.get_action_trace`, `reversi.plan_rollback`, `reversi.dry_run_rollback`, -`reversi.validate_manifest`. +| Tool | Does | +|---|---| +| `reversi.list_runs`, `reversi.get_run`, `reversi.get_action_trace` | Read your runs | +| `reversi.plan_rollback`, `reversi.dry_run_rollback` | What a rollback would do. Executes nothing | +| `reversi.validate_manifest` | Validate a manifest object | +| `reversi.preview_impact` | `{tool, arguments?}`: would this call be allowed, need approval or be blocked, with risk, approvers and rollback outlook. Executes nothing | +| `reversi.search` | Find your runs, or ledger records (`what: "records"`), by tool, status, text. `limit` up to 200 | On the **operator** endpoint, by role: | Tool | Role | Does | |---|---|---| -| `reversi.list_approvals`, `reversi.export_evidence`, and the read tools above (any run) | auditor | Read | +| The read tools above (any run, any actor) | auditor | Read | +| `reversi.classification` | auditor | Every upstream tool, classified against the manifests, plus the banner text | +| `reversi.suggest_manifest` | auditor | Draft manifests for upstream tools that have none. Drafts only | +| `reversi.list_approvals`, `reversi.export_evidence` | auditor | Read approvals; ledger records, signed if the gateway has a signing key | | `reversi.approve_action`, `reversi.deny_action` | approver | Decide a request | -| `reversi.execute_rollback` | operator | Dry run unless `confirm: true`; with `approve_compensations: true` also approves compensations that need it. Calls each compensation **on the upstream**, newest first, with retries and the idempotency key in `_meta["reversi/idempotency_key"]` | +| `reversi.execute_rollback` | operator | Dry run unless `confirm: true`; with `approve_compensations: true` also approves compensations that need it. Calls each compensation **on the upstream**, newest first, 3 attempts with backoff, idempotency key in `_meta["reversi/idempotency_key"]`, read-backs through the upstream | | `reversi.halt`, `reversi.resume` | admin | Kill switch | +| `reversi.reload` | admin | Re-read manifests and policy now. Returns `{changed, tools, policy_checksum, error?}` | -The agent cannot reach any of these: they are not on its endpoint, and calling one by name fails with "unknown tool". +The agent cannot reach the operator tools: they are not on its endpoint, and calling one by name fails with "unknown tool". Compensations bypass `allow`/`deny` and scope, since an operator is deliberately undoing something: keep permissions for destructive compensation tools out of agents' scopes. -Over **stdio** the gateway is a single unauthenticated agent session. Operate it with the CLI against the same data store -(`reversi approvals`, `reversi rollback`), or run it in HTTP mode. +Over **stdio** the gateway is a single unauthenticated agent session with the agent tools only. Operate it with the CLI against +the same data store (`reversi approvals`, `reversi rollback`), or run it in HTTP mode. ## Storage and notifications The gateway uses the same stores as the server (`--store files|sqlite|postgres`), so the console, CLI and gateway can all look -at one ledger and one approval queue. `REVERSI_APPROVAL_WEBHOOK` notifies approvers. See [storage](/operations/storage). +at one ledger and one approval queue. Approval and alert [notifications](/operations/notifications), traces and anchors are +configured in `reversi.config.yaml`; `REVERSI_APPROVAL_WEBHOOK` still works. See [storage](/operations/storage). ## Limits The gateway journals and forwards `tools/call`. It does not proxy resources, prompts or sampling. `tools/list` results are cached -for five seconds. Rollback executes against the upstream the compensation tool is exposed by. +for five seconds. Rollback executes against the upstream the compensation tool is exposed by. The schema checker is a subset +of JSON Schema. The behaviour on this page was exercised against stdio fakes and the filesystem connector in tests and by +hand; no third-party hosted MCP server was used. diff --git a/docs/index.mdx b/docs/index.mdx index 8202160..27c47e1 100644 --- a/docs/index.mdx +++ b/docs/index.mdx @@ -10,22 +10,28 @@ Reversi sits between an agent and its tools and gives you: - Declare per tool: how reversible it is, what undoes it, which permission it needs. + Declare per tool: how reversible it is, what undoes it, which permission it needs. Versioned, reviewable, signable. - Undo a run newest-first, idempotently, with retries, and never claim an irreversible step was undone. + Undo a run newest-first, idempotently, with retries and read-backs, and never claim an irreversible step was undone. Who did what and why. Edits, deletions and reordering are detectable. Credentials never reach it. - Rules as code, rate limits, content checks, and human approval bound to the exact call. + Rules as code, rate limits, content checks, and human approval bound to the exact call, with quorum, delegation and break-glass. - Put it in front of any MCP server. Journal, enforce, approve and roll back without changing the agent. + Put it in front of any MCP server. Starts in observe mode with no setup; journal, enforce, approve and roll back without changing the agent. + + + See runs, decide approvals, run rollbacks, edit manifests and policy. Off by default. + + + Replay captured calls through a candidate policy and see what would change. - Ed25519-signed bundles and auditor-readable reports. + Ed25519-signed bundles, auditor reports and externally anchored ledger heads. @@ -33,17 +39,21 @@ Reversi sits between an agent and its tools and gives you: | Piece | Package | |---|---| -| Engine, stores (files, SQLite, PostgreSQL), policy, content checks, evidence | `@multimindlab/reversi` | -| `reversi` CLI | `@multimindlab/reversi-cli` | +| Engine, stores (files, SQLite, PostgreSQL), policy, content checks, evidence, registry, simulation | `@multimindlab/reversi` | +| `reversi` CLI: init, doctor, lint, test, apply, inspect, rollback, approvals | `@multimindlab/reversi-cli` | | HTTP API, operator console, metrics | `@multimindlab/reversi-server` | | MCP gateway (stdio and Streamable HTTP) | `@multimindlab/reversi-gateway` | -| Filesystem, GitHub, Slack and Stripe connectors with working compensations | `@multimindlab/reversi-connectors` | -| Python client and offline verifier (standard library only) | `reversi-sdk` | +| Fourteen connectors with working compensations and a certification suite | `@multimindlab/reversi-connectors` | +| Python client, offline verifier, Pydantic bindings, framework adapters | `reversi-sdk` | | Browser trace viewer that verifies the ledger locally | `viewer/` | +| Dockerfile, Compose stack, Helm chart | `deploy/`, `Dockerfile` | Start with the [quickstart](/quickstart), or go straight to the [gateway](/gateway) if you already run MCP servers. +To run it for a team, see [GitOps](/operations/gitops) and [deployment](/operations/deployment). Reversi records and enforces what manifests declare. It cannot discover what a tool really does. A manifest that -lies makes Reversi lie. Read [limits](/limits) and the [threat model](/threat-model). +lies makes Reversi lie. Many parts (the connectors, notification channels, framework adapters, Docker and Helm files) are +built and tested against fakes but not against real services or clusters. Read [limits](/limits), the +[roadmap](/roadmap) and the [threat model](/threat-model). diff --git a/docs/integrations.mdx b/docs/integrations.mdx new file mode 100644 index 0000000..25d0251 --- /dev/null +++ b/docs/integrations.mdx @@ -0,0 +1,229 @@ +--- +title: Framework integrations +description: Put LangGraph, CrewAI, OpenAI Agents SDK, FastMCP and Claude Code tool calls behind Reversi with the Python SDK adapters. +--- + +The adapters live in `reversi_sdk.integrations`. Each one routes tool calls through a [run](/sdks/python), so every +call is authorized, journaled before it executes, and given an outcome afterwards. The SDK core stays standard +library only: each adapter imports its framework lazily, and `import reversi_sdk` never needs one. + + +**How these were tested.** The adapters are duck-typed against small fakes of each framework's interface (a tool with +`name` and `invoke`, a `FunctionTool`-shaped dataclass, a `tool()` registrar, and so on), run against a fake Reversi +server and, for the Claude Code hook, also the real built server. They were **not** run against installed copies of +LangGraph, CrewAI, the OpenAI Agents SDK, the `mcp` package or Claude Code itself. The shapes below target the +publicly documented interfaces of the current LangChain core `BaseTool`, CrewAI `BaseTool`, OpenAI Agents SDK +`function_tool` and guardrails, MCP Python SDK `FastMCP`, and Claude Code hooks; no specific version range has been +verified. If a release changes them, the adapter is the part to adjust. Treat a first run in your own environment as +the real test. + + +Every tool name you pass must have a [manifest](/concepts/manifests) on the server, and the run must hold its +permission. A refusal is a normal result, not an error: the adapters either return an explanation the model can read +(`on_block="message"`, the default) or raise (`on_block="raise"`). When the refusal is a pending approval, the message +carries the approval id so a human can approve it and the agent can retry once. + +```python +from reversi_sdk import ReversiClient + +client = ReversiClient("http://127.0.0.1:7421", api_key="agent-key") +run = client.start_run(["orders:write"], reason="support chat 4821") +``` + +## LangGraph and LangChain + +`reversible_tool` wraps a plain function (apply it under `@tool` so LangChain still sees the signature and docstring) +or a `BaseTool`-like object that has `name` and `invoke(input)`. Async functions and `ainvoke` are supported. + +```python +from langchain_core.tools import tool +from reversi_sdk.integrations.langgraph import reversible_tool, rollback_node + +@tool +@reversible_tool(run, "create_order", reason="customer asked") +def create_order(sku: str) -> dict: + """Create an order.""" + return shop.create_order(sku) + +# or wrap an existing tool object +safe_search = reversible_tool(run, "search_orders", on_block="raise")(search_tool) +``` + +`rollback_node` returns a function you can add to a graph as a node. It runs `RunHandle.rollback` (needs an +operator key) and puts the report in the state under `reversi_rollback`: + +```python +from langgraph.graph import StateGraph + +ops = ReversiClient("http://127.0.0.1:7421", api_key="operator-key") +graph.add_node("undo", rollback_node(ops, run.run_id, compensate=undo_one_call)) +``` + +Limits: `ToolNode` and `create_react_agent` need real `BaseTool` instances. Wrap the function before `@tool`, as +above; the proxy returned for an existing tool object is not itself a `BaseTool` subclass. + +## CrewAI + +`ReversibleCrewTool` has the shape of `crewai.tools.BaseTool` (`name`, `description`, `_run`) and wraps an existing +tool. CrewAI validates its `tools=` list against the real `BaseTool` class, so call `to_crewai()` to get a genuine +subclass instance that delegates to the adapter (it imports `crewai` at that moment). + +```python +from reversi_sdk.integrations.crewai import ReversibleCrewTool + +lookup = ReversibleCrewTool(run, "lookup_order", inner=OrderLookupTool(), reason="support ticket") +agent = Agent(role="support", goal="...", backstory="...", tools=[lookup.to_crewai()]) +``` + +Limits: `to_crewai()` builds the subclass dynamically and copies `name`, `description` and `args_schema`; custom +pydantic fields on your own tool class are not carried over. + +## OpenAI Agents SDK + +Put `reversi_function_tool` under `@function_tool`. A ready `FunctionTool` is also accepted: its +`on_invoke_tool(ctx, input_json)` is wrapped and the parsed JSON arguments are journaled. + +```python +from agents import Agent, function_tool +from reversi_sdk.integrations.openai_agents import reversi_function_tool, tool_error_formatter, guardrail + +@function_tool(failure_error_function=tool_error_formatter) +@reversi_function_tool(run, "create_order", on_block="raise") +def create_order(sku: str) -> str: + return shop.create_order(sku) +``` + +With `on_block="raise"` the SDK hands the `Blocked` error to `tool_error_formatter`, which turns it into text for the +model (including the approval id). With the default `"message"` the tool simply returns that text. + +`guardrail(client, run)` returns an input guardrail function that trips while the kill switch is on +(`client.status()["halted"]`). It also trips if Reversi cannot be reached. + +```python +from agents import InputGuardrail + +agent = Agent( + name="support", + tools=[create_order], + input_guardrails=[InputGuardrail(guardrail_function=guardrail(client, run))], +) +``` + +The guardrail returns `agents.GuardrailFunctionOutput` when that package is installed. The kill switch is also +enforced at every tool call, so the guardrail only stops a halted agent earlier. + +## FastMCP + +`reversible` replaces `@mcp.tool()`: it wraps the function and then registers it. `run_factory(ctx)` returns the run +for the call; `ctx` is the tool's `Context` parameter (named `ctx` or `context`) or `None`. `per_session_runs` is a +ready factory that opens one run per MCP session. + +```python +from mcp.server.fastmcp import FastMCP, Context +from reversi_sdk.integrations.fastmcp import reversible, per_session_runs, ReversiBlocked + +mcp = FastMCP("orders") +runs = per_session_runs(client, ["orders:write"]) + +@reversible(mcp, runs) +def create_order(sku: str, ctx: Context) -> dict: + return shop.create_order(sku) +``` + +A refusal raises `ReversiBlocked` (with `approval_id` and `reasons`). FastMCP returns its message to the client as a +tool error, so the model sees the approval id. Pass extra `mcp.tool` arguments as keyword arguments to `reversible`, +or `tool_name=` when the Reversi manifest name differs from the function name. + +Limits: sessions are keyed by the identity of `ctx.session`; a run is kept in memory for the life of the process. + +## Claude Code hook + +`reversi-claude-hook` implements Claude Code's `PreToolUse` and `PostToolUse` hooks. It opens one Reversi run per +Claude session (`claude-`), asks Reversi before each tool call, and records the outcome afterwards. + +| Reversi says | Hook decision | +| --- | --- | +| `allow`, `would_block` | `allow` | +| block with a pending approval | `ask`, with the approval id in the reason | +| any other block | `deny`, with Reversi's reasons | +| server unreachable, bad key, bad input | `deny` (fail closed) | + +Claude Code tool names are used as Reversi tool names (`Bash`, `Write`, ...), and `mcp__server__tool` becomes +`server.tool`. Register a manifest for each one you want to allow; an unknown tool is blocked by the server. + +Install the package, export the configuration in the shell that starts Claude Code, and add the hooks: + +```bash +pip install reversi-sdk +export REVERSI_URL=http://127.0.0.1:7421 +export REVERSI_AGENT_KEY=... # an agent key +export REVERSI_PERMISSIONS=claude:read,claude:shell +``` + +`.claude/settings.json`: + +```json +{ + "hooks": { + "PreToolUse": [ + { "matcher": "*", "hooks": [{ "type": "command", "command": "reversi-claude-hook" }] } + ], + "PostToolUse": [ + { "matcher": "*", "hooks": [{ "type": "command", "command": "reversi-claude-hook --post" }] } + ], + "PostToolUseFailure": [ + { "matcher": "*", "hooks": [{ "type": "command", "command": "reversi-claude-hook --post" }] } + ] + } +} +``` + +Keep the agent key out of the settings file, which is usually committed. + +| Variable | Meaning | +| --- | --- | +| `REVERSI_URL`, `REVERSI_AGENT_KEY` | Server address and agent key. Required. | +| `REVERSI_PERMISSIONS` | Comma list granted to each new run. | +| `REVERSI_HOOK_FAIL_OPEN=1` | Allow calls (unrecorded) when Reversi cannot be reached, for read-only exploration. Off by default. | +| `REVERSI_HOOK_STATE` | Directory for the small state files that link a Post event to the step recorded at Pre time. Default: a temp directory. | +| `REVERSI_TIMEOUT` | Seconds to wait for the server. Default 5. | + +Limits to know about: + +- `ask` shows Claude Code's own permission prompt. If the human says yes there, the tool runs even though the Reversi + approval is still pending, and no step is recorded for that call. For a hard gate, use `deny` semantics by not + granting the tool, or have the approver decide first and let the agent retry. +- The Post hook never blocks. If it cannot reach Reversi it prints a note on stderr and the step stays unresolved, + which the server reports instead of hiding. +- Reversi can only undo what a manifest describes. Claude Code's shell and file tools are usually `irreversible` or + `idempotent`, so rollback will list them as not undoable rather than undo them. +- Hook input fields (`tool_use_id`, `PostToolUseFailure`) follow Claude Code's documented format; without a + `tool_use_id` the hook falls back to matching on the tool name and input. + +## Webhook signatures + +If your Reversi server signs outbound webhooks, verify them with the raw body bytes before parsing: + +```python +from reversi_sdk import verify_webhook_signature + +ok = verify_webhook_signature(secret, request_body_bytes, request.headers.get("X-Reversi-Signature")) +``` + +The header has the form `sha256=`, the HMAC-SHA256 of the body under the shared secret. The comparison is +constant time, and a missing or malformed header returns `False`. The header name depends on the server +configuration; the helper only checks the value you pass. + +## Optional pydantic models + +`reversi_sdk.pydantic_models` provides pydantic v2 `ToolManifest` and `Compensation` models with the same rules as +the standard library validator. Install with `pip install 'reversi-sdk[pydantic]'`. The SDK does not depend on pydantic +and the standard library validator (`validate_manifest`) stays authoritative; a test checks both against the shared +conformance cases. + +```python +from reversi_sdk.pydantic_models import ToolManifest + +m = ToolManifest.model_validate(raw) # raises pydantic.ValidationError +manifest = m.normalized() # the same dict validate_manifest returns +``` diff --git a/docs/limits.mdx b/docs/limits.mdx index 0c0abfc..e8b0d8d 100644 --- a/docs/limits.mdx +++ b/docs/limits.mdx @@ -1,27 +1,49 @@ --- title: Limits -description: What Reversi does not do. Read this before relying on it. +description: What Reversi does not do, and what is built but not yet checked against real services. Read this before relying on it. --- +## What Reversi cannot do + - **Manifests are trusted.** Reversi cannot inspect what a tool really does. Wrong manifest, wrong guarantees. It also cannot - discover side effects nobody declared. + discover side effects nobody declared. MCP annotations (`readOnlyHint` and friends) are the server's own claims: they seed + draft manifests and the startup banner, and nothing more ([gateway](/gateway)). - **Compensation is not time travel.** Refunds leave statements, deletions can lose data, a Slack message may already have been read. Irreversible stays irreversible, and compensation can fail: the report says so. -- **Tamper-evident, not tamper-proof.** Anchor the ledger head externally. Whole-file rewrites and tail truncation are not - detectable from the data alone ([evidence](/concepts/evidence)). +- **Tamper-evident, not tamper-proof.** Anchor the ledger head externally (`reversi anchor publish`, or `anchor:` sinks in the project file). Whole-file rewrites and tail truncation are not + detectable from the data alone ([evidence](/concepts/evidence)). Retention deletes run state, approvals and captures, never ledger records. - **Policy regular expressions** are run against agent-supplied values. A pathological pattern written by a policy author can stall the process; keep patterns simple (policy files are trusted input). - **The HTTP API trusts the agent's reported outcomes** (results feed compensation arguments). Use the gateway when results must come from the system, not the agent. - **Content checks are heuristics.** They catch obvious credentials, hidden payloads and injection phrasing. A determined adversary can phrase around them. They are one layer, not a firewall. +- **Schema checks are a subset.** The gateway checks arguments and structured results against the upstream's published JSON Schema + with a small checker (types, enums, required, bounds, patterns, composition). It does not follow `$ref` or `if/then/else`, and it can only check what the upstream declares. - **Not an agent sandbox.** An agent that can call a tool directly, bypassing Reversi, is not governed. Put Reversi in the only path. - **One driver per run.** Several processes can share a SQLite or PostgreSQL store and append to one ledger safely, but run state is - last-write-wins: do not drive the same run from two processes. With the `files` store, share the directory only between processes on one host, never over a network filesystem. -- **Gateway scope.** It proxies `tools/call` and `tools/list`, not resources, prompts or sampling, and one session is one run. + last-write-wins: do not drive the same run from two processes. With the `files` store, share the directory only between processes on one host, never over a network filesystem. There is no clustering with run hand-off. +- **Gateway scope.** It proxies `tools/call` and `tools/list`, not MCP resources, prompts or sampling, and one session is one run. - **HTTP only.** Terminate TLS in front of the server and gateway. -- **Console cannot run compensations.** It shows the plan; execution happens through the CLI, API or gateway. -- **No OpenTelemetry export, no retention or purge policy, no hosted multi-tenant service.** Metrics, JSON logs and the ledger are - the operational record. Retention is yours to define with the store's own tools, and the ledger is append-only by design. -- **Connectors cover four systems** (filesystem, GitHub, Slack, Stripe) and were tested against fakes of those APIs, not live - services. Verify each against a sandbox account before trusting it ([connectors](/connectors)). -- **Python SDK** reads JSON manifests only, rejects some floats when hashing payloads, and cannot verify Ed25519 bundles. -- **Docker and CI files** were written but not executed in the project's own test pass. +- **Console sign-in is a pasted credential.** Login mode takes an API key or an OIDC access token. There is no browser redirect login to an identity provider. The open console has no login at all and is limited to loopback and a non-admin role. +- **Rollbacks from the console depend on a compensator.** The server console can execute a rollback only when `REVERSI_COMPENSATOR_CMD` is set. Declared read-backs run only if `REVERSI_READER_CMD` is set too; without it those steps stay `unverified`. The gateway console runs compensations and read-backs on the upstream. `reversi console` only shows the plan. +- **No hosted service.** Nothing here runs as a managed, multi-tenant control plane. +- **No Terraform provider and no Kubernetes operator.** Use `reversi apply` from CI and the Helm chart ([GitOps](/operations/gitops)). +- **SDKs.** TypeScript and Python only; no Go or Java. The Python SDK reads JSON manifests only, rejects non-whole floats outside 1e-4 to 1e16 when hashing payloads, and cannot verify Ed25519 bundles (use `reversi verify-bundle`). +- **Standalone server routes.** `/v1/connectors` is empty (the gateway console fills it in). +- **`manifests publish --status` overrides a `status:` in the file.** The flag (default `draft`) decides, so a file that says `status: approved` still lands as a draft unless you pass `--status approved`. +- **A draft in a registry is absent, not blocked.** Enforcement loads only the newest non-draft version of each tool, so a call to a tool that has only a draft reports "no manifest", not "is a draft". +- **`--detailed-exitcode`** exits 2 only together with `--dry-run`. +- **`reversi.classification` and `reversi.suggest_manifest`** are tools of the gateway's operator endpoint (`/operator/mcp`) only. + +## Built, but not verified against the real thing + +These are implemented and covered by tests that use fakes or local stand-ins, but have not been run against real services, +clusters or accounts. Do a first run in a throwaway environment. + +- **Fourteen connectors** (filesystem, GitHub, GitLab, Slack, Teams, Linear, Jira, Stripe, Google Calendar, Postgres, S3, Cloud Storage, Blob Storage, Kubernetes) and their certification suite run against fakes written from each vendor's documentation, not live services ([connectors](/connectors)). The filesystem connector is the exception: it is real and local. +- **Notification channels** (Slack, Teams, email over SMTP, PagerDuty, Opsgenie, signed webhook) and **OpenTelemetry export** were tested against local fake receivers, not the real products ([notifications](/operations/notifications)). +- **Framework integrations** (LangGraph, CrewAI, OpenAI Agents SDK, FastMCP, Claude Code hook) are tested against small fakes of each framework's interface, not installed copies ([integrations](/integrations)). +- **Dockerfile, Docker Compose stack and Helm chart** were written but not built or installed. `helm` was not available to lint the chart, and it has never been installed on a real cluster ([deployment](/operations/deployment)). +- **GitHub workflows** (`.github/workflows`) and the example workflow in [GitOps](/operations/gitops) have not been run on GitHub. +- **PostgreSQL** was exercised against a local instance; managed cloud PostgreSQL services were not. +- **Third-party MCP servers.** The gateway was run against stdio test servers and the bundled filesystem connector, not a range of hosted MCP servers. +- **An external security review** has not happened. See the [threat model](/threat-model) and `SECURITY.md`. diff --git a/docs/logo/dark.png b/docs/logo/dark.png new file mode 100644 index 0000000..7b7094a Binary files /dev/null and b/docs/logo/dark.png differ diff --git a/docs/logo/favicon.png b/docs/logo/favicon.png new file mode 100644 index 0000000..83f5ff8 Binary files /dev/null and b/docs/logo/favicon.png differ diff --git a/docs/logo/light.png b/docs/logo/light.png new file mode 100644 index 0000000..7b7094a Binary files /dev/null and b/docs/logo/light.png differ diff --git a/docs/operations/authentication.mdx b/docs/operations/authentication.mdx index 2a43680..9c5b54f 100644 --- a/docs/operations/authentication.mdx +++ b/docs/operations/authentication.mdx @@ -1,6 +1,6 @@ --- title: Authentication and roles -description: API keys with rotation, OIDC bearer tokens, and what each role may do. +description: API keys with rotation and groups, OIDC bearer tokens, console modes, and what each role may do. --- | Role | Capabilities | @@ -9,7 +9,7 @@ description: API keys with rotation, OIDC bearer tokens, and what each role may | `auditor` | Read runs, plans, ledger, evidence, metrics | | `approver` | Auditor + decide approvals | | `operator` | Approver + rollback, resolve | -| `admin` | Operator + kill switch | +| `admin` | Operator + kill switch, break-glass, holds, retention, anchors, key management | A principal may have several roles. Agents are scoped to their own runs; every other role sees all runs. @@ -17,13 +17,25 @@ A principal may have several roles. Agents are scoped to their own runs; every o ```json keys.json [ - { "name": "support-agent", "role": "agent", "key_sha256": "", "max_permissions": ["orders:write"] }, - { "name": "alice", "role": "approver", "key_sha256": "…" }, + { "name": "support-agent", "role": "agent", "key_sha256": "", "max_permissions": ["orders:write"], "tenant": "acme" }, + { "name": "fiona", "role": "approver", "groups": ["finance"], "key_sha256": "…" }, { "name": "ops", "roles": ["operator"], "key_sha256": "…", "not_before": "2026-11-01T00:00:00Z", "expires_at": "2027-05-01T00:00:00Z" } ] ``` -`max_permissions` caps what an agent's runs may request. Agents open runs in `enforce` mode only; set `allow_observe: true` on a key to let it open `observe` runs (which let violating calls proceed, so keep it for lab agents). +| Field | Meaning | +|---|---| +| `name` | The caller. For agents it is the `actor` recorded in the ledger. Several keys may share a name | +| `role` or `roles` | One role, or a list. Required | +| `key_sha256` | Lowercase hex SHA-256 of the key (64 characters). Or `key` for a plain key; prefer the hash | +| `groups` | Group names for [approval routing](/concepts/approvals), such as `finance` | +| `tenant` | The customer or team. Runs this key opens carry it (used by policy, holds and retention) | +| `max_permissions` | Caps what an agent's runs may request | +| `allow_observe` | Let this agent key open `observe` runs. Default false: agents open `enforce` runs only. Observe lets violating calls proceed, so keep it for lab agents | +| `not_before`, `expires_at` | ISO date or epoch milliseconds. The key is rejected before and from these times | +| `disabled` | `true` switches the key off immediately | + +A key needs `key` or `key_sha256` and a valid role, or the whole file is rejected. Hash keys (`printf '%s' "$KEY" | shasum -a 256`) rather than storing them. Generate keys with at least 128 bits of entropy (`openssl rand -hex 24`). Keys are compared in constant time across all entries. @@ -32,6 +44,21 @@ Hash keys (`printf '%s' "$KEY" | shasum -a 256`) rather than storing them. Gener hand out the new key, and remove the old entry later. The server reloads the file on `SIGHUP` or when it changes (checked every two seconds); a reload that fails to parse keeps the old keys. `disabled: true` switches a key off immediately. +### Managing keys from the server + +When the server runs with `REVERSI_KEYS`, an admin can manage the file over HTTP or in the console instead of editing it. +The server rewrites the file (mode `0600`) and applies the change at once. + +| Call | Does | +|---|---| +| `GET /v1/keys` | Lists keys with index, name, roles, groups, tenant, dates, `disabled` and a 12-character `fingerprint`. Never the key or its hash | +| `POST /v1/keys` | Body `name`, `role` or `roles`, optional `groups`, `tenant`, `max_permissions`, `expires_in_days`. Returns a new `rv_…` key **once** | +| `PATCH /v1/keys/{index}` | `disabled: true or false`, `expires_at` (a date, or `null` to clear) | +| `DELETE /v1/keys/{index}` | Removes the key. Later indexes shift down, so list again first | + +All four need the `admin` role. In the console, **Settings** has a Users and API keys table with create (name, role, +groups, tenant, days to expiry), disable, enable and remove; a new key is shown once, in the response. + ## OIDC Bearer JWTs from your identity provider: @@ -43,13 +70,63 @@ REVERSI_OIDC_ROLE_MAP='{"reversi-ops":"operator","reversi-approvers":"approver", REVERSI_OIDC_ROLE_CLAIM=realm_access.roles # default: roles ``` -Checked on every request: signature against the issuer's JWKS (fetched from the discovery document, cached, refreshed at most -every 10 seconds when an unknown `kid` appears), **RS256 or ES256 only** (never `none`, never HMAC), `iss`, `aud` containing your -audience, `exp` and `nbf` (60 s skew). The principal's name is `sub`, or the claim named by `REVERSI_OIDC_ACTOR_CLAIM`. Tokens with no mapped role are rejected. The audience check is what stops a token minted for another service -from working here: set it. +Checked on every request: signature against the issuer's JWKS, **RS256 or ES256 only** (never `none`, never HMAC), a `kid` in +the header, `iss`, `aud` containing your audience, `exp` and `nbf` (60 s skew). The audience check is what stops a token +minted for another service from working here: it is required. + +- **JWKS.** Found through the issuer's discovery document (`/.well-known/openid-configuration`) and cached. The key set is + re-read every 10 minutes even for known keys, so a key the issuer withdrew stops verifying, and when an unknown `kid` + appears, at most once every 10 seconds so a flood of bad tokens cannot hammer the issuer. +- **Roles.** The role claim may be a string, a list, or a nested path like `realm_access.roles`. Each value is looked up in + the role map; unmapped values are ignored, and a token with no mapped role is rejected. +- **Groups.** Every value of the role claim also counts as a group name, so an approval can be routed to an identity + provider group (`approvers: [finance-approvers]`) with no extra mapping. +- **Name.** `sub`, or the claim named by `REVERSI_OIDC_ACTOR_CLAIM`. +- **Tenant.** The verifier can read a tenant from a claim (`tenant_claim` in the library), but the server has no setting for + it yet, so OIDC principals carry no tenant there. The server also accepts API keys at the same time, which is handy for service accounts. -## Console login +## Console modes + +The [console](/console) is controlled by `REVERSI_CONSOLE` (or `console.mode` in `reversi.config.yaml`). + +| Mode | Meaning | +|---|---| +| `off` (default) | Nothing is served under `/console`; it answers 404 with a note on how to turn it on | +| `open` | No login. Acts as one fixed role (`REVERSI_CONSOLE_ROLE`: `auditor`, `approver` or `operator`; default `operator`). For your own machine; `reversi console` starts it | +| `login` | Paste an API key or an OIDC access token with a console role. Agent-only credentials are refused | + +`open` refuses to start unless `HOST` is loopback (`127.0.0.1`, `::1` or `localhost`), and refuses `admin` as its role. While +running it answers only requests whose `Host` is a loopback name and whose `Origin`, if any, is the same host; anything else +gets 403. So key management, break-glass, holds and the kill switch are not available in `open`: use `login` for those. + +OIDC browser redirect login (a "Sign in with SSO" button) is not built. In `login` mode a person pastes an OIDC access +token; there is no redirect flow. + +### Sessions + +A login produces a signed cookie (HMAC-SHA-256, `HttpOnly`, `SameSite=Strict`, eight hours). For API keys the cookie holds the +key's **fingerprint**, and every request looks the fingerprint up in the live key ring. Disabling, expiring or removing the +key ends the session at once, and changing its roles or groups changes what the session may do on the next click. An OIDC +session never outlives its token. Logging out invalidates the cookie. Set `REVERSI_SESSION_SECRET` so sessions survive a +restart, and `REVERSI_SECURE_COOKIES=1` behind HTTPS. + +## Slack approvals: user mapping + +Slack buttons approve as a Reversi person. The Slack user id is looked up in a map you provide. `reversi-server` turns the +callback (`POST /integrations/slack/actions`) on when both `REVERSI_SLACK_SIGNING_SECRET` and `REVERSI_SLACK_USERS` are set; +`REVERSI_SLACK_USERS` is the path of a JSON file: + +```json +{ "U123": { "name": "fiona", "roles": ["approver"], "groups": ["finance"] } } +``` + +Without the secret the route answers `404 Slack approvals are not enabled`; a secret without the map file stops the server at +start. In code the same map is `createApp({ slack: { signing_secret, users } })`. -Paste a key or token into the [console](/console) login form. A session is a signed cookie, re-checked against the key ring on every request: disabling, expiring or demoting the key ends or narrows the session at once. An OIDC session never outlives its token. Logging out invalidates the cookie. +`roles` defaults to `["approver"]`. A mapped user whose roles do not include `approver` or higher (an `auditor`, say) cannot +decide: they get the same "not set up as an approver" answer as an unmapped user. The request must carry a valid Slack +signature and a timestamp within five minutes (otherwise 401). Checked locally with signed requests: an unmapped user and a +mapped auditor are refused, a mapped approver decides the request. Not tested against real Slack. See +[approvals](/concepts/approvals). diff --git a/docs/operations/deployment.mdx b/docs/operations/deployment.mdx index c8efe1f..acfc66e 100644 --- a/docs/operations/deployment.mdx +++ b/docs/operations/deployment.mdx @@ -1,11 +1,20 @@ --- title: Deployment -description: Run the server and gateway in Docker or directly, configure them, and put TLS in front. +description: Run the server and gateway in Docker, Docker Compose or Kubernetes (Helm), configure them, and put TLS in front. --- + +**What has and has not been run.** The Dockerfile, the Compose file and the Helm chart were written and reviewed, but not +built, started or installed in the project's own test pass. In particular the Helm chart has **not been installed on a real +cluster**, and `helm` was not available on the machine that wrote it, so the chart has not even been `helm lint`ed or +rendered. Treat all three as a starting point, run `helm lint` and `helm template` yourself, and try them in a +non-production environment first. + + ## Docker -The repository's `Dockerfile` builds one image with the server (default command), the gateway and the CLI. +The repository's `Dockerfile` builds one image with the server (default command), the gateway and the CLI. It is a two-stage +build on `node:22-slim` (`npm ci`, `tsc -b`, then dev dependencies are pruned). ```bash docker build -t reversi . @@ -17,34 +26,123 @@ docker run --rm -p 127.0.0.1:7421:7421 -v reversi-data:/data \ docker run --rm reversi node packages/gateway/dist/main.js --config /config/gateway.yaml --manifests /config/manifests ``` -It runs as the unprivileged `node` user, keeps data in the `/data` volume, and has a health check on `/readyz`. +- Runs as the unprivileged `node` user, keeps data in the `/data` volume (`REVERSI_DATA=/data`), and has a health check on `/readyz`. +- It sets `HOST=0.0.0.0` and `PORT=7421`. Because of that, `REVERSI_CONSOLE=open` is refused (open mode is loopback only); use `REVERSI_CONSOLE=login`. +- A gateway that listens beyond loopback needs `auth` in its config, or it refuses to start. + +## Docker Compose + +`deploy/docker-compose.yml` runs the server on PostgreSQL 16 with a Postgres health check, JSON logs, and the **login console** +at `http://127.0.0.1:7421/console` (paste an API key). Follow the comments at its top: + +```bash +node packages/connectors/dist/main.js manifests fs --format yaml > examples/gateway/manifests/fs.yaml # after npm run build +cp deploy/keys.example.json deploy/keys.json # replace the placeholder hashes: printf '%s' "$KEY" | shasum -a 256 +POSTGRES_PASSWORD=$(openssl rand -hex 16) docker compose -f deploy/docker-compose.yml up --build +``` + +The stack mounts `deploy/keys.json`, `examples/gateway/manifests` and `examples/gateway/policy.yaml` read-only, publishes only on +`127.0.0.1`, and requires `POSTGRES_PASSWORD` (it fails without it). Replace the example manifests and policy with your own. -`deploy/docker-compose.yml` runs the server on PostgreSQL with a Postgres health check; follow the comments at its top -(generate the example manifests, copy and fill `deploy/keys.json`, set `POSTGRES_PASSWORD`). +## Kubernetes (Helm) - -The Docker files and compose stack are provided but were not built or run in the project's own test pass. Build and run them -in your environment before relying on them. - +The chart is at `deploy/helm/reversi`. It deploys the server, and optionally the gateway as a second container in the same pod. + +```bash +helm lint deploy/helm/reversi # not run by the author: helm was unavailable +helm template reversi deploy/helm/reversi -f my-values.yaml +helm install reversi deploy/helm/reversi -f my-values.yaml +``` + +Build and push the image yourself and set `image.repository` and `image.tag`. Nothing is created for secrets: you create the +Secrets and ConfigMaps and the chart refers to them by name, so no credential ever sits in `values.yaml`. + +```yaml my-values.yaml +image: { repository: registry.example.com/reversi, tag: "0.1.0" } +server: { console: login, environment: production } +manifests: { existingConfigMap: reversi-manifests } +policy: { existingConfigMap: reversi-policy } +keys: { existingSecret: reversi-keys } # a Secret with key keys.json +store: + kind: postgres + databaseUrlSecret: { name: reversi-db, key: DATABASE_URL } +ingress: { enabled: true, className: nginx, host: reversi.example.com, tls: { enabled: true, secretName: reversi-tls } } +``` + +### Values + +| Group | Key | Default | Meaning | +|---|---|---|---| +| `image` | `repository`, `tag`, `pullPolicy` | `reversi`, `0.1.0`, `IfNotPresent` | The image built from the Dockerfile | +| (top level) | `replicaCount` | `1` | More than one needs `store.kind: postgres` | +| `server` | `port` | `7421` | Container and Service port | +| | `console` | `off` | `off` or `login`. `open` is refused | +| | `environment` | `production` | `REVERSI_ENV`, matched by policy rules | +| | `logJson` | `true` | `REVERSI_LOG=json` | +| | `secureCookies` | `true` | `REVERSI_SECURE_COOKIES=1`; keep on behind TLS | +| `manifests` | `existingConfigMap` or `inline` | none | A ConfigMap of manifest files, or `{ "orders.yaml": "" }` rendered into one. Mounted read-only at `/config/manifests`, sets `REVERSI_MANIFESTS` | +| `policy` | `existingConfigMap` or `inline` | none | A ConfigMap holding `policy.yaml`, or the text inline. Mounted at `/config/policy`, sets `REVERSI_POLICY` | +| `registry` | `persistence.enabled`, `.size`, `.storageClass` | `false`, `1Gi`, none | A writable PersistentVolumeClaim (`-reversi-state`, ReadWriteOnce) mounted at `/state` in the server container. The server then reads manifests from `/state/manifests` and policy from `/state/policy` (`REVERSI_MANIFESTS`, `REVERSI_POLICY`) instead of the ConfigMap mounts. Needs `replicaCount: 1` | +| `keys` | `existingSecret`, `key` | none, `keys.json` | A Secret with the API key file (hashes; see `deploy/keys.example.json`). Sets `REVERSI_KEYS` | +| `store` | `kind` | `postgres` | `files`, `sqlite` or `postgres` | +| | `databaseUrlSecret.name`, `.key` | none, `DATABASE_URL` | Secret holding the PostgreSQL URL (required for postgres) | +| | `persistence.enabled`, `.size`, `.storageClass` | `false`, `5Gi` | A PersistentVolumeClaim for `files` and `sqlite`. Otherwise `/data` is an `emptyDir` and is lost with the pod | +| `oidc` | `enabled`, `issuer`, `audience`, `roleMap` | off | Sets the `REVERSI_OIDC_*` variables | +| `signing` | `existingSecret` | none | A Secret holding `signing.pem`, an Ed25519 key, for signed evidence | +| (top level) | `extraEnv` | `[]` | More environment entries, e.g. `{ name: REVERSI_CAPTURE, value: "1" }` | +| `gateway` | `enabled`, `port`, `existingConfigMap`, `console` | `false`, `7611`, none, `off` | Adds a gateway container. `existingConfigMap` must hold `gateway.yaml` (mounted at `/config/gateway`); the gateway console mode is passed as `--console` | +| `ingress` | `enabled`, `className`, `host`, `tls.enabled`, `tls.secretName`, `annotations` | off, `reversi.example.com`, TLS on | One Ingress to the server's HTTP port | +| (top level) | `serviceAccount.create`, `serviceAccount.annotations`, `resources`, `podAnnotations`, `nodeSelector`, `tolerations`, `affinity` | | As usual | + +### Fail-fast checks + +`templates/validate.yaml` stops the render with a message for: + +- `server.console=open`: "has no login and is refused on a cluster". +- `replicaCount` above 1 with a store other than `postgres`. +- `registry.persistence.enabled` with `replicaCount` above 1: "the volume is ReadWriteOnce". +- `server.console=login` with neither `keys.existingSecret` nor `oidc.enabled`. + +The templates also require `store.databaseUrlSecret.name` when the store is `postgres` (the default), and +`gateway.existingConfigMap` when the gateway is enabled. + +### What the chart does and does not do + +- Pod security: non-root user 1000, read-only root filesystem (writable `/data` and `/tmp`), all capabilities dropped, `RuntimeDefault` seccomp, no service account token mounted. Readiness probe `/readyz`, liveness `/healthz`. +- The server refuses to start without a credential source. The chart does not check that you set `keys.existingSecret` or `oidc.enabled` unless the console is `login`. +- The gateway container shares the pod's environment (store, manifests, policy, keys) and `/data`. Your `gateway.yaml` must listen on `0.0.0.0` and the port set in `gateway.port`, and use `auth.keys: /config/keys/keys.json` (or OIDC), since a gateway beyond loopback needs authentication. The Service exposes the `mcp` port inside the cluster; the Ingress routes only to the server. +- **Read-only by default.** With the ConfigMap sources, manifests and policy are read-only mounts, so the console's editors and `reversi apply --server` have nothing writable to work on. Update the ConfigMaps (`helm upgrade`) and let [hot reload](/operations/gitops) pick them up. I did not verify how quickly Kubernetes propagates a ConfigMap change into the pod or that the file watch sees it, so run `kill -HUP` or restart if it does not. +- **Writable registry (`registry.persistence.enabled: true`).** The console editors and `reversi apply --server` then work, because the server's manifests and policy are folders on the volume. Set `server.console: login` and use an admin key for `apply`. An init container creates `/state/manifests` and `/state/policy`, and the server and gateway treat an empty folder as an editable registry, so the first publish (for example `reversi apply --server --token `) creates it. The `/state` mount is in both the server and the gateway container. Checked locally with an empty folder and `apply --server`; the init container and mounts were read from the templates, not run on a cluster. +- There is no Terraform module and no operator. See [GitOps](/operations/gitops). ## Server environment +Everything can also be set in [`reversi.config.yaml`](/operations/gitops); the variable wins. + | Variable | Meaning | |---|---| -| `REVERSI_MANIFESTS` | Manifest file or directory (required) | -| `REVERSI_KEYS` | API key file; reloaded on `SIGHUP` or change | -| `REVERSI_OIDC_*` | OIDC: `ISSUER`, `AUDIENCE`, `ROLE_MAP`, `ROLE_CLAIM`, `ACTOR_CLAIM` ([authentication](/operations/authentication)) | -| `REVERSI_DATA`, `REVERSI_STORE`, `REVERSI_DATABASE_URL` | [Storage](/operations/storage) | -| `REVERSI_POLICY`, `REVERSI_ENV` | [Policy](/concepts/policy) file and environment name | +| `REVERSI_MANIFESTS` | Manifest file, folder, registry folder or https URL (required unless `manifests:` is in the project file) | +| `REVERSI_KEYS` | API key file; reloaded on `SIGHUP` or change. Also the file the console edits | +| `REVERSI_OIDC_ISSUER`, `_AUDIENCE`, `_ROLE_MAP`, `_ROLE_CLAIM`, `_ACTOR_CLAIM` | OIDC ([authentication](/operations/authentication)). Role map is JSON such as `{"reversi-ops":"operator"}` | +| `REVERSI_DATA`, `REVERSI_STORE`, `REVERSI_DATABASE_URL` | [Storage](/operations/storage). Defaults `./reversi-data`, `files` | +| `REVERSI_POLICY`, `REVERSI_ENV` | [Policy](/concepts/policy) file or versions folder, and environment name (default `default`) | | `REVERSI_KILL_FILE` | While this file exists, new calls are blocked | -| `REVERSI_SIGNING_KEY` | Ed25519 private key (PEM) for [signed bundles](/concepts/evidence) | -| `REVERSI_APPROVAL_WEBHOOK`, `REVERSI_CONSOLE_URL` | [Approval notifications](/concepts/approvals) | -| `REVERSI_SESSION_SECRET`, `REVERSI_SECURE_COOKIES` | [Console](/console) sessions | +| `REVERSI_SIGNING_KEY`, `REVERSI_MANIFEST_PUBLIC_KEY` | Ed25519 private key (PEM file) for [signed bundles](/concepts/evidence); public key manifests must be signed with | +| `REVERSI_APPROVAL_WEBHOOK`, `REVERSI_CONSOLE_URL` | [Approval notifications](/concepts/approvals) (sent through the egress guard) and the public URL linked from them | +| `REVERSI_CONFIG` | Project file; otherwise searched upward from the working directory | +| `REVERSI_CONSOLE`, `REVERSI_CONSOLE_ROLE` | [Console](/console) mode (`off`, `open`, `login`) and the role used in open mode | +| `REVERSI_COMPENSATOR_CMD` | Shell command run for each undo, call as JSON on stdin, exit 0 = done. Needed for rollbacks from the console and `POST /v1/runs/{id}/rollback` | +| `REVERSI_READER_CMD` | Shell command that runs a manifest's declared read-back after an undo: gets `{"tool","arguments"}` as JSON on stdin and prints the JSON result. Without it those steps are `unverified` | +| `REVERSI_SLACK_SIGNING_SECRET`, `REVERSI_SLACK_USERS` | Approve from Slack buttons: the signing secret, and the path of a JSON file mapping Slack user ids to `{"name","roles","groups"}` ([authentication](/operations/authentication)). Both are needed | +| `REVERSI_CAPTURE`, `REVERSI_CAPTURE_TTL_DAYS` | Keep redacted call arguments for replay and simulation | +| `REVERSI_OTLP_ENDPOINT` | Export OpenTelemetry traces ([monitoring](/operations/monitoring)) | +| `REVERSI_EGRESS_ALLOW`, `REVERSI_EGRESS_BLOCK_PRIVATE`, `REVERSI_REGION` | Where Reversi may send data; comma-separated host globs, `1`, a region label | +| `REVERSI_SESSION_SECRET`, `REVERSI_SECURE_COOKIES` | Console session signing; `1` behind HTTPS | | `REVERSI_METRICS_PUBLIC=1` | Serve `/metrics` without credentials (private networks only) | | `REVERSI_LOG=json` | One JSON line per request on stdout | | `PORT`, `HOST` | Listen address (default `127.0.0.1:7421`) | -At least one credential source (keys or OIDC) is required: the server will not start unauthenticated. +At least one credential source (keys or OIDC) is required: the server will not start unauthenticated. `node packages/server/dist/main.js --help` prints the list. ## TLS and networking @@ -56,21 +154,25 @@ agent endpoints. Set `allowed_hosts` on the gateway to your public hostname. One engine process per run is the supported model. With PostgreSQL you can run several server or gateway instances, each owning its own sessions and runs, behind a load balancer with session affinity for MCP sessions (the `Mcp-Session-Id` header). Ledger -appends from all of them stay on one chain. Active/active drivers of the *same* run are not supported. +appends from all of them stay on one chain. Active/active drivers of the *same* run are not supported, and there is no +clustering with run hand-off. ## Upgrades -Deploy the new version: PostgreSQL migrations run at start. Back up first ([storage](/operations/storage)). Manifests and -policy files are read at start; restart to load changes. Key files reload live. +Deploy the new version: PostgreSQL migrations run at start. Back up first ([storage](/operations/storage)). Manifests, the +policy and the manifest public key reload while running (file watch, `SIGHUP`, `POST /v1/reload`); other settings and +environment variables need a restart. Key files reload live. ## Incident runbook | Situation | Do | |---|---| -| Agent is misbehaving | Kill switch: console, `POST /v1/halt`, `reversi.halt`, or `reversi kill-switch on `. Rollbacks still work | -| Need to know what it did | Console run page, `reversi export --run`, or a signed bundle | -| Undo it | `reversi plan`, then `reversi rollback` / `reversi.execute_rollback`. Review `cannot be undone` and `unresolved` items by hand | +| Agent is misbehaving | Kill switch: console Evidence page, `POST /v1/halt` (can be scoped to a tool, actor, tenant or environment), `reversi.halt`, `reversi halt add`, or `reversi kill-switch on `. Rollbacks still work | +| Need to know what it did | Console run page, `reversi runs show`, `reversi export --run`, or a signed bundle | +| Undo it | `reversi plan`, then `reversi rollback`, the console Run the rollback button, or `reversi.execute_rollback`. Review `cannot be undone` and `unresolved` items by hand | | A step has an unknown outcome | Check the downstream system, then `reversi resolve` with a note | -| Compensation keeps failing | Read the error in the ledger (`compensation_failed`), fix the cause, re-run the rollback: it retries with the same idempotency key | -| Suspect ledger tampering | `reversi verify`, compare the head with your anchored copy | -| Key leaked | Set `disabled: true` (reloads within seconds) or remove it; review that principal's runs in the ledger | +| Compensation keeps failing | Read the error in the ledger (`compensation_failed`), fix the cause, re-run the rollback: it retries with the same idempotency key. Escalate to on-call from the run page | +| An approval is stuck and the incident is live | An admin can use break-glass (reason and ticket recorded) | +| Suspect ledger tampering | `reversi verify`, compare the head with your anchored copy (`reversi anchor verify`) | +| Key leaked | Set `disabled: true` (reloads within seconds) or disable it in the console; review that principal's runs in the ledger | +| A bad manifest or policy was published | `reversi manifests rollback ` or `reversi policy activate `; both are new versions, nothing is lost | diff --git a/docs/operations/gitops.mdx b/docs/operations/gitops.mdx new file mode 100644 index 0000000..ed0fab7 --- /dev/null +++ b/docs/operations/gitops.mdx @@ -0,0 +1,228 @@ +--- +title: GitOps +description: Keep manifests and policy in git, check them in CI, and make a registry match with `reversi apply`. The project file, hot reload, and why there is no Terraform provider. +--- + +Reversi's configuration is files. Keep them in git, review changes in pull requests, test them in CI, and apply the merged +result with one command. This page covers the project file, the versioned folders, `reversi apply`, hot reload and a CI example. + + +There is **no Terraform provider and no Kubernetes operator**. Neither is built. `reversi apply` (below) plus the +[Helm chart](/operations/deployment) is the alternative: a directory in git is the desired state, `apply` publishes only the +differences, and `--dry-run --detailed-exitcode` is the drift check. + + +## The project file: `reversi.config.yaml` + +The CLI, the server and the gateway all read it. It is found in the current folder or any parent (or `REVERSI_CONFIG`, or +`--config`). Every key is optional. `reversi init` writes a starter, and `reversi show-config` prints what was found. + +```yaml reversi.config.yaml +manifests: ./manifests # a folder of YAML/JSON, a registry folder, or an https URL of a served registry +policy: ./policy.yaml # a file, or a policy versions folder +data: + dir: ./reversi-data + store: sqlite # files | sqlite | postgres + database_url: ${DATABASE_URL} # only for postgres +environment: dev # matched by policy rules +kill_switch_file: ./KILL # while this file exists, new calls are blocked +approval: + ttl: 24h # lifetime of an approval request + webhook: ${APPROVAL_WEBHOOK} + console_url: https://reversi.example.com +signing_key: ./sign.key # Ed25519 private key for signed evidence +manifest_public_key: ./sign.pub # manifests must be signed by this key +capture: { enabled: true, ttl: 7d, redact_pii: true } +console: { mode: off } # off | open | login; open takes role +reload: { watch: true, interval: 2s } +``` + +The rest of the keys are described on their own pages: `notifications` (channels and routes, +[notifications](/operations/notifications)), `otel` (`endpoint`, `headers`, `service_name`, [monitoring](/operations/monitoring)), +`anchor` (`sinks` of `{type: file, path}` or `{type: webhook, url, secret}`, plus `interval`, [evidence](/concepts/evidence)), +`egress` (`allow`, `block_private`, `allow_metadata`, `region`) and `alerts` (`interval`, `remind`, `denial_spike`). + +| Key | Meaning | +|---|---| +| `manifests`, `policy` | Where they live. A folder created by `reversi manifests publish` / `reversi policy publish` is a **registry** / **policy versions** folder (see below) | +| `data.dir`, `data.store`, `data.database_url` | [Storage](/operations/storage) | +| `environment` | Name matched by policy rules (`dev`, `staging`, `prod`) | +| `kill_switch_file` | Path of the kill-switch file | +| `approval.ttl`, `approval.webhook`, `approval.console_url` | Request lifetime, notification webhook, public URL linked from notifications | +| `signing_key`, `manifest_public_key` | Evidence signing key; the key every manifest version must be signed with (loading fails if `manifests` is not a signed registry) | +| `capture.enabled`, `capture.ttl`, `capture.redact_pii` | Keep redacted call arguments for [simulation and replay](/concepts/simulation). Off by default | +| `notifications`, `otel`, `anchor`, `egress`, `alerts` | See the pages above | +| `console.mode`, `console.role` | The [console](/console). `role` is `auditor`, `approver` or `operator`; never `admin` | +| `reload.watch`, `reload.interval` | Hot reload (below). Default on, every 2 s | +| `log.level`, `log.format` | Engine messages (reloads, failed notifications, anchors, traces). `level` is `debug`, `info`, `warn` or `error`; `format` is `json` (redacted JSON lines on stderr) or `pretty` (`reversi [warn] message`). The per-request lines of `REVERSI_LOG=json` are separate | + +- Relative paths resolve against the folder holding the file. `http(s)` URLs and absolute paths are kept. +- `${NAME}` is replaced from the environment, and `${NAME:-default}` falls back. An unset variable without a default is an error, so secrets stay out of git. +- Durations: `30s`, `15m`, `2h`, `7d`. +- An unknown key is an error naming it. All problems are listed together. + +**Precedence:** command-line flags, then environment variables, then the file. + +| Environment variable | Overrides | +|---|---| +| `REVERSI_MANIFESTS`, `REVERSI_POLICY`, `REVERSI_ENV` | `manifests`, `policy`, `environment` | +| `REVERSI_DATA`, `REVERSI_STORE`, `REVERSI_DATABASE_URL` | `data.*` | +| `REVERSI_KILL_FILE`, `REVERSI_SIGNING_KEY`, `REVERSI_MANIFEST_PUBLIC_KEY` | `kill_switch_file`, `signing_key`, `manifest_public_key` | +| `REVERSI_APPROVAL_WEBHOOK`, `REVERSI_CONSOLE_URL` | `approval.webhook`, `approval.console_url` | +| `REVERSI_CAPTURE` (`1` or `on`), `REVERSI_CAPTURE_TTL_DAYS` | `capture` | +| `REVERSI_OTLP_ENDPOINT` | `otel.endpoint` | +| `REVERSI_EGRESS_ALLOW` (comma list), `REVERSI_EGRESS_BLOCK_PRIVATE=1`, `REVERSI_REGION` | `egress` | +| `REVERSI_CONSOLE`, `REVERSI_CONSOLE_ROLE` (server) | `console` | + +## Registry and policy versions + +A plain folder of manifest YAML is fine for a start. For review, history and editing, publish into a **registry folder**: every +change is a new numbered version with author, note, checksum, status (`draft`, `approved`, `deprecated`) and optionally an Ed25519 +signature. A **policy versions folder** does the same for the policy, with one active version. + +```bash +reversi manifests publish ./manifests --status approved --registry ./registry --key sign.key +reversi policy publish ./policy.yaml --dir ./policy-versions +``` + +```yaml reversi.config.yaml +manifests: ./registry +policy: ./policy-versions +``` + +- A folder counts as a registry once it holds `index.json`, and as policy versions once it holds `policy-index.json`. Both appear on the first publish. The server and gateway treat an empty folder as an editable registry or policy folder, so the first publish turns it into one. +- The server, gateway and console only offer editing when those folders are in use. With plain files they are read-only. +- Draft manifests are not enforceable, and a registry that holds only a draft of a tool simply has no manifest for it, so a call reports "no manifest" rather than "is a draft". Approving one writes a new, signed version; with `--four-eyes` the author cannot approve their own. +- `reversi manifests verify --trust sign.pub` recomputes every checksum and checks every signature. +- Deprecating keeps a tool working and rolling back but flags new calls. +- `reversi validate`, `lint`, `test`, `doctor` and `plan` read a registry folder as well as a folder of files (for a registry, the current non-draft version of each tool). + +## `reversi apply` + +`apply` makes a registry (and policy folder) match a directory. + +```text +state/ + manifests/ + orders.yaml one manifest, or a list of manifests, per file + payments.yaml + policy.yaml optional +``` + +If `state/manifests/` does not exist, the manifest files are read from `state/` itself. YAML and JSON both work. Every manifest +is validated first and a bad one aborts before anything changes; the same tool twice is an error. + +```text +$ reversi apply state --dry-run --detailed-exitcode; echo "exit $?" ++ manifest notes.read ++ manifest notes.write ++ manifest notes.restore ++ policy policy +4 changes (dry run: nothing applied) +exit 2 + +$ reversi apply state --key ci-key.key --note "from git" --by ci +~ manifest notes.read (was deprecated at v2) +... +applied notes.read v3 (approved) +applied policy v1 +``` + +| Symbol | Meaning | +|---|---| +| `+` | New tool or first policy | +| `~` | Content differs (version and status are ignored when comparing, and both sides are normalized so defaults are not differences), or the tool was deprecated and is in the directory again | +| `-` | With `--prune`: the tool is not in the directory and gets deprecated | +| (blank) | Unchanged | + +- **Idempotent.** A second run prints `up to date` and changes nothing. +- **Nothing is deleted.** `--prune` deprecates; the old versions stay in the registry. +- **Status.** New versions are `approved` by default, on the basis that the directory is the reviewed source of truth. `--status draft` publishes drafts for a person to approve in the console or with `reversi manifests approve`. +- **Signing.** `--key` signs each new manifest version. Policy versions are not signed. +- **Not atomic.** Manifests are applied before the policy. If the policy step fails (for example the policy folder does not exist), the manifests are already applied; fix the cause and run again. +- **`--detailed-exitcode`** exits 2 when there are changes, and only together with `--dry-run`. Without it a dry run exits 0. + +### Local or server + +| | Local | `--server --token ` | +|---|---|---| +| Target | `--registry ` and `--policy-dir `, or `manifests:` and `policy:` in the project file | A running server's `/v1/registry` API | +| Credential | none (file access) | Admin API key to apply; an operator key is enough to dry-run. `--token` or `REVERSI_TOKEN` | +| Running servers | Notice the change by file watch or on `SIGHUP` | The server writes the files and reloads itself | +| Precondition | The registry folder is empty or already a registry. The policy folder must already exist | The server was started with `manifests:` pointing at a registry folder or an existing empty folder (and `policy:` at a policy versions folder or an empty one) | + +A server whose `manifests:` points to plain files refuses `apply --server` with `no manifest registry folder is configured`: +start the server on an empty folder (the first `apply --server` turns it into a registry), or bootstrap with `reversi manifests publish` or a local `reversi apply`. +Both modes were exercised by hand and by the `apply` tests; the server routes are in the [admin API](/api-reference/admin). + +## Hot reload + +The server and the gateway re-read manifests, the policy and the manifest public key while running. A configuration that +fails to load is rejected and the previous one keeps serving. A successful change writes a `reloaded` record to the ledger. + +- **File watch**: on by default, polling every 2 s. Turn off or change with `reload: { watch: false }` or `reload: { interval: 10s }`. +- **`SIGHUP`**: `kill -HUP `. +- **On demand**: `POST /v1/reload` (admin) on the server, `reversi.reload` (admin) on the gateway's operator endpoint. +- Publishing from the console, `apply --server` and the registry endpoints reload as part of the request. If the new configuration does not load, the request answers 409 `saved, but the new configuration did not load: ...`. + +API key files reload separately (`SIGHUP` or a change to the file). The gateway's own config file and `reversi.config.yaml` keys other than manifests and policy are read at start. + +## CI example + +Lint and test on every pull request; apply on merge; check for drift on a schedule. The commands are the ones run locally for +this page; the workflow file itself has not been run on GitHub. + +```yaml .github/workflows/reversi.yml +name: reversi +on: + pull_request: + push: { branches: [main] } + schedule: [{ cron: "0 6 * * *" }] + +jobs: + check: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-node@v4 + with: { node-version: 22 } + - run: npm install -g @multimindlab/reversi-cli + - run: reversi lint --strict state/manifests + - run: reversi test -m state/manifests --policy state/policy.yaml tests/manifests.test.yaml + - name: Would this change anything? + if: github.event_name == 'pull_request' + run: reversi apply state --server "$REVERSI_URL" --dry-run --detailed-exitcode || [ $? -eq 2 ] + env: + REVERSI_URL: ${{ vars.REVERSI_URL }} + REVERSI_TOKEN: ${{ secrets.REVERSI_OPERATOR_KEY }} + - name: Drift check + if: github.event_name == 'schedule' + run: reversi apply state --server "$REVERSI_URL" --prune --dry-run --detailed-exitcode + env: + REVERSI_URL: ${{ vars.REVERSI_URL }} + REVERSI_TOKEN: ${{ secrets.REVERSI_OPERATOR_KEY }} + + apply: + if: github.event_name == 'push' + needs: check + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-node@v4 + with: { node-version: 22 } + - run: npm install -g @multimindlab/reversi-cli + - run: reversi apply state --server "$REVERSI_URL" --prune --note "git ${{ github.sha }}" + env: + REVERSI_URL: ${{ vars.REVERSI_URL }} + REVERSI_TOKEN: ${{ secrets.REVERSI_ADMIN_KEY }} +``` + +- `lint --strict` and `test` exit 1 on warnings or failures, which fails the job. +- The pull-request step prints the pending changes and passes on exit 2; the scheduled step fails on exit 2, which is drift (someone changed the registry by hand). +- Use a dedicated admin key for the apply job and keep it in the CI secret store. Keys are managed in the [console](/console) or [authentication](/operations/authentication). +- The server must be reachable from the runner. Reaching a private server from GitHub-hosted runners needs a self-hosted runner or a tunnel, which is outside Reversi. + +## What is not built + +- A Terraform provider and a Kubernetes operator (custom resources reconciled in-cluster). Use `reversi apply` from CI. +- Applying is a push. There is no agent in the cluster that pulls from git. diff --git a/docs/operations/monitoring.mdx b/docs/operations/monitoring.mdx index a4b3aa6..632b304 100644 --- a/docs/operations/monitoring.mdx +++ b/docs/operations/monitoring.mdx @@ -1,6 +1,6 @@ --- title: Monitoring -description: Health, readiness, Prometheus metrics and structured logs. +description: Health, Prometheus metrics, structured logs, alerts and OpenTelemetry traces. --- ## Health @@ -29,14 +29,93 @@ config), unless `REVERSI_METRICS_PUBLIC=1` (only on a private network). | `reversi_uptime_seconds` | gauge | | Useful alerts: `reversi_halted == 1`, a rising rate of `reversi_ledger_events_total{event="compensation_failed"}`, -`reversi_approvals{status="pending"}` above zero for longer than your approval SLA, and `readyz` failing. +`reversi_approvals{status="pending"}` above zero for longer than your approval SLA, and `readyz` failing. Most of these are +also available as built-in alerts below, delivered to the places you already watch. ## Logs -`REVERSI_LOG=json` writes one line per request: `{"ts","method","path","status","ms","principal"}`. Bodies, headers and -credentials are never logged. The ledger itself is the audit log. +`REVERSI_LOG=json` makes the server write one line per request: + +```json +{"ts":"2026-10-16T17:21:10.820Z","method":"POST","path":"/v1/approvals/apr_…/approve","status":200,"ms":21.1,"principal":"fred"} +``` + +Bodies, headers and credentials are never logged. The ledger itself is the audit log. + +For your own code, `createLogger` from `@multimindlab/reversi` writes one JSON object per line +(`{"level","time","msg",…fields}`) at `debug`, `info`, `warn` or `error`, with `child` loggers for fixed fields. It redacts +before writing: credential patterns inside any string, and the whole value of any field whose name looks like a secret +(`password`, `secret`, `token`, `authorization`, `api_key`, `cookie`, `credential`, `private_key`, `signature`). The `log:` +section of `reversi.config.yaml` configures the engine's own logger, which writes reloads, failed notifications, anchors and +trace export errors to stderr: + +```yaml reversi.config.yaml +log: { level: info, format: json } # level: debug | info | warn | error; format: json | pretty +``` + +`json` writes redacted JSON lines (`{"level":"warn","time":"…","msg":"…"}`); `pretty` writes `reversi [warn] message`. +`REVERSI_LOG=json` is separate: it only controls the per-request line of the server. + +## Alerts + +Alerts are conditions worth a human, computed from the ledger, run state, approvals and (on the gateway) upstream health. +They are always visible: the console **Alerts** page and `GET /v1/alerts` (auditor and up) evaluate them live. When +[notification channels](/operations/notifications) are configured, an alert manager also sends them. + +| Kind | Severity | Fires when | +|---|---|---| +| `compensation_failed` | critical | A step's compensation failed and the run is not rolled back | +| `verification_failed` | critical | An undo ran but reading back its effect failed | +| `break_glass` | critical | A [break-glass](/concepts/approvals) approval was recorded | +| `upstream_unhealthy` | critical | A gateway upstream does not answer its ping | +| `budget_threshold` | warning, critical at 100% | A budget crossed a threshold | +| `rollback_unverified` | warning | An undo ran but could not be verified | +| `irreversible_executed` | warning | An irreversible tool succeeded | +| `unclassified_mutation` | warning | A tool with no manifest was called | +| `policy_denial_spike` | warning | Blocked calls reach the threshold (default 10) in the window | +| `unresolved_outcome` | warning | A step has been `intended` or `uncertain` for over an hour | +| `approval_timeout` | warning | An approval expired unanswered or escalated | +| `halted` | warning | A scoped halt is active | + +Event-based kinds look at the last **15 minutes** of the ledger, so they clear on their own. `compensation_failed`, +`unresolved_outcome` and `halted` are state-based and stay until fixed or lifted. + +```yaml reversi.config.yaml +alerts: + interval: 1m # how often to evaluate (default 1m) + remind: 1h # repeat a still-active alert (default 1h) + denial_spike: 10 # blocked calls in the window (default 10) +``` + +The manager sends an alert **once** when it appears, again every `remind` while it persists, and a `Resolved:` message +(which closes the PagerDuty or Opsgenie incident) when the condition clears. State is kept in the data store, so several +processes sharing it do not repeat each other. `denial_spike` is honoured by the manager, `GET /v1/alerts` and the console Alerts page alike. The window and the one-hour unresolved limit are not configurable in the +project file. The manager only runs when at least one channel is configured; with none, use the console or `GET /v1/alerts`. ## Tracing -OpenTelemetry export is not built. The ledger and metrics cover audit and operations; the structured request log carries -the principal and timings for correlation. +Reversi exports OpenTelemetry traces over OTLP/HTTP with JSON. There is no SDK to install. + +```yaml reversi.config.yaml +otel: + endpoint: http://collector:4318 # spans go to /v1/traces + service_name: reversi + headers: { x-api-key: "${OTEL_KEY}" } +``` + +`REVERSI_OTLP_ENDPOINT` sets the endpoint without a file. The server and gateway export while running. + +| Span | From | To | Status | +|---|---|---|---| +| `reversi.call ` | `intended` | `succeeded`, `failed` or `uncertain` | OK only if `succeeded` | +| `reversi.compensate ` | `compensating` | `compensated` or `compensation_failed` | OK only if `compensated` | +| `reversi.call.blocked ` | the block | the block | error for `blocked`, OK for `would_block` | + +All spans of a run share one trace id (derived from the run id), so a run is one trace. Attributes: +`reversi.run_id`, `reversi.step`, `reversi.tool`, `reversi.actor`, `reversi.effect` (from the manifest), +`reversi.event`, `reversi.reason`, `reversi.detail`, `reversi.seq.start` and `reversi.seq.end` (the ledger positions, so a +span links back to the evidence). The resource carries `service.name` (default `reversi`) and `deployment.environment` +(from `environment:`). Spans carry names and ids, never arguments or results. + +Spans are sent in batches of 50 or every five seconds, with a 10 s timeout, through the +[egress guard](/operations/retention). A failing collector is logged to stderr and never affects calls. diff --git a/docs/operations/notifications.mdx b/docs/operations/notifications.mdx new file mode 100644 index 0000000..4bb8cae --- /dev/null +++ b/docs/operations/notifications.mdx @@ -0,0 +1,132 @@ +--- +title: Notifications +description: Tell people about approvals, failed rollbacks and alerts through Slack, Teams, email, PagerDuty, Opsgenie or a signed webhook. +--- + +Reversi can push events to the places your team already watches. A notification never changes what Reversi decided: a +channel that is down, slow or misconfigured is logged as a warning and the call, approval or rollback carries on. + +## Configure + +Channels and routes live under `notifications:` in `reversi.config.yaml`. `${NAME}` is replaced from the environment, so +secrets stay out of the file (a missing variable is a config error). + +```yaml reversi.config.yaml +approval: + console_url: https://reversi.example.com # linked from messages +notifications: + channels: + - { type: slack, name: ops, url: "${SLACK_WEBHOOK_URL}", interactive: true } + - { type: pagerduty, name: oncall, routing_key: "${PD_ROUTING_KEY}" } + - { type: webhook, name: siem, url: "https://siem.example.com/reversi", secret: "${HOOK_SECRET}" } + routes: + - { events: [approval_requested, approval_decided, approval_escalated], channels: [ops] } + - { events: ["*"], channels: [oncall], min_severity: critical } + - { events: ["*"], channels: [siem] } +``` + +- **Routes** pick channels by event name (`"*"` for all) and an optional `min_severity` (`info`, `warning` or `critical`; + default `info`). A message goes to every channel named by any matching route, once. +- **No `routes`:** every event goes to every channel. +- A route naming a channel that does not exist, a duplicate channel `name`, or a missing required field fails when the + config loads, with the field named. +- Notifications are sent by the process that sees the event: the server, gateway, console, or the CLI when it decides an + approval (`reversi approvals approve|deny` sends `approval_decided` to the channels in the project file; see + [approvals](/concepts/approvals)). + +## Channels + +| `type` | Required | Optional | Notes | +|---|---|---|---| +| `slack` | `url` (incoming webhook) | `interactive` | Header, text and a link to the console. `interactive: true` adds Approve and Deny buttons to approval requests | +| `teams` | `url` (incoming webhook) | | A `MessageCard` coloured by severity with an "Open in Reversi" action | +| `email` | `host`, `port`, `from`, `to` (list) | `secure`, `starttls`, `user`, `pass`, `verify_tls` | Plain-text mail, subject `[Reversi warning] ` | +| `pagerduty` | `routing_key` | `url` | Events API v2: `trigger`, then `resolve` with the same dedup key | +| `opsgenie` | `api_key` | `region` (`us` or `eu`), `url` | Alert with the dedup key as alias; priority P5, P3 or P1 by severity; resolve closes it | +| `webhook` | `url` | `secret`, `headers` | Signed JSON, below | + +**Email** speaks SMTP directly with no dependency: `secure: true` for implicit TLS (port 465), otherwise STARTTLS is used +when the server offers it (`starttls: false` turns that off), `user` and `pass` authenticate with AUTH PLAIN, and the +certificate is verified unless `verify_tls: false`. A permanent rejection (a 5xx reply) is not retried. + +**Delivery.** Every channel sends through the [egress guard](/operations/retention), with a 10 second timeout and up to +three attempts (waits of 0.2 s and 0.4 s) on network errors, `429` and `5xx`. A `4xx` fails at once: it means a wrong URL +or key. Error messages have `key=`, `token=` and `secret=` values blanked. + +## Events + +| Event | Severity | When | +|---|---|---| +| `approval_requested` | warning | A call is waiting for approval. Carries the approval id and argument names | +| `approval_progress` | info | One approver of a quorum has voted; more are needed | +| `approval_decided` | info | The request was approved or denied. Resolves the matching incident | +| `approval_escalated` | warning | Nobody answered in time and the request was widened | +| `break_glass` | critical | An admin forced an approval through | +| `rollback_failed` | critical | A compensation failed | +| `verification_failed` | critical | An undo ran but could not be read back | +| `rollback_unverified` | warning | An undo ran with nothing to verify it | +| `irreversible_executed` | warning | An irreversible tool ran | +| `unclassified_mutation` | warning | A tool with no manifest was called | +| `retention_purged` | info | Run data was deleted by [retention](/operations/retention) | +| `alert` | per alert | An [alert](/operations/monitoring), and its `Resolved:` message | + +Messages carry names and ids (tool, run, actor, argument names), never argument values. + +## The signed webhook + +A `webhook` channel POSTs JSON: + +```json +{"event":"approval_requested","ts":"2026-10-16T17:22:36.534Z","severity":"warning","title":"Approval needed: payments.refund", + "text":"Reversi approval needed: call `payments.refund` in run run_… (arguments: amount). Why: policy `big-refund`: large refund; manifest requires approval. Decide: https://reversi.example.com/console/approvals", + "link":"https://reversi.example.com/console/approvals","dedup_key":"approval:apr_…","resolve":false, + "data":{"approval_id":"apr_…","run_id":"run_…","tool":"payments.refund","status":"pending","quorum":2,"arg_keys":["amount"]}} +``` + +with headers `x-reversi-event` (the event name) and, when `secret` is set, `x-reversi-signature: sha256=<hex>`, the +HMAC-SHA-256 of the **raw request body** under the secret. Compute it over the bytes exactly as received, before parsing +JSON, and compare in constant time. + +<CodeGroup> +```python Python +from reversi_sdk import verify_webhook_signature + +ok = verify_webhook_signature(secret, raw_body, request.headers.get("x-reversi-signature")) +``` + +```js Node +import { createHmac, timingSafeEqual } from "node:crypto"; + +function verify(secret, rawBody, header) { + if (!header?.startsWith("sha256=")) return false; + const want = createHmac("sha256", secret).update(rawBody).digest("hex"); + const got = header.slice(7).trim().toLowerCase(); + return got.length === want.length && timingSafeEqual(Buffer.from(got), Buffer.from(want)); +} +``` +</CodeGroup> + +`verify_webhook_signature` returns `False` for a missing header, a missing `sha256=` prefix or any change to the body. It is +also `ReversiClient.verify_webhook_signature`. Anchor webhooks use the same header and scheme. + +## Test it + +```bash +reversi doctor --send-test +# ok egress allows 127.0.0.1:7553 +# ok test message: sent to ops-hook +``` + +`doctor` checks every channel URL against the egress rules and then sends a test message through all channels. Admins can +also press **Send a test notification** under **Settings** in the console, which lists the channels that were reached and +the ones that failed. + +## The older approval webhook + +`REVERSI_APPROVAL_WEBHOOK` (or `approval.webhook`) is a single URL that gets each new approval request with a `text` field. +It keeps working beside channels. Prefer channels: they add retries, routing, signing, decisions and the egress guard. + +## Not verified here + +The Slack, Teams, PagerDuty, Opsgenie and SMTP payloads are covered by this repository's tests against local fakes. They +have not been exercised against the real services. diff --git a/docs/operations/retention.mdx b/docs/operations/retention.mdx new file mode 100644 index 0000000..b01b1a2 --- /dev/null +++ b/docs/operations/retention.mdx @@ -0,0 +1,151 @@ +--- +title: Retention, holds and data residency +description: How long run data is kept, how a legal hold stops deletion, where Reversi may send data, and how personal data is masked. +--- + +The ledger is append-only and holds hashes, reasons and details, never argument values, so **it is never deleted**. +Retention removes the working data around a run: its run state, its approval requests and its captured (redacted) +arguments. Without a `retention:` section in the policy, nothing is ever deleted. + +## The retention policy + +```yaml policy.yaml +retention: + default_days: 30 + rules: + - { match: { tool: "mail.*" }, days: 365 } + - { match: { data_class: pii, environment: prod }, days: 90 } +``` + +- `default_days` is required (a positive integer). +- Each rule has a `match` and `days`. `match` fields, all optional and all must hold: `tool` (glob, matched against the + tools the run called), `tenant`, `environment`, `data_class` (from the manifest's `risk.data_class`). +- A run is kept for the **longest** of its default, every matching rule, and `audit.retention_days` on the manifest of any + tool it called. +- A manifest with `audit.evidence_export: required` means the run cannot be deleted until its evidence has been archived as + a signed bundle (see below). + +```yaml tools.yaml +- tool: mail.send + effect: irreversible + permission: mail:send + audit: { retention_days: 365, evidence_export: required } +``` + +Retention runs when you ask it to: there is no built-in timer, so run `reversi retention apply` from cron or a job. + +## Plan and apply + +```bash +reversi retention plan +# r1 0d old keep 30d keep +reversi retention apply --dry-run +reversi retention apply --archive-dir ./archive --key signing.key +# deleted: r1 +# archived r1 -> archive/r1.json +``` + +`plan` lists every run with its age, how long it is kept and what would happen: `keep`, `due`, `due; needs an evidence +bundle first`, or `HELD (<reason>)`. `--days <n>` replaces the policy's setting for a one-off run. `apply` deletes the +due runs, prints `deleted: …` (`would delete: …` with `--dry-run`) and a `kept <run>: <reason>` line for each run it left +alone. + +- **Holds win.** A held run is skipped with `legal hold hold_…: <reason>`. +- **Evidence export.** With `--archive-dir` and a signing key (`--key`, or the configured `signing_key`) every deleted run + first gets a signed bundle at `<archive-dir>/<run_id>.json`, verifiable with `reversi verify-bundle`. A run whose + manifest requires export and that has no archive folder or key is skipped: + `a manifest requires an evidence bundle before deletion; give an archive directory and a signing key`. +- **Recorded.** Each deletion appends a `purged` record to the ledger (`retention: older than N days`, with the archive + path) and sends a `retention_purged` [notification](/operations/notifications). + +The same actions exist on the server and in the console: + +| Where | Plan | Apply | +|---|---|---| +| HTTP | `GET /v1/retention/plan` (auditor) | `POST /v1/retention/apply` with `{"dry_run": true}` or `{"archive_dir": "…"}` (admin) | +| Console | **Settings**, Retention: how many runs are due | Dry run and Apply now (admin). The console has no archive folder, so runs that require export are skipped there | + +Both answer `404 no retention settings` when the policy has no `retention:` section. The server archives with its signing +key (`REVERSI_SIGNING_KEY`) and writes to the `archive_dir` you give it, on the server's disk. + +## Legal holds + +A hold stops retention for the runs it covers, for as long as it is active. + +```bash +reversi hold add --run r1 --reason "case 42" +# hold_3e81e022-… holding run_id=r1 +reversi hold add --tenant acme --tool payments.refund --reason "audit 2026-Q4" +reversi hold list +reversi hold release hold_3e81e022-… +``` + +- **Scope.** Any of `--run`, `--tenant`, `--actor`, `--tool`; all given fields must match. `--tool` is an exact tool name and matches a + run that called that tool at least once. A hold with no scope is refused (`a legal hold needs a scope: a run, tenant, actor or + tool`), and the reason must be at least 5 characters (a case or matter reference). +- **Effect.** Retention skips covered runs. A hold does not touch the ledger, which is never deleted anyway, and it does not + extend the time-to-live of [captured](#personal-data-in-capture) arguments. +- **Release** keeps the record (`hold list --all` shows it as released) and makes the run due again. +- **HTTP:** `GET /v1/holds` (auditor, `?all=1` for released ones), `POST /v1/holds` with `run_id`, `tenant`, `actor`, + `tool`, `reason` (admin), `DELETE /v1/holds/{id}` (admin). **Console:** Settings, Legal holds (admin). + +## Data residency and egress controls + +Reversi itself sends data out in a few places: notification channels, anchor webhooks, OpenTelemetry export, remote +manifest registries and the gateway's upstream servers. The `egress` settings control where. + +```yaml reversi.config.yaml +egress: + allow: ["*.eu.example.com", "hooks.slack.com"] # host globs; unset means any host + block_private: true # refuse loopback, private and link-local addresses + region: eu-west-1 # a label recorded in anchors and PDF reports +``` + +| Setting | Env | Default | Effect | +|---|---|---|---| +| `allow` | `REVERSI_EGRESS_ALLOW` (comma separated) | unset: any host | A destination must match one of these globs | +| `block_private` | `REVERSI_EGRESS_BLOCK_PRIVATE=1` | off, so local development works | Refuses hosts that are, or resolve to, loopback, private, link-local and carrier-grade NAT addresses | +| `allow_metadata` | none | off | Cloud metadata addresses (`169.254.169.254` and friends) are **always refused**; this is the only way to lift it and you should not | +| `region` | `REVERSI_REGION` | none | A label: recorded in anchors and PDF reports. It enforces nothing by itself; `allow` does | + +Only `http` and `https` are allowed. Hostnames are resolved before connecting, so a name that points inside the network is +caught too, and every redirect (at most three) is checked the same way. Refusals read, for example: + +```text +hooks.other.com is not in the egress allowlist (region eu-west-1) +127.0.0.1 resolves to a private or loopback address (127.0.0.1) and egress.block_private is on +169.254.169.254 resolves to a cloud metadata address (169.254.169.254), which is never allowed +``` + +`reversi doctor` checks each configured notification, trace and webhook URL against these rules. The single +`REVERSI_APPROVAL_WEBHOOK` goes through the guard too. The server's own Settings page +says whether egress rules are in force. + +## Personal data in capture + +The ledger never stores arguments. Replay, simulating a policy against history and impact previews need them, so there is +an opt-in **capture** store, off by default. + +```yaml reversi.config.yaml +capture: + enabled: true # or REVERSI_CAPTURE=1 + ttl: 7d # default 7 days (REVERSI_CAPTURE_TTL_DAYS) + redact_pii: true # default true +``` + +Everything is redacted on the way in, before it is stored: + +```json +{"note":"mail [PII:email] card [PII:credit_card] ssn [PII:ssn]","api_key":"[REDACTED:stripe_key]","amount":5} +``` + +- **Credentials** (API keys, tokens, private keys and similar patterns) are always replaced, in any field. +- **Personal data**, while `redact_pii` is on: emails, phone numbers, card numbers (must pass the Luhn check), US social + security numbers and IBANs (must pass the checksum). Patterns are conservative on purpose, so an ordinary number is not + masked. Set `redact_pii: false` only if you accept storing them. +- **Named fields.** A manifest can list more paths: `audit.redaction: ["$args.client_secret", "$result.token"]`. They become + `[REDACTED]`. +- **Size.** Strings are cut at 20,000 characters, and long lists and objects at 200 entries. +- **Expiry.** Records past their time-to-live are no longer returned; `reversi capture purge` deletes them, and + `reversi capture purge --run <id>` deletes everything captured for a run. Retention also purges a run's captures when it + deletes the run. diff --git a/docs/quickstart.mdx b/docs/quickstart.mdx index 2f3b5c1..ffa9c76 100644 --- a/docs/quickstart.mdx +++ b/docs/quickstart.mdx @@ -43,6 +43,10 @@ Open `viewer/index.html` in a browser and choose that file. The chain is verifie reversi validate tools.yaml ``` +Or start from a project: `reversi init my-project` writes a config, example manifests, a starter policy and tests, then +`reversi doctor` checks the setup and `reversi test` runs the contract tests and the policy's own tests without touching any +system. `reversi lint` flags unsafe or incomplete manifests (a missing owner, no timeout, a mutating tool with no undo). + ## 4. Wrap calls <CodeGroup> @@ -88,11 +92,22 @@ reversi rollback <run_id> -m tools.yaml -d ./reversi-data --compensator-cmd "./u ## 6. Or put the gateway in front of an MCP server -No code changes in the agent: point its MCP client at `reversi-gateway`, and it journals, enforces policy, -asks for approval and can roll the run back. See the [gateway guide](/gateway). +No code changes in the agent: point its MCP client at `reversi-gateway`. You do not need manifests to begin. With none, the +gateway starts in **observe** mode: everything runs and is journaled, nothing is blocked, and a banner on stderr lists which +upstream tools are unclassified. + +```bash +reversi gateway --upstream-cmd npx --upstream-arg -y --upstream-arg my-mcp-server +``` + +Draft manifests from what the server actually offers, review them, approve them, then enforce: ```bash -reversi-gateway --manifests tools.yaml --upstream-cmd npx --upstream-arg -y --upstream-arg my-mcp-server +reversi suggest-manifest --upstream-cmd npx --upstream-arg -y --upstream-arg my-mcp-server -o drafts +reversi manifests publish drafts --registry ./registry --by bob # drafts are not enforceable yet +reversi manifests approve my.tool --registry ./registry --by alice --four-eyes +reversi gateway --manifests ./registry --upstream-cmd npx --upstream-arg -y --upstream-arg my-mcp-server ``` -Run it in `--mode observe` first to see what would have been blocked before you enforce anything. +Look at what happened in a browser with `reversi console` (open mode, your machine only). The [gateway guide](/gateway) covers +modes, schema checks, approvals and rollback; [GitOps](/operations/gitops) covers keeping all of this in git. diff --git a/docs/roadmap.mdx b/docs/roadmap.mdx index 1a9ba2c..61234c9 100644 --- a/docs/roadmap.mdx +++ b/docs/roadmap.mdx @@ -1,26 +1,56 @@ --- title: Roadmap -description: What is built, and what is not. +description: What is built, what is built but unverified against real systems, and what is not built. --- ## Built -Manifests, write-ahead ledger, dependency-aware rollback with retries, verification and reconciliation, scopes, approvals -(queued, argument-bound, single-use, expiring, persisted, notified), observe mode, [policy as code](/concepts/policy), -[content checks](/concepts/content-checks), kill switch, secret redaction, [signed evidence](/concepts/evidence) and reports, -files / SQLite / PostgreSQL stores, HTTP API with [roles, key rotation and OIDC](/operations/authentication), the -[operator console](/console), Prometheus metrics, the [MCP gateway](/gateway) (enforce and observe, multiple upstreams, -allow/deny, approvals, operator endpoint with rollback tools, Streamable HTTP with auth), [connectors](/connectors) for -filesystem, GitHub, Slack and Stripe, the CLI, the Python SDK, the trace viewer, Docker and CI definitions. +**Core.** Manifests, write-ahead ledger, dependency-aware rollback with retries, verification read-backs and reconciliation, +scopes, observe and advise modes, risk scoring, [policy as code](/concepts/policy) with tests and versions, +[content checks](/concepts/content-checks), kill switches (global and scoped), secret redaction, [signed evidence](/concepts/evidence), +reports (HTML, PDF, period reports) and files / SQLite / PostgreSQL stores. + +**Approvals.** Queued, argument-bound, single use, expiring, persisted; quorum and group routing, escalation on timeout, +delegation, break-glass with a recorded reason and evidence reference ([approvals](/concepts/approvals)). + +**Manifests and policy as files.** Versioned, signable manifest registry with drafts, approval (four-eyes), deprecation and +rollback; policy versions; `reversi lint`, `reversi test`, manifest suggestion from MCP tools, schema-drift detection; +`reversi apply` for GitOps; hot reload ([GitOps](/operations/gitops)). + +**Gateway.** Observe-first start with no manifests, classification banner, enforce / advise / observe, input and output schema +validation, per-manifest timeout and retry, verification through the upstream, several upstreams, allow/deny, approvals, +operator endpoint with rollback tools, Streamable HTTP with auth, egress check of upstream URLs, the console on its own port +([gateway](/gateway)). + +**What-if.** Capture of redacted calls, policy simulation, replay with changes, speculative comparison of plans, impact preview +([simulation](/concepts/simulation)). + +**Operations.** HTTP API with [roles, key rotation and OIDC](/operations/authentication); the [operator console](/console) with +three modes, manifest and policy editors, rollback execution, alerts, keys, holds and retention; Prometheus metrics, OpenTelemetry +traces, structured logs and alerts; notification channels (Slack, Teams, email, PagerDuty, Opsgenie, signed webhook) with +routing ([notifications](/operations/notifications)); retention, legal holds and ledger anchoring ([retention](/operations/retention)); +egress allowlist and data-residency labels. + +**Ecosystem.** The CLI ([cli](/cli)), TypeScript and Python SDKs, Pydantic bindings, [framework adapters](/integrations) +(LangGraph, CrewAI, OpenAI Agents SDK, FastMCP, Claude Code hook), fourteen [connectors](/connectors) with a certification +suite, the trace viewer, a Dockerfile, a Compose stack, a Helm chart (with an optional writable registry volume) and CI definitions. + +## Built, not yet verified against real systems + +Tested with fakes or local stand-ins only. See [limits](/limits) for the detail. + +- The fourteen connectors against live services (the filesystem connector is local and real). +- Notification channels and OpenTelemetry export against the real products. +- The Python framework adapters against installed LangGraph, CrewAI, OpenAI Agents and FastMCP. +- The Dockerfile, Compose stack and Helm chart (never built or installed; `helm lint` was not run). +- The GitHub workflows, and the CI example in [GitOps](/operations/gitops). ## Not built -- OpenTelemetry traces (Prometheus metrics and JSON logs exist). -- Risk scoring, delegated approval and break-glass flows beyond the single approver role. -- Retention and purge tooling, legal hold, data residency controls. -- Gateway: resources, prompts and sampling passthrough; clustering with run hand-off. -- More connectors (Linear, Jira, Calendar, PostgreSQL, Salesforce, cloud providers) and a connector certification suite. -- Framework adapters (LangGraph, CrewAI, OpenAI Agents) and Go/Java SDKs. -- Speculative execution and branch comparison. -- Helm chart, external anchoring service, hosted control plane. -- An external security review of the code. +- A Terraform provider and a Kubernetes operator. `reversi apply` and the Helm chart are the alternative. +- OIDC browser redirect login for the console (login mode takes a pasted API key or OIDC access token). +- A hosted control plane. +- Go and Java SDKs. +- Clustering with run hand-off between gateway or server processes. +- Gateway passthrough of MCP resources, prompts and sampling. +- An independent external security review of the code. diff --git a/docs/testing/adoption.mdx b/docs/testing/adoption.mdx new file mode 100644 index 0000000..b06440f --- /dev/null +++ b/docs/testing/adoption.mdx @@ -0,0 +1,1216 @@ +--- +title: Manual testing - adoption +description: Try by hand, in one terminal, everything that takes a team from "an MCP server we do not trust yet" to enforced, signed manifests kept in git. +--- + +Every command here was run, in this order, against this repository, in a fresh scratch directory. The expected output is copied +from that run and trimmed (`...` marks cut text; ids, times and your scratch path will differ). For the rest of the manual tests +see the [main testing guide](/testing). + +Everything runs from **one terminal**. Servers are started in the background, write to a log file, and are stopped with `kill`. +Ports used: 7700, 7701, 7702 (all on 127.0.0.1). + +<Note> +This page covers what is built. There is no Terraform provider and no Kubernetes operator: `reversi apply` (scenario 6) and the +Helm chart are the supported ways to manage Reversi from files. +</Note> + +## 0. Setup + +You need Node 22.13 or newer (`node -v`), `curl`, `openssl` and `shasum`, and a repository built with `npm install && npm run build`. + +```bash +cd /path/to/Reversi # the repository root +export REPO=$PWD +export T=$(mktemp -d) # everything below is created in here +export NODE_NO_WARNINGS=1 # hides Node's "SQLite is experimental" notice +rv() { node $REPO/packages/cli/dist/main.js "$@"; } # the reversi CLI +mcp() { node $REPO/examples/gateway/mcp.mjs "$@"; } # a tiny MCP client +sha() { printf '%s' "$1" | shasum -a 256 | cut -d' ' -f1; } +waitfor() { until curl -sf "http://127.0.0.1:$1/readyz" >/dev/null; do sleep 0.3; done; } +echo $T +``` + +## 1. `reversi init` and `reversi doctor` + +`init` creates a project: a config file, example manifests, a starter policy, a contract-test file and a data store. + +```bash +mkdir $T/proj && cd $T/proj +rv init +``` + +```text +created reversi.config.yaml +created manifests/example.yaml +created policy.yaml +created tests/manifests.test.yaml +updated .gitignore (reversi-data/) +created reversi-data/ (sqlite store) + +Next: + reversi doctor check the setup + ... +``` + +`reversi.config.yaml` is found from the current folder or any parent, by the CLI, the server and the gateway. Relative paths in it +are relative to the file. See what Reversi resolved: + +```bash +rv show-config +``` + +```text +root: <scratch>/proj +file: <scratch>/proj/reversi.config.yaml +manifests: <scratch>/proj/manifests +policy: <scratch>/proj/policy.yaml +environment: dev +data: + dir: <scratch>/proj/reversi-data + store: sqlite +``` + +`doctor` checks the Node version, the config, that the manifests load and lint, that the data store opens and the ledger chain is +intact, the policy and its own tests, signing key, notification channels, network rules, that the data directory is writable, and +(with `--upstream-cmd`, `--upstream-url` or `--gateway-config`) that an MCP server answers. + +```bash +rv doctor; echo "exit $?" +``` + +```text +ok Node 22.14.0 +ok config <scratch>/proj/reversi.config.yaml +ok 3 manifests load +warn lint: 2 warnings (reversi lint) +ok data store (sqlite) opens +ok ledger: 0 records, chain intact +info no signing key: signed evidence bundles and anchors are off +ok policy: 1 rules, 2 tests +info no notification channels configured +ok data directory <scratch>/proj/reversi-data is writable +ok, with 1 warning +exit 0 +``` + +The example contract tests and the policy's own tests run without touching anything: + +```bash +rv lint +rv test +``` + +```text +warning L011 notes.restore: mutating tool has no owner: nobody is named to ask when it misbehaves +warning L012 notes.restore: no risk.data_class: policy and retention cannot treat sensitive data differently +info L022 notes.restore: no timeout_ms: a hung call is only bounded by the gateway default +0 errors, 2 warnings, 1 notes +ok writing a note can be undone, and the undo is read back +ok a failed write leaves nothing to undo +ok policy: ordinary tools are allowed +ok policy: drop needs approval +not covered by any test: notes.restore +4 passed +``` + +**A failing doctor, and the fix.** Break a manifest, run `doctor`, then put it back: + +```bash +sed -i.bak 's/effect: reversible/effect: sort_of/' manifests/example.yaml +rv doctor > doctor.out; echo "exit $?" +grep -E 'FAIL|problem' doctor.out +mv manifests/example.yaml.bak manifests/example.yaml +rv doctor | tail -1 +``` + +```text +exit 1 +FAIL engine: <scratch>/proj/manifests/example.yaml: `notes.write`: `effect` must be one of read_only, idempotent, reversible, compensable, irreversible +1 problem to fix +ok, with 1 warning +``` + +**What this proves:** a project is one folder with one config file; `doctor` finds a broken setup, names the file and the rule, and +exits 1; the example contract and policy tests pass. + +## 2. Observe-first: a gateway with no manifests + +The aim is to put Reversi in front of an MCP server on day one, change nothing about the agent, and see what it does. The upstream +is the real filesystem connector. + +```bash +mkdir -p $T/gw/sandbox $T/gw/manifests && cd $T/gw +echo hello > sandbox/notes.txt +cat > reversi.config.yaml <<EOF +manifests: ./manifests +data: { dir: ./data, store: files } +EOF +cat > gateway.yaml <<EOF +upstreams: + - name: fs + command: node + args: ["$REPO/packages/connectors/dist/main.js", "serve", "fs", "--root", "$T/gw/sandbox"] + namespace: false +listen: { host: 127.0.0.1, port: 7700 } +EOF +node $REPO/packages/gateway/dist/main.js --config gateway.yaml > gw.log 2>&1 & +echo $! > gw.pid; waitfor 7700; cat gw.log +``` + +`manifests/` is empty and the config names no mode, so the gateway starts in **observe** mode. The startup banner lists every +upstream tool and how it is classified: + +```text +reversi-gateway: no manifests yet, so starting in observe mode: everything runs and is journaled. Run `reversi suggest-manifest` to draft manifests for what the upstream offers. +reversi-connectors: serving fs on stdio +Reversi gateway running in observe mode. + +WARN fs.read_file - no manifest; treated as a change that could not be undone +WARN fs.list_dir - no manifest; treated as a change that could not be undone +WARN fs.write_file - no manifest; treated as a change that could not be undone +... +0 classified, 0 unclassified reads, 9 unclassified changes, 0 destructive without a manifest. + +9 tools have no manifest. They run (nothing is blocked) and are journaled as unclassified. Draft manifests with: reversi suggest-manifest. +reversi-gateway: observe mode, agent endpoint http://127.0.0.1:7700/mcp, files store +``` + +Now act as an agent. `--run obs-1` makes every one-shot call continue the same run, as a real agent session would. + +```bash +U=http://127.0.0.1:7700/mcp +mcp $U list --run obs-1 | head -4 +mcp $U call fs.read_file '{"path":"notes.txt"}' --run obs-1 +mcp $U call fs.write_file '{"path":"todo.txt","content":"buy milk"}' --run obs-1 --reason "save a todo" +mcp $U call fs.delete_file '{"path":"todo.txt"}' --run obs-1 +ls sandbox +``` + +```text +fs.read_file Read a UTF-8 text file under the root. +fs.list_dir List a directory under the root (default: the root itself). +fs.write_file Create or overwrite a text file. The previous content is saved so roll +fs.delete_file Delete a file. It is moved into the backup directory so rollback can r +RESULT: +{"path":"notes.txt","content":"hello\n","size":6} +RESULT: +{"path":"todo.txt","backup_id":"a2401029-...","existed":false} +RESULT: +{"path":"todo.txt","backup_id":"ce469e3d-..."} +notes.txt +``` + +All three calls ran (the delete really deleted). Look at the journal: + +```bash +rv runs list +rv runs show obs-1 +``` + +```text +obs-1 actor=mcp-client mode=observe steps=3 complete +run obs-1 actor=mcp-client mode=observe started 2026-10-16 17:19:18Z +scope: + +steps + 0 fs.read_file no manifest succeeded + 1 fs.write_file no manifest succeeded + 2 fs.delete_file no manifest succeeded + +rollback: NOT fully reversible + step 2 fs.delete_file unmanifested no manifest was registered for this tool, so its effect is unknown + ... + +events + 0 ... run_started - by mcp-client | mode=observe scope_seal=e3b0c442... + 1 ... would_block fs.read_file by mcp-client | no manifest registered for tool `fs.read_file` + 2 ... intended fs.read_file by mcp-client + 3 ... succeeded fs.read_file by mcp-client + ... +``` + +Each call is a `would_block` (what enforce mode would have done), then `intended` (written before the call is forwarded), then the +outcome. The same records as JSON Lines, and a check that the hash chain holds: + +```bash +rv export --run obs-1 | wc -l +rv export --run obs-1 | sed -n 3p | cut -c1-230 +rv verify +``` + +```text +10 +{"seq":2,"ts_ms":1792171158850,"run_id":"obs-1","step":0,"actor":"mcp-client","tool":"fs.read_file","payload_hash":"327e0978...","reason":"MCP tools/call fs.read_file",... +ok: 10 records, head d5bc5ac3... +``` + +**What this proves:** with no manifests and no changes to the agent, every call is forwarded and journaled with its reason, arguments +only as a hash, and a hash-chained record; nothing is blocked; the banner tells you what to classify next. + +## 3. From unclassified to signed, approved manifests + +### 3.1 Draft manifests from the live server + +```bash +kill $(cat gw.pid) +rv suggest-manifest --gateway-config gateway.yaml --out drafts +``` + +```text +reversi-connectors: serving fs on stdio +drafts/fs.read_file.yaml low +drafts/fs.list_dir.yaml low +drafts/fs.write_file.yaml medium +drafts/fs.delete_file.yaml medium +drafts/fs.restore_file.yaml medium +drafts/fs.make_dir.yaml low +drafts/fs.remove_dir.yaml medium +drafts/fs.move_file.yaml medium +drafts/fs.unmove_file.yaml medium +9 drafted, 0 skipped (already have manifests) +``` + +The word after each file is the confidence. Every draft has `status: draft`, so none can be enforced until a person approves it. +Read one: + +```bash +cat drafts/fs.write_file.yaml +``` + +```text +# DRAFT (medium confidence). Review before enforcing. +# - undo guessed from the tool names: `fs.restore_file`; check the argument map and decide whether it is a true inverse (reversible) or leaves residue (compensable) +tool: fs.write_file +description: Create or overwrite a text file. The previous content is saved so rollback can restore it. +effect: compensable +permission: fs:write +status: draft +risk: + data_class: internal + blast_radius: medium +compensation: + tool: fs.restore_file + argument_map: + path: $args.path + backup_id: $result.backup_id + kind: compensation + residue: + - TODO describe what the undo cannot take back +schema_hash: 2e6a54497ec6003e +``` + +Tools with no obvious undo are drafted as `irreversible` with `approval: always` (the safe default); read-only names are drafted +`read_only`. `schema_hash` is used in 5.5. Review means editing: name an owner, set a timeout, replace the `TODO`. Also remove +two drafts on purpose, so two tools stay unclassified for scenario 4: + +```bash +sed -i.bak -e 's/^ - TODO describe .*/ - the previous content comes back from the backup; the file modification time does not/' \ + -e 's/^effect: compensable/effect: compensable\nowner: platform@example.com\ntimeout_ms: 30s/' drafts/fs.write_file.yaml +rm drafts/fs.write_file.yaml.bak drafts/fs.move_file.yaml drafts/fs.unmove_file.yaml +rv lint drafts | tail -4 +``` + +```text +warning L024 fs.restore_file: draft manifest: it is not enforceable until approved +info L016 fs.write_file: the undo is not verified: add compensation.verification to read back that it worked +warning L024 fs.write_file: draft manifest: it is not enforceable until approved +0 errors, 11 warnings, 8 notes +``` + +`lint` finds unsafe, incomplete or contradictory manifests (exit 1 on errors, or on warnings with `--strict`). With +`--upstream-cmd`, `--upstream-url` or `--gateway-config` it also checks the manifests against the server's real tools. + +### 3.2 Contract tests for manifests + +A test file is a list of cases. Each runs in a sandbox: no system is touched. Write three: + +```bash +mkdir tests +cat > tests/fs.test.yaml <<'EOF' +- name: writing a file can be undone from its backup + tool: fs.write_file + args: { path: todo.txt, content: buy milk } + result: { path: todo.txt, backup_id: b1, existed: false } + expect_plan: compensate + expect_handle: { path: todo.txt, backup_id: b1 } +- name: a failed write leaves nothing to undo + tool: fs.write_file + args: { path: todo.txt, content: x } + fail: disk full +- name: deleting a file is never undone + tool: fs.delete_file + args: { path: todo.txt } + result: { path: todo.txt, backup_id: b2 } + expect_plan: unrecoverable +EOF +rv test tests/fs.test.yaml -m drafts; echo "exit $?" +``` + +Drafts cannot be enforced, so the sandbox refuses them: + +```text +FAIL writing a file can be undone from its backup + the call was refused: the manifest for `fs.write_file` is a draft: publish it before it can be enforced + ... + expected plan kind compensate, got none +FAIL a failed write leaves nothing to undo + ... +FAIL deleting a file is never undone + ... +not covered by any test: fs.make_dir, fs.remove_dir, fs.restore_file +3 of 3 failed +exit 1 +``` + +Test the content as it will be enforced, using a throwaway copy marked approved: + +```bash +rm -rf reviewed && cp -r drafts reviewed +sed -i.bak 's/^status: draft/status: approved/' reviewed/*.yaml && rm reviewed/*.bak +rv test tests/fs.test.yaml -m reviewed; echo "exit $?" +``` + +```text +ok writing a file can be undone from its backup +ok a failed write leaves nothing to undo +ok deleting a file is never undone +not covered by any test: fs.make_dir, fs.remove_dir, fs.restore_file +3 passed +exit 0 +``` + +Break a test on purpose (the wrong backup id) to see a real failure: + +```bash +sed 's/backup_id: b1 }/backup_id: b9 }/' tests/fs.test.yaml > tests/broken.yaml +rv test tests/broken.yaml -m reviewed > broken.out; echo "exit $?" +head -3 broken.out; rm tests/broken.yaml broken.out +``` + +```text +exit 1 +FAIL writing a file can be undone from its backup + compensation argument `backup_id` is "b1", expected "b9" +ok a failed write leaves nothing to undo +``` + +### 3.3 Publish, approve and verify with a signing key + +```bash +mkdir keys +rv keygen keys/signing +rv manifests publish drafts --registry registry --by alice --key keys/signing.key --note "first drafts from suggest-manifest" +rv manifests approve fs.read_file --registry registry --by alice --four-eyes --key keys/signing.key; echo "exit $?" +``` + +```text +wrote keys/signing.key and keys/signing.pub (key id ae0ce962c5978747) +fs.delete_file v1 draft 16c5f8580b19 +fs.list_dir v1 draft d7f9551d1c2a +... +fs.write_file v1 draft 21b8d6991e45 +error: the author of a draft cannot approve it: ask someone else +exit 1 +``` + +`publish` makes drafts by default. `--four-eyes` forbids the author approving their own draft. A different person approves +(each approval is a new, signed version): + +```bash +for t in $(ls drafts | sed 's/\.yaml$//'); do + rv manifests approve $t --registry registry --by bob --four-eyes --key keys/signing.key --note reviewed +done +rv manifests list --registry registry +rv manifests history fs.write_file --registry registry +rv manifests diff fs.write_file 1 2 --registry registry | grep '^[-+]' +rv manifests verify --registry registry --trust keys/signing.pub +``` + +```text +fs.delete_file v2 approved by bob +... +fs.delete_file v2 approved +fs.list_dir v2 approved +fs.make_dir v2 approved +fs.read_file v2 approved +fs.remove_dir v2 approved +fs.restore_file v2 approved +fs.write_file v2 approved +v1 draft 2026-10-16T17:22:36Z by alice 21b8d6991e45 signed - first drafts from suggest-manifest +v2 approved 2026-10-16T17:22:37Z by alice, approved by bob 3cec779960c1 signed - reviewed +- "version": 1, +- "status": "draft", ++ "version": 2, ++ "status": "approved", +ok: 7 tools, every version verifies +``` + +(`manifests deprecate` and `manifests rollback <tool> <version>` complete the set; not run here.) + +### 3.4 Use the registry as the manifests source, then tamper with it + +Point the project at the registry and require its signature. The gateway, the server and the CLI all read these two settings: + +```bash +cat > reversi.config.yaml <<EOF +manifests: ./registry +manifest_public_key: ./keys/signing.pub +data: { dir: ./data, store: files } +EOF +rmdir manifests +rv doctor | grep -E 'registry|manifests load' +rv validate registry +``` + +```text +ok manifest registry <scratch>/gw/registry: every version verifies +ok 7 manifests load +ok: 7 tool manifests valid +``` + +`validate`, `lint`, `test` and `plan` read a registry folder too (the current non-draft version of each tool). + +Now change one registry file behind the registry's back (grant `fs.write_file` a weaker permission): + +```bash +cp registry/tools/fs.write_file/v2.json $T/v2.good +sed -i.bak 's/"permission": "fs:write"/"permission": "fs:read"/' registry/tools/fs.write_file/v2.json && rm registry/tools/fs.write_file/v2.json.bak +rv manifests verify --registry registry --trust keys/signing.pub; echo "exit $?" +rv doctor | grep FAIL +node $REPO/packages/gateway/dist/main.js --config gateway.yaml 2>&1 | head -1 +``` + +```text +BAD `fs.write_file` v2 was changed outside the registry: its checksum does not match the index +exit 1 +FAIL manifest registry: `fs.write_file` v2 was changed outside the registry: its checksum does not match the index +FAIL engine: `fs.write_file` v2 was changed outside the registry: its checksum does not match the index +error: `fs.write_file` v2 was changed outside the registry: its checksum does not match the index +``` + +`verify`, `doctor` and the gateway all refuse the tampered registry; the gateway does not start: it prints that one-line error (exit code 64) followed by its usage text, not a stack trace. Restore +the file: + +```bash +cp $T/v2.good registry/tools/fs.write_file/v2.json +rv manifests verify --registry registry --trust keys/signing.pub +``` + +```text +ok: 7 tools, every version verifies +``` + +**What this proves:** drafts are generated from the real server, tests and lint run before anything is enforceable, approval is a +separate signed step by a second person, and any after-the-fact edit to the registry is detected. + +## 4. Switch to enforce + +Create keys for an agent, an approver, an operator and an admin (only SHA-256 hashes are stored), and an enforcing config: + +```bash +AGENT=agent-$(openssl rand -hex 8); APPR=appr-$(openssl rand -hex 8); OPS=ops-$(openssl rand -hex 8); ROOT=root-$(openssl rand -hex 8) +cat > keys.json <<EOF +[ + { "name": "fs-agent", "role": "agent", "key_sha256": "$(sha $AGENT)", "max_permissions": ["fs:read","fs:write","fs:admin"] }, + { "name": "alice", "role": "approver", "key_sha256": "$(sha $APPR)" }, + { "name": "dana", "role": "operator", "key_sha256": "$(sha $OPS)" }, + { "name": "root", "role": "admin", "key_sha256": "$(sha $ROOT)" } +] +EOF +sed -i.bak 's/^upstreams:/mode: enforce\nupstreams:/' gateway.yaml && rm gateway.yaml.bak +echo 'auth: { keys: keys.json }' >> gateway.yaml +node $REPO/packages/gateway/dist/main.js --config gateway.yaml > gw.log 2>&1 & +echo $! > gw.pid; waitfor 7700; cat gw.log +``` + +```text +reversi-connectors: serving fs on stdio +Reversi gateway running in enforce mode. + +OK fs.read_file - read_only +OK fs.list_dir - read_only +OK fs.write_file - compensable +OK fs.delete_file - irreversible +OK fs.restore_file - idempotent +OK fs.make_dir - irreversible +OK fs.remove_dir - irreversible +WARN fs.move_file - no manifest; treated as a change that could not be undone +WARN fs.unmove_file - no manifest; treated as a change that could not be undone + +7 classified, 0 unclassified reads, 2 unclassified changes, 0 destructive without a manifest. + +2 tools have no manifest and are blocked. Draft them with: reversi suggest-manifest, review, then reversi manifests publish. +reversi-gateway: enforce mode, agent endpoint http://127.0.0.1:7700/mcp, files store +``` + +The registry's manifests are enforced, and the observe-mode journal from scenario 2 is still in the same data directory. + +```bash +agent() { mcp $U "$@" --token $AGENT --run enf-1; } +echo "-- unclassified tool" +agent call fs.move_file '{"from":"notes.txt","to":"moved.txt"}' +echo "-- allowed call" +agent call fs.write_file '{"path":"notes.txt","content":"changed by agent"}' --reason "update notes" +cat sandbox/notes.txt; echo +echo "-- approval required" +agent call fs.delete_file '{"path":"notes.txt"}' +rv approvals list +``` + +```text +-- unclassified tool +ERROR RESULT: +Blocked by Reversi: no manifest registered for tool `fs.move_file` +-- allowed call +RESULT: +{"path":"notes.txt","backup_id":"a5cda514-...","existed":true} +changed by agent +-- approval required +ERROR RESULT: +Blocked by Reversi: approval required: request apr_39763c04-... is pending an operator decision; retry after it is approved +apr_39763c04-... pending run=enf-1 call fs.delete_file args=[path] why: manifest requires approval +``` + +Approve from the CLI (it shares the data directory with the gateway), then the agent retries: + +```bash +ID=$(rv approvals list | awk '{print $1}' | head -1) +rv approvals approve $ID --as alice --note "checked the ticket" +agent call fs.delete_file '{"path":"notes.txt"}' +ls sandbox +rv approvals list +rv export --run enf-1 | jq -r 'select(.event=="approved") | .actor + ": " + .detail' | cut -c1-110 +``` + +```text +apr_39763c04-... approved by alice +RESULT: +{"path":"notes.txt","backup_id":"62beadea-..."} +apr_39763c04-... consumed run=enf-1 call fs.delete_file args=[path] why: manifest requires approval +alice: approval apr_39763c04-... approved by alice; the call must be retried +``` + +The CLI decides through the engine, so the decision is in the ledger (an `approved` record with the approver as actor), delegations apply, and notification channels in the config are told. + +The approval is single-use (`consumed`). `ls sandbox` prints nothing: the file is gone. + +### Rollback + +Two writes in a new run, then see what a rollback would do: + +```bash +echo restore-me > sandbox/keep.txt +a3() { mcp $U "$@" --token $AGENT --run enf-3; } +a3 call fs.write_file '{"path":"keep.txt","content":"overwritten"}' > /dev/null +a3 call fs.write_file '{"path":"new.txt","content":"brand new"}' > /dev/null +ls sandbox; cat sandbox/keep.txt; echo +rv plan enf-3 +``` + +```text +keep.txt +new.txt +overwritten +rollback plan for enf-3 + step 1 fs.write_file compensate -> fs.restore_file + step 0 fs.write_file compensate -> fs.restore_file +fully reversible (assuming each compensation succeeds) +``` + +The gateway holds the connection to the upstream, so the rollback that works against a gateway is the **operator endpoint**: it +calls each compensation on the upstream, newest first. Without `confirm` it is a dry run. + +```bash +op() { mcp http://127.0.0.1:7700/operator/mcp "$@" --token $OPS; } +mcp http://127.0.0.1:7700/operator/mcp list --token $AGENT +op call reversi.execute_rollback '{"run_id":"enf-3"}' | head -12 +op call reversi.execute_rollback '{"run_id":"enf-3","confirm":true}' | grep -E 'dry_run|complete' +ls sandbox; cat sandbox/keep.txt +rv runs show enf-3 | tail -5 +rv verify +``` + +```text +could not connect: Streamable HTTP error: Error POSTing to endpoint: {"error":"auditor role or higher required"} +RESULT: +{ + "dry_run": true, + "plan": { + "run_id": "enf-3", + "actions": [ + { + "step": 1, + "tool": "fs.write_file", + "kind": "compensate", + "compensation_tool": "fs.restore_file", +... + "dry_run": false, + "complete": true +keep.txt +restore-me + 24 ... compensating fs.restore_file by dana | idempotency_key=f7b454af... + 25 ... compensated fs.restore_file by dana + 26 ... compensating fs.restore_file by dana | idempotency_key=f49bdada... + 27 ... compensated fs.restore_file by dana +ok: 28 records, head 4141ffba... +``` + +`new.txt` is gone and `keep.txt` has its old text. The agent key cannot even connect to the operator endpoint. + +<Note> +`reversi rollback <run> --compensator-cmd '...'` is the CLI path for a compensator you supply (a script that undoes a call). It +only trusts that command's exit code, so it is not used here: a command that exits 0 without undoing anything marks the step +compensated. For a gateway, use the operator endpoint above (or the console). +</Note> + +**What this proves:** the unclassified tool is blocked, an allowed call runs, an approval-required call is blocked until a named +approver decides and then runs once, and a run is rolled back through the same upstream with every step in the ledger. + +Stop this gateway: + +```bash +kill $(cat gw.pid) +``` + +## 5. Gateway features + +A small upstream of our own, so every feature can be triggered. It has four tools: `notes_get` and `notes_slow` and `notes_flaky` +(read-only hint) and `notes_add`. `notes_flaky` fails twice per id, then works. Creating `drift.flag` changes `notes_add`'s schema. + +```bash +mkdir $T/feat && cd $T/feat +ln -s $REPO/node_modules node_modules # so the script finds @modelcontextprotocol/sdk +cat > notes-server.mjs <<'EOF' +import { existsSync } from "node:fs"; +import { Server } from "@modelcontextprotocol/sdk/server/index.js"; +import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js"; +import { CallToolRequestSchema, ListToolsRequestSchema } from "@modelcontextprotocol/sdk/types.js"; + +const notes = new Map(); +const attempts = new Map(); +const tools = () => [ + { name: "notes_get", description: "Read a note.", annotations: { readOnlyHint: true }, + inputSchema: { type: "object", properties: { id: { type: "string", minLength: 1 } }, required: ["id"], additionalProperties: false } }, + { name: "notes_add", description: "Create a note (safe to repeat).", + inputSchema: { type: "object", properties: { title: { type: "string", minLength: 1 }, priority: { type: "integer", minimum: 1, maximum: 5 }, + ...(existsSync("drift.flag") ? { tags: { type: "array", items: { type: "string" } } } : {}) }, required: ["title", "priority"], additionalProperties: false } }, + { name: "notes_slow", description: "Answer after a delay.", annotations: { readOnlyHint: true }, + inputSchema: { type: "object", properties: { ms: { type: "integer" } }, required: ["ms"] } }, + { name: "notes_flaky", description: "Fails twice per id, then works.", annotations: { readOnlyHint: true }, + inputSchema: { type: "object", properties: { id: { type: "string" } }, required: ["id"] } }, +]; +const server = new Server({ name: "notes", version: "1" }, { capabilities: { tools: {} } }); +server.setRequestHandler(ListToolsRequestSchema, async () => ({ tools: tools() })); +server.setRequestHandler(CallToolRequestSchema, async (req) => { + const { name, arguments: a = {} } = req.params; + const say = (o) => ({ content: [{ type: "text", text: JSON.stringify(o) }] }); + if (name === "notes_add") { const id = `n${notes.size + 1}`; notes.set(id, a); return say({ id, ...a }); } + if (name === "notes_get") return say(notes.get(a.id) ?? { error: "no such note" }); + if (name === "notes_slow") { await new Promise((r) => setTimeout(r, a.ms)); return say({ waited: a.ms }); } + if (name === "notes_flaky") { + const n = (attempts.get(a.id) ?? 0) + 1; attempts.set(a.id, n); + if (n < 3) throw new Error(`temporary failure (attempt ${n})`); + return say({ id: a.id, attempts: n }); + } + throw new Error(`unknown tool ${name}`); +}); +await server.connect(new StdioServerTransport()); +EOF +rv suggest-manifest --upstream-cmd node --upstream-arg notes-server.mjs --server notes --out drafts > /dev/null 2>&1 +mkdir manifests && cp drafts/*.yaml manifests/ +sed -i.bak -e '/^# /d' -e 's/^status: draft/status: approved/' -e 's/^effect: irreversible/effect: idempotent/' -e 's/^approval: always/approval: never/' manifests/*.yaml && rm manifests/*.bak +echo 'timeout_ms: 500ms' >> manifests/notes_slow.yaml +printf 'timeout_ms: 2s\nretry: { attempts: 3, base_ms: 100 }\n' >> manifests/notes_flaky.yaml +rv validate manifests +rv lint manifests | tail -1 +``` + +```text +ok: 4 tool manifests valid +0 errors, 1 warnings, 1 notes +``` + +Here the reviewed drafts were approved in bulk (the page's point is the gateway, not the review). `notes_add` is declared +`idempotent` and `notes_slow` has a 500 ms timeout; `notes_flaky` allows 3 attempts (retry is only accepted for `read_only` and +`idempotent` tools, because repeating anything else could repeat its effect). + +Keys for this gateway, a config in **advise** mode, and start it: + +```bash +AGENT=agent-$(openssl rand -hex 8); OPS=ops-$(openssl rand -hex 8) +cat > keys.json <<EOF +[ + { "name": "notes-agent", "role": "agent", "key_sha256": "$(sha $AGENT)", "max_permissions": ["notes:read","notes:admin"] }, + { "name": "dana", "role": "operator", "key_sha256": "$(sha $OPS)" } +] +EOF +cat > gateway.yaml <<EOF +mode: advise +upstreams: + - { name: notes, command: node, args: ["$T/feat/notes-server.mjs"], cwd: "$T/feat", namespace: false } +listen: { host: 127.0.0.1, port: 7701 } +auth: { keys: keys.json } +EOF +node $REPO/packages/gateway/dist/main.js --config gateway.yaml --manifests manifests --data data5 > gw.log 2>&1 & +echo $! > gw.pid; waitfor 7701; tail -9 gw.log +``` + +```text +Reversi gateway running in advise mode. + +OK notes_get - read_only +OK notes_add - idempotent +OK notes_slow - read_only +OK notes_flaky - read_only + +4 classified, 0 unclassified reads, 0 unclassified changes, 0 destructive without a manifest. +reversi-gateway: advise mode, agent endpoint http://127.0.0.1:7701/mcp, files store +``` + +### 5.1 Advise mode: an advisory is appended, nothing is blocked + +```bash +U=http://127.0.0.1:7701/mcp +ag() { mcp $U "$@" --token $AGENT --run f-1; } +ag call notes_add '{"title":"buy milk","priority":2}' +ag call notes_add '{"title":"","priority":9}' +rv search --event would_block --run f-1 -d data5 -m manifests +``` + +```text +RESULT: +{"id":"n1","title":"buy milk","priority":2} +RESULT: +{"id":"n2","title":"","priority":9} +Reversi advisory (nothing was blocked): arguments do not match the tool's input schema: $.title: shorter than 1; $.priority: above maximum 5 + 3 ... f-1#1 would_block notes_add notes-agent | arguments do not match the tool's input schema: $.title: shorter than 1; $.priority: above +``` + +The call ran, the result carries an extra text block saying what enforce mode would have done, and the ledger has a `would_block`. + +### 5.2 Enforce: schema validation, timeout, retry + +Gateway arguments are checked against the upstream tool's own `inputSchema` (and results against its `outputSchema`) unless +`validate_schemas: false`. Restart in enforce mode: + +```bash +kill $(cat gw.pid) +sed -i.bak 's/^mode: advise/mode: enforce/' gateway.yaml && rm gateway.yaml.bak +node $REPO/packages/gateway/dist/main.js --config gateway.yaml --manifests manifests --data data5 > gw.log 2>&1 & +echo $! > gw.pid; waitfor 7701 +ag() { mcp $U "$@" --token $AGENT --run f-2; } +ag call notes_add '{"title":"buy milk","priority":2}' +ag call notes_add '{"title":"","priority":9}' +ag call notes_add '{"title":"x"}' +echo "-- timeout" +ag call notes_slow '{"ms":100}' +ag call notes_slow '{"ms":2000}' +echo "-- retry" +ag call notes_flaky '{"id":"a"}' +``` + +```text +RESULT: +{"id":"n1","title":"buy milk","priority":2} +ERROR RESULT: +Blocked by Reversi: arguments do not match the tool's input schema: $.title: shorter than 1; $.priority: above maximum 5 +ERROR RESULT: +Blocked by Reversi: arguments do not match the tool's input schema: $: missing required `priority` +-- timeout +RESULT: +{"waited":100} +request failed: MCP error -32603: upstream did not answer within 500 ms +-- retry +RESULT: +{"id":"a","attempts":3} +``` + +The 2000 ms call hits the manifest's `timeout_ms` and is journaled as `uncertain` (the gateway cannot know whether it finished; see +`reversi runs show f-2`). `notes_flaky` threw twice; the gateway retried (100 ms, then 200 ms backoff) and the third attempt +succeeded: `"attempts":3` is the upstream's own count. Retry never applies to a tool that is not `read_only` or `idempotent`. + +### 5.3 Hot reload: a manifest edit, and a broken edit + +The gateway polls the manifest files (every 2 s by default; `reload.watch` and `reload.interval` in the project config; `SIGHUP` +forces it). Remove one manifest while it runs: + +```bash +mkdir held && mv manifests/notes_get.yaml held/ +sleep 4; tail -1 gw.log +mcp $U call notes_get '{"id":"n1"}' --token $AGENT --run f-3 +``` + +```text +reversi [info] reloaded: 3 manifests, policy none +ERROR RESULT: +Blocked by Reversi: no manifest registered for tool `notes_get` +``` + +Now a broken edit, then restore the file. The bad file must be rejected and the old configuration kept: + +```bash +cp manifests/notes_slow.yaml held/ +sed -i.bak 's/^effect: read_only/effect: nonsense/' manifests/notes_slow.yaml && rm manifests/notes_slow.yaml.bak +sleep 4; tail -2 gw.log +mcp $U call notes_slow '{"ms":10}' --token $AGENT --run f-3 +cp held/notes_slow.yaml manifests/notes_slow.yaml +``` + +```text +reversi [info] reloaded: 3 manifests, policy none +reversi [error] reload failed, keeping the previous manifests and policy: manifests/notes_slow.yaml: `notes_slow`: `effect` must be one of read_only, idempotent, reversible, compensable, irreversible +RESULT: +{"waited":10} +``` + +The call still works under the previous, valid configuration. + +### 5.4 `unclassified_reads: observe` + +A tool with no manifest that the server itself annotates read-only can be let through (journaled as unclassified, in a separate +observe-mode run named `<run>-reads`) while every other unclassified tool stays blocked. `notes_get` has no manifest right now: + +```bash +kill $(cat gw.pid) +sed -i.bak 's/^mode: enforce/mode: enforce\nunclassified_reads: observe/' gateway.yaml && rm gateway.yaml.bak +node $REPO/packages/gateway/dist/main.js --config gateway.yaml --manifests manifests --data data5 >> gw.log 2>&1 & +echo $! > gw.pid; waitfor 7701 +grep -E '^NOTE' gw.log | tail -1 +mcp $U call notes_get '{"id":"n1"}' --token $AGENT --run f-3 +rv runs list -d data5 -m manifests | grep f-3 +rv runs show f-3-reads -d data5 -m manifests | grep -E 'unclassified read|^steps|^ +0 notes' +``` + +```text +NOTE notes_get - no manifest; the server says it is read-only (unverified) +RESULT: +{"error":"no such note"} +f-3 actor=notes-agent mode=enforce steps=1 complete +f-3-reads actor=notes-agent mode=observe steps=1 complete +steps + 0 notes_get no manifest succeeded + 24 ... intended notes_get by notes-agent | unclassified read: the server says it is read-only; this is unverified +``` + +(The `no such note` answer is the upstream's: it was restarted, so its notes are gone.) The call ran and is marked unverified. Put +the manifest back; the running gateway picks it up: + +```bash +mv held/notes_get.yaml manifests/ +sleep 4; tail -1 gw.log +``` + +```text +reversi [info] reloaded: 4 manifests, policy none +``` + +### 5.5 Schema drift + +Each manifest can carry the `schema_hash` of the tool's schema as it was when someone reviewed it (`suggest-manifest` writes it). +If the upstream later changes the schema, the gateway refuses the call until the manifest is reviewed again. Change the upstream's +schema, wait out the gateway's 5 s tool-list cache, and call: + +```bash +touch drift.flag; sleep 6 +mcp $U call notes_add '{"title":"after drift","priority":1}' --token $AGENT --run f-4 +``` + +```text +ERROR RESULT: +Blocked by Reversi: the tool's input schema changed since its manifest was approved: review the manifest again +``` + +Review, then record the new hash. The new value comes from a fresh draft; the running gateway reloads the manifest: + +```bash +rv suggest-manifest --upstream-cmd node --upstream-arg notes-server.mjs --server notes --out drafts2 > /dev/null 2>&1 +grep schema_hash manifests/notes_add.yaml drafts2/notes_add.yaml +NEW=$(grep '^schema_hash' drafts2/notes_add.yaml | cut -d' ' -f2) +sed -i.bak "s/^schema_hash: .*/schema_hash: $NEW/" manifests/notes_add.yaml && rm manifests/notes_add.yaml.bak +sleep 4 +mcp $U call notes_add '{"title":"after drift","priority":1,"tags":["a"]}' --token $AGENT --run f-4 +``` + +```text +manifests/notes_add.yaml:schema_hash: 7941405d5073f08d +drafts2/notes_add.yaml:schema_hash: f26233af61faf95f +RESULT: +{"id":"n1","title":"after drift","priority":1,"tags":["a"]} +``` + +### 5.6 `reversi.classification`, `reversi.preview_impact`, `reversi.suggest_manifest` + +`reversi.preview_impact` is on the agent endpoint (an agent can ask "would this be allowed, and could it be undone?" before it +tries). `reversi.classification` and `reversi.suggest_manifest` are on the **operator** endpoint only, so they need an operator +key. The tiny client is the same `mcp.mjs` used throughout. + +```bash +op() { mcp http://127.0.0.1:7701/operator/mcp "$@" --token $OPS; } +mcp $U call reversi.preview_impact '{"tool":"notes_add","arguments":{"title":"x","priority":1}}' --token $AGENT --run f-4 | grep -E '"(verdict|effect)"' +mcp $U call reversi.preview_impact '{"tool":"nope"}' --token $AGENT --run f-4 | grep -E '"(verdict|known)"|unclassified' +op call reversi.classification | grep -E '"(name|class)"' +op call reversi.suggest_manifest +mcp $U list --token $AGENT --run f-4 | grep -c -E 'classification|suggest_manifest' +``` + +```text + "effect": "idempotent", + "verdict": "allow", + "known": false, + "verdict": "block", + "unclassified tool" + "name": "notes_get", + "class": "classified", + "name": "notes_add", + "class": "classified", + "name": "notes_slow", + "class": "classified", + "name": "notes_flaky", + "class": "classified", +RESULT: +{ + "suggestions": [], + "skipped": [ + "notes_get", + "notes_add", + "notes_slow", + "notes_flaky" + ] +} +0 +``` + +`suggestions` is empty because every upstream tool already has a manifest (they are `skipped`). The last line shows the agent +cannot list the other two tools. Stop the gateway: + +```bash +kill $(cat gw.pid) +``` + +**What this proves:** advise mode never blocks; enforce mode validates arguments against the upstream's schema, applies per-tool +timeouts and safe retries, and notices schema drift; manifest edits take effect without a restart, and a broken one is rejected +while the old set stays in force; the agent can ask what a call would do before making it. + +## 6. GitOps with `reversi apply` + +`reversi apply <dir>` makes a registry (and policy) match a directory kept in git. Only differences are published; **nothing is +ever deleted** (`--prune` deprecates tools that left the directory). The directory holds `manifests/*.yaml` (or manifest files +at the top level) and, optionally, `policy.yaml`. + +```bash +mkdir -p $T/ap/desired/manifests && cd $T/ap +cp $T/feat/manifests/*.yaml desired/manifests/ +cat > desired/policy.yaml <<'EOF' +version: 1 +rules: + - id: no-big-priority + match: { tool: notes_add } + when: [{ path: $args.priority, op: gt, value: 3 }] + action: require_approval + reason: high priority notes need a person +tests: + - { name: low priority is fine, input: { tool: notes_add, args: { priority: 1 } }, expect: allow } + - { name: high priority needs approval, input: { tool: notes_add, args: { priority: 5 } }, expect: require_approval } +EOF +ls -R desired +``` + +```text +desired: +manifests +policy.yaml + +desired/manifests: +notes_add.yaml +notes_flaky.yaml +notes_get.yaml +notes_slow.yaml +``` + +### 6.1 Dry run and the drift check + +```bash +rv apply desired --registry reg --policy-dir pol --by gitops --dry-run --detailed-exitcode; echo "exit $?" +ls reg pol 2>&1 +``` + +```text ++ manifest notes_add ++ manifest notes_flaky ++ manifest notes_get ++ manifest notes_slow ++ policy policy +5 changes (dry run: nothing applied) +exit 2 +ls: pol: No such file or directory +ls: reg: No such file or directory +``` + +Exit 2 means "there are changes" and nothing was written. `--detailed-exitcode` sets exit 2 only together with `--dry-run` +(without `--dry-run` a successful apply exits 0). Use it as a CI drift check. + +### 6.2 Apply, re-apply, edit, prune + +```bash +rv apply desired --registry reg --policy-dir pol --by gitops --key $T/gw/keys/signing.key --note "initial import" +echo "-- again" +rv apply desired --registry reg --policy-dir pol --by gitops --key $T/gw/keys/signing.key; echo "exit $?" +rv manifests verify --registry reg --trust $T/gw/keys/signing.pub +``` + +```text ++ manifest notes_add +... +5 changes +applied notes_add v1 (approved) +applied notes_flaky v1 (approved) +applied notes_get v1 (approved) +applied notes_slow v1 (approved) +applied policy v1 +-- again + manifest notes_add unchanged + ... + policy policy unchanged +up to date +exit 0 +ok: 4 tools, every version verifies +``` + +Manifests are applied as `approved` because the directory in git is the reviewed source of truth (`--status draft` to apply drafts). +The second run changes nothing. Edit one file and apply again: + +```bash +sed -i.bak 's/^timeout_ms: 500ms/timeout_ms: 1s/' desired/manifests/notes_slow.yaml && rm desired/manifests/*.bak +rv apply desired --registry reg --policy-dir pol --by gitops --key $T/gw/keys/signing.key --dry-run --detailed-exitcode; echo "exit $?" +rv apply desired --registry reg --policy-dir pol --by gitops --key $T/gw/keys/signing.key --note "slow timeout 1s" +rv manifests diff notes_slow 1 2 --registry reg | grep '^[-+]' +``` + +```text + manifest notes_add unchanged + manifest notes_flaky unchanged + manifest notes_get unchanged +~ manifest notes_slow (v1 differs) + policy policy unchanged +1 change (dry run: nothing applied) +exit 2 +... +applied notes_slow v2 (approved) +- "version": 1, ++ "version": 2, +- "timeout_ms": 500, ++ "timeout_ms": 1000, +``` + +Remove a file from the directory. Without `--prune` the dry run reports nothing; with it the tool is deprecated, not deleted: + +```bash +rm desired/manifests/notes_flaky.yaml +rv apply desired --registry reg --policy-dir pol --by gitops --dry-run | tail -1 +rv apply desired --registry reg --policy-dir pol --by gitops --key $T/gw/keys/signing.key --prune +rv manifests list --registry reg +``` + +```text +up to date + manifest notes_add unchanged + manifest notes_get unchanged + manifest notes_slow unchanged +- manifest notes_flaky (not in the directory) + policy policy unchanged +1 change +deprecated notes_flaky +notes_add v1 approved +notes_flaky v2 deprecated +notes_get v1 approved +notes_slow v2 approved +``` + +### 6.3 Against a running server + +A server only accepts registry changes if its manifests and policy are versioned folders, so seed one with `apply`, then start the +server on them with an admin key and an operator key: + +```bash +mkdir -p seed/manifests && cp $T/feat/manifests/notes_get.yaml seed/manifests/ && printf 'version: 1\nrules: []\n' > seed/policy.yaml +rv apply seed --registry srv-reg --policy-dir srv-pol --by seed --key $T/gw/keys/signing.key | tail -2 +SROOT=root-$(openssl rand -hex 8); SOPS=ops-$(openssl rand -hex 8) +cat > srv-keys.json <<EOF +[ + { "name": "dana", "role": "operator", "key_sha256": "$(sha $SOPS)" }, + { "name": "root", "role": "admin", "key_sha256": "$(sha $SROOT)" } +] +EOF +REVERSI_MANIFESTS=srv-reg REVERSI_POLICY=srv-pol REVERSI_KEYS=srv-keys.json REVERSI_DATA=srv-data \ +REVERSI_SIGNING_KEY=$T/gw/keys/signing.key PORT=7702 node $REPO/packages/server/dist/main.js > srv.log 2>&1 & +echo $! > srv.pid; waitfor 7702; cat srv.log +``` + +```text +applied notes_get v1 (approved) +applied policy v1 +reversi-server listening on http://127.0.0.1:7702 (1 tools, files store, console off) +``` + +```bash +S=http://127.0.0.1:7702 +rv apply desired --server $S --token $SROOT --dry-run --detailed-exitcode; echo "exit $?" +echo "-- operator key" +rv apply desired --server $S --token $SOPS; echo "exit $?" +echo "-- admin key" +rv apply desired --server $S --token $SROOT --note "from git"; echo "exit $?" +rv apply desired --server $S --token $SROOT | tail -1 +cat srv.log | tail -3 +``` + +```text ++ manifest notes_add + manifest notes_get unchanged ++ manifest notes_slow +~ policy policy +3 changes (dry run: nothing applied) +exit 2 +-- operator key ++ manifest notes_add + manifest notes_get unchanged ++ manifest notes_slow +~ policy policy +3 changes +error: POST /v1/registry/manifests: 403 admin role required +exit 1 +-- admin key +... +applied notes_add v1 (approved) +applied notes_slow v1 (approved) +applied policy v2 +exit 0 +up to date +reversi [info] reloaded: 2 manifests, policy 10ddd32c +reversi [info] reloaded: 3 manifests, policy 10ddd32c +reversi [info] reloaded: 3 manifests, policy 4cd2f41a +``` + +An operator key can read the registry (so it can compute the plan) but is refused when the plan would publish an approved +manifest or a policy: that takes the admin role. The server reloads its configuration after each applied change. Stop everything +and clean up: + +```bash +kill $(cat srv.pid) +rm -rf $T +``` + +**What this proves:** a directory in git is the source of truth; `--dry-run --detailed-exitcode` detects drift without changing +anything; apply is idempotent and signs what it publishes; removal is a deprecation, never a deletion; and a server applies only +for an admin key. + +## Not covered here + +- The console (browser) screens for manifests, policy versions and connector health; see [the console](/console). +- `reversi manifests deprecate` and `manifests rollback`, and policy versions on their own. +- A remote `https` manifests source and `manifest_public_key` against it; only a local registry folder was exercised. +- `reversi rollback --compensator-cmd` with a real compensator (shown only as a caution in scenario 4), and `--verify-cmd` and `--reader-cmd`. +- Gateway OIDC authentication, `allow` and `deny` lists, and several upstreams at once. +- The Helm chart, Dockerfile and live third-party connectors are written but not exercised against real clusters or services in this + repository's own test pass. +- OIDC browser-redirect login for the console is not built (login is an API key or an OIDC access token). diff --git a/docs/testing/governance.mdx b/docs/testing/governance.mdx new file mode 100644 index 0000000..641acc7 --- /dev/null +++ b/docs/testing/governance.mdx @@ -0,0 +1,1031 @@ +--- +title: Manual testing - policy, approvals and evidence analysis +description: Hand-test rich manifests, policy rules, budgets, freezes, approvals, delegation, halts, simulation, replay and the console editors, with the output to expect. +--- + +Every command here was run in a scratch folder against the built code, in the order shown. Run them from the **repository root** +in `bash` or `zsh`. Scratch files go in `try/gov/` (delete the folder to start over). The general guide is +[Manual testing guide](/testing); this page covers the governance features only. + +<Note> +You need Node 22.13 or newer, a built repository (`npm install && npm run build`), `curl`, `openssl` and `jq`. +The server listens on port 7750 here. Ids (`apr_...`), timestamps and hashes differ on every run; the rest of each output should match. +</Note> + +## 0. Setup + +### 0.1 Helpers and files + +```bash +mkdir -p try/gov +rv() { node packages/cli/dist/main.js "$@"; } +rvg() { rv --config try/gov/reversi.config.yaml "$@"; } # the CLI with this guide's config file +``` + +Nine tool manifests. Read them: they show the risk fields, the idempotency key, the timeout and the undo of each tool. + +```bash +cat > try/gov/manifests.yaml <<'EOF2' +- tool: read_balance + effect: read_only + permission: payments:read +- tool: refund + effect: idempotent + permission: payments:write +- tool: charge_card + description: Charge a customer's card + effect: compensable + permission: payments:write + owner: payments-team + oncall: "#payments-oncall" + risk: { data_class: financial, blast_radius: high } + timeout_ms: 5s + idempotency: { key: "charge-{{args.order_id}}", scope: run } + compensation: + tool: refund + kind: compensation + residue: ["the charge stays visible on the customer's statement"] + argument_map: { charge_id: $result.charge_id } +- tool: payout + effect: compensable + permission: payments:write + risk: { data_class: financial, blast_radius: medium } + compensation: + tool: refund + argument_map: { charge_id: $result.payout_id } +- tool: revoke_access + effect: idempotent + permission: access:write +- tool: grant_access + effect: reversible + permission: access:write + compensation: + tool: revoke_access + argument_map: { user: $args.user } +- tool: rollback_release + effect: idempotent + permission: deploy:write +- tool: deploy_release + effect: reversible + permission: deploy:write + compensation: + tool: rollback_release + argument_map: { version: $args.version } +- tool: old_charge + effect: compensable + permission: payments:write + compensation: + tool: refund + argument_map: { charge_id: $result.charge_id } +EOF2 +cat > try/gov/draft.yaml <<'EOF2' +- tool: export_report + description: Export a customer report + effect: read_only + permission: reports:read +EOF2 +``` + +The baseline policy (version 1): a rule that routes big payouts to two people of the `finance` group, one that routes access grants to the `security` +group and escalates after 3 seconds, a rate limit, a cumulative budget and risk scoring. + +```bash +cat > try/gov/policy.yaml <<'EOF2' +version: 1 +rules: + - id: big-payouts + match: { tool: payout } + when: [{ path: $args.amount, op: gt, value: 1000 }] + action: require_approval + reason: payouts over 1000 need two finance approvers + approvers: [finance] + quorum: 2 + - id: grant-needs-security + match: { tool: grant_access } + action: require_approval + reason: access grants need the security group + approvers: [security] + approval_timeout: 3s + on_timeout: escalate + escalate_to: [operator] +limits: + - { id: balance-polling, match: { tool: read_balance }, max_calls: 3, window_seconds: 60, per: run } +budgets: + - id: charge-budget + match: { tool: charge_card } + metric: sum + path: $args.amount + max: 1000 + window_seconds: 86400 + per: actor + action: deny +risk: + amount: { path: $args.amount, bands: [{ over: 500, points: 15 }, { over: 5000, points: 30 }] } + thresholds: { approve: 80, deny: 100 } +tests: + - { name: small charge is allowed, input: { tool: charge_card, effect: compensable, args: { order_id: o1, amount: 100 } }, expect: allow } + - { name: big payout needs approval, input: { tool: payout, effect: compensable, args: { amount: 5000 } }, expect: require_approval, rule: big-payouts } +EOF2 +``` + +### 0.2 Registry, policy versions, config and keys + +The console can only edit manifests that live in a **registry folder** and a policy that lives in a **policy versions folder**. Create both +with the CLI, and a project config file that points at them. + +```bash +rvg manifests publish try/gov/manifests.yaml --registry try/gov/registry --status approved --by root --note "initial set" > /dev/null +rvg manifests publish try/gov/draft.yaml --registry try/gov/registry --by root --note "new report export" +rvg manifests deprecate old_charge --registry try/gov/registry --replaced-by charge_card --by root +rvg policy publish try/gov/policy.yaml --dir try/gov/policies --by root --note baseline +cat > try/gov/reversi.config.yaml <<'EOF2' +manifests: registry +policy: policies +data: { dir: data } +EOF2 +``` + +Expect: + +```text +export_report v1 draft b49830181892 +old_charge v2 deprecated; use charge_card +policy v1 2 tests passed active ce2efb2232f0 +``` + +`manifests publish` stores every manifest with the status you pass (`draft` by default) and ignores a `status:` written in the file. A draft is not enforced. + +Keys: an agent, a second agent that belongs to tenant `acme`, an auditor, two approvers in the `finance` group, +one in `security`, an operator and an admin. + +```bash +sha() { printf '%s' "$1" | shasum -a 256 | cut -d' ' -f1; } +for n in AGENT ACME AUD ALICE BOB SAM DANA ROOT; do eval "$n=$(echo $n | tr A-Z a-z)-$(openssl rand -hex 8)"; done +cat > try/gov/keys.json <<EOF2 +[ + { "name": "pay-agent", "role": "agent", "key_sha256": "$(sha $AGENT)", "max_permissions": ["payments:read","payments:write","access:write","deploy:write","reports:read","customers:read"] }, + { "name": "acme-agent", "role": "agent", "key_sha256": "$(sha $ACME)", "tenant": "acme", "max_permissions": ["payments:read"] }, + { "name": "audrey", "role": "auditor", "key_sha256": "$(sha $AUD)" }, + { "name": "alice", "role": "approver", "key_sha256": "$(sha $ALICE)", "groups": ["finance"] }, + { "name": "bob", "role": "approver", "key_sha256": "$(sha $BOB)", "groups": ["finance"] }, + { "name": "sam", "role": "approver", "key_sha256": "$(sha $SAM)", "groups": ["security"] }, + { "name": "dana", "role": "operator", "key_sha256": "$(sha $DANA)" }, + { "name": "root", "role": "admin", "key_sha256": "$(sha $ROOT)" } +] +EOF2 +printf 'AGENT=%s\nACME=%s\nAUD=%s\nALICE=%s\nBOB=%s\nSAM=%s\nDANA=%s\nROOT=%s\n' $AGENT $ACME $AUD $ALICE $BOB $SAM $DANA $ROOT > try/gov/.env.sh # new terminals: . try/gov/.env.sh +``` + +### 0.3 Start the server (terminal A) + +```bash Terminal A +REVERSI_CONFIG=try/gov/reversi.config.yaml REVERSI_KEYS=try/gov/keys.json REVERSI_CONSOLE=login PORT=7750 \ +node packages/server/dist/main.js +``` + +Expect `reversi-server listening on http://127.0.0.1:7750 (9 tools, files store, console login)`. The registry holds ten tools but the draft is not loaded. + +### 0.4 Helpers (terminal B) + +```bash +. try/gov/.env.sh +rv() { node packages/cli/dist/main.js "$@"; } +rvg() { rv --config try/gov/reversi.config.yaml "$@"; } +j() { curl -s -H "Authorization: Bearer $1" -H 'content-type: application/json' "${@:2}"; echo; } +B=localhost:7750 +PERMS='"permissions":["payments:read","payments:write","access:write","deploy:write","reports:read","customers:read"]' +newrun() { j $AGENT -X POST $B/v1/runs -d "{\"run_id\":\"$1\",$PERMS}" | jq -c '{run_id,mode}'; } +call() { j $AGENT -X POST $B/v1/runs/$1/intents -d "{\"tool\":\"$2\",\"args\":$3,\"reason\":\"manual test\"}"; } +finish() { j $AGENT -X POST $B/v1/runs/$1/steps/$2/outcome -d "{\"status\":\"succeeded\",\"result\":$3}"; } +pend() { rvg approvals list --status pending | grep " run=$1 " | awk '{print $1}' | head -1; } +curl -s $B/readyz | jq -c . +``` + +Expect `{"ok":true,"ledger_head":"0000...","halted":null}`. + +## 1. Rich manifests in action + +**What this proves:** the extra manifest fields change what the engine does, and the ledger says why. + +### 1.1 The fields are stored + +```bash +jq -c '{tool,owner,oncall,risk,timeout_ms,idempotency,status}' try/gov/registry/tools/charge_card/v1.json +``` + +Expect owner `payments-team`, risk `financial` / `high`, `timeout_ms:"5s"`, and the idempotency template. + +### 1.2 A duplicate call is blocked + +```bash +newrun m-1 +call m-1 charge_card '{"order_id":"A100","amount":200}' +finish m-1 0 '{"charge_id":"ch_1"}' +call m-1 charge_card '{"order_id":"A100","amount":200}' +``` + +Expect `allow`, then the second identical order is refused: + +```text +{"decision":"block","reasons":["duplicate call: idempotency key matches step 0 of run m-1 (succeeded)"]} +``` + +The key is `charge-A100`, rendered from the arguments, and `scope: run` means a different run could charge A100 again (use `scope: actor` or `global` to widen it). + +### 1.3 A deprecated tool still runs, and is flagged + +```bash +call m-1 old_charge '{"amount":10}' +finish m-1 1 '{"charge_id":"ch_old"}' +rvg search --run m-1 --event intended,blocked | cut -c1-170 +``` + +Expect the call to be allowed (`{"decision":"allow","step":1}`), and the events to carry the risk score, the duplicate refusal and the deprecation note: + +```text + 1 ...Z m-1#0 intended charge_card pay-agent | risk 70 (effect compensable +30, blast radius high +20, data class financial +20) + 3 ...Z m-1 blocked charge_card pay-agent | duplicate call: idempotency key matches step 0 of run m-1 (succeeded) + 4 ...Z m-1#1 intended old_charge pay-agent | manifest deprecated; use charge_card; risk 30 (effect compensable +30) +``` + +### 1.4 A draft is not enforced + +```bash +call m-1 export_report '{}' +``` + +With a registry, a draft is simply not loaded, so the call is refused as unknown: + +```text +{"decision":"block","reasons":["no manifest registered for tool `export_report`"]} +``` + +With a plain manifest file, `status: draft` is loaded and refused by name. Try it on a throwaway second server (port 7751): + +```bash +cat > try/gov/draft-demo.yaml <<'EOF2' +- tool: export_report + effect: read_only + permission: reports:read + status: draft +EOF2 +REVERSI_MANIFESTS=try/gov/draft-demo.yaml REVERSI_KEYS=try/gov/keys.json REVERSI_DATA=try/gov/draft-data PORT=7751 node packages/server/dist/main.js & +SRV=$!; sleep 2 +j $AGENT -X POST localhost:7751/v1/runs -d '{"run_id":"d-1","permissions":["reports:read"]}' | jq -c .run_id +j $AGENT -X POST localhost:7751/v1/runs/d-1/intents -d '{"tool":"export_report","args":{},"reason":"x"}' +kill $SRV +``` + +Expect `{"decision":"block","reasons":["the manifest for `export_report` is a draft: publish it before it can be enforced"]}`. + +### 1.5 Timeout + +`timeout_ms` bounds the **upstream call made by the [gateway](/gateway)**: after that many milliseconds the gateway reports +`upstream did not answer within N ms`. The HTTP server does not call tools itself, so there is nothing to time out here; this page does not exercise it (see "Not covered here"). + +## 2. Policy: rules, limits, budgets, risk, windows, freezes, versions + +**What this proves:** policy can refuse or hold a call for reasons other than the manifest, and each refusal names the rule. + +### 2.1 Check and test the file + +```bash +rvg policy check try/gov/policy.yaml +rvg policy test try/gov/policy.yaml +``` + +```text +ok: 2 rules, 1 limits, default allow +ok small charge is allowed +ok big payout needs approval +2 passed +``` + +### 2.2 A rate limit + +```bash +newrun p-1 +for i in 1 2 3 4; do call p-1 read_balance '{"account":"acc1"}'; done +``` + +Expect three `allow` lines, then `{"decision":"block","reasons":["limit `balance-polling`: 3 calls per 60s exceeded"]}`. + +### 2.3 Risk scoring + +Thresholds are `approve: 80`, `deny: 100`. A 600 charge scores 30 + 20 + 20 + 15 = 85, so a human is needed. A 9000 charge scores 100 and is refused. + +```bash +call p-1 charge_card '{"order_id":"R1","amount":600}' +call p-1 charge_card '{"order_id":"R2","amount":9000}' +``` + +```text +{"decision":"block","reasons":["approval required: request apr_... is pending an operator decision; retry after it is approved"]} +{"decision":"block","reasons":["budget `charge-budget`: 200 of 1000 used in 86400s, this call would make it 9200","risk score 100 is at or above the deny threshold 100 (effect compensable +30, blast radius high +20, data class financial +20, amount over 5000 +30)"]} +``` + +Both reasons are reported: the 9000 charge also breaks the budget of section 2.4. + +The pending request for R1 is decided in section 3. + +### 2.4 A cumulative budget + +The budget is 1000 per actor per day, and `m-1` already spent 200. Two charges of 300 fit; the warning at 80 percent is written to the ledger; the third does not fit. + +```bash +call p-1 charge_card '{"order_id":"B1","amount":300}' +call p-1 charge_card '{"order_id":"B2","amount":300}' +call p-1 charge_card '{"order_id":"B3","amount":300}' +rvg search --run p-1 --text budget | cut -c1-200 +``` + +```text +{"decision":"allow","step":3} +{"decision":"allow","step":4} +{"decision":"block","reasons":["budget `charge-budget`: 800 of 1000 used in 86400s, this call would make it 1100"]} + 13 ...Z p-1 blocked charge_card pay-agent | budget `charge-budget`: 200 of 1000 used in 86400s, this call would make it 9200; risk sco + 15 ...Z p-1#4 intended charge_card pay-agent | budget `charge-budget` at 80% (800 of 1000); risk 70 (effect compensable +30, blast radius high +20, ... + 16 ...Z p-1 blocked charge_card pay-agent | budget `charge-budget`: 800 of 1000 used in 86400s, this call would make it 1100 +``` + +The budget counts only calls that were allowed to start, and is shared by every run of the same actor (`per: actor`). + +### 2.5 Version 2: a time window, a freeze, and a window that is closed + +The next file is version 1 plus the lines marked `new`. The day names and the freeze period are computed so that one window covers **now** +and one does not. All times are UTC. + +```bash +TODAY=$(node -e 'console.log(["sun","mon","tue","wed","thu","fri","sat"][new Date().getUTCDay()])') +TOMORROW=$(node -e 'console.log(["sun","mon","tue","wed","thu","fri","sat"][(new Date().getUTCDay()+1)%7])') +FROM=$(node -e 'console.log(new Date(Date.now()-3600e3).toISOString())') +TO=$(node -e 'console.log(new Date(Date.now()+3600e3).toISOString())') +cat > try/gov/policy-v2.yaml <<EOF2 +version: 1 +rules: + - id: big-payouts + match: { tool: payout } + when: [{ path: \$args.amount, op: gt, value: 1000 }] + action: require_approval + reason: payouts over 1000 need two finance approvers + approvers: [finance] + quorum: 2 + - id: grant-needs-security + match: { tool: grant_access } + action: require_approval + reason: access grants need the security group + approvers: [security] + approval_timeout: 3s + on_timeout: escalate + escalate_to: [operator] + - id: payouts-paused-today # new: the window covers now + match: { tool: payout } + window: { days: [$TODAY] } + action: deny + reason: the payout run is paused today + - id: refunds-closed-tomorrow # new: the window does not cover now + match: { tool: refund } + window: { days: [$TOMORROW] } + action: deny + reason: refunds are closed tomorrow +freezes: # new + - id: release-freeze + from: "$FROM" + to: "$TO" + reason: quarterly release freeze + match: { tool: "deploy_*" } +limits: + - { id: balance-polling, match: { tool: read_balance }, max_calls: 3, window_seconds: 60, per: run } +budgets: + - id: charge-budget + match: { tool: charge_card } + metric: sum + path: \$args.amount + max: 1000 + window_seconds: 86400 + per: actor + action: deny +risk: + amount: { path: \$args.amount, bands: [{ over: 500, points: 15 }, { over: 5000, points: 30 }] } + thresholds: { approve: 80, deny: 100 } +tests: + - { name: small charge is allowed, input: { tool: charge_card, effect: compensable, args: { order_id: o1, amount: 100 } }, expect: allow } + - { name: payouts are paused today, input: { tool: payout, effect: compensable, args: { amount: 50 } }, expect: deny, rule: payouts-paused-today } + - { name: tomorrow's window does not fire today, input: { tool: refund, effect: idempotent, args: {} }, expect: allow } + - { name: deploys are frozen, input: { tool: deploy_release, effect: reversible, args: {} }, expect: deny, rule: release-freeze } + - { name: reads are never frozen, input: { tool: read_balance, effect: read_only, args: {} }, expect: allow } +EOF2 +rvg policy test try/gov/policy-v2.yaml +``` + +```text +ok small charge is allowed +ok payouts are paused today +ok tomorrow's window does not fire today +ok deploys are frozen +ok reads are never frozen +5 passed +``` + +<Note> +A window rule fires only while the time is inside the window (`days`, `between: "09:00-17:00"`, `timezone`; `outside: true` inverts it). +A freeze with no `match` blocks every call that is not `read_only`. Both read the real clock, so this file is only valid on the day you create it. +I dropped the version-1 "big payout" test because payouts are now paused. +</Note> + +### 2.6 Publish it, watch it enforce, then roll back + +```bash +rvg policy publish try/gov/policy-v2.yaml --by root --note "pause payouts today, release freeze, closed-tomorrow refunds" +sleep 3 # the server notices the new active version within a few seconds +newrun p-2 +call p-2 payout '{"amount":50}' +call p-2 deploy_release '{"version":"1.4"}' +call p-2 refund '{"charge_id":"ch_1"}' +call p-2 read_balance '{"account":"x"}' +``` + +```text +policy v2 5 tests passed active ... +{"run_id":"p-2","mode":"enforce"} +{"decision":"block","reasons":["policy `payouts-paused-today`: the payout run is paused today"]} +{"decision":"block","reasons":["freeze `release-freeze`: quarterly release freeze (until 2026-...Z)"]} +{"decision":"allow","step":0} +{"decision":"allow","step":1} +``` + +The open window blocks the payout, the freeze blocks the deploy, the closed window lets the refund through, and a read is never frozen. +Now the history, the diff, and the rollback to version 1 (a policy version is activated, never edited): + +```bash +rvg policy history +rvg policy diff 1 2 | grep -E '^[+-]' | head -12 +rvg policy activate 1 +sleep 3 +call p-2 deploy_release '{"version":"1.4"}' +``` + +```text +v1 ...Z by root 2 tests ce2efb2232f0 - baseline +v2 *active* ...Z by root 5 tests ... - pause payouts today, release freeze, closed-tomorrow refunds ++ - id: payouts-paused-today +... +policy v1 is now active +{"decision":"allow","step":2} +``` + +(`history` prints `*active*` on v2 before the rollback; the last command shows the deploy allowed again once v1 is back.) + +## 3. Approvals + +**What this proves:** approvals can be routed to groups, need N different people, can be delegated, escalate when nobody answers, and can be forced +only by an admin with evidence. The requester can never approve their own call. + +### 3.1 Routing to a group, quorum 2 + +```bash +newrun a-1 +call a-1 payout '{"amount":5000,"to":"acct_9"}' +rvg approvals list --status pending +ID=$(pend a-1); echo $ID +j $AGENT -X POST $B/v1/approvals/$ID/approve -d '{}' +j $SAM -X POST $B/v1/approvals/$ID/approve -d '{}' +j $ALICE -X POST $B/v1/approvals/$ID/approve -d '{"note":"invoice 8841 matches"}' | jq -c '{status,quorum,decisions:[.decisions[]|{by,basis}]}' +j $ALICE -X POST $B/v1/approvals/$ID/approve -d '{"note":"again"}' +j $BOB -X POST $B/v1/approvals/$ID/approve -d '{"note":"second look done"}' | jq -c '{status,decided_by}' +call a-1 payout '{"amount":5000,"to":"acct_9"}' +finish a-1 0 '{"payout_id":"po_1"}' +``` + +```text +{"decision":"block","reasons":["approval required: request apr_... is pending an operator decision (needs 2 approvers from finance); retry after it is approved"]} +apr_... pending run=p-1 call charge_card args=[order_id,amount] why: risk score 85 is at or above the approval threshold 80 (...) +apr_... pending run=a-1 call payout args=[amount,to] why: policy `big-payouts`: payouts over 1000 need two finance approvers +{"error":"approver role required"} +{"error":"this request needs an approver from: finance"} +{"status":"pending","quorum":2,"decisions":[{"by":"alice","basis":"group:finance"}]} +{"error":"you have already decided this request"} +{"status":"approved","decided_by":"alice, bob"} +{"decision":"allow","step":0} +``` + +The list also holds the risk-score request from section 2.3. It shows argument **names** (`amount`, `to`), never values. The approval is bound to the exact arguments and used once. + +### 3.2 The same from the CLI, with `--groups` + +The CLI trusts the host, so you say who you are and which groups you decide as: + +```bash +call a-1 payout '{"amount":2500,"to":"acct_7"}' | cut -c1-80 +ID=$(pend a-1) +rvg approvals approve $ID --as sam --groups security --note "wrong group" +rvg approvals approve $ID --as alice --groups finance --note ok +rvg approvals approve $ID --as bob --groups finance --note ok +rvg approvals list --status approved | tail -1 | cut -c1-60 +``` + +```text +error: this request needs an approver from: finance +apr_... pending by null +apr_... approved by alice, bob +apr_... approved run=a-1 call payout ... +``` + +(`pending by null` after the first vote means the quorum is not reached yet.) The CLI decides through the engine, so the +decision is in the ledger too, and notification channels in the config would be told: + +```bash +rvg export --run a-1 | jq -r 'select(.event=="approved") | .actor + ": " + .detail' | tail -1 | cut -c1-120 +``` + +```text +bob: approval apr_... approved by alice, bob [quorum 2]; the call must be retried +``` + +Deny works the same way, and a denied call stays blocked with the reason: + +```bash +call a-1 payout '{"amount":8000,"to":"acct_6"}' | cut -c1-70 +ID=$(pend a-1); rvg approvals deny $ID --as alice --groups finance --note "not today" +call a-1 payout '{"amount":8000,"to":"acct_6"}' +``` + +```text +{"decision":"block","reasons":["approval required: request apr_... +apr_... denied by alice +{"decision":"block","reasons":["approval apr_... was denied: not today"]} +``` + +### 3.3 Delegation: grant, use, expire + +Alice hands her `finance` authority to dana for a very short time (0.005 hours is 18 seconds; the console's form takes whole hours). Dana is an operator, not in `finance`. + +```bash +call a-1 payout '{"amount":3000,"to":"acct_5"}' | cut -c1-70 +ID=$(pend a-1) +j $DANA -X POST $B/v1/approvals/$ID/approve -d '{}' +rvg delegate add --from alice --to dana --hours 0.005 --reason "alice on leave" --group finance +rvg delegate list +j $DANA -X POST $B/v1/approvals/$ID/approve -d '{"note":"covering for alice"}' | jq -c '{status,decisions:[.decisions[]|{by,on_behalf_of,basis}]}' +j $ALICE -X POST $B/v1/approvals/$ID/approve -d '{}' +j $BOB -X POST $B/v1/approvals/$ID/approve -d '{}' | jq -c '{status,decided_by}' +``` + +```text +{"error":"this request needs an approver from: finance"} +dlg_... alice -> dana until ...Z +dlg_... alice -> dana until ...Z active alice on leave +{"status":"pending","decisions":[{"by":"dana","on_behalf_of":"alice","basis":"delegated by alice (finance)"}]} +{"error":"you have already decided this request"} +{"status":"approved","decided_by":"dana, bob"} +``` + +A delegate's vote counts as the delegator's, so Alice cannot also vote. After 18 seconds the delegation is spent: + +```bash +call a-1 payout '{"amount":4000,"to":"acct_4"}' | cut -c1-70 +ID4=$(pend a-1) +sleep 20 +rvg delegate list +j $DANA -X POST $B/v1/approvals/$ID4/approve -d '{}' +``` + +```text +dlg_... alice -> dana until ...Z expired alice on leave +{"error":"this request needs an approver from: finance"} +``` + +Delegation is honoured everywhere a decision is made, including the CLI on the host. Repeat the first half with a fresh request, and let Dana decide with `reversi approvals approve` instead of the API (no `--groups`; the delegation supplies `finance`): + +```bash +call a-1 payout '{"amount":3500,"to":"acct_3"}' | cut -c1-70 +ID=$(pend a-1) +rvg delegate add --from alice --to dana --hours 1 --reason "alice on leave" --group finance +rvg approvals approve $ID --as dana --note "covering for alice" +rvg approvals approve $ID --as bob --groups finance +rvg export --run a-1 | jq -r 'select(.event=="approved") | .detail' | tail -1 | cut -c1-120 +``` + +```text +apr_... pending by null +apr_... approved by dana, bob +approval apr_... approved by dana, bob [quorum 2]; the call must be retried +``` + +That delegation stays active for an hour; end it with `rvg delegate revoke <id>` (the id is in `rvg delegate list`) before the later sections. + +### 3.4 Escalation on timeout + +`grant_access` goes to the `security` group with a 3 second timeout, then to `operator`. Approve quickly after the escalation: the approval itself lapses at the same deadline. + +```bash +newrun e-1 +call e-1 grant_access '{"user":"newhire","role":"admin"}' +j $DANA "$B/v1/approvals?status=pending&run_id=e-1" | jq -c '.approvals[] | {required_roles,escalated,on_timeout,escalate_to}' +sleep 4 +j $DANA "$B/v1/approvals?status=pending&run_id=e-1" | jq -c '.approvals[] | {required_roles,escalated}' +ID=$(pend e-1) +j $SAM -X POST $B/v1/approvals/$ID/approve -d '{}' +j $DANA -X POST $B/v1/approvals/$ID/approve -d '{"note":"security did not answer"}' | jq -c '{status,decided_by}'; call e-1 grant_access '{"user":"newhire","role":"admin"}' +rvg search --run e-1 | cut -c1-150 +``` + +```text +{"decision":"block","reasons":["approval required: request apr_... is pending ... (needs an approver from security); ..."]} +{"required_roles":["security"],"escalated":false,"on_timeout":"escalate","escalate_to":["operator"]} +{"required_roles":["operator"],"escalated":true} +{"error":"this request needs an approver from: operator"} +{"status":"approved","decided_by":"dana"} +{"decision":"allow","step":0} + ... e-1 escalated grant_access reversi | escalated to operator +``` + +After escalation `sam` (security) is refused and any `operator` is accepted. + +### 3.5 Break-glass + +Only an admin can force a pending approval, with a reason of at least 10 characters and a reference to the evidence. It is recorded as break-glass, not as a normal approval. + +```bash +j $DANA -X POST $B/v1/approvals/$ID4/break-glass -d '{"reason":"payroll outage, vendor will cancel","evidence":"INC-2041"}' +j $ROOT -X POST $B/v1/approvals/$ID4/break-glass -d '{"reason":"too short","evidence":"INC-2041"}' +j $ROOT -X POST $B/v1/approvals/$ID4/break-glass -d '{"reason":"payroll outage, vendor will cancel","evidence":"INC-2041"}' | jq -c '{status,decided_by,decision_note,break_glass}' +call a-1 payout '{"amount":6000,"to":"acct_3"}' | cut -c1-60 +ID5=$(pend a-1) +rvg break-glass $ID5 --reason "audit deadline today" --evidence INC-2042 --by root +rvg search --run a-1 --event approved | grep BREAK | cut -c1-170 +``` + +```text +{"error":"admin role required"} +{"error":"break-glass needs a reason of at least 10 characters"} +{"status":"approved","decided_by":"root","decision_note":"BREAK-GLASS: payroll outage, vendor will cancel","break_glass":{...,"evidence":"INC-2041",...}} +apr_... approved by break-glass (root); evidence INC-2042 +... a-1 approved payout root | BREAK-GLASS: approval apr_... forced by root; evidence: INC-2041 +``` + +### 3.6 Decide in the console (login mode, key paste) + +1. Create a request: + +```bash +call a-1 payout '{"amount":7000,"to":"acct_2"}' | cut -c1-70 +``` + +2. Open [http://localhost:7750/console/login](http://localhost:7750/console/login), paste the value of `$SAM` (print it with `echo $SAM`), **Sign in**. +3. Click **Approvals**. In the *Pending* table the row for `payout` shows **Who may decide: finance (0/2)**, the policy reason, and the argument names. Type a note and click **Approve**: the page says *Could not decide: this request needs an approver from: finance*. +4. **Log out**, sign in with `$ALICE`, **Approvals**, **Approve**. The row now shows `finance (1/2)`. Alice cannot vote twice. +5. **Log out**, sign in with `$BOB`, **Approve**. The request moves to *Recent decisions* as `approved` by `alice, bob`. +6. Sign in with `$ROOT` (admin): pending rows also have a **Break-glass** form (reason and ticket). Under *Delegation* you can hand authority to someone for whole hours and **Revoke** it. +7. Sign in with `$AUD` (auditor): the page has no **Approve** buttons. + +Then let the agent use it: + +```bash +call a-1 payout '{"amount":7000,"to":"acct_2"}' +``` + +Expect `{"decision":"allow","step":...}`. + +## 4. Scoped halts and kill switches + +**What this proves:** a halt can be narrowed to a tool, an actor or a tenant, only an admin can set it, and lifting it restores service. + +```bash +rvg halt add "drill: stop payouts" --tool payout --by dana +rvg halt list +call a-1 payout '{"amount":10,"to":"x"}' +call a-1 read_balance '{"account":"x"}' +rvg halt lift $(rvg halt list | awk '{print $1}' | head -1) +``` + +```text +halt_... drill: stop payouts ({"tool":"payout"}) +halt_... drill: stop payouts scope={"tool":"payout"} by dana +{"decision":"block","reasons":["kill switch: drill: stop payouts (scope: tool=payout)"]} +{"decision":"allow","step":...} +lifted 1 +``` + +Scope by **actor** from the API (admin only), then by **tenant** from the CLI. The `acme-agent` key carries `tenant: acme`: + +```bash +j $ROOT -X POST $B/v1/halt -d '{"reason":"stop one agent","actor":"pay-agent"}' | jq -c .halt.scope +call a-1 read_balance '{"account":"y"}' +j $DANA -X POST $B/v1/halt -d '{"reason":"x"}' +j $ROOT -X POST $B/v1/resume -d '{}' +rvg halt add "acme maintenance" --tenant acme --by dana +j $ACME -X POST $B/v1/runs -d '{"run_id":"t-1","permissions":["payments:read"]}' | jq -c .run_id +j $ACME -X POST $B/v1/runs/t-1/intents -d '{"tool":"read_balance","args":{},"reason":"x"}' +call a-1 read_balance '{"account":"z"}' +rvg halt lift +``` + +```text +{"actor":"pay-agent"} +{"decision":"block","reasons":["kill switch: stop one agent (by root) (scope: actor=pay-agent)"]} +{"error":"admin role required"} +{"halted":null,"lifted":1} +"t-1" +{"decision":"block","reasons":["kill switch: acme maintenance (scope: tenant=acme)"]} +{"decision":"allow","step":...} +lifted 1 +``` + +In the console: sign in as `$ROOT`, **Evidence**, *Kill switches*. Fill **reason** and one of **tool glob / actor / tenant / environment**, click **Block new calls**; the new row shows its scope and a +**Lift** button. A red *Kill switch active* card appears on every page only for a global halt (no scope) or the kill-switch file; a scoped halt appears in the table only. An operator sees the table but not the form. + +## 5. Capture, simulation, replay, impact and speculation + +**What this proves:** with capture off the history features say so and nothing breaks; with capture on you can test a policy change against real calls before you publish it. + +### 5.1 With capture off (the default) + +Create a candidate policy first (it is used again below), then ask for each history feature: + +```bash +cat > try/gov/candidate.yaml <<'EOF2' +version: 1 +rules: + - id: payouts-need-approval + match: { tool: payout } + when: [{ path: $args.amount, op: gt, value: 400 }] + action: require_approval + reason: any payout over 400 needs a human +EOF2 +rvg capture list +rvg simulate --policy try/gov/candidate.yaml; echo "exit $?" +rvg replay a-1 --set amount=9000; echo "exit $?" +j $DANA -X POST $B/v1/simulate -d '{}' +``` + +```text +capture is off. Turn it on with `capture: { enabled: true }` in reversi.config.yaml, or REVERSI_CAPTURE=1 +error: simulation needs captured calls: turn capture on first (it keeps redacted arguments for a limited time) +exit 1 +error: replay needs captured calls: turn capture on first +exit 1 +{"error":"capture is off, so there is no history to simulate against"} +``` + +In the browser (sign in as `$DANA` or `$ROOT`): open `/console/replay/a-1`, type path `amount`, value `9000`, **Replay**: *Nothing is captured for that run (capture must be on, and captures expire).* +**Policy**, then **Simulate against captured history**: *Capture is off, so there is no history to simulate against.* + +Impact preview and speculation work without capture, because they only read manifests and policy: + +```bash +rvg impact payout --args '{"amount":5000}' +rvg impact charge_card --args '{"order_id":"Z1","amount":600}' +``` + +```text +payout: compensable +decision now: needs_approval (approval required: simulated) +risk 75: effect compensable +30, blast radius medium +10, data class financial +20, amount over 500 +15 +data class financial, blast radius medium +rollback: undo by calling refund +warning: the undo is not verified: nothing reads back that it worked +``` + +The second command prints the same layout for `charge_card`: `decision now: needs_approval`, `risk 85`, the owner, and +`rollback: but the charge stays visible on the customer's statement` (the residue from its manifest). + +Compare three plans without running anything: + +```bash +cat > try/gov/plans.yaml <<'EOF2' +plans: + - name: one big payout + calls: + - { tool: payout, args: { amount: 2000, to: acct_1 }, result: { payout_id: po_1 } } + - name: two small payouts + calls: + - { tool: payout, args: { amount: 900, to: acct_1 }, result: { payout_id: po_1 } } + - { tool: payout, args: { amount: 900, to: acct_1 }, result: { payout_id: po_2 } } + - name: ship it + calls: + - { tool: deploy_release, args: { version: "2.1" } } + - { tool: export_report, args: {} } +EOF2 +rvg speculate try/gov/plans.yaml +``` + +```text +two small payouts <- recommended + can run to the end; approvals needed: 0; highest risk: 75; fully reversible +one big payout + can run to the end; approvals needed: 1; highest risk: 75; fully reversible +ship it + blocked at call 1: no manifest registered for tool `export_report`; approvals needed: 0; highest risk: 15; fully reversible + note: some results were synthesized from the manifests, so the rollback outlook assumes compensation arguments are available +``` + +It also shows a real gap: splitting one payout in two avoids the per-call approval rule. A budget on payouts (like `charge-budget`) is the way to close it. + +The console equivalent is **Approvals**, the *impact preview* link on a pending row, or `/console/impact`: type a tool, JSON arguments, **Preview**. +The API equivalents are `GET /v1/impact?tool=...&args=...` and `POST /v1/speculate`: + +```bash +curl -s -G -H "Authorization: Bearer $AUD" $B/v1/impact --data-urlencode tool=payout --data-urlencode 'args={"amount":5000}' | jq -c '{tool,verdict,risk_score}' +``` + +### 5.2 Turn capture on + +Stop terminal A with Ctrl-C, add the setting, start it again with the same command as in 0.3. (Browser sessions end when the server restarts.) + +```bash +printf 'capture: { enabled: true }\n' >> try/gov/reversi.config.yaml +``` + +```bash Terminal A +REVERSI_CONFIG=try/gov/reversi.config.yaml REVERSI_KEYS=try/gov/keys.json REVERSI_CONSOLE=login PORT=7750 \ +node packages/server/dist/main.js +``` + +Now make a run to analyse. Only calls made from now on are captured, with secrets redacted and a time limit. + +```bash +newrun c-1 +call c-1 read_balance '{"account":"acc1"}' +call c-1 payout '{"amount":500,"to":"acct_2"}' +finish c-1 1 '{"payout_id":"po_c1"}' +call c-1 payout '{"amount":1500,"to":"acct_3"}' | cut -c1-90 +call c-1 deploy_release '{"version":"2.0"}' +rvg capture list +``` + +```text +...Z c-1#0 read_balance allow args={"account":"acc1"} +...Z c-1#1 payout allow succeeded args={"amount":500,"to":"acct_2"} +...Z c-1 payout block args={"amount":1500,"to":"acct_3"} +...Z c-1#2 deploy_release allow args={"version":"2.0"} +``` + +### 5.3 Simulate a policy against that history + +```bash +rvg simulate --policy try/gov/candidate.yaml +``` + +```text +4 captured calls in 1 runs: 3 unchanged + +newly needing approval (1) + c-1#1 payout allow -> needs_approval approval required: simulated +``` + +The 500 payout was allowed; under the candidate (over 400 needs approval) it would have needed one. In the browser: **Policy**, edit the text (change `value: 1000` to `value: 400`), +click **Simulate against captured history**: *Last 7 days of captured calls ... Newly needing approval*. Nothing is published by simulating. + +### 5.4 Replay one run with a changed argument + +```bash +rvg replay c-1 --set 'tool:payout/amount=9000' +``` + +```text +run c-1: 4 captured calls + 0 read_balance as recorded: allow with your changes: allow + 1 payout as recorded: allow with your changes: needs_approval (approval required: simulated) + 2 payout as recorded: needs_approval with your changes: needs_approval (approval required: simulated) + 3 deploy_release as recorded: allow with your changes: allow +1 decision would change +rollback as recorded: fully reversible; with changes: fully reversible +``` + +In the browser: **Runs**, open `c-1`: the page links **Replay with a change** (only when capture is on). Fill *Tool* `payout`, *Argument path* `amount`, *New value* `9000`, **Replay**: +a table of *As recorded* against *With your change*, ending *1 decision would change*. Replay needs the operator role. + +Equivalent API calls: `POST /v1/simulate` with `{"policy":"<yaml text>"}` and `POST /v1/runs/c-1/replay` with `{"changes":[{"tool":"payout","path":"amount","value":9000}]}`, both with an operator key. + +## 6. Console editors + +**What this proves:** a manifest or policy change can be authored, validated, reviewed and activated in the browser, with the right role at each step. + +Requirements: manifests must come from a **registry folder** (created in 0.2) and the policy from a **policy versions folder**. If you point the server at plain files the +console shows them read-only and says so. Roles: *operator* can save a manifest as a draft; only *admin* can approve a draft, publish as approved, and publish or activate a policy. + +### 6.1 A manifest with the form builder + +1. Sign in as `$DANA` (operator). **Manifests** lists the 9 enforced tools and, in a box, *Drafts waiting for approval: export_report*. +2. Click **New manifest**. Fill the form: Tool name `lookup_customer`, Description `Look up a customer record`, Effect `read_only`, Permission `customers:read`, Owner `support-team`, Data class `pii`, Blast radius `low`, Timeout `10s`. Leave Undo empty. Click **Build the YAML**. +3. Below, *The YAML* shows the manifest and a green *Valid. Lint: no findings*. Click **Validate** to re-check (edit the text to see errors, for example set `effect: nope`). +4. Add a note and click **Save as draft**. You land on the manifest page: status *draft*, v1, with a *History* table. +5. Check the agent cannot use it yet: + +```bash +newrun m-2 +call m-2 lookup_customer '{"id":"c1"}' +``` + +```text +{"decision":"block","reasons":["no manifest registered for tool `lookup_customer`"]} +``` + +6. As operator you cannot approve: there is no **Approve the latest draft** button. Log out, sign in as `$ROOT` (admin), open **Manifests**, click `lookup_customer`, tick *the author may not approve it*, click **Approve the latest draft**. + History now has v2 *approved*, "by dana, approved by root". The server reloads itself. +7. Reload the agent side: the same call is now allowed. + +```bash +call m-2 lookup_customer '{"id":"c1"}' +rvg manifests history lookup_customer +``` + +```text +{"decision":"allow","step":0} +v1 draft ...Z by dana ... - customer lookup for support +v2 approved ...Z by dana, approved by root ... - approved from v1 +``` + +Also on that page: *Contract tests* (a YAML list, **Run the tests**), **Deprecate** (with a replacement tool), and **Restore** on any older version. + +A manifest's `approval:` object (roles, quorum, timeout) and the *Approver groups* field are kept intact by save, approve, deprecate and restore, and the registry loads with them. This page routes approvals with policy rules instead; see section 3 for the effect of each. + +### 6.2 The policy editor + +1. Sign in as `$ROOT`. Click **Policy**. The top line reads *2 rules, 1 limits, 1 budgets, 0 freezes ... active v1* and *2 of 2 of the policy's own tests pass*. +2. Use the **Rule builder** if you like (it appends a rule to the text), or edit the text directly: change the `big-payouts` condition `value: 1000` to `value: 400`. +3. Click **Validate and run its tests**: *Valid. 2 tests pass* and a *Changes from the active version* diff. +4. Click **Simulate against captured history**: the 500 payout is *Newly needing approval*. +5. Type a note (`lower payout threshold to 400`) and click **Publish and activate** (admin). The *Versions* table now lists v1, v2 and v3; v3 has the *active* badge. +6. Click **diff** on the v3 row (v2 to v3), then **Activate** on any other row to roll back. The server reloads on every activation. + +```bash +rvg policy history +call c-1 payout '{"amount":500,"to":"acct_9"}' | cut -c1-110 +``` + +```text +v1 ...Z by root 2 tests ce2efb2232f0 - baseline +v2 ...Z by root 5 tests ... - pause payouts today, release freeze, closed-tomorrow refunds +v3 *active* ...Z by root 2 tests ... - lower payout threshold to 400 +{"decision":"block","reasons":["approval required: request apr_... is pending an operator decision (needs 2 approvers from finance); retry after it is approved"]} +``` + +## 7. Run inspection and search + +**What this proves:** every run and call can be found and explained afterwards, from the CLI, the API and the console. + +```bash +rvg runs list +rvg runs list --status needs_attention --tool payout +rvg actions list --tool payout --limit 3 +rvg actions show a-1 0 +rvg actions show c-1 1 | grep captured +rvg search --event blocked --text budget | cut -c1-150 +``` + +```text +a-1 actor=pay-agent mode=enforce steps=4 needs_attention UNRESOLVED=1,2,3 +c-1 actor=pay-agent mode=enforce steps=3 needs_attention UNRESOLVED=0,2 +e-1 actor=pay-agent mode=enforce steps=1 needs_attention UNRESOLVED=0 +m-1 actor=pay-agent mode=enforce steps=2 complete +... +t-1 actor=acme-agent mode=enforce steps=0 blocked_only tenant=acme +a-1 actor=pay-agent mode=enforce steps=4 needs_attention UNRESOLVED=1,2,3 +c-1 ... +p-2 ... +2026-10-16 ...Z a-1#0 payout compensable succeeded pay-agent risk 75 +... +a-1 step 0: payout [compensable] succeeded +why: manual test +manifest v1, risk 75 +undo: refund (compensation) +undo arguments: {"charge_id":"po_1"} +result hash: ... +arguments are not stored (only their hash is in the ledger); turn on capture to keep redacted copies +... + approval apr_...: consumed by alice, bob +... +captured arguments (redacted): {"amount":500,"to":"acct_2"} + 13 ...Z p-1 blocked charge_card pay-agent | budget `charge-budget`: 200 of 1000 used in 86400s, this call would make it 9200; risk sco + 16 ...Z p-1 blocked charge_card pay-agent | budget `charge-budget`: 800 of 1000 used in 86400s, this call would make it 1100 +``` + +Runs with an unresolved step (an intent whose outcome was never reported, like most of these test calls) show as `needs_attention`. +`actions show` lists the approvals behind a call; `a-1` was run before capture was on, so it can only say the arguments were not kept, while `c-1` shows the redacted copy. + +The search API needs an auditor key or better: + +```bash +j $AUD "$B/v1/search/runs?tool=payout&limit=2" | jq -c '{runs:[.runs[]|{run_id,status,tools}],next}' +j $AUD "$B/v1/search/records?event=blocked&text=limit" | jq -c '.records[]|{seq,run_id,event,detail}' +``` + +```text +{"runs":[{"run_id":"a-1","status":"needs_attention","tools":["payout","read_balance"]},{"run_id":"c-1",...}],"next":"c-1"} +{"seq":...,"run_id":"p-1","event":"blocked","detail":"limit `balance-polling`: 3 calls per 60s exceeded"} +``` + +Both take `actor`, `tool`, `since`, `until`, `text` and `limit` (runs also `status`, `tenant`, `mode`, `after`; records also `run`, `event`, `after_seq`). Page through with `next`. + +In the console: **Runs** has filters (actor, tenant, tool, *needs attention*). Open any run for steps, the rollback plan, approvals and the event trail. Click **Compare two runs** (or open +`/console/compare?a=a-1&b=c-1`): the two step lists sit side by side. + +## Not covered here + +- Manifest `timeout_ms` is enforced by the gateway on the upstream call; it was checked by reading the code, not run. +- Manifest-level `approval:` roles, quorum and timeout, and the *Approver groups* field of the form builder, are not exercised by this guide. +- Delegation expiry in the console (its form takes whole hours); expiry was tested with the CLI. +- Policy `environment` rules, `default: deny`, `content:` checks, PII handling in capture and the capture time limit. +- OIDC login, notifications to Slack or email, `reversi apply`, the Helm chart, and the gateway. diff --git a/docs/testing/operations.mdx b/docs/testing/operations.mdx new file mode 100644 index 0000000..212f6c6 --- /dev/null +++ b/docs/testing/operations.mdx @@ -0,0 +1,1952 @@ +--- +title: Operations, security and integrations +description: Hand-test the console modes, API keys, OIDC, notifications, alerts, tracing, evidence, retention, egress rules and the three stores, with the output to expect. +--- + +This page tests what you need in order to run Reversi for other people: who may sign in, who gets told, what evidence you can +hand to an auditor, how long data is kept, where it may be sent, and where it lives. The first guide, [Manual testing](/testing), +covers the core product; do its section 5.1 and 5.2 first if you have never run the server. + +Run everything from the **repository root**, in `bash` or `zsh`, after `npm install && npm run build`. Scratch files go in +`try/ops/` (delete the folder to start over). Services use ports 7800 to 7849 on `127.0.0.1` only; change them if they are busy. + +<Note> +Every command and output below comes from a real run against this repository, in the order shown. IDs, timestamps and hashes +will differ on your machine. Slack, Teams, email, PagerDuty, Opsgenie and OpenTelemetry are exercised **only against fakes you +start on localhost**, never against the real services. The console is checked with `curl` (it is server-rendered HTML); open the +same URLs in a browser to see them drawn. See [Not covered here](#not-covered-here) for the full list. +</Note> + +A block that starts with `# Terminal A` runs in its own terminal, which you leave running; every other block goes in your +working terminal, called B here. Stop a terminal with Ctrl-C when the text says so. + +## 0. Setup + +You need Node 22.13 or newer, Python 3, `curl`, `openssl` and `shasum`. The `psql` client is optional (section 9). + +Manifests, a policy (an approval rule, secret detection and a `retention:` section) and one API key per role: + +```bash +mkdir -p try/ops/manifests +cat > try/ops/manifests/tools.yaml <<'EOF2' +- tool: lookup_customer + effect: read_only + permission: customers:read + +- tool: create_order + effect: reversible + permission: orders:write + compensation: + tool: cancel_order + argument_map: + order_id: $result.order_id + +- tool: cancel_order + effect: idempotent + permission: orders:write + +- tool: charge_card + effect: compensable + permission: payments:write + compensation: + tool: refund + argument_map: + charge_id: $result.charge_id + amount: $args.amount + audit: + evidence_export: required + +- tool: refund + effect: idempotent + permission: payments:write + +- tool: delete_customer + effect: irreversible + permission: customers:admin +EOF2 +cat > try/ops/policy.yaml <<'EOF2' +version: 1 +rules: + - id: big-charges + match: { tool: charge_card } + when: [{ path: $args.amount, op: gt, value: 100 }] + action: require_approval + reason: charges over 100 need a human +content: + secrets: deny +retention: + default_days: 30 +EOF2 +sha() { printf '%s' "$1" | shasum -a 256 | cut -d' ' -f1; } +AGENT=agent-$(openssl rand -hex 8); AUD=aud-$(openssl rand -hex 8); APPR=appr-$(openssl rand -hex 8) +OPS=ops-$(openssl rand -hex 8); ROOT=root-$(openssl rand -hex 8) +cat > try/ops/keys.json <<EOF2 +[ + { "name": "order-agent", "role": "agent", "key_sha256": "$(sha $AGENT)", "max_permissions": ["customers:read","orders:write","payments:write"] }, + { "name": "audrey", "role": "auditor", "key_sha256": "$(sha $AUD)" }, + { "name": "alice", "role": "approver", "key_sha256": "$(sha $APPR)" }, + { "name": "dana", "role": "operator", "key_sha256": "$(sha $OPS)" }, + { "name": "root", "role": "admin", "key_sha256": "$(sha $ROOT)" } +] +EOF2 +``` + +The keys, the environment every terminal shares, and a few helpers: `rv` is the CLI, `j` calls the API with a key, `post` calls a +console page and prints its status and text, `login` signs a key in to the console and keeps the cookie in `try/ops/jar-NAME`. +Each new terminal starts with `. try/ops/.env.sh`. + +```bash +cat > try/ops/.env.sh <<EOF2 +AGENT=$AGENT; AUD=$AUD; APPR=$APPR; OPS=$OPS; ROOT=$ROOT +export HOOK_SECRET=$(openssl rand -hex 16) +EOF2 +cat >> try/ops/.env.sh <<'EOF2' +export REVERSI_MANIFESTS=try/ops/manifests REVERSI_KEYS=try/ops/keys.json REVERSI_POLICY=try/ops/policy.yaml +export REVERSI_SIGNING_KEY=try/ops/signing.key REVERSI_DATA=try/ops/data +rv() { node packages/cli/dist/main.js "$@"; } +j() { curl -s -H "Authorization: Bearer $1" -H 'content-type: application/json' "${@:2}"; echo; } +msg() { sed -n 's/.*<main>\(.*\)<\/main>.*/\1/p' | sed -E 's#</(p|h1|h2|td|th|li|tr)># #g; s/<[^>]*>//g' | tr -s ' '; } +post() { curl -s -o try/ops/out.html -w "%{http_code} " "$@"; msg < try/ops/out.html; echo; } +login() { rm -f try/ops/jar-$1; curl -s -c try/ops/jar-$1 -o /dev/null -w "$1: login %{http_code}\n" --data-urlencode "key=$2" localhost:${3:-7800}/console/login; } +csrf() { curl -s -b try/ops/jar-$1 localhost:${2:-7800}/console/runs | grep -o 'name="csrf" value="[^"]*"' | head -1 | cut -d'"' -f4; } +kix() { curl -s -H "Authorization: Bearer $ROOT" localhost:7800/v1/keys | python3 -c "import sys,json;print([k['index'] for k in json.load(sys.stdin)['keys'] if k['name']=='$1'][0])"; } +pending() { curl -s -H "Authorization: Bearer $APPR" "localhost:7800/v1/approvals?status=pending" | python3 -c "import sys,json;print([a['id'] for a in json.load(sys.stdin)['approvals'] if a['run_id']=='$1'][0])"; } +EOF2 +. try/ops/.env.sh; B=localhost:7800 +``` + +```bash +rv validate $REVERSI_MANIFESTS +rv keygen try/ops/signing +``` + +Expect: + +```text +ok: 6 tool manifests valid +wrote try/ops/signing.key and try/ops/signing.pub (key id d67bfe0ef2ea4b10) +``` + +## 1. Console modes + +The console is **off by default**. Choose a mode with `REVERSI_CONSOLE` on the server, `--console` on the gateway, or +`console: { mode: … }` in `reversi.config.yaml`: + +| Mode | Login | Who can reach it | Role | +|---|---|---|---| +| `off` | none | nobody: `/console` answers 404 | none | +| `open` | none | requests to `localhost` from the same origin only, on a loopback listener | fixed: `operator` by default, or `auditor` or `approver`; never `admin` | +| `login` | paste an API key or an OIDC access token | anyone who can reach the port and holds a credential | the credential's roles | + +### 1.1 Off + +```bash +# Terminal A +. try/ops/.env.sh +PORT=7800 node packages/server/dist/main.js +``` + +Expect: `reversi-server listening on http://127.0.0.1:7800 (6 tools, files store, console off)`. + +```bash +curl -s -i $B/console | head -1 +curl -s $B/console; echo +j $OPS $B/v1/status +``` + +Expect: + +```text +HTTP/1.1 404 Not Found +The console is off. Start Reversi with REVERSI_CONSOLE=open (on your own machine, no login) or REVERSI_CONSOLE=login. +{"halted":null,"environment":"default","runs":0,"pending_approvals":0} +``` + +```bash +# Terminal A: press Ctrl-C +``` + +**What this proves:** with no setting the console does not exist, while the API works. + +### 1.2 Open: no login, this machine only, never admin + +```bash +# Terminal A +. try/ops/.env.sh +REVERSI_CONSOLE=open PORT=7800 node packages/server/dist/main.js +``` + +Expect `… (6 tools, files store, console open)`. Give the console something to show: a charge over 100 waits for approval. + +```bash +j $AGENT -X POST $B/v1/runs -d '{"run_id":"c-1","permissions":["payments:write"]}' +j $AGENT -X POST $B/v1/runs/c-1/intents -d '{"tool":"charge_card","args":{"amount":500},"reason":"pay"}' +curl -s $B/console/runs | grep -o '<title>[^<]*' +curl -s $B/console/runs | grep -o 'local-console[^<]*' +``` + +Expect: + +```text +{"run_id":"c-1","mode":"enforce","scope_seal":"3ef3832df527…"} +{"decision":"block","reasons":["approval required: request apr_… is pending an operator decision; retry after it is approved"]} +Runs · Reversi Console +local-console (operator) - open console, no login +``` + +Open [http://localhost:7800/console](http://localhost:7800/console) in a browser too: there is no login page, and the top bar says +"local-console (operator) - open console, no login". + +Now abuse it. Open mode answers only requests addressed to `localhost` and made from the same origin: + +```bash +post -H 'Host: evil.example.com' http://127.0.0.1:7800/console/runs +CSRF=$(curl -s $B/console/approvals | grep -o 'name="csrf" value="[^"]*"' | head -1 | cut -d'"' -f4) +post -X POST -H 'Origin: http://evil.example.com' -d "csrf=$CSRF&reason=x" $B/console/halt +``` + +Expect: + +```text +403 Not allowed The console in open mode only answers requests made from this machine to localhost. Use REVERSI_CONSOLE=login to serve it to other people. +403 Not allowed The console in open mode only answers requests made from this machine to localhost. Use REVERSI_CONSOLE=login to serve it to other people. +``` + +The open console is never admin, so the kill switch and publishing are refused even from the right origin with a valid CSRF +token. (The token sits on any page with a form; the approvals page has them.) + +```bash +post -X POST -H "Origin: http://$B" -d "csrf=$CSRF&reason=x" $B/console/halt +post -X POST -H "Origin: http://$B" -d "csrf=$CSRF" $B/console/manifests/charge_card/approve +post -X POST -H "Origin: http://$B" -d "reason=x" $B/console/halt +``` + +Expect: + +```text +403 Forbidden Your role does not allow this. +403 Forbidden Your role does not allow this. +403 Forbidden Invalid or missing CSRF token. Go back, reload the page and try again. +``` + +What it can do is what its role allows. As operator it can approve, and the ledger names it `local-console`: + +```bash +ID=$(pending c-1) +post -X POST -H "Origin: http://$B" -d "csrf=$CSRF¬e=looks+fine" $B/console/approvals/$ID/approve +j $APPR "$B/v1/approvals?status=approved" | python3 -c 'import sys,json;a=json.load(sys.stdin)["approvals"][0];print(a["status"],a["decided_by"])' +curl -s -o /dev/null -w "the API without a key: %{http_code}\n" $B/v1/status +``` + +Expect: + +```text +303 +approved local-console +the API without a key: 401 +``` + +```bash +# Terminal A: press Ctrl-C +``` + +The server also **refuses to start** open on a non-loopback address, or open as admin: + +```bash +HOST=0.0.0.0 REVERSI_CONSOLE=open PORT=7801 node packages/server/dist/main.js; echo "exit $?" +REVERSI_CONSOLE=open REVERSI_CONSOLE_ROLE=admin PORT=7801 node packages/server/dist/main.js; echo "exit $?" +``` + +Expect: + +```text +error: the open console has no login, so it only runs on a loopback HOST (127.0.0.1, ::1, localhost), not 0.0.0.0. Use REVERSI_CONSOLE=login. +exit 64 +error: the open console never runs as admin +exit 64 +``` + +`reversi console` is the shortcut for your own laptop: it opens the engine on your data directory and serves the open console +(default port 7421, here 7802). Its `--role` is `auditor`, `approver` or `operator`. + +```bash +# Terminal A +. try/ops/.env.sh +rv console --port 7802 --role auditor +``` + +```bash +sleep 1 +curl -s localhost:7802/console/runs | grep -o 'local-console[^<]*' +post -H 'Host: evil.example.com' http://127.0.0.1:7802/console/runs | cut -c1-48 +``` + +Expect: + +```text +local-console (auditor) - open console, no login +403 Not allowed The console in open mode only an +``` + +Expect the terminal to print `console: http://localhost:7802/console (open mode as auditor; Ctrl-C to stop)`. + +```bash +# Terminal A: press Ctrl-C +``` + +**What this proves:** open mode needs no credential but refuses any request that is not plainly your own browser talking to +`localhost`; it can never become admin; and it cannot be started where other people could reach it. + +### 1.3 Login: paste a key, roles apply, a disabled key ends the session + +```bash +# Terminal A +. try/ops/.env.sh +REVERSI_CONSOLE=login REVERSI_SESSION_SECRET=$(openssl rand -hex 16) PORT=7800 node packages/server/dist/main.js 2>&1 | tee try/ops/a.log +``` + +Expect `… console login)`. In a browser, open [http://localhost:7800/console](http://localhost:7800/console): it sends you to a sign-in form. +Here the same with `curl`. Wrong keys and agent keys are refused; then sign in as the auditor, approver and admin: + +```bash +j $AGENT -X POST $B/v1/runs -d '{"run_id":"l-1","permissions":["payments:write"]}' | cut -c1-40 +j $AGENT -X POST $B/v1/runs/l-1/intents -d '{"tool":"charge_card","args":{"amount":500},"reason":"pay"}' | cut -c1-60 +curl -s -o /dev/null -w "no session: %{http_code} -> %{redirect_url}\n" $B/console/runs +post -d "key=nope" $B/console/login +post -d "key=$AGENT" $B/console/login +login aud $AUD; login appr $APPR; login root $ROOT +``` + +Expect: + +```text +{"run_id":"l-1","mode":"enforce","scope_ +{"decision":"block","reasons":["approval required: request a +no session: 302 -> http://localhost:7800/console/login +401 Sign in That credential was not accepted, or it has no console role. Try again +401 Sign in That credential was not accepted, or it has no console role. Try again +aud: login 303 +appr: login 303 +root: login 303 +``` + +Roles differ. The auditor has no approve button and no kill-switch form; the approver can approve; only the admin has the kill switch: + +```bash +for u in aud appr root; do + a=$(curl -s -b try/ops/jar-$u $B/console/approvals | grep -q '/approve"' && echo yes || echo no) + h=$(curl -s -b try/ops/jar-$u $B/console/evidence | grep -q 'action="/console/halt"' && echo yes || echo no) + echo "$u: approve buttons=$a kill-switch form=$h" +done +ID=$(pending l-1) +post -b try/ops/jar-aud -d "csrf=$(csrf aud)" $B/console/approvals/$ID/approve +curl -s -b try/ops/jar-root $B/console/runs | grep -o 'root ([a-z]*)' +``` + +Expect: + +```text +aud: approve buttons=no kill-switch form=no +appr: approve buttons=yes kill-switch form=no +root: approve buttons=yes kill-switch form=yes +403 Forbidden Your role does not allow this. +root (admin) +``` + +The cookie is `HttpOnly; SameSite=Strict`, and it follows the key. Disable the auditor's key and the session ends at once: + +```bash +curl -s -D - -o /dev/null -d "key=$AUD" $B/console/login | grep -i set-cookie | sed 's/rv_session=[^;]*/rv_session=…/' +curl -s -b try/ops/jar-aud -o /dev/null -w "auditor session: %{http_code}\n" $B/console/runs +j $ROOT -X PATCH $B/v1/keys/$(kix audrey) -d '{"disabled":true}' +curl -s -b try/ops/jar-aud -o /dev/null -w "after disabling the key: %{http_code} -> %{redirect_url}\n" $B/console/runs +j $AUD $B/v1/status +j $ROOT -X PATCH $B/v1/keys/$(kix audrey) -d '{"disabled":false}' +curl -s -b try/ops/jar-aud -o /dev/null -w "after enabling it again: %{http_code}\n" $B/console/runs +``` + +Expect: + +```text +set-cookie: rv_session=…; Path=/console; HttpOnly; SameSite=Strict; Max-Age=28800 +auditor session: 200 +{"ok":true} +after disabling the key: 302 -> http://localhost:7800/console/login +{"error":"missing or invalid API key"} +{"ok":true} +after enabling it again: 200 +``` + +Leave Terminal A running for section 2. + +**What this proves:** login mode needs a credential with a console role (agent keys are refused), pages and buttons follow the role, +a forged POST from a lower role is refused even with a valid CSRF token, and disabling a key ends its sessions without a restart. + +### 1.4 The same switch on the gateway + +The gateway serves the console on its own port, so it needs a `listen:` section. Build its config and manifests: + +```bash +mkdir -p try/ops/sandbox try/ops/gw-manifests +node packages/connectors/dist/main.js manifests fs --format yaml > try/ops/gw-manifests/fs.yaml +cat > try/ops/gateway.yaml <<'EOF2' +mode: enforce +upstreams: + - name: fs + command: node + args: ["packages/connectors/dist/main.js", "serve", "fs", "--root", "try/ops/sandbox"] + namespace: false +listen: { host: 127.0.0.1, port: 7806 } +auth: { keys: try/ops/keys.json } +EOF2 +``` + +```bash +# Terminal C +. try/ops/.env.sh +node packages/gateway/dist/main.js --config try/ops/gateway.yaml --manifests try/ops/gw-manifests --data try/ops/gw-data +``` + +```bash +sleep 2 +curl -s -w " [%{http_code}]\n" localhost:7806/console/runs +``` + +Expect: + +```text +{"error":"not found"} [404] +``` + +```bash +# Terminal C: press Ctrl-C +``` + +Restart it with `--console open` and with `--console login`: + +```bash +# Terminal C +. try/ops/.env.sh +node packages/gateway/dist/main.js --config try/ops/gateway.yaml --manifests try/ops/gw-manifests --data try/ops/gw-data --console open +``` + +```bash +sleep 2 +curl -s localhost:7806/console/runs | grep -o 'local-console[^<]*' +curl -s -o /dev/null -w "foreign Host header: %{http_code}\n" -H 'Host: evil.example.com' http://127.0.0.1:7806/console/runs +curl -s -o /dev/null -w "the MCP endpoint still needs a key: %{http_code}\n" -X POST localhost:7806/mcp +``` + +Expect: + +```text +local-console (operator) - open console, no login +foreign Host header: 403 +the MCP endpoint still needs a key: 401 +``` + +```bash +# Terminal C: press Ctrl-C +``` + +```bash +# Terminal C +. try/ops/.env.sh +node packages/gateway/dist/main.js --config try/ops/gateway.yaml --manifests try/ops/gw-manifests --data try/ops/gw-data --console login +``` + +```bash +sleep 2 +curl -s -o /dev/null -w "no session: %{http_code} -> %{redirect_url}\n" localhost:7806/console/runs +login gw $AUD 7806 +curl -s -b try/ops/jar-gw localhost:7806/console/runs | grep -o 'audrey ([a-z]*)' +``` + +Expect: + +```text +no session: 302 -> http://localhost:7806/console/login +gw: login 303 +audrey (auditor) +``` + +```bash +# Terminal C: press Ctrl-C +``` + +The gateway also refuses an unsafe open console at start: + +```bash +sed 's/host: 127.0.0.1/host: 0.0.0.0/' try/ops/gateway.yaml > try/ops/gateway-lan.yaml +node packages/gateway/dist/main.js --config try/ops/gateway-lan.yaml --manifests try/ops/gw-manifests --data try/ops/gw-data --console open 2>&1 | grep error +node packages/gateway/dist/main.js --config try/ops/gateway.yaml --manifests try/ops/gw-manifests --data try/ops/gw-data --console open --console-role admin 2>&1 | grep error +``` + +Expect: + +```text +error: the open console has no login, so the gateway must listen on loopback, not 0.0.0.0. Use --console login. +error: the open console never runs as admin +``` + +## 2. API keys: file format, console, API, rotation, expiry + +A key file is a JSON list. Each entry has a `name` (it becomes the actor in the ledger), `role` or `roles`, and `key_sha256` (the hex +SHA-256 of the key; the key itself is never stored). Optional: `max_permissions` (agents), `groups`, `tenant`, `not_before`, +`expires_at`, `disabled`, `allow_observe`. The server re-reads the file on `SIGHUP` or when it changes (checked every two seconds). +Terminal A from section 1.3 is still running. + +```bash +j $ROOT $B/v1/keys | python3 -c 'import sys,json;[print(k["index"],k["name"],k["roles"],k["fingerprint"]) for k in json.load(sys.stdin)["keys"]]' +j $AUD $B/v1/keys -w "[%{http_code}]\n" +``` + +Expect: + +```text +0 order-agent ['agent'] edba5bee3cb7 +1 audrey ['auditor'] 5e36ca533901 +2 alice ['approver'] 52513b4ee156 +3 dana ['operator'] 568a2f681422 +4 root ['admin'] 9a8e43fe03e4 +{"error":"admin role required"}[403] +``` + +### 2.1 Create, use, expire and remove a key over the API + +`POST /v1/keys` is admin only. The new key is shown **once**: + +```bash +j $ROOT -X POST $B/v1/keys -d '{"name":"temp-auditor","role":"auditor","expires_in_days":1}' > try/ops/newkey.json +sed 's/"key":"[^"]*"/"key":"rv_…"/' try/ops/newkey.json +TEMP=$(python3 -c 'import json;print(json.load(open("try/ops/newkey.json"))["key"])') +curl -s -o /dev/null -w "the new key: %{http_code}\n" -H "Authorization: Bearer $TEMP" $B/v1/status +j $ROOT $B/v1/keys | python3 -c 'import sys,json;[print(k["index"],k["name"],k.get("expires_at")) for k in json.load(sys.stdin)["keys"] if k["name"]=="temp-auditor"]' +j $ROOT -X PATCH $B/v1/keys/$(kix temp-auditor) -d '{"expires_at":"2000-01-01T00:00:00Z"}' +curl -s -w " [%{http_code}] expired\n" -H "Authorization: Bearer $TEMP" $B/v1/status +j $ROOT -X PATCH $B/v1/keys/$(kix temp-auditor) -d '{"expires_at":null}' +curl -s -o /dev/null -w "expiry cleared: %{http_code}\n" -H "Authorization: Bearer $TEMP" $B/v1/status +j $ROOT -X DELETE $B/v1/keys/$(kix temp-auditor) +curl -s -w " [%{http_code}] removed\n" -H "Authorization: Bearer $TEMP" $B/v1/status +``` + +Expect: + +```text +{"name":"temp-auditor","roles":["auditor"],"key":"rv_…","note":"this is the only time the key is shown"} +the new key: 200 +5 temp-auditor 2026-10-17T18:12:26.944Z +{"ok":true} +{"error":"missing or invalid API key"} [401] expired +{"ok":true} +expiry cleared: 200 +{"ok":true} +{"error":"missing or invalid API key"} [401] removed +``` + +### 2.2 The same in the console Settings page + +Sign in as the admin and create, disable and remove a key from **Settings**. The new key is shown once, on the page after the form: + +```bash +login root $ROOT +CS=$(csrf root) +curl -s -b try/ops/jar-root -d "csrf=$CS&name=console-auditor&role=auditor&days=7" $B/console/settings/keys > try/ops/ck.html +msg < try/ops/ck.html | cut -c1-70 +CONSOLEKEY=$(grep -o '
[^<]*
' try/ops/ck.html | sed 's/<[^>]*>//g') +curl -s -o /dev/null -w "the new key: %{http_code}\n" -H "Authorization: Bearer $CONSOLEKEY" $B/v1/status +curl -s -b try/ops/jar-root $B/console/settings | grep -o 'console-auditor.\{0,110\}' | sed 's/<[^>]*>/ /g' | tr -s ' ' | cut -c1-62 +post -b try/ops/jar-root -d "csrf=$CS" $B/console/settings/keys/$(kix console-auditor)/disable +curl -s -w " [%{http_code}] disabled\n" -H "Authorization: Bearer $CONSOLEKEY" $B/v1/status +post -b try/ops/jar-root -d "csrf=$CS" $B/console/settings/keys/$(kix console-auditor)/remove +python3 -c 'import json;print([k["name"] for k in json.load(open("try/ops/keys.json"))])' +``` + +Expect: + +```text +root: login 303 +Key created Copy this key now. It is not stored and cannot be shown ag +the new key: 200 +console-auditor auditor 2026-10-23T18:12:27.225Z e58a04603d90 +303 +{"error":"missing or invalid API key"} [401] disabled +303 +['order-agent', 'audrey', 'alice', 'dana', 'root'] +``` + +### 2.3 Rotation and expiry by editing the file + +Rotate "carol" without downtime: add her new key beside the old one, give the old entry an `expires_at`, and the server follows the file. +A file that does not parse is ignored and the old keys stay in force. + +```bash +CAROL_OLD=carol-$(openssl rand -hex 8); CAROL_NEW=carol-$(openssl rand -hex 8) +cat > try/ops/keyedit.py <<'EOF2' +import json, sys, hashlib +f = "try/ops/keys.json" +ks = json.load(open(f)) +mode, key = sys.argv[1], sys.argv[2] if len(sys.argv) > 2 else "" +if mode == "add": + ks.append({"name": "carol", "role": "operator", "key_sha256": hashlib.sha256(key.encode()).hexdigest()}) +elif mode == "expire-old": + for k in ks: + if k["name"] == "carol": + k["expires_at"] = "2000-01-01T00:00:00Z" +elif mode == "drop": + ks = [k for k in ks if k["name"] != "carol"] +json.dump(ks, open(f, "w"), indent=2) +EOF2 +python3 try/ops/keyedit.py add $CAROL_OLD; sleep 3 +curl -s -o /dev/null -w "old key: %{http_code}\n" -H "Authorization: Bearer $CAROL_OLD" $B/v1/status +python3 try/ops/keyedit.py expire-old; python3 try/ops/keyedit.py add $CAROL_NEW; sleep 3 +curl -s -o /dev/null -w "old key after rotation: %{http_code}\n" -H "Authorization: Bearer $CAROL_OLD" $B/v1/status +curl -s -o /dev/null -w "new key: %{http_code}\n" -H "Authorization: Bearer $CAROL_NEW" $B/v1/status +cp try/ops/keys.json try/ops/keys.good; echo '{ broken' > try/ops/keys.json; sleep 3 +curl -s -o /dev/null -w "new key while the file is broken: %{http_code}\n" -H "Authorization: Bearer $CAROL_NEW" $B/v1/status +cp try/ops/keys.good try/ops/keys.json; python3 try/ops/keyedit.py drop; sleep 3 +curl -s -w " [%{http_code}] after dropping carol\n" -H "Authorization: Bearer $CAROL_NEW" $B/v1/status +grep -a "API keys\|key reload" try/ops/a.log | sed 's/: Expected.*//' | tail -6 +``` + +Expect: + +```text +old key: 200 +old key after rotation: 401 +new key: 200 +new key while the file is broken: 200 +{"error":"missing or invalid API key"} [401] after dropping carol +reversi-server: reloaded 5 API keys +reversi-server: reloaded 6 API keys +reversi-server: reloaded 7 API keys +reversi-server: key reload failed, keeping the old keys +reversi-server: reloaded 5 API keys +``` + +```bash +# Terminal A: press Ctrl-C +``` + +**What this proves:** keys can be created, expired, disabled and removed from the API, the console or the file; only admins can; +expired and disabled keys stop working without a restart; a broken key file never locks everyone out or in. + +## 3. OIDC with the mock identity provider + +`examples/oidc/mock-idp.mjs` is a throwaway OpenID provider that mints signed tokens. It is for trying this out, never for production. +It listens on `PORT` (default 9000; here 7810). + +```bash +# Terminal C +. try/ops/.env.sh +PORT=7810 node examples/oidc/mock-idp.mjs +``` + +```bash +# Terminal A +. try/ops/.env.sh +REVERSI_CONSOLE=login REVERSI_SESSION_SECRET=$(openssl rand -hex 16) PORT=7800 \ +REVERSI_OIDC_ISSUER=http://127.0.0.1:7810 REVERSI_OIDC_AUDIENCE=reversi \ +REVERSI_OIDC_ROLE_MAP='{"reversi-ops":"operator","reversi-agents":"agent"}' node packages/server/dist/main.js +``` + +The bearer token works on the API. The audience, expiry and role map are checked; a token with no mapped role is refused: + +```bash +sleep 1 +I=http://127.0.0.1:7810 +OPSTOK=$(curl -s "$I/token?sub=dana&roles=reversi-ops"); AGTOK=$(curl -s "$I/token?sub=bot-1&roles=reversi-agents") +curl -s -H "Authorization: Bearer $OPSTOK" $B/v1/status; echo +curl -s -H "Authorization: Bearer $AGTOK" $B/v1/status; echo +curl -s -w " [%{http_code}] wrong audience\n" -H "Authorization: Bearer $(curl -s "$I/token?sub=x&roles=reversi-ops&aud=other")" $B/v1/status +curl -s -w " [%{http_code}] expired\n" -H "Authorization: Bearer $(curl -s "$I/token?sub=x&roles=reversi-ops&ttl=-100")" $B/v1/status +curl -s -w " [%{http_code}] no mapped role\n" -H "Authorization: Bearer $(curl -s "$I/token?sub=x&roles=something-else")" $B/v1/status +curl -s -H "Authorization: Bearer $AGTOK" -H 'content-type: application/json' -X POST $B/v1/runs -d '{"run_id":"oidc-1","permissions":["customers:read"]}' | cut -c1-40; echo +rv runs list | grep oidc +j $AUD $B/v1/status +``` + +Expect: + +```text +{"halted":null,"environment":"default","runs":2,"pending_approvals":1} +{"error":"auditor role required"} +{"error":"missing or invalid API key"} [401] wrong audience +{"error":"missing or invalid API key"} [401] expired +{"error":"missing or invalid API key"} [401] no mapped role +{"run_id":"oidc-1","mode":"enforce","sco + +oidc-1 actor=bot-1 mode=enforce steps=0 empty +{"halted":null,"environment":"default","runs":3,"pending_approvals":1} +``` + +API keys keep working next to OIDC (the last line is the auditor's key). Now paste a token into the console login. The session never outlives the token: + +```bash +curl -s -c try/ops/jar-oidc -o /dev/null -w "console login with a token: %{http_code}\n" --data-urlencode "key=$OPSTOK" $B/console/login +curl -s -b try/ops/jar-oidc $B/console/runs | grep -o 'dana ([a-z]*)' +curl -s -c try/ops/jar-short -o /dev/null --data-urlencode "key=$(curl -s "$I/token?sub=dana&roles=reversi-ops&ttl=5")" $B/console/login +curl -s -b try/ops/jar-short -o /dev/null -w "short token, at once: %{http_code}\n" $B/console/runs +sleep 6 +curl -s -b try/ops/jar-short -o /dev/null -w "after the token expired: %{http_code} -> %{redirect_url}\n" $B/console/runs +curl -s -o /dev/null -w "an agent token in the console: %{http_code}\n" --data-urlencode "key=$AGTOK" $B/console/login +``` + +Expect: + +```text +console login with a token: 303 +dana (operator) +short token, at once: 200 +after the token expired: 302 -> http://localhost:7800/console/login +an agent token in the console: 401 +``` + +```bash +# Terminals A and C: press Ctrl-C +``` + +**What this proves:** OIDC bearer tokens authenticate the API and the console login, roles come from the claim map, the principal in the +ledger is the token's `sub` (`bot-1`), and a console session ends when its token does. Redirect (browser SSO) login is not built. + +## 4. Notifications + +Notifications are configured under `notifications:` in `reversi.config.yaml` (see [Notifications](/operations/notifications)). Three fakes +stand in for the services, all on localhost. First the scripts. The fake HTTP endpoint prints one line per POST and checks the webhook +signature; the fake SMTP server accepts a message and prints it decoded: + +```bash +cat > try/ops/fake.mjs <<'EOF2' +// Prints every POST it receives as one JSON line. Usage: node fake.mjs [signing-secret] +import { createServer } from "node:http"; +import { createHmac } from "node:crypto"; +import { appendFileSync, writeFileSync } from "node:fs"; +const [port, secret] = [Number(process.argv[2]), process.argv[3]]; +createServer((req, res) => { + let body = ""; + req.on("data", (c) => (body += c)); + req.on("end", () => { + const want = secret && "sha256=" + createHmac("sha256", secret).update(body).digest("hex"); + const sig = req.headers["x-reversi-signature"]; + if (sig) { writeFileSync("try/ops/last-webhook.body", body); writeFileSync("try/ops/last-webhook.sig", sig); } + let spans; if (req.url.endsWith("/v1/traces")) { appendFileSync("try/ops/traces.ndjson", body + "\n"); spans = JSON.parse(body).resourceSpans.flatMap((r) => r.scopeSpans.flatMap((s) => s.spans)).map((s) => `${s.name} status=${s.status.code}`); } + console.log(JSON.stringify({ path: req.url, spans, event: req.headers["x-reversi-event"], signature: sig ? (sig === want ? "valid" : "INVALID") : undefined, body: spans ? undefined : body.slice(0, 700) })); + res.writeHead(200).end("{}"); + }); +}).listen(port, "127.0.0.1", () => console.error("fake endpoint on " + port)); +EOF2 +cat > try/ops/smtp-sink.mjs <<'EOF2' +// A fake SMTP server: accepts messages and prints them. Usage: node smtp-sink.mjs +import { createServer } from "node:net"; +createServer((sock) => { + let inData = false, buf = "", text = ""; + const say = (s) => sock.write(s + "\r\n"); + say("220 fake-smtp ready"); + sock.on("data", (d) => { + buf += d.toString(); + let i; + while ((i = buf.indexOf("\r\n")) >= 0) { + const line = buf.slice(0, i); buf = buf.slice(i + 2); + if (inData) { + if (line === ".") { inData = false; const [head, body] = text.split("\n\n"); console.log(head.split("\n").filter((l) => /^(To|Subject):/.test(l)).join(" | ") + "\n" + Buffer.from((body ?? "").replace(/\s/g, ""), "base64").toString()); say("250 queued"); } else text += line + "\n"; + } else if (/^EHLO/i.test(line)) say("250 fake-smtp"); + else if (/^DATA/i.test(line)) { inData = true; text = ""; say("354 go ahead"); } + else if (/^QUIT/i.test(line)) { say("221 bye"); sock.end(); } + else say("250 ok"); + } + }); +}).listen(Number(process.argv[2]), "127.0.0.1", () => console.error("fake smtp on " + process.argv[2])); +EOF2 +cat > try/ops/notify.yaml <<'EOF2' +manifests: ./manifests +policy: ./policy.yaml +data: { dir: ./data } +approval: { console_url: "http://127.0.0.1:7800" } +log: { format: json, level: info } +notifications: + channels: + - { type: webhook, name: hook, url: "http://127.0.0.1:7811/webhook", secret: "${HOOK_SECRET}" } + - { type: slack, name: slack, url: "http://127.0.0.1:7811/slack" } + - { type: teams, name: teams, url: "http://127.0.0.1:7811/teams" } + - { type: pagerduty, name: pd, routing_key: fake-routing-key, url: "http://127.0.0.1:7811/pagerduty" } + - { type: opsgenie, name: og, api_key: fake-api-key, url: "http://127.0.0.1:7811/opsgenie" } + - { type: email, name: mail, host: 127.0.0.1, port: 7813, starttls: false, from: reversi@example.test, to: [ops@example.test] } + routes: + - { events: ["*"], channels: [hook, slack, teams, pd, og, mail] } +EOF2 +``` + +Start the fakes and the server. The example webhook sink (`examples/webhook-sink.mjs`) takes the older single approval webhook, `REVERSI_APPROVAL_WEBHOOK`, +which works beside the channels: + +```bash +# Terminal C +. try/ops/.env.sh +node try/ops/fake.mjs 7811 $HOOK_SECRET 2>&1 | tee try/ops/fake.log +``` + +```bash +# Terminal D +. try/ops/.env.sh +node try/ops/smtp-sink.mjs 7813 2>&1 | tee try/ops/smtp.log +``` + +```bash +# Terminal E +. try/ops/.env.sh +PORT=7814 node examples/webhook-sink.mjs 2>&1 | tee try/ops/sink.log +``` + +```bash +# Terminal A +. try/ops/.env.sh +REVERSI_CONFIG=try/ops/notify.yaml REVERSI_APPROVAL_WEBHOOK=http://127.0.0.1:7814/hook \ +REVERSI_CONSOLE=login REVERSI_SESSION_SECRET=$(openssl rand -hex 16) PORT=7800 node packages/server/dist/main.js 2>&1 | tee try/ops/a.log +``` + +### 4.1 A test message from the CLI and from the console + +`reversi doctor --send-test` sends one message through every channel in the config, and checks each URL against the egress rules. + +```bash +rv --config try/ops/notify.yaml doctor --send-test | grep "test message" +sleep 1 +tail -n 5 try/ops/fake.log | cut -c1-150 +tail -n 3 try/ops/smtp.log +``` + +Expect: + +```text +ok test message: sent to hook, slack, teams, pd, og, mail +{"path":"/webhook","event":"doctor_test","signature":"valid","body":"{\"event\":\"doctor_test\",\"ts\":\"2026-10-16T18:13:04.622Z\",\"severity\":\"inf +{"path":"/slack","body":"{\"text\":\"Reversi test notification\\nIf you can read this, the channel works.\",\"blocks\":[{\"type\":\"header\",\"text\": +{"path":"/teams","body":"{\"@type\":\"MessageCard\",\"@context\":\"https://schema.org/extensions\",\"summary\":\"Reversi test notification\",\"themeCo +{"path":"/pagerduty","body":"{\"routing_key\":\"fake-routing-key\",\"event_action\":\"trigger\",\"dedup_key\":\"doctor_test:Reversi test notification\ +{"path":"/opsgenie/v2/alerts","body":"{\"message\":\"Reversi test notification\",\"alias\":\"doctor_test:Reversi test notification\",\"description\":\ +To: ops@example.test | Subject: [Reversi info] Reversi test notification +Reversi test notification +If you can read this, the channel works. +``` + +In the console, **Settings** lists the channels and, for an admin, has a **Send a test notification** button: + +```bash +login root $ROOT +curl -s -b try/ops/jar-root $B/console/settings | grep -o 'Notification channels.\{0,120\}' | sed 's/<[^>]*>/ /g' | tr -s ' ' | cut -c1-80 +post -b try/ops/jar-root -d "csrf=$(csrf root)" $B/console/settings/notify-test +sleep 1 +tail -n 5 try/ops/fake.log | cut -c1-110 +tail -n 3 try/ops/smtp.log +``` + +Expect: + +```text +root: login 303 +Notification channels hook , slack , teams , pd , og , mail `. The fake endpoint saved the last +signed delivery. The fake already printed `signature: valid`; check it yourself with the Python helper in the SDK: + +```bash +cat try/ops/last-webhook.sig | cut -c1-30 +PYTHONPATH=packages/sdk-py python3 - <<'EOF2' +import os +from reversi_sdk import verify_webhook_signature +body = open("try/ops/last-webhook.body", "rb").read() +sig = open("try/ops/last-webhook.sig").read() +secret = os.environ["HOOK_SECRET"] +print("genuine delivery:", verify_webhook_signature(secret, body, sig)) +print("body changed: ", verify_webhook_signature(secret, body.replace(b"approval_requested", b"approval_decided"), sig)) +print("wrong secret: ", verify_webhook_signature("not-the-secret", body, sig)) +print("no header: ", verify_webhook_signature(secret, body, None)) +EOF2 +``` + +Expect: + +```text +sha256=a16e5174b5118aca43e32ee +genuine delivery: True +body changed: False +wrong secret: False +no header: False +``` + +### 4.4 A dead channel never changes a decision + +Stop the three fakes (Ctrl-C in terminals C, D and E), then ask for another approval. Because `notify.yaml` has `log: { format: json }`, the +server writes its warnings as JSON lines on stderr (the logger redacts credential-looking fields and values; section 5.5 shows it): + +```bash +# Terminals C, D and E: press Ctrl-C +``` + +```bash +j $AGENT -X POST $B/v1/runs/n-1/intents -d '{"tool":"charge_card","args":{"amount":752},"reason":"pay"}' | cut -c1-70 +sleep 4 +grep -a '"level":"warn"' try/ops/a.log | cut -c1-190 +``` + +Expect: + +```text +{"decision":"block","reasons":["approval required: request apr_3d52766 +{"level":"warn","time":"2026-10-16T18:13:10.072Z","msg":"approval webhook failed: fetch failed"} +{"level":"warn","time":"2026-10-16T18:13:10.688Z","msg":"notification to mail failed: email mail: connect ECONNREFUSED 127.0.0.1:7813"} +{"level":"warn","time":"2026-10-16T18:13:10.688Z","msg":"notification to hook failed: webhook hook: fetch failed"} +{"level":"warn","time":"2026-10-16T18:13:10.688Z","msg":"notification to slack failed: slack slack: fetch failed"} +{"level":"warn","time":"2026-10-16T18:13:10.689Z","msg":"notification to teams failed: teams teams: fetch failed"} +{"level":"warn","time":"2026-10-16T18:13:10.689Z","msg":"notification to pd failed: pagerduty pd: fetch failed"} +{"level":"warn","time":"2026-10-16T18:13:10.689Z","msg":"notification to og failed: opsgenie og: fetch failed"} +``` + +```bash +# Terminal A: press Ctrl-C +``` + +**What this proves:** the six channel types send the documented payloads (Slack blocks, a Teams card, a PagerDuty event, an Opsgenie +alert, an SMTP message, a signed webhook); the signature verifies and breaks if the body or secret changes; messages carry names, never values; and +a channel that is down is a logged warning, not a failed or allowed call. + + +Slack, Teams, SMTP, PagerDuty and Opsgenie were tested only against these local fakes, which check that a request arrives and what it says. They +were not tested against the real services, so real-world details (rate limits, TLS and AUTH on a real mail relay, Slack's response format, +PagerDuty routing keys) are untested. + + +## 5. Alerts, tracing, logs and metrics + +One config turns on alerts every two seconds with a denial threshold of 3, sends only `alert` events to the webhook, and exports +traces to an OTLP endpoint. It uses its own data directory so the numbers are easy to read. + +```bash +cat > try/ops/alerts.yaml <<'EOF2' +manifests: ./manifests +policy: ./policy.yaml +data: { dir: ./data-alerts } +approval: { console_url: "http://127.0.0.1:7800" } +notifications: + channels: + - { type: webhook, name: hook, url: "http://127.0.0.1:7811/webhook", secret: "${HOOK_SECRET}" } + routes: + - { events: ["alert"], channels: [hook] } +otel: { endpoint: "http://127.0.0.1:7812", service_name: reversi-try } +alerts: { interval: 2s, denial_spike: 3 } +EOF2 +``` + +```bash +# Terminal C +. try/ops/.env.sh +node try/ops/fake.mjs 7811 $HOOK_SECRET 2>&1 | tee try/ops/fake.log +``` + +```bash +# Terminal D +. try/ops/.env.sh +node try/ops/fake.mjs 7812 2>&1 | tee try/ops/otlp.log +``` + +```bash +# Terminal A +. try/ops/.env.sh +REVERSI_LOG=json REVERSI_CONFIG=try/ops/alerts.yaml REVERSI_DATA=try/ops/data-alerts \ +REVERSI_CONSOLE=login REVERSI_SESSION_SECRET=$(openssl rand -hex 16) PORT=7800 node packages/server/dist/main.js 2>&1 | tee try/ops/a.log +``` + +### 5.1 A denial spike and an unclassified tool + +```bash +j $AGENT -X POST $B/v1/runs -d '{"run_id":"al-1","permissions":["customers:read","orders:write","payments:write"]}' | cut -c1-40 +j $AGENT -X POST $B/v1/runs/al-1/intents -d '{"tool":"create_order","args":{"sku":"B"},"reason":"order"}' +j $AGENT -X POST $B/v1/runs/al-1/steps/0/outcome -d '{"status":"succeeded","result":{"order_id":"o1"}}' +for i in 1 2 3; do j $AGENT -X POST $B/v1/runs/al-1/intents -d '{"tool":"delete_customer","args":{"id":"c1"},"reason":"probing"}' | cut -c1-100; done +j $AGENT -X POST $B/v1/runs/al-1/intents -d '{"tool":"mystery_tool","args":{},"reason":"x"}' | cut -c1-100 +sleep 4 +tail -n 2 try/ops/fake.log | tr -d '\000' | cut -c1-250 +``` + +Expect: + +```text +{"run_id":"al-1","mode":"enforce","scope +{"decision":"allow","step":0} +{"step":0,"status":"succeeded"} +{"decision":"block","reasons":["permission `customers:admin` is not in the run's scope"]} +{"decision":"block","reasons":["permission `customers:admin` is not in the run's scope"]} +{"decision":"block","reasons":["permission `customers:admin` is not in the run's scope"]} +{"decision":"block","reasons":["no manifest registered for tool `mystery_tool`"]} +{"path":"/webhook","event":"alert","signature":"valid","body":"{\"event\":\"alert\",\"ts\":\"2026-10-16T18:13:24.288Z\",\"severity\":\"warning\",\"title\":\"4 calls blocked in the last 15... +{"path":"/webhook","event":"alert","signature":"valid","body":"{\"event\":\"alert\",\"ts\":\"2026-10-16T18:13:24.318Z\",\"severity\":\"warning\",\"title\":\"Unclassified tool called: myst... +``` + +The threshold of 3 fired a notification ("4 calls blocked"). The live views, `GET /v1/alerts` and the console **Alerts** page, use the +same `alerts.denial_spike`, so they list the spike too: + +```bash +j $AUD $B/v1/alerts | python3 -c 'import sys,json;[print(a["severity"],a["title"]) for a in json.load(sys.stdin)["alerts"]]' +login aud $AUD +curl -s -b try/ops/jar-aud $B/console/alerts | msg | grep -o 'warning.*' | cut -c1-110 +``` + +Expect: + +```text +warning 4 calls blocked in the last 15 minutes +warning Unclassified tool called: mystery_tool +aud: login 303 +warning 4 calls blocked in the last 15 minutes An agent may be looping or probing. Check the blocked reasons i +``` + +### 5.2 An alert that resolves + +Alerts are sent once, repeated as reminders (`alerts.remind`, default one hour) and resolved when the condition clears. A kill switch is the simplest condition to clear: + +```bash +j $ROOT -X POST $B/v1/halt -d '{"reason":"drill"}' | cut -c1-60 +sleep 3 +j $ROOT -X POST $B/v1/resume +sleep 3 +tail -n 2 try/ops/fake.log | tr -d '\000' | python3 -c ' +import sys, json +for l in sys.stdin: + d = json.loads(l); b = json.loads(d["body"]) + print(d["signature"], b["severity"], b["title"], "resolve=" + str(b["resolve"]))' +``` + +Expect: + +```text +{"halted":"drill (by root)","halt":{"id":"halt_7318705a-a165 +{"halted":null,"lifted":1} +valid warning New calls are blocked resolve=False +valid info Resolved: New calls are blocked resolve=True +``` + +### 5.3 An unresolved step + +An outcome that stays unknown for more than **one hour** raises "Outcome still unknown". The hour is fixed, so this guide cannot wait for +it. Leave one behind and look again later: + +```bash +j $AGENT -X POST $B/v1/runs -d '{"run_id":"al-3","permissions":["orders:write"]}' | cut -c1-40 +j $AGENT -X POST $B/v1/runs/al-3/intents -d '{"tool":"create_order","args":{"sku":"B"},"reason":"order"}' +j $AGENT -X POST $B/v1/runs/al-3/steps/0/outcome -d '{"status":"uncertain","error":"timeout"}' +j $AUD $B/v1/alerts | python3 -c 'import sys,json;print([a["title"] for a in json.load(sys.stdin)["alerts"] if "unknown" in a["title"]])' +``` + +Expect: + +```text +{"run_id":"al-3","mode":"enforce","scope +{"decision":"allow","step":0} +{"step":0,"status":"uncertain"} +[] +``` + +The empty list is right for now (the step is a minute old). An hour later the same query lists `Outcome still unknown: create_order in al-3`. +This part is **not verified** here. + +### 5.4 OpenTelemetry traces + +Spans are posted as OTLP JSON to `/v1/traces` every five seconds. Terminal D received them: + +```bash +j $AGENT -X POST $B/v1/runs -d '{"run_id":"al-2","permissions":["customers:read","orders:write","payments:write"]}' > /dev/null +j $AGENT -X POST $B/v1/runs/al-2/intents -d '{"tool":"create_order","args":{"sku":"B"},"reason":"order"}' > /dev/null +j $AGENT -X POST $B/v1/runs/al-2/steps/0/outcome -d '{"status":"succeeded","result":{"order_id":"o1"}}' > /dev/null +j $AGENT -X POST $B/v1/runs/al-2/intents -d '{"tool":"delete_customer","args":{"id":"c1"},"reason":"x"}' > /dev/null +sleep 6 +tr -d '\000' < try/ops/otlp.log | tail -n 2 | cut -c1-200 +python3 - <<'EOF2' +import json +batches = [json.loads(l) for l in open("try/ops/traces.ndjson")] +vals = lambda attrs: {a["key"]: list(a["value"].values())[0] for a in attrs} +print(vals(batches[0]["resourceSpans"][0]["resource"]["attributes"])) +spans = [s for b in batches for r in b["resourceSpans"] for sc in r["scopeSpans"] for s in sc["spans"]] +s = next(x for x in spans if x["name"] == "reversi.call create_order" and vals(x["attributes"])["reversi.run_id"] == "al-2") +print(s["name"], "| status", s["status"], "| kind", s["kind"], "| traceId", s["traceId"][:12] + "…") +print(vals(s["attributes"])) +EOF2 +grep -c '"stringValue":"\(o1\|c1\|B\)"' try/ops/traces.ndjson +``` + +Expect: + +```text +{"path":"/v1/traces","spans":["reversi.call.blocked delete_customer status=2","reversi.call.blocked delete_customer status=2","reversi.call.blocked delete_customer status=2","reversi.call... +{"path":"/v1/traces","spans":["reversi.call create_order status=2","reversi.call create_order status=1","reversi.call.blocked delete_customer status=2"]} +{'service.name': 'reversi-try', 'deployment.environment': 'default'} +reversi.call create_order | status {'code': 1} | kind 1 | traceId 0ff933828a98… +{'reversi.run_id': 'al-2', 'reversi.step': '0', 'reversi.tool': 'create_order', 'reversi.actor': 'order-agent', 'reversi.effect': 'reversible', 'reversi.event': 'succeeded', 'reversi.reas... +0 +``` + +The last number is 0: none of the argument or result values (`o1`, `c1`, `B`) appear in any span. Spans carry names and ids only. + +### 5.5 Structured logs and Prometheus metrics + +`REVERSI_LOG=json` writes one JSON line per request. Bodies and credentials are never logged: + +```bash +tail -n 2 try/ops/a.log | cut -c1-200 +echo "agent key in the log: $(grep -ac "$AGENT" try/ops/a.log); request bodies in the log: $(grep -ac probing try/ops/a.log)" +curl -s -H "Authorization: Bearer $AUD" $B/metrics | grep -E '^reversi_(ledger_events_total|ledger_records|runs|approvals\{status="pending"\}|halted)' +curl -s -o /dev/null -w "metrics without a credential: %{http_code}\n" $B/metrics +``` + +Expect: + +```text +{"ts":"2026-10-16T18:13:33.419Z","method":"POST","path":"/v1/runs/al-2/steps/0/outcome","status":200,"ms":10.5,"principal":"order-agent"} +{"ts":"2026-10-16T18:13:33.439Z","method":"POST","path":"/v1/runs/al-2/intents","status":200,"ms":5.6,"principal":"order-agent"} +agent key in the log: 0; request bodies in the log: 0 +reversi_ledger_events_total{event="blocked"} 12 +reversi_ledger_events_total{event="intended"} 3 +reversi_ledger_events_total{event="run_started"} 3 +reversi_ledger_events_total{event="succeeded"} 2 +reversi_ledger_events_total{event="uncertain"} 1 +reversi_ledger_records 21 +reversi_runs 3 +reversi_approvals{status="pending"} 0 +reversi_halted 0 +metrics without a credential: 401 +``` + +The library logger (`createLogger`, used by embedders) redacts credentials by field name and by pattern. Try it directly: + +```bash +node -e 'import("@multimindlab/reversi").then(({createLogger})=>{const l=createLogger({base:{service:"demo"}});l.info("charge failed",{api_key:"sk_live_abcdefghijklmnop123456",authorization:"Bearer abc",note:"saw AKIAABCDEFGHIJKLMNOP in output",amount:5})})' +``` + +Expect: + +```text +{"level":"info","time":"2026-10-16T18:13:39.604Z","msg":"charge failed","service":"demo","api_key":"[REDACTED]","authorization":"[REDACTED]","note":"saw [REDACTED:aws_access_key] in outpu... +``` + + +Two logs exist. `REVERSI_LOG=json` writes the per-request line above (server only). Warnings from the engine itself (a failed notification, anchor or trace +export) go through the `log:` section of `reversi.config.yaml`: `format: json` makes them redacted JSON lines on stderr, as section 4.4 showed, and +`format: pretty` writes `reversi [warn] message`, and `level` is `debug`, `info`, `warn` or `error`. The code above is the same logger, for embedders. + + +```bash +# Terminals A, C and D: press Ctrl-C +``` + +**What this proves:** alerts are computed live and also pushed once, with a resolve when they clear; traces leave as OTLP/HTTP JSON +without argument values; logs and metrics carry no credentials; `/metrics` needs an auditor key. + +## 6. Evidence: ledger, bundles, reports, anchors + +This section runs on the main data directory. The config adds one anchor sink, a file, and a data `region` label (section 8). A server start writes one anchor. + +```bash +cat > try/ops/evidence.yaml <<'EOF2' +manifests: ./manifests +policy: ./policy.yaml +data: { dir: ./data } +egress: { region: eu-west } +anchor: + sinks: + - { type: file, path: ./anchors.jsonl } + interval: 1h +EOF2 +``` + +```bash +# Terminal A +. try/ops/.env.sh +REVERSI_CONFIG=try/ops/evidence.yaml REVERSI_CONSOLE=login REVERSI_SESSION_SECRET=$(openssl rand -hex 16) PORT=7800 node packages/server/dist/main.js +``` + +A run with a success, an approval and a charge: + +```bash +j $AGENT -X POST $B/v1/runs -d '{"run_id":"ev-1","permissions":["customers:read","orders:write","payments:write"]}' | cut -c1-40 +j $AGENT -X POST $B/v1/runs/ev-1/intents -d '{"tool":"create_order","args":{"sku":"BOOK-1"},"reason":"customer asked"}' +j $AGENT -X POST $B/v1/runs/ev-1/steps/0/outcome -d '{"status":"succeeded","result":{"order_id":"ord_1"}}' +j $AGENT -X POST $B/v1/runs/ev-1/intents -d '{"tool":"charge_card","args":{"amount":500},"reason":"pay"}' | cut -c1-70 +j $APPR -X POST $B/v1/approvals/$(pending ev-1)/approve -d '{"note":"checked ticket 123"}' | cut -c1-70 +j $AGENT -X POST $B/v1/runs/ev-1/intents -d '{"tool":"charge_card","args":{"amount":500},"reason":"pay"}' +j $AGENT -X POST $B/v1/runs/ev-1/steps/1/outcome -d '{"status":"succeeded","result":{"charge_id":"ch_1"}}' +``` + +Expect: + +```text +{"run_id":"ev-1","mode":"enforce","scope +{"decision":"allow","step":0} +{"step":0,"status":"succeeded"} +{"decision":"block","reasons":["approval required: request apr_730d46f +{"id":"apr_…","run_id":"ev-1","step +{"decision":"allow","step":1} +{"step":1,"status":"succeeded"} +``` + +### 6.1 Verify the ledger and make signed bundles + +```bash +rv verify +j $AUD $B/v1/ledger/verify +rv bundle --key try/ops/signing.key --out try/ops/ledger-bundle.json +rv verify-bundle try/ops/ledger-bundle.json --pub try/ops/signing.pub +sed 's/charge_card/delete_all/' try/ops/ledger-bundle.json > try/ops/tampered.json +rv verify-bundle try/ops/tampered.json --pub try/ops/signing.pub; echo "exit $?" +rv keygen try/ops/other > /dev/null +rv verify-bundle try/ops/ledger-bundle.json --pub try/ops/other.pub; echo "exit $?" +rv bundle --key try/ops/signing.key --out try/ops/run-bundle.json --run ev-1 +rv verify-bundle try/ops/run-bundle.json +curl -s -H "Authorization: Bearer $AUD" $B/v1/runs/ev-1/bundle -o try/ops/served-bundle.json +rv verify-bundle try/ops/served-bundle.json --pub try/ops/signing.pub +``` + +Expect: + +```text +ok: 17 records, head 0678eb13f71f… +{"ok":true,"records":17,"head":"0678eb13f71f…"} +wrote try/ops/ledger-bundle.json: 17 records, key d67bfe0ef2ea4b10 +ok: 17 records, hash chain verified, key d67bfe0ef2ea4b10 (trusted) +INVALID: records were modified after signing +exit 1 +INVALID: signature does not match the trusted key +exit 1 +wrote try/ops/run-bundle.json: 8 records, key d67bfe0ef2ea4b10 +ok: 8 records, record hashes verified (single run), key d67bfe0ef2ea4b10 (NOT checked against a trusted key) +ok: 8 records, record hashes verified (single run), key d67bfe0ef2ea4b10 (trusted) +``` + +### 6.2 HTML and PDF run reports + +```bash +rv report ev-1 --out try/ops/report.html +rv --config try/ops/evidence.yaml report ev-1 --out try/ops/report.pdf +head -c 5 try/ops/report.pdf; echo +file try/ops/report.pdf | cut -d, -f1-3 +grep -o '[^<]*' try/ops/report.html +echo "data region in the PDF: $(grep -a -c 'Data region: eu-west' try/ops/report.pdf)" +echo "approver's note in the report: $(grep -c 'checked ticket 123' try/ops/report.html); argument values in the report: $(grep -c 'BOOK-1' try/ops/report.html)" +``` + +Expect: + +```text +wrote try/ops/report.html +wrote try/ops/report.pdf +%PDF- +try/ops/report.pdf: PDF document, version 1.4, 1 pages +Reversi report: ev-1 +data region in the PDF: 1 +approver's note in the report: 1; argument values in the report: 0 +``` + +Open `try/ops/report.html` in a browser, and the PDF in a viewer; the PDF is a one-page summary. The console links the same files on the run page: + +```bash +login root $ROOT +curl -s -b try/ops/jar-root $B/console/runs/ev-1 | grep -o 'href="[^"]*\(report\|bundle\)[^"]*"' | sed 's/href=//; s/"//g' +curl -s -b try/ops/jar-root -o try/ops/console-report.pdf -w "console PDF: %{http_code} %{content_type}\n" $B/console/runs/ev-1/report.pdf +head -c 5 try/ops/console-report.pdf; echo +curl -s -b try/ops/jar-root -o /dev/null -w "console HTML report: %{http_code} %{content_type}\n" $B/console/runs/ev-1/report +curl -s -b try/ops/jar-root -o try/ops/console-bundle.json -w "console bundle: %{http_code}\n" $B/console/bundle +rv verify-bundle try/ops/console-bundle.json --pub try/ops/signing.pub +curl -s -b try/ops/jar-root $B/console/evidence | tr '\n' ' ' | msg | cut -c1-120 +``` + +Expect: + +```text +root: login 303 +/console/runs/ev-1/report +/console/runs/ev-1/report.pdf +/console/runs/ev-1/bundle +console PDF: 200 application/pdf +%PDF- +console HTML report: 200 text/html; charset=UTF-8 +console bundle: 200 +ok: 17 records, hash chain verified, key d67bfe0ef2ea4b10 (trusted) +Evidence Ledger: chain verified 17 records, head 0678eb13f71f… Downlo +``` + +### 6.3 The period report + +With no run id, `reversi report` summarises a period for an auditor: what agents did, what was blocked, approvals, rollbacks. Formats are `md`, `html` or `json`. + +```bash +rv report --since 1d | head -24 +rv report --since 1d --format html --out try/ops/period.html +j $AUD "$B/v1/report?since=1d" | cut -c1-120 +``` + +Expect: + +```text +# Reversi report + +Period: 2026-10-15 18:13Z to 2026-10-16 18:13Z + +**2** tool calls in **5** runs. 2 succeeded, 0 failed, 0 with an unknown outcome. + +## What the agents did +Calls by tool: +- charge_card: 1 +- create_order: 1 + +By reversibility class: +- compensable: 1 +- reversible: 1 + +By actor: +- order-agent: 2 + +## Blocked: 0 +- none + +## Would have been blocked (observe and advise modes): 0 +These calls ran. Enforcing would have stopped them: +- none +wrote try/ops/period.html +{"from_ms":1792088025907,"to_ms":1792174425907,"runs":5,"calls":2,"actors":[{"key":"order-agent","count":2}],"tools":[{" +``` + +### 6.4 Anchoring + +An anchor is a signed note of the ledger's length and head hash, written somewhere you control. It lets you prove later that history was +not cut back or rewritten: the check compares the anchored hash with the ledger's record at that position, so a tail that was cut off or a chain that was rebuilt differently both fail. The server already wrote one when it started; +write another through the API and the CLI, then verify: + +```bash +cat try/ops/anchors.jsonl | python3 -c 'import sys,json;[print({k:(v[:16] if isinstance(v,str) else v) for k,v in json.loads(l).items()}) for l in sys.stdin]' | cut -c1-230 +j $ROOT -X POST $B/v1/anchor | cut -c1-110 +rv anchor publish --file try/ops/anchors.jsonl --key try/ops/signing.key +rv anchor verify try/ops/anchors.jsonl --pub try/ops/signing.pub +``` + +Expect: + +```text +{'format': 'reversi-anchor', 'version': 1, 'ts_ms': 1792174420750, 'head': '0949ae344fa949f6', 'records': 9, 'region': 'eu-west', 'key_id': 'd67bfe0ef2ea4b10', 'signature': 'e1wRFiMtSdUIY... +{"anchor":{"format":"reversi-anchor","version":1,"ts_ms":1792174425961,"head":"0678eb13f71fc95b9e3de85d59a380f +anchored 17 records, head 0678eb13f71fc95b… (signed) + delivered to file:try/ops/anchors.jsonl +2026-10-16T18:13:40Z 9 records ok (signature trusted) +2026-10-16T18:13:45Z 17 records ok (signature trusted) +2026-10-16T18:13:46Z 17 records ok (signature trusted) +``` + +A truncated copy of the data directory still passes `reversi verify` (the chain is intact up to where it stops) but fails the anchors: + +```bash +rm -rf try/ops/data-copy; cp -R try/ops/data try/ops/data-copy +head -n 14 try/ops/data/ledger.jsonl > try/ops/data-copy/ledger.jsonl +rv verify -d try/ops/data-copy; echo "exit $?" +rv anchor verify try/ops/anchors.jsonl --pub try/ops/signing.pub -d try/ops/data-copy; echo "exit $?" +rv anchor verify try/ops/anchors.jsonl --pub try/ops/other.pub; echo "exit $?" +``` + +Expect: + +```text +ok: 14 records, head ba1f26e7ed9b… +exit 0 +2026-10-16T18:13:40Z 9 records ok (signature trusted) +2026-10-16T18:13:45Z 17 records FAIL: the ledger has 14 records but was anchored at 17: it has been truncated +2026-10-16T18:13:46Z 17 records FAIL: the ledger has 14 records but was anchored at 17: it has been truncated +exit 1 +2026-10-16T18:13:40Z 9 records FAIL: the anchor's signature does not match the trusted key +2026-10-16T18:13:45Z 17 records FAIL: the anchor's signature does not match the trusted key +2026-10-16T18:13:46Z 17 records FAIL: the anchor's signature does not match the trusted key +exit 1 +``` + +A rewritten history, here a different but internally valid ledger (the one from section 5) put in the copy's place, also passes `reversi verify` and fails the anchors: + +```bash +cp try/ops/data-alerts/ledger.jsonl try/ops/data-copy/ledger.jsonl +rv verify -d try/ops/data-copy | cut -c1-40 +rv anchor verify try/ops/anchors.jsonl --pub try/ops/signing.pub -d try/ops/data-copy; echo "exit $?" +``` + +Expect: + +```text +ok: 21 records, head a357813bdf7caa0bdbe +2026-10-16T18:13:40Z 9 records FAIL: record 8 no longer has the anchored hash: history was rewritten +2026-10-16T18:13:45Z 17 records FAIL: record 16 no longer has the anchored hash: history was rewritten +2026-10-16T18:13:46Z 17 records FAIL: record 16 no longer has the anchored hash: history was rewritten +exit 1 +``` + +With no `--file`, `anchor publish` uses the `anchor:` sinks of the config (the CLI reads `anchor:`, `notifications:` and `alerts:` from the project file). Sign it with `--key`: + +```bash +rv --config try/ops/evidence.yaml anchor publish --key try/ops/signing.key; echo "exit $?" +tail -n 1 try/ops/anchors.jsonl | jq -c '{records,key_id,signed:(.signature != null)}' +``` + +Expect (the record count and head depend on your run): + +```text +anchored 21 records, head a357813bdf7cxxxx… (signed) + delivered to file:/try/ops/anchors.jsonl +exit 0 +{"records":21,"key_id":"...","signed":true} +``` + +```bash +# Terminal A: press Ctrl-C +``` + +**What this proves:** the ledger verifies end to end; a signed bundle verifies against a key you trust and fails if one character +changes or the wrong key is used; reports exist as HTML and PDF (a real `%PDF` file) and are linked in the console; reports show names and +notes, never argument values; anchors catch a truncated or rewritten history that a chain check alone accepts. + +## 7. Retention and legal holds + +Retention deletes the working data around a run (run state, approval requests and captured arguments). The **ledger is never +deleted**; a `purged` record is appended instead. The policy's `retention:` section sets `default_days` (at least 1) and optional rules by tool, +tenant, environment or data class. A manifest can raise a run's retention with `audit.retention_days` and can require an evidence bundle first with +`audit.evidence_export: required` (our `charge_card` does). We work on a copy so the main data survives. + +```bash +rm -rf try/ops/data-ret try/ops/archive; cp -R try/ops/data try/ops/data-ret +export REVERSI_DATA=try/ops/data-ret +``` + +```bash +rv runs list +rv retention plan +``` + +Expect: + +```text +c-1 actor=order-agent mode=enforce steps=0 blocked_only +ev-1 actor=order-agent mode=enforce steps=2 complete +l-1 actor=order-agent mode=enforce steps=0 blocked_only +n-1 actor=order-agent mode=enforce steps=0 blocked_only +oidc-1 actor=bot-1 mode=enforce steps=0 empty +c-1 0d old keep 30d keep +ev-1 0d old keep 30d keep +l-1 0d old keep 30d keep +n-1 0d old keep 30d keep +oidc-1 0d old keep 30d keep +``` + +Everything is new, so nothing is due. To test without waiting a month, override the policy with `--days 0`, which makes every run due: + +```bash +rv retention plan --days 0 +``` + +Expect: + +```text +c-1 0d old keep 0d due +ev-1 0d old keep 0d due; needs an evidence bundle first +l-1 0d old keep 0d due +n-1 0d old keep 0d due +oidc-1 0d old keep 0d due +``` + +A legal hold blocks deletion. Hold `c-1`, then do a dry run, a real run without an archive, and a real run with one: + +```bash +rv hold add --run c-1 --reason "case 4411 litigation hold" --by legal +rv hold list +rv retention apply --days 0 --dry-run +rv retention apply --days 0 --archive-dir try/ops/archive --key try/ops/signing.key +ls try/ops/archive +rv verify-bundle try/ops/archive/ev-1.json --pub try/ops/signing.pub +rv runs list +rv verify +rv export | grep '"purged"' | head -1 | cut -c1-230 +``` + +Expect: + +```text +hold_… holding run_id=c-1 +hold_… run_id=c-1 case 4411 litigation hold by legal +would delete: l-1, n-1, oidc-1 +kept c-1: legal hold hold_…: case 4411 litigation hold +kept ev-1: a manifest requires an evidence bundle before deletion; give an archive directory and a signing key +deleted: ev-1, l-1, n-1, oidc-1 +archived ev-1 -> try/ops/archive/ev-1.json +archived l-1 -> try/ops/archive/l-1.json +archived n-1 -> try/ops/archive/n-1.json +archived oidc-1 -> try/ops/archive/oidc-1.json +kept c-1: legal hold hold_…: case 4411 litigation hold +ev-1.json +l-1.json +n-1.json +oidc-1.json +ok: 8 records, record hashes verified (single run), key d67bfe0ef2ea4b10 (trusted) +c-1 actor=order-agent mode=enforce steps=0 blocked_only +ok: 21 records, head 9935752ef21a… +{"seq":17,"ts_ms":1792174429560,"run_id":"ev-1","step":null,"actor":"cli","tool":"-","payload_hash":"74234e98afe7…","reason":"retention: older than 0 days","event":"purged","detai +``` + +`ev-1` uses `charge_card`, whose manifest requires an evidence bundle, so it was kept in the dry run (no archive directory given) and deleted only once a directory and key were. With them, every deleted run gets a signed bundle, not only the ones that require it. +The held run `c-1` survived and says why. Release the hold and apply again: + +```bash +rv hold release $(rv hold list | awk '{print $1}') +rv hold list --all | cut -c1-110 +rv retention apply --days 0 --archive-dir try/ops/archive --key try/ops/signing.key +rv runs list; rv verify +export REVERSI_DATA=try/ops/data +``` + +Expect: + +```text +released hold_… +hold_… run_id=c-1 case 4411 litigation hold by legal RELEASED +deleted: c-1 +archived c-1 -> try/ops/archive/c-1.json +ok: 22 records, head 4b0b6597193b… +``` + +The same is available on the server: `GET /v1/retention/plan`, `POST /v1/holds`, `POST /v1/retention/apply` (admin), and the console **Settings** page with Dry run and Apply buttons. +Against the main data (nothing is due under the 30 day policy): + +```bash +# Terminal A +. try/ops/.env.sh +REVERSI_CONSOLE=login REVERSI_SESSION_SECRET=$(openssl rand -hex 16) PORT=7800 node packages/server/dist/main.js +``` + +```bash +sleep 1 +j $AUD $B/v1/retention/plan | cut -c1-150 +j $ROOT -X POST $B/v1/holds -d '{"run_id":"ev-1","reason":"case 7 under review"}' | cut -c1-110 +login root $ROOT +curl -s -b try/ops/jar-root $B/console/settings | grep -o 'Retention.\{0,120\}' | sed 's/<[^>]*>/ /g' | tr -s ' ' | cut -c1-110 +post -b try/ops/jar-root -d "csrf=$(csrf root)&dry=1" $B/console/settings/retention +j $ROOT -X DELETE $B/v1/holds/$(j $AUD $B/v1/holds | python3 -c 'import sys,json;print(json.load(sys.stdin)["holds"][0]["id"])') +``` + +Expect: + +```text +{"config":{"default_days":30,"rules":[]},"runs":[{"run_id":"c-1","age_days":0,"retention_days":30,"due":false,"needs_export":false},{"run_id":"ev-1"," +{"id":"hold_…","scope":{"run_id":"ev-1"},"reason":"case 7 under review","by +root: login 303 +Retention Default 30 days. 5 runs tracked, 0 due for deletion now (holds respected).
try/ops/egress.yaml <<'EOF2' +manifests: ./manifests +policy: ./policy.yaml +data: { dir: ./data } +egress: + allow: ["127.0.0.1", "169.254.169.254"] + region: eu-west +notifications: + channels: + - { type: webhook, name: listed, url: "http://127.0.0.1:7811/webhook" } + - { type: webhook, name: not-listed, url: "http://localhost:7811/webhook" } + - { type: webhook, name: metadata, url: "http://169.254.169.254/latest/meta-data" } +EOF2 +cat > try/ops/egress-private.yaml <<'EOF2' +manifests: ./manifests +policy: ./policy.yaml +data: { dir: ./data } +egress: + block_private: true +notifications: + channels: + - { type: webhook, name: loopback, url: "http://127.0.0.1:7811/webhook" } + - { type: webhook, name: lan, url: "http://10.1.2.3/hook" } +EOF2 +``` + +`reversi doctor` checks every configured destination without sending anything: + +```bash +rv --config try/ops/egress.yaml doctor | grep " egress" +rv --config try/ops/egress-private.yaml doctor | grep " egress" +``` + +Expect: + +```text +ok egress allows 127.0.0.1:7811 +FAIL egress: localhost is not in the egress allowlist (region eu-west) +FAIL egress: 169.254.169.254 resolves to a cloud metadata address (169.254.169.254), which is never allowed +FAIL egress: 127.0.0.1 resolves to a private or loopback address (127.0.0.1) and egress.block_private is on +FAIL egress: 10.1.2.3 resolves to a private or loopback address (10.1.2.3) and egress.block_private is on +``` + +Now for real, from a running server. The fake endpoint is on 127.0.0.1 (listed); the same fake reached as `localhost` is not in `allow`, and the metadata address is refused even though it is listed: + +```bash +# Terminal C +. try/ops/.env.sh +node try/ops/fake.mjs 7811 2>&1 | tee try/ops/fake.log +``` + +```bash +# Terminal A +. try/ops/.env.sh +REVERSI_CONFIG=try/ops/egress.yaml REVERSI_CONSOLE=login REVERSI_SESSION_SECRET=$(openssl rand -hex 16) PORT=7800 node packages/server/dist/main.js 2>&1 | tee try/ops/a.log +``` + +```bash +login root $ROOT +post -b try/ops/jar-root -d "csrf=$(csrf root)" $B/console/settings/notify-test +sleep 1 +grep -a "warn" try/ops/a.log | cut -c1-170 +tr -d '\000' < try/ops/fake.log | grep -a path | cut -c1-90 +curl -s -b try/ops/jar-root $B/console/settings | grep -o 'Data residency.\{0,120\}' | sed 's/<[^>]*>//g' | cut -c1-90 +``` + +Expect: + +```text +root: login 303 +200 Test notification Sent to: listed. Failed: not-listed, metadata. Back +reversi [warn] notification to not-listed failed: webhook not-listed: localhost is not in the egress allowlist (region eu-west) +reversi [warn] notification to metadata failed: webhook metadata: 169.254.169.254 resolves to a cloud metadata address (169.254.169.254), which is never allowed +{"path":"/webhook","event":"test","body":"{\"event\":\"test\",\"ts\":\"2026-10-16T18:14:01 +Data residency: outbound addresses are checked against the egress rules +``` + +```bash +# Terminals A and C: press Ctrl-C +``` + +**What this proves:** an allow list stops data going to hosts you did not name, `block_private` stops it going to internal addresses, metadata +endpoints are never reachable, and the refusal is a warning that does not affect any decision. The same guard covers anchor webhooks, OTLP export, +remote manifest URLs and gateway upstream URLs; only notifications are exercised here. + +## 9. The three stores, with the same scenario + +The same four calls against each store: open a run, create an order, ask for a charge over 100 (blocked for approval), and try a tool out of scope. +Then a **second server on the same store** approves the charge and the first server lets it through, which shows that the ledger and approvals are shared. + +```bash +scenario() { local B=localhost:$1 R=$2 + j $AGENT -X POST $B/v1/runs -d "{\"run_id\":\"$R\",\"permissions\":[\"orders:write\",\"payments:write\"]}" > /dev/null + j $AGENT -X POST $B/v1/runs/$R/intents -d '{"tool":"create_order","args":{"sku":"B"},"reason":"order"}' + j $AGENT -X POST $B/v1/runs/$R/steps/0/outcome -d '{"status":"succeeded","result":{"order_id":"o1"}}' + j $AGENT -X POST $B/v1/runs/$R/intents -d '{"tool":"charge_card","args":{"amount":500},"reason":"pay"}' | cut -c1-90 + j $AGENT -X POST $B/v1/runs/$R/intents -d '{"tool":"delete_customer","args":{"id":"c1"},"reason":"x"}' | cut -c1-110 +} +shared() { local R=$1 P1=$2 P2=$3 + ID=$(curl -s -H "Authorization: Bearer $APPR" "localhost:$P2/v1/approvals?status=pending" | python3 -c "import sys,json;print([a['id'] for a in json.load(sys.stdin)['approvals'] if a['run_id']=='$R'][0])") + curl -s -X POST -H "Authorization: Bearer $APPR" -H 'content-type: application/json' localhost:$P2/v1/approvals/$ID/approve -d '{"note":"ok"}' | python3 -c 'import sys,json;a=json.load(sys.stdin);print("approved on the second server:",a["status"],a["decided_by"])' + j $AGENT -X POST localhost:$P1/v1/runs/$R/intents -d '{"tool":"charge_card","args":{"amount":500},"reason":"pay"}' + j $AGENT -X POST localhost:$P1/v1/runs/$R/steps/1/outcome -d '{"status":"succeeded","result":{"charge_id":"ch_1"}}' +} +ledgers() { for p in "$@"; do curl -s -H "Authorization: Bearer $AUD" localhost:$p/v1/ledger/verify | python3 -c "import sys,json;d=json.load(sys.stdin);print('port $p:',d['ok'],d['records'],'records, head',d['head'][:16])"; done; } +TAG=$(date +%s) +``` + +### 9.1 Files (the default) + +```bash +# Terminal A +. try/ops/.env.sh +REVERSI_STORE=files REVERSI_DATA=try/ops/st-files PORT=7800 node packages/server/dist/main.js +``` + +```bash +# Terminal C +. try/ops/.env.sh +REVERSI_STORE=files REVERSI_DATA=try/ops/st-files PORT=7803 node packages/server/dist/main.js +``` + +```bash +sleep 1 +scenario 7800 files-$TAG +shared files-$TAG 7800 7803 +rv runs list -d try/ops/st-files --store files +rv verify -d try/ops/st-files --store files | cut -c1-40 +ledgers 7800 7803 +``` + +Expect: + +```text +{"decision":"allow","step":0} +{"step":0,"status":"succeeded"} +{"decision":"block","reasons":["approval required: request apr_6220eef2-2144-4b14-8094-1e6 +{"decision":"block","reasons":["permission `customers:admin` is not in the run's scope"]} +approved on the second server: approved alice +{"decision":"allow","step":1} +{"step":1,"status":"succeeded"} +files-", "data directory", "./reversi-data") - .option("--store ", "files | sqlite | postgres (default: REVERSI_STORE or files)") - .option("--database-url ", "PostgreSQL URL (default: REVERSI_DATABASE_URL)"); - const dataOpts = (c: Command) => storeOpts(c.requiredOption("-m, --manifests ", "manifest file or directory (.json, .yaml)")); - - interface StoreFlags { - data: string; - store?: string; - databaseUrl?: string; - } - const dataOptionsOf = (o: StoreFlags) => { - const env = dataOptionsFromEnv(); - return { - dir: o.data, - store: o.store ? storeKindFrom(o.store) : env.store, - postgres_url: o.databaseUrl ?? env.postgres_url, - }; - }; - const dataExists = (o: StoreFlags) => { - const d = dataOptionsOf(o); - if (d.store === "files" && !existsSync(o.data)) throw new Fail(`data directory not found: ${o.data}`); - }; - /** Open the engine; the caller closes the store. Policy only matters for new calls, which the CLI never makes. */ - const engine = async (o: StoreFlags & { manifests: string }): Promise => { - dataExists(o); - return openEngine({ manifests: o.manifests, data: dataOptionsOf(o) }); - }; - const withEngine = async (o: Parameters[0], fn: (e: EngineHandle) => Promise | T): Promise => { - const e = await engine(o); - try { - return await fn(e); - } finally { - e.data.close(); - } - }; - const readDoc = (path: string): unknown => { - const text = readFileSync(path, "utf8"); - return extname(path).toLowerCase() === ".json" ? JSON.parse(text) : parseYaml(text); - }; + program.option("--config ", "project configuration file (default: reversi.config.yaml in this or a parent folder)"); + const ctx = makeContext(program, io); + const { storeOpts, dataOpts, dataOptionsOf, dataExists, withEngine, readDoc } = ctx; program .command("validate ") @@ -157,21 +149,10 @@ export async function main(argv: string[], io: IO = DEFAULT_IO): Promise if (v.ok) io.out(`ok: ${v.records} records, head ${v.head}`); else { io.err(`TAMPERED or corrupt at record ${v.seq}: ${v.error}`); - code = 1; + ctx.code = 1; } }); - dataOpts(program.command("runs").description("list runs and their step counts")).action((o) => - withEngine(o, (rv) => { - for (const id of rv.reversi.runIds()) { - const run = rv.reversi.openRun(id); - const s = run.state(); - const un = run.unresolved(); - io.out(`${id} actor=${s.actor} mode=${s.mode} steps=${s.steps.length}${un.length ? ` UNRESOLVED=${un.join(",")}` : ""}`); - } - }), - ); - dataOpts(program.command("plan ").description("show what a rollback would do; executes nothing")) .option("--json", "machine-readable output") .action((run_id: string, o) => @@ -179,13 +160,14 @@ export async function main(argv: string[], io: IO = DEFAULT_IO): Promise const p = rv.reversi.openRun(run_id).plan(); if (o.json) io.out(JSON.stringify(p, null, 2)); else for (const l of formatPlan(p)) io.out(l); - if (!p.fully_reversible) code = 2; + if (!p.fully_reversible) ctx.code = 2; }), ); dataOpts(program.command("rollback ").description("undo a run with an operator-supplied compensator command")) .requiredOption("--compensator-cmd ", "shell command; receives the compensation call as JSON on stdin; exit 0 = undone") .option("--verify-cmd ", "shell command run after each compensation with the same JSON; exit 0 = the undo is confirmed") + .option("--reader-cmd ", "shell command that runs a manifest's declared read-back: gets {tool, arguments} as JSON on stdin, prints the result as JSON. Without it, steps that declare a verification end up unverified") .option("--approve-compensations", "approve compensations that require approval (operator decision)") .option("--retries ", "attempts per compensation, with exponential backoff and the same idempotency key", "1") .option("--timeout-ms ", "per-compensation timeout", "30000") @@ -206,6 +188,7 @@ export async function main(argv: string[], io: IO = DEFAULT_IO): Promise actor: "cli", retry: { attempts }, compensate: (c) => runCompensator(o.compensatorCmd, c, timeout), + verifier: o.readerCmd ? (tool, args) => runReader(o.readerCmd, tool, args, timeout) : undefined, verify: o.verifyCmd ? (c) => runCompensator(o.verifyCmd, c, timeout).then(() => true, () => false) : undefined, }); if (o.json) io.out(JSON.stringify(report, null, 2)); @@ -214,13 +197,14 @@ export async function main(argv: string[], io: IO = DEFAULT_IO): Promise io.out(`skipped (no undo needed): ${report.skipped.join(", ") || "-"}`); for (const f of report.failed) io.out(`FAILED step ${f.step}: ${f.error}`); if (report.unrecoverable.length) io.out(`cannot be undone: ${report.unrecoverable.join(", ")}`); + if (report.unverified.length) io.out(`undone but not verified (declared read-back not run; pass --reader-cmd): ${report.unverified.join(", ")}`); if (report.unresolved.length) io.out(`outcome unknown, reconcile manually (reversi resolve): ${report.unresolved.join(", ")}`); if (report.unmanifested.length) io.out(`no manifest: ${report.unmanifested.join(", ")}`); for (const d of report.deferred) io.out(`deferred step ${d.step}: waits for ${d.blocked_by.join(", ")}`); if (report.awaiting_approval.length) io.out(`awaiting approval: ${report.awaiting_approval.join(", ")}`); io.out(report.complete ? "rollback complete" : "rollback INCOMPLETE"); } - if (!report.complete) code = 2; + if (!report.complete) ctx.code = 2; }); }); @@ -262,16 +246,21 @@ export async function main(argv: string[], io: IO = DEFAULT_IO): Promise } }); for (const decision of ["approve", "deny"] as const) { - storeOpts(approvals.command(`${decision} `).description(`${decision} a pending request`).option("--note ", "recorded with the decision").option("--as ", "who is deciding", "cli")).action(async (id: string, o) => { + storeOpts(approvals.command(`${decision} `).description(`${decision} a pending request`).option("--note ", "recorded with the decision").option("--as ", "who is deciding", "cli").option("--groups ", "groups you decide as, comma separated (for approvals routed to groups)").option("--roles ", "roles you decide as (default operator)")).action(async (id: string, o) => { dataExists(o); - const d = await openData(dataOptionsOf(o)); - try { - const a = new Approvals(d.approvals); - const r = decision === "approve" ? a.approve(id, o.as, o.note) : a.deny(id, o.as, o.note); + // Through the engine, so the decision is journaled, delegations apply, and notifications go out like any other decision. + const m = (() => { + try { + return ctx.manifestsPath(o); + } catch { + return undefined; + } + })(); + await ctx.withEngine({ ...o, manifests: m ?? "-" }, (e) => { + const who = { roles: o.roles ? String(o.roles).split(",") : ["operator"], groups: o.groups ? String(o.groups).split(",").map((x: string) => x.trim()).filter(Boolean) : [] }; + const r = decision === "approve" ? e.approvals.approve(id, o.as, o.note, who) : e.approvals.deny(id, o.as, o.note, who); io.out(`${r.id} ${r.status} by ${r.decided_by}`); - } finally { - d.close(); - } + }, { manifests: m ?? "", allow_empty_manifests: !m }); }); } @@ -301,7 +290,7 @@ export async function main(argv: string[], io: IO = DEFAULT_IO): Promise for (const r of results) io.out(`${r.ok ? "ok " : "FAIL"} ${r.name}${r.ok ? "" : `: expected ${r.expected}, got ${r.got}${r.message ? ` (${r.message})` : ""}`}`); const bad = results.filter((r) => !r.ok).length; io.out(bad ? `${bad} of ${results.length} failed` : `${results.length} passed`); - if (bad) code = 1; + if (bad) ctx.code = 1; }); policy .command("check ") @@ -370,23 +359,42 @@ export async function main(argv: string[], io: IO = DEFAULT_IO): Promise const r = verifyBundle(JSON.parse(readFileSync(file, "utf8")), { trusted_public_pem: o.pub ? readFileSync(o.pub, "utf8") : undefined }); if (!r.ok) { io.err(`INVALID: ${r.error}`); - code = 1; + ctx.code = 1; return; } io.out(`ok: ${r.records} records, ${r.chain === "full" ? "hash chain verified" : "record hashes verified (single run)"}, key ${r.key_id}${r.trusted_key ? " (trusted)" : " (NOT checked against a trusted key)"}`); }); - dataOpts(program.command("report ").description("write a self-contained HTML report for auditors").requiredOption("--out ", "output file")).action((run_id: string, o) => - withEngine(o, (rv) => { + dataOpts( + program + .command("report [run_id]") + .description("an auditor report for one run (HTML or PDF by file extension), or with no run id the report for a period: what agents did, what was blocked or would have been, approvals, rollbacks") + .option("--out ", "output file (.html, .pdf, .md, .json)") + .option("--since ", "for the period report: 7d, 2026-10-01, ... (default 7d)") + .option("--format ", "for the period report: md | html | json"), + ).action(async (run_id: string | undefined, o: EngineFlags & { out?: string; since?: string; format?: string }) => { + if (!run_id) return periodReportCommand(ctx, o); + if (!o.out) throw new Fail("--out is required for a run report"); + await withEngine(o, (rv) => { const run = rv.reversi.openRun(run_id); - writeFileSync(o.out, renderReport({ run: run.state(), records: rv.reversi.ledger.forRun(run_id), plan: run.plan(), ledger_head: rv.reversi.ledger.head() })); + const args = { run: run.state(), records: rv.reversi.ledger.forRun(run_id), plan: run.plan(), ledger_head: rv.reversi.ledger.head() }; + if (o.out?.toLowerCase().endsWith(".pdf")) writeFileSync(o.out, renderReportPdf({ ...args, region: rv.egress.region })); + else writeFileSync(o.out as string, renderReport(args)); io.out(`wrote ${o.out}`); - }), - ); + }); + }); + + registerProjectCommands(program, ctx); + registerManifestCommands(program, ctx); + registerInspectCommands(program, ctx); + registerSimulationCommands(program, ctx); + registerGovernanceCommands(program, ctx); + registerApplyCommand(program, ctx); + registerServeCommands(program, ctx); try { await program.parseAsync(argv, { from: "user" }); - return code; + return ctx.code; } catch (e) { if (e instanceof CommanderError) return e.exitCode === 0 ? 0 : 64; if (e instanceof ManifestError || e instanceof PolicyError) { diff --git a/packages/cli/src/commands/apply.ts b/packages/cli/src/commands/apply.ts new file mode 100644 index 0000000..4fb8dbf --- /dev/null +++ b/packages/cli/src/commands/apply.ts @@ -0,0 +1,214 @@ +/** + * `reversi apply `: make a registry (and policy) match a directory in git. A directory holds manifests (`manifests/*.yaml`, + * or yaml/json files at the top) and optionally `policy.yaml`. It is compared with the target, a local registry or a running + * server, and only what differs is published. Nothing is deleted: `--prune` deprecates tools that left the directory. + */ +import { createPrivateKey, createPublicKey } from "node:crypto"; +import { existsSync, readFileSync, readdirSync, statSync } from "node:fs"; +import { userInfo } from "node:os"; +import { join } from "node:path"; +import type { Command } from "commander"; +import { ManifestRegistry, PolicyVersions, canonicalize, sha256Hex, validateManifest, type SigningKey } from "@multimindlab/reversi"; +import { Fail, type Ctx } from "../context.js"; +import { readManifests } from "./manifests.js"; + +interface Current { + manifests: Map }>; + policy_checksum: string | null; +} +interface Target { + read(): Promise; + publish(manifest: Record, status: "draft" | "approved", note: string | undefined): Promise; + deprecate(tool: string): Promise; + publishPolicy(text: string, note: string | undefined): Promise; +} + +export interface ApplyChange { + kind: "manifest" | "policy"; + action: "create" | "update" | "deprecate" | "unchanged"; + name: string; + detail?: string; +} + +const me = () => { + try { + return userInfo().username; + } catch { + return "cli"; + } +}; + +/** Content that matters when comparing: everything except the registry-assigned version and status. */ +const essence = (m: Record): string => { + const r = validateManifest(m); // both sides in normalized form, so defaults do not show up as differences + const { version: _v, status: _s, ...rest } = (r.ok ? r.manifest : m) as unknown as Record; + return canonicalize(rest); +}; + +export function readDesired(dir: string): { manifests: Record[]; policy?: string } { + if (!existsSync(dir) || !statSync(dir).isDirectory()) throw new Fail(`not a directory: ${dir}`); + const mdir = existsSync(join(dir, "manifests")) ? join(dir, "manifests") : dir; + const files = mdir === dir ? readManifestsTop(dir) : readManifests(mdir); + const manifests: Record[] = []; + const seen = new Set(); + for (const raw of files) { + const r = validateManifest(raw); + if (!r.ok) throw new Fail(`invalid manifest ${(raw as { tool?: string }).tool ?? "?"}: ${r.errors.join("; ")}`); + if (seen.has(r.manifest.tool)) throw new Fail(`duplicate manifest for \`${r.manifest.tool}\` in ${dir}`); + seen.add(r.manifest.tool); + manifests.push(raw as Record); + } + const pf = ["policy.yaml", "policy.yml", "policy.json"].map((f) => join(dir, f)).find((f) => existsSync(f)); + return { manifests, ...(pf ? { policy: readFileSync(pf, "utf8") } : {}) }; +} + +/** A directory that is itself the manifest folder: policy.* files are not manifests. */ +function readManifestsTop(dir: string): unknown[] { + const all = readManifests(dir) as Array>; + return all.filter((m) => typeof m === "object" && m !== null && "tool" in m); +} + +export function plan(desired: ReturnType, current: Current, o: { prune: boolean }): ApplyChange[] { + const out: ApplyChange[] = []; + for (const raw of desired.manifests) { + const tool = (raw as { tool: string }).tool; + const have = current.manifests.get(tool); + if (!have) out.push({ kind: "manifest", action: "create", name: tool }); + else if (have.status === "deprecated" && (raw as { status?: string }).status !== "deprecated") out.push({ kind: "manifest", action: "update", name: tool, detail: `was deprecated at v${have.version}` }); + else if (essence(have.manifest) !== essence(raw)) out.push({ kind: "manifest", action: "update", name: tool, detail: `v${have.version} differs` }); + else out.push({ kind: "manifest", action: "unchanged", name: tool }); + } + if (o.prune) { + const wanted = new Set(desired.manifests.map((m) => (m as { tool: string }).tool)); + for (const [tool, have] of current.manifests) if (!wanted.has(tool) && have.status !== "deprecated") out.push({ kind: "manifest", action: "deprecate", name: tool, detail: "not in the directory" }); + } + if (desired.policy !== undefined) { + const sum = sha256Hex(desired.policy); + out.push({ kind: "policy", action: current.policy_checksum === null ? "create" : current.policy_checksum === sum ? "unchanged" : "update", name: "policy" }); + } + return out; +} + +function localTarget(registryDir: string, policyDir: string | undefined, by: string, key: SigningKey | undefined): Target { + const reg = new ManifestRegistry(registryDir); + return { + async read() { + const manifests: Current["manifests"] = new Map(); + if (ManifestRegistry.isRegistry(registryDir)) { + for (const t of reg.tools()) { + const cur = reg.history(t).at(-1); + if (cur) manifests.set(t, { version: cur.version, status: cur.status, manifest: reg.get(t, cur.version) as unknown as Record }); + } + } + let policy_checksum: string | null = null; + if (policyDir) { + const pv = new PolicyVersions(policyDir); + const a = pv.activeVersion(); + policy_checksum = a === null ? null : (pv.history().find((h) => h.version === a)?.checksum ?? null); + } + return { manifests, policy_checksum }; + }, + async publish(m, status, note) { + return reg.publish(m, { by, note, status, key }).version; + }, + async deprecate(tool) { + reg.deprecate(tool, { by, key }); + }, + async publishPolicy(text, note) { + if (!policyDir) throw new Fail("no policy folder: pass --policy-dir, or leave policy.yaml out of the directory"); + return new PolicyVersions(policyDir).publish(text, { by, note, activate: true }).version; + }, + }; +} + +function remoteTarget(url: string, token: string): Target { + const base = url.replace(/\/$/, ""); + const call = async (method: string, path: string, body?: unknown): Promise => { + const res = await fetch(`${base}${path}`, { method, headers: { authorization: `Bearer ${token}`, "content-type": "application/json" }, body: body === undefined ? undefined : JSON.stringify(body) }); + const text = await res.text(); + let json: any; + try { + json = text ? JSON.parse(text) : {}; + } catch { + json = { error: text }; + } + if (!res.ok) throw new Fail(`${method} ${path}: ${res.status} ${json.error ?? text}`); + return json; + }; + return { + async read() { + const j = await call("GET", "/v1/registry"); + const manifests: Current["manifests"] = new Map(); + for (const m of j.manifests as Array<{ tool: string; version: number; status: string; manifest: Record }>) manifests.set(m.tool, m); + return { manifests, policy_checksum: j.policy?.checksum ?? null }; + }, + async publish(manifest, status, note) { + return (await call("POST", "/v1/registry/manifests", { manifest, status, note })).version; + }, + async deprecate(tool) { + await call("POST", `/v1/registry/manifests/${encodeURIComponent(tool)}/deprecate`, {}); + }, + async publishPolicy(text, note) { + return (await call("POST", "/v1/registry/policy", { text, note, activate: true })).version; + }, + }; +} + +export function registerApplyCommand(program: Command, ctx: Ctx): void { + const { io } = ctx; + program + .command("apply ") + .description("make a registry (and policy) match a directory kept in git; only differences are published. Nothing is ever deleted") + .option("--registry ", "the local registry folder (default: `manifests:` in the config)") + .option("--policy-dir ", "the local policy versions folder (default: `policy:` in the config)") + .option("--server ", "apply to a running Reversi server instead (needs --token)") + .option("--token ", "admin API key for --server (default: REVERSI_TOKEN)") + .option("--status ", "approved (default: the directory is the reviewed source of truth) | draft", "approved") + .option("--prune", "deprecate tools that are in the target but not in the directory") + .option("--dry-run", "show what would change and change nothing") + .option("--detailed-exitcode", "exit 2 when there are changes (with --dry-run: for CI drift checks)") + .option("--by ", "who is applying (default: your user name)") + .option("--key ", "signing key for the new versions") + .option("--note ", "recorded with each new version") + .action(async (dir: string, o) => { + if (o.status !== "draft" && o.status !== "approved") throw new Fail("--status must be draft or approved"); + const desired = readDesired(dir); + let target: Target; + if (o.server) { + const token = o.token ?? process.env.REVERSI_TOKEN; + if (!token) throw new Fail("--server needs --token (or REVERSI_TOKEN)"); + target = remoteTarget(o.server, token); + } else { + const p = ctx.project(); + const registryDir = o.registry ?? p?.manifests; + if (!registryDir) throw new Fail("no registry folder: pass --registry or --server , or set `manifests:` in reversi.config.yaml"); + if (existsSync(registryDir) && !ManifestRegistry.isRegistry(registryDir) && statSync(registryDir).isDirectory() && readdirSync(registryDir).length > 0) { + throw new Fail(`${registryDir} holds manifest files but is not a registry: apply into an empty or registry folder`); + } + const policyDir = o.policyDir ?? (p?.policy && existsSync(p.policy) && statSync(p.policy).isDirectory() ? p.policy : undefined); + const key = o.key ? (() => { const private_pem = readFileSync(o.key, "utf8"); return { private_pem, public_pem: createPublicKey(createPrivateKey(private_pem)).export({ type: "spki", format: "pem" }).toString() }; })() : undefined; + target = localTarget(registryDir, policyDir, o.by ?? me(), key); + } + const changes = plan(desired, await target.read(), { prune: o.prune === true }); + const pending = changes.filter((c) => c.action !== "unchanged"); + const sign = { create: "+", update: "~", deprecate: "-", unchanged: " " } as const; + for (const c of changes) io.out(`${sign[c.action]} ${c.kind.padEnd(8)} ${c.name}${c.detail ? ` (${c.detail})` : ""}${c.action === "unchanged" ? " unchanged" : ""}`); + io.out(pending.length ? `${pending.length} change${pending.length === 1 ? "" : "s"}${o.dryRun ? " (dry run: nothing applied)" : ""}` : "up to date"); + if (!o.dryRun) { + for (const c of pending) { + if (c.kind === "policy") { + const v = await target.publishPolicy(desired.policy as string, o.note); + io.out(`applied policy v${v}`); + } else if (c.action === "deprecate") { + await target.deprecate(c.name); + io.out(`deprecated ${c.name}`); + } else { + const m = desired.manifests.find((x) => (x as { tool: string }).tool === c.name) as Record; + const v = await target.publish(m, o.status, o.note); + io.out(`applied ${c.name} v${v} (${o.status})`); + } + } + } + if (o.detailedExitcode && pending.length && o.dryRun) ctx.code = 2; + }); +} diff --git a/packages/cli/src/commands/governance.ts b/packages/cli/src/commands/governance.ts new file mode 100644 index 0000000..acc1f29 --- /dev/null +++ b/packages/cli/src/commands/governance.ts @@ -0,0 +1,154 @@ +/** Retention, legal holds, anchoring, delegation, break-glass and scoped halts. */ +import { readFileSync, writeFileSync } from "node:fs"; +import type { Command } from "commander"; +import { + Ledger, + anchorLedger, + applyRetention, + planRetention, + readAnchors, + verifyAnchor, + type HoldScope, + type RetentionConfig, + type SigningKey, +} from "@multimindlab/reversi"; +import { createPrivateKey, createPublicKey } from "node:crypto"; +import { Fail, type Ctx, type EngineFlags } from "../context.js"; + +function key(path: string | undefined): SigningKey | undefined { + if (!path) return undefined; + const private_pem = readFileSync(path, "utf8"); + return { private_pem, public_pem: createPublicKey(createPrivateKey(private_pem)).export({ type: "spki", format: "pem" }).toString() }; +} + +export function registerGovernanceCommands(program: Command, ctx: Ctx): void { + const { io } = ctx; + + // ---- retention ---- + const retention = program.command("retention").description("delete run data past its retention period (the ledger itself is never deleted)"); + const retentionConfig = async (e: Awaited>, days?: string): Promise => { + if (days) return { default_days: Number(days), rules: [] }; + const { loadPolicyFile } = await import("@multimindlab/reversi"); + const p = e.config.policy; + const doc = p ? (await import("node:fs")).statSync(p).isFile() ? loadPolicyFile(p).doc : undefined : undefined; + if (!doc?.retention) throw new Fail("no retention settings: add a `retention:` section to the policy file, or pass --days"); + return doc.retention; + }; + ctx.dataOpts(retention.command("plan").description("which runs are due for deletion, and which are held").option("--days ", "use this retention instead of the policy's")).action((o: EngineFlags & { days?: string }) => + ctx.withEngine(o, async (e) => { + const cfg = await retentionConfig(e, o.days); + const plan = planRetention(e.reversi, cfg, e.holds); + for (const p of plan) io.out(`${p.run_id} ${p.age_days}d old keep ${p.retention_days}d ${p.held ? `HELD (${p.held.reason})` : p.due ? (p.needs_export ? "due; needs an evidence bundle first" : "due") : "keep"}`); + if (!plan.length) io.out("no runs"); + }), + ); + ctx + .dataOpts(retention.command("apply").description("delete run state, approvals and captures for runs past retention").option("--days ").option("--dry-run").option("--archive-dir ", "write a signed evidence bundle for each run before deleting it").option("--key ", "signing key for the bundles")) + .action((o: EngineFlags & { days?: string; dryRun?: boolean; archiveDir?: string; key?: string }) => + ctx.withEngine(o, async (e) => { + const cfg = await retentionConfig(e, o.days); + const r = applyRetention(e.reversi, cfg, { holds: e.holds, approvals: e.approvals, capture: e.capture, dry_run: o.dryRun, archive_dir: o.archiveDir, key: key(o.key) ?? e.signing_key, actor: "cli" }); + io.out(`${o.dryRun ? "would delete" : "deleted"}: ${r.purged.join(", ") || "nothing"}`); + for (const x of r.exported) io.out(`archived ${x.run_id} -> ${x.file}`); + for (const s of r.skipped) io.out(`kept ${s.run_id}: ${s.reason}`); + }), + ); + + // ---- legal hold ---- + const hold = program.command("hold").description("legal holds: stop retention from deleting runs"); + ctx.dataOpts(hold.command("add").description("hold runs matching a scope").option("--run ").option("--tenant ").option("--actor ").option("--tool ").requiredOption("--reason ", "the case or matter").option("--by ", "who", "cli")).action((o: EngineFlags & HoldScope & { reason: string; by: string; run?: string }) => + ctx.withEngine(o, (e) => { + const h = e.holds.add({ run_id: o.run, tenant: o.tenant, actor: o.actor, tool: o.tool }, o.reason, o.by); + io.out(`${h.id} holding ${Object.entries(h.scope).filter(([, v]) => v).map(([k, v]) => `${k}=${v}`).join(", ")}`); + }), + ); + ctx.dataOpts(hold.command("list").description("active holds").option("--all", "include released ones")).action((o: EngineFlags & { all?: boolean }) => + ctx.withEngine(o, (e) => { + for (const h of e.holds.list({ all: o.all })) io.out(`${h.id} ${Object.entries(h.scope).filter(([, v]) => v).map(([k, v]) => `${k}=${v}`).join(", ")} ${h.reason} by ${h.by}${h.released_ms ? " RELEASED" : ""}`); + }), + ); + ctx.dataOpts(hold.command("release ")).action((id: string, o: EngineFlags) => + ctx.withEngine(o, (e) => { + if (!e.holds.release(id)) throw new Fail(`no active hold ${id}`); + io.out(`released ${id}`); + }), + ); + + // ---- anchors ---- + const anchor = program.command("anchor").description("publish the ledger head somewhere you control, and prove later that history is unchanged"); + ctx + .dataOpts(anchor.command("publish").description("write the current head to the sinks in the config (or --file / --webhook)").option("--file ", "append to this file").option("--webhook ").option("--secret ", "sign the webhook body with this secret").option("--key ", "sign the anchor")) + .action((o: EngineFlags & { file?: string; webhook?: string; secret?: string; key?: string }) => + ctx.withEngine(o, async (e) => { + const sinks = [...(o.file ? [{ type: "file" as const, path: o.file }] : []), ...(o.webhook ? [{ type: "webhook" as const, url: o.webhook, secret: o.secret }] : []), ...(!o.file && !o.webhook ? (e.config.anchor?.sinks ?? []) : [])]; + if (!sinks.length) throw new Fail("no anchor sinks: pass --file or --webhook, or add `anchor:` to reversi.config.yaml"); + const r = await anchorLedger(e.reversi.ledger, sinks, { key: key(o.key) ?? e.signing_key, egress: e.egress }); + io.out(`anchored ${r.anchor.records} records, head ${r.anchor.head.slice(0, 16)}…${r.anchor.signature ? " (signed)" : ""}`); + for (const d of r.delivered) io.out(` delivered to ${d}`); + for (const x of r.errors) io.err(` FAILED ${x}`); + if (r.errors.length) ctx.code = 1; + }), + ); + ctx + .dataOpts(anchor.command("verify ").description("check every anchor in a file against the ledger").option("--pub ", "trusted public key the anchors must be signed by")) + .action((file: string, o: EngineFlags & { pub?: string }) => + ctx.withEngine(o, (e) => { + const ledger: Ledger = e.reversi.ledger; + const anchors = readAnchors(file); + if (!anchors.length) throw new Fail(`no anchors in ${file}`); + let bad = 0; + for (const a of anchors) { + const r = verifyAnchor(ledger, a, o.pub ? readFileSync(o.pub, "utf8") : undefined); + io.out(`${new Date(a.ts_ms).toISOString().slice(0, 19)}Z ${a.records} records ${r.ok ? `ok${r.signed ? " (signature trusted)" : ""}` : `FAIL: ${r.error}`}`); + if (!r.ok) bad++; + } + if (bad) ctx.code = 1; + }), + ); + + // ---- delegation ---- + const delegate = program.command("delegate").description("hand your approval authority to someone for a while"); + ctx.dataOpts(delegate.command("add").requiredOption("--from ").requiredOption("--to ").requiredOption("--hours ").requiredOption("--reason ").option("--group ", "only this group (repeatable)", (v: string, p: string[] = []) => [...p, v])).action((o: EngineFlags & { from: string; to: string; hours: string; reason: string; group?: string[] }) => + ctx.withEngine(o, (e) => { + const d = e.delegations.grant({ from: o.from, to: o.to, until_ms: Date.now() + Number(o.hours) * 3_600_000, reason: o.reason, groups: o.group }); + io.out(`${d.id} ${d.from} -> ${d.to} until ${new Date(d.until_ms).toISOString().slice(0, 16)}Z`); + }), + ); + ctx.dataOpts(delegate.command("list")).action((o: EngineFlags) => ctx.withEngine(o, (e) => { + for (const d of e.delegations.list()) io.out(`${d.id} ${d.from} -> ${d.to} until ${new Date(d.until_ms).toISOString().slice(0, 16)}Z ${d.revoked ? "REVOKED" : d.until_ms < Date.now() ? "expired" : "active"} ${d.reason}`); + })); + ctx.dataOpts(delegate.command("revoke ")).action((id: string, o: EngineFlags) => ctx.withEngine(o, (e) => { + if (!e.delegations.revoke(id)) throw new Fail(`no delegation ${id}`); + io.out(`revoked ${id}`); + })); + + // ---- break-glass ---- + ctx + .dataOpts(program.command("break-glass ").description("emergency override of a pending approval: needs a real reason and a reference to the evidence, and is recorded as such").requiredOption("--reason ", "at least 10 characters").requiredOption("--evidence ", "ticket or incident").option("--by ", "who", "cli")) + .action((id: string, o: EngineFlags & { reason: string; evidence: string; by: string }) => + ctx.withEngine(o, (e) => { + const r = e.approvals.breakGlass(id, o.by, { reason: o.reason, evidence: o.evidence }, { roles: ["admin"], groups: [] }); + io.out(`${r.id} approved by break-glass (${o.by}); evidence ${o.evidence}`); + }), + ); + + // ---- halts ---- + const halt = program.command("halt").description("scoped kill switches (shared by every process using the same data store)"); + ctx + .dataOpts(halt.command("add ").description("block new calls, everywhere or narrowed by tool, actor, tenant or environment").option("--tool ").option("--actor ").option("--tenant ").option("--environment ").option("--by ", "who", "cli")) + .action((reason: string, o: EngineFlags & { tool?: string; actor?: string; tenant?: string; environment?: string; by: string }) => + ctx.withEngine(o, (e) => { + const h = e.reversi.halt(reason, { tool: o.tool, actor: o.actor, tenant: o.tenant, environment: o.environment } as never, o.by); + io.out(`${h.id} ${reason}${Object.keys(h.scope).length ? ` (${JSON.stringify(h.scope)})` : " (everything)"}`); + }), + ); + ctx.dataOpts(halt.command("list")).action((o: EngineFlags) => ctx.withEngine(o, (e) => { + for (const h of e.reversi.halts()) io.out(`${h.id} ${h.reason} scope=${JSON.stringify(h.scope)} by ${h.by}`); + if (!e.reversi.halts().length) io.out("no halts"); + })); + ctx.dataOpts(halt.command("lift [id]").description("lift one halt, or all of them")).action((id: string | undefined, o: EngineFlags) => ctx.withEngine(o, (e) => { + io.out(`lifted ${e.reversi.resume(id)}`); + })); + + void writeFileSync; +} diff --git a/packages/cli/src/commands/inspect.ts b/packages/cli/src/commands/inspect.ts new file mode 100644 index 0000000..41e5782 --- /dev/null +++ b/packages/cli/src/commands/inspect.ts @@ -0,0 +1,182 @@ +/** Looking at what happened: runs, actions, ledger search, and the periodic report. */ +import { writeFileSync } from "node:fs"; +import type { Command } from "commander"; +import { + buildPeriodReport, + parseDuration, + renderPeriodHtml, + renderPeriodMarkdown, + searchRecords, + searchRuns, + type EngineHandle, + type LedgerEvent, + type RunStatus, +} from "@multimindlab/reversi"; +import { Fail, type Ctx, type EngineFlags } from "../context.js"; + +/** "7d", "12h", "30m", an ISO date, or epoch ms -> a point in time. Durations count back from now. */ +export function sinceMs(v: string | undefined, now = Date.now()): number | undefined { + if (v === undefined) return undefined; + const d = parseDuration(v); + if (d !== undefined && /^\d+(\.\d+)?\s*(ms|s|m|h|d)$/.test(v.trim())) return now - d; + const t = Date.parse(v); + if (Number.isFinite(t)) return t; + if (/^\d{10,}$/.test(v)) return Number(v); + throw new Fail(`cannot read "${v}" as a time: use a duration like 7d, a date like 2026-10-01, or epoch milliseconds`); +} + +const when = (ms: number) => new Date(ms).toISOString().replace("T", " ").slice(0, 19) + "Z"; + +export function registerInspectCommands(program: Command, ctx: Ctx): void { + const { io } = ctx; + + const runs = program.command("runs").description("list runs, or show one"); + ctx + .dataOpts( + runs + .command("list", { isDefault: true }) + .description("list runs, newest last") + .option("--actor ") + .option("--tenant ") + .option("--tool ") + .option("--status ", "needs_attention | rolled_back | complete | blocked_only | empty") + .option("--mode ", "enforce | observe | advise") + .option("--since ", "7d, 2026-10-01, ...") + .option("--until ") + .option("--text ", "part of the run id") + .option("--limit ", "how many", "100") + .option("--json", "machine-readable"), + ) + .action((o: EngineFlags & Record & { json?: boolean }) => + ctx.withEngine(o, (e) => { + const r = searchRuns(e.reversi, { actor: o.actor, tenant: o.tenant, tool: o.tool, status: o.status as RunStatus | undefined, mode: o.mode, since_ms: sinceMs(o.since), until_ms: sinceMs(o.until), text: o.text, limit: Number(o.limit) }); + if (o.json) return io.out(JSON.stringify(r, null, 2)); + for (const s of r.runs) io.out(`${s.run_id} actor=${s.actor} mode=${s.mode} steps=${s.steps} ${s.status}${s.tenant ? ` tenant=${s.tenant}` : ""}${s.unresolved.length ? ` UNRESOLVED=${s.unresolved.join(",")}` : ""}`); + if (r.next) io.out(`(more: --text or narrower filters; the next page starts after ${r.next})`); + }), + ); + + ctx + .dataOpts(runs.command("show ").description("one run: steps, rollback outlook, approvals and the event trail").option("--json")) + .action((id: string, o: EngineFlags & { json?: boolean }) => + ctx.withEngine(o, (e) => { + const run = e.reversi.openRun(id); + const st = run.state(); + const plan = run.plan(); + const recs = e.reversi.ledger.forRun(id); + const approvals = e.approvals.list({ run_id: id }); + if (o.json) return io.out(JSON.stringify({ state: st, plan, approvals, records: recs }, null, 2)); + io.out(`run ${id} actor=${st.actor} mode=${st.mode}${st.tenant ? ` tenant=${st.tenant}` : ""} started ${when(st.created_ms)}`); + io.out(`scope: ${st.scope.permissions.join(" ")}`); + io.out(""); + io.out("steps"); + for (const s of st.steps) io.out(` ${String(s.step).padStart(3)} ${s.tool.padEnd(28)} ${(s.effect ?? "no manifest").padEnd(12)} ${s.status}${s.risk_score !== undefined ? ` risk ${s.risk_score}` : ""}${s.error ? ` error: ${s.error}` : ""}`); + if (st.steps.length === 0) io.out(" (none)"); + io.out(""); + io.out(plan.fully_reversible ? "rollback: fully reversible (assuming each compensation succeeds)" : "rollback: NOT fully reversible"); + for (const a of plan.actions) io.out(` step ${a.step} ${a.tool} ${a.kind}${a.compensation_tool ? ` -> ${a.compensation_tool}` : ""}${a.residue?.length ? ` (residue: ${a.residue.join("; ")})` : ""}${a.note ? ` ${a.note}` : ""}`); + if (approvals.length) { + io.out(""); + io.out("approvals"); + for (const a of approvals) io.out(` ${a.id} ${a.status} ${a.kind} ${a.tool} ${a.decided_by ? `by ${a.decided_by}` : "waiting"}${a.quorum && a.quorum > 1 ? ` (${(a.decisions ?? []).filter((d) => d.decision === "approve").length}/${a.quorum})` : ""}`); + } + io.out(""); + io.out("events"); + for (const r of recs) io.out(` ${String(r.seq).padStart(5)} ${when(r.ts_ms)} ${r.event.padEnd(19)} ${r.tool} by ${r.actor}${r.detail ? ` | ${r.detail.slice(0, 100)}` : ""}`); + }), + ); + + const actions = program.command("actions").description("individual tool calls across runs"); + ctx + .dataOpts( + actions + .command("list", { isDefault: true }) + .description("steps across runs") + .option("--run ") + .option("--tool ") + .option("--actor ") + .option("--status ", "succeeded | failed | uncertain | intended | compensated | ...") + .option("--since ") + .option("--limit ", "how many", "200") + .option("--json"), + ) + .action((o: EngineFlags & Record & { json?: boolean }) => + ctx.withEngine(o, (e) => { + const since = sinceMs(o.since); + const rows: Array<{ run: string; step: number; tool: string; actor: string; effect: string | null; status: string; ts_ms: number; risk?: number }> = []; + for (const id of o.run ? [o.run] : e.reversi.runIds().sort()) { + const st = e.reversi.openRun(id).state(); + if (o.actor && st.actor !== o.actor) continue; + for (const s of st.steps) if ((!o.tool || s.tool === o.tool) && (!o.status || s.status === o.status) && (!since || s.ts_ms >= since)) rows.push({ run: id, step: s.step, tool: s.tool, actor: st.actor, effect: s.effect, status: s.status, ts_ms: s.ts_ms, risk: s.risk_score }); + } + const out = rows.sort((a, b) => a.ts_ms - b.ts_ms).slice(-Number(o.limit)); + if (o.json) return io.out(JSON.stringify(out, null, 2)); + for (const r of out) io.out(`${when(r.ts_ms)} ${r.run}#${r.step} ${r.tool.padEnd(28)} ${(r.effect ?? "no manifest").padEnd(12)} ${r.status.padEnd(18)} ${r.actor}${r.risk !== undefined ? ` risk ${r.risk}` : ""}`); + if (out.length === 0) io.out("no matching actions"); + }), + ); + ctx + .dataOpts(actions.command("show ").description("one call: its state, its ledger records and, if capture is on, its redacted arguments").option("--json")) + .action((id: string, step: string, o: EngineFlags & { json?: boolean }) => + ctx.withEngine(o, (e) => { + const n = Number(step); + const st = e.reversi.openRun(id).state(); + const s = st.steps.find((x) => x.step === n); + if (!s) throw new Fail(`run ${id} has no step ${step}`); + const recs = e.reversi.ledger.forRun(id).filter((r) => r.step === n); + const cap = e.capture?.get(id, n); + const approvals = e.approvals.list({ run_id: id }).filter((a) => a.tool === s.tool || a.tool === s.compensation_tool); + if (o.json) return io.out(JSON.stringify({ step: s, records: recs, captured: cap, approvals }, null, 2)); + io.out(`${id} step ${n}: ${s.tool} [${s.effect ?? "no manifest"}] ${s.status}${s.error ? ` error: ${s.error}` : ""}`); + io.out(`why: ${s.reason}`); + if (s.manifest_version) io.out(`manifest v${s.manifest_version}${s.risk_score !== undefined ? `, risk ${s.risk_score}` : ""}`); + if (s.compensation_tool) io.out(`undo: ${s.compensation_tool} (${s.compensation_kind ?? "compensation"})${s.residue?.length ? `; leaves: ${s.residue.join("; ")}` : ""}${s.verification ? `; verified by reading ${s.verification.tool}` : ""}`); + if (s.handle) io.out(`undo arguments: ${JSON.stringify(s.handle)}`); + if (s.result_hash) io.out(`result hash: ${s.result_hash}`); + io.out(cap ? `captured arguments (redacted): ${JSON.stringify(cap.args)}` : "arguments are not stored (only their hash is in the ledger); turn on capture to keep redacted copies"); + for (const r of recs) io.out(` ${String(r.seq).padStart(5)} ${when(r.ts_ms)} ${r.event.padEnd(19)} by ${r.actor}${r.detail ? ` | ${r.detail}` : ""}`); + for (const a of approvals) io.out(` approval ${a.id}: ${a.status}${a.decided_by ? ` by ${a.decided_by}` : ""}`); + }), + ); + + ctx + .storeOpts( + program + .command("search") + .description("search ledger records") + .option("-m, --manifests ", "(only needed to label tools)") + .option("--run ") + .option("--actor ") + .option("--tool ") + .option("--event ", "one event name, or several separated by commas") + .option("--text ", "words in the reason or detail") + .option("--since ") + .option("--until ") + .option("--limit ", "how many", "200") + .option("--json"), + ) + .action((o: EngineFlags & Record & { json?: boolean }) => + ctx.withEngine({ ...o, manifests: o.manifests ?? ctx.project(o)?.manifests ?? process.env.REVERSI_MANIFESTS }, (e) => { + const r = searchRecords(e.reversi, { run_id: o.run, actor: o.actor, tool: o.tool, event: o.event ? (o.event.split(",") as LedgerEvent[]) : undefined, text: o.text, since_ms: sinceMs(o.since), until_ms: sinceMs(o.until), limit: Number(o.limit) }); + if (o.json) return io.out(JSON.stringify(r.records, null, 2)); + for (const x of r.records) io.out(`${String(x.seq).padStart(6)} ${when(x.ts_ms)} ${x.run_id}${x.step !== null ? `#${x.step}` : ""} ${x.event.padEnd(19)} ${x.tool} ${x.actor}${x.detail ? ` | ${x.detail.slice(0, 90)}` : ""}`); + if (r.records.length === 0) io.out("no matching records"); + }), + ); + + // the periodic report is `reversi report` with no run id + void ((e: EngineHandle) => e); +} + +export async function periodReportCommand(ctx: Ctx, o: EngineFlags & { since?: string; format?: string; out?: string }): Promise { + await ctx.withEngine(o, (e) => { + const from = sinceMs(o.since ?? "7d") as number; + const rep = buildPeriodReport(e.reversi, e.approvals, from); + const fmt = o.format ?? (o.out?.endsWith(".html") ? "html" : o.out?.endsWith(".json") ? "json" : "md"); + const text = fmt === "json" ? JSON.stringify(rep, null, 2) : fmt === "html" ? renderPeriodHtml(rep) : renderPeriodMarkdown(rep); + if (o.out) { + writeFileSync(o.out, text); + ctx.io.out(`wrote ${o.out}`); + } else ctx.io.out(text.replace(/\n$/, "")); + }); +} diff --git a/packages/cli/src/commands/manifests.ts b/packages/cli/src/commands/manifests.ts new file mode 100644 index 0000000..f94859b --- /dev/null +++ b/packages/cli/src/commands/manifests.ts @@ -0,0 +1,178 @@ +/** The manifest registry and versioned policies: publish, approve, deprecate, history, diff, roll back, verify. */ +import { createPrivateKey, createPublicKey } from "node:crypto"; +import { existsSync, readFileSync, readdirSync, statSync } from "node:fs"; +import { userInfo } from "node:os"; +import { extname, join } from "node:path"; +import type { Command } from "commander"; +import { parse as parseYaml } from "yaml"; +import { ManifestRegistry, PolicyVersions, RegistryError, loadRegistry, type SigningKey } from "@multimindlab/reversi"; +import { Fail, type Ctx } from "../context.js"; + +const me = () => { + try { + return userInfo().username; + } catch { + return "cli"; + } +}; + +function loadKey(path: string | undefined): SigningKey | undefined { + if (!path) return undefined; + const private_pem = readFileSync(path, "utf8"); + return { private_pem, public_pem: createPublicKey(createPrivateKey(private_pem)).export({ type: "spki", format: "pem" }).toString() }; +} + +export function manifestFiles(path: string): string[] { + const st = statSync(path); + if (st.isFile()) return [path]; + return readdirSync(path) + .filter((f) => /\.(ya?ml|json)$/i.test(f)) + .sort() + .map((f) => join(path, f)); +} + +export function readManifests(path: string): unknown[] { + return manifestFiles(path).flatMap((f) => { + const text = readFileSync(f, "utf8"); + const doc = extname(f).toLowerCase() === ".json" ? JSON.parse(text) : parseYaml(text); + return Array.isArray(doc) ? doc : [doc]; + }); +} + +export function registerManifestCommands(program: Command, ctx: Ctx): void { + const { io } = ctx; + const registryDir = (o: { registry?: string }): string => { + const dir = o.registry ?? ctx.project()?.manifests; + if (!dir) throw new Fail("no registry folder: pass --registry , or set `manifests:` to a registry folder in reversi.config.yaml"); + if (existsSync(dir) && statSync(dir).isDirectory() && readdirSync(dir).length > 0 && !ManifestRegistry.isRegistry(dir)) { + throw new Fail(`${dir} holds manifest files but is not a registry: publish them with \`reversi manifests publish\` into an empty or registry folder`); + } + return dir; + }; + const opened = (o: { registry?: string; trust?: string }) => + new ManifestRegistry(registryDir(o), { trusted_public_pem: o.trust ? readFileSync(o.trust, "utf8") : undefined }); + const common = (c: Command) => + c.option("--registry ", "the registry folder (default: `manifests:` in the config)").option("--by ", "who is doing this (default: your user name)").option("--key ", "signing key (from `reversi keygen`) to sign the new version").option("--note ", "why"); + + const group = program.command("manifests").description("a versioned, signable registry of manifests"); + + common(group.command("publish ").description("add manifests (a file or folder) to the registry as new versions").option("--status ", "draft | approved (default draft: someone else approves)", "draft")).action((path: string, o: { registry?: string; by?: string; key?: string; note?: string; status: string }) => { + if (o.status !== "draft" && o.status !== "approved") throw new Fail("--status must be draft or approved"); + const reg = opened(o); + const key = loadKey(o.key); + for (const m of readManifests(path)) { + const r = reg.publish(m, { by: o.by ?? me(), note: o.note, status: o.status, key }); + io.out(`${r.tool} v${r.version} ${o.status} ${r.checksum.slice(0, 12)}`); + } + }); + + common(group.command("approve ").description("promote a draft to approved (a new, signed version)").option("--version ", "the draft version (default: the latest)").option("--four-eyes", "the author may not approve their own draft")).action((tool: string, o: { registry?: string; by?: string; key?: string; note?: string; version?: string; fourEyes?: boolean }) => { + const r = opened(o).approve(tool, { by: o.by ?? me(), version: o.version ? Number(o.version) : undefined, note: o.note, key: loadKey(o.key), require_other: o.fourEyes }); + io.out(`${r.tool} v${r.version} approved by ${o.by ?? me()}`); + }); + + common(group.command("deprecate ").description("keep a tool working and rolling back, but flag new calls and point to its replacement").option("--replaced-by ")).action((tool: string, o: { registry?: string; by?: string; key?: string; note?: string; replacedBy?: string }) => { + const r = opened(o).deprecate(tool, { by: o.by ?? me(), replaced_by: o.replacedBy, note: o.note, key: loadKey(o.key) }); + io.out(`${r.tool} v${r.version} deprecated${o.replacedBy ? `; use ${o.replacedBy}` : ""}`); + }); + + common(group.command("rollback ").description("publish an old version again as the newest")).action((tool: string, version: string, o: { registry?: string; by?: string; key?: string; note?: string }) => { + const r = opened(o).rollbackTo(tool, Number(version), { by: o.by ?? me(), key: loadKey(o.key), note: o.note }); + io.out(`${r.tool} v${r.version} (restored from v${version})`); + }); + + group + .command("history ") + .option("--registry ") + .description("every version of a tool") + .action((tool: string, o: { registry?: string }) => { + for (const v of opened(o).history(tool)) io.out(`v${v.version} ${v.status.padEnd(10)} ${new Date(v.created_ms).toISOString().slice(0, 19)}Z by ${v.by}${v.approved_by ? `, approved by ${v.approved_by}` : ""} ${v.checksum.slice(0, 12)}${v.signature ? " signed" : ""}${v.note ? ` - ${v.note}` : ""}`); + }); + + group + .command("diff ") + .option("--registry ") + .description("what changed between two versions") + .action((tool: string, a: string, b: string, o: { registry?: string }) => { + const d = opened(o).diff(tool, Number(a), Number(b)); + io.out(d || "(no differences)"); + }); + + group + .command("list") + .option("--registry ") + .description("the tools in the registry and their current versions") + .action((o: { registry?: string }) => { + const r = opened(o); + for (const t of r.tools()) { + const h = r.history(t); + const last = h[h.length - 1]; + io.out(`${t.padEnd(40)} v${last?.version} ${last?.status}`); + } + if (r.tools().length === 0) io.out("(empty)"); + }); + + group + .command("verify") + .option("--registry ") + .option("--trust ", "public key every version must be signed by") + .description("check every version's checksum (and signature, with --trust)") + .action((o: { registry?: string; trust?: string }) => { + const problems = opened(o).verifyAll(); + if (problems.length) { + for (const p of problems) io.err(`BAD ${p}`); + ctx.code = 1; + } else io.out(`ok: ${opened(o).tools().length} tools, every version verifies`); + }); + + group + .command("check ") + .description("validate manifests in a file or folder (same as `reversi validate`)") + .action((path: string) => { + const reg = loadRegistry(path); + io.out(`ok: ${reg.size} tool manifests valid`); + }); + + // ---- policy versions, added to the existing `policy` group ---- + const policy = program.commands.find((c) => c.name() === "policy") ?? program.command("policy").description("policy as code"); + const pdir = (o: { dir?: string }) => { + const p = o.dir ?? ctx.project()?.policy; + if (!p) throw new Fail("no policy folder: pass --dir , or point `policy:` at a folder in reversi.config.yaml"); + if (existsSync(p) && statSync(p).isFile()) throw new Fail(`${p} is a single policy file; versions live in a folder`); + return new PolicyVersions(p); + }; + policy + .command("publish ") + .description("validate a policy and its tests, store it as a new version, and activate it") + .option("--dir ", "the policy versions folder (default: `policy:` in the config)") + .option("--by ") + .option("--note ") + .option("--no-activate", "store it without making it the active policy") + .action((file: string, o: { dir?: string; by?: string; note?: string; activate: boolean }) => { + const v = pdir(o).publish(readFileSync(file, "utf8"), { by: o.by ?? me(), note: o.note, activate: o.activate }); + io.out(`policy v${v.version} ${v.tests_passed} tests passed${o.activate ? " active" : ""} ${v.checksum.slice(0, 12)}`); + }); + policy + .command("history") + .option("--dir ") + .description("every policy version") + .action((o: { dir?: string }) => { + const pv = pdir(o); + for (const v of pv.history()) io.out(`v${v.version}${pv.activeVersion() === v.version ? " *active*" : " "} ${new Date(v.created_ms).toISOString().slice(0, 19)}Z by ${v.by} ${v.tests_passed} tests ${v.checksum.slice(0, 12)}${v.note ? ` - ${v.note}` : ""}`); + }); + policy + .command("diff ") + .option("--dir ") + .description("what changed between two policy versions") + .action((a: string, b: string, o: { dir?: string }) => io.out(pdir(o).diff(Number(a), Number(b)) || "(no differences)")); + policy + .command("activate ") + .option("--dir ") + .description("make a version the active policy (this is also how you roll back)") + .action((v: string, o: { dir?: string }) => { + pdir(o).activate(Number(v)); + io.out(`policy v${v} is now active`); + }); +} + +export { RegistryError }; diff --git a/packages/cli/src/commands/project.ts b/packages/cli/src/commands/project.ts new file mode 100644 index 0000000..27c0a18 --- /dev/null +++ b/packages/cli/src/commands/project.ts @@ -0,0 +1,338 @@ +/** init, doctor, lint, test: getting a project set up and keeping its manifests honest. */ +import { existsSync, mkdirSync, readFileSync, readdirSync, statSync, writeFileSync } from "node:fs"; +import { dirname, join, resolve } from "node:path"; +import type { Command } from "commander"; +import { stringify } from "yaml"; +import { + ManifestRegistry, + Notifier, + PolicyVersions, + createChannel, + checkUrl, + hasErrors, + lintManifests, + loadPolicyFile, + loadRegistry, + openData, + runManifestTests, + runPolicyTests, + starterConfig, + type ManifestTestCase, + type StoreKind, +} from "@multimindlab/reversi"; +import { Fail, type Ctx, type EngineFlags } from "../context.js"; +import { fetchUpstreamTools, hasUpstream, upstreamOpts, type UpstreamFlags } from "./upstream.js"; + +const EXAMPLE_MANIFESTS = `# One manifest per tool: how reversible it is, what undoes it, who may use it. +# Replace these two with your own (reversi suggest-manifest drafts them from an MCP server). +- tool: notes.read + description: Read a note. + effect: read_only + permission: notes:read + +- tool: notes.write + description: Create or overwrite a note. The previous text is kept so it can be restored. + effect: reversible + permission: notes:write + owner: you@example.com + risk: { data_class: internal, blast_radius: low } + timeout_ms: 30s + compensation: + tool: notes.restore + kind: inverse + argument_map: + name: $args.name + backup: $result.backup_id + verification: + tool: notes.read + arguments: { name: $args.name } + expect: { restored: true } + +- tool: notes.restore + description: Put a note back the way a backup says it was. + effect: idempotent + permission: notes:write +`; + +const EXAMPLE_POLICY = `version: 1 +# Rules add restrictions on top of the manifests; nothing here can grant a permission the run's scope lacks. +rules: + - id: no-destructive-names + description: anything called drop, wipe or purge needs a person + match: { tool: ["*.drop*", "*.wipe*", "*.purge*"] } + action: require_approval + reason: destructive name +limits: + - id: runaway-guard + match: {} + max_calls: 200 + window_seconds: 60 + per: run +content: + secrets: flag + encoded_payloads: require_approval + injection: require_approval +tests: + - { name: ordinary tools are allowed, input: { tool: notes.write }, expect: allow } + - { name: drop needs approval, input: { tool: db.drop_table }, expect: require_approval, rule: no-destructive-names } +`; + +const EXAMPLE_TESTS = `# Contract tests for manifests: run with \`reversi test\`. +- name: writing a note can be undone, and the undo is read back + tool: notes.write + args: { name: todo } + result: { backup_id: b1 } + expect_plan: compensate + expect_handle: { name: todo, backup: b1 } + verify_with: { restored: true } + expect_verified: true +- name: a failed write leaves nothing to undo + tool: notes.write + args: { name: todo } + fail: disk full +`; + +type Check = { level: "ok" | "warn" | "fail" | "info"; text: string }; + +export function registerProjectCommands(program: Command, ctx: Ctx): void { + const { io } = ctx; + + program + .command("init [dir]") + .description("create a Reversi project: config, example manifests, a starter policy, tests and a data store") + .option("--store ", "files | sqlite", "sqlite") + .option("--force", "overwrite files that already exist") + .action(async (dir: string | undefined, o: { store: string; force?: boolean }) => { + const root = resolve(dir ?? "."); + if (!["files", "sqlite"].includes(o.store)) throw new Fail("--store must be files or sqlite (use postgres by editing reversi.config.yaml)"); + mkdirSync(root, { recursive: true }); + const made: string[] = []; + const put = (rel: string, text: string) => { + const f = join(root, rel); + if (existsSync(f) && !o.force) { + io.out(`kept ${rel} (exists; --force to overwrite)`); + return; + } + mkdirSync(dirname(f), { recursive: true }); + writeFileSync(f, text); + made.push(rel); + io.out(`created ${rel}`); + }; + put("reversi.config.yaml", starterConfig({ store: o.store as StoreKind })); + put("manifests/example.yaml", EXAMPLE_MANIFESTS); + put("policy.yaml", EXAMPLE_POLICY); + put("tests/manifests.test.yaml", EXAMPLE_TESTS); + const ignore = join(root, ".gitignore"); + const line = "reversi-data/\n"; + if (!existsSync(ignore) || !readFileSync(ignore, "utf8").includes("reversi-data")) { + writeFileSync(ignore, (existsSync(ignore) ? readFileSync(ignore, "utf8").replace(/\n?$/, "\n") : "") + line); + io.out("updated .gitignore (reversi-data/)"); + } + const data = await openData({ dir: join(root, "reversi-data"), store: o.store as StoreKind }); + data.close(); + io.out(`created reversi-data/ (${o.store} store)`); + loadRegistry(join(root, "manifests")); + io.out(""); + io.out("Next:"); + io.out(" reversi doctor check the setup"); + io.out(" reversi test run the example contract and policy tests"); + io.out(" reversi suggest-manifest --upstream-cmd draft manifests from a real server"); + io.out(" reversi gateway --upstream-cmd --mode observe see what an agent does before enforcing anything"); + }); + + upstreamOpts( + ctx.dataOpts( + program + .command("doctor") + .description("check the setup: config, manifests, policy, data store, keys, notifications, network rules and (optionally) an upstream") + .option("--policy ", "policy file or folder") + .option("--send-test", "send a test message through every configured notification channel"), + ), + ).action(async (o: EngineFlags & UpstreamFlags & { sendTest?: boolean }) => { + const checks: Check[] = []; + const add = (level: Check["level"], text: string) => checks.push({ level, text }); + const [maj, min] = process.versions.node.split(".").map(Number) as [number, number]; + if (maj > 22 || (maj === 22 && (min ?? 0) >= 13)) add("ok", `Node ${process.versions.node}`); + else add("fail", `Node ${process.versions.node}: Reversi needs 22.13 or newer`); + + let project; + try { + project = ctx.project(o); + add(project ? "ok" : "info", project ? `config ${project.file}` : "no reversi.config.yaml (defaults and environment variables are used; `reversi init` creates one)"); + } catch (e) { + add("fail", `config: ${e instanceof Error ? e.message : String(e)}`); + } + + let registryTools = 0; + try { + const path = ctx.manifestsPath(o); + if (!/^https?:\/\//.test(path)) { + if (!existsSync(path)) throw new Error(`${path} does not exist`); + if (statSync(path).isDirectory() && ManifestRegistry.isRegistry(path)) { + const problems = new ManifestRegistry(path).verifyAll(); + add(problems.length ? "fail" : "ok", problems.length ? `manifest registry: ${problems.join("; ")}` : `manifest registry ${path}: every version verifies`); + } + } + const reg = ctx.engineConfigOf(o).manifests; + void reg; + await ctx.withEngine(o, (e) => { + registryTools = e.registry.size; + const findings = lintManifests(e.registry); + add("ok", `${e.registry.size} manifests load`); + const errs = findings.filter((f) => f.level === "error").length; + const warns = findings.filter((f) => f.level === "warning").length; + if (errs) add("fail", `lint: ${errs} errors (reversi lint)`); + else if (warns) add("warn", `lint: ${warns} warnings (reversi lint)`); + else add("ok", "lint: clean"); + add("ok", `data store (${e.data.kind}) opens`); + const v = e.reversi.ledger.verify(); + add(v.ok ? "ok" : "fail", v.ok ? `ledger: ${v.records} records, chain intact` : `ledger: TAMPERED or corrupt at record ${v.seq}: ${v.error}`); + const un = e.reversi.runIds().filter((id) => e.reversi.openRun(id).unresolved().length); + if (un.length) add("warn", `${un.length} runs have steps with an unknown outcome (reversi runs list --status needs_attention)`); + const halted = e.reversi.halted(); + if (halted) add("warn", `new calls are halted: ${halted}`); + if (e.signing_key) add("ok", "signing key loads"); + else add("info", "no signing key: signed evidence bundles and anchors are off"); + if (e.config.policy) { + try { + const f = existsSync(e.config.policy) && statSync(e.config.policy).isDirectory() ? undefined : loadPolicyFile(e.config.policy); + if (f) { + const results = runPolicyTests(f.doc); + const bad = results.filter((r) => !r.ok).length; + add(bad ? "fail" : "ok", `policy: ${f.doc.rules.length} rules, ${results.length} tests${bad ? `, ${bad} failing` : ""}`); + } else add("ok", `policy versions in ${e.config.policy}`); + } catch (err) { + add("fail", `policy: ${err instanceof Error ? err.message : String(err)}`); + } + } else add("info", "no policy file: only manifests and scopes restrict calls"); + if (e.notifier) add("ok", `notifications: ${e.notifier.channelNames().join(", ")}`); + else add("info", "no notification channels configured"); + if (e.capture) add("ok", "capture is on (redacted arguments are kept for replay and simulation)"); + return undefined; + }); + } catch (e) { + add("fail", `engine: ${e instanceof Error ? e.message : String(e)}`); + } + + // data directory writable + try { + const d = ctx.dataOptionsOf(o); + if (d.store !== "postgres") { + mkdirSync(d.dir, { recursive: true }); + const probe = join(d.dir, `.doctor-${process.pid}`); + writeFileSync(probe, ""); + (await import("node:fs")).unlinkSync(probe); + add("ok", `data directory ${d.dir} is writable`); + } + } catch (e) { + add("fail", `data directory: ${e instanceof Error ? e.message : String(e)}`); + } + + // network destinations + const proj = project; + const urls = [...(proj?.notifications?.channels ?? []).flatMap((c) => ("url" in c && c.url ? [c.url] : [])), proj?.otel?.endpoint, proj?.approval?.webhook].filter((u): u is string => !!u); + for (const u of urls) { + try { + await checkUrl(proj?.egress, u); + add("ok", `egress allows ${new URL(u).host}`); + } catch (e) { + add("fail", `egress: ${e instanceof Error ? e.message : String(e)}`); + } + } + if (o.sendTest && proj?.notifications?.channels.length) { + const n = new Notifier(proj.notifications.channels.map((c) => createChannel(c, { egress: proj.egress })), [{ events: ["*"], channels: proj.notifications.channels.map((c) => c.name) }]); + const r = await n.notify({ event: "doctor_test", severity: "info", title: "Reversi test notification", text: "If you can read this, the channel works." }); + add(r.failed.length ? "fail" : "ok", `test message: sent to ${r.sent.join(", ") || "nobody"}${r.failed.length ? `; failed: ${r.failed.join(", ")}` : ""}`); + } + + if (hasUpstream(o)) { + try { + const { tools } = await fetchUpstreamTools(o); + add("ok", `upstream answers: ${tools.length} tools`); + if (registryTools) { + const reg = loadRegistry(ctx.manifestsPath(o)); + const missing = tools.filter((t) => !reg.has(t.name)).length; + add(missing ? "warn" : "ok", missing ? `${missing} upstream tools have no manifest (reversi suggest-manifest)` : "every upstream tool has a manifest"); + } + } catch (e) { + add("fail", `upstream: ${e instanceof Error ? e.message : String(e)}`); + } + } + + const sym = { ok: "ok ", warn: "warn", fail: "FAIL", info: "info" } as const; + for (const c of checks) io.out(`${sym[c.level]} ${c.text}`); + const fails = checks.filter((c) => c.level === "fail").length; + const warns = checks.filter((c) => c.level === "warn").length; + io.out(fails ? `${fails} problem${fails === 1 ? "" : "s"} to fix` : warns ? `ok, with ${warns} warning${warns === 1 ? "" : "s"}` : "everything checks out"); + if (fails) ctx.code = 1; + }); + + upstreamOpts( + program + .command("lint [path]") + .description("find unsafe, incomplete or contradictory manifests; with an upstream, also check them against its real tools") + .option("--strict", "exit 1 on warnings too") + .option("--json", "machine-readable findings"), + ).action(async (path: string | undefined, o: UpstreamFlags & { strict?: boolean; json?: boolean }) => { + const reg = loadRegistry(path ?? ctx.manifestsPath({})); + const tools = hasUpstream(o) ? (await fetchUpstreamTools(o)).tools : undefined; + const findings = lintManifests(reg, { tools }); + if (o.json) io.out(JSON.stringify(findings, null, 2)); + else { + for (const f of findings) io.out(`${f.level.padEnd(7)} ${f.code} ${f.tool}: ${f.message}`); + const n = (l: string) => findings.filter((f) => f.level === l).length; + io.out(findings.length ? `${n("error")} errors, ${n("warning")} warnings, ${n("info")} notes` : `ok: ${reg.size} manifests, no findings`); + } + if (hasErrors(findings) || (o.strict && findings.some((f) => f.level === "warning"))) ctx.code = 1; + }); + + ctx + .dataOpts(program.command("test [fixtures...]").description("run contract tests for manifests, and the policy's own tests, without touching any system").option("--policy ", "policy file")) + .action(async (fixtures: string[], o: EngineFlags) => { + const reg = loadRegistry(ctx.manifestsPath(o)); + const root = ctx.project(o)?.root ?? process.cwd(); + let files = fixtures; + if (files.length === 0) { + const dir = join(root, "tests"); + files = existsSync(dir) ? readdirSync(dir).filter((f) => /\.(test\.)?(ya?ml|json)$/.test(f)).map((f) => join(dir, f)) : []; + } + const cases: ManifestTestCase[] = []; + for (const f of files) { + const doc = ctx.readDoc(f); + const list = Array.isArray(doc) ? doc : (doc as { manifests?: unknown[] })?.manifests; + if (!Array.isArray(list)) throw new Fail(`${f}: expected a list of test cases`); + cases.push(...(list as ManifestTestCase[])); + } + let failed = 0; + let policyDoc; + const polPath = o.policy ?? ctx.project(o)?.policy; + if (polPath && existsSync(polPath) && !statSync(polPath).isDirectory()) policyDoc = loadPolicyFile(polPath).doc; + else if (polPath && existsSync(polPath) && PolicyVersions.isVersioned(polPath)) policyDoc = new PolicyVersions(polPath).doc(new PolicyVersions(polPath).activeVersion() as number); + const { results, untested } = await runManifestTests(reg, cases, { policy: policyDoc, environment: ctx.project(o)?.environment }); + for (const r of results) { + io.out(`${r.ok ? "ok " : "FAIL"} ${r.name}`); + for (const p of r.problems) io.out(` ${p}`); + if (!r.ok) failed++; + } + if (policyDoc) { + const pr = runPolicyTests(policyDoc); + for (const r of pr) { + io.out(`${r.ok ? "ok " : "FAIL"} policy: ${r.name}${r.ok ? "" : `: expected ${r.expected}, got ${r.got}`}`); + if (!r.ok) failed++; + } + results.push(...pr.map((r) => ({ name: r.name, tool: "", ok: r.ok, problems: [] }))); + } + if (results.length === 0) io.out("no tests found: add files under tests/ or pass them as arguments"); + if (untested.length) io.out(`not covered by any test: ${untested.join(", ")}`); + io.out(failed ? `${failed} of ${results.length} failed` : `${results.length} passed`); + if (failed) ctx.code = 1; + }); + + program + .command("show-config") + .description("print the project configuration Reversi found, with defaults filled in") + .action(() => { + const p = ctx.project({}); + io.out(p ? stringify(p) : "no reversi.config.yaml found"); + }); +} diff --git a/packages/cli/src/commands/serve.ts b/packages/cli/src/commands/serve.ts new file mode 100644 index 0000000..90fd97f --- /dev/null +++ b/packages/cli/src/commands/serve.ts @@ -0,0 +1,73 @@ +/** Starting things from the CLI: draft manifests from a server, run the gateway, open the console on your machine. */ +import { spawn } from "node:child_process"; +import { existsSync, mkdirSync, writeFileSync } from "node:fs"; +import { dirname, join } from "node:path"; +import { fileURLToPath } from "node:url"; +import { serve } from "@hono/node-server"; +import type { Command } from "commander"; +import { stringify } from "yaml"; +import { Registry, suggestManifests } from "@multimindlab/reversi"; +import { createApp } from "@multimindlab/reversi-server"; +import { Fail, type Ctx } from "../context.js"; +import { fetchUpstreamTools, hasUpstream, upstreamOpts } from "./upstream.js"; +import { loadRegistry } from "@multimindlab/reversi"; + +export function registerServeCommands(program: Command, ctx: Ctx): void { + const { io } = ctx; + + upstreamOpts( + program + .command("suggest-manifest") + .description("connect to an MCP server and draft a manifest for every tool that has none. Drafts need a person's approval before they are enforced") + .option("-m, --manifests ", "existing manifests: tools that already have one are skipped") + .option("-o, --out ", "write one YAML file per tool here (default: print them)") + .option("--server ", "name used for permissions, like :read"), + ).action(async (o) => { + if (!hasUpstream(o)) throw new Fail("give --upstream-cmd, --upstream-url or --gateway-config"); + const { tools, server } = await fetchUpstreamTools(o); + let existing: Registry | undefined; + const mpath = o.manifests ?? process.env.REVERSI_MANIFESTS ?? ctx.project()?.manifests; + if (mpath && existsSync(mpath)) existing = loadRegistry(mpath); + const r = suggestManifests(tools, { server: o.server ?? server, existing }); + if (o.out) mkdirSync(o.out, { recursive: true }); + for (const s of r.suggestions) { + const text = stringify(s.manifest, { lineWidth: 100 }); + const name = String(s.manifest.tool); + if (o.out) { + const file = join(o.out, `${name.replace(/[^A-Za-z0-9._-]/g, "_")}.yaml`); + writeFileSync(file, `# DRAFT (${s.confidence} confidence). Review before enforcing.\n${s.notes.map((n) => `# - ${n}`).join("\n")}\n${text}`); + io.out(`${file} ${s.confidence}`); + } else io.out(`# ${name} (${s.confidence} confidence)\n${s.notes.map((n) => `# ${n}`).join("\n")}\n${text}---`); + } + io.err(`${r.suggestions.length} drafted, ${r.skipped.length} skipped (already have manifests)${r.ignored.length ? `, ${r.ignored.length} ignored` : ""}`); + }); + + program + .command("gateway [args...]") + .description("run the MCP gateway (same options as reversi-gateway; try `reversi gateway --help`)") + .allowUnknownOption() + .helpOption(false) + .action(async (args: string[]) => { + const main = join(dirname(fileURLToPath(import.meta.resolve("@multimindlab/reversi-gateway"))), "main.js"); + const child = spawn(process.execPath, [main, ...args], { stdio: "inherit" }); + ctx.code = await new Promise((resolve) => child.on("close", (c) => resolve(c ?? 1))); + }); + + ctx + .dataOpts(program.command("console").description("open the console on this machine: no login, loopback only, never admin. Stop it with Ctrl-C")) + .option("--policy ", "policy file") + .option("--port ", "port (default 7421)", "7421") + .option("--role ", "auditor | approver | operator (default operator)", "operator") + .action(async (o) => { + if (!["auditor", "approver", "operator"].includes(o.role)) throw new Fail("--role must be auditor, approver or operator (the open console is never admin)"); + const e = await ctx.engine(o, { allow_empty_manifests: !o.manifests && !ctx.project(o)?.manifests }); + const app = createApp({ reversi: e.reversi, approvals: e.approvals, keys: [], engine: e, signing_key: e.signing_key, console: { mode: "open", role: o.role } }); + const server = serve({ fetch: app.fetch, port: Number(o.port), hostname: "127.0.0.1" }, (i) => io.out(`console: http://localhost:${i.port}/console (open mode as ${o.role}; Ctrl-C to stop)`)); + await new Promise((resolve) => { + const stop = () => server.close(() => resolve()); + process.once("SIGINT", stop); + process.once("SIGTERM", stop); + }); + await e.close(); + }); +} diff --git a/packages/cli/src/commands/simulate.ts b/packages/cli/src/commands/simulate.ts new file mode 100644 index 0000000..3e2ce69 --- /dev/null +++ b/packages/cli/src/commands/simulate.ts @@ -0,0 +1,180 @@ +/** Capture, policy simulation against history, replay with a change, speculative plans, and impact previews. */ +import { existsSync, readFileSync, statSync } from "node:fs"; +import type { Command } from "commander"; +import { + PolicyVersions, + loadPolicyFile, + parsePolicy, + previewImpact, + replayRun, + simulatePolicy, + speculate, + type ArgChange, + type PlanInput, + type PolicyDoc, +} from "@multimindlab/reversi"; +import { Fail, type Ctx, type EngineFlags } from "../context.js"; +import { sinceMs } from "./inspect.js"; + +function loadPolicyDoc(ctx: Ctx, path: string | undefined, o: EngineFlags): PolicyDoc | undefined { + const p = path ?? o.policy ?? ctx.project(o)?.policy; + if (!p) return undefined; + if (!existsSync(p)) throw new Fail(`policy not found: ${p}`); + if (statSync(p).isDirectory()) { + const pv = new PolicyVersions(p); + const a = pv.activeVersion(); + return a === null ? undefined : pv.doc(a); + } + return loadPolicyFile(p).doc; +} + +function parseChange(spec: string): ArgChange { + // [step.]path=json e.g. 1.amount=9000 or tool:pay.charge.amount=9000 or amount=9000 + const eq = spec.indexOf("="); + if (eq < 1) throw new Fail(`--set wants path=value, like 1.amount=9000 or tool:pay.charge/amount=9000 (got "${spec}")`); + let lhs = spec.slice(0, eq); + const raw = spec.slice(eq + 1); + let value: unknown = raw; + try { + value = JSON.parse(raw); + } catch { + /* a plain string */ + } + const change: ArgChange = { path: "", value }; + const t = /^tool:([^/]+)\/(.+)$/.exec(lhs); + if (t) { + change.tool = t[1]; + lhs = t[2] as string; + } else { + const s = /^(\d+)\.(.+)$/.exec(lhs); + if (s) { + change.step = Number(s[1]); + lhs = s[2] as string; + } + } + change.path = lhs; + return change; +} + +export function registerSimulationCommands(program: Command, ctx: Ctx): void { + const { io } = ctx; + + const capture = program.command("capture").description("redacted call arguments, kept for a limited time when you turn capture on"); + ctx.dataOpts(capture.command("list").description("captured calls").option("--run ").option("--tool ").option("--since ").option("--json")).action((o: EngineFlags & { run?: string; tool?: string; since?: string; json?: boolean }) => + ctx.withEngine(o, (e) => { + if (!e.capture) return io.out("capture is off. Turn it on with `capture: { enabled: true }` in reversi.config.yaml, or REVERSI_CAPTURE=1"); + const rows = e.capture.list({ run_id: o.run, tool: o.tool, since_ms: sinceMs(o.since) }); + if (o.json) return io.out(JSON.stringify(rows, null, 2)); + for (const r of rows) io.out(`${new Date(r.ts_ms).toISOString().slice(0, 19)}Z ${r.run_id}${r.step !== null ? `#${r.step}` : ""} ${r.tool.padEnd(26)} ${r.decision.padEnd(11)} ${r.outcome ?? ""} args=${JSON.stringify(r.args).slice(0, 80)}`); + if (!rows.length) io.out("nothing captured"); + }), + ); + ctx.dataOpts(capture.command("purge").description("delete expired captures, or everything for a run").option("--run ")).action((o: EngineFlags & { run?: string }) => + ctx.withEngine(o, (e) => { + if (!e.capture) return io.out("capture is off"); + io.out(`removed ${e.capture.purge({ run_id: o.run })} captured calls`); + }, { capture: { enabled: true } }), + ); + + ctx + .dataOpts(program.command("simulate").description("replay captured calls through a different policy and report what would have been decided differently").option("--policy ", "the candidate policy (file or folder)").option("--since ", "only calls since then (default 7d)").option("--run ").option("--json")) + .action((o: EngineFlags & { since?: string; run?: string; json?: boolean }) => + ctx.withEngine(o, async (e) => { + if (!e.capture) throw new Fail("simulation needs captured calls: turn capture on first (it keeps redacted arguments for a limited time)"); + const policy = loadPolicyDoc(ctx, o.policy, {}) ?? undefined; + if (!policy) throw new Fail("pass --policy with the policy to try"); + const records = e.capture.list({ run_id: o.run, since_ms: o.run ? undefined : sinceMs(o.since ?? "7d") }); + if (!records.length) throw new Fail("no captured calls in that period"); + const rep = await simulatePolicy(records, { registry: e.registry, policy, environment: e.reversi.environment }); + if (o.json) return io.out(JSON.stringify(rep, null, 2)); + io.out(`${rep.calls} captured calls in ${rep.runs} runs: ${rep.unchanged} unchanged`); + const show = (title: string, list: typeof rep.newly_blocked) => { + if (!list.length) return; + io.out(`\n${title} (${list.length})`); + for (const c of list.slice(0, 50)) io.out(` ${c.run}${c.step !== null ? `#${c.step}` : ""} ${c.tool} ${c.before} -> ${c.after} ${c.reasons[0] ?? ""}`); + }; + show("newly blocked", rep.newly_blocked); + show("newly needing approval", rep.newly_needs_approval); + show("newly allowed", rep.newly_allowed); + for (const n of rep.notes) io.out(`note: ${n}`); + if (rep.newly_blocked.length) ctx.code = 2; + }), + ); + + ctx + .dataOpts( + program + .command("replay ") + .description("replay one captured run in a sandbox, once as recorded and once with changes, and compare") + .option("--set ", "change an argument: 1.amount=9000, tool:pay.charge/amount=9000, amount=9000 (repeatable)", (v: string, prev: string[] = []) => [...prev, v]) + .option("--policy ", "also try a different policy") + .option("--json"), + ) + .action((id: string, o: EngineFlags & { set?: string[]; json?: boolean }) => + ctx.withEngine(o, async (e) => { + if (!e.capture) throw new Fail("replay needs captured calls: turn capture on first"); + const records = e.capture.list({ run_id: id }); + if (!records.length) throw new Fail(`nothing captured for run ${id} (captures expire; capture must have been on)`); + const policy = loadPolicyDoc(ctx, o.policy, {}) ?? undefined; + const rep = await replayRun(records, { registry: e.registry, policy, environment: e.reversi.environment }, (o.set ?? []).map(parseChange)); + if (o.json) return io.out(JSON.stringify(rep, null, 2)); + io.out(`run ${id}: ${records.length} captured calls`); + for (const [i, v] of rep.variant.entries()) { + const b = rep.baseline[i]; + io.out(` ${i} ${v.tool.padEnd(28)} as recorded: ${b?.verdict} with your changes: ${v.verdict}${v.verdict !== "allow" ? ` (${v.reasons[0] ?? ""})` : ""}`); + } + io.out(rep.differences.length ? `${rep.differences.length} decision${rep.differences.length === 1 ? "" : "s"} would change` : "no decision would change"); + io.out(`rollback as recorded: ${rep.baseline_plan?.fully_reversible ? "fully reversible" : "not fully reversible"}; with changes: ${rep.variant_plan?.fully_reversible ? "fully reversible" : "not fully reversible"}`); + }), + ); + + ctx + .dataOpts(program.command("speculate ").description("compare alternative plans without running them: decisions, approvals, risk, residue and rollback for each").option("--policy ").option("--actor ").option("--json")) + .action((file: string, o: EngineFlags & { actor?: string; json?: boolean }) => + ctx.withEngine(o, async (e) => { + const doc = ctx.readDoc(file) as { plans?: PlanInput[] } | PlanInput[]; + const plans = Array.isArray(doc) ? doc : doc.plans; + if (!plans?.length) throw new Fail(`${file}: expected a list of plans, each with a name and a list of calls`); + const policy = loadPolicyDoc(ctx, o.policy, {}) ?? undefined; + const rep = await speculate(plans, { registry: e.registry, policy, environment: e.reversi.environment }, { actor: o.actor }); + if (o.json) return io.out(JSON.stringify(rep, null, 2)); + for (const p of rep.plans) { + io.out(`${p.name}${p.name === rep.recommended ? " <- recommended" : ""}`); + io.out(` ${p.blocked_at !== undefined ? `blocked at call ${p.blocked_at}: ${p.steps[p.blocked_at]?.reasons[0]}` : "can run to the end"}; approvals needed: ${p.approvals_needed}; highest risk: ${p.max_risk}; ${p.fully_reversible ? "fully reversible" : "NOT fully reversible"}`); + if (p.irreversible.length) io.out(` irreversible: ${p.irreversible.join(", ")}`); + if (p.residue.length) io.out(` leaves behind: ${p.residue.join("; ")}`); + for (const n of p.notes) io.out(` note: ${n}`); + } + if (!rep.recommended) io.out("no plan can run to the end"); + }), + ); + + ctx + .dataOpts(program.command("impact ").description("what approving a call to this tool would do and what could be undone").option("--args ", "the call's arguments").option("--policy ").option("--actor ").option("--json")) + .action((tool: string, o: EngineFlags & { args?: string; actor?: string; json?: boolean }) => + ctx.withEngine(o, async (e) => { + let args: unknown; + if (o.args) { + try { + args = JSON.parse(o.args); + } catch { + throw new Fail("--args must be JSON"); + } + } + const policy = loadPolicyDoc(ctx, o.policy, {}) ?? undefined; + const p = await previewImpact({ registry: e.registry, policy, environment: e.reversi.environment }, { tool, args, actor: o.actor }); + if (o.json) return io.out(JSON.stringify(p, null, 2)); + io.out(`${p.tool}: ${p.known ? p.effect : "no manifest"}${p.status && p.status !== "approved" ? ` (${p.status})` : ""}`); + io.out(`decision now: ${p.verdict}${p.reasons.length ? ` (${p.reasons[0]})` : ""}`); + if (p.risk_score) io.out(`risk ${p.risk_score.score}: ${p.risk_score.factors.join(", ") || "no factors"}`); + if (p.risk) io.out(`data class ${p.risk.data_class ?? "?"}, blast radius ${p.risk.blast_radius ?? "?"}${p.owner ? `, owner ${p.owner}` : ""}`); + if (p.requires.length) io.out(`needs first: ${p.requires.join(", ")}`); + if (p.approval) io.out(`approvers: ${p.approval.roles.join(" or ") || "any"}, ${p.approval.quorum} needed`); + for (const l of p.rollback_outlook) io.out(`rollback: ${l}`); + for (const w of p.warnings) io.out(`warning: ${w}`); + }), + ); + + void readFileSync; + void parsePolicy; +} diff --git a/packages/cli/src/commands/upstream.ts b/packages/cli/src/commands/upstream.ts new file mode 100644 index 0000000..cd7bada --- /dev/null +++ b/packages/cli/src/commands/upstream.ts @@ -0,0 +1,42 @@ +/** Connecting to an MCP server from the CLI: to lint against it, draft manifests from it, or check it is healthy. */ +import { loadGatewayConfig, parseGatewayConfig, UpstreamPool, type UpstreamConfig } from "@multimindlab/reversi-gateway"; +import type { McpToolInfo } from "@multimindlab/reversi"; +import type { Command } from "commander"; +import { Fail } from "../context.js"; + +export interface UpstreamFlags { + upstreamCmd?: string; + upstreamArg?: string[]; + upstreamUrl?: string; + gatewayConfig?: string; +} + +export function upstreamOpts(c: Command): Command { + return c + .option("--gateway-config ", "read the upstreams from a gateway config file") + .option("--upstream-cmd ", "an MCP server to start over stdio") + .option("--upstream-arg ", "an argument for --upstream-cmd (repeatable)", (v: string, prev: string[] = []) => [...prev, v]) + .option("--upstream-url ", "a Streamable HTTP MCP server"); +} + +export function hasUpstream(o: UpstreamFlags): boolean { + return !!(o.upstreamCmd || o.upstreamUrl || o.gatewayConfig); +} + +export function upstreamConfigs(o: UpstreamFlags): UpstreamConfig[] { + if (o.gatewayConfig) return loadGatewayConfig(o.gatewayConfig).upstreams; + if (!!o.upstreamCmd === !!o.upstreamUrl) throw new Fail("give --gateway-config, or exactly one of --upstream-cmd / --upstream-url"); + return parseGatewayConfig({ upstreams: [o.upstreamUrl ? { name: "upstream", url: o.upstreamUrl } : { name: "upstream", command: o.upstreamCmd, args: o.upstreamArg ?? [] }] }).upstreams; +} + +/** Connect, list the tools (named as the gateway would expose them), and disconnect. */ +export async function fetchUpstreamTools(o: UpstreamFlags): Promise<{ tools: McpToolInfo[]; server: string }> { + const cfgs = upstreamConfigs(o); + const pool = await UpstreamPool.connect(cfgs); + try { + const tools = (await pool.listTools()) as unknown as McpToolInfo[]; + return { tools, server: cfgs.length === 1 ? (cfgs[0] as UpstreamConfig).name : "upstream" }; + } finally { + await pool.close(); + } +} diff --git a/packages/cli/src/context.ts b/packages/cli/src/context.ts new file mode 100644 index 0000000..8e37731 --- /dev/null +++ b/packages/cli/src/context.ts @@ -0,0 +1,139 @@ +/** Shared plumbing for CLI commands: option handling, project config defaults, opening the engine. */ +import { existsSync, readFileSync } from "node:fs"; +import { extname } from "node:path"; +import type { Command } from "commander"; +import { parse as parseYaml } from "yaml"; +import { + dataOptionsFromEnv, + findProjectConfig, + loadProjectConfig, + openEngine, + storeKindFrom, + type DataOptions, + type EngineConfig, + type EngineHandle, + type ProjectConfig, +} from "@multimindlab/reversi"; + +export interface IO { + out: (s: string) => void; + err: (s: string) => void; +} + +/** An error whose message is for the user, not a stack trace. */ +export class Fail extends Error {} + +export interface StoreFlags { + data?: string; + store?: string; + databaseUrl?: string; + config?: string; +} +export type EngineFlags = StoreFlags & { manifests?: string; policy?: string }; + +export interface Ctx { + io: IO; + /** Exit code the command wants (0 ok, 1 error, 2 incomplete). */ + code: number; + program: Command; + project(flags?: StoreFlags): ProjectConfig | undefined; + storeOpts(c: Command): Command; + /** Adds the storage options plus `-m, --manifests` (optional when the project config names them). */ + dataOpts(c: Command): Command; + dataOptionsOf(o: StoreFlags): DataOptions; + dataExists(o: StoreFlags): void; + engineConfigOf(o: EngineFlags, extra?: Partial): EngineConfig; + engine(o: EngineFlags, extra?: Partial): Promise; + withEngine(o: EngineFlags, fn: (e: EngineHandle) => Promise | T, extra?: Partial): Promise; + readDoc(path: string): unknown; + manifestsPath(o: EngineFlags): string; +} + +export function makeContext(program: Command, io: IO): Ctx { + let cached: { key: string; value: ProjectConfig | undefined } | undefined; + + const ctx: Ctx = { + io, + code: 0, + program, + + project(flags = {}) { + const explicit = flags.config ?? (program.opts() as { config?: string }).config; + const key = explicit ?? process.cwd(); + if (cached?.key === key) return cached.value; + const file = findProjectConfig(process.cwd(), explicit); + cached = { key, value: file ? loadProjectConfig(file) : undefined }; + return cached.value; + }, + + storeOpts(c) { + return c + .option("-d, --data ", "data directory (default: reversi.config.yaml, REVERSI_DATA or ./reversi-data)") + .option("--store ", "files | sqlite | postgres (default: the config file, REVERSI_STORE or files)") + .option("--database-url ", "PostgreSQL URL (default: REVERSI_DATABASE_URL)"); + }, + + dataOpts(c) { + return ctx.storeOpts(c.option("-m, --manifests ", "manifests: a file, a folder, a registry folder or an https URL (default: the config file or REVERSI_MANIFESTS)")); + }, + + dataOptionsOf(o) { + const env = dataOptionsFromEnv(); + const p = ctx.project(o)?.data; + return { + dir: o.data ?? process.env.REVERSI_DATA ?? p?.dir ?? "./reversi-data", + store: o.store ? storeKindFrom(o.store) : process.env.REVERSI_STORE ? env.store : (p?.store ?? env.store), + postgres_url: o.databaseUrl ?? env.postgres_url ?? p?.database_url, + }; + }, + + dataExists(o) { + const d = ctx.dataOptionsOf(o); + if (d.store === "files" && !existsSync(d.dir)) throw new Fail(`data directory not found: ${d.dir} (run \`reversi init\`, or pass -d)`); + }, + + manifestsPath(o) { + const m = o.manifests ?? process.env.REVERSI_MANIFESTS ?? ctx.project(o)?.manifests; + if (!m) throw new Fail("no manifests: pass -m , set REVERSI_MANIFESTS, or add `manifests:` to reversi.config.yaml"); + return m; + }, + + engineConfigOf(o, extra = {}) { + const p = ctx.project(o); + return { + manifests: ctx.manifestsPath(o), + data: ctx.dataOptionsOf(o), + policy: o.policy ?? process.env.REVERSI_POLICY ?? p?.policy, + environment: process.env.REVERSI_ENV ?? p?.environment, + manifest_public_key: process.env.REVERSI_MANIFEST_PUBLIC_KEY ?? p?.manifest_public_key, + signing_key: process.env.REVERSI_SIGNING_KEY ?? p?.signing_key, + egress: p?.egress, + anchor: p?.anchor, + notifications: p?.notifications, + alerts: p?.alerts, + capture: process.env.REVERSI_CAPTURE === "1" || process.env.REVERSI_CAPTURE === "on" ? { enabled: true, ttl_ms: process.env.REVERSI_CAPTURE_TTL_DAYS ? Number(process.env.REVERSI_CAPTURE_TTL_DAYS) * 86_400_000 : p?.capture?.ttl_ms, redact_pii: p?.capture?.redact_pii } : p?.capture, + ...extra, + }; + }, + + async engine(o, extra) { + ctx.dataExists(o); + return openEngine(ctx.engineConfigOf(o, { log: () => {}, ...extra })); + }, + + async withEngine(o, fn, extra) { + const e = await ctx.engine(o, extra); + try { + return await fn(e); + } finally { + await e.close(); + } + }, + + readDoc(path) { + const text = readFileSync(path, "utf8"); + return extname(path).toLowerCase() === ".json" ? JSON.parse(text) : parseYaml(text); + }, + }; + return ctx; +} diff --git a/packages/cli/test/apply.test.ts b/packages/cli/test/apply.test.ts new file mode 100644 index 0000000..9554cdd --- /dev/null +++ b/packages/cli/test/apply.test.ts @@ -0,0 +1,169 @@ +import { mkdirSync, mkdtempSync, readFileSync, readdirSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join, resolve } from "node:path"; +import { afterEach, beforeEach, describe, expect, it } from "vitest"; +import { serve } from "@hono/node-server"; +import { Approvals, Ledger, ManifestRegistry, Policy, PolicyVersions, Registry, Reversi } from "@multimindlab/reversi"; +import { createApp } from "@multimindlab/reversi-server"; +import { main } from "../src/cli.js"; + +let dir: string; +beforeEach(() => { + dir = mkdtempSync(join(tmpdir(), "reversi-apply-")); +}); +afterEach(() => rmSync(dir, { recursive: true, force: true })); + +async function cli(...argv: string[]) { + const out: string[] = []; + const err: string[] = []; + const code = await main(argv, { out: (s) => out.push(s), err: (s) => err.push(s) }); + return { code, out: out.join("\n"), err: err.join("\n") }; +} + +const desiredDir = () => join(dir, "desired"); +function write(name: string, text: string) { + mkdirSync(join(desiredDir(), "manifests"), { recursive: true }); + writeFileSync(join(desiredDir(), "manifests", name), text); +} +const A = "tool: shop.read\neffect: read_only\npermission: shop:read\n"; +const B = "tool: shop.write\neffect: idempotent\npermission: shop:write\n"; +const POLICY = "version: 1\nrules:\n - id: r\n match: { tool: 'shop.write' }\n action: deny\ntests:\n - name: denied\n input: { tool: shop.write, args: {} }\n expect: deny\n"; + +describe("reversi apply", () => { + it("creates, is idempotent, updates, and prunes by deprecating", async () => { + const reg = join(dir, "registry"); + const pol = join(dir, "policy"); + write("a.yaml", A); + write("b.yaml", B); + writeFileSync(join(desiredDir(), "policy.yaml"), POLICY); + const base = ["apply", desiredDir(), "--registry", reg, "--policy-dir", pol, "--by", "ci"]; + + const dry = await cli(...base, "--dry-run", "--detailed-exitcode"); + expect(dry.out).toContain("+ manifest shop.read"); + expect(dry.out).toContain("3 changes (dry run: nothing applied)"); + expect(dry.code).toBe(2); + expect(readdirSync(dir)).not.toContain("registry"); + + const first = await cli(...base); + expect(first.code).toBe(0); + expect(first.out).toContain("applied shop.read v1 (approved)"); + expect(first.out).toContain("applied policy v1"); + const registry = new ManifestRegistry(reg); + expect(registry.history("shop.write").at(-1)?.status).toBe("approved"); + expect(new PolicyVersions(pol).activeVersion()).toBe(1); + + const again = await cli(...base, "--dry-run", "--detailed-exitcode"); + expect(again.out).toContain("up to date"); + expect(again.code).toBe(0); + + write("b.yaml", "tool: shop.write\neffect: idempotent\npermission: shop:write\nowner: team@example.com\n"); + const upd = await cli(...base); + expect(upd.out).toContain("~ manifest shop.write"); + expect(registry.history("shop.write")).toHaveLength(2); + expect(registry.history("shop.read")).toHaveLength(1); // untouched + + rmSync(join(desiredDir(), "manifests", "a.yaml")); + expect((await cli(...base)).out).toContain("up to date"); // no prune: nothing removed + const pruned = await cli(...base, "--prune"); + expect(pruned.out).toContain("- manifest shop.read"); + expect(registry.history("shop.read").at(-1)?.status).toBe("deprecated"); + }); + + it("refuses an invalid directory and a registry that is just loose files", async () => { + write("bad.yaml", "tool: x\neffect: nonsense\npermission: p\n"); + const bad = await cli("apply", desiredDir(), "--registry", join(dir, "r")); + expect(bad.code).toBe(1); + expect(bad.err).toContain("invalid manifest"); + rmSync(desiredDir(), { recursive: true }); + write("a.yaml", A); + const loose = join(dir, "loose"); + mkdirSync(loose); + writeFileSync(join(loose, "x.yaml"), A); + const r = await cli("apply", desiredDir(), "--registry", loose); + expect(r.code).toBe(1); + expect(r.err).toContain("not a registry"); + }); +}); + +describe("reversi apply --server", () => { + it("applies to a running server with an admin key, and refuses a lesser key", async () => { + const approvals = new Approvals(); + const reversi = new Reversi({ registry: new Registry(), ledger: new Ledger(), approver: approvals.asApprover(), policy: Policy.from({ rules: [] }) }); + const app = createApp({ reversi, approvals, keys: [{ name: "root", role: "admin", key: "k-root" }, { name: "ops", role: "operator", key: "k-ops" }], registry_dir: join(dir, "sreg"), policy_dir: join(dir, "spol") }); + const server = await new Promise>((r) => { + const s = serve({ fetch: app.fetch, port: 0, hostname: "127.0.0.1" }, () => r(s)); + }); + try { + const port = (server.address() as { port: number }).port; + write("a.yaml", A); + writeFileSync(join(desiredDir(), "policy.yaml"), POLICY); + const url = `http://127.0.0.1:${port}`; + const denied = await cli("apply", desiredDir(), "--server", url, "--token", "k-ops"); + expect(denied.code).toBe(1); + const ok = await cli("apply", desiredDir(), "--server", url, "--token", "k-root"); + expect(ok.out).toContain("applied shop.read v1 (approved)"); + expect(new ManifestRegistry(join(dir, "sreg")).history("shop.read")).toHaveLength(1); + expect((await cli("apply", desiredDir(), "--server", url, "--token", "k-root", "--dry-run")).out).toContain("up to date"); + } finally { + server.close(); + } + }); +}); + +describe("suggest-manifest", () => { + const fs = resolve(__dirname, "../../connectors/dist/main.js"); + it("drafts manifests from a real MCP server and writes them out", async () => { + const sandbox = join(dir, "sb"); + mkdirSync(sandbox); + const out = join(dir, "drafts"); + const r = await cli("suggest-manifest", "--upstream-cmd", "node", "--upstream-arg", fs, "--upstream-arg", "serve", "--upstream-arg", "fs", "--upstream-arg", "--root", "--upstream-arg", sandbox, "-o", out); + expect(r.code).toBe(0); + const files = readdirSync(out); + expect(files).toContain("fs.write_file.yaml"); + expect(readFileSync(join(out, "fs.write_file.yaml"), "utf8")).toContain("DRAFT"); + expect(r.err).toContain("drafted"); + }); +}); + +describe("rollback read-back and approvals through the engine", () => { + it("--reader-cmd runs a manifest's declared verification; without it the step is reported unverified", async () => { + const m = join(dir, "m.yaml"); + writeFileSync(m, `- tool: shop.cancel\n effect: idempotent\n permission: s:w\n- tool: shop.get\n effect: read_only\n permission: s:r\n- tool: shop.create\n effect: reversible\n permission: s:w\n compensation:\n tool: shop.cancel\n argument_map: { id: $result.id }\n verification:\n tool: shop.get\n arguments: { id: $result.id }\n expect: { status: canceled }\n retries: 1\n backoff_ms: 1\n`); + const data = join(dir, "d"); + const { Reversi, Ledger, JsonlLedgerStore, FileRunStore, SealedScope, loadRegistry } = await import("@multimindlab/reversi"); + const rv = new Reversi({ registry: loadRegistry(m), ledger: new Ledger(new JsonlLedgerStore(join(data, "ledger.jsonl"))), runs: new FileRunStore(join(data, "runs")) }); + const run = rv.startRun({ run_id: "v1", actor: "a", scope: SealedScope.seal(["s:w", "s:r"]) }); + await run.call({ tool: "shop.create", reason: "x", execute: () => ({ id: "o9" }) }); + const base = ["rollback", "v1", "-m", m, "-d", data, "--compensator-cmd", "cat > /dev/null"]; + const no = await cli(...base); + expect(no.code).toBe(2); + expect(no.out).toContain("not verified"); + const ok = await cli(...base, "--reader-cmd", `cat > /dev/null; echo '{"status":"canceled"}'`); + expect(ok.out).toContain("rollback complete"); + }); + + it("an approval decided from the CLI is journaled in the ledger", async () => { + const data = join(dir, "d2"); + const m = join(dir, "m2.yaml"); + writeFileSync(m, "- tool: shop.pay\n effect: idempotent\n permission: s:w\n approval: always\n"); + const { Reversi, Ledger, JsonlLedgerStore, FileRunStore, FileApprovalStore, Approvals, SealedScope, loadRegistry } = await import("@multimindlab/reversi"); + const approvals = new Approvals(new FileApprovalStore(join(data, "approvals.json"))); + const rv = new Reversi({ registry: loadRegistry(m), ledger: new Ledger(new JsonlLedgerStore(join(data, "ledger.jsonl"))), runs: new FileRunStore(join(data, "runs")), approver: approvals.asApprover() }); + const run = rv.startRun({ run_id: "p1", actor: "a", scope: SealedScope.seal(["s:w"]) }); + await run.call({ tool: "shop.pay", reason: "x", execute: () => 1 }); + const id = approvals.list({ status: "pending" })[0]!.id; + const r = await cli("approvals", "approve", id, "-d", data, "--as", "dana", "--note", "ok"); + expect(r.out).toBe(`${id} approved by dana`); + expect(readFileSync(join(data, "ledger.jsonl"), "utf8")).toContain(`approval ${id} approved by dana`); + }); +}); + +describe("commands that read manifests accept a registry folder", () => { + it("validate and lint work on it", async () => { + const reg = join(dir, "r"); + new ManifestRegistry(reg).publish({ tool: "shop.read", effect: "read_only", permission: "s:r" }, { by: "t", status: "approved" }); + expect((await cli("validate", reg)).code).toBe(0); + const lint = await cli("lint", reg); + expect(lint.code).toBe(0); + }); +}); diff --git a/packages/cli/tsconfig.json b/packages/cli/tsconfig.json index b223a6f..8130fc9 100644 --- a/packages/cli/tsconfig.json +++ b/packages/cli/tsconfig.json @@ -2,5 +2,5 @@ "extends": "../../tsconfig.base.json", "compilerOptions": { "rootDir": "src", "outDir": "dist" }, "include": ["src"], - "references": [{ "path": "../reversi" }] + "references": [{ "path": "../reversi" }, { "path": "../gateway" }, { "path": "../server" }] } diff --git a/packages/connectors/package.json b/packages/connectors/package.json index 172bb3c..12c5144 100644 --- a/packages/connectors/package.json +++ b/packages/connectors/package.json @@ -1,7 +1,7 @@ { "name": "@multimindlab/reversi-connectors", "version": "0.1.0", - "description": "Real tool connectors with Reversi manifests and working compensations: filesystem, GitHub, Slack, Stripe", + "description": "Real tool connectors with Reversi manifests and working compensations: filesystem, GitHub, GitLab, Slack, Teams, Linear, Jira, Stripe, Google Calendar, Postgres, S3, Cloud Storage, Blob Storage, Kubernetes", "license": "Apache-2.0", "repository": { "type": "git", @@ -30,6 +30,7 @@ "dependencies": { "@modelcontextprotocol/sdk": "^1.32.1", "@multimindlab/reversi": "0.1.0", + "pg": "^8.23.1", "yaml": "^2.8.0" }, "devDependencies": { diff --git a/packages/connectors/src/aws.ts b/packages/connectors/src/aws.ts new file mode 100644 index 0000000..df79020 --- /dev/null +++ b/packages/connectors/src/aws.ts @@ -0,0 +1,307 @@ +import { EffectUncertainError } from "@multimindlab/reversi"; +import { + HttpError, + defineConnector, + matches, + optPosInt, + optStr, + redact, + send, + xmlBlocks, + xmlText, + BACKUP_PREFIX, + type Connector, + type Fetch, + type Sent, +} from "./common.js"; +import { + backupName, + contentBytes, + contentOut, + contentProps, + isBackupName, + maxBytes, + mediaType, + newBackupName, + objectName, +} from "./objects.js"; +import { amzDateOf, canonicalQueryString, sha256Hex, signV4, uriEncode } from "./sigv4.js"; + +export interface AwsConfig { + accessKeyId: string; + secretAccessKey: string; + sessionToken?: string; + region: string; + fetch?: Fetch; + /** An S3-compatible endpoint (MinIO, LocalStack). Requests then use path-style URLs. */ + endpoint?: string; + timeoutMs?: number; + /** Clock for the signature; tests pin it. */ + now?: () => Date; +} + +const BUCKET = /^[a-z0-9][a-z0-9.-]{1,61}[a-z0-9]$/; +const REGION = /^[a-z]{2}(-[a-z]+)+-\d$/; +const EMPTY_HASH = sha256Hex(""); + +export function createAwsConnector(cfg: AwsConfig): Connector { + if (!cfg.accessKeyId) throw new Error("aws connector needs an `accessKeyId`"); + if (!cfg.secretAccessKey) throw new Error("aws connector needs a `secretAccessKey`"); + if (!cfg.region || !REGION.test(cfg.region)) throw new Error("aws connector needs a `region` such as us-east-1"); + const f = cfg.fetch ?? fetch; + const secrets = [cfg.secretAccessKey, cfg.sessionToken ?? ""].filter(Boolean); + const timeoutMs = cfg.timeoutMs ?? 30000; + const now = cfg.now ?? (() => new Date()); + const endpoint = cfg.endpoint ? new URL(cfg.endpoint.replace(/\/+$/, "")) : undefined; + + function location(bucket: string, key: string): { origin: string; host: string; path: string } { + if (endpoint) { + const prefix = endpoint.pathname.replace(/\/+$/, ""); + return { + origin: endpoint.origin, + host: endpoint.host, + path: `${prefix}/${bucket}${key ? "/" + uriEncode(key, true) : ""}`, + }; + } + if (bucket.includes(".")) { + const host = `s3.${cfg.region}.amazonaws.com`; + return { origin: `https://${host}`, host, path: `/${bucket}${key ? "/" + uriEncode(key, true) : ""}` }; + } + const host = `${bucket}.s3.${cfg.region}.amazonaws.com`; + return { origin: `https://${host}`, host, path: `/${key ? uriEncode(key, true) : ""}` }; + } + + /** A signed S3 request. XML error bodies become HttpError unless the status is tolerated. */ + async function s3( + method: "GET" | "HEAD" | "PUT" | "DELETE", + bucket: string, + key: string, + o: { + query?: Array<[string, string]>; + headers?: Record; + body?: Uint8Array; + binary?: boolean; + tolerate?: number[]; + } = {}, + ): Promise { + const loc = location(bucket, key); + const body = o.body; + const headers: Record = { + host: loc.host, + "x-amz-date": amzDateOf(now()), + "x-amz-content-sha256": body ? sha256Hex(body) : EMPTY_HASH, + ...(o.headers ?? {}), + }; + const signed = signV4({ + method, + canonicalUri: loc.path || "/", + query: o.query, + headers, + payloadHash: headers["x-amz-content-sha256"]!, + region: cfg.region, + service: "s3", + accessKeyId: cfg.accessKeyId, + secretAccessKey: cfg.secretAccessKey, + sessionToken: cfg.sessionToken, + amzDate: headers["x-amz-date"]!, + }); + const { host: _host, ...wire } = signed.headers; + void _host; + const qs = o.query && o.query.length ? `?${canonicalQueryString(o.query)}` : ""; + const label = `S3 ${method} ${bucket}/${key}`.trim(); + const res = await send( + f, + `${loc.origin}${loc.path || "/"}${qs}`, + { method, headers: { ...wire, Authorization: signed.authorization }, body }, + { label, mutating: method === "PUT" || method === "DELETE", secrets, timeoutMs, binary: o.binary }, + ); + if (!res.ok && !(o.tolerate ?? []).includes(res.status)) { + const code = xmlText(res.text, "Code"); + const msg = xmlText(res.text, "Message"); + const detail = code || msg ? `${code ?? ""}${code && msg ? ": " : ""}${msg ?? ""}` : ""; + throw new HttpError(`${label} failed with status ${res.status}${detail ? `: ${redact(detail, secrets).slice(0, 200)}` : ""}`, res.status); + } + return res; + } + + /** CopyObject can answer 200 with an body. */ + async function copy(bucket: string, from: string, to: string): Promise { + const r = await s3("PUT", bucket, to, { + headers: { "x-amz-copy-source": `/${bucket}/${uriEncode(from, true)}` }, + }); + if (//.test(r.text)) { + const code = xmlText(r.text, "Code") ?? "unknown"; + throw new HttpError(`S3 copy ${from} to ${to} failed: ${code}`, 500); + } + } + + const bucketProp = { type: "string", description: "Bucket name" }; + const keyProp = { type: "string", description: "Object key" }; + const bucketOf = (args: Record) => { + const b = matches(args, "bucket", BUCKET, "bucket name"); + if (b.includes("..")) throw new Error("`bucket` is not a valid bucket name"); + return b; + }; + + /** HEAD, then copy the current object aside. Returns the backup name, or null if the key did not exist. */ + async function backUp(bucket: string, key: string): Promise<{ existed: boolean; backup_key: string | null }> { + const head = await s3("HEAD", bucket, key, { tolerate: [404] }); + if (head.status === 404) return { existed: false, backup_key: null }; + const backup_key = newBackupName(); + await copy(bucket, key, backup_key); + return { existed: true, backup_key }; + } + + async function dropBackup(bucket: string, backup_key: string | null): Promise { + if (backup_key) await s3("DELETE", bucket, backup_key).catch(() => undefined); + } + + const restoreMap = { bucket: "$args.bucket", key: "$args.key", backup_key: "$result.backup_key" }; + + return defineConnector("aws", [ + { + name: "get_object", + description: "Read an object (at most max_bytes, default 5 MiB). Needs s3:GetObject.", + effect: "read_only", + permission: "aws:s3:read", + properties: { bucket: bucketProp, key: keyProp, max_bytes: { type: "integer", minimum: 1 } }, + required: ["bucket", "key"], + handler: async (args) => { + const bucket = bucketOf(args); + const key = objectName(args, "key", { allowBackup: true }); + const max = maxBytes(args); + const r = await s3("GET", bucket, key, { headers: { range: `bytes=0-${max - 1}` }, binary: true }); + const bytes = r.bytes ?? new Uint8Array(); + const total = /\/(\d+)$/.exec(r.headers.get("content-range") ?? "")?.[1]; + const size = total ? Number(total) : bytes.byteLength; + return { + bucket, + key, + ...contentOut(bytes), + size, + truncated: size > bytes.byteLength, + content_type: r.headers.get("content-type"), + etag: r.headers.get("etag"), + }; + }, + }, + { + name: "list_objects", + description: "List objects (ListObjectsV2). Needs s3:ListBucket. Backup objects are hidden.", + effect: "read_only", + permission: "aws:s3:read", + properties: { + bucket: bucketProp, + prefix: { type: "string" }, + max_keys: { type: "integer", minimum: 1, maximum: 1000 }, + continuation_token: { type: "string" }, + }, + required: ["bucket"], + handler: async (args) => { + const bucket = bucketOf(args); + const query: Array<[string, string]> = [["list-type", "2"]]; + const prefix = optStr(args, "prefix", { max: 1024 }); + const max = optPosInt(args, "max_keys"); + if (max !== undefined && max > 1000) throw new Error("`max_keys` must be at most 1000"); + const token = optStr(args, "continuation_token", { max: 2048 }); + if (prefix) query.push(["prefix", prefix]); + if (max) query.push(["max-keys", String(max)]); + if (token) query.push(["continuation-token", token]); + const r = await s3("GET", bucket, "", { query }); + const objects = xmlBlocks(r.text, "Contents") + .map((c) => ({ + key: xmlText(c, "Key") ?? "", + size: Number(xmlText(c, "Size") ?? 0), + last_modified: xmlText(c, "LastModified") ?? null, + etag: xmlText(c, "ETag") ?? null, + })) + .filter((o) => o.key && !isBackupName(o.key)); + return { + objects, + truncated: xmlText(r.text, "IsTruncated") === "true", + next_continuation_token: xmlText(r.text, "NextContinuationToken") ?? null, + }; + }, + }, + { + name: "put_object", + description: `Write an object. If one exists it is first copied to ${BACKUP_PREFIX} in the same bucket, so rollback restores it (or deletes the new object). Needs s3:GetObject, s3:PutObject and s3:ListBucket.`, + effect: "reversible", + permission: "aws:s3:write", + compensation: { tool: "restore_object", argument_map: restoreMap }, + properties: { bucket: bucketProp, key: keyProp, ...contentProps }, + required: ["bucket", "key", "content"], + handler: async (args) => { + const bucket = bucketOf(args); + const key = objectName(args, "key"); + const body = contentBytes(args); + const type = mediaType(args); + const { existed, backup_key } = await backUp(bucket, key); + try { + const r = await s3("PUT", bucket, key, { body, headers: { "content-type": type } }); + return { bucket, key, existed, backup_key, etag: r.headers.get("etag") }; + } catch (e) { + if (e instanceof EffectUncertainError && backup_key) { + throw new EffectUncertainError(`${e.message}; the previous version is saved at ${backup_key}`); + } + if (!(e instanceof EffectUncertainError)) await dropBackup(bucket, backup_key); + throw e; + } + }, + }, + { + name: "delete_object", + description: `Delete an object. It is first copied to ${BACKUP_PREFIX} so rollback can restore it. Needs s3:GetObject, s3:DeleteObject and s3:ListBucket.`, + effect: "reversible", + permission: "aws:s3:write", + compensation: { tool: "restore_object", argument_map: restoreMap }, + properties: { bucket: bucketProp, key: keyProp }, + required: ["bucket", "key"], + handler: async (args) => { + const bucket = bucketOf(args); + const key = objectName(args, "key"); + const { existed, backup_key } = await backUp(bucket, key); + if (!existed) return { bucket, key, existed, backup_key }; + try { + await s3("DELETE", bucket, key); + } catch (e) { + if (e instanceof EffectUncertainError) { + throw new EffectUncertainError(`${e.message}; the previous version is saved at ${backup_key}`); + } + await dropBackup(bucket, backup_key); + throw e; + } + return { bucket, key, existed, backup_key }; + }, + }, + { + name: "restore_object", + description: + "Undo put_object or delete_object: copy the saved backup back over the key, or delete the key when there was no previous object (backup_key null). Safe to repeat. The backup object is kept.", + effect: "idempotent", + permission: "aws:s3:write", + properties: { + bucket: bucketProp, + key: keyProp, + backup_key: { type: ["string", "null"], description: "Backup object name, or null if the key did not exist" }, + }, + required: ["bucket", "key", "backup_key"], + handler: async (args) => { + const bucket = bucketOf(args); + const key = objectName(args, "key"); + const backup = backupName(args, "backup_key"); + if (backup === null) { + await s3("DELETE", bucket, key); + return { bucket, key, restored: true, action: "deleted" }; + } + try { + await copy(bucket, backup, key); + } catch (e) { + if (e instanceof HttpError && e.status === 404) throw new Error(`backup ${backup} was not found`); + throw e; + } + return { bucket, key, restored: true, action: "restored" }; + }, + }, + ]); +} diff --git a/packages/connectors/src/azure.ts b/packages/connectors/src/azure.ts new file mode 100644 index 0000000..b71db4e --- /dev/null +++ b/packages/connectors/src/azure.ts @@ -0,0 +1,278 @@ +import { EffectUncertainError } from "@multimindlab/reversi"; +import { + BACKUP_PREFIX, + HttpError, + defineConnector, + matches, + optPosInt, + optStr, + redact, + send, + xmlBlocks, + xmlText, + type Connector, + type Fetch, + type Sent, +} from "./common.js"; +import { + backupName, + contentBytes, + contentOut, + contentProps, + isBackupName, + maxBytes, + mediaType, + newBackupName, + objectName, +} from "./objects.js"; + +export interface AzureConfig { + /** Storage account name. */ + account: string; + /** A Microsoft Entra bearer token for https://storage.azure.com/. */ + token: string; + fetch?: Fetch; + /** Override the endpoint, e.g. for Azurite. Default https://.blob.core.windows.net */ + baseUrl?: string; + timeoutMs?: number; +} + +const ACCOUNT = /^[a-z0-9]{3,24}$/; +const CONTAINER = /^[a-z0-9](?:[a-z0-9]|-(?=[a-z0-9])){2,62}$/; +const API_VERSION = "2023-11-03"; + +export function createAzureConnector(cfg: AzureConfig): Connector { + if (!cfg.token) throw new Error("azure connector needs a `token`"); + if (!cfg.account || !ACCOUNT.test(cfg.account)) throw new Error("azure connector needs a storage `account` name"); + const f = cfg.fetch ?? fetch; + const base = (cfg.baseUrl ?? `https://${cfg.account}.blob.core.windows.net`).replace(/\/+$/, ""); + const secrets = [cfg.token]; + const timeoutMs = cfg.timeoutMs ?? 30000; + + const encode = (name: string) => name.split("/").map(encodeURIComponent).join("/"); + const blobUrl = (container: string, name: string) => `${base}/${container}/${encode(name)}`; + + async function blobApi( + method: "GET" | "HEAD" | "PUT" | "DELETE", + url: string, + o: { + query?: Array<[string, string]>; + headers?: Record; + body?: Uint8Array; + binary?: boolean; + tolerate?: number[]; + } = {}, + ): Promise { + const q = new URLSearchParams(o.query ?? []).toString(); + const label = `Blob ${method} ${url.slice(base.length)}`; + const res = await send( + f, + `${url}${q ? `?${q}` : ""}`, + { + method, + headers: { + Authorization: `Bearer ${cfg.token}`, + "x-ms-version": API_VERSION, + "x-ms-date": new Date().toUTCString(), + ...(o.headers ?? {}), + }, + body: o.body, + }, + { label, mutating: method === "PUT" || method === "DELETE", secrets, timeoutMs, binary: o.binary }, + ); + if (!res.ok && !(o.tolerate ?? []).includes(res.status)) { + const code = xmlText(res.text, "Code") ?? res.headers.get("x-ms-error-code") ?? ""; + const msg = xmlText(res.text, "Message") ?? ""; + const detail = [code, msg.split("\n")[0]].filter(Boolean).join(": "); + throw new HttpError(`${label} failed with status ${res.status}${detail ? `: ${redact(detail, secrets).slice(0, 200)}` : ""}`, res.status); + } + return res; + } + + /** Copy a blob within the account, synchronously, optionally only if the source still has `etag`. */ + async function copy(container: string, from: string, to: string, etag?: string): Promise { + const r = await blobApi("PUT", blobUrl(container, to), { + headers: { + "x-ms-copy-source": blobUrl(container, from), + "x-ms-requires-sync": "true", + "x-ms-copy-source-authorization": `Bearer ${cfg.token}`, + ...(etag ? { "x-ms-source-if-match": etag } : {}), + }, + }); + const status = r.headers.get("x-ms-copy-status"); + if (status && status !== "success") throw new Error(`Blob copy of ${from} ended with status ${status}`); + } + + async function dropBackup(container: string, backup: string | null): Promise { + if (backup) await blobApi("DELETE", blobUrl(container, backup), { tolerate: [404] }).catch(() => undefined); + } + + const containerProp = { type: "string", description: "Container name" }; + const blobProp = { type: "string", description: "Blob name" }; + const containerOf = (args: Record) => matches(args, "container", CONTAINER, "container name"); + const restoreMap = { container: "$args.container", blob: "$args.blob", backup_blob: "$result.backup_blob" }; + + /** HEAD the blob; if it exists, copy it aside bound to its ETag. */ + async function backUp(container: string, blob: string) { + const head = await blobApi("HEAD", blobUrl(container, blob), { tolerate: [404] }); + if (head.status === 404) return { existed: false, etag: null, backup_blob: null as string | null }; + const etag = head.headers.get("etag"); + if (!etag) throw new Error("Blob storage returned no ETag"); + const backup_blob = newBackupName(); + await copy(container, blob, backup_blob, etag); + return { existed: true, etag, backup_blob: backup_blob as string | null }; + } + + return defineConnector("azure", [ + { + name: "get_blob", + description: "Read a blob (at most max_bytes, default 5 MiB). Needs Storage Blob Data Reader.", + effect: "read_only", + permission: "azure:blob:read", + properties: { container: containerProp, blob: blobProp, max_bytes: { type: "integer", minimum: 1 } }, + required: ["container", "blob"], + handler: async (args) => { + const container = containerOf(args); + const blob = objectName(args, "blob", { allowBackup: true }); + const max = maxBytes(args); + const r = await blobApi("GET", blobUrl(container, blob), { headers: { Range: `bytes=0-${max - 1}` }, binary: true }); + const bytes = r.bytes ?? new Uint8Array(); + const total = /\/(\d+)$/.exec(r.headers.get("content-range") ?? "")?.[1]; + const size = total ? Number(total) : bytes.byteLength; + return { + container, + blob, + ...contentOut(bytes), + size, + truncated: size > bytes.byteLength, + content_type: r.headers.get("content-type"), + etag: r.headers.get("etag"), + }; + }, + }, + { + name: "list_blobs", + description: "List blobs in a container. Needs Storage Blob Data Reader. Backup blobs are hidden.", + effect: "read_only", + permission: "azure:blob:read", + properties: { + container: containerProp, + prefix: { type: "string" }, + max_results: { type: "integer", minimum: 1, maximum: 5000 }, + marker: { type: "string" }, + }, + required: ["container"], + handler: async (args) => { + const container = containerOf(args); + const query: Array<[string, string]> = [ + ["restype", "container"], + ["comp", "list"], + ]; + const prefix = optStr(args, "prefix", { max: 1024 }); + const max = optPosInt(args, "max_results"); + if (max !== undefined && max > 5000) throw new Error("`max_results` must be at most 5000"); + const marker = optStr(args, "marker", { max: 2048 }); + if (prefix) query.push(["prefix", prefix]); + if (max) query.push(["maxresults", String(max)]); + if (marker) query.push(["marker", marker]); + const r = await blobApi("GET", `${base}/${container}`, { query }); + const blobs = xmlBlocks(r.text, "Blob") + .map((b) => ({ + blob: xmlText(b, "Name") ?? "", + size: Number(xmlText(b, "Content-Length") ?? 0), + last_modified: xmlText(b, "Last-Modified") ?? null, + etag: xmlText(b, "Etag") ?? null, + })) + .filter((b) => b.blob && !isBackupName(b.blob)); + return { blobs, next_marker: xmlText(r.text, "NextMarker") || null }; + }, + }, + { + name: "put_blob", + description: `Write a block blob. If one exists it is first copied to ${BACKUP_PREFIX} in the same container and the write is conditional on its ETag, so rollback restores exactly what was overwritten (or deletes the new blob). Needs Storage Blob Data Contributor.`, + effect: "reversible", + permission: "azure:blob:write", + compensation: { tool: "restore_blob", argument_map: restoreMap }, + properties: { container: containerProp, blob: blobProp, ...contentProps }, + required: ["container", "blob", "content"], + handler: async (args) => { + const container = containerOf(args); + const blob = objectName(args, "blob"); + const body = contentBytes(args); + const type = mediaType(args); + const { existed, etag, backup_blob } = await backUp(container, blob); + try { + const r = await blobApi("PUT", blobUrl(container, blob), { + body, + headers: { + "x-ms-blob-type": "BlockBlob", + "Content-Type": type, + ...(etag ? { "If-Match": etag } : { "If-None-Match": "*" }), + }, + }); + return { container, blob, existed, backup_blob, etag: r.headers.get("etag") }; + } catch (e) { + if (e instanceof EffectUncertainError && backup_blob) { + throw new EffectUncertainError(`${e.message}; the previous version is saved at ${backup_blob}`); + } + if (!(e instanceof EffectUncertainError)) await dropBackup(container, backup_blob); + throw e; + } + }, + }, + { + name: "delete_blob", + description: `Delete a blob. It is first copied to ${BACKUP_PREFIX} and the delete is conditional on its ETag, so rollback restores what was deleted. Needs Storage Blob Data Contributor.`, + effect: "reversible", + permission: "azure:blob:write", + compensation: { tool: "restore_blob", argument_map: restoreMap }, + properties: { container: containerProp, blob: blobProp }, + required: ["container", "blob"], + handler: async (args) => { + const container = containerOf(args); + const blob = objectName(args, "blob"); + const { existed, etag, backup_blob } = await backUp(container, blob); + if (!existed) return { container, blob, existed, backup_blob }; + try { + await blobApi("DELETE", blobUrl(container, blob), { headers: { "If-Match": etag! } }); + } catch (e) { + if (e instanceof EffectUncertainError) { + throw new EffectUncertainError(`${e.message}; the previous version is saved at ${backup_blob}`); + } + await dropBackup(container, backup_blob); + throw e; + } + return { container, blob, existed, backup_blob }; + }, + }, + { + name: "restore_blob", + description: + "Undo put_blob or delete_blob: copy the saved backup back over the blob, or delete the blob when there was no previous one (backup_blob null). Safe to repeat. The backup blob is kept.", + effect: "idempotent", + permission: "azure:blob:write", + properties: { + container: containerProp, + blob: blobProp, + backup_blob: { type: ["string", "null"], description: "Backup blob name, or null if the blob did not exist" }, + }, + required: ["container", "blob", "backup_blob"], + handler: async (args) => { + const container = containerOf(args); + const blob = objectName(args, "blob"); + const backup = backupName(args, "backup_blob"); + if (backup === null) { + await blobApi("DELETE", blobUrl(container, blob), { tolerate: [404] }); + return { container, blob, restored: true, action: "deleted" }; + } + try { + await copy(container, backup, blob); + } catch (e) { + if (e instanceof HttpError && e.status === 404) throw new Error(`backup ${backup} was not found`); + throw e; + } + return { container, blob, restored: true, action: "restored" }; + }, + }, + ]); +} diff --git a/packages/connectors/src/certify.ts b/packages/connectors/src/certify.ts new file mode 100644 index 0000000..e72d612 --- /dev/null +++ b/packages/connectors/src/certify.ts @@ -0,0 +1,118 @@ +import { canonicalize, sha256Hex } from "@multimindlab/reversi"; +import type { Connector } from "./common.js"; + +/** + * The connector certification suite: the twelve checks every connector must pass (or declare not applicable with a + * reason). The checks are run against fake backends by `test/certification.ts`; this module holds the shared + * vocabulary and the manifest drift detector that the gateway and CLI can use at runtime. + */ + +export const CHECK_IDS = [ + "forward_success", + "forward_failure", + "duplicate_call", + "retry", + "compensation_success", + "compensation_failure", + "verification_failure", + "permission_denial", + "timeout", + "redaction", + "dependency_block", + "manifest_drift", +] as const; +export type CheckId = (typeof CHECK_IDS)[number]; + +export type CheckStatus = "pass" | "fail" | "not_applicable"; + +export interface CheckResult { + check: CheckId; + status: CheckStatus; + /** Required for `not_applicable`; for `fail`, what went wrong. */ + reason?: string; + /** Tools the check ran against. */ + tools: string[]; +} + +export interface Certification { + connector: string; + results: CheckResult[]; +} + +export const CHECKS: Array<{ id: CheckId; title: string; verifies: string }> = [ + { id: "forward_success", title: "Forward success", verifies: "every tool does its job against the fake backend, and read-only tools change nothing" }, + { id: "forward_failure", title: "Forward failure", verifies: "a rejected call surfaces as an error naming the status, changes nothing, and does not contain the credential" }, + { id: "duplicate_call", title: "Duplicate call", verifies: "the same call with the same idempotency key a second time has no further effect" }, + { id: "retry", title: "Retry", verifies: "a transport failure or 5xx on a write is reported as uncertain (EffectUncertainError) or an HTTP error, and running the call again succeeds" }, + { id: "compensation_success", title: "Compensation success", verifies: "a real engine rolls the step back and the fake backend is back in its prior state" }, + { id: "compensation_failure", title: "Compensation failure", verifies: "a backend error during the undo leaves the step compensation_failed and the rollback incomplete; a retry then completes" }, + { id: "verification_failure", title: "Verification failure", verifies: "a failing verify callback makes the rollback report the step as failed, never compensated" }, + { id: "permission_denial", title: "Permission denial", verifies: "a 401 or 403 surfaces as an error naming the status, without the credential" }, + { id: "timeout", title: "Timeout", verifies: "a backend that never answers ends in EffectUncertainError for writes and a plain error for reads" }, + { id: "redaction", title: "Redaction", verifies: "the credential appears in no error message, no returned JSON and no request URL or body" }, + { id: "dependency_block", title: "Dependency block", verifies: "with a `requires` edge, rolling back while the later step stands defers the earlier one" }, + { id: "manifest_drift", title: "Manifest drift", verifies: "a changed, added or removed tool input schema is detected against the stored hashes" }, +]; + +/** Hash of every tool's input schema, keyed by tool name. Store this as the baseline. */ +export function connectorSchemaHashes(connector: Pick): Record { + const out: Record = {}; + for (const [tool, schema] of Object.entries(connector.inputSchemas)) out[tool] = sha256Hex(canonicalize(schema)); + return out; +} + +export interface ManifestDrift { + added: string[]; + removed: string[]; + changed: string[]; +} + +/** + * Compare the current tools with a stored `schema_hash` map (tool name to hash, as made by + * {@link connectorSchemaHashes}). Without an explicit map, a `schema_hash` field on each manifest object is used; + * with neither, every tool counts as added. + */ +export function detectManifestDrift( + manifests: unknown[], + inputSchemas: Record, + stored?: Record, +): ManifestDrift { + const names = new Set(Object.keys(inputSchemas)); + const fromManifests: Record = {}; + for (const m of manifests) { + if (typeof m !== "object" || m === null) continue; + const r = m as { tool?: unknown; schema_hash?: unknown }; + if (typeof r.tool !== "string") continue; + names.add(r.tool); + if (typeof r.schema_hash === "string") fromManifests[r.tool] = r.schema_hash; + } + const baseline = stored ?? fromManifests; + const current = connectorSchemaHashes({ inputSchemas }); + const drift: ManifestDrift = { added: [], removed: [], changed: [] }; + for (const tool of [...names].sort()) { + const have = current[tool]; + const want = baseline[tool]; + if (want === undefined) drift.added.push(tool); + else if (have === undefined || have !== want) drift.changed.push(tool); + } + for (const tool of Object.keys(baseline).sort()) if (!names.has(tool)) drift.removed.push(tool); + return drift; +} + +export function certificationFailures(c: Certification): string[] { + return c.results.filter((r) => r.status === "fail").map((r) => `${c.connector}: ${r.check}: ${r.reason ?? "failed"}`); +} + +/** A markdown table of connectors by checks: `pass` or `n/a`. Fails render as `FAIL`. */ +export function renderCertificationMatrix(certs: Certification[]): string { + const head = ["Connector", ...CHECKS.map((c) => c.title)]; + const rows = certs.map((c) => [ + c.connector, + ...CHECK_IDS.map((id) => { + const r = c.results.find((x) => x.check === id); + return !r ? "missing" : r.status === "pass" ? "pass" : r.status === "fail" ? "FAIL" : "n/a"; + }), + ]); + const line = (cells: string[]) => `| ${cells.join(" | ")} |`; + return [line(head), line(head.map(() => "---")), ...rows.map(line)].join("\n"); +} diff --git a/packages/connectors/src/common.ts b/packages/connectors/src/common.ts index b343d9d..d77193e 100644 --- a/packages/connectors/src/common.ts +++ b/packages/connectors/src/common.ts @@ -161,6 +161,9 @@ export interface Sent { ok: boolean; json: unknown; text: string; + /** Raw body, set only when the request was sent with `binary: true`. */ + bytes?: Uint8Array; + headers: Headers; } export interface SendOptions { @@ -168,6 +171,8 @@ export interface SendOptions { mutating: boolean; secrets: string[]; timeoutMs: number; + /** Read the body as bytes (object downloads) instead of text. */ + binary?: boolean; } /** @@ -177,9 +182,15 @@ export interface SendOptions { export async function send(f: Fetch, url: string, init: RequestInit, o: SendOptions): Promise { let res: Response; let text: string; + let bytes: Uint8Array | undefined; try { res = await f(url, { ...init, signal: AbortSignal.timeout(o.timeoutMs) }); - text = await res.text(); + if (o.binary) { + bytes = new Uint8Array(await res.arrayBuffer()); + text = res.status >= 200 && res.status < 300 ? "" : new TextDecoder().decode(bytes); + } else { + text = await res.text(); + } } catch (e) { const msg = redact(e instanceof Error ? e.message : String(e), o.secrets); if (o.mutating) throw new EffectUncertainError(`${o.label}: no response (${msg}); the effect may have happened`); @@ -191,7 +202,7 @@ export async function send(f: Fetch, url: string, init: RequestInit, o: SendOpti } catch { json = undefined; } - return { status: res.status, ok: res.status >= 200 && res.status < 300, json, text }; + return { status: res.status, ok: res.status >= 200 && res.status < 300, json, text, bytes, headers: res.headers }; } export function snippet(s: Sent, secrets: string[]): string { @@ -209,3 +220,39 @@ export function snippet(s: Sent, secrets: string[]): string { export function asRecord(x: unknown): Args { return isRecord(x) ? x : {}; } + +/** Text of the first `` element, with the five XML entities decoded. */ +export function xmlText(xml: string, tag: string): string | undefined { + const m = new RegExp(`<${tag}(?:\\s[^>]*)?>([\\s\\S]*?)`).exec(xml); + return m ? xmlDecode(m[1]!) : undefined; +} + +/** Inner XML of every `` element. */ +export function xmlBlocks(xml: string, tag: string): string[] { + return [...xml.matchAll(new RegExp(`<${tag}(?:\\s[^>]*)?>([\\s\\S]*?)`, "g"))].map((m) => m[1]!); +} + +export function xmlDecode(s: string): string { + return s + .replace(/</g, "<") + .replace(/>/g, ">") + .replace(/"/g, '"') + .replace(/'/g, "'") + .replace(/&/g, "&"); +} + +/** A zero-or-more integer argument. */ +export function nonNegInt(args: Args, key: string): number { + const v = args[key]; + if (typeof v !== "number" || !Number.isSafeInteger(v) || v < 0) { + throw new Error(`\`${key}\` must be a non-negative integer`); + } + return v; +} + +/** Prefix under which object-store connectors keep the previous version of an object they overwrite or delete. */ +export const BACKUP_PREFIX = ".reversi-backups/"; + +export function assertNotBackupKey(key: string, what: string): void { + if (key.startsWith(BACKUP_PREFIX)) throw new Error(`\`${what}\` points into ${BACKUP_PREFIX}, which is reserved`); +} diff --git a/packages/connectors/src/gcalendar.ts b/packages/connectors/src/gcalendar.ts new file mode 100644 index 0000000..e2b77b5 --- /dev/null +++ b/packages/connectors/src/gcalendar.ts @@ -0,0 +1,245 @@ +import { + HttpError, + asRecord, + defineConnector, + matches, + optMatches, + optPosInt, + optStr, + send, + snippet, + str, + type Connector, + type Fetch, +} from "./common.js"; + +export interface GcalendarConfig { + token: string; + fetch?: Fetch; + baseUrl?: string; + timeoutMs?: number; +} + +const CALENDAR_ID = /^[A-Za-z0-9._@%+-]{1,256}$/; +const EVENT_ID = /^[A-Za-z0-9_-]{1,1024}$/; +const DATE = /^\d{4}-\d{2}-\d{2}$/; +const DATE_TIME = /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}(:\d{2}(\.\d+)?)?(Z|[+-]\d{2}:\d{2})$/; +const FIELDS = ["summary", "description", "location", "start", "end"] as const; + +export function createGcalendarConnector(cfg: GcalendarConfig): Connector { + if (!cfg.token) throw new Error("gcalendar connector needs a `token`"); + const f = cfg.fetch ?? fetch; + const base = (cfg.baseUrl ?? "https://www.googleapis.com/calendar/v3").replace(/\/+$/, ""); + const secrets = [cfg.token]; + const timeoutMs = cfg.timeoutMs ?? 30000; + + async function api( + method: "GET" | "POST" | "PATCH" | "DELETE", + p: string, + opts: { body?: unknown; tolerate?: number[] } = {}, + ): Promise<{ status: number; json: Record }> { + const label = `Google Calendar ${method} ${p.split("?")[0]}`; + const res = await send( + f, + base + p, + { + method, + headers: { + Authorization: `Bearer ${cfg.token}`, + Accept: "application/json", + ...(opts.body !== undefined ? { "Content-Type": "application/json" } : {}), + }, + body: opts.body !== undefined ? JSON.stringify(opts.body) : undefined, + }, + { label, mutating: method !== "GET", secrets, timeoutMs }, + ); + if (!res.ok && !(opts.tolerate ?? []).includes(res.status)) { + throw new HttpError(`${label} failed with status ${res.status}: ${snippet(res, secrets)}`, res.status); + } + return { status: res.status, json: asRecord(res.json) }; + } + + const calendarOf = (args: Record) => + optMatches(args, "calendar_id", CALENDAR_ID, "calendar id") ?? "primary"; + const calPath = (id: string) => `/calendars/${encodeURIComponent(id)}/events`; + + /** `2026-05-01` is an all-day date; anything else must be an RFC 3339 date-time. */ + function when(args: Record, key: "start" | "end", zone: string | undefined): Record { + const v = str(args, key, { max: 64 }); + if (DATE.test(v)) return { date: v }; + if (!DATE_TIME.test(v)) throw new Error(`\`${key}\` must be a date (2026-05-01) or an RFC 3339 date-time`); + return zone ? { dateTime: v, timeZone: zone } : { dateTime: v }; + } + + /** PATCH merges nested objects, so the unused date/dateTime key must be cleared explicitly. */ + const exclusive = (t: unknown): unknown => { + if (typeof t !== "object" || t === null || Array.isArray(t)) return t; + const r = t as Record; + if (typeof r.dateTime === "string") return { ...r, date: null }; + if (typeof r.date === "string") return { ...r, dateTime: null }; + return t; + }; + + const cal = { type: "string", description: "Calendar ID; default primary" }; + const eventId = { type: "string" }; + const timeProps = { + start: { type: "string", description: "Date or RFC 3339 date-time" }, + end: { type: "string", description: "Date or RFC 3339 date-time" }, + time_zone: { type: "string", description: "IANA zone such as Europe/Berlin" }, + }; + + return defineConnector("gcalendar", [ + { + name: "list_events", + description: "List events, optionally within a time window.", + effect: "read_only", + permission: "gcalendar:events:read", + properties: { + calendar_id: cal, + time_min: { type: "string", description: "RFC 3339" }, + time_max: { type: "string", description: "RFC 3339" }, + max_results: { type: "integer", minimum: 1, maximum: 250 }, + page_token: { type: "string" }, + }, + handler: async (args) => { + const q = new URLSearchParams({ singleEvents: "true", orderBy: "startTime" }); + const tmin = optMatches(args, "time_min", DATE_TIME, "RFC 3339 date-time"); + const tmax = optMatches(args, "time_max", DATE_TIME, "RFC 3339 date-time"); + const max = optPosInt(args, "max_results"); + if (max !== undefined && max > 250) throw new Error("`max_results` must be at most 250"); + const token = optStr(args, "page_token", { max: 2048 }); + if (tmin) q.set("timeMin", tmin); + if (tmax) q.set("timeMax", tmax); + if (max) q.set("maxResults", String(max)); + if (token) q.set("pageToken", token); + const { json } = await api("GET", `${calPath(calendarOf(args))}?${q.toString()}`); + const events = (Array.isArray(json.items) ? json.items : []).map((e) => { + const r = asRecord(e); + return { event_id: r.id, summary: r.summary ?? null, start: r.start, end: r.end, status: r.status }; + }); + return { events, next_page_token: typeof json.nextPageToken === "string" ? json.nextPageToken : null }; + }, + }, + { + name: "create_event", + description: + "Create an event. No invitation emails are sent (sendUpdates=none). Rollback deletes the event.", + effect: "reversible", + permission: "gcalendar:events:write", + compensation: { + tool: "delete_event", + argument_map: { calendar_id: "$result.calendar_id", event_id: "$result.event_id" }, + }, + properties: { + calendar_id: cal, + summary: { type: "string" }, + description: { type: "string" }, + location: { type: "string" }, + ...timeProps, + }, + required: ["summary", "start", "end"], + handler: async (args) => { + const calendar_id = calendarOf(args); + const zone = optStr(args, "time_zone", { max: 64 }); + const body: Record = { + summary: str(args, "summary", { max: 1024 }), + start: when(args, "start", zone), + end: when(args, "end", zone), + }; + const description = optStr(args, "description", { max: 8192 }); + const location = optStr(args, "location", { max: 1024 }); + if (description !== undefined) body.description = description; + if (location !== undefined) body.location = location; + const { json } = await api("POST", `${calPath(calendar_id)}?sendUpdates=none`, { body }); + if (typeof json.id !== "string") throw new Error("Google Calendar response had no event id"); + return { calendar_id, event_id: json.id, html_link: json.htmlLink ?? null }; + }, + }, + { + name: "delete_event", + description: "Delete an event without notifying attendees. An event that is already gone counts as success.", + effect: "idempotent", + permission: "gcalendar:events:delete", + properties: { calendar_id: cal, event_id: eventId }, + required: ["event_id"], + handler: async (args) => { + const calendar_id = calendarOf(args); + const id = matches(args, "event_id", EVENT_ID, "event id"); + const r = await api("DELETE", `${calPath(calendar_id)}/${encodeURIComponent(id)}?sendUpdates=none`, { + tolerate: [404, 410], + }); + return { calendar_id, event_id: id, deleted: r.status !== 404 && r.status !== 410 }; + }, + }, + { + name: "update_event", + description: + "Change summary, description, location, start or end of an event (no notifications). The event is read first; the previous values of exactly the changed fields are returned so rollback can patch them back.", + effect: "reversible", + permission: "gcalendar:events:write", + compensation: { + tool: "restore_event", + argument_map: { + calendar_id: "$result.calendar_id", + event_id: "$result.event_id", + previous: "$result.previous", + }, + }, + properties: { + calendar_id: cal, + event_id: eventId, + summary: { type: "string" }, + description: { type: "string" }, + location: { type: "string" }, + ...timeProps, + }, + required: ["event_id"], + handler: async (args) => { + const calendar_id = calendarOf(args); + const id = matches(args, "event_id", EVENT_ID, "event id"); + const zone = optStr(args, "time_zone", { max: 64 }); + const patch: Record = {}; + for (const k of ["summary", "description", "location"] as const) { + const v = optStr(args, k, { max: k === "summary" || k === "location" ? 1024 : 8192 }); + if (v !== undefined) patch[k] = v; + } + if (args.start !== undefined) patch.start = exclusive(when(args, "start", zone)); + if (args.end !== undefined) patch.end = exclusive(when(args, "end", zone)); + if (Object.keys(patch).length === 0) throw new Error("nothing to update: give at least one field"); + const p = `${calPath(calendar_id)}/${encodeURIComponent(id)}`; + const current = (await api("GET", p)).json; + const previous: Record = {}; + for (const k of Object.keys(patch)) previous[k] = current[k] ?? null; + await api("PATCH", `${p}?sendUpdates=none`, { body: patch }); + return { calendar_id, event_id: id, previous }; + }, + }, + { + name: "restore_event", + description: + "Patch an event back to previously saved field values (a null removes the field). Safe to repeat. Used to undo update_event.", + effect: "idempotent", + permission: "gcalendar:events:write", + properties: { + calendar_id: cal, + event_id: eventId, + previous: { type: "object", description: "Field values saved by update_event" }, + }, + required: ["event_id", "previous"], + handler: async (args) => { + const calendar_id = calendarOf(args); + const id = matches(args, "event_id", EVENT_ID, "event id"); + const prev = args.previous; + if (typeof prev !== "object" || prev === null || Array.isArray(prev)) throw new Error("`previous` must be an object"); + const body: Record = {}; + for (const [k, v] of Object.entries(prev)) { + if (!(FIELDS as readonly string[]).includes(k)) throw new Error(`\`previous.${k}\` is not a restorable field`); + body[k] = k === "start" || k === "end" ? exclusive(v) : v; + } + if (Object.keys(body).length === 0) throw new Error("`previous` has no fields"); + await api("PATCH", `${calPath(calendar_id)}/${encodeURIComponent(id)}?sendUpdates=none`, { body }); + return { calendar_id, event_id: id, restored: Object.keys(body) }; + }, + }, + ]); +} diff --git a/packages/connectors/src/gcp.ts b/packages/connectors/src/gcp.ts new file mode 100644 index 0000000..8c655e0 --- /dev/null +++ b/packages/connectors/src/gcp.ts @@ -0,0 +1,274 @@ +import { EffectUncertainError } from "@multimindlab/reversi"; +import { + BACKUP_PREFIX, + HttpError, + asRecord, + defineConnector, + matches, + optPosInt, + optStr, + send, + snippet, + type Connector, + type Fetch, + type Sent, +} from "./common.js"; +import { + backupName, + contentBytes, + contentOut, + contentProps, + isBackupName, + maxBytes, + mediaType, + newBackupName, + objectName, +} from "./objects.js"; + +export interface GcpConfig { + /** An OAuth 2.0 access token with a Cloud Storage scope. */ + accessToken: string; + fetch?: Fetch; + baseUrl?: string; + timeoutMs?: number; +} + +const BUCKET = /^[a-z0-9][a-z0-9._-]{1,61}[a-z0-9]$/; +const MAX_REWRITE_ROUNDS = 1000; + +export function createGcpConnector(cfg: GcpConfig): Connector { + if (!cfg.accessToken) throw new Error("gcp connector needs an `accessToken`"); + const f = cfg.fetch ?? fetch; + const base = (cfg.baseUrl ?? "https://storage.googleapis.com").replace(/\/+$/, ""); + const secrets = [cfg.accessToken]; + const timeoutMs = cfg.timeoutMs ?? 30000; + + async function api( + method: "GET" | "POST" | "DELETE", + p: string, + o: { + query?: Array<[string, string]>; + body?: Uint8Array | string; + contentType?: string; + headers?: Record; + binary?: boolean; + tolerate?: number[]; + } = {}, + ): Promise { + const q = new URLSearchParams(o.query ?? []).toString(); + const label = `GCS ${method} ${p}`; + const res = await send( + f, + `${base}${p}${q ? `?${q}` : ""}`, + { + method, + headers: { + Authorization: `Bearer ${cfg.accessToken}`, + ...(o.contentType ? { "Content-Type": o.contentType } : {}), + ...(o.headers ?? {}), + }, + body: o.body, + }, + { label, mutating: method !== "GET", secrets, timeoutMs, binary: o.binary }, + ); + if (!res.ok && !(o.tolerate ?? []).includes(res.status)) { + throw new HttpError(`${label} failed with status ${res.status}: ${snippet(res, secrets)}`, res.status); + } + return res; + } + + const objPath = (bucket: string, name: string) => `/storage/v1/b/${bucket}/o/${encodeURIComponent(name)}`; + + /** Current generation of an object, or null if it does not exist. */ + async function generationOf(bucket: string, name: string): Promise { + const r = await api("GET", objPath(bucket, name), { query: [["fields", "generation"]], tolerate: [404] }); + if (r.status === 404) return null; + const g = asRecord(r.json).generation; + if (typeof g !== "string" && typeof g !== "number") throw new Error("Cloud Storage response had no generation"); + return String(g); + } + + /** Server-side copy that survives large objects: rewrite until `done`. */ + async function rewrite(bucket: string, from: string, to: string, sourceGeneration?: string): Promise { + let token: string | undefined; + for (let i = 0; i < MAX_REWRITE_ROUNDS; i++) { + const query: Array<[string, string]> = []; + if (sourceGeneration) query.push(["sourceGeneration", sourceGeneration]); + if (token) query.push(["rewriteToken", token]); + const r = await api("POST", `${objPath(bucket, from)}/rewriteTo/b/${bucket}/o/${encodeURIComponent(to)}`, { + query, + body: "{}", + contentType: "application/json", + }); + const j = asRecord(r.json); + if (j.done === true) return; + if (typeof j.rewriteToken !== "string") throw new Error("Cloud Storage rewrite did not finish and gave no token"); + token = j.rewriteToken; + } + throw new Error("Cloud Storage rewrite did not finish"); + } + + const bucketProp = { type: "string", description: "Bucket name" }; + const keyProp = { type: "string", description: "Object name" }; + const bucketOf = (args: Record) => matches(args, "bucket", BUCKET, "bucket name"); + const restoreMap = { bucket: "$args.bucket", key: "$args.key", backup_key: "$result.backup_key" }; + + async function dropBackup(bucket: string, backup_key: string | null): Promise { + if (backup_key) await api("DELETE", objPath(bucket, backup_key), { tolerate: [404] }).catch(() => undefined); + } + + return defineConnector("gcp", [ + { + name: "get_object", + description: "Read an object (at most max_bytes, default 5 MiB). Needs storage.objects.get.", + effect: "read_only", + permission: "gcp:storage:read", + properties: { bucket: bucketProp, key: keyProp, max_bytes: { type: "integer", minimum: 1 } }, + required: ["bucket", "key"], + handler: async (args) => { + const bucket = bucketOf(args); + const key = objectName(args, "key", { allowBackup: true }); + const max = maxBytes(args); + const r = await api("GET", objPath(bucket, key), { + query: [["alt", "media"]], + headers: { Range: `bytes=0-${max - 1}` }, + binary: true, + }); + const bytes = r.bytes ?? new Uint8Array(); + const total = /\/(\d+)$/.exec(r.headers.get("content-range") ?? "")?.[1]; + const size = total ? Number(total) : bytes.byteLength; + return { + bucket, + key, + ...contentOut(bytes), + size, + truncated: size > bytes.byteLength, + content_type: r.headers.get("content-type"), + }; + }, + }, + { + name: "list_objects", + description: "List objects. Needs storage.objects.list. Backup objects are hidden.", + effect: "read_only", + permission: "gcp:storage:read", + properties: { + bucket: bucketProp, + prefix: { type: "string" }, + max_results: { type: "integer", minimum: 1, maximum: 1000 }, + page_token: { type: "string" }, + }, + required: ["bucket"], + handler: async (args) => { + const bucket = bucketOf(args); + const query: Array<[string, string]> = [["fields", "items(name,size,updated,contentType),nextPageToken"]]; + const prefix = optStr(args, "prefix", { max: 1024 }); + const max = optPosInt(args, "max_results"); + if (max !== undefined && max > 1000) throw new Error("`max_results` must be at most 1000"); + const token = optStr(args, "page_token", { max: 2048 }); + if (prefix) query.push(["prefix", prefix]); + if (max) query.push(["maxResults", String(max)]); + if (token) query.push(["pageToken", token]); + const j = asRecord((await api("GET", `/storage/v1/b/${bucket}/o`, { query })).json); + const objects = (Array.isArray(j.items) ? j.items : []) + .map(asRecord) + .filter((o) => typeof o.name === "string" && !isBackupName(o.name)) + .map((o) => ({ key: o.name, size: Number(o.size ?? 0), updated: o.updated ?? null, content_type: o.contentType ?? null })); + return { objects, next_page_token: typeof j.nextPageToken === "string" ? j.nextPageToken : null }; + }, + }, + { + name: "put_object", + description: `Write an object. If one exists it is first copied to ${BACKUP_PREFIX} in the same bucket and the upload is conditional on that generation, so rollback restores exactly what was overwritten (or deletes the new object). Needs storage.objects.get, create, delete and list.`, + effect: "reversible", + permission: "gcp:storage:write", + compensation: { tool: "restore_object", argument_map: restoreMap }, + properties: { bucket: bucketProp, key: keyProp, ...contentProps }, + required: ["bucket", "key", "content"], + handler: async (args) => { + const bucket = bucketOf(args); + const key = objectName(args, "key"); + const body = contentBytes(args); + const type = mediaType(args); + const generation = await generationOf(bucket, key); + const existed = generation !== null; + const backup_key = existed ? newBackupName() : null; + if (backup_key) await rewrite(bucket, key, backup_key, generation!); + try { + const r = await api("POST", `/upload/storage/v1/b/${bucket}/o`, { + query: [ + ["uploadType", "media"], + ["name", key], + ["ifGenerationMatch", generation ?? "0"], + ], + body, + contentType: type, + }); + return { bucket, key, existed, backup_key, generation: asRecord(r.json).generation ?? null }; + } catch (e) { + if (e instanceof EffectUncertainError && backup_key) { + throw new EffectUncertainError(`${e.message}; the previous version is saved at ${backup_key}`); + } + if (!(e instanceof EffectUncertainError)) await dropBackup(bucket, backup_key); + throw e; + } + }, + }, + { + name: "delete_object", + description: `Delete an object. It is first copied to ${BACKUP_PREFIX} and the delete is conditional on that generation, so rollback restores what was deleted. Needs storage.objects.get, create, delete and list.`, + effect: "reversible", + permission: "gcp:storage:write", + compensation: { tool: "restore_object", argument_map: restoreMap }, + properties: { bucket: bucketProp, key: keyProp }, + required: ["bucket", "key"], + handler: async (args) => { + const bucket = bucketOf(args); + const key = objectName(args, "key"); + const generation = await generationOf(bucket, key); + if (generation === null) return { bucket, key, existed: false, backup_key: null }; + const backup_key = newBackupName(); + await rewrite(bucket, key, backup_key, generation); + try { + await api("DELETE", objPath(bucket, key), { query: [["ifGenerationMatch", generation]] }); + } catch (e) { + if (e instanceof EffectUncertainError) { + throw new EffectUncertainError(`${e.message}; the previous version is saved at ${backup_key}`); + } + await dropBackup(bucket, backup_key); + throw e; + } + return { bucket, key, existed: true, backup_key }; + }, + }, + { + name: "restore_object", + description: + "Undo put_object or delete_object: copy the saved backup back over the key, or delete the key when there was no previous object (backup_key null). Safe to repeat. The backup object is kept.", + effect: "idempotent", + permission: "gcp:storage:write", + properties: { + bucket: bucketProp, + key: keyProp, + backup_key: { type: ["string", "null"], description: "Backup object name, or null if the key did not exist" }, + }, + required: ["bucket", "key", "backup_key"], + handler: async (args) => { + const bucket = bucketOf(args); + const key = objectName(args, "key"); + const backup = backupName(args, "backup_key"); + if (backup === null) { + await api("DELETE", objPath(bucket, key), { tolerate: [404] }); + return { bucket, key, restored: true, action: "deleted" }; + } + try { + await rewrite(bucket, backup, key); + } catch (e) { + if (e instanceof HttpError && e.status === 404) throw new Error(`backup ${backup} was not found`); + throw e; + } + return { bucket, key, restored: true, action: "restored" }; + }, + }, + ]); +} diff --git a/packages/connectors/src/gitlab.ts b/packages/connectors/src/gitlab.ts new file mode 100644 index 0000000..a1d3874 --- /dev/null +++ b/packages/connectors/src/gitlab.ts @@ -0,0 +1,219 @@ +import { + HttpError, + asRecord, + defineConnector, + matches, + optStr, + optStrList, + posInt, + send, + snippet, + str, + type Connector, + type Fetch, +} from "./common.js"; + +export interface GitlabConfig { + token: string; + fetch?: Fetch; + /** Instance root such as https://gitlab.example.com (self-hosted). Default https://gitlab.com. `/api/v4` is added. */ + baseUrl?: string; + timeoutMs?: number; +} + +const PROJECT = /^(\d{1,20}|[A-Za-z0-9._-]+(\/[A-Za-z0-9._-]+)+)$/; + +function branchName(args: Record, key: string): string { + const v = str(args, key, { max: 250 }); + if ( + !/^[A-Za-z0-9._/-]+$/.test(v) || + v.includes("..") || + v.includes("//") || + v.startsWith("/") || + v.endsWith("/") || + v.endsWith(".lock") || + v.endsWith(".") || + v.startsWith("-") + ) { + throw new Error(`\`${key}\` is not a valid branch name`); + } + return v; +} + +export function createGitlabConnector(cfg: GitlabConfig): Connector { + if (!cfg.token) throw new Error("gitlab connector needs a `token`"); + const f = cfg.fetch ?? fetch; + let base = (cfg.baseUrl ?? "https://gitlab.com").replace(/\/+$/, ""); + if (!base.endsWith("/api/v4")) base += "/api/v4"; + const secrets = [cfg.token]; + const timeoutMs = cfg.timeoutMs ?? 30000; + + async function api( + method: "GET" | "POST" | "PUT" | "DELETE", + p: string, + opts: { body?: unknown; tolerate?: number[] } = {}, + ): Promise<{ status: number; json: Record }> { + const label = `GitLab ${method} ${p}`; + const res = await send( + f, + base + p, + { + method, + headers: { + "PRIVATE-TOKEN": cfg.token, + Accept: "application/json", + ...(opts.body !== undefined ? { "Content-Type": "application/json" } : {}), + }, + body: opts.body !== undefined ? JSON.stringify(opts.body) : undefined, + }, + { label, mutating: method !== "GET", secrets, timeoutMs }, + ); + if (!res.ok && !(opts.tolerate ?? []).includes(res.status)) { + throw new HttpError(`${label} failed with status ${res.status}: ${snippet(res, secrets)}`, res.status); + } + return { status: res.status, json: asRecord(res.json) }; + } + + const projectOf = (args: Record) => { + const project = matches(args, "project", PROJECT, "project id or path"); + if (project.split("/").some((s) => s === "." || s === "..")) throw new Error("`project` is not a valid project id or path"); + return { project, p: `/projects/${encodeURIComponent(project)}` }; + }; + + const target = { project: { type: "string", description: "Numeric project ID or full path such as group/project" } }; + const iid = { type: "integer", minimum: 1, description: "Issue IID (the number shown in the project)" }; + + return defineConnector("gitlab", [ + { + name: "get_issue", + description: "Read an issue.", + effect: "read_only", + permission: "gitlab:issues:read", + properties: { ...target, issue_iid: iid }, + required: ["project", "issue_iid"], + handler: async (args) => { + const { p } = projectOf(args); + const n = posInt(args, "issue_iid"); + const { json } = await api("GET", `${p}/issues/${n}`); + return { + issue_iid: n, + title: json.title, + state: json.state, + description: json.description ?? null, + labels: Array.isArray(json.labels) ? json.labels : [], + web_url: json.web_url, + }; + }, + }, + { + name: "create_issue", + description: "Open an issue. Rollback closes it.", + effect: "reversible", + permission: "gitlab:issues:write", + compensation: { + tool: "close_issue", + argument_map: { project: "$result.project", issue_iid: "$result.issue_iid" }, + }, + properties: { + ...target, + title: { type: "string" }, + description: { type: "string" }, + labels: { type: "array", items: { type: "string" } }, + }, + required: ["project", "title"], + handler: async (args) => { + const { project, p } = projectOf(args); + const body: Record = { title: str(args, "title", { max: 255 }) }; + const description = optStr(args, "description", { max: 1000000 }); + if (description !== undefined) body.description = description; + const labels = optStrList(args, "labels"); + if (labels) body.labels = labels.join(","); + const { json } = await api("POST", `${p}/issues`, { body }); + if (typeof json.iid !== "number") throw new Error("GitLab response had no issue iid"); + return { project, issue_iid: json.iid, web_url: json.web_url, state: json.state }; + }, + }, + { + name: "close_issue", + description: "Close an issue. Closing a closed issue succeeds.", + effect: "idempotent", + permission: "gitlab:issues:write", + properties: { ...target, issue_iid: iid }, + required: ["project", "issue_iid"], + handler: async (args) => { + const { p } = projectOf(args); + const n = posInt(args, "issue_iid"); + const { json } = await api("PUT", `${p}/issues/${n}`, { body: { state_event: "close" } }); + return { issue_iid: n, state: json.state ?? "closed" }; + }, + }, + { + name: "add_note", + description: "Comment on an issue. Rollback deletes the note.", + effect: "reversible", + permission: "gitlab:issues:write", + compensation: { + tool: "delete_note", + argument_map: { project: "$args.project", issue_iid: "$args.issue_iid", note_id: "$result.note_id" }, + }, + properties: { ...target, issue_iid: iid, body: { type: "string" } }, + required: ["project", "issue_iid", "body"], + handler: async (args) => { + const { p } = projectOf(args); + const n = posInt(args, "issue_iid"); + const { json } = await api("POST", `${p}/issues/${n}/notes`, { body: { body: str(args, "body", { max: 1000000 }) } }); + if (typeof json.id !== "number") throw new Error("GitLab response had no note id"); + return { note_id: json.id }; + }, + }, + { + name: "delete_note", + description: "Delete an issue note. A note that is already gone counts as success.", + effect: "idempotent", + permission: "gitlab:issues:write", + properties: { ...target, issue_iid: iid, note_id: { type: "integer", minimum: 1 } }, + required: ["project", "issue_iid", "note_id"], + handler: async (args) => { + const { p } = projectOf(args); + const n = posInt(args, "issue_iid"); + const id = posInt(args, "note_id"); + const r = await api("DELETE", `${p}/issues/${n}/notes/${id}`, { tolerate: [404] }); + return { note_id: id, deleted: r.status !== 404 }; + }, + }, + { + name: "create_branch", + description: "Create a branch from a branch, tag or commit. Rollback deletes the branch.", + effect: "reversible", + permission: "gitlab:branch:write", + compensation: { + tool: "delete_branch", + argument_map: { project: "$args.project", branch: "$args.branch" }, + }, + properties: { ...target, branch: { type: "string" }, ref: { type: "string", description: "Branch, tag or commit SHA to branch from" } }, + required: ["project", "branch", "ref"], + handler: async (args) => { + const { p } = projectOf(args); + const branch = branchName(args, "branch"); + const ref = str(args, "ref", { max: 250 }); + const { json } = await api("POST", `${p}/repository/branches`, { body: { branch, ref } }); + return { branch, commit: asRecord(json.commit).id ?? null }; + }, + }, + { + name: "delete_branch", + description: + "Delete a branch. A branch that is already gone counts as success. Used to undo create_branch; deleting a branch with commits loses them unless you keep the SHA, so withhold this permission from agents.", + effect: "idempotent", + permission: "gitlab:branch:delete", + properties: { ...target, branch: { type: "string" } }, + required: ["project", "branch"], + handler: async (args) => { + const { p } = projectOf(args); + const branch = branchName(args, "branch"); + const r = await api("DELETE", `${p}/repository/branches/${encodeURIComponent(branch)}`, { tolerate: [404] }); + return { branch, deleted: r.status !== 404 }; + }, + }, + ]); +} diff --git a/packages/connectors/src/index.ts b/packages/connectors/src/index.ts index 393dc11..db64c1b 100644 --- a/packages/connectors/src/index.ts +++ b/packages/connectors/src/index.ts @@ -1,8 +1,18 @@ import type { Connector } from "./common.js"; +import { createAwsConnector, type AwsConfig } from "./aws.js"; +import { createAzureConnector, type AzureConfig } from "./azure.js"; import { createFilesystemConnector, type FilesystemConfig } from "./filesystem.js"; +import { createGcalendarConnector, type GcalendarConfig } from "./gcalendar.js"; +import { createGcpConnector, type GcpConfig } from "./gcp.js"; import { createGithubConnector, type GithubConfig } from "./github.js"; +import { createGitlabConnector, type GitlabConfig } from "./gitlab.js"; +import { createJiraConnector, type JiraConfig } from "./jira.js"; +import { createKubernetesConnector, type KubernetesConfig } from "./kubernetes.js"; +import { createLinearConnector, type LinearConfig } from "./linear.js"; +import { createPostgresConnector, type PostgresConfig } from "./postgres.js"; import { createSlackConnector, type SlackConfig } from "./slack.js"; import { createStripeConnector, type StripeConfig } from "./stripe.js"; +import { createTeamsConnector, type TeamsConfig } from "./teams.js"; export { HttpError, @@ -12,17 +22,58 @@ export { type ToolHandler, } from "./common.js"; export { compensatorFor } from "./compensate.js"; +export { createAwsConnector, type AwsConfig } from "./aws.js"; +export { createAzureConnector, type AzureConfig } from "./azure.js"; export { createFilesystemConnector, type FilesystemConfig } from "./filesystem.js"; +export { createGcalendarConnector, type GcalendarConfig } from "./gcalendar.js"; +export { createGcpConnector, type GcpConfig } from "./gcp.js"; export { createGithubConnector, type GithubConfig } from "./github.js"; +export { createGitlabConnector, type GitlabConfig } from "./gitlab.js"; +export { createJiraConnector, type JiraConfig } from "./jira.js"; +export { caFetch, createKubernetesConnector, type KubernetesConfig } from "./kubernetes.js"; +export { createLinearConnector, type LinearConfig } from "./linear.js"; +export { + createPgPool, + createPostgresConnector, + type PgClientLike, + type PgLike, + type PgResult, + type PostgresConfig, +} from "./postgres.js"; export { createConnectorServer } from "./server.js"; export { createSlackConnector, type SlackConfig } from "./slack.js"; export { createStripeConnector, type StripeConfig } from "./stripe.js"; +export { createTeamsConnector, type TeamsConfig } from "./teams.js"; +export { amzDateOf, signV4, signingKey, uriEncode, type SigV4Input, type SigV4Output } from "./sigv4.js"; +export { + CHECKS, + CHECK_IDS, + certificationFailures, + connectorSchemaHashes, + detectManifestDrift, + renderCertificationMatrix, + type Certification, + type CheckId, + type CheckResult, + type CheckStatus, + type ManifestDrift, +} from "./certify.js"; export interface AllConnectorsConfig { fs?: FilesystemConfig; github?: GithubConfig; slack?: SlackConfig; stripe?: StripeConfig; + gitlab?: GitlabConfig; + teams?: TeamsConfig; + linear?: LinearConfig; + jira?: JiraConfig; + gcalendar?: GcalendarConfig; + postgres?: PostgresConfig; + aws?: AwsConfig; + gcp?: GcpConfig; + azure?: AzureConfig; + kubernetes?: KubernetesConfig; } /** Build every connector that has a config entry. */ @@ -32,5 +83,15 @@ export function allConnectors(config: AllConnectorsConfig): Connector[] { if (config.github) out.push(createGithubConnector(config.github)); if (config.slack) out.push(createSlackConnector(config.slack)); if (config.stripe) out.push(createStripeConnector(config.stripe)); + if (config.gitlab) out.push(createGitlabConnector(config.gitlab)); + if (config.teams) out.push(createTeamsConnector(config.teams)); + if (config.linear) out.push(createLinearConnector(config.linear)); + if (config.jira) out.push(createJiraConnector(config.jira)); + if (config.gcalendar) out.push(createGcalendarConnector(config.gcalendar)); + if (config.postgres) out.push(createPostgresConnector(config.postgres)); + if (config.aws) out.push(createAwsConnector(config.aws)); + if (config.gcp) out.push(createGcpConnector(config.gcp)); + if (config.azure) out.push(createAzureConnector(config.azure)); + if (config.kubernetes) out.push(createKubernetesConnector(config.kubernetes)); return out; } diff --git a/packages/connectors/src/jira.ts b/packages/connectors/src/jira.ts new file mode 100644 index 0000000..969b19e --- /dev/null +++ b/packages/connectors/src/jira.ts @@ -0,0 +1,245 @@ +import { + HttpError, + asRecord, + defineConnector, + matches, + optStr, + redact, + send, + snippet, + str, + type Connector, + type Fetch, +} from "./common.js"; + +export interface JiraConfig { + /** Site root such as https://yourco.atlassian.net */ + baseUrl: string; + email: string; + apiToken: string; + fetch?: Fetch; + timeoutMs?: number; +} + +const ISSUE_KEY = /^[A-Z][A-Z0-9_]{1,9}-\d{1,9}$/; +const PROJECT_KEY = /^[A-Z][A-Z0-9_]{1,9}$/; +const NUMERIC_ID = /^\d{1,18}$/; + +/** Atlassian Document Format for plain text: one paragraph per line. */ +function adf(text: string): unknown { + return { + type: "doc", + version: 1, + content: text.split("\n").map((line) => ({ + type: "paragraph", + content: line ? [{ type: "text", text: line }] : [], + })), + }; +} + +export function createJiraConnector(cfg: JiraConfig): Connector { + if (!cfg.baseUrl) throw new Error("jira connector needs a `baseUrl`"); + if (!cfg.email) throw new Error("jira connector needs an `email`"); + if (!cfg.apiToken) throw new Error("jira connector needs an `apiToken`"); + const f = cfg.fetch ?? fetch; + const base = cfg.baseUrl.replace(/\/+$/, ""); + let parsed: URL; + try { + parsed = new URL(base); + } catch { + throw new Error("jira `baseUrl` must be an http(s) URL"); + } + if (!/^https?:$/.test(parsed.protocol) || parsed.username || parsed.password) { + throw new Error("jira `baseUrl` must be an http(s) URL without credentials"); + } + const basic = Buffer.from(`${cfg.email}:${cfg.apiToken}`).toString("base64"); + const secrets = [cfg.apiToken, basic]; + const timeoutMs = cfg.timeoutMs ?? 30000; + + async function api( + method: "GET" | "POST" | "DELETE", + p: string, + opts: { body?: unknown; tolerate?: number[] } = {}, + ): Promise<{ status: number; json: Record }> { + const label = `Jira ${method} ${p.split("?")[0]}`; + const res = await send( + f, + `${base}/rest/api/3${p}`, + { + method, + headers: { + Authorization: `Basic ${basic}`, + Accept: "application/json", + ...(opts.body !== undefined ? { "Content-Type": "application/json" } : {}), + }, + body: opts.body !== undefined ? JSON.stringify(opts.body) : undefined, + }, + { label, mutating: method !== "GET", secrets, timeoutMs }, + ); + if (!res.ok && !(opts.tolerate ?? []).includes(res.status)) { + let msg = snippet(res, secrets); + const messages = asRecord(res.json).errorMessages; + if (Array.isArray(messages) && messages.length > 0) msg = redact(String(messages[0]), secrets).slice(0, 200); + throw new HttpError(`${label} failed with status ${res.status}: ${msg}`, res.status); + } + return { status: res.status, json: asRecord(res.json) }; + } + + const keyProp = { type: "string", description: "Issue key such as PROJ-123" }; + const statusOf = (issue: Record) => { + const s = asRecord(asRecord(issue.fields).status); + return { id: typeof s.id === "string" ? s.id : null, name: typeof s.name === "string" ? s.name : null }; + }; + + return defineConnector("jira", [ + { + name: "get_issue", + description: "Read an issue's summary and status.", + effect: "read_only", + permission: "jira:issues:read", + properties: { issue_key: keyProp }, + required: ["issue_key"], + handler: async (args) => { + const key = matches(args, "issue_key", ISSUE_KEY, "issue key"); + const { json } = await api("GET", `/issue/${key}?fields=summary,status`); + const st = statusOf(json); + return { issue_key: key, summary: asRecord(json.fields).summary ?? null, status: st.name, status_id: st.id }; + }, + }, + { + name: "create_issue", + description: "Create an issue. Rollback deletes it.", + effect: "reversible", + permission: "jira:issues:write", + compensation: { tool: "delete_issue", argument_map: { issue_key: "$result.issue_key" } }, + properties: { + project_key: { type: "string" }, + summary: { type: "string" }, + issue_type: { type: "string", description: "Issue type name, default Task" }, + description: { type: "string", description: "Plain text" }, + }, + required: ["project_key", "summary"], + handler: async (args) => { + const fields: Record = { + project: { key: matches(args, "project_key", PROJECT_KEY, "project key") }, + summary: str(args, "summary", { max: 255 }), + issuetype: { name: optStr(args, "issue_type", { max: 60 }) ?? "Task" }, + }; + const description = optStr(args, "description", { max: 32000 }); + if (description !== undefined) fields.description = adf(description); + const { json } = await api("POST", "/issue", { body: { fields } }); + if (typeof json.key !== "string") throw new Error("Jira response had no issue key"); + return { issue_key: json.key, issue_id: json.id ?? null }; + }, + }, + { + name: "delete_issue", + description: + "Delete an issue permanently (subtasks are not deleted, Jira refuses if there are any). An issue that is already gone counts as success. Used to undo create_issue; deletion cannot be undone, so withhold this permission from agents.", + effect: "idempotent", + permission: "jira:issues:delete", + properties: { issue_key: keyProp }, + required: ["issue_key"], + handler: async (args) => { + const key = matches(args, "issue_key", ISSUE_KEY, "issue key"); + const r = await api("DELETE", `/issue/${key}?deleteSubtasks=false`, { tolerate: [404] }); + return { issue_key: key, deleted: r.status !== 404 }; + }, + }, + { + name: "add_comment", + description: "Comment on an issue. Rollback deletes the comment.", + effect: "reversible", + permission: "jira:comments:write", + compensation: { + tool: "delete_comment", + argument_map: { issue_key: "$args.issue_key", comment_id: "$result.comment_id" }, + }, + properties: { issue_key: keyProp, body: { type: "string", description: "Plain text" } }, + required: ["issue_key", "body"], + handler: async (args) => { + const key = matches(args, "issue_key", ISSUE_KEY, "issue key"); + const { json } = await api("POST", `/issue/${key}/comment`, { body: { body: adf(str(args, "body", { max: 32000 })) } }); + if (typeof json.id !== "string") throw new Error("Jira response had no comment id"); + return { comment_id: json.id }; + }, + }, + { + name: "delete_comment", + description: "Delete a comment. A comment that is already gone counts as success.", + effect: "idempotent", + permission: "jira:comments:write", + properties: { issue_key: keyProp, comment_id: { type: "string" } }, + required: ["issue_key", "comment_id"], + handler: async (args) => { + const key = matches(args, "issue_key", ISSUE_KEY, "issue key"); + const id = matches(args, "comment_id", NUMERIC_ID, "comment id"); + const r = await api("DELETE", `/issue/${key}/comment/${id}`, { tolerate: [404] }); + return { comment_id: id, deleted: r.status !== 404 }; + }, + }, + { + name: "transition_issue", + description: + "Move an issue through a workflow transition (by transition id or name, or the name of the status it leads to). The previous status is read first and returned; rollback transitions back to it if the workflow allows that.", + effect: "compensable", + permission: "jira:issues:transition", + compensation: { + tool: "restore_status", + argument_map: { issue_key: "$args.issue_key", status_id: "$result.previous_status_id" }, + }, + properties: { + issue_key: keyProp, + transition: { type: "string", description: "Transition id, transition name, or target status name" }, + }, + required: ["issue_key", "transition"], + handler: async (args) => { + const key = matches(args, "issue_key", ISSUE_KEY, "issue key"); + const wanted = str(args, "transition", { max: 100 }); + const before = statusOf((await api("GET", `/issue/${key}?fields=status`)).json); + if (before.id === null) throw new Error("Jira response had no status"); + const list = (await api("GET", `/issue/${key}/transitions`)).json.transitions; + const options = (Array.isArray(list) ? list : []).map(asRecord); + const lower = wanted.toLowerCase(); + const match = + options.find((t) => t.id === wanted) ?? + options.find((t) => String(t.name ?? "").toLowerCase() === lower) ?? + options.find((t) => String(asRecord(t.to).name ?? "").toLowerCase() === lower); + if (!match || typeof match.id !== "string") { + throw new Error(`no transition \`${wanted}\` is available from status ${before.name ?? before.id}`); + } + await api("POST", `/issue/${key}/transitions`, { body: { transition: { id: match.id } } }); + const to = asRecord(match.to); + return { + issue_key: key, + previous_status: before.name, + previous_status_id: before.id, + status: to.name ?? null, + status_id: to.id ?? null, + }; + }, + }, + { + name: "restore_status", + description: + "Move an issue back to a status by finding a transition that leads to it. Succeeds without changes if the issue is already there; fails if the workflow has no such transition. Used to undo transition_issue.", + effect: "idempotent", + permission: "jira:issues:transition", + properties: { issue_key: keyProp, status_id: { type: "string" } }, + required: ["issue_key", "status_id"], + handler: async (args) => { + const key = matches(args, "issue_key", ISSUE_KEY, "issue key"); + const statusId = matches(args, "status_id", NUMERIC_ID, "status id"); + const now = statusOf((await api("GET", `/issue/${key}?fields=status`)).json); + if (now.id === statusId) return { issue_key: key, status_id: statusId, moved: false }; + const list = (await api("GET", `/issue/${key}/transitions`)).json.transitions; + const match = (Array.isArray(list) ? list : []).map(asRecord).find((t) => asRecord(t.to).id === statusId); + if (!match || typeof match.id !== "string") { + throw new Error(`the workflow has no transition from status ${now.name ?? now.id} back to status ${statusId}`); + } + await api("POST", `/issue/${key}/transitions`, { body: { transition: { id: match.id } } }); + return { issue_key: key, status_id: statusId, moved: true }; + }, + }, + ]); +} diff --git a/packages/connectors/src/kubernetes.ts b/packages/connectors/src/kubernetes.ts new file mode 100644 index 0000000..449fc19 --- /dev/null +++ b/packages/connectors/src/kubernetes.ts @@ -0,0 +1,320 @@ +import { readFileSync } from "node:fs"; +import http from "node:http"; +import https from "node:https"; +import { + HttpError, + asRecord, + defineConnector, + matches, + nonNegInt, + optMatches, + optPosInt, + send, + snippet, + type Connector, + type Fetch, + type Sent, +} from "./common.js"; + +export interface KubernetesConfig { + /** API server URL such as https://10.0.0.1:6443 */ + apiServer: string; + /** A bearer token (service account token). */ + token: string; + /** Path to the cluster CA certificate (PEM). When set and no `fetch` is injected, requests use node:https with this CA. */ + caCertPath?: string; + fetch?: Fetch; + timeoutMs?: number; +} + +const NAMESPACE = /^[a-z0-9]([-a-z0-9]{0,61}[a-z0-9])?$/; +const NAME = /^[a-z0-9]([-a-z0-9.]{0,251}[a-z0-9])?$/; +const UID = /^[0-9a-fA-F-]{8,64}$/; +const SELECTOR = /^[A-Za-z0-9_./=!,() -]{1,512}$/; +const DATA_KEY = /^[-._a-zA-Z0-9]{1,253}$/; +const MAX_CONFIGMAP_BYTES = 900 * 1024; +const MAX_REPLICAS = 10000; + +/** A fetch built on node:http(s) that trusts the given CA, for clusters with a private certificate authority. */ +export function caFetch(ca: string | Buffer): Fetch { + return ((input: string | URL | Request, init?: RequestInit) => + new Promise((resolve, reject) => { + const url = new URL(typeof input === "string" || input instanceof URL ? input : input.url); + const tls = url.protocol === "https:"; + const headers: Record = {}; + new Headers(init?.headers).forEach((v, k) => (headers[k] = v)); + const signal = init?.signal; + if (signal?.aborted) return reject(signal.reason instanceof Error ? signal.reason : new Error("aborted")); + const req = (tls ? https : http).request( + url, + { method: init?.method ?? "GET", headers, ...(tls ? { ca } : {}) }, + (res) => { + const chunks: Buffer[] = []; + res.on("data", (c: Buffer) => chunks.push(c)); + res.on("error", reject); + res.on("end", () => { + const h = new Headers(); + for (const [k, v] of Object.entries(res.headers)) { + if (Array.isArray(v)) for (const x of v) h.append(k, x); + else if (v !== undefined) h.set(k, v); + } + const status = res.statusCode ?? 0; + resolve(new Response(status === 204 || status === 304 ? null : Buffer.concat(chunks), { status, headers: h })); + }); + }, + ); + req.on("error", reject); + signal?.addEventListener("abort", () => req.destroy(signal.reason instanceof Error ? signal.reason : new Error("aborted")), { + once: true, + }); + if (init?.body !== undefined && init.body !== null) req.write(typeof init.body === "string" ? init.body : Buffer.from(init.body as Uint8Array)); + req.end(); + })) as Fetch; +} + +export function createKubernetesConnector(cfg: KubernetesConfig): Connector { + if (!cfg.apiServer) throw new Error("kubernetes connector needs an `apiServer`"); + if (!cfg.token) throw new Error("kubernetes connector needs a `token`"); + const f = cfg.fetch ?? (cfg.caCertPath ? caFetch(readFileSync(cfg.caCertPath)) : fetch); + const base = cfg.apiServer.replace(/\/+$/, ""); + const secrets = [cfg.token]; + const timeoutMs = cfg.timeoutMs ?? 30000; + + async function api( + method: "GET" | "POST" | "PATCH" | "DELETE", + p: string, + opts: { body?: unknown; query?: Array<[string, string]>; tolerate?: number[] } = {}, + ): Promise }> { + const label = `Kubernetes ${method} ${p}`; + const q = new URLSearchParams(opts.query ?? []).toString(); + const res = await send( + f, + `${base}${p}${q ? `?${q}` : ""}`, + { + method, + headers: { + Authorization: `Bearer ${cfg.token}`, + Accept: "application/json", + ...(opts.body !== undefined + ? { "Content-Type": method === "PATCH" ? "application/merge-patch+json" : "application/json" } + : {}), + }, + body: opts.body !== undefined ? JSON.stringify(opts.body) : undefined, + }, + { label, mutating: method !== "GET", secrets, timeoutMs }, + ); + if (!res.ok && !(opts.tolerate ?? []).includes(res.status)) { + throw new HttpError(`${label} failed with status ${res.status}: ${snippet(res, secrets)}`, res.status); + } + return { ...res, object: asRecord(res.json) }; + } + + const ns = (args: Record) => matches(args, "namespace", NAMESPACE, "namespace (DNS-1123 label)"); + const nm = (args: Record) => matches(args, "name", NAME, "name (DNS-1123 subdomain)"); + const deployPath = (namespace: string, name: string) => `/apis/apps/v1/namespaces/${namespace}/deployments/${name}`; + + const target = { + namespace: { type: "string", description: "Namespace (DNS-1123 label)" }, + name: { type: "string", description: "Object name (DNS-1123 subdomain)" }, + }; + const replicasProp = { type: "integer", minimum: 0, maximum: MAX_REPLICAS }; + const replicasArg = (args: Record) => { + const n = nonNegInt(args, "replicas"); + if (n > MAX_REPLICAS) throw new Error(`\`replicas\` must be at most ${MAX_REPLICAS}`); + return n; + }; + + return defineConnector("kubernetes", [ + { + name: "get_deployment", + description: "Read a deployment's replica counts and images. RBAC: get on deployments.apps.", + effect: "read_only", + permission: "kubernetes:deployments:read", + properties: target, + required: ["namespace", "name"], + handler: async (args) => { + const namespace = ns(args); + const name = nm(args); + const d = (await api("GET", deployPath(namespace, name))).object; + const spec = asRecord(d.spec); + const status = asRecord(d.status); + const containers = asRecord(asRecord(spec.template).spec).containers; + return { + namespace, + name, + replicas: spec.replicas ?? 1, + ready_replicas: status.readyReplicas ?? 0, + updated_replicas: status.updatedReplicas ?? 0, + available_replicas: status.availableReplicas ?? 0, + generation: asRecord(d.metadata).generation ?? null, + images: (Array.isArray(containers) ? containers : []).map((c) => asRecord(c).image), + }; + }, + }, + { + name: "list_pods", + description: "List pods in a namespace, optionally by label selector. RBAC: list on pods.", + effect: "read_only", + permission: "kubernetes:pods:read", + properties: { + namespace: target.namespace, + label_selector: { type: "string", description: "Such as app=web,tier!=cache" }, + limit: { type: "integer", minimum: 1, maximum: 500 }, + }, + required: ["namespace"], + handler: async (args) => { + const namespace = ns(args); + const selector = optMatches(args, "label_selector", SELECTOR, "label selector"); + const limit = optPosInt(args, "limit"); + if (limit !== undefined && limit > 500) throw new Error("`limit` must be at most 500"); + const query: Array<[string, string]> = []; + if (selector) query.push(["labelSelector", selector]); + if (limit) query.push(["limit", String(limit)]); + const j = (await api("GET", `/api/v1/namespaces/${namespace}/pods`, { query })).object; + const pods = (Array.isArray(j.items) ? j.items : []).map((i) => { + const r = asRecord(i); + const statuses = (Array.isArray(asRecord(r.status).containerStatuses) ? (asRecord(r.status).containerStatuses as unknown[]) : []).map(asRecord); + return { + name: asRecord(r.metadata).name, + phase: asRecord(r.status).phase ?? null, + node_name: asRecord(r.spec).nodeName ?? null, + ready: statuses.length > 0 && statuses.every((s) => s.ready === true), + restarts: statuses.reduce((n, s) => n + (typeof s.restartCount === "number" ? s.restartCount : 0), 0), + }; + }); + const next = asRecord(j.metadata).continue; + return { pods, continue: typeof next === "string" && next ? next : null }; + }, + }, + { + name: "scale_deployment", + description: + "Set a deployment's replica count through the scale subresource. The current count is read first and returned as previous_replicas; rollback scales back to it. RBAC: get and patch on deployments/scale.", + effect: "reversible", + permission: "kubernetes:deployments:scale", + compensation: { + tool: "restore_replicas", + argument_map: { namespace: "$args.namespace", name: "$args.name", replicas: "$result.previous_replicas" }, + }, + properties: { ...target, replicas: replicasProp }, + required: ["namespace", "name", "replicas"], + handler: async (args) => { + const namespace = ns(args); + const name = nm(args); + const replicas = replicasArg(args); + const p = `${deployPath(namespace, name)}/scale`; + const current = asRecord((await api("GET", p)).object.spec).replicas; + if (typeof current !== "number") throw new Error("Kubernetes scale response had no spec.replicas"); + await api("PATCH", p, { body: { spec: { replicas } } }); + return { namespace, name, previous_replicas: current, replicas }; + }, + }, + { + name: "restore_replicas", + description: "Set a deployment's replica count back to a saved value. Safe to repeat. Used to undo scale_deployment.", + effect: "idempotent", + permission: "kubernetes:deployments:scale", + properties: { ...target, replicas: replicasProp }, + required: ["namespace", "name", "replicas"], + handler: async (args) => { + const namespace = ns(args); + const name = nm(args); + const replicas = replicasArg(args); + await api("PATCH", `${deployPath(namespace, name)}/scale`, { body: { spec: { replicas } } }); + return { namespace, name, replicas }; + }, + }, + { + name: "create_configmap", + description: + "Create a ConfigMap. Fails if it already exists. Rollback deletes it, guarded by its UID so a different object with the same name is never deleted. RBAC: create on configmaps.", + effect: "reversible", + permission: "kubernetes:configmaps:write", + compensation: { + tool: "delete_configmap", + argument_map: { namespace: "$args.namespace", name: "$args.name", uid: "$result.uid" }, + }, + properties: { ...target, data: { type: "object", additionalProperties: { type: "string" } } }, + required: ["namespace", "name", "data"], + handler: async (args) => { + const namespace = ns(args); + const name = nm(args); + const raw = args.data; + if (typeof raw !== "object" || raw === null || Array.isArray(raw)) throw new Error("`data` must be an object of strings"); + const data: Record = {}; + let size = 0; + for (const [k, v] of Object.entries(raw)) { + if (!DATA_KEY.test(k)) throw new Error(`\`data\` has an invalid key \`${k.slice(0, 64)}\``); + if (typeof v !== "string") throw new Error(`\`data.${k}\` must be a string`); + size += Buffer.byteLength(k) + Buffer.byteLength(v); + data[k] = v; + } + if (size > MAX_CONFIGMAP_BYTES) throw new Error("`data` is larger than a ConfigMap may be"); + const { object } = await api("POST", `/api/v1/namespaces/${namespace}/configmaps`, { + body: { apiVersion: "v1", kind: "ConfigMap", metadata: { name }, data }, + }); + const uid = asRecord(object.metadata).uid; + if (typeof uid !== "string") throw new Error("Kubernetes response had no uid"); + return { namespace, name, uid, resource_version: asRecord(object.metadata).resourceVersion ?? null }; + }, + }, + { + name: "delete_configmap", + description: + "Delete a ConfigMap. With `uid`, only the object with that UID is deleted. A ConfigMap that is already gone, or was replaced by another with a different UID, counts as success. RBAC: delete on configmaps; the agent should not hold it.", + effect: "idempotent", + permission: "kubernetes:configmaps:delete", + properties: { ...target, uid: { type: "string" } }, + required: ["namespace", "name"], + handler: async (args) => { + const namespace = ns(args); + const name = nm(args); + const uid = optMatches(args, "uid", UID, "uid"); + const r = await api("DELETE", `/api/v1/namespaces/${namespace}/configmaps/${name}`, { + body: uid ? { apiVersion: "v1", kind: "DeleteOptions", preconditions: { uid } } : undefined, + tolerate: uid ? [404, 409] : [404], + }); + return { namespace, name, deleted: r.status !== 404 && r.status !== 409 }; + }, + }, + { + name: "rollout_restart", + description: + "Restart a deployment's pods by stamping the pod template (what `kubectl rollout restart` does). Pods are replaced and the old ones are gone, so it is irreversible and always needs approval. RBAC: patch on deployments.apps.", + effect: "irreversible", + approval: "always", + permission: "kubernetes:deployments:restart", + properties: target, + required: ["namespace", "name"], + handler: async (args) => { + const namespace = ns(args); + const name = nm(args); + const restarted_at = new Date().toISOString(); + await api("PATCH", deployPath(namespace, name), { + body: { spec: { template: { metadata: { annotations: { "kubectl.kubernetes.io/restartedAt": restarted_at } } } } }, + }); + return { namespace, name, restarted_at }; + }, + }, + { + name: "delete_pod", + description: + "Delete a pod. Its state is gone for good, so it is irreversible and always needs approval. RBAC: delete on pods.", + effect: "irreversible", + approval: "always", + permission: "kubernetes:pods:delete", + properties: { ...target, grace_period_seconds: { type: "integer", minimum: 0, maximum: 3600 } }, + required: ["namespace", "name"], + handler: async (args) => { + const namespace = ns(args); + const name = nm(args); + const grace = args.grace_period_seconds === undefined ? undefined : nonNegInt(args, "grace_period_seconds"); + if (grace !== undefined && grace > 3600) throw new Error("`grace_period_seconds` must be at most 3600"); + await api("DELETE", `/api/v1/namespaces/${namespace}/pods/${name}`, { + body: grace === undefined ? undefined : { apiVersion: "v1", kind: "DeleteOptions", gracePeriodSeconds: grace }, + }); + return { namespace, name, deleted: true }; + }, + }, + ]); +} diff --git a/packages/connectors/src/linear.ts b/packages/connectors/src/linear.ts new file mode 100644 index 0000000..fd1c9eb --- /dev/null +++ b/packages/connectors/src/linear.ts @@ -0,0 +1,208 @@ +import { + HttpError, + asRecord, + defineConnector, + matches, + optPosInt, + optStr, + redact, + send, + snippet, + str, + type Connector, + type Fetch, +} from "./common.js"; + +export interface LinearConfig { + /** A personal API key (sent as-is in `Authorization`), or an OAuth token prefixed with `Bearer `. */ + apiKey: string; + fetch?: Fetch; + baseUrl?: string; + timeoutMs?: number; +} + +const UUID = /^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$/; +const ISSUE_REF = /^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}|[A-Za-z][A-Za-z0-9]{0,9}-\d{1,9})$/; + +export function createLinearConnector(cfg: LinearConfig): Connector { + if (!cfg.apiKey) throw new Error("linear connector needs an `apiKey`"); + const f = cfg.fetch ?? fetch; + const url = (cfg.baseUrl ?? "https://api.linear.app/graphql").replace(/\/+$/, ""); + const secrets = [cfg.apiKey, cfg.apiKey.replace(/^Bearer\s+/i, "")]; + const timeoutMs = cfg.timeoutMs ?? 30000; + + /** One GraphQL operation. `tolerateNotFound` turns an "Entity not found" error into `notFound: true`. */ + async function gql( + op: string, + query: string, + variables: Record, + o: { mutating: boolean; tolerateNotFound?: boolean }, + ): Promise<{ data: Record; notFound: boolean }> { + const label = `Linear ${op}`; + const res = await send( + f, + url, + { + method: "POST", + headers: { Authorization: cfg.apiKey, "Content-Type": "application/json" }, + body: JSON.stringify({ query, variables }), + }, + { label, mutating: o.mutating, secrets, timeoutMs }, + ); + const body = asRecord(res.json); + const errors = Array.isArray(body.errors) ? body.errors : []; + if (errors.length > 0) { + const first = asRecord(errors[0]); + const message = redact(String(first.message ?? "unknown error"), secrets); + if (o.tolerateNotFound && /entity not found|not found/i.test(message)) return { data: {}, notFound: true }; + throw new HttpError(`${label} failed with status ${res.status}: ${message}`, res.status); + } + if (!res.ok) throw new HttpError(`${label} failed with status ${res.status}: ${snippet(res, secrets)}`, res.status); + return { data: asRecord(body.data), notFound: false }; + } + + const issueId = { type: "string", description: "Issue UUID" }; + const commentId = { type: "string", description: "Comment UUID" }; + + return defineConnector("linear", [ + { + name: "get_issue", + description: "Read an issue by UUID or identifier such as ENG-123.", + effect: "read_only", + permission: "linear:issues:read", + properties: { id: { type: "string", description: "Issue UUID or identifier such as ENG-123" } }, + required: ["id"], + handler: async (args) => { + const id = matches(args, "id", ISSUE_REF, "issue id or identifier"); + const { data } = await gql( + "issue", + "query Issue($id: String!) { issue(id: $id) { id identifier title description url archivedAt state { name } } }", + { id }, + { mutating: false }, + ); + const i = asRecord(data.issue); + if (typeof i.id !== "string") throw new Error("Linear response had no issue"); + return { + issue_id: i.id, + identifier: i.identifier, + title: i.title, + description: i.description ?? null, + state: asRecord(i.state).name ?? null, + archived: typeof i.archivedAt === "string", + url: i.url, + }; + }, + }, + { + name: "create_issue", + description: "Create an issue in a team. Rollback archives it.", + effect: "reversible", + permission: "linear:issues:write", + compensation: { tool: "archive_issue", argument_map: { issue_id: "$result.issue_id" } }, + properties: { + team_id: { type: "string", description: "Team UUID" }, + title: { type: "string" }, + description: { type: "string" }, + priority: { type: "integer", minimum: 1, maximum: 4, description: "1 urgent, 2 high, 3 normal, 4 low" }, + }, + required: ["team_id", "title"], + handler: async (args) => { + const input: Record = { + teamId: matches(args, "team_id", UUID, "team UUID"), + title: str(args, "title", { max: 512 }), + }; + const description = optStr(args, "description", { max: 100000 }); + if (description !== undefined) input.description = description; + const priority = optPosInt(args, "priority"); + if (priority !== undefined) { + if (priority > 4) throw new Error("`priority` must be between 1 and 4"); + input.priority = priority; + } + const { data } = await gql( + "issueCreate", + "mutation IssueCreate($input: IssueCreateInput!) { issueCreate(input: $input) { success issue { id identifier url title } } }", + { input }, + { mutating: true }, + ); + const r = asRecord(data.issueCreate); + const issue = asRecord(r.issue); + if (r.success !== true || typeof issue.id !== "string") throw new Error("Linear did not create the issue"); + return { issue_id: issue.id, identifier: issue.identifier, url: issue.url }; + }, + }, + { + name: "archive_issue", + description: "Archive an issue (issueArchive). An issue that is already archived or gone counts as success.", + effect: "idempotent", + permission: "linear:issues:write", + properties: { issue_id: issueId }, + required: ["issue_id"], + handler: async (args) => { + const id = matches(args, "issue_id", UUID, "issue UUID"); + const cur = await gql( + "issue", + "query Issue($id: String!) { issue(id: $id) { id archivedAt } }", + { id }, + { mutating: false, tolerateNotFound: true }, + ); + if (cur.notFound || typeof asRecord(cur.data.issue).archivedAt === "string") { + return { issue_id: id, archived: false }; + } + const { data } = await gql( + "issueArchive", + "mutation IssueArchive($id: String!) { issueArchive(id: $id) { success } }", + { id }, + { mutating: true, tolerateNotFound: true }, + ); + const ok = asRecord(data.issueArchive).success; + if (ok !== true && Object.keys(data).length > 0) throw new Error("Linear did not archive the issue"); + return { issue_id: id, archived: ok === true }; + }, + }, + { + name: "add_comment", + description: "Comment on an issue. Rollback deletes the comment.", + effect: "reversible", + permission: "linear:comments:write", + compensation: { tool: "delete_comment", argument_map: { comment_id: "$result.comment_id" } }, + properties: { issue_id: issueId, body: { type: "string", description: "Markdown" } }, + required: ["issue_id", "body"], + handler: async (args) => { + const input = { + issueId: matches(args, "issue_id", UUID, "issue UUID"), + body: str(args, "body", { max: 100000 }), + }; + const { data } = await gql( + "commentCreate", + "mutation CommentCreate($input: CommentCreateInput!) { commentCreate(input: $input) { success comment { id url } } }", + { input }, + { mutating: true }, + ); + const r = asRecord(data.commentCreate); + const c = asRecord(r.comment); + if (r.success !== true || typeof c.id !== "string") throw new Error("Linear did not create the comment"); + return { comment_id: c.id, url: c.url ?? null }; + }, + }, + { + name: "delete_comment", + description: "Delete a comment (commentDelete). A comment that is already gone counts as success.", + effect: "idempotent", + permission: "linear:comments:write", + properties: { comment_id: commentId }, + required: ["comment_id"], + handler: async (args) => { + const id = matches(args, "comment_id", UUID, "comment UUID"); + const r = await gql( + "commentDelete", + "mutation CommentDelete($id: String!) { commentDelete(id: $id) { success } }", + { id }, + { mutating: true, tolerateNotFound: true }, + ); + if (r.notFound) return { comment_id: id, deleted: false }; + if (asRecord(r.data.commentDelete).success !== true) throw new Error("Linear did not delete the comment"); + return { comment_id: id, deleted: true }; + }, + }, + ]); +} diff --git a/packages/connectors/src/main.ts b/packages/connectors/src/main.ts index e6d73ca..a10de8f 100644 --- a/packages/connectors/src/main.ts +++ b/packages/connectors/src/main.ts @@ -4,24 +4,56 @@ import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js" import { stringify } from "yaml"; import { compensatorFor } from "./compensate.js"; import type { Connector } from "./common.js"; -import { allConnectors, type AllConnectorsConfig } from "./index.js"; +import { CHECKS } from "./certify.js"; +import { allConnectors, createPgPool, type AllConnectorsConfig, type PgLike } from "./index.js"; import { createConnectorServer } from "./server.js"; const USAGE = `reversi-connectors: real tool connectors with Reversi manifests and working compensations. - serve [--root dir] [--backup-dir dir] - Serve one connector as an MCP server on stdio. Credentials come from the environment: - GITHUB_TOKEN, SLACK_BOT_TOKEN, STRIPE_SECRET_KEY. Never from flags. - manifests [--format json|yaml] + serve [--root dir] [--backup-dir dir] + Serve one connector as an MCP server on stdio. Credentials come from the environment, never from flags: + github GITHUB_TOKEN + slack SLACK_BOT_TOKEN + stripe STRIPE_SECRET_KEY + gitlab GITLAB_TOKEN (optional GITLAB_BASE_URL for self-hosted) + teams TEAMS_TOKEN + linear LINEAR_API_KEY + jira JIRA_BASE_URL, JIRA_EMAIL, JIRA_API_TOKEN + gcalendar GOOGLE_CALENDAR_TOKEN + postgres POSTGRES_URL + aws AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION (optional AWS_SESSION_TOKEN) + gcp GCP_ACCESS_TOKEN + azure AZURE_STORAGE_TOKEN, AZURE_STORAGE_ACCOUNT + kubernetes KUBE_API_SERVER, KUBE_TOKEN (optional KUBE_CA_CERT, a path) + fs none (rooted at --root) + manifests [--format json|yaml] Print the connector's Reversi tool manifests (default yaml). compensate [--root dir] [--backup-dir dir] Read one CompensationCall as JSON on stdin and run it. Connectors are enabled by their credentials in the environment (fs is always on, rooted at --root, default the current directory). Use as: reversi rollback --compensator-cmd "reversi-connectors compensate --root ." + certify + Print the twelve certification checks and where they are enforced. The checks run against fake backends + in the test suite (npm test), not against live services. Logs go to stderr; with \`serve\`, stdout is the MCP channel.`; -const NAMES = ["fs", "github", "slack", "stripe"] as const; +const NAMES = [ + "fs", + "github", + "slack", + "stripe", + "gitlab", + "teams", + "linear", + "jira", + "gcalendar", + "postgres", + "aws", + "gcp", + "azure", + "kubernetes", +] as const; type Name = (typeof NAMES)[number]; function fail(msg: string): never { @@ -38,8 +70,14 @@ function need(name: string): string { return env(name) ?? fail(`${name} is not set`); } -function configFor(which: Name, root: string, backupDir: string | undefined, placeholder: boolean): AllConnectorsConfig { - const secret = (n: string) => (placeholder ? "unused" : need(n)); +const noPool: PgLike = { + query: async () => { + throw new Error("no database connection in placeholder mode"); + }, +}; + +async function configFor(which: Name, root: string, backupDir: string | undefined, placeholder: boolean): Promise { + const secret = (n: string, dummy = "unused") => (placeholder ? dummy : need(n)); switch (which) { case "fs": return { fs: { root, backupDir } }; @@ -49,7 +87,100 @@ function configFor(which: Name, root: string, backupDir: string | undefined, pla return { slack: { token: secret("SLACK_BOT_TOKEN") } }; case "stripe": return { stripe: { secretKey: secret("STRIPE_SECRET_KEY") } }; + case "gitlab": + return { gitlab: { token: secret("GITLAB_TOKEN"), baseUrl: env("GITLAB_BASE_URL") } }; + case "teams": + return { teams: { token: secret("TEAMS_TOKEN") } }; + case "linear": + return { linear: { apiKey: secret("LINEAR_API_KEY") } }; + case "jira": + return { + jira: { + baseUrl: secret("JIRA_BASE_URL", "https://example.atlassian.net"), + email: secret("JIRA_EMAIL"), + apiToken: secret("JIRA_API_TOKEN"), + }, + }; + case "gcalendar": + return { gcalendar: { token: secret("GOOGLE_CALENDAR_TOKEN") } }; + case "postgres": { + if (placeholder) return { postgres: { pool: noPool } }; + const url = need("POSTGRES_URL"); + let password = ""; + try { + password = decodeURIComponent(new URL(url).password); + } catch { + /* not a URL: the whole string is still redacted */ + } + return { postgres: { pool: await createPgPool(url), secrets: [url, password] } }; + } + case "aws": + return { + aws: { + accessKeyId: secret("AWS_ACCESS_KEY_ID"), + secretAccessKey: secret("AWS_SECRET_ACCESS_KEY"), + sessionToken: env("AWS_SESSION_TOKEN"), + region: placeholder ? (env("AWS_REGION") ?? "us-east-1") : need("AWS_REGION"), + }, + }; + case "gcp": + return { gcp: { accessToken: secret("GCP_ACCESS_TOKEN") } }; + case "azure": + return { + azure: { + token: secret("AZURE_STORAGE_TOKEN"), + account: placeholder ? (env("AZURE_STORAGE_ACCOUNT") ?? "unused") : need("AZURE_STORAGE_ACCOUNT"), + }, + }; + case "kubernetes": + return { + kubernetes: { + apiServer: secret("KUBE_API_SERVER", "https://unused.invalid"), + token: secret("KUBE_TOKEN"), + caCertPath: placeholder ? undefined : env("KUBE_CA_CERT"), + }, + }; + } +} + +/** Connectors whose credentials are present in the environment, plus the filesystem. */ +async function configFromEnv(root: string, backupDir: string | undefined): Promise { + const config: AllConnectorsConfig = { fs: { root, backupDir } }; + const trigger: Array<[Name, string]> = [ + ["github", "GITHUB_TOKEN"], + ["slack", "SLACK_BOT_TOKEN"], + ["stripe", "STRIPE_SECRET_KEY"], + ["gitlab", "GITLAB_TOKEN"], + ["teams", "TEAMS_TOKEN"], + ["linear", "LINEAR_API_KEY"], + ["jira", "JIRA_API_TOKEN"], + ["gcalendar", "GOOGLE_CALENDAR_TOKEN"], + ["postgres", "POSTGRES_URL"], + ["aws", "AWS_ACCESS_KEY_ID"], + ["gcp", "GCP_ACCESS_TOKEN"], + ["azure", "AZURE_STORAGE_TOKEN"], + ["kubernetes", "KUBE_API_SERVER"], + ]; + for (const [name, variable] of trigger) { + if (env(variable)) Object.assign(config, await configFor(name, root, backupDir, false)); } + return config; +} + +const CERT_TESTS = + "packages/connectors/test/certify.test.ts (runs every check for every connector), driven by test/certification.ts and the per-connector fixtures in test/fixtures/.ts"; + +function printCertification(which: string): void { + const names = which === "all" ? [...NAMES] : [which]; + const out = [ + "Connector certification: the checks below are enforced by the test suite against fake backends,", + "not against live services. The suite fails if any connector fails any check.", + "", + ]; + for (const c of CHECKS) out.push(` ${c.id.padEnd(22)} ${c.verifies}`); + out.push("", `Enforced in: ${CERT_TESTS}`, `Connectors: ${names.join(", ")}`); + out.push("Run: npm test (a connector that does not apply to a check must say why; see docs/connectors.mdx)"); + process.stdout.write(out.join("\n") + "\n"); } async function readStdin(): Promise { @@ -78,7 +209,7 @@ async function main(): Promise { if (cmd === "serve") { if (!NAMES.includes(name as Name)) fail(`serve needs one of: ${NAMES.join(", ")}`); - const connectors = allConnectors(configFor(name as Name, root, backupDir, false)); + const connectors = allConnectors(await configFor(name as Name, root, backupDir, false)); const server = createConnectorServer(connectors); await server.connect(new StdioServerTransport()); process.stderr.write(`reversi-connectors: serving ${name} on stdio\n`); @@ -88,9 +219,10 @@ async function main(): Promise { if (cmd === "manifests") { let connectors: Connector[]; if (name === "all") { - connectors = NAMES.flatMap((n) => allConnectors(configFor(n, root, backupDir, true))); + connectors = []; + for (const n of NAMES) connectors.push(...allConnectors(await configFor(n, root, backupDir, true))); } else if (NAMES.includes(name as Name)) { - connectors = allConnectors(configFor(name as Name, root, backupDir, true)); + connectors = allConnectors(await configFor(name as Name, root, backupDir, true)); } else { return fail(`manifests needs one of: ${NAMES.join(", ")}, all`); } @@ -120,10 +252,7 @@ async function main(): Promise { ) { return fail("stdin must be a CompensationCall with `tool`, `args` and `idempotency_key`"); } - const config: AllConnectorsConfig = { fs: { root, backupDir } }; - if (env("GITHUB_TOKEN")) config.github = { token: need("GITHUB_TOKEN") }; - if (env("SLACK_BOT_TOKEN")) config.slack = { token: need("SLACK_BOT_TOKEN") }; - if (env("STRIPE_SECRET_KEY")) config.stripe = { secretKey: need("STRIPE_SECRET_KEY") }; + const config = await configFromEnv(root, backupDir); try { await compensatorFor(allConnectors(config))(call as Parameters>[0]); } catch (e) { @@ -133,6 +262,12 @@ async function main(): Promise { return; } + if (cmd === "certify") { + if (name !== "all" && !NAMES.includes(name as Name)) return fail(`certify needs one of: ${NAMES.join(", ")}, all`); + printCertification(name as string); + return; + } + fail(`unknown command \`${cmd}\`\n\n${USAGE}`); } diff --git a/packages/connectors/src/objects.ts b/packages/connectors/src/objects.ts new file mode 100644 index 0000000..9df80e6 --- /dev/null +++ b/packages/connectors/src/objects.ts @@ -0,0 +1,84 @@ +import { randomUUID } from "node:crypto"; +import { BACKUP_PREFIX, assertNotBackupKey, str } from "./common.js"; + +/** Shared argument handling for the object-store connectors (S3, Cloud Storage, Blob Storage). */ + +export const MAX_OBJECT_BYTES = 5 * 1024 * 1024; +const BACKUP_KEY = new RegExp(`^${BACKUP_PREFIX.replace(/[.\\/]/g, "\\$&")}[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$`); + +type Args = Record; + +/** An object name: 1-1024 bytes, no control characters, and not inside the reserved backup prefix. */ +export function objectName(args: Args, key: string, opts: { allowBackup?: boolean } = {}): string { + const v = str(args, key, { max: 1024 }); + // eslint-disable-next-line no-control-regex + if (/[\u0000-\u001f\u007f]/.test(v)) throw new Error(`\`${key}\` contains a control character`); + if (Buffer.byteLength(v, "utf8") > 1024) throw new Error(`\`${key}\` is too long`); + if (!opts.allowBackup) assertNotBackupKey(v, key); + return v; +} + +/** The backup object name returned by put/delete, or null when the object did not exist before. */ +export function backupName(args: Args, key: string): string | null { + const v = args[key]; + if (v === null) return null; + if (typeof v !== "string" || !BACKUP_KEY.test(v)) throw new Error(`\`${key}\` is not a backup object name`); + return v; +} + +export function newBackupName(): string { + return BACKUP_PREFIX + randomUUID(); +} + +/** `content` as bytes: UTF-8 text by default, or base64 with `encoding: "base64"`. */ +export function contentBytes(args: Args): Uint8Array { + const content = args.content; + if (typeof content !== "string") throw new Error("`content` must be a string"); + const enc = args.encoding ?? "utf8"; + if (enc !== "utf8" && enc !== "base64") throw new Error("`encoding` must be utf8 or base64"); + let bytes: Uint8Array; + if (enc === "base64") { + if (!/^[A-Za-z0-9+/]*={0,2}$/.test(content) || content.length % 4 !== 0) throw new Error("`content` is not valid base64"); + bytes = Buffer.from(content, "base64"); + } else { + bytes = Buffer.from(content, "utf8"); + } + if (bytes.byteLength > MAX_OBJECT_BYTES) throw new Error(`\`content\` is larger than ${MAX_OBJECT_BYTES} bytes`); + return bytes; +} + +/** Bytes as UTF-8 text when they are valid UTF-8, otherwise base64. */ +export function contentOut(bytes: Uint8Array): { content: string; encoding: "utf8" | "base64" } { + try { + return { content: new TextDecoder("utf-8", { fatal: true }).decode(bytes), encoding: "utf8" }; + } catch { + return { content: Buffer.from(bytes).toString("base64"), encoding: "base64" }; + } +} + +export function maxBytes(args: Args): number { + const v = args.max_bytes; + if (v === undefined) return MAX_OBJECT_BYTES; + if (typeof v !== "number" || !Number.isSafeInteger(v) || v < 1 || v > MAX_OBJECT_BYTES) { + throw new Error(`\`max_bytes\` must be an integer from 1 to ${MAX_OBJECT_BYTES}`); + } + return v; +} + +export const contentProps = { + content: { type: "string" }, + encoding: { type: "string", enum: ["utf8", "base64"], description: "How `content` is encoded; default utf8" }, + content_type: { type: "string", description: "Media type; default application/octet-stream" }, +}; + +const MEDIA_TYPE = /^[A-Za-z0-9!#$&^_.+-]{1,127}\/[A-Za-z0-9!#$&^_.+-]{1,127}(;\s?[A-Za-z0-9=._-]+)*$/; +export function mediaType(args: Args): string { + const v = args.content_type; + if (v === undefined) return "application/octet-stream"; + if (typeof v !== "string" || !MEDIA_TYPE.test(v)) throw new Error("`content_type` is not a valid media type"); + return v; +} + +export function isBackupName(name: string): boolean { + return name.startsWith(BACKUP_PREFIX); +} diff --git a/packages/connectors/src/postgres.ts b/packages/connectors/src/postgres.ts new file mode 100644 index 0000000..87a3912 --- /dev/null +++ b/packages/connectors/src/postgres.ts @@ -0,0 +1,361 @@ +import { EffectUncertainError } from "@multimindlab/reversi"; +import { defineConnector, matches, optMatches, optPosInt, redact, str, type Connector } from "./common.js"; + +export interface PgResult { + rows: Record[]; + rowCount: number | null; +} +export interface PgClientLike { + query(text: string, values?: unknown[]): Promise; + /** A `pg` pool client: pass `true` to drop a connection whose state is unknown. */ + release?(destroy?: boolean | Error): void; +} +/** What the connector needs from a `pg` Pool. Without `connect`, `query` is assumed to be one session. */ +export interface PgLike { + query(text: string, values?: unknown[]): Promise; + connect?(): Promise; +} + +export interface PostgresConfig { + pool: PgLike; + /** Text to scrub from error messages, such as the connection string and password. */ + secrets?: string[]; + timeoutMs?: number; + /** Most rows `query` returns. Default 1000. */ + maxRows?: number; +} + +/** A `pg` Pool for a connection string. `pg` is loaded on demand. */ +export async function createPgPool(connectionString: string): Promise }> { + const name: string = "pg"; + const mod = (await import(name)) as { default?: { Pool: new (o: object) => unknown }; Pool?: new (o: object) => unknown }; + const Pool = mod.Pool ?? mod.default?.Pool; + if (!Pool) throw new Error("the `pg` package is not installed"); + return new Pool({ connectionString }) as PgLike & { end(): Promise }; +} + +export const IDENT = /^[A-Za-z_][A-Za-z0-9_]{0,62}$/; +const MAX_COLUMNS = 100; + +/** An application-level failure inside a session: the transaction is rolled back and the error surfaces as is. */ +class ToolError extends Error {} +class TimedOut extends Error {} + +type Scalar = string | number | boolean | null; +type Q = (text: string, values?: unknown[], mutation?: boolean) => Promise; + +const quote = (id: string) => `"${id}"`; + +function ident(args: Record, key: string): string { + return matches(args, key, IDENT, "SQL identifier (letters, digits, underscore; max 63)"); +} + +/** An optional identifier; `null` (as produced by a `$result` reference) counts as absent. */ +function optIdent(args: Record, key: string): string | undefined { + if (args[key] === null) return undefined; + return optMatches(args, key, IDENT, "SQL identifier (letters, digits, underscore; max 63)"); +} + +function table(args: Record): string { + const schema = optIdent(args, "schema"); + const t = ident(args, "table"); + return schema ? `${quote(schema)}.${quote(t)}` : quote(t); +} + +/** A bindable parameter: scalars as is, objects and arrays as JSON text (for json/jsonb columns). */ +function bindable(v: unknown, what: string): unknown { + if (v === null || typeof v === "string" || typeof v === "boolean") return v; + if (typeof v === "number") { + if (!Number.isFinite(v)) throw new Error(`${what} must be a finite number`); + return v; + } + if (v instanceof Date) return v.toISOString(); + if (typeof v === "object") { + if (Buffer.isBuffer(v)) throw new Error(`${what} is binary, which this connector does not handle`); + return JSON.stringify(v); + } + throw new Error(`${what} has an unsupported type`); +} + +function keyValue(args: Record): string | number { + const v = args.key; + if (typeof v === "string" && v.length > 0 && v.length <= 512) return v; + if (typeof v === "number" && Number.isSafeInteger(v)) return v; + throw new Error("`key` must be a non-empty string or an integer"); +} + +function columnValues(args: Record, field: string): Array<[string, unknown]> { + const v = args[field]; + if (typeof v !== "object" || v === null || Array.isArray(v)) throw new Error(`\`${field}\` must be an object of column values`); + const entries = Object.entries(v); + if (entries.length === 0) throw new Error(`\`${field}\` must not be empty`); + if (entries.length > MAX_COLUMNS) throw new Error(`\`${field}\` has more than ${MAX_COLUMNS} columns`); + return entries.map(([c, val]) => { + if (!IDENT.test(c)) throw new Error(`\`${field}\` has an invalid column name \`${c.slice(0, 64)}\``); + return [c, bindable(val, `\`${field}.${c}\``)]; + }); +} + +/** JSON-safe copy of a row: dates as ISO text, bigints as strings, binary as hex. */ +function jsonSafe(v: unknown): unknown { + if (v instanceof Date) return v.toISOString(); + if (typeof v === "bigint") return v.toString(); + if (Buffer.isBuffer(v)) return `\\x${v.toString("hex")}`; + if (Array.isArray(v)) return v.map(jsonSafe); + if (typeof v === "object" && v !== null) { + return Object.fromEntries(Object.entries(v).map(([k, x]) => [k, jsonSafe(x)])); + } + return v; +} + +function sqlState(e: unknown): string | undefined { + const c = typeof e === "object" && e !== null && "code" in e ? (e as { code: unknown }).code : undefined; + return typeof c === "string" && /^[0-9A-Z]{5}$/.test(c) ? c : undefined; +} + +/** Only SELECT and WITH, and a single statement; the READ ONLY transaction is the real guard. */ +function readOnlySql(text: string): string { + const t = text.trim().replace(/;+\s*$/, ""); + if (!/^(select|with)\b/i.test(t)) throw new Error("`text` must be a SELECT (or WITH ... SELECT) statement"); + if (t.includes(";")) throw new Error("`text` must be a single statement without `;`"); + if (t.includes("\0")) throw new Error("`text` contains a NUL byte"); + return t; +} + +export function createPostgresConnector(cfg: PostgresConfig): Connector { + if (!cfg.pool || typeof cfg.pool.query !== "function") throw new Error("postgres connector needs a `pool`"); + const secrets = cfg.secrets ?? []; + const timeoutMs = cfg.timeoutMs ?? 30000; + const maxRows = cfg.maxRows ?? 1000; + + function race(p: Promise): Promise { + return new Promise((resolve, reject) => { + const t = setTimeout(() => reject(new TimedOut(`no response within ${timeoutMs} ms`)), timeoutMs); + p.then( + (v) => { + clearTimeout(t); + resolve(v); + }, + (e: unknown) => { + clearTimeout(t); + reject(e); + }, + ); + }); + } + + /** + * Run `fn` on one session. A driver error with an SQLSTATE means the server rejected the statement, so nothing + * was applied. A failure without one (connection lost, timeout) after a write was sent leaves the outcome unknown. + */ + async function session(label: string, mutating: boolean, fn: (q: Q) => Promise): Promise { + let client: PgClientLike | undefined; + let mutationSent = false; + let drop = false; + const q: Q = (text, values, mutation = false) => { + if (mutation) mutationSent = true; + return (client ?? cfg.pool).query(text, values); + }; + try { + return await race( + (async () => { + if (cfg.pool.connect) client = await cfg.pool.connect(); + return fn(q); + })(), + ); + } catch (e) { + const code = sqlState(e); + if (e instanceof ToolError || code) { + await (client ?? cfg.pool).query("ROLLBACK").catch(() => undefined); + if (e instanceof ToolError) throw new Error(`${label}: ${redact(e.message, secrets)}`); + const err = new Error(`${label}: Postgres error ${code}: ${redact((e as Error).message, secrets)}`); + (err as Error & { code?: string }).code = code; + throw err; + } + drop = true; + const msg = redact(e instanceof Error ? e.message : String(e), secrets); + if (mutating && mutationSent) { + throw new EffectUncertainError(`${label}: connection failed (${msg}); the effect may have happened`); + } + throw new Error(`${label}: connection failed (${msg})`); + } finally { + client?.release?.(drop); + } + } + + const tableProps = { + table: { type: "string", description: "Table name (letters, digits, underscore)" }, + schema: { type: ["string", "null"], description: "Optional schema name" }, + }; + const keyProps = { + key_column: { type: "string", description: "Primary key column; default id" }, + key: { description: "Primary key value (string or integer)" }, + }; + + return defineConnector("postgres", [ + { + name: "query", + description: + "Run one SELECT with bound parameters ($1, $2, ...). It runs in a READ ONLY transaction, so it cannot change data.", + effect: "read_only", + permission: "postgres:rows:read", + properties: { + text: { type: "string", description: "A single SELECT statement using $1, $2, ... placeholders" }, + values: { type: "array", items: {}, description: "Parameter values, bound and never interpolated" }, + limit: { type: "integer", minimum: 1, description: `At most this many rows (cap ${maxRows})` }, + }, + required: ["text"], + handler: async (args) => { + const text = readOnlySql(str(args, "text", { max: 100000 })); + const raw = args.values ?? []; + if (!Array.isArray(raw) || raw.length > 200) throw new Error("`values` must be an array of at most 200 items"); + const values = raw.map((v, i) => bindable(v, `\`values[${i}]\``)); + const limit = Math.min(optPosInt(args, "limit") ?? maxRows, maxRows); + return session("Postgres query", false, async (q) => { + await q("BEGIN READ ONLY"); + const r = await q(text, values); + await q("ROLLBACK"); + return { + rows: r.rows.slice(0, limit).map(jsonSafe), + row_count: r.rows.length, + truncated: r.rows.length > limit, + }; + }); + }, + }, + { + name: "insert_row", + description: + "Insert one row with bound values and return its primary key (INSERT ... RETURNING). Rollback deletes the row by that key.", + effect: "reversible", + permission: "postgres:rows:write", + compensation: { + tool: "delete_row", + argument_map: { + table: "$result.table", + schema: "$result.schema", + key_column: "$result.key_column", + key: "$result.key", + }, + }, + properties: { + ...tableProps, + key_column: keyProps.key_column, + values: { type: "object", description: "Column name to value" }, + }, + required: ["table", "values"], + handler: async (args) => { + const t = table(args); + const keyColumn = optMatches(args, "key_column", IDENT, "SQL identifier") ?? "id"; + const cols = columnValues(args, "values"); + const text = `INSERT INTO ${t} (${cols.map(([c]) => quote(c)).join(",")}) VALUES (${cols + .map((_, i) => `$${i + 1}`) + .join(",")}) RETURNING ${quote(keyColumn)}`; + const r = await session("Postgres insert_row", true, (q) => + q( + text, + cols.map(([, v]) => v), + true, + ), + ); + const key = jsonSafe(r.rows[0]?.[keyColumn]); + if (typeof key !== "string" && typeof key !== "number") throw new Error("Postgres did not return a usable primary key"); + return { + table: str(args, "table"), + schema: optIdent(args, "schema") ?? null, + key_column: keyColumn, + key, + }; + }, + }, + { + name: "delete_row", + description: + "Delete one row by primary key. A row that is already gone counts as success. Used to undo insert_row; the data cannot be recovered, so withhold this permission from agents.", + effect: "idempotent", + permission: "postgres:rows:delete", + properties: { ...tableProps, ...keyProps }, + required: ["table", "key"], + handler: async (args) => { + const t = table(args); + const keyColumn = optMatches(args, "key_column", IDENT, "SQL identifier") ?? "id"; + const key = keyValue(args); + const r = await session("Postgres delete_row", true, (q) => + q(`DELETE FROM ${t} WHERE ${quote(keyColumn)} = $1`, [key], true), + ); + return { deleted: (r.rowCount ?? 0) > 0 }; + }, + }, + { + name: "update_row", + description: + "Set columns of one row by primary key. In one transaction it locks the row, reads the old values of exactly those columns, updates, and returns the old values so rollback can set them back.", + effect: "reversible", + permission: "postgres:rows:write", + compensation: { + tool: "restore_row", + argument_map: { + table: "$result.table", + schema: "$result.schema", + key_column: "$result.key_column", + key: "$result.key", + values: "$result.previous", + }, + }, + properties: { ...tableProps, ...keyProps, set: { type: "object", description: "Column name to new value" } }, + required: ["table", "key", "set"], + handler: async (args) => { + const t = table(args); + const keyColumn = optMatches(args, "key_column", IDENT, "SQL identifier") ?? "id"; + const key = keyValue(args); + const cols = columnValues(args, "set"); + if (cols.some(([c]) => c === keyColumn)) throw new Error("`set` must not change the key column"); + const names = cols.map(([c]) => c); + const prev = await session("Postgres update_row", true, async (q) => { + await q("BEGIN"); + const sel = await q(`SELECT ${names.map(quote).join(",")} FROM ${t} WHERE ${quote(keyColumn)} = $1 FOR UPDATE`, [key]); + if (sel.rows.length !== 1) throw new ToolError("no row with that key"); + const upd = await q( + `UPDATE ${t} SET ${cols.map(([c], i) => `${quote(c)} = $${i + 2}`).join(", ")} WHERE ${quote(keyColumn)} = $1`, + [key, ...cols.map(([, v]) => v)], + true, + ); + if (upd.rowCount !== 1) throw new ToolError(`expected to update 1 row, updated ${upd.rowCount ?? 0}`); + await q("COMMIT", undefined, true); + return sel.rows[0] as Record; + }); + return { + table: str(args, "table"), + schema: optIdent(args, "schema") ?? null, + key_column: keyColumn, + key, + previous: jsonSafe(prev), + }; + }, + }, + { + name: "restore_row", + description: + "Set columns of one row back to saved values. Safe to repeat; fails if the row no longer exists. Used to undo update_row.", + effect: "idempotent", + permission: "postgres:rows:write", + properties: { ...tableProps, ...keyProps, values: { type: "object", description: "Column name to saved value" } }, + required: ["table", "key", "values"], + handler: async (args) => { + const t = table(args); + const keyColumn = optMatches(args, "key_column", IDENT, "SQL identifier") ?? "id"; + const key = keyValue(args); + const cols = columnValues(args, "values"); + const r = await session("Postgres restore_row", true, (q) => + q( + `UPDATE ${t} SET ${cols.map(([c], i) => `${quote(c)} = $${i + 2}`).join(", ")} WHERE ${quote(keyColumn)} = $1`, + [key, ...cols.map(([, v]) => v)], + true, + ), + ); + if ((r.rowCount ?? 0) === 0) throw new Error("Postgres restore_row: no row with that key"); + return { restored: cols.map(([c]) => c) }; + }, + }, + ]); +} diff --git a/packages/connectors/src/sigv4.ts b/packages/connectors/src/sigv4.ts new file mode 100644 index 0000000..ada05f2 --- /dev/null +++ b/packages/connectors/src/sigv4.ts @@ -0,0 +1,87 @@ +import { createHash, createHmac } from "node:crypto"; + +/** AWS Signature Version 4 request signing (https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_sigv4-create-signed-request.html). */ + +export interface SigV4Input { + method: string; + /** Already URI-encoded path, starting with `/`. For S3 it is encoded once. */ + canonicalUri: string; + /** Raw (not yet encoded) query parameters. */ + query?: Array<[string, string]>; + /** Every header to sign, including `host` and `x-amz-date`. Names are case-insensitive. */ + headers: Record; + /** Lowercase hex SHA-256 of the body. */ + payloadHash: string; + region: string; + service: string; + accessKeyId: string; + secretAccessKey: string; + sessionToken?: string; + /** `YYYYMMDDTHHMMSSZ` */ + amzDate: string; +} + +export interface SigV4Output { + authorization: string; + signedHeaders: string; + canonicalRequest: string; + stringToSign: string; + signature: string; + /** The headers that were signed, lowercased (including `x-amz-security-token` when a session token is used). */ + headers: Record; +} + +export const sha256Hex = (data: string | Uint8Array): string => createHash("sha256").update(data).digest("hex"); + +const hmac = (key: string | Buffer, data: string): Buffer => createHmac("sha256", key).update(data, "utf8").digest(); + +/** RFC 3986 percent-encoding as AWS defines it: only A-Z a-z 0-9 - _ . ~ stay; `/` stays when `keepSlash`. */ +export function uriEncode(s: string, keepSlash = false): string { + let out = ""; + for (const ch of s) { + if (/^[A-Za-z0-9\-_.~]$/.test(ch) || (keepSlash && ch === "/")) out += ch; + else out += [...Buffer.from(ch, "utf8")].map((b) => "%" + b.toString(16).toUpperCase().padStart(2, "0")).join(""); + } + return out; +} + +/** Query parameters encoded and sorted the way SigV4 requires; use the same string in the request URL. */ +export function canonicalQueryString(query: Array<[string, string]>): string { + return query + .map(([k, v]) => [uriEncode(k), uriEncode(v)] as const) + .sort((a, b) => (a[0] < b[0] ? -1 : a[0] > b[0] ? 1 : a[1] < b[1] ? -1 : a[1] > b[1] ? 1 : 0)) + .map(([k, v]) => `${k}=${v}`) + .join("&"); +} + +export function signingKey(secretAccessKey: string, dateStamp: string, region: string, service: string): Buffer { + const kDate = hmac("AWS4" + secretAccessKey, dateStamp); + const kRegion = hmac(kDate, region); + const kService = hmac(kRegion, service); + return hmac(kService, "aws4_request"); +} + +export function signV4(i: SigV4Input): SigV4Output { + const headers: Record = {}; + for (const [k, v] of Object.entries(i.headers)) headers[k.toLowerCase()] = v.trim().replace(/\s+/g, " "); + if (i.sessionToken) headers["x-amz-security-token"] = i.sessionToken; + const names = Object.keys(headers).sort(); + const canonicalHeaders = names.map((n) => `${n}:${headers[n]}\n`).join(""); + const signedHeaders = names.join(";"); + const canonicalQuery = canonicalQueryString(i.query ?? []); + const canonicalRequest = [i.method.toUpperCase(), i.canonicalUri, canonicalQuery, canonicalHeaders, signedHeaders, i.payloadHash].join( + "\n", + ); + const dateStamp = i.amzDate.slice(0, 8); + const scope = `${dateStamp}/${i.region}/${i.service}/aws4_request`; + const stringToSign = ["AWS4-HMAC-SHA256", i.amzDate, scope, sha256Hex(canonicalRequest)].join("\n"); + const signature = createHmac("sha256", signingKey(i.secretAccessKey, dateStamp, i.region, i.service)) + .update(stringToSign, "utf8") + .digest("hex"); + const authorization = `AWS4-HMAC-SHA256 Credential=${i.accessKeyId}/${scope}, SignedHeaders=${signedHeaders}, Signature=${signature}`; + return { authorization, signedHeaders, canonicalRequest, stringToSign, signature, headers }; +} + +export function amzDateOf(d: Date): string { + return d.toISOString().replace(/[:-]|\.\d{3}/g, ""); +} diff --git a/packages/connectors/src/teams.ts b/packages/connectors/src/teams.ts new file mode 100644 index 0000000..220b2a6 --- /dev/null +++ b/packages/connectors/src/teams.ts @@ -0,0 +1,133 @@ +import { + HttpError, + asRecord, + defineConnector, + matches, + optPosInt, + optMatches, + send, + snippet, + str, + type Connector, + type Fetch, +} from "./common.js"; + +export interface TeamsConfig { + /** A Microsoft Graph access token (delegated for sending and deleting messages). */ + token: string; + fetch?: Fetch; + baseUrl?: string; + timeoutMs?: number; +} + +const GUID = /^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$/; +const CHANNEL = /^19:[A-Za-z0-9_.@=-]{1,200}$/; +const MESSAGE = /^\d{8,20}$/; + +export function createTeamsConnector(cfg: TeamsConfig): Connector { + if (!cfg.token) throw new Error("teams connector needs a `token`"); + const f = cfg.fetch ?? fetch; + const base = (cfg.baseUrl ?? "https://graph.microsoft.com/v1.0").replace(/\/+$/, ""); + const secrets = [cfg.token]; + const timeoutMs = cfg.timeoutMs ?? 30000; + + async function api( + method: "GET" | "POST", + p: string, + opts: { body?: unknown; tolerate?: number[] } = {}, + ): Promise<{ status: number; json: Record }> { + const label = `Graph ${method} ${p}`; + const res = await send( + f, + base + p, + { + method, + headers: { + Authorization: `Bearer ${cfg.token}`, + Accept: "application/json", + ...(opts.body !== undefined ? { "Content-Type": "application/json" } : {}), + }, + body: opts.body !== undefined ? JSON.stringify(opts.body) : undefined, + }, + { label, mutating: method !== "GET", secrets, timeoutMs }, + ); + if (!res.ok && !(opts.tolerate ?? []).includes(res.status)) { + throw new HttpError(`${label} failed with status ${res.status}: ${snippet(res, secrets)}`, res.status); + } + return { status: res.status, json: asRecord(res.json) }; + } + + const channelPath = (args: Record) => { + const team = matches(args, "team_id", GUID, "team ID (GUID)"); + const channel = matches(args, "channel_id", CHANNEL, "channel ID such as 19:...@thread.tacv2"); + return { team, channel, p: `/teams/${team}/channels/${encodeURIComponent(channel)}` }; + }; + + const team = { type: "string", description: "Team (group) ID, a GUID" }; + const channel = { type: "string", description: "Channel ID such as 19:abc@thread.tacv2" }; + + return defineConnector("teams", [ + { + name: "list_channels", + description: "List the channels of a team. Needs Channel.ReadBasic.All.", + effect: "read_only", + permission: "teams:channels:read", + properties: { team_id: team, top: { type: "integer", minimum: 1, maximum: 999 } }, + required: ["team_id"], + handler: async (args) => { + const teamId = matches(args, "team_id", GUID, "team ID (GUID)"); + const top = optPosInt(args, "top"); + if (top !== undefined && top > 999) throw new Error("`top` must be at most 999"); + const { json } = await api("GET", `/teams/${teamId}/channels${top ? `?$top=${top}` : ""}`); + const channels = (Array.isArray(json.value) ? json.value : []).map((c) => { + const r = asRecord(c); + return { id: r.id, display_name: r.displayName, membership_type: r.membershipType ?? null }; + }); + const next = json["@odata.nextLink"]; + return { channels, next_link: typeof next === "string" ? next : null }; + }, + }, + { + name: "post_message", + description: + "Post a message to a channel as the signed-in user. Needs the delegated ChannelMessage.Send permission. Rollback soft-deletes it.", + effect: "reversible", + permission: "teams:messages:write", + compensation: { + tool: "delete_message", + argument_map: { team_id: "$args.team_id", channel_id: "$args.channel_id", message_id: "$result.message_id" }, + }, + properties: { + team_id: team, + channel_id: channel, + content: { type: "string" }, + content_type: { type: "string", enum: ["text", "html"] }, + }, + required: ["team_id", "channel_id", "content"], + handler: async (args) => { + const { p, team: t, channel: c } = channelPath(args); + const contentType = optMatches(args, "content_type", /^(text|html)$/, "content type (text or html)") ?? "text"; + const { json } = await api("POST", `${p}/messages`, { + body: { body: { contentType, content: str(args, "content", { max: 28000 }) } }, + }); + if (typeof json.id !== "string") throw new Error("Graph response had no message id"); + return { team_id: t, channel_id: c, message_id: json.id, web_url: json.webUrl ?? null }; + }, + }, + { + name: "delete_message", + description: + "Soft-delete a channel message (POST .../softDelete). Needs the delegated ChannelMessage.ReadWrite permission and only works on messages the signed-in user may delete. A message that is already gone counts as success.", + effect: "idempotent", + permission: "teams:messages:delete", + properties: { team_id: team, channel_id: channel, message_id: { type: "string" } }, + required: ["team_id", "channel_id", "message_id"], + handler: async (args) => { + const { p } = channelPath(args); + const id = matches(args, "message_id", MESSAGE, "message ID"); + const r = await api("POST", `${p}/messages/${id}/softDelete`, { tolerate: [404] }); + return { message_id: id, deleted: r.status !== 404 }; + }, + }, + ]); +} diff --git a/packages/connectors/test/certification.selftest.test.ts b/packages/connectors/test/certification.selftest.test.ts new file mode 100644 index 0000000..9df4d7c --- /dev/null +++ b/packages/connectors/test/certification.selftest.test.ts @@ -0,0 +1,221 @@ +import { describe, expect, it } from "vitest"; +import { HttpError, defineConnector, matches, send, snippet, str, type Connector } from "../src/common.js"; +import { connectorSchemaHashes, createSlackConnector, detectManifestDrift } from "../src/index.js"; +import { certifyConnector, httpBackend, type Scenario, type Subject } from "./certification.js"; + +const TOKEN = "kv-SECRET-TOKEN-9999"; + +function kvBackend() { + const kv = new Map(); + return Object.assign( + httpBackend({ + state: () => [...kv].sort(), + handle: (c) => { + const u = new URL(c.url); + const k = decodeURIComponent(u.pathname.replace("/kv/", "")); + if (c.method === "POST" && u.pathname === "/kv") { + const b = JSON.parse(c.body) as { k: string; v: string }; + kv.set(b.k, b.v); + return { status: 201, json: { k: b.k } }; + } + if (c.method === "DELETE") return kv.delete(k) ? { status: 204 } : { status: 404, json: { message: "no" } }; + if (c.method === "GET") return kv.has(k) ? { json: { k, v: kv.get(k) } } : { status: 404, json: { message: "no" } }; + return { status: 500 }; + }, + }), + { kv }, + ); +} + +/** A correct connector over the key-value fake, optionally with `requires` edges. */ +function goodKv(f: typeof fetch, timeoutMs = 30000): Connector { + const secrets = [TOKEN]; + const call = async (method: string, p: string, body?: unknown, tolerate: number[] = []) => { + const r = await send( + f, + `https://kv.example${p}`, + { method, headers: { Authorization: `Bearer ${TOKEN}` }, body: body === undefined ? undefined : JSON.stringify(body) }, + { label: `KV ${method} ${p}`, mutating: method !== "GET", secrets, timeoutMs }, + ); + if (!r.ok && !tolerate.includes(r.status)) throw new HttpError(`KV ${method} ${p} failed with status ${r.status}: ${snippet(r, secrets)}`, r.status); + return r; + }; + const props = { key: { type: "string" }, value: { type: "string" } }; + const c = defineConnector("kv", [ + { + name: "open", + description: "Create a key.", + effect: "reversible", + permission: "kv:write", + compensation: { tool: "close", argument_map: { key: "$args.key" } }, + properties: props, + required: ["key", "value"], + handler: async (a) => { + await call("POST", "/kv", { k: str(a, "key"), v: str(a, "value") }); + return { key: a.key }; + }, + }, + { + name: "close", + description: "Delete a key.", + effect: "idempotent", + permission: "kv:write", + properties: { key: props.key }, + required: ["key"], + handler: async (a) => { + const r = await call("DELETE", `/kv/${encodeURIComponent(matches(a, "key", /^[a-z0-9]+$/, "key"))}`, undefined, [404]); + return { deleted: r.status !== 404 }; + }, + }, + { + name: "annotate", + description: "Create a second key that depends on the first.", + effect: "reversible", + permission: "kv:write", + compensation: { tool: "close", argument_map: { key: "$args.key" } }, + properties: props, + required: ["key", "value"], + handler: async (a) => { + await call("POST", "/kv", { k: str(a, "key"), v: str(a, "value") }); + return { key: a.key }; + }, + }, + ]); + return c; +} + +function subjectOf(make: (f: typeof fetch, t?: number) => Connector, extra: Partial = {}): Subject { + return { + name: "kv", + secrets: [TOKEN], + make: (o) => { + const backend = kvBackend(); + return { backend, connector: make(backend.fetch, o?.timeoutMs) }; + }, + ...extra, + }; +} + +const kvScenarios: Scenario[] = [ + { tool: "open", args: { key: "alpha", value: "1" } }, + { tool: "close", args: { key: "alpha" }, setup: (b) => void (b as ReturnType).kv.set("alpha", "1") }, + { tool: "annotate", args: { key: "beta", value: "2" } }, +]; + +describe("certification suite", () => { + it("passes a correct connector", async () => { + const cert = await certifyConnector(subjectOf(goodKv), kvScenarios); + expect(cert.results.filter((r) => r.status === "fail")).toEqual([]); + expect(cert.results.find((r) => r.check === "dependency_block")?.status).toBe("not_applicable"); + }); + + it("runs the dependency block check when a manifest declares requires, and passes", async () => { + const make = (f: typeof fetch, t?: number) => { + const c = goodKv(f, t); + (c.manifests.find((m) => (m as { tool: string }).tool === "kv.annotate") as { requires?: string[] }).requires = ["kv.open"]; + return c; + }; + const cert = await certifyConnector(subjectOf(make), kvScenarios); + const r = cert.results.find((x) => x.check === "dependency_block")!; + expect(r.status, r.reason).toBe("pass"); + expect(r.tools).toEqual(["kv.annotate"]); + expect(cert.results.filter((x) => x.status === "fail")).toEqual([]); + }); + + it("fails a connector that leaks the credential, loses uncertainty, and does not undo", async () => { + const bad = (f: typeof fetch): Connector => { + const c = defineConnector("kv", [ + { + name: "open", + description: "Create a key.", + effect: "reversible", + permission: "kv:write", + compensation: { tool: "close", argument_map: { key: "$args.key", extra: "$args.value" } }, + properties: { key: { type: "string" }, value: { type: "string" } }, + required: ["key", "value"], + handler: async (a) => { + let r: Response; + try { + r = await f("https://kv.example/kv", { method: "POST", headers: { Authorization: `Bearer ${TOKEN}` }, body: JSON.stringify({ k: a.key, v: a.value }) }); + } catch (e) { + throw new Error(`transport: ${(e as Error).message}`); // plain Error: should have been EffectUncertainError + } + if (!r.ok) throw new Error(`failed ${r.status}: ${await r.text()} (token ${TOKEN})`); // leaks the credential + return { key: a.key, echo: `auth was ${TOKEN}` }; + }, + }, + { + name: "close", + description: "Pretend to delete.", + effect: "idempotent", + permission: "kv:write", + properties: { key: { type: "string" } }, + required: ["key"], + handler: async () => ({ deleted: true }), // does nothing + }, + ]); + return c; + }; + const cert = await certifyConnector(subjectOf((f) => bad(f)), [ + { tool: "open", args: { key: "alpha", value: "1" } }, + { tool: "close", args: { key: "alpha" }, setup: (b) => void (b as ReturnType).kv.set("alpha", "1") }, + ]); + const failed = (id: string) => cert.results.find((r) => r.check === id)?.status === "fail"; + expect(failed("forward_failure")).toBe(true); + expect(failed("retry")).toBe(true); + expect(failed("compensation_success")).toBe(true); + expect(failed("redaction")).toBe(true); + expect(failed("timeout")).toBe(true); // it never applies a timeout, so the call hangs until the suite gives up + expect(failed("forward_success")).toBe(true); // `close` changed nothing although it reported success + }); + + it("fails a scenario list that does not cover every tool", async () => { + const cert = await certifyConnector(subjectOf(goodKv), kvScenarios.slice(0, 1)); + const r = cert.results.find((x) => x.check === "forward_success")!; + expect(r.status).toBe("fail"); + expect(r.reason).toContain("no scenario for"); + }); + + it("requires a reason for every not_applicable result", async () => { + const cert = await certifyConnector(subjectOf(goodKv, { notApplicable: { timeout: "declared by the subject" } }), kvScenarios); + expect(cert.results.find((r) => r.check === "timeout")).toMatchObject({ status: "not_applicable", reason: "declared by the subject" }); + for (const r of cert.results) if (r.status === "not_applicable") expect(r.reason).toBeTruthy(); + }); +}); + +describe("manifest drift", () => { + const slack = createSlackConnector({ token: "t" }); + const baseline = connectorSchemaHashes(slack); + + it("hashes every tool and reports no drift against itself", () => { + expect(Object.keys(baseline).sort()).toEqual(Object.keys(slack.tools).sort()); + for (const h of Object.values(baseline)) expect(h).toMatch(/^[0-9a-f]{64}$/); + expect(detectManifestDrift(slack.manifests, slack.inputSchemas, baseline)).toEqual({ added: [], removed: [], changed: [] }); + }); + + it("detects a changed input schema", () => { + const schemas = structuredClone(slack.inputSchemas) as Record }>; + schemas["slack.post_message"]!.properties.blocks = { type: "array" }; + expect(detectManifestDrift(slack.manifests, schemas, baseline)).toEqual({ added: [], removed: [], changed: ["slack.post_message"] }); + }); + + it("detects added and removed tools", () => { + const fewer = { ...baseline }; + delete fewer["slack.add_reaction"]; + expect(detectManifestDrift(slack.manifests, slack.inputSchemas, fewer).added).toEqual(["slack.add_reaction"]); + const more = { ...baseline, "slack.gone": "0".repeat(64) }; + expect(detectManifestDrift(slack.manifests, slack.inputSchemas, more).removed).toEqual(["slack.gone"]); + }); + + it("reads a schema_hash on the manifests when no baseline is passed", () => { + const stamped = slack.manifests.map((m) => ({ ...(m as object), schema_hash: baseline[(m as { tool: string }).tool] })); + expect(detectManifestDrift(stamped, slack.inputSchemas)).toEqual({ added: [], removed: [], changed: [] }); + expect(detectManifestDrift(slack.manifests, slack.inputSchemas).added).toHaveLength(slack.manifests.length); + }); + + it("is stable across key order", () => { + const a = { inputSchemas: { t: { type: "object", properties: { a: 1, b: 2 } } } }; + const b = { inputSchemas: { t: { properties: { b: 2, a: 1 }, type: "object" } } }; + expect(connectorSchemaHashes(a)).toEqual(connectorSchemaHashes(b)); + }); +}); diff --git a/packages/connectors/test/certification.ts b/packages/connectors/test/certification.ts new file mode 100644 index 0000000..ef12f15 --- /dev/null +++ b/packages/connectors/test/certification.ts @@ -0,0 +1,568 @@ +import { EffectUncertainError, Registry } from "@multimindlab/reversi"; +import { expect } from "vitest"; +import { + CHECK_IDS, + compensatorFor, + connectorSchemaHashes, + detectManifestDrift, + type Certification, + type CheckId, + type CheckResult, + type Connector, +} from "../src/index.js"; +import { engine, type Call, type FakeReply } from "./helpers.js"; + +/** + * The shared certification helper. `certifyConnector(subject, scenarios)` drives every tool of a connector against a + * fake backend and records pass / fail / not_applicable, with a reason, for each of the twelve checks. + */ + +export type Args = Record; + +export type Fault = + | { kind: "status"; status: number; body?: FakeReply; on?: "any" | "mutating" } + | { kind: "throw"; message: string; on?: "any" | "mutating" } + | { kind: "hang"; on?: "any" | "mutating" }; + +export interface Backend { + /** The active fault; set by the checks. */ + fault: Fault | null; + /** Every request the connector made, including ones a fault swallowed. */ + calls: Call[]; + /** A JSON snapshot of everything the fake service holds. */ + state(): unknown; + isMutating(c: Call): boolean; + /** False for backends with no transport to break (the local filesystem). */ + supportsFaults: boolean; +} + +export interface HttpBackend extends Backend { + fetch: typeof fetch; +} + +/** A fake HTTP service with fault injection. `handle` answers requests; `state` snapshots what it holds. */ +export function httpBackend(o: { + handle: (c: Call) => FakeReply | Promise; + state: () => unknown; + isMutating?: (c: Call) => boolean; +}): HttpBackend { + const calls: Call[] = []; + const isMutating = o.isMutating ?? ((c: Call) => c.method !== "GET" && c.method !== "HEAD"); + const b: HttpBackend = { + fault: null, + calls, + state: o.state, + isMutating, + supportsFaults: true, + fetch: (async (url: string | URL | Request, init?: RequestInit) => { + const headers: Record = {}; + new Headers(init?.headers).forEach((v, k) => (headers[k] = v)); + let body = ""; + if (init?.body instanceof Uint8Array) body = Buffer.from(init.body).toString("latin1"); + else if (init?.body !== undefined && init?.body !== null) body = String(init.body); + const call: Call = { url: String(url), method: init?.method ?? "GET", headers, body }; + calls.push(call); + const f = b.fault; + if (f && (f.on !== "mutating" || isMutating(call))) { + if (f.kind === "throw") throw new TypeError(f.message); + if (f.kind === "hang") { + return new Promise((_, reject) => { + const s = init?.signal; + if (s?.aborted) reject(s.reason); + s?.addEventListener("abort", () => reject(s.reason), { once: true }); + }); + } + return toResponse({ status: f.status, ...(f.body ?? { json: { message: `fault ${f.status}` } }) }); + } + return toResponse(await o.handle(call)); + }) as typeof fetch, + }; + return b; +} + +export function toResponse(r: FakeReply & { bytes?: Uint8Array; headers?: Record }): Response { + const status = r.status ?? 200; + const body = (r as { bytes?: Uint8Array }).bytes ?? r.text ?? (r.json === undefined ? "" : JSON.stringify(r.json)); + return new Response(status === 204 || status === 304 ? null : body, { + status, + headers: (r as { headers?: Record }).headers, + }); +} + +export interface Subject { + name: string; + /** Every credential that must never leak. */ + secrets: string[]; + /** A fresh connector wired to a fresh fake backend. `timeoutMs` is passed through to the connector. */ + make(opts?: { timeoutMs?: number }): { connector: Connector; backend: Backend }; + /** How a backend error with this HTTP status must read in the error message. Default: the number itself. */ + statusPattern?: (status: number) => RegExp; + /** An error body in the service's own format that echoes `secret`, so redaction is exercised. */ + faultBody?: (status: number, secret: string) => FakeReply; + /** Checks that cannot apply to this connector, with the reason. */ + notApplicable?: Partial>; +} + +export interface Scenario { + /** Short tool name, such as `post_message`. */ + tool: string; + args: Args; + /** Seed the fake backend so `args` refer to things that exist. */ + setup?: (b: Backend) => void | Promise; + /** Assertions on a successful result. */ + expect?: (result: unknown, b: Backend) => void; + /** Make the tool fail definitively without a transport fault (for the filesystem). */ + fail?: (b: Backend) => void; + /** For tools that pass the idempotency key to the service: does this request carry `key`? */ + usesKey?: (c: Call, key: string) => boolean; + /** The call changes nothing observable even when it succeeds. */ + noEffect?: boolean; + fingerprint?: (b: Backend) => unknown; + /** Compare the state before the call with the state after rollback. Default: equal. */ + restored?: (before: unknown, after: unknown, b: Backend) => void; + /** Make the undo fail / heal it again. Default: a 500 fault on every request. */ + breakCompensation?: (b: Backend) => void; + healCompensation?: (b: Backend) => void; +} + +type Manifest = { tool: string; effect: string; compensation?: { tool: string; argument_map: Record }; requires?: string[] }; + +const manifestsOf = (c: Connector): Manifest[] => c.manifests as Manifest[]; + +function shortName(c: Connector, full: string): string { + return full.startsWith(`${c.name}.`) ? full.slice(c.name.length + 1) : full; +} + +function leaks(text: string, secrets: string[]): string | undefined { + return secrets.find((s) => s && text.includes(s)); +} + +function noLeak(what: string, text: string, secrets: string[]): void { + const s = leaks(text, secrets); + if (s) throw new Error(`${what} contains the credential`); +} + +const mutatingFault = (mutating: boolean): "any" | "mutating" => (mutating ? "mutating" : "any"); + +export async function certifyConnector(subject: Subject, scenarios: Scenario[]): Promise { + const probeMade = subject.make(); + const probe = probeMade.connector; + const supportsFaults = probeMade.backend.supportsFaults; + const manifests = manifestsOf(probe); + const manifestFor = (sc: Scenario) => { + const m = manifests.find((x) => x.tool === `${probe.name}.${sc.tool}`); + if (!m) throw new Error(`scenario names unknown tool \`${sc.tool}\``); + return m; + }; + const statusRe = subject.statusPattern ?? ((s: number) => new RegExp(`\\b${s}\\b`)); + const bodyFor = (status: number, secret: string): FakeReply => + subject.faultBody?.(status, secret) ?? { json: { message: `rejected: ${secret}`, error: { message: `rejected: ${secret}` } } }; + const secret0 = subject.secrets[0] ?? ""; + + const fp = (sc: Scenario, b: Backend) => JSON.parse(JSON.stringify(sc.fingerprint ? sc.fingerprint(b) : b.state() ?? null)); + const prepare = async (sc: Scenario, timeoutMs?: number) => { + const { connector, backend } = subject.make(timeoutMs ? { timeoutMs } : undefined); + await sc.setup?.(backend); + const tool = connector.tools[`${connector.name}.${sc.tool}`]; + if (!tool) throw new Error(`connector has no tool \`${sc.tool}\``); + return { connector, backend, tool, before: fp(sc, backend) }; + }; + const errorOf = async (p: Promise): Promise => { + try { + await p; + } catch (e) { + if (e instanceof Error) return e; + throw new Error(`a non-Error was thrown: ${String(e)}`); + } + throw new Error("expected the call to fail, but it succeeded"); + }; + + async function each( + id: CheckId, + list: Scenario[], + body: (sc: Scenario, m: Manifest) => Promise, + naReason: string, + ): Promise { + const declared = subject.notApplicable?.[id]; + if (declared) return { check: id, status: "not_applicable", reason: declared, tools: [] }; + if (list.length === 0) return { check: id, status: "not_applicable", reason: naReason, tools: [] }; + const failures: string[] = []; + for (const sc of list) { + try { + await body(sc, manifestFor(sc)); + } catch (e) { + failures.push(`${sc.tool}: ${e instanceof Error ? e.message.split("\n")[0] : String(e)}`); + } + } + return { + check: id, + status: failures.length ? "fail" : "pass", + ...(failures.length ? { reason: failures.join("; ") } : {}), + tools: list.map((s) => s.tool), + }; + } + + const mutatingOf = (m: Manifest) => m.effect !== "read_only"; + const compensable = scenarios.filter((s) => manifestFor(s).compensation); + const results: CheckResult[] = []; + + // 1. forward success + { + const r = await each( + "forward_success", + scenarios, + async (sc, m) => { + const { backend, tool, before } = await prepare(sc); + const mark = backend.calls.length; + const result = await tool(sc.args, {}); + JSON.stringify(result ?? null); + sc.expect?.(result, backend); + const after = fp(sc, backend); + if (m.effect === "read_only") { + expect(after).toEqual(before); + if (backend.supportsFaults) { + const writes = backend.calls.slice(mark).filter((c) => backend.isMutating(c)); + expect(writes, "a read-only tool made a mutating request").toEqual([]); + } + } else if (!sc.noEffect) { + expect(after, "the tool reported success but the backend did not change").not.toEqual(before); + } + }, + "no scenarios", + ); + const known = new Set(scenarios.map((s) => `${probe.name}.${s.tool}`)); + const missing = manifests.map((m) => m.tool).filter((t) => !known.has(t)); + if (missing.length) { + r.status = "fail"; + r.reason = [r.reason, `no scenario for: ${missing.join(", ")}`].filter(Boolean).join("; "); + } + results.push(r); + } + + // 2. forward failure + results.push( + await each( + "forward_failure", + scenarios.filter((s) => s.fail || supportsFaults), + async (sc) => { + const { backend, tool, before: seeded } = await prepare(sc); + let before = seeded; + if (sc.fail) { + sc.fail(backend); + before = fp(sc, backend); // the hook itself sets the scene; the call must not change it further + } else backend.fault = { kind: "status", status: 422, body: bodyFor(422, secret0) }; + const err = await errorOf(Promise.resolve().then(() => tool(sc.args, {}))); + expect(err).not.toBeInstanceOf(EffectUncertainError); + noLeak("the error message", err.message, subject.secrets); + if (!sc.fail) expect(err.message).toMatch(statusRe(422)); + backend.fault = null; + expect(fp(sc, backend), "a failed call changed the backend").toEqual(before); + }, + "no scenarios", + ), + ); + + // 3. duplicate call + results.push( + await each( + "duplicate_call", + scenarios.filter((s) => manifestFor(s).effect === "idempotent" || s.usesKey), + async (sc) => { + const { backend, tool } = await prepare(sc); + const key = "idem-0123456789abcdef"; + const mark = backend.calls.length; + await tool(sc.args, { idempotency_key: key }); + const first = fp(sc, backend); + const mid = backend.calls.length; + await tool(sc.args, { idempotency_key: key }); + expect(fp(sc, backend), "the repeat had a second effect").toEqual(first); + if (sc.usesKey) { + for (const c of backend.calls.slice(mark).filter((x) => backend.isMutating(x))) { + expect(sc.usesKey(c, key), `a write did not carry the idempotency key: ${c.method} ${c.url}`).toBe(true); + } + expect(backend.calls.length).toBeGreaterThan(mid); + } + }, + "no tool takes an idempotency key and none is naturally idempotent", + ), + ); + + // 4. retry + results.push( + await each( + "retry", + scenarios, + async (sc, m) => { + const mutating = mutatingOf(m); + for (const mode of ["transport", "5xx"] as const) { + const { backend, tool, before } = await prepare(sc); + const on = mutatingFault(mutating); + backend.fault = + mode === "transport" + ? { kind: "throw", message: `socket hang up ${secret0}`, on } + : { kind: "status", status: 503, body: bodyFor(503, secret0), on }; + const err = await errorOf(Promise.resolve().then(() => tool(sc.args, {}))); + if (mode === "transport") { + if (mutating) expect(err, "a lost connection on a write must be uncertain").toBeInstanceOf(EffectUncertainError); + else expect(err).not.toBeInstanceOf(EffectUncertainError); + } else if (!(err instanceof EffectUncertainError)) { + expect(err.message).toMatch(statusRe(503)); + } + noLeak("the error message", err.message, subject.secrets); + expect(fp(sc, backend), "the failed attempt changed the backend").toEqual(before); + backend.fault = null; + await tool(sc.args, {}); + if (mutating && !sc.noEffect) expect(fp(sc, backend), "the re-run did not take effect").not.toEqual(before); + } + }, + "no scenarios", + ), + ); + + /** Forward-run one scenario through a real engine, first running the scenarios of any tools it `requires`. */ + const rollbackDrive = async (sc: Scenario, m: Manifest) => { + const { connector, backend } = subject.make(); + const prereqs = (m.requires ?? []).map((r) => { + const found = scenarios.find((s) => `${probe.name}.${s.tool}` === r); + if (!found) throw new Error(`no scenario for required tool ${r}`); + return found; + }); + for (const p of prereqs) await p.setup?.(backend); + await sc.setup?.(backend); + const before = fp(sc, backend); + const eng = engine([connector]); + for (const p of prereqs) expect((await eng.call(`${connector.name}.${p.tool}`, p.args)).status).toBe("executed"); + const step = await eng.call(`${connector.name}.${sc.tool}`, sc.args); + expect(step.status).toBe("executed"); + if (!sc.noEffect) expect(fp(sc, backend)).not.toEqual(before); + return { connector, backend, before, eng, comp: compensatorFor([connector]), steps: prereqs.length + 1 }; + }; + const assertRestored = (sc: Scenario, before: unknown, b: Backend) => { + const after = fp(sc, b); + if (sc.restored) sc.restored(before, after, b); + else expect(after, "the backend is not back in its prior state").toEqual(before); + }; + const standing = (eng: ReturnType, status: string) => + eng.run.state().steps.filter((s) => s.status === status).length; + + // 5. compensation success + results.push( + await each( + "compensation_success", + compensable, + async (sc, m) => { + const { connector, backend, before, eng, comp, steps } = await rollbackDrive(sc, m); + // the compensation is a registered idempotent tool that accepts exactly what the argument_map produces + const reg = Registry.from(connector.manifests); + const ct = reg.get(m.compensation!.tool)!; + expect(ct.effect, "the compensation tool must be idempotent").toBe("idempotent"); + const schema = connector.inputSchemas[ct.tool] as { properties: object; required: string[] }; + const mapped = Object.keys(m.compensation!.argument_map); + for (const k of mapped) expect(Object.keys(schema.properties), `argument_map key \`${k}\` is not an argument of ${ct.tool}`).toContain(k); + for (const k of schema.required) expect(mapped, `${ct.tool} requires \`${k}\` but argument_map does not provide it`).toContain(k); + const report = await eng.run.rollback({ compensate: comp }); + expect(report.complete, JSON.stringify(report)).toBe(true); + expect(report.compensated).toHaveLength(steps); + assertRestored(sc, before, backend); + }, + "no tool declares a compensation", + ), + ); + + // 6. compensation failure + results.push( + await each( + "compensation_failure", + compensable, + async (sc, m) => { + const { backend, before, eng, comp } = await rollbackDrive(sc, m); + if (sc.breakCompensation) sc.breakCompensation(backend); + else backend.fault = { kind: "status", status: 500, body: bodyFor(500, secret0) }; + const report = await eng.run.rollback({ compensate: comp }); + expect(report.complete, "a failed compensation must never report complete").toBe(false); + expect(report.failed).toHaveLength(1); + expect(report.compensated).toEqual([]); + expect(standing(eng, "compensation_failed")).toBe(1); + noLeak("the rollback report", JSON.stringify(report), subject.secrets); + if (sc.healCompensation) sc.healCompensation(backend); + else backend.fault = null; + const again = await eng.run.rollback({ compensate: comp }); + expect(again.complete, JSON.stringify(again)).toBe(true); + assertRestored(sc, before, backend); + }, + "no tool declares a compensation", + ), + ); + + // 7. verification failure + results.push( + await each( + "verification_failure", + compensable, + async (sc, m) => { + const { backend, before, eng, comp } = await rollbackDrive(sc, m); + const report = await eng.run.rollback({ compensate: comp, verify: () => false }); + expect(report.complete).toBe(false); + expect(report.compensated, "an unverified undo must not be reported as compensated").toEqual([]); + expect(report.failed).toHaveLength(1); + expect(report.failed[0]!.error).toContain("verification failed"); + expect(standing(eng, "compensated")).toBe(0); + const again = await eng.run.rollback({ compensate: comp, verify: () => true }); + expect(again.complete, JSON.stringify(again)).toBe(true); + assertRestored(sc, before, backend); + }, + "no tool declares a compensation", + ), + ); + + // 8. permission denial + results.push( + await each( + "permission_denial", + supportsFaults ? scenarios : [], + async (sc) => { + for (const status of [401, 403]) { + const { backend, tool, before } = await prepare(sc); + backend.fault = { kind: "status", status, body: bodyFor(status, secret0) }; + const err = await errorOf(Promise.resolve().then(() => tool(sc.args, {}))); + expect(err).not.toBeInstanceOf(EffectUncertainError); + expect(err.message).toMatch(statusRe(status)); + noLeak("the error message", err.message, subject.secrets); + backend.fault = null; + expect(fp(sc, backend), "a denied call changed the backend").toEqual(before); + } + }, + "no scenarios", + ), + ); + + // 9. timeout + results.push( + await each( + "timeout", + scenarios, + async (sc, m) => { + const mutating = mutatingOf(m); + const { backend, tool } = await prepare(sc, 40); + backend.fault = { kind: "hang", on: mutatingFault(mutating) }; + const deadline = new Promise((_, reject) => { + const t = setTimeout(() => reject(new Error("the call never returned: the connector applies no timeout")), 2000); + t.unref(); + }); + const err = await errorOf(Promise.race([Promise.resolve().then(() => tool(sc.args, {})), deadline])); + if (mutating) expect(err, "a write that never answers must be uncertain").toBeInstanceOf(EffectUncertainError); + else expect(err, "a read that never answers is a plain error").not.toBeInstanceOf(EffectUncertainError); + noLeak("the error message", err.message, subject.secrets); + }, + "no scenarios", + ), + ); + + // 10. redaction + results.push( + await each( + "redaction", + scenarios, + async (sc, m) => { + const mutating = mutatingOf(m); + { + const { backend, tool } = await prepare(sc); + const result = await tool(sc.args, {}); + noLeak("the returned JSON", JSON.stringify(result ?? null), subject.secrets); + for (const c of backend.calls) noLeak(`the request ${c.method} ${c.url} (URL or body)`, `${c.url}\n${c.body}`, subject.secrets); + } + for (const fault of [ + { kind: "throw", message: `ECONNRESET talking to host with ${secret0}` }, + { kind: "status", status: 500, body: bodyFor(500, secret0) }, + ] as Fault[]) { + const { backend, tool } = await prepare(sc); + backend.fault = { ...fault, on: mutatingFault(mutating) } as Fault; + const err = await errorOf(Promise.resolve().then(() => tool(sc.args, {}))); + noLeak("the error message", err.message, subject.secrets); + } + }, + "no scenarios", + ), + ); + + // 11. dependency block + { + const edges = manifests.flatMap((m) => (m.requires ?? []).map((req) => ({ later: m.tool, earlier: req }))); + const usable = edges.filter( + (e) => scenarios.some((s) => `${probe.name}.${s.tool}` === e.later) && scenarios.some((s) => `${probe.name}.${s.tool}` === e.earlier), + ); + const declared = subject.notApplicable?.dependency_block; + if (declared) results.push({ check: "dependency_block", status: "not_applicable", reason: declared, tools: [] }); + else if (edges.length === 0) { + results.push({ check: "dependency_block", status: "not_applicable", reason: "no tool manifest declares `requires`", tools: [] }); + } else { + const failures: string[] = []; + if (usable.length === 0) failures.push("a `requires` edge has no scenario for both tools"); + for (const e of usable) { + try { + const first = scenarios.find((s) => `${probe.name}.${s.tool}` === e.earlier)!; + const second = scenarios.find((s) => `${probe.name}.${s.tool}` === e.later)!; + const { connector, backend } = subject.make(); + await first.setup?.(backend); + await second.setup?.(backend); + const eng = engine([connector]); + const a = await eng.call(e.earlier, first.args); + const b = await eng.call(e.later, second.args); + expect([a.status, b.status]).toEqual(["executed", "executed"]); + backend.fault = { kind: "status", status: 500 }; + const report = await eng.run.rollback({ compensate: compensatorFor([connector]) }); + expect(report.complete).toBe(false); + expect(report.deferred, "the earlier step must be deferred while the later one stands").not.toEqual([]); + const stepA = (a as { step: number }).step; + const stepB = (b as { step: number }).step; + expect(report.deferred.find((d) => d.step === stepA)?.blocked_by).toContain(stepB); + } catch (err) { + failures.push(`${e.later} requires ${e.earlier}: ${err instanceof Error ? err.message.split("\n")[0] : String(err)}`); + } + } + results.push({ + check: "dependency_block", + status: failures.length ? "fail" : "pass", + ...(failures.length ? { reason: failures.join("; ") } : {}), + tools: usable.map((e) => e.later), + }); + } + } + + // 12. manifest drift + { + const declared = subject.notApplicable?.manifest_drift; + if (declared) results.push({ check: "manifest_drift", status: "not_applicable", reason: declared, tools: [] }); + else { + let reason: string | undefined; + try { + const c = subject.make().connector; + Registry.from(c.manifests); + const names = manifestsOf(c).map((m) => m.tool); + expect(Object.keys(c.inputSchemas).sort(), "every manifest needs an input schema").toEqual([...names].sort()); + const baseline = connectorSchemaHashes(c); + expect(Object.keys(baseline).sort()).toEqual([...names].sort()); + expect(detectManifestDrift(c.manifests, c.inputSchemas, baseline)).toEqual({ added: [], removed: [], changed: [] }); + const victim = names[0]!; + const changed = { ...c.inputSchemas, [victim]: { ...(c.inputSchemas[victim] as object), properties: { extra: { type: "string" } } } }; + expect(detectManifestDrift(c.manifests, changed, baseline).changed).toEqual([victim]); + const smaller = { ...c.inputSchemas }; + delete smaller[victim]; + const dropped = detectManifestDrift( + c.manifests.filter((m) => (m as { tool: string }).tool !== victim), + smaller, + baseline, + ); + expect(dropped.removed).toEqual([victim]); + const added = detectManifestDrift(c.manifests, c.inputSchemas, Object.fromEntries(Object.entries(baseline).filter(([k]) => k !== victim))); + expect(added.added).toEqual([victim]); + } catch (e) { + reason = e instanceof Error ? e.message.split("\n")[0] : String(e); + } + results.push({ check: "manifest_drift", status: reason ? "fail" : "pass", ...(reason ? { reason } : {}), tools: manifests.map((m) => shortName(probe, m.tool)) }); + } + } + + results.sort((a, b) => CHECK_IDS.indexOf(a.check) - CHECK_IDS.indexOf(b.check)); + return { connector: probe.name, results }; +} diff --git a/packages/connectors/test/certify.test.ts b/packages/connectors/test/certify.test.ts new file mode 100644 index 0000000..79ac89c --- /dev/null +++ b/packages/connectors/test/certify.test.ts @@ -0,0 +1,44 @@ +import { readFileSync } from "node:fs"; +import { fileURLToPath } from "node:url"; +import { afterAll, describe, expect, it } from "vitest"; +import { CHECK_IDS, certificationFailures, renderCertificationMatrix, type Certification } from "../src/index.js"; +import { certifyConnector } from "./certification.js"; +import { cleanupFs, fixtures } from "./fixtures/index.js"; + +afterAll(cleanupFs); + +// Certification runs once per connector; the tests below assert on the recorded results. +const certifications: Certification[] = []; +for (const f of fixtures) certifications.push(await certifyConnector(f.subject, f.scenarios)); + +describe("connector certification", () => { + it("covers all fourteen connectors", () => { + expect(certifications.map((c) => c.connector)).toEqual([ + "fs", "github", "slack", "stripe", "gitlab", "teams", "linear", "jira", "gcalendar", "postgres", "aws", "gcp", "azure", "kubernetes", + ]); + }); + + describe.each(certifications)("$connector", (cert) => { + it("records a result for each of the twelve checks", () => { + expect(cert.results.map((r) => r.check)).toEqual([...CHECK_IDS]); + }); + + it.each(CHECK_IDS)("%s", (id) => { + const r = cert.results.find((x) => x.check === id)!; + expect(r.status, `${cert.connector}: ${id}: ${r.reason ?? ""}`).not.toBe("fail"); + if (r.status === "not_applicable") expect(r.reason, "not_applicable needs a reason").toBeTruthy(); + }); + + it("has no failing check", () => { + expect(certificationFailures(cert)).toEqual([]); + }); + }); + + it("documents the matrix and every not-applicable reason in docs/connectors.mdx", () => { + const docs = readFileSync(fileURLToPath(new URL("../../../docs/connectors.mdx", import.meta.url)), "utf8"); + for (const line of renderCertificationMatrix(certifications).split("\n")) expect(docs).toContain(line); + for (const c of certifications) { + for (const r of c.results) if (r.status === "not_applicable") expect(docs, `${c.connector}/${r.check}`).toContain(r.reason!); + } + }); +}); diff --git a/packages/connectors/test/cli.test.ts b/packages/connectors/test/cli.test.ts new file mode 100644 index 0000000..13b6c3f --- /dev/null +++ b/packages/connectors/test/cli.test.ts @@ -0,0 +1,64 @@ +import { spawnSync } from "node:child_process"; +import { existsSync } from "node:fs"; +import path from "node:path"; +import { fileURLToPath } from "node:url"; +import { Registry } from "@multimindlab/reversi"; +import { describe, expect, it } from "vitest"; +import { CHECK_IDS } from "../src/index.js"; + +const main = path.join(path.dirname(fileURLToPath(import.meta.url)), "..", "dist", "main.js"); +const run = (args: string[], env: Record = {}, input?: string) => + spawnSync(process.execPath, [main, ...args], { input, encoding: "utf8", env: { PATH: process.env.PATH ?? "", ...env } }); +const built = existsSync(main); + +describe("reversi-connectors CLI for the new connectors", () => { + it.skipIf(!built).each([ + ["gitlab", "GITLAB_TOKEN"], + ["teams", "TEAMS_TOKEN"], + ["linear", "LINEAR_API_KEY"], + ["jira", "JIRA_BASE_URL"], + ["gcalendar", "GOOGLE_CALENDAR_TOKEN"], + ["postgres", "POSTGRES_URL"], + ["aws", "AWS_ACCESS_KEY_ID"], + ["gcp", "GCP_ACCESS_TOKEN"], + ["azure", "AZURE_STORAGE_TOKEN"], + ["kubernetes", "KUBE_API_SERVER"], + ])("serve %s needs %s in the environment, not a flag", (name, variable) => { + const r = run(["serve", name]); + expect(r.status).toBe(1); + expect(r.stderr).toContain(`${variable} is not set`); + expect(run(["serve", name, "--token", "abc"]).status).toBe(1); + }); + + it.skipIf(!built)("prints valid manifests for each new connector without any credentials", () => { + for (const name of ["gitlab", "teams", "linear", "jira", "gcalendar", "postgres", "aws", "gcp", "azure", "kubernetes"]) { + const out = JSON.parse(run(["manifests", name, "--format", "json"]).stdout) as Array<{ tool: string }>; + expect(out.length).toBeGreaterThan(2); + expect(out.every((m) => m.tool.startsWith(`${name}.`))).toBe(true); + Registry.from(out); + } + }); + + it.skipIf(!built)("compensate enables a connector from its environment variables and runs its compensation", () => { + // no network: the Kubernetes API server is an unroutable name, so the compensation fails with a clear, redacted error + const call = { run_id: "r", step: 0, tool: "kubernetes.restore_replicas", original_tool: "kubernetes.scale_deployment", args: { namespace: "prod", name: "web", replicas: 3 }, idempotency_key: "k" }; + const r = run(["compensate", "--root", "."], { KUBE_API_SERVER: "http://127.0.0.1:1", KUBE_TOKEN: "SECRET-KUBE-TOKEN" }, JSON.stringify(call)); + expect(r.status).toBe(1); + expect(r.stderr).toContain("Kubernetes PATCH"); + expect(r.stderr).not.toContain("SECRET-KUBE-TOKEN"); + const without = run(["compensate", "--root", "."], {}, JSON.stringify(call)); + expect(without.stderr).toContain("no connector provides"); + }); + + it.skipIf(!built)("certify prints the twelve checks and says where they are enforced", () => { + const all = run(["certify", "all"]); + expect(all.status).toBe(0); + for (const id of CHECK_IDS) expect(all.stdout).toContain(id); + expect(all.stdout).toContain("fake backends"); + expect(all.stdout).toContain("certify.test.ts"); + expect(run(["certify", "gitlab"]).status).toBe(0); + const bad = run(["certify", "nope"]); + expect(bad.status).toBe(1); + expect(bad.stderr).toContain("certify needs one of"); + }); +}); diff --git a/packages/connectors/test/fixtures/aws.ts b/packages/connectors/test/fixtures/aws.ts new file mode 100644 index 0000000..84d4dfa --- /dev/null +++ b/packages/connectors/test/fixtures/aws.ts @@ -0,0 +1,178 @@ +import { expect } from "vitest"; +import { createAwsConnector, signV4 } from "../../src/index.js"; +import { httpBackend, type Backend, type Scenario, type Subject } from "../certification.js"; +import type { Call, FakeReply } from "../helpers.js"; + +export const ACCESS_KEY_ID = "AKIAEXAMPLEKEYID1234"; +export const SECRET = "SECRETACCESSKEY/abcdefghij0123456789+xyz"; +export const SESSION = "FwoGZXIvYXdzSESSIONTOKENexample"; +export const BUCKET = "my-bucket"; +export const REGION = "us-east-1"; +export const NOW = new Date("2026-05-01T12:00:00Z"); +export const BK = ".reversi-backups/00000000-0000-4000-8000-000000000001"; + +type Reply = FakeReply & { bytes?: Uint8Array; headers?: Record }; + +const xmlError = (status: number, code: string, message = code): Reply => ({ + status, + text: `${code}${message}`, +}); +const esc = (s: string) => s.replace(/&/g, "&").replace(/ = {}; + for (const name of m[4]!.split(";")) { + if (name === "host") headers.host = u.host; + else if (c.headers[name] === undefined) return `signed header ${name} was not sent`; + else headers[name] = c.headers[name]!; + } + delete headers["x-amz-security-token"]; + const query = [...u.searchParams.entries()] as Array<[string, string]>; + const signed = signV4({ + method: c.method, + canonicalUri: u.pathname, + query, + headers, + payloadHash: c.headers["x-amz-content-sha256"] ?? "", + region: REGION, + service: "s3", + accessKeyId: ACCESS_KEY_ID, + secretAccessKey: SECRET, + sessionToken: c.headers["x-amz-security-token"], + amzDate: c.headers["x-amz-date"] ?? "", + }); + return signed.signature === m[5] ? undefined : "signature mismatch"; +} + +export function fakeS3() { + const objects = new Map(); + let n = 0; + const b = httpBackend({ + state: () => ({ objects: [...objects].map(([k, v]) => [k, v.bytes.toString("base64"), v.type]).sort() }), + handle: (c): Reply => { + const bad = verifySignature(c); + if (bad) return xmlError(403, "SignatureDoesNotMatch", bad); + const u = new URL(c.url); + if (u.host !== `${BUCKET}.s3.${REGION}.amazonaws.com`) return xmlError(404, "NoSuchBucket"); + const key = decodeURIComponent(u.pathname.slice(1)); + if (c.method === "GET" && key === "") { + const prefix = u.searchParams.get("prefix") ?? ""; + const rows = [...objects].filter(([k]) => k.startsWith(prefix)).sort(([a], [b2]) => (a < b2 ? -1 : 1)); + return { + text: + `false` + + rows.map(([k, v]) => `${esc(k)}${v.bytes.length}2026-01-01T00:00:00.000Z"e"`).join("") + + ``, + }; + } + const copySource = c.headers["x-amz-copy-source"]; + if (c.method === "PUT" && copySource) { + const src = objects.get(decodeURIComponent(copySource.replace(new RegExp(`^/${BUCKET}/`), ""))); + if (!src) return xmlError(404, "NoSuchKey", "The specified key does not exist."); + objects.set(key, { bytes: Buffer.from(src.bytes), type: src.type }); + return { text: `"c${++n}"` }; + } + if (c.method === "PUT") { + objects.set(key, { bytes: Buffer.from(c.body, "latin1"), type: c.headers["content-type"] ?? "" }); + return { headers: { etag: `"p${++n}"` } }; + } + if (c.method === "DELETE") { + objects.delete(key); + return { status: 204 }; + } + const o = objects.get(key); + if (!o) return c.method === "HEAD" ? { status: 404 } : xmlError(404, "NoSuchKey", "The specified key does not exist."); + if (c.method === "HEAD") return { headers: { etag: '"h"', "content-length": String(o.bytes.length) } }; + const range = /^bytes=0-(\d+)$/.exec(c.headers.range ?? ""); + const end = range ? Number(range[1]) + 1 : o.bytes.length; + const part = o.bytes.subarray(0, end); + return { + status: range && part.length < o.bytes.length ? 206 : 200, + bytes: part, + headers: { "content-type": o.type || "application/octet-stream", etag: '"g"', "content-range": `bytes 0-${part.length - 1}/${o.bytes.length}` }, + }; + }, + }); + return Object.assign(b, { objects }); +} +export type FakeS3 = ReturnType; +const s3 = (b: Backend) => b as FakeS3; +const put = (b: Backend, key: string, content: string, type = "text/plain") => void s3(b).objects.set(key, { bytes: Buffer.from(content), type }); +const seed = (b: Backend) => put(b, "docs/a.txt", "v1"); +const visible = (b: Backend) => (s3(b).state() as { objects: string[][] }).objects.filter(([k]) => !k!.startsWith(".reversi-backups/")); +const backups = (b: Backend) => [...s3(b).objects.keys()].filter((k) => k.startsWith(".reversi-backups/")); + +export const subject: Subject = { + name: "aws", + secrets: [SECRET, SESSION], + make: (o) => { + const backend = fakeS3(); + return { + backend, + connector: createAwsConnector({ + accessKeyId: ACCESS_KEY_ID, + secretAccessKey: SECRET, + sessionToken: SESSION, + region: REGION, + fetch: backend.fetch, + now: () => NOW, + timeoutMs: o?.timeoutMs, + }), + }; + }, + faultBody: (status, secret) => ({ text: `AccessDeniedstatus ${status} denied ${secret}` }), +}; + +const base = { fingerprint: visible }; +export const scenarios: Scenario[] = [ + { tool: "get_object", args: { bucket: BUCKET, key: "docs/a.txt" }, setup: seed, expect: (r) => expect(r).toMatchObject({ content: "v1", encoding: "utf8", size: 2, truncated: false }) }, + { + tool: "list_objects", + args: { bucket: BUCKET, prefix: "docs/", max_keys: 10 }, + setup: (b) => { + seed(b); + put(b, BK, "hidden"); + put(b, "other/b.txt", "x"); + }, + expect: (r) => expect((r as { objects: Array<{ key: string }> }).objects.map((o) => o.key)).toEqual(["docs/a.txt"]), + }, + { + tool: "put_object", + args: { bucket: BUCKET, key: "docs/new file.txt", content: "hello" }, + ...base, + expect: (r, b) => { + expect(r).toMatchObject({ existed: false, backup_key: null }); + expect(backups(b)).toEqual([]); + }, + }, + { + tool: "put_object", + args: { bucket: BUCKET, key: "docs/a.txt", content: "v2", content_type: "text/plain" }, + setup: seed, + ...base, + expect: (r, b) => { + expect(r).toMatchObject({ existed: true }); + expect(backups(b)).toHaveLength(1); + }, + }, + { tool: "delete_object", args: { bucket: BUCKET, key: "docs/a.txt" }, setup: seed, ...base, expect: (r, b) => (expect(r).toMatchObject({ existed: true }), expect(backups(b)).toHaveLength(1)) }, + { + tool: "restore_object", + args: { bucket: BUCKET, key: "docs/a.txt", backup_key: BK }, + setup: (b) => (put(b, "docs/a.txt", "v2"), put(b, BK, "v1")), + ...base, + expect: (_r, b) => expect(s3(b).objects.get("docs/a.txt")!.bytes.toString()).toBe("v1"), + }, + { + tool: "restore_object", + args: { bucket: BUCKET, key: "docs/new.txt", backup_key: null }, + setup: (b) => put(b, "docs/new.txt", "x"), + ...base, + }, +]; diff --git a/packages/connectors/test/fixtures/azure.ts b/packages/connectors/test/fixtures/azure.ts new file mode 100644 index 0000000..06ec4f4 --- /dev/null +++ b/packages/connectors/test/fixtures/azure.ts @@ -0,0 +1,118 @@ +import { expect } from "vitest"; +import { createAzureConnector } from "../../src/index.js"; +import { httpBackend, type Backend, type Scenario, type Subject } from "../certification.js"; +import type { FakeReply } from "../helpers.js"; + +export const TOKEN = "eyJ0eXAi.SECRET-AZURE-STORAGE-TOKEN.sig"; +export const ACCOUNT = "myaccount"; +export const CONTAINER = "docs"; +export const BK = ".reversi-backups/00000000-0000-4000-8000-000000000003"; + +type Reply = FakeReply & { bytes?: Uint8Array; headers?: Record }; +const xmlErr = (status: number, code: string, message = code): Reply => ({ + status, + headers: { "x-ms-error-code": code }, + text: `${code}${message}\nRequestId:abc`, +}); +const esc = (s: string) => s.replace(/&/g, "&").replace(/(); + let n = 0; + const b = httpBackend({ + state: () => ({ blobs: [...blobs].map(([k, v]) => [k, v.bytes.toString("base64"), v.type]).sort() }), + handle: (c): Reply => { + if (c.headers.authorization !== `Bearer ${TOKEN}`) return xmlErr(401, "AuthenticationFailed"); + if (!c.headers["x-ms-version"]) return xmlErr(400, "MissingRequiredHeader"); + const u = new URL(c.url); + if (u.host !== `${ACCOUNT}.blob.core.windows.net`) return xmlErr(404, "ResourceNotFound"); + const parts = u.pathname.slice(1).split("/"); + const container = parts.shift(); + const name = decodeURIComponent(parts.join("/")); + if (container !== CONTAINER) return xmlErr(404, "ContainerNotFound"); + if (c.method === "GET" && name === "") { + const prefix = u.searchParams.get("prefix") ?? ""; + const rows = [...blobs].filter(([k]) => k.startsWith(prefix)).sort(([a], [x]) => (a < x ? -1 : 1)); + return { + text: + `` + + rows.map(([k, v]) => `${esc(k)}Wed, 01 Jan 2026 00:00:00 GMT${v.etag}${v.bytes.length}`).join("") + + ``, + }; + } + const cur = blobs.get(name); + const copySource = c.headers["x-ms-copy-source"]; + if (c.method === "PUT" && copySource) { + if (c.headers["x-ms-requires-sync"] !== "true") return xmlErr(400, "InvalidHeaderValue", "copy must be synchronous"); + if (c.headers["x-ms-copy-source-authorization"] !== `Bearer ${TOKEN}`) return xmlErr(403, "CannotVerifyCopySource"); + const srcName = decodeURIComponent(new URL(copySource).pathname.slice(1).split("/").slice(1).join("/")); + const src = blobs.get(srcName); + if (!src) return xmlErr(404, "BlobNotFound"); + const want = c.headers["x-ms-source-if-match"]; + if (want && want !== src.etag) return xmlErr(412, "SourceConditionNotMet"); + blobs.set(name, { bytes: Buffer.from(src.bytes), type: src.type, etag: `"0x${++n}"` }); + return { status: 202, headers: { "x-ms-copy-status": "success", etag: `"0x${n}"` } }; + } + if (c.method === "PUT") { + if (c.headers["x-ms-blob-type"] !== "BlockBlob") return xmlErr(400, "MissingRequiredHeader"); + const ifMatch = c.headers["if-match"]; + if (ifMatch && (!cur || cur.etag !== ifMatch)) return xmlErr(412, "ConditionNotMet"); + if (c.headers["if-none-match"] === "*" && cur) return xmlErr(409, "BlobAlreadyExists"); + const etag = `"0x${++n}"`; + blobs.set(name, { bytes: Buffer.from(c.body, "latin1"), type: c.headers["content-type"] ?? "", etag }); + return { status: 201, headers: { etag } }; + } + if (c.method === "DELETE") { + if (!cur) return xmlErr(404, "BlobNotFound"); + const ifMatch = c.headers["if-match"]; + if (ifMatch && cur.etag !== ifMatch) return xmlErr(412, "ConditionNotMet"); + blobs.delete(name); + return { status: 202 }; + } + if (!cur) return c.method === "HEAD" ? { status: 404, headers: { "x-ms-error-code": "BlobNotFound" } } : xmlErr(404, "BlobNotFound"); + if (c.method === "HEAD") return { headers: { etag: cur.etag, "content-length": String(cur.bytes.length) } }; + const range = /^bytes=0-(\d+)$/.exec(c.headers.range ?? ""); + const part = cur.bytes.subarray(0, range ? Number(range[1]) + 1 : cur.bytes.length); + return { status: part.length < cur.bytes.length ? 206 : 200, bytes: part, headers: { "content-type": cur.type, etag: cur.etag, "content-range": `bytes 0-${part.length - 1}/${cur.bytes.length}` } }; + }, + }); + return Object.assign(b, { blobs }); +} +export type FakeBlob = ReturnType; +const az = (b: Backend) => b as FakeBlob; +const put = (b: Backend, key: string, content: string) => void az(b).blobs.set(key, { bytes: Buffer.from(content), type: "text/plain", etag: '"0x0"' }); +const seed = (b: Backend) => put(b, "a/b.txt", "v1"); +const visible = (b: Backend) => (az(b).state() as { blobs: string[][] }).blobs.filter(([k]) => !k!.startsWith(".reversi-backups/")); +const backups = (b: Backend) => [...az(b).blobs.keys()].filter((k) => k.startsWith(".reversi-backups/")); + +export const subject: Subject = { + name: "azure", + secrets: [TOKEN], + make: (o) => { + const backend = fakeBlob(); + return { backend, connector: createAzureConnector({ account: ACCOUNT, token: TOKEN, fetch: backend.fetch, timeoutMs: o?.timeoutMs }) }; + }, + faultBody: (status, secret) => ({ text: `AuthorizationFailurestatus ${status} ${secret}`, headers: { "x-ms-error-code": "AuthorizationFailure" } } as FakeReply), +}; + +const base = { fingerprint: visible }; +export const scenarios: Scenario[] = [ + { tool: "get_blob", args: { container: CONTAINER, blob: "a/b.txt" }, setup: seed, expect: (r) => expect(r).toMatchObject({ content: "v1", encoding: "utf8", size: 2 }) }, + { + tool: "list_blobs", + args: { container: CONTAINER, prefix: "a/" }, + setup: (b) => (seed(b), put(b, BK, "hidden")), + expect: (r) => expect((r as { blobs: Array<{ blob: string }> }).blobs.map((x) => x.blob)).toEqual(["a/b.txt"]), + }, + { tool: "put_blob", args: { container: CONTAINER, blob: "a/new.txt", content: "hello" }, ...base, expect: (r) => expect(r).toMatchObject({ existed: false, backup_blob: null }) }, + { tool: "put_blob", args: { container: CONTAINER, blob: "a/b.txt", content: "v2" }, setup: seed, ...base, expect: (r, b) => (expect(r).toMatchObject({ existed: true }), expect(backups(b)).toHaveLength(1)) }, + { tool: "delete_blob", args: { container: CONTAINER, blob: "a/b.txt" }, setup: seed, ...base, expect: (_r, b) => expect(backups(b)).toHaveLength(1) }, + { + tool: "restore_blob", + args: { container: CONTAINER, blob: "a/b.txt", backup_blob: BK }, + setup: (b) => (put(b, "a/b.txt", "v2"), put(b, BK, "v1")), + ...base, + expect: (_r, b) => expect(az(b).blobs.get("a/b.txt")!.bytes.toString()).toBe("v1"), + }, + { tool: "restore_blob", args: { container: CONTAINER, blob: "a/new.txt", backup_blob: null }, setup: (b) => put(b, "a/new.txt", "x"), ...base }, +]; diff --git a/packages/connectors/test/fixtures/fs.ts b/packages/connectors/test/fixtures/fs.ts new file mode 100644 index 0000000..805daf8 --- /dev/null +++ b/packages/connectors/test/fixtures/fs.ts @@ -0,0 +1,133 @@ +import { mkdirSync, mkdtempSync, readFileSync, readdirSync, rmSync, statSync, unlinkSync, writeFileSync, rmdirSync } from "node:fs"; +import { tmpdir } from "node:os"; +import path from "node:path"; +import { expect } from "vitest"; +import { createFilesystemConnector } from "../../src/index.js"; +import type { Backend, Scenario, Subject } from "../certification.js"; + +const dirs: string[] = []; +export function cleanup(): void { + for (const d of dirs.splice(0)) rmSync(d, { recursive: true, force: true }); +} + +const BACKUP = "00000000-0000-4000-8000-0000000000aa"; + +function tree(root: string, rel = ""): Record { + const out: Record = {}; + for (const name of readdirSync(path.join(root, rel)).sort()) { + const r = rel ? `${rel}/${name}` : name; + if (r === ".reversi-backups") continue; + if (statSync(path.join(root, r)).isDirectory()) { + out[r + "/"] = null; + Object.assign(out, tree(root, r)); + } else out[r] = readFileSync(path.join(root, r), "utf8"); + } + return out; +} + +export function fakeFs() { + const root = mkdtempSync(path.join(tmpdir(), "rv-cert-fs-")); + dirs.push(root); + const p = (rel: string) => path.join(root, rel); + const b: Backend & { root: string; p: (rel: string) => string } = { + fault: null, + calls: [], + supportsFaults: false, + isMutating: () => false, + state: () => tree(root), + root, + p, + }; + return b; +} +type FakeFs = ReturnType; +const fx = (b: Backend) => b as FakeFs; +const file = (b: Backend, rel: string, content: string) => { + mkdirSync(path.dirname(fx(b).p(rel)), { recursive: true }); + writeFileSync(fx(b).p(rel), content); +}; +const seedBackup = (b: Backend, rel: string, content: string | null) => { + const dir = path.join(fx(b).root, ".reversi-backups", BACKUP); + mkdirSync(dir, { recursive: true }); + if (content !== null) writeFileSync(path.join(dir, "content"), content); + writeFileSync(path.join(dir, "meta.json"), JSON.stringify({ path: rel, existed: content !== null, restored: false })); +}; +/** Swap a file for a directory (and back) so the undo has something it must refuse to overwrite. */ +const blockWithDir = (rel: string) => (b: Backend) => { + rmSync(fx(b).p(rel), { force: true }); + mkdirSync(fx(b).p(rel)); +}; +const unblock = (rel: string) => (b: Backend) => rmdirSync(fx(b).p(rel)); + +export const subject: Subject = { + name: "fs", + secrets: [], + make: () => { + const backend = fakeFs(); + return { backend, connector: createFilesystemConnector({ root: backend.root }) }; + }, + notApplicable: { + retry: "the local filesystem has no network transport that can fail transiently; errors are definite", + permission_denial: "access control is the operating system's; the connector confines paths to its root instead (covered in filesystem.test.ts)", + timeout: "local file operations have no network wait to time out", + redaction: "the connector holds no credential", + }, +}; + +export const scenarios: Scenario[] = [ + { tool: "read_file", args: { path: "a.txt" }, setup: (b) => file(b, "a.txt", "one"), fail: (b) => unlinkSync(fx(b).p("a.txt")), expect: (r) => expect(r).toMatchObject({ content: "one", size: 3 }) }, + { + tool: "list_dir", + args: { path: "d" }, + setup: (b) => file(b, "d/a.txt", "one"), + fail: (b) => (rmSync(fx(b).p("d"), { recursive: true }), writeFileSync(fx(b).p("d"), "now a file")), + expect: (r) => expect(r).toMatchObject({ entries: [{ name: "a.txt", type: "file" }] }), + }, + { + tool: "write_file", + args: { path: "new.txt", content: "hello" }, + fail: (b) => mkdirSync(fx(b).p("new.txt")), + breakCompensation: blockWithDir("new.txt"), + healCompensation: unblock("new.txt"), + }, + { + tool: "write_file", + args: { path: "a.txt", content: "two" }, + setup: (b) => file(b, "a.txt", "one"), + fail: (b) => (unlinkSync(fx(b).p("a.txt")), mkdirSync(fx(b).p("a.txt"))), + breakCompensation: blockWithDir("a.txt"), + healCompensation: unblock("a.txt"), + }, + { + tool: "delete_file", + args: { path: "a.txt" }, + setup: (b) => file(b, "a.txt", "one"), + fail: (b) => unlinkSync(fx(b).p("a.txt")), + breakCompensation: (b) => mkdirSync(fx(b).p("a.txt")), + healCompensation: unblock("a.txt"), + }, + { + tool: "restore_file", + args: { path: "a.txt", backup_id: BACKUP }, + setup: (b) => (file(b, "a.txt", "two"), seedBackup(b, "a.txt", "one")), + fail: (b) => rmSync(path.join(fx(b).root, ".reversi-backups"), { recursive: true, force: true }), + expect: (_r, b) => expect(readFileSync(fx(b).p("a.txt"), "utf8")).toBe("one"), + }, + { + tool: "make_dir", + args: { path: "d" }, + fail: (b) => mkdirSync(fx(b).p("d")), + breakCompensation: (b) => writeFileSync(fx(b).p("d/inside.txt"), "x"), + healCompensation: (b) => unlinkSync(fx(b).p("d/inside.txt")), + }, + { tool: "remove_dir", args: { path: "d" }, setup: (b) => mkdirSync(fx(b).p("d")), fail: (b) => writeFileSync(fx(b).p("d/inside.txt"), "x") }, + { + tool: "move_file", + args: { from: "a.txt", to: "b.txt" }, + setup: (b) => file(b, "a.txt", "one"), + fail: (b) => unlinkSync(fx(b).p("a.txt")), + breakCompensation: (b) => writeFileSync(fx(b).p("a.txt"), "intruder"), + healCompensation: (b) => unlinkSync(fx(b).p("a.txt")), + }, + { tool: "unmove_file", args: { from: "a.txt", to: "b.txt" }, setup: (b) => file(b, "b.txt", "one"), fail: (b) => unlinkSync(fx(b).p("b.txt")) }, +]; diff --git a/packages/connectors/test/fixtures/gcalendar.ts b/packages/connectors/test/fixtures/gcalendar.ts new file mode 100644 index 0000000..8b50315 --- /dev/null +++ b/packages/connectors/test/fixtures/gcalendar.ts @@ -0,0 +1,99 @@ +import { expect } from "vitest"; +import { createGcalendarConnector } from "../../src/index.js"; +import { httpBackend, type Backend, type Scenario, type Subject } from "../certification.js"; + +export const TOKEN = "ya29.SECRET-GOOGLE-CALENDAR-TOKEN"; + +type Ev = Record; + +function merge(target: Ev, patch: Ev): void { + for (const [k, v] of Object.entries(patch)) { + if (v === null) delete target[k]; + else if (typeof v === "object" && !Array.isArray(v) && typeof target[k] === "object" && target[k] !== null) merge(target[k], v); + else target[k] = typeof v === "object" ? JSON.parse(JSON.stringify(v, (_k, x) => (x === null ? undefined : x))) : v; + } +} + +export function fakeCalendar() { + const events = new Map(); + let n = 0; + const b = httpBackend({ + state: () => ({ events: [...events].sort() }), + handle: (c) => { + const u = new URL(c.url); + const p = u.pathname.replace("/calendar/v3/calendars/primary/events", ""); + if ((c.method === "POST" || c.method === "PATCH" || c.method === "DELETE") && u.searchParams.get("sendUpdates") !== "none") { + return { status: 400, json: { error: { message: "writes must pass sendUpdates=none" } } }; + } + if (c.method === "POST" && p === "") { + const id = `ev${++n}`; + const ev = { id, ...JSON.parse(c.body) }; + events.set(id, ev); + return { json: { id, htmlLink: `https://calendar.example/${id}` } }; + } + if (c.method === "GET" && p === "") { + return { json: { items: [...events.values()].map((e) => ({ id: e.id, summary: e.summary, start: e.start, end: e.end, status: "confirmed" })), nextPageToken: undefined } }; + } + const m = p.match(/^\/([A-Za-z0-9_-]+)$/); + if (m) { + const ev = events.get(m[1]!); + if (!ev) return { status: 404, json: { error: { code: 404, message: "Not Found" } } }; + if (c.method === "GET") return { json: ev }; + if (c.method === "PATCH") { + merge(ev, JSON.parse(c.body)); + return { json: ev }; + } + if (c.method === "DELETE") { + events.delete(m[1]!); + return { status: 204 }; + } + } + return { status: 500, json: { error: { message: `unexpected ${c.method} ${c.url}` } } }; + }, + }); + return Object.assign(b, { events }); +} +type FakeCal = ReturnType; +const cal = (b: Backend) => b as FakeCal; +const seed = (b: Backend) => + void cal(b).events.set("ev9", { + id: "ev9", + summary: "Standup", + start: { dateTime: "2026-05-01T09:00:00Z" }, + end: { dateTime: "2026-05-01T09:15:00Z" }, + }); + +export const subject: Subject = { + name: "gcalendar", + secrets: [TOKEN], + make: (o) => { + const backend = fakeCalendar(); + return { backend, connector: createGcalendarConnector({ token: TOKEN, fetch: backend.fetch, timeoutMs: o?.timeoutMs }) }; + }, + faultBody: (_s, secret) => ({ json: { error: { code: 403, message: `forbidden ${secret}` } } }), +}; + +export const scenarios: Scenario[] = [ + { tool: "list_events", args: { time_min: "2026-05-01T00:00:00Z", max_results: 10 }, setup: seed, expect: (r) => expect((r as { events: unknown[] }).events).toHaveLength(1) }, + { + tool: "create_event", + args: { summary: "Lunch", start: "2026-05-02T12:00:00+02:00", end: "2026-05-02T13:00:00+02:00", time_zone: "Europe/Berlin", location: "Cafe" }, + expect: (r) => expect(r).toMatchObject({ calendar_id: "primary", event_id: "ev1" }), + }, + { tool: "delete_event", args: { event_id: "ev9" }, setup: seed }, + { + tool: "update_event", + args: { event_id: "ev9", summary: "Planning", location: "Room 1", start: "2026-05-01T10:00:00Z", end: "2026-05-01T10:30:00Z" }, + setup: seed, + expect: (r) => + expect(r).toMatchObject({ + event_id: "ev9", + previous: { summary: "Standup", location: null, start: { dateTime: "2026-05-01T09:00:00Z" }, end: { dateTime: "2026-05-01T09:15:00Z" } }, + }), + }, + { + tool: "restore_event", + args: { event_id: "ev9", previous: { summary: "Standup", location: null } }, + setup: (b) => void cal(b).events.set("ev9", { id: "ev9", summary: "Planning", location: "Room 1" }), + }, +]; diff --git a/packages/connectors/test/fixtures/gcp.ts b/packages/connectors/test/fixtures/gcp.ts new file mode 100644 index 0000000..212f846 --- /dev/null +++ b/packages/connectors/test/fixtures/gcp.ts @@ -0,0 +1,106 @@ +import { expect } from "vitest"; +import { createGcpConnector } from "../../src/index.js"; +import { httpBackend, type Backend, type Scenario, type Subject } from "../certification.js"; +import type { FakeReply } from "../helpers.js"; + +export const TOKEN = "ya29.SECRET-GCP-ACCESS-TOKEN"; +export const BUCKET = "my-bucket"; +export const BK = ".reversi-backups/00000000-0000-4000-8000-000000000002"; + +type Reply = FakeReply & { bytes?: Uint8Array; headers?: Record }; +const err = (status: number, message: string): Reply => ({ status, json: { error: { code: status, message } } }); + +export function fakeGcs() { + const objects = new Map(); + let gen = 1000; + const b = httpBackend({ + state: () => ({ objects: [...objects].map(([k, v]) => [k, v.bytes.toString("base64"), v.type]).sort() }), + handle: (c): Reply => { + if (c.headers.authorization !== `Bearer ${TOKEN}`) return err(401, "Invalid Credentials"); + const u = new URL(c.url); + const p = u.pathname; + const q = u.searchParams; + let m: RegExpMatchArray | null; + if (c.method === "POST" && p === `/upload/storage/v1/b/${BUCKET}/o`) { + const name = q.get("name")!; + const want = q.get("ifGenerationMatch"); + const have = objects.get(name)?.generation ?? 0; + if (want !== null && Number(want) !== have) return err(412, "Precondition Failed"); + objects.set(name, { bytes: Buffer.from(c.body, "latin1"), type: c.headers["content-type"] ?? "", generation: ++gen }); + return { json: { name, generation: String(gen) } }; + } + if (c.method === "GET" && p === `/storage/v1/b/${BUCKET}/o`) { + const prefix = q.get("prefix") ?? ""; + const items = [...objects].filter(([k]) => k.startsWith(prefix)).sort(([a], [x]) => (a < x ? -1 : 1)).map(([name, v]) => ({ name, size: String(v.bytes.length), updated: "2026-01-01T00:00:00Z", contentType: v.type })); + return { json: { items } }; + } + if ((m = p.match(new RegExp(`^/storage/v1/b/${BUCKET}/o/([^/]+)/rewriteTo/b/${BUCKET}/o/([^/]+)$`))) && c.method === "POST") { + const src = objects.get(decodeURIComponent(m[1]!)); + if (!src) return err(404, "No such object"); + const want = q.get("sourceGeneration"); + if (want !== null && Number(want) !== src.generation) return err(412, "source generation changed"); + if (q.get("rewriteToken") !== "round-2") return { json: { done: false, rewriteToken: "round-2" } }; + objects.set(decodeURIComponent(m[2]!), { bytes: Buffer.from(src.bytes), type: src.type, generation: ++gen }); + return { json: { done: true } }; + } + if ((m = p.match(new RegExp(`^/storage/v1/b/${BUCKET}/o/([^/]+)$`)))) { + const name = decodeURIComponent(m[1]!); + const o = objects.get(name); + if (c.method === "DELETE") { + if (!o) return err(404, "No such object"); + const want = q.get("ifGenerationMatch"); + if (want !== null && Number(want) !== o.generation) return err(412, "Precondition Failed"); + objects.delete(name); + return { status: 204 }; + } + if (c.method === "GET") { + if (!o) return err(404, "No such object"); + if (q.get("alt") !== "media") return { json: { generation: String(o.generation) } }; + const range = /^bytes=0-(\d+)$/.exec(c.headers.range ?? ""); + const part = o.bytes.subarray(0, range ? Number(range[1]) + 1 : o.bytes.length); + return { status: part.length < o.bytes.length ? 206 : 200, bytes: part, headers: { "content-type": o.type, "content-range": `bytes 0-${part.length - 1}/${o.bytes.length}` } }; + } + } + return err(500, `unexpected ${c.method} ${c.url}`); + }, + }); + return Object.assign(b, { objects }); +} +export type FakeGcs = ReturnType; +const gcs = (b: Backend) => b as FakeGcs; +const put = (b: Backend, key: string, content: string) => void gcs(b).objects.set(key, { bytes: Buffer.from(content), type: "text/plain", generation: 5 }); +const seed = (b: Backend) => put(b, "docs/a.txt", "v1"); +const visible = (b: Backend) => (gcs(b).state() as { objects: string[][] }).objects.filter(([k]) => !k!.startsWith(".reversi-backups/")); +const backups = (b: Backend) => [...gcs(b).objects.keys()].filter((k) => k.startsWith(".reversi-backups/")); + +export const subject: Subject = { + name: "gcp", + secrets: [TOKEN], + make: (o) => { + const backend = fakeGcs(); + return { backend, connector: createGcpConnector({ accessToken: TOKEN, fetch: backend.fetch, timeoutMs: o?.timeoutMs }) }; + }, + faultBody: (status, secret) => ({ json: { error: { code: status, message: `denied ${secret}` } } }), +}; + +const base = { fingerprint: visible }; +export const scenarios: Scenario[] = [ + { tool: "get_object", args: { bucket: BUCKET, key: "docs/a.txt" }, setup: seed, expect: (r) => expect(r).toMatchObject({ content: "v1", encoding: "utf8", size: 2 }) }, + { + tool: "list_objects", + args: { bucket: BUCKET, prefix: "docs/" }, + setup: (b) => (seed(b), put(b, BK, "hidden")), + expect: (r) => expect((r as { objects: Array<{ key: string }> }).objects.map((o) => o.key)).toEqual(["docs/a.txt"]), + }, + { tool: "put_object", args: { bucket: BUCKET, key: "docs/new.txt", content: "aGk=", encoding: "base64" }, ...base, expect: (r) => expect(r).toMatchObject({ existed: false, backup_key: null }) }, + { tool: "put_object", args: { bucket: BUCKET, key: "docs/a.txt", content: "v2" }, setup: seed, ...base, expect: (r, b) => (expect(r).toMatchObject({ existed: true }), expect(backups(b)).toHaveLength(1)) }, + { tool: "delete_object", args: { bucket: BUCKET, key: "docs/a.txt" }, setup: seed, ...base, expect: (_r, b) => expect(backups(b)).toHaveLength(1) }, + { + tool: "restore_object", + args: { bucket: BUCKET, key: "docs/a.txt", backup_key: BK }, + setup: (b) => (put(b, "docs/a.txt", "v2"), put(b, BK, "v1")), + ...base, + expect: (_r, b) => expect(gcs(b).objects.get("docs/a.txt")!.bytes.toString()).toBe("v1"), + }, + { tool: "restore_object", args: { bucket: BUCKET, key: "docs/new.txt", backup_key: null }, setup: (b) => put(b, "docs/new.txt", "x"), ...base }, +]; diff --git a/packages/connectors/test/fixtures/github.ts b/packages/connectors/test/fixtures/github.ts new file mode 100644 index 0000000..7715481 --- /dev/null +++ b/packages/connectors/test/fixtures/github.ts @@ -0,0 +1,101 @@ +import { expect } from "vitest"; +import { createGithubConnector } from "../../src/index.js"; +import { httpBackend, type Backend, type Scenario, type Subject } from "../certification.js"; + +export const TOKEN = "ghp_SECRETGITHUBTOKEN0123456789abcdef"; +export const SHA = "a".repeat(40); +const T = { owner: "acme", repo: "app" }; + +export function fakeGithub() { + const issues = new Map }>(); + const comments = new Map(); + const branches = new Set(["main"]); + let nIssue = 0; + let nComment = 100; + const b = httpBackend({ + state: () => ({ + issues: [...issues].map(([n, i]) => [n, i.title, i.state, [...i.labels].sort()]), + comments: [...comments], + branches: [...branches].sort(), + }), + handle: (c) => { + const p = new URL(c.url).pathname.replace("/repos/acme/app", ""); + const body = c.body ? JSON.parse(c.body) : {}; + let m: RegExpMatchArray | null; + if (c.method === "POST" && p === "/issues") { + const n = ++nIssue; + issues.set(n, { title: body.title, state: "open", labels: new Set() }); + return { status: 201, json: { number: n, html_url: `https://github.example/${n}`, state: "open" } }; + } + if ((m = p.match(/^\/issues\/(\d+)$/))) { + const i = issues.get(Number(m[1])); + if (!i) return { status: 404, json: { message: "Not Found" } }; + if (c.method === "GET") return { json: { title: i.title, state: i.state, body: null, labels: [...i.labels].map((name) => ({ name })), html_url: "u" } }; + if (c.method === "PATCH") { + i.state = body.state; + return { json: { state: i.state } }; + } + } + if (c.method === "POST" && (m = p.match(/^\/issues\/(\d+)\/comments$/))) { + if (!issues.has(Number(m[1]))) return { status: 404, json: { message: "Not Found" } }; + const id = ++nComment; + comments.set(id, Number(m[1])); + return { status: 201, json: { id, html_url: `https://github.example/c/${id}` } }; + } + if (c.method === "DELETE" && (m = p.match(/^\/issues\/comments\/(\d+)$/))) { + return comments.delete(Number(m[1])) ? { status: 204 } : { status: 404, json: { message: "Not Found" } }; + } + if (c.method === "POST" && p === "/git/refs") { + const name = String(body.ref).replace("refs/heads/", ""); + if (branches.has(name)) return { status: 422, json: { message: "Reference already exists" } }; + branches.add(name); + return { status: 201, json: { ref: body.ref, object: { sha: body.sha } } }; + } + if (c.method === "DELETE" && (m = p.match(/^\/git\/refs\/heads\/(.+)$/))) { + return branches.delete(decodeURIComponent(m[1]!).replace(/%2F/g, "/")) ? { status: 204 } : { status: 422, json: { message: "Reference does not exist" } }; + } + if (c.method === "POST" && (m = p.match(/^\/issues\/(\d+)\/labels$/))) { + const i = issues.get(Number(m[1])); + if (!i) return { status: 404, json: { message: "Not Found" } }; + for (const l of body.labels) i.labels.add(l); + return { json: [...i.labels].map((name) => ({ name })) }; + } + if (c.method === "DELETE" && (m = p.match(/^\/issues\/(\d+)\/labels\/(.+)$/))) { + const i = issues.get(Number(m[1])); + return i?.labels.delete(decodeURIComponent(m[2]!)) ? { json: [] } : { status: 404, json: { message: "Label does not exist" } }; + } + return { status: 500, json: { message: `unexpected ${c.method} ${c.url}` } }; + }, + }); + return Object.assign(b, { issues, comments, branches }); +} +type FakeGithub = ReturnType; +const gh = (b: Backend) => b as FakeGithub; +const seed = (b: Backend) => void gh(b).issues.set(1, { title: "seed", state: "open", labels: new Set(["bug"]) }); + +export const subject: Subject = { + name: "github", + secrets: [TOKEN], + make: (o) => { + const backend = fakeGithub(); + return { backend, connector: createGithubConnector({ token: TOKEN, fetch: backend.fetch, timeoutMs: o?.timeoutMs }) }; + }, +}; + +export const scenarios: Scenario[] = [ + { tool: "get_issue", args: { ...T, issue_number: 1 }, setup: seed, expect: (r) => expect(r).toMatchObject({ title: "seed", state: "open", labels: ["bug"] }) }, + { + tool: "create_issue", + args: { ...T, title: "Bug", labels: ["bug"] }, + expect: (r) => expect(r).toMatchObject({ issue_number: 1 }), + // GitHub cannot delete issues: rollback closes it, and it stays visible + restored: (_b, after) => expect((after as { issues: unknown[][] }).issues.map((i) => i[2])).toEqual(["closed"]), + }, + { tool: "close_issue", args: { ...T, issue_number: 1 }, setup: seed }, + { tool: "add_comment", args: { ...T, issue_number: 1, body: "hello" }, setup: seed, expect: (r) => expect(r).toMatchObject({ comment_id: 101 }) }, + { tool: "delete_comment", args: { ...T, comment_id: 55 }, setup: (b) => (seed(b), void gh(b).comments.set(55, 1)) }, + { tool: "create_branch", args: { ...T, branch: "feature/x", sha: SHA } }, + { tool: "delete_branch", args: { ...T, branch: "old" }, setup: (b) => void gh(b).branches.add("old") }, + { tool: "add_labels", args: { ...T, issue_number: 1, labels: ["bug", "new"] }, setup: seed, expect: (r) => expect(r).toMatchObject({ added: ["new"] }) }, + { tool: "remove_labels", args: { ...T, issue_number: 1, labels: ["bug"] }, setup: seed }, +]; diff --git a/packages/connectors/test/fixtures/gitlab.ts b/packages/connectors/test/fixtures/gitlab.ts new file mode 100644 index 0000000..29840ff --- /dev/null +++ b/packages/connectors/test/fixtures/gitlab.ts @@ -0,0 +1,83 @@ +import { expect } from "vitest"; +import { createGitlabConnector } from "../../src/index.js"; +import { httpBackend, type Backend, type Scenario, type Subject } from "../certification.js"; + +export const TOKEN = "glpat-SECRET-GITLAB-TOKEN-123"; +export const PROJECT = "group/proj"; + +export function fakeGitlab() { + const issues = new Map(); + const notes = new Map(); + const branches = new Set(["main"]); + let nextIid = 1; + let nextNote = 100; + const b = httpBackend({ + state: () => ({ issues: [...issues], notes: [...notes], branches: [...branches].sort() }), + handle: (c) => { + const u = new URL(c.url); + const p = u.pathname.replace("/api/v4/projects/group%2Fproj", ""); + const body = c.body ? JSON.parse(c.body) : {}; + let m: RegExpMatchArray | null; + if (c.method === "POST" && p === "/issues") { + const iid = nextIid++; + issues.set(iid, { title: body.title, state: "opened" }); + return { status: 201, json: { iid, state: "opened", web_url: `https://gitlab.example/g/p/-/issues/${iid}` } }; + } + if ((m = p.match(/^\/issues\/(\d+)$/))) { + const i = issues.get(Number(m[1])); + if (!i) return { status: 404, json: { message: "404 Not found" } }; + if (c.method === "GET") return { json: { iid: Number(m[1]), title: i.title, state: i.state, description: null, labels: [], web_url: "u" } }; + if (c.method === "PUT") { + i.state = "closed"; + return { json: { iid: Number(m[1]), state: "closed" } }; + } + } + if (c.method === "POST" && (m = p.match(/^\/issues\/(\d+)\/notes$/))) { + if (!issues.has(Number(m[1]))) return { status: 404, json: { message: "404 Not found" } }; + const id = nextNote++; + notes.set(id, Number(m[1])); + return { status: 201, json: { id } }; + } + if (c.method === "DELETE" && (m = p.match(/^\/issues\/(\d+)\/notes\/(\d+)$/))) { + return notes.delete(Number(m[2])) ? { status: 204 } : { status: 404, json: { message: "404 Not found" } }; + } + if (c.method === "POST" && p === "/repository/branches") { + if (branches.has(body.branch)) return { status: 400, json: { message: "Branch already exists" } }; + branches.add(body.branch); + return { status: 201, json: { name: body.branch, commit: { id: "abc123" } } }; + } + if (c.method === "DELETE" && (m = p.match(/^\/repository\/branches\/(.+)$/))) { + return branches.delete(decodeURIComponent(m[1]!)) ? { status: 204 } : { status: 404, json: { message: "404 Branch Not Found" } }; + } + return { status: 500, json: { message: `unexpected ${c.method} ${c.url}` } }; + }, + }); + return Object.assign(b, { issues, notes, branches }); +} +export type FakeGitlab = ReturnType; +const gl = (b: Backend) => b as FakeGitlab; +const seedIssue = (b: Backend) => void gl(b).issues.set(1, { title: "seed", state: "opened" }); +const seedNote = (b: Backend) => { + seedIssue(b); + gl(b).notes.set(7, 1); +}; + +export const subject: Subject = { + name: "gitlab", + secrets: [TOKEN], + make: (o) => { + const backend = fakeGitlab(); + return { backend, connector: createGitlabConnector({ token: TOKEN, fetch: backend.fetch, timeoutMs: o?.timeoutMs }) }; + }, +}; + +export const scenarios: Scenario[] = [ + { tool: "get_issue", args: { project: PROJECT, issue_iid: 1 }, setup: seedIssue, expect: (r) => expect(r).toMatchObject({ title: "seed", state: "opened" }) }, + { tool: "create_issue", args: { project: PROJECT, title: "T", labels: ["a", "b"] }, expect: (r) => expect(r).toMatchObject({ project: PROJECT, issue_iid: 1 }), + restored: (_before, after) => expect((after as { issues: Array<[number, { state: string }]> }).issues[0]![1].state).toBe("closed") }, + { tool: "close_issue", args: { project: PROJECT, issue_iid: 1 }, setup: seedIssue }, + { tool: "add_note", args: { project: PROJECT, issue_iid: 1, body: "hello" }, setup: seedIssue, expect: (r) => expect(r).toMatchObject({ note_id: 100 }) }, + { tool: "delete_note", args: { project: PROJECT, issue_iid: 1, note_id: 7 }, setup: seedNote }, + { tool: "create_branch", args: { project: PROJECT, branch: "feature/x", ref: "main" } }, + { tool: "delete_branch", args: { project: PROJECT, branch: "old" }, setup: (b) => void gl(b).branches.add("old") }, +]; diff --git a/packages/connectors/test/fixtures/index.ts b/packages/connectors/test/fixtures/index.ts new file mode 100644 index 0000000..d548f0e --- /dev/null +++ b/packages/connectors/test/fixtures/index.ts @@ -0,0 +1,25 @@ +import type { Scenario, Subject } from "../certification.js"; +import * as aws from "./aws.js"; +import * as azure from "./azure.js"; +import * as fs from "./fs.js"; +import * as gcalendar from "./gcalendar.js"; +import * as gcp from "./gcp.js"; +import * as github from "./github.js"; +import * as gitlab from "./gitlab.js"; +import * as jira from "./jira.js"; +import * as kubernetes from "./kubernetes.js"; +import * as linear from "./linear.js"; +import * as postgres from "./postgres.js"; +import * as slack from "./slack.js"; +import * as stripe from "./stripe.js"; +import * as teams from "./teams.js"; + +export interface Fixture { + subject: Subject; + scenarios: Scenario[]; +} + +/** Every connector, in the order the docs list them. */ +export const fixtures: Fixture[] = [fs, github, slack, stripe, gitlab, teams, linear, jira, gcalendar, postgres, aws, gcp, azure, kubernetes]; + +export { cleanup as cleanupFs } from "./fs.js"; diff --git a/packages/connectors/test/fixtures/jira.ts b/packages/connectors/test/fixtures/jira.ts new file mode 100644 index 0000000..3d6c2d7 --- /dev/null +++ b/packages/connectors/test/fixtures/jira.ts @@ -0,0 +1,107 @@ +import { expect } from "vitest"; +import { createJiraConnector } from "../../src/index.js"; +import { httpBackend, type Backend, type Scenario, type Subject } from "../certification.js"; + +export const TOKEN = "ATATT3-SECRET-JIRA-API-TOKEN"; +export const EMAIL = "bot@example.com"; +export const BASIC = Buffer.from(`${EMAIL}:${TOKEN}`).toString("base64"); +export const BASE = "https://example.atlassian.net"; + +const STATUSES: Record = { "1": "To Do", "2": "In Progress", "3": "Done" }; +const TRANSITIONS: Array<{ id: string; name: string; to: string }> = [ + { id: "11", name: "Reopen", to: "1" }, + { id: "21", name: "Start progress", to: "2" }, + { id: "31", name: "Finish", to: "3" }, +]; + +export function fakeJira() { + const issues = new Map(); + const comments = new Map(); + let nIssue = 0; + let nComment = 10000; + const gone = { status: 404, json: { errorMessages: ["Issue does not exist or you do not have permission to see it."] } }; + const b = httpBackend({ + state: () => ({ issues: [...issues], comments: [...comments] }), + handle: (c) => { + const u = new URL(c.url); + const p = u.pathname.replace("/rest/api/3", ""); + let m: RegExpMatchArray | null; + if (c.method === "POST" && p === "/issue") { + const key = `PROJ-${++nIssue}`; + issues.set(key, { summary: JSON.parse(c.body).fields.summary, status: "1" }); + return { status: 201, json: { id: String(10000 + nIssue), key } }; + } + if ((m = p.match(/^\/issue\/([A-Z]+-\d+)$/))) { + const i = issues.get(m[1]!); + if (!i) return gone; + if (c.method === "GET") return { json: { key: m[1], fields: { summary: i.summary, status: { id: i.status, name: STATUSES[i.status] } } } }; + if (c.method === "DELETE") { + issues.delete(m[1]!); + for (const [id, k] of [...comments]) if (k === m[1]) comments.delete(id); + return { status: 204 }; + } + } + if ((m = p.match(/^\/issue\/([A-Z]+-\d+)\/comment$/)) && c.method === "POST") { + if (!issues.has(m[1]!)) return gone; + const id = String(++nComment); + comments.set(id, m[1]!); + return { status: 201, json: { id } }; + } + if ((m = p.match(/^\/issue\/([A-Z]+-\d+)\/comment\/(\d+)$/)) && c.method === "DELETE") { + return comments.delete(m[2]!) ? { status: 204 } : { status: 404, json: { errorMessages: ["comment not found"] } }; + } + if ((m = p.match(/^\/issue\/([A-Z]+-\d+)\/transitions$/))) { + const i = issues.get(m[1]!); + if (!i) return gone; + if (c.method === "GET") { + return { + json: { + transitions: TRANSITIONS.filter((t) => t.to !== i.status).map((t) => ({ id: t.id, name: t.name, to: { id: t.to, name: STATUSES[t.to] } })), + }, + }; + } + const t = TRANSITIONS.find((x) => x.id === JSON.parse(c.body).transition.id && x.to !== i.status); + if (!t) return { status: 400, json: { errorMessages: ["transition not valid"] } }; + i.status = t.to; + return { status: 204 }; + } + return { status: 500, json: { errorMessages: [`unexpected ${c.method} ${c.url}`] } }; + }, + }); + return Object.assign(b, { issues, comments }); +} +type FakeJira = ReturnType; +const jr = (b: Backend) => b as FakeJira; +const seed = (b: Backend) => void jr(b).issues.set("PROJ-9", { summary: "seed", status: "1" }); +const seedComment = (b: Backend) => { + seed(b); + jr(b).comments.set("555", "PROJ-9"); +}; + +export const subject: Subject = { + name: "jira", + secrets: [TOKEN, BASIC], + make: (o) => { + const backend = fakeJira(); + return { + backend, + connector: createJiraConnector({ baseUrl: BASE, email: EMAIL, apiToken: TOKEN, fetch: backend.fetch, timeoutMs: o?.timeoutMs }), + }; + }, + faultBody: (_s, secret) => ({ json: { errorMessages: [`not allowed ${secret} ${BASIC}`] } }), +}; + +export const scenarios: Scenario[] = [ + { tool: "get_issue", args: { issue_key: "PROJ-9" }, setup: seed, expect: (r) => expect(r).toMatchObject({ summary: "seed", status: "To Do", status_id: "1" }) }, + { tool: "create_issue", args: { project_key: "PROJ", summary: "New", description: "line one\nline two" }, expect: (r) => expect(r).toMatchObject({ issue_key: "PROJ-1" }) }, + { tool: "delete_issue", args: { issue_key: "PROJ-9" }, setup: seed }, + { tool: "add_comment", args: { issue_key: "PROJ-9", body: "hi" }, setup: seed, expect: (r) => expect(r).toMatchObject({ comment_id: "10001" }) }, + { tool: "delete_comment", args: { issue_key: "PROJ-9", comment_id: "555" }, setup: seedComment }, + { + tool: "transition_issue", + args: { issue_key: "PROJ-9", transition: "Start progress" }, + setup: seed, + expect: (r) => expect(r).toMatchObject({ previous_status: "To Do", previous_status_id: "1", status: "In Progress", status_id: "2" }), + }, + { tool: "restore_status", args: { issue_key: "PROJ-9", status_id: "1" }, setup: (b) => void jr(b).issues.set("PROJ-9", { summary: "seed", status: "2" }) }, +]; diff --git a/packages/connectors/test/fixtures/kubernetes.ts b/packages/connectors/test/fixtures/kubernetes.ts new file mode 100644 index 0000000..158388c --- /dev/null +++ b/packages/connectors/test/fixtures/kubernetes.ts @@ -0,0 +1,107 @@ +import { expect } from "vitest"; +import { createKubernetesConnector } from "../../src/index.js"; +import { httpBackend, type Backend, type Scenario, type Subject } from "../certification.js"; +import type { FakeReply } from "../helpers.js"; + +export const TOKEN = "eyJhbGciOi.SECRET-KUBE-SERVICE-ACCOUNT-TOKEN.sig"; +export const SEED_UID = "11111111-0000-4000-8000-000000000001"; +export const SERVER = "https://kube.example:6443"; + +const status = (code: number, reason: string, message: string): FakeReply => ({ status: code, json: { kind: "Status", status: "Failure", message, reason, code } }); + +export function fakeKube() { + const deployments = new Map(); + const configmaps = new Map }>(); + const pods = new Map; phase: string }>(); + let n = 0; + const b = httpBackend({ + state: () => ({ deployments: [...deployments], configmaps: [...configmaps], pods: [...pods] }), + handle: (c) => { + if (c.headers.authorization !== `Bearer ${TOKEN}`) return status(401, "Unauthorized", "Unauthorized"); + const p = new URL(c.url).pathname; + const q = new URL(c.url).searchParams; + const body = c.body ? JSON.parse(c.body) : {}; + let m: RegExpMatchArray | null; + if ((m = p.match(/^\/apis\/apps\/v1\/namespaces\/([^/]+)\/deployments\/([^/]+)(\/scale)?$/))) { + const d = deployments.get(`${m[1]}/${m[2]}`); + if (!d) return status(404, "NotFound", `deployments.apps "${m[2]}" not found`); + if (c.method === "GET" && m[3]) return { json: { kind: "Scale", spec: { replicas: d.replicas } } }; + if (c.method === "GET") { + return { + json: { + metadata: { generation: 4 }, + spec: { replicas: d.replicas, template: { spec: { containers: [{ image: "web:1.0" }] } } }, + status: { readyReplicas: d.replicas, updatedReplicas: d.replicas, availableReplicas: d.replicas }, + }, + }; + } + if (c.method === "PATCH") { + if (c.headers["content-type"] !== "application/merge-patch+json") return status(415, "UnsupportedMediaType", "bad patch type"); + if (m[3]) d.replicas = body.spec.replicas; + else if (body.spec?.template?.metadata?.annotations?.["kubectl.kubernetes.io/restartedAt"]) d.restarts++; + return { json: {} }; + } + } + if (c.method === "POST" && (m = p.match(/^\/api\/v1\/namespaces\/([^/]+)\/configmaps$/))) { + const key = `${m[1]}/${body.metadata.name}`; + if (configmaps.has(key)) return status(409, "AlreadyExists", `configmaps "${body.metadata.name}" already exists`); + const uid = `00000000-0000-4000-8000-${String(++n).padStart(12, "0")}`; + configmaps.set(key, { uid, data: body.data }); + return { status: 201, json: { metadata: { name: body.metadata.name, uid, resourceVersion: "1" } } }; + } + if (c.method === "DELETE" && (m = p.match(/^\/api\/v1\/namespaces\/([^/]+)\/configmaps\/([^/]+)$/))) { + const cm = configmaps.get(`${m[1]}/${m[2]}`); + if (!cm) return status(404, "NotFound", "not found"); + if (body.preconditions?.uid && body.preconditions.uid !== cm.uid) return status(409, "Conflict", "Precondition failed: UID in precondition does not match"); + configmaps.delete(`${m[1]}/${m[2]}`); + return { json: { kind: "Status", status: "Success" } }; + } + if (c.method === "DELETE" && (m = p.match(/^\/api\/v1\/namespaces\/([^/]+)\/pods\/([^/]+)$/))) { + return pods.delete(`${m[1]}/${m[2]}`) ? { json: { kind: "Pod" } } : status(404, "NotFound", "pod not found"); + } + if (c.method === "GET" && (m = p.match(/^\/api\/v1\/namespaces\/([^/]+)\/pods$/))) { + const sel = q.get("labelSelector"); + const items = [...pods] + .filter(([k]) => k.startsWith(`${m![1]}/`)) + .filter(([, v]) => !sel || sel.split(",").every((kv) => v.labels[kv.split("=")[0]!] === kv.split("=")[1])) + .map(([k, v]) => ({ + metadata: { name: k.split("/")[1] }, + spec: { nodeName: "node-1" }, + status: { phase: v.phase, containerStatuses: [{ ready: true, restartCount: 2 }] }, + })); + return { json: { items, metadata: {} } }; + } + return status(500, "InternalError", `unexpected ${c.method} ${c.url}`); + }, + }); + return Object.assign(b, { deployments, configmaps, pods }); +} +export type FakeKube = ReturnType; +const k = (b: Backend) => b as FakeKube; +const seedDeploy = (b: Backend) => void k(b).deployments.set("prod/web", { replicas: 3, restarts: 0 }); +const seedPods = (b: Backend) => { + k(b).pods.set("prod/web-1", { labels: { app: "web" }, phase: "Running" }); + k(b).pods.set("prod/web-2", { labels: { app: "web" }, phase: "Running" }); + k(b).pods.set("prod/db-1", { labels: { app: "db" }, phase: "Running" }); +}; + +export const subject: Subject = { + name: "kubernetes", + secrets: [TOKEN], + make: (o) => { + const backend = fakeKube(); + return { backend, connector: createKubernetesConnector({ apiServer: SERVER, token: TOKEN, fetch: backend.fetch, timeoutMs: o?.timeoutMs }) }; + }, + faultBody: (code, secret) => status(code, "Forbidden", `forbidden for token ${secret}`), +}; + +export const scenarios: Scenario[] = [ + { tool: "get_deployment", args: { namespace: "prod", name: "web" }, setup: seedDeploy, expect: (r) => expect(r).toMatchObject({ replicas: 3, ready_replicas: 3, images: ["web:1.0"] }) }, + { tool: "list_pods", args: { namespace: "prod", label_selector: "app=web", limit: 50 }, setup: seedPods, expect: (r) => expect((r as { pods: Array<{ name: string }> }).pods.map((p) => p.name)).toEqual(["web-1", "web-2"]) }, + { tool: "scale_deployment", args: { namespace: "prod", name: "web", replicas: 0 }, setup: seedDeploy, expect: (r) => expect(r).toMatchObject({ previous_replicas: 3, replicas: 0 }) }, + { tool: "restore_replicas", args: { namespace: "prod", name: "web", replicas: 3 }, setup: (b) => void k(b).deployments.set("prod/web", { replicas: 0, restarts: 0 }) }, + { tool: "create_configmap", args: { namespace: "prod", name: "settings", data: { "app.properties": "a=b" } }, expect: (r) => expect(r).toMatchObject({ uid: "00000000-0000-4000-8000-000000000001" }) }, + { tool: "delete_configmap", args: { namespace: "prod", name: "settings", uid: SEED_UID }, setup: (b) => void k(b).configmaps.set("prod/settings", { uid: SEED_UID, data: {} }) }, + { tool: "rollout_restart", args: { namespace: "prod", name: "web" }, setup: seedDeploy }, + { tool: "delete_pod", args: { namespace: "prod", name: "web-1", grace_period_seconds: 5 }, setup: seedPods }, +]; diff --git a/packages/connectors/test/fixtures/linear.ts b/packages/connectors/test/fixtures/linear.ts new file mode 100644 index 0000000..3650de7 --- /dev/null +++ b/packages/connectors/test/fixtures/linear.ts @@ -0,0 +1,83 @@ +import { expect } from "vitest"; +import { createLinearConnector } from "../../src/index.js"; +import { httpBackend, type Backend, type Scenario, type Subject } from "../certification.js"; + +export const KEY = "lin_api_SECRETLINEARKEY0123456789"; +export const TEAM = "cccccccc-0000-4000-8000-000000000001"; +export const ISSUE = "aaaaaaaa-0000-4000-8000-000000000001"; +export const COMMENT = "dddddddd-0000-4000-8000-000000000001"; + +export function fakeLinear() { + const issues = new Map(); + const comments = new Set(); + let n = 0; + const gen = (prefix: string) => `${prefix}-0000-4000-8000-${String(++n).padStart(12, "0")}`; + const notFound = { json: { errors: [{ message: "Entity not found: Issue" }] } }; + const b = httpBackend({ + state: () => ({ issues: [...issues], comments: [...comments].sort() }), + isMutating: (c) => (JSON.parse(c.body || "{}").query ?? "").trim().startsWith("mutation"), + handle: (c) => { + const { query, variables } = JSON.parse(c.body) as { query: string; variables: Record }; + if (query.includes("issueCreate")) { + const id = gen("bbbbbbbb"); + issues.set(id, { title: variables.input.title, archived: false }); + return { json: { data: { issueCreate: { success: true, issue: { id, identifier: "ENG-1", url: "https://linear.app/x/ENG-1", title: variables.input.title } } } } }; + } + if (query.includes("issueArchive")) { + const i = issues.get(variables.id); + if (!i) return notFound; + i.archived = true; + return { json: { data: { issueArchive: { success: true } } } }; + } + if (query.includes("commentCreate")) { + if (!issues.has(variables.input.issueId)) return notFound; + const id = gen("eeeeeeee"); + comments.add(id); + return { json: { data: { commentCreate: { success: true, comment: { id, url: "https://linear.app/c" } } } } }; + } + if (query.includes("commentDelete")) { + if (!comments.delete(variables.id)) return { json: { errors: [{ message: "Entity not found: Comment" }] } }; + return { json: { data: { commentDelete: { success: true } } } }; + } + if (query.includes("issue(id")) { + const i = issues.get(variables.id); + if (!i) return notFound; + return { + json: { + data: { + issue: { id: variables.id, identifier: "ENG-1", title: i.title, description: null, url: "https://linear.app/x/ENG-1", archivedAt: i.archived ? "2026-01-01T00:00:00Z" : null, state: { name: "Todo" } }, + }, + }, + }; + } + return { status: 400, json: { errors: [{ message: "unexpected query" }] } }; + }, + }); + return Object.assign(b, { issues, comments }); +} +type FakeLinear = ReturnType; +const ln = (b: Backend) => b as FakeLinear; +const seedIssue = (b: Backend) => void ln(b).issues.set(ISSUE, { title: "seed", archived: false }); + +export const subject: Subject = { + name: "linear", + secrets: [KEY], + make: (o) => { + const backend = fakeLinear(); + return { backend, connector: createLinearConnector({ apiKey: KEY, fetch: backend.fetch, timeoutMs: o?.timeoutMs }) }; + }, + faultBody: (_s, secret) => ({ json: { errors: [{ message: `rejected ${secret}` }] } }), +}; + +export const scenarios: Scenario[] = [ + { tool: "get_issue", args: { id: ISSUE }, setup: seedIssue, expect: (r) => expect(r).toMatchObject({ title: "seed", archived: false }) }, + { + tool: "create_issue", + args: { team_id: TEAM, title: "Bug", priority: 2 }, + // an archived issue still exists in Linear; it is hidden, not erased + restored: (_b, after) => expect((after as { issues: Array<[string, { archived: boolean }]> }).issues.every(([, i]) => i.archived)).toBe(true), + }, + { tool: "archive_issue", args: { issue_id: ISSUE }, setup: seedIssue }, + { tool: "add_comment", args: { issue_id: ISSUE, body: "note" }, setup: seedIssue }, + { tool: "delete_comment", args: { comment_id: COMMENT }, setup: (b) => void ln(b).comments.add(COMMENT) }, +]; diff --git a/packages/connectors/test/fixtures/postgres.ts b/packages/connectors/test/fixtures/postgres.ts new file mode 100644 index 0000000..2d579e6 --- /dev/null +++ b/packages/connectors/test/fixtures/postgres.ts @@ -0,0 +1,164 @@ +import { expect } from "vitest"; +import { createPostgresConnector, type PgClientLike, type PgLike, type PgResult } from "../../src/index.js"; +import type { Backend, Fault, Scenario, Subject } from "../certification.js"; +import type { Call } from "../helpers.js"; + +export const URL_SECRET = "postgres://app:SECRETPGPASSWORD@db.internal:5432/prod"; +export const PASSWORD = "SECRETPGPASSWORD"; + +type Row = Record; + +class PgError extends Error { + constructor( + message: string, + readonly code: string, + ) { + super(message); + } +} + +const STATUS_CODE: Record = { 401: "28P01", 403: "42501", 422: "22P02", 500: "XX000", 503: "57P03" }; + +/** An in-memory Postgres that understands exactly the SQL shapes the connector emits. Every statement is logged. */ +export function fakePg() { + const tables = new Map([["users", []]]); + const seq = new Map(); + let txn: { readOnly: boolean; snapshot: string } | null = null; + const calls: Call[] = []; + const snapshot = () => JSON.stringify([...tables]); + const restore = (s: string) => { + tables.clear(); + for (const [k, v] of JSON.parse(s) as Array<[string, Row[]]>) tables.set(k, v); + }; + const isMutating = (c: Call) => /^(INSERT|UPDATE|DELETE|COMMIT)/.test(c.url); + + const b: Backend & { tables: typeof tables; pg: PgLike; withConnect: PgLike; released: Array } = { + fault: null as Fault | null, + calls, + supportsFaults: true, + isMutating, + state: () => ({ users: tables.get("users") }), + tables, + released: [], + pg: undefined as unknown as PgLike, + withConnect: undefined as unknown as PgLike, + }; + + const rowsOf = (t: string): Row[] => { + const r = tables.get(t); + if (!r) throw new PgError(`relation "${t}" does not exist`, "42P01"); + return r; + }; + const writable = () => { + if (txn?.readOnly) throw new PgError("cannot execute a write in a read-only transaction", "25006"); + }; + + async function query(text: string, values: unknown[] = []): Promise { + const call: Call = { url: text, method: "SQL", headers: {}, body: JSON.stringify(values) }; + calls.push(call); + text = text.replace(/"\w+"\.("\w+")/g, "$1"); // schema-qualified names are matched as plain table names + const f = b.fault; + if (f && (f.on !== "mutating" || isMutating(call))) { + if (f.kind === "throw") throw new Error(f.message); + if (f.kind === "hang") return new Promise(() => undefined); + throw new PgError(`fault ${f.status}`, STATUS_CODE[f.status] ?? "XX000"); + } + let m: RegExpMatchArray | null; + if (text === "BEGIN" || text === "BEGIN READ ONLY") { + if (txn) restore(txn.snapshot); // the previous session died mid-transaction + txn = { readOnly: text !== "BEGIN", snapshot: snapshot() }; + return { rows: [], rowCount: null }; + } + if (text === "ROLLBACK") { + if (txn) restore(txn.snapshot); + txn = null; + return { rows: [], rowCount: null }; + } + if (text === "COMMIT") { + txn = null; + return { rows: [], rowCount: null }; + } + if ((m = text.match(/^INSERT INTO "(\w+)" \(([^)]*)\) VALUES \(([^)]*)\) RETURNING "(\w+)"$/))) { + writable(); + const cols = m[2]!.split(",").map((c) => c.replace(/"/g, "")); + const n = (seq.get(m[1]!) ?? 0) + 1; + seq.set(m[1]!, n); + const row: Row = { id: n }; + cols.forEach((c, i) => (row[c] = values[i])); + rowsOf(m[1]!).push(row); + return { rows: [{ [m[4]!]: row[m[4]!] }], rowCount: 1 }; + } + if ((m = text.match(/^DELETE FROM "(\w+)" WHERE "(\w+)" = \$1$/))) { + writable(); + const rows = rowsOf(m[1]!); + const keep = rows.filter((r) => r[m![2]!] !== values[0]); + const n = rows.length - keep.length; + tables.set(m[1]!, keep); + return { rows: [], rowCount: n }; + } + if ((m = text.match(/^SELECT ((?:"\w+",?)+) FROM "(\w+)" WHERE "(\w+)" = \$1 FOR UPDATE$/))) { + const cols = m[1]!.split(",").map((c) => c.replace(/"/g, "")); + const hit = rowsOf(m[2]!).filter((r) => r[m![3]!] === values[0]); + return { rows: hit.map((r) => Object.fromEntries(cols.map((c) => [c, r[c] ?? null]))), rowCount: hit.length }; + } + if ((m = text.match(/^UPDATE "(\w+)" SET (.+) WHERE "(\w+)" = \$1$/))) { + writable(); + const sets = [...m[2]!.matchAll(/"(\w+)" = \$(\d+)/g)].map((x) => [x[1]!, Number(x[2])] as const); + const hit = rowsOf(m[1]!).filter((r) => r[m![3]!] === values[0]); + for (const r of hit) for (const [c, i] of sets) r[c] = values[i - 1]; + return { rows: [], rowCount: hit.length }; + } + if ((m = text.match(/^(?:SELECT|WITH) /i))) { + if (/\b(DELETE FROM|INSERT INTO|UPDATE )/i.test(text)) writable(); + const rows = rowsOf("users").filter((r) => (values.length ? Number(r.age) > Number(values[0]) : true)); + return { rows: rows.map((r) => ({ ...r })), rowCount: rows.length }; + } + throw new PgError(`syntax error near "${text.slice(0, 20)}"`, "42601"); + } + + b.pg = { query }; + b.withConnect = { + query, + connect: async () => ({ query, release: (d?: boolean | Error) => void b.released.push(d) }) as PgClientLike, + }; + return b; +} +export type FakePg = ReturnType; +const pg = (b: Backend) => b as FakePg; +const seedUser = (b: Backend) => void pg(b).tables.get("users")!.push({ id: 7, name: "Ann", age: 30, data: null }); + +export const subject: Subject = { + name: "postgres", + secrets: [URL_SECRET, PASSWORD], + make: (o) => { + const backend = fakePg(); + return { + backend, + connector: createPostgresConnector({ pool: backend.pg, secrets: [URL_SECRET, PASSWORD], timeoutMs: o?.timeoutMs }), + }; + }, + statusPattern: (s) => new RegExp(STATUS_CODE[s] ?? String(s)), + faultBody: undefined, +}; + +export const scenarios: Scenario[] = [ + { + tool: "query", + args: { text: "SELECT * FROM users WHERE age > $1", values: [20] }, + setup: seedUser, + expect: (r) => expect(r).toMatchObject({ rows: [{ id: 7, name: "Ann" }], row_count: 1, truncated: false }), + }, + { tool: "insert_row", args: { table: "users", values: { name: "Bo", age: 41 } }, expect: (r) => expect(r).toMatchObject({ table: "users", schema: null, key_column: "id", key: 1 }) }, + { tool: "delete_row", args: { table: "users", key: 7 }, setup: seedUser }, + { + tool: "update_row", + args: { table: "users", key: 7, set: { name: "Zed", age: 31 } }, + setup: seedUser, + expect: (r) => expect(r).toMatchObject({ key: 7, previous: { name: "Ann", age: 30 } }), + }, + { + tool: "restore_row", + args: { table: "users", key: 7, values: { name: "Ann", age: 30 } }, + setup: (b) => void pg(b).tables.get("users")!.push({ id: 7, name: "Zed", age: 31, data: null }), + }, +]; diff --git a/packages/connectors/test/fixtures/slack.ts b/packages/connectors/test/fixtures/slack.ts new file mode 100644 index 0000000..8973660 --- /dev/null +++ b/packages/connectors/test/fixtures/slack.ts @@ -0,0 +1,65 @@ +import { expect } from "vitest"; +import { createSlackConnector } from "../../src/index.js"; +import { httpBackend, type Backend, type Scenario, type Subject } from "../certification.js"; + +export const TOKEN = "xoxb-SECRET-SLACK-TOKEN"; +export const CH = "C0123ABCDE"; +const TS = "1700000000.000001"; + +export function fakeSlack() { + const messages = new Map>(); + let n = 1; + const b = httpBackend({ + state: () => ({ messages: [...messages].map(([ts, r]) => [ts, [...r].sort()]) }), + handle: (c) => { + const method = new URL(c.url).pathname.replace("/api/", ""); + const body = c.body ? JSON.parse(c.body) : {}; + switch (method) { + case "chat.postMessage": { + const ts = `1700000000.${String(++n).padStart(6, "0")}`; + messages.set(ts, new Set()); + return { json: { ok: true, channel: body.channel, ts } }; + } + case "chat.delete": + return messages.delete(body.ts) ? { json: { ok: true } } : { json: { ok: false, error: "message_not_found" } }; + case "reactions.add": { + const m = messages.get(body.timestamp); + if (!m) return { json: { ok: false, error: "message_not_found" } }; + if (m.has(body.name)) return { json: { ok: false, error: "already_reacted" } }; + m.add(body.name); + return { json: { ok: true } }; + } + case "reactions.remove": { + const m = messages.get(body.timestamp); + if (!m) return { json: { ok: false, error: "message_not_found" } }; + return m.delete(body.name) ? { json: { ok: true } } : { json: { ok: false, error: "no_reaction" } }; + } + case "conversations.list": + return { json: { ok: true, channels: [{ id: CH, name: "general", is_private: false }], response_metadata: { next_cursor: "" } } }; + } + return { status: 404 }; + }, + }); + return Object.assign(b, { messages }); +} +type FakeSlack = ReturnType; +const sl = (b: Backend) => b as FakeSlack; +const seed = (b: Backend) => void sl(b).messages.set(TS, new Set()); + +export const subject: Subject = { + name: "slack", + secrets: [TOKEN], + make: (o) => { + const backend = fakeSlack(); + return { backend, connector: createSlackConnector({ token: TOKEN, fetch: backend.fetch, timeoutMs: o?.timeoutMs }) }; + }, + faultBody: (_s, secret) => ({ json: { ok: false, error: `invalid_auth ${secret}` } }), +}; + +export const scenarios: Scenario[] = [ + { tool: "list_channels", args: { limit: 10 }, expect: (r) => expect(r).toMatchObject({ channels: [{ id: CH }] }) }, + { tool: "post_message", args: { channel: CH, text: "hi" }, expect: (r) => expect(r).toMatchObject({ channel: CH, ts: "1700000000.000002" }) }, + { tool: "delete_message", args: { channel: CH, ts: TS }, setup: seed }, + { tool: "add_reaction", args: { channel: CH, timestamp: TS, name: "eyes" }, setup: seed }, + { tool: "remove_reaction", args: { channel: CH, timestamp: TS, name: "eyes" }, setup: (b) => void sl(b).messages.set(TS, new Set(["eyes"])) }, +]; diff --git a/packages/connectors/test/fixtures/stripe.ts b/packages/connectors/test/fixtures/stripe.ts new file mode 100644 index 0000000..ae0fb1a --- /dev/null +++ b/packages/connectors/test/fixtures/stripe.ts @@ -0,0 +1,86 @@ +import { expect } from "vitest"; +import { createStripeConnector } from "../../src/index.js"; +import { httpBackend, type Backend, type Scenario, type Subject } from "../certification.js"; +import type { Call, FakeReply } from "../helpers.js"; + +export const KEY = "sk_test_SUPERSECRETKEY999"; + +/** A tiny Stripe, including the Idempotency-Key replay behaviour that makes retried writes safe. */ +export function fakeStripe() { + const customers = new Set(); + const intents = new Map(); + const refunds: string[] = []; + const replay = new Map(); + let n = 0; + const route = (c: Call): FakeReply => { + const p = new URL(c.url).pathname; + const form = new URLSearchParams(c.body); + let m: RegExpMatchArray | null; + if (c.method === "POST" && p === "/v1/customers") { + const id = `cus_${++n}`; + customers.add(id); + return { json: { id, email: form.get("email"), name: form.get("name") } }; + } + if (c.method === "DELETE" && (m = p.match(/^\/v1\/customers\/(.+)$/))) { + return customers.delete(m[1]!) ? { json: { id: m[1], deleted: true } } : { status: 404, json: { error: { code: "resource_missing", message: "No such customer" } } }; + } + if (c.method === "POST" && p === "/v1/payment_intents") { + const id = `pi_${++n}`; + intents.set(id, "requires_payment_method"); + return { json: { id, status: "requires_payment_method", amount: Number(form.get("amount")), currency: form.get("currency") } }; + } + if (c.method === "POST" && (m = p.match(/^\/v1\/payment_intents\/(.+)\/cancel$/))) { + const st = intents.get(m[1]!); + if (st === undefined) return { status: 404, json: { error: { code: "resource_missing" } } }; + if (st === "canceled" || st === "succeeded") return { status: 400, json: { error: { code: "payment_intent_unexpected_state", message: "bad state" } } }; + intents.set(m[1]!, "canceled"); + return { json: { id: m[1], status: "canceled" } }; + } + if (c.method === "GET" && (m = p.match(/^\/v1\/payment_intents\/(.+)$/))) return { json: { id: m[1], status: intents.get(m[1]!) } }; + if (c.method === "POST" && p === "/v1/refunds") { + refunds.push(`re_${++n}`); + return { json: { id: refunds.at(-1), status: "succeeded", amount: Number(form.get("amount") ?? 1000), currency: "usd" } }; + } + if (c.method === "GET" && (m = p.match(/^\/v1\/charges\/(.+)$/))) { + return { json: { id: m[1], amount: 1000, amount_refunded: 0, currency: "usd", status: "succeeded", paid: true, refunded: false } }; + } + return { status: 500, json: { error: { message: "unexpected" } } }; + }; + const b = httpBackend({ + state: () => ({ customers: [...customers].sort(), intents: [...intents], refunds }), + handle: (c) => { + const key = c.headers["idempotency-key"]; + if (c.method === "POST" && key && replay.has(key)) return replay.get(key)!; + const r = route(c); + if (c.method === "POST" && key && (r.status ?? 200) < 500) replay.set(key, r); + return r; + }, + }); + return Object.assign(b, { customers, intents, refunds }); +} +type FakeStripe = ReturnType; +const st = (b: Backend) => b as FakeStripe; +const hasKey = (c: Call, key: string) => c.headers["idempotency-key"] === key; + +export const subject: Subject = { + name: "stripe", + secrets: [KEY], + make: (o) => { + const backend = fakeStripe(); + return { backend, connector: createStripeConnector({ secretKey: KEY, fetch: backend.fetch, timeoutMs: o?.timeoutMs }) }; + }, + faultBody: (_s, secret) => ({ json: { error: { message: `Invalid API Key provided: ${secret}` } } }), +}; + +export const scenarios: Scenario[] = [ + { tool: "retrieve_charge", args: { charge_id: "ch_123" }, expect: (r) => expect(r).toMatchObject({ amount: 1000, paid: true }) }, + { tool: "create_customer", args: { email: "a@example.com", name: "Ann", metadata: { plan: "pro" } }, usesKey: hasKey, expect: (r) => expect(r).toMatchObject({ customer_id: "cus_1" }) }, + { tool: "delete_customer", args: { customer_id: "cus_7" }, setup: (b) => void st(b).customers.add("cus_7") }, + { tool: "create_payment_intent", args: { amount: 1500, currency: "EUR" }, usesKey: hasKey, + expect: (r) => expect(r).toMatchObject({ payment_intent_id: "pi_1", currency: "eur" }), + // a canceled intent stays on the account, in its final state + restored: (_b, after) => expect((after as { intents: Array<[string, string]> }).intents.map((i) => i[1])).toEqual(["canceled"]), + }, + { tool: "cancel_payment_intent", args: { payment_intent_id: "pi_7" }, usesKey: hasKey, setup: (b) => void st(b).intents.set("pi_7", "requires_payment_method") }, + { tool: "create_refund", args: { charge: "ch_123", amount: 500 }, usesKey: hasKey, expect: (r) => expect(r).toMatchObject({ status: "succeeded" }) }, +]; diff --git a/packages/connectors/test/fixtures/teams.ts b/packages/connectors/test/fixtures/teams.ts new file mode 100644 index 0000000..204177d --- /dev/null +++ b/packages/connectors/test/fixtures/teams.ts @@ -0,0 +1,64 @@ +import { expect } from "vitest"; +import { createTeamsConnector } from "../../src/index.js"; +import { httpBackend, type Backend, type Scenario, type Subject } from "../certification.js"; + +export const TOKEN = "eyJ0eXAi.TEAMS-SECRET-GRAPH-TOKEN.sig"; +export const TEAM = "11111111-2222-3333-4444-555555555555"; +export const CHANNEL = "19:abc123@thread.tacv2"; + +export function fakeTeams() { + const messages = new Map(); + let n = 0; + const b = httpBackend({ + state: () => ({ messages: [...messages] }), + handle: (c) => { + const p = decodeURIComponent(new URL(c.url).pathname).replace("/v1.0", ""); + const base = `/teams/${TEAM}/channels`; + if (c.method === "GET" && p === base) { + return { json: { value: [{ id: CHANNEL, displayName: "General", membershipType: "standard" }], "@odata.nextLink": null } }; + } + if (c.method === "POST" && p === `${base}/${CHANNEL}/messages`) { + const id = String(1700000000000 + ++n); + messages.set(id, { content: JSON.parse(c.body).body.content, deleted: false }); + return { status: 201, json: { id, webUrl: `https://teams.example/${id}` } }; + } + const m = p.match(new RegExp(`^${base}/${CHANNEL}/messages/(\\d+)/softDelete$`)); + if (c.method === "POST" && m) { + const msg = messages.get(m[1]!); + if (!msg || msg.deleted) return { status: 404, json: { error: { code: "NotFound", message: "Message not found" } } }; + msg.deleted = true; + return { status: 204 }; + } + return { status: 500, json: { error: { message: `unexpected ${c.method} ${c.url}` } } }; + }, + }); + return Object.assign(b, { messages }); +} +type FakeTeams = ReturnType; +const tm = (b: Backend) => b as FakeTeams; + +export const subject: Subject = { + name: "teams", + secrets: [TOKEN], + make: (o) => { + const backend = fakeTeams(); + return { backend, connector: createTeamsConnector({ token: TOKEN, fetch: backend.fetch, timeoutMs: o?.timeoutMs }) }; + }, + faultBody: (_status, secret) => ({ json: { error: { code: "Forbidden", message: `denied ${secret}` } } }), +}; + +export const scenarios: Scenario[] = [ + { tool: "list_channels", args: { team_id: TEAM }, expect: (r) => expect(r).toMatchObject({ channels: [{ id: CHANNEL, display_name: "General" }] }) }, + { + tool: "post_message", + args: { team_id: TEAM, channel_id: CHANNEL, content: "hello" }, + expect: (r) => expect(r).toMatchObject({ message_id: "1700000000001" }), + // a soft-deleted message stays in the channel as a placeholder + restored: (_b, after) => expect((after as { messages: Array<[string, { deleted: boolean }]> }).messages.every(([, m]) => m.deleted)).toBe(true), + }, + { + tool: "delete_message", + args: { team_id: TEAM, channel_id: CHANNEL, message_id: "1700000000099" }, + setup: (b) => void tm(b).messages.set("1700000000099", { content: "x", deleted: false }), + }, +]; diff --git a/packages/connectors/test/kubernetes.test.ts b/packages/connectors/test/kubernetes.test.ts new file mode 100644 index 0000000..e6caccc --- /dev/null +++ b/packages/connectors/test/kubernetes.test.ts @@ -0,0 +1,198 @@ +import { mkdtempSync, writeFileSync } from "node:fs"; +import http from "node:http"; +import type { AddressInfo } from "node:net"; +import { tmpdir } from "node:os"; +import path from "node:path"; +import { Registry } from "@multimindlab/reversi"; +import { describe, expect, it } from "vitest"; +import { caFetch, compensatorFor, createKubernetesConnector } from "../src/index.js"; +import { SEED_UID, SERVER, TOKEN, fakeKube } from "./fixtures/kubernetes.js"; +import { engine } from "./helpers.js"; + +const make = () => { + const b = fakeKube(); + const c = createKubernetesConnector({ apiServer: SERVER, token: TOKEN, fetch: b.fetch }); + return { b, c, t: (n: string, a: Record) => c.tools[`kubernetes.${n}`]!(a, {}) }; +}; + +describe("kubernetes manifests", () => { + it("validate; the two destructive tools are irreversible and always need approval", () => { + const { c } = make(); + const reg = Registry.from(c.manifests); + expect(reg.size).toBe(8); + for (const tool of ["kubernetes.rollout_restart", "kubernetes.delete_pod"]) { + expect(reg.get(tool)).toMatchObject({ effect: "irreversible", approval: "always" }); + expect(reg.get(tool)?.compensation).toBeUndefined(); + } + expect(reg.get("kubernetes.scale_deployment")?.compensation?.tool).toBe("kubernetes.restore_replicas"); + expect(reg.get("kubernetes.create_configmap")?.compensation).toMatchObject({ + tool: "kubernetes.delete_configmap", + argument_map: { namespace: "$args.namespace", name: "$args.name", uid: "$result.uid" }, + }); + expect(reg.get("kubernetes.get_deployment")?.effect).toBe("read_only"); + expect(reg.get("kubernetes.list_pods")?.effect).toBe("read_only"); + }); +}); + +describe("kubernetes tools", () => { + it("scale reads the current replicas through the scale subresource, then merge-patches", async () => { + const { b, t } = make(); + b.deployments.set("prod/web", { replicas: 3, restarts: 0 }); + const r = await t("scale_deployment", { namespace: "prod", name: "web", replicas: 5 }); + expect(r).toEqual({ namespace: "prod", name: "web", previous_replicas: 3, replicas: 5 }); + expect(b.calls.map((c) => `${c.method} ${new URL(c.url).pathname}`)).toEqual([ + "GET /apis/apps/v1/namespaces/prod/deployments/web/scale", + "PATCH /apis/apps/v1/namespaces/prod/deployments/web/scale", + ]); + expect(b.calls[1]!.headers["content-type"]).toBe("application/merge-patch+json"); + expect(JSON.parse(b.calls[1]!.body)).toEqual({ spec: { replicas: 5 } }); + expect(b.calls[0]!.headers.authorization).toBe(`Bearer ${TOKEN}`); + }); + + it("scaling to zero and back through a real engine", async () => { + const { b, c } = make(); + b.deployments.set("prod/web", { replicas: 3, restarts: 0 }); + const { call, run } = engine([c]); + await call("kubernetes.scale_deployment", { namespace: "prod", name: "web", replicas: 0 }); + expect(b.deployments.get("prod/web")!.replicas).toBe(0); + const report = await run.rollback({ compensate: compensatorFor([c]) }); + expect(report.complete).toBe(true); + expect(b.deployments.get("prod/web")!.replicas).toBe(3); + }); + + it("deletes a created ConfigMap only if its UID still matches", async () => { + const { b, c, t } = make(); + const { call, run } = engine([c]); + const made = await call("kubernetes.create_configmap", { namespace: "prod", name: "settings", data: { a: "b" } }); + const uid = (made as { result: { uid: string } }).result.uid; + // someone deletes it and creates a different one with the same name + b.configmaps.set("prod/settings", { uid: SEED_UID, data: { theirs: "1" } }); + const report = await run.rollback({ compensate: compensatorFor([c]) }); + expect(report.complete).toBe(true); + expect(b.configmaps.get("prod/settings")).toEqual({ uid: SEED_UID, data: { theirs: "1" } }); + const del = b.calls.at(-1)!; + expect(del.method).toBe("DELETE"); + expect(JSON.parse(del.body)).toEqual({ apiVersion: "v1", kind: "DeleteOptions", preconditions: { uid } }); + expect(await t("delete_configmap", { namespace: "prod", name: "settings", uid: SEED_UID })).toMatchObject({ deleted: true }); + }); + + it("create_configmap refuses to overwrite an existing one", async () => { + const { b, t } = make(); + b.configmaps.set("prod/settings", { uid: SEED_UID, data: {} }); + const err = await t("create_configmap", { namespace: "prod", name: "settings", data: { a: "b" } }).catch((e) => e); + expect(err.message).toContain("status 409"); + expect(err.message).toContain("already exists"); + }); + + it("rollout_restart patches the pod template annotation; delete_pod can pass a grace period", async () => { + const { b, t } = make(); + b.deployments.set("prod/web", { replicas: 3, restarts: 0 }); + b.pods.set("prod/web-1", { labels: {}, phase: "Running" }); + const r = (await t("rollout_restart", { namespace: "prod", name: "web" })) as { restarted_at: string }; + expect(JSON.parse(b.calls[0]!.body)).toEqual({ spec: { template: { metadata: { annotations: { "kubectl.kubernetes.io/restartedAt": r.restarted_at } } } } }); + await t("delete_pod", { namespace: "prod", name: "web-1", grace_period_seconds: 0 }); + expect(JSON.parse(b.calls[1]!.body)).toEqual({ apiVersion: "v1", kind: "DeleteOptions", gracePeriodSeconds: 0 }); + await expect(t("delete_pod", { namespace: "prod", name: "web-1" })).rejects.toThrow("status 404"); + }); + + it("rollback reports an irreversible step as unrecoverable and never complete", async () => { + const { b, c } = make(); + b.deployments.set("prod/web", { replicas: 3, restarts: 0 }); + const { call, run } = engine([c]); // the test engine approves everything + await call("kubernetes.rollout_restart", { namespace: "prod", name: "web" }); + const report = await run.rollback({ compensate: compensatorFor([c]) }); + expect(report.unrecoverable).toHaveLength(1); + expect(report.complete).toBe(false); + }); + + it("lists pods by label selector", async () => { + const { b, t } = make(); + b.pods.set("prod/web-1", { labels: { app: "web" }, phase: "Running" }); + b.pods.set("prod/db-1", { labels: { app: "db" }, phase: "Pending" }); + const r = (await t("list_pods", { namespace: "prod", label_selector: "app=db" })) as { pods: Array<{ name: string; phase: string; restarts: number }> }; + expect(r.pods).toEqual([{ name: "db-1", phase: "Pending", node_name: "node-1", ready: true, restarts: 2 }]); + expect(b.calls[0]!.url).toContain("labelSelector=app%3Ddb"); + }); + + it("validates namespaces and names as DNS-1123 before any request", async () => { + const { b, t } = make(); + for (const bad of ["Prod", "pr_od", "-prod", "prod-", "a/b", "..", "", "x".repeat(64), "pro d", "prod%2Fother"]) { + await expect(t("get_deployment", { namespace: bad, name: "web" })).rejects.toThrow("namespace"); + } + for (const bad of ["Web", "we_b", ".web", "web.", "a/b", "../x", "x".repeat(254)]) { + await expect(t("get_deployment", { namespace: "prod", name: bad })).rejects.toThrow("name"); + } + await expect(t("scale_deployment", { namespace: "prod", name: "web", replicas: -1 })).rejects.toThrow("non-negative"); + await expect(t("scale_deployment", { namespace: "prod", name: "web", replicas: 1.5 })).rejects.toThrow("non-negative"); + await expect(t("scale_deployment", { namespace: "prod", name: "web", replicas: 10001 })).rejects.toThrow("at most"); + await expect(t("create_configmap", { namespace: "prod", name: "x", data: { "bad key!": "v" } })).rejects.toThrow("invalid key"); + await expect(t("create_configmap", { namespace: "prod", name: "x", data: { k: 1 } })).rejects.toThrow("must be a string"); + await expect(t("delete_configmap", { namespace: "prod", name: "x", uid: "not a uid!" })).rejects.toThrow("uid"); + await expect(t("list_pods", { namespace: "prod", label_selector: "app=$(rm -rf)" })).rejects.toThrow("label selector"); + expect(b.calls).toHaveLength(0); + await t("get_deployment", { namespace: "a", name: "a.b-c" }).catch(() => undefined); + expect(b.calls[0]!.url).toBe(`${SERVER}/apis/apps/v1/namespaces/a/deployments/a.b-c`); + }); + + it("maps Kubernetes Status errors to HTTP errors without the token", async () => { + const { t } = make(); + const err = await t("get_deployment", { namespace: "prod", name: "missing" }).catch((e) => e); + expect(err.status).toBe(404); + expect(err.message).toContain("not found"); + const c = createKubernetesConnector({ apiServer: SERVER, token: TOKEN, fetch: (async () => new Response(JSON.stringify({ kind: "Status", message: `forbidden: ${TOKEN}`, code: 403 }), { status: 403 })) as typeof fetch }); + const e2 = await c.tools["kubernetes.get_deployment"]!({ namespace: "prod", name: "web" }, {}).catch((e) => e); + expect(e2.message).toContain("status 403"); + expect(e2.message).not.toContain(TOKEN); + }); +}); + +describe("kubernetes transport with a private CA", () => { + it("caFetch speaks plain http/https requests and maps responses, headers, bodies and 204s", async () => { + const seen: Array<{ method: string; url: string; auth: string | undefined; body: string }> = []; + const server = http.createServer((req, res) => { + const chunks: Buffer[] = []; + req.on("data", (c: Buffer) => chunks.push(c)); + req.on("end", () => { + seen.push({ method: req.method!, url: req.url!, auth: req.headers.authorization, body: Buffer.concat(chunks).toString() }); + if (req.url!.endsWith("/gone")) return void res.writeHead(204).end(); + if (req.url!.endsWith("/slow")) return; // never answers + res.writeHead(req.url!.endsWith("/missing") ? 404 : 200, { "content-type": "application/json", "x-multi": ["a", "b"] }).end(JSON.stringify({ echoed: Buffer.concat(chunks).toString() })); + }); + }); + await new Promise((r) => server.listen(0, "127.0.0.1", r)); + const base = `http://127.0.0.1:${(server.address() as AddressInfo).port}`; + try { + const f = caFetch("-----BEGIN CERTIFICATE-----\nunused for http\n-----END CERTIFICATE-----"); + const ok = await f(`${base}/echo`, { method: "PATCH", headers: { Authorization: "Bearer t", "Content-Type": "application/merge-patch+json" }, body: '{"a":1}' }); + expect(ok.status).toBe(200); + expect(await ok.json()).toEqual({ echoed: '{"a":1}' }); + expect(ok.headers.get("x-multi")).toBe("a, b"); + expect((await f(`${base}/gone`, { method: "DELETE" })).status).toBe(204); + expect((await f(`${base}/missing`)).status).toBe(404); + expect(seen[0]).toEqual({ method: "PATCH", url: "/echo", auth: "Bearer t", body: '{"a":1}' }); + await expect(f(`${base}/slow`, { signal: AbortSignal.timeout(50) })).rejects.toThrow(); + await expect(f(`${base}/echo`, { signal: AbortSignal.abort() })).rejects.toThrow(); + } finally { + server.closeAllConnections(); + server.close(); + } + }); + + it("with KUBE_CA_CERT the connector uses the CA transport; a timeout on a write is uncertain", async () => { + const server = http.createServer(() => undefined); // accepts and never answers + await new Promise((r) => server.listen(0, "127.0.0.1", r)); + const port = (server.address() as AddressInfo).port; + const ca = path.join(mkdtempSync(path.join(tmpdir(), "rv-ca-")), "ca.pem"); + writeFileSync(ca, "-----BEGIN CERTIFICATE-----\nx\n-----END CERTIFICATE-----\n"); + try { + const c = createKubernetesConnector({ apiServer: `http://127.0.0.1:${port}`, token: TOKEN, caCertPath: ca, timeoutMs: 80 }); + const err = await c.tools["kubernetes.restore_replicas"]!({ namespace: "prod", name: "web", replicas: 1 }, {}).catch((e) => e); + expect(err.name).toBe("EffectUncertainError"); + expect(err.message).not.toContain(TOKEN); + expect(() => createKubernetesConnector({ apiServer: SERVER, token: TOKEN, caCertPath: "/nonexistent/ca.pem" })).toThrow(); + } finally { + server.closeAllConnections(); + server.close(); + } + }); +}); diff --git a/packages/connectors/test/postgres.test.ts b/packages/connectors/test/postgres.test.ts new file mode 100644 index 0000000..4e63a21 --- /dev/null +++ b/packages/connectors/test/postgres.test.ts @@ -0,0 +1,239 @@ +import { EffectUncertainError, Registry } from "@multimindlab/reversi"; +import { describe, expect, it } from "vitest"; +import { compensatorFor, createPostgresConnector } from "../src/index.js"; +import { fakePg, PASSWORD, URL_SECRET } from "./fixtures/postgres.js"; +import { engine } from "./helpers.js"; + +function setup(opts: { connect?: boolean; timeoutMs?: number; maxRows?: number } = {}) { + const pg = fakePg(); + const connector = createPostgresConnector({ + pool: opts.connect ? pg.withConnect : pg.pg, + secrets: [URL_SECRET, PASSWORD], + timeoutMs: opts.timeoutMs, + maxRows: opts.maxRows, + }); + const t = (n: string, a: Record) => connector.tools[`postgres.${n}`]!(a, {}); + const sql = () => pg.calls.map((c) => c.url); + const log = () => pg.calls.map((c) => [c.url, JSON.parse(c.body)]); + return { pg, connector, t, sql, log }; +} + +const HOSTILE = `x'); DROP TABLE users; --`; + +describe("postgres manifests", () => { + it("validate, and every compensation is a registered idempotent tool", () => { + const { connector } = setup(); + const reg = Registry.from(connector.manifests); + expect(reg.size).toBe(5); + expect(reg.get("postgres.query")?.effect).toBe("read_only"); + expect(reg.get("postgres.insert_row")?.compensation?.tool).toBe("postgres.delete_row"); + expect(reg.get("postgres.update_row")?.compensation?.tool).toBe("postgres.restore_row"); + expect(reg.get("postgres.delete_row")?.effect).toBe("idempotent"); + expect(reg.get("postgres.restore_row")?.effect).toBe("idempotent"); + }); +}); + +describe("postgres emits exact parameterized SQL", () => { + it("insert_row", async () => { + const { t, log } = setup(); + expect(await t("insert_row", { table: "users", values: { name: HOSTILE, age: 41, data: { a: [1, 2] }, note: null } })).toEqual({ + table: "users", + schema: null, + key_column: "id", + key: 1, + }); + expect(log()).toEqual([ + ['INSERT INTO "users" ("name","age","data","note") VALUES ($1,$2,$3,$4) RETURNING "id"', [HOSTILE, 41, '{"a":[1,2]}', null]], + ]); + }); + + it("insert_row with a schema and a custom key column", async () => { + const { pg, t, sql } = setup(); + pg.tables.set("users", []); + await expect(t("insert_row", { table: "users", schema: "app", key_column: "id", values: { name: "x" } })).resolves.toMatchObject({ schema: "app" }); + expect(sql()).toEqual(['INSERT INTO "app"."users" ("name") VALUES ($1) RETURNING "id"']); + }); + + it("delete_row", async () => { + const { pg, t, log } = setup(); + pg.tables.get("users")!.push({ id: 7, name: "Ann" }); + expect(await t("delete_row", { table: "users", key: 7 })).toEqual({ deleted: true }); + expect(await t("delete_row", { table: "users", key: 7 })).toEqual({ deleted: false }); + expect(log()[0]).toEqual(['DELETE FROM "users" WHERE "id" = $1', [7]]); + }); + + it("update_row reads the old values in one transaction and returns them", async () => { + const { pg, t, log } = setup(); + pg.tables.get("users")!.push({ id: 7, name: "Ann", age: 30 }); + const r = await t("update_row", { table: "users", key: 7, set: { name: HOSTILE, age: 31 } }); + expect(r).toMatchObject({ key: 7, key_column: "id", previous: { name: "Ann", age: 30 } }); + expect(log()).toEqual([ + ["BEGIN", []], + ['SELECT "name","age" FROM "users" WHERE "id" = $1 FOR UPDATE', [7]], + ['UPDATE "users" SET "name" = $2, "age" = $3 WHERE "id" = $1', [7, HOSTILE, 31]], + ["COMMIT", []], + ]); + expect(pg.tables.get("users")![0]).toMatchObject({ name: HOSTILE, age: 31 }); + }); + + it("restore_row sets the saved values back", async () => { + const { pg, t, log } = setup(); + pg.tables.get("users")!.push({ id: 7, name: "Zed", age: 31 }); + await t("restore_row", { table: "users", key: 7, values: { name: "Ann", age: 30 } }); + expect(log()).toEqual([['UPDATE "users" SET "name" = $2, "age" = $3 WHERE "id" = $1', [7, "Ann", 30]]]); + await expect(t("restore_row", { table: "users", key: 99, values: { name: "x" } })).rejects.toThrow("no row with that key"); + }); + + it("query binds values and runs in a READ ONLY transaction", async () => { + const { pg, t, log } = setup(); + pg.tables.get("users")!.push({ id: 7, name: "Ann", age: 30 }); + const r = await t("query", { text: "SELECT * FROM users WHERE age > $1;", values: [20] }); + expect(r).toMatchObject({ row_count: 1, truncated: false, rows: [{ id: 7 }] }); + expect(log()).toEqual([ + ["BEGIN READ ONLY", []], + ["SELECT * FROM users WHERE age > $1", [20]], + ["ROLLBACK", []], + ]); + }); + + it("never interpolates a value into SQL text", async () => { + const { pg, t } = setup(); + pg.tables.get("users")!.push({ id: 7, name: "Ann", age: 30 }); + await t("insert_row", { table: "users", values: { name: HOSTILE } }); + await t("update_row", { table: "users", key: HOSTILE, set: { name: HOSTILE } }).catch(() => undefined); + await t("delete_row", { table: "users", key: HOSTILE }); + await t("restore_row", { table: "users", key: HOSTILE, values: { name: HOSTILE } }).catch(() => undefined); + await t("query", { text: "SELECT * FROM users WHERE age > $1", values: [HOSTILE] }).catch(() => undefined); + expect(pg.calls.length).toBeGreaterThan(8); + for (const c of pg.calls) { + expect(c.url).not.toContain("DROP"); + expect(c.url).not.toContain("x'"); + } + expect(pg.calls.some((c) => c.body.includes("DROP TABLE"))).toBe(true); // it travelled as a parameter + }); +}); + +describe("postgres validation", () => { + it("accepts only plain identifiers and quotes them", async () => { + const { t, pg } = setup(); + for (const bad of ["users; DROP TABLE x", 'us"ers', "1users", "", "a".repeat(64), "us ers", "users--", "üsers"]) { + await expect(t("insert_row", { table: bad, values: { a: 1 } })).rejects.toThrow(/table|identifier/i); + await expect(t("insert_row", { table: "users", values: { [bad]: 1 } })).rejects.toThrow(/column|values|must/i); + await expect(t("delete_row", { table: "users", key_column: bad, key: 1 })).rejects.toThrow(/identifier|key_column/); + await expect(t("delete_row", { table: "users", schema: bad, key: 1 })).rejects.toThrow(/identifier|schema/); + } + expect(pg.calls).toHaveLength(0); + pg.tables.set("a".repeat(63), []); + await t("insert_row", { table: "a".repeat(63), values: { _ok1: 1 } }); + expect(pg.calls[0]!.url).toBe(`INSERT INTO "${"a".repeat(63)}" ("_ok1") VALUES ($1) RETURNING "id"`); + }); + + it("rejects bad keys, empty or oversized value sets, unknown arguments and key-column updates", async () => { + const { t, pg } = setup(); + await expect(t("delete_row", { table: "users", key: {} })).rejects.toThrow("`key`"); + await expect(t("delete_row", { table: "users", key: 1.5 })).rejects.toThrow("`key`"); + await expect(t("insert_row", { table: "users", values: {} })).rejects.toThrow("must not be empty"); + await expect(t("insert_row", { table: "users", values: Object.fromEntries(Array.from({ length: 101 }, (_, i) => [`c${i}`, i])) })).rejects.toThrow("more than 100"); + await expect(t("insert_row", { table: "users", values: { a: Infinity } })).rejects.toThrow("finite"); + await expect(t("insert_row", { table: "users", values: { a: 1 }, extra: 1 })).rejects.toThrow("unknown argument"); + await expect(t("update_row", { table: "users", key: 1, set: { id: 2 } })).rejects.toThrow("key column"); + expect(pg.calls).toHaveLength(0); + }); + + it("query accepts only a single SELECT or WITH statement", async () => { + const { t, pg } = setup(); + for (const text of ["DELETE FROM users", "UPDATE users SET a = 1", "INSERT INTO users VALUES (1)", "DROP TABLE users", "SELECT 1; DROP TABLE users", "SELECT 1; COMMIT; DELETE FROM users", "", " --SELECT 1"]) { + await expect(t("query", { text })).rejects.toThrow(); + } + expect(pg.calls).toHaveLength(0); + }); + + it("a write hidden in a WITH is stopped by the READ ONLY transaction and rolled back", async () => { + const { pg, t, sql } = setup(); + pg.tables.get("users")!.push({ id: 7, name: "Ann", age: 30 }); + const err = await t("query", { text: "WITH gone AS (DELETE FROM users RETURNING *) SELECT * FROM gone" }).catch((e) => e); + expect(err.message).toContain("25006"); + expect(pg.tables.get("users")).toHaveLength(1); + expect(sql()).toEqual(["BEGIN READ ONLY", "WITH gone AS (DELETE FROM users RETURNING *) SELECT * FROM gone", "ROLLBACK"]); + }); + + it("query caps the rows it returns", async () => { + const { pg, t } = setup({ maxRows: 2 }); + for (let i = 1; i <= 5; i++) pg.tables.get("users")!.push({ id: i, name: "n", age: 30 }); + const r = (await t("query", { text: "SELECT * FROM users" })) as { rows: unknown[]; row_count: number; truncated: boolean }; + expect(r.rows).toHaveLength(2); + expect(r).toMatchObject({ row_count: 5, truncated: true }); + expect(((await t("query", { text: "SELECT * FROM users", limit: 1 })) as { rows: unknown[] }).rows).toHaveLength(1); + }); +}); + +describe("postgres failure handling", () => { + it("update_row on a missing row rolls back and changes nothing", async () => { + const { t, sql } = setup(); + await expect(t("update_row", { table: "users", key: 99, set: { name: "x" } })).rejects.toThrow("no row with that key"); + expect(sql()).toEqual(['BEGIN', 'SELECT "name" FROM "users" WHERE "id" = $1 FOR UPDATE', "ROLLBACK"]); + }); + + it("a server error carries its SQLSTATE and the transaction is rolled back", async () => { + const { pg, t } = setup(); + pg.tables.get("users")!.push({ id: 7, name: "Ann" }); + pg.fault = { kind: "status", status: 403, on: "mutating" }; + const err = await t("update_row", { table: "users", key: 7, set: { name: "x" } }).catch((e) => e); + expect(err).not.toBeInstanceOf(EffectUncertainError); + expect(err.message).toContain("42501"); + expect(err.code).toBe("42501"); + expect(pg.calls.at(-1)!.url).toBe("ROLLBACK"); + }); + + it("a lost connection after a write was sent is uncertain; before it, a plain error", async () => { + const a = setup(); + a.pg.tables.get("users")!.push({ id: 7, name: "Ann" }); + a.pg.fault = { kind: "throw", message: `read ECONNRESET ${URL_SECRET}`, on: "mutating" }; + const err = await a.t("update_row", { table: "users", key: 7, set: { name: "x" } }).catch((e) => e); + expect(err).toBeInstanceOf(EffectUncertainError); + expect(err.message).not.toContain(PASSWORD); + expect(err.message).toContain("[redacted]"); + + const b = setup(); + b.pg.fault = { kind: "throw", message: "connection refused", on: "any" }; + const e2 = await b.t("insert_row", { table: "users", values: { name: "x" } }).catch((e) => e); + expect(e2).toBeInstanceOf(EffectUncertainError); // the INSERT itself was attempted + const c = setup(); + c.pg.fault = { kind: "throw", message: "connection refused", on: "any" }; + const e3 = await c.t("query", { text: "SELECT 1" }).catch((e) => e); + expect(e3).not.toBeInstanceOf(EffectUncertainError); + }); + + it("uses a pooled client and releases it, dropping it when its state is unknown", async () => { + const { pg, t, sql } = setup({ connect: true }); + pg.tables.get("users")!.push({ id: 7, name: "Ann", age: 30 }); + await t("update_row", { table: "users", key: 7, set: { age: 31 } }); + expect(pg.released).toEqual([false]); + pg.fault = { kind: "throw", message: "terminating connection", on: "mutating" }; + await t("delete_row", { table: "users", key: 7 }).catch(() => undefined); + expect(pg.released).toEqual([false, true]); + expect(sql()[0]).toBe("BEGIN"); + }); + + it("times out a query that never answers", async () => { + const { pg, t } = setup({ timeoutMs: 30 }); + pg.fault = { kind: "hang", on: "mutating" }; + await expect(t("delete_row", { table: "users", key: 1 })).rejects.toBeInstanceOf(EffectUncertainError); + }); +}); + +describe("postgres end to end", () => { + it("rolls back an insert and an update through a real engine", async () => { + const { pg, connector } = setup(); + pg.tables.get("users")!.push({ id: 7, name: "Ann", age: 30 }); + const before = JSON.stringify(pg.state()); + const { call, run } = engine([connector]); + await call("postgres.insert_row", { table: "users", values: { name: "Bo", age: 5 } }); + await call("postgres.update_row", { table: "users", key: 7, set: { name: "Zed" } }); + await call("postgres.query", { text: "SELECT * FROM users" }); + expect(JSON.stringify(pg.state())).not.toBe(before); + const report = await run.rollback({ compensate: compensatorFor([connector]) }); + expect(report.complete).toBe(true); + expect(JSON.stringify(pg.state())).toBe(before); + }); +}); diff --git a/packages/connectors/test/server.test.ts b/packages/connectors/test/server.test.ts index 5452621..a0d3375 100644 --- a/packages/connectors/test/server.test.ts +++ b/packages/connectors/test/server.test.ts @@ -87,7 +87,7 @@ describe("reversi-connectors binary", () => { expect(list.map((m) => m.tool)).toContain("stripe.create_refund"); Registry.from(list); const json = JSON.parse(run(["manifests", "all", "--format", "json"]).stdout) as unknown[]; - expect(Registry.from(json).size).toBe(29); + expect(Registry.from(json).size).toBe(84); }); it.skipIf(!existsSync(main))("serve refuses to start without its credential and never takes it from flags", () => { diff --git a/packages/connectors/test/sigv4.test.ts b/packages/connectors/test/sigv4.test.ts new file mode 100644 index 0000000..b453ca7 --- /dev/null +++ b/packages/connectors/test/sigv4.test.ts @@ -0,0 +1,98 @@ +import { describe, expect, it } from "vitest"; +import { amzDateOf, signV4, signingKey, uriEncode } from "../src/index.js"; +import { sha256Hex } from "../src/sigv4.js"; + +// The documented credentials used throughout the AWS Signature Version 4 test suite and examples. +const ACCESS_KEY_ID = "AKIDEXAMPLE"; +const SECRET = "wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY"; + +describe("SigV4 against the AWS documented test vectors", () => { + it("get-vanilla from the SigV4 test suite", () => { + const out = signV4({ + method: "GET", + canonicalUri: "/", + headers: { Host: "example.amazonaws.com", "X-Amz-Date": "20150830T123600Z" }, + payloadHash: sha256Hex(""), + region: "us-east-1", + service: "service", + accessKeyId: ACCESS_KEY_ID, + secretAccessKey: SECRET, + amzDate: "20150830T123600Z", + }); + // get-vanilla.creq + expect(out.canonicalRequest).toBe( + [ + "GET", + "/", + "", + "host:example.amazonaws.com", + "x-amz-date:20150830T123600Z", + "", + "host;x-amz-date", + "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + ].join("\n"), + ); + // get-vanilla.sts + expect(out.stringToSign).toBe( + [ + "AWS4-HMAC-SHA256", + "20150830T123600Z", + "20150830/us-east-1/service/aws4_request", + "bb579772317eb040ac9ed261061d46c1f17a8133879d6129b6e1c25292927e63", + ].join("\n"), + ); + // get-vanilla.authz + expect(out.signature).toBe("5fa00fa31553b73ebf1942676e86291e8372ff2a2260956d9b8aae1d763fbf31"); + expect(out.authorization).toBe( + "AWS4-HMAC-SHA256 Credential=AKIDEXAMPLE/20150830/us-east-1/service/aws4_request, SignedHeaders=host;x-amz-date, Signature=5fa00fa31553b73ebf1942676e86291e8372ff2a2260956d9b8aae1d763fbf31", + ); + }); + + it("derives the documented signing key (IAM example, 20150830/us-east-1/iam)", () => { + expect(signingKey(SECRET, "20150830", "us-east-1", "iam").toString("hex")).toBe( + "c4afb1cc5771d871763a393e44b703571b55cc28424d1a5e86da6ed3c154a4b9", + ); + }); + + it("sorts and encodes query parameters, and orders signed headers", () => { + const out = signV4({ + method: "GET", + canonicalUri: "/", + query: [["b", "2"], ["a", "x y"], ["a", "1"]], + headers: { "X-Amz-Date": "20150830T123600Z", Host: "example.amazonaws.com" }, + payloadHash: sha256Hex(""), + region: "us-east-1", + service: "service", + accessKeyId: ACCESS_KEY_ID, + secretAccessKey: SECRET, + amzDate: "20150830T123600Z", + }); + expect(out.canonicalRequest.split("\n")[2]).toBe("a=1&a=x%20y&b=2"); + expect(out.signedHeaders).toBe("host;x-amz-date"); + }); + + it("signs the session token header when one is given", () => { + const out = signV4({ + method: "GET", + canonicalUri: "/", + headers: { host: "example.amazonaws.com", "x-amz-date": "20150830T123600Z" }, + payloadHash: sha256Hex(""), + region: "us-east-1", + service: "service", + accessKeyId: ACCESS_KEY_ID, + secretAccessKey: SECRET, + sessionToken: "TOKEN", + amzDate: "20150830T123600Z", + }); + expect(out.signedHeaders).toBe("host;x-amz-date;x-amz-security-token"); + expect(out.headers["x-amz-security-token"]).toBe("TOKEN"); + expect(out.signature).not.toBe("5fa00fa31553b73ebf1942676e86291e8372ff2a2260956d9b8aae1d763fbf31"); + }); + + it("encodes URIs the way AWS requires", () => { + expect(uriEncode("a b/c~d-e_f.g")).toBe("a%20b%2Fc~d-e_f.g"); + expect(uriEncode("a b/c", true)).toBe("a%20b/c"); + expect(uriEncode("é*!'()")).toBe("%C3%A9%2A%21%27%28%29"); + expect(amzDateOf(new Date("2015-08-30T12:36:00.123Z"))).toBe("20150830T123600Z"); + }); +}); diff --git a/packages/connectors/test/storage.test.ts b/packages/connectors/test/storage.test.ts new file mode 100644 index 0000000..63ca5b5 --- /dev/null +++ b/packages/connectors/test/storage.test.ts @@ -0,0 +1,343 @@ +import { EffectUncertainError, Registry } from "@multimindlab/reversi"; +import { describe, expect, it } from "vitest"; +import { compensatorFor, createAwsConnector, createAzureConnector, createGcpConnector } from "../src/index.js"; +import { ACCESS_KEY_ID, BUCKET, NOW, REGION, SECRET, SESSION, fakeS3, verifySignature } from "./fixtures/aws.js"; +import { ACCOUNT, CONTAINER, TOKEN as AZ_TOKEN, fakeBlob } from "./fixtures/azure.js"; +import { BUCKET as GCS_BUCKET, TOKEN as GCP_TOKEN, fakeGcs } from "./fixtures/gcp.js"; +import { engine } from "./helpers.js"; + +const route = (c: { method: string; url: string }) => `${c.method} ${decodeURIComponent(new URL(c.url).pathname)}`; + +describe("aws (S3)", () => { + const make = (o: { endpoint?: string } = {}) => { + const b = fakeS3(); + const c = createAwsConnector({ accessKeyId: ACCESS_KEY_ID, secretAccessKey: SECRET, sessionToken: SESSION, region: REGION, fetch: b.fetch, now: () => NOW, ...o }); + return { b, c, t: (n: string, a: Record) => c.tools[`aws.${n}`]!(a, {}) }; + }; + const put = (b: ReturnType, k: string, v: string) => b.objects.set(k, { bytes: Buffer.from(v), type: "text/plain" }); + + it("manifests validate; one restore tool undoes both put and delete", () => { + const { c } = make(); + const reg = Registry.from(c.manifests); + expect(reg.size).toBe(5); + expect(reg.get("aws.put_object")?.compensation).toMatchObject({ tool: "aws.restore_object", argument_map: { bucket: "$args.bucket", key: "$args.key", backup_key: "$result.backup_key" } }); + expect(reg.get("aws.delete_object")?.compensation?.tool).toBe("aws.restore_object"); + expect(reg.get("aws.get_object")?.effect).toBe("read_only"); + expect(reg.get("aws.list_objects")?.effect).toBe("read_only"); + }); + + it("signs every request with SigV4 (verified by the fake from the request as received)", async () => { + const { b, t } = make(); + put(b, "docs/a.txt", "v1"); + await t("get_object", { bucket: BUCKET, key: "docs/a.txt" }); + await t("put_object", { bucket: BUCKET, key: "docs/a b+c.txt", content: "x" }); + await t("list_objects", { bucket: BUCKET, prefix: "docs/" }); + expect(b.calls.length).toBeGreaterThanOrEqual(4); + for (const c of b.calls) { + expect(verifySignature(c)).toBeUndefined(); + expect(c.headers["x-amz-date"]).toBe("20260501T120000Z"); + expect(c.headers["x-amz-security-token"]).toBe(SESSION); + expect(c.headers.authorization).toMatch(/^AWS4-HMAC-SHA256 Credential=AKIAEXAMPLEKEYID1234\/20260501\/us-east-1\/s3\/aws4_request, SignedHeaders=.*host.*x-amz-content-sha256.*x-amz-date.*x-amz-security-token, Signature=[0-9a-f]{64}$/); + expect(c.headers.authorization).not.toContain(SECRET); + } + }); + + it("a tampered request fails verification", async () => { + const { b, t } = make(); + await t("put_object", { bucket: BUCKET, key: "k", content: "x" }); + const call = b.calls.at(-1)!; + expect(verifySignature({ ...call, headers: { ...call.headers, "x-amz-date": "20260501T120001Z" } })).toBe("signature mismatch"); + }); + + it("uses virtual-hosted URLs, path-style for dotted buckets, and a custom endpoint", async () => { + const { b, t } = make(); + await t("put_object", { bucket: BUCKET, key: "a/b.txt", content: "x" }); + expect(b.calls[0]!.url).toBe(`https://${BUCKET}.s3.us-east-1.amazonaws.com/a/b.txt`); + const dotted = make(); + await dotted.t("get_object", { bucket: "my.dotted.bucket", key: "a" }).catch(() => undefined); + expect(dotted.b.calls[0]!.url).toBe("https://s3.us-east-1.amazonaws.com/my.dotted.bucket/a"); + const local = make({ endpoint: "http://localhost:9000" }); + await local.t("list_objects", { bucket: BUCKET, max_keys: 5 }).catch(() => undefined); + expect(local.b.calls[0]!.url).toBe(`http://localhost:9000/${BUCKET}?list-type=2&max-keys=5`); + }); + + it("put over an existing object: HEAD, copy to a backup key, then PUT", async () => { + const { b, t } = make(); + put(b, "docs/a.txt", "v1"); + const r = (await t("put_object", { bucket: BUCKET, key: "docs/a.txt", content: "v2" })) as { existed: boolean; backup_key: string }; + expect(r.existed).toBe(true); + expect(r.backup_key).toMatch(/^\.reversi-backups\/[0-9a-f-]{36}$/); + expect(b.calls.map(route)).toEqual(["HEAD /docs/a.txt", `PUT /${r.backup_key}`, "PUT /docs/a.txt"]); + expect(b.calls[1]!.headers["x-amz-copy-source"]).toBe(`/${BUCKET}/docs/a.txt`); + expect(b.objects.get(r.backup_key)!.bytes.toString()).toBe("v1"); + expect(b.objects.get("docs/a.txt")!.bytes.toString()).toBe("v2"); + }); + + it("put of a new key makes no backup", async () => { + const { b, t } = make(); + const r = await t("put_object", { bucket: BUCKET, key: "new.txt", content: "x", content_type: "text/csv" }); + expect(r).toMatchObject({ existed: false, backup_key: null }); + expect(b.calls.map((c) => c.method)).toEqual(["HEAD", "PUT"]); + expect(b.calls[1]!.headers["content-type"]).toBe("text/csv"); + expect([...b.objects.keys()]).toEqual(["new.txt"]); + }); + + it("a definite failure of the PUT removes the backup; an uncertain one keeps it and names it", async () => { + const definite = make(); + put(definite.b, "k", "v1"); + definite.b.fault = { kind: "status", status: 403, on: "mutating" }; + // the copy is the first write; let it through, fail only the second write + let writes = 0; + const real = definite.b.fetch; + const c1 = createAwsConnector({ + accessKeyId: ACCESS_KEY_ID, secretAccessKey: SECRET, region: REGION, now: () => NOW, + fetch: (async (u: string, i: RequestInit) => { + const isWrite = i.method === "PUT"; + if (isWrite && ++writes === 2) return new Response("AccessDenied", { status: 403 }); + definite.b.fault = null; + return real(u, i); + }) as typeof fetch, + }); + const err = await c1.tools["aws.put_object"]!({ bucket: BUCKET, key: "k", content: "v2" }, {}).catch((e) => e); + expect(err.message).toContain("AccessDenied"); + expect([...definite.b.objects.keys()]).toEqual(["k"]); + + const lost = make(); + put(lost.b, "k", "v1"); + let n = 0; + const real2 = lost.b.fetch; + const c2 = createAwsConnector({ + accessKeyId: ACCESS_KEY_ID, secretAccessKey: SECRET, region: REGION, now: () => NOW, + fetch: (async (u: string, i: RequestInit) => { + if (i.method === "PUT" && ++n === 2) throw new TypeError("socket hang up"); + return real2(u, i); + }) as typeof fetch, + }); + const e2 = await c2.tools["aws.put_object"]!({ bucket: BUCKET, key: "k", content: "v2" }, {}).catch((e) => e); + expect(e2).toBeInstanceOf(EffectUncertainError); + const backup = [...lost.b.objects.keys()].find((k) => k.startsWith(".reversi-backups/"))!; + expect(e2.message).toContain(backup); + }); + + it("delete copies aside then deletes, and restore brings it back; the backup stays as residue", async () => { + const { b, c } = make(); + put(b, "docs/a.txt", "v1"); + const { call, run } = engine([c]); + await call("aws.delete_object", { bucket: BUCKET, key: "docs/a.txt" }); + expect(b.objects.has("docs/a.txt")).toBe(false); + expect([...b.objects.keys()].filter((k) => k.startsWith(".reversi-backups/"))).toHaveLength(1); + const report = await run.rollback({ compensate: compensatorFor([c]) }); + expect(report.complete).toBe(true); + expect(b.objects.get("docs/a.txt")!.bytes.toString()).toBe("v1"); + expect([...b.objects.keys()].filter((k) => k.startsWith(".reversi-backups/"))).toHaveLength(1); + }); + + it("deleting a missing key is a no-op whose rollback leaves the key absent", async () => { + const { b, t } = make(); + expect(await t("delete_object", { bucket: BUCKET, key: "nope" })).toMatchObject({ existed: false, backup_key: null }); + expect(b.calls.map((c) => c.method)).toEqual(["HEAD"]); + }); + + it("returns binary content as base64 and honours max_bytes", async () => { + const { b, t } = make(); + b.objects.set("bin", { bytes: Buffer.from([0xff, 0xfe, 0x00, 0x01]), type: "application/octet-stream" }); + expect(await t("get_object", { bucket: BUCKET, key: "bin" })).toMatchObject({ encoding: "base64", content: "//4AAQ==", size: 4 }); + put(b, "big", "0123456789"); + const r = (await t("get_object", { bucket: BUCKET, key: "big", max_bytes: 4 })) as { content: string; truncated: boolean; size: number }; + expect(r).toMatchObject({ content: "0123", truncated: true, size: 10 }); + expect(b.calls.at(-1)!.headers.range).toBe("bytes=0-3"); + await t("put_object", { bucket: BUCKET, key: "bin2", content: "//4AAQ==", encoding: "base64" }); + expect([...b.objects.get("bin2")!.bytes]).toEqual([0xff, 0xfe, 0x00, 0x01]); + }); + + it("lists with XML entities decoded and hides backups", async () => { + const { b, t } = make(); + put(b, "a&b.txt", "x"); + put(b, ".reversi-backups/00000000-0000-4000-8000-000000000009", "x"); + const r = (await t("list_objects", { bucket: BUCKET })) as { objects: Array<{ key: string }> }; + expect(r.objects.map((o) => o.key)).toEqual(["a&b.txt"]); + }); + + it("surfaces S3 error codes with the status, not the credential, and rejects a copy that answers 200 with an error", async () => { + const denied = createAwsConnector({ + accessKeyId: ACCESS_KEY_ID, secretAccessKey: SECRET, region: REGION, + fetch: (async () => new Response(`AccessDeniedno ${SECRET}`, { status: 403 })) as typeof fetch, + }); + const err = await denied.tools["aws.get_object"]!({ bucket: BUCKET, key: "k" }, {}).catch((e) => e); + expect(err.message).toContain("status 403: AccessDenied"); + expect(err.message).not.toContain(SECRET); + const sneaky = createAwsConnector({ + accessKeyId: ACCESS_KEY_ID, secretAccessKey: SECRET, region: REGION, + fetch: (async (_u: string, i: RequestInit) => + i.method === "HEAD" ? new Response(null, { status: 200 }) : new Response("InternalError", { status: 200 })) as typeof fetch, + }); + await expect(sneaky.tools["aws.delete_object"]!({ bucket: BUCKET, key: "k" }, {})).rejects.toThrow("InternalError"); + }); + + it("validates input and refuses the reserved backup prefix for writes", async () => { + const { b, t } = make(); + await expect(t("put_object", { bucket: "Bad_Bucket", key: "k", content: "x" })).rejects.toThrow("bucket name"); + await expect(t("put_object", { bucket: BUCKET, key: ".reversi-backups/x", content: "x" })).rejects.toThrow("reserved"); + await expect(t("delete_object", { bucket: BUCKET, key: ".reversi-backups/x" })).rejects.toThrow("reserved"); + await expect(t("put_object", { bucket: BUCKET, key: "k", content: "!!!", encoding: "base64" })).rejects.toThrow("base64"); + await expect(t("put_object", { bucket: BUCKET, key: "k", content: "x", encoding: "latin1" })).rejects.toThrow("encoding"); + await expect(t("put_object", { bucket: BUCKET, key: "k", content: "x".repeat(5 * 1024 * 1024 + 1) })).rejects.toThrow("larger than"); + await expect(t("restore_object", { bucket: BUCKET, key: "k", backup_key: "somewhere/else" })).rejects.toThrow("backup object name"); + await expect(t("put_object", { bucket: BUCKET, key: "k\nx", content: "x" })).rejects.toThrow("control character"); + await expect(t("list_objects", { bucket: BUCKET, max_keys: 5000 })).rejects.toThrow("at most 1000"); + expect(b.calls).toHaveLength(0); + expect(() => createAwsConnector({ accessKeyId: "a", secretAccessKey: "b", region: "mars" })).toThrow("region"); + }); +}); + +describe("gcp (Cloud Storage)", () => { + const make = () => { + const b = fakeGcs(); + const c = createGcpConnector({ accessToken: GCP_TOKEN, fetch: b.fetch }); + return { b, c, t: (n: string, a: Record) => c.tools[`gcp.${n}`]!(a, {}) }; + }; + const put = (b: ReturnType, k: string, v: string, generation = 5) => b.objects.set(k, { bytes: Buffer.from(v), type: "text/plain", generation }); + + it("manifests validate", () => { + const reg = Registry.from(make().c.manifests); + expect(reg.size).toBe(5); + expect(reg.get("gcp.put_object")?.compensation?.tool).toBe("gcp.restore_object"); + expect(reg.get("gcp.delete_object")?.compensation?.tool).toBe("gcp.restore_object"); + }); + + it("overwrites conditionally on the generation it backed up, following a multi-round rewrite", async () => { + const { b, t } = make(); + put(b, "docs/a.txt", "v1", 42); + const r = (await t("put_object", { bucket: GCS_BUCKET, key: "docs/a.txt", content: "v2" })) as { backup_key: string }; + const seen = b.calls.map((c) => `${c.method} ${new URL(c.url).pathname}${new URL(c.url).search}`); + expect(seen[0]).toBe(`GET /storage/v1/b/${GCS_BUCKET}/o/docs%2Fa.txt?fields=generation`); + const rewrites = seen.filter((s) => s.includes("/rewriteTo/")); + expect(rewrites).toHaveLength(2); + expect(rewrites[0]).toContain("sourceGeneration=42"); + expect(rewrites[1]).toContain("rewriteToken=round-2"); + expect(seen.at(-1)).toBe(`POST /upload/storage/v1/b/${GCS_BUCKET}/o?uploadType=media&name=docs%2Fa.txt&ifGenerationMatch=42`); + expect(b.calls.every((c) => c.headers.authorization === `Bearer ${GCP_TOKEN}`)).toBe(true); + expect(b.objects.get(r.backup_key)!.bytes.toString()).toBe("v1"); + }); + + it("a new object is created only if absent (ifGenerationMatch=0)", async () => { + const { b, t } = make(); + await t("put_object", { bucket: GCS_BUCKET, key: "new.txt", content: "x" }); + expect(b.calls.at(-1)!.url).toContain("ifGenerationMatch=0"); + // a concurrent writer wins the race: the upload is refused and nothing is overwritten + const { b: b2, c } = make(); + const real = b2.fetch; + const racing = createGcpConnector({ + accessToken: GCP_TOKEN, + fetch: (async (u: string, i: RequestInit) => { + if (i.method === "POST" && u.includes("/upload/")) put(b2, "new.txt", "theirs", 77); + return real(u, i); + }) as typeof fetch, + }); + void c; + const err = await racing.tools["gcp.put_object"]!({ bucket: GCS_BUCKET, key: "new.txt", content: "mine" }, {}).catch((e) => e); + expect(err.message).toContain("status 412"); + expect(b2.objects.get("new.txt")!.bytes.toString()).toBe("theirs"); + }); + + it("delete is conditional on the generation and fully restorable", async () => { + const { b, c } = make(); + put(b, "docs/a.txt", "v1", 9); + const { call, run } = engine([c]); + await call("gcp.delete_object", { bucket: GCS_BUCKET, key: "docs/a.txt" }); + expect(b.calls.at(-1)!.url).toContain("ifGenerationMatch=9"); + expect(b.objects.has("docs/a.txt")).toBe(false); + expect((await run.rollback({ compensate: compensatorFor([c]) })).complete).toBe(true); + expect(b.objects.get("docs/a.txt")!.bytes.toString()).toBe("v1"); + }); + + it("validates input", async () => { + const { b, t } = make(); + await expect(t("get_object", { bucket: "UPPER", key: "k" })).rejects.toThrow("bucket name"); + await expect(t("put_object", { bucket: GCS_BUCKET, key: ".reversi-backups/a", content: "x" })).rejects.toThrow("reserved"); + await expect(t("restore_object", { bucket: GCS_BUCKET, key: "k", backup_key: "x" })).rejects.toThrow("backup object name"); + expect(b.calls).toHaveLength(0); + }); +}); + +describe("azure (Blob Storage)", () => { + const make = () => { + const b = fakeBlob(); + const c = createAzureConnector({ account: ACCOUNT, token: AZ_TOKEN, fetch: b.fetch }); + return { b, c, t: (n: string, a: Record) => c.tools[`azure.${n}`]!(a, {}) }; + }; + const put = (b: ReturnType, k: string, v: string, etag = '"0x0"') => b.blobs.set(k, { bytes: Buffer.from(v), type: "text/plain", etag }); + + it("manifests validate", () => { + const reg = Registry.from(make().c.manifests); + expect(reg.size).toBe(5); + expect(reg.get("azure.put_blob")?.compensation).toMatchObject({ tool: "azure.restore_blob", argument_map: { container: "$args.container", blob: "$args.blob", backup_blob: "$result.backup_blob" } }); + }); + + it("creates new blobs only if absent, and overwrites only the version it backed up", async () => { + const { b, t } = make(); + await t("put_blob", { container: CONTAINER, blob: "a/new.txt", content: "x", content_type: "text/plain" }); + const create = b.calls.at(-1)!; + expect(create.url).toBe(`https://${ACCOUNT}.blob.core.windows.net/${CONTAINER}/a/new.txt`); + expect(create.headers["if-none-match"]).toBe("*"); + expect(create.headers["x-ms-blob-type"]).toBe("BlockBlob"); + expect(create.headers["x-ms-version"]).toBe("2023-11-03"); + expect(create.headers.authorization).toBe(`Bearer ${AZ_TOKEN}`); + + put(b, "a/b.txt", "v1", '"0xABC"'); + const r = (await t("put_blob", { container: CONTAINER, blob: "a/b.txt", content: "v2" })) as { backup_blob: string }; + const calls = b.calls.slice(-3); + expect(calls.map((c) => c.method)).toEqual(["HEAD", "PUT", "PUT"]); + expect(calls[1]!.headers["x-ms-copy-source"]).toBe(`https://${ACCOUNT}.blob.core.windows.net/${CONTAINER}/a/b.txt`); + expect(calls[1]!.headers["x-ms-requires-sync"]).toBe("true"); + expect(calls[1]!.headers["x-ms-source-if-match"]).toBe('"0xABC"'); + expect(calls[2]!.headers["if-match"]).toBe('"0xABC"'); + expect(b.blobs.get(r.backup_blob)!.bytes.toString()).toBe("v1"); + }); + + it("an overwrite by someone else in between is refused", async () => { + const { b } = make(); + put(b, "a/b.txt", "v1", '"0x1"'); + const real = b.fetch; + const c = createAzureConnector({ + account: ACCOUNT, + token: AZ_TOKEN, + fetch: (async (u: string, i: RequestInit) => { + const res = await real(u, i); + if (i.method === "PUT" && (i.headers as Record)["x-ms-copy-source"]) put(b, "a/b.txt", "theirs", '"0x2"'); + return res; + }) as typeof fetch, + }); + const err = await c.tools["azure.put_blob"]!({ container: CONTAINER, blob: "a/b.txt", content: "mine" }, {}).catch((e) => e); + expect(err.message).toContain("status 412"); + expect(err.message).toContain("ConditionNotMet"); + expect(b.blobs.get("a/b.txt")!.bytes.toString()).toBe("theirs"); + expect([...b.blobs.keys()].filter((k) => k.startsWith(".reversi-backups/"))).toEqual([]); // the backup was cleaned up + }); + + it("delete is conditional on the ETag and restorable", async () => { + const { b, c } = make(); + put(b, "a/b.txt", "v1", '"0x7"'); + const { call, run } = engine([c]); + await call("azure.delete_blob", { container: CONTAINER, blob: "a/b.txt" }); + expect(b.calls.filter((x) => x.method === "DELETE")[0]!.headers["if-match"]).toBe('"0x7"'); + expect((await run.rollback({ compensate: compensatorFor([c]) })).complete).toBe(true); + expect(b.blobs.get("a/b.txt")!.bytes.toString()).toBe("v1"); + }); + + it("validates input and names Azure error codes without the token", async () => { + const { b, t } = make(); + await expect(t("get_blob", { container: "Bad_Container", blob: "x" })).rejects.toThrow("container name"); + await expect(t("put_blob", { container: CONTAINER, blob: ".reversi-backups/x", content: "x" })).rejects.toThrow("reserved"); + expect(b.calls).toHaveLength(0); + const denied = createAzureConnector({ + account: ACCOUNT, + token: AZ_TOKEN, + fetch: (async () => new Response(`AuthorizationPermissionMismatch${AZ_TOKEN}\nRequestId:1`, { status: 403 })) as typeof fetch, + }); + const err = await denied.tools["azure.get_blob"]!({ container: CONTAINER, blob: "x" }, {}).catch((e) => e); + expect(err.message).toContain("status 403: AuthorizationPermissionMismatch"); + expect(err.message).not.toContain(AZ_TOKEN); + expect(() => createAzureConnector({ account: "BAD", token: "t" })).toThrow("account"); + }); +}); diff --git a/packages/connectors/test/tickets.test.ts b/packages/connectors/test/tickets.test.ts new file mode 100644 index 0000000..733b562 --- /dev/null +++ b/packages/connectors/test/tickets.test.ts @@ -0,0 +1,313 @@ +import { Registry } from "@multimindlab/reversi"; +import { describe, expect, it } from "vitest"; +import { + compensatorFor, + createGcalendarConnector, + createGitlabConnector, + createJiraConnector, + createLinearConnector, + createTeamsConnector, +} from "../src/index.js"; +import { fakeCalendar, TOKEN as CAL_TOKEN } from "./fixtures/gcalendar.js"; +import { fakeGitlab, TOKEN as GL_TOKEN, PROJECT } from "./fixtures/gitlab.js"; +import { BASE, BASIC, EMAIL, fakeJira, TOKEN as JIRA_TOKEN } from "./fixtures/jira.js"; +import { fakeLinear, ISSUE, KEY, TEAM as LIN_TEAM } from "./fixtures/linear.js"; +import { CHANNEL, fakeTeams, TEAM, TOKEN as TEAMS_TOKEN } from "./fixtures/teams.js"; +import { engine } from "./helpers.js"; + +describe("gitlab", () => { + const make = (baseUrl?: string) => { + const b = fakeGitlab(); + const c = createGitlabConnector({ token: GL_TOKEN, fetch: b.fetch, baseUrl }); + return { b, c, t: (n: string, a: Record) => c.tools[`gitlab.${n}`]!(a, {}) }; + }; + + it("manifests validate and compensations are registered", () => { + const { c } = make(); + const reg = Registry.from(c.manifests); + expect(reg.size).toBe(7); + expect(reg.get("gitlab.create_issue")?.compensation?.tool).toBe("gitlab.close_issue"); + expect(reg.get("gitlab.add_note")?.compensation?.tool).toBe("gitlab.delete_note"); + expect(reg.get("gitlab.create_branch")?.compensation?.tool).toBe("gitlab.delete_branch"); + }); + + it("sends the token in PRIVATE-TOKEN and encodes the project path", async () => { + const { b, t } = make(); + await t("create_issue", { project: PROJECT, title: "T", labels: ["a", "b"] }); + const call = b.calls[0]!; + expect(call.url).toBe("https://gitlab.com/api/v4/projects/group%2Fproj/issues"); + expect(call.headers["private-token"]).toBe(GL_TOKEN); + expect(call.headers.authorization).toBeUndefined(); + expect(JSON.parse(call.body)).toEqual({ title: "T", labels: "a,b" }); + }); + + it("works against a self-hosted instance", async () => { + const { b, t } = make("https://git.corp.example/"); + await t("create_branch", { project: PROJECT, branch: "x/y", ref: "main" }); + expect(b.calls[0]!.url).toBe("https://git.corp.example/api/v4/projects/group%2Fproj/repository/branches"); + expect(JSON.parse(b.calls[0]!.body)).toEqual({ branch: "x/y", ref: "main" }); + const { b: b2, t: t2 } = make("https://git.corp.example/api/v4"); + await t2("delete_branch", { project: PROJECT, branch: "x/y" }); + expect(b2.calls[0]!.url).toBe("https://git.corp.example/api/v4/projects/group%2Fproj/repository/branches/x%2Fy"); + }); + + it("validates arguments before any request", async () => { + const { b, t } = make(); + await expect(t("get_issue", { project: "../etc", issue_iid: 1 })).rejects.toThrow("project id or path"); + await expect(t("get_issue", { project: PROJECT, issue_iid: 0 })).rejects.toThrow("positive integer"); + await expect(t("create_branch", { project: PROJECT, branch: "a..b", ref: "main" })).rejects.toThrow("branch name"); + await expect(t("create_branch", { project: PROJECT, branch: "-x", ref: "main" })).rejects.toThrow("branch name"); + await expect(t("close_issue", { project: PROJECT, issue_iid: 1, nope: 1 })).rejects.toThrow("unknown argument"); + expect(b.calls).toHaveLength(0); + }); + + it("closing is the undo for create_issue, and the issue stays visible", async () => { + const { b, c } = make(); + const { call, run } = engine([c]); + await call("gitlab.create_issue", { project: PROJECT, title: "T" }); + const r = await run.rollback({ compensate: compensatorFor([c]) }); + expect(r.complete).toBe(true); + expect([...b.issues.values()]).toEqual([{ title: "T", state: "closed" }]); + expect(b.calls.at(-1)!.method).toBe("PUT"); + }); +}); + +describe("teams (Microsoft Graph)", () => { + const make = () => { + const b = fakeTeams(); + const c = createTeamsConnector({ token: TEAMS_TOKEN, fetch: b.fetch }); + return { b, c, t: (n: string, a: Record) => c.tools[`teams.${n}`]!(a, {}) }; + }; + + it("uses the Graph channel message endpoints with a bearer token", async () => { + const { b, t } = make(); + const posted = (await t("post_message", { team_id: TEAM, channel_id: CHANNEL, content: "hi", content_type: "html" })) as { message_id: string }; + const post = b.calls[0]!; + expect(post.method).toBe("POST"); + expect(post.url).toBe(`https://graph.microsoft.com/v1.0/teams/${TEAM}/channels/19%3Aabc123%40thread.tacv2/messages`); + expect(post.headers.authorization).toBe(`Bearer ${TEAMS_TOKEN}`); + expect(JSON.parse(post.body)).toEqual({ body: { contentType: "html", content: "hi" } }); + await t("delete_message", { team_id: TEAM, channel_id: CHANNEL, message_id: posted.message_id }); + const del = b.calls[1]!; + expect(del.method).toBe("POST"); + expect(del.url).toBe(`https://graph.microsoft.com/v1.0/teams/${TEAM}/channels/19%3Aabc123%40thread.tacv2/messages/${posted.message_id}/softDelete`); + await t("list_channels", { team_id: TEAM, top: 5 }); + expect(b.calls[2]!.url).toBe(`https://graph.microsoft.com/v1.0/teams/${TEAM}/channels?$top=5`); + }); + + it("treats a missing message as already deleted and validates ids", async () => { + const { t, b } = make(); + expect(await t("delete_message", { team_id: TEAM, channel_id: CHANNEL, message_id: "1700000000555" })).toMatchObject({ deleted: false }); + const n = b.calls.length; + await expect(t("post_message", { team_id: "nope", channel_id: CHANNEL, content: "x" })).rejects.toThrow("team ID"); + await expect(t("post_message", { team_id: TEAM, channel_id: "general", content: "x" })).rejects.toThrow("channel ID"); + await expect(t("delete_message", { team_id: TEAM, channel_id: CHANNEL, message_id: "../x" })).rejects.toThrow("message ID"); + await expect(t("post_message", { team_id: TEAM, channel_id: CHANNEL, content: "x", content_type: "markdown" })).rejects.toThrow("content type"); + expect(b.calls).toHaveLength(n); + }); + + it("declares the permission each tool needs", () => { + const { c } = make(); + const reg = Registry.from(c.manifests); + expect(reg.get("teams.list_channels")?.permission).toBe("teams:channels:read"); + expect(reg.get("teams.post_message")?.permission).toBe("teams:messages:write"); + expect(reg.get("teams.delete_message")?.permission).toBe("teams:messages:delete"); + expect(reg.get("teams.post_message")?.description).toContain("ChannelMessage.Send"); + expect(reg.get("teams.delete_message")?.description).toContain("ChannelMessage.ReadWrite"); + expect(reg.get("teams.list_channels")?.description).toContain("Channel.ReadBasic.All"); + }); +}); + +describe("linear", () => { + const make = () => { + const b = fakeLinear(); + const c = createLinearConnector({ apiKey: KEY, fetch: b.fetch }); + return { b, c, t: (n: string, a: Record) => c.tools[`linear.${n}`]!(a, {}) }; + }; + const body = (b: { calls: Array<{ body: string }> }, i: number) => JSON.parse(b.calls[i]!.body) as { query: string; variables: Record }; + + it("posts GraphQL with the key as the whole Authorization value", async () => { + const { b, t } = make(); + await t("create_issue", { team_id: LIN_TEAM, title: "Bug", description: "d", priority: 2 }); + const c = b.calls[0]!; + expect(c.url).toBe("https://api.linear.app/graphql"); + expect(c.headers.authorization).toBe(KEY); + expect(body(b, 0).query).toContain("issueCreate(input: $input)"); + expect(body(b, 0).variables).toEqual({ input: { teamId: LIN_TEAM, title: "Bug", description: "d", priority: 2 } }); + }); + + it("archives with the issueArchive mutation only when the issue is not already archived", async () => { + const { b, t } = make(); + b.issues.set(ISSUE, { title: "x", archived: false }); + expect(await t("archive_issue", { issue_id: ISSUE })).toMatchObject({ archived: true }); + expect(body(b, 1).query).toContain("issueArchive(id: $id)"); + expect(body(b, 1).variables).toEqual({ id: ISSUE }); + const n = b.calls.length; + expect(await t("archive_issue", { issue_id: ISSUE })).toMatchObject({ archived: false }); + expect(b.calls.length).toBe(n + 1); // a read, no second mutation + expect(await t("archive_issue", { issue_id: "ffffffff-0000-4000-8000-000000000009" })).toMatchObject({ archived: false }); + }); + + it("surfaces GraphQL errors returned with HTTP 200, without the key", async () => { + const b = fakeLinear(); + const c = createLinearConnector({ + apiKey: KEY, + fetch: (async () => new Response(JSON.stringify({ errors: [{ message: `Argument Validation Error ${KEY}` }] }), { status: 200 })) as typeof fetch, + }); + const err = await c.tools["linear.create_issue"]!({ team_id: LIN_TEAM, title: "x" }, {}).catch((e) => e); + expect(err.message).toContain("Argument Validation Error"); + expect(err.message).not.toContain(KEY); + void b; + }); + + it("validates ids and accepts identifiers for reads", async () => { + const { b, t } = make(); + await expect(t("get_issue", { id: "not an id" })).rejects.toThrow("issue id or identifier"); + await expect(t("add_comment", { issue_id: "ENG-1", body: "x" })).rejects.toThrow("issue UUID"); + await expect(t("create_issue", { team_id: LIN_TEAM, title: "x", priority: 9 })).rejects.toThrow("between 1 and 4"); + expect(b.calls).toHaveLength(0); + b.issues.set("ENG-1", { title: "viaIdentifier", archived: false }); + expect(await t("get_issue", { id: "ENG-1" })).toMatchObject({ title: "viaIdentifier" }); + }); +}); + +describe("jira", () => { + const make = () => { + const b = fakeJira(); + const c = createJiraConnector({ baseUrl: `${BASE}/`, email: EMAIL, apiToken: JIRA_TOKEN, fetch: b.fetch }); + return { b, c, t: (n: string, a: Record) => c.tools[`jira.${n}`]!(a, {}) }; + }; + + it("uses REST v3 with basic auth and sends descriptions as Atlassian Document Format", async () => { + const { b, t } = make(); + await t("create_issue", { project_key: "PROJ", summary: "S", issue_type: "Bug", description: "a\nb" }); + const c = b.calls[0]!; + expect(c.url).toBe(`${BASE}/rest/api/3/issue`); + expect(c.headers.authorization).toBe(`Basic ${BASIC}`); + expect(JSON.parse(c.body)).toEqual({ + fields: { + project: { key: "PROJ" }, + summary: "S", + issuetype: { name: "Bug" }, + description: { + type: "doc", + version: 1, + content: [ + { type: "paragraph", content: [{ type: "text", text: "a" }] }, + { type: "paragraph", content: [{ type: "text", text: "b" }] }, + ], + }, + }, + }); + }); + + it("reads the status first, transitions, and returns the previous status", async () => { + const { b, t } = make(); + b.issues.set("PROJ-9", { summary: "s", status: "1" }); + for (const [transition, to] of [["21", "In Progress"], ["Finish", "Done"], ["To Do", "To Do"]] as const) { + const r = await t("transition_issue", { issue_key: "PROJ-9", transition }); + expect(r).toMatchObject({ status: to }); + } + const first = b.calls.slice(0, 3).map((c) => `${c.method} ${new URL(c.url).pathname.replace("/rest/api/3", "")}`); + expect(first).toEqual(["GET /issue/PROJ-9", "GET /issue/PROJ-9/transitions", "POST /issue/PROJ-9/transitions"]); + await expect(t("transition_issue", { issue_key: "PROJ-9", transition: "Teleport" })).rejects.toThrow("no transition"); + }); + + it("rolls a transition back to the previous status through a real engine", async () => { + const { b, c } = make(); + b.issues.set("PROJ-9", { summary: "s", status: "2" }); + const { call, run } = engine([c]); + await call("jira.transition_issue", { issue_key: "PROJ-9", transition: "Finish" }); + expect(b.issues.get("PROJ-9")!.status).toBe("3"); + const report = await run.rollback({ compensate: compensatorFor([c]) }); + expect(report.complete).toBe(true); + expect(b.issues.get("PROJ-9")!.status).toBe("2"); + }); + + it("restore_status fails clearly when the workflow has no way back", async () => { + const { b, t } = make(); + b.issues.set("PROJ-9", { summary: "s", status: "1" }); + await expect(t("restore_status", { issue_key: "PROJ-9", status_id: "99" })).rejects.toThrow("no transition"); + expect(await t("restore_status", { issue_key: "PROJ-9", status_id: "1" })).toMatchObject({ moved: false }); + }); + + it("validates keys and the base URL", async () => { + const { b, t } = make(); + await expect(t("get_issue", { issue_key: "proj-1" })).rejects.toThrow("issue key"); + await expect(t("delete_issue", { issue_key: "PROJ-1/../../x" })).rejects.toThrow("issue key"); + await expect(t("delete_comment", { issue_key: "PROJ-1", comment_id: "abc" })).rejects.toThrow("comment id"); + await expect(t("create_issue", { project_key: "p", summary: "x" })).rejects.toThrow("project key"); + expect(b.calls).toHaveLength(0); + expect(() => createJiraConnector({ baseUrl: "https://u:p@x.atlassian.net", email: "a", apiToken: "b" })).toThrow("without credentials"); + expect(() => createJiraConnector({ baseUrl: "ftp://x", email: "a", apiToken: "b" })).toThrow("http(s)"); + expect(() => createJiraConnector({ baseUrl: BASE, email: "", apiToken: "b" })).toThrow("email"); + }); + + it("scrubs both the token and the encoded basic credential from errors", async () => { + const c = createJiraConnector({ + baseUrl: BASE, + email: EMAIL, + apiToken: JIRA_TOKEN, + fetch: (async () => new Response(JSON.stringify({ errorMessages: [`bad ${JIRA_TOKEN} and ${BASIC}`] }), { status: 400 })) as typeof fetch, + }); + const err = await c.tools["jira.get_issue"]!({ issue_key: "PROJ-1" }, {}).catch((e) => e); + expect(err.message).toContain("status 400"); + expect(err.message).not.toContain(JIRA_TOKEN); + expect(err.message).not.toContain(BASIC); + }); +}); + +describe("gcalendar", () => { + const make = () => { + const b = fakeCalendar(); + const c = createGcalendarConnector({ token: CAL_TOKEN, fetch: b.fetch }); + return { b, c, t: (n: string, a: Record) => c.tools[`gcalendar.${n}`]!(a, {}) }; + }; + + it("never notifies attendees and defaults to the primary calendar", async () => { + const { b, t } = make(); + await t("create_event", { summary: "Trip", start: "2026-06-01", end: "2026-06-03" }); + const c = b.calls[0]!; + expect(c.url).toBe("https://www.googleapis.com/calendar/v3/calendars/primary/events?sendUpdates=none"); + expect(c.headers.authorization).toBe(`Bearer ${CAL_TOKEN}`); + expect(JSON.parse(c.body)).toEqual({ summary: "Trip", start: { date: "2026-06-01" }, end: { date: "2026-06-03" } }); + await t("delete_event", { event_id: "ev1" }); + expect(b.calls[1]!.url).toBe("https://www.googleapis.com/calendar/v3/calendars/primary/events/ev1?sendUpdates=none"); + }); + + it("update_event reads first and returns exactly the previous values of the fields it changed", async () => { + const { b, t } = make(); + b.events.set("ev9", { id: "ev9", summary: "Standup", description: "keep me", start: { dateTime: "2026-05-01T09:00:00Z" }, end: { dateTime: "2026-05-01T09:15:00Z" } }); + const r = await t("update_event", { event_id: "ev9", summary: "Planning", location: "Room 1" }); + expect(r).toMatchObject({ previous: { summary: "Standup", location: null } }); + expect(Object.keys((r as { previous: object }).previous).sort()).toEqual(["location", "summary"]); + expect(b.calls.map((c) => c.method)).toEqual(["GET", "PATCH"]); + expect(JSON.parse(b.calls[1]!.body)).toEqual({ summary: "Planning", location: "Room 1" }); + expect(b.events.get("ev9")!.description).toBe("keep me"); + await t("restore_event", { event_id: "ev9", previous: (r as { previous: object }).previous }); + expect(b.events.get("ev9")).toMatchObject({ summary: "Standup", description: "keep me" }); + expect(b.events.get("ev9")!.location).toBeUndefined(); + }); + + it("moving between a date-time and an all-day date clears the other key", async () => { + const { b, t } = make(); + b.events.set("ev9", { id: "ev9", summary: "S", start: { dateTime: "2026-05-01T09:00:00Z" }, end: { dateTime: "2026-05-01T10:00:00Z" } }); + const r = (await t("update_event", { event_id: "ev9", start: "2026-05-02", end: "2026-05-03" })) as { previous: object }; + expect(b.events.get("ev9")).toMatchObject({ start: { date: "2026-05-02" }, end: { date: "2026-05-03" } }); + expect(b.events.get("ev9")!.start.dateTime).toBeUndefined(); + await t("restore_event", { event_id: "ev9", previous: r.previous }); + expect(b.events.get("ev9")).toMatchObject({ start: { dateTime: "2026-05-01T09:00:00Z" }, end: { dateTime: "2026-05-01T10:00:00Z" } }); + expect(b.events.get("ev9")!.start.date).toBeUndefined(); + }); + + it("validates arguments", async () => { + const { b, t } = make(); + await expect(t("create_event", { summary: "x", start: "tomorrow", end: "2026-06-01" })).rejects.toThrow("RFC 3339"); + await expect(t("update_event", { event_id: "ev1" })).rejects.toThrow("nothing to update"); + await expect(t("restore_event", { event_id: "ev1", previous: { attendees: [] } })).rejects.toThrow("not a restorable field"); + await expect(t("restore_event", { event_id: "ev1", previous: {} })).rejects.toThrow("no fields"); + await expect(t("delete_event", { event_id: "a/b" })).rejects.toThrow("event id"); + await expect(t("list_events", { max_results: 999 })).rejects.toThrow("at most 250"); + expect(b.calls).toHaveLength(0); + }); +}); diff --git a/packages/gateway/package.json b/packages/gateway/package.json index 27d2a93..e91aec0 100644 --- a/packages/gateway/package.json +++ b/packages/gateway/package.json @@ -28,8 +28,10 @@ "node": ">=22" }, "dependencies": { + "@hono/node-server": "^1.19.0", "@modelcontextprotocol/sdk": "^1.32.1", "@multimindlab/reversi": "0.1.0", + "@multimindlab/reversi-server": "0.1.0", "yaml": "^2.8.0" } } diff --git a/packages/gateway/src/config.ts b/packages/gateway/src/config.ts index 377a59d..bfb58a7 100644 --- a/packages/gateway/src/config.ts +++ b/packages/gateway/src/config.ts @@ -29,7 +29,15 @@ export interface OidcConfig { export interface GatewayConfig { /** `observe` journals and never blocks; `enforce` blocks. Default enforce. */ - mode: "observe" | "enforce"; + mode: "observe" | "enforce" | "advise"; + /** Whether the file or flags named a mode. Without one, a gateway that has no manifests yet starts in observe mode. */ + mode_explicit?: boolean; + /** Tools with no manifest that the server calls read-only: `block` (default) or `observe` (run, journal as unclassified). */ + unclassified_reads?: "block" | "observe"; + /** Check arguments and results against each tool's declared schemas. Default true. */ + validate_schemas?: boolean; + /** The operator console on the gateway's own port. `off` (default), `open` (no login, loopback only, fixed role) or `login`. */ + console?: { mode: "off" | "open" | "login"; role?: Role }; /** Actor recorded for unauthenticated sessions (stdio, or HTTP on loopback with no auth). */ actor: string; /** Permissions for each session's scope. Default: every permission in the manifests. */ @@ -70,10 +78,21 @@ export function expandEnv(s: string, env: Record = p export function parseGatewayConfig(raw: unknown): GatewayConfig { const errors: string[] = []; if (!isRec(raw)) throw new ConfigError(["config must be an object"]); - const known = ["mode", "actor", "permissions", "upstreams", "allow", "deny", "listen", "resource_url", "allowed_hosts", "allowed_origins", "auth"]; + const known = ["mode", "unclassified_reads", "validate_schemas", "console", "actor", "permissions", "upstreams", "allow", "deny", "listen", "resource_url", "allowed_hosts", "allowed_origins", "auth"]; for (const k of Object.keys(raw)) if (!known.includes(k)) errors.push(`unknown field \`${k}\``); const mode = raw.mode ?? "enforce"; - if (mode !== "observe" && mode !== "enforce") errors.push("`mode` must be observe or enforce"); + if (mode !== "observe" && mode !== "enforce" && mode !== "advise") errors.push("`mode` must be observe, advise or enforce"); + if (raw.unclassified_reads !== undefined && raw.unclassified_reads !== "block" && raw.unclassified_reads !== "observe") errors.push("`unclassified_reads` must be block or observe"); + if (raw.validate_schemas !== undefined && typeof raw.validate_schemas !== "boolean") errors.push("`validate_schemas` must be true or false"); + let consoleCfg: GatewayConfig["console"]; + if (raw.console !== undefined) { + const c = raw.console; + if (!isRec(c) || !["off", "open", "login"].includes(c.mode as string)) errors.push("`console.mode` must be off, open or login"); + else { + if (c.role !== undefined && !["auditor", "approver", "operator"].includes(c.role as string)) errors.push("`console.role` must be auditor, approver or operator (the open console is never admin)"); + consoleCfg = { mode: c.mode as "off" | "open" | "login", ...(c.role ? { role: c.role as Role } : {}) }; + } + } if (raw.actor !== undefined && (typeof raw.actor !== "string" || !raw.actor)) errors.push("`actor` must be a non-empty string"); for (const k of ["permissions", "allow", "deny", "allowed_hosts", "allowed_origins"]) { if (raw[k] !== undefined && !strList(raw[k])) errors.push(`\`${k}\` must be a list of strings`); @@ -134,6 +153,10 @@ export function parseGatewayConfig(raw: unknown): GatewayConfig { if (errors.length) throw new ConfigError(errors); return { mode: mode as GatewayConfig["mode"], + mode_explicit: raw.mode !== undefined, + unclassified_reads: raw.unclassified_reads as GatewayConfig["unclassified_reads"], + validate_schemas: raw.validate_schemas as boolean | undefined, + console: consoleCfg, actor: (raw.actor as string | undefined) ?? "mcp-client", permissions: raw.permissions as string[] | undefined, upstreams, diff --git a/packages/gateway/src/gateway.ts b/packages/gateway/src/gateway.ts index b0c7891..3320734 100644 --- a/packages/gateway/src/gateway.ts +++ b/packages/gateway/src/gateway.ts @@ -19,10 +19,20 @@ import { } from "@modelcontextprotocol/sdk/types.js"; import { SealedScope, + banner, can, + classifyTools, createBundle, globAny, + hintsOf, + previewImpact, + schemaHash, + searchRecords, + searchRuns, + suggestManifests, validateManifest, + type McpToolInfo, + type ToolManifest, type Approvals, type Capability, type Outcome, @@ -32,6 +42,7 @@ import { type SigningKey, toJsonl, } from "@multimindlab/reversi"; +import { validateSchema } from "./schema.js"; import { UnknownToolError, type UpstreamPool } from "./upstream.js"; export const REASON_META_KEY = "reversi/reason"; @@ -41,7 +52,16 @@ export interface GatewayOptions { reversi: Reversi; approvals: Approvals; pool: UpstreamPool; - mode: "observe" | "enforce"; + mode: "observe" | "enforce" | "advise"; + /** What to do with a tool that has no manifest but whose server says it only reads. `observe` lets it run and journals it as unclassified; `block` (default) treats it like any unclassified tool. */ + unclassified_reads?: "block" | "observe"; + /** Check arguments and results against the upstream tool's declared schemas. Default true. */ + validate_schemas?: boolean; + /** Re-reads manifests and policy (the engine's reload); used by the operator tool and the hot reload. */ + reload?: (by: string) => Promise<{ changed: boolean; tools: number; error?: string }>; + /** Capture store, for the simulate tool. */ + capture?: import("@multimindlab/reversi").CaptureStore; + environment?: string; /** Permissions for each session's scope. Default: every permission named in the manifests. */ permissions?: string[]; /** Hide tools from `tools/list` that the allow/deny lists forbid. */ @@ -67,7 +87,7 @@ export function resultForHandle(result: object): unknown { } /** Errors that say the call was refused are failures; anything else may have taken effect. */ -function outcomeForError(e: unknown): Outcome { +function outcomeForError(e: unknown, _manifest?: ToolManifest): Outcome { const error = e instanceof Error ? e.message : String(e); if (e instanceof UnknownToolError) return { status: "failed", error }; // never reached any upstream const refused = e instanceof McpError && (e.code === ErrorCode.InvalidParams || e.code === ErrorCode.MethodNotFound); @@ -139,6 +159,22 @@ export class Gateway { }); } + /** Server-claimed read-only tools without a manifest run in a shadow observe-mode run, so they are journaled and never block. */ + private readRunFor(principal: Principal, main: Run): Run { + const id = `${main.run_id}-reads`; + const existing = (() => { + try { + return this.o.reversi.openRun(id); + } catch { + return null; + } + })(); + return ( + existing ?? + this.o.reversi.startRun({ run_id: id, actor: principal.name, scope: SealedScope.seal(this.scopeFor(principal)), mode: "observe", reason: `unclassified reads of ${main.run_id}` }) + ); + } + private scopeFor(p: Principal): string[] { const all = this.o.permissions ?? [...new Set(this.o.reversi.registry.list().map((m) => m.permission))]; return p.max_permissions ? all.filter((x) => p.max_permissions?.includes(x)) : all; @@ -176,7 +212,50 @@ export class Gateway { return r.ok ? ok({ valid: true, manifest: r.manifest }) : ok({ valid: false, errors: r.errors }); }, }, + { + name: "reversi.preview_impact", + description: "Before calling a tool: would the call be allowed, need approval or be blocked, and could it be undone? Executes nothing.", + inputSchema: obj({ tool: { type: "string" }, arguments: { type: "object" } }, ["tool"]), + ...read, + run: async (a, ctx) => ok(await previewImpact({ registry: rv.registry, policy: rv.policy?.doc, environment: this.o.environment }, { tool: String(a.tool), args: a.arguments, actor: ctx.who.name, tenant: ctx.who.tenant })), + }, + { + name: "reversi.suggest_manifest", + description: "Draft manifests for upstream tools that have none, from their names, schemas and annotations. A draft needs a person's approval before it is enforced.", + inputSchema: obj(), + ...read, + run: async () => { + const r = suggestManifests(await this.toolInfos(), { server: "upstream", existing: rv.registry }); + return ok({ suggestions: r.suggestions.map((x) => ({ manifest: x.manifest, confidence: x.confidence, notes: x.notes })), skipped: r.skipped }); + }, + }, + { + name: "reversi.classification", + description: "Which upstream tools are classified by a manifest and which are not.", + inputSchema: obj(), + ...read, + run: async () => ok(await this.classification()), + }, + { + name: "reversi.search", + description: "Find runs by actor, tool, status or time, or ledger records by event.", + inputSchema: obj({ what: { type: "string", enum: ["runs", "records"] }, actor: { type: "string" }, tool: { type: "string" }, status: { type: "string" }, text: { type: "string" }, limit: { type: "number" } }), + ...read, + run: (a, ctx) => { + const limit = typeof a.limit === "number" ? Math.min(a.limit, 200) : 50; + const actor = ctx.own_only ? ctx.who.name : typeof a.actor === "string" ? a.actor : undefined; + if (a.what === "records") return ok(searchRecords(rv, { actor, tool: typeof a.tool === "string" ? a.tool : undefined, text: typeof a.text === "string" ? a.text : undefined, limit })); + return ok(searchRuns(rv, { actor, tool: typeof a.tool === "string" ? a.tool : undefined, status: a.status as never, text: typeof a.text === "string" ? a.text : undefined, limit })); + }, + }, // ---- operator only (never listed on the agent endpoint) ---- + { + name: "reversi.reload", + description: "Re-read manifests and policy now. A configuration that fails to load is rejected and the old one stays.", + inputSchema: obj(), + cap: "halt", + run: async (_a, ctx) => (this.o.reload ? ok(await this.o.reload(ctx.who.name)) : fail("this gateway has no reloadable configuration")), + }, { name: "reversi.list_approvals", description: "Approval requests, optionally by status.", inputSchema: obj({ status: { type: "string", enum: ["pending", "approved", "denied", "consumed", "expired"] } }), ...read, run: (a) => ok({ approvals: approvals.list({ status: a.status as never }) }) }, { name: "reversi.approve_action", @@ -209,6 +288,11 @@ export class Gateway { } const report = await run.rollback({ actor: ctx.who.name, + verifier: async (tool, vargs) => { + const r = await this.o.pool.call(tool, vargs); + if (r.isError) throw new Error(`read-back ${tool} returned an error`); + return resultForHandle(r); + }, retry: { attempts: 3, base_ms: 250 }, compensate: async (call) => { const r = await this.o.pool.call(call.tool, call.args, { [IDEMPOTENCY_META_KEY]: call.idempotency_key, "reversi/run_id": call.run_id, "reversi/step": call.step }); @@ -238,7 +322,7 @@ export class Gateway { ]; } - private static readonly AGENT_TOOLS = new Set(["reversi.list_runs", "reversi.get_run", "reversi.get_action_trace", "reversi.plan_rollback", "reversi.dry_run_rollback", "reversi.validate_manifest"]); + private static readonly AGENT_TOOLS = new Set(["reversi.list_runs", "reversi.get_run", "reversi.get_action_trace", "reversi.plan_rollback", "reversi.dry_run_rollback", "reversi.validate_manifest", "reversi.preview_impact", "reversi.search"]); private async upstreamTools(): Promise { if (this.cache && Date.now() - this.cache.at < 5_000) return this.cache.tools; @@ -247,6 +331,63 @@ export class Gateway { return tools; } + /** What the upstream says about each tool, as MCP tool info (annotations, schemas). */ + async toolInfos(): Promise { + return (await this.upstreamTools()) as unknown as McpToolInfo[]; + } + + /** Every tool the upstreams offer, classified against the manifests, with the startup banner text. */ + async classification(): Promise<{ rows: ReturnType; banner: string }> { + const rows = classifyTools(this.o.reversi.registry, await this.toolInfos()); + const unclassified = rows.filter((r) => r.class !== "classified").length; + const next = unclassified + ? this.o.mode === "enforce" + ? `${unclassified} tools have no manifest and are blocked. Draft them with: reversi suggest-manifest, review, then reversi manifests publish.` + : `${unclassified} tools have no manifest. They run (nothing is blocked) and are journaled as unclassified. Draft manifests with: reversi suggest-manifest.` + : undefined; + return { rows, banner: banner(rows, { mode: this.o.mode, next }) }; + } + + /** One row per upstream for the console: health, how many of its tools have manifests. */ + async connectors(): Promise> { + const status = await this.o.pool.status(); + await this.upstreamTools().catch(() => undefined); + const owners = this.o.pool.owners(); + return this.o.pool.entries.map((e) => { + const mine = [...owners].filter(([, up]) => up === e.name).map(([tool]) => tool); + const classified = mine.filter((t) => this.o.reversi.registry.get(t)).length; + return { name: e.name, kind: e.kind, healthy: status.find((x) => x.name === e.name)?.ok ?? false, tools: mine.length, classified, unclassified: mine.length - classified, credential: "n/a" as const }; + }); + } + + /** Call the upstream, with the manifest's timeout, and its retries only when repeating the call is safe (reads and idempotent tools). */ + private async forward(name: string, args: Record | undefined, manifest: ToolManifest | undefined): Promise { + const safe = manifest?.effect === "read_only" || manifest?.effect === "idempotent"; + const attempts = safe ? Math.max(1, manifest?.retry?.attempts ?? 1) : 1; + let last: unknown; + for (let i = 0; i < attempts; i++) { + try { + const call = this.o.pool.call(name, args); + if (!manifest?.timeout_ms) return await call; + let timer: NodeJS.Timeout | undefined; + const timeout = new Promise((_r, rej) => { + timer = setTimeout(() => rej(new Error(`upstream did not answer within ${manifest.timeout_ms} ms`)), manifest.timeout_ms); + }); + try { + return await Promise.race([call, timeout]); + } finally { + clearTimeout(timer); + call.catch(() => undefined); + } + } catch (e) { + last = e; + if (e instanceof UnknownToolError) throw e; + if (i + 1 < attempts) await new Promise((r) => setTimeout(r, Math.min(manifest?.retry?.max_ms ?? 5000, (manifest?.retry?.base_ms ?? 200) * 2 ** i))); + } + } + throw last; + } + /** * A new MCP session. `operator: false` is the agent endpoint: one run, upstream tools, read-only * reversi tools. `operator: true` exposes only the operator tools, limited by the principal's roles. @@ -255,6 +396,7 @@ export class Gateway { const { principal } = opts; const server = new Server({ name: "reversi-gateway", version: "0.1.0" }, { capabilities: { tools: {} } }); const rtools = this.tools().filter((t) => (opts.operator ? can(principal, t.cap) : Gateway.AGENT_TOOLS.has(t.name))); + let readRun: Run | undefined; const ctx: Ctx = { who: principal, own_only: !opts.operator }; const run = opts.operator ? null : this.runFor(principal, opts.run_id); @@ -277,20 +419,43 @@ export class Gateway { const metaReason = _meta?.[REASON_META_KEY]; const reason = typeof metaReason === "string" && metaReason.trim() ? metaReason : `MCP tools/call ${name}`; - const begin = await run.begin({ tool: name, args, reason }); + + const infos = await this.toolInfos().catch(() => [] as McpToolInfo[]); + const info = infos.find((t) => t.name === name); + const manifest = this.o.reversi.registry.get(name); + const extra: string[] = []; + if (this.o.validate_schemas !== false && info?.inputSchema) { + const problems = validateSchema(info.inputSchema, args ?? {}); + if (problems.length) extra.push(`arguments do not match the tool's input schema: ${problems.slice(0, 5).join("; ")}`); + } + if (manifest?.schema_hash && info && schemaHash(info) !== manifest.schema_hash) { + extra.push("the tool's input schema changed since its manifest was approved: review the manifest again"); + } + + // A tool without a manifest that the server calls read-only may run in a shadow observe run, if the operator allowed that. + const shadow = !manifest && this.o.unclassified_reads === "observe" && info !== undefined && hintsOf(info).read_only === true && run.state().mode === "enforce"; + const active = shadow ? (readRun ??= this.readRunFor(principal, run)) : run; + const begin = await active.begin({ tool: name, args, reason, ...(extra.length ? { extra_violations: extra } : {}), ...(shadow ? { extra_notes: ["unclassified read: the server says it is read-only; this is unverified"] } : {}) }); if (begin.decision === "block") { return fail(`Blocked by Reversi: ${begin.reasons.join("; ")}`, { reversi: { blocked: true, reasons: begin.reasons } }); } let result: CallToolResult; try { - result = await this.o.pool.call(name, args); + result = await this.forward(name, args, manifest); } catch (e) { - run.complete(begin.step, outcomeForError(e)); + active.complete(begin.step, outcomeForError(e, manifest)); throw e; } - if (result.isError) run.complete(begin.step, { status: "failed", error: "upstream returned isError" }); - else run.complete(begin.step, { status: "succeeded", result: resultForHandle(result) }); + if (result.isError) active.complete(begin.step, { status: "failed", error: "upstream returned isError" }); + else { + const handle = resultForHandle(result); + const bad = this.o.validate_schemas !== false && info?.outputSchema && (result as { structuredContent?: unknown }).structuredContent !== undefined ? validateSchema(info.outputSchema, (result as { structuredContent?: unknown }).structuredContent) : []; + active.complete(begin.step, bad.length ? { status: "succeeded", result: handle, invalid_result: true, notes: [`result does not match the tool's output schema: ${bad.slice(0, 3).join("; ")}`] } : { status: "succeeded", result: handle }); + } + if (begin.decision === "would_block" && this.o.mode === "advise") { + return { ...result, content: [...result.content, { type: "text", text: `Reversi advisory (nothing was blocked): ${begin.reasons.join("; ")}` }] }; + } return result; }); diff --git a/packages/gateway/src/http.ts b/packages/gateway/src/http.ts index d4ab48a..6b17531 100644 --- a/packages/gateway/src/http.ts +++ b/packages/gateway/src/http.ts @@ -30,6 +30,8 @@ export interface HttpGatewayOptions { anonymous_actor: string; allowed_hosts?: string[]; allowed_origins?: string[]; + /** Serves `/console/*` (the server's console, mounted on this port). Its own checks guard it, so the MCP Origin rule does not apply. */ + console?: (req: IncomingMessage, res: ServerResponse) => unknown; /** Idle sessions are closed after this long. Default 30 minutes. */ session_ttl_ms?: number; max_body_bytes?: number; @@ -157,10 +159,11 @@ export function createHttpGateway(o: HttpGatewayOptions): { server: HttpServer; const hostname = host.startsWith("[") ? host.slice(0, host.indexOf("]") + 1) : host.split(":")[0] ?? ""; const hostOk = allowedHosts.size ? allowedHosts.has(host) || allowedHosts.has(hostname) : !isLoopback(o.host) || isLoopback(hostname); if (!hostOk) return json(res, 403, { error: "host not allowed" }); - const origin = req.headers.origin; - if (origin && !allowedOrigins.has(origin)) return json(res, 403, { error: "origin not allowed" }); - const path = (req.url ?? "/").split("?")[0]; + const origin = req.headers.origin; + const isConsole = o.console !== undefined && (path === "/console" || path?.startsWith("/console/")); + if (origin && !isConsole && !allowedOrigins.has(origin)) return json(res, 403, { error: "origin not allowed" }); + if (isConsole) return void (await o.console?.(req, res)); if (path === "/healthz") return json(res, 200, { ok: true }); if (path === "/readyz") { // Unauthenticated, and it pings every upstream: answer from a two-second cache. @@ -180,8 +183,14 @@ export function createHttpGateway(o: HttpGatewayOptions): { server: HttpServer; } catch (e) { const status = (e as { status?: number }).status ?? 500; if (!res.headersSent) { - // An oversized upload is not read to the end: close the connection once the answer is out. - if (status === 413) res.once("finish", () => req.destroy()); + // An oversized upload is answered at once and the rest is read and discarded (never buffered), so the client sees + // the 413 instead of a reset connection. A slow sender is cut off after five seconds. + if (status === 413) { + req.resume(); + const t = setTimeout(() => req.destroy(), 5000); + t.unref(); + req.once("close", () => clearTimeout(t)); + } json(res, status, { error: status === 500 ? "internal error" : (e as Error).message }, status === 413 ? { Connection: "close" } : {}); } if (status === 500) console.error(e); diff --git a/packages/gateway/src/main.ts b/packages/gateway/src/main.ts index fe985d3..20e7e91 100644 --- a/packages/gateway/src/main.ts +++ b/packages/gateway/src/main.ts @@ -1,30 +1,51 @@ #!/usr/bin/env node -import { readFileSync } from "node:fs"; +import { existsSync, readFileSync, readdirSync, statSync } from "node:fs"; import { parseArgs } from "node:util"; +import { getRequestListener } from "@hono/node-server"; import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js"; -import { KeyRing, OidcVerifier, dataOptionsFromEnv, openEngine, storeKindFrom, type ApiKey, type SigningKey } from "@multimindlab/reversi"; +import { + KeyRing, + ManifestRegistry, + OidcVerifier, + PolicyVersions, + checkUrl, + dataOptionsFromEnv, + engineConfigFromEnv, + findProjectConfig, + loadProjectConfig, + openEngine, + storeKindFrom, + type ApiKey, + type ProjectConfig, + type Role, +} from "@multimindlab/reversi"; +import { createApp } from "@multimindlab/reversi-server"; import { ConfigError, allowDenyPolicy, loadGatewayConfig, parseGatewayConfig, type GatewayConfig } from "./config.js"; -import { Gateway } from "./gateway.js"; +import { Gateway, IDEMPOTENCY_META_KEY, resultForHandle } from "./gateway.js"; import { createHttpGateway } from "./http.js"; import { UpstreamPool } from "./upstream.js"; const USAGE = `reversi-gateway: journal, police and roll back an MCP server's tool calls. - --config gateway config (YAML/JSON): upstreams, mode, allow/deny, listen, auth - --manifests tool manifests (file or directory) (required) + --config gateway config (YAML/JSON): upstreams, mode, allow/deny, listen, auth, console + --manifests tool manifests (file, directory, registry or URL). Optional: with none, the gateway starts in observe + mode, lets everything through and journals it, and shows what to classify. ./manifests is used if it exists. --policy policy rules, limits and content checks --data data directory (default ./reversi-data) --store files | sqlite | postgres (default files) --database-url PostgreSQL URL (or REVERSI_DATABASE_URL) --environment environment name matched by policy rules --signing-key Ed25519 private key for signed evidence bundles + --console off (default) | open | login: the operator console on the gateway's own port (needs "listen") + --console-role role in open mode: auditor | approver | operator (default operator; never admin) Shortcuts instead of --config, for one upstream over stdio on stdio: --upstream-cmd [--upstream-arg ]... or --upstream-url - --mode observe|enforce (default enforce) + --mode observe|advise|enforce (default enforce; observe when there are no manifests yet) --actor actor recorded for unauthenticated sessions (default mcp-client) --run-id stdio mode: the run id to use, or continue if it exists (HTTP clients send a Reversi-Run-Id header) +Reversi reads reversi.config.yaml from the current directory or above (or REVERSI_CONFIG) for everything else. Without a "listen" section the gateway speaks MCP on stdio, so logs go to stderr.`; const { values: v } = parseArgs({ @@ -37,6 +58,8 @@ const { values: v } = parseArgs({ "database-url": { type: "string" }, environment: { type: "string" }, "signing-key": { type: "string" }, + console: { type: "string" }, + "console-role": { type: "string" }, "upstream-cmd": { type: "string" }, "upstream-arg": { type: "string", multiple: true }, "upstream-url": { type: "string" }, @@ -57,7 +80,6 @@ if (v.help) { console.error(USAGE); process.exit(0); } -if (!v.manifests) die("--manifests is required"); let cfg: GatewayConfig; try { @@ -65,53 +87,131 @@ try { else { if (!!v["upstream-cmd"] === !!v["upstream-url"]) die("give --config, or exactly one of --upstream-cmd / --upstream-url"); cfg = parseGatewayConfig({ - mode: v.mode ?? "enforce", + ...(v.mode ? { mode: v.mode } : {}), actor: v.actor, upstreams: [v["upstream-url"] ? { name: "upstream", url: v["upstream-url"] } : { name: "upstream", command: v["upstream-cmd"], args: v["upstream-arg"] ?? [] }], }); } + if (v.mode && v.config) cfg = { ...cfg, mode: v.mode as GatewayConfig["mode"], mode_explicit: true }; } catch (e) { if (e instanceof ConfigError) die(e.errors.join("\n ")); throw e; } -const dataOpts = dataOptionsFromEnv(); +const env = process.env; +let project: ProjectConfig | undefined; +try { + const file = findProjectConfig(process.cwd(), undefined, env); + project = file ? loadProjectConfig(file, env) : undefined; +} catch (e) { + die(e instanceof Error ? e.message : String(e)); +} + +const dataOpts = dataOptionsFromEnv({ ...(project?.data?.dir ? { REVERSI_DATA: project.data.dir } : {}), ...(project?.data?.store ? { REVERSI_STORE: project.data.store } : {}), ...(project?.data?.database_url ? { REVERSI_DATABASE_URL: project.data.database_url } : {}), ...env }); if (v.data) dataOpts.dir = v.data; if (v.store) dataOpts.store = storeKindFrom(v.store); if (v["database-url"]) dataOpts.postgres_url = v["database-url"]; -const engine = await openEngine({ - manifests: v.manifests, - data: dataOpts, - policy: v.policy ?? process.env.REVERSI_POLICY, - environment: v.environment ?? process.env.REVERSI_ENV, - kill_switch_file: process.env.REVERSI_KILL_FILE, - approval_webhook: process.env.REVERSI_APPROVAL_WEBHOOK, - console_url: process.env.REVERSI_CONSOLE_URL, +const manifests = v.manifests ?? env.REVERSI_MANIFESTS ?? project?.manifests ?? (existsSync("./manifests") ? "./manifests" : undefined); +const base = engineConfigFromEnv({ ...env, REVERSI_MANIFESTS: manifests ?? "-" }, dataOpts, project); +let engine: Awaited>; +try { + engine = await openEngine({ + ...base, + manifests: manifests ?? "", + allow_empty_manifests: !manifests, + policy: v.policy ?? base.policy, + environment: v.environment ?? base.environment, + signing_key: v["signing-key"] ?? base.signing_key, extra_policy: allowDenyPolicy(cfg), }); +} catch (e) { + die(e instanceof Error ? e.message : String(e)); +} -let signing_key: SigningKey | undefined; -if (v["signing-key"]) { - const { createPrivateKey, createPublicKey } = await import("node:crypto"); - const private_pem = readFileSync(v["signing-key"], "utf8"); - signing_key = { private_pem, public_pem: createPublicKey(createPrivateKey(private_pem)).export({ type: "spki", format: "pem" }).toString() }; +// Observe-first: no manifests and no explicit mode means nothing is blocked while the picture builds up. +let mode = cfg.mode; +if (engine.registry.size === 0 && !cfg.mode_explicit) { + mode = "observe"; + console.error("reversi-gateway: no manifests yet, so starting in observe mode: everything runs and is journaled. Run `reversi suggest-manifest` to draft manifests for what the upstream offers."); } +for (const u of cfg.upstreams) if (u.url) await checkUrl(engine.egress, u.url).catch((e) => die(`upstream ${u.name}: ${e instanceof Error ? e.message : e}`)); const pool = await UpstreamPool.connect(cfg.upstreams); -const gateway = new Gateway({ reversi: engine.reversi, approvals: engine.approvals, pool, mode: cfg.mode, permissions: cfg.permissions, allow: cfg.allow, deny: cfg.deny, signing_key }); +const gateway = new Gateway({ + reversi: engine.reversi, + approvals: engine.approvals, + pool, + mode, + permissions: cfg.permissions, + allow: cfg.allow, + deny: cfg.deny, + signing_key: engine.signing_key, + unclassified_reads: cfg.unclassified_reads, + validate_schemas: cfg.validate_schemas, + reload: (by) => engine.reload(by), + capture: engine.capture, + environment: engine.config.environment, +}); + +try { + console.error((await gateway.classification()).banner); +} catch (e) { + console.error(`reversi-gateway: could not list the upstream tools: ${e instanceof Error ? e.message : e}`); +} + +engine.start({ health: () => pool.status() }); +if (project?.reload?.watch !== false) engine.watch(project?.reload?.interval_ms); +process.on("SIGHUP", () => void engine.reload("sighup")); const shutdown = async (code = 0) => { await pool.close(); - engine.data.close(); + await engine.close(); process.exit(code); }; process.on("SIGINT", () => void shutdown()); process.on("SIGTERM", () => void shutdown()); +const dirOf = (p: string | undefined, versioned: (d: string) => boolean) => (p && existsSync(p) && statSync(p).isDirectory() && (versioned(p) || readdirSync(p).length === 0) ? p : undefined); + if (cfg.listen) { const keys = cfg.auth?.keys ? new KeyRing(JSON.parse(readFileSync(cfg.auth.keys, "utf8")) as ApiKey[]) : undefined; const oidc = cfg.auth?.oidc ? new OidcVerifier({ ...cfg.auth.oidc }) : undefined; + + const consoleMode = (v.console ?? cfg.console?.mode ?? project?.console?.mode ?? "off") as "off" | "open" | "login"; + const consoleRole = (v["console-role"] ?? cfg.console?.role ?? project?.console?.role ?? "operator") as Role; + if (!["off", "open", "login"].includes(consoleMode)) die("--console must be off, open or login"); + if (consoleMode === "open") { + if (!["127.0.0.1", "::1", "localhost"].includes(cfg.listen.host)) die(`the open console has no login, so the gateway must listen on loopback, not ${cfg.listen.host}. Use --console login.`); + if (consoleRole === "admin") die("the open console never runs as admin"); + } + if (consoleMode === "login" && !keys && !oidc) die("--console login needs auth.keys or auth.oidc to log in with"); + const consoleApp = + consoleMode === "off" + ? undefined + : createApp({ + reversi: engine.reversi, + approvals: engine.approvals, + keys: keys ?? [], + oidc, + engine, + signing_key: engine.signing_key, + console: { mode: consoleMode, role: consoleRole as never }, + secure_cookies: env.REVERSI_SECURE_COOKIES === "1", + session_secret: env.REVERSI_SESSION_SECRET, + registry_dir: dirOf(engine.config.manifests, (d) => ManifestRegistry.isRegistry(d)), + policy_dir: dirOf(engine.config.policy, (d) => PolicyVersions.isVersioned(d)), + compensator: async (call) => { + const r = await pool.call(call.tool, call.args, { [IDEMPOTENCY_META_KEY]: call.idempotency_key, "reversi/run_id": call.run_id, "reversi/step": call.step }); + if (r.isError) throw new Error(`compensation ${call.tool} returned an error`); + }, + verifier: async (tool, a) => resultForHandle(await pool.call(tool, a)), + connectors: () => gateway.connectors(), + upstream_tools: () => gateway.toolInfos(), + health: () => pool.status(), + }); + const listener = consoleApp ? getRequestListener(consoleApp.fetch) : undefined; + const http = createHttpGateway({ gateway, pool, @@ -123,13 +223,16 @@ if (cfg.listen) { anonymous_actor: cfg.actor, allowed_hosts: cfg.allowed_hosts, allowed_origins: cfg.allowed_origins, + console: listener, }); http.server.listen(cfg.listen.port, cfg.listen.host, () => { const a = http.server.address(); - console.error(`reversi-gateway: ${cfg.mode} mode, agent endpoint http://${cfg.listen?.host}:${typeof a === "object" && a ? a.port : cfg.listen?.port}/mcp, ${engine.data.kind} store`); + const port = typeof a === "object" && a ? a.port : cfg.listen?.port; + console.error(`reversi-gateway: ${mode} mode, agent endpoint http://${cfg.listen?.host}:${port}/mcp, ${engine.data.kind} store${consoleApp ? `, console http://${cfg.listen?.host}:${port}/console (${consoleMode})` : ""}`); }); } else { + if (v.console && v.console !== "off") die("--console needs an HTTP gateway: add a `listen` section to the config"); const { server, run } = gateway.createSession({ principal: { name: cfg.actor, roles: ["agent"], via: "api_key" }, operator: false, run_id: v["run-id"] }); await server.connect(new StdioServerTransport()); - console.error(`reversi-gateway: run ${run?.run_id}, ${cfg.mode} mode, ${engine.data.kind} store`); + console.error(`reversi-gateway: run ${run?.run_id}, ${mode} mode, ${engine.data.kind} store`); } diff --git a/packages/gateway/src/schema.ts b/packages/gateway/src/schema.ts new file mode 100644 index 0000000..4e9e196 --- /dev/null +++ b/packages/gateway/src/schema.ts @@ -0,0 +1,58 @@ +/** A small JSON Schema checker for MCP tool arguments and results: type, enum, const, required, properties, additionalProperties, items, bounds, pattern, composition. */ +type Schema = Record; + +const typeOf = (v: unknown): string => (v === null ? "null" : Array.isArray(v) ? "array" : Number.isInteger(v) ? "integer" : typeof v); +const isRec = (x: unknown): x is Record => typeof x === "object" && x !== null && !Array.isArray(x); +const same = (a: unknown, b: unknown) => JSON.stringify(a) === JSON.stringify(b); + +export function validateSchema(schema: unknown, value: unknown, path = "$", depth = 0): string[] { + if (!isRec(schema) || depth > 12) return []; + const s = schema as Schema; + const out: string[] = []; + const t = typeOf(value); + + if (s.type !== undefined) { + const allowed = (Array.isArray(s.type) ? s.type : [s.type]) as string[]; + const okType = allowed.some((x) => x === t || (x === "number" && t === "integer")); + if (!okType) return [`${path}: expected ${allowed.join(" or ")}, got ${t}`]; + } + if (Array.isArray(s.enum) && !s.enum.some((e) => same(e, value))) out.push(`${path}: must be one of ${JSON.stringify(s.enum).slice(0, 80)}`); + if ("const" in s && !same(s.const, value)) out.push(`${path}: must equal ${JSON.stringify(s.const).slice(0, 60)}`); + + if (typeof value === "number") { + if (typeof s.minimum === "number" && value < s.minimum) out.push(`${path}: below minimum ${s.minimum}`); + if (typeof s.maximum === "number" && value > s.maximum) out.push(`${path}: above maximum ${s.maximum}`); + if (typeof s.exclusiveMinimum === "number" && value <= s.exclusiveMinimum) out.push(`${path}: must be above ${s.exclusiveMinimum}`); + if (typeof s.exclusiveMaximum === "number" && value >= s.exclusiveMaximum) out.push(`${path}: must be below ${s.exclusiveMaximum}`); + if (typeof s.multipleOf === "number" && s.multipleOf > 0 && Math.abs(value / s.multipleOf - Math.round(value / s.multipleOf)) > 1e-9) out.push(`${path}: must be a multiple of ${s.multipleOf}`); + } + if (typeof value === "string") { + if (typeof s.minLength === "number" && value.length < s.minLength) out.push(`${path}: shorter than ${s.minLength}`); + if (typeof s.maxLength === "number" && value.length > s.maxLength) out.push(`${path}: longer than ${s.maxLength}`); + if (typeof s.pattern === "string") { + try { + if (!new RegExp(s.pattern, "u").test(value)) out.push(`${path}: does not match ${s.pattern}`); + } catch { + /* an unreadable pattern in the upstream's schema is not the caller's fault */ + } + } + } + if (Array.isArray(value)) { + if (typeof s.minItems === "number" && value.length < s.minItems) out.push(`${path}: fewer than ${s.minItems} items`); + if (typeof s.maxItems === "number" && value.length > s.maxItems) out.push(`${path}: more than ${s.maxItems} items`); + if (isRec(s.items)) value.slice(0, 200).forEach((v, i) => out.push(...validateSchema(s.items, v, `${path}[${i}]`, depth + 1))); + } + if (isRec(value)) { + const props = isRec(s.properties) ? s.properties : {}; + for (const r of Array.isArray(s.required) ? (s.required as string[]) : []) if (!(r in value)) out.push(`${path}: missing required \`${r}\``); + for (const [k, v] of Object.entries(value)) { + if (k in props) out.push(...validateSchema(props[k], v, `${path}.${k}`, depth + 1)); + else if (s.additionalProperties === false) out.push(`${path}: unexpected \`${k}\``); + else if (isRec(s.additionalProperties)) out.push(...validateSchema(s.additionalProperties, v, `${path}.${k}`, depth + 1)); + } + } + if (Array.isArray(s.allOf)) for (const sub of s.allOf) out.push(...validateSchema(sub, value, path, depth + 1)); + if (Array.isArray(s.anyOf) && s.anyOf.length && !s.anyOf.some((sub) => validateSchema(sub, value, path, depth + 1).length === 0)) out.push(`${path}: matches none of the allowed shapes`); + if (Array.isArray(s.oneOf) && s.oneOf.length && s.oneOf.filter((sub) => validateSchema(sub, value, path, depth + 1).length === 0).length !== 1) out.push(`${path}: must match exactly one allowed shape`); + return out.slice(0, 10); +} diff --git a/packages/gateway/src/upstream.ts b/packages/gateway/src/upstream.ts index 5aa4863..286d82f 100644 --- a/packages/gateway/src/upstream.ts +++ b/packages/gateway/src/upstream.ts @@ -5,10 +5,19 @@ import { StreamableHTTPClientTransport } from "@modelcontextprotocol/sdk/client/ import type { CallToolResult, Tool } from "@modelcontextprotocol/sdk/types.js"; import { expandEnv, type UpstreamConfig } from "./config.js"; +/** + * The SDK client would reject a result that breaks the tool's output schema and throw away the result, even though the call took + * effect. The gateway checks schemas itself and records a mismatch in the ledger, so the client's own check is turned off. + */ +export const PASS_THROUGH_VALIDATOR = { + getValidator: () => (input: unknown) => ({ valid: true as const, data: input, errorMessage: undefined }), +}; + export interface UpstreamEntry { name: string; client: Client; namespace: boolean; + kind: "stdio" | "http"; } /** The gateway has no upstream for this tool name, so nothing was sent anywhere. */ @@ -30,7 +39,7 @@ export class UpstreamPool { const entries: UpstreamEntry[] = []; try { for (const c of cfgs) { - const client = new Client({ name: "reversi-gateway", version: "0.1.0" }); + const client = new Client({ name: "reversi-gateway", version: "0.1.0" }, { jsonSchemaValidator: PASS_THROUGH_VALIDATOR as never }); const expandAll = (m: Record | undefined) => Object.fromEntries(Object.entries(m ?? {}).map(([k, v]) => [k, expandEnv(v, env)])); const transport = c.url ? new StreamableHTTPClientTransport(new URL(c.url), { requestInit: { headers: expandAll(c.headers) } }) @@ -42,7 +51,7 @@ export class UpstreamPool { stderr: "inherit", }); await client.connect(transport); - entries.push({ name: c.name, client, namespace: c.namespace ?? many }); + entries.push({ name: c.name, client, namespace: c.namespace ?? many, kind: c.url ? "http" : "stdio" }); } } catch (e) { await Promise.allSettled(entries.map((x) => x.client.close())); @@ -73,6 +82,11 @@ export class UpstreamPool { return out; } + /** Which upstream each exposed tool name belongs to (as of the last listing). */ + owners(): Map { + return new Map([...this.routes].map(([exposed, r]) => [exposed, r.entry.name])); + } + async resolve(exposed: string): Promise<{ entry: UpstreamEntry; original: string } | undefined> { if (!this.routes.has(exposed)) await this.listTools(); return this.routes.get(exposed); diff --git a/packages/gateway/test/features.test.ts b/packages/gateway/test/features.test.ts new file mode 100644 index 0000000..bdf270b --- /dev/null +++ b/packages/gateway/test/features.test.ts @@ -0,0 +1,129 @@ +import { describe, expect, it } from "vitest"; +import { AGENT, MANIFESTS, OPERATOR, build, connect, fakeUpstream, json, textOf } from "./helpers.js"; +import { validateSchema } from "../src/schema.js"; +import { schemaHash } from "@multimindlab/reversi"; + +describe("schema checker", () => { + const schema = { type: "object", required: ["amount"], additionalProperties: false, properties: { amount: { type: "number", minimum: 1, maximum: 100 }, note: { type: "string", maxLength: 5 }, kind: { enum: ["a", "b"] } } }; + it("accepts good values and names what is wrong", () => { + expect(validateSchema(schema, { amount: 5 })).toEqual([]); + expect(validateSchema(schema, {})).toEqual([expect.stringContaining("missing required `amount`")]); + expect(validateSchema(schema, { amount: 500 })[0]).toContain("above maximum"); + expect(validateSchema(schema, { amount: "5" })[0]).toContain("expected number"); + expect(validateSchema(schema, { amount: 5, extra: 1 })[0]).toContain("unexpected `extra`"); + expect(validateSchema(schema, { amount: 5, kind: "z" })[0]).toContain("one of"); + expect(validateSchema({ anyOf: [{ type: "string" }, { type: "number" }] }, true)[0]).toContain("none of"); + }); +}); + +describe("gateway features", () => { + it("blocks arguments that do not match the tool's input schema", async () => { + const f = await fakeUpstream("up", { create_order: () => json({ id: "o1" }), cancel_order: () => json({}), lookup: () => json({}), refund: () => json({}) }, false, { + create_order: { inputSchema: { required: ["sku"], properties: { sku: { type: "string" } } } }, + }); + const { gateway } = build({ fakes: [f] }); + const { client } = await connect(gateway, AGENT); + const bad = await client.callTool({ name: "create_order", arguments: { sku: 5 } }); + expect(bad.isError).toBe(true); + expect(textOf(bad)).toContain("input schema"); + expect(f.calls).toHaveLength(0); + expect((await client.callTool({ name: "create_order", arguments: { sku: "x" } })).isError).toBeFalsy(); + }); + + it("marks a result that breaks the output schema, and does not trust it for the undo", async () => { + const f = await fakeUpstream("up", { create_order: () => ({ content: [{ type: "text", text: "{}" }], structuredContent: { id: 7 } }), cancel_order: () => json({}), lookup: () => json({}), refund: () => json({}) }, false, { + create_order: { outputSchema: { type: "object", required: ["id"], properties: { id: { type: "string" } } } }, + }); + const { gateway, reversi } = build({ fakes: [f] }); + const { client, run } = await connect(gateway, AGENT); + await client.callTool({ name: "create_order", arguments: {} }); + const detail = reversi.ledger.forRun(run!.run_id).map((r) => r.detail ?? "").join(" "); + expect(detail).toContain("output schema"); + }); + + it("notices a tool whose schema changed since the manifest was approved", async () => { + const tool = { name: "create_order", inputSchema: { type: "object", properties: { sku: { type: "string" } } } }; + const manifests = [{ ...MANIFESTS[0]!, schema_hash: schemaHash(tool as never) }, ...MANIFESTS.slice(1)]; + const f = await fakeUpstream("up", { create_order: () => json({ id: "o1" }), cancel_order: () => json({}), lookup: () => json({}), refund: () => json({}) }, false, { + create_order: { inputSchema: { properties: { sku: { type: "string" }, extra: { type: "string" } } } }, // upstream added a field + }); + const { gateway } = build({ fakes: [f], manifests }); + const { client } = await connect(gateway, AGENT); + const r = await client.callTool({ name: "create_order", arguments: {} }); + expect(r.isError).toBe(true); + expect(textOf(r)).toContain("schema changed"); + }); + + it("lets an unclassified read through when asked to, and still blocks unclassified changes", async () => { + const f = await fakeUpstream("up", { peek: () => json({ ok: 1 }), mutate: () => json({}), create_order: () => json({ id: "o" }), cancel_order: () => json({}), lookup: () => json({}), refund: () => json({}) }, false, { + peek: { annotations: { readOnlyHint: true } }, + }); + const { gateway } = build({ fakes: [f], gw: { unclassified_reads: "observe" } }); + const { client } = await connect(gateway, AGENT); + expect((await client.callTool({ name: "peek", arguments: {} })).isError).toBeFalsy(); + expect((await client.callTool({ name: "mutate", arguments: {} })).isError).toBe(true); + const strict = build({ fakes: [f] }); + const s = await connect(strict.gateway, AGENT); + expect((await s.client.callTool({ name: "peek", arguments: {} })).isError).toBe(true); + }); + + it("adds an advisory note in advise mode and blocks nothing", async () => { + const f = await fakeUpstream("up", { mystery: () => json({ ok: 1 }) }); + const { gateway } = build({ fakes: [f], mode: "advise" }); + const { client } = await connect(gateway, AGENT); + const r = await client.callTool({ name: "mystery", arguments: {} }); + expect(r.isError).toBeFalsy(); + expect(JSON.stringify(r.content)).toContain("Reversi advisory"); + }); + + it("times out a slow tool as uncertain, and retries only safe tools", async () => { + let slow = 0; + let flaky = 0; + const f = await fakeUpstream("up", { + slow: async () => { + slow++; + await new Promise((r) => setTimeout(r, 300)); + return json({}); + }, + flaky: () => { + if (++flaky < 3) throw new Error("boom"); + return json({ ok: true }); + }, + }); + const manifests = [ + { tool: "slow", effect: "idempotent", permission: "p", timeout_ms: 50 }, + { tool: "flaky", effect: "read_only", permission: "p", retry: { attempts: 3, base_ms: 1, max_ms: 5 } }, + ]; + const { gateway } = build({ fakes: [f], manifests }); + const { client, run } = await connect(gateway, AGENT); + expect((await client.callTool({ name: "flaky", arguments: {} })).isError).toBeFalsy(); + expect(flaky).toBe(3); + await client.callTool({ name: "slow", arguments: {} }).catch(() => undefined); + expect(slow).toBe(1); + expect(run!.state().steps.find((s) => s.tool === "slow")?.status).not.toBe("succeeded"); + }); + + it("classifies the upstream tools and previews impact", async () => { + const f = await fakeUpstream("up", { create_order: () => json({}), cancel_order: () => json({}), lookup: () => json({}), refund: () => json({}), novel: () => json({}) }); + const { gateway } = build({ fakes: [f] }); + const c = await gateway.classification(); + expect(c.rows.find((r) => r.name === "novel")?.class).toBe("unclassified_mutation"); + expect(c.banner).toContain("no manifest"); + const { client } = await connect(gateway, AGENT); + const prev = (await client.callTool({ name: "reversi.preview_impact", arguments: { tool: "create_order" } })).structuredContent as { effect: string; rollback_outlook: string[] }; + expect(prev.effect).toBe("reversible"); + expect(prev.rollback_outlook.length).toBeGreaterThan(0); + const names = (await client.listTools()).tools.map((t) => t.name); + expect(names).toContain("reversi.search"); + expect(names).not.toContain("reversi.reload"); + const op = await connect(gateway, OPERATOR, true); + expect((await op.client.listTools()).tools.map((t) => t.name)).toContain("reversi.reload"); + }); + + it("reports connectors with their classification counts", async () => { + const f = await fakeUpstream("up", { create_order: () => json({}), novel: () => json({}) }); + const { gateway } = build({ fakes: [f] }); + const rows = await gateway.connectors(); + expect(rows[0]).toMatchObject({ name: "up", kind: "stdio", healthy: true, tools: 2, classified: 1, unclassified: 1 }); + }); +}); diff --git a/packages/gateway/test/helpers.ts b/packages/gateway/test/helpers.ts index 8f6bfca..b05e88e 100644 --- a/packages/gateway/test/helpers.ts +++ b/packages/gateway/test/helpers.ts @@ -4,20 +4,23 @@ import { Server } from "@modelcontextprotocol/sdk/server/index.js"; import { CallToolRequestSchema, ListToolsRequestSchema, type CallToolResult } from "@modelcontextprotocol/sdk/types.js"; import { Approvals, ContentChecker, Ledger, Policy, Registry, Reversi, type Principal, type PolicyDoc } from "@multimindlab/reversi"; import { Gateway, UpstreamPool, type GatewayOptions } from "../src/index.js"; +import { PASS_THROUGH_VALIDATOR } from "../src/upstream.js"; export type Handler = (args: Record | undefined, meta: Record | undefined) => CallToolResult | Promise; export interface Fake { - entry: { name: string; client: Client; namespace: boolean }; + entry: { name: string; client: Client; namespace: boolean; kind: "stdio" | "http" }; calls: Array<{ name: string; args: unknown; meta: unknown }>; } export const json = (v: unknown): CallToolResult => ({ content: [{ type: "text", text: JSON.stringify(v) }] }); -export async function fakeUpstream(name: string, tools: Record, namespace = false): Promise { +export type ToolMeta = Record }>; + +export async function fakeUpstream(name: string, tools: Record, namespace = false, meta: ToolMeta = {}): Promise { const calls: Fake["calls"] = []; const server = new Server({ name, version: "1" }, { capabilities: { tools: {} } }); server.setRequestHandler(ListToolsRequestSchema, async () => ({ - tools: Object.keys(tools).map((n) => ({ name: n, description: n, inputSchema: { type: "object" as const, properties: {} } })), + tools: Object.keys(tools).map((n) => ({ name: n, description: n, inputSchema: { type: "object" as const, properties: {}, ...(meta[n]?.inputSchema ?? {}) }, ...(meta[n]?.outputSchema ? { outputSchema: meta[n]?.outputSchema } : {}), ...(meta[n]?.annotations ? { annotations: meta[n]?.annotations } : {}) })), })); server.setRequestHandler(CallToolRequestSchema, async (req) => { calls.push({ name: req.params.name, args: req.params.arguments, meta: req.params._meta }); @@ -27,9 +30,9 @@ export async function fakeUpstream(name: string, tools: Record, }); const [a, b] = InMemoryTransport.createLinkedPair(); await server.connect(a); - const client = new Client({ name: "gw", version: "1" }); + const client = new Client({ name: "gw", version: "1" }, { jsonSchemaValidator: PASS_THROUGH_VALIDATOR as never }); await client.connect(b); - return { entry: { name, client, namespace }, calls }; + return { entry: { name, client, namespace, kind: "stdio" }, calls }; } export const MANIFESTS = [ @@ -42,7 +45,7 @@ export const MANIFESTS = [ export const AGENT: Principal = { name: "agent-1", roles: ["agent"], via: "api_key" }; export const OPERATOR: Principal = { name: "ops", roles: ["admin"], via: "api_key" }; -export function build(opts: { fakes: Fake[]; manifests?: unknown[]; mode?: "observe" | "enforce"; policy?: PolicyDoc; content?: boolean; gw?: Partial }) { +export function build(opts: { fakes: Fake[]; manifests?: unknown[]; mode?: "observe" | "enforce" | "advise"; policy?: PolicyDoc; content?: boolean; gw?: Partial }) { const approvals = new Approvals(); const reversi = new Reversi({ registry: Registry.from(opts.manifests ?? MANIFESTS), diff --git a/packages/gateway/test/http-config.test.ts b/packages/gateway/test/http-config.test.ts index 50a5070..828d554 100644 --- a/packages/gateway/test/http-config.test.ts +++ b/packages/gateway/test/http-config.test.ts @@ -131,9 +131,15 @@ describe("HTTP gateway with API keys", () => { }); it("rejects oversized bodies", async () => { - const { url } = await serve({ keys: KEYS }); - const r = await post(`${url}/mcp`, { authorization: "Bearer k-agent" }, { pad: "x".repeat(5_000_000) }); - expect(r.status).toBe(413); + const { url, reversi } = await serve({ keys: KEYS }); + // The server answers 413 as soon as the limit is crossed. A client still sending its megabytes may see that answer or, + // if the connection closes under it first, a failed fetch: what must never happen is the request being processed. + const r = await post(`${url}/mcp`, { authorization: "Bearer k-agent" }, { pad: "x".repeat(5_000_000) }).then( + (res) => res.status, + () => "reset", + ); + expect([413, "reset"]).toContain(r); + expect(reversi.runIds()).toEqual([]); }); }); @@ -199,7 +205,7 @@ describe("config", () => { } catch (e) { expect(e).toBeInstanceOf(ConfigError); const m = (e as ConfigError).errors.join("\n"); - expect(m).toContain("`mode` must be observe or enforce"); + expect(m).toContain("`mode` must be observe, advise or enforce"); expect(m).toContain("name must be letters"); expect(m).toContain("exactly one of"); expect(m).toContain("duplicate name"); diff --git a/packages/gateway/tsconfig.json b/packages/gateway/tsconfig.json index b223a6f..2c38d84 100644 --- a/packages/gateway/tsconfig.json +++ b/packages/gateway/tsconfig.json @@ -1,6 +1,18 @@ { "extends": "../../tsconfig.base.json", - "compilerOptions": { "rootDir": "src", "outDir": "dist" }, - "include": ["src"], - "references": [{ "path": "../reversi" }] + "compilerOptions": { + "rootDir": "src", + "outDir": "dist" + }, + "include": [ + "src" + ], + "references": [ + { + "path": "../reversi" + }, + { + "path": "../server" + } + ] } diff --git a/packages/reversi/src/alerts.ts b/packages/reversi/src/alerts.ts new file mode 100644 index 0000000..e3885b1 --- /dev/null +++ b/packages/reversi/src/alerts.ts @@ -0,0 +1,139 @@ +/** + * Alerts: conditions worth a human's attention, computed from the ledger, approvals and health. `evaluateAlerts` is pure + * (the console shows its result live); the {@link AlertManager} sends each alert once, reminds while it persists, and + * resolves it when the condition clears, sharing its state through the KV store so several processes do not repeat it. + */ +import type { Approvals } from "./approvals.js"; +import type { Notifier, Severity } from "./channels.js"; +import type { Reversi } from "./engine.js"; +import type { KvStore } from "./kv.js"; + +export interface Alert { + id: string; + kind: string; + severity: Severity; + title: string; + detail: string; + ts_ms: number; + run_id?: string; +} + +export interface AlertOptions { + now?: number; + /** How far back event-based alerts look. Default 15 minutes. */ + window_ms?: number; + /** Blocked calls in the window that count as a spike. Default 10. */ + denial_spike?: number; + /** A step whose outcome has been unknown this long is flagged. Default 1 hour. */ + unresolved_ms?: number; + /** Upstream health (the gateway supplies it). */ + health?: Array<{ name: string; ok: boolean }>; +} + +export function evaluateAlerts(rv: Reversi, approvals: Approvals | undefined, o: AlertOptions = {}): Alert[] { + const now = o.now ?? Date.now(); + const since = now - (o.window_ms ?? 900_000); + const out = new Map(); + const add = (a: Alert) => { + if (!out.has(a.id)) out.set(a.id, a); + }; + + let denials = 0; + for (const r of rv.ledger.records()) { + if (r.ts_ms < since) continue; + if (r.event === "blocked") denials++; + if (r.event === "verification_failed") add({ id: `verify:${r.run_id}:${r.step}`, kind: "verification_failed", severity: "critical", title: `Undo not verified: ${r.tool}`, detail: r.detail, ts_ms: r.ts_ms, run_id: r.run_id }); + if (r.event === "unverified") add({ id: `unverified:${r.run_id}:${r.step}`, kind: "rollback_unverified", severity: "warning", title: `Undo could not be verified: ${r.tool}`, detail: r.detail, ts_ms: r.ts_ms, run_id: r.run_id }); + if (r.event === "succeeded" && rv.registry.get(r.tool)?.effect === "irreversible") add({ id: `irreversible:${r.run_id}:${r.step}`, kind: "irreversible_executed", severity: "warning", title: `Irreversible action executed: ${r.tool}`, detail: `${r.actor} in run ${r.run_id}`, ts_ms: r.ts_ms, run_id: r.run_id }); + if ((r.event === "blocked" || r.event === "would_block") && /no manifest registered/.test(r.detail) && !rv.registry.has(r.tool)) add({ id: `unclassified:${r.tool}`, kind: "unclassified_mutation", severity: "warning", title: `Unclassified tool called: ${r.tool}`, detail: `${r.actor} (${r.event})`, ts_ms: r.ts_ms, run_id: r.run_id }); + const budget = /budget `([^`]+)` at (\d+)%/.exec(r.detail); + if (budget) add({ id: `budget:${budget[1]}`, kind: "budget_threshold", severity: Number(budget[2]) >= 100 ? "critical" : "warning", title: `Budget ${budget[1]} at ${budget[2]}%`, detail: r.detail, ts_ms: r.ts_ms, run_id: r.run_id }); + if (r.event === "approved" && r.detail.startsWith("BREAK-GLASS")) add({ id: `breakglass:${r.seq}`, kind: "break_glass", severity: "critical", title: `Break-glass approval for ${r.tool}`, detail: `${r.actor}: ${r.reason}`, ts_ms: r.ts_ms, run_id: r.run_id }); + } + if (denials >= (o.denial_spike ?? 10)) add({ id: "denial-spike", kind: "policy_denial_spike", severity: "warning", title: `${denials} calls blocked in the last ${Math.round((o.window_ms ?? 900_000) / 60_000)} minutes`, detail: "An agent may be looping or probing. Check the blocked reasons in the ledger.", ts_ms: now }); + + for (const id of rv.runIds()) { + const st = rv.openRun(id).state(); + for (const s of st.steps) { + if (s.status === "compensation_failed") add({ id: `comp:${id}:${s.step}`, kind: "compensation_failed", severity: "critical", title: `Rollback failed: ${s.tool} in ${id}`, detail: s.error ?? "", ts_ms: s.ts_ms, run_id: id }); + if ((s.status === "intended" || s.status === "uncertain") && now - s.ts_ms > (o.unresolved_ms ?? 3_600_000)) add({ id: `stuck:${id}:${s.step}`, kind: "unresolved_outcome", severity: "warning", title: `Outcome still unknown: ${s.tool} in ${id}`, detail: `step ${s.step} has been ${s.status} for over ${Math.round((now - s.ts_ms) / 60_000)} minutes; check the downstream system and resolve it`, ts_ms: s.ts_ms, run_id: id }); + } + } + + for (const a of approvals?.list() ?? []) { + if ((a.status === "expired" || a.escalated) && a.requested_ms >= since - (a.ttl_ms ?? 0)) add({ id: `approval-timeout:${a.id}`, kind: "approval_timeout", severity: "warning", title: `Approval ${a.status === "expired" ? "expired unanswered" : "escalated"}: ${a.tool}`, detail: `run ${a.run_id}, requested by ${a.requested_by ?? "unknown"}`, ts_ms: a.expires_ms, run_id: a.run_id }); + } + for (const h of o.health ?? []) if (!h.ok) add({ id: `upstream:${h.name}`, kind: "upstream_unhealthy", severity: "critical", title: `Upstream ${h.name} is not answering`, detail: "The gateway cannot reach it; calls to its tools will fail.", ts_ms: now }); + for (const h of rv.halts()) add({ id: `halt:${h.id}`, kind: "halted", severity: "warning", title: "New calls are blocked", detail: `${h.reason}${Object.keys(h.scope).length ? ` (scope: ${Object.entries(h.scope).map(([k, v]) => `${k}=${v}`).join(", ")})` : ""}`, ts_ms: h.ts_ms }); + return [...out.values()].sort((a, b) => (a.severity === b.severity ? b.ts_ms - a.ts_ms : a.severity === "critical" ? -1 : b.severity === "critical" ? 1 : 0)); +} + +interface AlertState { + active: boolean; + first_ms: number; + last_notified_ms: number; + title: string; +} + +const NS = "alert-state"; + +export interface AlertManagerOptions extends Omit { + remind_ms?: number; + console_url?: string; + /** Upstream health, asked for on every tick (the gateway supplies it). */ + health?: () => Promise>; +} + +export class AlertManager { + constructor( + private readonly rv: Reversi, + private readonly approvals: Approvals | undefined, + private readonly kv: KvStore, + private readonly notifier: Notifier, + private readonly o: AlertManagerOptions = {}, + ) {} + + private timer: NodeJS.Timeout | undefined; + + setHealth(fn: AlertManagerOptions["health"]): void { + this.o.health = fn; + } + + start(intervalMs = 60_000): void { + this.stop(); + this.timer = setInterval(() => void this.tick().catch(() => {}), intervalMs); + this.timer.unref(); + } + stop(): void { + if (this.timer) clearInterval(this.timer); + } + + /** Evaluate once: notify new alerts (and reminders), resolve cleared ones. Returns what was sent. */ + async tick(now = Date.now()): Promise<{ fired: string[]; resolved: string[] }> { + const health = this.o.health ? await this.o.health().catch(() => undefined) : undefined; + const { health: _fn, remind_ms: _r, console_url: _c, ...evalOpts } = this.o; + const alerts = evaluateAlerts(this.rv, this.approvals, { ...evalOpts, now, health }); + const remind = this.o.remind_ms ?? 3_600_000; + const fired: string[] = []; + const resolved: string[] = []; + const live = new Set(alerts.map((a) => a.id)); + for (const a of alerts) { + const cur = this.kv.get(NS, a.id); + let send = false; + if (!cur) send = this.kv.cas(NS, a.id, { active: true, first_ms: now, last_notified_ms: now, title: a.title }, 0); + else if (!cur.value.active) send = this.kv.cas(NS, a.id, { active: true, first_ms: now, last_notified_ms: now, title: a.title }, cur.version); + else if (now - cur.value.last_notified_ms >= remind) send = this.kv.cas(NS, a.id, { ...cur.value, last_notified_ms: now }, cur.version); + if (send) { + fired.push(a.id); + await this.notifier.notify({ event: "alert", severity: a.severity, title: a.title, text: a.detail, dedup_key: a.id, data: { kind: a.kind, run_id: a.run_id }, link: this.o.console_url ? `${this.o.console_url.replace(/\/$/, "")}/console/alerts` : undefined }); + } + } + for (const e of this.kv.list(NS)) { + if (e.value.active && !live.has(e.key) && this.kv.cas(NS, e.key, { ...e.value, active: false }, e.version)) { + resolved.push(e.key); + await this.notifier.notify({ event: "alert", severity: "info", title: `Resolved: ${e.value.title}`, text: "The condition has cleared.", dedup_key: e.key, resolve: true }); + } + } + return { fired, resolved }; + } +} diff --git a/packages/reversi/src/anchor.ts b/packages/reversi/src/anchor.ts new file mode 100644 index 0000000..7b136ae --- /dev/null +++ b/packages/reversi/src/anchor.ts @@ -0,0 +1,94 @@ +/** + * External anchoring: periodically publish the ledger head somewhere the operator cannot quietly rewrite (an append-only + * file on other storage, a webhook into a system you do not control), optionally signed. Later, `verifyAnchor` proves the + * ledger still contains the anchored head, so truncation or a rewrite of history is caught. + */ +import { createHmac, createPublicKey, sign, verify } from "node:crypto"; +import { appendFileSync, existsSync, readFileSync } from "node:fs"; +import { canonicalize } from "./canonical.js"; +import { guardedFetch, type EgressConfig } from "./egress.js"; +import { keyId, type SigningKey } from "./evidence.js"; +import type { Ledger } from "./ledger.js"; + +export type AnchorSink = { type: "file"; path: string } | { type: "webhook"; url: string; secret?: string }; + +export interface AnchorRecord { + format: "reversi-anchor"; + version: 1; + ts_ms: number; + /** Hash of the last record when anchored. */ + head: string; + /** How many records the ledger had. */ + records: number; + region?: string; + key_id?: string; + signature?: string; +} + +const body = (a: AnchorRecord) => canonicalize({ format: a.format, version: a.version, ts_ms: a.ts_ms, head: a.head, records: a.records, region: a.region ?? null }); + +export async function anchorLedger( + ledger: Ledger, + sinks: AnchorSink[], + o: { key?: SigningKey; egress?: EgressConfig; fetch?: typeof fetch; now?: number } = {}, +): Promise<{ anchor: AnchorRecord; delivered: string[]; errors: string[] }> { + const records = ledger.records().length; + const anchor: AnchorRecord = { format: "reversi-anchor", version: 1, ts_ms: o.now ?? Date.now(), head: ledger.head(), records, ...(o.egress?.region ? { region: o.egress.region } : {}) }; + if (o.key) { + anchor.key_id = keyId(o.key.public_pem); + anchor.signature = sign(null, Buffer.from(body(anchor)), (await import("node:crypto")).createPrivateKey(o.key.private_pem)).toString("base64"); + } + const delivered: string[] = []; + const errors: string[] = []; + const f = guardedFetch(o.egress, o.fetch); + for (const s of sinks) { + try { + if (s.type === "file") { + appendFileSync(s.path, JSON.stringify(anchor) + "\n"); + delivered.push(`file:${s.path}`); + } else { + const text = JSON.stringify(anchor); + const headers: Record = { "content-type": "application/json" }; + if (s.secret) headers["x-reversi-signature"] = `sha256=${createHmac("sha256", s.secret).update(text).digest("hex")}`; + const res = await f(s.url, { method: "POST", headers, body: text, signal: AbortSignal.timeout(10_000) }); + if (!res.ok) throw new Error(`answered ${res.status}`); + delivered.push(`webhook:${new URL(s.url).host}`); + } + } catch (e) { + errors.push(`${s.type === "file" ? s.path : s.url}: ${e instanceof Error ? e.message : String(e)}`); + } + } + return { anchor, delivered, errors }; +} + +export function readAnchors(path: string): AnchorRecord[] { + if (!existsSync(path)) return []; + return readFileSync(path, "utf8").split("\n").filter((l) => l.trim()).map((l) => JSON.parse(l) as AnchorRecord); +} + +export type AnchorCheck = { ok: true; records: number; signed: boolean } | { ok: false; error: string }; + +/** Does the ledger still contain this anchor's head at the position it claims? */ +export function verifyAnchor(ledger: Ledger, a: AnchorRecord, trusted_public_pem?: string): AnchorCheck { + if (a?.format !== "reversi-anchor") return { ok: false, error: "not a Reversi anchor" }; + const v = ledger.verify(); + if (!v.ok) return { ok: false, error: `the ledger itself is broken at record ${v.seq}: ${v.error}` }; + let signed = false; + if (a.signature) { + if (!trusted_public_pem) { + signed = false; // present but nothing to check it against + } else { + try { + signed = verify(null, Buffer.from(body(a)), createPublicKey(trusted_public_pem), Buffer.from(a.signature, "base64")); + } catch { + signed = false; + } + if (!signed) return { ok: false, error: "the anchor's signature does not match the trusted key" }; + } + } else if (trusted_public_pem) return { ok: false, error: "the anchor is unsigned but a trusted key was required" }; + const recs = ledger.records(); + if (a.records > recs.length) return { ok: false, error: `the ledger has ${recs.length} records but was anchored at ${a.records}: it has been truncated` }; + const at = recs[a.records - 1]; + if (a.records === 0 ? false : at?.hash !== a.head) return { ok: false, error: `record ${a.records - 1} no longer has the anchored hash: history was rewritten` }; + return { ok: true, records: a.records, signed }; +} diff --git a/packages/reversi/src/approvals.ts b/packages/reversi/src/approvals.ts index a5c6838..63e4728 100644 --- a/packages/reversi/src/approvals.ts +++ b/packages/reversi/src/approvals.ts @@ -8,6 +8,7 @@ import { randomUUID } from "node:crypto"; import { closeSync, existsSync, fsyncSync, mkdirSync, openSync, readFileSync, renameSync, writeSync } from "node:fs"; import { dirname } from "node:path"; +import type { Delegations } from "./delegations.js"; import type { ApprovalDecision, ApprovalRequest, Approver } from "./engine.js"; import { redactSecrets } from "./redact.js"; @@ -32,6 +33,21 @@ export interface ApprovalRecord { decided_by: string | null; decided_ms: number | null; decision_note: string | null; + /** The actor of the run that asked. A requester may not approve their own request. */ + requested_by?: string; + /** Reversi roles or group names that may approve. Empty: any approver. */ + required_roles?: string[]; + /** How many distinct approvers are needed (default 1). */ + quorum?: number; + /** Counts decisions so far; lets stores detect two approvers racing. */ + decision_count?: number; + decisions?: Array<{ by: string; on_behalf_of?: string; basis: string; decision: "approve" | "deny"; note: string | null; ms: number }>; + on_timeout?: "deny" | "escalate"; + escalate_to?: string[]; + escalated?: boolean; + /** The lifetime applied to this request, used again when it escalates. */ + ttl_ms?: number; + break_glass?: { by: string; reason: string; evidence: string; ms: number }; } export interface ApprovalStore { @@ -43,7 +59,9 @@ export interface ApprovalStore { * not in that status (someone else got there first). This is what makes "single use" and "decided once" hold * when several processes share a store. */ - transition(id: string, from: ApprovalStatus, patch: Partial): boolean; + transition(id: string, from: ApprovalStatus, patch: Partial, expectCount?: number): boolean; + /** Remove a request (retention). */ + delete(id: string): boolean; } export class MemoryApprovalStore implements ApprovalStore { @@ -58,9 +76,12 @@ export class MemoryApprovalStore implements ApprovalStore { list() { return [...this.items.values()].map((r) => structuredClone(r)); } - transition(id: string, from: ApprovalStatus, patch: Partial) { + delete(id: string) { + return this.items.delete(id); + } + transition(id: string, from: ApprovalStatus, patch: Partial, expectCount?: number) { const r = this.items.get(id); - if (!r || r.status !== from) return false; + if (!r || r.status !== from || (expectCount !== undefined && (r.decision_count ?? 0) !== expectCount)) return false; this.items.set(id, { ...r, ...patch }); return true; } @@ -93,10 +114,25 @@ export class FileApprovalStore implements ApprovalStore { list() { return Object.values(this.read()); } + delete(id: string) { + const all = this.read(); + if (!all[id]) return false; + delete all[id]; + const tmp = `${this.path}.${process.pid}.tmp`; + const fd = openSync(tmp, "w"); + try { + writeSync(fd, JSON.stringify(all, null, 2)); + fsyncSync(fd); + } finally { + closeSync(fd); + } + renameSync(tmp, this.path); + return true; + } /** Read, compare and rename-over: atomic enough for one host; use SQLite or PostgreSQL to share across hosts. */ - transition(id: string, from: ApprovalStatus, patch: Partial) { + transition(id: string, from: ApprovalStatus, patch: Partial, expectCount?: number) { const r = this.read()[id]; - if (!r || r.status !== from) return false; + if (!r || r.status !== from || (expectCount !== undefined && (r.decision_count ?? 0) !== expectCount)) return false; this.put({ ...r, ...patch }); return true; } @@ -109,35 +145,70 @@ export class ApprovalError extends Error { } } +/** Who is deciding: their Reversi roles and group names (from the key file or the identity provider). */ +export interface ApproverIdentity { + roles?: string[]; + groups?: string[]; +} + export interface ApprovalsOptions { /** How long a pending or approved request stays valid. Default 1 hour. */ ttl_ms?: number; now?: () => number; /** Called when a new request is queued (to notify an approver). Must not throw; failures are the callback's to handle. */ on_request?: (rec: ApprovalRecord) => void; - /** Called after an approver approves or denies, so the decision can be written to the ledger. */ + /** Called after the final decision (approve, deny, break-glass), so it can be written to the ledger. */ on_decision?: (rec: ApprovalRecord, decision: "approved" | "denied") => void; + /** Called when one approver of a quorum has approved but more are needed. */ + on_progress?: (rec: ApprovalRecord) => void; + /** Called when an unanswered request escalates to other approvers. */ + on_escalate?: (rec: ApprovalRecord) => void; + /** Used to let delegates approve on behalf of someone. */ + delegations?: Delegations; } +const DECISION_LIMIT = 10; + export class Approvals { private readonly ttl: number; private readonly now: () => number; - private readonly onRequest: ((rec: ApprovalRecord) => void) | undefined; - private readonly onDecision: ((rec: ApprovalRecord, d: "approved" | "denied") => void) | undefined; + private readonly o: ApprovalsOptions; constructor( readonly store: ApprovalStore = new MemoryApprovalStore(), opts: ApprovalsOptions = {}, ) { this.ttl = opts.ttl_ms ?? 3_600_000; this.now = opts.now ?? Date.now; - this.onRequest = opts.on_request; - this.onDecision = opts.on_decision; + this.o = opts; } + private safely(fn: (() => void) | undefined): void { + try { + fn?.(); + } catch { + /* notifications and ledger mirroring must never change an approval decision */ + } + } + + /** Expire or escalate a request whose time is up. */ private sweep(r: ApprovalRecord): ApprovalRecord { if ((r.status === "pending" || r.status === "approved") && this.now() >= r.expires_ms) { - r.status = "expired"; - this.store.put(r); + if (r.status === "pending" && r.on_timeout === "escalate" && !r.escalated) { + const patch: Partial = { + escalated: true, + required_roles: r.escalate_to && r.escalate_to.length ? r.escalate_to : r.required_roles, + expires_ms: this.now() + (r.ttl_ms ?? this.ttl), + decision_count: (r.decision_count ?? 0) + 1, + }; + if (this.store.transition(r.id, "pending", patch, r.decision_count ?? 0)) { + const next = { ...r, ...patch }; + this.safely(() => this.o.on_escalate?.(next)); + return next; + } + return this.store.get(r.id) ?? r; + } + if (this.store.transition(r.id, r.status, { status: "expired" })) return { ...r, status: "expired" }; + return this.store.get(r.id) ?? r; } return r; } @@ -160,6 +231,11 @@ export class Approvals { return (req) => this.decide(req); } + private pendingMessage(rec: ApprovalRecord): string { + const need = (rec.quorum ?? 1) > 1 ? ` (needs ${rec.quorum} approvers${rec.required_roles?.length ? ` from ${rec.required_roles.join(" or ")}` : ""})` : rec.required_roles?.length ? ` (needs an approver from ${rec.required_roles.join(" or ")})` : ""; + return `approval required: request ${rec.id} is pending an operator decision${need}; retry after it is approved`; + } + private decide(req: ApprovalRequest): ApprovalDecision { const mine = this.list({ run_id: req.run_id }).filter( (r) => r.kind === req.kind && r.tool === req.tool && (r.args_hash === null || r.args_hash === (req.args_hash ?? null)), @@ -170,8 +246,14 @@ export class Approvals { } const denied = mine.find((r) => r.status === "denied" && r.args_hash === (req.args_hash ?? null) && r.step === req.step); if (denied) return { approved: false, message: `approval ${denied.id} was denied${denied.decision_note ? `: ${denied.decision_note}` : ""}` }; + const expired = mine.find((r) => r.status === "expired" && r.args_hash === (req.args_hash ?? null) && r.step === req.step && this.now() - r.expires_ms < 60_000); const pending = mine.find((r) => r.status === "pending"); - if (pending) return { approved: false, message: `approval required: request ${pending.id} is pending an operator decision; retry after it is approved` }; + if (pending) return { approved: false, message: this.pendingMessage(pending) }; + if (expired && expired.on_timeout !== "escalate") { + // Nobody answered in time and the rule says deny: say so once, then allow a fresh request. + return { approved: false, message: `approval ${expired.id} expired unanswered and was denied; the call must be requested again` }; + } + const ttl = req.timeout_ms ?? this.ttl; const rec: ApprovalRecord = { id: `apr_${randomUUID()}`, run_id: req.run_id, @@ -184,42 +266,114 @@ export class Approvals { args_hash: req.args_hash ?? null, status: "pending", requested_ms: this.now(), - expires_ms: this.now() + this.ttl, + expires_ms: this.now() + ttl, decided_by: null, decided_ms: null, decision_note: null, + requested_by: req.requested_by, + required_roles: req.required_roles ?? [], + quorum: req.quorum ?? 1, + decision_count: 0, + decisions: [], + on_timeout: req.on_timeout ?? "deny", + escalate_to: req.escalate_to ?? [], + escalated: false, + ttl_ms: ttl, }; this.store.put(rec); - try { - this.onRequest?.(rec); - } catch { - /* a broken notifier must never change an approval decision */ + this.safely(() => this.o.on_request?.(rec)); + return { approved: false, message: this.pendingMessage(rec) }; + } + + /** + * Work out whether `by` may decide `r`, and on whose authority. Returns the basis ("role:operator", "group:finance", + * "delegated by alice") or throws. With no `who` the caller is trusted (the CLI on the host that owns the data). + */ + private authorize(r: ApprovalRecord, by: string, who: ApproverIdentity | undefined): { basis: string; on_behalf_of?: string } { + if (!who) return { basis: "trusted caller" }; + if (r.requested_by && r.requested_by === by) throw new ApprovalError("you cannot decide a request made by your own run"); + const required = r.required_roles ?? []; + const mine = new Set([...(who.roles ?? []), ...(who.groups ?? [])]); + if (required.length === 0) return { basis: "approver" }; + const direct = required.find((x) => mine.has(x)); + if (direct) return { basis: `${(who.roles ?? []).includes(direct) ? "role" : "group"}:${direct}` }; + for (const d of this.o.delegations?.activeFor(by) ?? []) { + const delegated = d.groups.length ? d.groups : required; + const hit = required.find((x) => delegated.includes(x)); + if (hit) return { basis: `delegated by ${d.from} (${hit})`, on_behalf_of: d.from }; } - return { approved: false, message: `approval required: request ${rec.id} is pending an operator decision; retry after it is approved` }; + throw new ApprovalError(`this request needs an approver from: ${required.join(", ")}`); } - private decideOn(id: string, status: "approved" | "denied", by: string, note?: string): ApprovalRecord { + private decideOn(id: string, decision: "approve" | "deny", by: string, note?: string, who?: ApproverIdentity): ApprovalRecord { const r = this.get(id); if (!r) throw new ApprovalError(`approval \`${id}\` not found`); if (r.status !== "pending") throw new ApprovalError(`approval \`${id}\` is ${r.status}, not pending`); - const patch = { status, decided_by: by, decided_ms: this.now(), decision_note: note ? redactSecrets(note) : null }; - if (!this.store.transition(id, "pending", patch)) { - throw new ApprovalError(`approval \`${id}\` was decided by someone else first`); + const auth = this.authorize(r, by, who); + const identity = auth.on_behalf_of ?? by; + const prior = r.decisions ?? []; + if (prior.some((d) => (d.on_behalf_of ?? d.by) === identity || d.by === by)) throw new ApprovalError("you have already decided this request"); + if (auth.on_behalf_of && prior.some((d) => d.by === auth.on_behalf_of)) throw new ApprovalError(`${auth.on_behalf_of} has already decided this request, so their delegate cannot add a second vote`); + const entry = { by, ...(auth.on_behalf_of ? { on_behalf_of: auth.on_behalf_of } : {}), basis: auth.basis, decision, note: note ? redactSecrets(note) : null, ms: this.now() }; + const decisions = [...prior, entry].slice(-DECISION_LIMIT); + const count = r.decision_count ?? 0; + const quorum = r.quorum ?? 1; + const approvedVotes = decisions.filter((d) => d.decision === "approve").length; + let patch: Partial; + if (decision === "deny") { + patch = { status: "denied", decided_by: by, decided_ms: this.now(), decision_note: entry.note, decisions, decision_count: count + 1 }; + } else if (approvedVotes >= quorum) { + patch = { + status: "approved", + decided_by: decisions.filter((d) => d.decision === "approve").map((d) => d.by).join(", "), + decided_ms: this.now(), + decision_note: entry.note, + decisions, + decision_count: count + 1, + }; + } else { + patch = { decisions, decision_count: count + 1 }; // one more vote, still pending } + if (!this.store.transition(id, "pending", patch, count)) throw new ApprovalError(`approval \`${id}\` changed while you were deciding: reload it and try again`); const done = { ...r, ...patch }; - try { - this.onDecision?.(done, status); - } catch { - /* the decision stands even if recording it elsewhere failed; the approval store is the source of truth */ - } + if (done.status === "approved" || done.status === "denied") this.safely(() => this.o.on_decision?.(done, done.status === "approved" ? "approved" : "denied")); + else this.safely(() => this.o.on_progress?.(done)); return done; } - approve(id: string, by: string, note?: string): ApprovalRecord { - return this.decideOn(id, "approved", by, note); + approve(id: string, by: string, note?: string, who?: ApproverIdentity): ApprovalRecord { + return this.decideOn(id, "approve", by, note, who); + } + deny(id: string, by: string, note?: string, who?: ApproverIdentity): ApprovalRecord { + return this.decideOn(id, "deny", by, note, who); } - deny(id: string, by: string, note?: string): ApprovalRecord { - return this.decideOn(id, "denied", by, note); + + /** + * Emergency override: an admin approves without the usual approvers or quorum. A reason and a pointer to the evidence + * (ticket, incident) are mandatory, and the ledger records it as a break-glass approval. + */ + breakGlass(id: string, by: string, info: { reason: string; evidence: string }, who?: ApproverIdentity): ApprovalRecord { + if (who && !(who.roles ?? []).includes("admin")) throw new ApprovalError("break-glass needs the admin role"); + if (info.reason.trim().length < 10) throw new ApprovalError("break-glass needs a reason of at least 10 characters"); + if (!info.evidence.trim()) throw new ApprovalError("break-glass needs a reference to the evidence (ticket or incident)"); + const r = this.get(id); + if (!r) throw new ApprovalError(`approval \`${id}\` not found`); + if (r.status !== "pending") throw new ApprovalError(`approval \`${id}\` is ${r.status}, not pending`); + const count = r.decision_count ?? 0; + const bg = { by, reason: redactSecrets(info.reason), evidence: redactSecrets(info.evidence), ms: this.now() }; + const patch: Partial = { + status: "approved", + decided_by: by, + decided_ms: bg.ms, + decision_note: `BREAK-GLASS: ${bg.reason}`, + break_glass: bg, + decision_count: count + 1, + decisions: [...(r.decisions ?? []), { by, basis: "break-glass", decision: "approve" as const, note: bg.reason, ms: bg.ms }].slice(-DECISION_LIMIT), + }; + if (!this.store.transition(id, "pending", patch, count)) throw new ApprovalError(`approval \`${id}\` changed while you were deciding: reload it and try again`); + const done = { ...r, ...patch }; + this.safely(() => this.o.on_decision?.(done, "approved")); + return done; } /** Operator pre-approval for a tool in a run, valid for `count` uses and any arguments. */ diff --git a/packages/reversi/src/auth.ts b/packages/reversi/src/auth.ts index c39e81f..9f77e9f 100644 --- a/packages/reversi/src/auth.ts +++ b/packages/reversi/src/auth.ts @@ -30,6 +30,10 @@ export interface Principal { /** Recorded as the ledger `actor` for agents and compensations. */ name: string; roles: Role[]; + /** Group names, for approval routing ("finance-approver"). From the key file or the identity provider. */ + groups?: string[]; + /** The customer or team this principal belongs to. Runs it opens carry it. */ + tenant?: string; max_permissions?: string[]; /** May open runs in `observe` mode. Observe mode lets violating calls run, so it is not for agents by default. */ allow_observe?: boolean; @@ -57,6 +61,8 @@ export interface ApiKey { max_permissions?: string[]; /** Agent keys: allow opening runs in observe mode (default: enforce only). */ allow_observe?: boolean; + groups?: string[]; + tenant?: string; /** ISO date or epoch ms. The key is rejected before this time. Use with `expires_at` to rotate without downtime. */ not_before?: string | number; /** ISO date or epoch ms. The key is rejected from this time on. */ @@ -79,6 +85,8 @@ interface Compiled { digest: Buffer; max_permissions?: string[]; allow_observe: boolean; + groups: string[]; + tenant?: string; not_before?: number; expires_at?: number; disabled: boolean; @@ -107,6 +115,8 @@ export class KeyRing { digest, max_permissions: k.max_permissions, allow_observe: k.allow_observe === true, + groups: k.groups ?? [], + tenant: k.tenant, not_before: toMs(k.not_before, "not_before", k.name), expires_at: toMs(k.expires_at, "expires_at", k.name), disabled: k.disabled === true, @@ -124,7 +134,7 @@ export class KeyRing { } private principal(k: Compiled): Principal { - return { name: k.name, roles: k.roles, max_permissions: k.max_permissions, allow_observe: k.allow_observe, via: "api_key", fingerprint: k.digest.toString("hex").slice(0, 32) }; + return { name: k.name, roles: k.roles, groups: k.groups, ...(k.tenant ? { tenant: k.tenant } : {}), max_permissions: k.max_permissions, allow_observe: k.allow_observe, via: "api_key", fingerprint: k.digest.toString("hex").slice(0, 32) }; } authenticate(token: string): Principal | null { @@ -159,6 +169,8 @@ export interface OidcOptions { role_map: Record; /** Claim used as the principal name. Default `sub`. */ actor_claim?: string; + /** Claim holding the tenant (customer or team). */ + tenant_claim?: string; /** Caps for agent principals. */ max_permissions?: string[]; fetch?: typeof fetch; @@ -260,7 +272,16 @@ export class OidcVerifier { if (roles.length === 0) return null; const name = claims[this.o.actor_claim ?? "sub"]; if (typeof name !== "string" || !name) return null; - return { name, roles, max_permissions: this.o.max_permissions, via: "oidc", exp_ms: claims.exp * 1000 }; + const tenant = this.o.tenant_claim ? claims[this.o.tenant_claim] : undefined; + return { + name, + roles, + groups: values, // every value of the role claim is also a group name, so approvals can be routed to it + ...(typeof tenant === "string" && tenant ? { tenant } : {}), + max_permissions: this.o.max_permissions, + via: "oidc", + exp_ms: claims.exp * 1000, + }; } catch { return null; } diff --git a/packages/reversi/src/capture.ts b/packages/reversi/src/capture.ts new file mode 100644 index 0000000..6984f3f --- /dev/null +++ b/packages/reversi/src/capture.ts @@ -0,0 +1,152 @@ +/** + * Opt-in capture of redacted call arguments and results, kept for a limited time. + * + * The ledger never stores arguments (only hashes). Replay, policy simulation against history and impact previews need + * them, so capture is a separate, explicit choice: off by default, redacted on the way in (credentials, optional + * personal data, and the paths a manifest lists under `audit.redaction`), and deleted when it expires. + */ +import { randomBytes } from "node:crypto"; +import { kvUpdate, type KvStore } from "./kv.js"; +import { maskPii } from "./pii.js"; +import { redactSecrets } from "./redact.js"; + +export interface CaptureConfig { + enabled: boolean; + /** How long a record is kept. Default 7 days. */ + ttl_ms: number; + /** Also mask emails, phone numbers, card numbers, SSNs and IBANs. Default true. */ + redact_pii: boolean; + /** Largest captured value in characters; bigger ones are cut. Default 20,000. */ + max_chars: number; +} + +export const DEFAULT_CAPTURE: CaptureConfig = { enabled: false, ttl_ms: 7 * 86_400_000, redact_pii: true, max_chars: 20_000 }; + +export interface CaptureRecord { + run_id: string; + /** Null for a call that was blocked before it became a step. */ + step: number | null; + tool: string; + actor: string; + tenant?: string; + environment: string; + args: unknown; + result?: unknown; + /** What the engine decided at the time. */ + decision: "allow" | "would_block" | "block"; + reasons: string[]; + outcome?: "succeeded" | "failed" | "uncertain"; + manifest_version?: number; + risk_score?: number; + ts_ms: number; + expires_ms: number; +} + +const NS = "capture"; +const MASK = "[REDACTED]"; + +/** Replace the value at `$args.a.b` inside `root` (which is the args or result object itself). */ +function setPath(root: unknown, path: string, value: unknown): void { + const segs = path.split(".").slice(1); + let cur = root as Record | unknown[]; + for (let i = 0; i < segs.length; i++) { + const k = segs[i] as string; + const next = (cur as Record)[k]; + if (i === segs.length - 1) { + if (typeof cur === "object" && cur !== null && k in cur) (cur as Record)[k] = value; + return; + } + if (typeof next !== "object" || next === null) return; + cur = next as Record; + } +} + +/** A redacted deep copy: credentials and (optionally) personal data masked, listed paths replaced, long values cut. */ +export function redactForCapture(value: unknown, opts: { redact_pii: boolean; max_chars: number; paths?: string[]; root?: "args" | "result" }): unknown { + const clean = (v: unknown, depth: number): unknown => { + if (depth > 12) return "[too deep]"; + if (typeof v === "string") { + let s = redactSecrets(v); + if (opts.redact_pii) s = maskPii(s); + return s.length > opts.max_chars ? `${s.slice(0, opts.max_chars)}…[cut]` : s; + } + if (Array.isArray(v)) return v.slice(0, 200).map((x) => clean(x, depth + 1)); + if (typeof v === "object" && v !== null) return Object.fromEntries(Object.entries(v).slice(0, 200).map(([k, x]) => [k, clean(x, depth + 1)])); + if (typeof v === "number" && !Number.isFinite(v)) return String(v); + return v; + }; + const out = clean(value, 0); + for (const p of opts.paths ?? []) if (p.startsWith(`$${opts.root ?? "args"}.`)) setPath(out, p, MASK); + return out; +} + +export class CaptureStore { + readonly config: CaptureConfig; + constructor( + private readonly kv: KvStore, + config: Partial = {}, + private readonly now: () => number = Date.now, + ) { + this.config = { ...DEFAULT_CAPTURE, ...config }; + } + + get enabled(): boolean { + return this.config.enabled; + } + + private key(run: string, step: number | null): string { + return `${run}/${step === null ? `blocked-${this.now()}-${randomBytes(3).toString("hex")}` : String(step).padStart(6, "0")}`; + } + + /** Redact and store a call. Returns the key, so the outcome can be added later. */ + record(rec: Omit & { args: unknown; result?: unknown; redact_paths?: string[] }): string | null { + if (!this.config.enabled) return null; + const { redact_paths, ...rest } = rec; + const o = { redact_pii: this.config.redact_pii, max_chars: this.config.max_chars, paths: redact_paths }; + const stored: CaptureRecord = { + ...rest, + args: redactForCapture(rec.args, { ...o, root: "args" }), + ...(rec.result !== undefined ? { result: redactForCapture(rec.result, { ...o, root: "result" }) } : {}), + ts_ms: this.now(), + expires_ms: this.now() + this.config.ttl_ms, + }; + const key = this.key(rec.run_id, rec.step); + this.kv.put(NS, key, stored); + return key; + } + + /** Add what happened to a captured call. */ + complete(run: string, step: number, outcome: "succeeded" | "failed" | "uncertain", result: unknown, redact_paths?: string[]): void { + if (!this.config.enabled) return; + const o = { redact_pii: this.config.redact_pii, max_chars: this.config.max_chars, paths: redact_paths, root: "result" as const }; + kvUpdate(this.kv, NS, `${run}/${String(step).padStart(6, "0")}`, (cur) => + cur ? { ...cur, outcome, ...(result !== undefined ? { result: redactForCapture(result, o) } : {}) } : cur, + ); + } + + private live(r: CaptureRecord): boolean { + return r.expires_ms > this.now(); + } + + list(filter: { run_id?: string; tool?: string; since_ms?: number } = {}): CaptureRecord[] { + return this.kv + .list(NS, filter.run_id ? `${filter.run_id}/` : "") + .map((e) => e.value) + .filter((r) => r && this.live(r) && (!filter.tool || r.tool === filter.tool) && (!filter.since_ms || r.ts_ms >= filter.since_ms)) + .sort((a, b) => a.ts_ms - b.ts_ms || (a.step ?? 0) - (b.step ?? 0)); + } + + get(run: string, step: number): CaptureRecord | undefined { + const r = this.kv.get(NS, `${run}/${String(step).padStart(6, "0")}`)?.value; + return r && this.live(r) ? r : undefined; + } + + /** Delete expired records (and, with `run_id`, everything captured for that run). Returns how many were removed. */ + purge(opts: { run_id?: string } = {}): number { + let n = 0; + for (const e of this.kv.list(NS, opts.run_id ? `${opts.run_id}/` : "")) { + if (opts.run_id || !this.live(e.value)) if (this.kv.delete(NS, e.key)) n++; + } + return n; + } +} diff --git a/packages/reversi/src/channels.ts b/packages/reversi/src/channels.ts new file mode 100644 index 0000000..82442cb --- /dev/null +++ b/packages/reversi/src/channels.ts @@ -0,0 +1,297 @@ +/** + * Notification channels: Slack, Microsoft Teams, email, PagerDuty, Opsgenie and a signed generic webhook. Every channel + * sends through the egress guard with a timeout and retries transient failures; a failure is reported to `onError` and + * never changes what Reversi decided. + */ +import { createHmac } from "node:crypto"; +import { guardedFetch, type EgressConfig } from "./egress.js"; +import { sendMail } from "./smtp.js"; + +export type Severity = "info" | "warning" | "critical"; +const RANK: Record = { info: 0, warning: 1, critical: 2 }; + +export interface NotifyMessage { + /** Machine-readable event name (`approval_requested`, `rollback_failed`, ...). */ + event: string; + severity: Severity; + title: string; + text: string; + /** Alerts with the same key update one incident instead of opening new ones. */ + dedup_key?: string; + /** Close the incident named by `dedup_key` (PagerDuty resolve, Opsgenie close). */ + resolve?: boolean; + link?: string; + /** Structured details (names and ids, never argument values). */ + data?: Record; + /** For interactive approval buttons. */ + approval?: { id: string; run_id: string; tool: string }; +} + +export type ChannelConfig = + | { type: "slack"; name: string; url: string; interactive?: boolean } + | { type: "teams"; name: string; url: string } + | { type: "email"; name: string; host: string; port: number; secure?: boolean; starttls?: boolean; user?: string; pass?: string; from: string; to: string[]; verify_tls?: boolean } + | { type: "pagerduty"; name: string; routing_key: string; url?: string } + | { type: "opsgenie"; name: string; api_key: string; region?: "us" | "eu"; url?: string } + | { type: "webhook"; name: string; url: string; secret?: string; headers?: Record }; + +export interface Channel { + name: string; + type: ChannelConfig["type"]; + send(msg: NotifyMessage): Promise; +} + +export interface ChannelDeps { + fetch?: typeof fetch; + egress?: EgressConfig; + /** Pause between retries. */ + sleep?: (ms: number) => Promise; + attempts?: number; +} + +const COLOR: Record = { info: "0076D7", warning: "FFA500", critical: "D70000" }; +const PD_SEV: Record = { info: "info", warning: "warning", critical: "critical" }; +const OG_PRIO: Record = { info: "P5", warning: "P3", critical: "P1" }; + +export function createChannel(cfg: ChannelConfig, deps: ChannelDeps = {}): Channel { + const f = guardedFetch(deps.egress, deps.fetch); + const sleep = deps.sleep ?? ((ms: number) => new Promise((r) => setTimeout(r, ms))); + const attempts = deps.attempts ?? 3; + + /** POST JSON, retrying network errors, 429 and 5xx. 4xx (a wrong URL or key) fails at once. */ + async function post(url: string, body: unknown, headers: Record = {}): Promise { + const text = typeof body === "string" ? body : JSON.stringify(body); + let last = ""; + for (let i = 1; i <= attempts; i++) { + try { + const res = await f(url, { method: "POST", headers: { "content-type": "application/json", ...headers }, body: text, signal: AbortSignal.timeout(10_000) }); + if (res.ok) return; + last = `answered ${res.status}`; + if (res.status < 500 && res.status !== 429) break; + } catch (e) { + last = e instanceof Error ? e.message : String(e); + } + if (i < attempts) await sleep(200 * 2 ** (i - 1)); + } + throw new Error(`${cfg.type} ${cfg.name}: ${last.replace(/(key|token|secret)=[^&\s]+/gi, "$1=[redacted]")}`); + } + + const plain = (m: NotifyMessage) => `${m.title}\n${m.text}${m.link ? `\n${m.link}` : ""}`; + + switch (cfg.type) { + case "slack": + return { + name: cfg.name, + type: cfg.type, + send: (m) => { + const blocks: unknown[] = [ + { type: "header", text: { type: "plain_text", text: m.title.slice(0, 150) } }, + { type: "section", text: { type: "mrkdwn", text: m.text.slice(0, 2900) } }, + ]; + if (m.link) blocks.push({ type: "context", elements: [{ type: "mrkdwn", text: `<${m.link}|Open in the Reversi console>` }] }); + if (cfg.interactive && m.approval) { + blocks.push({ + type: "actions", + block_id: "reversi_approval", + elements: [ + { type: "button", text: { type: "plain_text", text: "Approve" }, style: "primary", action_id: "reversi_approve", value: m.approval.id }, + { type: "button", text: { type: "plain_text", text: "Deny" }, style: "danger", action_id: "reversi_deny", value: m.approval.id }, + ], + }); + } + return post(cfg.url, { text: plain(m), blocks }); + }, + }; + case "teams": + return { + name: cfg.name, + type: cfg.type, + send: (m) => + post(cfg.url, { + "@type": "MessageCard", + "@context": "https://schema.org/extensions", + summary: m.title, + themeColor: COLOR[m.severity], + title: m.title, + text: m.text, + ...(m.link ? { potentialAction: [{ "@type": "OpenUri", name: "Open in Reversi", targets: [{ os: "default", uri: m.link }] }] } : {}), + }), + }; + case "pagerduty": + return { + name: cfg.name, + type: cfg.type, + send: (m) => + post(cfg.url ?? "https://events.pagerduty.com/v2/enqueue", { + routing_key: cfg.routing_key, + event_action: m.resolve ? "resolve" : "trigger", + dedup_key: m.dedup_key ?? `${m.event}:${m.title}`.slice(0, 255), + payload: { summary: m.title.slice(0, 1024), source: "reversi", severity: PD_SEV[m.severity], custom_details: { text: m.text, ...m.data } }, + ...(m.link ? { links: [{ href: m.link, text: "Reversi console" }] } : {}), + }), + }; + case "opsgenie": { + const base = cfg.url ?? (cfg.region === "eu" ? "https://api.eu.opsgenie.com" : "https://api.opsgenie.com"); + const auth = { authorization: `GenieKey ${cfg.api_key}` }; + return { + name: cfg.name, + type: cfg.type, + send: (m) => { + const alias = (m.dedup_key ?? `${m.event}:${m.title}`).slice(0, 512); + if (m.resolve) return post(`${base}/v2/alerts/${encodeURIComponent(alias)}/close?identifierType=alias`, { note: m.text.slice(0, 25000), source: "reversi" }, auth); + return post(`${base}/v2/alerts`, { message: m.title.slice(0, 130), alias, description: m.text.slice(0, 15000), priority: OG_PRIO[m.severity], source: "reversi", details: Object.fromEntries(Object.entries(m.data ?? {}).map(([k, v]) => [k, String(v)])) }, auth); + }, + }; + } + case "email": + return { + name: cfg.name, + type: cfg.type, + send: async (m) => { + let last = ""; + for (let i = 1; i <= attempts; i++) { + try { + await sendMail({ host: cfg.host, port: cfg.port, secure: cfg.secure, starttls: cfg.starttls, user: cfg.user, pass: cfg.pass, from: cfg.from, verify_tls: cfg.verify_tls }, { to: cfg.to, subject: `[Reversi ${m.severity}] ${m.title}`, text: plain(m) }); + return; + } catch (e) { + last = e instanceof Error ? e.message : String(e); + if (/^SMTP .*(5\d\d) /.test(last) || /invalid address/.test(last)) break; + if (i < attempts) await sleep(200 * 2 ** (i - 1)); + } + } + throw new Error(`email ${cfg.name}: ${last}`); + }, + }; + case "webhook": + return { + name: cfg.name, + type: cfg.type, + send: (m) => { + const body = JSON.stringify({ event: m.event, ts: new Date().toISOString(), severity: m.severity, title: m.title, text: m.text, link: m.link, dedup_key: m.dedup_key, resolve: m.resolve === true, data: m.data ?? {} }); + const sig: Record = cfg.secret ? { "x-reversi-signature": `sha256=${createHmac("sha256", cfg.secret).update(body).digest("hex")}` } : {}; + return post(cfg.url, body, { ...(cfg.headers ?? {}), ...sig, "x-reversi-event": m.event }); + }, + }; + } +} + +export function parseChannels(raw: unknown, errors: string[]): ChannelConfig[] { + if (raw === undefined) return []; + if (!Array.isArray(raw)) { + errors.push("`channels` must be a list"); + return []; + } + const names = new Set(); + const out: ChannelConfig[] = []; + raw.forEach((c: unknown, i: number) => { + const w = `channels[${i}]`; + const o = c as Record; + if (typeof o !== "object" || o === null) return void errors.push(`${w}: must be an object`); + if (typeof o.name !== "string" || !o.name) errors.push(`${w}: name is required`); + else if (names.has(o.name)) errors.push(`${w}: duplicate name \`${o.name}\``); + else names.add(o.name); + const need = (keys: string[]) => keys.forEach((k) => typeof o[k] !== "string" && errors.push(`${w}: ${k} is required`)); + const url = (k: string) => { + try { + const u = new URL(String(o[k])); + if (u.protocol !== "https:" && u.protocol !== "http:") errors.push(`${w}: ${k} must be http(s)`); + } catch { + errors.push(`${w}: ${k} is not a valid URL`); + } + }; + switch (o.type) { + case "slack": + case "teams": + need(["url"]); + if (typeof o.url === "string") url("url"); + break; + case "webhook": + need(["url"]); + if (typeof o.url === "string") url("url"); + break; + case "pagerduty": + need(["routing_key"]); + break; + case "opsgenie": + need(["api_key"]); + if (o.region !== undefined && o.region !== "us" && o.region !== "eu") errors.push(`${w}: region must be us or eu`); + break; + case "email": + need(["host", "from"]); + if (!Number.isInteger(o.port)) errors.push(`${w}: port is required`); + if (!Array.isArray(o.to) || o.to.length === 0 || o.to.some((t) => typeof t !== "string")) errors.push(`${w}: to must be a list of addresses`); + break; + default: + errors.push(`${w}: type must be slack, teams, email, pagerduty, opsgenie or webhook`); + } + out.push(o as unknown as ChannelConfig); + }); + return out; +} + +export interface Route { + /** Event names, or `*`. */ + events: string[]; + channels: string[]; + min_severity?: Severity; +} + +export class Notifier { + private readonly byName = new Map(); + constructor( + channels: Channel[], + private readonly routes: Route[], + private readonly onError: (e: Error, channel: string) => void = (e) => console.error(`notification failed: ${e.message}`), + ) { + for (const c of channels) this.byName.set(c.name, c); + for (const r of routes) for (const n of r.channels) if (!this.byName.has(n)) throw new Error(`route names unknown channel \`${n}\``); + } + + channelNames(): string[] { + return [...this.byName.keys()]; + } + + /** Send to every channel a matching route names. Returns once all have finished; never throws. */ + async notify(msg: NotifyMessage): Promise<{ sent: string[]; failed: string[] }> { + const targets = new Set(); + for (const r of this.routes) { + if ((r.events.includes("*") || r.events.includes(msg.event)) && RANK[msg.severity] >= RANK[r.min_severity ?? "info"]) r.channels.forEach((c) => targets.add(c)); + } + const sent: string[] = []; + const failed: string[] = []; + await Promise.all( + [...targets].map(async (n) => { + try { + await (this.byName.get(n) as Channel).send(msg); + sent.push(n); + } catch (e) { + failed.push(n); + this.onError(e instanceof Error ? e : new Error(String(e)), n); + } + }), + ); + return { sent, failed }; + } + + /** Fire and forget, for hooks that must not wait. */ + notifyLater(msg: NotifyMessage): void { + void this.notify(msg); + } +} + +export function parseRoutes(raw: unknown, channels: ChannelConfig[], errors: string[]): Route[] { + if (raw === undefined) return channels.length ? [{ events: ["*"], channels: channels.map((c) => c.name), min_severity: "info" }] : []; + if (!Array.isArray(raw)) { + errors.push("`routes` must be a list"); + return []; + } + const names = new Set(channels.map((c) => c.name)); + return raw.map((r: unknown, i: number) => { + const o = r as Record; + if (!Array.isArray(o?.events) || o.events.length === 0) errors.push(`routes[${i}]: events must be a list (or ["*"])`); + if (!Array.isArray(o?.channels) || o.channels.length === 0) errors.push(`routes[${i}]: channels must be a list`); + else for (const c of o.channels as string[]) if (!names.has(c)) errors.push(`routes[${i}]: unknown channel \`${c}\``); + if (o?.min_severity !== undefined && !["info", "warning", "critical"].includes(o.min_severity as string)) errors.push(`routes[${i}]: min_severity must be info, warning or critical`); + return o as unknown as Route; + }); +} diff --git a/packages/reversi/src/classify.ts b/packages/reversi/src/classify.ts new file mode 100644 index 0000000..36ad837 --- /dev/null +++ b/packages/reversi/src/classify.ts @@ -0,0 +1,34 @@ +/** What a gateway knows about each upstream tool at a glance: classified by a manifest, or not, and what the server claims. */ +import type { Registry } from "./manifest.js"; +import { hintsOf, type Hints, type McpToolInfo } from "./suggest.js"; + +export type ToolClass = "classified" | "unclassified_read_hint" | "unclassified_mutation" | "destructive_no_manifest"; + +export interface ClassifiedTool { + name: string; + class: ToolClass; + effect?: string; + status?: string; + hints: Hints; + /** Short human text for the startup banner. */ + line: string; +} + +export function classifyTools(reg: Registry, tools: McpToolInfo[]): ClassifiedTool[] { + return tools.map((t) => { + const m = reg.get(t.name); + const hints = hintsOf(t); + if (m) return { name: t.name, class: "classified", effect: m.effect, status: m.status, hints, line: `OK ${t.name} - ${m.effect}${m.status !== "approved" ? ` (${m.status})` : ""}` }; + if (hints.read_only === true) return { name: t.name, class: "unclassified_read_hint", hints, line: `NOTE ${t.name} - no manifest; the server says it is read-only (unverified)` }; + if (hints.destructive === true) return { name: t.name, class: "destructive_no_manifest", hints, line: `BLOCK ${t.name} - destructive per the server, no manifest` }; + return { name: t.name, class: "unclassified_mutation", hints, line: `WARN ${t.name} - no manifest; treated as a change that could not be undone` }; + }); +} + +export function banner(rows: ClassifiedTool[], opts: { mode: string; next?: string }): string { + const n = (c: ToolClass) => rows.filter((r) => r.class === c).length; + const lines = [`Reversi gateway running in ${opts.mode} mode.`, "", ...rows.map((r) => r.line), ""]; + lines.push(`${n("classified")} classified, ${n("unclassified_read_hint")} unclassified reads, ${n("unclassified_mutation")} unclassified changes, ${n("destructive_no_manifest")} destructive without a manifest.`); + if (opts.next) lines.push("", opts.next); + return lines.join("\n"); +} diff --git a/packages/reversi/src/config.ts b/packages/reversi/src/config.ts index daa261a..d002c02 100644 --- a/packages/reversi/src/config.ts +++ b/packages/reversi/src/config.ts @@ -1,22 +1,45 @@ -/** Build a fully configured engine from files and environment: shared by the server, gateway and CLI. */ -import { readFileSync } from "node:fs"; -import { extname } from "node:path"; +/** + * Build a fully configured engine from files and environment: shared by the server, gateway and CLI. + * + * `openEngine` wires everything the product does around the core engine: where manifests and policy come from (a folder, + * a signed registry, a URL), the data store, approvals with routing and notifications, capture, alerts, tracing, + * anchoring and egress rules. `reload()` re-reads manifests and policy while running; `watch()` does it when they change. + */ +import { createHash } from "node:crypto"; +import { existsSync, readFileSync, readdirSync, statSync } from "node:fs"; +import { extname, join } from "node:path"; import { parse as parseYaml } from "yaml"; +import { AlertManager } from "./alerts.js"; +import { anchorLedger, type AnchorSink } from "./anchor.js"; import { Approvals } from "./approvals.js"; -import { webhookNotifier } from "./notify.js"; +import { CaptureStore } from "./capture.js"; +import { Notifier, createChannel, type ChannelConfig, type Route } from "./channels.js"; import { ContentChecker } from "./content.js"; import { openData, type Data, type DataOptions } from "./data.js"; +import { Delegations } from "./delegations.js"; +import { guardedFetch, type EgressConfig } from "./egress.js"; +import { createLogger } from "./logger.js"; +import { Reversi } from "./engine.js"; +import type { SigningKey } from "./evidence.js"; +import { approvalNotification, attachNotifications } from "./hub.js"; import { Ledger } from "./ledger.js"; import { loadRegistry } from "./local.js"; -import { Reversi } from "./engine.js"; -import type { Registry } from "./manifest.js"; +import { Registry } from "./manifest.js"; +import { ManifestRegistry, loadRemoteRegistry } from "./registry.js"; +import { webhookNotifier } from "./notify.js"; +import { OtelExporter } from "./otel.js"; import { Policy, mergePolicyDocs, parsePolicy, type PolicyDoc } from "./policy.js"; +import { PolicyVersions } from "./policyversions.js"; +import { LegalHolds } from "./retention.js"; +import { findProjectConfig, loadProjectConfig, type ProjectConfig } from "./projectconfig.js"; export interface EngineConfig { - /** Manifest file or directory. */ + /** A manifest file or folder, a registry folder, or an http(s) URL of a served registry. Empty with `allow_empty_manifests`. */ manifests: string; + /** Start with no manifests (the gateway's observe-first mode). */ + allow_empty_manifests?: boolean; data: DataOptions; - /** Policy file (.yaml/.yml/.json) with rules, limits, tests and an optional `content:` section. */ + /** Policy file (.yaml/.yml/.json), or a folder of versioned policies. */ policy?: string; environment?: string; kill_switch_file?: string; @@ -26,15 +49,50 @@ export interface EngineConfig { approval_ttl_ms?: number; /** Webhook (Slack/Teams incoming-webhook compatible) told about each new approval request. */ approval_webhook?: string; - /** Public URL of the console, linked from the notification. */ + /** Public URL of the console, linked from notifications. */ console_url?: string; + /** PEM file with the public key manifests in a registry must be signed by. */ + manifest_public_key?: string; + /** PEM file with an Ed25519 private key: signs evidence, anchors and retention archives. */ + signing_key?: string; + capture?: { enabled: boolean; ttl_ms?: number; redact_pii?: boolean }; + notifications?: { channels: ChannelConfig[]; routes: Route[] }; + otel?: { endpoint: string; headers?: Record; service_name?: string }; + anchor?: { sinks: AnchorSink[]; interval_ms?: number }; + egress?: EgressConfig; + alerts?: { interval_ms?: number; remind_ms?: number; denial_spike?: number }; + /** Where problems and reloads are reported. Default: stderr. */ + log?: (level: "info" | "warn" | "error", msg: string) => void; +} + +export interface ReloadResult { + changed: boolean; + tools: number; + policy_checksum: string | null; + error?: string; } export interface EngineHandle { reversi: Reversi; approvals: Approvals; + delegations: Delegations; + holds: LegalHolds; registry: Registry; data: Data; + capture?: CaptureStore; + notifier?: Notifier; + alerts?: AlertManager; + otel?: OtelExporter; + signing_key?: SigningKey; + egress: EgressConfig; + config: EngineConfig; + /** Re-read manifests and policy. A failure keeps the old ones and says why. */ + reload(by?: string): Promise; + /** Reload whenever the manifest or policy files change. */ + watch(intervalMs?: number): void; + /** Start background work: alerts, anchoring, trace export. */ + start(opts?: { health?: () => Promise> }): void; + close(): Promise; } export function loadPolicyFile(path: string) { @@ -43,44 +101,247 @@ export function loadPolicyFile(path: string) { return { doc, policy: new Policy(doc), content: doc.content ? new ContentChecker(doc.content) : undefined }; } +async function loadManifests(cfg: EngineConfig, egress: EgressConfig): Promise { + if (!cfg.manifests) { + if (cfg.allow_empty_manifests) return new Registry(); + throw new Error("no manifests: set REVERSI_MANIFESTS, `manifests:` in reversi.config.yaml, or pass --manifests"); + } + const trusted = cfg.manifest_public_key ? readFileSync(cfg.manifest_public_key, "utf8") : undefined; + if (/^https?:\/\//.test(cfg.manifests)) return loadRemoteRegistry(cfg.manifests, { trusted_public_pem: trusted, egress }); + if (existsSync(cfg.manifests) && statSync(cfg.manifests).isDirectory() && ManifestRegistry.isRegistry(cfg.manifests)) return new ManifestRegistry(cfg.manifests, { trusted_public_pem: trusted }).load(); + if (trusted) throw new Error("manifest_public_key is set but the manifests are not in a signed registry"); + return loadRegistry(cfg.manifests); +} + +function loadPolicyDoc(cfg: EngineConfig): { doc: PolicyDoc | undefined; checksum: string | null } { + let file: PolicyDoc | undefined; + let text = ""; + if (cfg.policy) { + if (existsSync(cfg.policy) && statSync(cfg.policy).isDirectory()) { + const pv = new PolicyVersions(cfg.policy); + text = pv.activeText() ?? ""; + if (text) file = parsePolicy(parseYaml(text)); + } else { + text = readFileSync(cfg.policy, "utf8"); + file = parsePolicy(extname(cfg.policy).toLowerCase() === ".json" ? JSON.parse(text) : parseYaml(text)); + } + } + const merged = mergePolicyDocs(file, cfg.extra_policy); + return { doc: merged, checksum: merged ? createHash("sha256").update(text + JSON.stringify(cfg.extra_policy ?? null)).digest("hex") : null }; +} + +/** Modification fingerprint of the files a config depends on. */ +function fingerprint(paths: Array): string { + const parts: string[] = []; + const walk = (p: string, depth: number) => { + try { + const st = statSync(p); + parts.push(`${p}:${st.mtimeMs}:${st.size}`); + if (st.isDirectory() && depth < 4) for (const f of readdirSync(p).slice(0, 2000)) walk(join(p, f), depth + 1); + } catch { + parts.push(`${p}:missing`); + } + }; + for (const p of paths) if (p && !/^https?:\/\//.test(p)) walk(p, 0); + return parts.join("|"); +} + export async function openEngine(cfg: EngineConfig): Promise { - const registry = loadRegistry(cfg.manifests); + const egress = cfg.egress ?? {}; + const log = cfg.log ?? ((level, msg) => console.error(`reversi [${level}] ${msg}`)); + const registry = await loadManifests(cfg, egress); const data = await openData(cfg.data); const ledger = new Ledger(data.ledger); + + const signing_key: SigningKey | undefined = cfg.signing_key + ? await (async () => { + const { createPrivateKey, createPublicKey } = await import("node:crypto"); + const private_pem = readFileSync(cfg.signing_key as string, "utf8"); + return { private_pem, public_pem: createPublicKey(createPrivateKey(private_pem)).export({ type: "spki", format: "pem" }).toString() }; + })() + : undefined; + + // ---- notifications ---- + const onNotifyError = (e: Error, channel: string) => log("warn", `notification to ${channel} failed: ${e.message}`); + const notifier = cfg.notifications?.channels.length + ? new Notifier( + cfg.notifications.channels.map((c) => createChannel(c, { egress })), + cfg.notifications.routes, + onNotifyError, + ) + : undefined; + const legacyWebhook = cfg.approval_webhook ? webhookNotifier({ url: cfg.approval_webhook, console_url: cfg.console_url, fetch: guardedFetch(egress), on_error: (e) => log("warn", `approval webhook failed: ${e.message}`) }) : undefined; + + const delegations = new Delegations(data.kv); const approvals = new Approvals(data.approvals, { - on_decision: (rec, decision) => + delegations, + ttl_ms: cfg.approval_ttl_ms, + on_request: (rec) => { + legacyWebhook?.(rec); + notifier?.notifyLater(approvalNotification(rec, "requested", cfg.console_url)); + }, + on_progress: (rec) => notifier?.notifyLater(approvalNotification(rec, "progress", cfg.console_url)), + on_decision: (rec, decision) => { + const bg = rec.break_glass; + const last = rec.decisions?.[rec.decisions.length - 1]; ledger.append({ ts_ms: Date.now(), run_id: rec.run_id, step: rec.step, - actor: rec.decided_by ?? "approver", + actor: last?.by ?? rec.decided_by ?? "approver", tool: rec.tool, reason: rec.decision_note || `operator ${decision} request ${rec.id}`, event: decision === "approved" ? "approved" : "blocked", - detail: `approval ${rec.id} ${decision} by ${rec.decided_by}${decision === "approved" ? "; the call must be retried" : ""}`, - }), - ttl_ms: cfg.approval_ttl_ms, - on_request: cfg.approval_webhook ? webhookNotifier({ url: cfg.approval_webhook, console_url: cfg.console_url }) : undefined, + detail: bg + ? `BREAK-GLASS: approval ${rec.id} forced by ${bg.by}; evidence: ${bg.evidence}` + : `approval ${rec.id} ${decision} by ${rec.decided_by}${last?.on_behalf_of ? ` (on behalf of ${last.on_behalf_of})` : ""}${(rec.quorum ?? 1) > 1 ? ` [quorum ${rec.quorum}]` : ""}${decision === "approved" ? "; the call must be retried" : ""}`, + }); + notifier?.notifyLater(approvalNotification(rec, "decided", cfg.console_url)); + }, + on_escalate: (rec) => { + ledger.append({ ts_ms: Date.now(), run_id: rec.run_id, step: rec.step, actor: "reversi", tool: rec.tool, reason: `approval ${rec.id} was not answered in time`, event: "escalated", detail: `escalated to ${(rec.required_roles ?? []).join(", ") || "any approver"}` }); + notifier?.notifyLater(approvalNotification(rec, "escalated", cfg.console_url)); + }, }); - const file = cfg.policy ? loadPolicyFile(cfg.policy) : undefined; - const merged = mergePolicyDocs(file?.doc, cfg.extra_policy); - const pol = merged ? { policy: new Policy(merged), content: merged.content ? new ContentChecker(merged.content) : undefined } : undefined; + + const pol = loadPolicyDoc(cfg); + const capture = cfg.capture?.enabled ? new CaptureStore(data.kv, { enabled: true, ...(cfg.capture.ttl_ms ? { ttl_ms: cfg.capture.ttl_ms } : {}), ...(cfg.capture.redact_pii !== undefined ? { redact_pii: cfg.capture.redact_pii } : {}) }) : undefined; const reversi = new Reversi({ registry, ledger, runs: data.runs, approver: approvals.asApprover(), - policy: pol?.policy, - content: pol?.content, + policy: pol.doc ? new Policy(pol.doc, undefined, data.kv) : undefined, + content: pol.doc?.content ? new ContentChecker(pol.doc.content) : undefined, environment: cfg.environment, kill_switch_file: cfg.kill_switch_file, + kv: data.kv, + capture, }); - return { reversi, approvals, registry, data }; + if (notifier) attachNotifications({ reversi, notifier, console_url: cfg.console_url }); + + let currentPolicySum = pol.checksum; + let currentTools = registry.size; + const stoppers: Array<() => void> = []; + + const otel = cfg.otel ? new OtelExporter({ endpoint: cfg.otel.endpoint, headers: cfg.otel.headers, service_name: cfg.otel.service_name, environment: cfg.environment, egress, onError: (e) => log("warn", `trace export failed: ${e.message}`) }, reversi) : undefined; + const alerts = notifier ? new AlertManager(reversi, approvals, data.kv, notifier, { remind_ms: cfg.alerts?.remind_ms, denial_spike: cfg.alerts?.denial_spike, console_url: cfg.console_url }) : undefined; + + const handle: EngineHandle = { + reversi, + approvals, + delegations, + holds: new LegalHolds(data.kv), + get registry() { + return reversi.registry; + }, + data, + capture, + notifier, + alerts, + otel, + signing_key, + egress, + config: cfg, + + async reload(by = "reload") { + try { + const reg = await loadManifests(cfg, egress); + const p = loadPolicyDoc(cfg); + const changed = p.checksum !== currentPolicySum || reg.size !== currentTools || JSON.stringify(reg.list()) !== JSON.stringify(reversi.registry.list()); + if (changed) { + reversi.reload({ registry: reg, policy: p.doc ? new Policy(p.doc, undefined, data.kv) : undefined, content: p.doc?.content ? new ContentChecker(p.doc.content) : undefined }); + currentPolicySum = p.checksum; + currentTools = reg.size; + ledger.append({ ts_ms: Date.now(), run_id: "-", actor: by, tool: "-", reason: "configuration reloaded", event: "reloaded", detail: `${reg.size} manifests; policy ${p.checksum ? p.checksum.slice(0, 8) : "none"}` }); + log("info", `reloaded: ${reg.size} manifests, policy ${p.checksum ? p.checksum.slice(0, 8) : "none"}`); + } + return { changed, tools: reg.size, policy_checksum: p.checksum }; + } catch (e) { + const error = e instanceof Error ? e.message : String(e); + log("error", `reload failed, keeping the previous manifests and policy: ${error}`); + return { changed: false, tools: currentTools, policy_checksum: currentPolicySum, error }; + } + }, + + watch(intervalMs = 2000) { + let last = fingerprint([cfg.manifests, cfg.policy, cfg.manifest_public_key]); + const t = setInterval(() => { + const now = fingerprint([cfg.manifests, cfg.policy, cfg.manifest_public_key]); + if (now !== last) { + last = now; + void handle.reload("file watcher"); + } + }, intervalMs); + t.unref(); + stoppers.push(() => clearInterval(t)); + }, + + start(opts = {}) { + otel?.start(); + if (alerts) { + alerts.setHealth(opts.health); + alerts.start(cfg.alerts?.interval_ms ?? 60_000); + stoppers.push(() => alerts.stop()); + } + if (cfg.anchor?.sinks.length) { + const f = guardedFetch(egress); + const run = () => { + void anchorLedger(ledger, cfg.anchor?.sinks ?? [], { key: signing_key, egress, fetch: f }).then((r) => r.errors.forEach((e) => log("warn", `anchor sink failed: ${e}`))); + }; + run(); + const t = setInterval(run, cfg.anchor.interval_ms ?? 3_600_000); + t.unref(); + stoppers.push(() => clearInterval(t)); + } + }, + + async close() { + for (const stop of stoppers.splice(0)) stop(); + await otel?.stop(); + data.close(); + }, + }; + return handle; } -/** Engine settings from the environment (REVERSI_MANIFESTS, REVERSI_POLICY, REVERSI_ENV, REVERSI_KILL_FILE, plus the data variables). */ -export function engineConfigFromEnv(env: Record, dataOpts: DataOptions): EngineConfig { - const manifests = env.REVERSI_MANIFESTS; - if (!manifests) throw new Error("REVERSI_MANIFESTS is required (a manifest file or directory)"); - return { manifests, data: dataOpts, policy: env.REVERSI_POLICY, environment: env.REVERSI_ENV, kill_switch_file: env.REVERSI_KILL_FILE, approval_webhook: env.REVERSI_APPROVAL_WEBHOOK, console_url: env.REVERSI_CONSOLE_URL }; +/** Engine settings from the environment (REVERSI_MANIFESTS, REVERSI_POLICY, ...), over the project file if there is one. */ +export function engineConfigFromEnv(env: Record, dataOpts: DataOptions, project?: ProjectConfig): EngineConfig { + const manifests = env.REVERSI_MANIFESTS ?? project?.manifests; + if (!manifests) throw new Error("REVERSI_MANIFESTS is required (a manifest file or directory), or set `manifests:` in reversi.config.yaml"); + return { + manifests, + data: dataOpts, + policy: env.REVERSI_POLICY ?? project?.policy, + environment: env.REVERSI_ENV ?? project?.environment, + kill_switch_file: env.REVERSI_KILL_FILE ?? project?.kill_switch_file, + approval_webhook: env.REVERSI_APPROVAL_WEBHOOK ?? project?.approval?.webhook, + console_url: env.REVERSI_CONSOLE_URL ?? project?.approval?.console_url, + approval_ttl_ms: project?.approval?.ttl_ms, + manifest_public_key: env.REVERSI_MANIFEST_PUBLIC_KEY ?? project?.manifest_public_key, + signing_key: env.REVERSI_SIGNING_KEY ?? project?.signing_key, + capture: env.REVERSI_CAPTURE === "1" || env.REVERSI_CAPTURE === "on" ? { enabled: true, ttl_ms: env.REVERSI_CAPTURE_TTL_DAYS ? Number(env.REVERSI_CAPTURE_TTL_DAYS) * 86_400_000 : project?.capture?.ttl_ms, redact_pii: project?.capture?.redact_pii } : project?.capture, + notifications: project?.notifications, + otel: env.REVERSI_OTLP_ENDPOINT ? { endpoint: env.REVERSI_OTLP_ENDPOINT, headers: project?.otel?.headers } : project?.otel, + anchor: project?.anchor, + egress: { ...(project?.egress ?? {}), ...(env.REVERSI_EGRESS_ALLOW ? { allow: env.REVERSI_EGRESS_ALLOW.split(",").map((s) => s.trim()).filter(Boolean) } : {}), ...(env.REVERSI_EGRESS_BLOCK_PRIVATE === "1" ? { block_private: true } : {}), ...(env.REVERSI_REGION ? { region: env.REVERSI_REGION } : {}) }, + alerts: project?.alerts, + log: logFrom(project?.log), + }; } + +const LEVEL_RANK = { debug: 10, info: 20, warn: 30, error: 40 } as const; + +/** The `log:` section of the project file: a minimum level, and JSON lines (secrets redacted) instead of plain text. */ +function logFrom(cfg: ProjectConfig["log"]): EngineConfig["log"] { + if (!cfg) return undefined; + const min = LEVEL_RANK[cfg.level ?? "info"]; + const json = cfg.format === "json" ? createLogger({ level: cfg.level ?? "info", write: (l) => process.stderr.write(l + "\n") }) : undefined; + return (level, msg) => { + if (LEVEL_RANK[level] < min) return; + if (json) json[level](msg); + else console.error(`reversi [${level}] ${msg}`); + }; +} + +export { findProjectConfig, loadProjectConfig }; diff --git a/packages/reversi/src/content.ts b/packages/reversi/src/content.ts index 4e56c1a..aef1a28 100644 --- a/packages/reversi/src/content.ts +++ b/packages/reversi/src/content.ts @@ -7,11 +7,14 @@ * they do not prove a call is safe. */ import { globAny } from "./glob.js"; +import { detectPii } from "./pii.js"; import { findSecrets } from "./redact.js"; export type CheckAction = "flag" | "require_approval" | "deny"; export interface ContentConfig { + /** Personal data (emails, phone numbers, card numbers, SSNs, IBANs) in arguments. Off unless set. */ + pii?: CheckAction; secrets: CheckAction; encoded_payloads: CheckAction; injection: CheckAction; @@ -36,7 +39,7 @@ export const DEFAULT_CONTENT: ContentConfig = { }; export interface Finding { - kind: "secret" | "encoded_payload" | "injection" | "cumulative"; + kind: "secret" | "encoded_payload" | "injection" | "cumulative" | "pii"; action: CheckAction; detail: string; } @@ -140,6 +143,10 @@ export class ContentChecker { for (const h of direct(s)) { add({ kind: h.kind, action: this.config[h.kind === "secret" ? "secrets" : "injection"], detail: h.what }); } + if (this.config.pii) { + const kinds = detectPii(s); + if (kinds.length) add({ kind: "pii", action: this.config.pii, detail: `personal data (${kinds.join(", ")})` }); + } let layer = [{ via: "", text: s }]; for (let depth = 1; depth <= MAX_DEPTH; depth++) { const next: Array<{ via: string; text: string }> = []; diff --git a/packages/reversi/src/counters.ts b/packages/reversi/src/counters.ts new file mode 100644 index 0000000..2e3f6f1 --- /dev/null +++ b/packages/reversi/src/counters.ts @@ -0,0 +1,42 @@ +/** + * Sliding-window counters for rate limits and budgets. With a {@link KvStore} the windows are shared by every + * process using the same data store; without one they live in memory. + */ +import { kvUpdate, type KvStore } from "./kv.js"; + +type Events = Array<[number, number]>; // [timestamp ms, amount] +const NS = "windows"; +const MAX_EVENTS = 20_000; + +export class WindowCounters { + private readonly mem = new Map(); + constructor( + private readonly kv: KvStore | undefined, + private readonly clock: () => number = Date.now, + ) {} + + private read(key: string): Events { + return this.kv ? ((this.kv.get(NS, key)?.value) ?? []) : (this.mem.get(key) ?? []); + } + + /** Calls and summed amount inside the last `windowMs`. */ + total(key: string, windowMs: number): { count: number; sum: number } { + const horizon = this.clock() - windowMs; + let count = 0; + let sum = 0; + for (const [t, a] of this.read(key)) { + if (t > horizon) { + count++; + sum += a; + } + } + return { count, sum }; + } + + add(key: string, amount: number, windowMs: number): void { + const now = this.clock(); + const trim = (ev: Events | undefined) => [...(ev ?? []).filter(([t]) => t > now - windowMs), [now, amount] as [number, number]].slice(-MAX_EVENTS); + if (this.kv) kvUpdate(this.kv, NS, key, trim); + else this.mem.set(key, trim(this.mem.get(key))); + } +} diff --git a/packages/reversi/src/data.ts b/packages/reversi/src/data.ts index d0102d3..24c9830 100644 --- a/packages/reversi/src/data.ts +++ b/packages/reversi/src/data.ts @@ -5,6 +5,7 @@ */ import { join } from "node:path"; import { FileApprovalStore, type ApprovalStore } from "./approvals.js"; +import { FileKvStore, type KvStore } from "./kv.js"; import { JsonlLedgerStore, type LedgerStore } from "./ledger.js"; import { FileRunStore, type RunStore } from "./store.js"; @@ -23,6 +24,8 @@ export interface Data { ledger: LedgerStore; runs: RunStore; approvals: ApprovalStore; + /** Settings and state other than the ledger, run state and approvals. */ + kv: KvStore; close(): void; } @@ -42,19 +45,20 @@ export async function openData(opts: DataOptions): Promise { if (kind === "sqlite") { const { openSqliteStores } = await import("./sqlite.js"); const s = openSqliteStores(join(opts.dir, "reversi.db")); - return { kind, ledger: s.ledger, runs: s.runs, approvals: s.approvals, close: () => s.db.close() }; + return { kind, ledger: s.ledger, runs: s.runs, approvals: s.approvals, kv: s.kv, close: () => s.db.close() }; } if (kind === "postgres") { if (!opts.postgres_url) throw new Error("the postgres store needs a connection URL (REVERSI_DATABASE_URL or --database-url)"); const { openPostgresStores } = await import("./postgres.js"); const s = openPostgresStores(opts.postgres_url); - return { kind, ledger: s.ledger, runs: s.runs, approvals: s.approvals, close: () => s.pg.close() }; + return { kind, ledger: s.ledger, runs: s.runs, approvals: s.approvals, kv: s.kv, close: () => s.pg.close() }; } return { kind, ledger: new JsonlLedgerStore(join(opts.dir, "ledger.jsonl")), runs: new FileRunStore(join(opts.dir, "runs")), approvals: new FileApprovalStore(join(opts.dir, "approvals.json")), + kv: new FileKvStore(join(opts.dir, "settings.json")), close: () => {}, }; } diff --git a/packages/reversi/src/delegations.ts b/packages/reversi/src/delegations.ts new file mode 100644 index 0000000..fef1b03 --- /dev/null +++ b/packages/reversi/src/delegations.ts @@ -0,0 +1,55 @@ +/** + * Delegation: an approver hands their approval authority to someone else for a while (leave, rotation). The delegate + * approves as themselves, "on behalf of" the delegator, who may not also approve the same request; every use is + * recorded in the decision. + */ +import { randomUUID } from "node:crypto"; +import type { KvStore } from "./kv.js"; + +export interface Delegation { + id: string; + from: string; + to: string; + /** Groups delegated. Empty: all of the delegator's groups. */ + groups: string[]; + reason: string; + created_ms: number; + until_ms: number; + revoked?: boolean; +} + +const NS = "delegations"; + +export class Delegations { + constructor( + private readonly kv: KvStore, + private readonly now: () => number = Date.now, + ) {} + + grant(d: { from: string; to: string; until_ms: number; groups?: string[]; reason: string }): Delegation { + if (!d.from || !d.to) throw new Error("a delegation needs from and to"); + if (d.from === d.to) throw new Error("you cannot delegate to yourself"); + if (!(d.until_ms > this.now())) throw new Error("a delegation must end in the future"); + if (d.until_ms - this.now() > 90 * 86_400_000) throw new Error("a delegation can last at most 90 days"); + if (!d.reason.trim()) throw new Error("a delegation needs a reason"); + const rec: Delegation = { id: `dlg_${randomUUID()}`, from: d.from, to: d.to, groups: d.groups ?? [], reason: d.reason, created_ms: this.now(), until_ms: d.until_ms }; + this.kv.put(NS, rec.id, rec); + return rec; + } + + revoke(id: string): boolean { + const e = this.kv.get(NS, id); + if (!e) return false; + this.kv.put(NS, id, { ...e.value, revoked: true }); + return true; + } + + list(): Delegation[] { + return this.kv.list(NS).map((e) => e.value); + } + + /** Delegations in force that let `to` act for someone else. */ + activeFor(to: string): Delegation[] { + return this.list().filter((d) => d.to === to && !d.revoked && d.until_ms > this.now()); + } +} diff --git a/packages/reversi/src/diff.ts b/packages/reversi/src/diff.ts new file mode 100644 index 0000000..2e51d22 --- /dev/null +++ b/packages/reversi/src/diff.ts @@ -0,0 +1,53 @@ +/** A small line diff (longest common subsequence) for comparing manifest and policy versions. */ + +export interface DiffLine { + op: " " | "+" | "-"; + text: string; +} + +export function diffLines(a: string, b: string): DiffLine[] { + const x = a.split("\n"); + const y = b.split("\n"); + const n = x.length; + const m = y.length; + // lcs[i][j] = length of the longest common subsequence of x[i..] and y[j..] + const lcs: number[][] = Array.from({ length: n + 1 }, () => new Array(m + 1).fill(0)); + for (let i = n - 1; i >= 0; i--) for (let j = m - 1; j >= 0; j--) lcs[i]![j] = x[i] === y[j] ? (lcs[i + 1]![j + 1] as number) + 1 : Math.max(lcs[i + 1]![j] as number, lcs[i]![j + 1] as number); + const out: DiffLine[] = []; + let i = 0; + let j = 0; + while (i < n && j < m) { + if (x[i] === y[j]) { + out.push({ op: " ", text: x[i] as string }); + i++; + j++; + } else if ((lcs[i + 1]![j] as number) >= (lcs[i]![j + 1] as number)) out.push({ op: "-", text: x[i++] as string }); + else out.push({ op: "+", text: y[j++] as string }); + } + while (i < n) out.push({ op: "-", text: x[i++] as string }); + while (j < m) out.push({ op: "+", text: y[j++] as string }); + return out; +} + +/** The changed lines with `context` lines around them, in unified-diff style (without hunk headers). */ +export function renderDiff(a: string, b: string, context = 2): string { + const d = diffLines(a, b); + const keep = new Set(); + d.forEach((l, i) => { + if (l.op !== " ") for (let k = Math.max(0, i - context); k <= Math.min(d.length - 1, i + context); k++) keep.add(k); + }); + const out: string[] = []; + let gap = false; + d.forEach((l, i) => { + if (!keep.has(i)) { + gap = true; + return; + } + if (gap && out.length) out.push("..."); + gap = false; + out.push(`${l.op}${l.text}`); + }); + return out.join("\n"); +} + +export const hasChanges = (a: string, b: string) => a !== b; diff --git a/packages/reversi/src/egress.ts b/packages/reversi/src/egress.ts new file mode 100644 index 0000000..651c517 --- /dev/null +++ b/packages/reversi/src/egress.ts @@ -0,0 +1,86 @@ +/** + * Control over where Reversi itself sends data: webhooks, anchors, notifications, trace export and the upstream servers + * the gateway connects to. An allowlist of hosts (also the data-residency control: only destinations in your region), + * and protection against being pointed at internal addresses (SSRF), with the cloud metadata endpoints always refused. + */ +import { lookup } from "node:dns/promises"; +import { isIP } from "node:net"; +import { globMatch } from "./glob.js"; + +export interface EgressConfig { + /** Host globs destinations must match (`*.eu.example.com`). Unset: any host. */ + allow?: string[]; + /** Refuse loopback, private and link-local addresses. Off by default so local development works. */ + block_private?: boolean; + /** Allow cloud metadata addresses (169.254.169.254 and friends). Never needed; off. */ + allow_metadata?: boolean; + /** Where this deployment's data lives, recorded in bundles and reports. */ + region?: string; +} + +export class EgressError extends Error { + constructor(message: string) { + super(message); + this.name = "EgressError"; + } +} + +function ipv4(ip: string): number[] | null { + const m = /^(\d{1,3})\.(\d{1,3})\.(\d{1,3})\.(\d{1,3})$/.exec(ip); + return m ? m.slice(1).map(Number) : null; +} + +export function isMetadataAddress(ip: string): boolean { + const v = ip.toLowerCase().replace(/^::ffff:/, ""); + return v === "169.254.169.254" || v === "100.100.100.200" || v === "fd00:ec2::254" || v === "192.0.0.192"; +} + +export function isPrivateAddress(ipIn: string): boolean { + const ip = ipIn.toLowerCase().replace(/^::ffff:/, ""); + const v4 = ipv4(ip); + if (v4) { + const [a, b] = v4 as [number, number, number, number]; + return a === 10 || a === 127 || a === 0 || (a === 172 && b >= 16 && b <= 31) || (a === 192 && b === 168) || (a === 169 && b === 254) || (a === 100 && b >= 64 && b <= 127); + } + return ip === "::1" || ip === "::" || ip.startsWith("fc") || ip.startsWith("fd") || ip.startsWith("fe8") || ip.startsWith("fe9") || ip.startsWith("fea") || ip.startsWith("feb"); +} + +/** Throws EgressError unless the URL may be contacted. Resolves the host, so a name that points inside is caught too. */ +export async function checkUrl(cfg: EgressConfig | undefined, raw: string | URL): Promise { + const url = typeof raw === "string" ? new URL(raw) : raw; + if (url.protocol !== "http:" && url.protocol !== "https:") throw new EgressError(`only http(s) destinations are allowed, not ${url.protocol}`); + const host = url.hostname.replace(/^\[|\]$/g, "").toLowerCase(); + if (cfg?.allow?.length && !cfg.allow.some((g) => globMatch(g.toLowerCase(), host) || globMatch(g.toLowerCase(), url.host.toLowerCase()))) { + throw new EgressError(`${host} is not in the egress allowlist${cfg.region ? ` (region ${cfg.region})` : ""}`); + } + const addrs = isIP(host) ? [host] : (await lookup(host, { all: true }).catch(() => [])).map((a) => a.address); + for (const a of addrs) { + if (!cfg?.allow_metadata && isMetadataAddress(a)) throw new EgressError(`${host} resolves to a cloud metadata address (${a}), which is never allowed`); + if (cfg?.block_private && isPrivateAddress(a)) throw new EgressError(`${host} resolves to a private or loopback address (${a}) and egress.block_private is on`); + } + return url; +} + +/** `fetch` that checks every destination first, and every redirect (up to 3) the same way. */ +export function guardedFetch(cfg: EgressConfig | undefined, f: typeof fetch = fetch): typeof fetch { + return (async (input: string | URL | Request, init?: RequestInit) => { + let url = await checkUrl(cfg, typeof input === "string" || input instanceof URL ? input : input.url); + for (let hop = 0; ; hop++) { + const res = await f(url, { ...init, redirect: "manual" }); + if (res.status >= 300 && res.status < 400 && res.headers.get("location")) { + if (hop >= 3) throw new EgressError("too many redirects"); + url = await checkUrl(cfg, new URL(res.headers.get("location") as string, url)); + continue; + } + return res; + } + }) as typeof fetch; +} + +export function egressFromEnv(env: Record = process.env): EgressConfig { + return { + allow: env.REVERSI_EGRESS_ALLOW ? env.REVERSI_EGRESS_ALLOW.split(",").map((s) => s.trim()).filter(Boolean) : undefined, + block_private: env.REVERSI_EGRESS_BLOCK_PRIVATE === "1", + region: env.REVERSI_REGION, + }; +} diff --git a/packages/reversi/src/engine.ts b/packages/reversi/src/engine.ts index 83e233e..ca1c6aa 100644 --- a/packages/reversi/src/engine.ts +++ b/packages/reversi/src/engine.ts @@ -18,7 +18,11 @@ import type { ContentChecker } from "./content.js"; import { Ledger, type LedgerEvent } from "./ledger.js"; import { redactSecrets } from "./redact.js"; import { Registry, resolveRef } from "./manifest.js"; -import type { Policy, PolicyInput } from "./policy.js"; +import type { Verification } from "./manifest.js"; +import type { CaptureStore } from "./capture.js"; +import type { KvStore } from "./kv.js"; +import type { ApprovalRequirement, Policy, PolicyInput } from "./policy.js"; +import { globAny } from "./glob.js"; import { SealedScope } from "./scope.js"; import { MemoryRunStore, @@ -68,6 +72,14 @@ export interface ApprovalRequest { triggers?: string[]; /** Names of the arguments (never values). */ arg_keys?: string[]; + /** The actor of the run, so the requester cannot approve their own request. */ + requested_by?: string; + /** Roles or groups that may approve; empty means any approver. */ + required_roles?: string[]; + quorum?: number; + timeout_ms?: number; + on_timeout?: "deny" | "escalate"; + escalate_to?: string[]; } /** `true`/`false`, or a decision with a message that is shown to the caller when not approved. */ export type ApprovalDecision = boolean | { approved: boolean; message?: string }; @@ -86,6 +98,25 @@ export interface ReversiOptions { environment?: string; /** If this file exists, every new call is blocked (rollback still works). */ kill_switch_file?: string; + /** Opt-in capture of redacted arguments and results, for replay, simulation and impact previews. */ + capture?: CaptureStore; + /** Shared settings store: halts are kept here so every process using the same data store sees them. */ + kv?: KvStore; +} + +/** What a halt applies to. An empty scope halts everything. */ +export interface HaltScope { + tool?: string; + actor?: string; + tenant?: string; + environment?: string; +} +export interface Halt { + id: string; + reason: string; + scope: HaltScope; + by: string; + ts_ms: number; } export interface StartRunOptions { @@ -95,6 +126,8 @@ export interface StartRunOptions { /** `enforce` blocks violations. `observe` journals them as `would_block` and lets the call run. */ mode?: Mode; reason?: string; + /** Customer or team the run belongs to. */ + tenant?: string; } export interface BeginInput { @@ -103,6 +136,10 @@ export interface BeginInput { reason: string; /** Steps this call depends on beyond the manifest's `requires`. */ depends_on?: number[]; + /** Violations found by the caller before asking (for example schema validation in the gateway). Treated like policy denials. */ + extra_violations?: string[]; + /** Notes for the ledger record of the intent. */ + extra_notes?: string[]; } export type BeginResult = @@ -111,7 +148,14 @@ export type BeginResult = | { decision: "block"; reasons: string[] }; export type Outcome = - | { status: "succeeded"; result?: unknown } + | { + status: "succeeded"; + result?: unknown; + /** Appended to the ledger detail. */ + notes?: string[]; + /** The result does not match what the tool promised: its `$result` references cannot be trusted for compensation. */ + invalid_result?: boolean; + } | { status: "failed"; error?: string } | { status: "uncertain"; error?: string }; @@ -125,6 +169,11 @@ export interface PlanAction { step: number; tool: string; kind: PlanKind; + /** What the undo will not take back. */ + residue?: string[]; + compensation_kind?: "inverse" | "compensation"; + /** The undo is read back before it counts as complete. */ + verified?: boolean; compensation_tool?: string; args?: Record; idempotency_key?: string; @@ -167,6 +216,8 @@ export interface RollbackReport { skipped: number[]; failed: Array<{ step: number; error: string }>; unrecoverable: number[]; + /** Compensations that ran but whose declared read-back could not be performed (no verifier) or was not possible. */ + unverified: number[]; unresolved: number[]; unmanifested: number[]; deferred: Array<{ step: number; blocked_by: number[] }>; @@ -179,6 +230,52 @@ export function idempotencyKey(run_id: string, step: number, tool: string): stri return sha256Hex(canonicalize([run_id, step, tool])); } +/** Several approval requirements on one call: any listed role may approve, the biggest quorum wins, the shortest timeout wins. */ +export function mergeRequirements(reqs: ApprovalRequirement[]): Pick { + if (reqs.length === 0) return {}; + const timeouts = reqs.map((r) => r.timeout_ms).filter((t): t is number => t !== undefined); + return { + required_roles: [...new Set(reqs.flatMap((r) => r.roles))], + quorum: Math.max(...reqs.map((r) => r.quorum)), + ...(timeouts.length ? { timeout_ms: Math.min(...timeouts) } : {}), + on_timeout: reqs.some((r) => r.on_timeout === "deny" || r.on_timeout === undefined) ? "deny" : "escalate", + escalate_to: [...new Set(reqs.flatMap((r) => r.escalate_to ?? []))], + }; +} + +/** `{{args.a.b}}` placeholders rendered from the call arguments; undefined if any is missing or not a scalar. */ +export function renderIdempotencyKey(template: string, args: unknown): string | undefined { + let missing = false; + const out = template.replace(/\{\{\s*(args(?:\.[A-Za-z0-9_-]+)+)\s*\}\}/g, (_m, path: string) => { + const v = resolveRef(`$${path}`, "args", args); + if (typeof v === "string" || typeof v === "number" || typeof v === "boolean") return String(v); + missing = true; + return ""; + }); + return missing ? undefined : out; +} + +const LIVE_STATUSES = new Set(["intended", "succeeded", "uncertain", "compensating", "compensation_failed"]); + +/** Compare a verification result with its `expect` (dotted path -> value). Returns a description of the first mismatch, or null. */ +export function expectMismatch(actual: unknown, expect: Record): string | null { + for (const [path, want] of Object.entries(expect)) { + let cur: unknown = actual; + for (const seg of path.split(".")) { + if (Array.isArray(cur) && /^\d+$/.test(seg)) cur = cur[Number(seg)]; + else if (typeof cur === "object" && cur !== null && Object.prototype.hasOwnProperty.call(cur, seg)) cur = (cur as Record)[seg]; + else { + cur = undefined; + break; + } + } + if (canonicalize(cur === undefined ? null : cur) !== canonicalize(want === undefined ? null : want) || (cur === undefined && want !== undefined)) { + return `\`${path}\` is ${cur === undefined ? "missing" : JSON.stringify(cur)}, expected ${JSON.stringify(want)}`; + } + } + return null; +} + function isArgObject(v: unknown): v is Record { return typeof v === "object" && v !== null && !Array.isArray(v); } @@ -228,6 +325,9 @@ function classify(state: RunState, s: StepState): PlanAction | null { args: s.handle ?? {}, idempotency_key: idempotencyKey(state.run_id, s.step, s.tool), needs_approval: s.compensation_requires_approval, + ...(s.residue?.length ? { residue: s.residue } : {}), + ...(s.compensation_kind ? { compensation_kind: s.compensation_kind } : {}), + ...(s.verification ? { verified: true } : {}), }; } case "skip": @@ -251,18 +351,22 @@ function classify(state: RunState, s: StepState): PlanAction | null { } export class Reversi { - readonly registry: Registry; + registry: Registry; readonly ledger: Ledger; readonly runs: RunStore; - readonly policy: Policy | undefined; - readonly content: ContentChecker | undefined; + policy: Policy | undefined; + content: ContentChecker | undefined; readonly environment: string; private readonly approver: Approver | undefined; private readonly clock: () => number; private readonly killFile: string | undefined; - private haltReason: string | null = null; + private readonly memoryHalts = new Map(); + private readonly kv: KvStore | undefined; + readonly capture: CaptureStore | undefined; constructor(opts: ReversiOptions) { + this.kv = opts.kv; + this.capture = opts.capture; this.policy = opts.policy; this.content = opts.content; this.environment = opts.environment ?? "default"; @@ -288,6 +392,7 @@ export class Reversi { parent_seal: opts.scope.parentSeal, }, created_ms: this.clock(), + ...(opts.tenant ? { tenant: opts.tenant } : {}), steps: [], }; this.ledger.append({ @@ -313,16 +418,56 @@ export class Reversi { return this.runs.list(); } - /** Emergency stop: block every new call until {@link Reversi.resume}. Rollbacks are not affected. */ - halt(reason: string): void { - this.haltReason = reason || "halted by operator"; + /** + * Swap in new manifests, policy or content checks without a restart (hot reload). Calls already in flight finish under + * the rules they started with; the next call sees the new ones. + */ + reload(parts: { registry?: Registry; policy?: Policy | undefined; content?: ContentChecker | undefined }): void { + if (parts.registry) this.registry = parts.registry; + if ("policy" in parts) this.policy = parts.policy; + if ("content" in parts) this.content = parts.content; + } + + /** + * Emergency stop: block new calls until {@link Reversi.resume}. With no scope it stops everything; a scope narrows it to + * a tool (glob), actor, tenant or environment. Rollbacks are never affected. Halts live in the shared settings store when + * there is one, so every process sees them. + */ + halt(reason: string, scope: HaltScope = {}, by = "operator"): Halt { + const h: Halt = { id: `halt_${randomUUID()}`, reason: reason || "halted by operator", scope, by, ts_ms: this.clock() }; + if (this.kv) this.kv.put("halts", h.id, h); + else this.memoryHalts.set(h.id, h); + return h; + } + + /** Lift one halt by id, or all of them. Returns how many were lifted. */ + resume(id?: string): number { + const all = this.halts().filter((h) => !id || h.id === id); + for (const h of all) { + if (this.kv) this.kv.delete("halts", h.id); + else this.memoryHalts.delete(h.id); + } + return all.length; } - resume(): void { - this.haltReason = null; + + halts(): Halt[] { + return this.kv ? this.kv.list("halts").map((e) => e.value) : [...this.memoryHalts.values()]; } - /** The reason new calls are blocked, or null. Also true while the kill-switch file exists. */ - halted(): string | null { - if (this.haltReason !== null) return this.haltReason; + + /** The reason new calls (matching `ctx`, if given) are blocked, or null. The kill-switch file halts everything. */ + halted(ctx?: { tool?: string; actor?: string; tenant?: string; environment?: string }): string | null { + for (const h of this.halts()) { + const s = h.scope; + const applies = + (s.tool === undefined || (ctx?.tool !== undefined && globAny(s.tool, ctx.tool))) && + (s.actor === undefined || (ctx?.actor !== undefined && globAny(s.actor, ctx.actor))) && + (s.tenant === undefined || (ctx?.tenant !== undefined && globAny(s.tenant, ctx.tenant))) && + (s.environment === undefined || (ctx?.environment !== undefined && globAny(s.environment, ctx.environment))); + if (applies) { + const scope = Object.entries(s).map(([k, v]) => `${k}=${v}`).join(", "); + return scope ? `${h.reason} (scope: ${scope})` : h.reason; + } + } if (this.killFile && existsSync(this.killFile)) return `kill switch file ${this.killFile} exists`; return null; } @@ -411,6 +556,29 @@ export class Run { else violations.push(`depends_on step ${d} does not exist`); } + if (manifest?.status === "draft") violations.push(`the manifest for \`${call.tool}\` is a draft: publish it before it can be enforced`); + if (call.extra_violations?.length) violations.push(...call.extra_violations); + + // Duplicate-call guard: the same rendered idempotency key must not run twice while the first still stands. + let idemKey: string | undefined; + if (manifest?.idempotency) { + const rendered = renderIdempotencyKey(manifest.idempotency.key, call.args); + if (rendered === undefined) { + violations.push(`idempotency key \`${manifest.idempotency.key}\` needs arguments that are missing or not simple values`); + } else { + idemKey = `${call.tool}|${rendered}`; + const scope = manifest.idempotency.scope; + const runs = scope === "run" ? [state] : this.rv.runIds().map((id) => this.rv.runs.get(id)).filter((r): r is RunState => !!r && (scope === "global" || r.actor === state.actor)); + for (const r of runs) { + const dup = r.steps.find((x) => x.idem_key === idemKey && LIVE_STATUSES.has(x.status)); + if (dup) { + violations.push(`duplicate call: idempotency key matches step ${dup.step} of run ${r.run_id} (${dup.status})${dup.status === "intended" ? ", which is still in flight" : ""}`); + break; + } + } + } + } + // Arguments with no canonical form (Infinity, NaN, BigInt...) would all share one hash, which would let an // approval granted for one call cover another. Treat them as a violation and never match an approval. let argsHash: string; @@ -421,8 +589,9 @@ export class Run { violations.push(`arguments cannot be canonicalized: ${e instanceof Error ? e.message : "unsupported value"}`); } const triggers: string[] = []; - const notes: string[] = []; - const halted = this.rv.halted(); + const notes: string[] = [...(call.extra_notes ?? [])]; + if (manifest?.status === "deprecated") notes.push(`manifest deprecated${manifest.replaced_by ? `; use ${manifest.replaced_by}` : ""}`); + const halted = this.rv.halted({ tool: call.tool, actor: state.actor, tenant: state.tenant, environment: this.rv.environment }); if (halted) violations.push(`kill switch: ${halted}`); const policyInput: PolicyInput = { tool: call.tool, @@ -432,11 +601,26 @@ export class Run { run_id: this.run_id, environment: this.rv.environment, args: call.args, + tenant: state.tenant, + data_class: manifest?.risk?.data_class, + blast_radius: manifest?.risk?.blast_radius, }; + const requirements: ApprovalRequirement[] = []; + let riskScore: number | undefined; if (this.rv.policy) { const d = this.rv.policy.evaluate(policyInput); violations.push(...d.deny); triggers.push(...d.approve); + requirements.push(...d.requirements); + notes.push(...d.warnings); + if (d.risk) { + riskScore = d.risk.score; + notes.push(`risk ${d.risk.score}${d.risk.factors.length ? ` (${d.risk.factors.join(", ")})` : ""}`); + } + } + if (manifest?.approval_rules) { + const a = manifest.approval_rules; + requirements.push({ roles: a.roles, quorum: a.quorum, timeout_ms: a.timeout_ms, on_timeout: a.on_timeout, escalate_to: a.escalate_to, source: `manifest \`${manifest.tool}\`` }); } if (this.rv.content) { const succeeded = state.steps.filter((x) => x.status === "succeeded").map((x) => x.tool); @@ -453,6 +637,7 @@ export class Run { const enforce = state.mode === "enforce"; if (violations.length && enforce) { this.emit("blocked", null, call.tool, call.args, call.reason, violations.join("; ")); + this.captureCall(state, call, manifest?.audit?.redaction, null, "block", violations, riskScore, manifest?.version); return { decision: "block", reasons: violations }; } const observed = [...violations]; @@ -469,6 +654,8 @@ export class Run { args_hash: argsHash, triggers, arg_keys: isArgObject(call.args) ? Object.keys(call.args).slice(0, 50) : [], + requested_by: state.actor, + ...mergeRequirements(requirements), }) : false; const approved = typeof decision === "boolean" ? decision : decision.approved; @@ -483,6 +670,7 @@ export class Run { : "approval required but no approver is configured"; if (enforce) { this.emit("blocked", null, call.tool, call.args, call.reason, `${msg} [${triggers.join("; ")}]`); + this.captureCall(state, call, manifest?.audit?.redaction, null, "block", [msg], riskScore, manifest?.version); return { decision: "block", reasons: [msg] }; } observed.push(msg); @@ -522,11 +710,83 @@ export class Run { handle_missing: missing, error: null, ts_ms: this.rv.internals.now(), + ...(manifest ? { manifest_version: manifest.version } : {}), + ...(manifest?.compensation + ? { + compensation_kind: manifest.compensation.kind, + residue: manifest.compensation.residue, + ...(manifest.compensation.timeout_ms ? { compensation_timeout_ms: manifest.compensation.timeout_ms } : {}), + ...(manifest.compensation.verification ? { verification: this.resolveVerification(manifest.compensation.verification, call.args, undefined, "call") } : {}), + } + : {}), + ...(idemKey ? { idem_key: idemKey } : {}), + ...(riskScore !== undefined ? { risk_score: riskScore } : {}), }); this.rv.runs.put(st); + this.captureCall(state, call, manifest?.audit?.redaction, step, observed.length ? "would_block" : "allow", observed, riskScore, manifest?.version); return observed.length ? { decision: "would_block", step, reasons: observed } : { decision: "allow", step }; } + /** + * Resolve a verification's argument references. At call time only `$args.*` are known; when the call succeeds the + * `$result.*` ones are filled in (`result` given). Anything that cannot be resolved is listed in `missing`, and the + * verification is then reported as unverifiable instead of being skipped. + */ + private resolveVerification( + v: Verification, + args: unknown, + result: unknown, + stage: "call" | "result", + previous?: NonNullable, + ): NonNullable { + const resolved: Record = { ...(previous?.args ?? {}) }; + const missing = new Set(previous?.missing ?? []); + for (const [key, ref] of Object.entries(v.arguments)) { + const fromArgs = ref.startsWith("$args."); + if (fromArgs && stage === "call") { + const val = resolveRef(ref, "args", args); + if (val === undefined) missing.add(key); + else resolved[key] = val; + } else if (!fromArgs && stage === "result") { + const val = result === undefined ? undefined : resolveRef(ref, "result", result); + if (val === undefined) missing.add(key); + else resolved[key] = val; + } + } + return { tool: v.tool, refs: v.arguments, args: resolved, expect: v.expect, retries: v.retries, backoff_ms: v.backoff_ms, missing: [...missing] }; + } + + private captureCall( + state: RunState, + call: BeginInput, + redact: string[] | undefined, + step: number | null, + decision: "allow" | "would_block" | "block", + reasons: string[], + risk: number | undefined, + manifest_version: number | undefined, + ): void { + if (!this.rv.capture?.enabled) return; + try { + this.rv.capture.record({ + run_id: this.run_id, + step, + tool: call.tool, + actor: state.actor, + tenant: state.tenant, + environment: this.rv.environment, + args: call.args, + decision, + reasons, + manifest_version, + risk_score: risk, + redact_paths: redact, + }); + } catch { + /* capture is a convenience: it must never stop a call */ + } + } + /** Record the outcome of a call that {@link Run.begin} allowed. */ complete(step: number, outcome: Outcome): { step: number; status: StepStatus } { return this.applyOutcome(step, outcome, ["intended"]); @@ -591,13 +851,26 @@ export class Run { const missing = [...s.handle_missing]; for (const [key, ref] of Object.entries(s.argument_map)) { if (!ref.startsWith("$result.")) continue; - const v = resolveRef(ref, "result", outcome.result); + const v = outcome.invalid_result ? undefined : resolveRef(ref, "result", outcome.result); if (v === undefined) missing.push(key); else handle[key] = v; } + if (s.verification) { + const sp = s.verification; + s.verification = this.resolveVerification( + { tool: sp.tool, arguments: sp.refs, expect: sp.expect, retries: sp.retries, backoff_ms: sp.backoff_ms }, + undefined, + outcome.invalid_result ? undefined : outcome.result, + "result", + sp, + ); + } + s.result_hash = hashPayload(outcome.result); const flagged = this.rv.content?.inspect(outcome.result).filter((f) => f.kind === "injection" || f.kind === "encoded_payload") ?? []; const detail = [ missing.length ? `compensation arguments unavailable: ${missing.join(", ")}` : "", + outcome.invalid_result ? "result does not match the tool's declared output" : "", + ...(outcome.notes ?? []), ...flagged.map((f) => `result content check (${f.kind}): ${f.detail}`), who, ].filter(Boolean); @@ -612,6 +885,13 @@ export class Run { s.error = error || null; } this.rv.runs.put(st); + if (this.rv.capture?.enabled && (s.status === "succeeded" || s.status === "failed" || s.status === "uncertain")) { + try { + this.rv.capture.complete(this.run_id, step, s.status, outcome.status === "succeeded" ? outcome.result : undefined, this.rv.registry.get(s.tool)?.audit?.redaction); + } catch { + /* see above */ + } + } return { step, status: s.status }; } @@ -792,6 +1072,44 @@ export class Run { this.rv.runs.put(st); } + private async runVerification( + step: number, + v: NonNullable, + verifier: ((tool: string, args: Record) => unknown | Promise) | undefined, + actor: string | undefined, + sleep: (ms: number) => Promise, + _base: number, + _cap: number, + ): Promise<"verified" | "failed" | "impossible" | "unverified"> { + const reason = `rollback of run ${this.run_id}: verify the undo of step ${step}`; + const who = actor ?? "operator"; + if (v.missing.length) { + this.emit("verification_failed", step, v.tool, null, reason, `cannot verify: arguments unavailable (${v.missing.join(", ")})`, who); + return "impossible"; + } + if (!verifier) { + this.emit("unverified", step, v.tool, v.args, reason, "no verifier is available to run the declared read-back", who); + return "unverified"; + } + let last = ""; + for (let attempt = 1; attempt <= v.retries; attempt++) { + try { + const got = await verifier(v.tool, v.args); + const mismatch = expectMismatch(got, v.expect); + if (!mismatch) { + this.emit("verified", step, v.tool, v.args, reason, `${v.tool} matched the expected state${attempt > 1 ? ` on attempt ${attempt}` : ""}`, who); + return "verified"; + } + last = mismatch; + } catch (e) { + last = `verifier error: ${messageOf(e)}`; + } + if (attempt < v.retries) await sleep(v.backoff_ms * 2 ** (attempt - 1)); + } + this.emit("verification_failed", step, v.tool, v.args, reason, `after ${v.retries} attempts: ${last}`, who); + return "failed"; + } + /** * Undo the run: newest step first, skipping nothing silently. A step is never undone while a step * depending on it is still standing. Failures do not stop independent steps. Safe to call again: @@ -804,6 +1122,12 @@ export class Run { retry?: { attempts?: number; base_ms?: number; max_ms?: number; sleep?: (ms: number) => Promise }; /** After a compensation reports success, confirm the undo is visible downstream. `false` or a throw counts as failure. */ verify?: (call: CompensationCall) => boolean | Promise; + /** + * Executes a manifest-declared verification: call `tool` with `args` and return what it returned (parsed). The engine + * compares it with the declared `expect`, retrying with backoff. Without a verifier, a step that declares one is + * reported as `unverified` and the rollback is not complete. + */ + verifier?: (tool: string, args: Record) => unknown | Promise; }): Promise { const attempts = Math.max(1, opts.retry?.attempts ?? 1); const base = opts.retry?.base_ms ?? 200; @@ -819,6 +1143,7 @@ export class Run { unmanifested: [], deferred: [], awaiting_approval: [], + unverified: [], complete: false, }; const order = this.load() @@ -841,9 +1166,20 @@ export class Run { if (attempt < attempts) await sleep(Math.min(cap, base * 2 ** (attempt - 1)) * (0.5 + Math.random() / 2)); } } + let unverified = false; + if (error === null) { + const sv = this.load().steps.find((x) => x.step === n)?.verification; + if (sv) { + const outcome = await this.runVerification(n, sv, opts.verifier, opts.actor, sleep, base, cap); + if (outcome === "failed") error = `verification failed: ${sv.tool} did not show the expected state after the undo`; + else if (outcome === "impossible") error = `verification impossible: arguments unavailable (${sv.missing.join(", ")})`; + else if (outcome === "unverified") unverified = true; + } + } if (error === null) { this.completeCompensation(n, { status: "compensated" }, { actor: opts.actor }); report.compensated.push(n); + if (unverified) report.unverified.push(n); } else { const msg = attempts > 1 ? `${error} (after ${attempts} attempts)` : error; this.completeCompensation(n, { status: "failed", error: msg }, { actor: opts.actor }); @@ -873,7 +1209,8 @@ export class Run { report.unresolved.length === 0 && report.unmanifested.length === 0 && report.deferred.length === 0 && - report.awaiting_approval.length === 0; + report.awaiting_approval.length === 0 && + report.unverified.length === 0; return report; } } diff --git a/packages/reversi/src/hub.ts b/packages/reversi/src/hub.ts new file mode 100644 index 0000000..4f5786f --- /dev/null +++ b/packages/reversi/src/hub.ts @@ -0,0 +1,61 @@ +/** Turns ledger records and approval events into notifications. */ +import type { Approvals, ApprovalRecord } from "./approvals.js"; +import type { Reversi } from "./engine.js"; +import type { LedgerRecord } from "./ledger.js"; +import { approvalMessage } from "./notify.js"; +import type { Notifier, NotifyMessage } from "./channels.js"; + +const link = (base: string | undefined, path: string) => (base ? `${base.replace(/\/$/, "")}${path}` : undefined); + +/** What a ledger record means to a human, or null if it is routine. */ +export function describeRecord(r: LedgerRecord, rv: Reversi, consoleUrl?: string): NotifyMessage | null { + const run = link(consoleUrl, `/console/runs/${encodeURIComponent(r.run_id)}`); + const data = { run_id: r.run_id, step: r.step, tool: r.tool, actor: r.actor, seq: r.seq }; + switch (r.event) { + case "compensation_failed": + return { event: "rollback_failed", severity: "critical", title: `Rollback failed: ${r.tool} in ${r.run_id}`, text: `Undoing step ${r.step} failed: ${r.detail || "no detail"}. The run is not fully rolled back.`, dedup_key: `comp:${r.run_id}:${r.step}`, link: run, data }; + case "verification_failed": + return { event: "verification_failed", severity: "critical", title: `Undo not verified: ${r.tool} in ${r.run_id}`, text: r.detail, dedup_key: `verify:${r.run_id}:${r.step}`, link: run, data }; + case "unverified": + return { event: "rollback_unverified", severity: "warning", title: `Undo ran but could not be verified: ${r.run_id}`, text: r.detail, link: run, data }; + case "escalated": + return { event: "approval_escalated", severity: "warning", title: `Approval escalated: ${r.tool}`, text: `${r.reason}. ${r.detail}`, link: link(consoleUrl, "/console/approvals"), data }; + case "succeeded": { + const effect = rv.registry.get(r.tool)?.effect; + if (effect === "irreversible") return { event: "irreversible_executed", severity: "warning", title: `Irreversible action executed: ${r.tool}`, text: `${r.actor} ran an irreversible tool in run ${r.run_id}. It cannot be undone.`, link: run, data }; + return null; + } + case "approved": + if (r.detail.startsWith("BREAK-GLASS")) return { event: "break_glass", severity: "critical", title: `Break-glass approval for ${r.tool}`, text: `${r.actor}: ${r.reason}. ${r.detail}`, link: link(consoleUrl, "/console/approvals"), data }; + return null; + case "blocked": + case "would_block": + if (/no manifest registered/.test(r.detail) && rv.registry.get(r.tool) === undefined) return { event: "unclassified_mutation", severity: "warning", title: `Call to an unclassified tool: ${r.tool}`, text: `${r.actor} called ${r.tool}, which has no manifest (${r.event === "blocked" ? "blocked" : "allowed in a non-enforcing mode"}).`, dedup_key: `unclassified:${r.tool}`, link: run, data }; + return null; + case "purged": + return { event: "retention_purged", severity: "info", title: `Run data purged: ${r.run_id}`, text: r.detail, data }; + default: + return null; + } +} + +export function approvalNotification(rec: ApprovalRecord, kind: "requested" | "decided" | "escalated" | "progress", consoleUrl?: string): NotifyMessage { + const m = approvalMessage(rec, consoleUrl); + const approval = { id: rec.id, run_id: rec.run_id, tool: rec.tool }; + const data = { approval_id: rec.id, run_id: rec.run_id, tool: rec.tool, status: rec.status, quorum: rec.quorum ?? 1, arg_keys: rec.arg_keys }; + if (kind === "requested") return { event: "approval_requested", severity: "warning", title: `Approval needed: ${rec.tool}`, text: m.text, dedup_key: `approval:${rec.id}`, link: link(consoleUrl, "/console/approvals"), data, approval }; + if (kind === "escalated") return { event: "approval_escalated", severity: "warning", title: `Approval escalated: ${rec.tool}`, text: `${m.text} Nobody answered in time, so it now needs ${(rec.required_roles ?? []).join(" or ") || "an approver"}.`, dedup_key: `approval:${rec.id}`, link: link(consoleUrl, "/console/approvals"), data, approval }; + if (kind === "progress") return { event: "approval_progress", severity: "info", title: `Approval progress: ${rec.tool}`, text: `${(rec.decisions ?? []).filter((d) => d.decision === "approve").length} of ${rec.quorum ?? 1} approvals so far.`, data }; + return { event: "approval_decided", severity: "info", title: `Approval ${rec.status}: ${rec.tool}`, text: `${rec.decided_by ?? "someone"} ${rec.status === "approved" ? "approved" : "denied"} the request${rec.decision_note ? `: ${rec.decision_note}` : ""}.`, resolve: true, dedup_key: `approval:${rec.id}`, link: link(consoleUrl, "/console/approvals"), data }; +} + +/** + * Wire an engine and its approvals to a notifier. Ledger-derived events are sent as they are journaled by this process; + * approval events come from the approvals object (so create it with {@link Approvals} options that call these). + */ +export function attachNotifications(o: { reversi: Reversi; notifier: Notifier; console_url?: string }): () => void { + return o.reversi.ledger.onAppend((r) => { + const m = describeRecord(r, o.reversi, o.console_url); + if (m) o.notifier.notifyLater(m); + }); +} diff --git a/packages/reversi/src/index.ts b/packages/reversi/src/index.ts index 99330ef..7bf7de9 100644 --- a/packages/reversi/src/index.ts +++ b/packages/reversi/src/index.ts @@ -1,12 +1,18 @@ export { canonicalize, hashPayload, sha256Hex } from "./canonical.js"; export { + BLAST_RADII, EFFECTS, + parseDuration, ManifestError, Registry, REF_PATTERN, resolveRef, + authorForm, validateManifest, + type BlastRadius, type Compensation, + type ManifestStatus, + type Verification, type Effect, type ToolManifest, type ValidateResult, @@ -44,8 +50,13 @@ export { Reversi, ReversiError, Run, + expectMismatch, + mergeRequirements, idempotencyKey, + renderIdempotencyKey, type ApprovalDecision, + type Halt, + type HaltScope, type ApprovalRequest, type Approver, type BeginInput, @@ -72,6 +83,8 @@ export { mergePolicyDocs, parsePolicy, runPolicyTests, + type ApprovalRequirement, + type Budget, type Condition, type Limit, type Match, @@ -90,6 +103,7 @@ export { FileApprovalStore, MemoryApprovalStore, type ApprovalRecord, + type ApproverIdentity, type ApprovalStatus, type ApprovalStore, type ApprovalsOptions, @@ -104,8 +118,10 @@ export { type EvidenceBundle, type SigningKey, } from "./evidence.js"; +export { FileKvStore, KvConflictError, MemoryKvStore, kvUpdate, type KvEntry, type KvStore } from "./kv.js"; export { dataOptionsFromEnv, openData, storeKindFrom, type Data, type DataOptions, type StoreKind } from "./data.js"; -export { engineConfigFromEnv, loadPolicyFile, openEngine, type EngineConfig, type EngineHandle } from "./config.js"; +export { engineConfigFromEnv, loadPolicyFile, openEngine, type EngineConfig, type EngineHandle, type ReloadResult } from "./config.js"; +export { CONFIG_NAMES, ProjectConfigError, findProjectConfig, loadProjectConfig, starterConfig, type ProjectConfig } from "./projectconfig.js"; export { KeyRing, OidcVerifier, @@ -119,3 +135,48 @@ export { type Role, } from "./auth.js"; export { approvalMessage, webhookNotifier, type WebhookOptions } from "./notify.js"; +export { Delegations, type Delegation } from "./delegations.js"; +export { WindowCounters } from "./counters.js"; +export { DEFAULT_WEIGHTS, parseRisk, scoreRisk, type RiskConfig, type RiskInput, type RiskResult } from "./risk.js"; +export { frozenAt, inWindow, type Day, type Freeze, type Window } from "./schedule.js"; +export { LegalHolds, applyRetention, parseRetention, planRetention, retentionDays, type Hold, type HoldScope, type RetentionConfig, type RetentionItem, type RetentionReport } from "./retention.js"; +export { searchRecords, searchRuns, summarizeRun, type RecordFilter, type RunFilter, type RunStatus, type RunSummary } from "./query.js"; +export { CaptureStore, DEFAULT_CAPTURE, redactForCapture, type CaptureConfig, type CaptureRecord } from "./capture.js"; +export { detectPii, maskPii, type PiiKind } from "./pii.js"; +export { + Sandbox, + previewImpact, + replayRun, + simulatePolicy, + speculate, + synthesizeResult, + type ArgChange, + type Change, + type ImpactPreview, + type PlanInput, + type PlanReport, + type ReplayReport, + type SandboxCall, + type SandboxContext, + type SandboxStep, + type SimulationReport, + type SpeculationReport, + type Verdict, +} from "./simulate.js"; +export { EgressError, checkUrl, egressFromEnv, guardedFetch, isMetadataAddress, isPrivateAddress, type EgressConfig } from "./egress.js"; +export { anchorLedger, readAnchors, verifyAnchor, type AnchorCheck, type AnchorRecord, type AnchorSink } from "./anchor.js"; +export { renderPdf, renderReportPdf } from "./pdf.js"; +export { Notifier, createChannel, parseChannels, parseRoutes, type Channel, type ChannelConfig, type ChannelDeps, type NotifyMessage, type Route, type Severity } from "./channels.js"; +export { sendMail, type MailMessage, type SmtpConfig } from "./smtp.js"; +export { approvalNotification, attachNotifications, describeRecord } from "./hub.js"; +export { AlertManager, evaluateAlerts, type Alert, type AlertManagerOptions, type AlertOptions } from "./alerts.js"; +export { OtelExporter, type OtelOptions } from "./otel.js"; +export { createLogger, redactFields, type Level, type Logger } from "./logger.js"; +export { diffLines, renderDiff, type DiffLine } from "./diff.js"; +export { ManifestRegistry, RegistryError, loadRemoteRegistry, type RegistryOptions, type RegistryVersion } from "./registry.js"; +export { PolicyVersionError, PolicyVersions, type PolicyVersion } from "./policyversions.js"; +export { hasErrors, lintManifests, type KnownTool, type LintFinding } from "./lint.js"; +export { compensateMetaOf, hintsOf, schemaHash, suggestManifests, type CompensateMeta, type Hints, type McpToolInfo, type SuggestOptions, type Suggestion } from "./suggest.js"; +export { banner, classifyTools, type ClassifiedTool, type ToolClass } from "./classify.js"; +export { runManifestTests, type ManifestTestCase, type ManifestTestResult } from "./manifesttest.js"; +export { buildPeriodReport, renderPeriodHtml, renderPeriodMarkdown, type PeriodReport, type TopEntry } from "./weekly.js"; diff --git a/packages/reversi/src/kv.ts b/packages/reversi/src/kv.ts new file mode 100644 index 0000000..8cbfe56 --- /dev/null +++ b/packages/reversi/src/kv.ts @@ -0,0 +1,154 @@ +/** + * A small versioned key-value store for the settings and state that are not the ledger, run state or + * approvals: halts, delegations, legal holds, budgets, capture records, alert state, tenants. + * + * Every backend offers compare-and-set, so several processes can update the same key without losing writes. + */ +import { closeSync, existsSync, fsyncSync, mkdirSync, openSync, readFileSync, renameSync, statSync, unlinkSync, writeSync } from "node:fs"; +import { dirname } from "node:path"; + +export interface KvEntry { + key: string; + value: T; + /** Starts at 1 and increases with every write. */ + version: number; +} + +export interface KvStore { + get(ns: string, key: string): KvEntry | undefined; + /** Unconditional write. */ + put(ns: string, key: string, value: unknown): void; + /** Write only if the stored version equals `expected` (0 means the key must not exist). */ + cas(ns: string, key: string, value: unknown, expected: number): boolean; + delete(ns: string, key: string): boolean; + list(ns: string, prefix?: string): KvEntry[]; +} + +const clone = (v: T): T => (v === undefined ? v : (JSON.parse(JSON.stringify(v)) as T)); + +export class KvConflictError extends Error { + constructor(ns: string, key: string) { + super(`could not update ${ns}/${key}: too many concurrent writers`); + this.name = "KvConflictError"; + } +} + +/** Read-modify-write with retries: `fn` gets the current value (undefined if absent) and returns the new one. */ +export function kvUpdate(kv: KvStore, ns: string, key: string, fn: (current: T | undefined) => T, tries = 25): T { + for (let i = 0; i < tries; i++) { + const cur = kv.get(ns, key); + const next = fn(cur ? clone(cur.value) : undefined); + if (kv.cas(ns, key, next, cur?.version ?? 0)) return next; + } + throw new KvConflictError(ns, key); +} + +export class MemoryKvStore implements KvStore { + private readonly m = new Map(); + private id = (ns: string, key: string) => `${ns}\u0000${key}`; + get(ns: string, key: string) { + const e = this.m.get(this.id(ns, key)); + return e ? ({ key, value: clone(e.value), version: e.version } as KvEntry) : undefined; + } + put(ns: string, key: string, value: unknown) { + const e = this.m.get(this.id(ns, key)); + this.m.set(this.id(ns, key), { key, value: clone(value), version: (e?.version ?? 0) + 1 }); + } + cas(ns: string, key: string, value: unknown, expected: number) { + const e = this.m.get(this.id(ns, key)); + if ((e?.version ?? 0) !== expected) return false; + this.m.set(this.id(ns, key), { key, value: clone(value), version: expected + 1 }); + return true; + } + delete(ns: string, key: string) { + return this.m.delete(this.id(ns, key)); + } + list(ns: string, prefix = "") { + const out: KvEntry[] = []; + for (const [id, e] of this.m) if (id.startsWith(`${ns}\u0000${prefix}`)) out.push({ key: e.key, value: clone(e.value) as T, version: e.version }); + return out.sort((a, b) => a.key.localeCompare(b.key)); + } +} + +/** One JSON file, rewritten atomically under a lock file. Fine for processes on one host. */ +export class FileKvStore implements KvStore { + constructor(private readonly path: string) { + mkdirSync(dirname(path), { recursive: true }); + } + private read(): Record> { + return existsSync(this.path) ? (JSON.parse(readFileSync(this.path, "utf8")) as Record>) : {}; + } + private locked(fn: (all: Record>) => { out: T; write?: boolean }): T { + const lock = `${this.path}.lock`; + const deadline = Date.now() + 10_000; + for (;;) { + try { + closeSync(openSync(lock, "wx")); + break; + } catch (e) { + if ((e as NodeJS.ErrnoException).code !== "EEXIST") throw e; + try { + if (Date.now() - statSync(lock).mtimeMs > 30_000) unlinkSync(lock); + } catch { + /* released meanwhile */ + } + if (Date.now() > deadline) throw new Error("could not lock the settings file"); + Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, 5); + } + } + try { + const all = this.read(); + const r = fn(all); + if (r.write) { + const tmp = `${this.path}.${process.pid}.tmp`; + const fd = openSync(tmp, "w"); + try { + writeSync(fd, JSON.stringify(all)); + fsyncSync(fd); + } finally { + closeSync(fd); + } + renameSync(tmp, this.path); + } + return r.out; + } finally { + try { + unlinkSync(lock); + } catch { + /* gone */ + } + } + } + get(ns: string, key: string) { + const e = this.read()[ns]?.[key]; + return e ? ({ key, value: e.value, version: e.version } as KvEntry) : undefined; + } + put(ns: string, key: string, value: unknown) { + this.locked((all) => { + const bucket = (all[ns] ??= {}); + bucket[key] = { key, value: clone(value), version: (bucket[key]?.version ?? 0) + 1 }; + return { out: undefined, write: true }; + }); + } + cas(ns: string, key: string, value: unknown, expected: number) { + return this.locked((all) => { + const bucket = (all[ns] ??= {}); + if ((bucket[key]?.version ?? 0) !== expected) return { out: false }; + bucket[key] = { key, value: clone(value), version: expected + 1 }; + return { out: true, write: true }; + }); + } + delete(ns: string, key: string) { + return this.locked((all) => { + if (!all[ns]?.[key]) return { out: false }; + delete (all[ns] as Record)[key]; + return { out: true, write: true }; + }); + } + list(ns: string, prefix = "") { + return Object.values(this.read()[ns] ?? {}) + .filter((e) => e.key.startsWith(prefix)) + .map((e) => ({ key: e.key, value: e.value as T, version: e.version })) + .sort((a, b) => a.key.localeCompare(b.key)); + } +} diff --git a/packages/reversi/src/ledger.ts b/packages/reversi/src/ledger.ts index 29f0d85..a079051 100644 --- a/packages/reversi/src/ledger.ts +++ b/packages/reversi/src/ledger.ts @@ -27,7 +27,13 @@ export type LedgerEvent = | "compensated" | "compensation_failed" | "unrecoverable" - | "deferred"; + | "deferred" + | "verified" + | "verification_failed" + | "unverified" + | "escalated" + | "purged" + | "reloaded"; export interface LedgerRecord { seq: number; @@ -240,6 +246,7 @@ export interface AppendInput { } export class Ledger { + private readonly listeners = new Set<(r: LedgerRecord) => void>(); constructor(private readonly store: LedgerStore = new MemoryLedgerStore()) { const v = verifyRecords(store.all()); if (!v.ok) { @@ -270,6 +277,13 @@ export class Ledger { const record: LedgerRecord = { ...body, hash: hashOf(body) }; try { this.store.append(record); + for (const l of this.listeners) { + try { + l(record); + } catch { + /* a listener (notifications, tracing) must never break journaling */ + } + } return record; } catch (e) { if (e instanceof LedgerConflictError && attempt < 8) continue; // re-read the tail and try again @@ -278,6 +292,12 @@ export class Ledger { } } + /** Be told about every record this process appends. Returns an unsubscribe function. */ + onAppend(listener: (r: LedgerRecord) => void): () => void { + this.listeners.add(listener); + return () => this.listeners.delete(listener); + } + records(): readonly LedgerRecord[] { return this.store.all(); } diff --git a/packages/reversi/src/lint.ts b/packages/reversi/src/lint.ts new file mode 100644 index 0000000..ba9e96c --- /dev/null +++ b/packages/reversi/src/lint.ts @@ -0,0 +1,84 @@ +/** Manifest linting: unsafe, incomplete or contradictory definitions that are still syntactically valid. */ +import type { Registry, ToolManifest } from "./manifest.js"; + +export interface LintFinding { + code: string; + level: "error" | "warning" | "info"; + tool: string; + message: string; +} + +/** What an MCP server says about a tool, when known. */ +export interface KnownTool { + name: string; + inputSchema?: { properties?: Record; required?: string[] }; +} + +const MUTATING = (m: ToolManifest) => m.effect !== "read_only"; + +export function lintManifests(reg: Registry, opts: { tools?: KnownTool[] } = {}): LintFinding[] { + const out: LintFinding[] = []; + const all = reg.list(); + const add = (code: string, level: LintFinding["level"], m: ToolManifest, message: string) => out.push({ code, level, tool: m.tool, message }); + const schemas = new Map((opts.tools ?? []).map((t) => [t.name, t])); + const writePerms = new Set(all.filter(MUTATING).map((m) => m.permission)); + + for (const m of all) { + if (MUTATING(m) && !m.description) add("L010", "warning", m, "mutating tool has no description: reviewers cannot tell what it does"); + if (MUTATING(m) && !m.owner) add("L011", "warning", m, "mutating tool has no owner: nobody is named to ask when it misbehaves"); + if (MUTATING(m) && !m.risk?.data_class) add("L012", "warning", m, "no risk.data_class: policy and retention cannot treat sensitive data differently"); + if (m.effect === "compensable" && !(m.compensation?.residue.length)) add("L013", "warning", m, "compensable means the undo leaves residue; declare what it leaves in compensation.residue"); + if (m.effect === "reversible" && m.compensation?.kind === "compensation" && m.compensation.residue.length) add("L030", "warning", m, "declared reversible but the compensation lists residue: it is compensable, not reversible"); + if (m.effect === "compensable" && m.compensation?.kind === "inverse") add("L031", "warning", m, "declared compensable but the compensation is an exact inverse: it may be reversible"); + if (m.compensation) { + const ct = reg.get(m.compensation.tool); + if (ct && ct.effect !== "idempotent") add("L014", "warning", m, `compensation \`${ct.tool}\` is ${ct.effect}, not idempotent: rollback retries it, so running twice must be safe`); + if (!m.compensation.verification) add("L016", "info", m, "the undo is not verified: add compensation.verification to read back that it worked"); + if (Object.keys(m.compensation.argument_map).length === 0 && m.effect !== "irreversible") add("L032", "info", m, "compensation takes no arguments from the call: confirm it can identify what to undo"); + } + if (m.effect === "irreversible") { + if (!m.risk?.blast_radius && !m.approval_rules) add("L015", "warning", m, "irreversible with no blast_radius and no named approvers"); + if (m.audit?.evidence_export !== "required") add("L021", "info", m, "irreversible actions should set audit.evidence_export: required"); + } + if (MUTATING(m) && !m.timeout_ms) add("L022", "info", m, "no timeout_ms: a hung call is only bounded by the gateway default"); + if (m.effect === "read_only" && writePerms.has(m.permission)) add("L019", "warning", m, `read-only tool shares permission \`${m.permission}\` with tools that change things: a read-only agent could not be scoped separately`); + if (m.status === "draft") add("L024", "warning", m, "draft manifest: it is not enforceable until approved"); + if (!m.schema_hash && opts.tools) add("L027", "info", m, "no schema_hash: the gateway cannot notice if the upstream tool's input schema changes"); + + const known = schemas.get(m.tool); + if (known?.inputSchema?.properties) { + const props = Object.keys(known.inputSchema.properties); + const first = (ref: string) => ref.replace(/^\$args\./, "").split(".")[0] as string; + for (const [k, ref] of Object.entries(m.compensation?.argument_map ?? {})) if (ref.startsWith("$args.") && !props.includes(first(ref))) add("L017", "error", m, `compensation.argument_map.${k} reads \`${ref}\` but the tool's input schema has no \`${first(ref)}\``); + for (const ref of Object.values(m.compensation?.verification?.arguments ?? {})) if (ref.startsWith("$args.") && !props.includes(first(ref))) add("L017", "error", m, `verification reads \`${ref}\` but the tool's input schema has no \`${first(ref)}\``); + for (const mm of m.idempotency?.key.matchAll(/\{\{\s*args\.([A-Za-z0-9_-]+)/g) ?? []) if (!props.includes(mm[1] as string)) add("L020", "error", m, `idempotency key uses args.${mm[1]} which the tool's input schema does not have`); + for (const p of m.audit?.redaction ?? []) if (p.startsWith("$args.") && !props.includes(first(p))) add("L033", "warning", m, `redaction path \`${p}\` names an argument the input schema does not have`); + } else if (opts.tools && !known) { + add("L034", "warning", m, "no tool with this name was found on the upstream: the manifest may be stale or misnamed"); + } + } + + // requires cycles + const state = new Map(); + const stack: string[] = []; + const visit = (name: string): void => { + if (state.get(name) === 2) return; + if (state.get(name) === 1) { + const m = reg.get(name); + if (m) add("L023", "error", m, `\`requires\` forms a cycle: ${[...stack.slice(stack.indexOf(name)), name].join(" -> ")}`); + return; + } + state.set(name, 1); + stack.push(name); + for (const r of reg.get(name)?.requires ?? []) visit(r); + stack.pop(); + state.set(name, 2); + }; + for (const m of all) visit(m.tool); + + // upstream tools with no manifest + for (const t of opts.tools ?? []) if (!reg.has(t.name)) out.push({ code: "L035", level: "warning", tool: t.name, message: "the upstream offers this tool but no manifest describes it: it will be blocked in enforce mode" }); + return out; +} + +export const hasErrors = (f: LintFinding[]) => f.some((x) => x.level === "error"); diff --git a/packages/reversi/src/local.ts b/packages/reversi/src/local.ts index 7b47d71..a6d9a1a 100644 --- a/packages/reversi/src/local.ts +++ b/packages/reversi/src/local.ts @@ -7,6 +7,7 @@ import { join, extname } from "node:path"; import { parse as parseYaml } from "yaml"; import { JsonlLedgerStore, Ledger } from "./ledger.js"; import { ManifestError, Registry } from "./manifest.js"; +import { ManifestRegistry } from "./registry.js"; import { Reversi, type Approver } from "./engine.js"; import { FileRunStore } from "./store.js"; @@ -24,6 +25,7 @@ function filesIn(path: string): string[] { /** Load manifests from a file or a directory of `.json`/`.yaml`/`.yml` files. Each file holds one manifest or an array. */ export function loadRegistry(path: string): Registry { if (!existsSync(path)) throw new ManifestError([`manifest path not found: ${path}`]); + if (statSync(path).isDirectory() && ManifestRegistry.isRegistry(path)) return new ManifestRegistry(path).load(); // a versioned registry: its current versions const reg = new Registry(); const errors: string[] = []; for (const file of filesIn(path)) { diff --git a/packages/reversi/src/logger.ts b/packages/reversi/src/logger.ts new file mode 100644 index 0000000..67ba2f4 --- /dev/null +++ b/packages/reversi/src/logger.ts @@ -0,0 +1,37 @@ +/** Structured JSON logging with redaction: credentials never reach a log line, whatever field they arrive in. */ +import { redactSecrets } from "./redact.js"; + +export type Level = "debug" | "info" | "warn" | "error"; +const RANK: Record = { debug: 10, info: 20, warn: 30, error: 40 }; +const SENSITIVE_KEY = /pass(word|wd)?|secret|token|authorization|api[_-]?key|cookie|credential|private[_-]?key|signature/i; + +export interface Logger { + debug(msg: string, fields?: Record): void; + info(msg: string, fields?: Record): void; + warn(msg: string, fields?: Record): void; + error(msg: string, fields?: Record): void; + child(fields: Record): Logger; +} + +export function redactFields(v: unknown, depth = 0): unknown { + if (depth > 6) return "[too deep]"; + if (typeof v === "string") return redactSecrets(v); + if (v instanceof Error) return { name: v.name, message: redactSecrets(v.message) }; + if (Array.isArray(v)) return v.slice(0, 50).map((x) => redactFields(x, depth + 1)); + if (typeof v === "object" && v !== null) return Object.fromEntries(Object.entries(v).map(([k, x]) => [k, SENSITIVE_KEY.test(k) ? "[REDACTED]" : redactFields(x, depth + 1)])); + return v; +} + +export function createLogger(o: { level?: Level; write?: (line: string) => void; base?: Record; now?: () => Date } = {}): Logger { + const min = RANK[o.level ?? "info"]; + const write = o.write ?? ((l: string) => process.stdout.write(l + "\n")); + const now = o.now ?? (() => new Date()); + const make = (base: Record): Logger => { + const log = (level: Level) => (msg: string, fields: Record = {}) => { + if (RANK[level] < min) return; + write(JSON.stringify({ level, time: now().toISOString(), msg: redactSecrets(msg), ...(redactFields({ ...base, ...fields }) as object) })); + }; + return { debug: log("debug"), info: log("info"), warn: log("warn"), error: log("error"), child: (f) => make({ ...base, ...f }) }; + }; + return make(o.base ?? {}); +} diff --git a/packages/reversi/src/manifest.ts b/packages/reversi/src/manifest.ts index 05fb063..49d7c1e 100644 --- a/packages/reversi/src/manifest.ts +++ b/packages/reversi/src/manifest.ts @@ -7,9 +7,32 @@ export const EFFECTS = ["read_only", "idempotent", "reversible", "compensable", "irreversible"] as const; export type Effect = (typeof EFFECTS)[number]; +export type ManifestStatus = "draft" | "approved" | "deprecated"; +export const BLAST_RADII = ["low", "medium", "high", "critical"] as const; +export type BlastRadius = (typeof BLAST_RADII)[number]; + +/** Read back upstream state after a compensation and require an expected condition before calling the undo complete. */ +export interface Verification { + /** A read-only tool that returns the state to check. */ + tool: string; + /** Arguments for it, as references (`$args.x`, `$result.y`) resolved when the original call succeeded. */ + arguments: Record; + /** Dotted path in the verification result -> expected value (deep equal). */ + expect: Record; + /** Attempts, with `backoff_ms` between them (doubling), because undo is often eventually consistent. */ + retries: number; + backoff_ms: number; +} + export interface Compensation { /** Name of the tool that undoes this one. */ tool: string; + /** `inverse` restores the exact prior state. `compensation` (default) makes amends and may leave residue. */ + kind: "inverse" | "compensation"; + /** What the undo cannot take back, in plain words (an emailed notification, a line on a statement). */ + residue: string[]; + verification?: Verification; + timeout_ms?: number; /** * How to build the compensation's arguments. Keys are the compensation tool's argument names; * values are references: `$args.path.to.value` (the original call) or `$result.path.to.value`. @@ -29,6 +52,24 @@ export interface ToolManifest { compensation?: Compensation; /** Tools that must have succeeded earlier in the same run. Also creates rollback ordering edges. */ requires: string[]; + /** Manifest version (a registry increments it on every publish). */ + version: number; + /** `draft` manifests are not enforceable: calls are blocked. `deprecated` ones still run and still roll back. */ + status: ManifestStatus; + replaced_by?: string; + owner?: string; + oncall?: string; + risk?: { data_class?: string; blast_radius?: BlastRadius; environment?: string }; + timeout_ms?: number; + /** Retry transient failures of the forward call (only honoured for read_only and idempotent tools). */ + retry?: { attempts: number; base_ms: number; max_ms: number }; + /** Calls with the same rendered key are duplicates. `{{args.order_id}}` templates over the call arguments. */ + idempotency?: { key: string; scope: "run" | "actor" | "global" }; + audit?: { retention_days?: number; evidence_export?: "required" | "optional"; redaction?: string[] }; + /** Who may approve, how many must, and what happens when nobody answers in time. */ + approval_rules?: { roles: string[]; quorum: number; timeout_ms?: number; on_timeout: "deny" | "escalate"; escalate_to?: string[] }; + /** Hash of the upstream tool's inputSchema when this manifest was written; lets the gateway notice drift. */ + schema_hash?: string; } export class ManifestError extends Error { @@ -50,8 +91,30 @@ const TOP_KEYS = new Set([ "approval", "compensation", "requires", + "version", + "status", + "replaced_by", + "owner", + "oncall", + "risk", + "timeout_ms", + "retry", + "idempotency", + "audit", + "schema_hash", ]); -const COMP_KEYS = new Set(["tool", "argument_map", "requires_approval"]); +const COMP_KEYS = new Set(["tool", "argument_map", "requires_approval", "kind", "residue", "verification", "timeout_ms"]); +const VERIFY_KEYS = new Set(["tool", "arguments", "expect", "retries", "backoff_ms"]); + +/** "15m", "30s", "2h", "500ms", "1d" or a number of milliseconds. */ +export function parseDuration(v: unknown): number | undefined { + if (typeof v === "number") return Number.isFinite(v) && v > 0 ? Math.floor(v) : undefined; + if (typeof v !== "string") return undefined; + const m = /^(\d+(?:\.\d+)?)\s*(ms|s|m|h|d)$/.exec(v.trim()); + if (!m) return undefined; + const mult = { ms: 1, s: 1000, m: 60_000, h: 3_600_000, d: 86_400_000 }[m[2] as "ms"]; + return Math.floor(Number(m[1]) * mult); +} function isRecord(x: unknown): x is Record { return typeof x === "object" && x !== null && !Array.isArray(x); @@ -60,9 +123,11 @@ function isRecord(x: unknown): x is Record { export type ValidateResult = { ok: true; manifest: ToolManifest } | { ok: false; errors: string[] }; /** Validate one manifest and fill in defaults. Does not check references to other tools. */ -export function validateManifest(raw: unknown): ValidateResult { +export function validateManifest(input: unknown): ValidateResult { const errors: string[] = []; - if (!isRecord(raw)) return { ok: false, errors: ["manifest must be an object"] }; + if (!isRecord(input)) return { ok: false, errors: ["manifest must be an object"] }; + // A normalized manifest (as stored and returned by this library) carries `approval_rules`; read it back as the author's `approval` object. + const raw = isRecord(input.approval_rules) && typeof input.approval === "string" ? authorForm(input as unknown as ToolManifest) : input; const label = typeof raw.tool === "string" && raw.tool ? `\`${raw.tool}\`` : "manifest"; const err = (m: string) => errors.push(`${label}: ${m}`); @@ -82,9 +147,33 @@ export function validateManifest(raw: unknown): ValidateResult { if (!EFFECTS.includes(effect)) err(`\`effect\` must be one of ${EFFECTS.join(", ")}`); let approval: "never" | "always" = "never"; + let approval_rules: ToolManifest["approval_rules"]; if (raw.approval !== undefined) { if (raw.approval === "never" || raw.approval === "always") approval = raw.approval; - else err("`approval` must be `never` or `always`"); + else if (isRecord(raw.approval)) { + const a = raw.approval; + for (const k of Object.keys(a)) if (!["required", "roles", "quorum", "timeout", "on_timeout", "escalate_to"].includes(k)) err(`unknown approval field \`${k}\``); + if (a.required !== undefined && typeof a.required !== "boolean") err("`approval.required` must be a boolean"); + approval = a.required === false ? "never" : "always"; + const roles = a.roles === undefined ? [] : a.roles; + if (!Array.isArray(roles) || roles.some((r) => typeof r !== "string" || !r)) err("`approval.roles` must be a list of role or group names"); + const quorum = a.quorum === undefined ? 1 : a.quorum; + if (!Number.isInteger(quorum) || (quorum as number) < 1 || (quorum as number) > 10) err("`approval.quorum` must be an integer from 1 to 10"); + let timeout_ms: number | undefined; + if (a.timeout !== undefined) { + timeout_ms = parseDuration(a.timeout); + if (timeout_ms === undefined) err("`approval.timeout` must be a duration like 15m, 30s or 2h"); + } + if (a.on_timeout !== undefined && a.on_timeout !== "deny" && a.on_timeout !== "escalate") err("`approval.on_timeout` must be deny or escalate"); + if (Array.isArray(roles) && Array.isArray(roles) && (quorum as number) > 1 && roles.length === 0) err("a quorum needs `approval.roles` to draw approvers from"); + approval_rules = { + roles: Array.isArray(roles) ? (roles as string[]) : [], + quorum: Number.isInteger(quorum) ? (quorum as number) : 1, + timeout_ms, + on_timeout: a.on_timeout === "escalate" ? "escalate" : "deny", + escalate_to: Array.isArray(a.escalate_to) ? (a.escalate_to as string[]) : undefined, + }; + } else err("`approval` must be `never`, `always` or an object with roles, quorum and timeout"); } if (effect === "irreversible") { if (raw.approval === "never") err("irreversible tools must require approval"); @@ -93,10 +182,13 @@ export function validateManifest(raw: unknown): ValidateResult { let requires: string[] = []; if (raw.requires !== undefined) { - if (!Array.isArray(raw.requires) || raw.requires.some((r) => typeof r !== "string" || !r)) { - err("`requires` must be a list of tool names"); + const normalized = Array.isArray(raw.requires) + ? raw.requires.map((r) => (typeof r === "string" ? r : isRecord(r) && typeof r.tool === "string" && (r.status === undefined || r.status === "succeeded") ? r.tool : "")) + : []; + if (!Array.isArray(raw.requires) || normalized.some((r) => !r)) { + err("`requires` must be a list of tool names (or { tool, status: succeeded })"); } else { - requires = raw.requires as string[]; + requires = normalized; if (new Set(requires).size !== requires.length) err("`requires` contains duplicates"); if (requires.includes(raw.tool as string)) err("a tool cannot require itself"); } @@ -131,11 +223,53 @@ export function validateManifest(raw: unknown): ValidateResult { if (c.requires_approval !== undefined && typeof c.requires_approval !== "boolean") { err("`compensation.requires_approval` must be a boolean"); } + if (c.kind !== undefined && c.kind !== "inverse" && c.kind !== "compensation") err("`compensation.kind` must be inverse or compensation"); + let residue: string[] = []; + if (c.residue !== undefined) { + if (!Array.isArray(c.residue) || c.residue.some((r) => typeof r !== "string" || !r)) err("`compensation.residue` must be a list of strings"); + else residue = c.residue as string[]; + } + if (c.kind === "inverse" && residue.length) err("an `inverse` compensation restores exact state, so it cannot declare residue; use kind: compensation"); + let comp_timeout: number | undefined; + if (c.timeout_ms !== undefined) { + comp_timeout = parseDuration(c.timeout_ms); + if (comp_timeout === undefined) err("`compensation.timeout_ms` must be a positive number or a duration"); + } + let verification: Verification | undefined; + if (c.verification !== undefined) { + const v = c.verification; + if (!isRecord(v)) err("`compensation.verification` must be an object"); + else { + for (const k of Object.keys(v)) if (!VERIFY_KEYS.has(k)) err(`unknown verification field \`${k}\``); + if (typeof v.tool !== "string" || !TOOL_NAME.test(v.tool)) err("`verification.tool` must be a tool name"); + const vargs: Record = {}; + if (v.arguments !== undefined) { + if (!isRecord(v.arguments)) err("`verification.arguments` must be an object"); + else + for (const [k, ref] of Object.entries(v.arguments)) { + if (typeof ref !== "string" || !REF_PATTERN.test(ref)) err(`verification.arguments.${k} must be a reference like \`$result.id\``); + else vargs[k] = ref; + } + } + if (!isRecord(v.expect) || Object.keys(v.expect).length === 0) err("`verification.expect` must name at least one path and its expected value"); + const retries = v.retries === undefined ? 3 : v.retries; + if (!Number.isInteger(retries) || (retries as number) < 1 || (retries as number) > 10) err("`verification.retries` must be 1..10"); + const backoff = v.backoff_ms === undefined ? 1000 : parseDuration(v.backoff_ms); + if (backoff === undefined) err("`verification.backoff_ms` must be a positive number or a duration like 2s"); + if (typeof v.tool === "string" && isRecord(v.expect) && Number.isInteger(retries) && backoff !== undefined) { + verification = { tool: v.tool, arguments: vargs, expect: v.expect, retries: retries as number, backoff_ms: backoff }; + } + } + } if (typeof c.tool === "string") { compensation = { tool: c.tool, argument_map, requires_approval: c.requires_approval === true, + kind: c.kind === "inverse" ? "inverse" : "compensation", + residue, + ...(verification ? { verification } : {}), + ...(comp_timeout !== undefined ? { timeout_ms: comp_timeout } : {}), }; } } @@ -147,6 +281,71 @@ export function validateManifest(raw: unknown): ValidateResult { err(`${effect} tools must not declare a compensation`); } + // ---- metadata, risk, timeouts, retry, idempotency, audit ---- + const version = raw.version === undefined ? 1 : raw.version; + if (!Number.isInteger(version) || (version as number) < 1) err("`version` must be a positive integer"); + const status = raw.status === undefined ? "approved" : raw.status; + if (status !== "draft" && status !== "approved" && status !== "deprecated") err("`status` must be draft, approved or deprecated"); + for (const k of ["replaced_by", "owner", "oncall", "schema_hash"] as const) { + if (raw[k] !== undefined && (typeof raw[k] !== "string" || !raw[k])) err(`\`${k}\` must be a non-empty string`); + } + let risk: ToolManifest["risk"]; + if (raw.risk !== undefined) { + const r = raw.risk; + if (!isRecord(r)) err("`risk` must be an object"); + else { + for (const k of Object.keys(r)) if (!["data_class", "blast_radius", "environment"].includes(k)) err(`unknown risk field \`${k}\``); + if (r.data_class !== undefined && (typeof r.data_class !== "string" || !r.data_class)) err("`risk.data_class` must be a non-empty string"); + if (r.blast_radius !== undefined && !BLAST_RADII.includes(r.blast_radius as BlastRadius)) err(`\`risk.blast_radius\` must be one of ${BLAST_RADII.join(", ")}`); + if (r.environment !== undefined && typeof r.environment !== "string") err("`risk.environment` must be a string"); + risk = r as ToolManifest["risk"]; + } + } + let timeout_ms: number | undefined; + if (raw.timeout_ms !== undefined) { + timeout_ms = parseDuration(raw.timeout_ms); + if (timeout_ms === undefined) err("`timeout_ms` must be a positive number or a duration like 30s"); + } + let retry: ToolManifest["retry"]; + if (raw.retry !== undefined) { + const r = raw.retry; + if (!isRecord(r)) err("`retry` must be an object"); + else { + for (const k of Object.keys(r)) if (!["attempts", "base_ms", "max_ms"].includes(k)) err(`unknown retry field \`${k}\``); + const attempts = r.attempts === undefined ? 3 : r.attempts; + const base = r.base_ms === undefined ? 200 : parseDuration(r.base_ms); + const max = r.max_ms === undefined ? 5000 : parseDuration(r.max_ms); + if (!Number.isInteger(attempts) || (attempts as number) < 1 || (attempts as number) > 10) err("`retry.attempts` must be 1..10"); + if (base === undefined || max === undefined) err("`retry.base_ms` and `retry.max_ms` must be positive durations"); + else retry = { attempts: attempts as number, base_ms: base, max_ms: max }; + } + if (effect !== "read_only" && effect !== "idempotent") err("forward retry is only allowed for read_only and idempotent tools: retrying anything else could repeat its effect"); + } + let idempotency: ToolManifest["idempotency"]; + if (raw.idempotency !== undefined) { + const i = raw.idempotency; + if (!isRecord(i)) err("`idempotency` must be an object"); + else { + for (const k of Object.keys(i)) if (!["key", "scope"].includes(k)) err(`unknown idempotency field \`${k}\``); + if (typeof i.key !== "string" || !/\{\{\s*args(\.[A-Za-z0-9_-]+)+\s*\}\}/.test(i.key)) err("`idempotency.key` must be a template using at least one `{{args.field}}`"); + if (i.scope !== undefined && !["run", "actor", "global"].includes(i.scope as string)) err("`idempotency.scope` must be run, actor or global"); + if (typeof i.key === "string") idempotency = { key: i.key, scope: (i.scope as "run" | "actor" | "global") ?? "run" }; + } + } + let audit: ToolManifest["audit"]; + if (raw.audit !== undefined) { + const a = raw.audit; + if (!isRecord(a)) err("`audit` must be an object"); + else { + for (const k of Object.keys(a)) if (!["retention_days", "evidence_export", "redaction"].includes(k)) err(`unknown audit field \`${k}\``); + if (a.retention_days !== undefined && (!Number.isInteger(a.retention_days) || (a.retention_days as number) < 1)) err("`audit.retention_days` must be a positive integer"); + if (a.evidence_export !== undefined && a.evidence_export !== "required" && a.evidence_export !== "optional") err("`audit.evidence_export` must be required or optional"); + if (a.redaction !== undefined && (!Array.isArray(a.redaction) || a.redaction.some((p) => typeof p !== "string" || !/^\$(args|result)(\.[A-Za-z0-9_-]+)+$/.test(p)))) err("`audit.redaction` must be a list of paths like `$args.client_secret`"); + audit = a as ToolManifest["audit"]; + } + } + if (status === "deprecated" && raw.replaced_by === raw.tool) err("a manifest cannot be replaced by itself"); + if (errors.length) return { ok: false, errors }; const manifest: ToolManifest = { tool: raw.tool as string, @@ -154,7 +353,16 @@ export function validateManifest(raw: unknown): ValidateResult { permission: raw.permission as string, approval, requires, + version: version as number, + status: status as ManifestStatus, }; + if (approval_rules) manifest.approval_rules = approval_rules; + for (const k of ["replaced_by", "owner", "oncall", "schema_hash"] as const) if (typeof raw[k] === "string") manifest[k] = raw[k] as string; + if (risk) manifest.risk = risk; + if (timeout_ms !== undefined) manifest.timeout_ms = timeout_ms; + if (retry) manifest.retry = retry; + if (idempotency) manifest.idempotency = idempotency; + if (audit) manifest.audit = audit; if (typeof raw.description === "string") manifest.description = raw.description; if (compensation) manifest.compensation = compensation; return { ok: true, manifest }; @@ -191,6 +399,12 @@ export class Registry { for (const r of m.requires) { if (!this.tools.has(r)) errors.push(`\`${m.tool}\`: required tool \`${r}\` is not registered`); } + const v = m.compensation?.verification; + if (v && !this.tools.has(v.tool)) errors.push(`\`${m.tool}\`: verification tool \`${v.tool}\` is not registered`); + if (v && this.tools.get(v.tool) && this.tools.get(v.tool)?.effect !== "read_only") { + errors.push(`\`${m.tool}\`: verification tool \`${v.tool}\` must be read_only: a check must not change anything`); + } + if (m.replaced_by && !this.tools.has(m.replaced_by)) errors.push(`\`${m.tool}\`: replaced_by \`${m.replaced_by}\` is not registered`); } return errors; } @@ -235,3 +449,23 @@ export function resolveRef(ref: string, source: "args" | "result", obj: unknown) } return cur; } + +/** + * A normalized manifest written back in the form an author writes (the `approval` object instead of the internal + * `approval_rules`), so it passes the validator again. Used wherever a stored manifest is republished or edited. + */ +export function authorForm(m: ToolManifest): Record { + const { approval_rules: r, ...rest } = m as ToolManifest & Record; + if (!r) return rest; + return { + ...rest, + approval: { + required: m.approval === "always", + roles: r.roles, + quorum: r.quorum, + ...(r.timeout_ms !== undefined ? { timeout: `${r.timeout_ms}ms` } : {}), + on_timeout: r.on_timeout, + ...(r.escalate_to ? { escalate_to: r.escalate_to } : {}), + }, + }; +} diff --git a/packages/reversi/src/manifesttest.ts b/packages/reversi/src/manifesttest.ts new file mode 100644 index 0000000..5a6d169 --- /dev/null +++ b/packages/reversi/src/manifesttest.ts @@ -0,0 +1,99 @@ +/** + * Contract tests for manifests, run without touching any system: given a fake call and a fake result, does the manifest + * produce the compensation arguments you expect, the verification you expect, the right plan, and the right behaviour + * on failure, duplicates and missing prerequisites? + */ +import { expectMismatch } from "./engine.js"; +import type { Registry } from "./manifest.js"; +import { Sandbox, type SandboxContext } from "./simulate.js"; + +export interface ManifestTestCase { + name?: string; + tool: string; + args?: unknown; + /** The fake result of a successful call. */ + result?: unknown; + /** Make the call fail with this message instead. */ + fail?: string; + /** Run these tools first (with synthesized results) so `requires` is satisfied. Default: the manifest's `requires`. */ + after?: string[]; + /** Compensation arguments the plan must contain. */ + expect_handle?: Record; + /** The plan action kind expected for this step: compensate | skip | unrecoverable. */ + expect_plan?: string; + /** Substring a block reason must contain (the call is expected to be refused). */ + expect_blocked?: string; + /** Verification arguments the manifest must resolve to. */ + expect_verification?: Record; + /** What the read-back would return, and whether the manifest's `expect` should accept it. */ + verify_with?: unknown; + expect_verified?: boolean; + /** Run the same call a second time and expect it to be refused as a duplicate. */ + expect_duplicate?: boolean; +} + +export interface ManifestTestResult { + name: string; + tool: string; + ok: boolean; + problems: string[]; +} + +export async function runManifestTests(reg: Registry, cases: ManifestTestCase[], ctx: Omit = {}): Promise<{ results: ManifestTestResult[]; untested: string[] }> { + const results: ManifestTestResult[] = []; + for (const [i, c] of cases.entries()) { + const problems: string[] = []; + const name = c.name ?? `${c.tool} #${i + 1}`; + const m = reg.get(c.tool); + if (!m) { + results.push({ name, tool: c.tool, ok: false, problems: [`no manifest for \`${c.tool}\``] }); + continue; + } + const sb = new Sandbox({ ...ctx, registry: reg }, { assume_approved: true }); + for (const pre of c.after ?? m.requires) await sb.seed({ run: "t", tool: pre }); + const judge = new Sandbox({ ...ctx, registry: reg }); + for (const pre of c.after ?? m.requires) await judge.seed({ run: "t", tool: pre }); + const verdict = await judge.call({ run: "t", tool: c.tool, args: c.args }); + if (c.expect_blocked !== undefined) { + if (verdict.verdict === "allow") problems.push(`expected the call to be refused (${c.expect_blocked}) but it was allowed`); + else if (!verdict.reasons.some((r) => r.includes(c.expect_blocked as string))) problems.push(`refused, but not for "${c.expect_blocked}": ${verdict.reasons.join("; ")}`); + results.push({ name, tool: c.tool, ok: problems.length === 0, problems }); + continue; + } + if (verdict.verdict === "block") problems.push(`the call was refused: ${verdict.reasons.join("; ")}`); + const step = await sb.call({ run: "t", tool: c.tool, args: c.args, result: c.result, outcome: c.fail ? "failed" : "succeeded" }); + if (step.verdict === "block") problems.push(`the call was refused: ${step.reasons.join("; ")}`); + const plan = sb.plan("t"); + const action = plan?.actions.find((a) => a.tool === c.tool); + if (c.fail) { + if (action) problems.push(`a failed call should leave nothing to undo, but the plan has ${action.kind}`); + } else { + if (c.expect_plan && (action?.kind ?? "skip") !== c.expect_plan) problems.push(`expected plan kind ${c.expect_plan}, got ${action?.kind ?? "none"}${action?.note ? ` (${action.note})` : ""}`); + if (c.expect_handle) { + const got = action?.args ?? {}; + for (const [k, v] of Object.entries(c.expect_handle)) if (JSON.stringify(got[k]) !== JSON.stringify(v)) problems.push(`compensation argument \`${k}\` is ${JSON.stringify(got[k])}, expected ${JSON.stringify(v)}`); + } + const ver = sb.reversi.openRun([...sb.reversi.runIds()][0] as string).state().steps.find((s) => s.tool === c.tool)?.verification; + if (c.expect_verification) { + if (!ver) problems.push("expected a verification but the manifest declares none"); + else for (const [k, v] of Object.entries(c.expect_verification)) if (JSON.stringify(ver.args[k]) !== JSON.stringify(v)) problems.push(`verification argument \`${k}\` is ${JSON.stringify(ver.args[k])}, expected ${JSON.stringify(v)}`); + } + if (c.verify_with !== undefined) { + if (!ver) problems.push("verify_with given but the manifest declares no verification"); + else { + const mismatch = expectMismatch(c.verify_with, ver.expect); + const accepted = mismatch === null; + if (c.expect_verified !== undefined && accepted !== c.expect_verified) problems.push(accepted ? "the read-back was accepted but should have been rejected" : `the read-back was rejected (${mismatch}) but should have been accepted`); + } + } + } + if (c.expect_duplicate) { + const again = await sb.call({ run: "t", tool: c.tool, args: c.args, result: c.result }); + if (again.verdict === "allow") problems.push("a second identical call was allowed; expected it to be refused as a duplicate"); + else if (!again.reasons.some((r) => r.includes("duplicate"))) problems.push(`the second call was refused, but not as a duplicate: ${again.reasons.join("; ")}`); + } + results.push({ name, tool: c.tool, ok: problems.length === 0, problems }); + } + const tested = new Set(cases.map((c) => c.tool)); + return { results, untested: reg.list().filter((m) => m.effect !== "read_only" && !tested.has(m.tool)).map((m) => m.tool) }; +} diff --git a/packages/reversi/src/otel.ts b/packages/reversi/src/otel.ts new file mode 100644 index 0000000..2e48893 --- /dev/null +++ b/packages/reversi/src/otel.ts @@ -0,0 +1,150 @@ +/** + * OpenTelemetry trace export (OTLP over HTTP/JSON, no SDK). Each tool call becomes a span from its journaled intent to + * its outcome, each compensation a span from `compensating` to its result, and blocked calls become short error spans. + * Spans carry names and ids, never arguments or results. Point `endpoint` at a collector (`http://collector:4318`). + */ +import { createHash } from "node:crypto"; +import type { Reversi } from "./engine.js"; +import { guardedFetch, type EgressConfig } from "./egress.js"; +import type { LedgerRecord } from "./ledger.js"; + +type AttrValue = { stringValue: string } | { intValue: string } | { boolValue: boolean }; +interface Span { + traceId: string; + spanId: string; + parentSpanId?: string; + name: string; + kind: number; + startTimeUnixNano: string; + endTimeUnixNano: string; + attributes: Array<{ key: string; value: AttrValue }>; + status: { code: number; message?: string }; +} + +const hex = (s: string, bytes: number) => createHash("sha256").update(s).digest("hex").slice(0, bytes * 2); +const nano = (ms: number) => String(BigInt(ms) * 1_000_000n); +const attr = (key: string, v: string | number | boolean | null | undefined): { key: string; value: AttrValue } | null => + v === null || v === undefined ? null : { key, value: typeof v === "boolean" ? { boolValue: v } : typeof v === "number" ? { intValue: String(Math.trunc(v)) } : { stringValue: String(v).slice(0, 500) } }; + +export interface OtelOptions { + /** Base URL of an OTLP/HTTP collector; spans go to `/v1/traces`. */ + endpoint: string; + headers?: Record; + service_name?: string; + service_version?: string; + environment?: string; + egress?: EgressConfig; + fetch?: typeof fetch; + /** Send when this many spans are waiting (default 50) or every `interval_ms` (default 5000). */ + batch?: number; + interval_ms?: number; + onError?: (e: Error) => void; +} + +export class OtelExporter { + private readonly pending: Span[] = []; + private readonly open = new Map(); + private timer: NodeJS.Timeout | undefined; + private unsubscribe: (() => void) | undefined; + private readonly f: typeof fetch; + constructor( + private readonly o: OtelOptions, + private readonly rv?: Reversi, + ) { + this.f = guardedFetch(o.egress, o.fetch); + } + + /** Start listening to the engine's ledger and flush on a timer. */ + start(): void { + if (this.rv) this.unsubscribe = this.rv.ledger.onAppend((r) => this.record(r)); + this.timer = setInterval(() => void this.flush(), this.o.interval_ms ?? 5000); + this.timer.unref(); + } + + async stop(): Promise { + this.unsubscribe?.(); + if (this.timer) clearInterval(this.timer); + await this.flush(); + } + + private span(start: LedgerRecord, end: LedgerRecord, name: string, ok: boolean): Span { + const effect = this.rv?.registry.get(start.tool)?.effect; + return { + traceId: hex(start.run_id, 16), + spanId: hex(`${start.run_id}/${start.step}/${name}/${start.seq}`, 8), + name: `${name} ${start.tool}`, + kind: 1, + startTimeUnixNano: nano(start.ts_ms), + endTimeUnixNano: nano(Math.max(end.ts_ms, start.ts_ms)), + attributes: [ + attr("reversi.run_id", start.run_id), + attr("reversi.step", start.step), + attr("reversi.tool", start.tool), + attr("reversi.actor", start.actor), + attr("reversi.effect", effect), + attr("reversi.event", end.event), + attr("reversi.reason", start.reason), + attr("reversi.detail", end.detail), + attr("reversi.seq.start", start.seq), + attr("reversi.seq.end", end.seq), + ].filter((a): a is { key: string; value: AttrValue } => a !== null), + status: ok ? { code: 1 } : { code: 2, message: end.detail.slice(0, 200) || end.event }, + }; + } + + /** Feed a ledger record. Exposed for tests and for replaying history. */ + record(r: LedgerRecord): void { + const key = `${r.run_id}/${r.step}`; + switch (r.event) { + case "intended": + case "compensating": + this.open.set(`${key}/${r.event}`, r); + return; + case "succeeded": + case "failed": + case "uncertain": { + const s = this.open.get(`${key}/intended`); + if (s) this.push(this.span(s, r, "reversi.call", r.event === "succeeded")); + this.open.delete(`${key}/intended`); + return; + } + case "compensated": + case "compensation_failed": { + const s = this.open.get(`${key}/compensating`); + if (s) this.push(this.span(s, r, "reversi.compensate", r.event === "compensated")); + this.open.delete(`${key}/compensating`); + return; + } + case "blocked": + case "would_block": + this.push(this.span(r, r, "reversi.call.blocked", r.event === "would_block")); + return; + default: + return; + } + } + + private push(s: Span) { + this.pending.push(s); + if (this.pending.length >= (this.o.batch ?? 50)) void this.flush(); + } + + async flush(): Promise { + if (this.pending.length === 0) return; + const spans = this.pending.splice(0); + const body = { + resourceSpans: [ + { + resource: { attributes: [attr("service.name", this.o.service_name ?? "reversi"), attr("service.version", this.o.service_version), attr("deployment.environment", this.o.environment ?? this.rv?.environment)].filter((a): a is { key: string; value: AttrValue } => a !== null) }, + scopeSpans: [{ scope: { name: "reversi" }, spans }], + }, + ], + }; + try { + const res = await this.f(`${this.o.endpoint.replace(/\/$/, "")}/v1/traces`, { method: "POST", headers: { "content-type": "application/json", ...(this.o.headers ?? {}) }, body: JSON.stringify(body), signal: AbortSignal.timeout(10_000) }); + if (!res.ok) throw new Error(`collector answered ${res.status}`); + } catch (e) { + (this.o.onError ?? ((err) => console.error(`trace export failed: ${err.message}`)))(e instanceof Error ? e : new Error(String(e))); + } + } +} diff --git a/packages/reversi/src/pdf.ts b/packages/reversi/src/pdf.ts new file mode 100644 index 0000000..58a98e5 --- /dev/null +++ b/packages/reversi/src/pdf.ts @@ -0,0 +1,111 @@ +/** + * A small, dependency-free PDF writer for the auditor report: monospaced text on A4 pages. It writes a valid PDF 1.4 + * file (cross-reference table included) that any viewer opens, and that text extractors can read. + */ +import type { LedgerRecord } from "./ledger.js"; +import type { RunState } from "./store.js"; + +const PAGE_W = 595; +const PAGE_H = 842; +const MARGIN = 40; +const SIZE = 9; +const LEAD = 12; +const CHAR_W = 0.6 * SIZE; // Courier is monospaced at 600/1000 em +const COLS = Math.floor((PAGE_W - 2 * MARGIN) / CHAR_W); +const ROWS = Math.floor((PAGE_H - 2 * MARGIN) / LEAD); + +/** Latin-1 safe text for a PDF string literal. */ +function pdfText(s: string): string { + return s + .replace(/[^\x20-\x7e\xa0-\xff]/g, "?") + .replace(/\\/g, "\\\\") + .replace(/\(/g, "\\(") + .replace(/\)/g, "\\)"); +} + +function wrap(line: string): string[] { + if (line.length <= COLS) return [line]; + const out: string[] = []; + let rest = line; + while (rest.length > COLS) { + let cut = rest.lastIndexOf(" ", COLS); + if (cut < COLS / 2) cut = COLS; + out.push(rest.slice(0, cut)); + rest = " " + rest.slice(cut).trimStart(); + } + out.push(rest); + return out; +} + +/** Lay lines out on pages. A line starting with `#` is a heading (bold). */ +export function renderPdf(lines: string[], title: string): Buffer { + const rows = lines.flatMap((l) => (l.startsWith("# ") ? [l] : wrap(l))); + const pages: string[][] = []; + for (let i = 0; i < rows.length; i += ROWS) pages.push(rows.slice(i, i + ROWS)); + if (pages.length === 0) pages.push([""]); + + const objs: string[] = []; + const add = (s: string) => objs.push(s); + // 1 catalog, 2 pages, 3 font, 4 bold font, then per page: content, page + add("<< /Type /Catalog /Pages 2 0 R >>"); + const kids = pages.map((_, i) => `${7 + i * 2} 0 R`).join(" "); + add(`<< /Type /Pages /Kids [${kids}] /Count ${pages.length} >>`); + add("<< /Type /Font /Subtype /Type1 /BaseFont /Courier /Encoding /WinAnsiEncoding >>"); + add("<< /Type /Font /Subtype /Type1 /BaseFont /Courier-Bold /Encoding /WinAnsiEncoding >>"); + add(`<< /Title (${pdfText(title)}) /Producer (Reversi) >>`); // 5 info + pages.forEach((rowsOnPage, i) => { + let y = PAGE_H - MARGIN - SIZE; + const ops: string[] = []; + for (const r of rowsOnPage) { + const heading = r.startsWith("# "); + ops.push(`BT /${heading ? "F2" : "F1"} ${heading ? SIZE + 3 : SIZE} Tf ${MARGIN} ${y} Td (${pdfText(heading ? r.slice(2) : r)}) Tj ET`); + y -= LEAD; + } + const stream = ops.join("\n"); + add(`<< /Length ${Buffer.byteLength(stream, "latin1")} >>\nstream\n${stream}\nendstream`); + add(`<< /Type /Page /Parent 2 0 R /MediaBox [0 0 ${PAGE_W} ${PAGE_H}] /Contents ${5 + i * 2 + 1} 0 R /Resources << /Font << /F1 3 0 R /F2 4 0 R >> >> >>`); + }); + + let out = "%PDF-1.4\n"; + const offsets: number[] = []; + objs.forEach((body, i) => { + offsets.push(Buffer.byteLength(out, "latin1")); + out += `${i + 1} 0 obj\n${body}\nendobj\n`; + }); + const xref = Buffer.byteLength(out, "latin1"); + out += `xref\n0 ${objs.length + 1}\n0000000000 65535 f \n`; + for (const o of offsets) out += `${String(o).padStart(10, "0")} 00000 n \n`; + out += `trailer\n<< /Size ${objs.length + 1} /Root 1 0 R /Info 5 0 R >>\nstartxref\n${xref}\n%%EOF\n`; + return Buffer.from(out, "latin1"); +} + +const t = (ms: number) => new Date(ms).toISOString().replace("T", " ").slice(0, 19) + "Z"; + +/** The auditor report as a PDF: the same facts as the HTML report. */ +export function renderReportPdf(o: { + run: RunState; + records: readonly LedgerRecord[]; + plan?: { fully_reversible: boolean; actions: Array<{ step: number; tool: string; kind: string; note?: string; residue?: string[] }> }; + ledger_head?: string; + region?: string; + generated_ms?: number; +}): Buffer { + const { run, records } = o; + const L: string[] = []; + L.push(`# Reversi report: run ${run.run_id}`, ""); + L.push(`Actor: ${run.actor} Mode: ${run.mode} Tenant: ${run.tenant ?? "-"}`); + L.push(`Started: ${t(run.created_ms)} Scope seal: ${run.scope.seal.slice(0, 16)}`); + L.push(`Permissions: ${run.scope.permissions.join(" ")}`); + if (o.region) L.push(`Data region: ${o.region}`); + L.push("", "# Steps"); + for (const s of run.steps) L.push(`${String(s.step).padStart(3)} ${s.tool} [${s.effect ?? "no manifest"}] ${s.status}${s.error ? ` error: ${s.error}` : ""}`); + if (!run.steps.length) L.push("(no steps)"); + if (o.plan) { + L.push("", "# Rollback outlook", o.plan.fully_reversible ? "Every standing step can be undone, assuming each compensation succeeds." : "NOT fully reversible. See below."); + for (const a of o.plan.actions) L.push(`step ${a.step} ${a.tool}: ${a.kind}${a.note ? ` - ${a.note}` : ""}${a.residue?.length ? ` (residue: ${a.residue.join("; ")})` : ""}`); + } + L.push("", "# Event trail"); + for (const r of records) L.push(`${String(r.seq).padStart(5)} ${t(r.ts_ms)} ${r.event.padEnd(19)} ${r.tool} by ${r.actor}${r.step !== null ? ` step ${r.step}` : ""}`, ` why: ${r.reason}${r.detail ? ` detail: ${r.detail}` : ""} hash ${r.hash.slice(0, 12)}`); + L.push("", `Ledger head at generation: ${o.ledger_head ?? "n/a"}`, "Arguments and results are never stored; each record carries only their hash.", `Generated ${t(o.generated_ms ?? Date.now())}.`); + return renderPdf(L, `Reversi report ${run.run_id}`); +} diff --git a/packages/reversi/src/pii.ts b/packages/reversi/src/pii.ts new file mode 100644 index 0000000..e8a6701 --- /dev/null +++ b/packages/reversi/src/pii.ts @@ -0,0 +1,76 @@ +/** + * Detection and masking of personal data in free text. Patterns are deliberately conservative (a number has to pass its + * checksum to count as a card or IBAN) so that masking and flagging stay trustworthy. + */ + +export type PiiKind = "email" | "phone" | "credit_card" | "ssn" | "iban"; + +const EMAIL = /\b[A-Za-z0-9._%+-]{1,64}@[A-Za-z0-9.-]{1,255}\.[A-Za-z]{2,24}\b/g; +const PHONE = /(?= 0; i--) { + let d = digits.charCodeAt(i) - 48; + if (dbl) { + d *= 2; + if (d > 9) d -= 9; + } + sum += d; + dbl = !dbl; + } + return sum % 10 === 0; +} + +function ibanOk(raw: string): boolean { + const s = raw.replace(/ /g, ""); + if (s.length < 15 || s.length > 34) return false; + const rearranged = s.slice(4) + s.slice(0, 4); + let rem = 0; + for (const ch of rearranged) { + const v = /[A-Z]/.test(ch) ? String(ch.charCodeAt(0) - 55) : ch; + for (const d of v) rem = (rem * 10 + (d.charCodeAt(0) - 48)) % 97; + } + return rem === 1; +} + +interface Hit { + kind: PiiKind; + start: number; + end: number; +} + +function scan(text: string): Hit[] { + const hits: Hit[] = []; + const run = (re: RegExp, kind: PiiKind, ok: (m: string) => boolean = () => true) => { + re.lastIndex = 0; + for (let m = re.exec(text); m; m = re.exec(text)) if (ok(m[0])) hits.push({ kind, start: m.index, end: m.index + m[0].length }); + }; + run(EMAIL, "email"); + run(CARD, "credit_card", (m) => { + const d = m.replace(/[ -]/g, ""); + return d.length >= 13 && d.length <= 19 && luhn(d); + }); + run(IBAN, "iban", ibanOk); + run(SSN, "ssn"); + run(PHONE, "phone", (m) => m.replace(/\D/g, "").length >= 9 && m.replace(/\D/g, "").length <= 15 && !hits.some((h) => h.start <= text.indexOf(m) && h.end >= text.indexOf(m) + m.length)); + // A span claimed by a stronger kind (a card number also looks like a phone number) is not counted twice. + return hits.filter((h, i) => !hits.some((o, j) => j !== i && o.start <= h.start && o.end >= h.end && (o.end - o.start > h.end - h.start || (o.end - o.start === h.end - h.start && j < i)))); +} + +/** Which kinds of personal data appear in the text. */ +export function detectPii(text: string): PiiKind[] { + return [...new Set(scan(text).map((h) => h.kind))]; +} + +/** Replace personal data with `[PII:kind]`. */ +export function maskPii(text: string): string { + const hits = scan(text).sort((a, b) => b.start - a.start); + let out = text; + for (const h of hits) out = out.slice(0, h.start) + `[PII:${h.kind}]` + out.slice(h.end); + return out; +} diff --git a/packages/reversi/src/policy.ts b/packages/reversi/src/policy.ts index dcc4ffc..e48d0ca 100644 --- a/packages/reversi/src/policy.ts +++ b/packages/reversi/src/policy.ts @@ -5,7 +5,12 @@ */ import { globAny } from "./glob.js"; import type { CheckAction, ContentConfig } from "./content.js"; -import { EFFECTS, resolveRef, type Effect } from "./manifest.js"; +import { WindowCounters } from "./counters.js"; +import type { KvStore } from "./kv.js"; +import { EFFECTS, parseDuration, resolveRef, type Effect } from "./manifest.js"; +import { parseRetention, type RetentionConfig } from "./retention.js"; +import { parseRisk, scoreRisk, type RiskConfig, type RiskResult } from "./risk.js"; +import { frozenAt, inWindow, parseFreeze, parseWindow, type Freeze, type Window } from "./schedule.js"; export type PolicyAction = "allow" | "deny" | "require_approval"; const OPS = ["eq", "ne", "gt", "gte", "lt", "lte", "in", "not_in", "matches", "exists", "contains"] as const; @@ -24,6 +29,20 @@ export interface Match { permission?: string | string[]; actor?: string | string[]; environment?: string | string[]; + tenant?: string | string[]; + data_class?: string | string[]; + blast_radius?: string | string[]; +} + +/** Who must approve, how many, and what happens when nobody answers. Several requirements on one call are merged. */ +export interface ApprovalRequirement { + /** Reversi roles or group names. An approver must hold at least one. Empty: any approver. */ + roles: string[]; + quorum: number; + timeout_ms?: number; + on_timeout?: "deny" | "escalate"; + escalate_to?: string[]; + source: string; } export interface PolicyRule { @@ -31,8 +50,30 @@ export interface PolicyRule { description?: string; match: Match; when?: Condition[]; + /** Only applies inside (or, with `outside`, outside) this time window. */ + window?: Window; action: PolicyAction; reason?: string; + /** For `require_approval`: who may approve, and how many of them. */ + approvers?: string[]; + quorum?: number; + approval_timeout?: string | number; + on_timeout?: "deny" | "escalate"; + escalate_to?: string[]; +} + +export interface Budget { + id: string; + match: Match; + metric: "calls" | "sum"; + /** For `sum`: `$args.amount`. */ + path?: string; + max: number; + window_seconds: number; + per: "run" | "actor" | "tenant" | "global"; + action: "deny" | "require_approval"; + /** Warn (in the ledger) from this share of the budget. Default 0.8. */ + warn_at: number; } export interface Limit { @@ -40,7 +81,7 @@ export interface Limit { match: Match; max_calls: number; window_seconds: number; - per: "run" | "actor" | "global"; + per: "run" | "actor" | "tenant" | "global"; } export interface PolicyTestCase { @@ -57,6 +98,12 @@ export interface PolicyDoc { default: "allow" | "deny"; rules: PolicyRule[]; limits: Limit[]; + budgets: Budget[]; + /** Change freezes: no non-read call in these periods. */ + freezes: Freeze[]; + risk?: RiskConfig; + /** How long run state, approvals and captured data are kept. */ + retention?: RetentionConfig; tests: PolicyTestCase[]; /** Content-check settings, if the file has a `content:` section. */ content?: Partial; @@ -70,6 +117,9 @@ export interface PolicyInput { run_id: string; environment: string; args: unknown; + tenant?: string; + data_class?: string; + blast_radius?: string; } export interface PolicyDecision { @@ -77,6 +127,11 @@ export interface PolicyDecision { deny: string[]; approve: string[]; fired: string[]; + /** Who must approve what needs approval, from rules that name approvers or a quorum. */ + requirements: ApprovalRequirement[]; + risk?: RiskResult; + /** Things worth a note in the ledger that do not block: budget nearly spent. */ + warnings: string[]; } export class PolicyError extends Error { @@ -96,7 +151,7 @@ function validateMatch(m: unknown, where: string, errors: string[]): Match { return {}; } for (const k of Object.keys(m)) { - if (!["tool", "effect", "permission", "actor", "environment"].includes(k)) errors.push(`${where}: unknown match field \`${k}\``); + if (!["tool", "effect", "permission", "actor", "environment", "tenant", "data_class", "blast_radius"].includes(k)) errors.push(`${where}: unknown match field \`${k}\``); else if (!strOrList(m[k])) errors.push(`${where}: match.${k} must be a string or list of strings`); } const eff = m.effect === undefined ? [] : Array.isArray(m.effect) ? m.effect : [m.effect]; @@ -131,9 +186,9 @@ function parseContent(raw: unknown, errors: string[]): Partial | } const out: Partial = {}; for (const k of Object.keys(raw)) { - if (!["secrets", "encoded_payloads", "injection", "cumulative", "max_string_bytes"].includes(k)) errors.push(`content: unknown field \`${k}\``); + if (!["secrets", "encoded_payloads", "injection", "pii", "cumulative", "max_string_bytes"].includes(k)) errors.push(`content: unknown field \`${k}\``); } - for (const k of ["secrets", "encoded_payloads", "injection"] as const) { + for (const k of ["secrets", "encoded_payloads", "injection", "pii"] as const) { if (raw[k] === undefined) continue; if (!CHECK_ACTIONS.includes(raw[k] as CheckAction)) errors.push(`content.${k} must be ${CHECK_ACTIONS.join(", ")}`); else out[k] = raw[k] as CheckAction; @@ -162,7 +217,7 @@ function parseContent(raw: unknown, errors: string[]): Partial | export function parsePolicy(raw: unknown): PolicyDoc { const errors: string[] = []; if (!isRec(raw)) throw new PolicyError(["policy must be an object"]); - for (const k of Object.keys(raw)) if (!["version", "default", "rules", "limits", "tests", "content"].includes(k)) errors.push(`unknown field \`${k}\``); + for (const k of Object.keys(raw)) if (!["version", "default", "rules", "limits", "budgets", "freezes", "risk", "retention", "tests", "content"].includes(k)) errors.push(`unknown field \`${k}\``); if (raw.version !== undefined && raw.version !== 1) errors.push("`version` must be 1"); const def = raw.default ?? "allow"; if (def !== "allow" && def !== "deny") errors.push("`default` must be allow or deny"); @@ -174,7 +229,7 @@ export function parsePolicy(raw: unknown): PolicyDoc { errors.push(`${where}: must be an object`); continue; } - for (const k of Object.keys(r)) if (!["id", "description", "match", "when", "action", "reason"].includes(k)) errors.push(`${where}: unknown field \`${k}\``); + for (const k of Object.keys(r)) if (!["id", "description", "match", "when", "window", "action", "reason", "approvers", "quorum", "approval_timeout", "on_timeout", "escalate_to"].includes(k)) errors.push(`${where}: unknown field \`${k}\``); if (typeof r.id !== "string" || !r.id) errors.push(`${where}: id is required`); else if (ids.has(r.id)) errors.push(`${where}: duplicate id \`${r.id}\``); else ids.add(r.id); @@ -182,7 +237,13 @@ export function parsePolicy(raw: unknown): PolicyDoc { const match = validateMatch(r.match, where, errors); const when = r.when === undefined ? [] : Array.isArray(r.when) ? r.when : (errors.push(`${where}: when must be a list`), []); when.forEach((c, j) => validateCondition(c, `${where}.when[${j}]`, errors)); - rules.push({ ...(r as unknown as PolicyRule), match, when: when as Condition[] }); + const window = parseWindow(r.window, where, errors); + if (r.approvers !== undefined && (!Array.isArray(r.approvers) || r.approvers.some((a) => typeof a !== "string" || !a))) errors.push(`${where}: approvers must be a list of role or group names`); + if (r.quorum !== undefined && (!Number.isInteger(r.quorum) || (r.quorum as number) < 1 || (r.quorum as number) > 10)) errors.push(`${where}: quorum must be 1..10`); + if (r.approval_timeout !== undefined && parseDuration(r.approval_timeout) === undefined) errors.push(`${where}: approval_timeout must be a duration like 15m`); + if (r.on_timeout !== undefined && r.on_timeout !== "deny" && r.on_timeout !== "escalate") errors.push(`${where}: on_timeout must be deny or escalate`); + if ((r.approvers !== undefined || r.quorum !== undefined || r.approval_timeout !== undefined) && r.action !== "require_approval") errors.push(`${where}: approvers, quorum and approval_timeout only apply to require_approval rules`); + rules.push({ ...(r as unknown as PolicyRule), match, when: when as Condition[], ...(window ? { window } : {}) }); } const limits: Limit[] = []; for (const [i, l] of (Array.isArray(raw.limits) ? raw.limits : raw.limits === undefined ? [] : (errors.push("`limits` must be a list"), [])).entries()) { @@ -194,13 +255,39 @@ export function parsePolicy(raw: unknown): PolicyDoc { if (typeof l.id !== "string" || !l.id) errors.push(`${where}: id is required`); if (!Number.isInteger(l.max_calls) || (l.max_calls as number) < 0) errors.push(`${where}: max_calls must be a non-negative integer`); if (typeof l.window_seconds !== "number" || !(l.window_seconds > 0)) errors.push(`${where}: window_seconds must be positive`); - if (l.per !== undefined && !["run", "actor", "global"].includes(l.per as string)) errors.push(`${where}: per must be run, actor or global`); + if (l.per !== undefined && !["run", "actor", "tenant", "global"].includes(l.per as string)) errors.push(`${where}: per must be run, actor, tenant or global`); limits.push({ ...(l as unknown as Limit), per: (l.per as Limit["per"]) ?? "run", match: validateMatch(l.match, where, errors) }); } + const budgets: Budget[] = []; + for (const [i, b] of (Array.isArray(raw.budgets) ? raw.budgets : raw.budgets === undefined ? [] : (errors.push("`budgets` must be a list"), [])).entries()) { + const where = `budgets[${i}]`; + if (!isRec(b)) { + errors.push(`${where}: must be an object`); + continue; + } + for (const k of Object.keys(b)) if (!["id", "match", "metric", "path", "max", "window_seconds", "per", "action", "warn_at"].includes(k)) errors.push(`${where}: unknown field \`${k}\``); + if (typeof b.id !== "string" || !b.id) errors.push(`${where}: id is required`); + if (b.metric !== "calls" && b.metric !== "sum") errors.push(`${where}: metric must be calls or sum`); + if (b.metric === "sum" && (typeof b.path !== "string" || !/^\$args(\.[A-Za-z0-9_-]+)+$/.test(b.path))) errors.push(`${where}: a sum budget needs path like $args.amount`); + if (typeof b.max !== "number" || !(b.max >= 0)) errors.push(`${where}: max must be a non-negative number`); + if (typeof b.window_seconds !== "number" || !(b.window_seconds > 0)) errors.push(`${where}: window_seconds must be positive`); + if (b.per !== undefined && !["run", "actor", "tenant", "global"].includes(b.per as string)) errors.push(`${where}: per must be run, actor, tenant or global`); + if (b.action !== undefined && b.action !== "deny" && b.action !== "require_approval") errors.push(`${where}: action must be deny or require_approval`); + if (b.warn_at !== undefined && (typeof b.warn_at !== "number" || b.warn_at <= 0 || b.warn_at > 1)) errors.push(`${where}: warn_at must be above 0 and at most 1`); + budgets.push({ ...(b as unknown as Budget), per: (b.per as Budget["per"]) ?? "run", action: (b.action as Budget["action"]) ?? "deny", warn_at: (b.warn_at as number) ?? 0.8, match: validateMatch(b.match, where, errors) }); + } + const freezes: Freeze[] = []; + if (raw.freezes !== undefined && !Array.isArray(raw.freezes)) errors.push("`freezes` must be a list"); + else for (const [i, f] of ((raw.freezes as unknown[] | undefined) ?? []).entries()) { + const fr = parseFreeze(f, `freezes[${i}]`, errors); + if (fr) freezes.push(fr); + } + const risk = parseRisk(raw.risk, errors); + const retention = parseRetention(raw.retention, errors); const tests = raw.tests === undefined ? [] : Array.isArray(raw.tests) ? (raw.tests as PolicyTestCase[]) : (errors.push("`tests` must be a list"), []); const content = parseContent(raw.content, errors); if (errors.length) throw new PolicyError(errors); - return { version: 1, default: def as "allow" | "deny", rules, limits, tests, ...(content ? { content } : {}) }; + return { version: 1, default: def as "allow" | "deny", rules, limits, budgets, freezes, ...(risk ? { risk } : {}), ...(retention ? { retention } : {}), tests, ...(content ? { content } : {}) }; } function matches(m: Match, i: PolicyInput): boolean { @@ -209,7 +296,10 @@ function matches(m: Match, i: PolicyInput): boolean { (m.effect === undefined || (i.effect !== null && (Array.isArray(m.effect) ? m.effect : [m.effect]).includes(i.effect))) && (m.permission === undefined || (i.permission !== null && globAny(m.permission, i.permission))) && globAny(m.actor, i.actor) && - globAny(m.environment, i.environment) + globAny(m.environment, i.environment) && + (m.tenant === undefined || (i.tenant !== undefined && globAny(m.tenant, i.tenant))) && + (m.data_class === undefined || (i.data_class !== undefined && globAny(m.data_class, i.data_class))) && + (m.blast_radius === undefined || (i.blast_radius !== undefined && globAny(m.blast_radius, i.blast_radius))) ); } @@ -258,59 +348,118 @@ function holds(c: Condition, args: unknown, failClosed = false): boolean { } export class Policy { - /** Call timestamps per limit bucket, for sliding-window limits. */ - private readonly calls = new Map(); + private readonly counters: WindowCounters; constructor( readonly doc: PolicyDoc, private readonly clock: () => number = Date.now, - ) {} + /** Share counters (rate limits, budgets) between processes by passing the data store's KV. */ + kv?: KvStore, + ) { + this.counters = new WindowCounters(kv, clock); + } + + static from(raw: unknown, clock?: () => number, kv?: KvStore): Policy { + return new Policy(parsePolicy(raw), clock, kv); + } + + /** The amount a call adds to a budget: 1 for `calls`, the numeric argument for `sum`. */ + private amountOf(b: Budget, input: PolicyInput): number { + if (b.metric === "calls") return 1; + const n = numeric(resolveRef(b.path as string, "args", input.args)); + return n ?? 0; + } - static from(raw: unknown, clock?: () => number): Policy { - return new Policy(parsePolicy(raw), clock); + private scopeKey(id: string, per: "run" | "actor" | "tenant" | "global", i: PolicyInput): string { + return JSON.stringify([id, per === "run" ? i.run_id : per === "actor" ? i.actor : per === "tenant" ? (i.tenant ?? "") : ""]); + } + + /** Risk score for a call, using this policy's weights (or the defaults when it has none). */ + score(input: PolicyInput): RiskResult { + return scoreRisk(this.doc.risk, { effect: input.effect, blast_radius: input.blast_radius, data_class: input.data_class, environment: input.environment, actor: input.actor, args: input.args }); } evaluate(input: PolicyInput): PolicyDecision { const deny: string[] = []; const approve: string[] = []; const fired: string[] = []; + const warnings: string[] = []; + const requirements: ApprovalRequirement[] = []; + const now = this.clock(); let allowed = false; for (const r of this.doc.rules) { if (!matches(r.match, input) || !(r.when ?? []).every((c) => holds(c, input.args, r.action !== "allow"))) continue; + if (r.window && !inWindow(r.window, now)) continue; fired.push(r.id); const why = `policy \`${r.id}\`${r.reason ? `: ${r.reason}` : ""}`; if (r.action === "deny") deny.push(why); - else if (r.action === "require_approval") approve.push(why); - else allowed = true; + else if (r.action === "require_approval") { + approve.push(why); + if (r.approvers || r.quorum || r.approval_timeout) { + requirements.push({ + roles: r.approvers ?? [], + quorum: r.quorum ?? 1, + ...(r.approval_timeout !== undefined ? { timeout_ms: parseDuration(r.approval_timeout) } : {}), + ...(r.on_timeout ? { on_timeout: r.on_timeout } : {}), + ...(r.escalate_to ? { escalate_to: r.escalate_to } : {}), + source: `policy \`${r.id}\``, + }); + } + } else allowed = true; } if (this.doc.default === "deny" && !allowed && deny.length === 0) deny.push("policy: no allow rule matches this call (default deny)"); + + for (const f of this.doc.freezes) { + if (!frozenAt(f, now)) continue; + const m = f.match; + const applies = m + ? globAny(m.tool, input.tool) && (m.effect === undefined || (input.effect !== null && (Array.isArray(m.effect) ? m.effect : [m.effect]).includes(input.effect))) && globAny(m.environment, input.environment) + : input.effect !== "read_only"; + if (applies) { + deny.push(`freeze \`${f.id}\`: ${f.reason} (until ${f.to})`); + fired.push(f.id); + } + } + for (const l of this.doc.limits) { if (!matches(l.match, input)) continue; - const key = this.bucket(l, input); - const horizon = this.clock() - l.window_seconds * 1000; - const recent = (this.calls.get(key) ?? []).filter((t) => t > horizon); - if (recent.length >= l.max_calls) { + const { count } = this.counters.total(`limit:${this.scopeKey(l.id, l.per, input)}`, l.window_seconds * 1000); + if (count >= l.max_calls) { deny.push(`limit \`${l.id}\`: ${l.max_calls} calls per ${l.window_seconds}s exceeded`); fired.push(l.id); } } - return { action: deny.length ? "deny" : approve.length ? "require_approval" : "allow", deny, approve, fired }; + + for (const b of this.doc.budgets) { + if (!matches(b.match, input)) continue; + const { count, sum } = this.counters.total(`budget:${this.scopeKey(b.id, b.per, input)}`, b.window_seconds * 1000); + const used = b.metric === "calls" ? count : sum; + const after = used + this.amountOf(b, input); + if (after > b.max) { + (b.action === "deny" ? deny : approve).push(`budget \`${b.id}\`: ${used} of ${b.max} used in ${b.window_seconds}s, this call would make it ${after}`); + fired.push(b.id); + } else if (b.max > 0 && after >= b.max * b.warn_at) { + warnings.push(`budget \`${b.id}\` at ${Math.round((after / b.max) * 100)}% (${after} of ${b.max})`); + } + } + + let risk: RiskResult | undefined; + if (this.doc.risk) { + risk = this.score(input); + const t = this.doc.risk.thresholds; + if (t?.deny !== undefined && risk.score >= t.deny) deny.push(`risk score ${risk.score} is at or above the deny threshold ${t.deny} (${risk.factors.join(", ")})`); + else if (t?.approve !== undefined && risk.score >= t.approve) approve.push(`risk score ${risk.score} is at or above the approval threshold ${t.approve} (${risk.factors.join(", ")})`); + } + return { action: deny.length ? "deny" : approve.length ? "require_approval" : "allow", deny, approve, fired, requirements, ...(risk ? { risk } : {}), warnings }; } - /** Count a call that was actually allowed to start, for the limits. */ + /** Count a call that was actually allowed to start, for the limits and budgets. */ record(input: PolicyInput): void { - const now = this.clock(); for (const l of this.doc.limits) { - if (!matches(l.match, input)) continue; - const key = this.bucket(l, input); - const horizon = now - l.window_seconds * 1000; - const list = (this.calls.get(key) ?? []).filter((t) => t > horizon); - list.push(now); - this.calls.set(key, list); + if (matches(l.match, input)) this.counters.add(`limit:${this.scopeKey(l.id, l.per, input)}`, 1, l.window_seconds * 1000); + } + for (const b of this.doc.budgets) { + if (matches(b.match, input)) this.counters.add(`budget:${this.scopeKey(b.id, b.per, input)}`, this.amountOf(b, input), b.window_seconds * 1000); } - } - - private bucket(l: Limit, i: PolicyInput): string { - return JSON.stringify([l.id, l.per === "run" ? i.run_id : l.per === "actor" ? i.actor : ""]); } } @@ -356,6 +505,10 @@ export function mergePolicyDocs(...docs: Array): PolicyDo default: present.some((d) => d.default === "deny") ? "deny" : "allow", rules: present.flatMap((d) => d.rules), limits: present.flatMap((d) => d.limits), + budgets: present.flatMap((d) => d.budgets), + freezes: present.flatMap((d) => d.freezes), + risk: present.find((d) => d.risk)?.risk, + retention: present.find((d) => d.retention)?.retention, tests: present.flatMap((d) => d.tests), content: present.find((d) => d.content)?.content, }; diff --git a/packages/reversi/src/policyversions.ts b/packages/reversi/src/policyversions.ts new file mode 100644 index 0000000..4a98e61 --- /dev/null +++ b/packages/reversi/src/policyversions.ts @@ -0,0 +1,115 @@ +/** + * Versioned policy files. Publishing validates the policy and runs its own tests (a policy whose tests fail is refused); + * activating a version makes it the one the engine loads; rollback re-activates an older one. Everything keeps its + * history, checksum and author. + */ +import { existsSync, mkdirSync, readFileSync, renameSync, writeFileSync } from "node:fs"; +import { join } from "node:path"; +import { parse as parseYaml } from "yaml"; +import { sha256Hex } from "./canonical.js"; +import { renderDiff } from "./diff.js"; +import { parsePolicy, runPolicyTests, type PolicyDoc } from "./policy.js"; + +export interface PolicyVersion { + version: number; + checksum: string; + created_ms: number; + by: string; + note?: string; + tests_passed: number; +} +interface PolicyIndex { + format: "reversi-policy-versions"; + version: 1; + active: number | null; + versions: PolicyVersion[]; +} + +export class PolicyVersionError extends Error { + constructor(message: string) { + super(message); + this.name = "PolicyVersionError"; + } +} + +export class PolicyVersions { + constructor( + readonly dir: string, + private readonly now: () => number = Date.now, + ) {} + + static isVersioned(dir: string): boolean { + try { + return (JSON.parse(readFileSync(join(dir, "policy-index.json"), "utf8")) as PolicyIndex).format === "reversi-policy-versions"; + } catch { + return false; + } + } + + private read(): PolicyIndex { + const p = join(this.dir, "policy-index.json"); + return existsSync(p) ? (JSON.parse(readFileSync(p, "utf8")) as PolicyIndex) : { format: "reversi-policy-versions", version: 1, active: null, versions: [] }; + } + private write(ix: PolicyIndex): void { + mkdirSync(this.dir, { recursive: true }); + const tmp = join(this.dir, `policy-index.json.${process.pid}.tmp`); + writeFileSync(tmp, JSON.stringify(ix, null, 2)); + renameSync(tmp, join(this.dir, "policy-index.json")); + } + private file(v: number) { + return join(this.dir, `policy-v${v}.yaml`); + } + + history(): PolicyVersion[] { + return this.read().versions; + } + activeVersion(): number | null { + return this.read().active; + } + + text(version: number): string { + const v = this.read().versions.find((x) => x.version === version); + if (!v) throw new PolicyVersionError(`there is no policy version ${version}`); + const text = readFileSync(this.file(version), "utf8"); + if (sha256Hex(text) !== v.checksum) throw new PolicyVersionError(`policy v${version} was changed outside the version history: its checksum does not match`); + return text; + } + + activeText(): string | undefined { + const a = this.read().active; + return a === null ? undefined : this.text(a); + } + + doc(version: number): PolicyDoc { + return parsePolicy(parseYaml(this.text(version))); + } + + /** Validate, run the policy's own tests, store as a new version, and (by default) make it active. */ + publish(text: string, o: { by: string; note?: string; activate?: boolean; allow_failing_tests?: boolean }): PolicyVersion { + const doc = parsePolicy(parseYaml(text)); + const results = runPolicyTests(doc); + const failed = results.filter((r) => !r.ok); + if (failed.length && !o.allow_failing_tests) throw new PolicyVersionError(`${failed.length} of the policy's own tests fail: ${failed.map((f) => f.name).join(", ")}`); + const ix = this.read(); + const version = (ix.versions.at(-1)?.version ?? 0) + 1; + mkdirSync(this.dir, { recursive: true }); + writeFileSync(this.file(version), text); + const rec: PolicyVersion = { version, checksum: sha256Hex(text), created_ms: this.now(), by: o.by, ...(o.note ? { note: o.note } : {}), tests_passed: results.length - failed.length }; + ix.versions.push(rec); + if (o.activate !== false) ix.active = version; + this.write(ix); + return rec; + } + + activate(version: number): void { + const ix = this.read(); + if (!ix.versions.some((v) => v.version === version)) throw new PolicyVersionError(`there is no policy version ${version}`); + this.text(version); // refuses a tampered file + ix.active = version; + this.write(ix); + } + + diff(a: number, b: number): string { + return renderDiff(this.text(a), this.text(b), 3); + } +} diff --git a/packages/reversi/src/postgres.ts b/packages/reversi/src/postgres.ts index b3b936d..fa9c67c 100644 --- a/packages/reversi/src/postgres.ts +++ b/packages/reversi/src/postgres.ts @@ -12,6 +12,7 @@ import { createRequire } from "node:module"; import { MessageChannel, Worker, receiveMessageOnPort, type MessagePort } from "node:worker_threads"; import type { ApprovalRecord, ApprovalStatus, ApprovalStore } from "./approvals.js"; +import type { KvEntry, KvStore } from "./kv.js"; import { LedgerConflictError, type LedgerRecord, type LedgerStore } from "./ledger.js"; import { RUN_ID_PATTERN, type RunState, type RunStore } from "./store.js"; @@ -114,6 +115,10 @@ export const MIGRATIONS: Array<{ version: number; sql: string }> = [ CREATE TABLE approvals (id TEXT PRIMARY KEY, record JSONB NOT NULL); `, }, + { + version: 2, + sql: `CREATE TABLE kv (ns TEXT NOT NULL, key TEXT NOT NULL, value JSONB NOT NULL, version BIGINT NOT NULL, PRIMARY KEY (ns, key));`, + }, ]; export function migrate(pg: SyncPg): number[] { @@ -175,6 +180,9 @@ export class PgRunStore implements RunStore { JSON.stringify(state), ]); } + delete(run_id: string): boolean { + return this.pg.query("DELETE FROM runs WHERE run_id = $1 RETURNING run_id", [this.check(run_id)]).length === 1; + } list(): string[] { return this.pg.query<{ run_id: string }>("SELECT run_id FROM runs ORDER BY run_id").map((r) => r.run_id); } @@ -191,16 +199,47 @@ export class PgApprovalStore implements ApprovalStore { list(): ApprovalRecord[] { return this.pg.query<{ record: ApprovalRecord }>("SELECT record FROM approvals").map((r) => r.record); } - transition(id: string, from: ApprovalStatus, patch: Partial): boolean { - return ( - this.pg.query("UPDATE approvals SET record = record || $1::jsonb WHERE id = $2 AND record->>'status' = $3 RETURNING id", [JSON.stringify(patch), id, from]) - .length === 1 - ); + delete(id: string): boolean { + return this.pg.query("DELETE FROM approvals WHERE id = $1 RETURNING id", [id]).length === 1; + } + transition(id: string, from: ApprovalStatus, patch: Partial, expectCount?: number): boolean { + const params: unknown[] = [JSON.stringify(patch), id, from]; + let extra = ""; + if (expectCount !== undefined) { + params.push(expectCount); + extra = " AND COALESCE((record->>'decision_count')::int, 0) = $4"; + } + return this.pg.query(`UPDATE approvals SET record = record || $1::jsonb WHERE id = $2 AND record->>'status' = $3${extra} RETURNING id`, params).length === 1; + } +} + +export class PgKvStore implements KvStore { + constructor(private readonly pg: SyncPg) {} + get(ns: string, key: string): KvEntry | undefined { + const r = this.pg.query<{ value: T; version: string }>("SELECT value, version FROM kv WHERE ns = $1 AND key = $2", [ns, key])[0]; + return r ? { key, value: r.value, version: Number(r.version) } : undefined; + } + put(ns: string, key: string, value: unknown): void { + this.pg.query("INSERT INTO kv (ns, key, value, version) VALUES ($1, $2, $3, 1) ON CONFLICT (ns, key) DO UPDATE SET value = EXCLUDED.value, version = kv.version + 1", [ns, key, JSON.stringify(value)]); + } + cas(ns: string, key: string, value: unknown, expected: number): boolean { + if (expected === 0) { + return this.pg.query("INSERT INTO kv (ns, key, value, version) VALUES ($1, $2, $3, 1) ON CONFLICT (ns, key) DO NOTHING RETURNING key", [ns, key, JSON.stringify(value)]).length === 1; + } + return this.pg.query("UPDATE kv SET value = $3, version = version + 1 WHERE ns = $1 AND key = $2 AND version = $4 RETURNING key", [ns, key, JSON.stringify(value), expected]).length === 1; + } + delete(ns: string, key: string): boolean { + return this.pg.query("DELETE FROM kv WHERE ns = $1 AND key = $2 RETURNING key", [ns, key]).length === 1; + } + list(ns: string, prefix = ""): KvEntry[] { + return this.pg + .query<{ key: string; value: T; version: string }>("SELECT key, value, version FROM kv WHERE ns = $1 AND left(key, $2) = $3 ORDER BY key", [ns, prefix.length, prefix]) + .map((r) => ({ key: r.key, value: r.value, version: Number(r.version) })); } } -export function openPostgresStores(url: string): { pg: SyncPg; ledger: PgLedgerStore; runs: PgRunStore; approvals: PgApprovalStore } { +export function openPostgresStores(url: string): { pg: SyncPg; ledger: PgLedgerStore; runs: PgRunStore; approvals: PgApprovalStore; kv: PgKvStore } { const pg = new SyncPg(url); migrate(pg); - return { pg, ledger: new PgLedgerStore(pg), runs: new PgRunStore(pg), approvals: new PgApprovalStore(pg) }; + return { pg, ledger: new PgLedgerStore(pg), runs: new PgRunStore(pg), approvals: new PgApprovalStore(pg), kv: new PgKvStore(pg) }; } diff --git a/packages/reversi/src/projectconfig.ts b/packages/reversi/src/projectconfig.ts new file mode 100644 index 0000000..a7e1b33 --- /dev/null +++ b/packages/reversi/src/projectconfig.ts @@ -0,0 +1,220 @@ +/** + * The project file, `reversi.config.yaml`: one place that says where manifests and policy live, where data is kept, who + * is told about what, and so on. The CLI, the server and the gateway all read it; environment variables and command-line + * flags override it. `${NAME}` in a string is replaced from the environment, so credentials stay out of the file. + */ +import { existsSync, readFileSync } from "node:fs"; +import { dirname, extname, isAbsolute, join, resolve } from "node:path"; +import { parse as parseYaml } from "yaml"; +import { ROLES, type Role } from "./auth.js"; +import type { AnchorSink } from "./anchor.js"; +import { parseChannels, parseRoutes, type ChannelConfig, type Route } from "./channels.js"; +import type { StoreKind } from "./data.js"; +import type { EgressConfig } from "./egress.js"; +import { parseDuration } from "./manifest.js"; + +export interface ProjectConfig { + /** Directory the file was found in; relative paths are resolved against it. */ + root: string; + file: string; + manifests?: string; + policy?: string; + data?: { dir?: string; store?: StoreKind; database_url?: string }; + environment?: string; + kill_switch_file?: string; + approval?: { ttl_ms?: number; webhook?: string; console_url?: string }; + signing_key?: string; + manifest_public_key?: string; + capture?: { enabled: boolean; ttl_ms?: number; redact_pii?: boolean }; + notifications?: { channels: ChannelConfig[]; routes: Route[] }; + otel?: { endpoint: string; headers?: Record; service_name?: string }; + anchor?: { sinks: AnchorSink[]; interval_ms?: number }; + egress?: EgressConfig; + alerts?: { interval_ms?: number; remind_ms?: number; denial_spike?: number }; + console?: { mode: "off" | "open" | "login"; role?: Role }; + log?: { level?: "debug" | "info" | "warn" | "error"; format?: "json" | "pretty" }; + reload?: { watch?: boolean; interval_ms?: number }; +} + +export class ProjectConfigError extends Error { + constructor(public readonly errors: string[]) { + super(errors.join("; ")); + this.name = "ProjectConfigError"; + } +} + +export const CONFIG_NAMES = ["reversi.config.yaml", "reversi.config.yml", "reversi.config.json"]; + +/** Look for the config file in `startDir` and its parents. `REVERSI_CONFIG` or an explicit path wins. */ +export function findProjectConfig(startDir = process.cwd(), explicit?: string, env: Record = process.env): string | undefined { + const pinned = explicit ?? env.REVERSI_CONFIG; + if (pinned) { + if (!existsSync(pinned)) throw new ProjectConfigError([`config file not found: ${pinned}`]); + return resolve(pinned); + } + for (let dir = resolve(startDir); ; dir = dirname(dir)) { + for (const n of CONFIG_NAMES) if (existsSync(join(dir, n))) return join(dir, n); + if (dirname(dir) === dir) return undefined; + } +} + +function expand(v: unknown, env: Record, errors: string[], where: string): unknown { + if (typeof v === "string") { + return v.replace(/\$\{([A-Za-z_][A-Za-z0-9_]*)(?::-([^}]*))?\}/g, (_m, name: string, dflt: string | undefined) => { + const val = env[name] ?? dflt; + if (val === undefined) { + errors.push(`${where}: environment variable ${name} is not set`); + return ""; + } + return val; + }); + } + if (Array.isArray(v)) return v.map((x, i) => expand(x, env, errors, `${where}[${i}]`)); + if (typeof v === "object" && v !== null) return Object.fromEntries(Object.entries(v).map(([k, x]) => [k, expand(x, env, errors, `${where}.${k}`)])); + return v; +} + +const isRec = (x: unknown): x is Record => typeof x === "object" && x !== null && !Array.isArray(x); + +export function loadProjectConfig(file: string, env: Record = process.env): ProjectConfig { + const errors: string[] = []; + let raw: unknown; + try { + const text = readFileSync(file, "utf8"); + raw = extname(file).toLowerCase() === ".json" ? JSON.parse(text) : parseYaml(text); + } catch (e) { + throw new ProjectConfigError([`${file}: cannot read: ${e instanceof Error ? e.message : String(e)}`]); + } + if (raw === null || raw === undefined) raw = {}; + if (!isRec(raw)) throw new ProjectConfigError([`${file}: the config must be a mapping`]); + const cfg = expand(raw, env, errors, "config") as Record; + const root = dirname(resolve(file)); + const path = (v: unknown, where: string): string | undefined => { + if (v === undefined) return undefined; + if (typeof v !== "string" || !v) { + errors.push(`${where} must be a path`); + return undefined; + } + return /^https?:\/\//.test(v) || isAbsolute(v) ? v : resolve(root, v); + }; + const dur = (v: unknown, where: string): number | undefined => { + if (v === undefined) return undefined; + const d = parseDuration(v); + if (d === undefined) errors.push(`${where} must be a duration like 30s or 15m`); + return d; + }; + const known = ["manifests", "policy", "data", "environment", "kill_switch_file", "approval", "signing_key", "manifest_public_key", "capture", "notifications", "otel", "anchor", "egress", "alerts", "console", "log", "reload"]; + for (const k of Object.keys(cfg)) if (!known.includes(k)) errors.push(`unknown setting \`${k}\``); + const out: ProjectConfig = { root, file: resolve(file) }; + out.manifests = path(cfg.manifests, "manifests"); + out.policy = path(cfg.policy, "policy"); + out.signing_key = path(cfg.signing_key, "signing_key"); + out.manifest_public_key = path(cfg.manifest_public_key, "manifest_public_key"); + out.kill_switch_file = path(cfg.kill_switch_file, "kill_switch_file"); + if (cfg.environment !== undefined) typeof cfg.environment === "string" ? (out.environment = cfg.environment) : errors.push("environment must be a string"); + + if (cfg.data !== undefined) { + if (!isRec(cfg.data)) errors.push("data must be a mapping"); + else { + for (const k of Object.keys(cfg.data)) if (!["dir", "store", "database_url"].includes(k)) errors.push(`data: unknown setting \`${k}\``); + if (cfg.data.store !== undefined && !["files", "sqlite", "postgres"].includes(cfg.data.store as string)) errors.push("data.store must be files, sqlite or postgres"); + out.data = { dir: path(cfg.data.dir, "data.dir"), store: cfg.data.store as StoreKind | undefined, database_url: cfg.data.database_url as string | undefined }; + } + } + if (cfg.approval !== undefined) { + if (!isRec(cfg.approval)) errors.push("approval must be a mapping"); + else { + for (const k of Object.keys(cfg.approval)) if (!["ttl", "webhook", "console_url"].includes(k)) errors.push(`approval: unknown setting \`${k}\``); + out.approval = { ttl_ms: dur(cfg.approval.ttl, "approval.ttl"), webhook: cfg.approval.webhook as string | undefined, console_url: cfg.approval.console_url as string | undefined }; + } + } + if (cfg.capture !== undefined) { + if (!isRec(cfg.capture)) errors.push("capture must be a mapping"); + else { + for (const k of Object.keys(cfg.capture)) if (!["enabled", "ttl", "redact_pii"].includes(k)) errors.push(`capture: unknown setting \`${k}\``); + if (cfg.capture.enabled !== undefined && typeof cfg.capture.enabled !== "boolean") errors.push("capture.enabled must be true or false"); + out.capture = { enabled: cfg.capture.enabled === true, ttl_ms: dur(cfg.capture.ttl, "capture.ttl"), redact_pii: cfg.capture.redact_pii as boolean | undefined }; + } + } + if (cfg.notifications !== undefined) { + if (!isRec(cfg.notifications)) errors.push("notifications must be a mapping"); + else { + for (const k of Object.keys(cfg.notifications)) if (!["channels", "routes"].includes(k)) errors.push(`notifications: unknown setting \`${k}\``); + const channels = parseChannels(cfg.notifications.channels, errors); + out.notifications = { channels, routes: parseRoutes(cfg.notifications.routes, channels, errors) }; + } + } + if (cfg.otel !== undefined) { + if (!isRec(cfg.otel) || typeof cfg.otel.endpoint !== "string") errors.push("otel needs an endpoint"); + else out.otel = cfg.otel as unknown as ProjectConfig["otel"]; + } + if (cfg.anchor !== undefined) { + if (!isRec(cfg.anchor) || !Array.isArray(cfg.anchor.sinks) || cfg.anchor.sinks.length === 0) errors.push("anchor needs a list of sinks"); + else { + const sinks = (cfg.anchor.sinks as Array>).map((s, i) => { + if (s.type === "file" && typeof s.path === "string") return { type: "file", path: path(s.path, `anchor.sinks[${i}].path`) as string } as AnchorSink; + if (s.type === "webhook" && typeof s.url === "string") return { type: "webhook", url: s.url, ...(typeof s.secret === "string" ? { secret: s.secret } : {}) } as AnchorSink; + errors.push(`anchor.sinks[${i}] must be { type: file, path } or { type: webhook, url }`); + return undefined; + }); + out.anchor = { sinks: sinks.filter((s): s is AnchorSink => !!s), interval_ms: dur(cfg.anchor.interval, "anchor.interval") }; + } + } + if (cfg.egress !== undefined) { + if (!isRec(cfg.egress)) errors.push("egress must be a mapping"); + else { + for (const k of Object.keys(cfg.egress)) if (!["allow", "block_private", "region", "allow_metadata"].includes(k)) errors.push(`egress: unknown setting \`${k}\``); + out.egress = cfg.egress as EgressConfig; + } + } + if (cfg.alerts !== undefined) { + if (!isRec(cfg.alerts)) errors.push("alerts must be a mapping"); + else out.alerts = { interval_ms: dur(cfg.alerts.interval, "alerts.interval"), remind_ms: dur(cfg.alerts.remind, "alerts.remind"), denial_spike: cfg.alerts.denial_spike as number | undefined }; + } + if (cfg.console !== undefined) { + if (!isRec(cfg.console)) errors.push("console must be a mapping"); + else { + if (!["off", "open", "login"].includes(cfg.console.mode as string)) errors.push("console.mode must be off, open or login"); + if (cfg.console.role !== undefined && (!ROLES.includes(cfg.console.role as Role) || cfg.console.role === "agent" || cfg.console.role === "admin")) errors.push("console.role must be auditor, approver or operator (an unauthenticated console is never admin)"); + out.console = { mode: cfg.console.mode as "off" | "open" | "login", role: cfg.console.role as Role | undefined }; + } + } + if (cfg.log !== undefined) { + if (!isRec(cfg.log)) errors.push("log must be a mapping"); + else out.log = cfg.log as ProjectConfig["log"]; + } + if (cfg.reload !== undefined) { + if (!isRec(cfg.reload)) errors.push("reload must be a mapping"); + else out.reload = { watch: cfg.reload.watch as boolean | undefined, interval_ms: dur(cfg.reload.interval, "reload.interval") }; + } + if (errors.length) throw new ProjectConfigError(errors.map((e) => `${file}: ${e}`)); + return out; +} + +/** A starter config for `reversi init`. */ +export function starterConfig(opts: { store: StoreKind }): string { + return `# Reversi project configuration. Every setting is optional; see the docs for the full list. +manifests: ./manifests # a folder of YAML/JSON files, a registry folder (reversi manifests publish), or an https URL +policy: ./policy.yaml +data: + dir: ./reversi-data + store: ${opts.store} # files | sqlite | postgres +environment: dev # matched by policy rules (dev, staging, prod) + +# Who is told about what. Secrets come from the environment: \${SLACK_WEBHOOK_URL}. +# notifications: +# channels: +# - { type: slack, name: ops, url: "\${SLACK_WEBHOOK_URL}", interactive: false } +# routes: +# - { events: [approval_requested, rollback_failed, alert], channels: [ops] } + +# Keep redacted arguments for replay and policy simulation (off by default). +# capture: +# enabled: true +# ttl: 7d + +# console: +# mode: open # off (default) | open (no login, localhost only) | login (keys or SSO) +# role: operator +`; +} diff --git a/packages/reversi/src/query.ts b/packages/reversi/src/query.ts new file mode 100644 index 0000000..f42aea6 --- /dev/null +++ b/packages/reversi/src/query.ts @@ -0,0 +1,115 @@ +/** Searching runs and ledger records: filters, summaries and pagination. */ +import type { Reversi } from "./engine.js"; +import type { LedgerEvent, LedgerRecord } from "./ledger.js"; +import type { RunState } from "./store.js"; + +export type RunStatus = "needs_attention" | "rolled_back" | "complete" | "blocked_only" | "empty"; + +export interface RunSummary { + run_id: string; + actor: string; + tenant?: string; + mode: string; + started_ms: number; + steps: number; + tools: string[]; + status: RunStatus; + unresolved: number[]; + failed_compensations: number; +} + +export interface RunFilter { + actor?: string; + tenant?: string; + tool?: string; + status?: RunStatus; + mode?: string; + since_ms?: number; + until_ms?: number; + /** Matches the run id. */ + text?: string; + limit?: number; + /** Run id to continue after (the last one of the previous page). */ + after?: string; +} + +const ATTENTION = new Set(["uncertain", "intended", "compensation_failed", "unrecoverable"]); + +export function summarizeRun(state: RunState, unresolved: number[], blocked: number): RunSummary { + const failed = state.steps.filter((s) => s.status === "compensation_failed").length; + const standing = state.steps.filter((s) => !["failed", "compensated", "no_action"].includes(s.status)); + let status: RunStatus; + if (state.steps.length === 0) status = blocked > 0 ? "blocked_only" : "empty"; + else if (unresolved.length || state.steps.some((s) => ATTENTION.has(s.status)) || standing.some((s) => s.effect === "irreversible")) status = "needs_attention"; + else if (state.steps.every((s) => ["compensated", "no_action", "failed"].includes(s.status)) && state.steps.some((s) => s.status === "compensated")) status = "rolled_back"; + else status = "complete"; + return { + run_id: state.run_id, + actor: state.actor, + ...(state.tenant ? { tenant: state.tenant } : {}), + mode: state.mode, + started_ms: state.created_ms, + steps: state.steps.length, + tools: [...new Set(state.steps.map((s) => s.tool))], + status, + unresolved, + failed_compensations: failed, + }; +} + +/** Runs matching the filter, oldest first by id order, with the next page cursor. */ +export function searchRuns(rv: Reversi, f: RunFilter = {}): { runs: RunSummary[]; next?: string } { + const limit = Math.min(Math.max(f.limit ?? 100, 1), 1000); + const blocked = new Map(); + for (const r of rv.ledger.records()) if (r.event === "blocked" && r.step === null) blocked.set(r.run_id, (blocked.get(r.run_id) ?? 0) + 1); + const ids = rv.runIds().sort(); + const out: RunSummary[] = []; + for (const id of ids) { + if (f.after && id <= f.after) continue; + const run = rv.openRun(id); + const st = run.state(); + if ((f.actor && st.actor !== f.actor) || (f.tenant && st.tenant !== f.tenant) || (f.mode && st.mode !== f.mode)) continue; + if ((f.since_ms && st.created_ms < f.since_ms) || (f.until_ms && st.created_ms > f.until_ms)) continue; + if (f.text && !id.includes(f.text)) continue; + if (f.tool && !st.steps.some((s) => s.tool === f.tool) && !rv.ledger.forRun(id).some((r) => r.tool === f.tool)) continue; + const sum = summarizeRun(st, run.unresolved(), blocked.get(id) ?? 0); + if (f.status && sum.status !== f.status) continue; + out.push(sum); + if (out.length === limit) return { runs: out, next: id }; + } + return { runs: out }; +} + +export interface RecordFilter { + run_id?: string; + actor?: string; + tool?: string; + event?: LedgerEvent | LedgerEvent[]; + since_ms?: number; + until_ms?: number; + /** Matches reason or detail. */ + text?: string; + limit?: number; + /** Return records with seq above this (cursor). */ + after_seq?: number; +} + +export function searchRecords(rv: Reversi, f: RecordFilter = {}): { records: LedgerRecord[]; next_seq?: number } { + const limit = Math.min(Math.max(f.limit ?? 200, 1), 5000); + const events = f.event === undefined ? undefined : new Set(Array.isArray(f.event) ? f.event : [f.event]); + const text = f.text?.toLowerCase(); + const out: LedgerRecord[] = []; + for (const r of rv.ledger.records()) { + if (f.after_seq !== undefined && r.seq <= f.after_seq) continue; + if (f.run_id && r.run_id !== f.run_id) continue; + if (f.actor && r.actor !== f.actor) continue; + if (f.tool && r.tool !== f.tool) continue; + if (events && !events.has(r.event)) continue; + if (f.since_ms && r.ts_ms < f.since_ms) continue; + if (f.until_ms && r.ts_ms > f.until_ms) continue; + if (text && !`${r.reason} ${r.detail}`.toLowerCase().includes(text)) continue; + out.push(r); + if (out.length === limit) return { records: out, next_seq: r.seq }; + } + return { records: out }; +} diff --git a/packages/reversi/src/registry.ts b/packages/reversi/src/registry.ts new file mode 100644 index 0000000..1287801 --- /dev/null +++ b/packages/reversi/src/registry.ts @@ -0,0 +1,251 @@ +/** + * A versioned manifest registry on disk (or served over HTTP as static files). + * + * /index.json the list of tools and their versions, each with a checksum + * /tools//v.json the manifest at that version + * + * Every publish makes a new immutable version. Versions start as `draft` and are enforceable only once approved; a + * deprecated tool keeps its history and still rolls back. Each version's checksum (and, if a signing key is used, an + * Ed25519 signature over it) is checked when the registry is loaded, so a manifest changed outside the workflow is refused. + */ +import { createPrivateKey, createPublicKey, sign, verify } from "node:crypto"; +import { existsSync, mkdirSync, readFileSync, renameSync, writeFileSync } from "node:fs"; +import { join } from "node:path"; +import { canonicalize, sha256Hex } from "./canonical.js"; +import { renderDiff } from "./diff.js"; +import { guardedFetch, type EgressConfig } from "./egress.js"; +import { keyId, type SigningKey } from "./evidence.js"; +import { Registry, authorForm, validateManifest, type ManifestStatus, type ToolManifest } from "./manifest.js"; + +export interface RegistryVersion { + version: number; + checksum: string; + status: ManifestStatus; + created_ms: number; + by: string; + note?: string; + approved_by?: string; + signature?: string; + key_id?: string; +} + +interface IndexFile { + format: "reversi-registry"; + version: 1; + tools: Record; +} + +export class RegistryError extends Error { + constructor(message: string) { + super(message); + this.name = "RegistryError"; + } +} + +const fileFor = (dir: string, tool: string, v: number) => join(dir, "tools", encodeURIComponent(tool), `v${v}.json`); +const signedText = (tool: string, version: number, checksum: string) => `reversi-manifest:${tool}@${version}:${checksum}`; +const checksumOf = (m: unknown) => sha256Hex(canonicalize(m)); + +export interface RegistryOptions { + now?: () => number; + /** If set, every version must carry a valid signature from this key. */ + trusted_public_pem?: string; +} + +export class ManifestRegistry { + private readonly now: () => number; + constructor( + readonly dir: string, + private readonly opts: RegistryOptions = {}, + ) { + this.now = opts.now ?? Date.now; + } + + static isRegistry(dir: string): boolean { + try { + return existsSync(join(dir, "index.json")) && (JSON.parse(readFileSync(join(dir, "index.json"), "utf8")) as IndexFile).format === "reversi-registry"; + } catch { + return false; + } + } + + private readIndex(): IndexFile { + const p = join(this.dir, "index.json"); + if (!existsSync(p)) return { format: "reversi-registry", version: 1, tools: {} }; + return JSON.parse(readFileSync(p, "utf8")) as IndexFile; + } + + private writeIndex(ix: IndexFile): void { + mkdirSync(this.dir, { recursive: true }); + const tmp = join(this.dir, `index.json.${process.pid}.tmp`); + writeFileSync(tmp, JSON.stringify(ix, null, 2)); + renameSync(tmp, join(this.dir, "index.json")); + } + + tools(): string[] { + return Object.keys(this.readIndex().tools).sort(); + } + + history(tool: string): RegistryVersion[] { + const t = this.readIndex().tools[tool]; + if (!t) throw new RegistryError(`no tool \`${tool}\` in the registry`); + return t.versions; + } + + get(tool: string, version: number): ToolManifest { + const v = this.history(tool).find((x) => x.version === version); + if (!v) throw new RegistryError(`\`${tool}\` has no version ${version}`); + const raw = JSON.parse(readFileSync(fileFor(this.dir, tool, version), "utf8")) as unknown; + this.check(tool, v, raw); + const r = validateManifest(raw); + if (!r.ok) throw new RegistryError(`\`${tool}\` v${version} is not a valid manifest: ${r.errors.join("; ")}`); + return r.manifest; + } + + /** Verify a version's checksum, and its signature when a trusted key is configured. */ + private check(tool: string, v: RegistryVersion, raw: unknown): void { + if (checksumOf(raw) !== v.checksum) throw new RegistryError(`\`${tool}\` v${v.version} was changed outside the registry: its checksum does not match the index`); + if (this.opts.trusted_public_pem) { + let ok = false; + try { + ok = !!v.signature && verify(null, Buffer.from(signedText(tool, v.version, v.checksum)), createPublicKey(this.opts.trusted_public_pem), Buffer.from(v.signature, "base64")); + } catch { + ok = false; + } + if (!ok) throw new RegistryError(`\`${tool}\` v${v.version} is not signed by the trusted key`); + } + } + + /** Add a new version. It is a draft unless `status` says otherwise. */ + publish(manifest: unknown, o: { by: string; note?: string; status?: ManifestStatus; key?: SigningKey }): { tool: string; version: number; checksum: string } { + const first = validateManifest(manifest); + if (!first.ok) throw new RegistryError(first.errors.join("; ")); + const tool = first.manifest.tool; + const ix = this.readIndex(); + const versions = (ix.tools[tool] ??= { versions: [] }).versions; + const version = (versions.at(-1)?.version ?? 0) + 1; + const status = o.status ?? "draft"; + const body = { ...(manifest as Record), version, status }; + const again = validateManifest(body); + if (!again.ok) throw new RegistryError(again.errors.join("; ")); + const checksum = checksumOf(body); + const rec: RegistryVersion = { version, checksum, status, created_ms: this.now(), by: o.by, ...(o.note ? { note: o.note } : {}) }; + if (o.key) { + rec.signature = sign(null, Buffer.from(signedText(tool, version, checksum)), createPrivateKey(o.key.private_pem)).toString("base64"); + rec.key_id = keyId(o.key.public_pem); + } + mkdirSync(join(this.dir, "tools", encodeURIComponent(tool)), { recursive: true }); + writeFileSync(fileFor(this.dir, tool, version), JSON.stringify(body, null, 2)); + versions.push(rec); + this.writeIndex(ix); + return { tool, version, checksum }; + } + + /** + * Promote a draft to approved. It republishes the manifest as a new `approved` version, so the approval is itself a + * versioned, signed event, and the approver is recorded. With `require_other` the approver may not be the author. + */ + approve(tool: string, o: { by: string; version?: number; note?: string; key?: SigningKey; require_other?: boolean }): { tool: string; version: number } { + const hist = this.history(tool); + const target = hist.find((v) => v.version === (o.version ?? hist.at(-1)?.version)); + if (!target) throw new RegistryError(`\`${tool}\` has no version ${o.version}`); + if (target.status !== "draft") throw new RegistryError(`\`${tool}\` v${target.version} is ${target.status}, not a draft`); + if (o.require_other && target.by === o.by) throw new RegistryError("the author of a draft cannot approve it: ask someone else"); + const m = this.get(tool, target.version); + const out = this.publish(authorForm(m), { by: target.by, note: o.note ?? `approved from v${target.version}`, status: "approved", key: o.key }); + const ix = this.readIndex(); + const rec = (ix.tools[tool]?.versions ?? []).find((v) => v.version === out.version); + if (rec) { + rec.approved_by = o.by; + this.writeIndex(ix); + } + return { tool, version: out.version }; + } + + /** Mark a tool deprecated (new calls are noted; rollback keeps working), optionally pointing at its replacement. */ + deprecate(tool: string, o: { by: string; replaced_by?: string; note?: string; key?: SigningKey }): { tool: string; version: number } { + const cur = this.current(tool); + if (!cur) throw new RegistryError(`\`${tool}\` has no approved version to deprecate`); + const out = this.publish({ ...authorForm(cur), status: "deprecated", ...(o.replaced_by ? { replaced_by: o.replaced_by } : {}) }, { by: o.by, note: o.note ?? "deprecated", status: "deprecated", key: o.key }); + return { tool, version: out.version }; + } + + /** Publish an old version again as the newest one (a rollback of the manifest, with history intact). */ + rollbackTo(tool: string, version: number, o: { by: string; key?: SigningKey; note?: string }): { tool: string; version: number } { + const m = this.get(tool, version); + const out = this.publish(authorForm(m), { by: o.by, note: o.note ?? `restored v${version}`, status: m.status === "draft" ? "draft" : "approved", key: o.key }); + return { tool, version: out.version }; + } + + /** The newest approved or deprecated version of a tool (drafts are not enforceable). */ + current(tool: string, includeDrafts = false): ToolManifest | undefined { + const t = this.readIndex().tools[tool]; + const v = [...(t?.versions ?? [])].reverse().find((x) => includeDrafts || x.status !== "draft"); + return v ? this.get(tool, v.version) : undefined; + } + + diff(tool: string, a: number, b: number): string { + const pretty = (v: number) => JSON.stringify(this.get(tool, v), null, 2); + return renderDiff(pretty(a), pretty(b), 3); + } + + /** Everything enforceable, checksum- and signature-verified. */ + load(o: { drafts?: boolean } = {}): Registry { + const reg = new Registry(); + for (const t of this.tools()) { + const m = this.current(t, o.drafts); + if (m) reg.register(m); + } + reg.assertValid(); + return reg; + } + + /** Check every version of every tool; returns problems instead of throwing at the first. */ + verifyAll(): string[] { + const problems: string[] = []; + for (const t of this.tools()) { + let expected = 1; + for (const v of this.history(t)) { + if (v.version !== expected++) problems.push(`\`${t}\`: version numbers are not continuous at v${v.version}`); + try { + this.get(t, v.version); + } catch (e) { + problems.push(e instanceof Error ? e.message : String(e)); + } + } + } + return problems; + } +} + +/** Load a registry served as static files: `/index.json` and `/tools//v.json`. */ +export async function loadRemoteRegistry(url: string, o: { trusted_public_pem?: string; egress?: EgressConfig; fetch?: typeof fetch; headers?: Record } = {}): Promise { + const f = guardedFetch(o.egress, o.fetch); + const base = url.replace(/\/$/, ""); + const get = async (path: string) => { + const res = await f(`${base}/${path}`, { headers: o.headers, signal: AbortSignal.timeout(15_000) }); + if (!res.ok) throw new RegistryError(`${base}/${path} answered ${res.status}`); + return (await res.json()) as unknown; + }; + const ix = (await get("index.json")) as IndexFile; + if (ix?.format !== "reversi-registry") throw new RegistryError(`${base}/index.json is not a Reversi registry`); + const reg = new Registry(); + for (const [tool, t] of Object.entries(ix.tools)) { + const v = [...t.versions].reverse().find((x) => x.status !== "draft"); + if (!v) continue; + const raw = await get(`tools/${encodeURIComponent(tool)}/v${v.version}.json`); + if (checksumOf(raw) !== v.checksum) throw new RegistryError(`\`${tool}\` v${v.version} does not match its checksum in the index`); + if (o.trusted_public_pem) { + let ok = false; + try { + ok = !!v.signature && verify(null, Buffer.from(signedText(tool, v.version, v.checksum)), createPublicKey(o.trusted_public_pem), Buffer.from(v.signature, "base64")); + } catch { + ok = false; + } + if (!ok) throw new RegistryError(`\`${tool}\` v${v.version} is not signed by the trusted key`); + } + reg.register(raw); + } + reg.assertValid(); + return reg; +} diff --git a/packages/reversi/src/retention.ts b/packages/reversi/src/retention.ts new file mode 100644 index 0000000..7407717 --- /dev/null +++ b/packages/reversi/src/retention.ts @@ -0,0 +1,185 @@ +/** + * Retention and legal hold. + * + * The ledger is append-only and holds only hashes, reasons and details, so it is never purged. What retention removes + * is the working data around a run: its run state, its approval requests and its captured (redacted) arguments. A legal + * hold stops that for the runs it covers, and a manifest can require an evidence bundle to be archived first. + */ +import { randomUUID } from "node:crypto"; +import { mkdirSync, writeFileSync } from "node:fs"; +import { join } from "node:path"; +import type { Approvals } from "./approvals.js"; +import type { CaptureStore } from "./capture.js"; +import type { Reversi } from "./engine.js"; +import { createBundle, type SigningKey } from "./evidence.js"; +import { globAny } from "./glob.js"; +import type { KvStore } from "./kv.js"; +import type { RunState } from "./store.js"; + +export interface HoldScope { + run_id?: string; + tenant?: string; + actor?: string; + tool?: string; +} +export interface Hold { + id: string; + scope: HoldScope; + reason: string; + by: string; + ts_ms: number; + released_ms?: number; +} + +const NS = "holds"; + +export class LegalHolds { + constructor( + private readonly kv: KvStore, + private readonly now: () => number = Date.now, + ) {} + + add(scope: HoldScope, reason: string, by: string): Hold { + if (!Object.values(scope).some(Boolean)) throw new Error("a legal hold needs a scope: a run, tenant, actor or tool"); + if (reason.trim().length < 5) throw new Error("a legal hold needs a reason (case or matter reference)"); + const h: Hold = { id: `hold_${randomUUID()}`, scope, reason, by, ts_ms: this.now() }; + this.kv.put(NS, h.id, h); + return h; + } + + release(id: string): boolean { + const e = this.kv.get(NS, id); + if (!e || e.value.released_ms) return false; + this.kv.put(NS, id, { ...e.value, released_ms: this.now() }); + return true; + } + + list(opts: { all?: boolean } = {}): Hold[] { + return this.kv.list(NS).map((e) => e.value).filter((h) => opts.all || !h.released_ms); + } + + /** The hold that covers this run, if any. */ + coverOf(run: RunState): Hold | undefined { + return this.list().find((h) => { + const s = h.scope; + return (!s.run_id || s.run_id === run.run_id) && (!s.tenant || s.tenant === run.tenant) && (!s.actor || s.actor === run.actor) && (!s.tool || run.steps.some((x) => x.tool === s.tool)); + }); + } +} + +export interface RetentionConfig { + default_days: number; + rules: Array<{ match: { tool?: string; tenant?: string; environment?: string; data_class?: string }; days: number }>; +} + +export function parseRetention(raw: unknown, errors: string[]): RetentionConfig | undefined { + if (raw === undefined) return undefined; + if (typeof raw !== "object" || raw === null || Array.isArray(raw)) { + errors.push("`retention` must be an object"); + return undefined; + } + const r = raw as Record; + for (const k of Object.keys(r)) if (!["default_days", "rules"].includes(k)) errors.push(`retention: unknown field \`${k}\``); + if (typeof r.default_days !== "number" || !Number.isInteger(r.default_days) || r.default_days < 1) errors.push("retention.default_days must be a positive integer"); + const rules = Array.isArray(r.rules) ? r.rules : r.rules === undefined ? [] : (errors.push("retention.rules must be a list"), []); + rules.forEach((x: unknown, i: number) => { + const o = x as { match?: Record; days?: unknown }; + if (!o || typeof o.match !== "object" || o.match === null) errors.push(`retention.rules[${i}]: match is required`); + else for (const k of Object.keys(o.match)) if (!["tool", "tenant", "environment", "data_class"].includes(k)) errors.push(`retention.rules[${i}]: unknown match field \`${k}\``); + if (!Number.isInteger(o?.days) || (o.days as number) < 1) errors.push(`retention.rules[${i}]: days must be a positive integer`); + }); + return { default_days: r.default_days as number, rules: rules as RetentionConfig["rules"] }; +} + +/** How long to keep a run: the longest of its default, the matching rules and each tool's manifest `audit.retention_days`. */ +export function retentionDays(state: RunState, rv: Reversi, cfg: RetentionConfig): number { + let days = cfg.default_days; + for (const s of state.steps) { + const m = rv.registry.get(s.tool); + days = Math.max(days, m?.audit?.retention_days ?? 0); + for (const r of cfg.rules) { + const x = r.match; + if ((!x.tool || globAny(x.tool, s.tool)) && (!x.tenant || x.tenant === state.tenant) && (!x.environment || x.environment === rv.environment) && (!x.data_class || x.data_class === m?.risk?.data_class)) days = Math.max(days, r.days); + } + } + return days; +} + +export interface RetentionItem { + run_id: string; + age_days: number; + retention_days: number; + held?: Hold; + due: boolean; + needs_export: boolean; +} + +export function planRetention(rv: Reversi, cfg: RetentionConfig, holds: LegalHolds | undefined, now = Date.now()): RetentionItem[] { + return rv.runIds().flatMap((id) => { + const st = rv.openRun(id).state(); + const age = (now - st.created_ms) / 86_400_000; + const days = retentionDays(st, rv, cfg); + return [ + { + run_id: id, + age_days: Math.floor(age), + retention_days: days, + held: holds?.coverOf(st), + due: age >= days, + needs_export: st.steps.some((s) => rv.registry.get(s.tool)?.audit?.evidence_export === "required"), + }, + ]; + }); +} + +export interface RetentionReport { + dry_run: boolean; + purged: string[]; + skipped: Array<{ run_id: string; reason: string }>; + exported: Array<{ run_id: string; file: string }>; +} + +export function applyRetention( + rv: Reversi, + cfg: RetentionConfig, + o: { holds?: LegalHolds; approvals?: Approvals; capture?: CaptureStore; dry_run?: boolean; archive_dir?: string; key?: SigningKey; now?: number; actor?: string }, +): RetentionReport { + const now = o.now ?? Date.now(); + const rep: RetentionReport = { dry_run: o.dry_run === true, purged: [], skipped: [], exported: [] }; + for (const item of planRetention(rv, cfg, o.holds, now)) { + if (!item.due) continue; + if (item.held) { + rep.skipped.push({ run_id: item.run_id, reason: `legal hold ${item.held.id}: ${item.held.reason}` }); + continue; + } + const needExport = item.needs_export || (o.archive_dir !== undefined && o.key !== undefined); + if (item.needs_export && (!o.archive_dir || !o.key)) { + rep.skipped.push({ run_id: item.run_id, reason: "a manifest requires an evidence bundle before deletion; give an archive directory and a signing key" }); + continue; + } + let file: string | undefined; + if (needExport && o.archive_dir && o.key && !o.dry_run) { + mkdirSync(o.archive_dir, { recursive: true }); + const bundle = createBundle({ records: rv.ledger.forRun(item.run_id), ledger_head: rv.ledger.head(), run_id: item.run_id, key: o.key, now }); + file = join(o.archive_dir, `${item.run_id}.json`); + writeFileSync(file, JSON.stringify(bundle, null, 2)); + rep.exported.push({ run_id: item.run_id, file }); + } + if (!o.dry_run) { + rv.runs.delete(item.run_id); + for (const a of o.approvals?.list({ run_id: item.run_id }) ?? []) o.approvals?.store.delete(a.id); + o.capture?.purge({ run_id: item.run_id }); + rv.ledger.append({ + ts_ms: now, + run_id: item.run_id, + actor: o.actor ?? "retention", + tool: "-", + reason: `retention: older than ${item.retention_days} days`, + event: "purged", + detail: `run state, approvals and captured data deleted${file ? `; evidence bundle archived at ${file}` : ""}`, + }); + } + rep.purged.push(item.run_id); + } + return rep; +} diff --git a/packages/reversi/src/risk.ts b/packages/reversi/src/risk.ts new file mode 100644 index 0000000..2f84328 --- /dev/null +++ b/packages/reversi/src/risk.ts @@ -0,0 +1,111 @@ +/** + * Risk scoring: a transparent 0..100 number from the tool's effect class, data class, blast radius, environment, + * the call's amount and the actor. Every factor is listed, so a reviewer can see why a call scored what it did. + */ +import { resolveRef } from "./manifest.js"; + +export interface RiskConfig { + weights?: { + effect?: Record; + blast_radius?: Record; + data_class?: Record; + environment?: Record; + /** Points for a tool with no manifest. */ + unclassified?: number; + }; + /** Points added when a numeric argument is over a band. The highest matching band applies. */ + amount?: { path: string; bands: Array<{ over: number; points: number }> }; + /** Extra points for specific actors (a new or untrusted agent). */ + actors?: Record; + /** At or above `approve` a call needs approval; at or above `deny` it is refused. */ + thresholds?: { approve?: number; deny?: number }; +} + +export const DEFAULT_WEIGHTS = { + effect: { read_only: 0, idempotent: 5, reversible: 15, compensable: 30, irreversible: 60 }, + blast_radius: { low: 0, medium: 10, high: 20, critical: 35 }, + data_class: { public: 0, internal: 5, pii: 15, financial: 20, phi: 25, secret: 25 }, + environment: { prod: 10, production: 10 }, + unclassified: 40, +} as const; + +export interface RiskInput { + effect: string | null; + blast_radius?: string; + data_class?: string; + environment: string; + actor: string; + args: unknown; +} + +export interface RiskResult { + score: number; + factors: string[]; +} + +export function scoreRisk(cfg: RiskConfig | undefined, i: RiskInput): RiskResult { + const w = { ...DEFAULT_WEIGHTS, ...(cfg?.weights ?? {}) } as Record | number>; + const factors: string[] = []; + let score = 0; + const add = (points: number | undefined, label: string) => { + if (!points) return; + score += points; + factors.push(`${label} ${points > 0 ? "+" : ""}${points}`); + }; + if (i.effect === null) add(w.unclassified as number, "no manifest"); + else add((w.effect as Record)[i.effect], `effect ${i.effect}`); + if (i.blast_radius) add((w.blast_radius as Record)[i.blast_radius], `blast radius ${i.blast_radius}`); + if (i.data_class) add((w.data_class as Record)[i.data_class], `data class ${i.data_class}`); + add((w.environment as Record)[i.environment], `environment ${i.environment}`); + if (cfg?.amount) { + const v = resolveRef(cfg.amount.path, "args", i.args); + const n = typeof v === "number" ? v : typeof v === "string" && v.trim() !== "" ? Number(v) : NaN; + if (Number.isFinite(n)) { + const band = [...cfg.amount.bands].sort((a, b) => b.over - a.over).find((b) => n > b.over); + if (band) add(band.points, `amount over ${band.over}`); + } + } + add(cfg?.actors?.[i.actor], `actor ${i.actor}`); + return { score: Math.max(0, Math.min(100, score)), factors }; +} + +export function parseRisk(raw: unknown, errors: string[]): RiskConfig | undefined { + if (raw === undefined) return undefined; + if (typeof raw !== "object" || raw === null || Array.isArray(raw)) { + errors.push("`risk` must be an object"); + return undefined; + } + const r = raw as Record; + for (const k of Object.keys(r)) if (!["weights", "amount", "actors", "thresholds"].includes(k)) errors.push(`risk: unknown field \`${k}\``); + const numMap = (v: unknown, where: string) => { + if (v === undefined) return; + if (typeof v !== "object" || v === null || Object.values(v).some((x) => typeof x !== "number" || !Number.isFinite(x))) errors.push(`${where} must map names to numbers`); + }; + if (r.weights !== undefined) { + const w = r.weights as Record; + if (typeof w !== "object" || w === null) errors.push("risk.weights must be an object"); + else { + for (const k of Object.keys(w)) { + if (!["effect", "blast_radius", "data_class", "environment", "unclassified"].includes(k)) errors.push(`risk.weights: unknown field \`${k}\``); + else if (k === "unclassified") { + if (typeof w[k] !== "number") errors.push("risk.weights.unclassified must be a number"); + } else numMap(w[k], `risk.weights.${k}`); + } + } + } + numMap(r.actors, "risk.actors"); + if (r.amount !== undefined) { + const a = r.amount as Record; + if (typeof a?.path !== "string" || !/^\$args(\.[A-Za-z0-9_-]+)+$/.test(a.path)) errors.push("risk.amount.path must look like `$args.amount`"); + if (!Array.isArray(a?.bands) || a.bands.some((b: unknown) => typeof (b as { over?: unknown }).over !== "number" || typeof (b as { points?: unknown }).points !== "number")) errors.push("risk.amount.bands must be a list of { over, points }"); + } + if (r.thresholds !== undefined) { + const t = r.thresholds as Record; + for (const k of Object.keys(t ?? {})) { + if (!["approve", "deny"].includes(k)) errors.push(`risk.thresholds: unknown field \`${k}\``); + else if (typeof t[k] !== "number" || (t[k] as number) < 0 || (t[k] as number) > 100) errors.push(`risk.thresholds.${k} must be 0..100`); + } + if (typeof t?.approve === "number" && typeof t?.deny === "number" && t.approve > t.deny) errors.push("risk.thresholds.approve must not be above deny"); + } + return r as RiskConfig; +} diff --git a/packages/reversi/src/schedule.ts b/packages/reversi/src/schedule.ts new file mode 100644 index 0000000..c2cb2ba --- /dev/null +++ b/packages/reversi/src/schedule.ts @@ -0,0 +1,82 @@ +/** Time-window conditions for policy rules (business hours, maintenance windows) and freeze periods. */ + +const DAYS = ["sun", "mon", "tue", "wed", "thu", "fri", "sat"] as const; +export type Day = (typeof DAYS)[number]; + +export interface Window { + /** Days of the week the window covers. Default: every day. */ + days?: Day[]; + /** `HH:MM-HH:MM` in `timezone`; may wrap midnight (`22:00-06:00`). Default: all day. */ + between?: string; + /** IANA time zone, default UTC. */ + timezone?: string; + /** Invert: the window matches when the time is *outside* the described span. */ + outside?: boolean; +} + +export interface Freeze { + id: string; + /** ISO timestamps. */ + from: string; + to: string; + reason: string; + /** Which calls are frozen. Default: every call that is not read_only. */ + match?: { tool?: string | string[]; effect?: string | string[]; environment?: string | string[] }; +} + +const HHMM = /^([01]?\d|2[0-3]):([0-5]\d)-([01]?\d|2[0-3]):([0-5]\d)$/; + +export function parseWindow(raw: unknown, where: string, errors: string[]): Window | undefined { + if (raw === undefined) return undefined; + if (typeof raw !== "object" || raw === null || Array.isArray(raw)) { + errors.push(`${where}: window must be an object`); + return undefined; + } + const w = raw as Record; + for (const k of Object.keys(w)) if (!["days", "between", "timezone", "outside"].includes(k)) errors.push(`${where}: unknown window field \`${k}\``); + if (w.days !== undefined && (!Array.isArray(w.days) || w.days.some((d) => !DAYS.includes(d as Day)))) errors.push(`${where}: window.days must be a list of ${DAYS.join(", ")}`); + if (w.between !== undefined && (typeof w.between !== "string" || !HHMM.test(w.between))) errors.push(`${where}: window.between must look like 09:00-17:00`); + if (w.timezone !== undefined) { + try { + new Intl.DateTimeFormat("en-US", { timeZone: String(w.timezone) }); + } catch { + errors.push(`${where}: window.timezone is not a valid IANA time zone`); + } + } + if (w.outside !== undefined && typeof w.outside !== "boolean") errors.push(`${where}: window.outside must be a boolean`); + return w as Window; +} + +/** True when `nowMs` falls in the window (or outside it, for `outside: true`). */ +export function inWindow(w: Window, nowMs: number): boolean { + const parts = new Intl.DateTimeFormat("en-US", { timeZone: w.timezone ?? "UTC", weekday: "short", hour: "2-digit", minute: "2-digit", hour12: false }).formatToParts(new Date(nowMs)); + const get = (t: string) => parts.find((p) => p.type === t)?.value ?? ""; + const day = get("weekday").slice(0, 3).toLowerCase() as Day; + const minutes = (Number(get("hour")) % 24) * 60 + Number(get("minute")); + let inside = !w.days || w.days.includes(day); + if (inside && w.between) { + const m = HHMM.exec(w.between) as RegExpExecArray; + const from = Number(m[1]) * 60 + Number(m[2]); + const to = Number(m[3]) * 60 + Number(m[4]); + inside = from <= to ? minutes >= from && minutes < to : minutes >= from || minutes < to; + } + return w.outside ? !inside : inside; +} + +export function parseFreeze(raw: unknown, where: string, errors: string[]): Freeze | undefined { + if (typeof raw !== "object" || raw === null || Array.isArray(raw)) { + errors.push(`${where}: must be an object`); + return undefined; + } + const f = raw as Record; + for (const k of Object.keys(f)) if (!["id", "from", "to", "reason", "match"].includes(k)) errors.push(`${where}: unknown field \`${k}\``); + if (typeof f.id !== "string" || !f.id) errors.push(`${where}: id is required`); + const from = Date.parse(String(f.from)); + const to = Date.parse(String(f.to)); + if (!Number.isFinite(from) || !Number.isFinite(to)) errors.push(`${where}: from and to must be ISO timestamps`); + else if (to <= from) errors.push(`${where}: to must be after from`); + if (typeof f.reason !== "string" || !f.reason) errors.push(`${where}: a freeze needs a reason`); + return f as unknown as Freeze; +} + +export const frozenAt = (f: Freeze, nowMs: number): boolean => nowMs >= Date.parse(f.from) && nowMs < Date.parse(f.to); diff --git a/packages/reversi/src/simulate.ts b/packages/reversi/src/simulate.ts new file mode 100644 index 0000000..7f82473 --- /dev/null +++ b/packages/reversi/src/simulate.ts @@ -0,0 +1,393 @@ +/** + * Look before you leap, and look back: run calls through a throwaway engine (in-memory ledger and run state, the real + * manifests and policy, nothing executed anywhere) to answer "what would this policy have done to last week's calls", + * "what if this one argument had been different", "which of these plans is safest" and "what would approving this do". + */ +import { randomUUID } from "node:crypto"; +import type { CaptureRecord } from "./capture.js"; +import type { ContentChecker } from "./content.js"; +import { Reversi, type RollbackPlan } from "./engine.js"; +import { Ledger } from "./ledger.js"; +import type { Registry, ToolManifest } from "./manifest.js"; +import { Policy, type PolicyDoc } from "./policy.js"; +import { SealedScope } from "./scope.js"; + +export interface SandboxContext { + registry: Registry; + policy?: PolicyDoc; + content?: ContentChecker; + environment?: string; + /** Pretend the policy's clock is here (for time windows and freezes). */ + now?: () => number; +} + +export type Verdict = "allow" | "needs_approval" | "block"; + +export interface SandboxCall { + run?: string; + tool: string; + args?: unknown; + actor?: string; + tenant?: string; + /** What the tool returned (used for compensation arguments). If omitted a result is synthesized from the manifest. */ + result?: unknown; + /** The recorded outcome, when replaying. Default succeeded. */ + outcome?: "succeeded" | "failed" | "uncertain"; +} + +export interface SandboxStep { + run: string; + index: number; + tool: string; + verdict: Verdict; + reasons: string[]; + risk?: number; + /** The call ran in the sandbox (allowed or would be blocked in non-enforce modes). */ + executed: boolean; + simulated_result: boolean; +} + +const APPROVAL = "approval required"; + +/** A result containing every `$result.*` path the manifest's compensation and verification refer to. */ +export function synthesizeResult(m: ToolManifest | undefined): unknown { + const refs = [...Object.values(m?.compensation?.argument_map ?? {}), ...Object.values(m?.compensation?.verification?.arguments ?? {})].filter((r) => r.startsWith("$result.")); + const out: Record = {}; + for (const r of refs) { + const segs = r.split(".").slice(1); + let cur = out; + segs.forEach((seg, i) => { + if (i === segs.length - 1) cur[seg] = ``; + else cur = (cur[seg] ??= {}) as Record; + }); + } + return out; +} + +export class Sandbox { + readonly reversi: Reversi; + private runs = new Map>(); + private index = new Map(); + private assume: boolean; + constructor( + private readonly ctx: SandboxContext, + opts: { assume_approved?: boolean } = {}, + ) { + this.assume = opts.assume_approved === true; + const registry = ctx.registry; + this.reversi = new Reversi({ + registry, + ledger: new Ledger(), + policy: ctx.policy ? new Policy(ctx.policy, ctx.now) : undefined, + content: ctx.content, + environment: ctx.environment, + now: ctx.now, + approver: () => (this.assume ? true : { approved: false, message: `${APPROVAL}: simulated` }), + }); + } + + private runFor(id: string, actor: string, tenant: string | undefined) { + let run = this.runs.get(id); + if (!run) { + const perms = [...new Set(this.ctx.registry.list().map((m) => m.permission))]; + run = this.reversi.startRun({ run_id: `sim_${randomUUID().slice(0, 8)}_${id}`.slice(0, 100).replace(/[^A-Za-z0-9._-]/g, "_"), actor, scope: SealedScope.seal(perms), tenant }); + this.runs.set(id, run); + } + return run; + } + + async call(c: SandboxCall): Promise { + const id = c.run ?? "plan"; + const run = this.runFor(id, c.actor ?? "simulation", c.tenant); + const manifest = this.ctx.registry.get(c.tool); + const i = this.index.get(id) ?? 0; + this.index.set(id, i + 1); + const b = await run.begin({ tool: c.tool, args: c.args, reason: "simulation" }); + if (b.decision === "block") { + const approval = b.reasons.some((r) => r.startsWith(APPROVAL)); + return { run: id, index: i, tool: c.tool, verdict: approval ? "needs_approval" : "block", reasons: b.reasons, executed: false, simulated_result: false }; + } + const simulated = c.result === undefined; + const result = c.result ?? synthesizeResult(manifest); + const outcome = c.outcome ?? "succeeded"; + run.complete(b.step, outcome === "succeeded" ? { status: "succeeded", result } : { status: outcome, error: "simulated" }); + const risk = run.state().steps.find((s) => s.step === b.step)?.risk_score; + return { run: id, index: i, tool: c.tool, verdict: "allow", reasons: b.decision === "would_block" ? b.reasons : [], risk, executed: true, simulated_result: simulated }; + } + + /** Run a prerequisite as if it had already been approved and had succeeded, without changing how later calls are judged. */ + async seed(c: SandboxCall): Promise { + const before = this.assume; + this.assume = true; + try { + return await this.call(c); + } finally { + this.assume = before; + } + } + + plan(run = "plan"): RollbackPlan | undefined { + return this.runs.get(run)?.plan(); + } +} + +// ---- policy simulation against captured history ---- + +export interface Change { + before: Verdict; + after: Verdict; + run: string; + step: number | null; + tool: string; + reasons: string[]; +} + +export interface SimulationReport { + calls: number; + runs: number; + unchanged: number; + newly_blocked: Change[]; + newly_needs_approval: Change[]; + newly_allowed: Change[]; + /** Calls the history had blocked for a reason other than policy (scope, kill switch) and that still are. */ + notes: string[]; +} + +function historical(r: CaptureRecord): Verdict { + if (r.decision !== "block") return "allow"; + return r.reasons.some((x) => x.startsWith(APPROVAL)) ? "needs_approval" : "block"; +} + +/** Replay captured calls through a candidate policy and say what would have been decided differently. */ +export async function simulatePolicy(records: CaptureRecord[], ctx: SandboxContext): Promise { + const sandbox = new Sandbox(ctx); + const rep: SimulationReport = { calls: 0, runs: new Set(records.map((r) => r.run_id)).size, unchanged: 0, newly_blocked: [], newly_needs_approval: [], newly_allowed: [], notes: [] }; + for (const r of [...records].sort((a, b) => a.ts_ms - b.ts_ms)) { + rep.calls++; + const step = await sandbox.call({ run: r.run_id, tool: r.tool, args: r.args, actor: r.actor, tenant: r.tenant, result: r.result, outcome: r.outcome }); + const before = historical(r); + const after = step.verdict; + if (before === after) { + rep.unchanged++; + continue; + } + const change: Change = { before, after, run: r.run_id, step: r.step, tool: r.tool, reasons: step.reasons }; + if (after === "block") rep.newly_blocked.push(change); + else if (after === "needs_approval") rep.newly_needs_approval.push(change); + else rep.newly_allowed.push(change); + } + const redacted = records.some((r) => JSON.stringify(r.args).includes("[REDACTED]") || JSON.stringify(r.args).includes("[PII:")); + if (redacted) rep.notes.push("some captured arguments were redacted, so rules that look inside those values may behave differently than they did live"); + return rep; +} + +// ---- replay one run with a change ---- + +export interface ArgChange { + /** Step number, or omit to apply to every call of `tool`. */ + step?: number; + tool?: string; + /** `$args.amount` or `amount`. */ + path: string; + value: unknown; +} + +function applyChange(args: unknown, path: string, value: unknown): unknown { + const clone = JSON.parse(JSON.stringify(args ?? {})) as Record; + const segs = path.replace(/^\$args\./, "").split("."); + let cur = clone as Record; + segs.forEach((seg, i) => { + if (i === segs.length - 1) cur[seg] = value; + else { + if (typeof cur[seg] !== "object" || cur[seg] === null) cur[seg] = {}; + cur = cur[seg] as Record; + } + }); + return clone; +} + +export interface ReplayReport { + baseline: SandboxStep[]; + variant: SandboxStep[]; + differences: Array<{ index: number; tool: string; before: Verdict; after: Verdict; reasons: string[] }>; + baseline_plan?: RollbackPlan; + variant_plan?: RollbackPlan; +} + +/** Replay one captured run twice, as recorded and with the changes, in the sandbox. Nothing touches production. */ +export async function replayRun(records: CaptureRecord[], ctx: SandboxContext, changes: ArgChange[]): Promise { + const calls = [...records].sort((a, b) => a.ts_ms - b.ts_ms); + const go = async (mutate: boolean) => { + const sb = new Sandbox(ctx, { assume_approved: true }); + const steps: SandboxStep[] = []; + for (const [i, r] of calls.entries()) { + let args = r.args; + if (mutate) for (const c of changes) if ((c.step === undefined || c.step === (r.step ?? i)) && (c.tool === undefined || c.tool === r.tool)) args = applyChange(args, c.path, c.value); + steps.push(await sb.call({ run: "replay", tool: r.tool, args, actor: r.actor, tenant: r.tenant, result: r.result, outcome: r.outcome })); + } + return { steps, plan: sb.plan("replay") }; + }; + // Judge decisions without assuming approval (so "needs approval" shows), then compute the plans on the approved path. + const judge = async (mutate: boolean) => { + const sb = new Sandbox(ctx); + const steps: SandboxStep[] = []; + for (const [i, r] of calls.entries()) { + let args = r.args; + if (mutate) for (const c of changes) if ((c.step === undefined || c.step === (r.step ?? i)) && (c.tool === undefined || c.tool === r.tool)) args = applyChange(args, c.path, c.value); + steps.push(await sb.call({ run: "replay", tool: r.tool, args, actor: r.actor, tenant: r.tenant, result: r.result, outcome: r.outcome })); + } + return steps; + }; + const [bj, vj, bp, vp] = await Promise.all([judge(false), judge(true), go(false), go(true)]); + return { + baseline: bj, + variant: vj, + differences: vj.flatMap((v, i) => (v.verdict !== bj[i]?.verdict ? [{ index: i, tool: v.tool, before: (bj[i] as SandboxStep).verdict, after: v.verdict, reasons: v.reasons }] : [])), + baseline_plan: bp.plan, + variant_plan: vp.plan, + }; +} + +// ---- speculative execution: compare alternative plans ---- + +export interface PlanInput { + name: string; + calls: Array<{ tool: string; args?: unknown; result?: unknown }>; +} + +export interface PlanReport { + name: string; + steps: SandboxStep[]; + /** Index of the first call that would be refused, if any. */ + blocked_at?: number; + approvals_needed: number; + max_risk: number; + total_risk: number; + irreversible: string[]; + residue: string[]; + fully_reversible: boolean; + rollback: RollbackPlan | undefined; + /** Lower is safer. Blocked plans rank last. */ + rank: number; + notes: string[]; +} + +export interface SpeculationReport { + plans: PlanReport[]; + /** Name of the safest plan that can run to the end, if any. */ + recommended?: string; +} + +export async function speculate(plans: PlanInput[], ctx: SandboxContext, o: { actor?: string; tenant?: string } = {}): Promise { + const reports: PlanReport[] = []; + for (const p of plans) { + const sb = new Sandbox(ctx, { assume_approved: true }); + const judge = new Sandbox(ctx); + const steps: SandboxStep[] = []; + let blocked_at: number | undefined; + for (const [i, c] of p.calls.entries()) { + // What would be decided if nobody had approved yet? + const asked = await judge.call({ run: "p", tool: c.tool, args: c.args, actor: o.actor, tenant: o.tenant, result: c.result }); + const step = await sb.call({ run: "p", tool: c.tool, args: c.args, actor: o.actor, tenant: o.tenant, result: c.result }); + steps.push(asked.verdict === "needs_approval" ? { ...step, verdict: "needs_approval", reasons: asked.reasons } : step); + if (step.verdict === "block") { + blocked_at = i; + break; + } + } + const rollback = sb.plan("p"); + const executed = steps.filter((s) => s.executed); + const manifests = executed.map((s) => ctx.registry.get(s.tool)); + const irreversible = executed.filter((_, i) => manifests[i]?.effect === "irreversible").map((s) => s.tool); + const residue = [...new Set((rollback?.actions ?? []).flatMap((a) => a.residue ?? []))]; + const approvals = steps.filter((s) => s.verdict === "needs_approval").length; + const risks = executed.map((s) => s.risk ?? 0); + const max_risk = risks.length ? Math.max(...risks) : 0; + const notes: string[] = []; + if (steps.some((s) => s.simulated_result)) notes.push("some results were synthesized from the manifests, so the rollback outlook assumes compensation arguments are available"); + if (!ctx.policy) notes.push("no policy loaded: decisions come from manifests and scope only"); + reports.push({ + name: p.name, + steps, + ...(blocked_at !== undefined ? { blocked_at } : {}), + approvals_needed: approvals, + max_risk, + total_risk: risks.reduce((a, b) => a + b, 0), + irreversible, + residue, + fully_reversible: rollback?.fully_reversible ?? true, + rollback, + rank: (blocked_at !== undefined ? 10_000 : 0) + approvals * 10 + max_risk + irreversible.length * 50 + (rollback?.fully_reversible === false ? 25 : 0), + notes, + }); + } + reports.sort((a, b) => a.rank - b.rank); + return { plans: reports, recommended: reports.find((r) => r.blocked_at === undefined)?.name }; +} + +// ---- impact preview ---- + +export interface ImpactPreview { + tool: string; + known: boolean; + status?: string; + effect: string | null; + owner?: string; + risk?: { data_class?: string; blast_radius?: string }; + verdict: Verdict; + reasons: string[]; + risk_score?: { score: number; factors: string[] }; + approval?: { roles: string[]; quorum: number; timeout_ms?: number; on_timeout?: string }; + compensation?: { tool: string; kind: string; residue: string[]; verified: boolean; needs_approval: boolean }; + requires: string[]; + rollback_outlook: string[]; + warnings: string[]; +} + +/** What would approving this call do, and what could be undone afterwards? */ +export async function previewImpact(ctx: SandboxContext, call: { tool: string; args?: unknown; actor?: string; tenant?: string }): Promise { + const m = ctx.registry.get(call.tool); + if (!m) { + return { tool: call.tool, known: false, effect: null, verdict: "block", reasons: ["no manifest is registered for this tool"], requires: [], rollback_outlook: ["unknown: without a manifest Reversi cannot say whether this can be undone"], warnings: ["unclassified tool"] }; + } + const sb = new Sandbox(ctx); + // Satisfy `requires` so the verdict and outlook are about this call, not about missing prerequisites. + for (const r of m.requires) await sb.seed({ run: "preview", tool: r, actor: call.actor, tenant: call.tenant }); + const judged = await sb.call({ run: "preview", tool: call.tool, args: call.args, actor: call.actor, tenant: call.tenant }); + const sb2 = new Sandbox(ctx, { assume_approved: true }); + for (const r of m.requires) await sb2.call({ run: "preview", tool: r, actor: call.actor, tenant: call.tenant }); + await sb2.call({ run: "preview", tool: call.tool, args: call.args, actor: call.actor, tenant: call.tenant }); + const plan = sb2.plan("preview"); + const mine = plan?.actions.find((a) => a.tool === call.tool); + const outlook: string[] = []; + if (!mine) outlook.push("nothing to undo"); + else if (mine.kind === "compensate") { + outlook.push(`undo by calling ${mine.compensation_tool}${mine.needs_approval ? " (needs approval)" : ""}`); + if (mine.residue?.length) outlook.push(`but ${mine.residue.join("; ")}`); + if (mine.verified) outlook.push("the undo is read back and verified"); + } else if (mine.kind === "skip") outlook.push("no undo needed: this does not change anything that needs undoing"); + else outlook.push(mine.note ?? "cannot be undone"); + const warnings: string[] = []; + if (m.effect === "irreversible") warnings.push("irreversible: once approved, this cannot be taken back"); + if (m.status === "draft") warnings.push("the manifest is a draft"); + if (m.status === "deprecated") warnings.push(`the manifest is deprecated${m.replaced_by ? `; use ${m.replaced_by}` : ""}`); + if (m.compensation && !m.compensation.verification) warnings.push("the undo is not verified: nothing reads back that it worked"); + const policy = ctx.policy ? new Policy(ctx.policy, ctx.now) : undefined; + const score = policy?.score({ tool: m.tool, effect: m.effect, permission: m.permission, actor: call.actor ?? "agent", run_id: "preview", environment: ctx.environment ?? "default", args: call.args, tenant: call.tenant, data_class: m.risk?.data_class, blast_radius: m.risk?.blast_radius }); + return { + tool: m.tool, + known: true, + status: m.status, + effect: m.effect, + owner: m.owner, + risk: m.risk, + verdict: judged.verdict, + reasons: judged.reasons, + ...(score ? { risk_score: score } : {}), + ...(m.approval_rules ? { approval: { roles: m.approval_rules.roles, quorum: m.approval_rules.quorum, timeout_ms: m.approval_rules.timeout_ms, on_timeout: m.approval_rules.on_timeout } } : {}), + ...(m.compensation ? { compensation: { tool: m.compensation.tool, kind: m.compensation.kind, residue: m.compensation.residue, verified: !!m.compensation.verification, needs_approval: m.compensation.requires_approval } } : {}), + requires: m.requires, + rollback_outlook: outlook, + warnings, + }; +} diff --git a/packages/reversi/src/smtp.ts b/packages/reversi/src/smtp.ts new file mode 100644 index 0000000..2940c21 --- /dev/null +++ b/packages/reversi/src/smtp.ts @@ -0,0 +1,143 @@ +/** + * A minimal SMTP client: EHLO, optional STARTTLS, AUTH PLAIN, one message. Enough to send notification mail through an + * ordinary relay without a dependency. It does not queue or retry (the notifier retries), and it only speaks UTF-8 text. + */ +import { randomUUID } from "node:crypto"; +import { connect as netConnect, type Socket } from "node:net"; +import { connect as tlsConnect, type TLSSocket } from "node:tls"; + +export interface SmtpConfig { + host: string; + port: number; + /** Implicit TLS (port 465). */ + secure?: boolean; + /** Upgrade with STARTTLS when the server offers it (default true). */ + starttls?: boolean; + user?: string; + pass?: string; + from: string; + /** Reject servers whose certificate does not verify (default true). */ + verify_tls?: boolean; + timeout_ms?: number; +} + +export interface MailMessage { + to: string[]; + subject: string; + text: string; +} + +class Lines { + private buf = ""; + private waiters: Array<(l: string) => void> = []; + private queue: string[] = []; + private error: Error | null = null; + private rejecters: Array<(e: Error) => void> = []; + constructor(private socket: Socket | TLSSocket) { + this.attach(socket); + } + attach(s: Socket | TLSSocket) { + this.socket = s; + s.on("data", (d) => { + this.buf += d.toString("utf8"); + let i: number; + while ((i = this.buf.indexOf("\r\n")) >= 0) { + const line = this.buf.slice(0, i); + this.buf = this.buf.slice(i + 2); + const w = this.waiters.shift(); + if (w) { + this.rejecters.shift(); + w(line); + } else this.queue.push(line); + } + }); + s.on("error", (e) => this.fail(e)); + s.on("close", () => this.fail(new Error("connection closed by the server"))); + } + private fail(e: Error) { + this.error = this.error ?? e; + for (const r of this.rejecters.splice(0)) r(this.error); + this.waiters.splice(0); + } + next(): Promise { + if (this.queue.length) return Promise.resolve(this.queue.shift() as string); + if (this.error) return Promise.reject(this.error); + return new Promise((res, rej) => { + this.waiters.push(res); + this.rejecters.push(rej); + }); + } + /** One full reply (multi-line replies end at a line whose 4th character is a space). */ + async reply(): Promise<{ code: number; lines: string[] }> { + const lines: string[] = []; + for (;;) { + const l = await this.next(); + lines.push(l); + if (l.length < 4 || l[3] === " ") return { code: Number(l.slice(0, 3)), lines }; + } + } +} + +const b64 = (s: string) => Buffer.from(s, "utf8").toString("base64"); +const asciiHeader = (s: string) => (/^[\x20-\x7e]*$/.test(s) ? s.replace(/[\r\n]/g, " ") : `=?UTF-8?B?${b64(s)}?=`); + +export async function sendMail(cfg: SmtpConfig, msg: MailMessage): Promise { + if (msg.to.length === 0) throw new Error("no recipients"); + for (const a of [cfg.from, ...msg.to]) if (/[\r\n<>]/.test(a)) throw new Error("invalid address"); + const timeout = cfg.timeout_ms ?? 15_000; + const tlsOpts = { host: cfg.host, port: cfg.port, servername: cfg.host, rejectUnauthorized: cfg.verify_tls !== false }; + const sock: Socket | TLSSocket = await new Promise((resolve, reject) => { + const s = cfg.secure ? tlsConnect(tlsOpts) : netConnect({ host: cfg.host, port: cfg.port }); + s.setTimeout(timeout, () => { + s.destroy(new Error(`SMTP timed out after ${timeout}ms`)); + }); + s.once(cfg.secure ? "secureConnect" : "connect", () => resolve(s)); + s.once("error", reject); + }); + let lines = new Lines(sock); + let current: Socket | TLSSocket = sock; + const send = async (cmd: string, expect: number[]) => { + current.write(cmd + "\r\n"); + const r = await lines.reply(); + if (!expect.includes(r.code)) throw new Error(`SMTP ${cmd.split(" ")[0]} failed: ${r.lines.join(" ")}`.replace(cfg.pass ?? "\u0000", "[redacted]")); + return r; + }; + try { + const hello = await lines.reply(); + if (hello.code !== 220) throw new Error(`SMTP greeting failed: ${hello.lines.join(" ")}`); + let ehlo = await send("EHLO reversi", [250]); + if (!cfg.secure && cfg.starttls !== false && ehlo.lines.some((l) => /STARTTLS/i.test(l))) { + await send("STARTTLS", [220]); + const upgraded: TLSSocket = await new Promise((resolve, reject) => { + const t = tlsConnect({ socket: current as Socket, servername: cfg.host, rejectUnauthorized: cfg.verify_tls !== false }); + t.once("secureConnect", () => resolve(t)); + t.once("error", reject); + }); + current = upgraded; + lines = new Lines(upgraded); + ehlo = await send("EHLO reversi", [250]); + } + if (cfg.user) await send(`AUTH PLAIN ${b64(`\u0000${cfg.user}\u0000${cfg.pass ?? ""}`)}`, [235]); + await send(`MAIL FROM:<${cfg.from}>`, [250]); + for (const to of msg.to) await send(`RCPT TO:<${to}>`, [250, 251]); + await send("DATA", [354]); + const headers = [ + `From: ${cfg.from}`, + `To: ${msg.to.join(", ")}`, + `Subject: ${asciiHeader(msg.subject)}`, + `Date: ${new Date().toUTCString()}`, + `Message-ID: <${randomUUID()}@reversi>`, + "MIME-Version: 1.0", + "Content-Type: text/plain; charset=utf-8", + "Content-Transfer-Encoding: base64", + ]; + const body = (b64(msg.text).match(/.{1,76}/g) ?? []).join("\r\n"); + current.write(`${headers.join("\r\n")}\r\n\r\n${body}\r\n.\r\n`); + const done = await lines.reply(); + if (done.code !== 250) throw new Error(`SMTP message rejected: ${done.lines.join(" ")}`); + current.write("QUIT\r\n"); + await Promise.race([lines.reply().catch(() => null), new Promise((r) => setTimeout(r, 1000))]); // let the server say goodbye + } finally { + current.destroy(); + } +} diff --git a/packages/reversi/src/sqlite.ts b/packages/reversi/src/sqlite.ts index 9fee498..91762f0 100644 --- a/packages/reversi/src/sqlite.ts +++ b/packages/reversi/src/sqlite.ts @@ -9,6 +9,7 @@ import { mkdirSync } from "node:fs"; import { dirname } from "node:path"; import { DatabaseSync } from "node:sqlite"; import type { ApprovalRecord, ApprovalStatus, ApprovalStore } from "./approvals.js"; +import type { KvEntry, KvStore } from "./kv.js"; import { LedgerConflictError, type LedgerRecord, type LedgerStore } from "./ledger.js"; import { RUN_ID_PATTERN, type RunState, type RunStore } from "./store.js"; @@ -31,6 +32,13 @@ export function openDatabase(path: string): DatabaseSync { run_id TEXT PRIMARY KEY, state TEXT NOT NULL ); + CREATE TABLE IF NOT EXISTS kv ( + ns TEXT NOT NULL, + key TEXT NOT NULL, + value TEXT NOT NULL, + version INTEGER NOT NULL, + PRIMARY KEY (ns, key) + ); CREATE TABLE IF NOT EXISTS approvals ( id TEXT PRIMARY KEY, record TEXT NOT NULL @@ -82,6 +90,9 @@ export class SqliteRunStore implements RunStore { .prepare("INSERT INTO runs (run_id, state) VALUES (?, ?) ON CONFLICT(run_id) DO UPDATE SET state = excluded.state") .run(this.check(state.run_id), JSON.stringify(state)); } + delete(run_id: string): boolean { + return Number(this.db.prepare("DELETE FROM runs WHERE run_id = ?").run(this.check(run_id)).changes) === 1; + } list(): string[] { return (this.db.prepare("SELECT run_id FROM runs ORDER BY run_id").all() as Array<{ run_id: string }>).map( (r) => r.run_id, @@ -103,21 +114,55 @@ export class SqliteApprovalStore implements ApprovalStore { list(): ApprovalRecord[] { return (this.db.prepare("SELECT record FROM approvals").all() as Array<{ record: string }>).map((r) => JSON.parse(r.record) as ApprovalRecord); } - transition(id: string, from: ApprovalStatus, patch: Partial): boolean { + delete(id: string): boolean { + return Number(this.db.prepare("DELETE FROM approvals WHERE id = ?").run(id).changes) === 1; + } + transition(id: string, from: ApprovalStatus, patch: Partial, expectCount?: number): boolean { const res = this.db - .prepare("UPDATE approvals SET record = json_patch(record, ?) WHERE id = ? AND json_extract(record, '$.status') = ?") - .run(JSON.stringify(patch), id, from); + .prepare( + `UPDATE approvals SET record = json_patch(record, ?) WHERE id = ? AND json_extract(record, '$.status') = ?${expectCount === undefined ? "" : " AND COALESCE(json_extract(record, '$.decision_count'), 0) = ?"}`, + ) + .run(...([JSON.stringify(patch), id, from, ...(expectCount === undefined ? [] : [expectCount])] as [string, string, string])); return Number(res.changes) === 1; } } +export class SqliteKvStore implements KvStore { + constructor(private readonly db: DatabaseSync) {} + get(ns: string, key: string): KvEntry | undefined { + const r = this.db.prepare("SELECT value, version FROM kv WHERE ns = ? AND key = ?").get(ns, key) as { value: string; version: number } | undefined; + return r ? { key, value: JSON.parse(r.value) as T, version: Number(r.version) } : undefined; + } + put(ns: string, key: string, value: unknown): void { + this.db + .prepare("INSERT INTO kv (ns, key, value, version) VALUES (?, ?, ?, 1) ON CONFLICT(ns, key) DO UPDATE SET value = excluded.value, version = version + 1") + .run(ns, key, JSON.stringify(value)); + } + cas(ns: string, key: string, value: unknown, expected: number): boolean { + if (expected === 0) { + const r = this.db.prepare("INSERT INTO kv (ns, key, value, version) VALUES (?, ?, ?, 1) ON CONFLICT(ns, key) DO NOTHING").run(ns, key, JSON.stringify(value)); + return Number(r.changes) === 1; + } + const r = this.db.prepare("UPDATE kv SET value = ?, version = version + 1 WHERE ns = ? AND key = ? AND version = ?").run(JSON.stringify(value), ns, key, expected); + return Number(r.changes) === 1; + } + delete(ns: string, key: string): boolean { + return Number(this.db.prepare("DELETE FROM kv WHERE ns = ? AND key = ?").run(ns, key).changes) === 1; + } + list(ns: string, prefix = ""): KvEntry[] { + const rows = this.db.prepare("SELECT key, value, version FROM kv WHERE ns = ? AND substr(key, 1, ?) = ? ORDER BY key").all(ns, prefix.length, prefix) as Array<{ key: string; value: string; version: number }>; + return rows.map((r) => ({ key: r.key, value: JSON.parse(r.value) as T, version: Number(r.version) })); + } +} + /** Open one database file and return the stores over it. */ export function openSqliteStores(path: string): { db: DatabaseSync; ledger: SqliteLedgerStore; runs: SqliteRunStore; approvals: SqliteApprovalStore; + kv: SqliteKvStore; } { const db = openDatabase(path); - return { db, ledger: new SqliteLedgerStore(db), runs: new SqliteRunStore(db), approvals: new SqliteApprovalStore(db) }; + return { db, ledger: new SqliteLedgerStore(db), runs: new SqliteRunStore(db), approvals: new SqliteApprovalStore(db), kv: new SqliteKvStore(db) }; } diff --git a/packages/reversi/src/store.ts b/packages/reversi/src/store.ts index cd37283..7eb93f0 100644 --- a/packages/reversi/src/store.ts +++ b/packages/reversi/src/store.ts @@ -14,6 +14,7 @@ import { readFileSync, readdirSync, renameSync, + unlinkSync, writeSync, } from "node:fs"; import { join } from "node:path"; @@ -50,9 +51,24 @@ export interface StepState { handle_missing: string[]; error: string | null; ts_ms: number; + /** Manifest version at call time. */ + manifest_version?: number; + /** `inverse` restores exact state; `compensation` may leave `residue`. Snapshotted with the compensation. */ + compensation_kind?: "inverse" | "compensation"; + residue?: string[]; + /** Read-back to run after the compensation: arguments resolved from this call's args and result. */ + verification?: { tool: string; refs: Record; args: Record; expect: Record; retries: number; backoff_ms: number; missing: string[] }; + compensation_timeout_ms?: number; + /** Rendered idempotency key, for duplicate detection. */ + idem_key?: string; + /** Hash of the result: a reference to it that is safe to keep. */ + result_hash?: string; + /** Risk score 0..100 computed when the call began, if risk scoring is configured. */ + risk_score?: number; } -export type Mode = "enforce" | "observe"; +/** `advise` behaves like `observe` (nothing is blocked) and tells the caller what would have been. */ +export type Mode = "enforce" | "observe" | "advise"; export interface RunState { run_id: string; @@ -60,6 +76,8 @@ export interface RunState { mode: Mode; scope: { permissions: string[]; seal: string; parent_seal: string | null }; created_ms: number; + /** Customer or team the run belongs to, for tenant-scoped policy and queries. */ + tenant?: string; steps: StepState[]; } @@ -67,6 +85,8 @@ export interface RunStore { get(run_id: string): RunState | undefined; put(state: RunState): void; list(): string[]; + /** Remove a run's state (retention). The ledger records about it stay: the ledger is append-only. */ + delete(run_id: string): boolean; } export const RUN_ID_PATTERN = /^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$/; @@ -83,6 +103,9 @@ export class MemoryRunStore implements RunStore { list(): string[] { return [...this.runs.keys()]; } + delete(run_id: string): boolean { + return this.runs.delete(run_id); + } } /** One JSON file per run. Writes go to a temp file, are fsynced, then renamed into place. */ @@ -116,6 +139,13 @@ export class FileRunStore implements RunStore { renameSync(tmp, f); } + delete(run_id: string): boolean { + const f = this.file(run_id); + if (!existsSync(f)) return false; + unlinkSync(f); + return true; + } + list(): string[] { return readdirSync(this.dir) .filter((n) => n.endsWith(".json")) diff --git a/packages/reversi/src/suggest.ts b/packages/reversi/src/suggest.ts new file mode 100644 index 0000000..810d37d --- /dev/null +++ b/packages/reversi/src/suggest.ts @@ -0,0 +1,259 @@ +/** + * Drafting manifests from what an MCP server says about its tools: names, descriptions, input schemas and annotations + * (`readOnlyHint`, `destructiveHint`, `idempotentHint`, `openWorldHint`), plus the `mcp-compensate` convention when a server + * declares it. Annotations are the server's own claims, so they only ever seed a **draft**: nothing here grants anything. + */ +import { createHash } from "node:crypto"; +import { canonicalize } from "./canonical.js"; +import type { Registry } from "./manifest.js"; + +export interface McpToolInfo { + name: string; + description?: string; + inputSchema?: { properties?: Record; required?: string[]; type?: string }; + outputSchema?: { properties?: Record } | undefined; + annotations?: { readOnlyHint?: boolean; destructiveHint?: boolean; idempotentHint?: boolean; openWorldHint?: boolean; title?: string } & Record; + _meta?: Record; +} + +export interface Hints { + read_only?: boolean; + destructive?: boolean; + idempotent?: boolean; + open_world?: boolean; +} + +export function hintsOf(t: McpToolInfo): Hints { + const a = t.annotations ?? {}; + return { + ...(typeof a.readOnlyHint === "boolean" ? { read_only: a.readOnlyHint } : {}), + ...(typeof a.destructiveHint === "boolean" ? { destructive: a.destructiveHint } : {}), + ...(typeof a.idempotentHint === "boolean" ? { idempotent: a.idempotentHint } : {}), + ...(typeof a.openWorldHint === "boolean" ? { open_world: a.openWorldHint } : {}), + }; +} + +/** Hash of a tool's input schema, stored in a manifest as `schema_hash` so drift can be noticed. */ +export function schemaHash(t: McpToolInfo): string { + return createHash("sha256").update(canonicalize(t.inputSchema ?? {})).digest("hex").slice(0, 16); +} + +export interface CompensateMeta { + readonly?: boolean; + compensable?: boolean; + irreversible?: boolean; + compensation?: { tool: string; mapping: Record }; +} + +/** + * Read the `mcp-compensate` convention from a tool: under `_meta["mcp-compensate"]`, `_meta.compensate` or + * `annotations["mcp-compensate"]`, an object with `readonly`, `compensable`, `irreversible` flags and a + * `compensation: { tool, parameters | params | mapping }` whose mapping is `{ compensationArg: "$result.x" | "$args.y" }`. + * (This is the shape Reversi accepts; check it against your server's documentation.) + */ +export function compensateMetaOf(t: McpToolInfo): CompensateMeta | undefined { + const raw = (t._meta?.["mcp-compensate"] ?? t._meta?.compensate ?? t.annotations?.["mcp-compensate"]) as Record | undefined; + if (typeof raw !== "object" || raw === null) return undefined; + const out: CompensateMeta = {}; + if (typeof raw.readonly === "boolean") out.readonly = raw.readonly; + if (typeof raw.compensable === "boolean") out.compensable = raw.compensable; + if (typeof raw.irreversible === "boolean") out.irreversible = raw.irreversible; + const c = raw.compensation as Record | string | undefined; + const tool = typeof c === "string" ? c : typeof c?.tool === "string" ? c.tool : undefined; + if (tool) { + const m = (typeof c === "object" ? (c.mapping ?? c.parameters ?? c.params) : undefined) as Record | undefined; + const mapping = Object.fromEntries(Object.entries(m ?? {}).filter(([, v]) => typeof v === "string" && /^\$(args|result)(\.[A-Za-z0-9_-]+)+$/.test(v as string)) as Array<[string, string]>); + out.compensation = { tool, mapping }; + } + return Object.keys(out).length ? out : undefined; +} + +const READ_VERBS = ["get", "list", "read", "search", "find", "describe", "fetch", "query", "show", "view", "check", "lookup", "count", "head", "stat", "retrieve", "browse", "export"]; +const DESTRUCTIVE_VERBS = ["delete", "remove", "drop", "destroy", "purge", "wipe", "erase", "truncate", "send", "publish", "charge", "pay", "transfer", "deploy", "merge", "execute", "exec", "run", "kill", "terminate", "revoke"]; +// forward verb -> verbs that undo it +const INVERSES: Record = { + create: ["delete", "remove", "cancel", "close", "archive", "destroy"], + add: ["remove", "delete", "unset"], + open: ["close"], + put: ["restore", "revert", "delete"], + write: ["restore", "revert", "delete"], + set: ["unset", "restore", "revert"], + update: ["restore", "revert"], + edit: ["restore", "revert"], + post: ["delete", "remove"], + enable: ["disable"], + start: ["stop"], + scale: ["restore", "unscale"], + charge: ["refund"], + grant: ["revoke"], + lock: ["unlock"], + subscribe: ["unsubscribe"], + schedule: ["cancel", "unschedule"], + insert: ["delete", "restore"], + move: ["unmove", "restore"], + transition: ["restore"], + assign: ["unassign"], + tag: ["untag"], + label: ["unlabel"], + react: ["unreact"], +}; + +function split(name: string): { prefix: string; verb: string; noun: string } { + const dot = Math.max(name.lastIndexOf("."), name.lastIndexOf("/"), name.lastIndexOf(":")); + const prefix = dot >= 0 ? name.slice(0, dot + 1) : ""; + const rest = name.slice(dot + 1); + const words = rest.split(/[_\-\s]+|(?<=[a-z])(?=[A-Z])/).filter(Boolean).map((w) => w.toLowerCase()); + return { prefix, verb: words[0] ?? "", noun: words.slice(1).join("_") }; +} + +export interface Suggestion { + manifest: Record; + confidence: "low" | "medium" | "high"; + notes: string[]; +} + +export interface SuggestOptions { + /** Server name, used for permission names (`:read`). */ + server?: string; + /** Tools that already have manifests are skipped. */ + existing?: Registry; + /** Draft only these tools. */ + only?: string[]; +} + +function dataClassOf(name: string, desc: string): string { + const s = `${name} ${desc}`.toLowerCase(); + if (/(secret|token|credential|password|api[_ -]?key|private[_ -]?key)/.test(s)) return "secret"; + if (/(payment|invoice|charge|refund|billing|card|payout|ledger|price)/.test(s)) return "financial"; + if (/(customer|user|email|contact|person|employee|patient|profile|address|phone)/.test(s)) return "pii"; + return "internal"; +} + +function mapArgs(forward: McpToolInfo, comp: McpToolInfo, notes: string[]): Record { + const fProps = Object.keys(forward.inputSchema?.properties ?? {}); + const cProps = Object.keys(comp.inputSchema?.properties ?? {}); + const required = comp.inputSchema?.required ?? cProps; + const out: Record = {}; + for (const p of required.filter((x) => cProps.includes(x))) { + if (fProps.includes(p) && !/(^|_)(id|number|key|ts|uuid)$/.test(p)) out[p] = `$args.${p}`; + else if (/(^|_)(id|number|key|ts|uuid|name)$/.test(p) || p === "id") out[p] = `$result.${p}`; + else if (fProps.includes(p)) out[p] = `$args.${p}`; + else { + out[p] = `$result.${p}`; + notes.push(`could not tell where the undo's \`${p}\` comes from: assumed \`$result.${p}\`, check the tool's output`); + } + } + return out; +} + +export function suggestManifests(tools: McpToolInfo[], o: SuggestOptions = {}): { suggestions: Suggestion[]; skipped: string[]; ignored: string[] } { + const byName = new Map(tools.map((t) => [t.name, t])); + const server = (o.server ?? "upstream").replace(/[^A-Za-z0-9_.-]/g, "_"); + const suggestions: Suggestion[] = []; + const skipped: string[] = []; + const claimedAsUndo = new Set(); + + const parsed = tools.map((t) => ({ t, ...split(t.name), hints: hintsOf(t), cm: compensateMetaOf(t) })); + const isReadLike = (p: (typeof parsed)[number]) => p.cm?.readonly === true || p.hints.read_only === true || (READ_VERBS.includes(p.verb) && p.hints.destructive !== true); + const findInverse = (p: (typeof parsed)[number]): McpToolInfo | undefined => { + for (const inv of INVERSES[p.verb] ?? []) { + const cand = parsed.find((q) => q.t.name !== p.t.name && q.prefix === p.prefix && q.verb === inv && (q.noun === p.noun || (p.noun === "" && q.noun === ""))); + if (cand) return cand.t; + } + return undefined; + }; + + // Which tools are somebody's undo? They get drafted as idempotent helpers rather than as irreversible actions. + const undoOf = new Set(); + for (const p of parsed) { + if (p.cm?.compensation) undoOf.add(p.cm.compensation.tool); + else if (!isReadLike(p) && p.hints.destructive !== true) { + const inv = findInverse(p); + if (inv) undoOf.add(inv.name); + } + } + for (const p of parsed) { + if (o.only && !o.only.includes(p.t.name)) continue; + if (o.existing?.has(p.t.name)) { + skipped.push(p.t.name); + continue; + } + const notes: string[] = []; + let confidence: Suggestion["confidence"] = "low"; + const desc = p.t.description?.split("\n")[0]?.trim() ?? ""; + let effect: string; + let compensation: Record | undefined; + + const readByName = READ_VERBS.includes(p.verb); + if (p.cm?.readonly === true || p.hints.read_only === true) { + effect = "read_only"; + confidence = p.cm?.readonly ? "high" : "medium"; + notes.push(p.cm?.readonly ? "read-only per the server's mcp-compensate metadata" : "read-only per the server's readOnlyHint (a claim by the server, not verified)"); + } else if (p.cm?.irreversible === true) { + effect = "irreversible"; + confidence = "high"; + notes.push("irreversible per the server's mcp-compensate metadata"); + } else if (p.cm?.compensation) { + effect = p.cm.compensable === false ? "reversible" : "compensable"; + confidence = "high"; + const c = byName.get(p.cm.compensation.tool); + compensation = { tool: p.cm.compensation.tool, argument_map: p.cm.compensation.mapping }; + if (!c) notes.push(`mcp-compensate names \`${p.cm.compensation.tool}\` as the undo but the server does not list it`); + claimedAsUndo.add(p.cm.compensation.tool); + } else if (readByName && p.hints.destructive !== true) { + effect = "read_only"; + confidence = "low"; + notes.push(`classified read_only from the name \`${p.verb}_…\` only: confirm it changes nothing`); + } else { + const inv = findInverse(p); + if (inv && p.hints.destructive !== true) { + const map = mapArgs(p.t, inv, notes); + effect = "compensable"; + compensation = { tool: inv.name, argument_map: map, kind: "compensation", residue: ["TODO describe what the undo cannot take back"] }; + confidence = "medium"; + notes.push(`undo guessed from the tool names: \`${inv.name}\`; check the argument map and decide whether it is a true inverse (reversible) or leaves residue (compensable)`); + claimedAsUndo.add(inv.name); + } else if (p.hints.idempotent === true && p.hints.destructive !== true) { + effect = "idempotent"; + confidence = "medium"; + notes.push("idempotent per the server's idempotentHint (unverified)"); + } else if (undoOf.has(p.t.name)) { + effect = "idempotent"; + confidence = "medium"; + notes.push("used as another tool's undo, so it is drafted idempotent: rollback may call it more than once; confirm that is safe"); + } else { + effect = "irreversible"; + confidence = p.hints.destructive === true || DESTRUCTIVE_VERBS.includes(p.verb) ? "medium" : "low"; + notes.push(p.hints.destructive === true ? "marked destructive by the server and no undo tool was found" : "no undo tool was found, so it is drafted as irreversible: the safe default until you say otherwise"); + } + } + const props = Object.keys(p.t.inputSchema?.properties ?? {}); + const m: Record = { + tool: p.t.name, + ...(desc ? { description: desc } : { description: `${p.t.name} (describe what this does)` }), + effect, + permission: `${server}:${effect === "read_only" ? "read" : effect === "irreversible" ? "admin" : "write"}`, + status: "draft", + ...(effect === "read_only" ? {} : { risk: { data_class: dataClassOf(p.t.name, desc), blast_radius: effect === "irreversible" ? "high" : "medium" } }), + ...(compensation ? { compensation } : {}), + ...(p.t.inputSchema ? { schema_hash: schemaHash(p.t) } : {}), + }; + if (effect === "irreversible") m.approval = "always"; + if (props.length === 0 && effect !== "read_only") notes.push("the tool takes no arguments, so an undo cannot identify what to undo"); + if (p.hints.open_world === true) notes.push("openWorldHint: it reaches outside systems; consider a lower blast-radius budget and egress rules"); + suggestions.push({ manifest: m, confidence, notes }); + } + // The undo tools themselves are idempotent helpers: draft them too if the server offers them and they were not drafted. + for (const name of claimedAsUndo) { + if (suggestions.some((s) => s.manifest.tool === name) || o.existing?.has(name) || (o.only && !o.only.includes(name))) continue; + const t = byName.get(name); + if (!t) continue; + suggestions.push({ + manifest: { tool: name, description: t.description?.split("\n")[0]?.trim() || `${name} (undo helper)`, effect: "idempotent", permission: `${server}:write`, status: "draft", ...(t.inputSchema ? { schema_hash: schemaHash(t) } : {}) }, + confidence: "medium", + notes: ["used as another tool's undo, so it is drafted idempotent: rollback may call it more than once; confirm that is safe"], + }); + } + const names = new Set(suggestions.map((s) => s.manifest.tool as string)); + return { suggestions, skipped, ignored: tools.filter((t) => !names.has(t.name) && !skipped.includes(t.name)).map((t) => t.name) }; +} diff --git a/packages/reversi/src/weekly.ts b/packages/reversi/src/weekly.ts new file mode 100644 index 0000000..f4a3e42 --- /dev/null +++ b/packages/reversi/src/weekly.ts @@ -0,0 +1,210 @@ +/** + * The periodic report: what agents did, what was blocked or would have been (the observe/advise findings), what needed + * approval, how rollbacks went, and what needs attention. Built from the ledger, run state and approvals. + */ +import type { Approvals } from "./approvals.js"; +import type { Reversi } from "./engine.js"; +import { searchRuns } from "./query.js"; + +export interface TopEntry { + key: string; + count: number; +} + +export interface PeriodReport { + from_ms: number; + to_ms: number; + runs: number; + calls: number; + actors: TopEntry[]; + tools: TopEntry[]; + /** Calls by the effect class of the manifest (or "no manifest"). */ + by_effect: TopEntry[]; + outcomes: { succeeded: number; failed: number; uncertain: number }; + blocked: { total: number; reasons: TopEntry[] }; + would_block: { total: number; reasons: TopEntry[] }; + unclassified_tools: TopEntry[]; + approvals: { requested: number; approved: number; denied: number; expired: number; escalated: number; median_latency_s: number | null }; + rollback: { compensated: number; failed: number; unverified: number; verified: number; success_rate: number | null }; + irreversible_executed: number; + needs_attention: Array<{ run_id: string; actor: string; status: string }>; + budget_warnings: TopEntry[]; + days: Array<{ day: string; calls: number; blocked: number }>; +} + +const top = (m: Map, n = 10): TopEntry[] => [...m].sort((a, b) => b[1] - a[1] || a[0].localeCompare(b[0])).slice(0, n).map(([key, count]) => ({ key, count })); +const bump = (m: Map, k: string) => m.set(k, (m.get(k) ?? 0) + 1); +const reasonKey = (detail: string) => detail.replace(/apr_[0-9a-f-]+/g, "apr_…").replace(/`[^`]*`/g, (x) => x).slice(0, 140) || "(no detail)"; + +export function buildPeriodReport(rv: Reversi, approvals: Approvals | undefined, from_ms: number, to_ms: number = Date.now()): PeriodReport { + const actors = new Map(); + const tools = new Map(); + const byEffect = new Map(); + const blocked = new Map(); + const wouldBlock = new Map(); + const unclassified = new Map(); + const budget = new Map(); + const days = new Map(); + const runsSeen = new Set(); + const out = { succeeded: 0, failed: 0, uncertain: 0 }; + const rb = { compensated: 0, failed: 0, unverified: 0, verified: 0 }; + let calls = 0; + let irreversible = 0; + let blockedTotal = 0; + let wouldTotal = 0; + const day = (ms: number) => new Date(ms).toISOString().slice(0, 10); + + for (const r of rv.ledger.records()) { + if (r.ts_ms < from_ms || r.ts_ms > to_ms) continue; + if (r.run_id !== "-") runsSeen.add(r.run_id); + const d = days.get(day(r.ts_ms)) ?? { calls: 0, blocked: 0 }; + switch (r.event) { + case "intended": + calls++; + d.calls++; + bump(actors, r.actor); + bump(tools, r.tool); + bump(byEffect, rv.registry.get(r.tool)?.effect ?? "no manifest"); + break; + case "succeeded": + out.succeeded++; + if (rv.registry.get(r.tool)?.effect === "irreversible") irreversible++; + break; + case "failed": + out.failed++; + break; + case "uncertain": + out.uncertain++; + break; + case "blocked": + if (!r.detail.startsWith("approval ") || r.detail.includes("was denied")) { + blockedTotal++; + d.blocked++; + bump(blocked, reasonKey(r.detail)); + } + if (/no manifest registered/.test(r.detail)) bump(unclassified, r.tool); + break; + case "would_block": + wouldTotal++; + bump(wouldBlock, reasonKey(r.detail)); + if (/no manifest registered/.test(r.detail)) bump(unclassified, r.tool); + break; + case "compensated": + rb.compensated++; + break; + case "compensation_failed": + rb.failed++; + break; + case "verified": + rb.verified++; + break; + case "unverified": + rb.unverified++; + break; + default: + break; + } + const b = /budget `([^`]+)` at \d+%/.exec(r.detail); + if (b) bump(budget, b[1] as string); + days.set(day(r.ts_ms), d); + } + + const appr = (approvals?.list() ?? []).filter((a) => a.requested_ms >= from_ms && a.requested_ms <= to_ms); + const lat = appr.filter((a) => a.decided_ms !== null).map((a) => ((a.decided_ms as number) - a.requested_ms) / 1000).sort((a, b) => a - b); + const attention = searchRuns(rv, { status: "needs_attention", limit: 100 }).runs.map((r) => ({ run_id: r.run_id, actor: r.actor, status: r.status })); + const attempted = rb.compensated + rb.failed; + return { + from_ms, + to_ms, + runs: runsSeen.size, + calls, + actors: top(actors), + tools: top(tools), + by_effect: top(byEffect), + outcomes: out, + blocked: { total: blockedTotal, reasons: top(blocked) }, + would_block: { total: wouldTotal, reasons: top(wouldBlock) }, + unclassified_tools: top(unclassified), + approvals: { + requested: appr.length, + approved: appr.filter((a) => a.status === "approved" || a.status === "consumed").length, + denied: appr.filter((a) => a.status === "denied").length, + expired: appr.filter((a) => a.status === "expired").length, + escalated: appr.filter((a) => a.escalated).length, + median_latency_s: lat.length ? (lat[Math.floor(lat.length / 2)] as number) : null, + }, + rollback: { ...rb, success_rate: attempted ? Math.round((rb.compensated / attempted) * 1000) / 10 : null }, + irreversible_executed: irreversible, + needs_attention: attention, + budget_warnings: top(budget), + days: [...days].sort((a, b) => a[0].localeCompare(b[0])).map(([d, v]) => ({ day: d, ...v })), + }; +} + +const list = (t: TopEntry[]) => (t.length ? t.map((e) => `- ${e.key}: ${e.count}`).join("\n") : "- none"); + +export function renderPeriodMarkdown(r: PeriodReport, title = "Reversi report"): string { + const iso = (ms: number) => new Date(ms).toISOString().slice(0, 16).replace("T", " ") + "Z"; + const lines = [ + `# ${title}`, + "", + `Period: ${iso(r.from_ms)} to ${iso(r.to_ms)}`, + "", + `**${r.calls}** tool calls in **${r.runs}** runs. ${r.outcomes.succeeded} succeeded, ${r.outcomes.failed} failed, ${r.outcomes.uncertain} with an unknown outcome.`, + "", + "## What the agents did", + "Calls by tool:", + list(r.tools), + "", + "By reversibility class:", + list(r.by_effect), + "", + "By actor:", + list(r.actors), + "", + `## Blocked: ${r.blocked.total}`, + list(r.blocked.reasons), + "", + `## Would have been blocked (observe and advise modes): ${r.would_block.total}`, + "These calls ran. Enforcing would have stopped them:", + list(r.would_block.reasons), + "", + "## Tools called without a manifest", + list(r.unclassified_tools), + "", + "## Approvals", + `${r.approvals.requested} requested: ${r.approvals.approved} approved, ${r.approvals.denied} denied, ${r.approvals.expired} expired unanswered, ${r.approvals.escalated} escalated.${r.approvals.median_latency_s !== null ? ` Median time to a decision: ${Math.round(r.approvals.median_latency_s)}s.` : ""}`, + "", + "## Rollback", + `${r.rollback.compensated} compensations succeeded, ${r.rollback.failed} failed${r.rollback.success_rate !== null ? ` (${r.rollback.success_rate}% success)` : ""}. ${r.rollback.verified} verified, ${r.rollback.unverified} could not be verified.`, + `Irreversible actions executed: ${r.irreversible_executed}.`, + "", + "## Needs attention", + r.needs_attention.length ? r.needs_attention.map((x) => `- ${x.run_id} (${x.actor})`).join("\n") : "- nothing", + "", + "## Budgets near their limit", + list(r.budget_warnings), + "", + "## Calls per day", + r.days.length ? r.days.map((d) => `- ${d.day}: ${d.calls} calls, ${d.blocked} blocked`).join("\n") : "- none", + "", + ]; + return lines.join("\n"); +} + +const esc = (s: string) => s.replace(/[&<>"']/g, (c) => ({ "&": "&", "<": "<", ">": ">", '"': """, "'": "'" })[c] as string); + +export function renderPeriodHtml(r: PeriodReport, title = "Reversi report"): string { + const md = renderPeriodMarkdown(r, title); + const body = md + .split("\n") + .map((l) => { + if (l.startsWith("# ")) return `

${esc(l.slice(2))}

`; + if (l.startsWith("## ")) return `

${esc(l.slice(3))}

`; + if (l.startsWith("- ")) return `
  • ${esc(l.slice(2))}
    • `; + return l ? `

    ${esc(l).replace(/\*\*(.+?)\*\*/g, "$1")}

    ` : ""; + }) + .join("\n") + .replace(/(
  • .*<\/li>\n?)+/g, (m) => `
      ${m}
    `); + return `${esc(title)}${body}`; +} diff --git a/packages/reversi/test/capture-simulate.test.ts b/packages/reversi/test/capture-simulate.test.ts new file mode 100644 index 0000000..5858888 --- /dev/null +++ b/packages/reversi/test/capture-simulate.test.ts @@ -0,0 +1,207 @@ +import { describe, expect, it } from "vitest"; +import { + CaptureStore, + ContentChecker, + Ledger, + MemoryKvStore, + Policy, + Registry, + Reversi, + SealedScope, + detectPii, + maskPii, + parsePolicy, + previewImpact, + redactForCapture, + replayRun, + simulatePolicy, + speculate, +} from "../src/index.js"; + +describe("personal data detection", () => { + it("finds and masks emails, cards, IBANs, SSNs and phone numbers, and only with valid checksums", () => { + const text = "mail ada@example.com, card 4111 1111 1111 1111, iban GB82 WEST 1234 5698 7654 32, ssn 123-45-6789, call +1 415-555-0132"; + expect(detectPii(text).sort()).toEqual(["credit_card", "email", "iban", "phone", "ssn"]); + expect(maskPii(text)).toBe("mail [PII:email], card [PII:credit_card], iban [PII:iban], ssn [PII:ssn], call [PII:phone]"); + expect(detectPii("order 4111 1111 1111 1112 failed")).toEqual([]); // fails Luhn + expect(detectPii("GB82 WEST 1234 5698 7654 33")).toEqual([]); // fails the IBAN check + expect(detectPii("version 1.2.3 build 20261015")).toEqual([]); + }); + + it("a pii content check flags, asks or denies", () => { + const c = new ContentChecker({ pii: "require_approval" }); + expect(c.inspect({ note: "contact ada@example.com" })).toEqual([{ kind: "pii", action: "require_approval", detail: "personal data (email)" }]); + expect(new ContentChecker().inspect({ note: "contact ada@example.com" })).toEqual([]); // off unless configured + expect(() => parsePolicy({ content: { pii: "sometimes" } })).toThrow(); + }); +}); + +describe("capture", () => { + const cap = (cfg = {}, t = () => 1_000) => new CaptureStore(new MemoryKvStore(), { enabled: true, ...cfg }, t); + + it("redacts credentials, personal data and manifest-listed paths on the way in", () => { + const out = redactForCapture({ to: "ada@example.com", token: "ghp_" + "a".repeat(36), client_secret: "s3cr3t", nested: { n: 1 } }, { redact_pii: true, max_chars: 100, paths: ["$args.client_secret"], root: "args" }) as Record; + expect(out).toEqual({ to: "[PII:email]", token: "[REDACTED:github_token]", client_secret: "[REDACTED]", nested: { n: 1 } }); + expect(redactForCapture({ big: "x".repeat(500) }, { redact_pii: false, max_chars: 10 })).toEqual({ big: "xxxxxxxxxx…[cut]" }); + expect(redactForCapture({ n: Infinity }, { redact_pii: false, max_chars: 10 })).toEqual({ n: "Infinity" }); + }); + + it("does nothing unless enabled", () => { + const c = new CaptureStore(new MemoryKvStore()); + expect(c.record({ run_id: "r", step: 0, tool: "t", actor: "a", environment: "e", args: {}, decision: "allow", reasons: [] })).toBeNull(); + expect(c.list()).toEqual([]); + }); + + it("expires, purges and lists by run and tool", () => { + let t = 1_000; + const c = cap({ ttl_ms: 5_000 }, () => t); + c.record({ run_id: "r1", step: 0, tool: "a", actor: "x", environment: "e", args: { k: 1 }, decision: "allow", reasons: [] }); + t = 2_000; + c.record({ run_id: "r2", step: 0, tool: "b", actor: "x", environment: "e", args: {}, decision: "block", reasons: ["nope"] }); + expect(c.list()).toHaveLength(2); + expect(c.list({ run_id: "r1" }).map((r) => r.tool)).toEqual(["a"]); + expect(c.list({ tool: "b" })).toHaveLength(1); + t = 7_000; // r1 (kept until 6,000) has expired, r2 (until 7,000) too + expect(c.list()).toEqual([]); + expect(c.purge()).toBe(2); + expect(c.purge()).toBe(0); + t = 8_000; + c.record({ run_id: "r3", step: 0, tool: "a", actor: "x", environment: "e", args: {}, decision: "allow", reasons: [] }); + expect(c.purge({ run_id: "r3" })).toBe(1); + }); + + it("the engine captures decisions, blocked calls and outcomes, with redaction", async () => { + const capture = cap(); + const rv = new Reversi({ + registry: Registry.from([ + { tool: "send", effect: "irreversible", permission: "p", audit: { redaction: ["$args.api_key", "$result.secret"] } }, + { tool: "read", effect: "read_only", permission: "p" }, + ]), + ledger: new Ledger(), + capture, + approver: () => true, + }); + const run = rv.startRun({ run_id: "r1", actor: "agent", scope: SealedScope.seal(["p"]), tenant: "acme" }); + await run.call({ tool: "send", args: { to: "ada@example.com", api_key: "k-123" }, reason: "x", execute: () => ({ ok: true, secret: "shh" }) }); + await run.call({ tool: "nope", args: { q: 1 }, reason: "x", execute: () => 1 }); + const recs = capture.list({ run_id: "r1" }); + expect(recs[0]).toMatchObject({ tool: "send", step: 0, decision: "allow", outcome: "succeeded", tenant: "acme", args: { to: "[PII:email]", api_key: "[REDACTED]" }, result: { ok: true, secret: "[REDACTED]" } }); + expect(recs[1]).toMatchObject({ tool: "nope", step: null, decision: "block" }); + expect(capture.get("r1", 0)?.tool).toBe("send"); + }); +}); + +const MANIFESTS = [ + { tool: "orders.create", effect: "reversible", permission: "p", compensation: { tool: "orders.cancel", argument_map: { id: "$result.id" }, residue: [] } }, + { tool: "orders.cancel", effect: "idempotent", permission: "p" }, + { tool: "pay.charge", effect: "compensable", permission: "p", risk: { data_class: "financial", blast_radius: "high" }, requires: ["orders.create"], compensation: { tool: "pay.refund", residue: ["statement line stays"], argument_map: { charge: "$result.charge_id", amount: "$args.amount" }, verification: { tool: "pay.get", arguments: { charge: "$result.charge_id" }, expect: { status: "refunded" } } } }, + { tool: "pay.refund", effect: "idempotent", permission: "p" }, + { tool: "pay.get", effect: "read_only", permission: "p" }, + { tool: "mail.send", effect: "irreversible", permission: "p" }, +]; +const registry = () => Registry.from(MANIFESTS); + +function rec(over: object): import("../src/index.js").CaptureRecord { + return { run_id: "r1", step: 0, tool: "orders.create", actor: "agent", environment: "prod", args: {}, decision: "allow", reasons: [], outcome: "succeeded", ts_ms: 1, expires_ms: 9e12, ...over } as never; +} + +describe("policy simulation against history", () => { + const history = [ + rec({ step: 0, ts_ms: 1, tool: "orders.create", result: { id: "o1" } }), + rec({ step: 1, ts_ms: 2, tool: "pay.charge", args: { amount: 50 }, result: { charge_id: "c1" } }), + rec({ step: 2, ts_ms: 3, tool: "pay.charge", args: { amount: 5000 }, result: { charge_id: "c2" } }), + rec({ run_id: "r2", step: null, ts_ms: 4, tool: "mail.send", decision: "block", reasons: ["approval required: request apr_1 is pending"] }), + ]; + + it("reports what a stricter policy would have blocked, what needs approval, and what it would newly allow", async () => { + const policy = parsePolicy({ + rules: [ + { id: "big", match: { tool: "pay.charge" }, when: [{ path: "$args.amount", op: "gt", value: 1000 }], action: "deny", reason: "too large" }, + { id: "mail-ok", match: { tool: "mail.send" }, action: "allow" }, + ], + }); + const rep = await simulatePolicy(history, { registry: registry(), policy, environment: "prod" }); + expect(rep).toMatchObject({ calls: 4, runs: 2 }); + expect(rep.newly_blocked).toEqual([expect.objectContaining({ tool: "pay.charge", step: 2, before: "allow", after: "block" })]); + expect(rep.newly_blocked[0]?.reasons[0]).toContain("too large"); + // mail.send is irreversible so the manifest still wants approval; it was already waiting for it: unchanged + expect(rep.unchanged).toBe(3); + const none = await simulatePolicy(history, { registry: registry(), environment: "prod" }); + expect(none.newly_blocked).toEqual([]); + const tight = await simulatePolicy(history.slice(0, 2), { registry: registry(), policy: parsePolicy({ rules: [{ id: "all", match: { tool: "pay.*" }, action: "require_approval" }] }) }); + expect(tight.newly_needs_approval.map((c) => c.tool)).toEqual(["pay.charge"]); + }); + + it("does not touch real state: the sandbox has its own ledger and counters", async () => { + const policy = parsePolicy({ limits: [{ id: "one", match: { tool: "orders.create" }, max_calls: 1, window_seconds: 60, per: "global" }] }); + const a = await simulatePolicy([rec({ step: 0 }), rec({ step: 1, ts_ms: 2 })], { registry: registry(), policy }); + expect(a.newly_blocked).toHaveLength(1); + const b = await simulatePolicy([rec({ step: 0 })], { registry: registry(), policy }); // a fresh sandbox: the limit is not remembered + expect(b.newly_blocked).toHaveLength(0); + }); +}); + +describe("replay with a change", () => { + it("shows the decision that changes and how the rollback outlook changes", async () => { + const run = [ + rec({ step: 0, ts_ms: 1, tool: "orders.create", result: { id: "o1" } }), + rec({ step: 1, ts_ms: 2, tool: "pay.charge", args: { amount: 50 }, result: { charge_id: "c1" } }), + ]; + const policy = parsePolicy({ rules: [{ id: "big", match: { tool: "pay.charge" }, when: [{ path: "$args.amount", op: "gt", value: 1000 }], action: "deny", reason: "too large" }] }); + const rep = await replayRun(run, { registry: registry(), policy }, [{ tool: "pay.charge", path: "$args.amount", value: 9000 }]); + expect(rep.baseline.map((s) => s.verdict)).toEqual(["allow", "allow"]); + expect(rep.variant.map((s) => s.verdict)).toEqual(["allow", "block"]); + expect(rep.differences).toEqual([expect.objectContaining({ index: 1, tool: "pay.charge", before: "allow", after: "block" })]); + expect(rep.baseline_plan?.actions.map((a) => a.tool)).toEqual(["pay.charge", "orders.create"]); + expect(rep.baseline_plan?.actions[0]).toMatchObject({ residue: ["statement line stays"], verified: true }); + }); +}); + +describe("speculative execution", () => { + it("compares alternative plans and recommends the safest one that can run", async () => { + const ctx = { registry: registry(), policy: parsePolicy({ risk: { thresholds: { deny: 95 } } }), environment: "prod" }; + const report = await speculate( + [ + { name: "charge then email", calls: [{ tool: "orders.create" }, { tool: "pay.charge", args: { amount: 20 } }, { tool: "mail.send" }] }, + { name: "charge only", calls: [{ tool: "orders.create" }, { tool: "pay.charge", args: { amount: 20 } }] }, + { name: "charge without an order", calls: [{ tool: "pay.charge", args: { amount: 20 } }] }, + ], + ctx, + ); + expect(report.recommended).toBe("charge only"); + expect(report.plans.map((p) => p.name)).toEqual(["charge only", "charge then email", "charge without an order"]); + const [only, withMail, bad] = report.plans as [(typeof report.plans)[number], (typeof report.plans)[number], (typeof report.plans)[number]]; + expect(only).toMatchObject({ fully_reversible: true, irreversible: [], approvals_needed: 0 }); + expect(only.residue).toEqual(["statement line stays"]); + expect(withMail).toMatchObject({ irreversible: ["mail.send"], approvals_needed: 1, fully_reversible: false }); + expect(withMail.rank).toBeGreaterThan(only.rank); + expect(bad.blocked_at).toBe(0); + expect(bad.steps[0]?.reasons[0]).toContain("requires `orders.create`"); + expect(only.notes[0]).toContain("synthesized"); + }); +}); + +describe("impact preview", () => { + it("says what approving would do and what could be undone", async () => { + const policy = parsePolicy({ risk: { thresholds: { approve: 50 } } }); + const p = await previewImpact({ registry: registry(), policy, environment: "prod" }, { tool: "pay.charge", args: { amount: 20 }, actor: "agent" }); + expect(p).toMatchObject({ + known: true, + effect: "compensable", + verdict: "needs_approval", + risk: { data_class: "financial", blast_radius: "high" }, + requires: ["orders.create"], + compensation: { tool: "pay.refund", residue: ["statement line stays"], verified: true }, + }); + expect(p.risk_score?.score).toBe(80); + expect(p.rollback_outlook).toEqual(["undo by calling pay.refund", "but statement line stays", "the undo is read back and verified"]); + expect(p.warnings).toEqual([]); + + const mail = await previewImpact({ registry: registry() }, { tool: "mail.send" }); + expect(mail.warnings).toContain("irreversible: once approved, this cannot be taken back"); + expect(mail.rollback_outlook[0]).toContain("cannot be undone"); + + const unknown = await previewImpact({ registry: registry() }, { tool: "mystery" }); + expect(unknown).toMatchObject({ known: false, verdict: "block", warnings: ["unclassified tool"] }); + }); +}); diff --git a/packages/reversi/test/engine-config.test.ts b/packages/reversi/test/engine-config.test.ts new file mode 100644 index 0000000..97a1c65 --- /dev/null +++ b/packages/reversi/test/engine-config.test.ts @@ -0,0 +1,254 @@ +import { createServer } from "node:http"; +import { mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import type { AddressInfo } from "node:net"; +import { afterEach, describe, expect, it } from "vitest"; +import { + ManifestRegistry, + PolicyVersions, + ProjectConfigError, + findProjectConfig, + generateSigningKey, + loadProjectConfig, + openEngine, + readAnchors, + SealedScope, + starterConfig, + type EngineHandle, +} from "../src/index.js"; + +const dirs: string[] = []; +const open: EngineHandle[] = []; +afterEach(async () => { + while (open.length) await (open.pop() as EngineHandle).close(); + while (dirs.length) rmSync(dirs.pop() as string, { recursive: true, force: true }); +}); +const tmp = () => { + const d = mkdtempSync(join(tmpdir(), "rv-cfg-")); + dirs.push(d); + return d; +}; +const log = () => { + const lines: string[] = []; + return { lines, fn: (l: string, m: string) => lines.push(`${l}: ${m}`) }; +}; + +const MANIFESTS = [ + { tool: "orders.create", effect: "reversible", permission: "p", compensation: { tool: "orders.cancel", argument_map: { id: "$result.id" } } }, + { tool: "orders.cancel", effect: "idempotent", permission: "p" }, + { tool: "mail.send", effect: "irreversible", permission: "p" }, +]; +const scope = () => SealedScope.seal(["p"]); + +function folder(manifests = MANIFESTS) { + const dir = tmp(); + writeFileSync(join(dir, "tools.json"), JSON.stringify(manifests)); + return dir; +} + +describe("openEngine: sources", () => { + it("loads a folder, a signed registry and a versioned policy", async () => { + const dir = tmp(); + const key = generateSigningKey(); + const reg = new ManifestRegistry(join(dir, "registry")); + for (const m of MANIFESTS) reg.publish(m, { by: "ci", status: "approved", key }); + writeFileSync(join(dir, "pub.pem"), key.public_pem); + const pv = new PolicyVersions(join(dir, "policy")); + pv.publish("version: 1\nrules:\n - id: no-mail\n match: { tool: mail.send }\n action: deny\n", { by: "ci" }); + const h = await openEngine({ manifests: join(dir, "registry"), manifest_public_key: join(dir, "pub.pem"), policy: join(dir, "policy"), data: { dir: join(dir, "data") }, log: () => {} }); + open.push(h); + expect(h.registry.size).toBe(3); + const run = h.reversi.startRun({ actor: "a", scope: scope() }); + expect((await run.call({ tool: "mail.send", reason: "x", execute: () => 1 })).status).toBe("blocked"); + // an unsigned manifest slipped into the registry is refused at start + new ManifestRegistry(join(dir, "registry")).publish({ tool: "evil", effect: "read_only", permission: "p" }, { by: "mallory", status: "approved" }); + await expect(openEngine({ manifests: join(dir, "registry"), manifest_public_key: join(dir, "pub.pem"), data: { dir: join(dir, "d2") }, log: () => {} })).rejects.toThrow(/not signed by the trusted key/); + }); + + it("loads from a URL through the egress rules", async () => { + const dir = tmp(); + const reg = new ManifestRegistry(join(dir, "registry")); + for (const m of MANIFESTS) reg.publish(m, { by: "ci", status: "approved" }); + const server = createServer((req, res) => { + try { + res.writeHead(200).end(readFileSync(join(dir, "registry", decodeURIComponent((req.url ?? "").slice(1))), "utf8")); + } catch { + res.writeHead(404).end(); + } + }); + await new Promise((r) => server.listen(0, "127.0.0.1", r)); + const url = `http://127.0.0.1:${(server.address() as AddressInfo).port}`; + try { + const h = await openEngine({ manifests: url, data: { dir: join(dir, "data") }, log: () => {} }); + open.push(h); + expect(h.registry.size).toBe(3); + await expect(openEngine({ manifests: url, egress: { allow: ["*.example.com"] }, data: { dir: join(dir, "d2") }, log: () => {} })).rejects.toThrow(/allowlist/); + } finally { + server.close(); + } + }); + + it("starts with no manifests only when asked (the observe-first gateway)", async () => { + const dir = tmp(); + await expect(openEngine({ manifests: "", data: { dir } })).rejects.toThrow(/no manifests/); + const h = await openEngine({ manifests: "", allow_empty_manifests: true, data: { dir }, log: () => {} }); + open.push(h); + expect(h.registry.size).toBe(0); + }); +}); + +describe("hot reload", () => { + it("picks up changed manifests and policy, records it, and keeps the old ones when the new ones are broken", async () => { + const dir = folder(); + const manifests = join(dir, "manifests"); + mkdirSync(manifests); + writeFileSync(join(manifests, "tools.json"), readFileSync(join(dir, "tools.json"))); + const policy = join(dir, "policy.yaml"); + writeFileSync(policy, "version: 1\nrules: []\n"); + const l = log(); + const h = await openEngine({ manifests, policy, data: { dir: join(dir, "data") }, log: l.fn }); + open.push(h); + const run = h.reversi.startRun({ actor: "a", scope: scope() }); + expect((await run.call({ tool: "orders.cancel", reason: "x", execute: () => 1 })).status).toBe("executed"); + expect(await h.reload()).toMatchObject({ changed: false, tools: 3 }); + + writeFileSync(policy, "version: 1\nrules:\n - id: freeze-cancel\n match: { tool: orders.cancel }\n action: deny\n"); + expect(await h.reload("alice")).toMatchObject({ changed: true }); + expect((await run.call({ tool: "orders.cancel", reason: "x", execute: () => 1 })).status).toBe("blocked"); + const ev = h.reversi.ledger.records().filter((r) => r.event === "reloaded"); + expect(ev).toHaveLength(1); + expect(ev[0]).toMatchObject({ actor: "alice", run_id: "-" }); + expect(ev[0]?.detail).toMatch(/^3 manifests; policy [0-9a-f]{8}$/); + + // a broken edit changes nothing and says why + writeFileSync(policy, "rules: [{ id: a, action: nope }]"); + const bad = await h.reload(); + expect(bad.changed).toBe(false); + expect(bad.error).toContain("action must be"); + expect((await run.call({ tool: "orders.cancel", reason: "x", execute: () => 1 })).status).toBe("blocked"); // still the previous policy + expect(l.lines.some((x) => x.startsWith("error: reload failed, keeping the previous"))).toBe(true); + + // manifests too + writeFileSync(join(manifests, "tools.json"), JSON.stringify([...MANIFESTS, { tool: "extra", effect: "read_only", permission: "p" }])); + writeFileSync(policy, "version: 1\nrules: []\n"); + expect(await h.reload()).toMatchObject({ changed: true, tools: 4 }); + expect(h.reversi.registry.has("extra")).toBe(true); + }); + + it("watch() reloads by itself when a file changes", async () => { + const dir = folder(); + const h = await openEngine({ manifests: dir, data: { dir: join(dir, "data") }, log: () => {} }); + open.push(h); + h.watch(30); + await new Promise((r) => setTimeout(r, 80)); + writeFileSync(join(dir, "tools.json"), JSON.stringify([...MANIFESTS, { tool: "watched", effect: "read_only", permission: "p" }])); + for (let i = 0; i < 50 && !h.reversi.registry.has("watched"); i++) await new Promise((r) => setTimeout(r, 40)); + expect(h.reversi.registry.has("watched")).toBe(true); + }); +}); + +describe("openEngine: wiring", () => { + it("sends approval notifications through configured channels, and capture and anchors work", async () => { + const dir = folder(); + const posts: any[] = []; + const server = createServer((req, res) => { + let b = ""; + req.on("data", (c) => (b += c)); + req.on("end", () => { + posts.push({ url: req.url, body: JSON.parse(b) }); + res.end("ok"); + }); + }); + await new Promise((r) => server.listen(0, "127.0.0.1", r)); + const base = `http://127.0.0.1:${(server.address() as AddressInfo).port}`; + try { + const key = generateSigningKey(); + writeFileSync(join(dir, "sign.key"), key.private_pem); + const h = await openEngine({ + manifests: dir, + data: { dir: join(dir, "data"), store: "sqlite" }, + console_url: "https://r.example.com", + signing_key: join(dir, "sign.key"), + capture: { enabled: true }, + notifications: { channels: [{ type: "webhook", name: "ops", url: `${base}/ops` }], routes: [{ events: ["approval_requested", "approval_decided", "rollback_failed"], channels: ["ops"] }] }, + anchor: { sinks: [{ type: "file", path: join(dir, "anchors.jsonl") }], interval_ms: 3_600_000 }, + log: () => {}, + }); + open.push(h); + h.start(); + const run = h.reversi.startRun({ actor: "agent", scope: scope() }); + await run.call({ tool: "mail.send", args: { to: "ada@example.com" }, reason: "x", execute: () => 1 }); // needs approval + const pending = h.approvals.list({ status: "pending" })[0]!; + h.approvals.approve(pending.id, "alice"); + await new Promise((r) => setTimeout(r, 100)); + expect(posts.map((p) => p.body.event)).toEqual(["approval_requested", "approval_decided"]); + expect(posts[0].body.link).toBe("https://r.example.com/console/approvals"); + expect(JSON.stringify(posts)).not.toContain("ada@example.com"); + expect(h.capture?.list({ run_id: run.run_id })[0]).toMatchObject({ tool: "mail.send", args: { to: "[PII:email]" }, decision: "block" }); + await new Promise((r) => setTimeout(r, 100)); + expect(readAnchors(join(dir, "anchors.jsonl"))[0]).toMatchObject({ format: "reversi-anchor", key_id: expect.any(String) }); + expect(h.signing_key?.public_pem).toBe(key.public_pem); + } finally { + server.close(); + } + }); +}); + +describe("project config file", () => { + it("is found by walking up, expands environment variables, resolves paths against the file, and rejects mistakes", () => { + const root = tmp(); + mkdirSync(join(root, "a", "b"), { recursive: true }); + writeFileSync( + join(root, "reversi.config.yaml"), + `manifests: ./manifests +policy: https://example.com/policy.yaml +data: { dir: ./data, store: sqlite } +approval: { ttl: 30m, webhook: "\${HOOK}" } +capture: { enabled: true, ttl: 2d } +notifications: + channels: [{ type: webhook, name: w, url: "\${HOOK}" }] +anchor: { sinks: [{ type: file, path: ./anchors.jsonl }], interval: 1h } +console: { mode: open, role: operator } +`, + ); + expect(findProjectConfig(join(root, "a", "b"), undefined, {})).toBe(join(root, "reversi.config.yaml")); + expect(findProjectConfig(tmp(), undefined, {})).toBeUndefined(); + const c = loadProjectConfig(join(root, "reversi.config.yaml"), { HOOK: "https://hooks.example.com/x" }); + expect(c).toMatchObject({ + manifests: join(root, "manifests"), + policy: "https://example.com/policy.yaml", + data: { dir: join(root, "data"), store: "sqlite" }, + approval: { ttl_ms: 1_800_000, webhook: "https://hooks.example.com/x" }, + capture: { enabled: true, ttl_ms: 172_800_000 }, + anchor: { sinks: [{ type: "file", path: join(root, "anchors.jsonl") }], interval_ms: 3_600_000 }, + console: { mode: "open", role: "operator" }, + }); + expect(c.notifications?.routes[0]).toMatchObject({ events: ["*"], channels: ["w"] }); // no routes given: everything goes to every channel + expect(() => loadProjectConfig(join(root, "reversi.config.yaml"), {})).toThrow(/environment variable HOOK is not set/); + }); + + it("reports every mistake", () => { + const f = join(tmp(), "reversi.config.yaml"); + writeFileSync(f, "mystery: 1\ndata: { store: oracle }\nconsole: { mode: open, role: admin }\ncapture: { ttl: soon }\nanchor: { sinks: [] }\n"); + try { + loadProjectConfig(f, {}); + expect.unreachable(); + } catch (e) { + expect(e).toBeInstanceOf(ProjectConfigError); + const m = (e as ProjectConfigError).errors.join("\n"); + expect(m).toContain("unknown setting `mystery`"); + expect(m).toContain("data.store must be files, sqlite or postgres"); + expect(m).toContain("never admin"); + expect(m).toContain("capture.ttl must be a duration"); + expect(m).toContain("anchor needs a list of sinks"); + } + expect(() => findProjectConfig(tmp(), "/nonexistent/reversi.config.yaml", {})).toThrow(/not found/); + }); + + it("the starter config from `reversi init` is itself valid", () => { + const f = join(tmp(), "reversi.config.yaml"); + writeFileSync(f, starterConfig({ store: "sqlite" })); + expect(loadProjectConfig(f, {})).toMatchObject({ data: { store: "sqlite" }, environment: "dev" }); + }); +}); diff --git a/packages/reversi/test/evidence-ops.test.ts b/packages/reversi/test/evidence-ops.test.ts new file mode 100644 index 0000000..84ddb9f --- /dev/null +++ b/packages/reversi/test/evidence-ops.test.ts @@ -0,0 +1,312 @@ +import { createServer } from "node:http"; +import { mkdtempSync, readFileSync, rmSync, writeFileSync, existsSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import type { AddressInfo } from "node:net"; +import { afterEach, describe, expect, it } from "vitest"; +import { + Approvals, + CaptureStore, + EgressError, + Ledger, + LegalHolds, + MemoryKvStore, + Registry, + Reversi, + SealedScope, + anchorLedger, + applyRetention, + checkUrl, + generateSigningKey, + guardedFetch, + isMetadataAddress, + isPrivateAddress, + parsePolicy, + planRetention, + readAnchors, + renderPdf, + renderReportPdf, + searchRecords, + searchRuns, + verifyAnchor, + verifyBundle, + type EvidenceBundle, +} from "../src/index.js"; + +const dirs: string[] = []; +afterEach(() => { + while (dirs.length) rmSync(dirs.pop() as string, { recursive: true, force: true }); +}); +const tmp = () => { + const d = mkdtempSync(join(tmpdir(), "rv-ev-")); + dirs.push(d); + return d; +}; + +const MANIFESTS = [ + { tool: "orders.create", effect: "reversible", permission: "p", compensation: { tool: "orders.cancel", argument_map: { id: "$result.id" } } }, + { tool: "orders.cancel", effect: "idempotent", permission: "p" }, + { tool: "mail.send", effect: "irreversible", permission: "p", audit: { evidence_export: "required", retention_days: 400 } }, + { tool: "read", effect: "read_only", permission: "p" }, +]; + +async function world(clock: { t: number }) { + const kv = new MemoryKvStore(); + const approvals = new Approvals(); + const capture = new CaptureStore(kv, { enabled: true, ttl_ms: 800 * 86_400_000 }, () => clock.t); + const rv = new Reversi({ registry: Registry.from(MANIFESTS), ledger: new Ledger(), approver: approvals.asApprover(), kv, capture, now: () => clock.t }); + return { rv, kv, approvals, capture, holds: new LegalHolds(kv, () => clock.t) }; +} + +describe("search", () => { + it("filters runs and records, with pagination", async () => { + const clock = { t: Date.parse("2026-10-01T00:00:00Z") }; + const { rv } = await world(clock); + const mk = async (id: string, actor: string, tenant: string | undefined, tool: string, fail = false) => { + const run = rv.startRun({ run_id: id, actor, tenant, scope: SealedScope.seal(["p"]) }); + await run.call({ tool, reason: "x", execute: () => (fail ? Promise.reject(new Error("boom")) : { id: 1 }) }).catch(() => {}); + clock.t += 1000; + return run; + }; + const a = await mk("run-a", "alice", "acme", "orders.create"); + await a.rollback({ compensate: () => {} }); + await mk("run-b", "bob", "globex", "read"); + await mk("run-c", "alice", "acme", "mail.send"); // blocked: needs approval; no steps + const d = rv.startRun({ run_id: "run-d", actor: "bob", scope: SealedScope.seal(["p"]) }); + const bg = await d.begin({ tool: "orders.create", reason: "x" }); + if (bg.decision === "allow") d.complete(bg.step, { status: "uncertain", error: "timeout" }); + + const status = (id: string) => searchRuns(rv).runs.find((r) => r.run_id === id)?.status; + expect(status("run-a")).toBe("rolled_back"); + expect(status("run-b")).toBe("complete"); + expect(status("run-c")).toBe("blocked_only"); + expect(status("run-d")).toBe("needs_attention"); + expect(searchRuns(rv, { actor: "alice" }).runs.map((r) => r.run_id)).toEqual(["run-a", "run-c"]); + expect(searchRuns(rv, { tenant: "globex" }).runs.map((r) => r.run_id)).toEqual(["run-b"]); + expect(searchRuns(rv, { tool: "orders.create" }).runs.map((r) => r.run_id)).toEqual(["run-a", "run-d"]); + expect(searchRuns(rv, { status: "needs_attention" }).runs.map((r) => r.run_id)).toEqual(["run-d"]); + expect(searchRuns(rv, { since_ms: Date.parse("2026-10-01T00:00:02Z") }).runs.map((r) => r.run_id)).toEqual(["run-c", "run-d"].filter((x) => rv.openRun(x).state().created_ms >= Date.parse("2026-10-01T00:00:02Z"))); + const page1 = searchRuns(rv, { limit: 2 }); + expect(page1.runs).toHaveLength(2); + expect(searchRuns(rv, { limit: 2, after: page1.next }).runs.map((r) => r.run_id)).toEqual(["run-c", "run-d"]); + + expect(searchRecords(rv, { event: "compensated" }).records).toHaveLength(1); + expect(searchRecords(rv, { run_id: "run-c", event: ["blocked"] }).records[0]?.detail).toContain("approval required"); + expect(searchRecords(rv, { text: "TIMEOUT" }).records).toHaveLength(1); + const p = searchRecords(rv, { limit: 3 }); + expect(p.records).toHaveLength(3); + expect(searchRecords(rv, { limit: 3, after_seq: p.next_seq }).records[0]?.seq).toBe((p.next_seq ?? 0) + 1); + }); +}); + +describe("retention and legal hold", () => { + const DAY = 86_400_000; + it("purges only runs past their retention, never held ones, and records it in the ledger", async () => { + const clock = { t: Date.parse("2026-01-01T00:00:00Z") }; + const { rv, approvals, capture, holds } = await world(clock); + for (const id of ["old", "held", "fresh"]) { + const run = rv.startRun({ run_id: id, actor: "a", tenant: id === "held" ? "acme" : undefined, scope: SealedScope.seal(["p"]) }); + await run.call({ tool: "read", args: { q: id }, reason: "x", execute: () => 1 }); + clock.t += id === "old" ? 60 * DAY : id === "held" ? 40 * DAY : 0; + } + holds.add({ tenant: "acme" }, "case 2026-77 litigation", "legal"); + const cfg = { default_days: 30, rules: [] }; + const now = clock.t; + const plan = planRetention(rv, cfg, holds, now); + expect(plan.map((p) => [p.run_id, p.due, !!p.held])).toEqual([["old", true, false], ["held", true, true], ["fresh", false, false]]); + + const dry = applyRetention(rv, cfg, { holds, approvals, capture, dry_run: true, now }); + expect(dry).toMatchObject({ dry_run: true, purged: ["old"] }); + expect(rv.runIds()).toHaveLength(3); // dry run changed nothing + + const rep = applyRetention(rv, cfg, { holds, approvals, capture, now }); + expect(rep.purged).toEqual(["old"]); + expect(rep.skipped).toEqual([{ run_id: "held", reason: expect.stringContaining("legal hold") }]); + expect(rv.runIds().sort()).toEqual(["fresh", "held"]); + expect(capture.list({ run_id: "old" })).toEqual([]); + expect(capture.list({ run_id: "held" })).toHaveLength(1); + const purged = rv.ledger.records().filter((r) => r.event === "purged"); + expect(purged).toHaveLength(1); + expect(purged[0]).toMatchObject({ run_id: "old", reason: "retention: older than 30 days" }); + expect(rv.ledger.verify().ok).toBe(true); // the ledger stays intact and keeps every record + holds.release(holds.list()[0]!.id); + expect(applyRetention(rv, cfg, { holds, approvals, capture, now }).purged).toEqual(["held"]); + }); + + it("a manifest that requires evidence export keeps the run until an archive and key are given, then archives a verifiable bundle", async () => { + const clock = { t: Date.parse("2026-01-01T00:00:00Z") }; + const { rv, approvals, capture } = await world(clock); + const run = rv.startRun({ run_id: "m1", actor: "a", scope: SealedScope.seal(["p"]) }); + const asked = await run.begin({ tool: "mail.send", reason: "x" }); + expect(asked.decision).toBe("block"); + // an approved, executed irreversible call + const approvals2 = approvals; + approvals2.add("m1", "call", "mail.send", 1, "ops"); + await run.call({ tool: "mail.send", reason: "x", execute: () => "sent" }); + clock.t += 500 * DAY; // past mail.send's 400-day retention + const cfg = { default_days: 30, rules: [] }; + expect(applyRetention(rv, cfg, { approvals, capture, now: clock.t }).skipped[0]?.reason).toContain("requires an evidence bundle"); + expect(rv.runIds()).toEqual(["m1"]); + const dir = tmp(); + const key = generateSigningKey(); + const rep = applyRetention(rv, cfg, { approvals, capture, archive_dir: join(dir, "archive"), key, now: clock.t }); + expect(rep.purged).toEqual(["m1"]); + const bundle = JSON.parse(readFileSync(rep.exported[0]!.file, "utf8")) as EvidenceBundle; + expect(verifyBundle(bundle, { trusted_public_pem: key.public_pem })).toMatchObject({ ok: true, chain: "per-record" }); + expect(rv.ledger.records().at(-1)?.detail).toContain("evidence bundle archived"); + }); + + it("holds need a scope and a reason", () => { + const h = new LegalHolds(new MemoryKvStore()); + expect(() => h.add({}, "reason here", "x")).toThrow(/scope/); + expect(() => h.add({ run_id: "r" }, "hi", "x")).toThrow(/reason/); + const hold = h.add({ tool: "mail.send" }, "matter 12345", "legal"); + expect(h.list({ all: true })).toHaveLength(1); + expect(h.release(hold.id)).toBe(true); + expect(h.release(hold.id)).toBe(false); + expect(h.list()).toHaveLength(0); + }); + + it("retention config is validated and lives in the policy file", () => { + expect(parsePolicy({ retention: { default_days: 90, rules: [{ match: { tool: "pay.*" }, days: 400 }] } }).retention?.default_days).toBe(90); + expect(() => parsePolicy({ retention: { default_days: 0 } })).toThrow(/positive integer/); + expect(() => parsePolicy({ retention: { default_days: 5, rules: [{ match: { colour: "x" }, days: 1 }] } })).toThrow(/unknown match field/); + }); +}); + +describe("egress control", () => { + it("classifies addresses", () => { + for (const ip of ["10.1.2.3", "127.0.0.1", "192.168.1.1", "172.16.0.1", "172.31.255.255", "169.254.1.1", "100.64.0.1", "::1", "fd12::1", "fe80::1", "::ffff:10.0.0.1"]) expect(isPrivateAddress(ip), ip).toBe(true); + for (const ip of ["8.8.8.8", "172.32.0.1", "1.1.1.1", "2606:4700::1111"]) expect(isPrivateAddress(ip), ip).toBe(false); + expect(isMetadataAddress("169.254.169.254")).toBe(true); + expect(isMetadataAddress("::ffff:169.254.169.254")).toBe(true); + expect(isMetadataAddress("8.8.8.8")).toBe(false); + }); + + it("applies the allowlist, the private block and the always-on metadata block", async () => { + await expect(checkUrl(undefined, "http://169.254.169.254/latest/meta-data")).rejects.toThrow(/metadata/); + await expect(checkUrl(undefined, "ftp://example.com")).rejects.toThrow(/only http/); + await expect(checkUrl({ block_private: true }, "http://127.0.0.1:9000/x")).rejects.toThrow(/private or loopback/); + await expect(checkUrl({ block_private: true }, "http://[::1]/x")).rejects.toThrow(EgressError); + expect((await checkUrl(undefined, "http://127.0.0.1:9000/x")).port).toBe("9000"); // fine for local development + await expect(checkUrl({ allow: ["*.eu.example.com"], region: "eu" }, "https://hooks.us.example.com/x")).rejects.toThrow(/not in the egress allowlist \(region eu\)/); + await expect(checkUrl({ allow: ["*.eu.example.com"] }, "http://127.0.0.1/x")).rejects.toThrow(/allowlist/); + expect((await checkUrl({ allow: ["127.0.0.1"] }, "http://127.0.0.1:80/x")).hostname).toBe("127.0.0.1"); + }); + + it("guardedFetch checks redirects too", async () => { + const seen: string[] = []; + const f = (async (url: URL | string) => { + seen.push(String(url)); + return String(url).includes("start") ? new Response(null, { status: 302, headers: { location: "http://169.254.169.254/steal" } }) : new Response("ok"); + }) as unknown as typeof fetch; + await expect(guardedFetch(undefined, f)("http://127.0.0.1/start")).rejects.toThrow(/metadata/); + expect(seen).toEqual(["http://127.0.0.1/start"]); + const ok = await guardedFetch(undefined, (async () => new Response("fine")) as unknown as typeof fetch)("http://127.0.0.1/a"); + expect(await ok.text()).toBe("fine"); + }); +}); + +describe("external anchoring", () => { + async function ledgerWith(n: number) { + const l = new Ledger(); + for (let i = 0; i < n; i++) l.append({ ts_ms: i, run_id: "r", actor: "a", tool: "t", reason: `e${i}`, event: "run_started" }); + return l; + } + + it("writes signed anchors to a file and a webhook, and proves later that history is unchanged", async () => { + const dir = tmp(); + const key = generateSigningKey(); + const l = await ledgerWith(3); + const posts: Array<{ body: string; sig?: string }> = []; + const server = createServer((req, res) => { + let b = ""; + req.on("data", (c) => (b += c)); + req.on("end", () => { + posts.push({ body: b, sig: req.headers["x-reversi-signature"] as string }); + res.end("ok"); + }); + }); + await new Promise((r) => server.listen(0, "127.0.0.1", r)); + const url = `http://127.0.0.1:${(server.address() as AddressInfo).port}/anchor`; + try { + const file = join(dir, "anchors.jsonl"); + const out = await anchorLedger(l, [{ type: "file", path: file }, { type: "webhook", url, secret: "s3" }], { key, egress: { region: "eu-west-1" }, now: 5 }); + expect(out.errors).toEqual([]); + expect(out.delivered).toHaveLength(2); + expect(out.anchor).toMatchObject({ records: 3, head: l.head(), region: "eu-west-1" }); + expect(posts[0]?.sig).toMatch(/^sha256=[0-9a-f]{64}$/); + const stored = readAnchors(file)[0]!; + expect(verifyAnchor(l, stored, key.public_pem)).toEqual({ ok: true, records: 3, signed: true }); + + // the ledger grows: the old anchor still verifies + l.append({ ts_ms: 9, run_id: "r", actor: "a", tool: "t", reason: "more", event: "run_started" }); + expect(verifyAnchor(l, stored)).toMatchObject({ ok: true }); + // truncation and rewrites are caught + const truncated = await ledgerWith(2); + expect(verifyAnchor(truncated, stored)).toMatchObject({ ok: false, error: expect.stringContaining("truncated") }); + const rewritten = new Ledger(); + for (let i = 0; i < 3; i++) rewritten.append({ ts_ms: i, run_id: "r", actor: "a", tool: "t", reason: `different${i}`, event: "run_started" }); + expect(verifyAnchor(rewritten, stored)).toMatchObject({ ok: false, error: expect.stringContaining("rewritten") }); + // a forged signature is rejected when a trusted key is required + expect(verifyAnchor(l, { ...stored, head: stored.head }, generateSigningKey().public_pem)).toMatchObject({ ok: false, error: expect.stringContaining("signature") }); + expect(verifyAnchor(l, { ...stored, signature: undefined }, key.public_pem)).toMatchObject({ ok: false, error: expect.stringContaining("unsigned") }); + } finally { + server.close(); + } + }); + + it("reports sink failures without losing the anchor, and honours egress rules", async () => { + const l = await ledgerWith(1); + const out = await anchorLedger(l, [{ type: "webhook", url: "http://169.254.169.254/x" }, { type: "file", path: "/nonexistent-dir/a.jsonl" }]); + expect(out.delivered).toEqual([]); + expect(out.errors).toHaveLength(2); + expect(out.errors[0]).toContain("metadata"); + expect(existsSync("/nonexistent-dir")).toBe(false); + }); +}); + +describe("PDF report", () => { + function parsePdf(buf: Buffer) { + const text = buf.toString("latin1"); + const startxref = Number(/startxref\n(\d+)\n%%EOF\n$/.exec(text)?.[1]); + expect(text.slice(startxref, startxref + 4)).toBe("xref"); + const entries = [...text.slice(startxref).matchAll(/(\d{10}) \d{5} n /g)].map((m) => Number(m[1])); + entries.forEach((off, i) => expect(text.slice(off, off + `${i + 1} 0 obj`.length), `object ${i + 1}`).toBe(`${i + 1} 0 obj`)); + const pages = [...text.matchAll(/\/Type \/Page \//g)].length; + const kids = /\/Kids \[([^\]]*)\]/.exec(text)?.[1]?.match(/\d+ 0 R/g) ?? []; + for (const k of kids) { + const n = Number(k.split(" ")[0]); + expect(text.slice(entries[n - 1]).split("endobj")[0]).toContain("/Type /Page "); + } + // stream lengths are right + for (const m of text.matchAll(/\/Length (\d+) >>\nstream\n([\s\S]*?)\nendstream/g)) expect(Buffer.byteLength(m[2] as string, "latin1")).toBe(Number(m[1])); + return { text, pages, kids: kids.length, entries: entries.length }; + } + + it("writes a structurally valid multi-page PDF with wrapped text and escaped characters", () => { + const lines = Array.from({ length: 200 }, (_, i) => `line ${i} (parens) back\\slash ünïcode ✓ ${"x".repeat(i % 3 === 0 ? 200 : 5)}`); + const pdf = renderPdf(["# Title", ...lines], "Test (doc)"); + expect(pdf.subarray(0, 8).toString()).toBe("%PDF-1.4"); + const info = parsePdf(pdf); + expect(info.pages).toBeGreaterThan(2); + expect(info.kids).toBe(info.pages); + expect(info.text).toContain("\\(parens\\)"); + expect(info.text).toContain("back\\\\slash"); + expect(info.text).toContain("\\(doc\\)"); + expect(parsePdf(renderPdf([], "empty")).pages).toBe(1); + }); + + it("renders a run report", async () => { + const clock = { t: Date.parse("2026-10-01T00:00:00Z") }; + const { rv } = await world(clock); + const run = rv.startRun({ run_id: "rep-1", actor: "agent", scope: SealedScope.seal(["p"]) }); + await run.call({ tool: "orders.create", reason: "customer asked", execute: () => ({ id: 1 }) }); + const pdf = renderReportPdf({ run: run.state(), records: rv.ledger.forRun("rep-1"), plan: run.plan(), ledger_head: rv.ledger.head(), region: "eu-west-1", generated_ms: clock.t }); + const { text } = parsePdf(pdf); + expect(text).toContain("Reversi report: run rep-1"); + expect(text).toContain("orders.create"); + expect(text).toContain("Data region: eu-west-1"); + expect(text).toContain("customer asked"); + }); +}); diff --git a/packages/reversi/test/governance.test.ts b/packages/reversi/test/governance.test.ts new file mode 100644 index 0000000..0b1a121 --- /dev/null +++ b/packages/reversi/test/governance.test.ts @@ -0,0 +1,336 @@ +import { describe, expect, it } from "vitest"; +import { + ApprovalError, + Approvals, + Delegations, + Ledger, + MemoryKvStore, + Policy, + PolicyError, + Registry, + Reversi, + SealedScope, + inWindow, + mergeRequirements, + parsePolicy, + scoreRisk, +} from "../src/index.js"; + +const base = { tool: "pay", effect: "compensable" as const, permission: "p", actor: "agent", run_id: "r1", environment: "prod", args: {} }; +const at = (iso: string) => () => Date.parse(iso); + +describe("risk scoring", () => { + it("adds transparent factors and caps at 100", () => { + const r = scoreRisk({ amount: { path: "$args.amount", bands: [{ over: 100, points: 10 }, { over: 1000, points: 25 }] }, actors: { newbie: 15 } }, { effect: "compensable", blast_radius: "high", data_class: "financial", environment: "prod", actor: "newbie", args: { amount: 5000 } }); + expect(r.score).toBe(100); // 30+20+20+10+25+15 = 120, capped + expect(r.factors).toEqual(["effect compensable +30", "blast radius high +20", "data class financial +20", "environment prod +10", "amount over 1000 +25", "actor newbie +15"]); + expect(scoreRisk(undefined, { effect: "irreversible", blast_radius: "critical", data_class: "secret", environment: "prod", actor: "a", args: {} }).score).toBe(100); + expect(scoreRisk(undefined, { effect: null, environment: "dev", actor: "a", args: {} })).toMatchObject({ score: 40, factors: ["no manifest +40"] }); + }); + + it("thresholds turn a score into an approval or a denial", () => { + const p = Policy.from({ risk: { thresholds: { approve: 40, deny: 90 } } }); + const d = (over: object) => p.evaluate({ ...base, ...over }); + expect(d({ effect: "read_only", environment: "dev" }).action).toBe("allow"); + expect(d({ effect: "compensable", blast_radius: "high" }).action).toBe("require_approval"); // 30+20+10 = 60 + expect(d({ effect: "irreversible", blast_radius: "critical", data_class: "secret" }).action).toBe("deny"); + expect(d({ effect: "read_only", environment: "dev" }).risk?.score).toBe(0); + }); + + it("rejects bad risk config", () => { + expect(() => parsePolicy({ risk: { thresholds: { approve: 90, deny: 50 } } })).toThrow(/approve must not be above deny/); + expect(() => parsePolicy({ risk: { weights: { effect: { x: "a" } } } })).toThrow(PolicyError); + }); +}); + +describe("time windows and freezes", () => { + it("evaluates days, hours, wrap-around and time zones", () => { + const biz = { days: ["mon", "tue", "wed", "thu", "fri"] as never, between: "09:00-17:00" }; + expect(inWindow(biz, Date.parse("2026-10-14T10:00:00Z"))).toBe(true); // Wednesday + expect(inWindow(biz, Date.parse("2026-10-14T18:00:00Z"))).toBe(false); + expect(inWindow(biz, Date.parse("2026-10-17T10:00:00Z"))).toBe(false); // Saturday + expect(inWindow({ between: "22:00-06:00" }, Date.parse("2026-10-14T23:30:00Z"))).toBe(true); + expect(inWindow({ between: "22:00-06:00" }, Date.parse("2026-10-14T05:59:00Z"))).toBe(true); + expect(inWindow({ between: "22:00-06:00" }, Date.parse("2026-10-14T12:00:00Z"))).toBe(false); + expect(inWindow({ between: "09:00-17:00", timezone: "America/New_York" }, Date.parse("2026-10-14T14:00:00Z"))).toBe(true); // 10:00 in New York + expect(inWindow({ between: "09:00-17:00", timezone: "America/New_York", outside: true }, Date.parse("2026-10-14T14:00:00Z"))).toBe(false); + }); + + it("a rule with a window only fires inside it", () => { + const doc = { rules: [{ id: "after-hours", match: { tool: "pay" }, window: { between: "09:00-17:00", outside: true }, action: "require_approval" }] }; + expect(Policy.from(doc, at("2026-10-14T10:00:00Z")).evaluate(base).action).toBe("allow"); + expect(Policy.from(doc, at("2026-10-14T20:00:00Z")).evaluate(base).action).toBe("require_approval"); + }); + + it("a freeze denies non-read calls for its period, with its reason", () => { + const doc = { freezes: [{ id: "q4-freeze", from: "2026-12-15T00:00:00Z", to: "2027-01-05T00:00:00Z", reason: "holiday change freeze" }] }; + const during = Policy.from(doc, at("2026-12-24T12:00:00Z")); + expect(during.evaluate(base)).toMatchObject({ action: "deny", deny: [expect.stringContaining("holiday change freeze")] }); + expect(during.evaluate({ ...base, effect: "read_only" }).action).toBe("allow"); + expect(Policy.from(doc, at("2027-01-06T00:00:00Z")).evaluate(base).action).toBe("allow"); + expect(() => parsePolicy({ freezes: [{ id: "x", from: "2026-01-02", to: "2026-01-01", reason: "r" }] })).toThrow(/after from/); + expect(() => parsePolicy({ rules: [{ id: "a", action: "deny", window: { between: "9-5" } }] })).toThrow(/between/); + }); +}); + +describe("budgets and shared limits", () => { + it("a sum budget refuses the call that would exceed it, warns near the end, and is shared through the KV store", () => { + const kv = new MemoryKvStore(); + let t = 1_000_000; + const doc = { budgets: [{ id: "daily-spend", match: { tool: "pay" }, metric: "sum", path: "$args.amount", max: 1000, window_seconds: 86400, per: "actor", warn_at: 0.8 }] }; + const a = new Policy(parsePolicy(doc), () => t, kv); + const b = new Policy(parsePolicy(doc), () => t, kv); // another process + const call = (p: Policy, amount: number) => { + const i = { ...base, args: { amount } }; + const d = p.evaluate(i); + if (d.action !== "deny") p.record(i); + return d; + }; + expect(call(a, 500).action).toBe("allow"); + const second = call(b, 350); // 850 of 1000 now: over the 80% warning + expect(second.action).toBe("allow"); + expect(second.warnings[0]).toBe("budget `daily-spend` at 85% (850 of 1000)"); + const third = call(a, 300); + expect(third).toMatchObject({ action: "deny", deny: [expect.stringContaining("850 of 1000 used")] }); + t += 86_400_001; // the window slides past the earlier spending + expect(call(b, 300).action).toBe("allow"); + }); + + it("a budget can ask for approval instead of refusing, and is per actor", () => { + const p = Policy.from({ budgets: [{ id: "calls", match: {}, metric: "calls", max: 1, window_seconds: 60, per: "actor", action: "require_approval" }] }); + p.record(base); + expect(p.evaluate(base).action).toBe("require_approval"); + expect(p.evaluate({ ...base, actor: "other" }).action).toBe("allow"); + }); + + it("rate limits are shared the same way", () => { + const kv = new MemoryKvStore(); + const doc = { limits: [{ id: "burst", match: {}, max_calls: 1, window_seconds: 60, per: "global" }] }; + const a = new Policy(parsePolicy(doc), undefined, kv); + const b = new Policy(parsePolicy(doc), undefined, kv); + a.record(base); + expect(b.evaluate(base).deny[0]).toContain("burst"); + }); + + it("validates budgets", () => { + expect(() => parsePolicy({ budgets: [{ id: "x", metric: "sum", max: 5, window_seconds: 60 }] })).toThrow(/needs path/); + expect(() => parsePolicy({ budgets: [{ id: "x", metric: "calls", max: -1, window_seconds: 0, per: "world" }] })).toThrow(PolicyError); + }); +}); + +describe("tenants and match fields", () => { + it("matches tenant, data class and blast radius", () => { + const p = Policy.from({ rules: [{ id: "acme-pii", match: { tenant: "acme", data_class: "pii", blast_radius: ["high", "critical"] }, action: "deny" }] }); + expect(p.evaluate({ ...base, tenant: "acme", data_class: "pii", blast_radius: "high" }).action).toBe("deny"); + expect(p.evaluate({ ...base, tenant: "other", data_class: "pii", blast_radius: "high" }).action).toBe("allow"); + expect(p.evaluate({ ...base, data_class: "pii", blast_radius: "high" }).action).toBe("allow"); // no tenant known: does not match + }); +}); + +describe("approval routing, quorum, delegation, escalation, break-glass", () => { + const fin = { roles: ["approver"], groups: ["finance"] }; + const ops = { roles: ["approver"], groups: ["ops"] }; + function setup(opts: { now?: () => number } = {}) { + const kv = new MemoryKvStore(); + const delegations = new Delegations(kv, opts.now); + const events: string[] = []; + const approvals = new Approvals(undefined, { delegations, now: opts.now, on_decision: (r, d) => events.push(`${d}:${r.decided_by}`), on_escalate: (r) => events.push(`escalated:${r.required_roles?.join("|")}`) }); + const asked = (extra: object = {}) => + approvals.asApprover()({ run_id: "r1", step: null, tool: "pay", reason: "x", kind: "call", args_hash: "h", requested_by: "agent-1", required_roles: ["finance"], quorum: 1, ...extra }); + return { approvals, delegations, events, asked }; + } + + it("only a listed role or group may approve; the requester never may", () => { + const { approvals, asked } = setup(); + asked(); + const id = approvals.list()[0]!.id; + expect(() => approvals.approve(id, "olga", undefined, ops)).toThrow(/needs an approver from: finance/); + expect(() => approvals.approve(id, "agent-1", undefined, { roles: ["approver"], groups: ["finance"] })).toThrow(/your own run/); + expect(approvals.approve(id, "fiona", "ok", fin)).toMatchObject({ status: "approved", decided_by: "fiona" }); + expect(approvals.get(id)?.decisions?.[0]).toMatchObject({ by: "fiona", basis: "group:finance" }); + }); + + it("a quorum needs distinct approvers and stays pending until it is met", () => { + const { approvals, asked, events } = setup(); + asked({ quorum: 2 }); + const id = approvals.list()[0]!.id; + expect(approvals.approve(id, "fiona", undefined, fin).status).toBe("pending"); + expect(() => approvals.approve(id, "fiona", undefined, fin)).toThrow(/already decided/); + const done = approvals.approve(id, "fred", undefined, fin); + expect(done).toMatchObject({ status: "approved", decided_by: "fiona, fred" }); + expect(events).toEqual(["approved:fiona, fred"]); // only the final decision reaches the ledger hook + expect(approvals.asApprover()({ run_id: "r1", step: null, tool: "pay", reason: "x", kind: "call", args_hash: "h" })).toBe(true); + }); + + it("one denial ends it", () => { + const { approvals, asked } = setup(); + asked({ quorum: 2 }); + const id = approvals.list()[0]!.id; + approvals.approve(id, "fiona", undefined, fin); + expect(approvals.deny(id, "fred", "no", fin).status).toBe("denied"); + }); + + it("two approvers racing on the same vote cannot lose one", () => { + const { approvals, asked } = setup(); + asked({ quorum: 2 }); + const id = approvals.list()[0]!.id; + const stale = approvals.get(id)!; + approvals.approve(id, "fiona", undefined, fin); + // fred decides from the stale copy the store no longer matches + expect(approvals.store.transition(id, "pending", { decisions: [{ by: "fred", basis: "x", decision: "approve", note: null, ms: 1 }], decision_count: 1 }, stale.decision_count ?? 0)).toBe(false); + expect(approvals.get(id)?.decisions).toHaveLength(1); + }); + + it("a delegate approves on behalf of the delegator, once, and the delegator cannot also vote", () => { + let t = 1_000_000; + const { approvals, delegations, asked } = setup({ now: () => t }); + delegations.grant({ from: "fiona", to: "dan", until_ms: t + 3_600_000, reason: "on leave" }); + asked({ quorum: 2 }); + const id = approvals.list()[0]!.id; + const dan = { roles: ["approver"], groups: [] }; + expect(approvals.approve(id, "dan", undefined, dan).status).toBe("pending"); + expect(approvals.get(id)?.decisions?.[0]).toMatchObject({ by: "dan", on_behalf_of: "fiona", basis: "delegated by fiona (finance)" }); + expect(() => approvals.approve(id, "fiona", undefined, fin)).toThrow(/already decided/); + expect(approvals.approve(id, "fred", undefined, fin).status).toBe("approved"); + // expired or revoked delegations stop working + const second = setup({ now: () => t }); + const d = second.delegations.grant({ from: "fiona", to: "dan", until_ms: t + 1000, reason: "short" }); + second.asked(); + t += 5000; + expect(() => second.approvals.approve(second.approvals.list()[0]!.id, "dan", undefined, dan)).toThrow(/needs an approver/); + void d; + }); + + it("delegation is validated", () => { + const { delegations } = setup(); + expect(() => delegations.grant({ from: "a", to: "a", until_ms: Date.now() + 1000, reason: "x" })).toThrow(/yourself/); + expect(() => delegations.grant({ from: "a", to: "b", until_ms: Date.now() - 1, reason: "x" })).toThrow(/future/); + expect(() => delegations.grant({ from: "a", to: "b", until_ms: Date.now() + 200 * 86_400_000, reason: "x" })).toThrow(/90 days/); + expect(() => delegations.grant({ from: "a", to: "b", until_ms: Date.now() + 1000, reason: " " })).toThrow(/reason/); + const d = delegations.grant({ from: "a", to: "b", until_ms: Date.now() + 5000, reason: "r" }); + expect(delegations.activeFor("b")).toHaveLength(1); + delegations.revoke(d.id); + expect(delegations.activeFor("b")).toHaveLength(0); + }); + + it("an unanswered request is denied on timeout, or escalates to other approvers once", () => { + let t = 1_000_000; + const denying = setup({ now: () => t }); + denying.asked({ timeout_ms: 1000, on_timeout: "deny" }); + t += 1500; + expect(denying.approvals.list()[0]?.status).toBe("expired"); + expect(denying.asked({ timeout_ms: 1000, on_timeout: "deny" })).toMatchObject({ approved: false, message: expect.stringContaining("expired unanswered and was denied") }); + + t = 2_000_000; + const esc = setup({ now: () => t }); + esc.asked({ timeout_ms: 1000, on_timeout: "escalate", escalate_to: ["cfo"] }); + t += 1500; + const rec = esc.approvals.list()[0]!; + expect(rec).toMatchObject({ status: "pending", escalated: true, required_roles: ["cfo"] }); + expect(esc.events).toEqual(["escalated:cfo"]); + expect(() => esc.approvals.approve(rec.id, "fiona", undefined, fin)).toThrow(/needs an approver from: cfo/); + expect(esc.approvals.approve(rec.id, "carl", undefined, { roles: ["approver"], groups: ["cfo"] }).status).toBe("approved"); + t += 5_000_000; // escalation happens once; the next timeout expires it + const again = setup({ now: () => t }); + again.asked({ timeout_ms: 1000, on_timeout: "escalate", escalate_to: ["cfo"] }); + t += 1500; + again.approvals.list(); + t += 1500; + expect(again.approvals.list()[0]?.status).toBe("expired"); + }); + + it("break-glass needs admin, a real reason and evidence, and is recorded as such", () => { + const { approvals, asked, events } = setup(); + asked({ quorum: 2 }); + const id = approvals.list()[0]!.id; + expect(() => approvals.breakGlass(id, "oscar", { reason: "outage, customers affected", evidence: "INC-42" }, ops)).toThrow(/admin role/); + const admin = { roles: ["admin"], groups: [] }; + expect(() => approvals.breakGlass(id, "root", { reason: "short", evidence: "INC-42" }, admin)).toThrow(/at least 10/); + expect(() => approvals.breakGlass(id, "root", { reason: "outage, customers affected", evidence: " " }, admin)).toThrow(/evidence/); + const done = approvals.breakGlass(id, "root", { reason: "outage, customers affected", evidence: "INC-42" }, admin); + expect(done).toMatchObject({ status: "approved", break_glass: { by: "root", evidence: "INC-42" } }); + expect(done.decision_note).toContain("BREAK-GLASS"); + expect(events).toEqual(["approved:root"]); + expect(() => approvals.breakGlass(id, "root", { reason: "outage, customers affected", evidence: "INC-42" }, admin)).toThrow(ApprovalError); + }); + + it("merges several requirements: any listed role, biggest quorum, shortest timeout, deny wins", () => { + expect(mergeRequirements([])).toEqual({}); + expect(mergeRequirements([ + { roles: ["finance"], quorum: 1, timeout_ms: 900000, on_timeout: "escalate", escalate_to: ["cfo"], source: "a" }, + { roles: ["risk"], quorum: 2, timeout_ms: 600000, source: "b" }, + ])).toEqual({ required_roles: ["finance", "risk"], quorum: 2, timeout_ms: 600000, on_timeout: "deny", escalate_to: ["cfo"] }); + }); +}); + +describe("engine: requirements, scoped halts, advise mode, tenants", () => { + const MANIFESTS = [ + { tool: "pay", effect: "compensable", permission: "p", risk: { data_class: "financial", blast_radius: "high" }, approval: { roles: ["finance"], quorum: 2, timeout: "10m", on_timeout: "escalate", escalate_to: ["cfo"] }, compensation: { tool: "refund" } }, + { tool: "refund", effect: "idempotent", permission: "p" }, + { tool: "read", effect: "read_only", permission: "p" }, + ]; + function make(opts: Partial[0]> = {}) { + const kv = new MemoryKvStore(); + const approvals = new Approvals(); + const rv = new Reversi({ registry: Registry.from(MANIFESTS), ledger: new Ledger(), approver: approvals.asApprover(), kv, ...opts }); + return { rv, approvals, kv }; + } + + it("manifest approval rules and policy rules reach the approval request", async () => { + const policy = Policy.from({ rules: [{ id: "big", match: { tool: "pay" }, action: "require_approval", approvers: ["risk"], quorum: 1 }], risk: { thresholds: { approve: 50 } } }); + const { rv, approvals } = make({ policy }); + const run = rv.startRun({ actor: "agent", scope: SealedScope.seal(["p"]) }); + const r = await run.call({ tool: "pay", args: { n: 1 }, reason: "x", execute: () => ({}) }); + expect(r).toMatchObject({ status: "blocked", reasons: [expect.stringContaining("needs 2 approvers from risk or finance")] }); + expect(approvals.list()[0]).toMatchObject({ requested_by: "agent", required_roles: ["risk", "finance"], quorum: 2, on_timeout: "deny" }); + expect(rv.ledger.records().at(-1)?.detail).toContain("policy `big`"); + expect(rv.ledger.records().at(-1)?.detail).toContain("manifest requires approval"); + expect(rv.ledger.records().at(-1)?.detail).toContain("risk score"); + }); + + it("the risk score is recorded in the intent and on the step", async () => { + const policy = Policy.from({ risk: {} }); + const { rv } = make({ policy }); + const run = rv.startRun({ actor: "agent", scope: SealedScope.seal(["p"]) }); + await run.call({ tool: "read", reason: "x", execute: () => 1 }); + expect(run.state().steps[0]?.risk_score).toBe(0); // read_only in the default environment + expect(rv.ledger.records().find((r) => r.event === "intended")?.detail).toBe("risk 0"); + }); + + it("scoped halts block only what they name, are shared through the KV, and rollback still works", async () => { + const { rv, kv } = make(); + const run = rv.startRun({ actor: "agent-7", scope: SealedScope.seal(["p"]), tenant: "acme" }); + rv.halt("pay is broken", { tool: "pay" }); + const other = new Reversi({ registry: Registry.from(MANIFESTS), ledger: new Ledger(), kv }); // another process + expect(other.halted({ tool: "pay" })).toContain("pay is broken (scope: tool=pay)"); + expect(other.halted({ tool: "read" })).toBeNull(); + expect((await run.call({ tool: "pay", reason: "x", execute: () => ({}) })).status).toBe("blocked"); + expect((await run.call({ tool: "read", reason: "x", execute: () => 1 })).status).toBe("executed"); + rv.halt("tenant freeze", { tenant: "acme" }); + expect((await run.call({ tool: "read", reason: "x", execute: () => 1 })).status).toBe("blocked"); + rv.halt("agent paused", { actor: "agent-*" }); + expect(rv.halts()).toHaveLength(3); + expect(rv.resume(rv.halts()[0]!.id)).toBe(1); + expect(rv.resume()).toBe(2); + expect((await run.call({ tool: "read", reason: "x", execute: () => 1 })).status).toBe("executed"); + }); + + it("advise mode runs the call and reports what would have happened", async () => { + const policy = Policy.from({ rules: [{ id: "no", match: { tool: "read" }, action: "deny" }] }); + const { rv } = make({ policy }); + const run = rv.startRun({ actor: "a", scope: SealedScope.seal(["p"]), mode: "advise" }); + const r = await run.call({ tool: "read", reason: "x", execute: () => 1 }); + expect(r).toMatchObject({ status: "executed", observed_violations: [expect.stringContaining("policy `no`")] }); + expect(rv.ledger.records().some((x) => x.event === "would_block")).toBe(true); + }); + + it("tenant flows from the run into policy matching", async () => { + const policy = Policy.from({ rules: [{ id: "acme-no-pay", match: { tenant: "acme", tool: "pay" }, action: "deny" }] }); + const { rv } = make({ policy }); + const acme = rv.startRun({ actor: "a", scope: SealedScope.seal(["p"]), tenant: "acme" }); + const globex = rv.startRun({ actor: "b", scope: SealedScope.seal(["p"]), tenant: "globex" }); + expect((await acme.call({ tool: "pay", reason: "x", execute: () => ({}) })).status).toBe("blocked"); + const g = await globex.begin({ tool: "pay", reason: "x" }); + expect(g).toMatchObject({ decision: "block", reasons: [expect.stringContaining("approval required")] }); // not denied by the tenant rule; the manifest still wants approval + }); +}); diff --git a/packages/reversi/test/kv.test.ts b/packages/reversi/test/kv.test.ts new file mode 100644 index 0000000..8e449ec --- /dev/null +++ b/packages/reversi/test/kv.test.ts @@ -0,0 +1,110 @@ +import { randomBytes } from "node:crypto"; +import { mkdtempSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { afterEach, describe, expect, it } from "vitest"; +import { FileKvStore, KvConflictError, MemoryKvStore, kvUpdate, type KvStore } from "../src/index.js"; +import { openSqliteStores } from "../src/sqlite.js"; + +const dirs: string[] = []; +afterEach(() => { + while (dirs.length) rmSync(dirs.pop() as string, { recursive: true, force: true }); +}); +const tmp = () => { + const d = mkdtempSync(join(tmpdir(), "rv-kv-")); + dirs.push(d); + return d; +}; + +function suite(name: string, make: () => { kv: KvStore; close?: () => void }) { + describe(`KvStore: ${name}`, () => { + it("stores, versions, lists by prefix and deletes", () => { + const { kv, close } = make(); + expect(kv.get("n", "a")).toBeUndefined(); + kv.put("n", "a", { x: 1 }); + expect(kv.get("n", "a")).toEqual({ key: "a", value: { x: 1 }, version: 1 }); + kv.put("n", "a", { x: 2 }); + expect(kv.get("n", "a")?.version).toBe(2); + kv.put("n", "ab/1", 1); + kv.put("n", "b", 2); + kv.put("other", "a", 3); + expect(kv.list("n").map((e) => e.key)).toEqual(["a", "ab/1", "b"]); + expect(kv.list("n", "a").map((e) => e.key)).toEqual(["a", "ab/1"]); + expect(kv.delete("n", "a")).toBe(true); + expect(kv.delete("n", "a")).toBe(false); + expect(kv.get("other", "a")?.value).toBe(3); + close?.(); + }); + + it("compare-and-set only succeeds with the expected version", () => { + const { kv, close } = make(); + expect(kv.cas("n", "k", "first", 0)).toBe(true); + expect(kv.cas("n", "k", "again", 0)).toBe(false); // already exists + expect(kv.cas("n", "k", "stale", 5)).toBe(false); + expect(kv.cas("n", "k", "second", 1)).toBe(true); + expect(kv.get("n", "k")).toMatchObject({ value: "second", version: 2 }); + close?.(); + }); + + it("kvUpdate retries until its write lands, even when another writer interleaves", () => { + const { kv, close } = make(); + let interfered = false; + const out = kvUpdate(kv, "n", "counter", (cur) => { + if (!interfered) { + interfered = true; + kv.put("n", "counter", 100); // someone else writes between our read and write + } + return (cur ?? 0) + 1; + }); + expect(out).toBe(101); + expect(kv.get("n", "counter")?.value).toBe(101); + close?.(); + }); + + it("kvUpdate gives up with a clear error under endless contention", () => { + const { kv, close } = make(); + expect(() => kvUpdate(kv, "n", "k", (c) => { kv.put("n", "k", Math.random()); return (c ?? 0) + 1; }, 3)).toThrow(KvConflictError); + close?.(); + }); + }); +} + +suite("memory", () => ({ kv: new MemoryKvStore() })); +suite("file", () => ({ kv: new FileKvStore(join(tmp(), "nested", "settings.json")) })); +suite("sqlite", () => { + const s = openSqliteStores(join(tmp(), "r.db")); + return { kv: s.kv, close: () => s.db.close() }; +}); + +describe("file KvStore shared by two instances", () => { + it("sees each other's writes", () => { + const f = join(tmp(), "s.json"); + const a = new FileKvStore(f); + const b = new FileKvStore(f); + a.put("n", "k", 1); + expect(b.get("n", "k")?.value).toBe(1); + expect(b.cas("n", "k", 2, 1)).toBe(true); + expect(a.get("n", "k")?.version).toBe(2); + }); +}); + +const pgUrl = process.env.REVERSI_TEST_PG; +describe.skipIf(!pgUrl)("KvStore: postgres", () => { + it("behaves like the others", async () => { + const { SyncPg, openPostgresStores } = await import("../src/postgres.js"); + const schema = `rv_kv_${randomBytes(5).toString("hex")}`; + const admin = new SyncPg(pgUrl as string); + admin.query(`CREATE SCHEMA ${schema}`); + admin.close(); + const u = new URL(pgUrl as string); + u.searchParams.set("options", `-c search_path=${schema}`); + const s = openPostgresStores(u.toString()); + s.kv.put("n", "a", { x: 1 }); + expect(s.kv.cas("n", "a", { x: 2 }, 1)).toBe(true); + expect(s.kv.cas("n", "a", { x: 3 }, 1)).toBe(false); + expect(s.kv.cas("n", "new", 1, 0)).toBe(true); + expect(s.kv.list("n").map((e) => e.key)).toEqual(["a", "new"]); + expect(s.kv.delete("n", "a")).toBe(true); + s.pg.close(); + }); +}); diff --git a/packages/reversi/test/manifest-engine.test.ts b/packages/reversi/test/manifest-engine.test.ts new file mode 100644 index 0000000..b6fbf9e --- /dev/null +++ b/packages/reversi/test/manifest-engine.test.ts @@ -0,0 +1,177 @@ +import { describe, expect, it } from "vitest"; +import { Ledger, Registry, Reversi, SealedScope, expectMismatch, renderIdempotencyKey } from "../src/index.js"; + +const MANIFESTS = [ + { + tool: "pay.create", + effect: "compensable", + permission: "p", + idempotency: { key: "{{args.order_id}}:{{args.amount}}", scope: "run" }, + compensation: { + tool: "pay.cancel", + kind: "compensation", + residue: ["customer may see a notification"], + argument_map: { id: "$result.id" }, + verification: { tool: "pay.get", arguments: { id: "$result.id", order: "$args.order_id" }, expect: { status: "canceled", "meta.reason": "customer" }, retries: 3, backoff_ms: 10 }, + }, + }, + { tool: "pay.cancel", effect: "idempotent", permission: "p" }, + { tool: "pay.get", effect: "read_only", permission: "p" }, + { tool: "global.once", effect: "idempotent", permission: "p", idempotency: { key: "{{args.k}}", scope: "global" } }, + { tool: "actor.once", effect: "idempotent", permission: "p", idempotency: { key: "{{args.k}}", scope: "actor" } }, + { tool: "wip", effect: "read_only", permission: "p", status: "draft" }, + { tool: "old", effect: "read_only", permission: "p", status: "deprecated", replaced_by: "new" }, + { tool: "new", effect: "read_only", permission: "p" }, +]; + +function make() { + const rv = new Reversi({ registry: Registry.from(MANIFESTS), ledger: new Ledger() }); + const run = (id = "r1", actor = "agent") => rv.startRun({ run_id: id, actor, scope: SealedScope.seal(["p"]) }); + return { rv, run }; +} +const ok = (v: unknown = { id: "pi_1" }) => () => v; + +describe("manifest status", () => { + it("blocks calls to a draft manifest and notes a deprecated one", async () => { + const { rv, run } = make(); + const r = run(); + expect(await r.call({ tool: "wip", reason: "x", execute: ok() })).toMatchObject({ status: "blocked", reasons: [expect.stringContaining("is a draft")] }); + expect((await r.call({ tool: "old", reason: "x", execute: ok() })).status).toBe("executed"); + expect(rv.ledger.records().find((x) => x.event === "intended")?.detail).toBe("manifest deprecated; use new"); + }); +}); + +describe("idempotency keys", () => { + it("renders templates and refuses missing or non-scalar values", () => { + expect(renderIdempotencyKey("{{args.a}}-{{ args.b.c }}", { a: 1, b: { c: "x" } })).toBe("1-x"); + expect(renderIdempotencyKey("{{args.a}}", {})).toBeUndefined(); + expect(renderIdempotencyKey("{{args.a}}", { a: { n: 1 } })).toBeUndefined(); + expect(renderIdempotencyKey("{{args.a}}", null)).toBeUndefined(); + }); + + it("blocks a duplicate in the same run while the first still stands, but allows it after the first failed or was undone", async () => { + const { run } = make(); + const r = run(); + const args = { order_id: "o1", amount: 5 }; + expect((await r.call({ tool: "pay.create", args, reason: "x", execute: ok() })).status).toBe("executed"); + const dup = await r.call({ tool: "pay.create", args, reason: "x", execute: ok() }); + expect(dup).toMatchObject({ status: "blocked", reasons: [expect.stringContaining("duplicate call: idempotency key matches step 0")] }); + expect((await r.call({ tool: "pay.create", args: { order_id: "o1", amount: 6 }, reason: "x", execute: ok() })).status).toBe("executed"); // different key + // a failed first attempt does not count + const r2 = run("r2"); + await r2.call({ tool: "pay.create", args, reason: "x", execute: () => { throw new Error("nope"); } }).catch(() => {}); + expect((await r2.call({ tool: "pay.create", args, reason: "x", execute: ok() })).status).toBe("executed"); + }); + + it("treats an in-flight call as a duplicate", async () => { + const { run } = make(); + const r = run(); + const first = await r.begin({ tool: "pay.create", args: { order_id: "o", amount: 1 }, reason: "x" }); + expect(first.decision).toBe("allow"); + const second = await r.begin({ tool: "pay.create", args: { order_id: "o", amount: 1 }, reason: "x" }); + expect(second).toMatchObject({ decision: "block", reasons: [expect.stringContaining("still in flight")] }); + }); + + it("blocks a call whose key cannot be rendered", async () => { + const { run } = make(); + const r = await run().call({ tool: "pay.create", args: { order_id: "o" }, reason: "x", execute: ok() }); + expect(r).toMatchObject({ status: "blocked", reasons: [expect.stringContaining("missing or not simple values")] }); + }); + + it("actor scope spans that actor's runs only; global spans everyone", async () => { + const { run } = make(); + const a1 = run("a1", "alice"); + const a2 = run("a2", "alice"); + const b1 = run("b1", "bob"); + await a1.call({ tool: "actor.once", args: { k: "x" }, reason: "x", execute: ok() }); + expect((await a2.call({ tool: "actor.once", args: { k: "x" }, reason: "x", execute: ok() })).status).toBe("blocked"); + expect((await b1.call({ tool: "actor.once", args: { k: "x" }, reason: "x", execute: ok() })).status).toBe("executed"); + await a1.call({ tool: "global.once", args: { k: "g" }, reason: "x", execute: ok() }); + expect((await b1.call({ tool: "global.once", args: { k: "g" }, reason: "x", execute: ok() })).status).toBe("blocked"); + }); +}); + +describe("compensation metadata and verification", () => { + const args = { order_id: "o1", amount: 5 }; + + it("the plan shows residue, kind and that the undo is verified; results are referenced by hash", async () => { + const { run } = make(); + const r = run(); + await r.call({ tool: "pay.create", args, reason: "x", execute: ok() }); + expect(r.plan().actions[0]).toMatchObject({ kind: "compensate", residue: ["customer may see a notification"], compensation_kind: "compensation", verified: true }); + const s = r.state().steps[0]; + expect(s?.result_hash).toMatch(/^[0-9a-f]{64}$/); + expect(s?.verification).toMatchObject({ tool: "pay.get", args: { id: "pi_1", order: "o1" }, missing: [] }); + }); + + it("verifies the undo by reading back, retrying with backoff until the state matches", async () => { + const { rv, run } = make(); + const r = run(); + await r.call({ tool: "pay.create", args, reason: "x", execute: ok() }); + const calls: unknown[] = []; + const sleeps: number[] = []; + let n = 0; + const rep = await r.rollback({ + compensate: () => {}, + retry: { sleep: async (ms) => void sleeps.push(ms) }, + verifier: (tool, a) => { + calls.push([tool, a]); + return ++n < 3 ? { status: "pending" } : { status: "canceled", meta: { reason: "customer" } }; + }, + }); + expect(rep).toMatchObject({ complete: true, compensated: [0], unverified: [] }); + expect(calls).toHaveLength(3); + expect(calls[0]).toEqual(["pay.get", { id: "pi_1", order: "o1" }]); + expect(sleeps).toEqual([10, 20]); + expect(rv.ledger.records().map((x) => x.event).slice(-3)).toEqual(["compensating", "verified", "compensated"]); + }); + + it("a verification that never matches fails the compensation and is never reported complete", async () => { + const { rv, run } = make(); + const r = run(); + await r.call({ tool: "pay.create", args, reason: "x", execute: ok() }); + const rep = await r.rollback({ compensate: () => {}, retry: { sleep: async () => {} }, verifier: () => ({ status: "succeeded" }) }); + expect(rep.complete).toBe(false); + expect(rep.failed[0]?.error).toContain("verification failed"); + expect(r.state().steps[0]?.status).toBe("compensation_failed"); + const ev = rv.ledger.records().find((x) => x.event === "verification_failed"); + expect(ev?.detail).toContain("after 3 attempts"); + expect(ev?.detail).toContain("`status` is \"succeeded\", expected \"canceled\""); + }); + + it("without a verifier the step is compensated but reported unverified, and the rollback is incomplete", async () => { + const { rv, run } = make(); + const r = run(); + await r.call({ tool: "pay.create", args, reason: "x", execute: ok() }); + const rep = await r.rollback({ compensate: () => {} }); + expect(rep).toMatchObject({ compensated: [0], unverified: [0], complete: false }); + expect(rv.ledger.records().some((x) => x.event === "unverified")).toBe(true); + }); + + it("when the verification's arguments are missing it says so instead of skipping", async () => { + const { run } = make(); + const r = run(); + await r.call({ tool: "pay.create", args, reason: "x", execute: ok({ no_id: true }) }); // result has no `id` + const plan = r.plan(); + expect(plan.actions[0]).toMatchObject({ kind: "unrecoverable" }); // compensation args missing too + expect(r.state().steps[0]?.verification?.missing).toContain("id"); + }); + + it("an invalid result (does not match the tool's declared output) makes its $result references unusable", async () => { + const { run } = make(); + const r = run(); + const b = await r.begin({ tool: "pay.create", args, reason: "x" }); + if (b.decision !== "allow") throw new Error("expected allow"); + r.complete(b.step, { status: "succeeded", result: { id: "pi_9" }, invalid_result: true, notes: ["output schema: id must be a number"] }); + const s = r.state().steps[0]!; + expect(s.handle_missing).toContain("id"); + expect(r.plan().actions[0]).toMatchObject({ kind: "unrecoverable" }); + }); + + it("expectMismatch handles nested paths, arrays and missing values", () => { + expect(expectMismatch({ a: { b: [1, { c: 2 }] } }, { "a.b.1.c": 2 })).toBeNull(); + expect(expectMismatch({ a: 1 }, { a: 2 })).toContain("expected 2"); + expect(expectMismatch({}, { "a.b": null })).toContain("missing"); + expect(expectMismatch({ s: { x: 1, y: 2 } }, { s: { y: 2, x: 1 } })).toBeNull(); // key order does not matter + }); +}); diff --git a/packages/reversi/test/manifest-tooling.test.ts b/packages/reversi/test/manifest-tooling.test.ts new file mode 100644 index 0000000..41678bd --- /dev/null +++ b/packages/reversi/test/manifest-tooling.test.ts @@ -0,0 +1,298 @@ +import { createServer } from "node:http"; +import { mkdtempSync, readFileSync, readdirSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import type { AddressInfo } from "node:net"; +import { afterEach, describe, expect, it } from "vitest"; +import { stringify } from "yaml"; +import { + ManifestRegistry, + PolicyVersionError, + PolicyVersions, + RegistryError, + Registry, + banner, + classifyTools, + compensateMetaOf, + generateSigningKey, + hasErrors, + hintsOf, + lintManifests, + loadRemoteRegistry, + renderDiff, + runManifestTests, + schemaHash, + suggestManifests, + type McpToolInfo, +} from "../src/index.js"; + +const dirs: string[] = []; +afterEach(() => { + while (dirs.length) rmSync(dirs.pop() as string, { recursive: true, force: true }); +}); +const tmp = () => { + const d = mkdtempSync(join(tmpdir(), "rv-tool-")); + dirs.push(d); + return d; +}; + +const create = { tool: "issues.create", effect: "reversible", permission: "gh:write", compensation: { tool: "issues.close", argument_map: { number: "$result.number" } } }; +const close = { tool: "issues.close", effect: "idempotent", permission: "gh:write" }; + +describe("diff", () => { + it("shows only what changed, with context", () => { + const a = ["a", "b", "c", "d", "e", "f", "g", "h"].join("\n"); + const b = ["a", "b", "c", "D", "e", "f", "g", "h", "i"].join("\n"); + expect(renderDiff(a, b, 1)).toBe(["...", " c", "-d", "+D", " e", "...", " h", "+i"].join("\n").replace(/^\.\.\.\n/, "") ); + }); +}); + +describe("manifest registry", () => { + it("publishes immutable versions, approves drafts, deprecates and restores, with a diff between versions", () => { + const dir = tmp(); + let t = 1000; + const reg = new ManifestRegistry(dir, { now: () => t++ }); + reg.publish(close, { by: "alice", status: "approved" }); + expect(reg.publish(create, { by: "alice" })).toMatchObject({ tool: "issues.create", version: 1 }); + expect(reg.history("issues.create")[0]).toMatchObject({ status: "draft", by: "alice" }); + expect(reg.load().size).toBe(1); // the draft is not enforceable + expect(reg.load({ drafts: true }).size).toBe(2); + expect(() => reg.approve("issues.create", { by: "alice", require_other: true })).toThrow(/author of a draft cannot approve/); + const approved = reg.approve("issues.create", { by: "bob", require_other: true, note: "reviewed" }); + expect(approved.version).toBe(2); + expect(reg.history("issues.create")[1]).toMatchObject({ status: "approved", approved_by: "bob", by: "alice" }); + expect(reg.load().size).toBe(2); + + reg.publish({ ...create, description: "Open an issue" }, { by: "alice", status: "approved" }); + expect(reg.diff("issues.create", 2, 3)).toContain('+ "description": "Open an issue",'); + const dep = reg.deprecate("issues.create", { by: "carol", replaced_by: "issues.close" }); + expect(reg.current("issues.create")).toMatchObject({ status: "deprecated", replaced_by: "issues.close", version: dep.version }); + expect(reg.load().get("issues.create")?.status).toBe("deprecated"); // still loaded: it must keep rolling back + const restored = reg.rollbackTo("issues.create", 2, { by: "carol" }); + expect(restored.version).toBe(dep.version + 1); + expect(reg.current("issues.create")?.status).toBe("approved"); + expect(reg.tools()).toEqual(["issues.close", "issues.create"]); + expect(reg.verifyAll()).toEqual([]); + expect(() => reg.history("nope")).toThrow(RegistryError); + expect(() => reg.publish({ tool: "x", effect: "reversible", permission: "p" }, { by: "a" })).toThrow(/must declare a compensation/); + }); + + it("refuses a manifest edited outside the workflow, and a version whose index entry was edited to match", () => { + const dir = tmp(); + const reg = new ManifestRegistry(dir); + reg.publish(close, { by: "a", status: "approved" }); + const file = join(dir, "tools", "issues.close", "v1.json"); + writeFileSync(file, readFileSync(file, "utf8").replace('"gh:write"', '"gh:admin"')); + expect(() => reg.load()).toThrow(/changed outside the registry/); + expect(reg.verifyAll()[0]).toContain("checksum does not match"); + }); + + it("with a trusted key every version must be signed by it", () => { + const dir = tmp(); + const key = generateSigningKey(); + const signed = new ManifestRegistry(dir, { trusted_public_pem: key.public_pem }); + signed.publish(close, { by: "a", status: "approved", key }); + expect(signed.load().size).toBe(1); + // someone with write access adds a manifest without the key + new ManifestRegistry(dir).publish({ tool: "other", effect: "read_only", permission: "p" }, { by: "mallory", status: "approved" }); + expect(() => signed.load()).toThrow(/not signed by the trusted key/); + const wrongKey = new ManifestRegistry(dir, { trusted_public_pem: generateSigningKey().public_pem }); + expect(() => wrongKey.get("issues.close", 1)).toThrow(/not signed/); + }); + + it("loads from a plain HTTP server that serves the same files, checking checksums", async () => { + const dir = tmp(); + const reg = new ManifestRegistry(dir); + reg.publish(create, { by: "a", status: "approved" }); + reg.publish(close, { by: "a", status: "approved" }); + let tamper = false; + const server = createServer((req, res) => { + const path = decodeURIComponent((req.url ?? "").split("?")[0] as string).replace(/^\//, ""); + try { + let body = readFileSync(join(dir, path), "utf8"); + if (tamper && path.endsWith("issues.close/v1.json")) body = body.replace("gh:write", "gh:root"); + res.writeHead(200, { "content-type": "application/json" }).end(body); + } catch { + res.writeHead(404).end(); + } + }); + await new Promise((r) => server.listen(0, "127.0.0.1", r)); + try { + const url = `http://127.0.0.1:${(server.address() as AddressInfo).port}`; + expect((await loadRemoteRegistry(url)).size).toBe(2); + tamper = true; + await expect(loadRemoteRegistry(url)).rejects.toThrow(/does not match its checksum/); + await expect(loadRemoteRegistry("http://169.254.169.254")).rejects.toThrow(/metadata/); + await expect(loadRemoteRegistry(`${url}/nope`)).rejects.toThrow(/answered 404/); + } finally { + server.close(); + } + }); +}); + +describe("policy versions", () => { + const good = "version: 1\nrules:\n - id: r\n match: { tool: x }\n action: deny\ntests:\n - { name: x denied, input: { tool: x }, expect: deny }\n"; + it("publishes, refuses a failing or invalid policy, activates, diffs and detects tampering", () => { + const dir = tmp(); + const pv = new PolicyVersions(dir); + expect(PolicyVersions.isVersioned(dir)).toBe(false); + const v1 = pv.publish(good, { by: "alice", note: "first" }); + expect(v1).toMatchObject({ version: 1, tests_passed: 1 }); + expect(PolicyVersions.isVersioned(dir)).toBe(true); + expect(pv.activeVersion()).toBe(1); + expect(() => pv.publish(good.replace("expect: deny", "expect: allow"), { by: "a" })).toThrow(/1 of the policy's own tests fail: x denied/); + expect(() => pv.publish("rules: [{ id: a, action: nope }]", { by: "a" })).toThrow(); + const v2 = pv.publish(good.replace("tool: x", "tool: y").replace("{ tool: x }", "{ tool: y }"), { by: "bob", activate: false }); + expect(pv.activeVersion()).toBe(1); + pv.activate(v2.version); + expect(pv.activeVersion()).toBe(2); + expect(pv.diff(1, 2)).toContain("+ match: { tool: y }"); + pv.activate(1); // rollback + expect(pv.doc(pv.activeVersion() as number).rules[0]?.id).toBe("r"); + expect(pv.history().map((h) => h.by)).toEqual(["alice", "bob"]); + writeFileSync(join(dir, "policy-v1.yaml"), good.replace("deny", "allow")); + expect(() => pv.activeText()).toThrow(PolicyVersionError); + expect(() => pv.activate(9)).toThrow(/no policy version 9/); + expect(readdirSync(dir).sort()).toEqual(["policy-index.json", "policy-v1.yaml", "policy-v2.yaml"]); + }); +}); + +describe("lint", () => { + it("flags unsafe, incomplete and contradictory manifests, and checks against the upstream's schemas", () => { + const reg = Registry.from([ + { tool: "a.read", effect: "read_only", permission: "a:write" }, + { tool: "a.write", effect: "compensable", permission: "a:write", compensation: { tool: "a.undo", argument_map: { id: "$args.missing" } }, idempotency: { key: "{{args.ghost}}" } }, + { tool: "a.undo", effect: "reversible", permission: "a:write", compensation: { tool: "a.redo", kind: "compensation", residue: ["x"] } }, + { tool: "a.redo", effect: "read_only", permission: "a:read" }, + { tool: "a.drop", effect: "irreversible", permission: "a:admin", status: "draft" }, + { tool: "loop.1", effect: "read_only", permission: "l", requires: ["loop.2"] }, + { tool: "loop.2", effect: "read_only", permission: "l", requires: ["loop.1"] }, + ]); + const f = lintManifests(reg, { tools: [{ name: "a.write", inputSchema: { properties: { id: {} } } }, { name: "a.unknown" }] }); + const codes = (tool: string) => f.filter((x) => x.tool === tool).map((x) => x.code); + expect(codes("a.read")).toContain("L019"); + expect(codes("a.write")).toEqual(expect.arrayContaining(["L010", "L011", "L012", "L013", "L014", "L017", "L020", "L016"])); + expect(codes("a.undo")).toContain("L030"); + expect(codes("a.drop")).toEqual(expect.arrayContaining(["L015", "L021", "L024"])); + expect(codes("loop.1")).toContain("L023"); + expect(f.find((x) => x.code === "L035")?.tool).toBe("a.unknown"); + expect(f.find((x) => x.code === "L034")?.tool).toBe("a.read"); + expect(hasErrors(f)).toBe(true); + const clean = Registry.from([{ tool: "r", effect: "read_only", permission: "r:read", description: "reads" }]); + expect(lintManifests(clean)).toEqual([]); + }); +}); + +const TOOLS: McpToolInfo[] = [ + { name: "gh.get_issue", description: "Get an issue", annotations: { readOnlyHint: true }, inputSchema: { properties: { number: {} } } }, + { name: "gh.create_issue", description: "Open an issue\nlong text", inputSchema: { properties: { title: {}, body: {} }, required: ["title"] } }, + { name: "gh.close_issue", description: "Close an issue", annotations: { idempotentHint: true }, inputSchema: { properties: { number: {} }, required: ["number"] } }, + { name: "gh.delete_repo", description: "Delete a repo", annotations: { destructiveHint: true }, inputSchema: { properties: { name: {} } } }, + { name: "gh.list_labels", inputSchema: { properties: {} } }, + { name: "pay.charge", description: "Charge a customer card", inputSchema: { properties: { amount: {}, customer: {} } }, _meta: { "mcp-compensate": { compensable: true, compensation: { tool: "pay.refund", mapping: { charge_id: "$result.id", amount: "$args.amount", bogus: "nope" } } } } }, + { name: "pay.refund", description: "Refund a charge", inputSchema: { properties: { charge_id: {}, amount: {} } } }, + { name: "dns.set_record", description: "Set a record", annotations: { openWorldHint: true }, inputSchema: { properties: { name: {}, value: {} } } }, +]; + +describe("suggest manifests", () => { + it("drafts effects, undo pairs and argument maps from names, schemas and hints, always as drafts", () => { + const r = suggestManifests(TOOLS, { server: "github" }); + const by = Object.fromEntries(r.suggestions.map((s) => [s.manifest.tool as string, s])); + expect(by["gh.get_issue"]?.manifest).toMatchObject({ effect: "read_only", permission: "github:read", status: "draft" }); + expect(by["gh.get_issue"]?.notes[0]).toContain("readOnlyHint"); + expect(by["gh.list_labels"]).toMatchObject({ confidence: "low", manifest: { effect: "read_only" } }); + expect(by["gh.create_issue"]?.manifest).toMatchObject({ effect: "compensable", compensation: { tool: "gh.close_issue", argument_map: { number: "$result.number" } } }); + expect(by["gh.create_issue"]?.manifest.description).toBe("Open an issue"); + expect(by["gh.delete_repo"]?.manifest).toMatchObject({ effect: "irreversible", approval: "always", permission: "github:admin", risk: { blast_radius: "high" } }); + expect(by["gh.close_issue"]?.manifest.effect).toBe("idempotent"); + expect(by["dns.set_record"]?.manifest.effect).toBe("irreversible"); // no inverse found: the safe default + expect(by["dns.set_record"]?.notes.join()).toContain("openWorldHint"); + // mcp-compensate wins over guessing, keeps only valid references, and the undo is drafted too + expect(by["pay.charge"]).toMatchObject({ confidence: "high", manifest: { effect: "compensable", compensation: { tool: "pay.refund", argument_map: { charge_id: "$result.id", amount: "$args.amount" } }, risk: { data_class: "financial" } } }); + expect(by["pay.refund"]?.manifest.effect).toBe("idempotent"); + expect(Object.values(by).every((s) => s.manifest.status === "draft")).toBe(true); + // the drafts are valid manifests once the TODO residue note is accepted + const reg = Registry.from(r.suggestions.map((s) => s.manifest)); + expect(reg.size).toBe(r.suggestions.length); + expect(r.suggestions.every((s) => typeof s.manifest.schema_hash === "string" || !TOOLS.find((t) => t.name === s.manifest.tool)?.inputSchema)).toBe(true); + }); + + it("skips tools that already have manifests, can be limited, and reports schema hashes that change with the schema", () => { + const existing = Registry.from([{ tool: "gh.get_issue", effect: "read_only", permission: "p" }]); + const r = suggestManifests(TOOLS, { existing, only: ["gh.get_issue", "gh.delete_repo"] }); + expect(r.skipped).toEqual(["gh.get_issue"]); + expect(r.suggestions.map((s) => s.manifest.tool)).toEqual(["gh.delete_repo"]); + const a = schemaHash({ name: "t", inputSchema: { properties: { a: {} } } }); + expect(schemaHash({ name: "t", inputSchema: { properties: { a: {} } } })).toBe(a); + expect(schemaHash({ name: "t", inputSchema: { properties: { a: {}, b: {} } } })).not.toBe(a); + }); + + it("reads annotations and the mcp-compensate convention in its accepted shapes", () => { + expect(hintsOf({ name: "x", annotations: { readOnlyHint: true, destructiveHint: false, idempotentHint: true, openWorldHint: true, title: "t" } })).toEqual({ read_only: true, destructive: false, idempotent: true, open_world: true }); + expect(compensateMetaOf({ name: "x", annotations: { "mcp-compensate": { irreversible: true } } })).toEqual({ irreversible: true }); + expect(compensateMetaOf({ name: "x", _meta: { compensate: { compensation: "undo_x" } } })).toEqual({ compensation: { tool: "undo_x", mapping: {} } }); + expect(compensateMetaOf({ name: "x", _meta: { "mcp-compensate": { readonly: true } } })).toEqual({ readonly: true }); + expect(compensateMetaOf({ name: "x" })).toBeUndefined(); + expect(compensateMetaOf({ name: "x", _meta: { "mcp-compensate": "yes" } })).toBeUndefined(); + }); + + it("the drafts survive a YAML round trip", () => { + const text = stringify(suggestManifests(TOOLS.slice(0, 3)).suggestions.map((s) => s.manifest)); + expect(text).toContain("status: draft"); + }); +}); + +describe("classification banner", () => { + it("separates classified tools, unclassified reads, unclassified changes and destructive ones", () => { + const reg = Registry.from([{ tool: "gh.get_issue", effect: "read_only", permission: "p" }, { tool: "gh.create_issue", effect: "idempotent", permission: "p", status: "draft" }]); + const rows = classifyTools(reg, TOOLS.slice(0, 5)); + expect(rows.map((r) => [r.name, r.class])).toEqual([ + ["gh.get_issue", "classified"], + ["gh.create_issue", "classified"], + ["gh.close_issue", "unclassified_mutation"], + ["gh.delete_repo", "destructive_no_manifest"], + ["gh.list_labels", "unclassified_mutation"], + ]); + const text = banner(rows, { mode: "observe", next: "Next: reversi suggest-manifest" }); + expect(text).toContain("Reversi gateway running in observe mode."); + expect(text).toContain("OK gh.get_issue - read_only"); + expect(text).toContain("(draft)"); + expect(text).toContain("BLOCK gh.delete_repo"); + expect(text).toContain("2 unclassified changes, 1 destructive"); + expect(text.trim().endsWith("Next: reversi suggest-manifest")).toBe(true); + expect(classifyTools(Registry.from([]), [{ name: "r", annotations: { readOnlyHint: true } }])[0]?.class).toBe("unclassified_read_hint"); + }); +}); + +describe("manifest contract tests", () => { + const reg = Registry.from([ + { tool: "orders.create", effect: "reversible", permission: "p", compensation: { tool: "orders.cancel", argument_map: { id: "$result.id" } } }, + { tool: "orders.cancel", effect: "idempotent", permission: "p" }, + { tool: "pay.charge", effect: "compensable", permission: "p", requires: ["orders.create"], idempotency: { key: "{{args.order}}" }, compensation: { tool: "pay.refund", argument_map: { charge: "$result.charge_id", amount: "$args.amount" }, verification: { tool: "pay.get", arguments: { charge: "$result.charge_id" }, expect: { status: "refunded" } } } }, + { tool: "pay.refund", effect: "idempotent", permission: "p" }, + { tool: "pay.get", effect: "read_only", permission: "p" }, + { tool: "never.tested", effect: "idempotent", permission: "p" }, + ]); + + it("passes a correct manifest and names every way a wrong expectation fails", async () => { + const { results, untested } = await runManifestTests(reg, [ + { name: "order maps its id", tool: "orders.create", result: { id: 7 }, expect_handle: { id: 7 }, expect_plan: "compensate" }, + { name: "charge maps charge, amount and verification", tool: "pay.charge", args: { order: "o1", amount: 20 }, result: { charge_id: "c1" }, expect_handle: { charge: "c1", amount: 20 }, expect_verification: { charge: "c1" }, verify_with: { status: "refunded" }, expect_verified: true }, + { name: "read-back that disagrees is rejected", tool: "pay.charge", args: { order: "o1", amount: 20 }, result: { charge_id: "c1" }, verify_with: { status: "pending" }, expect_verified: false }, + { name: "a failure leaves nothing to undo", tool: "orders.create", fail: "boom" }, + { name: "duplicates are refused", tool: "pay.charge", args: { order: "o1", amount: 1 }, result: { charge_id: "c" }, expect_duplicate: true }, + { name: "missing key argument is refused", tool: "pay.charge", args: {}, expect_blocked: "missing or not simple" }, + { name: "wrong expectation", tool: "orders.create", result: { id: 7 }, expect_handle: { id: 8 } }, + { name: "result lacks the id", tool: "orders.create", result: { nope: 1 }, expect_plan: "compensate" }, + { name: "should have been refused", tool: "orders.cancel", expect_blocked: "x" }, + { name: "unknown", tool: "ghost" }, + ]); + const by = Object.fromEntries(results.map((r) => [r.name, r])); + for (const ok of ["order maps its id", "charge maps charge, amount and verification", "read-back that disagrees is rejected", "a failure leaves nothing to undo", "duplicates are refused", "missing key argument is refused"]) expect(by[ok]?.problems, ok).toEqual([]); + expect(by["wrong expectation"]?.problems[0]).toContain("`id` is 7, expected 8"); + expect(by["result lacks the id"]?.problems[0]).toContain("expected plan kind compensate, got unrecoverable"); + expect(by["should have been refused"]?.problems[0]).toContain("expected the call to be refused"); + expect(by["unknown"]?.problems[0]).toContain("no manifest for `ghost`"); + expect(untested.sort()).toEqual(["never.tested", "pay.refund"]); + }); +}); diff --git a/packages/reversi/test/manifest.test.ts b/packages/reversi/test/manifest.test.ts index 71bf669..e618e31 100644 --- a/packages/reversi/test/manifest.test.ts +++ b/packages/reversi/test/manifest.test.ts @@ -31,6 +31,16 @@ const FIXTURES: Array<[string, unknown, boolean]> = [ ["irreversible", { tool: "send_email", effect: "irreversible", permission: "mail:send" }, true], ["irreversible, explicit approval", { tool: "send_email", effect: "irreversible", permission: "mail:send", approval: "always" }, true], ["with requires", { tool: "b", effect: "read_only", permission: "p", requires: ["a"] }, true], + [ + "rich manifest", + { tool: "pay.create", effect: "compensable", permission: "p", version: 2, status: "deprecated", replaced_by: "pay.v2", owner: "o", risk: { data_class: "financial", blast_radius: "high" }, timeout_ms: "30s", idempotency: { key: "{{args.order_id}}", scope: "actor" }, audit: { retention_days: 30, evidence_export: "required", redaction: ["$args.secret"] }, approval: { roles: ["fin"], quorum: 2, timeout: "15m", on_timeout: "escalate" }, compensation: comp("pay.cancel", { kind: "compensation", residue: ["notification sent"], verification: { tool: "pay.get", arguments: { id: "$result.id" }, expect: { status: "canceled" }, retries: 3, backoff_ms: "2s" } }) }, + true, + ], + ["retry on read_only", { tool: "x", effect: "read_only", permission: "p", retry: { attempts: 3 } }, true], + ["retry on compensable", { tool: "x", effect: "compensable", permission: "p", retry: { attempts: 3 }, compensation: comp("y") }, false], + ["bad status", { tool: "x", effect: "read_only", permission: "p", status: "retired" }, false], + ["idempotency without template", { tool: "x", effect: "read_only", permission: "p", idempotency: { key: "plain" } }, false], + ["verification without expect", { tool: "x", effect: "reversible", permission: "p", compensation: comp("y", { verification: { tool: "g" } }) }, false], ["compensable without compensation", { tool: "x", effect: "compensable", permission: "p" }, false], ["reversible without compensation", { tool: "x", effect: "reversible", permission: "p" }, false], ["read_only with compensation", { tool: "x", effect: "read_only", permission: "p", compensation: comp("y") }, false], @@ -156,3 +166,28 @@ describe("resolveRef", () => { expect(resolveRef("$args.__proto__", "args", {})).toBeUndefined(); }); }); + +describe("richer manifests", () => { + it("normalizes defaults and parses durations", () => { + const r = validateManifest({ tool: "x", effect: "reversible", permission: "p", timeout_ms: "30s", compensation: { tool: "y", timeout_ms: "1m", verification: { tool: "g", expect: { a: 1 }, backoff_ms: "2s" } } }); + expect(r.ok && r.manifest).toMatchObject({ version: 1, status: "approved", timeout_ms: 30000, compensation: { kind: "compensation", residue: [], timeout_ms: 60000, verification: { retries: 3, backoff_ms: 2000 } } }); + }); + + it("the approval object normalizes into rules and still forces approval", () => { + const r = validateManifest({ tool: "x", effect: "read_only", permission: "p", approval: { roles: ["fin"], quorum: 2, timeout: "15m", on_timeout: "escalate", escalate_to: ["cfo"] } }); + expect(r.ok && r.manifest).toMatchObject({ approval: "always", approval_rules: { roles: ["fin"], quorum: 2, timeout_ms: 900000, on_timeout: "escalate", escalate_to: ["cfo"] } }); + }); + + it("accepts requires as objects and an inverse without residue", () => { + const r = validateManifest({ tool: "x", effect: "reversible", permission: "p", requires: [{ tool: "a", status: "succeeded" }, "b"], compensation: { tool: "y", kind: "inverse" } }); + expect(r.ok && r.manifest.requires).toEqual(["a", "b"]); + }); + + it("registry checks that a verification tool exists and is read-only, and that replaced_by exists", () => { + const base = { tool: "x", effect: "reversible", permission: "p", compensation: { tool: "y", verification: { tool: "get_x", expect: { gone: true } } } }; + expect(() => Registry.from([base, { tool: "y", effect: "idempotent", permission: "p" }])).toThrow(/verification tool `get_x` is not registered/); + expect(() => Registry.from([base, { tool: "y", effect: "idempotent", permission: "p" }, { tool: "get_x", effect: "idempotent", permission: "p" }])).toThrow(/must be read_only/); + expect(() => Registry.from([{ tool: "old", effect: "read_only", permission: "p", status: "deprecated", replaced_by: "new" }])).toThrow(/replaced_by `new` is not registered/); + expect(Registry.from([base, { tool: "y", effect: "idempotent", permission: "p" }, { tool: "get_x", effect: "read_only", permission: "p" }]).size).toBe(3); + }); +}); diff --git a/packages/reversi/test/notifications.test.ts b/packages/reversi/test/notifications.test.ts new file mode 100644 index 0000000..bb25532 --- /dev/null +++ b/packages/reversi/test/notifications.test.ts @@ -0,0 +1,396 @@ +import { createHmac } from "node:crypto"; +import { createServer as createHttp } from "node:http"; +import { createServer as createTcp, type Socket } from "node:net"; +import type { AddressInfo } from "node:net"; +import { afterEach, describe, expect, it } from "vitest"; +import { + AlertManager, + Approvals, + Ledger, + MemoryKvStore, + Notifier, + OtelExporter, + Policy, + Registry, + Reversi, + SealedScope, + approvalNotification, + attachNotifications, + createChannel, + createLogger, + describeRecord, + evaluateAlerts, + parseChannels, + parseRoutes, + sendMail, + type ChannelConfig, + type NotifyMessage, +} from "../src/index.js"; + +const closers: Array<() => void> = []; +afterEach(() => { + while (closers.length) (closers.pop() as () => void)(); +}); + +async function httpSink(status = 200) { + const got: Array<{ url: string; headers: Record; body: any; raw: string }> = []; + let failNext = 0; + const server = createHttp((req, res) => { + let raw = ""; + req.on("data", (c) => (raw += c)); + req.on("end", () => { + got.push({ url: req.url ?? "", headers: req.headers, raw, body: raw ? JSON.parse(raw) : undefined }); + if (failNext > 0) { + failNext--; + res.writeHead(503).end("busy"); + } else res.writeHead(status).end("ok"); + }); + }); + await new Promise((r) => server.listen(0, "127.0.0.1", r)); + closers.push(() => server.close()); + return { url: `http://127.0.0.1:${(server.address() as AddressInfo).port}`, got, fail: (n: number) => (failNext = n) }; +} + +const msg = (over: Partial = {}): NotifyMessage => ({ event: "approval_requested", severity: "warning", title: "Approval needed: pay", text: "details here", link: "https://reversi.example.com/console/approvals", dedup_key: "approval:apr_1", approval: { id: "apr_1", run_id: "r1", tool: "pay" }, data: { run_id: "r1" }, ...over }); +const noSleep = async () => {}; + +describe("channels", () => { + it("slack: text, blocks, and interactive buttons only when asked", async () => { + const s = await httpSink(); + await createChannel({ type: "slack", name: "s", url: `${s.url}/hook` }).send(msg()); + await createChannel({ type: "slack", name: "s2", url: `${s.url}/hook`, interactive: true }).send(msg()); + expect(s.got[0]?.body.text).toContain("Approval needed: pay"); + expect(s.got[0]?.body.blocks.some((b: any) => b.type === "actions")).toBe(false); + const actions = s.got[1]?.body.blocks.find((b: any) => b.type === "actions"); + expect(actions.elements.map((e: any) => [e.action_id, e.value])).toEqual([["reversi_approve", "apr_1"], ["reversi_deny", "apr_1"]]); + }); + + it("teams: a MessageCard with a link", async () => { + const s = await httpSink(); + await createChannel({ type: "teams", name: "t", url: s.url }).send(msg({ severity: "critical" })); + expect(s.got[0]?.body).toMatchObject({ "@type": "MessageCard", themeColor: "D70000", title: "Approval needed: pay", potentialAction: [{ name: "Open in Reversi" }] }); + }); + + it("pagerduty: trigger then resolve with the same dedup key", async () => { + const s = await httpSink(); + const ch = createChannel({ type: "pagerduty", name: "pd", routing_key: "RK", url: `${s.url}/v2/enqueue` }); + await ch.send(msg({ event: "alert", severity: "critical", title: "Rollback failed", dedup_key: "comp:r1:2" })); + await ch.send(msg({ event: "alert", severity: "info", title: "Resolved", dedup_key: "comp:r1:2", resolve: true })); + expect(s.got[0]?.body).toMatchObject({ routing_key: "RK", event_action: "trigger", dedup_key: "comp:r1:2", payload: { severity: "critical", source: "reversi", summary: "Rollback failed" } }); + expect(s.got[1]?.body).toMatchObject({ event_action: "resolve", dedup_key: "comp:r1:2" }); + }); + + it("opsgenie: creates with GenieKey and closes by alias", async () => { + const s = await httpSink(); + const ch = createChannel({ type: "opsgenie", name: "og", api_key: "KEY123", url: s.url }); + await ch.send(msg({ severity: "critical", dedup_key: "a/b" })); + await ch.send(msg({ resolve: true, dedup_key: "a/b" })); + expect(s.got[0]?.headers.authorization).toBe("GenieKey KEY123"); + expect(s.got[0]).toMatchObject({ url: "/v2/alerts", body: { alias: "a/b", priority: "P1" } }); + expect(s.got[1]?.url).toBe("/v2/alerts/a%2Fb/close?identifierType=alias"); + }); + + it("generic webhook: signed with HMAC-SHA-256 and tagged with the event", async () => { + const s = await httpSink(); + await createChannel({ type: "webhook", name: "w", url: s.url, secret: "shh", headers: { "x-team": "ops" } }).send(msg()); + const r = s.got[0]!; + expect(r.headers["x-reversi-signature"]).toBe(`sha256=${createHmac("sha256", "shh").update(r.raw).digest("hex")}`); + expect(r.headers["x-reversi-event"]).toBe("approval_requested"); + expect(r.headers["x-team"]).toBe("ops"); + expect(r.body).toMatchObject({ event: "approval_requested", severity: "warning", resolve: false }); + }); + + it("retries transient failures, gives up on client errors, and never leaks the key", async () => { + const s = await httpSink(); + s.fail(2); + await createChannel({ type: "webhook", name: "w", url: s.url }, { sleep: noSleep }).send(msg()); + expect(s.got).toHaveLength(3); + const bad = await httpSink(404); + await expect(createChannel({ type: "pagerduty", name: "pd", routing_key: "SECRET-RK", url: `${bad.url}?token=SECRETTOKEN` }, { sleep: noSleep }).send(msg())).rejects.toThrow(/answered 404/); + expect(bad.got).toHaveLength(1); + await expect(createChannel({ type: "webhook", name: "w", url: "http://169.254.169.254/x" }, { sleep: noSleep }).send(msg())).rejects.toThrow(/metadata/); + }); + + it("validates channel and route config", () => { + const errors: string[] = []; + const ch = parseChannels([{ type: "slack", name: "a", url: "not a url" }, { type: "email", name: "a", host: "h" }, { type: "carrier-pigeon", name: "p" }, { type: "opsgenie", name: "o", api_key: "k", region: "mars" }], errors); + expect(ch).toHaveLength(4); + expect(errors.join("\n")).toMatch(/not a valid URL/); + expect(errors.join("\n")).toMatch(/duplicate name/); + expect(errors.join("\n")).toMatch(/port is required/); + expect(errors.join("\n")).toMatch(/to must be a list/); + expect(errors.join("\n")).toMatch(/type must be slack/); + expect(errors.join("\n")).toMatch(/region must be us or eu/); + const e2: string[] = []; + parseRoutes([{ events: [], channels: ["ghost"] }], [{ type: "webhook", name: "w", url: "http://x" } as ChannelConfig], e2); + expect(e2.join("\n")).toMatch(/events must be a list/); + expect(e2.join("\n")).toMatch(/unknown channel `ghost`/); + expect(parseRoutes(undefined, [{ type: "webhook", name: "w", url: "http://x" } as ChannelConfig], [])[0]).toMatchObject({ events: ["*"], channels: ["w"] }); + }); +}); + +describe("notifier routing", () => { + it("routes by event and severity, reports failures, and never throws", async () => { + const a = await httpSink(); + const b = await httpSink(); + const errors: string[] = []; + const n = new Notifier( + [createChannel({ type: "webhook", name: "all", url: a.url }), createChannel({ type: "webhook", name: "pager", url: b.url }), createChannel({ type: "webhook", name: "dead", url: "http://127.0.0.1:1/x" }, { sleep: noSleep, attempts: 1 })], + [{ events: ["*"], channels: ["all"] }, { events: ["rollback_failed"], channels: ["pager", "dead"], min_severity: "critical" }], + (e, c) => errors.push(`${c}: ${e.message.slice(0, 30)}`), + ); + expect(await n.notify(msg({ event: "approval_requested" }))).toEqual({ sent: ["all"], failed: [] }); + expect(await n.notify(msg({ event: "rollback_failed", severity: "warning" }))).toEqual({ sent: ["all"], failed: [] }); // below min severity for the pager route + const r = await n.notify(msg({ event: "rollback_failed", severity: "critical" })); + expect(r.sent.sort()).toEqual(["all", "pager"]); + expect(r.failed).toEqual(["dead"]); + expect(errors).toHaveLength(1); + expect(() => new Notifier([], [{ events: ["*"], channels: ["nope"] }])).toThrow(/unknown channel/); + }); +}); + +describe("SMTP", () => { + function fakeSmtp(opts: { rejectRcpt?: boolean; requireAuth?: boolean } = {}) { + const session: string[] = []; + let data = ""; + const server = createTcp((sock: Socket) => { + let buf = ""; + let inData = false; + sock.on("error", () => {}); + const reply = (s: string) => sock.write(s + "\r\n"); + reply("220 fake ESMTP"); + sock.on("data", (d) => { + buf += d.toString(); + let i: number; + while ((i = buf.indexOf("\r\n")) >= 0) { + const line = buf.slice(0, i); + buf = buf.slice(i + 2); + if (inData) { + if (line === ".") { + inData = false; + reply("250 queued"); + } else data += line + "\n"; + continue; + } + session.push(line.startsWith("AUTH") ? "AUTH " : line); + if (line.startsWith("EHLO")) { + sock.write("250-fake\r\n250 AUTH PLAIN\r\n"); + } else if (line.startsWith("AUTH PLAIN")) reply(Buffer.from(line.slice(11), "base64").toString() === "\u0000mailer\u0000pw" ? "235 ok" : "535 bad credentials"); + else if (line.startsWith("MAIL FROM")) reply("250 ok"); + else if (line.startsWith("RCPT TO")) reply(opts.rejectRcpt ? "550 no such user" : "250 ok"); + else if (line === "DATA") { + inData = true; + reply("354 go"); + } else if (line === "QUIT") sock.end("221 bye\r\n"); + else reply("500 what"); + } + }); + }); + return new Promise<{ port: number; session: string[]; data: () => string }>((r) => + server.listen(0, "127.0.0.1", () => { + closers.push(() => server.close()); + r({ port: (server.address() as AddressInfo).port, session, data: () => data }); + }), + ); + } + + it("sends a message with authentication, base64 body and an encoded subject", async () => { + const s = await fakeSmtp(); + await sendMail({ host: "127.0.0.1", port: s.port, starttls: false, user: "mailer", pass: "pw", from: "reversi@example.com" }, { to: ["a@example.com", "b@example.com"], subject: "Approval needed: café", text: "Line one\nLine two ✓" }); + expect(s.session).toEqual(["EHLO reversi", "AUTH ", "MAIL FROM:", "RCPT TO:", "RCPT TO:", "DATA", "QUIT"]); + const d = s.data(); + expect(d).toContain("Subject: =?UTF-8?B?"); + expect(d).toContain("To: a@example.com, b@example.com"); + const body = d.split("\n\n")[1]!.replace(/\n/g, ""); + expect(Buffer.from(body, "base64").toString("utf8")).toBe("Line one\nLine two ✓"); + }); + + it("surfaces rejections without leaking the password, and refuses header injection", async () => { + const s = await fakeSmtp(); + await expect(sendMail({ host: "127.0.0.1", port: s.port, starttls: false, user: "mailer", pass: "wrong", from: "a@example.com" }, { to: ["x@example.com"], subject: "s", text: "t" })).rejects.toThrow(/SMTP AUTH failed: 535 bad credentials/); + const r = await fakeSmtp({ rejectRcpt: true }); + await expect(sendMail({ host: "127.0.0.1", port: r.port, starttls: false, from: "a@example.com" }, { to: ["x@example.com"], subject: "s", text: "t" })).rejects.toThrow(/550/); + await expect(sendMail({ host: "127.0.0.1", port: s.port, from: "a@example.com" }, { to: ["x@example.com\r\nBcc: evil@example.com"], subject: "s", text: "t" })).rejects.toThrow(/invalid address/); + await expect(sendMail({ host: "127.0.0.1", port: 1, from: "a@example.com" }, { to: ["x@example.com"], subject: "s", text: "t" })).rejects.toThrow(); + }); + + it("the email channel uses it and does not retry a permanent rejection", async () => { + const s = await fakeSmtp({ rejectRcpt: true }); + await expect(createChannel({ type: "email", name: "m", host: "127.0.0.1", port: s.port, starttls: false, from: "a@example.com", to: ["x@example.com"] }, { sleep: noSleep }).send(msg())).rejects.toThrow(/550/); + expect(s.session.filter((l) => l.startsWith("EHLO"))).toHaveLength(1); + const ok = await fakeSmtp(); + await createChannel({ type: "email", name: "m", host: "127.0.0.1", port: ok.port, starttls: false, from: "a@example.com", to: ["x@example.com"] }).send(msg({ severity: "critical" })); + expect(ok.data()).toContain("Subject: [Reversi critical] Approval needed: pay"); + }); +}); + +const MANIFESTS = [ + { tool: "pay", effect: "compensable", permission: "p", compensation: { tool: "refund" } }, + { tool: "refund", effect: "idempotent", permission: "p" }, + { tool: "mail", effect: "irreversible", permission: "p" }, +]; + +function engine(over: object = {}) { + const approvals = new Approvals(); + const rv = new Reversi({ registry: Registry.from(MANIFESTS), ledger: new Ledger(), approver: approvals.asApprover(), kv: new MemoryKvStore(), ...over }); + return { rv, approvals }; +} + +describe("event hub", () => { + it("describes the records a human cares about, and ignores routine ones", async () => { + const { rv, approvals } = engine(); + const seen: NotifyMessage[] = []; + const notifier = { notifyLater: (m: NotifyMessage) => seen.push(m) } as unknown as Notifier; + attachNotifications({ reversi: rv, notifier, console_url: "https://r.example.com" }); + const run = rv.startRun({ run_id: "r1", actor: "agent", scope: SealedScope.seal(["p"]) }); + await run.call({ tool: "unknown", reason: "x", execute: () => 1 }); + approvals.add("r1", "call", "mail", 1, "ops"); + await run.call({ tool: "mail", reason: "x", execute: () => "sent" }); + await run.call({ tool: "pay", reason: "x", execute: () => ({}) }); + await run.rollback({ compensate: () => { throw new Error("gateway timeout"); } }); + expect(seen.map((m) => m.event)).toEqual(["unclassified_mutation", "irreversible_executed", "rollback_failed"]); + expect(seen[0]).toMatchObject({ severity: "warning", dedup_key: "unclassified:unknown", link: "https://r.example.com/console/runs/r1" }); + expect(seen[2]).toMatchObject({ severity: "critical", dedup_key: "comp:r1:1" }); + expect(seen[2]?.text).toContain("gateway timeout"); + }); + + it("builds approval notifications with buttons data and no argument values", () => { + const rec = { id: "apr_9", run_id: "r1", step: null, kind: "call", tool: "pay", reason: "x", triggers: ["policy `big`"], arg_keys: ["amount"], args_hash: "h", status: "pending", requested_ms: 1, expires_ms: 2, decided_by: null, decided_ms: null, decision_note: null, quorum: 2 } as never; + const req = approvalNotification(rec, "requested", "https://r.example.com"); + expect(req).toMatchObject({ event: "approval_requested", approval: { id: "apr_9", tool: "pay" }, link: "https://r.example.com/console/approvals" }); + expect(JSON.stringify(req)).not.toMatch(/amount":\s*\d/); + expect(approvalNotification({ ...(rec as object), status: "approved", decided_by: "alice" } as never, "decided").resolve).toBe(true); + expect(approvalNotification(rec, "escalated").text).toContain("Nobody answered in time"); + }); +}); + +describe("alerts", () => { + async function troubled() { + const { rv, approvals } = engine({ policy: Policy.from({ budgets: [{ id: "daily", match: { tool: "pay" }, metric: "calls", max: 2, window_seconds: 3600, per: "global", warn_at: 0.5 }] }) }); + const run = rv.startRun({ run_id: "r1", actor: "agent", scope: SealedScope.seal(["p"]) }); + await run.call({ tool: "pay", reason: "x", execute: () => ({}) }); + await run.call({ tool: "pay", reason: "x", execute: () => ({}) }); // at 100%: warning recorded + await run.rollback({ compensate: () => { throw new Error("down"); } }); + for (let i = 0; i < 10; i++) await run.call({ tool: "ghost", reason: "x", execute: () => 1 }); + const b = await run.begin({ tool: "pay", reason: "x" }); // blocked by the budget + void b; + approvals.add("r1", "call", "mail", 1); + await run.call({ tool: "mail", reason: "x", execute: () => "sent" }); + rv.halt("incident", { tool: "pay" }); + return { rv, approvals, run }; + } + + it("finds failed rollbacks, spikes, irreversible actions, budgets, unclassified tools, halts and unhealthy upstreams", async () => { + const { rv, approvals } = await troubled(); + const alerts = evaluateAlerts(rv, approvals, { health: [{ name: "github", ok: false }, { name: "fs", ok: true }] }); + const kinds = alerts.map((a) => a.kind); + for (const k of ["compensation_failed", "policy_denial_spike", "irreversible_executed", "budget_threshold", "unclassified_mutation", "halted", "upstream_unhealthy"]) expect(kinds, k).toContain(k); + expect(alerts[0]?.severity).toBe("critical"); // critical first + expect(alerts.find((a) => a.kind === "budget_threshold")?.title).toContain("daily"); + expect(new Set(alerts.map((a) => a.id)).size).toBe(alerts.length); + const quiet = engine(); + expect(evaluateAlerts(quiet.rv, quiet.approvals)).toEqual([]); + }); + + it("flags outcomes unknown for too long and approvals that timed out", async () => { + const { rv, approvals } = engine(); + const run = rv.startRun({ run_id: "r1", actor: "a", scope: SealedScope.seal(["p"]) }); + const b = await run.begin({ tool: "pay", reason: "x" }); + if (b.decision === "allow") run.complete(b.step, { status: "uncertain", error: "timeout" }); + const later = Date.now() + 2 * 3_600_000; + expect(evaluateAlerts(rv, approvals, { now: later }).map((a) => a.kind)).toContain("unresolved_outcome"); + expect(evaluateAlerts(rv, approvals, {}).map((a) => a.kind)).not.toContain("unresolved_outcome"); + }); + + it("the manager sends once, reminds later, and resolves when the condition clears", async () => { + const { rv, approvals } = engine(); + const sink = await httpSink(); + const notifier = new Notifier([createChannel({ type: "webhook", name: "w", url: sink.url })], [{ events: ["alert"], channels: ["w"] }]); + const kv = new MemoryKvStore(); + const mgr = new AlertManager(rv, approvals, kv, notifier, { remind_ms: 1000 }); + const run = rv.startRun({ run_id: "r1", actor: "a", scope: SealedScope.seal(["p"]) }); + await run.call({ tool: "pay", reason: "x", execute: () => ({}) }); + await run.rollback({ compensate: () => { throw new Error("down"); } }); + const t0 = Date.now(); + expect((await mgr.tick(t0)).fired).toEqual(["comp:r1:0"]); + expect((await mgr.tick(t0 + 10)).fired).toEqual([]); // already told + expect((await mgr.tick(t0 + 2000)).fired).toEqual(["comp:r1:0"]); // reminder + expect(sink.got.map((g) => g.body.resolve)).toEqual([false, false]); + // fixed by a retry + await run.rollback({ compensate: () => {} }); + const res = await mgr.tick(t0 + 3000); + expect(res.resolved).toEqual(["comp:r1:0"]); + expect(sink.got.at(-1)?.body).toMatchObject({ resolve: true, dedup_key: "comp:r1:0", severity: "info" }); + expect((await mgr.tick(t0 + 4000)).resolved).toEqual([]); + // a second manager sharing the store does not repeat what the first sent + await run.call({ tool: "pay", reason: "x", execute: () => ({}) }); + await run.rollback({ compensate: () => { throw new Error("down again"); } }); + const other = new AlertManager(rv, approvals, kv, notifier, { remind_ms: 1000 }); + const [a, b] = await Promise.all([mgr.tick(t0 + 5000), other.tick(t0 + 5000)]); + expect(a.fired.length + b.fired.length).toBe(1); + }); +}); + +describe("OpenTelemetry export", () => { + it("exports a span per call and per compensation, with names and ids but no arguments", async () => { + const s = await httpSink(); + const { rv } = engine(); + const otel = new OtelExporter({ endpoint: s.url, headers: { "x-api-key": "k" }, service_version: "1.2.3", environment: "prod", interval_ms: 60_000 }, rv); + otel.start(); + const run = rv.startRun({ run_id: "r1", actor: "agent", scope: SealedScope.seal(["p"]) }); + await run.call({ tool: "pay", args: { card: "4111 1111 1111 1111" }, reason: "customer asked", execute: () => ({ id: 1 }) }); + await run.call({ tool: "ghost", reason: "x", execute: () => 1 }); // blocked + await run.call({ tool: "pay", reason: "x", execute: () => { throw new Error("boom"); } }).catch(() => {}); + await run.rollback({ compensate: () => {} }); + await otel.stop(); + expect(s.got).toHaveLength(1); + expect(s.got[0]?.url).toBe("/v1/traces"); + expect(s.got[0]?.headers["x-api-key"]).toBe("k"); + const rs = s.got[0]?.body.resourceSpans[0]; + const res = Object.fromEntries(rs.resource.attributes.map((a: any) => [a.key, Object.values(a.value)[0]])); + expect(res).toMatchObject({ "service.name": "reversi", "service.version": "1.2.3", "deployment.environment": "prod" }); + const spans = rs.scopeSpans[0].spans; + expect(spans.map((x: any) => x.name)).toEqual(["reversi.call pay", "reversi.call.blocked ghost", "reversi.call pay", "reversi.compensate refund"]); + expect(spans.map((x: any) => x.status.code)).toEqual([1, 2, 2, 1]); + expect(spans[0].traceId).toMatch(/^[0-9a-f]{32}$/); + expect(spans[0].spanId).toMatch(/^[0-9a-f]{16}$/); + expect(new Set(spans.map((x: any) => x.traceId)).size).toBe(1); // one trace per run + expect(new Set(spans.map((x: any) => x.spanId)).size).toBe(4); + expect(BigInt(spans[0].endTimeUnixNano)).toBeGreaterThanOrEqual(BigInt(spans[0].startTimeUnixNano)); + const attrs = Object.fromEntries(spans[0].attributes.map((a: any) => [a.key, Object.values(a.value)[0]])); + expect(attrs).toMatchObject({ "reversi.run_id": "r1", "reversi.tool": "pay", "reversi.effect": "compensable", "reversi.actor": "agent", "reversi.reason": "customer asked" }); + expect(JSON.stringify(s.got[0]?.body)).not.toContain("4111"); + }); + + it("batches, and reports a failing collector without throwing", async () => { + const s = await httpSink(500); + const errors: string[] = []; + const { rv } = engine(); + const otel = new OtelExporter({ endpoint: s.url, batch: 2, onError: (e) => errors.push(e.message) }, rv); + otel.start(); + const run = rv.startRun({ actor: "a", scope: SealedScope.seal(["p"]) }); + await run.call({ tool: "pay", reason: "x", execute: () => 1 }); + expect(s.got).toHaveLength(0); + await run.call({ tool: "pay", reason: "x", execute: () => 1 }); + await new Promise((r) => setTimeout(r, 50)); + expect(s.got).toHaveLength(1); // flushed at two spans + await otel.stop(); + expect(errors[0]).toBe("collector answered 500"); + }); +}); + +describe("structured logger", () => { + it("writes JSON lines, filters by level, and redacts by key and by value", () => { + const lines: string[] = []; + const log = createLogger({ level: "info", write: (l) => lines.push(l), base: { app: "reversi" }, now: () => new Date("2026-10-15T00:00:00Z") }); + log.debug("hidden"); + log.info("request", { path: "/v1/runs", authorization: "Bearer abc", nested: { apiKey: "k", note: "token ghp_" + "a".repeat(36) }, err: new Error("failed with password=hunter2hunter2") }); + log.child({ run: "r1" }).warn("careful", { token: "x" }); + expect(lines).toHaveLength(2); + const a = JSON.parse(lines[0] as string); + expect(a).toMatchObject({ level: "info", time: "2026-10-15T00:00:00.000Z", msg: "request", app: "reversi", path: "/v1/runs", authorization: "[REDACTED]", nested: { apiKey: "[REDACTED]", note: "token [REDACTED:github_token]" } }); + expect(a.err.message).toContain("[REDACTED:credential_assignment]"); + expect(JSON.parse(lines[1] as string)).toMatchObject({ level: "warn", run: "r1", token: "[REDACTED]" }); + expect(lines.join("")).not.toContain("hunter2"); + }); +}); diff --git a/packages/reversi/test/registry-roundtrip.test.ts b/packages/reversi/test/registry-roundtrip.test.ts new file mode 100644 index 0000000..7801918 --- /dev/null +++ b/packages/reversi/test/registry-roundtrip.test.ts @@ -0,0 +1,39 @@ +import { mkdtempSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { describe, expect, it } from "vitest"; +import { ManifestRegistry, authorForm, validateManifest } from "../src/index.js"; + +const M = { tool: "pay.refund", effect: "idempotent", permission: "pay:w", approval: { roles: ["finance"], quorum: 2, timeout: "15m", on_timeout: "escalate", escalate_to: ["cfo"] } }; + +describe("a manifest with an approval object survives the registry", () => { + it("approve, deprecate and restore keep the rules", () => { + const reg = new ManifestRegistry(mkdtempSync(join(tmpdir(), "rv-reg-"))); + reg.publish(M, { by: "a", status: "draft" }); + reg.approve("pay.refund", { by: "b" }); + const cur = reg.current("pay.refund"); + expect(cur?.approval_rules).toMatchObject({ roles: ["finance"], quorum: 2, timeout_ms: 900_000, on_timeout: "escalate", escalate_to: ["cfo"] }); + reg.deprecate("pay.refund", { by: "b" }); + reg.rollbackTo("pay.refund", 2, { by: "b" }); + expect(reg.current("pay.refund")?.approval_rules?.quorum).toBe(2); + expect(reg.verifyAll()).toEqual([]); + }); + + it("authorForm output validates to the same manifest", () => { + const v = validateManifest(M); + if (!v.ok) throw new Error("setup"); + const again = validateManifest(authorForm(v.manifest)); + expect(again.ok && again.manifest).toEqual(v.manifest); + }); +}); + +describe("loading a registry that holds an approval object", () => { + it("load() and a hand-written normalized manifest both validate", () => { + const reg = new ManifestRegistry(mkdtempSync(join(tmpdir(), "rv-reg-"))); + reg.publish(M, { by: "a", status: "approved" }); + const loaded = reg.load(); + expect(loaded.get("pay.refund")?.approval_rules?.quorum).toBe(2); + const v = validateManifest(loaded.get("pay.refund")); + expect(v.ok).toBe(true); + }); +}); diff --git a/packages/sdk-py/pyproject.toml b/packages/sdk-py/pyproject.toml index 0701117..e9d005e 100644 --- a/packages/sdk-py/pyproject.toml +++ b/packages/sdk-py/pyproject.toml @@ -20,5 +20,11 @@ Homepage = "https://reversi.multimind.dev" Documentation = "https://reversi-docs.multimind.dev" Repository = "https://github.com/multimindlab/reversi" +[project.optional-dependencies] +pydantic = ["pydantic>=2"] + +[project.scripts] +reversi-claude-hook = "reversi_sdk.integrations.claude_code_hook:main" + [tool.setuptools] -packages = ["reversi_sdk"] +packages = ["reversi_sdk", "reversi_sdk.integrations"] diff --git a/packages/sdk-py/reversi_sdk/__init__.py b/packages/sdk-py/reversi_sdk/__init__.py index 86ffe48..e7bf496 100644 --- a/packages/sdk-py/reversi_sdk/__init__.py +++ b/packages/sdk-py/reversi_sdk/__init__.py @@ -1,6 +1,6 @@ """Reversi Python SDK.""" from .canonical import canonicalize, hash_payload, sha256_hex -from .client import Blocked, EffectUncertain, ReversiClient, ReversiError, RunHandle +from .client import Blocked, EffectUncertain, ReversiClient, ReversiError, RunHandle, verify_webhook_signature from .ledger import GENESIS_HASH, VerifyResult, parse_jsonl, verify_file, verify_records from .manifest import EFFECTS, ManifestError, Registry, validate_manifest @@ -8,5 +8,5 @@ __all__ = [ "Blocked", "EFFECTS", "EffectUncertain", "GENESIS_HASH", "ManifestError", "Registry", "ReversiClient", "ReversiError", "RunHandle", "VerifyResult", "canonicalize", "hash_payload", "parse_jsonl", "sha256_hex", - "validate_manifest", "verify_file", "verify_records", + "validate_manifest", "verify_file", "verify_records", "verify_webhook_signature", ] diff --git a/packages/sdk-py/reversi_sdk/client.py b/packages/sdk-py/reversi_sdk/client.py index cbca023..740bf09 100644 --- a/packages/sdk-py/reversi_sdk/client.py +++ b/packages/sdk-py/reversi_sdk/client.py @@ -1,13 +1,15 @@ """HTTP client for the Reversi server (standard library only).""" from __future__ import annotations +import hashlib +import hmac import json import random import re import time import urllib.error import urllib.request -from typing import Any, Callable, Dict, List, Optional +from typing import Any, Callable, Dict, List, Optional, Union class ReversiError(Exception): @@ -40,6 +42,15 @@ def approval_id(self) -> Optional[str]: return m.group(1) if m else None +def verify_webhook_signature(secret: Union[str, bytes], body: Union[str, bytes], signature_header: Optional[str]) -> bool: + if not signature_header or not signature_header.startswith("sha256="): + return False + key = secret.encode("utf-8") if isinstance(secret, str) else secret + data = body.encode("utf-8") if isinstance(body, str) else body + expected = hmac.new(key, data, hashlib.sha256).hexdigest() + return hmac.compare_digest(expected.encode("ascii"), signature_header[len("sha256="):].strip().lower().encode("utf-8", "replace")) + + class ReversiClient: def __init__(self, base_url: str, api_key: str, timeout: float = 10.0): self.base_url = base_url.rstrip("/") @@ -64,6 +75,14 @@ def _request(self, method: str, path: str, body: Any = None, raw: bool = False) raise ReversiError(e.code, msg) from None return text if raw else json.loads(text) + @staticmethod + def verify_webhook_signature(secret: Union[str, bytes], body: Union[str, bytes], signature_header: Optional[str]) -> bool: + """True if `signature_header` (`sha256=`) is the HMAC-SHA256 of the raw request body under `secret`. + + Pass the body exactly as received (bytes), before any JSON parsing. + """ + return verify_webhook_signature(secret, body, signature_header) + # --- agent plane --- def start_run(self, permissions: List[str], run_id: Optional[str] = None, mode: str = "enforce", reason: Optional[str] = None) -> "RunHandle": diff --git a/packages/sdk-py/reversi_sdk/integrations/__init__.py b/packages/sdk-py/reversi_sdk/integrations/__init__.py new file mode 100644 index 0000000..d196246 --- /dev/null +++ b/packages/sdk-py/reversi_sdk/integrations/__init__.py @@ -0,0 +1 @@ +"""Framework adapters. Each module imports its framework lazily, so `import reversi_sdk` never needs one.""" diff --git a/packages/sdk-py/reversi_sdk/integrations/_common.py b/packages/sdk-py/reversi_sdk/integrations/_common.py new file mode 100644 index 0000000..801458c --- /dev/null +++ b/packages/sdk-py/reversi_sdk/integrations/_common.py @@ -0,0 +1,86 @@ +"""Shared plumbing for the adapters: journaled calls, block messages, JSON-safe args.""" +from __future__ import annotations + +import asyncio +import json +from typing import Any, Callable, Dict, Tuple + +from ..client import Blocked, EffectUncertain + +_CONTEXT_TYPES = {"RunContextWrapper", "ToolContext", "Context", "RunnableConfig"} + + +def check_on_block(on_block: str) -> str: + if on_block not in ("message", "raise"): + raise ValueError('on_block must be "message" or "raise"') + return on_block + + +def jsonable(v: Any) -> Any: + """Round-trip through JSON so the journal never fails on framework objects (they become their repr).""" + try: + return json.loads(json.dumps(v, default=repr)) + except (TypeError, ValueError): + return repr(v) + + +def journal_args(args: Tuple[Any, ...], kwargs: Dict[str, Any]) -> Any: + """What to record as the call's arguments: kwargs, or a single dict positional, without framework contexts.""" + args = tuple(a for a in args if type(a).__name__ not in _CONTEXT_TYPES) + kwargs = {k: v for k, v in kwargs.items() if type(v).__name__ not in _CONTEXT_TYPES} + if not args: + return jsonable(kwargs) + if len(args) == 1 and not kwargs and isinstance(args[0], dict): + return jsonable(args[0]) + return jsonable({"args": list(args), "kwargs": kwargs}) + + +def block_message(b: Blocked, tool: str) -> str: + text = f"Reversi blocked the call to `{tool}`: {'; '.join(b.reasons)}. The tool did not run." + if b.approval_id: + text += (f" Approval request {b.approval_id} is pending. Ask an approver to approve it, " + "then retry the same call once.") + return text + + +def call_sync(run: Any, tool: str, reason: str, fn: Callable[[], Any], args: Any, on_block: str) -> Any: + holder: list = [] + + def execute(_: Any) -> Any: + r = fn() + holder.append(r) + return jsonable(r) + + try: + run.call(tool, reason, execute, args=args) + except Blocked as b: + if on_block == "raise": + raise + return block_message(b, tool) + return holder[0] + + +async def call_async(run: Any, tool: str, reason: str, fn: Callable[[], Any], args: Any, on_block: str) -> Any: + """Same journaling as RunHandle.call for a coroutine function; HTTP runs in a worker thread.""" + loop = asyncio.get_running_loop() + d = await loop.run_in_executor(None, lambda: run.begin(tool, reason, args)) + if d["decision"] == "block": + b = Blocked(d["reasons"]) + if on_block == "raise": + raise b + return block_message(b, tool) + step = d["step"] + + async def done(status: str, **kw: Any) -> None: + await loop.run_in_executor(None, lambda: run.complete(step, status, **kw)) + + try: + result = await fn() + except EffectUncertain as e: + await done("uncertain", error=str(e)[:300]) + raise + except Exception as e: + await done("failed", error=str(e)[:300]) + raise + await done("succeeded", result=jsonable(result)) + return result diff --git a/packages/sdk-py/reversi_sdk/integrations/claude_code_hook.py b/packages/sdk-py/reversi_sdk/integrations/claude_code_hook.py new file mode 100644 index 0000000..fceeb3a --- /dev/null +++ b/packages/sdk-py/reversi_sdk/integrations/claude_code_hook.py @@ -0,0 +1,178 @@ +"""Claude Code PreToolUse / PostToolUse hook backed by a Reversi run per Claude session. + +Pre: `reversi-claude-hook` reads the hook JSON on stdin, asks Reversi, prints the permission decision. +Post: `reversi-claude-hook --post` reports the outcome of the step recorded at Pre time. + +Environment: REVERSI_URL, REVERSI_AGENT_KEY, REVERSI_PERMISSIONS (comma list), REVERSI_HOOK_FAIL_OPEN=1, +REVERSI_HOOK_STATE (state directory), REVERSI_TIMEOUT (seconds, default 5). +""" +from __future__ import annotations + +import hashlib +import json +import os +import re +import sys +import tempfile +from typing import Any, Dict, Mapping, Optional, TextIO + +from ..client import ReversiClient, ReversiError, Blocked + + +def map_tool_name(name: str) -> str: + """`mcp__server__tool` becomes `server.tool`; every other name is used as is.""" + m = re.fullmatch(r"mcp__(.+?)__(.+)", name) + return f"{m.group(1)}.{m.group(2)}" if m else name + + +def _state_dir(env: Mapping[str, str]) -> str: + d = env.get("REVERSI_HOOK_STATE") or os.path.join(tempfile.gettempdir(), "reversi-claude-hook") + os.makedirs(d, exist_ok=True) + return d + + +def _state_path(env: Mapping[str, str], session_id: str) -> str: + return os.path.join(_state_dir(env), re.sub(r"[^A-Za-z0-9_-]", "_", session_id) + ".json") + + +def _call_key(hook: Dict[str, Any]) -> str: + tid = hook.get("tool_use_id") + if isinstance(tid, str) and tid: + return tid + digest = hashlib.sha256(json.dumps(hook.get("tool_input"), sort_keys=True, default=repr).encode()).hexdigest()[:16] + return f"{hook.get('tool_name')}:{digest}" + + +def _load(path: str) -> Dict[str, Any]: + try: + with open(path, encoding="utf-8") as f: + d = json.load(f) + return d if isinstance(d, dict) else {} + except (OSError, ValueError): + return {} + + +def _save(path: str, data: Dict[str, Any]) -> None: + tmp = f"{path}.{os.getpid()}.tmp" + with open(tmp, "w", encoding="utf-8") as f: + json.dump(data, f) + os.replace(tmp, path) + + +def _decision(kind: str, reason: str) -> Dict[str, Any]: + return {"hookSpecificOutput": {"hookEventName": "PreToolUse", "permissionDecision": kind, + "permissionDecisionReason": reason}} + + +def _client(env: Mapping[str, str]) -> ReversiClient: + url, key = env.get("REVERSI_URL"), env.get("REVERSI_AGENT_KEY") + if not url or not key: + raise RuntimeError("REVERSI_URL and REVERSI_AGENT_KEY must be set") + return ReversiClient(url, key, timeout=float(env.get("REVERSI_TIMEOUT") or 5)) + + +def _ensure_run(client: ReversiClient, run_id: str, env: Mapping[str, str]) -> Any: + run = client.run(run_id) + try: + run.state() + return run + except ReversiError as e: + if e.status != 404: + raise + perms = [p.strip() for p in (env.get("REVERSI_PERMISSIONS") or "").split(",") if p.strip()] + try: + return client.start_run(perms, run_id=run_id, reason="Claude Code session") + except ReversiError as e: + if e.status == 409: # another hook process opened it first + return run + raise + + +def pre_tool_use(hook: Dict[str, Any], env: Mapping[str, str]) -> Dict[str, Any]: + fail_open = env.get("REVERSI_HOOK_FAIL_OPEN") == "1" + try: + session = str(hook.get("session_id") or "") + tool_name = hook.get("tool_name") + if not session or not isinstance(tool_name, str): + raise ValueError("hook input needs session_id and tool_name") + tool_input = hook.get("tool_input") + run = _ensure_run(_client(env), f"claude-{session}", env) + tool = map_tool_name(tool_name) + reason = "" + if isinstance(tool_input, dict) and isinstance(tool_input.get("description"), str): + reason = tool_input["description"] + d = run.begin(tool, (reason or f"Claude Code called {tool_name}")[:200], tool_input) + except Exception as e: # noqa: BLE001 - anything wrong means Reversi could not vouch for the call + if fail_open: + return _decision("allow", f"Reversi unavailable ({e}); REVERSI_HOOK_FAIL_OPEN=1 so the call is allowed unrecorded") + return _decision("deny", f"Reversi unavailable ({e}); failing closed") + + if d["decision"] in ("allow", "would_block"): + path = _state_path(env, session) + state = _load(path) + state[_call_key(hook)] = d["step"] + _save(path, state) + note = "; ".join(d.get("reasons", [])) + return _decision("allow", f"Reversi step {d['step']}" + (f" (would block: {note})" if note else "")) + b = Blocked(d.get("reasons", [])) + if b.approval_id: + return _decision("ask", f"Reversi requires approval: request {b.approval_id} is pending. " + f"Have a Reversi approver approve it, then retry. ({'; '.join(b.reasons)})") + return _decision("deny", "Reversi blocked this call: " + "; ".join(b.reasons)) + + +def post_tool_use(hook: Dict[str, Any], env: Mapping[str, str]) -> Optional[str]: + """Report the outcome for the step recorded at Pre time. Returns a message for stderr on trouble, else None.""" + session = str(hook.get("session_id") or "") + if not session: + return "no session_id" + path = _state_path(env, session) + state = _load(path) + key = _call_key(hook) + if key not in state: + return None # no Pre record (allowed by Claude Code's own prompt, or the hook was not installed then) + step = state[key] + resp = hook.get("tool_response") + failed = hook.get("hook_event_name") == "PostToolUseFailure" or ( + isinstance(resp, dict) and (resp.get("is_error") is True or bool(resp.get("error")))) + run = _client(env).run(f"claude-{session}") + if failed: + err = hook.get("error") or (resp.get("error") if isinstance(resp, dict) else None) or "tool failed" + run.complete(step, "failed", error=str(err)[:300]) + else: + small = isinstance(resp, dict) and len(json.dumps(resp, default=repr)) < 100_000 + run.complete(step, "succeeded", result=resp if small else None) + del state[key] + _save(path, state) + return None + + +def main(argv: Optional[list] = None, stdin: Optional[TextIO] = None, stdout: Optional[TextIO] = None, + env: Optional[Mapping[str, str]] = None) -> int: + argv = sys.argv[1:] if argv is None else argv + stdin, stdout, env = stdin or sys.stdin, stdout or sys.stdout, os.environ if env is None else env + post = "--post" in argv + try: + hook = json.loads(stdin.read()) + if not isinstance(hook, dict): + raise ValueError("hook input must be a JSON object") + except ValueError as e: + if not post: + stdout.write(json.dumps(_decision("deny", f"unreadable hook input: {e}")) + "\n") + else: + sys.stderr.write(f"reversi-claude-hook: unreadable hook input: {e}\n") + return 0 + if post: + try: + msg = post_tool_use(hook, env) + except Exception as e: # noqa: BLE001 - the tool already ran; never fail the session over reporting + msg = str(e) + if msg: + sys.stderr.write(f"reversi-claude-hook: could not report outcome: {msg}\n") + return 0 + stdout.write(json.dumps(pre_tool_use(hook, env)) + "\n") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/packages/sdk-py/reversi_sdk/integrations/crewai.py b/packages/sdk-py/reversi_sdk/integrations/crewai.py new file mode 100644 index 0000000..d8d3203 --- /dev/null +++ b/packages/sdk-py/reversi_sdk/integrations/crewai.py @@ -0,0 +1,53 @@ +"""CrewAI adapter, duck-typed on `crewai.tools.BaseTool` (`name`, `description`, `_run`).""" +from __future__ import annotations + +from typing import Any, Optional + +from ._common import call_sync, check_on_block, journal_args + + +class ReversibleCrewTool: + """Journals every `_run` through a Reversi run. + + Wrap an existing tool (anything with `_run` or `run`), or subclass and override `_execute`:: + + tool = ReversibleCrewTool(run, "lookup_order", inner=MyCrewTool(), reason="support ticket") + agent = Agent(..., tools=[tool.to_crewai()]) + """ + + def __init__(self, run: Any, tool_name: str, inner: Any = None, name: Optional[str] = None, + description: Optional[str] = None, reason: str = "agent tool call", on_block: str = "message"): + self.run_handle = run + self.tool_name = tool_name + self.inner = inner + self.name = name or getattr(inner, "name", tool_name) + self.description = description or getattr(inner, "description", "") + self.args_schema = getattr(inner, "args_schema", None) + self.reason = reason + self.on_block = check_on_block(on_block) + + def _execute(self, *args: Any, **kwargs: Any) -> Any: + if self.inner is None: + raise NotImplementedError("pass inner=... or override _execute") + fn = getattr(self.inner, "_run", None) or self.inner.run + return fn(*args, **kwargs) + + def _run(self, *args: Any, **kwargs: Any) -> Any: + return call_sync(self.run_handle, self.tool_name, self.reason, lambda: self._execute(*args, **kwargs), + journal_args(args, kwargs), self.on_block) + + def run(self, *args: Any, **kwargs: Any) -> Any: + return self._run(*args, **kwargs) + + def to_crewai(self) -> Any: + """Build a real `crewai.tools.BaseTool` that delegates to this adapter (imports crewai lazily).""" + from crewai.tools import BaseTool # type: ignore[import-not-found] + + outer = self + attrs = {"__annotations__": {"name": str, "description": str}, + "name": self.name, "description": self.description, + "_run": lambda s, *a, **kw: outer._run(*a, **kw)} + if self.args_schema is not None: + attrs["__annotations__"]["args_schema"] = type(self.args_schema) + attrs["args_schema"] = self.args_schema + return type("Reversible" + "".join(c for c in self.name.title() if c.isalnum()), (BaseTool,), attrs)() diff --git a/packages/sdk-py/reversi_sdk/integrations/fastmcp.py b/packages/sdk-py/reversi_sdk/integrations/fastmcp.py new file mode 100644 index 0000000..cab8e07 --- /dev/null +++ b/packages/sdk-py/reversi_sdk/integrations/fastmcp.py @@ -0,0 +1,93 @@ +"""FastMCP adapter: make `@mcp.tool()` functions run through a Reversi run (duck-typed on the server object).""" +from __future__ import annotations + +import functools +import inspect +import threading +import uuid +from typing import Any, Callable, Dict, List, Optional + +from ..client import Blocked, ReversiClient +from ._common import block_message, call_async, call_sync, journal_args + + +class ReversiBlocked(Exception): + """Raised from a tool when Reversi refused the call. FastMCP returns the message to the client as a tool error.""" + + def __init__(self, message: str, approval_id: Optional[str] = None, reasons: Optional[List[str]] = None): + super().__init__(message) + self.approval_id = approval_id + self.reasons = reasons or [] + + +def _find_ctx(args: tuple, kwargs: dict) -> Any: + for k in ("ctx", "context"): + if k in kwargs: + return kwargs[k] + for v in list(kwargs.values()) + list(args): + if type(v).__name__ == "Context": + return v + return None + + +def _as_blocked(tool: str, e: Blocked) -> ReversiBlocked: + return ReversiBlocked(block_message(e, tool), e.approval_id, e.reasons) + + +def reversible(mcp: Any, run_factory: Callable[[Any], Any], reason: str = "MCP tool call", + tool_name: Optional[str] = None, **tool_kwargs: Any) -> Callable[[Callable[..., Any]], Callable[..., Any]]: + """Decorator replacing `@mcp.tool(**tool_kwargs)`:: + + @reversible(mcp, per_session_runs(client, ["orders:write"])) + def create_order(sku: str) -> dict: ... + + `run_factory(ctx)` returns the RunHandle for this call; ctx is the tool's FastMCP `Context` (a parameter named + `ctx` or `context`) or None. Pass `mcp=None` to only wrap, without registering. A refusal raises `ReversiBlocked`. + """ + def deco(fn: Callable[..., Any]) -> Callable[..., Any]: + name = tool_name or fn.__name__ + if inspect.iscoroutinefunction(fn): + @functools.wraps(fn) + async def awrapper(*a: Any, **kw: Any) -> Any: + run = run_factory(_find_ctx(a, kw)) + try: + return await call_async(run, name, reason, lambda: fn(*a, **kw), journal_args(a, kw), "raise") + except Blocked as b: + raise _as_blocked(name, b) from None + wrapped: Callable[..., Any] = awrapper + else: + @functools.wraps(fn) + def wrapper(*a: Any, **kw: Any) -> Any: + run = run_factory(_find_ctx(a, kw)) + try: + return call_sync(run, name, reason, lambda: fn(*a, **kw), journal_args(a, kw), "raise") + except Blocked as b: + raise _as_blocked(name, b) from None + wrapped = wrapper + if mcp is not None: + registered = mcp.tool(**tool_kwargs)(wrapped) + return registered if registered is not None else wrapped + return wrapped + + return deco + + +def per_session_runs(client: ReversiClient, permissions: List[str], prefix: str = "mcp") -> Callable[[Any], Any]: + """A `run_factory` that opens one run per MCP session (keyed by the session object) and reuses it.""" + runs: Dict[Any, Any] = {} + lock = threading.Lock() + + def factory(ctx: Any) -> Any: + key: Any = "default" + if ctx is not None: + try: + key = id(ctx.session) + except Exception: # noqa: BLE001 - not a real Context: share one run + key = "default" + with lock: + if key not in runs: + runs[key] = client.start_run(permissions, run_id=f"{prefix}-{uuid.uuid4().hex[:12]}", + reason="MCP session") + return runs[key] + + return factory diff --git a/packages/sdk-py/reversi_sdk/integrations/langgraph.py b/packages/sdk-py/reversi_sdk/integrations/langgraph.py new file mode 100644 index 0000000..542a5a6 --- /dev/null +++ b/packages/sdk-py/reversi_sdk/integrations/langgraph.py @@ -0,0 +1,81 @@ +"""LangChain / LangGraph adapter (duck-typed: no langchain import is needed).""" +from __future__ import annotations + +import functools +import inspect +from typing import Any, Callable, Dict + +from ._common import call_async, call_sync, check_on_block, journal_args + + +class _ReversibleTool: + """Proxy for a BaseTool-like object: `.name`, `.invoke(input)` (or callable). Other attributes pass through.""" + + def __init__(self, run: Any, tool_name: str, reason: str, on_block: str, target: Any): + self._run = run + self._tool_name = tool_name + self._reason = reason + self._on_block = on_block + self._target = target + + def __getattr__(self, item: str) -> Any: + return getattr(self._target, item) + + def invoke(self, input: Any, *a: Any, **kw: Any) -> Any: + fn = self._target.invoke if hasattr(self._target, "invoke") else self._target + return call_sync(self._run, self._tool_name, self._reason, lambda: fn(input, *a, **kw), + journal_args((input,), {}), self._on_block) + + async def ainvoke(self, input: Any, *a: Any, **kw: Any) -> Any: + if hasattr(self._target, "ainvoke"): + fn = self._target.ainvoke + return await call_async(self._run, self._tool_name, self._reason, lambda: fn(input, *a, **kw), + journal_args((input,), {}), self._on_block) + return self.invoke(input, *a, **kw) + + def __call__(self, *a: Any, **kw: Any) -> Any: + if hasattr(self._target, "invoke") and len(a) == 1 and not kw: + return self.invoke(a[0]) + return call_sync(self._run, self._tool_name, self._reason, lambda: self._target(*a, **kw), + journal_args(a, kw), self._on_block) + + +def reversible_tool(run: Any, tool_name: str, reason: str = "agent tool call", on_block: str = "message", + tool: Any = None) -> Any: + """Route every call of a tool through `run.call`. + + Use as a decorator on the plain function, *before* `@tool` so LangChain still sees the signature:: + + @tool + @reversible_tool(run, "create_order", reason="customer asked") + def create_order(sku: str) -> dict: ... + + or wrap an existing BaseTool-like object (`reversible_tool(run, "x")(my_tool)`, or pass `tool=`). + A blocked call returns an explanatory string (`on_block="message"`) or raises `Blocked` (`on_block="raise"`). + """ + check_on_block(on_block) + + def wrap(target: Any) -> Any: + if hasattr(target, "invoke") or not callable(target): + return _ReversibleTool(run, tool_name, reason, on_block, target) + if inspect.iscoroutinefunction(target): + @functools.wraps(target) + async def awrapper(*a: Any, **kw: Any) -> Any: + return await call_async(run, tool_name, reason, lambda: target(*a, **kw), journal_args(a, kw), on_block) + return awrapper + + @functools.wraps(target) + def wrapper(*a: Any, **kw: Any) -> Any: + return call_sync(run, tool_name, reason, lambda: target(*a, **kw), journal_args(a, kw), on_block) + return wrapper + + return wrap(tool) if tool is not None else wrap + + +def rollback_node(client: Any, run_id: str, compensate: Callable[[Dict[str, Any]], None], + state_key: str = "reversi_rollback", **rollback_kwargs: Any) -> Callable[[Any], Dict[str, Any]]: + """A LangGraph node that undoes the run (operator key needed). Returns `{state_key: report}`.""" + def node(state: Any = None) -> Dict[str, Any]: + return {state_key: client.run(run_id).rollback(compensate, **rollback_kwargs)} + node.__name__ = "reversi_rollback" + return node diff --git a/packages/sdk-py/reversi_sdk/integrations/openai_agents.py b/packages/sdk-py/reversi_sdk/integrations/openai_agents.py new file mode 100644 index 0000000..3b43191 --- /dev/null +++ b/packages/sdk-py/reversi_sdk/integrations/openai_agents.py @@ -0,0 +1,92 @@ +"""OpenAI Agents SDK adapter (duck-typed; the `agents` package is imported lazily, only by `guardrail`).""" +from __future__ import annotations + +import asyncio +import dataclasses +import functools +import inspect +import json +from typing import Any, Callable + +from ..client import Blocked, ReversiClient +from ._common import block_message, call_async, call_sync, check_on_block, journal_args + + +def reversi_function_tool(run: Any, tool_name: str, reason: str = "agent tool call", + on_block: str = "message") -> Callable[[Any], Any]: + """Wrap the function you pass to `@function_tool` (put this decorator *under* it):: + + @function_tool + @reversi_function_tool(run, "create_order") + def create_order(sku: str) -> str: ... + + It also accepts a ready `FunctionTool` (it wraps `on_invoke_tool(ctx, input_json)`). + Use `on_block="raise"` with `tool_error_formatter` to let the SDK turn a block into model-visible text. + """ + check_on_block(on_block) + + def deco(target: Any) -> Any: + if hasattr(target, "on_invoke_tool"): + inner = target.on_invoke_tool + + async def on_invoke(ctx: Any, input_json: str) -> Any: + try: + args = json.loads(input_json) if input_json else {} + except ValueError: + args = {"raw": input_json} + return await call_async(run, tool_name, reason, lambda: inner(ctx, input_json), args, on_block) + + if dataclasses.is_dataclass(target): + return dataclasses.replace(target, on_invoke_tool=on_invoke) + target.on_invoke_tool = on_invoke + return target + if inspect.iscoroutinefunction(target): + @functools.wraps(target) + async def awrapper(*a: Any, **kw: Any) -> Any: + return await call_async(run, tool_name, reason, lambda: target(*a, **kw), journal_args(a, kw), on_block) + return awrapper + + @functools.wraps(target) + def wrapper(*a: Any, **kw: Any) -> Any: + return call_sync(run, tool_name, reason, lambda: target(*a, **kw), journal_args(a, kw), on_block) + return wrapper + + return deco + + +def tool_error_formatter(ctx: Any, error: Exception) -> str: + """For `function_tool(failure_error_function=...)`: a `Blocked` becomes an instruction to the model, not a crash.""" + if isinstance(error, Blocked): + return block_message(error, "this tool") + return f"An error occurred while running the tool: {error}" + + +class GuardrailResult: + """Stand-in for `agents.GuardrailFunctionOutput` when the package is not installed.""" + + def __init__(self, output_info: Any, tripwire_triggered: bool): + self.output_info = output_info + self.tripwire_triggered = tripwire_triggered + + +def guardrail(client: ReversiClient, run: Any = None) -> Callable[..., Any]: + """Input guardrail that trips while the Reversi kill switch is on (the key must be allowed to read `/v1/status`):: + + Agent(..., input_guardrails=[InputGuardrail(guardrail_function=guardrail(client, run))]) + + If Reversi cannot be reached the guardrail trips (fails closed). + """ + async def reversi_kill_switch(ctx: Any = None, agent: Any = None, input: Any = None) -> Any: + try: + st = await asyncio.get_running_loop().run_in_executor(None, client.status) + halted = bool(st.get("halted")) + info = {"halted": halted, "run_id": getattr(run, "run_id", None)} + except Exception as e: # noqa: BLE001 + halted, info = True, {"error": f"could not reach Reversi: {e}"} + try: + from agents import GuardrailFunctionOutput # type: ignore[import-not-found] + except ImportError: + return GuardrailResult(info, halted) + return GuardrailFunctionOutput(output_info=info, tripwire_triggered=halted) + + return reversi_kill_switch diff --git a/packages/sdk-py/reversi_sdk/manifest.py b/packages/sdk-py/reversi_sdk/manifest.py index 00c5f14..560f10d 100644 --- a/packages/sdk-py/reversi_sdk/manifest.py +++ b/packages/sdk-py/reversi_sdk/manifest.py @@ -9,8 +9,31 @@ _TOOL = re.compile(r"[A-Za-z0-9_][A-Za-z0-9_.:/-]{0,127}") _PERM = re.compile(r"\S{1,128}") _REF = re.compile(r"\$(args|result)(\.[A-Za-z0-9_-]+)+") -_TOP = {"tool", "description", "effect", "permission", "approval", "compensation", "requires"} -_COMP = {"tool", "argument_map", "requires_approval"} +_TOP = {"tool", "description", "effect", "permission", "approval", "compensation", "requires", "version", "status", + "replaced_by", "owner", "oncall", "risk", "timeout_ms", "retry", "idempotency", "audit", "schema_hash"} +_COMP = {"tool", "argument_map", "requires_approval", "kind", "residue", "verification", "timeout_ms"} +_VERIFY = {"tool", "arguments", "expect", "retries", "backoff_ms"} +BLAST_RADII = ("low", "medium", "high", "critical") +_DURATION = re.compile(r"(\d+(?:\.\d+)?)\s*(ms|s|m|h|d)") +_MULT = {"ms": 1, "s": 1000, "m": 60_000, "h": 3_600_000, "d": 86_400_000} +_IDEM = re.compile(r"\{\{\s*args(\.[A-Za-z0-9_-]+)+\s*\}\}") +_REDACT = re.compile(r"\$(args|result)(\.[A-Za-z0-9_-]+)+") + + +def parse_duration(v: Any) -> Optional[int]: + """'15m', '30s', '2h', '500ms', '1d' or a number of milliseconds. None if invalid.""" + if isinstance(v, bool): + return None + if isinstance(v, (int, float)): + return int(v) if v == v and v not in (float("inf"), float("-inf")) and v > 0 else None + if not isinstance(v, str): + return None + m = _DURATION.fullmatch(v.strip()) + return int(float(m.group(1)) * _MULT[m.group(2)]) if m else None + + +def _is_int(x: Any) -> bool: + return isinstance(x, int) and not isinstance(x, bool) class ManifestError(ValueError): @@ -46,11 +69,44 @@ def err(m: str) -> None: err(f"`effect` must be one of {', '.join(EFFECTS)}") approval = "never" + approval_rules: Optional[Dict[str, Any]] = None if "approval" in raw: - if raw["approval"] in ("never", "always") and isinstance(raw["approval"], str): - approval = raw["approval"] + ap = raw["approval"] + if isinstance(ap, str) and ap in ("never", "always"): + approval = ap + elif isinstance(ap, dict): + for k in ap: + if k not in ("required", "roles", "quorum", "timeout", "on_timeout", "escalate_to"): + err(f"unknown approval field `{k}`") + if "required" in ap and not isinstance(ap["required"], bool): + err("`approval.required` must be a boolean") + approval = "never" if ap.get("required") is False else "always" + roles = ap.get("roles", []) + if not isinstance(roles, list) or any(not isinstance(r, str) or not r for r in roles): + err("`approval.roles` must be a list of role or group names") + quorum = ap.get("quorum", 1) + if not _is_int(quorum) or quorum < 1 or quorum > 10: + err("`approval.quorum` must be an integer from 1 to 10") + timeout_ms = None + if "timeout" in ap: + timeout_ms = parse_duration(ap["timeout"]) + if timeout_ms is None: + err("`approval.timeout` must be a duration like 15m, 30s or 2h") + if "on_timeout" in ap and ap["on_timeout"] not in ("deny", "escalate"): + err("`approval.on_timeout` must be deny or escalate") + if isinstance(roles, list) and _is_int(quorum) and quorum > 1 and len(roles) == 0: + err("a quorum needs `approval.roles` to draw approvers from") + approval_rules = { + "roles": list(roles) if isinstance(roles, list) else [], + "quorum": quorum if _is_int(quorum) else 1, + "on_timeout": "escalate" if ap.get("on_timeout") == "escalate" else "deny", + } + if timeout_ms is not None: + approval_rules["timeout_ms"] = timeout_ms + if isinstance(ap.get("escalate_to"), list): + approval_rules["escalate_to"] = ap["escalate_to"] else: - err("`approval` must be `never` or `always`") + err("`approval` must be `never`, `always` or an object with roles, quorum and timeout") if effect == "irreversible": if raw.get("approval") == "never": err("irreversible tools must require approval") @@ -59,10 +115,19 @@ def err(m: str) -> None: requires: List[str] = [] if "requires" in raw: r = raw["requires"] - if not isinstance(r, list) or any(not isinstance(x, str) or not x for x in r): - err("`requires` must be a list of tool names") + normalized = [] + if isinstance(r, list): + for x in r: + if isinstance(x, str): + normalized.append(x) + elif isinstance(x, dict) and isinstance(x.get("tool"), str) and x.get("status", "succeeded") == "succeeded": + normalized.append(x["tool"]) + else: + normalized.append("") + if not isinstance(r, list) or any(not x for x in normalized): + err("`requires` must be a list of tool names (or { tool, status: succeeded })") else: - requires = list(r) + requires = normalized if len(set(requires)) != len(requires): err("`requires` contains duplicates") if name in requires: @@ -94,8 +159,60 @@ def err(m: str) -> None: amap[k] = v if "requires_approval" in c and not isinstance(c["requires_approval"], bool): err("`compensation.requires_approval` must be a boolean") + if "kind" in c and c["kind"] not in ("inverse", "compensation"): + err("`compensation.kind` must be inverse or compensation") + residue: List[str] = [] + if "residue" in c: + if not isinstance(c["residue"], list) or any(not isinstance(x, str) or not x for x in c["residue"]): + err("`compensation.residue` must be a list of strings") + else: + residue = list(c["residue"]) + if c.get("kind") == "inverse" and residue: + err("an `inverse` compensation restores exact state, so it cannot declare residue; use kind: compensation") + comp_timeout = None + if "timeout_ms" in c: + comp_timeout = parse_duration(c["timeout_ms"]) + if comp_timeout is None: + err("`compensation.timeout_ms` must be a positive number or a duration") + verification = None + if "verification" in c: + v = c["verification"] + if not isinstance(v, dict): + err("`compensation.verification` must be an object") + else: + for k in v: + if k not in _VERIFY: + err(f"unknown verification field `{k}`") + vtool = v.get("tool") + if not isinstance(vtool, str) or not _TOOL.fullmatch(vtool): + err("`verification.tool` must be a tool name") + vargs: Dict[str, str] = {} + if "arguments" in v: + if not isinstance(v["arguments"], dict): + err("`verification.arguments` must be an object") + else: + for k, ref in v["arguments"].items(): + if not isinstance(ref, str) or not _REF.fullmatch(ref): + err(f"verification.arguments.{k} must be a reference like `$result.id`") + else: + vargs[k] = ref + if not isinstance(v.get("expect"), dict) or len(v["expect"]) == 0: + err("`verification.expect` must name at least one path and its expected value") + retries = v.get("retries", 3) + if not _is_int(retries) or retries < 1 or retries > 10: + err("`verification.retries` must be 1..10") + backoff = 1000 if "backoff_ms" not in v else parse_duration(v["backoff_ms"]) + if backoff is None: + err("`verification.backoff_ms` must be a positive number or a duration like 2s") + if isinstance(vtool, str) and isinstance(v.get("expect"), dict) and _is_int(retries) and backoff is not None: + verification = {"tool": vtool, "arguments": vargs, "expect": v["expect"], "retries": retries, "backoff_ms": backoff} if isinstance(ctool, str): - compensation = {"tool": ctool, "argument_map": amap, "requires_approval": c.get("requires_approval") is True} + compensation = {"tool": ctool, "argument_map": amap, "requires_approval": c.get("requires_approval") is True, + "kind": "inverse" if c.get("kind") == "inverse" else "compensation", "residue": residue} + if verification: + compensation["verification"] = verification + if comp_timeout is not None: + compensation["timeout_ms"] = comp_timeout if effect in ("reversible", "compensable"): if "compensation" not in raw: @@ -103,9 +220,109 @@ def err(m: str) -> None: elif effect_ok and "compensation" in raw: err(f"{effect} tools must not declare a compensation") + version = raw.get("version", 1) + if not _is_int(version) or version < 1: + err("`version` must be a positive integer") + status = raw.get("status", "approved") + if status not in ("draft", "approved", "deprecated"): + err("`status` must be draft, approved or deprecated") + for k in ("replaced_by", "owner", "oncall", "schema_hash"): + if k in raw and (not isinstance(raw[k], str) or not raw[k]): + err(f"`{k}` must be a non-empty string") + risk = None + if "risk" in raw: + r = raw["risk"] + if not isinstance(r, dict): + err("`risk` must be an object") + else: + for k in r: + if k not in ("data_class", "blast_radius", "environment"): + err(f"unknown risk field `{k}`") + if "data_class" in r and (not isinstance(r["data_class"], str) or not r["data_class"]): + err("`risk.data_class` must be a non-empty string") + if "blast_radius" in r and r["blast_radius"] not in BLAST_RADII: + err(f"`risk.blast_radius` must be one of {', '.join(BLAST_RADII)}") + if "environment" in r and not isinstance(r["environment"], str): + err("`risk.environment` must be a string") + risk = r + timeout_ms = None + if "timeout_ms" in raw: + timeout_ms = parse_duration(raw["timeout_ms"]) + if timeout_ms is None: + err("`timeout_ms` must be a positive number or a duration like 30s") + retry = None + if "retry" in raw: + r = raw["retry"] + if not isinstance(r, dict): + err("`retry` must be an object") + else: + for k in r: + if k not in ("attempts", "base_ms", "max_ms"): + err(f"unknown retry field `{k}`") + attempts = r.get("attempts", 3) + base = 200 if "base_ms" not in r else parse_duration(r["base_ms"]) + mx = 5000 if "max_ms" not in r else parse_duration(r["max_ms"]) + if not _is_int(attempts) or attempts < 1 or attempts > 10: + err("`retry.attempts` must be 1..10") + if base is None or mx is None: + err("`retry.base_ms` and `retry.max_ms` must be positive durations") + elif _is_int(attempts): + retry = {"attempts": attempts, "base_ms": base, "max_ms": mx} + if effect not in ("read_only", "idempotent"): + err("forward retry is only allowed for read_only and idempotent tools: retrying anything else could repeat its effect") + idempotency = None + if "idempotency" in raw: + i = raw["idempotency"] + if not isinstance(i, dict): + err("`idempotency` must be an object") + else: + for k in i: + if k not in ("key", "scope"): + err(f"unknown idempotency field `{k}`") + if not isinstance(i.get("key"), str) or not _IDEM.search(i["key"]): + err("`idempotency.key` must be a template using at least one `{{args.field}}`") + if "scope" in i and i["scope"] not in ("run", "actor", "global"): + err("`idempotency.scope` must be run, actor or global") + if isinstance(i.get("key"), str): + idempotency = {"key": i["key"], "scope": i.get("scope", "run")} + audit = None + if "audit" in raw: + a = raw["audit"] + if not isinstance(a, dict): + err("`audit` must be an object") + else: + for k in a: + if k not in ("retention_days", "evidence_export", "redaction"): + err(f"unknown audit field `{k}`") + if "retention_days" in a and (not _is_int(a["retention_days"]) or a["retention_days"] < 1): + err("`audit.retention_days` must be a positive integer") + if "evidence_export" in a and a["evidence_export"] not in ("required", "optional"): + err("`audit.evidence_export` must be required or optional") + if "redaction" in a and (not isinstance(a["redaction"], list) or any(not isinstance(p, str) or not _REDACT.fullmatch(p) for p in a["redaction"])): + err("`audit.redaction` must be a list of paths like `$args.client_secret`") + audit = a + if status == "deprecated" and raw.get("replaced_by") == name: + err("a manifest cannot be replaced by itself") + if errors: return None, errors - out: Dict[str, Any] = {"tool": name, "effect": effect, "permission": perm, "approval": approval, "requires": requires} + out: Dict[str, Any] = {"tool": name, "effect": effect, "permission": perm, "approval": approval, "requires": requires, + "version": version, "status": status} + if approval_rules: + out["approval_rules"] = approval_rules + for k in ("replaced_by", "owner", "oncall", "schema_hash"): + if isinstance(raw.get(k), str): + out[k] = raw[k] + if risk: + out["risk"] = risk + if timeout_ms is not None: + out["timeout_ms"] = timeout_ms + if retry: + out["retry"] = retry + if idempotency: + out["idempotency"] = idempotency + if audit: + out["audit"] = audit if isinstance(raw.get("description"), str): out["description"] = raw["description"] if compensation: @@ -142,6 +359,13 @@ def validate(self) -> List[str]: for r in m["requires"]: if r not in self._tools: out.append(f"`{m['tool']}`: required tool `{r}` is not registered") + v = (c or {}).get("verification") + if v and v["tool"] not in self._tools: + out.append(f"`{m['tool']}`: verification tool `{v['tool']}` is not registered") + if v and v["tool"] in self._tools and self._tools[v["tool"]]["effect"] != "read_only": + out.append(f"`{m['tool']}`: verification tool `{v['tool']}` must be read_only: a check must not change anything") + if m.get("replaced_by") and m["replaced_by"] not in self._tools: + out.append(f"`{m['tool']}`: replaced_by `{m['replaced_by']}` is not registered") return out @classmethod diff --git a/packages/sdk-py/reversi_sdk/pydantic_models.py b/packages/sdk-py/reversi_sdk/pydantic_models.py new file mode 100644 index 0000000..721fecc --- /dev/null +++ b/packages/sdk-py/reversi_sdk/pydantic_models.py @@ -0,0 +1,95 @@ +"""Optional pydantic v2 bindings for the manifest. Import this module only if pydantic is installed. + +The stdlib validator in `manifest.py` stays the single authority on what a valid manifest is: every model runs it first +and reports its errors, so the typed models and the validator cannot disagree. The models add IDE types and +`model_dump()` / JSON Schema generation on top. +""" +from __future__ import annotations + +from typing import Any, Dict, List, Optional + +try: + from pydantic import BaseModel, ConfigDict, model_validator +except ImportError as e: # pragma: no cover + raise ImportError("reversi_sdk.pydantic_models needs pydantic v2: pip install 'reversi-sdk[pydantic]'") from e + +from .manifest import validate_manifest + + +def _plain(v: Any) -> Any: + """Models nested in the input become plain dicts, so the stdlib validator sees what the caller meant.""" + if isinstance(v, BaseModel): + return _plain(v.model_dump(exclude_unset=True)) + if isinstance(v, dict): + return {k: _plain(x) for k, x in v.items()} + if isinstance(v, list): + return [_plain(x) for x in v] + return v + + +class Verification(BaseModel): + """Read back state after a compensation and require it to match `expect` before the undo counts as done.""" + model_config = ConfigDict(extra="forbid") + + tool: str + arguments: Dict[str, str] = {} + expect: Dict[str, Any] + retries: int = 3 + backoff_ms: int = 1000 + + +class Compensation(BaseModel): + model_config = ConfigDict(extra="forbid") + + tool: str + argument_map: Dict[str, str] = {} + requires_approval: bool = False + kind: str = "compensation" + residue: List[str] = [] + verification: Optional[Verification] = None + timeout_ms: Optional[int] = None + + +class ToolManifest(BaseModel): + model_config = ConfigDict(extra="forbid") + + tool: str + effect: str + permission: str + description: Optional[str] = None + approval: str = "never" + approval_rules: Optional[Dict[str, Any]] = None + requires: List[str] = [] + compensation: Optional[Compensation] = None + version: int = 1 + status: str = "approved" + replaced_by: Optional[str] = None + owner: Optional[str] = None + oncall: Optional[str] = None + risk: Optional[Dict[str, Any]] = None + timeout_ms: Optional[int] = None + retry: Optional[Dict[str, int]] = None + idempotency: Optional[Dict[str, str]] = None + audit: Optional[Dict[str, Any]] = None + schema_hash: Optional[str] = None + + @model_validator(mode="before") + @classmethod + def _authoritative(cls, data: Any) -> Any: + raw = _plain(data) + normalized, errors = validate_manifest(raw) + if normalized is None: + raise ValueError("; ".join(errors)) + return normalized + + def normalized(self) -> Dict[str, Any]: + """The same dict `validate_manifest` returns.""" + return self.model_dump(exclude_none=True) + + +def validate_with_pydantic(raw: Any) -> Optional[Dict[str, Any]]: + """Normalized manifest, or None if invalid (use ToolManifest.model_validate for the error details).""" + try: + return ToolManifest.model_validate(raw).normalized() + except ValueError: # pydantic.ValidationError subclasses ValueError + return None diff --git a/packages/sdk-py/tests/_fake_reversi.py b/packages/sdk-py/tests/_fake_reversi.py new file mode 100644 index 0000000..7ba0f42 --- /dev/null +++ b/packages/sdk-py/tests/_fake_reversi.py @@ -0,0 +1,85 @@ +"""A tiny in-process fake of the Reversi agent API, enough for the adapters and the Claude Code hook.""" +import json +import threading +from http.server import BaseHTTPRequestHandler, HTTPServer + +APPROVAL = "apr_0a1b2c-3d" + + +class FakeReversi: + """Decisions by tool name: `blocked*` -> block, `approve*` -> pending approval, `would*` -> would_block, else allow.""" + + def __init__(self): + self.runs = {} # run_id -> {"permissions": [...], "steps": [...]} + self.requests = [] # (method, path, body) + self.halted = False + self.down = False + outer = self + + class H(BaseHTTPRequestHandler): + def log_message(self, *a): + pass + + def _send(self, code, obj): + data = json.dumps(obj).encode() + self.send_response(code) + self.send_header("Content-Type", "application/json") + self.send_header("Content-Length", str(len(data))) + self.end_headers() + self.wfile.write(data) + + def _handle(self, method): + n = int(self.headers.get("Content-Length") or 0) + body = json.loads(self.rfile.read(n)) if n else None + outer.requests.append((method, self.path, body)) + if self.headers.get("Authorization") != "Bearer agent-key": + return self._send(401, {"error": "bad key"}) + parts = self.path.strip("/").split("/") + if method == "GET" and self.path == "/v1/status": + return self._send(200, {"halted": outer.halted}) + if method == "POST" and self.path == "/v1/runs": + rid = body.get("run_id") or "gen-1" + if rid in outer.runs: + return self._send(409, {"error": f"run `{rid}` already exists"}) + outer.runs[rid] = {"permissions": body["permissions"], "steps": []} + return self._send(201, {"run_id": rid, "mode": "enforce"}) + if len(parts) >= 3 and parts[:2] == ["v1", "runs"]: + run = outer.runs.get(parts[2]) + if run is None: + return self._send(404, {"error": "run not found"}) + if method == "GET" and len(parts) == 3: + return self._send(200, {"run_id": parts[2], "steps": run["steps"], "unresolved": []}) + if method == "POST" and parts[3:] == ["intents"]: + tool = body["tool"] + if tool.startswith("blocked"): + return self._send(200, {"decision": "block", "reasons": ["tool is not in scope"]}) + if tool.startswith("approve"): + return self._send(200, {"decision": "block", "reasons": [ + f"approval required: request {APPROVAL} is pending an operator decision; retry after it is approved"]}) + step = len(run["steps"]) + run["steps"].append({"step": step, "tool": tool, "args": body.get("args"), "outcome": None}) + if tool.startswith("would"): + return self._send(200, {"decision": "would_block", "step": step, "reasons": ["dry run"]}) + return self._send(200, {"decision": "allow", "step": step}) + if method == "POST" and len(parts) == 6 and parts[3] == "steps" and parts[5] == "outcome": + run["steps"][int(parts[4])]["outcome"] = body + return self._send(200, {"ok": True}) + self._send(404, {"error": "not found"}) + + def do_GET(self): + self._handle("GET") + + def do_POST(self): + self._handle("POST") + + self.server = HTTPServer(("127.0.0.1", 0), H) + self.url = f"http://127.0.0.1:{self.server.server_address[1]}" + self.thread = threading.Thread(target=self.server.serve_forever, daemon=True) + + def start(self): + self.thread.start() + return self + + def stop(self): + self.server.shutdown() + self.server.server_close() diff --git a/packages/sdk-py/tests/test_claude_hook.py b/packages/sdk-py/tests/test_claude_hook.py new file mode 100644 index 0000000..69d7f8a --- /dev/null +++ b/packages/sdk-py/tests/test_claude_hook.py @@ -0,0 +1,218 @@ +"""The Claude Code hook, end to end: against an in-process fake Reversi, and against the real built server.""" +import hashlib +import io +import json +import os +import re +import shutil +import subprocess +import tempfile +import unittest + +from reversi_sdk import ReversiClient +from reversi_sdk.integrations import claude_code_hook as hook + +from ._fake_reversi import APPROVAL, FakeReversi + +ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "..", "..")) +MAIN = os.path.join(ROOT, "packages", "server", "dist", "main.js") +CLI = os.path.join(ROOT, "packages", "cli", "dist", "main.js") + + +def run_hook(payload, env, post=False): + out = io.StringIO() + code = hook.main(["--post"] if post else [], io.StringIO(json.dumps(payload)), out, env) + assert code == 0 + return json.loads(out.getvalue())["hookSpecificOutput"] if out.getvalue() else None + + +def pre(session="s1", tool="Read", inp=None, tid="tu1"): + return {"hook_event_name": "PreToolUse", "session_id": session, "tool_name": tool, + "tool_input": inp or {"file_path": "/a"}, "tool_use_id": tid} + + +class ToolNameMapping(unittest.TestCase): + def test_mapping(self): + self.assertEqual(hook.map_tool_name("Bash"), "Bash") + self.assertEqual(hook.map_tool_name("mcp__github__create_issue"), "github.create_issue") + self.assertEqual(hook.map_tool_name("mcp__my_server__do__it"), "my_server.do__it") + + +class HookAgainstFake(unittest.TestCase): + @classmethod + def setUpClass(cls): + cls.fake = FakeReversi().start() + + @classmethod + def tearDownClass(cls): + cls.fake.stop() + + def setUp(self): + self.state = tempfile.mkdtemp(prefix="hookstate-") + self.addCleanup(shutil.rmtree, self.state, True) + self.env = {"REVERSI_URL": self.fake.url, "REVERSI_AGENT_KEY": "agent-key", "REVERSI_PERMISSIONS": "a, b", + "REVERSI_HOOK_STATE": self.state} + self.session = "sess-" + self.id().split(".")[-1] + + def steps(self): + return self.fake.runs[f"claude-{self.session}"]["steps"] + + def test_allow_creates_one_run_per_session_and_maps_mcp_names(self): + d = run_hook(pre(self.session, "mcp__srv__lookup"), self.env) + self.assertEqual(d["hookEventName"], "PreToolUse") + self.assertEqual(d["permissionDecision"], "allow") + run_hook(pre(self.session, "Read", tid="tu2"), self.env) + self.assertEqual([s["tool"] for s in self.steps()], ["srv.lookup", "Read"]) + self.assertEqual(self.fake.runs[f"claude-{self.session}"]["permissions"], ["a", "b"]) + creates = [r for r in self.fake.requests if r[:2] == ("POST", "/v1/runs") and r[2]["run_id"] == f"claude-{self.session}"] + self.assertEqual(len(creates), 1) + + def test_would_block_is_allowed(self): + d = run_hook(pre(self.session, "would_x"), self.env) + self.assertEqual(d["permissionDecision"], "allow") + self.assertIn("dry run", d["permissionDecisionReason"]) + + def test_pending_approval_asks_and_names_the_request(self): + d = run_hook(pre(self.session, "approve_deploy"), self.env) + self.assertEqual(d["permissionDecision"], "ask") + self.assertIn(APPROVAL, d["permissionDecisionReason"]) + + def test_other_block_denies(self): + d = run_hook(pre(self.session, "blocked_x"), self.env) + self.assertEqual(d["permissionDecision"], "deny") + self.assertIn("tool is not in scope", d["permissionDecisionReason"]) + + def test_post_reports_success_for_the_recorded_step(self): + run_hook(pre(self.session, "Read", tid="a"), self.env) + run_hook(pre(self.session, "Read", tid="b"), self.env) + self.assertIsNone(run_hook({"hook_event_name": "PostToolUse", "session_id": self.session, "tool_name": "Read", + "tool_use_id": "b", "tool_response": {"content": "x"}}, self.env, post=True)) + self.assertIsNone(self.steps()[0]["outcome"]) + self.assertEqual(self.steps()[1]["outcome"], {"status": "succeeded", "result": {"content": "x"}}) + # reporting twice does nothing: the state entry is consumed + before = len(self.fake.requests) + run_hook({"session_id": self.session, "tool_name": "Read", "tool_use_id": "b"}, self.env, post=True) + self.assertEqual(len(self.fake.requests), before) + + def test_post_failure_and_error_response_report_failed(self): + run_hook(pre(self.session, "Bash", tid="f1"), self.env) + run_hook(pre(self.session, "Bash", tid="f2"), self.env) + run_hook({"hook_event_name": "PostToolUseFailure", "session_id": self.session, "tool_name": "Bash", + "tool_use_id": "f1", "error": "exit 1"}, self.env, post=True) + run_hook({"session_id": self.session, "tool_name": "Bash", "tool_use_id": "f2", + "tool_response": {"is_error": True, "error": "nope"}}, self.env, post=True) + self.assertEqual([s["outcome"]["status"] for s in self.steps()], ["failed", "failed"]) + self.assertEqual(self.steps()[0]["outcome"]["error"], "exit 1") + + def test_post_without_pre_record_is_a_noop(self): + before = len(self.fake.requests) + run_hook({"session_id": self.session, "tool_name": "Read", "tool_use_id": "zzz"}, self.env, post=True) + self.assertEqual(len(self.fake.requests), before) + + def test_post_works_without_tool_use_id(self): + run_hook({**pre(self.session, "Read"), "tool_use_id": None}, self.env) + run_hook({"session_id": self.session, "tool_name": "Read", "tool_input": {"file_path": "/a"}}, self.env, post=True) + self.assertEqual(self.steps()[0]["outcome"]["status"], "succeeded") + + def test_fail_closed_when_unreachable_and_fail_open_when_configured(self): + env = dict(self.env, REVERSI_URL="http://127.0.0.1:9", REVERSI_TIMEOUT="1") + d = run_hook(pre(self.session), env) + self.assertEqual(d["permissionDecision"], "deny") + self.assertIn("failing closed", d["permissionDecisionReason"]) + d = run_hook(pre(self.session), dict(env, REVERSI_HOOK_FAIL_OPEN="1")) + self.assertEqual(d["permissionDecision"], "allow") + self.assertIn("FAIL_OPEN", d["permissionDecisionReason"]) + + def test_missing_config_bad_key_and_bad_input_deny(self): + self.assertEqual(run_hook(pre(self.session), {})["permissionDecision"], "deny") + d = run_hook(pre(self.session), dict(self.env, REVERSI_AGENT_KEY="wrong")) + self.assertEqual(d["permissionDecision"], "deny") + out = io.StringIO() + hook.main([], io.StringIO("not json"), out, self.env) + self.assertEqual(json.loads(out.getvalue())["hookSpecificOutput"]["permissionDecision"], "deny") + d = run_hook({"tool_name": "Read"}, self.env) + self.assertEqual(d["permissionDecision"], "deny") + + def test_post_never_raises_when_server_is_down(self): + run_hook(pre(self.session, "Read", tid="d1"), self.env) + env = dict(self.env, REVERSI_URL="http://127.0.0.1:9", REVERSI_TIMEOUT="1") + self.assertIsNone(run_hook({"session_id": self.session, "tool_name": "Read", "tool_use_id": "d1"}, env, post=True)) + + def test_console_script_is_declared(self): + with open(os.path.join(ROOT, "packages", "sdk-py", "pyproject.toml"), encoding="utf-8") as f: + text = f.read() + self.assertIn('reversi-claude-hook = "reversi_sdk.integrations.claude_code_hook:main"', text) + + def test_runs_as_a_subprocess_like_claude_code_does(self): + env = dict(os.environ, **self.env) + p = subprocess.run(["python3", "-m", "reversi_sdk.integrations.claude_code_hook"], + input=json.dumps(pre(self.session, "Read")), capture_output=True, text=True, env=env, + cwd=os.path.join(ROOT, "packages", "sdk-py")) + self.assertEqual(p.returncode, 0, p.stderr) + self.assertEqual(json.loads(p.stdout)["hookSpecificOutput"]["permissionDecision"], "allow") + + +MANIFESTS = [ + {"tool": "Read", "effect": "read_only", "permission": "claude:read"}, + {"tool": "Bash", "effect": "irreversible", "permission": "claude:shell"}, + {"tool": "Write", "effect": "idempotent", "permission": "claude:write"}, +] + + +@unittest.skipUnless(shutil.which("node") and os.path.exists(MAIN), "needs node and `npm run build`") +class HookAgainstRealServer(unittest.TestCase): + @classmethod + def setUpClass(cls): + cls.tmp = tempfile.mkdtemp(prefix="reversi-hook-") + with open(os.path.join(cls.tmp, "tools.json"), "w") as f: + json.dump(MANIFESTS, f) + with open(os.path.join(cls.tmp, "keys.json"), "w") as f: + json.dump([{"name": "claude", "role": "agent", "key": "agent-key"}, + {"name": "appr", "role": "approver", "key": "appr-key"}, + {"name": "ops", "role": "operator", "key_sha256": hashlib.sha256(b"ops-key").hexdigest()}], f) + subprocess.run(["node", CLI, "keygen", os.path.join(cls.tmp, "sign")], check=True, capture_output=True) + env = dict(os.environ, REVERSI_SIGNING_KEY=os.path.join(cls.tmp, "sign.key"), + REVERSI_MANIFESTS=os.path.join(cls.tmp, "tools.json"), REVERSI_KEYS=os.path.join(cls.tmp, "keys.json"), + REVERSI_DATA=os.path.join(cls.tmp, "data"), PORT="0", HOST="127.0.0.1") + cls.proc = subprocess.Popen(["node", MAIN], env=env, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True) + line = cls.proc.stdout.readline() + m = re.search(r"http://127\.0\.0\.1:(\d+)", line) + if not m: + cls.proc.kill() + raise RuntimeError(f"server did not start: {line!r}") + cls.url = f"http://127.0.0.1:{m.group(1)}" + + @classmethod + def tearDownClass(cls): + cls.proc.terminate() + cls.proc.wait(timeout=5) + cls.proc.stdout.close() + shutil.rmtree(cls.tmp, ignore_errors=True) + + def test_allow_ask_approve_allow_deny_and_outcome(self): + state = os.path.join(self.tmp, "hookstate") + env = {"REVERSI_URL": self.url, "REVERSI_AGENT_KEY": "agent-key", "REVERSI_PERMISSIONS": "claude:read,claude:shell", + "REVERSI_HOOK_STATE": state} + d = run_hook(pre("live", "Read", tid="t1"), env) + self.assertEqual(d["permissionDecision"], "allow", d) + run_hook({"session_id": "live", "tool_name": "Read", "tool_use_id": "t1", "tool_response": {"ok": True}}, env, post=True) + + d = run_hook(pre("live", "Bash", {"command": "ls"}, tid="t2"), env) + self.assertEqual(d["permissionDecision"], "ask", d) + aid = re.search(r"apr_[0-9a-f-]+", d["permissionDecisionReason"]).group(0) + appr = ReversiClient(self.url, "appr-key") + self.assertEqual([a["id"] for a in appr.approvals(status="pending")], [aid]) + appr.approve(aid, "ok") + d = run_hook(pre("live", "Bash", {"command": "ls"}, tid="t3"), env) + self.assertEqual(d["permissionDecision"], "allow", d) + + d = run_hook(pre("live", "Write", tid="t4"), env) # claude:write was not granted to the run + self.assertEqual(d["permissionDecision"], "deny", d) + + run_hook({"session_id": "live", "tool_name": "Bash", "tool_use_id": "t3", "tool_response": {"stdout": ""}}, env, post=True) + st = ReversiClient(self.url, "ops-key").run("claude-live").state() + self.assertEqual(st["unresolved"], []) + + +if __name__ == "__main__": + unittest.main() diff --git a/packages/sdk-py/tests/test_integrations.py b/packages/sdk-py/tests/test_integrations.py new file mode 100644 index 0000000..fd5ca67 --- /dev/null +++ b/packages/sdk-py/tests/test_integrations.py @@ -0,0 +1,350 @@ +"""Framework adapters, tested against small fakes of the framework interfaces (the frameworks are not installed).""" +import dataclasses +import sys +import types +import unittest + +from reversi_sdk import Blocked, EffectUncertain, ReversiClient +from reversi_sdk.integrations import crewai as rcrew +from reversi_sdk.integrations import fastmcp as rmcp +from reversi_sdk.integrations import langgraph as rlg +from reversi_sdk.integrations import openai_agents as roa + +from ._fake_reversi import APPROVAL, FakeReversi + + +class Base(unittest.TestCase): + @classmethod + def setUpClass(cls): + cls.fake = FakeReversi().start() + + @classmethod + def tearDownClass(cls): + cls.fake.stop() + + def setUp(self): + self.client = ReversiClient(self.fake.url, "agent-key") + self.run = self.client.start_run(["p"], run_id=f"r-{self.id().split('.')[-1]}") + self.steps = lambda: self.fake.runs[self.run.run_id]["steps"] + + +class FakeBaseTool: + """Mimics langchain_core.tools.BaseTool: name, description, invoke(input).""" + name = "lookup" + description = "look things up" + + def __init__(self): + self.calls = [] + + def invoke(self, input, config=None): + self.calls.append(input) + return {"found": input} + + +class LangGraphTests(Base): + def test_function_is_journaled_and_keeps_signature(self): + @rlg.reversible_tool(self.run, "lookup", reason="test") + def lookup(sku: str, qty: int = 1): + """doc""" + return {"sku": sku} + + self.assertEqual(lookup("A1", qty=2), {"sku": "A1"}) + self.assertEqual(lookup.__name__, "lookup") + self.assertEqual(lookup.__doc__, "doc") + self.assertEqual(self.steps()[0]["args"], {"args": ["A1"], "kwargs": {"qty": 2}}) + self.assertEqual(self.steps()[0]["outcome"]["status"], "succeeded") + + def test_basetool_proxy_journals_invoke_and_passes_attributes(self): + inner = FakeBaseTool() + t = rlg.reversible_tool(self.run, "lookup")(inner) + self.assertEqual(t.name, "lookup") + self.assertEqual(t.description, "look things up") + self.assertEqual(t.invoke({"q": 1}), {"found": {"q": 1}}) + self.assertEqual(self.steps()[0]["args"], {"q": 1}) + self.assertEqual(inner.calls, [{"q": 1}]) + + def test_blocked_returns_message_without_running(self): + inner = FakeBaseTool() + out = rlg.reversible_tool(self.run, "blocked_tool", tool=inner).invoke({"q": 1}) + self.assertIn("tool is not in scope", out) + self.assertIn("did not run", out) + self.assertEqual(inner.calls, []) + self.assertEqual(self.steps(), []) + + def test_pending_approval_id_is_in_the_message(self): + out = rlg.reversible_tool(self.run, "approve_me")(lambda **kw: 1)(x=1) + self.assertIn(APPROVAL, out) + + def test_on_block_raise(self): + f = rlg.reversible_tool(self.run, "approve_me", on_block="raise")(lambda **kw: 1) + with self.assertRaises(Blocked) as cm: + f(x=1) + self.assertEqual(cm.exception.approval_id, APPROVAL) + with self.assertRaises(ValueError): + rlg.reversible_tool(self.run, "x", on_block="nope") + + def test_failure_and_uncertainty_are_journaled_and_reraised(self): + def boom(**kw): + raise RuntimeError("db down") + + def unsure(**kw): + raise EffectUncertain("timeout") + + with self.assertRaises(RuntimeError): + rlg.reversible_tool(self.run, "t1")(boom)() + with self.assertRaises(EffectUncertain): + rlg.reversible_tool(self.run, "t2")(unsure)() + self.assertEqual([s["outcome"]["status"] for s in self.steps()], ["failed", "uncertain"]) + + def test_non_json_results_do_not_break_journaling(self): + obj = object() + self.assertIs(rlg.reversible_tool(self.run, "t")(lambda: obj)(), obj) + + def test_callable_without_invoke(self): + class C: + def __call__(self, x): + return x + 1 + self.assertEqual(rlg.reversible_tool(self.run, "t", tool=C())(1), 2) + + +class LangGraphAsyncTests(unittest.IsolatedAsyncioTestCase): + async def test_async_function_and_block(self): + fake = FakeReversi().start() + self.addCleanup(fake.stop) + run = ReversiClient(fake.url, "agent-key").start_run(["p"], run_id="a1") + + @rlg.reversible_tool(run, "alookup") + async def alookup(x): + return x * 2 + + self.assertEqual(await alookup(3), 6) + self.assertEqual(fake.runs["a1"]["steps"][0]["outcome"]["status"], "succeeded") + blocked = rlg.reversible_tool(run, "blocked_x")(alookup) + self.assertIn("blocked", await blocked(3)) + + +class RollbackNodeTests(unittest.TestCase): + def test_node_returns_report_in_state(self): + calls = [] + + class FakeRunH: + def rollback(self, compensate, **kw): + calls.append(kw) + return {"complete": True, "compensated": [1]} + + class FakeClient: + def run(self, rid): + calls.append(rid) + return FakeRunH() + + node = rlg.rollback_node(FakeClient(), "run-9", lambda c: None, attempts=2) + self.assertEqual(node({"messages": []}), {"reversi_rollback": {"complete": True, "compensated": [1]}}) + self.assertEqual(calls, ["run-9", {"attempts": 2}]) + + +class CrewTests(Base): + class Inner: + name = "Order lookup" + description = "finds orders" + args_schema = None + + def _run(self, order_id): + return f"order {order_id}" + + def test_journals_and_exposes_crewai_shape(self): + t = rcrew.ReversibleCrewTool(self.run, "lookup_order", inner=self.Inner()) + self.assertEqual((t.name, t.description), ("Order lookup", "finds orders")) + self.assertEqual(t._run(order_id="7"), "order 7") + self.assertEqual(t.run("8"), "order 8") + self.assertEqual(self.steps()[0]["tool"], "lookup_order") + self.assertEqual(self.steps()[0]["args"], {"order_id": "7"}) + + def test_block_message_and_raise(self): + t = rcrew.ReversibleCrewTool(self.run, "approve_x", inner=self.Inner()) + self.assertIn(APPROVAL, t._run(order_id="1")) + t = rcrew.ReversibleCrewTool(self.run, "approve_x", inner=self.Inner(), on_block="raise") + with self.assertRaises(Blocked): + t._run(order_id="1") + + def test_to_crewai_builds_subclass_of_real_basetool(self): + class BaseTool: # fake crewai.tools.BaseTool + def run(self, *a, **kw): + return self._run(*a, **kw) + + mod, tools = types.ModuleType("crewai"), types.ModuleType("crewai.tools") + tools.BaseTool = BaseTool + mod.tools = tools + saved = {k: sys.modules.get(k) for k in ("crewai", "crewai.tools")} + sys.modules.update({"crewai": mod, "crewai.tools": tools}) + try: + real = rcrew.ReversibleCrewTool(self.run, "lookup_order", inner=self.Inner()).to_crewai() + finally: + for k, v in saved.items(): + if v is None: + sys.modules.pop(k, None) + else: + sys.modules[k] = v + self.assertIsInstance(real, BaseTool) + self.assertEqual(real.name, "Order lookup") + self.assertEqual(real.run(order_id="3"), "order 3") + self.assertEqual(len(self.steps()), 1) + + +class OpenAIAgentsTests(Base): + def test_function_wrapper_drops_context_and_journals_json_args(self): + class RunContextWrapper: + pass + + @roa.reversi_function_tool(self.run, "create_order") + def create_order(ctx, sku: str): + return f"made {sku}" + + self.assertEqual(create_order(RunContextWrapper(), sku="A1"), "made A1") + self.assertEqual(self.steps()[0]["args"], {"sku": "A1"}) + + def test_block_modes_and_error_formatter(self): + f = roa.reversi_function_tool(self.run, "approve_x")(lambda a: 1) + self.assertIn(APPROVAL, f({"a": 1})) + g = roa.reversi_function_tool(self.run, "approve_x", on_block="raise")(lambda a: 1) + with self.assertRaises(Blocked) as cm: + g({"a": 1}) + text = roa.tool_error_formatter(None, cm.exception) + self.assertIn(APPROVAL, text) + self.assertIn("boom", roa.tool_error_formatter(None, RuntimeError("boom"))) + + def test_function_tool_object_is_wrapped(self): + @dataclasses.dataclass + class FunctionTool: # mimics agents.FunctionTool + name: str + on_invoke_tool: object + + async def invoke(ctx, input_json): + return "ok" + + import asyncio + tool = roa.reversi_function_tool(self.run, "create_order")(FunctionTool("create_order", invoke)) + self.assertEqual(asyncio.run(tool.on_invoke_tool(None, '{"sku": "A1"}')), "ok") + self.assertEqual(self.steps()[0]["args"], {"sku": "A1"}) + blocked = roa.reversi_function_tool(self.run, "blocked_x")(FunctionTool("x", invoke)) + self.assertIn("blocked", asyncio.run(blocked.on_invoke_tool(None, "{}"))) + + +class GuardrailTests(unittest.IsolatedAsyncioTestCase): + async def test_trips_when_halted_and_fails_closed(self): + fake = FakeReversi().start() + self.addCleanup(fake.stop) + client = ReversiClient(fake.url, "agent-key", timeout=2) + g = roa.guardrail(client) + self.assertFalse((await g(None, None, "hi")).tripwire_triggered) + fake.halted = True + r = await g(None, None, "hi") + self.assertTrue(r.tripwire_triggered) + self.assertTrue(r.output_info["halted"]) + dead = roa.guardrail(ReversiClient("http://127.0.0.1:9", "k", timeout=1)) + r = await dead(None, None, "hi") + self.assertTrue(r.tripwire_triggered) + self.assertIn("could not reach", r.output_info["error"]) + + async def test_uses_agents_result_class_when_installed(self): + class Out: + def __init__(self, output_info, tripwire_triggered): + self.output_info, self.tripwire_triggered = output_info, tripwire_triggered + + mod = types.ModuleType("agents") + mod.GuardrailFunctionOutput = Out + fake = FakeReversi().start() + self.addCleanup(fake.stop) + sys.modules["agents"] = mod + try: + r = await roa.guardrail(ReversiClient(fake.url, "agent-key"))(None, None, "x") + finally: + del sys.modules["agents"] + self.assertIsInstance(r, Out) + + +class FastMCPTests(Base): + class FakeMCP: # mimics FastMCP: tool(**kw) returns a decorator + def __init__(self): + self.tools = {} + + def tool(self, **kw): + def deco(fn): + self.tools[kw.get("name", fn.__name__)] = fn + return fn + return deco + + def test_registers_and_journals_with_ctx_passed_to_factory(self): + mcp, seen = self.FakeMCP(), [] + + def factory(ctx): + seen.append(ctx) + return self.run + + class Context: + pass + + @rmcp.reversible(mcp, factory) + def create_order(sku: str, ctx: Context = None): + return {"sku": sku} + + ctx = Context() + self.assertEqual(mcp.tools["create_order"]("A1", ctx=ctx), {"sku": "A1"}) + self.assertEqual(mcp.tools["create_order"]("A2"), {"sku": "A2"}) + self.assertIs(seen[0], ctx) + self.assertIsNone(seen[1]) + self.assertEqual(self.steps()[0]["args"], {"args": ["A1"], "kwargs": {}}) + self.assertEqual(self.steps()[0]["tool"], "create_order") + + def test_blocked_raises_reversi_blocked_with_approval(self): + mcp = self.FakeMCP() + rmcp.reversible(mcp, lambda ctx: self.run, tool_name="approve_refund", name="refund")(lambda amount: amount) + with self.assertRaises(rmcp.ReversiBlocked) as cm: + mcp.tools["refund"](amount=5) + self.assertEqual(cm.exception.approval_id, APPROVAL) + self.assertIn(APPROVAL, str(cm.exception)) + self.assertEqual(self.steps(), []) + + def test_per_session_runs_reuses_one_run_per_session(self): + factory = rmcp.per_session_runs(self.client, ["p"], prefix="sess") + + def ctx(session): + return types.SimpleNamespace(session=session) + + s1, s2 = object(), object() + a, b, c = factory(ctx(s1)), factory(ctx(s1)), factory(ctx(s2)) + self.assertIs(a, b) + self.assertNotEqual(a.run_id, c.run_id) + self.assertTrue(a.run_id.startswith("sess-")) + + +class FastMCPAsyncTests(unittest.IsolatedAsyncioTestCase): + async def test_async_tool_and_block(self): + fake = FakeReversi().start() + self.addCleanup(fake.stop) + run = ReversiClient(fake.url, "agent-key").start_run(["p"], run_id="m1") + + @rmcp.reversible(None, lambda ctx: run) + async def read_doc(path: str): + return path.upper() + + self.assertEqual(await read_doc("a"), "A") + blocked = rmcp.reversible(None, lambda ctx: run, tool_name="blocked_doc")(read_doc) + with self.assertRaises(rmcp.ReversiBlocked): + await blocked("a") + + +class ImportIsolation(unittest.TestCase): + def test_core_import_needs_no_framework(self): + import subprocess + code = ("import sys, reversi_sdk, reversi_sdk.integrations.langgraph, reversi_sdk.integrations.crewai, " + "reversi_sdk.integrations.openai_agents, reversi_sdk.integrations.fastmcp, " + "reversi_sdk.integrations.claude_code_hook;" + "bad=[m for m in ('langchain','langgraph','crewai','agents','mcp','pydantic') if m in sys.modules];" + "sys.exit(1 if bad else 0)") + root = __import__("os").path.abspath(__import__("os").path.join(__import__("os").path.dirname(__file__), "..")) + r = subprocess.run([sys.executable, "-c", code], cwd=root, capture_output=True, text=True) + self.assertEqual(r.returncode, 0, r.stderr) + + +if __name__ == "__main__": + unittest.main() diff --git a/packages/sdk-py/tests/test_pydantic_and_webhook.py b/packages/sdk-py/tests/test_pydantic_and_webhook.py new file mode 100644 index 0000000..986cd63 --- /dev/null +++ b/packages/sdk-py/tests/test_pydantic_and_webhook.py @@ -0,0 +1,79 @@ +"""Webhook signature helper, and the optional pydantic bindings (skipped when pydantic is not installed).""" +import hashlib +import hmac +import json +import os +import unittest + +from reversi_sdk import ReversiClient, validate_manifest, verify_webhook_signature + +CONF = os.path.join(os.path.dirname(__file__), "..", "..", "..", "conformance", "manifest-cases.json") + + +def sign(secret: bytes, body: bytes) -> str: + return "sha256=" + hmac.new(secret, body, hashlib.sha256).hexdigest() + + +class WebhookSignature(unittest.TestCase): + def test_valid_and_invalid(self): + body = b'{"event":"approval.requested","id":"apr_1"}' + good = sign(b"s3cret", body) + self.assertTrue(verify_webhook_signature("s3cret", body, good)) + self.assertTrue(ReversiClient.verify_webhook_signature(b"s3cret", body.decode(), good)) + self.assertTrue(verify_webhook_signature("s3cret", body, good.upper().replace("SHA256=", "sha256="))) + self.assertFalse(verify_webhook_signature("other", body, good)) + self.assertFalse(verify_webhook_signature("s3cret", body + b" ", good)) + self.assertFalse(verify_webhook_signature("s3cret", body, good[len("sha256="):])) # missing prefix + self.assertFalse(verify_webhook_signature("s3cret", body, "sha1=" + good[7:])) + self.assertFalse(verify_webhook_signature("s3cret", body, good[:-2])) # truncated + self.assertFalse(verify_webhook_signature("s3cret", body, None)) + self.assertFalse(verify_webhook_signature("s3cret", body, "")) + self.assertFalse(verify_webhook_signature("s3cret", body, "sha256=éé")) # non-ascii must not raise + + def test_uses_constant_time_compare(self): + import reversi_sdk.client as c + calls = [] + real = c.hmac.compare_digest + c.hmac.compare_digest = lambda a, b: calls.append(1) or real(a, b) + try: + verify_webhook_signature("k", b"x", sign(b"k", b"x")) + finally: + c.hmac.compare_digest = real + self.assertEqual(calls, [1]) + + +try: + import pydantic # noqa: F401 + HAVE_PYDANTIC = True +except ImportError: + HAVE_PYDANTIC = False + + +@unittest.skipUnless(HAVE_PYDANTIC, "pydantic v2 is not installed") +class PydanticAgreesWithStdlib(unittest.TestCase): + def test_every_conformance_case(self): + from reversi_sdk.pydantic_models import validate_with_pydantic + with open(CONF, encoding="utf-8") as f: + cases = json.load(f) + for i, case in enumerate(cases): + with self.subTest(i=i, input=case["input"]): + std, _ = validate_manifest(case["input"]) + pyd = validate_with_pydantic(case["input"]) + self.assertEqual(std is not None, case["valid"]) + self.assertEqual(pyd is not None, case["valid"]) + self.assertEqual(pyd, std) + + def test_models_are_typed_and_report_errors(self): + from pydantic import ValidationError + from reversi_sdk.pydantic_models import Compensation, ToolManifest + m = ToolManifest(tool="charge", effect="compensable", permission="pay", + compensation=Compensation(tool="refund", argument_map={"id": "$result.id"})) + self.assertEqual(m.normalized()["compensation"]["tool"], "refund") + with self.assertRaises(ValidationError): + ToolManifest(tool="x", effect="irreversible", permission="p", approval="never") + with self.assertRaises(ValidationError): + ToolManifest(tool="x", effect="read_only", permission="p", surprise=1) + + +if __name__ == "__main__": + unittest.main() diff --git a/packages/server/src/app.ts b/packages/server/src/app.ts index 35b259a..ce7eecd 100644 --- a/packages/server/src/app.ts +++ b/packages/server/src/app.ts @@ -9,6 +9,7 @@ import { type ApiKey, type Capability, type Principal, + type Role, ApprovalError, LedgerError, RUN_ID_PATTERN, @@ -20,11 +21,13 @@ import { renderReport, toJsonl, type Approvals, + type Delegations, type Mode, type Outcome, type SigningKey, } from "@multimindlab/reversi"; -import { mountConsole } from "./console.js"; +import { mountConsole, type ConsoleMode } from "./console.js"; +import { ApiError, mountExtras, type ExtraOptions } from "./extras.js"; import { Metrics } from "./metrics.js"; export interface LogEntry { @@ -36,10 +39,14 @@ export interface LogEntry { principal: string | null; } -export interface AppOptions { +export type { ConnectorInfo, RollbackState } from "./extras.js"; + +export interface AppOptions extends ExtraOptions { /** The engine's approver should be `approvals.asApprover()`. */ reversi: Reversi; approvals: Approvals; + /** Enables delegation endpoints. */ + delegations?: Delegations; keys: ApiKey[] | KeyRing; oidc?: OidcVerifier; /** Enables signed evidence bundles. */ @@ -52,6 +59,12 @@ export interface AppOptions { /** Signs console session cookies. Random per process by default, so sessions end on restart. */ session_secret?: string; secure_cookies?: boolean; + /** + * The operator console. `off` (default) serves nothing. `open` has no login, answers only requests addressed to the + * loopback interface from the same origin, and acts with a fixed role (default operator, never admin). `login` asks + * for an API key or OIDC token. + */ + console?: { mode: ConsoleMode; role?: Role }; now?: () => number; } @@ -202,9 +215,9 @@ export function createApp(opts: AppOptions): Hono { if (extra.length) throw new HttpError(403, `permissions exceed this key's cap: ${extra.join(", ")}`); } const mode = b.mode ?? "enforce"; - if (mode !== "enforce" && mode !== "observe") throw new HttpError(400, "`mode` must be enforce or observe"); - // Observe mode lets violating calls run, so an agent must not be able to pick it for itself. - if (mode === "observe" && !who.allow_observe) { + if (mode !== "enforce" && mode !== "observe" && mode !== "advise") throw new HttpError(400, "`mode` must be enforce, observe or advise"); + // Observe and advise mode let violating calls run, so an agent must not be able to pick them for itself. + if (mode !== "enforce" && !who.allow_observe) { throw new HttpError(403, "this credential may not open observe-mode runs (key option `allow_observe`)"); } const run = reversi.startRun({ @@ -212,6 +225,7 @@ export function createApp(opts: AppOptions): Hono { actor: who.name, scope: SealedScope.seal(perms as string[]), mode: mode as Mode, + tenant: who.tenant, reason: str(b, "reason", { optional: true }), }); const s = run.state(); @@ -314,21 +328,73 @@ export function createApp(opts: AppOptions): Hono { return c.json({ approvals: approvals.list({ status: status as never, run_id: c.req.query("run_id") }) }); }); + const identityOf = (p: Principal) => ({ roles: p.roles as string[], groups: p.groups ?? [] }); for (const decision of ["approve", "deny"] as const) { app.post(`/v1/approvals/:id/${decision}`, async (c) => { need(c, "approve"); const b = c.req.header("content-length") === "0" ? {} : await body(c).catch(() => ({})); const note = typeof b.note === "string" && b.note.trim() ? b.note : undefined; - const by = c.get("who").name; + const who = c.get("who"); try { - return c.json(decision === "approve" ? approvals.approve(c.req.param("id"), by, note) : approvals.deny(c.req.param("id"), by, note)); + return c.json( + decision === "approve" + ? approvals.approve(c.req.param("id"), who.name, note, identityOf(who)) + : approvals.deny(c.req.param("id"), who.name, note, identityOf(who)), + ); } catch (e) { - if (e instanceof ApprovalError) throw new HttpError(/not found/.test(e.message) ? 404 : 409, e.message); + if (e instanceof ApprovalError) throw new HttpError(/not found/.test(e.message) ? 404 : /needs an approver|cannot decide|already decided/.test(e.message) ? 403 : 409, e.message); throw e; } }); } + /** Emergency override of an approval: admin only, with a mandatory reason and evidence reference. */ + app.post("/v1/approvals/:id/break-glass", async (c) => { + need(c, "halt"); + const b = await body(c); + const who = c.get("who"); + try { + return c.json(approvals.breakGlass(c.req.param("id"), who.name, { reason: String(b.reason ?? ""), evidence: String(b.evidence ?? "") }, identityOf(who))); + } catch (e) { + if (e instanceof ApprovalError) throw new HttpError(/not found/.test(e.message) ? 404 : /needs the admin/.test(e.message) ? 403 : /needs a (reason|reference)/.test(e.message) ? 400 : 409, e.message); + throw e; + } + }); + + // ---- delegation: hand your approval authority to someone for a while ---- + app.get("/v1/delegations", (c) => { + need(c, "read"); + if (!opts.delegations) throw new HttpError(404, "delegation is not enabled on this server"); + return c.json({ delegations: opts.delegations.list() }); + }); + app.post("/v1/delegations", async (c) => { + need(c, "approve"); + if (!opts.delegations) throw new HttpError(404, "delegation is not enabled on this server"); + const b = await body(c); + const who = c.get("who"); + // You delegate your own authority. An admin may delegate on behalf of someone else. + const from = typeof b.from === "string" && b.from !== who.name ? (can(who, "halt") ? b.from : (() => { throw new HttpError(403, "you can only delegate your own approval authority"); })()) : who.name; + const hours = Number(b.hours); + if (!Number.isFinite(hours) || hours <= 0) throw new HttpError(400, "`hours` must be a positive number"); + try { + return c.json( + opts.delegations.grant({ from, to: str(b, "to") as string, until_ms: now() + hours * 3_600_000, groups: Array.isArray(b.groups) ? (b.groups as string[]) : undefined, reason: str(b, "reason") as string }), + 201, + ); + } catch (e) { + throw new HttpError(400, e instanceof Error ? e.message : "invalid delegation"); + } + }); + app.delete("/v1/delegations/:id", (c) => { + need(c, "approve"); + if (!opts.delegations) throw new HttpError(404, "delegation is not enabled on this server"); + const d = opts.delegations.list().find((x) => x.id === c.req.param("id")); + if (!d) throw new HttpError(404, "delegation not found"); + if (d.from !== c.get("who").name && !can(c.get("who"), "halt")) throw new HttpError(403, "only the delegator or an admin can revoke it"); + opts.delegations.revoke(d.id); + return c.json({ ok: true }); + }); + /** Pre-approve a tool in a run, for any arguments (counts down per use). */ app.post("/v1/runs/:id/approvals", async (c) => { need(c, "approve"); @@ -378,15 +444,31 @@ export function createApp(opts: AppOptions): Hono { app.post("/v1/halt", async (c) => { need(c, "halt"); const b = await body(c); - reversi.halt(`${str(b, "reason")} (by ${c.get("who").name})`); - return c.json({ halted: reversi.halted() }); + const scope: Record = {}; + for (const k of ["tool", "actor", "tenant", "environment"]) { + if (b[k] !== undefined) { + if (typeof b[k] !== "string" || !b[k]) throw new HttpError(400, `\`${k}\` must be a non-empty string`); + scope[k] = b[k] as string; + } + } + const h = reversi.halt(`${str(b, "reason")} (by ${c.get("who").name})`, scope, c.get("who").name); + return c.json({ halted: reversi.halted(), halt: h }); }); - app.post("/v1/resume", (c) => { + app.get("/v1/halts", (c) => { + need(c, "read"); + return c.json({ halts: reversi.halts() }); + }); + app.post("/v1/resume", async (c) => { need(c, "halt"); - reversi.resume(); - return c.json({ halted: reversi.halted() }); + const b = c.req.header("content-length") === "0" || !c.req.header("content-type") ? {} : await body(c).catch(() => ({})); + const lifted = reversi.resume(typeof b.id === "string" ? b.id : undefined); + return c.json({ halted: reversi.halted(), lifted }); }); + const extras = mountExtras(app, { opts, ring, now, need, body, str, identityOf }); + + const openRole = opts.console?.role ?? "operator"; + if (opts.console?.mode === "open" && openRole === "admin") throw new Error("the open console never runs as admin: use operator, or log in"); mountConsole(app as unknown as Hono, { reversi, approvals, @@ -399,11 +481,16 @@ export function createApp(opts: AppOptions): Hono { report: runReport, bundle: evidenceBundle, now, + mode: opts.console?.mode ?? "off", + open_role: openRole, + opts, + extras, + ring, }); app.notFound((c) => c.json({ error: "not found" }, 404)); app.onError((e, c) => { - if (e instanceof HttpError) return c.json({ error: e.message }, e.status); + if (e instanceof HttpError || e instanceof ApiError) return c.json({ error: e.message }, e.status); if (e instanceof LedgerError) return c.json({ error: "ledger unavailable; request refused" }, 503); if (e instanceof ReversiError) return c.json({ error: e.message }, 409); if (e instanceof Error && /^invalid run id/.test(e.message)) return c.json({ error: e.message }, 400); diff --git a/packages/server/src/console-admin.ts b/packages/server/src/console-admin.ts new file mode 100644 index 0000000..fb3de45 --- /dev/null +++ b/packages/server/src/console-admin.ts @@ -0,0 +1,177 @@ +/** Console pages for operating Reversi: alerts, connectors, tool inventory, run comparison, and settings (keys, roles, holds, retention, channels). */ +import { ROLES, applyRetention, evaluateAlerts, planRetention, searchRuns, type Role } from "@multimindlab/reversi"; +import { P, esc, time, type Kit } from "./console-kit.js"; + +const CAP_TABLE: Array<[Role, string]> = [ + ["agent", "open and use its own runs"], + ["auditor", "read runs, evidence and reports"], + ["approver", "read, and approve or deny calls"], + ["operator", "read, approve, and roll back"], + ["admin", "everything, including kill switches, publishing, keys and holds"], +]; + +export function mountAdminPages(k: Kit): void { + const { app, guarded, render, form, deps } = k; + const engine = () => deps.opts.engine; + const sev = (s: string) => (s === "critical" ? "bad" : s === "warning" ? "warn" : ""); + + app.get( + "/console/alerts", + guarded({ cap: "read" }, async (c, s) => { + const health = deps.opts.health ? await deps.opts.health().catch(() => undefined) : undefined; + const alerts = evaluateAlerts(k.rv, deps.approvals, { now: deps.now(), health, ...(engine()?.config.alerts?.denial_spike ? { denial_spike: engine()?.config.alerts?.denial_spike } : {}) }); + const rows = alerts + .map((a) => `${esc(a.severity)}${esc(a.title)}${esc(a.detail)}${time(a.ts_ms)}${a.run_id ? `${esc(a.run_id)}` : ""}`) + .join(""); + return render(c, s, "Alerts", `

    Computed live from the ledger, approvals and upstream health. ${engine()?.alerts ? "Each alert is also sent once to the notification channels, with reminders while it lasts." : "Notifications are not configured."}

    ${rows || ''}
    SeverityWhatDetailSinceRun
    Nothing needs attention.
    `); + }), + ); + + app.get( + "/console/connectors", + guarded({ cap: "read" }, async (c, s) => { + const list = deps.opts.connectors ? await deps.opts.connectors().catch(() => []) : []; + const rows = list + .map((x) => `${esc(x.name)}${esc(x.kind)}${x.healthy ? "healthy" : "unreachable"}${x.tools}${x.classified}${x.unclassified ? `${x.unclassified}` : 0}${esc(x.credential ?? "")}${esc(x.detail ?? "")}`) + .join(""); + return render(c, s, "Connectors", `

    The systems whose tools Reversi governs. Unclassified tools have no manifest: they are blocked in enforce mode and only observed in observe mode.

    ${rows || ''}
    NameKindHealthToolsWith manifestUnclassifiedCredentialDetail
    No connectors are reported. The gateway supplies this list.
    `); + }), + ); + + app.get( + "/console/tools", + guarded({ cap: "read" }, async (c, s) => { + const manifested = new Set(k.rv.registry.list().map((m) => m.tool)); + let upstream: Array<{ name: string }> = []; + try { + upstream = deps.opts.upstream_tools ? await deps.opts.upstream_tools() : []; + } catch { + /* upstream unreachable: show the manifests only */ + } + const used = new Map(); + for (const r of searchRuns(k.rv, { limit: 500 }).runs) for (const t of r.tools) used.set(t, (used.get(t) ?? 0) + 1); + const unclassified = upstream.filter((t) => !manifested.has(t.name)); + const names = [...new Set([...manifested, ...upstream.map((t) => t.name), ...used.keys()])].sort(); + const rows = names + .map((n) => { + const m = k.rv.registry.get(n); + return `${esc(n)}${m ? esc(m.effect) : 'unclassified'}${m ? k.statusBadge(m.status) : ""}${upstream.some((t) => t.name === n) ? "yes" : upstream.length ? "no" : ""}${used.get(n) ?? 0}${m ? `manifest` : k.allowed(s, "rollback") ? 'draft one' : ""}`; + }) + .join(""); + return render(c, s, "Tool inventory", `

    ${manifested.size} classified, ${unclassified.length} unclassified${upstream.length ? ` of ${upstream.length} offered by the upstream` : ""}. Use counts come from the 500 most recent runs.

    ${rows || ''}
    ToolEffectStatusOffered nowRuns
    No tools yet.
    `); + }), + ); + + app.get( + "/console/compare", + guarded({ cap: "read" }, (c, s) => { + const a = c.req.query("a") ?? ""; + const b = c.req.query("b") ?? ""; + const side = (id: string) => { + const st = id ? k.rv.runs.get(id) : undefined; + if (!st) return id ? `

    No run ${esc(id)}.

    ` : ""; + return `${st.steps.map((x) => ``).join("")}
    #ToolEffectStatus
    ${x.step}${esc(x.tool)}${esc(x.effect)}${esc(x.status)}

    ${esc(st.actor)} · ${st.mode} · ${time(st.created_ms)}

    `; + }; + return render(c, s, "Compare runs", `
    ${side(a)}
    ${side(b)}
    `); + }), + ); + + // ================= settings ================= + app.get( + "/console/settings", + guarded({ cap: "read" }, (c, s) => { + const e = engine(); + const admin = k.allowed(s, "halt"); + const keys = deps.extras.keys; + const keyRows = keys.enabled() + ? keys + .list() + .map((x) => `${esc(x.name)}${esc(x.roles.join(", "))}${esc(x.groups.join(", "))}${esc(x.tenant ?? "")}${x.expires_at ? esc(typeof x.expires_at === "number" ? time(x.expires_at) : x.expires_at) : ""}${esc(x.fingerprint.slice(0, 12))}${x.disabled ? 'disabled' : 'active'}${admin ? form(s, `/console/settings/keys/${x.index}/${x.disabled ? "enable" : "disable"}`, "", x.disabled ? "Enable" : "Disable") + " " + form(s, `/console/settings/keys/${x.index}/remove`, "", "Remove", "bad") : ""}`) + .join("") + : ""; + const holds = e?.holds.list() ?? []; + const cfg = k.rv.policy?.doc.retention; + const plan = cfg && e ? planRetention(k.rv, cfg, e.holds, deps.now()) : []; + const due = plan.filter((p) => p.due && !p.held).length; + const channels = e?.notifier?.channelNames() ?? []; + const body = ` +

    Environment

    Environment ${esc(k.rv.environment ?? "(not set)")} · console mode ${esc(deps.mode)}${deps.mode === "open" ? ` as ${esc(deps.open_role)}` : ""} · ledger head ${esc(String(k.rv.ledger.head()).slice(0, 16))}${e?.config.capture?.enabled ? " · capture on" : " · capture off"}${e?.config.egress ? " · egress guard on" : ""}
    +

    Roles

    ${CAP_TABLE.map(([r, d]) => ``).join("")}
    RoleMay
    ${r}${esc(d)}
    +

    Users and API keys

    ${keys.enabled() ? `${keyRows}
    NameRolesGroupsTenantExpiresFingerprintState
    ${admin ? `

    ` : ""}` : '

    Keys are managed outside the server (no keys file is configured), or sign-in uses OIDC.

    '} +

    Legal holds

    ${e ? `${holds.map((h) => ``).join("") || ''}
    IdScopeReasonBySince
    ${esc(h.id)}${esc(JSON.stringify(h.scope))}${esc(h.reason)}${esc(h.by)}${time(h.ts_ms)}${admin ? form(s, `/console/settings/holds/${encodeURIComponent(h.id)}/release`, "", "Release") : ""}
    No holds.
    ${admin ? `

    ` : ""}` : '

    Needs the engine services.

    '} +

    Retention

    ${cfg ? `

    Default ${cfg.default_days} days. ${plan.length} runs tracked, ${due} due for deletion now (holds respected).

    ${admin ? `${form(s, "/console/settings/retention", '', "Dry run")} ${form(s, "/console/settings/retention", "", "Apply now (deletes runs)", "bad")}` : ""}` : '

    No retention: section in the policy: nothing is ever deleted.

    '} +

    Notification channels

    ${channels.length ? `

    ${channels.map((n) => `${esc(n)}`).join(", ")}

    ${admin ? form(s, "/console/settings/notify-test", "", "Send a test notification") : ""}` : '

    None configured (notifications: in the project file).

    '} +

    Integrations

    • Slack approvals: ${deps.opts.slack ? "enabled" : "off"}
    • OpenTelemetry traces: ${e?.otel ? "on" : "off"}
    • Anchoring: ${e?.config.anchor ? `${e.config.anchor.sinks.length} sink(s)` : "off"}
    • Evidence signing: ${deps.signing_available ? "on" : "off"}
    • Data residency: ${e?.config.egress ? "outbound addresses are checked against the egress rules" : "no egress rules; set egress: to restrict where data can be sent"}
    `; + return render(c, s, "Settings", body); + }), + ); + + const back = (c: Parameters[0]) => k.redirect(c, "/console/settings"); + const fail = (c: Parameters[0], s: Parameters[1], e: unknown) => render(c, s, "Could not do that", `

    ${esc(e instanceof Error ? e.message : e)}

    `, 409); + + app.post( + "/console/settings/keys", + guarded({ cap: "halt", post: true }, (c, s, f) => { + try { + const g = (n: string) => String(f?.get(n) ?? "").trim(); + const role = g("role") as Role; + if (!ROLES.includes(role)) throw new Error("unknown role"); + const key = deps.extras.keys.create({ name: g("name"), roles: [role], groups: g("groups").split(/[,\s]+/).filter(Boolean), tenant: g("tenant") || undefined, expires_in_days: g("days") ? Number(g("days")) : undefined }); + return render(c, s, "Key created", `

    Copy this key now. It is not stored and cannot be shown again.

    ${esc(key)}

    Back to settings

    `); + } catch (e) { + return fail(c, s, e); + } + }), + ); + for (const act of ["disable", "enable", "remove"] as const) { + app.post( + `/console/settings/keys/:i/${act}`, + guarded({ cap: "halt", post: true }, (c, s) => { + try { + const i = Number(P(c, "i")); + if (act === "remove") deps.extras.keys.remove(i); + else deps.extras.keys.disable(i, act === "disable"); + return back(c); + } catch (e) { + return fail(c, s, e); + } + }), + ); + } + app.post( + "/console/settings/holds", + guarded({ cap: "halt", post: true }, (c, s, f) => { + try { + const g = (n: string) => String(f?.get(n) ?? "").trim() || undefined; + engine()?.holds.add({ run_id: g("run_id"), tenant: g("tenant"), actor: g("actor"), tool: g("tool") }, g("reason") ?? "", s.name); + return back(c); + } catch (e) { + return fail(c, s, e); + } + }), + ); + app.post( + "/console/settings/holds/:id/release", + guarded({ cap: "halt", post: true }, (c, s) => { + engine()?.holds.release(P(c, "id")); + return back(c); + }), + ); + app.post( + "/console/settings/retention", + guarded({ cap: "halt", post: true }, (c, s, f) => { + const e = engine(); + const cfg = k.rv.policy?.doc.retention; + if (!e || !cfg) return fail(c, s, "no retention settings"); + const rep = applyRetention(k.rv, cfg, { holds: e.holds, approvals: e.approvals, capture: e.capture, dry_run: f?.get("dry") === "1", key: e.signing_key, now: deps.now(), actor: s.name }); + return render(c, s, "Retention", `

    ${rep.dry_run ? "Dry run: nothing deleted. Would delete" : "Deleted"} ${rep.purged.length} runs; skipped ${rep.skipped.length}.

      ${rep.skipped.map((x) => `
    • ${esc(x.run_id)}: ${esc(x.reason)}
      • `).join("")}

    Back

    `); + }), + ); + app.post( + "/console/settings/notify-test", + guarded({ cap: "halt", post: true }, async (c, s) => { + const r = await engine()?.notifier?.notify({ event: "test", severity: "info", title: "Reversi test notification", text: `Sent from the console by ${s.name}.` }); + return render(c, s, "Test notification", `

    Sent to: ${esc((r?.sent ?? []).join(", ") || "nobody")}. Failed: ${esc((r?.failed ?? []).join(", ") || "none")}.

    Back

    `); + }), + ); +} diff --git a/packages/server/src/console-editors.ts b/packages/server/src/console-editors.ts new file mode 100644 index 0000000..b01de21 --- /dev/null +++ b/packages/server/src/console-editors.ts @@ -0,0 +1,518 @@ +/** Console pages for editing: manifests (form, YAML, diff, publish workflow, import), policy (editor, builder, versions, simulation), and impact previews. */ +import { readFileSync } from "node:fs"; +import { parse as parseYaml, stringify } from "yaml"; +import { + ManifestRegistry, + PolicyVersions, + Registry, + authorForm, + lintManifests, + parsePolicy, + previewImpact, + replayRun, + renderDiff, + runManifestTests, + runPolicyTests, + simulatePolicy, + suggestManifests, + validateManifest, + type ArgChange, + type McpToolInfo, + type PolicyDoc, +} from "@multimindlab/reversi"; +import { P, esc, time, type Kit } from "./console-kit.js"; + +const YAML_OPTS = { lineWidth: 100 } as const; + +function dumpManifest(m: unknown): string { + return stringify(typeof m === "object" && m !== null && "approval_rules" in m ? authorForm(m as never) : m, YAML_OPTS); +} + +/** Parse a manifest from the editor: YAML or JSON, one manifest. */ +function readManifestText(text: string): { manifest?: Record; errors: string[] } { + try { + const doc = parseYaml(text) as unknown; + if (typeof doc !== "object" || doc === null || Array.isArray(doc)) return { errors: ["the text must be one manifest (a mapping)"] }; + const v = validateManifest(doc); + return v.ok ? { manifest: doc as Record, errors: [] } : { errors: v.errors }; + } catch (e) { + return { errors: [`cannot read it as YAML: ${e instanceof Error ? e.message : String(e)}`] }; + } +} + +const TEMPLATE = `tool: service.do_thing +description: What this tool does, in a sentence. +effect: reversible # read_only | idempotent | reversible | compensable | irreversible +permission: service:write +owner: team@example.com +risk: { data_class: internal, blast_radius: low } +timeout_ms: 30s +compensation: + tool: service.undo_thing + kind: inverse # inverse = restores exact state; compensation = may leave residue + argument_map: + id: $result.id +`; + +export function mountEditorPages(k: Kit): void { + const { app, guarded, render, form, deps } = k; + const reg = (): ManifestRegistry | undefined => (deps.opts.registry_dir ? new ManifestRegistry(deps.opts.registry_dir) : undefined); + const pver = (): PolicyVersions | undefined => (deps.opts.policy_dir ? new PolicyVersions(deps.opts.policy_dir) : undefined); + const findings = (r: Registry) => lintManifests(r); + const reload = async (by: string) => (await deps.opts.engine?.reload(by))?.error; + + // ================= manifests ================= + app.get( + "/console/manifests", + guarded({ cap: "read" }, (c, s) => { + const live = k.rv.registry.list(); + const lint = findings(k.rv.registry); + const r = reg(); + const rows = live + .map((m) => { + const f = lint.filter((x) => x.tool === m.tool); + const errs = f.filter((x) => x.level === "error").length; + const warns = f.filter((x) => x.level === "warning").length; + return `${esc(m.tool)}${esc(m.effect)}${k.statusBadge(m.status)}v${m.version}${esc(m.owner ?? "")}${esc(m.risk?.data_class ?? "")}${m.compensation ? `${esc(m.compensation.tool)}${m.compensation.verification ? " (verified)" : ""}` : 'none'}${errs ? `${errs} errors ` : ""}${warns ? `${warns} warnings` : ""}`; + }) + .join(""); + const drafts = r ? r.tools().filter((t) => r.history(t).at(-1)?.status === "draft") : []; + return render( + c, + s, + "Manifests", + `

    ${live.length} enforced manifests. ${r ? `Registry: ${esc(deps.opts.registry_dir)}.` : 'The manifests are plain files, so the console shows them but cannot change them. Publish them into a registry folder (reversi manifests publish) and point the server at it to edit them here.'}

    +

    ${r && k.allowed(s, "rollback") ? 'New manifest · Draft from an MCP server\'s tools · ' : ""}Tool inventory

    +${drafts.length ? `
    Drafts waiting for approval: ${drafts.map((t) => `${esc(t)}`).join(", ")}
    ` : ""} +${rows || ''}
    ToolEffectStatusVersionOwnerData classUndoLint
    No manifests.
    `, + ); + }), + ); + + // Form-mode builder: a form that produces the YAML. + const builderForm = (s: Parameters[0], seed: Record = {}) => { + const v = (n: string) => esc(seed[n] ?? ""); + const rows = [0, 1, 2, 3].map((i) => ``).join(""); + return `
    +

    +

    +

    +

    +

    +

    +

    Undo (reversible and compensable tools)

    +

    +

    +${rows}
    Undo argumentWhere it comes fromPath
    +

    Verify the undo (optional)

    +

    +
    `; + }; + + const editor = (s: Parameters[0], o: { action: string; text: string; title?: string; extra?: string; messages?: string }) => `${o.messages ?? ""} +
    +

    +

    ${k.allowed(s, "rollback") && reg() ? '' : ""} ${k.allowed(s, "halt") && reg() ? '' : ""}

    ${o.extra ?? ""}
    `; + + const validation = (text: string, currentText?: string): string => { + const r = readManifestText(text); + if (!r.manifest) return `
    Not valid:
      ${r.errors.map((e) => `
    • ${esc(e)}
      • `).join("")}
    `; + let lint = ""; + try { + const m = r.manifest; + const trial = new Registry(); + for (const x of k.rv.registry.list()) if (x.tool !== m.tool) trial.register(x); + trial.register(m); + const f = lintManifests(trial).filter((x) => x.tool === m.tool); + lint = f.length ? `
      ${f.map((x) => `
    • ${esc(x.level)} ${esc(x.code)}: ${esc(x.message)}
      • `).join("")}
    ` : "no findings"; + const problems = trial.validate(); + if (problems.length) lint += `

    references: ${esc(problems.join("; "))}

    `; + } catch (e) { + lint = `${esc(e instanceof Error ? e.message : e)}`; + } + const diff = currentText && currentText.trim() !== text.trim() ? `
    ${esc(renderDiff(currentText, text, 3))}
    ` : currentText ? '

    no change from the current version

    ' : ""; + return `
    Valid. Lint: ${lint}${diff ? `
    Changes${diff}` : ""}
    `; + }; + + app.get( + "/console/manifests/new", + guarded({ cap: "rollback" }, (c, s) => render(c, s, "New manifest", `

    Fill in the form, or write the YAML directly.

    ${builderForm(s)}

    Or write the YAML

    ${editor(s, { action: "/console/manifests/save", text: TEMPLATE })}`)), + ); + + app.post( + "/console/manifests/build", + guarded({ cap: "rollback", post: true }, (c, s, f) => { + const g = (n: string) => String(f?.get(n) ?? "").trim(); + const seed: Record = {}; + for (const n of ["tool", "description", "effect", "permission", "owner", "data_class", "blast_radius", "timeout", "approval", "approvers", "quorum", "comp_tool", "comp_kind", "residue", "ver_tool", "ver_arg", "ver_ref", "ver_expect"]) seed[n] = g(n); + const m: Record = { tool: g("tool"), ...(g("description") ? { description: g("description") } : {}), effect: g("effect"), permission: g("permission") }; + if (g("owner")) m.owner = g("owner"); + if (g("data_class") || g("blast_radius")) m.risk = { ...(g("data_class") ? { data_class: g("data_class") } : {}), ...(g("blast_radius") ? { blast_radius: g("blast_radius") } : {}) }; + if (g("timeout")) m.timeout_ms = g("timeout"); + if (g("approvers")) m.approval = { roles: g("approvers").split(/[,\s]+/).filter(Boolean), quorum: Number(g("quorum") || 1) }; + else if (g("approval") === "always") m.approval = "always"; + if (g("comp_tool")) { + const argument_map: Record = {}; + for (let i = 0; i < 4; i++) { + const n = g(`map_name${i}`); + const p = g(`map_path${i}`); + seed[`map_name${i}`] = n; + seed[`map_path${i}`] = p; + seed[`map_src${i}`] = g(`map_src${i}`); + if (n && p) argument_map[n] = `${g(`map_src${i}`) || "$result"}.${p}`; + } + const comp: Record = { tool: g("comp_tool"), kind: g("comp_kind") || "compensation", argument_map }; + const residue = g("residue").split("\n").map((x) => x.trim()).filter(Boolean); + if (residue.length) comp.residue = residue; + if (g("ver_tool") && g("ver_expect")) { + const expect: Record = {}; + for (const part of g("ver_expect").split(",")) { + const [pa, ...va] = part.split("="); + if (pa?.trim()) expect[pa.trim()] = va.join("=").trim(); + } + comp.verification = { tool: g("ver_tool"), arguments: g("ver_arg") && g("ver_ref") ? { [g("ver_arg")]: g("ver_ref") } : {}, expect }; + } + m.compensation = comp; + } + const text = dumpManifest(m); + return render(c, s, "New manifest", `${builderForm(s, seed)}

    The YAML

    ${editor(s, { action: "/console/manifests/save", text, messages: validation(text) })}`); + }), + ); + + app.post( + "/console/manifests/save", + guarded({ cap: "rollback", post: true }, async (c, s, f) => { + const text = String(f?.get("text") ?? ""); + const action = String(f?.get("do") ?? "validate"); + const parsed = readManifestText(text); + const back = (msgs: string, status = 200) => render(c, s, "Manifest", editor(s, { action: "/console/manifests/save", text, messages: msgs }), status); + if (action === "validate" || !parsed.manifest) return back(validation(text)); + const r = reg(); + if (!r) return back('
    No registry folder is configured, so this cannot be saved.
    ', 409); + if (action === "approved" && !k.allowed(s, "halt")) return back('
    Publishing as approved needs the admin role. Save it as a draft for an admin to approve.
    ', 403); + try { + const out = r.publish(parsed.manifest, { by: s.name, note: String(f?.get("note") ?? "").trim() || undefined, status: action === "approved" ? "approved" : "draft", key: deps.opts.engine?.signing_key }); + if (action === "approved") { + const err = await reload(s.name); + if (err) return back(`
    Saved v${out.version}, but the new configuration did not load: ${esc(err)}
    `, 409); + } + return c.redirect(`/console/manifests/${encodeURIComponent(out.tool)}`, 303); + } catch (e) { + return back(`
    ${esc(e instanceof Error ? e.message : e)}
    `, 400); + } + }), + ); + + app.get( + "/console/manifests/import", + guarded({ cap: "rollback" }, async (c, s) => { + let fetched = ""; + let note = ""; + if (deps.opts.upstream_tools) { + try { + fetched = JSON.stringify(await deps.opts.upstream_tools(), null, 2); + note = "These are the tools the connected upstream offers right now."; + } catch (e) { + note = `Could not fetch them: ${e instanceof Error ? e.message : e}`; + } + } + return render(c, s, "Draft manifests from MCP tools", `

    Paste the tools array from an MCP tools/list response. Reversi drafts a manifest for each tool from its name, schema and annotations. Annotations are the server's own claims: they only ever produce drafts that a person approves. ${esc(note)}

    +

    `); + }), + ); + app.post( + "/console/manifests/import", + guarded({ cap: "rollback", post: true }, (c, s, f) => { + let tools: McpToolInfo[]; + try { + const raw = JSON.parse(String(f?.get("tools") ?? "[]")) as unknown; + tools = (Array.isArray(raw) ? raw : (raw as { tools?: McpToolInfo[] }).tools) as McpToolInfo[]; + if (!Array.isArray(tools)) throw new Error("expected an array of tools"); + } catch (e) { + return render(c, s, "Draft manifests", `

    ${esc(e instanceof Error ? e.message : e)}

    `, 400); + } + const r = suggestManifests(tools, { server: String(f?.get("server") ?? "upstream"), existing: k.rv.registry }); + const body = r.suggestions + .map((sg) => { + const text = dumpManifest(sg.manifest); + return `
    ${esc(String(sg.manifest.tool))} ${sg.confidence} confidence
      ${sg.notes.map((n) => `
    • ${esc(n)}
      • `).join("")}
    ${editor(s, { action: "/console/manifests/save", text })}
    `; + }) + .join(""); + return render(c, s, "Draft manifests", `

    ${r.suggestions.length} drafted, ${r.skipped.length} skipped because they already have manifests.

    ${body}`); + }), + ); + + app.get( + "/console/manifests/:tool", + guarded({ cap: "read" }, (c, s) => { + const tool = P(c, "tool"); + const r = reg(); + const live = k.rv.registry.get(tool); + const hist = r && r.tools().includes(tool) ? r.history(tool) : []; + const current = r && hist.length ? r.current(tool, true) : live; + if (!current) return render(c, s, "Manifest not found", `

    No manifest for ${esc(tool)}.

    `, 404); + const text = dumpManifest(current); + const f = lintManifests(k.rv.registry).filter((x) => x.tool === tool); + const testText = `- name: forward success maps the undo\n tool: ${tool}\n args: {}\n expect_plan: ${current.compensation ? "compensate" : "skip"}\n`; + return render( + c, + s, + `Manifest ${tool}`, + `

    ${k.statusBadge(current.status)} v${current.version} · effect ${esc(current.effect)}${current.owner ? ` · owner ${esc(current.owner)}` : ""} · impact preview

    +${f.length ? `
    Lint
      ${f.map((x) => `
    • ${esc(x.level)} ${esc(x.code)}: ${esc(x.message)}
      • `).join("")}
    ` : ""} +

    Edit

    ${r ? editor(s, { action: "/console/manifests/save", text }) : `
    ${esc(text)}

    Read-only: the manifests are plain files.

    `} +${ + r && hist.length + ? `

    History

    ${hist + .map((h, i) => ``) + .join("")}
    VersionStatusWhenByNote
    v${h.version}${k.statusBadge(h.status)}${time(h.created_ms)}${esc(h.by)}${h.approved_by ? `, approved by ${esc(h.approved_by)}` : ""}${h.signature ? " (signed)" : ""}${esc(h.note ?? "")}${i > 0 ? `diff ` : ""}${k.allowed(s, "halt") ? form(s, `/console/manifests/${encodeURIComponent(tool)}/restore`, ``, "Restore") : ""}
    +${hist.at(-1)?.status === "draft" && k.allowed(s, "halt") ? form(s, `/console/manifests/${encodeURIComponent(tool)}/approve`, ` `, "Approve the latest draft", "ok") : ""} +${current.status === "approved" && k.allowed(s, "halt") ? form(s, `/console/manifests/${encodeURIComponent(tool)}/deprecate`, ``, "Deprecate") : ""}` + : "" +} +

    Contract tests

    Run these without touching any system. The YAML is the same format as reversi test.

    +

    `, + ); + }), + ); + + app.get( + "/console/manifests/:tool/diff", + guarded({ cap: "read" }, (c, s) => { + const r = reg(); + if (!r) return render(c, s, "Diff", "

    No registry.

    ", 404); + try { + const d = r.diff(P(c, "tool"), Number(c.req.query("a")), Number(c.req.query("b"))); + return render(c, s, `${P(c, "tool")}: v${c.req.query("a")} to v${c.req.query("b")}`, `
    ${esc(d || "(no differences)")}
    `); + } catch (e) { + return render(c, s, "Diff", `

    ${esc(e instanceof Error ? e.message : e)}

    `, 404); + } + }), + ); + for (const action of ["approve", "deprecate", "restore"] as const) { + app.post( + `/console/manifests/:tool/${action}`, + guarded({ cap: "halt", post: true }, async (c, s, f) => { + const r = reg(); + const tool = P(c, "tool"); + try { + if (!r) throw new Error("no registry folder is configured"); + const key = deps.opts.engine?.signing_key; + if (action === "approve") r.approve(tool, { by: s.name, key, require_other: f?.get("four_eyes") === "1" }); + else if (action === "deprecate") r.deprecate(tool, { by: s.name, key, replaced_by: String(f?.get("replaced_by") ?? "").trim() || undefined }); + else r.rollbackTo(tool, Number(f?.get("version")), { by: s.name, key }); + const err = await reload(s.name); + if (err) throw new Error(`saved, but the new configuration did not load: ${err}`); + } catch (e) { + return render(c, s, "Could not do that", `

    ${esc(e instanceof Error ? e.message : e)}

    `, 409); + } + return c.redirect(`/console/manifests/${encodeURIComponent(tool)}`, 303); + }), + ); + } + app.post( + "/console/manifests/:tool/test", + guarded({ cap: "read", post: true }, async (c, s, f) => { + let cases; + try { + cases = parseYaml(String(f?.get("tests") ?? "")) as never[]; + if (!Array.isArray(cases)) throw new Error("expected a list of test cases"); + } catch (e) { + return render(c, s, "Contract tests", `

    ${esc(e instanceof Error ? e.message : e)}

    `, 400); + } + const { results } = await runManifestTests(k.rv.registry, cases, { policy: k.rv.policy?.doc, environment: k.rv.environment }); + return render(c, s, "Contract test results", `${results.map((x) => ``).join("")}
    TestResult
    ${esc(x.name)}${x.ok ? 'pass' : `fail
      ${x.problems.map((p) => `
    • ${esc(p)}
      • `).join("")}
    `}

    Back

    `); + }), + ); + + // ================= policy ================= + app.get( + "/console/policy", + guarded({ cap: "read" }, (c, s) => { + const pv = pver(); + const doc = k.rv.policy?.doc; + let text = ""; + let active: number | null = null; + if (pv) { + active = pv.activeVersion(); + text = active === null ? "version: 1\nrules: []\n" : pv.text(active); + } else if (deps.opts.engine?.config.policy) { + try { + text = readFileSync(deps.opts.engine.config.policy, "utf8"); + } catch { + text = ""; + } + } + const tests = doc ? runPolicyTests(doc) : []; + const hist = pv ? pv.history() : []; + const body = `

    ${doc ? `${doc.rules.length} rules, ${doc.limits.length} limits, ${doc.budgets.length} budgets, ${doc.freezes.length} freezes, default ${esc(doc.default)}${doc.risk ? ", risk scoring on" : ""}.` : "No policy is loaded: only manifests and scopes restrict calls."} ${pv ? `Versions folder: ${esc(deps.opts.policy_dir)}, active ${active === null ? "none" : `v${active}`}.` : 'The policy is a plain file, so the console shows it but cannot change it. Point policy: at a folder (reversi policy publish) to edit it here.'}

    +${tests.length ? `

    ${tests.filter((t) => t.ok).length} of ${tests.length} of the policy's own tests pass.

    ` : ""} +

    Rule builder

    +
    +

    +

    When the call matches:

    +

    and, optionally:

    +

    for approvals:

    +
    +

    Policy

    +

    +

    ${pv && k.allowed(s, "halt") ? ' ' : ""}

    +${hist.length ? `

    Versions

    ${hist.map((h, i) => ``).join("")}
    VersionWhenByTestsNote
    v${h.version}${h.version === active ? ' active' : ""}${time(h.created_ms)}${esc(h.by)}${h.tests_passed}${esc(h.note ?? "")}${i > 0 ? `diff ` : ""}${k.allowed(s, "halt") && h.version !== active ? form(s, "/console/policy/activate", ``, "Activate") : ""}
    ` : ""}`; + return render(c, s, "Policy", body); + }), + ); + + app.post( + "/console/policy/rule", + guarded({ cap: "read", post: true }, (c, s, f) => { + const g = (n: string) => String(f?.get(n) ?? "").trim(); + let doc: Record; + try { + doc = (parseYaml(g("text")) as Record) ?? { version: 1 }; + } catch { + doc = { version: 1 }; + } + const rule: Record = { id: g("id"), match: {}, action: g("action") }; + const match = rule.match as Record; + for (const n of ["tool", "effect", "environment", "tenant", "data_class"]) if (g(n)) match[n] = g(n); + if (g("path")) { + let value: unknown = g("value"); + try { + value = JSON.parse(g("value")); + } catch { + /* a plain string */ + } + rule.when = [{ path: g("path"), op: g("op"), ...(g("op") === "exists" ? {} : { value }) }]; + } + if (g("reason")) rule.reason = g("reason"); + if (g("action") === "require_approval" && g("approvers")) { + rule.approvers = g("approvers").split(/[,\s]+/).filter(Boolean); + rule.quorum = Number(g("quorum") || 1); + } + doc.rules = [...((doc.rules as unknown[]) ?? []), rule]; + const text = stringify(doc, YAML_OPTS); + return render(c, s, "Policy", `

    Rule added below. Validate it, then publish.

    ${pver() && k.allowed(s, "halt") ? '' : ""}

    `); + }), + ); + + app.post( + "/console/policy/save", + guarded({ cap: "read", post: true }, async (c, s, f) => { + const text = String(f?.get("text") ?? ""); + const action = String(f?.get("do") ?? "validate"); + const again = (msgs: string, status = 200) => + render(c, s, "Policy", `${msgs}

    ${pver() && k.allowed(s, "halt") ? '' : ""}

    `, status); + let doc: PolicyDoc; + try { + doc = parsePolicy(parseYaml(text)); + } catch (e) { + const errs = (e as { errors?: string[] }).errors ?? [e instanceof Error ? e.message : String(e)]; + return again(`
    Not valid:
      ${errs.map((x) => `
    • ${esc(x)}
      • `).join("")}
    `); + } + const results = runPolicyTests(doc); + const bad = results.filter((r) => !r.ok); + const testHtml = `
    ${bad.length ? `${bad.length} of ${results.length} tests fail` : `Valid. ${results.length} tests pass`}.
      ${results.map((r) => `
    • ${r.ok ? "ok" : "FAIL"}: ${esc(r.name)}${r.ok ? "" : ` (expected ${esc(r.expected)}, got ${esc(r.got)})`}
      • `).join("")}
    `; + const pv = pver(); + const activeText = pv && pv.activeVersion() !== null ? pv.text(pv.activeVersion() as number) : undefined; + const diff = activeText !== undefined && activeText !== text ? `
    Changes from the active version
    ${esc(renderDiff(activeText, text, 3))}
    ` : ""; + if (action === "validate") return again(testHtml + diff); + if (action === "simulate") { + const cap = deps.opts.engine?.capture; + if (!cap) return again(`${testHtml}
    Capture is off, so there is no history to simulate against. Turn it on (capture: enabled: true) to keep redacted arguments for a limited time.
    `); + const rep = await simulatePolicy(cap.list({ since_ms: deps.now() - 7 * 86_400_000 }), { registry: k.rv.registry, policy: doc, environment: k.rv.environment }); + const row = (title: string, l: typeof rep.newly_blocked) => (l.length ? `

    ${title} (${l.length})

    ${l.slice(0, 100).map((x) => ``).join("")}
    RunToolChangeWhy
    ${esc(x.run)}${esc(x.tool)}${esc(x.before)} to ${esc(x.after)}${esc(x.reasons[0] ?? "")}
    ` : ""); + return again(`${testHtml}
    Last 7 days of captured calls: ${rep.calls} calls in ${rep.runs} runs, ${rep.unchanged} unchanged.${row("Newly blocked", rep.newly_blocked)}${row("Newly needing approval", rep.newly_needs_approval)}${row("Newly allowed", rep.newly_allowed)}${rep.notes.map((n) => `

    ${esc(n)}

    `).join("")}
    `); + } + if (!pv || !k.allowed(s, "halt")) return again(`${testHtml}
    Publishing needs a policy versions folder and the admin role.
    `, 403); + try { + pv.publish(text, { by: s.name, note: String(f?.get("note") ?? "").trim() || undefined, activate: action === "publish" }); + const err = action === "publish" ? await reload(s.name) : undefined; + if (err) return again(`
    Saved, but the new policy did not load: ${esc(err)}
    `, 409); + } catch (e) { + return again(`${testHtml}
    ${esc(e instanceof Error ? e.message : e)}
    `, 400); + } + return c.redirect("/console/policy", 303); + }), + ); + app.get( + "/console/policy/diff", + guarded({ cap: "read" }, (c, s) => { + const pv = pver(); + if (!pv) return render(c, s, "Policy diff", "

    No policy versions folder.

    ", 404); + try { + return render(c, s, `Policy v${c.req.query("a")} to v${c.req.query("b")}`, `
    ${esc(pv.diff(Number(c.req.query("a")), Number(c.req.query("b"))) || "(no differences)")}
    `); + } catch (e) { + return render(c, s, "Policy diff", `

    ${esc(e instanceof Error ? e.message : e)}

    `, 404); + } + }), + ); + app.post( + "/console/policy/activate", + guarded({ cap: "halt", post: true }, async (c, s, f) => { + try { + pver()?.activate(Number(f?.get("version"))); + const err = await reload(s.name); + if (err) throw new Error(err); + } catch (e) { + return render(c, s, "Could not activate", `

    ${esc(e instanceof Error ? e.message : e)}

    `, 409); + } + return c.redirect("/console/policy", 303); + }), + ); + + // ================= impact, replay ================= + app.get( + "/console/impact", + guarded({ cap: "read" }, async (c, s) => { + const tool = c.req.query("tool") ?? ""; + const argsText = c.req.query("args") ?? ""; + let result = ""; + if (tool) { + let args: unknown; + let bad = ""; + if (argsText) { + try { + args = JSON.parse(argsText); + } catch { + bad = "The arguments are not valid JSON."; + } + } + if (bad) result = `
    ${bad}
    `; + else { + const p = await previewImpact({ registry: k.rv.registry, policy: k.rv.policy?.doc, environment: k.rv.environment }, { tool, args, actor: c.req.query("actor") || undefined }); + result = `

    ${esc(p.tool)}: ${p.known ? esc(p.effect) : "no manifest"}${p.status && p.status !== "approved" ? ` (${esc(p.status)})` : ""}. Decision now: ${esc(p.verdict)}${p.reasons.length ? ` (${esc(p.reasons[0])})` : ""}

    +${p.risk_score ? `

    Risk score ${p.risk_score.score}: ${esc(p.risk_score.factors.join(", ") || "no factors")}

    ` : ""} +${p.risk ? `

    Data class ${esc(p.risk.data_class ?? "?")}, blast radius ${esc(p.risk.blast_radius ?? "?")}${p.owner ? `, owner ${esc(p.owner)}` : ""}

    ` : ""} +${p.requires.length ? `

    Needs first: ${p.requires.map((r) => `${esc(r)}`).join(", ")}

    ` : ""} +${p.approval ? `

    Approvers: ${esc(p.approval.roles.join(" or ") || "any")}, ${p.approval.quorum} needed

    ` : ""} +

    If approved, rollback outlook:

      ${p.rollback_outlook.map((l) => `
    • ${esc(l)}
      • `).join("")}
    +${p.warnings.length ? `
      ${p.warnings.map((w) => `
    • ${esc(w)}
      • `).join("")}
    ` : ""}
    `; + } + } + const names = k.rv.registry.list().map((m) => `