Document alert.trigger/resolve and add alert console screenshots

content/automation/alerts.md

@@ -23,6 +23,8 @@ lead: 'An alert rule watches one metric on one deployment and fires when it stay
 
 ## Create a rule
 
+{{< shot src="/img/alert-list.png" url="console.deploys.app/alert?project=acme" alt="The alert rules list with ok, firing, nodata, and disabled statuses" caption="Each rule shows its target, condition, status, and last evaluated value." >}}
+
 From the console, open **Alerts** and click **Create rule**. Or use the CLI:
 
 ```bash
@@ -57,6 +59,8 @@ minute is tolerated — see [When a rule fires](#when-a-rule-fires).
 | **Renotify** | Re-send `alert.trigger` every N minutes while still firing. `0` disables it (notify only on transitions) — see [Renotify](#renotify). |
 | **Disabled** | A disabled rule keeps its config but stops evaluating. Saving any edit — including disabling — resets the rule's status to `ok`, so it starts fresh when re-enabled. |
 
+{{< shot src="/img/alert-create.png" url="console.deploys.app/alert/create?project=acme" alt="The create-alert-rule form" caption="Pick a deployment, a metric, a threshold, and how long it must hold. Delivery is configured on notification channels, not on the rule." >}}
+
 ## Metric vocabulary
 
 | `Metric` | Meaning | Threshold unit | Bucket aggregation |
@@ -65,6 +69,8 @@ minute is tolerated — see [When a rule fires](#when-a-rule-fires).
 | `memory` | Memory usage as a share of the deployment's limit, averaged across pods | percent (same headroom as `cpu`) | avg per minute |
 | `requests` | Request rate, summed across pods | requests/min | sum per minute |
 | `egress` | Egress traffic, summed across pods | bytes/min | sum per minute |
+| `value` | A [custom metric](/deployments/custom-metrics/) gauge (kind=custom) | the series unit | avg per minute |
+| `rate` | A custom-metric counter's per-minute increase (kind=custom) | series unit / min | max−min per minute |
 
 `cpu` and `memory` are computed the same way the Metric tab's chart lines are —
 `avg(usage) / avg(limit)` per one-minute bucket — so the threshold you set lines
@@ -112,7 +118,11 @@ sample doesn't flap it straight back to firing.
 | `nodata` | Not enough recent data to evaluate — the deployment is stopped or deleted, or (for `cpu`/`memory`) has no limit set. Does not notify, and does not resolve an active `firing` alert. |
 
 The console list and detail pages show a rule's current status, last evaluated
-value, and — while firing — how long it's been firing. From the CLI:
+value, and — while firing — how long it's been firing:
+
+{{< shot src="/img/alert-detail.png" url="console.deploys.app/alert/detail?project=acme&name=api-memory-high" alt="An alert rule detail page while firing, with evaluator state and transition history" caption="The detail page shows the current value, how long the rule has been firing, and the 30-day transition history." >}}
+
+From the CLI:
 
 ```bash
 deploys alert list --project acme

content/automation/notification-channels.md

@@ -258,6 +258,38 @@ auto-detected failure.
 See [Monitoring → react to failures without polling](/deployments/monitoring/)
 for the end-to-end agent loop this enables.
 
+## Metric alerts: `alert.trigger` and `alert.resolve`
+
+A [metric alert rule](/automation/alerts/) has no delivery config of its own —
+it reuses these channels. When the rule's rolling window becomes a breach, or
+clears, the platform emits a notification (not an audit row), the same
+notify-only path as `deployment.health`:
+
+| Event | Fires when | Outcome |
+|---|---|---|
+| `alert.trigger` | the rule's window is a breach, or a renotify while it is still firing | `failure` (red in Discord) |
+| `alert.resolve` | the metric is back inside the threshold | `success` (green in Discord) |
+
+Subscribe a channel to those two names, or to `alert.*` (which also matches
+audited rule create / update / delete):
+
+```bash
+deploys notification create --project acme --name alerts-discord \
+  --type discord \
+  --url https://discord.com/api/webhooks/123/abc \
+  --event alert.trigger --event alert.resolve
+```
+
+`alert.trigger` / `alert.resolve` are written by the **system**, not a user, so
+they carry an **empty actor** (`actorType` `user`, no email). The `message` is
+a one-line summary of the condition and the value that crossed it, e.g.
+`web: cpu >= 90% for 10m (current 94.2%)`. A project with no matching channel
+still evaluates the rule and still shows `firing` in the console — it just has
+nowhere to send the page.
+
+See [Alerts](/automation/alerts/) for the metric vocabulary, window semantics,
+and how to create a rule.
+
 ## Test and the delivery log
 
 Use **Send test** (console) or `notification test` to deliver a synthetic change

content/deployments/custom-metrics.md

@@ -0,0 +1,149 @@
+---
+title: 'Custom metrics'
+linkTitle: 'Custom metrics'
+weight: 9
+description: 'Scrape your own Prometheus /metrics endpoint, chart the series, and alert on them.'
+lead: 'A metric source tells the platform to scrape a Prometheus text endpoint on one of your deployments once a minute. The series are stored for 30 days, charted in the console, and can be the target of an alert rule. Sources are project-scoped and run on Deploys.app — there is nothing extra to host.'
+---
+
+## What you get
+
+- **Own-deployment scrape** — the collector inside the location GETs
+  `http://<service>:<port><path>` on your deployment. You pick the deployment,
+  port, and path (`/metrics` by default). There is no URL field; the platform
+  builds the in-cluster address so nothing outside your project can be scraped.
+- **Gauges and counters** — Prometheus gauges, counters, and untyped series are
+  stored. Histogram and summary families (including `_bucket`) are dropped.
+- **Hard caps** — 4 sources per project, 100 series per source. Hitting the
+  series cap marks the source **truncated** instead of silently dropping extras.
+- **Charts** — the same line-chart shape as platform metrics, over 1h / 6h /
+  12h / 1d / 7d / 30d.
+- **Alerts** — a rule can target `kind=custom` with a source + exact series key
+  and metric `value` (gauge) or `rate` (counter, per-minute increase). The
+  [alert window](/automation/alerts/#when-a-rule-fires) is the same as for
+  CPU / memory / requests / egress.
+
+## Create a source
+
+From the console, open **Metric sources** and create a source. Or use the CLI:
+
+```bash
+deploys metricsource set \
+  --project acme \
+  --name web \
+  --location gke.cluster-rcf2 \
+  --deployment web \
+  --port 9090 \
+  --path /metrics
+```
+
+`set` is an upsert: the first call creates the source, later calls replace the
+config. Path defaults to `/metrics`.
+
+### Fields
+
+| Field | Description |
+|---|---|
+| **Name** | A project-unique name (lowercase, e.g. `web`). |
+| **Location** | The location the target deployment runs in. |
+| **Deployment** | Your own deployment in that location. External hosts are not allowed. |
+| **Port** | 1–65535. Must be reachable on the deployment's in-cluster Service. |
+| **Path** | Path only, leading `/`, no host, no `://`. Default `/metrics`. |
+| **Disabled** | Keep the config but skip scraping. |
+
+{{< callout type="note" >}}
+The scrape target is always in-cluster DNS derived from the deployment — never
+a free-form URL. That is what keeps this from becoming an SSRF trampoline.
+{{< /callout >}}
+
+## Series identity
+
+Each sample is stored under `name{sortedLabels}`, for example
+`queue_depth{queue="email"}`. Labels are sorted by name. A series with no
+labels is just `name`.
+
+On each scrape the collector keeps gauges, counters, and untyped values, and
+drops histogram/summary families. Counters are stored **raw** (monotonic);
+rate is computed at query / alert time as a per-minute increase, with resets
+clamped at zero.
+
+If a scrape would add an 101st series, the extra series are not stored and the
+source is marked **truncated**. The console shows a banner; nothing is dropped
+quietly.
+
+## Charts
+
+Query a source over the same short windows as cache/WAF activity (1h, 6h, 12h,
+1d, 7d, 30d):
+
+```bash
+deploys metricsource query \
+  --project acme \
+  --name web \
+  --series 'queue_depth{queue="email"}' \
+  --timerange 1h
+```
+
+Empty `--series` lets the server pick the most recently seen series (capped at
+100). Gauges average inside each bucket; counters use `max − min` per bucket
+(never negative).
+
+A deployment's Metric tab also grows a **Custom** section when a source targets
+that deployment. Scrape errors surface as `lastError` on the source.
+
+## Alert on a custom series
+
+Create an alert rule with `kind=custom` instead of a deployment metric:
+
+```bash
+deploys alert create \
+  --project acme \
+  --name email-queue-depth \
+  --kind custom \
+  --source web \
+  --series 'queue_depth{queue="email"}' \
+  --metric value \
+  --op ">=" \
+  --threshold 1000 \
+  --for 5
+```
+
+`value` compares the gauge (or untyped) sample. `rate` compares the
+per-minute increase of a counter. Delivery is still
+[`alert.trigger` / `alert.resolve`](/automation/notification-channels/#metric-alerts-alerttrigger-and-alertresolve)
+on your notification channels. Window semantics — 80% present to fire, a data
+gap does not resolve a firing rule — are the same as [platform-metric
+alerts](/automation/alerts/#when-a-rule-fires).
+
+A series that has not been scraped yet evaluates as `nodata`. You can create
+the rule before the first scrape.
+
+## Limits
+
+| Limit | Value |
+|---|---|
+| Sources per project | 4 |
+| Series per source | 100 (further series mark the source truncated) |
+| Path length | 256 |
+| Scrape timeout | 5 seconds |
+| Scrape body | 1 MiB |
+| Retention | 30 days |
+
+Custom metrics at these caps are included; there is no per-series SKU in v1.
+
+## Using the API directly
+
+`metricSource.set`, `metricSource.get`, `metricSource.list`,
+`metricSource.delete`, `metricSource.series`, and `metricSource.query`. Also
+on the [MCP server](/automation/mcp/).
+
+## Permissions
+
+| Action | Permission |
+|---|---|
+| Create / edit | `metricSource.set` |
+| View / list / series / query | `metricSource.get` / `metricSource.list` |
+| Delete | `metricSource.delete` |
+
+Nothing in the payload is secret, so `metricSource.get` / `metricSource.list`
+are grantable to public principals like most read permissions.

content/deployments/monitoring.md

@@ -14,7 +14,8 @@ gap between them tells you whether you're under- or over-provisioned. The
 Replicas chart is available replica count (HPA scale, crash-loop drop); it is
 hidden for Static and CronJob deployments, which have no k8s Deployment.
 The time-range selector spans **1 hour aggregate**, **1 day**, **7 days**, and
-**30 days**.
+**30 days**. To chart series from your own Prometheus `/metrics` endpoint, add
+a [custom metric source](/deployments/custom-metrics/).
 
 {{< shot src="/img/deployment-metrics.png" url="console.deploys.app/deployment/metrics?project=acme&location=gke.cluster-rcf2&name=web" alt="Live CPU, memory, request rate, and egress charts for the web deployment" caption="Solid lines are real usage; dashed lines are the request you've allocated." >}}
 

scripts/screenshots/README.md

@@ -33,9 +33,12 @@ Prereqs:
 
 - The console repo is cloned alongside this one: `~/Projects/deploys-app/console`
   (or wherever your deploys-app workspace lives — the script resolves it as
-  `../console`).
-- The console repo's working tree on `src/lib/server/mock.js` is clean (the
-  script refuses to run otherwise so it never clobbers your own edits).
+  `../console`). From a docs git worktree set `CONSOLE_DIR` to the console
+  checkout that has `node_modules` (e.g. `.worktrees/console-custom-metrics`).
+- The console repo's working tree on `src/lib/server/mock.ts` (or `mock.js`)
+  is clean (the script refuses to run otherwise so it never clobbers your own
+  edits). If `mock-enrichment.patch` does not apply, capture still runs against
+  the stock mock fixtures.
 - `bun` is on PATH for `bun dev:mock`.
 - Playwright Chromium is installed in the console repo (`bunx playwright install`
   if not — the console already ships `@playwright/test`).

scripts/screenshots/capture.mjs

@@ -52,7 +52,10 @@ const screens = [
 	['cache-manage',          `/cache/manage?${P}&${LOC}`, { h: 1150 }],
 	['cache-metrics',         `/cache/metrics?${P}&${LOC}`, { h: 1150 }],
 	['scheduler-list',        `/scheduler?${P}`],
-	['scheduler-detail',      `/scheduler/detail?${P}&name=daily-health-check`, { h: 1150 }]
+	['scheduler-detail',      `/scheduler/detail?${P}&name=daily-health-check`, { h: 1150 }],
+	['alert-list',            `/alert?${P}`],
+	['alert-create',          `/alert/create?${P}`, { h: 1150 }],
+	['alert-detail',          `/alert/detail?${P}&name=api-memory-high`, { h: 1150 }]
 ]
 
 const browser = await chromium.launch()

scripts/screenshots/refresh.sh

@@ -14,30 +14,46 @@ set -euo pipefail
 
 here=$(cd "$(dirname "$0")" && pwd)
 docs=$(cd "$here/../.." && pwd)
-console=$(cd "$docs/../console" && pwd)
+# CONSOLE_DIR overrides the sibling-checkout default so a docs worktree can
+# point at a console worktree (../../console from .worktrees/docs-* is wrong).
+if [[ -n "${CONSOLE_DIR:-}" ]]; then
+	console=$(cd "$CONSOLE_DIR" && pwd)
+else
+	console=$(cd "$docs/../console" && pwd)
+fi
 patch="$here/mock-enrichment.patch"
+mock_file=""
+for candidate in src/lib/server/mock.ts src/lib/server/mock.js; do
+	if [[ -f "$console/$candidate" ]]; then
+		mock_file=$candidate
+		break
+	fi
+done
 
 [[ -d "$console" ]] || { echo "console repo not found at $console" >&2; exit 1; }
+[[ -n "$mock_file" ]] || { echo "console mock fixture not found (tried mock.ts / mock.js)" >&2; exit 1; }
 [[ -f "$patch" ]] || { echo "patch not found: $patch" >&2; exit 1; }
 
 cd "$console"
 
-# refuse to run if mock.js has uncommitted changes — we'd revert them
-if ! git diff --quiet -- src/lib/server/mock.js; then
-	echo "src/lib/server/mock.js has uncommitted changes — commit or stash first." >&2
+# refuse to run if the mock fixture has uncommitted changes — we'd revert them
+if ! git diff --quiet -- "$mock_file"; then
+	echo "$mock_file has uncommitted changes — commit or stash first." >&2
 	exit 1
 fi
 
 runner=""
 cleanup () {
 	[[ -n "${mock_pid:-}" ]] && kill "$mock_pid" 2>/dev/null || true
 	[[ -n "$runner" ]] && rm -f "$runner" || true
-	git checkout -- src/lib/server/mock.js 2>/dev/null || true
+	git checkout -- "$mock_file" 2>/dev/null || true
 }
 trap cleanup EXIT
 
-echo "==> applying mock-enrichment.patch"
-git apply "$patch"
+echo "==> applying mock-enrichment.patch (against $mock_file)"
+if ! git apply "$patch"; then
+	echo "warning: mock-enrichment.patch did not apply; capturing against stock mock fixtures" >&2
+fi
 
 echo "==> starting bun dev:mock"
 MOCK_API=1 bun run dev --port 5173 > /tmp/console-mock.log 2>&1 &