Skip to content

Add metricSource contract (metric alerts phase 2) - #140

Open
acoshift wants to merge 4 commits into
mainfrom
feat/custom-metrics
Open

Add metricSource contract (metric alerts phase 2)#140
acoshift wants to merge 4 commits into
mainfrom
feat/custom-metrics

Conversation

@acoshift

@acoshift acoshift commented Sep 2, 2026

Copy link
Copy Markdown
Member

Phase 2 of metric alerts: the API contract for scraping a project's own Prometheus /metrics endpoint, charting the series, and alerting on them.

metricSource resource

Project-scoped scrape sources addressed by (project, name), with location in config (same shape as alert).

  • RPCs: metricSource.set / .get / .list / .delete / .series / .query
  • Set is a full upsert (cache.set style): first Set of a name creates, subsequent Sets of the same name replace config. Max 4 sources per project is server-enforced when creating a new name (MetricSourceMaxPerProject); Valid() does not count existing sources.
  • Scrape target is (deployment, port, path) only — no URL field. Path defaults to /metrics, must start with /, and must not contain :// or a host. That is the v1 SSRF bound: apiserver resolves http://<kubeName>.<ns>:port/path for the collector.
  • metricSource.query returns []*DeploymentMetricsLine so Chart.svelte is unchanged. Time range is the waf/cache short windows (1h|6h|12h|1d|7d|30d), not deployment 1hagg.
  • Caps (shared with apiserver/collector): 4 sources/project, 100 series/source, path ≤ 256, scrape timeout 5s, body ≤ 1 MiB.

Permissions: metricSource.* / .set / .get / .list / .delete. Series/query are gated by metricSource.get. Reads are public-bindable (nothing secret).

AlertTarget.Kind=custom

Additive on the existing alert contract:

  • Kind empty or "deployment": existing platform metrics (cpu|memory|requests|egress); Location+Deployment required; Source/Series must be empty.
  • Kind="custom": Source (metricSource name) + Series (exact name{sortedLabels} key); Location/Deployment must be empty (location lives on the source). Condition metric is value (gauge) or rate (counter per-minute).

Existing deployment creates with empty Kind still validate.

Collector RPCs (internal)

  • collector.listMetricSources {location} → enabled sources with platform-resolved URL. The request type has no URL field.
  • collector.setCustomUsage {location, list: [{projectID, sourceID, series, value, at}]} — list may be empty.

Merge first

Dependents (apiserver, collector, console, mcp, CLI, docs) should wait to re-pin until this PR is merged; do not pin the branch SHA.

Phase 2: scrape sources (own deployment + port + path, no URL field)
and AlertTarget.Kind=custom (value/rate). Collector listMetricSources /
setCustomUsage types for the in-cluster scrape loop.
The collector caps samples at 100 before ingest, so apiserver cannot
infer overflow from list length. Truncated/LastError ride the same
RPC so the source row can show a banner and scrape errors.
A failed scrape has no samples, so LastError needs a source id on the
request itself.
Without Type on the ingest item, apiserver stored every series as
untyped and never took the counter increase path on live scrapes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant