Metrics and observability: no Prometheus/OpenTelemetry export

[jason-shen]

Problem

The only things a running server exposes today are /health (a literal ok, see #30) and the per-turn timing events pushed down the DataChannel to the connected client:

// internal/pipeline/types.go
type timingMsg struct {
	Type  string `json:"type"`
	Stage string `json:"stage"`
	Ms    int64  `json:"ms"`
}

Those numbers are real and already measured at the right places — internal/pipeline/agent.go stamps time.Since(turnStart) at endpointing, first LLM token, and first TTS audio. But they only ever reach the browser that caused them. Nothing aggregates them, nothing survives the call, and there is no way to answer "what is p95 time-to-first-audio across the last hour" without asking a user to open devtools.

internal/procstat samples process CPU into an atomic and is read by nothing but a future health handler. internal/session.Manager knows the live session count. Both are one line away from being a gauge.

For a latency-sensitive media server this is the gap that keeps it out of production: you cannot alert on it, you cannot capacity-plan with it, and #56 (load testing) has nowhere to write its results.

Proposed change

Add a Prometheus scrape endpoint on a separate internal listener, not on the public mux. POST /whip is internet-facing and rate-limited per IP; metrics should not be. A second http.Server bound to metrics.bind (default 127.0.0.1:9090, disabled when empty) keeps the public surface unchanged and makes the sidecar/Cloud Run story a config choice rather than a fork.

Use prometheus/client_golang. It is the only new direct dependency and OpenTelemetry can scrape a Prometheus endpoint, so this does not foreclose OTel later.

Metrics worth having on day one, all labelled sparingly (never by session_id — that is unbounded cardinality and will kill the scrape):

Metric	Type	Labels
streamcore_sessions_active	gauge	—
streamcore_sessions_total	counter	outcome = ended / reaped / panicked
streamcore_session_duration_seconds	histogram	—
streamcore_turn_latency_seconds	histogram	stage = endpoint / llm_first_token / tts_first_audio
streamcore_barge_ins_total	counter	—
streamcore_provider_requests_total	counter	kind = stt/llm/tts, provider, outcome
streamcore_provider_latency_seconds	histogram	kind, provider
streamcore_whip_requests_total	counter	result = ok / rate_limited / capped / unauthorized
streamcore_process_cpu_percent	gauge	— (from procstat)

Feed the turn histograms from the same call sites that already build timingMsg, so the DataChannel event and the metric cannot drift apart. Put the recording behind a small interface in a new internal/metrics package with a no-op implementation, so the pipeline does not take a hard dependency on Prometheus and tests stay quiet.

Bucket boundaries matter more than usual here: the interesting range for time-to-first-audio is 200 ms to 3 s, and the default prometheus.DefBuckets wastes most of its resolution below 100 ms. Pick explicit buckets.

Also ship the Grafana dashboard JSON under infrastructure/. A metrics endpoint nobody has a dashboard for gets scraped and ignored.

Out of scope

Tracing. Per-turn spans across STT → LLM → TTS are worth having, but they are a different dependency, a different sampling story, and a different issue. Metrics first.

Acceptance criteria

• [metrics] bind = "127.0.0.1:9090" in config.toml.example, documented in docs/configuration.md, and off when unset.
• GET /metrics on that listener returns a valid Prometheus exposition; promtool check metrics is clean.
• No metric carries session_id or any other unbounded label.
• Turn latency histograms are recorded at the same call sites that emit timingMsg.
• The public mux does not serve /metrics.
• Test asserting the counters move for a session that starts, takes one turn, and ends.
• Grafana dashboard JSON committed under infrastructure/.

Pointers

• internal/pipeline/agent.go — timing.EndpointMs ~L139, timing events ~L218 and ~L281
• internal/pipeline/types.go — timingMsg ~L42
• internal/session/manager.go — live and resumable session maps
• internal/procstat/procstat.go — CPUPercent(), already sampled
• main.go — where the second listener goes, alongside the existing mux ~L80
• Related: Add /version endpoint and richer /health with build info #30 (/version and richer /health)