Cloud Action Costs Dashboard

[lainecsmith]

What was changed

Adds a new community Grafana dashboard for Temporal Cloud, sourced entirely from the OpenMetrics endpoint, plus a companion alert rules file:

temporal_cloud_action_costs.json — dashboard with two sections:

• Summary: 1-day / 7-day / month-to-date Action totals.
• Action Spike Detection: current Action rate vs. the same time 24h and 7d ago, percent-change indicators, a breakdown by Workflow Type and Action Type, and a ranked table of top Workflow Types by Action volume.

temporal_cloud_action_costs_alerts.yaml — optional Grafana unified-alerting provisioning file with 2 rules (action rate spike vs 24h ago, vs 7d ago).

Scope is intentionally limited to Action volume — Temporal Cloud doesn't expose price or credit-ledger balance via metrics, so this dashboard doesn't show cost in dollars or track spend against a commitment. That's called out explicitly in the dashboard's intro panel, with a pointer to Billing Center / the Billing API for anyone who needs authoritative $ figures.

Why?

Two of the most common cost-support conversations with customers are (1) someone ships a change or a new workflow that quietly drives a lot more Actions than expected, and nobody notices until the bill, and (2) there's no proactive signal — you only find out by going and looking. This gives customers a self-serve starting point for both: a dashboard to see it, and alert rules so it can page/notify instead of requiring someone to be staring at Grafana when it happens.

Checklist

How was this tested:

• Created a Temporal Cloud Service Account with the Metrics Read-Only role and generated an API key; verified the OpenMetrics endpoint directly with curl before touching Grafana.
• Ran Prometheus in a homelab Docker Compose stack, added a scrape job against metrics.temporal.io with Bearer auth, and confirmed the temporal-cloud job showed UP in Prometheus's own Targets page.
• Imported the dashboard into a homelab Grafana instance against that Prometheus data source; confirmed the temporal_namespace variable populates from label_values(), and confirmed (separately, via a direct endpoint query) that namespaces without recent Action traffic simply don't appear — expected behavior given OpenMetrics' sparse reporting, not a bug — and added a note about this to the dashboard's intro panel so it's not mistaken for one.
• Verified the account has 85 namespaces total but only 2 have recent traffic; confirmed the dashboard's namespace variable correctly surfaces those 2 and that switching to "All" sums correctly against real data.
• Sanity-checked the Actions MTD figure against the Cloud UI's own Usage Dashboard for the same namespace/timeframe — same order of magnitude (exact match not expected; different counting methodology, called out in the docs).

Any docs updates needed?
Updated cloud/README.md in this PR with a link to the new dashboard the alerts.
No docs.temporal.io changes made as part of this PR.

[prathyushpv]

does this file temporal_cloud_alert_rules.yaml exist?

[lainecsmith]

It does, except I renamed it and forgot to update here! Fixed, thanks!

[prathyushpv]

Should it be Estimated Actions?

[lainecsmith]

Sure, changed.

[Copilot]

Pull request overview

Adds a new community Grafana dashboard (plus optional unified-alerting provisioning) to help Temporal Cloud users monitor Action volume month-to-date and detect sudden Action-rate spikes using the OpenMetrics endpoint.

Changes:

• Added a new Grafana dashboard for Action MTD totals and spike-detection breakdowns.
• Added Grafana unified-alerting provisioning rules to alert on action-rate spikes vs 24h/7d baselines.
• Updated cloud README to link the new dashboard and sample alerts.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 7 comments.

File	Description
cloud/temporal_cloud_action_costs.json	New Grafana dashboard for Action MTD totals, spike comparisons vs 24h/7d ago, and workflow/action-type breakdowns.
cloud/temporal_cloud_action_costs_alerts.yaml	New Grafana unified-alerting provisioning file with two spike-detection alert rules.
cloud/README.md	Adds links to the new dashboard and sample alert rules.

Suppressed comments (3)

cloud/temporal_cloud_action_costs.json:206

• This panel description says "billable" actions, but the query uses temporal_cloud_v1_total_action_count (not ..._billable_action_count). Either switch the query to the billable metric or adjust the description to avoid implying it is billable.

      "description": "Estimated total billable actions over the last 7 days",

cloud/temporal_cloud_action_costs.json:614

• This percent-change expression can divide by zero when the offset value is 0 (or missing), producing +Inf/NaN in the Stat panel and making the paired alerting behavior noisy. Clamp the denominator to a tiny positive value to keep the expression finite.

          "expr": "(sum(temporal_cloud_v1_total_action_count{temporal_namespace=~\"$temporal_namespace\"}) - sum(temporal_cloud_v1_total_action_count{temporal_namespace=~\"$temporal_namespace\"} offset 7d)) / sum(temporal_cloud_v1_total_action_count{temporal_namespace=~\"$temporal_namespace\"} offset 7d) * 100",

cloud/temporal_cloud_action_costs_alerts.yaml:115

• This alert expression can divide by zero when the offset value is 0 (or missing), producing +Inf/NaN and making the alert noisy. Clamp the denominator to a tiny positive value to keep the evaluation finite.

                (sum(temporal_cloud_v1_total_action_count{temporal_namespace=~".*"})
                - sum(temporal_cloud_v1_total_action_count{temporal_namespace=~".*"} offset 7d))
                / sum(temporal_cloud_v1_total_action_count{temporal_namespace=~".*"} offset 7d) * 100

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[prathyushpv]

This is from an AI review: avg_over_time divides by samples present, not minutes elapsed, and Cloud omits datapoints for idle minutes — so totals over-report by 1/duty_cycle, always upward, worst on the least-active namespaces.

Measured against known truth: a namespace reporting 1h/day reads 8,640,000 vs a true 360,000 (24x); account total +53%.

The sibling's sum_over_time(m[1d])/1440*86400 is exact at 60s scrape but double-counts at 30s (Temporal's recommended interval), so don't copy that either. This is correct at both:

sum(sum_over_time(temporal_cloud_v1_total_action_count{temporal_namespace=~"$temporal_namespace"}[1d:1m])) * 60

Same fix on lines 261, 331, 962.

[prathyushpv]

This is from an AI agent: Same avg_over_time bug, but here it inverts the ranking — a workflow type reporting 1h/day is scored as if it ran all week. On my data the true #4 (2,520,000) displays as #1 (60,480,000), above the real leader (54,432,000).

topk(20, sum by (temporal_workflow_type, temporal_namespace) (
  sum_over_time(temporal_cloud_v1_billable_action_count{temporal_namespace=~"$temporal_namespace"}[$__range:1m])
) * 60)

Also: the footer sums only the top-20 rows but reads as a total.

[prathyushpv]

This is from an ai agent: clamp_min doesn't fix this — the denominator is absent, not zero, and clamp_min can't create a missing sample. Verified: on a namespace with no data 24h ago it still returns an empty vector, so the tile is blank.

(Scoped to All it's fine — the account denominator is present. This bites once the variable is narrowed to the new namespace.)

or vector(0) works (returns 5,000 on the same data):

(sum(M) - (sum(M offset 1d) or vector(0)))
  / clamp_min(sum(M offset 1d) or vector(0), 1) * 100
# M = temporal_cloud_v1_total_action_count{temporal_namespace=~"$temporal_namespace"}

Also 1e-9 as a floor yields 5e13% where it does apply; 1 gives 50,000%.

[prathyushpv]

This is from an AI agent: clamp_min from line 540 wasn't applied here, so the 24h and 7d panels now differ — same for the two alert rules (alerts.yaml:64 vs :115). Whatever form you pick should land on all four.

[prathyushpv]

This is from an AI agent: Default range is now-7d and this panel uses $__range, so "Actions Spent This Month (MTD)" and "7 Day Actions" render the same number. Suggest "time": {"from": "now/M", "to": "now"}.

Related: the intro text and this panel's description say to use "This month", but in Grafana that's now/M -> now/M — to rounds up to month end, so $__range_s is a full 31 days regardless of the date (~3x inflation mid-month). You want "This month so far".

[prathyushpv]

DS_PROMETHEUS is declared here. But I don't see any references anywhere. Is it needed?

[prathyushpv]

From an AI agent: Bare sum() with no by (temporal_namespace) collapses the account to one number (=~".*" is a no-op matcher). NS-A steady at 10,000/sec plus NS-B jumping 10 -> 500/sec (~42M extra actions/day) = 4.9% account-wide, under the 50 threshold. Per-namespace that's 4900%.

by (temporal_namespace) gives one instance per namespace. Two caveats: or vector(0) won't survive it (no namespace label — leaves a stray {} = 0), and a brand-new namespace drops out of the result entirely, so cold start needs its own rule:

sum by (temporal_namespace) (temporal_cloud_v1_total_action_count) > 5
unless
sum by (temporal_namespace) (temporal_cloud_v1_total_action_count offset 1d)

That will also fire for an irregular namespace idle at that exact minute yesterday, so a windowed baseline may be better. See :540 re the clamp_min here.

[prathyushpv]

From AI agent: clamp_min wasn't applied to the 7d rule, so it now differs from the 24h one. Same for the grouping point above.

[prathyushpv]

This is from an AI agent: {{ $values.A }} renders unrounded — a live rule gave me "Action rate is 257.14285714285717% higher...". A is an instant query so format it directly, but guard it (printf on a missing value renders %!f(<nil>)):

summary: '{{ if $values.A }}Action rate is {{ printf "%.1f" $values.A.Value }}% above{{ else }}Action rate change unknown vs{{ end }} the same time yesterday.'

.Value is required — $values.A is a struct. Add {{ $labels.temporal_namespace }} once grouping is in.

Minor: the description says "moved more than 50% away from", but gt is one-sided — a collapse never fires.

[prathyushpv]

This cites three thresholds — 50%, 75%, 80% — but only 50 and 75 exist in the file. Looks like a leftover from a removed third rule.