feat: change to generic webhook

[dewabisma]

Okay, found the bug!

The slack webhook always considered error that is why it keep firing.

[n13]

Reviewed against the pinned stack: Grafana 11.3.0 (grafana/Dockerfile:1) → grafana/alerting @22cee00b280e, and Rocket.Chat processWebhookMessage.ts. The diagnosis in the PR comment is right — Grafana's Slack notifier unmarshals Rocket's {"success":true} looking for an ok field, gets OK=false, and fails every send. But the fix trades a noisy-but-working delivery path for a silent one that delivers nothing.

Blocking

1. Silent alert blackhole — the generic webhook payload is unreadable by a Rocket.Chat incoming webhook (contactpoints.rocket.fragment.yml:7)
Grafana 11.3's webhook notifier POSTs its fixed envelope (receiver/status/alerts/…/title/state/message) with no text, msg, or attachments key — and 11.3 has no custom-payload templating (that landed in 12.x). Rocket.Chat's script-less incoming webhook builds msg: trim(text || msg || '') → '', posts a blank message, and returns 200 {"success":true}. Grafana's webhook notifier judges success on HTTP 2xx only, so it records the notification as delivered — and with repeat_interval: 8736h (policies.production.yml:10) the alert is not re-sent for a year. Since "Rocket Notifications" is the default receiver and the terminal route for every non-critical alert with no continue: true fan-out, the entire warning tier goes dark with green health on both sides. The old type: slack payload ({channel, text, username, attachments…}) is exactly what Rocket renders natively — delivery worked; only response parsing was broken. "The retries stopped" is also what silent non-delivery looks like, so please verify with an actual message in the channel, not the absence of errors.
Fixes that keep delivery working: (a) enable Script Enabled on the Rocket.Chat incoming webhook with a script mapping title/message → text, and document that requirement in README/env.example; (b) upgrade to Grafana ≥12 and use the webhook custom payload to emit {"text": …}; or (c) keep type: slack and accept the retry noise until either is done.

2. Alert name and firing/resolved state now exist only in title, a field Rocket.Chat never reads (contactpoints.rocket.fragment.yml:12)
The old per-alert header 🚨 *{{ .Labels.alertname }}* — {{ .Status | toUpper }} was deleted from message, so even in the best future case (a Rocket-side script mapping message → text), notifications carry no alert name, and — with disableResolveMessage: false — a resolve renders byte-identical to the fire. Rocket.Chat's accepted payload keys (text/msg/attachments/alias/emoji/avatar/channel…) include no title. Put the alertname + status line back inside the range (or have the script concatenate title + message).

3. The secret-bearing webhook URL silently downgrades from encrypted to plaintext (contactpoints.rocket.fragment.yml:9)
For type: slack, url is a secure setting (encrypted into secure_settings); for type: webhook it is a plain setting. Same uid, same value — but after this change the full https://…/hooks/<id>/<token> sits cleartext in the alert_configuration table, in DB backups, and unredacted in GET /api/v1/provisioning/contact-points and the contact-point edit UI. Worth an explicit decision (and ideally token rotation if the DB has broader readership).

Non-blocking

4. username: Grafana Monitor drop loses the sender alias — and can't be re-added under this type (contactpoints.rocket.fragment.yml:8). Under slack it became the displayed sender (Rocket maps payload username → alias). The webhook notifier's username setting is the HTTP Basic Auth user (a silent no-op alone; adds an Authorization: Basic header if a password is ever added). Dropping it was correct, but the identity loss is undocumented and the field name is a trap for the next editor.

5. {{ .CommonLabels.alertname }} silently depends on alertname staying in every Rocket route's group_by (contactpoints.rocket.fragment.yml:12). Safe today (all Rocket routes inherit ["alertname","grafana_folder","instance"]), but Grafana templates run with missingkey=zero — if a future route drops alertname from group_by, the title degrades to 🚨 — FIRING with no error anywhere. Per-alert .Labels.alertname in the body (finding 2) removes the coupling.

6. The Rocket and Telegram message templates were byte-identical before this PR; now they're two divergent hand-synced copies (contactpoints.rocket.fragment.yml:13 vs contactpoints.telegram.fragment.yml). The ✅-on-resolved improvement stops at Rocket, and every future field addition is a two-file edit. Grafana 11.3 supports provisioned shared templates (templates: + {{ define }}/{{ template }}); a templates.yml dropped in provisioning/alerting/ is picked up with no entrypoint/Dockerfile changes.

7. Nits (contactpoints.rocket.fragment.yml:11): title: | leaves a literal \n in the payload title — use |-; six of the {{- left-trims are no-ops (the preceding -}} already trimmed); each alert block emits a trailing blank line; httpMethod: POST restates the default. Rendering was simulated for all field combinations — nothing actually glues together, so these are polish only.

Rollout / repo notes (pre-existing, but they decide whether this PR works)

• docker compose restart grafana will NOT apply this change. The fragments are baked into the image (grafana/Dockerfile:20-26) and nothing bind-mounts them; README.md:360-363 instructs restart after alert-config changes. Deploy needs docker compose up -d --build grafana — otherwise the retry spam "mysteriously" continues.
• Testing trap: with only ROCKET_WEBHOOK_URL set, docker-entrypoint.sh:31 installs policies.local.yml (email-only, Rocket receiver unreferenced) — all three of TELEGRAM_BOT_TOKEN/TELEGRAM_CHAT_ID/ROCKET_WEBHOOK_URL must be set to exercise this path.
• README contradicts the live routing ("Warning Alerts → Email only", README.md:220) and never mentions Rocket at all — which would mask the blackhole in finding 1 ("Rocket isn't supposed to get warnings anyway"). .env.example (which README.md:123 points at) lacks ROCKET_WEBHOOK_URL entirely; env.example has it.
• Prettier hazard: npm run format globs grafana/**/*.yml and would de-indent this fragment's load-bearing 2-space root indentation, breaking the assembled contactpoints.yml for ALL contact points — it has happened before (commit 081543f "fix: contactpoints identation"). Consider adding the fragments to .prettierignore.
• envsubst trap: the fragment passes through unrestricted envsubst (docker-entrypoint.sh:23), so any future $labels-style Go-template idiom (as used throughout rules.yml) would be silently blanked at container start. Consider envsubst '${ROCKET_WEBHOOK_URL}' to whitelist the one variable.

---

Verdict: Request changes — as merged, this stops the retry noise by making delivery silently fail: Rocket.Chat cannot render Grafana's generic webhook envelope, and Grafana will record every blank post as a successful delivery. Pair the type switch with a Rocket-side transform script (or a Grafana ≥12 custom payload), put alertname/status back in the message body, and confirm an actual rendered message in the channel before merging.