feat(ops): GitOps pipeline, incident management, cost optimization, backup verification

[knightqueen-sketch]

Warning

Not tested — implementation only. None of these changes have been built, run or tested (no cargo build/cargo test, no kustomize build, no script or workflow runs). Please review and test before merging.

Closes #1066
Closes #1067
Closes #1068
Closes #1069

#1069 — GitOps deployment pipeline (Argo CD)

• deploy/k8s/base (Deployment, Service, HPA, PDB) + overlays/staging and overlays/production (Kustomize)
• deploy/argocd/: AppProject (allow-list, production sync window), Applications with automated sync (prune, selfHeal), argocd-cm build options
• .github/workflows/gitops-promote.yml: merge to main builds the image and bumps staging; workflow_dispatch opens a PR that bumps production, and merging it deploys
• .github/workflows/gitops-validate.yml: renders and kubeconform-validates overlays, then posts the rendered diff on the PR
• scripts/gitops-rollback.sh: rollback via git revert of the promotion commit
• docs/gitops.md

#1068 — Incident management integration

• api-server/src/incidents.rs: PagerDuty (Events API v2) / Opsgenie integration (INCIDENT_PROVIDER), incident store with dedup, timeline, acknowledge/resolve (propagated to the provider and to the in-process alert pipeline to stop escalation), postmortems, MTTA/MTTR stats and metrics
• Automatic incident creation: critical in-process alert notifications (hooked in commitment_monitoring) and an Alertmanager webhook receiver (POST /v1/admin/incidents/alertmanager)
• Routes under /v1/admin/incidents wired into both routers in main.rs
• alertmanager.yml: incident-tracker webhook route and opsgenie-oncall receiver
• docs/incident-response.md, docs/postmortem-template.md

#1067 — Cost optimization automation

• HPA for api-server (CPU/memory, conservative scale-down) + PDB; per-environment bounds in overlays
• scripts/ops/cost-optimize.sh: batched orphaned/expired data cleanup, zstd compression of rotated audit logs, lz4 for old partitions, S3 lifecycle tiering, savings ledger + Pushgateway metrics, Markdown cost report
• Nightly CronJob (deploy/k8s/jobs) and weekly .github/workflows/cost-report.yml (adds AWS Cost Explorer spend)
• docs/cost-optimization.md

#1066 — Automated backup verification

• scripts/ops/backup-verify.sh: freshness (RPO) checks, SHA256 + pg_verifybackup integrity, restore + WAL replay into a disposable Postgres, schema/row/amcheck validation, measured recovery time vs RTO, metrics, and a direct Alertmanager alert on failure
• Daily in-cluster CronJob and weekly off-cluster .github/workflows/backup-verification.yml (opens an issue on failure)
• monitoring/prometheus/operations-rules.yml: backup, cost/autoscaling and Argo CD alerts
• docs/backup-strategy.md; cross-links added to docs/disaster-recovery.md, monitoring/README.md, CHANGELOG.md

Required setup (out of band)

Secrets: KUBE_CONFIG is not needed (Argo CD pulls). Needed: PAGERDUTY_ROUTING_KEY / OPSGENIE_API_KEY, BACKUP_BUCKET, BACKUP_VERIFY_ROLE_ARN, COST_REPORT_ROLE_ARN, COST_LEDGER_URI, PUSHGATEWAY_URL, ALERTMANAGER_URL; an ops-toolbox image with aws-cli, psql, jq, zstd and docker-cli.

🤖 Generated with Claude Code

[drips-wave]

@knightqueen-sketch Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits