CLO-188 Add events and more metrics to orchestratord

[Alphadelta14]

Motivation

There is a desire to be able to easier troubleshoot upgrades going sour or even progress of environment rollouts.
We can't create dashboards until we have appropriate metrics and events.

Description

This improves the observability experience associated with orchestratord.
We generally want events for failed reconciliations because those are super actionable.
We also need some metrics to be able monitor orchestratord.

Verification

self-managed deployment:

diff --git a/aws/modules/operator/main.tf b/aws/modules/operator/main.tf
index 26e958c..2c9729c 100644
--- a/aws/modules/operator/main.tf
+++ b/aws/modules/operator/main.tf
@@ -36,8 +36,8 @@ locals {
         enableLicenseKeyChecks = var.enable_license_key_checks
         installV1CRD           = var.install_v1_crd
       }
-      image = var.orchestratord_version == null ? {} : {
-        tag = var.orchestratord_version
+      image = {
+        tag = "v26.40.0-dev.0--pr.ge0cb06ca4fc037dee47f976d607b3bd2ff3750df"
       },
       cloudProvider = {
         type   = "aws"

manual rbac (not upgrading my chart)

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: heather-mzmon-materialize-operator-events
  labels:
    app.kubernetes.io/name: materialize-operator
    # Applied by hand to test PR 38529 ahead of the chart change that adds this
    # rule to the operator's own ClusterRole. Safe to delete once the chart
    # carrying it is installed.
    materialize.cloud/manual-grant: "pr-38529"
rules:
# The controllers publish a warning event on the Materialize, Balancer, or
# Console resource whose reconciliation failed. Patch is for the aggregation of
# a repeated failure into one event with a count.
- apiGroups: ["events.k8s.io"]
  resources:
  - events
  verbs:
  - create
  - patch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: heather-mzmon-materialize-operator-events
  labels:
    app.kubernetes.io/name: materialize-operator
    materialize.cloud/manual-grant: "pr-38529"
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: heather-mzmon-materialize-operator-events
subjects:
- kind: ServiceAccount
  name: orchestratord
  namespace: materialize

Non-leader doesn't have new metrics.
Leader however has a new set of metrics:

metric results

curl http://127.0.0.1:52682/metrics # HELP environmentd_needs_update Count of organizations in this cluster which are running outdated pod templates. Only the operator replica holding the leadership lease reconciles, so the others report zero. # TYPE environmentd_needs_update gauge environmentd_needs_update 0 # HELP orchestratord_is_leader Whether this operator replica holds the controller leadership lease, and is therefore the replica reconciling. Summed across the replicas this should be 1. A sustained 0 means no replica can take the lease, for instance because the service account lacks permission on leases, and the operator is reconciling nothing. # TYPE orchestratord_is_leader gauge orchestratord_is_leader 1 # HELP orchestratord_reconciliation_duration_seconds Time spent in one reconciliation pass. This covers only the reconciler itself, not the finalizer bookkeeping around it, and a pass that waits on a rollout returns promptly rather than blocking, so this measures work done and not the wall-clock length of a rollout. # TYPE orchestratord_reconciliation_duration_seconds histogram orchestratord_reconciliation_duration_seconds_bucket{controller="balancer",event_type="apply",le="0.001"} 0 orchestratord_reconciliation_duration_seconds_bucket{controller="balancer",event_type="apply",le="0.002"} 0 orchestratord_reconciliation_duration_seconds_bucket{controller="balancer",event_type="apply",le="0.004"} 0 orchestratord_reconciliation_duration_seconds_bucket{controller="balancer",event_type="apply",le="0.008"} 0 orchestratord_reconciliation_duration_seconds_bucket{controller="balancer",event_type="apply",le="0.016"} 0 orchestratord_reconciliation_duration_seconds_bucket{controller="balancer",event_type="apply",le="0.032"} 0 orchestratord_reconciliation_duration_seconds_bucket{controller="balancer",event_type="apply",le="0.064"} 8 orchestratord_reconciliation_duration_seconds_bucket{controller="balancer",event_type="apply",le="0.128"} 12 orchestratord_reconciliation_duration_seconds_bucket{controller="balancer",event_type="apply",le="0.256"} 12 orchestratord_reconciliation_duration_seconds_bucket{controller="balancer",event_type="apply",le="0.512"} 12 orchestratord_reconciliation_duration_seconds_bucket{controller="balancer",event_type="apply",le="1"} 12 orchestratord_reconciliation_duration_seconds_bucket{controller="balancer",event_type="apply",le="2"} 12 orchestratord_reconciliation_duration_seconds_bucket{controller="balancer",event_type="apply",le="4"} 12 orchestratord_reconciliation_duration_seconds_bucket{controller="balancer",event_type="apply",le="8"} 12 orchestratord_reconciliation_duration_seconds_bucket{controller="balancer",event_type="apply",le="16"} 12 orchestratord_reconciliation_duration_seconds_bucket{controller="balancer",event_type="apply",le="32"} 12 orchestratord_reconciliation_duration_seconds_bucket{controller="balancer",event_type="apply",le="64"} 12 orchestratord_reconciliation_duration_seconds_bucket{controller="balancer",event_type="apply",le="128"} 12 orchestratord_reconciliation_duration_seconds_bucket{controller="balancer",event_type="apply",le="256"} 12 orchestratord_reconciliation_duration_seconds_bucket{controller="balancer",event_type="apply",le="512"} 12 orchestratord_reconciliation_duration_seconds_bucket{controller="balancer",event_type="apply",le="+Inf"} 12 orchestratord_reconciliation_duration_seconds_sum{controller="balancer",event_type="apply"} 0.700450801 orchestratord_reconciliation_duration_seconds_count{controller="balancer",event_type="apply"} 12 orchestratord_reconciliation_duration_seconds_bucket{controller="console",event_type="apply",le="0.001"} 0 orchestratord_reconciliation_duration_seconds_bucket{controller="console",event_type="apply",le="0.002"} 0 orchestratord_reconciliation_duration_seconds_bucket{controller="console",event_type="apply",le="0.004"} 0 orchestratord_reconciliation_duration_seconds_bucket{controller="console",event_type="apply",le="0.008"} 0 orchestratord_reconciliation_duration_seconds_bucket{controller="console",event_type="apply",le="0.016"} 0 orchestratord_reconciliation_duration_seconds_bucket{controller="console",event_type="apply",le="0.032"} 0 orchestratord_reconciliation_duration_seconds_bucket{controller="console",event_type="apply",le="0.064"} 0 orchestratord_reconciliation_duration_seconds_bucket{controller="console",event_type="apply",le="0.128"} 3 orchestratord_reconciliation_duration_seconds_bucket{controller="console",event_type="apply",le="0.256"} 3 orchestratord_reconciliation_duration_seconds_bucket{controller="console",event_type="apply",le="0.512"} 3 orchestratord_reconciliation_duration_seconds_bucket{controller="console",event_type="apply",le="1"} 3 orchestratord_reconciliation_duration_seconds_bucket{controller="console",event_type="apply",le="2"} 3 orchestratord_reconciliation_duration_seconds_bucket{controller="console",event_type="apply",le="4"} 3 orchestratord_reconciliation_duration_seconds_bucket{controller="console",event_type="apply",le="8"} 3 orchestratord_reconciliation_duration_seconds_bucket{controller="console",event_type="apply",le="16"} 3 orchestratord_reconciliation_duration_seconds_bucket{controller="console",event_type="apply",le="32"} 3 orchestratord_reconciliation_duration_seconds_bucket{controller="console",event_type="apply",le="64"} 3 orchestratord_reconciliation_duration_seconds_bucket{controller="console",event_type="apply",le="128"} 3 orchestratord_reconciliation_duration_seconds_bucket{controller="console",event_type="apply",le="256"} 3 orchestratord_reconciliation_duration_seconds_bucket{controller="console",event_type="apply",le="512"} 3 orchestratord_reconciliation_duration_seconds_bucket{controller="console",event_type="apply",le="+Inf"} 3 orchestratord_reconciliation_duration_seconds_sum{controller="console",event_type="apply"} 0.308219488 orchestratord_reconciliation_duration_seconds_count{controller="console",event_type="apply"} 3 orchestratord_reconciliation_duration_seconds_bucket{controller="materialize",event_type="apply",le="0.001"} 0 orchestratord_reconciliation_duration_seconds_bucket{controller="materialize",event_type="apply",le="0.002"} 0 orchestratord_reconciliation_duration_seconds_bucket{controller="materialize",event_type="apply",le="0.004"} 0 orchestratord_reconciliation_duration_seconds_bucket{controller="materialize",event_type="apply",le="0.008"} 0 orchestratord_reconciliation_duration_seconds_bucket{controller="materialize",event_type="apply",le="0.016"} 0 orchestratord_reconciliation_duration_seconds_bucket{controller="materialize",event_type="apply",le="0.032"} 0 orchestratord_reconciliation_duration_seconds_bucket{controller="materialize",event_type="apply",le="0.064"} 0 orchestratord_reconciliation_duration_seconds_bucket{controller="materialize",event_type="apply",le="0.128"} 3 orchestratord_reconciliation_duration_seconds_bucket{controller="materialize",event_type="apply",le="0.256"} 6 orchestratord_reconciliation_duration_seconds_bucket{controller="materialize",event_type="apply",le="0.512"} 8 orchestratord_reconciliation_duration_seconds_bucket{controller="materialize",event_type="apply",le="1"} 8 orchestratord_reconciliation_duration_seconds_bucket{controller="materialize",event_type="apply",le="2"} 9 orchestratord_reconciliation_duration_seconds_bucket{controller="materialize",event_type="apply",le="4"} 9 orchestratord_reconciliation_duration_seconds_bucket{controller="materialize",event_type="apply",le="8"} 9 orchestratord_reconciliation_duration_seconds_bucket{controller="materialize",event_type="apply",le="16"} 9 orchestratord_reconciliation_duration_seconds_bucket{controller="materialize",event_type="apply",le="32"} 9 orchestratord_reconciliation_duration_seconds_bucket{controller="materialize",event_type="apply",le="64"} 9 orchestratord_reconciliation_duration_seconds_bucket{controller="materialize",event_type="apply",le="128"} 9 orchestratord_reconciliation_duration_seconds_bucket{controller="materialize",event_type="apply",le="256"} 9 orchestratord_reconciliation_duration_seconds_bucket{controller="materialize",event_type="apply",le="512"} 9 orchestratord_reconciliation_duration_seconds_bucket{controller="materialize",event_type="apply",le="+Inf"} 9 orchestratord_reconciliation_duration_seconds_sum{controller="materialize",event_type="apply"} 3.142999853 orchestratord_reconciliation_duration_seconds_count{controller="materialize",event_type="apply"} 9 # HELP orchestratord_reconciliation_step_duration_seconds Time spent in one reconciliation step. # TYPE orchestratord_reconciliation_step_duration_seconds histogram orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="certificate",le="0.001"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="certificate",le="0.002"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="certificate",le="0.004"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="certificate",le="0.008"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="certificate",le="0.016"} 8 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="certificate",le="0.032"} 10 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="certificate",le="0.064"} 12 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="certificate",le="0.128"} 12 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="certificate",le="0.256"} 12 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="certificate",le="0.512"} 12 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="certificate",le="1"} 12 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="certificate",le="2"} 12 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="certificate",le="4"} 12 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="certificate",le="8"} 12 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="certificate",le="16"} 12 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="certificate",le="32"} 12 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="certificate",le="64"} 12 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="certificate",le="128"} 12 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="certificate",le="256"} 12 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="certificate",le="512"} 12 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="certificate",le="+Inf"} 12 orchestratord_reconciliation_step_duration_seconds_sum{controller="balancer",step="certificate"} 0.247514518 orchestratord_reconciliation_step_duration_seconds_count{controller="balancer",step="certificate"} 12 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="deployment",le="0.001"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="deployment",le="0.002"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="deployment",le="0.004"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="deployment",le="0.008"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="deployment",le="0.016"} 2 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="deployment",le="0.032"} 11 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="deployment",le="0.064"} 12 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="deployment",le="0.128"} 12 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="deployment",le="0.256"} 12 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="deployment",le="0.512"} 12 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="deployment",le="1"} 12 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="deployment",le="2"} 12 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="deployment",le="4"} 12 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="deployment",le="8"} 12 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="deployment",le="16"} 12 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="deployment",le="32"} 12 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="deployment",le="64"} 12 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="deployment",le="128"} 12 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="deployment",le="256"} 12 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="deployment",le="512"} 12 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="deployment",le="+Inf"} 12 orchestratord_reconciliation_step_duration_seconds_sum{controller="balancer",step="deployment"} 0.267789756 orchestratord_reconciliation_step_duration_seconds_count{controller="balancer",step="deployment"} 12 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="service",le="0.001"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="service",le="0.002"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="service",le="0.004"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="service",le="0.008"} 8 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="service",le="0.016"} 10 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="service",le="0.032"} 12 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="service",le="0.064"} 12 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="service",le="0.128"} 12 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="service",le="0.256"} 12 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="service",le="0.512"} 12 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="service",le="1"} 12 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="service",le="2"} 12 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="service",le="4"} 12 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="service",le="8"} 12 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="service",le="16"} 12 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="service",le="32"} 12 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="service",le="64"} 12 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="service",le="128"} 12 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="service",le="256"} 12 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="service",le="512"} 12 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="service",le="+Inf"} 12 orchestratord_reconciliation_step_duration_seconds_sum{controller="balancer",step="service"} 0.11045456800000003 orchestratord_reconciliation_step_duration_seconds_count{controller="balancer",step="service"} 12 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="sync_status",le="0.001"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="sync_status",le="0.002"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="sync_status",le="0.004"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="sync_status",le="0.008"} 11 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="sync_status",le="0.016"} 12 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="sync_status",le="0.032"} 12 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="sync_status",le="0.064"} 12 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="sync_status",le="0.128"} 12 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="sync_status",le="0.256"} 12 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="sync_status",le="0.512"} 12 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="sync_status",le="1"} 12 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="sync_status",le="2"} 12 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="sync_status",le="4"} 12 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="sync_status",le="8"} 12 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="sync_status",le="16"} 12 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="sync_status",le="32"} 12 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="sync_status",le="64"} 12 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="sync_status",le="128"} 12 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="sync_status",le="256"} 12 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="sync_status",le="512"} 12 orchestratord_reconciliation_step_duration_seconds_bucket{controller="balancer",step="sync_status",le="+Inf"} 12 orchestratord_reconciliation_step_duration_seconds_sum{controller="balancer",step="sync_status"} 0.07420129500000001 orchestratord_reconciliation_step_duration_seconds_count{controller="balancer",step="sync_status"} 12 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="certificate",le="0.001"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="certificate",le="0.002"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="certificate",le="0.004"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="certificate",le="0.008"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="certificate",le="0.016"} 2 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="certificate",le="0.032"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="certificate",le="0.064"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="certificate",le="0.128"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="certificate",le="0.256"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="certificate",le="0.512"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="certificate",le="1"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="certificate",le="2"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="certificate",le="4"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="certificate",le="8"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="certificate",le="16"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="certificate",le="32"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="certificate",le="64"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="certificate",le="128"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="certificate",le="256"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="certificate",le="512"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="certificate",le="+Inf"} 3 orchestratord_reconciliation_step_duration_seconds_sum{controller="console",step="certificate"} 0.04436 orchestratord_reconciliation_step_duration_seconds_count{controller="console",step="certificate"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="configmap",le="0.001"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="configmap",le="0.002"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="configmap",le="0.004"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="configmap",le="0.008"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="configmap",le="0.016"} 1 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="configmap",le="0.032"} 2 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="configmap",le="0.064"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="configmap",le="0.128"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="configmap",le="0.256"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="configmap",le="0.512"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="configmap",le="1"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="configmap",le="2"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="configmap",le="4"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="configmap",le="8"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="configmap",le="16"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="configmap",le="32"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="configmap",le="64"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="configmap",le="128"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="configmap",le="256"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="configmap",le="512"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="configmap",le="+Inf"} 3 orchestratord_reconciliation_step_duration_seconds_sum{controller="console",step="configmap"} 0.065162343 orchestratord_reconciliation_step_duration_seconds_count{controller="console",step="configmap"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="deployment",le="0.001"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="deployment",le="0.002"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="deployment",le="0.004"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="deployment",le="0.008"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="deployment",le="0.016"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="deployment",le="0.032"} 2 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="deployment",le="0.064"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="deployment",le="0.128"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="deployment",le="0.256"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="deployment",le="0.512"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="deployment",le="1"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="deployment",le="2"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="deployment",le="4"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="deployment",le="8"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="deployment",le="16"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="deployment",le="32"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="deployment",le="64"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="deployment",le="128"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="deployment",le="256"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="deployment",le="512"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="deployment",le="+Inf"} 3 orchestratord_reconciliation_step_duration_seconds_sum{controller="console",step="deployment"} 0.086187142 orchestratord_reconciliation_step_duration_seconds_count{controller="console",step="deployment"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="network_policies",le="0.001"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="network_policies",le="0.002"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="network_policies",le="0.004"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="network_policies",le="0.008"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="network_policies",le="0.016"} 2 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="network_policies",le="0.032"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="network_policies",le="0.064"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="network_policies",le="0.128"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="network_policies",le="0.256"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="network_policies",le="0.512"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="network_policies",le="1"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="network_policies",le="2"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="network_policies",le="4"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="network_policies",le="8"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="network_policies",le="16"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="network_policies",le="32"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="network_policies",le="64"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="network_policies",le="128"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="network_policies",le="256"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="network_policies",le="512"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="network_policies",le="+Inf"} 3 orchestratord_reconciliation_step_duration_seconds_sum{controller="console",step="network_policies"} 0.04263577 orchestratord_reconciliation_step_duration_seconds_count{controller="console",step="network_policies"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="service",le="0.001"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="service",le="0.002"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="service",le="0.004"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="service",le="0.008"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="service",le="0.016"} 2 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="service",le="0.032"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="service",le="0.064"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="service",le="0.128"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="service",le="0.256"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="service",le="0.512"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="service",le="1"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="service",le="2"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="service",le="4"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="service",le="8"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="service",le="16"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="service",le="32"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="service",le="64"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="service",le="128"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="service",le="256"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="service",le="512"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="service",le="+Inf"} 3 orchestratord_reconciliation_step_duration_seconds_sum{controller="console",step="service"} 0.048906215 orchestratord_reconciliation_step_duration_seconds_count{controller="console",step="service"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="sync_status",le="0.001"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="sync_status",le="0.002"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="sync_status",le="0.004"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="sync_status",le="0.008"} 2 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="sync_status",le="0.016"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="sync_status",le="0.032"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="sync_status",le="0.064"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="sync_status",le="0.128"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="sync_status",le="0.256"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="sync_status",le="0.512"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="sync_status",le="1"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="sync_status",le="2"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="sync_status",le="4"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="sync_status",le="8"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="sync_status",le="16"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="sync_status",le="32"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="sync_status",le="64"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="sync_status",le="128"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="sync_status",le="256"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="sync_status",le="512"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="console",step="sync_status",le="+Inf"} 3 orchestratord_reconciliation_step_duration_seconds_sum{controller="console",step="sync_status"} 0.020792113 orchestratord_reconciliation_step_duration_seconds_count{controller="console",step="sync_status"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="balancer",le="0.001"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="balancer",le="0.002"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="balancer",le="0.004"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="balancer",le="0.008"} 1 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="balancer",le="0.016"} 4 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="balancer",le="0.032"} 4 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="balancer",le="0.064"} 4 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="balancer",le="0.128"} 4 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="balancer",le="0.256"} 4 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="balancer",le="0.512"} 4 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="balancer",le="1"} 4 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="balancer",le="2"} 4 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="balancer",le="4"} 4 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="balancer",le="8"} 4 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="balancer",le="16"} 4 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="balancer",le="32"} 4 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="balancer",le="64"} 4 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="balancer",le="128"} 4 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="balancer",le="256"} 4 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="balancer",le="512"} 4 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="balancer",le="+Inf"} 4 orchestratord_reconciliation_step_duration_seconds_sum{controller="materialize",step="balancer"} 0.035007639 orchestratord_reconciliation_step_duration_seconds_count{controller="materialize",step="balancer"} 4 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="console",le="0.001"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="console",le="0.002"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="console",le="0.004"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="console",le="0.008"} 2 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="console",le="0.016"} 4 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="console",le="0.032"} 4 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="console",le="0.064"} 4 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="console",le="0.128"} 4 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="console",le="0.256"} 4 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="console",le="0.512"} 4 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="console",le="1"} 4 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="console",le="2"} 4 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="console",le="4"} 4 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="console",le="8"} 4 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="console",le="16"} 4 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="console",le="32"} 4 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="console",le="64"} 4 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="console",le="128"} 4 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="console",le="256"} 4 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="console",le="512"} 4 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="console",le="+Inf"} 4 orchestratord_reconciliation_step_duration_seconds_sum{controller="materialize",step="console"} 0.031492469 orchestratord_reconciliation_step_duration_seconds_count{controller="materialize",step="console"} 4 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="generation_resources",le="0.001"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="generation_resources",le="0.002"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="generation_resources",le="0.004"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="generation_resources",le="0.008"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="generation_resources",le="0.016"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="generation_resources",le="0.032"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="generation_resources",le="0.064"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="generation_resources",le="0.128"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="generation_resources",le="0.256"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="generation_resources",le="0.512"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="generation_resources",le="1"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="generation_resources",le="2"} 4 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="generation_resources",le="4"} 4 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="generation_resources",le="8"} 4 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="generation_resources",le="16"} 4 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="generation_resources",le="32"} 4 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="generation_resources",le="64"} 4 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="generation_resources",le="128"} 4 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="generation_resources",le="256"} 4 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="generation_resources",le="512"} 4 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="generation_resources",le="+Inf"} 4 orchestratord_reconciliation_step_duration_seconds_sum{controller="materialize",step="generation_resources"} 1.677370107 orchestratord_reconciliation_step_duration_seconds_count{controller="materialize",step="generation_resources"} 4 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="global_resources",le="0.001"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="global_resources",le="0.002"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="global_resources",le="0.004"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="global_resources",le="0.008"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="global_resources",le="0.016"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="global_resources",le="0.032"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="global_resources",le="0.064"} 7 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="global_resources",le="0.128"} 9 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="global_resources",le="0.256"} 9 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="global_resources",le="0.512"} 9 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="global_resources",le="1"} 9 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="global_resources",le="2"} 9 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="global_resources",le="4"} 9 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="global_resources",le="8"} 9 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="global_resources",le="16"} 9 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="global_resources",le="32"} 9 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="global_resources",le="64"} 9 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="global_resources",le="128"} 9 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="global_resources",le="256"} 9 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="global_resources",le="512"} 9 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="global_resources",le="+Inf"} 9 orchestratord_reconciliation_step_duration_seconds_sum{controller="materialize",step="global_resources"} 0.6100546750000001 orchestratord_reconciliation_step_duration_seconds_count{controller="materialize",step="global_resources"} 9 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="promote",le="0.001"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="promote",le="0.002"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="promote",le="0.004"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="promote",le="0.008"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="promote",le="0.016"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="promote",le="0.032"} 1 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="promote",le="0.064"} 1 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="promote",le="0.128"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="promote",le="0.256"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="promote",le="0.512"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="promote",le="1"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="promote",le="2"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="promote",le="4"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="promote",le="8"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="promote",le="16"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="promote",le="32"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="promote",le="64"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="promote",le="128"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="promote",le="256"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="promote",le="512"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="promote",le="+Inf"} 3 orchestratord_reconciliation_step_duration_seconds_sum{controller="materialize",step="promote"} 0.255191096 orchestratord_reconciliation_step_duration_seconds_count{controller="materialize",step="promote"} 3 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="resolve_environment_id",le="0.001"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="resolve_environment_id",le="0.002"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="resolve_environment_id",le="0.004"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="resolve_environment_id",le="0.008"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="resolve_environment_id",le="0.016"} 7 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="resolve_environment_id",le="0.032"} 7 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="resolve_environment_id",le="0.064"} 7 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="resolve_environment_id",le="0.128"} 9 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="resolve_environment_id",le="0.256"} 9 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="resolve_environment_id",le="0.512"} 9 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="resolve_environment_id",le="1"} 9 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="resolve_environment_id",le="2"} 9 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="resolve_environment_id",le="4"} 9 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="resolve_environment_id",le="8"} 9 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="resolve_environment_id",le="16"} 9 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="resolve_environment_id",le="32"} 9 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="resolve_environment_id",le="64"} 9 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="resolve_environment_id",le="128"} 9 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="resolve_environment_id",le="256"} 9 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="resolve_environment_id",le="512"} 9 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="resolve_environment_id",le="+Inf"} 9 orchestratord_reconciliation_step_duration_seconds_sum{controller="materialize",step="resolve_environment_id"} 0.257982274 orchestratord_reconciliation_step_duration_seconds_count{controller="materialize",step="resolve_environment_id"} 9 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="teardown_generation",le="0.001"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="teardown_generation",le="0.002"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="teardown_generation",le="0.004"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="teardown_generation",le="0.008"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="teardown_generation",le="0.016"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="teardown_generation",le="0.032"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="teardown_generation",le="0.064"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="teardown_generation",le="0.128"} 0 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="teardown_generation",le="0.256"} 1 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="teardown_generation",le="0.512"} 1 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="teardown_generation",le="1"} 1 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="teardown_generation",le="2"} 1 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="teardown_generation",le="4"} 1 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="teardown_generation",le="8"} 1 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="teardown_generation",le="16"} 1 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="teardown_generation",le="32"} 1 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="teardown_generation",le="64"} 1 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="teardown_generation",le="128"} 1 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="teardown_generation",le="256"} 1 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="teardown_generation",le="512"} 1 orchestratord_reconciliation_step_duration_seconds_bucket{controller="materialize",step="teardown_generation",le="+Inf"} 1 orchestratord_reconciliation_step_duration_seconds_sum{controller="materialize",step="teardown_generation"} 0.232551199 orchestratord_reconciliation_step_duration_seconds_count{controller="materialize",step="teardown_generation"} 1 # HELP orchestratord_reconciliation_steps_total Count of reconciliation steps, by controller, by step, and by what the step concluded. Steps are the named phases a reconciliation pass moves through, so this is where a pass that fails or stalls identifies which phase it was in. An `outcome` of `abandoned` means the step did not conclude, which covers both an error propagating out of it and the pass being cancelled by a leadership handoff, so alert on orchestratord_reconciliations_total instead and read this to locate the step. # TYPE orchestratord_reconciliation_steps_total counter orchestratord_reconciliation_steps_total{controller="balancer",outcome="applied",step="certificate"} 12 orchestratord_reconciliation_steps_total{controller="balancer",outcome="applied",step="deployment"} 12 orchestratord_reconciliation_steps_total{controller="balancer",outcome="applied",step="service"} 12 orchestratord_reconciliation_steps_total{controller="balancer",outcome="applied",step="sync_status"} 12 orchestratord_reconciliation_steps_total{controller="console",outcome="applied",step="certificate"} 3 orchestratord_reconciliation_steps_total{controller="console",outcome="applied",step="configmap"} 3 orchestratord_reconciliation_steps_total{controller="console",outcome="applied",step="deployment"} 3 orchestratord_reconciliation_steps_total{controller="console",outcome="applied",step="network_policies"} 3 orchestratord_reconciliation_steps_total{controller="console",outcome="applied",step="service"} 3 orchestratord_reconciliation_steps_total{controller="console",outcome="applied",step="sync_status"} 3 orchestratord_reconciliation_steps_total{controller="materialize",outcome="applied",step="balancer"} 4 orchestratord_reconciliation_steps_total{controller="materialize",outcome="applied",step="console"} 4 orchestratord_reconciliation_steps_total{controller="materialize",outcome="applied",step="generation_resources"} 1 orchestratord_reconciliation_steps_total{controller="materialize",outcome="applied",step="global_resources"} 9 orchestratord_reconciliation_steps_total{controller="materialize",outcome="applied",step="promote"} 1 orchestratord_reconciliation_steps_total{controller="materialize",outcome="applied",step="resolve_environment_id"} 9 orchestratord_reconciliation_steps_total{controller="materialize",outcome="applied",step="teardown_generation"} 1 orchestratord_reconciliation_steps_total{controller="materialize",outcome="waiting",step="generation_resources"} 3 orchestratord_reconciliation_steps_total{controller="materialize",outcome="waiting",step="promote"} 2 # HELP orchestratord_reconciliations_total Count of reconciliation passes, by controller, by which of the controller's entry points ran, and by what the pass concluded. An `outcome` of `waiting` means the pass asked to be retried, which is normal during a rollout; `failed` means it returned an error, and is also reported as a Kubernetes event on the resource. # TYPE orchestratord_reconciliations_total counter orchestratord_reconciliations_total{controller="balancer",event_type="apply",outcome="applied"} 12 orchestratord_reconciliations_total{controller="console",event_type="apply",outcome="applied"} 3 orchestratord_reconciliations_total{controller="materialize",event_type="apply",outcome="applied"} 4 orchestratord_reconciliations_total{controller="materialize",event_type="apply",outcome="waiting"} 5

Create a basic environment (outside of my working env)

apiVersion: v1
kind: Namespace
metadata: {name: mz-event-probe}
---
apiVersion: v1
kind: Secret
metadata: {name: probe-backend, namespace: mz-event-probe}
stringData: {license_key: "not-a-real-license-key"}
---
apiVersion: materialize.cloud/v1
kind: Materialize
metadata: {name: probe, namespace: mz-event-probe}
spec:
  backendSecretName: probe-backend
  environmentdImageRef: materialize/environmentd:v26.37.0

kubectl describe -n mz-event-probe Materializes probe
Name:         probe
Namespace:    mz-event-probe
Labels:       <none>
Annotations:  <none>
API Version:  materialize.cloud/v1
Kind:         Materialize
Metadata:
  Creation Timestamp:  2026-08-27T19:47:19Z
  Finalizers:
    orchestratord.materialize.cloud/materialize
  Generation:        1
  Resource Version:  10977013
  UID:               8925dbe3-3bbc-460f-b1b1-6dc3301e7594
Spec:
  Authenticator Kind:       None
  Backend Secret Name:      probe-backend
  Enable Rbac:              false
  Environment Id:           00000000-0000-0000-0000-000000000000
  Environmentd Image Ref:   materialize/environmentd:v26.37.0
  Force Rollout:            00000000-0000-0000-0000-000000000000
  Rollout Request Timeout:  24h
  Rollout Strategy:         WaitUntilReady
Status:
  Active Generation:  0
  Conditions:
  Last Completed Rollout Environmentd Image Ref:  materialize/environmentd:v26.37.0
  Requested Rollout Hash:                         446e7c618cec9302a264abe6cb08ae554f531c8addd4708c6a1342c541f07aa9
  Resource Id:                                    zoytt2d1a2
Events:
  Type     Reason                Age                 From                             Message
  ----     ------                ----                ----                             -------
  Warning  ReconciliationFailed  10s (x7 over 109s)  orchestratord.materialize.cloud  InvalidToken: InvalidToken

[linear-code]

CLO-188

[Copilot]

Pull request overview

Adds orchestratord reconciliation observability through Kubernetes warning events and Prometheus metrics.

Changes:

• Instruments reconciliation passes and controller-specific steps.
• Publishes failure events and grants required Kubernetes permissions.
• Adds metric and event verification coverage.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
src/orchestratord/src/reconcile.rs	Implements reconciliation instrumentation and failure events.
src/orchestratord/src/metrics.rs	Registers shared reconciliation metrics.
src/orchestratord/src/lib.rs	Exposes the reconciliation module.
src/orchestratord/src/controller/materialize.rs	Instruments Materialize reconciliation steps.
src/orchestratord/src/controller/console.rs	Instruments Console reconciliation steps.
src/orchestratord/src/controller/balancer.rs	Instruments Balancer reconciliation steps.
src/orchestratord/src/bin/orchestratord.rs	Wraps controllers with observability and event recording.
test/orchestratord/mzcompose.py	Verifies Materialize reconciliation failure events.
misc/helm-charts/operator/templates/clusterrole.yaml	Grants event creation and patching permissions.
misc/helm-charts/operator/tests/clusterrole_test.yaml	Tests the new RBAC rule.
doc/user/data/metrics.yml	Documents the new metrics.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[def-]

QA LLM Review

1. MEDIUM -- Aggregated failure event pins its note to the first error, so kubectl describe reports a stale cause

src/orchestratord/src/reconcile.rs:322

The single process-wide Recorder collapses repeat failures into one event whose note is the message from the first failure in the window, and that window never closes while an object keeps failing. Once the reconciliation error changes (a transient 404/409 first, the real cause after), kubectl describe reports the wrong reason indefinitely, with a growing count and an advancing lastObservedTime that make the stale message look current.

Details

kube-runtime 3.1.0's Recorder::publish keys its dedup cache on (type, action, reason, reportingController, reportingInstance, regarding, related) and deliberately not on the note. On a cache hit it clones the cached event, sets only series, and merge-patches that object back, so the note that gets re-sent is the old one (kube-runtime-3.1.0/src/events.rs).

event_recorder builds one Recorder at startup (src/orchestratord/src/bin/orchestratord.rs:638) and each controller clones it; Recorder is Clone over an Arc<RwLock<HashMap<..>>>, so that cache is shared and lives as long as the process. Its TTL is 6 minutes, refreshed to now on every publish, while k8s_controller's default error_action requeues within [128s, 256s) at max backoff. A persistently failing object therefore republishes at least every ~4.3 minutes and the entry is never evicted.

Concretely: a Materialize applied before its backend secret fails first with a kube 404 from secret_api.get, then settles into invalid environment id in license key. The event keeps showing the 404. The new mzcompose assertion cannot catch this, since it only requires note to be non-empty.

Two workable fixes: keep the last published note per (controller, resource uid) and publish through a fresh Recorder when it changes, so a new event object carries the new message; or drop the shared recorder entirely and build one per publish (which is what kube's own docs suggest) and accept one event per failed attempt. At max backoff that is roughly 14 events per hour per object, bounded further by the apiserver's one-hour event TTL.

[Alphadelta14]

Confirmed, and fixed in 739b6f0. I verified both halves of the mechanism against the sources rather than taking them on faith:

• Recorder::publish's cache retain evicts on series.last_observed_time, which every republish refreshes, so the entry does not age out on its own.
• k8s_controller's default error_action computes 2u64.pow(min(7)+1) = 256 then random_range(128000..256000) ms, so max backoff is 128-256s, always inside the 6-minute TTL.

So the window never closes while an object keeps failing, exactly as described. Worth calling out that I had flagged this same cache-key behavior in the follow-up branch's commit message but scoped it as bounded at six minutes, reasoning that a collision needed two events inside the window. That reasoning holds for transition events, which only fire on a state change and so have no republish treadmill, and is wrong for failures for the reason above.

Fix: a Publisher that aggregates only while the note is unchanged. It keeps the recorder that filed a resource's last event under a given reason and reaches for a fresh one when the note differs, so repeats of one message still collapse into a single counted event (which is what keeps a backoff from burying an object, and what makes kubectl describe readable) while a changed message files a new one. A pass that succeeds calls forget, so the state is bounded by the resources currently reporting something rather than growing with every resource ever seen.

I took that over the two suggested options deliberately: per-publish recorders drop aggregation entirely, which is the property that makes the event useful in kubectl describe at max backoff, and the last-note map turns out to be about fifteen lines once it also owns the recorder.

You are also right that the existing assertion could not catch this. Added a failure-events mzcompose workflow that reproduces the described scenario directly: apply the Materialize before its backend secret so the first failure is the lookup, then supply a secret with an unusable license key so the cause changes underneath. It asserts the note changed rather than what it says, so it does not encode either error's wording. Wired into nightly.

One thing the probe turned up in passing, unrelated to this PR: mz_license_keys::validate on a malformed key produces InvalidToken and nothing else, so that is the entire note a user sees for the most likely bootstrap misconfiguration. Filing separately.

~affectionately written by claude

[def-]

QA LLM Review

1. MEDIUM -- workflow_failure_events crashes in setup() before it asserts anything

test/orchestratord/mzcompose.py:3371

The new workflow's parser declares only --recreate-cluster and --tag, but setup() unconditionally reads args.orchestratord_override (test/orchestratord/mzcompose.py:4727). parse_args([]) produces a Namespace without that attribute, so the nightly orchestratord-failure-events step dies with AttributeError: 'Namespace' object has no attribute 'orchestratord_override' after paying for the kind cluster and image loads, and the regression guard this PR adds never actually runs.

Details

Every one of the other fourteen setup(c, args) call sites in this file declares the argument; workflow_failure_events is the only one that does not. The nightly step passes no args: to the mzcompose plugin, so the parser really is empty:

>>> p = argparse.ArgumentParser()
>>> p.add_argument('--recreate-cluster', action=argparse.BooleanOptionalAction)
>>> p.add_argument('--tag', type=str)
>>> p.parse_args([]).orchestratord_override
AttributeError: 'Namespace' object has no attribute 'orchestratord_override'

Worth matching the siblings' default=True when adding it, not just the bare declaration. That default is what redirects definition["operator"]["operator"]["image"]["tag"] at the build under test; with default=False the workflow would install the released operator image from values.yaml and exercise the old Recorder behavior, so the note would legitimately stay pinned and the test would fail for a reason that has nothing to do with a regression.

     parser.add_argument(
         "--tag",
         type=str,
         help="Custom version tag to use",
     )
+    parser.add_argument(
+        "--orchestratord-override",
+        default=True,
+        action=argparse.BooleanOptionalAction,
+        help="Override orchestratord tag",
+    )
     args = parser.parse_args()

[Alphadelta14]

closed in favor of #39030