feat(monitoring): scrape etcd, scheduler and controller-manager [DO NOT MERGE until Talos apply] - #170
Open
jdwillmsen wants to merge 1 commit into
Open
feat(monitoring): scrape etcd, scheduler and controller-manager [DO NOT MERGE until Talos apply]#170jdwillmsen wants to merge 1 commit into
jdwillmsen wants to merge 1 commit into
Conversation
The cluster had no telemetry for the three control-plane components whose degradation is cluster-fatal, so an etcd latency spike on 2026-07-24 had to be diagnosed from apiserver logs alone and was diagnosed wrongly. Scheduler and controller-manager need no address list: both run as hostNetwork static pods labelled component=kube-scheduler / component=kube-controller-manager, which the chart's default headless-Service selector already matches, and their pod IP is the node IP. Enabling the stock exporters is sufficient. etcd is a Talos host service with no pod, so the chart's selector matches nothing and its only fallback is a hardcoded list of control-plane addresses. Those addresses are DHCP-assigned, so a ScrapeConfig using node service discovery is used instead - Prometheus re-resolves them every discovery cycle and no address is committed. kubeEtcd.enabled stays true with only its Service and ServiceMonitor disabled: the bundled etcd rules are gated on that flag as well as on defaultRules.rules.etcd, so disabling it ships metrics with no alerts. Refs JDWLABS-197 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Comment on lines
+6
to
+40
| spec: | ||
| # The chart's kubeEtcd exporter cannot be used: etcd here is a host-level | ||
| # service with no pod, so the default component=etcd Service selector matches | ||
| # nothing and the only fallback is a hardcoded address list. Node service | ||
| # discovery re-resolves control-plane addresses from the API on every | ||
| # discovery cycle, so a DHCP address change is followed instead of silently | ||
| # ending the scrape. | ||
| # | ||
| # The job name must contain "etcd": every rule in the bundled etcd group | ||
| # selects on job=~".*etcd.*", so a different name loads the rules but leaves | ||
| # them permanently without data. | ||
| jobName: kube-etcd | ||
| # Port 2381 is etcd's dedicated metrics listener, which serves only /metrics | ||
| # and /health over plain HTTP. The client port (2379) also serves metrics but | ||
| # demands client certificates; 2381 exposes no client API at all, so no | ||
| # scrape-side TLS or credentials are involved. | ||
| scheme: HTTP | ||
| kubernetesSDConfigs: | ||
| - role: Node | ||
| relabelings: | ||
| # The control-plane role label is set with an EMPTY value, so matching on | ||
| # the label's value keeps nothing. labelpresent is the only reliable test. | ||
| - action: keep | ||
| sourceLabels: [__meta_kubernetes_node_labelpresent_node_role_kubernetes_io_control_plane] | ||
| regex: "true" | ||
| # Node discovery addresses the kubelet by default; retarget to the node's | ||
| # InternalIP on the etcd metrics port. | ||
| - action: replace | ||
| sourceLabels: [__meta_kubernetes_node_address_InternalIP] | ||
| regex: (.+) | ||
| targetLabel: __address__ | ||
| replacement: $1:2381 | ||
| - action: replace | ||
| sourceLabels: [__meta_kubernetes_node_name] | ||
| targetLabel: node |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
DO NOT MERGE until the Talos machine config is applied
Every ArgoCD app here runs
prune+selfHeal, so merging this is deploying it. The scrape targets this adds cannot come up until the control-plane bind addresses are widened by a Talos machine-config apply — which is human-gated and has not run yet.Merging early does not fail safe: Prometheus records
up == 0for six targets and the newly-enabledKubeSchedulerDown/KubeControllerManagerDown/etcdInsufficientMembersrules fire, describing a control-plane outage that is not happening.Gate:
talops reconcile --plan→ review →talops reconcile→ confirm0.0.0.0:2381,0.0.0.0:10257,0.0.0.0:10259LISTEN on all three control planes → then merge this.The Talos side is already merged: infrastructure#63 (
e189311) addslisten-metrics-urlsand bothbind-addressvalues tobootstrap/internal/talos/patches/control-plane.yaml. It has never been applied.Verified live on all three control planes at the time of writing — unchanged from the 2026-07-27 and 2026-07-29 baselines:
2381is absent on all three.What this changes
tenants/platform/services/kube-prometheus-stack/values.yamlkubeScheduler.enabled/kubeControllerManager.enabled→true(stock exporters)kubeEtcd.enabled→true, withservice.enabledandserviceMonitor.enabled→falsedefaultRules.rules:etcd,kubeControllerManager,kubeSchedulerAlerting,kubeSchedulerRecording→trueprometheus.prometheusSpec.scrapeConfigSelectorNilUsesHelmValues→false, matching the four sibling selector flags already theretenants/platform/services/kube-prometheus-stack/postInstall/scrapeconfig-kube-etcd.yaml(new) — aScrapeConfigfor etcd using node service discovery.Two design decisions, both driven by live evidence
1. Scheduler and controller-manager need no address list — the premise on the ticket was wrong.
The ticket proposed node service discovery for all three components. That is unnecessary for two of them. Both run as
hostNetworkstatic pods inkube-systemcarrying exactly the labels the chart's default selector uses, and their pod IP is the node IP:Labels present on all three (
kubectl get pods -n kube-system -l tier=control-plane --show-labels):component=kube-scheduler,component=kube-controller-manager. So enabling the stock exporters commits no addresses and adds no custom config. Rendered result:insecureSkipVerifyis the chart default on Kubernetes >= 1.23 and is required, not incidental: both components generate a self-signed serving cert at process start. Authz is covered — the Prometheus ClusterRole already grantsnonResourceURLs: [/metrics].2. etcd genuinely cannot use the chart exporter.
etcd on Talos is a host-level service, not a pod:
So the chart's
component=etcdselector matches nothing, and the only fallback iskubeEtcd.endpoints— a literal list of control-plane IPs. Those are DHCP-assigned here, and a scrape config that stops resolving the moment an address moves is useless precisely during the incident class these metrics exist to diagnose. Node service discovery re-resolves from the API every cycle and commits no address.Two details in that ScrapeConfig worth reviewing:
__meta_kubernetes_node_labelpresent_node_role_kubernetes_io_control_plane, not the label's value.node-role.kubernetes.io/control-planeis set with an empty value on these nodes, so a value regex keeps nothing.jobName: kube-etcd— every rule in the bundled etcd group selects onjob=~".*etcd.*". A name without "etcd" in it loads the rules and leaves them permanently without data.The trap this nearly shipped with
kubeEtcd.enabled: falsedoes not only disable the etcd exporter — it also gates the etcd rules:My first draft set
defaultRules.rules.etcd: truewhile leavingkubeEtcd.enabled: false, and the render came back with no etcd rule group at all — metrics arriving with nothing watching them, and a green diff. Settingenabled: truewith onlyservice/serviceMonitoroff renders the rules while creating neither of the two objects that cannot work without a pod to select. Theendpoints.yamltemplate is separately gated on a non-emptykubeEtcd.endpoints, so noEndpointsobject appears either.Validation
Chart rendered at the pinned revision (87.19.1) against the live Kubernetes version, via
rtk proxyso the output is not truncated:Object-level diff against
origin/main's values, same chart and flags — 8 added, 0 removed, 1 modified:The only Prometheus CR field that moved, and a scan of the 8 added objects for committed addresses:
Rule groups now rendering:
Repo gates, all green:
The ScrapeConfig is genuinely schema-checked, not skipped (
Skipped: 0). It also validates against the live cluster's CRD schema rather than only the catalog copy:CRD ownership — no bootstrap-ordering change needed
scrapeconfigs.monitoring.coreos.comis not vendored inbootstrap/crds/foundation-crds.yaml(which carries onlyservicemonitors,podmonitors,prometheusrules). It does not need to be:crds.enabled: true), annotatedoperator.prometheus.io/version: 0.92.1.postInstall/is a source in the same Application as the chart (helm-charts/tenant-envelope/templates/services-appset.yaml), and that Application setsSkipDryRunOnMissingResource=true, which is exactly the CRD-created-in-the-same-sync case.ignoreDifferencesentry is required — unlike the three vendored CRDs, which have one each.What cannot be verified until after the apply
Nothing below is claimed as done. All of it is post-deploy work:
count by (job) (up)includingkube-etcd,kube-scheduler,kube-controller-manager— 3 targets eachetcd_disk_wal_fsync_duration_secondsandetcd_server_leader_changes_seen_totalreturning data:10257/:10259accept the Prometheus service-account token against the Talos-issued serving cert in practice — the config is right in principle, but only a live scrape proves it:2381truly needs no scrape-side TLS (expected plain HTTP, since the merged Talos patch sets the scheme tohttp)inactiverather thanfiringplatform-kube-prometheus-stackSynced + HealthyKnown gaps, deliberately not in this PR
DASHBOARDS.md), andgrafana.enabled: falsemeans the chart's bundled etcd dashboard never renders. No etcd dashboard exists in the repo. Adding a large JSON blob whose datasource wiring cannot be checked until data exists belongs in its own change, after the apply./metricson127.0.0.1:10249with no metrics-bind-address override in its args, so it has the same problem and needs the same kind of machine-config fix. Out of scope here.2381is unauthenticated (serves only/metricsand/health, no client API) and becomes LAN-reachable.10257/10259keep their authn/authz. Recorded against the API-exposure-restriction work rather than left undocumented; note NetworkPolicy is currently unenforced cluster-wide, so there is no in-cluster restriction either.Test plan after the apply
Also re-run the cert-approver dual-ownership pre-flight recorded on the ticket around the apply: confirm the
kubelet-serving-cert-approverDeployment's managers stayargocd-controller (Apply)+kube-controller-manager (Update)with no Talos manager, and its image stays pinned at0.11.0.Refs JDWLABS-197
🤖 Generated with Claude Code