Add reconciliation metrics and Kubernetes events

[Alphadelta14]

What

Adds controller-agnostic building blocks for observing and reporting on reconciliation:

• Metrics: Controller::with_observer(Arc<dyn ReconcileObserver>) reports each pass with its controller name, resource, Phase (init/apply/cleanup/delete), Outcome (completed/waiting/skipped/failed/abandoned) and duration.
  • Reconcilers time named steps with metadata.step("...") on the TraceMetadata they already receive.
  • metadata.set_outcome(...) overrides a pass's outcome.
  • A pass cancelled mid-way (for example, on losing the leadership lease) is recorded as abandoned.
  • PrometheusMetrics, behind a new optional prometheus feature, implements the trait. It's a Collector, so it works with a plain prometheus::Registry or mz_ore's register_collector. Default histogram buckets run from 10ms to 30s ([0.01, 0.05, 0.25, 1, 5, 30]); with_buckets overrides them.
• Events: the new events::EventRecorder publishes events.k8s.io/v1 events on behalf of one controller.
  • Each controller gets its own recorder, whose Reporter names it (a qualified name such as example.com/widgets). The reporter is the only thing in an event that shows which controller published it, even when several controllers reconcile the same kind of resource.
  • An identical repeat increments the existing event's series.count; any change (including to the note) starts a new event. Notes over 1 KiB are truncated. A series whose event the API server has already expired is recreated.
  • Controller::with_event_recorder publishes a warning event whenever a pass fails. The event comes from the new Context::failure_event hook, which has a default implementation and can be overridden to reword or suppress it.
  • Reconcilers can publish their own events with metadata.publish_event(...). Code outside a reconcile (for example, a background task) can publish on the controller's behalf with EventRecorder::publish.
  • Each publish gives up after a timeout (5s by default; EventRecorder::with_timeout overrides it), returning PublishError::Timeout.
• Conditions: the new conditions module maintains standard metav1.Conditions, following the same rules as apimachinery's meta.SetStatusCondition.
  • conditions::set applies a DesiredCondition. It keeps lastTransitionTime unless the status changes, and returns Unchanged/Updated/Transitioned, which tells the caller whether the status needs writing.
  • conditions::find_observed ignores conditions determined from an older generation of the resource.
  • Writing the status is left to the reconciler.
• Controller::with_name sets the controller label. It defaults to FINALIZER_NAME, or failing that the resource kind.
• Error is now exported, along with Error::display_chain. Before this, Context::error_action couldn't be overridden outside the crate.

Why

Downstream controllers (orchestratord in materialize, and the cloud environment/region controllers and stack-deployer) have no metrics for reconciliation, and no way for a stuck object to explain itself in kubectl describe. Materialize's orchestratord has a prototype of the metrics and events as a Context wrapper (CLO-188). Building them into the crate:

• covers finalizer add/remove failures, which a wrapper never sees;
• removes the gotcha where step and pass labels had to be kept in sync by hand;
• lets cancelled passes be recorded.

Every downstream CRD already uses metav1.Condition, but each repo updates conditions by hand, and several get lastTransitionTime wrong (resetting it when only the message changes, or stamping it on every write). conditions gives them one correct implementation to move to.

Reviewer notes

• Every change is additive. The one removal is the #[doc(hidden)] Context::_reconcile, which became a private function.
• Publishes to the same event series are serialized with a per-series async lock held across the API request. Concurrent identical events therefore aggregate instead of each creating an event, and patches can't race on the count. Publishes to different series still run in parallel.
• A successful pass resets the aggregation of the controller's automatic failure events for that resource, so a failure that recurs after a recovery becomes a new event. It does not reset events published with publish_event or EventRecorder::publish; otherwise a reconciler that publishes the same event every pass while it waits would create a new event each time. This is why a recorder must not be shared between controllers: one controller's success would reset another's failure series.
• Failure events are published inline, before the failed pass returns to kube-runtime, so the timeout is what stops a slow or hung events API from holding up that object's requeue. Giving up mid-publish is harmless: at worst one count increment is lost or an extra event is created.
• Ok(Some(Action::requeue(..))) counts as waiting; None and await_change() count as completed. A controller that requeues as a periodic resync after converging would show as waiting unless it calls set_outcome.
• Event publishing needs create and patch on events.k8s.io/events. This is documented in the events module.
• The tests use an in-process mock API server (src/test_util.rs). The async tests run on a small helper runtime instead of #[tokio::test], because tokio's macros feature pulls in a second syn version that cargo-deny rejects.
• I haven't compiled cloud or materialize against this branch.
• The version is not bumped; the CHANGELOG has an Unreleased section.

🤖 Generated with Claude Code

[Copilot]

Copilot review overview

🟡 Changes recommended

Event publishing has a critical concurrency flaw, with additional error rendering, test coverage, and CI gaps.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 1 · 1

Open (2)

• Serialize concurrent recorder state transitions per series · New
• Avoid suppressing causes based on substring matches · New

What changed in this PR

Adds controller-wide reconciliation metrics and Kubernetes event recording, including optional Prometheus support.

Changes:

• Adds reconciliation phases, outcomes, observers, and step timing.
• Adds aggregated Kubernetes events and failure hooks.
• Updates controller integration, tests, exports, and documentation.

File	Description and findings
src/​test_util.rs	Adds mock API-server utilities. Nit (1 vote): Fix the “Creates respond” documentation wording.
src/​prometheus.rs	Implements Prometheus metrics collection. No direct issue identified.
src/​observe.rs	Defines observation APIs. Nits (1 vote each): Fix feature-gated documentation link and inaccurate ReconcileRecord and finish_with behavior descriptions.
src/​lib.rs	Exports and documents new APIs. Nit (1 vote): Default-feature documentation links to feature-gated PrometheusMetrics.
src/​events.rs	Implements event recording and aggregation. Critical (4 votes): Concurrent publishes can lose counts, create duplicate series, or corrupt cached state; serialize transitions per SeriesKey and test concurrency.
src/​controller.rs	Integrates instrumentation and failure events. Moderate (2 votes): display_chain can suppress real causes. Moderate (1 vote): Finalizer phases and PATCH failures lack tests. Nit (1 vote): Cleanup documentation incorrectly claims success_action behavior matches apply.
README.md	Documents observability features. No issue identified.
CHANGELOG.md	Records the unreleased functionality. No issue identified.
Cargo.toml	Adds Prometheus and test dependencies. Moderate (1 vote): CI does not compile or test the optional Prometheus feature.
Cargo.lock	Updates dependency resolution. No issue identified.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[alex-hunt-materialize]

Overall, this looks good.

[alex-hunt-materialize]

If the recorder is is shared by every controller, how do we know which one published the event? It's pretty common to have multiple controllers reconciling the same object.

[Alphadelta14]

the Reporter has a reportingController and reportingInstance

[alex-hunt-materialize]

Maybe I'm just missing something, but don't we also only have one reporter?

I think the term "controller" might be causing some confusion between us. I mean the individual instances of things implementing the traits in this repo, of which there can be multiple in a single pod. These controllers don't even need to point at different resource types. We could have three controllers in a single pod all pointing at Materialize CRs, for example. In that case, how would we know which one generated the event if they all share the same EventRecorder (and therefore the same Reporter)?

[Alphadelta14]

oh, traditionally you'd want a separate reporter per controller implementation. Bad comment.

[alex-hunt-materialize]

It is a bit unclear to me why this would ever not be Some. Shouldn't we always have access to the name of the controller to be able to set this?

[Alphadelta14]

mostly because standard kubernetes allows it to be None. EventRecorder::publish can be called outside of the controller (like a failed to refresh a cache on a timer).