Skip to content

feat: add operator observability metrics and Grafana dashboard - #35

Merged
pierinho13 merged 3 commits into
mainfrom
adding_more_observability
Aug 8, 2026
Merged

feat: add operator observability metrics and Grafana dashboard#35
pierinho13 merged 3 commits into
mainfrom
adding_more_observability

Conversation

@pierinho13

Copy link
Copy Markdown
Owner

What

Add a small set of custom Prometheus metrics for github-platform-operator and provide a ready-to-import Grafana dashboard.

The operator already exposes the standard controller-runtime, Go and process metrics. This change complements them with GitHub-specific signals that are useful to understand API usage and rate-limit behavior without duplicating metrics already provided by controller-runtime.

Custom metrics

The following metrics are now exposed:

  • github_platform_operator_github_api_requests_total
  • github_platform_operator_github_api_request_duration_seconds
  • github_platform_operator_github_api_transport_errors_total
  • github_platform_operator_github_rate_limit_limit
  • github_platform_operator_github_rate_limit_remaining
  • github_platform_operator_github_rate_limit_reset_timestamp_seconds
  • github_platform_operator_github_rate_limit_events_total
  • github_platform_operator_github_rate_limit_blocked_until_timestamp_seconds

The instrumentation is implemented at the shared GitHub HTTP transport layer so all controllers using the common GitHub client benefit from the same metrics without duplicating instrumentation in each reconciler.

Labels are intentionally kept low-cardinality and do not include repository names, usernames, organizations or tokens.

Grafana dashboard

Add a Grafana dashboard under:

dashboards/grafana/github-platform-operator.json

The dashboard combines the new GitHub-specific metrics with existing controller-runtime metrics and includes panels for:

  • GitHub API request rate
  • HTTP errors
  • GitHub API p95 latency
  • Rate-limit remaining and usage percentage
  • Rate-limit events
  • Shared rate-limit gate blocked duration
  • Reconciliation rate and errors by controller
  • Reconciliation p95 latency
  • Workqueue depth
  • Transport errors
  • CPU, memory and goroutines
  • Rate-limit reset countdown

Tests

Add contract tests covering:

  • GitHub API request metrics
  • rate-limit headers and gauges
  • transport errors
  • rate-limit events
  • shared rate-limit blocked state

The existing metrics E2E test is also extended to verify that custom operator metrics are exposed through /metrics.

Validation

make fmt   ✅
make vet   ✅
make test  ✅
make lint  ✅

@pierinho13
pierinho13 enabled auto-merge August 8, 2026 18:58
@pierinho13
pierinho13 disabled auto-merge August 8, 2026 19:17
@pierinho13
pierinho13 merged commit c5fbdca into main Aug 8, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant