Skip to content

operations.md

Latest commit

 

History

History
316 lines (238 loc) · 8.99 KB

operations.md

File metadata and controls

316 lines (238 loc) · 8.99 KB
Raw

Operations

Observe reconciliation

List all managed resources:

kubectl get \
  ghprovider,ghorgmember,ghteam,ghteammember,ghrepo,ghruleset,ghteamaccess,ghcollab,ghenv,ghsecret,ghvar \
  -A

Inspect a resource and its conditions:

kubectl get ghrepo example-repository -o yaml

Common Ready condition reasons include:

RepositoryCreated
RepositoryUpdated
RepositoryAvailable
RulesetCreated
RulesetUpdated
RulesetAvailable
BypassActorUnavailable
OrganizationMembershipConfigured
OrganizationMembershipUpdated
TeamCreated
TeamUpdated
TeamAvailable
TeamMembershipConfigured
TeamMembershipUpdated
InvitationPending
AccessConfigured
EnvironmentCreated
SecretCreated
SecretUpdated
VariableCreated
VariableUpdated
ReconciliationSuspended
DependencyUnavailable
InvalidDesiredState
ReconciliationFailed

View controller logs:

kubectl logs -f \
  deployment/github-platform-operator \
  --namespace github-platform-operator-system

Deletion policies

The default policy is non-destructive.

Resource Safe default Explicit remote action
GitHubRepository Orphan Archive, Delete
GitHubOrganizationMember Orphan Revoke
GitHubTeam Orphan Delete
GitHubTeamMembership Orphan Revoke
GitHubRepositoryRuleset Orphan Delete
GitHubRepositoryTeamAccess Orphan Revoke
GitHubRepositoryCollaborator Orphan Revoke
GitHubEnvironment Orphan Delete
GitHubActionsSecret Orphan Revoke
GitHubActionsVariable Orphan Revoke

Review the manifest before changing a deletion policy. Kubernetes finalizers keep the custom resource present until the requested GitHub cleanup succeeds.

Suspend and resume a provider

Suspend reconciliation before credential maintenance or a planned GitHub change:

kubectl patch ghprovider default \
  --type merge \
  -p '{"spec":{"suspended":true}}'

Resources that use the provider report:

Ready=False
Reason=ReconciliationSuspended

No GitHub API requests or credential reads are made while the provider is suspended. Remote Delete and Revoke finalizers wait for the provider to be resumed. Orphan resources can still be removed because they require no remote operation.

Resume reconciliation:

kubectl patch ghprovider default \
  --type merge \
  -p '{"spec":{"suspended":false}}'

GitHub API rate limits

All controllers share one reactive rate-limit gate inside the manager process. When GitHub returns a primary, secondary or abuse rate-limit response, the operator honors Retry-After or the GitHub reset time and requeues affected resources without producing a continuous error storm.

A normal authorization or feature-plan 403 is not treated as a rate limit. Inspect the response message in the resource condition or controller logs.

Metrics and Grafana

The manager exposes the standard controller-runtime, Go runtime and process metrics together with a small set of GitHub-specific Prometheus metrics:

Metric Purpose
github_platform_operator_github_api_requests_total GitHub API responses by HTTP method and status code
github_platform_operator_github_api_request_duration_seconds GitHub API HTTP round-trip latency histogram by method
github_platform_operator_github_api_transport_errors_total Requests that failed before an HTTP response was received
github_platform_operator_github_rate_limit_limit Last observed GitHub rate-limit ceiling by resource
github_platform_operator_github_rate_limit_remaining Last observed remaining requests by rate-limit resource
github_platform_operator_github_rate_limit_reset_timestamp_seconds Last observed rate-limit reset timestamp
github_platform_operator_github_rate_limit_events_total Detected primary, secondary or unknown rate-limit events
github_platform_operator_github_rate_limit_blocked_until_timestamp_seconds Timestamp until which the shared reactive rate-limit gate is blocked

The metrics are instrumented in the shared GitHub HTTP transport, so the same measurements cover repository, organization, team, ruleset, Actions and GitHub App token requests without duplicating instrumentation in every controller. Rate-limit gauges represent the last response observed by the shared HTTP client; they are intentionally not labeled by provider, organization, repository or credential in order to avoid sensitive labels and unbounded cardinality.

Prometheus must scrape the manager metrics Service. The Kustomize deployment contains a ServiceMonitor example under config/prometheus. With secure metrics, the scraping service account must also be authorized for the non-resource /metrics URL.

An importable Grafana dashboard is included at:

dashboards/grafana/github-platform-operator.json

The dashboard combines these GitHub-specific metrics with existing metrics such as controller_runtime_reconcile_total, controller_runtime_reconcile_time_seconds, workqueue_depth, process_cpu_seconds_total, process_resident_memory_bytes and go_goroutines.

Secret and variable rotation

Update the referenced Kubernetes Secret:

kubectl patch secret payments-actions-values \
  --namespace default \
  --type merge \
  -p '{"stringData":{"region":"eu-central-1"}}'

The secret watch enqueues only the Actions resources that reference the changed Secret. The status records the source Secret UID and resource version without recording its value.

Helm configuration

Show all values:

helm show values \
  oci://ghcr.io/pierinho13/charts/github-platform-operator \
  --version 0.1.0

Example override:

replicaCount: 2

resources:
  requests:
    cpu: 50m
    memory: 96Mi
  limits:
    cpu: 500m
    memory: 256Mi

metrics:
  enabled: true
  secure: true
helm upgrade --install github-platform-operator \
  oci://ghcr.io/pierinho13/charts/github-platform-operator \
  --version 0.1.0 \
  --namespace github-platform-operator-system \
  --create-namespace \
  --values values.yaml

Upgrade the operator

Helm installs files under the chart's crds/ directory only during the first installation. Helm does not upgrade or delete those CRDs automatically.

For a release that changes CRD schemas:

  1. Download or check out the target release.
  2. Apply the target CRDs.
  3. Upgrade the chart.
kubectl apply -f config/crd/bases/

helm upgrade github-platform-operator \
  oci://ghcr.io/pierinho13/charts/github-platform-operator \
  --version <target-version> \
  --namespace github-platform-operator-system

Review release notes before upgrading between v1alpha1 releases.

Uninstall

helm uninstall github-platform-operator \
  --namespace github-platform-operator-system

The operator Deployment and RBAC are removed. Helm intentionally leaves CRDs and existing custom resources in the cluster.

Before deleting CRDs, inspect all finalizers and remote deletion policies:

kubectl get \
  ghorgmember,ghteam,ghteammember,ghrepo,ghruleset,ghteamaccess,ghcollab,ghenv,ghsecret,ghvar \
  -A

Troubleshooting

Resource remains without READY

Confirm that the controller is running:

kubectl get pods -n github-platform-operator-system
kubectl logs \
  deployment/github-platform-operator \
  -n github-platform-operator-system

DependencyUnavailable

Check referenced resources and namespaces:

kubectl get ghprovider
kubectl get ghorgmember,ghteam,ghteammember,ghrepo,ghenv -A
kubectl get secret -A

GitHub returns 403

The token or GitHub App installation may not have enough permission, the organization policy may block the operation, or the selected GitHub plan may not support that feature.

For rulesets, a response such as the following indicates a GitHub plan restriction rather than an operator authentication failure:

Upgrade to GitHub Pro or make this repository public to enable this feature.

BypassActorUnavailable

A teamSlug or username could not be resolved. Confirm the identifier and the provider organization. Team resolution also requires organization Members: read permission; classic personal access tokens need read:org. The condition message includes the failed actor and the GitHub API error.

Ruleset returns 422

Inspect the condition message and the exact rule parameters. Ref-name conditions must contain at least one include entry. Empty exclusions are supported and are sent as an empty JSON array.

Provider is suspended

kubectl get ghprovider default -o yaml

Set spec.suspended back to false when remote reconciliation should resume.

Organization or team membership remains pending

The GitHub user must accept the organization invitation. The organization and team membership controllers will observe the active membership during a later reconciliation.

Collaborator remains pending

The invited GitHub user must accept the repository invitation. The controller will observe the accepted access during a later reconciliation.