Skip to content

Record final observability closeout - #23

Merged
natnaela-devops merged 39 commits into
mainfrom
final-observability-closeout
Oct 2, 2026
Merged

natnaela-devops merged 39 commits into
mainfrom
final-observability-closeout

Conversation

@natnaela-devops

@natnaela-devops natnaela-devops commented Oct 2, 2026 •

Copy link
Copy Markdown
Owner

Summary

Final sanitized closeout synchronization for the validated OpenSearch observability platform.

This PR records:

  • final Prometheus rule naming and the live 16-alert / 0-recording-rule baseline;
  • final Alertmanager receiver/template naming, 30s / 5m / 4h timing, four warning-to-critical inhibition rules, and grouped Telegram presentation;
  • the live Data Prepper -> Prometheus span_derived RED metric path, with recording/reporting rules optional rather than required;
  • RKE2 OpenTelemetry Collector/Operator onboarding and Java instrumentation guardrails;
  • reusable Linux host node_exporter onboarding with pinned version/checksum, backup, rollback, and validation workflow;
  • final log/APM naming and separation standards;
  • the six-dashboard sanitized saved-object reference with only actually referenced Explore objects;
  • backup-first change control and tested automatic rollback evidence.

Dashboard reference

The public dashboard reference contains:

  • 6 dashboards;
  • 74 dashboard-referenced Explore panels;
  • 4 data views;
  • 2 saved searches.

The live Prometheus connection is replaced by reference-prometheus. Workspace metadata, private host identity, private addresses, deployment-specific application namespaces, and transient saved-object metadata are removed. A dedicated CI validator checks counts, dependency integrity, generic namespace/environment/node mappings, documentation-only IPs, opaque UUID absence, and secret-like material.

Safety / rollback

  • backup-first workflow used for every live mutation;
  • previous Prometheus and Alertmanager configuration preserved before changes;
  • previous OpenSearch Dashboards saved objects preserved before UI naming changes;
  • rollback helpers generated and retained;
  • failed backend-standardization attempts automatically restored the prior configuration;
  • post-rollback verification confirmed all 16 Prometheus rules healthy;
  • no destructive live change was accepted without validation.

Validation

  • Prometheus staged/live rule validation passed;
  • 16 alert names, PromQL expressions, durations, labels, and annotations were preserved during naming standardization;
  • Alertmanager configuration validation passed with four inhibition rules;
  • final rule evaluation: 16 alerting / 0 recording / 16 healthy;
  • dashboard reference dependency validation passed;
  • final PR privacy scan found no known live customer/environment identifiers in added content;
  • GitHub Repository validation workflow is green.

Public-repository hygiene

No production credentials, Telegram bot token, real chat ID, private certificate/key, kubeconfig, private workspace ID, live host address, live hostname, trace payload, or transaction payload is intentionally committed.

@natnaela-devops
natnaela-devops marked this pull request as ready for review October 2, 2026 17:40
@natnaela-devops
natnaela-devops merged commit 4a940f6 into main Oct 2, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant