Record final observability closeout

.github/workflows/validate.yml

@@ -22,6 +22,10 @@ jobs:
             bash -n "$script"
           done < <(find scripts -maxdepth 1 -type f -name '*.sh' | sort)
 
+      - name: Dashboard reference validation
+        shell: bash
+        run: python3 scripts/validate-dashboard-reference.py
+
       - name: Version invariants
         shell: bash
         run: |

README.md

@@ -13,7 +13,7 @@ This implementation replaces the repository's former SigNoz/Grafana experiment.
 | Kubernetes | OpenTelemetry Collector Contrib | 0.156.0 | OTLP gateway, Kubernetes enrichment, and signal routing |
 | Kubernetes | Fluent Bit | 4.1.0 | Container stdout collection for workloads not exporting OTLP logs |
 | Processing | Data Prepper | 2.16.0 | Trace/log ingestion and v2 service-map generation |
-| Metrics | Prometheus | 3.14.0 | Metrics storage, RED rules, SLO calculations, and reporting metrics |
+| Metrics | Prometheus | 3.14.0 | Metrics storage, remote-write ingestion, alert evaluation, and optional derived/reporting rules |
 | Alerting | Alertmanager | 0.31.1 | Alert routing and notification delivery |
 | Host metrics | node_exporter | 1.12.1 | Linux host CPU, memory, disk, filesystem, network, load, and uptime metrics |
 | Host processes | process-exporter | 0.8.7 | Selected process-level metrics on the observability host |
@@ -69,7 +69,7 @@ Every Java container must run exactly one OpenTelemetry Java agent.
 
 Applications exporting logs directly through OTLP use `fluentbit.io/exclude: "true"` to prevent Fluent Bit from sending an uncorrelated duplicate.
 
-See [Kubernetes Telemetry Onboarding](docs/05-kubernetes-telemetry-onboarding.md) for both manifests, the double-agent guardrail, validation, and rollback.
+See [Kubernetes Telemetry Onboarding](docs/05-kubernetes-telemetry-onboarding.md) for the in-cluster OpenTelemetry Collector/Operator manifests, the double-agent guardrail, validation, and rollback. Linux host metrics are onboarded separately with [node_exporter](docs/16-linux-host-metrics-onboarding.md).
 
 ## Repository layout
 
@@ -92,15 +92,16 @@ See [Kubernetes Telemetry Onboarding](docs/05-kubernetes-telemetry-onboarding.md
 │   ├── 12-developer-search-standardization.md
 │   ├── 13-explore-log-datasets.md
 │   ├── 14-alerting-standard.md
-│   └── 15-ubuntu-2404-cleanroom-qualification.md
+│   ├── 15-ubuntu-2404-cleanroom-qualification.md
+│   └── 16-linux-host-metrics-onboarding.md
 ├── evidence/                           # Evidence policy; generated output stays local
 ├── kubernetes/
 │   ├── collector/                       # Collector, Service, RBAC, and configuration
 │   ├── examples/                        # Mutually exclusive Java onboarding modes
 │   ├── instrumentation/                 # Shared opt-in Instrumentation resource
 │   └── operator/                        # Pinned Operator Helm values
 ├── prometheus/
-│   └── rules/                           # Platform alerts, RED, SLO, and report rules
+│   └── rules/                           # Standardized alerts plus optional recording/reporting rules
 ├── scripts/
 │   └── validate-platform.sh             # Read-only end-to-end acceptance checks
 ├── prometheus.yml                       # Remote-write receiver and local scrape baseline
@@ -118,7 +119,7 @@ The functional lab has demonstrated:
 - a gateway-to-payment distributed trace and v2 service-map relationship;
 - trace- and span-correlated application logs;
 - duplicate-log prevention between OTLP and Fluent Bit;
-- automatic Prometheus reporting discovery for newly instrumented services;
+- automatic service-derived RED metric discovery for newly instrumented services;
 - request, 4xx, 5xx, latency, availability, and error-budget calculations;
 - a fresh Ubuntu 24.04.5 LTS dedicated-host clean-room bootstrap, secured activation, and five-minute runtime soak with all seven observability services healthy, all configured Prometheus targets UP, and zero unhealthy Prometheus rules.
 
@@ -143,6 +144,7 @@ Read the repository in this order:
 13. [Explore Logs Dataset Separation](docs/13-explore-log-datasets.md)
 14. [Alerting Standard](docs/14-alerting-standard.md)
 15. [Ubuntu 24.04 Clean-Room Qualification](docs/15-ubuntu-2404-cleanroom-qualification.md)
+16. [Linux Host Metrics Onboarding](docs/16-linux-host-metrics-onboarding.md)
 
 ## Lab-only OpenSearch start
 

config/reference.env.example

@@ -28,3 +28,8 @@ ALERTMANAGER_TELEGRAM_CHAT_ID=123456789
 ALERTMANAGER_TELEGRAM_TOKEN_FILE=/etc/alertmanager/secrets/telegram_bot_token
 
 KUBELET_BEARER_TOKEN_FILE=/run/secrets/kubelet_bearer_token
+
+# Live UAT receives span_derived RED metrics directly from Data Prepper.
+# Enable only for environments that intentionally use the optional Prometheus
+# recording/reporting-rule path.
+ENABLE_APM_RECORDING_RULES=false

dashboards/README.md

@@ -1,15 +1,56 @@
-# OpenSearch Dashboards exports
+# OpenSearch Dashboards reference exports
 
-This directory is reserved for sanitized OpenSearch Dashboards saved-object exports created from the validated observability workspace.
+This directory contains the sanitized saved-object reference captured from the final validated observability workspace.
 
 The former Grafana sample was removed because Grafana is not part of the current platform.
 
-Before committing an export:
+## Final dashboard set
 
-1. remove data-source identifiers that are environment-specific;
-2. remove internal addresses, hostnames, usernames, and credentials;
-3. remove customer, account, transaction, trace, span, pod, and node identifiers;
-4. confirm that the object imports into a clean lab workspace;
-5. record its required index patterns and minimum OpenSearch Dashboards version.
+The reference contains six dashboards:
 
-Planned exports include the Kubernetes platform overview, application SLO report, service health overview, and ingestion pipeline health.
+1. Infrastructure Health
+2. Deployment & Workload Health
+3. Application Performance
+4. Developer Investigation
+5. Real-Time Channel Performance
+6. Kubernetes Platform Health
+
+The complete sanitized dependency set is under [reference/](reference/):
+
+- `dashboards.json` — 6 dashboard objects;
+- `explore-panels.json` — 74 Explore objects actually referenced by the six dashboards;
+- `data-views.json` — 4 standardized log/APM data views;
+- `saved-searches.json` — 2 saved searches used by Developer Investigation;
+- `SANITIZATION.md` — mapping and import requirements;
+- `SHA256SUMS` — checksums for the captured sanitized reference files.
+
+Unreferenced legacy Explore objects from the live workspace are intentionally excluded.
+
+## Privacy and sanitization
+
+The public reference does not retain the live workspace ID, customer identity, private hostnames, private addresses, deployment-specific application namespaces, credentials, tokens, trace IDs, transaction payloads, or the live Prometheus connection identity.
+
+The Prometheus source is represented by the placeholder `reference-prometheus`. Host identities use documentation labels and TEST-NET-1 addresses.
+
+Run:
+
+```bash
+python3 scripts/validate-dashboard-reference.py
+```
+
+to verify object counts, reference integrity, the intentional Prometheus placeholder, and the private-identity guardrails.
+
+## Import / adaptation
+
+These files are sanitized reference collections, not a blind production restore. Before using them in another environment:
+
+1. create/select the target Prometheus data connection;
+2. map `reference-prometheus` to that connection identifier;
+3. replace the `reference` telemetry environment label;
+4. adapt the generic application/channel namespace selectors;
+5. map the documentation host inventory to the target hosts;
+6. import/restore data views and saved searches before dashboards;
+7. refresh data-view fields;
+8. validate every query and visualization against the target telemetry.
+
+See [reference/SANITIZATION.md](reference/SANITIZATION.md) for the exact mapping contract.

dashboards/reference/SANITIZATION.md

@@ -0,0 +1,49 @@
+# Sanitized dashboard reference
+
+This directory is generated from the final operated saved-object snapshot and is safe for public reference after the checks described below.
+
+## Included objects
+
+- 6 dashboards
+- 74 dashboard-referenced Explore panels
+- 4 data views
+- 2 saved searches used by Developer Investigation
+
+Unreferenced legacy Explore objects from the live workspace are intentionally excluded.
+
+## Sanitization map
+
+Private deployment identity is replaced before publication:
+
+- Prometheus source identity -> `reference-prometheus`;
+- workspace metadata -> removed;
+- environment label -> `reference`;
+- application workload namespace selector -> `^application-.*$`;
+- channel deployment namespaces -> `channel-mobile` and `channel-ussd`;
+- host names -> `host-01`, `host-02`, `host-03`, and `observability-host-01`;
+- internal host addresses -> TEST-NET-1 documentation addresses;
+- customer/environment display text -> generic reference terminology;
+- transient saved-object metadata (`updated_at`, `version`, workspace/namespace/score fields) -> removed;
+- data-view field caches -> removed so fields can be refreshed after import.
+
+The public artifacts retain the generic Mobile and USSD channel concepts because they are part of the dashboard design, but deployment-specific namespace and host identity is removed.
+
+## Prometheus source mapping
+
+The operated workspace did not contain a normal `index-pattern` saved object with the live Prometheus connection name. Instead, metric Explore objects referenced the Prometheus `data-connection` connection identifier through the saved-object reference field.
+
+For that reason `reference-prometheus` is intentionally a placeholder, not a complete Prometheus connection definition. Before import/rendering, map it to the target OpenSearch Dashboards Prometheus connection identifier.
+
+## Import notes
+
+These files are sanitized reference collections rather than raw live exports. Before importing into another environment:
+
+1. create or select the target Prometheus data connection;
+2. replace `reference-prometheus` with that connection identifier;
+3. replace `environment="reference"` with the target telemetry environment;
+4. adapt `^application-.*$`, `channel-mobile`, and `channel-ussd` selectors to the target namespace policy;
+5. map the documentation host identities/addresses to the target host inventory;
+6. import data views and saved searches before dashboards;
+7. refresh data-view fields and validate every dashboard panel.
+
+No raw workspace ID, private IP address, customer hostname, customer name, credential, token, trace ID, or transaction payload is retained in these files.

dashboards/reference/SHA256SUMS

@@ -0,0 +1,5 @@
+6503112ee4c81673665fd32c16d0eb847524ae35cbe90dce773a978ff190de55  dashboards.json
+7133d12f7788df621d5d23640720de8a8fc00b282bf8751f1e0c233c5f9aacab  explore-panels.json
+038c2dbf3020036a94e4e380a07fdf97024d8598f0dc0f0e7127107d9d545137  data-views.json
+fe91d451c24584cdb21784a3961eff71ed7650407875e66674467e6a3001590c  saved-searches.json
+53ad361042085871a8687af9a5c2d482913a2245e0a3c5116efc81feb5585eff  SANITIZATION.md