Repository navigation
Phase F foundation: OpenSearch security, independent alerting, snapshots - #11
Merged
Merged
Conversation
- TLS everywhere on the main cluster: transport + HTTP, certs from new scripts/07-generate-opensearch-certs.sh (root CA + node/admin certs, RFC2253 DNs matched in node configs; guards against docker-created bind-mount directories) - users/roles: admin, logstash writer (scoped to the six SIEM index patterns + templates), readonly (Grafana + exporter), kibanaserver; rendered + applied by scripts/08b-init-opensearch-security.sh (idempotent securityadmin flow, verifies auth-200 and anon-401) - every client updated: Logstash outputs (https, logstash user, CA verification), Grafana datasources (readonly basicAuth via env interpolation), opensearch-dashboards (security plugin kept on, kibanaserver service user), all scripts' curl calls, ISM/smoketest/ ingest-test, change-passwords rotation for the four new passwords - prometheus: dead /_prometheus scrape jobs removed (plugin not shipped in the stock image) — metrics path moves to the exporter in the next commit; direct-scrape recipe kept as a comment - InfluxDB auth enabled with self-healing bootstrap (CREATE USER exception on 1.8); Grafana + unifi-poller wired with credentials - healthchecks accept 401 (auth working) and never carry the admin password into container env; deploy rsync protects server-side generated certs from --delete - note: this commit's compose also carries mounts/services wiring for the alerting and snapshot commits that follow on this branch Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015XBQ5bbaaoKreyoWivRwVQ
- alertmanager service (pinned, loopback-only API) with a route tree by severity and an n8n webhook receiver; tracked template + envsubst render script, default rendering works out of the box - elasticsearch-exporter service (works against OpenSearch) as the sole authenticated metrics path into the secured cluster — no credentials in prometheus.yml - prometheus rule_files + 8 SIEM health rules: cluster red/yellow, ingest-rate-zero (a silent pipeline must page, not look quiet), target down, container restart churn, filesystem filling, rule-evaluation failures (promtool: SUCCESS) - CI now also runs promtool check rules and amtool check-config - new tests/python/test_yaml_configs.py: rules schema/severity invariants, route/receiver consistency, datasource https+auth invariants, node security config invariants (68 tests total) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015XBQ5bbaaoKreyoWivRwVQ
- path.repo on both nodes backed by /data/warm/snapshots; new scripts/10-snapshot-setup.sh registers the fs repository and a daily Snapshot Management policy (retain 14) over all six SIEM index patterns, via the secured endpoints - docs/backup-restore.md: manual snapshots, the scheduled policy, a restore drill (restore-to-renamed-index, verify, delete), offsite copy guidance; Wazuh indexer snapshots flagged as follow-up - docs/phase-f-testing.md: ordered per-cycle checklist for live-machine testing — .env additions, cert generation, live-deployment migration order (Influx admin before auth flip, securityadmin before client restarts), fresh-install bring-up, authenticated/anonymous curl verification, exporter metrics, forced test alert, snapshot+restore drill, and a full rollback recipe - README service table + game-plan Phase F statuses updated Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015XBQ5bbaaoKreyoWivRwVQ
ChiefGyk3D
marked this pull request as ready for review
September 2, 2026 22:42
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The first three foundation items from docs/game-plan.md Phase F, built as three commits so they can be tested and reverted independently. This branch is for live-machine test cycles — start with
docs/phase-f-testing.md, which has the.envadditions, the migration order for your live deployment, verification commands, and a rollback recipe.F1 — The SIEM can now defend itself
scripts/07-generate-opensearch-certs.sh), users/roles applied idempotently byscripts/08b-init-opensearch-security.sh(admin / scopedlogstashwriter /readonlyfor Grafana+exporter /kibanaserver)change-passwords.shF2 note
Doppler (issue #4) intentionally not in this branch — the new
OPENSEARCH_*env vars are designed to drop into the Doppler migration unchanged.F3 — Alerting that survives Grafana dying
rule_fileswith 8 health rules: cluster red/yellow, ingest-rate-zero (a silent pipeline pages instead of looking quiet), target down, container restart churn, filesystem filling, rule-eval failures/_prometheusscrape jobs (plugin never shipped in the stock image) are removedF4 — A disk failure is no longer total evidence loss
path.repoon the warm tier, daily Snapshot Management policy (retain 14) over all six SIEM index patterns, and a documented restore drill indocs/backup-restore.mdValidated statically (all green)
compose renders without .env;
promtool check config+check rulesSUCCESS; shellcheck error-severity clean; 68 pytest tests (new YAML-invariant suite covering rules schema, route/receiver consistency, datasource https+auth, node security config); cert chain verified end-to-end locally (SANs, RFC2253 DNs match node configs, PKCS#8 keys).Needs live-machine verification (the test cycles)
securityadmin.sh/hash.shinvocation against a real OpenSearch 2.19 containerssl,ssl_certificate_verification,cacert) accepted at boot (--config.test_and_exitruns in CI but the full handshake needs the cluster)OPENSEARCH_USERNAME/PASSWORD/SSL_VERIFICATIONMODE) on the 2.19 image_plugins/_sm/policies) on 2.19readonlyrole (may need a one-time permission widening)amtool check-configand the Logstash config test run in CI (no docker daemon in the build environment)Known follow-ups after F1 lands:
docs/maintenance.mdstill shows pre-security example commands; Wazuh-indexer snapshots.🤖 Generated with Claude Code
https://claude.ai/code/session_015XBQ5bbaaoKreyoWivRwVQ
Generated by Claude Code