AIOps-Enabled Closed-Loop SRE Platform
Prometheus telemetry, dependency-aware diagnosis, grounded AI evidence,
deterministic remediation policy, and post-action recovery verification.
Architecture • Verified evidence • Run locally • Demo guide • Safety model
CloudMind is a controlled AIOps laboratory built around five Dockerized Flask services. It observes the system, correlates service and dependency signals, proposes a bounded response, and records every decision.
| 1 · Observe | 2 · Diagnose | 3 · Govern | 4 · Verify |
|---|---|---|---|
| Prometheus captures service and dependency health. | Gemini structured output or deterministic rules identify the probable cause. | Grounded evidence, allowlists, cooldowns, budgets, and circuit breakers decide what is safe. | Prometheus and active probes determine whether the service recovered. |
The LLM never controls Docker. It provides advisory diagnosis only. Local deterministic policy owns every execution decision.
Operator view Live service health, controlled stress actions, incident dialogue, and persisted AIOps decisions. |
Policy observability Live InfraMirror policy-decision metrics rendered through Prometheus and Grafana. |
The screenshots above are authentic captures from the local Docker Compose runtime. See the live recommend-mode validation record.
flowchart TB
subgraph OBSERVE["1 · OBSERVE"]
direction LR
S["Five Flask services"] --> P["Prometheus"]
P --> A["Alertmanager"]
end
subgraph UNDERSTAND["2 · UNDERSTAND"]
direction LR
A --> I["InfraMirror snapshot"]
I --> D["Gemini schema<br/>or rules fallback"]
D --> G["Evidence grounding"]
end
subgraph GOVERN["3 · GOVERN"]
direction LR
G --> E["Deterministic<br/>evidence score"]
E --> Q{"Policy decision"}
Q -->|Recommend| AUDIT["Persist for operator"]
Q -->|Execute| GUARDS["Grace · cooldown · lease<br/>budget · circuit breaker"]
end
subgraph RECOVER["4 · ACT AND VERIFY"]
direction LR
GUARDS --> R["Allowlisted restart"]
R --> V["Recovery verification"]
V --> AUDIT
AUDIT --> O["Dashboard · Grafana · audit trail"]
end
| Layer | Responsibility | Trust boundary |
|---|---|---|
| Services | Produce workload, health, error, latency, and dependency signals | Never receive model-generated commands |
| Prometheus + Alertmanager | Observe and route authenticated alerts | Telemetry is treated as bounded input |
| InfraMirror diagnosis | Correlate the snapshot and propose a structured cause | Gemini is optional and advisory |
| Evidence grounding | Replace model numeric claims with snapshot truth | Ungrounded evidence cannot approve execution |
| Policy + guards | Score evidence and enforce allowlists, leases, budgets, and circuits | Sole authority for remediation |
| Recovery + audit | Probe health and persist the complete decision | No recovery claim without observed evidence |
Read docs/architecture.md for the component map, data contracts, and Docker trust boundary.
| Capability | CloudMind implementation |
|---|---|
| Telemetry correlation | CPU, latency, request rate, error ratio, availability, alerts, incidents, and dependency health |
| Root-cause analysis | Dependency-aware diagnosis across API, database, cache, auth, and frontend |
| AI-assisted operations | Schema-constrained Gemini diagnosis with deterministic fallback |
| Evidence integrity | Model-selected signals are resolved against the immutable telemetry snapshot |
| Governed action | Model confidence is separated from the deterministic policy evidence score |
| Closed-loop verification | Recovery requires Prometheus health plus active dependency probes |
| Operational auditability | Diagnosis, evidence, decision, execution, guard state, and recovery are persisted |
CloudMind does not claim learned anomaly detection, formal causal inference, self-learning, or autonomous production operations.
recommend mode by default
+
allowlisted action and target
+
grounded target-consistent evidence
+
deterministic evidence score
+
startup grace + cooldown + per-target lease
+
restart budget + recovery circuit breaker
=
one governed remediation decision
- Safe defaults are
AIOPS_EXECUTION_MODE=recommendandHEALING_ENABLED=false. - Execute mode requires both
AIOPS_EXECUTION_MODE=executeandHEALING_ENABLED=true. - The only supported actions are
restart_serviceandno_action. - Only
frontend,api,database,cache, andauthcan be targeted. - A single weak signal cannot authorize a restart.
- Gemini keys are sent in the
x-goog-api-keyheader, never in a URL. - Per-target restart budgets and circuit breakers prevent remediation loops.
- Docker socket access is privileged; CloudMind is intended for an operator-owned local environment.
Read the full safety model and security policy.
| Evidence | Verified result |
|---|---|
| Automated tests | 224 passed |
| Safety-critical branch coverage | 80.17% |
| Deterministic validation scenarios | 10 |
| Fixture root-cause accuracy | 100% |
| Fixture recommendation accuracy | 100% |
| Transient no-action accuracy | 100% |
| Unsafe actions executed | 0 |
| Live recommend-mode scenarios | 5 passed |
| Prometheus scrape targets | 6 up |
| Live Gemini requests used for validation | 0 |
| External API cost | $0 |
Accuracy percentages describe deterministic fixtures—not production accuracy or provider reliability. Execute-mode recovery rate and MTTR remain unmeasured.
- Machine-readable evidence: artifacts/aiops-validation-results.json
- Generated report: docs/validation-results.md
- Live recommend-mode record: docs/live-validation-results.md
- Full hardening report: CLOUDMIND_AIOPS_HARDENING_REPORT.md
- A service exposes operational and dependency telemetry.
- Prometheus evaluates the signals and Alertmanager sends an authenticated event.
- InfraMirror captures a bounded snapshot and deterministic incident fingerprint.
- Gemini returns schema-constrained advice, or rules produce a deterministic fallback.
- Evidence grounding rejects invented signals and replaces numeric claims with observed values.
- Policy computes an evidence score independent of model confidence.
- Recommend mode records the decision without changing a container.
- Execute mode rechecks grace, cooldown, lease, restart budget, and circuit state.
- One allowlisted restart may occur, followed by recovery verification.
- The complete trail is persisted and exported as bounded-label metrics.
- Docker Desktop with Docker Compose
- Python 3.12+
git clone https://github.com/Mukeshkr-19/CLOUDMIND.git
cd CLOUDMIND
cp .env.example .envSet unique local values for WHISPER_TOKEN and GRAFANA_ADMIN_PASSWORD. Gemini and Discord are optional; leave GEMINI_API_KEY and DISCORD_WEBHOOK_URL blank to use deterministic/local fallbacks.
docker compose config --quiet
docker compose up -d --build
docker compose ps| Open | Local URL |
|---|---|
| CloudMind operator dashboard | http://127.0.0.1:5050 |
| InfraMirror metrics | http://127.0.0.1:5055/metrics |
| Prometheus | http://127.0.0.1:9090 |
| Alertmanager | http://127.0.0.1:9093 |
| Grafana | http://127.0.0.1:3000 |
Stop the environment with docker compose down.
make dev-setup
make verify
make lint
make type-check
make coverage
make validation-reportRun the controlled live scenarios in safe recommend mode:
venv/bin/python scripts/run_aiops_scenarios.py all \
--expect-mode recommend \
--requests 10 \
--incident-timeout 30No Gemini request or container restart is required. See the demo guide before enabling any execute-mode behavior.
| Variable | Safe default | Purpose |
|---|---|---|
GEMINI_API_KEY |
blank | Optional provider key |
GEMINI_MODEL |
gemini-3.6-flash |
Configurable diagnosis model |
GEMINI_API_VERSION |
v1beta |
Centralized REST API version |
AIOPS_EXECUTION_MODE |
recommend |
Records decisions without execution |
HEALING_ENABLED |
false |
Second explicit gate for container changes |
AIOPS_CONFIDENCE_THRESHOLD |
0.75 |
Advisory model-confidence floor |
AIOPS_EVIDENCE_SCORE_THRESHOLD |
0.55 |
Deterministic evidence floor |
AIOPS_MAX_RESTARTS_PER_SERVICE_PER_HOUR |
3 |
Rolling per-target restart budget |
AIOPS_MAX_FAILED_RECOVERIES |
2 |
Failed recoveries before circuit opening |
AIOPS_CIRCUIT_BREAKER_RESET_SEC |
900 |
Automatic circuit reset interval |
AIOPS_EXECUTION_GRACE_SEC |
30 |
Startup period forced to recommend mode |
See .env.example for all bounded tuning options.
| Path | Role |
|---|---|
microservices/ |
Five Flask services and operational signals |
inframirror/gemini_client.py |
Secure structured-output client and retry boundary |
inframirror/incident_intelligence.py |
Provider diagnosis and dependency-aware rules |
inframirror/evidence_grounding.py |
Snapshot-backed evidence authority |
inframirror/policy_engine.py |
Deterministic policy evidence assessment |
inframirror/remediation_guard.py |
Restart budgets and circuit breakers |
inframirror/recovery_verifier.py |
Post-action health verification |
inframirror/incident_store.py |
Atomic bounded incident audit store |
inframirror/aiops_metrics.py |
Bounded-label AIOps Prometheus metrics |
scripts/ |
Live scenarios and deterministic validation matrix |
grafana/provisioning/ |
Service and AIOps dashboards |
.github/workflows/ |
SHA-pinned CI, security, and CodeQL |
CloudMind is deliberately optimized as an operator-owned Docker Compose reference implementation: small enough to run locally, but complete enough to demonstrate the full observe → diagnose → govern → remediate → verify loop.
| Engineering focus | Current design boundary |
|---|---|
| Dependency-aware operational reasoning | Correlates real telemetry and service topology without presenting heuristics as formal causal inference |
| Safe AI-assisted diagnosis | Gemini is optional, schema-constrained, and backed by deterministic rules |
| Governed closed-loop remediation | Local policy, budgets, leases, and circuit breakers retain execution authority |
| Reproducible incident laboratory | Five Dockerized services provide controlled failure and recovery scenarios |
Deployment boundary: InfraMirror's Docker socket access is intentionally limited to an operator-owned lab. A broader deployment should replace it with a narrower, authenticated remediation adapter.
Detailed constraints and threat boundaries live in the safety model and architecture documentation, keeping this overview focused on the working system.
- Expand meaningful safety-path coverage beyond the current 80.17% gate.
- Publish isolated execute-mode recovery benchmarks.
- Introduce a narrower remediation adapter behind the existing policy interface.
- Add a concise end-to-end demonstration video.
CloudMind gives infrastructure components distinct voices so an incident can be followed like a conversation. That personality layer makes the demo memorable, while telemetry, evidence grounding, policy, execution guards, and recovery verification remain deterministic and auditable.
CloudMind is available under the MIT License.