Skip to content

feat: log aggregation and conditional alert rules - #11

Merged
btcneves merged 1 commit into
mainfrom
feat/log-aggregation
May 21, 2026
Merged

btcneves merged 1 commit into
mainfrom
feat/log-aggregation

Conversation

@btcneves

@btcneves btcneves commented May 21, 2026 •

Copy link
Copy Markdown
Owner

Objective

Adds two independent features to AutoFlowOps:

  1. Log aggregation — structured log shipping to Loki and Elasticsearch with a consistent label schema across all service components.
  2. Conditional alert rules — per-job rules that trigger Alerts based on configurable conditions evaluated after every execution.

Changes

Log Aggregation

  • Structured JSON logging via structlog with context propagation through contextvars
  • Loki and Elasticsearch log shipping; configurable via environment variables (LOG_SINK, LOKI_URL, ELASTICSEARCH_URL)
  • Consistent label schema: service, env, job_id, execution_id, workspace_id
  • docker-compose.observability.yml with Loki + Promtail stack for local development
  • Grafana provisioning datasource for Loki auto-configured
  • docs/log-aggregation.md: label schema, sink configuration, local setup and production guidance
  • Deployment docs updated with observability environment variables

Conditional Alert Rules

  • AlertRule model with CASCADE FK to jobs, workspace isolation via SET NULL FK to workspaces
  • Alembic migration c3d4e5f6a7b8 creates the alert_rules table with index on job_id
  • Four condition types: http_status_gte, duration_ms_gte, response_body_contains, consecutive_failures_gte
  • CRUD API under GET|POST /api/jobs/{job_id}/alert-rules and PATCH|DELETE /api/jobs/{job_id}/alert-rules/{rule_id}
  • RBAC: viewers read-only; operators and admins manage rules (enforced via require_operator)
  • evaluate_alert_rules() in http_runner.py; Celery worker wired post-retry to avoid premature alerts during retrying state
  • React section in JobDetailPage with create form, inline toggle and delete; operators-only write actions
  • Client-side validation mirrors backend constraints (integer thresholds, HTTP status range 100–599)
  • i18n in English and Portuguese-Brazilian
  • Audit logging on all mutations via log_action()
  • API reference documentation updated with all four endpoints and curl examples

Test Plan

  • uv run pytest — 260 backend tests pass (7 new alert rule tests, 3 new notification tests)
  • npm test in frontend/ — 76 frontend tests pass (1 new JobDetailPage test)
  • npx tsc --noEmit in frontend/ — no TypeScript errors
  • uv run ruff check backend/ — no linting errors
  • Alembic migration applies cleanly on a fresh database: alembic upgrade head
  • Log aggregation: set LOG_SINK=loki and LOKI_URL=http://localhost:3100 — verify structured logs reach Loki
  • Alert rule CRUD via API: create a rule, trigger an execution, verify Alert is created
  • Workspace isolation: confirm rules are not accessible across workspace headers

Notes

  • Log aggregation is additive and backwards-compatible; the default sink is stdout (existing behavior)
  • Alert rules are evaluated after every execution regardless of execution status, enabling both failure-based and performance-based alerting
  • consecutive_failures_gte queries the last 50 completed executions to determine the streak
  • Frontend bundle includes a pre-existing chunk size warning (803 kB) unrelated to this PR

Security Checklist

  • No real secrets, tokens, passwords or API keys in any committed file
  • .env.production is not tracked
  • All credentials masked in API responses, UI and delivery records
  • No sensitive configuration details exposed in public documentation
  • No git push --force, no history rewrite, no --no-verify bypass

Introduces per-job AlertRule configuration with four condition types:
http_status_gte, duration_ms_gte, response_body_contains and
consecutive_failures_gte. Rules are evaluated after every execution
(both inline and via Celery worker) and create Alerts with configurable
severity when triggered.

- AlertRule model with CASCADE FK to jobs and workspace isolation
- Alembic migration c3d4e5f6a7b8 creates the alert_rules table
- CRUD API under /api/jobs/{job_id}/alert-rules (operator required)
- evaluate_alert_rules() in http_runner; Celery worker wired post-retry
- React section in JobDetailPage with create form, toggle and delete
- RBAC: viewers read-only; operators manage rules
- i18n in English and Portuguese
- 7 backend integration tests; 1 frontend component test
@btcneves
btcneves merged commit 9c83252 into main May 21, 2026
5 checks passed
@btcneves
btcneves deleted the feat/log-aggregation branch May 21, 2026 12:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant