Skip to content

Write an operator runbook for stuck circuit breakers #1337

Description

@greatest0fallt1me

Summary

README's 'Circuit Breaker Stuck Open' section suggests waiting for cooldown, but does not mention the breaker state metric, the Postgres-backed store table, or how to reset state safely.

Why this matters

During incidents operators need concrete steps; restarting the process is currently the only option and drops traffic.

Scope

Expand the section into docs/ with: how to read gateway upstream breaker state from /api/metrics, SQL to inspect the breaker table created by PostgresCircuitBreakerStore, the admin reset endpoint once mounted, and env tuning (PROXY_BREAKER_*).

Relevant code in CalloraOrg/Callora-Backend:

  • README.md
  • RESILIENCE.md
  • src/lib/circuitBreaker.ts
  • src/config/env.ts

Priority

Low

Acceptance criteria

  • The runbook lists metric names exactly as registered in src/metrics.ts
  • SQL uses the actual table name and columns
  • Tuning variables match src/config/env.ts
  • README links to the runbook

Validation

Validate metric names against curl /api/metrics output in a local run.

Non-goals

  • Typo-only, formatting-only, or cosmetic changes.
  • Unrelated refactors, dependency upgrades, or broad rewrites.
  • Removing safeguards or weakening validation to make tests pass.

Contributor application

Before coding, describe the affected modules, proposed state/invariant changes, test strategy, compatibility considerations, and an estimate. Wait for assignment before starting implementation.

PR requirements

Use a feature branch and include Closes #<issue-number>. Address every criterion, map criteria to code and tests, explain security and failure-mode handling, and ensure CI passes.

Quality review

A maintainer will assess correctness, completeness, test depth, compatibility, observability, and adverse-case handling. A substantive implementation is required; merge or CI status alone does not guarantee reward eligibility.


Generated by Claude Code

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

Stellar WaveStellar Wave Program issue

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions