Skip to content

feat(backend): add observability, indexer dead-letter replay, and str… - #1462

Merged
blurbeast merged 2 commits into
LabsCrypt:mainfrom
Mikey-222:feat/observability-admin-deadletter-simulation
Sep 26, 2026
Merged

blurbeast merged 2 commits into
LabsCrypt:mainfrom
Mikey-222:feat/observability-admin-deadletter-simulation

Conversation

@Mikey-222

Copy link
Copy Markdown
Contributor

Description

Type of Change

  • 🐛 Bug fix (non-breaking change which fixes an issue)
  • ✨ New feature (non-breaking change which adds functionality)
  • 💥 Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • 📚 Documentation update
  • 🔧 Refactoring (no functional changes)
  • ⚡ Performance improvement
  • 🧪 Test addition or update

Related Issues

Closes #

Changes Made

Testing

Test Coverage

  • Unit tests added/updated
  • Integration tests added/updated
  • Manual testing performed

Test Steps

Breaking Changes

Breaking Changes:

Migration Guide:

Screenshots/Demo

Checklist

  • My code follows the project's style guidelines
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • Updated Postman/Hoppscotch API collections if routes changed
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • Any dependent changes have been merged and published
  • I have checked for breaking changes and documented them if applicable

Additional Notes

…eam simulation

Adds production visibility and two operational safety nets for the indexer
and stream endpoints.

Observability
- Prometheus registry covering indexer progress, RPC failovers/circuit
  breaker, SSE connections and drops, DB pool/query timings, HTTP requests
  and Node process stats.
- GET /metrics is fail-closed: unavailable in production unless a guard is
  configured, and when both METRICS_BEARER_TOKEN and METRICS_ALLOWED_CIDRS
  are set both must pass. It also bypasses the global rate limiter.
- OpenTelemetry bootstrap with OTLP HTTP export, plus domain spans for
  indexer batches, RPC calls and dead-letter replay. Honours
  OTEL_SDK_DISABLED and degrades to no-op spans when no provider exists.
- Importable Grafana dashboard and setup docs under
  backend/docs/observability/.

Indexer dead-letter queue
- New IndexerDeadLetterEvent model; events that fail processing are
  quarantined with their payload, error and attempt count instead of being
  dropped, so a contract bug or RPC outage no longer loses events.
- Admin endpoints to list, replay, replay-all and discard entries, behind
  the existing router-wide requireAdmin guard.

Stream simulation
- POST /api/v1/streams/simulate assembles an unsigned transaction and runs
  it against Soroban simulateTransaction, returning fee estimates, resource
  usage and the decoded return value, with contract error codes decoded
  into typed ApiError responses.

Fixes found while testing
- withSpan re-ran the wrapped function on error, so any failing operation
  executed twice and its error was reported as success. The callback is now
  invoked exactly once and errors propagate to the caller. The tracer is
  also resolved per call instead of being pinned at module load, which
  matters when the SDK initialises after this module or is disabled.
- Route normalisation used global regexes with .test(), so stateful lastIndex
  made UUID and long-hash route labels alternate between raw and normalised
  forms between requests.
- Array route templates produced double-slash labels such as /v1//streams/:id.
- The simulation placeholder account was not a valid Stellar address.
@blurbeast
blurbeast merged commit 88ed9ae into LabsCrypt:main Sep 26, 2026
2 of 6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment