[Backend] Prometheus Metrics Collection & OpenTelemetry Distributed Tracing

[blurbeast]

Overview & Background

As FlowFi scales to support real-time high-throughput streaming events, infrastructure visibility is critical. The backend currently uses Winston for structured logging, but lacks time-series APM metrics and distributed tracing. Production monitoring requires standard Prometheus metrics to track indexer synchronization lag, RPC latency, SSE client pool saturation, and database connection pool utilization.

---

Detailed Problem Statement

1. Invisible Indexer Lag:
   • Operators cannot monitor how many ledgers the indexer is lagging behind the live Stellar ledger sequence.
2. No RPC Pool Health Visibility:
   • Failovers and circuit breaker trips across multi-RPC nodes are not exposed as trackable metrics.
3. SSE Connection Saturation:
   • Memory and file descriptor exhaustion from long-lived SSE connections cannot be anticipated without real-time gauges.

---

Technical Specification & Architecture

1. Prometheus Metrics Instrumentation (prom-client)

Implement backend/src/lib/metrics.ts exporting:

• flowfi_indexer_current_ledger: Gauge of last indexed ledger.
• flowfi_indexer_network_ledger: Gauge of current network ledger sequence.
• flowfi_indexer_lag_ledgers: Computed gauge (network - current).
• flowfi_sse_active_connections: Gauge of connected SSE clients by topic/user.
• flowfi_rpc_request_duration_seconds: Histogram of Stellar RPC requests partitioned by method (simulateTransaction, sendTransaction, getEvents).
• flowfi_db_query_duration_seconds: Histogram of Prisma query durations.
• flowfi_http_requests_total: Counter of API requests by status code and route.

2. Metrics Endpoint

• Expose GET /metrics in backend/src/app.ts.
• Restrict to internal cluster network or protect via Prometheus bearer token.

3. OpenTelemetry Tracing Integration

• Initialize @opentelemetry/sdk-node in backend/src/index.ts to trace HTTP request lifecycles through controllers, database operations, and outbound RPC calls.

---

Target Files

• backend/src/lib/metrics.ts
• backend/src/routes/health.routes.ts
• backend/src/services/indexer.service.ts
• backend/src/services/sse.service.ts
• backend/src/services/sorobanService.ts

---

Acceptance Criteria

• GET /metrics outputs standard Prometheus scrape format.
• Indexer ledger lag correctly updates on every polling cycle.
• Active SSE client count increases and decreases accurately on connect/disconnect.
• Grafana dashboard JSON template provided in backend/docs/observability/grafana-dashboard.json.

[Notdayo]

@Notdayo has applied to work on this issue as part of the Stellar Wave Program's 9th wave.

Hello Maintainer, I’ve carefully reviewed the issue and understand the requirements. I’m interested in working on it and believe I can deliver a proper solution that meets the expected criteria. I’d appreciate being considered for the assignment.

ℹ️ Repo Maintainers: To accept this application, review their application or assign @Notdayo to this issue.

[salienne]

@salienne has applied to work on this issue as part of the Stellar Wave Program's 9th wave.

Hi Maintainer, I've reviewed the issue and identified both the affected implementation and the validation gaps. My approach would be to implement the security fix and add tests to prevent regressions. Kindly assign it to me.

ℹ️ Repo Maintainers: To accept this application, review their application or assign @salienne to this issue.

[Mikey-222]

@Mikey-222 has applied to work on this issue as part of the Stellar Wave Program's 9th wave.

can i take this

ℹ️ Repo Maintainers: To accept this application, review their application or assign @Mikey-222 to this issue.

[drips-wave]

Congratulations, @Mikey-222! 🎉 Your application was accepted by the repo's maintainers, and the issue is due on September 30, 2026.

🧑‍💻 @Mikey-222: Please resolve the issue such that the repo's maintainers have enough time to review your contribution before the due date. You'll earn Points for completing the issue on-time, which will make you eligible for a share of the Stellar Wave Program's reward pool.

Warning

When opening a PR, please link it to this issue to ensure it gets tracked accurately. Points are awarded when this issue is marked as completed by the maintainer.

🤠 Repo maintainers: Please keep an eye on the contributor's progress and review their work before the due date. You can manage this issue, including adjusting its complexity and points, here.

🌊 Happy Wave 🌊