Feature/be add prometheus alert metrics and health telemetry to exchange rate oracle cache

backend/docs/EXCHANGE_RATE_ORACLE_CACHE.md

@@ -3,7 +3,9 @@
 Caching and concurrency control for path-payment exchange-rate quotes
 (`GET /api/path-payment-quote/:id`).
 
-Covers issues **#1445** (distributed concurrency control and locking) and
+Covers issues **#1443** (Prometheus alert metrics and health telemetry),
+**#1444** (automated retry with exponential backoff),
+**#1445** (distributed concurrency control and locking) and
 **#1446** (integration and stress test suite).
 
 ---
@@ -16,6 +18,9 @@ Covers issues **#1445** (distributed concurrency control and locking) and
 | `src/lib/exchange-rate-coordinator.js` | Cross-instance coordination: Redis lock + shared quote store |
 | `src/services/exchangeRateService.js` | `getExchangeRateQuote()` composes both layers around the Horizon query |
 | `src/lib/path-payment-metrics.js` | Prometheus series (existing cache metrics + concurrency metrics) |
+| `src/lib/exchange-rate-oracle-telemetry.js` | Alert metrics, rolling-window health, separate registry (#1443) |
+| `src/lib/exchange-rate-oracle-retry.js` | Full-jitter exponential backoff around the Horizon read (#1444) |
+| `docs/alerts/exchange-rate-oracle-cache.rules.yml` | Prometheus alert rules (#1443, #1444) |
 
 ```
 getExchangeRateQuote(key)
@@ -154,11 +159,96 @@ HTTP bursts wait until every request has joined the in-flight load, using
 Horizon response. supertest opens a separate server per request, so arrival
 order is otherwise not guaranteed.
 
+## 8. Alert metrics and health (#1443)
+
+Lookups and loads are counted in `exchange-rate-oracle-telemetry.js`, separate
+from the path-payment series so a scrape can alert on the cache itself.
+`/metrics` merges the registry. Every label is from a fixed set (`hit`,
+`miss`, `stale`, `success`, `error`, `timeout`, `not_found`). Asset codes,
+issuers, amounts and cache keys are never labels.
+
+| Metric | Type | Labels |
+|---|---|---|
+| `exchange_rate_oracle_cache_lookups_total` | counter | `result` (hit/miss/stale) |
+| `exchange_rate_oracle_cache_loads_total` | counter | `outcome` (success/error/timeout/not_found) |
+| `exchange_rate_oracle_cache_load_duration_seconds` | histogram | `outcome` |
+| `exchange_rate_oracle_cache_health_state` | gauge | 0 healthy, 1 degraded, 2 unhealthy |
+| `exchange_rate_oracle_cache_error_ratio` | gauge | rolling window |
+| `exchange_rate_oracle_cache_timeout_ratio` | gauge | rolling window |
+| `exchange_rate_oracle_cache_stale_ratio` | gauge | rolling window |
+| `exchange_rate_oracle_cache_last_load_timestamp_seconds` | gauge | none |
+
+`not_found` is a normal "Horizon has no path" result. It is counted, but it
+does not move the error ratio and cannot mark the cache unhealthy.
+
+`GET /health/exchange-rate-oracle-cache` (public, no quote or account data):
+
+| Status | Condition | HTTP |
+|---|---|---|
+| `unhealthy` | ≥ `min_samples` loads and internal error ratio ≥ threshold | 503 |
+| `degraded` | ≥ `min_samples` loads and timeout ratio ≥ threshold, **or** ≥ `min_samples` lookups and stale ratio ≥ threshold | 200 |
+| `healthy` | otherwise, including no traffic | 200 |
+
+`GET /health` also reports `services.exchange_rate_oracle_cache`. That value
+does not change `ok` or the status code. Gauges refresh at scrape time, so
+they decay when traffic stops. The window is a fixed ring of 5-second buckets.
+
+| Variable | Default |
+|---|---|
+| `EXCHANGE_RATE_ORACLE_HEALTH_WINDOW_MS` | `300000` |
+| `EXCHANGE_RATE_ORACLE_HEALTH_MIN_SAMPLES` | `20` |
+| `EXCHANGE_RATE_ORACLE_ERROR_RATIO_THRESHOLD` | `0.05` |
+| `EXCHANGE_RATE_ORACLE_TIMEOUT_RATIO_THRESHOLD` | `0.2` |
+| `EXCHANGE_RATE_ORACLE_STALE_RATIO_THRESHOLD` | `0.5` |
+
+`docs/alerts/exchange-rate-oracle-cache.rules.yml` alerts on unhealthy state,
+error ratio, repeated timeouts, a high stale ratio, and p99 load latency
+above 5s. A unit test checks that every metric named in the rules file is
+registered.
+
+Failed loads are logged at `warn` with outcome, duration, status and error
+message. The cache key is not logged. A failure inside the metrics client is
+swallowed so it cannot replace the loader's error.
+
+| Metric | Type | Labels |
+|---|---|---|
+| `exchange_rate_oracle_cache_retries_total` | counter | `result` (scheduled/recovered/exhausted) |
+
+## 9. Retry with exponential backoff (#1444)
+
+`getExchangeRateQuote()` wraps the Horizon read in `withOracleRetry()`. The
+wrapper sits inside `ExchangeRateCache.getOrLoad()`, so concurrent callers
+for the same key share one retry loop. Cache hits do not retry. Redis lock
+polling is unchanged.
+
+Delay is full jitter: `random(0, min(maxDelayMs, baseDelayMs * 2^n))`.
+Defaults are 3 attempts, 100ms base, 1000ms cap. Env and callers cannot
+exceed 6 attempts or 10s of delay.
+
+| Variable | Default |
+|---|---|
+| `EXCHANGE_RATE_ORACLE_RETRY_MAX_ATTEMPTS` | `3` |
+| `EXCHANGE_RATE_ORACLE_RETRY_BASE_DELAY_MS` | `100` |
+| `EXCHANGE_RATE_ORACLE_RETRY_MAX_DELAY_MS` | `1000` |
+
+Retried: network errors, HTTP 408, 429, and 5xx except 501.
+Not retried: `NoPathFoundError` (404), other 4xx, `CacheLoadTimeoutError`,
+and any error with `retryable: false`. The last error is rethrown with
+`retryAttempts` set, and it is not cached. The outer load timeout still
+bounds how long HTTP waiters block.
+
+The quote is a read of public DEX data, so a retry cannot submit a payment.
+`ExchangeRateOracleCacheRetriesExhausted` fires when a load gives up.
+
+## 10. Tests
+
 The suites were mutation-checked against `exchange-rate-cache.js`. Disabling
 single-flight fails 16 tests. Dropping the invalidation guard fails 4.
 
 ```
-npx vitest run src/lib/exchange-rate-cache.test.js \
+npx vitest run src/lib/exchange-rate-oracle-telemetry.test.js \
+               src/lib/exchange-rate-oracle-retry.test.js \
+               src/lib/exchange-rate-cache.test.js \
                src/lib/exchange-rate-coordinator.test.js \
                src/services/exchangeRateService.test.js \
                tests/integration/exchange-rate-cache.test.js

backend/docs/alerts/exchange-rate-oracle-cache.rules.yml

@@ -0,0 +1,89 @@
+# Prometheus alerting rules for the Exchange Rate Oracle Cache (issue #1443).
+#
+# Load with:   rule_files: ["docs/alerts/exchange-rate-oracle-cache.rules.yml"]
+# Validate:    promtool check rules docs/alerts/exchange-rate-oracle-cache.rules.yml
+#
+# Metric reference: docs/EXCHANGE_RATE_ORACLE_CACHE.md
+
+groups:
+  - name: exchange-rate-oracle-cache
+    rules:
+      - alert: ExchangeRateOracleCacheUnhealthy
+        expr: max(exchange_rate_oracle_cache_health_state) >= 2
+        for: 5m
+        labels:
+          severity: critical
+        annotations:
+          summary: Exchange rate oracle cache reports unhealthy
+          description: >
+            Rolling-window internal error ratio is above threshold. Quote loads
+            are failing. See GET /health/exchange-rate-oracle-cache for counts
+            and reasons. "No path" results are excluded from this ratio.
+
+      - alert: ExchangeRateOracleCacheHighErrorRatio
+        expr: |
+          sum(rate(exchange_rate_oracle_cache_loads_total{outcome="error"}[10m]))
+            /
+          clamp_min(sum(rate(exchange_rate_oracle_cache_loads_total[10m])), 1e-9)
+            > 0.05
+          and sum(increase(exchange_rate_oracle_cache_loads_total[10m])) >= 20
+        for: 10m
+        labels:
+          severity: critical
+        annotations:
+          summary: More than 5% of exchange-rate oracle loads are internal errors
+          description: >
+            Horizon or the quote loader is failing. Check logs for
+            "Exchange rate oracle cache load failed".
+
+      - alert: ExchangeRateOracleCacheLoadTimeouts
+        expr: sum(increase(exchange_rate_oracle_cache_loads_total{outcome="timeout"}[10m])) > 5
+        for: 10m
+        labels:
+          severity: warning
+        annotations:
+          summary: Exchange-rate oracle loads are timing out
+          description: >
+            More than 5 loads hit the load timeout in 10m. Waiters receive 504.
+            A hung Horizon call is the usual cause.
+
+      - alert: ExchangeRateOracleCacheHighStaleRatio
+        expr: |
+          sum(rate(exchange_rate_oracle_cache_lookups_total{result="stale"}[10m]))
+            /
+          clamp_min(sum(rate(exchange_rate_oracle_cache_lookups_total[10m])), 1e-9)
+            > 0.5
+          and sum(increase(exchange_rate_oracle_cache_lookups_total[10m])) >= 20
+        for: 10m
+        labels:
+          severity: warning
+        annotations:
+          summary: Most exchange-rate cache lookups are past the fresh TTL
+          description: >
+            Quotes are being refreshed constantly. The TTL may be shorter than
+            the request interval, or invalidation is too aggressive.
+
+      - alert: ExchangeRateOracleCacheRetriesExhausted
+        expr: sum(increase(exchange_rate_oracle_cache_retries_total{result="exhausted"}[10m])) > 0
+        for: 5m
+        labels:
+          severity: warning
+        annotations:
+          summary: Exchange-rate oracle retries were exhausted
+          description: >
+            A quote load kept failing after exponential backoff. The last error
+            is returned to the caller and is not cached. Check Horizon.
+
+      - alert: ExchangeRateOracleCacheSlowLoads
+        expr: |
+          histogram_quantile(0.99,
+            sum by (le) (rate(exchange_rate_oracle_cache_load_duration_seconds_bucket[10m]))
+          ) > 5
+        for: 10m
+        labels:
+          severity: warning
+        annotations:
+          summary: Exchange-rate oracle load p99 latency above 5s
+          description: >
+            Upstream quote latency is high. The HTTP load timeout is 15s by
+            default (EXCHANGE_RATE_LOAD_TIMEOUT_MS).

backend/src/app.js

@@ -48,6 +48,7 @@ import {
 import { versionDeprecationMiddleware } from "./lib/version-deprecation.js";
 import oracleRouter from "./routes/oracle.js";
 import { getPaymentSessionValidatorHealth } from "./lib/payment-session-validator.js";
+import { getExchangeRateOracleHealth } from "./lib/exchange-rate-oracle-telemetry.js";
 import { configureExchangeRateCoordination } from "./services/exchangeRateService.js";
 
 export async function createApp({ redisClient }) {
@@ -260,6 +261,7 @@ export async function createApp({ redisClient }) {
         redis: redisAvailable ? "ok" : "unavailable",
         // Informational only — does not affect `ok` / the status code.
         payment_session_validator: getPaymentSessionValidatorHealth().status,
+        exchange_rate_oracle_cache: getExchangeRateOracleHealth().status,
       },
     });
   });
@@ -288,6 +290,30 @@ export async function createApp({ redisClient }) {
     res.status(health.status === "unhealthy" ? 503 : 200).json(health);
   });
 
+  /**
+   * @swagger
+   * /health/exchange-rate-oracle-cache:
+   *   get:
+   *     summary: Exchange Rate Oracle Cache health telemetry
+   *     description: >
+   *       Rolling-window load and lookup counts plus derived status for the
+   *       exchange-rate quote cache (issue #1443). Returns 503 only when the
+   *       cache is unhealthy (internal error ratio above threshold). A
+   *       degraded status (timeouts or stale lookups) still returns 200.
+   *       The body contains no quotes, asset codes, or account ids.
+   *     tags: [Health]
+   *     security: []
+   *     responses:
+   *       200:
+   *         description: Cache healthy or degraded
+   *       503:
+   *         description: Cache unhealthy
+   */
+  app.get("/health/exchange-rate-oracle-cache", (_req, res) => {
+    const health = getExchangeRateOracleHealth();
+    res.status(health.status === "unhealthy" ? 503 : 200).json(health);
+  });
+
   const verifyPaymentRateLimit = createVerifyPaymentRateLimit({
     store: redisAvailable ? createRedisRateLimitStore({ client: redisClient }) : undefined,
   });

backend/src/lib/exchange-rate-cache.js

@@ -36,6 +36,11 @@ import {
   exchangeRateCacheLoadTimeouts,
   exchangeRateCacheStaleWritesPrevented,
 } from './path-payment-metrics.js';
+import {
+  classifyOracleLoadError,
+  recordOracleLoad,
+  recordOracleLookup,
+} from './exchange-rate-oracle-telemetry.js';
 
 /**
  * Default cache metrics wired to the granular path-payment series (issue #1048)
@@ -142,6 +147,7 @@ export class ExchangeRateCache {
     const entry = this.cache.get(key);
     if (!entry) {
       this.metrics?.miss?.inc?.({ cache: 'exchange_rate' });
+      recordOracleLookup('miss');
       return { hit: false, data: null, stale: false };
     }
 
@@ -150,11 +156,13 @@ export class ExchangeRateCache {
     if (age > this.staleToleranceMs) {
       this.cache.delete(key);
       this.metrics?.miss?.inc?.({ cache: 'exchange_rate' });
+      recordOracleLookup('miss');
       return { hit: false, data: null, stale: false };
     }
 
     const stale = age > this.ttlMs;
     this.metrics?.hit?.inc?.({ cache: 'exchange_rate', stale: stale ? '1' : '0' });
+    recordOracleLookup(stale ? 'stale' : 'hit');
 
     // Refresh recency in LRU order
     this.cache.delete(key);
@@ -204,6 +212,7 @@ export class ExchangeRateCache {
 
     const entry = { promise: null, invalidated: false };
     entry.promise = (async () => {
+      const started = Date.now();
       try {
         // Invoke the loader on a later microtask so the entry is registered
         // in `inflight` first — even a synchronously throwing loader then
@@ -217,7 +226,11 @@ export class ExchangeRateCache {
         } else {
           this.set(key, data);
         }
+        recordOracleLoad('success', Date.now() - started);
         return data;
+      } catch (err) {
+        recordOracleLoad(classifyOracleLoadError(err), Date.now() - started, err);
+        throw err;
       } finally {
         // Only remove our own entry; an invalidation may already have
         // detached it and a newer load may occupy the slot.