diff --git a/backend/docs/EXCHANGE_RATE_ORACLE_CACHE.md b/backend/docs/EXCHANGE_RATE_ORACLE_CACHE.md new file mode 100644 index 00000000..7197058e --- /dev/null +++ b/backend/docs/EXCHANGE_RATE_ORACLE_CACHE.md @@ -0,0 +1,166 @@ +# Exchange Rate Oracle Cache + +Caching and concurrency control for path-payment exchange-rate quotes +(`GET /api/path-payment-quote/:id`). + +Covers issues **#1445** (distributed concurrency control and locking) and +**#1446** (integration and stress test suite). + +--- + +## 1. Layout + +| File | Role | +|---|---| +| `src/lib/exchange-rate-cache.js` | In-process LRU + TTL cache with single-flight `getOrLoad()` | +| `src/lib/exchange-rate-coordinator.js` | Cross-instance coordination: Redis lock + shared quote store | +| `src/services/exchangeRateService.js` | `getExchangeRateQuote()` composes both layers around the Horizon query | +| `src/lib/path-payment-metrics.js` | Prometheus series (existing cache metrics + concurrency metrics) | + +``` +getExchangeRateQuote(key) + │ + ├─ L1: ExchangeRateCache.getOrLoad(key) per process + │ fresh hit ─────────────────────────────▶ return (cached: true) + │ load in flight ────────────────────────▶ join it (cached: true) + │ otherwise run ONE loader ↓ + │ + └─ L2: ExchangeRateCoordinator.load(key) across instances (if Redis) + shared quote present ──────────────────▶ return (cached: true) + SET exrate:lock: NX PX ── won ─────▶ query Horizon, publish, release + └ lost ────▶ poll shared store / retry lock + until waitTimeoutMs → direct query +``` + +## 2. The problems this solves (#1445) + +| Problem | Before | After | +|---|---|---| +| Thundering herd, one process | N concurrent misses → N Horizon calls | 1 call; the rest join it | +| Thundering herd, many instances | 1 call per instance | 1 call total (lock leader); peers reuse the shared quote | +| Invalidation race | a load started before `invalidateExchangeRateQuote()` wrote the old quote back afterwards | the detached load serves its own callers but never writes to the cache | +| Hung Horizon call | every waiter hung with it | loads time out (`EXCHANGE_RATE_LOAD_TIMEOUT_MS`, 504) and the slot is freed | +| Invalidation across instances | only the local process forgot the quote | the shared quote is deleted too | + +## 3. In-process single-flight (`ExchangeRateCache.getOrLoad`) + +- The first miss for a key registers an in-flight entry, then calls the + loader. Later callers await the same promise. +- The loader is invoked on a later microtask, so a synchronously throwing + loader still reaches the cleanup code. +- Errors reach every waiter and are **not** cached. The next call retries. +- `delete(key)` / `clear()` flag the in-flight entry as invalidated and + detach it. The next caller starts a fresh load, and the old load cannot + write back. Memory is bounded by the number of active loads, with no + per-key history. +- A stale-but-tolerable entry is refreshed through the same single-flight + path. + +## 4. Distributed coordination (`ExchangeRateCoordinator`) + +Enabled by `createApp()` whenever the Redis client is connected +(`configureExchangeRateCoordination`). Without Redis, only the in-process +layer runs. + +**Lock.** `SET exrate:lock: PX NX`. Release uses a +Lua compare-and-delete, so a holder whose lease expired can never delete the +next owner's lock. No fencing token is needed: the protected work (a +read-only Horizon query followed by an idempotent cache write) is safe to +duplicate in the rare lease-expiry case. That case is logged with a hint to +raise `EXCHANGE_RATE_LOCK_TTL_MS`. + +**Shared store.** `exrate:quote:` holds `{ v: 1, insertedAt, data }` with +`PX = sharedTtlMs`, so any present entry is fresh. Keys are SHA-256 hashes, so +client input never reaches Redis key names. + +**Follower loop.** Each iteration reads the shared store (hit → done), then +tries the lock (won → become leader, double-check the store, load). If the +lock is still held, the follower sleeps `pollIntervalMs` and repeats. If the +leader fails, or crashes and its lease expires, a follower wins the lock on a +later iteration. After `waitTimeoutMs`, the follower queries Horizon directly. + +**Failure policy: fail open.** Any Redis error, or a closed client, falls +back to a direct Horizon query and is counted in +`exchange_rate_coordination_fallbacks_total{reason="redis_error"}`. Loader +errors are never mistaken for Redis errors, so they are never retried by the +coordinator. + +**Validation of shared data.** Entries read from Redis must parse, carry +`v: 1` and pass `isValidQuote` (string amounts matching +`^\d{1,19}(\.\d{1,7})?$` and an array `path`). Anything else counts as a miss +(`result="invalid"`) and is overwritten by the next leader. A corrupted or +tampered entry can never reach a payer as `send_max`. + +### Configuration + +| Variable | Default | Meaning | +|---|---|---| +| `EXCHANGE_RATE_CACHE_TTL_MS` | `30000` | L1 freshness and shared-quote lifetime | +| `EXCHANGE_RATE_LOCK_TTL_MS` | `5000` | Lock lease; keep above a typical Horizon call | +| `EXCHANGE_RATE_LOCK_WAIT_MS` | `2000` | Max follower wait before querying directly | +| `EXCHANGE_RATE_LOCK_POLL_MS` | `50` | Follower poll interval | +| `EXCHANGE_RATE_LOAD_TIMEOUT_MS` | `15000` | Upper bound on one load; waiters get 504 | + +## 5. Metrics + +Added to the path-payment registry, which `/metrics` already serves: + +| Metric | Type | Labels | +|---|---|---| +| `exchange_rate_cache_coalesced_requests_total` | counter | `cache` | +| `exchange_rate_cache_inflight_loads` | gauge | `cache` | +| `exchange_rate_cache_load_timeouts_total` | counter | `cache` | +| `exchange_rate_cache_stale_writes_prevented_total` | counter | `cache` | +| `exchange_rate_lock_acquisitions_total` | counter | `result` (acquired/contended/error) | +| `exchange_rate_lock_wait_seconds` | histogram | `outcome` (shared_hit/acquired/timeout/error) | +| `exchange_rate_shared_cache_lookups_total` | counter | `result` (hit/miss/invalid/error) | +| `exchange_rate_coordination_fallbacks_total` | counter | `reason` (wait_timeout/redis_error) | + +`path_payment_quote_cache_size` is now updated on every write and delete, +not only on `prune()`. + +## 6. Security notes + +- Redis is treated as trusted infrastructure, but its content is still + validated before use (see above). +- Lock tokens are random UUIDs, and release is owner-checked atomically. +- Fail-open is deliberate: the quote is public DEX data and coordination is + only an optimization. An attacker who can break Redis gains nothing beyond + the pre-#1445 behavior of one Horizon query per request. +- The existing per-IP rate limit on the quote endpoint still applies. + Coalescing reduces the Horizon load an attacker can cause by bursting + identical requests. +- Unchanged from before: the cache key covers asset pair, amount and issuers, + not `slippage` or `source_account`. The route always uses the default + slippage. + +## 7. Tests (#1446) + +| Suite | Scope | +|---|---| +| `src/lib/exchange-rate-cache.test.js` | LRU/TTL plus single-flight, invalidation races, sync-throwing loaders, timeouts, gauges | +| `src/lib/exchange-rate-coordinator.test.js` | lock ownership and expiry, leader/follower, leader failure and crash takeover, wait-timeout fallback, fail-open, poisoned shared entries | +| `src/services/exchangeRateService.test.js` | service wiring, burst coalescing, `NoPathFoundError` fan-out, shared reuse, cross-instance invalidation, Redis outage | +| `tests/integration/exchange-rate-cache.test.js` | real HTTP stack on `GET /api/path-payment-quote/:id`: 40-request burst → 1 Horizon call, distinct pairs, 404/502 fan-out without caching, 504 on hung Horizon, invalidation mid-flight, `/metrics`, Redis coordination, poisoned entry, Redis down | +| `load-tests/exchange-rate-cache-stress.test.js` | 5k concurrent → 1 call; 20k over 250 keys → 250 calls; LRU churn; invalidation storm; failure isolation; 8 simulated instances sharing Redis → 1 call per key; cold-instance reuse; lock-holder crash; Redis outage mid-burst; stuck lock; leak check | + +`tests/helpers/fake-redis.js` is an in-memory Redis (SET NX/PX, GET, DEL, +compare-and-delete EVAL, TTL expiry, injectable latency and failures). Many +coordinators can share one instance to simulate a scaled deployment +deterministically, without a live Redis. + +HTTP bursts wait until every request has joined the in-flight load, using +`exchange_rate_cache_coalesced_requests_total`, before releasing the stubbed +Horizon response. supertest opens a separate server per request, so arrival +order is otherwise not guaranteed. + +The suites were mutation-checked against `exchange-rate-cache.js`. Disabling +single-flight fails 16 tests. Dropping the invalidation guard fails 4. + +``` +npx vitest run src/lib/exchange-rate-cache.test.js \ + src/lib/exchange-rate-coordinator.test.js \ + src/services/exchangeRateService.test.js \ + tests/integration/exchange-rate-cache.test.js +npm run test:load -- exchange-rate-cache-stress +``` diff --git a/backend/docs/PAYMENT_SESSION_VALIDATOR.md b/backend/docs/PAYMENT_SESSION_VALIDATOR.md new file mode 100644 index 00000000..c209835b --- /dev/null +++ b/backend/docs/PAYMENT_SESSION_VALIDATOR.md @@ -0,0 +1,197 @@ +# Payment Session Validator + +Validation stage that runs before a payment session is persisted, used by both +`POST /api/create-payment` / `POST /api/sessions` (`src/routes/payments.js`) +and `paymentService.createPaymentSession` (`src/services/paymentService.js`). + +Covers issues **#1447** (payload sanitization and strict validation) and +**#1448** (Prometheus alert metrics and health telemetry). Builds on the shared +rules from #1087 (see `PAYMENT_PROCESSOR.md`). + +--- + +## 1. Layout + +| File | Role | +|---|---| +| `src/lib/payment-session-rules.js` | Pure rule functions: no I/O, logging or metrics | +| `src/lib/payment-session-validator.js` | `validatePaymentSession()`: runs the rules in order, records metrics, logs, tracks health | +| `src/lib/payment-session-validator-metrics.js` | Separate Prometheus registry, merged into `/metrics` | +| `docs/alerts/payment-session-validator.rules.yml` | Prometheus alerting rules | + +Request pipeline for the HTTP routes: + +``` +zod schema (paymentSessionZodSchema) type coercion, required fields, memo rules +sanitizeMetadataMiddleware metadata XSS/NoSQL scrubbing, forbidden keys → 400 +validatePaymentSession({ source: "http" }) + sanitization → payload → issuer → limits → allowlist first failure → 400 +insert into payments uses the *sanitized* payload +``` + +In the service layer, the validator now runs **before** the on-chain issuer +lookup (`AssetIssuerErrorRecovery.verifyIssuerOnChain`). An invalid request +never costs a Horizon round-trip. + +## 2. Rules + +| Rule | Function | Rejection reasons | +|---|---|---| +| `sanitization` | `sanitizeSessionPayload` | `malformed_payload`, `forbidden_key`, `field_too_long` | +| `payload` | `validateSessionAsset`, `validateSessionAmount` | `invalid_asset`, `invalid_amount` | +| `issuer` | `resolveAndValidateIssuer` | `missing_issuer`, `invalid_issuer` | +| `limits` | `validatePerAssetLimits` | `below_min`, `above_max` (response includes `min`/`max` + `delta`) | +| `allowlist` | `validateAllowedIssuers` | `issuer_not_allowed` | + +### Sanitization (#1447) + +- The body must be a plain JSON object. Arrays, primitives and class + instances are rejected. +- `__proto__`, `constructor` and `prototype` keys are rejected at any depth + (walk capped at 8 levels). `JSON.parse` creates `__proto__` as an own + property, so the walk sees it. +- For the string fields `asset`, `asset_issuer`, `recipient`, `description`, + `message`, `memo`, `memo_type`, `webhook_url` and `client_id`, each value is: + - NFC-normalized + - stripped of C0/C1 control characters, bidi overrides/isolates + (U+202A–U+202E, U+2066–U+2069, U+200E/F) and zero-width characters + - trimmed + - length-checked **after** stripping (limits in `SESSION_FIELD_MAX_LENGTHS`) +- An optional field that sanitizes to `""` is dropped. A required field + (`asset`, `recipient`) stays `""` and fails later with an explicit error. +- The input is never mutated. Callers persist `validation.payload`. + +### Strict validation (#1447) + +- `asset` must match `^[A-Z0-9]{1,12}$` after upper-casing. +- `amount` must be a finite number > 0, ≤ `922337203685.4775807` (int64 + stroops) and have at most 7 decimal places. Floating-point artifacts such as + `0.1 + 0.2` are rejected. + +### Hardening of existing rules + +- Limit lookup uses `Object.hasOwn`, so asset codes like `CONSTRUCTOR` or + `__proto__` can never resolve to inherited properties. +- Limit bounds are coerced with `Number()`, so legacy numeric-string configs + such as `"10"` keep working. Unusable bounds (`"abc"`, objects) are + **ignored** and reported as config anomalies instead of silently comparing + as `NaN`. +- Allowlist entries that are not strings are ignored rather than coerced. + +### Related fixes + +- `src/lib/request-schemas.js` called `isValidStellarPublicKey` without + importing it. Every non-XLM session with an issuer hit a `ReferenceError` + inside schema validation. +- `src/lib/sanitize-metadata.js` copied keys into a fresh object with + `sanitized[key] = …`. A `__proto__` key therefore replaced that object's + prototype. Metadata containing forbidden keys is now rejected with 400. + +## 3. Metrics (#1448) + +Every label value comes from a fixed server-side set. None comes from request +data, so clients cannot inflate cardinality. Unknown `source` values collapse +to `unknown`. + +| Metric | Type | Labels | +|---|---|---| +| `payment_session_validator_evaluations_total` | counter | `source` (http/service/unknown), `outcome` (accepted/rejected/error) | +| `payment_session_validator_rejections_total` | counter | `source`, `rule`, `reason` | +| `payment_session_validator_duration_seconds` | histogram | `source`, `outcome` | +| `payment_session_validator_sanitized_fields_total` | counter | `field` | +| `payment_session_validator_suspicious_payloads_total` | counter | `signal` (forbidden_key/malformed_payload/bidi_control/oversized_field) | +| `payment_session_validator_config_anomalies_total` | counter | `kind` (invalid_entry/invalid_min/invalid_max/min_greater_than_max) | +| `payment_session_validator_health_state` | gauge | 0 healthy, 1 degraded, 2 unhealthy | +| `payment_session_validator_rejection_ratio` | gauge | rolling window | +| `payment_session_validator_error_ratio` | gauge | rolling window | +| `payment_session_validator_last_evaluation_timestamp_seconds` | gauge | none | + +The rolling-window gauges are recomputed at scrape time, so they decay back to +healthy when traffic stops. + +## 4. Health telemetry (#1448) + +`GET /health/payment-session-validator` (public, no merchant data): + +```json +{ + "status": "healthy", + "reasons": [], + "window_ms": 300000, + "total": 42, "accepted": 40, "rejected": 2, "errors": 0, "suspicious": 0, + "rejection_ratio": 0.0476, "error_ratio": 0, + "last_evaluation_at": "2026-09-26T12:00:00.000Z", + "thresholds": { "min_samples": 20, "error_ratio": 0.05, "rejection_ratio": 0.5, "suspicious": 10 } +} +``` + +| Status | Condition | HTTP | +|---|---|---| +| `unhealthy` | ≥ `min_samples` evaluations and error ratio ≥ threshold | 503 | +| `degraded` | ≥ `min_samples` and rejection ratio ≥ threshold, **or** suspicious count ≥ threshold | 200 | +| `healthy` | otherwise (including no traffic) | 200 | + +`GET /health` also reports `services.payment_session_validator`. The value is +informational only and does not change `ok` or the status code. + +The window uses 5-second buckets in a fixed ring, so memory stays constant +under any load. Tuning via environment: + +| Variable | Default | +|---|---| +| `PAYMENT_SESSION_VALIDATOR_HEALTH_WINDOW_MS` | `300000` | +| `PAYMENT_SESSION_VALIDATOR_HEALTH_MIN_SAMPLES` | `20` | +| `PAYMENT_SESSION_VALIDATOR_ERROR_RATIO_THRESHOLD` | `0.05` | +| `PAYMENT_SESSION_VALIDATOR_REJECTION_RATIO_THRESHOLD` | `0.5` | +| `PAYMENT_SESSION_VALIDATOR_SUSPICIOUS_THRESHOLD` | `10` | + +## 5. Alerts + +`docs/alerts/payment-session-validator.rules.yml` defines: + +| Alert | Severity | Fires when | +|---|---|---| +| `PaymentSessionValidatorInternalErrors` | critical | any `outcome="error"` in 5m | +| `PaymentSessionValidatorUnhealthy` | critical | `health_state >= 2` for 5m | +| `PaymentSessionValidatorHighRejectionRatio` | warning | > 50% rejected over 10m (≥ 20 samples) | +| `PaymentSessionValidatorSuspiciousPayloadSpike` | warning | > 10 suspicious payloads in 5m | +| `PaymentSessionValidatorPrototypePollutionAttempt` | warning | any `forbidden_key` in 15m | +| `PaymentSessionValidatorMerchantConfigAnomaly` | info | any config anomaly in 30m | +| `PaymentSessionValidatorSlow` | warning | p99 > 25ms for 10m | + +A unit test checks that every metric referenced in the rules file exists in +the registry. + +## 6. Logging + +- Rejection: `info` with `{ merchantId, source, rule, reason }` +- Suspicious payload: `warn` with `{ merchantId, source, signals }` +- Config anomaly: `warn` with `{ merchantId, source, anomalies }` +- Internal error: `error` with `{ err, merchantId, source }`, then rethrown + +The raw payload is never logged. + +## 7. Security notes + +- **Fail-closed on validator errors:** unexpected exceptions are recorded and + rethrown, so the route returns 500 and no session is created. +- **Config anomalies fail open, but only per bound:** a malformed bound is + ignored, and the other bound and all other rules still apply. The choice + favors availability for merchants with bad configs. It is visible through + the `MerchantConfigAnomaly` alert. +- **Trojan Source / display spoofing:** bidi and zero-width characters are + stripped before the text reaches the database or the hosted checkout page. +- **DoS:** limits and allowlist checks now run before the Horizon issuer + lookup, and the forbidden-key walk is depth-capped. +- **Behavior changes clients may notice:** amounts with more than 7 decimals, + amounts above the Stellar maximum, `description` > 1000 chars and + `client_id` > 128 chars are now rejected with 400. + +## 8. Tests + +``` +npx vitest run src/lib/payment-session-rules.test.js \ + src/lib/payment-session-validator.test.js \ + src/lib/sanitize-metadata.test.js \ + tests/integration/payment-session-validator.test.js +``` diff --git a/backend/docs/alerts/payment-session-validator.rules.yml b/backend/docs/alerts/payment-session-validator.rules.yml new file mode 100644 index 00000000..a1a982ec --- /dev/null +++ b/backend/docs/alerts/payment-session-validator.rules.yml @@ -0,0 +1,94 @@ +# Prometheus alerting rules for the Payment Session Validator (issue #1448). +# +# Load with: rule_files: ["docs/alerts/payment-session-validator.rules.yml"] +# Validate: promtool check rules docs/alerts/payment-session-validator.rules.yml +# +# Metric reference: docs/PAYMENT_SESSION_VALIDATOR.md + +groups: + - name: payment-session-validator + rules: + - alert: PaymentSessionValidatorInternalErrors + expr: sum(increase(payment_session_validator_evaluations_total{outcome="error"}[5m])) > 0 + for: 0m + labels: + severity: critical + annotations: + summary: Payment session validator is throwing internal errors + description: > + {{ $value | humanize }} validations failed with an internal error in + the last 5m. Sessions are not being created. Check logs for + "Payment session validator failed unexpectedly". + + - alert: PaymentSessionValidatorUnhealthy + expr: max(payment_session_validator_health_state) >= 2 + for: 5m + labels: + severity: critical + annotations: + summary: Payment session validator reports unhealthy + description: > + Rolling-window error ratio is above threshold. See + GET /health/payment-session-validator for counts and reasons. + + - alert: PaymentSessionValidatorHighRejectionRatio + expr: | + sum(rate(payment_session_validator_evaluations_total{outcome="rejected"}[10m])) + / + clamp_min(sum(rate(payment_session_validator_evaluations_total[10m])), 1e-9) + > 0.5 + and sum(increase(payment_session_validator_evaluations_total[10m])) >= 20 + for: 10m + labels: + severity: warning + annotations: + summary: More than half of payment sessions are being rejected + description: > + Likely a broken client integration or a scripted probe. Break down + by rule/reason with payment_session_validator_rejections_total. + + - alert: PaymentSessionValidatorSuspiciousPayloadSpike + expr: sum(increase(payment_session_validator_suspicious_payloads_total[5m])) > 10 + for: 0m + labels: + severity: warning + annotations: + summary: Spike in suspicious payment session payloads + description: > + {{ $value | humanize }} payloads carried forbidden keys, bidi control + characters, oversized fields or were not JSON objects in the last + 5m. Correlate with the "Suspicious payment session payload" log + entries (they carry merchantId) to identify the source. + + - alert: PaymentSessionValidatorPrototypePollutionAttempt + expr: sum(increase(payment_session_validator_suspicious_payloads_total{signal="forbidden_key"}[15m])) > 0 + for: 0m + labels: + severity: warning + annotations: + summary: Prototype-pollution keys submitted to payment session creation + description: The payload was rejected; this alert exists for security visibility. + + - alert: PaymentSessionValidatorMerchantConfigAnomaly + expr: sum by (kind) (increase(payment_session_validator_config_anomalies_total[30m])) > 0 + for: 0m + labels: + severity: info + annotations: + summary: "Merchant payment_limits misconfigured ({{ $labels.kind }})" + description: > + A merchant's limit bound is non-numeric or min > max; the unusable + bound is ignored. Find the merchant via the + "Merchant payment_limits misconfigured" log entry. + + - alert: PaymentSessionValidatorSlow + expr: | + histogram_quantile(0.99, + sum by (le) (rate(payment_session_validator_duration_seconds_bucket[10m])) + ) > 0.025 + for: 10m + labels: + severity: warning + annotations: + summary: Payment session validation p99 latency above 25ms + description: Validation is pure CPU work and normally completes in well under 1ms. diff --git a/backend/load-tests/exchange-rate-cache-stress.test.js b/backend/load-tests/exchange-rate-cache-stress.test.js new file mode 100644 index 00000000..7a23a295 --- /dev/null +++ b/backend/load-tests/exchange-rate-cache-stress.test.js @@ -0,0 +1,335 @@ +/** + * Exchange Rate Oracle Cache — stress suite (issue #1446). + * + * Hammers the real ExchangeRateCache and ExchangeRateCoordinator with + * thousands of concurrent requests. Several "instances" (each with its own + * in-memory cache) share one in-memory Redis with injected latency, which + * simulates a horizontally scaled deployment. No network and no Horizon. + * + * Invariants checked under load: + * - Horizon is queried exactly once per key, per process and across instances + * - LRU capacity is never exceeded and no in-flight entries or locks leak + * - an invalidated load never writes back + * - Redis outages and crashed lock holders degrade to direct queries, + * never to failed requests + * + * Run with: npm run test:load -- exchange-rate-cache-stress + */ +import { describe, it, expect, vi, beforeEach } from 'vitest'; + +vi.mock('../src/lib/logger.js', () => ({ + logger: { debug: vi.fn(), info: vi.fn(), warn: vi.fn(), error: vi.fn() }, +})); + +import { ExchangeRateCache } from '../src/lib/exchange-rate-cache.js'; +import { ExchangeRateCoordinator } from '../src/lib/exchange-rate-coordinator.js'; +import { createFakeRedis } from '../tests/helpers/fake-redis.js'; + +const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms)); + +const quoteFor = (key, extra = {}) => ({ + sourceAsset: 'XLM', + sourceAmount: '0.5000000', + sendMax: '0.5050000', + path: [], + key, + ...extra, +}); + +/** Horizon stand-in that counts calls per key and takes `latencyMs`. */ +function createHorizon({ latencyMs = 20 } = {}) { + const calls = new Map(); + return { + calls, + get total() { + let n = 0; + for (const c of calls.values()) n += c; + return n; + }, + fetch(key) { + return async () => { + calls.set(key, (calls.get(key) ?? 0) + 1); + await sleep(latencyMs); + return quoteFor(key); + }; + }, + }; +} + +/** One simulated API instance: private L1 cache + shared-Redis coordinator. */ +function createInstance(redis, opts = {}) { + const cache = new ExchangeRateCache({ ttlMs: 30_000, staleToleranceMs: 60_000, maxEntries: 1_000 }); + const coordinator = redis + ? new ExchangeRateCoordinator({ + redisClient: redis, + sharedTtlMs: 30_000, + lockTtlMs: 2_000, + waitTimeoutMs: 3_000, + pollIntervalMs: 5, + metrics: null, + ...opts, + }) + : null; + return { + cache, + coordinator, + get(key, horizon) { + const load = horizon.fetch(key); + const loader = coordinator ? async () => (await coordinator.load(key, load)).data : load; + return cache.getOrLoad(key, loader, { timeoutMs: 10_000 }); + }, + }; +} + +function percentile(sorted, p) { + return sorted[Math.min(sorted.length - 1, Math.floor((p / 100) * sorted.length))]; +} + +async function timed(promises) { + const started = performance.now(); + const latencies = []; + const results = await Promise.all( + promises.map(async (make) => { + const t0 = performance.now(); + const r = await make(); + latencies.push(performance.now() - t0); + return r; + }), + ); + latencies.sort((a, b) => a - b); + return { + results, + wallMs: performance.now() - started, + p50: percentile(latencies, 50), + p99: percentile(latencies, 99), + }; +} + +function lockKeys(redis) { + return [...redis.store.keys()].filter((k) => k.startsWith('exrate:lock:')); +} + +describe('Exchange Rate Oracle Cache — stress (issue #1446)', () => { + let horizon; + + beforeEach(() => { + horizon = createHorizon(); + }); + + describe('single process', () => { + it('5,000 concurrent requests for one quote → 1 Horizon call', async () => { + const node = createInstance(null); + const { results, p99, wallMs } = await timed( + Array.from({ length: 5_000 }, () => () => node.get('hot', horizon)), + ); + expect(horizon.total).toBe(1); + expect(results.every((r) => r.data.key === 'hot')).toBe(true); + expect(results.filter((r) => r.source === 'loader')).toHaveLength(1); + expect(node.cache.inflightCount).toBe(0); + // Everyone waits roughly one Horizon round-trip, not 5,000 of them. + expect(p99).toBeLessThan(1_000); + expect(wallMs).toBeLessThan(2_000); + }); + + it('20,000 requests over 250 keys → exactly 250 Horizon calls', async () => { + const node = createInstance(null); + const keys = Array.from({ length: 250 }, (_, i) => `pair-${i}`); + await timed( + Array.from({ length: 20_000 }, (_, i) => () => node.get(keys[(i * 7919) % keys.length], horizon)), + ); + expect(horizon.calls.size).toBe(250); + expect([...horizon.calls.values()].every((c) => c === 1)).toBe(true); + expect(node.cache.size).toBe(250); + }); + + it('respects LRU capacity under concurrent churn', async () => { + const cache = new ExchangeRateCache({ ttlMs: 30_000, staleToleranceMs: 60_000, maxEntries: 50 }); + const keys = Array.from({ length: 500 }, (_, i) => `k${i}`); + let maxObserved = 0; + await Promise.all( + Array.from({ length: 5_000 }, (_, i) => + cache + .getOrLoad(keys[(i * 31) % keys.length], horizon.fetch(keys[(i * 31) % keys.length])) + .then(() => { + maxObserved = Math.max(maxObserved, cache.size); + }), + ), + ); + expect(maxObserved).toBeLessThanOrEqual(50); + expect(cache.size).toBeLessThanOrEqual(50); + expect(cache.inflightCount).toBe(0); + }); + + it('never caches a result from before the latest invalidation (invalidation storm)', async () => { + const cache = new ExchangeRateCache({ ttlMs: 30_000, staleToleranceMs: 60_000, maxEntries: 1_000 }); + const keys = Array.from({ length: 20 }, (_, i) => `k${i}`); + const version = new Map(keys.map((k) => [k, 0])); + const versionedLoader = (key) => async () => { + const startedAt = version.get(key); + await sleep(Math.random() * 10); + return { key, version: startedAt }; + }; + + const ops = []; + for (let i = 0; i < 4_000; i++) { + const key = keys[i % keys.length]; + if (i % 13 === 0) { + ops.push( + sleep(Math.random() * 30).then(() => { + version.set(key, version.get(key) + 1); + cache.delete(key); + }), + ); + } else { + ops.push(sleep(Math.random() * 30).then(() => cache.getOrLoad(key, versionedLoader(key)))); + } + } + await Promise.all(ops); + await sleep(20); // let any detached loads settle + + for (const key of keys) { + const entry = cache.get(key); + if (entry.hit) { + expect(entry.data.version).toBe(version.get(key)); + } + } + expect(cache.inflightCount).toBe(0); + }); + + it('isolates failures: a failing key does not affect healthy keys', async () => { + const node = createInstance(null); + let failures = 0; + const flaky = { + fetch: (key) => + key === 'broken' + ? async () => { + failures++; + await sleep(5); + throw new Error('horizon 503'); + } + : horizon.fetch(key), + }; + const settled = await Promise.allSettled( + Array.from({ length: 2_000 }, (_, i) => node.get(i % 2 ? 'broken' : 'healthy', flaky)), + ); + const ok = settled.filter((s) => s.status === 'fulfilled'); + const bad = settled.filter((s) => s.status === 'rejected'); + expect(ok).toHaveLength(1_000); + expect(bad).toHaveLength(1_000); + expect(failures).toBe(1); // coalesced failure, not 1,000 retries + expect(horizon.calls.get('healthy')).toBe(1); + expect(node.cache.inflightCount).toBe(0); + }); + }); + + describe('multiple instances sharing Redis', () => { + it('8 instances × 250 concurrent requests for one quote → 1 Horizon call', async () => { + const redis = createFakeRedis({ latencyMs: 2 }); + const nodes = Array.from({ length: 8 }, () => createInstance(redis)); + const { results, p99 } = await timed( + nodes.flatMap((node) => Array.from({ length: 250 }, () => () => node.get('hot', horizon))), + ); + expect(horizon.total).toBe(1); + expect(results.every((r) => r.data.key === 'hot')).toBe(true); + expect(lockKeys(redis)).toEqual([]); + expect(nodes.every((n) => n.cache.inflightCount === 0)).toBe(true); + expect(p99).toBeLessThan(2_000); + }); + + it('8 instances × 50 keys → exactly one Horizon call per key', async () => { + const redis = createFakeRedis({ latencyMs: 1 }); + const nodes = Array.from({ length: 8 }, () => createInstance(redis)); + const keys = Array.from({ length: 50 }, (_, i) => `pair-${i}`); + await Promise.all( + nodes.flatMap((node, n) => + Array.from({ length: 200 }, (_, i) => node.get(keys[(i + n * 7) % keys.length], horizon)), + ), + ); + expect(horizon.calls.size).toBe(50); + expect([...horizon.calls.values()].every((c) => c === 1)).toBe(true); + expect(lockKeys(redis)).toEqual([]); + }); + + it('a cold instance reuses quotes published by its peers', async () => { + const redis = createFakeRedis({ latencyMs: 1 }); + const warm = createInstance(redis); + await Promise.all(Array.from({ length: 20 }, (_, i) => warm.get(`pair-${i}`, horizon))); + expect(horizon.total).toBe(20); + + const cold = createInstance(redis); + const results = await Promise.all(Array.from({ length: 20 }, (_, i) => cold.get(`pair-${i}`, horizon))); + expect(horizon.total).toBe(20); + expect(results.every((r, i) => r.data.key === `pair-${i}`)).toBe(true); + }); + + it('recovers when the lock holder crashes (lease expiry)', async () => { + const redis = createFakeRedis({ latencyMs: 1 }); + // A crashed instance left the lock behind and never published a quote. + await redis.sendCommand(['SET', 'exrate:lock:hot', 'crashed-node', 'PX', '150', 'NX']); + const nodes = Array.from({ length: 4 }, () => createInstance(redis)); + const started = performance.now(); + const results = await Promise.all( + nodes.flatMap((node) => Array.from({ length: 100 }, () => node.get('hot', horizon))), + ); + const elapsed = performance.now() - started; + expect(results).toHaveLength(400); + expect(horizon.total).toBe(1); // one survivor takes over, peers reuse its quote + expect(elapsed).toBeGreaterThanOrEqual(140); + expect(elapsed).toBeLessThan(1_500); + }); + + it('keeps serving through a Redis outage mid-burst', async () => { + const redis = createFakeRedis({ latencyMs: 2 }); + const nodes = Array.from({ length: 6 }, () => createInstance(redis)); + const keys = Array.from({ length: 30 }, (_, i) => `pair-${i}`); + + setTimeout(() => redis.setFailure(new Error('ECONNRESET')), 5); + const settled = await Promise.allSettled( + nodes.flatMap((node) => Array.from({ length: 300 }, (_, i) => node.get(keys[i % keys.length], horizon))), + ); + + expect(settled.every((s) => s.status === 'fulfilled')).toBe(true); + // Fail-open costs at most one query per key per instance, and L1 + // single-flight still holds within each instance. + expect(horizon.total).toBeLessThanOrEqual(nodes.length * keys.length); + for (const count of horizon.calls.values()) { + expect(count).toBeLessThanOrEqual(nodes.length); + } + }); + + it('falls back to direct queries when a peer holds the lock too long', async () => { + const redis = createFakeRedis({ latencyMs: 1 }); + await redis.sendCommand(['SET', 'exrate:lock:hot', 'stuck-node', 'PX', '60000', 'NX']); + const nodes = Array.from({ length: 3 }, () => createInstance(redis, { waitTimeoutMs: 100 })); + const results = await Promise.all( + nodes.flatMap((node) => Array.from({ length: 50 }, () => node.get('hot', horizon))), + ); + expect(results).toHaveLength(150); + // Each instance falls back once (L1 still coalesces its own callers). + expect(horizon.total).toBe(3); + }); + + it('does not leak state after a large mixed workload', async () => { + const redis = createFakeRedis({ latencyMs: 1 }); + const nodes = Array.from({ length: 4 }, () => createInstance(redis)); + const keys = Array.from({ length: 100 }, (_, i) => `pair-${i}`); + const ops = []; + for (let i = 0; i < 4_000; i++) { + const node = nodes[i % nodes.length]; + const key = keys[(i * 17) % keys.length]; + if (i % 50 === 0) { + node.cache.delete(key); + ops.push(node.coordinator.invalidate(key)); + } else { + ops.push(node.get(key, horizon)); + } + } + await Promise.all(ops); + expect(lockKeys(redis)).toEqual([]); + expect(nodes.every((n) => n.cache.inflightCount === 0)).toBe(true); + for (const node of nodes) { + expect(node.cache.size).toBeLessThanOrEqual(keys.length); + } + }); + }); +}); diff --git a/backend/src/app.js b/backend/src/app.js index c606090f..503fc3cd 100644 --- a/backend/src/app.js +++ b/backend/src/app.js @@ -47,11 +47,17 @@ import { } from "./lib/transaction-signer.js"; import { versionDeprecationMiddleware } from "./lib/version-deprecation.js"; import oracleRouter from "./routes/oracle.js"; +import { getPaymentSessionValidatorHealth } from "./lib/payment-session-validator.js"; +import { configureExchangeRateCoordination } from "./services/exchangeRateService.js"; export async function createApp({ redisClient }) { const app = express(); const redisAvailable = Boolean(redisClient && redisClient.isOpen); + // Cross-instance exchange-rate quote coordination (issue #1445). Without + // Redis the cache still coalesces concurrent misses within this process. + configureExchangeRateCoordination({ redisClient: redisAvailable ? redisClient : null }); + const __filename = fileURLToPath(import.meta.url); const __dirname = path.dirname(__filename); const publicDir = path.join(__dirname, "..", "public"); @@ -252,10 +258,36 @@ export async function createApp({ redisClient }) { database: dbOk ? "ok" : "unavailable", horizon: horizonOk ? "ok" : "unavailable", redis: redisAvailable ? "ok" : "unavailable", + // Informational only — does not affect `ok` / the status code. + payment_session_validator: getPaymentSessionValidatorHealth().status, }, }); }); + /** + * @swagger + * /health/payment-session-validator: + * get: + * summary: Payment Session Validator health telemetry + * description: > + * Rolling-window counts, rejection / error ratios and derived status + * for payment session validation (issue #1448). Returns 503 only when + * the validator is unhealthy (internal error ratio above threshold); + * a degraded status (high rejection ratio or suspicious payload spike) + * still returns 200. + * tags: [Health] + * security: [] + * responses: + * 200: + * description: Validator healthy or degraded + * 503: + * description: Validator unhealthy + */ + app.get("/health/payment-session-validator", (_req, res) => { + const health = getPaymentSessionValidatorHealth(); + res.status(health.status === "unhealthy" ? 503 : 200).json(health); + }); + const verifyPaymentRateLimit = createVerifyPaymentRateLimit({ store: redisAvailable ? createRedisRateLimitStore({ client: redisClient }) : undefined, }); diff --git a/backend/src/lib/exchange-rate-cache.js b/backend/src/lib/exchange-rate-cache.js index 26db34b9..2ef9b781 100644 --- a/backend/src/lib/exchange-rate-cache.js +++ b/backend/src/lib/exchange-rate-cache.js @@ -11,6 +11,17 @@ * - Prometheus metrics for hit/miss/eviction counts * - Stale-while-revalidate tolerance (separate staleness window) * - Thread-safe via synchronous Map operations (Node.js single-threaded) + * + * Concurrency control (issue #1445): + * - getOrLoad() coalesces concurrent misses for the same key into ONE loader + * call (single-flight), so a burst of identical quote requests produces a + * single Horizon query per process. + * - delete()/clear() mark any in-flight load for the key as invalidated and + * detach it. That load still resolves its own callers, but it can never + * write its (possibly outdated) result back into the cache. Tracking this on + * the in-flight entry keeps memory bounded by the number of active loads. + * - Loads are bounded by a timeout so a hung loader cannot pin waiters forever. + * Cross-process coordination lives in exchange-rate-coordinator.js. */ import { createHash } from 'node:crypto'; @@ -20,16 +31,38 @@ import { pathPaymentQuoteCacheMisses, pathPaymentQuoteCacheEvictions, pathPaymentQuoteCacheSize, + exchangeRateCacheCoalescedRequests, + exchangeRateCacheInflightLoads, + exchangeRateCacheLoadTimeouts, + exchangeRateCacheStaleWritesPrevented, } from './path-payment-metrics.js'; -/** Default cache metrics wired to the granular path-payment series (issue #1048). */ +/** + * Default cache metrics wired to the granular path-payment series (issue #1048) + * plus the concurrency-control series (issue #1445). + */ const DEFAULT_METRICS = { hit: pathPaymentQuoteCacheHits, miss: pathPaymentQuoteCacheMisses, eviction: pathPaymentQuoteCacheEvictions, size: pathPaymentQuoteCacheSize, + coalesced: exchangeRateCacheCoalescedRequests, + inflight: exchangeRateCacheInflightLoads, + loadTimeout: exchangeRateCacheLoadTimeouts, + staleWritePrevented: exchangeRateCacheStaleWritesPrevented, }; +export class CacheLoadTimeoutError extends Error { + constructor(timeoutMs) { + super(`Exchange rate load timed out after ${timeoutMs}ms`); + this.name = 'CacheLoadTimeoutError'; + // `status` is what the Express error handler reads; `statusCode` + // matches ExchangeRateError. + this.status = 504; + this.statusCode = 504; + } +} + const DEFAULT_TTL_MS = Number.parseInt( process.env.EXCHANGE_RATE_CACHE_TTL_MS || '30000', 10, @@ -79,6 +112,24 @@ export class ExchangeRateCache { this.metrics = metrics; /** @type {Map} */ this.cache = new Map(); + /** + * In-flight loads (single-flight). + * @type {Map, invalidated: boolean}>} + */ + this.inflight = new Map(); + } + + /** Detach the in-flight load for `key` (if any) and forbid its write-back. */ + _invalidateInflight(key) { + const entry = this.inflight.get(key); + if (!entry) return; + entry.invalidated = true; + this.inflight.delete(key); + this._updateInflightGauge(); + } + + _updateInflightGauge() { + this.metrics?.inflight?.set?.({ cache: 'exchange_rate' }, this.inflight.size); } /** @@ -125,11 +176,78 @@ export class ExchangeRateCache { logger.debug(`ExchangeRateCache: evicted oldest entry (key prefix: ${oldestKey?.slice(0, 8)})`); } this.cache.set(key, { data, insertedAt: Date.now() }); + this.metrics?.size?.set?.({ cache: 'exchange_rate' }, this.cache.size); } - /** Remove a specific entry (e.g. after a payment status change makes its quote stale). */ + /** + * Return a fresh cached value, or run `loader` exactly once per key no + * matter how many callers ask concurrently. + * + * @template T + * @param {string} key + * @param {() => Promise} loader + * @param {object} [opts] + * @param {number} [opts.timeoutMs] reject waiters if the load exceeds this + * @returns {Promise<{data: T, source: 'cache'|'loader'|'coalesced'}>} + */ + async getOrLoad(key, loader, { timeoutMs = 0 } = {}) { + const cached = this.get(key); + if (cached.hit && !cached.stale) { + return { data: cached.data, source: 'cache' }; + } + + const existing = this.inflight.get(key); + if (existing) { + this.metrics?.coalesced?.inc?.({ cache: 'exchange_rate' }); + return { data: await existing.promise, source: 'coalesced' }; + } + + const entry = { promise: null, invalidated: false }; + entry.promise = (async () => { + try { + // Invoke the loader on a later microtask so the entry is registered + // in `inflight` first — even a synchronously throwing loader then + // goes through the cleanup in `finally`. + const data = await withTimeout(Promise.resolve().then(loader), timeoutMs, () => { + this.metrics?.loadTimeout?.inc?.({ cache: 'exchange_rate' }); + }); + if (entry.invalidated) { + this.metrics?.staleWritePrevented?.inc?.({ cache: 'exchange_rate' }); + logger.debug(`ExchangeRateCache: dropped write for invalidated key (prefix: ${key?.slice(0, 8)})`); + } else { + this.set(key, data); + } + return data; + } finally { + // Only remove our own entry; an invalidation may already have + // detached it and a newer load may occupy the slot. + if (this.inflight.get(key) === entry) { + this.inflight.delete(key); + this._updateInflightGauge(); + } + } + })(); + this.inflight.set(key, entry); + this._updateInflightGauge(); + + return { data: await entry.promise, source: 'loader' }; + } + + /** Number of keys currently being loaded. */ + get inflightCount() { + return this.inflight.size; + } + + /** + * Remove a specific entry (e.g. after a payment status change makes its + * quote stale). Also detaches any in-flight load so the next caller starts + * a fresh one; the detached load cannot write its result back. + */ delete(key) { - return this.cache.delete(key); + this._invalidateInflight(key); + const removed = this.cache.delete(key); + this.metrics?.size?.set?.({ cache: 'exchange_rate' }, this.cache.size); + return removed; } /** Evict all entries older than ttlMs. Returns the number of evicted entries. */ @@ -153,10 +271,33 @@ export class ExchangeRateCache { return this.cache.size; } - /** Clear all entries — intended for test isolation. */ + /** Clear all entries and invalidate every in-flight load. */ clear() { + for (const key of [...this.inflight.keys()]) { + this._invalidateInflight(key); + } this.cache.clear(); + this.metrics?.size?.set?.({ cache: 'exchange_rate' }, 0); + } +} + +/** + * Race `promise` against a timer. `timeoutMs <= 0` disables the timeout. + * The timer is always cleared so it never keeps the event loop alive. + */ +function withTimeout(promise, timeoutMs, onTimeout) { + if (!timeoutMs || timeoutMs <= 0) { + return promise; } + let timer; + const timeout = new Promise((_, reject) => { + timer = setTimeout(() => { + onTimeout?.(); + reject(new CacheLoadTimeoutError(timeoutMs)); + }, timeoutMs); + timer.unref?.(); + }); + return Promise.race([promise, timeout]).finally(() => clearTimeout(timer)); } let defaultInstance = null; diff --git a/backend/src/lib/exchange-rate-cache.test.js b/backend/src/lib/exchange-rate-cache.test.js index a2d30a43..f4581c3d 100644 --- a/backend/src/lib/exchange-rate-cache.test.js +++ b/backend/src/lib/exchange-rate-cache.test.js @@ -93,9 +93,10 @@ describe('ExchangeRateCache', () => { cache.set('fresh', 'value'); vi.advanceTimersByTime(50); cache.set('stale-but-tolerable', 'value2'); - vi.advanceTimersByTime(300); // moves first entry past staleToleranceMs + vi.advanceTimersByTime(160); // first entry at 210ms (> staleToleranceMs=200), second at 160ms const pruned = cache.prune(); expect(pruned).toBe(1); + expect(cache.get('stale-but-tolerable').hit).toBe(true); vi.useRealTimers(); }); @@ -138,3 +139,170 @@ describe('getExchangeRateCache singleton', () => { expect(a).not.toBe(b); }); }); + +describe('ExchangeRateCache.getOrLoad — concurrency control (issue #1445)', () => { + const deferred = () => { + let resolve; + let reject; + const promise = new Promise((res, rej) => { + resolve = res; + reject = rej; + }); + return { promise, resolve, reject }; + }; + + const makeMetrics = () => ({ + hit: { inc: vi.fn() }, + miss: { inc: vi.fn() }, + eviction: { inc: vi.fn() }, + size: { set: vi.fn() }, + coalesced: { inc: vi.fn() }, + inflight: { set: vi.fn() }, + loadTimeout: { inc: vi.fn() }, + staleWritePrevented: { inc: vi.fn() }, + }); + + let cache; + let metrics; + + beforeEach(() => { + metrics = makeMetrics(); + cache = new ExchangeRateCache({ ttlMs: 1000, maxEntries: 100, staleToleranceMs: 2000, metrics }); + }); + + afterEach(() => { + vi.useRealTimers(); + }); + + it('serves fresh entries without calling the loader', async () => { + cache.set('k', { rate: 1 }); + const loader = vi.fn(); + await expect(cache.getOrLoad('k', loader)).resolves.toEqual({ data: { rate: 1 }, source: 'cache' }); + expect(loader).not.toHaveBeenCalled(); + }); + + it('coalesces concurrent misses into a single loader call', async () => { + const d = deferred(); + const loader = vi.fn(() => d.promise); + + const pending = Array.from({ length: 50 }, () => cache.getOrLoad('k', loader)); + expect(cache.inflightCount).toBe(1); + d.resolve({ rate: 2 }); + const results = await Promise.all(pending); + + expect(loader).toHaveBeenCalledTimes(1); + expect(results.filter((r) => r.source === 'loader')).toHaveLength(1); + expect(results.filter((r) => r.source === 'coalesced')).toHaveLength(49); + expect(results.every((r) => r.data.rate === 2)).toBe(true); + expect(metrics.coalesced.inc).toHaveBeenCalledTimes(49); + expect(cache.inflightCount).toBe(0); + expect(cache.get('k').data).toEqual({ rate: 2 }); + }); + + it('keeps different keys independent', async () => { + const loader = vi.fn(async () => ({ rate: Math.random() })); + await Promise.all([cache.getOrLoad('a', loader), cache.getOrLoad('b', loader), cache.getOrLoad('a', loader)]); + expect(loader).toHaveBeenCalledTimes(2); + }); + + it('refreshes a stale entry through a single load', async () => { + vi.useFakeTimers(); + cache.set('k', { rate: 1 }); + vi.advanceTimersByTime(1500); // stale but tolerable + const loader = vi.fn(async () => ({ rate: 9 })); + const [a, b] = await Promise.all([cache.getOrLoad('k', loader), cache.getOrLoad('k', loader)]); + expect(loader).toHaveBeenCalledTimes(1); + expect(a.data).toEqual({ rate: 9 }); + expect(b.data).toEqual({ rate: 9 }); + }); + + it('propagates loader errors to every waiter and caches nothing', async () => { + const d = deferred(); + const loader = vi.fn(() => d.promise); + const pending = [cache.getOrLoad('k', loader), cache.getOrLoad('k', loader)]; + d.reject(new Error('horizon down')); + const settled = await Promise.allSettled(pending); + expect(settled.every((s) => s.status === 'rejected' && s.reason.message === 'horizon down')).toBe(true); + expect(cache.inflightCount).toBe(0); + expect(cache.size).toBe(0); + + // The next call retries rather than replaying the failure. + const ok = await cache.getOrLoad('k', async () => ({ rate: 3 })); + expect(ok.source).toBe('loader'); + }); + + it('cleans up after a synchronously throwing loader', async () => { + const loader = () => { + throw new Error('sync boom'); + }; + await expect(cache.getOrLoad('k', loader)).rejects.toThrow('sync boom'); + expect(cache.inflightCount).toBe(0); + }); + + it('does not write back a load that was invalidated mid-flight', async () => { + const first = deferred(); + const pending = cache.getOrLoad('k', () => first.promise); + + cache.delete('k'); // e.g. payment status changed + expect(cache.inflightCount).toBe(0); + + // A caller after the invalidation must start a fresh load, not join the old one. + const second = vi.fn(async () => ({ rate: 'new' })); + const fresh = await cache.getOrLoad('k', second); + expect(second).toHaveBeenCalledTimes(1); + expect(fresh.data).toEqual({ rate: 'new' }); + + first.resolve({ rate: 'old' }); + const old = await pending; + expect(old.data).toEqual({ rate: 'old' }); // original caller still served + expect(cache.get('k').data).toEqual({ rate: 'new' }); // but cache not clobbered + expect(metrics.staleWritePrevented.inc).toHaveBeenCalledTimes(1); + }); + + it('clear() invalidates every in-flight load', async () => { + const a = deferred(); + const b = deferred(); + const pa = cache.getOrLoad('a', () => a.promise); + const pb = cache.getOrLoad('b', () => b.promise); + cache.clear(); + a.resolve(1); + b.resolve(2); + await Promise.all([pa, pb]); + expect(cache.size).toBe(0); + expect(cache.inflightCount).toBe(0); + expect(metrics.staleWritePrevented.inc).toHaveBeenCalledTimes(2); + }); + + it('times out a hung loader, rejects waiters and frees the slot', async () => { + vi.useFakeTimers(); + const hung = new Promise(() => {}); + const pending = [ + cache.getOrLoad('k', () => hung, { timeoutMs: 100 }), + cache.getOrLoad('k', () => hung, { timeoutMs: 100 }), + ]; + const assertion = expect(Promise.all(pending)).rejects.toMatchObject({ + name: 'CacheLoadTimeoutError', + statusCode: 504, + }); + await vi.advanceTimersByTimeAsync(100); + await assertion; + expect(metrics.loadTimeout.inc).toHaveBeenCalledTimes(1); + expect(cache.inflightCount).toBe(0); + }); + + it('reports the in-flight gauge as loads start and finish', async () => { + const d = deferred(); + const pending = cache.getOrLoad('k', () => d.promise); + expect(metrics.inflight.set).toHaveBeenLastCalledWith({ cache: 'exchange_rate' }, 1); + d.resolve(1); + await pending; + expect(metrics.inflight.set).toHaveBeenLastCalledWith({ cache: 'exchange_rate' }, 0); + }); + + it('works without any metrics object', async () => { + const bare = new ExchangeRateCache({ ttlMs: 1000 }); + await expect(bare.getOrLoad('k', async () => 1)).resolves.toEqual({ data: 1, source: 'loader' }); + bare.delete('k'); + bare.clear(); + }); +}); diff --git a/backend/src/lib/exchange-rate-coordinator.js b/backend/src/lib/exchange-rate-coordinator.js new file mode 100644 index 00000000..27e3c4ab --- /dev/null +++ b/backend/src/lib/exchange-rate-coordinator.js @@ -0,0 +1,289 @@ +/** + * Distributed concurrency control for the Exchange Rate Oracle Cache + * (issue #1445). + * + * The in-memory ExchangeRateCache already coalesces concurrent misses within + * one process. With N API instances behind a load balancer, a popular quote + * can still trigger N simultaneous Horizon queries. This module coordinates + * those instances through Redis: + * + * 1. Shared quote store – `exrate:quote:` holds the latest quote for + * ttlMs so any instance can reuse a peer's result. + * 2. Distributed lock – `exrate:lock:` (SET NX PX + random token) + * elects one instance to query Horizon. Release is a compare-and-delete + * Lua script, so a holder whose lock already expired can never delete a + * lock now owned by someone else. + * 3. Followers poll the shared store. If the leader fails or crashes (lock + * released or expired with no quote written), the next poll can take + * the lock itself. After waitTimeoutMs a follower gives up and queries + * Horizon directly. + * + * Failure policy: FAIL OPEN. Quotes are read-only public DEX data, so any + * Redis error (or a closed client) falls back to a direct Horizon query + * rather than failing the request. Coordination is purely an optimization. + * + * Data read from Redis is validated before use. A corrupted or foreign entry + * counts as a miss, never as a quote. + * + * All commands go through `sendCommand`, which node-redis v4/v5 and the + * project's no-op fallback client all support. + */ + +import { randomUUID } from 'node:crypto'; +import { logger } from './logger.js'; +import { + exchangeRateLockAcquisitions, + exchangeRateLockWaitDuration, + exchangeRateSharedCacheLookups, + exchangeRateCoordinationFallbacks, +} from './path-payment-metrics.js'; + +const DEFAULT_METRICS = { + lock: exchangeRateLockAcquisitions, + wait: exchangeRateLockWaitDuration, + shared: exchangeRateSharedCacheLookups, + fallback: exchangeRateCoordinationFallbacks, +}; + +/** Compare-and-delete: only the token holder may release the lock. */ +const RELEASE_LOCK_SCRIPT = + "if redis.call('get', KEYS[1]) == ARGV[1] then return redis.call('del', KEYS[1]) else return 0 end"; + +const SHARED_ENTRY_VERSION = 1; + +/** Stellar amounts: up to 7 decimal places, no sign or exponent. */ +const AMOUNT_PATTERN = /^\d{1,19}(\.\d{1,7})?$/; + +function readIntEnv(name, fallback) { + const raw = Number.parseInt(process.env[name] ?? '', 10); + return Number.isFinite(raw) && raw > 0 ? raw : fallback; +} + +const sleep = (ms) => new Promise((resolve) => { + const timer = setTimeout(resolve, ms); + timer.unref?.(); +}); + +/** + * Default shape check for a cached exchange-rate quote. Guards against + * corrupted or tampered Redis content reaching payers as a send_max. + */ +export function isValidQuote(data) { + return ( + data !== null && + typeof data === 'object' && + typeof data.sourceAsset === 'string' && + typeof data.sourceAmount === 'string' && + AMOUNT_PATTERN.test(data.sourceAmount) && + typeof data.sendMax === 'string' && + AMOUNT_PATTERN.test(data.sendMax) && + Array.isArray(data.path) + ); +} + +/** + * Minimal Redis lock with owner tokens and TTL-based expiry. + */ +export class RedisLock { + constructor(client, { prefix = 'exrate:lock:' } = {}) { + this.client = client; + this.prefix = prefix; + } + + /** @returns {Promise} owner token when acquired, else null */ + async acquire(name, ttlMs) { + const token = randomUUID(); + const reply = await this.client.sendCommand([ + 'SET', this.prefix + name, token, 'PX', String(ttlMs), 'NX', + ]); + return reply === 'OK' ? token : null; + } + + /** @returns {Promise} whether this token still owned the lock */ + async release(name, token) { + const reply = await this.client.sendCommand([ + 'EVAL', RELEASE_LOCK_SCRIPT, '1', this.prefix + name, token, + ]); + return Number(reply) === 1; + } +} + +/** + * Coordinates quote loads across API instances. See module docs. + */ +export class ExchangeRateCoordinator { + /** + * @param {object} opts + * @param {object} opts.redisClient node-redis client (must expose sendCommand) + * @param {number} [opts.sharedTtlMs] how long a shared quote is reusable + * @param {number} [opts.lockTtlMs] lock lease; should exceed a typical Horizon call + * @param {number} [opts.waitTimeoutMs] max time a follower waits before querying directly + * @param {number} [opts.pollIntervalMs] follower poll cadence + * @param {string} [opts.keyPrefix] + * @param {(data: unknown) => boolean} [opts.validate] shared-entry validator + * @param {object|null} [opts.metrics] + */ + constructor({ + redisClient, + sharedTtlMs = readIntEnv('EXCHANGE_RATE_CACHE_TTL_MS', 30_000), + lockTtlMs = readIntEnv('EXCHANGE_RATE_LOCK_TTL_MS', 5_000), + waitTimeoutMs = readIntEnv('EXCHANGE_RATE_LOCK_WAIT_MS', 2_000), + pollIntervalMs = readIntEnv('EXCHANGE_RATE_LOCK_POLL_MS', 50), + keyPrefix = 'exrate:', + validate = isValidQuote, + metrics = DEFAULT_METRICS, + } = {}) { + if (!redisClient || typeof redisClient.sendCommand !== 'function') { + throw new TypeError('ExchangeRateCoordinator requires a redis client with sendCommand'); + } + this.client = redisClient; + this.sharedTtlMs = sharedTtlMs; + this.lockTtlMs = lockTtlMs; + this.waitTimeoutMs = waitTimeoutMs; + this.pollIntervalMs = pollIntervalMs; + this.quotePrefix = `${keyPrefix}quote:`; + this.lock = new RedisLock(redisClient, { prefix: `${keyPrefix}lock:` }); + this.validate = validate; + this.metrics = metrics; + } + + /** Whether the underlying client can currently be used. */ + get available() { + return this.client.isOpen !== false; + } + + async _readShared(key) { + const raw = await this.client.sendCommand(['GET', this.quotePrefix + key]); + if (raw === null || raw === undefined) { + this.metrics?.shared?.inc?.({ result: 'miss' }); + return null; + } + try { + const entry = JSON.parse(String(raw)); + if (entry?.v === SHARED_ENTRY_VERSION && this.validate(entry.data)) { + this.metrics?.shared?.inc?.({ result: 'hit' }); + return entry.data; + } + } catch { + // fall through: treat unparsable content as a miss + } + this.metrics?.shared?.inc?.({ result: 'invalid' }); + logger.warn({ keyPrefix: key.slice(0, 8) }, 'Ignoring invalid shared exchange-rate cache entry'); + return null; + } + + async _writeShared(key, data) { + const payload = JSON.stringify({ v: SHARED_ENTRY_VERSION, insertedAt: Date.now(), data }); + await this.client.sendCommand([ + 'SET', this.quotePrefix + key, payload, 'PX', String(this.sharedTtlMs), + ]); + } + + /** Remove a quote from the shared store (cross-instance invalidation). */ + async invalidate(key) { + if (!this.available) return false; + const removed = await this.client.sendCommand(['DEL', this.quotePrefix + key]); + return Number(removed) > 0; + } + + /** + * Run `loader` at most once across all coordinated instances for `key` + * (subject to lock TTL and wait timeout — see module docs). + * + * @template T + * @param {string} key already-hashed cache key + * @param {() => Promise} loader queries Horizon + * @returns {Promise<{data: T, source: 'shared'|'leader'|'fallback'}>} + */ + async load(key, loader) { + if (!this.available) { + return { data: await loader(), source: 'fallback' }; + } + + const startedAt = Date.now(); + const deadline = startedAt + this.waitTimeoutMs; + const observeWait = (outcome) => + this.metrics?.wait?.observe?.({ outcome }, (Date.now() - startedAt) / 1000); + + // Only Redis operations run inside this try; the loader is always + // invoked outside it so loader errors are never mistaken for Redis ones. + let token = null; + try { + for (;;) { + const shared = await this._readShared(key); + if (shared !== null) { + observeWait('shared_hit'); + return { data: shared, source: 'shared' }; + } + + token = await this.lock.acquire(key, this.lockTtlMs); + if (token) { + this.metrics?.lock?.inc?.({ result: 'acquired' }); + // Double-check: a previous leader may have written between our + // GET and SET NX. + const written = await this._readShared(key); + if (written !== null) { + await this._release(key, token); + observeWait('shared_hit'); + return { data: written, source: 'shared' }; + } + observeWait('acquired'); + break; + } + + this.metrics?.lock?.inc?.({ result: 'contended' }); + if (Date.now() >= deadline) { + observeWait('timeout'); + this.metrics?.fallback?.inc?.({ reason: 'wait_timeout' }); + logger.warn( + { keyPrefix: key.slice(0, 8), waitedMs: Date.now() - startedAt }, + 'Exchange-rate lock wait timed out; querying Horizon directly', + ); + break; + } + await sleep(this.pollIntervalMs); + } + } catch (err) { + // Redis failure during coordination: fail open. + if (token) await this._release(key, token); + token = null; + observeWait('error'); + this.metrics?.lock?.inc?.({ result: 'error' }); + this.metrics?.fallback?.inc?.({ reason: 'redis_error' }); + logger.warn({ err: err?.message }, 'Exchange-rate coordination unavailable; querying Horizon directly'); + } + + if (!token) { + return { data: await loader(), source: 'fallback' }; + } + + // Leader path. The loader's own errors propagate to the caller; the lock + // is always released so followers can take over immediately. + try { + const data = await loader(); + try { + await this._writeShared(key, data); + } catch (err) { + logger.warn({ err: err?.message }, 'Failed to publish exchange-rate quote to shared cache'); + } + return { data, source: 'leader' }; + } finally { + await this._release(key, token); + } + } + + async _release(key, token) { + try { + const released = await this.lock.release(key, token); + if (!released) { + logger.warn( + { keyPrefix: key.slice(0, 8), lockTtlMs: this.lockTtlMs }, + 'Exchange-rate lock expired before release; consider raising EXCHANGE_RATE_LOCK_TTL_MS', + ); + } + } catch (err) { + // The lease expires on its own; nothing else to do. + logger.warn({ err: err?.message }, 'Failed to release exchange-rate lock'); + } + } +} diff --git a/backend/src/lib/exchange-rate-coordinator.test.js b/backend/src/lib/exchange-rate-coordinator.test.js new file mode 100644 index 00000000..e5a07f30 --- /dev/null +++ b/backend/src/lib/exchange-rate-coordinator.test.js @@ -0,0 +1,284 @@ +import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest'; + +vi.mock('./logger.js', () => ({ + logger: { debug: vi.fn(), info: vi.fn(), warn: vi.fn(), error: vi.fn() }, +})); + +import { logger } from './logger.js'; +import { + ExchangeRateCoordinator, + RedisLock, + isValidQuote, +} from './exchange-rate-coordinator.js'; +import { createFakeRedis } from '../../tests/helpers/fake-redis.js'; + +const QUOTE = Object.freeze({ + sourceAsset: 'XLM', + sourceAmount: '0.5000000', + sendMax: '0.5050000', + path: [], +}); + +const makeMetrics = () => ({ + lock: { inc: vi.fn() }, + wait: { observe: vi.fn() }, + shared: { inc: vi.fn() }, + fallback: { inc: vi.fn() }, +}); + +const make = (redis, opts = {}) => + new ExchangeRateCoordinator({ + redisClient: redis, + sharedTtlMs: 1_000, + lockTtlMs: 500, + waitTimeoutMs: 300, + pollIntervalMs: 10, + metrics: makeMetrics(), + ...opts, + }); + +describe('isValidQuote', () => { + it('accepts a well-formed quote', () => { + expect(isValidQuote(QUOTE)).toBe(true); + }); + + it.each([ + null, + 'string', + { ...QUOTE, sendMax: 123 }, + { ...QUOTE, sendMax: '-1' }, + { ...QUOTE, sendMax: '1e9' }, + { ...QUOTE, sourceAmount: '0.12345678' }, + { ...QUOTE, path: 'x' }, + { ...QUOTE, sourceAsset: undefined }, + ])('rejects malformed quote %#', (value) => { + expect(isValidQuote(value)).toBe(false); + }); +}); + +describe('RedisLock', () => { + let redis; + let lock; + + beforeEach(() => { + redis = createFakeRedis(); + lock = new RedisLock(redis, { prefix: 't:' }); + }); + + afterEach(() => vi.useRealTimers()); + + it('grants the lock to exactly one contender', async () => { + const tokens = await Promise.all(Array.from({ length: 20 }, () => lock.acquire('k', 1000))); + expect(tokens.filter(Boolean)).toHaveLength(1); + }); + + it('uses SET NX PX with a random token', async () => { + const token = await lock.acquire('k', 750); + expect(token).toMatch(/^[0-9a-f-]{36}$/); + expect(redis.calls[0]).toEqual(['SET', 't:k', token, 'PX', '750', 'NX']); + }); + + it('only the owner can release', async () => { + const token = await lock.acquire('k', 1000); + expect(await lock.release('k', 'someone-else')).toBe(false); + expect(await lock.acquire('k', 1000)).toBeNull(); + expect(await lock.release('k', token)).toBe(true); + expect(await lock.acquire('k', 1000)).not.toBeNull(); + }); + + it('an expired holder cannot delete the next owner’s lock', async () => { + vi.useFakeTimers(); + const stale = await lock.acquire('k', 100); + vi.advanceTimersByTime(150); + const fresh = await lock.acquire('k', 1000); + expect(fresh).not.toBeNull(); + expect(await lock.release('k', stale)).toBe(false); + expect(await lock.acquire('k', 1000)).toBeNull(); // fresh lock still held + }); +}); + +describe('ExchangeRateCoordinator', () => { + let redis; + + beforeEach(() => { + redis = createFakeRedis(); + vi.clearAllMocks(); + }); + + it('requires a client with sendCommand', () => { + expect(() => new ExchangeRateCoordinator({ redisClient: {} })).toThrow(TypeError); + expect(() => new ExchangeRateCoordinator({})).toThrow(TypeError); + }); + + it('leader loads, publishes to the shared store and releases the lock', async () => { + const c = make(redis); + const loader = vi.fn(async () => QUOTE); + const result = await c.load('k', loader); + expect(result).toEqual({ data: QUOTE, source: 'leader' }); + expect(loader).toHaveBeenCalledTimes(1); + expect(redis.store.has('exrate:lock:k')).toBe(false); + const stored = JSON.parse(redis.store.get('exrate:quote:k').value); + expect(stored).toMatchObject({ v: 1, data: QUOTE }); + expect(c.metrics.lock.inc).toHaveBeenCalledWith({ result: 'acquired' }); + }); + + it('serves a peer’s shared quote without calling the loader', async () => { + await make(redis).load('k', async () => QUOTE); + const loader = vi.fn(); + const c = make(redis); + await expect(c.load('k', loader)).resolves.toEqual({ data: QUOTE, source: 'shared' }); + expect(loader).not.toHaveBeenCalled(); + expect(c.metrics.shared.inc).toHaveBeenCalledWith({ result: 'hit' }); + }); + + it('shared quotes expire after sharedTtlMs', async () => { + vi.useFakeTimers(); + await make(redis).load('k', async () => QUOTE); + vi.advanceTimersByTime(1_001); + const loader = vi.fn(async () => QUOTE); + await make(redis).load('k', loader); + expect(loader).toHaveBeenCalledTimes(1); + vi.useRealTimers(); + }); + + it('followers wait for the leader instead of querying Horizon', async () => { + const leader = make(redis); + const follower = make(redis); + let release; + const slow = new Promise((resolve) => { + release = resolve; + }); + const leaderLoader = vi.fn(() => slow); + const followerLoader = vi.fn(async () => QUOTE); + + const p1 = leader.load('k', leaderLoader); + await new Promise((r) => setTimeout(r, 5)); // leader takes the lock first + const p2 = follower.load('k', followerLoader); + await new Promise((r) => setTimeout(r, 30)); + release(QUOTE); + + const [r1, r2] = await Promise.all([p1, p2]); + expect(r1.source).toBe('leader'); + expect(r2).toEqual({ data: QUOTE, source: 'shared' }); + expect(followerLoader).not.toHaveBeenCalled(); + expect(follower.metrics.lock.inc).toHaveBeenCalledWith({ result: 'contended' }); + expect(follower.metrics.wait.observe).toHaveBeenCalledWith({ outcome: 'shared_hit' }, expect.any(Number)); + }); + + it('a follower takes over when the leader fails', async () => { + const leader = make(redis); + const follower = make(redis); + let fail; + const failing = new Promise((_, reject) => { + fail = reject; + }); + + const p1 = leader.load('k', () => failing); + await new Promise((r) => setTimeout(r, 5)); + const followerLoader = vi.fn(async () => QUOTE); + const p2 = follower.load('k', followerLoader); + await new Promise((r) => setTimeout(r, 20)); + fail(new Error('no path')); + + await expect(p1).rejects.toThrow('no path'); + await expect(p2).resolves.toEqual({ data: QUOTE, source: 'leader' }); + expect(followerLoader).toHaveBeenCalledTimes(1); + }); + + it('a follower takes over when the leader crashes and its lease expires', async () => { + // Simulate a crashed instance: lock held, never released, no quote written. + await redis.sendCommand(['SET', 'exrate:lock:k', 'dead-instance', 'PX', '50', 'NX']); + const loader = vi.fn(async () => QUOTE); + const result = await make(redis).load('k', loader); + expect(result.source).toBe('leader'); + expect(loader).toHaveBeenCalledTimes(1); + }); + + it('falls back to a direct load after waitTimeoutMs', async () => { + await redis.sendCommand(['SET', 'exrate:lock:k', 'slow-peer', 'PX', '10000', 'NX']); + const c = make(redis, { waitTimeoutMs: 50 }); + const loader = vi.fn(async () => QUOTE); + const started = Date.now(); + const result = await c.load('k', loader); + expect(result).toEqual({ data: QUOTE, source: 'fallback' }); + expect(Date.now() - started).toBeGreaterThanOrEqual(45); + expect(c.metrics.fallback.inc).toHaveBeenCalledWith({ reason: 'wait_timeout' }); + expect(logger.warn).toHaveBeenCalledWith(expect.anything(), expect.stringContaining('timed out')); + }); + + it('fails open when Redis errors, calling the loader exactly once', async () => { + redis.setFailure(new Error('ECONNRESET')); + const c = make(redis); + const loader = vi.fn(async () => QUOTE); + await expect(c.load('k', loader)).resolves.toEqual({ data: QUOTE, source: 'fallback' }); + expect(loader).toHaveBeenCalledTimes(1); + expect(c.metrics.fallback.inc).toHaveBeenCalledWith({ reason: 'redis_error' }); + }); + + it('does not retry the loader when the loader itself fails during fallback', async () => { + await redis.sendCommand(['SET', 'exrate:lock:k', 'slow-peer', 'PX', '10000', 'NX']); + const c = make(redis, { waitTimeoutMs: 20 }); + const loader = vi.fn(async () => { + throw new Error('no path'); + }); + await expect(c.load('k', loader)).rejects.toThrow('no path'); + expect(loader).toHaveBeenCalledTimes(1); + expect(c.metrics.fallback.inc).not.toHaveBeenCalledWith({ reason: 'redis_error' }); + }); + + it('still returns the quote if publishing to the shared store fails', async () => { + const c = make(redis); + const loader = vi.fn(async () => { + redis.setFailure(new Error('READONLY')); + return QUOTE; + }); + await expect(c.load('k', loader)).resolves.toEqual({ data: QUOTE, source: 'leader' }); + expect(logger.warn).toHaveBeenCalledWith(expect.anything(), expect.stringContaining('publish')); + }); + + it('skips coordination entirely when the client is closed', async () => { + redis.isOpen = false; + const loader = vi.fn(async () => QUOTE); + await expect(make(redis).load('k', loader)).resolves.toEqual({ data: QUOTE, source: 'fallback' }); + expect(redis.calls).toHaveLength(0); + }); + + it.each([ + ['unparsable JSON', 'not-json{'], + ['wrong version', JSON.stringify({ v: 99, data: QUOTE })], + ['tampered sendMax', JSON.stringify({ v: 1, data: { ...QUOTE, sendMax: '-5' } })], + ['foreign value', 'mocked_hash'], + ])('treats %s in the shared store as a miss', async (_label, raw) => { + redis.store.set('exrate:quote:k', { value: raw, expiresAt: null }); + const c = make(redis); + const loader = vi.fn(async () => QUOTE); + const result = await c.load('k', loader); + expect(result.source).toBe('leader'); + expect(loader).toHaveBeenCalledTimes(1); + expect(c.metrics.shared.inc).toHaveBeenCalledWith({ result: 'invalid' }); + // The leader overwrites the bad entry with a valid one. + expect(JSON.parse(redis.store.get('exrate:quote:k').value).data).toEqual(QUOTE); + }); + + it('invalidate() removes the shared quote', async () => { + const c = make(redis); + await c.load('k', async () => QUOTE); + expect(await c.invalidate('k')).toBe(true); + expect(redis.store.has('exrate:quote:k')).toBe(false); + expect(await c.invalidate('k')).toBe(false); + }); + + it('warns when the lease expired before release', async () => { + vi.useFakeTimers({ toFake: ['Date'] }); + const c = make(redis, { lockTtlMs: 10 }); + await c.load('k', async () => { + vi.setSystemTime(Date.now() + 50); // loader outlives the lease + return QUOTE; + }); + expect(logger.warn).toHaveBeenCalledWith( + expect.objectContaining({ lockTtlMs: 10 }), + expect.stringContaining('EXCHANGE_RATE_LOCK_TTL_MS'), + ); + vi.useRealTimers(); + }); +}); diff --git a/backend/src/lib/path-payment-metrics.js b/backend/src/lib/path-payment-metrics.js index b3cac0c6..ac7954f0 100644 --- a/backend/src/lib/path-payment-metrics.js +++ b/backend/src/lib/path-payment-metrics.js @@ -72,6 +72,63 @@ export const pathPaymentQuoteCacheSize = new client.Gauge({ labelNames: ["cache"], }); +/** + * Exchange-rate cache concurrency control (issue #1445). + * In-process single-flight + cross-instance Redis lock coordination. + */ +export const exchangeRateCacheCoalescedRequests = new client.Counter({ + name: "exchange_rate_cache_coalesced_requests_total", + help: "Quote requests that joined an in-flight load instead of querying Horizon", + labelNames: ["cache"], +}); + +export const exchangeRateCacheInflightLoads = new client.Gauge({ + name: "exchange_rate_cache_inflight_loads", + help: "Exchange-rate quote loads currently in flight in this process", + labelNames: ["cache"], +}); + +export const exchangeRateCacheLoadTimeouts = new client.Counter({ + name: "exchange_rate_cache_load_timeouts_total", + help: "Exchange-rate quote loads that exceeded the load timeout", + labelNames: ["cache"], +}); + +export const exchangeRateCacheStaleWritesPrevented = new client.Counter({ + name: "exchange_rate_cache_stale_writes_prevented_total", + help: "Loads whose result was not cached because the key was invalidated mid-flight", + labelNames: ["cache"], +}); + +/** result: acquired | contended | error */ +export const exchangeRateLockAcquisitions = new client.Counter({ + name: "exchange_rate_lock_acquisitions_total", + help: "Distributed exchange-rate lock acquisition attempts, by result", + labelNames: ["result"], +}); + +/** outcome: shared_hit | acquired | timeout | error */ +export const exchangeRateLockWaitDuration = new client.Histogram({ + name: "exchange_rate_lock_wait_seconds", + help: "Time spent coordinating with other instances before a quote was available", + labelNames: ["outcome"], + buckets: [0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5], +}); + +/** result: hit | miss | invalid | error */ +export const exchangeRateSharedCacheLookups = new client.Counter({ + name: "exchange_rate_shared_cache_lookups_total", + help: "Lookups against the Redis-backed shared exchange-rate quote cache, by result", + labelNames: ["result"], +}); + +/** reason: wait_timeout | redis_error */ +export const exchangeRateCoordinationFallbacks = new client.Counter({ + name: "exchange_rate_coordination_fallbacks_total", + help: "Quote loads that bypassed distributed coordination and queried Horizon directly", + labelNames: ["reason"], +}); + /** Number of intermediate assets in the returned path (0 = direct pair). */ export const pathPaymentQuotePathHops = new client.Histogram({ name: "path_payment_quote_path_hops", @@ -96,6 +153,14 @@ register.registerMetric(pathPaymentQuoteCacheHits); register.registerMetric(pathPaymentQuoteCacheMisses); register.registerMetric(pathPaymentQuoteCacheEvictions); register.registerMetric(pathPaymentQuoteCacheSize); +register.registerMetric(exchangeRateCacheCoalescedRequests); +register.registerMetric(exchangeRateCacheInflightLoads); +register.registerMetric(exchangeRateCacheLoadTimeouts); +register.registerMetric(exchangeRateCacheStaleWritesPrevented); +register.registerMetric(exchangeRateLockAcquisitions); +register.registerMetric(exchangeRateLockWaitDuration); +register.registerMetric(exchangeRateSharedCacheLookups); +register.registerMetric(exchangeRateCoordinationFallbacks); register.registerMetric(pathPaymentQuotePathHops); register.registerMetric(pathPaymentQuoteRate); diff --git a/backend/src/lib/payment-session-rules.js b/backend/src/lib/payment-session-rules.js index ab6b1e59..6797349e 100644 --- a/backend/src/lib/payment-session-rules.js +++ b/backend/src/lib/payment-session-rules.js @@ -18,13 +18,291 @@ * 3. Per-asset limits – merchant-configured min/max for the asset * 4. Allowed issuers – merchant allowlist (when non-empty) * + * Payload sanitization and strict validation (issue #1447) run BEFORE those + * business rules: + * + * 0a. sanitizeSessionPayload – rejects non-object bodies and prototype + * pollution keys, strips control / bidi / zero-width characters from + * string fields, NFC-normalizes them and enforces field length caps. + * 0b. validateSessionAsset – asset code must be 1-12 alphanumerics. + * 0c. validateSessionAmount – finite, positive, <= 7 decimal places and + * within Stellar's int64 stroop range. + * * The module is intentionally free of I/O, logging and metrics so it can be - * unit-tested exhaustively and safely reused. + * unit-tested exhaustively and safely reused. Metrics, logging and health + * telemetry live in payment-session-validator.js (issue #1448). */ import { isValidStellarPublicKey } from "./stellar.js"; import { resolveAssetIssuer } from "../constants/assetConstants.js"; +/** Largest amount representable on Stellar: (2^63 - 1) stroops. */ +const MAX_STELLAR_AMOUNT = 922337203685.4775807; + +/** Stellar amounts carry at most 7 decimal places (1 stroop = 1e-7). */ +const STELLAR_AMOUNT_DECIMALS = 7; + +const ASSET_CODE_PATTERN = /^[A-Z0-9]{1,12}$/; + +/** Keys that can mutate an object's prototype chain when copied/merged. */ +const FORBIDDEN_KEYS = new Set(["__proto__", "constructor", "prototype"]); + +/** Nested objects deeper than this are not walked for forbidden keys. */ +const MAX_INSPECTION_DEPTH = 8; + +/** + * Maximum lengths (in UTF-16 code units, after sanitization) for free-form + * string fields on a payment session. Generous enough for legitimate use, + * tight enough to stop oversized values reaching the database or the hosted + * checkout page. + */ +const SESSION_FIELD_MAX_LENGTHS = Object.freeze({ + asset: 12, + asset_issuer: 56, + recipient: 256, + description: 1000, + message: 28, + memo: 64, + memo_type: 16, + webhook_url: 2048, + client_id: 128, +}); + +const REQUIRED_STRING_FIELDS = new Set(["asset", "recipient"]); + +// C0/C1 control characters (TAB/LF/CR included — none of these fields are +// multi-line), and DEL. +// eslint-disable-next-line no-control-regex +const CONTROL_CHARS = /[\u0000-\u001F\u007F-\u009F]/g; +// Bidirectional overrides/isolates ("Trojan Source") that can make the +// description or memo render differently from what is stored. +const BIDI_CONTROL_CHARS = /[\u202A-\u202E\u2066-\u2069\u200E\u200F]/g; +// Zero-width characters used to disguise look-alike values. +const ZERO_WIDTH_CHARS = /[\u200B-\u200D\u2060\uFEFF]/g; + +function isPlainObject(value) { + if (value === null || typeof value !== "object" || Array.isArray(value)) { + return false; + } + const proto = Object.getPrototypeOf(value); + return proto === Object.prototype || proto === null; +} + +/** + * Collect the paths of prototype-pollution keys anywhere in `value`. + * Walks own enumerable keys only (JSON.parse creates `__proto__` as an own + * property, so it is visible here) and stops at MAX_INSPECTION_DEPTH. + */ +function findForbiddenKeys(value, path = "", depth = 0, found = []) { + if (value === null || typeof value !== "object" || depth > MAX_INSPECTION_DEPTH) { + return found; + } + for (const key of Object.keys(value)) { + const childPath = path ? `${path}.${key}` : key; + if (FORBIDDEN_KEYS.has(key)) { + found.push(childPath); + continue; + } + findForbiddenKeys(value[key], childPath, depth + 1, found); + } + return found; +} + +/** + * Clean a single string value. + * @returns {{ value: string, changed: boolean, hadBidi: boolean }} + */ +function sanitizeSessionString(raw) { + const hadBidi = BIDI_CONTROL_CHARS.test(raw); + BIDI_CONTROL_CHARS.lastIndex = 0; + const value = raw + .normalize("NFC") + .replace(BIDI_CONTROL_CHARS, "") + .replace(ZERO_WIDTH_CHARS, "") + .replace(CONTROL_CHARS, "") + .trim(); + return { value, changed: value !== raw, hadBidi }; +} + +/** + * Sanitize a payment session request body (issue #1447). + * + * Never mutates the input. Returns a shallow copy in which every known string + * field has been cleaned; non-string values are left for strict validation. + * + * @param {unknown} body + * @returns {{ + * payload: object|null, + * modifiedFields: string[], + * suspicious: string[], + * rejection: {reason: string, message: string, details?: object}|null, + * }} + */ +function sanitizeSessionPayload(body) { + if (!isPlainObject(body)) { + return { + payload: null, + modifiedFields: [], + suspicious: ["malformed_payload"], + rejection: { + reason: "malformed_payload", + message: "Payment session payload must be a JSON object", + }, + }; + } + + const forbidden = findForbiddenKeys(body); + if (forbidden.length > 0) { + return { + payload: null, + modifiedFields: [], + suspicious: ["forbidden_key"], + rejection: { + reason: "forbidden_key", + message: "Payment session payload contains a forbidden key", + details: { fields: forbidden }, + }, + }; + } + + const payload = { ...body }; + const modifiedFields = []; + const suspicious = new Set(); + + for (const [field, maxLength] of Object.entries(SESSION_FIELD_MAX_LENGTHS)) { + if (typeof payload[field] !== "string") { + continue; + } + + const { value, changed, hadBidi } = sanitizeSessionString(payload[field]); + if (hadBidi) { + suspicious.add("bidi_control"); + } + if (changed) { + modifiedFields.push(field); + } + + if (value.length > maxLength) { + suspicious.add("oversized_field"); + return { + payload: null, + modifiedFields, + suspicious: [...suspicious], + rejection: { + reason: "field_too_long", + message: `${field} must be at most ${maxLength} characters`, + details: { field, max_length: maxLength }, + }, + }; + } + + // Optional fields that sanitize down to nothing are treated as absent; + // required ones stay "" so strict validation rejects them explicitly. + payload[field] = + value === "" && !REQUIRED_STRING_FIELDS.has(field) ? undefined : value; + } + + return { + payload, + modifiedFields, + suspicious: [...suspicious], + rejection: null, + }; +} + +/** + * Strict asset-code validation (issue #1447). + * @param {unknown} asset Normalized (uppercase) asset code + * @returns {{reason:"invalid_asset", message}|null} + */ +function validateSessionAsset(asset) { + if (typeof asset !== "string" || !ASSET_CODE_PATTERN.test(asset)) { + return { + reason: "invalid_asset", + message: "asset must be 1-12 alphanumeric characters", + }; + } + return null; +} + +/** + * Strict amount validation (issue #1447). Rejects values Stellar cannot + * represent so they fail here rather than at transaction build time. + * + * @param {unknown} amount + * @returns {{reason:"invalid_amount", message}|null} + */ +function validateSessionAmount(amount) { + if (typeof amount !== "number" || !Number.isFinite(amount) || amount <= 0) { + return { + reason: "invalid_amount", + message: "Amount must be a positive number", + }; + } + + if (amount > MAX_STELLAR_AMOUNT) { + return { + reason: "invalid_amount", + message: `Amount must not exceed ${MAX_STELLAR_AMOUNT}`, + }; + } + + if (Number(amount.toFixed(STELLAR_AMOUNT_DECIMALS)) !== amount) { + return { + reason: "invalid_amount", + message: `Amount must have at most ${STELLAR_AMOUNT_DECIMALS} decimal places`, + }; + } + + return null; +} + +/** + * Coerce a merchant-configured limit bound to a finite number. + * Returns undefined for absent or unusable values (NaN, objects, ...). + */ +function toFiniteBound(value) { + if (value === undefined || value === null || value === "") { + return undefined; + } + const num = typeof value === "number" ? value : Number(value); + return Number.isFinite(num) ? num : undefined; +} + +/** + * Report problems with a merchant's payment_limits entry for `rawAsset` + * without changing validation outcome. Used by the validator to surface + * misconfiguration through metrics/logs (issue #1448). + * + * @returns {string[]} anomaly kinds: "invalid_min" | "invalid_max" | "min_greater_than_max" + */ +function inspectPaymentLimitsConfig({ rawAsset, paymentLimits }) { + if (!paymentLimits || typeof paymentLimits !== "object") { + return []; + } + if (!Object.hasOwn(paymentLimits, rawAsset)) { + return []; + } + const assetLimits = paymentLimits[rawAsset]; + if (!assetLimits || typeof assetLimits !== "object") { + return ["invalid_entry"]; + } + + const anomalies = []; + const min = toFiniteBound(assetLimits.min); + const max = toFiniteBound(assetLimits.max); + if (assetLimits.min !== undefined && assetLimits.min !== null && min === undefined) { + anomalies.push("invalid_min"); + } + if (assetLimits.max !== undefined && assetLimits.max !== null && max === undefined) { + anomalies.push("invalid_max"); + } + if (min !== undefined && max !== undefined && min > max) { + anomalies.push("min_greater_than_max"); + } + return anomalies; +} + /** * Resolve and validate the asset issuer for a payment session. * @@ -77,29 +355,39 @@ function validatePerAssetLimits({ rawAsset, amount, paymentLimits }) { return null; } + // Own-property lookup only: an asset code such as "constructor" must never + // resolve to something inherited from Object.prototype. + if (typeof rawAsset !== "string" || !Object.hasOwn(paymentLimits, rawAsset)) { + return null; + } const assetLimits = paymentLimits[rawAsset]; - if (!assetLimits) { + if (!assetLimits || typeof assetLimits !== "object") { return null; } - if (assetLimits.min !== undefined && amount < assetLimits.min) { + // Non-numeric bounds are ignored here and reported separately by + // inspectPaymentLimitsConfig so they can be alerted on. + const min = toFiniteBound(assetLimits.min); + const max = toFiniteBound(assetLimits.max); + + if (min !== undefined && amount < min) { return { reason: "below_min", message: `Amount is below the minimum for ${rawAsset}`, details: { - min: assetLimits.min, - delta: Number((assetLimits.min - amount).toFixed(7)), + min, + delta: Number((min - amount).toFixed(7)), }, }; } - if (assetLimits.max !== undefined && amount > assetLimits.max) { + if (max !== undefined && amount > max) { return { reason: "above_max", message: `Amount exceeds the maximum for ${rawAsset}`, details: { - max: assetLimits.max, - delta: Number((amount - assetLimits.max).toFixed(7)), + max, + delta: Number((amount - max).toFixed(7)), }, }; } @@ -126,7 +414,8 @@ function validateAllowedIssuers({ asset, assetIssuer, allowedIssuers }) { return null; } - if (!assetIssuer || !allowedIssuers.includes(assetIssuer)) { + const allowed = allowedIssuers.filter((issuer) => typeof issuer === "string"); + if (!assetIssuer || !allowed.includes(assetIssuer)) { return { reason: "issuer_not_allowed", message: "asset_issuer is not in the merchant's list of allowed issuers", @@ -137,6 +426,12 @@ function validateAllowedIssuers({ asset, assetIssuer, allowedIssuers }) { } export { + MAX_STELLAR_AMOUNT, + SESSION_FIELD_MAX_LENGTHS, + sanitizeSessionPayload, + validateSessionAsset, + validateSessionAmount, + inspectPaymentLimitsConfig, resolveAndValidateIssuer, validatePerAssetLimits, validateAllowedIssuers, diff --git a/backend/src/lib/payment-session-rules.test.js b/backend/src/lib/payment-session-rules.test.js index ab81e177..6150f446 100644 --- a/backend/src/lib/payment-session-rules.test.js +++ b/backend/src/lib/payment-session-rules.test.js @@ -11,6 +11,12 @@ vi.mock("../constants/assetConstants.js", () => ({ })); import { + MAX_STELLAR_AMOUNT, + SESSION_FIELD_MAX_LENGTHS, + sanitizeSessionPayload, + validateSessionAsset, + validateSessionAmount, + inspectPaymentLimitsConfig, resolveAndValidateIssuer, validatePerAssetLimits, validateAllowedIssuers, @@ -188,3 +194,214 @@ describe("validateAllowedIssuers (issue #1087)", () => { expect(rejection.reason).toBe("issuer_not_allowed"); }); }); + +describe("sanitizeSessionPayload (issue #1447)", () => { + const base = { amount: 10, asset: "USDC", asset_issuer: VALID_ISSUER, recipient: VALID_ISSUER }; + + it("returns an equal copy for a clean payload without mutating the input", () => { + const input = { ...base, description: "Order #42" }; + const result = sanitizeSessionPayload(input); + expect(result.rejection).toBeNull(); + expect(result.payload).toEqual(input); + expect(result.payload).not.toBe(input); + expect(result.modifiedFields).toEqual([]); + expect(result.suspicious).toEqual([]); + }); + + it.each([null, undefined, "string", 42, [], [base], new Date()])( + "rejects non-plain-object body %#", + (body) => { + const result = sanitizeSessionPayload(body); + expect(result.payload).toBeNull(); + expect(result.rejection.reason).toBe("malformed_payload"); + expect(result.suspicious).toContain("malformed_payload"); + }, + ); + + it("accepts null-prototype objects", () => { + const body = Object.assign(Object.create(null), base); + expect(sanitizeSessionPayload(body).rejection).toBeNull(); + }); + + it("rejects a top-level __proto__ key created by JSON.parse", () => { + const body = JSON.parse(`{"amount":1,"asset":"XLM","recipient":"x","__proto__":{"polluted":true}}`); + const result = sanitizeSessionPayload(body); + expect(result.rejection.reason).toBe("forbidden_key"); + expect(result.rejection.details.fields).toEqual(["__proto__"]); + expect(result.suspicious).toEqual(["forbidden_key"]); + expect({}.polluted).toBeUndefined(); + }); + + it("reports nested forbidden key paths (metadata, branding, arrays)", () => { + const body = { + ...base, + metadata: { a: { constructor: { prototype: {} } } }, + branding_overrides: JSON.parse(`{"__proto__":{}}`), + tags: [{ prototype: 1 }], + }; + const result = sanitizeSessionPayload(body); + expect(result.rejection.details.fields).toEqual([ + "metadata.a.constructor", + "branding_overrides.__proto__", + "tags.0.prototype", + ]); + }); + + it("does not walk beyond the inspection depth limit", () => { + let deep = { __proto_safe: true }; + let cursor = deep; + for (let i = 0; i < 20; i++) { + cursor.next = {}; + cursor = cursor.next; + } + cursor.constructor = {}; // own key, far below the depth cap + expect(() => sanitizeSessionPayload({ ...base, metadata: deep })).not.toThrow(); + }); + + it("strips control, zero-width and bidi characters and records the fields", () => { + const result = sanitizeSessionPayload({ + ...base, + description: "Pay\u0000 now\u200B\u202E gnp.exe", + client_id: "abc\t\n", + }); + expect(result.rejection).toBeNull(); + expect(result.payload.description).toBe("Pay now gnp.exe"); + expect(result.payload.client_id).toBe("abc"); + expect(result.modifiedFields).toEqual(["description", "client_id"]); + expect(result.suspicious).toEqual(["bidi_control"]); + }); + + it("NFC-normalizes strings so look-alike encodings compare equal", () => { + const decomposed = "Cafe\u0301"; + const result = sanitizeSessionPayload({ ...base, description: decomposed }); + expect(result.payload.description).toBe("Caf\u00e9"); + expect(result.modifiedFields).toContain("description"); + }); + + it("drops optional fields that sanitize to empty, keeps required ones as empty", () => { + const result = sanitizeSessionPayload({ + ...base, + recipient: "\u200B", + description: " \u0007 ", + }); + expect(result.payload.recipient).toBe(""); + expect(result.payload.description).toBeUndefined(); + }); + + it("leaves non-string values untouched for strict validation to judge", () => { + const result = sanitizeSessionPayload({ ...base, amount: "10", description: 5 }); + expect(result.payload.amount).toBe("10"); + expect(result.payload.description).toBe(5); + }); + + it.each(Object.entries(SESSION_FIELD_MAX_LENGTHS))( + "rejects %s longer than %i characters", + (field, max) => { + const result = sanitizeSessionPayload({ ...base, [field]: "A".repeat(max + 1) }); + expect(result.rejection).toEqual({ + reason: "field_too_long", + message: `${field} must be at most ${max} characters`, + details: { field, max_length: max }, + }); + expect(result.suspicious).toContain("oversized_field"); + }, + ); + + it("measures length after stripping, so padding with invisible chars cannot bypass or trip the cap", () => { + const max = SESSION_FIELD_MAX_LENGTHS.client_id; + const padded = "A".repeat(max) + "\u200B".repeat(50); + expect(sanitizeSessionPayload({ ...base, client_id: padded }).rejection).toBeNull(); + }); +}); + +describe("validateSessionAsset (issue #1447)", () => { + it.each(["XLM", "USDC", "A", "ABCDEFGHIJKL", "USD1"])("accepts %s", (asset) => { + expect(validateSessionAsset(asset)).toBeNull(); + }); + + it.each(["", "ABCDEFGHIJKLM", "US-DC", "US DC", "usdc", "ÜSDC", null, 1, undefined])( + "rejects %j", + (asset) => { + expect(validateSessionAsset(asset)).toEqual({ + reason: "invalid_asset", + message: "asset must be 1-12 alphanumeric characters", + }); + }, + ); +}); + +describe("validateSessionAmount (issue #1447)", () => { + it.each([0.0000001, 1, 10.5, 1234567.1234567, MAX_STELLAR_AMOUNT])("accepts %d", (amount) => { + expect(validateSessionAmount(amount)).toBeNull(); + }); + + it.each([0, -1, NaN, Infinity, -Infinity, "10", null, undefined, {}])( + "rejects non-positive / non-numeric %j", + (amount) => { + expect(validateSessionAmount(amount)?.message).toBe("Amount must be a positive number"); + }, + ); + + it("rejects amounts above the int64 stroop range", () => { + expect(validateSessionAmount(MAX_STELLAR_AMOUNT * 2)?.message).toMatch(/must not exceed/); + expect(validateSessionAmount(1e21)?.reason).toBe("invalid_amount"); + }); + + it("rejects more than 7 decimal places, including float artifacts", () => { + expect(validateSessionAmount(0.00000001)?.message).toMatch(/7 decimal places/); + expect(validateSessionAmount(0.1 + 0.2)?.message).toMatch(/7 decimal places/); + }); +}); + +describe("inspectPaymentLimitsConfig (issue #1448)", () => { + it("reports nothing for valid, absent or unrelated config", () => { + expect(inspectPaymentLimitsConfig({ rawAsset: "USDC", paymentLimits: null })).toEqual([]); + expect(inspectPaymentLimitsConfig({ rawAsset: "USDC", paymentLimits: { XLM: { min: "x" } } })).toEqual([]); + expect(inspectPaymentLimitsConfig({ rawAsset: "USDC", paymentLimits: { USDC: { min: 1, max: "5" } } })).toEqual([]); + }); + + it("flags non-numeric bounds, inverted ranges and malformed entries", () => { + expect(inspectPaymentLimitsConfig({ rawAsset: "USDC", paymentLimits: { USDC: { min: "abc", max: {} } } })) + .toEqual(["invalid_min", "invalid_max"]); + expect(inspectPaymentLimitsConfig({ rawAsset: "USDC", paymentLimits: { USDC: { min: 10, max: 1 } } })) + .toEqual(["min_greater_than_max"]); + expect(inspectPaymentLimitsConfig({ rawAsset: "USDC", paymentLimits: { USDC: 5 } })) + .toEqual(["invalid_entry"]); + }); + + it("ignores inherited properties", () => { + expect(inspectPaymentLimitsConfig({ rawAsset: "constructor", paymentLimits: {} })).toEqual([]); + }); +}); + +describe("validatePerAssetLimits hardening (issue #1447)", () => { + it("never resolves limits from Object.prototype", () => { + for (const rawAsset of ["constructor", "__proto__", "toString", "hasOwnProperty"]) { + expect(validatePerAssetLimits({ rawAsset, amount: 1, paymentLimits: {} })).toBeNull(); + } + }); + + it("coerces numeric-string bounds (legacy configs) and ignores unusable ones", () => { + expect( + validatePerAssetLimits({ rawAsset: "USDC", amount: 0.5, paymentLimits: { USDC: { min: "1" } } })?.reason, + ).toBe("below_min"); + expect( + validatePerAssetLimits({ rawAsset: "USDC", amount: 0.5, paymentLimits: { USDC: { min: "abc", max: null } } }), + ).toBeNull(); + }); + + it("ignores non-object asset entries", () => { + expect(validatePerAssetLimits({ rawAsset: "USDC", amount: 1, paymentLimits: { USDC: 5 } })).toBeNull(); + }); +}); + +describe("validateAllowedIssuers hardening (issue #1447)", () => { + it("ignores non-string allowlist entries rather than matching them", () => { + const rejection = validateAllowedIssuers({ + asset: "USDC", + assetIssuer: VALID_ISSUER, + allowedIssuers: [{ toString: () => VALID_ISSUER }, 42, null], + }); + expect(rejection?.reason).toBe("issuer_not_allowed"); + }); +}); diff --git a/backend/src/lib/payment-session-validator-metrics.js b/backend/src/lib/payment-session-validator-metrics.js new file mode 100644 index 00000000..7030f768 --- /dev/null +++ b/backend/src/lib/payment-session-validator-metrics.js @@ -0,0 +1,115 @@ +import client from "prom-client"; + +/** + * Payment Session Validator metrics (issue #1448). + * + * Tracks the validation stage that runs before a payment session is + * persisted (see payment-session-validator.js): + * + * - evaluation outcomes and latency, per call site + * - WHICH rule rejected a session and WHY + * - payload sanitization activity and suspicious-payload signals + * - merchant payment_limits misconfiguration + * - rolling-window health state for alerting + * + * Label values are all drawn from fixed, server-defined sets — never from + * request data such as the asset code — so a client cannot inflate series + * cardinality. + * + * The metrics live in their own registry so they can be unit-tested in + * isolation; the /metrics endpoint merges this registry with the main one. + */ + +const register = new client.Registry(); + +register.setDefaultLabels({ + app: "stellar-payment-api", +}); + +/** + * source: http | service | unknown + * outcome: accepted | rejected | error + */ +export const sessionValidatorEvaluationsTotal = new client.Counter({ + name: "payment_session_validator_evaluations_total", + help: "Total number of payment session validations, by call site and outcome", + labelNames: ["source", "outcome"], +}); + +/** + * rule: sanitization | payload | issuer | limits | allowlist + * reason: the rejection reason emitted by payment-session-rules.js + */ +export const sessionValidatorRejectionsTotal = new client.Counter({ + name: "payment_session_validator_rejections_total", + help: "Total number of payment sessions rejected by the validator, by rule and reason", + labelNames: ["source", "rule", "reason"], +}); + +export const sessionValidatorDuration = new client.Histogram({ + name: "payment_session_validator_duration_seconds", + help: "Time spent validating a payment session in seconds", + labelNames: ["source", "outcome"], + buckets: [0.0001, 0.0005, 0.001, 0.0025, 0.005, 0.01, 0.025, 0.05, 0.1], +}); + +/** field: one of the sanitized string fields (see SESSION_FIELD_MAX_LENGTHS) */ +export const sessionValidatorSanitizedFieldsTotal = new client.Counter({ + name: "payment_session_validator_sanitized_fields_total", + help: "Total number of payload fields altered by sanitization, by field", + labelNames: ["field"], +}); + +/** signal: forbidden_key | malformed_payload | bidi_control | oversized_field */ +export const sessionValidatorSuspiciousPayloadsTotal = new client.Counter({ + name: "payment_session_validator_suspicious_payloads_total", + help: "Total number of payment session payloads carrying a suspicious signal", + labelNames: ["signal"], +}); + +/** kind: invalid_entry | invalid_min | invalid_max | min_greater_than_max */ +export const sessionValidatorConfigAnomaliesTotal = new client.Counter({ + name: "payment_session_validator_config_anomalies_total", + help: "Total number of merchant payment_limits misconfigurations encountered during validation", + labelNames: ["kind"], +}); + +/** + * Rolling-window health gauges. Their values are refreshed at scrape time via + * the collect hook installed by payment-session-validator.js so they decay + * correctly when traffic stops. + * + * health_state: 0 = healthy, 1 = degraded, 2 = unhealthy + */ +export const sessionValidatorHealthState = new client.Gauge({ + name: "payment_session_validator_health_state", + help: "Payment session validator health (0 = healthy, 1 = degraded, 2 = unhealthy)", +}); + +export const sessionValidatorRejectionRatio = new client.Gauge({ + name: "payment_session_validator_rejection_ratio", + help: "Share of validations rejected over the rolling health window", +}); + +export const sessionValidatorErrorRatio = new client.Gauge({ + name: "payment_session_validator_error_ratio", + help: "Share of validations that failed with an internal error over the rolling health window", +}); + +export const sessionValidatorLastEvaluationTimestamp = new client.Gauge({ + name: "payment_session_validator_last_evaluation_timestamp_seconds", + help: "Unix time of the most recent payment session validation", +}); + +register.registerMetric(sessionValidatorEvaluationsTotal); +register.registerMetric(sessionValidatorRejectionsTotal); +register.registerMetric(sessionValidatorDuration); +register.registerMetric(sessionValidatorSanitizedFieldsTotal); +register.registerMetric(sessionValidatorSuspiciousPayloadsTotal); +register.registerMetric(sessionValidatorConfigAnomaliesTotal); +register.registerMetric(sessionValidatorHealthState); +register.registerMetric(sessionValidatorRejectionRatio); +register.registerMetric(sessionValidatorErrorRatio); +register.registerMetric(sessionValidatorLastEvaluationTimestamp); + +export { register as paymentSessionValidatorRegister }; diff --git a/backend/src/lib/payment-session-validator.js b/backend/src/lib/payment-session-validator.js new file mode 100644 index 00000000..88925706 --- /dev/null +++ b/backend/src/lib/payment-session-validator.js @@ -0,0 +1,301 @@ +/** + * payment-session-validator.js + * + * Instrumented entry point for payment session validation (issues #1447, + * #1448). Wraps the pure rules in payment-session-rules.js with: + * + * - payload sanitization + strict validation, run before business rules + * - Prometheus metrics (payment-session-validator-metrics.js) + * - structured logging of rejections and suspicious payloads + * - a rolling-window health monitor exposed via /health and gauges + * + * Rule order (first failure wins): + * + * sanitization → payload (asset, amount) → issuer → limits → allowlist + * + * All checks are synchronous and free of network I/O, so callers should run + * this before anything expensive (e.g. on-chain issuer lookups). + * + * Logs never include the raw payload — only the rule, reason, call site and + * merchant id — so attacker-controlled content does not reach log sinks. + */ + +import { logger } from "./logger.js"; +import { + sanitizeSessionPayload, + validateSessionAsset, + validateSessionAmount, + inspectPaymentLimitsConfig, + resolveAndValidateIssuer, + validatePerAssetLimits, + validateAllowedIssuers, +} from "./payment-session-rules.js"; +import { + sessionValidatorEvaluationsTotal, + sessionValidatorRejectionsTotal, + sessionValidatorDuration, + sessionValidatorSanitizedFieldsTotal, + sessionValidatorSuspiciousPayloadsTotal, + sessionValidatorConfigAnomaliesTotal, + sessionValidatorHealthState, + sessionValidatorRejectionRatio, + sessionValidatorErrorRatio, + sessionValidatorLastEvaluationTimestamp, +} from "./payment-session-validator-metrics.js"; + +const KNOWN_SOURCES = new Set(["http", "service"]); + +const HEALTH_STATE_VALUES = { healthy: 0, degraded: 1, unhealthy: 2 }; + +function readNumberEnv(name, fallback) { + const raw = Number.parseFloat(process.env[name] ?? ""); + return Number.isFinite(raw) && raw >= 0 ? raw : fallback; +} + +export const DEFAULT_HEALTH_OPTIONS = Object.freeze({ + windowMs: readNumberEnv("PAYMENT_SESSION_VALIDATOR_HEALTH_WINDOW_MS", 300_000), + bucketMs: 5_000, + minSamples: readNumberEnv("PAYMENT_SESSION_VALIDATOR_HEALTH_MIN_SAMPLES", 20), + errorRatioThreshold: readNumberEnv("PAYMENT_SESSION_VALIDATOR_ERROR_RATIO_THRESHOLD", 0.05), + rejectionRatioThreshold: readNumberEnv("PAYMENT_SESSION_VALIDATOR_REJECTION_RATIO_THRESHOLD", 0.5), + suspiciousThreshold: readNumberEnv("PAYMENT_SESSION_VALIDATOR_SUSPICIOUS_THRESHOLD", 10), +}); + +/** + * Rolling-window health tracker. Counts are kept in fixed-size time buckets + * so memory stays constant regardless of traffic. + */ +export class SessionValidatorHealthMonitor { + constructor(options = {}, now = () => Date.now()) { + this.options = { ...DEFAULT_HEALTH_OPTIONS, ...options }; + this.now = now; + this.bucketCount = Math.max(1, Math.ceil(this.options.windowMs / this.options.bucketMs)); + this.reset(); + } + + reset() { + this.buckets = Array.from({ length: this.bucketCount }, () => ({ + slot: -1, + accepted: 0, + rejected: 0, + errors: 0, + suspicious: 0, + })); + this.lastEvaluationAt = null; + } + + _bucketFor(time) { + const slot = Math.floor(time / this.options.bucketMs); + const bucket = this.buckets[slot % this.bucketCount]; + if (bucket.slot !== slot) { + bucket.slot = slot; + bucket.accepted = 0; + bucket.rejected = 0; + bucket.errors = 0; + bucket.suspicious = 0; + } + return bucket; + } + + /** + * @param {"accepted"|"rejected"|"error"} outcome + * @param {{ suspicious?: boolean }} [flags] + */ + record(outcome, { suspicious = false } = {}) { + const time = this.now(); + const bucket = this._bucketFor(time); + if (outcome === "accepted") bucket.accepted++; + else if (outcome === "rejected") bucket.rejected++; + else bucket.errors++; + if (suspicious) bucket.suspicious++; + this.lastEvaluationAt = time; + } + + snapshot() { + const time = this.now(); + const currentSlot = Math.floor(time / this.options.bucketMs); + const oldestSlot = currentSlot - this.bucketCount + 1; + const counts = { accepted: 0, rejected: 0, errors: 0, suspicious: 0 }; + + for (const bucket of this.buckets) { + if (bucket.slot < oldestSlot || bucket.slot > currentSlot) continue; + counts.accepted += bucket.accepted; + counts.rejected += bucket.rejected; + counts.errors += bucket.errors; + counts.suspicious += bucket.suspicious; + } + + const total = counts.accepted + counts.rejected + counts.errors; + const rejectionRatio = total > 0 ? counts.rejected / total : 0; + const errorRatio = total > 0 ? counts.errors / total : 0; + const { minSamples, errorRatioThreshold, rejectionRatioThreshold, suspiciousThreshold } = + this.options; + + const reasons = []; + let status = "healthy"; + if (total >= minSamples && errorRatio >= errorRatioThreshold) { + status = "unhealthy"; + reasons.push("error_ratio_exceeded"); + } + if (total >= minSamples && rejectionRatio >= rejectionRatioThreshold) { + if (status === "healthy") status = "degraded"; + reasons.push("rejection_ratio_exceeded"); + } + if (counts.suspicious >= suspiciousThreshold) { + if (status === "healthy") status = "degraded"; + reasons.push("suspicious_payload_spike"); + } + + return { + status, + reasons, + window_ms: this.bucketCount * this.options.bucketMs, + total, + ...counts, + rejection_ratio: Number(rejectionRatio.toFixed(4)), + error_ratio: Number(errorRatio.toFixed(4)), + last_evaluation_at: + this.lastEvaluationAt === null ? null : new Date(this.lastEvaluationAt).toISOString(), + thresholds: { + min_samples: minSamples, + error_ratio: errorRatioThreshold, + rejection_ratio: rejectionRatioThreshold, + suspicious: suspiciousThreshold, + }, + }; + } +} + +const healthMonitor = new SessionValidatorHealthMonitor(); + +// Refresh rolling-window gauges at scrape time so they decay when idle. +sessionValidatorHealthState.collect = function collectHealthState() { + const snap = healthMonitor.snapshot(); + this.set(HEALTH_STATE_VALUES[snap.status]); + sessionValidatorRejectionRatio.set(snap.rejection_ratio); + sessionValidatorErrorRatio.set(snap.error_ratio); +}; + +/** Current validator health, for /health endpoints. */ +export function getPaymentSessionValidatorHealth() { + return healthMonitor.snapshot(); +} + +/** Test-only: clear the rolling health window. */ +export function resetPaymentSessionValidatorHealth() { + healthMonitor.reset(); +} + +/** + * Validate a payment session request. + * + * @param {object} params + * @param {unknown} params.body Request body (already schema-parsed on HTTP) + * @param {object} params.merchant Merchant record (payment_limits, allowed_issuers, id) + * @param {"http"|"service"} [params.source] Call site, used as a metric label + * @returns {{ ok: true, payload: object, asset: string, assetIssuer: string|null } + * | { ok: false, rejection: { rule: string, reason: string, message: string, details?: object } }} + * @throws Re-throws unexpected internal errors after recording them. + */ +export function validatePaymentSession({ body, merchant, source = "unknown" }) { + const label = KNOWN_SOURCES.has(source) ? source : "unknown"; + const startedAt = process.hrtime.bigint(); + let suspicious = false; + + const finish = (outcome) => { + const seconds = Number(process.hrtime.bigint() - startedAt) / 1e9; + sessionValidatorEvaluationsTotal.inc({ source: label, outcome }); + sessionValidatorDuration.observe({ source: label, outcome }, seconds); + sessionValidatorLastEvaluationTimestamp.set(Date.now() / 1000); + healthMonitor.record(outcome, { suspicious }); + }; + + const reject = (rule, rejection) => { + sessionValidatorRejectionsTotal.inc({ source: label, rule, reason: rejection.reason }); + finish("rejected"); + logger.info( + { merchantId: merchant?.id, source: label, rule, reason: rejection.reason }, + "Payment session rejected by validator", + ); + return { ok: false, rejection: { rule, ...rejection } }; + }; + + try { + const sanitized = sanitizeSessionPayload(body); + + for (const field of sanitized.modifiedFields) { + sessionValidatorSanitizedFieldsTotal.inc({ field }); + } + if (sanitized.suspicious.length > 0) { + suspicious = true; + for (const signal of sanitized.suspicious) { + sessionValidatorSuspiciousPayloadsTotal.inc({ signal }); + } + logger.warn( + { merchantId: merchant?.id, source: label, signals: sanitized.suspicious }, + "Suspicious payment session payload", + ); + } + if (sanitized.rejection) { + return reject("sanitization", sanitized.rejection); + } + + const payload = sanitized.payload; + const asset = payload.asset?.toUpperCase(); + + const payloadRejection = validateSessionAsset(asset) || validateSessionAmount(payload.amount); + if (payloadRejection) { + return reject("payload", payloadRejection); + } + + const { assetIssuer, rejection: issuerRejection } = resolveAndValidateIssuer( + asset, + payload.asset_issuer, + ); + if (issuerRejection) { + return reject("issuer", issuerRejection); + } + + const anomalies = inspectPaymentLimitsConfig({ + rawAsset: payload.asset, + paymentLimits: merchant?.payment_limits, + }); + if (anomalies.length > 0) { + for (const kind of anomalies) { + sessionValidatorConfigAnomaliesTotal.inc({ kind }); + } + logger.warn( + { merchantId: merchant?.id, source: label, anomalies }, + "Merchant payment_limits misconfigured; affected bounds are ignored", + ); + } + + const limitRejection = validatePerAssetLimits({ + rawAsset: payload.asset, + amount: payload.amount, + paymentLimits: merchant?.payment_limits, + }); + if (limitRejection) { + return reject("limits", limitRejection); + } + + const allowlistRejection = validateAllowedIssuers({ + asset, + assetIssuer, + allowedIssuers: merchant?.allowed_issuers, + }); + if (allowlistRejection) { + return reject("allowlist", allowlistRejection); + } + + finish("accepted"); + return { ok: true, payload, asset, assetIssuer }; + } catch (err) { + finish("error"); + logger.error( + { err, merchantId: merchant?.id, source: label }, + "Payment session validator failed unexpectedly", + ); + throw err; + } +} diff --git a/backend/src/lib/payment-session-validator.test.js b/backend/src/lib/payment-session-validator.test.js new file mode 100644 index 00000000..a1b924ed --- /dev/null +++ b/backend/src/lib/payment-session-validator.test.js @@ -0,0 +1,317 @@ +import { describe, it, expect, vi, beforeEach } from "vitest"; +import { readFileSync } from "node:fs"; + +vi.mock("./stellar.js", () => ({ + isValidStellarPublicKey: vi.fn( + (value) => typeof value === "string" && /^G[A-Z2-7]{55}$/.test(value), + ), +})); + +vi.mock("../constants/assetConstants.js", () => ({ + resolveAssetIssuer: vi.fn((_asset, issuer) => issuer || null), +})); + +vi.mock("./logger.js", () => ({ + logger: { debug: vi.fn(), info: vi.fn(), warn: vi.fn(), error: vi.fn() }, +})); + +import { logger } from "./logger.js"; +import { + validatePaymentSession, + getPaymentSessionValidatorHealth, + resetPaymentSessionValidatorHealth, + SessionValidatorHealthMonitor, +} from "./payment-session-validator.js"; +import { paymentSessionValidatorRegister } from "./payment-session-validator-metrics.js"; + +const VALID_ISSUER = "GBBD47IF6LWK7P7MDEVSCWR7DPUWV3NY3DTQEVFL4NAT4AQH3ZLLFLA5"; +const OTHER_ISSUER = "GA5XIGA5C7FBPTVQ3CWHKNC7D2ZBHB24G3KUJG5WZ6S4EYWSSBFVL45T"; + +const validBody = () => ({ + amount: 10, + asset: "USDC", + asset_issuer: VALID_ISSUER, + recipient: VALID_ISSUER, + description: "Order 1", +}); + +async function metricValue(name, labels = {}) { + const metric = paymentSessionValidatorRegister.getSingleMetric(name); + const { values } = await metric.get(); + const match = values.find( + (v) => + !v.metricName?.endsWith("_bucket") && + !v.metricName?.endsWith("_sum") && + Object.entries(labels).every(([k, val]) => v.labels[k] === val), + ); + return match ? match.value : 0; +} + +describe("validatePaymentSession (issues #1447, #1448)", () => { + beforeEach(() => { + paymentSessionValidatorRegister.resetMetrics(); + resetPaymentSessionValidatorHealth(); + vi.clearAllMocks(); + }); + + it("accepts a valid payload and returns the sanitized copy + resolved issuer", async () => { + const result = validatePaymentSession({ body: validBody(), merchant: { id: "m1" }, source: "http" }); + expect(result).toEqual({ + ok: true, + payload: validBody(), + asset: "USDC", + assetIssuer: VALID_ISSUER, + }); + expect( + await metricValue("payment_session_validator_evaluations_total", { source: "http", outcome: "accepted" }), + ).toBe(1); + expect( + await metricValue("payment_session_validator_duration_seconds", { source: "http", outcome: "accepted" }), + ).toBe(1); // _count + }); + + it("uppercases the asset from the sanitized payload", () => { + const result = validatePaymentSession({ + body: { ...validBody(), asset: " usdc​ " }, + merchant: {}, + source: "service", + }); + expect(result.ok).toBe(true); + expect(result.asset).toBe("USDC"); + expect(result.payload.asset).toBe("usdc"); + }); + + it.each([ + ["sanitization", "malformed_payload", () => "not-an-object", {}], + ["sanitization", "forbidden_key", () => JSON.parse(`{"__proto__":{"x":1},"amount":1}`), {}], + ["sanitization", "field_too_long", () => ({ ...validBody(), client_id: "x".repeat(200) }), {}], + ["payload", "invalid_asset", () => ({ ...validBody(), asset: "US$C" }), {}], + ["payload", "invalid_amount", () => ({ ...validBody(), amount: 1.123456789 }), {}], + ["payload", "invalid_amount", () => ({ ...validBody(), amount: "10" }), {}], + ["issuer", "missing_issuer", () => ({ ...validBody(), asset_issuer: undefined }), {}], + ["issuer", "invalid_issuer", () => ({ ...validBody(), asset_issuer: "GNOTVALID" }), {}], + ["limits", "below_min", () => validBody(), { payment_limits: { USDC: { min: 50 } } }], + ["limits", "above_max", () => validBody(), { payment_limits: { USDC: { max: 5 } } }], + ["allowlist", "issuer_not_allowed", () => validBody(), { allowed_issuers: [OTHER_ISSUER] }], + ])("rejects at rule=%s reason=%s and records it", async (rule, reason, makeBody, merchant) => { + const result = validatePaymentSession({ body: makeBody(), merchant: { id: "m1", ...merchant }, source: "http" }); + expect(result.ok).toBe(false); + expect(result.rejection.rule).toBe(rule); + expect(result.rejection.reason).toBe(reason); + expect(typeof result.rejection.message).toBe("string"); + expect( + await metricValue("payment_session_validator_rejections_total", { source: "http", rule, reason }), + ).toBe(1); + expect( + await metricValue("payment_session_validator_evaluations_total", { source: "http", outcome: "rejected" }), + ).toBe(1); + expect(logger.info).toHaveBeenCalledWith( + { merchantId: "m1", source: "http", rule, reason }, + "Payment session rejected by validator", + ); + }); + + it("evaluates rules in order: sanitization before payload before issuer before limits", () => { + const merchant = { payment_limits: { USDC: { max: 1 } }, allowed_issuers: [OTHER_ISSUER] }; + const result = validatePaymentSession({ + body: { ...validBody(), asset_issuer: "bad", amount: 0.123456789 }, + merchant, + }); + expect(result.rejection.rule).toBe("payload"); + const next = validatePaymentSession({ body: { ...validBody(), asset_issuer: "bad" }, merchant }); + expect(next.rejection.rule).toBe("issuer"); + }); + + it("passes limit details through on limit rejections", () => { + const result = validatePaymentSession({ + body: validBody(), + merchant: { payment_limits: { USDC: { max: 4 } } }, + }); + expect(result.rejection.details).toEqual({ max: 4, delta: 6 }); + }); + + it("never logs the raw payload", () => { + validatePaymentSession({ + body: { ...validBody(), description: "secret‮", asset: "!!" }, + merchant: { id: "m1" }, + source: "http", + }); + const logged = JSON.stringify([...logger.info.mock.calls, ...logger.warn.mock.calls]); + expect(logged).not.toContain("secret"); + expect(logged).not.toContain(VALID_ISSUER); + }); + + it("counts sanitized fields and suspicious signals", async () => { + validatePaymentSession({ + body: { ...validBody(), description: "hi‮", client_id: "c\u0000" }, + merchant: { id: "m1" }, + source: "http", + }); + expect(await metricValue("payment_session_validator_sanitized_fields_total", { field: "description" })).toBe(1); + expect(await metricValue("payment_session_validator_sanitized_fields_total", { field: "client_id" })).toBe(1); + expect(await metricValue("payment_session_validator_suspicious_payloads_total", { signal: "bidi_control" })).toBe(1); + expect(logger.warn).toHaveBeenCalledWith( + { merchantId: "m1", source: "http", signals: ["bidi_control"] }, + "Suspicious payment session payload", + ); + }); + + it("records merchant config anomalies without blocking the session", async () => { + const result = validatePaymentSession({ + body: validBody(), + merchant: { id: "m1", payment_limits: { USDC: { min: "abc", max: 5 } } }, + source: "service", + }); + expect(result.rejection?.reason).toBe("above_max"); // valid max still enforced + expect(await metricValue("payment_session_validator_config_anomalies_total", { kind: "invalid_min" })).toBe(1); + expect(logger.warn).toHaveBeenCalledWith( + expect.objectContaining({ anomalies: ["invalid_min"] }), + expect.stringContaining("misconfigured"), + ); + }); + + it("collapses unknown sources into a single label to bound cardinality", async () => { + validatePaymentSession({ body: validBody(), merchant: {}, source: "attacker-controlled-" + Math.random() }); + expect( + await metricValue("payment_session_validator_evaluations_total", { source: "unknown", outcome: "accepted" }), + ).toBe(1); + }); + + it("records internal errors, logs them and rethrows", async () => { + // Force an unexpected failure mid-validation via a throwing getter. + const merchant = { + id: "m1", + get allowed_issuers() { + throw new Error("db row corrupted"); + }, + }; + expect(() => validatePaymentSession({ body: validBody(), merchant, source: "http" })).toThrow( + "db row corrupted", + ); + expect( + await metricValue("payment_session_validator_evaluations_total", { source: "http", outcome: "error" }), + ).toBe(1); + expect(logger.error).toHaveBeenCalled(); + expect(getPaymentSessionValidatorHealth().errors).toBe(1); + }); + + it("refreshes health gauges at scrape time", async () => { + for (let i = 0; i < 25; i++) { + validatePaymentSession({ body: { ...validBody(), asset: "!" }, merchant: {} }); + } + const text = await paymentSessionValidatorRegister.metrics(); + expect(text).toMatch(/payment_session_validator_health_state\{[^}]*\} 1/); + expect(text).toMatch(/payment_session_validator_rejection_ratio\{[^}]*\} 1/); + expect(text).toContain("payment_session_validator_last_evaluation_timestamp_seconds"); + }); +}); + +describe("SessionValidatorHealthMonitor (issue #1448)", () => { + let clock; + const make = (opts = {}) => + new SessionValidatorHealthMonitor( + { + windowMs: 60_000, + bucketMs: 1_000, + minSamples: 10, + errorRatioThreshold: 0.1, + rejectionRatioThreshold: 0.5, + suspiciousThreshold: 3, + ...opts, + }, + () => clock, + ); + + beforeEach(() => { + clock = 1_700_000_000_000; + }); + + it("is healthy with no traffic", () => { + const snap = make().snapshot(); + expect(snap).toMatchObject({ status: "healthy", total: 0, rejection_ratio: 0, last_evaluation_at: null }); + }); + + it("does not alarm below the minimum sample size", () => { + const m = make(); + for (let i = 0; i < 9; i++) m.record("error"); + expect(m.snapshot().status).toBe("healthy"); + }); + + it("goes unhealthy when the error ratio crosses the threshold", () => { + const m = make(); + for (let i = 0; i < 9; i++) m.record("accepted"); + m.record("error"); + m.record("error"); + const snap = m.snapshot(); + expect(snap.status).toBe("unhealthy"); + expect(snap.reasons).toContain("error_ratio_exceeded"); + }); + + it("goes degraded on a high rejection ratio or a suspicious spike", () => { + const m = make(); + for (let i = 0; i < 10; i++) m.record("rejected"); + expect(m.snapshot()).toMatchObject({ status: "degraded", reasons: ["rejection_ratio_exceeded"] }); + + const s = make(); + for (let i = 0; i < 3; i++) s.record("rejected", { suspicious: true }); + expect(s.snapshot()).toMatchObject({ status: "degraded", reasons: ["suspicious_payload_spike"] }); + }); + + it("unhealthy takes precedence but all reasons are reported", () => { + const m = make(); + for (let i = 0; i < 10; i++) m.record("error", { suspicious: true }); + const snap = m.snapshot(); + expect(snap.status).toBe("unhealthy"); + expect(snap.reasons).toEqual(["error_ratio_exceeded", "suspicious_payload_spike"]); + }); + + it("forgets events that fall out of the rolling window", () => { + const m = make(); + for (let i = 0; i < 20; i++) m.record("error"); + expect(m.snapshot().status).toBe("unhealthy"); + clock += 61_000; + const snap = m.snapshot(); + expect(snap.total).toBe(0); + expect(snap.status).toBe("healthy"); + expect(snap.last_evaluation_at).not.toBeNull(); + }); + + it("reuses bucket slots without leaking old counts", () => { + const m = make({ windowMs: 3_000 }); + m.record("error"); + clock += 3_000; // same ring index, new slot + m.record("accepted"); + expect(m.snapshot()).toMatchObject({ total: 1, accepted: 1, errors: 0 }); + }); + + it("keeps memory constant under heavy load", () => { + const m = make(); + for (let i = 0; i < 100_000; i++) { + clock += 7; + m.record(i % 2 ? "accepted" : "rejected"); + } + expect(m.buckets).toHaveLength(60); + expect(m.snapshot().total).toBeLessThanOrEqual(60_000 / 7 + 1); + }); +}); + +describe("alert rules (issue #1448)", () => { + it("only reference metrics that the validator registry exposes", () => { + const rules = readFileSync( + new URL("../../docs/alerts/payment-session-validator.rules.yml", import.meta.url), + "utf8", + ); + const referenced = new Set( + [...rules.matchAll(/\b(payment_session_validator_[a-z_]+)/g)].map(([, name]) => + name.replace(/_(bucket|sum|count)$/, ""), + ), + ); + const registered = new Set( + paymentSessionValidatorRegister.getMetricsAsArray().map((m) => m.name), + ); + expect(referenced.size).toBeGreaterThan(0); + for (const name of referenced) { + expect(registered, `alert rule references unknown metric ${name}`).toContain(name); + } + }); +}); diff --git a/backend/src/lib/request-schemas.js b/backend/src/lib/request-schemas.js index 3804bb83..f1a9c37d 100644 --- a/backend/src/lib/request-schemas.js +++ b/backend/src/lib/request-schemas.js @@ -3,6 +3,7 @@ import { HEX_COLOR_REGEX } from "./branding.js"; import { isValidAssetCode, isValidStellarAccountId, + isValidStellarPublicKey, validateMemo, } from "./stellar.js"; import { resolveAssetIssuer } from "../constants/assetConstants.js"; diff --git a/backend/src/lib/sanitize-metadata.js b/backend/src/lib/sanitize-metadata.js index 441c6144..b52f8850 100644 --- a/backend/src/lib/sanitize-metadata.js +++ b/backend/src/lib/sanitize-metadata.js @@ -7,6 +7,9 @@ const MAX_NESTING_DEPTH = 4; const MAX_STRING_LENGTH = 1000; const MAX_KEYS = 50; +// Keys that rewrite an object's prototype chain when assigned (issue #1447). +const FORBIDDEN_KEYS = new Set(['__proto__', 'constructor', 'prototype']); + // Patterns that indicate potential XSS or injection attacks const DANGEROUS_PATTERNS = [ /]*>.*?<\/script>/gi, @@ -65,6 +68,21 @@ function countKeys(obj) { return count; } +/** + * Whether any own key anywhere in obj is a prototype-pollution key. + */ +function hasForbiddenKey(obj) { + if (typeof obj !== 'object' || obj === null) { + return false; + } + for (const key of Object.keys(obj)) { + if (FORBIDDEN_KEYS.has(key) || hasForbiddenKey(obj[key])) { + return true; + } + } + return false; +} + /** * Sanitize string values to remove dangerous patterns */ @@ -108,6 +126,9 @@ function sanitizeObject(obj) { const sanitized = {}; for (const [key, value] of Object.entries(obj)) { + // Never assign prototype-pollution keys: `sanitized.__proto__ = x` + // would replace the object's prototype instead of adding a key. + if (FORBIDDEN_KEYS.has(key)) continue; // Sanitize key names too const cleanKey = sanitizeString(key); sanitized[cleanKey] = sanitizeObject(value); @@ -151,6 +172,13 @@ export function validateMetadata(metadata) { }; } + if (hasForbiddenKey(metadata)) { + return { + valid: false, + error: 'Metadata contains a forbidden key (__proto__, constructor or prototype)' + }; + } + // Sanitize the metadata const sanitized = sanitizeObject(metadata); diff --git a/backend/src/lib/sanitize-metadata.test.js b/backend/src/lib/sanitize-metadata.test.js index b61c0160..5d197ec0 100644 --- a/backend/src/lib/sanitize-metadata.test.js +++ b/backend/src/lib/sanitize-metadata.test.js @@ -177,6 +177,27 @@ describe('validateMetadata', () => { }) }) +describe('validateMetadata — prototype pollution (issue #1447)', () => { + it('rejects a top-level __proto__ key parsed from JSON', () => { + const metadata = JSON.parse('{"__proto__": {"isAdmin": true}, "ok": 1}') + const result = validateMetadata(metadata) + expect(result.valid).toBe(false) + expect(result.error).toContain('forbidden key') + expect({}.isAdmin).toBeUndefined() + }) + + it('rejects nested constructor / prototype keys', () => { + expect(validateMetadata({ a: { constructor: { prototype: {} } } }).valid).toBe(false) + expect(validateMetadata({ list: [{ prototype: 1 }] }).valid).toBe(false) + }) + + it('still accepts keys that merely contain the forbidden words', () => { + const result = validateMetadata({ constructor_name: 'x', proto: 'y' }) + expect(result.valid).toBe(true) + expect(result.sanitized.constructor_name).toBe('x') + }) +}) + describe('sanitizeMetadataMiddleware', () => { it('passes through when no metadata in request', () => { const req = { body: {} } diff --git a/backend/src/routes/payments.js b/backend/src/routes/payments.js index dd9c5294..fe2c5744 100644 --- a/backend/src/routes/payments.js +++ b/backend/src/routes/payments.js @@ -40,11 +40,7 @@ import { paymentFailedCounter, } from "../lib/metrics.js"; import { sanitizeMetadataMiddleware } from "../lib/sanitize-metadata.js"; -import { - resolveAndValidateIssuer, - validatePerAssetLimits, - validateAllowedIssuers, -} from "../lib/payment-session-rules.js"; +import { validatePaymentSession } from "../lib/payment-session-validator.js"; import { getSupabaseClient } from "../lib/supabase-client.js"; import { paymentProcessorSessionsTotal, @@ -256,61 +252,35 @@ function createPaymentsRouter({ const sessionStart = Date.now(); try { const supabase = await getSupabaseClient(); - const body = req.body; - const asset = body.asset?.toUpperCase(); - logger.info({ merchantId: req.merchant?.id, amount: body.amount, asset: body.asset }, "DEBUG: createSession started"); - - // Shared business-rule validation (issue #1087) — issuer presence/format. - const { assetIssuer, rejection: issuerRejection } = resolveAndValidateIssuer( - asset, - body.asset_issuer, - ); - if (issuerRejection) { - paymentFailedCounter.inc({ asset: body.asset, reason: issuerRejection.reason }); - paymentProcessorSessionsTotal.inc({ asset: body.asset, outcome: "validation_failed" }); - paymentProcessorSessionDuration.observe( - { asset: body.asset, outcome: "validation_failed" }, - (Date.now() - sessionStart) / 1000, - ); - return res.status(400).json({ error: issuerRejection.message }); - } - - // Shared business-rule validation (issue #1087) — per-asset limits (#153). - const limitRejection = validatePerAssetLimits({ - rawAsset: body.asset, - amount: body.amount, - paymentLimits: req.merchant.payment_limits, + logger.info({ merchantId: req.merchant?.id, amount: req.body?.amount, asset: req.body?.asset }, "DEBUG: createSession started"); + + // Sanitization, strict payload checks and shared business rules + // (issues #1087, #1447) with validator metrics/health (#1448). + const validation = validatePaymentSession({ + body: req.body, + merchant: req.merchant, + source: "http", }); - if (limitRejection) { - paymentFailedCounter.inc({ asset: body.asset, reason: limitRejection.reason }); - paymentProcessorSessionsTotal.inc({ asset: body.asset, outcome: "validation_failed" }); + if (!validation.ok) { + const { rejection } = validation; + const assetLabel = req.body?.asset; + paymentFailedCounter.inc({ + asset: assetLabel, + reason: rejection.reason === "issuer_not_allowed" ? "invalid_issuer" : rejection.reason, + }); + paymentProcessorSessionsTotal.inc({ asset: assetLabel, outcome: "validation_failed" }); paymentProcessorSessionDuration.observe( - { asset: body.asset, outcome: "validation_failed" }, + { asset: assetLabel, outcome: "validation_failed" }, (Date.now() - sessionStart) / 1000, ); return res.status(400).json({ - error: limitRejection.message, - ...limitRejection.details, + error: rejection.message, + ...(rejection.rule === "limits" ? rejection.details : {}), }); } - // Shared business-rule validation (issue #1087) — allowed-issuers check: - // if the merchant has configured a non-empty allowlist, only those - // issuer addresses may be used. - const allowedIssuerRejection = validateAllowedIssuers({ - asset, - assetIssuer, - allowedIssuers: req.merchant.allowed_issuers, - }); - if (allowedIssuerRejection) { - paymentFailedCounter.inc({ asset: body.asset, reason: "invalid_issuer" }); - paymentProcessorSessionsTotal.inc({ asset: body.asset, outcome: "validation_failed" }); - paymentProcessorSessionDuration.observe( - { asset: body.asset, outcome: "validation_failed" }, - (Date.now() - sessionStart) / 1000, - ); - return res.status(400).json({ error: allowedIssuerRejection.message }); - } + const body = validation.payload; + const { asset, assetIssuer } = validation; const isSandbox = body.sandbox === true; const baseId = randomUUID(); diff --git a/backend/src/routes/prometheus.js b/backend/src/routes/prometheus.js index 7347ada1..bd812212 100644 --- a/backend/src/routes/prometheus.js +++ b/backend/src/routes/prometheus.js @@ -9,6 +9,9 @@ import { trustlineManagerRegister } from "../lib/trustline-manager-metrics.js"; // Granular Path Payment Service metrics live in their own registry (issue #1048) // and are merged into the scrape output below. import { pathPaymentRegister } from "../lib/path-payment-metrics.js"; +// Payment Session Validator metrics live in their own registry (issue #1448) +// and are merged into the scrape output below. +import { paymentSessionValidatorRegister } from "../lib/payment-session-validator-metrics.js"; const router = express.Router(); @@ -17,7 +20,7 @@ const router = express.Router(); * /metrics: * get: * summary: Expose Prometheus metrics - * description: Returns the current state of Prometheus metrics for the application, including granular payment processor, trustline manager and path payment metrics. + * description: Returns the current state of Prometheus metrics for the application, including granular payment processor, trustline manager, path payment and payment session validator metrics. * tags: [Monitoring] * responses: * 200: @@ -29,17 +32,15 @@ const router = express.Router(); */ router.get("/metrics", async (req, res) => { try { - const [coreMetrics, processorMetrics, trustlineMetrics, pathPaymentMetrics] = - await Promise.all([ - register.metrics(), - paymentProcessorRegister.metrics(), - trustlineManagerRegister.metrics(), - pathPaymentRegister.metrics(), - ]); + const scrapes = await Promise.all([ + register.metrics(), + paymentProcessorRegister.metrics(), + trustlineManagerRegister.metrics(), + pathPaymentRegister.metrics(), + paymentSessionValidatorRegister.metrics(), + ]); res.set("Content-Type", register.contentType); - res.end( - `${coreMetrics}\n${processorMetrics}\n${trustlineMetrics}\n${pathPaymentMetrics}`, - ); + res.end(scrapes.join("\n")); } catch (err) { res.status(500).end(err); } diff --git a/backend/src/services/exchangeRateService.js b/backend/src/services/exchangeRateService.js index 88d5b0e6..571cddb5 100644 --- a/backend/src/services/exchangeRateService.js +++ b/backend/src/services/exchangeRateService.js @@ -8,6 +8,14 @@ * 3. Slippage application + response shaping * 4. Prometheus metrics * + * Concurrency control (issue #1445): + * - Concurrent misses for the same quote share ONE load per process + * (ExchangeRateCache.getOrLoad single-flight). + * - When configureExchangeRateCoordination() has been given a live Redis + * client, that load is further coordinated across instances by a + * distributed lock + shared quote store (exchange-rate-coordinator.js). + * Coordination fails open to a direct Horizon query. + * * The route handler calls getExchangeRateQuote() and only handles HTTP concerns; * all exchange-rate logic lives here. */ @@ -17,10 +25,43 @@ import { getExchangeRateCache, generateRateCacheKey, } from '../lib/exchange-rate-cache.js'; +import { ExchangeRateCoordinator } from '../lib/exchange-rate-coordinator.js'; import { logger } from '../lib/logger.js'; const DEFAULT_SLIPPAGE = parseFloat(process.env.PATH_PAYMENT_SLIPPAGE ?? '0.01'); +/** Upper bound on a single quote load, so waiters are never pinned forever. */ +const LOAD_TIMEOUT_MS = (() => { + const raw = Number.parseInt(process.env.EXCHANGE_RATE_LOAD_TIMEOUT_MS ?? '', 10); + return Number.isFinite(raw) && raw > 0 ? raw : 15_000; +})(); + +/** @type {ExchangeRateCoordinator|null} */ +let coordinator = null; + +/** + * Enable cross-instance coordination. Call once at startup with a connected + * Redis client; passing a missing/closed client leaves coordination disabled. + * + * @param {object} opts + * @param {object|null} opts.redisClient + * @returns {boolean} whether coordination is enabled + */ +export function configureExchangeRateCoordination({ redisClient, ...options } = {}) { + if (!redisClient?.isOpen || typeof redisClient.sendCommand !== 'function') { + coordinator = null; + return false; + } + coordinator = new ExchangeRateCoordinator({ redisClient, ...options }); + logger.info('Exchange-rate cache distributed coordination enabled'); + return true; +} + +/** Disable cross-instance coordination (test isolation / shutdown). */ +export function resetExchangeRateCoordination() { + coordinator = null; +} + export class ExchangeRateError extends Error { constructor(message, statusCode = 502) { super(message); @@ -67,18 +108,50 @@ export async function getExchangeRateQuote({ destAssetIssuer, ); - const cached = cache.get(cacheKey); - if (cached.hit && !cached.stale) { - logger.debug('exchange_rate_cache: HIT'); - return { ...cached.data, cached: true }; - } + const fetchFromHorizon = () => fetchQuoteFromHorizon({ + sourceAssetCode, + sourceAssetIssuer, + destAssetCode, + destAssetIssuer, + destAmount, + sourceAccount, + slippage, + }); - if (cached.hit && cached.stale) { - logger.debug('exchange_rate_cache: STALE — revalidating'); + // Captured per call so a later reconfiguration cannot change an in-flight load. + const activeCoordinator = coordinator; + const loader = activeCoordinator + ? async () => { + const { data, source } = await activeCoordinator.load(cacheKey, fetchFromHorizon); + // A quote reused from a peer instance never reached Horizon here. + return { ...data, cached: source === 'shared' }; + } + : fetchFromHorizon; + + const { data, source } = await cache.getOrLoad(cacheKey, loader, { + timeoutMs: LOAD_TIMEOUT_MS, + }); + + if (source === 'loader') { + logger.debug('exchange_rate_cache: MISS — loaded'); + return data; } - logger.debug('exchange_rate_cache: MISS — querying Horizon'); + // 'cache' (fresh hit) or 'coalesced' (joined another caller's load): + // either way this request did not query Horizon itself. + logger.debug(`exchange_rate_cache: ${source === 'cache' ? 'HIT' : 'COALESCED'}`); + return { ...data, cached: true }; +} +async function fetchQuoteFromHorizon({ + sourceAssetCode, + sourceAssetIssuer, + destAssetCode, + destAssetIssuer, + destAmount, + sourceAccount, + slippage, +}) { const path = await findStrictReceivePaths({ sourceAccount, destAssetCode, @@ -107,7 +180,6 @@ export async function getExchangeRateQuote({ cached: false, }; - cache.set(cacheKey, quote); return quote; } @@ -124,5 +196,14 @@ export function invalidateExchangeRateQuote( ) { const cache = getExchangeRateCache(); const key = generateRateCacheKey(sourceAsset, destAsset, destAmount, sourceAssetIssuer, destAssetIssuer); - return cache.delete(key); + const removed = cache.delete(key); + + // Propagate to peers via the shared store. Fire-and-forget keeps this + // function synchronous for existing callers; failures only mean peers + // keep the quote until its short TTL expires. + coordinator?.invalidate(key).catch((err) => { + logger.warn({ err: err?.message }, 'Failed to invalidate shared exchange-rate quote'); + }); + + return removed; } diff --git a/backend/src/services/exchangeRateService.test.js b/backend/src/services/exchangeRateService.test.js index 9c940fd1..a2b0b56f 100644 --- a/backend/src/services/exchangeRateService.test.js +++ b/backend/src/services/exchangeRateService.test.js @@ -2,17 +2,20 @@ import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest'; import { getExchangeRateQuote, invalidateExchangeRateQuote, + configureExchangeRateCoordination, + resetExchangeRateCoordination, NoPathFoundError, ExchangeRateError, } from './exchangeRateService.js'; -import { resetExchangeRateCache } from '../lib/exchange-rate-cache.js'; +import { resetExchangeRateCache, generateRateCacheKey } from '../lib/exchange-rate-cache.js'; +import { createFakeRedis } from '../../tests/helpers/fake-redis.js'; vi.mock('../lib/stellar.js', () => ({ findStrictReceivePaths: vi.fn(), })); vi.mock('../lib/logger.js', () => ({ - logger: { debug: vi.fn(), warn: vi.fn(), error: vi.fn() }, + logger: { debug: vi.fn(), info: vi.fn(), warn: vi.fn(), error: vi.fn() }, })); import { findStrictReceivePaths } from '../lib/stellar.js'; @@ -136,7 +139,90 @@ describe('invalidateExchangeRateQuote', () => { await getExchangeRateQuote({ sourceAssetCode: 'XLM', destAssetCode: 'USDC', destAmount: '1.0' }); await getExchangeRateQuote({ sourceAssetCode: 'XLM', destAssetCode: 'USDC', destAmount: '2.0' }); invalidateExchangeRateQuote('XLM', 'USDC', '1.0'); - await getExchangeRateQuote({ sourceAssetCode: 'XLM', destAssetCode: 'USDC', destAmount: '2.0' }); - expect(findStrictReceivePaths).toHaveBeenCalledTimes(3); + const quote = await getExchangeRateQuote({ sourceAssetCode: 'XLM', destAssetCode: 'USDC', destAmount: '2.0' }); + // The 2.0 quote is still cached, so no third Horizon call is made. + expect(findStrictReceivePaths).toHaveBeenCalledTimes(2); + expect(quote.cached).toBe(true); + }); +}); + +describe('concurrency control (issue #1445)', () => { + const params = { sourceAssetCode: 'XLM', destAssetCode: 'USDC', destAmount: '1.0' }; + + beforeEach(() => { + resetExchangeRateCache(); + resetExchangeRateCoordination(); + vi.clearAllMocks(); + findStrictReceivePaths.mockResolvedValue(MOCK_PATH); + }); + + afterEach(() => { + resetExchangeRateCache(); + resetExchangeRateCoordination(); + }); + + it('coalesces a burst of identical requests into one Horizon call', async () => { + let release; + findStrictReceivePaths.mockImplementation( + () => new Promise((resolve) => { release = () => resolve(MOCK_PATH); }), + ); + const pending = Array.from({ length: 25 }, () => getExchangeRateQuote(params)); + await vi.waitFor(() => expect(findStrictReceivePaths).toHaveBeenCalled()); + release(); + const quotes = await Promise.all(pending); + expect(findStrictReceivePaths).toHaveBeenCalledTimes(1); + expect(quotes.filter((q) => q.cached === false)).toHaveLength(1); + expect(quotes.filter((q) => q.cached === true)).toHaveLength(24); + expect(new Set(quotes.map((q) => q.sendMax))).toEqual(new Set(['0.5050000'])); + }); + + it('propagates NoPathFoundError to every coalesced caller', async () => { + findStrictReceivePaths.mockResolvedValue(null); + const settled = await Promise.allSettled([getExchangeRateQuote(params), getExchangeRateQuote(params)]); + expect(settled.every((s) => s.status === 'rejected' && s.reason instanceof NoPathFoundError)).toBe(true); + expect(findStrictReceivePaths).toHaveBeenCalledTimes(1); + }); + + it('configureExchangeRateCoordination ignores missing or closed clients', () => { + expect(configureExchangeRateCoordination({ redisClient: null })).toBe(false); + expect(configureExchangeRateCoordination({ redisClient: { isOpen: false, sendCommand: vi.fn() } })).toBe(false); + expect(configureExchangeRateCoordination({ redisClient: { isOpen: true } })).toBe(false); + expect(configureExchangeRateCoordination()).toBe(false); + expect(configureExchangeRateCoordination({ redisClient: createFakeRedis() })).toBe(true); + }); + + it('publishes quotes to the shared store and reuses a peer’s quote', async () => { + const redis = createFakeRedis(); + configureExchangeRateCoordination({ redisClient: redis, pollIntervalMs: 5 }); + + const first = await getExchangeRateQuote(params); + expect(first.cached).toBe(false); + const key = generateRateCacheKey('XLM', 'USDC', '1.0'); + expect(redis.store.has(`exrate:quote:${key}`)).toBe(true); + + // A second instance with an empty in-memory cache. + resetExchangeRateCache(); + const second = await getExchangeRateQuote(params); + expect(second.cached).toBe(true); + expect(second.sendMax).toBe(first.sendMax); + expect(findStrictReceivePaths).toHaveBeenCalledTimes(1); + }); + + it('invalidation also clears the shared quote', async () => { + const redis = createFakeRedis(); + configureExchangeRateCoordination({ redisClient: redis }); + await getExchangeRateQuote(params); + invalidateExchangeRateQuote('XLM', 'USDC', '1.0'); + await vi.waitFor(() => expect(redis.count('DEL')).toBe(1)); + expect(redis.store.size).toBe(0); + }); + + it('keeps serving quotes when Redis fails', async () => { + const redis = createFakeRedis(); + configureExchangeRateCoordination({ redisClient: redis }); + redis.setFailure(new Error('ECONNREFUSED')); + const quote = await getExchangeRateQuote(params); + expect(quote.sourceAmount).toBe('0.5000000'); + expect(() => invalidateExchangeRateQuote('XLM', 'USDC', '1.0')).not.toThrow(); }); }); diff --git a/backend/src/services/paymentService.js b/backend/src/services/paymentService.js index 70dbfb71..3d99b51a 100644 --- a/backend/src/services/paymentService.js +++ b/backend/src/services/paymentService.js @@ -27,11 +27,7 @@ import { } from "../lib/metrics.js"; import { paymentSignatureVerifier } from "../lib/payment-signature-verification.js"; import { logger } from "../lib/logger.js"; -import { - resolveAndValidateIssuer, - validatePerAssetLimits, - validateAllowedIssuers, -} from "../lib/payment-session-rules.js"; +import { validatePaymentSession } from "../lib/payment-session-validator.js"; import { getSupabaseClient } from "../lib/supabase-client.js"; import { paymentProcessorSessionsTotal, @@ -460,26 +456,35 @@ export const paymentService = { async createPaymentSession(merchant, body) { const sessionStart = Date.now(); const recordSessionOutcome = (outcome) => { - paymentProcessorSessionsTotal.inc({ asset: body.asset, outcome }); + paymentProcessorSessionsTotal.inc({ asset: body?.asset, outcome }); paymentProcessorSessionDuration.observe( - { asset: body.asset, outcome }, + { asset: body?.asset, outcome }, (Date.now() - sessionStart) / 1000, ); }; const supabase = await getSupabaseClient(); - const asset = body.asset?.toUpperCase(); - // Shared business-rule validation (issue #1087) — issuer presence/format. - const { assetIssuer, rejection: issuerRejection } = resolveAndValidateIssuer( - asset, - body.asset_issuer, - ); - if (issuerRejection) { + // Sanitization, strict payload checks and shared business rules + // (issues #1087, #1447) with validator metrics/health (#1448). These are + // pure and cheap, so they run before the on-chain issuer lookup below — + // an invalid request never costs a Horizon round-trip. + const validation = validatePaymentSession({ body, merchant, source: "service" }); + if (!validation.ok) { + const { rejection } = validation; + paymentFailedCounter.inc({ + asset: body?.asset, + reason: rejection.reason === "issuer_not_allowed" ? "invalid_issuer" : rejection.reason, + }); recordSessionOutcome("validation_failed"); - const error = new Error(issuerRejection.message); + const error = new Error(rejection.message); error.status = 400; + if (rejection.rule === "limits") { + error.details = rejection.details; + } throw error; } + body = validation.payload; + const { asset, assetIssuer } = validation; // Task #756: Dynamic Issuer Verification with Error Recovery if (asset !== "XLM" && assetIssuer) { @@ -501,36 +506,6 @@ export const paymentService = { } } - // Shared business-rule validation (issue #1087) — per-asset limits. - const limitRejection = validatePerAssetLimits({ - rawAsset: body.asset, - amount: body.amount, - paymentLimits: merchant.payment_limits, - }); - if (limitRejection) { - paymentFailedCounter.inc({ asset: body.asset, reason: limitRejection.reason }); - recordSessionOutcome("validation_failed"); - const error = new Error(limitRejection.message); - error.status = 400; - error.details = limitRejection.details; - throw error; - } - - // Shared business-rule validation (issue #1087) — allowed issuers. - const allowedIssuers = merchant.allowed_issuers; - const allowlistRejection = validateAllowedIssuers({ - asset, - assetIssuer, - allowedIssuers, - }); - if (allowlistRejection) { - paymentFailedCounter.inc({ asset: body.asset, reason: "invalid_issuer" }); - recordSessionOutcome("validation_failed"); - const error = new Error(allowlistRejection.message); - error.status = 400; - throw error; - } - const paymentId = randomUUID(); const now = new Date().toISOString(); const paymentLinkBase = process.env.PAYMENT_LINK_BASE || "http://localhost:3000"; diff --git a/backend/tests/helpers/fake-redis.js b/backend/tests/helpers/fake-redis.js new file mode 100644 index 00000000..6de8dbc0 --- /dev/null +++ b/backend/tests/helpers/fake-redis.js @@ -0,0 +1,79 @@ +/** + * In-memory stand-in for the subset of Redis used by the exchange-rate + * coordinator (issues #1445, #1446): SET [PX ms] [NX], GET, DEL and the + * compare-and-delete EVAL script. Several coordinators can share one + * instance to simulate multiple API processes against the same Redis. + * + * Commands resolve asynchronously (optionally after `latencyMs`) so they + * interleave the way network round-trips do. Expiry follows Date.now(), so + * it works with vi.useFakeTimers(). + */ +export function createFakeRedis({ latencyMs = 0 } = {}) { + const store = new Map(); // key -> { value, expiresAt } + const calls = []; + let failWith = null; + + const live = (key) => { + const entry = store.get(key); + if (!entry) return null; + if (entry.expiresAt !== null && Date.now() >= entry.expiresAt) { + store.delete(key); + return null; + } + return entry; + }; + + const execute = (args) => { + const [cmd, ...rest] = args; + switch (String(cmd).toUpperCase()) { + case 'GET': + return live(rest[0])?.value ?? null; + case 'SET': { + const [key, value, ...opts] = rest; + const upper = opts.map((o) => String(o).toUpperCase()); + const pxIndex = upper.indexOf('PX'); + const ttl = pxIndex >= 0 ? Number(opts[pxIndex + 1]) : null; + if (upper.includes('NX') && live(key)) return null; + store.set(key, { value: String(value), expiresAt: ttl ? Date.now() + ttl : null }); + return 'OK'; + } + case 'DEL': + return rest.reduce((n, key) => n + (live(key) && store.delete(key) ? 1 : 0), 0); + case 'EVAL': { + // Only the compare-and-delete release script is supported. + const [, , key, token] = rest; + if (live(key)?.value === token) { + store.delete(key); + return 1; + } + return 0; + } + default: + throw new Error(`fake-redis: unsupported command ${cmd}`); + } + }; + + return { + isOpen: true, + calls, + store, + /** Make every subsequent command reject with `error` (null to heal). */ + setFailure(error) { + failWith = error; + }, + async sendCommand(args) { + calls.push(args); + if (latencyMs > 0) { + await new Promise((resolve) => setTimeout(resolve, latencyMs)); + } else { + await Promise.resolve(); + } + if (failWith) throw failWith; + return execute(args); + }, + /** Test helper: count calls by command name. */ + count(cmd) { + return calls.filter(([c]) => String(c).toUpperCase() === cmd).length; + }, + }; +} diff --git a/backend/tests/integration/exchange-rate-cache.test.js b/backend/tests/integration/exchange-rate-cache.test.js new file mode 100644 index 00000000..f8ded940 --- /dev/null +++ b/backend/tests/integration/exchange-rate-cache.test.js @@ -0,0 +1,302 @@ +/** + * Exchange Rate Oracle Cache — HTTP integration (issue #1446). + * + * Drives GET /api/path-payment-quote/:id through the real Express app, the + * real ExchangeRateCache / ExchangeRateCoordinator and the real service. + * Only the external edges are mocked: Horizon (findStrictReceivePaths), + * Supabase and Redis (in-memory fake). + */ +import request from 'supertest'; +import { describe, it, expect, beforeAll, beforeEach, afterAll, afterEach, vi } from 'vitest'; + +vi.hoisted(() => { + process.env.PATH_PAYMENT_QUOTE_RATE_LIMIT_MAX = '100000'; + // Wide enough that every request in a burst joins the load before it times out. + process.env.EXCHANGE_RATE_LOAD_TIMEOUT_MS = '1000'; +}); + +import { createApp } from '../../src/app.js'; +import { closePool } from '../../src/lib/db.js'; +import { findStrictReceivePaths } from '../../src/lib/stellar.js'; +import { resetExchangeRateCache, generateRateCacheKey } from '../../src/lib/exchange-rate-cache.js'; +import { + configureExchangeRateCoordination, + resetExchangeRateCoordination, + invalidateExchangeRateQuote, +} from '../../src/services/exchangeRateService.js'; +import { exchangeRateCacheCoalescedRequests } from '../../src/lib/path-payment-metrics.js'; +import { createFakeRedis } from '../helpers/fake-redis.js'; + +const USDC_ISSUER = 'GBBD47IF6LWK7P7MDEVSCWR7DPUWV3NY3DTQEVFL4NAT4AQH3ZLLFLA5'; +const SOURCE_ACCOUNT = 'GA5XIGA5C7FBPTVQ3CWHKNC7D2ZBHB24G3KUJG5WZ6S4EYWSSBFVL45T'; + +const payments = vi.hoisted(() => new Map()); + +vi.mock('../../src/lib/stellar.js', () => ({ + findMatchingPayment: vi.fn(), + findAnyRecentPayment: vi.fn(), + findStrictReceivePaths: vi.fn(), + getNetworkFeeStats: vi.fn(), + isHorizonReachable: vi.fn(async () => true), + isValidAssetCode: vi.fn((v) => typeof v === 'string' && /^[A-Z0-9]{1,12}$/.test(v.trim().toUpperCase())), + isValidStellarAccountId: vi.fn((v) => typeof v === 'string' && /^G[A-Z2-7]{55}$/.test(v)), + isValidStellarPublicKey: vi.fn((v) => typeof v === 'string' && /^G[A-Z2-7]{55}$/.test(v)), + validateMemo: vi.fn(() => ({ valid: true })), + verifyTransactionSignature: vi.fn(), + withHorizonRetry: vi.fn(), +})); + +vi.mock('../../src/lib/supabase.js', () => ({ + supabase: { + from: vi.fn(() => ({ + select: vi.fn(() => ({ limit: vi.fn(() => Promise.resolve({ data: [], error: null })) })), + })), + }, +})); + +vi.mock('../../src/lib/supabase-client.js', () => ({ + getSupabaseClient: vi.fn(async () => { + const filters = {}; + const query = { + select: () => query, + eq: (col, val) => { + filters[col] = val; + return query; + }, + is: () => query, + maybeSingle: async () => ({ data: payments.get(filters.id) ?? null, error: null }), + }; + return { from: () => query }; + }), +})); + +const paymentId = (n) => `00000000-0000-4000-8000-${String(n).padStart(12, '0')}`; + +function seedPayment(n, amount) { + payments.set(paymentId(n), { + id: paymentId(n), + amount, + asset: 'USDC', + asset_issuer: USDC_ISSUER, + recipient: SOURCE_ACCOUNT, + status: 'pending', + }); +} + +const horizonPath = (destAmount) => ({ + source_asset_code: 'XLM', + source_asset_issuer: null, + source_amount: (Number(destAmount) * 0.5).toFixed(7), + destination_amount: destAmount, + path: [], +}); + +const quote = (app, n = 1) => + request(app) + .get(`/api/path-payment-quote/${paymentId(n)}`) + .query({ source_asset: 'XLM', source_account: SOURCE_ACCOUNT }); + +async function coalescedCount() { + const { values } = await exchangeRateCacheCoalescedRequests.get(); + return values.reduce((sum, v) => sum + v.value, 0); +} + +/** + * Fire `n` identical requests and wait until all of them are attached to the + * single in-flight load (1 leader + n-1 coalesced). supertest opens a + * separate server per request, so arrival order is otherwise not guaranteed. + */ +async function burstJoined(app, horizon, n) { + const before = await coalescedCount(); + const burst = Array.from({ length: n }, () => quote(app).then((r) => r)); + await vi.waitFor(async () => { + expect(horizon.pending).toBe(1); + expect((await coalescedCount()) - before).toBe(n - 1); + }, { timeout: 5_000, interval: 10 }); + return burst; +} + +/** Horizon stub whose responses are released manually. */ +function gatedHorizon() { + const gates = []; + findStrictReceivePaths.mockImplementation( + ({ destAmount }) => + new Promise((resolve, reject) => + gates.push({ resolve: (value = horizonPath(destAmount)) => resolve(value), reject }), + ), + ); + return { + get pending() { + return gates.length; + }, + releaseAll(value) { + gates.splice(0).forEach((g) => g.resolve(value)); + }, + failAll(err) { + gates.splice(0).forEach((g) => g.reject(err)); + }, + }; +} + +describe('Exchange Rate Oracle Cache — HTTP integration', () => { + let app; + + beforeAll(async () => { + ({ app } = await createApp({ redisClient: null })); + }); + + beforeEach(() => { + payments.clear(); + for (let i = 1; i <= 5; i++) seedPayment(i, `${i}.0000000`); + resetExchangeRateCache(); + resetExchangeRateCoordination(); + findStrictReceivePaths.mockReset(); + findStrictReceivePaths.mockImplementation(async ({ destAmount }) => horizonPath(destAmount)); + }); + + afterEach(() => { + resetExchangeRateCoordination(); + }); + + afterAll(async () => { + await closePool().catch(() => {}); + }); + + it('serves a quote and then serves it from cache', async () => { + const first = await quote(app); + const second = await quote(app); + expect(first.status).toBe(200); + expect(first.body).toMatchObject({ source_asset: 'XLM', source_amount: '0.5000000', send_max: '0.5050000' }); + expect(second.body).toEqual(first.body); + expect(findStrictReceivePaths).toHaveBeenCalledTimes(1); + }); + + it('coalesces a burst of identical requests into one Horizon call', async () => { + const horizon = gatedHorizon(); + const burst = await burstJoined(app, horizon, 40); + horizon.releaseAll(); + const responses = await Promise.all(burst); + + expect(findStrictReceivePaths).toHaveBeenCalledTimes(1); + expect(responses.every((r) => r.status === 200)).toBe(true); + expect(new Set(responses.map((r) => r.body.send_max))).toEqual(new Set(['0.5050000'])); + }); + + it('keeps distinct asset/amount pairs independent under concurrency', async () => { + const responses = await Promise.all( + [1, 2, 3, 4, 5, 1, 2, 3, 4, 5].map((n) => quote(app, n)), + ); + expect(findStrictReceivePaths).toHaveBeenCalledTimes(5); + for (const [i, res] of responses.entries()) { + const n = (i % 5) + 1; + expect(res.body.destination_amount).toBe(`${n}.0000000`); + expect(res.body.source_amount).toBe((n * 0.5).toFixed(7)); + } + }); + + it('returns 404 to every coalesced caller when no path exists, then retries', async () => { + const horizon = gatedHorizon(); + const burst = await burstJoined(app, horizon, 10); + horizon.releaseAll(null); // Horizon found no path + const responses = await Promise.all(burst); + expect(responses.every((r) => r.status === 404)).toBe(true); + expect(findStrictReceivePaths).toHaveBeenCalledTimes(1); + + findStrictReceivePaths.mockImplementation(async ({ destAmount }) => horizonPath(destAmount)); + expect((await quote(app)).status).toBe(200); + expect(findStrictReceivePaths).toHaveBeenCalledTimes(2); + }); + + it('fails every waiter on a Horizon error without caching it', async () => { + const horizon = gatedHorizon(); + const burst = await burstJoined(app, horizon, 10); + horizon.failAll(Object.assign(new Error('Horizon 503'), { status: 502 })); + const responses = await Promise.all(burst); + expect(responses.every((r) => r.status === 502)).toBe(true); + + findStrictReceivePaths.mockImplementation(async ({ destAmount }) => horizonPath(destAmount)); + expect((await quote(app)).status).toBe(200); + }); + + it('returns 504 when Horizon hangs past the load timeout and recovers afterwards', async () => { + findStrictReceivePaths.mockImplementation(() => new Promise(() => {})); + const responses = await Promise.all(Array.from({ length: 5 }, () => quote(app))); + expect(responses.every((r) => r.status === 504)).toBe(true); + expect(findStrictReceivePaths).toHaveBeenCalledTimes(1); + + findStrictReceivePaths.mockImplementation(async ({ destAmount }) => horizonPath(destAmount)); + expect((await quote(app)).status).toBe(200); + }); + + it('does not let an invalidated in-flight quote repopulate the cache', async () => { + const horizon = gatedHorizon(); + const inflight = quote(app).then((r) => r); + await vi.waitFor(() => expect(horizon.pending).toBe(1)); + + invalidateExchangeRateQuote('XLM', 'USDC', '1.0000000', null, USDC_ISSUER); + horizon.releaseAll(); + expect((await inflight).status).toBe(200); + + findStrictReceivePaths.mockImplementation(async ({ destAmount }) => horizonPath(destAmount)); + await quote(app); + expect(findStrictReceivePaths).toHaveBeenCalledTimes(2); // re-queried, not served stale + }); + + it('exposes concurrency metrics on /metrics', async () => { + const horizon = gatedHorizon(); + const burst = await burstJoined(app, horizon, 5); + horizon.releaseAll(); + await Promise.all(burst); + + const res = await request(app).get('/metrics'); + expect(res.status).toBe(200); + expect(res.text).toMatch(/exchange_rate_cache_coalesced_requests_total\{[^}]*\} [1-9]/); + expect(res.text).toContain('exchange_rate_cache_inflight_loads'); + expect(res.text).toContain('exchange_rate_lock_acquisitions_total'); + expect(res.text).toContain('exchange_rate_coordination_fallbacks_total'); + }); + + describe('with Redis coordination', () => { + let redis; + + beforeEach(() => { + redis = createFakeRedis({ latencyMs: 1 }); + expect(configureExchangeRateCoordination({ redisClient: redis, pollIntervalMs: 5 })).toBe(true); + }); + + it('publishes the quote and lets a cold instance reuse it without Horizon', async () => { + const first = await quote(app); + expect(first.status).toBe(200); + const key = generateRateCacheKey('XLM', 'USDC', '1.0000000', null, USDC_ISSUER); + expect(redis.store.has(`exrate:quote:${key}`)).toBe(true); + expect(redis.store.has(`exrate:lock:${key}`)).toBe(false); + + resetExchangeRateCache(); // simulate a different instance's empty L1 + const second = await quote(app); + expect(second.body).toEqual(first.body); + expect(findStrictReceivePaths).toHaveBeenCalledTimes(1); + }); + + it('ignores a poisoned shared entry and serves a verified quote', async () => { + const key = generateRateCacheKey('XLM', 'USDC', '1.0000000', null, USDC_ISSUER); + redis.store.set(`exrate:quote:${key}`, { + value: JSON.stringify({ v: 1, data: { sourceAsset: 'XLM', sourceAmount: '0.0000001', sendMax: '-1', path: [] } }), + expiresAt: null, + }); + const res = await quote(app); + expect(res.status).toBe(200); + expect(res.body.send_max).toBe('0.5050000'); + expect(findStrictReceivePaths).toHaveBeenCalledTimes(1); + }); + + it('keeps serving quotes while Redis is down', async () => { + redis.setFailure(new Error('ECONNREFUSED')); + const horizon = gatedHorizon(); + const burst = await burstJoined(app, horizon, 10); + horizon.releaseAll(); + const responses = await Promise.all(burst); + expect(responses.every((r) => r.status === 200)).toBe(true); + expect(findStrictReceivePaths).toHaveBeenCalledTimes(1); // L1 single-flight still applies + }); + }); +}); diff --git a/backend/tests/integration/payment-session-validator.test.js b/backend/tests/integration/payment-session-validator.test.js new file mode 100644 index 00000000..1bf73ad8 --- /dev/null +++ b/backend/tests/integration/payment-session-validator.test.js @@ -0,0 +1,207 @@ +/** + * Payment Session Validator — HTTP integration (issues #1447, #1448). + * + * Drives POST /api/sessions through the real Express app (schema validation, + * metadata sanitizer, route handler, validator) with only the external edges + * mocked: Stellar/Horizon, Supabase, API-key auth and idempotency storage. + */ +import request from "supertest"; +import { describe, it, expect, beforeAll, beforeEach, afterAll, vi } from "vitest"; +import { createApp } from "../../src/app.js"; +import { closePool } from "../../src/lib/db.js"; +import { resetPaymentSessionValidatorHealth } from "../../src/lib/payment-session-validator.js"; + +const VALID_ISSUER = "GBBD47IF6LWK7P7MDEVSCWR7DPUWV3NY3DTQEVFL4NAT4AQH3ZLLFLA5"; +const OTHER_ISSUER = "GA5XIGA5C7FBPTVQ3CWHKNC7D2ZBHB24G3KUJG5WZ6S4EYWSSBFVL45T"; + +const mockRedisClient = { + ping: vi.fn().mockResolvedValue("PONG"), + on: vi.fn(), + sendCommand: vi.fn().mockResolvedValue("mocked_hash"), +}; + +const state = vi.hoisted(() => ({ + merchant: null, + inserted: [], +})); + +vi.mock("../../src/lib/stellar.js", () => ({ + findMatchingPayment: vi.fn(), + findAnyRecentPayment: vi.fn(), + findStrictReceivePaths: vi.fn(), + getNetworkFeeStats: vi.fn(), + isHorizonReachable: vi.fn(async () => true), + isValidAssetCode: vi.fn(() => true), + isValidStellarAccountId: vi.fn(() => true), + isValidStellarPublicKey: vi.fn((v) => typeof v === "string" && /^G[A-Z2-7]{55}$/.test(v)), + validateMemo: vi.fn(() => ({ valid: true })), + verifyTransactionSignature: vi.fn(), + withHorizonRetry: vi.fn(), +})); + +vi.mock("../../src/lib/supabase.js", () => ({ + supabase: { + from: vi.fn(() => ({ + select: vi.fn(() => ({ + limit: vi.fn(() => Promise.resolve({ data: [], error: null })), + })), + })), + }, +})); + +vi.mock("../../src/lib/supabase-client.js", () => ({ + getSupabaseClient: vi.fn(async () => ({ + from: vi.fn(() => ({ + insert: vi.fn(async (row) => { + state.inserted.push(row); + return { error: null }; + }), + })), + })), +})); + +vi.mock("../../src/lib/auth.js", async (importOriginal) => { + const actual = await importOriginal(); + return { + ...actual, + requireApiKeyAuth: () => (req, _res, next) => { + req.merchant = state.merchant; + next(); + }, + }; +}); + +vi.mock("../../src/lib/idempotency.js", () => ({ + idempotencyMiddleware: (_req, _res, next) => next(), +})); + +const baseMerchant = () => ({ + id: "merchant-1", + payment_limits: null, + allowed_issuers: [], + branding_config: null, +}); + +const validSession = () => ({ + amount: 25, + asset: "USDC", + asset_issuer: VALID_ISSUER, + recipient: VALID_ISSUER, + description: "Order 1001", +}); + +describe("Payment Session Validator — HTTP integration", () => { + let app; + + beforeAll(async () => { + ({ app } = await createApp({ redisClient: mockRedisClient })); + }); + + beforeEach(() => { + state.merchant = baseMerchant(); + state.inserted = []; + resetPaymentSessionValidatorHealth(); + }); + + afterAll(async () => { + await closePool().catch(() => {}); + }); + + it("creates a session and persists sanitized fields", async () => { + const res = await request(app) + .post("/api/sessions") + .send({ ...validSession(), description: "Order​ 1001‮", client_id: "cli\u0000ent" }); + + expect(res.status).toBe(201); + expect(state.inserted).toHaveLength(1); + expect(state.inserted[0]).toMatchObject({ + asset: "USDC", + asset_issuer: VALID_ISSUER, + description: "Order 1001", + client_id: "client", + }); + }); + + it.each([ + ["an amount with more than 7 decimals", { amount: 1.123456789 }, /7 decimal places/], + ["an amount above the Stellar maximum", { amount: 1e13 }, /must not exceed/], + ["an asset code with symbols", { asset: "US$C" }, /alphanumeric/], + ["an oversized description", { description: "x".repeat(1001) }, /description must be at most 1000/], + ])("rejects %s with 400 and persists nothing", async (_label, patch, message) => { + const res = await request(app).post("/api/sessions").send({ ...validSession(), ...patch }); + expect(res.status).toBe(400); + expect(res.body.error).toMatch(message); + expect(state.inserted).toHaveLength(0); + }); + + it("rejects prototype-pollution keys in metadata", async () => { + const res = await request(app) + .post("/api/sessions") + .set("Content-Type", "application/json") + .send(`{"amount":25,"asset":"USDC","asset_issuer":"${VALID_ISSUER}","recipient":"${VALID_ISSUER}","metadata":{"__proto__":{"isAdmin":true}}}`); + expect(res.status).toBe(400); + expect(res.body.error).toMatch(/forbidden key/); + expect({}.isAdmin).toBeUndefined(); + expect(state.inserted).toHaveLength(0); + }); + + it("enforces merchant limits and returns limit details", async () => { + state.merchant.payment_limits = { USDC: { max: 10 } }; + const res = await request(app).post("/api/sessions").send(validSession()); + expect(res.status).toBe(400); + expect(res.body).toMatchObject({ error: "Amount exceeds the maximum for USDC", max: 10, delta: 15 }); + }); + + it("does not resolve limits from Object.prototype for exotic asset codes", async () => { + state.merchant.payment_limits = {}; + const res = await request(app) + .post("/api/sessions") + .send({ ...validSession(), asset: "CONSTRUCTOR" }); + expect(res.status).toBe(201); + }); + + it("enforces the merchant issuer allowlist", async () => { + state.merchant.allowed_issuers = [OTHER_ISSUER]; + const res = await request(app).post("/api/sessions").send(validSession()); + expect(res.status).toBe(400); + expect(res.body.error).toMatch(/allowed issuers/); + }); + + it("exposes validator metrics on /metrics", async () => { + await request(app).post("/api/sessions").send(validSession()); + await request(app).post("/api/sessions").send({ ...validSession(), amount: 0.123456789 }); + + const res = await request(app).get("/metrics"); + expect(res.status).toBe(200); + expect(res.text).toMatch( + /payment_session_validator_evaluations_total\{[^}]*source="http"[^}]*outcome="accepted"[^}]*\} [1-9]/, + ); + expect(res.text).toMatch( + /payment_session_validator_rejections_total\{[^}]*rule="payload"[^}]*reason="invalid_amount"[^}]*\} [1-9]/, + ); + expect(res.text).toContain("payment_session_validator_duration_seconds_bucket"); + expect(res.text).toContain("payment_session_validator_health_state"); + }); + + it("reports validator health on /health/payment-session-validator and /health", async () => { + await request(app).post("/api/sessions").send(validSession()); + + const res = await request(app).get("/health/payment-session-validator"); + expect(res.status).toBe(200); + expect(res.body).toMatchObject({ status: "healthy", accepted: 1, total: 1 }); + expect(res.body.thresholds).toBeDefined(); + + const health = await request(app).get("/health"); + expect(health.body.services.payment_session_validator).toBe("healthy"); + }); + + it("reports degraded (still 200) when most sessions are rejected", async () => { + for (let i = 0; i < 20; i++) { + await request(app).post("/api/sessions").send({ ...validSession(), asset: "BAD-ASSET" }); + } + const res = await request(app).get("/health/payment-session-validator"); + expect(res.status).toBe(200); + expect(res.body.status).toBe("degraded"); + expect(res.body.reasons).toContain("rejection_ratio_exceeded"); + }); +});