feat(onchain): transactional outbox and durable slashing saga (#396 #397)

.env.example

@@ -435,6 +435,18 @@ HEALTH_READY_SUCCESS_THRESHOLD=2
 HEALTH_EVENT_LOOP_MAX_LAG_MS=1000
 # Soroban RPC endpoints for the quorum check (default: SOROBAN_RPC_URL).
 SOROBAN_RPC_HEALTH_URLS=
+# ── Transactional outbox relay (issue #396) ──────────────────────────────────
+OUTBOX_RELAY_ENABLED=true
+OUTBOX_RELAY_INTERVAL_MS=2000
+OUTBOX_RELAY_BATCH_SIZE=10
+OUTBOX_MAX_ATTEMPTS=8
+# Must exceed the signed transaction's 30 s time bound.
+OUTBOX_LEASE_SECONDS=120
+
+# ── Slashing saga (issue #397) ───────────────────────────────────────────────
+SLASH_CHALLENGE_WINDOW_SECONDS=600
+SLASH_CLOCK_SKEW_TOLERANCE_SECONDS=30
+SLASH_MAX_SUBMIT_ATTEMPTS=5
 
 # Maximum number of concurrent WebSocket connections accepted by the gateway.
 # Connections beyond this limit are rejected with close code 1013 (try again later).

.env.mainnet.example

@@ -310,6 +310,18 @@ HEALTH_READY_SUCCESS_THRESHOLD=2
 HEALTH_EVENT_LOOP_MAX_LAG_MS=1000
 # Soroban RPC endpoints for the quorum check (default: SOROBAN_RPC_URL).
 SOROBAN_RPC_HEALTH_URLS=
+# ── Transactional outbox relay (issue #396) ──────────────────────────────────
+OUTBOX_RELAY_ENABLED=true
+OUTBOX_RELAY_INTERVAL_MS=2000
+OUTBOX_RELAY_BATCH_SIZE=10
+OUTBOX_MAX_ATTEMPTS=8
+# Must exceed the signed transaction's 30 s time bound.
+OUTBOX_LEASE_SECONDS=120
+
+# ── Slashing saga (issue #397) ───────────────────────────────────────────────
+SLASH_CHALLENGE_WINDOW_SECONDS=600
+SLASH_CLOCK_SKEW_TOLERANCE_SECONDS=30
+SLASH_MAX_SUBMIT_ATTEMPTS=5
 
 # ─── Killswitch ──────────────────────────────────────────────────────────────
 # Keep this well under 5000 so worst-case propagation stays inside the 5 s requirement.

.env.staging.example

@@ -206,3 +206,15 @@ HEALTH_READY_SUCCESS_THRESHOLD=2
 HEALTH_EVENT_LOOP_MAX_LAG_MS=1000
 # Soroban RPC endpoints for the quorum check (default: SOROBAN_RPC_URL).
 SOROBAN_RPC_HEALTH_URLS=
+# ── Transactional outbox relay (issue #396) ──────────────────────────────────
+OUTBOX_RELAY_ENABLED=true
+OUTBOX_RELAY_INTERVAL_MS=2000
+OUTBOX_RELAY_BATCH_SIZE=10
+OUTBOX_MAX_ATTEMPTS=8
+# Must exceed the signed transaction's 30 s time bound.
+OUTBOX_LEASE_SECONDS=120
+
+# ── Slashing saga (issue #397) ───────────────────────────────────────────────
+SLASH_CHALLENGE_WINDOW_SECONDS=600
+SLASH_CLOCK_SKEW_TOLERANCE_SECONDS=30
+SLASH_MAX_SUBMIT_ATTEMPTS=5

.env.testnet.example

@@ -265,3 +265,15 @@ HEALTH_READY_SUCCESS_THRESHOLD=2
 HEALTH_EVENT_LOOP_MAX_LAG_MS=1000
 # Soroban RPC endpoints for the quorum check (default: SOROBAN_RPC_URL).
 SOROBAN_RPC_HEALTH_URLS=
+# ── Transactional outbox relay (issue #396) ──────────────────────────────────
+OUTBOX_RELAY_ENABLED=true
+OUTBOX_RELAY_INTERVAL_MS=2000
+OUTBOX_RELAY_BATCH_SIZE=10
+OUTBOX_MAX_ATTEMPTS=8
+# Must exceed the signed transaction's 30 s time bound.
+OUTBOX_LEASE_SECONDS=120
+
+# ── Slashing saga (issue #397) ───────────────────────────────────────────────
+SLASH_CHALLENGE_WINDOW_SECONDS=600
+SLASH_CLOCK_SKEW_TOLERANCE_SECONDS=30
+SLASH_MAX_SUBMIT_ATTEMPTS=5

CHANGELOG.md

@@ -28,6 +28,17 @@ Commit message format is enforced via [commitlint](https://commitlint.js.org/) s
 ## [Unreleased]
 
 ### Added
+- Transactional outbox for on-chain writes: `onchain_outbox` table, intent change + outbox
+  row committed in one Prisma transaction, `OutboxRelayService` (SKIP LOCKED claims, per-intent
+  ordering, envelope hash persisted before submit, dead-lettering with alert),
+  `TxConfirmationService`, live `StellarTxService.invokeContract` submit path, and
+  `POST /api/v1/admin/outbox/:id/requeue` (Closes #396)
+- Durable solver slashing saga: `pending_slashes` table (exactly-once per intent),
+  configurable challenge window, on-chain re-verification with clock-skew tolerance,
+  solver fill-proof and admin cancellation endpoints, compensation via `rollbackPenalty`,
+  metrics, alert rules and `docs/runbooks/slash-cancellation.md` (Closes #397)
+- Admin slash cancellation and outbox requeue use the shared `AdminGuard` RBAC (`x-admin-key`)
+  and are recorded in `admin_audit_log`
 - `scripts/generate-client.ts` — generates a typed TypeScript API client from the live
   OpenAPI spec using `openapi-typescript` v7; output committed to `src/generated/`
   (Closes #134)
@@ -72,6 +83,10 @@ Commit message format is enforced via [commitlint](https://commitlint.js.org/) s
   `npm run test:scripts` (see `prisma/migrations/README.md`)
 
 ### Fixed
+- `SorobanModule` referenced `forwardRef`/`IntentsModule` without importing them; it now
+  imports `SolversModule` (what `EventIngestionService` actually needs)
+- e2e `@stellar/stellar-sdk` mock now re-exports the real SDK and stubs only
+  `SorobanRpc.Server` (it previously lacked `Networks`, `Keypair`, … so no e2e suite could load)
 - `IntentsService.create()` idempotency-key handling is now race-safe — concurrent
   requests carrying the same key synchronously claim an in-flight slot before any
   `await`, so exactly one intent is created and the losers replay its result

docs/architecture/onchain-settlement.md

@@ -1,5 +1,10 @@
 # Architecture: On-Chain Settlement (Target Design)
 
+> **Update (issues #396 / #397):** the write-side plumbing below now exists —
+> see [Transactional outbox](#transactional-outbox) and
+> [Slashing saga](#slashing-saga). Contract method names remain provisional
+> until the ADR (issue #19) fixes the interface.
+>
 > **Status: target architecture, not yet implemented.** As of this writing,
 > `IntentsService` and `SolversService` are in-memory `Map`s
 > (`src/intents/intents.service.ts`, `src/solvers/solvers.service.ts`), and
@@ -166,6 +171,84 @@ transactions.
   above on a narrower surface (sweeper-triggered only, no user-facing HTTP
   write path).
 
+## Transactional outbox
+
+*Implemented — issue #396.* Stage 1 above ("HTTP request → Soroban tx") no
+longer submits inside the request. Submitting a transaction and writing
+Postgres as two separate steps is a dual write: a crash in between leaves the
+database saying "accepted" while the transaction never went out, or the
+reverse. Instead:
+
+```mermaid
+sequenceDiagram
+    participant API as IntentsService
+    participant DB as Postgres
+    participant Relay as OutboxRelayService
+    participant Chain as Soroban
+
+    API->>DB: BEGIN; UPDATE intents …; INSERT onchain_outbox (pending); COMMIT
+    loop every OUTBOX_RELAY_INTERVAL_MS
+        Relay->>DB: claim due head-of-intent rows (FOR UPDATE SKIP LOCKED) → processing
+        Relay->>Relay: build + simulate + sign
+        Relay->>DB: store envelope_hash (fenced on attempts)
+        Relay->>Chain: sendTransaction
+        Relay->>DB: status = submitted, tx_hash
+        Relay->>Chain: getTransaction(tx_hash) (TxConfirmationService)
+        Relay->>DB: status = confirmed
+    end
+```
+
+- **Atomicity.** `IntentsService` runs `create` / `acceptIfOpen` /
+  `fillIfAccepted` / `cancelIfOpen` through `IIntentsUnitOfWork`
+  (`src/intents/intents.unit-of-work.ts`). The Prisma adapter wraps the
+  intent write and the `onchain_outbox` insert in one `$transaction`. A
+  transition whose guard fails (lost race) enqueues nothing. With
+  `ONCHAIN_INTENTS_ENABLED=false` the outbox is bypassed entirely.
+- **Fail fast.** The payload is encoded to contract arguments at enqueue time,
+  so malformed input (bad address, non-integer amount) fails the HTTP request
+  rather than becoming a poison row.
+- **Ordering.** `onchain_outbox.id` is a sequence. A row is only claimable when
+  every earlier row for the same `intent_id` is `confirmed` or `simulated`,
+  so one intent's operations apply in order while different intents run in
+  parallel. `SKIP LOCKED` lets several relay instances share the work.
+- **Crash idempotency.** The signed envelope hash is persisted *before*
+  broadcast. A worker that dies mid-submit leaves the row `processing`; after
+  `OUTBOX_LEASE_SECONDS` it is reclaimed, and the relay first looks the stored
+  hash up: `SUCCESS` → confirm without resubmitting; `FAILED` → retry;
+  `NOT_FOUND` → rebuild. `NOT_FOUND` is conclusive only because the lease
+  (120 s) outlives the transaction's 30 s time bound
+  (`INVOKE_TX_TIMEOUT_SECONDS`). Every post-claim write is fenced on
+  `attempts`, so a worker whose lease expired cannot clobber a reclaimed row.
+- **Retries and poison rows.** Failures back off exponentially (1 s doubling,
+  capped at 5 min). After `OUTBOX_MAX_ATTEMPTS` claims a row becomes `dead`,
+  `vortex_outbox_dead_total` increments (alerted), and it blocks its intent
+  until requeued via `POST /api/v1/admin/outbox/:id/requeue`.
+- **Dry run.** Under `ONCHAIN_DRY_RUN=true` rows end as `simulated` (terminal).
+  They are not replayed when dry-run is later switched off.
+- **Out of scope:** cross-service delivery (Kafka etc.).
+
+Row lifecycle: `pending → processing → submitted → confirmed`, with
+`processing → simulated` (dry run), back to `pending` on retry, and `dead`
+past the attempt limit.
+
+## Slashing saga
+
+*Implemented — issue #397.* The `accepted → slashed` row of the mapping table
+runs as a durable saga (`src/intents/slashing-pipeline.service.ts`, table
+`pending_slashes`):
+
+`detected → challenge_window → submitted → confirmed | cancelled`
+
+The sweeper only *detects*. The slash is broadcast after a configurable
+challenge window, and only after the chain has been re-checked for a fill that
+landed by `fillDeadline + SLASH_CLOCK_SKEW_TOLERANCE_SECONDS` (by ledger close
+time, so server clock skew can't cause a wrong slash). A unique constraint on
+`intent_id` makes it exactly-once. Cancellation (solver fill-proof, admin, or
+giving up after `SLASH_MAX_SUBMIT_ATTEMPTS`) runs the compensation
+(`SolversService.rollbackPenalty`, intent leaves `slashed`) exactly once. The
+operator procedure is in
+[`docs/runbooks/slash-cancellation.md`](../runbooks/slash-cancellation.md).
+
 ## Persistence layer
 
 Both `IntentsService` and `SolversService` delegate all storage to an

docs/runbooks/alerts/onchain-writes.rules.yml

@@ -0,0 +1,33 @@
+# Prometheus alerting rules for the on-chain write path.
+# Metric names are defined in src/metrics/metrics.service.ts — keep in sync.
+groups:
+  - name: vortex-onchain-writes
+    rules:
+      # Issue #396 — a poison outbox row was dead-lettered. Later operations for
+      # the same intent are blocked until it is requeued.
+      - alert: VortexOutboxRowDead
+        expr: increase(vortex_outbox_dead_total[5m]) > 0
+        labels:
+          severity: page
+        annotations:
+          summary: "Outbox row moved to dead after exhausting OUTBOX_MAX_ATTEMPTS"
+          runbook: docs/runbooks/on-call.md#scenario-g--outbox-rows-dead-or-backlogged
+
+      # Issue #396 — the relay is not draining (RPC outage, relay disabled, signer broken).
+      - alert: VortexOutboxBacklog
+        expr: sum(vortex_outbox_rows{status=~"pending|processing|submitted"}) > 100
+        for: 15m
+        labels:
+          severity: warn
+        annotations:
+          summary: "More than 100 on-chain writes waiting in the outbox for 15m"
+          runbook: docs/runbooks/on-call.md#scenario-g--outbox-rows-dead-or-backlogged
+
+      # Issue #397 — the slashing saga gave up on a slash (submit kept failing).
+      - alert: VortexSlashSubmitFailed
+        expr: increase(vortex_slash_pipeline_transitions_total{to_state="cancelled",reason="submit_failed"}[15m]) > 0
+        labels:
+          severity: page
+        annotations:
+          summary: "A detected solver slash was cancelled after repeated submission failures"
+          runbook: docs/runbooks/slash-cancellation.md#submit-failed-slashes