[437] [High] Replace Polling Sweeper with a Deadline-Scheduled Job Queue

.env.example

@@ -452,6 +452,9 @@ HEALTH_READY_SUCCESS_THRESHOLD=2
 HEALTH_EVENT_LOOP_MAX_LAG_MS=1000
 # Soroban RPC endpoints for the quorum check (default: SOROBAN_RPC_URL).
 SOROBAN_RPC_HEALTH_URLS=
+# Low-frequency scan that catches deadline jobs the queue did not run (issue #437).
+SAFETY_SWEEP_INTERVAL_MS=300000
+
 # JSON map of EVM chain name to HTTPS JSON-RPC URL for admin token verification.
 # Example: {"ethereum":"https://ethereum.example/rpc","base":"https://base.example/rpc"}
 EVM_RPC_URLS={}
@@ -464,7 +467,6 @@ DATASETS_SALT_RETENTION_WINDOWS=2
 DATASETS_PUBLIC_BUCKET=vortex-public-datasets
 DATASETS_STORAGE_KIND=memory
 DATASETS_LOCAL_DIR=./data/datasets
-
 # ─── Intents store (issue #404) ──────────────────────────────────────────────
 # memory | dual | postgres — supersedes INTENTS_PERSISTENCE above.
 #   memory   — in-process only; everything is lost on restart (dev/test)
@@ -515,3 +517,13 @@ WS_MAX_CONNECTIONS=1000
 WS_BACKPLANE=memory
 REDIS_URL=redis://localhost:6379
 
+# Public anonymised datasets (RFC 0001). Disabled until an operator opts in.
+DATASETS_ENABLED=false
+DATASETS_ANONYMIZE=true
+DATASETS_SALT=
+DATASETS_SALT_ROTATION_HOURS=24
+DATASETS_SALT_RETENTION_WINDOWS=2
+DATASETS_PUBLIC_BUCKET=vortex-public-datasets
+DATASETS_STORAGE_KIND=memory
+DATASETS_LOCAL_DIR=./data/datasets
+

CHANGELOG.md

@@ -0,0 +1,152 @@
+# Changelog
+
+All notable changes to `vortex-backend` are documented here.
+
+This project follows [Conventional Commits](https://www.conventionalcommits.org/)
+and [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) conventions.
+Commit message format is enforced via [commitlint](https://commitlint.js.org/) starting with v0.2.0.
+
+> **How to update this file**
+>
+> Add entries to the `[Unreleased]` section as you work on features and fixes.
+> Organize by subsection (`Added`, `Fixed`, `Removed`, etc.) as defined below.
+>
+> **Mapping from Conventional Commits:**
+> - `feat(scope): ...` → `[Added]`
+> - `fix(scope): ...` → `[Fixed]`
+> - `perf(scope): ...` → `[Changed]` (with note about performance improvement)
+> - `refactor(scope): ...` → `[Changed]` (with scope of refactoring)
+> - `docs(scope): ...` → `[Documentation]` (if user-facing; skip if internal only)
+> - `chore(scope): ...` → Skip (internal tooling, no user-visible change)
+>
+> On version release, the `[Unreleased]` section is renamed to `[X.Y.Z]` with the
+> release date, and a new `[Unreleased]` section is created. Version must be bumped
+> in `package.json` to match.
+
+---
+
+## [Unreleased]
+
+### Added
+- Deadline jobs `expire-intent` and `fill-window-expired` replace the 30s sweeper poll. A safety sweep (`SAFETY_SWEEP_INTERVAL_MS`, default 5 min) catches lost jobs and increments `vortex_sweeper_safety_caught_total` (Closes #437).
+- `scripts/generate-client.ts` — generates a typed TypeScript API client from the live
+  OpenAPI spec using `openapi-typescript` v7; output committed to `src/generated/`
+  (Closes #134)
+- `CONTRIBUTING.md` — backend-specific onboarding guide covering prerequisites,
+  commands, DTO conventions, logger usage, module wiring, and PR checklist
+  (Closes #135)
+- `.env.testnet.example` — environment template for local testnet development with
+  safe defaults and blank contract IDs (Closes #136)
+- `.env.mainnet.example` — environment template for production mainnet with strict
+  CORS, required signing key, and all mandatory fields marked (Closes #136)
+- `commitlint.config.js` — enforces Conventional Commits via `@commitlint/config-conventional`
+  (Closes #137)
+- Husky `commit-msg` hook — runs commitlint on every local commit (Closes #137)
+- CI job `commitlint` — validates commit messages on every push/PR in GitHub Actions
+  (Closes #137)
+- `src/common/amount.ts` — shared, unit-tested base-units ↔ decimal conversion plus
+  the protocol fee (0.05 %) and quote-variance helpers; `IntentsController.quote()`
+  and `fill()` now use it instead of duplicated inline `BigInt`/decimal math
+  (Closes #272)
+- `PATCH /api/v1/solvers/:address` (`UpdateSolverDto`, `buildUpdateSolverMessage`) —
+  signature-verified partial update of a solver's mutable profile fields
+  (`name`, `supportedChains`, `supportedTokens`, `avgFillTime`); immutable fields
+  are stripped by the DTO whitelist (Closes #273)
+- Typed Swagger response documentation for every `SorobanController` and
+  `TokensController` route, including the account route's 400/429 responses
+  (Closes #271)
+- `src/disputes/` — structured slash-dispute (appeal) workflow: authenticated
+  `POST /api/v1/solvers/disputes`, evidence auto-verification (fill-verifier),
+  reviewer lifecycle (`open → under_review → upheld | overturned`) with SLA
+  deadlines and RBAC (`ReviewerGuard`), treasury refund requests on overturn,
+  and public anonymised statistics (`docs/governance/dispute-reviewers.md`)
+- `src/analytics/` — analytics layer over TimescaleDB continuous aggregates
+  (ADR-0001) with `GET /api/v1/analytics/{volume,fees,latency,solver-share}`
+  endpoints (`interval`, `from`, `to`, `chain`, `token` params), idempotent
+  event-driven ingestion, 1m/1h/1d rollups with retention, and historical backfill
+- CI job `migration-lint` — lints the `prisma/migrations/**/migration.sql` a
+  change adds or modifies (via the `scripts/check-migrations.ts` squawk-equivalent
+  checker) for unsafe DDL: non-concurrent index builds/drops, column type
+  rewrites, `NOT NULL` without a default, and `LOCK TABLE`; also requires a
+  `down.sql` in every changed migration. Overrides use `-- squawk-ignore <rule>`
+  with a mandatory `-- justification:`; fixture tests run via
+  `npm run test:scripts` (see `prisma/migrations/README.md`)
+
+### Fixed
+- `IntentsService.create()` idempotency-key handling is now race-safe — concurrent
+  requests carrying the same key synchronously claim an in-flight slot before any
+  `await`, so exactly one intent is created and the losers replay its result
+  (Closes #274)
+- `.github/workflows/ci.yml` failed to parse because the `backend` job's
+  `runs-on` was on the same line as its `name`, so no CI job could run
+- `TokensModule` was missing `exports: [TokensService]` — `IntentsController`
+  could not inject `TokensService` outside the Jest test environment
+- `IntentsModule` was missing `exports: [IntentsGateway]` — `StatsService`
+  could not inject `IntentsGateway` outside the Jest test environment
+
+---
+
+## [0.1.0] — 2026-07-01
+
+Initial versioned release of the NestJS rewrite.
+
+### Added (features)
+- Full NestJS rebuild replacing the original Express server
+- `ConfigModule` with Joi-based env validation and production signing-key enforcement
+- `GET/POST /api/v1/intents` — intent listing, creation, accept, fill, cancel, quote
+- `GET /api/v1/solvers` — solver leaderboard and stats
+- `GET /api/v1/tokens` — supported token registry with chain filtering
+- `GET /api/v1/stats` — protocol-level statistics
+- `GET /health` — service health with database and Soroban RPC checks
+- `WS /ws` — real-time intent feed with snapshot on connect and event replay buffer
+- `GET /docs` — Swagger / OpenAPI UI (spec available at `/docs-json`)
+- `GET /api/v1/chain/*` — Soroban RPC read endpoints (health, ledger, network, account)
+- Soroban transaction signer service with fee estimation and retry/backoff
+- Soroban contract event ingestion service
+- Settlement contract integration for on-chain intent registration
+- Solver registry contract integration (solver registry, deregistration, liveness)
+- Routing service with address validation and price oracle integration
+- Ed25519 signature verification on cancel/accept/fill/register endpoints
+- Per-user intent rate limiting (10 creates / 60 s) on top of global IP throttle
+- Idempotency key support on `POST /api/v1/intents`
+- Intent expiry sweeper (30 s interval, configurable)
+- WebSocket heartbeat and dead-client cleanup
+- Topic-based filtering for WS subscriptions
+- WS subscriber count cap (configurable via `WS_MAX_CONNECTIONS`)
+- Request ID correlation across HTTP logs and error responses
+- Structured Winston logging with configurable log level
+- Prometheus metrics endpoint (`/metrics`)
+- OpenTelemetry tracing instrumentation
+- Sentry error alerting integration
+- Helmet HTTP security headers with Swagger-compatible CSP
+- Global CORS enforcement (wildcard rejected in `NODE_ENV=production`)
+- 10 KB body size limit on all routes
+- Append-only intent audit log
+- Prisma ORM with PostgreSQL schema and migration tooling
+- Reference solver bot (`npm run solver:demo`) with exponential reconnect backoff
+- Seed script (`npm run seed`)
+- Database backup/restore runbook (`RUNBOOK_BACKUP_RESTORE.md`)
+- Load tests for concurrent intent accept race conditions and WS broadcast fanout
+- Full unit and e2e test suite (coverage threshold: 70 % on all axes)
+
+### Fixed
+- Precision loss in quote calculation for large bigint amounts
+- `fillAmount` format validation before `BigInt` parsing
+- `limit`/`offset` query param validation on intent listing
+- `minDstAmount` `BigInt` parsing guard in `fill()`
+- CORS wildcard default replaced with strict origin validation
+- Log injection vectors sanitised
+
+### Security
+- CORS wildcard (`*`) rejected at startup in `NODE_ENV=production`
+- Ed25519 Stellar signature authentication on all state-mutating endpoints
+- `SOROBAN_SIGNING_KEY` validated against Stellar secret seed format at startup
+  in production; process refuses to start without a well-formed key
+- Secrets scanning via Gitleaks in CI
+- `npm audit` at `--audit-level=high` in CI
+- WS connection limit to prevent resource exhaustion
+
+---
+
+[Unreleased]: https://github.com/vortex-protocol/vortex-backend/compare/v0.1.0...HEAD
+[0.1.0]: https://github.com/vortex-protocol/vortex-backend/releases/tag/v0.1.0

docs/adr/0005-deadline-jobs.md

@@ -0,0 +1,25 @@
+# ADR 0005: Deadline jobs for intent expiry
+
+- **Status**: Accepted
+- **Date**: 2026-09-29
+- **Technical Story**: #437 — replace the polling sweeper with deadline-scheduled jobs
+
+## Context
+
+`IntentsSweeperService` scanned every open and accepted intent every 30 seconds. Expiry latency was bounded by that interval, and the scan grew with intent volume.
+
+The job queue from ADR 0002 is already in the process. It has delayed enqueue and idempotency keys. It does not have a cancel or replace API.
+
+## Decision
+
+Arm `expire-intent` when an intent is created and `fill-window-expired` when it is accepted or its fill deadline is extended. The delay is the time remaining until the stored deadline. The idempotency key is `job:intentId:deadline`.
+
+A moved deadline enqueues a new job. The previous job still runs. The handler loads the intent and returns without writing when the state is terminal or the stored deadline is not the one in the payload.
+
+The leader-elected sweep stays, at `SAFETY_SWEEP_INTERVAL_MS` (default 5 minutes), for jobs lost to a crash or a missed timer. Each intent it expires or slashes increments `vortex_sweeper_safety_caught_total`. That counter should stay near zero. `triggerManualSweep` is unchanged.
+
+## Consequences
+
+- Expiry no longer waits for the scan interval. A job fires at the deadline.
+- Stale jobs are ignored rather than cancelled.
+- Operators tune the safety net with `SAFETY_SWEEP_INTERVAL_MS`. The queue driver remains `JOBS_DRIVER` (`memory` or `bullmq`); this change does not add a second queue.