fix(queue): fence job claims and drain in-flight work on shutdown

packages/outpost/db/prisma/migrations/20260820120000_add_job_claim_fencing/migration.sql

@@ -0,0 +1,27 @@
+-- Fences writes to the worker execution that owns a claim, and records the
+-- deadline that execution was granted so reclaim has a single authority.
+--
+-- Additive and nullable with no backfill, which is deliberate: a NULL claimToken
+-- on an in-flight row means "claimed before this deployed", and the reclaim's
+-- legacy branch handles those on an absolute ceiling instead of a granted
+-- deadline.
+--
+-- On ordering: `apps/worker/start.sh` runs `migrate deploy` at container start,
+-- so this ships WITH the new code rather than ahead of it. There is no two-phase
+-- deploy here. What makes that safe is the additive-nullable shape, not any
+-- sequencing guarantee -- during the rollout an old replica's unfenced
+-- `prisma.job.update` writes race the new replica's fenced ones, and the old
+-- replica simply does not see these columns.
+--
+-- Takes ACCESS EXCLUSIVE on "Job" for the length of this transaction. That is
+-- brief for a nullable ADD COLUMN with no default (a catalog-only change in
+-- PG11+), but it does queue behind in-flight claim UPDATEs and block everything
+-- behind it while it waits. Deliberately no `lock_timeout`: a statement that
+-- times out here aborts the migration, and `migrate deploy` records the failure
+-- in `_prisma_migrations` with `finished_at = NULL`, which is the P3009 state
+-- `start.sh` exists to diagnose and which needs a manual `migrate resolve`.
+-- Waiting is recoverable; a half-recorded migration is not. Same trade `start.sh`
+-- makes when it leaves `migrate deploy` unbounded.
+ALTER TABLE "Job"
+ADD COLUMN "claimToken" TEXT,
+ADD COLUMN "lockUntil" TIMESTAMP(3);

packages/outpost/db/prisma/migrations/20260820120100_index_job_status_lockuntil/migration.sql

@@ -0,0 +1,20 @@
+-- Index the columns the reclaim sweep scans.
+--
+-- The sweep runs once per poll interval on every replica:
+--   status = 'PROCESSING' AND "lockUntil" < NOW() - grace
+-- plus a NULL-"lockUntil" arm for rows claimed before the column existed. The
+-- first arm keeps "lockUntil" bare with the interval arithmetic on the other
+-- side, which is what lets the second column do any work; the legacy arms
+-- discriminate on "lockedAt"/"updatedAt" and ride the (status, "lockUntil" IS
+-- NULL) prefix, then filter. That is fine -- those arms empty out after one
+-- rollout. See reclaimStaleJobs() in packages/outpost/queue/src/worker.ts.
+--
+-- Its own migration on purpose. Prisma wraps each migration file in one
+-- transaction, so keeping this with the ADD COLUMN would hold that statement's
+-- ACCESS EXCLUSIVE lock across the index build and block reads for the duration.
+-- Split, the ALTER commits first and this takes only SHARE: writes wait, reads
+-- do not. Same reason CONCURRENTLY is not used -- it cannot run inside Prisma's
+-- transaction at all.
+--
+-- Purely additive: creates an index only, no column or data changes.
+CREATE INDEX "Job_status_lockUntil_idx" ON "Job"("status", "lockUntil");

packages/outpost/db/prisma/schema.prisma

@@ -241,13 +241,42 @@ model Job {
     progress    Int?      // Optional percentage 0-100
     runAt       DateTime  @default(now())
     lockedAt    DateTime?
+    // The deadline the CLAIMING worker was granted, written at claim time from
+    // that worker's own timeout config. Reclaim compares against this rather than
+    // re-deriving a window from whichever worker happens to notice, so a replica
+    // on an older config cannot decide a live claim has expired.
+    lockUntil   DateTime?
+    claimToken  String?   // Fences writes to the worker execution that owns this claim
     completedAt DateTime?
     error       String?
     createdAt   DateTime  @default(now())
     updatedAt   DateTime  @updatedAt
 
     @@index([status, runAt])
     @@index([type])
+    // Serves the reclaim sweep, which runs once per poll interval per replica.
+    // [status, runAt] has the wrong second column for it and [type] is far too
+    // low-selectivity to help.
+    //
+    // Only usable while the sweep's predicate keeps lockUntil bare on one side —
+    // see reclaimStaleJobs() in packages/outpost/queue/src/worker.ts. Wrapping
+    // the column in a CASE, or moving the interval arithmetic onto it, leaves the
+    // planner with the status prefix and a filter over every PROCESSING row.
+    //
+    // Worth stating what it does NOT buy, because the gain is smaller than it
+    // looks: PROCESSING is a tiny set (bounded by replicas × maxConcurrency, and
+    // JOB_CLEANUP keeps the table small besides), and [status, runAt]'s prefix
+    // already narrows to it. Meanwhile status and lockUntil both change on every
+    // claim and every release, so this index adds maintenance to the two hottest
+    // write paths on the table. It is here because the sweep's cost should not be
+    // a function of table size as the queue grows, not because it pays for itself
+    // at today's volumes.
+    //
+    // Ideally partial (WHERE status = 'PROCESSING'), which schema.prisma cannot
+    // express. Taking the wider index rather than hand-writing one, because a
+    // hand-written index puts the database permanently in drift against the CI
+    // gate that caught the missing claimToken field on this same branch.
+    @@index([status, lockUntil])
 }
 
 enum JobStatus {