AI_RESPONSE can double-post under timeout-retry

[NathanTarbert]

Pre-existing in main, not introduced by #172. Closed by #170's delta. Filed so it stays visible while #172 is in production.

runWithTimeout in packages/outpost/queue/src/worker.ts is a Promise.race between the handler and a timer. When the timer wins, the race resolves — but the handler keeps running. Nothing cancels it.

AI_RESPONSE has a 120s timeout (apps/worker/src/index.ts) over a deliberately slow pipeline (Pathfinder retrieval, generation, confidence scoring, groundedness gate), so hitting it is not exotic.

The sequence

1. Attempt 1 starts, runs past 120s. The race resolves as a timeout; the handler continues.
2. The worker marks the job for retry; attempt 2 starts roughly a second later.
3. Attempt 2 loads the ticket's messages before attempt 1 reaches its message.create, so it sees no prior AI response and passes the re-answer gate.
4. Both attempts generate, and both post to the platform.

Two AI answers in one thread — the exact behaviour #172 exists to prevent, reachable without any duplicate inbound event.

Why the gate alone cannot close it

The gate is a check-then-write against a snapshot. Two concurrent readers both see "no response yet." No amount of care inside the handler fixes that; it needs the database to arbitrate.

#170's delta does exactly that: a unique index on Message(ticketId, responseKey), claimed via message.create before any platform post-back, so the loser catches a narrowly-scoped P2002 and returns without posting. Plus Job.claimToken fencing so a timed-out attempt cannot write back over the attempt that replaced it.

Two things worth fixing regardless of #170

• runWithTimeout doesn't cancel. Even with the uniqueness constraint, a timed-out handler keeps burning a pipeline slot and model tokens to produce a response that will be discarded. An AbortSignal threaded into the handler would stop the work rather than just ignoring it.
• 120s may be too tight for this pipeline. If timeouts are routine rather than exceptional, the retry storm is the real problem and the double-post is a symptom.

Exposure while #172 is in production

Needs an AI_RESPONSE job to exceed 120s. Worth checking how often that actually happens before judging severity — [AI Response] Ticket … latency= log lines carry the number.

[jerelvelarde]

Confirmed by execution during a review round on #224, with a concrete timeline — recording it here since this issue has been open on reasoning rather than a repro.

runWithTimeout loses the Promise.race but nothing cancels the handler; there is no AbortSignal on JobHandlerContext. handleFailure then immediately writes status: 'PENDING', claimToken: null, lockUntil: null, so the row is claimable by any replica while the original handler is still executing.

With defaultTimeoutMs: 50 and a handler that posts at t=500ms:

• t=60ms — row is already PENDING with claimToken: null. Claimable.
• t=560ms — the orphaned handler performs its external side effect.

Three things worth pinning down in the fix:

1. lockUntil gives zero protection here. The deadline is cleared at timeout, not held. fix(queue): fence job claims and drain in-flight work on shutdown #224's fencing covers crash-abandonment at the row level; it does not touch this path, which is the common case rather than the exceptional one.
2. The fence cannot even report it. The claim token fences database writes, not Discord posts or GitHub comments. The orphan usually never reaches a fenced write, so nothing is logged. If it does reach one, the operator's first and only signal is a warnIfFenced line.
3. activeJobs.delete(job.id) runs in the finally, so concurrency is under-reported while the orphan is still working.

Minimum viable fix is an AbortSignal on JobHandlerContext, aborted in runWithTimeout's finally, with handlers doing network I/O passing it to fetch. Short of that, the timeout path should at least log distinctly — right now handleFailure records it as an ordinary retry, indistinguishable from a handler that returned a clean failure.

[NathanTarbert]

Closing — the double-post this describes is prevented on main.

The sequence here is that runWithTimeout does not cancel the handler, so attempt 1 keeps running while attempt 2 starts, and both reach the platform post-back.

#191 closed the second half of that. Message.responseKey carries a unique constraint on (ticketId, responseKey) (schema.prisma:136), and PRIMARY_AI_RESPONSE is claimed before any post-back, so the losing attempt catches a narrowly-identified P2002 and returns without posting. Two attempts can still race; only one can publish.

The other half of the original description — that the orphaned handler keeps running after the race resolves — is still true and is tracked separately as #266, including the fact that it is invisible to the worker's own job accounting.