fix(jobs): recover a delivery stranded in PENDING - #142
Merged
Merged
Conversation
A committed job whose delivery reached PENDING could be left with nothing able to move it. The worker no-ops a payload whose delivery_attempt no longer matches and then marks the outbox row DELIVERED, and resumeDelivery re-queued only NOT_REQUESTED or RETRIEVAL_FAILED, so the job kept a committed payment, an unretrieved result, and no path forward. PENDING alone is still not treated as resumable. The deciding evidence is whether a fulfill_supplier_order row is still queued for that job: a row a worker currently holds is status PENDING and so counts as queued, which means an in-flight retrieval is never duplicated. Only a job with no such row left is re-queued, under a fresh delivery_attempt that fences the old retrieval. The claim is a compare-and-set against the state read under the job's FOR UPDATE lock, so two concurrent resumes cannot both take one delivery. No payment work is created on this path; the committed settlement is untouched.
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
oneshot | 996e07b | Sep 13 2026, 02:19 PM |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to #141, which fixed the stale snapshot behind "Retrieving result". This fixes the state-machine hole behind the same symptom.
A committed job whose delivery reached
PENDINGcould be left with nothing able to move it. The worker no-ops a payload whosedelivery_attemptno longer matches and then marks the outbox rowDELIVERED, whileresumeDeliveryre-queued onlyNOT_REQUESTEDorRETRIEVAL_FAILED. Such a job kept a committed payment, an unretrieved result, and no path forward — and #141's bounded re-reads would simply give up on it.PENDINGalone is still not treated as resumable. The deciding evidence is whether afulfill_supplier_orderrow is still queued for that job: a row a worker currently holds isstatus = 'PENDING'and so counts as queued, which means an in-flight retrieval is never duplicated. Only a job with no such row left is re-queued, under a freshdelivery_attemptthat fences the old retrieval.Scope and acceptance criteria
PENDINGwith no queued fulfilment work can be recovered.Product and security invariants
Invariant notes:
This path creates supplier-retrieval work only: no
submit_settlementrow, no attempt row, no payment call, asserted by test.1 intent / at most 1 committed settlementis untouched and the committed payment is preserved, which is the point — the job is recovered without paying again.Claiming is a compare-and-set against the state read under the job's
FOR UPDATElock, so two concurrent resumes cannot both take one delivery, and the queued-row check runs under that same lock. The freshdelivery_attemptfences any older retrieval from completing, using the existingdelivery_attemptguard incompleteDeliveryandfailDelivery.Selected
.agent/TEST_MATRIX.mdcases: downstream failure after payment (payment durable, no replacement payment) and duplicate/parallel claim (one delivery claimed once).Validation
Commands and results:
The new coverage is fake-client unit tests at the SQL boundary. The Postgres integration suite (
pnpm test:integration) was not run here and no database was available; CI or a reviewer with a database should exercise it.Independent review evidence
Gate A — exact candidate tree before push
Reviewer tool:
free-pi-cliVerdict: skipped, carried over from the requester's instruction on fix(web,jobs): reach Result ready instead of sticking on Retrieving result #141
Findings or residual risks: Gate A was not run. Recorded deviation from
.agent/IMPLEMENTATION_LOOP.md§4-5, not a pass. This change touches durable job state, so it is a better Gate A candidate than fix(web,jobs): reach Result ready instead of sticking on Retrieving result #141 was.The reviewed tree equals the committed tree.
Gate B — exact remote PR head
Pull request URL/number: this PR
Reviewer tool:
free-pi-cliVerdict: skipped, carried over from the requester's instruction on fix(web,jobs): reach Result ready instead of sticking on Retrieving result #141
Findings or residual risks: Gate B was not run. Recorded deviation from §7, not a pass.
Gate B reviewed the current remote head and matches Gate A's approved tree, or a fresh Gate A was run for the changed tree.
Agent policy / repository-policyand all applicable CI checks pass.Risk and rollback
POST /v1/jobs/:jobId/resume, so recovery still takes an API call. The control that used to do it was removed deliberately ina2903a1; re-adding one is a product decision.PENDINGdelivery.Human merge