Skip to content

fix(jobs): recover a delivery stranded in PENDING - #142

Merged
selezenart merged 1 commit into
developfrom
fix/stranded-pending-delivery
Sep 13, 2026
Merged

selezenart merged 1 commit into
developfrom
fix/stranded-pending-delivery

Conversation

@selezenart

Copy link
Copy Markdown
Collaborator

Summary

Follow-up to #141, which fixed the stale snapshot behind "Retrieving result". This fixes the state-machine hole behind the same symptom.

A committed job whose delivery reached PENDING could be left with nothing able to move it. The worker no-ops a payload whose delivery_attempt no longer matches and then marks the outbox row DELIVERED, while resumeDelivery re-queued only NOT_REQUESTED or RETRIEVAL_FAILED. Such a job kept a committed payment, an unretrieved result, and no path forward — and #141's bounded re-reads would simply give up on it.

PENDING alone is still not treated as resumable. The deciding evidence is whether a fulfill_supplier_order row is still queued for that job: a row a worker currently holds is status = 'PENDING' and so counts as queued, which means an in-flight retrieval is never duplicated. Only a job with no such row left is re-queued, under a fresh delivery_attempt that fences the old retrieval.

Scope and acceptance criteria

  • The change is limited to the stated milestone or issue.
  • Acceptance criteria are listed and satisfied.
  • No unrelated cleanup is included.
  1. A delivery stranded in PENDING with no queued fulfilment work can be recovered.
  2. A delivery that is genuinely queued, or in flight in a worker, is left alone.
  3. No payment or settlement behaviour changes.

Product and security invariants

  • Tenant isolation remains fail-closed.
  • Sponsor authorization, auditability, and daily caps remain enforced where applicable.
  • Recipients cannot modify sponsor controls or access sponsor-only data.
  • No secret, token, production identifier, or personal data is committed or pasted into review prompts.
  • Any non-applicable invariant is explained below.

Invariant notes:

This path creates supplier-retrieval work only: no submit_settlement row, no attempt row, no payment call, asserted by test. 1 intent / at most 1 committed settlement is untouched and the committed payment is preserved, which is the point — the job is recovered without paying again.

Claiming is a compare-and-set against the state read under the job's FOR UPDATE lock, so two concurrent resumes cannot both take one delivery, and the queued-row check runs under that same lock. The fresh delivery_attempt fences any older retrieval from completing, using the existing delivery_attempt guard in completeDelivery and failDelivery.

Selected .agent/TEST_MATRIX.md cases: downstream failure after payment (payment durable, no replacement payment) and duplicate/parallel claim (one delivery claimed once).

Validation

Commands and results:

pnpm format:check: PASS
pnpm lint: PASS
pnpm typecheck: PASS
pnpm test: PASS - 1063 tests, includes build
pnpm --filter @oneshot/storage-postgres exec vitest run test/jobs.test.ts: PASS - 7 tests
Local Node 24.20.0 differs from the pinned 24.19.0; CI must validate the
  pinned runtime

The new coverage is fake-client unit tests at the SQL boundary. The Postgres integration suite (pnpm test:integration) was not run here and no database was available; CI or a reviewer with a database should exercise it.

Independent review evidence

Gate A — exact candidate tree before push

Gate B — exact remote PR head

  • Pull request URL/number: this PR

  • Reviewer tool: free-pi-cli

  • Verdict: skipped, carried over from the requester's instruction on fix(web,jobs): reach Result ready instead of sticking on Retrieving result #141

  • Findings or residual risks: Gate B was not run. Recorded deviation from §7, not a pass.

  • Gate B reviewed the current remote head and matches Gate A's approved tree, or a fresh Gate A was run for the changed tree.

  • Agent policy / repository-policy and all applicable CI checks pass.

Risk and rollback

  • Residual risks:
    • Neither gate was run, on a change to durable job state.
    • Not reproduced against live data: the stranded case is inferred from the state machine. Whether any such job exists today is unverified.
    • Nothing in the web UI calls POST /v1/jobs/:jobId/resume, so recovery still takes an API call. The control that used to do it was removed deliberately in a2903a1; re-adding one is a product decision.
    • Integration tests against Postgres were not run.
  • Rollback or recovery plan: revert the single commit. It changes no schema and no settlement state; reverting restores the previous refusal to resume a PENDING delivery.

Human merge

  • A human owner has reviewed the evidence and will perform the merge.

A committed job whose delivery reached PENDING could be left with nothing able
to move it. The worker no-ops a payload whose delivery_attempt no longer
matches and then marks the outbox row DELIVERED, and resumeDelivery re-queued
only NOT_REQUESTED or RETRIEVAL_FAILED, so the job kept a committed payment, an
unretrieved result, and no path forward.

PENDING alone is still not treated as resumable. The deciding evidence is
whether a fulfill_supplier_order row is still queued for that job: a row a
worker currently holds is status PENDING and so counts as queued, which means
an in-flight retrieval is never duplicated. Only a job with no such row left is
re-queued, under a fresh delivery_attempt that fences the old retrieval.

The claim is a compare-and-set against the state read under the job's FOR
UPDATE lock, so two concurrent resumes cannot both take one delivery. No
payment work is created on this path; the committed settlement is untouched.
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Sep 13, 2026

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
✅ Deployment successful!
View logs
oneshot 996e07b Sep 13 2026, 02:19 PM

@selezenart
selezenart marked this pull request as ready for review September 13, 2026 14:24
@selezenart
selezenart merged commit 5075217 into develop Sep 13, 2026
5 checks passed
@selezenart
selezenart deleted the fix/stranded-pending-delivery branch September 13, 2026 14:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant