Skip to content

docs: require min-instances and no-cpu-throttling for the worker - #145

Merged
selezenart merged 1 commit into
developfrom
docs/worker-deploy-flags
Sep 13, 2026
Merged

selezenart merged 1 commit into
developfrom
docs/worker-deploy-flags

Conversation

@selezenart

Copy link
Copy Markdown
Collaborator

Summary

The documented worker deploy omitted --min-instances=1 and --no-cpu-throttling. Following it as written reproduces the outage we just had.

The worker is not request-driven: it drains the outbox on its own timer, so it only works while its process is alive and holding CPU. Without those flags Cloud Run stops the idle container and throttles CPU between requests, and the timer stops with it. The failure is quiet — payments still settle through the API, because user-wallet settlement never touches the worker — but supplier deliveries sit in PENDING and every request stays on "Retrieving result".

It is also intermittent, which is what made it expensive to diagnose: any request that reaches the service boots an instance, and RestartRunner.start() drains the backlog before the timer takes over, so the queue appears to clear itself and then stalls again.

Scope and acceptance criteria

  • The change is limited to the stated milestone or issue.
  • Acceptance criteria are listed and satisfied.
  • No unrelated cleanup is included.
  1. The documented deploy command produces a worker that actually drains the outbox.
  2. The reason both flags are required is stated, so neither is dropped as noise later.

Documentation only. No source, schema, or behaviour change.

Product and security invariants

  • Tenant isolation remains fail-closed.
  • Sponsor authorization, auditability, and daily caps remain enforced where applicable.
  • Recipients cannot modify sponsor controls or access sponsor-only data.
  • No secret, token, production identifier, or personal data is committed or pasted into review prompts.
  • Any non-applicable invariant is explained below.

Invariant notes:

Not applicable: no code path changes. The added flags affect instance lifecycle only and grant no new authority.

Validation

markdownlint-cli2 docs/MAINNET_READINESS.md: PASS
Verified in production: applying these flags to the running oneshot-worker
  cleared the stalled deliveries.

Independent review evidence

Gate A — exact candidate tree before push

Gate B — exact remote PR head

  • Reviewer tool: free-pi-cli

  • Verdict: not run, same reason.

  • Agent policy / repository-policy and all applicable CI checks pass.

Risk and rollback

  • Residual risks: --min-instances=1 means the worker no longer scales to zero, which costs more than an idle service did. That is the intended trade: a settlement worker that sleeps does not do its job.
  • Rollback or recovery plan: revert the commit. It changes documentation only.

Human merge

  • A human owner has reviewed the evidence and will perform the merge.

The worker drains the outbox on its own timer rather than per request, so it
only works while its process is alive and holding CPU. The documented deploy
omitted both flags, and a deployment without them fails quietly: payments still
settle through the API, but supplier deliveries sit in PENDING and requests stay
on "Retrieving result".

The failure is intermittent, which makes it easy to misdiagnose as a UI or
state-machine fault. Any request that reaches the service boots an instance, and
startup drains the backlog before the timer takes over, so the queue appears to
clear itself and then stalls again.
@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
✅ Deployment successful!
View logs
oneshot 314106e Sep 13 2026, 03:01 PM

@selezenart
selezenart merged commit 8e89d5f into develop Sep 13, 2026
5 checks passed
@selezenart
selezenart deleted the docs/worker-deploy-flags branch September 13, 2026 15:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant