Skip to content

Batch completion: skip the batch row read for jobs that aren't last (1/3) - #8

Open
jpcamara wants to merge 1 commit into
mainfrom
batch-completion-fastpath
Open

Batch completion: skip the batch row read for jobs that aren't last (1/3)#8
jpcamara wants to merge 1 commit into
mainfrom
batch-completion-fastpath

Conversation

@jpcamara

@jpcamara jpcamara commented Aug 24, 2026

Copy link
Copy Markdown
Owner

Stack 1 of 3. Reduces the batch completion path from 5 statements per finishing job to 4.

The problem

Every job that finishes a batch runs this chain:

UPDATE solid_queue_jobs SET finished_at = ...
SELECT solid_queue_batch_executions.*  WHERE job_id = ?     -- has_one load
DELETE FROM solid_queue_batch_executions WHERE id = ?
SELECT id, finished_at, enqueued_at FROM solid_queue_batches WHERE id = ?
SELECT 1 FROM solid_queue_batch_executions WHERE batch_id = ? LIMIT 1

finish_batch reads the batch row and then calls #finish, which immediately returns if the batch still has outstanding tracking rows. So every job but the last one in a batch fetches a row it never uses — 499,999 wasted reads in a 500k batch.

The change

Ask the cheap question first. The batch row is only read by the job that actually finishes the batch.

#finish still performs the same check itself, along with the finished?/enqueued? guards, the CAS with without_executions in its WHERE, and finalize's PostgreSQL stale-snapshot re-check. Nothing here decides anything on its own — it only reorders two read-only checks.

Measured

120k-job batch, PostgreSQL, 4 worker processes × 20 threads:

before after
statements per completion 5 4
completion time 4.63ms 3.90ms
drain rate 345/s 372/s
wall clock 383s 346s

Every run verified 120,000/120,000 completed with zero tracking rows left behind.

Testing

Batch suites green on SQLite and PostgreSQL. RuboCop clean.

Every job that finishes asks whether it was the last one in its batch,
and almost never is. The batch row was being read before that question
was asked, so all but one job per batch paid for a row they didn't use.

Checking for outstanding tracking rows first drops the completion path
from five statements to four. #finish still repeats the check, and the
CAS it guards is unchanged, so nothing here decides anything on its own.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant