Merge latest batch refinements from rails/solid_queue#142

.rubocop.yml

@@ -7,3 +7,4 @@ AllCops:
   TargetRubyVersion: 3.3
   Exclude:
     - "**/*_schema.rb"
+    - "lib/generators/solid_queue/update/templates/db/*"

README.md

@@ -31,6 +31,10 @@ Solid Queue can be used with SQL databases such as MySQL, PostgreSQL, or SQLite,
 - [Failed jobs and retries](#failed-jobs-and-retries)
   - [Error reporting on jobs](#error-reporting-on-jobs)
 - [Batch jobs](#batch-jobs)
+  - [Batch progress and counters](#batch-progress-and-counters)
+  - [Batch maintenance](#batch-maintenance)
+  - [Clearing batches](#clearing-batches)
+  - [Upgrading existing installations](#upgrading-existing-installations)
 - [Puma plugin](#puma-plugin)
 - [Jobs and transactional integrity](#jobs-and-transactional-integrity)
 - [Recurring tasks](#recurring-tasks)
@@ -641,21 +645,19 @@ class ApplicationMailer < ActionMailer::Base
 
 ## Batch jobs
 
-SolidQueue offers support for batching jobs. This allows you to track progress of a set of jobs,
-and optionally trigger callbacks based on their status. It supports the following:
+Solid Queue supports grouping jobs into batches, so you can track the progress of the set as a whole and optionally fire callbacks based on its status. Batches support the following:
 
 - Relating jobs to a batch, to track their status
 - Three available callbacks to fire:
-  - `on_finish`: Fired when all jobs have finished, including retries. Fires even when some jobs have failed.
-  - `on_success`: Fired when all jobs have succeeded, including retries. Will not fire if any jobs have failed, but will fire if jobs have been discarded using `discard_on`
-  - `on_failure`: Fired when all jobs have finished, including retries. Will only fire if one or more jobs have failed.
-- If a job is part of a batch, it can enqueue more jobs for that batch using `batch#enqueue`
-- Attaching arbitrary metadata to a batch
+  - `on_finish`: fired when all jobs have finished, including retries, even when some jobs have failed.
+  - `on_success`: fired when all jobs have succeeded, including retries. It won't fire if any jobs have failed, but it will fire if jobs have been discarded using `discard_on`.
+  - `on_failure`: fired when all jobs have finished, including retries, and one or more of them have failed.
+- Enqueuing more jobs for a batch from inside one of its jobs, with `batch.enqueue`
+- Attaching a description and arbitrary metadata to a batch
 
-Callback jobs are regular jobs: they don't receive any extra arguments, and they can access
-the batch they belong to through the `batch` accessor:
+Callback jobs are regular jobs: the batch doesn't pass them any arguments (although you can configure your own), and they can access the batch they belong to through the `batch` accessor:
 
-```rb
+```ruby
 class SleepyJob < ApplicationJob
   def perform(seconds_to_sleep)
     Rails.logger.info "Feeling #{seconds_to_sleep} seconds sleepy..."
@@ -687,65 +689,35 @@ SolidQueue::Batch.enqueue(
   on_failure: BatchFailureJob,
   user_id: 123
 ) do
-  5.times.map { |i| SleepyJob.perform_later(i) }
+  5.times { |i| SleepyJob.perform_later(i) }
 end
 ```
 
-A job joins the batch that's active when its enqueue is requested. This also works when
-Rails defers the actual enqueue until after the surrounding transaction commits.
+A job joins the batch that's active *when its enqueue is requested*—this also works when Rails defers the actual enqueue until after the surrounding transaction commits. In particular:
 
 - A job created outside a batch and enqueued inside one joins that batch.
-- Creating a job inside a batch without enqueueing it doesn't keep the batch open.
-- If a job already carries a batch ID but is enqueued inside another active batch, the
-  active batch takes precedence.
-
-Callbacks can be given as a job class or as a configured job instance, e.g.
-`on_finish: BatchFinishJob.new.set(queue: :batches)`. Note that the job is serialized when
-the batch is created, so options resolved at that point (like `wait_until:` timestamps) are
-relative to batch creation, not to when the callback is eventually enqueued.
-
-### Batch options
-
-In the case of an empty batch, a `SolidQueue::Batch::EmptyJob` is enqueued.
+- Creating a job inside a batch without enqueueing it doesn't keep the batch open: if the batch finishes before the job is finally enqueued, the enqueue raises `SolidQueue::Batch::AlreadyFinished`.
+- If a job already carries a batch ID but is enqueued inside another active batch, the active batch takes precedence.
 
-By default, this job runs on the `default` queue. You can specify an alternative queue for it in an initializer:
+Besides the callbacks, `SolidQueue::Batch.enqueue` accepts a `description:`, to label the batch, and a `metadata:` hash; any other keyword arguments (like `user_id: 123` above) are merged into the batch's `metadata`.
 
-```rb
-Rails.application.config.after_initialize do # or to_prepare
-  SolidQueue::Batch::EmptyJob.queue_as "my_batch_queue"
-end
-```
+Callbacks can be given as a job class or as a configured job instance—for example, `on_finish: BatchFinishJob.new.set(queue: :batches)` or `on_success: BatchSuccessJob.new("some argument")`. Note that the job is serialized when the batch is created, so options resolved at that point (like `wait_until:` timestamps) are relative to batch creation, not to when the callback is eventually enqueued.
 
-The empty job and batch callback jobs always enqueue through Solid Queue, even when the
-job classes involved (or the application default) use a different Active Job adapter.
+Callback jobs always enqueue through Solid Queue, even when the job classes involved (or the application default) use a different Active Job adapter. And a batch that ends up with no jobs finishes as soon as it starts, firing its callbacks right away.
 
 ### Batch progress and counters
 
-Batches track `total_jobs`, `completed_jobs`, `failed_jobs` and `pending_jobs`, plus a
-`progress_percentage` helper. A couple of accounting details to be aware of:
+Batches track `total_jobs`, `completed_jobs`, `failed_jobs` and `pending_jobs`, plus a `progress_percentage` helper. A couple of accounting details to be aware of:
 
-- Every *attempt* counts: when a job is retried via `retry_on`, each retry is enqueued as a
-  new job in the batch, so a job that fails twice and then succeeds contributes 3 to
-  `total_jobs` (2 completed retries + 1 success).
-- Jobs discarded via `discard_on`, concurrency's `on_conflict: :discard`, or manual
-  discarding count as completed, not failed.
-- Manually retrying a failed job (via `SolidQueue::FailedExecution#retry`) doesn't re-add it
-  to its batch: if the batch already finished as failed, a successful manual retry won't
-  change the batch's status.
+- Counters track *logical* jobs, matching what you enqueued: a retry via `retry_on` keeps the job's Active Job ID, so a job that fails twice and then succeeds still contributes 1 to `total_jobs`. Each attempt does get its own row in the batch's `jobs` relation, though.
+- Jobs discarded via `discard_on`, concurrency's `on_conflict: :discard`, or manual discarding count as completed, not failed.
+- Manually retrying a failed job (via `SolidQueue::FailedExecution#retry`) doesn't re-add it to its batch: if the batch already finished as failed, a successful manual retry won't change the batch's status.
 
 ### Batch maintenance
 
-Batch completion is normally detected as jobs finish, without ever locking the batch row
-outside a single once-per-batch moment. A few edge cases can't trigger that detection: jobs
-removed via bulk discards (which delete jobs without callbacks), a process that crashed
-after enqueueing jobs but before starting its batch, or a completion whose callback
-enqueueing failed and rolled back.
+Batch completion is normally detected as jobs finish, without ever locking the batch row outside a single once-per-batch moment. A few edge cases can't trigger that detection: jobs removed via bulk discards (which delete jobs without callbacks), a process that crashed after enqueueing jobs but before starting its batch, or a completion whose callback enqueueing failed and rolled back.
 
-The dispatcher sweeps these up automatically via `SolidQueue::Batch.sweep_stalled`, as part
-of its regular maintenance (every `concurrency_maintenance_interval` seconds, sharing a
-single maintenance timer and database connection). If you disable `batch_maintenance` (or
-don't run a dispatcher), you can run the sweep yourself, for example as a
-[recurring task](#recurring-tasks):
+The dispatcher sweeps these up automatically via `SolidQueue::Batch.sweep_stalled`, as part of its regular maintenance (every `concurrency_maintenance_interval` seconds, sharing a single maintenance timer and database connection). If you disable `batch_maintenance` (or don't run a dispatcher), you can run the sweep yourself, for example as a [recurring task](#recurring-tasks):
 
 ```yml
 batch_maintenance:
@@ -755,57 +727,25 @@ batch_maintenance:
 
 ### Clearing batches
 
-Finished, non-failed batches are cleared after `config.solid_queue.clear_finished_jobs_after`,
-but only when you invoke it: like jobs, batches are cleared with
-`SolidQueue::Batch.clear_finished_in_batches`, which you'd typically run periodically
-alongside `SolidQueue::Job.clear_finished_in_batches`. Failed batches are kept, like failed
-jobs, so you can inspect them.
+Finished, non-failed batches are cleared with `SolidQueue::Batch.clear_finished_in_batches` after `config.solid_queue.clear_finished_jobs_after`, but only when you invoke it. Failed batches are kept, like failed jobs, so you can inspect them. Installing Solid Queue configures [a recurring task](#recurring-tasks) that clears finished jobs every hour; you can add a matching entry for batches to your `recurring.yml`:
+
+```yml
+clear_solid_queue_finished_batches:
+  command: "SolidQueue::Batch.clear_finished_in_batches(sleep_between_batches: 0.3)"
+  schedule: every hour at minute 12
+```
 
 ### Upgrading existing installations
 
-If you installed Solid Queue before batches existed, add the new tables with a migration in
-`db/queue_migrate`:
+If you installed Solid Queue before batches existed, copy the migration that adds the new tables to your app and run it:
 
-```ruby
-class AddSolidQueueBatches < ActiveRecord::Migration[7.1]
-  def change
-    create_table :solid_queue_batches do |t|
-      t.string :active_job_batch_id
-      t.string :description
-      t.text :on_finish
-      t.text :on_success
-      t.text :on_failure
-      t.text :metadata
-      t.integer :total_jobs, default: 0, null: false
-      t.integer :completed_jobs, default: 0, null: false
-      t.integer :failed_jobs, default: 0, null: false
-      t.datetime :enqueued_at
-      t.datetime :finished_at
-      t.datetime :failed_at
-      t.timestamps
-
-      t.index :active_job_batch_id, unique: true
-      t.index :finished_at
-    end
-
-    create_table :solid_queue_batch_executions do |t|
-      t.bigint :job_id, null: false
-      t.bigint :batch_id, null: false
-      t.datetime :created_at, null: false
-
-      t.index :job_id, unique: true
-      t.index :batch_id
-    end
-
-    add_column :solid_queue_jobs, :batch_id, :bigint
-    add_index :solid_queue_jobs, :batch_id
-
-    add_foreign_key :solid_queue_batch_executions, :solid_queue_batches, column: :batch_id, on_delete: :cascade
-    add_foreign_key :solid_queue_batch_executions, :solid_queue_jobs, column: :job_id, on_delete: :cascade
-  end
-end
+```bash
+bin/rails solid_queue:update
+bin/rails db:migrate
 ```
 
+Until you do, Solid Queue works exactly as before—jobs enqueue and run without any batch bookkeeping, trying to start a batch raises, and the dispatcher logs a deprecation warning to remind you the migration is pending. It becomes part of the base schema in Solid Queue 2.0.
+
 ## Puma plugin
 
 We provide a Puma plugin if you want to run the Solid Queue's supervisor together with Puma and have Puma monitor and manage it. You just need to add

UPGRADING.md

@@ -1,3 +1,13 @@
+# Upgrading to version 1.7.x
+This version introduces support for grouping jobs into batches, which needs new tables. Fresh installs get them with the base schema; existing installations need to copy the migration that adds them and run it:
+
+```bash
+bin/rails solid_queue:update
+bin/rails db:migrate
+```
+
+The migration is optional for now: until you run it, everything works as before, batches aside. It will become part of the required schema in Solid Queue 2.0.
+
 # Upgrading to version 1.5.x
 Ruby 3.1 is no longer supported, as it reached end-of-life in March 2025. Solid Queue now requires Ruby 3.2 or newer. If you're still on Ruby 3.1, Bundler will continue to resolve solid_queue 1.4.x for you, but you won't receive any new versions until you upgrade Ruby.