Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 19 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ Solid Queue can be used with SQL databases such as MySQL, PostgreSQL, or SQLite,
- [Threads, processes, and signals](#threads-processes-and-signals)
- [Database configuration](#database-configuration)
- [Other configuration settings](#other-configuration-settings)
- [Validating the configuration](#validating-the-configuration)
- [Lifecycle hooks](#lifecycle-hooks)
- [Errors when enqueuing](#errors-when-enqueuing)
- [Concurrency controls](#concurrency-controls)
Expand Down Expand Up @@ -264,7 +265,7 @@ Here's an overview of the different options:
```

This will create a worker fetching jobs from all queues starting with `staging`. The wildcard `*` is only allowed on its own or at the end of a queue name; you can't specify queue names such as `*_some_queue`. These will be ignored.

Also, if a wildcard (*) is included alongside explicit queue names, for example: `queues: [default, backend, *]`, then it would behave like `queues: *`

Finally, you can combine prefixes with exact names, like `[ staging*, background ]`, and the behaviour with respect to order will be the same as with only exact names.
Expand Down Expand Up @@ -408,6 +409,22 @@ There are several settings that control how Solid Queue works that you can set a
- `clear_finished_jobs_after`: period to keep finished jobs around, in case `preserve_finished_jobs` is true — defaults to 1 day. When installing Solid Queue, [a recurring job](#recurring-tasks) is automatically configured to clear finished jobs every hour on the 12th minute in batches. You can edit the `recurring.yml` configuration to change this as you see fit.
- `default_concurrency_control_period`: the value to be used as the default for the `duration` parameter in [concurrency controls](#concurrency-controls). It defaults to 3 minutes.

### Validating the configuration

You can validate the Solid Queue configuration ahead of time, which is handy in deploy scripts or CI:

```bash
# Using the bin/jobs binstub
bin/jobs check

# Or via rake
bin/rails solid_queue:check
```

Both commands validate the same configuration for the current Rails environment, exit non-zero on any error, and print `Solid Queue configuration is valid` on success. They are tolerant of a missing database connection so they can run on CI or deploy hosts without DB credentials.

`bin/jobs check` accepts the same flags as `bin/jobs start` (e.g. `--config_file`, `--recurring_schedule_file`, `--skip-recurring`). The rake task is configured via the same environment variables Solid Queue already honors: `SOLID_QUEUE_CONFIG`, `SOLID_QUEUE_RECURRING_SCHEDULE`, and `SOLID_QUEUE_SKIP_RECURRING`. To validate a specific environment's configuration, prefix with `RAILS_ENV`, for example `RAILS_ENV=production bin/jobs check`.


## Lifecycle hooks

Expand Down Expand Up @@ -781,7 +798,7 @@ SolidQueue.unschedule_recurring_task("my_dynamic_task")

Only dynamic tasks can be unscheduled at runtime. Attempting to unschedule a static task (defined in `config/recurring.yml`) will raise an `ActiveRecord::RecordNotFound` error.

Tasks scheduled like this persist between Solid Queue's restarts and won't stop running until you manually unschedule them.
Tasks scheduled like this persist between Solid Queue's restarts and won't stop running until you manually unschedule them.

## Inspiration

Expand Down
6 changes: 6 additions & 0 deletions lib/solid_queue/cli.rb
Original file line number Diff line number Diff line change
Expand Up @@ -30,5 +30,11 @@ def self.exit_on_failure?
def start
SolidQueue::Supervisor.start(**options.symbolize_keys)
end

desc :check, "Validates the Solid Queue configuration for the current Rails env without starting anything. Exits non-zero on errors."
def check
configuration = SolidQueue::Configuration.new(**options.symbolize_keys, skip_db_checks: true)
exit 1 unless configuration.check!
end
end
end
24 changes: 23 additions & 1 deletion lib/solid_queue/configuration.rb
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,19 @@ def standalone?
mode.fork? || @options[:standalone]
end

def check!(out: $stdout, err: $stderr)
if valid?
out.puts "Solid Queue configuration is valid."
return true
end

err.puts "Invalid Solid Queue configuration:"
errors.full_messages.each do |message|
message.each_line { |line| err.puts " #{line.chomp}" }
end
false
end

private
attr_reader :options

Expand All @@ -89,12 +102,21 @@ def ensure_valid_recurring_tasks
end

def ensure_correctly_sized_thread_pool
if (db_pool_size = SolidQueue::Record.connection_pool&.size) && db_pool_size < estimated_number_of_threads
if (db_pool_size = database_connection_pool_size) && db_pool_size < estimated_number_of_threads
errors.add(:base, "Solid Queue is configured to use #{estimated_number_of_threads} threads but the " +
"database connection pool is #{db_pool_size}. Increase it in `config/database.yml`")
end
end

# Returns nil instead of raising when skip_db_checks is set, so
# `bin/jobs check` works on CI/deploy hosts without DB credentials.
def database_connection_pool_size
SolidQueue::Record.connection_pool&.size
rescue ActiveRecord::ActiveRecordError
raise unless options[:skip_db_checks]
nil
end

def default_options
{
mode: ENV["SOLID_QUEUE_SUPERVISOR_MODE"] || :fork,
Expand Down
6 changes: 6 additions & 0 deletions lib/solid_queue/tasks.rb
Original file line number Diff line number Diff line change
Expand Up @@ -8,4 +8,10 @@
task start: :environment do
SolidQueue::Supervisor.start
end

desc "validate the Solid Queue configuration for the current Rails env without starting any process"
task check: :environment do
configuration = SolidQueue::Configuration.new(skip_db_checks: true)
exit 1 unless configuration.check!
end
end
35 changes: 35 additions & 0 deletions test/unit/cli_test.rb
Original file line number Diff line number Diff line change
Expand Up @@ -36,11 +36,46 @@ class CliTest < ActiveSupport::TestCase
end
end

test "check exits 0 and prints OK message for a valid configuration" do
out, err = capture_io do
assert_nothing_raised { SolidQueue::Cli.start([ "check", "--skip-recurring" ]) }
end

assert_match "Solid Queue configuration is valid.", out
assert_empty err
end

test "check exits 1 and prints invalid recurring task errors" do
out, err, exit_status = capture_check_run(
"--recurring_schedule_file", config_file_path(:recurring_with_invalid).to_s
)

assert_equal 1, exit_status
assert_match "Invalid Solid Queue configuration", err
assert_match "periodic_invalid_class", err
assert_match "periodic_incorrect_schedule", err
assert_empty out
end

private
def configuration_from_cli(**cli_options)
cli = SolidQueue::Cli.new([], cli_options)
options = cli.options.symbolize_keys.compact

SolidQueue::Configuration.new(**options)
end

# capture_io re-raises SystemExit before returning the captured strings.
# It swallows it inside the block and returns its status alongside the IO.
def capture_check_run(*args)
status = nil
out, err = capture_io do
begin
SolidQueue::Cli.start([ "check", *args ])
rescue SystemExit => e
status = e.status
end
end
[ out, err, status ]
end
end
48 changes: 48 additions & 0 deletions test/unit/configuration_test.rb
Original file line number Diff line number Diff line change
Expand Up @@ -172,6 +172,54 @@ class ConfigurationTest < ActiveSupport::TestCase
configuration.errors.full_messages.first
end

test "skips DB-pool validation gracefully when skip_db_checks is set and the pool is unavailable" do
SolidQueue::Record.stubs(:connection_pool).raises(ActiveRecord::ConnectionNotEstablished)

configuration = SolidQueue::Configuration.new(
workers: [ { queues: "background", threads: 50, polling_interval: 10 } ],
skip_recurring: true,
skip_db_checks: true
)

assert configuration.valid?,
"Expected configuration to be valid when DB pool is unavailable, got: #{configuration.errors.full_messages.inspect}"
end

test "still raises DB connection errors during normal validation when skip_db_checks is not set" do
SolidQueue::Record.stubs(:connection_pool).raises(ActiveRecord::ConnectionNotEstablished)

configuration = SolidQueue::Configuration.new(
workers: [ { queues: "background", threads: 50, polling_interval: 10 } ],
skip_recurring: true
)

assert_raises(ActiveRecord::ConnectionNotEstablished) { configuration.valid? }
end

test "check! prints success message and returns true for a valid configuration" do
configuration = SolidQueue::Configuration.new(skip_recurring: true)
out = StringIO.new
err = StringIO.new

assert configuration.check!(out: out, err: err)
assert_match "Solid Queue configuration is valid.", out.string
assert_empty err.string
end

test "check! prints errors to err and returns false for an invalid configuration" do
configuration = SolidQueue::Configuration.new(
recurring_schedule_file: config_file_path(:recurring_with_invalid),
skip_db_checks: true
)
out = StringIO.new
err = StringIO.new

assert_not configuration.check!(out: out, err: err)
assert_empty out.string
assert_match "Invalid Solid Queue configuration:", err.string
assert_match "periodic_invalid_class", err.string
end

private
def assert_processes(configuration, kind, count, **attributes)
processes = configuration.configured_processes.select { |p| p.kind == kind }
Expand Down
50 changes: 50 additions & 0 deletions test/unit/rake_tasks_test.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
# frozen_string_literal: true

require "test_helper"
require "rake"

class RakeTasksTest < ActiveSupport::TestCase
setup do
@previous_rake_application = Rake.application
@rake = Rake::Application.new
Rake.application = @rake
Rake::Task.define_task(:environment)
load File.expand_path("../../lib/solid_queue/tasks.rb", __dir__)
end

teardown do
Rake.application = @previous_rake_application
end

test "solid_queue:check exits 0 and prints OK message for a valid configuration" do
SolidQueue::Configuration.any_instance.stubs(:skip_recurring_tasks?).returns(true)

out, err = capture_io do
assert_nothing_raised { @rake["solid_queue:check"].invoke }
end

assert_match "Solid Queue configuration is valid.", out
assert_empty err
end

test "solid_queue:check exits 1 and prints errors for an invalid configuration" do
SolidQueue::Configuration.any_instance.stubs(:invalid_tasks).returns(
[ stub(key: "broken", errors: stub(full_messages: [ "is invalid" ])) ]
)
SolidQueue::Configuration.any_instance.stubs(:skip_recurring_tasks?).returns(false)

status = nil
out, err = capture_io do
begin
@rake["solid_queue:check"].invoke
rescue SystemExit => e
status = e.status
end
end

assert_equal 1, status
assert_empty out
assert_match "Invalid Solid Queue configuration:", err
assert_match "broken", err
end
end
Loading