From 8ea6d5d7b4487bff769f1dddf877471a5ef98a14 Mon Sep 17 00:00:00 2001 From: Cole Robertson Date: Wed, 9 Sep 2026 04:41:39 +0000 Subject: [PATCH] Update to Litestream 0.5.17 without a daemon socket The bundled binary moves from 0.3.13 to 0.5.17. Release tarballs are pinned by sha256 in Upstream::CHECKSUMS and verified before extraction; the darwin zip assets are gone upstream, so rubyzip goes too. Litestream 0.5 removed generations, snapshots and wal in favour of TXID-numbered LTX files. Commands gains ltx and status, both with a json: true option that parses the CLI's -json output. The dashboard is rebuilt on those two commands plus databases, so it keeps working when litestream replicate runs in a different container from Rails (Kamal, Docker): status reads local state and ltx reads the replica, neither needs the daemon's IPC socket. Process detection via systemctl and ps is unchanged and still matches a 0.5 replicate. The generator template and README use the 0.5 config shape (single replica: per database, root-level snapshot: block). restore keeps its signature; 0.5 auto-detects 0.3 backups and prefers whichever is newer. --- CHANGELOG.md | 9 +- Gemfile.lock | 4 +- README.md | 124 ++++---- app/views/litestream/processes/show.html.erb | 107 ++++--- lib/litestream.rb | 28 +- lib/litestream/commands.rb | 24 +- .../litestream/templates/config.yml.erb | 17 +- lib/litestream/upstream.rb | 15 +- lib/litestream/version.rb | 2 +- lib/tasks/litestream_tasks.rake | 22 +- litestream.gemspec | 1 - rakelib/package.rake | 26 +- test/controllers/test_processes_controller.rb | 26 +- test/dummy/config/litestream.yml | 18 +- test/generators/test_install.rb | 4 + test/litestream/test_commands.rb | 289 ++++++++---------- test/tasks/test_litestream_tasks.rb | 105 ++++--- test/test_litestream.rb | 38 +++ 18 files changed, 429 insertions(+), 430 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5b61a0f..f98d2e8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,4 +1,11 @@ -## [Unreleased] +## [0.15.0] - Unreleased + +- Bundle Litestream 0.5.17 release archives and verify their SHA-256 checksums during packaging. +- Replace the removed `generations`, `snapshots`, and `wal` wrappers and rake tasks with `ltx` and `status`. +- Support JSON CLI output and use daemon-free `databases`, `status`, and `ltx` calls for dashboard data. +- Show local status, LTX levels, snapshots, latest transaction IDs, and transaction lag in the dashboard. +- Generate Litestream 0.5 configuration with a single replica and global snapshot settings. +- Document restoration and migration from Litestream 0.3 backups. ## [0.14.0] - 2025-06-14 diff --git a/Gemfile.lock b/Gemfile.lock index 51f11ba..7ac01dd 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -1,7 +1,7 @@ PATH remote: . specs: - litestream (0.14.0) + litestream (0.15.0) actionpack (>= 7.0) actionview (>= 7.0) activejob (>= 7.0) @@ -204,7 +204,6 @@ GEM rubocop (>= 1.48.1, < 2.0) rubocop-ast (>= 1.30.0, < 2.0) ruby-progressbar (1.13.0) - rubyzip (2.3.2) securerandom (0.4.1) sqlite3 (2.6.0-arm64-darwin) sqlite3 (2.6.0-x86_64-linux-gnu) @@ -244,7 +243,6 @@ DEPENDENCIES minitest (~> 5.0) rails rake (~> 13.0) - rubyzip standard (~> 1.3) BUNDLED WITH diff --git a/README.md b/README.md index e5c3056..92f7711 100644 --- a/README.md +++ b/README.md @@ -85,16 +85,22 @@ The gem streamlines the configuration process by providing a default configurati The default configuration file looks like this if you only have one SQLite database: ```yaml +snapshot: + interval: 24h + retention: 24h + dbs: - path: storage/production.sqlite3 - replicas: - - type: s3 - path: storage/production.sqlite3 - bucket: $LITESTREAM_REPLICA_BUCKET - access-key-id: $LITESTREAM_ACCESS_KEY_ID - secret-access-key: $LITESTREAM_SECRET_ACCESS_KEY + replica: + type: s3 + path: storage/production.sqlite3 + bucket: $LITESTREAM_REPLICA_BUCKET + access-key-id: $LITESTREAM_ACCESS_KEY_ID + secret-access-key: $LITESTREAM_SECRET_ACCESS_KEY ``` +In Litestream 0.5, snapshot interval and retention are global settings rather than per-replica settings. + This is the default for Amazon S3. The full range of possible replica types (e.g. other S3-compatible object storage servers) are covered in Litestream's [replica guides](https://litestream.io/guides/#replica-guides). The gem also provides a default initializer file at `config/initializers/litestream.rb` that allows you to configure various variables referenced in the configuration file in Ruby. By providing a Ruby interface to these environment variables, you can use your preferred method of storing secrets. For example, the default generated file uses Rails' encrypted credentials to store your secrets. @@ -179,7 +185,7 @@ You can restore any replicated database at any point using the gem's provided `l > [!NOTE] > During the restoration process, you need to prevent any interaction with ActiveRecord/SQLite, such as from a running `rails server` or `rails console` instance. If there is any interaction, Rails might regenerate the production database and prevent restoration via litestream. If this happens, you might get a "cannot restore, output path already exists" error. -1. Rename the production (`production.sqlite3`, `production.sqlite3-shm`, and `production.sqlite3-wal`) databases (**recommended**) or alternatively delete. To delete the production databases locally, you can run the following at your own risk: +1. Rename the production database and its SQLite sidecar files (**recommended**) or alternatively delete them. To delete the production databases locally, you can run the following at your own risk: ```shell # DANGEROUS OPERATION, consider renaming database files instead bin/rails db:drop DISABLE_DATABASE_ENVIRONMENT_CHECK=1 @@ -215,26 +221,19 @@ You can forward arguments in whatever order you like, you simply need to ensure -if-replica-exists Returns exit code of 0 if no backups found. --parallelism NUM - Determines the number of WAL files downloaded in parallel. - Defaults to 8 - --replica NAME - Restore from a specific replica. - Defaults to replica with latest data. - --generation NAME - Restore from a specific generation. - Defaults to generation with latest data. - --index NUM - Restore up to a specific WAL index (inclusive). - Defaults to use the highest available index. +-txid TXID + Restore through a specific 16-character hexadecimal transaction ID. -timestamp TIMESTAMP Restore to a specific point-in-time. Defaults to use the latest available backup. +-json + Print the restore result as JSON. + +-dry-run + Print the restore plan without writing the database. + -config PATH Specifies the configuration file. Defaults to /etc/litestream.yml @@ -387,79 +386,59 @@ bin/rails litestream:databases This will return a list of databases and their configured replicas: ``` -path replicas +path replica /Users/you/Code/your-app/storage/production.sqlite3 s3 ``` -You can also list the generations of a specific database: - -```shell -bin/rails litestream:generations -- --database=storage/production.sqlite3 -``` - -This will list all generations for the specified database, including stats about their lag behind the primary database and the time range they cover: - -``` -name generation lag start end -s3 a295b16a796689f3 -156ms 2024-04-17T00:01:19Z 2024-04-17T00:01:19Z -``` - -You can list the snapshots available for a database: +You can inspect local replication status for all databases or one specific database: ```shell -bin/rails litestream:snapshots -- --database=storage/production.sqlite3 +bin/rails litestream:status -- --database=storage/production.sqlite3 ``` -This command lists snapshots available for that specified database: +This reads local state and does not require a running Litestream process: ``` -replica generation index size created -s3 a295b16a796689f3 1 4645465 2024-04-17T00:01:19Z +database status local_txid wal_size +/Users/you/Code/your-app/storage/production.sqlite3 ok 000000000000000a 128 kB ``` -Finally, you can list the wal files available for a database: +You can list all remote LTX files, including the level-9 snapshot: ```shell -bin/rails litestream:wal -- --database=storage/production.sqlite3 +bin/rails litestream:ltx -- --database=storage/production.sqlite3 --level=all ``` -This command lists wal files available for that specified database: +The command returns the compaction level, transaction range, byte size, and creation time: ``` -replica generation index offset size created -s3 a295b16a796689f3 1 0 2036 2024-04-17T00:01:19Z +level min_txid max_txid size created +0 0000000000000008 000000000000000a 1013 2026-09-08T03:16:43Z ``` ### Running commands from Ruby In addition to the provided rake tasks, you can also run Litestream commands directly from Ruby. The gem provides a `Litestream::Commands` module that wraps the Litestream CLI commands. This is particularly useful for the introspection commands, as you can use the output in your Ruby code. -The `Litestream::Commands.databases` method returns an array of hashes with the "path" and "replicas" keys for each database: +Pass `json: true` to return the 0.5 CLI's JSON values with string keys. The `Litestream::Commands.databases` method returns each database path and replica type: ```ruby -Litestream::Commands.databases -# => [{"path"=>"/Users/you/Code/your-app/storage/production.sqlite3", "replicas"=>"s3"}] +Litestream::Commands.databases(json: true) +# => [{"path"=>"/Users/you/Code/your-app/storage/production.sqlite3", "replica"=>"s3"}] ``` -The `Litestream::Commands.generations` method returns an array of hashes with the "name", "generation", "lag", "start", and "end" keys for each generation: +The `status` method returns local state without connecting to the replica or requiring a daemon: ```ruby -Litestream::Commands.generations('storage/production.sqlite3') -# => [{"name"=>"s3", "generation"=>"5f4341bc3d22d615", "lag"=>"3s", "start"=>"2024-04-17T19:48:09Z", "end"=>"2024-04-17T19:48:09Z"}] +Litestream::Commands.status("storage/production.sqlite3", json: true) +# => [{"database"=>"storage/production.sqlite3", "status"=>"ok", "local_txid"=>"000000000000000a", "wal_size"=>"128 kB"}] ``` -The `Litestream::Commands.snapshots` method returns an array of hashes with the "replica", "generation", "index", "size", and "created" keys for each snapshot: +The `ltx` method lists remote LTX files. An unreplicated database returns an empty array: ```ruby -Litestream::Commands.snapshots('storage/production.sqlite3') -# => [{"replica"=>"s3", "generation"=>"5f4341bc3d22d615", "index"=>"0", "size"=>"4645465", "created"=>"2024-04-17T19:48:09Z"}] -``` - -The `Litestream::Commands.wal` method returns an array of hashes with the "replica", "generation", "index", "offset","size", and "created" keys for each wal: - -```ruby -Litestream::Commands.wal('storage/production.sqlite3') -# => [{"replica"=>"s3", "generation"=>"5f4341bc3d22d615", "index"=>"0", "offset"=>"0", "size"=>"2036", "created"=>"2024-04-17T19:48:09Z"}] +Litestream::Commands.ltx("storage/production.sqlite3", json: true, "--level" => "all") +# => [{"level"=>0, "min_txid"=>"0000000000000008", "max_txid"=>"000000000000000a", "size"=>1013, "timestamp"=>"2026-09-08T03:16:43Z"}] ``` You can also restore a database programmatically using the `Litestream::Commands.restore` method, which returns the path to the restored database: @@ -479,14 +458,27 @@ The full set of commands available to the `litestream` executable are covered in ```shell litestream databases [arguments] -litestream generations [arguments] DB_PATH|REPLICA_URL +litestream info [arguments] +litestream list [arguments] +litestream ltx [arguments] DB_PATH +litestream register [arguments] litestream replicate [arguments] -litestream restore [arguments] DB_PATH|REPLICA_URL -litestream snapshots [arguments] DB_PATH|REPLICA_URL +litestream reset [arguments] +litestream restore [arguments] DB_PATH +litestream start [arguments] +litestream status [arguments] [DB_PATH] +litestream stop [arguments] +litestream sync [arguments] +litestream unregister [arguments] litestream version -litestream wal [arguments] DB_PATH|REPLICA_URL ``` +### Upgrading from 0.3 + +Litestream 0.5 changes each database from a `replicas:` list to a single `replica:` map and moves retention and snapshot settings into the root-level `snapshot:` block. Update existing configuration files before upgrading. + +The 0.5 `restore` command auto-detects both 0.3 and LTX backups and prefers whichever backup is newer. Existing 0.3 `generations/` objects are left in place; you can delete them after the LTX history covers your full retention window. To roll back, reinstall a 0.14.x release of this gem, which bundles Litestream 0.3. + ### Using in development By default, if you install the gem and configure via `puma.rb` or `Procfile`, Litestream will not start in development. diff --git a/app/views/litestream/processes/show.html.erb b/app/views/litestream/processes/show.html.erb index d8c8e31..070c905 100644 --- a/app/views/litestream/processes/show.html.erb +++ b/app/views/litestream/processes/show.html.erb @@ -56,62 +56,61 @@
-
- <% database['generations'].each do |generation| %> -
- - <%= generation['generation'] %> - (<%= generation['lag'] %> lag) - - -
-
Start
-
- - - -
+
+ <% if database['error'] %> +

Error: <%= database['error'] %>

+ <% else %> + <% status = database['status'] || {} %> +

+ Status: <%= status['status'] || 'unknown' %> + · Local TXID: <%= status['local_txid'] || '-' %> + · WAL size: <%= status['wal_size'] || '-' %> +

-
End
-
- - - -
+
+
Latest TXID
+
<%= database.dig('latest', 'max_txid') || '-' %>
+
Snapshot
+
+ <% if database['snapshot'] %> + + (<%= number_to_human_size database['snapshot']['size'] %>) + <% else %> + none + <% end %> +
+
Levels
+
<%= database['levels'].presence&.sort&.map { |level, count| "L#{level}: #{count}" }&.join(', ') || 'none' %>
+
Lag
+
<%= database['lag_txids'].nil? ? '-' : "#{database['lag_txids']} txids" %>
+
-
-
Snapshots
-
- - - - - - - - - - - <% generation['snapshots'].each do |snapshot| %> - - - - - - <% end %> - -
Created atIndexSize
- - - - - <%= snapshot['index'] %> - - <%= number_to_human_size snapshot['size'] %> -
-
-
-
+
+ + LTX files (<%= database['ltx'].size %>) + + + + + + + + + + + + <% database['ltx'].each do |ltx| %> + + + + + + + <% end %> + +
LevelTXID rangeSizeCreated
<%= ltx['level'] %><%= ltx['min_txid'] %>–<%= ltx['max_txid'] %><%= number_to_human_size ltx['size'] %> + +
<% end %>
diff --git a/lib/litestream.rb b/lib/litestream.rb index 159faa9..602bde1 100644 --- a/lib/litestream.rb +++ b/lib/litestream.rb @@ -107,20 +107,26 @@ def replicate_process end def databases - databases = Commands.databases + databases = Commands.databases(json: true) databases.each do |db| - generations = Commands.generations(db["path"]) - snapshots = Commands.snapshots(db["path"]) - db["path"] = db["path"].gsub(Rails.root.to_s, "[ROOT]") - - db["generations"] = generations.map do |generation| - id = generation["generation"] - replica = generation["name"] - generation["snapshots"] = snapshots.select { |snapshot| snapshot["generation"] == id && snapshot["replica"] == replica } - .map { |s| s.slice("index", "size", "created") } - generation.slice("generation", "name", "lag", "start", "end", "snapshots") + path = db["path"] + begin + db["status"] = Commands.status(path, json: true).first + db["ltx"] = Commands.ltx(path, **{"json" => true, "--level" => "all"}) + db["levels"] = db["ltx"].each_with_object(Hash.new(0)) { |entry, levels| levels[entry["level"]] += 1 } + db["snapshot"] = db["ltx"].select { |entry| entry["level"] == 9 }.max_by { |entry| entry["timestamp"] } + db["latest"] = db["ltx"].max_by { |entry| Integer(entry["max_txid"], 16) } + + local_txid = db.dig("status", "local_txid") + db["lag_txids"] = if local_txid && local_txid != "-" && db["latest"] + Integer(local_txid, 16) - Integer(db["latest"]["max_txid"], 16) + end + rescue => error + db["error"] = error.message end + + db["path"] = path.gsub(Rails.root.to_s, "[ROOT]") end end diff --git a/lib/litestream/commands.rb b/lib/litestream/commands.rb index 0d86aad..6745c46 100644 --- a/lib/litestream/commands.rb +++ b/lib/litestream/commands.rb @@ -1,4 +1,5 @@ require_relative "upstream" +require "json" module Litestream module Commands @@ -114,22 +115,14 @@ def databases(**argv) execute("databases", argv) end - def generations(database, **argv) - raise DatabaseRequiredException, "database argument is required for generations command, e.g. litestream:generations -- --database=path/to/database.sqlite" if database.nil? + def ltx(database, **argv) + raise DatabaseRequiredException, "database argument is required for ltx command, e.g. litestream:ltx -- --database=path/to/database.sqlite" if database.nil? - execute("generations", argv, database) + execute("ltx", argv, database) end - def snapshots(database, **argv) - raise DatabaseRequiredException, "database argument is required for snapshots command, e.g. litestream:snapshots -- --database=path/to/database.sqlite" if database.nil? - - execute("snapshots", argv, database) - end - - def wal(database, **argv) - raise DatabaseRequiredException, "database argument is required for wal command, e.g. litestream:wal -- --database=path/to/database.sqlite" if database.nil? - - execute("wal", argv, database) + def status(database = nil, **argv) + execute("status", argv, database) end private @@ -152,9 +145,11 @@ def prepare(command, argv = {}, database = nil) ENV["LITESTREAM_ACCESS_KEY_ID"] ||= Litestream.replica_key_id ENV["LITESTREAM_SECRET_ACCESS_KEY"] ||= Litestream.replica_access_key + argv = argv.stringify_keys + argv["-json"] = nil if argv.delete("json") args = { "--config" => Litestream.config_path.to_s - }.merge(argv.stringify_keys).to_a.flatten.compact + }.merge(argv).to_a.flatten.compact cmd = [executable, command, *args, database].compact puts cmd.inspect if ENV["DEBUG"] @@ -163,6 +158,7 @@ def prepare(command, argv = {}, database = nil) def run(cmd, tabled_output:) stdout = `#{cmd.join(" ")}`.chomp + return JSON.parse(stdout) if cmd.include?("--json") || cmd.include?("-json") return stdout unless tabled_output keys, *rows = stdout.split("\n").map { _1.split(/\s+/) } diff --git a/lib/litestream/generators/litestream/templates/config.yml.erb b/lib/litestream/generators/litestream/templates/config.yml.erb index 43009db..f789f8a 100644 --- a/lib/litestream/generators/litestream/templates/config.yml.erb +++ b/lib/litestream/generators/litestream/templates/config.yml.erb @@ -9,13 +9,18 @@ # `replicate` command. # # For more details, see: https://litestream.io/reference/config/ +# Retention and snapshot settings are global in Litestream 0.5. +snapshot: + interval: 24h + retention: 24h + dbs: <%- production_sqlite_databases.each do |database| -%> - path: <%= database %> - replicas: - - type: s3 - bucket: $LITESTREAM_REPLICA_BUCKET - path: <%= database %> - access-key-id: $LITESTREAM_ACCESS_KEY_ID - secret-access-key: $LITESTREAM_SECRET_ACCESS_KEY + replica: + type: s3 + bucket: $LITESTREAM_REPLICA_BUCKET + path: <%= database %> + access-key-id: $LITESTREAM_ACCESS_KEY_ID + secret-access-key: $LITESTREAM_SECRET_ACCESS_KEY <%- end -%> diff --git a/lib/litestream/upstream.rb b/lib/litestream/upstream.rb index 0029efd..0209e39 100644 --- a/lib/litestream/upstream.rb +++ b/lib/litestream/upstream.rb @@ -1,14 +1,21 @@ module Litestream module Upstream - VERSION = "v0.3.13" + VERSION = "0.5.17" # rubygems platform name => upstream release filename NATIVE_PLATFORMS = { "aarch64-linux" => "litestream-#{VERSION}-linux-arm64.tar.gz", - "arm64-darwin" => "litestream-#{VERSION}-darwin-arm64.zip", + "arm64-darwin" => "litestream-#{VERSION}-darwin-arm64.tar.gz", "arm64-linux" => "litestream-#{VERSION}-linux-arm64.tar.gz", - "x86_64-darwin" => "litestream-#{VERSION}-darwin-amd64.zip", - "x86_64-linux" => "litestream-#{VERSION}-linux-amd64.tar.gz" + "x86_64-darwin" => "litestream-#{VERSION}-darwin-x86_64.tar.gz", + "x86_64-linux" => "litestream-#{VERSION}-linux-x86_64.tar.gz" + } + + CHECKSUMS = { + "litestream-0.5.17-linux-x86_64.tar.gz" => "cfb371176d164437ae869f8351cfde49bd1804ae71c61923f75c9cba9c9c006d", + "litestream-0.5.17-linux-arm64.tar.gz" => "f8ca4a050095c1efbda2c4365172e61bf9d955ea0d9ac42f448b52e51819baa5", + "litestream-0.5.17-darwin-x86_64.tar.gz" => "891875af09db152e93a4b31a8a79f538ce7ce702c132803cfe0a831e7cb1b7db", + "litestream-0.5.17-darwin-arm64.tar.gz" => "e211f68ff7658d19f193f2914417afdf8f89a053ff8f263e5d6b3b1d3bbc7b08" } end end diff --git a/lib/litestream/version.rb b/lib/litestream/version.rb index df8a12c..e203145 100644 --- a/lib/litestream/version.rb +++ b/lib/litestream/version.rb @@ -1,3 +1,3 @@ module Litestream - VERSION = "0.14.0" + VERSION = "0.15.0" end diff --git a/lib/tasks/litestream_tasks.rake b/lib/tasks/litestream_tasks.rake index 387701b..b51418f 100644 --- a/lib/tasks/litestream_tasks.rake +++ b/lib/tasks/litestream_tasks.rake @@ -32,30 +32,20 @@ namespace :litestream do puts Litestream::Commands::Output.format(Litestream::Commands.databases(**options)) end - desc "List all generations for a database or replica, for example `rake litestream:generations -- -database=storage/production.sqlite3`" - task generations: :environment do + desc "List all LTX files for a database or replica, for example `rake litestream:ltx -- -database=storage/production.sqlite3`" + task ltx: :environment do options = parse_argv_options database = options.delete(:"--database") || options.delete(:"-database") - puts Litestream::Commands::Output.format(Litestream::Commands.generations(database, **options)) + puts Litestream::Commands::Output.format(Litestream::Commands.ltx(database, **options)) end - desc "List all snapshots for a database or replica, for example `rake litestream:snapshots -- -database=storage/production.sqlite3`" - task snapshots: :environment do + desc "Show replication status, for example `rake litestream:status -- -database=storage/production.sqlite3`" + task status: :environment do options = parse_argv_options database = options.delete(:"--database") || options.delete(:"-database") - puts Litestream::Commands::Output.format(Litestream::Commands.snapshots(database, **options)) - end - - desc "List all wal files for a database or replica, for example `rake litestream:wal -- -database=storage/production.sqlite3`" - task wal: :environment do - options = parse_argv_options - database = options.delete(:"--database") || options.delete(:"-database") - - puts Litestream::Commands::Output.format( - Litestream::Commands.wal(database, **options) - ) + puts Litestream::Commands::Output.format(Litestream::Commands.status(database, **options)) end private diff --git a/litestream.gemspec b/litestream.gemspec index e004350..e19400f 100644 --- a/litestream.gemspec +++ b/litestream.gemspec @@ -34,7 +34,6 @@ Gem::Specification.new do |spec| spec.add_dependency "railties", rails_version end spec.add_development_dependency "rails" - spec.add_development_dependency "rubyzip" # For more information and examples about making a new gem, check out our # guide at: https://bundler.io/guides/creating_gem.html diff --git a/rakelib/package.rake b/rakelib/package.rake index f1a047f..eb0c45c 100644 --- a/rakelib/package.rake +++ b/rakelib/package.rake @@ -58,12 +58,13 @@ require "rubygems/package" require "rubygems/package_task" require "open-uri" +require "digest" +require "stringio" require "zlib" -require "zip" require_relative "../lib/litestream/upstream" def litestream_download_url(filename) - "https://github.com/benbjohnson/litestream/releases/download/#{Litestream::Upstream::VERSION}/#{filename}" + "https://github.com/benbjohnson/litestream/releases/download/v#{Litestream::Upstream::VERSION}/#{filename}" end LITESTREAM_RAILS_GEMSPEC = Bundler.load_gemspec("litestream.gemspec") @@ -95,16 +96,17 @@ Litestream::Upstream::NATIVE_PLATFORMS.each do |platform, filename| # lazy, but fine for now. URI.open(release_url) do |remote| # standard:disable Security/Open - if release_url.end_with?(".zip") - Zip::File.open_buffer(remote) do |zip_file| - zip_file.extract("litestream", exepath) - end - elsif release_url.end_with?(".gz") - Zlib::GzipReader.wrap(remote) do |gz| - Gem::Package::TarReader.new(gz) do |reader| - reader.seek("litestream") do |file| - File.binwrite(exepath, file.read) - end + archive = remote.read + expected_checksum = Litestream::Upstream::CHECKSUMS.fetch(filename) + actual_checksum = Digest::SHA256.hexdigest(archive) + unless actual_checksum == expected_checksum + raise "Checksum mismatch for #{filename}: expected #{expected_checksum}, got #{actual_checksum}" + end + + Zlib::GzipReader.wrap(StringIO.new(archive)) do |gz| + Gem::Package::TarReader.new(gz) do |reader| + reader.seek("litestream") do |file| + File.binwrite(exepath, file.read) end end end diff --git a/test/controllers/test_processes_controller.rb b/test/controllers/test_processes_controller.rb index 3f15a6e..4c3271b 100644 --- a/test/controllers/test_processes_controller.rb +++ b/test/controllers/test_processes_controller.rb @@ -5,17 +5,15 @@ class Litestream::TestProcessesController < ActionDispatch::IntegrationTest stubbed_process = {pid: "12345", status: "sleeping", started: DateTime.now} stubbed_databases = [ {"path" => "[ROOT]/storage/test.sqlite3", - "replicas" => "s3", - "generations" => [ - {"generation" => SecureRandom.hex, - "name" => "s3", - "lag" => "23h59m59s", - "start" => "2024-05-02T11:32:16Z", - "end" => "2024-05-02T11:33:10Z", - "snapshots" => [ - {"index" => "0", "size" => "4145735", "created" => "2024-05-02T11:32:16Z"} - ]} - ]} + "replica" => "s3", + "status" => {"status" => "ok", "local_txid" => "000000000000000a", "wal_size" => "128 kB"}, + "ltx" => [ + {"level" => 9, "min_txid" => "0000000000000001", "max_txid" => "0000000000000009", "size" => 4_145_735, "timestamp" => "2026-09-08T03:16:43Z"} + ], + "levels" => {9 => 1}, + "snapshot" => {"level" => 9, "max_txid" => "0000000000000009", "size" => 4_145_735, "timestamp" => "2026-09-08T03:16:43Z"}, + "latest" => {"level" => 9, "max_txid" => "0000000000000009"}, + "lag_txids" => 1} ] Litestream.stub :replicate_process, stubbed_process do Litestream.stub :databases, stubbed_databases do @@ -30,7 +28,11 @@ class Litestream::TestProcessesController < ActionDispatch::IntegrationTest assert_select "#databases li", 1 do assert_select "h2 code", stubbed_databases[0]["path"] - assert_select "details##{stubbed_databases[0]["generations"][0]["generation"]}" + assert_select "p", /Status:.*ok/ + assert_select "details#ltx" + assert_select "tbody tr", 1 + assert_select "td", "9" + assert_select "code", /0000000000000001.*0000000000000009/ end end end diff --git a/test/dummy/config/litestream.yml b/test/dummy/config/litestream.yml index 53280aa..79194f7 100644 --- a/test/dummy/config/litestream.yml +++ b/test/dummy/config/litestream.yml @@ -9,12 +9,16 @@ # `replicate` command. # # For more details, see: https://litestream.io/reference/config/ +snapshot: + interval: 24h + retention: 24h + dbs: - path: storage/test.sqlite3 - replicas: - - type: s3 - bucket: $LITESTREAM_REPLICA_BUCKET - path: test - endpoint: http://localhost:9000 - access-key-id: $LITESTREAM_ACCESS_KEY_ID - secret-access-key: $LITESTREAM_SECRET_ACCESS_KEY + replica: + type: s3 + bucket: $LITESTREAM_REPLICA_BUCKET + path: test + endpoint: http://localhost:9000 + access-key-id: $LITESTREAM_ACCESS_KEY_ID + secret-access-key: $LITESTREAM_SECRET_ACCESS_KEY diff --git a/test/generators/test_install.rb b/test/generators/test_install.rb index ad7cda9..6805f0c 100644 --- a/test/generators/test_install.rb +++ b/test/generators/test_install.rb @@ -22,6 +22,10 @@ def after_teardown assert_match "- path: storage/test.sqlite3", content assert_match "- path: storage/queue.sqlite3", content assert_match "- path: storage/errors.sqlite3", content + assert_match "snapshot:", content + assert_match "interval: 24h", content + assert_match "retention: 24h", content + assert_match "replica:", content assert_match "bucket: $LITESTREAM_REPLICA_BUCKET", content assert_match "access-key-id: $LITESTREAM_ACCESS_KEY_ID", content assert_match "secret-access-key: $LITESTREAM_SECRET_ACCESS_KEY", content diff --git a/test/litestream/test_commands.rb b/test/litestream/test_commands.rb index 9b96224..3f1c0ba 100644 --- a/test/litestream/test_commands.rb +++ b/test/litestream/test_commands.rb @@ -441,75 +441,82 @@ def test_databases_read_from_custom_configured_litestream_config_path end end - class TestGenerationsCommand < TestCommands - def test_generations_with_no_options + class TestLtxCommand < TestCommands + def test_ltx_with_no_options stub = proc do |cmd| executable, command, *argv = cmd assert_match Regexp.new("exe/test/litestream"), executable - assert_equal "generations", command + assert_equal "ltx", command assert_equal 3, argv.size assert_equal "--config", argv[0] assert_match Regexp.new("dummy/config/litestream.yml"), argv[1] assert_equal "db/test.sqlite3", argv[2] end Litestream::Commands.stub :run, stub do - Litestream::Commands.generations("db/test.sqlite3") + Litestream::Commands.ltx("db/test.sqlite3") end end - def test_generations_with_boolean_option + def test_ltx_with_boolean_option stub = proc do |cmd| executable, command, *argv = cmd assert_match Regexp.new("exe/test/litestream"), executable - assert_equal "generations", command + assert_equal "ltx", command assert_equal 4, argv.size assert_equal "--config", argv[0] assert_match Regexp.new("dummy/config/litestream.yml"), argv[1] - assert_equal "--if-db-not-exists", argv[2] + assert_equal "--level", argv[2] assert_equal "db/test.sqlite3", argv[3] end Litestream::Commands.stub :run, stub do - Litestream::Commands.generations("db/test.sqlite3", "--if-db-not-exists" => nil) + Litestream::Commands.ltx("db/test.sqlite3", "--level" => nil) end end - def test_generations_with_string_option + def test_ltx_with_string_option stub = proc do |cmd| executable, command, *argv = cmd assert_match Regexp.new("exe/test/litestream"), executable - assert_equal "generations", command + assert_equal "ltx", command assert_equal 5, argv.size assert_equal "--config", argv[0] assert_match Regexp.new("dummy/config/litestream.yml"), argv[1] - assert_equal "--parallelism", argv[2] - assert_equal 10, argv[3] + assert_equal "--level", argv[2] + assert_equal "all", argv[3] assert_equal "db/test.sqlite3", argv[4] end Litestream::Commands.stub :run, stub do - Litestream::Commands.generations("db/test.sqlite3", "--parallelism" => 10) + Litestream::Commands.ltx("db/test.sqlite3", "--level" => "all") end end - def test_generations_with_config_option + def test_ltx_with_config_option stub = proc do |cmd| executable, command, *argv = cmd assert_match Regexp.new("exe/test/litestream"), executable - assert_equal "generations", command + assert_equal "ltx", command assert_equal 3, argv.size assert_equal "--config", argv[0] assert_equal "CONFIG", argv[1] assert_equal "db/test.sqlite3", argv[2] end Litestream::Commands.stub :run, stub do - Litestream::Commands.generations("db/test.sqlite3", "--config" => "CONFIG") + Litestream::Commands.ltx("db/test.sqlite3", "--config" => "CONFIG") + end + end + + def test_ltx_requires_database + error = assert_raises Litestream::Commands::DatabaseRequiredException do + Litestream::Commands.ltx(nil) end + assert_match "database argument is required for ltx command", error.message end - def test_generations_sets_replica_bucket_env_var_from_config_when_env_var_not_set + def test_ltx_sets_replica_bucket_env_var_from_config_when_env_var_not_set Litestream.replica_bucket = "mybkt" Litestream::Commands.stub :run, nil do - Litestream::Commands.generations("db/test.sqlite3") + Litestream::Commands.ltx("db/test.sqlite3") end assert_equal "mybkt", ENV["LITESTREAM_REPLICA_BUCKET"] @@ -517,11 +524,11 @@ def test_generations_sets_replica_bucket_env_var_from_config_when_env_var_not_se assert_nil ENV["LITESTREAM_SECRET_ACCESS_KEY"] end - def test_generations_sets_replica_key_id_env_var_from_config_when_env_var_not_set + def test_ltx_sets_replica_key_id_env_var_from_config_when_env_var_not_set Litestream.replica_key_id = "mykey" Litestream::Commands.stub :run, nil do - Litestream::Commands.generations("db/test.sqlite3") + Litestream::Commands.ltx("db/test.sqlite3") end assert_nil ENV["LITESTREAM_REPLICA_BUCKET"] @@ -529,11 +536,11 @@ def test_generations_sets_replica_key_id_env_var_from_config_when_env_var_not_se assert_nil ENV["LITESTREAM_SECRET_ACCESS_KEY"] end - def test_generations_sets_replica_access_key_env_var_from_config_when_env_var_not_set + def test_ltx_sets_replica_access_key_env_var_from_config_when_env_var_not_set Litestream.replica_access_key = "access" Litestream::Commands.stub :run, nil do - Litestream::Commands.generations("db/test.sqlite3") + Litestream::Commands.ltx("db/test.sqlite3") end assert_nil ENV["LITESTREAM_REPLICA_BUCKET"] @@ -541,13 +548,13 @@ def test_generations_sets_replica_access_key_env_var_from_config_when_env_var_no assert_equal "access", ENV["LITESTREAM_SECRET_ACCESS_KEY"] end - def test_generations_sets_all_env_vars_from_config_when_env_vars_not_set + def test_ltx_sets_all_env_vars_from_config_when_env_vars_not_set Litestream.replica_bucket = "mybkt" Litestream.replica_key_id = "mykey" Litestream.replica_access_key = "access" Litestream::Commands.stub :run, nil do - Litestream::Commands.generations("db/test.sqlite3") + Litestream::Commands.ltx("db/test.sqlite3") end assert_equal "mybkt", ENV["LITESTREAM_REPLICA_BUCKET"] @@ -555,7 +562,7 @@ def test_generations_sets_all_env_vars_from_config_when_env_vars_not_set assert_equal "access", ENV["LITESTREAM_SECRET_ACCESS_KEY"] end - def test_generations_does_not_set_env_var_from_config_when_env_vars_already_set + def test_ltx_does_not_set_env_var_from_config_when_env_vars_already_set ENV["LITESTREAM_REPLICA_BUCKET"] = "original_bkt" ENV["LITESTREAM_ACCESS_KEY_ID"] = "original_key" ENV["LITESTREAM_SECRET_ACCESS_KEY"] = "original_access" @@ -565,217 +572,134 @@ def test_generations_does_not_set_env_var_from_config_when_env_vars_already_set Litestream.replica_access_key = "access" Litestream::Commands.stub :run, nil do - Litestream::Commands.generations("db/test.sqlite3") + Litestream::Commands.ltx("db/test.sqlite3") end assert_equal "original_bkt", ENV["LITESTREAM_REPLICA_BUCKET"] assert_equal "original_key", ENV["LITESTREAM_ACCESS_KEY_ID"] assert_equal "original_access", ENV["LITESTREAM_SECRET_ACCESS_KEY"] end - end - class TestSnapshotsCommand < TestCommands - def test_snapshots_with_no_options + def test_json_convenience_adds_json_flag stub = proc do |cmd| - executable, command, *argv = cmd - assert_match Regexp.new("exe/test/litestream"), executable - assert_equal "snapshots", command - assert_equal 3, argv.size - assert_equal "--config", argv[0] - assert_match Regexp.new("dummy/config/litestream.yml"), argv[1] - assert_equal "db/test.sqlite3", argv[2] + assert_includes cmd, "-json" + refute_includes cmd, "json" end Litestream::Commands.stub :run, stub do - Litestream::Commands.snapshots("db/test.sqlite3") + Litestream::Commands.ltx("db/test.sqlite3", json: true) end end - def test_snapshots_with_boolean_option - stub = proc do |cmd| - executable, command, *argv = cmd - assert_match Regexp.new("exe/test/litestream"), executable - assert_equal "snapshots", command - assert_equal 4, argv.size - assert_equal "--config", argv[0] - assert_match Regexp.new("dummy/config/litestream.yml"), argv[1] - assert_equal "--if-db-not-exists", argv[2] - assert_equal "db/test.sqlite3", argv[3] + def test_run_parses_json_output + stdout = '[{"level":0,"min_txid":"0000000000000001"}]' + Litestream::Commands.stub :`, stdout do + result = Litestream::Commands.send(:run, ["litestream", "ltx", "-json"], tabled_output: true) + assert_equal [{"level" => 0, "min_txid" => "0000000000000001"}], result end - Litestream::Commands.stub :run, stub do - Litestream::Commands.snapshots("db/test.sqlite3", "--if-db-not-exists" => nil) + end + + def test_run_parses_json_output_with_double_dash_flag + stdout = '[{"level":9,"max_txid":"000000000000000a"}]' + Litestream::Commands.stub :`, stdout do + result = Litestream::Commands.send(:run, ["litestream", "ltx", "--json"], tabled_output: true) + assert_equal 9, result.first["level"] end end - def test_snapshots_with_string_option + def test_json_false_does_not_add_json_flag stub = proc do |cmd| - executable, command, *argv = cmd - assert_match Regexp.new("exe/test/litestream"), executable - assert_equal "snapshots", command - assert_equal 5, argv.size - assert_equal "--config", argv[0] - assert_match Regexp.new("dummy/config/litestream.yml"), argv[1] - assert_equal "--parallelism", argv[2] - assert_equal 10, argv[3] - assert_equal "db/test.sqlite3", argv[4] + refute_includes cmd, "-json" + refute_includes cmd, "--json" end Litestream::Commands.stub :run, stub do - Litestream::Commands.snapshots("db/test.sqlite3", "--parallelism" => 10) + Litestream::Commands.ltx("db/test.sqlite3", json: false) end end + end - def test_snapshots_with_config_option + class TestStatusCommand < TestCommands + def test_status_with_no_database stub = proc do |cmd| executable, command, *argv = cmd assert_match Regexp.new("exe/test/litestream"), executable - assert_equal "snapshots", command - assert_equal 3, argv.size + assert_equal "status", command + assert_equal 2, argv.size assert_equal "--config", argv[0] - assert_equal "CONFIG", argv[1] - assert_equal "db/test.sqlite3", argv[2] + assert_match Regexp.new("dummy/config/litestream.yml"), argv[1] end Litestream::Commands.stub :run, stub do - Litestream::Commands.snapshots("db/test.sqlite3", "--config" => "CONFIG") - end - end - - def test_snapshots_sets_replica_bucket_env_var_from_config_when_env_var_not_set - Litestream.replica_bucket = "mybkt" - - Litestream::Commands.stub :run, nil do - Litestream::Commands.snapshots("db/test.sqlite3") - end - - assert_equal "mybkt", ENV["LITESTREAM_REPLICA_BUCKET"] - assert_nil ENV["LITESTREAM_ACCESS_KEY_ID"] - assert_nil ENV["LITESTREAM_SECRET_ACCESS_KEY"] - end - - def test_snapshots_sets_replica_key_id_env_var_from_config_when_env_var_not_set - Litestream.replica_key_id = "mykey" - - Litestream::Commands.stub :run, nil do - Litestream::Commands.snapshots("db/test.sqlite3") - end - - assert_nil ENV["LITESTREAM_REPLICA_BUCKET"] - assert_equal "mykey", ENV["LITESTREAM_ACCESS_KEY_ID"] - assert_nil ENV["LITESTREAM_SECRET_ACCESS_KEY"] - end - - def test_snapshots_sets_replica_access_key_env_var_from_config_when_env_var_not_set - Litestream.replica_access_key = "access" - - Litestream::Commands.stub :run, nil do - Litestream::Commands.snapshots("db/test.sqlite3") - end - - assert_nil ENV["LITESTREAM_REPLICA_BUCKET"] - assert_nil ENV["LITESTREAM_ACCESS_KEY_ID"] - assert_equal "access", ENV["LITESTREAM_SECRET_ACCESS_KEY"] - end - - def test_snapshots_sets_all_env_vars_from_config_when_env_vars_not_set - Litestream.replica_bucket = "mybkt" - Litestream.replica_key_id = "mykey" - Litestream.replica_access_key = "access" - - Litestream::Commands.stub :run, nil do - Litestream::Commands.snapshots("db/test.sqlite3") - end - - assert_equal "mybkt", ENV["LITESTREAM_REPLICA_BUCKET"] - assert_equal "mykey", ENV["LITESTREAM_ACCESS_KEY_ID"] - assert_equal "access", ENV["LITESTREAM_SECRET_ACCESS_KEY"] - end - - def test_snapshots_does_not_set_env_var_from_config_when_env_vars_already_set - ENV["LITESTREAM_REPLICA_BUCKET"] = "original_bkt" - ENV["LITESTREAM_ACCESS_KEY_ID"] = "original_key" - ENV["LITESTREAM_SECRET_ACCESS_KEY"] = "original_access" - - Litestream.replica_bucket = "mybkt" - Litestream.replica_key_id = "mykey" - Litestream.replica_access_key = "access" - - Litestream::Commands.stub :run, nil do - Litestream::Commands.snapshots("db/test.sqlite3") + Litestream::Commands.status end - - assert_equal "original_bkt", ENV["LITESTREAM_REPLICA_BUCKET"] - assert_equal "original_key", ENV["LITESTREAM_ACCESS_KEY_ID"] - assert_equal "original_access", ENV["LITESTREAM_SECRET_ACCESS_KEY"] end - end - class TestWalCommand < TestCommands - def test_wal_with_no_options + def test_status_with_database stub = proc do |cmd| executable, command, *argv = cmd assert_match Regexp.new("exe/test/litestream"), executable - assert_equal "wal", command + assert_equal "status", command assert_equal 3, argv.size assert_equal "--config", argv[0] assert_match Regexp.new("dummy/config/litestream.yml"), argv[1] assert_equal "db/test.sqlite3", argv[2] end Litestream::Commands.stub :run, stub do - Litestream::Commands.wal("db/test.sqlite3") + Litestream::Commands.status("db/test.sqlite3") end end - def test_wal_with_boolean_option + def test_status_with_boolean_option stub = proc do |cmd| executable, command, *argv = cmd assert_match Regexp.new("exe/test/litestream"), executable - assert_equal "wal", command + assert_equal "status", command assert_equal 4, argv.size assert_equal "--config", argv[0] assert_match Regexp.new("dummy/config/litestream.yml"), argv[1] - assert_equal "--if-db-not-exists", argv[2] + assert_equal "--no-expand-env", argv[2] assert_equal "db/test.sqlite3", argv[3] end Litestream::Commands.stub :run, stub do - Litestream::Commands.wal("db/test.sqlite3", "--if-db-not-exists" => nil) + Litestream::Commands.status("db/test.sqlite3", "--no-expand-env" => nil) end end - def test_wal_with_string_option + def test_status_with_string_option stub = proc do |cmd| executable, command, *argv = cmd assert_match Regexp.new("exe/test/litestream"), executable - assert_equal "wal", command + assert_equal "status", command assert_equal 5, argv.size assert_equal "--config", argv[0] assert_match Regexp.new("dummy/config/litestream.yml"), argv[1] - assert_equal "--parallelism", argv[2] - assert_equal 10, argv[3] + assert_equal "--log-level", argv[2] + assert_equal "debug", argv[3] assert_equal "db/test.sqlite3", argv[4] end Litestream::Commands.stub :run, stub do - Litestream::Commands.wal("db/test.sqlite3", "--parallelism" => 10) + Litestream::Commands.status("db/test.sqlite3", "--log-level" => "debug") end end - def test_wal_with_config_option + def test_status_with_config_option stub = proc do |cmd| executable, command, *argv = cmd assert_match Regexp.new("exe/test/litestream"), executable - assert_equal "wal", command + assert_equal "status", command assert_equal 3, argv.size assert_equal "--config", argv[0] assert_equal "CONFIG", argv[1] assert_equal "db/test.sqlite3", argv[2] end Litestream::Commands.stub :run, stub do - Litestream::Commands.wal("db/test.sqlite3", "--config" => "CONFIG") + Litestream::Commands.status("db/test.sqlite3", "--config" => "CONFIG") end end - def test_wal_sets_replica_bucket_env_var_from_config_when_env_var_not_set + def test_status_sets_replica_bucket_env_var_from_config_when_env_var_not_set Litestream.replica_bucket = "mybkt" Litestream::Commands.stub :run, nil do - Litestream::Commands.wal("db/test.sqlite3") + Litestream::Commands.status("db/test.sqlite3") end assert_equal "mybkt", ENV["LITESTREAM_REPLICA_BUCKET"] @@ -783,11 +707,11 @@ def test_wal_sets_replica_bucket_env_var_from_config_when_env_var_not_set assert_nil ENV["LITESTREAM_SECRET_ACCESS_KEY"] end - def test_wal_sets_replica_key_id_env_var_from_config_when_env_var_not_set + def test_status_sets_replica_key_id_env_var_from_config_when_env_var_not_set Litestream.replica_key_id = "mykey" Litestream::Commands.stub :run, nil do - Litestream::Commands.wal("db/test.sqlite3") + Litestream::Commands.status("db/test.sqlite3") end assert_nil ENV["LITESTREAM_REPLICA_BUCKET"] @@ -795,11 +719,11 @@ def test_wal_sets_replica_key_id_env_var_from_config_when_env_var_not_set assert_nil ENV["LITESTREAM_SECRET_ACCESS_KEY"] end - def test_wal_sets_replica_access_key_env_var_from_config_when_env_var_not_set + def test_status_sets_replica_access_key_env_var_from_config_when_env_var_not_set Litestream.replica_access_key = "access" Litestream::Commands.stub :run, nil do - Litestream::Commands.wal("db/test.sqlite3") + Litestream::Commands.status("db/test.sqlite3") end assert_nil ENV["LITESTREAM_REPLICA_BUCKET"] @@ -807,13 +731,13 @@ def test_wal_sets_replica_access_key_env_var_from_config_when_env_var_not_set assert_equal "access", ENV["LITESTREAM_SECRET_ACCESS_KEY"] end - def test_wal_sets_all_env_vars_from_config_when_env_vars_not_set + def test_status_sets_all_env_vars_from_config_when_env_vars_not_set Litestream.replica_bucket = "mybkt" Litestream.replica_key_id = "mykey" Litestream.replica_access_key = "access" Litestream::Commands.stub :run, nil do - Litestream::Commands.wal("db/test.sqlite3") + Litestream::Commands.status("db/test.sqlite3") end assert_equal "mybkt", ENV["LITESTREAM_REPLICA_BUCKET"] @@ -821,7 +745,7 @@ def test_wal_sets_all_env_vars_from_config_when_env_vars_not_set assert_equal "access", ENV["LITESTREAM_SECRET_ACCESS_KEY"] end - def test_wal_does_not_set_env_var_from_config_when_env_vars_already_set + def test_status_does_not_set_env_var_from_config_when_env_vars_already_set ENV["LITESTREAM_REPLICA_BUCKET"] = "original_bkt" ENV["LITESTREAM_ACCESS_KEY_ID"] = "original_key" ENV["LITESTREAM_SECRET_ACCESS_KEY"] = "original_access" @@ -831,20 +755,47 @@ def test_wal_does_not_set_env_var_from_config_when_env_vars_already_set Litestream.replica_access_key = "access" Litestream::Commands.stub :run, nil do - Litestream::Commands.wal("db/test.sqlite3") + Litestream::Commands.status("db/test.sqlite3") end assert_equal "original_bkt", ENV["LITESTREAM_REPLICA_BUCKET"] assert_equal "original_key", ENV["LITESTREAM_ACCESS_KEY_ID"] assert_equal "original_access", ENV["LITESTREAM_SECRET_ACCESS_KEY"] end + + def test_status_json_convenience_adds_json_flag + stub = proc do |cmd| + assert_includes cmd, "-json" + end + Litestream::Commands.stub :run, stub do + Litestream::Commands.status(json: true) + end + end + + def test_run_parses_json_hash_output + stdout = '{"db_path":"db/test.sqlite3","txid":"000000000000000a"}' + Litestream::Commands.stub :`, stdout do + result = Litestream::Commands.send(:run, ["litestream", "restore", "-json"], tabled_output: false) + assert_equal "000000000000000a", result["txid"] + end + end + + def test_status_all_databases_with_json + stub = proc do |cmd| + assert_equal "status", cmd[1] + assert_equal "-json", cmd.last + end + Litestream::Commands.stub :run, stub do + Litestream::Commands.status(json: true) + end + end end class TestOutput < ActiveSupport::TestCase def test_output_formatting_generates_table_with_data data = [ - {path: "/storage/database.db", replicas: "s3"}, - {path: "/storage/another-database.db", replicas: "s3"} + {"path" => "/storage/database.db", "replica" => "s3"}, + {"path" => "/storage/another-database.db", "replica" => "s3"} ] result = Litestream::Commands::Output.format(data) @@ -853,23 +804,23 @@ def test_output_formatting_generates_table_with_data assert_equal 3, lines.length assert_includes lines[0], "path" - assert_includes lines[0], "replicas" + assert_includes lines[0], "replica" assert_includes lines[1], "/storage/database.db" assert_includes lines[2], "/storage/another-database.db" end def test_output_formatting_generates_formatted_table data = [ - {path: "/storage/database.db", replicas: "s3"}, - {path: "/storage/another-database.db", replicas: "s3"} + {path: "/storage/database.db", replica: "s3"}, + {path: "/storage/another-database.db", replica: "s3"} ] result = Litestream::Commands::Output.format(data) lines = result.split("\n") - replicas_pos = lines[0].index("replicas") - assert_equal replicas_pos, lines[1].index("s3") - assert_equal replicas_pos, lines[2].index("s3") + replica_pos = lines[0].index("replica") + assert_equal replica_pos, lines[1].index("s3") + assert_equal replica_pos, lines[2].index("s3") end end end diff --git a/test/tasks/test_litestream_tasks.rb b/test/tasks/test_litestream_tasks.rb index 344c3ec..27e3327 100644 --- a/test/tasks/test_litestream_tasks.rb +++ b/test/tasks/test_litestream_tasks.rb @@ -9,9 +9,8 @@ def setup Rake::Task["litestream:replicate"].reenable Rake::Task["litestream:restore"].reenable Rake::Task["litestream:databases"].reenable - Rake::Task["litestream:generations"].reenable - Rake::Task["litestream:snapshots"].reenable - Rake::Task["litestream:wal"].reenable + Rake::Task["litestream:ltx"].reenable + Rake::Task["litestream:status"].reenable end def teardown @@ -138,127 +137,127 @@ def test_databases_task_with_arguments_without_separator end end - class TestGenerationsTask < TestLitestreamTasks - def test_generations_task_with_only_database_using_single_dash + class TestLtxTask < TestLitestreamTasks + def test_ltx_task_with_only_database_using_single_dash ARGV.replace ["--", "-database=db/test.sqlite3"] fake = Minitest::Mock.new fake.expect :call, nil, ["db/test.sqlite3"] - Litestream::Commands.stub :generations, fake do - Rake.application.invoke_task "litestream:generations" + Litestream::Commands.stub :ltx, fake do + Rake.application.invoke_task "litestream:ltx" end fake.verify end - def test_generations_task_with_only_database_using_double_dash + def test_ltx_task_with_only_database_using_double_dash ARGV.replace ["--", "--database=db/test.sqlite3"] fake = Minitest::Mock.new fake.expect :call, nil, ["db/test.sqlite3"] - Litestream::Commands.stub :generations, fake do - Rake.application.invoke_task "litestream:generations" + Litestream::Commands.stub :ltx, fake do + Rake.application.invoke_task "litestream:ltx" end fake.verify end - def test_generations_task_with_arguments - ARGV.replace ["--", "-database=db/test.sqlite3", "--if-db-not-exists"] + def test_ltx_task_with_arguments + ARGV.replace ["--", "-database=db/test.sqlite3", "--level=all"] fake = Minitest::Mock.new - fake.expect :call, nil, ["db/test.sqlite3"], "--if-db-not-exists": nil - Litestream::Commands.stub :generations, fake do - Rake.application.invoke_task "litestream:generations" + fake.expect :call, nil, ["db/test.sqlite3"], "--level": "all" + Litestream::Commands.stub :ltx, fake do + Rake.application.invoke_task "litestream:ltx" end fake.verify end - def test_generations_task_with_arguments_without_separator + def test_ltx_task_with_arguments_without_separator ARGV.replace ["-database=db/test.sqlite3"] fake = Minitest::Mock.new fake.expect :call, nil, [nil] - Litestream::Commands.stub :generations, fake do - Rake.application.invoke_task "litestream:generations" + Litestream::Commands.stub :ltx, fake do + Rake.application.invoke_task "litestream:ltx" end fake.verify end end - class TestSnapshotsTask < TestLitestreamTasks - def test_snapshots_task_with_only_database_using_single_dash + class TestStatusTask < TestLitestreamTasks + def test_status_task_with_only_database_using_single_dash ARGV.replace ["--", "-database=db/test.sqlite3"] fake = Minitest::Mock.new fake.expect :call, nil, ["db/test.sqlite3"] - Litestream::Commands.stub :snapshots, fake do - Rake.application.invoke_task "litestream:snapshots" + Litestream::Commands.stub :status, fake do + Rake.application.invoke_task "litestream:status" end fake.verify end - def test_snapshots_task_with_only_database_using_double_dash + def test_status_task_with_only_database_using_double_dash ARGV.replace ["--", "--database=db/test.sqlite3"] fake = Minitest::Mock.new fake.expect :call, nil, ["db/test.sqlite3"] - Litestream::Commands.stub :snapshots, fake do - Rake.application.invoke_task "litestream:snapshots" + Litestream::Commands.stub :status, fake do + Rake.application.invoke_task "litestream:status" end fake.verify end - def test_snapshots_task_with_arguments - ARGV.replace ["--", "-database=db/test.sqlite3", "--if-db-not-exists"] + def test_status_task_with_arguments + ARGV.replace ["--", "-database=db/test.sqlite3", "--json"] fake = Minitest::Mock.new - fake.expect :call, nil, ["db/test.sqlite3"], "--if-db-not-exists": nil - Litestream::Commands.stub :snapshots, fake do - Rake.application.invoke_task "litestream:snapshots" + fake.expect :call, nil, ["db/test.sqlite3"], "--json": nil + Litestream::Commands.stub :status, fake do + Rake.application.invoke_task "litestream:status" end fake.verify end - def test_snapshots_task_with_arguments_without_separator + def test_status_task_with_arguments_without_separator ARGV.replace ["-database=db/test.sqlite3"] fake = Minitest::Mock.new fake.expect :call, nil, [nil] - Litestream::Commands.stub :snapshots, fake do - Rake.application.invoke_task "litestream:snapshots" + Litestream::Commands.stub :status, fake do + Rake.application.invoke_task "litestream:status" end fake.verify end end - class TestWalTask < TestLitestreamTasks - def test_wal_task_with_only_database_using_single_dash - ARGV.replace ["--", "-database=db/test.sqlite3"] + class TestStatusWithoutDatabaseTask < TestLitestreamTasks + def test_status_task_with_no_arguments + ARGV.replace ["--"] fake = Minitest::Mock.new - fake.expect :call, nil, ["db/test.sqlite3"] - Litestream::Commands.stub :wal, fake do - Rake.application.invoke_task "litestream:wal" + fake.expect :call, nil, [nil] + Litestream::Commands.stub :status, fake do + Rake.application.invoke_task "litestream:status" end fake.verify end - def test_wal_task_with_only_database_using_double_dash - ARGV.replace ["--", "--database=db/test.sqlite3"] + def test_status_task_with_boolean_option_and_no_database + ARGV.replace ["--", "--json"] fake = Minitest::Mock.new - fake.expect :call, nil, ["db/test.sqlite3"] - Litestream::Commands.stub :wal, fake do - Rake.application.invoke_task "litestream:wal" + fake.expect :call, nil, [nil], "--json": nil + Litestream::Commands.stub :status, fake do + Rake.application.invoke_task "litestream:status" end fake.verify end - def test_wal_task_with_arguments - ARGV.replace ["--", "-database=db/test.sqlite3", "--if-db-not-exists"] + def test_status_task_with_config_and_no_database + ARGV.replace ["--", "--config=config/litestream.yml"] fake = Minitest::Mock.new - fake.expect :call, nil, ["db/test.sqlite3"], "--if-db-not-exists": nil - Litestream::Commands.stub :wal, fake do - Rake.application.invoke_task "litestream:wal" + fake.expect :call, nil, [nil], "--config": "config/litestream.yml" + Litestream::Commands.stub :status, fake do + Rake.application.invoke_task "litestream:status" end fake.verify end - def test_wal_task_with_arguments_without_separator - ARGV.replace ["-database=db/test.sqlite3"] + def test_status_task_ignores_arguments_without_separator + ARGV.replace ["--json"] fake = Minitest::Mock.new fake.expect :call, nil, [nil] - Litestream::Commands.stub :wal, fake do - Rake.application.invoke_task "litestream:wal" + Litestream::Commands.stub :status, fake do + Rake.application.invoke_task "litestream:status" end fake.verify end diff --git a/test/test_litestream.rb b/test/test_litestream.rb index 5c1f41d..291ea58 100644 --- a/test/test_litestream.rb +++ b/test/test_litestream.rb @@ -23,6 +23,7 @@ def test_replicate_process_systemd " └─1179656 /usr/bin/litestream replicate", "", "Warning: some journal files were not opened due to insufficient permissions."].join("\n") + system("true") Litestream.stub :`, stubbed_status do info = Litestream.replicate_process @@ -46,6 +47,7 @@ def test_replicate_process_systemd_custom_command "Warning: some journal files were not opened due to insufficient permissions."].join("\n") Litestream.systemctl_command = "systemctl --user status myapp-litestream.service" + system("true") Litestream.stub :`, stubbed_status do info = Litestream.replicate_process @@ -77,6 +79,7 @@ def test_replicate_process_ps end end + system("true") Litestream.stub :`, stubbed_backticks do info = Litestream.replicate_process @@ -85,4 +88,39 @@ def test_replicate_process_ps assert_equal info[:started].class, DateTime end end + + def test_databases_derives_replication_summary_and_isolates_errors + database_path = Rails.root.join("storage/test.sqlite3").to_s + failing_path = Rails.root.join("storage/failing.sqlite3").to_s + databases = [ + {"path" => database_path, "replica" => "file"}, + {"path" => failing_path, "replica" => "file"} + ] + status = [{"database" => database_path, "status" => "ok", "local_txid" => "000000000000000a", "wal_size" => "128 kB"}] + ltx = [ + {"level" => 0, "min_txid" => "0000000000000008", "max_txid" => "0000000000000008", "size" => 100, "timestamp" => "2026-09-08T01:00:00Z"}, + {"level" => 0, "min_txid" => "0000000000000009", "max_txid" => "0000000000000009", "size" => 110, "timestamp" => "2026-09-08T02:00:00Z"}, + {"level" => 9, "min_txid" => "0000000000000001", "max_txid" => "0000000000000007", "size" => 1_013, "timestamp" => "2026-09-08T03:00:00Z"} + ] + + status_stub = proc { |path, **| (path == database_path) ? status : raise("status unavailable") } + ltx_stub = proc { |path, **| (path == database_path) ? ltx : flunk("ltx should not run after status fails") } + + Litestream::Commands.stub :databases, databases do + Litestream::Commands.stub :status, status_stub do + Litestream::Commands.stub :ltx, ltx_stub do + result = Litestream.databases + + assert_equal "[ROOT]/storage/test.sqlite3", result[0]["path"] + assert_equal status.first, result[0]["status"] + assert_equal({0 => 2, 9 => 1}, result[0]["levels"]) + assert_equal ltx[2], result[0]["snapshot"] + assert_equal ltx[1], result[0]["latest"] + assert_equal 1, result[0]["lag_txids"] + assert_equal "status unavailable", result[1]["error"] + assert_equal "[ROOT]/storage/failing.sqlite3", result[1]["path"] + end + end + end + end end