Skip to content

Update to Litestream 0.5.17 #78

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
cole-robertson wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Update to Litestream 0.5.17 #78

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
8a78baf
Run Litestream commands without a shell and surface failures
cole-robertson Sep 9, 2026
859616c
Check the exit status of the in-process replicate
cole-robertson Sep 14, 2026
f63ea85
Update to Litestream 0.5.17
cole-robertson Sep 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,14 @@
## [Unreleased]

- Run Litestream commands without a shell and raise on command failures and timeouts.
- Support parsed JSON output from Litestream 0.5 commands with `json: true`.
- Bundle Litestream 0.5.17 release archives and verify their SHA-256 checksums during packaging.
- Replace the removed `generations`, `snapshots`, and `wal` wrappers and rake tasks with `ltx` and `status`.
- Use daemon-free `databases`, `status`, and `ltx` calls for dashboard data.
- Show local status and LTX files in the dashboard.
- Generate Litestream 0.5 configuration with a single replica and global snapshot settings.
- Document restoration and migration from Litestream 0.3 backups.

## [0.14.0] - 2025-06-14

- Change async behaviour of replicate and other commands ([@hschne](https://github.com/fractaledmind/litestream-ruby/pull/62))
Expand Down
2 changes: 0 additions & 2 deletions Gemfile.lock
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -204,7 +204,6 @@ GEM
rubocop (>= 1.48.1, < 2.0)
rubocop-ast (>= 1.30.0, < 2.0)
ruby-progressbar (1.13.0)
rubyzip (2.3.2)
securerandom (0.4.1)
sqlite3 (2.6.0-arm64-darwin)
sqlite3 (2.6.0-x86_64-linux-gnu)
Expand Down Expand Up @@ -244,7 +243,6 @@ DEPENDENCIES
minitest (~> 5.0)
rails
rake (~> 13.0)
rubyzip
standard (~> 1.3)

BUNDLED WITH
Expand Down
126 changes: 60 additions & 66 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,16 +85,22 @@ The gem streamlines the configuration process by providing a default configurati
The default configuration file looks like this if you only have one SQLite database:

```yaml
snapshot:
interval: 24h
retention: 24h

dbs:
- path: storage/production.sqlite3
replicas:
- type: s3
path: storage/production.sqlite3
bucket: $LITESTREAM_REPLICA_BUCKET
access-key-id: $LITESTREAM_ACCESS_KEY_ID
secret-access-key: $LITESTREAM_SECRET_ACCESS_KEY
replica:
type: s3
path: storage/production.sqlite3
bucket: $LITESTREAM_REPLICA_BUCKET
access-key-id: $LITESTREAM_ACCESS_KEY_ID
secret-access-key: $LITESTREAM_SECRET_ACCESS_KEY
```

In Litestream 0.5, snapshot interval and retention are global settings rather than per-replica settings.

This is the default for Amazon S3. The full range of possible replica types (e.g. other S3-compatible object storage servers) are covered in Litestream's [replica guides](https://litestream.io/guides/#replica-guides).

The gem also provides a default initializer file at `config/initializers/litestream.rb` that allows you to configure various variables referenced in the configuration file in Ruby. By providing a Ruby interface to these environment variables, you can use your preferred method of storing secrets. For example, the default generated file uses Rails' encrypted credentials to store your secrets.
Expand Down Expand Up @@ -179,7 +185,7 @@ You can restore any replicated database at any point using the gem's provided `l
> [!NOTE]
> During the restoration process, you need to prevent any interaction with ActiveRecord/SQLite, such as from a running `rails server` or `rails console` instance. If there is any interaction, Rails might regenerate the production database and prevent restoration via litestream. If this happens, you might get a "cannot restore, output path already exists" error.

1. Rename the production (`production.sqlite3`, `production.sqlite3-shm`, and `production.sqlite3-wal`) databases (**recommended**) or alternatively delete. To delete the production databases locally, you can run the following at your own risk:
1. Rename the production database and its SQLite sidecar files (**recommended**) or alternatively delete them. To delete the production databases locally, you can run the following at your own risk:
```shell
# DANGEROUS OPERATION, consider renaming database files instead
bin/rails db:drop DISABLE_DATABASE_ENVIRONMENT_CHECK=1
Expand Down Expand Up @@ -215,26 +221,19 @@ You can forward arguments in whatever order you like, you simply need to ensure
-if-replica-exists
Returns exit code of 0 if no backups found.

-parallelism NUM
Determines the number of WAL files downloaded in parallel.
Defaults to 8

-replica NAME
Restore from a specific replica.
Defaults to replica with latest data.

-generation NAME
Restore from a specific generation.
Defaults to generation with latest data.

-index NUM
Restore up to a specific WAL index (inclusive).
Defaults to use the highest available index.
-txid TXID
Restore through a specific 16-character hexadecimal transaction ID.

-timestamp TIMESTAMP
Restore to a specific point-in-time.
Defaults to use the latest available backup.

-json
Print the restore result as JSON.

-dry-run
Print the restore plan without writing the database.

-config PATH
Specifies the configuration file.
Defaults to /etc/litestream.yml
Expand Down Expand Up @@ -387,79 +386,59 @@ bin/rails litestream:databases
This will return a list of databases and their configured replicas:

```
path replicas
path replica
/Users/you/Code/your-app/storage/production.sqlite3 s3
```

You can also list the generations of a specific database:
You can inspect local replication status for all databases or one specific database:

```shell
bin/rails litestream:generations -- --database=storage/production.sqlite3
bin/rails litestream:status -- --database=storage/production.sqlite3
```

This will list all generations for the specified database, including stats about their lag behind the primary database and the time range they cover:
This reads local state and does not require a running Litestream process:

```
name generation lag start end
s3 a295b16a796689f3 -156ms 2024-04-17T00:01:19Z 2024-04-17T00:01:19Z
database status local_txid wal_size
/Users/you/Code/your-app/storage/production.sqlite3 ok 000000000000000a 128 kB
```

You can list the snapshots available for a database:
You can list all remote LTX files, including the level-9 snapshot:

```shell
bin/rails litestream:snapshots -- --database=storage/production.sqlite3
bin/rails litestream:ltx -- --database=storage/production.sqlite3 --level=all
```

This command lists snapshots available for that specified database:
The command returns the compaction level, transaction range, byte size, and creation time:

```
replica generation index size created
s3 a295b16a796689f3 1 4645465 2024-04-17T00:01:19Z
```

Finally, you can list the wal files available for a database:

```shell
bin/rails litestream:wal -- --database=storage/production.sqlite3
```

This command lists wal files available for that specified database:

```
replica generation index offset size created
s3 a295b16a796689f3 1 0 2036 2024-04-17T00:01:19Z
level min_txid max_txid size created
0 0000000000000008 000000000000000a 1013 2026-09-08T03:16:43Z
```

### Running commands from Ruby

In addition to the provided rake tasks, you can also run Litestream commands directly from Ruby. The gem provides a `Litestream::Commands` module that wraps the Litestream CLI commands. This is particularly useful for the introspection commands, as you can use the output in your Ruby code.

The `Litestream::Commands.databases` method returns an array of hashes with the "path" and "replicas" keys for each database:
Pass `json: true` to return the 0.5 CLI's JSON values with string keys. The `Litestream::Commands.databases` method returns each database path and replica type:

```ruby
Litestream::Commands.databases
# => [{"path"=>"/Users/you/Code/your-app/storage/production.sqlite3", "replicas"=>"s3"}]
Litestream::Commands.databases(json: true)
# => [{"path"=>"/Users/you/Code/your-app/storage/production.sqlite3", "replica"=>"s3"}]
```

The `Litestream::Commands.generations` method returns an array of hashes with the "name", "generation", "lag", "start", and "end" keys for each generation:
The `status` method returns local state without connecting to the replica or requiring a daemon:

```ruby
Litestream::Commands.generations('storage/production.sqlite3')
# => [{"name"=>"s3", "generation"=>"5f4341bc3d22d615", "lag"=>"3s", "start"=>"2024-04-17T19:48:09Z", "end"=>"2024-04-17T19:48:09Z"}]
Litestream::Commands.status("storage/production.sqlite3", json: true)
# => [{"database"=>"storage/production.sqlite3", "status"=>"ok", "local_txid"=>"000000000000000a", "wal_size"=>"128 kB"}]
```

The `Litestream::Commands.snapshots` method returns an array of hashes with the "replica", "generation", "index", "size", and "created" keys for each snapshot:
The `ltx` method lists remote LTX files. An unreplicated database returns an empty array:

```ruby
Litestream::Commands.snapshots('storage/production.sqlite3')
# => [{"replica"=>"s3", "generation"=>"5f4341bc3d22d615", "index"=>"0", "size"=>"4645465", "created"=>"2024-04-17T19:48:09Z"}]
```

The `Litestream::Commands.wal` method returns an array of hashes with the "replica", "generation", "index", "offset","size", and "created" keys for each wal:

```ruby
Litestream::Commands.wal('storage/production.sqlite3')
# => [{"replica"=>"s3", "generation"=>"5f4341bc3d22d615", "index"=>"0", "offset"=>"0", "size"=>"2036", "created"=>"2024-04-17T19:48:09Z"}]
Litestream::Commands.ltx("storage/production.sqlite3", json: true, "--level" => "all")
# => [{"level"=>0, "min_txid"=>"0000000000000008", "max_txid"=>"000000000000000a", "size"=>1013, "timestamp"=>"2026-09-08T03:16:43Z"}]
```

You can also restore a database programmatically using the `Litestream::Commands.restore` method, which returns the path to the restored database:
Expand All @@ -479,14 +458,29 @@ The full set of commands available to the `litestream` executable are covered in

```shell
litestream databases [arguments]
litestream generations [arguments] DB_PATH|REPLICA_URL
litestream info [arguments]
litestream list [arguments]
litestream ltx [arguments] DB_PATH
litestream register [arguments]
litestream replicate [arguments]
litestream restore [arguments] DB_PATH|REPLICA_URL
litestream snapshots [arguments] DB_PATH|REPLICA_URL
litestream reset [arguments]
litestream restore [arguments] DB_PATH
litestream start [arguments]
litestream status [arguments] [DB_PATH]
litestream stop [arguments]
litestream sync [arguments]
litestream unregister [arguments]
litestream version
litestream wal [arguments] DB_PATH|REPLICA_URL
```

### Upgrading from 0.3

Litestream 0.5 changes each database from a `replicas:` list to a single `replica:` map and moves retention and snapshot settings into the root-level `snapshot:` block. Existing `replicas:` configuration files must be edited by hand before upgrading. The `databases` command and Ruby method now return a `"replica"` key instead of `"replicas"`.

The removed `generations`, `snapshots`, and `wal` command methods and rake tasks now raise an error pointing to `ltx`, which replaces all three forms of remote backup introspection.

The 0.5 `restore` command auto-detects both 0.3 and LTX backups and prefers whichever backup is newer. Existing 0.3 `generations/` objects are left in place; you can delete them after the LTX history covers your full retention window. To roll back, reinstall a 0.14.x release of this gem, which bundles Litestream 0.3.

### Using in development

By default, if you install the gem and configure via `puma.rb` or `Procfile`, Litestream will not start in development.
Expand Down
98 changes: 43 additions & 55 deletions app/views/litestream/processes/show.html.erb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,63 +56,51 @@
</div>

<br />
<section id="generations" class="ml-6">
<% database['generations'].each do |generation| %>
<details id="<%= generation['generation'] %>" open="open">
<summary class="cursor-pointer rounded p-2 hover:bg-gray-50 dark:hover:bg-gray-800">
<code><%= generation['generation'] %></code>
(<em><%= generation['lag'] %> lag</em>)
</summary>
<section class="ml-6">
<% if database['error'] %>
<p class="text-red-600 dark:text-red-400"><%= database['error'] %></p>
<% else %>
<dl class="grid grid-cols-[fit-content(100%)_1fr] gap-x-4">
<dt class="font-bold">Status</dt>
<dd><%= database.dig('status', 'status') %></dd>
<dt class="font-bold">Local TXID</dt>
<dd><code><%= database.dig('status', 'local_txid') %></code></dd>
<dt class="font-bold">WAL size</dt>
<dd><%= number_to_human_size database.dig('status', 'wal_size') %></dd>
</dl>

<dl class="ml-7 grid grid-cols-[fit-content(100%)_1fr] gap-x-4">
<dt class="font-bold">Start</dt>
<dd class="">
<abbr title="<%= generation['start'] %>" class="underline decoration-dashed decoration-gray-500 cursor-help">
<time datetime="<%= generation['start'] %>"><%= DateTime.parse(generation['start']).to_formatted_s(:db) %></time>
</abbr>
</dd>
<br />
<% if database['ltx'] == [] %>
<p>No LTX files yet</p>
<% else %>
<table class="min-w-full divide-y divide-gray-300">
<thead>
<tr>
<th scope="col" class="whitespace-nowrap px-2 py-2 text-right text-sm font-semibold text-gray-900 dark:text-gray-100">Level</th>
<th scope="col" class="whitespace-nowrap px-2 py-2 text-left text-sm font-semibold text-gray-900 dark:text-gray-100">Min TXID</th>
<th scope="col" class="whitespace-nowrap px-2 py-2 text-left text-sm font-semibold text-gray-900 dark:text-gray-100">Max TXID</th>
<th scope="col" class="whitespace-nowrap px-2 py-2 text-right text-sm font-semibold text-gray-900 dark:text-gray-100">Size</th>
<th scope="col" class="whitespace-nowrap px-2 py-2 text-left text-sm font-semibold text-gray-900 dark:text-gray-100">Created</th>
</tr>
</thead>

<dt class="font-bold">End</dt>
<dd class="">
<abbr title="<%= generation['end'] %>" class="underline decoration-dashed decoration-gray-500 cursor-help">
<time datetime="<%= generation['end'] %>"><%= DateTime.parse(generation['end']).to_formatted_s(:db) %></time>
</abbr>
</dd>

<div class="col-span-2">
<dt class="font-bold">Snapshots</dt>
<dd class="">
<table class="min-w-full divide-y divide-gray-300">
<thead>
<tr>
<th scope="col" class="whitespace-nowrap px-2 py-2 text-left text-sm font-semibold text-gray-900 dark:text-gray-100">Created at</th>
<th scope="col" class="whitespace-nowrap px-2 py-2 text-right text-sm font-semibold text-gray-900 dark:text-gray-100">Index</th>
<th scope="col" class="whitespace-nowrap px-2 py-2 text-right text-sm font-semibold text-gray-900 dark:text-gray-100">Size</th>
</tr>
</thead>

<tbody class="bg-white dark:bg-gray-900">
<% generation['snapshots'].each do |snapshot| %>
<tr class="align-top even:bg-gray-50 dark:even:bg-gray-800">
<td scope="col" class="whitespace-nowrap px-2 py-2 text-sm text-gray-900 dark:text-gray-100">
<abbr title="<%= snapshot['created'] %>" class="underline decoration-dashed decoration-gray-500 cursor-help">
<time datetime="<%= snapshot['created'] %>"><%= DateTime.parse(snapshot['created']).to_formatted_s(:db) %></time>
</abbr>
</td>
<td scope="col" class="whitespace-nowrap px-2 py-2 text-sm text-gray-900 dark:text-gray-100 text-right">
<%= snapshot['index'] %>
</td>
<td scope="col" class="whitespace-nowrap px-2 py-2 text-sm text-gray-900 dark:text-gray-100 text-right">
<%= number_to_human_size snapshot['size'] %>
</td>
</tr>
<% end %>
</tbody>
</table>
</dd>
</div>
</dl>
</details>
<tbody class="bg-white dark:bg-gray-900">
<% database['ltx'].each do |ltx| %>
<tr class="align-top even:bg-gray-50 dark:even:bg-gray-800">
<td class="whitespace-nowrap px-2 py-2 text-sm text-gray-900 dark:text-gray-100 text-right"><%= ltx['level'] %></td>
<td class="whitespace-nowrap px-2 py-2 text-sm text-gray-900 dark:text-gray-100"><code><%= ltx['min_txid'] %></code></td>
<td class="whitespace-nowrap px-2 py-2 text-sm text-gray-900 dark:text-gray-100"><code><%= ltx['max_txid'] %></code></td>
<td class="whitespace-nowrap px-2 py-2 text-sm text-gray-900 dark:text-gray-100 text-right"><%= number_to_human_size ltx['size'] %></td>
<td class="whitespace-nowrap px-2 py-2 text-sm text-gray-900 dark:text-gray-100">
<abbr title="<%= ltx['timestamp'] %>" class="underline decoration-dashed decoration-gray-500 cursor-help">
<time datetime="<%= ltx['timestamp'] %>"><%= DateTime.parse(ltx['timestamp']).to_formatted_s(:db) %></time>
</abbr>
</td>
</tr>
<% end %>
</tbody>
</table>
<% end %>
<% end %>
</section>
</li>
Expand Down
20 changes: 9 additions & 11 deletions lib/litestream.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -107,20 +107,18 @@ def replicate_process
end

def databases
databases = Commands.databases
databases = Commands.databases(json: true)

databases.each do |db|
generations = Commands.generations(db["path"])
snapshots = Commands.snapshots(db["path"])
db["path"] = db["path"].gsub(Rails.root.to_s, "[ROOT]")

db["generations"] = generations.map do |generation|
id = generation["generation"]
replica = generation["name"]
generation["snapshots"] = snapshots.select { |snapshot| snapshot["generation"] == id && snapshot["replica"] == replica }
.map { |s| s.slice("index", "size", "created") }
generation.slice("generation", "name", "lag", "start", "end", "snapshots")
path = db["path"]
begin
db["status"] = Commands.status(path, json: true).first
db["ltx"] = Commands.ltx(path, :json => true, "--level" => "all")
rescue Commands::CommandFailedException => error
db["error"] = error.message
end

db["path"] = path.gsub(Rails.root.to_s, "[ROOT]")
end
end

Expand Down
Loading