Litestream 0.5.17 caveats: deploy overlap, stalled compaction, and a connection leak (fixed upstream, unreleased) #84

Context for #81 / 0.16.0, from running Litestream 0.5.17 in production at Rebulk (Railwatch Cloud), where it caused an outage on 2026-09-16. Nothing here blocks 0.16.0; it's what we'd want gem users to know and what we'd suggest next.

What can go wrong on 0.5.17

1. Two replicate processes on one database. 0.5.17 takes no lock on the database it replicates. During a Kamal deploy the new container boots before the old one stops, so with the Puma plugin both containers run replicate against the same SQLite file on the shared volume until the old one is stopped. Both stage the next L0 through the same temp path. Usually nothing happens; one of our deploys left an L0 in S3 with a bad checksum. #77 ends the case where the daemon outlived Puma; this is the overlap that remains while old and new containers both run.

2. One corrupt L0 stalls compaction forever. Every L1 pass includes it and fails, and the L0 backlog grows with no visible error to the app (ours reached 63k objects / 20 GB). Related: benbjohnson/litestream#1467 (open).

3. Connection leak during failed compactions. If the upload fails while compaction is still reading its sources, each source stream is reopened after close and never closed (benbjohnson/litestream#1492). With a large stuck backlog against S3 this exhausted our host's TCP memory and wedged it. Fixed by benbjohnson/litestream#1500, merged 2026-10-08 but not in a release yet; 0.5.17 is still the latest.

How much this applies: our databases were 8-21 GB of append-heavy telemetry, which is what made the leak escalate in days. A typical app is more likely to see the overlap and the stalled compaction without the outage. The overlap that starts it, though, is the standard Rails + Kamal + SQLite deploy.

Suggested follow-ups

1. Bump to the first Litestream release that includes fix(replica): reconcile resumable reader recovery benbjohnson/litestream#1500. We're happy to send the PR (pin + checksums) when it's tagged.
2. A lock in the gem. Have the Puma plugin and litestream:replicate take an exclusive flock on a file beside the database before starting replicate, and wait for it. On Kamal the volume is a host directory, so the lock works across containers: the new container's replicate starts as soon as the old one exits. This is the shape we run in production (our supervisor holds the same lock). We can send this as a PR if you'd like it.
3. README note on deploy overlap until the lock and the bump land, and a hint to watch the L0 count (litestream ltx -level 0 -json) so a stalled compaction doesn't go unnoticed.

Upstream Litestream has no instance lock and no open issue for one; we carry one in our fork (Rebulk/litestream@08e5d3e).