Fix silent cleanup failure that filled the DB disk - #1
Open
digitalhen wants to merge 1 commit into
Open
Conversation
The nightly cleanup ran cleanup_old_data.sql in a single transaction. Its last drop_chunks targets route_station_5min_stats, and TimescaleDB chunk ops are gated on table OWNERSHIP — not GRANTs. The ML feature tables had been created by a superuser (peer auth on supermac), so as `subway` that drop_chunks raised "must be owner of hypertable". Because everything ran in one transaction, the error rolled back the realtime drop_chunks too, so the firehose was never pruned: 16 days of stop_time_updates (~69 GB) accumulated, filled supermac's disk, and crash-looped Postgres (June 2026). - cleanup_db: run each statement in its own transaction (autocommit) and log+continue on failure, so a late drop_chunks error can't undo the earlier, critical realtime drops. - migration: ALTER TABLE ... OWNER TO subway for alerts_unique and route_station_5min_stats, so ownership is deterministic regardless of who applies it (the prior GRANT-only fix in 8ee59e9 was insufficient). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
digitalhen
force-pushed
the
fix/cleanup-resilience-and-ml-ownership
branch
from
July 23, 2026 03:25
69f0a3a to
e781343
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What broke
The subway DB on supermac grew to 87 GB and filled the disk on June 12, crash-looping Postgres into recovery mode.
Root cause: the nightly cleanup (
capture/cron/daily_stats.py::cleanup_db) randb/cleanup_old_data.sqlin a single transaction. That script ends withdrop_chunks('route_station_5min_stats', …). TimescaleDB chunk ops are gated on table ownership, not GRANTs — and the ML feature tables (route_station_5min_stats,alerts_unique) were created by a superuser via peer auth, so assubwaythatdrop_chunksraisedmust be owner of hypertable. The error rolled back the realtimedrop_chunksearlier in the same transaction, so the firehose was never pruned. 16 days ofstop_time_updates(~69 GB) piled up → disk full → crash.Logged nightly since May 30 as
FAILED computing <day>: must be owner of hypertable "route_station_5min_stats". The earlier8ee59e9GRANT-only fix didn't help because ownership ≠ privileges.Changes
cleanup_db: run each statement in its own transaction (autocommit), log + continue on failure. A latedrop_chunkserror can no longer undo the earlier, critical realtime drops.ALTER TABLE … OWNER TO subwayfor both ML tables so ownership is deterministic regardless of who applies it.Prod already remediated (by hand)
ALTER TABLE … OWNER TO subwayapplied on supermac.subway(verified the fix): 87 GB → 722 MB, disk 88% → 43% (after thinning the APFS local snapshots that were pinning the freed blocks).🤖 Generated with Claude Code