Repository navigation
feat(services): compact lake files after player erasure - #41
Merged
Merged
Conversation
TheMeinerLP
force-pushed
the
feat/compact-lake-after-erasure
branch
from
October 15, 2026 22:15
b038819 to
43db09f
Compare
Rewrite events and raw lake files without the sessions of erased players. Each file is streamed in bounded batches, uploaded through the staging protocol and swapped in one catalog transaction under a per-file advisory lock. Interrupted runs are resumed: replaced objects are purged on the next run, unreferenced new objects are left to housekeeping. Migration 0005 adds erasure_requests.compacted_at, lake_files.replaced_by and purge_pending, and the erased_sessions view.
Snapshots are immutable, so their manifest still names erased sessions
and files that compaction replaced. Add GET /snapshots/{id}/sessions
with exclude_erased and keyset paging, and GET /snapshots/{id}/files
which follows replaced_by to the live files. Both need the
data_scientist role.
Hourly by default, non-overlapping, hardened like housekeeping, with service account, network policy, values and schema entries.
…point Items carry session_id, pseudonym, started_at, segments, labels and the erased flag, as the ml jobs read them.
…d_at Job tokens may call the snapshot sessions and files endpoints; the erasure request output carries compacted_at.
TheMeinerLP
force-pushed
the
feat/compact-lake-after-erasure
branch
from
October 15, 2026 22:28
43db09f to
bfc29fe
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
corvus-compact: one-shot job that rewriteseventsand still-existingrawlake files without the sessions of erased players. Streams in bounded batches, uses the ingest staging protocol, swaps files in one catalog transaction under a per-filepg_try_advisory_lock, then purges the old object. Idempotent and resumable (purge_pending).0005_compaction.sql:erasure_requests.compacted_at,lake_files.replaced_by/purge_pending, viewcorvus.erased_sessions.GET /api/v1/snapshots/{id}/sessions?exclude_erased=true(paged) andGET /api/v1/snapshots/{id}/files(followsreplaced_by), roledata_scientist. Snapshots stay immutable.compactCronJob (hourly, hardened like housekeeping), values, schema, helm-unittest.Test plan
helm lint,helm template,helm unittest