Repository navigation
Keep detached rows visible through Parquet publication - #58
Merged
Merged
Conversation
VibeSWE
commented
Oct 21, 2026
VibeSWE
changed the base branch from
fix/current-minute-query-visibility
to
main
October 21, 2026 17:41
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A query between writer detachment and Parquet publication can omit rows that were already accepted. Retain detached writers' Arrow buffers until their corresponding Parquet is published, and capture memory plus local Parquet membership under the same publication lock. Rows remain visible exactly once across that local transition, including failed conversion and retry.
This extends the current-writer retention from merged PR #53 to detached writers. Physical part files are exclusively created with collision fallback. Startup recovery skips attached writers, coordinates with detachment/sealing, and transfers or discards retained-memory ownership along with the source file. Encoding and query execution remain outside the publication lock; Arrow payloads are shared.
Validation includes publication barriers, failed conversion and retry, late appends after failed sealing, overlapping detaches, and same-instance startup recovery. The three recovery regressions were run before the fix and all failed with the expected retained-memory duplication or stale rows. Fresh independent Sol review approved the recovery fix. After integrating #57, the writer, stream and query-provider suites passed: 86 passed, 1 ignored. A new combined publication/lease regression also passes, including ten independent reruns: it pauses publication, captures memory plus real file leases, deletes the original Parquet and verifies exact row IDs and lease cleanup. Fresh Sol and Opus reviews approved the integration; Opus also approved the corrected parallel test barriers.
Limits are documented in
docs/staging-conversion-visibility.md: memory staging must be enabled; failed sealing/conversion retains buffers until recovery and can increase RAM usage. PR50's broader recovery/converter ownership work remains separate. Staged-file lifetime and overlap with uploaded copies are addressed in #57. This does not establish an atomic snapshot spanning memory and remote manifests. This branch includes merged #57 and takes its file leases insidecapture_staging, before refreshing the remote snapshot. Capture-time cleanup and footer validation also run under the publication read lock and can delay publication on slow filesystems.This backend fix is separate from the merged Prism UI retirement in #55. The branch includes current main, including #53, #55 and #56, and includes the now-merged #57. This PR targets main.