[core] Reduce memory for global index-only commits - #8896
Draft
leaves12138 wants to merge 1 commit into
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
BinaryManifestEntryfields directly instead of materializing all data files asSimpleFileEntryrowIdCheckFromSnapshotWhy
A global index-only commit previously read every live data entry in the affected partitions into Java objects and retained that base-file list through snapshot preparation. It also invoked data-manifest sorting even though no data file changed.
On the large test table, conflict detection materialized 4,105,298
SimpleFileEntryinstances and the subsequent manifest sort rewrote 32 data manifests. These phases overlapped in memory.The new path still validates that every global-index row-id range is covered by current live data files. It performs the check with projected binary manifest entries, applies DELETE entries by full file identity, and retains only primitive ranges intersecting the requested index ranges.
Measured impact
Snapshot 3496 metadata, Linux x86,
-Xmx24g:bbcac771ff)903aa197a1The index commit produced the next snapshot successfully, and the tests assert that its data-manifest list is unchanged.