[flink] Resolve per-partition bucket count in sink bucket shuffle - #4311
Draft
Kaixuan-Duan wants to merge 2 commits into
Draft
[flink] Resolve per-partition bucket count in sink bucket shuffle#4311Kaixuan-Duan wants to merge 2 commits into
Kaixuan-Duan wants to merge 2 commits into
Conversation
Kaixuan-Duan
marked this pull request as draft
September 14, 2026 03:23
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Closes #4310.
After
ALTER TABLE SET ('bucket.num' = N)rescales a partitioned table, partitions createdbefore the rescale keep their original bucket counts while new partitions use the new one.
However, the primary-key-table Flink sink's pre-write bucket shuffle still sharded records
with the table-level numBuckets captured at job submission time (a constructor argument
of
FlinkRowDataChannelComputer). The sharding denominator then mismatched the actualper-partition layout, scattering the records of one bucket across multiple writer subtasks.
Each writer reports its own log end offset for the same bucket, so the WriterState ends up
with conflicting per-bucket offsets, and
RecoveryOffsetManager.putMergedOffsetthrowsIllegalStateExceptionon restore — the job can never recover from checkpoint/savepoint.This PR restores the "one bucket, exactly one writer" invariant by making the bucket shuffle
resolve each partition's actual bucket count at runtime.
Brief change log
PartitionBucketCountResolver: resolves a partition's authoritative bucket count fromcluster metadata at runtime and caches it locally. A partition's bucket count is immutable
once created, so cached entries are valid forever and need no invalidation. Cache misses
are handled in three ways:
listPartitionInfoscall warms upevery partition of the table;
writer, so blocking here would deadlock) → fall back to the current table-level count,
which is exactly what the partition will be created with;
FlinkRowDataChannelComputernow uses the per-partition count for both the bucket id andthe sharding-mode decision, replacing the stale table-level value.
FlinkSinkpassesTablePath/Configurationto the channel computer at both call sites.Tests
PartitionBucketCountResolverTest: three-way miss handling, immutable caching,bounded-retry fail-fast — pure JVM with an injected fake metadata source.
PartitionBucketCountResolverITCase: cold-start authoritativeresolution (the pre-rescale partition keeps its count while the table-level value differs),
fallback matching the count the partition is actually created with, and metadata-call
counting across cache hits. Runs at parallelism 3 so both the pre- and post-rescale
partitions exercise the combine-mode sharding formula.
UndoRecoveryITCase#testPartitionedTableRecoveryAfterBucketRescale(end-to-end, threephases): write → stop-with-savepoint → rescale → a new submission restores the savepoint
and writes to the pre-rescale partition → a second recovery writes to the post-rescale
partition; asserts exact per-key sums at every phase. Before the fix, the second restore
failed with conflicting per-bucket offsets.
bucket 1 of partition 2024-01: channels [0, 2](one bucket written by two subtasks —the exact failure this PR fixes); bypassing the resolver makes all 5 unit tests fail.
fluss-flink-commonsink suite andfluss-commonpass.Known limitations
count lands between the fallback and the actual partition creation, that partition keeps
sharding with the stale count until a restart (sub-second window; rescales are rare).
must be discarded.
API and Format
No public API or RPC/storage format changes.