[server] Forbid ALTER bucket.num on aggregation merge engine tables - #4315
Merged
Merged
Conversation
platinumhamburg
previously requested changes
Sep 14, 2026
| } | ||
|
|
||
| @Test | ||
| void testAlterMergeEngineToAggregationRejectedAfterRescale() throws Exception { |
Contributor
There was a problem hiding this comment.
table.merge-engine cannot be altered regardless of whether the table has been rescaled, so this test only exercises the existing option whitelist. The rescale setup adds no coverage for the guard introduced by this PR. Could we remove this test and keep testAlterBucketNumRejectedOnAggregationTable to cover the new behavior?
Member
|
most CI are passed, merging... |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Refs #4280 , #4310.
Brief change log
ALTER TABLE SET ('bucket.num' = N)(rescale) on tables whose merge engine isaggregation. The validation runs before the lake propagation and the ZK commit, so a rejected ALTER has no side effects.table.merge-engineis not in the alterable-options whitelist and is already rejected by the existing validation; a new test locks this behavior down.Background and Motivation
The Flink sink for aggregation tables restores from checkpoints via undo recovery, whose correctness relies on the bucket shuffle keeping the "one bucket, one writer" invariant: the write offset (WriterState) of each bucket has a single authoritative reporter.
After a rescale, the pre-write shuffle of the Flink job still uses the table-level bucket count captured at job submission time. When it mismatches the partition's actual bucket count, records of the same bucket get scattered across multiple writer subtasks. The checkpoint state fragments, and on restore
RecoveryOffsetManager.putMergedOffsetthrows anIllegalStateException— the job can no longer recover from checkpoint/savepoint and can only be restarted from scratch with discarded state.Until the runtime fix that resolves per-partition bucket counts (#4310) lands, reject the combination up front to move the failure from "explodes at recovery time" to "rejected at ALTER submission time", guaranteeing data correctness. This mutual exclusion follows the same pattern as the existing historical-partition exclusion.
Test Plan
testAlterBucketNumRejectedOnAggregationTable: the ALTER bucket.num on an aggregation table is rejected, and the bucket count, bucketCountEpoch, and partition bucket counts are all untouched.testAlterMergeEngineToAggregationRejectedAfterRescale: after a rescale (epoch=1), switching the merge engine is rejected by the existing whitelist validation, and the bucket layout stays at the post-rescale state.AlterBucketNumTestpasses 25/25 (including the 2 new cases).API and Format
Documentation