Skip to content

[server] Forbid ALTER bucket.num on aggregation merge engine tables - #4315

Merged
wuchong merged 1 commit into
apache:mainfrom
Kaixuan-Duan:forbid-rescale-on-agg-table
Sep 14, 2026
Merged

[server] Forbid ALTER bucket.num on aggregation merge engine tables#4315
wuchong merged 1 commit into
apache:mainfrom
Kaixuan-Duan:forbid-rescale-on-agg-table

Conversation

@Kaixuan-Duan

Copy link
Copy Markdown
Contributor

Purpose

Refs #4280 , #4310.

Brief change log

  • Forbid executing ALTER TABLE SET ('bucket.num' = N) (rescale) on tables whose merge engine is aggregation. The validation runs before the lake propagation and the ZK commit, so a rejected ALTER has no side effects.
  • The reverse direction (switching a rescaled table to the aggregation engine) needs no new code: table.merge-engine is not in the alterable-options whitelist and is already rejected by the existing validation; a new test locks this behavior down.

Background and Motivation

The Flink sink for aggregation tables restores from checkpoints via undo recovery, whose correctness relies on the bucket shuffle keeping the "one bucket, one writer" invariant: the write offset (WriterState) of each bucket has a single authoritative reporter.

After a rescale, the pre-write shuffle of the Flink job still uses the table-level bucket count captured at job submission time. When it mismatches the partition's actual bucket count, records of the same bucket get scattered across multiple writer subtasks. The checkpoint state fragments, and on restore RecoveryOffsetManager.putMergedOffset throws an IllegalStateException — the job can no longer recover from checkpoint/savepoint and can only be restarted from scratch with discarded state.

Until the runtime fix that resolves per-partition bucket counts (#4310) lands, reject the combination up front to move the failure from "explodes at recovery time" to "rejected at ALTER submission time", guaranteeing data correctness. This mutual exclusion follows the same pattern as the existing historical-partition exclusion.

Test Plan

  • Added testAlterBucketNumRejectedOnAggregationTable: the ALTER bucket.num on an aggregation table is rejected, and the bucket count, bucketCountEpoch, and partition bucket counts are all untouched.
  • Added testAlterMergeEngineToAggregationRejectedAfterRescale: after a rescale (epoch=1), switching the merge engine is rejected by the existing whitelist validation, and the bucket layout stays at the post-rescale state.
  • AlterBucketNumTest passes 25/25 (including the 2 new cases).

API and Format

Documentation

@wuchong wuchong left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

}

@Test
void testAlterMergeEngineToAggregationRejectedAfterRescale() throws Exception {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

table.merge-engine cannot be altered regardless of whether the table has been rescaled, so this test only exercises the existing option whitelist. The rescale setup adds no coverage for the guard introduced by this PR. Could we remove this test and keep testAlterBucketNumRejectedOnAggregationTable to cover the new behavior?

@wuchong

wuchong commented Sep 14, 2026

Copy link
Copy Markdown
Member

most CI are passed, merging...

@wuchong
wuchong merged commit d920be6 into apache:main Sep 14, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants