[core] Refactor sort compact to produce COMPACT commits by hbgstc123 · Pull Request #8631 · apache/paimon

hbgstc123 · 2026-07-15T01:54:44Z

Purpose

Follow up on #7595 per this review comment: sort compact should not be modeled as an OVERWRITE commit with extra base-snapshot handling. Instead, it should produce a normal COMPACT commit so the existing commit protocol can perform compact validation and conflict detection.

This PR fixes the concurrent-write data loss risk in sort compact by refactoring Flink/Spark sort compact for append tables to:

Write sorted output in write-only mode.
Rewrite the written append commit messages into CompactIncrement messages.
Commit them through the normal compact commit path.

No new overwritePartition(baseSnapshotId) APIs or special overwrite conflict logic are introduced.

This is a follow-up to #8571, scoped to append tables only per review feedback.

Additional hardening on top of the COMPACT-commit refactor:

Capture base-snapshot deletion-vector metadata in SortCompactPlanMetadata at planning time, so Flink commit recovery can still clean up DVs even if the base snapshot has expired.
At rewrite time, merge latest-snapshot DV metadata so concurrent DV writes are also cleaned up.
Align Flink commit failure behavior with Spark: abort newly written files when commit fails and the COMPACT snapshot is not yet visible.
Add sort-compaction.warn-input-files / sort-compaction.max-input-files to guard large sort compact plans whose input files are embedded in the Flink job graph.

Tests

Core

SortCompactCommitMessageRewriterTest
- rewrite to compact messages
- multi-bucket / cross-bucket rewrite
- deletion-vector cleanup, including concurrent DV writes and expired base snapshots
- captured plan metadata serialization / round trip
- compact commit success detection, including expired snapshot gaps
- inline compaction rejection

Flink

SortCompactCommitterTest
- merge partial committables before rewrite
- delete-only compact when all input rows are filtered out
- commit recovery does not delete planned input or create DV index files
- commit failure aborts written messages only when COMPACT snapshot is not yet visible
SortCompactActionForAppendTableITCase
- latest snapshot kind is COMPACT
- concurrent append between read and commit does not lose data
CommitterOperatorTest
CompactProcedureITCase updated to expect COMPACT

Spark

SortCompactSparkCommitTest
- commit failure aborts written messages only when COMPACT snapshot is not yet visible
CompactProcedureTestBase updated to expect COMPACT

Merge latest-snapshot deletion-vector metadata at rewrite time so concurrent DV writes are cleaned up, while keeping captured plan metadata for expired snapshots. Align Flink commit failure behavior with Spark by aborting write output when the COMPACT snapshot is not yet visible. Co-authored-by: Cursor <cursoragent@cursor.com>

…sed by subclasses Commit 0f5aecf changed commitUser, state, and write from protected to private, but GlobalFullCompactionSinkWrite and LookupSinkWrite still access them directly. This causes JDK 8 compilation failures. Restore protected visibility to fix the build. Co-authored-by: Cursor <cursoragent@cursor.com>

…pact restricts to append tables SortCompactAction now explicitly rejects primary-key tables and only supports bucket-unaware append tables. The dynamic-bucket test case exercises unsupported behavior and fails on Flink 2.x CI, so remove it. Co-authored-by: Cursor <cursoragent@cursor.com>

Sort compact now commits as COMPACT instead of OVERWRITE. Concurrent merge and sort compact on V2 delta row-level DV append tables can leave duplicate visible rows, so skip that combination and keep coverage on the V1 write path. Co-authored-by: Cursor <cursoragent@cursor.com>

JingsongLi · 2026-07-16T05:46:03Z

There is a data correctness issue with “Sort compact” and concurrent DV updates. Since “Sort compact” generates output based on an old snapshot but deletes the latest DV from the input file upon commit, this can cause deleted data to be restored.

hbgstc123 · 2026-07-17T05:00:39Z

There is a data correctness issue with “Sort compact” and concurrent DV updates. Since “Sort compact” generates output based on an old snapshot but deletes the latest DV from the input file upon commit, this can cause deleted data to be restored.

Right, just pushed a fix

…pend_only

Resolve conflicts in FlinkSink and FlinkSinkBuilder by keeping both sort-compact writeProviderOverride/createAppendTableSink hooks and master's BLOB descriptor reader factory wiring. Co-authored-by: Cursor <cursoragent@cursor.com>

JingsongLi · 2026-07-23T13:09:52Z

+        return this;
+    }
+
+    private void validateSortCompactInput(List<DataSplit> compactInputSplits) {


Remove this method and two options. We don't need to restrict this.

Drop validateSortCompactInput and the warn/max input file options per review feedback. Co-authored-by: Cursor <cursoragent@cursor.com>

…ct_append_only Resolve conflict in BaseAppendDeleteFileMaintainer by keeping upstream javadoc. Co-authored-by: Cursor <cursoragent@cursor.com>

JingsongLi · 2026-07-26T11:49:16Z

dvMaintainer.persist() has already written a new DV file during the rewrite phase, but Flink/Spark’s failure cleanup only calls abort on the original writtenMessages; the new DV exists only in the rewritten compact messages and will not be cleaned up. During a delete-only compact, writtenMessages is even empty, and cleanup returns immediately. Retry attempts following conflicts or commit failures will continue to generate orphaned index files.

Co-authored-by: Cursor <cursoragent@cursor.com>

hbgstc123 · 2026-07-27T07:49:20Z

dvMaintainer.persist() has already written a new DV file during the rewrite phase, but Flink/Spark’s failure cleanup only calls abort on the original writtenMessages; the new DV exists only in the rewritten compact messages and will not be cleaned up. During a delete-only compact, writtenMessages is even empty, and cleanup returns immediately. Retry attempts following conflicts or commit failures will continue to generate orphaned index files.

done

JingsongLi · 2026-07-28T13:48:29Z


    DataFileMeta assignSequenceNumber(long minSequenceNumber, long maxSequenceNumber);

+    default DataFileMeta assignFileSource(FileSource fileSource) {


Why add this method to interface default?

DataFileMeta is @public, so a default method avoids breaking external implementations (same pattern as Table#newHybridSearchBuilder). However the only implementation in Paimon project is PojoDataFileMeta, and it has a real override. Happy to make it abstract

…append_only

After making DataFileMeta#assignFileSource abstract, BinaryDataFileMeta from master must provide the method explicitly.

hbgstc123 mentioned this pull request Jul 15, 2026

[core] Refactor sort compact to produce COMPACT commits #8571

Closed

hbgstc123 force-pushed the fix_sort_compact_append_only branch from c8a1b81 to de2e634 Compare July 15, 2026 08:04

hbg and others added 5 commits July 16, 2026 09:58

[core] Refactor sort compact to produce COMPACT commits

6d4da26

hbgstc123 force-pushed the fix_sort_compact_append_only branch from de2e634 to d462567 Compare July 16, 2026 02:01

address comment, detect concurrent dv conflict

92e4afa

hbg and others added 3 commits July 17, 2026 14:02

Merge remote-tracking branch 'apache/master' into fix_sort_compact_ap…

dcd3ba7

…pend_only

report correct error message in ConflictDetection

13ba744

Merge branch 'master' into fix_sort_compact_append_only

66235f7

Resolve conflicts in FlinkSink and FlinkSinkBuilder by keeping both sort-compact writeProviderOverride/createAppendTableSink hooks and master's BLOB descriptor reader factory wiring. Co-authored-by: Cursor <cursoragent@cursor.com>

JingsongLi reviewed Jul 23, 2026

View reviewed changes

hbg and others added 2 commits July 24, 2026 00:49

[flink] Remove sort compact input file count limits

626bb3f

Drop validateSortCompactInput and the warn/max input file options per review feedback. Co-authored-by: Cursor <cursoragent@cursor.com>

Merge branch 'master' of github.com:apache/paimon into fix_sort_compa…

0ab8ac0

…ct_append_only Resolve conflict in BaseAppendDeleteFileMaintainer by keeping upstream javadoc. Co-authored-by: Cursor <cursoragent@cursor.com>

hbg and others added 2 commits July 27, 2026 12:06

delete new dv index files when error thrown in rewrite.

1423d4f

Retrigger CI for Flink 1.x Others workflow.

ffc40c2

Co-authored-by: Cursor <cursoragent@cursor.com>

JingsongLi reviewed Jul 28, 2026

View reviewed changes

hbg added 4 commits July 29, 2026 10:22

address comment, make DataFileMeta#assignFileSource abstract

5a39d9c

Merge remote-tracking branch 'upstream/master' into fix_sort_compact_…

0231843

…append_only

[core] Implement assignFileSource for BinaryDataFileMeta

5830cab

After making DataFileMeta#assignFileSource abstract, BinaryDataFileMeta from master must provide the method explicitly.

Trigger CI re-run.

604c6ab

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[core] Refactor sort compact to produce COMPACT commits - #8631

[core] Refactor sort compact to produce COMPACT commits#8631
hbgstc123 wants to merge 17 commits into
apache:masterfrom
hbgstc123:fix_sort_compact_append_only

hbgstc123 commented Jul 15, 2026 •

edited

Loading

Uh oh!

JingsongLi commented Jul 16, 2026

Uh oh!

hbgstc123 commented Jul 17, 2026

Uh oh!

JingsongLi Jul 23, 2026

Uh oh!

hbgstc123 Jul 23, 2026

Uh oh!

JingsongLi commented Jul 26, 2026

Uh oh!

hbgstc123 commented Jul 27, 2026

Uh oh!

JingsongLi Jul 28, 2026

Uh oh!

hbgstc123 Jul 29, 2026

Uh oh!

hbgstc123 Jul 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants


		DataFileMeta assignSequenceNumber(long minSequenceNumber, long maxSequenceNumber);

		default DataFileMeta assignFileSource(FileSource fileSource) {

Uh oh!

Conversation

hbgstc123 commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Tests

Uh oh!

JingsongLi commented Jul 16, 2026

Uh oh!

hbgstc123 commented Jul 17, 2026

Uh oh!

JingsongLi Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

hbgstc123 Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

JingsongLi commented Jul 26, 2026

Uh oh!

hbgstc123 commented Jul 27, 2026

Uh oh!

JingsongLi Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

hbgstc123 Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

hbgstc123 Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

hbgstc123 commented Jul 15, 2026 •

edited

Loading