Iceberg compaction always fails for primary-key tables — RewriteFiles committed after RowDelta

[amccartney-r7]

Search before asking

• I searched in the issues and found nothing similar.

Fluss version

0.9.1-incubating

Please describe the bug 🐞

IcebergLakeCommitter.commit() commits the RowDelta (new data + equality deletes) before the RewriteFiles (compaction). For PK tables using merge-on-read, every RowDelta adds equality delete files targeting existing data files in the same partitions. When commitRewrite() runs immediately after, validateFromSnapshot() detects those new deletes and rejects the commit.

Log evidence:

ERROR org.apache.fluss.lake.iceberg.tiering.IcebergLakeCommitter - Failed to commit rewrite files to iceberg, delete rewrite added files [...]
org.apache.iceberg.exceptions.ValidationException: Cannot commit, found new delete for replaced data file: GenericDataFile{content=data, ...}
    at org.apache.iceberg.exceptions.ValidationException.check(ValidationException.java:49)

This happens on every commit cycle for PK tables. The snapshot history confirms compaction never commits — every operation is overwrite (RowDelta), never replace (RewriteFiles):

operation    total-data-files
overwrite    312
overwrite    310
overwrite    308
overwrite    306

The file count grows monotonically (one file per bucket per commit), with no compaction.

Root cause: Commit ordering in IcebergLakeCommitter.commit() (line 132 vs 139). The RowDelta commits first, adding equality deletes that target the files the rewrite wants to replace.

// Current: RowDelta first, RewriteFiles second
long snapshotId = commit(snapshotUpdate, snapshotProperties);  // line 132 — RowDelta adds deletes

List<RewriteDataFileResult> rewriteDataFileResults =
        committable.rewriteDataFileResults();
if (!rewriteDataFileResults.isEmpty()) {
    Long rewriteCommitSnapshotId =
            commitRewrite(rewriteDataFileResults, snapshotProperties);  // line 139 — always fails
}

Append-only tables are unaffected because AppendFiles adds no delete files.

Test coverage gap: The existing IcebergRewriteTest only covers append-only tables (no identifier fields in the schema). PK tables with equality deletes are untested.

Also affects main (as of dd2d30a).

Solution

Swap the commit order — RewriteFiles first, RowDelta second. The rewrite validates from the snapshot it was planned against; committing it before the RowDelta means no new deletes exist yet, so validation passes. The RowDelta's equality deletes then correctly target the compacted files (which have a lower sequence number).

After applying the fix, replace operations appear and the file count drops immediately:

operation    total-data-files
overwrite    4
replace      2
overwrite    6

Are you willing to submit a PR?

• I'm willing to submit a PR!