branch-4.1: [feature](cloud) support single rowset grouped compaction (#65907)

[mymeiyi]

pick #65907

[hello-stephen]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?

[Copilot]

🟡 Changes recommended

Three moderate unresolved comments in grouped compaction must be addressed before approval.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Adds cloud single-rowset grouped compaction to reduce peak memory usage while preserving grouped metadata across storage and cloud workflows.

Changes:

• Adds configurable grouped compaction and range-based merging.
• Updates readers, writers, metadata, snapshots, schema changes, and row-ID conversion.
• Adds unit, integration, and cloud regression coverage.

File summaries

File	Summary
regression-test/suites/cloud_p0/compaction/test_cloud_single_rowset_grouped_compaction.groovy	Adds end-to-end grouped compaction scenarios.
gensrc/proto/olap_file.proto	Adds grouped overlap metadata.
be/test/storage/segment/inverted_index_file_writer_test.cpp	Updates segment writer test coverage.
be/test/storage/rowset/rowset_meta_test.cpp	Tests grouped metadata semantics.
be/test/storage/rowid_conversion_test.cpp	Tests grouped row-ID conversion.
be/test/storage/pb_convert_test.cpp	Tests grouped metadata conversion.
be/test/storage/iterator/vertical_block_reader_test.cpp	Tests grouped iterator initialization.
be/test/storage/cloud_file_cache_write_index_only_test.cpp	Tests grouped segment allocation tracking.
be/test/cloud/cloud_snapshot_mgr_test.cpp	Tests snapshot metadata preservation.
be/test/cloud/cloud_cumulative_compaction_policy_test.cpp	Tests grouped compaction policy behavior.
be/test/cloud/cloud_compaction_test.cpp	Tests grouped selection and range execution.
be/src/storage/rowset/vertical_beta_rowset_writer.h	Declares grouped vertical writer APIs.
be/src/storage/rowset/vertical_beta_rowset_writer.cpp	Retains segment metadata across range flushes.
be/src/storage/rowset/rowset_writer.h	Updates rowset writer interfaces.
be/src/storage/rowset/rowset_meta.h	Stores and validates grouped metadata.
be/src/storage/rowset/beta_rowset_writer.h	Updates beta writer declarations.
be/src/storage/rowset/beta_rowset_writer_v2.h	Updates versioned beta writer declarations.
be/src/storage/rowid_conversion.h	Supports reused mappings across grouped ranges.
be/src/storage/merger.h	Updates merge interfaces for segment ranges.
be/src/storage/merger.cpp	Propagates segment ranges through merges.
be/src/storage/iterator/vertical_block_reader.h	Declares grouped reader helpers.
be/src/storage/iterator/vertical_block_reader.cpp	Lazily initializes grouped segment iterators.
be/src/storage/compaction/compaction.h	Adds range-aware merge interfaces.
be/src/storage/compaction/compaction.cpp	Supports range-based merge execution.
be/src/cloud/pb_convert.cpp	Converts grouped metadata between protobufs.
be/src/cloud/config.h	Declares grouped compaction settings.
be/src/cloud/config.cpp	Defines grouped compaction settings.
be/src/cloud/cloud_snapshot_mgr.cpp	Preserves grouped metadata in snapshots.
be/src/cloud/cloud_schema_change_job.cpp	Clears grouped layout during rewrites.
be/src/cloud/cloud_cumulative_compaction.h	Adds grouped compaction state and APIs.
be/src/cloud/cloud_cumulative_compaction.cpp	Implements grouped selection and execution.

Review details

Suppressed comments (2)

be/src/cloud/cloud_cumulative_compaction.cpp:853

• Grouped vertical compaction now calls vertical_merge_rowsets once per segment range, but that routine assigns each range's sample_info directly into the tablet-wide cumu_sample_infos entries. After this loop the cached bytes/rows/group sizes describe only the last range, so the next estimate_batch_size() can choose a badly sized batch for the table's actual data and undermine the memory-saving goal. Aggregate the per-range samples or update the tablet cache once with statistics covering all ranges.

        RETURN_IF_ERROR(execute_merge(group_readers, range.merge_way_num, &group_stats,
                                      std::make_pair(range.segment_start, range.segment_end),
                                      {.total_ranges = cast_set<int64_t>(segment_ranges.size()),
                                       .range_index = cast_set<int64_t>(range_index)}));

be/src/cloud/cloud_cumulative_compaction.cpp:334

• The grouped-mode flag suppresses cumulative-point advancement for the entire task, even when this pass emits only one output range and update_output_rowset_after_build() leaves the rowset as ordinary NONOVERLAPPING. In the non-parallel path, modify_rowsets() therefore leaves the cumulative point behind a completed rowset; the next picker rejects the sole non-overlapping rowset, so it can remain stranded in the cumulative layer. Base this decision on the built output overlap state rather than only on whether grouped mode was selected, and cover the one-range case.

    if (!_single_rowset_compaction_segment_group_size.has_value()) {
        return true;
    }

    DORIS_CHECK_EQ(_input_rowsets.size(), 1);

• Files reviewed: 31/31 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.