feat: support parquet page row count limit

[lucasfang]

Purpose

Linked issue: close #xxx

This change adds the write option parquet.page.row.count.limit to the Parquet
format layer, aligning with parquet-mr's parquet.page.row.count.limit
(PARQUET-1414, parquet-mr 1.11.0) in both semantics and default value (20000):

• Arrow patch (cmake_modules/arrow.diff): extends parquet::WriterProperties
  with data_page_row_count_limit (default DEFAULT_DATA_PAGE_ROW_COUNT_LIMIT = 20000). TypedColumnWriterImpl now finishes the current data page when
  either the estimated encoded size reaches parquet.page.size or the buffered
  row count reaches the limit, whichever comes first — the same rule parquet-mr
  uses. The limit is checked at write-batch granularity.
• Paimon format layer: defines the option key in parquet_format_defs.h and
  plumbs it through ParquetWriterBuilder::PrepareWriterProperties into the
  Arrow WriterProperties::Builder.

Bounding rows per page keeps the column index (page index) fine-grained, so
page-level predicate pushdown stays effective on pages that would otherwise
hold a huge number of highly-compressible rows.

Behavior notes (default changed from "unlimited" to 20000, matching Java):

• All-null / highly compressible columns now finish pages early, which sharply
  reduces writer memory (e.g. the all-null case in TestMemoryControl drops
  from ~20 MB to ~0.36 MB peak).
• The parquet.writer.max.memory.use peak can transiently reach ~2.2x the
  budget: the budget is only checked between batches, and with row-count
  splitting the budget is mostly held as finished page buffers, which
  BufferedPageWriter copies into its in-memory sink when the row group is
  flushed (2x plus one batch is the theoretical bound regardless of encoding).
  TestMemoryControl was updated accordingly with the measured rationale in
  comments.

The commit also hardens logging_test.cpp: glog global state (log destination
and flags) is now restored via ScopeGuard on every exit path, and the
<program>.INFO symlink is disabled so temporary-directory cleanup cannot
break on a dangling link.

Tests

Unit tests (all run locally and passing):

• ParquetWriterBuilderTest.DefaultPrepareWriterProperties — asserts the
  default data_page_row_count_limit is 20000.
• ParquetWriterBuilderTest.PrepareWriterProperties — sets
  parquet.page.row.count.limit=40000 and asserts it overrides the default.
• Arrow-side TestColumnWriter.WriteDataPagesSplitOnRowCountLimit (in
  arrow.diff) — verifies pages split exactly at the row-count limit for both
  flat and repeated columns, and repeated columns do not split records across
  pages.
• ParquetFormatWriterTest.TestMemoryControl — re-baselined for the new
  default: all-null runs assert the writer stays below the memory budget;
  non-null runs assert the budget triggers row-group splitting with the peak
  bounded by 2.5x.

API and Format

• No public API change under include/paimon/.
• New format-layer option key: parquet.page.row.count.limit (write path,
  default 20000, aligned with parquet-mr).
• The on-disk format remains fully compatible; only data page boundaries become
  finer for row-count-bound pages, which increases page index entries per
  column chunk.

Documentation

No standalone documentation change; the option key and its semantics are
documented in parquet_format_defs.h next to the definition.

Generative AI tooling

Generated-by: Qoder

[Copilot]

Pull request overview

This PR adds support for configuring a Parquet data page row count limit via Paimon’s Parquet writer properties, alongside related dependency patch updates and test adjustments.

Changes:

• Add a new Parquet write option parquet.page.row.count.limit and plumb it into ParquetWriterBuilder.
• Extend ParquetWriterBuilder unit tests to cover the row count limit property.
• Improve LoggerTest.TestGlogWritesLogFileToDisk cleanup by using ScopeGuard to restore process-wide glog state.
• Expand the bundled Arrow/Parquet patch (cmake_modules/arrow.diff) to introduce/adjust Parquet writer properties (including the row count limit) and other Arrow/Parquet behaviors.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
src/paimon/format/parquet/parquet_writer_builder.cpp	Reads the new option and sets Parquet writer data_page_row_count_limit.
src/paimon/format/parquet/parquet_writer_builder_test.cpp	Adds assertions for default and configured data_page_row_count_limit.
src/paimon/format/parquet/parquet_format_defs.h	Defines the new option key parquet.page.row.count.limit and documents its semantics.
src/paimon/common/logging/logging_test.cpp	Uses ScopeGuard to reliably restore glog global flags/destinations after the test.
cmake_modules/arrow.diff	Updates the vendored Arrow/Parquet patch to add row count limit support and additional related/unrelated changes.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.