[feat](paimon) support paimon-rust reader on BE

[pzhdfy]

What problem does this PR solve?

Problem Summary:

another pr for branch-4.1: #66227
we current will keep iterating on branch-4.1, because our internal doris forks from branch-4.1

and this pr for master will be closed later, becuase paimon-rust reader is based on scanner v1, and all the scanner v1 code will be removed, we will refactor paimon-rust reader to scanner v2 in the future, and open a new pull request.

Adds a third BE-side reader for Paimon FORMAT_JNI splits, backed by the
paimon-rust C bindings (libpaimon_c.a, statically linked). Rust Reader is gated by the session variable
enable_paimon_rust_reader (default false).

Compared with the existing readers:

• vs paimon-jni: no JVM round-trip on the hot path — the split is
  deserialized in-process and rows arrive via the Arrow C Data Interface,
  which shortens the scanner critical section and removes JNI copy cost.
• vs paimon-cpp: reuses paimon's official Rust implementation. paimon-rust tracks upstream paimon spec directly
  (schema, DV, snapshots, predicates), so features/fixes propagate may be more fast and different.
  Our goal is not to replace paimon-cpp, just give user a choice to switch at query time.

Architecture

FE (PaimonScanNode): when enable_paimon_rust_reader is on and the split
is a DataSplit, encode it with paimon's native binary serialization
(PaimonUtil.encodeDataSplitToString), stamp the range with
TPaimonReaderType.PAIMON_RUST, and attach db_name / table_name /
paimon_table so BE can open the table through a catalog.

BE (FileScanner::_get_next_reader): dispatches on reader_type. On
PAIMON_RUST it instantiates PaimonRustReader, which:

1. paimon_catalog_create({warehouse, metastore=filesystem, ...})
   • warehouse is derived from the table root path by stripping
     /<db>.db/<table> (paimon FS convention).
2. paimon_identifier_new(db, tbl) → paimon_catalog_get_table
3. paimon_table_new_read_builder → projection (case-insensitive) →
   optional filter from PaimonRustPredicateConverter (translates Doris
   push-down conjuncts into paimon-rust predicates, best-effort so any
   non-representable conjunct is silently dropped and re-applied by the
   engine).
4. paimon_plan_from_split_bytes(split_bytes) — one BE scanner reads
   exactly its FE-planned split (no re-planning).
5. paimon_table_read_to_arrow → streamed arrow record batches → block.

Partition columns are back-filled in on_after_read_block from
FE-provided columns_from_path*, mirroring PaimonCppReader. OSS/S3
hadoop-style credential keys (fs.oss.* / fs.s3a.*) are remapped to
AWS_ACCESS_KEY / AWS_ENDPOINT / ... so paimon-rust's FileIO can
authenticate.

Thirdparty

• Introduces paimon-rust-2fb5e49 (built with Rust 1.91.0, feature
  paimon/storage-hdfs, --locked).
• Auto-installs cbindgen and regenerates the C header from the Rust
  extern "C" surface via a locally-written cbindgen.toml
  (language=C, include_guard=PAIMON_C_H, pragma_once=true, cpp_compat=true),
  so the shipped paimon.h always matches the built .a.

Release note

Add enable_paimon_rust_reader session variable. When enabled, Paimon
non-native reads on BE use the paimon-rust reader instead of paimon-jni /
paimon-cpp.

Check List (For Author)

• Test

  • Unit Test
  • Manual test (add detailed scripts or steps below)
    1. CREATE CATALOG paimon_fs PROPERTIES ("type"="paimon", "paimon.catalog.type"="filesystem", "warehouse"="file:///media/ssd1/test_paimon/warehouse/"); when using "warehouse"="hdfs://.../warehouse/", add HADOOP_CONF_DIR=xx in be.conf
    2. SET enable_paimon_rust_reader=true; SET force_jni_scanner=1;
    3. SELECT from a partitioned/DV paimon table; results must match enable_paimon_rust_reader=false and force_jni_scanner=0.
    4. SELECT count(*) — should hit table-level row count push-down.
    5. WHERE a=1 AND b>2 — BE INFO log should show paimon-rust predicate pushdown: applied.
  • Regression test
  • No need to test or manual test.

• Behavior changed:

  • No. Guarded by enable_paimon_rust_reader (default false); when off, all paths are unchanged.

• Does this need documentation?

  • No.
  • Yes.

Check List (For Reviewer who merge this PR)

• Confirm the release note
• Confirm test cases
• Confirm document
• Add branch pick label

[hello-stephen]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?

[suxiaogang223]

BE should not reopen the catalog to obtain the table here. FE has already resolved and serialized the Paimon table in TFileScanRangeParams.serialized_table and planned the exact split; the JNI reader restores that table context and executes the split directly. Re-resolving it through paimon_catalog_get_table may observe different table/schema state and unnecessarily restricts this path to the derived filesystem-catalog layout. Please deserialize/restore the FE-provided table information for the Rust reader instead of doing a catalog lookup in BE.

[pzhdfy]

跟paimon_rust社区同学沟通回复（后面发布后 会切换table获取接口）：
paimon-rust 将提供通用的 catalog-free 接口 paimon_table_from_schema_json。Doris FE 直接传递 table_path、完整的 TableSchema JSON、database/table_name、branch 和文件系统访问配置，Rust 侧直接构造 Table，不创建 Catalog，也不会重新加载
或覆盖 FE 已解析的 schema。branch 仅决定 schema/snapshot/tag 的读取路径，读取仍可使用 FE 生成的 splits；如需 Rust 侧 time travel，可继续使用现有的 read-builder options。POC 代码已推送到 JunRuiLee/paimon-rust 的 feat/resolved-table-c-api 分
支，commit 为 9c39365。

[lxy-9602]

@pzhdfy Thank you for the contibution! Just to clarify, paimon-cpp has already been donated from the Alibaba repository Alibaba repo to the Apache repo and is currently in the release process. It is now the official Paimon standard-format implementation under the Apache community, and we expect it to continue tracking the upstream standard as the Paimon community evolves. So, I’m not sure “bespoke C++ port” is a accurate wording here. Also, paimon-cpp has been widely used in production within Alibaba for about 2 years, with both stability and performance validated in large-scale scenarios. Given that Doris’s backend is implemented in C++, I wonder whether the C++ path might be a more natural fit to prioritize here.

[pzhdfy]

@pzhdfy Thank you for the contibution! Just to clarify, paimon-cpp has already been donated from the Alibaba repository Alibaba repo to the Apache repo and is currently in the release process. It is now the official Paimon standard-format implementation under the Apache community, and we expect it to continue tracking the upstream standard as the Paimon community evolves. So, I’m not sure “bespoke C++ port” is a accurate wording here. Also, paimon-cpp has been widely used in production within Alibaba for about 2 years, with both stability and performance validated in large-scale scenarios. Given that Doris’s backend is implemented in C++, I wonder whether the C++ path might be a more natural fit to prioritize here.

1. yes paimon-cpp has moved to Apache repo. But supported features may be different.
   Our goal is not to replace paimon-cpp, just give user a choice to switch at query time.

2. paimon-cpp(opensource version) has poor performance，when doing merge-on-read or predicate pushdown paimon-cpp spends 4x~6x time than paimon-java.

• paimon-cpp use arrow parquet, which has poor read performance, and predicate pushdown is not well implemented.
• paimon-cpp merge on read is not well implemented in opensource version
• all above is base on the opensource version integrated in doris which is tagged in 2026.2 (a very old version) We do not test the newest version, and do not know whether has more optimization‌ in paimon-cpp Alibaba internal implemention

3.paimon-rust is 2x~3x faster than paimon-java when read MOR ,and close to doris native reader when read DV（ only slow 10%，because arrow convert to doris block）

• rust native parquet reader has nice performance
• better Vectorization and predicate pushdown

[lxy-9602]

@pzhdfy Thanks for the reply. If paimon-cpp currently looks slower than the Java implementation, I wonder whether it could be related to configuration. A few points that may be worth double-checking:

• Some read-path optimizations in paimon-cpp are disabled by default, so it might be worth testing again with EnablePrefetch, EnableMultiThreadRowToBatch, and SetRowToBatchThreadNumber(3) enabled.
• In our internal tests, we observed roughly a 2–4x E2E performance improvement over the Java-based path after enabling the relevant optimizations.
• For pushdown performance, paimon-cpp master has already been optimized to support page-level pushdown, which reduces latency by an order of magnitude compared with the previous implementation (for example, refact: Reuse RowGroupPageIndexReader across columns to improve page-level predicate pushdown performance regression on wide tables alibaba/paimon-cpp#316 334 375 424)

Looking forward to seeing more testing and results.