branch-4.1:[feat](paimon) support paimon-rust reader on BE

[pzhdfy]

What problem does this PR solve?

Related PR: #65883 (for master)

we current will keep iterating on branch-4.1, because our internal doris forks from branch-4.1

and the pr for master will be closed later, becuase paimon-rust reader is based on scanner v1, and all the scanner v1 code will be removed, we will refactor paimon-rust reader to scanner v2 in the future, and open a new pull request.

Problem Summary:

Adds a third BE-side reader for Paimon FORMAT_JNI splits, backed by the
paimon-rust C bindings (libpaimon_c.a, statically linked). Rust Reader is gated by the session variable
enable_paimon_rust_reader (default false).

attention:
when using enable_paimon_rust_reader=true, must set enable_file_scanner_v2=false;
because now will all route to scanner_v2 when enable_file_scanner_v2=true; after #66008

Compared with the existing readers:

• vs paimon-jni: no JVM round-trip on the hot path — the split is
  deserialized in-process and rows arrive via the Arrow C Data Interface,
  which shortens the scanner critical section and removes JNI copy cost.
• vs paimon-cpp: reuses paimon's official Rust implementation. paimon-rust tracks upstream paimon spec directly
  (schema, DV, snapshots, predicates), so features/fixes propagate may be more fast and different.
  Our goal is not to replace paimon-cpp, just give user a choice to switch at query time.

Architecture

FE (PaimonScanNode): when enable_paimon_rust_reader is on and the split
is a DataSplit, encode it with paimon's native binary serialization
(PaimonUtil.encodeDataSplitToString), stamp the range with
TPaimonReaderType.PAIMON_RUST, and attach db_name / table_name /
paimon_table so BE can open the table through a catalog.

BE (FileScanner::_get_next_reader): dispatches on reader_type. On
PAIMON_RUST it instantiates PaimonRustReader, which:

1. paimon_catalog_create({warehouse, metastore=filesystem, ...})
   • warehouse is derived from the table root path by stripping
     /<db>.db/<table> (paimon FS convention).
2. paimon_identifier_new(db, tbl) → paimon_catalog_get_table
3. paimon_table_new_read_builder → projection (case-insensitive) →
   optional filter from PaimonRustPredicateConverter (translates Doris
   push-down conjuncts into paimon-rust predicates, best-effort so any
   non-representable conjunct is silently dropped and re-applied by the
   engine).
4. paimon_plan_from_split_bytes(split_bytes) — one BE scanner reads
   exactly its FE-planned split (no re-planning).
5. paimon_table_read_to_arrow → streamed arrow record batches → block.

Partition columns are back-filled in on_after_read_block from
FE-provided columns_from_path*, mirroring PaimonCppReader. OSS/S3
hadoop-style credential keys (fs.oss.* / fs.s3a.*) are remapped to
AWS_ACCESS_KEY / AWS_ENDPOINT / ... so paimon-rust's FileIO can
authenticate.

Thirdparty

• Introduces paimon-rust-2fb5e49 (built with Rust 1.91.0, feature
  paimon/storage-hdfs, --locked).
• Auto-installs cbindgen and regenerates the C header from the Rust
  extern "C" surface via a locally-written cbindgen.toml
  (language=C, include_guard=PAIMON_C_H, pragma_once=true, cpp_compat=true),
  so the shipped paimon.h always matches the built .a.

Release note

Add enable_paimon_rust_reader session variable. When enabled, Paimon
non-native reads on BE use the paimon-rust reader instead of paimon-jni /
paimon-cpp.

Check List (For Author)

• Test

  • Unit Test
  • Manual test (add detailed scripts or steps below)
    1. CREATE CATALOG paimon_fs PROPERTIES ("type"="paimon", "paimon.catalog.type"="filesystem", "warehouse"="file:///media/ssd1/test_paimon/warehouse/"); when using "warehouse"="hdfs://.../warehouse/", add HADOOP_CONF_DIR=xx in be.conf
    2. SET enable_paimon_rust_reader=true; SET force_jni_scanner=1;set enable_file_scanner_v2=0;
    3. SELECT from a partitioned/DV paimon table; results must match enable_paimon_rust_reader=false and force_jni_scanner=0 and enable_file_scanner_v2=1.
    4. SELECT count(*) — should hit table-level row count push-down.
    5. WHERE a=1 AND b>2 — BE INFO log should show paimon-rust predicate pushdown: applied.
  • Regression test
  • No need to test or manual test.

• Behavior changed:

  • No. Guarded by enable_paimon_rust_reader (default false); when off, all paths are unchanged.

• Does this need documentation?

  • No.
  • Yes.

Check List (For Reviewer who merge this PR)

• Confirm the release note
• Confirm test cases
• Confirm document
• Add branch pick label

[hello-stephen]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?