[python] Expand native planning across resolved tables and scan modes

[JingsongLi]

Purpose

Expand native planning across resolved table contexts and scan modes while preserving Java semantics. This draft depends on apache/paimon-rust#847. Native CI continues to build Rust main; the dependency must merge before the new native coverage can pass there.

• Preserve the Python table's complete resolved schema and effective options for filesystem, JDBC and REST catalogs. Enable stale schemas, historical schemas, copy() overrides/removals, and REST branches whose schemas are only available from the catalog. JDBC planning uses filesystem snapshots without another database connection. REST tables retain catalog snapshot loading, credentials and token refresh; mismatched locations and unreproducible custom contexts retain fallback. Pass structured identifiers to Rust so database and table names containing dots retain their identity, including branch reads.
• Enable catalogless planning with standard local, PyArrow and resolving FileIO. FileStoreTable.from_path() now uses the FileIO factory and accepts storage options. Resolve each file's concrete FileIO before PyArrow reading, and correctly decode a REST response with a null snapshot.
• Use Rust file and deletion-vector planning for append/DE chunk shuffle, including incremental windows. Retain Python's live-row chunk sizing, seeded ordering and balanced worker assignment, and keep aligned column files before chunk construction.
• Enable native planning for fully materialized first-row/DV clustering files across levels, all-L0 first-row runs, and L0 runs with materialized files in separate raw splits. Retain fallback when a first-row plan containing L0 requires merging materialized clustered files.
• Preserve Python raw-reader filters across schema evolution: evaluate cross-schema predicates after field-ID normalization and before projection/limit, avoiding unsafe physical-name pushdown for renamed/re-added columns.
• Preserve physical row positions for deletion vectors and indexed PK ranges: postpone file filtering until position selection and merging, and carry the DV iterator position across batches. This prevents deleted rows reappearing and live rows being incorrectly dropped with small batches.

REST snapshot results, including empty results and SNAPSHOT 404, are authoritative. Permission and service errors (including HTTP 501) remain errors, matching Java. Query authorization, precomputed PK global-index results, continuous streaming and write planning retain their existing Python paths.

Tests

• Rebuilt the Rust dependency wheel; 489 native/planner/schema/reader tests passed, exercising 642 native plans. Newly enabled paths fail explicitly if planning falls back to Python.
• 244 tests passed without the Rust runtime in the normal Python lane, including JDBC and REST catalogs, snapshot loading, resolving FileIO, predicates, deletion vectors and schema evolution.
• Coverage includes append/PK/DE schema rename and drop/re-add across filesystem/JDBC/REST catalogs, removed selectors/options, local paths and file URIs, REST empty/latest/branch snapshots and dotted names, service errors and expired-token refresh, clustered DV files and overlapping first-row L0 runs, chunk row counts/order/shards, aligned blob files, time travel, incremental events and indexes. Reader regressions run without Rust and cover raw/merge paths, indexed positions, deleted hits and batch sizes 1/1024.
• The previous Rust wheel incorrectly read filesystem snapshots on REST 403/404/500; the new error regressions fail against it. Materialized-DV packing and evolved-schema filtering also reproduced failures before their fixes.
• The new JDBC, dotted-identifier and L0 tests failed on the prior gates. The DV tests reproduced unsafe predicate pushdown and per-batch position resets before their fixes.
• Flake8 for all changed Python files and git diff --check passed.

[leaves12138]

Reviewed 0135158. I did not find a blocking correctness issue.

The review covered resolved schema/options propagation for filesystem/JDBC/REST and catalogless tables, preservation of REST snapshot/credential semantics, capability-based fallback, incremental/chunk-shuffle planning, and first-row/DV split boundaries. In particular, cross-schema filtering now runs after field-ID normalization, while DV and indexed PK reads preserve physical positions before residual filtering and merging.

Independent validation against this head:

• 441 targeted tests passed with a wheel built from paimon-rust #847 at efd99f5555a7f4b9e51457d17d3c1fb1c6c95b7f, exercising 627 native plans.
• 148 reader/schema/DV/FileIO/snapshot/JDBC tests passed without the Rust runtime.
• 8 additional temporary regression cases passed across Python/native planning and append/PK-raw/PK-merge/first-row readers, combining batch size 1, a deleted predicate hit, rename plus same-name re-add, projection excluding the filter column, and limit. Native cases explicitly rejected Python-planner fallback.
• Flake8 passed for all changed Python files.

Validation limits: a broader pure-Python REST run reached 244 passing tests, with 7 failures due to missing optional Lance/DuckDB dependencies, before I interrupted the subsequent Ray tests. This was not a complete REST-suite pass. At the final CI check, the regular Python lanes were green, while Python / Rust Plan was still building Rust main. The Rust #847 dependency is now merged.

Non-blocking performance follow-up: split_read.py:596 disables file-level predicate pushdown whenever schema IDs differ, including otherwise compatible changes such as an unrelated added column. This is a defensible correctness-first choice, but may increase read work on evolved tables. A later optimization could retain pushdown after validating the referenced field IDs/names/types; I did not benchmark the performance impact here.