Fix filter column projection in REST scan planning

[carlosjourdan]

Superseded by apache#4105, which proposes this branch directly to apache/iceberg-python:main. This fork-only PR is closed without merging.

Addresses apache#4104.

Rationale for this change

Server-side planning can return files containing only the requested columns and omit residual-filter. If the original filter references an unselected column, the client evaluates it against a missing value and silently returns no rows.

Include the filter's columns in the plan request. The existing output projection keeps the result limited to the caller's selected columns.

select x; filter y = 10
  plan requests x, y → client filters on y → result contains x

Are these changes tested?

• Before: both Parquet regression cases return [] instead of [{"x": 1}, {"x": 3}].
• After: both pass, covering servers that apply the filter and servers that leave it to the client. Request tests cover case sensitivity, nested fields, deduplication, wildcard selection, and current-schema binding with a historical snapshot.
• Python 3.13: make test PYTEST_ARGS='-q --tb=short': 4,271 passed, 5 skipped.
• make lint: passed.

CI: Python 3.10 is blocked by the existing Google warning-filter problem tracked in apache#4100 (failed job). A minimal import reproduces the collection error with both the base (b32392dc) and branch pytest configurations on Python 3.10.21, using the locked dependency versions; applying apache#4100's filter to the reproducer makes it pass. Linux 3.11–3.14 jobs were cancelled by fail-fast; Windows 3.12 passed. Linux CI still needs a successful rerun after the upstream fix lands.

Local performance measurements

Compared the upstream and patched request methods on the same Python 3.13 / PyArrow 25 environment, with request JSON serialization, a stub catalog, and nine interleaved timing blocks. On a 1,000-column schema:

Request preparation	Upstream	Patched
Warm schema, one predicate	6.7 µs	16.6 µs
Warm schema, ten predicates	17.6 µs	215 µs
First schema use, one predicate	7.3 µs	3.43 ms

Local million-row Parquet scans with identical correct results had medians around 14–15 ms on both versions. These measurements do not establish strict performance equivalence or measure live server processing and transfer costs. Unity Catalog has not been tested live.

Are there any user-facing changes?

Filtered server-planned scans now return the matching rows when filter columns were omitted from selected_fields. Returned columns are unchanged.

AI assistance: Codex was used for the implementation, regression tests, validation, and PR description. Validation limitations and uncertainties are documented above.