[SPARK-59622][PYTHON] Migrate Arrow map/iter UDF eval types to the EvalTypeHandler pipeline - #58900

Closed

Yicong-Huang wants to merge 20 commits into

apache:masterfrom

Yicong-Huang:SPARK-59415-arrow-map-iter

Commits on Sep 17, 2026

refactor: migrate Arrow map/iter UDF eval types to eval handlers

Yicong-Huang
and
Isaac
committed
948a183
fix: address review - top-level pyarrow, no type-ignore, non-string annotations

Yicong-Huang
and
Isaac
committed
ea5295d
refactor: keep extract_key_value_indexes in worker_util, relax its import guard

Yicong-Huang
and
Isaac
committed
9e246fc

Commits on Sep 18, 2026

refactor: use have_pyarrow to guard the _arrow handler import

Yicong-Huang
and
Isaac
committed
ed98d60
refactor: drop _arrow_return_schema, reuse to_arrow_schema for grouped-map output

Yicong-Huang
and
Isaac
committed
6c21aef
refactor: order verification helpers and _arrow handlers alphabetically

Yicong-Huang
and
Isaac
committed
b96e1df
refactor: move extract_key_value_indexes to eval_handlers/_util, revert worker_util guard

Yicong-Huang
and
Isaac
committed
cec36aa
test: split handler tests into test_base_ and test_arrow_ eval type handlers

Yicong-Huang
and
Isaac
committed
e7c02b4
test: hoist pyarrow import to file level in test_arrow_eval_type_handlers

Yicong-Huang
and
Isaac
committed
897d22d
refactor: guard _arrow import with require_minimum_pyarrow_version, not the test helper

Yicong-Huang
and
Isaac
committed
ae5466c
refactor: rename eval_handlers/_util.py to _utils.py

Yicong-Huang
and
Isaac
committed
760a083
refactor: name the helper module eval_handlers/utils.py (no leading underscore)

Yicong-Huang
and
Isaac
committed
81080c8
test: unify the Arrow handler tests

Yicong-Huang
and
Isaac
committed
6b53587
test: move eval type handler tests under pyspark.sql.tests

Yicong-Huang
and
Isaac
committed
57612de
test: co-locate eval type handler tests in pyspark.eval_handlers.tests

Yicong-Huang
and
Isaac
committed
b8ac03b
test: group eval_handlers test goals together in modules.py

Yicong-Huang
and
Isaac
committed
21ccf36
test: order the eval_handlers test goals alphabetically

Yicong-Huang
and
Isaac
committed
21e65d2
test: list eval_handlers test goals before the pyspark.sql ones

Yicong-Huang
and
Isaac
committed
6824550
test: exempt pyspark.eval_handlers.tests from mypy like other test packages

Yicong-Huang
and
Isaac
committed
bce1fe7

Commits on Sep 21, 2026

refactor: register Arrow handlers without pyarrow, check version at run

Yicong-Huang
and
Isaac
committed
d200d85