Rebuild Python UDF with typed revisions and Arrow Flight execution

[iamlinjunhong]

What type of PR is this?

• API-change
• BUG
• Improvement
• Documentation
• Feature
• Test and CI
• Code Refactoring

Which issue(s) this PR fixes:

Related to #28132. This delivers the external unisolated adapter stage; it does not close the issue's broader sandbox/production-isolation requirements.

What this PR does / why we need it:

Replace the demo's per-value protobuf transport with typed, revision-bound Python UDF execution over Arrow Flight. CN remains MO-vector-native and owns guarded expression evaluation; user code runs in external handler processes.

• Share immutable Catalog revisions, complete type descriptors and exact RoutineCall identities with SQL UDF. Validate syntax/handler contracts at CREATE/REPLACE, publish digest-addressed artifacts, and invalidate cached/prepared calls when the full overload namespace changes. DROP distinguishes exact decimal scales.
• Execute SCALAR and VECTOR handlers through ExternalRoutineEval, preserving CASE selection, NULL policy, zero-row/zero-argument calls, nested evaluation and DML error atomicity.
• Bound Gateway admission, stream windows (W=1), handler slots and terminal records. Close EndInput/result ACK/Finish semantics, independent cancellation, generation fencing and retryable cleanup after partial initialization. Container init reaps orphaned descendants.
• Enable the worker in the existing ordinary CI/BVT path, with one worker endpoint per CN in the multi-CN launch. Add correctness cases for Arrow compute, revision/overload changes and cross-account snapshot restore. Remove the old demo execution path.

Implementation and rollout contract: docs/design/python_udf.md. Namespace validation: docs/design/routine_namespace_validation.md. Local usage and cases: Python BVT README.

Current CI repair validation

The CI repair fixes the Arrow import ownership checks, preserves existing UDF SQL error codes/messages while adopting repository error wrapping, installs timezone data in both CI runtime image variants, and aligns the Python local launch FileServices with TN. Catalog result fixtures now assert the added revision table and six identity columns, including cross-account snapshot restore; Python error/result oracles remain unchanged. Real-worker test cleanup owns the process before readiness and records completion before repeated cleanup.

• macOS arm64: fresh native and service builds; owning frontend/colexec/planner/function/CN tests; UDF/protocol race tests and six real-worker race tests; SQL diagnostic regression controls.
• Fresh real SQL → Gateway → Flight → handler → MO result: 32 unique affected BVT scripts, 4004 passed, 0 failed, 22 pre-existing ignored (4026 total). Python subset 352/352. One script ran in both batches and is counted once. This current SQL run is single-CN macOS, not a claimed new Linux multi-CN run.
• Full affected-package lint: 0 issues; make err-check; CI actionlint; Python 3.12 coverage helper tests 6 passed.
• Actual CI prebuilt runtime image assembly and Python worker image both contain tzdata 2026c. The original runtime base had no timezone data. Runtime assembly validation used a cached executable; current SQL used the freshly built source binary.
• Linux current-source Go/Flight/race result: Linux amd64: final-source ordinary UDF/real-Flight and import-boundary tests passed; protocol race passed. Linux arm64 (local Docker, Go 1.26.4, Python 3.12/PyArrow 24): full Python package race 81 top-level tests passed, including all six real-worker tests, with no skips. The interrupted amd64 Python race is not counted as passed.

Existing host UT and coverage workflows use matrixorigin/CI#457 to install the checked-out worker requirements in a Python 3.12 venv and expose that interpreter to Go tests. CI #457 merged into matrixorigin/CI main at c1a30c45dfce0352d1d4fb9573dda2a0feedb4d9; this changes existing jobs only. At the pre-rebase head c560dc168944c6ed158c8db36284173c38d255b1, MatrixOne ALL CI run 35810950106 had not reached a terminal result at the last recorded check. It does not validate the rebased head. Per user request, hosted CI was not awaited; this PR update will trigger a fresh run, and no hosted result is claimed.

Latest-main rebase and current local validation

The branch was rebased onto main at ca71f51e2f962c0fbe0a92d9d01bb071bb1bb9c9 after nine new main commits made the previous PR head conflict. The merged MySQL grammar was regenerated with the repository goyacc target and its full parser package passed. Current-head validation also passed: Python UDF race, protocol race, full frontend, CN service and colexec packages; go vet; incremental golangci-lint (0 issues); and git diff --check. The official CI coverage parser, using the latest-main PR diff plus CI UT/BVT profiles and current local package profiles, reports 2,727/3,626 = 75.21%, strictly above the 75% gate. Hosted CI was not awaited, as requested.

Prior Linux integration evidence

Validated on Linux amd64 with the repository CI tester image, Go 1.26.4, Python 3.13.5 and PyArrow 24.0.0. Source included main merge 550ca2830d and repair 37889f08b7; these results predate the CI repair above and are retained as earlier deployment/semantic evidence.

• Native build and repository CGo-wrapper tests for UDF/protocol/Gateway; UDF packages with -race.
• Owning colexec, function, rule, MySQL parser and v4_0_7 upgrade packages; focused frontend Catalog/drop/revision/restore tests.
• Python worker suite: 142 passed, 29 subtests. The subsequently changed bounded orphan-reap assertion passed separately.
• Real SQL → CN Gateway → Flight → handler → MO result: Python BVT 352/352 on each of two CNs; SQL UDF 82/82; external-table clone 108/108, with no skips. Uses the repository CI's nometa mode.
• Cross-account snapshot restore executes revision A in a fresh target after replacing the source with B; checks exact restored identities/revisions/bodies, SQL UDF control and cleanup.
• Linux Supervisor worker SIGKILL: SQL error, handler/descendant reaping, no transparent replay, ordinary SQL preserved; replacement worker/lease executes a fresh call successfully.
• Actual worker image: three real Flight leader-exit cases leave no owned descendant PID; PID 1 is tini. Companion Operator command/render tests preserve init and loopback binding.

GPT-6 medium design and integrated mo-self-review covered the branch and corrected exact-DROP, partial-initialization ownership and orphan-reaping findings. Hosted CI is pending; local/remote test passes are not reported as hosted CI results.

Compatibility and remaining scope

Python is disabled in generic launch and requires explicit unisolated opt-in. Legacy demo definitions/plans are rejected and must be explicitly recreated; existing SQL UDFs remain supported. Worker/Gateway capability and tzdata/environment digests must match before execution. Operator integrations must use an init-capable worker image and preserve its init command.

This does not provide sandbox tenant isolation or authenticated/TLS Flight, imported dependency environments, W>1/cumulative ACK, or full production Operator rollout acceptance. Native Linux dual-CN tests and image tests do not claim all deployment topologies were exercised. Prior optional strict-metadata SQL checking found six header differences; those expectations were not changed to hide them, and this PR reports the existing CI comparison mode explicitly.

[qodo-code-review]

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

[aptend]

Review at exact head 4ca2294 (re-review of 92a89d8).

[P1] The design-approval gate is still open. In docs/design/python_udf_current_stage_approval.md:45-51, the new approval ledger explicitly leaves independent Architecture and SQL/Planner approval of this exact current-stage contract pending; Security and Cloud/Operator approval also remain pending for broader enablement. The only recorded acceptance is by the feature owner for a development/test adapter. Issue #28132 requires formal cross-subsystem review of a stable versioned design before production implementation begins. This PR writes persistent catalog/revision and SQL/planner semantics into main, even though rollout is opt-in. For a concrete counterexample, if overload or restore semantics in this implementation disagree with the eventual SQL/Planner decision, test-stage databases can already persist revisions under the old contract and need an incompatible migration; the opt-in label does not undo that persisted state. The new document accurately records the missing approvals but does not close this gate. Please obtain and link decisions on the exact design revision from the relevant Architecture and SQL/Planner owners (and keep the Security/Operator production limits explicit) before this implementation is merged.

I inspected the old review and the rebased incremental change, including the new approval record, against the full PR design/scope. This is a design-gate finding, not a claim that a specific runtime failure was reproduced. Exact-head required CI is also currently red in coverage; I did not attribute those failures to this PR.