[SPARK-59364][UDF] Build external UDF payload and Init messages

[haiyangsun-db]

What changes were proposed in this pull request?

This PR adds the session initialization metadata needed by the language-neutral external UDF
framework:

• Extend worker specifications with declarations for static and named dynamic configuration,
  authorized environment variables, and required resource directories.
• Build complete Init messages from ExternalUserDefinedFunction, including logical input
  metadata, named arguments, invocation identity, payload format, and evaluation type.
• Add a versioned PySpark UDF payload and a bridge from scalar Arrow PythonUDF expressions.
• Reuse the Arrow Python runner's configuration declarations when constructing Python worker
  specifications.

The existing Python UDF execution path is not switched to unified execution by this PR.

Why are the changes needed?

The external UDF framework can describe and launch workers, but it does not yet have a generic
contract for the per-session engine context or invocation metadata a worker needs. Without that
contract, physical execution would need language-specific knowledge, and Catalyst child rewrites
could leave serialized input metadata stale.

These changes keep worker requirements declarative, forward only engine-authorized context, and
rebuild logical input metadata from the current expression children.

Does this PR introduce any user-facing change?

No. This extends unreleased external UDF infrastructure and does not change the execution path of
existing Python UDFs.

How was this patch tested?

Added PythonExternalUserDefinedFunctionSuite coverage for context selection, required and
optional dynamic configuration, conflicting targets, named resource directories, rewritten and
named input metadata, Python worker configuration declarations, unsupported evaluation types, and
versioned payload decoding.

The protocol module and SQL tests were compiled, then the focused suites were run with:

build/sbt -java-home /usr/lib/jvm/java-17-openjdk-amd64 \
  udf-worker-proto/compile \
  sql/Test/compile \
  'sql/testOnly org.apache.spark.sql.execution.externalUDF.PythonExternalUserDefinedFunctionSuite org.apache.spark.sql.execution.externalUDF.PythonUDFWorkerSpecificationSuite org.apache.spark.sql.execution.externalUDF.PlanExternalUDFsSuite'

All 39 executed tests passed. One environment-gated Python worker test was canceled because
PySpark was unavailable in the local Python environment.

Was this patch authored or co-authored using generative AI tooling?

Yes

[Yicong-Huang]

@haiyangsun-db please let me know once this PR is ready for review