[INFRA] Distinguish source and test modules in test selection

[zhengruifeng]

What changes were proposed in this pull request?

This PR separates source ownership from test ownership in
dev/sparktestsupport/modules.py. Source and test nodes are declared explicitly as
SourceModule and TestModule instances, and each node lists the files it owns. The test nodes
retain the existing CI module names and runnable test goals.

A test module can depend on both source modules and other test modules. Source changes continue to
propagate through the source dependency graph, while test changes propagate only through explicit
test-to-test dependencies. JVM files under src/test belong to test nodes, and PySpark test files
belong to the test node that lists them in python_test_goals.

Why are the changes needed?

The previous module graph used the same node for source code and test code. As a result, changing a
test was treated like changing the module's source and triggered every dependent module. For
example, #58883 changes only a Spark Connect Python test, but it triggered all pandas API on Spark
test shards because those modules depend on PySpark SQL and Spark Connect source code.

Explicit source and test nodes avoid this unnecessary CI fan-out while preserving dependencies
between tests that reuse test code, including Scala test dependencies.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Added doctests covering explicit mixed source/test dependencies, Spark Connect Python test
ownership, and Scala test ownership. Also ran:

PYTHONPATH=dev conda run -n spark-dev-313 python -m doctest dev/sparktestsupport/modules.py
PYTHONPATH=dev conda run -n spark-dev-313 python -m doctest dev/sparktestsupport/utils.py
conda run -n spark-dev-313 dev/lint-python --ruff

Representative Scala source, Scala test, Scala test-resource, and PySpark test paths were also
checked directly against determine_modules_for_files and determine_modules_to_test.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: OpenAI Codex (GPT-5)

[nchammas]

Nice. This is right on topic for SPARK-59180.

@zhengruifeng - Are you interested in this topic beyond this specific PR? I have a lot of work planned in this area and would love to have someone to collaborate with.

[zhengruifeng]

@nchammas nice plan, we also have some discussion in the dev maillist and @gaogaotiantian had make the post-merge python test more smarter