Add agent instructions, plan and ADR process

[pang-wu]

What does this PR do?

Adds AGENTS.md — the instruction file that agent harnesses load automatically — plus CLAUDE.md pointing at it so Claude Code reads the same content. It also introduces a plan and ADR process under doc/plan/ and doc/adr/, each with a template.

AGENTS.md mostly writes down what is already true about this repo, so an agent does not have to rediscover it each session:

• The Maven root is core/, not the repo root, and -am is required when building a single module.
• The shim architecture: version-specific code goes behind SparkShimLoader, never Spark-version if branches in raydp-main.
• The gates: ./build.sh (which runs JVM tests via mvn verify in CI) and pytest python/raydp/tests/, across the Python × Spark × Ray matrix.
• The style checks that actually fail CI: scalastyle's 100-column limit and license-header check, checkstyle, and pylint.

Two conventions are new rather than descriptive:

A checklist for adding Spark version support. Shim patch ranges, the <sparkNMM.version> property, the pyspark upper bound in python/setup.py, the Scala version, and the CI matrix all have to move together. A partial job compiles cleanly and then fails at shim resolution or lets pip install an unsupported PySpark. The checklist also distinguishes a new patch (widen SUPPORTED_PATCHES) from a new minor (new shim module plus one matrix entry), and says explicitly that patch versions do not belong in the matrix.

A design, testing, plan, and ADR bar. doc/plan/ and doc/adr/ are treated as historical records: a merged plan or ADR is superseded by a new document rather than rewritten, so the decision trail stays auditable. ADRs land before the plan that schedules the work, so a plan never depends on an unrecorded decision.

Why is this change needed?

Agent-assisted contributions are already landing here, and the context an agent needs is currently spread across pom.xml files, build.sh, and CI workflow YAML. Several things are genuinely non-obvious and easy to get wrong:

• Building raydp-main without -am fails, because it depends on the shim modules.
• The pyspark cap in setup.py exists to match shim coverage — raising it too far silently allows an install with no shim.
• maven-surefire-plugin before 2.22 has no JUnit Platform provider and auto-selects the TestNG provider instead, discovering zero JUnit 5 tests while leaving the build green. Lowering that version would silently disable the JVM test lane.
• The Scala incremental compiler can report "Nothing to compile - all classes are up to date" while target/classes holds only the Java classes, producing bogus "not found: type" errors until a clean build.

Writing these down is cheap and stops each of them from being rediscovered the hard way.

Related Issue

Fixes #

Type of Change

• Bug fix
• New feature
• Documentation update
• Test update
• Refactoring
• Build / CI / dependency change
• Other

How was this tested?

Documentation only — no code, build, or CI configuration changes, so no behavior to test.

Every path, filename, plugin version, and configuration value referenced in the new files was verified against the tree at 01ca3bf, including:

grep -n "pyspark >=" python/setup.py                    # 4.0.0 <= x <= 4.1.1
grep -n "SUPPORTED_PATCHES" core/shims/spark410/src/main/scala/com/intel/raydp/shims/SparkShimProvider.scala
grep -n "spark400.version\|spark410.version\|scala.version" core/pom.xml
grep -n "spark-version:" .github/workflows/raydp.yml
ls core/shims/spark410/src/main/resources/META-INF/services/

Internal markdown links between AGENTS.md, doc/plan/, and doc/adr/ were checked to resolve.

Checklist

• I have added or updated tests if needed
• I have updated documentation if needed
• I have run relevant tests locally
• I have checked that this change is backward compatible

[carsonwang]

can we add the ADR/plan template/readme only when we write the first actual ADR/plan so it will be much clear?
We can merge the AGENTS.md and CLAUDE.md first. As #490 was merged, need some updates like Spark 4.2 info.

[pang-wu]

can we add the ADR/plan template/readme only when we write the first actual ADR/plan so it will be much clear? We can merge the AGENTS.md and CLAUDE.md first. As #490 was merged, need some updates like Spark 4.2 info.

But when someone is trying to create the plan they will need a template first? I don't quite follow the concern here though

[carsonwang]

can we add the ADR/plan template/readme only when we write the first actual ADR/plan so it will be much clear? We can merge the AGENTS.md and CLAUDE.md first. As #490 was merged, need some updates like Spark 4.2 info.

But when someone is trying to create the plan they will need a template first? I don't quite follow the concern here though

These plan/ADR process and templates do not look like standards. I don't see them widely adopted in other popular projects. It remains a questions if we should adopt it. Different people might have different preferences whiling developing with agents. can we only include minimal and necessary info in agents.md and keep other templates and docs in your local for now?