chore: Codegen fallback explain for plan

[comphead]

Adds a new spark.comet.explainCodegen.enabled config (default false) that annotates each Spark expression routed through the JVM codegen dispatcher with a [COMET-INFO: <expr>: routes through JVM codegen dispatcher] message on the surrounding Comet operator. Also fills in the previously silent fallback branches inside CometScalaUDF.emitJvmCodegenDispatch so every None return now tags the expression via
withFallbackReason with an <expr>: <reason> prefix.

What changed

• spark/src/main/scala/org/apache/comet/CometConf.scala — new COMET_EXPLAIN_CODEGEN_ENABLED config in the CATEGORY_EXEC_EXPLAIN category, modeled after COMET_EXPLAIN_FALLBACK_ENABLED. Default
  false, so behavior is off unless opted in.
• spark/src/main/scala/org/apache/comet/serde/CometScalaUDF.scala
  • Every silent return None in emitJvmCodegenDispatch now calls withFallbackReason(expr, "<prettyName>: ...") first (payload/data-arg/return-type serialization failures).
  • Existing fallback branches (flag disabled, canHandle reject) get the same <prettyName>: prefix so multiple expressions in one projection each surface distinctly (Set[String] no longer collapses
    identical strings).
  • Success path emits withInfo(expr, "<prettyName>: routes through JVM codegen dispatcher") when the new flag is on. withInfo writes only EXTENSION_INFO, so no planning rule flips and the operator
    remains Comet-native. CometExecRule.rollUpInfoMessages propagates it to CometProjectExec / etc. for verbose explain rendering.
• spark/src/test/scala/org/apache/comet/CometCodegenSuite.scala — new test explainCodegen.enabled surfaces routed expressions in COMET-INFO runs SELECT hypot(a,b), levenshtein(s1,s2), asserts the
  plan is Comet-native (checkSparkAnswerAndOperator), and asserts both expressions surface in the [COMET-INFO: ...] block.
• docs/source/user-guide/latest/understanding-comet-plans.md — adds a row + section for the new config alongside the existing explainFallback / explain.format / explain.native configs, with an
  example showing the exact rendered output.

Example

spark.conf.set("spark.comet.exec.scalaUDF.codegen.enabled", "true")
spark.conf.set("spark.comet.explainCodegen.enabled", "true")

val df = spark.sql("SELECT hypot(a, b), levenshtein(s1, s2) FROM t")
println(new org.apache.comet.ExtendedExplainInfo()
  .generateExtendedInfo(df.queryExecution.executedPlan))

CometNativeColumnarToRow
+- CometProject [COMET-INFO: hypot: routes through JVM codegen dispatcher, levenshtein: routes through JVM codegen dispatcher]
   +- CometNativeScan parquet spark_catalog.default.t

Comet accelerated 2 out of 2 eligible operators (100%). Final plan contains 1 transitions between Spark and Comet.

Why

Previously, expressions running through the JVM codegen dispatcher were indistinguishable from natively-executed ones in explain output. Users investigating perf / native coverage had no way to tell which
sub-expressions were running Spark's own doGenCode inside a Comet kernel vs a true DataFusion implementation. And when the dispatcher declined (return-type unsupported, closure-serialize failed, etc.), the
plan silently fell back to Spark with no annotation on the expression.

[andygrove]

Is it possible to combine the list of expressions to avoid duplication e.g. hypot and Levenshtein route through the JVM codegen dispatcher ?

[parthchandra]

Why make this configurable? Don't we always want this?
Also, do we have any place in the documentation that explains (to the end user) what the phrase routes through JVM codegen dispatcher means.

[andygrove]

Why make this configurable? Don't we always want this?

This new tagging seems extremely verbose and seems more like a debugging tool. I don't think end users need to know about internal implementation details as long as Comet is executing the expression on the Arrow-native path.

Also, do we have any place in the documentation that explains (to the end user) what the phrase routes through JVM codegen dispatcher means.

We should explain this in https://datafusion.apache.org/comet/user-guide/0.17/understanding-comet-plans.html

[andygrove]

I was thinking about another approach for this. I haven't explored how practical it is to implement though.

The extended explain plan just shows operators, but what if we include expressions and use a symbol to annotate whether the expression is rust or jvm?

For example, here * could mean Rust and + could mean JVM.

CometProject: expr1*, expr2* expr3+, expr4+
  CometFilter: expr*

[parthchandra]

I was thinking about another approach for this. I haven't explored how practical it is to implement though.

The extended explain plan just shows operators, but what if we include expressions and use a symbol to annotate whether the expression is rust or jvm?

For example, here * could mean Rust and + could mean JVM.

CometProject: expr1*, expr2* expr3+, expr4+
  CometFilter: expr*

The way to do it, I think, would be to have a new tag for the expression nodes and then the explain plan generator can format it however we want it to be displayed.

[comphead]

The extended explain plan just shows operators, but what if we include expressions and use a symbol to annotate whether the expression is rust or jvm?

Maybe lets have at least possibility to recognize fallback codegen using this tool and we can improve it in follow up PR?
Spark uses ! to identify the operator has problems, we can use some other symbol yes

Why make this configurable? Don't we always want this?

Did it configurable so it won't bleed to golden files, WDYT would be better?