Follow-ups from the 1.1.0 user guide review

[andygrove]

The 1.1.0 user guide review in #6168 left these items open. Some depend on other PRs, some need a decision, and some need a code change or a test rather than a doc edit.

Before the 1.1.0 branch cut

• Native Parquet scan multiplies rows for a struct with duplicate field names #5783: a Parquet struct with byte-identical child names silently multiplies rows. This is a regression from feat: support unicode case sensitive field names for reading parquet #5602, which is new in 1.1.0. fix: reject duplicate Parquet field names before decoding #5786 (approved, closes Native Parquet scan multiplies rows for a struct with duplicate field names #5783) rejects the ambiguous reads with a clear error. If it doesn't land before the cut, add Native Parquet scan multiplies rows for a struct with duplicate field names #5783 to the incorrect-results list in compatibility/scans.md.
• Iceberg transform residuals: a residual such as bucket(4, id) = 2 AND x is pushed to iceberg-rust as id = 2, which drops rows. This is also true in 1.0.0, and the fix is fix(iceberg): don't push transform residuals as their source column, fail on residual errors #6154. Once it lands, the transform-residual bullet under "Current limitations" in iceberg.md should say that a single transform predicate falls back, while a combined one stays native without pushing the residual. If it misses 1.1.0, document it as a known wrong-results issue instead.
• feat: remove memory accounting from Comet's on-heap mode #6066 removes three keys that shipped in 1.0.0: spark.comet.memoryOverhead, spark.comet.exec.onHeap.memoryPool and spark.comet.shuffle.jvm.memoryFactor. They're in the Testing category, but the versioning policy only allows removing a key in a major release and doesn't exempt testing keys. Either keep them as deprecated no-ops or clarify the policy.
• Check that the scheduled CI runs have been healthy over the release window, the other release preparation step.

Docs decisions

• Spark 4.2 has no expression-compatibility page. SPARK_PROFILES in docs/build.sh and dev/generate-release-docs.sh stops at 4.1, while compatibility/expressions/index.md tells readers to pick their version's page. The registrations match 4.1, but aggregates.scala has isSpark42Plus branches for collect_list / collect_set and mode, so a 4.2 page would differ.
• The links in expressions.md to compatibility/expressions/<category>.md (array, cast, datetime, map) don't resolve on the published site, because those pages only exist under spark-<ver>/ at build time. 1.0.0 has the same problem. Should they point at the Spark 4.1 pages, or at the index? The reverse row also links array.md, while its generated notes land in string.md.
• In understanding-comet-plans.md, the from_unixtime extended-explain example no longer falls back, since CometFromUnixTime routes the incompatible case through the codegen dispatcher. Replace it with real output from an expression that does fall back. The compatibility guide URL quoted in it is also stale.
• tuning.md "Remote Shuffle with Celeborn" describes the native push path in detail. However, CelebornReflectionCompatibilitySuite asserts that released Celeborn clients are refused native push, so today users only get ordinary Celeborn shuffle underneath Comet operators. Say so up front, or move the detail to the contributor guide.

Stale text generated from code

• The spark.comet.exec.scalaUDF.codegen.enabled description says the regex family routes through the dispatcher by default. In-subset rlike literals have run natively since feat: run rlike natively by default for Java-equivalent literal patterns #5415, and the description also understates that the flag gates every dispatch path.
• The spark.comet.memoryOverhead description still talks about additional memory allocated per executor. After fix: remove the ineffective spark.executor.memoryOverhead adjustment from the driver plugin #6054 it only sizes the native pool in on-heap mode.
• The TruncTimestamp non-UTC incompatibility reason in serde/datetime.scala cites date_trunc incorrect results in non-UTC timezone #2649, which was closed on 2026-06-30. Check whether the incompatibility still holds, then update or drop the reason. It renders on the generated datetime compatibility page.
• The Iceberg write toggles, spark.comet.write.iceberg.splitOperator.enabled and spark.comet.iceberg.write.enabled, are in the Testing category. So configs.md lists them as testing settings, even though iceberg-writes.md documents them for users.
• GenerateDocs Implementation column:
  • Builder-registered functions publish as "—". Examples are hour, minute, second, make_timestamp, mode, to_char and to_varchar.
  • regexp_extract and regexp_extract_all show Native, although they dispatch by default.
  • Each profile's pass rewrites expressions.md, so the last pass wins.
• Other GenerateDocs gaps:
  • The cast tables use Cast.canCast, so casts that Spark rejects under ANSI show U rather than N/A.
  • There is no hash, predicate or json category, so some notes are never published. For example, hash, xxhash64 and sha* fall back for decimal precision above 18 and for TIME.

Needs a test or a check before changing the docs

• compatibility/scans.md says decimals encoded in binary format fall back. There is no plan-time check for this, and arrow-rs reads BYTE_ARRAY decimals as Decimal128, so the claim is likely stale.
• iceberg.md says REST-vended credentials aren't consumed automatically. But the scan forwards FileIO.properties(), so static vended s3.* keys probably do reach the reader. It may be only refresh, or credentials delivered through storage-credentials, that are unsupported.
• expressions.md "Not currently planned": since feat: route unrecognized StaticInvoke and Invoke through the codegen dispatcher #5692, several of these should route through the codegen dispatcher rather than fall back. That covers is_valid_utf8, make_valid_utf8, validate_utf8, try_validate_utf8, quote, sentences and the scalar bitmap_* functions. Add a fixture before changing the text.
• These are probably supported but untested:
  • current_time, time_diff and time_trunc
  • the two-argument TIME forms of make_timestamp_ltz / make_timestamp_ntz
  • the datetime form of try_add
• Reported by the review, not yet confirmed: iceberg.md "Current limitations" may also be missing two fallbacks, equality deletes on struct or variant columns and fixed partition columns.
• datasources.md doesn't cover native Parquet scans of gs://, including which settings are honoured. It also doesn't explain fs.comet.libhdfs.schemes routing.

Other gaps that predate 1.1.0

• ANSI arithmetic overflow errors: wrong error class for Byte/Short, wrong type names, missing try_ suggestions #5071 was auto-closed by fix: skip null slots when checking overflow in unary negation #5162, but two of its items remain:

  • Byte / Short Add, Subtract and Multiply overflow raises ARITHMETIC_OVERFLOW instead of BINARY_ARITHMETIC_OVERFLOW.
  • Integral overflow messages omit Spark's try_ suggestion.

  Reopen it or file a follow-up, so the compatibility guide entry can link to it. Filed as ANSI integral overflow errors: wrong error class for Byte/Short arithmetic and missing try_ suggestion #6217.

• iceberg.md "Current limitations" doesn't mention several fallbacks:

  • Iceberg metadata tables
  • metadata columns other than _file, _pos, _spec_id and _partition
  • FileIO implementations outside the supported list

• mode is marked ✅, although every input falls back unless spark.comet.expression.Mode.allowIncompatible=true is set. That stretches the status legend.