fix: Joom 1.1-5 — nested booleans exported to the JVM with a non-zero bit offset (backport #6339)

[msafonov]

A boolean nested in a struct, list or map could come back to the JVM with the values and validity of neighbouring rows (null or flipped). In production this nulled nested booleans in 19 rows of mongo.finance_order_costs_daily_snapshot. The bug is upstream, not in our fork changes. Proposed version name: joom-1.1-5.

Cause

Comet exports batches to the JVM through the Arrow C Data Interface. arrow-rs folds a slice into the buffers for every Spark type except boolean, which keeps its bit offset in ArrayData::offset, and Arrow Java 18.3 ignores the imported offset (apache/arrow-java#88). The existing workaround in prepare_output (apache#2051) only checked the top level of each column; a struct around a sliced boolean has offset 0, so the nested boolean was read from bit 0.

Changes

• Backport of fix: zero sliced boolean offsets at every level before exporting to the JVM apache/datafusion-comet#6339 (closes Sliced booleans nested in structs, and sliced inputs to the JVM UDF bridge, reach the JVM misaligned and return wrong results apache/datafusion-comet#6288), cherry-picked from apache branch-1.1 (fix: [branch-1.1] zero sliced boolean offsets at every level before exporting to the JVM (#6339) apache/datafusion-comet#6449, 6680083). zero_offsets re-slices boolean bitmaps to start at bit 0 at every level, sharing every other buffer, and is applied in move_to_spark to every exported batch and decoded shuffle block (replacing the top-level take in prepare_output) and to the JVM UDF bridge inputs. The only conflict was the module list in native/common/src/lib.rs.
• Regression test for the production shape: a nullable boolean inside a nullable struct inside a struct, sliced by ORDER BY … LIMIT … OFFSET.

No upstream default is changed.

Tests

On joom-1.1-4 code the new tests fail and with this PR they pass:

test	joom-1.1-4	this PR
Rust test_move_to_spark_zeroes_nested_boolean_offsets	FAILED	ok
CometExecSuite: 3 upstream tests + nested struct-in-struct test	4 failed (wrong results)	4 passed
CometCodegenSuite: 3 upstream UDF-bridge tests	3 failed	3 passed

Locally on this PR: cargo fmt, clippy -D warnings, spotless and scalastyle clean; Rust workspace 1918 passed, 0 failed; CometExecSuite 155, CometCodegenSuite 103, CometSmjJoinFilterFuzzSuite default part 8, CometNativeShuffleSuite 70, CometFuzzTestSuite 45, CometArrayExpressionSuite 65, CometMapExpressionSuite 20, CometUdfBridgeSuite, CometShuffleReadCoalesceSuite, CometBlockStoreShuffleReaderSuite: all passed, 0 failed.

Cluster check

BoolSliceRepro (offset slice, sliced aggregate output, row_number dedup over a struct-in-struct boolean as in production), rows with a wrong nested boolean against the top-level copy of the same value; vanilla Spark gives 0 in all three:

query	joom-1.1-4	this PR
ORDER BY … LIMIT 40 OFFSET 17	12 / 40	0 / 40
GROUP BY over 100-row batches	695 / 2000	0 / 2000
row_number dedup, 20M rows	9783 / 4,948,454	0 / 4,948,454

🤖 Generated with Claude Code