From 248abe036ca14939181362bd6f713afa5b50b1a4 Mon Sep 17 00:00:00 2001 From: Felipe Pessoto Date: Thu, 20 Aug 2026 01:08:29 +0000 Subject: [PATCH 1/3] [GLUTEN-12377][CI] Stop quarantining the Delta DV row-index failures The native Delta bitmap aggregator intermittently aborted during a MERGE writing deletion vectors, with a garbage row index. Because it landed on a different *DVs*Suite test each run, it was quarantined by error signature rather than by test name. The root cause was not in the aggregator. It was a Velox scan defect: with no child readers, a filter and no deletion, 'outputRows()' was empty while the result carried numValues rows, so the row-index child came back with zero rows and downstream reads ran off the end of a zero-length buffer. Fixed upstream in facebookincubator/velox#18536, so the aggregator now receives real row indexes and the suite can be enforced again. Remove both patterns and the two cross-references that named them. The README keeps the example, marked historical, since it documents the mechanism. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../workflows/util/delta-spark-ut/README.md | 11 ++++-- .../delta-spark-ut/flaky-error-patterns.txt | 36 ++----------------- .../util/delta-spark-ut/flaky-tests.txt | 4 +-- 3 files changed, 11 insertions(+), 40 deletions(-) diff --git a/.github/workflows/util/delta-spark-ut/README.md b/.github/workflows/util/delta-spark-ut/README.md index b3119d7d399..4feba8ebfea 100644 --- a/.github/workflows/util/delta-spark-ut/README.md +++ b/.github/workflows/util/delta-spark-ut/README.md @@ -176,10 +176,10 @@ and blank lines allowed: ### Quarantine by error signature Some bugs surface on a **different test each run** — for example the native Delta -DV bitmap row-index error (the aggregator gets a garbage row index during a MERGE -that writes deletion vectors and aborts, e.g. `Delta RoaringBitmapArray row index +DV bitmap row-index error (the aggregator got a garbage row index during a MERGE +that writes deletion vectors and aborted, e.g. `Delta RoaringBitmapArray row index ... exceeds max representable value` or `Delta bitmap row index cannot be -negative: ...`) lands on a different `*DVs*Suite` MERGE test every time. Chasing +negative: ...`) landed on a different `*DVs*Suite` MERGE test every time. Chasing those by name is whack-a-mole, so quarantine them by **root cause** in **`flaky-error-patterns.txt`** instead: each line is a regex matched against a failed test's ``/`` text. Any failure that matches is treated as @@ -193,6 +193,11 @@ Delta RoaringBitmapArray row index \d+ exceeds max representable value Delta bitmap row index cannot be negative: -?\d+ ``` +That example is historical: the root cause was a Velox scan bug +([velox#18535](https://github.com/facebookincubator/velox/issues/18535)), fixed +upstream, so both patterns have since been removed and the suite is enforced +again. It is kept here because it shows the shape of the mechanism. + This is more precise than a name glob: a *different* real failure in the same suite is still caught, because only failures carrying the signature are ignored. diff --git a/.github/workflows/util/delta-spark-ut/flaky-error-patterns.txt b/.github/workflows/util/delta-spark-ut/flaky-error-patterns.txt index dcac80e3dc1..f366726879b 100644 --- a/.github/workflows/util/delta-spark-ut/flaky-error-patterns.txt +++ b/.github/workflows/util/delta-spark-ut/flaky-error-patterns.txt @@ -10,37 +10,5 @@ # surfaces on a different test each run, so matching by test name is # whack-a-mole. Prefer fixing the underlying bug and REMOVING the entry. # -# --------------------------------------------------------------------------- -# Native Delta bitmap aggregator receives an INVALID row index during a MERGE -# that writes deletion vectors, and aborts. The garbage row index trips one of -# two bounds checks, so the same root cause shows up with two messages: -# -# too large (Long.MAX_VALUE): -# VeloxRuntimeError INVALID_STATE -# Reason: Delta RoaringBitmapArray row index 9223372036854775807 exceeds max -# representable value 9223372030412324864 -# Expression: value <= kMaxRepresentableValue -# Function: addSafe File: .../velox/compute/delta/RoaringBitmapArray.cpp:92 -# -# negative (garbage): -# VeloxRuntimeError INVALID_STATE -# Reason: Delta bitmap row index cannot be negative: -6254810385378525259 -# Expression: value >= 0 -# Function: addRowIndex File: .../operators/functions/delta/DeltaBitmapAggregator.cc:44 -# -# Both are the same root cause (garbage row index into the DV bitmap aggregator) -# but distinct native errors, so each has its own explicit pattern below -- -# deliberately specific (bound to the exact error string) rather than a broad -# "Delta bitmap row index" match, to avoid masking an unrelated failure. Add a -# new line if a further bounds-check variant appears. Intermittent (depends on -# the runtime plan/scan/scheduling), so it hits a different *DVs*Suite MERGE test -# on each run. Remove these once the row-index materialization is fixed in the -# native backend. Tracked upstream (DV bitmap invalid row index). -# --------------------------------------------------------------------------- -# too-large (Long.MAX_VALUE) -- RoaringBitmapArray.cpp addSafe, value <= kMaxRepresentableValue -Delta RoaringBitmapArray row index \d+ exceeds max representable value -# negative garbage -- DeltaBitmapAggregator.cc addRowIndex, value >= 0 -# The check is `value >= 0`, so the reported index is always negative: match the -# sign explicitly rather than `-?` so this can't quarantine an unrelated failure -# if the message format ever changes. -Delta bitmap row index cannot be negative: -\d+ +# There are currently no quarantined error signatures. Add one below when a +# nondeterministic bug starts landing on a different test each run. diff --git a/.github/workflows/util/delta-spark-ut/flaky-tests.txt b/.github/workflows/util/delta-spark-ut/flaky-tests.txt index e42348f0b0d..e06a65b2bd9 100644 --- a/.github/workflows/util/delta-spark-ut/flaky-tests.txt +++ b/.github/workflows/util/delta-spark-ut/flaky-tests.txt @@ -18,7 +18,5 @@ # # NOTE: when a bug surfaces on a DIFFERENT test each run (so matching by name is # whack-a-mole), quarantine it by ERROR SIGNATURE in flaky-error-patterns.txt -# instead. The native Delta DV bitmap row-index bug (RoaringBitmapArray -# Long.MAX_VALUE) is handled there, which is why no `*DVs*Suite` MERGE entries -# are listed below. +# instead. From 9fee7d041c694c756aea60099b4f7e496ba1b765 Mon Sep 17 00:00:00 2001 From: Felipe Pessoto Date: Fri, 21 Aug 2026 23:56:45 +0000 Subject: [PATCH 2/3] [GLUTEN-12377][CI] Address review: correct the README regex example The example regex allowed an optional minus sign, which contradicted both the pattern it documents and the line above it saying patterns are deliberately specific. The file's real pattern was `-\d+`, with a comment explaining that the check is `value >= 0` so the index is always negative and the sign should be matched explicitly. Also name the Velox PR that fixed the root cause and the pin that picked it up, so the historical note says where the fix landed rather than just that it did. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/workflows/util/delta-spark-ut/README.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/.github/workflows/util/delta-spark-ut/README.md b/.github/workflows/util/delta-spark-ut/README.md index 4feba8ebfea..1a18fc4018a 100644 --- a/.github/workflows/util/delta-spark-ut/README.md +++ b/.github/workflows/util/delta-spark-ut/README.md @@ -190,13 +190,15 @@ list so it can't leak into the baseline): # regex matched against the failure message + stack (enforce mode). # one explicit pattern per known error, deliberately specific. Delta RoaringBitmapArray row index \d+ exceeds max representable value -Delta bitmap row index cannot be negative: -?\d+ +Delta bitmap row index cannot be negative: -\d+ ``` That example is historical: the root cause was a Velox scan bug ([velox#18535](https://github.com/facebookincubator/velox/issues/18535)), fixed -upstream, so both patterns have since been removed and the suite is enforced -again. It is kept here because it shows the shape of the mechanism. +by [velox#18536](https://github.com/facebookincubator/velox/pull/18536) and +picked up by the `dft-2026_08_21` Velox pin, so both patterns were removed and +the suite is enforced again. It is kept here because it shows the shape of the +mechanism. This is more precise than a name glob: a *different* real failure in the same suite is still caught, because only failures carrying the signature are ignored. From 45de5eb8d5f738f4771043292bf3d770da9fa692 Mon Sep 17 00:00:00 2001 From: Felipe Pessoto Date: Sat, 22 Aug 2026 00:08:45 +0000 Subject: [PATCH 3/3] [GLUTEN-12377][CI] Address review: mark the README example as historical The regex block sits above the paragraph explaining that the patterns are no longer active, so a reader skimming for config could mistake it for the current contents of flaky-error-patterns.txt and copy it back in. Mark it inside the block, where it cannot be missed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/workflows/util/delta-spark-ut/README.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/.github/workflows/util/delta-spark-ut/README.md b/.github/workflows/util/delta-spark-ut/README.md index 1a18fc4018a..86bceb67402 100644 --- a/.github/workflows/util/delta-spark-ut/README.md +++ b/.github/workflows/util/delta-spark-ut/README.md @@ -187,18 +187,20 @@ flaky regardless of which test it hit (and is dropped from the shard's failures list so it can't leak into the baseline): ``` +# HISTORICAL EXAMPLE -- no longer active; flaky-error-patterns.txt has no +# entries. Do not copy these back in: the bug they matched is fixed. # regex matched against the failure message + stack (enforce mode). # one explicit pattern per known error, deliberately specific. Delta RoaringBitmapArray row index \d+ exceeds max representable value Delta bitmap row index cannot be negative: -\d+ ``` -That example is historical: the root cause was a Velox scan bug +The root cause was a Velox scan bug ([velox#18535](https://github.com/facebookincubator/velox/issues/18535)), fixed by [velox#18536](https://github.com/facebookincubator/velox/pull/18536) and picked up by the `dft-2026_08_21` Velox pin, so both patterns were removed and -the suite is enforced again. It is kept here because it shows the shape of the -mechanism. +the suite is enforced again. The example is kept because it shows the shape of +the mechanism. This is more precise than a name glob: a *different* real failure in the same suite is still caught, because only failures carrying the signature are ignored.