diff --git a/.github/workflows/util/delta-spark-ut/README.md b/.github/workflows/util/delta-spark-ut/README.md index b3119d7d39..86bceb6740 100644 --- a/.github/workflows/util/delta-spark-ut/README.md +++ b/.github/workflows/util/delta-spark-ut/README.md @@ -176,10 +176,10 @@ and blank lines allowed: ### Quarantine by error signature Some bugs surface on a **different test each run** — for example the native Delta -DV bitmap row-index error (the aggregator gets a garbage row index during a MERGE -that writes deletion vectors and aborts, e.g. `Delta RoaringBitmapArray row index +DV bitmap row-index error (the aggregator got a garbage row index during a MERGE +that writes deletion vectors and aborted, e.g. `Delta RoaringBitmapArray row index ... exceeds max representable value` or `Delta bitmap row index cannot be -negative: ...`) lands on a different `*DVs*Suite` MERGE test every time. Chasing +negative: ...`) landed on a different `*DVs*Suite` MERGE test every time. Chasing those by name is whack-a-mole, so quarantine them by **root cause** in **`flaky-error-patterns.txt`** instead: each line is a regex matched against a failed test's ``/`` text. Any failure that matches is treated as @@ -187,12 +187,21 @@ flaky regardless of which test it hit (and is dropped from the shard's failures list so it can't leak into the baseline): ``` +# HISTORICAL EXAMPLE -- no longer active; flaky-error-patterns.txt has no +# entries. Do not copy these back in: the bug they matched is fixed. # regex matched against the failure message + stack (enforce mode). # one explicit pattern per known error, deliberately specific. Delta RoaringBitmapArray row index \d+ exceeds max representable value -Delta bitmap row index cannot be negative: -?\d+ +Delta bitmap row index cannot be negative: -\d+ ``` +The root cause was a Velox scan bug +([velox#18535](https://github.com/facebookincubator/velox/issues/18535)), fixed +by [velox#18536](https://github.com/facebookincubator/velox/pull/18536) and +picked up by the `dft-2026_08_21` Velox pin, so both patterns were removed and +the suite is enforced again. The example is kept because it shows the shape of +the mechanism. + This is more precise than a name glob: a *different* real failure in the same suite is still caught, because only failures carrying the signature are ignored. diff --git a/.github/workflows/util/delta-spark-ut/flaky-error-patterns.txt b/.github/workflows/util/delta-spark-ut/flaky-error-patterns.txt index dcac80e3dc..f366726879 100644 --- a/.github/workflows/util/delta-spark-ut/flaky-error-patterns.txt +++ b/.github/workflows/util/delta-spark-ut/flaky-error-patterns.txt @@ -10,37 +10,5 @@ # surfaces on a different test each run, so matching by test name is # whack-a-mole. Prefer fixing the underlying bug and REMOVING the entry. # -# --------------------------------------------------------------------------- -# Native Delta bitmap aggregator receives an INVALID row index during a MERGE -# that writes deletion vectors, and aborts. The garbage row index trips one of -# two bounds checks, so the same root cause shows up with two messages: -# -# too large (Long.MAX_VALUE): -# VeloxRuntimeError INVALID_STATE -# Reason: Delta RoaringBitmapArray row index 9223372036854775807 exceeds max -# representable value 9223372030412324864 -# Expression: value <= kMaxRepresentableValue -# Function: addSafe File: .../velox/compute/delta/RoaringBitmapArray.cpp:92 -# -# negative (garbage): -# VeloxRuntimeError INVALID_STATE -# Reason: Delta bitmap row index cannot be negative: -6254810385378525259 -# Expression: value >= 0 -# Function: addRowIndex File: .../operators/functions/delta/DeltaBitmapAggregator.cc:44 -# -# Both are the same root cause (garbage row index into the DV bitmap aggregator) -# but distinct native errors, so each has its own explicit pattern below -- -# deliberately specific (bound to the exact error string) rather than a broad -# "Delta bitmap row index" match, to avoid masking an unrelated failure. Add a -# new line if a further bounds-check variant appears. Intermittent (depends on -# the runtime plan/scan/scheduling), so it hits a different *DVs*Suite MERGE test -# on each run. Remove these once the row-index materialization is fixed in the -# native backend. Tracked upstream (DV bitmap invalid row index). -# --------------------------------------------------------------------------- -# too-large (Long.MAX_VALUE) -- RoaringBitmapArray.cpp addSafe, value <= kMaxRepresentableValue -Delta RoaringBitmapArray row index \d+ exceeds max representable value -# negative garbage -- DeltaBitmapAggregator.cc addRowIndex, value >= 0 -# The check is `value >= 0`, so the reported index is always negative: match the -# sign explicitly rather than `-?` so this can't quarantine an unrelated failure -# if the message format ever changes. -Delta bitmap row index cannot be negative: -\d+ +# There are currently no quarantined error signatures. Add one below when a +# nondeterministic bug starts landing on a different test each run. diff --git a/.github/workflows/util/delta-spark-ut/flaky-tests.txt b/.github/workflows/util/delta-spark-ut/flaky-tests.txt index e42348f0b0..e06a65b2bd 100644 --- a/.github/workflows/util/delta-spark-ut/flaky-tests.txt +++ b/.github/workflows/util/delta-spark-ut/flaky-tests.txt @@ -18,7 +18,5 @@ # # NOTE: when a bug surfaces on a DIFFERENT test each run (so matching by name is # whack-a-mole), quarantine it by ERROR SIGNATURE in flaky-error-patterns.txt -# instead. The native Delta DV bitmap row-index bug (RoaringBitmapArray -# Long.MAX_VALUE) is handled there, which is why no `*DVs*Suite` MERGE entries -# are listed below. +# instead.