Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 13 additions & 4 deletions .github/workflows/util/delta-spark-ut/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -176,23 +176,32 @@ and blank lines allowed:
### Quarantine by error signature

Some bugs surface on a **different test each run** — for example the native Delta
DV bitmap row-index error (the aggregator gets a garbage row index during a MERGE
that writes deletion vectors and aborts, e.g. `Delta RoaringBitmapArray row index
DV bitmap row-index error (the aggregator got a garbage row index during a MERGE
that writes deletion vectors and aborted, e.g. `Delta RoaringBitmapArray row index
... exceeds max representable value` or `Delta bitmap row index cannot be
negative: ...`) lands on a different `*DVs*Suite` MERGE test every time. Chasing
negative: ...`) landed on a different `*DVs*Suite` MERGE test every time. Chasing
those by name is whack-a-mole, so quarantine them by **root cause** in
**`flaky-error-patterns.txt`** instead: each line is a regex matched against a
failed test's `<failure>`/`<error>` text. Any failure that matches is treated as
flaky regardless of which test it hit (and is dropped from the shard's failures
list so it can't leak into the baseline):

```
# HISTORICAL EXAMPLE -- no longer active; flaky-error-patterns.txt has no
# entries. Do not copy these back in: the bug they matched is fixed.
# regex matched against the failure message + stack (enforce mode).
# one explicit pattern per known error, deliberately specific.
Delta RoaringBitmapArray row index \d+ exceeds max representable value
Delta bitmap row index cannot be negative: -?\d+
Delta bitmap row index cannot be negative: -\d+
Comment thread
felipepessoto marked this conversation as resolved.
```

The root cause was a Velox scan bug
([velox#18535](https://github.com/facebookincubator/velox/issues/18535)), fixed
by [velox#18536](https://github.com/facebookincubator/velox/pull/18536) and
picked up by the `dft-2026_08_21` Velox pin, so both patterns were removed and
the suite is enforced again. The example is kept because it shows the shape of
the mechanism.

This is more precise than a name glob: a *different* real failure in the same
suite is still caught, because only failures carrying the signature are ignored.

Expand Down
36 changes: 2 additions & 34 deletions .github/workflows/util/delta-spark-ut/flaky-error-patterns.txt
Original file line number Diff line number Diff line change
Expand Up @@ -10,37 +10,5 @@
# surfaces on a different test each run, so matching by test name is
# whack-a-mole. Prefer fixing the underlying bug and REMOVING the entry.
#
# ---------------------------------------------------------------------------
# Native Delta bitmap aggregator receives an INVALID row index during a MERGE
# that writes deletion vectors, and aborts. The garbage row index trips one of
# two bounds checks, so the same root cause shows up with two messages:
#
# too large (Long.MAX_VALUE):
# VeloxRuntimeError INVALID_STATE
# Reason: Delta RoaringBitmapArray row index 9223372036854775807 exceeds max
# representable value 9223372030412324864
# Expression: value <= kMaxRepresentableValue
# Function: addSafe File: .../velox/compute/delta/RoaringBitmapArray.cpp:92
#
# negative (garbage):
# VeloxRuntimeError INVALID_STATE
# Reason: Delta bitmap row index cannot be negative: -6254810385378525259
# Expression: value >= 0
# Function: addRowIndex File: .../operators/functions/delta/DeltaBitmapAggregator.cc:44
#
# Both are the same root cause (garbage row index into the DV bitmap aggregator)
# but distinct native errors, so each has its own explicit pattern below --
# deliberately specific (bound to the exact error string) rather than a broad
# "Delta bitmap row index" match, to avoid masking an unrelated failure. Add a
# new line if a further bounds-check variant appears. Intermittent (depends on
# the runtime plan/scan/scheduling), so it hits a different *DVs*Suite MERGE test
# on each run. Remove these once the row-index materialization is fixed in the
# native backend. Tracked upstream (DV bitmap invalid row index).
# ---------------------------------------------------------------------------
# too-large (Long.MAX_VALUE) -- RoaringBitmapArray.cpp addSafe, value <= kMaxRepresentableValue
Delta RoaringBitmapArray row index \d+ exceeds max representable value
# negative garbage -- DeltaBitmapAggregator.cc addRowIndex, value >= 0
# The check is `value >= 0`, so the reported index is always negative: match the
# sign explicitly rather than `-?` so this can't quarantine an unrelated failure
# if the message format ever changes.
Delta bitmap row index cannot be negative: -\d+
# There are currently no quarantined error signatures. Add one below when a
# nondeterministic bug starts landing on a different test each run.
4 changes: 1 addition & 3 deletions .github/workflows/util/delta-spark-ut/flaky-tests.txt
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,5 @@
#
# NOTE: when a bug surfaces on a DIFFERENT test each run (so matching by name is
# whack-a-mole), quarantine it by ERROR SIGNATURE in flaky-error-patterns.txt
# instead. The native Delta DV bitmap row-index bug (RoaringBitmapArray
# Long.MAX_VALUE) is handled there, which is why no `*DVs*Suite` MERGE entries
# are listed below.
# instead.

Loading