Skip to content

fix(test): parallel consistency test flakes on CI (Ubuntu + Alpine simultaneously) #1048

Description

@Chemaclass

Observed

On #1044, Ubuntu - parallel extended and Alpine - parallel extended failed together with an identical diff:

✗ Failed: Parallel does not lose same named files in different dirs
    Expected '2 passed 2 total 2 passed 2 total '
    but got  '0 total 0 total 2 passed 2 total 2 passed 2 total '

gh run rerun --failed on the same commit went green, so it is not deterministic — but two independent platforms failing in the same run, the same way, is not the shape of ordinary flakiness and is worth understanding rather than papering over with reruns.

What the diff says

tests/acceptance/bashunit_parallel_consistency_test.sh:51 scrapes counts with
grep -oE '[0-9]+ (passed|total)' over the whole captured output of a nested run. The parallel
side picked up an extra 0 total pair ahead of the real summary — i.e. the capture contains
the tail of a second bashunit run that found no tests ("Tests: 0 total" / "Assertions: 0 total").

So the question is where a zero-test summary comes from inside that capture. Two candidates,
neither confirmed:

  1. Output from a concurrently running acceptance test leaking into this one's capture under
    --parallel (the suite runs many nested ./bashunit invocations).
  2. The nested run genuinely selecting nothing on the first of its two invocations, under load.

Why the assertion cannot tell us

The test compares scraped counts, not structure, so any stray summary text anywhere in the
capture changes the result. It already carries a assert_contains "2 total" guard against both
sides collapsing vacuously — but that guard passes here, because the real counts are present;
they just have extra ones in front.

Suggested direction

  • Anchor the scrape to the summary block rather than the whole capture (e.g. take the last
    passed/total pair, or match the Tests: line specifically), so unrelated output cannot
    shift the comparison.
  • If leakage is the cause, that is worth knowing on its own: it would affect every acceptance
    test that captures a nested run's output.

Repro

Not reproducible locally: 3× targeted and 2× full ./bashunit --parallel tests/ on macOS all
green. Needs CI load, or --repeat against the acceptance suite.

Acceptance criteria

  • The cause is identified (leakage vs genuine empty selection), not just suppressed
  • The assertion is anchored so unrelated output cannot change its verdict
  • If output leakage between concurrent nested runs is real, it is filed separately

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

Status
No status

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions