Skip to content

test: harden stress_no_fd_leak against shared-runner fd churn#575

Open
enricopiovesan wants to merge 2 commits into
mainfrom
fix/stress-no-fd-leak-flake
Open

test: harden stress_no_fd_leak against shared-runner fd churn#575
enricopiovesan wants to merge 2 commits into
mainfrom
fix/stress-no-fd-leak-flake

Conversation

@enricopiovesan

Copy link
Copy Markdown
Collaborator

Summary

Harden the flaky stress_no_fd_leak stress test so it no longer fails on unrelated descriptor churn on shared CI runners (it failed on docs-only PR #572 on x86_64-unknown-linux-gnu, then passed on re-run), while keeping it a meaningful leak detector.

Governing Spec

  • 047-thread-pool-executor

Acceptance scenario 5.2 of spec 047 ("Given repeated executor create-use-drop cycles, When resource checks run, Then no file descriptor or handle leak is detected") still holds: the hardened detector fires on any genuine per-cycle leak (verified by injecting one leaked descriptor per cycle, which fails the test).

Project Item

What Changed

  • Contracts changed: none.
  • Runtime behavior changed: none — test-only change in crates/traverse-runtime/tests/thread_pool_stress.rs.
  • Compatibility impact: none.
  • ADR needed or linked: no.

Test hardening details:

  • 50 warm-up create-use-drop cycles run before the baseline is captured, so lazy process-wide initialization (stdio, threading, platform probes) opens its descriptors outside the measured window.
  • Each measurement is the minimum of 5 samples 100ms apart, filtering descriptors that unrelated runner activity opens transiently.
  • The assertion is growth-based with a fixed tolerance of 8 descriptors over 1,000 measured cycles, replacing the noise-sensitive absolute after <= before check. A real per-cycle leak produces growth of ~1,000 descriptors, so the tolerance cannot mask one.
  • Up to 3 re-measurements (500ms apart) run before the assertion fails, letting transient churn settle.

Definition of Done

  • stress_no_fd_leak uses warm-up baseline + growth-with-tolerance assertion + retried measurement
  • Any real per-cycle fd leak still fails the test (validated by temporarily injecting std::mem::forget(File::open(...)) per cycle — test failed as expected)
  • Full stress test suite passes locally

Validation

  • Spec alignment checked (BASE_SHA=origin/main bash scripts/ci/spec_alignment_check.sh on this body)
  • Contract alignment checked (no contract surface touched)
  • Tests updated and passing (cargo test -p traverse-runtime --test thread_pool_stress -- --ignored --nocapture: 9 passed, 0 failed)
  • Core coverage preserved (test-only change; no runtime code touched)
  • Required validation gates passing
  • cargo fmt --check and cargo clippy -p traverse-runtime --tests clean
  • Leak-detector sensitivity re-verified: injected 1 fd leak per cycle → test fails; removed → test passes

Notes

Fixes the flake reported in #574. The tolerance (8 fds) is deliberately small relative to the 1,000-cycle window so the test cannot be silenced by threshold inflation — any per-cycle leak exceeds it by two orders of magnitude.

enricopiovesan and others added 2 commits July 7, 2026 18:51
The absolute after<=before assertion failed on unrelated descriptor
activity on shared CI runners (docs-only PR #572). Take the baseline
after warm-up cycles, measure as the minimum of several samples,
assert on growth with a small fixed tolerance, and retry the
measurement before failing. A real per-cycle leak over 1,000 cycles
still exceeds the tolerance by two orders of magnitude.

Closes #574

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky stress_no_fd_leak on shared CI runners: absolute fd-count assertion is noise-sensitive

1 participant