Skip to content

test: Warm PowerShell before Windows round trips - #380

Merged
leongdl merged 1 commit into
OpenJobDescription:mainlinefrom
leongdl:fix/pr379-flaky-test
Oct 16, 2026
Merged

leongdl merged 1 commit into
OpenJobDescription:mainlinefrom
leongdl:fix/pr379-flaky-test

Conversation

@leongdl

@leongdl leongdl commented Oct 15, 2026

Copy link
Copy Markdown
Contributor

Hold: do not merge until the 0.14.0 release (#379) has shipped.

Fixes: no issue filed. Flaky failure seen on #379, run 38812686088.

What was the problem/requirement? (What/Why)

TestReprPwshWindowsValidation in test/openjd/expr/test_strings.py fails intermittently on windows-latest with subprocess.TimeoutExpired after 10 seconds. On a fresh hosted runner, the first PowerShell launch has a one-time startup delay that can run past the 10-second per-call timeout in _pwsh_eval. Whichever test launches PowerShell first gets killed, and a killed launch can leave the next one cold as well.

Evidence:

  • Run 38812686088 on chore(release): 0.14.0 #379, a CHANGELOG-only change, lost the first four tests of the class at about 10.1 s each. The other nine tests passed.
  • The same timeout appears in 8 of the last 60 Code Quality runs, on Python 3.10, 3.12, 3.13 and 3.14. Every one of them includes test_pwsh_string_simple_roundtrip, the first test in the class.
  • Run 35389550086 passed on a rerun at the same SHA.

What was the solution? (How)

  • Add a class-scoped autouse fixture that pays the startup delay once, with a 120-second deadline. It goes through the same _pwsh_eval path the tests use: a .ps1 file run with -File that calls Write-Output. It asserts the output, so a broken PowerShell still fails the class instead of being skipped.
  • Mark the class @pytest.mark.timeout(180). pytest-timeout counts class-scoped fixture setup against the first test, and the project-wide --timeout=30 would otherwise cut a long warm-up short.
  • The per-call timeout stays at 10 seconds. No assertion changes, and there are no retries or skips.

What is the impact of this change?

Test-only. One file, 13 lines added. No change to src/.

How was this change tested?

  • Have you run the unit tests? Yes. hatch run test passes 6330 tests both before and after the change. hatch run lint and hatch run typing are clean.

The class is Windows-only and I developed this on macOS, so the evidence is a simulation, not a Windows reproduction. A shim in front of PowerShell 7 simulated a slow first launch:

Cold start Before fix After fix
12 s 1 failure (test_pwsh_string_simple_roundtrip) 13/13 pass
41 s 4 failures, the same four as run 38812686088, at 10.01 s each 13/13 pass

Each of these mutants fails the class under a 41 s cold start: no warm-up fixture, a 10 s warm-up deadline, and no timeout marker.

Only repeated green runs on real Windows runners prove the fix. The failure rate is roughly one in 50 Windows jobs, so a single green run is weak evidence. Please rerun the Code Quality Windows matrix a few times before merging.

Was this change documented?

  • Are relevant docstrings in the code base updated? N/A. A short comment on the fixture explains the warm-up.

Is this a breaking change?

No.

Does this change impact security?

No. It is a test-only change.


By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

TestReprPwshWindowsValidation fails intermittently on windows-latest
with subprocess.TimeoutExpired after 10 seconds. The first PowerShell
launch on a fresh hosted runner pays a one-time cold start that can
exceed the 10-second per-call timeout in _pwsh_eval, so whichever test
launches PowerShell first is killed, and a killed launch can leave the
next one cold too. Run 38812686088 (PR OpenJobDescription#379, a CHANGELOG-only change)
lost the first four tests of the class this way, each at about 10.1 s,
while the other nine passed. The same timeout appears in 8 of the last
60 Code Quality runs, on Python 3.10, 3.12, 3.13 and 3.14, and run
35389550086 passed on rerun at the same SHA.

Add a class-scoped autouse fixture that pays the cold start once, with
a 120-second deadline, through the same _pwsh_eval path the tests use:
a .ps1 file run with -File that calls Write-Output. It asserts the
output, so a broken PowerShell still fails the class instead of being
skipped. The per-call timeout stays at 10 seconds and no test
assertion changes. Mark the class timeout(180), because pytest-timeout
counts class-scoped fixture setup against the first test and the
project-wide --timeout=30 would otherwise cut a long warm-up short.

Verified on macOS with PowerShell 7.6.7 behind a shim that simulates
the cold start. The class is Windows-only, so this is a simulation,
not a Windows reproduction. Without the fix, a 41 s cold start fails
the same four tests as run 38812686088, at 10.01 s each, and a 12 s
cold start fails test_pwsh_string_simple_roundtrip alone. With the fix
both pass 13/13, and the cold start lands in the first test's setup.
Each of these mutants fails the class under a 41 s cold start: no
warm-up, a 10 s warm-up deadline, and no timeout marker.

Only repeated green Code Quality runs on real Windows runners prove
the fix. At roughly one failure in 50 Windows jobs, one green run is
weak evidence.

Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>
@github-actions

Copy link
Copy Markdown

Claude review · advisory

Reviewed 02cd712 (full review): 0 findings.

Open: 0 blocking · 0 should-fix · 0 nit. ✅ Nothing blocking.

Fix or reply to each thread; the next revision's review re-checks open threads and resolves those it agrees are handled. Resolving a thread yourself also closes it. Later revisions review only what changed.

@leongdl leongdl mentioned this pull request Oct 15, 2026
@leongdl
leongdl enabled auto-merge (squash) October 16, 2026 19:35
@leongdl
leongdl merged commit 1eaa3c0 into OpenJobDescription:mainline Oct 16, 2026
65 of 68 checks passed
@leongdl
leongdl deleted the fix/pr379-flaky-test branch October 16, 2026 21:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants