Overlap the WSL and container setups with the build again, with five-minute step timeouts

[johnsimons]

Why

#5873 moved "Wait for build" ahead of the WSL and container setups because two Windows jobs had hung with "The hosted runner lost communication with the server" while the build and the 8 GB WSL2 VM shared the runner. The job history since then says the serialisation did not buy what it was meant to buy:

• 8 Aug to 9 Sep, overlap in place, 500 runs: two Windows jobs ran past 25 minutes. One was a dead runner that recorded no steps, the other a test run whose SQL Server container had become unreachable.
• 10 Sep to 8 Oct, serialised, 315 runs: eleven Windows container jobs hit the 30 minute cap, across all four container categories, always one job per run with the sibling Windows jobs green. Seven recorded no steps at all. The four from 6 and 7 Oct stalled inside "Setup WSL" with the build already finished.

The hang lives in the WSL step and the runner, and the serial order costs the critical path about two minutes on every run.

What changes

• "Wait for build" moves back below "Setup IBM MQ", so WSL and the four container setups run while the build is still going. It stays ahead of "Azure login", because the Azure steps create real cloud resources and a broken build must not provision them (6ec08ce).
• "Build" and "Setup WSL" get timeout-minutes: 5. A healthy build takes up to about three minutes on the heaviest category and WSL under a minute, so a stall now fails the job in five minutes instead of occupying a runner for thirty.
• The comments on "Setup WSL" and on the VM memory input say why the order is what it is.

Measured on this branch

Three CI runs with a sampler step that recorded free memory and CPU every five seconds during the overlapped stretch. The final file does not include the sampler.

Critical path, Windows-SqlServer:

• master, serial, 5 and 6 Oct: job 667 to 707s. Build 128 to 150s, Setup WSL 44 to 50s, Setup SQL Server 111 to 114s, about 328s before the tests start.
• this branch, run 37861906054: job 458s. Build 170s, Setup WSL 41s, Setup SQL Server 130s, Wait for build 0s, 217s before the tests start.

The build and the SQL Server setup each run slower when they share the box, which is why the saving is 110s and not the 160s the step lengths add up to.

Other Windows container jobs: PostgreSql 365s to 276s, RabbitMQ 293s to 253s, IBMMQ 205s to 153s.

Memory was not the constraint. Minimum free memory during the overlap was 3.9 GB on Windows-SqlServer, 8 to 10 GB on the other three Windows container jobs, and above 12 GB on Linux. CPU sat at 100% on every Windows container job for the duration, so cores are what the build and the VM compete for.

One of the eight overlapped Windows container jobs hung: run 37859193494, Windows-SqlServer, with "Build" and "Setup WSL" both in progress. That commit predates the step timeouts, so it ran to the 30 minute cap.

What to expect

• The hang rate under the overlap rests on eight jobs. If it is higher than the serial order's one in a hundred, it shows up as red Windows-SqlServer jobs over the next weeks, each costing five minutes rather than thirty. The lasting fix is in setup-wsl-action, whose Invoke-Wsl calls have no timeout.
• No run has exercised a step timeout yet, and "Build" is a background: true step. If timeout-minutes does not apply to background steps, a hung build still runs to the job cap. The "Setup WSL" timeout is on a plain step.

Verification

The three experiment runs above, on the branch with the sampler steps; the workflow file parses locally. The CI run on this pull request is the first with the exact final file.

[warwickschroeder]

Setting the timeout to 5min could introduce a flaky build. The Windows-DefaultCore build can sometimes run over 5min, albeit rarely. Over the last ~56 runs its p90 is 212s, and the longest was 308s. An easy fix would be to raise this timeout.