Flaky test: KillSupervisorsCustomDutyTest.test_customDuty_removesHistoryOfTerminatedSupervisor #20445

This issue was generated automatically by Claude Code (Anthropic's AI coding agent) running a scheduled CI-triage routine on behalf of @FrankChen021. Analysis and suggested fixes are AI-produced; please verify before acting on them.

Status: Resolved by #20458
Subject: KillSupervisorsCustomDutyTest.test_customDuty_removesHistoryOfTerminatedSupervisor (embedded-tests, unit tests (25, K*,U*,Z*,Y*,X*))
Failures: 1 · First seen: 2026-09-28 · Last seen: 2026-09-28

Root cause

The coordinator in this test runs the killSupervisors duty every 100 ms (cleanupMetadata.period=PT0.1S) with durationToRetain=PT0M. Once terminateSupervisor returns, the duty can delete both the spec and its tombstone before the test's second getSupervisorHistory call at line 87, which is meant to assert 2 entries. That call then gets 404 No history for [autocompact__datasource_…]. #19150 fixed an earlier race in the final 404 check. This failure comes one step earlier. Commit 8da9a1b only changes TaskRunnerListener signatures and does not touch supervisor history or the kill duty. The same job passed on the neighbouring commits d2bff3c and 1afd36b.

Suggested fix

The post-termination "history has 2 entries" assertion can't be reliable while the duty deletes terminated history every 100 ms with no retention. Either remove it and rely on the metadata/kill/supervisor/count sum of 2, or raise durationToRetain to a few seconds so the tombstone survives the read. Also check that the metric wait still fits within its timeout after that change. Fixed by #20458 (merged 2026-10-01).

Occurrences

Failed push-triggered master jobs only. The daily triage routine adds one row per new failed job.