Repository navigation
Regression: KafkaIndexFaultToleranceTest.test_supervisorRecovers_afterCoordinatorRestart fails since #20505 #20604
Description
This issue was generated automatically by Claude Code (Anthropic's AI coding agent) running a scheduled CI-triage routine on behalf of @FrankChen021. Analysis and suggested fixes are AI-produced; please verify before acting on them.
Status: Fix PR #20651 open
Subject: KafkaIndexFaultToleranceTest.test_supervisorRecovers_afterCoordinatorRestart (embedded-tests, unit tests / unit tests(main) (25, K*,U*,Z*,Y*,X*))
Failures: 19 · First seen: 2026-10-14 · Last seen: 2026-10-20
Root cause
Regression from #20505 (merged 2026-10-14 as 7c89096), which makes the Overlord hold a streaming task's publish until earlier task groups on the same partitions finish. The Coordinator restart delays handoff past the test's completion timeout; the supervisor kills the earlier group and stops the later one, then marks the earlier group done, which releases the later task's held publish. The Overlord commits that task's segments and offsets, but the task is interrupted before it sees the result and never emits ingest/rows/published, so the metric stays short of 3000 and verifyAndTearDown times out. Since #20505 it has failed on 19 master commits (7 of 8 on 2026-10-20), always as the only failing test; the data itself is correct. Diagnosis from #20651.
Suggested fix
Merge #20651: when the supervisor stops or kills a task (killTask, killTaskWithSuccess, stopTask), it first answers the task's held publish with "not ready", so a later release from the completed group has no effect. Until it lands, this test fails on most master commits.
Occurrences
Failed push-triggered master jobs only. The daily triage routine adds one row per new failed job.
| Date | Commit | Job | Failure log | Detail | Reported in |
|---|---|---|---|---|---|
| 2026-10-14 | afe71f5 (#20577) | unit tests / unit tests(main) (25, K*,U*,Z*,Y*,X*) / test-jdk25-[K*,U*,Z*,Y*,X*] |
job 116011534392 | ISE: Timed out waiting for event after [240,000]ms in verifyAndTearDown:78 (published rows < 3000); 1/1 attempt; run attempt 2 |
|
| 2026-10-15 | e78e3b5 (#20588) | unit tests / unit tests(main) (25, K*,U*,Z*,Y*,X*) / test-jdk25-[K*,U*,Z*,Y*,X*] |
job 116026662477 | ISE: Timed out waiting for event after [240,000]ms in verifyAndTearDown:78 (published rows < 3000); 1/1 attempt |
|
| 2026-10-15 | b4b0fe5 (#20591) | unit tests / unit tests(main) (25, K*,U*,Z*,Y*,X*) / test-jdk25-[K*,U*,Z*,Y*,X*] |
job 116057779540 | ISE: Timed out waiting for event after [240,000]ms in verifyAndTearDown:78 (published rows < 3000); 1/1 attempt; run attempt 1, passed on run attempt 2 |
|
| 2026-10-15 | d5d0f11 (#20593) | unit tests / unit tests(main) (25, K*,U*,Z*,Y*,X*) / test-jdk25-[K*,U*,Z*,Y*,X*] |
job 116279969326 | ISE: Timed out waiting for event after [240,000]ms in verifyAndTearDown:78 (published rows < 3000); 1/1 attempt |
|
| 2026-10-15 | 2a2e474 (#20236) | unit tests / unit tests(main) (25, K*,U*,Z*,Y*,X*) / test-jdk25-[K*,U*,Z*,Y*,X*] |
job 116455192136 | ISE: Timed out waiting for event after [240,000]ms in verifyAndTearDown:78 (published rows < 3000); 1/1 attempt |
|
| 2026-10-16 | fde5b65 (#20585) | unit tests / unit tests(main) (25, K*,U*,Z*,Y*,X*) / test-jdk25-[K*,U*,Z*,Y*,X*] |
job 116764041195 | ISE: Timed out waiting for event after [240,000]ms in verifyAndTearDown:78 (published rows < 3000); 1/1 attempt |
|
| 2026-10-16 | 17d5ad0 (#20584) | unit tests / unit tests(main) (25, K*,U*,Z*,Y*,X*) / test-jdk25-[K*,U*,Z*,Y*,X*] |
job 116764225495 | ISE: Timed out waiting for event after [240,000]ms in verifyAndTearDown:78 (published rows < 3000); 1/1 attempt |
|
| 2026-10-16 | 92fe6f5 (#20581) | unit tests / unit tests(main) (25, K*,U*,Z*,Y*,X*) / test-jdk25-[K*,U*,Z*,Y*,X*] |
job 116764748745 | ISE: Timed out waiting for event after [240,000]ms in verifyAndTearDown:78 (published rows < 3000); 1/1 attempt |
|
| 2026-10-16 | 206e3c9 (#20615) | unit tests / unit tests(main) (25, K*,U*,Z*,Y*,X*) / test-jdk25-[K*,U*,Z*,Y*,X*] |
job 116971271378 | ISE: Timed out waiting for event after [240,000]ms in verifyAndTearDown:78 (published rows < 3000); 1/1 attempt |
|
| 2026-10-17 | 3d8ec4f (#20597) | unit tests / unit tests(main) (25, K*,U*,Z*,Y*,X*) / test-jdk25-[K*,U*,Z*,Y*,X*] |
job 117275157294 | ISE: Timed out waiting for event after [240,000]ms in verifyAndTearDown:78 (published rows < 3000); 1/1 attempt |
|
| 2026-10-17 | f8a6753 (#20596) | unit tests / unit tests(main) (25, K*,U*,Z*,Y*,X*) / test-jdk25-[K*,U*,Z*,Y*,X*] |
job 117275111369 | ISE: Timed out waiting for event after [240,000]ms in verifyAndTearDown:78 (published rows < 3000); 1/1 attempt |
|
| 2026-10-18 | 433fd4d (#20622) | unit tests / unit tests(main) (25, K*,U*,Z*,Y*,X*) / test-jdk25-[K*,U*,Z*,Y*,X*] |
job 117556036750 | ISE: Timed out waiting for event after [240,000]ms in verifyAndTearDown:78 (published rows < 3000); 1/1 attempt |
|
| 2026-10-20 | b14e41a (#20586) | unit tests / unit tests(main) (25, K*,U*,Z*,Y*,X*) / test-jdk25-[K*,U*,Z*,Y*,X*] |
job 118417805402 | ISE: Timed out waiting for event after [240,000]ms in verifyAndTearDown:78 (published rows < 3000); 1/1 attempt |
|
| 2026-10-20 | 8803f6c (#20447) | unit tests / unit tests(main) (25, K*,U*,Z*,Y*,X*) / test-jdk25-[K*,U*,Z*,Y*,X*] |
job 118622709253 | ISE: Timed out waiting for event after [240,000]ms in verifyAndTearDown:78 (published rows < 3000); 1/1 attempt |
|
| 2026-10-20 | ff28cd4 (#20638) | unit tests / unit tests(main) (25, K*,U*,Z*,Y*,X*) / test-jdk25-[K*,U*,Z*,Y*,X*] |
job 118697261450 | ISE: Timed out waiting for event after [240,000]ms in verifyAndTearDown:78 (published rows < 3000); 1/1 attempt |
|
| 2026-10-20 | 8a3e4c3 (#20471) | unit tests / unit tests(main) (25, K*,U*,Z*,Y*,X*) / test-jdk25-[K*,U*,Z*,Y*,X*] |
job 118701431012 | ISE: Timed out waiting for event after [240,000]ms in verifyAndTearDown:78 (published rows < 3000); 1/1 attempt |
|
| 2026-10-20 | c55da2a (#20476) | unit tests / unit tests(main) (25, K*,U*,Z*,Y*,X*) / test-jdk25-[K*,U*,Z*,Y*,X*] |
job 118702055296 | ISE: Timed out waiting for event after [240,000]ms in verifyAndTearDown:78 (published rows < 3000); 1/1 attempt |
|
| 2026-10-20 | 9e5b027 (#20639) | unit tests / unit tests(main) (25, K*,U*,Z*,Y*,X*) / test-jdk25-[K*,U*,Z*,Y*,X*] |
job 118705474850 | ISE: Timed out waiting for event after [240,000]ms in verifyAndTearDown:78 (published rows < 3000); 1/1 attempt |
|
| 2026-10-20 | d61f8cf (#20530) | unit tests / unit tests(main) (25, K*,U*,Z*,Y*,X*) / test-jdk25-[K*,U*,Z*,Y*,X*] |
job 118719815364 | ISE: Timed out waiting for event after [240,000]ms in verifyAndTearDown:78 (published rows < 3000); 1/1 attempt |
Failures on open dependency PRs
Pull-request runs (Dependabot version bumps, no code changes in this area), for reference only; not counted above. Each failed with the same 240 s timeout in verifyAndTearDown:78.
| PR | Failure log | Run attempt |
|---|---|---|
| #20596 | job 116130974204 | 1 |
| #20597 | job 116168312684 | 1 |
| #20597 | job 116427559860 | 2 |
| #20573 | job 116427650479 | 1 |
| #20598 | job 116450390123 | 1 |
| #20599 | job 116451010475 | 1 |
| #20610 | job 116734381485 | 1 |
| #20611 | job 116734620148 | 1 |
| #20613 | job 116735576583 | 1 |
Activity
[-]Flaky test: KafkaIndexFaultToleranceTest.test_supervisorRecovers_afterCoordinatorRestart[/-][+]Regression: KafkaIndexFaultToleranceTest.test_supervisorRecovers_afterCoordinatorRestart fails since #20505[/+]FrankChen021 commentedon Oct 16, 2026
@kfaraz could you take a look? This test has failed on 8 of 19
K*,U*,Z*,Y*,X*master job runs since #20505 merged on 2026-10-14. It passed on every master run from 2026-10-12 until then, and it is now failing most open dependency PRs. The failure is always the 240 s wait for published rows after a Coordinator restart, which suggests the new wait for earlier task groups to publish never completes in that scenario. Details and job links are in the issue body.FrankChen021 commentedon Oct 16, 2026
Failed again on 2026-10-15: e78e3b5 (#20588) job 116026662477, b4b0fe5 (#20591) job 116057779540, d5d0f11 (#20593) job 116279969326, 2a2e474 (#20236) job 116455192136, same 240 s published-rows timeout in
verifyAndTearDown:78; 4 of 8 master commits that day failed only on this test.FrankChen021 commentedon Oct 17, 2026
Failed again on 2026-10-16: fde5b65 (#20585) job 116764041195, 17d5ad0 (#20584) job 116764225495, 92fe6f5 (#20581) job 116764748745, 206e3c9 (#20615) job 116971271378, same 240 s published-rows timeout in
verifyAndTearDown:78; 4 of 9 master commits that day failed, all only on this test.FrankChen021 commentedon Oct 18, 2026
Failed again on 2026-10-17: 3d8ec4f job 117275157294 and f8a6753 job 117275111369, same 240 s timeout in
verifyAndTearDown:78(1/1 attempt each); the job passed on the other 10 master commits that day.FrankChen021 commentedon Oct 19, 2026
Failed again on 2026-10-18: 433fd4d (#20622, Guava test-dependency bump) job 117556036750, same 240 s published-rows timeout in
verifyAndTearDown:78(1/1 attempt); the test passed on the other 2 master commits that ran unit tests that day.FrankChen021 commentedon Oct 21, 2026
Failed again on 2026-10-20: on 7 of 8 master commits whose
K*,U*,Z*,Y*,X*job ran, most recently d61f8cf job 118719815364, same 240 s timeout inverifyAndTearDown:78. Fix PR #20651 is open.fix: Do not release a stopped streaming task to publish after a prior…