Repository navigation
Flaky test: IngestionSmokeTest.test_streamLogs_ofCancelledTask (ConsulDiscoveryPlainDockerTest) #20435
Description
This issue was generated automatically by Claude Code (Anthropic's AI coding agent) running a scheduled CI-triage routine on behalf of @FrankChen021. Analysis and suggested fixes are AI-produced; please verify before acting on them.
Status: Fix PR #20549 open
Subject: IngestionSmokeTest.test_streamLogs_ofCancelledTask, seen via ConsulDiscoveryPlainDockerTest (embedded-tests, docker-tests)
Failures: 1 · First seen: 2026-09-25 · Last seen: 2026-09-25
Root cause
Race: the test polls TaskLogStreamer.streamTaskLog(taskId, 0) only until the Optional is present. In the Docker setup, the peon's log is pushed after the process exits, so the stream can be present but empty, which fails assertFalse(logs.isEmpty()) at IngestionSmokeTest.java:398. The TLS and MTLS variants passed in the same job.
Suggested fix
Move the read into the waitForResult supplier, and wait until the log contains "Running task[%s] for [%d] millis" instead of just Optional::isPresent. #20549 implements this.
Occurrences
Failed push-triggered master jobs only. The daily triage routine adds one row per new failed job.
| Date | Commit | Job | Failure log | Detail | Reported in |
|---|---|---|---|---|---|
| 2026-09-25 | 53e814e (#20425) | docker-tests |
job 108039867472 | streamed task log was empty; single attempt (no surefire retries) | #20428 |
Activity
rishi-rana commentedon Oct 10, 2026
I'd like to pick this up — will move the log read into the
waitForResultpoll loop so it waits for the expected log line (Running task[%s] for [%d] millis) rather than just stream presence, per the suggested fix above. Will open a PR shortly.ykisana commentedon Oct 10, 2026
@rishi-rana I think this fixes the test. But I believe there may be an underlying bug here.
The Overlord's HttpRemoteTaskRunner.streamTaskLog passes the worker's response through without checking the HTTP status. A 404, which happens once the worker has dropped a finished task, becomes a present-but-empty log stream.
SwitchingTaskLogStreamer then never falls back to deep storage, where the log actually is.
https://github.com/apache/druid/blob/98bbcb4c7f78/indexing-service/src/main/java/org/apache/druid/indexing/overlord/hrtr/HttpRemoteTaskRunner.java#L936-L940
ykisana commentedon Oct 10, 2026
Nvm ignore me, looking at logs, the test flake happened before this bug would've opened up
fix: avoid race in IngestionSmokeTest.test_streamLogs_ofCancelledTask (…