Summary
10 distinct tests failed against committed code (Timer or Post Merge Action builds on main) in the 24 hours ending 2026-06-10T10:00 UTC. None reproduced locally with the original seed, consistent with non-deterministic (timing/environment-dependent) flakiness.
Summary Table (sorted by total builds affected)
| # |
Test |
Builds Affected |
First Seen |
Trend |
Reproduced? |
| 1 |
RemoteStoreStatsIT |
217 |
2024-03-27 |
Worsening — recent months show increased failure rate |
No — passed with seed |
| 2 |
MixedClusterClientYamlTestSuiteIT |
148 |
2024-03-25 |
Worsening — recent months show increased failure rate |
N/A — requires multi-version cluster |
| 3 |
SharedClusterSnapshotRestoreIT |
108 |
2024-08-31 |
Worsening — recent months show increased failure rate |
No — passed with seed |
| 4 |
SecureReactorNetty4HttpServerTransportTests |
67 |
2025-02-19 |
Improving — recent months show decreased failure rate |
No — passed with seed |
| 5 |
IngestFromKafkaIT |
54 |
2025-10-15 |
Worsening — recent months show increased failure rate |
No — passed with seed |
| 6 |
AggregationProfilerIT |
26 |
2025-08-07 |
Worsening — recent months show increased failure rate |
No — passed with seed |
| 7 |
SegmentReplicationUsingRemoteStoreIT |
12 |
2024-07-15 |
Worsening — recent months show increased failure rate |
No — passed with seed |
| 8 |
IndexRecoveryIT |
12 |
2024-06-05 |
Worsening — recent months show increased failure rate |
No — passed with seed |
| 9 |
AdmissionControlSingleNodeTests |
8 |
2025-04-14 |
Stable — consistent low-level flakiness |
No — passed with seed |
| 10 |
IngestPipelineFromKafkaIT |
3 |
2026-03-30 |
Stable — consistent low-level flakiness |
No — passed with seed |
Detailed Findings
1. RemoteStoreStatsIT.testNonZeroPrimaryStatsOnNewlyCreatedIndexWithZeroDocs
| Field |
Value |
| Build |
#79997 |
| Seed |
98627D977CB6673C:5595A84B3AA9B0FA |
| Error |
java.lang.AssertionError |
| First failure |
2024-03-27 |
| Total builds affected |
217 |
| Recent monthly failures (2026) |
2026-01=7, 2026-02=2, 2026-03=9, 2026-04=8, 2026-05=14, 2026-06=12 |
| Reproduced locally |
No — passed with seed |
| Trend |
Worsening — recent months show increased failure rate |
2. MixedClusterClientYamlTestSuiteIT.test {p0=cluster.health/10_basic/cluster health with closed index}
| Field |
Value |
| Build |
#79975 |
| Seed |
6AD408F9F4AB03EB:E28037235A576E13 |
| Error |
java.lang.AssertionError: Failure at [cluster.health/10_basic:182]: expected [2xx] status code but api [cluster.health] returned [408 Request Timeout] [{"cluster_name":"v3.6.1","status":"red","timed_o |
| First failure |
2024-03-25 |
| Total builds affected |
148 |
| Recent monthly failures (2026) |
2026-01=1, 2026-04=9, 2026-05=11, 2026-06=13 |
| Reproduced locally |
N/A — requires multi-version cluster |
| Trend |
Worsening — recent months show increased failure rate |
3. SharedClusterSnapshotRestoreIT.testSnapshotFileFailureDuringSnapshot
| Field |
Value |
| Build |
#80058 |
| Seed |
CA37A560C385DC0F:F545090A3AB8BA12 |
| Error |
`java.lang.AssertionError: |
| Expected: <0L> |
|
| First failure | 2024-08-31 |
| Total builds affected | 108 |
| Recent monthly failures (2026) | 2026-01=3, 2026-03=6, 2026-04=10, 2026-05=9, 2026-06=5 |
| Reproduced locally | No — passed with seed |
| Trend | Worsening — recent months show increased failure rate |
4. SecureReactorNetty4HttpServerTransportTests.testBadRequest
| Field |
Value |
| Build |
#79964 |
| Seed |
E43DC6930946AF27:F50C8D304839DBDC |
| Error |
reactor.core.Exceptions$ReactiveException: reactor.netty.http.client.PrematureCloseException: Connection prematurely closed BEFORE response |
| First failure |
2025-02-19 |
| Total builds affected |
67 |
| Recent monthly failures (2026) |
2026-06=2 |
| Reproduced locally |
No — passed with seed |
| Trend |
Improving — recent months show decreased failure rate |
5. IngestFromKafkaIT.testAllActiveOffsetBasedLag
| Field |
Value |
| Build |
#80024 |
| Seed |
C12A3978C955D4CA:62CBB61E56B3475 |
| Error |
java.lang.AssertionError |
| First failure |
2025-10-15 |
| Total builds affected |
54 |
| Recent monthly failures (2026) |
2026-03=8, 2026-04=13, 2026-05=25, 2026-06=6 |
| Reproduced locally |
No — passed with seed |
| Trend |
Worsening — recent months show increased failure rate |
6. AggregationProfilerIT.testTopHitsAggregationFetchProfiling {p0={"search.concurrent_segment_search.enabled":"true"}}
| Field |
Value |
| Build |
#80029 |
| Seed |
2ACBF530BFD46BA8:51059F0E421632E3 |
| Error |
org.opensearch.action.search.SearchPhaseExecutionException: all shards failed |
| First failure |
2025-08-07 |
| Total builds affected |
26 |
| Recent monthly failures (2026) |
2026-03=2, 2026-04=1, 2026-05=5, 2026-06=4 |
| Reproduced locally |
No — passed with seed |
| Trend |
Worsening — recent months show increased failure rate |
7. SegmentReplicationUsingRemoteStoreIT.testPrimaryStopped_ReplicaPromoted
| Field |
Value |
| Build |
#79924 |
| Seed |
3A9E9A3DB968E533:2B969B5AA61A120B |
| Error |
java.lang.AssertionError: Count is 2 hits but 3 was expected. Total shards: 1 Successful shards: 1 & 0 shard failures: |
| First failure |
2024-07-15 |
| Total builds affected |
12 |
| Recent monthly failures (2026) |
2026-04=3, 2026-05=4, 2026-06=1 |
| Reproduced locally |
No — passed with seed |
| Trend |
Worsening — recent months show increased failure rate |
8. IndexRecoveryIT.testOngoingRecoveryAndClusterManagerFailOver
| Field |
Value |
| Build |
#80020 |
| Seed |
B82047EE4FA4447F:AE78B75A37F1BB9E |
| Error |
java.lang.AssertionError: timed out waiting for green state |
| First failure |
2024-06-05 |
| Total builds affected |
12 |
| Recent monthly failures (2026) |
2026-05=4, 2026-06=3 |
| Reproduced locally |
No — passed with seed |
| Trend |
Worsening — recent months show increased failure rate |
9. AdmissionControlSingleNodeTests.testAdmissionControlRejectionDisabledMode
| Field |
Value |
| Build |
#80024 |
| Seed |
C12A3978C955D4CA:10A2707B0AD35D4 |
| Error |
java.lang.AssertionError: expected:<3> but was:<6> |
| First failure |
2025-04-14 |
| Total builds affected |
8 |
| Recent monthly failures (2026) |
2026-03=1, 2026-06=1 |
| Reproduced locally |
No — passed with seed |
| Trend |
Stable — consistent low-level flakiness |
10. IngestPipelineFromKafkaIT.testCombinedScriptTransformations
| Field |
Value |
| Build |
#79989 |
| Seed |
78A1D1604AFDB051:7F2405C1BA8BBD63 |
| Error |
org.awaitility.core.ConditionTimeoutException: Condition org.opensearch.plugin.kafka.KafkaUtils$$Lambda/0x000000002a6bb978 was not fulfilled within 1 minutes. |
| First failure |
2026-03-30 |
| Total builds affected |
3 |
| Recent monthly failures (2026) |
2026-03=1, 2026-06=2 |
| Reproduced locally |
No — passed with seed |
| Trend |
Stable — consistent low-level flakiness |
Methodology
- Queried
gradle-check-* indices on the OpenSearch metrics cluster for test_status=FAILED in the past 24h, filtered to committed-code builds (invoke_type=Timer, git_reference=main or invoke_type=Post Merge Action).
- Queried full historical failure data across all build types (including PR builds) with monthly date histogram and cardinality aggregation on
build_number.
- Extracted test seeds from Jenkins test reports (the
SeedInfo.seed([...]) line in stack traces).
- Attempted local reproduction:
./gradlew :<module>:<task> --tests "<class>.<method>" -Dtests.seed=<SEED>.
- None of the 9 locally-runnable tests reproduced. The
MixedClusterClientYamlTestSuiteIT requires a multi-version backward-compatibility cluster and could not be run locally with a simple gradle command.
Observations
- None of the seeds are deterministic reproducers. This is expected for timing-dependent flakes —
RandomizedRunner seeds control Random streams but not thread scheduling, network timing, GC pauses, or JIT warmup.
- CI runner migration (April 2026): The move from
m5.8xlarge to m7a.8xlarge correlates with upticks in MixedClusterClientYamlTestSuiteIT, SegmentReplicationUsingRemoteStoreIT, and IndexRecoveryIT.
RemoteStoreStatsIT is the most chronic offender (217 builds since March 2024) with consistent ~5-15 builds/month. No sign of improvement.IngestFromKafkaIT is worsening rapidly (2 builds in Oct 2025 → 25 in May 2026).SecureReactorNetty4HttpServerTransportTests had a spike in Sep 2025 (33 builds) but has been mostly quiet since. The June 2026 occurrence may be a recurrence.
Summary
10 distinct tests failed against committed code (Timer or Post Merge Action builds on
main) in the 24 hours ending 2026-06-10T10:00 UTC. None reproduced locally with the original seed, consistent with non-deterministic (timing/environment-dependent) flakiness.Summary Table (sorted by total builds affected)
RemoteStoreStatsITMixedClusterClientYamlTestSuiteITSharedClusterSnapshotRestoreITSecureReactorNetty4HttpServerTransportTestsIngestFromKafkaITAggregationProfilerITSegmentReplicationUsingRemoteStoreITIndexRecoveryITAdmissionControlSingleNodeTestsIngestPipelineFromKafkaITDetailed Findings
1.
RemoteStoreStatsIT.testNonZeroPrimaryStatsOnNewlyCreatedIndexWithZeroDocs98627D977CB6673C:5595A84B3AA9B0FAjava.lang.AssertionError2.
MixedClusterClientYamlTestSuiteIT.test {p0=cluster.health/10_basic/cluster health with closed index}6AD408F9F4AB03EB:E28037235A576E13java.lang.AssertionError: Failure at [cluster.health/10_basic:182]: expected [2xx] status code but api [cluster.health] returned [408 Request Timeout] [{"cluster_name":"v3.6.1","status":"red","timed_o3.
SharedClusterSnapshotRestoreIT.testSnapshotFileFailureDuringSnapshotCA37A560C385DC0F:F545090A3AB8BA12| First failure | 2024-08-31 |
| Total builds affected | 108 |
| Recent monthly failures (2026) | 2026-01=3, 2026-03=6, 2026-04=10, 2026-05=9, 2026-06=5 |
| Reproduced locally | No — passed with seed |
| Trend | Worsening — recent months show increased failure rate |
4.
SecureReactorNetty4HttpServerTransportTests.testBadRequestE43DC6930946AF27:F50C8D304839DBDCreactor.core.Exceptions$ReactiveException: reactor.netty.http.client.PrematureCloseException: Connection prematurely closed BEFORE response5.
IngestFromKafkaIT.testAllActiveOffsetBasedLagC12A3978C955D4CA:62CBB61E56B3475java.lang.AssertionError6.
AggregationProfilerIT.testTopHitsAggregationFetchProfiling {p0={"search.concurrent_segment_search.enabled":"true"}}2ACBF530BFD46BA8:51059F0E421632E3org.opensearch.action.search.SearchPhaseExecutionException: all shards failed7.
SegmentReplicationUsingRemoteStoreIT.testPrimaryStopped_ReplicaPromoted3A9E9A3DB968E533:2B969B5AA61A120Bjava.lang.AssertionError: Count is 2 hits but 3 was expected. Total shards: 1 Successful shards: 1 & 0 shard failures:8.
IndexRecoveryIT.testOngoingRecoveryAndClusterManagerFailOverB82047EE4FA4447F:AE78B75A37F1BB9Ejava.lang.AssertionError: timed out waiting for green state9.
AdmissionControlSingleNodeTests.testAdmissionControlRejectionDisabledModeC12A3978C955D4CA:10A2707B0AD35D4java.lang.AssertionError: expected:<3> but was:<6>10.
IngestPipelineFromKafkaIT.testCombinedScriptTransformations78A1D1604AFDB051:7F2405C1BA8BBD63org.awaitility.core.ConditionTimeoutException: Condition org.opensearch.plugin.kafka.KafkaUtils$$Lambda/0x000000002a6bb978 was not fulfilled within 1 minutes.Methodology
gradle-check-*indices on the OpenSearch metrics cluster fortest_status=FAILEDin the past 24h, filtered to committed-code builds (invoke_type=Timer, git_reference=mainorinvoke_type=Post Merge Action).build_number.SeedInfo.seed([...])line in stack traces)../gradlew :<module>:<task> --tests "<class>.<method>" -Dtests.seed=<SEED>.MixedClusterClientYamlTestSuiteITrequires a multi-version backward-compatibility cluster and could not be run locally with a simple gradle command.Observations
RandomizedRunnerseeds controlRandomstreams but not thread scheduling, network timing, GC pauses, or JIT warmup.m5.8xlargetom7a.8xlargecorrelates with upticks inMixedClusterClientYamlTestSuiteIT,SegmentReplicationUsingRemoteStoreIT, andIndexRecoveryIT.RemoteStoreStatsITis the most chronic offender (217 builds since March 2024) with consistent ~5-15 builds/month. No sign of improvement.IngestFromKafkaITis worsening rapidly (2 builds in Oct 2025 → 25 in May 2026).SecureReactorNetty4HttpServerTransportTestshad a spike in Sep 2025 (33 builds) but has been mostly quiet since. The June 2026 occurrence may be a recurrence.