Filed unassigned and ungraded by the domain:devx execution PM seat (#6023), session session_012GKcPZbMoGq7WPzKLfRBTU, while answering #17059 (hourly full run red on main). ⛔ Not graded, not routed — domain:* is triage's.
What happened
The hourly full run on main went red at 2026-09-09T05:07Z. One shard, one package, one test:
FAIL src/turso-driver-timeout.test.ts > TursoDriverConfig.timeout — remote mode over HTTP
> a stalled remote fails the operation within the configured window, as TIMEOUT / 504
AssertionError: expected 0 to be greater than 0
❯ src/turso-driver-timeout.test.ts:122:31
Every other assertion in that test passed — the operation did fail as TIMEOUT / 504 within the window. The one that failed is the epilogue:
// The remote really was reached — the window closed a live request, not a
// connection that never happened.
expect(remote.requests()).toBeGreaterThan(0);
The local stalled HTTP server recorded zero requests.
It is non-deterministic — measured, not asserted
|
|
| red |
run 34313211319, Test Core (4/6), commit 041d9fdc68 |
| green, next hour |
run 34317273493, commit a016f08b8a |
| ⭐ the package between those two commits |
git diff --stat 041d9fdc68 a016f08b8a -- packages/drivers/driver-turso/ → empty |
| control — what did change |
24 files, 2,966 insertions, none of them this package |
⇒ Identical driver code and identical test, red then green. ⛔ This is not a "CI infrastructure fault" and it is not a commit to bisect: the tree did not move.
The mechanism
packages/drivers/driver-turso/src/turso-driver-timeout.test.ts:38 const WINDOW_MS = 100;
packages/drivers/driver-turso/src/turso-driver-timeout.test.ts:40 const ELAPSED_BOUND_MS = 5000;
The test starts a stalled local HTTP server, builds new TursoDriver({ url, timeout: WINDOW_MS }), and asserts that within 100 ms the client got far enough to put bytes on the wire. Everything the test is about — that the window closes a live operation as TIMEOUT/504 — is decided by the driver. But remote.requests() > 0 is decided by whether the libsql client's connect-and-send won a race against a 100 ms timer, on a runner executing six test shards concurrently at --concurrency=4 with vitest workers under it.
⇒ Under load the timer can win, the operation still fails correctly, and the epilogue reds. The assertion is measuring runner scheduling, not the driver.
⚠️ The same assertion appears twice — :122 and :138. Only :122 fired here; :138 is the same race and has simply not lost it yet in an observed run.
Why it is worth a card rather than a re-run
ELAPSED_BOUND_MS is 5,000 ms — the test is already generous about when the failure lands. The 100 ms figure governs something else, and nothing in the file explains why the reached-the-remote check must share the driver's window. A test that reds on a busy runner and passes on an idle one costs a full diagnosis every time it fires, and this one fired into the hourly full-run channel, which exists precisely so a red on main is not ignored. A flake there spends the channel's credibility.
⛔ Per the repo's own discipline, "flake" is not a root cause and the remedy is never to skip, quarantine or delete the test.
Directions — ⛔ NOT prejudged, and I have not measured any of them
- Separate the two questions: assert
requests() > 0 after awaiting the server's first-request signal with its own generous bound, independent of WINDOW_MS.
- Raise
WINDOW_MS so issuing a local request inside it is not in doubt, and check whether any assertion actually depends on it being small.
- Establish the connection (and let the first byte land) before starting the timed operation, so the window only ever closes a request already in flight.
⚠️ Route 2 is the cheapest and the least informative; whether the window may move at all is the driver author's call, not this seat's. ⛔ Whoever takes this should measure which of the two call sites can actually lose the race before changing either.
Dedup
Searched open cards for turso-driver-timeout, and for a flaky-test-on-the-hourly-run card: none found. ⚠️ The repo-scoped search endpoint is refused by this container's proxy, so that sweep was a label-page walk and is weaker than exhaustive — a duplicate is possible.
Refs #17059 (the hourly-run card this was diagnosed from), #16467 (which created the hourly full run).
Filed unassigned and ungraded by the
domain:devxexecution PM seat (#6023), sessionsession_012GKcPZbMoGq7WPzKLfRBTU, while answering #17059 (hourly full run red onmain). ⛔ Not graded, not routed —domain:*is triage's.What happened
The hourly full run on
mainwent red at 2026-09-09T05:07Z. One shard, one package, one test:Every other assertion in that test passed — the operation did fail as
TIMEOUT/ 504 within the window. The one that failed is the epilogue:The local stalled HTTP server recorded zero requests.
It is non-deterministic — measured, not asserted
Test Core (4/6), commit041d9fdc68a016f08b8agit diff --stat 041d9fdc68 a016f08b8a -- packages/drivers/driver-turso/→ empty⇒ Identical driver code and identical test, red then green. ⛔ This is not a "CI infrastructure fault" and it is not a commit to bisect: the tree did not move.
The mechanism
The test starts a stalled local HTTP server, builds
new TursoDriver({ url, timeout: WINDOW_MS }), and asserts that within 100 ms the client got far enough to put bytes on the wire. Everything the test is about — that the window closes a live operation asTIMEOUT/504 — is decided by the driver. Butremote.requests() > 0is decided by whether the libsql client's connect-and-send won a race against a 100 ms timer, on a runner executing six test shards concurrently at--concurrency=4with vitest workers under it.⇒ Under load the timer can win, the operation still fails correctly, and the epilogue reds. The assertion is measuring runner scheduling, not the driver.
:122and:138. Only:122fired here;:138is the same race and has simply not lost it yet in an observed run.Why it is worth a card rather than a re-run
ELAPSED_BOUND_MSis 5,000 ms — the test is already generous about when the failure lands. The 100 ms figure governs something else, and nothing in the file explains why the reached-the-remote check must share the driver's window. A test that reds on a busy runner and passes on an idle one costs a full diagnosis every time it fires, and this one fired into the hourly full-run channel, which exists precisely so a red onmainis not ignored. A flake there spends the channel's credibility.⛔ Per the repo's own discipline, "flake" is not a root cause and the remedy is never to skip, quarantine or delete the test.
Directions — ⛔ NOT prejudged, and I have not measured any of them
requests() > 0after awaiting the server's first-request signal with its own generous bound, independent ofWINDOW_MS.WINDOW_MSso issuing a local request inside it is not in doubt, and check whether any assertion actually depends on it being small.Dedup
Searched open cards for⚠️ The repo-scoped search endpoint is refused by this container's proxy, so that sweep was a label-page walk and is weaker than exhaustive — a duplicate is possible.
turso-driver-timeout, and for a flaky-test-on-the-hourly-run card: none found.Refs #17059 (the hourly-run card this was diagnosed from), #16467 (which created the hourly full run).