[fforres]

Impact

A short D1 outage no longer fails every tool call that reads from the database. The Cloudflare host now waits and tries again for up to about 3.6 seconds before it returns an error.

Incident

On 2026-10-16, from 16:45:38 to about 16:48 UTC, executor-cloudflare logged 63 D1_ERROR: internal error; reference = ... errors (9, 24, 22 and 8 per minute). Then D1 recovered by itself. During those ~3 minutes, every tool call that used D1 failed (machine_run_status, machine_exec, machine_open, github/search). One of these failures stopped an unattended PR-review run.

What changed

• New apps/host-cloudflare/src/db/d1-retry.ts: withD1TransientRetry(db) wraps the D1 binding. It covers prepare().bind().first/all/run/raw, batch (it gives the binding its own statements), exec and withSession.
• createD1ExecutorDb wraps the binding once, at the top. All D1 traffic in this host goes through that function: drizzle runtime queries, the schema fingerprint, and the data migrations. Both callers (app.ts and the MCP session Durable Object) get the retry, and no call site changes.

What is retried, and what is not

The classification matches the documented D1 error list (https://developers.cloudflare.com/d1/observability/debug-d1/). The wrapper reads the error message and its cause chain.

Class	Messages	Retried for
not executed (D1 was never reached)	Cannot resolve D1 DB due to transient issue on remote node, Internal error while starting up D1 DB storage caused object to be reset, D1 DB is overloaded. Too many requests queued	all statements, including writes
maybe executed	D1_ERROR: internal error (the incident), Network connection lost, Replica disconnected from primary, Internal error in D1 DB storage caused object to be reset, D1 DB reset because its code was updated, Can't read from request stream because client disconnected, D1 DB is overloaded. Requests queued for too long	read-only statements only
permanent	everything else: SQL and syntax errors, constraint violations, no such table, D1_TYPE_ERROR, size and free-tier limits, and the resets that the query itself causes (storage timeout, isolate memory, CPU limit)	never

Writes. When a write fails with a "maybe executed" error such as internal error, the write may or may not have been applied. The binding cannot tell which. So the wrapper retries a write only when the error shows that D1 was never reached. A write that fails with internal error returns the original error, as before. A statement counts as read-only only if it starts with SELECT, WITH or EXPLAIN and contains no keyword that can write. This is the same rule D1 uses for its own automatic retries. A batch is retried after a "maybe executed" error only if all of its statements are read-only.

Timeouts. The brief listed timeouts. The only documented D1 timeout (storage operation exceeded timeout which caused object to be reset) is caused by a write that is too large, so a retry fails again. It is not retried.

Overload. The D1 DB is overloaded errors are retried, but the attempts are few and spaced with jitter, so the extra load stays small. Cloudflare's advice for overload is to reduce the load.

Backoff

Schedule.exponential(200ms), with each delay capped at 2s, then Schedule.jittered (80% to 120%), with Schedule.recurs(4). This gives at most 5 attempts. The delays are about 200, 400, 800 and 1600 ms, so the worst case is about 3.6 s before the original error is returned. The while predicate stops at once on a failure that cannot be retried. Each retry logs Effect.logWarning("D1 transient failure, retrying") with d1.operation, d1.read_only, d1.attempt, d1.max_attempts, d1.failure_class and d1.failure.

Tests

apps/host-cloudflare/src/db/d1-retry.test.ts (Effect Vitest, TestClock):

• the classifier table: 20 literal D1 messages, each mapped to its class, plus the cause chain and a non-Error rejection
• the read-only SQL table
• a read that fails 4 times with transient errors succeeds on attempt 5
• after 5 failures, the original error is returned (5 calls)
• a constraint violation is not retried (1 call)
• a write that fails with internal error is not retried (1 call)
• a write retries when D1 was not reached
• the retry waits before the second attempt
• through the wrapper: a bound read retries and returns its rows, a write rejects with the binding's own error, and a batch gives the binding its own statements

apps/host-cloudflare/src/db/d1.test.ts: createD1ExecutorDb gets through one D1_ERROR: internal error on the fingerprint read. This test fails if the wrapper is removed.

I checked the tests by mutation. Each of these changes made at least one test fail: removing the attempt bound, retrying writes after "maybe executed" errors, dropping the "not executed" class, passing the wrapped statements to batch, and removing the backoff delay.

Gates

• vitest run src/db (host-cloudflare): 49 passed. Full host-cloudflare bun run test: 136 passed.
• bun run typecheck: passed (45/45 tasks).
• bun run format:check: fails on 3 files that this PR does not change: packages/core/execution/src/index.ts, jev-discovery.test.ts and jev.test.ts.
• bun run lint: 0 findings in the changed files. It fails on 2 errors that this PR does not touch: executor(no-unknown-error-message) at packages/core/execution/src/jev-discovery.ts:174:26 and packages/core/sdk/src/executor.ts:4240:24.
• bun run test (turbo, --continue): 38 of 39 tasks pass. The one failure was already there before this PR, in @executor-js/cloudflare, which this PR does not change: src/mcp/agents-post-stream-loss.test.ts > covers the sandbox execution ceiling plus the browser-approval wait plus margin with expected 660000 to be greater than or equal to 3901000 (raise MCP_POST_RESPONSE_DEADLINE_MS in patches/agents@0.17.3.patch). The likely cause is the sixty-minute execute default in d8393e3.

Out of scope

R2 (the blob store), Durable Object stubs and service bindings do not go through a shared seam in this host, and their transient failures look different. The incident was D1 only, so this PR does not change them.

https://claude.ai/code/session_01UBfgbNPCqMRR2hryXb1FDL