Skip to content

Converting a FetchHttpClient response stream drops workerd's known-length capability #8205

Description

@third774

What version of Effect is running?

  • Effect 4.0.0-rc.112
  • @cloudflare/vitest-pool-workers 0.22.0
  • Miniflare 5.20260815.0-alpha / workerd 1.20260815.1

What steps can reproduce the bug?

import { env } from "cloudflare:workers"
import { Effect, Stream } from "effect"
import {
  FetchHttpClient,
  HttpClient,
  HttpClientRequest
} from "effect/unstable/http"
import { expect, it, vi } from "vitest"

it("drops the native body's known length after Effect conversion", async () => {
  const bytes = new Uint8Array([1, 2, 3])
  const url = "https://example.com/archive.tgz"
  vi.stubGlobal("fetch", () => Promise.resolve(new Response(bytes)))

  const nativeResponse = await fetch(url)
  if (nativeResponse.body === null) throw new Error("Missing response body")
  await expect(
    env.BACKUP.put("native.tgz", nativeResponse.body)
  ).resolves.not.toBeNull()

  const program = Effect.gen(function*() {
    const request = HttpClientRequest.get(url)
    const response = yield* HttpClient.execute(request)
    const body = yield* Stream.toReadableStreamEffect(response.stream)

    yield* Effect.tryPromise(() => env.BACKUP.put("effect.tgz", body))
  }).pipe(Effect.provide(FetchHttpClient.layer))

  await expect(Effect.runPromise(program)).rejects.toThrow(
    "Provided readable stream must have a known length"
  )
})

The R2 operation rejects with:

Provided readable stream must have a known length
(request/response body or readable half of FixedLengthStream)

In the same runtime, the native response-body write succeeds. FixedLengthStream.readable also succeeds.

What is the expected behavior?

For the direct conversion shown above, Stream.toReadableStreamEffect(response.stream) should produce a Web stream that retains the source body's runtime-known length and is accepted by R2.

What do you see instead?

The public HttpClientResponse body API is an Effect Stream. Converting it with Stream.toReadableStreamEffect() creates a generic Web ReadableStream, which does not retain the runtime-internal identity-length capability that R2 queries.

The body bytes are unchanged, but R2 rejects the converted stream. This prevents direct streaming through HttpClientResponse.stream without reconstructing the length or buffering the body.

Additional information

HttpClientResponse.fromWeb() keeps the original Response in a private field. HttpClientResponse.stream wraps Response.body with Stream.fromReadableStream. The HTTP client also adds interruption and finalization wrappers. Stream.toReadableStreamEffect() then creates a new Web stream, so workerd can no longer query the source body's identity length.

Cloudflare documents ReadableStream as a valid R2Bucket.put() value, but workerd's R2 implementation requires a stream to report an identity length before it starts the write. This check is in workerd itself, not only in Miniflare.

This request concerns direct conversion of response.stream. An arbitrary transformed, filtered, concatenated, or generated Effect stream does not necessarily have a known length.

A caller can read Content-Length, create a Cloudflare FixedLengthStream, and pump the Effect stream into it with Sink.fromWritableStream. This is a platform-specific manual bridge, and it works only when the header is present and equals the number of bytes emitted by the stream. Buffering through response.arrayBuffer is another workaround when the body fits safely in memory.

Our workaround fetches archives with the platform fetch() and passes the original Response.body to R2. Effect still owns retries, errors, and timeouts around that operation. Packument requests continue to use FetchHttpClient.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions