Skip to content

Support streaming put payloads - #817

Open
senzzzi wants to merge 2 commits into
apache:mainfrom
senzzzi:bruno.rodrigues/put-supports-streaming
Open

Support streaming put payloads#817
senzzzi wants to merge 2 commits into
apache:mainfrom
senzzzi:bruno.rodrigues/put-supports-streaming

Conversation

@senzzzi

@senzzzi senzzzi commented Jul 30, 2026

Copy link
Copy Markdown

Which issue does this PR close?

Closes #281.

Rationale for this change

PutPayload currently only supports collections of Bytes. As a result, uploading a file through put or put_opts requires loading the entire file into memory or using a multipart upload.

This change allows callers to stream a payload with a known content length through a single PUT request. File-backed payloads are a primary use case.

What changes are included in this PR?

  • Extends PutPayload to support replayable streaming bodies with a known content length.
  • Adds PutPayload::from_stream for constructing streaming payloads.
  • Adds PutPayload::from_file, which reads files in 16 KiB chunks by default.
  • Adds PutPayload::from_file_with_chunk_size for configuring the file chunk size.
  • Updates HTTP request bodies to poll payload streams asynchronously.
  • Recreates the stream for every request attempt so retryable uploads can resend the complete payload.
  • Updates S3 signing and checksum calculation to consume streaming payloads.
  • Updates the local filesystem and in-memory backends to accept streaming payloads.
  • Adds regression tests covering:
    • the default 16 KiB file chunk size;
    • replaying cloned streaming request bodies; and
    • retrying an upload using a file-backed streaming payload.

Validation performed:

  • cargo test
  • cargo test --all-features
  • cargo clippy --all-targets --all-features -- -D warnings
  • compilation with only the fs feature
  • compilation without default features

Are there any user-facing changes?

Yes. Callers can now upload a file without first loading the complete file into memory:

let payload = PutPayload::from_file(file_path).await?;
store.put(&location, payload).await?;

Custom streaming payloads can be created with PutPayload::from_stream. The stream factory must produce a new stream from the beginning for every invocation, and the
supplied content length must exactly match the number of bytes yielded.

Existing byte-backed PutPayload construction and iteration behavior is unchanged. Synchronous APIs such as iter, AsRef<[Bytes]>, and conversion into Bytes are not supported for streaming payloads; callers should use PutPayload::stream or PutPayload::bytes instead.

API compatibility consideration

PutPayload remains a public struct backed by a private enum so that existing byte-backed construction and usage remain source compatible:

enum PutPayloadInner {
    Bytes(Arc<[Bytes]>),
    Streaming {
        factory: StreamFactory,
        content_length: usize,
    },
}

There is an API-design limitation around the existing synchronous accessors. Methods and trait implementations such as PutPayload::iter, AsRef<[Bytes]>, IntoIterator, and conversion into Bytes cannot synchronously represent an asynchronous, fallible stream.

In this implementation, their existing behavior is unchanged for byte-backed payloads, but calling them with a streaming payload panics. Streaming-aware implementations should instead use PutPayload::stream or PutPayload::bytes.

Changing the existing synchronous APIs to return Option or Result would avoid the panic but would be a breaking public API change. Returning an empty iterator for streaming payloads was rejected because it could silently result in incomplete uploads.

Feedback from maintainers on the preferred long-term API shape would be appreciated. Possible alternatives include:

  • accepting the documented limitation for synchronous APIs;
  • changing these APIs in a future breaking release; or
  • introducing streaming uploads through a separate payload type or method.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

object_store: Support for streaming body in put/put_opts

1 participant