Support streaming put payloads - #817
Open
senzzzi wants to merge 2 commits into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #281.
Rationale for this change
PutPayloadcurrently only supports collections ofBytes. As a result, uploading a file throughputorput_optsrequires loading the entire file into memory or using a multipart upload.This change allows callers to stream a payload with a known content length through a single PUT request. File-backed payloads are a primary use case.
What changes are included in this PR?
PutPayloadto support replayable streaming bodies with a known content length.PutPayload::from_streamfor constructing streaming payloads.PutPayload::from_file, which reads files in 16 KiB chunks by default.PutPayload::from_file_with_chunk_sizefor configuring the file chunk size.Validation performed:
cargo testcargo test --all-featurescargo clippy --all-targets --all-features -- -D warningsfsfeatureAre there any user-facing changes?
Yes. Callers can now upload a file without first loading the complete file into memory:
Custom streaming payloads can be created with PutPayload::from_stream. The stream factory must produce a new stream from the beginning for every invocation, and the
supplied content length must exactly match the number of bytes yielded.
Existing byte-backed PutPayload construction and iteration behavior is unchanged. Synchronous APIs such as iter, AsRef<[Bytes]>, and conversion into Bytes are not supported for streaming payloads; callers should use PutPayload::stream or PutPayload::bytes instead.
API compatibility consideration
PutPayloadremains a public struct backed by a private enum so that existing byte-backed construction and usage remain source compatible:There is an API-design limitation around the existing synchronous accessors. Methods and trait implementations such as PutPayload::iter, AsRef<[Bytes]>, IntoIterator, and conversion into Bytes cannot synchronously represent an asynchronous, fallible stream.
In this implementation, their existing behavior is unchanged for byte-backed payloads, but calling them with a streaming payload panics. Streaming-aware implementations should instead use PutPayload::stream or PutPayload::bytes.
Changing the existing synchronous APIs to return Option or Result would avoid the panic but would be a breaking public API change. Returning an empty iterator for streaming payloads was rejected because it could silently result in incomplete uploads.
Feedback from maintainers on the preferred long-term API shape would be appreciated. Possible alternatives include: