Fix: stream chunk uploads from disk to avoid OOM on large files#5
Open
mamont wants to merge 1 commit into
Open
Fix: stream chunk uploads from disk to avoid OOM on large files#5mamont wants to merge 1 commit into
mamont wants to merge 1 commit into
Conversation
Both upload paths read each part fully into memory before PUTting it: Android allocated ByteArray(size) in readChunk, and iOS read the whole chunk into Data before writing the temp file. With large chunk sizes and several parts prepared in parallel, peak memory scaled with chunkSize x parallelism and reliably got OOM-killed on lower-heap devices. Replace both with a fixed 256 KB streaming copy: - Android: a custom RequestBody (streamingFileChunkBody) seeks to the part offset and streams through a reusable 256 KB buffer. OkHttp may invoke writeTo() more than once on retry, so the file is reopened per call. Content-Length is now derived from contentLength(). - iOS: streamChunkToFile copies the part into the temp file through a 256 KB buffer instead of buffering the whole chunk in Data. Both fail fast on a short read so a truncated part can't corrupt the S3 multipart upload. Per-part memory is now the buffer size regardless of chunk size. Fix linter warnings
mamont
force-pushed
the
fix/stream-chunks-to-avoid-oom
branch
from
09e1ced to
93f1c13
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes high memory usage/OOM during multipart uploads by streaming chunk data instead of loading entire chunks into memory.
Changes
ByteArraychunk allocation with a streaming OkHttpRequestBody.Datachunk allocation with buffered copy to the temporary upload file.Validation
yarn typecheckyarn lintyarn testyarn example androidyarn example ios