Repository navigation
feat(sdk): accept io.Reader in CreateTDF and drop the 64 GB payload cap [backport to release/otdfctl/v0.38] - #4114
Merged
elizabethhealy merged 1 commit intoSep 28, 2026
Conversation
1 of 3 tasks
…ap (#3945) > **Part 16 of 20** in the DSPX-2604 re-cut. Base branch: `dspx-2604-07-readfull`. > > This stack replaces #3782 / #3865 / #3921, which stay open and untouched > until it lands. Nothing here is a rebase of those branches — the work was > re-cut from the ticket so each PR stands on its own. ### Proposed Changes CreateTDF and CreateTDFContext took an io.ReadSeeker, so a caller with a pipe, a socket, or any other one-pass source had to spool the whole payload to disk or memory first. That is the block DSPX-2604 exists to remove: the Everfox re-wrap pipeline hands us a stream it cannot rewind. Both now take an io.Reader and consume it from its current position through EOF. Seekability was only ever used to measure the input. The length still matters, but it is now resolved rather than required: - WithInputSize(n) declares it outright, for a reader that cannot report it; - failing that, a reader that happens to implement io.Seeker is probed, and the cursor restored to wherever it was; - failing both, the payload is unmeasurable and is read until it ends. The one thing an unmeasurable payload gives up is the compact ZIP32 layout. The ZIP64 decision is baked into the payload's local file header, which is emitted ahead of the first segment, so it cannot be revisited once the archive has started; a payload that might exceed a 32-bit offset has to be written as ZIP64 from the outset. WithInputSize exists to buy that back — declaring the length of a piped payload keeps it in ZIP32 when it fits. The read loop no longer computes a segment count up front. It reads a buffer at a time until EOF, which is what makes an unknown length workable, and happens to be the same code path for a short final segment. An empty payload still produces one empty segment. The segment count is still passed to the archive writer when it is known, because that is what keeps a large declared count from being clamped to a one-segment capacity hint. Three behavior changes worth calling out. The first is the only one that can silently shorten a payload; the other two fail loudly. - A seekable reader is no longer rewound. The old code seeked to the end to measure and then back to byte 0, so it encrypted the whole file no matter where the caller had left the cursor. resolveInputSize saves the current position and restores that one, so the payload is whatever remains from there. Concretely: a caller that sniffs the first 512 bytes for content-type detection and then hands the same *os.File to CreateTDF used to get the whole file and now gets the file minus 512 bytes — no compile error, no runtime error, just a shorter TDF that looks complete. Reading from the current position is the right semantics for an io.Reader-shaped API, and `a seekable reader is encrypted from its current position` pins it, but a caller relying on the rewind has to seek to 0 itself now. - The 64 GB cap (maxFileSizeSupported/errFileTooLarge) is gone. It could only ever be enforced on a measurable payload, so keeping it would have meant `encrypt bigfile` failing where `encrypt < bigfile` succeeded. Both were unexported; nothing outside the package referenced them. - A declared size is exact, not an upper bound. A reader that reaches EOF early now fails the call with errInputShorterThanDeclared instead of returning a TDF that is silently short of the payload the caller asked to encrypt. Reading still stops at the declared size if the reader has more. What bounds a payload now, with the cap gone. maxPayloadSegments refuses a size needing more than MaxInt32 segments, but segmentCount is the only place it is enforced and that runs only when the length resolves — an unmeasurable stream has no segment ceiling. Underneath that, the real limit is manifest memory: one manifest segment and one archive-writer entry per segment, all live until finalize, roughly 500k entries per terabyte at the 2 MiB default. Neither is reachable today — 4 PiB at that segment size — so this is a note about where the wall moved to, not a regression. DSPX-4905 tracks bounding and measuring both. Testing: Test_CreateTDF_StreamingInput covers the three measurement modes across empty, sub-segment, exact-multiple, and partial-final-segment payloads, asserting the ZIP64 choice, the segment count, and a full round trip through LoadTDF. Test_CreateTDF_InputSizeBounds covers the negative, over-long, short, and mid-stream-start cases. Both guards were mutation-checked: removing the io.LimitReader fails "declared size bounds the read", and dropping the unknown-size ZIP64 rule fails every unmeasurable case. ### Checklist - [x] I have added or updated unit tests - [ ] I have added or updated integration tests (if appropriate) - [ ] I have added or updated documentation ### Testing Instructions ``` cd sdk && go test ./... -race ``` `Test_CreateTDF_StreamingInput` and `Test_CreateTDF_InputSizeBounds` are the new coverage. Both guards were mutation-checked: removing the `io.LimitReader` fails "declared size bounds the read", and dropping the unknown-size ZIP64 rule fails every unmeasurable case. <details> <summary><b>The full DSPX-2604 stack — 20 PRs</b></summary> | # | PR | Based on | |---|----|----------| | 01 | #3930 chore: bump go.work toolchain to go1.25.12 and simplify an rt_test condition | `main` | | 02 | #3931 feat(sdk): make the zipstream clock injectable for deterministic ZIP output | `main` | | 03 | #3932 fix(sdk): reject a zipstream write set that omits segment 0 | #3931 | | 04 | #3933 fix(sdk): map ReadAt plaintext offsets from cumulative segment sizes | `main` | | 05 | #3934 chore(sdk): extract integrityAlgorithmString, createPolicyBinding, signAssertions | `main` | | 06 | #3935 chore(sdk): add direct tests for createKeyAccess, encryptMetadata and tdfSalt | `main` | | 07 | #3936 fix(sdk): fill each segment with io.ReadFull and size the buffer to the input | `main` | | 08 | #3937 chore(cli): move streaming IO helpers into pkg | `main` | | 09 | #3938 fix(cli): stream encrypt instead of buffering the whole payload | #3937 | | 10 | #3939 fix(cli): stream decrypt and inspect instead of buffering | #3938 | | 11 | #3940 feat(sdk): add a chunked segment writer (experimental) | `dspx-2604-base-11` = #3932 + #3934 + #3935 | | 12 | #3941 fix(sdk): stop GetManifest from splitting the key under the lock | #3940 | | 13 | #3942 fix(sdk): reject a chunked split naming a KAS with no resolved public key | #3941 | | 14 | #3943 chore(sdk): alias experimental/tdf manifest and assertion types | #3942 | | 15 | #3944 fix(sdk): emit spec-compliant key access in experimental/tdf and delegate Writer | #3943 | | 16 | #3945 feat(sdk): accept io.Reader in CreateTDF and drop the 64 GB payload cap | #3936 | | 17 | #3946 chore(sdk): rewrite CreateTDF on top of the chunked writer | `dspx-2604-base-17` = #3944 + #3945 | | 18 | #3947 chore(sdk): drop dead TDFConfig fields and deprecate the TDFFormat enum | #3946 | | 19 | #3948 fix(cli): drop the encrypt-side stdin spool | `dspx-2604-base-19` = #3947 + #3939 | | 20 | #3949 feat(sdk): graduate the chunked writer to stable API | #3948 | **Reviewable in parallel right now**, since they sit directly on `main` and depend on nothing else: 01, 02, 04, 05, 06, 07, 08. **Why three PRs have a `dspx-2604-base-*` base.** A GitHub PR takes one base branch, but 11, 17 and 19 each build on more than one parent. The `base-*` branches are empty merge commits that exist only to join those parents so the PR diff shows exactly its own change and nothing else. They contain no code, have no PR of their own, and go away once their parents land — retarget the child onto `main` at that point. **Wants a cross-SDK xtest run before merge:** 15, 17 (and therefore 20). They touch the KAS wire format. **Red checks you may see are network flakes, not this stack.** Four distinct ones hit this batch and all clear on re-run: `golangci-lint config verify` timing out on `https://golangci-lint.run/.../golangci.v2.8.jsonschema.json` (fails the whole `go (<module>)` job and fail-fast cancels its siblings), the bats installer getting a 403, Docker Hub timing out on `keycloak/keycloak:26.4`, and `buf` reporting "the server hosted at that remote is unavailable" while the Java SDK generates sources. The `govulncheck` step also emits `##[error]` annotations against the go1.25.11 stdlib, but it is `continue-on-error: true` and never fails a job — 01 bumps the toolchain and clears those annotations. </details> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * TDF creation now supports streaming inputs, including non-seekable readers such as pipes and network streams. * Added `WithInputSize` to declare the exact payload size when it cannot be determined automatically. * Payloads are processed in segments, supporting unknown sizes and ZIP64 archives when needed. * **Bug Fixes** * Invalid or inaccurate declared sizes now result in clear errors, including when input ends too early. * Payloads exceeding the supported segment limit are rejected. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Signed-off-by: Dave Mihalcik <dmihalcik@virtru.com> (cherry picked from commit 20ff505)
opentdf-automation
Bot
force-pushed
the
backport-3945-to-release/otdfctl/v0.38
branch
from
September 28, 2026 17:47
b8fe16e to
7d2bb3c
Compare
|
Important Review skippedBot user detected. To trigger a single review, invoke the ⚙️ Run configurationConfiguration used: Repository UI Review profile: ASSERTIVE Plan: Advanced Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
github-actions
Bot
added
comp:sdk
Contributor
X-Test Failure Report |
Contributor
Benchmark results, click to expandBenchmark authorization.GetDecisions Results:
Benchmark authorization.v2.GetMultiResourceDecision Results:
Benchmark Statistics
Bulk Benchmark Results
TDF3 Benchmark Results:
|
Contributor
Benchmark results, click to expandBenchmark authorization.GetDecisions Results:
Benchmark authorization.v2.GetMultiResourceDecision Results:
Benchmark Statistics
Bulk Benchmark Results
TDF3 Benchmark Results:
|
Contributor
|
4 tasks
Contributor
elizabethhealy
approved these changes
Sep 28, 2026
opentdf-automation Bot
added a commit
that referenced
this pull request
Sep 28, 2026
## Summary `release-otdfctl.yaml` runs `make build` with the root `go.work`, so the published otdfctl binaries are compiled against the **in-repo** `sdk/`, `protocol/go`, `lib/*` instead of the versions pinned in `otdfctl/go.mod`. That's inconsistent with: - the release-please PR check, which runs `.github/scripts/work-init.sh` to drop `./sdk` from the workspace and validate against the pinned versions - what users get from `go install github.com/opentdf/platform/otdfctl@vX` On release branches this can break a release after it's published: e.g. on `release/otdfctl/v0.38`, otdfctl pins `sdk v0.33.0` (which includes #3945), but the in-tree `sdk/` did not until #4114. The release-please check would pass while the post-publish binary build would fail to compile, leaving a release with no artifacts. This sets `GOWORK=off` on the build step so binaries are built strictly from `otdfctl/go.mod`. `setup-go` still reads the Go version from `go.work`. ## Test plan - [ ] actionlint passes - [ ] Locally: `cd otdfctl && GOWORK=off make build` succeeds on `main` - [ ] Next `otdfctl/v*` release uploads binaries successfully - [ ] Consider backporting to active `release/otdfctl/*` branches <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Updated the release build to use dependencies specified for `otdfctl`. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Co-authored-by: CoopAgent <coopagent@users.noreply.github.com> (cherry picked from commit e8a2f7f)
pflynn-virtru
pushed a commit
to pflynn-virtru/platform
that referenced
this pull request
Sep 30, 2026
## Summary `release-otdfctl.yaml` runs `make build` with the root `go.work`, so the published otdfctl binaries are compiled against the **in-repo** `sdk/`, `protocol/go`, `lib/*` instead of the versions pinned in `otdfctl/go.mod`. That's inconsistent with: - the release-please PR check, which runs `.github/scripts/work-init.sh` to drop `./sdk` from the workspace and validate against the pinned versions - what users get from `go install github.com/opentdf/platform/otdfctl@vX` On release branches this can break a release after it's published: e.g. on `release/otdfctl/v0.38`, otdfctl pins `sdk v0.33.0` (which includes opentdf#3945), but the in-tree `sdk/` did not until opentdf#4114. The release-please check would pass while the post-publish binary build would fail to compile, leaving a release with no artifacts. This sets `GOWORK=off` on the build step so binaries are built strictly from `otdfctl/go.mod`. `setup-go` still reads the Go version from `go.work`. ## Test plan - [ ] actionlint passes - [ ] Locally: `cd otdfctl && GOWORK=off make build` succeeds on `main` - [ ] Next `otdfctl/v*` release uploads binaries successfully - [ ] Consider backporting to active `release/otdfctl/*` branches <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Updated the release build to use dependencies specified for `otdfctl`. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Co-authored-by: CoopAgent <coopagent@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Backport of #3945 to
release/otdfctl/v0.38.