Skip to content

chore(sdk): rewrite CreateTDF on top of the chunked writer - #3946

Open
dmihalcik-virtru wants to merge 1 commit into
dspx-2604-base-17from
dspx-2604-17-createtdf-delegates
Open

dmihalcik-virtru wants to merge 1 commit into
dspx-2604-base-17from
dspx-2604-17-createtdf-delegates

Conversation

@dmihalcik-virtru

@dmihalcik-virtru dmihalcik-virtru commented Sep 1, 2026

Copy link
Copy Markdown
Member

Part 17 of 20 in the DSPX-2604 re-cut. Base branch: dspx-2604-base-17.

This stack replaces #3782 / #3865 / #3921, which stay open and untouched
until it lands. Nothing here is a rebase of those branches — the work was
re-cut from the ticket so each PR stands on its own.

Proposed Changes

CreateTDFContext and the chunked writer had grown two full implementations of
the same thing: build a policy, split the DEK, wrap each share to its KAS,
encrypt segments, accumulate an aggregate hash, sign assertions, emit a
manifest. Two copies of TDF construction is one too many — every spec change
has to land twice, and the second copy is the one that gets forgotten.

CreateTDF now delegates. It keeps the parts that are genuinely its own — the
KAO template, autoconfigure, the input-size resolution and read loop from the
previous commit — and hands each segment to the chunked writer, which owns
manifest assembly for both paths from here on.

The two paths differ in when key access is resolved, so that is what the
writer is now parameterized on. The chunked writer defers to a KeySplitter at
Finalize, because a caller may still be adding attributes while segments are
in flight. CreateTDF cannot: it knows its attributes up front and wants an
unreachable KAS to fail the call before a single payload byte reaches the
output writer. Both are expressed as a keyAccessResolver, with the DEK now
injectable so CreateTDF can wrap it ahead of time and hand the writer a
staticKeyAccess.

Everything downstream of that split is shared: resolvePolicyAndKeyAccess and
buildKeyAccessObjects replace prepareManifest's inline loop and the chunked
writer's buildChunkedPolicy/buildChunkedKeyAccessObjects, so both paths now
emit byte-identical policy and key access for the same attributes.

Two things fall out of the unification, both moving the chunked writer onto
the shipped classic behavior:

  • With zero attributes the policy body's "dataAttributes" and "dissem" are
    now null rather than []. createPolicyObjectFromFQNs initializes them
    inside the attribute loop; the deleted buildChunkedPolicy did so
    unconditionally. The classic path has always emitted null here.

  • A KAS named by a split but missing a public key is still rejected rather
    than skipped (the check moved into buildKeyAccessObjects), but the error
    now names the missing PEM rather than the absent map entry — the two cases
    were indistinguishable in practice and only the outcome matters.

The writer also gained an explicit segment size. It previously reported the
first segment's actual length as defaultSegmentSize, which is only correct
when every segment is full; a single-segment TDF would advertise a short
default. CreateTDF knows the configured size and now says so.

TDFObject loses aesGcm and payloadKey, which only ever existed to carry state
between prepareManifest and the encrypt loop.

Since #3940 (revised) fixed the chunked writer's root to HS256 and its
segments to GMAC, there is nothing to plumb through here: CreateTDF stops
passing TDFConfig.rootIntegrityAlg and segmentIntegrityAlg to the writer.
Those were already the only values the defaults could hold and no exported
option set either, so the manifest is unchanged.

Deliberately not in this commit: removing enableEncryption, tdfFormat,
readActionName, and the two now-unread integrity fields, which are dead but
unrelated; they are a separate cleanup (#3947).

Testing: the existing TDFSuite round trips pin byte-level output across
segment sizes, target modes, and multi-KAS splits, and pass unchanged. Also
verified against the streaming-input and input-size
coverage added in the previous commit, the experimental chunked writer suite,
and cross-module builds of examples, otdfctl, service, and tests-bdd.

Checklist

  • I have added or updated unit tests
  • I have added or updated integration tests (if appropriate)
  • I have added or updated documentation

Testing Instructions

cd sdk && go test ./... -race

The existing TDFSuite round trips pin byte-level output across segment
sizes, target modes and multi-KAS splits, and pass unchanged — that is the
main assurance here.

This touches key access construction on the shipped path, so it wants a
cross-SDK run before merge. This is the branch to pin it to for the whole
writer-delegation half of the stack: xtest drives the Go side through
otdfctlSDK.CreateTDF, and this is the first commit where that call
reaches the chunked writer at all.

gh workflow run xtest.yml --repo opentdf/tests --ref main \
  -f platform-ref=dspx-2604-17-createtdf-delegates \
  -f otdfctl-ref=dspx-2604-17-createtdf-delegates \
  -f java-ref=main -f js-ref=main

otdfctl-ref must name the branch, not main — otherwise the CLI is built
against main's sdk/ and the run passes without executing any of this. The
job label should read go@dspx-2604-17-createtdf-delegates.

The full DSPX-2604 stack — 20 PRs
# PR Based on
01 #3930 chore: bump go.work toolchain to go1.25.12 and simplify an rt_test condition main
02 #3931 feat(sdk): make the zipstream clock injectable for deterministic ZIP output main
03 #3932 fix(sdk): reject a zipstream write set that omits segment 0 #3931
04 #3933 fix(sdk): map ReadAt plaintext offsets from cumulative segment sizes main
05 #3934 chore(sdk): extract integrityAlgorithmString, createPolicyBinding, signAssertions main
06 #3935 chore(sdk): add direct tests for createKeyAccess, encryptMetadata and tdfSalt main
07 #3936 fix(sdk): fill each segment with io.ReadFull and size the buffer to the input main
08 #3937 chore(cli): move streaming IO helpers into pkg main
09 #3938 fix(cli): stream encrypt instead of buffering the whole payload #3937
10 #3939 fix(cli): stream decrypt and inspect instead of buffering #3938
11 #3940 feat(sdk): add a chunked segment writer (experimental) dspx-2604-base-11 = #3932 + #3934 + #3935
12 #3941 fix(sdk): stop GetManifest from splitting the key under the lock #3940
13 #3942 fix(sdk): reject a chunked split naming a KAS with no resolved public key #3941
14 #3943 chore(sdk): alias experimental/tdf manifest and assertion types #3942
15 #3944 fix(sdk): emit spec-compliant key access in experimental/tdf and delegate Writer #3943
16 #3945 feat(sdk): accept io.Reader in CreateTDF and drop the 64 GB payload cap #3936
17 #3946 chore(sdk): rewrite CreateTDF on top of the chunked writer dspx-2604-base-17 = #3944 + #3945
18 #3947 chore(sdk): drop dead TDFConfig fields and deprecate the TDFFormat enum #3946
19 #3948 fix(cli): drop the encrypt-side stdin spool dspx-2604-base-19 = #3947 + #3939
20 #3949 feat(sdk): graduate the chunked writer to stable API #3948

Reviewable in parallel right now, since they sit directly on main and depend on
nothing else: 01, 02, 04, 05, 06, 07, 08.

Why three PRs have a dspx-2604-base-* base. A GitHub PR takes one base branch,
but 11, 17 and 19 each build on more than one parent. The base-* branches are empty
merge commits that exist only to join those parents so the PR diff shows exactly its
own change and nothing else. They contain no code, have no PR of their own, and go
away once their parents land — retarget the child onto main at that point.

Wants a cross-SDK xtest run before merge: 15, 17 (and therefore 20). They touch
the KAS wire format. Dispatch it against 17 or 20, never 15 on its own: xtest drives
the Go side through otdfctl -> SDK.CreateTDF, and 17 is the first commit where
that call reaches the rewritten writer. Set otdfctl-ref to the same branch as
platform-ref -- it defaults to main, which builds the CLI against main's sdk/
and makes the run vacuous.

Red checks you may see are network flakes, not this stack. Four distinct ones hit
this batch and all clear on re-run: golangci-lint config verify timing out on
https://golangci-lint.run/.../golangci.v2.8.jsonschema.json (fails the whole go (<module>) job and fail-fast cancels its siblings), the bats installer getting a 403,
Docker Hub timing out on keycloak/keycloak:26.4, and buf reporting "the server
hosted at that remote is unavailable" while the Java SDK generates sources. The
govulncheck step also emits ##[error] annotations against the go1.25.11 stdlib, but
it is continue-on-error: true and never fails a job — 01 bumps the toolchain and
clears those annotations.

@dmihalcik-virtru
dmihalcik-virtru requested review from a team as code owners September 1, 2026 02:57
@coderabbitai

coderabbitai Bot commented Sep 1, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 000b96bb-7321-48ac-85f9-8f0a96242887

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added comp:sdk A software development kit, including library, for client applications and inter-service communicati size/l labels Sep 1, 2026
@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor
Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 260.573394ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 135.553264ms

Benchmark Statistics

Name № Requests Avg Duration Min Duration Max Duration

Bulk Benchmark Results

Metric Value
Total Decrypts 100
Successful Decrypts 100
Failed Decrypts 0
Total Time 417.368815ms
Throughput 239.60 requests/second

TDF3 Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 43.737562519s
Average Latency 436.48954ms
Throughput 114.32 requests/second

@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

X-Test Failure Report

@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor
Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 275.391918ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 139.401877ms

Benchmark Statistics

Name № Requests Avg Duration Min Duration Max Duration

Bulk Benchmark Results

Metric Value
Total Decrypts 100
Successful Decrypts 100
Failed Decrypts 0
Total Time 443.371855ms
Throughput 225.54 requests/second

TDF3 Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 44.279606203s
Average Latency 442.099151ms
Throughput 112.92 requests/second

@github-actions

Copy link
Copy Markdown
Contributor
Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 234.93444ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 144.427265ms

Benchmark Statistics

Name № Requests Avg Duration Min Duration Max Duration

Bulk Benchmark Results

Metric Value
Total Decrypts 100
Successful Decrypts 100
Failed Decrypts 0
Total Time 413.909931ms
Throughput 241.60 requests/second

TDF3 Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 1m0.936519293s
Average Latency 607.939701ms
Throughput 82.05 requests/second

@github-actions

Copy link
Copy Markdown
Contributor
Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 263.815863ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 144.745371ms

Benchmark Statistics

Name № Requests Avg Duration Min Duration Max Duration

Bulk Benchmark Results

Metric Value
Total Decrypts 100
Successful Decrypts 100
Failed Decrypts 0
Total Time 420.373722ms
Throughput 237.88 requests/second

TDF3 Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 59.777998055s
Average Latency 596.492681ms
Throughput 83.64 requests/second

@dmihalcik-virtru
dmihalcik-virtru force-pushed the dspx-2604-17-createtdf-delegates branch from befe24c to a7fe096 Compare September 15, 2026 23:57
@github-actions

Copy link
Copy Markdown
Contributor
Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 185.082817ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 125.499241ms

Benchmark Statistics

Name № Requests Avg Duration Min Duration Max Duration

Bulk Benchmark Results

Metric Value
Total Decrypts 100
Successful Decrypts 100
Failed Decrypts 0
Total Time 397.576672ms
Throughput 251.52 requests/second

TDF3 Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 47.415822831s
Average Latency 473.186341ms
Throughput 105.45 requests/second

@github-actions

Copy link
Copy Markdown
Contributor
Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 149.054023ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 86.311093ms

Benchmark Statistics

Name № Requests Avg Duration Min Duration Max Duration

Bulk Benchmark Results

Metric Value
Total Decrypts 100
Successful Decrypts 100
Failed Decrypts 0
Total Time 285.496185ms
Throughput 350.27 requests/second

TDF3 Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 39.707009248s
Average Latency 396.152306ms
Throughput 125.92 requests/second

@github-actions

Copy link
Copy Markdown
Contributor
Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 152.249735ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 82.041472ms

Benchmark Statistics

Name № Requests Avg Duration Min Duration Max Duration

Bulk Benchmark Results

Metric Value
Total Decrypts 100
Successful Decrypts 100
Failed Decrypts 0
Total Time 275.810075ms
Throughput 362.57 requests/second

TDF3 Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 37.228174134s
Average Latency 371.507677ms
Throughput 134.31 requests/second

@github-actions

Copy link
Copy Markdown
Contributor

X-Test Failure Report

@github-actions

Copy link
Copy Markdown
Contributor

X-Test Failure Report

@github-actions

Copy link
Copy Markdown
Contributor
Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 248.33506ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 139.331593ms

Benchmark Statistics

Name № Requests Avg Duration Min Duration Max Duration

Bulk Benchmark Results

Metric Value
Total Decrypts 100
Successful Decrypts 100
Failed Decrypts 0
Total Time 411.407024ms
Throughput 243.07 requests/second

TDF3 Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 58.874616647s
Average Latency 586.871067ms
Throughput 84.93 requests/second

@github-actions

Copy link
Copy Markdown
Contributor
Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 250.048851ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 133.210824ms

Benchmark Statistics

Name № Requests Avg Duration Min Duration Max Duration

Bulk Benchmark Results

Metric Value
Total Decrypts 100
Successful Decrypts 100
Failed Decrypts 0
Total Time 433.798824ms
Throughput 230.52 requests/second

TDF3 Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 59.671460328s
Average Latency 595.248665ms
Throughput 83.79 requests/second

CreateTDFContext and the chunked writer had grown two full implementations of
the same thing: build a policy, split the DEK, wrap each share to its KAS,
encrypt segments, accumulate an aggregate hash, sign assertions, emit a
manifest. Two copies of TDF construction is one too many — every spec change
has to land twice, and the second copy is the one that gets forgotten.

CreateTDF now delegates. It keeps the parts that are genuinely its own — the
KAO template, autoconfigure, the input-size resolution and read loop from the
previous commit — and hands each segment to the chunked writer, which owns
manifest assembly for both paths from here on.

The two paths differ in when key access is resolved, so that is what the
writer is now parameterized on. The chunked writer defers to a KeySplitter at
Finalize, because a caller may still be adding attributes while segments are
in flight. CreateTDF cannot: it knows its attributes up front and wants an
unreachable KAS to fail the call before a single payload byte reaches the
output writer. Both are expressed as a keyAccessResolver, with the DEK now
injectable so CreateTDF can wrap it ahead of time and hand the writer a
staticKeyAccess.

Everything downstream of that split is shared: resolvePolicyAndKeyAccess and
buildKeyAccessObjects replace prepareManifest's inline loop and the chunked
writer's buildChunkedPolicy/buildChunkedKeyAccessObjects, so both paths now
emit byte-identical policy and key access for the same attributes. The
splitter-contract checks move with them into splitterKeyAccess.resolve, which
is now the single place a third-party KeySplitter's output is validated and
verified to XOR back to the DEK before anything is wrapped.

Two things fall out of the unification, both moving the chunked writer onto
the shipped classic behavior:

  - With zero attributes the policy body's "dataAttributes" and "dissem" are
    now null rather than []. createPolicyObjectFromFQNs initializes them
    inside the attribute loop; the deleted buildChunkedPolicy did so
    unconditionally. The classic path has always emitted null here.

  - A KAS named by a split but missing a public key is still rejected rather
    than skipped (the check moved into buildKeyAccessObjects), but the error
    now names the missing PEM rather than the absent map entry — the two cases
    were indistinguishable in practice and only the outcome matters.

The writer also gained an explicit segment size. It previously reported the
first segment's actual length as defaultSegmentSize, which is only correct
when every segment is full; a single-segment TDF would advertise a short
default. CreateTDF knows the configured size and now says so.

TDFObject loses aesGcm and payloadKey, which only ever existed to carry state
between prepareManifest and the encrypt loop.

Deliberately not in this commit: removing enableEncryption, tdfFormat, and
readActionName, which are dead but unrelated; they are a separate cleanup.

Testing: the existing TDFSuite round trips pin byte-level output across
integrity algorithms, segment sizes, target modes, and multi-KAS splits, and
pass unchanged. Also verified against the streaming-input and input-size
coverage added in the previous commit, the experimental chunked writer suite,
and cross-module builds of examples, otdfctl, service, and tests-bdd.

Signed-off-by: Dave Mihalcik <dmihalcik@virtru.com>
@github-actions

Copy link
Copy Markdown
Contributor

X-Test Failure Report

@github-actions

Copy link
Copy Markdown
Contributor
Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 230.854193ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 137.554461ms

Benchmark Statistics

Name № Requests Avg Duration Min Duration Max Duration

Bulk Benchmark Results

Metric Value
Total Decrypts 100
Successful Decrypts 100
Failed Decrypts 0
Total Time 433.277731ms
Throughput 230.80 requests/second

TDF3 Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 58.976073606s
Average Latency 588.692831ms
Throughput 84.78 requests/second

@github-actions

Copy link
Copy Markdown
Contributor
Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 239.782989ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 126.946176ms

Benchmark Statistics

Name № Requests Avg Duration Min Duration Max Duration

Bulk Benchmark Results

Metric Value
Total Decrypts 100
Successful Decrypts 100
Failed Decrypts 0
Total Time 448.890853ms
Throughput 222.77 requests/second

TDF3 Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 56.375727363s
Average Latency 562.363283ms
Throughput 88.69 requests/second

@github-actions

Copy link
Copy Markdown
Contributor

⚠️ Govulncheck found vulnerabilities ⚠️

The following modules have known vulnerabilities:

  • otdfctl
  • service
  • tests-bdd

See the workflow run for details.

@github-actions

Copy link
Copy Markdown
Contributor

⚠️ Govulncheck found vulnerabilities ⚠️

The following modules have known vulnerabilities:

  • otdfctl
  • service
  • tests-bdd

See the workflow run for details.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp:sdk A software development kit, including library, for client applications and inter-service communicati size/l

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant